Numpy 2 compatibility

[matbryan52]

Looks like we are OK for numpy 2 here, just hdbscan which is not yet compatible. Their new version 0.8.37 released a few days ago pins numpy < 2, so new installs will work. Older installs that are upgraded will likely crash with the ValueError for the dtype size change on the import of hdbscan.

See scikit-learn-contrib/hdbscan#642

We might also want to checkout fast_hdbscan mentioned in that issue: https://github.com/TutteInstitute/fast_hdbscan

[sk1p]

We might also want to checkout fast_hdbscan mentioned in that issue: https://github.com/TutteInstitute/fast_hdbscan

I didn't look in detail yet, but the description sounds good - we already depend on numba, so installation (and maintenance on their side) should be a lot less painful. The better performance is then just the cherry on top.

[matbryan52]

We might also want to checkout fast_hdbscan mentioned in that issue: https://github.com/TutteInstitute/fast_hdbscan

I didn't look in detail yet, but the description sounds good - we already depend on numba, so installation (and maintenance on their side) should be a lot less painful. The better performance is then just the cherry on top.

In the end not such a simple change, yet.

Looks like fast_hdbscan is not quite yet numpy2 compatible (use of np.bool8 at least). Dropping to older numpy does mean that tests pass with fast_hdbscan, which is a good sign, but the import time is huge (10+ seconds on ptycho), I think because they run a fit to warmup numba at import : https://github.com/TutteInstitute/fast_hdbscan/blob/main/fast_hdbscan/__init__.py

Might want to wait and see, for now.

[sk1p]

For reference, this is the numpy2 related issue: TutteInstitute/fast_hdbscan#17 (needs a release)
And this is the slow import issue: TutteInstitute/fast_hdbscan#26

[sk1p]

fast-hdbscan has released a new version in pypi some time ago, but the at least the slow import is still an issue. The latest release of the original HDBSCAN should now support numpy 2.x though.

Is there something we need to do on our side? Should we require numpy 2.x and a recent hdbscan to ensure things match?