Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip dispatching to GPU for unimplemented metrics in UMAP #6224

Open
wants to merge 4 commits into
base: branch-25.02
Choose a base branch
from
+43 −4
Open

Skip dispatching to GPU for unimplemented metrics in UMAP #6224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2788304
Skip dispatching to GPU for unimplemented metrics
betatim Jan 14, 2025
72d7e33
Update python/cuml/cuml/tests/experimental/accel/estimators_hyperpara…
betatim Jan 15, 2025
3f90e99
Typo fix
betatim Jan 15, 2025
18dc640
Make conda quieter
betatim Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ci/test_python_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2022-2024, NVIDIA CORPORATION.
# Copyright (c) 2022-2025, NVIDIA CORPORATION.

set -euo pipefail

Expand All @@ -17,7 +17,7 @@ rapids-dependency-file-generator \
--prepend-channel "${CPP_CHANNEL}" \
--prepend-channel "${PYTHON_CHANNEL}" | tee env.yaml

rapids-mamba-retry env create --yes -f env.yaml -n test
rapids-mamba-retry env create --quiet --yes -f env.yaml -n test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the goal of adding this --quiet?

For context... we'd recently tried to do that globally in CI images for RAPIDS, but found that it seemed to have the side effect of suppressing exception names from mamba, making rapids-mamba-retry ineffective:

rapidsai/ci-imgs#220

rapids-mamba-retry / rapids-conda-retry work by string-matching on Python exceptions:

https://github.com/rapidsai/gha-tools/blob/0558ffce255e4e7da5d5312e79f35dd81e444144/tools/rapids-conda-retry#L82

So adding --quiet here might mean you're trading quieter logs for more need to manually retry failures from conda.

Copy link
Member Author

@betatim betatim Jan 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly added it here to reduce the number of lines I have to scroll past to see the output from the pytest command. It is somewhat annoying that the progress bars seem to take up many many many many lines :-/ but yeah, not really interested in negotiating with the rest of RAPIDS about this change (I'll revert it when I'm done with this PR) :D

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to reduce the number of lines I have to scroll past to see the output from the pytest command

I agree, it's annoying :/

That's why we'd attempted to make quiet: true the global setting for RAPIDS CI: rapidsai/ci-imgs#217

But breaking the retry mechanism was just not worth it... I think having to scroll past some logs lines is better than having to manually re-run CI.

Anyway, the work to actually find the root cause of these excessive empty lines is still something we should do. Put up rapidsai/ci-imgs#228 to track that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed that it is something we'd have to fix in conda/mamba? Somehow making it aware of the fact that no human is watching so that it can either not output a progress bar or some such. At least it seems like there are CLI tools out there that somehow adjust the fancyness of their output. Alas, I have no idea how they do it :(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, it might also be a side effect of the RAPIDS-specific wrappers we have around those tools. Put up one idea at rapidsai/ci-imgs#228 (comment)

Anyway, that ci-imgs issue is a good tracking issue for this. Hopefully we can get more specific reproducible examples and make some progress there. I'll try to look into it when I can.


# Temporarily allow unbound variables for conda activation.
set +u
Expand Down
16 changes: 16 additions & 0 deletions python/cuml/cuml/manifold/umap.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,22 @@ class UMAP(UniversalBase,
_cpu_estimator_import_path = 'umap.UMAP'
embedding_ = CumlArrayDescriptor(order='C')

_hyperparam_interop_translator = {
"metric": {
"sokalsneath": "NotImplemented",
"rogerstanimoto": "NotImplemented",
"sokalmichener": "NotImplemented",
"yule": "NotImplemented",
"ll_dirichlet": "NotImplemented",
"russellrao": "NotImplemented",
"kulsinski": "NotImplemented",
"dice": "NotImplemented",
"wminkowski": "NotImplemented",
"mahalanobis": "NotImplemented",
"haversine": "NotImplemented",
}
}

@device_interop_preparation
def __init__(self, *,
n_neighbors=15,
Expand Down
27 changes: 25 additions & 2 deletions python/cuml/cuml/tests/experimental/accel/estimators_hyperparams/test_accel_umap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, NVIDIA CORPORATION.
# Copyright (c) 2024-2025, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -46,10 +46,33 @@ def test_umap_min_dist(manifold_data, min_dist):


@pytest.mark.parametrize(
"metric", ["euclidean", "manhattan", "chebyshev", "cosine"]
"metric",
[
"euclidean",
"manhattan",
"chebyshev",
"cosine",
# These metrics are currently not supported in cuml,
# we test them here to make sure no exception is raised
"sokalsneath",
"rogerstanimoto",
"sokalmichener",
"yule",
"ll_dirichlet",
"russellrao",
betatim marked this conversation as resolved.
Show resolved Hide resolved
"kulsinski",
"dice",
"wminkowski",
"mahalanobis",
"haversine",
],
)
def test_umap_metric(manifold_data, metric):
X = manifold_data
# haversine only works for 2D data
if metric == "haversine":
X = X[:, :2]

umap = UMAP(metric=metric, random_state=42)
X_embedded = umap.fit_transform(X)
trust = trustworthiness(X, X_embedded, n_neighbors=5)
Expand Down
Loading