Skip dispatching to GPU for unimplemented metrics in UMAP #6224
Conversation
python/cuml/cuml/tests/experimental/accel/estimators_hyperparams/test_accel_umap.py
Show resolved
Hide resolved
…ms/test_accel_umap.py Co-authored-by: Dante Gama Dessavre <[email protected]>
|
I feel like I am missing something. The tests are still failing with |
|
The answer is that the docs call it |
betatim
changed the title
betatim
force-pushed
the
blacklist-unsupported-metrics
branch
from
df524b9 to
18dc640
Compare
There was a problem hiding this comment.
I recommend reverting that rapids-mamba-retry --quiet. I've explained the tradeoffs in a comment.
Marking this "approve" so you're not blocked waiting on ci-codeowners, and so you can merge it without further review if you all want to make a different choice than I would in the face of those tradeoffs.
| @@ -17,7 +17,7 @@ rapids-dependency-file-generator \ | |||
| --prepend-channel "${CPP_CHANNEL}" \ | |||
| --prepend-channel "${PYTHON_CHANNEL}" | tee env.yaml | |||
|
|
|||
| rapids-mamba-retry env create --yes -f env.yaml -n test | |||
| rapids-mamba-retry env create --quiet --yes -f env.yaml -n test | |||
There was a problem hiding this comment.
What is the goal of adding this --quiet?
For context... we'd recently tried to do that globally in CI images for RAPIDS, but found that it seemed to have the side effect of suppressing exception names from mamba, making rapids-mamba-retry ineffective:
rapids-mamba-retry / rapids-conda-retry work by string-matching on Python exceptions:
So adding --quiet here might mean you're trading quieter logs for more need to manually retry failures from conda.
There was a problem hiding this comment.
Mostly added it here to reduce the number of lines I have to scroll past to see the output from the pytest command. It is somewhat annoying that the progress bars seem to take up many many many many lines :-/ but yeah, not really interested in negotiating with the rest of RAPIDS about this change (I'll revert it when I'm done with this PR) :D
There was a problem hiding this comment.
to reduce the number of lines I have to scroll past to see the output from the pytest command
I agree, it's annoying :/
That's why we'd attempted to make quiet: true the global setting for RAPIDS CI: rapidsai/ci-imgs#217
But breaking the retry mechanism was just not worth it... I think having to scroll past some logs lines is better than having to manually re-run CI.
Anyway, the work to actually find the root cause of these excessive empty lines is still something we should do. Put up rapidsai/ci-imgs#228 to track that.
There was a problem hiding this comment.
I assumed that it is something we'd have to fix in conda/mamba? Somehow making it aware of the fact that no human is watching so that it can either not output a progress bar or some such. At least it seems like there are CLI tools out there that somehow adjust the fancyness of their output. Alas, I have no idea how they do it :(
There was a problem hiding this comment.
Maybe, it might also be a side effect of the RAPIDS-specific wrappers we have around those tools. Put up one idea at rapidsai/ci-imgs#228 (comment)
Anyway, that ci-imgs issue is a good tracking issue for this. Hopefully we can get more specific reproducible examples and make some progress there. I'll try to look into it when I can.
This lists more metrics that the CPU umap library supports but cuml doesn't yet support. By listing them as not implemented we don't dispatch to the GPU when a user selects them.
Also added them to a test to check that they do not raise an error. It would be nice to check that they ran on the CPU when the accelerator is enabled, but I couldn't find a nice way to do it :-/ Ideas welcome.