introduce libcugraph wheels #4804

jameslamb · 2024-12-05T16:06:17Z

Replaces #4340, contributes to rapidsai/build-planning#33.

Proposes packaging libcugraph as a wheel, which is then re-used by cugraph-cu{11,12} and pylibcugraph-cu{11,12} wheels.

Notes for Reviewers

Benefits of these changes

smaller wheels (see "Size Changes" below)
- no more pylibcugraph and cugraph both holding copies of libcugraph.so
faster compile times
- no more re-compiling RAFT, thanks to introduce libraft wheels raft#2531
- no more recompiling libcugraph.so in both pylibcugraph and cugraph wheel builds
other benefits mentioned in Support dynamic linking between RAPIDS wheels build-planning#33

Wheel contents

libcugraph:

libcugraph.so (shared library)
cuGraph headers
vendored dependencies (fmt, spdlog, CCCL, cuco)

pylibcugraph:

pylibcugraph Python / Cython code and compiled Cython extensions

cugraph:

cugraph Python / Cython code and compiled Cython extension

Dependency Flows

In short.... libcugraph contains libcugraph.so and libcugraph_c.so dynamic libraries and the headers to link against it.

Anything that needs to link against cuGraph at build time pulls in libcugraph wheels as a build dependency.
Anything that needs cuGraph's symbols at runtime pulls it in as a runtime dependency, and calls libcugraph.load_library().

For more details and some flowcharts, see rapidsai/build-planning#33 (comment)

Size changes (CUDA 12, Python 3.12, x86_64)

wheel	num files (before)	num files (this PR)	size (before)	size (this PR)
`libcugraph`	---	1762	---	903M
`pylibcugraph`	190	187	901M	2M
`cugraph`	315	313	899M	3M
TOTAL	505	2,262	1,800M	908M

NOTES: size = compressed, "before" = 2025-01-13 nightlies

This is a cuGraph-specific slice of the table from rapidsai/raft#2531. See that PR for details.

How I tested this

These other PRs:

…t working yet

bdice · 2025-01-14T20:16:56Z

ci/validate_wheel.sh

Are we going to run the size checks for libcugraph wheels for each CUDA version? I see max_allowed_size_compressed = '1.3G' in the pyproject.toml but that seems like it would be for CUDA 11?

I don't see a meaningful size difference across CUDA versions for libcugraph wheels here.

For example, look at these builds from this PR:

CUDA 11.8.0, Python 3.12, amd64 (build link)

file size * compressed size: 0.9G * uncompressed size: 1.4G

CUDA 12.5.1, Python 3.12, amd64 (build link)

file size * compressed size: 0.9G * uncompressed size: 1.4G

I think that cuGraph doesn't actually directly need any of the CUDA math libraries (cuBLAS, cuSolver, etc.)... just indirectly via RAFT needing them. We're observing this now that libcugraph wheels are linking against a pre-built RAFT delivered by the libraft wheels, instead of building RAFT from source here.

I don't see any of those libraries referenced in code here (except some cuRAND usage in C++ tests)

git grep -i blas cpp/ git grep -i blas python/ git grep -i fft cpp/ git grep -i fft python/ git grep -i solver cpp/ git grep -i solver python/ git grep -i curand cpp/ git grep -i curand python/

Does that sound plausible? If so, I'll try dropping the dependency on CUDA math wheels for libcugraph wheels here.

I agree with your assessment (confirmed independently). Let's try dropping it.

I did a bit more research to validate my understanding.

On the conda side, RAFT has no run exports that would cause a host: dependency on RAFT to add run: dependencies on the related CUDA math libraries. We do this because we don't know which math libraries would be relevant for any given RAFT consumer. However, because libraft wheels already supply those math libraries, we won't need to specify them for cuGraph. Transitive discovery is sufficient.

Ok perfect, thank you.

I've pushed commits here removing the direct dependencies on CUDA math libraries for cuGraph wheels. There's one caveat... they do need to be listed out explicitly in the --exclude args passed to auditwheel repair, or otherwise auditwheel ends up vendoring them. I've done that in ci/build_wheel_libcugraph.sh.

…ures

…graph-wheel

Replaces #2306, contributes to rapidsai/build-planning#33. Proposes packaging `libraft` as a wheel, which is then re-used by: * `pylibraft-cu{11,12}` and `raft-cu{11,12}` (this PR) * `libcugraph-cu{11,12}`, `pylibcugraph-cu{11,12}`, and `cugraph-cu{11,12}` in rapidsai/cugraph#4804 * `libcuml-cu{11,12}` and `cuml-cu{11,12}` in rapidsai/cuml#6199 As part of this, also proposes: * introducing a new CMake option, `RAFT_COMPILE_DYNAMIC_ONLY`, to allow building/installing only the dynamic shared library (i.e. skipping the static library) * enforcing `rapids-cmake`'s preferred CMake style (#2531 (comment)) * making wheel-building CI jobs always depend on other wheel-building CI jobs, not tests or `*-publish` (to reduce end-to-end CI time) ## Notes for Reviewers ### Benefits of these changes * smaller wheels (see "Size Changes" below) * faster compile times (no more re-compiling RAFT in cuGraph and cuML CI) * other benefits mentioned in rapidsai/build-planning#33 ### Wheel contents `libraft`: * `libraft.so` (shared library) * RAFT headers * vendored dependencies (`fmt`, CCCL, `cuco`, `cute`, `cutlass`) `pylibraft`: * `pylibraft` Python / Cython code and compiled Cython extensions `raft-dask`: * `raft-dask` Python / Cython code and compiled Cython extension ### Dependency Flows In short.... `libraft` contains a `libraft.so` dynamic library and the headers to link against it. * Anything that needs to link against RAFT at build time pulls in `libraft` wheels as a build dependency. * Anything that needs RAFT's symbols at runtime pulls it in as a runtime dependency, and calls `libraft.load_library()`. For more details and some flowcharts, see rapidsai/build-planning#33 (comment) ### Size changes (CUDA 12, Python 3.12, x86_64) | wheel | num files (before) | num files (these PRs) | size (before) | size (these PRs) | |:---------------:|------------------:|-----------------:|--------------:|-------------:| | `libraft`. | --- | 3169 | --- | 19M | | `pylibraft` | 64 | 63 | 11M | 1M | | `raft-dask` | 29 | 28 | 188M | 188M | | `libcugraph` | --- | 1762 | --- | 903M | | `pylibcugraph` | 190 | 187 | 901M | 2M | | `cugraph` | 315 | 313 | 899M | 3.0M | | `libcuml` | --- | 1766 | --- | 289M | | `cuml` | 442 | --- | 517M | --- | |**TOTAL** | **1,040** | **7,268** | **2,516M** | **1,405M** | *NOTES: size = compressed, "before" = 2025-01-13 nightlies* <details><summary>how I calculated those (click me)</summary> * `cugraph`: nightly commit = rapidsai/cugraph@8507cbf, PR = rapidsai/cugraph#4804 * `cuml`: nightly commit = rapidsai/cuml@7c715c4, PR = rapidsai/cuml#6199 * `raft`: nightly commit = 1b62c41, PR = this PR ```shell docker run \ --rm \ --network host \ --env RAPIDS_NIGHTLY_DATE=2025-01-13 \ --env CUGRAPH_NIGHTLY_SHA=8507cbf63db2f349136b266d3e6e787b189f45a0 \ --env CUGRAPH_PR="pull-request/4804" \ --env CUGRAPH_PR_SHA="2ef32eaa006a84c0bd16220bb8e8af34198fbee8" \ --env CUML_NIGHTLY_SHA=7c715c494dff71274d0fdec774bdee12a7e78827 \ --env CUML_PR="pull-request/6199" \ --env CUML_PR_SHA="2ef32eaa006a84c0bd16220bb8e8af34198fbee8" \ --env RAFT_NIGHTLY_SHA=1b62c4117a35b11ce3c830daae248e32ebf75e3f \ --env RAFT_PR="pull-request/2531" \ --env RAFT_PR_SHA="0d6597b08919f2aae8ac268f1a68d6a8fe5beb4e" \ --env RAPIDS_PY_CUDA_SUFFIX=cu12 \ --env WHEEL_DIR_BEFORE=/tmp/wheels-before \ --env WHEEL_DIR_AFTER=/tmp/wheels-after \ -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.12 \ bash # --- nightly wheels --- # mkdir -p ./wheels-before export RAPIDS_BUILD_TYPE=branch export RAPIDS_REF_NAME="branch-25.02" # pylibraft RAPIDS_PY_WHEEL_NAME="pylibraft_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_SHA=${RAFT_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # raft-dask RAPIDS_PY_WHEEL_NAME="raft_dask_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_SHA=${RAFT_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # cugraph RAPIDS_PY_WHEEL_NAME="cugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_SHA=${CUGRAPH_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # pylibcugraph RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_SHA=${CUGRAPH_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # cuml RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuml \ RAPIDS_SHA=${CUML_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # --- wheels from CI --- # mkdir -p ./wheels-after export RAPIDS_BUILD_TYPE="pull-request" # libraft RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_REF_NAME="${RAFT_PR}" \ RAPIDS_SHA="${RAFT_PR_SHA}" \ rapids-download-wheels-from-s3 cpp ./wheels-after # pylibraft RAPIDS_PY_WHEEL_NAME="pylibraft_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_REF_NAME="${RAFT_PR}" \ RAPIDS_SHA="${RAFT_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # raft-dask RAPIDS_PY_WHEEL_NAME="raft_dask_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_REF_NAME="${RAFT_PR}" \ RAPIDS_SHA="${RAFT_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # libcugraph RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_REF_NAME="${CUGRAPH_PR}" \ RAPIDS_SHA="${CUGRAPH_PR_SHA}" \ rapids-download-wheels-from-s3 cpp ./wheels-after # pylibcugraph RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_REF_NAME="${CUGRAPH_PR}" \ RAPIDS_SHA="${CUGRAPH_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # cugraph RAPIDS_PY_WHEEL_NAME="cugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_REF_NAME="${CUGRAPH_PR}" \ RAPIDS_SHA="${CUGRAPH_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # libcuml RAPIDS_PY_WHEEL_NAME="libcuml_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuml \ RAPIDS_REF_NAME="${CUML_PR}" \ RAPIDS_SHA="${CUML_PR_SHA}" \ rapids-download-wheels-from-s3 cpp ./wheels-after # cuml RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuml \ RAPIDS_REF_NAME="${CUML_PR}" \ RAPIDS_SHA="${CUML_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after pip install pydistcheck pydistcheck \ --inspect \ --select 'distro-too-large-compressed' \ ./wheels-before/*.whl \ | grep -E '^checking|files: | compressed' \ > ./before.txt # get more exact sizes du -sh ./wheels-before/* pydistcheck \ --inspect \ --select 'distro-too-large-compressed' \ ./wheels-after/*.whl \ | grep -E '^checking|files: | compressed' \ > ./after.txt # get more exact sizes du -sh ./wheels-after/* ``` </details> ### How I tested this These other PRs: * rapidsai/devcontainers#435 * rapidsai/cugraph-gnn#110 * rapidsai/cuml#6199 * rapidsai/cugraph#4804

bdice · 2025-01-17T00:29:29Z

I bumped this to use published libraft wheels.

jameslamb · 2025-01-17T15:03:53Z

Python tests here are failing as a result of these RMM changes: rapidsai/rmm#1775

E FutureWarning: The rmm._cuda.stream module is deprecated in 25.02 and will be removed in a future release. Use rmm.pylibrmm.stream instead.

(build link)

Think that might be coming through via cudf. It's being discussed at rapidsai/cuml#6229 (comment)

jameslamb · 2025-01-17T15:23:26Z

ci/release/update-version.sh

@@ -109,4 +110,3 @@ find .devcontainer/ -type f -name devcontainer.json -print0 | while IFS= read -r
 done

 sed_runner "s/:[0-9][0-9]\.[0-9][0-9]/:${NEXT_SHORT_TAG}/" ./notebooks/README.md
-sed_runner "s/branch-[0-9][0-9].[0-9][0-9]/branch-${NEXT_SHORT_TAG}/" ./docs/cugraph/source/nx_cugraph/nx_cugraph.md


To test the changes in this PR, I tested update-version.sh like this:

ci/release/update-version.sh '25.04.00'

That revealed that this file doesn't exist any more... because all the docs were moved over to https://github.com/rapidsai/cugraph-docs in #4837

Thought it wasn't worth a new PR and CI run just for this, so just adding it here.

bdice

Some comments to address.

.github/workflows/build.yaml

.github/workflows/pr.yaml

ci/build_wheel.sh

bdice · 2025-01-17T16:00:55Z

ci/build_wheel_libcugraph.sh

+PARALLEL_LEVEL=$(python -c \
+  "from math import ceil; from multiprocessing import cpu_count; print(ceil(cpu_count()/4))")


This is copied from pylibcugraph, so I want to question it a bit. Are we really limited to only 1/4 of the available cores? I am guessing it's due to memory requirements at compile time?

Can we try to be more aggressive in a follow-up PR? Or can we add comments to explain why this is so limited?

am guessing it's due to memory requirements at compile time?

Exactly, this was added back in #4489 (comment) to deal with out-of-memory issues building wheels.

Since most of the heavy compilation would move to libcugraph wheels in this PR, I've moved this limiting (and requesting a cpu32 node) over to libcugraph here.

HOWEVER... maybe now that these builds aren't also compiling RAFT and since #4720 reduced the number of TUs being compiled, we could get away with a higher degree of parallelism.

I'll push a change here trying to remove this limit, let's see what happens.

I just pushed a changed removing this limiting of PARALLEL_LEVEL. I left this requesting a cpu32 node for now. Let's see how it goes.

bdice · 2025-01-17T16:03:54Z

dependencies.yaml

+      - depends_on_libraft
+      - depends_on_librmm


It surprises me that we have so many C++ wheel dependencies in pylibcugraph. Does pylibcugraph directly use RAFT / RMM C++ headers? I would have expected that pylibraft / rmm would have sufficed here.

Does pylibcugraph directly use RAFT / RMM C++ headers?

pylibcugraph doesn't, but RAFT / RMM appear in libcugraph's public headers and are therefore PUBLIC dependencies in CMake. So those headers need to get into the build environment to link against libcugraph.so.

cugraph/cpp/include/cugraph/algorithms.hpp

Lines 26 to 30 in 0b50bf9

#include <raft/core/device_span.hpp>

#include <raft/core/handle.hpp>

#include <raft/random/rng_state.hpp>

#include <rmm/resource_ref.hpp>

cugraph/cpp/CMakeLists.txt

Lines 482 to 485 in 0b50bf9

target_link_libraries(cugraph

PUBLIC

rmm::rmm

raft::raft

This is related to the discussions from rapidsai/build-planning#92 , and it's for similar reasons that e.g. pylibcudf has a librmm build dependency despite not directly including any RMM headers:

https://github.com/rapidsai/cudf/blob/a4bbd0930a0e4922f69586560b064a0bd9e6aedc/python/pylibcudf/pyproject.toml#L115

Co-authored-by: Bradley Dice <[email protected]>

…libcugraph-wheel

Contributes to rapidsai/build-planning#33 Adjusts `rapids-build-utils` manifest for release 25.02 to account for the introduction of new `libcugraph` wheels (rapidsai/cugraph#4804). ## Notes for Reviewers This shouldn't be merged still pointing at my forks. Plan: 1. admin-merge rapidsai/cugraph#4804 once everything except devcontainers CI there is passing 2. point this PR at upstream `rapidsai/cugraph` 3. observe CI passing and merge this normally (or admin-merge to save time) --------- Co-authored-by: Bradley Dice <[email protected]> Co-authored-by: Paul Taylor <[email protected]>

bdice · 2025-01-18T01:08:07Z

/merge

jameslamb added 2 commits December 4, 2024 14:01

distribute libcugraph wheels

2898e78

exclude libcugraph.so from cugraph/pylibcugraph

0ecd5a5

jameslamb added the DO NOT MERGE Hold off on merging; see PR for details label Dec 5, 2024

github-actions bot added cuGraph CMake python ci labels Dec 5, 2024

jameslamb mentioned this pull request Dec 5, 2024

remove 'wget' conda dependency, re-organize dependencies.yaml #4805

Merged

jameslamb added 11 commits December 5, 2024 09:01

fix pr-builder

f1645ac

remove more cugraph-ops, fix variables

6e1d3d9

remove Python interpreter stuff

3aa693c

Merge branch 'branch-25.02' into libcugraph-wheel

7c55d20

temporarily raise size threshold

c5cb53d

declare librmm dependency, fix CI dependency tree

e73606c

run a subet of wheel-testing CI

fc37020

skip devcontainer jobs

e80d566

skip 'checks' job too

695f627

just run Python 3.12 builds

486bf96

move CUDA libs deps to wheels, ignore deprecation warning

fce4fcf

This comment was marked as resolved.

Sign in to view

jameslamb and others added 10 commits December 6, 2024 13:49

more dependency fiddling

796b54e

fix CI configs, remove unnecessary stuff from pylibcugraph CMake

157f63e

load libcugraph_c too

4b4c4b2

merge branch-25.02

e5aa3c2

fix typo

089109f

header install sort of working

881dc25

Merge branch 'branch-25.02' into libcugraph-wheel

3484186

install working (no libraft.a/libraft.so, just headers) ... export no…

ead60e4

…t working yet

install working, exports still not quite working

7dd4409

remove debugging stuff

1d60712

jameslamb marked this pull request as ready for review January 14, 2025 16:59

jameslamb requested review from a team as code owners January 14, 2025 16:59

stop installing RMM

540c1a4

github-actions bot added the cuGraph label Jan 14, 2025

bdice reviewed Jan 14, 2025

View reviewed changes

jameslamb added 3 commits January 16, 2025 10:54

remove DETECT_CONDA_ENV, remove CUDA_MATH_WHEELS, init CUDA architect…

5573920

…ures

Merge branch 'branch-25.02' of github.com:rapidsai/cugraph into libcu…

72978c7

…graph-wheel

language CUDA

5bc7523

bdice added 2 commits January 16, 2025 18:21

Remove libraft testing changes.

47672c9

Revert copyright-only changes.

5b6fd89

revert testing-only changes in GitHub Actions configs

abd6f18

jameslamb commented Jan 17, 2025

View reviewed changes

bdice reviewed Jan 17, 2025

View reviewed changes

jameslamb and others added 3 commits January 17, 2025 10:31

Apply suggestions from code review

96fb23a

Co-authored-by: Bradley Dice <[email protected]>

do not artificially restrict PARALLEL_LEVEL for libcugraph builds

ba73e56

Merge branch 'libcugraph-wheel' of github.com:jameslamb/cugraph into …

2dbd946

…libcugraph-wheel

BradReesWork approved these changes Jan 17, 2025

View reviewed changes

bdice approved these changes Jan 17, 2025

View reviewed changes

jameslamb added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change and removed DO NOT MERGE Hold off on merging; see PR for details labels Jan 17, 2025

rapids-bot bot merged commit 8ebff3b into rapidsai:branch-25.02 Jan 18, 2025
82 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce libcugraph wheels #4804

introduce libcugraph wheels #4804

jameslamb commented Dec 5, 2024 •

edited

Loading

This comment was marked as resolved.

bdice Jan 14, 2025

jameslamb Jan 16, 2025

bdice Jan 16, 2025 •

edited

Loading

bdice Jan 16, 2025

jameslamb Jan 17, 2025

bdice commented Jan 17, 2025

jameslamb commented Jan 17, 2025

jameslamb Jan 17, 2025

bdice left a comment

bdice Jan 17, 2025

jameslamb Jan 17, 2025

jameslamb Jan 17, 2025

bdice Jan 17, 2025

jameslamb Jan 17, 2025

bdice commented Jan 18, 2025

		PARALLEL_LEVEL=$(python -c \
		"from math import ceil; from multiprocessing import cpu_count; print(ceil(cpu_count()/4))")

	#include <raft/core/device_span.hpp>
	#include <raft/core/handle.hpp>
	#include <raft/random/rng_state.hpp>

	#include <rmm/resource_ref.hpp>

introduce libcugraph wheels #4804

introduce libcugraph wheels #4804

Conversation

jameslamb commented Dec 5, 2024 • edited Loading

Notes for Reviewers

Benefits of these changes

Wheel contents

Dependency Flows

Size changes (CUDA 12, Python 3.12, x86_64)

How I tested this

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice commented Jan 17, 2025

jameslamb commented Jan 17, 2025

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice commented Jan 18, 2025

jameslamb commented Dec 5, 2024 •

edited

Loading

bdice Jan 16, 2025 •

edited

Loading