diff --git a/DEVELOP.md b/DEVELOP.md
index b6f1dfa5..49de1cd9 100644
--- a/DEVELOP.md
+++ b/DEVELOP.md
@@ -1,11 +1,33 @@
# Contributing to the RAPIDS devcontainers
+This document contains implementation details and design overviews for
+[rapidsai/devcontainers](https://github.com/rapidsai/devcontainers) as a
+centralized source of common scripts and patterns for providing devcontainers.
+
+For the user-level overview providing instructions on how to use the
+devcontainer as a development environment, see
+[USAGE_IN_PROJECT.md](USAGE_IN_PROJECT.md)
+
+For a project maintainer-level overview providing instructions on how to add
+and change .devcontainer.json to suit your project, see [USAGE.md](USAGE.md)
+
+The code in this repository fits into a few main categories:
+
+* Features
+* GitHub Actions automations
+* Scripts
+* matrix.yml
+* Dockerfiles
+
+## Features
+
From the official devcontainer [documentation on Features](https://containers.dev/implementors/features/):
> Development container "Features" are self-contained, shareable units of installation code and development container configuration.
-In short, a "feature" is a layer in a Dockerfile which encapsulates a reusable bit of logic executed when building a Docker image.
-
-This repository defines features to install the following dev tools, compilers, and SDKs:
+Each "feature" specified becomes a `RUN` statement in a temporary Dockerfile,
+and as such each "feature" results in an image layer. The
+[rapidsai/devcontainers repository](https://github.com/rapidsai/devcontainers)
+defines `features` to install the following dev tools, compilers, and SDKs:
* [CMake](features/src/cmake/)
* [CUDA Toolkit](features/src/cuda/)
@@ -21,3 +43,144 @@ This repository defines features to install the following dev tools, compilers,
* [sccache](features/src/sccache/)
* [devcontainer-utils](features/src/utils/)
* [rapids-build-utils](features/src/rapids-build-utils/)
+
+Many of these feature scripts use apt utilities and thus only run on debian-based
+images.
+
+### Utility features
+
+A few of the features install custom tools for the devcontainer environment,
+rather than just installation scripts of external tools and libraries. A brief
+overview of responsibilities for these features follows.
+
+#### `rapids-build-utils`
+
+The `rapids-build-utils` feature is a good place to start when investigating an
+unknown command or behavior in a devcontainer. Most of the scripts in the this
+feature prepare the devcontainer prior to use. The scripts from this project are
+installed by [`install.sh`](./features/src/rapids-build-utils/install.sh). `install.sh`
+creates aliases that begin with a `rapids-` prefix for the .sh scripts in the
+[`rapids-build-utils` bin
+folder](features/src/rapids-build-utils/opt/rapids-build-utils/bin).
+
+The `rapids-build-utils` scripts can also be run manually to force an update the
+devcontainer scripts after modifying a project's manifest.yml file to add new
+projects or change dependencies. See
+[USAGE.md](./USAGE.md#generating-scripts-for-other-projects-manifestyaml-file)
+for more information on customizing manifest.yml.
+
+> **NOTE:** Some scripts in downstream projects, especially those in `ci/`
+folders, call scripts that start with the `rapids-` prefix but are not provided
+by the `rapids-build-utils` feature. These `rapids-` prefixed scripts come from
+the [gha-tools](https://github.com/rapidsai/gha-tools/tree/main/tools)
+repository, which is part of the older RAPIDS unified CI configuration and not
+currently part of the devcontainer features. To run the scripts in a project's
+`ci` folder, use the images at
+[rapidsai/ci-imgs](https://github.com/rapidsai/ci-imgs) instead of the
+devcontainers.
+
+#### devcontainers-utils
+
+These scripts handle git-related configuration, setting up SSH deploy
+keys, GitHub authorization, and the vault setup for S3. The commands here are
+prefixed with `devcontainer-` during installation.
+
+## Github Actions automations
+
+Github Actions runs the build matrix of the many base images defined in [matrix.yml](./matrix.yml).
+These actions are broken up into many reusable pieces. The image build jobs start in
+[release.yml](./.github/workflows/release.yml).
+
+```mermaid
+flowchart TD
+ subgraph Legend
+ workflow
+ action(action)
+ script{{script}}
+ end
+ subgraph Image build Github Actions flowchart
+ release.yml[release.yml workflow dispatch] --> release-features.yml["Publish features (release-features.yml)"]
+ release-features.yml --> image-matrix("Determine image matrix (image-matrix/action.yml)")
+ image-matrix --> write-matrix{{Write matrix by calling image-matrix/action.sh}}
+ write-matrix --> build-test-and-push-linux-image.yml
+ write-matrix --> build-test-and-push-windows-image.yml
+ build-test-and-push-linux-image.yml --> write-json("Write .devcontainer.json (devcontainer-json/action.yml)")
+ write-json --> write-script{{Write devcontainer.json by calling devcontainer-json/action.sh}}
+ build-test-and-push-windows-image.yml --> build-windows("Build windows image (build-windows-image/action.yml)")
+ write-script --> build-linux-image("Build linux limage (build-linux-image/action.yml)")
+ end
+```
+
+These are divided into 3 categories in the diagram:
+* Workflows are `.yml` files in `.github/workflows`
+* Actions are folders in `.github/actions`. One folder per action. Actions must have a `action.yml` file, but may
+also have accompanying shell scripts.
+* Scripts are the shell scripts that are in some of the actions folders. They are broken out in this diagram to help
+show that functionality can be defined outside of action.yml files.
+
+## Base images (matrix.yml)
+
+Base images are composed from individual features in [matrix.yml](./matrix.yml)
+using [YAML
+anchors](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/).
+These get built on Github Actions as described
+[above](#github-actions-automations) The devcontainers repo has some generic
+scripts for generating, building and launch devcontainers in the scripts folder.
+Some downstream repos have related, but more specialized scripts to facilitate
+working with their subspace of the configuration matrix. An example is CCCL's
+[make_devcontainers.sh](https://github.com/NVIDIA/cccl/blob/main/.devcontainer/make_devcontainers.sh)
+script.
+
+## Dockerfiles
+
+Dockerfiles do not play much role in this scheme. They serve to [set a few global
+variables and extend from the base image](./.devcontainer/rapids.Dockerfile).
+Changes made to a Dockerfile are usually better achieved by adding or changing a feature
+script instead.
+
+## Build caching with `sccache`
+
+The devcontainers configure CMake to use
+[sccache](https://github.com/mozilla/sccache) as C, C++, CUDA, and Rust compiler
+launchers. Refer to the [sccache
+docs](https://github.com/mozilla/sccache/tree/main/docs) for configuring the
+various storage back-ends.
+
+### Build caching with private S3 buckets
+
+A private S3 bucket can be used as the `sccache` storage back-end.
+
+If you're using a [GitHub action](https://github.com/aws-actions/configure-aws-credentials) to assume AWS roles in CI, or are comfortable distributing and managing S3 credentials, you can define the `SCCACHE_BUCKET`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` variables in the container environment.
+
+### Using GitHub OAuth to issue S3 credentials via Hashicorp Vault
+
+The [`devcontainer-utils`](features/src/utils/) feature includes a `devcontainer-utils-vault-s3-init` script that uses GitHub OAuth and Hashicorp Vault to issue temporary S3 credentials to authorized users.
+
+> **NOTE:** This script runs in the devcontainer's [`postAttachCommand`](https://containers.dev/implementors/json_reference/#lifecycle-scripts), but it does nothing unless `SCCACHE_BUCKET` and `VAULT_HOST` are in the container environment.
+
+The `devcontainer-utils-vault-s3-init` script performs the following actions, exiting early if any step is unsuccessful:
+
+1. Log in via the [GitHub CLI](https://cli.github.com/)
+2. Authenticate via [Vault's GitHub auth method](https://developer.hashicorp.com/vault/docs/auth/github#authentication)
+3. Use Vault to [generate temporary AWS credentials](https://developer.hashicorp.com/vault/api-docs/secret/aws#generate-credentials)
+4. Store results in `~/.aws` and install crontab to re-authenticate
+
+The above steps can be customized via the following environment variables:
+```
+# The hostname of the Vault instance to use
+VAULT_HOST="https://vault.ops.k8s.rapids.ai"
+
+# List of GitHub organizations for which Vault can generate credentials.
+# The scripts assumes the Vault instance exposes an authentication endpoint
+# for each org at `$VAULT_HOST/v1/auth/github-$org/login`.
+# https://developer.hashicorp.com/vault/docs/auth/github#authentication
+VAULT_GITHUB_ORGS="nvidia nv-morpheus nv-legate rapids"
+
+# The TTL for the generated AWS credentials
+VAULT_S3_TTL=43200
+
+# The URI to the Vault API that generates AWS credentials
+# The full URL expands to `$VAULT_HOST/$VAULT_S3_URI?ttl=$VAULT_S3_TTL`
+# https://developer.hashicorp.com/vault/api-docs/secret/aws#generate-credentials
+VAULT_S3_URI="v1/aws/creds/devs"
+```
\ No newline at end of file
diff --git a/README.md b/README.md
index ef43c894..8cc0d1bb 100644
--- a/README.md
+++ b/README.md
@@ -1,11 +1,23 @@
# RAPIDS [devcontainers](https://containers.dev/)
-This repository contains features and workflows for building development containers to support local dev and CI for NVIDIA [RAPIDS](https://github.com/rapidsai), [CCCL](https://github.com/nvidia/cccl), and [Legate](https://github.com/nv-legate).
+This repository contains centralized features and workflows for building
+development containers ([devcontainers](https://containers.dev/)) to support
+local dev and CI for projects in NVIDIA [RAPIDS](https://github.com/rapidsai),
+[CCCL](https://github.com/nvidia/cccl), and
+[Legate](https://github.com/nv-legate).
-We've chosen to use a monorepo, but it is similar in spirit to the official [devcontainers/features](https://github.com/devcontainers/features) and [devcontainers/images](https://github.com/devcontainers/images) repositories.
+[Devcontainers](https://containers.dev/) are an open standard for specifying the
+creation and execution of Docker containers for developing a codebase.
-### For details on using the RAPIDS devcontainers, see [`USAGE.md`](USAGE.md).
+Downstream repositories that utilize devcontainers use both the `feature`
+scripts that install software, as well as docker images that serve to cache sets
+of installed software. These images serve as base images for the devcontainers
+specified in the downstream repositories.
-### For details on contributing to this repository, see [`DEVELOP.md`](DEVELOP.md).
+## Usage
-### See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+### [Using devcontainers to provide a build environment on a project](./USAGE_IN_PROJECT.md)
+
+### [Setting up and maintaining devcontainer configuration in other projects](./USAGE.md)
+
+### [Developing the centralized `feature` scripts and base images in this repository](DEVELOP.md).
diff --git a/USAGE.md b/USAGE.md
index e06c7f89..0d2f4a5c 100644
--- a/USAGE.md
+++ b/USAGE.md
@@ -1,19 +1,121 @@
-# Using the RAPIDS devcontainers
+# Adding and adapting devcontainers to a project
-### See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+This document describes how to add devcontainers to a project that does not yet
+have them and how to customize the devcontainer to fit a project's
+needs.
-* [Using the pre-built images](#using-the-pre-built-images)
- * [Using in `devcontainer.json`](#using-in-devcontainerjson)
-* [Build caching with sccache](#build-caching-with-sccache)
+For how to use devcontainers to provide development environments, see
+[USAGE_IN_PROJECT.md](./USAGE_IN_PROJECT.md)
+For how to change the centralized installation and configuration scripts that
+are shared among projects, see [DEVELOP.md](./DEVELOP.md).
+
+## Adding devcontainers to a project
+
+Adding devcontainers to a project means adding one or more `devcontainer.json`
+files. One devcontainer is equivalent to one `devcontainer.json` file. Projects
+such as [CCCL](https://github.com/NVIDIA/cccl/blob/main/.devcontainer/README.md)
+have additional scripts that manage many configurations. New projects needing
+only cuda and python can bootstrap themselves by copying the `cuda-*` folders
+from [rapids/devcontainers](./.devcontainer).
+
+## Devcontainers helper scripts
+
+### Generated build scripts
+
+Several scripts are generated based on the contents of `manifest.yaml`. By
+default, `manifest.yaml` comes from [the RAPIDS/devcontainers
+repo](./features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml),
+but [a local `manifest.yaml` file can be used instead](#generating-scripts-for-other-projects-manifestyaml-file).
+
+The generated scripts are:
+* `/usr/bin/build-*`
+* `/usr/bin/clean-*`
+* `/usr/bin/clone-*`
+* `/usr/bin/configure-*`
+
+Here the `*` is a placeholder for the project name and the kind of
+build. For example, a project with a python component in `manifest.yaml`
+will have `build-cudf-python`.
+
+These scripts may trigger side-effects. For example, `build-*` scripts are only
+generated for projects that exist in the workspace. When working on `cudf`,
+which depends on `rmm`, the default workspace only mounts `cudf` and generates
+`build-cudf` scripts. To obtain `rmm` build scripts also, run `clone-rmm`,
+which will clone `rmm` into the workspace and generate build scripts for it.
+
+### rapids-build-utils
+
+[`rapids-build-utils`](./features/src/rapids-build-utils/opt/rapids-build-utils/bin)
+scripts are meta-build scripts. They assist with setting up the workspace and
+reloading things when important configuration changes happen.
+`rapids-build-utils` scripts are most often called indirectly by devcontainer
+lifecycle hooks, but are also useful for forcing updates in the workspace.
+`rapids-build-utils` scripts are described in more detail in
+[DEVELOP.md](./DEVELOP.md#rapids-build-utils).
+
+### Generating scripts for other projects: `manifest.yaml` file
+
+Build script generation is controlled with a `manifest.yaml` file. This file
+comes from [the rapidsai/devcontainers
+repo](./features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml),
+but can be [overridden with a local manifest.yaml file](#local-changes). New
+projects and changes to dependencies can be submitted in a PR to the
+[rapidsai/devcontainers](https://github.com/rapidsai/devcontainers) repo.
+
+> **NOTE** manifest.yaml's content is pinned in
+`.devcontainer/*/devcontainer.json` indirectly via the pin on rapids-build-utils.
+For changes in manifest.yaml to be reflected in the devcontainer, the manifest.yaml
+changes need to be present in a release of `rapidsai/devcontainers`, and the pin
+on `rapids-build-utils` must be updated in the project's devcontainer.json file(s).
+
+```
+ "features": {
+ "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.2": {}
+ },
+```
+
+In this `devcontainer.json` excerpt, the `24.2` value is the pin that would need
+to be updated.
+
+#### Local changes
+
+To work around the delay in getting a change merged into the devcontainers repo
+and have a new tag in place, a local manifest.yaml file can be used. Start by
+copying `manifest.yaml` from the rapidsai/devcontainers repo. This file can
+live anywhere, but this example has it in `.devcontainer/manifest.yaml`.
+
+In the desired devcontainer.json file, add a top-level key with this:
+
+```
+ "containerEnv": {
+ "PROJECT_MANIFEST_YML": "/home/coder/${localWorkspaceFolderBasename}/.devcontainer/manifest.yaml"
+ },
+```
+
+Rebuild or re-open the devcontainer, and the scripts (`build-*`, `clone-*`,
+etc.) will be regenerated using the local `.devcontainer/manifest.yaml` instead
+of the manifest.yaml that comes from the devcontainer.json-pinned
+`rapids-build-utils` version. The `rapids-generate-scripts` command from
+[rapids-build-utils](./features/src/rapids-build-utils/opt/rapids-build-utils/bin)
+will also force a refresh of these scripts without rebuilding the devcontainer.
## Using the pre-built images
-We publish a [matrix](matrix.yml) of pre-built images to DockerHub to accelerate initializing local devcontainers, GitHub Codespaces, and CI jobs.
+The choice of using a pre-built image refers to the `build/args/BASE` or `image`
+entry in `devcontainer.json`. The pre-built images are not meant to be used
+directly. We publish a [matrix](matrix.yml) of pre-built images to DockerHub to
+accelerate initializing local devcontainers, GitHub Codespaces, and CI jobs.
+These use the "feature" scripts to install their components, so these base
+containers effectively cache running their feature scripts.
-The features that comprise the image are noted in the image tags. If no version is defined for a tool or SDK, the image includes the latest available version at image build time.
+The features that comprise the image are noted in the image tags. If no version
+is defined for a tool or SDK, the image includes the latest available version at
+image build time.
-> **NOTE:** `git`, `git-lfs`, `github-cli`, `gitlab-cli`, `cmake`, `ninja`, `sccache`, and our devcontainer-utils [are included](image/.devcontainer/devcontainer.json#L12-L33) in each pre-built image.
+> **NOTE:** `git`, `git-lfs`, `github-cli`, `gitlab-cli`, `cmake`, `ninja`, `sccache`, and [devcontainer-utils](./features/src/utils) [are included](image/.devcontainer/devcontainer.json#L12-L33) in each pre-built image.
+
+See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
### Using in [`devcontainer.json`](https://containers.dev/implementors/json_reference/#image-specific)
@@ -21,21 +123,32 @@ The pre-built images can be used as the `"image"`, or as the base of a Dockerfil
devcontainer.json using pre-built image
{
"image": "rapidsai/devcontainers:24.06-cpp-llvm16-cuda12.0-nvhpc23.5-ubuntu22.04",
"hostRequirements": { "gpu": true },
"workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind"
}devcontainer.json using individual features
{
"image": "ubuntu:22.04",
"features": {
"ghcr.io/rapidsai/devcontainers/features/cmake:24.6": {},
"ghcr.io/rapidsai/devcontainers/features/ninja:24.6": {},
"ghcr.io/rapidsai/devcontainers/features/sccache:24.6": {
"version": "0.5.4"
}
},
"overrideFeatureInstallOrder": [
"ghcr.io/rapidsai/devcontainers/features/cmake",
"ghcr.io/rapidsai/devcontainers/features/ninja",
"ghcr.io/rapidsai/devcontainers/features/sccache"
],
"workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind"
}devcontainer.json extending base image with additional features
{
"rapidsai/devcontainers:24.06-cpp-llvm16-cuda12.0-nvhpc23.5-ubuntu22.04",
"features": {
"ghcr.io/rapidsai/devcontainers/features/cmake:24.02": {},
"ghcr.io/rapidsai/devcontainers/features/ninja:24.02": {},
"ghcr.io/rapidsai/devcontainers/features/sccache:24.02": {
"version": "0.5.4"
}
},
"overrideFeatureInstallOrder": [
"ghcr.io/rapidsai/devcontainers/features/cmake",
"ghcr.io/rapidsai/devcontainers/features/ninja",
"ghcr.io/rapidsai/devcontainers/features/sccache"
],
"workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind"
}