Release tag mechanism for locked-down installs

Push a labeled version of crucible that passes ci after merging,
including all pinned commits of the sub-projects. This release
mechanism will be used for regression testing and also by users
that need more determinsm from crucible stable installs that do
not change over time.

Additional changes are required to implement the installation
bits that will take advantage of this release tag mechanism.

.github/workflows/crucible-release.yaml

@@ -0,0 +1,125 @@
+name: crucible-release
+
+on:
+  workflow_call:
+    inputs:
+      dry-run:
+        required: false
+        type: boolean
+        description: Do NOT push new generated tag to the upstream repositories
+        default: false
+
+env:
+  GITHUB_WORKSPACE: ${{ inputs.github_workspace }}
+
+jobs:
+  release-tag:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    outputs:
+      tag: ${{ steps.gen-release-tag.outputs.tag }}
+    steps:
+    - name: Generate release tag
+      id: gen-release-tag
+      run: |
+          year="$(date +%Y)"
+          month="$(date +%m)"
+          quarter="$(((month-1)/3+1))"
+          echo "tag=$year.$quarter" >> $GITHUB_OUTPUT
+    - name: Get release tag
+      id: get-release-tag
+      env:
+          TAG: ${{ needs.release-tag.outputs.tag }}
+      run: echo "$TAG"
+
+  remote-tag:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    - name: checkout crucible default
+      uses: actions/checkout@v4
+      with:
+        repository: perftool-incubator/crucible
+        ref: master
+        path: crucible
+    - name: check if tag exists on the crucible repo
+      env:
+        TAG: ${{ needs.release-tag.outputs.tag }}
+      run: |
+          cd $GITHUB_WORKSPACE
+          cd crucible
+          git ls-remote --tags origin | grep -v "$TAG"
+
+  sub-projects:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    outputs:
+      repos: ${{ steps.get-repos.outputs.repos }}
+    - name: Get the list of sub-projects
+      id: get-repos
+      run: |
+          cd $GITHUB_WORKSPACE
+          cd crucible
+          # get first column (repo name) and add to a bash array ( a b c)
+          projects=( $(grep -v ^# config/default_subprojects | awk {'print $1'} ) )
+          # builtin implict join array a,b,c
+          printf -v list '%s,' "${projects[@]}"
+          # convert to a comma separated list [a,b,c]
+          echo "repos=[${list%,}]" >> $GITHUB_OUTPUT
+
+  push-tags:
+    needs:
+    - release-tag
+    - remote-tag
+    - sub-projects
+    strategy:
+        matrix:
+            repository: ${{ needs.sub-projects.outputs.repos }}
+    steps:
+    - name: Display sub-project repository name
+      run: |
+          echo "repository=${{ matrix.repository }}"
+
+    - name: checkout sub-project repository
+      uses: actions/checkout@v4
+      with:
+        repository: perftool-incubator/${{ matrix.repository }}
+        ref: ''
+        path: ''
+
+    - name: create release tag
+      env:
+        TAG: ${{ needs.release-tag.outputs.tag }}
+      run: |
+          cd $GITHUB_WORKSPACE
+          cd ${{ matrix.repository }}
+          git tag $TAG
+
+    - name: push release tag
+      if: ${{ inputs.dry_run == false }}
+      env:
+        TAG: ${{ needs.release-tag.outputs.tag }}
+      run: |
+          cd $GITHUB_WORKSPACE
+          cd ${{ matrix.repository }}
+          ######git push origin $TAG
+          echo "git push origin $TAG"
+
+    - name: check if tag has been pushed
+      if: ${{ inputs.dry_run == false }}
+      env:
+        TAG: ${{ needs.release-tag.outputs.tag }}
+      run: |
+          cd $GITHUB_WORKSPACE
+          cd ${{ matrix.repository }}
+          git ls-remote --tags origin | grep "$TAG"
+          git fetch -t
+          git tag -l "$TAG"
+
+  crucible-stable-release-complete:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    needs:
+    - push-tags
+    steps:
+    - name: Confirm Success
+      run: echo "crucible-stable-release-complete"

.github/workflows/crucible-scheduled.yaml

@@ -16,3 +16,9 @@ jobs:
     secrets:
       production_registry_auth: ${{ secrets.CRUCIBLE_PRODUCTION_ENGINES_REGISTRY_AUTH }}
       quay_oauth_token: ${{ secrets.CRUCIBLE_QUAYIO_OAUTH_TOKEN }}
+
+  call-crucible-release:
+    uses: perftool-incubator/crucible/.github/workflows/crucible-release.yaml@main
+    needs: call-core-crucible-ci_scheduled
+    with:
+      github_workspace: "$GITHUB_WORKSPACE"

spec/release-tag.md

@@ -0,0 +1,67 @@
+# Release tag
+Implement a "Release" mechanism for installing "consistent" versions of
+Crucible.
+
+## Problem description
+We need to be able to install a "version" of Crucible that is "locked down"
+and won't change at all if it is installed over time.
+
+Crucible installations cannot be used for regression testing in a consistent
+manner.
+
+bin/update will override the installed repository by pulling the latest commit
+if the installed commit is an older one.
+
+Crucible installer clones Crucible repository and all sub-projects from the
+latest "main" branch. A default installation contains all the Crucible
+repositories from the latest merged commits.
+
+```
+#project-name        project-type   git-repo-url                     branch
+rickshaw             core           /rickshaw                        master
+multiplex            core           /multiplex                       master
+roadblock            core           /roadblock                       master
+...
+hwlatdetect          benchmark      /bench-hwlatdetect               main
+tracer               benchmark      /bench-tracer                    main
+...
+ftrace               tool           /tool-ftrace                     master
+testing              doc            /testing-repo                    master
+...
+```
+
+## Proposed change
+"Locked down" release by pinning all sub-projects to a commit that is
+verified by CI after code merges. Sub-projects includes core and benchmark
+repositories. Stable releases must be labeled to be a "locked-down" version
+of Crucible installation.
+
+### Tagging
+Create a new CI job that depends on the existing periodic CI job
+(scheduled) that runs against merged commits that passed pre-merge jobs.
+The new CI job will run only if the existing scheduled CI job succeeds.
+
+                           PASS
+CI workflow:
+crucible-core-ci-scheduled --> crucible-release
+       (existing)           (new / to be created)
+
+The "crucible-release" job should check for the tag YYYY.N, and tag
+every repository, including the main Crucible project if the tag does
+NOT exist. Example: 2024.1, 2025.3, etc.
+
+YYYY	4-digit Year
+N	Quarter (1=Jan-Mar, 2=Apr-Jun, 3=Jul-Sept, 4=Oct-Dec)
+
+The job should exit and do nothing if the tag already exists.
+
+Two inputs should be created with the workflow_dispatch (manual trigger):
+ - A `force` option will override the tag if specified.
+ - A `dry-run` mode will skip the push tag step for debugging purposes.
+
+### Installer
+- Add a new `--release=<tag>` to the installation to specify the release
+that will be installed.
+
+### Crucible command
+- Add a new `crucible repo release` to list tagged releases