Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add CI/CD architecture documentation #9202

Merged
merged 1 commit into from
Feb 4, 2025
+376 −0
Merged

docs: add CI/CD architecture documentation #9202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
360 changes: 360 additions & 0 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,360 @@
# Zebra CI/CD Architecture

This document provides a comprehensive overview of Zebra's Continuous Integration and Continuous Deployment (CI/CD) system. It serves as a guide for contributors, maintainers, and new team members.

## Table of Contents

1. [System Overview](#system-overview)
2. [CI/CD Workflow Diagram](#cicd-workflow-diagram)
3. [Core Infrastructure](#core-infrastructure)
4. [Workflow Organization](#workflow-organization)
5. [Test Execution Strategy](#test-execution-strategy)
6. [Infrastructure Details](#infrastructure-details)
7. [Best Practices](#best-practices)
8. [Known Issues](#known-issues)

## System Overview

Zebra's CI/CD system is built on GitHub Actions, providing a unified platform for automation. The system ensures code quality, maintains stability, and automates routine tasks through specialized workflows.

## CI/CD Workflow Diagram

Below is a Mermaid diagram illustrating how our CI workflows relate to each other, with a focus on parallel execution patterns and job dependencies. The diagram shows the main CI pipeline, integration test flow, unit test flow, underlying infrastructure, and the various triggers that initiate the pipeline.

```mermaid
graph TB
%% Define Triggers subgraph with parallel triggers
subgraph "Triggers"
direction TB
P[Pull Request] & Q[Push to main] & R[Weekly Schedule] & S[Manual Trigger] & T[Merge Queue]
end

%% Main CI Pipeline with parallel flows after build
subgraph "Main CI Pipeline"
direction TB
A[ci-tests.yml]
B[sub-build-docker-image.yml]
A --> B
end

%% Infrastructure dependencies
subgraph "Infrastructure"
direction TB
M[Docker Build Cloud]
N[GCP Resources]
O[GitHub Runners]
end

%% Unit Test Flow with parallel test execution
subgraph "Unit Test Flow"
direction TB
C[sub-ci-unit-tests-docker.yml]
H[test-all] & I[test-fake-activation-heights] & J[test-empty-sync] & K[test-lightwalletd-integration] & L[test-configuration-file]
C --> H
C --> I
C --> J
C --> K
C --> L
end

%% Integration Test Flow with some parallel and some sequential steps
subgraph "Integration Test Flow"
direction TB
D[sub-ci-integration-tests-gcp.yml]
E[sub-find-cached-disks.yml]
F[sub-deploy-integration-tests-gcp.yml]
G[sub-test-zebra-config.yml]
D --> E
D --> F
E --> F
F --> G
end

%% Connect triggers to main pipeline
P --> A
Q --> A
R --> A
S --> A
T --> A

%% Connect infrastructure to respective components
M --> B
N --> D
O --> C

%% Connect main pipeline to test flows
B --> C
B --> D

%% Style definitions
classDef primary fill:#2374ab,stroke:#2374ab,color:white
classDef secondary fill:#48a9a6,stroke:#48a9a6,color:white
classDef infra fill:#4b4e6d,stroke:#4b4e6d,color:white
classDef trigger fill:#95a5a6,stroke:#95a5a6,color:white

%% Apply styles
class A,B primary
class C,D,E,F,G secondary
class H,I,J,K,L secondary
class M,N,O infra
class P,Q,R,S,T trigger
```

*The diagram above illustrates the parallel execution patterns in our CI/CD system. All triggers can initiate the pipeline concurrently, unit tests run in parallel after the Docker image build, and integration tests follow a mix of parallel and sequential steps. The infrastructure components support their respective workflow parts concurrently.*

## Core Infrastructure

### 1. GitHub Actions

- Primary CI/CD platform
- Workflow automation and orchestration
- Integration with other services

### 2. Infrastructure as Code

- Uses [Cloud Foundation Fabric](https://github.com/ZcashFoundation/cloud-foundation-fabric) for GCP infrastructure
- Terraform-based architecture, networking, and permissions
- Resources (VMs, Disks, Images, etc.) deployed via GitHub Actions pipelines

### 3. Build and Registry Services

#### Docker-based Testing

- Most tests run in containers defined by our [Dockerfile](http://../../docker/Dockerfile)
- The [entrypoint script](http://../../docker/entrypoint.sh) manages:
- Test execution
- Environment configuration
- Resource cleanup

#### [Docker Build Cloud](https://www.docker.com/products/build-cloud/)

- Optimized build times (~10 min for non-cached, ~30 sec for cached)
- More efficient than GitHub Runners
- Addresses [Rust caching limitations](https://github.com/ZcashFoundation/zebra/issues/6169#issuecomment-1712776391)

#### Container Registries

- Google Cloud Registry: Internal CI artifacts
- [Docker Hub](https://hub.docker.com/): Public release artifacts
- Ensures proper artifact distribution

### 4. Test Infrastructure

#### GitHub-hosted Runners

- All Unit Tests jobs
- Standard CI/CD operations
- Limited to 6-hour runtime

#### Self-hosted Runners (GKE)

- All Integration Tests jobs (deployed to GCP)
- Support for tests exceeding 6 hours
- Extended logging capabilities
- Full GitHub Actions console integration

**Note**: Self-hosted Runners are just used to keep the logs running in the GitHub Actions UI for over 6 hours, the Integration Tests are not run in the Self-hosted Runner itself, but in the deployed VMs in GCP through GitHub Actions.

### 5. Queue Management

[Mergify](https://mergify.yml)

- Automated PR merging and queue-based testing
- Priority management
- Ensures code quality before merge
- See our [`.mergify.yml`](http://../../.mergify.yml) for configuration

## Workflow Organization

### Main Workflows

- **CI Tests** (`ci-*.yml`): Core testing workflows
- Unit tests
- Integration tests
- Code coverage
- Linting
- **CD Deployments** (`cd-*.yml`): Deployment workflows
- Node deployment to GCP
- Documentation deployment
- **Release Management** (`release-*.yml`): Version and release workflows

### Supporting Workflows

- **Sub-workflows** (`sub-*.yml`): Reusable workflow components
- Docker image building
- Test configurations
- GCP resource management
- **Patch Workflows** (`*.patch.yml`, `*.patch-external.yml`): Handle GitHub Actions limitations for required checks

### Patch Workflows Rationale

Our use of patch workflows (`.patch.yml` and `.patch-external.yml`) is a workaround for a [known limitation in GitHub Actions](https://github.com/orgs/community/discussions/44490) regarding path filters and required checks. When a workflow is marked as required for PR merging:

1. **Path Filtering Limitation**: GitHub Actions does not properly handle the case where a required workflow is skipped due to path filters. Instead of marking the check as "skipped" or "passed", it remains in a "pending" state, blocking PR merges.

2. **Our Solution**: We maintain parallel "patch" workflows that:

- Run without path filters
- Contain minimal steps that always pass when the original workflow would have been skipped
- Allow PRs to merge when changes don't affect relevant paths

3. **Impact**:

- Doubled number of workflow files to maintain
- Additional complexity in workflow management
- Extra status checks in PR UI

## Test Execution Strategy

### Test Orchestration

Our test execution is centralized through our Docker [entrypoint script](http://../../docker/entrypoint.sh), providing a unified way to run tests both in CI and locally.

#### Environment Variable-driven Testing

```bash
# Full test suite
docker run --rm -e RUN_ALL_TESTS=1 zebra-tests

# Specific test suites
docker run --rm -e TEST_LWD_INTEGRATION=1 zebra-tests
```

#### Test Categories

- Full suite (`RUN_ALL_TESTS`)
- Experimental features (`RUN_ALL_EXPERIMENTAL_TESTS`)
- Integration tests (`TEST_LWD_INTEGRATION`)
- Network sync (`TEST_ZEBRA_EMPTY_SYNC`, `TEST_UPDATE_SYNC`)
- State management (`TEST_DISK_REBUILD`)

### Pull Request Testing

#### Continuous Validation

- Tests run automatically on each commit
- Contributors get immediate feedback on their changes
- Regressions are caught early in the development process
- Reduces manual testing burden on reviewers

#### Fast Feedback Loop

- Linting: Code style and formatting
- Unit tests: Function and component behavior
- Basic integration tests: Core functionality
- All results are reported directly in the PR interface

#### Deep Validation

- Full integration test suite
- Cross-platform compatibility checks
- Performance benchmarks
- State management validation

### Scheduled Testing

Weekly runs include:

- Full Mainnet synchronization
- Extended integration suites
- Resource cleanup

## Infrastructure Details

### VM-based Test Infrastructure

#### Test-specific Requirements

- Some integration tests need a fully synced network
- Certain tests validate against specific chain heights
- Network state persistence between test runs
- Not all tests require this infrastructure - many run in standard containers

#### State Management Complexity

- **Creation**: Initial sync and state building for test environments
- **Versioning**: Multiple state versions for different test scenarios
- **Caching**: Reuse of existing states to avoid re-sync
- **Attachment**: Dynamic VM disk mounting for tests
- **Cleanup**: Automated state and resource cleanup

#### Infrastructure Implications

- GCP VM infrastructure for state-dependent tests
- Complex disk image management for test states
- State versioning and compatibility checks
- Resource lifecycle management

#### Future Considerations

- Potential migration of state-dependent tests to container-native environments
- Would require solving state persistence in Kubernetes
- Need to balance containerization benefits with test requirements
- Opportunity to reduce infrastructure complexity

## Best Practices

### For Contributors

#### Local Testing

```bash
# Build and run tests
docker build -t zebra-tests --target tests .
docker run --rm zebra-tests
```

#### PR Guidelines

- Use descriptive labels
- Mark as draft when needed
- Address test failures

### For Maintainers

#### Workflow Maintenance

- Regular review of workflow efficiency
- Update resource allocations as needed
- Monitor test execution times

#### Security Considerations

- Regular secret rotation
- Access control review
- Dependency updates

## Known Issues

### External Contributor Limitations

#### GCP Dependencies

- Most CI workflows depend on Google Cloud Platform resources
- Docker artifacts and VM images are tied to GCP
- External contributors cannot run full CI suite in their forks
- Integration tests require GCP infrastructure access
- This particularly impacts:
- Integration test execution
- Node deployment testing
- State storage and caching validation

#### GitHub Actions Variables Restriction

- Due to a [GitHub Actions limitation](https://github.com/orgs/community/discussions/44322), workflows in forked repositories cannot access repository variables
- This affects external contributors' ability to run CI workflows
- Required configuration values are not available in fork-based PRs
- Currently no workaround available from GitHub
- Impact on external contributors:
- Cannot run workflows requiring GCP credentials
- Unable to access configuration variables
- Limited ability to test infrastructure changes

### Mitigation Through Mergify

- When external PRs enter the merge queue, they are tested with full access to variables and resources
- All CI workflows run in the context of our repository, not the fork
- This provides a safety net, ensuring no untested code reaches production
- External contributors can still get feedback through code review before their changes are tested in the queue

These safeguards help maintain code quality while working around the platform limitations for external contributions.
Loading
Loading