Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patcher edify: Concepts, Architecture, and Installation files #2315

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
11 changes: 6 additions & 5 deletions docs/2.0/docs/patcher/architecture/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,15 @@

### Environments

Patcher allows teams to define environments as a grouping of folders using glob patterns. Patcher commands (on the CLI and in GitHub Actions) accept commands to match those folders, such as the argument to `patcher-action` -- `include_dirs: "{*prod*}/**"` which would match all folders with "prod" in the name. A given environment can include 1 or many (without limit) folders. Patcher will scan the entire group of folders at once for potential updates and changes.
Patcher allows teams to define environments as groupings of folders using glob patterns. Patcher commands on the CLI and in GitHub Actions accept arguments to match these folders. For example, the `patcher-action` argument `include_dirs: "{*prod*}/**"` matches all folders containing "prod" in their name. A single environment can include one or multiple folders without any set limit. Patcher scans the entire group of folders simultaneously for potential updates and changes.

There is no limit on how many environments you can have, or other limit on the naming structure for those environments.
There are no restrictions on the number of environments or the naming structure you can use.

In the future it is planned to model environments using a configuration based system (to be shared with Gruntwork Pipelines) which will allow for even more flexibility in your folder structure.
In the future, you'll model environments using a configuration-based system integrated with Gruntwork Pipelines, offering greater flexibility in organizing folder structures.

### Dependencies

A `dependency` in Patcher workflows is a reference to code that is versioned and in use by your codebase, generally a Terraform or Tofu module that exists in a git repo using a specific git tag for versioning. For example, if you are using `gruntwork-io/terraform-aws-messaging.git//modules/sqs?ref=v0.8.0` as a terraform source module, then your dependency would be `gruntwork-io/terraform-aws-messaging/sqs`.
A `dependency` in Patcher workflows refers to versioned code that your codebase references, typically a Terraform or OpenTofu module stored in a Git repository and tagged with a specific version. For example, if your Terraform source module is `gruntwork-io/terraform-aws-messaging.git//modules/sqs?ref=v0.8.0`, the corresponding dependency is `gruntwork-io/terraform-aws-messaging/sqs`.

Patcher generally models promotion workflows around the idea of grouping changes together per-dependency. Patcher would then identify all usages of `gruntwork-io/terraform-aws-messaging/sqs` within a given environment and create a single pull request to update to the next appropriate version.

Patcher organizes promotion workflows by grouping changes per dependency. It identifies all instances of `gruntwork-io/terraform-aws-messaging/sqs` within a specified environment and generates a single pull request to update the dependency to the next appropriate version.
97 changes: 50 additions & 47 deletions docs/2.0/docs/patcher/concepts/grouping.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
# Update Grouping & Pull Request Strategy

## Overview
When working with enterprise IaC repositories you'll often find that you tend to reuse modules many times across a single repository. This creates a scenario where when a dependent module releases a new version, you are prompted with updating your reference to that dependency many times. This is in contrast to the traditional dependency update scenario for application codebases in e.g. a Java or NodeJS codebase, where a single dependency update may result in a change in a single change in a gradle.build or package.json file. In those environments, a pull request to bump a version would likely be only a few lines of code, and the build with those updates would progress through CI to reach higher environments in sequence. In IaC, you often have those environments repeated within a codebase, and references to those dependencies in many places. Patcher thus has a choice - when a dependency releases a new version, how to divide the diff of applying those updates into pull requests.

The naive option is to simply make one pull request per dependency update found within a repository. At scale, this quickly becomes unmanageable. Patcher does support this behavior, but it also includes the ability to create PRs to address all of those changes in a single PR, to create a single PR per environment (e.g. update everything in dev), to create a PR per dependency update (e.g. update every instance of dependency XYZ), or to make a PR per dependency update per environment (update dependency XYZ in dev).

In summary, there are five ways to group pull requests generated by Patcher:
1. **Full-Consolidation**: One PR to update everything.
2. **No-Consolidation**: One PR for each update.
3. **Dependency-Only Consolidation**: One PR per dependency. This allows you to merge one dependency at a time to all environments where they are used.
4. **Environment-Only Consolidation**: One PR per environment. This allows you to promote all dependencies as a group through environments one at a time.
5. **(Environment x Dependency) Consolidation**: One per dependency per environment. This allows you to promote a single dependency at a time through environments

## Grouping Examples
To clarify via example, let's imagine we have the following repository
Enterprise IaC repositories often reuse modules extensively across a single repository. As a result, updating a dependent module version can require changes in multiple references across the codebase. Unlike application codebases in environments like Java or Node.js—where a single version bump typically involves updating centralized files such as `gradle.build` or `package.json`—IaC repositories frequently repeat environments and dependencies across folders. This repetition creates a challenge: how to group dependency updates into pull requests for optimal efficiency and maintainability?

The simplest approach is to create one pull request for each dependency update. While simple, this method can quickly become overwhelming at scale. Patcher supports this approach but also offers more flexible options for grouping updates. Pull requests can be consolidated based on the following strategies:

- **Full-consolidation**: A single pull request updates all dependencies across the repository.
- **No-consolidation**: A separate pull request is created for each individual update.
- **Dependency-only consolidation**: A single pull request is created for each dependency, updating it across all environments.
- **Environment-only consolidation**: A single pull request is created for each environment, updating all dependencies within that environment.
- **(Environment x Dependency) consolidation**: A single pull request is created for each dependency within each environment.

## Grouping examples
To demonstrate these strategies, consider the following example repository:
```
/dev/unit1/terragrunt.hcl -> [email protected]
/dev/unit2/terragrunt.hcl -> [email protected]
Expand All @@ -24,51 +23,54 @@ To clarify via example, let's imagine we have the following repository
/prod/unit4/terragrunt.hcl -> [email protected]
```

In this example, assuming all 3 dependencies have newer versions available, the 5 strategies would result in:
1. **Full-Consolidation**: One pull request with an update to all 7 units
2. **No-Consolidation**: Seven pull requests, one for each unit
3. **Dependency-Only Consolidation**: Three pull requests, one for each dependency, with the PRs for `dependency1` and `dependency2` updating both `dev` and `prod`.
4. **Environment-Only Consolidation**: Two pull requests, one for `dev` and one for `prod`.
5. **(Environment x Dependency) Consolidation**: Five pull requests, two for `dev` and three for `prod`.
Assuming newer versions are available for all three dependencies, the strategies would result in:
- **Full-consolidation**: One pull request updating all seven units.
- **No-consolidation**: Seven separate pull requests, one per unit.
- **Dependency-only consolidation**: Three pull requestsone for each dependency. For example, `dependency1` would be updated across both `dev` and 'prod'.
- **Environment-only consolidation**: Two pull requestsone for all updates in `dev` and one for all updates in `prod`.
- **(Environment x Dependency) consolidation**: Five pull requeststwo for updates in `dev` and three for updates in `prod`.

## Terminology
* `unit` A unit refers to a folder containing a `terragrunt.hcl` file, and thus a single corresponding OpenTofu state file. A unit may specify one or multiple modules as `dependencies`.
* `dependency` (also referred to as `target`) A dependency is an OpenTofu module that is referenced by `ref` (usually a full source code path AND a version number) inside your `unit`. Patcher understands the semantics of semantic versioning on dependency `ref`s.
* `environment` is a logical grouping of infrastructure to represent your application environments, such as `dev` or `prod`. An environment usually contains multiple `units` and thus many `dependencies`. Generally IaC environments are similar to each other, and represented as a folder structure in your repository.
* `update` is the the act of changing a single instance of a dependency to reference a newer version and to accommodate any breaking changes.
* `unit`: A folder containing a `terragrunt.hcl` file, representing a single OpenTofu state file. A unit may reference one or more modules as dependencies.
* `dependency` (or `target`): An OpenTofu module referenced using a `ref` (typically a full source path and version) in a `unit`. Patcher interprets semantic versioning for dependency updates.
* `environment`: A logical grouping of infrastructure representing application stages, such as `dev` or `prod`. Environments are generally organized as folders in the repository and include multiple units and dependencies.
* `update`: The action of modifying a dependency reference to use a newer version and accommodating any associated breaking changes.

:::info
As of November 2024 Patcher's understanding of environments is limited to groupings of folders matched with glob patterns. E.g. `dev` is all folders matching `dev-*`, `prod` is all folders matching `prod-*`. Pipelines has a sophisticated HCL configuration syntax that allows for much more powerful definitions of environments. It is planned that Patcher will be able to leverage this method of defining environments in the future. Let us know if this expanded definitional capability is important to your use case.

As of November 2024, Patcher recognizes environments using folder groupings matched with glob patterns. For example, `dev` may correspond to `dev-*` folders and 'prod' to `prod-*` folders. A more sophisticated environment definition using HCL syntax (similar to Pipelines) is planned for future releases. Let us know if this capability is important for your use case.

:::

## Implementation Discussion

When generating pull requests in CI, the workflow generally involves first invoking `patcher report` to identify what updates are available, then `patcher update` to create PRs applying those updates. Patcher does not have a built-in way to specify the grouping strategy by name, instead the grouping strategies are implemented with flags to the report and update commands.
## Implementation discussion

### Patcher Report
In CI workflows, pull requests are typically generated by first running a patcher report to identify updates, followed by `patcher update` to apply those updates. Patcher does not include a single option to specify grouping strategies by name. Instead, grouping is implemented through combinations of `report` and `update` command flags.

Patcher [report](/2.0/reference/patcher/#report) accepts a `--include-dirs` argument. When passed, this will cause `patcher report` to only output updates matching the glob pattern specified. This allows the developer to in essence limit the resulting PR to only a single environment. By invoking `patcher report` multiple times with different `--include-dirs` arguments, the developer can in effect create different PR workflows for each environment.
### Patcher report

The `patcher report` command accepts the `--include-dirs` flag, which filters updates based on matching glob patterns. This allows developers to limit updates to specific environments. By running `patcher report` multiple times with different `--include-dirs` values, you can create distinct workflows for each environment.
Patcher `report` outputs in JSON which can be inspected or iterated over to achieve desired behaviors.

### Patcher Update
The patcher report output is JSON-formatted and can be inspected or iterated for further customization.

Patcher [update](/2.0/reference/patcher/#update) accepts a `--target` argument. This argument can be specified one or many times, and limits which dependencies patcher will include in the generated pull request. By invoking `patcher update` multiple times with different `--target` arguments, the developer can in effect create different PR workflows for each dependency.
### Patcher update

## Implementation Walkthrough
The `patcher update` command accepts a `--target` flag, which specifies one or more dependencies to update. By running `patcher update` with different `--target` values, you can control which dependencies are included in each pull request.

### Full-Consolidation
To implement a full-consolidation workflow, do not pass either `--include-dirs` or `--target` arguments to patcher `report` and `update`. The output of patcher `report` will include all updates that are available, and the effect of patcher `update` will be to create a single PR that includes all targets (dependencies) across all folders.
## Implementation walkthrough

### Full-consolidation
For full consolidation, omit the `--include-dirs` and `--target` arguments. This approach generates a single pull request containing all updates.

**Pseudocode:**
```
run patcher report
run patcher update
```

### No-consolidation

### No-Consolidation

Patcher `report` has two types of outputs, a `plan` and a `spec`. Generally we use the `spec` file as its simpler and contains the information needed for most workflows. To achieve a no-consolidation workflow, however, we need the full `plan`.
To create one pull request per update, use the `plan` output of `patcher report` and iterate over each dependency and its instances.

**Pseudocode:**
```
Expand All @@ -79,19 +81,19 @@ for each dependency in plan.json
run patcher update --target $dependency.org/$dependency.repo/$dependency.module
```

### Dependency-Only Consolidation
To implement a dependency-only consolidation workflow, run patcher `update` without any `--include-dirs` arguments. This will create a single report output JSON file for all updates. Iterate over the targets in that JSON file and run patcher `update` once per identified target.
### Dependency-only consolidation
To group by dependency, run `patcher update` for each target without filtering environments.

**Pseudocode:**
```
run patcher report without an include-dirs argument
for each $target in the output of patcher report
for each $target in output
run patcher update --target=$target
```


### Environment-Only Consolidation
To implement a environment-only consolidation workflow you need to iterate over each of your environments, passing in the environment to `--include-dirs` to `report` and then pass the complete output from report to a single `update` command.
### Environment-only consolidation
To group updates by environment, use `--include-dirs` to filter by environment and run `patcher update` without specifying targets.

**Pseudocode:**
```
Expand All @@ -100,16 +102,17 @@ for each $environment
run patcher update without any target argument
```

### (Environment x Dependency) Consolidation
To implement a environment x dependency consolidation workflow you will need to iterate over each of your environments, passing in the environment to `--include-dirs` to `report` and then iterating over each target in the report output and passing `--target` to `update`.
### (Environment x Dependency) consolidation
To create pull requests for each dependency within specific environments, combine `--include-dirs` with `--target`.


**Pseudocode:**
```
for each $environment (identified as a glob pattern of folders)
for each $environment e.g., glob patterns like dev-* or prod-*)
run patcher report --include-dirs=$environment
for each $target in the output of patcher report
for each $target output
run patcher update --target=$target
```


**Pseudocode:**
**Pseudocode:**
8 changes: 4 additions & 4 deletions docs/2.0/docs/patcher/concepts/index.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# What is Gruntwork Patcher?

Gruntwork Patcher helps you automatically keep your infrastructure code ([Terragrunt](https://terragrunt.gruntwork.io/) and [OpenTofu](https://opentofu.org/)) up to date, including patching your code to make it work with backward incompatible module releases.
Gruntwork Patcher automates the process of keeping your infrastructure code ([Terragrunt](https://terragrunt.gruntwork.io/) and [OpenTofu](https://opentofu.org/)) up to date, including applying patches to ensure compatibility with backward-incompatible module releases.

Without Patcher, the manual process of discovering updates and determining if they can be safely applied can take hours of an engineer's time for each module dependency.
Manually identifying updates and assessing whether they can be safely applied can consume significant engineering time for each module dependency. Patcher eliminates this inefficiency by streamlining the update process.

You can use Patcher to maintain your dependencies on the Gruntwork IaC Library which includes patches for recent breaking changes to Gruntwork modules. Patcher can also be used to update dependencies on your own modules and dependencies on open source modules using semantic versioning to decide which versions can be safely updated and which require manual intervention.
You can use Gruntwork Patcher to manage dependencies on the Gruntwork IaC Library, which includes patches for recent breaking changes to Gruntwork modules. Patcher also supports updating dependencies for your own modules or open-source projects, using semantic versioning to identify safe updates and highlight those that need manual review.

Gruntwork Patcher provides a straightforward README driven workflow for use when applying breaking changes that don't have patches. Patcher updates the dependency to the version with a breaking changes, generates a README file into the folder that contains the file that has been updated, and then stops. The README will contain the release note for the breaking change. The user reads the README file, takes any action that is required and then deletes the README file before running Patcher again.
Gruntwork Patcher provides a clear, README-driven workflow for breaking changes without available patches. When Patcher detects breaking changes, it updates the relevant dependency to the version containing those changes and generates a README file in the folder with the updated file. This README outlines the release notes and details the breaking changes. Users must review the README, address any necessary actions, and remove the file before rerunning Patcher.
Loading