diff --git a/docs/2.0/docs/pipelines/architecture/actions.md b/docs/2.0/docs/pipelines/architecture/actions.md index bc855323d..b19c70b30 100644 --- a/docs/2.0/docs/pipelines/architecture/actions.md +++ b/docs/2.0/docs/pipelines/architecture/actions.md @@ -10,10 +10,10 @@ When a pull request is created, Pipelines will automatically execute `terragrunt When a pull request is merged, Pipelines will automatically execute either `terragrunt apply` or `terragrunt destroy` on every infrastructure change, depending on the type of infrastructure change. For example, if the pull request deletes a `terragrunt.hcl` file, Pipelines will run `terragrunt destroy`. -## Skipping Runs +## Skipping runs Sometimes you find it necessary to make a change without going through the full pipelines process. This can be accomplished using GitHub's built in method for skipping workflow runs [by adding [skip ci] to your commit message](https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/skipping-workflow-runs). -## Other Actions +## Other actions If you'd like to request a new Pipelines action, please email us at [feedback@gruntwork.io](mailto:feedback@gruntwork.io). diff --git a/docs/2.0/docs/pipelines/architecture/audit-logs.md b/docs/2.0/docs/pipelines/architecture/audit-logs.md index 555633055..28a25a813 100644 --- a/docs/2.0/docs/pipelines/architecture/audit-logs.md +++ b/docs/2.0/docs/pipelines/architecture/audit-logs.md @@ -1,18 +1,18 @@ -# Audit logs +# Audit Logs -Gruntwork Pipelines provides an audit log of which GitHub user performed _what_ operation in your AWS accounts as a result of a [Pipelines Action](/2.0/docs/pipelines/architecture/actions.md). +Gruntwork Pipelines provides an audit log that records which GitHub user performed specific operations in your AWS accounts as a result of a [Pipelines Action](/2.0/docs/pipelines/architecture/actions.md). -Accessing AWS environments from a CI/CD system is commonly done by assuming temporary credentials using [OpenID Connect (OIDC)](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). Usage of shared credentials complicates the task of maintaining an accurate log of who did what in your AWS accounts. Gruntwork Pipelines solves this issue by leveraging [AWS CloudTrail](https://aws.amazon.com/cloudtrail/) and a naming scheme based on context from the triggering Pipelines Action in GitHub. This setup associates every single API operation performed by Gruntwork Pipelines with a GitHub username and a pull request or branch. This allows your security team to quickly triage access related issues and perform analytics on usage patterns of individual users from your version control system in your AWS accounts. +Accessing AWS environments from a CI/CD system often involves assuming temporary credentials using [OpenID Connect (OIDC)](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). Shared credentials can complicate tracking who performed specific actions in AWS accounts. Gruntwork Pipelines addresses this challenge by using [AWS CloudTrail](https://aws.amazon.com/cloudtrail/) with a naming convention that includes context from the triggering Pipelines Action in GitHub. This approach associates every API operation performed by Pipelines with a GitHub username and a specific pull request or branch, enabling your security team to efficiently investigate access-related issues and analyze individual user actions. ## How it works -Gruntwork Pipelines provides an audit log of which user performed what action in which account. To accomplish this, Pipelines sets the [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session name using a combination of the initiating GitHub user, the name of Pipelines itself, and the pull request or branch from which the action was triggered. All logging for Gruntwork Pipelines AWS access is uses [AWS CloudTrail](https://aws.amazon.com/cloudtrail/). Session names are set in the `User name` field in CloudTrail, allowing those searching the data to clearly identify which user performed an action. For more information on querying the logs see [where you can find logs](#where-you-can-find-logs) and [querying data](#querying-data). +Gruntwork Pipelines creates an audit log that tracks which user performed what action in which AWS account. It does this by setting the [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session name to include the initiating GitHub username, the Pipelines name, and the pull request or branch that triggered the action. Logging is handled through [AWS CloudTrail](https://aws.amazon.com/cloudtrail/), where session names appear in the `User name` field, making it easy to identify which user performed an action. For information on locating logs, see [where you can find logs](#where-you-can-find-logs) and [querying data](#querying-data). ### What gets logged -Logs are available for all operations performed in every AWS account by Gruntwork Pipelines. Gruntwork Pipelines takes advantage of [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session names to clearly label all sessions with the GitHub user who requested the change and the pull request or branch that triggered the change. +Logs are generated for all operations performed by Gruntwork Pipelines across every AWS account. These logs leverage [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) session names to clearly label sessions with the GitHub username that requested the change and the associated pull request or branch. -The `userIdentity` field in each CloudTrail event associated with API calls performed by Pipelines [Actions](/2.0/docs/pipelines/architecture/actions.md) will contain the session name. For example, if the GitHub user `SomeUserInYourOrg` made a pull request that was the 123rd pull request in your repository, an associated CloudTrail event would have information that looked like the following in the `userIdentity` field. +Each CloudTrail event linked to API calls from Pipelines [Actions](/2.0/docs/pipelines/architecture/actions.md) includes the session name in the `userIdentity` field. For example, if the GitHub user `SomeUserInYourOrg` initiated the 123rd pull request in your repository, the `userIdentity` field in a corresponding CloudTrail event would provide details such as the following. ```json { @@ -45,26 +45,31 @@ The `userIdentity` field in each CloudTrail event associated with API calls perf } ``` -Due to the 64 character limit for STS session names, we do not include the originating repository. To determine the originating repo, you can search repositories for commits containing the user and branch/PR-number combination, or reach out to the GitHub user directly. +Due to the 64-character limit for STS session names, the originating repository is not included in the session name. To identify the originating repository, you can search through repositories for commits matching the user and branch/PR-number combination or contact the GitHub user directly. -Combined with a [query service](#querying-data), you can use data from CloudTrail to perform analytics on usage of Gruntwork Pipelines in your AWS accounts. +By combining this data with a [query service](#querying-data), you can analyze the usage patterns of Gruntwork Pipelines in your AWS accounts using CloudTrail logs. ### Who gets logged -Pipelines uses a naming scheme that combines the GitHub user who triggered the Pipelines [Action](/2.0/docs/pipelines/architecture/actions.md) and the pull request or branch from which the action was initiated. Pipelines will set the AWS STS session name according to the following format: `-via-GWPipelines@(PR-|)`. +Pipelines employs a naming scheme that integrates the GitHub user who triggered the Pipelines [Action](/2.0/docs/pipelines/architecture/actions.md) along with the pull request or branch that initiated the action. The AWS STS session name is formatted as follows: +`-via-GWPipelines@(PR-|)`. -Pipelines runs Pipelines Actions when a pull request is opened, updated, re-opened, or merged to the deploy branch (e.g., main). The naming scheme will use different values for pull request events and pushes to the deploy branch (e.g. merged PRs). +#### For pull request events +When Pipelines runs in response to a pull request event (opened, updated, or reopened), the session name includes the user who made the most recent commit on the branch and the pull request number assigned by GitHub. For instance: +- If the user `SomeUserInYourOrg` created pull request number `123`, the session name would be: + `SomeUserInYourOrg-via-GWPipelines@PR-123`. -When Pipelines is run in response to a pull request event, the user that created the most recent commit on the branch will be used in the log, in addition to the pull request number assigned by GitHub. For example, if the user `SomeUserInYourOrg` created a pull request that was given the number `123` by GitHub, the resulting session name would be `SomeUserInYourOrg-via-GWPipelines@PR-123`. - -When Pipelines is run in response to a pull request being merged, the user that performed the merge (e.g., the user who clicked the `merge` button on the pull request) will be used in the log in addition the to deploy branch (e.g., `main`). For example, if the user `SomeUserInYourOrg` merged a pull request to your deploy branch named `main` the session name would be `SomeUserInYourOrg-via-GWPipelines@main` +#### For merged pull requests +When Pipelines runs after a pull request is merged, the session name reflects the user who performed the merge (e.g., clicked the `merge` button) and the deploy branch name (e.g., `main`). For example: +- If the user `SomeUserInYourOrg` merged a pull request to the branch `main`, the session name would be: + `SomeUserInYourOrg-via-GWPipelines@main`. ## Where you can find logs -Gruntwork Pipelines leverages AWS CloudTrail to log all actions taken by Pipelines in your AWS account. Due to our naming scheme, identifying operations performed in your AWS account by Gruntwork Pipelines are clearly identified. +Gruntwork Pipelines uses AWS CloudTrail to log all actions performed in your AWS accounts. Thanks to the naming scheme, actions initiated by Gruntwork Pipelines are clearly identified in CloudTrail. -Accessing CloudTrail and querying data is dependent on your organization's policies and settings. If you are a Gruntwork Account Factory customer, see the documentation on [logging](/2.0/docs/accountfactory/architecture/logging) for information on how to access and query your CloudTrail data. +Accessing and querying CloudTrail data depends on your organization's specific policies and configurations. If you are a Gruntwork Account Factory customer, refer to the documentation on [logging](/2.0/docs/accountfactory/architecture/logging) for details on accessing and querying CloudTrail data. ## Querying data -CloudTrail can be configured to automatically store events in an S3 bucket of your choosing. Storing data in S3, when correctly configured, allows you to limit access to only the users that need it. Data in stored in S3 can be searched directly using a query service like [Amazon Athena](https://aws.amazon.com/athena/) or forwarded to other logging systems. +CloudTrail can be configured to automatically store events in an S3 bucket of your choice. When stored in S3 with proper configuration, access can be restricted to authorized users only. You can query stored data using services such as [Amazon Athena](https://aws.amazon.com/athena/) or forward the data to other logging systems for further analysis. diff --git a/docs/2.0/docs/pipelines/architecture/components.md b/docs/2.0/docs/pipelines/architecture/components.md index 70b82931e..24838a9c2 100644 --- a/docs/2.0/docs/pipelines/architecture/components.md +++ b/docs/2.0/docs/pipelines/architecture/components.md @@ -1,23 +1,23 @@ # Components Architecture -There are two major components in pipelines - the orchestrator and the executor. The orchestrator determines the jobs to be executed and the executor actually executes the jobs and makes any necessary updates in AWS. +Pipelines consists of two main components: the orchestrator and the executor. The orchestrator identifies necessary jobs, while the executor performs those tasks and updates AWS resources accordingly. ## Orchestrator -The orchestrator identifies each infrastructure change in a pull request or git commit, classifies the type of infrastructure change it is (e.g. `AccountsAdded`, `ModuleChanged`, `EnvCommonChanged`), and determines the right pipelines actions (e.g., `terragrunt plan`, `apply`, or `destroy`) to run based on that infrastructure change. +The orchestrator analyzes each infrastructure change in a pull request or git commit, categorizes the type of change (e.g., `AccountsAdded`, `ModuleChanged`, `EnvCommonChanged`), and identifies the appropriate pipelines actions (e.g., `terragrunt plan`, `apply`, or `destroy`) to execute based on the type of change. ## Executor -The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added. +The executor receives a pipeline action and an infrastructure change as input and executes the specified action on the change. For example, when responding to `AccountsAdded` events on merge, the executor may create a follow-up pull request in the `infrastructure-live-root` repository to include additional IaC code for baselining the newly added accounts. -## Execution Flow +## Execution flow -Pipelines starts with an event in GitHub - a pull request being created, updated, or re-opened, and a PR being merged to `main` (or any other push to `main`). From there, the orchestrator determines the infra-change set and appropriate action to run for each infra-change. For each change in the infra-change set, the executor runs the appropriate action. The executor then logs the results of the action back in GitHub on the pull request that initiated the flow. +Pipelines begins with an event in GitHub, such as the creation, update, or reopening of a pull request, or a push to `main` (e.g., merging a pull request). The orchestrator determines the set of infrastructure changes (`infra-change set`) and selects the appropriate action for each change. For every change in the set, the executor performs the necessary action and logs the results in GitHub, attaching them to the pull request that triggered the workflow. -## Trust Boundaries +## Trust boundaries -Vital to the architecture of Pipelines is understanding of the trust model inherent to the system. Pipelines is designed to be run in a CI/CD system, which means that it has privileged access to your AWS accounts. +A critical aspect of Pipelines' architecture is understanding its trust model. Since Pipelines runs within a CI/CD system, it has privileged access to your AWS accounts. -One principle to keep in mind is that anyone with trust to edit code on the `main` branch of your repositories is trusted to make the corresponding changes in your AWS accounts. This is why we recommend that you follow the [Repository Access](/2.0/docs/pipelines/installation/viamachineusers#repository-access) guidelines to ensure that only the right people have the right access to your repositories. +Anyone with the ability to edit code in the `main` branch of your repositories inherently has the authority to make corresponding changes in your AWS accounts. For this reason, it is important to follow the [Repository Access](/2.0/docs/pipelines/installation/viamachineusers#repository-access) guidelines to ensure appropriate access control. -In addition, each AWS IAM role provisioned as part of DevOps Foundations only trusts a single repository (and for apply roles, only a single branch). If you find that you are expanding the permissions of a given role too wide, you should consider creating a new role that has more granular permissions for the specific use-case. Utilize the `infrastructure-live-access-control` repository to support that need. +Additionally, each AWS IAM role provisioned through DevOps Foundations is configured to trust a single repository (and, for apply roles, a single branch). If a role's permissions become overly broad, consider creating a new role with more granular permissions tailored to the specific use case. Use the `infrastructure-live-access-control` repository to define and manage these roles. diff --git a/docs/2.0/docs/pipelines/architecture/github-workflows.md b/docs/2.0/docs/pipelines/architecture/github-workflows.md index 8e0482a35..00437bf93 100644 --- a/docs/2.0/docs/pipelines/architecture/github-workflows.md +++ b/docs/2.0/docs/pipelines/architecture/github-workflows.md @@ -1,6 +1,6 @@ # GitHub Workflows -Pipelines integrates via GitHub Workflows in your repositories that integrate with [Reusable Workflows](https://docs.github.com/en/actions/sharing-automations/reusing-workflows) in Gruntwork's [pipelines-workflows](https://github.com/gruntwork-io/pipelines-workflows) repository. The workflows in your repositories depend on the Gruntwork workflows via the `uses` clause within a job, this will appear like: +Pipelines integrates with your repositories through GitHub Workflows, leveraging [Reusable Workflows](https://docs.github.com/en/actions/sharing-automations/reusing-workflows) from Gruntwork's [pipelines-workflows](https://github.com/gruntwork-io/pipelines-workflows) repository. The workflows in your repositories rely on Gruntwork workflows through the uses clause within a job. This is structured as follows: ```yml jobs: @@ -8,47 +8,44 @@ jobs: uses: gruntwork-io/pipelines-workflows/.github/workflows/pipelines-root.yml@v3 ``` -## Workflow Versioning +## Workflow versioning -Gruntwork uses [Semantic Versioning](https://semver.org/) for `pipelines-workflows` releases. We create new git tags to track new releases of the workflows following the `v.MAJOR.MINOR.PATCH` format. We also publish a major tag e.g. `v.MAJOR` that we update to reference the latest release within that version. For example when we publish a patch from `v3.0.1` to `v3.0.2` we will update the `v3` tag to reference the newer version. +Gruntwork follows [Semantic Versioning](https://semver.org/) for `pipelines-workflows` releases. New releases are tracked using git tags in the `v.MAJOR.MINOR.PATCH` format. A major tag, such as `v.MAJOR`, is also maintained and updated to point to the latest release within that major version. For example, when releasing a patch update from `v3.0.1` to `v3.0.2`, the `v3` tag will be updated to reference the newer version. -When calling a workflow the final part of the `uses` clause contains the version reference. E.g. `pipelines-root.yml@v3`. We recommend using the major version e.g. `v3` within your workflows to receive the latest fixes and performance improvements, but you are also able to pin to a specific tag. +When referencing a workflow, the version is specified in the `uses` clause. For example: `pipelines-root.yml@v3`. Using the major version, like v3, in your workflows ensures you receive the latest updates and performance enhancements.. However, you can choose to pin to a specific version if needed. -## Modifying Workflows +## Modifying workflows -If you make modifications to the workflows in _your_ repositories, these changes will only affect that specific repository. E.g. if you want to add a customization to the `pipelines.yml` in your `infrastructure-live-root` repository it will only affect this repository, and not your vended delegated repositories. +Changes made to workflows in your repositories only affect the specific repository where the modification occurs. For instance, customizing the `pipelines.yml` workflow in your `infrastructure-live-root` repository will not impact workflows in other repositories, such as delegated repositories. -If you choose to [Fork the Gruntwork Workflows](https://docs.gruntwork.io/2.0/docs/pipelines/guides/extending-pipelines#extend-the-github-actions-workflow) you will be able to make changes that affect multiple repositories. It's important to understand exactly which workflows in the `pipelines-workflows` repository affect which of your repositories. See a full list of dependencies below. +If you [fork the Gruntwork Workflows](https://docs.gruntwork.io/2.0/docs/pipelines/guides/extending-pipelines#extend-the-github-actions-workflow), you can make changes that affect multiple repositories. Be sure to understand the dependencies between workflows in the `pipelines-workflows` repository and your repositories. The dependencies are detailed below. -## Workflow Dependencies +## Workflow dependencies -The Gruntwork pipelines-workflows repository contains the following reusable workflows: +The `pipelines-workflows` repository includes the following reusable workflows: -- `pipelines-drift-detection.yml` - A workflow for [Pipelines Drift Detection](/2.0/docs/pipelines/concepts/drift-detection), used by all repositories where Drift Detection is installed. -- `pipelines-root.yml` - The core Pipelines workflow used only by your `infrastructure-live-root` repository. This workflow provides core plan/apply as well as account vending functionality. -- `pipelines-unlock.yml` - A workflow used to manually unlock state files, used by all repositories. -- `pipelines.yml` - The core Pipelines workflow used by your `infrastructure-live-access-control` and delegated repositories for plan/apply operations. +- `pipelines-drift-detection.yml` - Used for [Pipelines Drift Detection](/2.0/docs/pipelines/concepts/drift-detection) in all repositories with Drift Detection installed. +- `pipelines-root.yml` - The core Pipelines workflow for the `infrastructure-live-root` repository, providing core plan/apply functionality and account vending. +- `pipelines-unlock.yml` - Used to manually unlock state files in all repositories. +- `pipelines.yml` - The core Pipelines workflow for `infrastructure-live-access-control` and delegated repositories, supporting plan/apply operations. - -In your repositories you will have the following set of workflows: +In your repositories, the following workflows are typically present: #### infrastructure-live-root -- `account-factory.yml` - A standalone workflow that does not use a workflow from `pipelines-workflows`. -- `pipelines-drift-detection.yml` (enterprise only) - Uses the Gruntwork `pipelines-drift-detection.yml` drift detection workflow. -- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` unlock workflow. +- `account-factory.yml` - A standalone workflow independent of `pipelines-workflows`. +- `pipelines-drift-detection.yml` (Enterprise only) - Uses the Gruntwork `pipelines-drift-detection.yml` workflow. +- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` workflow. - `pipelines.yml` - Uses `pipelines-root.yml`. - #### infrastructure-live-access-control -- `pipelines-drift-detection.yml` - (enterprise only) - Uses the Gruntwork `pipelines-drift-detection.yml` drift detection workflow. -- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` unlock workflow. +- `pipelines-drift-detection.yml` (Enterprise only) - Uses the Gruntwork `pipelines-drift-detection.yml` workflow. +- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` workflow. - `pipelines.yml` - Uses `pipelines.yml`. #### infrastructure-live-delegated ([Vended Delegated Repositories](/2.0/docs/accountfactory/guides/delegated-repositories)) -- `pipelines-drift-detection.yml` - Uses the Gruntwork `pipelines-drift-detection.yml` drift detection workflow. -- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` unlock workflow. +- `pipelines-drift-detection.yml` - Uses the Gruntwork `pipelines-drift-detection.yml` workflow. +- `pipelines-unlock.yml` - Uses the Gruntwork `pipelines-unlock.yml` workflow. - `pipelines.yml` - Uses `pipelines.yml`. - diff --git a/docs/2.0/docs/pipelines/architecture/index.md b/docs/2.0/docs/pipelines/architecture/index.md index c84ba03d2..1794afe91 100644 --- a/docs/2.0/docs/pipelines/architecture/index.md +++ b/docs/2.0/docs/pipelines/architecture/index.md @@ -1,24 +1,25 @@ # Architecture -Gruntwork Pipelines is designed to allow you to utilize the components you need in order to manage your infrastructure in a way that makes sense for your organization. +Gruntwork Pipelines is designed to provide flexibility, enabling you to utilize the components you need to manage your infrastructure in a way that aligns with your organization's requirements. + -Understanding what's included, and how it's structured will give you a better idea of how to use the Pipelines and associated IaC effectively. +Understanding the components and their structure will help you use Pipelines and associated Infrastructure as Code (IaC) effectively. ## `infrastructure-live-root` -The `infrastructure-live-root` repository is the root of your infrastructure. It's provisioned using the [infrastructure-live-root-template](https://github.com/gruntwork-io/infrastructure-live-root-template). +The `infrastructure-live-root` repository serves as the root of your infrastructure and is provisioned using the [infrastructure-live-root-template](https://github.com/gruntwork-io/infrastructure-live-root-template) template repository. -This repository is where you'll manage sensitive resources like the Landing Zone and Organization services for AWS. This repository is typically only given access to a small number of trusted users in your organization. +This repository is where you manage sensitive resources such as the Landing Zone and Organization services for AWS. Typically, access to this repository is restricted to a small group of trusted users. -All of the other infrastructure that is managed using Gruntwork software will ultimately rely on something managed in this repository to function. +All other infrastructure managed with Gruntwork software ultimately depends on resources configured in this repository. ### Workflows -- **Account Factory** - This workflow provides a convenient API for interacting with Gruntwork Account Factory. - +- **Account Factory:** Provides an API for interacting with the Gruntwork Account Factory. It uses a [repository dispatch](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#repository_dispatch) to create AWS account requests. + This workflow uses a [repository dispatch](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#repository_dispatch) to create a standard AWS account creation request in the repository. - The payload for the workflow dispatch is a JSON object that can be conveniently constructed using the sample HTML file located next to the workflow file in the repository. + The workflow payload is a JSON object, which can be constructed using the sample HTML file included in the repository. This file can be customized for organizational needs, such as adding tagging fields or additional context. :::tip @@ -30,23 +31,20 @@ All of the other infrastructure that is managed using Gruntwork software will ul ::: -- **Pipelines** - This workflow will be used by the `infrastructure-live-root` repository to manage infrastructure in response to changes in the repository in a GitOps fashion. - - While the permissions for this workflow are largely read-only when proposing infrastructure changes, the workflow will have the ability to make changes to relevant infrastructure when the changes are merged. +- **Pipelines:** Manages infrastructure changes in a GitOps fashion. While the workflow permissions are mostly read-only for proposing changes, they include write access to apply infrastructure changes upon merging. :::tip -The `infrastructure-live-root` repository does not have to be named "infrastructure-live-root". You can customize the name during the bootstrap process. It is highly recommended that the repository is named something similar to `infrastructure-live-root` to make it clear what it is when reading Gruntwork documentation, however. - +The `infrastructure-live-root` repository can be renamed during the bootstrap process, but giving it a similar name to `infrastructure-live-root` is recommended for clarity when using Gruntwork documentation. Multiple root repositories can be created if needed, but this increases complexity and operational overhead. It also doesn't have to be the only root repository in your organization. You can have multiple root repositories if you have a complex organization structure that requires it. Make sure to evaluate the trade-offs of having multiple root repositories before doing so. It can be a significant source of complexity and operational overhead. ::: ## `infrastructure-live-access-control` -The `infrastructure-live-access-control` repository is used to manage access control for your infrastructure. It's provisioned when the "Bootstrap the infrastructure-access-control repository" checkbox is checked when running the Bootstrap workflow in the `infrastructure-live-root` repository. +The `infrastructure-live-access-control` repository manages access control for your infrastructure and is provisioned during the bootstrap process in the `infrastructure-live-root` repository. While only necessary for Enterprise customers, it is recommended for all users. -This repository is only necessary for Enterprise customers, but is recommended for all customers. It's where you'll manage IAM roles, policies, and permissions for *delegated* infrastructure management in your organization. +This repository handles IAM roles, policies, and permissions for delegated infrastructure management. It allows application engineers to propose access control changes, while a central platform team reviews and approves them. More access can be granted to this repository than the `infrastructure-live-root` repository, but it should still be treated as a sensitive resource. Organizations typically find it useful to have restrictions on who can approve and merge changes to this repository, and allow for users to propose changes in a self-service fashion. This allows for application workload engineers to propose changes to the access control their workflows need, while allowing a central platform team to review and approve those changes instead of having to author the changes themselves. @@ -80,25 +78,22 @@ It also doesn't have to be the only access control repository in your organizati ## `infrastructure-catalog` -The `infrastructure-catalog` repository is used to store OpenTofu/Terraform modules that your organization has authored and intends to use within your organization. It's provisioned when the "Bootstrap the infrastructure-catalog repository" checkbox is checked when running the Bootstrap workflow in the `infrastructure-live-root` repository. +The `infrastructure-catalog` repository stores OpenTofu/Terraform modules created for internal use. It is optionally provisioned during the bootstrap process of the `infrastructure-live-root` repository. -This repository is optional, but is recommended for all customers. It's where you'll store reusable infrastructure code that can be shared across your organization. - -A common pattern used by customers is to leverage the Gruntwork IaC library whenever possible, and to author custom modules here if there's ever a need to extend or modify the behavior of Gruntwork modules. +This repository is optional but recommended for managing reusable infrastructure code. Customers often combine Gruntwork modules with custom modules stored here to extend functionality. :::tip -The `infrastructure-catalog` repository does not have to be named "infrastructure-catalog". You can customize the name during the bootstrap process. It is highly recommended that the repository is named something similar to `infrastructure-catalog` to make it clear what it is when reading Gruntwork documentation, however. +While `infrastructure-catalog` can be renamed, keeping a consistent name is recommended for clarity in documentation. Multiple module repositories can be created if necessary, but consider the trade-offs between centralized and decentralized approaches. -It also doesn't have to be the only modules repository in your organization. You can have multiple modules repositories if you have a complex organization structure that requires it. Make sure to evaluate the trade-offs of having multiple modules repositories before doing so. It can be advantageous to have one repository for all modules to make it easier to find and share modules across your organization. However, it can also be advantageous to have multiple repositories if you have different teams that need to manage their own modules, or want to have different modules available to different teams within your organization. +It can be advantageous to have one repository for all modules to make it easier to find and share modules across your organization. However, it can also be advantageous to have multiple repositories if you have different teams that need to manage their own modules, or want to have different modules available to different teams within your organization. ::: ### Workflows -- **Tests** - This workflow will be used by the `infrastructure-catalog` repository to run tests on the modules in the repository. This workflow is typically run on every push to ensure that the modules are functioning as expected by provisioning them in real environments, running [Terratests](https://github.com/gruntwork-io/terratest) against them, then tearing them down. +- **Tests:** Validates module functionality by provisioning them in real environments, running [Terratests](https://github.com/gruntwork-io/terratest), and tearing them down. This workflow ensures modules work as expected. - The configurations for this workflow are configured out of the box to run tests on the examples written in the repository. ## `infrastructure-live-delegated` @@ -107,8 +102,7 @@ One of the primary benefits of IaC Foundations is the streamlined delegation of These repositories can be created manually by customers for specific purposes. For example, an application repository may need permissions to build and push a container image to AWS ECR, or to deploy an update to an ECS service. These permissions can be delegated by the `infrastructure-live-access-control` repository to a specific repository that needs those permissions. Enterprise customers can also expect the creation and management of delegated repositories centrally in the `infrastructure-live-root` repository. This is an Enterprise-only feature that allows for the creation of delegated repositories with largely the same degree of infrastructure management as the `infrastructure-live-root` repository itself. This is useful for organizations that want to allow large degrees of infrastructure management (e.g. entire AWS accounts) without having to manually provision and manage the repositories that need those permissions. - -## Entity Relationship Diagram +## Entity relationship diagram ```mermaid erDiagram @@ -122,14 +116,14 @@ erDiagram :::note -`infrastructure` is shortened to `infra` in the diagram for brevity. +The term `infrastructure` is abbreviated as `infra` in the diagram for simplicity. ::: -The central `infrastructure-live-root` repository is the root of all infrastructure management. +The `infrastructure-live-root` repository serves as the central hub for managing all infrastructure. -If users opt-in to delegating access control to the `infrastructure-live-access-control` repository, they can provision the relevant `pipelines-access-control` roles in AWS accounts where they want to delegate access control using the `infrastructure-live-root` repository, then manage access control using those roles in the `infrastructure-live-access-control` repository. +- Users who choose to delegate access control can use the `infrastructure-live-root` repository to provision the necessary `pipelines-access-control` roles in AWS accounts. These roles allow access control to be managed within the `infrastructure-live-access-control` repository. -If users opt-in to delegating infrastructure management to the `infrastructure-live-delegated` repositories, they can provision the relevant `pipelines-delegated` roles in AWS accounts where they want to delegate infrastructure management using the `infrastructure-live-access-control` repository, then manage infrastructure using those roles in the `infrastructure-live-delegated` repositories. +- Users who opt to delegate infrastructure management can use the `infrastructure-live-access-control` repository to provision the required `pipelines-delegated` roles in AWS accounts. These roles enable infrastructure management to be handled through `infrastructure-live-delegated` repositories. -If users opt-in to creating and managing modules centrally, they can create and manage modules in the `infrastructure-catalog` repository. This repository can be used by any repository in the organization to share reusable, vetted modules. +- Users who prefer to centralize module management can utilize the `infrastructure-catalog` repository to create and maintain reusable, vetted modules. These modules can be shared across the organization and accessed by any repository as needed. diff --git a/docs/2.0/docs/pipelines/architecture/security-controls.md b/docs/2.0/docs/pipelines/architecture/security-controls.md index 372fc753a..e111e2d9b 100644 --- a/docs/2.0/docs/pipelines/architecture/security-controls.md +++ b/docs/2.0/docs/pipelines/architecture/security-controls.md @@ -1,54 +1,49 @@ # Controls -Pipelines takes a defense in depth approach to securing workflows. This document provides an overview of the controls that Pipelines employs to ensure that only infrastructure that has been written in code and approved by a reviewer can be deployed in your AWS accounts. +Pipelines employs a defense-in-depth approach to secure workflows. This document outlines the controls Pipelines uses to ensure that only infrastructure written in code and approved by a reviewer can be deployed to your AWS accounts. ## Least privilege principle -Pipelines leverages the principle of least privilege to ensure that only the necessary permissions are used when performing infrastructure actions. +Pipelines adheres to the principle of least privilege, granting only the necessary permissions for infrastructure actions. -By default, the only repository needed to interact with infrastructure using Pipelines as part of DevOps Foundations is the `infrastructure-live-root` repository. This repository contains the infrastructure code that controls your management, logs, security and shared accounts. Access to this repository should be limited to a small group of trusted individuals who are responsible for defining critical infrastructure for your organization (think of the `root` user in Unix systems). +By default, the only repository required to interact with infrastructure using Pipelines in DevOps Foundations is the `infrastructure-live-root` repository. This repository contains infrastructure code for management, logs, security, and shared accounts. Access to this repository should be limited to a small, trusted group responsible for defining critical infrastructure, similar to the role of the `root` user in Unix systems. -The [AWS IAM role assumed via OIDC](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) by Pipelines when pull requests are opened or updated in this repository has a trust policy that limits access to the repository itself, and only has read permissions. This means that interaction with AWS resources when opening or updating pull requests is limited to reading the current state of infrastructure in AWS. This is important for ensuring that all updates to infrastructure are properly reviewed. +- The [AWS IAM role assumed via OIDC](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) when pull requests are opened or updated in this repository has a trust policy that restricts access to the repository itself and provides read-only permissions. This guarantees that all infrastructure updates are reviewed before deployment. +- The AWS IAM role assumed via OIDC when pull requests are merged into the `main` branch has a trust policy limiting access to the repository’s `main` branch and granting write permissions. This allows changes to be applied to your AWS infrastructure. To prevent unauthorized changes, ensure that only trusted individuals have write access to the `main` branch and implement appropriate branch protections. -The AWS IAM role assumed via OIDC by Pipelines when pull requests are merged to the `main` branch has a trust policy that limits access to the repository itself on the `main` branch, and has write permissions. This is what allows merged pull requests to make changes to your AWS infrastructure. As a consequence, you should ensure that only trusted individuals have write access to the `main` branch of the `infrastructure-live-root` repository, and that sufficient protections are in place to prevent unauthorized changes to that branch. +## Infrastructure access control -## Infrastructure Access Control +An optional `infrastructure-live-access-control` repository can manage access control for infrastructure provisioned in AWS accounts. Using this repository is a best practice for centralized and auditable access management. -In addition to the `infrastructure-live-root` repository, an optional dedicated repository for access control (the `infrastructure-live-access-control` repository) can be used to manage access to infrastructure provisioned in your AWS accounts as part of DevOps Foundations. Using this repository is a recommended best practice, as it allows you to manage access to your infrastructure in a centralized and auditable way. +- Access to the `main` branch of the `infrastructure-live-access-control` repository should be restricted to a small, trusted group managing infrastructure access. +- The same least privilege principles apply: roles assumed by Pipelines for pull requests have read-only permissions, while roles for merged pull requests have write permissions. -The same principles of least privilege apply to the `infrastructure-live-access-control` repository. Access to the `main` branch of this repository should be limited to a small group of trusted individuals who are responsible for managing access to your infrastructure. Similar restrictions exist for the two roles assumed by Pipelines when interacting with this repository, with read permissions for pull requests and write permissions for merged pull requests. - -One key difference between the `infrastructure-live-root` and `infrastructure-live-access-control` repositories is that the `infrastructure-live-access-control` repository is used to manage access to your infrastructure, rather than defining the infrastructure itself. This might mean that you want to grant write access to the repository to a broader set of individuals who are responsible for managing access to your infrastructure, while retaining the same restrictions on the `main` branch. - -This encourages a dialogue between the individuals responsible for defining critical infrastructure that keeps your AWS accounts functional and secure, and the individuals responsible for deploying workloads to those accounts. Consider encouraging a culture of collaboration between these groups to ensure that access control is continuously reviewed, refined and restricted to the minimum necessary permissions for workloads to be deployed. +Unlike the `infrastructure-live-root` repository, this repository focuses on managing access control rather than defining infrastructure. You might grant write access to a broader group for managing access while maintaining tight control over the `main` branch. Encourage collaboration between platform teams and application engineers to review and refine access control continuously. ## Token strategy -Gruntwork Pipelines (when setup via Machine Users and not the GitHub App) uses a series of GitHub Personal Access Tokens (PAT) to allow cross-repository code and workflow access. This approach ensures that each token has the minimal required permissions to perform its tasks. The full list of tokens and required permissions are listed below. - -- `PIPELINES_READ_TOKEN` - A [classic PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with repository read access to your `infrastructure-live` and `infrastructure-modules` repositories, as well as Gruntwork IaC Library modules and the Pipelines CLI. -- `INFRA_ROOT_WRITE_TOKEN` - A [fine-grained PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) that has read and write access to your `infrastructure-live-root` repository. This token is used in Pipelines to create pull requests after code generation and add pull request comments. -- `ORG_REPO_ADMIN_TOKEN` - A fine-grained PAT that can update all repositories in your GitHub organization. It is used by Pipelines to do the following: - - Bootstrap repositories during DevOps Foundations setup. If you are a DevOps Foundations Teams customer, you can delete this token once it is no longer needed. - - Enterprise only: This token is also used by Pipelines to create new repositories for delegated management of infrastructure, and to create pull requests that baseline those repositories. - -Steps to create a PAT can be found in the official documentation. Refer to [creating a personal access token classic](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) and [creating a fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token), respectively. +Gruntwork Pipelines (when set up via Machine Users rather than the GitHub App) uses GitHub Personal Access Tokens (PATs) to enable cross-repository access. Each token is scoped with minimal permissions required for its purpose. -To learn more about GitHub PATs, refer to their documentation on [managing personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens). +### Required tokens +- `PIPELINES_READ_TOKEN`: A [classic PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with read access to `infrastructure-live`, `infrastructure-modules`, Gruntwork IaC Library modules, and the Pipelines CLI. +- `INFRA_ROOT_WRITE_TOKEN`: A [fine-grained PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) with read/write access to `infrastructure-live-root`. Used to create pull requests and add pull request comments. +- `ORG_REPO_ADMIN_TOKEN`: A fine-grained PAT with admin access to repositories in your GitHub organization. Used for DevOps Foundations setup, including bootstrapping repositories and, for Enterprise customers, creating repositories for delegated infrastructure management. -## AWS Credentials +Refer to [creating a personal access token classic](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) and [creating a fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token) for setup instructions. -Pipelines requires the use of IAM Roles configured with a trust policy to allow GitHub Actions to use OpenId Connect (OIDC) with to run actions in your AWS accounts. This prevents the requirement of creating long-lived AWS credentials and storing them as secrets in GitHub Actions. At execution time, Pipelines exchanges the GitHub OIDC token for short-lived AWS credentials generated using AWS STS to run actions in your AWS accounts that only have requisite permissions for the operations they need to perform. +## AWS credentials -These credentials, although temporary, should still be treated as secrets. The Pipelines role in the `infrastructure-live-root` has administrative permissions to many AWS resources to allow you to deploy many resources without needing to update the policy. +Pipelines requires IAM roles configured with trust policies to use OpenID Connect (OIDC) with GitHub Actions. This eliminates the need for long-lived AWS credentials stored as secrets. During execution, Pipelines exchanges a GitHub OIDC token for temporary AWS credentials generated with AWS STS, scoped to the necessary permissions for the operation. -Leverage the `infrastructure-live-access-control` repository to grant limited access to select infrastructure from other repositories. +While temporary, these credentials should still be treated as sensitive. The Pipelines role in the `infrastructure-live-root` repository typically has administrative permissions for deploying resources. Use the `infrastructure-live-access-control` repository to grant more limited permissions for specific cases. ### OpenID Connect -Pipelines provisions an OpenId Connect identity provider in AWS IAM in every account, setting up GitHub as the provider and setting the audience to AWS STS and your GitHub organization. To learn more about provisioning OpenId Connect providers in AWS, refer to the [official documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html). +Pipelines provisions an OpenID Connect identity provider in AWS IAM for each account, setting GitHub as the provider and restricting the audience to AWS STS and your GitHub organization. The Pipelines IAM role's trust policy ensures: +- Only a single repository in your GitHub organization can assume the role for plans. +- Only a single branch can assume the role for applies/destroys. -The Pipelines IAM role uses a trust policy to limit access to assume the role to a single repository in your GitHub Organization for plans, and an additional restriction requiring that a single branch can access the role used for applies/destroys. The OpenID Connect provider is set as the principal, and a condition is added to the limit access to the relevant repository in your GitHub organization. An example of the trust policy can be found below. +For more details, see the [official AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html). Below is an example of a trust policy used by Pipelines. ```json { @@ -71,80 +66,54 @@ The Pipelines IAM role uses a trust policy to limit access to assume the role to } ``` -Refer to [Configuring OpenId Connect in Amazon Web Services](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) to learn more. - -### Roles Provisioned By DevOps Foundations - -Pipelines will automatically look for certain roles in AWS accounts to assume when executing certain actions. These are roles that are provisioned as part of DevOps Foundations to ensure that Pipelines can perform necessary infrastructure operations. - -These roles follow the naming convention `-pipelines-`. +Refer to [Configuring OpenId Connect in Amazon Web Services](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) for additional details. -e.g. The `root-pipelines-plan` role is assumed by Pipelines to plan changes in the `infrastructure-live-root` repository. +### Roles provisioned by DevOps Foundations -All of the roles provisioned by DevOps Foundations are meant to be assumed by Pipelines for operation in a single repository, and have a trust policy that only allows GitHub Actions workflows triggered by that repository to assume the role. +Pipelines automatically provisions specific roles in AWS accounts to support required infrastructure operations. These roles follow the naming pattern `-pipelines-`. -In addition, these roles are always provisioned in pairs, a `plan` role and an `apply` role. +For example: +- The `root-pipelines-plan` role is used by Pipelines to plan changes in the `infrastructure-live-root` repository. -- `plan` roles are used to plan changes to infrastructure. They have read-only permissions to AWS resources, and are used to execute Terragrunt plans for open Pull Requests. They are allowed to be assumed by workflows started by any branch in the repository that trusts them. -- `apply` roles are used to apply changes to infrastructure. They have read/write permissions to AWS resources, and are used to execute Terragrunt apply and destroy actions for merged Pull Requests and direct pushes to the deploy branch (usually `main`). They are only allowed to be assumed by workflows started by changes to the deploy branch in the repository that trusts them. +These roles are designed to operate in a single repository and include a trust policy that only permits GitHub Actions workflows triggered by that repository to assume the role. Each role is provisioned in pairs: +- `plan` roles, with read-only permissions, are used to execute Terragrunt plans for open pull requests. +- `apply` roles, with read/write permissions, are used to apply or destroy infrastructure changes for merged pull requests or direct pushes to the deploy branch (commonly `main`). -Breaking these permissions up into two roles ensures that controls like [branch protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) and [CODEOWNERS files](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) can be used to ensure that only appropriately privileged parties can drive changes to your infrastructure. - -The explicit enumeration of these roles are as follows: +This separation ensures that controls like [branch protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) and [CODEOWNERS files](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) can effectively govern infrastructure changes. #### `root-pipelines-plan` -The read-only plan role for the `infrastructure-live-root` repository. - -These are the first roles created when setting up DevOps Foundations, and exist in every account. The initial set of these are provisioned manually by the customer setting up DevOps Foundations, and is how all of the other roles get bootstrapped. - -These roles are generally the most permissive of the (read-only) roles created by DevOps Foundations, as they are used to handle everything involved in setting up an AWS account for use by other resources. +A read-only plan role for the `infrastructure-live-root` repository. +- This role is one of the first created when setting up DevOps Foundations. It is provisioned manually by the customer during the platform setup process. +- It exists in all accounts and handles tasks necessary for setting up AWS accounts. +- These roles are highly permissive among read-only roles as they manage foundational AWS account setups. #### `root-pipelines-apply` -The read/write apply role for the `infrastructure-live-root` repository. - -These are the first roles created when setting up DevOps Foundations, and exist in every account. The initial set of these are provisioned manually by the customer setting up DevOps Foundations, and is how all of the other roles get bootstrapped. - -These roles are generally the most permissive of the roles created by DevOps Foundations, as they are used to handle everything involved in setting up an AWS account for use by other resources. +A read/write apply role for the `infrastructure-live-root` repository. +- Like the plan role, this is one of the initial roles created during setup. +- It is broadly permissive to support foundational account setups and bootstrapping. #### `access-control-pipelines-plan` -The read-only plan role for the `infrastructure-live-access-control` repository. - -These are created as part of vending any new account, and are intentionally absent from the _core_ AWS accounts: - -- `management` -- `logs` -- `security` -- `shared` - -These roles are used by the `infrastructure-live-access-control` repository to grant external repositories access to infrastructure in vended accounts, and thus are not needed in the core accounts by default. - -These roles have (read-only) permissions to manage IAM roles and policies within the accounts they are provisioned in. +A read-only plan role for the `infrastructure-live-access-control` repository. +- These roles are provisioned for new accounts but are not included in core accounts such as `management`, `logs`, `security`, or `shared`. +- They manage IAM roles and policies for vended accounts, facilitating infrastructure access control. #### `access-control-pipelines-apply` -The read/write apply role for the `infrastructure-live-access-control` repository. - -These are created as part of vending any new account, and are intentionally absent from the _core_ AWS accounts: - -- `management` -- `logs` -- `security` -- `shared` - -These roles are used by the `infrastructure-live-access-control` repository to grant external repositories access to infrastructure in vended accounts, and thus are not needed in the core accounts by default. - -These roles have permissions to manage IAM roles and policies within the accounts they are provisioned in. +A read/write apply role for the `infrastructure-live-access-control` repository. +- Similar to the plan role, these roles are provisioned for vended accounts but excluded from core accounts. +- They have permissions to manage IAM roles and policies for the accounts where they are provisioned. #### `delegated-pipelines-plan` -The read-only plan role for delegated repositories. +A read-only plan role for delegated repositories, used by Pipelines Enterprise customers. -The `infrastructure-live-access-control` repository is vended ready to provision these roles with the configuration baked in. +- These roles are pre-configured to have minimal permissions, primarily for managing OpenTofu/Terraform state. +- A pull request will be opened in `infrastructure-live-access-control` during provisioning include documentation for adding additional permissions if necessary. +- Users should ensure that only the necessary _read-only_ permissions are granted for the specific delegated repository. -Enterprise customers will see pull requests opened automatically in the `infrastructure-live-access-control` repository to vend these roles for every new account vended as a delegated account. Given that delegated accounts have their own repositories that are separate from the main `infrastructure-live-root` and `infrastructure-live-access-control` repositories, these roles are used to ensure that the delegated repositories have adequate access to the infrastructure they need to manage. :::note @@ -156,16 +125,11 @@ It is up to the user provisioning these roles to ensure that this role has only #### `delegated-pipelines-apply` -The read/write apply role for delegated repositories. - -The `infrastructure-live-access-control` repository is vended ready to provision these roles with the configuration baked in. - -Enterprise customers will see pull requests opened automatically in the `infrastructure-live-access-control` repository to vend these roles for every new account vended as a delegated account. Given that delegated accounts have their own repositories that are separate from the main `infrastructure-live-root` and `infrastructure-live-access-control` repositories, these roles are used to ensure that the delegated repositories have adequate access to the infrastructure they need to manage. +A read/write apply role for delegated repositories. +- Similar to the plan role, these roles are pre-configured with minimal permissions and are intended for managing OpenTofu/Terraform state. +- A pull request will be opened in `infrastructure-live-access-control` during provisioning include documentation for adding additional permissions if necessary. +- Users must ensure that the role has only the necessary _read/write_ permissions required for the delegated repository. :::note - -These roles have almost no permissions by default. They are pre-configured by default to only have access to OpenTofu/Terraform state, and the pull requests that are opened to provision them include documentation on how to add additional permissions as appropriate. - -It is up to the user provisioning these roles to ensure that this role has only the necessary _read/write_ permissions required to manage infrastructure changes relevant to the delegated repository. - -::: +The `delegated-pipelines-plan` and `delegated-pipelines-apply` roles are automatically provisioned for new delegated accounts. Enterprise customers will see pull requests created in the `infrastructure-live-access-control` repository to vend these roles with proper configurations. +::: \ No newline at end of file diff --git a/docs/2.0/docs/pipelines/architecture/usage-data.md b/docs/2.0/docs/pipelines/architecture/usage-data.md index 2b6da20bf..7bc78674b 100644 --- a/docs/2.0/docs/pipelines/architecture/usage-data.md +++ b/docs/2.0/docs/pipelines/architecture/usage-data.md @@ -1,3 +1,3 @@ # Usage Data -Gruntwork Pipelines captures usage data to better understand how our customers use our products. This information includes the duration of pipelines runs, the number of jobs you run, customer name, and application errors that occur during execution. If you would like to disable telemetry, please set the environment variable `PIPELINES_TELEMETRY_OPT_OUT=1` in your CI job. +Gruntwork Pipelines collects usage data to gain insights into how customers interact with the product. This data includes information such as the duration of pipeline runs, the number of jobs executed, the customer name, and any application errors encountered during execution. \ No newline at end of file diff --git a/docs/2.0/docs/pipelines/concepts/cloud-auth.md b/docs/2.0/docs/pipelines/concepts/cloud-auth.md index 1a0129d65..cf9594570 100644 --- a/docs/2.0/docs/pipelines/concepts/cloud-auth.md +++ b/docs/2.0/docs/pipelines/concepts/cloud-auth.md @@ -1,20 +1,20 @@ # Authenticating to the Cloud -## Authenticating to AWS Accounts +## Authenticating to AWS accounts -Pipelines can intelligently figure out which AWS account to authenticate to based on the infrastructure changes proposed in your Pull Request. +Pipelines automatically determines which AWS account to authenticate to based on the infrastructure changes proposed in your pull request. ### How Pipelines authenticates to AWS -To perform the actions that Pipelines detects, each AWS account needs to assume an AWS IAM Role using [OIDC](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). To explain OIDC at a high level, AWS considers GitHub an "identity provider," trusts GitHub's request for a temporary IAM Role, and then issues AWS credentials that are valid for the duration of a GitHub Action workflow. +To execute the actions detected by Pipelines, each AWS account must assume an AWS IAM Role using [OIDC](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). At a high level, OIDC works as follows: AWS recognizes GitHub as an "identity provider," trusts GitHub’s request to assume a temporary IAM Role, and then issues AWS credentials valid for the duration of the GitHub Actions workflow. -Every time you create a new AWS account, you need to update the AWS OIDC configuration to create an IAM role for this account and allow it to be assumed by GitHub. When you use [Gruntwork Account Factory](/2.0/docs/accountfactory/architecture), this role is automatically created when [adding a new AWS account](/2.0/docs/accountfactory/guides/vend-aws-account). +When creating a new AWS account, it is necessary to update the AWS OIDC configuration to include an IAM role that GitHub can assume. When using the [Gruntwork Account Factory](/2.0/docs/accountfactory/architecture), this update is performed automatically during the process of [adding a new AWS account](/2.0/docs/accountfactory/guides/vend-aws-account). ### How Pipelines knows what AWS account to authenticate to -Pipelines assumes that each top level directory in your `infrastructure-live` repository maps to a single AWS account, excluding the directory used for [module defaults](/2.0/docs/library/concepts/module-defaults). Each account-mapped directory must have an entry in the `/accounts.yml` file with a key matching the directory name and containing key/value pairs for the AWS account ID and the account's root user email address. +Pipelines assumes that each top-level directory in your `infrastructure-live` repository corresponds to a single AWS account, excluding the directory reserved for [module defaults](/2.0/docs/library/concepts/module-defaults). Each account-mapped directory must have an entry in the `accounts.yml` file. The entry should include a key matching the directory name and key/value pairs for the AWS account ID and the root user email address of the account. -For example, the following entry in `accounts.yml` would be mapped to a directory called `my-cool-account` in your `infrastructure-live` repository. +For instance, the following `accounts.yml` entry maps to a directory named `my-cool-account` in your `infrastructure-live` repository: ```yml title=accounts.yml "my-cool-account": @@ -39,35 +39,35 @@ For example, the following entry in `accounts.yml` would be mapped to a director :::note -This section contains some advanced topics pertaining to Pipelines, how it differentiates between types of changes, and how it handles planning and applying changes. +This section covers advanced topics related to Pipelines, how it differentiates between types of changes, and how it manages planning and applying changes. -From the perspective of the end user, you will still only need one step in your CI job. Pipelines knows how to do the rest. +For end users, only a single step is required in the CI job. Pipelines handles all additional processes automatically. ::: -Pipelines can handle two main types of infrastructure-change events: +Pipelines manages two main types of infrastructure-change events: -1. Adding/changing/deleting Terragrunt files -2. Creating new AWS accounts +1. Adding, changing, or deleting Terragrunt files. +2. Creating new AWS accounts. -For the first type of infrastructure change (add/change/delete Terragrunt files) Pipelines authenticates to the specific AWS account where the affected resources live. For the second type of change (creating new AWS accounts), Pipelines uses the Management Account. +For the first type (add/change/delete Terragrunt files), Pipelines authenticates directly to the AWS account containing the affected resources. For the second type (creating new AWS accounts), Pipelines uses the Management Account. -#### Management Account +#### Management account -Gruntwork's Account Factory is built on top of AWS Control Tower, and AWS Control Tower requires that you create new AWS accounts using the [Control Tower Management AWS Account](https://docs.aws.amazon.com/controltower/latest/userguide/how-control-tower-works.html#what-is-mgmt). +Gruntwork's Account Factory is built on AWS Control Tower, which requires that new AWS accounts be created through the [Control Tower Management AWS Account](https://docs.aws.amazon.com/controltower/latest/userguide/how-control-tower-works.html#what-is-mgmt). -When new AWS account requests are approved and the AWS account is created, Pipelines will create a Pull Request that contains an account baseline. Since Pipelines does not yet have access to the AWS account, the baseline will be applied by authenticating to the Management Account. +When a new AWS account request is approved and the account is created, Pipelines generates a Pull Request containing the account baseline. Since Pipelines does not initially have access to the new AWS account, it authenticates through the Management Account to apply the baseline. -When applying the baseline and provisioning the Pipelines role in the new child account, Pipelines will first assume the management account Pipelines role, then will assume an automatically provisioned role in the child account (this process is known as [role chaining](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html)). Once Pipelines has temporary credentials in the new account, it will run a plan or apply to provision the requested resources. After the initial baseline and Pipelines role has been applied in the child account, all subsequent events for that account will be handled by the Pipelines role directly in the child account. +During baseline application and provisioning of the Pipelines role in the new child account, Pipelines first assumes the management account's Pipelines role. It then assumes an automatically provisioned role in the child account (a process known as [role chaining](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html)). Once Pipelines has temporary credentials in the new account, it executes a plan or apply to provision the requested resources. After the initial baseline and Pipelines role are applied, all subsequent events for that account are managed directly through the Pipelines role in the child account. :::note -The AWS IAM Role for the management account must have permissions to provision new AWS Accounts (via AWS Control Tower) and assume a role in all child accounts. +The AWS IAM Role in the Management Account must have permissions to provision new AWS accounts (via AWS Control Tower) and assume roles in all child accounts. ::: -#### Child Accounts +#### Child accounts -Each child account (e.g., `dev`, `stage`, `prod,` etc.) contains an AWS IAM role that Pipelines can assume from GitHub Actions using OIDC. This role is automatically provisioned as a part of the [account baseline process](/2.0/docs/accountfactory/guides/vend-aws-account). Once this role is in place in the child account, users may submit Pull Requests to add/modify/delete resources in the child account. +Each child account (e.g., `dev`, `stage`, `prod`, etc.) contains an AWS IAM role that Pipelines can assume from GitHub Actions using OIDC. This role is automatically provisioned during the [account baseline process](/2.0/docs/accountfactory/guides/vend-aws-account). Once the role is established in the child account, users can submit Pull Requests to add, modify, or delete resources in that account. -When a Pull Request is created or synchronized, or a push to the `main` branch occurs, Pipelines will detect the changes, map them to the new account, assume the role in the child account, then run a `terragrunt plan` or `terragrunt apply` job (plan for PRs, apply for pushes to main). +When a Pull Request is created or synchronized, or when changes are pushed to the `main` branch, Pipelines detects the changes, maps them to the appropriate account, assumes the role in the child account, and executes a `terragrunt plan` (for Pull Requests) or `terragrunt apply` (for pushes to `main`). diff --git a/docs/2.0/docs/pipelines/concepts/drift-detection.md b/docs/2.0/docs/pipelines/concepts/drift-detection.md index 24a668535..e47e15d8f 100644 --- a/docs/2.0/docs/pipelines/concepts/drift-detection.md +++ b/docs/2.0/docs/pipelines/concepts/drift-detection.md @@ -1,15 +1,15 @@ # Pipelines Drift Detection :::note -Pipelines Drift Detection is only available to Devops Foundations Enterprise customers. +Pipelines Drift Detection is available exclusively to DevOps Foundations Enterprise customers. ::: ## What is Pipelines Drift Detection -Infrastructure Drift occurs when the applied terragrunt cloud configuration differs from the committed Infrastructure as Code (IaC). +Infrastructure drift occurs when the applied Terragrunt cloud configuration no longer matches the committed Infrastructure as Code (IaC). -Pipelines Drift Detection helps to mitigate Drift in your repositories by running `terragrunt plan` on infrastructure units. If the plan detects any units have drifted from their applied configuration Pipelines will open a Drift Detected Pull Request tracking this drift in your repository. +Pipelines Drift Detection helps address drift in your repositories by running `terragrunt plan` on infrastructure units. If drift is detected, Pipelines creates a Drift Detected Pull Request to track the changes in your repository. -When the Drift Detected Pull Request is merged, Pipelines will run `terragrunt apply` on all units where drift was detected to ensure resources once again match what is specified in code. +When the Drift Detected Pull Request is merged, Pipelines runs `terragrunt apply` on all units where drift was identified, ensuring that resources align with the specifications in your code. -See [Setting up Drift Detection](/2.0/docs/pipelines/configuration/driftdetection) and [Running Drift Detection](/2.0/docs/pipelines/guides/running-drift-detection) for more information on how to detect and remediate drift in your repositories. +For more information on detecting and resolving drift, see [Setting up Drift Detection](/2.0/docs/pipelines/configuration/driftdetection) and [Running Drift Detection](/2.0/docs/pipelines/guides/running-drift-detection). diff --git a/docs/2.0/docs/pipelines/concepts/hcl-config-language.md b/docs/2.0/docs/pipelines/concepts/hcl-config-language.md index 778d7d4c9..c7c290b2f 100644 --- a/docs/2.0/docs/pipelines/concepts/hcl-config-language.md +++ b/docs/2.0/docs/pipelines/concepts/hcl-config-language.md @@ -1,6 +1,7 @@ # Primer on HCL Terminology -HCL is an extensible configuration language developed by HashiCorp. See [this](https://github.com/hashicorp/hcl/blob/main/hclsyntax/spec.md) for the full specification. +HCL is an extensible configuration language created by HashiCorp. For the full specification, see [this document](https://github.com/hashicorp/hcl/blob/main/hclsyntax/spec.md). + Example: ```hcl @@ -11,32 +12,32 @@ environment "an_environment" { } ``` -The main terminology you need to know to understand the documentation below includes: +The key terminology to understand the documentation below includes: ### Blocks -A block is a collection of nested configurations that are defined within curly braces `{}`, immediately after the identifier of the block. +A block is a collection of nested configurations defined within curly braces `{}`, immediately following the block's identifier. + +For example, a `filter` block is nested within the `environment` block in the example above. -A `filter` block is nested in the `environment` block in the example above. +Blocks can be one of three types: `bare`, `labeled`, or `fully qualified`. -Blocks come in three variants: `bare`, `labeled` or `fully qualified`. -
    -
  • The filter block in the above example is `bare` - it has no additional qualification or labelling beyond the name `filter.
    • +- The `filter` block in the example is a `bare` block, as it lacks additional qualification or labeling beyond its name, `filter`. -
  • The environment block is a `labeled` block, and its label is `an_environment`.
    • +- The `environment` block is a `labeled` block, with `an_environment` as its label. -
  • Most terraform you've seen uses `fully qualified` blocks that have both a type and a label. For example, `resource "aws_instance" "my_instance" { ... }` - in this case the type is `aws_instance` and the label is `my_instance`.
    • +- Most OpenTofu/Terraform configurations use `fully qualified` blocks, which include both a type and a label. For instance, `resource "aws_instance" "my_instance" { ... }` has a type of `aws_instance` and a label of `my_instance`. -
### Attributes -An attribute is a key-value pair separated by an `=` that is defined within a block. +An attribute is a key-value pair separated by `=` and defined within a block. + +For example, the `paths` attribute is defined within the `filter` block in the example above. -The `paths` attribute is defined within the `filter` block in the example above. ### Labels -A label is one or more strings that are used to qualify a block. +A label is one or more strings used to qualify a block. -The `an_environment` label is used to qualify the `environment` block in the example above. \ No newline at end of file +For example, the `an_environment` label qualifies the `environment` block in the example above. diff --git a/docs/2.0/docs/pipelines/concepts/overview.md b/docs/2.0/docs/pipelines/concepts/overview.md index 92d88e827..d3784b254 100644 --- a/docs/2.0/docs/pipelines/concepts/overview.md +++ b/docs/2.0/docs/pipelines/concepts/overview.md @@ -1,15 +1,15 @@ # What is Gruntwork Pipelines? -**Gruntwork Pipelines is designed to enable your organization to deploy infrastructure changes to cloud environments simply, with control and confidence.** +**Gruntwork Pipelines enables your organization to deploy infrastructure changes to cloud environments with simplicity, control, and confidence.** -Having worked with hundreds of organizations to help them improve DevOps, we've discovered two truths about making changes to infrastructure: +After working with hundreds of organizations to improve DevOps processes, we’ve identified two key truths about managing infrastructure changes: 1. Teams want to control exactly how infrastructure change gets rolled out 2. Deploying infrastructure changes can be scary! -To address your need for _control_, we've designed Gruntwork Pipelines to use [configuration as code](/2.0/reference/pipelines/configurations-as-code/api), where you use HCL (a popular alternative to JSON and YAML) to set configuration values that apply to your entire git repo, to just one environment, or to a single deployable unit of infrastructure. For example, you can specify a unique AWS authentication strategy for each deployable unit of infrastructure, one per environment, or a single strategy for the entire git repo. +To address the need for **control**, Gruntwork Pipelines uses [configuration as code](/2.0/reference/pipelines/configurations-as-code/api). Using HCL (a widely adopted alternative to JSON and YAML), you can define configuration values for your entire Git repository, specific environments, or individual deployable infrastructure units. For instance, you can specify unique AWS authentication strategies at different levels, from individual units to the entire repository. -To address your need for _assurance_ that an infrastructure change is safe to apply, we include a thoughtfully formatted `terragrunt plan` user experience, and the ability to customize Gruntwork Pipelines to support arbitrary steps that your organization needs to establish confidence in a deployment. Building assurance also factors heavily into our roadmap. +To address the need for **assurance**, Gruntwork Pipelines provides a clear and user-friendly `terragrunt plan` experience. It also allows customization to include any additional steps your organization requires to build confidence in a deployment. Building assurance also factors heavily into our roadmap. ## Built for Terragrunt @@ -19,40 +19,35 @@ Gruntwork is the creator and maintainer of [Terragrunt](https://terragrunt.grunt - `terragrunt destroy` - `terragrunt run-all` -We also make updates directly to Terragrunt to support the functionality we want to see in Gruntwork Pipelines. +Gruntwork actively contributes to Terragrunt, ensuring it supports features critical to Gruntwork Pipelines. -## Reduces the complexity of CI/CD +## Simplifies CI/CD Complexity One of the things we've discovered over the years helping customers automate their infrastructure management is that it can be _very_ costly and time-consuming to build and maintain a CI/CD pipeline that can efficiently handle the complexity of infrastructure changes. Customers typically don't want to trigger an update to _all_ of their infrastructure whenever _any_ component changes, and they typically want to have related changes coordinated and rolled out correctly. A driving design goal of Gruntwork Pipelines is to allow for a minimal setup experience, followed by a very intuitive model for driving infrastructure updates. Most customers can get Pipelines configured in less than an hour, then drive all of their infrastructure changes directly via pull requests to Infrastructure as Code. Most of the time, you do not need to think about how Gruntwork Pipelines works, or how it makes decisions about what to do. You simply update your Infrastructure as Code to reflect the desired state of your infrastructure, have the pull request reviewed and merged, then Gruntwork Pipelines takes care of the rest. -## Runs directly in GitHub Actions +## Runs in GitHub Actions -Gruntwork Pipelines runs directly in your GitHub Actions workflows, and uses a pull request-centric workflow. This means that all information about a proposed infrastructure change is added as comments to the applicable pull request, and that you apply the infrastructure change by interacting with the pull request. +Gruntwork Pipelines integrates directly with GitHub Actions, using a pull request-centric workflow. All information about a proposed infrastructure change is added as comments to the relevant pull request, and infrastructure changes are applied by interacting with the pull request. -This also means that Gruntwork Pipelines does not depend on Gruntwork servers to perform any `terragrunt` operations. You are fully in control over the execution of your infrastructure automation, and can customize the behavior of Gruntwork Pipelines to suit your organization's needs. +This approach ensures Gruntwork Pipelines operates independently of Gruntwork servers. All `terragrunt` operations are executed within GitHub Actions, giving you full control over your infrastructure automation. Secrets and state files remain secure within GitHub Actions. -Gruntwork does not need access to your secrets or state files, as these remain in GitHub Actions. At the same time, we continually push new features and updates to Gruntwork Pipelines so that your user experience, security posture, and feature set continues to improve with no effort on your part. +We continually update Gruntwork Pipelines, delivering new features and security improvements without requiring any action on your part. -## Common Terms +## Common terms -Gruntwork Pipelines uses specific terminology to describe code changes and operations that occur as a result of changes. This section will familiarize you with the terminology used throughout the Gruntwork Pipelines documentation. +Gruntwork Pipelines uses specific terminology to describe infrastructure changes and associated operations. Below are key terms used in the documentation: ### Infrastructure change -When you edit any Infrastructure As Code (IAC) that needs to be "applied" to your cloud account (e.g. AWS or GCP), you are making an _infrastructure change_. We sometime call these an "infra-change" for short. - -Infra-changes can involve updates to OpenTofu/Terraform or Terragrunt code, or any other type of file that represents a desired state of your infrastructure and that needs to be somehow applied. A classic example is changing some variables on an instance of an OpenTofu module. Gruntwork Pipelines assumes that infra-changes are committed via git, usually by first opening a pull request. When you open a pull request, you are essentially "proposing" an infra-change. +An _infrastructure change_ occurs when you edit Infrastructure as Code (IaC) that needs to be applied to your cloud account (e.g., AWS or GCP). These changes are often referred to as "infra-changes." Examples include modifying variables in OpenTofu/Terraform or Terragrunt code. Gruntwork Pipelines assumes infra-changes are committed via git, typically through a pull request, which "proposes" the change. ### Infrastructure change set -Sometimes users create a pull request that changes more than one file at a time. And sometimes a change to a single file affects how multiple other files will be "applied" to your live cloud account. For example, many users of Terragrunt use a pattern known as "envcommon" as a way to specify a default set of values for modules. - -When more than one infra-change is made at a time, we call this an _infrastructure-change set_. +An _infrastructure change set_ refers to pull requests that involve multiple infra-changes. For example, changes to an "envcommon" file in Terragrunt may affect several modules. A change set represents all related infra-changes that need to be applied together. ### Pipelines actions -When a user proposes to make an infra-change by opening a pull request, we want to take some kind of "action" in response to that proposed-change. For example, we may want to run a `terragrunt plan` and estimate the cost of applying this Terragrunt plan. We call these _pipelines actions_. +_Pipelines actions_ describe the operations triggered by proposed infra-changes, such as running a `terragrunt plan` or estimating costs. Gruntwork Pipelines supports a growing library of actions and continues to expand its capabilities over time. -Gruntwork is responsible for adding support for a growing library of Pipelines Actions and we will continue to add more over time.