Skip to content

Commit

Permalink
feat: enable additional metadata collection (under feature flag) (#33232
Browse files Browse the repository at this point in the history
)

### Issue # (if applicable)

Closes #33260

Relevant discussion #33198

### Note

The majority of the code changes are auto generated so you'll see
hundreds of `addConstructMetadata` method call across different L2
resources.

This method comes from this change
https://github.com/aws/aws-cdk/pull/33232/files#diff-81f821b1205e7040fc3103bf7c0114060a6d5c43ebd2994aa4ed5906e42c9c5fR33.
The main code change that needs to be reviewed is in
`packages/aws-cdk-lib/core` as well as
`tools/@aws-cdk/construct-metadata-updater`

### Reason for this change

This discussion aims to expand the scope of usage data collected by the
AWS CDK to better inform CDK development and improve communication for
customer-impacting topics. Currently, for those that opt in, the CDK
collects usage data on your CDK version and which L2 constructs you use.

### Description of changes

1. Update CDK synthesis code to additionally handle resource metadata.
On feature flag set to true, synthesis will not only inject Metadata
usage like version and construct name, it will additionally look for any
construct/method/feature flag metadata injected during resource
creation. On feature flag set to false, it should be the same as before.
2. One-time tool metadata-updater to automatically find the right
classes and add import statements and add metadata statements. The tool
can be run multiple times and should not add additional import or add
metadata statements to files that already been added. An action item is
to link the tool to a GHA to periodically run this.
3. Build a workflow (that will be linked to GHA in the future) and when
redacting, redact based on the value. The workflow on run will parse all
files in aws-cdk repository and built a JSON file that contains all
constructs and loggable properties of the construct. When redacting,
only log the property key if the key exists in the JSON file. The value
will be logged only if the value is not a * in the JSON file. Everything
else is redacted for safely.
4. Build a JSON blueprint of the ENUM values and do not redact ENUM
values.

Consider the following example

```
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from 'aws-cdk-lib/aws-s3';

class MyStack extends cdk.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create an S3 bucket (L2 construct)
    const myBucket = new s3.Bucket(this, 'MyBucket', {
      bucketName: 'my-cdk-example-bucket', // String type
      versioned: true,                    // Boolean type
      removalPolicy: cdk.RemovalPolicy.DESTROY, // ENUM type
      lifecycleRules: [{                  // Array of object type
        expirationDate: new Date('2019-10-01'),
        objectSizeLessThan: 600,
        objectSizeGreaterThan: 500,
      }],
    });

    // Use a method of the L2 construct to define additional properties
    myBucket.addLifecycleRule({
      id: 'ExpireOldObjects',
      enabled: true, // Boolean
      expiration: cdk.Duration.days(90), // Expire objects after 90 days
    });
  }
}

// Define the CDK app and stack
const app = new cdk.App();
new MyStack(app, 'MyStack');
app.synth();
```

At synthesis, usage data is collected, compressed, and stored in the
`AWS::CDK::Metadata resource`. Based on **current** behavior, the
following is an example of the usage data that will be collected from
our example app:
```
{ 
    "fqn": "aws-cdk-lib.aws-s3.Bucket", 
    "version": "v2.170.0" 
}
```

With this proposal, the following usage data will be collected. The *
value replaces property values that will be redacted from data
collection:
```
{ 
    "fqn": "aws-cdk-lib.aws_s3.Bucket", 
    "version": "2.170.0", 
    "metadata": [ 
        { 
            "type": "aws:cdk:analytics:construct", 
            "data": { 
                "bucketName": "*", 
                "versioned": true, 
                "removalPolicy": "cdk.RemovalPolicy.DESTROY", 
                "lifecycleRules": [ 
                    { 
                        "expirationDate": "*", 
                        "objectSizeLessThan": "*", 
                        "objectSizeGreaterThan": "*" 
                    } 
                ] 
            } 
        }, 
        { 
            "type": "aws:cdk:analytics:method", 
            "data": { 
                "name": "addLifecycleRule", 
                "prop": { 
                    "id": "*", 
                    "enabled": true, 
                    "expiration": "*",
                } 
            } 
        } 
    ] 
}
```

### Describe any new or updated permissions being added

No

### Description of how you validated changes

Many new unit tests added to verify different behaviour of various
functions and methods introduced. One integ test file is added to test
the deployability.

### Checklist
- [x] My code adheres to the [CONTRIBUTING
GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and
[DESIGN
GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license*

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
  • Loading branch information
GavinZZ and mergify[bot] authored Feb 1, 2025
1 parent 0bb3d6f commit 6b9e47a
Show file tree
Hide file tree
Showing 382 changed files with 32,110 additions and 19 deletions.
1 change: 1 addition & 0 deletions lerna.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"tools/@aws-cdk/yarn-cling",
"tools/@aws-cdk/lazify",
"tools/@aws-cdk/lambda-integration-test-updater",
"tools/@aws-cdk/construct-metadata-updater",
"scripts/@aws-cdk/script-tests"
],
"rejectCycles": true,
Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
"tools/@aws-cdk/yarn-cling",
"tools/@aws-cdk/lazify",
"tools/@aws-cdk/lambda-integration-test-updater",
"tools/@aws-cdk/construct-metadata-updater",
"scripts/@aws-cdk/script-tests"
],
"nohoist": [
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 6b9e47a

Please sign in to comment.