Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend metadata with descriptors of CF Ref handling #1836

Merged
merged 57 commits into from
Nov 23, 2024
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
ff27a56
WIP on categorizing Ref docu
t0yv0 Nov 19, 2024
7f53cbd
Colocate property names
t0yv0 Nov 19, 2024
9443d13
Adding a guessing game
t0yv0 Nov 19, 2024
5f2766c
Guessing a few more resources
t0yv0 Nov 19, 2024
cc199e1
Add a Makefile target
t0yv0 Nov 19, 2024
469a10c
More information
t0yv0 Nov 19, 2024
caaa962
Fix typo
t0yv0 Nov 19, 2024
e32087a
Sort prop options by primary key index
t0yv0 Nov 19, 2024
4d6b326
Label some more resources
t0yv0 Nov 19, 2024
39fd303
Print stats; more labels
t0yv0 Nov 19, 2024
d38dd01
Allow marking Not Supported
t0yv0 Nov 19, 2024
d9de7d2
More labels
t0yv0 Nov 19, 2024
d8640c8
More labels
t0yv0 Nov 19, 2024
2c83641
N=62 labels
t0yv0 Nov 19, 2024
b0fa8fa
N=81 labels
t0yv0 Nov 19, 2024
d25399e
N=91 labels
t0yv0 Nov 20, 2024
8a7d4ae
Note JSON-encoded Properties
t0yv0 Nov 20, 2024
a09a34c
N=121 labels
t0yv0 Nov 20, 2024
9f5d997
Improve RefReturnsArn recognizer
t0yv0 Nov 20, 2024
8018e8a
Implement automatic heuristic labelling
t0yv0 Nov 20, 2024
9d7fdff
Add a target for automatic heuristic labelling
t0yv0 Nov 20, 2024
48f53db
Auto label some resources
t0yv0 Nov 20, 2024
ac58717
Recognize RefReturnsName categories
t0yv0 Nov 20, 2024
98039a3
Name based heuristic
t0yv0 Nov 20, 2024
fe3c226
Apply more heuristics
t0yv0 Nov 20, 2024
add63d4
Improve the Name heuristic
t0yv0 Nov 20, 2024
a64434a
Apply the name heuristic
t0yv0 Nov 20, 2024
bf70e2b
Further improve the name heuristic
t0yv0 Nov 20, 2024
c99525d
Apply the improved Name heuristic; N=266
t0yv0 Nov 20, 2024
04fea9c
Improve Arn category recognizer
t0yv0 Nov 20, 2024
574b095
Apply heuristics
t0yv0 Nov 20, 2024
eadeb3f
More ARN recognizers
t0yv0 Nov 20, 2024
a75e510
Add an ID heuristic
t0yv0 Nov 20, 2024
91f1501
Better ARN heuristic
t0yv0 Nov 20, 2024
9d181fd
Improved name heuristic
t0yv0 Nov 20, 2024
550486a
Better ARN heuristic
t0yv0 Nov 20, 2024
c0f9cff
Better ARN heuristic
t0yv0 Nov 20, 2024
0794355
Correct a bug
t0yv0 Nov 20, 2024
fc38dfd
Better ID heuristic
t0yv0 Nov 20, 2024
cdc53c4
Better ID heuristic
t0yv0 Nov 20, 2024
30ca419
Improved ID heuristic
t0yv0 Nov 20, 2024
f6bdc04
Better ARN heuristic
t0yv0 Nov 20, 2024
e466077
More key resources
t0yv0 Nov 20, 2024
e31e437
Toward including top-100 used resources
t0yv0 Nov 20, 2024
fb7e897
Add a README folder for the data
t0yv0 Nov 20, 2024
aae3db6
Add a header
t0yv0 Nov 20, 2024
26a19d6
Move CfRefBehavior into the medatada
t0yv0 Nov 20, 2024
09660cb
Share refdb code
t0yv0 Nov 20, 2024
c77ffb4
Weave CF Ref metadata into the provider metadata
t0yv0 Nov 20, 2024
f91f587
Fix failing test
t0yv0 Nov 20, 2024
e908ce1
Basic data quality reporting
t0yv0 Nov 21, 2024
3e6d1e4
Manually cover AWS::EC2::SecurityGroup
t0yv0 Nov 21, 2024
67433c3
Run CF to confirm AWS::IAM::RolePolicy behavior and schematize it
t0yv0 Nov 21, 2024
29053da
Confirmed experimentally that Secret.Id is also the ARN
t0yv0 Nov 21, 2024
10fe2d1
More data cleaning
t0yv0 Nov 22, 2024
ff9c0ab
Regenerate metadata
t0yv0 Nov 22, 2024
eb0829b
PR feedback
t0yv0 Nov 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -185,4 +185,19 @@ install_sdks:: install_dotnet_sdk install_python_sdk install_nodejs_sdk
# Required for the codegen action that runs in pulumi/pulumi
only_build:: build

ref-db::
mkdir -p bin
(cd provider/tools/ref-parser && go build -o ../../../bin/ref-parser)
./bin/ref-parser -guide ./aws-cloudformation-user-guide -schema ./aws-cloudformation-schema -db ./meta/ref-db.json

ref-db-auto::
mkdir -p bin
(cd provider/tools/ref-parser && go build -o ../../../bin/ref-parser)
./bin/ref-parser -guide ./aws-cloudformation-user-guide -schema ./aws-cloudformation-schema -db ./meta/ref-db.json -auto

ref-db-report::
mkdir -p bin
(cd provider/tools/ref-parser && go build -o ../../../bin/ref-parser)
./bin/ref-parser -guide ./aws-cloudformation-user-guide -schema ./aws-cloudformation-schema -db ./meta/ref-db.json -report

.PHONY: ensure generate_schema generate build_provider build
21 changes: 21 additions & 0 deletions meta/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# meta

Reference data used by the `pulumi-aws-native` provider.

## ref-db.json

Metadata describing the behavior of the [CloudFormation Ref Intrinsic](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html).

What Ref returns varies from resource to resource. The only available information on this is free-form text in CloudFormation documentation and anything that can be extracted experimentally. To facilitate building tooling such as `pulumi/pulumi-cdk` that emulates CF behavior, `ref-db.json` collects schematized information describing the behavior of `Ref`.

`ref-db.json` can be edited directly to correct errors or add entries for resources not yet covered.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add an example here on what it looks like to update this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You send a PR and edit it? I'm not sure I understand. It just has to conform to the schema.


## regions.json

AWS regions and their names.

Please do not edit this file directly.

This list can be refreshed by extracting from `botocore`. See `.botocore.version`.

make docs
Loading
Loading