Intermitent errors creating aws-native:ec2:SubnetRouteTableAssociation

[mjeffryes]

What happened?

Pulumi-cdk has been experiencing flaky test runs (pulumi/pulumi-cdk#96) due to the following error:

 aws-native:ec2:SubnetRouteTableAssociation (VPCPublicSubnet1RouteTableAssociation0B0896DC):
      error: reading resource state: reading resource state: operation error CloudControl: GetResource, https response error StatusCode: 400, RequestID: fca75fd6-63d7-4569-8853-49f8558a8242, ResourceNotFoundException: AWS::EC2::SubnetRouteTableAssociation Handler returned status FAILED: No route tables Found with association rtbassoc-03189a84b8caa818f (HandlerErrorCode: NotFound, RequestToken: 81dddeca-f9da-44a5-9b93-9ff127dfb1c4)

This error is encountered here: https://github.com/pulumi/pulumi-aws-native/blob/master/provider/pkg/provider/provider.go#L803 when cloud control fails to find the resource it has just created.

We'll need to work this down to a repro case to send to the CC API maintainers; we can also look into a workaround in aws-native (eg. adding our own retries for this particular case).

Example

https://github.com/pulumi/pulumi-cdk/tree/main/examples/alb is our best repro case at the moment.

Output of pulumi about

N/A

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

I have been looking a this a little bit, it appears notoriously difficult to reproduce locally. I have boiled down the relevant resource-set from the CDK examples to something minimal as follows:

import * as aws from "@pulumi/aws-native";

const vpc = new aws.ec2.Vpc("my-vpc", {
    cidrBlock: "10.0.0.0/16",
    enableDnsHostnames: true,
    enableDnsSupport: true,
    instanceTenancy: "default",
    tags: [
        {
            key: "Name",
            value: "fargatestack/MyVpc"
        }
    ]
});


const mySubnet = new aws.ec2.Subnet("my-subnet", {
    availabilityZone: "us-west-2a",
    cidrBlock: "10.0.0.0/18",
    mapPublicIpOnLaunch: true,
    tags: [
        {
            key: "aws-cdk:subnet-name",
            value: "Public"
        },
        {
            key: "aws-cdk:subnet-type",
            value: "Public"
        },
        {
            key: "Name",
            value: "fargatestack/MyVpc/PublicSubnet1"
        }
    ],
    vpcId: vpc.id,
});

const myRT = new aws.ec2.RouteTable("my-rt", {
    vpcId: vpc.id,
    tags: [
        {
            key: "Name",
            value: "fargatestack/MyVpc/MyRouteTable"
        }
    ]
})

const myRTA = new aws.ec2.SubnetRouteTableAssociation("my-rta", {
    routeTableId: myRT.id,
    subnetId: mySubnet.id,
})

export const routeTableID = myRTA.id;

Unfortunately standing this up and down does not quite reproduce the issue.

[t0yv0]

Linking some more context in.

Control reaches this code:

pulumi-aws-native/provider/pkg/client/client.go

Line 92 in c2f11ba

return nil, nil, fmt.Errorf("reading resource state: %w", err)

We have been staring at this with @flostadler . The code seems to be written well. What is happening is that the awaiter c.awaiter.WaitForResourceOpCompletion succeeded. Therefore waitErr == nil. We hit this code:

		// Creation succeeded but reading failed - return the read error.
		return nil, nil, fmt.Errorf("reading resource state: %w", err)

The message is

No route tables Found with association rtbassoc-03189a84b8caa818f.

[t0yv0]

Something that feels suspect is that the CDK workflow is scheduled with

  schedule:
    - cron: '0 7 * * *'

https://github.com/pulumi/pulumi-cdk/blob/main/.github/workflows/main.yml#L89

It seems not entirely impossible that this would race with the aws-account-cleanup lambda scheduled to run every 12hrs:

https://github.com/pulumi/aws-account-cleanup/blob/master/pkg/cleanvpc/cleanvpc.go#L208

Specifically there is no code to cleanup RTAs, but there is code to cleanup RTs.

[t0yv0]

We could try tagging resources with "ExpiresBy": "2025-01-01" and teach the cleaner to respect this tag and avoid deleting resources until the expiry date is reached. In this case, we could ensure the VPC is tagged, which will bypass premature cleanup by the lambda.

[t0yv0]

Based on standup: Anton to search in CloudTrail by resource name and try to see what that gives it.

[t0yv0]

We looked at CloudControl events, and it seems consistent with an eventual consistency issue.

Adding aws-cloudformation/cloudformation-coverage-roadmap#2178

[t0yv0]

Expecting this to be fixed by #1809 introducing retry but not being able to repro outside of pulumi-cdk CI.

I will close for now and we reopen as needed.

[t0yv0]

Looks like what finally solved it is moving tests to another region. @corymhall suspected interference with pulumi-eks tests.