Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workflow failure: main #96

Closed
pulumi-bot opened this issue Nov 15, 2023 · 2 comments · Fixed by #97
Closed

Workflow failure: main #96

pulumi-bot opened this issue Nov 15, 2023 · 2 comments · Fixed by #97
Assignees
@mjeffryes
Labels
kind/engineering Work that is not visible to an external user p1 A bug severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed

Comments

@pulumi-bot
Copy link
Contributor

pulumi-bot commented Nov 15, 2023 •
edited
Loading

Workflow Failure

main has failed. See the list of failures below:
@pulumi-bot pulumi-bot added kind/engineering Work that is not visible to an external user needs-triage Needs attention from the triage team p1 A bug severe enough to be the next item assigned to an engineer labels Nov 15, 2023
@lukehoban
Copy link
Contributor

The error in all cases above was:

 aws-native:ec2:SubnetRouteTableAssociation (VPCPublicSubnet1RouteTableAssociation0B0896DC):
      error: reading resource state: reading resource state: operation error CloudControl: GetResource, https response error StatusCode: 400, RequestID: fca75fd6-63d7-4569-8853-49f8558a8242, ResourceNotFoundException: AWS::EC2::SubnetRouteTableAssociation Handler returned status FAILED: No route tables Found with association rtbassoc-03189a84b8caa818f (HandlerErrorCode: NotFound, RequestToken: 81dddeca-f9da-44a5-9b93-9ff127dfb1c4)

This mostly happened in the ALB test, but also once in the Fargate test.

@mikhailshilkov mikhailshilkov removed the needs-triage Needs attention from the triage team label Nov 18, 2023
@mjeffryes
Copy link
Member

AFAICT, this is an intermittent bug with the CC API where an AWS::EC2::SubnetRouteTableAssociation isn't always found immediately after it's created.

I don't think "Not Found" errors are retried by default in the cloud control client used by aws-native (and doing so would create artificial slowdowns when there are legit not founds).

We can file a bug against Cloud Control, but in the meantime, we will need to either disable these tests or retry them to avoid the flaky results

@mjeffryes mjeffryes mentioned this issue Nov 23, 2023
Closed
mjeffryes added a commit that referenced this issue Nov 27, 2023
@mjeffryes
Retry flakey integration tests
5c0589e 
#96
@mjeffryes mjeffryes mentioned this issue Nov 27, 2023
Merged
mjeffryes added a commit that referenced this issue Nov 28, 2023
@mjeffryes
Retry flakey integration tests (#97)
bce6cd1 
As per, pulumi/pulumi-aws-native#1186 we're
seeing intermittent errors in the creation of
`aws-native:ec2:SubnetRouteTableAssociation` This change enables retries
for the affected tests to reduce the noise until we have an upstream fix

Fixes #96
@pulumi-bot pulumi-bot added the resolution/fixed This issue was fixed label Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mjeffryes mjeffryes

Labels
kind/engineering Work that is not visible to an external user p1 A bug severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Retry flakey integration tests pulumi/pulumi-cdk
4 participants
@lukehoban @mjeffryes @mikhailshilkov @pulumi-bot