Api Gateway Stage Tags cannot be added after Creation

[Bhutania]

What happened?

Updating an API Gateway stage that does not already have tags results in an error. Adding tags during creation or outside of IaC seems to clear up this issue.

Example

Minimal Reproduction:

		restApi, err := apigatewayNative.NewRestApi(ctx, "api", &apigatewayNative.RestApiArgs{
			Name: pulumi.String("arnav-test"),
		},
			pulumi.Provider(awsNativeProvider),
		)

		method, err := apigatewayNative.NewMethod(ctx, "api-method", &apigatewayNative.MethodArgs{
			HttpMethod:        pulumi.String("GET"),
			ResourceId:        restApi.RootResourceId,
			RestApiId:         restApi.ID(),
			AuthorizationType: pulumi.String("NONE"),
			Integration: &apigatewayNative.MethodIntegrationArgs{
				Uri:                   pulumi.String("https://www.google.com"),
				Type:                  apigatewayNative.MethodIntegrationTypeHttp,
				IntegrationHttpMethod: pulumi.String("GET"),
			},
		},
			pulumi.Provider(awsNativeProvider),
		)
		if err != nil {
			return err
		}

		deployment, err := apigatewayNative.NewDeployment(ctx, "api-deployment", &apigatewayNative.DeploymentArgs{
			RestApiId: restApi.ID(),
		},
			pulumi.Provider(awsNativeProvider),
			pulumi.DependsOn([]pulumi.Resource{method}),
		)
		if err != nil {
			return err
		}

		// uncomment this after create
		tags := awsNative.TagArray{
			// &awsNative.TagArgs{
			// 	Key:   pulumi.String("foo"),
			// 	Value: pulumi.String("bar"),
			// },
		}

		_, err = apigatewayNative.NewStage(ctx, "api-stage", &apigatewayNative.StageArgs{
			RestApiId:    restApi.ID(),
			StageName:    pulumi.String("prod"),
			DeploymentId: deployment.DeploymentId,
			Tags:         tags,
		},
			pulumi.Provider(awsNativeProvider),
		)
		if err != nil {
			return err
		}

Output:

stderr: error: operation error CloudControl: UpdateResource, https response error StatusCode: 400, RequestID: ee812dc4-0070-4c7c-9900-ecda5cde19af, api error ValidationException: [REPLACE Operation] noSuchPath in source, path provided : //Tags
error: update failed
exit status 1

Output of pulumi about

pulumi about
CLI          
Version      3.145.0
Go Version   go1.23.4
Go Compiler  gc

Host     
OS       darwin
Version  15.2
Arch     arm64

Backend        
Name           pulumi.com
URL            https://app.pulumi.com
User           Unknown
Organizations  
Token type     personal

I'm using autostack so the provider versions are not apparent but this what is in my go mod:
github.com/pulumi/pulumi-aws-native/sdk v1.22.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Thanks Bhutania for filing this issue. I cannot reproduce with these versions.

Can you provide a complete repro? Perhaps there is something special about the awsNativeProvider setup?

CLI          
Version      3.141.0
Go Version   go1.23.3
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  aws-native  1.22.0
language  go          unknown

Host     
OS       darwin
Version  14.6.1
Arch     arm64

This project is written in go: executable='/run/current-system/sw/bin/go' version='go version go1.23.3 darwin/arm64'

Current Stack: anton-pulumi-corp/repro-aws-native-1972/repro-pulumi-aws-native-1972

TYPE                              URN
pulumi:pulumi:Stack               urn:pulumi:repro-pulumi-aws-native-1972::repro-aws-native-1972::pulumi:pulumi:Stack::repro-aws-native-1972-repro-pulumi-aws-native-1972
pulumi:providers:aws-native       urn:pulumi:repro-pulumi-aws-native-1972::repro-aws-native-1972::pulumi:providers:aws-native::default_1_22_0
aws-native:apigateway:RestApi     urn:pulumi:repro-pulumi-aws-native-1972::repro-aws-native-1972::aws-native:apigateway:RestApi::api
aws-native:apigateway:Method      urn:pulumi:repro-pulumi-aws-native-1972::repro-aws-native-1972::aws-native:apigateway:Method::api-method
aws-native:apigateway:Deployment  urn:pulumi:repro-pulumi-aws-native-1972::repro-aws-native-1972::aws-native:apigateway:Deployment::api-deployment
aws-native:apigateway:Stage       urn:pulumi:repro-pulumi-aws-native-1972::repro-aws-native-1972::aws-native:apigateway:Stage::api-stage


Found no pending operations associated with repro-pulumi-aws-native-1972

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/anton-pulumi-corp
User           anton-pulumi-corp
Organizations  anton-pulumi-corp, moolumi, demo, pulumi
Token type     personal

Dependencies:
NAME                                     VERSION
github.com/pulumi/pulumi-aws-native/sdk  v1.22.0
github.com/pulumi/pulumi/sdk/v3          v3.144.1

Pulumi locates its logs in /var/folders/gd/3ncjb1lj5ljgk8xl5ssn_gvc0000gn/T/com.apple.shortcuts.mac-helper// by default

[Bhutania]

Hmm, just to be clear, the error occurs on update, not on create. First the APIG stage needs to be created (with no tags), and then updated to add new tags.

Here is an example diff that I would expect in a preview on the update operation:

    ~ aws-native:apigateway:Stage: (update)
        [id=acwgno1f5k|prod]
        [urn=urn:pulumi:arnav-test::arnav-program-backend::aws-native:apigateway:Stage::api-stage]
        [provider=urn:pulumi:arnav-test::arnav-program-backend::pulumi:providers:aws-native::program-account-aws-native-provider::54a7fd5e-d63b-4c6e-9773-62e237cc138f]
      ~ tags: [
          + [0]: {
                  + key  : "foo"
                  + value: "bar"
                }
        ]

Aws Native Provider is fairly simple:

awsNative.NewProvider(ctx, "aws-native-provider", &awsNative.ProviderArgs{
		Region:     pulumi.String("us-west-2"),
		Profile:    pulumi.String("foo"),
		MaxRetries: pulumi.Int(10),
},

I'll work on getting you a full repro.

[flostadler]

@Bhutania I was able to repro it with your example.

This is the full program I've used (uncommenting the tags after the first apply):

package main

import (
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
	awsNative "github.com/pulumi/pulumi-aws-native/sdk/go/aws"
	apigatewayNative "github.com/pulumi/pulumi-aws-native/sdk/go/aws/apigateway"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {

		awsNativeProvider, err := awsNative.NewProvider(ctx, "aws-native-provider", &awsNative.ProviderArgs{
			Region:     pulumi.String("us-west-2"),
			Profile:    pulumi.String("pulumi-dev-sandbox"),
			MaxRetries: pulumi.Int(10),
		})
		if err != nil {
			return err
		}

		restApi, err := apigatewayNative.NewRestApi(ctx, "api", &apigatewayNative.RestApiArgs{
			Name: pulumi.String("arnav-test"),
		},
			pulumi.Provider(awsNativeProvider),
		)
		if err != nil {
			return err
		}

		method, err := apigatewayNative.NewMethod(ctx, "api-method", &apigatewayNative.MethodArgs{
			HttpMethod:        pulumi.String("GET"),
			ResourceId:        restApi.RootResourceId,
			RestApiId:         restApi.ID(),
			AuthorizationType: pulumi.String("NONE"),
			Integration: &apigatewayNative.MethodIntegrationArgs{
				Uri:                   pulumi.String("https://www.google.com"),
				Type:                  apigatewayNative.MethodIntegrationTypeHttp,
				IntegrationHttpMethod: pulumi.String("GET"),
			},
		},
			pulumi.Provider(awsNativeProvider),
		)
		if err != nil {
			return err
		}

		deployment, err := apigatewayNative.NewDeployment(ctx, "api-deployment", &apigatewayNative.DeploymentArgs{
			RestApiId: restApi.ID(),
		},
			pulumi.Provider(awsNativeProvider),
			pulumi.DependsOn([]pulumi.Resource{method}),
		)
		if err != nil {
			return err
		}

		// uncomment this after create
		tags := awsNative.TagArray{
			// &awsNative.TagArgs{
			//	Key:   pulumi.String("foo"),
			//	Value: pulumi.String("bar"),
			// },
		}

		_, err = apigatewayNative.NewStage(ctx, "api-stage", &apigatewayNative.StageArgs{
			RestApiId:    restApi.ID(),
			StageName:    pulumi.String("prod"),
			DeploymentId: deployment.DeploymentId,
			Tags:         tags,
		},
			pulumi.Provider(awsNativeProvider),
		)
		if err != nil {
			return err
		}
		return nil
	})
}

I'll investigate what's causing this.

[flostadler]

At a first glance, what the provider seems to be doing is correct. It's sending a replace operation for the Path /Tags. It seems like AWS Cloud Control itself is adding the second slash in the property path.

[Bhutania]

Not sure if this info helps, but instead using the provider's DefaultTags does not result in an error when updating the stage. I ended up applying some default tags to the provider resource and then applying the stage specific tags afterwards as a separate operation.

My (poor) diagram:
create (no tags) -> up (provider defaultTags) -> up (stage tags)

[flostadler]

I was able to repro it just with the AWS CLI:

aws cloudcontrol update-resource --type-name AWS::ApiGateway::Stage \
--identifier 'vzcrrrjhj8|prod' --patch-document '[{"op":"replace","path":"/Tags","value":[{"Key":"foo","Value":"bar"}]}]'

So this seems to be a bug on the AWS Cloud Control side. But it's interesting that it works with the default tags. It could be an issue with updating from no tags (i.e. empty tag list) to tags. Let me verify that theory

[flostadler]

Indeed, after adding tags to the stage manually it works.

The issue here seems to be how AWS CloudControl handles an empty Tags array. Going from an empty Tags array to some tags is a replace operation, but this trips them up.

As a workaround you could completely omit the Tags property of the Stage if the array is empty. In that case they get correctly added in the future.

AWS Cloud Control issue: aws-cloudformation/cloudformation-coverage-roadmap#2236