[Bug]: Error: configmaps "aws-auth" is forbidden: User "system:anonymous" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

[stafot]

Terraform Core Version

<=1.8.1

AWS Provider Version

>=5.58.0

Affected Resource(s)

• kubernetes_config_map
• kubernetes_cluster_role_binding
• kubernetes_cluster_role

Expected Behavior

Successful plan

Actual Behavior

Plan fails.

Relevant Error/Panic Output Snippet

Error: configmaps "aws-auth" is forbidden: User "system:anonymous" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

  with module.cluster.kubernetes_config_map.aws_auth_configmap,
  on .terraform/modules/cluster/aws/cluster/aws_auth_configmap.tf line 73, in resource "kubernetes_config_map" "aws_auth_configmap":
  73: resource "kubernetes_config_map" "aws_auth_configmap" {


Error: clusterrolebindings.rbac.authorization.k8s.io "eks-console-dashboard-full-access-binding" is forbidden: User "system:anonymous" cannot get resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope

  with module.cluster.kubernetes_cluster_role_binding.console_access,
  on .terraform/modules/cluster/aws/cluster/aws_auth_configmap.tf line 109, in resource "kubernetes_cluster_role_binding" "console_access":
 109: resource "kubernetes_cluster_role_binding" "console_access" {


Error: clusterroles.rbac.authorization.k8s.io "eks-console-dashboard-full-access-clusterrole" is forbidden: User "system:anonymous" cannot get resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope

  with module.cluster.kubernetes_cluster_role.console_access,
  on .terraform/modules/cluster/aws/cluster/aws_auth_configmap.tf line 127, in resource "kubernetes_cluster_role" "console_access":
 127: resource "kubernetes_cluster_role" "console_access" {

If we use though aws provider <= 5.57.0 which fixes the above behavior we get another error

Warning: Failed to decode resource from state

Error decoding "module.cluster.aws_eks_cluster.cluster" from prior state:
unsupported attribute "bootstrap_self_managed_addons"

Error: Resource instance managed by newer provider version

The current state of module.cluster.aws_eks_cluster.cluster was created by a
newer provider version than is currently selected. Upgrade the aws provider
to work with this state.

Thus we are in a deadend

Terraform Configuration Files

https://github.com/mattermost/mattermost-cloud-monitoring/tree/master/aws/cluster

We are using the above module being set in our internal infrastructure.
Cannot share the whole internal infrastructure structure
but
when we set

    aws = {
      source  = "hashicorp/aws"
      version = ">=5.41.0"
    }

fails because it uses provider 5.60.0(latest)

but when we set

    aws = {
      source  = "hashicorp/aws"
      version = "=5.57.0"
    }

works

Steps to Reproduce

Use the above files we shared and run a plan

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

Additional information:

We cannot use terraform >= 1.8.1 because there is an open bug in atlantis, although we can test it if you want to see if a recent terraform version fixes it. ref. runatlantis/atlantis#4471

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[james0209]

@stafot FWIW, the "We cannot use terraform >= 1.8.1 because there is an open bug in atlantis" is fixed in the latest versions of Atlantis. Reference here.

[stafot]

Thanks @james0209 I will try to upgrade to check if this resolves permissions issue and if not I will update the description.

[justinretzolk]

Hey @stafot 👋 Thank you for taking the time to raise this! In reviewing the report, it looks like the initial errors your receiving are coming from the Kubernetes provider, which is something you'd need to report in that provider's repository.

If we use though aws provider <= 5.57.0 which fixes the above behavior we get another error

The error you received here was caused by the attempt at using a provider version that's older than the one that the resources are currently managed under, which isn't supported.

Since this appears to be expected behavior as far as the AWS Provider is concerned, and there's no further action for the provider team to take, I'm going to close this issue. If you encounter unexpected behavior with the provider in the future, please do let us know!

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[stafot]

@justinretzolk
Just 2 clarifications because I might misinterpreted.

1. we are sure that the trigger is the aws provider change since kubernetes provider is unchanged.
2. The more recent versions (5.60.0) do not work while the older (5.57.0) work, so could you elaborate how this is expected?

I am going to create a bug issue on the other provider but I need to understand why do you think that is expected a version upgrade in this provider is expected to cause another provider to break.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.