DWX-19990: Add Log Bucket and its Access for Managed ARN Restricted Policies #21
Conversation
|
@roohisyeda this issue affects R41C so could you please review and merge this as soon as possible? |
| ], | ||
| "Resource": [ | ||
| "arn:aws:s3:::${DATALAKE_BUCKET}/cf-templates/*", | ||
| "arn:aws:s3:::${DATALAKE_BUCKET}/backup/*" | ||
| "arn:aws:s3:::${DATALAKE_BUCKET}/backup/*", | ||
| "arn:aws:s3:::${LOG_BUCKET}/cdw/*" |
There was a problem hiding this comment.
backup bucket can be defined as well, see backupStorageLocationBase in https://cloudera.github.io/cdp-dev-docs/cli-docs/environments/create-aws-environment.html
When you create an environment all 3 properties can be configured: data bucket and location, log bucket and loction and backup bucket and location; these 3 can point to different or the same bucktes
There was a problem hiding this comment.
yes, good catch. Added it here in the latest commit
|
Aren't we supposed to fix here https://github.com/cloudera/cdw-cloud-policies/blob/main/aws-iam-policies/managedArn-node-inline-policy.json#L266 for this issue or may be both? |
| ], | ||
| "Resource": [ | ||
| "arn:aws:s3:::${DATALAKE_BUCKET}/cf-templates/*", | ||
| "arn:aws:s3:::${DATALAKE_BUCKET}/backup/*" | ||
| "arn:aws:s3:::${DATALAKE_BUCKET}/backup/*", | ||
| "arn:aws:s3:::${LOG_BUCKET}/cdw/*", |
There was a problem hiding this comment.
oh sorry, we need access to LOG_LOCATION, not really the bucket and we should not restrict it under /cdw. The location is bucket + custom_path like: s3://my-bucket/my/custom/path/ This is exactly what the user can configure when they create a data lake and an environment.
Same for the backup bucket
| @@ -121,7 +121,9 @@ | |||
| ], | |||
| "Resource": [ | |||
| "arn:aws:s3:::${DATALAKE_BUCKET}/cf-templates/*", | |||
| "arn:aws:s3:::${DATALAKE_BUCKET}/backup/*" | |||
| "arn:aws:s3:::${DATALAKE_BUCKET}/backup/*", | |||
| "arn:aws:s3:::${LOG_BUCKET}/cdw/*", | |||
Based on QE test changes, we got S3 Access denied when a different log bucket was used. This is because we only specify permissions to work on the DL bucket. To mitigate this issue, we can add another bucket for Logs in the Sid: S3PutGetObject.
Pull Request checklist:
Description:
Changes to any files in generated folder(not to be touched)
Is the DWX JIRA PR using a new AWS SDK API which needs new permission(s), not included in either of the files under
https://github.com/cloudera/cdw-cloud-policies/tree/main/aws-iam-policies/docs then create a PR to include the new action
Is the dwx-cf-template.yaml being modified, that is specifically any new resources are added or if new permissions/actions are needed.
Are new permissions added in the policies section of "NodeInstanceRole" section of dwx-cf-template.yaml
Is this a bug fix for an old release, with missing permission, example DWX-15473
Backward compatibility
Testing: