Spring Cloud Data Flow S3 Source giving access denied from eks eventhough other applications able to access S3

[Noushath]

Our team is trying to do a poc , where we are trying to use spring cloud dataflow (SCDF) as our data-pipeline integrator.
We have deployed scdf on aws eks and the nodes have the respective roles to access S3. But we see when we are trying to deploy a stream in SCDF with s3 as source we are getting Access Denied. Is there any specific app parameters we need to specify, while using it from aws eks cluster so the stream is able to take access related information from node roles. While running the same stream on windows laptop with local .aws/credentials where the access key and secret access key are set in default profile via aws cli aws config it works fine with below config:

app.file.directory=C:/Users/xxx/Documents/xxx
app.s3.auto-create-local-dir=false
app.s3.file.consumer.mode=lines
app.s3.local-dir=C:/Users/xxx/Documents/yyy
app.s3.remote-dir=xxxx
spring.cloud.dataflow.skipper.platformName=default
app.s3.cloud.aws.stack.auto=false
app.s3.cloud.aws.stack=false
app.s3.cloud.aws.credentials.profileName=default
app.s3.supplier.filename-regex=.*\.csv$
app.s3.cloud.aws.region.static=us-east-1
app.s3.logging.level.com.amazonaws.internal.InstanceMetadataServiceResourceFetcher=error
app.s3.logging.level.com.amazonaws.util.EC2MetadataUtils=error

But when deployed in eks cluster the stream is not able to take credentials from node role. Appreciate if some one can direct what is the config params to be used in cluster.

[maciejwalkowiak]

I can't tell for sure, but as far as I know, when using AWS SDK on EKS, you need a credentials provider from aws-java-sdk-sts module in AWS SDK. You should also remove app.s3.cloud.aws.credentials.profileName=default from your config. This will make Spring Cloud AWS use default credentials chain provided by the SDK that will automatically use STS once the dependency to aws-java-sdk-sts is on the classpath. Let me know if it worked!

[Noushath]

In eks we are using a slightly different config as below, we are getting errors like

org.springframework.messaging.MessagingException: Problem occurred while synchronizing 'bucket' to local directory; nested exception is org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: us-east-1. Please use this region to retry the request

Actually our bucket is in us-east-2 and we are specifying same as in below stream definition
s3 --auto-create-local-dir=false --local-dir=/tmp --supplier.filename-regex='.*\.csv$' --cloud.aws.credentials.useDefaultAwsCredentialsChain=true --cloud.aws.region.auto=false --cloud.aws.region.static=us-east-2 --remote-dir=xxxxx --file.consumer.mode=lines --cloud.aws.stack.auto=false > :someKafkaRawTopic

It would be helpful if we can get a sample on how the stream definition should look like for access s3 bucket form aws eks deployed SCDF.

[maciejwalkowiak]

Please consult Spring Cloud Data Flow team. To be honest I don't even know if it uses Spring Cloud AWS under the hood and which version if it does.

com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: us-east-1

This exception comes from AWS SDK which means that this bucket really is in us-east-1 region.

[artembilan]

We do use Spring Cloud AWS and rely on its auto-configuration:

	@Bean
	@ConditionalOnMissingBean
	public AmazonS3 amazonS3(AWSCredentialsProvider awsCredentialsProvider, RegionProvider regionProvider) {
		return AmazonS3ClientBuilder.standard()
				.withCredentials(awsCredentialsProvider)
				.withRegion(regionProvider.getRegion().getName())
				.build();
	}

So, technically there is nothing to do from Spring Cloud Dataflow perspective: everything has to be done from auto-configuration which may really rely on the default credentials chain from AWS SDK.
The version we use at the moment is this <spring-cloud-aws.version>2.2.4.RELEASE</spring-cloud-aws.version> and fully rely in the AWS SDK pulled transitively.

[Noushath]

Is there a way to config parameter to tell AWS how to tell what is the credentials provider and region from eks?

This is the stream definition used in eks which fails with access denied even though the k8s node has the required roles

s3 --auto-create-local-dir=false --local-dir="/tmp" --supplier.filename-regex=".*.csv$" --cloud.aws.region.auto=false --cloud.aws.region.static=us-east-1 --remote-dir=xxxxx --file.consumer.mode=lines --cloud.aws.stack.auto=false --cloud.aws.credentials.use-default-aws-credentials-chain=true > :scdfRawTopic