The same here:

17/05/12 13:57:56 WARN redshift.Utils$: An error occurred while trying to read the S3 bucket lifecycle configuration
java.lang.NullPointerException
	at java.lang.String.startsWith(String.java:1405)
	at java.lang.String.startsWith(String.java:1434)
	at com.databricks.spark.redshift.Utils$$anonfun$5.apply(Utils.scala:140)
	at com.databricks.spark.redshift.Utils$$anonfun$5.apply(Utils.scala:136)
	at scala.collection.Iterator$class.exists(Iterator.scala:919)
	at scala.collection.AbstractIterator.exists(Iterator.scala:1336)
	at scala.collection.IterableLike$class.exists(IterableLike.scala:77)
	at scala.collection.AbstractIterable.exists(Iterable.scala:54)
	at com.databricks.spark.redshift.Utils$.checkThatBucketHasObjectLifecycleConfiguration(Utils.scala:136)
	at com.databricks.spark.redshift.RedshiftWriter.saveToRedshift(RedshiftWriter.scala:389)
	at com.databricks.spark.redshift.DefaultSource.createRelation(DefaultSource.scala:108)
	at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:426)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)
...
...

I thought it had to do with not setting a bucket prefix when configuring the lifecycle policy but even after setting it, it keeps showing (although the operation succeeds)

+1

I think I have a PR that fixes this (you need to upgrade AWS SDK dependencies). See: #357

Disclaimer: New to Scala
I am using the jars from http://repo1.maven.org/maven2/com/databricks/ and I am getting this same error when trying to write to redshift. I am running this from the Spark shell for debugging and every time I do I get this error and my shell hangs but the operation never completes.

+1 seeing this in pyspark

+1

It seems to me that this issue has something to do with the fact that com.amazonaws.aws-java-sdk-s3 dependency is provided. When I run spark job locally from ide it works, but it does not work on AWS EMR. I think when deployed jar uses libraries, provided by aws environment, which are probably new and conflict with spark-redshift. Meanwhile locally it uses old libraries and as a result, it works. As a temporary solution a suggest to use fix by @BorePlusPlus .

    <repositories>
        <repository>
            <id>jitpack.io</id>
            <url>https://jitpack.io</url>
        </repository>
    </repositories>
...
    <dependency>
        <groupId>com.github.BorePlusPlus</groupId>
        <artifactId>spark-redshift_2.11</artifactId>
        <version>bucket-lifecycle-check-upgrade-SNAPSHOT</version>
    </dependency>

I agree that this is a super annoying error, since the stack trace is so long. This solution worked for me:

spark.sparkContext.setLogLevel("ERROR")

I got the suggestion from here.

+1

For us it turned out "the file is not there" - that is being attempted to be read and thus

"An error occurred while trying to read the S3 bucket lifecycle configuration
java.lang.NullPointerException"

and a subsequent

"S3ServiceException:Access Denied,Status 403,Error AccessDenied,"

It would seem we are reading before the file is available - parallel processing woes?

Object not found results in 403 (access denied) rather than 404 (not found) because different return codes would provide an attacker with useful information - it leaks information that an object of a given name actually exists. A simple dictionary-style attack could then enumerate all of the objects in someone's bucket. For a similar reason, a login page should never emit "Invalid user" and "Invalid password" for the two authentication failure scenarios; it should always emit "Invalid credentials".

A fix would then be

Check the regions.

For example: It was because the region was set to "us-west-2" that was visible on the aws console link. However the contents were hosted on ap-southeast-1.

Check Permissions.

By default, permissions are given to the AWS user only. If you use IAM authentication with access keys, you must add permissions to "authenticated users" in S3.

"...If the object you request does not exist, the error Amazon S3 returns depends on whether you also have the s3:ListBucket permission.

If you have the s3:ListBucket permission on the bucket, Amazon S3 will return an HTTP status code 404 ("no such key") error. if you don’t have the s3:ListBucket permission, Amazon S3 will return an HTTP status code 403 ("access denied") error."

Keep your role policy as in the helloV post.
Go to S3. Select your bucket. Click Permissions. Click Bucket Policy.
Try something like this:
{
"Version": "2012-10-17",
"Id": "Lambda access bucket policy",
"Statement": [
{
"Sid": "All on objects in bucket lambda",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWSACCOUNTID:root"
},
"Action": "s3:",
"Resource": "arn:aws:s3:::BUCKET-NAME/"
},
{
"Sid": "All on bucket by lambda",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWSACCOUNTID:root"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::BUCKET-NAME"
}
]
}

Your architecture chooses the right solution, hope this helps

By now, I have implemented multiple Spark applications with this library and the issue does not affect anything.

i solve the problem inverting this params
before :

sc._jsc.hadoopConfiguration().set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", aws_id)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", aws_key)

after:

sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", aws_id)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", aws_key)
sc._jsc.hadoopConfiguration().set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")

+1
Any news how to fix this properly?

+1 Just saw this happening using databricks runs (using Spark 3.2.1).

I was able to silence this by setting this piece of code's logger to ERROR

import org.apache.log4j.{Level, Logger}

// insert this line after spark session initiation
Logger.getLogger("com.databricks.spark.redshift.Utils$").setLevel(Level.ERROR)