Delta: Using AWS Glue ETL & Delta Lake issue encountered

[rubinder]

I was using the Delta Lake jar in a Glue ETL spark job and the output parquet files write out fine but ran into some issues writing the delta log. The delta log is not written. The error is

WARN EC2MetadataUtils: Unable to retrieve the requested metadata (/latest/user-data/). The requested metadata is not found at http://169.254.169.254/latest/user-data/
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: The requested metadata is not found at http://169.254.169.254/latest/user-data/
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:122)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:82)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:400)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:369)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:365)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.util.EC2MetadataUtils.getUserData(EC2MetadataUtils.java:361)
    at com.amazon.ws.emr.hadoop.fs.util.UserData.getUserData(UserData.java:64)
    at com.amazon.ws.emr.hadoop.fs.util.UserData.<init>(UserData.java:39)
    at com.amazon.ws.emr.hadoop.fs.util.UserData.ofDefaultResourceLocations(UserData.java:52)
    at com.amazon.ws.emr.hadoop.fs.util.AWSSessionCredentialsProviderFactory.buildSTSClient(AWSSessionCredentialsProviderFactory.java:52)
    at com.amazon.ws.emr.hadoop.fs.util.AWSSessionCredentialsProviderFactory.<clinit>(AWSSessionCredentialsProviderFactory.java:17)
    at com.amazon.ws.emr.hadoop.fs.rolemapping.DefaultS3CredentialsResolver.resolve(DefaultS3CredentialsResolver.java:22)
    at com.amazon.ws.emr.hadoop.fs.guice.CredentialsProviderOverrider.override(CredentialsProviderOverrider.java:25)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.executeOverriders(GlobalS3Executor.java:130)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:86)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:184)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.doesBucketExist(AmazonS3LiteClient.java:90)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.ensureBucketExists(Jets3tNativeFileSystemStore.java:139)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:116)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.initialize(S3NativeFileSystem.java:508)
    at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:111)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2859)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2896)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2878)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
    at org.apache.spark.deploy.DependencyUtils$.org$apache$spark$deploy$DependencyUtils$$resolveGlobPath(DependencyUtils.scala:191)
    at org.apache.spark.deploy.DependencyUtils$$anonfun$resolveGlobPaths$2.apply(DependencyUtils.scala:147)
    at org.apache.spark.deploy.DependencyUtils$$anonfun$resolveGlobPaths$2.apply(DependencyUtils.scala:145)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$3.apply(SparkSubmit.scala:354)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$3.apply(SparkSubmit.scala:354)
    at scala.Option.map(Option.scala:146)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:354)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
19/11/22 15:55:46 ERROR UserData: Error encountered while try to get user data
java.lang.NullPointerException
    at com.amazon.ws.emr.hadoop.fs.shaded.com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:871)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
    at com.amazon.ws.emr.hadoop.fs.util.UserData.getUserData(UserData.java:66)
    at com.amazon.ws.emr.hadoop.fs.util.UserData.<init>(UserData.java:39)
    at com.amazon.ws.emr.hadoop.fs.util.UserData.ofDefaultResourceLocations(UserData.java:52)
    at com.amazon.ws.emr.hadoop.fs.util.AWSSessionCredentialsProviderFactory.buildSTSClient(AWSSessionCredentialsProviderFactory.java:52)
    at com.amazon.ws.emr.hadoop.fs.util.AWSSessionCredentialsProviderFactory.<clinit>(AWSSessionCredentialsProviderFactory.java:17)
    at com.amazon.ws.emr.hadoop.fs.rolemapping.DefaultS3CredentialsResolver.resolve(DefaultS3CredentialsResolver.java:22)
    at com.amazon.ws.emr.hadoop.fs.guice.CredentialsProviderOverrider.override(CredentialsProviderOverrider.java:25)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.executeOverriders(GlobalS3Executor.java:130)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:86)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:184)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.doesBucketExist(AmazonS3LiteClient.java:90)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.ensureBucketExists(Jets3tNativeFileSystemStore.java:139)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:116)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.initialize(S3NativeFileSystem.java:508)
    at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:111)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2859)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2896)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2878)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
    at org.apache.spark.deploy.DependencyUtils$.org$apache$spark$deploy$DependencyUtils$$resolveGlobPath(DependencyUtils.scala:191)
    at org.apache.spark.deploy.DependencyUtils$$anonfun$resolveGlobPaths$2.apply(DependencyUtils.scala:147)
    at org.apache.spark.deploy.DependencyUtils$$anonfun$resolveGlobPaths$2.apply(DependencyUtils.scala:145)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$3.apply(SparkSubmit.scala:354)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$3.apply(SparkSubmit.scala:354)
    at scala.Option.map(Option.scala:146)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:354)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)   

I think the issue is in AWS Glue ETL the process running the spark job does not have access to the EC2 metadata since other jobs from other clients could be running on the host as opposed to AWS EMR.

Is Delta Lake only focused on AWS EMR support?
Not a critique just curious.
Thanks

[jose-torres]

The Delta Lake project is focused on general Spark support, and doesn't currently implement or compatibility with any particular third-party Spark execution service. This looks like an issue in the Hadoop filesystem implementation that your AWS Glue ETL process is using.

[ymwdalex]

@rubinder How did you start up Glue job iwth delta lake?

What I did is download the jar from here https://repo1.maven.org/maven2/io/delta/delta-core_2.11/0.2.0/delta-core_2.11-0.2.0.jar, put it as a glue jar library, I can use delta lake to read and write.

However, I cannot import "delta" pakcage, and get "ModuleNotFound error". How did you import the python package?

[jaceklaskowski]

@ymwdalex Please note that the latest version of delta is 0.5.0.

@rubinder You may have more luck with uber-packaging / shading delta as the exception seems to be due to version mismatch between jackson packages (that delta uses for json parsing).

[condamoor-sap]

Hi,
I am getting the same error..
20/03/12 23:30:22 WARN EC2MetadataUtils: Unable to retrieve the requested metadata (/latest/user-data/). The requested metadata is not found at http://169.254.169.254/latest/user-data/
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: The requested metadata is not found at http://169.254.169.254/latest/user-data/
.......
.......
ERROR UserData: Error encountered while try to get user data
java.lang.NullPointerException
at com.amazon.ws.emr.hadoop.fs.shaded.com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:871)

I was running the Spark code generated by AWS Glue. Didn't modify it in any way. What am I doing wrong?

[tdas]

If you are getting the same stack trace as in the description of this issue, then this not a Delta issue at all. From the stack trace, it seems that spark is trying to resolve the user through the EMRFilesystem. More specifically, spark-submit is trying to do that. So this probably happening even before your code has started, and likely due to some misconfiguration in EMR FileSystems, etc.

One way to isolate the problem is to run a simple, non-delta, spark job in the same setup and see whether this issue occurs again or not.

[pranavanand]

Hi @condamoor-sap ,

Did you try running a non-delta spark job with the same setup?

[condamoor-sap]

Hi Pranav,
It turned out to be a network configuration problem that required me to add an appropriately configured VPC/Subnet/Endpoint for my Glue Connector.
Thanks for checking.

-ravi

[pranavanand]

Glad to hear it!

[rubinder]

Hi Pranav,
It turned out to be a network configuration problem that required me to add an appropriately configured VPC/Subnet/Endpoint for my Glue Connector.
Thanks for checking.

-ravi

@condamoor-sap Were you able to Glue ETL + Delta Lake to work? Or just Glue ETL?

[tdas]

Closing this issue since the original problem seems to have been solved.