Spark-on-k8s-operator: What I am doing wrong

Created on 26 Sep 2019  路  17Comments  路  Source: GoogleCloudPlatform/spark-on-k8s-operator

I am trying to run a Pyspark application using operator. I can run it perfectly if I backed the python application in spark image but when I am trying to get them from s3, I am getting into all sort issue. Please advise what I am doing wrong:

Here is my YAML File

apiVersion: "sparkoperator.k8s.io/v1beta1"
kind: SparkApplication
metadata:
  name: generic-pyspark2.4.4
  namespace: random
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: 'pyspark-2.4.4-hadoop-2.7:v0.1"
  imagePullPolicy: Always
  class: org.apache.spark.deploy.PythonRunner
  mainApplicationFile: "s3a://buckets/pyspark/model.py"
  sparkConf:
    "spark.hadoop.fs.s3a.aws.credentials.provider": com.amazonaws.auth.InstanceProfileCredentialsProvider
    "spark.hadoop.fs.s3a.impl": org.apache.hadoop.fs.s3a.S3AFileSystem
    "spark.shuffle.service.enabled": false
    "spark.speculation": false
  deps:
    pyFiles:
      - "s3a://buckets/pyspark/aws_utils.py"
      - "s3a://buckets/pyspark/dataset.py"
  sparkVersion: "2.4.4"
  driver:
    cores: 2
    # coreLimit: "1200m"
    memory: "1024m"
    labels:
      version: 2.4.4
    serviceAccount: sparkoperator
  executor:
    cores: 4
    instances: 5
    memory: "10240m"
    labels:
      version: 2.4.4

I am using

  • spark - 2.4.4
  • aws-java-sdk-1.7.3.jar
  • hadoop-aws-2.7.3.jar
  • scala: 2.11
picture kodelint

Most helpful comment

Ahh...that's exactly what it was...when I updated my helm installation command to provide a pre-existing service account with IAM role attached to it, it worked fine. Thanks a ton for the guidance @bbenzikry. Very grateful :). Cheers :)

picture batCoder95 on 23 Nov 2020
👍2

All 17 comments

I always get the error:

19/09/26 01:22:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Traceback (most recent call last):
  File "/tmp/spark-de1303c8-6a8d-47d7-bbd1-23f953fd03a6/model.py", line 1, in <module>
    from aws_utils import *
ModuleNotFoundError: No module named 'aws_utils'

Which mean it is not able to locate the dependent files

kodelint picture kodelint on 26 Sep 2019

Please check out https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/404.

liyinan926 picture liyinan926 on 26 Sep 2019

@liyinan926 I already have the jars in the image. Here is my Dockerfile

FROM gcr.io/spark-operator/spark-py:v2.4.4
ENV PYTHONPATH "$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PYTHONPATH"
ENV PYSPARK_PYTHON "python3"
RUN rm -f /usr/bin/python && ln -s /usr/bin/python3 /usr/bin/python
RUN  apk update \
     && apk add make automake gcc g++ subversion python3-dev
RUN  pip3.6 install Pathlib numpy pandas==0.24.2 boto3
COPY jars/* /opt/spark/jars/

and the jars folder has following:

>> ls -l jars/ | awk '{print $9}'
aws-java-sdk-1.7.3.jar
hadoop-aws-2.7.3.jar
kodelint picture kodelint on 26 Sep 2019

See this comment: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/404#issuecomment-472259793. The default operator image may not have the necessary jars to resolve s3:// dependencies during submission. You need to make sure that the relevant jars are also in $SPARK_HOME/jars in the operator image.

liyinan926 picture liyinan926 on 26 Sep 2019

Thanks for pointer @liyinan926 Now this is what I have.

Spark Operator Image Dockerfile:

FROM gcr.io/spark-operator/spark-operator:v2.4.0-v1beta1-0.8.2

ADD http://central.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.3/hadoop-aws-2.7.3.jar $SPARK_HOME/jars
ADD http://central.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar $SPARK_HOME/jars

Spark Driver and Executer Image Dockerfile:

FROM gcr.io/spark-operator/spark-py:v2.4.4
ADD http://central.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.3/hadoop-aws-2.7.3.jar $SPARK_HOME/jars
ADD http://central.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar $SPARK_HOME/jars
ENV PYTHONPATH "$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PYTHONPATH"
ENV PYSPARK_PYTHON "python3"
RUN rm -f /usr/bin/python && ln -s /usr/bin/python3 /usr/bin/python
RUN  apk update \
     && apk add make automake gcc g++ subversion python3-dev
RUN  pip3.6 install Pathlib numpy pandas==0.24.2 boto3

and my YAML

apiVersion: "sparkoperator.k8s.io/v1beta1"
kind: SparkApplication
metadata:
  name: generic-pyspark2.4.4
  namespace: random
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: 'pyspark-2.4.4-hadoop-2.7:v0.1"
  imagePullPolicy: Always
  class: org.apache.spark.deploy.PythonRunner
  mainApplicationFile: "s3a://buckets/pyspark/model.py"
  sparkConf:
    "spark.hadoop.fs.s3a.aws.credentials.provider": com.amazonaws.auth.InstanceProfileCredentialsProvider
    "spark.hadoop.fs.s3a.impl": org.apache.hadoop.fs.s3a.S3AFileSystem
    "spark.shuffle.service.enabled": false
    "spark.speculation": false
  deps:
    pyFiles:
      - "s3a://buckets/pyspark/preprocess.zip"
  sparkVersion: "2.4.4"
  driver:
    cores: 2
    # coreLimit: "1200m"
    memory: "1024m"
    labels:
      version: 2.4.4
    serviceAccount: sparkoperator
  executor:
    cores: 4
    instances: 5
    memory: "10240m"
    labels:
      version: 2.4.4

I also added the following to my main PySpark Application model.py:

dep_file_path = pyspark.SparkFiles.get("preprocess.zip")
# Init Single Spark Context and Wrap with SQLContext for monkey patches
LOG.info("Initializing Spark Context")
sc = SparkContext.addPyFile(dep_file_path)
sc = SparkContext(appName="preprocessor")
sqlContext = SQLContext(sc)
sqlContext.sql("set spark.sql.caseSensitive=true")
sc._conf.set("spark.shuffle.service.enabled", "false")
sc._conf.set("spark.dynamicAllocation.enabled" , "false")
sc._conf.set("spark.hadoop.fs.s3a.fast.upload", "true")
sc._conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
sc._conf.set("spark.debug.maxToStringFields", "500")
print("Spark Context Configs:\n")
print(sc._conf.getAll())
kodelint picture kodelint on 30 Sep 2019

@kodelint is it working now?

liyinan926 picture liyinan926 on 3 Oct 2019

@liyinan926 still i can't get the s3a working for the dependency files.

kodelint picture kodelint on 3 Oct 2019

@kodelint - Were you able to get the s3a working for the dependency files?

batCoder95 picture batCoder95 on 20 Oct 2020

Hi @bbenzikry,

Just wanted to check if you have come across such an issue? Since GCP operator does not have S3 in itself by default, the parameter spec.deps is not accepting S3 files as dependencies. If you have seen such an issue, is it possible for you guide on this?

I also tried the above mentioned approach by @kodelint, where I tried adding aws-java-sdk and hadoop-aws jars, but still not able to get it working.

batCoder95 picture batCoder95 on 17 Nov 2020

Hi @batCoder95 , as @liyinan926 mentioned, this should be resolved by baking the jars into the operator image. Are you using my operator image / something based on that for hive-site.xml inclusion? If so, we do not include the S3 jars there. If you did add the jars to the operator image - what scheme are you using? s3? s3a? do they both fail?

bbenzikry picture bbenzikry on 18 Nov 2020

Hi @bbenzikry,
I'm kind of using a combination of above answer about baking in the JARS with operator and the dockerfile I got from your spark-eks repo.

Basically, I'm following below steps in a custom Dockerfile

  • Use the GCP operator image as base image
  • Add and Unzip Hadoop Distribution from Apache Mirrors
  • Add Hadoop JAR from your spark-glue repo
  • Add AWS-Java-SDK JAR in Spark-JARS folder (/opt/spark/jars/)
  • Add AWS-Java-SDK in Hadoop-Lib folder (/opt/hadoop/share/hadoop/tools/lib)
  • Declare Env variables

Below is the Dockerfile content for reference:

FROM gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0

ARG spark_version=3.0.1
ARG hadoop_version=3.3.0
ARG scala_version=2.12
ARG aws_java_sdk_version=1.11.868

WORKDIR /opt/
ADD http://mirrors.whoishostingthis.com/apache/hadoop/common/hadoop-${hadoop_version}/hadoop-${hadoop_version}.tar.gz .
RUN tar -xvzf hadoop-${hadoop_version}.tar.gz
RUN mv hadoop-${hadoop_version} hadoop

WORKDIR /opt/spark/jars
ADD https://github.com/bbenzikry/spark-glue/releases/download/${spark_version}/spark-hadoop-cloud_${scala_version}-${spark_version}.jar .
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${aws_java_sdk_version}/aws-java-sdk-bundle-${aws_java_sdk_version}.jar .

WORKDIR /opt/hadoop/share/hadoop/tools/lib
RUN rm ./aws-java-sdk-bundle-*.jar
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${aws_java_sdk_version}/aws-java-sdk-bundle-${aws_java_sdk_version}.jar .
RUN chmod 0644 aws-java-sdk-bundle*.jar

ENV SPARK_HOME /opt/spark
ENV HADOOP_HOME /opt/hadoop
ENV SPARK_DIST_CLASSPATH="$HADOOP_HOME/etc/hadoop:$HADOOP_HOME/share/hadoop/common/lib/*:$HADOOP_HOME/share/hadoop/common/*:$HADOOP_HOME/share/hadoop/hdfs:$HADOOP_HOME/share/hadoop/hdfs/lib/*:$HADOOP_HOME/share/hadoop/hdfs/*:$HADOOP_HOME/share/hadoop/yarn:$HADOOP_HOME/share/hadoop/yarn/lib/*:$HADOOP_HOME/share/hadoop/yarn/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*:$HADOOP_HOME/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:$HADOOP_HOME/share/hadoop/tools/lib/*"
ENV SPARK_EXTRA_CLASSPATH="$SPARK_DIST_CLASSPATH:/hadoop/share/hadoop/tools/lib/*:$SPARK_HOME/jars/*"
ENV LD_LIBRARY_PATH /lib64

When I deploy this image as my operator and then submit a SparkApplication, I see below error while describing the SparkApplication:

park: SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/spark/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2020-11-19 10:57:16,818 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
           at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:893)
           at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:869)
           at org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1580)
           at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:341)
           at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
           at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
           at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
           at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
           at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
           at org.apache.spark.deploy.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:191)
           at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:147)
           at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:145)
           at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
           at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
           at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
           at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
           at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
           at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
           at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
           at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
           at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$6(SparkSubmit.scala:365)
           at scala.Option.map(Option.scala:230)
           at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:365)
           at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
           at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
           at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
           at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
           at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
           at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
           at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Warning  SparkApplicationFailed  4s  spark-operator  SparkApplication testpyspark failed: failed to run spark-submit for SparkApplication dev/testpyspark: SLF4J: Class path contains multiple SLF4J bindings.

Please let me know if you would like me to provide more details. Would greatly appreciate any guidance with this :)

batCoder95 picture batCoder95 on 19 Nov 2020

There are a few things I see here;

  1. You're still using the spark version available in the operator, but you're adding the hadoop-cloud jar ( which is just for working with S3A commiters ) - this jar is compiled to work with the 3.0.1 distro from my repo - not sure what this will do.
    I'm pretty sure you don't need S3A commiters for anything in the operator itself, just for job execution dockerfiles ( if you intend to use commiters )

  2. There's no need to touch anything related to the operator distros ( hadoop / spark )
    Just add the jars relevant for AWS.
    For new versions of the SDK, you need to add an updated guava version - this should probably ( fingers crossed ) resolve the error you see.

https://github.com/bbenzikry/spark-eks/blob/12cbf7e6b85e5e0f928410e62278e71114f3b2d5/docker/spark3.Dockerfile#L38-L39

bbenzikry picture bbenzikry on 19 Nov 2020

Ahh....this is really helpful @bbenzikry . As per your suggestion I tried removing everything from the Dockerfile except AWS-Java-SDK for Spark, and also added the Guava Jar as per your git link.

But then I started to get Hadoop-S3A-File-System-Not-Found errors. So on investigating further, I realized that Hadoop JARS need to be copied from Apache-Mirrors Hadoop Distro into Spark-Jars-Path (/opt/spark/jars)

So I added these Hadoop Jars in the Spark Jars path of the Operator and it seems its able to go to S3 connection step to try and fetch the dependency files from S3. But in this step I am now getting the error "you must provide role arn". This is inspite of the fact that I have defined the sparkConf "spark.kubernetes.authenticate.driver.serviceAccountName" in YAML.

Below is my updated Dockerfile for reference:

FROM gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0

ARG spark_version=3.0.1
ARG hadoop_version=3.3.0
ARG scala_version=2.12
ARG aws_java_sdk_version=1.11.797

WORKDIR /opt/
ADD http://mirrors.whoishostingthis.com/apache/hadoop/common/hadoop-${hadoop_version}/hadoop-${hadoop_version}.tar.gz .
RUN tar -xvzf hadoop-${hadoop_version}.tar.gz
RUN mv hadoop-${hadoop_version} hadoop

WORKDIR /opt/spark/jars
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${aws_java_sdk_version}/aws-java-sdk-bundle-${aws_java_sdk_version}.jar .
RUN rm -f guava-*.jar
ADD https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.jar .
RUN rm -r hadoop-*.jar
RUN cp /opt/hadoop/share/hadoop/tools/lib/hadoop-*.jar /opt/spark/jars

WORKDIR /opt/hadoop/share/hadoop/tools/lib
RUN rm ./aws-java-sdk-bundle-*.jar
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${aws_java_sdk_version}/aws-java-sdk-bundle-${aws_java_sdk_version}.jar .
RUN chmod 0644 aws-java-sdk-bundle*.jar

ENV SPARK_HOME /opt/spark
ENV HADOOP_HOME /opt/hadoop
ENV SPARK_DIST_CLASSPATH="$HADOOP_HOME/etc/hadoop:$HADOOP_HOME/share/hadoop/common/lib/*:$HADOOP_HOME/share/hadoop/common/*:$HADOOP_HOME/share/hadoop/hdfs:$HADOOP_HOME/share/hadoop/hdfs/lib/*:$HADOOP_HOME/share/hadoop/hdfs/*:$HADOOP_HOME/share/hadoop/yarn:$HADOOP_HOME/share/hadoop/yarn/lib/*:$HADOOP_HOME/share/hadoop/yarn/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*:$HADOOP_HOME/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:$HADOOP_HOME/share/hadoop/tools/lib/*"
ENV SPARK_EXTRA_CLASSPATH="$SPARK_DIST_CLASSPATH:/hadoop/share/hadoop/tools/lib/*:$SPARK_HOME/jars/*"
ENV LD_LIBRARY_PATH /lib64

Below is the exact error for reference:

2020-11-19 20:13:22,300 WARN fs.FileSystem: Failed to initialize fileystem s3a://spark-bucket/DependencyCode.py: java.lang.NullPointerException: You must specify a value for roleArn and roleSessionName
Exception in thread "main" java.lang.NullPointerException: You must specify a value for roleArn and roleSessionName
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$Builder.<init>(STSAssumeRoleSessionCredentialsProvider.java:359)
        at com.amazonaws.services.securitytoken.internal.STSProfileCredentialsService.getAssumeRoleCredentialsProvider(STSProfileCredentialsService.java:31)
        at com.amazonaws.auth.profile.internal.securitytoken.STSProfileCredentialsServiceProvider.getProfileCredentialsProvider(STSProfileCredentialsServiceProvider.java:39)
        at com.amazonaws.auth.profile.internal.securitytoken.STSProfileCredentialsServiceProvider.getCredentials(STSProfileCredentialsServiceProvider.java:71)
        at com.amazonaws.auth.WebIdentityTokenCredentialsProvider.getCredentials(WebIdentityTokenCredentialsProvider.java:76)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1257)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:833)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:783)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5850)
        at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:5823)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5046)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
        at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3919)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1244)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1234)
        at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExistsV2$2(S3AFileSystem.java:575)
        at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
        at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:315)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
        at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:311)
        at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:286)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExistsV2(S3AFileSystem.java:574)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing(S3AFileSystem.java:494)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:397)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
        at org.apache.spark.deploy.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:191)
        at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:147)
        at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:145)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
        at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
        at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$6(SparkSubmit.scala:365)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:365)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Would you happen to know something about this?

batCoder95 picture batCoder95 on 20 Nov 2020

@batCoder95, I'm guessing the service account on the operator pod itself should have the proper credentials / IRSA role defined. as mentioned here - https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/404#issuecomment-582579692, if you're using deps and not mainApplicationFile, the operator resolves files using its own chain.

bbenzikry picture bbenzikry on 20 Nov 2020

Ahh...that's exactly what it was...when I updated my helm installation command to provide a pre-existing service account with IAM role attached to it, it worked fine. Thanks a ton for the guidance @bbenzikry. Very grateful :). Cheers :)

batCoder95 picture batCoder95 on 23 Nov 2020
👍2

@batCoder95 what kind of base image are you using for your sparkApplication image? Are you copying spark and hadoop in there as well?
I followed your spark-operator image, but now running sparkApplication gives me the following error in the sparkApplication driver pod logs 

21/02/17 22:15:32 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/app/spark-warehouse').
21/02/17 22:15:32 INFO SharedState: Warehouse path is 'file:/app/spark-warehouse'.
An error occurred while calling o58.parquet.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:46)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:361)
        at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:279)
        at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:268)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:268)
        at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:737)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
        ... 25 more
JunaidChaudry picture JunaidChaudry on 18 Feb 2021

@JunaidChaudry You can take a look at https://github.com/bbenzikry/spark-eks/blob/main/docker/spark3.Dockerfile for a reference

bbenzikry picture bbenzikry on 18 Feb 2021
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

vishnugs picture vishnugs  路  8Comments
mjschmidt picture mjschmidt  路  7Comments
liyinan926 picture liyinan926  路  7Comments
leandro-rouberte picture leandro-rouberte  路  4Comments
hongshaoyang picture hongshaoyang  路  3Comments