Delta: Premature end of Content-Length delimited message body

[NimeshSatam]

Hi,

We have been running Spark streaming jobs along with Delta Lake for past couple of months. But for last 2 days we have been getting the below error while writing to S3. Can you please let me know what we may be doing wrong here?

"org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 43,688; received: 28,617)"

Details of the streaming job:
Apache Spark: 3.0.1
Delta Lake: 0.7.0
Python: 3.7.9
Object Store: AWS S3

Can you let me know the probable cause of the error?

Pyspark Command used:
PYSPARK_DRIVER_PYTHON=ipython $SPARK_HOME/bin/pyspark --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1,org.apache.spark:spark-avro_2.12:3.0.1,za.co.absa:abris_2.12:3.2.1,io.delta:delta-core_2.12:0.7.0,org.apache.hadoop:hadoop-aws:2.7.4,org.apache.hadoop:hadoop-common:2.7.4 --repositories https://packages.confluent.io/maven/ --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore

Sample Code snippet used to replicate the error:

`
import boto3
from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession, Window
import pyspark.sql.functions as F

spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.getOrCreate()

spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3.ap-south-1.amazonaws.com")
spark.sparkContext._jsc.hadoopConfiguration().set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
spark.sparkContext.setSystemProperty("com.amazonaws.services.s3.enableV4", "true")
spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", "true")
spark.conf.set("spark.delta.merge.repartitionBeforeWrite", "true")
spark.conf.set("delta.compatibility.symlinkFormatManifest.enabled", "true")

from delta.tables import *
deltaTable = DeltaTable.forPath(spark, "s3a://BUCKET_NAME/STREAMING_PATH")
deltaTable.generate("symlink_format_manifest")

`

[zsxwing]

This may be a regression of S3. Could you try to reach AWS for this issue?

[twsweeting]

We hit this problem recently too after an apt update - when you type java -version what do you see? If it's openjdk version "1.8.0_272" you may need to rollback to openjdk version "1.8.0_265"

When we run the following on our linux machines, then restart spark, the problem is resolved for us:

wget https://launchpad.net/~openjdk-security/+archive/ubuntu/ppa/+build/19763089/+files/openjdk-8-jre_8u265-b01-0ubuntu2~18.04_amd64.deb
wget https://launchpad.net/~openjdk-security/+archive/ubuntu/ppa/+build/19763089/+files/openjdk-8-jdk_8u265-b01-0ubuntu2~18.04_amd64.deb
wget https://launchpad.net/~openjdk-security/+archive/ubuntu/ppa/+build/19763089/+files/openjdk-8-jdk-headless_8u265-b01-0ubuntu2~18.04_amd64.deb
wget https://launchpad.net/~openjdk-security/+archive/ubuntu/ppa/+build/19763089/+files/openjdk-8-jre-headless_8u265-b01-0ubuntu2~18.04_amd64.deb
sudo dpkg -i openjdk-8-jre-headless_8u265-b01-0ubuntu2~18.04_amd64.deb
sudo dpkg -i openjdk-8-jdk-headless_8u265-b01-0ubuntu2~18.04_amd64.deb
sudo dpkg -i openjdk-8-jre_8u265-b01-0ubuntu2~18.04_amd64.deb
sudo dpkg -i openjdk-8-jdk_8u265-b01-0ubuntu2~18.04_amd64.deb

[NimeshSatam]

Thanks guys for the reply.

We tried upgrading from spark 3.0.0 to 3.0.1 and used hadoop version 3.2. This helped us to the fix the problem.

Post the upgrade we tried running the jobs on the same delta locations and haven't faced the problem so far.

Output of "java -version" shows as

openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-8u272-b10-0ubuntu1~18.04-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)

If faced with the problem in future we will try to downgrade the same.