Druid: Druid 0.13.0 - Warning on Historicals "Not all bytes were read from the S3ObjectInputStream"

[Stephan3555]

Hello guys,

since our migration from Druid 0.12.3 to 0.13.0 the Historicals consistently throwing following warnings:

2019-01-23T12:45:06,949 WARN [ZkCoordinator] com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.

Is this something I have to be concerned? Because the cluster seems to work anyway.

Could be related to this:
https://github.com/aws/aws-sdk-java/issues/1211

[sascha-coenen]

interesting. I see the same thing on my middlemanagers. No idea yet why and how to overcome...
Thanks for creating the ticket about it.

[divyad1508]

I am also seeing this warning in my historical logs. Although, there has been no problem in the druid cluster so far.

Please let us know, if any update on the same.

Thanks!

[Stephan3555]

We skipped version 0.13.0 and went straight to 0.14.0. The Problem seems to be resolved in this version

[Stephan3555]

Nope, still occurs in Druid 0.14.0

2019-05-07T13:55:18,998 WARN [ZkCoordinator] com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.

Anyone experience this as well?

[gianm]

I've seen it too. Looking at the source of S3AbortableInputStream, this happens when a stream is closed without all bytes being read. I'm not sure what the cause might be. Maybe something in S3DataSegmentPuller that isn't fully reading the stream?

[trevortitlestad]

We are currently running 0.14.2-incubating and noticed the same thing on the Historical nodes when swapping out the servers. Once the Historical servers had finished loading the data back into the cluster and the load queues were empty, the log lines stopped and all appeared to go back to normal.

WARN [ZkCoordinator] com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.

[mlubavin-vg]

We are seeing this same thing on version 0.16.0

[sascha-coenen]

We are seeing this same thing on version 0.16.0
we see it too. its still happening in 0.16.0

[smokemonster99]

I see it in 0.17.0 on historicals.

[sh-homelab]

I see it also in 0.18.1.

[josh-pritchard-fcx]

Same here updating from 0.12.1 to 0.18.1

[dbrojas]

Got this error from the middleManagers on 0.19.0-rc1

[gianm]

I learned a lot about the ZIP format today 🙂

I did a little testing and I suspect this is due to the fact that ZipInputStream, which is used through CompressionUtils.unzip when pulling segment files, does not read the central directory at the end of zip files. It only reads the local file headers and local file data. (See https://users.cs.jmu.edu/buchhofp/forensics/formats/pkzip.html.) It closes the file when it first encounters the central directory, rather than reading the whole thing.

I think this is OK, because Druid itself is writing these zip files, and it isn't using any of the zip features (file replacement, deletion) that would require usage of the central directory. That means these messages are harmless, but they are annoying. We should be able to get rid of them by modifying CompressionUtils.unzip to read the rest of the stream after unzipping is done.

We should also implement a non-streaming version of CompressionUtils.unzip that is used in situations where Druid itself didn't write the zip file. In these situations, it's important to read the central directory.

[gianm]

10664 should fix this.