Aws-sdk-java: "Not all bytes were read from the S3ObjectInputStream" message is impossible to avoid with a ZipInputStream

Created on 10 Apr 2017 · 4Comments · Source: aws/aws-sdk-java

We store zip files on S3 and consume the stream of selected entries using the following code:

final ZipInputStream zipStream = new ZipInputStream(s3ObjectInputStream) {
    @Override
    public void close() throws IOException {
        // Drain before closing to keep S3 client happy
        while (getNextEntry() != null);
        while (read() >= 0);
        super.close();
    }
};
ZipEntry entry;
while ((entry = zipStream.getNextEntry()) != null) {
    if (entryFilter.matches(entry.getName())) {
        return zipStream;
    }
}

Even though we fully drain the stream before closing it, the warning persists. What are we doing wrong?

response-requested

Source

axelfontaine

Most helpful comment

ZIP archives have a (redundant) central directory structure at the end, so you can list the contents of the archive without scanning through the whole thing. Java's ZipInputStream never actually consumes this from the underlying stream; getNextEntry() returns null as soon as it finds the start of the central directory.

You might try adding while (in.read() >= 0); in your overridden close method to read through to the end of the underlying stream?