Tuf: Client explicitly asks server to not compress data

[sechkova]

Description of issue or feature request:

After the acceptance of TAP10 the framework no longer handles compressed metadata. Instead,

the specification should instead recommend that compression be implemented at the presentation layer of the OSI protocol stack, which is better suited to handle decompression of data. Once decompressed, metadata can be validated against the hashes and signatures listed in the Snapshot file.

However, the implementation explicitly asks server to not compress data to avoid potential issues with some server configurations.

Two questions coming from this observation:

• Do all the calculations up to now (for example the ones in PEP458) consider uncompressed metadata?
• Should the implementation match tap10 and allow handling of compressed metadata at a lower level?

Current behavior:
Client explicitly asks server to not compress data

Expected behavior:
Allow handling of compressed metadata at the presentation layer

[joshuagl]

Looks like this session header was introduced (https://github.com/theupdateframework/tuf/commit/d03dd0f2ecbaad0886bf2c22e7fbeb2dec9813b7) back in 2013, when adding support for SSL certificate verification, and has since been carried across multiple rewrites of tuf.download.

Use of this header was originally copy/pasted from pip (which pip introduced to handle some unexpected behaviour in urllib2). AFAICT from a brief perusal pip now uses the default headers implemented by requests of 'Accept-Encoding': ', '.join(('gzip', 'deflate')).

[joshuagl]

However, the implementation explicitly asks server to not compress data to avoid potential issues with some server configurations.

As far as I can tell with some git archaeology, the server configurations in question are those which supply unexpected encodings when Accept-Encoding is empty.

I vote that we remove any setting of 'Accept-Encoding' and use the default settings of the underlying requests framework. In future if users want a non-default encoding, they can implement their own fetcher (see #1250) to achieve this.

[trishankatdatadog]

Looks like this session header was introduced (d03dd0f) back in 2013, when adding support for SSL certificate verification, and has since been carried across multiple rewrites of tuf.download.

I did not sign that commit, so it might or might not have been me from a different life...

[jku]

I vote that we remove any setting of 'Accept-Encoding' and use the default settings of the underlying requests framework. In future if users want a non-default encoding, they can implement their own fetcher (see #1250) to achieve this.

I lean to the same solution... but I'll still document the alternative though (sechkova already said this but just for emphasis): allow compression _only for metadata_. We know the metadata is not compressed "on disk" so that should always be safe, even with webservers broken as described in the below comment.

AFAICT from a brief perusal pip now uses the default headers implemented by requests of 'Accept-Encoding': ', '.join(('gzip', 'deflate')).

In pip (src/_internal/network/utils.py) the same 'identity' encoding trick with the same reasoning is still used:

# We use Accept-Encoding: identity here because requests defaults to
# accepting compressed responses. This breaks in a variety of ways
# depending on how the server is configured.
# - Some servers will notice that the file isn't a compressible file
#   and will leave the file alone and with an empty Content-Encoding
# - Some servers will notice that the file is already compressed and
#   will leave the file alone, adding a Content-Encoding: gzip header
# - Some servers won't notice anything at all and will take a file
#   that's already been compressed and compress it again, and set
#   the Content-Encoding: gzip header
# By setting this to request only the identity encoding we're hoping
# to eliminate the third case.  Hopefully there does not exist a server
# which when given a file will notice it is already compressed and that
# you're not asking for a compressed file and will then decompress it
# before sending because if that's the case I don't think it'll ever be
# possible to make this work.
HEADERS = {'Accept-Encoding': 'identity'}  # type: Dict[str, str]

This is something I will have to deal with in the pip branch (to make sure metadata does get compressed).

[sechkova]

Another observation is that currently TUF deals with raw response content.

IIUC the requests' docs, we would like to deal with raw content for target files but use the requests framework to decode potentially compressed metadata files as shown in the example:

with open(filename, 'wb') as fd:
    for chunk in r.iter_content(chunk_size=128):
        fd.write(chunk)

[trishankatdatadog]

IIUC the requests' docs, we would like to deal with raw content for target files but use the requests framework to decode potentially compressed metadata files as shown in the example:

httpx has a similar API. My only concern here might be things like ZIP bomb attacks, but we should be able to avoid them as long as we iterate by reasonably-fixed-size chunks.

[joshuagl]

I'm leaning towards leaving this behaviour as-is for the current codebase, and addressing this in the reference fetcher during our refactor efforts.

[jku]

I'm leaning towards leaving this behaviour as-is for the current codebase, and addressing this in the reference fetcher during our refactor efforts.

:+1: This is fine for the pip use case if we manage to do the network io work -- then I can make sure metadata gets compressed in the pip-specific fetcher.