Tuf: updater can't handle missing content-length

[jku]

Updater apparently cannot handle missing Content-Length header for metadata files

Current behavior:
Trying to get targetinfo for a file fails with
TypeError: '<' not supported between instances of 'NoneType' and 'int'
if a metadata file does not have content-length header.

Expected behavior:

Either

• Fetching the file succeeds even without Content-Length or
• Content-Length requirement is explicitly documented

In either case no comparisons are tried between NoneType and int. Possible other header usage should maybe be reviewed at the same time

Log

<snip...>
  http://127.0.0.1:80 "GET /tuf/2.bins.json HTTP/1.1" 200 None
  ERROR: Could not get content length about <Response [200]> from server: int() can't convert non-string with explicit base
  Traceback (most recent call last):
    File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/download.py", line 432, in _get_content_length
      reported_length = int(reported_length, 10)
  TypeError: int() can't convert non-string with explicit base
  The server reported a length of None bytes.
  ERROR: Could not download URL: 'http://127.0.0.1/tuf/2.bins.json'
  Traceback (most recent call last):
    File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/download.py", line 270, in _download_file
      _check_content_length(reported_length, required_length,
    File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/download.py", line 479, in _check_content_length
      if reported_length < required_length:
  TypeError: '<' not supported between instances of 'NoneType' and 'int'
  ERROR: Update failed from http://127.0.0.1/tuf/2.bins.json.
  Traceback (most recent call last):
    File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/client/updater.py", line 1506, in _get_metadata_file
      file_object = tuf.download.unsafe_download(file_mirror,
    File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/download.py", line 150, in unsafe_download
      return _download_file(url, required_length, STRICT_REQUIRED_LENGTH=False)
    File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/download.py", line 270, in _download_file
      _check_content_length(reported_length, required_length,
    File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/download.py", line 479, in _check_content_length
      if reported_length < required_length:
  TypeError: '<' not supported between instances of 'NoneType' and 'int'
  ERROR: Failed to update '2.bins.json' from all mirrors: {'http://127.0.0.1/tuf/2.bins.json': TypeError("'<' not supported between instances of 'NoneType' and 'int'")}
  ERROR: Metadata for 'bins' cannot be updated.
ERROR: Exception:
Traceback (most recent call last):
  File "/home/jku/src/pip/src/pip/_internal/cli/base_command.py", line 208, in _main
    status = self.run(options, args)
  File "/home/jku/src/pip/src/pip/_internal/cli/req_command.py", line 184, in wrapper
    return func(self, options, args)
  File "/home/jku/src/pip/src/pip/_internal/commands/install.py", line 327, in run
    requirement_set = resolver.resolve(
  File "/home/jku/src/pip/src/pip/_internal/resolution/legacy/resolver.py", line 180, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/jku/src/pip/src/pip/_internal/resolution/legacy/resolver.py", line 385, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/jku/src/pip/src/pip/_internal/resolution/legacy/resolver.py", line 337, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/home/jku/src/pip/src/pip/_internal/operations/prepare.py", line 451, in prepare_linked_requirement
    local_file = unpack_url(
  File "/home/jku/src/pip/src/pip/_internal/operations/prepare.py", line 255, in unpack_url
    file = get_http_url(
  File "/home/jku/src/pip/src/pip/_internal/operations/prepare.py", line 129, in get_http_url
    from_path, content_type = downloader.download(link, temp_dir.path)
  File "/home/jku/src/pip/src/pip/_internal/network/download.py", line 255, in download
    target = self._updater.get_one_valid_targetinfo(path)
  File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/client/updater.py", line 2727, in get_one_valid_targetinfo
    target = self._preorder_depth_first_walk(target_filepath)
  File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/client/updater.py", line 2801, in _preorder_depth_first_walk
    self._refresh_targets_metadata(role_name,
  File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/client/updater.py", line 2525, in _refresh_targets_metadata
    self._update_metadata_if_changed(rolename)
  File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/client/updater.py", line 1951, in _update_metadata_if_changed
    self._update_metadata(metadata_role, upperbound_filelength,
  File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/client/updater.py", line 1785, in _update_metadata
    self._get_metadata_file(metadata_role, remote_filename,
  File "/home/jku/src/pip/venv/lib/python3.8/site-packages/tuf/client/updater.py", line 1602, in _get_metadata_file
    raise tuf.exceptions.NoWorkingMirrorError(file_mirror_errors)
tuf.exceptions.NoWorkingMirrorError: No working mirror was found:
  '127.0.0.1': TypeError("'<' not supported between instances of 'NoneType' and 'int'")

[jku]

The point of this code isn't very clear. _check_content_length() only logs a debug message and that's it: nothing is actually done.

In early pre-history there were exceptions raised if the Content-Length was not what was expected but they were removed in a massive "demo2" branch merge in August 2013.

[jku]

I think the only reasonable things that we could do with content-length are:

• abort before download if content_length > required_length
• when downloading, download (maximum) content_length bytes instead of required_length bytes

I'm just not totally convinced these are worth it: I believe they only protect against a not-quite-endless-data attacker who can insert large metadata files to server but can't modify response headers (and for target files the header should be useless since the correct way to protect against these attacks is including the length in the metadata)

I think I'll make a PR for removing the current content-length code: I can implement the above checks if there's strong preference for that

[lukpueh]

Thanks for your offer to implement these checks. I think we are fine without them.