Tuf: Should we accept metadata that includes unrecognised fields?

[joshuagl]

Description of issue or feature request:

The current reference implementation will ignore unrecognised fields in a metadata file when loading it. Furthermore, some TAPs (i.e. snapshot Merkle Tree) are relying on this behaviour to introduce new fields to the specification without making the metadata invalid for older clients which don't recognise the field.

Current behavior:

The current client implementation ignores unrecognised/unexpected fields.

The new metadata API currently does not support unrecognised fields. If a metadata file including an unexpected field is loaded, the load fails.

Expected behavior:

To Be Determined.

We should decide (and capture in an ADR) whether we expect to be able to load metadata which includes additional fields beyond wha the implementation expects.

See also a related issue in the in-toto reference implementation: https://github.com/in-toto/in-toto/issues/186

[trishankatdatadog]

I don't see the danger as long as the client doesn't change its behavior with respect to unrecognized fields, but I might be missing something.

[mnm678]

Allowing the additional fields allows for more flexibility in backwards-compatibility, plus it's implied in the spec. So I'd say we should continue to allow it, and add an ADR to that effect.

[houdini91]

I personally think it is useful to be able to add a custom application level metadata info.
for example it may be a Time server timestamp, apt release file, custom configuration .. etc

[MVrachev]

It feels like we all agree that we should just ignore unrecognized fields.

The next question is what to do with them exactly? What does it mean to ignore them?
While working on the new metadata Key class I just ignored the keyid_hash_algorithms, but then I had problems in the
tests/test_api.sign_verify().
The problem was that what I am verifying is a metadata object without keyid_hash_algorithms and the signature is calculated when it was there.

So, again: what should we do with those unrelated fields? Should we store them in our objects or just drop them?

If we decide to drop them this means that if we open a file with keyid_hash_algorithms and later save it again we would have
changed the content of the original file and the saved file won't contain it anymore.
This seems problematic when we think of an older TUF version that could rely on this field.

[trishankatdatadog]

I personally think it is useful to be able to add a custom application level metadata info.
for example it may be a Time server timestamp, apt release file, custom configuration .. etc

You should be able to attach most of this per-package custom metadata already using the custom targets metadata.

[trishankatdatadog]

The next question is what to do with them exactly? What does it mean to ignore them?

I think we should simply ignore them and not drop anything while reading and writing again.

[joshuagl]

Allowing the additional fields allows for more flexibility in backwards-compatibility, plus it's implied in the spec.

Wow, I'd somehow forgotten this line:

All of the formats described below include the ability to add more attribute-value fields for backwards-compatible format changes. If a backwards incompatible format change is needed, a new filename can be used.

The next question is what to do with them exactly? What does it mean to ignore them?

I think we should simply ignore them and not drop anything while reading and writing again.

Indeed, if we do otherwise we are not supporting the specification behaviour documented above and relied on by some TAPs.

[joshuagl]

Thinking about this a bit more, and looking at TAPs which reference or rely on this behaviour in the specification (TAPs 6, 10, 14, and 15), I think we (at least python-tuf) have three personas of metadata consumers:

1. TUF clients (updater), which only read the metadata (they write/download, but don't generate or modify, metadata files)
2. TUF repositories, which generate metadata – both from scratch, and modifying existing metadata files
3. developer tools, which generate and modify a subset of metadata

In the case of 1, the metadata API should be able to ignore (not drop, not choke on) unrecognised metadata fields. For 3 also, I think it makes sense to be able to modify metadata (i.e. add new signatures) without necessarily having to recognise and understand all of the fields in the metadata (see i.e. TAP 6).
For 2, I think it would be desirable to be able to (if not default to) only write and sign recognised metadata fields.

So, I think we want our metadata API to be able to read/write metadata without losing any unanticipated fields. But perhaps we should make writing unrecognised fields a boolean option (which defaults to off)?

[mnm678]

So, I think we want our metadata API to be able to read/write metadata without losing any unanticipated fields. But perhaps we should make writing unrecognised fields a boolean option (which defaults to off)?

I agree, adding a signature to something unknown may be a bit risky, but this would allow developers or repositories to add any additional metadata fields in a backwards-compatible manner for the updater.