Bids-specification: Need for checksums

[oesteban]

In order to ensure the integrity of data, the most widely used technique has traditionally been checksumming.

Although some related proposal is being built under #368 (see the shared doc there), I think checksums should not be considered part of provenance as their significance is much wider.

I would like to gauge the interest of @bids-standard/raw and @bids-standard/bep_leads to integrate ways of checking the integrity of BIDS datasets, either as a whole or granularly.

If the feedback to this proposal is sufficiently positive, I'd like to lead a BEP with the following:

1. A methodology to calculate checksums of a BIDS dataset, and it's implementation within the BIDS-Validator and PyBIDS.
2. Checksums generation should be unequivocal and defined at all BIDS levels:
   2.1. Content file + metadata (i.e., each _bold.nii.gz + _bold.json pair in the dataset).
   2.2. Data "families" (i.e., all runs of the same task + metadata)
   2.3. Modality (i.e., everything under anat/ + corresponding metadata)
   2.4. Session
   2.5. Subject
   2.6. Dataset
   
   
   
   • Important note: since metadata can be in an outer scope, metadata should be aggregated before checksumming following the inheritance principles of BIDS.
   
   
3. Define rules to write these checksums in the BIDS structure.
   
   
   
   • Whether there will be a special file or maybe a new "Checksum" key that should not be included during checksumming.
   
   
   • Whether it is possible to define metadata fields that will not be checksummed (probably not a good idea other than the checksum field itself).
   
   
   • Define checksums as OPTIONAL in all instances.
   
   

Beyond integrity checks (e.g., to ensure that we are using exactly the same version of data as in a previous experiment, if the versioning was corrupted at some point by mistake), this extension (more precisely: the unique/standard methodology to calculate checksums) would be extremely useful to workflow developers to be able to calculate whether a workflow should be re-run or not.

cc/ @satra @yarikoptic as their experience with Nipype and Datalad could be really useful here.

[robertoostenveld]

Checksums are valuable to check integrity following data transfer, but for storage the underlying file- or object-based storage systems have already appropriate methods to ensure integrity. E.g. ZFS checksums or iRODS checksums or S3 checksums. Or plain parity checks in RAID.

Therefore I consider checksumming to be _implicitly_ part of the storage system and therefore only needs to be _explicitly_ considered as part of the data transfer mechanism, which means that it might not have to be part of BIDS, but rather of the transfer mechanism (e.g. like it works for bittorrent).

Checking integrity after transfer can also be done using a SHA or MD5 manifest file for which tools exist like this that are by default available on every linux/macOS (and probably Windows) system. Adding the MANIFEST file to the .gitignore is easy. I don't see the reason to reinvent the wheel and specify something BIDS specific; that would also put an additional burden on BIDS tooling and discussions on "who" should check (e.g. pybids, bids-validator, or the BIDS apps or user software).

For the Donders repository we use SHA256 manifest files (which can be downloaded for each collection and are also included in the collection itself) as documented here.

[effigies]

I agree with @robertoostenveld. This is solved many ways with a lot of different protocols and storage tools, and I'm not sure I see an advantage to pushing any particular one.

It might nevertheless be useful to have a catalog of practice for how various strategies fit with BIDS. But that feels somewhat different from a BEP or a specification.

[oesteban]

I agree with @robertoostenveld. This is solved many ways with a lot of different protocols and storage tools, and I'm not sure I see an advantage to pushing any particular one.

I disagree, the problem I stated above is not solved. In all of the argued solutions, the message that is transmitted (or stored on some hardware) and checksum are files (as a stream of bits). In this proposal, the message are BIDS structures, which at the most basic level will generally be 2 files (a data file and a metadata file assuming there is no higher-level metadata file corresponding to the data file). In other words, the message goes beyond just a stream of bytes (fortunately, not as far beyond as not to be able to reuse existing technology).

In addition to that, these checksums are not intended (only) for checking data integrity. They would be most useful to quickly check data changes:

• reproducibility (i.e., to be able to check whether we are running exactly the same dataset without the need of version control infrastructure). If you write a paper, it would be great to be able to leave a "BIDS checksum" (or manifest if you like that name better) written. And, since that would be part of the spec, no one would not know how to calculate it. Same stands for sub-parts of the dataset (i.e. you reuse only the anatomical part of a dataset).
• economy: to avoid running the same workflow on the same data twice (unless that is the actual intent).

Checksums are valuable to check integrity following data transfer, but for storage the underlying file- or object-based storage systems have already appropriate methods to ensure integrity. E.g. ZFS checksums or iRODS checksums or S3 checksums. Or plain parity checks in RAID.

Sure, I'm not proposing to reinvent the wheel. And yes, I've used SHA and MD5 (and even knew the manpage of md5sum long before writing this).

Checking integrity after transfer can also be done using a SHA or MD5 manifest file for which tools exist like this that are by default available on every linux/macOS (and probably Windows) system. Adding the MANIFEST file to the .gitignore is easy. I don't see the reason to reinvent the wheel and specify something BIDS specific; that would also put an additional burden on BIDS tooling and discussions on "who" should check (e.g. pybids, bids-validator, or the BIDS apps or user software).

• MANIFEST files do not solve the problem stated above (for workflows to have granularity deciding whether input data changed) because all the tools will keep reimplementing the same problem internally (i.e., checking those MANIFESTS that affect the bit of compute you are deciding on, for instance, all the MANIFESTs of one subject). Additionally, to use md5sum to checksum one given subject of the BIDS structure, one would need to tar up that subject's folder, plus all the json files at different levels that will affect this particular subject. Yes, I agree this is doable but honestly impractical for BIDS users and error-prone. In other words - if this is not included in BIDS, we will need to make our own implementation (because we are not going to write a new implementation for every of our workflows). Chances are that: 1) people do not reinvent the wheel and reuse our implementation -that would be, in fact, an standardization but out of the reach of the BIDS community-; 2) people reinvent the wheel because they are not aware of this solution existed (bc. it is not part of BIDS) or because they have a better implementation (which would be a better standard). In both scenarios, the transparency of everything is much lower than a BIDS-supported initiative and the risk of ending up with many different and partial implementations (e.g. YAML, or an implementation that only cares about MRI datasets) is high.
• No one "should" check - but making it clear how checksums of BIDS subsets are to be computed will make it certainly easier to BIDS-Validator, pyBIDS or any other tool to calculate them.

It might nevertheless be useful to have a catalog of practice for how various strategies fit with BIDS. But that feels somewhat different from a BEP or a specification.

BIDS is no more than a meta-code (in the information theory sense). Therefore, it is sensible that it provides with checksumming protocols. As I said above, this proposal should try to reuse as much as we can from existing methods and libraries (i.e. MD5/SHA), but still, indicating how the hash of any given file AND metadata should be calculated would be great and included in the tree for tools to be able to access it when needed. Even more so for, e.g., a subject.

Finally, this proposal does not intend to replace existing mechanisms to generate MANIFESTS or ensure data integrity after transfers. I agree the existing tools are better for those particular tasks.

Hopefully, I've managed to clarify the intent of this proposal. If the general opinion is still along the lines above, I'll try to address this on my own and I'd be happy to generalize it to the BIDS spec if someone finds it useful.

EDIT: As a final note, I'd like to add that following the same argumentation, BIDS wouldn't have existed in the first place, as DICOM/NIFTI, filesystems, etc. already existed before.

[satra]

i think it's always good to version something. but that is an implementation detail that's different from a protocol/specification. one could say that any bids dataset should contain a checksum as a separate file during transmission that ensures cryptographic authenticity for the dataset. this is something openneuro can add to a dataset, and any user can compute/store for a dataset.

in terms of existing data structures for computing strong cryptographic hashes, the one i like quite a bit is a merkle tree (used by git, ipfs, etc.,.).

so i think the idea of transmitting a merkle tree (or the root checksum - can be used just for verification) with the dataset is a good idea, but i don't think the spec needs to have this in any specific way. one of the reasons the prov BEP is being created though is to provide such and other metadata about the objects. perhaps this checksum can go there.

ps. off-topic - cryptographic hash is another reason to avoid any symbolic links as part of bids specification. you would then have to define how symbolic links are stored/transmitted.

[yarikoptic]

Well, that is why we use git/git-annex and develop DataLad -- to guarantee data integrity (in addition to all other wonderful features ;)). It goes beyond relying on file system integrity (data could get garbled/truncated in transfer), and provides consistency checking as part of the process. Hey -- checksumming in the underlying principle of the stupid content tracker (i.e. git), and that is what we love it for ;)

And BIDS datasets are "dynamic" -- they do change, even if not so frequently. They get extended, curated, enhanced, etc. So, adding checksuming on top would result in at least partial re-implementation of what git/git-annex/datalad provides "natively".

So, with my DataLad hat, I am afraid that BIDS is the level where integrity checking should be prescribed. It should be provided by outside (possibly recommended somewhere within BIDS) tools.

[oesteban]

cryptographic hash is another reason to avoid any symbolic links as part of bids specification. you would then have to define how symbolic links are stored/transmitted.

Unless you prescribe how the hash of a symbolic link should be calculated - then it is a problem of compatibility between filesystems which you already had in first place.

And BIDS datasets are "dynamic" -- they do change, even if not so frequently. They get extended, curated, enhanced, etc. So, adding checksuming on top would result in at least partial re-implementation of what git/git-annex/datalad provides "natively".

I do not agree: the stupid content tracker/git-annex/datalad can track changes on the content of these files but not on the logical structure of the dataset. Any change is seen as a new logical structure. So, if I just change one metadata entry affecting one run of just one subject, any merkle tree/git/datalad etc. will say the whole dataset has changed.

If one wants to use datalad to make sure nothing changed, then one needs to check for changes within this subject and any other existing level of metadata higher up in the hierarchy. Obviously, this is doable but cumbersome. But if BIDS told me - look: this is how you checksum that subset, I can keep that record and check it later on rerun.

Anyways - it seems there is not a minimal mass of interest in this - I'll go ahead on my own.

[satra]

So, if I just change one metadata entry affecting one run of just one subject, any merkle tree/git/datalad etc. will say the whole dataset has changed.

and

But if BIDS told me - look: this is how you checksum that subset, I can keep that record and check it later on rerun.

that's exactly how git diff works - based on the merkle tree. and how datalad tracks changes with datalad run

[oesteban]

Sure, and both are like that because their purpose is general - not BIDS specific. You still need to parse and interpret the git diff to find the semantics (in BIDS terms) of the changes.

[satra]

You still need to parse and interpret the git diff to find the semantics (in BIDS terms) of the changes.

ah but that's a different question and involves a much more complicated scenario. i had initiated this at the ohbm hackathon in vancouver in a project called "same same but different". what a delta is between two items (metadata + blob) is non-trivial, and may involve multiple checksums for a single entity. as a trivial example, imagine in a nifti file you just change the sform. the diff should hopefully identify that the data are the same, but now it's mapping has changed. in a single file nifti, this would require some crafted property to annotate this.

thus, a checksum by itself won't cut it. it would simply assess at a bit level. if you really want to get to interpretive checksums we can have a brainstorming session :)

[oesteban]

hehehe I don't want to get at that fine level of interpretability of checksums. A fast answer to "has this subject changed?" is the core of the use case.

By fast I don't mean that you need to run things on a git-annex compatible system - or even install that. I just mean that, with a relatively clear specification the user can build their tool. If then the BIDS-Validator and/or pybids also implement that, well, much better.

[satra]

A fast answer to "has this subject changed?" is the core of the use case.

in this case i really do think a merkle tree can help as it can identify at any node of a tree whether there is a difference. so at any level of bids (and the levels are currently well defined), you can ask what objects have changed.

you may enjoy these:
https://media.consensys.net/ever-wonder-how-merkle-trees-work-c2f8b7100ed3
https://medium.com/textileio/whats-really-happening-when-you-add-a-file-to-ipfs-ae3b8b5e4b0f

[yarikoptic]

A fast answer to "has this subject changed?" is the core of the use case.

once again - git diff sub-XXXX is the answer. What I really mean is that with whatever you come up with would be something like git (but then why to re-implement it and start to worry about efficiency/scalability/caching/...?). One of the wonderful "git parable" style guides into git would be Matthew's https://matthew-brett.github.io/pydagogue/foundation.html if you haven't read it.

[oesteban]

• If a dataset-level metadata file changed (e.g., /task-rest.json)
  changed, git diff sub-xxxx would come out clean although it has changed in
  a BIDS sense.
• This checksum protocol would tell you other potential use cases
  (basically any entity).

And again, I'm not saying it cannot be done with existing tools. Similarly,
a query tool like pybids could have been done based on DICOM.

On Mon, Jan 6, 2020, 18:29 Yaroslav Halchenko [e-mail]
wrote:

A fast answer to "has this subject changed?" is the core of the use case.

once again - git diff sub-XXXX is the answer. What I really mean is that
with whatever you come up with would be something like git (but then why to
re-implement it and start to worry about
efficiency/scalability/caching/...?). One of the wonderful "git parable"
style guides into git would be Matthew's
https://matthew-brett.github.io/pydagogue/foundation.html if you haven't
read it.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/bids-standard/bids-specification/issues/388?email_source=notifications&email_token=AAESDRVFT4ESIIDLWKVXPO3Q4PSGZA5CNFSM4KDLR3O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIHPCAA#issuecomment-571404544,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAESDRRBBMB53HHWBAKRLZDQ4PSGZANCNFSM4KDLR3OQ
.