Libertem: Test files for datasets

[sk1p]

We need better test coverage for datasets. This could be done by having a small (<1MB?) example file for each file format we support in our repository, and testing all relevant functions on real data.

• [ ] MIB (different bit depths, different partitioning (number of frames per .mib file)) #129
• [x] RAW (different variants, including endian-ness #144 ,can easily be generated)
• [x] HDF5 (can easily be generated)
• [ ] BLO
• [ ] K2IS #63 #109
• [ ] FRMS6 #155
• [ ] MRC #152
• [ ] Norpix Seq #153
• [ ] TIA SER/EMI

For K2IS we should also have some additional files that exhibit the weird corner cases of the format.

[sk1p]

cc @woozey

[sk1p]

We need to make sure to not include the test files in the Python distribution, otherwise we will run into issues, see for example: https://github.com/hyperspy/hyperspy/issues/2048 - I'd put the files onto a dedicated hosting service (maybe bintray?) and download them for testing. That also allows for more realistic test files if we don't have to watch size as much (e.g. small but real scans instead of all 0s).

[sk1p]

There are multiple K2IS files available, for which we have the permission to publish them for testing purposes. They range from 8GB to 177GB. So: good for testing by users, not so good for automated testing in the CI pipeline.

@woozey will record new test data in the next weeks, which will be used for validating our reader. There is a minimum scan size of 16x16, which should be approx. 1.2GB of data in total. This could work with TravisCI where we should have ~9GB of storage available in total. If we don't care about the actual data, we could null those areas in the file and just keep the headers, and then heavily compress the result.

The testing should remove the files after all testcases that require them have run, which is easily doable with pytest fixtures.

As the disk space requirements and run time are not trivial, we should think about only running these tests when they are explicitly requested, and in CI.

[woozey]

@sk1p As far as I remember the acquisition software limits the scan size to 20x20. In any case I will double check it and generate new datasets for you also for troubleshooting reasons.

[sk1p]

We now have a 20x20 test dataset from the K2, thanks to @woozey ! We can convert this to raw binary (even different pixel formats), and set up I/O tests and benchmarks.

We would like to host this on zenodo, as open data, and download them on demand.

[sk1p]

The 20x20 K2IS dataset turned out to be just noise for some reason, so not so useful for testing. We should start for now with the larger 34x35 dataset (Capture52). We would only run the expensive I/O tests if explicitly requested by the user.

How do we get the ball rolling here? @uellue you have some experience with the zenodo publishing interface, do you know what metadata is required / useful?

[sk1p]

As an intermediate solution to this, I propose the following:

• setup a mirror on https://iffgit.fz-juelich.de
• run a gitlab CI runner on moellenstedt
• make our internal test data available on moellenstedt
• manually mirror the github repo to iffgit (the automated mirror feature is only available for paid gitlab licenses)

What this accomplishes:

• it means we can run the DataSet tests
• we can get test coverage results from the DataSet tests

But:

• we don't get a tight integration of GitLab/GitHub, like gitlab CI on github PR (that's only available for paid versions of gitlab)
• we only get results from master, not from PRs

[sk1p]

Hmm, actually, as the core functionality we need is the custom CI runner, which is AFAIK not available in Travis CI. It is available for Azure Pipelines, though: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser#install

That means with Azure Pipelines, we don't have to mirror the repo, just run an agent on moellenstedt, and we don't have the downsides of the gitlab solution. I think I'll try that one first!

[uellue]

@sk1p sounds good! 👍

[sk1p]

just run an agent on moellenstedt, and we don't have the downsides of the gitlab solution. I think I'll try that one first!

As there are security considerations when running all CI pipelines on our infra, I had another idea: we could trigger the dataset tests with a simple github bot command, after doing an initial review of the code, and after merging to master. That way we don't test code unseen, and still have okay-ish test frequency.

[sk1p]

Azure pipelines documentation on this topic: https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#run-pull-request-validation-only-when-authorized-by-your-team

You may not want to automatically build pull requests from unknown users until their changes can be reviewed. You can configure Azure Pipelines to build GitHub pull requests only when authorized by your team.

To enable this, in Azure Pipelines, select the Triggers tab in your pipeline's settings. Then, under Pull request validation, enable Only trigger builds for collaborators' pull request comments and save the pipeline. Now, the pull request validation build will not be triggered automatically. Only repository owners and collaborators with 'Write' permission can trigger the build by commenting on the pull request with /AzurePipelines run or /AzurePipelines run as described above.

So the solution is to add a second pipeline that runs the tests on the on-prem agent, which we can trigger after doing an initial review of the PR. Access to the agent pool needs to be confirmed on the first run, so now only the -data pipeline has access to the on-prem agent.