Feat/reproducibility

@karnem, @amunnich, @samartse, @qingtian, and FYI: @fangohr

This adds basic consistency testing for the correction notebooks of AGIPD and LPD, with reports also building into the documentation, see e.g:

https://in.xfel.eu/readthedocs/docs/european-xfel-offline-calibration/en/feat-reproducibility/test_results.html

and

https://in.xfel.eu/readthedocs/docs/european-xfel-offline-calibration/en/feat-reproducibility/testing.html

for a summary of how things work.

Note that in the documentation example, I've included quite a few failure cases, which are not part of this commit into the artefacts directory, these are removed for the latest commit.

Next steps after this is review and merged:

• add tests for Jungfrau and FastCCD correction notebooks
• add tests for characterisation notebooks. We should likely think about mock calibration database handling there, which resolves requests to always the same data, and provides verification for submissions to it.

Setting to WIP until the two update MRs are merged

marked as a Work In Progress

changed the description

added 1 commit

• bef738fb - Remove outdated artifacts

Compare with previous version

added 1 commit

• d9763efe - Correct a few typos

Compare with previous version

I'll take to ITDM to get better, user-independent locations for this.

Done. Data, together with calibration file are stored in the proposal p700001.

added 1 commit

• 8cea8b1b - Add karabo data self validation to tests

Compare with previous version

I really wonder if it was necessary to exchange the word to 'artefact', because "artifact" is widely used in IT technical literature.

Agreed. But since both are in use, I'd refrain from changing it back now, unless you deem it contextually wrong.

Thanks for review!

added 1 commit

• 0ab0f7cf - Fix spelling and grammar

Compare with previous version

changed target branch from upd/lpd to master

@karnem could you give this a review?

Few python packages have been missing: xmlrunner gitpython karabo_data

added 1 commit

• 83ca91c1 - Add missing requirements

Compare with previous version

I just tried on a fresh Karabo-2.3 install and did not have any problems. Could you try in that setup with the requirements.txt I've updated in the last commit.

Python package sklearn is also missing and required.