Install internal dependencies from Gitlab package index

Description

Installing iCalibrationDB and XFELDetectorAnalysis (pyDetLib) from Gitlab URLs is quite slow, because it has to clone git repositories, and pip doesn't cache the built packages. Installing from packages on our devpi server should be faster.

This requires a corresponding MR in deployment-tools to use the index: https://git.xfel.eu/calibration/deployment-tools/-/merge_requests/5

This does assume that we'll keep the devpi server up, and at the URL it currently is. The URL is written into requirements.txt files as part of calibration metadata, to be used for reproducibility - though we can do fudges in the reproduction code to replace the URL if we have to. I don't know if we want to take any steps to ensure our devpi server is reliable for the long term.

How Has This Been Tested?

TBD

Types of changes

• Bug fix (non-breaking change which fixes an issue) (sort of)

Reviewers

@roscar @schmidtp

added 1 commit

• df604add - Use devpi to install dependencies on Gitlab CI

Compare with previous version

added 1 commit

• abd3f133 - Fix writing --index-url in requirements.txt file

Compare with previous version

mentioned in merge request !767 (merged)

added 1956 commits

• abd3f133...5c57127a - 1952 commits from branch master
• 1a30e3bd - Rely on devpi to provide iCalibrationDB and pyDetLib
• 9f7d73fd - Record devpi index URL in requirements.txt files for reproducibility
• efc6cf47 - Use devpi to install dependencies on Gitlab CI
• 6732f351 - Fix writing --index-url in requirements.txt file

Compare with previous version

added 1 commit

• 27b22484 - Upgrade pyDetLib to version compatible with newer Numpy

Compare with previous version

added 1 commit

• a238c47f - CalParrot is now on devpi as well

Compare with previous version

added 1 commit

• c764fa44 - Install internal packages from Gitlab registry

Compare with previous version

added 1 commit

• 0dc32226 - Don't embed index URL in requirements.txt files in metadata folder

Compare with previous version

added 1 commit

• c4993901 - Specify index to get packages from for repeating calibration

Compare with previous version

This MR now depends on a PR in the separate env-cache project to allow specifying an index server when recreating environments for xfel-calibrate-repeat:

https://github.com/European-XFEL/env-cache/pull/1

Once we've reviewed this and we're happy, I'll merge that PR, release a new version of env-cache, and update the dependencies here to use the new version. So it's easy to make further changes there if we need to.

marked this merge request as ready

changed title from Install internal dependencies from devpi to Install internal dependencies from Gitlab package index

added 1 commit

• 73bed3ba - Use a released version of env-cache again

Compare with previous version

LGTM

added 137 commits

• 73bed3ba...ebd549e9 - 127 commits from branch master
• 1a29ed75 - Rely on devpi to provide iCalibrationDB and pyDetLib
• 102f2ab4 - Record devpi index URL in requirements.txt files for reproducibility
• e72ac9e0 - Use devpi to install dependencies on Gitlab CI
• 6394d8c8 - Fix writing --index-url in requirements.txt file
• d324a7bb - Upgrade pyDetLib to version compatible with newer Numpy
• 3ea8ea51 - CalParrot is now on devpi as well
• 081c72ad - Install internal packages from Gitlab registry
• 77c54d5a - Don't embed index URL in requirements.txt files in metadata folder
• b2a9e3ac - Specify index to get packages from for repeating calibration
• 6d365e89 - Use a released version of env-cache again

Compare with previous version

merged

mentioned in commit cb3c05c2

changed milestone to %3.16.0