Add tags of packages to requirements file

@ahmedk @kamile @kluyvert @tmichela @santosh

This MR stays to improve reproducibility.

A tag for each package is added. Tags here are based on current environment.

I additional I add pyDetLib in the requirements. PyDetLib have its own requirements file. In order to trigger that file I included requirements in the setup file of the pyDetLib (see branch https://git.xfel.eu/gitlab/karaboDevices/pyDetLib/tree/feat/update_setup).

This MR should be considered together (and goes after) with https://git.xfel.eu/gitlab/karaboDevices/pyDetLib/merge_requests/76.

Setup file of the pyDetLib requires numpy and Cython

So long as we are using a moderately recent version of pip, we can define Cython and numpy as build dependencies in pyproject.toml:

[build-system]
build-backend = "setuptools.build_meta:__legacy__"
requires = ["setuptools", "wheel", "cython", "numpy"]

Then, when you pip install pyDetLib from source, pip will install those dependencies in a temporary environment to build it with.

We'll need to ensure that it doesn't build against a newer numpy than the version it will be installed with, probably by explicitly selecting an older version of numpy in the list above - like "numpy<1.17" or "numpy==1.16.6".

I think we need to remove these access tokens.

changed this line in version 2 of the diff

If they will be removed installation would require a password. Also used, which installs the package should have enough rights.

added 1 commit

• 678990cb - Add tags of packages to requirements file

Compare with previous version

changed the description

@kluyvert I have another question. Is it fine have have requirements file with strict definition of tags (I mean == instead of >=). This may version conflicts if somebody is using pycalibration and have other packages installed.

What is the general recommendations on this point?

The general recommendation is that libraries should have loosely specified dependencies (e.g. >=2.5 or >=2.5, <3) in setup.py, and applications should have pinned dependencies (like ==2.6.3), either in setup.py or requirements.txt depending on how it's deployed. An application is the set of packages which get deployed and used together.

For us, pyDetLib would clearly be a library. Then we could either say:

• pycalibration is another library, and we have a requirements.txt file somewhere for the offline calibration pipeline which specifies versions of everything (including pycalibration)
• OR pycalibration is the application for offline calibration, and that specifies dependencies with specific versions

More info: https://caremad.io/posts/2013/07/setup-vs-requirement/

(Finally, if we really want to get into this, requirements.txt files are a fairly basic option. Newer tools like Poetry take a set of loosely defined requirements and create a 'lock file' with specific versions of all packages involved. Robert Rosca will tell you more about this with his trademark enthusiasm if you're interested ;-). But it's a bit of a jump to get used to.)

Thank you for clarification. At present requirements.txt file fixes environment and stays for reproducibility. Therefore I would leave it with pinned dependencies. It is not included in the setup.py.

changed the description

unmarked as a Work In Progress

added 1 commit

• 871f8faf - Add tags of packages to requirements file

Compare with previous version

@ahmedk @tmichela @kluyvert

New tag of pyDetLib is created and written in the pycalibration requirements. Could you review this MR again.

Any suggestion related to the token for installation of dependencies from git?

The packages are mostly in alphabetical order with a few exceptions. Let's fix the exceptions, because it's much easier to spot things when they're in order.

As we're fixing versions of the internal packages anyway, why don't we make some packages of them, put them somewhere on Maxwell and install from there? Then we can avoid needing any kind of password or access token. We have a folder /gpfs/exfel/sw/software which the DA team can put things in.

I don't have a permissions for this folder. How to install something there?

Currently, it's only people from the DA team who can put things there. I've just cloned the three repos we're using into /gpfs/exfel/sw/software/git/. So try replacing the git+https lines like this:

git+file:///gpfs/exfel/sw/software/git/nbparameterise@0.3

If this is a useful way to do it, we might be able to ask for other people to have access to the xsoft account which manages the software folder, or agree on an alternative directory to put these in.

added 1 commit

• 12d07639 - Add tags of packages to requirements file

Compare with previous version

added 1 commit

• 5102185c - Add tags of packages to requirements file

Compare with previous version

@kluyvert I updated requirements file and test it. Installation went fine.

The requirement file have in general an alphabetic order. First 3 lines correspond to tools installed from software folder. Last lines correspond to local package cal_tools. Any comments to this?

Yup, I think it's fine to leave those special cases at the top/bottom, with the alphabetic list of names between them.

LGTM

merged

mentioned in commit d0bd25ec