Fix/requirements into setuppy

Description

• Removes the requirements.txt file
• Move requirements into setup.py
• Reformat setup.py
• Update README to reflect no longer using requirements.txt
• Update CI to use new setup
  • Generates venv for tests each time to ensure runner local env is not used
  • Added --ignore=tests/legacy flag to pytest to ignore legacy test dir
  • Added --cov=cal_tools and --cov=xfel_calibrate to get coverage output for both modules
  • Added ssh key variable to CI, added .setup-ssh-euxfel-gitlab template to CI
• Change internal dependencies to use SSH instead of files (this is now possible as xsoft has a keypair on gitlab)

These changes make the installation of the package a bit more standard, removes the step of having to run two install commands (one for requirements.txt, and one for .), and let us use extras to define additional optional dependencies (e.g. test, dev, etc...).

How Has This Been Tested?

Tested locally by running pip install without extras, with test, with dev, and with test,dev, then ran the pytests to check that everything was still working correctly.

Also updated the CI definition to use the new setup style, so the tests also executed with the pipeline.

Relevant Documents (optional)

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

Reviewers

@danilevc @ahmedk @hammerd

Notes:

For our local dependencies the name of the package doesn't match the name of the repository, so I have to explicitly write out the name, and then use the git ssh url as a target, e.g. iCalibrationDB @ git+ssh://git@git.xfel.eu:10022/detectors/cal_db_interactive.git@2.0.1. @kluyvert does this look correct?

.gitlab-ci.yml

@@ -2,44 +2,45 @@ stages:
   - check
   - test
 
-.setup-ssh-euxfel-gitlab: &setup-ssh-euxfel-gitlab
-  - eval $(ssh-agent -s)
-  - echo "$SSH_PRIVATE_KEY_GITLAB" | tr -d '\r' | ssh-add -
-  #  Our self-hosted runners have persistent home directories, so here we store
-  #  the known_hosts file in the temporary project dir and tell git to use a ssh
-  #  command that uses this hosts file during operations
-  - export GIT_SSH_COMMAND="ssh UserKnownHostsFile=$CI_PROJECT_DIR/.ssh/known_hosts"
-  - mkdir -p $CI_PROJECT_DIR/.ssh
-  - chmod 700 $CI_PROJECT_DIR/.ssh
-  - ssh-keyscan -p 10022 git.xfel.eu > $CI_PROJECT_DIR/.ssh/known_hosts
+.before_script: &before_script
+  before_script:
+    - eval $(ssh-agent -s)
+    - echo "$SSH_PRIVATE_KEY_GITLAB" | tr -d '\r' | ssh-add -
+    #  Our self-hosted runners have persistent home directories, so here we store
+    #  the known_hosts file in the temporary project dir and tell git to use a ssh
+    #  command that uses this hosts file during operations
+    - export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=$CI_PROJECT_DIR/.ssh/known_hosts"
+    - mkdir -p $CI_PROJECT_DIR/.ssh
+    - chmod 700 $CI_PROJECT_DIR/.ssh
+    - ssh-keyscan -p 10022 git.xfel.eu > $CI_PROJECT_DIR/.ssh/known_hosts
+    - ls  $CI_PROJECT_DIR/.ssh
+    - echo $GIT_SSH_COMMAND
+    - python3 -m venv .venv
+    - source .venv/bin/activate
+    - python3 -m pip install --upgrade pip
 
 checks:
   stage: check
   only: [merge_requests]
   allow_failure: true
+  <<: *before_script
   script:
     - export PATH=/home/gitlab-runner/.local/bin:$PATH
-    # We'd like to run the pre-commit hooks only on files that are being
-    # modified by this merge request, however
-    # `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` is a 'premium' feature according to
-    # GitLab... so this is a workaround for extracting the hash
+    #  We'd like to run the pre-commit hooks only on files that are being
+    #  modified by this merge request, however
+    #  `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` is a 'premium' feature according to
+    #  GitLab... so this is a workaround for extracting the hash
     - export CI_MERGE_REQUEST_TARGET_BRANCH_SHA=$(git ls-remote origin $CI_MERGE_REQUEST_TARGET_BRANCH_NAME | cut -d$'\t' -f1)
-    - export FILES=$(git diff $CI_COMMIT_SHA  $CI_MERGE_REQUEST_TARGET_BRANCH_SHA --name-only | tr '\n' ' ')
+    - export FILES=$(git diff $CI_COMMIT_SHA $CI_MERGE_REQUEST_TARGET_BRANCH_SHA --name-only | tr '\n' ' ')
     - python3 -m pip install ".[test,dev]"
-    - echo "Running pre-commit on diff from  $CI_COMMIT_SHA to $CI_MERGE_REQUEST_TARGET_BRANCH_SHA ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME)"
+    - echo "Running pre-commit on diff from $CI_COMMIT_SHA to $CI_MERGE_REQUEST_TARGET_BRANCH_SHA ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME)"
     #  Pass list of modified files to pre-commit so that it only checks them
     - echo $FILES | xargs pre-commit run --color=always --files
 
 pytest:
   stage: test
   only: [merge_requests]
-  before_script:
-    - *setup-ssh-euxfel-gitlab
-    - ls  $CI_PROJECT_DIR/.ssh
-    - echo $GIT_SSH_COMMAND
+  <<: *before_script
   script:
-    - python3 -m venv .venv
-    - source .venv/bin/activate
-    - python3 -m pip install --upgrade pip
     - python3 -m pip install ".[test]"
     - python3 -m pytest --cov=cal_tools --cov=xfel_calibrate  --ignore=tests/legacy