Fix/requirements into setuppy

Description

• Removes the requirements.txt file
• Move requirements into setup.py
• Reformat setup.py
• Update README to reflect no longer using requirements.txt
• Update CI to use new setup
  • Generates venv for tests each time to ensure runner local env is not used
  • Added --ignore=tests/legacy flag to pytest to ignore legacy test dir
  • Added --cov=cal_tools and --cov=xfel_calibrate to get coverage output for both modules
  • Added ssh key variable to CI, added .setup-ssh-euxfel-gitlab template to CI
• Change internal dependencies to use SSH instead of files (this is now possible as xsoft has a keypair on gitlab)

These changes make the installation of the package a bit more standard, removes the step of having to run two install commands (one for requirements.txt, and one for .), and let us use extras to define additional optional dependencies (e.g. test, dev, etc...).

How Has This Been Tested?

Tested locally by running pip install without extras, with test, with dev, and with test,dev, then ran the pytests to check that everything was still working correctly.

Also updated the CI definition to use the new setup style, so the tests also executed with the pipeline.

Relevant Documents (optional)

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

Reviewers

@danilevc @ahmedk @hammerd

Notes:

For our local dependencies the name of the package doesn't match the name of the repository, so I have to explicitly write out the name, and then use the git ssh url as a target, e.g. iCalibrationDB @ git+ssh://git@git.xfel.eu:10022/detectors/cal_db_interactive.git@2.0.1. @kluyvert does this look correct?

.gitlab-ci.yml

@@ -2,13 +2,6 @@ stages:
   - check
   - test
 
-.setup-ssh-euxfel-gitlab: &setup-ssh-euxfel-gitlab
-  - eval $(ssh-agent -s)
-  - echo "$SSH_PRIVATE_KEY_GITLAB" | tr -d '\r' | ssh-add -
-  - mkdir -p ~/.ssh
-  - chmod 700 ~/.ssh
-  - ssh-keyscan -p 10022 git.xfel.eu > ~/.ssh/known_hosts
-
 checks:
   stage: check
   only: [merge_requests]
@@ -30,7 +23,17 @@ pytest:
   stage: test
   only: [merge_requests]
   before_script:
-    - *setup-ssh-euxfel-gitlab
+    - eval $(ssh-agent -s)
+    - echo "$SSH_PRIVATE_KEY_GITLAB" | tr -d '\r' | ssh-add -
+    #  Our self-hosted runners have persistent home directories, so here we store
+    #  the known_hosts file in the temporary project dir and tell git to use a ssh
+    #  command that uses this hosts file during operations
+    - export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=$CI_PROJECT_DIR/.ssh/known_hosts"
+    - mkdir -p $CI_PROJECT_DIR/.ssh
+    - chmod 700 $CI_PROJECT_DIR/.ssh
+    - ssh-keyscan -p 10022 git.xfel.eu > $CI_PROJECT_DIR/.ssh/known_hosts
+    - ls  $CI_PROJECT_DIR/.ssh
+    - echo $GIT_SSH_COMMAND
   script:
     - python3 -m venv .venv
     - source .venv/bin/activate