[AGIPD][CORRECT] Apply pulse index selection/validation after common-mode correction

Summary:

This MR consists of moving the part for choosing and applying selected pulse range max-pulses to the data being corrected.

As reported by Johannes from MID this should have better calibration quality for data with known empty cells along each detector row (32 cells) with the available signal.

This MR is a result of discussion at issue: https://git.xfel.eu/gitlab/detectors/calibration_workshop/issues/122

and is connected to issue: https://git.xfel.eu/gitlab/detectors/calibration_workshop/issues/224

Also this MR consists of two related fixes:

1- The call for get_trains_data had wrong order of parameters. 2- Loading pulse indices was affected a lot by how we read files using non-contiguous indices.

Tests:

This feature was tested in two different ways:

1- Using the below xfel-calibrate command line for current MR and current Master and comparing plots and timings.

/home/ahmedk/calibration2/cal2_venv/bin/xfel-calibrate AGIPD CORRECT \
--in-folder /gpfs/exfel/exp/MID/202002/p002718/raw \
--run 205 \
--out-folder /gpfs/exfel/data/scratch/ahmedk/test/mid_agipd_cm \
--karabo-da-control AGIPD1MCTRL00 \
--karabo-id-control MID_EXP_AGIPD1M1 \
--karabo-id MID_DET_AGIPD1M-1 \
--h5path-ctrl /CONTROL/{}/MDL/FPGA_COMP \
--common-mode \
--sequences 0 \
--max-pulses 65 352 2 \
--rel-gain

Test 1 conclusion:

1- Total process time is less by 1000 seconds (factor of 2 speed when selecting non-contiguous pulses.)

Test 2:

1- This is being tested by Johannes at the moment

Report for 1st test:

MR:AGIPDOfflineCorrection.pdf

Master:AGIPDOfflineCorrection.pdf

2- Using the below xfel-calibrate command line for current MR and current Master and validating that the image data at /INSTRUMENT/MID_DET_AGIPD1M-1/DET/{}CH0:xtdf/image/data are the equal.

/home/ahmedk/calibration2/cal2_venv/bin/xfel-calibrate AGIPD CORRECT \
--in-folder /gpfs/exfel/exp/MID/202002/p002718/raw \
--run 205 \
--out-folder /gpfs/exfel/data/scratch/ahmedk/test/mid_agipd \
--karabo-da-control AGIPD1MCTRL00 \
--karabo-id-control MID_EXP_AGIPD1M1 \
--karabo-id MID_DET_AGIPD1M-1 \
--h5path-ctrl /CONTROL/{}/MDL/FPGA_COMP \
--common-mode \
--sequences 0 \
--max-pulses 65 352 2 \
--rel-gain

Reports for 2nd test:

MR:AGIPDOfflineCorrection.pdf

Master:AGIPDOfflineCorrection.pdf

Reviewers:

@danilevc @roscar @hammerd