AGIPD FF Histogram

@ahmedk

This is MR contains features implemented by @karnem and is being moved from https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/309

Notebook is moved out of playground folder and written to be compatible with production notebooks.

changed title from WIP: creating branch after moving files from feat/update_AGIPD_FF to WIP: AGIPD FF Histogram

@jsztuk Before asking for code review I want to ask for review of the interface and functionality:

Following command line give an example, how notebook can be executed:

xfel-calibrate AGIPD FF_HISTS --out-folder  /gpfs/exfel/exp/MID/202030/p900137/scratch/karnem/r0319_0322_0342_v07 --in-folder /gpfs/exfel/exp/MID/202030/p900137/proc/ --karabo-id MID_DET_AGIPD1M-1 --runs 319 322 324 --chunk-size 5 --cells-list 'range(0,100)' 'range(56,155)' 'range(111,202)' --modules 0-16

As input parameters it takes list of runs, for each run range of cells is given. All sequences are considered. Range of modules is given.

An example of the output files including report is here: /gpfs/exfel/exp/MID/202030/p900137/scratch/karnem/r0319_0322_0342_v07. One h5 file per module, which contains all histograms. Apart histograms h5 file contains: cellIds, number of bins and bin range.

Report shows, which cells, files and runs have been processed. It shows summary information (mean, sum) over pixels across memory cells.

Does it make sense to include summary over cells (mean, sum) across pixels?

added 1 commit

• 4c114bc5 - creating branch after moving files from feat/update_AGIPD_FF

Compare with previous version

I had a look at the report and I have a few comments regarding the interface:

• cell_list --> description should be 'list of cells illuminated with X-rays' Is it also possible to give ranges as e.g range(0,202,2)? This kind of scenario (i.e. every nth cell is illuminated) should be also considered
• something went wrong with Module 3, it seems that cellId 0 has very high statistics (at least according to the report)
• Plots 'mean over module' should be modified. It is not a mean signal but mean of a histogram number of entries.., so it gives similar information to mean statistics plot. What we would like to have is the real mean value of the signal It means something like that (but better coded :))

`` rshist = np.reshape(hist,(cells, nBins,512,128))

mean_hist = np.zeros((cells,rshist.shape[2],rshist.shape[3]),dtype=float32)

sd_hist = np.zeros((cells,rshist.shape[2],rshist.shape[3]),dtype=float32)

x = np.linspace(hRange[0], hRange[1], nBins)

mids = 0.5*(x[1:] + x[:-1])

for k in range(0,cells):

for i in range(0,512):

    for j in range(0,128):

        probs = rshist[k,0:nBins-1,i,j] / (np.sum(rshist[k,0:nBins-1,i,j])-1)

        mean_hist[k,i,j] = np.sum(probs * mids)  

        sd_hist[k,i,j] = np.sqrt(np.sum(probs * (mids - mean_hist)**2))

average_hist_std = np.nanstd(mean_hist,axis=(1,2)) average_hist_mean = np.nanmean(mean_hist,axis=(1,2)) `` An example plot is attached

I do not think there is a need to add a summary over cells (mean, sum) across pixels. what would help to validate the data quality for the next step is possibility of having information (fraction of pixels) where histograms:

• are empty (i.e. no entries in the defined bin range)
• mean value of the signal mean_hist is smaller than defined threshold, eg. 30 ADU
• sd_hist is smaller than defined threshold, i.e. noise value 10 ADU

added 1 commit

• 3fbc4c7c - creating branch after moving files from feat/update_AGIPD_FF

Compare with previous version

added 1 commit

• c569ee27 - creating branch after moving files from feat/update_AGIPD_FF

Compare with previous version

added 1 commit

• ed386721 - creating branch after moving files from feat/update_AGIPD_FF

Compare with previous version

This is new version of report, which includes all discussed plots.

AGIPDFF_HISTSCalibration.pdf

Information about detector operation conditions:

• gain setting
• acquisition rate
• number of memory cells

is not included. In order to avoid code duplication it can be saved in the proc file e.g. during processing at correction notebook.

Alternatively it can be extracted from raw data just before sending constant to the DB.

unmarked as a Work In Progress

changed the description

added 1 commit

• 532b5f74 - creating branch after moving files from feat/update_AGIPD_FF

Compare with previous version

@jsztuk @ahmedk Histogramming notebook is updated and ready for review.

Number of processed runs and sequences is added to the output file.

Processing algorithm is now use multiprocessing with shared memory. Pressing time is significantly reduced (see details is the report).

Example of report is attached: AGIPDFF_HISTSCalibration.pdf