Interface CrystFEL with Karabo and allow automatic parameter tunning with rcrystfel

• Added CrystFELRunner to serve as an interface between Karabo and CrystFEL, and allow for automatic parameter optimization using rcrystfel.
• Added PickyBoi, which selects a set of trains to be sent forth for further processing.
• Added PpuKernel
• CrystFELRunner receives all command line options in a single vector of strings (crystfelArgs.misc).
• CrystFELRunner cleanly frees all resources after stopping CrystFEL.
• If an end-of-stream message is sent (ie: by PickyBoi), CrystFEL is automatically stopped.
• CrystFELRunner collects results of CrystFEL and shows them as properties (more may be added later).
• CrystFELRunner generates temporary geometry files based on its contents if the slot buildGeometry is called with a list of strings.
• PickyBoi now has a property breakStreamAfterEnd, which sends an end-of-stream message to all clients, alerting CrystFELRunners to stop CrystFEL automatically.

Other (by @hammerd, and about which I have no clue) -- this should be checked by @hammerd to avoid disaster:

• BaseArbiterKernel, ShmemTrainMatcher, DetectorAssembler and FrameSelectionArbiter have also been changed, relative to master.
• Many changes in the correction classes.

requested review from @hammerd

added 1 commit

• e3d4a37f - PickyBoi: add source filter, format

Compare with previous version

added 2 commits

• 91232763 - Filter data by source in PickyBoi.
• 4db44ada - Merged.

Compare with previous version

added 1 commit

• 2e29b821 - Using get instead of unsafe_get

Compare with previous version

Before going into actual review, would it make sense to target David's original crystfel-adapter branch here for a smailler diff?

I mentioned to David privately that this can be changed to a Draft, if this is not ok too. I just wanted to keep track of what is going on.

I wasn't aware of another branch, but I am not in a hurry. For me, this MR is more a matter of bookkeeping.

I was just thinking to suggest that someone review PickyBoi first. Then we merge that into master* (maybe after cherry-picking tweaks to it from this branch) and can focus on the CrystfelRunner which is the main contribution here.

*Or PickyBoi goes into a different package entirely because it's useful outside of calng. I think Philipp suggested that may be relevant. It's also not entirely obvious whether the CrystfelRunner needs to be part of core calng (I just make devices in the package I'm editing anyway)...

Yes, after posting I realized there's no need to have this in calng package. It's not required in almost all installations, and where it's used calng proper is actually not needed. In Karabo, large device packages are also somewhat discouraged.

I suggested the same for the PickyBoi given its much more general use, like the train matcher.

Sure thing, now CrystFEL integration and PickyBoi are their own repositories. I've just moved the current version of each from this branch as their respective initial commits. I would suggest opening a "cleanup" MR on each to stage reviews before tagging for production.

Having moved them out, this branch now mostly contains a few fixes here and there. Can either point to master or the train picker MR (which is now just adding the arbiter kernel). Can also close and apply the small fixes separately; will definitely squash either way.

I just propose you then change the description, because what I wrote before does not apply anymore.

changed target branch from master to crystfel-adapter

changed target branch from crystfel-adapter to master

added 1 commit

• 4ed8c643 - Move CrystfelRunner to crystfel-integration

Compare with previous version

added 1 commit

• fe4589be - Move PickyBoi to own repository

Compare with previous version

changed target branch from master to train-picker-device

changed target branch from train-picker-device to master

mentioned in merge request !47 (closed)

changed target branch from master to train-picker-device

merged

mentioned in commit 8e1508ca

mentioned in commit 3944f163