Send Kafka notifications when correction is complete

Description

We're experimenting with the migration system sending Kafka notifications, including when migration has completed and a raw run is available on Maxwell ('event': 'migration_complete'). It would be nice to have similar notifications when a corrected run is available, and the calibration webservice is the obvious place to send those.

This is very minimal at the moment. It would be nice to send more information (migration events also have run directory, instrument & cycle), but this isn't easy without changing the sqlite database, which is a bigger change than I wanted to make at the moment. It could be extended to add events when correction starts, and events for characterisation (e.g. dark processing).

It will also need some work to handle multiple kinds of processing running on the same run. We probably want events when each piece of processing completes (with a robust way to identify which is which), and another event when everything that writes to proc/r0123/ has finished. But we shouldn't let the perfect be the enemy of the good, IMO.

How Has This Been Tested?

Run on max-exfl017, re-corrected a CALLAB run while listening for the event. This is the result I got:

{'event': 'correction_complete',
 'proposal': '900113',
 'run': '9998',
 'detector': 'lpd',
 'success': True}

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code follows the code style of this project.

Reviewers

@danilevc @ahmedk

added 1 commit

• 90214234 - Use local copy of config file by default in update_mdc.py

Compare with previous version

added 1 commit

• f11fbd70 - Make update_mdc.py a bit more helpful

Compare with previous version

Luca tells me that if this was available, they'd be interested in using it in their upcoming experiments (14-15 and 28-29 of April). They can live without it, though, so it's not a priority.

added 6 commits

• f11fbd70...477e347e - 2 commits from branch master
• cabeb428 - Send Kafka notifications when correction completes
• 224cd50e - Kafka needs value serialised to bytes, not str
• 39f2a72e - Use local copy of config file by default in update_mdc.py
• 637a42b5 - Make update_mdc.py a bit more helpful

Compare with previous version

Looks cool!
I want to do a release that cherry picks the improvements you've made in previous MRs.
I suppose you'd like to see this in as well?

Is there a way to differentiate between the test and production installations in the future?
I would not want that people down the line have their scripts break when we do testing.

Lastly, could you provide a short example file (eg. the script you've been using) that can be used to check the kafka integration when doing release tests?
I'd like to have:

• a short script;
• a section in the release checklist that lists what we should look for (it can be simply a couple of points under Calibration Services).

Thanks Cyril!

• If you're happy with it, I think it would be good to get this deployed soon, so that Luca's group can try using it (with the understanding that it's in beta, of course).
• The obvious way to distinguish them would be with the topic, which is set in the config file. Consumers subscribe to a specific topic. I've set this to xfel-test-offline-cal to match the xfel-test-r2d2 topic for migration (though when they're both official, I guess we'll drop the 'test'). We could add a suffix like -staging in our test environment.
• Yes, I'll look at adding a script (in the repo, I assume) and a section in the release checklist. :-)

added 1 commit

• 2aec88d4 - Add script to print Kafka events

Compare with previous version

added 1 commit

• 9cbacf37 - Hide traceback after Ctrl-C

Compare with previous version

I've added a bit in the release checklist. Let me know if it needs expanding.

Ok, last comment: what happens when the brokers are unavailable?
Do we keep going?

Good question. Tomorrow, I will experiment with misconfiguring the broker list.

added 1 commit

• 718d64dd - Problems sending Kafka notifications should not prevent status updates

Compare with previous version

hehe :) much better

OK, with the latest change, it should now fail gracefully if it can't send Kafka notifications. I tested this by temporarily configuring the broker to no-such-server.desy.de, which it obviously can't connect to.

I think this should work for any kind of problem with Kafka, but I don't know all the possible problems. We could make a more paranoid design where we put messages in a queue and interact with Kafka on a separate thread. I think that should be unncessary complexity (kafka-python is already using threads internally), but it might be worth keeping in mind if we have problems in the future.

(I'm also now itching to refactor update_job_db and related functions, but I'll restrain myself for now. :-)

resolved all discussions

added 1 commit

• 87b15daa - Fix typo in log message

Compare with previous version

Very nice and educational, LGTM :D