Use / load balance with multiple GPUs where applicable
The "new" SASE1 nodes (except the bonus node) each have two sparkly A6000 GPUs. By default, CuPy just selects the first one, but it of course has options to select which GPU to use*. Ideally, we'd use this to distribute the load better. The question is: how does a correction device know which GPU to choose? Does the manager tell all SPB devices to use 0 and all FXE devices to use 1? Or does the manager keep track of number of devices per device server and tells each correction device their local index so they can do ex. modulo themselves?
*It might be tricky for one device to use multiple GPUs at a time, but that's not relevant yet.
Edited by David Hammer