Enforcing locality constraints on persistent rabbit storage

[jameshcorbett]

Users can allocate persistent XFS storage on rabbit nodes. Jobs that come in that want to use the storage will need to be constrained to using the same rack(s) that the persistent storage lives on. However, at the moment, Flux schedules them without considering the locality constraints at all.

@behlendorf remind me are there any constraints on persistent XFS storage in terms of the number of rabbits or nodes that can be used? It seems like it would be problematic if, for instance, a persistent XFS instance was created for three nodes and then a job came through with four nodes that tried to use it.

[behlendorf]

It seems like it would be problematic if, for instance, a persistent XFS instance was created for three nodes and then a job came through with four nodes that tried to use it.

This is the big constraint we identified early on. For this case, we probably need to require the job request the same number of nodes.