Add very basic version of job unstuck-ing for non-txn jobs that hang …

[bretthoerner]

…in 'running'

This is a v1 implementation so we don't forget to have something. The query checks for jobs that have been running for over 2 minutes and puts them back to available (the action that made them running already added an attempt).

Porting all of our retry time calculation logic into SQL didn't seem worth it, nor did SELECTing all the rows, doing the exact retry-time calculation per row in Rust, and updating them individually. We shouldn't have jobs getting stuck unless pods crash, so this isn't the common path for retries.

[xvello]

Worth adding a Harry-comment about:

• why we don't increment attempt when unlocking rows. My first instinct was to do so, but it's not desired
• what happens if for some reason the previous pod comes back to life and finishes it: I'm assuming that we'll duplicate the output, and record the success one or twice (depending on whether janitor runs between both updates?). Again, it's fine if that's the case for now (the events already have dupes from processing), just would love a quick list of tradeoffs while it's fresh in your head.