AGC design principles

[jirhiker]

1. Use high level language
2. divide into jobs
3. if software fails restart
4. checkpoint good state
5. hardware monitors software
6. Send Telemetry

pychron already follows 1 and 2 pretty strictly

3. if software fails restart. pychron should launch automatically on experiment computers. use information from (4) if available.
4. checkpoint good state. extend the events system to persist experiment state
5. Hardware monitors software. Watchdog is kinda like this. software monitoring software but in this context i think its ok
6. Send Telemetry. Add a telemetry to the watch dog plugin. added more events and context to experiment executor so more telemetry is send to the watchdog.

If the computer/experiment fails and the system passes all checks on relaunch then restart the experiment.