| title | date | severity | affectedsystems | resolved | |
|---|---|---|---|---|---|
Missing environment variable in router service |
2020-06-06T13:00:00.000Z |
major-outage |
|
true |
The router service failed due to the renaming of an environment variable necessary for the upcoming release of v3.0. This change was inadvertently deployed by our Flux-based continuous deployment.
Automatic updates should have been turned off prior to the release of this change. This has now been done and all images pinned to version v2.32.1.
Within the us-nw-1 cluster we are using Prometheus (deployed as the monitor service) for internal monitoring of services. However, monitor is not yet set up to monitor the router service. We are now using an external site monitoring service (Pingdom) to provide additional monitoring and alerting of https://hub.stenci.la.