Epic: Agent Observability

[nambrot]

Logging

Preview Give feedback

1. Agent debug logs should always be present when configured #3915
   agent +1
   
2. Epic: Agent Logging Improvements #2670
   1 of 7
   agent dx epic relayer tech-debt +5
   
3. indicate a relayer config flag was set incorrectly #3353
   agent dx relayer +3
4. Relayer "Ingesting leaf" log should say the index as well #3222
   agent bug good first issue +3
   
5. RPC rate limiting is not surfaced well by relayer logs #3320
   agent dx relayer +3
   
6. Agent crash dump output should include the entirety of configurable items #3146
   agent dx +2
   
7. Fallback provider errors should indicate the providers responsible for the errors #3408
   agent +1
8. BigQuery logs for relayer message delivery debugging #3546
   agent aw-infra +2
9. https://github.com/hyperlane-xyz/issues/issues/1220
10. Relayer retry API: add the # of messages prioritized in the queue to the http response #4059
    agent good first issue +2
    
11. Relayer API: get the contents of a queue #4080
    agent +1
12. https://github.com/hyperlane-xyz/issues/issues/1316

Metrics & Alerts

Preview Give feedback

1. Use metricAppContexts in the relayer to give more granularity on the contents of the prepare queue #3240
   agent monitoring +2
   
2. Roll out new metricAppContext-aware relayer queue, and update / add dashboards and alerts #3397
   agent aw-infra operations +3
   
3. Ensure all Injective warp routes are configured as metric app contexts in our infra #3415
   agent aw-infra partners +3
   
4. Agents: use configured nativeToken.decimals for balance metrics #3354
   agent good first issue +2
5. Gas usage metrics, refactor metrics logic #3238
   agent +1
   
6. Relayer MerkleTreeProcessorMetrics seemingly aren't registered with the Prometheus Registry #3221
   agent bug good first issue +3
   
7. Version control for grafana alerts #1698
   monitoring operations +2
   
8. Instrument Tokio Tasks with tokio-metrics #3239
   agent monitoring +2
   
9. Agent Flamegraphs #3455
   agent monitoring +2
10. Relayer metrics on app context validators are not resilient to aggregations of ISMs with different validator sets #4030
    agent +1
11. Relayer queue metrics should only be updated when an operation changes to a different stage/task #4068
    agent +1
    
12. Upon startup, validators should write some metadata about themselves to their checkpoint syncer #4226
    agent good first issue +2
    
13. https://github.com/hyperlane-xyz/issues/issues/1390

Community

Preview Give feedback

1. Grafana Dashboard for relayer agents #2155
   agent community monitoring operations validator +5
2. Better documentation around monitoring metrics #3194
   agent docs monitoring validator +4
   
3. https://github.com/hyperlane-xyz/issues/issues/508