Dashboards and alert rules reference metric names as strings — nothing stops a metric from being renamed in code while a dashboard panel or alert expr still queries the old name, silently going blank or never firing. scripts/verify/ catches this in CI, on every push, rather than relying on a human noticing during a live incident.

​
check_alert_coverage.py

Cross-checks docker/observability/alerts/seo-alerts.yml, alertmanager.yml, the coverage-map table in alerts/README.md, and docs/runbooks/:
  • both YAML files parse, and every alert has severity/summary/ description/runbook
  • every runbook: annotation resolves to a real file
  • every PromQL expr is paren/brace-balanced
  • every by (...)/without (...) label is actually declared on a metric the expression references
  • every metric in core/observability/metrics.py has a coverage-map row — alerted, or explicitly marked not-alerted with a reason
  • every runbook file is reachable from an alert or the runbooks index

​
check_dashboard_metrics.py

Cross-checks every Grafana panel’s expr against the same metric catalog — the exact class of drift a unit test can’t catch, since it’s about config-referencing-config, not application logic.

​
Run both together

Both run in CI as the verify-observability job on every push — see Observability.

Observability deployment