docs/runbooks/) — this page is a map to them, not a duplicate. Every
production alert’s PagerDuty payload links directly to the relevant one.
By symptom
API is slow or returning errors
API is slow or returning errors
Check
SEOApiP99LatencyHigh/SEOApiP95LatencyElevated and
SEOApiAvailabilityFastBurn/SlowBurn in Grafana’s seo-slo
dashboard first. Runbook: api-latency.md / api-availability.md
(the latter also owns the platform rollback procedure other runbooks
reference).Agent runs failing or timing out
Agent runs failing or timing out
SEOAgentFailureRatioHigh groups by agent_id deliberately — one
broken agent pages once; a platform-wide LLM outage pages as a storm
(check provider status first in that case).
SEOLlmBreakerOpen means a provider’s circuit breaker tripped —
traffic is already failing over automatically; this is a heads-up, not
an emergency. Runbook: agent-failures.md.Workflow stuck / not progressing
Workflow stuck / not progressing
Check
SEOWorkflowFailureRatioHigh and SEOApprovalBacklogGrowing.
If this started right after a deploy, it may be the rolling-deploy PEL
scenario — see Events
— which self-heals via XAUTOCLAIM within the idle threshold.
Runbook: workflow-failures.md / approval-backlog.md.Events backing up / DLQ growing
Events backing up / DLQ growing
SEOEventsDlqGrowth, SEOEventsDlqBacklog, SEOEventsPelDepthHigh,
SEOEventsConsumerStalled. DLQ payloads are preserved — this is
lossless but latent until replayed. Runbook: event-pipeline-dlq.md.LLM spend spiking
LLM spend spiking
SEOLlmSpendElevated (>200/h).
Almost always a retry loop or prompt-growth bug, not organic traffic.
Runbook: llm-cost-spike.md includes the kill-switch procedure.Celery queue silent or backed up
Celery queue silent or backed up
SEOCeleryTaskFailureRatioHigh, SEOCeleryHeartbeatLatencyHigh,
SEOCeleryQueueSilent. Runbook: celery-queue-health.md.Database pool exhaustion or slow queries
Database pool exhaustion or slow queries
SEODbPoolNearExhaustion, SEODbQueryLatencyP95High. Runbook:
db-pool-exhaustion.md.Cache degraded
Cache degraded
SEOCacheHitRatioLow — usually a symptom of the shared Redis instance
under memory pressure (broker + streams + cache + rate limiter share
one instance today). Runbook: cache-degradation.md.Suspected credential stuffing
Suspected credential stuffing
SEOAuthRateLimitSpike — per-IP auth-tier rate limiting, independent
of plan tier. Runbook: credential-stuffing.md.Data loss / need to restore from backup
Data loss / need to restore from backup
Not an alert — you’re here because data is lost or corrupted beyond
in-place repair. Objectives: RPO ≤24h, RTO under 30min. Always restore
onto the project’s own Postgres image, never vanilla
postgres — role
definitions and RLS defaults come from docker/init/postgres-init.sql.
Runbook: backup-restore.md, with an automated test harness
(tests/deploy/backup_restore_test.sh) that exercises the same
procedure.Local development issues
See Installation → Common pitfalls for PgBouncer/prepared-statements,uv vs pip, WSL2/Docker Desktop, and the
pytest coverage-gate-on-partial-runs gotcha.