Publishing
Every event publishes throughcore/events/publisher.py against a typed
catalog (core/events/catalog.py, 30 event types) — there’s no ad-hoc
redis.xadd() scattered through the codebase. Streams are bounded
(MAXLEN ~10_000) so an extended consumer outage can’t grow a stream
unbounded, at the cost of a real, monitored tradeoff: very high publish
volume during a long outage can trim undelivered events at the tail.
Consumer groups and the PEL
Redis Streams’ consumer-group model tracks delivered-but-unacknowledged messages in a Pending Entries List (PEL). A message is only removed from the PEL on explicit ACK — a consumer that crashes mid-handler leaves its in-flight message there, recoverable.Surviving a rolling deploy
Consumer identities are pod-unique ({hostname}-{pid}). Naively, that
means every rolling deploy — every pod replacement — permanently orphans
that pod’s PEL entries, since no future consumer shares its identity to
reclaim them.
EventBus.claim_stale() fixes this with XAUTOCLAIM (compatible with
both Redis 6.2 and 7 reply shapes): every EventConsumer runs a reclaim
sweep at startup (a fresh pod immediately rescues its rolled-over
predecessor’s PEL entries) and every 60 seconds thereafter (entries
idle more than 5 minutes). A slow-but-alive consumer’s in-flight entry is
never stolen — only genuinely abandoned entries move.
Dead-letter queue
A message that exhausts its retry budget — poison message, permanently failing handler — moves to the DLQ instead of retrying forever or being silently dropped. The payload is preserved; replay is a documented runbook procedure, not a data-loss event.Idempotent consumption
Event consumption is the third layer of MAEL’s idempotency design (after the Redis lock and the unique run-log constraint — see AI Agents): a duplicate delivery of the same event (redelivery after a crash, or anXAUTOCLAIM reclaim racing a
slow-but-still-alive original consumer) cannot cause the handler’s side
effects to apply twice.