Redis/Streams lab

Streams Lab: Consumer Crash And PEL Recovery

This lab shows how Redis Streams keep delivered-but-unacknowledged messages in a Pending Entries List, then recovers them with XAUTOCLAIM.

Break modeConsumer reads messages and crashes before acknowledging half of them.

Fix modeRecovery consumer claims idle pending messages and acknowledges them.

InvariantAll produced messages should be processed and PEL should end at zero.

XREADGROUP

crash

XAUTOCLAIM

XACK

Redis Commands

`XGROUP CREATE ... MKSTREAM`

Creates a consumer group. MKSTREAM creates the stream if missing.

`XADD stream * field value`

Appends a message to the stream. * asks Redis to assign the entry id.

`XREADGROUP GROUP g c STREAMS s >`

Reads new messages for a consumer group and assigns them to one consumer.

`XACK stream group id`

Marks a delivered message as processed and removes it from the PEL.

`XPENDING stream group`

Reports how many messages are pending and which consumers own them.

`XAUTOCLAIM stream group consumer min-idle start`

Claims pending messages that have been idle longer than the configured threshold.

Pending Entries List

The PEL is Redis Streams' memory of messages that were delivered to a consumer but not acknowledged. This is what makes streams useful for work queues where crash recovery matters.

State	Meaning	Command
New message	Entry exists in stream but has not been delivered to this group yet.	`XREADGROUP ... >`
Pending	Delivered to a consumer, but not acknowledged.	`XPENDING`
Claimed	Recovery consumer takes ownership after idle timeout.	`XAUTOCLAIM`
Done	Processed and removed from PEL.	`XACK`

Run Targets

The runnable lab is labs/redis/streams, backed by labs/redis/cmd/streams.

`make break`

Consumer crashes halfway. Look for XPENDING total above zero.

`make test`

Recovery consumer uses XAUTOCLAIM. Look for recovered messages and PEL zero.

`make load`

Runs the same recovery pattern with more messages.

Production Notes

Stream processing is at-least-once. A message can be processed more than once after crashes or retries.
Handlers should be idempotent, usually with a durable processed-message marker or business idempotency key.
XAUTOCLAIM needs a sane idle threshold. Too low can steal work from slow but healthy consumers.
Streams keep data until trimmed. Plan retention with XTRIM or maxlen policies.

Code Pointers

Code	Why it matters
`cmd/streams/main.go`	The consumer crash, pending-entry check, `XAUTOCLAIM`, and `XACK` recovery path.
`streams/Makefile`	Targets for crash, recovery, and larger stream runs.
`streams/compose.yaml`	Redis container used by the lab.
`streams/README.md`	Expected output for PEL recovery.

Cache Rate limit Locks Pub/Sub