How do queues, retries and idempotency work?

A queue smooths bursts and moves slow work (email, media processing, reconciliation, notifications) off the request path. Because delivery is usually at least once, design for duplicates:

• Idempotent consumers with a durable idempotency key and a unique business-action key
• Bounded retries with exponential backoff and jitter — unlimited retries cause retry storms
• Dead-letter queue plus a replay procedure for poison messages
• Transactional outbox — write the state change and the event in one database transaction so they never diverge
• Backpressure: concurrency limits and caps so a backlog can't take the database down
• Monitor queue age and lag, not just queue length

Payment and payroll consumers must not assume exactly-once processing.