Crypto pump drops one-shot messages on transient send failure (prod verification wedge) #60

Closed
opened 2026-08-07 11:00:10 +02:00 by thecrealm · 0 comments
Owner

Root cause of the persistent prod verification stall (#58 follow-through, user-reported 'Element does not get the info that katrix accepted'): SessionCrypto.pump marked a request FAILED (= pending.remove) on the first KatrixException and rethrew. Account-derived requests (keys upload/query) regenerate next round, but one-shot sends — verification ready/start/key/mac, signature uploads — have no regeneration path: a single 429/5xx on a real rate-limited server silently discarded them and wedged the flow, while the local phase already showed Ready. Containerized e2e Synapse has rate limits off, so every e2e stayed green. Fix: failed requests stay pending and retry on the next pump (the drain leg runs one per committed sync batch); a poison request drops after 8 attempts. PumpRetryTest covers it.

Root cause of the persistent prod verification stall (#58 follow-through, user-reported 'Element does not get the info that katrix accepted'): SessionCrypto.pump marked a request FAILED (= pending.remove) on the first KatrixException and rethrew. Account-derived requests (keys upload/query) regenerate next round, but one-shot sends — verification ready/start/key/mac, signature uploads — have no regeneration path: a single 429/5xx on a real rate-limited server silently discarded them and wedged the flow, while the local phase already showed Ready. Containerized e2e Synapse has rate limits off, so every e2e stayed green. Fix: failed requests stay pending and retry on the next pump (the drain leg runs one per committed sync batch); a poison request drops after 8 attempts. PumpRetryTest covers it.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
thecrealm/katrix#60
No description provided.