CI flake: TailEndToEndTest.kill_restart_no_loss_no_duplicates_offline_scrollback 120s timeout (run #138) #56

Closed
opened 2026-08-06 23:31:27 +02:00 by thecrealm · 3 comments
Owner

Build run #138 (b7ef236, roadmap-only commit — cannot be code-caused) failed with TimeoutCancellationException after 120s in the tail sample's Synapse e2e; #136 was green two commits earlier. Same deflake family as #55: find the slow leg (container pull? initial sync await?), don't just bump the timeout.

Build run #138 (b7ef236, roadmap-only commit — cannot be code-caused) failed with TimeoutCancellationException after 120s in the tail sample's Synapse e2e; #136 was green two commits earlier. Same deflake family as #55: find the slow leg (container pull? initial sync await?), don't just bump the timeout.
Author
Owner

Likely root cause identified while fixing #66: the kill/restart starvation signature matches a dead sliding pos. Synapse can reject a resumed connection position with M_UNKNOWN_POS, and SyncLoop treated that as a generic error — backoff, retry the same dead pos, forever — so run2 never surfaced the markers and the 120s/240s timeout fired. 5579d06 (D34 addendum) makes SlidingSyncSource restart the connection on M_UNKNOWN_POS (to-device token kept). Suggest keeping this open until a few CI runs confirm the flake is gone, then closing against 5579d06.

Likely root cause identified while fixing #66: the kill/restart starvation signature matches a dead sliding pos. Synapse can reject a resumed connection position with M_UNKNOWN_POS, and SyncLoop treated that as a generic error — backoff, retry the same dead pos, forever — so run2 never surfaced the markers and the 120s/240s timeout fired. 5579d06 (D34 addendum) makes SlidingSyncSource restart the connection on M_UNKNOWN_POS (to-device token kept). Suggest keeping this open until a few CI runs confirm the flake is gone, then closing against 5579d06.
Author
Owner

First CI run containing the M_UNKNOWN_POS restart (45f6c20, run for task 1018) is green — build + spec-lint, tail kill/restart e2e included. Leaving open per the earlier note until a few more runs confirm the starvation signature is gone.

First CI run containing the M_UNKNOWN_POS restart (45f6c20, run for task 1018) is green — build + spec-lint, tail kill/restart e2e included. Leaving open per the earlier note until a few more runs confirm the starvation signature is gone.
thecrealm added
now
and removed
next
labels 2026-08-10 09:46:26 +02:00
Author
Owner

Closing against 5579d06 (D34 addendum: SlidingSyncSource restarts the connection on M_UNKNOWN_POS, to-device token kept).

Evidence the starvation signature is gone — four consecutive green build runs since the fix, each verifiably executing :samples:katrix-tail:test (task line + job success in the logs; passing test names are not echoed):

  • run 182 (task 1018, 45f6c20) — the first post-fix run, already noted above
  • run 184 (task 1020, 43b9e1e)
  • run 186 (task 1022, b904e2a)
  • run 188 (task 1024, 165b510)

Also re-confirmed the original failure matches the diagnosis: run #138's test task started 19:51:07 and failed at 19:53:13 with TimeoutCancellationException after exactly one starved 120s await — run2 spinning on a dead sliding pos, never surfacing the markers.

Closing against 5579d06 (D34 addendum: SlidingSyncSource restarts the connection on M_UNKNOWN_POS, to-device token kept). Evidence the starvation signature is gone — four consecutive green build runs since the fix, each verifiably executing :samples:katrix-tail:test (task line + job success in the logs; passing test names are not echoed): - run 182 (task 1018, 45f6c20) — the first post-fix run, already noted above - run 184 (task 1020, 43b9e1e) - run 186 (task 1022, b904e2a) - run 188 (task 1024, 165b510) Also re-confirmed the original failure matches the diagnosis: run #138's test task started 19:51:07 and failed at 19:53:13 with TimeoutCancellationException after exactly one starved 120s await — run2 spinning on a dead sliding pos, never surfacing the markers.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
thecrealm/katrix#56
No description provided.