thecrealm/katrix
1
0
Fork 0
Code Issues 23 Pull requests Projects Releases Packages Wiki Activity Actions

Report upstream: continuwuity sliding sync stamps initial without honoring the required_state wildcard #71

New issue
Open
opened 2026-08-07 19:42:10 +02:00 by thecrealm · 3 comments
thecrealm commented 2026-08-07 19:42:10 +02:00
Owner
Copy link

Field finding from the #66 thread (D34 addendum 3): continuwuity 26.7.2 (prod, centoria.de) answers MSC4186 sliding sync with initial:true room payloads whose required_state omits state the [["",""]] wildcard should deliver (observed: m.room.encryption absent for an encrypted DM; Synapse 1.157.2 container e2e delivers it — SlidingStateDeliveryEndToEndTest). katrix now defends itself (transmit probe + explicit required_state entry), but the server behavior deviates from MSC4186 and hurts every client trusting initial payloads. Task: minimize a repro against a continuwuity container and file it upstream at forgejo.ellis.link/continuwuation/continuwuity; consider adding a continuwuity leg to the e2e matrix afterwards.

Field finding from the #66 thread (D34 addendum 3): continuwuity 26.7.2 (prod, centoria.de) answers MSC4186 sliding sync with initial:true room payloads whose required_state omits state the [["*","*"]] wildcard should deliver (observed: m.room.encryption absent for an encrypted DM; Synapse 1.157.2 container e2e delivers it — SlidingStateDeliveryEndToEndTest). katrix now defends itself (transmit probe + explicit required_state entry), but the server behavior deviates from MSC4186 and hurts every client trusting initial payloads. Task: minimize a repro against a continuwuity container and file it upstream at forgejo.ellis.link/continuwuation/continuwuity; consider adding a continuwuity leg to the e2e matrix afterwards.
thecrealm added
now
 and removed
next
 labels 2026-08-08 00:07:55 +02:00
thecrealm commented 2026-08-08 00:14:34 +02:00
Author
Owner
Copy link

Repro minimized against a pristine forgejo.ellis.link/continuwuation/continuwuity:latest container — which reports 26.7.2 (59c2649), same as prod. Probe matrix (each probe is a fresh sliding connection, so always the initial payload; room is an encrypted private chat the syncing user just joined):

required_state initial delivered types
[["*","*"]] true (empty)
[["m.room.encryption",""]] true m.room.encryption
[["*","*"],["m.room.encryption",""]] true m.room.encryption
[["m.room.member","*"]] true m.room.member

So it is sharper than the prod observation: the type-level wildcard delivers nothing at all — not merely missing m.room.encryption. The state_key wildcard on a concrete type works; explicit entries work (which confirms the D34-addendum-3 belt: our explicit entry beside the wildcard is exactly what saves us on this server).

Cause, confirmed in source (src/api/client/sync/v5.rs, collect_required_state, current main): the loop handles state_key == "*" by enumerating keys for a concrete type, but an event type of "*" is treated as the literal type string — room_state_get(room_id, "*", …) matches nothing, and the payload still goes out stamped initial: true.

Context worth knowing before filing: upstream issue 1661 ($ME/$LAZY not resolved) was closed as "not a Continuwuity issue" because those sentinels were removed in the current MSC4186 revision. The * wildcard argument is different — continuwuity accepts the tuple request format and expands the state_key wildcard, so dropping only the type wildcard is inconsistent with its own format handling, and Synapse 1.157.2 delivers full state for the identical request (SlidingStateDeliveryEndToEndTest).

Filing is blocked on a human: no auth for forgejo.ellis.link from this machine, and the maintainers there explicitly asked that AI not be used for communication on their tracker (1661 thread). Draft below — please review, put it in your own words where you see fit, and file it under your account.

Draft title: sliding sync: ["*","*"] type wildcard in required_state delivers no state — initial: true payloads arrive with empty required_state

Draft body:

Reproduced on a pristine container from forgejo.ellis.link/continuwuation/continuwuity:latest (reports 26.7.2, 59c2649).

Steps: create an encrypted private room (initial_state m.room.encryption), second user joins, then that user's first request to /_matrix/client/unstable/org.matrix.simplified_msc3575/sync with "required_state": [["*","*"]]. The room arrives with initial: true and an empty required_state. Requesting [["m.room.encryption",""]] explicitly delivers the event, and [["m.room.member","*"]] delivers all member events — so the tuple format and the state_key wildcard are honored; only the type-level * is not.

Cause appears to be collect_required_state in src/api/client/sync/v5.rs: state_key == "*" enumerates keys for a concrete type, but a type of "*" is looked up as the literal event type and matches nothing.

Impact: a client that subscribes with the wildcard and trusts an initial: true payload to carry the room's state silently sees no state at all. In our SDK's case the missing m.room.encryption made an encrypted room indistinguishable from an unencrypted one (a plaintext-into-encrypted-room hazard) until we added a per-room server confirmation. Synapse 1.157.2 answers the identical request with the full state set.

Repro script (docker + curl + python3): [attach continuwuity-repro.sh]

Repro script follows in the next comment.

Repro minimized against a pristine `forgejo.ellis.link/continuwuation/continuwuity:latest` container — which reports **26.7.2 (59c2649)**, same as prod. Probe matrix (each probe is a fresh sliding connection, so always the initial payload; room is an encrypted private chat the syncing user just joined): | required_state | initial | delivered types | |---|---|---| | `[["*","*"]]` | true | **(empty)** | | `[["m.room.encryption",""]]` | true | m.room.encryption | | `[["*","*"],["m.room.encryption",""]]` | true | m.room.encryption | | `[["m.room.member","*"]]` | true | m.room.member | So it is sharper than the prod observation: the **type-level wildcard delivers nothing at all** — not merely missing m.room.encryption. The state_key wildcard on a concrete type works; explicit entries work (which confirms the D34-addendum-3 belt: our explicit entry beside the wildcard is exactly what saves us on this server). Cause, confirmed in source (`src/api/client/sync/v5.rs`, `collect_required_state`, current main): the loop handles `state_key == "*"` by enumerating keys for a concrete type, but an event **type** of `"*"` is treated as the literal type string — `room_state_get(room_id, "*", …)` matches nothing, and the payload still goes out stamped `initial: true`. Context worth knowing before filing: upstream issue 1661 ($ME/$LAZY not resolved) was closed as "not a Continuwuity issue" because those sentinels were removed in the current MSC4186 revision. The `*` wildcard argument is different — continuwuity accepts the tuple request format and expands the state_key wildcard, so dropping only the type wildcard is inconsistent with its own format handling, and Synapse 1.157.2 delivers full state for the identical request (SlidingStateDeliveryEndToEndTest). **Filing is blocked on a human**: no auth for forgejo.ellis.link from this machine, and the maintainers there explicitly asked that AI not be used for communication on their tracker (1661 thread). Draft below — please review, put it in your own words where you see fit, and file it under your account. --- **Draft title:** sliding sync: `["*","*"]` type wildcard in `required_state` delivers no state — `initial: true` payloads arrive with empty `required_state` **Draft body:** Reproduced on a pristine container from `forgejo.ellis.link/continuwuation/continuwuity:latest` (reports 26.7.2, 59c2649). Steps: create an encrypted private room (initial_state m.room.encryption), second user joins, then that user's first request to `/_matrix/client/unstable/org.matrix.simplified_msc3575/sync` with `"required_state": [["*","*"]]`. The room arrives with `initial: true` and an empty `required_state`. Requesting `[["m.room.encryption",""]]` explicitly delivers the event, and `[["m.room.member","*"]]` delivers all member events — so the tuple format and the state_key wildcard are honored; only the type-level `*` is not. Cause appears to be `collect_required_state` in `src/api/client/sync/v5.rs`: `state_key == "*"` enumerates keys for a concrete type, but a type of `"*"` is looked up as the literal event type and matches nothing. Impact: a client that subscribes with the wildcard and trusts an `initial: true` payload to carry the room's state silently sees no state at all. In our SDK's case the missing `m.room.encryption` made an encrypted room indistinguishable from an unencrypted one (a plaintext-into-encrypted-room hazard) until we added a per-room server confirmation. Synapse 1.157.2 answers the identical request with the full state set. Repro script (docker + curl + python3): [attach continuwuity-repro.sh] --- Repro script follows in the next comment.
thecrealm commented 2026-08-08 00:14:43 +02:00
Author
Owner
Copy link

The self-contained repro script (docker + curl + python3; throwaway container, registration-token dance included):

#!/usr/bin/env bash
# Repro: continuwuity MSC4186 sliding sync ignores the ["*","*"] type
# wildcard in required_state — initial:true room payloads arrive with an
# EMPTY required_state, while explicit type entries (and state_key
# wildcards on a concrete type) deliver fine.
#
# Needs: docker, curl, python3. Starts a throwaway continuwuity on :6167.
set -euo pipefail

IMAGE="${IMAGE:-forgejo.ellis.link/continuwuation/continuwuity:latest}"
NAME=continuwuity-repro-$$
BASE="http://127.0.0.1:6167"

docker run -d --rm --name "$NAME" -p 6167:6167 \
  -e CONTINUWUITY_SERVER_NAME=repro.test \
  -e CONTINUWUITY_ADDRESS=0.0.0.0 \
  -e CONTINUWUITY_PORT=6167 \
  -e CONTINUWUITY_DATABASE_PATH=/tmp/db \
  -e CONTINUWUITY_ALLOW_REGISTRATION=true \
  -e CONTINUWUITY_REGISTRATION_TOKEN=repro-token \
  "$IMAGE" >/dev/null
trap 'docker stop "$NAME" >/dev/null 2>&1 || true' EXIT

echo -n "waiting for continuwuity"
for i in $(seq 1 60); do
  curl -fsS "$BASE/_matrix/client/versions" >/dev/null 2>&1 && break
  echo -n .; sleep 1
done
echo

# continuwuity mints a one-shot token for the FIRST account and only then
# honors the configured token — read it from the startup log
FIRST_TOKEN=$(docker logs "$NAME" 2>&1 | sed 's/\x1b\[[0-9;]*m//g' |
  sed -n 's/.*using the registration token \([A-Za-z0-9]*\) .*/\1/p' | head -1)

jsonget() { python3 -c "import sys,json;print(json.load(sys.stdin)$1)"; }
urlenc() { python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1],safe=''))" "$1"; }

reg() { # username, registration token -> access token
  local session
  # the UIA challenge is a 401 with a JSON body — no curl -f here
  session=$(curl -sS -X POST "$BASE/_matrix/client/v3/register" \
    -H 'Content-Type: application/json' \
    -d "{\"username\":\"$1\",\"password\":\"pass-$1-123\"}" | jsonget "['session']")
  curl -sS -X POST "$BASE/_matrix/client/v3/register" \
    -H 'Content-Type: application/json' \
    -d "{\"username\":\"$1\",\"password\":\"pass-$1-123\",\"auth\":{\"type\":\"m.login.registration_token\",\"token\":\"$2\",\"session\":\"$session\"}}" |
    jsonget "['access_token']"
}
ALICE=$(reg alice "$FIRST_TOKEN")
BOB=$(reg bob "repro-token")

ROOM=$(curl -fsS -X POST "$BASE/_matrix/client/v3/createRoom" \
  -H "Authorization: Bearer $ALICE" -H 'Content-Type: application/json' \
  -d '{"preset":"private_chat","invite":["@bob:repro.test"],
       "initial_state":[{"type":"m.room.encryption","state_key":"",
                         "content":{"algorithm":"m.megolm.v1.aes-sha2"}}]}' | jsonget "['room_id']")
curl -fsS -X POST "$BASE/_matrix/client/v3/join/$(urlenc "$ROOM")" \
  -H "Authorization: Bearer $BOB" -H 'Content-Type: application/json' -d '{}' >/dev/null
curl -fsS -X PUT "$BASE/_matrix/client/v3/rooms/$(urlenc "$ROOM")/send/m.room.message/t1" \
  -H "Authorization: Bearer $ALICE" -H 'Content-Type: application/json' \
  -d '{"msgtype":"m.text","body":"hello"}' >/dev/null

probe() { # label, required_state json -> prints delivered types
  curl -fsS -X POST "$BASE/_matrix/client/unstable/org.matrix.simplified_msc3575/sync?timeout=1000" \
    -H "Authorization: Bearer $BOB" -H 'Content-Type: application/json' \
    -d "{\"lists\":{\"all\":{\"ranges\":[[0,10]],\"required_state\":$2,\"timeline_limit\":5}}}" |
  python3 -c "
import sys, json
r = json.load(sys.stdin)
for rid, room in r.get('rooms', {}).items():
    types = sorted(set(e.get('type') for e in room.get('required_state', [])))
    print('$1 | initial:', room.get('initial'), '| delivered types:', types)
"
}
# each probe is a FRESH connection (no pos) -> always the initial payload
probe 'wildcard          ' '[["*","*"]]'
probe 'explicit          ' '[["m.room.encryption",""]]'
probe 'wildcard+explicit ' '[["*","*"],["m.room.encryption",""]]'
probe 'statekey-wildcard ' '[["m.room.member","*"]]'
The self-contained repro script (docker + curl + python3; throwaway container, registration-token dance included): ```bash #!/usr/bin/env bash # Repro: continuwuity MSC4186 sliding sync ignores the ["*","*"] type # wildcard in required_state — initial:true room payloads arrive with an # EMPTY required_state, while explicit type entries (and state_key # wildcards on a concrete type) deliver fine. # # Needs: docker, curl, python3. Starts a throwaway continuwuity on :6167. set -euo pipefail IMAGE="${IMAGE:-forgejo.ellis.link/continuwuation/continuwuity:latest}" NAME=continuwuity-repro-$$ BASE="http://127.0.0.1:6167" docker run -d --rm --name "$NAME" -p 6167:6167 \ -e CONTINUWUITY_SERVER_NAME=repro.test \ -e CONTINUWUITY_ADDRESS=0.0.0.0 \ -e CONTINUWUITY_PORT=6167 \ -e CONTINUWUITY_DATABASE_PATH=/tmp/db \ -e CONTINUWUITY_ALLOW_REGISTRATION=true \ -e CONTINUWUITY_REGISTRATION_TOKEN=repro-token \ "$IMAGE" >/dev/null trap 'docker stop "$NAME" >/dev/null 2>&1 || true' EXIT echo -n "waiting for continuwuity" for i in $(seq 1 60); do curl -fsS "$BASE/_matrix/client/versions" >/dev/null 2>&1 && break echo -n .; sleep 1 done echo # continuwuity mints a one-shot token for the FIRST account and only then # honors the configured token — read it from the startup log FIRST_TOKEN=$(docker logs "$NAME" 2>&1 | sed 's/\x1b\[[0-9;]*m//g' | sed -n 's/.*using the registration token \([A-Za-z0-9]*\) .*/\1/p' | head -1) jsonget() { python3 -c "import sys,json;print(json.load(sys.stdin)$1)"; } urlenc() { python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1],safe=''))" "$1"; } reg() { # username, registration token -> access token local session # the UIA challenge is a 401 with a JSON body — no curl -f here session=$(curl -sS -X POST "$BASE/_matrix/client/v3/register" \ -H 'Content-Type: application/json' \ -d "{\"username\":\"$1\",\"password\":\"pass-$1-123\"}" | jsonget "['session']") curl -sS -X POST "$BASE/_matrix/client/v3/register" \ -H 'Content-Type: application/json' \ -d "{\"username\":\"$1\",\"password\":\"pass-$1-123\",\"auth\":{\"type\":\"m.login.registration_token\",\"token\":\"$2\",\"session\":\"$session\"}}" | jsonget "['access_token']" } ALICE=$(reg alice "$FIRST_TOKEN") BOB=$(reg bob "repro-token") ROOM=$(curl -fsS -X POST "$BASE/_matrix/client/v3/createRoom" \ -H "Authorization: Bearer $ALICE" -H 'Content-Type: application/json' \ -d '{"preset":"private_chat","invite":["@bob:repro.test"], "initial_state":[{"type":"m.room.encryption","state_key":"", "content":{"algorithm":"m.megolm.v1.aes-sha2"}}]}' | jsonget "['room_id']") curl -fsS -X POST "$BASE/_matrix/client/v3/join/$(urlenc "$ROOM")" \ -H "Authorization: Bearer $BOB" -H 'Content-Type: application/json' -d '{}' >/dev/null curl -fsS -X PUT "$BASE/_matrix/client/v3/rooms/$(urlenc "$ROOM")/send/m.room.message/t1" \ -H "Authorization: Bearer $ALICE" -H 'Content-Type: application/json' \ -d '{"msgtype":"m.text","body":"hello"}' >/dev/null probe() { # label, required_state json -> prints delivered types curl -fsS -X POST "$BASE/_matrix/client/unstable/org.matrix.simplified_msc3575/sync?timeout=1000" \ -H "Authorization: Bearer $BOB" -H 'Content-Type: application/json' \ -d "{\"lists\":{\"all\":{\"ranges\":[[0,10]],\"required_state\":$2,\"timeline_limit\":5}}}" | python3 -c " import sys, json r = json.load(sys.stdin) for rid, room in r.get('rooms', {}).items(): types = sorted(set(e.get('type') for e in room.get('required_state', []))) print('$1 | initial:', room.get('initial'), '| delivered types:', types) " } # each probe is a FRESH connection (no pos) -> always the initial payload probe 'wildcard ' '[["*","*"]]' probe 'explicit ' '[["m.room.encryption",""]]' probe 'wildcard+explicit ' '[["*","*"],["m.room.encryption",""]]' probe 'statekey-wildcard ' '[["m.room.member","*"]]' ```
thecrealm added
blocked
 and removed
now
 labels 2026-08-08 00:14:43 +02:00
thecrealm commented 2026-08-08 00:35:12 +02:00
Author
Owner
Copy link

Format clarification (asked in review): the current MSC4186 revision no longer uses the tuple format at all — required_state is a typed object (RequiredStateRequest: include/exclude lists of {type?, state_key?} elements plus lazy_members), and wildcarding is expressed by omitting a field ({} matches everything). The [["*","*"]] tuple array with "*" sentinels is the earlier MSC4186/MSC3575 wire shape — the one Synapse serves on the org.matrix.simplified_msc3575 unstable endpoint, and the one rust-sdk and katrix send.

Why this strengthens the report rather than weakening it: continuwuity is also on the OLD shape. Its ruma request model deserializes required_state as (type, state_key) tuples — the current revision's object format would not even parse — and collect_required_state implements the old format's "*" sentinel for state_key while treating a "*" type as a literal event type. So the 1661-style response ("not in the current revision") does not transfer: measured against the current revision the endpoint's whole request shape is the old one; measured against the revision they actually implement, the type wildcard is half-implemented. Either way, answering initial: true with silently empty state is the defect. Worth folding this paragraph into the issue when filing.

Format clarification (asked in review): the current MSC4186 revision no longer uses the tuple format at all — `required_state` is a typed object (`RequiredStateRequest`: `include`/`exclude` lists of `{type?, state_key?}` elements plus `lazy_members`), and wildcarding is expressed by *omitting* a field (`{}` matches everything). The `[["*","*"]]` tuple array with `"*"` sentinels is the earlier MSC4186/MSC3575 wire shape — the one Synapse serves on the `org.matrix.simplified_msc3575` unstable endpoint, and the one rust-sdk and katrix send. Why this strengthens the report rather than weakening it: continuwuity is also on the OLD shape. Its ruma request model deserializes `required_state` as `(type, state_key)` tuples — the current revision's object format would not even parse — and `collect_required_state` implements the old format's `"*"` sentinel for state_key while treating a `"*"` type as a literal event type. So the 1661-style response ("not in the current revision") does not transfer: measured against the current revision the endpoint's whole request shape is the old one; measured against the revision they actually implement, the type wildcard is half-implemented. Either way, answering `initial: true` with silently empty state is the defect. Worth folding this paragraph into the issue when filing.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
spec-v1
Labels
Clear labels   
M0

   
M1

   
M2

   
M3

   
M4

   
M5

   
M6

   
blocked

   
next

   
now

   
priority
Jump the queue: work before other next items in the same milestone

No labels
M0
M1
M2
M3
M4
M5
M6
blocked
next
now
priority
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
thecrealm/katrix#71
No description provided.