Protocol

Normative. This document defines the wire behavior implemented three times (C++, Rust, TypeScript). Divergence is a bug in the code, never a reinterpretation of the spec. Implementations link back here with // spec: docs/08-protocol.md#<anchor> comments.

Design rules

1. Typed envelope, opaque media payloads. Routing/metadata fields are schema’d; SDP and ICE bodies ride as opaque strings (never modeled — they change with browser versions). (fleet-daemon lesson)
2. Correlate, don’t RPC. Every request carries a unique event_id; responses echo it. Long operations reply accept → feedback* → result, all with the same event_id.
3. Explicit reliability. Every DataChannel declares a reliability class; nothing defaults silently to reliable-ordered. (teleop-car lesson)
4. Versioned from message one. See Versioning.

Transport layers

Layer	Carries
WSS to fjarr-server	signaling envelopes (session setup, ICE), backend-consumer streams
WebRTC media tracks	camera/desktop video (RTP)
WebRTC DataChannels	everything else, per the topology below

Once a session is up, its signaling socket can go quiet for minutes: the heartbeat travels on fjarr:control, not on the socket. Proxies end idle WebSockets (Cloudflare after 100 s; the hosted demo’s operators were disconnected every 2 min 5 s, 2026-10-05), so the server sends a WebSocket ping on every socket every 30 s (Config::ws_keepalive). Peers answer with the protocol’s pong, which browsers and libsoup do by themselves; no envelope changes. A socket from which nothing at all (pong or message) has arrived for two intervals is gone: the server ends it as if it had dropped, so its sessions get peer-gone. Without that, the pings would keep alive a socket whose peer vanished without closing it, which the proxy’s idle limit used to end.

The agent applies the same rule from its side. It sends its own WebSocket ping every 20 s and treats a ping with no pong within 20 s as a dropped socket: it closes it (giving the silent peer libsoup’s 5 s for a close frame, so at most 45 s after the uplink died), ends its sessions and reconnects with the usual backoff. Without that, a robot whose uplink vanished without a FIN or RST (a Wi-Fi drop, a NAT that forgot the flow) believes it is online while the server has long since sent robot.offline, until the kernel abandons the TCP connection some 15 minutes later, and nobody can reach it in between (2026-10-23, a Jetson on the hosted demo).

RTP feedback and repair {#rtp-feedback}

Every video m-line the agent offers carries, and the agent honours:

Feedback	Offered as	The agent’s side
Congestion feedback	a=rtcp-fb:* transport-cc + the transport-wide sequence header extension	the per-peer estimator’s input (docs/23); browsers send it by default
Retransmission	a=rtcp-fb:* nack + an rtx payload per video payload	a lost packet is re-sent on request from a per-peer buffer of one GOP; a receiver waits at most one round trip instead of asking for a keyframe
Keyframe requests	a=rtcp-fb:* nack pli, ccm fir	a keyframe from the tier’s encoder, rate-limited (≥ 1 s, shared by every viewer of the tier); a no-op with a counted log line on a passthrough track (the camera decides)
Forward error correction	not offered	it spends bandwidth on every link to help the lossy ones; revisit with a customer on a link where retransmission’s round trip is too long

A track’s bandwidth-stats shows the effect (below): the estimate the agent holds for this peer and the tier it is actually sending.

Signaling messages {#signaling}

JSON text frames on the WSS connection. Common fields on every message:

{
  "v": 1,
  "type": "…",
  "event_id": "uuidv7",
  "ts": 1789467600123
}

type	Direction	Additional fields	Semantics
hello	client→server	role: "agent"|"operator", auth, agent_info/client_info, proto_versions	authenticate + advertise. auth for operators: {scheme:"grant", jwt}. For agents: {scheme:"dev-token", robot_id, dev_token} (the dev registry, until M5) or {scheme:"device-signature", robot_id, nonce_sig} (enrollment, M5). agent_info: {fjarr, os, arch, capabilities:[names]}
hello-ack	server→client	session_id?, proto_version, turn (urls + ephemeral credential + ttl)	accept; operator gets TURN creds here
session-request	server→agent	session_id, capabilities: [{name, params}], operator (display identity), turn?	grant-verified request to open a session; turn carries the agent’s ephemeral TURN credentials for this session (same minting and TTL as the operator’s hello-ack.turn, docs/10#turn — agents never hold static TURN secrets)
session-accept / session-reject	agent→server→operator	session_id, reject: reason	agent’s answer, relayed to the operator (policy hooks may refuse)
offer	agent→server→operator	session_id, sdp, tracks (manifest)	agent always offers
answer	operator→server→agent	session_id, sdp
ice	both, trickled	session_id, candidate, sdp_mline_index	trickle ICE is REQUIRED; candidate: "" = end of candidates
ice-restart	operator→server→agent	session_id	the operator asks the always-offering agent to re-offer with an ICE restart on the existing session (reconnection); the agent answers with a new offer
session-close	any	session_id, reason, retry? (bool, default false)	orderly teardown; retry: true means the closer expects the operator to open a new session immediately (agent media-plane restart, the ICE-restart fallback below) — the client treats it as a reconnection rung, never as a terminal close
peer-gone	server→other side	session_id, reason	server-side last-will: socket death is announced, never inferred (camera-streamer last-will lesson)
backend-stream	agent↔server	capability, payload (envelope)	backend-consumer envelope transport
session-peers (planned, M5)	server→all parties	session_id, peers: [{operator, role: "owner" | "viewer"}]	multi-operator presence alongside the docs/10 control domains; emitted on every change
error	server→client	code, message, caused_by (the offending message’s event_id)	see error codes

Reconnection {#reconnection}

Both sides reconnect with backoff (base 0.5 s ×2, cap 30 s, ±20 % jitter, reset only after 30 s stable (fleet-daemon backoff, copied verbatim)). The operator’s ladder, cheapest rung first:

1. ICE disconnected: wait a short grace (3 s) — ICE usually recovers on its own.
2. Still disconnected, or ICE failed: send ice-restart over signaling (the WSS is independent of the media path). An agent whose stack supports ICE restart re-offers with new ICE credentials and the same manifest; tracks and consumers stay attached. An agent that cannot (webrtcbin ignores the restart option through 1.28 — the spike) MUST answer session-close{reason:"ice-restart", retry:true} at once, so the operator climbs to rung 3 without waiting out the re-offer timeout (open question #21).
3. No new offer within 10 s, or the signaling socket itself is gone: a new signaling round with the same grant (a fresh hello, new session) — re-fetching the grant first if the server said grant-expired. Consumers keep their subscriptions and demand; the client re-flushes demand on the new session.
4. Attempts exhausted (host-configurable, default 5 rounds): failed, with an explicit retry available.

DataChannel topology {#datachannel-topology}

Channels are created by the agent at session setup, named fjarr:<class>[:<capability>]:

Channel	Class	Ordered	Retransmit	Used for
fjarr:control	control	yes	reliable	envelopes: capability control, telemetry, clipboard metadata, heartbeat
fjarr:realtime	realtime	no	0	pointer motion, joint states — newest-wins data only
fjarr:bulk:<cap>	bulk	yes	reliable	one per bulk-using capability; either blob frames (file chunks, clipboard payloads, snapshot bodies) or a raw byte stream — the capability declares which
fjarr:stream:<cap>	stream	no	0	lossy binary in either direction: framed framing for chunked frames with frame-level newest-wins (point clouds, depth maps, custom sensor data — ADR-0018), raw framing for one self-contained datagram per message (network packets)

Rules:

• KEY_DOWN/KEY_UP, button events → control (loss = stuck key; reliability required). Pointer motion → realtime.
• Bulk channels implement backpressure; control/realtime messages MUST stay ≤ 16 KiB.
• Stream frames under framed framing carry a 12-byte header (u32 frame_seq, u16 chunk_index, u16 chunk_count, u32 payload_len) so receivers reassemble whole frames and drop incomplete or stale ones; a frame larger than sctp.maxMessageSize is chunked, never queued behind a newer frame. Under raw framing there is no header and no reassembly: one binary message is one complete datagram, dropped or delivered whole.
• Session-level messages that belong to no capability use the reserved cap fjarr.core (below).
• Heartbeat: ping/pong envelope on control every 5 s, 3 missed → the peer is considered dead: the operator tears the media path down and climbs the reconnection ladder (its signaling socket death, or a session-close(reason="heartbeat") if the socket is still up, tells the server); the agent ends the session with session-close(reason="heartbeat").
• A bulk or stream channel’s framing is declared in the capability manifest, never guessed by a peer. Bulk: raw (terminal input — every binary message is bytes for the capability, no header) or blob (blob frames: files, clipboard payloads, introspection snapshots — chunks of a named, sized blob that an envelope refers to). Stream: framed (the 12-byte header above, the default) or raw. Control and realtime carry envelopes only.

Session-level messages (fjarr.core) {#fjarr-core}

type	kind	payload	notes
ping	request (operator → agent)	{"t0": ms}	every 5 s while connected; t0 = sender’s clock at send
pong	result (echoes the ping’s event_id)	{"ok": true, "t0", "t1", "t2"}	t1 = agent receive time, t2 = agent send time, agent clock
time-sync	request / result (result type is time-sync; only ping is answered as pong)	same payloads as ping/pong	an explicit on-demand probe (latency harness); heartbeats already keep the estimate fresh
ice-restart	—	—	not an envelope: it is a signaling message, because it must work while the media path is down
take-control	request → result	{"domain": "desktop" | "motion"}	claim a control domain now, from whoever holds it (docs/10). result{ok:true}; error{code:"capability-denied"} for a view_only grant or one without a capability in that domain. The previous holder’s input is released (for motion, the robot stops) and every session in the domain is sent control-state, both before this result is sent
release-control	request → result	{"domain": ...}	give a held domain up at once. Idempotent
control-state	event (agent → operator)	{"domains": {"desktop": {"holder": {"id", "label"} | null, "since"?: ms, "you": bool, "view_only": bool}, "motion": {...}}}	sent at session start and on every change of holder, to every session whose grant has a capability in that domain. domains lists only those domains; since is unix ms and absent when free; you is true for every session of the holding operator; view_only is true when this session can never claim the domain (its grant says so), so the client renders no input surface and no take-control button

t0..t3 are unix milliseconds (an agent using g_get_real_time divides by 1000). Clock offset and RTT follow NTP: with t3 = the operator’s receive time, offset = ((t1 − t0) + (t2 − t3)) / 2, rtt = (t3 − t0) − (t2 − t1). Implementations keep the sample with the smallest RTT over a sliding window (the fleet-daemon lesson: a single skewed sample must not jump the clock); consumers stamp outgoing commands as local + offset.

Track control (every track-owning capability) {#track-control}

select-tracks and bandwidth-stats are served by the agent core for every capability that declares tracks, under that capability’s cap (fjarr.camera/select-tracks, fjarr.desktop/select-tracks, fjarr.test/select-tracks); a capability never implements them. The client sends one request per track-owning capability (docs/21).

type	kind	payload	semantics
select-tracks	request → result	{"tracks": [{"track_id", "enabled": bool, "tier": "active" | "thumbnail", "preference"?: "motion" | "sharpness"}]}	full desired state for the tracks listed (unlisted = unchanged); the core flips valves, applies docs/16 tier params, requests a keyframe on enable, maps preference to the encoder’s degradation preference (motion, the default, keeps the frame rate and lets quality fall with the bitrate; sharpness keeps each frame’s quality and lowers the frame rate instead, docs/23); result{ok:true} once applied. Applied is the core’s desired state: media for a track flows only once this peer’s answer to the offer that carried its m-section has been applied, so an enable that arrives first (it travels on fjarr:control, the answer through the server) is held and takes effect — valve and keyframe — when that answer lands (docs/23). A track_id that is not in this capability’s manifest, or whose m-section this peer’s applied answer rejected (port 0 outside the BUNDLE group), makes the whole request fail — result{ok:false, error:{code:"payload-invalid", message:"unknown track <id>"}} or "track <id> was rejected by this peer's answer" — and nothing is applied
bandwidth-stats	event (agent → operator)	{"interval_ms": 1000, "tracks": [{"track_id", "enabled", "tier", "effective_tier", "estimate_bps", "adaptive", "bitrate_bps", "frames", "dropped", "nacks", "keyframe_requests"}]}	once per second while any of the capability’s tracks is enabled. tier = what the client asked for, effective_tier = what the agent sends (lower while this peer’s link is below the tier’s band — docs/23); estimate_bps = the agent’s estimate of this peer’s available bandwidth for the track; adaptive = false on a passthrough track without a lower stream (nothing the agent can do about congestion); nacks and keyframe_requests = repair requests served in the interval. bitrate_bps = outbound-rtp bytes sent over the interval × 8; frames = encoded frames pushed to this peer in the interval; dropped = frames skipped for this peer in the interval (FrameHub ring overrun + leaky-queue drops). Informational for the UI; the client’s health score uses its own getStats (docs/21)

Codec strings in the manifest are RTP encoding names, uppercase: H264, H265, VP8, VP9, OPUS. mid values are opaque strings; a GStreamer agent produces video0, application1, video2… (the spike).

The envelope {#envelope}

All control/backend messages share one JSON shape:

{
  "v": 1,
  "cap": "fjarr.camera",
  "type": "select-tracks",
  "event_id": "uuidv7",
  "kind": "request",
  "payload": { }
}

kind ∈ request | accept | feedback | result | event (unsolicited). accept/feedback/result echo the request’s event_id. result.payload always carries ok: bool and, on failure, error: {code, message, data?} (data is code-specific, see errors). A request to a capability not attached to the session is answered result{ok:false, error:{code:"capability-denied"}} when the agent has that capability but the grant did not include it, and capability-unknown when the agent has never heard of it (so a client can tell “ask for a wider grant” from “wrong robot”); an envelope with a v other than 1 on a DataChannel is dropped and counted, never answered (the signaling hello already negotiated the major).

Capability payload schemas live in protocol/schemas/ (JSON Schema), versioned with the capability; TS types and C++/Rust validators are generated from them (single source — fleet-daemon schema-drift lesson).

Blob references. A payload field may be a blob reference instead of the value itself: {"blob": "<blob_id>", "len": <bytes>, "type": "<media type>"} (the blob-ref fragment of envelope.schema.json). The bytes travel as blob frames on the capability’s bulk channel. A payload whose serialized form would exceed 8 KiB MUST use a reference — the 16 KiB control ceiling leaves headroom for the envelope around it, and the control channel never waits behind a large body.

Track manifest {#track-manifest}

Sent inside offer.tracks, before any media flows:

[
  {"track_id": "cam-front", "cap": "fjarr.camera", "kind": "video",
   "label": "Front", "codec": "H264", "pt": 96, "mid": "0", "monitor": null},
  {"track_id": "desk-del-dell-u2720q-8xk2n13", "cap": "fjarr.desktop",
   "kind": "video", "label": "Dell U2720Q (HDMI-1)", "codec": "H264",
   "pt": 97, "mid": "1",
   "monitor": {"id": "del-dell-u2720q-8xk2n13", "connector": "HDMI-1",
               "index": 0, "primary": true,
               "x": 0, "y": 0, "w": 1920, "h": 1080, "scale": 1.0,
               "name": "Dell U2720Q"}}
]

monitor.id is the stable identity. It is the monitor’s EDID vendor, model and serial, lowercased, with every run of other characters replaced by one - (del-dell-u2720q-8xk2n13). Desktop track_ids derive from it (desk-<id>), so a monitor keeps its identity across unplug and replug, across sessions, and on whichever connector it is plugged into. cursor: "embedded", when present, says this monitor’s video has the robot’s cursor drawn into it, so a client draws none of its own there; absent, the cursor arrives beside the video (cursor, cursor-position below). Connector names are not identities. A replug of a DisplayPort MST chain renamed DP-4, DP-6 and DP-8 to DP-5, DP-9 and DP-11 on the spike machine (ADR-0006). The current connector travels in connector, for display only.

Two cases are weaker, and the spec says so rather than hiding it. A monitor whose EDID has no serial, or one whose identity collides with another’s in the current set, uses its connector name instead, and keeps it only as long as the connector does. A virtual monitor (created for a headless robot or sized to an operator’s window) gets virtual-<n>, unique within the session and nothing more. index, primary, connector and the x/y placement are informational and change freely; name is the EDID model when known. Never key anything on index or connector.

kind is "video" or "audio" (audio tracks: docs/06 fjarr.audio, and fjarr.desktop’s one audio track, desktop-audio: what the robot’s speakers play, with monitor: null, offered with the desktop’s tracks while the robot has an output, docs/23). track_id is stable across renegotiations. mid is the SDP media identifier of the transceiver carrying the track (the agent knows it at offer time); receivers map incoming RTCTrackEvent.transceiver.mid → track_id directly — no SDP parsing, no payload-type guessing (docs/21). Dashboards MUST label from the manifest, not from SDP order. (camera-streamer manifest lesson)

Renegotiation and manifest updates {#renegotiation}

The track set can change mid-session (monitor hot-plug, a capability adding a track). The agent — which always offers — sends a new offer on the established session:

• tracks is the complete new manifest; unchanged tracks keep their track_id and mid; removed tracks are absent from the manifest — the agent MAY keep their transceiver alive but muted (sendonly, valve closed) for reuse, so receivers key on the manifest, never on transceiver liveness; new tracks get new transceivers. mid values are read from the offer SDP (a=mid) by both sides.
• manifest_version (monotonic u32, new offer field) orders manifests within a session: a new session_id starts a new sequence, and receivers reset their applied version with it. Receivers ignore an offer older than one already applied and re-apply an equal one (an ICE-restart re-offer carries the unchanged manifest and the unchanged version; the SDP is what changed).
• The agent serializes renegotiations: at most one un-answered offer per session; further changes are coalesced into the next offer.
• Media on unchanged tracks MUST continue uninterrupted throughout the renegotiation (docs/16 budget) — a hot-plug on one monitor never blips another.
• ICE restart travels in the offer’s SDP as usual; the same serialization applies.

Input events (fjarr.desktop)

Pointer motion (realtime channel, coalesced client-side to ≤ 60 Hz):

{"cap":"fjarr.desktop","type":"pointer","kind":"event",
 "payload":{"track_id":"desk-0","x":0.7341,"y":0.4288,"seq":58342}}

x/y are normalized [0,1] within that monitor’s track — DPI/scaling agnostic by construction. seq is monotonic per session, across every view a client has of it: one counter for the session, never one per view, so a view mounted later (another monitor in a layout, or a view remounted after the robot stopped sharing) never restarts it. Receivers drop stale. A counter per view made the agent drop every move from a remounted view as stale while its clicks still landed, where the pointer had been left (mini-PC, 2026-10-03). Control-channel input messages (all cap: "fjarr.desktop"):

type	payload	notes
button	{"button": "left" | "middle" | "right" | "back" | "forward", "down": bool, "track_id"?, "x"?, "y"?, "seq"?}	a press carries where it lands, the same normalized point and seq as pointer, and the agent moves the pointer there before pressing; a pointer older than its seq that arrives later is stale. Motion travels on realtime and the press on control, so a press could overtake its own motion and land where the pointer was. A release carries none: it ends where the pointer is
wheel	{"dx": px, "dy": px}	already normalized to pixels by the client
key	{"code": "KeyA", "down": bool}	code = KeyboardEvent.code (physical key), mapped to Linux keycodes agent-side; clients MUST NOT forward browser auto-repeat — the held key repeats natively
key-combo	{"codes": ["ControlLeft", "AltLeft", "Delete"]}	atomic press-and-release for combos the browser cannot capture
text (request → result)	{"text": "åäö"}	composed, IME or pasted text the physical-key path cannot express. The agent types it through the robot’s active keymap: each character becomes the key and level (Shift, AltGr) that produce it there, and keys the session holds are released first. A character that keymap cannot produce fails the whole request before anything is typed, with error{code:"unavailable", data:{untypable:["å"]}}: a robot with a US layout cannot type “å”, because no backend has a Unicode path (mutter 50 and libei 1.5; spike 2026-10-01). result{ok:true} once it is typed
release-all	{}	client-initiated on focus loss; the agent MUST also release everything on session-close. The one input message that is not input in the desktop control domain: it never claims it, so a viewer’s window losing focus cannot take control
clipboard-offer (agent → client, event)	{"offer_id": uuid, "types": ["text/plain"]}	the robot’s clipboard changed: something on the robot copied. types lists what it can be read as, in Fjarr’s names: text/plain is UTF-8 text and image/png a PNG image (both when the robot offers both); an empty list means the clipboard was cleared. Files on the clipboard are references to the robot’s own paths, not content: they wait for fjarr.files (M4). The bytes are not sent: a client that wants them asks with clipboard-read. Not sent for content a session wrote (clipboard-write), and a successful write withdraws the previous offer, so a session that starts later is offered nothing until the robot copies again. Never sent to a view_only session (docs/10)
clipboard-read (request → result)	{"offer_id", "type": "text/plain"}	result{ok:true, blob:{blob, len, type}}, and the bytes follow as blob frames on fjarr:bulk:fjarr.desktop. error{code:"payload-invalid"} for a type the offer did not list or an offer_id that is no longer the robot’s clipboard (it changed since); error{code:"unavailable", data:{reason:"too-large", limit}} above the type’s limit, 1 MiB of text or 8 MiB of PNG (an image’s bytes may arrive before its result, and the pending store holds 8 MiB, below); capability-denied for a view_only grant. Reading claims no control
clipboard-write (request → result)	{"type": "text/plain" | "image/png", "blob": {blob, len, type}}	puts text or a PNG image on the robot’s clipboard (another type is payload-invalid); the request is sent first and its bytes follow as blob frames on fjarr:bulk:fjarr.desktop. result{ok:true} once the robot’s clipboard holds it, so a paste keystroke sent after the result pastes this. It is input: it claims the desktop control domain like a key does (control-held, capability-denied for view_only). At most 1 MiB of text or 8 MiB of PNG (unavailable, too-large)
add-monitor (request → result)	{"width": 320–3840, "height": 240–2160}	a virtual monitor of that size for this session (slice 3.5): an extra screen on the robot’s desktop, gone when the session ends or on remove-monitor. result{ok:true, monitor: id} once it is in the layout; its monitors event and the re-offer with its desk-<id> track go out as for any hot-plug. Other sessions see it like any monitor. It changes the robot’s desktop, so it is input in the desktop control domain (control-held; capability-denied for view_only). error{code:"unavailable"} where the backend has none or after 2 per session; payload-invalid outside the sizes
remove-monitor (request → result)	{"id": "virtual-…"}	removes a virtual monitor this session added; payload-invalid for any other
cursor (agent → client, control class)	{"shape_id": string, "hidden"?: true, "hotspot": {x, y}, "image"?: {"w", "h", "blob": {blob, len, type}}}	the robot’s cursor changed shape, for local-cursor rendering (docs/22). shape_id names the shape by its content, so a client caches images by it; image comes only the first time a session is sent that id, its pixels (straight-alpha RGBA, w×h×4 bytes, at most 384×384) as a blob on fjarr:bulk:fjarr.desktop, type application/x-fjarr-rgba — an envelope is at most 16 KiB, which a 64×64 image in base64 already exceeds. hidden: true means no cursor is shown. Sent to every session at its start (the current shape) and on every change. Control class, because a lost shape would leave the wrong cursor until the next change
cursor-position (agent → client, realtime class)	{"track_id": "desk-…", "x": 0.43, "y": 0.61}	where the robot’s pointer is, normalized in that monitor’s track like pointer; at most 30 per second per session, newest wins. What a viewer who is not moving the pointer draws the cursor at; the operator moving it draws at their own pointer instead (docs/22)
desktop (agent → client, event)	{"state": "running" | "absent"}	whether the robot’s desktop session is there: absent while none is connected (the desktop is logged out, has not started yet, or is restarting, as after GNOME Shell crashed on an unplug, mini-PC 2026-10-04), running once it is. Sent at every session’s start and on each change, by a robot whose fjarr.desktop has a backend; its monitors say what that desktop shows. A client that has heard it knows the robot has a desktop, monitors or not (docs/22)
sharing (agent → client, event)	{"state": "on" | "stopped"}	stopped: the robot’s desktop ended the sharing without being asked. On GNOME, someone at the robot pressed stop on the screen-sharing indicator in the top bar. Every desk-* track goes unavailable (a re-offer without them), desktop input closes, and virtual monitors end with it. It stays stopped until a session sends resume-sharing; the agent never resumes on its own. on once it is resumed. Sent to every session on each change, and at a session’s start while stopped (docs/22)
resume-sharing (request → result)	{}	starts the desktop’s captures and input again after sharing: stopped. result{ok:true} once they are asked for; the tracks come back with a re-offer as each becomes available, like a hot-plug, and sharing: on goes to every session. ok:true too while sharing is on. Any session may send it, view_only included: it restores what every session sees and moves nothing on the robot, so it is not input and never claims the desktop control domain. Logged with the operator
monitors (agent → client, event)	{"monitors": [monitor…], "reason": "hotplug" | "mode-change" | "initial"}	emitted immediately on any change with the full current set, before the renegotiation completes, so UIs can show placeholders/arrangements at once; the offer’s manifest remains the source of truth for tracks

Full client-side semantics: docs/22.

Blob frames {#blob-frames}

A bulk channel declared with blob framing carries binary messages of one shape, whatever the capability — the same chunker, reassembler and backpressure pump serve introspection snapshots (slice 5), clipboard payloads (M3) and file transfer (M4):

offset  size  field
0       1     version (0x01)
1       16    blob_id (UUIDv7, binary)
17      8     byte_offset (u64 BE)
25      8     blob_len (u64 BE — the whole blob)
33      4     payload_len (u32 BE)
37      …     payload (≤ 256 KiB and ≤ sctp maxMessageSize)

Rules:

• A blob is complete when every byte of [0, blob_len) has arrived. Chunks of one blob are sent in offset order (the channel is ordered); chunks of different blobs may interleave. An empty blob is still sent: one chunk with byte_offset 0, blob_len 0 and no payload, so a reference to it completes like any other.
• A chunk whose version is not 1, whose payload_len disagrees with the message length, or whose byte_offset + payload_len exceeds blob_len is dropped and counted; the blob it belongs to is discarded.
• Order between channels. The sender emits the referencing envelope first, then the chunks. Control and bulk are different DataChannels, so receivers MUST accept either arrival order: chunks of a blob no envelope has named yet wait in a bounded pending store (8 MiB or 30 s per channel, oldest evicted); a reference whose blob has not completed 30 s after the envelope arrived fails locally (the caller of the request or the subscriber sees an error; nothing goes back on the wire). An unreferenced blob that completes is kept under the same bound until an envelope claims it.
• Backpressure (below) applies to blob senders like any bulk sender. A capability that produces blobs faster than the channel drains decides what to drop; the core never queues unboundedly (fjarr.introspect: the newest snapshot per pipeline wins, docs/24).
• A blob that must survive reconnect (a file) is resumed by ranges, below; a blob that is a value (a snapshot) is simply sent again by the new session. Receivers reassemble small blobs in memory (16 MiB cap by default) and consume large ones chunk by chunk.

File transfer (fjarr.files) {#file-frames}

Manifest/resume/complete are envelopes on control; the data is one blob per transfer with blob_id = transfer_id: file-offer (name, size, sha256, direction — the offer is the blob reference) → accept → chunks → file-complete(result). Resume: receiver sends file-resume {transfer_id, ranges:[[start,end],…]} after reconnect; sender fills gaps only. Integrity: whole-file SHA-256 verified before result.ok.

Terminal (fjarr.terminal) {#terminal}

A PTY on the robot. Bytes ride fjarr:bulk:fjarr.terminal with raw framing in both directions — every binary message is exactly the bytes for the pty, keystrokes one way and output the other, with no header of ours. Reliable and ordered, because a terminal that loses a character is worse than one that is slow.

One PTY per session at M2. That is why nothing here carries a terminal id: a second concurrent shell would need the channel multiplexed, which is a framing change and therefore a protocol decision to take when something actually needs it, not now.

type	kind	payload	semantics
open	request → result	{"cols": u16, "rows": u16, "term"?: string}	start the pty. result{ok:true} once the shell is running. error{code:"unavailable", message} when the robot has no terminal configured (docs/06) — a deployment choice, not a fault; error{code:"busy"} when a pty is already open on this session. A grant without fjarr.terminal never reaches the capability at all: the core answers capability-denied first (errors), and a client must treat that as “you were not given a shell” rather than as a failure
resize	request → result	{"cols": u16, "rows": u16}	TIOCSWINSZ + SIGWINCH. Sent on every client resize, coalesced by the client
close	request → result	{}	SIGHUP, then reap. Idempotent
exit	event (agent → operator)	{"code"?: int, "signal"?: string}	the shell ended on its own; exactly one of code or signal

The pty dies with the session, always: release_all_input on session end closes it (docs/15), so a dropped connection cannot leave an orphan shell holding the robot’s resources. The capability is input-bearing but in no control domain: every session has its own pty, so terminals run side by side (docs/10).

Talk (fjarr.audio) {#talk}

Talking to a person at the robot (M3.5 slice 3.5.3, docs/06). A session whose grant says fjarr.audio with params.talk: true, on a robot with a speaker configured, gets one audio m-section the agent offers recvonly in its first offer (Opus, 48 kHz). It carries no entry in offer.tracks: it is an uplink, not a track to label. The client answers it sendonly and attaches its microphone with replaceTrack(), with no renegotiation (docs/21). A session without talk is offered none.

What the agent plays is decided by messages, not by whether RTP arrives:

type	kind	payload	semantics
talk-start	request → result	{}	open this session’s gate to the speaker. result{ok:true} once it is open; the client attaches its microphone after it. capability-denied without talk; unavailable with no speaker or no uplink offered; busy while another session talks, with error.data {"holder": {"id", "label"}}. Idempotent for the session that talks
talk-stop	request → result	{}	close the gate. Idempotent. result{ok:true} once closed
talk-level	event (agent → the talking session)	{"rms_db": number, "peak_db": number, "frames": u64}	what reaches the speaker, after the volume cap, every 250 ms while the gate is open: dBFS, and the 20 ms frames played since talk-start. Lets the operator see their voice arrive

One session talks at a time. The gate closes on talk-stop, on session end (every reason, docs/15), and when the uplink m-section is rejected. Audio that arrives with the gate closed is dropped, never played: a client that sends RTP without talk-start is heard by no one. Each opening and closing is an audio-uplink session event for the audit log (docs/09, docs/10).

Network packets (fjarr.net) {#net-packets}

The network tunnel uses fjarr:stream:fjarr.net with raw framing: one binary message is exactly one IP packet, with no header of ours at all. The channel is unordered with no retransmission because the traffic inside carries its own recovery, and an outer retransmission would fight TCP’s and lose on a poor link.

• A packet larger than the negotiated MTU is dropped, never fragmented; the interface’s MTU makes this unreachable in practice and the counter exists to catch misconfiguration.
• Senders bound their queue and drop the tail when the channel’s buffered amount is high. Queueing is forbidden: a queue delivers a burst of stale packets after congestion, which damages the round-trip estimate of every connection inside the tunnel more than the loss would have.
• Both ends drop any packet whose source is not the expected peer, and any whose destination is neither their own tunnel address nor a multicast address (docs/27, ADR-0026). These checks are normative: the source check is what makes one robot unreachable from another, and the multicast exception is what lets DDS discovery work at all.

Control messages for the capability travel on fjarr:control as ordinary envelopes:

type	kind	payload	semantics
open	request → result	{}	brings the link up. result{ok:true, address, peer_address, mtu, policy:{forwarding:false, allow_ports}} — address is the robot’s tunnel address, peer_address the operator’s, and mtu is the interface’s own, which governs whatever the config says. error{code:"unavailable", message} when the tunnel is not enabled or the interface is absent, naming what to create (docs/26); error{code:"busy", data:{holder:{id, label}, since}} when another session holds the link, because the robot’s interface is point-to-point: the data names who holds it and since when (unix ms), so the client can say who to ask (docs/10). A grant without fjarr.net never reaches the capability at all: the core answers capability-denied first (errors)
close	request → result	{}	stops the flow; the interface itself stays, since it is persistent by design. error{code:"unavailable"} when this session has no open link
link-stats	event (agent → operator)	{"interval_ms": 1000, "tx_packets", "rx_packets", "tx_bytes", "rx_bytes", "dropped_no_peer", "dropped_policy", "dropped_queue", "dropped_mtu", "abandoned", "abandoned_error"}	once per second while the link is open; the four drop counters are what a support engineer reads first. abandoned is the fifth and the only one the sender could not have refused: messages usrsctp threw away after the channel accepted them, counted since the link opened, with abandoned_error usrsctp’s reason for the latest (docs/27). A transfer that dies with the four at zero and this one climbing is the transport, not the pump

Backpressure {#backpressure}

Senders pump only while bufferedAmount < HIGH_WATER (4 MiB) and resume on bufferedamountlow (LOW_WATER 1 MiB); GStreamer side mirrors with buffered-amount/on-buffered-amount-low. Unbounded sends are a spec violation, not a style issue.

Errors {#errors}

error.code is a stable string: auth-failed, grant-expired, capability-unknown, capability-denied, session-unknown, robot-offline, rate-limited, payload-invalid, internal, control-held, unavailable, busy. Codes are append-only.

control-held answers an input-bearing request from a session that does not hold that capability’s control domain. Its error.data is {"domain", "holder": {"id", "label"}, "since": ms}, so the client can say who to ask (docs/10). An input request from a view_only grant is answered capability-denied instead: waiting would not help. Other codes may carry error.data too (busy on fjarr.net/open); a client ignores data it does not know. unavailable and busy, used by capabilities since M2, are listed here as of M3.5 (talk-start, above).

Versioning {#versioning}

• v on every signaling message / envelope = protocol major. A peer receiving a higher major replies error(code="payload-invalid") and the connection renegotiates to the common proto_versions from hello.
• Within a major: fields are add-only; unknown fields MUST be ignored; semantics of existing fields never change (add a new type instead).
• Capability payloads version independently via capability semver (docs/05).
• The schemas in protocol/schemas/ are the machine-readable source; this doc explains semantics. CI (M0.5) fails if code and schemas drift.