Agent Core Architecture

Status: review — the slice-3 design, written before implementation (docs/13), the agent-side counterpart of docs/21. It turns the one-line idioms of docs/02 and the camera-streamer lessons into the structures slice 3 implements, and fills in the two types docs/09 left as forward declarations (SessionContext, BackendContext). Wire behaviour stays normative in docs/08; the webrtcbin spike supplies the element-level facts this design relies on.

Goals

Capabilities never see sockets, SDP, or GStreamer negotiation. They provide sources and handle envelopes; the core owns every session, pipeline and channel. A third-party capability is written against SessionContext alone.
One loop, one owner. Every mutation of core state happens on the core’s GLib main loop. Streaming threads touch exactly one structure (the FrameHub) through one lock. Late callbacks from torn-down sessions are no-ops by construction (generation counters), not by luck.
Produce once, fan out. N operators cost one encoder per track and tier; session churn never touches capture.
Recover in place, then escalate. A single session failing never affects another; the media plane can be rebuilt without dropping the signaling connection; only repeated failure escalates to a supervised process restart.
Every wire rule in docs/08 has one owner in the code — the offer builder, the renegotiation queue, the DC router, the heartbeat — with a // spec: backlink, so the C++ never re-interprets the protocol.
Testable without a robot or a browser. Unit tests on a real GLib loop; integration tests against the real fjarr-server with a C++ operator simulator that speaks the operator side of docs/08 and injects the docs/15 faults.

Non-goals for slice 3: any real capability (camera is slice 4), adaptive bitrate (slice 6), the desktop backends (M2), per-device enrollment (M5), store-and-forward (M7).

Process model

The agent is one process with two separately restartable subsystems, not two processes yet (ADR-0019):

Plane	Owns	Restartable without the other?
Control plane	signaling client, capability registry, session bookkeeping, backend bus, config, audit	it is the process; a crash here is a process restart
Media plane	the GStreamer producer pipelines, FrameHub, every per-session consumer pipeline and `webrtcbin`	yes: on an unrecoverable pipeline error the media plane is torn down and rebuilt; sessions are closed with `session-close(reason="media-restart")`, the signaling socket stays up, the robot stays online

Process separation (a fjarr-mediad child) is the documented upgrade path if driver hangs turn out to survive an in-process rebuild; the seam is the media-plane interface below, so nothing in the capability API changes.

Supervision contract — systemd is required on shipped robots (ADR-0019 addendum): the package ships a Type=notify unit with Restart=on-failure and WatchdogSec=30; embedders reach the same through Agent::supervision(); containers without systemd run with the watchdog off and say so at startup:

Exit code	Meaning	Supervisor action
0	clean stop (`stop()`, SIGTERM)	none
1	configuration/startup error (bad config, missing elements — the doctor’s job at runtime)	do not restart in a loop; log loudly
2	“restart me”: the recovery ladder is exhausted (docs/15 “agent SIGKILL mid-session” and “media plane hang” rows)	restart with the unit’s backoff

The daemon writes READY=1 once its capabilities are configured (the tunnel attached, docs/27) and the core loop runs — before the first hello-ack, since 2026-09-28 (ADR-0019, second addendum) — and STATUS= carries the connection (connecting to <url>, online, offline: <reason>), so systemctl status still tells the truth about an unreachable robot without the unit sitting in activating until the start timeout kills it. WATCHDOG=1 comes from the core loop every WatchdogSec/3; a wedged core loop is therefore killed by systemd, which is the only defense against a deadlock in our own code — which is why the packaged agent does not run without it.

Threading model

 core loop (GMainContext, one thread)      streaming threads (GStreamer)
 ┌──────────────────────────────────┐      ┌────────────────────────────┐
 │ signaling client (libsoup async) │      │ producer: capture→encode   │
 │ session state machines           │      │   → appsink ──┐            │
 │ DC router → capabilities         │      └───────────────┼────────────┘
 │ offer builder / renegotiation    │                      ▼
 │ heartbeat, watchdogs, stats      │           FrameHub (one mutex, ring per track)
 │ post_to_owner() sink             │                      │
 └──────────────▲───────────────────┘      ┌───────────────┼────────────┐
                │ g_idle_add_full           │ consumer (per session/track)│
        worker pool (GThreadPool)          │ appsrc ← pull ─┘ → valve →  │
        capability long work only          │ payloader → webrtcbin       │
                                           └────────────────────────────┘

Rules, each of which the camera streamer learned the hard way:

All core state is loop-affine. Session, PeerConnection, ChannelRouter, the registry, the config: read and written only from the core loop. There are no locks in the control plane.
post_to_owner(fn) is the only way any other thread (GStreamer bus and signal callbacks, streaming pads, worker pool, libsoup) reaches core state: it schedules fn on the core loop with g_idle_add_full (G_PRIORITY_DEFAULT) and carries a generation token; the sink drops the call if the token’s session generation has moved on.
Generation counters. Every Session has a monotonically increasing generation bumped on every teardown. Every callback context registered with GStreamer (g_signal_connect_data user data, pad-probe data) is a small heap object { weak session ref, generation } freed by GDestroyNotify; the callback compares generations before touching anything. This is what makes a disconnect storm safe.
Streaming threads own nothing. An appsink new-sample callback does one thing: push the GstSample into the FrameHub under its mutex. An appsrc need-data callback pulls from the ring under the same mutex. No session state is visible from there.
Capabilities run on the core loop (session_attached, on_message, …) and must return quickly; long work goes to SessionContext::worker() (a GThreadPool), whose completion is posted back with post_to_owner. A capability that blocks the loop trips the systemd watchdog — that is the contract, documented in docs/05.
Bus messages are delivered on the core loop (gst_bus_add_watch on the core context); per-pipeline error handling lives in the media plane.

Object model

Agent (public, pImpl)
 └─ Core
     ├─ Config                       parsed TOML + env overrides (below)
     ├─ CoreLoop                     GMainContext/GMainLoop, post_to_owner sink, worker pool
     ├─ SignalingClient              libsoup-3 WebSocket, hello/backoff, message dispatch
     ├─ CapabilityRegistry           name → Capability, manifests, config validation, dependencies
     ├─ ModuleLoader                 dlopen of in-tree optional modules (desktop backends, ADR-0021;
     │                                 slice 2c: versioned entry SYMBOL so an old module is not found
     │                                 rather than misread, and every failure carries what to install)
     ├─ MediaPlane
     │   ├─ FrameHub                 track_id/tier → ring of encoded GstSamples, subscribers
     │   ├─ SourceRegistry            type name → VideoSource factory (built-in + customer-registered)
     │   ├─ Producer (per output/tier) VideoSource bin → tee → EncoderAdapter → appsink → FrameHub
     │   └─ ConsumerPipeline (per session) appsrc/valve/payloader per track + one webrtcbin
     ├─ SessionManager               session_id → Session; control domains (docs/10)
     │   └─ Session
     │       ├─ generation, state, operator, granted capabilities
     │       ├─ PeerConnection       webrtcbin wrapper: offer builder, renegotiation queue,
     │       │                       ICE, DTLS/connection state, get-stats sampler
     │       ├─ ChannelRouter        DataChannels by class, envelope parsing, fjarr.core,
     │       │                       dispatch to capabilities, ChannelSender impls
     │       ├─ TrackSet             per-session tracks (manifest), demand (select-tracks)
     │       ├─ Heartbeat            answers ping; liveness timer; negotiation watchdog
     │       └─ SessionContextImpl   the capability-facing handle
     └─ BackendBus                   backend-stream envelopes ↔ capabilities (BackendContext)

Ownership is strictly top-down; nothing below holds an owning reference upward. Session objects are owned by the SessionManager and referenced elsewhere only through (weak_ptr, generation) pairs.

Signaling client

SignalingClient wraps a libsoup-3 SoupWebsocketConnection (ADR-0017) on the core context:

Hello. {role:"agent", auth, agent_info:{fjarr, os, arch, capabilities:[names]}, proto_versions:[1]}. Until enrollment lands (M5), auth is the dev scheme the Rust DevSharedTokenRegistry verifies: {scheme:"dev-token", robot_id, dev_token} (the token from FJARR_DEV_DEVICE_TOKEN — the same variable name on the server); the M5 scheme device-signature (Ed25519 over a server nonce, docs/10) slots into the same field. The credential never appears in logs.
TURN. The agent never holds TURN secrets: every session-request carries that session’s ephemeral credentials (turn, docs/08), minted by the server exactly like the operator’s, and the wrapper passes them to that session’s webrtcbin via add-turn-server.
Backoff on socket loss: docs/08 numbers (0.5 s ×2, cap 30 s, ±20 %, reset after 30 s stable), a pure Backoff class shared with tests.
Socket loss ends every session. The server announces peer-gone to the operators and forgets the sessions (slice 1); the agent therefore tears every session down locally with DetachReason::PeerGone and release_all_input. Operators climb their ladder and come back through a fresh session-request. Session resume across a signaling blip is open question #20 — it needs a protocol addition and is deliberately not in slice 3.
Dispatch. Every inbound message is validated structurally (the same rules as the TS guards; unknown types and unknown fields ignored, a higher v answered with error(payload-invalid) per docs/08) and routed by session_id to a Session, or to the SessionManager for session-request, or to the BackendBus for backend-stream.
Outbound goes through one queue on the core loop; nothing else calls soup_websocket_connection_send_text.

Agent-side session state machine

States: requested → building → offered → connected → closing → closed, with renegotiating as a flag on connected, not a state — media keeps flowing through it.

From	Event	To	Action
—	`session-request` (server-verified grant)	requested	lease check (docs/10 ownership); capability names re-checked against local config (docs/10); reject with `session-reject(reason)` on any failure
requested	attach	building	`session_attached` on every granted capability (they declare their per-session tracks); DataChannels created (`fjarr:control`, `fjarr:realtime`, `fjarr:bulk:<cap>` per declaration, `fjarr:stream:<cap>` when a capability declares it); consumer pipeline built; caps gate armed
building	all enabled tracks have fixed caps, or no media tracks	offered	`session-accept`, then `offer{sdp, tracks(manifest with mid), manifest_version:1}`; negotiation watchdog (15 s) armed with milestones
offered	`answer`	offered	set remote description; queued remote ICE applied; trickle continues
offered	DTLS connected and control DC open	connected	watchdog cleared; heartbeat liveness armed; `bandwidth-stats` sampler (1 s) started; `SessionEvent{started}` to the embedder
connected	`<cap>/select-tracks` (docs/08#track-control, served by the core for every track-owning capability)	connected	valves + tier + keyframe request — held for a track whose m-section is not answered yet, applied when the answer is; `result{ok}`; unknown `track_id`, or one this peer’s answer rejected → `payload-invalid`, nothing applied
connected	capability `update_tracks` / hot-plug	connected (renegotiating)	coalesce into the renegotiation queue: one un-answered offer at a time, `manifest_version++`, unchanged tracks keep `mid` and keep flowing (docs/08#renegotiation)
connected	`ice-restart` from the operator	closing → (operator reopens)	on every `webrtcbin` release to date (1.28 included): `session-close{reason:"ice-restart", retry:true}` — the operator opens a new session at once; on a stack with ICE restart: a new offer with fresh ICE credentials, same manifest and `manifest_version`, queued like a renegotiation
connected	operator ping	connected	`pong{t0,t1,t2}`; liveness timer reset
connected	no ping for 15 s (3 × 5 s)	closing	`session-close(reason="heartbeat")`
connected	ICE/DTLS `failed`	closing	`session-close(reason="ice-failed")` — the operator’s ladder decides what to do next; the agent never restarts ICE on its own initiative (it always offers, but only when asked)
any	`session-close` / `peer-gone` from the server	closing	—
any	pipeline error on this session’s consumer	closing	`session-close(reason="media-error")`; a media-plane rebuild is decided by the plane, not the session
any	watchdog expiry in `building`/`offered`	closing	`session-close(reason="negotiation-timeout:<last milestone>")` — the milestone name is the diagnostic. Only milestones that need the peer count (`offer-created`, `answer-received`, …): the local `local-description-set` / `ice-gathering-complete` neither re-arm the watchdog nor name the timeout, so an unanswered offer reads `negotiation-timeout:offer-created`
closing	—	closed	`Capability::release_all_input` on every capability whose manifest says `input_bearing` (unconditionally, before anything else), `session_detached(reason, detail)`, `session-close` sent, valves closed, FrameHub subscriptions dropped; then, after a bounded flush window (150 ms, so what the release emitted on `fjarr:control` leaves before the transport does), consumer pipeline to NULL, `webrtcbin` disposed on the loop, `SessionEvent{ended}`; generation bumped. Inbound envelopes are dropped from `closing` on; `close_all` (shutdown, socket loss, plane rebuild) skips the window

The order in closing is a safety behaviour with a regression test (docs/15): a session that ends mid-keydown must inject the key-up before the pipeline is touched, because pipeline teardown can block.

Milestones (the watchdog’s vocabulary, also snapshot triggers): attached, channels-created, caps-fixed, offer-created, local-description-set, answer-received, remote-description-set, ice-gathering-complete, ice-connected, dtls-connected, control-open, first-frame-sent. After connected, the further snapshot triggers are events, not milestones (the watchdog is off): select-tracks, renegotiation, state-changed, producer-restart, plane-rebuild. Every milestone/event is logged with the session id and elapsed time since attached; dot_dir set makes each dump the pipeline graph (docs/24).

Offer construction and renegotiation

The PeerConnection wrapper owns the only code that talks to webrtcbin. The rules it implements, with the element-level facts the webrtcbin spike established on GStreamer 1.24.2 and re-verified on 1.28.2 in slice 2.9 (its README has the exact API sequences):

bundle-policy=max-bundle, always. With the default policy the DataChannel transport never connects (spike Q6). One transport carries every track and channel.
Channels before media, pipeline ≥ READY first. create-data-channel asserts in NULL state; the wrapper brings the peer pipeline to READY, creates all DataChannels, then requests sink_%u pads, so the initial offer carries m=application and the control channel is open by the time DTLS completes (Q1).
Receive handlers exist before the channel opens. Each channel’s on-message-string/on-message-data handler is connected when the channel is created, and posts to the core loop from then on; on-open only sets up the sending side. Connecting them in the loop’s on-open handler lost the operator’s first request whenever the loop was busy as the channel came up: the client, which sends as soon as its end is open, got ahead of the posted on-open, webrtcbin had no handler, and the message was gone — the unanswered fjarr.net/open of #33 (docs/18, reproduced 2026-09-30 by blocking the loop while DTLS and SCTP came up). A message that reaches the loop before its channel’s on-open is held and delivered right after it.
And the same outbound. A capability may send events from session_attached, before fjarr:control is open (the cursor’s current shape, the robot’s clipboard offer). Those envelopes are held, at most 64, and sent in order the moment the channel opens, before anything else goes out on it. Dropping them lost the cursor’s shape for every session after the first (2026-10-03): the shape only goes out again when it changes.
Caps-gated offer. A GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM probe on each track’s payloader source pad waits for caps with fixed payload and ssrc (1–3 ms after PLAYING); the offer is created only when every track has fixed caps. Disabled tracks are still added — their caps come from the encoder configuration, so enabling later is a valve flip, never a renegotiation. An offer_creation_started latch makes the gate idempotent under on-negotiation-needed racing the probes (Q2).
A valve opens only on an answered m-section. webrtcbin holds a sink pad’s buffers until the transceiver behind it is negotiated — for a pad added by renegotiation, until the answer to that offer is applied; for an m-section the answer rejected, for good. A streaming thread parked there holds the payloader’s stream lock, and tearing the branch down (remove_track, which waits on that lock) then never returns: the core loop that would apply the releasing answer is the one stuck, so it is a permanent hang, not a delay (reproduced 2026-09-29 by enabling a renegotiated track before its answer, and by enabling one the answer rejected, then removing it). So the consumer records each track’s negotiation state from the applied answer, select-tracks on a track not yet answered records the demand and opens the valve (and requests the keyframe) when the answer is applied, and a track the answer rejected is refused. An enable before negotiation carried nothing to the peer anyway; this only stops it reaching webrtcbin.
Manifest mid comes from the offer SDP. The transceiver’s mid property stays NULL until the answer is applied (Q2), so the manifest builder parses a=mid: per m-section from the offer it just created. sink_%u pads take the next free m-line index (the DC occupies one), so the mapping pad → m-section → track_id is recorded at pad request time.
Renegotiation queue. update_tracks(new set) diffs by track_id. Additions request a new sink_%u pad, wait for its caps and re-offer; the existing m-sections keep their mid and media on them continues (spike Q3: 34–36 ms max gap on the untouched track, within the docs/16 budget). Removals (settled in slice 2.9, ADR-0022): close the track’s valve first, then set its transceiver to inactive and re-offer, the JSEP way; the track leaves the manifest and the transceiver stays in a pool, so the next addition of the same kind (re-plugging a monitor) reuses the m-section by flipping it back to sendonly. A reused m-section keeps its SSRC, so the rebuilt payloader continues the old one’s sequence numbers: one that started below them sent packet indexes SRTP had seen on that SSRC, libsrtp refused them as replays, and the session ended (a resumed desktop, mini-PC, 2026-10-03). The valve-first order matters: an offerer keeps pushing RTP on an inactive m-section until its valve closes (re-run). The 1.24 stall was the answerer sending EOS down its source pad, and it reproduces on a 1.28 answerer with default properties; with reuse-source-pads=TRUE it is gone (no RcvbufErrors, data channel both ways, the other track untouched). Every webrtcbin answerer Fjarr ships (fjarr-opsim, the loop tests) therefore sets that property and needs GStreamer ≥ 1.26; browsers never had the mechanic (Chromium check in slice 3a). The offerer side needs nothing version-specific, so an embedder on 1.24 gets the same removal path. The queue holds at most one un-answered offer; changes arriving in flight fold into the next offer; manifest_version increments per offer sent.
ICE restart: not on any GStreamer. webrtcbin ignores the offer options argument (a TODO in the source through 1.28), reuses the previous ICE credentials on every re-offer (three FIXME: deal with ICE restarts sites) and never calls libnice’s restart. The agent therefore answers the operator’s ice-restart with session-close{reason:"ice-restart", retry:true} (docs/08#reconnection): the operator opens a new session immediately and the agent builds a fresh peer connection. It is a new session: capabilities see session_detached then session_attached, the operator’s demand is re-flushed, and the docs/10 control claims — keyed on the operator identity and 30 s fail-open — carry across the gap, so no other operator can take control during a restart. The wrapper keeps a supports_ice_restart() probe so a stack that gains it (open question #21) switches to the in-place re-offer with no protocol change.
ICE. Trickled both ways; candidate: "" marks end-of-candidates; remote candidates arriving before a remote description is set are queued. libnice’s UPnP-IGD is off on every webrtcbin, set through its ICE agent’s agent before gathering. libnice’s default is on, which asks the robot’s gateway to map ports for its host candidates. A robot opening ports on a customer’s router is not Fjarr’s to decide, and TURN is the path for what direct candidates cannot reach. Found in the 2026-09-28 review: it had been on since M1, hidden behind a leak and a race suppression for its discovery thread. A Fjarr tunnel interface never carries ICE. The consumer hands libnice an explicit list of local addresses before it gathers — every address libnice would find itself (up, not loopback), minus those on the tunnel interfaces: the one fjarr.net is configured with (default fjarr0) and any other named fjarr*. Left to itself libnice offered the robot’s tunnel address as a host candidate (#34, docs/18, seen on the mini-PC): a pair between the two tunnel addresses works exactly while the link is up, and the link would then ride on itself. Excluded by name, not by the 100.64.0.0/10 range: that range is also carrier-grade NAT space a 4G robot really uses. If nothing is left after the exclusion, libnice gathers as it would have. TURN via add-turn-server (turn(s)://user:pass@host:port[?transport=tcp], Q6) with credentials from config (dev) or the docs/10 scheme (M5); ice-transport-policy from config for relay-only tests.
Stats. get-stats returns outbound-rtp with ssrc, bytes-sent, packets-sent (Q5) — keyed back to track_id through the ssrc recorded at the caps gate; that is the bandwidth-stats source.
Answerer facts for fjarr-opsim (a webrtcbin answerer): pad-added fires on the first RTP packet (~200 ms after the answer) with pads named src_<counter>, not by m-line — map through the pad’s transceiver property (Q3c).

Media plane

Video sources: one contract, three ways to provide one

Every track — a test pattern, a USB webcam, a stereo/depth camera with a vendor SDK, a network camera, a remote-desktop monitor — enters the media plane through the same VideoSource contract (docs/09): a GStreamer bin with one or more named output pads producing raw frames (or, when the device encodes itself, an elementary stream), plus availability and lifecycle. The core never knows what kind of device is behind a bin.

A customer provides a source in one of three ways, in increasing effort:

Tier	How	Covers	Code
1 — declarative	a GStreamer description string in `fjarr.toml`: `source = "v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720 ! jpegdec"`; the core wraps it with `gst_parse_bin_from_description` and a ghost `src` pad	anything with a GStreamer plugin: USB/CSI cameras (`v4l2src`, `libcamerasrc`), network cameras (`rtspsrc`), files (`filesrc ! decodebin`), test patterns (`videotestsrc`), vendor plugins the camera maker ships — e.g. Stereolabs ZED (`zedsrc` from the ZED GStreamer plugins, which also exposes left/right/depth), NVIDIA Jetson CSI (`nvarguscamerasrc`), Basler (`pylonsrc`), Allied Vision (`vimbaxsrc`)	none
2 — registered type	a C++ `VideoSource` implementation registered under a type name (`agent.register_source_type("acme.stereo", factory)`), referenced from config as `source = { type = "acme.stereo", serial = "…" }` with params validated against the type’s JSON Schema	SDK-backed devices without a GStreamer plugin (the bin wraps `appsrc` fed by the SDK — Intel RealSense through librealsense, a ZED through its SDK when depth post-processing must run on-device before encoding, industrial GigE cameras via their vendor SDK), multi-output devices (left/right/depth from one device), sources needing custom hot-plug or reconfiguration	one class
3 — capability	a capability that creates tracks from its own device model (the way `fjarr.desktop` turns monitors from a `DesktopBackend` into tracks)	devices whose control surface is the point, not just video	a capability

Vendor support is never a core dependency (ADR-0020): drivers Fjarr distributes are GStreamer plugins in separate, per-architecture packages (fjarr-gst-<vendor>) reached through tier 1; the core links no SDK, a customer installs only the packages for the hardware they own, and a configured track whose element is missing is unavailable with the reason (or a startup error if the track is required = true) while everything else runs. Tier 2 is for the embedding application’s own code. The same rule covers the desktop: capture is GStreamer (ximagesrc, pipewiresrc), while input, monitors, cursor and clipboard live in in-tree backend modules loaded at runtime from separate packages, so the core carries no X11, Wayland or PipeWire dependency (ADR-0021).

Tier 1 and tier 2 sources are both consumed by the built-in fjarr.camera capability, whose config is a list of tracks referencing sources (docs/06); a customer therefore adds a camera Fjarr has never heard of by editing config, and adds a whole new kind of camera by registering one class, without touching the capability or the core. A capability receives the registry as a SourceFactory in configure() (docs/09), so tier-2 types the embedding application registered resolve exactly like the built-ins. Built-in source types (slice 4): gst (tier 1, the default when source is a string), test (videotestsrc with pattern/size/fps params), v4l2 (device — a path or a /dev/v4l/by-id name — format mjpeg|yuyv|auto (auto = the device’s preferred raw format in system memory; a camera that only offers MJPEG takes mjpeg, and --probe-source says which), width, height, fps: a convenience over tier 1 with hot-plug from the kernel’s /dev/v4l/by-id tree, watched with GIO; no libudev in the core), rtsp (url, latency ms, protocols tcp|udp|auto, and from slice 6b passthrough and thumbnail_url — the camera’s own stream and its substream, below; without passthrough the stream is decoded to raw).

The contract, in prose (the C++ is in docs/09):

Outputs. A bin exposes one ghost pad src (single-output) or src_<name> pads (multi-output, e.g. src_left, src_right, src_depth); each output becomes one track. Output caps are raw (video/x-raw, any format — the core inserts videoconvert / vapostproc as needed), raw in device memory (memory:DMABuf, memory:VAMemory, NVMM later) for zero-copy into the hardware encoder, or an elementary stream (video/x-h264; video/x-h265 once browsers take it) for cameras with on-board encoders. Passthrough is that declaration: a source whose demanded output declares an elementary stream is parsed (h264parse config-interval=-1) and packetized, never decoded or encoded, so the track costs the robot no encoder at all. The rtsp type turns it on with passthrough = true (docs/06), a gst description with the same param, a registered type by declaring the caps — the core decides from the caps alone, so nothing else in the media plane knows about the setting. A passthrough track’s tiers are the camera’s own streams: active is the output, thumbnail a second output of that name (rtsp’s thumbnail_url), and without one the track has one tier and no adaptation (docs/16, adaptive: false). Keyframes cannot be requested from the camera: the hub’s keyframe gate still starts a joining viewer from the retained keyframe, and a PLI is counted and dropped. --probe-source reports the codec, profile and level a browser must accept. A source declares each output’s kind (video or audio; audio outputs feed fjarr.audio).
Lifecycle. describe() (outputs, declared caps, stable identity), create_bin() on demand, and the bin’s normal GStreamer state changes; the core creates the bin when the first tier of the first output is demanded and disposes it after the idle grace, so an unplugged or idle camera holds no device handle.
Availability and hot-plug. available() and on_availability_changed(callback): a USB camera unplugged mid-session makes its tracks disappear and reappear exactly like monitors do — through update_tracks and the docs/08 renegotiation — with the same stable track_id (from the config key, never from a device index). Sources without native events may poll; v4l2 watches the kernel’s /dev/v4l/by-id tree (udev populates it) through a GIO file monitor (a directory that does not exist yet is polled by GIO, so the first camera on a machine that had none is noticed within a few seconds).
Errors. A bin that errors on the bus is restarted with the producer backoff; a source may additionally report a permanent failure (unavailable(reason)) so the core stops retrying and the track shows the reason in the manifest label.
Threading. Source code runs on GStreamer streaming threads inside its bin, and its callbacks (availability) are marshaled by the core; a source never touches sessions, encoders or the FrameHub.

User-friendliness is a deliverable, not a hope: fjarr-agent --probe-source '<description or type>' builds the bin standalone, negotiates caps, prints the resolved format/size/fps and memory type, runs for two seconds and reports frame rate and any bus error — so a customer can validate a new camera on the robot without a browser or a server. The doctor gains a row per configured source.

The desktop descriptor handover (ADR-0028) {#desktop-descriptor-handover}

On GNOME the desktop backend module receives its PipeWire and EIS descriptors from fjarr-desktop-session (ADR-0028). It never opens the desktop user’s bus. This handover is measured, not assumed. On the spike machine on 2026-09-28 (GNOME 50.1, PipeWire 1.6.2, libei 1.5.0; spikes/desktop-helper), a process ran under its own system account, from a system unit with no session environment. It could not reach the user’s bus or PipeWire socket itself. It accepted the helper after the SO_PEERCRED check, captured the oracle’s magenta through pipewiresrc fd=, and injected two keys and a click through libei that the oracle logged. It rejected root and non-members. The ADR stands as written.

What the module and the package take from it:

PipeWire. The helper hands over a connection it opened, so the server has already taken the helper’s credentials. The agent passes the descriptor to pipewiresrc fd= together with the stream’s node id. As the spike did it, the connection carries all of the desktop user’s PipeWire rights, microphones included. So the helper narrows the connection before handing it over, as the portal’s OpenPipeWireRemote does. Proven 2026-10-01 (spikes/pipewire-narrowing; PipeWire 1.6.2, WirePlumber 0.5.13): an ordinary session-user process calls pw_client_update_permissions on its own connection with the core r-x, the stream’s node r-x, the client-node factory r-x (a capture stream is a client-node) and PW_ID_ANY 0, then pw_core_steal_fd. Through that descriptor the screen captures and a live microphone yields nothing, by serial, name, id or as the default source. The receiver cannot widen it, because it cannot see its own client object. This is the account boundary for audio, and CI checks it on every push (fixture-helper-check). The stream is damage-driven, so a static screen yields almost no buffers, and the source must not treat silence as capture loss.
EIS. The helper calls ConnectToEIS after Start(). mutter then offers one keyboard device and one absolute pointer device with buttons. The pointer’s region is the monitor’s logical rectangle, so the module maps a track-local point to the stream’s position plus that point. Keys are evdev codes. Input lands on the focused surface; the handover does nothing about focus.
The socket’s group. The agent has to be in fjarr-desktop (SupplementaryGroups= in the unit) to give /run/fjarr/desktop.sock that group, unless a .socket unit creates the socket. Adding the desktop account to the group takes effect only at its next login, because the running systemd --user keeps its old groups. The installer restarts the session or says that it must.

The helper protocol {#desktop-helper-protocol}

What crosses /run/fjarr/desktop.sock between fjarr-desktop-session and backend module E (specified 2026-09-30 for M3 slice 3.1; ADR-0028 and its addendum). The protocol is internal: both ends ship in the same packages, and its name is bumped with the module seam’s.

Transport. AF_UNIX, SOCK_SEQPACKET: every message is one datagram, so there is no framing to write, and descriptors travel with the message that names them (SCM_RIGHTS). A message is one JSON object, at most 64 KiB, with a type. The agent’s module listens (mode 0660, the desktop group), the helper connects and reconnects, and the module accepts a connection only when SO_PEERCRED says the configured desktop uid. One helper at a time; a second connection is refused while one is live.
Handshake. The helper sends hello {protocol: "fjarr-desktop-1", helper, session: {type: "wayland", desktop, compositor: "mutter"}}. The module answers welcome {protocol}, or refuse {reason} and closes. A protocol it does not speak is refused by name, so a mismatched package pair says so rather than failing later.
Monitors. The helper sends monitors {monitors: [{connector, name, identity: {vendor, product, serial}, x, y, width, height, scale, primary, kind: "physical" | "virtual"}]} after welcome, and again whenever the layout changes. identity comes from the EDID, which is what monitor ids are made of (docs/09).
Capture. The module sends start-capture {id, connector, cursor: "embedded" | "metadata" | "hidden"}. The helper records that monitor in a ScreenCast session of its own, not linked to the input session (below), starts it, and replies capture-started {id, node, x, y, width, height} with one descriptor: a PipeWire connection, or capture-failed {id, reason}. stop-capture {id} ends it. The module reads the stream with its own stream reader (below), which re-sends the last frame on a still screen: mutter’s stream is damage-driven and silent then, and the encoder must keep producing. Silence is never capture loss. The connection is narrowed to that node before it is handed over (above), so one connection serves one capture. One session per capture is what hot-plug needs (measured 2026-10-01, mutter 50). A started RemoteDesktop session takes no new linked ScreenCast session, and a RecordMonitor on its started ScreenCast session is accepted but never produces a stream. A capture of its own starts and stops without touching any other.
The stream reader and the cursor (slice 3.5; spikes 2026-10-03, spikes/desktop-comfort). mutter puts the cursor on the stream as SPA_META_Cursor: the position on every pointer move, the shape once per change, an empty shape when there is no cursor. pipewiresrc drops it. So module E reads each capture’s stream itself: a pw_stream on the handed connection, on a PipeWire thread of its own, linked as soon as the capture starts and the node’s only consumer. From every buffer it takes the frame and the cursor. The frame is a DMA-BUF handed on without a copy where the encoder can import it, and otherwise shared memory copied into a GstBuffer (ADR-0041; below). The frames go to the track’s source, an appsrc: the reader pushes each new one, re-sends the last one every keepalive interval (helper.keepalive_ms, 100 ms by default) on a still screen, and gives a newly built pipeline the last frame at once. A re-sent frame is marked as a repeat, so a capture’s new frames and its repeats are counted apart (below). Two consumers on one node do not work: mutter sends the shape, and on a still screen the only frame, to whoever is linked when it is produced, so a reader linked first starved pipewiresrc of its first frame, and one linked second never learnt the shape (both measured). A capture that streams without a frame falls back to the cursor in the video. In metadata mode mutter copies a monitor’s last painted image, and a freshly plugged monitor that has not painted yet gives none: every buffer then carries only the cursor (a 0-byte chunk flagged corrupted), on a still screen for good, and a new capture of the same monitor fared no better (measured on the fixture’s virtual monitors, about one hot-plug in four, 2026-10-03). Embedded mode renders the frame itself and never failed (10 of 10). So when a capture has streamed 500 ms without a frame, module E restarts it with cursor: "embedded", marks the monitor, and the capability announces it (monitors, mode-change) with cursor: "embedded" in its info (docs/08): that monitor’s video carries the cursor, and the client draws none there. It restarts a capture at most three times. After the third, it logs that the capture has no frame and leaves that last, embedded stream running: it is not stopped, and its frames begin when mutter first records one. Over SHM, a capture started while a window is scanned out directly hits this path, and the embedded restarts do not help (below). Each shape is named by a hash of its pixels and hotspot (mutter’s own id does not change with the shape); positions are normalized to the capture’s rectangle. Results reach the core loop by posting; the PipeWire thread never waits on it, because a buffer goes back to mutter only once its consumer has returned it.
Input. open-input {} makes the helper start its one RemoteDesktop session, if it has not, call ConnectToEIS on it, and reply input-opened {} with one descriptor: the EIS socket. The module is its libei sender (keyboard, absolute pointer, buttons, scroll), mapping a track-local point to the stream’s x, y plus that point. The session has no linked ScreenCast on purpose. Its absolute pointer then covers every monitor in the layout, and on a hot-plug mutter replaces the pointer device with one covering the new layout. With a linked stream, the pointer covers only the linked monitors, and a monitor plugged later is never added (both measured 2026-10-01).
Typing text. Keys are evdev codes, and the agent maps a client’s KeyboardEvent.code to one. text (docs/08) has no such path: mutter 50 and libei 1.5 have no Unicode input, and mutter’s own keysym call types only what the active keymap has (spike 2026-10-01: “a” and ”@” typed, “å” did not, on a US layout). So module E reads the keymap mutter hands over with the EIS keyboard device and finds, with xkbcommon, each character’s keycode and the modifiers of its level. It types over the same EIS socket as keys, so a key and the text after it arrive in order. A character no key produces makes the whole text fail by name before anything is typed. The layout is the robot’s: on GNOME, gnome-shell applies the configured input source. Headless mutter alone always uses US, which is why the fixture’s end-to-end test types Shift-level characters and refuses “å”, and a unit test types åäö against a Swedish keymap.
Clipboard (slice 3.5; spike 2026-10-03, spikes/desktop-comfort). mutter’s clipboard belongs to the RemoteDesktop session, and works on the unlinked input session. The helper calls EnableClipboard when it opens input. It sends clipboard-changed {types} when SelectionOwnerChanged says the robot owns the clipboard (session-is-owner: false), with mutter’s mime types; an echo of our own selection is not sent. The module maps them to Fjarr’s (text/plain when any text type is there, image/png when image/png is). clipboard-read {id, mime} makes the helper call SelectionRead and reply clipboard-data {id} with one descriptor, the read end, or clipboard-failed {id, reason}. The module reads at most the type’s limit from it (docs/08: 1 MiB of text, 8 MiB of PNG). The name is mime, never type, which names the message. The helper refuses a mime the robot’s current owner does not offer: an application answers only the names it offers, so a wrong name reads nothing, and the test fixture’s wl-copy answers any name, which hid exactly that (mini-PC, 2026-10-03). clipboard-set {id, types} arrives with one descriptor, a sealed memfd holding the bytes, and types naming them in Fjarr’s names. For text the helper calls SetSelection with every name an application may ask text for (text/plain;charset=utf-8, text/plain, UTF8_STRING, STRING, TEXT), for a PNG with image/png, and answers every SelectionTransfer(mime, serial) from the memfd (SelectionWrite, write, close, SelectionWriteDone), from a write that never blocks the helper on an application that reads slowly. It replies clipboard-set-done {id} once the selection is set, or clipboard-failed {id, reason}. It keeps the memfd until the next set, or until the robot copies something itself. mutter hands its descriptors over non-blocking. A paste on the robot needs a keyboard on the seat; the EIS keyboard device is that keyboard (the spike’s wl-paste refused without one).
Virtual monitors (slice 3.5). mutter’s RecordVirtual makes the monitor and is its stream: the monitor exists while that stream has a consumer, and takes its size from the consumer’s format. So add-virtual {id, width, height, cursor} makes the helper start a ScreenCast session with RecordVirtual and answer like a capture, capture-started {id, node, x, y, width, height, connector} with the narrowed connection, naming the connector the new monitor got (the one that appeared in the layout). stop-capture {id} ends the session and the monitor with it. Module E’s stream reader asks for exactly that size in its format, and when the capability then asks to capture the new monitor, module E hands it this stream’s source rather than recording the monitor a second time. A virtual monitor’s capture is never reported monitor-gone: it is the monitor. It is read through shared memory: a virtual monitor has no KMS CRTC, so mutter never scans a window out on it and records every frame into SHM (below).
Sound (M3.5 slice 3.5.2, below). The helper watches the desktop user’s PipeWire for the effective default output (the default metadata’s default.audio.sink) and sends speakers {sink: {name, description} | null} after welcome and whenever it changes. start-audio {id} makes it answer audio-started {id, node, name} with one descriptor: a PipeWire connection narrowed to that sink, exactly as a capture’s is narrowed to its node, or audio-failed {id, reason}. The helper keeps nothing for it: the agent ends the capture by closing its connection, so there is no stop-audio.
Loss. The helper sends capture-lost {id, reason: "monitor-gone" | "stream-stopped"} when a capture ends without being asked. mutter does not say when a recorded monitor goes away: there is no Stopped on the stream and no Closed on the session, only MonitorsChanged, and the session’s other streams keep producing (measured 2026-10-01, mutter 50, a monitor removed from a two-stream session). So on every layout change the helper compares the monitors with its captures, and sends monitor-gone for each capture whose connector has left. A closed socket is SessionEnded for every capture (CaptureLost, docs/09): the desktop session ended or the helper died, and the capability rebuilds its tracks when a helper connects again.
Growth. Later slices add message types (the clipboard and virtual monitors came in 3.5, the sound in 3.5.2) within fjarr-desktop-1. An end ignores a type it does not know. Changing the meaning of an existing type bumps the name.

Capture memory: DMA-BUF first, shared memory as the fallback {#desktop-capture-memory}

Module E’s reader takes a monitor’s frames as DMA-BUF wherever the encoder can import them, and falls back to shared memory (SHM: MemFd or MemPtr) otherwise (ADR-0041, decided 2026-10-17). SHM alone is not enough on a monitor with a KMS CRTC, a ghost screen included:

Direct scanout. With a hardware cursor, which metadata mode keeps, and nothing of the shell on screen, mutter 50.1 scans a fullscreen window that renders through dma-buf straight out to the monitor. While it does, it does not paint the stage, and records no frame into SHM monitor streams (before_stage_painted returns early for them; fixed upstream in mutter 50.5 and 51.0, not in Ubuntu 26.04’s 50.1: open question #49). Into DMA-BUF streams it copies the scanned-out buffer, every frame. On the mini-PC an SHM capture got 0 buffers in 12.4 s while the screen changed 24 times, and a DMA-BUF stream on the same monitor recorded 72 frames in 36 s of scanout, each the right colour (spikes/capture-scanout). Over SHM the reader’s keepalive kept the last composited frame going, so the video froze while every counter stayed green. A notification banner, a menu or the overview stops scanout while it is on screen, so the freeze comes and goes.
A capture that starts during scanout. Enabling a stream makes mutter queue a redraw, which during scanout copies the scanned-out buffer into DMA-BUF streams: a DMA-BUF capture got its first frame 6 to 18 ms after it started streaming, with a still screen and the pointer hidden, 10 starts of 10. An SHM capture gets none until scanout ends, and the reader’s fallback to the embedded cursor (above) does not end it: mutter 50.1 takes the hardware cursor away only at the next cursor update, and an application that hides the pointer keeps scanout on regardless. That covers an agent restart, a helper reconnect, a hot-plug and resume-sharing during a fullscreen application.

The import list. At startup the media plane asks the active encoder family which DMA-BUF formats it imports without a copy: for vaapi, the drm-format entries of vapostproc’s sink caps under memory:DMABuf, kept to the 8-bit RGB formats mutter produces (XR24, AR24, XB24, AB24). software imports none, and neither do nvcodec and nvv4l2 until they have a path of their own. The capability passes the list in each capture’s CaptureOptions.dmabuf (docs/09). On the mini-PC (radeonsi, Mesa 26.0.8) it holds one modifier for RGB, 0x0200000000000901, while mutter’s first choice is 0x0200000000401a01: offering mutter’s whole list would have negotiated a buffer VA cannot import.

Negotiation. For a physical monitor, with a non-empty import list and helper.capture_memory not "shm" (it is "auto" by default; "shm" keeps every capture on shared memory), the reader first asks mutter’s node for its EnumFormat params on the handed connection, which grants the node R and X (two round trips: a registry sync, then one after enum_params), and only then links. Its EnumFormat offers come in order of preference: one per format in the import list, with SPA_FORMAT_VIDEO_modifier set to that format’s modifiers that mutter also lists (flagged mandatory, and left for mutter, which allocates, to fixate), then the same formats without a modifier, then the other formats the reader takes over SHM (BGRx, BGRA, RGBx, RGBA), which is SHM. When mutter’s params cannot be read, the import list goes out whole and PipeWire intersects it; the reader logs a warning saying why (the node not in the registry, an error, or a node that lists no formats). mutter picks one. A Format whose modifier is still a choice is not fixed yet: the reader sets its buffer params and waits for mutter to send the fixed one. When the fixed Format carries a modifier, the reader asks for SPA_DATA_DmaBuf buffers; otherwise for MemFd | MemPtr, as before. A stream that offers DMA-BUF links without PW_STREAM_FLAG_MAP_BUFFERS (nothing of a DMA-BUF is mapped), so should mutter pick SHM after all, the reader maps a MemFd itself. The track’s source is available once that first fixed format has settled the memory, because the producer’s head depends on it (below). The capture keeps that memory for its life. A size change renegotiates within it, and a new DMA-BUF modifier then shows in /sources. A change of memory stops the reader feeding the track at once, since the track’s pipeline was built for the old memory, and restarts the capture, at most three times and never a virtual monitor’s (a restart would make another monitor). Past that the reader adopts the new memory: the source’s caps follow it, and the reader feeds the track again. Either way the producer built for the old memory fails to negotiate inside its source bin and is rebuilt from the source’s new caps on the slow source ladder (below, about 1 s), not through a re-offer, which would take the track away from its viewers. The reader logs the outcome once per capture, at info: capture of HDMI-1: DMA-BUF XR24:0x0200000000000901, into the encoder without a copy, or capture of Meta-0: shared memory (a virtual monitor) (a RecordVirtual monitor’s connector is known only after its stream has negotiated, so its line names it a virtual monitor). The reasons for SHM are named: a virtual monitor, capture_memory = "shm", an encoder that imports no DMA-BUF (naming the family), no modifier in common (both lists printed), mutter offering no DMA-BUF (no GPU it can allocate on, as in CI), mutter choosing SHM over the DMA-BUF offered (saying so when its formats could not be read first), a DMA-BUF buffer the reader could not wrap (below, naming why), or mutter moving a stream the reader adopted to SHM in a renegotiation. SHM on a physical monitor is logged as a warning, because that capture can still freeze behind a scanned-out window. The negotiated caps also show in /sources (docs/24).

Virtual monitors (mutter’s Meta- connectors: RecordVirtual ones and headless mutter’s own) are always read through SHM. They have no KMS CRTC, so mutter rejects them as scanout candidates and records every frame; their DMA-BUF path has not been measured.

DMA-BUF buffers.

Never mapped. A tiled DMA-BUF cannot be mapped (mmap gives EPERM), and reading a LINEAR one cost 48 ms per 1080p frame. A buffer has pixels when its first chunk is non-empty and not flagged SPA_CHUNK_FLAG_CORRUPTED, whatever its memory; a cursor-only buffer is told apart that way, as over SHM, never by mapping it.
Wrapped, not copied. The buffer’s plane becomes a GstMemory from GStreamer’s DMA-BUF allocator, on a duplicate of its descriptor, with its offset and stride in a GstVideoMeta. The appsrc caps are video/x-raw(memory:DMABuf), format=DMA_DRM, drm-format=<fourcc>:<modifier>, with the size and framerate=0/1. Each of mutter’s buffers is wrapped once and reused, as pipewiresrc’s pool does, so VA can keep its import of it. A buffer is wrapped whole or not at all. One with more than one plane (n_datas above 1: an auxiliary or compression plane, Intel CCS or AMD DCC) is not taken yet, because GStreamer 1.28’s VA import counts planes from the DMA_DRM format and maps a buffer with more, which would copy tiled or compressed bytes; no such buffer has been through the chain on any hardware, and the mini-PC’s modifier has one plane. Such a capture, and one whose descriptor would not duplicate or wrap, feeds nothing more and restarts on SHM with that reason. A new format, size or modifier drops the held frame, which was made for the old one.
Held while GStreamer holds the frame. Over SHM the reader copies and queues the buffer back at once. A DMA-BUF buffer goes back to mutter only when the last GstBuffer referencing it is released; the release is posted to the PipeWire thread, which queues it. Each delivery gets a token, an empty GstBuffer that every pushed buffer over it references through a GstParentBufferMeta, which a copy of the buffer carries too; the token’s end (a weak reference) posts the release, through a PipeWire loop event, so it never runs inside the thread that let go. A release for a buffer PipeWire removed meanwhile (a renegotiation) is dropped. The reader holds the newest one for the keepalive, so mutter never writes into a frame the encoder may still read or send again. It asks mutter for enough buffers (at least 8) that one held frame and a full appsrc queue still leave mutter one to record into. With none free, mutter is expected to skip a frame rather than wait; the first mini-PC runs of the reader check that, holding every buffer on purpose.
Implicit sync. mutter advertises the SyncTimeline meta for DMA-BUF streams. The reader does not announce it: mutter then hands over buffers whose rendering is ordered by the kernel’s implicit fences, which amdgpu honours (every blink right in the 3-second runs, every fresh buffer right). Adopting explicit sync later follows ADR-0041’s rules: the meta only together with a Buffers param naming it, acquire waits only for buffers with pixels, and the release only after VA is done.
The cursor is unchanged: SPA_META_Cursor arrives on DMA-BUF buffers, pixel and cursor-only alike, exactly as over SHM (measured 2026-10-17). A pointer the robot’s application hides arrives as an invalid cursor id (0), which the reader hands on as a hidden shape (docs/22). mutter sends the same id 0 to every other monitor’s stream while the pointer is on one (by mutter’s source, not measured), so module E takes a hidden shape from an invalid id only from the monitor the pointer was last seen on, and the pointer arriving on a monitor brings back that monitor’s latest shape unless that shape came from an invalid id (it may only say the pointer was elsewhere; the shape shown stays). A valid id without a bitmap after an invalid id brings back the shape shown before it, since mutter sends a bitmap only when the shape changes; after mutter’s own empty shape it does not. Both rules are pure code (ReaderCursor, CursorArbiter in src/desktop/cursor_image.hpp) with unit tests (test_cursor_image.cpp).

The keepalive with GPU buffers. On a still screen the reader pushes the held buffer again every keepalive interval, as a new GstBuffer over the same memory, so the encoder re-encodes the same surface; nothing is copied. A newly built pipeline gets it at once, as before.

New frames and repeats. The reader counts per capture the buffers mutter delivered with pixels (new frames), the buffers it pushed again (repeats: keepalive re-sends and the frame given to a newly built pipeline), and the cursor-only buffers, which is what GET /sources shows per capture through the source’s status() (docs/09, docs/24). Every repeat carries a GStreamer custom meta, FjarrFrameRepeat (gst_meta_register_custom_simple), which any source may add to a buffer that repeats content it already sent. The producer’s source pad counter counts buffers with and without it, for any source, which GET /stats shows per producer (source_buffers). A new frame is only “new from mutter”: mutter records one duplicate into every running DMA-BUF stream whenever another screencast starts, and occasionally redraws the same picture, and the reader cannot tell those apart from a change. Over SHM a still screen and a scanned-out window both give no new frames, so only the scanout check on real hardware tells them apart (docs/15).

Into the encoder. A DMA-BUF capture’s frames go to one vapostproc, which imports them, and on to vah264enc with no system-memory step (below). System memory is only for an encoder family that imports no DMA-BUF: it gets SHM from the reader, as today, and is then exposed to the scanout freeze; a GPU download for it (vapostproc to system memory, or GL at 17 to 30 % of a core) is not built until a GNOME robot without VA needs it.

Tests. Unit tests cover the offer for an import list (intersection, order, the SHM tail), every fallback reason, including the one settled when mutter chooses SHM, the pixels-or-cursor-only rule without mapping, the buffer hold and release, a format change dropping the held frame, a multi-plane buffer refused, a suspended capture feeding nothing, the restart-or-adopt rule for a change of memory, and the repeat marking, against hand-made PipeWire buffers in place of a pw_stream (agent/tests/test_capture_memory.cpp). The import list, the producer’s head, the software encoder’s refusal of DMA-BUF and the rebuild of a running producer whose source changed memory have their own (test_encoder_memory.cpp). Where the machine running the tests has VA-API (a developer’s workstation, never CI), that file also sends DMA-BUFs rendered by VA through the reader’s frames and the device-memory head into vah264enc, both tiers. The CI fixture has no GPU and only virtual monitors, so it keeps covering the SHM path. The DMA-BUF path is covered on the mini-PC, by the nightly’s scanout check (docs/15).

The desktop’s monitors (M3 slice 3.4) {#desktop-monitors}

fjarr.desktop offers one video track per monitor, desk-<wire id>, with the monitor’s geometry in the manifest (docs/08). On every monitors message from the helper, the capability diffs the new set by wire id, never by connector or index:

A monitor that arrived gets a capture (start_capture) and a track. Its source is unavailable until the helper hands the stream over, so the track joins the offer when it can deliver frames.
A monitor that left has its capture stopped and its track removed. Its wire id is remembered, so the same monitor returning gets the same track_id.
A monitor that changed (mode, scale, position, the primary flag) keeps its track and capture. Its MonitorInfo is updated in place, and the capture’s stream renegotiates its size by itself.
Every change sends fjarr.desktop monitors {monitors, reason} to each session before update_tracks re-offers, as docs/08 requires, so a client’s placeholders move at once. The reason is initial for a helper’s first set, hotplug when the set of ids changed, and mode-change otherwise.
Zero monitors is a valid state. The tracks go, the session stays, and input has nowhere to land until a monitor returns.

A capture costs little while nobody watches, because its producer starts on the first demand (above). Pointer input names its monitor by track_id, so it lands on that monitor whichever is primary.

Backend A, the X11 module (M3 slice 3.7) {#desktop-x11}

libfjarr-desktop-x11.so (package fjarr-desktop-x11, ADR-0006 and ADR-0021) implements DesktopBackend against an X server from the agent’s own process: no session helper (ADR-0028). The kiosk session grants the agent’s account with xhost +si:localuser:fjarr (done by fjarr-x11-session, below), and the module opens the display named by the capability’s display key (default :0) with Xlib. Everything runs on the core loop: the connection’s descriptor is watched there, and no Xlib call is made from another thread.

Probe. The module is usable when the display opens and has XTest, RandR ≥ 1.5 and XFixes; otherwise probe() says which is missing.
Monitors. RRGetMonitors (it covers real outputs and RandR 1.5 monitors alike). Identity: the EDID property of a monitor’s first output, through the shared monitor_identity (vendor, model, serial), so a monitor has the same wire id under X11 as under GNOME; a monitor with no output or EDID (Xvfb’s --setmonitor, a dummy plug) falls back to its name. A monitor whose output RandR reports disconnected is gone at once, though it keeps its CRTC until the kiosk session switches it off: its EDID is gone with it, and reported it was a nameless phantom. The set is re-read on RandR screen, output and CRTC events and every 500 ms, since a RandR 1.5 monitor change sent no event on Xvfb and the re-read must still meet the 2 s hot-plug budget (docs/06), and reported only when it changed; the capability diffs it by wire id (below).
Capture. One ximagesrc per monitor, cropped to its rectangle (startx/starty/endx/endy, inclusive), show-pointer=false (the cursor travels beside the video, local-cursor mode, as on GNOME) and use-damage=false (damage regions cost more than they save at 30 fps; ADR-0006 measured 14 ms paint-to-capture p50). ximagesrc reads through MIT-SHM, so the agent must share the X server’s IPC namespace: on a robot it does; a container needs ipc: host or the X server’s namespace, or it silently captures stale frames (the fixture did, 2026-10-04). A monitor’s capture is available from the start; a vanished monitor’s capture ends with it. A monitor that moves but keeps its size keeps its capture: the crop follows it in place (ximagesrc reads its origin on every frame), so nothing is re-offered (above). One that changes size is restarted on its new rectangle, since its caps change.
Input. XTest. Pointer: a monitor’s normalized point → the root’s x + nx·w, y + ny·h; buttons 1–3 and 8–9 (back, forward); the wheel as button 4–7 clicks, one per 48 px accumulated (three of the client’s 16 px lines). Keys: X keycode = evdev code + 8 (Xorg’s evdev rule); a key held through XTest auto-repeats on the robot as a real one does. type_text uses the shared KeymapIndex, built from the layout and variant the X server publishes on the root window (_XKB_RULES_NAMES), as on GNOME; characters no key produces are refused by name. release_all_input releases every key and button XTest pressed.
Cursor. XFixes: XFixesSelectCursorInput for shape changes, XFixesGetCursorImage for the pixels (premultiplied ARGB → the shared straight RGBA and content-hashed shape_id); the position is read with XQueryPointer at 30 Hz while it moves, and reported per monitor.
Presence. session_running() is true while the display is open. When the connection drops (the X server restarted, the session logged out), every capture ends (SessionEnded), the monitors become none, on_session_changed(false) fires, and the module reopens the display every 2 s until it is back (docs/08 desktop).
Not offered. Virtual monitors (RecordVirtual is mutter’s) and the sharing stop (GNOME’s indicator): Features says so and the capability refuses add-monitor as unavailable.
Sound (M3.5 slice 3.5.2, ADR-0034): through fjarr-x11-session, as a helper for the sound only (below).
Clipboard (slice 3.7c). X selections on CLIPBOARD, through an unmapped window the module owns: owner changes from XFixesSelectSelectionInput (the module’s own ownership is not a robot copy and is not reported); the offered types from TARGETS (mapped by the shared clipboard_types); reads by XConvertSelection into a property, one at a time, each step and each chunk answered within 5 s or failed; writes by owning the selection with a server timestamp and answering SelectionRequest for TARGETS, TIMESTAMP, every text name and image/png (MULTIPLE is refused). Contents above 256 KiB travel with the INCR protocol in both directions, in 256 KiB chunks; a fixed threshold rather than the server’s request size, which BIG-REQUESTS raises to 16 MiB and would leave the path untested under the 8 MiB limit. A paste that stops taking chunks for 10 s is dropped.

fjarr-x11-session is the session side, started by the kiosk session’s XDG autostart entry, as the session’s user. It hands the agent the desktop’s sound over the helper socket (below). It grants the agent’s account (XAddHost with the localuser server-interpreted family: the same grant as xhost +si:localuser:fjarr, with no authority file changing hands), lays out the outputs, and re-lays them on every RandR output or screen change: outputs that are disconnected but still hold a CRTC are switched off, connected ones are switched on at their preferred mode, left to right in connector order, each position set explicitly (shrinking the root moves windows). It keeps a ghost from being primary while a real monitor is connected (docs/26): with no primary, or a ghost or unplugged one, the leftmost real monitor becomes primary; a real monitor already primary stays so. ADR-0006 found a bare X kiosk lays out nothing and never reacts to a hot-plug; an MST chain’s middle monitor that does not come back after a hot-plug is a driver limit this does not fix.

Whether the desktop is there (M3 slice 3.5) {#desktop-presence}

Module E reports session_running() true from the helper’s hello until its connection closes, and calls on_session_changed on each change. The capability sends docs/08 desktop {state} to every session at its start and on each change, so a dashboard can tell “no display” from “no desktop” (an empty monitor set alone says neither). The new virtuals change DesktopBackend’s vtable; like the rest of M3 they amend fjarr_desktop_module_v1 in place, which module.hpp allows until the first M3 packages ship, and every change after that bumps the name.

GNOME’s screen-sharing indicator stops every capture and the input session from the robot’s side. A lost capture alone does not tell the stop apart: an unplugged monitor’s capture also ends, before the layout change arrives (fixture, 2026-10-03). The input session does: only the stop closes it. So the helper sends sharing-stopped {} when mutter closes the input session, beside capture-lost {reason: "stream-stopped"} per capture, and drops the dead input session so the next open-input makes a new one. Module E then reports CaptureLost::SharingStopped (docs/09) for every capture and asks for input again only with its next capture. A virtual monitor’s stream ends its monitor, so it is reported MonitorGone, leaves the layout as a hot-plug, and is not brought back. The desktop’s sound (below) stops with them: module E ends its audio capture and the source is unavailable until a capture starts again after resume-sharing.

The capability turns the first SharingStopped into sharing: stopped (docs/08) for every session: each stopped screen’s source is unavailable, so the re-offer drops its track. A SourceStopped alone changes nothing here. On resume-sharing it stops and starts the capture of every stopped screen through the backend (stop_capture, then start_capture with the same options), which brings the input back with it; each track returns as its new source becomes available, and sharing: on goes out. Nothing resumes by itself: a stop holds until a session asks (docs/10). Mutter refuses Stop on a session from anyone but its owner and the test fixture runs no gnome-shell, so CI drives this through the test module’s SourceStopped; the indicator itself is checked on the mini-PC.

Encoders and tiers

The core owns encoding through the EncoderAdapter seam, so a source never picks an encoder and hardware differences stay in one place. From M2.6 there are four families — software, vaapi, nvcodec, nvv4l2 — selected by media.encoder (ADR-0025, which also fixes the auto order and the rule that each family owes the nightly a runner).

The source’s memory type picks the upload path, not the encoder. A frame already on the device is never round-tripped through system memory, and a frame in system memory is converted once, on the CPU, and uploaded once: DMABuf → vapostproc; system memory → videoconvert ! vapostproc for VA-API, videoconvert ! cudaupload for nvcodec (or cudaupload ! cudaconvertscale where the CUDA runtime compiler is present); NVMM → nvvidconv. A tier’s scale and rate filters move to the device wherever a device-side equivalent exists. A pre-encoded output bypasses all of it (h264parse config-interval=-1 only).

The device-memory head (first used by the desktop’s DMA-BUF capture, ADR-0041). Until then every raw source was in system memory, and a producer’s shared head was videoconvert and an I420 capsfilter before the tee. A source whose caps carry memory:DMABuf gets no videoconvert: its head is a capsfilter on that memory feature (video/x-raw(memory:DMABuf), format=DMA_DRM), keeping the rawcaps role name, then one vapostproc (/import), which imports each buffer as a VA surface and converts it to NV12, a capsfilter video/x-raw(memory:VAMemory), format=NV12 (/vacaps), and the tee. Each tier branch is videorate drop-only=true with its rate filter stated on VA memory, then, for the thumbnail only, a second vapostproc that scales it on the GPU to the tier’s size, then vah264enc as before. No videoscale, no system-memory caps and no download anywhere on that path. One importer, not one per tier (as first designed): two vapostproc elements importing the same foreign DMA-BUF at once race in GStreamer 1.28’s VA import, whose second import replaces the surface the first is using, and crashed Intel’s iHD driver inside vaEndPicture (measured on a workstation, 2026-10-17); the import is also done once instead of per tier. A DMA-BUF source is never stamped: the frame-stamp painter (docs/25) maps every frame, which a DMA-BUF never is, and only fjarr.test sources are stamped (the producer logs a warning should one be). Measured at 1920×1080 and 30 fps on the mini-PC: about 6 % of one core for the pipeline, against about 28 % for the copy, videoconvert and vapostproc chain it replaces. A software or nvcodec producer never receives device memory from the desktop, because its import list is empty (above).

A family declares whether it can change bitrate while playing. Rate control (below) sets a tier’s target every 500 ms; a family that cannot take a live change says so, and its tracks fall back to tier switching alone through the same adaptive: false path passthrough uses. No family may make bandwidth-stats claim an adaptivity it does not have.

Tiers (docs/16) are separate producers of the same source output: the output goes through a tee; active encodes at full resolution/fps, thumbnail at 960×540@5 with a low bitrate cap. A producer starts lazily on first demand and stops when no session demands its tier for a grace period (10 s), so an idle robot encodes nothing. select-tracks therefore selects (track_id, tier) in the FrameHub, and switching tiers is a subscription change plus a keyframe request, not a renegotiation.

A tier’s branch is locked before it is torn down. Starting a tier on a playing producer syncs the new branch’s state with the pipeline and leaves the pipeline’s own asynchronous state change in flight; when it completes, the bin sets every child it still holds back to PLAYING. A stop in that window (stop_tier moments after start_tier) had its branch re-PLAYed between set_state(NULL) and removal: elements disposed un-NULLed (the parser’s buffers leaked) and, within a few dozen start/stop cycles, a stop that never returned. So stop_tier calls gst_element_set_locked_state(TRUE) on the branch’s elements first, which keeps the parent’s hands off them — the rule for any branch removed from a running pipeline (found by the nightly memcheck, 2026-09-29; LoopMedia.aTierStoppedWhileItsStartIsStillSettlingIsTornDownCleanly). The branch is also unlinked only from an IDLE probe on the tee’s pad: a branch set to NULL under a push in flight returns FLUSHING through the tee, and the source then pauses its task for good, freezing every tier while the pipeline still reads PLAYING (found under TSan with the CPU loaded, 2026-10-01, #38). The consumer’s branches have a different hazard, a thread parked on an un-negotiated webrtcbin pad, and a different rule (valves open only on answered m-sections).

FrameHub

One mutex, one entry per (track_id, tier): the latest keyframe, a ring of the last N encoded samples (N sized to one GOP + 1), and the subscriber list (weak_ptr<Subscriber>). Producers push under the lock; consumers subscribe with a keyframe gate: a new subscriber receives nothing until the next keyframe (or the retained one), then everything, with PTS rebased to the consumer’s clock so a late joiner starts at zero. A subscriber that falls behind (ring overrun) is skipped to the next keyframe, never fed stale frames — the same newest-wins discipline the web client applies to telemetry.

Keyframe requests (force-key-unit events upstream of the encoder) are issued on: subscribe, select-tracks enable, PLI/FIR from the peer (webrtcbin forwards them as upstream events), and tier switch. They are rate-limited per producer (≥ 1 s apart) so ten operators joining at once cost one keyframe.

Consumer pipeline (per session)

[per track]  appsrc(is-live, format=time) ! queue(leaky=downstream, max 1 GOP)
             ! valve ! rtp{h264,vp8,opus}pay(pt, ssrc, config-interval=-1)
             ! webrtcbin.sink_%u

valve drop=TRUE is the disabled state — the transceiver stays negotiated and no RTP flows; enabling flips the valve and requests a keyframe. The queue is bounded to one GOP so a stalled network drops frames rather than growing latency. bandwidth-stats (docs/06) is sampled every second from get-stats (outbound-rtp bytes/packets per ssrc → track_id) on the core loop and sent as an event on control.

Fan-out: what is shared, what is per consumer {#fan-out}

The whole point of the media plane is that adding an operator costs almost nothing. The cost model, stage by stage, for one source output with V viewers across T demanded tiers:

Stage	Count	Cost driver	Shared?
Capture / source bin	1 per output	device, format conversion	yes — one device handle, one capture thread
`tee` after the source	1	none (buffer refs)	yes
Scale + encode	1 per demanded tier (`T ≤ 2`)	the dominant cost: VA-API encode, or software if the adapter falls back	yes — the encoder never knows how many viewers exist
FrameHub ring	1 per `(output, tier)`	~1 GOP of encoded samples (≈ 0.5–2 MB at active tier)	yes
Delivery to a subscriber	`V`	one `gst_buffer_ref` + a shallow metadata-only copy for PTS rebase; no pixel copy, ever	per consumer, ≈ µs
`appsrc ! queue ! valve ! payloader`	`V`	RTP packetization of an already-encoded stream (fraction of a percent per viewer)	per consumer
`webrtcbin` (SRTP encryption, RTCP, ICE, DTLS, SCTP)	`V`	AES per packet — the real per-viewer cost, unavoidable since every peer has its own DTLS-SRTP keys	per consumer
Keyframe on join / enable	shared, rate-limited (≥ 1 s)	a keyframe is a bitrate spike for every viewer of that tier; ten joins in a second cost one	yes

Guarantees the implementation must keep (each with a loop test):

Zero-copy fan-out. Encoded samples are shared by reference from the producer’s appsink through the ring to every appsrc; the only per-consumer allocation is the GstBuffer metadata copy that carries the rebased timestamps (gst_buffer_copy_region with GST_BUFFER_COPY_METADATA, memory shared). The docs/16 “0 allocations per frame” budget is checked with heaptrack; a memcpy of pixel data anywhere after the encoder is a bug.
One encoder per demanded tier, none when idle. A tier’s producer exists only while at least one session demands it (select-tracks), with a 10 s grace so an operator toggling a tile does not restart the encoder. The hardware encoder count is bounded by outputs × tiers, which is what docs/16’s iGPU budget is written against; the encoder adapter reports its instance limit and the core refuses a tier (with a logged reason and the track marked unavailable in the manifest) rather than silently falling back to software.
A slow or dead consumer never touches the others. Each consumer has its own bounded queue(leaky=downstream) and its own appsrc in non-blocking mode; the hub’s delivery loop never waits on a consumer. A consumer that falls a GOP behind is resynced at the next keyframe.
Consumer churn never touches capture or encode. Subscribe and unsubscribe are list operations under the hub mutex; the producer pipeline’s state does not change when sessions come and go.

Why a hub and not one tee in one pipeline. The camera streamer tried both: its v2 linked every peer’s branch into the single producer pipeline, so one peer’s webrtcbin error took the encoders down for everyone and a stalled peer back-pressured the capture. Its v1 hub with per-peer pipelines had none of those problems and the same per-viewer cost, because tee also only passes buffer references. The hub adds one lock and one ring per tier — a price worth paying for the fault isolation, the keyframe gate for late joiners, and the ability to feed non-WebRTC subscribers from the same ring: the snapshot request (fjarr.camera/snapshot, decoded from the retained keyframe or taken from the raw tee at full resolution), a future local recorder, and the analytics adapter, none of which need a peer connection.

What is deliberately not shared. RTP packetization and SRTP could in principle be shared by teeing RTP packets into every webrtcbin with the same SSRC; the saving is a payloader per viewer (negligible) and it would cost per-peer PLI/NACK handling, per-peer valves and the ability to give different tiers to different viewers. Not worth it; revisit only if profiling shows payloading above 1 % per viewer.

Audio fans out the same way: one Opus encoder per source, the ring holds encoded frames, each session gets its own appsrc ! rtpopuspay.

Audio: the robot’s microphones (M3.5 slice 3.5.1) {#audio}

fjarr.audio offers each configured microphone as a kind: "audio" track, and the media plane carries it as it carries video, with less:

Where it comes from. A microphone is read through ALSA (alsasrc device=…, a USB or built-in device; the agent’s account is in audio): the agent is a system service, and a desktop user’s PipeWire or Pulse server is not its to open, nor there at all on a headless robot. The desktop’s own sound is different, and comes through the session helper (below). CI uses a test tone (audiotestsrc), a source type like the camera’s test. Config: [capabilities."fjarr.audio".mics.<id>] with label and source.
The producer: source ! audioconvert ! audioresample ! opusenc at 48 kHz mono, 20 ms frames, in-band FEC on: 32 kbps tuned for voice for a microphone, 64 kbps in the general-audio mode for a source that declares AudioContent::Music, the desktop’s sound (docs/16: 32–64 per track; voice tuning made music muffled on the mini-PC, 2026-10-12). Timestamps follow the samples (perfect-timestamp), not when each input buffer arrived: a desktop’s PipeWire buffers arrive unevenly, and their arrival times reached the RTP clock, about 60 of every 150 packets off by up to 6 ms in the fixture, which a browser plays as speeding up and slowing down. opsim’s desktop-audio now requires every RTP step to be one frame. Its encoded frames go into the source’s ring as video’s do. It starts on the first demand and stops after the tier grace, as video producers do. No keyframes, no tiers and no rate control: an Opus track’s bitrate is fixed and small next to video’s.
Per session: appsrc ! rtpopuspay ! webrtcbin, an audio m-section in the offer (sendonly from the agent), enabled and disabled by the same select-tracks as video.
Granted per session (docs/10). session_attached adds the microphone tracks only when the session’s grant carries listen: true, so a session without it is offered none, and the core refuses a select-tracks naming a track that session was not offered (docs/08). Because a producer starts on demand and is shared, a session without listen costs nothing: no pipeline, no encoder, no bytes. No core change is needed: every capability already chooses each session’s tracks in session_attached, from that session’s grant.

Desktop audio: what the robot’s desktop plays (M3.5 slice 3.5.2) {#desktop-audio}

The desktop’s sound is a kind: "audio" track of fjarr.desktop, not of fjarr.audio (docs/06): it follows the desktop grant, so whoever sees the screen hears it, and a view_only session too. It is the monitor of the effective default output: what the robot’s speakers play, whichever application plays it.

The handover (GNOME, backend E). The agent cannot open the desktop user’s PipeWire; the session helper hands it one connection narrowed to the default sink (the helper protocol, Sound). Measured 2026-10-12 in the desktop fixture (spikes/desktop-audio; PipeWire 1.6.2, WirePlumber 0.5.13), with a null sink as the default output, an application playing 440 Hz to it and a live microphone at 1000 Hz. Through a connection that sees only the core, the sink and the client-node factory, a capture stream with stream.capture.sink = true heard the 440 Hz tone whether it named the sink by serial or named no target. Naming the microphone by its serial heard nothing (it hangs, as in the narrowing spike), and asking for the default source linked the sink’s monitor again, never the microphone. The narrowing that keeps the microphones from the screen’s connection keeps them from the sound’s.
Where it is read. Module E’s audio reader: a pw_stream on the handed connection, on a PipeWire thread of its own like the stream reader, with stream.capture.sink = true and the sink as its target, asking for S16LE, 48 kHz, stereo (PipeWire converts). Each buffer goes into the track’s appsrc, timestamped on arrival. From there it is an audio track like a microphone’s (above): Opus at 32 kbps, one tier, demand driven. GStreamer’s pipewiresrc is not used, for the reason the stream reader replaced it: it reads PipeWire’s own client configuration, which the package does not ship.
Only while someone listens. The source is available while a helper is connected and reports an output, and is declared with the desktop’s other tracks. The capture itself runs only while the producer’s pipeline runs: it starts with the first select-tracks that enables the track, and ends when the producer stops after the tier grace (10 s). The producer keeps a stopped pipeline built rather than releasing its source, so module E watches the source’s appsrc on the core loop (after each build, then every 500 ms) and starts or ends the capture when it enters or leaves PAUSED/PLAYING. A capture link keeps a sink awake, and a robot nobody listens to should not pay for it; opsim’s desktop-audio checks that no capture is open before the track is enabled and none 15 s after it is disabled. A start-audio that fails leaves the track silent and is logged.
When the default output changes (headphones plugged in, HDMI chosen), the helper sends speakers again; a running capture is restarted on the new sink and its new reader feeds the pipeline already built, so the track and the session’s offer stay as they were. With no output at all (sink: null) the track goes unavailable and is re-offered without it.
When sharing stops (above), the sound stops with the screens: the desktop’s user ended the sharing, and the speakers are part of it. It comes back with resume-sharing.
Verified 2026-10-12 in the desktop fixture: opsim’s desktop-audio hears silence, then the tone the fixture plays (about 300 ms after it starts), keeps hearing it when the default output changes, and sees the capture open only while the track is enabled (ended 10.3 s after disable); the dashboard’s Sound toggle plays it in the lab browser. The X11 fixture passes the same scenario through fjarr-x11-session as its kiosk account. The stop with the sharing is checked on the mini-PC (3.5.4), as the indicator is: the fixture runs no GNOME Shell.
On X11 (backend A; ADR-0034) the agent cannot open the kiosk user’s sound server either, so fjarr-x11-session, already running for the whole kiosk session, is a helper for the sound only: it connects to the same socket, says hello with session: {type: "x11", desktop, compositor: "x11"}, and serves speakers and start-audio exactly as the GNOME helper does, with the same speaker watch and narrowing. Module A listens on the socket when a desktop account is configured (helper.user or uid, as module E) and reads the sound with the same audio reader. Its helper says nothing about monitors or whether the desktop is there: the module reads those from X. Without the socket the session program still grants and lays out.

The audio uplink: talking to the robot (M3.5 slice 3.5.3) {#audio-uplink}

The operator speaks through the robot’s speaker (docs/08). Inbound media is new to the agent, so it needs one core addition and one capability part:

The core: an inbound slot. SessionContext::add_inbound_audio in session_attached (docs/09) makes the consumer add one audio transceiver, recvonly, with Opus preferences, before the first offer. When webrtcbin’s source pad for it appears, the consumer links queue ! rtpopusdepay ! opusdec ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,rate=48000,channels=1 ! appsink and hands each buffer’s samples to the capability’s on_pcm on that streaming thread. Decoding is per session, as each session has its own pipeline; nothing else in the core changes, and no capability sees RTP or SDP. webrtcbin places an incoming stream by the SSRC the answer names for it (a=ssrc; browsers name it, and fjarr-opsim does as they do): an answer that names none leaves the stream on the first transceiver of the BUNDLE. A stream that matches no uplink is sunk into a fakesink and logged, because an unlinked pad stops webrtcbin’s receive path and the session with it (found building the opsim scenario).
The capability: one speaker, one gate. fjarr.audio owns the robot’s speaker pipeline, appsrc ! audioconvert ! volume ! output, built at talk-start and set to NULL at talk-stop: nothing queued plays on after a release, and the robot’s audio device is free while nobody talks. Each session’s on_pcm pushes into the appsrc only while that session holds the floor, under a mutex held for one 20 ms frame, so a closed gate drops audio on the streaming thread and nothing waits on the core loop. talk-start and talk-stop move the floor on the core loop. The appsrc holds at most 60 ms.
The output. speaker.output = "alsa" with speaker.device (default default) plays through alsasink with a 40 ms buffer, so a release goes quiet well inside docs/06’s 200 ms. "null" decodes, caps and meters, then discards: the demo robot (a container with no sound card) and CI use it. The volume cap speaker.max_volume (0–1, default 0.5) is the volume element’s fixed value: nothing on the wire can raise it.
What the operator sees. on_pcm measures what it pushes, scaled by the cap exactly as the volume element scales it, and every 250 ms the capability sends the talking session talk-level (RMS and peak in dBFS after the cap, and the frames played). It is the operator’s proof that their voice reaches the speaker, and the test oracle.
Grants and audit. The uplink is offered only to a session whose grant says talk and only when a speaker is configured; talk-start from any other session is capability-denied or unavailable. Every opening and closing is session_event("audio-uplink", "talk-start" | "talk-stop") and a log line naming the operator. Session end closes the gate through session_detached, like any other held input (docs/15 safety).
Config: [capabilities."fjarr.audio".speaker] with output, device and max_volume.
Verified (2026-10-12): opsim’s audio-talk against the demo robot (null output, cap 0.5) sends a full-scale sine from before talk-start: nothing of it is played before, 50 frames a second after, peaking at −6.0 dBFS; talk-stop answers in about 2 ms and no level follows. audio-untalked is offered no uplink and refused capability-denied. The floor, busy with its holder, and the gate closing on every session end are unit tests (test_audio_capability).

Rate control and tier switching {#rate-control-and-tier-switching}

docs/16 demands adaptive bitrate; the fan-out demands that one encoder serves every viewer of a tier. The two meet like this (ADR-0007; planned 2026-09-22, slice 6a):

Per peer: an estimate. Each consumer pipeline reads its rtpsession’s twcc-stats (per-packet send and arrival times and losses, from the browser’s transport-wide feedback) into a RateEstimator owned by the session: a delay-gradient trend detector plus a loss rule (loss below 2 % over a window: grow 5 % per second; above 10 %: cut to estimate × (1 − loss / 2); between: hold), bounded to [floor, 1.2 × tier target], with a step down applied within 200 ms of the feedback that caused it. The estimate is per (peer, track).
Per producer: a target inside a band. A tier encoder’s target is recomputed every 500 ms as the minimum estimate among its current subscribers, clamped to the tier’s band [band_low, target], where band_low is half the target for active and the docs/16 floor for thumbnail. The EncoderAdapter applies it live (set_bitrate; an encoder that cannot change bitrate while playing gets a keyframe and a reconfigure, which the adapter reports so the doctor can say so).
Demotion needs evidence, not just a low number. The estimator never believes in more bandwidth than 1.5× what arrived, because an encoder sending less than the estimate proves nothing about the ceiling. The mirror of that rule was missing and cost a real defect (found by the nightly, 2026-09-24): a track whose content compresses well — a static scene, a camera on a white wall, the test pattern — emits far below its target, so the estimate is pinned near what little arrived, falls under the band floor, and the viewer is demoted on a perfect link. An estimate the sender never pushed against is untested, not low. So a viewer is demoted only while the peer is actually sending at or above its estimate (≥ 0.9 × it, over the same feedback window); below that the demotion timer does not even start. Promotion is unaffected: an unexercised estimate that is high is still evidence a viewer can be restored.
Per peer: a tier. A subscriber whose estimate stays below its tier’s band_low for 2 s while that estimate is being tested is demoted to the next lower tier by the agent: its hub subscription moves, a keyframe is requested, bandwidth-stats reports effective_tier below tier. It is promoted back when its estimate stays above 1.2 × the higher tier’s band_low for 5 s. The client’s demanded tier is remembered and never overwritten; the agent’s override sits beside it. Only new feedback moves the clocks: the estimate is ticked every 200 ms but TWCC feedback arrives about once a second, and a tick with none is no evidence either way, so it neither starts, advances nor resets the demotion and promotion timers. (Counting it as “untested” reset the timer four ticks in five, and a viewer on a bad link took twice the 2 s to leave — open question #40.) A viewer belongs to one tier of a track at a time: when it reports a share for a tier, its share in the track’s other tiers is withdrawn at once, and a tier change withdraws it before the next report. A share left behind held the shared encoder at band_low until it aged out after 3 s (#40). A peer that has no lower tier to go to (the thumbnail producer refused for the encoder budget, or a passthrough track without a lower stream) keeps its tier with the per-consumer leaky queue as its only protection, and the track reports adaptive: false.
What this buys. One encoder per demanded tier, none when idle, as before. A lone viewer gets the whole adaptive range down to the floor. A bad receiver among good ones can pull the shared encoder down to band_low for at most 2 s, then leaves for the lower tier alone; the others never see less than half the target. The price is a quality step for the demoted viewer instead of a slide; a middle tier is one more encoder if the demos show the step is too coarse.
Per producer: motion or sharpness (slice 3.5). select-tracks carries a viewer’s preference (docs/08). motion, the default, keeps the frame rate and lets each frame’s quality fall with the bitrate. sharpness keeps each frame’s quality: when a tier’s target falls below its band’s top, its frame rate falls in proportion (videorate max-rate = the tier’s rate × target ÷ top, never below 5), so text stays legible on a slow link and a still screen costs nothing more. A producer serves sharpness while any of its viewers asks for it; a viewer’s preference goes with the viewer.
Where it shows. bandwidth-stats per track (docs/08), the session pipeline’s snapshot (the encoder’s current bitrate property, the estimator’s numbers under session:<sid8>/<track>/rate), /stats, and the web client’s health reasons (“tier reduced by the robot: link 900 kbps”).

Measurement. The loop test streams one 1080p30 source to N consumer branches ending in fakesink for N ∈ {1, 2, 5, 10} and records CPU per process, allocations per frame, and per-consumer delivery latency (hub push → appsrc push); fjarr-opsim repeats it with real webrtcbin answerers on the compose network. The docs/16 line ”≤ 5 % CPU per additional viewer” is asserted on the opsim numbers (the loop numbers must be far below it, since they exclude SRTP), and the trend is plotted per commit so a regression in the hub shows up before a customer’s ten-operator control room does.

Media-plane recovery

A bus ERROR on a consumer pipeline closes that session (media-error). A bus ERROR on a producer is classified by where it came from (slice 4):

Inside the source bin — a camera unplugged, an RTSP camera unreachable, a driver that died: that is the track’s failure. The producer is retried on a slow ladder (1 s → 30 s, for as long as something demands the track), the track is reported unavailable with the bus error as its reason (/sources, the doctor), and nothing else is affected: no session closes, the plane stays up. A source that also reports itself unavailable (a device node gone) parks the retry until it is back; the capability’s hot-plug callback removes and re-adds the track by renegotiation.
Anywhere else — the encode path, the tee, the sinks: the producer is restarted with backoff (0.5 s → 5 s, 5 attempts); subscribers see a gap and a fresh keyframe. If it cannot come back within its budget, or the pipeline fails to reach PLAYING after a rebuild, the media plane escalates: every session closes with media-restart, the plane is disposed and rebuilt from config; the third plane rebuild within 10 minutes exits with code 2.

Each step is a counted metric and a log line with the element name — the camera streamer’s ladder, with the numbers written down, and with the one distinction it lacked: a bad camera never takes the robot’s other cameras with it.

DataChannel router

ChannelRouter attaches to the GstWebRTCDataChannel objects the session created and implements docs/08#datachannel-topology:

Parsing. Text messages on control/realtime are parsed as envelopes and validated (v, cap pattern, kind, object payload); invalid ones are counted and dropped; oversized outbound envelopes (> 16 KiB UTF-8) are refused with an error to the sending capability.
fjarr.core is served by the router itself: ping → pong{ok,t0,t1,t2} immediately from the core loop (t1/t2 = g_get_real_time() / 1000, milliseconds), time-sync → a time-sync result with the same payload. Every ping or time-sync resets the session’s liveness timer.
Track control (<cap>/select-tracks, bandwidth-stats, docs/08#track-control) is served by the router for every capability that declared tracks, so no capability implements it.
Dispatch. cap → the capability attached to this session, on the core loop, with the session’s SessionContext. A capability not attached to the session gets no message; the router answers a request for one the agent has but the grant left out with capability-denied, and one the agent has never heard of with capability-unknown (docs/08#envelope; SessionDeps::known_capability asks the registry).
ChannelSender implementations per class:
- control: reliable ordered; send(Envelope); buffered_amount() from the channel’s buffered-amount property; on_drain from buffered-amount-low with the threshold at LOW_WATER.
- realtime: send(Envelope) on the unordered/unreliable channel; the sender never queues — if buffered_amount() exceeds 64 KiB the message is dropped and counted (newest-wins is the capability’s job; the core never lets a lossy channel grow a backlog).
- bulk (per capability): send_binary(frame) with the docs/08 watermarks (HIGH 4 MiB / LOW 1 MiB); above HIGH it returns false and sends nothing, so the capability pumps on on_drain — unbounded sends are a spec violation, so the sender refuses them. Frames larger than the negotiated SCTP max-message-size (65 536 on webrtcbin 1.24 and 1.28) are rejected with payload-invalid; chunking is the capability’s job (docs/08 blob chunks are ≤ 256 KiB and ≤ the SCTP limit).
- bulk framing (docs/08#blob-frames, slice 5a): a channel declared raw hands every binary message to Capability::on_binary(); one declared blob has its header parsed and checked by the router (version, lengths, offset within blob_len; a bad chunk is counted and its blob discarded) and delivered as on_blob_chunk(). The docs/08 pending store (chunks of blobs no envelope has named: 8 MiB or 30 s per channel, oldest evicted) is BlobAssembler, the helper over on_blob_chunk() for capabilities that want whole small blobs — the router itself keeps nothing. SessionContext::send_blob() is the one outbound pump (BlobPump, one per open bulk channel): it chunks to the SCTP limit minus the header, honours the watermarks, resumes on the channel’s drain signal, reports completion, and fails every queued blob with done(false) when the session closes — before the capabilities are detached. cancel_blob(id) drops one not yet sent.
- DataChannel parameters come only from the class table: control ordered=true reliable; realtime ordered=false, max-retransmits=0; bulk ordered=true reliable; stream ordered=false, max-retransmits=0; protocol empty, ids negotiated by SCTP (never negotiated=true), created in the order control, realtime, bulk…, stream… so the m=application index is stable.
Channel creation per session: control and realtime always; bulk:<cap> / stream:<cap> for each attached capability whose manifest declares them. Reliability parameters (ordered, max-retransmits) are set from the class table in docs/08 and nowhere else.

The concrete `SessionContext` and `BackendContext`

These replace the forward declarations in docs/09 (which is updated in the same commit as this document):

namespace fjarr {

struct TrackSpec {
  std::string track_id;            // stable; camera: the config key; desktop: "desk-<connector>"
  std::string label;
  TrackKind kind;                  // Video | Audio
  SourceRef source;                // a VideoSource (docs/09) + output name; owned by the media plane
  std::optional<MonitorInfo> monitor;
};

class SessionContext {
public:
  const SessionId& id() const;
  const OperatorInfo& operator_info() const;
  const nlohmann::json& granted_params(std::string_view cap) const;

  // Tracks: declared during session_attached (frozen into the first offer)
  // and changeable later (renegotiation, docs/08#renegotiation).
  void add_track(TrackSpec spec);                 // only valid inside session_attached
  void update_tracks(std::vector<TrackSpec> full_set);   // diffed by track_id, coalesced
  TrackState track_state(std::string_view track_id) const;  // enabled, tier, subscribers

  // Channels (docs/08 classes). Bulk/stream senders exist only if declared.
  ChannelSender& control();
  ChannelSender& realtime();
  ChannelSender& bulk();     // this capability's fjarr:bulk:<cap>
  ChannelSender& stream();   // this capability's fjarr:stream:<cap> (docs/08; fjarr.net, M4.5)

  // Correlation helpers (docs/08#envelope) — the only way to answer a request.
  void accept(const Envelope& request);
  void feedback(const Envelope& request, nlohmann::json payload);
  void result(const Envelope& request, nlohmann::json payload);   // payload.ok required
  void fail(const Envelope& request, std::string_view code, std::string_view message);
  void event(std::string_view type, nlohmann::json payload);      // kind=event on control

  // Async I/O on the core loop, never on a thread of the capability's own.
  // watch_readable was the extension API's first amendment (M2, fjarr.terminal's
  // pty); every() its second (M4.5, fjarr.net's per-second link-stats, which must
  // arrive on an idle link too). Both handles stop when destroyed.
  std::unique_ptr<FdWatch> watch_readable(int fd, std::function<bool()> on_readable);
  std::unique_ptr<Timer> every(std::chrono::milliseconds period, std::function<bool()> on_tick);

  // Off-loop work: run `job` on the pool, then `done` back on the core loop
  // — dropped if the session is gone by then (generation-guarded).
  void run_async(std::function<void()> job, std::function<void()> done);

  // Safety helpers (docs/15): a deadman the capability arms per input stream.
  std::unique_ptr<DeadmanHandle> arm_deadman(std::chrono::milliseconds budget, std::function<void()> on_expiry);

  void close(std::string_view reason);   // capability-initiated session end
};

class BackendContext {
public:
  ChannelSender& channel();        // backend-stream envelopes over the signaling connection
  bool online() const;             // signaling connected
  // Store-and-forward (M7): durable event types declared by the capability
  // are queued while offline, bounded per docs/16; everything else is
  // dropped when offline. Slice 3 ships the interface with drop-when-offline.
  void declare_durable(std::vector<std::string> event_types);
};

} // namespace fjarr

Rules: every method is core-loop-only (asserted in debug builds by thread id); add_track outside session_attached throws; result() on an event envelope throws; a capability that never answers a request is a bug the operator sees as a timeout — the core does not synthesize results.

Configuration

AgentConfig::from_file reads TOML (toml++, MIT — docs/14) with FJARR_* environment overrides winning:

[agent]
robot_id        = "robot-024"
server_url      = "wss://fjarr.acme.com/ws"
credential_file = "/etc/fjarr/device.key"  # M5 device key; until then the dev token:
dev_token       = ""                       # or FJARR_DEV_DEVICE_TOKEN (env wins); never logged
ice_policy      = "all"                    # all | relay (relay-only for tests)
log_level       = "info"                   # trace|debug|info|warn|error  (FJARR_LOG_LEVEL)
log_format      = "text"                   # text | json                  (FJARR_LOG_FORMAT)
dot_dir         = ""                       # FJARR_DOT_DIR: snapshots as files (docs/24)
watchdog_secs   = 0                        # 0 = WatchdogSec/3 from the unit; the packaged agent refuses to start outside systemd unless `allow_unsupervised = true` (containers, dev)
allow_unsupervised = false

[media]
encoder        = "auto"                    # auto | vaapi | software — never a silent fallback (below)
gop_seconds    = 2                         # keyframe interval; ring and consumer queue are sized to it
active_kbps    = 4000                      # the active tier's target; the encoder adapts in [active_kbps/2, active_kbps] (rate control, above)
active_floor_kbps = 250                    # a lone viewer may take the encoder down to here (docs/16)
thumbnail_kbps = 300                       # the thumbnail tier's target
tier_grace_ms  = 10000                     # producer lingers this long after the last demand

[introspect]                               # docs/24
enabled      = true
bind         = "127.0.0.1"                 # 0.0.0.0 requires `token`
port         = 7381
socket       = ""                          # Unix socket path instead of TCP
token        = ""
history      = 64                          # snapshots kept per pipeline

[capabilities."fjarr.test"]                # docs/06 — enabled by the install; hooks off in production
enabled    = true
test_hooks = false

[capabilities."fjarr.camera"]              # validated against the capability's JSON Schema
enabled = true

[capabilities."fjarr.camera".tracks.front]
label    = "Front"
source   = "v4l2src device=/dev/v4l/by-id/usb-Acme_Cam-video-index0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegdec"
required = false                           # true: a missing driver is a startup error (ADR-0020)

[capabilities."com.acme.arm-teach"]
enabled = true
privileges = ["fs-read:/var/lib/acme"]     # must match the manifest's requests

FJARR_<SECTION>_<KEY> environment variables override any key (FJARR_AGENT_SERVER_URL, FJARR_INTROSPECT_PORT); the short forms named in the comments are aliases kept for the docs and the compose files.

Encoder policy. auto picks VA-API when the doctor’s vah264enc smoke passes and otherwise refuses to start (exit 1) with the message that names the two options; software selects openh264enc (BSD, docs/14) explicitly — for CI, the devcontainer without /dev/dri, and robots that knowingly trade CPU for portability — and the doctor WARNs while it is in use. There is no runtime fallback from hardware to software: a robot that silently starts burning a core is the failure mode this rule exists to prevent.

Media-plane constants (initial values, tuned by the latency harness):

Constant	Value	Where it applies
GOP	`gop_seconds` × fps (2 s → 60 frames at 30 fps), one keyframe per GOP, no B-frames	encoder config, ring size (GOP + 1), consumer queue
Consumer queue	`queue leaky=downstream max-size-time=<GOP> max-size-buffers=0 max-size-bytes=0`	per track per session
`appsrc`	`is-live=true format=time do-timestamp=false block=false max-bytes=0`	per track per session
PTS rebase	first delivered keyframe’s PTS becomes 0; every later PTS and DTS get the same offset; durations unchanged	FrameHub delivery
Payload types	allocated per session from 96 upward in manifest order (video first, then audio); SSRC left to the payloader (random)	offer builder
Realtime drop threshold	64 KiB buffered	realtime `ChannelSender`
Keyframe request rate limit	≥ 1 s per producer	FrameHub
Producer restart backoff	0.5 s → 5 s doubling, 5 attempts	media plane
Plane rebuild escalation	third rebuild within 10 min → exit 2	media plane

Timers (all GLib sources on the core context, all generation-checked):

Timer	Period	Owner
Operator liveness	dead after 15 s without `ping`/`time-sync`	Session
Negotiation watchdog	15 s without a milestone, `building`/`offered` only	Session
`bandwidth-stats` + `get-stats` sample	1 s	Session
Counters log line	60 s	Core
Tier producer grace	`tier_grace_ms`	MediaPlane
`sd_notify` WATCHDOG=1	`WatchdogSec/3`	Core
Signaling backoff	docs/08	SignalingClient

The core validates each [capabilities.X] table against X’s config_schema (nlohmann json-schema-validator, MIT) before configure(); an unknown table is an error (exit 1), as is a grant naming a capability whose table has enabled = false (rejected per session, docs/10).

Observability

Logs: one line per event on stderr: <ISO-8601 ms UTC> <LEVEL> <component> <message> key=value… (values quoted when they contain spaces); log_format = "json" emits the same fields as one JSON object per line. Every line inside a session carries session=<id>; milestone lines carry ms=<elapsed since attached>. Credentials, grants and TURN passwords are never logged at any level.
Pipeline introspection (docs/24): snapshots (DOT, JSON, summary) at every milestone into a per-pipeline ring, served live by the local endpoint and the viewer, written to dot_dir when set, exposed over the session by fjarr.introspect — the camera streamer’s best debugging tool, made a product feature and a test oracle.
Counters (exposed later via observability, M7; for now logged every 60 s): sessions started/ended by reason, negotiation timeouts by milestone, dropped envelopes by cause, producer restarts, plane rebuilds, keyframe requests.
SessionEvent to the embedder: started, ended{reason}, error{code}, audio-uplink, with operator label — the audit hook of docs/10.

Safety behaviours (docs/15)

release_all_input() on every input-bearing capability is the first action of closing, on every path (operator close, peer-gone, heartbeat, media error, watchdog, signaling loss, process stop via SIGTERM).
A capability that consumes realtime commands arms a deadman (SessionContext::arm_deadman); the core fires on_expiry on the core loop when no feed() arrived within the budget, and again on session end. The teleop test capability in slice 3 uses it so the behaviour has a test before any real actuator exists.
SIGTERM: sessions close with session-close(reason="agent-shutdown"), input released, then exit 0 — bounded to 3 s, after which the process exits anyway.

Memory and lifetime discipline, and the tooling that enforces it

A C++ wrapper over a C object system fails in three ways — a missed unref (leak), an extra unref (use-after-free), and the wrong thread (data race) — and each has a tool that catches it before a robot does. Four layers, every one with a command an engineer or an AI agent runs and a text+JSON result it reads (docs/15):

Compile time. Every GObject/GstMiniObject lives in the RAII kit (GstElementPtr, GstSamplePtr, GstPromisePtr, SignalConnection, PadProbe, SourceGuard for GLib sources…), [[nodiscard]] on every factory. A /verify grep gate refuses raw g_object_unref, gst_object_unref, g_free, g_source_remove, g_signal_connect outside agent/src/core/glib/ — the kit is the only place that may touch reference counts. clang-tidy runs bugprone-*, cppcoreguidelines-owning-memory, misc-unused-*, -Werror.
Sanitizers, three presets, all in CI. asan (Address + Undefined + Leak, with lsan.supp for GLib’s intentional static allocations), tsan (the threading model’s proof: FrameHub’s lock, post_to_owner, the worker pool — a data race in the media plane is a TSan failure, not a Heisenbug), and release. Unit and loop tests run under all three; ASan and TSan are gates.
GStreamer’s own accounting, in every test and every scenario. The leaks tracer (GST_TRACERS="leaks(filters="GstElement,GstPad, GstBuffer,GstSample,GstPromise",stack-traces-flags=full)") is loaded by the test binaries and by the agent when GST_TRACERS is set (make agent-leaks does that for the demo robot); its action signals (activity-start-tracking, activity-get-checkpoint) bracket every test case and every fjarr-opsim scenario, so “created but never freed since the checkpoint” is an assertion, per test, with stack traces. GST_DEBUG=GST_REFCOUNTING:7 and GOBJECT_DEBUG=instance-count (where the platform GLib supports it) are one environment variable away; the doctor reports whether they are available.
Runtime census, always on. The RAII kit counts live wrappers by type at negligible cost (ObjectCensus: elements, pads, samples, promises, sources, sessions, pipelines), the FrameHub reports buffers held per ring, the channel senders their buffered bytes; the introspection endpoint serves it at GET /memory with RSS and a ?since=<checkpoint> diff (docs/24). The soak scenario (200 connect/disconnect cycles through fjarr-opsim) asserts the census returns to baseline and RSS growth stays inside the docs/16 budget — the leak test that runs against the real code path, not a mock.

Two more tools run nightly rather than per commit: valgrind memcheck on the loop tests (G_SLICE=always-malloc G_DEBUG=gc-friendly, GLib’s glib.supp, GStreamer’s gstreamer.supp and fjarr.supp) for the errors sanitizers structurally miss, and heaptrack on a streaming scenario for the hot-path allocation budget (docs/16): steady-state pushes through FrameHub and appsrc allocate one metadata-only GstBuffer header per subscriber per frame — the fan-out’s per-subscriber PTS rebase, accepted by design in slice 3c planning — and nothing else beyond the refs GStreamer itself takes.

The agent-first surface is deliberately small: make agent-test (release), make agent-test-asan, make agent-test-tsan, make agent-leaks SCENARIO=<opsim scenario> (prints the leaks-tracer diff), make agent-memcheck (valgrind), make agent-heaptrack, and curl localhost:7381/memory — each printing a one-screen verdict and writing JSON next to it.

Testing (docs/15)

Unit (GoogleTest, make agent-test, run under the ASan preset in CI): Backoff; generation-guarded post_to_owner (a torn-down session’s callback is dropped; a live one runs on the loop thread); FrameHub (keyframe gate, PTS rebase, ring overrun skips to the next keyframe, fan-out to N subscribers, producer restart); envelope parsing/validation against the golden fixtures in protocol/fixtures (all three tiers share them); ChannelSender watermarks with a fake channel; manifest builder; the renegotiation queue’s coalescing (three update_tracks while one offer is in flight → one further offer, manifest_version 2 then 3); config loading and schema validation; the milestone watchdog.

Loop tests (GStreamer in-process, no network): a videotestsrc producer through the real encoder path into the FrameHub and into a consumer branch ending in fakesink, asserting caps gating, valve toggling and keyframe-on-enable — the slice’s equivalent of the web mock.

Integration (fjarr-opsim): a C++ operator simulator built on the spike’s second webrtcbin plus a libsoup client speaking the operator side of docs/08. It connects through the real fjarr-server, requests the test capability, answers the offer, checks media arrives (buffer counts per track), drives select-tracks, ice-restart, hot-plug via a test hook, and injects the docs/15 faults: socket drop, silent operator (no pings), answer never sent (watchdog), SIGSTOP of the agent, TURN-only. It is the CI substitute for a browser until slice 5’s Playwright run, and remains the fault-injection tool afterwards.

Budgets: the loop test records CPU per additional consumer (docs/16 ”≤ 5 % per viewer”) on the CI runner as a trend, not a gate, until the latency harness (slice 7) exists.

`fjarr-opsim`: the operator simulator

A C++ binary in agent/tools/opsim/ (package fjarr-tools) that speaks the operator side of docs/08 against the real fjarr-server, answers with its own webrtcbin (the spike’s answerer, so its pad-added/src_N quirks are known), and runs named scenarios:

fjarr-opsim --server ws://fjarr-server:8080/ws --robot demo-robot-01 \
            --grant-secret "$FJARR_GRANT_HS256_SECRET" \
            --scenario <name> [--json out.json] [--timeout 60]

It mints its own operator grant (HS256 over the dev secret with GLib’s GHmac, the same claims the demo-backend uses) so it needs no browser and no backend.

What the simulator may not judge: rate control under jitter. Its webrtcbin receiver counts packets that arrive after it has sent feedback as lost. Measured 2026-09-24 on the wifi-ok profile, which injects 3 ms of jitter and no loss at all: the agent read loss of 34–51 % with a carried ratio of 0.47–0.57, and cut its estimate from 4 Mbps to 1.6 — correctly, because from where the agent sits half of what it sends is not arriving. A Chromium receiver does not do this under the same profile. So any netem profile with jitter records the rate rather than asserting it, and tests/stack/ratecontrol.spec.ts is where rate control is judged. The simulator is still the right tool for everything that does not depend on the receiver’s own loss reporting. Scenarios, each with the assertions it makes and the state it expects the agent to end in:

Scenario	Drives	Asserts
`smoke`	connect, `select-tracks` on, first frame, `echo`, close	media on both tracks’ stamps advance; echo round trip; `session-close` reaches the server
`toggle`	enable/disable `test-pattern` 20×	valve state via `/pipelines/session:<id>.json`; keyframe on every enable; no frames while disabled
`hotplug`	`hotplug{plugged:true}` then `false` while streaming	re-offer with `manifest_version` 2 then 3; `test-pattern` stamp counter has no gap > 1 frame; `test-second` flows then disappears
`rejected-track`	`hotplug{plugged:true}`, answer rejecting `test-second`’s m-line (port 0, out of BUNDLE), `select-tracks` on it, 1.5 s, then `hotplug{plugged:false}`	the enable is refused `payload-invalid` (“rejected by this peer’s answer”); the unplug is answered and re-offered promptly and `echo` still answers — before the valve rule the accepted enable parked a thread on the never-released pad and the unplug hung the core loop; `test-pattern` has no gap > 1 frame
`silent-operator`	stop sending pings	agent closes with `session-close{reason:"heartbeat"}` within 15–20 s; `release_all_input` observed as `deadman{expired}`
`no-answer`	never answer the offer	`session-close{reason:"negotiation-timeout:offer-created"}` at 15 s
`socket-drop`	drop the operator’s socket mid-stream	agent gets `peer-gone`, closes the session, census returns to baseline
`ice-restart`	send `ice-restart`	`session-close{reason:"ice-restart", retry:true}` within 100 ms; new session brokered and streaming
`deadman`	`drive` at 20 Hz, then stop	`deadman{state:"expired"}` within 600 ms of the last `drive`
`relay-only`	`--ice-policy relay` on both sides	media flows through coturn (relay candidates in both stats)
`soak`	N connect/stream/close cycles (`--cycles`, default 200; CI runs 20 per commit, 200 nightly and at the 3c gate)	`/memory` census equal to the baseline checkpointed after a warm-up of 40 cycles (a fifth of a shorter run) — the bounded snapshot history and log ring fill during it — RSS growth < 5 MB after it (the median of the last quarter of the cycles against the first quarter’s: two single readings swung by 6 MiB on one commit), no `error` counters
`netem-<profile>`	applies a docs/25 profile on the robot’s egress (`make opsim-netem NETEM_PROFILE=…`: netem on `eth0`, the introspection port exempt; opsim runs in `dev`, so the impairment is one-directional like a browser’s downlink — impairing `lo` for an opsim inside the robot shaped both directions through one queue and let feedback-path jitter read as loss, slice 6a), runs the smoke assertions with per-profile tolerances	frames keep arriving; health-relevant stats recorded. `wifi-ok`/`4g`/`lossy` run nightly; `bad` (15 % loss, 1.5 Mbit) needs loss recovery and adaptive bitrate (slice 6) and is expected to fail until then
`congested-viewer`	streams the test pattern at the active tier; expects `bad` applied toward it before it starts and cleared while it waits (driven by `tests/stack/ratecontrol.spec.ts` from `dev` with `netemToward`, slice 6a gate 2)	demoted to the thumbnail tier within 25 s (a session that starts behind the bad link first waits 3–4 s for TWCC to carry bitrates; the browser gate measures an established session’s reaction), frames keep arriving while demoted, promoted back within 60 s of the demotion, 20 frames within 8 s at the active tier

Output: a one-screen verdict per assertion (PASS/FAIL name: detail) and, with --json, the same as data plus the captured signaling and envelopes; exit 0 on all pass, 1 on any fail, 3 on timeout. A check that cannot run where it runs, because no fixture service provides it (desktop-control’s mode change without --desktop-plug), prints NOTE name: not run: why. The SUMMARY line and the JSON’s not_run name it again, and it changes no exit code. A target that needs the check looks for its PASS itself (make x11-e2e), because a check left out silently reads as a pass (docs/30, principle 8). make agent-leaks SCENARIO=<name> runs a scenario under the leaks tracer and prints the diff.

Slices 3a, 3b, 3c, 4, 5 and their gates {#slices-3a-3b-3c-and-their-gates}

Slice 3 as first written bundled four reviewable deliverables; it lands as three increments, each on main with its own retrospective review (docs/20), in this order:

3a — browser lab and loopback (web only, no C++ dependency): the browser compose service and lab profile, @fjarr/e2e with the stack/dashboard/cdp/loopback fixtures (the stack fixture tolerates a missing introspection endpoint until 3b), the wire tap in @fjarr/core, LoopbackAgent, the frame-stamp reader, fjarr-lab’s core commands, make lab-up/e2e, the CI job — plus a half-day spike running the existing webrtcbin probe’s offerer against the lab’s Chromium as answerer for the spike’s Q1/Q3/Q6, whose report attaches to ADR-0007. Gate: the slice-2 <VideoTile>/<VideoGrid>/push-to-talk suites pass in real Chromium against LoopbackAgent; the client ladder’s signaling rungs run under the CDP offline profile; the Chromium-answerer spike report exists. Met 2026-09-19 (make e2e: the loopback and stack projects; the ladder over real fjarr-server sockets found four defects the mock could not — docs/25).

3b — agent core, fjarr.test, fjarr-opsim: everything in this document through “Testing”, the docs/09 headers, the introspection walker with dot_dir and a minimal endpoint (/pipelines, /pipelines/<id>.{json,txt,dot}, /sources) with introspect.schema.json under the conformance gate, the ASan gate, TSan and the leaks tracer as trends, and the minimal demo wiring the gate needs: fjarr-server configured from the environment, the demo-backend minting real HS256 grants that list fjarr.test, demo-robot registering fjarr::TestCapability through the public API. Gate: (1) docker compose --profile demo up: the demo dashboard shows the test pattern within the docs/16 startup budget, select-tracks toggles it, <ConnectionQuality> reads real getStats; (2) every fjarr-opsim scenario except soak and netem-* passes in CI; (3) the lab’s ladder scenarios run against the real agent with the agent’s fault switches, the session-close{retry:true} rung included, and the frame stamp proves zero dropped frames on the untouched track during hotplug; (4) unit + loop tests green under ASan; the release_all_input and deadman regression tests exist; the headers match docs/09. Met 2026-09-20 — web/e2e/tests/stack/{agent,dashboard}.spec.ts (first frame in the dashboard 1.3 s after connect; hot-plug with max stamp gap 1; deadman 497 ms; session-close{retry:true} on ice-restart; the introspection JSON validated against the schema), make opsim-all, make agent-test-asan, agent/tests/test_session.cpp.

Implementation notes (slice 3b) — where the code settled something the text above left open, or learned from the lab:

Caps for the offer come from the transceiver’s codec-preferences, not from a data-driven pad probe: a disabled track pushes no buffers, so the payloader’s caps would never fix; the wrapper sets application/x-rtp, media=video, encoding-name=H264, payload=<pt>, clock-rate=90000, packetization-mode=1 on each transceiver at pad request time and the caps-fixed milestone is immediate. The pad probe on the payloader still records the negotiated caps for introspection.
Tier profiles: active is the source size at ≤ 30 fps, thumbnail is half the source capped at 960×540 at 5 fps (fjarr.test’s 1280×720 source therefore gives the docs/06 640×360). GOP = gop_seconds × fps.
Transport errors are not media errors: a bus ERROR from the SCTP/DTLS/ICE elements (the peer went away) closes the session with ice-failed; only errors elsewhere in the consumer pipeline are media-error. The lab found this when the client’s heartbeat tore its peer down and the agent blamed its own media plane.
The SSRC is signalled from the first offer. Without ssrc in the transceiver’s codec preferences, webrtcbin signals no a=ssrc for a track that has not flowed yet and a fresh one once it has; Chromium then recreates its receiver on the re-offer and loses a GOP on an untouched track during hot-plug (found by the lab as a 30-frame stamp gap with the browser’s inbound counters resetting). The payloader’s SSRC therefore goes into the preferences with the payload type.
PLI/FIR from the peer reach the producer. webrtcbin turns them into upstream force-key-unit events that would die at the consumer’s appsrc; a probe there relays them through the hub to the producer’s encoder (rate-limited and deferred by the media plane).
“Zero dropped frames” is measured at the receiver’s decoder: across a hot-plug the browser’s framesDropped and packetsLost for the untouched track stay at zero and framesDecoded keeps climbing, while the frame-stamp counter (read per presented frame) may skip a few presentations when Chromium applies the new remote description and starts a second software decoder — a presentation artifact of the browser on a shared host (1 frame on an idle machine, 3–4 on a loaded one, measured identically on the 3b and 3c agents), not a lost frame. The lab test asserts the wire-level counters exactly and budgets presentation at 6 frames; fjarr-opsim’s hotplug asserts a wire-level stamp gap ≤ 1 with a GStreamer receiver.
Snapshot coalescing is trailing-edge: a trigger inside the 250 ms window is deferred to the window’s end, never dropped, so the served snapshot is always the latest state within a quarter second.
Control domains (docs/10, revised 2026-09-28): every session is accepted with input allowed by its grant; nothing is claimed at open. The SessionManager keeps one claim per control_domain from the capability manifests. A session’s first input to a capability with a domain claims it if free. A non-holder’s events are dropped and counted, and its requests are answered control-held. take-control and release-control are served by the core under fjarr.core. desktop frees 5 s after its holder’s last input; motion only on release, disconnect or takeover, and a takeover runs the capability’s release_all_input first. Claims fail open after 30 s without the holder’s heartbeat. Capabilities without a domain (terminal, net, files) are never gated. As built: the rules live in ControlDomains, pure bookkeeping with the time passed in, and the SessionManager turns each change of holder into releases and control-state. A manifest’s control_inputs names the message types that are input (fjarr.test: drive), so echo never claims motion. The old holder’s release_all_input runs on every change away from them — takeover, release, stale — in desktop too, so no key stays down; a desktop freed by idleness is released only when someone else claims it, since the idle holder may be mid-drag. Any ping from any of the operator’s sessions is the heartbeat. view_only input is dropped, or answered capability-denied, and never reaches the SessionManager.
Closing has a flush window. release_all_input runs synchronously and what it emits on fjarr:control (deadman{expired}) must reach the operator; a NULL state change in the same loop turn discarded it. The session therefore detaches, tells the operator (session-close) and closes its valves at once, drops every inbound envelope from then on, and tears the consumer pipeline down 150 ms later (close_all — shutdown, socket loss, plane rebuild — skips the window). Track registrations on the plane are counted, because a reconnecting operator registers the same track id before the old session’s deferred close unregisters it.
The watchdog names only peer-facing milestones: local-description-set and ice-gathering-complete neither re-arm it nor name the timeout, so an unanswered offer closes as negotiation-timeout:offer-created.
A late joiner gets the ring, not just the keyframe: the hub delivers the whole current GOP (keyframe first, then its deltas, every one decodable in order) to the new subscriber only, deduplicated by a per-hub sequence number against frames still queued for fan-out; a ring that overflowed past its keyframe is no catch-up and the joiner waits for the requested one. The PTS base is set once per subscriber, so a resync keeps the RTP timeline monotonic.
A second tier starts downstream-first: the branch is synced sink → encoder → queue and only then linked to the tee; the other order let a PLAYING queue push into a not-yet-READY encoder bin, take FLUSHING and park the tee for good with no bus error (a regression test starts the thumbnail tier on a running producer).
A removed track’s branch goes; the transceiver stays. Parented elements cannot be renamed, so a re-added track rebuilds appsrc ! queue ! valve ! payloader under its own name and relinks it to the pooled sink_%u.
The input lease ends with the owner’s last session (docs/10): nobody waits out the 30 s fail-open after a clean close. Promotion of an already-open read-only session when the lease frees is not implemented: it reconnects.
Signals are loop callbacks: Agent::stop_on_signal(SIGTERM) installs a g_unix_signal source on the core context; shutdown never runs in async-signal context and Supervision::stop_deadline_ms bounds it. On shutdown the WebSocket close handshake is pumped (≤ 300 ms) so operators see session-close{agent-shutdown} rather than peer-gone.
TSan sees the kit’s hand-offs. GLib and GStreamer are not built with ThreadSanitizer, so a closure handed to the loop, a thread-pool job, a promise or a probe would look like a race between its two ends; the kit pairs every such hand-off with a release/acquire on one atomic (glib::handoff_release/acquire), and tsan.supp names the uninstrumented modules. With that, the 3b tests run clean under TSan and it is a gate (docs/15).
Nothing inbound may throw through GLib: the router, the signaling hooks and the RAII kit’s source trampolines catch std::exception (any granted operator can send {"tracks":[1]}); inbound envelopes above 16 KiB are dropped and counted, error messages echoing input are clamped.
Deferred to M4: the stream sender (ADR-0018), the backend bus, a producer bus-error → restart test under ASan, the duplicate deadman{expired} a capability emits before the core’s own expiry.

Implementation notes (slice 3c) — the memory ladder as built:

The leaks tracer is read through a ledger, never through get-live-objects. Its activity-get-checkpoint window lists what was created and what was removed since the last read and resets on every read, and its get-live-objects signal takes the listed objects’ references (it exists for process exit). The agent therefore keeps a process-wide multiset of created − removed, merged on every read; a checkpoint records that multiset and /memory?since= reports the difference, so reads are idempotent and any number of readers agree. make agent-leaks-selftest proves the bracketing fires on a deliberate leak — a gate that cannot fire is no gate (the ctest lesson of 3b).
valgrind needs three suppression files: GLib’s installed glib.supp, GStreamer’s gstreamer.supp vendored from the 1.28.2 tree, and agent/tests/valgrind/fjarr.supp for what those miss — GLib’s per-thread main-loop and source-attach bookkeeping, GIO module unloading, one GLib constructor allocation, and realloc(p, 0) in a JavaScript engine a plugin’s helper process loads. Every entry is library-internal by construction; a definite loss with a fjarr frame is never listed.
heaptrack found the one per-frame allocation the budget forbade: the delivery thread built its target list in a fresh vector per frame. It is now a member reused across frames; the remaining per-frame allocations are the buffer headers the budget allows (docs/16).
/events is one chunked libsoup response per client: our reference on the message and its finished signal define the client’s lifetime, a 15 s keep-alive comment keeps proxies from timing it out, and a client more than 1 MiB behind has its stream completed (its retry reconnects, Last-Event-ID replays what the ring still holds).
webrtcbin’s get-stats is not used. In GStreamer 1.28.2 its _get_data_channel_transport_stats obtains the RTP session element from rtpbin’s get-session (a new reference) and never releases it, once per call: a session sampling stats every second kept one reference per second after closing, about 18 MB per session in the lab. The stats sampler reads rtpbin’s own per-source counters (octets-sent, packets-sent of the internal sender sources) synchronously on the loop instead — the soak’s RSS budget is what caught it. Reported upstream; the workaround stays until the baseline carries the fix.
A stamped source stays on system memory. The frame-stamp painter maps every raw frame for writing; a VA-API branch proposes its own buffer pool upstream through the ALLOCATION query (system-memory caps, VA-backed buffers), and a write-map of such a buffer is a GPU round trip per frame. On a GPU at full clock the round trip hid inside the frame budget; at its floor clock the producer fell to 13 fps while the same chain without the painter did 29, which is how slice 3c found it. The producer answers the allocation query at the tee for stamped (fjarr.test) sources so the source allocates ordinary memory and the encoder branch uploads, as it does for any camera. Real sources are never stamped.
The log ring keeps info and above regardless of the configured level, so a bundle from a warn-level robot still shows what happened.

Implementation notes (slice 4) — sources as built:

Decodebin-based bins get a late ghost pad. The parser’s ghost-unlinked-pads would ghost decodebin’s internal typefind pad (and break rtspsrc’s delayed link), so rtsp and the sizeless v4l2 auto form parse without ghosting and add a targetless src that decodebin’s first video pad targets on pad-added — and re-targets after a NULL→PLAYING cycle, because decodebin rebuilds its pads and a stale target starved the second start of a producer (found by the lab).
Two hardware encodes at the iGPU’s floor clock do not make 30 fps each: with the 3c throttle still on this host, three viewers on two VA-API tracks presented 3–16 frames per two seconds while the software path streamed 30 to every viewer. The lab’s camera tests are gated on CI’s software path; VA-API multi-track throughput is a GPU-runner measurement (docs/12).
/sources lists configured sources before any session (Capability::configured_sources()), with each source’s own reason for being unavailable; the plane adds caps and tiers once a session registers the track.

Implementation notes (slice 5a) — blob frames and fjarr.introspect as built:

The first chunks leave on the next loop turn. send_blob() queues the transfer and posts the pump, so the capability’s synchronous code sends the referencing envelope first, as docs/08 asks — and a done() that runs inside the pump can enqueue the next blob without re-entering it.
Newest-wins is the capability’s, not the pump’s. fjarr.introspect counts blobs in flight per (session, pipeline); a snapshot that arrives while one is pumping is held (only the newest), and sent when the pump reports completion. The pump never drops, so a subscriber never sees a reference whose bytes were withdrawn.
The built-in is registered in the agent’s constructor with the store and the stats provider resolved lazily, because the rings exist only once the agent boots; the capability adds its store listener on the first attach. The SnapshotStore grew a listener list (the endpoint’s /events and the capability read the same ring).
On the web the receiver lives on the channel set, not the sender: session.bulk(cap) hands out a fresh sender per call, and a blob that arrives before anyone asked must survive that. It gives up its waiters when the channel closes or the peer is gone.
A blob chunk is 65 499 bytes: webrtcbin’s SCTP message limit minus the 37-byte header. A 87 KiB session DOT is two chunks.
The lab found nothing wrong in the agent this time; both defects it surfaced were in the test (producers are listed only once something streams; the error code travels on the error object, not in its message). The dashboard test renders the session graph with d3-graphviz in real Chromium and counts its SVG nodes — the “one viewer” gate’s first half, the served viewer being 5b.

3c — introspection completeness and the memory ladder: /events, the history ring and scrubbing, /stats, /memory with checkpoints, the diagnostics bundle (with the agent’s in-memory log ring, docs/24), make introspect and fjarr-lab introspect; the leaks tracer bracketing promoted to a gate, the soak and netem-* scenarios, the nightly workflow (valgrind, heaptrack, the full soak) prepared to run on a self-hosted GPU runner when one is registered (docs/12). TSan became a gate at the end of 3b. The bundled viewer moves to slice 5 so it is built once as <PipelineGraph> and served from GET / as a data file. Gate: docs/24 acceptance minus the viewer, via curl and fjarr-lab; the docs/15 memory rows green; the 200-cycle soak returns to the census baseline. Order settled in planning: 3c precedes slice 4 because real capture sources are where leaks and races surface, and the ladder must be in place to see them.

4 — fjarr.camera on real sources: the built-in capability whose config is a list of tracks referencing sources (docs/06), the v4l2 and rtsp source types beside gst and test, the SourceFactory seam in configure(), hot-plug of a v4l2 track from the /dev/v4l/by-id watch, required = true as a startup error, fjarr-agent --probe-source for every type incl. the memory type, a doctor/--check row per configured source, and the demo robot exposing fjarr.camera (a pattern track, an RTSP track from the lab’s RTSP simulator, the host webcam through an opt-in compose override). Gate: (1) three lab pages watch two fjarr.camera tracks of the demo robot concurrently (docs/06 acceptance) and a select-tracks toggle takes effect < 500 ms without renegotiation; (2) the RTSP track streams in CI from the simulator; a v4l2 track with no device is unavailable with its reason in /sources and absent from the manifest, and appears (and leaves) by renegotiation when the device arrives (unit tests on a temporary by-id directory and with a recording session context; for real only on bare metal — a container’s devices: cannot hot-plug); an unreachable RTSP camera degrades its own track only; (3) --probe-source reports caps, memory type and fps for test, gst, rtsp and a missing v4l2 device; (4) every 3b/3c gate still green, the soak included. Loss recovery, adaptive bitrate and elementary-stream passthrough are slice 6.

5 — demo wiring, planned 2026-09-21 as two increments, each with its own review:

5a — the protocol half. docs/08 blob frames on both tiers: the manifest’s bulk framing (raw / blob), the router’s header parsing, pending store and on_binary() / on_blob_chunk() dispatch, SessionContext::send_blob() and BlobAssembler; session.bulk(cap) .receive(ref) / .onChunk() in @fjarr/core with the same bounds; the blob-ref schema fragment under the conformance gate. On that: the fjarr.introspect capability (docs/24 message table incl. pipelines/snapshot, txt inline, json/dot always as blobs, newest-wins per pipeline under backpressure, grant-gated); the pipeline feeds (sessionPipelineFeed, httpPipelineFeed on streamed fetch) and their contract test; usePipelines / usePipelineSnapshot / usePipelineFeed / <PipelineGraph>; the demo backend’s operator/developer roles and the dashboard’s role picker; the Diagnostics tab. Gate: (1) docs/06 acceptance for fjarr.introspect in the lab: a developer-role page watches the session graph change while toggling a track and firing the hot-plug hook, an operator-role page gets capability-denied; (2) the feed contract suite passes against both feeds; (3) blob failure modes have tests on both tiers — chunks before the envelope, envelope before the chunks, a bad header, a blob that never completes, a channel that closes mid-blob, a subscriber above HIGH_WATER receiving only the newest snapshot; (4) the lab’s introspect command runs on the HTTP feed; (5) every earlier gate still green, the soak included (blob traffic must not move the census). Met 2026-09-21 (review).

5b — the delivery half. introspect.viewer_dir and the static serving rules (docs/24), the viewer app in the web workspace on the HTTP feed, the demo publishing the endpoint on host loopback with the dev token, make introspect opening it; docker/agent/Dockerfile (viewer, build, runtime and demo stages; the runtime stage fails on x264enc), the docker-compose.image.yml override and the CI job that builds the image and runs the lab smoke against it (docs/12). Gate: (1) the viewer opened by make introspect renders the demo robot’s session graph, and a lab test asserts the rendered nodes; (2) a request without the token is refused and one with it served; (3) serving tests: index at the root, traversal refused, a missing file a 404, API routes shadowing files, a robot without the directory still answers the text index; (4) CI builds the image, the image-based demo passes the lab smoke, the image contains no x264enc; (5) nothing is published. Met 2026-09-21 (review).

Implementation notes (slice 5b) — the served viewer and the image as built:

The endpoint decides static-or-API before the token. is_api() names the docs/24 routes; everything else is serve_static(): the file under viewer_dir resolved with weakly_canonical and required to stay under the directory (a symlink out of it is a 404, .. never reaches the filesystem), read whole, typed by extension, Cache-Control: immutable only for assets/*-<hash>.<ext>. GET / without a directory is the text index naming the key. The API shadows files by construction.
The HTTP feed did not seed “latest” from the list — the viewer’s first real run showed a session pipeline with no graph: its snapshots predated the SSE stream, and only the session feed’s replay had hidden the gap. setList() now seeds every pipeline’s snapshot store from its row on both feeds, and the contract suite reopens a feed on existing state.
The image builds from the repo root with a .dockerignore: a node stage builds the viewer (the lockfile’s whole importer set is copied as manifests so --frozen-lockfile holds), a build stage configures CMake directly (the presets want ccache), the runtime stage asserts the absence of x264enc at build time. The compose override uses !reset to drop the dev image’s mount and bash wrapper.
fjarr-opsim reads the token from its environment, so the make targets and CI needed no new arguments; the Makefile includes .env for the same token in its curls.
The simulator’s teardown crashes on the hosted runner are an upstream race. Two CI runs died after every check of a scenario had passed — a segfault in no-answer’s teardown, then the glibc tpp.c:83 assertion in ice-restart’s reconnect. fjarr-opsim under ThreadSanitizer against the demo robot reports data races inside OpenSSL between webrtcbin’s nicesrc and queue streaming threads during the DTLS handshake (BUF_MEM_grow, CRYPTO_clear_realloc, ASN1_STRING_set; the two threads hold disjoint mutex sets), i.e. GStreamer’s dtls elements driving one SSL object from two threads without a common lock. Heap corruption from that surfaces at free time, which is teardown, and the runner’s timing hits the window far more often than this host does. Until the baseline carries a fix, make opsim-all retries a scenario once when the simulator dies of a signal (a failed check still fails at once), and the agent’s own soak, which runs the same elements for 200 sessions, stays the watch for the same corruption on the robot side.

6 — loss recovery, rate control, passthrough, planned 2026-09-22 as two increments, each with its own review (decisions: ADR-0007 closed on webrtcbin with our own estimator; one thumbnail tier, on demand as today; NACK/RTX plus keyframe requests, no FEC; passthrough by config):

6a — repair and rate control. The offer carries transport-cc, nack/rtx and the keyframe feedback (docs/08#rtp-feedback; the transceiver’s do-nack, the RTX payload, the TWCC extension — a probe against the lab’s Chromium first, as the webrtcbin spike’s Q7: the cadence and content of twcc-stats with a browser receiver, and whether vah264enc / openh264enc take a bitrate change while playing). Then the RateEstimator per (peer, track), the producer’s banded target through EncoderAdapter::set_bitrate, tier demotion and promotion with their hysteresis, the bandwidth-stats fields, the rate node in the /stats, the web client’s tracks.agent store with effectiveTier and the health reason. Gate, in the lab under the docs/25 profiles (tests/stack/ratecontrol.spec.ts), read through the introspection port the impairment exempts — the per-second stats events themselves ride the impaired link and arrive seconds late: (1) bad (15 % loss, 100 ± 40 ms, 1.5 Mbit; 4g’s 8 Mbit does not constrain a 4 Mbps track) applied to a lone streaming viewer: the agent’s estimate is below 2 Mbps within 2.5 s and the encoder’s output below 1.7 Mbps within 6 s (the software encoder’s output converges over a GOP), the viewer is demoted and keeps decoding (3 or more frames in 6 s at the 5 fps thumbnail tier — 15 % random loss costs a repair round trip per frame), and once the link clears it is promoted back within 15 s and the encoder is back above 90 % of its target within 25 s; (2) three viewers, one behind bad on the robot’s egress toward it alone (fjarr-opsim congested-viewer in the dev container, netemToward): that viewer reports effective_tier = thumbnail, keeps receiving frames, and is promoted back once the link clears, while the two browser viewers keep the active tier and ≥ 90 % of their bitrate throughout; (3) lossy (5 % loss) on a 30 fps track over 20 s: nacks > 0, keyframe_requests ≤ 2, the bitrate above 1.5 Mbps (no cut on random loss); (4) covered by (1): the lone viewer takes the whole range; (5) the opsim netem-* scenarios assert rate-control per track from the wire (active with an estimate ≥ 2 Mbps on links that carry it, ≤ 2 Mbps on bad); (6) every earlier gate green, the soak included. Met 2026-09-22 (review).

Implementation notes (slice 6a) — repair and rate control as built:

The offer’s feedback lines are codec preferences. rtcp-fb-* fields and the TWCC extmap-3 on the transceiver’s caps, do-nack on the transceiver; webrtcbin adds the rtx payload and rtprtxsend itself and the payloader picks the extension writer from the negotiated caps. Chromium accepts all of it unchanged. The webrtcbin answerer (fjarr-opsim) needs do-nack on its own transceivers to be a fair viewer, and does send transport-wide feedback.
twcc-stats is one window: packets-sent/-recv, bitrate-sent/ -recv, packet-loss-pct, avg-delta-of-delta (ns). Windows are 5–30 packets; a window with packets-recv 0 is an unacknowledged one — a receiver without feedback would otherwise read as a lossless link and be driven to the ceiling. A lost feedback packet makes the next window report everything it did not cover as lost (50–70 % in one window), so a loss cut needs the throughput collapse in three windows in a row: a shaper collapses every window, a lost feedback packet one.
Three defects the first trace found in the estimator: persistent random loss spiralled it to the floor (a cut per window, and sending less never reduces random loss); one lost packet in a seven-packet window read as 14 % loss; and the growth cap at 1.5× what arrived kept a demoted viewer, which receives 300 kbps, from ever reaching the promotion line. The rules above are the answer, and the rolling-second loss, the collapse streak and the uncapped climb each have a unit test.
Netem’s shaper queue is the enemy of the data channel. A lone viewer clamped to the shared-encoder band sent 2 Mbps into a 1.5 Mbit shaper for the 2 s a tier switch takes; the kernel queue grew past what SCTP tolerates and the session died of a transport error. Hence the single-subscriber rule.
Measure the agent through the exempt port. Under bad the bandwidth-stats events queue behind the video and arrive 3 s late; the lab’s RobotContainer.netem now keeps port 7381 out of the impairment like docker/lab/netem.sh, and netemToward(profile, ip) impairs the robot’s egress toward one address only, which is how the three-viewer gate gets one bad link.
congested-viewer runs in dev, not the robot container: netem on lo shapes both directions through one queue and the simulator’s own pings died behind the video. The netem-* scenarios still impair lo both ways and the simulator’s pings died behind the video; the netem-* scenarios now run it from dev against the robot’s eth0 egress, one-directional like a browser’s downlink.
A webrtcbin receiver over-reports loss under jitter. With lossy (5 % loss, 30 ± 15 ms) on the media path alone, the simulator’s webrtcbin reports 20–50 % loss per feedback window and the estimate falls to the floor, while Chromium under the same profile reports ~5 % and the rate holds (tests/stack/ratecontrol.spec.ts): its feedback marks packets that arrive after the feedback that should have covered them as lost. The viewers Fjarr ships to are browsers, so netem-lossy records the agent’s estimate instead of asserting it and the browser lab is the oracle for that profile.

6b — passthrough. The rtsp source’s passthrough and thumbnail_url params and its elementary outputs, the same param on a gst description, the producer’s parse-only path with one allow-not-linked tee per stream, tiers that are the camera’s own streams, adaptive: false when no substream exists, the keyframe no-op, --probe-source codec/profile reporting, the lab’s RTSP simulator serving a second low-resolution mount, the demo robot’s RTSP track in passthrough. Gate: (1) the demo’s RTSP track streams to the browser and its pipeline contains no encoder and no decoder — only depayload and parse — with /stats reporting passthrough: true and no encoder target (tests/stack/passthrough.spec.ts); (2) select-tracks to the thumbnail tier gives the browser the camera’s 640×360 substream without a renegotiation, and back again; (3) no keyframe request reaches the camera while a viewer joins and streams; (4) a passthrough source without a substream reports one tier and adaptive: false, with a transcoded track reporting both (unit tests over the media plane); (5) every earlier gate green. Met 2026-09-22 (review).

Implementation notes (slice 6b) — passthrough as built:

The core decides from the caps, not from a flag. Producer::build() reads the demanded output’s declared_caps; an elementary stream skips convert, rawcaps and the shared tee entirely. set_bitrate refuses (the camera owns the rate), request_keyframe counts and drops, and MediaPlane::adaptive() answers the bandwidth-stats field from the source’s shape before a producer exists and from the producer after.
One tee per stream, allow-not-linked. A camera’s substream is connected as soon as the producer builds, but the thumbnail tier may never start; an unlinked ghost pad returns not-linked and rtspsrc fails the whole pipeline with “Internal data stream error”. Each passthrough output therefore goes through its own tolerant tee, as the raw path’s tee already did.
gst_parse_bin_from_description(…, TRUE) ghosts the depayloader’s sink pad, which counts as linked and makes rtspsrc’s delayed link fail (“Delayed linking failed”) — the same trap make_late_ghost_bin documents for decodebin. The passthrough chain parses with FALSE and ghosts the named parser’s src pad by hand.
create_bin() returns a floating ref (docs/09). Wrapping the chains in an outer bin with the RAII sink helper handed the caller a second reference and the leaks gate caught it on the first run.

7a — the remaining fault rows. The fault menu is mostly covered: fjarr-opsim already carries socket-drop, silent-operator, ice-restart, relay-only, deadman, congested-viewer and hotplug, and the netem profiles arrived with 6a. What is left is the robot’s own lifecycle, which nothing exercises today:

A killed agent. The operator-visible half: peer-gone within the heartbeat budget, the robot.offline webhook, and the robot streaming again after a restart. The supervised restart is M2.5’s systemd unit, so this slice restarts it from the test and measures the rest.
A wedged core loop. The watchdog half of the supervision contract exists in code and nothing exercises it, which is the exact shape of the safety bug this project keeps citing. A fake notify-socket supervisor in the agent tests (NOTIFY_SOCKET names a unix datagram socket, so no systemd is needed) asserts READY=1 after the first hello-ack, WATCHDOG=1 at WatchdogSec/3, and that the pings stop when a fjarr.test fault blocks the loop. A real watchdog kill on a systemd host is an M2.5 gate.
A pipeline error outside the source bin, injected through a fjarr.test fault that posts a bus ERROR on a chosen producer element: the 0.5 s → 5 s × 5 backoff, then the escalation that closes every session with media-restart while the signaling socket stays up and the robot stays online, and exit code 2 on the third plane rebuild within ten minutes (media-plane recovery). The in-source-bin case is already covered by slice 4’s per-track failure.
An expired grant and a skewed clock: one attempt, a fatal grant-expired, no retry storm, on the web ladder and in fjarr-opsim.

Gate: (1) each row above asserted in the lab or in the agent tests as noted, with the timings read against the docs/16 budgets; (2) the M1 gate’s “reconnect and ICE restart under fault injection” promoted from opsim-only into the stack suite against a real browser; (3) every earlier gate green. Met 2026-09-23 — agent/tests/test_supervision.cpp, agent/tests/test_loop_media.cpp, web/e2e/tests/stack/faults.spec.ts, signaling/…/hooks.rs tests.

Implementation notes (slice 7a) — what the rows cost and what they found:

A tier’s framerate was a target, not a ceiling — and it broke every slow camera. The pipeline-error test ran its source at 15 fps and the producer would not start: cannot link tee to tier. make_encode_bin filtered the rate with videorate drop-only=true ! capsfilter framerate=<fps>/1, an exact rate that drop-only can never reach from below, so the caps intersection through the queue was empty and the tee refused to link. No camera under 30 fps could stream on any tier. The filter is a range now. Every fixture in the repo runs at 30 fps, which is exactly why three milestones of testing never saw it — the bug needed a slower source, and we had never written one down.
The test is the supervisor. The watchdog row needs no systemd: bind a unix datagram socket, name it in NOTIFY_SOCKET, set WATCHDOG_USEC, and the daemon’s real sd_notify path runs against it. SIGSTOP is then an honest whole-process hang — the process genuinely cannot run, so it genuinely cannot ping, which is the property systemd relies on. The same rig pins the READY contract. Until 2026-09-28 that was “not before the first hello-ack”, so that systemctl status could not call an unreachable robot fine; it now asserts READY with no server at all, within a bounded time, plus a STATUS= line that says the server is not reached — the same truth, delivered without holding the unit in activating (ADR-0019, second addendum).
A killed agent is noticed by the server, not by the heartbeat. SIGKILL closes the agent’s socket, so the operator saw peer-gone:agent-disconnected 629 ms after the kill — far inside the heartbeat budget, which is the fallback for an agent that stops talking without dying. Streaming again 6.9 s after the kill, with the test playing the supervisor M2.5 will ship.
The stack test owns spacing, not arithmetic. The exact hello count under a rejected grant is the client’s ladder arithmetic and is pinned against the mock in review.test.ts; against the real server what matters is that retries are bounded and spread out. Observed gaps 45 / 11 / 476 / 1013 ms: the free refetch, then real backoff.
ice-restart against the real agent already existed (slice 3b, agent.spec.ts), so the M1 gate’s reconnect row needed only the other half: a signaling-server restart under the real C++ agent rather than the in-page one.

7b — the latency harness. Less new code than it looks: the frame stamp already carries a counter and a 48-bit sender timestamp, the per-frame reader already exists, and the reader already applies the fjarr.core/time-sync offset — so no stamp change and no protocol change. The work is assembly and reporting:

coturn joins the lab profile (today it exists only under turn), so the relay column is measurable at all; the browser forces it with iceTransportPolicy: "relay" and fjarr-opsim with --ice-policy relay.
p50 and p95 over a run under clean, lossy and relay, plus time-to-first-frame after enable, written to the run’s summary.txt and appended to a tracked CSV with a hardware label (docs/15).
Input-to-photon is not in this slice. It needs a robot-side input path, which is fjarr.desktop in M3. The harness is built so that adding it is a second measurement, not a second harness.

Gate: (1) glass-to-glass p50/p95 reported for all three conditions, the CI job failing only above the loose ceiling while a strict run on the prepared runner gates the docs/16 numbers themselves; (2) a labelled run appends to the tracked CSV; (3) time-to-first-frame reported; (4) every earlier gate green. Met 2026-09-24 — web/e2e/tests/stack/latency.spec.ts, make latency, web/e2e/latency.csv.

Implementation notes (slice 7b) — the harness as built:

Almost no new measurement code. The stamp already carried a counter and a 48-bit sender timestamp, the per-frame reader already applied the fjarr.core/time-sync offset, and stamps() already returned p50/p95. The slice is the rig around them: the three conditions, the relay path, the gates and the record.
coturn joined the lab profile, and FJARR_TURN_SECRET now defaults to coturn’s own dev secret, so the relay column needs no .env to exist. The relay test asserts the selected candidate pair is actually relayed — a relay measurement that silently went direct is worse than none — and reads it after the sampling window, because getStats has no candidate pair in the first moments of a session and reports “not relayed” for a session that is about to relay every packet.
The first condition measured pays the producer’s cold start, which made whichever test ran first the slowest and the ordering look like physics. A settle window before sampling removes it; two consecutive runs now agree within a few percent.
Every row says whether to believe it. On a developer laptop the software encoder cannot hold the source rate, so latency tracks CPU rather than the path. The row carries its decoded fps, a run below 60 % of the source rate is marked CPU-limited, and strict mode refuses such a run outright instead of failing it against a budget it was never measuring. This is the same starvation docs/25 records for the media suites.
Input-to-photon is not here: it needs a robot-side input path and lands with fjarr.desktop in M3, as a second measurement on this rig.

Where `fjarr.test` lives

fjarr::TestCapability is a public class in libfjarr (docs/06), enabled by [capabilities."fjarr.test"]. fjarr-agent registers it from config; demo-robot registers it through the public API like any customer capability, and the compose demo-robot service runs the demo-robot binary — the demo stays a customer-shaped embedding.

What is kept from the camera streamer, and what is changed

The engineering digest of the camera streamer’s sender (docs/11) settled several details of this design. Kept nearly verbatim (with file-level provenance recorded in the implementation, not here — the source is gitignored reference material):

the RAII kit (GstElementPtr and friends, SignalConnection, ScopeGuard) — extended with sample/buffer/promise/message wrappers and a SignalConnection that holds a weak reference to its instance;
the per-peer pipeline (one webrtcbin + branches per session in its own GstPipeline), so one peer’s failure never reaches the producers or the other peers — the v1 design, after v2’s single shared pipeline showed why;
the post_to_owner + generation-context discipline, the caps-gated offer with its latch, the teardown ordering of close_peer_session, and the milestone-fed negotiation watchdog;
the FrameHub worker thread pushing into appsrc (never from the appsink streaming thread), appsrc with block=FALSE and a bounded queue so a slow peer drops instead of stalling the hub;
DOT dumps at lifecycle points.

Changed deliberately, each a documented flaw in the original:

Original	Here
marshaling onto the default `GMainContext`; owner thread recorded late in `run()`	the core owns a private `GMainContext` created in the constructor; the owner thread id is fixed when the loop starts and every `post_to_owner` targets that context
timer callbacks bypass the marshaling layer and touch state directly	timers are sources on the core context and still go through the generation check — no undocumented invariants
callback contexts hold a raw pointer to the implementation and check the generation after dereferencing it	contexts hold `(weak_ptr<Session>, generation)`; the check precedes any access; the core object outlives every source because the loop drains before disposal
one reliable-ordered DataChannel for ping, control and stats alike	one channel per docs/08 class, reliability set from the class table only
no backpressure: `buffered-amount` never read	`ChannelSender` enforces the docs/08 watermarks; the realtime sender drops instead of queueing
keyframe forced on the shared encoder at every join/enable	keyframe requests rate-limited per producer; a retained keyframe serves late joiners
ring buffer filled but never read	the ring is the late-join catch-up and the keyframe gate
SDP round-tripped through text between the promise and `set-local-description`	the `GstWebRTCSessionDescription` object is passed through; text only at the wire
heartbeat timeout equal to one interval (a single lost pong kills a session)	the operator pings; the agent’s liveness budget is three intervals
static TURN with embedded credentials, config through environment defaults	TOML config, TURN credentials from the docs/10 scheme
process restart on any producer error	producer restart with backoff, then plane rebuild, then exit 2
bandwidth counted at `appsrc` push	`get-stats` `outbound-rtp` (bytes actually sent, retransmissions included)

Decisions folded into this design

In-process planes with a documented process-separation seam (ADR-0019).
One VideoSource contract for every kind of source, provided declaratively, as a registered type, or by a capability; the core owns encoding (EncoderAdapter) and never transcodes a pre-encoded output.
Tiers are separate producers, started on demand; tier switch is a subscription change, never a renegotiation.
The agent never initiates ICE restart; it answers the operator’s request (docs/08 keeps “agent always offers” true while the decision stays on the side that observes the failure).
Session resume across a signaling blip is not in slice 3 (open question #20).
fjarr-opsim is a first-class test artifact, like the web mock agent.
Module E captures physical monitors as DMA-BUF, into VA without a copy, with shared memory as the fallback, because SHM monitor streams freeze behind a directly scanned-out window (ADR-0041, capture memory).