Web Client Architecture
Status: review — the slice-2 design, written before implementation (docs/13); implemented in
web/packages/core+web/packages/react(slice 2, 2026-09-16), unit-tested against the@fjarr/core/testingmock agent and reviewed retrospectively (slice-2 review). Inspired by the fleet dashboard’s session provider (prior art): its good ideas are kept and pushed further; its structural problems are explicitly designed out.
- A session follows the user. Connecting to a robot is an app-level fact: navigate between pages, open/close panels, mount/unmount tiles — the WebRTC session stays up and every consumer re-attaches instantly.
- Consumers subscribe to slices, cheaply. A video tile, a floating overlay, a 3D map, a status pane and a terminal each get exactly the data they need, at the delivery cadence they need, without re-rendering each other.
- Bandwidth follows demand. A track is sent only while something on screen wants it, at the quality the largest visible consumer needs.
- Library, not app. No robot vocabulary, no toasts, no routing, no storage assumptions inside the library; the host dashboard composes it.
- Many sessions, one page. Connecting to three robots at once is one client with three independent sessions — separate state machines, heartbeats, track registries, TURN credentials — rendered side by side. There is no ambient “the connected robot” anywhere in the library: every hook and component takes an explicit session handle.
- Publish as easily as subscribe. Teleoperation commands, terminal keystrokes, clipboard payloads and file bytes all flow toward the robot through the same typed, channel-class-aware API — with coalescing for lossy realtime data and backpressure for bulk.
Non-goals: replacing the host’s state management, UI kit, or auth.
Layering
Section titled “Layering”@fjarr/react FjarrProvider · useSession · useSessionState · useMessage useTelemetry · useLatest · useVideoTrack · <VideoTile> <FloatingVideo> · <VideoGrid> · <SessionStatus> · <ConnectButton> │ (thin: hooks over core stores via useSyncExternalStore)@fjarr/core FjarrClient ─┬─ SessionManager ── Session (state machine)(zero deps) │ ├─ SignalingTransport (WSS, docs/08) │ ├─ PeerConnection wrapper (trickle ICE, │ │ restartIce, reconnect backoff) │ ├─ EnvelopeRouter (cap/type pub-sub, │ │ request → accept/feedback*/result) │ ├─ TrackRegistry (manifest, MediaStreamTrack │ │ by track_id, demand → select-tracks) │ ├─ Heartbeat (docs/08, 5 s / 3 missed) │ └─ Stores (state, telemetry, stats) └─ GrantProvider (injected: host owns auth)@fjarr/core is framework-agnostic and dependency-free (docs/14). Its
stores implement { subscribe, getSnapshot } so React binds through
useSyncExternalStore with no context value churn: the context holds one
stable FjarrClient; everything reactive is a store selector.
Sessions
Section titled “Sessions”Placement
Section titled “Placement”<FjarrProvider client={client}> mounts once, at the app root above the
router. Sessions are owned by the client’s SessionManager, keyed by
robot_id; React components hold handles, never own sessions. Unmounting a
page therefore never tears a session down (the fleet dashboard’s core
insight, made structural rather than incidental).
const client = createFjarrClient({ serverUrl: "wss://fjarr.acme.com/ws", grant: (robotId) => myBackend.fetchGrant(robotId), // host owns auth});<FjarrProvider client={client}><App/></FjarrProvider>Lifecycle and idle policy
Section titled “Lifecycle and idle policy”const session = client.sessions.open("robot-024"); // idempotentsession.close("operator-closed");open() returns the existing session if one exists; opening three robots
is three calls, each yielding an independent handle. Subtrees bind to one
robot with <SessionScope session={s}> so leaf components can call
useSession() without prop-drilling — but the scope is a convenience over
explicit handles, never a singleton (a page renders three scopes for three
robots). Optional
idle policy: client.sessions.open(id, { idle: { closeAfterMs } })
closes a session that has had zero consumers (no subscriptions, no
acquired tracks) for the grace period — the explicit choice replaces the
dashboard’s implicit “connected until the user clicks disconnect”.
Default: no auto-close (an operator who navigated away expects the
robot to still be there when they come back).
State machine
Section titled “State machine”States are exactly docs/09’s: idle → connecting → connected → reconnecting → (connected | failed) → closed. Transitions:
| From | Event | To | Action |
|---|---|---|---|
| idle | open() | connecting | fetch grant → WSS hello → await hello-ack (session_id, TURN creds) → session-request brokered by server → offer |
| connecting | offer | connecting | set remote, answer, trickle ICE both ways |
| connecting | DTLS up + control DC open | connected | start heartbeat; flush demand → select-tracks |
| connected | ICE disconnected (past a 3 s grace) or failed | reconnecting | send ice-restart over signaling; the agent re-offers; keep tracks/consumers attached |
| connected | heartbeat 3× missed / signaling socket lost / no re-offer within 10 s | reconnecting | new signaling round via same grant (if unexpired) |
| reconnecting | recovered | connected | re-flush demand (agent keyframes on enable) |
| reconnecting | attempts exhausted | failed | consumers see failed and an error event (reconnect-exhausted); retry() available |
| reconnecting | retry() | connecting round now | skips the remaining backoff (a host “reconnect now” button) |
| connected | restartIce() | reconnecting (rung 2) | a host “reconnect media” action: the same ice-restart request the ladder sends on ICE loss; an agent without in-place restart answers session-close{retry:true} |
| any | peer-gone / session-close from server | closed (reason) | release tracks, keep subscriptions registered for a possible open() again |
| connected / reconnecting | session-close{retry:true} (agent media restart — the ICE-restart fallback, since no webrtcbin release restarts ICE in place) | reconnecting | a counted round, started at once — the agent asked for a fresh session |
| any | error(grant-expired) | reconnecting | refetch grant via provider, then retry immediately — the first refresh per attempt is free (no backoff, not counted); a second consecutive grant-expired is an ordinary backed-off round, so a host minting rejected tokens can never hot-loop; never a generic failure |
| reconnecting | error(robot-offline) | reconnecting | the robot is re-registering after the same outage (server or agent restart): a counted, backed-off round. On a first connect the same error is fatal (failed, reason robot-offline) — the robot simply is not there |
| any | error(session-unknown) | closed (reason) | the server no longer knows our session (a message crossed peer-gone on the wire): an orderly close, not a failure |
The attempt budget (default 5 rounds) is renewed by a connection that stayed up for 30 s, the same rule the docs/08 backoff uses; a connect timeout (default 15 s, grant fetch included) bounds every round so a hanging backend or a socket that never opens still climbs the ladder.
The rungs are the docs/08 reconnection ladder;
backoff is docs/08’s (0.5 s ×2, cap 30 s, ±20 % jitter, reset after 30 s
stable). All of this is core logic, unit-tested against a fake transport
and a fake peer connection (docs/15) — the dashboard’s ad-hoc toggle → cleanup → start is the anti-pattern.
Subscriptions: three delivery modes
Section titled “Subscriptions: three delivery modes”Different consumers need different cadences. Forcing everything through React state is the dashboard’s main structural cost (its 20-field context value re-renders every consumer on every status tick). The library offers three modes over one router:
| Mode | API | Re-renders? | For |
|---|---|---|---|
| State | useTelemetry(session, selector, equals?) | on selected-value change only | status panes, badges, buttons |
| Stream | session.on(cap, type, handler) / useMessage(session, cap, type, handler) | never | logging, toasts, one-off events, side effects |
| Latest ref | useLatest(session, cap, type) → { current } | never | rAF consumers: 3D maps, gauges, canvas overlays |
The router keys envelopes by (cap, type) per docs/08; useTelemetry’s
selector reads from a per-session telemetry store that keeps the latest
envelope per (cap, type[, key]) — newest-wins, so a burst of 100 Hz
joint states never queues. Equality defaults to Object.is; pass a shallow
comparator for objects.
Requests use docs/08 correlation:
const result = await session.request("fjarr.camera", "select-tracks", payload);for await (const fb of session.requestStream("fjarr.files", "file-offer", payload)) { … }request() resolves on result, rejects with a typed FjarrError(code);
requestStream() yields accept/feedback* then returns the result.
When the refusal carries error.data (docs/08) the
error keeps it as FjarrError.data; heldBy(error) reads the holder of a
control-held or busy refusal as { domain, holder: {id, label}, since }
(or null), so a client can say who to ask.
Domain state stays in the host. The library ships no
systemHealth/battery concepts. A host builds its own
RobotStatusProvider on useTelemetry in ~30 lines; demos/demo-dashboard
demonstrates exactly that.
Publishing: sending toward the robot
Section titled “Publishing: sending toward the robot”Subscribing is half the job; teleoperation, terminal input, clipboard and files all send. The same router exposes a publish side, and the channel class (docs/08#datachannel-topology) is chosen by the capability declaration, never by the caller guessing:
| Need | API | Channel class | Semantics |
|---|---|---|---|
| One-off event/command | session.send(cap, type, payload) | control (reliable, ordered) | fire-and-forget envelope, kind: "event" |
| Command with outcome | session.request(cap, type, payload) | control | accept/feedback*/result correlation |
| Continuous lossy stream (joystick, pointer, joint targets) | session.publisher(cap, type, { key?, maxHz }) → .publish(payload) | realtime (unordered, no retransmit) | newest-wins per key, rate-capped (default 60 Hz); a burst never queues, the latest value always goes out |
| Byte stream (terminal input, clipboard payload) | session.channel(cap) → .write(bytes) | the capability’s bulk channel (fjarr:bulk:<cap>, reliable-ordered — docs/08); bytes written before the channel opens are queued (bounded) | ordered; bufferedAmount/onDrain exposed; incoming bytes via .onData(cb) |
| Large binary, either direction (file upload, a snapshot body) | session.bulk(cap) → .sendFrames(iter) / .sendBlob(bytes, type) / .receive(ref) / .onChunk(cb) | bulk (same DC), blob frames | send: pumps while below HIGH_WATER, resumes on bufferedamountlow. receive(ref) resolves a blob reference found in an envelope: whole bytes once complete (16 MiB cap), either arrival order, rejects after the docs/08 timeout or when the channel closes. onChunk is the streaming form for files (M4) |
| Lossy binary frames (point clouds, depth, custom sensors — either direction) | session.stream(cap) → .send(frame) / .onFrame(cb) | stream (unordered, no retransmit, binary; ADR-0018) | frame-level newest-wins with sequence numbers; chunked to the SCTP message limit; consumers get whole frames or nothing. Lands in M4 with the first stream-class capability (roadmap); slice 2 ships the surface without the chunker |
React bindings: usePublisher(session, cap, type, opts) returns a stable
publish function; useCommand(session, cap, type) wraps request() with
pending/error state for buttons.
Safety tie-in (docs/15#safety-behaviors). A realtime publisher can be
created with { deadman: { intervalMs } }: while the consumer holds it, the
library re-publishes the last value at that interval even if the input is
idle, so the agent’s deadman sees a live stream; releasing the publisher
stops the heartbeat and the agent’s deadman fires within its budget. This
is how a joystick component unmounting (or a tab going hidden) stops a
robot without any component author remembering to send “zero” — the
regression that silently broke the teleop-car deadman (prior art).
Worked example — a teleop stick:
const drive = usePublisher(session, "com.acme.teleop", "cmd_vel", { maxHz: 50, deadman: { intervalMs: 100 },});// gamepad loop:drive.publish({ linear: y * maxSpeed, angular: -x * 1.25 });and a terminal:
const pty = session.channel("fjarr.terminal"); // reliable-ordered per its manifestterm.onData((keys) => pty.write(encoder.encode(keys)));Control domains {#control-domains}
Section titled “Control domains {#control-domains}”Input-bearing capabilities share a robot by taking turns per control domain
(desktop, motion; docs/10). The session
keeps who holds each domain as state, from the agent’s
fjarr.core/control-state (docs/08):
| API | Semantics |
|---|---|
session.control: ReadonlyStore<ControlState | null> | { domains: { [domain]: { holder: {id, label} | null, since: number | null, you: boolean, viewOnly: boolean } } }. Only the domains this session’s grant has a capability in. Each event replaces the last; an unchanged domain keeps its object identity. null until the agent’s first control-state on this connection, and again after any disconnect (the next connection’s event is the truth). A malformed domain entry is dropped with a warning; unknown fields and domains are kept or ignored per docs/08 |
session.takeControl(domain) | take-control request; resolves on ok, rejects with FjarrError (capability-denied for a view_only grant or a grant without that domain) |
session.releaseControl(domain) | release-control request; idempotent |
useControl(session, domain) | { known, holder, since, you, free, viewOnly, takeControl, releaseControl }; re-renders only when that domain changes. known means control-state lists the domain; free is known and unheld |
since is unix ms on the agent’s clock: add timeSync.offsetMs to a
local timestamp before subtracting (“Anna (3 min)”). viewOnly comes from
the domain’s view_only in control-state: this session can never claim the
domain, so a client renders no input surface and no take-control button. It
is false when an older agent omits it, and such an agent refuses a
take-control from a view_only grant with capability-denied. The library
does not update the state optimistically on a successful takeControl: the
agent sends control-state before the result, so the new holder is already
in session.control when the promise resolves.
Claims happen on the first input in a domain, so a client whose deadman
publisher keeps re-sending after it lost control would re-claim the domain
the moment it frees. A driving component releases its publisher when you
turns false (the demo dashboard remounts its teleop panel).
Media: demand-driven track delivery
Section titled “Media: demand-driven track delivery”The efficiency idea the dashboard got right — enable tracks only when a video component is in the DOM — becomes automatic and richer.
Track registry
Section titled “Track registry”On offer, the manifest (docs/08) populates a per-session registry:
track_id → { label, cap, kind, codec, pt, mid, monitor }. Incoming
RTCTrackEvents map to track_id by transceiver.mid — the agent
includes mid in the manifest (protocol amendment),
so no SDP parsing or payload-type guessing is ever needed.
MediaStreamTracks live in the registry, independent of any element:
a <video> that mounts later simply attaches to the existing track.
Renegotiation-safe. A new offer mid-session (docs/08)
is diffed against the registry by track_id: new tracks are added
(pending until their RTCTrackEvent), removed tracks are released and their
consumers see status: "unavailable" while keeping their handles, and
untouched tracks are left strictly alone — no element re-attach, no
flicker. Demand is re-flushed only for tracks whose entry changed. Offers
with a manifest_version older than the applied one are ignored.
Fan-out is free. Any number of consumers (a grid tile, a floating
overlay, a picture-in-picture, a canvas overlay) attach to the same
track: the browser receives and decodes each RTP stream exactly once
regardless of how many elements render it, so a second tile costs zero
network bandwidth and zero decode — only compositing (the
receiver playground’s subscribeToStream idea, prior art,
made a guarantee of the registry). Demand is therefore aggregated per
track, never duplicated: ten consumers of cam-front produce one enabled
track at the highest tier any of them asks for.
Demand model
Section titled “Demand model”Every consumer declares what it needs; the registry folds demand into one per-track decision and tells the agent:
const handle = session.tracks.acquire("cam-front", { tier: "active" | "thumbnail", // quality it needs (docs/16 tiers) visible: true, // updated by the consumer as it changes});handle.update({ visible: false });handle.release();Per track: enabled = any consumer visible, tier = max over visible consumers, preference = sharpness if any visible consumer asks for it.
The agent may send a lower tier than asked while this peer’s link is
below the tier’s band (docs/23);
session.tracks.agent (a store of the agent’s per-second view per
track from bandwidth-stats: tier, effectiveTier, estimateBps,
adaptive, bitrateBps, nacks, keyframeRequests) then shows
effectiveTier below tier; it is kept beside the entries, not in
them, so the per-second tick re-renders only useTrackAgentStats(session, trackId) consumers. adaptive: false marks a passthrough track that
cannot adapt. The demand model never changes what it asks for because of
it — the agent’s override is the agent’s.
Changes are debounced (~250 ms) and coalesced into one select-tracks
request per track-owning capability (fjarr.camera for camera tracks,
fjarr.desktop for monitors — docs/06);
a demand change touching both produces two requests, never one per consumer.
Reconnection re-sends the full demand snapshot. Default is nothing
enabled — the dashboard’s “all tracks on until told otherwise” wastes
bandwidth on every connect.
Visibility
Section titled “Visibility”useVideoTrack (below) drives visible from IntersectionObserver on the
element plus document.visibilityState, with a short grace period so
scrolling past a tile doesn’t flap the stream. Consumers that must keep a
track warm off-screen (a floating overlay minimized to a corner, a
picture-in-picture) pass keepWarm: true. Re-enabling gets a keyframe
from the agent (docs/06), so the first frame after scroll-back is
immediate.
Audio tracks
Section titled “Audio tracks”Audio is a track like video (kind: "audio" in the manifest, docs/08),
served by the fjarr.audio capability (docs/06):
- Downlink (hear the robot’s surroundings, machine sounds): acquired
through the same demand model —
useAudioTrack(session, trackId)/<AudioSink>; default off; per-consumer mute. Browsers refuse to play unmuted audio without a user gesture, so<AudioSink>exposesstatus: "blocked-autoplay"and anunlock()to call from a click; the library never fakes a gesture. - Uplink (talk to a person at the robot): the operator’s microphone is a
published media track. Because the agent always offers (docs/08), the
agent pre-allocates a
recvonlyaudio transceiver in the initial offer whenever the grant includesfjarr.audiowithtalk: true; the browser attaches its mic withRTCRtpSender.replaceTrack()— no renegotiation.usePushToTalk(session)is the default UX (open mic is an explicit opt-in), echo cancellation/noise suppression flags pass through togetUserMedia, and every uplink start/stop is a session event for audit (docs/10 — a live microphone is as sensitive as a terminal).
Hooks and elements
Section titled “Hooks and elements”const { track, attach, status } = useVideoTrack(session, "cam-front", { tier: "active" });// attach(el) sets el.srcObject = new MediaStream([track]) and calls play();// status: "requested" | "streaming" | "disabled" | "unavailable"<VideoTile session trackId tier> is the headless-first default: a
<video muted playsInline autoPlay> with the hook wired, a slot for
overlays, and data-fjarr-status for styling. Stats (bandwidth-stats
envelopes, per docs/08) are a telemetry store: useTrackStats(session, trackId).
Stats and connection health
Section titled “Stats and connection health”RTCPeerConnection.getStats() is the ground truth for what the network is
doing; the dashboard tooling around it was the best part of the
receiver playground (prior art) and is
redesigned here as a core service rather than a component’s setInterval.
- Sampler in core, one per session (default 1 s, configurable), publishing
into the telemetry store so consumers use mode-1 selectors
(
useTrackStats(session, trackId),useSessionStats(session)) — no timers in React, no re-created intervals. - Per-track, not averaged.
inbound-rtpreports are keyed by theirmid→track_idvia the manifest. Per video track: bitrate (windowed, from byte deltas), packets lost (cumulative and windowed loss rate), jitter, jitter-buffer delay, frames decoded/dropped/per-second, resolution, freeze count and duration, key frames, PLI/FIR/NACK counts, average decode time,decoderImplementation+powerEfficientDecoder(is hardware decode engaged?). Per audio track: level, concealed samples/events. - Transport: RTT from the selected candidate pair only (not an
average over all pairs),
availableIncomingBitrate, and the pair’s local/remote candidate types — so the UI can say “relayed via TURN” versus “direct”, which explains most latency complaints on the spot. - Outbound (published audio, future uplinks): bytes sent,
qualityLimitationReason, retransmissions. Data channels: per-channel messages/bytes in and out,bufferedAmount. - Agent correlation: the agent’s
bandwidth-statsenvelope (docs/06) gives sent bytes per track; sent − received over the same window is the relay/path loss picture the latency harness (docs/15) plots. - Health score with reasons, not a color.
useSessionHealth(session)yields{ level: "good" | "degraded" | "poor", reasons: ["loss 6% > 5%", "rtt 340 ms > 300 ms"] }with thresholds taken from the performance budgets and hysteresis (a level changes only after three consecutive samples agree) so a single bad second doesn’t flap the badge. A tier reduced by the robot is a reason of its own (“tier reduced by the robot: link 900 kbps”, frombandwidth-stats), so the badge explains a quality step that the browser’s own stats would show as healthy.<ConnectionQuality>renders it; hosts can render their own from the same hook.
Components (@fjarr/react)
Section titled “Components (@fjarr/react)”All headless-first: logic in hooks, minimal default styling, every visual overridable; the host’s design system wins (docs/05).
| Component | Responsibility | Dashboard ancestor |
|---|---|---|
<SessionStatus> | reactive state chip; always re-renders on transition | ConnectionIcon |
<ConnectButton robotId> | open/close with correct labels per state | ConnectionButton |
<VideoTile> | one track, demand-managed | RunPage grid cells |
<VideoGrid> | N tiles from the manifest; layout order/labels from a host-provided ordering (no hard-coded camera names) | buildGridVideoTrackRows |
<FloatingVideo> | draggable/resizable/anchored overlay that persists across routes; keepWarm when collapsed | FrontCameraOverlay (generalized, robot-agnostic) |
<AudioSink> / usePushToTalk | downlink audio with autoplay-unlock status; PTT uplink | — |
useSessionPeers / <SessionPeers> | who else is on this robot and who owns input (docs/08 session-peers, M5) | — |
useGamepadHaptics | maps capability haptic events to the Gamepad vibration actuator | — |
<ConnectionQuality> | health level + reasons from the stats sampler | StreamSelector stats panel (generalized) |
<DesktopView> | the remote-desktop surface: one monitor’s video and its input surface, on DesktopInput from @fjarr/core. Design in docs/22, which also lists the core requirements slice 2 must satisfy for it | M3 3.1 (video), 3.2 (input) |
useTerminal(session) / <TerminalView> | the robot’s shell (M2, slice 2a): the hook carries no renderer, the component adds xterm behind optional peers on the @fjarr/react/terminal entry — the same shape <PipelineGraph> uses for d3, so a dashboard with no terminal pays nothing. “Not configured” and “not granted” are states to render, not errors to retry (docs/06, docs/08) | — |
<PipelineGraph feed pipelineId> / usePipelines(feed) / usePipelineSnapshot(feed, id) / usePipelineFeed(session) | live GStreamer pipeline graphs from fjarr.introspect (docs/24) over a pipeline feed; d3-graphviz rendering, history scrubbing (slice 5) | — |
Third-party capabilities register views with the same registry (docs/05).
Pipeline feeds {#pipeline-feeds}
Section titled “Pipeline feeds {#pipeline-feeds}”The pipeline viewer has two homes with two transports: the dashboard’s
Diagnostics tab reads snapshots over the session (fjarr.introspect,
blob references on the bulk channel), and the viewer served from the
robot’s own endpoint reads them over HTTP and Server-Sent Events
(docs/24). One component, one feed
interface, two implementations — otherwise there would be two viewers:
interface PipelineFeed { readonly pipelines: ReadonlyStore<PipelineInfo[]>; // id, kind, state, seq, last trigger readonly status: ReadonlyStore<{ live: boolean; error: string | null }>; // subscribed / stream open; capability-denied lands here snapshot(id: string): ReadonlyStore<SnapshotMeta | undefined>; // newest-wins per pipeline body(id: string, seq: number, form: "txt" | "json" | "dot"): Promise<string>; // bounded cache (64 bodies) history(id: string): Promise<SnapshotMeta[]>; refresh(): Promise<void>; // re-read the list close(): void;}sessionPipelineFeed(session, { forms? }): PipelineFeed // pipelines/subscribe (+ replay) + blob receive; re-subscribes on every new sessionhttpPipelineFeed(baseUrl, { token?, retryMs? }): PipelineFeed // GET /pipelines, /events (streamed fetch, Last-Event-ID), /pipelines/<id>.<form>?seqBoth live in @fjarr/core. The HTTP feed parses SSE from a streamed
fetch, not the browser’s EventSource, so it also runs in Node: the
lab’s introspect command is its third consumer
(docs/25). Stores use the same ReadonlyStore as
telemetry, so the hooks re-render only on change.
usePipelines, usePipelineSnapshot, usePipelineBody, useFeedStatus
and <PipelineGraph feed pipelineId seq?> take a feed, not a session —
the one deliberate deviation from the session-everywhere convention of
this document, made so the robot-local viewer and the dashboard share
every line above the transport. usePipelineFeed(session) creates one
session feed per session, shared by every consumer and closed by the
last, so a Diagnostics tab stays the promised twenty lines. The hooks
live in the main entry; <PipelineGraph> is the separate
@fjarr/react/pipelines entry because its default renderer pulls
d3-graphviz and the Graphviz wasm (optional peer dependencies, docs/14) —
pass renderDot to draw with anything else, and seq to scrub. The gate for “one
viewer” is a contract test that runs one suite against both feeds (the
session feed through the lab stack, the HTTP feed against the endpoint):
same list, same newest-wins behaviour under a burst, same bytes for a
given sequence, same recovery after the transport drops.
Capability-specific requirements on the core
Section titled “Capability-specific requirements on the core”Some capabilities need small hooks in the core that are cheap now and breaking later. Slice 2 provides them up front:
- Remote desktop (docs/22 core requirements):
track acquire options
preference+latencyMode(→select-tracks,jitterBufferTarget), a page-level keyboard focus registry, cursor events on the realtime class,useTimeSync. - Terminal: an ordered byte channel with
onDrain(publishing table). - Audio: autoplay-unlock status and
replaceTrackuplink on a pre-allocated transceiver (media section).
Host-app conveniences (opt-in, never automatic)
Section titled “Host-app conveniences (opt-in, never automatic)”- Persistence adapter:
createFjarrClient({ persistence })with{ get, set }so the host can remember the last robot (localStorage, URL, their store) — the library never toucheslocalStorageitself. useBeforeUnloadWhileConnected()— the dashboard’s warning-on-close, as a hook the host mounts if it wants it.- Events, not toasts:
client.on("session-event", …)emits typed lifecycle/errors; how they’re shown is the host’s job.
CLI login handoff {#cli-login}
Section titled “CLI login handoff {#cli-login}”fjarr-connect needs an operator credential and has no logged-in session of
its own (docs/27). @fjarr/react ships the
handoff as a component the host mounts on one authenticated route, so
their work is adding a route rather than building a flow:
<FjarrCliLogin mintOperatorCredential={() => api.post("/me/fjarr-cli-token")} approveCode={(code, credential) => api.put(`/fjarr/cli-codes/${code}`, { credential })}/>It reads which shape the CLI asked for from the page’s query string — ?port=&state=
for loopback, ?code= for a code, or nothing, in which case it offers a field to
type a code — and the host can pass request={{...}} to override that. Every
FjarrCliLogin string is a prop with a plain-English default, so the host can
translate or rebrand without forking it.
It renders inside the host’s app and therefore inherits the signed-in user; the host supplies one function that returns a credential for that user, which is usually a call to something they already have. The component handles both shapes the CLI can ask for:
- Loopback — posts the credential to a
127.0.0.1callback the CLI is listening on, after echoing the port and requiring an explicit click. It refuses any callback that is not loopback, and round-trips the CLI’sstatevalue. - Code — shows a short code the user confirms matches the one their terminal printed, for the headless case where no browser can be opened on the machine running the CLI.
It is a component, not a page: the host’s layout, and the host’s decision about who may mint a credential at all. The library never stores the credential and never sees the CLI.
Wire tap {#wire-tap}
Section titled “Wire tap {#wire-tap}”createFjarrClient({ wireTap: true }) enables client.on("wire", (e) => …)
with
{ robotId, sessionId, dir: "in" | "out", channel: "control" | "realtime" | "bulk" | "stream", cap, type, kind, eventId, bytes, ts }
(payloads on request via e.payload; envelopes only — bulk/stream report
sizes). Off by default and never enabled by @fjarr/react: envelopes carry
keystrokes and clipboard text (docs/10). The browser lab and fjarr-lab wire are its consumers (docs/25).
Audio uplink, as negotiated {#audio-uplink-negotiation}
Section titled “Audio uplink, as negotiated {#audio-uplink-negotiation}”The agent offers its push-to-talk transceiver recvonly; a browser creates
the matching transceiver at setRemoteDescription with direction
recvonly, so the core sets it to sendonly before createAnswer — or
replaceTrack() later moves no media at all (found in the browser lab,
slice 3a). What the agent plays is decided by messages
(docs/08, M3.5 3.5.3): audioUplink.replaceTrack(track)
sends talk-start and attaches the microphone only once it is granted, so a
refusal (busy, capability-denied, unavailable) leaves the microphone
detached and is reported as audioUplink.refused (code, message, holder);
replaceTrack(null) detaches and sends talk-stop. usePushToTalk exposes
the refusal as its error and the agent’s talk-level as robotLevel, so
the operator sees their voice arrive. getUserMedia only exists in secure contexts (https,
localhost): usePushToTalk reports that as its error instead of a
TypeError.
Testing (docs/15)
Section titled “Testing (docs/15)”- Golden fixtures against TS types — closes open question #15:
every
protocol/fixtures/valid/*.jsonmust parse into@fjarr/core’sEnvelope/Messagetypes; invalid ones must be rejected by the runtime guards (avitesttest next tocheck.mjs). - State machine unit tests with a scripted fake transport: every row of
the transition table, plus the fault menu (silent server,
grant-expired,peer-gone, ICEfailed). - Demand model unit tests: fold/debounce/coalesce, visibility flapping, reconnect re-flush.
- Publisher unit tests: newest-wins coalescing under burst, rate cap,
deadman re-publish while held and silence after release, bulk pump
honoring
bufferedAmountwatermarks. - Stats sampler unit tests against recorded
getStats()snapshots: per-track keying bymid, windowed deltas, selected-pair RTT, relay detection, health reasons + hysteresis. - Multi-session tests: three sessions on one client with independent state machines; a fault injected on one leaves the other two untouched.
- Browser e2e in the browser lab: first against
the in-browser loopback agent (a real
RTCPeerConnectionpair in the page — autoplay,srcObject, IntersectionObserver,getStats,replaceTrack, renegotiation and ICE restart on a real WebRTC stack), then against demo-robot via the demo stack once it streams. The core’s wire tap (client.on("wire", …)) makes DataChannel traffic observable there, since CDP cannot see inside SCTP. - All of the above except the browser e2e run in
make web-testwith no browser:@fjarr/core/testingprovides the scripted socket, peer connection andMockAgent(docs/15) that host dashboards can reuse.
Decisions folded into this design
Section titled “Decisions folded into this design”- External stores +
useSyncExternalStoreinstead of a state library — keeps@fjarr/coredependency-free and React-version-agnostic (zustand-style ergonomics without the dependency). - Multi-session from day one; single-session apps simply open one.
<SessionScope>is sugar over explicit handles, never a global. - Publish side mirrors subscribe side; channel class comes from the capability manifest, coalescing/backpressure from the channel class.
- Demand-driven delivery with visibility is on by default;
keepWarmis the escape hatch. - Agent supplies
midin the manifest (protocol amendment, docs/08).
Decided with the maintainer (2026-09-16): idle policy defaults to never
auto-close (hosts opt into closeAfterMs); <VideoGrid> ships in
@fjarr/react, headless-first, with host-provided ordering. The 250 ms
demand debounce is an initial value to be tuned against the latency
harness.