Remote Desktop Client

Status: review. The client half of fjarr.desktop (docs/06), built on the web client architecture. Implementation lands in M3; the core requirements section is a slice-2 obligation so the core needs no breaking change when the desktop view arrives.

Goals

A remote desktop in a browser is lost or won on details the camera use case never meets: whether Ctrl+W closes the robot’s window or your tab, whether the cursor lags, whether “å” arrives, whether a stuck Shift key survives a tab switch. This document pins every one of those.

Anatomy of <DesktopView>

<DesktopView session monitor={0}>
 ├─ VideoSurface        one <video> per monitor track (docs/21 demand model,
 │                      tier "active", preference "sharpness")
 ├─ InputSurface        transparent layer capturing pointer/keyboard/wheel/
 │                      touch; owns the focus state; renders the focus ring
 ├─ CursorOverlay       local cursor rendering (see Cursor strategy)
 ├─ Toolbar (slot)      monitor switch, fullscreen, special keys, clipboard,
 │                      view-only badge, quality (docs/21 health)
 └─ hooks: useDesktopInput, useDesktopFocus, useMonitors, useClipboardSync,
           usePresentation (multi-monitor fullscreen, see below)

Headless-first like every component (docs/05): the surfaces are hooks + minimal elements; the toolbar is a slot the host fills.

Where the logic lives (decided 2026-09-30, M3 planning). The input model is framework-agnostic and belongs to @fjarr/core, so a customer on Vue or plain TypeScript gets the hard part without React:

• @fjarr/core: DesktopInput, one per desktop view. It takes DOM events and a monitor’s geometry, and turns them into the wire messages (docs/08): code → evdev keycode, no auto-repeat, composition and beforeinput for text, held-key tracking and release-all, wheel normalization, pointer coalescing, and normalized per-monitor coordinates. It knows nothing about rendering and is unit-tested with synthetic events.
• @fjarr/react: <DesktopView> and the hooks above: the video and cursor surfaces, focus and Keyboard Lock, fullscreen, and presentation mode. They wire DesktopInput to the DOM and the session, and add nothing to the input model.
• The demo dashboard gets a Desktop panel for a robot that has a monitor: <DesktopView> with a toolbar of take-control, monitor switch, presentation mode and clipboard. It uses only the public API, like every demo.

Monitors and geometry

The manifest carries one video track per monitor with monitor: {id, index, primary, x, y, w, h, scale, name, connector} (docs/08). id is the stable identity: the EDID vendor-model-serial slug, not the connector name, which a replug can change. track_id is desk-<id>; index is display order only and changes when other monitors come and go — nothing in the client keys on it. useMonitors(session) is reactive: it reflects the monitors event immediately and the manifest after renegotiation. The view renders one monitor, a host layout can render several. Coordinates sent to the agent are normalized to the monitor (docs/08), so the client must map from screen pixels to the video’s content box — with object-fit: contain the element has letterbox bars that are not part of the monitor:

const box = contentBox(videoEl);           // accounts for object-fit + videoWidth/Height
const nx = (e.clientX - box.left) / box.width;   // outside [0,1] → no event, unless a button is held (below)

Display modes: fit (contain), fill, and native (1:1 pixels with scrolling, for pixel-exact work); HiDPI is handled by the agent’s scale being informational only — normalized coordinates make it irrelevant.

Dragging across monitors {#drag-across-monitors}

A window dragged on the robot crosses from one monitor to the next in one motion, as with a mouse at the robot (asked for 2026-10-11; until then the pointer stopped at the edge of the monitor the drag started on). While a button is held, a point outside the view’s monitor is not dropped: the client maps it through the robot’s own layout and sends it to the monitor it lands on.

• The point, unclamped, becomes robot coordinates: x + nx·w, y + ny·h of the view’s monitor (monitor.x/y/w/h from the manifest, the layout the robot’s display settings show). The monitor containing that point is the target; in a gap between monitors or beyond the outer edge, the nearest point of the nearest monitor is. The client sends pointer {track_id, x, y} normalized to the target, so the agent and the protocol are unchanged: the robot sees one pointer, pressed on one monitor and moved onto another, which both backends take as one drag.
• The robot’s arrangement decides, not the operator’s. Dragging right out of a monitor reaches the monitor right of it on the robot, wherever the operator has put its view or window. The robot’s cursor, drawn in every view (cursor strategy), shows where the pointer is. Following the operator’s own window placement instead was rejected for now: a browser on Wayland reports every window at 0,0.
• It holds for every view: the embedded <DesktopLayout>, a single view, and separate windows, fullscreen included. The view the drag started in keeps the pointer capture, and the browser keeps delivering its pointer moves, with coordinates past the view’s edges, for as long as the button is held, even over another window or screen.
• Without a held button nothing changes: a point outside the monitor sends nothing, and the operator’s pointer simply enters the next view.

The mapping is a pure function in @fjarr/core (throughLayout), tested against neighbours, gaps, offsets and monitors of different sizes; the lab browser drags from one fixture monitor into another and checks, in the robot’s test window, that the pointer crossed while the button was still down. Verified by Erik on the mini-PC (2026-10-11) in the embedded layout, in Chrome; separate fullscreen windows and Firefox are still to be tried.

Hot-plug

Monitors connect, disconnect, re-plug and change mode during sessions; the client must make this boring:

• <DesktopView monitorId="del-dell-u2422h-gk19rp3"> binds to the stable id. A host app offering a monitor picker shows name and connector and stores id. If that monitor disappears the view stays mounted and shows a “monitor disconnected” placeholder (its track handle is kept, demand released); when the monitor returns — same track_id — the view rebinds automatically, with no host-app involvement.
• <DesktopView> with no monitorId follows a policy: primary (default, tracks the primary flag as it moves), or first.
• <DesktopLayout session> renders every current monitor arranged by its x/y geometry — the same picture as the OS display settings — and reflows on every change; the host can override with tabs or a grid.
• The monitors event (docs/08) arrives before the renegotiation finishes, so placeholders and arrangement update instantly; frames follow within the docs/16 hot-plug budget. Tracks for untouched monitors are never re-attached (docs/21 renegotiation-safe registry) — no flicker on the monitors you were working on.
• Mode/DPI change on the same monitor: the <video> simply reports new videoWidth/Height; the content-box mapping recomputes; nothing re-binds.
• Zero monitors: the desktop view shows “no display connected”; the session, input focus and clipboard remain; the first monitor to appear is bound per policy.
• Mirrored outputs appear as separate tracks; <DesktopLayout> hides a monitor whose geometry exactly duplicates another’s unless showMirrors.

Keyboard focus is per session, not per monitor — keys go to the robot’s focused window wherever it is; pointer events carry the monitor’s track_id, so a single keyboard owner with several monitor views is the normal case.

Separate windows {#presentation-mode}

The goal: an operator can put a robot monitor in a window of its own, for instance on another local screen, fullscreen there with Keyboard Lock. The platform constraint: a document can be fullscreen on one screen only, in every browser, so each robot monitor shown fullscreen elsewhere is a window of its own.

One action, the same in every browser: “Open in separate window”. It opens one robot monitor in one new, normal window, one click per window. Fullscreen is its own, second click: the operator drags the window where they want it and uses the view’s fullscreen button there (enterFullscreen(), with Keyboard Lock, so Alt+Tab and Super reach the robot). A window never opens fullscreen by itself: two separate steps are easier to understand than one that does both. Nothing is placed or mapped automatically and no permission is asked: the Window Management API could open a window on a chosen screen, or fill two screens from one click through the fullscreen companion window, but measured in Chrome it still opens one popup per click, and it exists in Chromium only. One action that behaves the same everywhere is easier to understand (decided 2026-10-04, spikes/presentation-mode).

One session, portaled views (the spike settled it, closing open question #19). The dashboard page keeps the one session, the one grant and the one docs/10 desktop claim; a separate window is only a rendering surface. useSeparateWindows(session) in @fjarr/react:

1. open(monitorId), from a click, opens an empty same-origin popup (window.open("", …)), which stays in the opener’s browsing context group under any Cross-Origin-Opener-Policy the host sets. A popup that loaded a page of its own lost window.opener under COOP same-origin on the opener alone.
2. The hook renders <DesktopView monitorId={…}> into the popup’s document.body through a React portal (windows, which the host puts in its tree). The React tree, the session, the track handles and the focus registry stay in the opener; the <video> lives in the popup and plays the opener’s MediaStream. React attaches its event listeners to a portal’s container, so the input pipeline works there unchanged. The view uses its element’s own ownerDocument and defaultView for fullscreen, Keyboard Lock, focus, visibility and sizes, never the opener’s window and document.
3. Focus: a separate window that has the OS focus owns the keyboard; the focus registry treats each window’s focus/blur as its view’s, so one keyboard owner still holds across all windows.
4. Teardown: closing a separate window releases that monitor’s demand (unmount → release(), docs/21) and sends release-all for keys held from it; pagehide on the opener closes every separate window (a window with no session behind it is a bug). A robot monitor that goes away leaves its window showing the placeholder, as inline; re-plugging rebinds it.

Every <DesktopView> remains usable inline too: a separate window is a place to show it, never a different component.

Measured limit. Two fullscreen windows on two screens rendered ~15 fps each from a 30 fps stream on Chrome under X11 (one window alone: ~28), whatever the track sharing; a session per window would pay it too. It is the desktop’s presentation cost, to be measured again on Wayland and another GPU, not the design’s.

Not the browser’s job. A fixed control room (an operator desk with three screens permanently dedicated to one robot) is better served by Chrome kiosk mode or a Tauri/Electron shell hosting the same dashboard. Fjarr documents that recipe (docs/12 later) rather than building a shell.

Input pipeline

All input goes through the docs/21 publish side; classes per docs/08:

Input	Browser source	Message (docs/08)	Class
Pointer motion	pointermove (coalesced ≤ 60 Hz; getCoalescedEvents folded)	pointer {track_id, x, y, seq}	realtime
Buttons	pointerdown/up with setPointerCapture so drags that leave the element still deliver the up	button {button, down}	control
Wheel	wheel with deltaMode normalized to pixels (lines × 16, pages × viewport) and accumulated, sent at ≤ 60 Hz	wheel {dx, dy}	control
Keys	keydown/keyup using code (physical key); auto-repeat is not forwarded (repeat === true dropped — the agent’s held key auto-repeats natively)	key {code, down}	control
Composed text / IME	compositionend on a hidden contenteditable that holds the keyboard focus; keys during composition (isComposing) are not sent as keys. Paste-as-typing is a host action through typeText()	text {text} (a request: the robot types it through its keymap or refuses naming what it cannot type, docs/08)	control
Touch (tablets)	tap → click, long-press → right click, two-finger drag → wheel, pinch → local zoom (never sent)	as above	—
Special combos	toolbar buttons	key-combo {codes: ["ControlLeft","AltLeft","Delete"]}	control
Release everything	on blur, visibilitychange: hidden, focus loss, unmount	release-all	control

Held-state tracking: the client keeps the set of keys/buttons it has sent down for and sends ups (or release-all) itself when focus is lost — the agent’s own release_all_input on session end (docs/09) is the backstop, not the mechanism. A stuck Shift after Alt+Tab is a bug in this table, not an edge case.

Focus model

As built (M3 slice 3.2). <DesktopView> takes viewOnly and releaseKey (default "Escape", null for click-away only). Its ref is a DesktopViewHandle with enterFullscreen() (Keyboard Lock where the browser has it), exitFullscreen(), keyCombo(codes) and typeText(text), which is what a host toolbar drives. DesktopInput and contentBox() are exported from @fjarr/core for hosts that are not on React.

Exactly one input surface per page owns the keyboard: useDesktopFocus implements click-to-focus, a visible focus ring, Esc (configurable) to release, and releases on blur. Three robots on one page = three views, one keyboard owner; pointer works on hover in any of them (pointer events are per-element anyway). While not focused, a view forwards no keyboard events, so typing in the host app’s search box never reaches a robot.

Browser-reserved shortcuts

While a view is focused, keydown is preventDefault-ed for everything the page can intercept (Ctrl+W/T/N, Ctrl+Tab, F5, Ctrl+S…). What the page cannot intercept (Alt+Tab, Win/Super, Ctrl+Alt+Del on some platforms, Esc in fullscreen) needs the Keyboard Lock API — available only in fullscreen: <DesktopView> offers enterFullscreen() which calls navigator.keyboard.lock() and shows a one-time “press and hold Esc to exit” hint. Outside fullscreen, the special-keys toolbar covers those combos via key-combo. macOS Cmd is mapped to Super by default (configurable), because Cmd+letter combos mostly can’t be captured.

Cursor strategy

Two modes (docs/08 cursor, cursor-position):

Mode	How	Trade-off
Local cursor	agent captures without the cursor drawn in the video and sends cursor {shape_id, hotspot, image?} when the shape changes and cursor-position as the robot’s pointer moves; the client draws the shape at the local pointer while its operator is moving it, and at the robot’s position otherwise	the operator’s cursor feels instant (no round trip); everyone else still sees where the pointer is; needs cursor-shape access per backend: XFixes cursor image on X11, PipeWire cursor metadata on Wayland (below)
Embedded cursor	cursor rendered into the video by the agent; client sets cursor: none over the surface	works everywhere; cursor lags by the full glass-to-glass latency

Local mode is why remote desktops feel fast; embedded is the fallback for a backend with no cursor metadata. The mode is the backend’s, not each session’s: one capture serves every session watching a monitor, so they all get the same video.

<DesktopView> draws the local cursor in two ways. Under the operator’s own pointer, while the view may send input, it sets the surface’s CSS cursor to the shape (url(<image as PNG>) hotspot, default): the browser draws it, with no lag at all. Otherwise, a viewer or an operator whose pointer is elsewhere, an overlay draws the shape at the last cursor-position, offset by its hotspot. A hidden shape hides both. A monitor whose info says cursor: "embedded" has the cursor in its video already (the agent fell back for it, docs/23): there the view keeps the browser’s default cursor and draws no overlay.

On Wayland, the cursor comes from Fjarr’s own PipeWire reader. Mutter and the portal deliver the cursor as metadata on the PipeWire stream (cursor mode “metadata”: shape, hotspot and position beside frames drawn without it). But GStreamer’s pipewiresrc drops that metadata: its most verbose trace shows no cursor handling (ADR-0006). So the Wayland backends read the stream’s SPA_META_Cursor themselves, with libpipewire (already a dependency of fjarr-desktop-wayland, docs/14). They turn shape changes into cursor events and hand frames on to the media plane. It is built in M3, and local cursor is the default on Wayland as everywhere else. X11 needs nothing new: XFixes reports shape changes, and ximagesrc show-pointer=false captures without the cursor.

Capturing GNOME monitors as DMA-BUF (ADR-0041) changes nothing for the client: the cursor metadata arrives on DMA-BUF buffers exactly as on shared memory (position, shape and hotspot, measured on the mini-PC 2026-10-17), and metadata mode stays the default. A pointer that a robot application hides arrives as an invalid cursor id (0); the agent sends it as a hidden shape, and the view draws nothing. mutter sends the same id to a monitor the pointer is not on (by mutter’s source, not measured), so the agent takes it only from the monitor the pointer was last seen on, and restores that monitor’s shape when the pointer moves to it (docs/23). The DMA-BUF change’s unit tests check those rules (ReaderCursor.* and CursorArbiter.* in agent/tests/test_cursor_image.cpp). Neither CI fixture can hide the pointer from an application; the mini-PC’s scanout check runs with it hidden and asserts the hidden shape arrives (docs/15).

Virtual monitors (slice 3.5)

addDesktopMonitor(session, {width, height}) in @fjarr/core asks the robot for an extra screen of that size for this session (docs/08 add-monitor), for instance one sized to the operator’s window, and resolves with its monitor id once it is in the layout; removeDesktopMonitor(session, id) takes it away, and it goes by itself when the session ends. It arrives like any hot-plugged monitor: useMonitors lists it, <DesktopLayout> places it, and <DesktopView monitorId> shows it. The demo’s Desktop panel has “Add a screen”.

When the robot stops sharing

GNOME shows a screen-sharing indicator in the top bar while Fjarr captures, and its stop button ends the capture and the input (measured on the mini-PC, 2026-10-03). The agent reports it as sharing: stopped (docs/08) and keeps it stopped until a session sends resume-sharing. What happens then is the dashboard’s choice, so it is a prop:

• <DesktopView onRobotStop="ask"> (the default) shows “Sharing was stopped on the robot” over the view, with a Resume button, and sets data-fjarr-status="stopped-on-robot".
• <DesktopView onRobotStop="resume"> sends resume-sharing each time the stop arrives, so the person at the robot sees sharing come straight back. Nothing else is done: no backoff and no limit.

Both are available to a view_only view: resuming restores the screen and moves nothing (docs/10). Headless, useDesktopSharing(session) gives {state: "on" | "stopped", resume()} from DesktopSharing in @fjarr/core, one per session like DesktopClipboard. While stopped the robot offers no monitors, so a dashboard that shows its desktop only while useMonitors lists some must also show it while useDesktopSharing says stopped, or the message and the resume go with it (the demo did, 2026-10-03). The demo’s Desktop panel has a “Resume when the robot stops sharing” checkbox.

When there is nothing to show

<DesktopView> always says why it shows no picture, in its placeholder and in data-fjarr-status, and it never removes itself:

status	when	text
desktop-absent	desktop: absent (docs/08)	The robot’s desktop is not running. It may be restarting; this view comes back by itself.
stopped-on-robot	sharing: stopped	Sharing was stopped on the robot (and Resume, above)
monitor-disconnected	its monitorId left the layout	This monitor was disconnected from the robot. It comes back here when it is plugged in again.
no-display	the desktop runs with no monitor	No display is connected to the robot.

useDesktopState(session) gives "running" | "absent" | "unknown" (DesktopState in @fjarr/core, one per session); "unknown" until the robot has said, which a robot without a desktop never does. A dashboard that shows a desktop panel only while there are monitors loses all of this: the demo keeps its Desktop panel whenever the robot has a desktop (its state is known), and lets the view explain (it removed the panel during a GNOME Shell restart, mini-PC 2026-10-04).

Latency knobs (desktop-specific)

• RTCRtpReceiver.jitterBufferTarget = 0 on desktop tracks — trade smoothness for immediacy (cameras keep the default). Exposed as latencyMode: "interactive" | "smooth" on the track acquire options.
• select-tracks gains preference: "sharpness" | "motion" (docs/06): desktop text needs resolution, not fps; the agent maps it to encoder degradation preference and tier params.
• Input-to-photon measurement: requestVideoFrameCallback on the <video> plus fjarr.core/time-sync lets the client compute the docs/15 harness metric live; useDesktopLatency(session) surfaces it.

Clipboard

Browser side of the docs/06 offer/request model:

• Robot → browser: on clipboard-offer the client sends clipboard-read for text/plain and writes the text with navigator.clipboard.writeText(). The browser allows that only while the page is focused and the operator recently pressed a key or clicked, which is the case right after a copy on the robot (the operator’s Ctrl+C was that key). When the browser refuses, DesktopClipboard keeps the text and reports "needs-gesture", and the toolbar offers a “Copy from robot” button that writes it on click. There is one DesktopClipboard per session (acquireDesktopClipboard in @fjarr/core), however many monitors are shown, so a copy is read once; useDesktopClipboard(session) gives a toolbar its state and copyFromRobot().
• Browser → robot: the operator’s paste is caught on the focused input surface. A Ctrl+V or Cmd+V keydown is held back, not forwarded. The browser then fires paste on the surface with the clipboard’s text, without a permission prompt. The client sends clipboard-write, and once it is answered it sends Ctrl+V to the robot (key-combo). So the robot pastes what the operator pasted, never its previous clipboard. If no paste event follows within 300 ms (an empty or non-text clipboard), the held keys go to the robot as they were. Cmd+V on a Mac becomes Ctrl+V on the robot.
• Images (M3 3.5): a PNG copied on the robot is read as image/png when offered, in preference to text, and written with navigator.clipboard.write([new ClipboardItem({"image/png": blob})]), under the same gesture rule and “Copy from robot” fallback as text. A pasted image arrives in the same paste event as a file of type image/png; the client sends it as clipboard-write of image/png before the Ctrl+V, in preference to any text beside it. The limit is 8 MiB (docs/08).
• The browser hands the page its clipboard only on a paste in the view (Ctrl+V while it has focus). A paste started on the robot (right-click → Paste in its file manager) gets the robot’s own clipboard. Syncing on focus would close that; it is open question #41, unscheduled.
• Files on the clipboard route through fjarr.files (M4+); drag-and-drop onto the view uploads via the same capability. A browser cannot put files on the operator’s clipboard, so a copy of files on the robot can at most be offered as a download; real file copy in that direction is the native client’s (ADR-0029).

Sound (M3.5 slice 3.5.2) {#sound}

What the robot’s speakers play arrives as fjarr.desktop’s audio track, desktop-audio (docs/08, docs/23). <DesktopView> does not play it: sound is the host’s choice, as a microphone’s is, and a browser plays sound only after a gesture. The host mounts <AudioSink session trackId="desktop-audio"> (the fjarr.audio component; the track is demanded while it is mounted and nothing is sent otherwise) behind a control of its own, such as a “Sound” toggle in the toolbar slot, whose click is the gesture. The track is there only while the robot has an output, so the toggle shows only then (useTracks). It is one track for the whole desktop, played once in the page however many monitors or separate windows show the screen.

Ownership and view-only

Several operators can be on one desktop at once, the way people use a remote desktop to show a colleague something. Everyone sees the screen, and one of them has the pointer and keyboard (docs/10, the desktop domain):

• The toolbar shows who is in control, from control-state: “You”, “Anna (3 min)”, or nobody.
• Starting to type or move the pointer takes control when nobody holds it. The holder keeps it while they use it, and it frees 5 s after their last input.
• A take-control button claims it at once, even from an active holder, who sees “Bob took control”. A “request control” prompt comes with session-peers (docs/08, M5).
• Input sent without control is dropped by the agent. The client does not send it in the first place, and shows why.
• A grant with view_only: true renders without an input surface and without the button.

The web library exposes this as session.control, useControl(session, "desktop") and heldBy(error) (docs/21).

Core requirements for slice 2 (so M3 needs no core change)

1. Realtime and control publishers with the coalescing/deadman semantics of docs/21 (pointer at ≤ 60 Hz with seq; keys/buttons reliable).
2. Track acquire options: preference (→ select-tracks, an encoder-side knob) and latencyMode (→ the receiver’s jitterBufferTarget, a browser-side knob; never on the wire).
3. Envelope subscription for agent → client cursor events on the realtime class and clipboard-offer on control.
4. Manifest monitor geometry exposed via the track registry.
5. fjarr.core/time-sync available to consumers (useTimeSync).
6. A per-page focus registry in @fjarr/react (which view owns the keyboard) — small, but it must exist at the client level, not inside a component.
7. Renegotiation-safe track registry: manifest diffing by track_id, manifest_version ordering, and track handles that survive a track disappearing and rebind when it returns — the substrate for monitor hot-plug (docs/21).
8. Multi-window awareness for presentation mode: track handles expose the MediaStream so a view rendered into another same-origin document can attach it; the focus registry accepts a window per view (OS focus of a popup window = focus of that view); nothing in the core touches the global window/document without going through the view’s own — a portaled view must not observe the opener’s visibilitychange as its own.

Testing (docs/15)

Unit: letterbox math (fit/fill/native, HiDPI), held-state release on blur, repeat filtering, wheel normalization, focus arbitration with three views, manifest diffing on renegotiation (add/remove/mode-change/re-plug, stale manifest_version ignored), view rebinding policies. Browser (Playwright against robot-sim): type “åäö” and a Ctrl+Alt+Del combo into the sim’s xterm and read it back; Alt+Tab in fullscreen with keyboard lock; clipboard round trip both ways; cursor-shape event rendering; input-to-photon within docs/16 budgets; hot-plug via xrandr --setmonitor/--delmonitor on robot-sim (docs/07): add → visible < 2 s with zero dropped frames on the others, remove → placeholder, re-plug → same track_id, remove all → recover. Presentation mode (Chromium with the window-management permission pre-granted over CDP and a virtual two-screen display in CI): a click opens two fullscreen windows on two screens, each showing the mapped robot monitor with Keyboard Lock active; typing in either lands in the sim; closing one window releases its track and its held keys; closing the opener closes both; the mapping override survives a reload; the same test runs in mode: "route"; Firefox runs the degraded path (windows open, operator fullscreen per window) and must not error.