Development Environment

Everything runs in containers; the host needs only Docker + VS Code. The environment is self-verifying: make doctor is the authority on whether your setup can do what the specs assume.

Quickstart

git clone <repo> fjarr && cd fjarr
cp .env.example .env            # an .env from before 2026-09-24 needs FJARR_WEBHOOK_URL
                                # then check RENDER_GID matches your host:
stat -c %g /dev/dri/renderD128  # → RENDER_GID in .env
# recommended, once per host: let ThreadSanitizer run inside containers (below)
sudo sysctl -w vm.mmap_rnd_bits=28 && echo 'vm.mmap_rnd_bits=28' | sudo tee /etc/sysctl.d/60-tsan.conf
code .                          # "Reopen in Container"
# post-create runs pnpm install, cargo fetch, cmake configure, doctor
make doctor

Or without VS Code: docker compose up -d dev robot-sim, then docker compose exec dev bash.

Services (docker-compose.yml)

Service	Profile	Purpose
dev	default	the toolbox: Ubuntu 26.04 / C++ (clang 21) / GStreamer 1.28 + Rust + Node 22 (see the Dockerfile’s commented package groups; ADR-0022)
robot-sim	default	Xvfb fake robot desktop at :99 (openbox + moving apps); watch it at http://localhost:6080 (noVNC)
fjarr-server	demo, stack	the production sidecar image, built from signaling/ only
demo-backend	demo	the TS “customer backend” beside the sidecar — mints real HS256 grants with FJARR_GRANT_HS256_SECRET
demo-robot	demo	the C++ “customer robot” capturing robot-sim
demo-dashboard	demo	Vite dev server on http://localhost:5173
rtsp-sim	demo	a network-camera stand-in: GStreamer’s RTSP server on rtsp://rtsp-sim:8554/pattern, the demo robot’s fjarr.camera rtsp track (slice 4)
coturn	turn	TURN relay in use-auth-secret mode (ephemeral creds only)
browser	lab	headless Chromium with CDP on http://localhost:9222 and fake media devices — the browser lab
desktop-fixture	desktop	GNOME’s mutter headless with one virtual monitor, PipeWire and a test window: backend E’s capture and EIS input without a GPU or a lab machine. make desktop-fixture-test (docs/15)
desktop-robot	desktop	the reference daemon with backend module E from the build, attached to the fixture’s desktop through the session helper’s socket: make desktop-e2e (docs/23)

make demo-up = docker compose --profile demo up -d — the full customer topology. The dev and robot-sim containers share /tmp/.X11-unix via the x11sock volume, which is why DISPLAY=:99 works inside dev. The demo dashboard’s sidebar has a role picker (operator / developer): the demo backend mints the grant for the role on the next connect, and only the developer role’s grant carries fjarr.introspect, so its Diagnostics panel shows the robot’s live pipeline graphs while an operator sees “not available for this role” (docs/24). The demo backend and the dashboard run from the mounted sources: after pulling a change to either, docker compose --profile demo restart demo-backend demo-dashboard (and demo-robot after make agent-build).

The doctor {#doctor}

make doctor → .devcontainer/doctor.sh. Checks, and what a failure means:

Check	On failure
toolchains (cmake/cargo/node/…)	image build incomplete — rebuild dev
GStreamer elements (webrtcbin, nicesrc, ximagesrc, …)	missing plugin package — see Dockerfile groups. Encoder elements are not here: they belong to a family, below
x264enc absent	if present: GPL plugin leaked in — ADR-0011 violation, fix the image
/dev/dri accessible	RENDER_GID in .env ≠ host render group; fix + docker compose build dev
encoder family vaapi: the elements exist and encode	iHD driver problem — see GPU troubleshooting. The va plugin registers per VA device, so a machine with no usable one has no vah264enc at all rather than a broken one. A warning instead of a failure when FJARR_MEDIA_ENCODER=software: an NVIDIA or AMD runner has deliberately chosen the CPU path (ADR-0025), which is not a broken setup
DISPLAY=:99 + ximagesrc capture	robot-sim not up, or stale X socket — docker compose restart robot-sim
libei/pipewire/libevdev pkg-config	dev packages missing from image
/dev/uinput	WARN by default; opt in via the uinput override

The network tunnel in the lab

Where a target runs matters. A docker compose up or --force-recreate run from inside dev hands the host’s daemon /workspace as the bind source — a path that exists only in the container — and every service it recreates comes up with an empty workspace, reporting “demo-robot not built yet”. So targets that recreate services (tun-up, tunnel-isolation, tunnel-collision, m45-gate) run on the host. The browser suites are the opposite case: their defaults name compose services (browser:9222, demo-dashboard:5173), so they run inside dev, where those names resolve — make tunnel-login does the docker compose exec dev itself when invoked from the host, and CI provides the host’s view of the same stack through E2E_* instead. make m45-gate refuses to run inside dev for this reason.

fjarr.net (docs/27) attaches to a TUN interface that someone else created — on a robot the installer, in the lab make tun-up:

make tun-up          # fjarr0 on demo-robot and on dev, then the robot with fjarr.net on
make opsim-tunnel    # real IP over fjarr:stream:fjarr.net (docs/27 gate)
make tun-down

dev and demo-robot therefore carry /dev/net/tun and NET_ADMIN as a standing grant. Neither the agent nor fjarr-opsim uses the capability: docker/lab/tundev.sh creates the device as root and hands it to the agent’s user, and both processes then attach with none of their own — which is the property the lab exists to keep honest, since it is what lets a real robot run the agent unprivileged.

dev holds the operator end rather than demo-robot because fjarr-opsim normally shares the robot’s network namespace, where both tunnel addresses would be local and the kernel would short-circuit the link it is meant to test. make opsim-tunnel runs from dev for that reason, as opsim-netem does.

ROS 2 lives in sidecars on each end’s network namespace, started in the right order by make tun-up ROS=1 — a DDS participant created before the interface exists never sees it (docs/27#lifecycle), which is why the order matters and why the robot’s sidecar waits. make tunnel-ros runs the gate; docker/lab/dds-isolate.sh on|off|show is what makes it mean anything, by taking away the direct path between the two containers.

Opt-in overrides (conscious privilege grants)

Host desktop capture — develop against your real screen:

xhost +local:   # host, once per session
docker compose -f docker-compose.yml -f docker-compose.host-x11.yml up -d dev
# inside dev: DISPLAY is your host display now

Caveat: grants the container your entire screen + input access. Revert with xhost -local: and plain docker compose up -d dev.

uinput experiments:

sudo modprobe uinput
docker compose -f docker-compose.yml -f docker-compose.uinput.yml up -d dev

Caveat: the container can synthesize input on the host kernel.

Host webcam for fjarr.camera (slice 4) — the demo robot’s webcam track reads /dev/v4l/by-id/<name>; pass the host’s video devices in:

ls -l /dev/v4l/by-id/          # which node the …-video-index0 name points at
FJARR_WEBCAM_DEVICE=/dev/video2 FJARR_DEMO_WEBCAM=usb-Acme_Cam-video-index0 \
  docker compose -f docker-compose.yml -f docker-compose.camera.yml --profile demo up -d demo-robot

Without the override the track is unavailable (reason in curl localhost:7381/sources) and simply absent from the manifest — the docs/26 missing-device behaviour, exercised by CI. The demo reads /dev/video0 by default; set FJARR_DEMO_WEBCAM to a /dev/v4l/by-id name (the stable form real config uses) or another node. A container’s devices: are static, so unplug/replug is verified by the unit tests and on bare metal, not through this override. fjarr-agent --probe-source '{type = "v4l2", device = "/dev/video0"}' validates a camera before it goes into fjarr.toml.

GPU / VA-API troubleshooting

• vainfo errors inside dev → check LIBVA_DRIVER_NAME=iHD (set in the image) and that /dev/dri is mapped (compose devices:).
• Permission denied on renderD128 → .env RENDER_GID must equal stat -c %g /dev/dri/renderD128; rebuild dev after changing.
• No /dev/dri at all (CI, cloud VM) → doctor degrades to WARN; software encoders cover dev work but never performance claims.
• Machine-specific note (Meteor Lake NUCs and newer): only the iHD driver supports encode; mesa-va-drivers is present for completeness.
• The iGPU stuck at its floor clock (VA-API sessions at 10–15 fps while gst-launch alone manages 30, the software encoder streams 30, and the robot’s CPU is idle): the package power limit throttled the GPU and the clock stayed at its minimum afterwards. Read /sys/class/drm/card*/gt/gt0/rps_act_freq_mhz under load and throttle_reason_pl1; on a laptop check the power source and profile. A day of heavy builds and sanitizer runs did this once, helped along by the sim’s glxgears rendering unthrottled through llvmpipe on ~3.5 cores (removed since). The software encoder path — what CI measures — is unaffected; VA-API numbers taken in that state are not performance data.

TURN sanity check

docker compose --profile turn up -d coturn
# mint an ephemeral credential by hand (docs/10#turn):
u="$(($(date +%s)+600)):manual-test"
p=$(printf %s "$u" | openssl dgst -sha1 -hmac "$TURN_SECRET" -binary | base64)
docker run --rm --network host --entrypoint turnutils_uclient \
  coturn/coturn:4.6-alpine -u "$u" -w "$p" -p 3478 -e 127.0.0.1 -n 2 127.0.0.1
# Valid cred ⇒ allocation succeeds, then "403 (Forbidden IP)" on channel
# bind — that's coturn's default loopback-peer block, i.e. auth WORKED.
# A wrong cred fails earlier with "Cannot complete Allocation".

Caches & volumes

Named volumes keep rebuilds fast: cargo-registry, pnpm-store, ccache, x11sock. docker compose down -v wipes them (first build after that is slow again).

Browser lab

make lab-up starts the browser service (headless Chromium, CDP at http://127.0.0.1:9222/json from the host) beside fjarr-server; make e2e runs the Playwright/CDP suites from dev (make e2e-loopback needs no server); pnpm fjarr-lab <command> drives the same browser ad hoc (open, net lossy, wire --follow, profile cpu 10, memory --cycles 20…). Artifacts land in web/e2e/out/. The dev image carries the docker CLI with the host socket mounted so the lab can apply netem, pause or restart stack containers from inside — set DOCKER_GID in .env to the group owning /var/run/docker.sock. Details and the environment variables: docs/25.

ThreadSanitizer in the container

TSan keeps its shadow memory at fixed addresses; with the 32 bits of ASLR entropy recent kernels default to, the program’s own mappings can land on them and TSan refuses to start (“incompatible memory layout … unable to disable ASLR”). Its usual self-repair — re-exec with personality(ADDR_NO_RANDOMIZE) — is blocked by Docker’s default seccomp profile, and so is setarch -R. The fix is a host kernel setting, which every container inherits (the recommended one-time step in the quickstart):

sudo sysctl -w vm.mmap_rnd_bits=28                                # now
echo 'vm.mmap_rnd_bits=28' | sudo tee /etc/sysctl.d/60-tsan.conf   # after reboots

28 bits of randomisation instead of 32 is the sanitizer maintainers’ own recommendation and immaterial on a development machine. The doctor probes this functionally (it compiles and runs a one-line TSan program) and WARNs when make agent-test-tsan would not run. TSan is a gate (docs/15): CI’s cpp job runs inside a container and cannot set the sysctl, so the gate runs in the runner-level e2e job, which sets it first. agent/tests/tsan.supp lists the modules TSan cannot see into (GLib, GStreamer, libnice, libsoup are not instrumented, so their locks are invisible and every hand-off through their queues would otherwise be reported); the RAII kit pairs its own hand-offs (posts, sources, promises, thread-pool jobs) with an acquire/release so a race with both ends in fjarr code is always reported.

Ubuntu 26.04’s coreutils are uutils {#uutils}

The 26.04 base image ships the Rust rewrite of coreutils (install --version says uutils), and it is not GNU in every corner. The one that has bitten: install -o 1002 -g 1002 fails with “invalid user” when that uid has no passwd entry in the container, where GNU install takes the bare number. chown 1002:1002 does take it. A script that hands a file to a host-supplied uid (the lab’s robot-services publishes its ssh key to the operator’s uid) therefore does install -m 600 … and then chown (found 2026-09-29: every nightly tun-up failed on the self-hosted runner, uid 1002, while GitHub’s runner passed because its uid 1001 is the image’s robot user by chance). When a coreutils command behaves unlike GNU in a 26.04 container, suspect uutils first.

Debugging a hung process {#debugging-a-hung-process}

A test or agent that hangs says nothing until its stacks are read, and the dev container cannot attach a debugger: Docker’s default profile withholds CAP_SYS_PTRACE, so gdb -p fails (“ptrace: Inappropriate ioctl for device”) even as root.

• If it hangs reproducibly, launch it under gdb — a child needs no ptrace capability. timeout -s INT stops it where it hangs:

  timeout -s INT 40 gdb -q -batch -ex run -ex 'thread apply all bt 30' \
    --args build/debug/agent/tests/fjarr-tests --gtest_filter='Suite.name'

• If gdb changes the timing or the tool around it matters (heaptrack, valgrind), run it in a throwaway container from the same image with ptrace allowed, and attach as root once it hangs:

  docker run -d --name hang --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
    -v "$PWD":/workspace -w /workspace fjarr-dev sleep infinity
  docker exec -d hang sh -c 'heaptrack -o /tmp/ht ./build/release/agent/tests/fjarr-tests ... > /tmp/run.log 2>&1'
  docker exec hang pgrep -af fjarr-tests                      # its pid, once it hangs
  docker exec -u root hang gdb -q -batch -p <pid> -ex 'thread apply all bt 30' > bt.txt
  docker rm -f hang

  docker exec runs as the image’s user, which cannot ptrace: -u root is needed.

• Symbols for the distro’s libraries come from Ubuntu’s debuginfod: DEBUGINFOD_URLS=https://debuginfod.ubuntu.com gdb -ex 'set debuginfod enabled on' ….

• Read the stacks for two threads waiting on each other (a lock one holds that the other wants); g_cond_wait inside gst_pad_push is a blocking pad probe, and __lll_lock_wait under dl_iterate_phdr is the dynamic-loader lock — docs/15 has the heaptrack case this found.

Nightly CI and the self-hosted GPU runner

.github/workflows/nightly.yml (docs/15) runs the 200-cycle soak, valgrind, heaptrack, the netem-* scenarios and the latency harness every night and on demand. Its runs-on is the repository variable FJARR_NIGHTLY_RUNNER, falling back to GitHub’s ubuntu-24.04.

This repository is public, so a self-hosted runner may only serve workflows that untrusted code cannot reach. Anyone can open a pull request, and a workflow that runs on it executes their code on your machine. nightly.yml is safe by construction: it triggers on schedule and workflow_dispatch, which only ever run from the default branch and which a fork cannot trigger. ci.yml must stay on hosted runners — it runs on push and pull_request. Also set Settings → Actions → Fork pull request workflows to require approval for all outside collaborators. A machine that is also someone’s desktop shares its time through fjarr-lab: CI’s by night, theirs by day.

Setting one up:

1. On the machine: Ubuntu 24.04 or later, Docker Engine with Compose v2, the runner’s user in the docker group (and render/video when /dev/dri exists), vm.mmap_rnd_bits=28 in /etc/sysctl.d/ (TSan, above) or passwordless sudo for the job’s sysctl, make, and enough disk for the dev image plus build trees.
2. GitHub → the repository’s Settings → Actions → Runners → New self-hosted runner; install it as a service and give it a label you will point at. Register it on the repository, not the organisation: an org runner lives in a runner group, and a group does not serve public repositories unless you tick “Allow public repositories” — off by default, for the reason in the box above. The job simply waits forever with no error, labels matching and the runner idle. Flipping that toggle also exposes the machine to every public repository in the org, now and later; a repository-scoped runner is as capable and stays scoped.
3. GitHub → Settings → Secrets and variables → Actions → Variables: FJARR_NIGHTLY_RUNNER = <label>. Unset it to fall back to hosted.

A render node is not an Intel one. Our hardware encode path is VA-API (ADR-0022; NVENC is open question #2), and an NVIDIA card presents /dev/dri/renderD128 while offering no VA-API H.264 encoder. The nightly therefore probes the pipeline rather than trusting the device node, and sets media.encoder to software when the probe fails — the explicit, never-silent fallback of docs/23. The doctor treats a failed VA-API smoke as a warning when the encoder has been chosen as software, and as a failure otherwise. So a big AMD or NVIDIA box is a fine runner; it just will not exercise the VA-API path.

For the latency harness the same runner takes FJARR_LATENCY_LABEL (the hardware, for the CSV) and FJARR_LATENCY_STRICT=1 (hold the run to the docs/16 budgets). Turn strict on only for a machine nobody is using: the budgets assume a quiet host, and the harness refuses a run that could not decode the source rate anyway.

Lab machines and fjarr-lab {#lab-machines-and-fjarr-lab}

Some tests need real hardware: monitors and ghost screens, a GPU encoder, a camera, a reboot. Those run on lab machines, self-hosted runners that are also people’s machines by day (ADR-0033). The rules in the box above apply to all of them. They serve only workflows that trigger on schedule and workflow_dispatch from main, never push or pull_request, and they are registered on the repository.

fjarr-lab runs on each of them: a shell script with systemd units, no build step, the same on amd64 and on arm64 boards. It decides when the runner is online:

fjarr-lab status                # CI (job, since) · idle in the CI window · reserved (by whom, until) · next change
fjarr-lab reserve               # the machine is yours now: waits for a running job, then takes the runner offline
fjarr-lab release               # back to the schedule
fjarr-lab window 00:00-06:00    # the CI window, in /etc/fjarr-lab.conf
fjarr-lab window off            # no schedule: online unless reserved (a dedicated runner)
fjarr-lab reboot                # a lab job's reboot: waits until the runner is idle (the job has ended), then reboots

• The CI window. Two systemd timers start and stop the runner’s service at the window’s edges. At boot the unit works out the right state for the time of day, so a job that reboots the machine (the unattended-access test) gets its runner back.
• fjarr-lab alone starts the runner. The runner’s own unit stays disabled: enabled, it started at every boot beside the apply unit, which then waited for the job it found while the runner took the next queued one, and a reserved mini-PC ran a queue of rebooting lab jobs (2026-10-31). apply, reserve and release disable the unit where ./svc.sh install enabled it, and give it a drop-in (KillMode=mixed, TimeoutStopSec=60): GitHub’s KillMode=process left Runner.Listener taking jobs after a stop. A listener outside the unit counts as online, and going offline stops it.
• A reservation wins over the window until release, so an evening at the machine is not interrupted at midnight.
• Nothing is killed halfway. When the window closes, or someone reserves the machine, the runner stops taking jobs, waits for a running one (grace 30 min by default, then it is cancelled), and goes offline. status shows which of these is happening.
• Jobs start from a baseline: undo the previous configuration, install the build under test, apply the job’s own settings. Manual experiments and CI runs do not poison each other (docs/15).
• CI’s side. Lab jobs are scheduled for 22:15 UTC, shortly after the window opens in summer (00:15; in winter they wait for it), and must finish, reboots included, before it closes. GitHub starts them hours late (docs/15). A job dispatched by hand during the day waits in GitHub’s queue until the machine’s window opens; one that no runner takes within 24 hours is cancelled.
• A lab machine powers on by itself when power returns (its firmware’s restore-on-AC-power-loss setting: Power On), so a power cut costs at most the night it happens in. The mini-PC (an ASUS PN51) did not: it was off from 2026-10-14 to 10-16, most likely after a power cut during the lab’s power work (its journal ends without a shutdown), and two nights were lost. Pending (Erik): set its firmware. After power or cabling work at a lab machine, check fjarr-lab status on it before midnight.
• A red night reaches someone. nightly-alert (docs/30) keeps an issue labelled nightly-red open while a lab run or the Nightly is red, including a lab job still waiting for its runner after the window. It cannot tell a reservation from a machine that is off, so a reserved night is red too.

Workflows pick machines by label, never by name. Every lab machine carries fjarr-lab, its architecture, and what it can test:

Machine	Labels	Tests	Status
mini-PC (Ryzen 7 5700U, GNOME 50)	fjarr-lab amd64 desktop gnome vaapi	M3 desktop: backend E on hardware, ghost and chained monitors, unattended access after a reboot, input-to-photon	M3
GPU desktop (RTX 2080 Ti)	fjarr-lab amd64 nvcodec	the nvcodec family (M2.6) and the nightly media suites	M3 (in use by day; CI window at night)
Raspberry Pi 4	fjarr-lab arm64 rpi4 camera-libcamera v4l2m2m	the Pi camera; the Pi 4’s hardware H.264 encoder	planned
Raspberry Pi 5	fjarr-lab arm64 rpi5 camera-libcamera	the Pi camera; the Pi 5 has no hardware H.264 encoder, so the software path on ARM	planned
Jetson	fjarr-lab arm64 jetson camera-zed nvv4l2	the ZED vendor package; the nvv4l2 family	planned

A machine joins in four steps: register GitHub’s runner with its labels and install it as a service (the runner’s own ./svc.sh install); install fjarr-lab as root (make lab-runner-install, which puts the script, its apply unit and two timers in place); set RUNNER_SERVICE in /etc/fjarr-lab.conf and the window (sudo fjarr-lab window 00:00-06:00); and add the machine’s labels to the nightly matrix. make lab-test checks fjarr-lab’s logic anywhere, against a fake systemctl.

Pipeline introspection

With the demo profile up, make introspect (no arguments) opens the agent’s live pipeline viewer at http://localhost:7381/ (docs/24) and prints the token to paste. The endpoint is loopback-only inside the robot container, so the demo profile binds it on the container’s interface with the dev token FJARR_INTROSPECT_TOKEN (.env.example; the Makefile reads .env for it) and publishes it on the host’s loopback only; the viewer asks for the token once per tab. make introspect PIPELINE=<id> [FORMAT=json|dot] prints one pipeline, make introspect SUMMARIES=1 every pipeline’s summary, and from the host curl -H "Authorization: Bearer $FJARR_INTROSPECT_TOKEN" localhost:7381/pipelines is the raw API — the first thing to look at when media misbehaves, and what /verify and the e2e tests assert against.

Running the demo robot from the agent image

The demo robot runs in the dev image with the workspace mounted, so an agent change is make agent-build and a container restart, never an image rebuild. The product artifact is the fjarr-agent image (docker/agent/Dockerfile, repo-root context), built from the .debs since M2.5 (docs/26#releases), so run make deb first. The runtime stage is Ubuntu 26.04 plus the fjarr-agent package and VA-API drivers: no compilers, the fjarr user with the published uid 10001, the viewer under /usr/share/fjarr/viewer, a healthcheck on the live endpoint, and a build that fails if anything GPL the agent could load is present (x264enc, avdec_*, libx264, libxvidcore — ADR-0011, docs/14). A demo stage builds the demo robot against libfjarr-dev with find_package(fjarr) and puts it on the runtime image, which is the topology an embedding customer uses. To run the demo from it:

make deb
docker compose -f docker-compose.yml -f docker-compose.image.yml --profile demo up -d --build demo-robot

CI runs the lab smoke against this variant on every push and discards the image; publishing to GHCR arrives with M2.5’s release process (docs/26).

Make targets

doctor · agent-configure/build/test · agent-test-asan/tsan · agent-raii-gate · agent-leaks-selftest · opsim / opsim-all / opsim-soak (OPSIM_CYCLES) / opsim-netem (NETEM_PROFILE) (the docs/23 operator simulator against the demo robot) · agent-leaks (SCENARIO=, recreates the robot with GStreamer’s leaks tracer and prints what stayed alive) · agent-memcheck · agent-heaptrack (docs/15 memory safety, nightly) · introspect (PIPELINE=, FORMAT=: the endpoint from the terminal, docs/24) · signaling-run/test/clippy · web-dev/build/lint · sim-up · demo-up/down · stack-up · website-dev/build · docs-lint · docs-links · fmt · lint. .vscode/tasks.json wraps the same targets — terminal and IDE never diverge.