Skip to content

Slice 6a Review

Retrospective review of slice 6a per docs/13 and docs/20, run before the slice was committed. The slice closes ADR-0007 on webrtcbin with an estimator of our own and makes adaptive bitrate real: NACK/RTX and keyframe feedback in the offer, a per-peer estimate from transport-wide feedback, tier encoders that follow their viewers inside a band, and viewers moved between tiers on their own link. The web client shows the agent’s view per track. One reviewer pass on the C++, the simulator and the TypeScript, plus what the traces and the lab found — which, this time, was most of the design’s calibration.

Found byDefectFix
the first /stats trace under badthe estimator cut 7.5 % per window on 15 % random loss and reached the floor in 3 s: sending less never reduces random lossa loss cut needs throughput to be collapsing (the peer acknowledges under 70 % of what is sent) window after window, at most once per second; random loss with throughput intact is left to retransmission
the same traceone lost packet in a seven-packet feedback window read as 14 % lossloss is judged over a rolling second of windows, ≥ 10 packets
the same trace, recoverythe estimate could not climb out of the thumbnail tier: growth was capped at 1.5× what arrived, and a demoted viewer receives 300 kbpsgrowth is uncapped while nothing is lost (30 %/s below the level before the last cut); the delay rule catches an overshoot within three windows
the same trace, lossy±15 ms jitter tripped the delay rule (1 ms, two windows) at the thumbnail’s tiny windows3 ms for three windows in a row; a shaper’s queue is steady, jitter flips sign
lab, bad on a lone viewerthe session died of an SCTP transport error: clamped to the shared-encoder band (2 Mbps) above a 1.5 Mbit shaper for the 2 s a tier switch takes, the kernel queue grew past what the data channel toleratesa tier with one subscriber is clamped to the floor, not the band (docs/23)
lab, measuring the reactionthe bandwidth-stats events arrived 3 s late under bad: they ride the impaired link behind the videothe gate reads /stats through the introspection port, which the lab’s netem now exempts like docker/lab/netem.sh
lab, three viewersnetem on lo for the simulator viewer shaped both directions through one queue: its pings died behind the video and the session closed on the heartbeatthe simulator runs in dev and the robot’s egress toward that address alone is impaired (netemToward)
lab, three viewersthe simulator’s session read as a lossless link at the ceiling: its webrtcbin answerer sent no retransmission requests, and an unacknowledged feedback window was taken as “no loss”do-nack on the simulator’s transceivers; a window with no acknowledged packets is no evidence and holds the estimate
simulator netem-lossy (both directions of lo impaired)a lost feedback packet makes the next window report everything it did not cover as lost (50–70 % in one window), and 5 % reverse-path loss demoted the viewer to the floorthe loss cut needs three consecutive collapsed windows: a shaper collapses every window, a lost feedback packet one; unit-tested with a lost feedback packet every eighth window
simulator netem-lossy from dev, one-directionalwith only the media path jittered, the webrtcbin receiver still reported 20–50 % loss per window at 5 % real loss (Chromium reports ~5 % under the same profile) and the estimate fell to the floorthe profile’s rate control is recorded, not asserted, with a GStreamer receiver; the browser lab asserts it; the netem-* scenarios now run opsim from dev against the robot’s egress with the introspection port exempt on every device
lab, three viewers, run after the othersthe congested viewer was not demoted within the scenario’s 10 s: a session that starts already behind a bad link waits 3–4 s for transport-wide feedback to carry bitrates at all before the estimator has anything to judge, then 2 s below the band, then the 1 s samplethe scenario allows 25 s and says why; the browser gate measures an established session’s reaction, which is what docs/16 is about
lab, three viewers, onceselect-tracks got no result: its reply crosses the impaired link behind whatever video is already queued, and the shared encoder is held at 4 Mbps by the clean viewersa control request on that link gets a 15 s budget and one retry (the agent is unchanged: this is what a bad link does to a request)
the same traceavg-delta-of-delta is INT64_MIN before the first feedback, and the first windows carry packet counts with zero bitratesboth are read as “nothing to learn yet”; the session also treats an unchanged twcc-stats read as no new feedback, since rtpsession keeps the last window until the next arrives
simulatora nested wait_for deadlocked the congested-viewer scenarioinbox readers inside predicates take no lock
#SeverityFindingResolution
1highthe plan’s “estimate per (peer, track)” does not match the feedback, which is per transportone estimate per peer; the peer’s enabled tracks share it in proportion to their tier targets; docs/23 amended
2mediumthe promotion hold of 10 s plus the climb from the floor missed the 15 s gate5 s hold and 30 %/s recovery growth; measured 11.6 s to promotion, 12.6 s to 90 % of the target
3mediumthe plan put the estimator’s numbers into the session snapshot, which walks GStreamer elements onlythey are in /stats (rate, twcc, rtx, per-tier kbps) and in bandwidth-stats; docs/23 amended
4lowthe simulator’s netem-* comments still described an offer without NACK/RTXupdated; the scenarios gained a rate-control check

Measured (software encoder, the lab’s Chromium, this host)

Section titled “Measured (software encoder, the lab’s Chromium, this host)”
CaseResult
bad on a lone viewerestimate below 2 Mbps within ~1 s of the impairment, encoder output below 1.7 Mbps within ~3–4 s, demoted at ~2–3 s; 3–4 frames decoded per 6 s at the 5 fps thumbnail tier (a repair round trip per frame at 15 % loss); promoted 11.6 s and back above 3.6 Mbps 12.6 s after the link cleared
lossy (5 %) on a 30 fps track, 20 s1416 retransmission requests served, 0 keyframe requests, 2 freezes, 624 frames decoded, bitrate held
three viewers, one behind badthe congested viewer demoted 8–9 s after enabling (TWCC warm-up on a session that starts behind the link, then 2 s below the band, then the 1 s sample), kept receiving, promoted after the link cleared; the two clean viewers on the active tier at ≥ 3.5 Mbps throughout (clean 3.7 Mbps); three consecutive runs

1–5. tests/stack/ratecontrol.spec.ts (four tests) and the simulator’s netem-lossy, netem-4g, netem-bad with their rate-control check, plus congested-viewer — green on this host; the estimator, the tier policy and the client’s agent-stats store each have unit tests. 6. Every earlier gate green; see the commit.

  • The delay rule is a threshold on the average delta-of-delta, not a trendline filter with an adaptive threshold as in GCC; it held under every docs/25 profile, and a real Wi-Fi link is the next calibration.
  • A control run with retransmission disabled (gate 3 as planned) was not built: the numbers under lossy speak for themselves, and a config switch to turn repair off has no product use.
  • The bad profile at the thumbnail tier decodes slowly; forward error correction would help there at a bandwidth cost everywhere (docs/08 leaves it off).
  • Two test marginalities on this host were measured to be pre-existing, not regressions: the simulator’s netem-bad “frames keep arriving” bound (7–10 whole access units per 5 s window, unchanged with the encoder pinned at the pre-slice-6a fixed 300 kbps — 15 % loss costs most multi-packet frames, since only about 44 % of five-packet frames survive it; the bound is now 5 and the 3 s gap check is what guards a stall), and the slice-4 camera three-viewer test, which fails about one run in three on a loaded machine with the slice-6a changes stashed as well.
  • A viewer whose link is far below the shared encoder’s rate has its control channel queued behind the video for the two seconds the demotion hysteresis takes; the per-consumer leaky queue bounds what the agent holds, not what the network queues. A separate transport for control, or pacing a new subscriber’s first seconds, is an M3 question.
  • Passthrough tracks (adaptive: false) are slice 6b.