Rollouts
Canary, soak, concurrency, retries (started by hand or by the hub itself).
A rollout is a version, a target set and a policy. The hub turns it into one job per device and dispatches them as capacity allows. Start one from the fleet table (select boxes, or Update all).
Or take some vacations... and let the hub start them for you. See automatic rollouts.
The policy
| Field | What it does |
|---|---|
| Canary devices | The first N update alone; the rest wait for them to succeed. |
| Soak (min) | Extra wait after the canary succeeds, before the rest go. |
| Retry backoff (s) | Hold before the next attempt (60 by default, doubling, capped at 30 minutes). |
| Attempts per device | How many tries before a job is left failed. |
| Clean install | Uninstall first instead of upgrading in place. Wipes app data on every box. |
| Skip up-to-date | Leave boxes already on the version alone. Always off for a manual install. |
Targeting is one box, a selection, a device group, or the fleet. Groups are the useful unit past one site: they carry their own pre/post-install hooks and can cap concurrency below the fleet-wide limit.
Canary is worth setting even at three devices, most bad builds fail the same way on every box,
and the first three tell you as much as the first fifty. A failed canary parks the rollout at
PAUSED rather than pushing a bad build to the fleet.
The backoff exists for one failure in particular: a box that has just restarted
its system_server needs the best part of a minute, and a retry inside that
window fails instantly on hardware that is about to be fine. Zero retries on
the next scheduler tick; the Retry button ignores the hold entirely.
Concurrency
Settings → Tuning → Max concurrent jobs (10) caps the whole fleet; a group
can cap lower for a site on a thin uplink. The cap counts boxes
downloading or installing, which is what consumes your egress and their
/data.
A box itself takes one job at a time. Two rollouts that both include it queue a job each, and the second goes out when the first is finished, oldest first. A box that is swapping its own agent binary takes none until it is back on the new one, because the swap ends in a restart that would take an install down with it.
A rollout can also wait for another: it shows as pending, after the one it
waits for, and starts on its own once that one has finished with every job a
success. That is how a box joining its group gets its
MITM after its game. Start now sends a
waiting rollout without waiting.
A fleet-wide rollout to 200 boxes is roughly 35 GB off your server. Concurrency paces it; it does not reduce it. Check your traffic allowance before the first one. Check the troubleshooting topic for more information.
When things fail
| Situation | What happens |
|---|---|
| Device offline | The job stays queued and dispatches when the agent reconnects. |
| Job failed | Re-queued up to the attempt limit, after the backoff. Bytes already on disk make the retry cheap. |
| Attempts exhausted | Retry and Retry all failed in the UI. A rollout that had finished is reopened for it. |
| Job stalled | No progress for the stall timeout (Settings → Tuning, 900 s) and the hub re-queues it. |
| Hub restarted | Jobs survive in the database; the scheduler picks them up on the next tick. |
QUEUED ──► RUNNING ──┬─► COMPLETED
├─► PAUSED (canary failed — needs a human)
└─► CANCELLED (you stopped it)Each job carries the box's own log lines alongside its phase and percentage, so
a failure is diagnosable from the dashboard without adb: the pm output, a
sha256 mismatch, the free-space refusal.
Automatic rollouts
Per app, on Settings → Apps. With it on, the hub polls each source on its own interval, caches the newest approved version, and creates a rollout with that app's canary and soak (inside the time window, if set).
It never downgrades, never runs two rollouts for one app at once, does nothing when every box is already current, and waits out the cooldown.
| Field | What it does |
|---|---|
| Approve new versions automatically | Off, a human ticks approve on the Versions page first. |
| Window start / end | Jobs created outside it stay queued. Update only in your non peak hours. |
| Cooldown (min) | How long after the last automatic rollout finished before another may start. You can set this setting to 0 to ship immediately (not recommended). |
Polling interval is per source (Settings → Version sources → Check every, 15 minutes by default), because an index is somebody else's server. A changed interval takes effect within seconds.
Auto-approve plus no canary means an upstream mistake reaches N boxes unattended. Keep a canary of a few devices and a soak long enough that a dead scanner would show up, as a monitoring alert.