Monitoring and automations

A box can be online, heartbeating every twenty seconds, and scanning nothing: the scanner died, Pokémon GO fell out of focus, an ANR dialog is waiting for somebody to press ENTER, the loop stalled. None of that shows in the fleet table, because by every measure it has the box is fine.

Settings → Monitoring turns those into rules. A rule is a signal, how many bad readings in a row it takes to count, and a ladder of what to do, soft first, hard second.

The Monitoring page in the sidebar is it's history: what the rules saw and did across the fleet, which is where "why did that box reboot at 3am" is answered.

What it watches?Copy Anchor Link

The first five are knowable only on the box, so they need a recent agent, the tab counts how many boxes are behind, and rules skip a box that cannot answer. The rest the hub sees for itself and work on every box today; the Rotom ones need Rotom turned on.

What the box is costing itselfCopy Anchor Link

Three more, and these are the cheapest signals in the feature: no probe, no agent version, nothing new on the wire. The agent has reported all of it on every heartbeat since before monitoring existed — it is what the Load card draws — and the hub simply never compared it against anything.

The disk one is the one to turn on first, and not for the reason it looks: an install already refuses on low space and fails the job. A rule that clears caches at 10% turns a failed rollout into nothing happening at all.

The temperature one only tells you, and there is no second rung to write. No action on the catalog cools a box down, and rebooting one does not — it comes back to the same cupboard. The answer is air.

A box whose ROM exposes no thermal sensor reads unknown rather than cold, which is common on TV hardware and means the rule simply never fires there. That is the usual rule working: unknown neither counts nor resets.

The three Rotom signals are deliberately exclusive, so one fault never lights two ladders: a box Rotom has lost is only ever Rotom lost the box, a box with nothing allocated is only ever no workers, and doing nothing is read only where both of those are fine.

Two caveats worth knowing before switching them on:

• No workers is normal on a fleet with more boxes than accounts. The spares are idle on purpose. Scope that rule to a group, or raise its threshold.
• Doing nothing is silent unless Rotom measures request rates, which it only does in requests mode, or proxy mode with inspect. Where it does not measure, the signal reads unknown rather than idle — which is why turning it on cannot start rebooting a fleet that never reports a rate.

On a machineCopy Anchor Link

A rule declares which kind of device it is about — a box or a machine — and four signals exist for the second:

Swap counts as headroom in the first one, because on these machines it is exactly what stands between fifty workers and the OOM killer. A machine with no swap reports zero rather than nothing, so adding it is safe and a genuinely absent reading stays unknown.

The last two are read from what the machine said on its last heartbeat, not from its row:

• HOST_DOCKER_DOWN is the agent reporting while docker does not answer it. The machine's stored docker version deliberately keeps the last one ever seen, so a newer agent's finding is not wiped by an older one's silence — which makes the column history, and a daemon being down a question about right now. A hub that has just restarted reads unknown for one heartbeat, which the startup grace already covers.
• REDROID_INSTANCE_DOWN counts only the instances marked present: a removed one keeps its row, and counting it would have a drained machine alarm forever. All the known ones running is healthy even while another is still unknown, so a machine recovers on the container that came back instead of waiting for one nothing has looked at.

RESTART_INSTANCE is the narrowest remediation in the project: up -d --no-recreate, naming only the services that are down. A container that is running is not touched, nothing is removed, and the machine is never rebooted — which is the whole point of the action existing, since rebooting takes every other box on the machine down for one box's fault.

AGENT_OFFLINE needs nothing new: pointed at machines, it already means exactly the right thing, because a machine is a Device and its socket is gone the same way a box's is.

Why these are signals and not probe rulesCopy Anchor Link

A shell probe on the machine looks cheaper — one HEALTH_CHECK_FAILED rule per check, no enum values, no code. It does not work, and the way it fails is silent: rulesFor dedupes on signal + package name and keeps one winner per key, so three HEALTH_CHECK_FAILED rules with no package collapse into one. Two of your three checks would simply never run — no error, no probe, nothing in the feed to say why.

The ladder?Copy Anchor Link

A rule does the gentle thing first and the expensive thing only when the gentle one did not work:

Scanner offline
  failure 1 → restart the app
  failure 2 → reboot the box

The counter resets the moment the signal goes good, so a box that recovers never reaches the reboot. A rung already spent is not spent again — except the last one, which repeats at the cooldown, because there is nothing stronger left to try.

Restart one Redroid container is the host-side counterpart of restart the app: the smallest thing that fixes a wedged instance. Reboot the box is deliberately not overloaded to mean "reboot the VPS" — that takes every box on the machine down at once, which is a much larger decision than the signal that would reach for it.

It runs on the machine, over the machine's own socket, so a machine that is not answering has the rung refused rather than spent — the ladder still has it when the machine comes back. It is counted as a hard action for the same reason a reboot is: the container is the box, so restarting it cold-boots that Android and every app on it comes back from nothing.

The three Rotom actions travel Rotom's socket, not the agent's, which is what makes them the only remediation left on a box whose agent has died — every other action on the list needs the box reachable from Magnemite. The mirror of that is also true and worth planning a ladder around: Rotom cannot act on a box it has lost, so on a Rotom lost the box rule the bottom rung is the agent-side Reboot the box, and on the other two it is Reboot the box through Rotom.

Reboot the box through Rotom counts against reboots per box per day and sits under the post-reboot grace exactly like the agent-side one. A different socket, the same boot cycle.

Every shell-backed one takes an override command, {pkg} standing in for the rule's package. The catalog exists so an alert can say "cleared the cache and restarted" rather than quoting a command line; the override exists because your MITM is not the one this was written against.

ProbesCopy Anchor Link

Three signals need the box to look at something, and what to look at differs per MITM — so the probe is part of the rule:

• Shell command — runs as root, passes on exit 0, optionally also having to match a pattern. dumpsys activity services | grep -e MappingService is the whole "is the scanner up" check.
• Pattern in a log — reads the last N lines of a file and counts matches. Can also fail when nothing has been written for a while, which is what a stalled loop looks like from outside.
• HTTP endpoint — a GET from the box itself; 2xx is healthy.

A log probe can carry a second pattern for work getting done, and fail only when faults outnumber it — ten "seconds since last ping" lines mean nothing next to a hundred completed jobs and everything next to two:

Log file          /data/local/tmp/atlas.log
Lines to read     200
A fault           \[HEALTH CHECK\] ([0-9]+) seconds since last ping\.
Faults before     10
Work looks like   I \| Worker

Probes run every heartbeat, inside half the beat. Whatever does not finish in that budget reports nothing at all — never a failure. They are skipped entirely during an install, because an install force-stops the scanner on purpose and every probe would report the damage Magnemite asked for.

The refusalsCopy Anchor Link

Most of this feature is things it will not do. Each is a way a watchdog turns into a fleet rebooting itself in a loop:

The numbersCopy Anchor Link

The master switch is on Settings → Monitoring. Everything else is the How it watches card on Settings → Tuning, because every number in it is measured against the heartbeat or the offline timeout, which sit directly above.

The monitor pass itself is not configurable: it runs once per heartbeat, because a pass reads what the boxes last said and they say it once per beat. "Box is unreachable" counts from the offline timeout — to wait longer, raise that rule's own threshold, which is per group.

Where alerts goCopy Anchor Link

The channels — Discord, ntfy and web push — are set up on Settings → Notifications, beside everything else the fleet announces, and Notifications covers each of them. Configure none of them and the rules still run: remediation never depends on a notification going out.

Which rules reach which channel is decided on the rule, in Announce it — a dropdown of the three, and picking none is a rule that acts quietly. Set up a channel there; choose who uses it here.

None of the channels has a minimum severity. There used to be one per channel, and two filters in series meant a rule set to announce could silently not, with nothing on the rule to say why. One filter, on the rule, where somebody looking for the reason is already looking.

A rule's Severity changes how an alert arrives, never whether it does. It drives ntfy's priority dial, which is what decides whether a phone in a pocket makes a sound — INFO is quiet, WARN is high, and CRITICAL is max, which on Android gets through Do Not Disturb — and only CRITICAL pings the Discord role.

What the channels do share is the repeat window, on the Monitoring alerts card of the same tab: the same signal on the same box is not announced again inside it, 30 minutes by default. The point of it is that a flapping box does not announce itself thirty times, and a window per channel would let it do exactly that by taking turns. The action still happens and is still recorded.

Individual rules can be silenced without being switched off — Announce it set to nothing. "Pogo not in focus" ships that way: a box briefly on the launcher is common enough that announcing every one teaches people to ignore the channel. The same control is how a rule goes to Discord but not to a phone at 3am, without either channel having to know about it.

A rule that announces can also announce the all-clear — the second switch next to it in the rule dialog, off by default. On, the fault is followed by one more message when the signal goes good again, which is how "Box unreachable" becomes "Box unreachable recovered" in the channel rather than only in the feed. It goes to the same channels the fault did — a recovery announced somewhere the fault was not is a message about nothing — and it ignores the repeat window, which has just been filled by the fault it ends. Only an episode that actually reached the rule's threshold gets one — an all-clear for a fault nobody was told about is noise.

Press Send test on a channel's card before trusting it — see Notifications.

A channel that stops working never stops the remediation — the box is still restarted, and the Monitoring page records that the message did not arrive. An event counts as announced when any channel took it: a Discord message that landed is not made un-sent by ntfy being down at the same moment.

ScopeCopy Anchor Link

A rule applies to every box, or to one device group. A group-scoped rule replaces the fleet-wide rule for the same signal and package rather than adding to it, or two ladders would race over one fault and the box would be restarted twice. That is the seam for a fleet running Atlas at one site and Aegis at another.

Which kind of device is the other half of that filter, and it is not optional: a rule is about boxes or about machines, never both. Existing rules are about boxes, which is the default and what every seeded rule is.

Both mistakes would act, not just alert, which is why the filter is a field on the rule rather than something the evaluator infers.

The group is also where that site declares which MITM it runs and what to write into its config. A group-scoped rule and its group's MITM package are two halves of the same answer, so they are worth keeping in step: a rule probing Atlas's log on a group that runs Aegis is a rule that never fires.

The seeded rules are not invented: their thresholds and ladders are what aconf's atlas_monitor.sh has done on this hardware for years. The probe strings are Atlas's, which is exactly why they are editable rows.