Magnemite
Operating it

Grafana

Point Grafana straight at the database for load history, per-app usage and long-term charts.

The dashboard's Load history answers "what is this box doing"; Grafana answers "what has the fleet been doing", side by side with everything else you already graph. There is nothing to export and no agent to add: the readings are plain rows in the same MariaDB the hub writes to, so Grafana reads them with SQL.

Two provisioned dashboards ship in deploy/grafana/:

DashboardWhat it shows
FleetOnline/offline counts, load per core, memory, temperature, boxes reporting, monitor events. By group.
Per DeviceOne box: load, memory, storage, temperature, per-app CPU/RSS/process count, its monitor log.

Provision it

Copy the two provisioning files and the dashboards into your Grafana, then set the four variables the datasource reads:

# provisioning
cp deploy/grafana/provisioning/datasources/magnemite.yml \
   /etc/grafana/provisioning/datasources/
cp deploy/grafana/provisioning/dashboards/magnemite.yml \
   /etc/grafana/provisioning/dashboards/

# the dashboard JSON, at the path that provider points to
mkdir -p /var/lib/grafana/dashboards/magnemite
cp deploy/grafana/dashboards/*.json /var/lib/grafana/dashboards/magnemite/
MAGNEMITE_DB_HOST=db.internal:3306
MAGNEMITE_DB_NAME=magnemite
MAGNEMITE_DB_USER=grafana
MAGNEMITE_DB_PASSWORD=a-long-random-password
MAGNEMITE_DB_TIMEZONE=UTC

Restart Grafana. The dashboards land in a Magnemite folder.

Timezone

Rows are written with the hub's clock. MAGNEMITE_DB_TIMEZONE has to be the timezone that clock is in, or every point lands in the wrong hour — which looks like the fleet being busy overnight and idle all day.

Both provisioning files are marked not editable in the UI on purpose: the JSON in the repo is the source of truth, and an edit made in the browser is overwritten on the next scan. Change a panel by editing it there, exporting the JSON, and saving it back into deploy/grafana/dashboards/.

What the tables hold

TableOne row is
DeviceMetricSampleOne device at one moment: load, cores, CPU percent, memory, swap, storage, temperatures
DevicePackageMetricSampleOne tracked app on one box at one moment: CPU, RSS, process count
RotomSampleWhat Rotom said about one box at one moment: workers, request rate, its four flags
MonitorEventSomething the monitor saw or did
DeviceThe box as of its last heartbeat: live, not history

Three things about that data that decide how a panel should be written:

  • A missing reading is null, never zero. No thermal sensor, an agent too old to report memory, an app that is not running — all store nothing. Panels keep spanNulls: false so a gap draws as a gap. Connecting across nulls turns "the scanner was dead from 3am" into a straight line.
  • cpuCount is on the sample, not just on the device. A load average only means something against the core count of the box at the time, and a re-flashed box can come back with a different one. That is why every panel divides by s.cpuCount and not by d.cpuCount.
  • Per-app CPU is a share of one core. 200 means two cores pinned. It is the number top shows, deliberately not normalised against the core count.

Three newer columns

cpuPercent, swapTotalBytes and swapFreeBytes are on both Device and DeviceMetricSample, so they chart like everything else. Two things about them:

  • cpuPercent is a share of one core, the same convention as the per-app figures: 250 is two and a half cores, and an eight-core machine at full tilt reports 800. Divide by s.cpuCount for a 0–100 panel; nothing is clamped at 100, because clamping is how a saturated machine looks merely busy.
  • Swap zero is a real reading, null is not. A machine with no swap configured stores 0; a device whose agent never looked stores nothing. A free / total panel therefore has to guard total > 0 rather than trusting spanNulls to hide it, or every box in the fleet draws a divide-by-zero gap.

Containerised devices need their own panels

A Redroid box is a container, and inside a container the load average and the thermal zones are the host's. So ten boxes on one VPS report ten identical load curves and ten identical temperatures, none of which is theirs.

A per-device load alert fires once per instance for one machine's problem

Ten notifications, ten device names, one machine actually in trouble — and the fleet's own ceilings cannot damp it, because nothing on that path knows those ten rows are one machine. Per device, chart load and temperature for real hardware only; per machine, chart them once, on the machine's own row.

Which devices are containers is a join, not a flag — an instance row with a deviceId is a box that is a container:

-- Real hardware: boxes that are not a Redroid instance.
SELECT s.at, d.name, s.loadAvg1 / NULLIF(s.cpuCount, 0) AS `load per core`
FROM `DeviceMetricSample` s
JOIN `Device` d ON d.id = s.deviceId
LEFT JOIN `RedroidInstance` i ON i.deviceId = d.id
WHERE d.kind = 'BOX' AND i.id IS NULL AND $__timeFilter(s.at)
GROUP BY 1, 2 ORDER BY 1

To chart a machine instead, filter d.kind = 'HOST' — the readings are in the same table, which is the point of a machine being a Device. cpuPercent, memory, swap and storage stay perfectly good per instance; only load and temperature belong to the machine.

The shipped Fleet dashboard filters on kind; your own panels have to too

Every panel in magnemite-fleet.json now carries d.kind = 'BOX'. Without it a VPS's load average and its disk are averaged in with the boxes', and the fleet size is inflated by the machines running it. Per Device is deliberately left alone: it draws one device at a time, and picking a machine from the variable is a reasonable thing to want.

How far back it goes

Grafana does not extend the history — it only draws what is stored, and that is bounded by Settings → Tuning:

SettingWhat it does
Metric history → Device metrics → Sample every / Keep forThe two DeviceMetricSample tables
Metric history → Scanner (Rotom) → Sample every / Keep forRotomSample, on its own clock
Monitoring → Keep history forThe same idea for MonitorEvent

If you want a quarter of load history in Grafana, raise Keep for first.

Raising it does not recover what has already been pruned.

Big fleets

A 30-day panel over a large fleet scans a lot of rows. The samples are indexed on [deviceId, at] and [at], which covers both the per-box and the fleet-wide queries, so the usual fix is a wider $__interval (Grafana's min interval on the panel) rather than an index. If that stops being enough, an hourly rollup table fed by a scheduled job is the next step — nothing in the schema depends on the samples being raw.

Alerting

Grafana can alert on these queries, and for thresholds about trends, memory climbing all week, storage falling towards the point an install fails, it is the better place for it.

Do not move the fleet's own reflexes there. Settings → Monitoring acts on what a box reports and can restart an app or reboot the box; Grafana can only send a message. Anything whose right answer is "do something to the box" belongs in a monitor rule, not an alert. See Monitoring.

On this page