Grafana
Point Grafana straight at the database for load history, per-app usage and long-term charts.
The dashboard's Load history answers "what is this box doing"; Grafana answers "what has the fleet been doing", side by side with everything else you already graph. There is nothing to export and no agent to add: the readings are plain rows in the same MariaDB the hub writes to, so Grafana reads them with SQL.
Two provisioned dashboards ship in deploy/grafana/:
| Dashboard | What it shows |
|---|---|
| Fleet | Online/offline counts, load per core, memory, temperature, boxes reporting, monitor events. By group. |
| Per Device | One box: load, memory, storage, temperature, per-app CPU/RSS/process count, its monitor log. |
Provision it
Copy the two provisioning files and the dashboards into your Grafana, then set the four variables the datasource reads:
# provisioning
cp deploy/grafana/provisioning/datasources/magnemite.yml \
/etc/grafana/provisioning/datasources/
cp deploy/grafana/provisioning/dashboards/magnemite.yml \
/etc/grafana/provisioning/dashboards/
# the dashboard JSON, at the path that provider points to
mkdir -p /var/lib/grafana/dashboards/magnemite
cp deploy/grafana/dashboards/*.json /var/lib/grafana/dashboards/magnemite/MAGNEMITE_DB_HOST=db.internal:3306
MAGNEMITE_DB_NAME=magnemite
MAGNEMITE_DB_USER=grafana
MAGNEMITE_DB_PASSWORD=a-long-random-password
MAGNEMITE_DB_TIMEZONE=UTCRestart Grafana. The dashboards land in a Magnemite folder.
Timezone
Rows are written with the hub's clock. MAGNEMITE_DB_TIMEZONE has to be the timezone that clock
is in, or every point lands in the wrong hour — which looks like the fleet being busy overnight
and idle all day.
Both provisioning files are marked not editable in the UI on purpose: the JSON
in the repo is the source of truth, and an edit made in the browser is
overwritten on the next scan. Change a panel by editing it there, exporting the
JSON, and saving it back into deploy/grafana/dashboards/.
What the tables hold
| Table | One row is |
|---|---|
DeviceMetricSample | One device at one moment: load, cores, CPU percent, memory, swap, storage, temperatures |
DevicePackageMetricSample | One tracked app on one box at one moment: CPU, RSS, process count |
RotomSample | What Rotom said about one box at one moment: workers, request rate, its four flags |
MonitorEvent | Something the monitor saw or did |
Device | The box as of its last heartbeat: live, not history |
Three things about that data that decide how a panel should be written:
- A missing reading is null, never zero. No thermal sensor, an agent too
old to report memory, an app that is not running — all store nothing. Panels
keep
spanNulls: falseso a gap draws as a gap. Connecting across nulls turns "the scanner was dead from 3am" into a straight line. cpuCountis on the sample, not just on the device. A load average only means something against the core count of the box at the time, and a re-flashed box can come back with a different one. That is why every panel divides bys.cpuCountand not byd.cpuCount.- Per-app CPU is a share of one core. 200 means two cores pinned. It is the
number
topshows, deliberately not normalised against the core count.
Three newer columns
cpuPercent, swapTotalBytes and swapFreeBytes are on both Device and
DeviceMetricSample, so they chart like everything else. Two things about them:
cpuPercentis a share of one core, the same convention as the per-app figures: 250 is two and a half cores, and an eight-core machine at full tilt reports 800. Divide bys.cpuCountfor a 0–100 panel; nothing is clamped at 100, because clamping is how a saturated machine looks merely busy.- Swap zero is a real reading, null is not. A machine with no swap
configured stores
0; a device whose agent never looked stores nothing. Afree / totalpanel therefore has to guardtotal > 0rather than trustingspanNullsto hide it, or every box in the fleet draws a divide-by-zero gap.
Containerised devices need their own panels
A Redroid box is a container, and inside a container the load average and the thermal zones are the host's. So ten boxes on one VPS report ten identical load curves and ten identical temperatures, none of which is theirs.
A per-device load alert fires once per instance for one machine's problem
Ten notifications, ten device names, one machine actually in trouble — and the fleet's own ceilings cannot damp it, because nothing on that path knows those ten rows are one machine. Per device, chart load and temperature for real hardware only; per machine, chart them once, on the machine's own row.
Which devices are containers is a join, not a flag — an instance row with a
deviceId is a box that is a container:
-- Real hardware: boxes that are not a Redroid instance.
SELECT s.at, d.name, s.loadAvg1 / NULLIF(s.cpuCount, 0) AS `load per core`
FROM `DeviceMetricSample` s
JOIN `Device` d ON d.id = s.deviceId
LEFT JOIN `RedroidInstance` i ON i.deviceId = d.id
WHERE d.kind = 'BOX' AND i.id IS NULL AND $__timeFilter(s.at)
GROUP BY 1, 2 ORDER BY 1To chart a machine instead, filter d.kind = 'HOST' — the readings are in the
same table, which is the
point of a machine being a Device.
cpuPercent, memory, swap and storage stay perfectly good per instance; only
load and temperature belong to the machine.
The shipped Fleet dashboard filters on kind; your own panels have to too
Every panel in magnemite-fleet.json now carries d.kind = 'BOX'. Without it a VPS's load
average and its disk are averaged in with the boxes', and the fleet size is inflated by the
machines running it. Per Device is deliberately left alone: it draws one device at a time, and
picking a machine from the variable is a reasonable thing to want.
How far back it goes
Grafana does not extend the history — it only draws what is stored, and that is bounded by Settings → Tuning:
| Setting | What it does |
|---|---|
| Metric history → Device metrics → Sample every / Keep for | The two DeviceMetricSample tables |
| Metric history → Scanner (Rotom) → Sample every / Keep for | RotomSample, on its own clock |
| Monitoring → Keep history for | The same idea for MonitorEvent |
If you want a quarter of load history in Grafana, raise Keep for first.
Raising it does not recover what has already been pruned.
Big fleets
A 30-day panel over a large fleet scans a lot of rows. The samples are indexed on [deviceId, at]
and [at], which covers both the per-box and the fleet-wide queries, so the usual fix is a wider
$__interval (Grafana's min interval on the panel) rather than an index. If that stops being
enough, an hourly rollup table fed by a scheduled job is the next step — nothing in the schema
depends on the samples being raw.
Alerting
Grafana can alert on these queries, and for thresholds about trends, memory climbing all week, storage falling towards the point an install fails, it is the better place for it.
Do not move the fleet's own reflexes there. Settings → Monitoring acts on what a box reports and can restart an app or reboot the box; Grafana can only send a message. Anything whose right answer is "do something to the box" belongs in a monitor rule, not an alert. See Monitoring.