vps-health-api/agent/README.md

Workstation agent

Collects one snapshot of the machine it runs on and pushes it to POST /hosts/<id> on the health API. Cron runs it every 5 minutes; the dashboard reads the result through GET /hosts.

Zero dependencies — node:http / node:https and the metrics.js already shared with the server. Node 22+.

File	Role
push-metrics.js	Collect, build the payload, POST once, exit
run-push.sh	Cron wrapper: load the env file, check the install, exec node

This directory is not part of the server image: the Dockerfile copies package.json index.js metrics.js one by one, and a test pins that it stays that way. The agent ships by copying files onto a workstation, never by deploy.

What it deliberately does not do

No queue, no replay. One run is one attempt. A push that fails is dropped, logged, and the process exits non-zero — the next beat is five minutes away and a heartbeat from five minutes ago describes a machine that no longer exists. Nothing is spooled to disk, so nothing can ever be replayed to make the dashboard show a past that is no longer true.

No secret in the logs. run-push.sh is meant to be piped into logger, so everything the agent prints ends up in syslog and journald permanently. The failure path therefore reports only err.code and the HTTP status — never an error object, never the request options, never a response body, and never a header. That is also why run-push.sh must never gain a set -x: the shell would echo the token as it sources the env file.

The cost is real and accepted: a 400 tells you the server rejected the payload, not why. The reason is in the server's logs, and --dry-run (below) shows you the exact payload that was sent.

Install on a new workstation

1. Freeze a copy

The cron must point at a copy, never at the git working tree. The server is deployed by a manual trigger, so the checked-out repo drifts freely: a branch checkout in ~/claude-code/vps-health-api would silently change — or break — the heartbeat of the workstation, and nothing would say so.

push-metrics.js loads metrics.js from one directory up, so the copy keeps the repo layout:

mkdir -p ~/.local/share/maximus-host-agent/agent
cp metrics.js ~/.local/share/maximus-host-agent/
cp agent/push-metrics.js agent/run-push.sh ~/.local/share/maximus-host-agent/agent/
chmod +x ~/.local/share/maximus-host-agent/agent/run-push.sh

Re-run those three cp after any change to the agent — that copy step is the deployment.

2. Write the env file

~/.config/maximus-host-agent.env, mode 600 (the agent warns on any other mode). Use printf, not a heredoc: a trailing newline or space inside the token value is the most common install failure.

HOSTS_API_URL=https://health.lacompagniemaximus.com
HOSTS_INGEST_TOKEN=<the ingest token>
HOST_ID=thinkpad
# NODE_BIN=/usr/bin/node          # only if cron cannot find node (see below)
# HOST_AGENT_TIMEOUT_MS=10000     # push timeout, wall-clock; default 10000

chmod 600 ~/.config/maximus-host-agent.env

HOSTS_INGEST_TOKEN is the write-only token: it can push snapshots for an allowlisted id and nothing else. It is deliberately not HEALTH_TOKEN, so that the two can be rotated independently and so a workstation that only pushes never needs the read token.

Do not read more into that separation than it gives you. On the ThinkPad it buys no containment at all: that machine already stores HEALTH_TOKEN in cleartext for the defenseur-auto cron, so losing the laptop compromises both tokens. Rotate them together — see secret-rotation-ops.md.

HOST_ID must match ^[a-z0-9][a-z0-9-]{0,31}$ and be listed in the server's HOSTS_ALLOWED_IDS, otherwise the push comes back 403.

3. Dry run

Collect and print the payload without touching the network or needing any config:

node ~/.local/share/maximus-host-agent/agent/push-metrics.js --dry-run

Then one real push, by hand, before installing the cron:

~/.local/share/maximus-host-agent/agent/run-push.sh
# host-agent: ok id=thinkpad status=204 hostname=thinkpad-x1

Confirm the server side (from a machine that holds HEALTH_TOKEN):

curl -H "Authorization: Bearer $HEALTH_TOKEN" \
  https://health.lacompagniemaximus.com/hosts

The entry for your id should show online: true and a small ageSeconds.

4. Install the cron

crontab -e

*/5 * * * * $HOME/.local/share/maximus-host-agent/agent/run-push.sh 2>&1 | logger -t host-agent

Read what it did:

journalctl -t host-agent --since -1h

If the line works by hand but produces nothing under cron, it is almost always node: cron's PATH is /usr/bin:/bin, and a node installed by nvm or under /usr/local is not on it. Set NODE_BIN to an absolute path in the env file — the wrapper says so explicitly rather than failing silently.

One id per workstation

The server keeps one file per host id and the last write wins. Two workstations configured with the same HOST_ID therefore overwrite each other every five minutes, and nothing anywhere reports an error: the dashboard shows one host that looks perfectly healthy while its numbers come from whichever machine pushed last.

The signal to watch for is the hostname field in GET /hosts changing between polls while the id stays the same — plus CPU and memory that jump around without pattern. If you see that, two machines are sharing an id.

Give every workstation its own id and add it to HOSTS_ALLOWED_IDS on the server before installing the agent on it.

Exit codes

Code	Meaning
0	Snapshot accepted (server answered 2xx, normally 204)
1	Install problem: env file missing, a variable unset, malformed HOST_ID, node not found, metrics.js not found next to the agent
2	The push did not land: network error (code=…) or a non-2xx answer (HTTP …)

A 1 needs a human on the workstation. A 2 usually fixes itself on the next beat; a 2 that repeats for an hour means the API or the link is down.

Payload

Exactly the GET /health body minus logto and timestamp:

{ hostname, uptime,
  cpu:    { model, cores, loadAvg, usagePercent },
  memory: { totalGB, usedGB, freeGB, usagePercent },
  disk:   { totalGB, usedGB, freeGB, usagePercent } }

loadAvg is exactly three numbers (1, 5 and 15 minutes), as os.loadavg() returns them; the server answers 400 to any other length.

Sharing metrics.js with the server keeps the two in step, but it is not the guarantee — the frozen copy can lag behind a deployed server. The real contract is the server's own validation, and __tests__/agent.test.js checks the agent's payload against it (including a real snapshot through the real handler).

claude-usage agent (agent/claude-usage/)

A second, independent agent on the ThinkPad. Every 15 minutes it reads the Claude plan limits, aggregates the local Claude Code transcripts over the last 30 Toronto days, and pushes one schemaVersion: 1 snapshot to POST /claude-usage; the admin dashboard reads it through GET /claude-usage.

It lives in this repo, next to the server, because the snapshot contract and its sanitizer (claude-usage.js) are tested together: __tests__/agent-claude-usage.test.js builds a real snapshot from fixtures and checks that sanitizeClaudeUsage() keeps it unchanged. Like the host agent, it never enters the server image.

File	Role
aggregate.js	Read ~/.claude/projects/**/*.jsonl, build days and projects
limits.js	One GET to Anthropic's OAuth usage endpoint, mapped to limits.items
state.js	Local cache: frozen days and last good limits
push-claude-usage.js	Entry point: aggregate, limits, write the cache, push once, exit
run-claude-usage.sh	Cron wrapper: lock, load the env file, check the install, exec node

What it deliberately does not do

It never refreshes the OAuth token and never writes ~/.claude/.credentials.json. Refresh tokens are single use: refreshing from here would log Claude Code out. The file is only read; an expired token gives limits.status: "token_expired" and no call at all, until Claude Code itself refreshes it.

The Anthropic URL is pinned. It is a constant in limits.js, no environment variable changes it, and redirects are refused: the same process holds the OAuth bearer and a configurable push URL, and the bearer must only ever reach https://api.anthropic.com. CLAUDE_USAGE_API_URL must be https:// (the agent exits 1 otherwise).

One call, one push, no retry. The usage endpoint rate-limits hard and sends no Retry-After. When it fails, the snapshot carries the failure status (http_error, unavailable, ...) with the last good items and their fetchedAt, so the dashboard can show how old they are.

No secret in the logs. Same rule as the host agent: the output goes to logger -t claude-usage, so failures print fixed templates plus an HTTP status or an error code — never an error object, a response body, or a header — and run-claude-usage.sh must never gain a set -x.

Install

Freeze a copy — never point the cron at the git working tree:

mkdir -p ~/.local/share/maximus-claude-usage
cp agent/claude-usage/aggregate.js agent/claude-usage/limits.js agent/claude-usage/state.js agent/claude-usage/push-claude-usage.js agent/claude-usage/run-claude-usage.sh ~/.local/share/maximus-claude-usage/
chmod +x ~/.local/share/maximus-claude-usage/run-claude-usage.sh

Re-run that cp after any change to the agent: the copy step is the deployment. The code lives in ~/.local/share/, the cache in ~/.local/state/maximus-claude-usage/ (state.json and run.lock, created on the first run): wiping the copy never loses the cache, and vice versa.

Env file ~/.config/maximus-claude-usage.env, mode 600 (the wrapper warns on any other mode). Write it with printf, not a heredoc: a trailing newline in the token is refused with exit 1.

CLAUDE_USAGE_API_URL=https://health.lacompagniemaximus.com
CLAUDE_USAGE_INGEST_TOKEN=<the claude-usage ingest token>
# CLAUDE_USAGE_LIMITS=off        # skip the Anthropic call entirely
# NODE_BIN=/usr/bin/node         # only if cron cannot find node

chmod 600 ~/.config/maximus-claude-usage.env

CLAUDE_USAGE_INGEST_TOKEN is its own token: neither HOSTS_INGEST_TOKEN nor HEALTH_TOKEN works on POST /claude-usage, and it opens nothing else.

Dry run — prints the snapshot, pushes nothing, leaves the cache untouched, needs no env file (it still calls the Anthropic endpoint once unless CLAUDE_USAGE_LIMITS=off):

CLAUDE_USAGE_LIMITS=off node ~/.local/share/maximus-claude-usage/push-claude-usage.js --dry-run

Then one real run by hand, before installing the cron (the first one scans every transcript of the window and can take a while; the next ones only read files written since yesterday):

~/.local/share/maximus-claude-usage/run-claude-usage.sh
# claude-usage: ok status=204 limits=ok bytes=...

Crontab line:

*/15 * * * * $HOME/.local/share/maximus-claude-usage/run-claude-usage.sh 2>&1 | logger -t claude-usage

journalctl -t claude-usage --since -1h

A beat that finds the previous one still running (flock on ~/.local/state/maximus-claude-usage/run.lock) logs a run is already in progress, skipping and exits 0.

Exit codes

Code	Meaning
0	Snapshot accepted (2xx), or a skipped beat (lock held)
1	Install problem: env file missing, a variable unset, URL not https://, token with an invalid character, node or flock not found
2	The push did not land (network or non-2xx), or the snapshot exceeded 64 KiB (payload_too_large, nothing sent)

The limits failing is not an exit code: the snapshot is still pushed, with limits.status saying why.

Unreadable transcripts

A day is frozen in state.json two days after it ends, and never recomputed afterwards. A beat that cannot read part of ~/.claude/projects (a directory it cannot list, a transcript removed between the walk and its stat, a stream error) therefore freezes no new day: it still pushes its snapshot, exits as usual, and logs only a count:

claude-usage: warning: read_errors=N, no day frozen this beat

The next clean beat freezes the days that were waiting. A file that stays unreadable (left at chmod 000, for instance) blocks freezing on every beat: each run then rescans the whole 30-day window, slower but correct, and this log line is the only sign. Fix the file's permissions. To force a full recount after the fact, rm ~/.local/state/maximus-claude-usage/state.json.

Backfilling the history (one-shot, #55)

The server keeps every closed day in history/YYYY-MM.json from the first push after the deploy of #55. The days that already left the 30-day window then only live in the transcripts, which Claude Code deletes as they age: this backfill rebuilds them once. It runs from the repo, never from the cron install, and the agent itself does not change.

1. On the workstation, from the repo:

   node agent/claude-usage/backfill-history.js --out ~/claude-usage-backfill.json
   # claude-usage backfill: N day(s) from YYYY-MM-DD to YYYY-MM-DD, 24 window(s), read_errors=0
   

   It reads ~/.claude/projects only (never ~/.claude/.credentials.json), opens no connection, and writes only the --out file, in mode 600, refusing to overwrite it. It goes back --max-windows windows of 28 days (default 24, about 22 months; each window takes a few seconds), and stops at D-2: the last closed days are archived by the cron's next push anyway. Exit 1 with read_errors=N: some transcript could not be read, rerun before importing.

2. Copy it to the VPS and merge it inside the container, as node (the container's user, owner of the mount):

   scp ~/claude-usage-backfill.json ubuntu@vps-b0826277.tail3c811f.ts.net:/tmp/
   ssh ubuntu@vps-b0826277.tail3c811f.ts.net
   CTR=$(docker ps --format '{{.Names}}' | grep u8000gsg044wsk0oo0w884ok | head -1)
   docker exec -i "$CTR" node claude-usage-history.js --import - < /tmp/claude-usage-backfill.json
   # import: N day(s) read, A added, K already archived, M month(s) written
   rm /tmp/claude-usage-backfill.json
   

   An already archived date always wins: the import only adds the dates the archive lacks, so running it twice, or after the live archive has grown, changes nothing that is there. An import file with one invalid entry or a repeated date is refused whole (exit 1, nothing written). A month file the server cannot read back is skipped and left as is (exit 1, named on stderr).

3. Check through the API, with the read token: GET /claude-usage/history?from=<first month>&to=<last month>.

Then delete ~/claude-usage-backfill.json on the workstation.

PostgreSQL backup pull (agent/pg-backup-pull/)

A third, independent agent on the Lenovo. Once a day it pulls the age-encrypted dumps of the shared PostgreSQL cluster from the VPS, keeps 30 days of them in ~/backups/postgres/, and reports the pull to POST /backups/pull. The VPS side (backup-postgres.sh, 03:00 UTC), the status.json contract and the restore procedure live in la-compagnie-maximus/docs/postgres-ops.md, section Backup (la-compagnie-maximus#220).

File	Role
run-pg-backup-pull.sh	rsync over ssh, rotation, report; POSIX sh, no node
maximus-pg-backup-pull.service	systemd user unit, oneshot, retries on failure
maximus-pg-backup-pull.timer	OnCalendar=daily, Persistent=true

The only departure from the cron convention of the other agents is the timer. A cron line missed while the laptop is off is simply lost; Persistent=true runs the missed pass at the next boot. That catch-up run can start before Tailscale is up, hence Restart=on-failure every 15 minutes (8 tries at most in 4 hours).

What it never does. It never decrypts: the age private key is not on this machine (Vaultwarden plus a paper copy). It never deletes on the VPS side, and rsync only lets directories and *.age files through, so a misconfigured outbox cannot land a clear dump here. The key it uses (~/.ssh/pgbackup_pull_ed25519) can only run rrsync -ro on the outbox, from the Lenovo's Tailscale IP, on the classic sshd at 100.86.30.21:2222; the VPS host key is pinned in ~/.ssh/known_hosts_pgbackup (no trust on first use).

Install

mkdir -p ~/.local/share/maximus-pg-backup-pull ~/.config/systemd/user
cp agent/pg-backup-pull/run-pg-backup-pull.sh ~/.local/share/maximus-pg-backup-pull/
chmod +x ~/.local/share/maximus-pg-backup-pull/run-pg-backup-pull.sh
cp agent/pg-backup-pull/maximus-pg-backup-pull.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now maximus-pg-backup-pull.timer

As for the other agents, the copy step is the deployment: never point the unit at the git working tree. The user manager must run without an open session: loginctl show-user $USER -p Linger must say yes.

The env file ~/.config/maximus-pg-backup-pull.env (mode 600) is optional until the ingest route is deployed. Without it the pull runs and the report is skipped, with one log line. Then, written with printf:

BACKUPS_API_URL=https://health.lacompagniemaximus.com
BACKUPS_INGEST_TOKEN=<the backups ingest token>

The token reaches curl through a header file created 0600 and removed right after, never through argv.

Check

systemctl --user start maximus-pg-backup-pull.service
journalctl --user -t pg-backup-pull --since -1h
systemctl --user list-timers maximus-pg-backup-pull.timer

Exit codes

Code	Meaning
0	Pulled, rotated, reported (or report skipped without a token)
1	Install problem (missing key, known_hosts, binary) or rotation failure
2	The pull failed (pull_failed reported)
3	Pulled and rotated, but the report did not land