vps-health-api/agent/README.md

# Workstation agent

Collects one snapshot of the machine it runs on and pushes it to
`POST /hosts/<id>` on the health API. Cron runs it every 5 minutes; the
dashboard reads the result through `GET /hosts`.

Zero dependencies — `node:http` / `node:https` and the `metrics.js` already
shared with the server. Node 22+.

| File | Role |
|------|------|
| `push-metrics.js` | Collect, build the payload, POST once, exit |
| `run-push.sh` | Cron wrapper: load the env file, check the install, exec node |

This directory is **not** part of the server image: the `Dockerfile` copies
`package.json index.js metrics.js` one by one, and a test pins that it stays
that way. The agent ships by copying files onto a workstation, never by deploy.

## What it deliberately does not do

**No queue, no replay.** One run is one attempt. A push that fails is dropped,
logged, and the process exits non-zero — the next beat is five minutes away and
a heartbeat from five minutes ago describes a machine that no longer exists.
Nothing is spooled to disk, so nothing can ever be replayed to make the
dashboard show a past that is no longer true.

**No secret in the logs.** `run-push.sh` is meant to be piped into `logger`, so
everything the agent prints ends up in syslog and journald permanently. The
failure path therefore reports **only** `err.code` and the HTTP status — never
an error object, never the request options, never a response body, and never a
header. That is also why `run-push.sh` must never gain a `set -x`: the shell
would echo the token as it sources the env file.

The cost is real and accepted: a `400` tells you the server rejected the
payload, not why. The reason is in the server's logs, and `--dry-run` (below)
shows you the exact payload that was sent.

## Install on a new workstation

### 1. Freeze a copy

The cron must point at a copy, **never at the git working tree**. The server is
deployed by a manual trigger, so the checked-out repo drifts freely: a branch
checkout in `~/claude-code/vps-health-api` would silently change — or break —
the heartbeat of the workstation, and nothing would say so.

`push-metrics.js` loads `metrics.js` from one directory up, so the copy keeps
the repo layout:

```
mkdir -p ~/.local/share/maximus-host-agent/agent
cp metrics.js ~/.local/share/maximus-host-agent/
cp agent/push-metrics.js agent/run-push.sh ~/.local/share/maximus-host-agent/agent/
chmod +x ~/.local/share/maximus-host-agent/agent/run-push.sh
```

Re-run those three `cp` after any change to the agent — that copy step *is* the
deployment.

### 2. Write the env file

`~/.config/maximus-host-agent.env`, mode `600` (the agent warns on any other
mode). Use `printf`, not a heredoc: a trailing newline or space inside the token
value is the most common install failure.

```
HOSTS_API_URL=https://health.lacompagniemaximus.com
HOSTS_INGEST_TOKEN=<the ingest token>
HOST_ID=thinkpad
# NODE_BIN=/usr/bin/node          # only if cron cannot find node (see below)
# HOST_AGENT_TIMEOUT_MS=10000     # push timeout, wall-clock; default 10000
```

```
chmod 600 ~/.config/maximus-host-agent.env
```

`HOSTS_INGEST_TOKEN` is the write-only token: it can push snapshots for an
allowlisted id and nothing else. It is deliberately *not* `HEALTH_TOKEN`, so
that the two can be rotated independently and so a workstation that only pushes
never needs the read token.

Do not read more into that separation than it gives you. On the ThinkPad it
buys no containment at all: that machine already stores `HEALTH_TOKEN` in
cleartext for the `defenseur-auto` cron, so losing the laptop compromises both
tokens. **Rotate them together** — see `secret-rotation-ops.md`.

`HOST_ID` must match `^[a-z0-9][a-z0-9-]{0,31}$` and be listed in the server's
`HOSTS_ALLOWED_IDS`, otherwise the push comes back `403`.

### 3. Dry run

Collect and print the payload without touching the network or needing any
config:

```
node ~/.local/share/maximus-host-agent/agent/push-metrics.js --dry-run
```

Then one real push, by hand, before installing the cron:

```
~/.local/share/maximus-host-agent/agent/run-push.sh
# host-agent: ok id=thinkpad status=204 hostname=thinkpad-x1
```

Confirm the server side (from a machine that holds `HEALTH_TOKEN`):

```
curl -H "Authorization: Bearer $HEALTH_TOKEN" \
  https://health.lacompagniemaximus.com/hosts
```

The entry for your id should show `online: true` and a small `ageSeconds`.

### 4. Install the cron

```
crontab -e
```

```
*/5 * * * * $HOME/.local/share/maximus-host-agent/agent/run-push.sh 2>&1 | logger -t host-agent
```

Read what it did:

```
journalctl -t host-agent --since -1h
```

If the line works by hand but produces nothing under cron, it is almost always
`node`: cron's `PATH` is `/usr/bin:/bin`, and a node installed by nvm or under
`/usr/local` is not on it. Set `NODE_BIN` to an absolute path in the env file —
the wrapper says so explicitly rather than failing silently.

## One id per workstation

The server keeps **one file per host id** and the last write wins. Two
workstations configured with the same `HOST_ID` therefore overwrite each other
every five minutes, and nothing anywhere reports an error: the dashboard shows
one host that looks perfectly healthy while its numbers come from whichever
machine pushed last.

The signal to watch for is the `hostname` field in `GET /hosts` **changing
between polls** while the id stays the same — plus CPU and memory that jump
around without pattern. If you see that, two machines are sharing an id.

Give every workstation its own id and add it to `HOSTS_ALLOWED_IDS` on the
server before installing the agent on it.

## Exit codes

| Code | Meaning |
|------|---------|
| 0 | Snapshot accepted (server answered 2xx, normally `204`) |
| 1 | Install problem: env file missing, a variable unset, malformed `HOST_ID`, node not found, `metrics.js` not found next to the agent |
| 2 | The push did not land: network error (`code=…`) or a non-2xx answer (`HTTP …`) |

A `1` needs a human on the workstation. A `2` usually fixes itself on the next
beat; a `2` that repeats for an hour means the API or the link is down.

## Payload

Exactly the `GET /health` body minus `logto` and `timestamp`:

```
{ hostname, uptime,
  cpu:    { model, cores, loadAvg, usagePercent },
  memory: { totalGB, usedGB, freeGB, usagePercent },
  disk:   { totalGB, usedGB, freeGB, usagePercent } }
```

`loadAvg` is exactly three numbers (1, 5 and 15 minutes), as `os.loadavg()`
returns them; the server answers `400` to any other length.

Sharing `metrics.js` with the server keeps the two in step, but it is not the
guarantee — the frozen copy can lag behind a deployed server. The real contract
is the server's own validation, and `__tests__/agent.test.js` checks the agent's
payload against it (including a real snapshot through the real handler).

## claude-usage agent (`agent/claude-usage/`)

A second, independent agent on the ThinkPad. Every 15 minutes it reads the
Claude plan limits, aggregates the local Claude Code transcripts over the last
30 Toronto days, and pushes one `schemaVersion: 1` snapshot to
`POST /claude-usage`; the admin dashboard reads it through `GET /claude-usage`.

It lives in this repo, next to the server, because the snapshot contract and
its sanitizer (`claude-usage.js`) are tested together:
`__tests__/agent-claude-usage.test.js` builds a real snapshot from fixtures and
checks that `sanitizeClaudeUsage()` keeps it unchanged. Like the host agent, it
never enters the server image.

| File | Role |
|------|------|
| `aggregate.js` | Read `~/.claude/projects/**/*.jsonl`, build `days` and `projects` |
| `limits.js` | One GET to Anthropic's OAuth usage endpoint, mapped to `limits.items` |
| `state.js` | Local cache: frozen days and last good limits |
| `push-claude-usage.js` | Entry point: aggregate, limits, write the cache, push once, exit |
| `run-claude-usage.sh` | Cron wrapper: lock, load the env file, check the install, exec node |

### What it deliberately does not do

**It never refreshes the OAuth token and never writes
`~/.claude/.credentials.json`.** Refresh tokens are single use: refreshing from
here would log Claude Code out. The file is only read; an expired token gives
`limits.status: "token_expired"` and no call at all, until Claude Code itself
refreshes it.

**The Anthropic URL is pinned.** It is a constant in `limits.js`, no
environment variable changes it, and redirects are refused: the same process
holds the OAuth bearer and a configurable push URL, and the bearer must only
ever reach `https://api.anthropic.com`. `CLAUDE_USAGE_API_URL` must be
`https://` (the agent exits 1 otherwise).

**One call, one push, no retry.** The usage endpoint rate-limits hard and sends
no `Retry-After`. When it fails, the snapshot carries the failure status
(`http_error`, `unavailable`, ...) with the last good items and their
`fetchedAt`, so the dashboard can show how old they are.

**No secret in the logs.** Same rule as the host agent: the output goes to
`logger -t claude-usage`, so failures print fixed templates plus an HTTP status
or an error code — never an error object, a response body, or a header — and
`run-claude-usage.sh` must never gain a `set -x`.

### Install

Freeze a copy — never point the cron at the git working tree:

```
mkdir -p ~/.local/share/maximus-claude-usage
cp agent/claude-usage/aggregate.js agent/claude-usage/limits.js agent/claude-usage/state.js agent/claude-usage/push-claude-usage.js agent/claude-usage/run-claude-usage.sh ~/.local/share/maximus-claude-usage/
chmod +x ~/.local/share/maximus-claude-usage/run-claude-usage.sh
```

Re-run that `cp` after any change to the agent: the copy step *is* the
deployment. The code lives in `~/.local/share/`, the cache in
`~/.local/state/maximus-claude-usage/` (`state.json` and `run.lock`, created on
the first run): wiping the copy never loses the cache, and vice versa.

Env file `~/.config/maximus-claude-usage.env`, mode `600` (the wrapper warns on
any other mode). Write it with `printf`, not a heredoc: a trailing newline in
the token is refused with exit 1.

```
CLAUDE_USAGE_API_URL=https://health.lacompagniemaximus.com
CLAUDE_USAGE_INGEST_TOKEN=<the claude-usage ingest token>
# CLAUDE_USAGE_LIMITS=off        # skip the Anthropic call entirely
# NODE_BIN=/usr/bin/node         # only if cron cannot find node
```

```
chmod 600 ~/.config/maximus-claude-usage.env
```

`CLAUDE_USAGE_INGEST_TOKEN` is its own token: neither `HOSTS_INGEST_TOKEN` nor
`HEALTH_TOKEN` works on `POST /claude-usage`, and it opens nothing else.

Dry run — prints the snapshot, pushes nothing, leaves the cache untouched, needs
no env file (it still calls the Anthropic endpoint once unless
`CLAUDE_USAGE_LIMITS=off`):

```
CLAUDE_USAGE_LIMITS=off node ~/.local/share/maximus-claude-usage/push-claude-usage.js --dry-run
```

Then one real run by hand, before installing the cron (the first one scans every
transcript of the window and can take a while; the next ones only read files
written since yesterday):

```
~/.local/share/maximus-claude-usage/run-claude-usage.sh
# claude-usage: ok status=204 limits=ok bytes=...
```

Crontab line:

```
*/15 * * * * $HOME/.local/share/maximus-claude-usage/run-claude-usage.sh 2>&1 | logger -t claude-usage
```

```
journalctl -t claude-usage --since -1h
```

A beat that finds the previous one still running (`flock` on
`~/.local/state/maximus-claude-usage/run.lock`) logs
`a run is already in progress, skipping` and exits 0.

### Exit codes

| Code | Meaning |
|------|---------|
| 0 | Snapshot accepted (2xx), or a skipped beat (lock held) |
| 1 | Install problem: env file missing, a variable unset, URL not `https://`, token with an invalid character, node or flock not found |
| 2 | The push did not land (network or non-2xx), or the snapshot exceeded 64 KiB (`payload_too_large`, nothing sent) |

The limits failing is **not** an exit code: the snapshot is still pushed, with
`limits.status` saying why.

### Unreadable transcripts

A day is frozen in `state.json` two days after it ends, and never recomputed
afterwards. A beat that cannot read part of `~/.claude/projects` (a directory it
cannot list, a transcript removed between the walk and its `stat`, a stream
error) therefore freezes **no** new day: it still pushes its snapshot, exits as
usual, and logs only a count:

```
claude-usage: warning: read_errors=N, no day frozen this beat
```

The next clean beat freezes the days that were waiting. A file that stays
unreadable (left at `chmod 000`, for instance) blocks freezing on every beat:
each run then rescans the whole 30-day window, slower but correct, and this
log line is the only sign. Fix the file's permissions. To force a full
recount after the fact, `rm ~/.local/state/maximus-claude-usage/state.json`.

### Backfilling the history (one-shot, #55)

The server keeps every closed day in `history/YYYY-MM.json` from the first push
after the deploy of #55. The days that already left the 30-day window then only
live in the transcripts, which Claude Code deletes as they age: this backfill
rebuilds them once. It runs from the repo, never from the cron install, and the
agent itself does not change.

1. On the workstation, from the repo:

   ```bash
   node agent/claude-usage/backfill-history.js --out ~/claude-usage-backfill.json
   # claude-usage backfill: N day(s) from YYYY-MM-DD to YYYY-MM-DD, 24 window(s), read_errors=0
   ```

   It reads `~/.claude/projects` only (never `~/.claude/.credentials.json`),
   opens no connection, and writes only the `--out` file, in mode 600, refusing
   to overwrite it. It goes back `--max-windows` windows of 28 days (default 24,
   about 22 months; each window takes a few seconds), and stops at D-2: the
   last closed days are archived by the cron's next push anyway. Exit 1 with
   `read_errors=N`: some transcript could not be read, rerun before importing.

2. Copy it to the VPS and merge it inside the container, as `node` (the
   container's user, owner of the mount):

   ```bash
   scp ~/claude-usage-backfill.json ubuntu@vps-b0826277.tail3c811f.ts.net:/tmp/
   ssh ubuntu@vps-b0826277.tail3c811f.ts.net
   CTR=$(docker ps --format '{{.Names}}' | grep u8000gsg044wsk0oo0w884ok | head -1)
   docker exec -i "$CTR" node claude-usage-history.js --import - < /tmp/claude-usage-backfill.json
   # import: N day(s) read, A added, K already archived, M month(s) written
   rm /tmp/claude-usage-backfill.json
   ```

   **An already archived date always wins**: the import only adds the dates
   the archive lacks, so running it twice, or after the live archive has grown,
   changes nothing that is there. An import file with one invalid entry or a
   repeated date is refused whole (exit 1, nothing written). A month file the
   server cannot read back is skipped and left as is (exit 1, named on stderr).

3. Check through the API, with the read token:
   `GET /claude-usage/history?from=<first month>&to=<last month>`.

Then delete `~/claude-usage-backfill.json` on the workstation.

## PostgreSQL backup pull (`agent/pg-backup-pull/`)

A third, independent agent on the Lenovo. Once a day it pulls the
age-encrypted dumps of the shared PostgreSQL cluster from the VPS, keeps 30
days of them in `~/backups/postgres/`, and reports the pull to
`POST /backups/pull`. The VPS side (`backup-postgres.sh`, 03:00 UTC), the
`status.json` contract and the restore procedure live in
`la-compagnie-maximus/docs/postgres-ops.md`, section Backup
(la-compagnie-maximus#220).

| File | Role |
|------|------|
| `run-pg-backup-pull.sh` | rsync over ssh, rotation, report; POSIX `sh`, no node |
| `maximus-pg-backup-pull.service` | systemd user unit, oneshot, retries on failure |
| `maximus-pg-backup-pull.timer` | `OnCalendar=daily`, `Persistent=true` |

**The only departure from the cron convention of the other agents is the
timer.** A cron line missed while the laptop is off is simply lost;
`Persistent=true` runs the missed pass at the next boot. That catch-up run can
start before Tailscale is up, hence `Restart=on-failure` every 15 minutes (8
tries at most in 4 hours).

**What it never does.** It never decrypts: the age private key is not on this
machine (Vaultwarden plus a paper copy). It never deletes on the VPS side, and
rsync only lets directories and `*.age` files through, so a misconfigured
outbox cannot land a clear dump here. The key it uses
(`~/.ssh/pgbackup_pull_ed25519`) can only run `rrsync -ro` on the outbox, from
the Lenovo's Tailscale IP, on the classic sshd at `100.86.30.21:2222`; the VPS
host key is pinned in `~/.ssh/known_hosts_pgbackup` (no trust on first use).

### Install

```
mkdir -p ~/.local/share/maximus-pg-backup-pull ~/.config/systemd/user
cp agent/pg-backup-pull/run-pg-backup-pull.sh ~/.local/share/maximus-pg-backup-pull/
chmod +x ~/.local/share/maximus-pg-backup-pull/run-pg-backup-pull.sh
cp agent/pg-backup-pull/maximus-pg-backup-pull.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now maximus-pg-backup-pull.timer
```

As for the other agents, the copy step *is* the deployment: never point the
unit at the git working tree. The user manager must run without an open
session: `loginctl show-user $USER -p Linger` must say `yes`.

The env file `~/.config/maximus-pg-backup-pull.env` (mode `600`) is optional
until the ingest route is deployed. Without it the pull runs and the report is
skipped, with one log line. Then, written with `printf`:

```
BACKUPS_API_URL=https://health.lacompagniemaximus.com
BACKUPS_INGEST_TOKEN=<the backups ingest token>
```

The token reaches curl through a header file created `0600` and removed right
after, never through argv. An `EXIT` trap removes it too when the run is
stopped mid-report (systemd's `SIGTERM` is turned into an exit first).

### Check

```
systemctl --user start maximus-pg-backup-pull.service
journalctl --user -t pg-backup-pull --since -1h
systemctl --user list-timers maximus-pg-backup-pull.timer
```

A pass whose newest dated directory is not today's (UTC) logs
`warning: no directory for <today>, latest is <date>`: the VPS stopped
producing, or the pass ran before its 03:00 UTC run. A dated directory with no
`.age` file is never created (`--prune-empty-dirs`), and one found empty logs
`warning: no .age file in <date>`. The report still says `ok` with the newest
directory's count (partial files in `.rsync-partial/` not counted); freshness
is judged on the VPS side by `GET /backups`.

**Nothing the VPS sends may delete or alter a copy already here.**

- **No overwrite.** A file already pulled is never rewritten
  (`--ignore-existing`): the VPS writes each dump to a `.tmp` and renames it,
  so what lands is whole.
- **Directory names are checked.** A name is dated only if `date` gives it
  back unchanged: `2026-13-45` or `2026-02-30` is logged and ignored, never
  counted, never removed, and the pass goes on. A dated name after today
  (VPS clock off, or a VPS gone rogue) is logged and ignored the same way.
- **Rotation is anchored on the newest valid backup not after today.** It
  keeps that directory and the 29 days before it. Not on today alone: if the
  VPS stops producing (dead cron, failing `pg_dump`) or this machine's clock
  jumps ahead, the last off-site copy stays put instead of aging out after
  30 days. Not on the newest name alone: a future-dated directory would push
  the window over every real backup.

The script sticks to what busybox offers (no `find -printf`, no
`date -d "... days"`): the CI runs its tests in `node:22-alpine`.

### Exit codes

| Code | Meaning |
|------|---------|
| 0 | Pulled, rotated, reported (or report skipped without a token) |
| 1 | Install problem (missing key, known_hosts, binary) or rotation failure |
| 2 | The pull failed (`pull_failed` reported) |
| 3 | Pulled and rotated, but the report did not land |