feat(claude-usage): archive closed days by month, history route, one-shot backfill #57

Merged

maximus merged 4 commits from issue-55-claude-usage-history into main 2026-10-09 01:47:51 +00:00

Conversation 2 Commits 4 Files changed 14 +1443 -25

maximus commented 2026-10-09 00:47:16 +00:00

Owner

Copy link

Refs #55

What

Keep the closed days of the Claude usage snapshot past its 30-day window.

• Archive on POST /claude-usage. Once the snapshot is written, every day it carries except the most recent date (today, still open, whatever the order sent) and any date past the UTC reception day is merged into <dir of CLAUDE_USAGE_PATH>/history/YYYY-MM.json.
  • The latest snapshot wins. A day that leaves the window keeps the value the agent froze at D+2.
  • Each month file is written atomically: temp file in the same directory, mode 600, then renameSync.
  • A month file that cannot be read back, or is off-format, is never overwritten. That month is skipped and logged, and the other months are still written.
  • An archive failure never changes the 204. It is logged with the month and error code only, never the content.
• GET /claude-usage/history?from=YYYY-MM&to=YYYY-MM, behind the read token, is a new ROUTES entry with no change to checkAuth().
  • Default range: the 12 months ending with the last archived month, capped at the current month.
  • 400 on a bad format, on from > to, or on a span over 36 months.
  • Month files are re-read through the snapshot's per-day whitelist, sanitizeDay(), now exported from claude-usage.js.
• Import command: node claude-usage-history.js --import <file|->, run inside the container. An already archived date always wins. An invalid import file writes nothing.
• Backfill: agent/claude-usage/backfill-history.js runs by hand from the repo. The cron agent is unchanged.
  • It walks past windows of 28 kept days. Each window keeps a one-day margin on both sides, so a request that straddles a window boundary is counted once.
  • It stops at D-2 and is bounded by --max-windows (default 24).
  • It writes only --out (mode 600, never overwritten), opens no connection and never reads the credentials file.
  • Run on this machine, it produced 50 days from 2026-08-18 to 2026-10-06 in 2 min, with read_errors=0.
• Dockerfile COPY now ships claude-usage-history.js.
• Docs: CLAUDE.md and agent/README.md, which has the backfill and import procedure.

Deviations from the issue body

• Backfill stop rule. The issue says "back to the oldest transcript". I dropped the guess based on file mtimes: an mtime is a file's last write, and a resumed session holds lines older than it. The walk now runs a fixed --max-windows, then trims the leading empty days. A full scan takes about 5 s per window.
• Commits. There are two code commits instead of three. The module, the POST archive and the route form one commit, because the route and the import share the module.

Tests

Check	Before	After
npm test	634	731
bash test-curl.sh	45	51
sh scripts/boot-from-copy.sh	PASS	PASS

New and updated test files:

• new __tests__/claude-usage-history.test.js: 76 cases;
• agent-claude-usage.test.js: +10 backfill cases;
• auth.test.js: the route enters the GET matrix, plus one more 404 (not 401) case;
• writes.test.js: the POST case requires the rename of the month file; the history route enters the read routes; a corrupt month or a refused range writes nothing.

Mutations

Each mutation was applied from a cp backup, the suite was rerun, and the file was restored. All 30 turned the suite red (F1-F3 added after the review).

#	Mutation	Red
M1	Archive today too	9
M2	Overwrite an unreadable month	3
M3	Import ignores preferExisting	2
M4	No mode 600	4
M5	Temp file not removed on failure	1
M6	Day dated outside its month kept	2
M7	No 36-month cap	3
M8	No to = from default	1
M9	Stray file name taken as last month	1
M10	Archive before the snapshot write	5
M11	No try/catch around the archive	1
M12	Duplicate import date accepted	1
M13	One-day snapshot creates history/	1
M14	Route on the ingest token	26
M15	Unreadable month not logged	1
M16	No sanitize on read	2
M17	Merge result not sorted	4
B1	Backfill keeps D-1	2
B2	Keep all 30 days, step 28	2
B3	Step leaves a gap	1
B4	No window cap	4
B5	No leading trim	4
B6	--out overwritten	1
B7	--out without mode 600	1
B8	readErrors ignored	1
B9	Window ends on the kept end	2
B10	No right margin (late-best straddle)	1
F1	Exclude the last element, not the latest date (review #2)	2
F2	No filter on dates past reception (review #1)	2
F3	Route default not capped at the current month (review #1)	2

Post-merge checklist (hence Refs, not a closing keyword)

1. Redeploy on Coolify, then call GET /claude-usage/history. It should have content after the first POST (one cron beat).
2. Backfill: run backfill-history.js on the ThinkPad, scp the output, import it in the container, then check through the route. The procedure is in agent/README.md.
3. VPS backup: check whether /data/claude-usage/ (and /data/hosts, /data/defenseurs) is covered. If it is not, set it up and document it in coolify-ops.md.
4. #55 is closed by hand with the evidence once 1 to 3 are done.

🤖 Generated with Claude Code

Refs #55 ## What Keep the closed days of the Claude usage snapshot past its 30-day window. - **Archive on `POST /claude-usage`.** Once the snapshot is written, every day it carries except the most recent date (today, still open, whatever the order sent) and any date past the UTC reception day is merged into `<dir of CLAUDE_USAGE_PATH>/history/YYYY-MM.json`. - The latest snapshot wins. A day that leaves the window keeps the value the agent froze at D+2. - Each month file is written atomically: temp file in the same directory, mode 600, then `renameSync`. - A month file that cannot be read back, or is off-format, is never overwritten. That month is skipped and logged, and the other months are still written. - An archive failure never changes the 204. It is logged with the month and error code only, never the content. - **`GET /claude-usage/history?from=YYYY-MM&to=YYYY-MM`**, behind the read token, is a new `ROUTES` entry with no change to `checkAuth()`. - Default range: the 12 months ending with the last archived month, capped at the current month. - 400 on a bad format, on `from > to`, or on a span over 36 months. - Month files are re-read through the snapshot's per-day whitelist, `sanitizeDay()`, now exported from `claude-usage.js`. - **Import command:** `node claude-usage-history.js --import <file|->`, run inside the container. An already archived date always wins. An invalid import file writes nothing. - **Backfill:** `agent/claude-usage/backfill-history.js` runs by hand from the repo. The cron agent is unchanged. - It walks past windows of 28 kept days. Each window keeps a one-day margin on both sides, so a request that straddles a window boundary is counted once. - It stops at D-2 and is bounded by `--max-windows` (default 24). - It writes only `--out` (mode 600, never overwritten), opens no connection and never reads the credentials file. - Run on this machine, it produced 50 days from 2026-08-18 to 2026-10-06 in 2 min, with `read_errors=0`. - **Dockerfile** `COPY` now ships `claude-usage-history.js`. - **Docs:** `CLAUDE.md` and `agent/README.md`, which has the backfill and import procedure. ### Deviations from the issue body - **Backfill stop rule.** The issue says "back to the oldest transcript". I dropped the guess based on file mtimes: an mtime is a file's last write, and a resumed session holds lines older than it. The walk now runs a fixed `--max-windows`, then trims the leading empty days. A full scan takes about 5 s per window. - **Commits.** There are two code commits instead of three. The module, the POST archive and the route form one commit, because the route and the import share the module. ## Tests | Check | Before | After | |---|---|---| | `npm test` | 634 | 731 | | `bash test-curl.sh` | 45 | 51 | | `sh scripts/boot-from-copy.sh` | PASS | PASS | New and updated test files: - new `__tests__/claude-usage-history.test.js`: 76 cases; - `agent-claude-usage.test.js`: +10 backfill cases; - `auth.test.js`: the route enters the GET matrix, plus one more `404 (not 401)` case; - `writes.test.js`: the POST case requires the rename of the month file; the history route enters the read routes; a corrupt month or a refused range writes nothing. ### Mutations Each mutation was applied from a `cp` backup, the suite was rerun, and the file was restored. All 30 turned the suite red (F1-F3 added after the review). | # | Mutation | Red | |---|---|---| | M1 | Archive today too | 9 | | M2 | Overwrite an unreadable month | 3 | | M3 | Import ignores `preferExisting` | 2 | | M4 | No mode 600 | 4 | | M5 | Temp file not removed on failure | 1 | | M6 | Day dated outside its month kept | 2 | | M7 | No 36-month cap | 3 | | M8 | No `to = from` default | 1 | | M9 | Stray file name taken as last month | 1 | | M10 | Archive before the snapshot write | 5 | | M11 | No try/catch around the archive | 1 | | M12 | Duplicate import date accepted | 1 | | M13 | One-day snapshot creates `history/` | 1 | | M14 | Route on the ingest token | 26 | | M15 | Unreadable month not logged | 1 | | M16 | No sanitize on read | 2 | | M17 | Merge result not sorted | 4 | | B1 | Backfill keeps D-1 | 2 | | B2 | Keep all 30 days, step 28 | 2 | | B3 | Step leaves a gap | 1 | | B4 | No window cap | 4 | | B5 | No leading trim | 4 | | B6 | `--out` overwritten | 1 | | B7 | `--out` without mode 600 | 1 | | B8 | `readErrors` ignored | 1 | | B9 | Window ends on the kept end | 2 | | B10 | No right margin (late-best straddle) | 1 | | F1 | Exclude the last element, not the latest date (review #2) | 2 | | F2 | No filter on dates past reception (review #1) | 2 | | F3 | Route default not capped at the current month (review #1) | 2 | ## Post-merge checklist (hence `Refs`, not a closing keyword) 1. Redeploy on Coolify, then call `GET /claude-usage/history`. It should have content after the first POST (one cron beat). 2. Backfill: run `backfill-history.js` on the ThinkPad, scp the output, import it in the container, then check through the route. The procedure is in `agent/README.md`. 3. VPS backup: check whether `/data/claude-usage/` (and `/data/hosts`, `/data/defenseurs`) is covered. If it is not, set it up and document it in `coolify-ops.md`. 4. #55 is closed by hand with the evidence once 1 to 3 are done. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

maximus added 3 commits 2026-10-09 00:47:16 +00:00

feat(server): archive closed Claude usage days by month, serve them on /claude-usage/history ... 9084d90609

POST /claude-usage now merges every day of the sanitized snapshot except
the last one (today, still open) into <dir of CLAUDE_USAGE_PATH>/history/
YYYY-MM.json, after the snapshot itself is written. The latest snapshot
wins, so a day leaving the 30-day window keeps the value the agent froze
at D+2. Each month file is written atomically (same-dir temp, mode 600,
rename). A month file that cannot be read back is never overwritten, and
an archive failure is logged (months and error codes only) without
changing the 204.

GET /claude-usage/history?from=YYYY-MM&to=YYYY-MM (read token) serves the
archived days, defaulting to the 12 months up to the last archived month,
400 on a bad format, from > to or a span over 36 months. Month files are
re-read through the snapshot's per-day whitelist, now exported from
claude-usage.js as sanitizeDay().

The new module also carries the one-shot import command used by the
backfill (`node claude-usage-history.js --import <file|->`), where an
already archived date always wins.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(agent): one-shot backfill of the Claude usage history from local transcripts ... cf72292967

agent/claude-usage/backfill-history.js runs by hand from the repo (never
from the cron install, the agent is unchanged). It rebuilds the closed days
with the agent's own aggregate() on past windows and writes { days } to the
--out file, which the server merges with `claude-usage-history.js --import`
(an already archived date wins).

Each 30-day window keeps only its 28 inner days and the next steps back 28
days, so a request straddling a window boundary is deduplicated by the
window that keeps it and counted once. The first window stops at D-2. The
walk is bounded by --max-windows (default 24) rather than guessed from file
mtimes, which say nothing of the oldest line a resumed session holds.

It never reads the credentials file, opens no connection, writes only the
--out file (mode 600, never overwritten) and prints counts only.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: document the Claude usage history archive, its route and backfill ... b136749552

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

maximus commented 2026-10-09 01:23:04 +00:00

Author

Owner

Copy link

Verdict : APPROVE

Résumé. L'archive mensuelle, la route GET /claude-usage/history et le backfill tiennent ce que la PR annonce. L'ordre snapshot puis archive, le 204 inchangé, le mois illisible jamais écrasé, la route sur le token de lecture sans toucher checkAuth() et le COPY du Dockerfile sont en place. Les deux règles de fusion (POST : le dernier snapshot gagne ; import : la date archivée gagne) sont épinglées.

Ce qui a été exécuté

• npm test : 727/727 ; bash test-curl.sh : 51/51 ; sh scripts/boot-from-copy.sh : PASS (ports 3097-3099 libres, vérifiés avant).
• CI de la tête b136749 : success.
• 7 mutations du relecteur, distinctes de celles de l'auteur. Elles sont toutes rouges, et l'arbre a été restauré à l'identique (git status propre) :

#	Mutation	Rouges
R4	parseRange sans le refus from > to	2
R5	backfill sans marge gauche (slice(0, 28))	2
R6	lastArchivedMonth rend le premier mois au lieu du dernier	1
R8	writeMonth écrit directement le fichier final (plus de temporaire ni de rename)	2
R11	import tolérant (entrée invalide ou date dupliquée ignorée au lieu de refuser le fichier)	2
R12	la CLI d'import fusionne sans preferExisting	1
R13	le POST n'archive pas la veille (slice(0, -2))	14

Fermeture d'issue

La timeline de #55 porte pull_ref avec ref_action: "none" : aucun mot-clé de fermeture, conforme au Refs et à la fermeture manuelle annoncée.

Suggestions (non bloquantes)

1. Date future dans un snapshot (index.js, archiveClosedDays ; claude-usage-history.js, lastArchivedMonth). cleanDayDate vérifie qu'un jour existe, pas qu'il soit passé. Un snapshot portant un jour 2099-12-31 (horloge du poste déréglée, bug de l'agent) crée history/2099-12.json. Ce fichier fixe ensuite pour de bon la plage par défaut de GET /claude-usage/history sur 2099-01..2099-12, donc la route répond vide jusqu'à une suppression à la main. Deux parades possibles : n'archiver que les dates <= jour de receivedAt + 1, ou borner lastArchivedMonth au mois courant.
2. Ordre de days supposé (archiveClosedDays, days.slice(0, -1)). Le code suppose le dernier élément égal à aujourd'hui. Le contrat ordonne days du plus ancien au plus récent, mais sanitizeDays ne trie pas. Écarter la date maximale (au lieu du dernier élément) rendrait la règle indépendante de l'ordre reçu.
3. Les temporaires orphelins de history/ ne sont couverts par aucun balayage. C'est assumé et documenté, la remarque ne sert qu'à la trace.

Post-merge

La checklist de la PR (redeploy, backfill, sauvegarde de /data/claude-usage/, fermeture manuelle) reste à faire.

## Verdict : APPROVE **Résumé.** L'archive mensuelle, la route `GET /claude-usage/history` et le backfill tiennent ce que la PR annonce. L'ordre snapshot puis archive, le 204 inchangé, le mois illisible jamais écrasé, la route sur le token de lecture sans toucher `checkAuth()` et le `COPY` du Dockerfile sont en place. Les deux règles de fusion (POST : le dernier snapshot gagne ; import : la date archivée gagne) sont épinglées. ### Ce qui a été exécuté - `npm test` : 727/727 ; `bash test-curl.sh` : 51/51 ; `sh scripts/boot-from-copy.sh` : PASS (ports 3097-3099 libres, vérifiés avant). - CI de la tête `b136749` : `success`. - 7 mutations du relecteur, distinctes de celles de l'auteur. Elles sont toutes rouges, et l'arbre a été restauré à l'identique (`git status` propre) : | # | Mutation | Rouges | |---|---|---| | R4 | `parseRange` sans le refus `from > to` | 2 | | R5 | backfill sans marge gauche (`slice(0, 28)`) | 2 | | R6 | `lastArchivedMonth` rend le premier mois au lieu du dernier | 1 | | R8 | `writeMonth` écrit directement le fichier final (plus de temporaire ni de `rename`) | 2 | | R11 | import tolérant (entrée invalide ou date dupliquée ignorée au lieu de refuser le fichier) | 2 | | R12 | la CLI d'import fusionne sans `preferExisting` | 1 | | R13 | le POST n'archive pas la veille (`slice(0, -2)`) | 14 | ### Fermeture d'issue La timeline de #55 porte `pull_ref` avec `ref_action: "none"` : aucun mot-clé de fermeture, conforme au `Refs` et à la fermeture manuelle annoncée. ### Suggestions (non bloquantes) 1. **Date future dans un snapshot** (`index.js`, `archiveClosedDays` ; `claude-usage-history.js`, `lastArchivedMonth`). `cleanDayDate` vérifie qu'un jour existe, pas qu'il soit passé. Un snapshot portant un jour `2099-12-31` (horloge du poste déréglée, bug de l'agent) crée `history/2099-12.json`. Ce fichier fixe ensuite pour de bon la plage par défaut de `GET /claude-usage/history` sur 2099-01..2099-12, donc la route répond vide jusqu'à une suppression à la main. Deux parades possibles : n'archiver que les dates `<=` jour de `receivedAt` + 1, ou borner `lastArchivedMonth` au mois courant. 2. **Ordre de `days` supposé** (`archiveClosedDays`, `days.slice(0, -1)`). Le code suppose le dernier élément égal à aujourd'hui. Le contrat ordonne `days` du plus ancien au plus récent, mais `sanitizeDays` ne trie pas. Écarter la date maximale (au lieu du dernier élément) rendrait la règle indépendante de l'ordre reçu. 3. Les temporaires orphelins de `history/` ne sont couverts par aucun balayage. C'est assumé et documenté, la remarque ne sert qu'à la trace. ### Post-merge La checklist de la PR (redeploy, backfill, sauvegarde de `/data/claude-usage/`, fermeture manuelle) reste à faire.

maximus referenced this pull request from a commit 2026-10-09 01:34:57 +00:00

fix(server): archive by latest date, never a day past reception

maximus added 1 commit 2026-10-09 01:34:57 +00:00

fix(server): archive by latest date, never a day past reception ... c5ded9e6f8

Review of PR #57, suggestions 1 and 2:
- the excluded open day is now the most recent date of the snapshot,
  not its last element, so the rule no longer depends on the order the
  days were sent in (the sanitizer does not sort them);
- a day dated past the server's UTC reception day is never archived, and
  GET /claude-usage/history no longer takes a month file dated past the
  current month as its default end. A skewed workstation clock could
  otherwise open history/2099-12.json and leave the default range empty.

Refs #55

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

maximus merged commit c5ded9e6f8 into main 2026-10-09 01:47:51 +00:00

maximus deleted branch issue-55-claude-usage-history 2026-10-09 01:47:52 +00:00

maximus commented 2026-10-09 01:48:06 +00:00

Author

Owner

Copy link

Merged fast-forward on c5ded9e, CI green on that head.

c5ded9e (fix(server): archive by latest date, never a day past reception) was added after the APPROVE and was not reviewed. It applies review suggestions 1 and 2 at Max's request ("corrige et merge"):

• the excluded open day is the most recent date, not the last element sent;
• a day dated past the UTC reception day is never archived, and the route's default end ignores a month file dated past the current month.

Mutations on the fix, from a cp backup, tree restored after each one:

• F1 (exclude the last element, not the latest date): 2 red;
• F2 (no filter on dates past reception): 2 red;
• F3 (route default not capped at the current month): 2 red.

Suite: 727 tests before the fix, 731 after; test-curl 51/51; boot-from-copy PASS.

Suggestion 3 (no sweep of temp files in history/) is left as is, as documented.

#55 stays open until the post-merge checklist is done: redeploy, backfill, VPS backup check.

Merged fast-forward on `c5ded9e`, CI green on that head. `c5ded9e` (fix(server): archive by latest date, never a day past reception) was added after the APPROVE and was not reviewed. It applies review suggestions 1 and 2 at Max's request ("corrige et merge"): - the excluded open day is the most recent date, not the last element sent; - a day dated past the UTC reception day is never archived, and the route's default end ignores a month file dated past the current month. Mutations on the fix, from a `cp` backup, tree restored after each one: - F1 (exclude the last element, not the latest date): 2 red; - F2 (no filter on dates past reception): 2 red; - F3 (route default not capped at the current month): 2 red. Suite: 727 tests before the fix, 731 after; test-curl 51/51; boot-from-copy PASS. Suggestion 3 (no sweep of temp files in `history/`) is left as is, as documented. #55 stays open until the post-merge checklist is done: redeploy, backfill, VPS backup check.

maximus referenced this pull request 2026-10-09 01:48:06 +00:00

feat(claude-usage): conserver l’historique des jours clos au-delà de la fenêtre de 30 jours #55

maximus referenced this pull request 2026-10-12 23:29:40 +00:00

feat(backups): GET /backups et POST /backups/pull (sauvegardes PostgreSQL) #58

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

autopilot:pending-human

PR produite par /autopilot, attend review humaine (jamais mergee par le run)

  

source:analyste

Créée par le Super-Analyste

  

source:defenseur

Créée par l Escouade Défenseur

  

source:human

Créée par un humain

  

source:medic

Créée par le Medic (auto-détection)

  

status:approved

PR approuvée, prête à merger

  

status:blocked

Bloquée (question, dépendance, clarification)

  

status:in-progress

Un agent (Medic/Dev) travaille dessus

  

status:needs-clarification

Issue bloquee par une ambiguite LOW pendant un run autonome, Max doit trancher

  

status:needs-fix

Reviewer demande des corrections

  

status:ready

Analysée et documentée, prête à prendre

  

status:review

PR créée, en attente du Reviewer

  

status:triage

Nouvelle issue, pas encore analysée

  

type:bug

Correction de bug → Medic

  

type:feature

Nouvelle fonctionnalité → Dev

  

type:infra

Config, CI/CD, déploiement → Dev

  

type:refactor

Refactoring / amélioration technique → Dev

  

type:schema

Modèle de données / migrations → Dev

  

type:security

Issue de sécurité → Medic ou Dev

No labels

autopilot:pending-human

source:analyste

source:defenseur

source:human

source:medic

status:approved

status:blocked

status:in-progress

status:needs-clarification

status:needs-fix

status:ready

status:review

status:triage

type:bug

type:feature

type:infra

type:refactor

type:schema

type:security

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: maximus/vps-health-api#57

No description provided.