Compare commits
16 commits
108cc3801c
...
2d4caecae8
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2d4caecae8 | ||
|
|
89149d06a9 | ||
|
|
37b832e084 | ||
|
|
ef7de3cf9b | ||
|
|
484c4beb47 | ||
|
|
e2b8eb8b22 | ||
|
|
f377d760af | ||
|
|
c9872fc36b | ||
|
|
ce19efd476 | ||
|
|
bf608b9d67 | ||
|
|
6f64fc5c1a | ||
|
|
7b6f063094 | ||
|
|
7a604e0e0d | ||
|
|
bd1085c148 | ||
|
|
c88ebd862e | ||
|
|
b30c9fa5c1 |
62 changed files with 9463 additions and 785 deletions
|
|
@ -4,12 +4,28 @@
|
|||
|
||||
### Ajouté
|
||||
|
||||
- Import : ouvrir une source jamais configurée **détecte désormais son format tout seul** — délimiteur, encodage, format de date, et quelle colonne porte la date, la description et le montant. Les colonnes sont reconnues par leur libellé d'en-tête en français comme en anglais (`Montant`/`Amount`, `Débit`/`Withdrawal`, `Crédit`/`Deposit`…), et les formats d'export de **Desjardins, RBC, Banque Nationale et Tangerine** sont reconnus par leur nom. La détection rejoue ensuite sa propre décision sur toutes les lignes du fichier et affiche un **score de confiance** (« 147 des 150 lignes lues ») : au-dessus de 90 % le bandeau est neutre, en dessous il avertit et renvoie vers le mapping des colonnes et l'aperçu. Rien n'est jamais bloqué par le score — il dit ce qui a pu être lu, pas si le sens est bon. Le bouton baguette reste disponible pour relancer la détection à la demande, et une source qui a déjà un format enregistré n'est jamais re-détectée (#327, #328).
|
||||
- Import : l'**aperçu des données est désormais une étape obligatoire de chaque import**, et non plus une fenêtre optionnelle. Il affiche un récapitulatif signé de ce que le fichier va réellement écrire : combien de sorties et pour quel total, combien d'entrées et pour quel total, et combien de lignes n'ont pas pu être lues. Si les entrées et les sorties sont inversées, un bouton **« Inverser les signes »** les corrige — et comme il corrige la configuration de la source plutôt que le seul aperçu, le prochain fichier de cette banque se lira correctement de lui-même. L'écran de confirmation énonce maintenant aussi le mode de montant, la convention de signe et le mapping des colonnes nommé par en-tête, pour les vérifier avant que l'import ne parte (#329).
|
||||
- Import : quand une banque change la disposition de son export, l'assistant le **signale avant d'importer**. L'en-tête de chaque import réussi est mémorisé sur la source, et un fichier dont l'en-tête ne correspond plus ouvre un panneau listant les colonnes qui ont bougé, apparu ou disparu (« Montant : colonne 3 → 4 »), avec les deux seuls choix qui existent : adopter le format re-détecté, ou conserver celui enregistré. Un renommage purement cosmétique (`Montant` → `MONTANT ($)`) est reconnu comme le même en-tête et ne dit rien. Le panneau de dérive comme le bouton d'inversion portent en plus une note expliquant la **façon sûre de réparer un import déjà écrit de travers** : le supprimer d'abord dans l'historique des imports, puis rejouer le fichier — le réimporter par-dessus doublerait les lignes, les doublons étant repérés sur la date, la description et le montant ensemble (#330).
|
||||
- Import : vos **sources et modèles d'import font désormais partie de la sauvegarde chiffrée**. Exporter puis réimporter vos données effaçait toute configuration d'import et la remplaçait par une unique source synthétique « Data Import » : restaurer une sauvegarde obligeait à reconfigurer chaque banque à la main. Les anciens fichiers de sauvegarde, qui ne les contiennent pas, sont restaurés exactement comme avant (#331).
|
||||
- Migration des catégories : les catégories personnalisées sans correspondance standard peuvent désormais être **fusionnées** dans une catégorie standard, au lieu d'être seulement mises de côté. Chaque catégorie personnalisée du bloc « Catégories personnalisées » reçoit le même sélecteur de cible que les lignes du seed — choisissez une feuille standard et ses transactions, budgets, mots-clés et fournisseurs y sont réassignés, puis la catégorie personnalisée est retirée. Laisser une catégorie personnalisée non mappée conserve le comportement précédent (regroupée sous « Catégories personnalisées (migration) ») et ne bloque jamais la migration (#259).
|
||||
|
||||
### Modifié
|
||||
|
||||
- Import : le format d'une source — y compris le **mode de montant** (montant unique, ou colonnes débit et crédit séparées) et la **convention de signe** (dépenses écrites négatives ou positives) — est désormais enregistré en entier sur la source elle-même et restitué exactement tel que vous l'avez réglé. Jusqu'ici ces deux réglages n'étaient conservés que sur les modèles, et rouvrir une source configurée re-déduisait le mode puis remettait silencieusement la convention sur « dépenses négatives » : une source que vous aviez configurée en dépenses positives revenait sur le défaut à son deuxième import. Reconfigurer une source ne change pas, mais une source configurée une fois n'est plus jamais devinée. Le sélecteur de convention de signe est maintenant masqué en mode débit/crédit, où il ne s'appliquait pas (votre valeur enregistrée reste intacte), et votre configuration n'est écrite qu'au moment où l'import part réellement — abandonner l'assistant à l'étape des doublons ne laisse plus de configuration à moitié enregistrée (#323, #324, #328).
|
||||
|
||||
- L'application déverrouille désormais ses modules par édition — **Gratuite**, **Base** ou **Premium**, résolue depuis votre clé de licence. La Gratuite conserve le Tableau de bord, l'Import CSV, les Transactions, les Catégories, le rapport Tendances (et le hub Rapports), l'export/import chiffré et le journal des modifications, avec un seul profil. La **Base** débloque en plus le Budget, les Ajustements, les quatre rapports avancés (Faits saillants, Comparables, Analyse par catégorie, Cartes), les profils multiples et les mises à jour automatiques. La **Premium** débloque en plus le module Bilan complet (patrimoine, détail par titre, cours du marché). Les modules verrouillés restent visibles — un cadenas apparaît dans la barre latérale et sur les tuiles de rapports — et les ouvrir affiche un écran de déverrouillage avec un raccourci « J'ai déjà une clé » vers la carte de licence ; le bouton d'achat en ligne « Obtenir Base / Premium » est affiché mais désactivé pour l'instant (« bientôt disponible »). Le verrouillage n'est jamais destructif : quelle que soit l'édition, vos données sont conservées intactes et tout réapparaît dès qu'une clé valide est entrée — en Gratuite votre profil actif reste toujours accessible, seuls la création d'un profil supplémentaire ou le passage à un autre profil sont verrouillés (#297, #298, #299, #300, #301).
|
||||
|
||||
### Corrigé
|
||||
|
||||
- Import : un relevé dont les colonnes débit et crédit sont écrites dans cet ordre (`Date;Description;Crédit;Débit`) voyait **tous ses signes inversés** — les dépenses importées en revenus et les revenus en dépenses. Les deux colonnes étaient distinguées par leur position, jamais par leur libellé. Elles sont désormais identifiées par leur en-tête, l'ordre dans lequel elles apparaissent n'a donc plus d'importance (#327).
|
||||
- Import : chez les nombreuses banques qui écrivent `0,00` dans la colonne inutilisée au lieu de la laisser vide, **chaque dépense s'importait à 0,00** et disparaissait purement et simplement du grand livre, sans aucune erreur. Les montants sont maintenant calculés comme crédit moins débit, ce qui ne demande aucun cas particulier pour une cellule à zéro. Une ligne illisible dans les deux colonnes est signalée comme erreur au lieu d'être écrite en transaction gratuite à 0,00, et une colonne de montant laissée non mappée est signalée d'emblée plutôt que de retomber sur la lecture de la colonne de date (#325).
|
||||
- Import : les montants portant un code de devise ou un marqueur comptable en suffixe étaient lus **cent fois trop grands** — `100,00 CAD` s'importait à 10 000,00 et passait toutes les validations, seuls les chiffres de tête étant lus et le reste ignoré. Une telle valeur est désormais rejetée comme illisible et signalée sur sa ligne. Les formes comptables sont en revanche prises en charge : `(50,00)` et `50,00-` valent tous deux −50,00. La même correction s'applique à l'import CSV de titres dans un snapshot de bilan, où un prix écrit `150,25 CAD` était enregistré à 15 025 et une quantité illisible sauvegardée en silence comme une position de valeur nulle (#325).
|
||||
- Import : dans un fichier où une colonne utilise la virgule décimale et une autre le point, `1.234` pouvait être lu 1234 à un endroit et 1,234 à l'autre. Le séparateur décimal est maintenant tranché par colonne, à partir des valeurs de cette colonne (#325).
|
||||
- Import : un relevé à **montants positifs accompagnés d'une colonne d'indicateur « D » / « C »** était indiscernable d'un fichier tout-positif, donc configuré en « dépenses positives », et **chaque dépôt s'importait en dépense**. Ce format est désormais détecté et refusé avec une explication, au lieu d'être importé à l'envers. Réexportez le relevé avec des montants signés, ou avec des colonnes débit et crédit séparées (#327).
|
||||
- Import : une ligne d'en-tête dont la première cellule commence par des chiffres (une colonne intitulée `2025`, par exemple) était prise pour une ligne de données, et l'en-tête s'importait en transaction. Les lignes d'en-tête sont maintenant reconnues par leurs libellés autant que par leur forme (#325, #327).
|
||||
- Export/import de données : la restauration d'une sauvegarde n'était pas enveloppée dans une transaction — une violation de contrainte en cours de route abandonnait la restauration alors que les données existantes avaient déjà été effacées, détruisant l'historique financier sans retour arrière. L'effacement et la restauration réussissent ou échouent désormais d'un bloc. Restaurer dans un profil qui possédait déjà des modèles d'import n'échoue plus non plus sur une collision de nom (#331).
|
||||
|
||||
### Sécurité
|
||||
|
||||
- Mise à jour de deux dépendances Rust situées sur le chemin de la mise à jour automatique, corrigeant six advisories RustSec : `rustls-webpki` 0.103.9 → 0.103.13 (traitement des contraintes de nom de certificat et analyse des listes de révocation, dont une panique atteignable) et `tar` 0.4.44 → 0.4.46 (`chmod` de répertoires arbitraires en suivant des liens symboliques pendant l'extraction, et en-têtes de taille PAX mal pris en compte). Les deux sont tirées par `tauri-plugin-updater` : celle qui gère TLS travaille à chaque vérification et à chaque téléchargement de mise à jour, tandis que celle qui décompresse n'est atteinte que par des formats d'installeur que cette application ne livre pas — elle a été mise à jour quand même, plutôt que contournée par un raisonnement. Aucun changement de comportement. Une autre advisory reste signalée sur le fichier de verrouillage et ne s'applique pas au produit livré : `rsa` n'est jamais compilé, son seul parent étant un pilote MySQL qu'une application SQLite ne construit jamais. Elle est consignée comme acceptée, avec sa justification et les conditions de son retrait, plutôt que de laisser l'audit de sécurité quotidien rouge en permanence. Les deux advisories visant `quick-xml` ont été acceptées de la même façon pendant quelques heures, puis résolues pour de bon : `plist` 1.10.0 livrait déjà le `quick-xml` 0.41.0 corrigé (#310, #312).
|
||||
|
|
|
|||
16
CHANGELOG.md
16
CHANGELOG.md
|
|
@ -4,12 +4,28 @@
|
|||
|
||||
### Added
|
||||
|
||||
- Import: opening a source you have never configured now **detects its format on its own** — delimiter, encoding, date format, and which column holds the date, the description and the amount. Columns are recognised by their header label in French and in English (`Montant`/`Amount`, `Débit`/`Withdrawal`, `Crédit`/`Deposit`…), and the exported layouts of **Desjardins, RBC, National Bank and Tangerine** are recognised by name. Detection then replays its own decision over every row of the file and reports a **confidence score** ("147 of 150 rows read"): above 90 % the banner is neutral, below it the banner warns and points you at the column mapping and the preview. Nothing is ever blocked by the score — it says what could be read, not whether the meaning is right. The magic-wand button is still there to re-run detection on demand, and a source that already has a saved format is never re-detected (#327, #328).
|
||||
- Import: the data **preview is now a mandatory step of every import**, no longer an optional pop-up, and it shows a signed recap of what the file will actually write: how many outflows and for what total, how many inflows and for what total, and how many rows could not be read. If inflows and outflows are swapped, an **"Flip the signs"** button fixes them — and because it corrects the source's configuration rather than just this preview, the next file from that bank reads correctly on its own. The confirmation screen now also states the amount mode, the sign convention and the column mapping named by header, so you can check them before the import runs (#329).
|
||||
- Import: when a bank changes the layout of its export, the wizard now **says so before importing**. The header of every successful import is recorded on the source, and a file whose header no longer matches opens a panel listing the columns that moved, appeared or disappeared ("Amount: column 3 → 4"), with the two choices that exist: adopt the newly detected format, or keep the stored one. A purely cosmetic rename (`Montant` → `MONTANT ($)`) is recognised as the same header and says nothing. Both the drift panel and the sign-flip button now also carry a note explaining the **safe way to repair an import that was already written wrong**: delete it from the import history first, then replay the file — re-importing on top of it would double the rows, since duplicates are matched on date, description and amount together (#330).
|
||||
- Import: your **import sources and templates are now included in the encrypted backup**. Exporting and re-importing your data used to wipe every import configuration and replace it with a single synthetic "Data Import" source, so restoring a backup meant reconfiguring every bank by hand. Older backup files, which do not carry them, are still restored exactly as before (#331).
|
||||
- Category migration: custom categories that have no standard match can now be **merged** into a standard category instead of only being set aside. Each custom category in the "Custom categories" block gets the same target picker as the seeded rows — choose a standard leaf and its transactions, budgets, keywords and suppliers are reassigned to it, then the custom category is removed. Leaving a custom category unmapped keeps the previous behaviour (grouped under "Custom categories (migration)") and never blocks the migration (#259).
|
||||
|
||||
### Changed
|
||||
|
||||
- Import: the format of a source — including the **amount mode** (single amount, or separate debit and credit columns) and the **sign convention** (expenses written negative or positive) — is now stored in full on the source itself and restored exactly as you set it. Until now those two settings were only kept on templates, and reopening a configured source re-derived the mode and silently reset the convention to "expenses negative"; a source you had configured for positive expenses came back on the default at its second import. Reconfiguring a source is unchanged, but a source configured once is never guessed at again. The sign-convention selector is now hidden in debit/credit mode, where it never applied (your stored value is left untouched), and your configuration is written only once the import actually runs — abandoning the wizard at the duplicate-check step no longer leaves a half-saved configuration behind (#323, #324, #328).
|
||||
|
||||
- The application now unlocks its modules per edition — **Free**, **Base** or **Premium**, resolved from your license key. Free keeps the Dashboard, CSV Import, Transactions, Categories, the Trends report (and the Reports hub), encrypted export/import and the changelog, with a single profile. **Base** additionally unlocks Budget, Adjustments, the four advanced reports (Highlights, Compare, Category analysis, Cards), multiple profiles and automatic updates. **Premium** additionally unlocks the full Balance module (net worth, per-security detail, market prices). Locked modules stay visible — a lock badge shows in the sidebar and on the report tiles — and opening one shows an unlock screen with an "I already have a key" shortcut to the license card; the online "Get Base / Premium" purchase button is shown but disabled for now ("coming soon"). Locking is never destructive: whatever the edition, your data is kept untouched and everything reappears as soon as a valid key is entered — on Free your active profile always stays accessible, only creating or switching to another profile is locked (#297, #298, #299, #300, #301).
|
||||
|
||||
### Fixed
|
||||
|
||||
- Import: a statement whose debit and credit columns are written in that order (`Date;Description;Credit;Debit`) used to have **every single sign inverted** — expenses imported as income and income as expenses. The two columns were told apart by their position, never by their label. They are now identified by their header, so the order they appear in no longer matters (#327).
|
||||
- Import: on the many banks that write `0,00` in the unused column instead of leaving it empty, **every expense used to import as 0,00** and simply vanished from your ledger, with no error anywhere. Amounts are now computed as credit minus debit, which needs no special case for a zero cell. A row that cannot be read in either column is reported as an error instead of being silently written as a free 0,00 transaction, and an amount column left unmapped is now reported up front rather than falling back to reading the date column (#325).
|
||||
- Import: amounts carrying a trailing currency code or accounting marker were read **a hundred times too large** — `100,00 CAD` imported as 10 000,00 and passed every validation, since only the leading digits were read and the rest was discarded. Such a value is now rejected as unreadable and reported on its row. Accounting forms are handled properly instead: `(50,00)` and `50,00-` both read as −50,00. The same fix applies to the CSV import of securities into a balance snapshot, where a price written `150,25 CAD` used to be stored as 15 025 and an unreadable quantity was silently saved as a zero-value position (#325).
|
||||
- Import: in a file where one column uses a comma for decimals and another a period, `1.234` could be read as 1234 in one place and 1,234 in the other. The decimal separator is now decided per column, from that column's own values (#325).
|
||||
- Import: a statement with **positive amounts plus a separate "D" / "C" indicator column** used to be indistinguishable from an all-positive file, so it was configured as "expenses positive" and **every deposit imported as an expense**. This layout is now detected and refused with an explanation, instead of being imported backwards. Re-export the statement with signed amounts, or with separate debit and credit columns (#327).
|
||||
- Import: a header row whose first cell starts with digits (a column titled `2025`, for example) was mistaken for a data row, and the header was imported as a transaction. Header rows are now recognised by their labels as well as their shape (#325, #327).
|
||||
- Data export/import: restoring a backup was not wrapped in a transaction — a constraint failure partway through would abandon the restore after the existing data had already been wiped, destroying your financial history with no rollback. The wipe and the restore now succeed or fail as a whole. Restoring into a profile that already had import templates also no longer fails on a name collision (#331).
|
||||
|
||||
### Security
|
||||
|
||||
- Updated two Rust dependencies sitting on the automatic-update path, clearing six RustSec advisories: `rustls-webpki` 0.103.9 → 0.103.13 (certificate name-constraint handling and certificate-revocation-list parsing, including a reachable panic) and `tar` 0.4.44 → 0.4.46 (arbitrary directory `chmod` by following symlinks during extraction, and mishandled PAX size headers). Both are pulled in by `tauri-plugin-updater`: the TLS one runs on every update check and download, while the archive one is only reached by installer formats this app does not ship — it was updated anyway rather than reasoned around. No behaviour change. One further advisory is still reported against the lockfile and does not apply to the shipped product: `rsa` is never compiled at all, its only parent being a MySQL driver that a SQLite application never builds. It is recorded as accepted, with its justification and the conditions for removing it, instead of leaving the daily security audit permanently red. The two advisories against `quick-xml` were accepted the same way for a few hours, then resolved outright: `plist` 1.10.0 turned out to ship the fixed `quick-xml` 0.41.0 (#310, #312).
|
||||
|
|
|
|||
10
CLAUDE.md
10
CLAUDE.md
|
|
@ -43,7 +43,7 @@ src/
|
|||
│ ├── budget/ # Budget
|
||||
│ ├── categories/ # Catégories hiérarchiques
|
||||
│ ├── dashboard/ # Tableau de bord
|
||||
│ ├── import/ # Wizard d'import (13 composants)
|
||||
│ ├── import/ # Wizard d'import (14 composants)
|
||||
│ ├── layout/ # AppShell, Sidebar
|
||||
│ ├── profile/ # Profils (PIN, formulaire, switcher)
|
||||
│ ├── reports/ # Graphiques et rapports
|
||||
|
|
@ -70,7 +70,7 @@ src-tauri/
|
|||
│ │ ├── schema.sql # Schéma initial (v1)
|
||||
│ │ ├── seed_categories.sql # Seed catégories (v2)
|
||||
│ │ └── consolidated_schema.sql # Schéma complet (nouveaux profils)
|
||||
│ ├── lib.rs # Point d'entrée, 7 migrations inline, plugins
|
||||
│ ├── lib.rs # Point d'entrée, 17 migrations inline, plugins
|
||||
│ └── main.rs
|
||||
└── Cargo.toml
|
||||
```
|
||||
|
|
@ -86,7 +86,7 @@ src-tauri/
|
|||
|
||||
## Fonctionnalités principales
|
||||
|
||||
- **Import CSV** : wizard multi-étapes, détection auto de l'encodage/délimiteur, templates de config, déduplication par fichier
|
||||
- **Import CSV** : wizard multi-étapes, détection auto (encodage, délimiteur, colonnes par libellé FR/EN, formats de banques connus) avec score de confiance, **aperçu obligatoire à récap signé** avant tout import, templates de config, déduplication par fichier. Le format d'import est **persisté en entier sur la source et jamais re-deviné** — [ADR 0019](docs/adr/0019-format-import-persiste.md)
|
||||
- **Catégorisation** : automatique (mots-clés avec priorité) et manuelle, drag-and-drop pour réorganiser
|
||||
- **Transactions** : filtrage, tri, split sur plusieurs catégories, notes
|
||||
- **Budget** : grille 12 mois, templates réutilisables, budget vs réel
|
||||
|
|
@ -118,8 +118,8 @@ src-tauri/
|
|||
|
||||
## Base de données
|
||||
|
||||
- **20 tables** SQLite, **24 index** (voir `docs/architecture.md` pour le détail). Le module Bilan en représente 7 tables (`balance_categories`, `balance_accounts`, `balance_snapshots`, `balance_snapshot_lines`, `balance_account_transfers`, puis `balance_securities` + `balance_snapshot_holdings` ajoutées en Étape 2 — détail par titre) et 9 index
|
||||
- **16 migrations inline** dans `lib.rs` (v1→v16, via `tauri_plugin_sql::Migration`). Étape 2 (détail par titre) : v14 (`balance_securities` + `balance_snapshot_holdings` + 2 index), v15 (`balance_accounts.kind` + `detailed_since` + backfill), v16 (conversion des comptes cotés existants en détaillés 1-position). Voir [ADR 0015](docs/adr/0015-balance-detail-par-titre.md)
|
||||
- **20 tables** SQLite, **24 index** (voir `docs/architecture.md` pour le détail). Le module Bilan en représente 7 tables (`balance_categories`, `balance_accounts`, `balance_snapshots`, `balance_snapshot_lines`, `balance_account_transfers`, puis `balance_securities` + `balance_snapshot_holdings` ajoutées en Étape 2 — détail par titre) et 9 index. Ces deux comptes sont inchangés depuis v13 : v14-v16 ajoutent 2 tables + 2 index (déjà comptés), v17 est un `ALTER TABLE` pur
|
||||
- **17 migrations inline** dans `lib.rs` (v1→v17, via `tauri_plugin_sql::Migration`). Étape 2 du Bilan (détail par titre) : v14 (`balance_securities` + `balance_snapshot_holdings` + 2 index), v15 (`balance_accounts.kind` + `detailed_since` + backfill), v16 (conversion des comptes cotés existants en détaillés 1-position) — voir [ADR 0015](docs/adr/0015-balance-detail-par-titre.md). Import : v17 (`import_sources.amount_mode` + `sign_convention`, les deux avec `CHECK`, + `header_signature` + `template_id`) — voir [ADR 0019](docs/adr/0019-format-import-persiste.md)
|
||||
- **Schéma consolidé** (`consolidated_schema.sql`) pour l'initialisation des nouveaux profils
|
||||
- Les migrations appliquées sont protégées par checksum — ne jamais modifier une migration existante, toujours en créer une nouvelle
|
||||
|
||||
|
|
|
|||
7
STATE.md
7
STATE.md
|
|
@ -1,6 +1,6 @@
|
|||
# STATE — Simpl'Résultat
|
||||
|
||||
> Derniere MAJ : 2026-07-27 (**Advisories de dépendances traitées — #310 + #311 mergées** (`main` `a14258b`, ff-only, PRs #316/#318 stackées) : **`cargo audit` 9 → 0** et **`npm audit` 3 → 2 acceptées**. Les 6 advisories Rust atteignables tombent par `cargo update -p rustls-webpki -p tar` (0.103.13 / 0.4.46, bornes existantes, `Cargo.toml` intact) ; les 3 restantes sont **hors surface produit** — `quick-xml` ×2 n'est compilé sur **aucune cible livrée** (Apple-only via `plist`, ce qui révise le triage d'origine « bloqué en amont ») et `rsa` n'a aucun correctif publié ni aucun arbre — et passent en liste acceptée `.cargo/audit.toml`, **par ID d'advisory jamais par crate**, avec **garde-fou bloquant** dans `check-rust.yml` rejouant la preuve à chaque PR Rust (exit status testé séparément de la sortie + canari + log de chaque vérification). ADR 0018. Côté npm : `postcss` 8.5.23 **sans override** (la borne de vite l'autorisait déjà), CSS émis identique octet pour octet ; `react-router` accepté (mode RSC inatteignable, `react-router-dom` figé à 7.18.1). 5 suivis : #312-#315, #317. [Unreleased], non taggé — v0.14.0 reste le dernier tag ; candidate **v0.15.0**. Aucune migration DB (v1→v16). Précédente MAJ 2026-07-27 : **CI optimisée — #232 mergée** (`7779f7d`, `spec-ci-build-optimization` 3/4) — split `check.yml` → `check-rust.yml` + `check-frontend.yml` + `audit.yml`, caches Actions morts retirés (#234 non résolu), `cargo-audit` pré-buildé ; **rust 21m44 → 8m55**, **PR frontend-only 24m12 → 1m43**. Rappels infra : `main` est **protégée** (whitelist push `maximus`) + shadowing d'identité dans `~/.git-credentials` → remote ancré `https://maximus@…`)
|
||||
> Derniere MAJ : 2026-08-14 (**Chantier import CSV livré — milestone `planned-2026-08-12-import-csv-format` 10/10, pile de 10 PRs #333-#342 mergée ff-only** (`main` `37b832e`). Le symptôme rapporté par Max — « l'app oublie le format, y compris les colonnes montant positif/négatif » — n'était pas une faiblesse d'heuristique mais un **trou de persistance** : `import_sources` ne portait ni `amount_mode` ni `sign_convention` (ils n'existaient que sur `import_config_templates`), donc `useImportWizard.ts:323` réécrivait `signConvention: "negative_expense"` **en dur** à chaque restauration et une source réglée en montants positifs inversait tous ses montants au 2e import, sans erreur. **Migration v17** (4 colonnes + `CHECK`, backfill `LIKE '%debitAmount%'` reproduisant la règle runtime → aucune source ne change de comportement) ; codec unique `formatToRow`/`formatFromRow` à test de complétude ; `credit − debit` sur magnitudes ; `parseFrenchAmount` **ancré** ; détection par libellé d'en-tête ; score de confiance (seuil 90 %) ; **aperçu obligatoire à récap signé** ; signatures de banques + dérive de format ; sources et modèles enfin sauvegardés dans l'export SREF. **871 → 1181 vitest**, **106 → 111 Rust**. 20 tables / 24 index **inchangés** (v17 est un ALTER pur). ADR 0019. Candidate **v0.15.0**. Rappels infra : `main` est **protégée** (whitelist push `maximus`) + shadowing d'identité dans `~/.git-credentials` → remote ancré `https://maximus@…`)
|
||||
|
||||
## Position actuelle
|
||||
|
||||
|
|
@ -16,6 +16,7 @@ Audit critique de la page Bilan livré (`docs/audit-bilan-2026-05.md`, revue CPA
|
|||
|
||||
## Decisions recentes
|
||||
|
||||
- 2026-08-14 : **Import CSV — chantier complet livré (10 issues, migration v17)**. Cycle intégral en une session : revue d'implémentation → `/spec` → `/review-spec` → `/plan-run` → `/autopilot` → `/pr-review` ×10 → merge ff-only. **Le diagnostic a tenu, mais trois choses ont été trouvées en cours de route que personne n'avait vues.** (1) Au re-ancrage Phase 3b : `dataExportService` ne sérialise ni `import_sources` ni `import_config_templates` et fait `DELETE FROM import_sources` à la restauration — **restaurer une sauvegarde détruisait toutes les configurations d'import**, 3e voie de perte du format, indépendante du bug racine (→ #331, + `withTransaction` que le service n'avait nulle part). (2) Un worker a **mesuré** que l'exemple « Solde 2024 » de l'issue n'est pas défectueux (`parseFloat` s'arrête sur le `S`) et a ajouté la fixture qui l'est vraiment. (3) Un autre a mesuré que chez RBC `CAD$` et un `Cheque Number` presque vide **sont** sparse-complementary : le générique les apparie et une ligne de chèque s'importe à −234,05 au lieu de −6,95 — d'où l'override par signature. **Correction de la revue initiale** : `parseFrenchAmount("50,00-")` ne rend pas 50 mais **5000** — `"100,00 CAD"` → 10000, `"1 234,56 CR"` → 123456 : erreur de **facteur 100** qui passait `isNaN` et comptait donc la ligne comme VALIDE. Le ×100 atteignait aussi l'import de titres #245. Durcissement global assumé : un relevé à suffixe de devise produit désormais des **lignes en erreur** plutôt que des valeurs fausses. **`/pr-review` a bloqué 3 des 10 PRs, tous fondés** : (a) #336 — la règle débit/crédit testait `isNaN(debit) && isNaN(credit)`, donc une cellule illisible à côté du remplissage `0,00` calculait 0 − 0 = 0 et importait en silence, **le bug même que la PR fermait, une cellule plus loin** (rejoué sur sa propre fixture : 6 transactions à 0,00 $, 0 erreur) ; (b) #337 — `detectDescriptionColumn` retournait la colonne préférée **sans veto par les données**, et le mot-clé `transaction` capturait la colonne DEBIT/CREDIT de Tangerine, tuant la catégorisation ; (c) #340 — **régression réelle** : la signature Desjardins est 4 libellés génériques appariés par sous-ensemble, et une signature court-circuitait le scan sparse-complementary → un fichier `Date;Description;Débit;Crédit;Montant;Solde` lu correctement AVANT devenait `positive_expense` APRÈS, chaque dépôt en dépense. Corrigés en 3 commits sur la tête de pile (`484c4be`, `ef7de3c`, `37b832e`), **chacun vérifié par mutation** — et cette vérification a révélé qu'un de mes propres tests **passait la mutation** (le fix sur les signatures génériques empêchait la branche gardée d'être atteinte) : il a fallu un fichier reconnu par une signature *légitime* pour l'exercer. Leçon : un garde qui ne peut pas échouer ne garde rien, y compris quand c'est soi qui l'écrit. Incident run : le worker #331 tué par une limite de session pendant sa validation — travail récupéré depuis son worktree et revalidé de zéro (le rationale de sa PR est une reconstruction, signalé comme tel). Gotcha parallélisation : deux agents de revue partageaient `scratchpad/review.md`, l'un a écrasé l'autre entre Write et POST → **donner un chemin de scratch unique par agent**. Suivis ouverts : refus du format à indicateur borné à `amountCol ± 1` ; dérives de comptes préexistantes dans `CLAUDE.md`/`architecture.md` (mesurées et documentées, non corrigées). (ref #323-#332, PRs #333-#342)
|
||||
- 2026-07-27 : **#310 + #311 mergées — `cargo audit` 9→0, `npm audit` 3→2 acceptées** (`main` `a14258b`, ff-only ; PRs #316 → #318 stackées, CI verte des deux côtés). Parties d'un `/analyse-vulnerabilite` : rapport Défenseur **écarté comme source** (VPS injoignable — SSH Tailscale en attente d'auth navigateur, aucun lien imprimé ; rapport local du 2026-05-06, 82 jours) → vérité live = `cargo audit 0.22.2` + `npm audit` sur `main`, advisory-db `0bfde9d6`. **Le triage d'origine de #310 révisé sur un point** : `quick-xml` (2× 7.5 high) n'est **compilé sur aucune des deux cibles livrées** — `cargo tree -i` vide sur `x86_64-pc-windows-msvc` **et** `x86_64-unknown-linux-gnu`, présent seulement sur `x86_64-apple-darwin` via `plist`←`tauri` (dép. conditionnelle Apple). Il n'y avait donc **rien à attendre de Tauri**, contrairement au « probablement bloqué en amont » du corps. Méthode retenue : tracer **par triple explicite**, jamais `--target all` (qui répond « présent » pour des deps Apple jamais compilées ici) — mémoire [[reference-cargo-audit-reachability-par-cible]]. 6 advisories atteignables corrigées par `cargo update -p rustls-webpki -p tar` (0.103.9→**0.103.13** couvrant les 4, 0.4.44→**0.4.46**) sans toucher `Cargo.toml` ; dérive de lock expliquée et bornée (7 pointeurs `windows-sys` repointés vers des versions **déjà présentes**, 666 paquets avant/après, deps Windows-only → no-op sur Linux ; `--precise` donne le même résultat, c'est la re-résolution de cargo 1.94.1). Les 3 restantes → `.cargo/audit.toml` versionné, lu par les deux workflows sans qu'aucun ne le sache. **Le plan checker a levé un MAJOR fondé** : une liste de suppressions sans déclencheur de retrait **inverse** le problème de l'alarme (l'audit resterait vert si `quick-xml` redevenait atteignable) — d'où 3 règles en **ADR 0018** : admission sur preuve par cible livrée ou absence de correctif ; clé **par ID d'advisory jamais par crate** (une nouvelle advisory sur le même crate repasse au rouge) ; **garde-fou bloquant** dans `check-rust.yml`, placé après `cargo check` (index chaud) avec `--locked`, testant le **code de sortie séparément de la sortie** (un crate absent et un `cargo tree` en panne impriment tous deux du vide) + **canari** `tar`. Le pendant — suppression devenue *inutile* — est #312. **Leçon du 1er run CI** : le garde était **muet en cas de succès**, donc indiscernable dans les logs d'une étape non exécutée — exactement le mode d'échec silencieux que la PR combat (même famille que le « glob `src-tauri/**` non prouvé » de #232) → il logge maintenant chaque vérification (`Suppression guard: 4 checks, exit 0`, visible run 336). Preuve que `.cargo/audit.toml` est bien lu en conteneur : le `workflow_dispatch` de `audit.yml` sur la branche est **vert**, alors que le bump seul n'aurait éliminé que 6 des 9. **npm** (#311) : `postcss` 8.5.13→**8.5.23 sans override** — `vite` déclare `^8.5.3`, seul le lock était périmé (≠ cas #241 où le parent pinnait) ; `nanoid` suit dans sa borne ; **CSS émis identique octet pour octet** (postcss *est* le pipeline CSS, un build vert n'aurait prouvé que la compilation) ; `react-router` **accepté** — advisory mode **RSC**, or `App.tsx:109` monte un `BrowserRouter` client-only, et `react-router-dom` est **figé à 7.18.1** (v8 a fusionné le paquet dans `react-router`) donc le « fix » npm est un **downgrade** en 7.11.0 → sortir de la plage = migration, pas bump (#317, titre portant `react-router` pour la dédup par sous-chaîne de `/analyse-vulnerabilite`). Asymétrie à connaître : **aucun `npm audit` en CI**, donc rien ne rougit côté front — écrit dans `architecture.md` + `CLAUDE.md` pour que les 2 high permanents ne se lisent pas comme une régression. Gotchas Forgejo relevés : logs de job **uniquement via l'URL web** (l'API v1 répond 404 partout) et `head_branch` = `#<PR>` sur les runs `pull_request` → [[reference-forgejo-ci-logs-et-head-branch]]. 106 Rust + 871 vitest + builds verts. Aucune migration DB (v1→v16). (ref #310, #311, PRs #316/#318, #312-#315, #317)
|
||||
- 2026-07-27 : **#232 mergée — CI split, caches morts retirés, `cargo-audit` pré-buildé** (`main` `7779f7d`, PR #309 rebase, milestone `spec-ci-build-optimization` **3/4**). `/analyze` a confirmé la prémisse mais **révisé le cadrage sur 3 points mesurés** : (1) les 2 jobs ne tournent **pas en parallèle** — runner à capacité 1, `frontend` démarre à la seconde où `rust` finit sur tous les runs de l'historique → chaque PR payait rust+frontend ≈ 24,5 min ; (2) **3 des 40 derniers commits first-parent** touchent `src-tauri/`, dont 2 `chore: release` (push sur `main`, ne déclenche pas la CI) → le path-filter est le **gain dominant**, pas un effet de bord ; (3) le risque « required check skippé » est **nul** (`enable_status_check: false` sur la protection de `main`, vérifié API). Chrono re-mesuré du run 326 : **12m15s de gaspillage sur 21m44** — save `target/` 6m11 + save registry 43s (`reserveCache failed`, cause #234) + `cargo install cargo-audit` 4m41 + restores ~40s ; **nouveau vs baseline #231** : le restore **timeout aussi** désormais (`getCacheEntry failed`), le 30 juin n'avait qu'un MISS propre. Résultat : **rust 21m44 → 8m55** (−59 %), **frontend 2m28 → 1m43**, **PR frontend-only 24m12 → 1m43** (−93 %). **Trou de couverture fermé** : `branches: [main]` ne matchait pas une PR stackée → **#305-#308 (pile feature-gating) n'ont eu aucune CI**, seules #303/#304 (base `main`) ont tourné ; plus aucun filtre `branches:`. 2 écarts assumés vs le corps d'origine, tranchés sur mesure : cache npm retiré aussi (23s/run) et **denylist** pour le frontend (job à 2,5 min → le faire tourner pour rien est bon marché, ne PAS le faire tourner silencieusement ne l'est pas ; le job cher garde un allowlist strict). `/pr-review` **APPROVE** — a récupéré les logs CI plutôt que juger le diff, et levé que l'override `PATH` job-level aurait pu masquer le binaire d'`install-action` (vérifié : audit bien exécuté) ; 2 des 6 suggestions appliquées (`.claude/**` au denylist, ref `check.yml` périmée dans le skill `release`). **Skip prouvé empiriquement** : le 2e push (ni `src-tauri/**` ni `check-rust.yml`) n'a **pas** déclenché `check-rust`. **Reste non prouvé — le glob `src-tauri/**` lui-même** (seule l'entrée chemin-exact a matché) : mode d'échec **silencieux** sur la PR Rust (1/40, ni `cargo check` ni `cargo test`) → #310 (`cargo update` touchant `Cargo.lock` seul) sera le test isolé, à surveiller. Le retrait du `|| true` a **découvert 9 advisories RUSTSEC réelles** (préexistantes, seulement masquées) → **#310** ouverte : `rustls-webpki` ×4 + `tar` ×2 atteignables via `tauri-plugin-updater`/`reqwest` (correctifs **patch**), `quick-xml` ×2 bloqué en amont par `plist`←`tauri` (bump majeur), `rsa` sans correctif publié mais seul parent `sqlx-mysql` — absent de tout arbre de compilation (`cargo tree -i rsa --target all` vide, projet SQLite) donc vraisemblablement **inatteignable**. Conséquence : `audit.yml` (quotidien, bloquant par choix) sera **rouge dès son 1er run** tant que #310 n'est pas traitée — alarme permanente = alarme ignorée, donc à lander tôt. Aucune migration DB (v1→v16). (ref #232, PR #309, #310)
|
||||
- 2026-07-21 : **Milestone `planned-2026-07-19-feature-gating` livrée 6/6 et fermée — le gating par tier est sur `main`** ([Unreleased], candidate v0.15.0). Reprise du run `/autopilot` interrompu le 19 au soir (mort après la PR #303 ; worker #298 tué sans commit — 2 worktrees + branche vide nettoyés). Séquence : `/pr-review` #303 **REQUEST_CHANGES** (le retry backoff CWE-703 s'armait aussi sur un échec de `submitKey` → l'auto-refresh effaçait le message « clé invalide » ~1 s après ; et `status: "error"` persistant aurait laissé `ready=false` à jamais pour les futurs RequireFeature) → fix `b9e13b5` : validation de clé **orthogonale** au lifecycle de chargement (`validationError` dédié, `status` intouché, reducer exporté + 6 tests) → **APPROVE** → merge API. Relance `/autopilot` en session : **5 workers séquentiels en pile linéaire** (#302 dépend de tout ; CHANGELOG centralisé dans #302 → zéro conflit inter-maillons) → PRs #304-#308, 0 needs-clarification, 0 blocked ([rapport](reports/DAILY-REPORT-2026-07-20.md)). `/pr-review` **×5 parallèles → APPROVE ×5** ; seule retouche user-facing : coquille FR `docs.editions` (« tout la » → « tout de la », `6de9617`). Merge : tip cumulé validé (871 vitest + build + cargo 106) puis **ff-only** → push `main` **refusé : branche protégée** (whitelist `maximus`, règle du 2026-03-07, jamais rencontrée avant) — cause réelle : `~/.git-credentials` porte 3 identités Forgejo et `defenseur-auto-bot` en 1re position **shadow** `maximus` (git prend la 1re entrée du host ; les pushes de branches passaient, seule `main` whitelistée refusait). Fix scopé : `git remote set-url origin https://maximus@…` (ne PAS réordonner le fichier partagé — les agents defenseurs s'authentifient par lui). Push OK → 5 issues auto-fermées (`Resolves #N`), 5 PRs fermées + commentées, milestone 6/6 fermée, branches supprimées (+ purge de 7 branches `worktree-agent-*` du harness ; le « résidu `origin/issue-259` » signalé au warmup était un tracking ref périmé faute de `fetch --prune` — la branche remote avait bien été supprimée le 19). Décisions workers notables : tuiles `ProfileSelectionPage` **non** verrouillées (page atteinte seulement sans profil actif résoluble → un verrou heuristique risquerait un soft-lock hors de tous les profils) ; dev-override en **tête** de résolution (sinon un compte Premium actif masquerait `SR_DEV_EDITION`) ; `#[serde(default)]` préexistant sur `features` = les licences Base déjà émises sans le champ ne régressent pas. Gotcha process : un fork `/pr-review` lancé pendant que la CI est `pending` se termine « en attente du monitor CI » **sans jamais poster** (rien ne survit au fork) → monitorer la CI depuis le main loop, relancer le skill après le vert. Post-merge non bloquant : extraction `LockBadge`/`UpsellPanel` si une 3e surface apparaît ; label PIN « (annuler) » (bug cosmétique préexistant) ; job CI ponctuel `--features dev-override`. ADR 0017 Accepted. Aucune migration DB (v1→v16). (ref planned-2026-07-19-feature-gating, #297-#302, PRs #303-#308)
|
||||
|
|
@ -25,10 +26,10 @@ Audit critique de la page Bilan livré (`docs/audit-bilan-2026-05.md`, revue CPA
|
|||
- 2026-07-12 : **#259 recadrée — l'issue décrivait un flux qui n'existe pas** (rectifie les entrées des 2026-07-05 / 07-11 qui la classaient « mapping manuel compte sans similaire auto » et « reliquat de l'epic #260 » : les deux sont faux). Le corps d'origine, rédigé à chaud le 2026-07-05 pendant le run v0.12.0, avait **transposé par analogie** le signalement de Max vers le module Bilan sans ouvrir le fichier : il décrivait une auto-association des comptes standards aux comptes existants du profil dans `StarterAccountsModal`, avec « case désactivée quand aucun similaire n'est auto-identifié ». **Aucune notion de mapping n'existe dans ce flux** — la case est un simple « créer ce compte : oui/non », désactivée **quand une collision EST détectée** (`StarterAccountsModal.tsx:160`), soit la polarité inverse ; le commentaire `l.8` cité (« the matching checkbox ») désignait « la case **correspondante** », pas « la case de matching ». **Le vrai sujet** (confirmé par Max) est la **migration des catégories** : `computeMigrationPlan` range les catégories en 2 seaux et un seul est éditable — `plan.rows` (seed) passe par le moteur d'appariement + type-ahead par ligne (#246/#252), tandis que `plan.preserved` (catégories **custom**) est poussé avec `v1TargetId: null` **sans même être soumis au moteur d'appariement**, rendu en `<li>` texte brut (`StepSimulate.tsx:202-216`, aucun picker) et déversé d'office par le writer sous le fourre-tout « Catégories personnalisées (migration) ». Sémantique tranchée avec Max : **fusion** (choisir une feuille standard réassigne tx/budgets/mots-clés/fournisseurs et fait disparaître la custom ; ne rien choisir = comportement actuel, et ne doit **pas** bloquer le bouton « Suivant »). Re-parentage écarté. Travail = câblage sur 3 couches (UI `StepSimulate` → `MappingRow` réutilisable tel quel ; reducer `RESOLVE_ROW` qui ne voit que `plan.rows` ; 4 retouches du writer) — la machinerie de fusion est **déjà générique** (`buildMappingFromRows` filtre les cibles nulles, étapes 3-7 bouclent sur la Map). Complexité Medium. Piège à couvrir : fusionner une custom **parente ayant des enfants custom** (liste `preserved` plate → l'enfant non résolu retombe au fourre-tout, pas d'orphelin, mais aucun test ne le garantit). Leçon process : une issue rédigée par analogie en fin de run, sans lecture du code, peut inverser la prémisse **et** se tromper de module — le `/analyze` l'a rattrapée 6 jours plus tard. (ref #259)
|
||||
- 2026-07-11 : **M2 « rapports-parite » (#277-#279) shippée** — dernière grosse tranche de l'epic #260 « rapports uniformes ». Run `/autopilot` (3 workers en worktree → PRs #285/#286/#287), reviewée via **3 `/pr-review` adversariales parallèles** (read-only, `git show` sans checkout) : **APPROVE #285** (grille budget income-first, #278) + **APPROVE #286** (BVA réel-vs-budget income-first, #277) + **REQUEST_CHANGES #287** (dashboard Cartes, #279). Blocage #287 = **le point I7 prédit au plan** confirmé réel : `getCartesSnapshot` propageait `accountIds` aux sous-rapports top-movers/budget mais **pas** aux séries `fetchMonthlyFlows` (KPIs/sparklines/overlay 12 mois) ni `fetchSeasonality` → sur le dashboard (1re page à exposer le filtre par-dessus ces séries) les KPI montraient des totaux non filtrés à côté de barres/tendance filtrées. **Fix** (`9ee5ad3`) : threader `accountIds?` dans les 2 fetchers (`source_id IN (...)` paramétré via `inPlaceholders`, index `$3`/`$4`) + câblage `getCartesSnapshot` ; le CHANGELOG #279 promettait déjà le filtrage → **code aligné sur la promesse, texte inchangé** ; +2 assertions cartes. **Merge local de la pile** (3 merges `--no-ff`, conflit additif CHANGELOG ×2 résolu par union #277/#278/#279 — i18n **non conflictuel** car #277/#278 réutilisent des clés existantes, seul #279 touche les locales ; STATE non conflictuel car branches ne l'ont pas commité) → tip cumulé validé (tsc + vite build + **811 vitest** + cargo check + **98 Rust**) → push `main` `a982f9e`. Réconciliation Forgejo : 3 issues auto-fermées via `Resolves #N`, 3 PRs fermées manuellement (merges locaux non détectés *merged*), milestone `overnight-2026-07-08-rapports-parite` **3/3 fermée**, branches supprimées, worktrees leftover nettoyés (gotcha recursion). Aucune migration DB (v1→v16), non taggé ([Unreleased]). Wrinkle process : le fork du skill `/pr-review` avait posté un APPROVE prématuré sur #287 (I7 gradé non bloquant) → superséé + commentaire stale supprimé par la passe adversariale indépendante (trace code + « 1re page à exposer le filtre » + CHANGELOG à honorer) ; valeur de la double-vérif indépendante. Suivi non bloquant : `getDashboardSummary` = code mort à retirer. Reste de #260 : #259 (mapping manuel compte sans similaire auto). (ref #260, #277-#279, PRs #285-#287)
|
||||
- 2026-07-08 : **Suite #260 — M1 « filtres-fondation » mergée** (`main` `fe9ae01`), M2 prête. Planifiée via `/plan-overnight`+`/review-spec` (filtre **multi-comptes** `accountIds[]`/`IN(...)` tranché après analyse, révise le « sourceId singulier » initial). **M1 livrée** via `/autopilot` (5 workers séquentiels en worktree → PRs #280-#284 en **pile linéaire**) + `/pr-review` **APPROVE ×5** + **merge local fast-forward** : hook `useReportsPeriod` additif (+`accountIds` URL `sources` + type `ReportFilters`, #272), 7 services en `accountIds[]` via helper paramétré `sqlFilters.inPlaceholders` (#273), `<FilterPanel>` slot temporel + multi-select « sources d'import » (libellé distinct du Bilan, #274), adoption Tendances (#275) + Compare/Budget (#276 — incl. `CompareBudgetView` ; câblage budget via `getActualTotalsForYear`, le body citait à tort `getBudgetVsActualData`). Aucune migration DB. Build + **791 vitest** verts (728 + 63). NB : le « 4676 / 277 fichiers » vu en pré-merge était une **pollution de worktrees** — `vitest` récursait dans les 5 worktrees leftover sous `.claude/worktrees/` (791 × ~6) ; nettoyés depuis, compteur réel = 791. Gotcha : nettoyer les worktrees (ou exclure `.claude/worktrees/`) avant de valider un tip. Réconciliation Forgejo complète (5 issues auto-fermées via `Resolves #N`, 5 PRs fermées, milestone fermée, branches supprimées ; worktrees nettoyés). **M2 `overnight-2026-07-08-rapports-parite` (#277-#279) prête** → `/autopilot` (BVA + grille budget income-first + dashboard Cartes). Notes review pour M2 : séries `getCartesSnapshot` (`fetchMonthlyFlows`/`fetchSeasonality`) non filtrées (I7/#279) ; grille budget câblée via `getActualTotalsForYear`. (ref #260, #272-#276, PRs #280-#284)
|
||||
- 2026-07-07 : **Tendances hiérarchiques livrées** (1re slice de l'epic #260 « rapports uniformes »). Préparée via `/plan-overnight` (13 décisions drainées ; `/review-spec` 3 experts → 3 critiques résolus par l'approche **sidecar** : garder le pivot par mois pour chart **+ dashboard** (`DashboardPage:142` = consommateur partagé oublié), ajouter un arbre **clé-par-id** pour le tableau — pivot clé-par-nom aurait faussé le Résultat net sur homonymes une fois le `LIMIT 50` retiré), puis exécutée **le même jour** via `/autopilot` (session interactive, ciblée par nom de milestone **hors fenêtre horaire** — cf. mémoire [[feedback-plan-execute-temporal-decoupling]]). 4 workers en worktree → issues **#262-#265**, `/pr-review` **APPROVE ×4**, merge local de la pile (#266 standalone + chaîne #267→#268→#269 ; CHANGELOG **auto-résolu par section**, vérifié) ; tip cumulé validé build + **728 vitest** → push `main` `fe87313`, réconciliation Forgejo (4 PRs fermées, 4 issues auto-fermées via `Resolves #N`, milestone `overnight-2026-07-08-tendances-hierarchiques` fermée, branches supprimées). Contenu : tendance par catégorie → tableau income-statement **hiérarchique + collapse replié-par-défaut** (défaut byCategory+table = fix « paie invisible »), `getCategoryOverTime` en **arbre id-keyed sidecar** via `buildLeafDrivenTree<T>` générique extrait de `buildCompareTree` (behavior-preserving), Résultat avant transferts interleavé. Aucune migration DB, non taggé ([Unreleased]). 2 polish non bloquants : perf dashboard (arbre construit non lu, #264), DRY `typeOf` dupliqué (#265). Reste de #260 (BVA-au-standard, budget, dashboard, filtres partagés) différé. (ref #260, #262-#265, PRs #266-#269)
|
||||
## Blockers actifs
|
||||
|
||||
- Aucun blocker externe dur. `spec-monetisation` **fermée 12/12** (2026-07-08) — #50/#52/#53/#135/#136 livrées, ne sont plus des blockers.
|
||||
- Backlog ouvert (`status:ready`, non bloqué) : `spec-paiements` (#270 activation /v1 + product explicite + URL achat localisée — c'est elle qui activera le CTA « Obtenir » des UpsellGate, aujourd'hui désactivé « bientôt disponible » ; #271 absorbé par #301, livré) ; `spec-ci-build-optimization` **3/4** (#232 livrée le 2026-07-27 ; reste **#234** connectivité du serveur de cache runner, qui débloquerait `Swatinem/rust-cache` — accès hôte VPS requis).
|
||||
- Suivis ouverts des advisories (#310/#311 fermées) : **#314** — `audit.yml` **n'a jamais été déclenché** (zéro run `schedule` sur 420 tâches Actions) ; le `workflow_dispatch` fonctionne, donc le problème est isolé au **scheduler**, et tant qu'il n'est pas réglé la couverture quotidienne n'existe pas, quel que soit l'état vert du workflow. Puis **#312** (retirer les suppressions `quick-xml` quand `plist`/`tauri` passera à `>= 0.41.0` — le garde-fou attrape la suppression devenue injustifiée, #312 celle devenue inutile), **#313** (`tauri-plugin-deep-link 2.4.8` *yanked*, dépendance directe), **#315** (`/release` ne teste aucun cycle téléchargement+extraction réel, alors que `tar`/`rustls-webpki` ne sont ici que compilés), **#317** (re-évaluer `react-router` à une migration v8).
|
||||
- Release à considérer : le `[Unreleased]` porte le feature-gating **et** les deux lots d'advisories → candidate `v0.15.0` via `/release`.
|
||||
- Suivis ouverts du chantier import (milestone fermée) : le refus du format « montant absolu + indicateur D/C » ne scanne que `amountCol ± 1`, donc une colonne D/C placée ailleurs passe encore ; dérives de comptes préexistantes dans `CLAUDE.md` et `docs/architecture.md` (composants, pages, services, `Version actuelle : 0.6.3` vs tag v0.14.0) mesurées et documentées en revue, non corrigées.
|
||||
- Release à considérer : le `[Unreleased]` porte le feature-gating, les deux lots d'advisories **et** le chantier import (migration v17) → candidate `v0.15.0` via `/release`.
|
||||
|
|
|
|||
99
docs/adr/0019-format-import-persiste.md
Normal file
99
docs/adr/0019-format-import-persiste.md
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
# ADR 0019 — Le format d'import est une donnée persistée intégralement, jamais re-devinée
|
||||
|
||||
- Status: **Accepted**
|
||||
- Date: 2026-08-13
|
||||
- Issues: #326 (corpus de fixtures figeant le comportement fautif), #323 (migration v17), #324 (codec `formatToRow`/`formatFromRow`, racine du bug), #325 (règle débit/crédit + parseur ancré), #327 (détection lexicale par libellé), #328 (score de confiance + détection autonome), #329 (aperçu obligatoire + recap signé), #330 (signatures de banques + dérive d'en-tête), #331 (sources et modèles dans le format SREF), #332 (cette doc)
|
||||
- Spec: [`spec-decisions-import-csv-format.md`](../../spec-decisions-import-csv-format.md), [`spec-plan-import-csv-format.md`](../../spec-plan-import-csv-format.md)
|
||||
- Prolonge [ADR 0003](0003-sqlx-migrations.md) (migrations SQL inline additives) pour la migration v17
|
||||
|
||||
## Contexte
|
||||
|
||||
Un import CSV transforme une cellule de texte en un montant signé au grand livre. Deux informations décident du signe, et elles seules :
|
||||
|
||||
- **`amount_mode`** — le fichier porte-t-il un montant unique, ou deux colonnes débit et crédit ?
|
||||
- **`sign_convention`** — en montant unique, une dépense s'écrit-elle négative ou positive ?
|
||||
|
||||
Ces deux champs vivaient sur `import_config_templates` mais **pas** sur `import_sources`. La source — la configuration réellement utilisée à chaque import — ne portait que les réglages mécaniques (délimiteur, encodage, format de date, mapping de colonnes). L'asymétrie était structurelle, et le code la comblait en devinant : `useImportWizard.ts:321-323` ré-inférait le mode depuis `mapping.debitAmount !== undefined` et écrivait `signConvention: "negative_expense"` **en dur**.
|
||||
|
||||
Une source de carte de crédit configurée en dépenses positives revenait donc sur la convention par défaut à son **deuxième** import, et chaque montant était nié : les dépenses arrivaient en revenus. Sans erreur, sans avertissement, et sans contrôle agrégé capable de le voir — un signe inversé ne fait que changer le signe du total.
|
||||
|
||||
Le défaut n'avait pas une voie mais **trois**, indépendantes l'une de l'autre :
|
||||
|
||||
1. **La restauration en dur.** Recharger une source configurée écrasait sa convention par la valeur codée en dur (#324).
|
||||
2. **La dérive d'en-tête.** La banque déplace, renomme ou ajoute une colonne ; le mapping mémorisé continue de pointer sur des positions qui ne désignent plus les mêmes données, et l'import passe sans rien signaler (#330).
|
||||
3. **L'export de données.** `dataExportService` ne sérialisait ni les sources ni les modèles, puis exécutait `DELETE FROM import_sources` à la restauration et les remplaçait par une source synthétique « Data Import ». Restaurer une sauvegarde détruisait toute configuration d'import (#331).
|
||||
|
||||
Les trois produisent la même classe de panne — un format perdu, remplacé par une supposition — et une correction ponctuelle sur l'une n'apprend rien aux deux autres.
|
||||
|
||||
## Décision
|
||||
|
||||
**Le format d'import est une valeur persistée intégralement sur `import_sources`, lue telle quelle, jamais re-inférée.** Quatre règles la tiennent.
|
||||
|
||||
### 1. La base porte le format en entier (migration v17)
|
||||
|
||||
v17 ajoute quatre colonnes à `import_sources`, en additif pur (v1→v16 intactes) :
|
||||
|
||||
| Colonne | Définition |
|
||||
|---|---|
|
||||
| `amount_mode` | `TEXT NOT NULL DEFAULT 'single'` + `CHECK (amount_mode IN ('single','debit_credit','absolute_indicator'))` |
|
||||
| `sign_convention` | `TEXT NOT NULL DEFAULT 'negative_expense'` + `CHECK (sign_convention IN ('negative_expense','positive_expense'))` |
|
||||
| `header_signature` | `TEXT` — libellés normalisés de l'en-tête du dernier import réussi (détection de dérive) |
|
||||
| `template_id` | `INTEGER REFERENCES import_config_templates(id) ON DELETE SET NULL` |
|
||||
|
||||
Le `CHECK` d'`amount_mode` admet `absolute_indicator` **dès maintenant** : le troisième mode de montant pourra être livré sans nouvelle migration, tandis que la base refuse déjà toute valeur corrompue. Le backfill (`UPDATE … WHERE column_mapping LIKE '%debitAmount%'`) reproduit exactement ce que le code déduisait à la volée — v17 ne change aucun comportement observable. `sign_convention` n'est délibérément **pas** backfillée : son `DEFAULT` restitue précisément la valeur que le code codait en dur, la seule convention passée qui puisse être inférée ; en deviner une autre réécrirait silencieusement du sens.
|
||||
|
||||
`LIKE` plutôt que `json_extract`, pour que la migration ne dépende d'aucune extension JSON1 dans le SQLite embarqué.
|
||||
|
||||
### 2. Un codec unique, pas un type composé
|
||||
|
||||
La garantie de complétude vient de **`src/utils/importFormat.ts`**, seul point de conversion entre :
|
||||
|
||||
- `ImportFormatRow` — la forme persistée (snake_case, mapping en JSON, `has_header` normalisé en 0/1) ;
|
||||
- `ImportFormat` — la forme domaine consommée par le wizard et par `mapRow`.
|
||||
|
||||
Un type `ImportFormat` **composé**, partagé par les deux porteurs, était l'approche naturelle et elle est **structurellement impossible** : `ImportSource.has_header` est déclaré `boolean`, `ImportConfigTemplate.has_header` est un `number`, et `SourceConfig` est camelCase sur un mapping déjà parsé. Les porteurs sont incompatibles ; le point de passage, lui, peut être unique.
|
||||
|
||||
Ce qui donne des dents au codec est son **test de complétude**, sur deux niveaux :
|
||||
|
||||
- `FORMAT_FIELD_PAIRS` est typé `Record<keyof ImportFormat, keyof ImportFormatRow>` — un champ ajouté au format **ne compile pas** tant qu'il n'y figure pas ;
|
||||
- le test compare les clés réellement produites par chaque sens du codec à cette table — un champ déclaré mais non câblé **fait échouer le test**.
|
||||
|
||||
Supprimer `sign_convention` de `formatToRow` — la forme exacte du bug d'origine — fait tomber 14 tests. Les deux écrivains de modèles passent aussi par le codec, pour qu'un nouveau champ ne puisse pas atteindre une table et manquer l'autre.
|
||||
|
||||
`formatFromRow` **lève** sur une valeur qu'il ne sait pas lire au lieu de retomber sur un défaut : retomber sur la branche `single` lirait la mauvaise colonne à chaque ligne, et toute valeur autre que `positive_expense` signifierait silencieusement `negative_expense`. L'erreur porte une clé i18n et le wizard s'ouvre alors sur une configuration neuve — bloquer le wizard rendrait « reconfigurer cette source » impossible.
|
||||
|
||||
### 3. `template_id` est une étiquette de provenance, jamais relue comme format
|
||||
|
||||
Un modèle sert à **initialiser** une source ; il ne la gouverne pas. `template_id` est enregistré, restauré et affiché, mais **jamais relu pour dériver un format**. Éditer un modèle ne change aucune source liée — c'est un critère d'acceptation testé de bout en bout, avec vérification de non-vacuité (le modèle a bien changé). `ON DELETE SET NULL` : supprimer le modèle efface l'étiquette et laisse le format intact.
|
||||
|
||||
La raison est la même que celle du reste de l'ADR : si le format se déduisait du modèle, il redeviendrait une valeur devinée, portée par une ligne que l'utilisateur peut modifier ailleurs et à un autre moment.
|
||||
|
||||
### 4. Le format persisté gagne, et les deux autres voies de perte sont fermées
|
||||
|
||||
- **Détection** — elle se déclenche seule sur une source **sans format enregistré**, et **jamais** sur une source qui en a un (garde `!existing`). Le bouton baguette reste, pour la rejouer à la demande. Une source dont le format stocké échoue à décoder émet son erreur explicite plutôt qu'une nouvelle supposition.
|
||||
- **Dérive d'en-tête** — `header_signature` stocke les libellés normalisés du dernier import réussi, en **tableau JSON et non en hachage** : le panneau doit pouvoir nommer les colonnes qui ont bougé (« Montant : colonne 3 → 4 »). Un en-tête qui normalise différemment ouvre `FormatDriftPanel` au-dessus de l'aperçu, avec les deux seules issues qui existent — **adopter** le format re-détecté ou **conserver** celui enregistré. Un simple renommage cosmétique normalise à l'identique et ne dit rien. Une source dont les fichiers n'ont pas de ligne d'en-tête garde `header_signature` à NULL et la détection de dérive y est inopérante : c'est documenté, pas contourné — une signature inventée depuis les données se déclencherait à chaque import.
|
||||
- **Export/import de données** — `import_sources` et `import_config_templates` sont sérialisés dans l'enveloppe SREF derrière un `format_version` explicite. Les modèles sont restaurés **avant** les sources (`template_id` est une clé étrangère), upsertés par nom, et `template_id` est remappé sur les identifiants résolus. Le wipe et la restauration sont enveloppés dans `withTransaction`.
|
||||
|
||||
## Alternatives considérées
|
||||
|
||||
- **Un type `ImportFormat` composé, partagé par les deux tables.** Rejeté : impossible sans réécrire les deux porteurs (`has_header` `boolean` d'un côté, `number` de l'autre, mapping parsé vs JSON). Le codec obtient la même garantie sans toucher aux formes existantes, et son test la rend vérifiable — ce qu'un type partagé n'aurait pas donné à lui seul.
|
||||
- **Backfiller `sign_convention` en devinant depuis les données** (ex. « majorité de montants négatifs ⇒ `negative_expense` »). Rejeté : c'est exactement la re-inférence que cet ADR supprime, appliquée une fois pour toutes et sans possibilité de la relire. Le `DEFAULT` restitue la seule convention réellement en vigueur avant v17.
|
||||
- **Dériver le format depuis `template_id`.** Rejeté : voir la règle 3 — cela redonne au format un porteur mutable ailleurs.
|
||||
- **Hacher `header_signature`.** Rejeté : un hachage détecte la dérive mais ne peut pas la décrire, et le panneau de dérive n'aurait plus rien à montrer que « ça a changé ».
|
||||
- **Corriger rétroactivement les transactions déjà importées à l'envers.** Rejeté : on ne mute jamais des montants déjà écrits. La voie de réparation est `deleteImportWithTransactions` puis re-import — `RepairPathNotice` l'énonce dans l'interface, parce que ré-importer par-dessus **double** les lignes (`findDuplicates` apparie sur date + description + montant, donc une ligne corrigée ne s'apparie pas à la fautive).
|
||||
|
||||
## Conséquences
|
||||
|
||||
- **Positif.** Le format d'une source est lisible en base et rejoué à l'identique. Le codec et son test forment le point de contrôle unique : ajouter un champ de format sans le persister ne compile pas, ou ne passe pas. Les trois voies de perte sont fermées par construction, pas par vigilance.
|
||||
- **Positif.** L'aperçu signé, obligatoire à chaque import, montre ce que le format *signifie* (sorties, entrées, totaux signés, lignes en erreur) — un score de lecture parfait ne dit rien du signe.
|
||||
- **Coût.** Une source dont le format enregistré est illisible (valeur hors `CHECK` arrivée par un chemin non prévu, JSON corrompu) affiche une erreur et repart sur une configuration neuve, au lieu de continuer sur un défaut. C'est le comportement voulu, mais c'est un arrêt visible là où il y avait un silence.
|
||||
- **Coût.** `header_signature` n'est écrite qu'à l'import réussi et reste NULL pour les fichiers sans en-tête : la détection de dérive ne couvre pas ces sources.
|
||||
- **À surveiller.** Le `CHECK` d'`amount_mode` admet `absolute_indicator`, que l'application **refuse** aujourd'hui avec un message dédié plutôt que de l'importer de travers. Le jour où ce mode sera livré, le codec devra l'accepter — la migration, elle, n'est pas à refaire.
|
||||
- **Portée.** Les modules d'import restent entièrement en édition Gratuite ; ce chantier n'ajoute aucun gating ([ADR 0017](0017-feature-gating-par-tier.md)).
|
||||
|
||||
## Liens
|
||||
|
||||
- Spec : [`spec-decisions-import-csv-format.md`](../../spec-decisions-import-csv-format.md), [`spec-plan-import-csv-format.md`](../../spec-plan-import-csv-format.md)
|
||||
- [ADR 0003](0003-sqlx-migrations.md) — migrations SQL inline via tauri-plugin-sql (v17 en suit la convention additive)
|
||||
- [ADR 0004](0004-aes-256-gcm-encryption.md) — chiffrement de l'enveloppe SREF, dont le `format_version` est étendu par #331
|
||||
- `docs/architecture.md` — section « Import CSV — format persisté et détection »
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# Architecture technique — Simpl'Résultat
|
||||
|
||||
> Document mis à jour le 2026-07-20 — Version 0.14.x (gating par édition)
|
||||
> Document mis à jour le 2026-08-13 — Version 0.14.x (format d'import persisté, migration v17)
|
||||
|
||||
## Stack technique
|
||||
|
||||
|
|
@ -32,7 +32,7 @@ simpl-resultat/
|
|||
│ │ ├── budget/ # 5 composants
|
||||
│ │ ├── categories/ # 5 composants
|
||||
│ │ ├── dashboard/ # 2 composants
|
||||
│ │ ├── import/ # 13 composants (wizard d'import)
|
||||
│ │ ├── import/ # 14 composants (wizard d'import, dont FormatDriftPanel et RepairPathNotice)
|
||||
│ │ ├── layout/ # AppShell, Sidebar
|
||||
│ │ ├── profile/ # 3 composants (PIN, formulaire, switcher)
|
||||
│ │ ├── reports/ # ~25 composants (hub, faits saillants, tendances, comparables, zoom catégorie)
|
||||
|
|
@ -44,7 +44,7 @@ simpl-resultat/
|
|||
│ ├── pages/ # 14 pages (dont 4 sous-pages rapports)
|
||||
│ ├── services/ # 14 services métier
|
||||
│ ├── shared/ # Types, constantes, matrice d'entitlements (entitlements.ts), gate profils (profileGate.ts)
|
||||
│ ├── utils/ # 4 utilitaires (parsing, CSV, charts)
|
||||
│ ├── utils/ # 13 utilitaires purs (parsing montants/dates, détection CSV, codec de format d'import, dictionnaire lexical, signatures de banques, charts)
|
||||
│ ├── i18n/ # Config i18next + locales FR/EN
|
||||
│ ├── App.tsx # Router principal
|
||||
│ └── main.tsx # Point d'entrée
|
||||
|
|
@ -81,7 +81,7 @@ simpl-resultat/
|
|||
|
||||
| Table | Description |
|
||||
|-------|-------------|
|
||||
| `import_sources` | Configuration des sources d'import CSV |
|
||||
| `import_sources` | Configuration des sources d'import CSV. Porte le **format d'import en entier** depuis v17 : `amount_mode` (`CHECK ∈ {single, debit_credit, absolute_indicator}`) et `sign_convention` (`CHECK ∈ {negative_expense, positive_expense}`) s'ajoutent aux réglages mécaniques (délimiteur, encodage, format de date, `column_mapping`), plus `header_signature` (libellés normalisés du dernier import réussi, détection de dérive) et `template_id` (**étiquette de provenance**, `ON DELETE SET NULL`, jamais relue comme format) — voir [ADR 0019](adr/0019-format-import-persiste.md) et la section « Import CSV » |
|
||||
| `imported_files` | Suivi des fichiers importés (hash anti-doublons) |
|
||||
| `categories` | Catégories hiérarchiques (dépenses/revenus) |
|
||||
| `suppliers` | Fournisseurs avec auto-catégorisation |
|
||||
|
|
@ -159,6 +159,9 @@ Les migrations sont définies inline dans `src-tauri/src/lib.rs` via `tauri_plug
|
|||
| 14 | v14 | Étape 2 : `balance_securities` + `balance_snapshot_holdings` + 2 index (additive) — [ADR 0015](adr/0015-balance-detail-par-titre.md) |
|
||||
| 15 | v15 | Étape 2 : `balance_accounts.kind` (`simple`/`detailed`) + `detailed_since` + backfill depuis `category.kind` (`priced` → `detailed`) — [ADR 0015](adr/0015-balance-detail-par-titre.md) |
|
||||
| 16 | v16 | Étape 2 : conversion des comptes cotés existants en détaillés 1-position (security + holding miroir, gardée anti-perte, idempotente) — [ADR 0015](adr/0015-balance-detail-par-titre.md) |
|
||||
| 17 | v17 | Import : `import_sources.amount_mode` + `sign_convention` (les deux avec `CHECK`) + `header_signature` + `template_id` (FK `ON DELETE SET NULL`), backfill `amount_mode = 'debit_credit' WHERE column_mapping LIKE '%debitAmount%'` (additive, aucun changement de comportement observable) — [ADR 0019](adr/0019-format-import-persiste.md) |
|
||||
|
||||
v17 est purement additive : elle n'ajoute **ni table ni index** (20 tables / 24 index inchangés). Le `CHECK` d'`amount_mode` admet `absolute_indicator` dès maintenant, pour que le troisième mode de montant puisse être livré sans nouvelle migration, alors que l'application le **refuse** encore explicitement (voir plus bas). `LIKE` plutôt que `json_extract` : la migration ne dépend d'aucune extension JSON1 dans le SQLite embarqué. `sign_convention` n'est délibérément pas backfillée — son `DEFAULT` restitue la valeur que le code codait en dur, la seule convention passée inférable.
|
||||
|
||||
Pour les **nouveaux profils**, le fichier `consolidated_schema.sql` contient le schéma complet avec toutes les migrations pré-appliquées (pas besoin de rejouer les migrations).
|
||||
|
||||
|
|
@ -170,7 +173,7 @@ Pour les **nouveaux profils**, le fichier `consolidated_schema.sql` contient le
|
|||
| `profileService.ts` | Gestion des profils |
|
||||
| `categoryService.ts` | CRUD catégories hiérarchiques |
|
||||
| `transactionService.ts` | CRUD et filtrage des transactions ; détection d'erreurs FK RESTRICT pour transactions liées à un compte de bilan (typed `TransactionLinkedToBalanceError`) |
|
||||
| `importSourceService.ts` | Configuration des sources d'import |
|
||||
| `importSourceService.ts` | Configuration des sources d'import — lit et écrit le format complet via le codec `importFormat.ts` ([ADR 0019](adr/0019-format-import-persiste.md)) |
|
||||
| `importedFileService.ts` | Suivi des fichiers importés |
|
||||
| `importConfigTemplateService.ts` | Modèles de configuration d'import |
|
||||
| `categorizationService.ts` | Catégorisation automatique + helpers édition de mot-clé (`validateKeyword`, `previewKeywordMatches`, `applyKeywordWithReassignment`) |
|
||||
|
|
@ -178,7 +181,7 @@ Pour les **nouveaux profils**, le fichier `consolidated_schema.sql` contient le
|
|||
| `budgetService.ts` | Gestion budgétaire |
|
||||
| `dashboardService.ts` | Agrégation données tableau de bord : `getExpensesByCategory` (barres classées, `accountIds`), `getDashboardSummary` (non consommé depuis #279), `deriveNetWorthTile` (pur — tuile valeur nette du Bilan, réutilise `deriveLandingState`) |
|
||||
| `reportService.ts` | Génération de rapports : `getMonthlyTrends`, `getCategoryOverTime`, `getHighlights`, `getCompareMonthOverMonth`, `getCompareYearOverYear`, `getCategoryZoom` (CTE récursive bornée anti-cycle), `getCartesSnapshot` (snapshot dashboard Cartes, requêtes parallèles) |
|
||||
| `dataExportService.ts` | Export de données (chiffré) |
|
||||
| `dataExportService.ts` | Export de données (chiffré). L'enveloppe SREF porte un `format_version` explicite (`SREF_FORMAT_VERSION = 2`) et **inclut `import_sources` + `import_config_templates`** depuis #331 ; wipe et restauration sous `withTransaction`, modèles restaurés avant les sources avec remap de `template_id` |
|
||||
| `userPreferenceService.ts` | Stockage préférences utilisateur |
|
||||
| `logService.ts` | Capture des logs console (buffer circulaire, sessionStorage) |
|
||||
| `licenseService.ts` | Validation et gestion de la clé de licence (appels commandes Tauri) |
|
||||
|
|
@ -205,7 +208,7 @@ Chaque hook encapsule la logique d'état via `useReducer` :
|
|||
| `useCategories` | Catégories avec hiérarchie |
|
||||
| `useTransactions` | Transactions et filtrage |
|
||||
| `useDataImport` | Import de données |
|
||||
| `useImportWizard` | Assistant d'import multi-étapes |
|
||||
| `useImportWizard` | Assistant d'import multi-étapes. Restaure le format enregistré **tel quel** via `formatFromRow` (aucune ré-inférence), déclenche la détection seule sur une source sans format (garde `!existing`), traverse l'étape `file-preview` à **chaque** import, et n'écrit la configuration qu'à `executeImport` (point d'écriture unique) — voir la section « Import CSV » et l'[ADR 0019](adr/0019-format-import-persiste.md) |
|
||||
| `useImportHistory` | Historique des imports |
|
||||
| `useAdjustments` | Ajustements |
|
||||
| `useBudget` | Budget |
|
||||
|
|
@ -233,6 +236,55 @@ Chaque hook encapsule la logique d'état via `useReducer` :
|
|||
- `storageKey` **non nul** → l'état de repli est **persisté par profil** dans `user_preferences` (via `userPreferenceService`), donc détruit avec le profil, sans résidu `localStorage` ([ADR 0016](adr/0016-persistance-etat-ui-par-profil.md)). Quatre surfaces persistées (les 3 rapports + budget). Hydratation **asynchrone** : le défaut (« tout replié » pour rapports/budget) est rendu d'abord, un `useEffect` hydrate ensuite → pas de flash visible.
|
||||
- `storageKey === null` → état **purement en mémoire**, réinitialisé à chaque montage (les 2 arbres de catégories : navigation éphémère, rien à conserver ni à révéler).
|
||||
|
||||
## Import CSV — format persisté et détection
|
||||
|
||||
Le chantier #323-#332 a fait du **format d'import une donnée persistée intégralement, jamais re-devinée** — [ADR 0019](adr/0019-format-import-persiste.md) porte la décision, ses trois voies de perte et les alternatives rejetées. Quatre couches, du schéma à l'écran.
|
||||
|
||||
### 1. Le codec — `src/utils/importFormat.ts`
|
||||
|
||||
Point de conversion **unique** entre `ImportFormatRow` (persisté : snake_case, mapping en JSON, `has_header` normalisé 0/1) et `ImportFormat` (domaine). Il n'existe pas de type composé partagé par les deux porteurs — c'est structurellement impossible (`ImportSource.has_header` est `boolean`, `ImportConfigTemplate.has_header` est `number`), et la garantie de complétude vient du codec et de son test, pas d'une forme commune :
|
||||
|
||||
- `FORMAT_FIELD_PAIRS: Record<keyof ImportFormat, keyof ImportFormatRow>` — un champ ajouté au format **ne compile pas** tant qu'il n'y est pas listé ;
|
||||
- le test compare les clés réellement produites par `formatToRow` / `formatFromRow` à cette table — un champ listé mais non câblé **fait échouer le test**.
|
||||
|
||||
`formatFromRow` lève (`ImportFormatError`, clé i18n) sur une valeur illisible plutôt que de retomber sur un défaut. Le module porte aussi `mapRow` (la règle de ligne, pure), `detectAmountSeparators`, `summarizeParsedRows` (le récapitulatif signé de l'aperçu), `flipSignFormat` et `clearMappingForMode`.
|
||||
|
||||
**La règle de montant** : `amount = crédit − débit` sur des **magnitudes** (`Math.abs`), ce qui ne demande aucun cas particulier pour la colonne inutilisée remplie `0,00` (zéro est l'élément neutre de la soustraction) — le `isNaN(credit)` d'avant importait chaque débit d'un tel fichier à 0. Une ligne illisible dans les **deux** colonnes est une erreur de ligne, pas un 0,00 gratuit. `parseFrenchAmount` est **ancré** (plus de `parseFloat` rendant le plus long préfixe valide) : `100,00 CAD` rend NaN au lieu de 10000, `50,00-` rend −50, `(50,00)` rend −50. Le séparateur décimal est arbitré **par colonne** (`1.234` vaut 1234 en colonne française et 1.234 en anglaise ; aucune règle appliquée à la cellule seule ne peut trancher). L'écriture de la configuration se fait à `executeImport`, point d'écriture unique — un import abandonné à l'étape des doublons ne laisse aucune configuration derrière lui.
|
||||
|
||||
### 2. La détection — `csvAutoDetect.ts` + `headerDictionary.ts` + `bankSignatures.ts`
|
||||
|
||||
Trois modules distincts, évalués du plus spécifique au plus générique.
|
||||
|
||||
| Module | Rôle |
|
||||
|---|---|
|
||||
| `bankSignatures.ts` | Dispositions d'export documentées de **Desjardins, RBC, BNC et Tangerine** : un jeu de libellés normalisés + délimiteur + préambule. Évalué **avant** le dictionnaire générique. `MIN_SIGNATURE_LABELS = 4` (`Date;Description;Montant` est trop banal pour attribuer un nom de banque). Porte aussi la **signature d'en-tête stockée** (`buildHeaderSignature` / `parseHeaderSignature` / `detectHeaderDrift`) : même objet — une ligne d'en-tête réduite à ses libellés normalisés — vu deux fois |
|
||||
| `headerDictionary.ts` | Dictionnaire lexical FR/EN des rôles de colonnes (date ; description/libellé/détail/transaction ; montant/amount ; débit/retrait/déboursé/withdrawal ; crédit/dépôt/encaissement/deposit ; solde/balance) + `normalizeHeaderCell` / `matchHeaderColumn` / `matchTransactionHeaders`. **Module séparé de `csvAutoDetect.ts` par nécessité** : `montant` est le mot-clé *primaire* du montant pour les transactions et un token d'*exclusion* pour l'import de titres (`VALUE_HEADER_KEYWORDS`, #245) — deux tables qui ne doivent pas se percuter. Les helpers ont été **déplacés** plutôt qu'exportés-puis-réimportés, pour que la dépendance reste à sens unique |
|
||||
| `csvAutoDetect.ts` | Heuristiques de forme (délimiteur, encodage, en-tête, colonnes candidates) + `detectImportFormat`, l'entrée pleine fidélité qui rend soit une config soit une **raison** de refus, et le score |
|
||||
|
||||
**L'ordre débit/crédit vient du libellé, plus de la position.** La paire est toujours trouvée par la forme (colonnes creuses et complémentaires), mais laquelle est le débit est décidé par le dictionnaire — connaître l'une suffit. Avant, `Date;Description;Credit;Debit` inversait **tous** les signes, silencieusement (un total inversé ne fait que changer de signe). `detectHeader` gagne un **signal lexical** : une ligne que le test de forme rejette *uniquement* à cause d'un nombre est un en-tête si elle nomme ≥ 2 rôles (`MIN_HEADER_ROLE_MATCHES`). Chaque indice lexical reste une **préférence que les données peuvent contredire** ; sans en-tête, ou avec des libellés inconnus, tout retombe sur la forme exactement comme avant.
|
||||
|
||||
**Un format est explicitement refusé** : montants absolus accompagnés d'une colonne d'indicateur `D`/`C` (ou `DB`/`CR`) adjacente portant ≥ 2 valeurs distinctes. Il était auparavant indiscernable d'un fichier tout-positif, déduit `positive_expense`, et **chaque dépôt s'importait en dépense**. Message dédié (`import.errors.absoluteIndicatorFormat`) plutôt qu'un import de travers.
|
||||
|
||||
**Le score de confiance** (`DetectionScore` : `readRows`, `totalRows`, `ratio`, `confident`, seuil `CONFIDENCE_THRESHOLD = 0.9`) rejoue la configuration décidée sur **toutes** les lignes du fichier via `mapRow` — la fonction de production, pas un second mappeur — sous la même arbitration décimale. Il **colore, il ne bloque pas** : un score parfait ne dit rien du **signe** (le fixture `all-positive` score 100 % alors que chaque crédit s'importe en dépense). La détection se déclenche **seule** sur une source sans format enregistré, et **jamais** sur une source qui en a un (garde `!existing`) — le format stocké gagne. Le score est invalidé au changement de source, à l'édition manuelle du format et au chargement d'un modèle.
|
||||
|
||||
### 3. Le wizard — l'aperçu est obligatoire
|
||||
|
||||
L'étape `file-preview`, déclarée depuis l'origine dans `ImportWizardStep` sans qu'aucun dispatch ne la vise, est **rendue et traversée à chaque import** ; `FilePreviewModal` (l'aperçu optionnel qui l'avait supplantée) est supprimé. `checkDuplicates` en est le bouton « suivant », donc les lignes validées sont exactement les lignes vérifiées, sans second parsing entre les deux. L'étape n'est gatée par rien — surtout pas par le seuil de 90 %, puisque le cas qu'elle attrape est précisément le score parfait au signe inversé.
|
||||
|
||||
- **Récapitulatif signé** (`summarizeParsedRows`) : sorties et leur total, entrées et le leur, lignes en erreur. Les totaux restent **signés** — des magnitudes masqueraient la seule chose que le récap expose. Calculé sur tout le fichier, jamais sur les vingt lignes affichées.
|
||||
- **« Inverser les signes »** (`flipSignFormat`) agit sur la **configuration**, donc la correction est mémorisée avec la source. En mode `single` elle bascule la convention ; en `debit_credit` elle **échange les deux indices de colonnes**, parce que `mapRow` y calcule `crédit − débit` sur des magnitudes et ne lit jamais la convention — une bascule y aurait été inerte.
|
||||
- **`FormatDriftPanel`** s'ouvre au-dessus de l'aperçu quand l'en-tête normalise différemment de `header_signature`, colonne par colonne (« Montant : colonne 3 → 4 »), avec les deux seules issues qui existent : **adopter** le format re-détecté ou **conserver** celui enregistré. Un renommage cosmétique normalise à l'identique et ne dit rien. Une source dont les fichiers n'ont pas d'en-tête garde `header_signature` NULL et la détection de dérive y est inopérante — documenté, pas contourné.
|
||||
- **`RepairPathNotice`**, rendu dans le panneau de dérive **et** à côté du bouton d'inversion : `findDuplicates` apparie sur date + description + montant, donc ré-importer un fichier « maintenant qu'il se lit bien » ne corrige pas les lignes déjà écrites — il les **double**, et une inversion de signe produit des paires miroir qui s'annulent dans tous les rapports. La seule voie sûre est `deleteImportWithTransactions` puis rejeu. Aucun montant déjà écrit n'est jamais muté.
|
||||
- **`ImportConfirmation`** énonce désormais le mode de montant, la convention de signe (seulement dans le mode qui l'applique) et le mapping nommé par en-tête. Le sélecteur de convention est **masqué** en mode débit/crédit, où le parsing ne le lit pas — masqué, pas réinitialisé : la valeur stockée est intacte.
|
||||
|
||||
### 4. Le format SREF porte les sources et les modèles
|
||||
|
||||
`dataExportService` sérialisait uniquement catégories, fournisseurs, mots-clés et transactions, puis exécutait `DELETE FROM import_sources` à la restauration en les remplaçant par une source synthétique « Data Import » — restaurer une sauvegarde détruisait toute configuration d'import. Depuis #331 : `import_sources` et `import_config_templates` sont dans l'enveloppe derrière un `format_version` explicite (`SREF_FORMAT_VERSION = 2` ; une enveloppe sans le champ est l'ancien format, ses tableaux absents traités comme vides). Les **modèles sont restaurés avant les sources** (`template_id` est une FK), upsertés par nom, et `template_id` remappé sur les identifiants résolus. Wipe et restauration des deux chemins sont enveloppés dans `withTransaction` — le service n'en avait aucune, et une violation de contrainte en cours de restauration détruisait l'historique financier sans rollback. `amount_mode` et `sign_convention` sont whitelistés à la frontière d'import avec un message lisible (le `CHECK` v17 refuse déjà les mauvaises valeurs, mais une erreur de contrainte SQLite n'est pas un message utilisateur).
|
||||
|
||||
### Corpus de fixtures — `src/__fixtures__/csv/`
|
||||
|
||||
15 fichiers CSV **synthétiques** (aucune donnée réelle, aucun relevé réel n'entre dans ce dépôt — les signatures de banques sont écrites depuis des dispositions documentées). Ils couvrent des formes qu'un vrai relevé ne réunit jamais toutes : montant signé, débit/crédit, débit/crédit inversés, colonne inutilisée à `0,00`, préambule, en-tête contenant un nombre, en-tête à libellé numérique, sans en-tête, tout-positif, Desjardins quoté, montant absolu + indicateur, plus les quatre dispositions de banque. Posés en #326 **avant** la refonte, avec le comportement fautif figé et marqué `KNOWN DEFECT` + l'issue devant le corriger ; chaque maillon suivant a mis à jour l'attente et retiré le marqueur, sans jamais supprimer un test.
|
||||
|
||||
## Gating par édition (licence)
|
||||
|
||||
Depuis le chantier #297-#301, l'accès aux modules est gaté par l'édition de licence (`free` / `base` / `premium`) **côté interface uniquement** — un soft-paywall assumé (l'app est GPL ; le seul gate appliqué côté serveur reste la récupération de cours Premium via maximus-api). La matrice complète tier → modules, le rationale et les alternatives rejetées sont dans l'[ADR 0017](adr/0017-feature-gating-par-tier.md).
|
||||
|
|
@ -383,7 +435,7 @@ Les modules payants sont enveloppés dans des **layout-routes pathless `RequireF
|
|||
| Route | Page | Description |
|
||||
|-------|------|-------------|
|
||||
| `/` | `DashboardPage` | Tableau de bord (KPIs+deltas, top movers, adhérence budget, tuile valeur nette, barres classées, dépenses dans le temps — modèle Cartes, #279) |
|
||||
| `/import` | `ImportPage` | Assistant d'import CSV |
|
||||
| `/import` | `ImportPage` | Assistant d'import CSV : source → configuration (détection automatique + score) → sélection des fichiers → **aperçu obligatoire** (récap signé, inversion des signes, panneau de dérive) → doublons → confirmation |
|
||||
| `/transactions` | `TransactionsPage` | Liste avec filtres |
|
||||
| `/categories` | `CategoriesPage` | Gestion hiérarchique |
|
||||
| `/adjustments` | `AdjustmentsPage` | Ajustements manuels |
|
||||
|
|
@ -478,3 +530,5 @@ Les ADRs documentent les décisions techniques structurantes. Ils vivent dans `d
|
|||
| [0015](adr/0015-balance-detail-par-titre.md) | Bilan : détail par titre (holdings par snapshot, Étape 2) | 2026-06-06 | Accepted |
|
||||
| [0016](adr/0016-persistance-etat-ui-par-profil.md) | Persistance de l'état UI par profil : repli des catégories dans `user_preferences` | 2026-07-15 | Accepted |
|
||||
| [0017](adr/0017-feature-gating-par-tier.md) | Gating des fonctionnalités par édition : matrice UI statique, override signé fail-closed, soft-paywall assumé | 2026-07-20 | Accepted |
|
||||
| [0018](adr/0018-suppression-advisories-non-atteignables.md) | Suppression d'advisories non atteignables : admission sur preuve par cible livrée, clé par ID, garde-fou bloquant | 2026-07-27 | Accepted |
|
||||
| [0019](adr/0019-format-import-persiste.md) | Le format d'import est une donnée persistée intégralement, jamais re-devinée | 2026-08-13 | Accepted |
|
||||
|
|
|
|||
|
|
@ -100,8 +100,12 @@ Importez des relevés bancaires à partir de fichiers CSV à l'aide d'un assista
|
|||
|
||||
### Fonctionnalités
|
||||
|
||||
- Assistant d'import multi-étapes avec aperçu des données
|
||||
- Mapping de colonnes configurable, délimiteur et format de date
|
||||
- Détection automatique du format à l'ouverture d'une source jamais configurée : délimiteur, encodage, format de date et colonnes reconnues par leur libellé (français et anglais)
|
||||
- Formats reconnus par banque — Desjardins, RBC, Banque Nationale et Tangerine sont lus par leur nom
|
||||
- Score de confiance affiché après détection (« 147 des 150 lignes lues ») : il indique ce qui a pu être lu, jamais si le sens est bon
|
||||
- Aperçu obligatoire avant chaque import, avec un récapitulatif signé : sorties, entrées, totaux et lignes en erreur, plus un bouton « Inverser les signes »
|
||||
- Avertissement quand l'en-tête d'un fichier ne correspond plus à celui du dernier import réussi, colonne par colonne
|
||||
- Mapping de colonnes configurable, délimiteur, encodage et format de date modifiables à tout moment
|
||||
- Détection automatique des doublons (dans le lot et contre les données existantes)
|
||||
- Modèles d'import pour sauvegarder et réutiliser les configurations
|
||||
- Historique des imports avec possibilité de supprimer les imports précédents
|
||||
|
|
@ -110,16 +114,24 @@ Importez des relevés bancaires à partir de fichiers CSV à l'aide d'un assista
|
|||
|
||||
1. Définissez votre dossier d'import via le sélecteur de dossier en haut de la page
|
||||
2. Créez un sous-dossier pour chaque banque/source et placez-y les fichiers CSV
|
||||
3. Cliquez sur une source pour ouvrir l'assistant d'import
|
||||
4. Configurez le délimiteur, l'encodage, le format de date et le mapping des colonnes
|
||||
5. Sélectionnez les fichiers à importer et prévisualisez les données analysées
|
||||
6. Vérifiez les doublons, examinez le résumé, puis confirmez l'import
|
||||
3. Cliquez sur une source pour ouvrir l'assistant d'import — la première fois, le format est détecté automatiquement et un bandeau annonce le résultat
|
||||
4. Vérifiez la configuration proposée (délimiteur, encodage, format de date, mapping des colonnes) et corrigez-la si le bandeau signale un format incertain
|
||||
5. Sélectionnez les fichiers à importer
|
||||
6. **Examinez l'aperçu** : le récapitulatif indique combien de sorties et d'entrées le fichier produira, et pour quels montants
|
||||
7. Si les entrées et les sorties sont inversées, cliquez sur « Inverser les signes » — la correction est enregistrée avec la source, les prochains fichiers de cette banque se liront correctement d'eux-mêmes
|
||||
8. Vérifiez les doublons, examinez le résumé, puis confirmez l'import
|
||||
|
||||
### Astuces
|
||||
|
||||
- Sauvegardez votre configuration comme modèle pour ne pas avoir à reconfigurer à chaque fois
|
||||
- Le récapitulatif de l'aperçu est le seul contrôle qui voit le **sens** des montants : un score de 100 % signifie que chaque ligne a été lue, pas que vos dépenses ne sont pas comptées comme des revenus. Prenez deux secondes pour le lire.
|
||||
- Votre configuration est mémorisée intégralement sur la source, y compris le mode de montant et la convention de signe : une source configurée une fois n'est plus jamais re-devinée aux imports suivants
|
||||
- Si votre banque change la disposition de ses colonnes, un panneau le signale avant l'import et propose deux choix : adopter le nouveau format détecté, ou conserver celui enregistré
|
||||
- Pour réparer un import déjà écrit de travers, **ne ré-importez pas le fichier par-dessus** : les doublons sont repérés sur la date, la description ET le montant, donc une ligne corrigée s'ajoute au lieu de remplacer la fautive. Supprimez d'abord l'import fautif dans l'historique, puis rejouez le fichier.
|
||||
- Un relevé à montants positifs accompagné d'une colonne « D / C » est refusé plutôt qu'importé à l'envers : réexportez-le depuis votre banque avec des montants signés, ou avec deux colonnes débit et crédit séparées
|
||||
- Sauvegardez votre configuration comme modèle pour ne pas avoir à reconfigurer à chaque fois. Un modèle sert à initialiser une source ; le modifier ensuite ne change aucune source déjà configurée.
|
||||
- Les fichiers déjà importés sont marqués d'un badge — les ré-importer déclenchera la détection de doublons
|
||||
- Vous pouvez supprimer un import de l'historique pour retirer toutes ses transactions
|
||||
- Vos sources et vos modèles d'import font désormais partie de la sauvegarde chiffrée : restaurer un backup ne vous oblige plus à tout reconfigurer
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
79
spec-decisions-import-csv-format.md
Normal file
79
spec-decisions-import-csv-format.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
# Spec Decisions — Import CSV et reconnaissance de format
|
||||
|
||||
> Date: 2026-08-12
|
||||
> Projet: simpl-resultat
|
||||
> Statut: Draft
|
||||
> Slug: import-csv-format
|
||||
|
||||
## Contexte
|
||||
|
||||
L'import CSV est la porte d'entrée du produit : sans lui, aucune autre page n'a de données. Le module a été conçu au début du projet et n'a pas été retouché depuis, alors que le reste de l'app (Bilan, rapports, gating) a été refondu plusieurs fois. Une revue d'implémentation menée le 2026-08-12 a établi que le symptôme rapporté — « l'app oublie le format, y compris les colonnes de montant positif/négatif » — n'est pas une faiblesse d'heuristique mais un trou de persistance, doublé de plusieurs voies de corruption silencieuse.
|
||||
|
||||
**Le bug racine.** La table `import_sources` ne possède ni `amount_mode` ni `sign_convention` (`consolidated_schema.sql:8-20`) ; ces deux colonnes n'existent que sur `import_config_templates` (`consolidated_schema.sql:147-159`). À la restauration d'une source configurée, `useImportWizard.ts:323` écrit donc `signConvention: "negative_expense"` en dur. Une source réglée en `positive_expense` — cas classique d'un relevé de carte de crédit où les dépenses sont positives — revient au défaut inverse au deuxième import, et `useImportWizard.ts:514` applique alors la négation à contresens : **toutes les dépenses deviennent des revenus, sans la moindre erreur affichée**. Le mode de montant subit le même sort, re-deviné depuis la présence de `mapping.debitAmount` (`useImportWizard.ts:321`) plutôt que lu ; comme `ColumnMappingEditor.tsx:82-94` ne nettoie pas le mapping au changement de mode, un basculement non suivi d'une re-sélection de colonne se perd ou s'inverse au rechargement.
|
||||
|
||||
La documentation Desjardins confirme que le cas n'est pas théorique : le signe des montants diffère selon le type de compte, et un même utilisateur a couramment un compte-chèque et une carte de crédit dans deux conventions opposées.
|
||||
|
||||
**Corruption silencieuse au parsing.** `useImportWizard.ts:509` calcule `amount = isNaN(credit) ? -debit : credit` — le crédit gagne toujours. Beaucoup de banques écrivent `0,00` dans la colonne inutilisée plutôt que de la laisser vide : tous les débits deviennent alors 0 $, et la ligne passe la validation puisque `isNaN(0)` est faux. Les fallbacks `?? 0` des lignes 504-512 lisent la colonne 0 — souvent la date — quand le mapping est incomplet.
|
||||
|
||||
**Détection aveugle aux en-têtes.** `csvAutoDetect.ts:461-470` assigne le débit et le crédit par ordre de colonne, si bien qu'un fichier `Date;Description;Crédit;Débit` est inversé intégralement. `detectSingleAmount` (l.507-530) déduit la convention au vote majoritaire de négatifs sur 20 lignes. `detectHeader` (l.214-238) retourne `!hasDate && !hasNumber`, donc un en-tête contenant « Solde 2024 » passe pour une ligne de données. Le savoir-faire manquant existe pourtant dans le même fichier : le flux d'import de titres (#245, juillet 2026) fait du matching par libellé via `normalizeHeaderCell` et `matchHeaderColumn` (l.633-666).
|
||||
|
||||
**Rien ne rattrape l'erreur.** La détection auto n'est jamais lancée d'office — elle est derrière un bouton (`SourceConfigPanel.tsx:69-77`), et une source neuve démarre sur un défaut plausible (`;`, `DD/MM/YYYY`, colonnes 0/1/2) qui produit un import faux plutôt qu'une erreur franche. `autoDetectConfig` ne teste jamais sa propre config sur les données. L'aperçu est un modal optionnel de 20 lignes sans totaux. L'écran de confirmation affiche délimiteur, encodage, format de date et lignes ignorées, mais **ni le mode de montant, ni la convention de signe, ni le mapping** (`ImportConfirmation.tsx:57-80`). Enfin la config est écrite en base dès l'étape doublons (`useImportWizard.ts:588-606`), donc un import annulé persiste quand même une configuration potentiellement fausse.
|
||||
|
||||
**Aucun filet.** `csvAutoDetect.test.ts` ne couvre que le flux holdings. `autoDetectConfig`, `detectAmountMode`, `preprocessQuotedCSV` et l'intégralité de `useImportWizard` n'ont aucun test, sur les 871 que compte le projet.
|
||||
|
||||
## Objectif
|
||||
|
||||
Faire du format d'import une donnée persistée intégralement et vérifiée, plutôt qu'un ensemble de réglages partiellement mémorisés et re-devinés à chaque passage. La reconnaissance s'appuie sur les libellés d'en-tête plutôt que sur la seule forme des données, annonce sa confiance, et le wizard impose un contrôle visuel des montants signés avant toute écriture en base.
|
||||
|
||||
## Scope
|
||||
|
||||
### IN
|
||||
|
||||
- Migration v17 : `amount_mode` et `sign_convention` sur `import_sources`, avec backfill préservant le comportement actuel.
|
||||
- Persistance et restauration du format complet ; suppression de la valeur en dur et de la ré-inférence du mode.
|
||||
- Le mode de montant devient la source de vérité du mapping : changer de mode nettoie les colonnes de l'autre mode.
|
||||
- Règle débit/crédit corrigée (`crédit − débit` sur magnitudes), gestion de la colonne inutilisée à `0,00`, suppression des fallbacks `?? 0` au profit d'une erreur de ligne explicite.
|
||||
- Détection par libellé d'en-tête, dictionnaire FR/EN, réutilisant les helpers du flux holdings.
|
||||
- Détection auto lancée d'office sur une source non configurée.
|
||||
- Score de confiance : la config détectée est rejouée sur les données et le taux de lignes lues est affiché.
|
||||
- Aperçu promu en étape obligatoire du wizard, avec récapitulatif signé (sorties / entrées, totaux) et bascule de convention en un geste.
|
||||
- Récapitulatif du format complet — mode, convention, mapping — sur l'écran de confirmation.
|
||||
- La config n'est plus écrite en base avant confirmation de l'import.
|
||||
- Signatures de banques reconnues automatiquement (Desjardins, RBC, BNC, Tangerine), sans sélecteur de banque.
|
||||
- Détection de dérive : signature d'en-tête mémorisée, re-détection et présentation de l'écart au ré-import.
|
||||
- `parseFrenchAmount` : parenthèses comptables `(50,00)` et signe suffixe `50,00-`.
|
||||
- Sauvegarde et restauration des configurations de sources et des modèles dans l'export/import de données (format SREF).
|
||||
- Corpus de fixtures CSV synthétiques + tests sur la détection, le parsing et le cycle sauvegarde/restauration du format.
|
||||
|
||||
### OUT (explicitement exclu)
|
||||
|
||||
- Toute correction rétroactive des transactions déjà importées avec un signe inversé. La voie de réparation existe déjà (`deleteImportWithTransactions` — supprimer l'import fautif et le rejouer) et aucune mutation automatique de données financières déjà catégorisées et budgétées ne sera ajoutée.
|
||||
- Le troisième mode de montant « montant absolu + colonne indicateur » (`D`/`C`, `DB`/`CR`). Hors des formats des banques canadiennes personnelles visées.
|
||||
- Suppression ou fusion de `import_config_templates`.
|
||||
- Import d'autres formats que CSV (OFX, QFX, QIF, PDF).
|
||||
- Modification du gating : l'import reste entièrement en édition Free.
|
||||
|
||||
## Decisions prises
|
||||
|
||||
| Question | Decision | Raison |
|
||||
|----------|----------|--------|
|
||||
| Portée du chantier | Les trois vagues de la revue en un seul chantier | Les vagues 2 et 3 dépendent du socle de persistance de la vague 1 ; les séparer imposerait de rouvrir les mêmes fichiers trois fois. |
|
||||
| Transactions déjà importées à l'envers | Aucune correction rétroactive | `deleteImportWithTransactions` couvre déjà le besoin. Une inversion en masse mutilerait des données déjà catégorisées et budgétées pour un gain qu'un ré-import obtient sans risque. |
|
||||
| Presets banques | Signatures reconnues automatiquement | Cohérent avec la détection par en-tête, aucune liste à maintenir dans l'UI, aucun choix de plus à faire, et un preset ne peut pas être appliqué à tort. Un fichier inconnu retombe sur le dictionnaire générique. |
|
||||
| Corpus de tests | Fixtures synthétiques | Elles couvrent chaque clause du contrat, y compris les cas qu'un vrai relevé ne contient pas (débit/crédit inversés, colonne à `0,00`, en-tête numérique). Aucune donnée réelle au dépôt, démarrage immédiat. |
|
||||
| Dérive de format | Re-détecter et présenter l'écart | Ni blocage sec sur un écart bénin, ni passage en silence sur un mapping périmé. L'utilisateur arbitre sur un diff lisible. |
|
||||
| Statut de l'aperçu | Étape obligatoire du wizard | Décision de Max, contre la recommandation d'un aperçu conditionné à la confiance. L'étape `file-preview` existe déjà dans `ImportWizardStep` sans avoir jamais été rendue — le modal l'avait supplantée. |
|
||||
| Modèles de configuration | Conservés, schéma aligné sur celui des sources | Le modèle reste un format nommé réutilisable entre plusieurs comptes d'une même banque ; la source porte le format en vigueur. Mêmes champs des deux côtés, l'asymétrie qui a causé le bug devient impossible. |
|
||||
| Configurations de sources dans l'export de données | Sérialisées à l'export, restaurées à l'import | Découvert au re-ancrage de Phase 3b : l'export ne porte que catégories, fournisseurs, mots-clés et transactions, et l'import fait `DELETE FROM import_sources` (`dataExportService.ts:265` et `:362`) avant de créer une source factice « Data Import ». Restaurer une sauvegarde détruit donc toutes les configurations d'import. Troisième voie de perte du format, indépendante du bug racine et définitive ; la laisser ouverte viderait le chantier de son sens. |
|
||||
| Troisième mode de montant | Hors scope, sans le fermer | `amount_mode` reste sans contrainte `CHECK` en base, donc l'ajouter plus tard ne demandera aucune migration. Un fichier de ce type sera refusé explicitement plutôt qu'importé de travers. |
|
||||
| Exécution | Milestone `planned-2026-08-12-import-csv-format`, prête pour `/autopilot` | Bodies auto-suffisants, découpage en pile linéaire — la migration et le socle de détection sont des dépendances de presque tout le reste. |
|
||||
|
||||
## References
|
||||
|
||||
| Source | Pertinence |
|
||||
|--------|------------|
|
||||
| [CSV Format Bank Statement: UK Data Mapping Guide](https://snyp.ai/blog/csv-format-bank-statement) | Confirme la règle « une seule logique de montant par fichier » — ne jamais mélanger montant signé et colonnes débit/crédit séparées. Le bug `useImportWizard.ts:509` vient précisément d'un mélange mal arbitré. Recense les libellés sources à normaliser (Withdrawal, Deposit) vers un schéma cible Date / Description / Débit / Crédit / Montant / Solde — matière directe du dictionnaire d'en-têtes. |
|
||||
| [Bank Statement CSV Format for Clean Accounting Imports](https://thebankstatementconverter.app/bank-statement-csv-format) | Établit que dans un format à deux colonnes, débit et crédit sont **tous deux positifs** — d'où la règle `crédit − débit` sur magnitudes retenue, et non la comparaison de nullité actuelle. |
|
||||
| [How to Export a CSV from Desjardins (AccèsD)](https://www.flowvista.ca/guides/export-csv-desjardins) | Format Desjardins : Date, Description, Montant, Solde ; point-virgule ; virgule décimale ; en-têtes FR ou EN. Signale que certains exports de cartes de crédit n'ont pas de ligne d'en-tête et que **le signe des montants diffère selon le type de compte** — validation externe du bug racine. Base de la signature Desjardins. |
|
||||
| [Import a CSV bank statement — Xero Central](https://central.xero.com/0/article/Import-a-CSV-bank-statement) | Documente la troisième convention (montant absolu + indicateur `D`/`C`) et la pratique du multiplicateur `-1` sur la colonne crédit. Sert à cadrer ce qui est laissé hors scope et à garder `amount_mode` extensible. |
|
||||
| `csvAutoDetect.ts:633-737` (interne) | Le flux d'import de titres (#245) implémente déjà la détection par libellé d'en-tête avec normalisation des accents. Modèle à généraliser au flux transactions plutôt qu'à réécrire. |
|
||||
366
spec-plan-import-csv-format.md
Normal file
366
spec-plan-import-csv-format.md
Normal file
|
|
@ -0,0 +1,366 @@
|
|||
# Spec Plan — Import CSV et reconnaissance de format
|
||||
|
||||
> Date: 2026-08-12
|
||||
> Projet: simpl-resultat
|
||||
> Statut: Draft
|
||||
> Slug: import-csv-format
|
||||
> Decisions: [spec-decisions-import-csv-format.md](./spec-decisions-import-csv-format.md)
|
||||
|
||||
## Design
|
||||
|
||||
### UX / Interface
|
||||
|
||||
Le wizard passe de six étapes rendues à sept. `ImportWizardStep` déclare déjà `file-preview` (`types/index.ts:543`) sans que `ImportPage.tsx` ne la rende jamais — le modal optionnel l'avait supplantée. L'étape est restaurée et devient obligatoire.
|
||||
|
||||
**Étape configuration.** À l'ouverture d'une source jamais configurée, la détection se lance seule ; le bouton baguette magique reste pour la rejouer. Le panneau gagne un bandeau de résultat : soit une banque reconnue par signature (« Format Desjardins reconnu »), soit un score générique (« Format reconnu — 147 des 150 lignes lues »), soit un avertissement sous le seuil. Le sélecteur de convention de signe, aujourd'hui affiché en permanence alors que le parsing ne l'applique qu'en mode simple (`useImportWizard.ts:514`), n'apparaît plus qu'en mode montant unique.
|
||||
|
||||
**Étape aperçu.** Nouvelle étape traversée à chaque import. Au-dessus du tableau des vingt premières lignes, un récapitulatif signé : nombre de sorties et total, nombre d'entrées et total, nombre de lignes en erreur. C'est le contrôle qui rattrape visuellement toute erreur de convention, quelle qu'en soit la cause. Un bouton *Inverser les signes* bascule `sign_convention` et relance le parsing — il agit sur la configuration, jamais sur les données seules, pour que la correction soit mémorisée.
|
||||
|
||||
**Étape confirmation.** Le récapitulatif des réglages passe de quatre entrées à sept : le mode de montant, la convention de signe et le mapping des colonnes rejoignent délimiteur, encodage, format de date et lignes ignorées.
|
||||
|
||||
> **🟡 SECURITE** — « Inverser les signes » est inerte en mode débit/crédit. `sign_convention` n'est appliqué que dans la branche montant unique (`useImportWizard.ts:514`), et l'issue 6 masque son sélecteur en mode débit/crédit : le bouton présenté comme rattrapant « toute erreur de convention, quelle qu'en soit la cause » ne fait rien sur un fichier dont les deux colonnes sont mappées à l'envers.
|
||||
> **Resolution :** En mode débit/crédit, le même bouton permute `debitAmount` et `creditAmount` dans le mapping puis relance le parsing.
|
||||
|
||||
**Dérive de format.** Au ré-import d'une source connue dont l'en-tête ne correspond plus à la signature mémorisée, un panneau présente l'écart colonne par colonne (« Montant : 3 → 4 ») avec deux issues : adopter la configuration re-détectée, ou conserver l'ancienne.
|
||||
|
||||
### Donnees
|
||||
|
||||
**Migration v17** — additive, `v1` à `v16` intactes.
|
||||
|
||||
```sql
|
||||
ALTER TABLE import_sources ADD COLUMN amount_mode TEXT NOT NULL DEFAULT 'single'
|
||||
CHECK (amount_mode IN ('single','debit_credit','absolute_indicator'));
|
||||
ALTER TABLE import_sources ADD COLUMN sign_convention TEXT NOT NULL DEFAULT 'negative_expense'
|
||||
CHECK (sign_convention IN ('negative_expense','positive_expense'));
|
||||
ALTER TABLE import_sources ADD COLUMN header_signature TEXT;
|
||||
ALTER TABLE import_sources ADD COLUMN template_id INTEGER REFERENCES import_config_templates(id) ON DELETE SET NULL;
|
||||
UPDATE import_sources SET amount_mode = 'debit_credit'
|
||||
WHERE column_mapping LIKE '%debitAmount%';
|
||||
```
|
||||
|
||||
Le backfill reproduit exactement la règle appliquée aujourd'hui à la volée (`useImportWizard.ts:321`), donc aucune source ne change de comportement à la migration. Le test `LIKE` est préféré à `json_extract` pour ne dépendre d'aucune extension JSON1 dans le SQLite embarqué.
|
||||
|
||||
Les deux colonnes d'énumération portent un `CHECK`, conformément au pattern de la v15 sur `balance_accounts.kind`. Celui d'`amount_mode` accepte d'emblée `absolute_indicator`, la valeur du troisième mode laissé hors scope : la contrainte protège dès aujourd'hui contre une valeur corrompue, et le mode pourra être implémenté plus tard sans migration. La liste blanche applicative de la frontière SREF reste nécessaire — elle produit un message lisible là où la base ne rendrait qu'une erreur de contrainte.
|
||||
|
||||
> **🟢 ARCHITECTURE** — L'objectif (ajouter le 3e mode sans migration) s'atteint sans renoncer à la contrainte : `CHECK (amount_mode IN ('single','debit_credit','absolute_indicator'))` tient dès aujourd'hui et accepte déjà la valeur future.
|
||||
> **Resolution :** Écrire le `CHECK` élargi en v17 plutôt que de l'omettre.
|
||||
|
||||
> **🟡 SECURITE** — Sans `CHECK`, une valeur inconnue restaurée depuis une sauvegarde SREF éditable retombe en silence sur un défaut : `useImportWizard.ts:501` branche `if (amountMode === "debit_credit") … else`, donc toute autre valeur lit la mauvaise colonne, et toute valeur autre que `positive_expense` signifie `negative_expense`. C'est exactement la classe d'erreur silencieuse que le chantier existe pour tuer.
|
||||
> **Resolution :** Valider les deux champs contre une liste blanche à la frontière d'import SREF et à la lecture dans `importSourceService`, avec erreur visible plutôt que repli.
|
||||
> *Ref : CWE-20*
|
||||
|
||||
`consolidated_schema.sql` reçoit les quatre colonnes, non pour alimenter les nouveaux profils — ils les reçoivent de la v17, puisque le script consolidé s'exécute après toutes les migrations et n'utilise que `CREATE TABLE IF NOT EXISTS` — mais pour rester la définition de référence testée. Une constante `V17_SQL` miroir est ajoutée côté tests, conformément au pattern `V13_SQL` à `V16_SQL`, avec un test de parité sur le modèle de `consolidated_schema_has_holdings_tables_and_kind_at_parity` (`lib.rs:2824`) : sans lui, les `DEFAULT` et les `CHECK` peuvent diverger entre les deux définitions.
|
||||
|
||||
> **🟢 SECURITE + TECHNIQUE** — Le rationale est inversé, vérification faite. `get_new_profile_init_sql` s'exécute **après** que tauri-plugin-sql a appliqué toutes les migrations, et le script consolidé utilise `CREATE TABLE IF NOT EXISTS` : les quatre colonnes qui y sont ajoutées sont inertes en production. Les nouveaux profils les reçoivent de la v17 seule. Le miroir sert au test de parité, pas au chemin de production.
|
||||
> **Resolution :** Corriger la formulation et exiger un test de parité sur le modèle de `consolidated_schema_has_holdings_tables_and_kind_at_parity` (`lib.rs:2824`), sans quoi le `DEFAULT` de `sign_convention` peut diverger entre les deux définitions.
|
||||
|
||||
Après la v17, `import_sources` et `import_config_templates` portent les mêmes huit champs de format. L'asymétrie qui a causé le bug racine disparaît structurellement.
|
||||
|
||||
`template_id` est une **étiquette de provenance** : elle enregistre le modèle depuis lequel la source a été configurée et n'est **jamais relue comme format**. Le format en vigueur est toujours celui des huit colonnes de la source. La colonne sert à l'affichage (« configurée depuis le modèle Desjardins ») et au signalement de divergence ; éditer un modèle ne modifie aucune source liée, ce qu'un critère d'acceptation vérifie.
|
||||
|
||||
> **🟡 ARCHITECTURE** — `template_id` réintroduit une seconde source de vérité que la migration venait d'éliminer. Les huit champs sont déjà **copiés** sur la source ; or `updateConfigTemplate` modifie un modèle en place (`useImportWizard.ts:965`), donc le format d'une source liée diverge silencieusement de son modèle, sans qu'aucune règle ne dise lequel fait foi.
|
||||
> **Resolution :** Soit retirer `template_id` (YAGNI — le format est déjà copié), soit énoncer dans le plan et l'ADR qu'il est une **étiquette de provenance**, jamais relue comme format, avec un critère d'acceptation prouvant qu'éditer un modèle ne modifie aucune source liée.
|
||||
|
||||
**Format SREF.** Le fichier d'export gagne deux tableaux, `import_sources` et `import_config_templates`. À l'import, les sources sont restaurées au lieu d'être détruites ; la source factice « Data Import » n'est créée que pour rattacher des transactions orphelines. Une sauvegarde produite avant ce changement ne porte aucun de ces tableaux : l'import doit alors conserver le comportement actuel plutôt qu'échouer.
|
||||
|
||||
### Architecture
|
||||
|
||||
Deux types et un codec. `ImportFormatRow` est la forme persistée — snake_case, mapping en JSON, `has_header` normalisé — partagée par `import_sources` et `import_config_templates`. `ImportFormat` est la forme de domaine — camelCase, mapping parsé — manipulée par le wizard et la détection. Une paire `formatToRow` / `formatFromRow` est le **seul point de conversion** entre les deux.
|
||||
|
||||
La garantie recherchée ne vient donc pas de la structure des types, qui ne peut pas être commune (les porteurs actuels mélangent les casses et typent `has_header` en `boolean` d'un côté, `number` de l'autre), mais du codec et de son test : un champ ajouté au format sans être traversé par le codec fait échouer le test de complétude. C'est ce qui ferme la classe d'erreur à l'origine du chantier.
|
||||
|
||||
> **🔴 ARCHITECTURE + TECHNIQUE** — La composition est structurellement impossible telle qu'écrite : les porteurs ont des formes incompatibles, vérifiées dans `types/index.ts` — `ImportSource.has_header: boolean` (`:12`) et `column_mapping: string` (`:10`), `ImportConfigTemplate.has_header: number` (`:164`), `SourceConfig.hasHeader: boolean` en camelCase avec `columnMapping: ColumnMapping` objet (`:225`), plus `AutoDetectResult` qui ne porte que 7 des 8 champs (pas d'`encoding`). Un type unique composé dans les quatre ne peut pas exister sans trancher une casse et une représentation du mapping.
|
||||
> **Resolution :** Deux types et un codec : `ImportFormatRow` (persisté, snake_case, mapping en JSON, `has_header` normalisé) et `ImportFormat` (domaine, camelCase, mapping parsé), reliés par une paire `formatToRow`/`formatFromRow` **unique point de conversion**. La garantie de complétude vient alors du codec et de son test, pas de la structure du type.
|
||||
|
||||
La détection gagne une couche lexicale en amont de l'heuristique existante. `normalizeHeaderCell` et `matchHeaderColumn` (`csvAutoDetect.ts:633-666`) sont déjà au niveau module et sont réutilisés tels quels ; c'est le **dictionnaire** qui est nouveau, et il vit dans son propre module plutôt que dans `csvAutoDetect.ts`, déjà long de 856 lignes pour deux flux sans rapport. La séparation n'est pas cosmétique : `montant` est un token d'**exclusion** pour les titres (`VALUE_HEADER_KEYWORDS`, `:630`) et le mot-clé de montant **principal** pour les transactions — deux tables distinctes, deux flux qui ne se marchent pas dessus.
|
||||
|
||||
> **🟡 ARCHITECTURE** — Le « remontage » est un no-op : les deux fonctions sont déjà au niveau module du même fichier que `autoDetectConfig` (`:633`, `:646`). La mitigation du tableau des risques couvre donc un risque inexistant, tandis que le vrai couplage n'est pas traité — les tables de mots-clés sont partagées, et `montant` est un **token d'exclusion** pour les holdings (`VALUE_HEADER_KEYWORDS`, `:630`) alors qu'il est le mot-clé de montant principal pour les transactions.
|
||||
> **Resolution :** Retirer la tâche de remontage ; placer le dictionnaire transactions dans son propre module à côté de `bankSignatures.ts`, important les deux helpers et laissant les tables holdings intactes. `csvAutoDetect.ts` fait déjà 856 lignes pour deux flux sans rapport. Le dictionnaire FR/EN couvre date, description (libellé, détail), montant, débit (retrait, déboursé), crédit (dépôt, encaissement) et solde. Quand les libellés tranchent, ils priment ; quand ils sont muets — fichier sans en-tête, libellés inconnus — l'heuristique de forme actuelle reprend la main. C'est ce qui résout d'un coup l'ordre débit/crédit deviné par position (`csvAutoDetect.ts:461-470`), la détection d'en-tête aveugle aux nombres et le choix de la colonne description par longueur moyenne.
|
||||
|
||||
La règle de mapping d'une ligne — date, montant, application du signe — est extraite de `parseFilesInternal` en une fonction pure exportée `mapRow(raw, format): ParsedRow`. Elle devient le point unique consommé par le wizard, par le calcul de score et par le récapitulatif d'aperçu. Sans cette extraction, le score réimplémenterait la règle et pourrait afficher 100 % pendant que l'import écrit de mauvais signes — la divergence même que ce chantier combat. Elle rend au passage la règle testable unitairement, le dépôt n'ayant pas de jsdom (d'où le pattern d'export des reducers de `useSnapshotEditor.ts`).
|
||||
|
||||
`autoDetectConfig` retourne désormais un score : la configuration détectée est rejouée via `mapRow` sur les lignes de l'échantillon et le taux de lignes parsées sans erreur est remonté à l'appelant.
|
||||
|
||||
> **🔴 ARCHITECTURE + TECHNIQUE** — Rejouer la configuration exige la règle de mapping de ligne (date + montant + signe), qui vit dans `parseFilesInternal`, un `useCallback` de `useImportWizard.ts:461-557`. `autoDetectConfig` est un util pur qui ne reçoit que `rawContent` : calculer le score dans le module de détection en écrirait une **seconde implémentation**. Deux mappers, c'est précisément la classe de divergence que ce chantier existe pour tuer — le score pourrait afficher 100 % pendant que l'import écrit de mauvais signes.
|
||||
> **Resolution :** Extraire en issue 3 une fonction pure exportée `mapRow(raw, format): ParsedRow` dans `src/utils/`, consommée par le hook, le calcul de score et le récapitulatif d'aperçu. Elle rend au passage possibles les tests unitaires promis par l'issue 3 — le dépôt n'a pas de jsdom, d'où le pattern d'export des reducers et builders de `useSnapshotEditor.ts`.
|
||||
|
||||
Les signatures de banques sont un tableau de règles déclaratives — ensemble de libellés d'en-tête normalisés, délimiteur, particularités de préambule. Elles sont évaluées avant le dictionnaire générique et n'ajoutent aucune surface d'interface.
|
||||
|
||||
Le point d'écriture de la configuration en base migre de `checkDuplicatesInternal` (`useImportWizard.ts:588-606`) vers `executeImport`, pour qu'un import abandonné ne laisse plus de configuration derrière lui.
|
||||
|
||||
## Plan de travail
|
||||
|
||||
### Issue 1 — Migration v17 : le format complet sur les sources [type:schema]
|
||||
Dependances : Issue 4
|
||||
- [ ] Migration v17 dans `lib.rs` : quatre colonnes + backfill `amount_mode`
|
||||
- [ ] Miroir dans `consolidated_schema.sql`
|
||||
- [ ] Constante `V17_SQL` + test d'application sur une base v16
|
||||
- [ ] Test : le backfill reproduit la règle actuelle (source avec `debitAmount` → `debit_credit`, sans → `single`)
|
||||
- [ ] Test de parité entre `consolidated_schema.sql` et la chaîne v1→v17 (colonnes, `DEFAULT`, `CHECK`)
|
||||
- [ ] Test de non-régression : les chaînes SQL `v1` à `v16` absentes du diff
|
||||
|
||||
> **🟡 TECHNIQUE** — « Checksums intacts » n'est pas une assertion testable ici : aucun harnais de checksum n'existe. Les constantes `V10_SQL` à `V16_SQL` sont des copies manuelles appliquées via `execute_batch`, et les checksums ne vivent qu'au runtime dans `_sqlx_migrations` (réparés par `profile_commands.rs:223-275`). Un worker autonome va soit bâcler la case, soit y brûler un cycle.
|
||||
> **Resolution :** Remplacer par ce que le harnais vérifie réellement — les chaînes SQL v1 à v16 absentes du diff, plus l'application de `V17_SQL` sur une base v16 peuplée — et retirer le mot « checksums ». Ajouter le test de parité du schéma consolidé.
|
||||
|
||||
### Issue 2 — Persister et restaurer le format d'import [type:bug]
|
||||
Dependances : Issue 1
|
||||
- [ ] Type `ImportFormat` partagé ; `ImportSource` et `ImportConfigTemplate` le composent
|
||||
- [ ] `importSourceService` : les quatre nouvelles colonnes en création, mise à jour et lecture
|
||||
- [ ] `useImportWizard.selectSource` : lire `amount_mode` et `sign_convention` ; supprimer la valeur en dur (`:323`) et la ré-inférence du mode (`:321`)
|
||||
- [ ] `ColumnMappingEditor.onAmountModeChange` nettoie les colonnes du mode abandonné
|
||||
- [ ] Déplacer l'écriture de la config de `checkDuplicatesInternal` vers `executeImport`
|
||||
- [ ] Lier la source au modèle appliqué (`template_id`), persisté et restauré
|
||||
- [ ] Test de cycle : configurer → sauvegarder → recharger → le format est identique au bit près
|
||||
|
||||
### Issue 3 — Règle débit/crédit et parsing des montants [type:bug]
|
||||
Dependances : Issue 2
|
||||
- [ ] `amount = crédit − débit` sur magnitudes, en remplacement de la comparaison de nullité (`:509`)
|
||||
- [ ] Colonne inutilisée à `0,00` traitée comme absente
|
||||
- [ ] Supprimer les fallbacks `?? 0` (`:504-512`) au profit d'une erreur de ligne « colonne de montant non mappée »
|
||||
- [ ] `parseFrenchAmount` : parenthèses comptables `(50,00)` et signe suffixe `50,00-`
|
||||
- [ ] Séparateur décimal arbitré au niveau de la colonne et non de la cellule
|
||||
- [ ] **Validation ancrée** : `parseFrenchAmount` rend `NaN` sur tout caractère résiduel après normalisation — aujourd'hui `"50,00-"` rend 5000, `"1 234,56 CR"` rend 123456 et `"100,00 CAD"` rend 10000, erreurs de facteur 100 qui passent `isNaN`
|
||||
- [ ] Durcissement appliqué **partout** (décision tranchée) : vérifier les 11 sites d'appel — 8 dans `csvAutoDetect.ts` (`:224`, `:289`, `:323`, `:489`, `:517`, `:543-544`, `:712`) et 3 dans `useSnapshotEditor.ts:191-202` (import CSV de titres, #245)
|
||||
- [ ] Extraire `mapRow(raw, format): ParsedRow` pur et exporté depuis `parseFilesInternal`, consommé par le wizard, le score et l'aperçu
|
||||
- [ ] Tests unitaires sur chaque cas, dans le nouveau `src/utils/amountParser.test.ts`
|
||||
- [ ] Régression : les tests holdings existants restent verts
|
||||
|
||||
> **🔴 SECURITE** — Le défaut est plus grave que décrit, vérifié en exécutant la fonction : `parseFrenchAmount` termine sur `parseFloat`, qui s'arrête au premier caractère invalide au lieu de rejeter. `"50,00-"` rend **5000**, `"1 234,56 CR"` rend **123456**, `"100,00 CAD"` rend **10000** — une erreur de facteur 100, pas un signe perdu. Chacune passe `isNaN` et sera donc comptée comme ligne **valide** dans le nouveau récapitulatif signé, ce qui neutralise le filet de sécurité principal du chantier. Ajouter la forme `50,00-` traite un suffixe et laisse passer tous les autres.
|
||||
> **Resolution :** Valider la chaîne entière par une regex ancrée après normalisation et rendre `NaN` sur tout caractère résiduel. Ajouter ces trois chaînes aux tests unitaires.
|
||||
> *Ref : CWE-1284*
|
||||
|
||||
> **🔴 TECHNIQUE** — `parseFrenchAmount` a 11 sites d'appel que l'issue ne liste pas : 8 dans `csvAutoDetect.ts` (`:224` `detectHeader`, `:289`, `:323`, `:489`, `:517` `detectSingleAmount`, `:543-544` `isSparseComplementary`, `:712` holdings) et 3 dans `useSnapshotEditor.ts:191-202` — l'import CSV de titres (#245). `useSnapshotEditor.ts` est absent du tableau « Fichiers concernés ». Changer le parser déplace silencieusement `hasNumber`, `negCount` et les quantités de titres.
|
||||
> **Resolution :** Ajouter `csvAutoDetect.ts` et `useSnapshotEditor.ts` au périmètre de l'issue, avec une case de régression sur les tests holdings, et trancher explicitement si les nouvelles formes sont globales ou opt-in.
|
||||
|
||||
### Issue 4 — Corpus de fixtures CSV [type:feature]
|
||||
Dependances : aucune — **premier maillon de la pile**
|
||||
|
||||
> **🔴 TECHNIQUE + ARCHITECTURE** — Le corpus censé « figer le comportement avant la refonte » arrive **après** le changement qu'il doit servir de référence. L'issue 3 modifie `parseFrenchAmount`, dont dépendent `detectHeader`, `detectSingleAmount`, `pickBestAmountColumn` et `isSparseComplementary` : la ligne de base enregistrerait donc un comportement déjà modifié.
|
||||
> **Resolution :** Déplacer cette issue en première position — elle ne dépend de rien, ni migration ni persistance. Nouvel ordre : 4 → 1 → 2 → 3 → 5 → … Les issues 3, 5 et 6 se mesurent alors toutes à un contrat préexistant. Étendre les assertions aux sites d'appel holdings.
|
||||
- [ ] Fixtures synthétiques : montant signé, débit/crédit, débit/crédit inversés, colonne à `0,00`, préambule, en-tête contenant un nombre, sans en-tête, tout-positif, ligne entière entre guillemets
|
||||
- [ ] Tests de contrat figeant le comportement de `autoDetectConfig` avant sa refonte
|
||||
- [ ] Tests de bout en bout du parsing sur chaque fixture
|
||||
|
||||
### Issue 5 — Détection par libellé d'en-tête [type:feature]
|
||||
Dependances : Issue 3
|
||||
- [ ] Remonter `normalizeHeaderCell` et `matchHeaderColumn` en helpers partagés du module
|
||||
- [ ] Dictionnaire FR/EN : date, description, montant, débit, crédit, solde
|
||||
- [ ] Ordre débit/crédit résolu par libellé plutôt que par position
|
||||
- [ ] `detectHeader` : signal lexical en complément de la forme
|
||||
- [ ] Repli sur l'heuristique actuelle quand les libellés sont muets
|
||||
- [ ] Dictionnaire dans son propre module, hors de `csvAutoDetect.ts` — les tables holdings restent intactes (`montant` y est un token d'exclusion)
|
||||
- [ ] **Refus du troisième format** : détecter montants tous positifs + colonne voisine à une seule lettre `D`/`C`, et refuser explicitement avec un message dédié plutôt que d'importer chaque débit comme un revenu
|
||||
- [ ] Tests sur les fixtures, dont l'inversion crédit-avant-débit et le fichier à indicateur refusé
|
||||
|
||||
> **🟡 SECURITE** — Le document de décisions promet qu'un fichier au format « montant absolu + indicateur `D`/`C` » sera « refusé explicitement plutôt qu'importé de travers », mais aucune issue du plan ne le détecte ni ne le refuse. En l'état, un tel fichier présente une colonne de montants tous positifs, `detectSingleAmount` rend `positive_expense`, et chaque débit est importé comme un revenu.
|
||||
> **Resolution :** Ajouter ici une règle de détection (colonne de montants tous positifs plus une colonne voisine à une seule lettre `D`/`C`) et un message de refus explicite, avec un critère d'acceptation et une fixture dans l'issue de corpus.
|
||||
|
||||
### Issue 6 — Score de confiance et détection lancée d'office [type:feature]
|
||||
Dependances : Issue 5
|
||||
- [ ] `autoDetectConfig` rejoue sa configuration sur l'échantillon et retourne un taux
|
||||
- [ ] Détection déclenchée seule à l'ouverture d'une source non configurée
|
||||
- [ ] Bandeau de résultat dans `SourceConfigPanel` : reconnu, score, ou avertissement
|
||||
- [ ] Convention de signe masquée en mode débit/crédit
|
||||
- [ ] Clés i18n FR et EN
|
||||
|
||||
### Issue 7 — Aperçu obligatoire et récapitulatif signé [type:feature]
|
||||
Dependances : Issue 6
|
||||
- [ ] Rendre l'étape `file-preview` dans `ImportPage`, traversée à chaque import
|
||||
- [ ] Récapitulatif : sorties et total, entrées et total, lignes en erreur
|
||||
- [ ] Bouton *Inverser les signes* agissant sur la configuration, avec re-parsing
|
||||
- [ ] `ImportConfirmation` affiche mode, convention et mapping
|
||||
- [ ] `useImportWizard` : nouvelle transition vers `file-preview` dans le reducer, recâblage de `checkDuplicates` (aujourd'hui code mort) et de `parseAndCheckDuplicates` qui saute l'étape
|
||||
- [ ] `ImportPage` : remplacer la paire de boutons Aperçu / Vérifier-doublons par `WizardNavigation`
|
||||
- [ ] Retirer `FilePreviewModal`
|
||||
- [ ] En mode débit/crédit, *Inverser les signes* permute `debitAmount` et `creditAmount`
|
||||
- [ ] Clés i18n FR et EN
|
||||
|
||||
> **🟡 TECHNIQUE** — L'étape est un changement de machine à états, pas un rendu, et le périmètre annoncé est trop étroit. `file-preview` n'a **aucune transition** dans le reducer (`SET_STEP` ne la vise jamais), `parseAndCheckDuplicates` saute de `source-config` à `duplicate-check` (`:719`, commentaire « skips preview step »), `ImportPage` porte encore la paire de boutons Aperçu / Vérifier-doublons (`:126-140`) et le `FilePreviewModal` (`:196-202`) ; enfin `checkDuplicates` (`:685`, exporté `:1006`) est du code mort à recâbler. Modifier `FilePreviewTable`, seul fichier listé, mute aussi le modal encore vivant.
|
||||
> **Resolution :** Étendre le périmètre à `useImportWizard` (nouvelle transition + recâblage de `checkDuplicates`), `ImportPage` (remplacer la paire de boutons par `WizardNavigation`) et `FilePreviewModal` (le retirer).
|
||||
|
||||
### Issue 8 — Signatures de banques et dérive de format [type:feature]
|
||||
Dependances : Issue 7
|
||||
- [ ] Signatures déclaratives Desjardins, RBC, BNC, Tangerine, évaluées avant le dictionnaire générique
|
||||
- [ ] `header_signature` mémorisée à l'import réussi
|
||||
- [ ] Au ré-import : comparaison, re-détection, panneau d'écart avec adopter ou conserver
|
||||
- [ ] Clés i18n FR et EN
|
||||
- [ ] Le panneau de dérive et l'aperçu énoncent le seul chemin de réparation sûr : supprimer l'import fautif via l'historique avant de le rejouer — un ré-import corrigé ne s'apparie pas aux lignes déjà écrites et les double
|
||||
- [ ] Tests sur fixtures par banque et sur un cas de dérive
|
||||
|
||||
### Issue 9 — Sources et modèles dans l'export de données [type:bug]
|
||||
Dependances : Issue 8
|
||||
- [ ] Sérialiser `import_sources` et `import_config_templates` à l'export
|
||||
- [ ] Restaurer les sources à l'import au lieu du `DELETE` suivi d'une source factice (`:265`, `:362`)
|
||||
- [ ] **Envelopper purge et restauration dans `withTransaction`** — les deux fonctions n'en ont aucune aujourd'hui, une violation de contrainte à mi-course détruit l'historique sans retour arrière
|
||||
- [ ] Ordre de restauration : modèles avant sources ; stratégie d'identifiants explicite (upsert par nom + remap de `template_id`), `import_config_templates` n'étant dans aucune liste de purge
|
||||
- [ ] Liste blanche sur `amount_mode` et `sign_convention` à la frontière d'import, avec erreur lisible
|
||||
- [ ] Rétrocompatibilité : une sauvegarde antérieure sans ces tableaux s'importe comme aujourd'hui
|
||||
- [ ] Tests de cycle export/import préservant les configurations
|
||||
- [ ] Test : une restauration qui échoue à la ligne N laisse le profil intact
|
||||
|
||||
> **🔴 SECURITE** — La restauration SREF n'est enveloppée dans **aucune transaction** : `dataExportService.ts` ne contient pas une seule occurrence de `withTransaction` (vérifié), et enchaîne `DELETE FROM transactions / imported_files / import_sources / keywords / suppliers / categories` (`:263-268`, `:360-362`) puis des `db.execute` d'insertion. Cette issue ajoute deux boucles d'insertion de plus, sur des tables à contrainte `UNIQUE(name)` et une clé étrangère `template_id` : la moindre violation abandonne la restauration à mi-course et l'historique financier de l'utilisateur est perdu sans retour arrière.
|
||||
> **Resolution :** Envelopper l'ensemble purge + restauration des deux fonctions dans `withTransaction`, et ajouter un critère d'acceptation : une restauration qui échoue à la ligne N laisse le profil intact.
|
||||
> *Ref : CWE-460*
|
||||
|
||||
> **🔴 SECURITE** — `import_config_templates` n'apparaît dans aucune des deux listes de purge (vérifié) : réinsérer des modèles restaurés dans un profil qui en possède déjà échoue sur `UNIQUE constraint failed: import_config_templates.name`. Ce n'est pas un cas théorique — restaurer dans un profil existant est le chemin normal. L'ordre de restauration n'est pas non plus spécifié alors que `template_id` porte désormais une clé étrangère vers cette table.
|
||||
> **Resolution :** Spécifier l'ordre (modèles avant sources) et la stratégie d'identifiants : soit purger aussi `import_config_templates`, soit faire un upsert par nom et remapper `import_sources.template_id` vers les identifiants résolus.
|
||||
|
||||
### Issue 10 — Documentation [type:feature]
|
||||
Dependances : Issue 9
|
||||
- [ ] `docs/architecture.md` : migration v17, quatre colonnes, couche de détection lexicale, nouvelle étape du wizard
|
||||
- [ ] ADR 0019 — le format d'import est une donnée persistée intégralement, jamais re-devinée
|
||||
|
||||
> **🟢 ARCHITECTURE** — `.gitignore:71-72` ignore `spec-decisions-*.md` et `spec-plan-*.md` : l'ADR 0016 a livré une ligne `Spec:` morte pour cette raison exacte, corrigée par un force-add (PR #295).
|
||||
> **Resolution :** Ajouter `git add -f spec-decisions-import-csv-format.md spec-plan-import-csv-format.md` à la checklist, avant d'écrire la ligne `Spec:`.
|
||||
- [ ] `docs/guide-utilisateur.md` + clés `docs.*` FR et EN
|
||||
- [ ] `CHANGELOG.md` et `CHANGELOG.fr.md` sous `[Unreleased]`
|
||||
|
||||
### Ordre d'execution
|
||||
|
||||
```
|
||||
4 → 1 → 2 → 3 → 5 → 6 → 7 → 8 → 9 → 10
|
||||
```
|
||||
|
||||
Pile strictement linéaire. Chaque maillon touche `useImportWizard.ts`, `csvAutoDetect.ts` ou les deux ; une exécution en vagues parallèles produirait des conflits sur ces deux fichiers à chaque étage. Le CHANGELOG est centralisé dans l'issue 10 pour la même raison.
|
||||
|
||||
> **🔴 TECHNIQUE + ARCHITECTURE** — L'ordre place le corpus de contrat (issue 4) après le changement de parser qu'il doit servir de référence (issue 3). Ordre corrigé :
|
||||
> ```
|
||||
> 4 → 1 → 2 → 3 → 5 → 6 → 7 → 8 → 9 → 10
|
||||
> ```
|
||||
> **Resolution :** L'issue 4 ne dépend de rien et passe en tête ; la chaîne reste linéaire. Les issues Forgejo #323-#332 doivent être renumérotées en conséquence dans leurs lignes `Depends on`.
|
||||
|
||||
## Fichiers concernes
|
||||
|
||||
| Fichier | Action | Raison |
|
||||
|---------|--------|--------|
|
||||
| `src-tauri/src/lib.rs` | Modifier | Migration v17 + constante `V17_SQL` + tests |
|
||||
| `src-tauri/src/database/consolidated_schema.sql` | Modifier | Quatre colonnes sur `import_sources` pour les nouveaux profils |
|
||||
| `src/shared/types/index.ts` | Modifier | Type `ImportFormat`, composition dans `ImportSource` et `ImportConfigTemplate` |
|
||||
| `src/services/importSourceService.ts` | Modifier | Persistance des quatre colonnes en création, mise à jour, lecture |
|
||||
| `src/services/importConfigTemplateService.ts` | Modifier | Alignement sur `ImportFormat` |
|
||||
| `src/hooks/useImportWizard.ts` | Modifier | Restauration réelle, règle débit/crédit, point d'écriture, score, étape aperçu |
|
||||
| `src/utils/csvAutoDetect.ts` | Modifier | Couche lexicale, helpers partagés, score, signatures de banques |
|
||||
| `src/utils/amountParser.ts` | Modifier | Parenthèses comptables, signe suffixe, arbitrage par colonne |
|
||||
| `src/hooks/useSnapshotEditor.ts` | Modifier | **Manquant** — 3 appels à `parseFrenchAmount` (`:191-202`), import CSV de titres (#245) |
|
||||
| `src/utils/amountParser.test.ts` | Créer | **Manquant** — n'existe pas aujourd'hui, requis par l'issue de parsing |
|
||||
| `src/components/import/ColumnMappingEditor.tsx` | Modifier | Le mode nettoie le mapping opposé |
|
||||
| `src/components/import/SourceConfigPanel.tsx` | Modifier | Bandeau de détection, convention masquée en débit/crédit |
|
||||
| `src/components/import/FilePreviewTable.tsx` | Modifier | Récapitulatif signé et bascule de convention |
|
||||
| `src/components/import/ImportConfirmation.tsx` | Modifier | Mode, convention et mapping au récapitulatif |
|
||||
| `src/components/import/FormatDriftPanel.tsx` | Créer | Panneau d'écart au ré-import |
|
||||
| `src/pages/ImportPage.tsx` | Modifier | Étape `file-preview` rendue, panneau de dérive |
|
||||
| `src/services/dataExportService.ts` | Modifier | Sources et modèles sérialisés et restaurés |
|
||||
| `src/utils/bankSignatures.ts` | Créer | Signatures déclaratives par banque |
|
||||
| `src/utils/csvAutoDetect.test.ts` | Modifier | Tests du flux transactions, aujourd'hui absents |
|
||||
| `src/__fixtures__/csv/` | Créer | Corpus synthétique |
|
||||
|
||||
> **🟡 TECHNIQUE + ARCHITECTURE** — Le chemin `src/test/fixtures/csv/` inventait une troisième convention : `src/test/` n'existe pas (vérifié). Les tests sont colocalisés (`src/utils/*.test.ts`, `src/services/*.test.ts`), les fixtures vivent dans `src/__fixtures__/` et les tests d'intégration dans `src/__integration__/`. Corrigé dans le tableau ci-dessus.
|
||||
> **Resolution :** Corpus dans `src/__fixtures__/csv/`, tests de contrat dans `src/utils/csvAutoDetect.test.ts` et le nouveau `src/utils/amountParser.test.ts`.
|
||||
| `src/i18n/locales/{fr,en}.json` | Modifier | Clés de détection, aperçu, dérive, aide |
|
||||
| `docs/architecture.md` | Modifier | v17, détection lexicale, étape du wizard |
|
||||
| `docs/adr/0019-format-import-persiste.md` | Créer | Décision structurante |
|
||||
| `docs/guide-utilisateur.md` | Modifier | Nouveau parcours d'import |
|
||||
| `CHANGELOG.md`, `CHANGELOG.fr.md` | Modifier | Entrées sous `[Unreleased]` |
|
||||
|
||||
## Plan de tests
|
||||
|
||||
### Tests unitaires
|
||||
`parseFrenchAmount` sur chaque forme de montant, dont parenthèses et signe suffixe. `parseDate` inchangé, couvert en régression. La couche lexicale : chaque libellé du dictionnaire, l'ordre débit/crédit inversé, les libellés muets qui déclenchent le repli. `autoDetectConfig` sur chaque fixture, contrat complet — délimiteur, en-tête, lignes ignorées, format de date, mode, convention, mapping, score. Les signatures de banques, une par banque plus un fichier inconnu qui doit retomber sur le générique.
|
||||
|
||||
### Tests d'integration
|
||||
Le cycle de format configurer → sauvegarder → recharger, qui est le test qui aurait attrapé le bug racine. Le cycle export → import de données préservant sources et modèles, plus l'import d'une sauvegarde antérieure sans ces tableaux. L'application de la migration v17 sur une base v16 peuplée, avec vérification du backfill. Le parsing de bout en bout sur chaque fixture, du fichier brut aux montants signés.
|
||||
|
||||
### Tests de regression
|
||||
Les tests existants restent verts — 871 vitest et 106 Rust au dernier relevé. Le corpus de fixtures de l'issue 4 fige le comportement de `autoDetectConfig` avant sa refonte, de sorte que les issues 5 et 6 se mesurent à un contrat établi plutôt qu'à une intention. Les migrations v1 à v16 conservent leurs checksums.
|
||||
|
||||
## Criteres d'acceptation
|
||||
|
||||
- [ ] Une source réglée en montants positifs conserve sa convention au deuxième import, et à tous les suivants
|
||||
- [ ] Un fichier `Date;Description;Crédit;Débit` est mappé dans le bon ordre sans intervention
|
||||
- [ ] Un fichier dont la colonne inutilisée porte `0,00` produit les bons montants, aucun n'est nul
|
||||
- [ ] Un mapping incomplet produit une erreur de ligne explicite, jamais une lecture de la colonne 0
|
||||
- [ ] Basculer de mode puis recharger la source restitue le mode choisi
|
||||
- [ ] Une source jamais configurée déclenche la détection sans action de l'utilisateur
|
||||
- [ ] L'aperçu est traversé à chaque import et affiche sorties, entrées et totaux
|
||||
- [ ] L'écran de confirmation affiche mode, convention et mapping
|
||||
- [ ] Un import abandonné ne laisse aucune configuration en base
|
||||
- [ ] Un fichier Desjardins est reconnu par signature et annoncé comme tel
|
||||
- [ ] Un en-tête modifié depuis le dernier import déclenche le panneau d'écart
|
||||
- [ ] Exporter puis réimporter ses données préserve toutes les configurations de sources
|
||||
- [ ] Une sauvegarde produite avant ce chantier s'importe sans erreur
|
||||
- [ ] Les migrations v1 à v16 sont inchangées ; la v17 s'applique sur une base v16 peuplée
|
||||
|
||||
## Edge cases et risques
|
||||
|
||||
| Cas | Mitigation |
|
||||
|-----|------------|
|
||||
| Le backfill v17 ne peut pas deviner la convention passée d'une source existante | Aucune source ne change de comportement — le défaut restitue exactement ce que le code appliquait déjà. Une source qui souffrait du bug continue de souffrir jusqu'au prochain import, où le score de confiance et l'aperçu obligatoire exposent l'écart. C'est une amélioration franche sans mutation silencieuse, cohérente avec la décision d'exclure toute correction rétroactive. |
|
||||
| Un ré-import corrigé double-compte les lignes déjà importées de travers | `findDuplicates` (`transactionService.ts:164`) apparie sur `date AND description AND amount`. Les issues 2 et 3 changent les montants produits par une source — signe inversé, débits qui ne valent plus 0 : le ré-import que la ligne précédente invite à faire **ne reconnaîtra pas** les lignes fautives et les ajoutera en double, une inversion de signe produisant en prime des paires miroir qui se compensent à ~0 au lieu de sauter aux yeux. Le seul chemin de réparation sûr est `deleteImportWithTransactions` sur l'import fautif **avant** de le rejouer — à énoncer dans l'interface de dérive et d'aperçu, pas seulement ici. |
|
||||
| Un fichier sans ligne d'en-tête ne bénéficie d'aucun signal lexical | Repli explicite sur l'heuristique de forme actuelle, qui reste testée par le corpus. Le score de confiance sera mécaniquement plus bas, ce qui est l'information juste. |
|
||||
| Un fichier sans en-tête n'a pas de signature à mémoriser | `header_signature` reste nulle et la détection de dérive est inopérante sur ces sources. Documenté plutôt que contourné : inventer une signature sur les données produirait de faux positifs à chaque changement de contenu. |
|
||||
| `json_extract` indisponible dans le SQLite embarqué | Backfill écrit en `LIKE '%debitAmount%'`, sans dépendance à l'extension JSON1. |
|
||||
| L'aperçu obligatoire ajoute un clic à chaque import mensuel | Coût accepté par décision explicite, contre la recommandation d'un aperçu conditionné à la confiance. |
|
||||
| Les signatures de banques sont écrites sans relevés réels | Elles reposent sur les formats documentés et sur `preprocessQuotedCSV`, qui prouve déjà le cas Desjardins. Un fichier non reconnu retombe sur le dictionnaire générique — l'échec d'une signature dégrade, il ne casse pas. |
|
||||
| Modifier `csvAutoDetect.ts` risque de régresser le flux holdings (#245) | Les helpers sont remontés sans changer leur comportement ; les tests holdings existants font foi et doivent rester verts à chaque maillon. |
|
||||
| Une pile de dix issues sur deux fichiers centraux dérive au rebase | Ordre strictement linéaire, CHANGELOG centralisé dans le dernier maillon, chaque PR basée sur la précédente. |
|
||||
|
||||
## Revision — Synthese
|
||||
|
||||
> Date: 2026-08-12 | Experts: Securite, Architecture, Technique
|
||||
|
||||
### Verdict
|
||||
|
||||
🟡 **CRITIQUES ADRESSEES — plan corrige le 2026-08-13** — A la revue, deux fondations du plan etaient fausses telles qu'ecrites (le type `ImportFormat` compose, l'ordre des issues) et trois defauts du code se sont reveles plus graves que ce que le plan enoncait. Les 7 critiques et les 9 ameliorations sont desormais integrees au corps du document ; les annotations restent en trace de revue. Decisions tranchees : 4 (voir ci-dessous).
|
||||
|
||||
### Resume
|
||||
|
||||
| Expert | 🔴 | 🟡 | 🟢 | Points cles |
|
||||
|--------|-----|-----|-----|-------------|
|
||||
| Securite | 3 | 4 | 1 | Restauration SREF hors transaction ; `parseFloat` accepte les suffixes et rend une magnitude ×100 ; modeles absents de la liste de purge |
|
||||
| Architecture | 3 | 3 | 2 | `ImportFormat` compose est structurellement impossible ; le score dupliquerait la regle de parsing ; `template_id` recree une seconde source de verite |
|
||||
| Technique | 4 | 3 | 1 | Le corpus de contrat arrive apres le changement qu'il fige ; 11 sites d'appel non listes ; l'etape apercu est un changement de machine a etats |
|
||||
|
||||
Trois constats ont ete verifies a la main avant annotation, et un constat d'agent a ete durci plutot que repris : `parseFrenchAmount("50,00-")` rend **5000**, pas 50 — la revue d'implementation initiale sous-estimait ce defaut.
|
||||
|
||||
### Actions requises
|
||||
|
||||
1. 🔴 **`ImportFormat` compose** — remplacer par `ImportFormatRow` + `ImportFormat` relies par un codec unique ; les quatre porteurs ont des casses et des types incompatibles.
|
||||
2. 🔴 **Ordre des issues** — le corpus de contrat passe en tete : `4 → 1 → 2 → 3 → 5 → …`.
|
||||
3. 🔴 **`parseFrenchAmount`** — validation ancree rendant `NaN` sur tout residu ; les suffixes produisent aujourd'hui une erreur de facteur 100 qui passe `isNaN`.
|
||||
4. 🔴 **Sites d'appel du parser** — ajouter `csvAutoDetect.ts` et `useSnapshotEditor.ts` au perimetre ; 11 appels non listes, dont le flux holdings.
|
||||
5. 🔴 **Score de confiance** — extraire `mapRow(raw, format)` pur en amont, sinon le score reimplemente la regle de parsing.
|
||||
6. 🔴 **Restauration SREF** — envelopper purge et restauration dans `withTransaction` ; aujourd'hui aucune.
|
||||
7. 🔴 **Modeles a la restauration** — ordre et strategie d'identifiants a specifier ; `import_config_templates` n'est dans aucune liste de purge.
|
||||
8. 🟡 Whitelist des valeurs `amount_mode` / `sign_convention` a la frontiere SREF.
|
||||
9. 🟡 `template_id` — le retirer ou le declarer etiquette de provenance.
|
||||
10. 🟡 Bouton d'inversion inerte en mode debit/credit.
|
||||
11. 🟡 Le refus du troisieme mode de montant est promis sans livrable.
|
||||
12. 🟡 Un re-import corrige double-compte les lignes fautives (`findDuplicates` apparie sur le montant).
|
||||
13. 🟡 Perimetre de l'etape apercu — machine a etats, pas rendu.
|
||||
14. 🟡 Le remontage des helpers est un no-op ; le vrai couplage est le dictionnaire partage.
|
||||
15. 🟡 « Checksums intacts » n'est pas testable ; chemin des fixtures corrige en `src/__fixtures__/csv/`.
|
||||
|
||||
|
||||
### Decisions tranchees (2026-08-13)
|
||||
|
||||
| Decision | Retenu | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| Contrainte `CHECK` sur `amount_mode` | `CHECK` elargi a `absolute_indicator` | Revise la decision de cadrage « pas de CHECK ». L'objectif d'origine — ajouter le 3e mode sans migration — est atteint par la valeur future deja admise dans la contrainte, sans renoncer a la garantie en base. `sign_convention` recoit le meme traitement. |
|
||||
| Role de `template_id` | Etiquette de provenance | La colonne enregistre d'ou vient la configuration et n'est jamais relue comme format ; les huit colonnes de la source font foi. Un critere d'acceptation verifie qu'editer un modele ne modifie aucune source liee. |
|
||||
| Troisieme format de montant | Detecte et refuse explicitement | Le document de decisions promettait un refus explicite sans qu'aucune issue ne le livre. Sans la regle, un tel fichier importe chaque debit comme un revenu — l'echec silencieux que le chantier combat. |
|
||||
| Portee du durcissement de `parseFrenchAmount` | Global, tous les sites d'appel | Un `"100,00 CAD"` qui rend 10000 est un bug partout, y compris a l'import de titres. Une seule fonction a raisonner, au prix d'une passe de regression sur les tests holdings existants. |
|
||||
|
||||
### Corrections integrees sans arbitrage
|
||||
|
||||
`ImportFormatRow` + `ImportFormat` relies par un codec unique (la composition d'un type unique etait structurellement impossible) ; ordre des issues `4 → 1 → 2 → 3 → 5 → …` ; extraction de `mapRow` avant le calcul de score ; `withTransaction` sur la purge et la restauration SREF ; ordre et strategie d'identifiants a la restauration des modeles ; perimetre de l'etape apercu etendu au reducer, a `ImportPage` et au modal ; dictionnaire lexical dans son propre module ; `useSnapshotEditor.ts` et `src/utils/amountParser.test.ts` ajoutes au tableau des fichiers ; chemin des fixtures corrige en `src/__fixtures__/csv/` ; assertion « checksums » remplacee par ce que le harnais verifie reellement ; chemin de reparation sur re-import enonce dans l'interface.
|
||||
|
||||
|
||||
### Decisions de planification (2026-08-13, seance /plan-run)
|
||||
|
||||
| Question | Decision | Portee |
|
||||
|----------|----------|--------|
|
||||
| Les specs sont gitignorees, les workers ne les verraient pas | Force-add et commit des deux fichiers (precedent PR #295) **et** bodies d'issues auto-suffisants | Toutes les issues ; debloque la ligne `Spec:` de l'ADR 0019 |
|
||||
| Seuil de confiance non chiffre | **90 %** de lignes lues. En dessous : bandeau d'avertissement + score detaille (« 132/150 lignes »). Au-dessus : bandeau neutre. L'apercu obligatoire reste le filet reel quel que soit le score | Issues 6 et 7 (#328, #329) |
|
||||
| Contenu de `header_signature` | Tableau JSON des libelles normalises par `normalizeHeaderCell`, ex. `["date","description","montant","solde"]`. Un hash rendrait impossible l'ecart colonne par colonne promis par le panneau de derive | Issue 8 (#330) |
|
||||
| Compatibilite du format SREF | Champ de version explicite dans le fichier exporte. A l'import, son absence signifie « format anterieur » et les tableaux manquants sont traites comme vides | Issue 9 (#331) |
|
||||
| Emplacement du codec et de `mapRow` | `src/utils/importFormat.ts` — meme dossier que `amountParser`, `dateParser` et `csvAutoDetect`, qui portent deja la logique pure du domaine. Les types restent dans `src/shared/types/` | Issues 2 et 3 (#324, #325) |
|
||||
|
|
@ -15,6 +15,22 @@ CREATE TABLE IF NOT EXISTS import_sources (
|
|||
column_mapping TEXT NOT NULL,
|
||||
skip_lines INTEGER NOT NULL DEFAULT 0,
|
||||
has_header INTEGER NOT NULL DEFAULT 1,
|
||||
-- Full import format (migration v17). These four columns are INERT on the
|
||||
-- production path: this script runs AFTER tauri-plugin-sql has applied every
|
||||
-- migration and only uses CREATE TABLE IF NOT EXISTS, so a brand-new profile
|
||||
-- actually receives them from v17. They are mirrored here to keep this file
|
||||
-- the tested reference definition -- a parity test compares it against the
|
||||
-- v1->v17 chain so the DEFAULT and CHECK of the two cannot drift apart.
|
||||
-- `amount_mode` admits 'absolute_indicator' from the start so the third mode
|
||||
-- can ship without another migration. `template_id` is a provenance tag
|
||||
-- only, never re-read as format: the eight format columns above are
|
||||
-- authoritative, so editing a template changes no linked source.
|
||||
amount_mode TEXT NOT NULL DEFAULT 'single'
|
||||
CHECK (amount_mode IN ('single','debit_credit','absolute_indicator')),
|
||||
sign_convention TEXT NOT NULL DEFAULT 'negative_expense'
|
||||
CHECK (sign_convention IN ('negative_expense','positive_expense')),
|
||||
header_signature TEXT,
|
||||
template_id INTEGER REFERENCES import_config_templates(id) ON DELETE SET NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
|
|
|||
|
|
@ -313,6 +313,58 @@ pub fn run() {
|
|||
DROP TABLE _v16_guard;",
|
||||
kind: MigrationKind::Up,
|
||||
},
|
||||
// Migration v17 — the full import format on the sources themselves (#323).
|
||||
//
|
||||
// Until now `import_sources` carried only the mechanical CSV settings
|
||||
// (delimiter, encoding, date format, column mapping). The two fields
|
||||
// that decide how an amount is READ — `amount_mode` and
|
||||
// `sign_convention` — existed only on `import_config_templates`. That
|
||||
// asymmetry is the root bug of this chantier: restoring a saved source
|
||||
// re-inferred the mode from the mapping and hardcoded
|
||||
// `signConvention: "negative_expense"` (`useImportWizard.ts:321-323`),
|
||||
// so a source configured with positive expenses silently flipped back
|
||||
// on its second import. After v17 both tables carry the same eight
|
||||
// format fields and the asymmetry is gone structurally.
|
||||
//
|
||||
// All four columns are additive and either defaulted or nullable, so
|
||||
// the ALTERs are safe on a populated v16 database:
|
||||
// - `amount_mode` / `sign_convention` carry a CHECK, same pattern as
|
||||
// the v15 `balance_accounts.kind`. `amount_mode` admits
|
||||
// 'absolute_indicator' from the start: that third mode (absolute
|
||||
// amount + a D/C indicator column) is out of scope today but must
|
||||
// be implementable without another migration, and the constraint
|
||||
// still refuses a corrupted value right now.
|
||||
// - `header_signature` stores the normalized header labels seen at
|
||||
// the last successful import, for drift detection. NULL until an
|
||||
// import records one, and permanently NULL for headerless files.
|
||||
// - `template_id` is a PROVENANCE TAG only, never re-read as format:
|
||||
// the eight columns of the source are authoritative, so editing a
|
||||
// template must not change any linked source. ON DELETE SET NULL
|
||||
// keeps the source and its format intact when a template is deleted.
|
||||
//
|
||||
// The backfill reproduces EXACTLY the rule the wizard applies on the
|
||||
// fly today (`useImportWizard.ts:321` — a mapping carrying
|
||||
// `debitAmount` means debit/credit, anything else means single amount),
|
||||
// so no existing source changes behaviour on migration. The test is
|
||||
// `LIKE '%debitAmount%'` rather than `json_extract` so the migration
|
||||
// depends on no JSON1 extension in the bundled SQLite. `sign_convention`
|
||||
// is deliberately NOT backfilled: its DEFAULT restores precisely the
|
||||
// value the code hardcoded, which is the only past convention that can
|
||||
// be inferred — guessing anything else would silently rewrite meaning.
|
||||
Migration {
|
||||
version: 17,
|
||||
description: "add amount_mode, sign_convention, header_signature and template_id to import_sources",
|
||||
sql: "ALTER TABLE import_sources ADD COLUMN amount_mode TEXT NOT NULL DEFAULT 'single' \
|
||||
CHECK (amount_mode IN ('single','debit_credit','absolute_indicator')); \
|
||||
ALTER TABLE import_sources ADD COLUMN sign_convention TEXT NOT NULL DEFAULT 'negative_expense' \
|
||||
CHECK (sign_convention IN ('negative_expense','positive_expense')); \
|
||||
ALTER TABLE import_sources ADD COLUMN header_signature TEXT; \
|
||||
ALTER TABLE import_sources ADD COLUMN template_id INTEGER \
|
||||
REFERENCES import_config_templates(id) ON DELETE SET NULL; \
|
||||
UPDATE import_sources SET amount_mode = 'debit_credit' \
|
||||
WHERE column_mapping LIKE '%debitAmount%';",
|
||||
kind: MigrationKind::Up,
|
||||
},
|
||||
];
|
||||
|
||||
tauri::Builder::default()
|
||||
|
|
@ -3415,5 +3467,460 @@ mod tests {
|
|||
.unwrap();
|
||||
assert_eq!(xyz_secs, 0, "no security for a priced account without asset_type");
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Migration v17 — the full import format on import_sources (#323)
|
||||
// -------------------------------------------------------------------------
|
||||
// What these tests guarantee:
|
||||
// - v17 applies on a POPULATED v16 database with zero loss: every existing
|
||||
// source keeps its identity and its mechanical CSV settings, and the
|
||||
// child rows that reference it survive the ALTERs.
|
||||
// - the backfill reproduces EXACTLY the rule the wizard applied on the fly
|
||||
// (`useImportWizard.ts:321`), so no source changes behaviour on
|
||||
// migration — a mapping carrying `debitAmount` becomes 'debit_credit',
|
||||
// everything else stays 'single', and `sign_convention` lands on the
|
||||
// value the code used to hardcode.
|
||||
// - the two CHECKs really bite (an unknown enum value is refused, and
|
||||
// 'absolute_indicator' is already admitted so the third amount mode
|
||||
// needs no further migration).
|
||||
// - `template_id` is a provenance tag: deleting the template it points at
|
||||
// leaves the source alive with a NULL tag, and editing a template
|
||||
// changes no linked source's format.
|
||||
// - consolidated_schema.sql and the v1→v17 chain agree column for column,
|
||||
// DEFAULT for DEFAULT, CHECK for CHECK, FK for FK.
|
||||
//
|
||||
// "v1..v16 unchanged" is NOT asserted here: no checksum harness exists in
|
||||
// this crate (V10_SQL..V17_SQL are hand-kept copies, and the real checksums
|
||||
// only live at runtime in `_sqlx_migrations`). It is verified by the diff —
|
||||
// v17 adds strictly new lines and touches no earlier migration string.
|
||||
// =========================================================================
|
||||
|
||||
/// Production v3 SQL — kept in sync with the Migration { version: 3 } entry.
|
||||
const V3_SQL: &str =
|
||||
"ALTER TABLE import_sources ADD COLUMN has_header INTEGER NOT NULL DEFAULT 1;";
|
||||
|
||||
/// Production v5 SQL — kept in sync with the Migration { version: 5 } entry.
|
||||
const V5_SQL: &str = "CREATE TABLE IF NOT EXISTS import_config_templates (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
delimiter TEXT NOT NULL DEFAULT ';',
|
||||
encoding TEXT NOT NULL DEFAULT 'utf-8',
|
||||
date_format TEXT NOT NULL DEFAULT 'DD/MM/YYYY',
|
||||
skip_lines INTEGER NOT NULL DEFAULT 0,
|
||||
has_header INTEGER NOT NULL DEFAULT 1,
|
||||
column_mapping TEXT NOT NULL,
|
||||
amount_mode TEXT NOT NULL DEFAULT 'single',
|
||||
sign_convention TEXT NOT NULL DEFAULT 'negative_expense',
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);";
|
||||
|
||||
/// Production v17 SQL — kept in sync with the Migration { version: 17 } entry.
|
||||
const V17_SQL: &str = "ALTER TABLE import_sources ADD COLUMN amount_mode TEXT NOT NULL DEFAULT 'single' \
|
||||
CHECK (amount_mode IN ('single','debit_credit','absolute_indicator')); \
|
||||
ALTER TABLE import_sources ADD COLUMN sign_convention TEXT NOT NULL DEFAULT 'negative_expense' \
|
||||
CHECK (sign_convention IN ('negative_expense','positive_expense')); \
|
||||
ALTER TABLE import_sources ADD COLUMN header_signature TEXT; \
|
||||
ALTER TABLE import_sources ADD COLUMN template_id INTEGER \
|
||||
REFERENCES import_config_templates(id) ON DELETE SET NULL; \
|
||||
UPDATE import_sources SET amount_mode = 'debit_credit' \
|
||||
WHERE column_mapping LIKE '%debitAmount%';";
|
||||
|
||||
/// Build the pre-v17 state of the two tables v17 touches. `import_sources`
|
||||
/// and `import_config_templates` are shaped by exactly three migrations —
|
||||
/// v1 (creation), v3 (`has_header`) and v5 (the templates table). None of
|
||||
/// v2, v4 and v6→v16 alters either table, so this IS their v16 shape; the
|
||||
/// parity test below re-proves it by comparing the result against the
|
||||
/// consolidated reference definition.
|
||||
fn db_pre_v17() -> Connection {
|
||||
let conn = Connection::open_in_memory().expect("open in-memory db");
|
||||
conn.execute("PRAGMA foreign_keys = ON;", [])
|
||||
.expect("enable FKs");
|
||||
conn.execute_batch(crate::database::SCHEMA)
|
||||
.expect("apply v1 SCHEMA");
|
||||
conn.execute_batch(V3_SQL).expect("apply v3");
|
||||
conn.execute_batch(V5_SQL).expect("apply v5");
|
||||
conn
|
||||
}
|
||||
|
||||
/// Insert a source and return its id. Only `name` and `column_mapping` have
|
||||
/// no default at v16, so everything else is left to the schema.
|
||||
fn seed_source(conn: &Connection, name: &str, mapping: &str) -> i64 {
|
||||
conn.execute(
|
||||
"INSERT INTO import_sources (name, column_mapping) VALUES (?1, ?2)",
|
||||
rusqlite::params![name, mapping],
|
||||
)
|
||||
.unwrap();
|
||||
conn.last_insert_rowid()
|
||||
}
|
||||
|
||||
/// (name, type, notnull, dflt_value) for every column of `table`, sorted by
|
||||
/// name. Sorting drops physical column order on purpose: ALTER TABLE can
|
||||
/// only append, while a CREATE TABLE places a column where it reads best —
|
||||
/// the order is not part of the contract, the definitions are.
|
||||
fn column_shape(conn: &Connection, table: &str) -> Vec<(String, String, i64, Option<String>)> {
|
||||
let mut cols: Vec<(String, String, i64, Option<String>)> = conn
|
||||
.prepare(&format!(
|
||||
"SELECT name, type, \"notnull\", dflt_value FROM pragma_table_info('{table}')"
|
||||
))
|
||||
.unwrap()
|
||||
.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?)))
|
||||
.unwrap()
|
||||
.map(|r| r.unwrap())
|
||||
.collect();
|
||||
cols.sort();
|
||||
cols
|
||||
}
|
||||
|
||||
/// (referenced table, from column, to column, ON DELETE action) per FK.
|
||||
fn fk_shape(conn: &Connection, table: &str) -> Vec<(String, String, String, String)> {
|
||||
let mut fks: Vec<(String, String, String, String)> = conn
|
||||
.prepare(&format!(
|
||||
"SELECT \"table\", \"from\", \"to\", on_delete FROM pragma_foreign_key_list('{table}')"
|
||||
))
|
||||
.unwrap()
|
||||
.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?)))
|
||||
.unwrap()
|
||||
.map(|r| r.unwrap())
|
||||
.collect();
|
||||
fks.sort();
|
||||
fks
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn migration_v17_applies_on_a_populated_v16_db() {
|
||||
let conn = db_pre_v17();
|
||||
|
||||
// A realistic populated v16 profile: a configured source, a template and
|
||||
// an imported file hanging off the source by FK.
|
||||
let src = seed_source(
|
||||
&conn,
|
||||
"Desjardins",
|
||||
r#"{"date":"Date","description":"Description","amount":"Montant"}"#,
|
||||
);
|
||||
conn.execute(
|
||||
"UPDATE import_sources SET delimiter = ',', encoding = 'windows-1252', \
|
||||
date_format = '%Y-%m-%d', skip_lines = 3, has_header = 0 WHERE id = ?1",
|
||||
[src],
|
||||
)
|
||||
.unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO import_config_templates (name, column_mapping) VALUES ('Modèle A', '{}')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO imported_files (source_id, filename, file_hash) \
|
||||
VALUES (?1, 'janvier.csv', 'deadbeef')",
|
||||
[src],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
conn.execute_batch(V17_SQL).expect("apply v17 on a v16 db");
|
||||
|
||||
// The four columns landed.
|
||||
let cols: Vec<String> = column_shape(&conn, "import_sources")
|
||||
.into_iter()
|
||||
.map(|(n, _, _, _)| n)
|
||||
.collect();
|
||||
for expected in &[
|
||||
"amount_mode",
|
||||
"sign_convention",
|
||||
"header_signature",
|
||||
"template_id",
|
||||
] {
|
||||
assert!(
|
||||
cols.contains(&expected.to_string()),
|
||||
"v17 must add {expected} to import_sources"
|
||||
);
|
||||
}
|
||||
|
||||
// Nothing the source already carried was touched.
|
||||
let (name, delim, enc, fmt, skip, header, mapping): (
|
||||
String,
|
||||
String,
|
||||
String,
|
||||
String,
|
||||
i64,
|
||||
i64,
|
||||
String,
|
||||
) = conn
|
||||
.query_row(
|
||||
"SELECT name, delimiter, encoding, date_format, skip_lines, has_header, \
|
||||
column_mapping FROM import_sources WHERE id = ?1",
|
||||
[src],
|
||||
|r| {
|
||||
Ok((
|
||||
r.get(0)?,
|
||||
r.get(1)?,
|
||||
r.get(2)?,
|
||||
r.get(3)?,
|
||||
r.get(4)?,
|
||||
r.get(5)?,
|
||||
r.get(6)?,
|
||||
))
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(name, "Desjardins");
|
||||
assert_eq!(delim, ",");
|
||||
assert_eq!(enc, "windows-1252");
|
||||
assert_eq!(fmt, "%Y-%m-%d");
|
||||
assert_eq!(skip, 3);
|
||||
assert_eq!(header, 0);
|
||||
assert_eq!(
|
||||
mapping,
|
||||
r#"{"date":"Date","description":"Description","amount":"Montant"}"#
|
||||
);
|
||||
|
||||
// The new columns land on their defaults: a source that never carried a
|
||||
// format now carries the exact one the code applied to it.
|
||||
let (mode, sign, sig, tpl): (String, String, Option<String>, Option<i64>) = conn
|
||||
.query_row(
|
||||
"SELECT amount_mode, sign_convention, header_signature, template_id \
|
||||
FROM import_sources WHERE id = ?1",
|
||||
[src],
|
||||
|r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?)),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(mode, "single");
|
||||
assert_eq!(sign, "negative_expense");
|
||||
assert!(sig.is_none(), "no header signature until an import records one");
|
||||
assert!(tpl.is_none(), "an existing source has no known provenance");
|
||||
|
||||
// The child row and its FK survived the ALTERs.
|
||||
let files: i64 = conn
|
||||
.query_row(
|
||||
"SELECT COUNT(*) FROM imported_files WHERE source_id = ?1",
|
||||
[src],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(files, 1, "imported_files must survive the v17 ALTERs");
|
||||
let templates: i64 = conn
|
||||
.query_row("SELECT COUNT(*) FROM import_config_templates", [], |r| {
|
||||
r.get(0)
|
||||
})
|
||||
.unwrap();
|
||||
assert_eq!(templates, 1, "templates are untouched by v17");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn migration_v17_backfill_reproduces_the_wizard_rule() {
|
||||
let conn = db_pre_v17();
|
||||
|
||||
// The rule being frozen (`useImportWizard.ts:321`):
|
||||
// mapping.debitAmount !== undefined ? "debit_credit" : "single"
|
||||
let cases: &[(&str, &str, &str)] = &[
|
||||
(
|
||||
"Deux colonnes",
|
||||
r#"{"date":"Date","description":"Libellé","debitAmount":"Débit","creditAmount":"Crédit"}"#,
|
||||
"debit_credit",
|
||||
),
|
||||
(
|
||||
"Débit seul",
|
||||
r#"{"date":"Date","description":"Libellé","debitAmount":"Retrait"}"#,
|
||||
"debit_credit",
|
||||
),
|
||||
(
|
||||
"Montant unique",
|
||||
r#"{"date":"Date","description":"Libellé","amount":"Montant"}"#,
|
||||
"single",
|
||||
),
|
||||
(
|
||||
"Crédit seul",
|
||||
r#"{"date":"Date","description":"Libellé","creditAmount":"Dépôt"}"#,
|
||||
"single",
|
||||
),
|
||||
(
|
||||
"Mapping minimal",
|
||||
r#"{"date":"Date","description":"Libellé"}"#,
|
||||
"single",
|
||||
),
|
||||
];
|
||||
for (name, mapping, _) in cases {
|
||||
seed_source(&conn, name, mapping);
|
||||
}
|
||||
|
||||
conn.execute_batch(V17_SQL).expect("apply v17");
|
||||
|
||||
for (name, _, expected_mode) in cases {
|
||||
let (mode, sign): (String, String) = conn
|
||||
.query_row(
|
||||
"SELECT amount_mode, sign_convention FROM import_sources WHERE name = ?1",
|
||||
[name],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
mode, *expected_mode,
|
||||
"{name}: the backfill must reproduce the wizard rule"
|
||||
);
|
||||
// The wizard hardcoded this on every restore, so every migrated
|
||||
// source must land on it — that is what "no behaviour change" means.
|
||||
assert_eq!(
|
||||
sign, "negative_expense",
|
||||
"{name}: sign_convention restores the previously hardcoded value"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn migration_v17_checks_reject_unknown_enum_values() {
|
||||
let conn = db_pre_v17();
|
||||
conn.execute_batch(V17_SQL).expect("apply v17");
|
||||
|
||||
let set = |col: &str, val: &str| {
|
||||
conn.execute(
|
||||
&format!("INSERT INTO import_sources (name, column_mapping, {col}) VALUES (?1, '{{}}', ?2)"),
|
||||
rusqlite::params![format!("{col}-{val}"), val],
|
||||
)
|
||||
};
|
||||
|
||||
// The third amount mode is admitted from the start: implementing it later
|
||||
// must not require another migration.
|
||||
assert!(
|
||||
set("amount_mode", "absolute_indicator").is_ok(),
|
||||
"absolute_indicator must be accepted by the v17 CHECK"
|
||||
);
|
||||
assert!(set("amount_mode", "debit_credit").is_ok());
|
||||
assert!(set("sign_convention", "positive_expense").is_ok());
|
||||
|
||||
// A corrupted value — e.g. restored from a hand-edited SREF backup — is
|
||||
// refused by the database rather than silently mis-read at parse time.
|
||||
assert!(
|
||||
set("amount_mode", "montants_bizarres").is_err(),
|
||||
"an unknown amount_mode must be refused"
|
||||
);
|
||||
assert!(
|
||||
set("sign_convention", "whatever").is_err(),
|
||||
"an unknown sign_convention must be refused"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn migration_v17_template_id_is_a_nullable_provenance_tag() {
|
||||
let conn = db_pre_v17();
|
||||
conn.execute_batch(V17_SQL).expect("apply v17");
|
||||
|
||||
conn.execute(
|
||||
"INSERT INTO import_config_templates (name, column_mapping, amount_mode, sign_convention) \
|
||||
VALUES ('Desjardins', '{}', 'single', 'positive_expense')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
let tpl = conn.last_insert_rowid();
|
||||
let src = seed_source(&conn, "Compte chèque", "{}");
|
||||
conn.execute(
|
||||
"UPDATE import_sources SET template_id = ?1, sign_convention = 'positive_expense' \
|
||||
WHERE id = ?2",
|
||||
[tpl, src],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
// Editing the template changes NO linked source: the eight format columns
|
||||
// on the source are authoritative, the tag only records provenance.
|
||||
conn.execute(
|
||||
"UPDATE import_config_templates SET amount_mode = 'debit_credit', \
|
||||
sign_convention = 'negative_expense' WHERE id = ?1",
|
||||
[tpl],
|
||||
)
|
||||
.unwrap();
|
||||
let (mode, sign): (String, String) = conn
|
||||
.query_row(
|
||||
"SELECT amount_mode, sign_convention FROM import_sources WHERE id = ?1",
|
||||
[src],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(mode, "single", "editing a template must not touch a source");
|
||||
assert_eq!(sign, "positive_expense");
|
||||
|
||||
// Deleting the template keeps the source and its format — only the tag
|
||||
// goes (ON DELETE SET NULL).
|
||||
conn.execute("DELETE FROM import_config_templates WHERE id = ?1", [tpl])
|
||||
.unwrap();
|
||||
let (tag, sign_after): (Option<i64>, String) = conn
|
||||
.query_row(
|
||||
"SELECT template_id, sign_convention FROM import_sources WHERE id = ?1",
|
||||
[src],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
.unwrap();
|
||||
assert!(tag.is_none(), "deleting a template nulls the provenance tag");
|
||||
assert_eq!(
|
||||
sign_after, "positive_expense",
|
||||
"deleting a template must not alter the source format"
|
||||
);
|
||||
|
||||
// A dangling tag is refused outright.
|
||||
assert!(
|
||||
conn.execute(
|
||||
"UPDATE import_sources SET template_id = 9999 WHERE id = ?1",
|
||||
[src]
|
||||
)
|
||||
.is_err(),
|
||||
"template_id must point at a real template"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn consolidated_schema_matches_v17_chain_on_import_sources_at_parity() {
|
||||
// The consolidated schema is the tested reference definition for new
|
||||
// profiles. Without this test the DEFAULT or the CHECK of the four v17
|
||||
// columns could drift between the two definitions unnoticed — the
|
||||
// columns are inert on the production path (this script runs after every
|
||||
// migration and only uses CREATE TABLE IF NOT EXISTS), so nothing else
|
||||
// would surface the divergence.
|
||||
let chain = db_pre_v17();
|
||||
chain.execute_batch(V17_SQL).expect("apply v17");
|
||||
let consolidated = consolidated_db();
|
||||
|
||||
// Same columns, same types, same NOT NULL flags, same DEFAULTs.
|
||||
assert_eq!(
|
||||
column_shape(&consolidated, "import_sources"),
|
||||
column_shape(&chain, "import_sources"),
|
||||
"consolidated import_sources must match the v1→v17 chain"
|
||||
);
|
||||
// Same FK, same ON DELETE action.
|
||||
assert_eq!(
|
||||
fk_shape(&consolidated, "import_sources"),
|
||||
fk_shape(&chain, "import_sources"),
|
||||
"consolidated import_sources must carry the same template_id FK"
|
||||
);
|
||||
assert_eq!(
|
||||
fk_shape(&consolidated, "import_sources"),
|
||||
vec![(
|
||||
"import_config_templates".to_string(),
|
||||
"template_id".to_string(),
|
||||
"id".to_string(),
|
||||
"SET NULL".to_string(),
|
||||
)],
|
||||
);
|
||||
|
||||
// The CHECKs are not exposed by pragma, so prove them behaviourally on
|
||||
// BOTH definitions — the same values must be accepted and refused.
|
||||
for conn in [&consolidated, &chain] {
|
||||
for (col, val, ok) in [
|
||||
("amount_mode", "absolute_indicator", true),
|
||||
("amount_mode", "debit_credit", true),
|
||||
("amount_mode", "nope", false),
|
||||
("sign_convention", "positive_expense", true),
|
||||
("sign_convention", "nope", false),
|
||||
] {
|
||||
let res = conn.execute(
|
||||
&format!(
|
||||
"INSERT INTO import_sources (name, column_mapping, {col}) \
|
||||
VALUES (?1, '{{}}', ?2)"
|
||||
),
|
||||
rusqlite::params![format!("{col}-{val}"), val],
|
||||
);
|
||||
assert_eq!(
|
||||
res.is_ok(),
|
||||
ok,
|
||||
"{col} = {val} must behave identically in both definitions"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
7
src/__fixtures__/csv/absolute-indicator.csv
Normal file
7
src/__fixtures__/csv/absolute-indicator.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Montant;Sens
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;84,32;D
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00;C
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;142,18;D
|
||||
22/01/2025;RESTAURANT LE BISTRO;56,75;D
|
||||
27/01/2025;VIREMENT RECU;300,00;C
|
||||
31/01/2025;FRAIS MENSUELS;6,95;D
|
||||
|
7
src/__fixtures__/csv/all-positive.csv
Normal file
7
src/__fixtures__/csv/all-positive.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Montant
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;84,32
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;142,18
|
||||
22/01/2025;RESTAURANT LE BISTRO;56,75
|
||||
27/01/2025;VIREMENT RECU;300,00
|
||||
31/01/2025;FRAIS MENSUELS;6,95
|
||||
|
7
src/__fixtures__/csv/bank-bnc.csv
Normal file
7
src/__fixtures__/csv/bank-bnc.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Catégorie;Débit;Crédit;Solde
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;Alimentation;84,32;;915,68
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;Revenu;;1250,00;2165,68
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;Services publics;142,18;;2023,50
|
||||
22/01/2025;RESTAURANT LE BISTRO;Restaurants;56,75;;1966,75
|
||||
27/01/2025;VIREMENT RECU;Transferts;;300,00;2266,75
|
||||
30/01/2025;FRAIS MENSUELS;Frais bancaires;6,95;;2259,80
|
||||
|
7
src/__fixtures__/csv/bank-desjardins.csv
Normal file
7
src/__fixtures__/csv/bank-desjardins.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Montant;Solde
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;-84,32;915,68
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00;2165,68
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;-142,18;2023,50
|
||||
22/01/2025;RESTAURANT LE BISTRO;-56,75;1966,75
|
||||
27/01/2025;VIREMENT RECU;300,00;2266,75
|
||||
30/01/2025;FRAIS MENSUELS;-6,95;2259,80
|
||||
|
7
src/__fixtures__/csv/bank-rbc.csv
Normal file
7
src/__fixtures__/csv/bank-rbc.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Account Type,Account Number,Transaction Date,Cheque Number,Description 1,Description 2,CAD$,USD$
|
||||
Chequing,1234567,05/01/2025,,EPICERIE METRO SAINTE-FOY,,-84.32,
|
||||
Chequing,1234567,15/01/2025,,DEPOT PAIE EMPLOYEUR,PAIE,1250.00,
|
||||
Chequing,1234567,18/01/2025,,HYDRO QUEBEC PREAUTORISE,,-142.18,
|
||||
Chequing,1234567,22/01/2025,,RESTAURANT LE BISTRO,,-56.75,
|
||||
Chequing,1234567,27/01/2025,,VIREMENT RECU,,300.00,
|
||||
Chequing,1234567,30/01/2025,241,FRAIS MENSUELS,,-6.95,
|
||||
|
7
src/__fixtures__/csv/bank-tangerine.csv
Normal file
7
src/__fixtures__/csv/bank-tangerine.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date,Transaction,Name,Memo,Amount
|
||||
05/01/2025,DEBIT,EPICERIE METRO SAINTE-FOY,,-84.32
|
||||
15/01/2025,CREDIT,DEPOT PAIE EMPLOYEUR,Paie bimensuelle,1250.00
|
||||
18/01/2025,DEBIT,HYDRO QUEBEC PREAUTORISE,,-142.18
|
||||
22/01/2025,DEBIT,RESTAURANT LE BISTRO,,-56.75
|
||||
27/01/2025,CREDIT,VIREMENT RECU,,300.00
|
||||
30/01/2025,DEBIT,FRAIS MENSUELS,,-6.95
|
||||
|
7
src/__fixtures__/csv/debit-credit-reversed.csv
Normal file
7
src/__fixtures__/csv/debit-credit-reversed.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Credit;Debit
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;;84,32
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00;
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;;142,18
|
||||
22/01/2025;RESTAURANT LE BISTRO;;56,75
|
||||
27/01/2025;VIREMENT RECU;300,00;
|
||||
31/01/2025;FRAIS MENSUELS;;6,95
|
||||
|
7
src/__fixtures__/csv/debit-credit.csv
Normal file
7
src/__fixtures__/csv/debit-credit.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Debit;Credit
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;84,32;
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;;1250,00
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;142,18;
|
||||
22/01/2025;RESTAURANT LE BISTRO;56,75;
|
||||
27/01/2025;VIREMENT RECU;;300,00
|
||||
31/01/2025;FRAIS MENSUELS;6,95;
|
||||
|
7
src/__fixtures__/csv/desjardins-quoted.csv
Normal file
7
src/__fixtures__/csv/desjardins-quoted.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
"Date,""Description"",Montant"
|
||||
"05/01/2025,""EPICERIE METRO SAINTE-FOY"",-84.32"
|
||||
"15/01/2025,""DEPOT PAIE EMPLOYEUR"",1250.00"
|
||||
"18/01/2025,""HYDRO QUEBEC PREAUTORISE"",-142.18"
|
||||
"22/01/2025,""RESTAURANT LE BISTRO"",-56.75"
|
||||
"27/01/2025,""VIREMENT RECU"",300.00"
|
||||
"31/01/2025,""FRAIS MENSUELS"",-6.95"
|
||||
|
7
src/__fixtures__/csv/header-numeric-label.csv
Normal file
7
src/__fixtures__/csv/header-numeric-label.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;2025 Montant
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;-84,32
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;-142,18
|
||||
22/01/2025;RESTAURANT LE BISTRO;-56,75
|
||||
27/01/2025;VIREMENT RECU;300,00
|
||||
31/01/2025;FRAIS MENSUELS;-6,95
|
||||
|
7
src/__fixtures__/csv/header-with-number.csv
Normal file
7
src/__fixtures__/csv/header-with-number.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Montant;Solde 2024
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;-84,32;1415,68
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00;2665,68
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;-142,18;2523,50
|
||||
22/01/2025;RESTAURANT LE BISTRO;-56,75;2466,75
|
||||
27/01/2025;VIREMENT RECU;300,00;2766,75
|
||||
31/01/2025;FRAIS MENSUELS;-6,95;2759,80
|
||||
|
68
src/__fixtures__/csv/index.ts
Normal file
68
src/__fixtures__/csv/index.ts
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
/**
|
||||
* Synthetic CSV corpus for the import-format work (issue #326).
|
||||
*
|
||||
* Every file in this directory is INVENTED. No real bank statement, no real
|
||||
* account number, no real transaction ever lands in the repository — the app is
|
||||
* privacy-first and the corpus has to be readable by anyone. Synthetic is not a
|
||||
* compromise here: these files deliberately cover shapes a single real statement
|
||||
* never contains at once (a reversed debit/credit pair, an unused column filled
|
||||
* with `0,00`, a header cell carrying a number), which is precisely the point.
|
||||
*
|
||||
* The corpus is the reference the detection rewrite (#323-#332) measures itself
|
||||
* against. Read `csvAutoDetect.test.ts` for the frozen contract, including the
|
||||
* cases that are frozen as DEFECTIVE.
|
||||
*/
|
||||
|
||||
import { readFileSync } from "fs";
|
||||
import { resolve } from "path";
|
||||
|
||||
/** Stable identifier of each corpus case. */
|
||||
export type CsvFixtureName =
|
||||
| "signed-amount"
|
||||
| "debit-credit"
|
||||
| "debit-credit-reversed"
|
||||
| "unused-column-zero"
|
||||
| "preamble"
|
||||
| "header-with-number"
|
||||
| "header-numeric-label"
|
||||
| "no-header"
|
||||
| "all-positive"
|
||||
| "desjardins-quoted"
|
||||
| "absolute-indicator"
|
||||
// Bank-signature cases (#330). Each one is the DOCUMENTED header layout of a
|
||||
// bank, invented row by row like the rest of the corpus. Three of them are
|
||||
// shapes the generic dictionary reads WRONG — Tangerine's `Transaction` type
|
||||
// column taken for the description, RBC's `CAD$` paired with the near-empty
|
||||
// `Cheque Number` as a debit/credit pair — which is what the signatures exist
|
||||
// for.
|
||||
| "bank-desjardins"
|
||||
| "bank-rbc"
|
||||
| "bank-bnc"
|
||||
| "bank-tangerine";
|
||||
|
||||
/** Every case in the corpus, in the order the issue lists them. */
|
||||
export const CSV_FIXTURE_NAMES: readonly CsvFixtureName[] = [
|
||||
"signed-amount",
|
||||
"debit-credit",
|
||||
"debit-credit-reversed",
|
||||
"unused-column-zero",
|
||||
"preamble",
|
||||
"header-with-number",
|
||||
"header-numeric-label",
|
||||
"no-header",
|
||||
"all-positive",
|
||||
"desjardins-quoted",
|
||||
"absolute-indicator",
|
||||
"bank-desjardins",
|
||||
"bank-rbc",
|
||||
"bank-bnc",
|
||||
"bank-tangerine",
|
||||
];
|
||||
|
||||
/**
|
||||
* Read one fixture as raw text, exactly as `useImportWizard` receives it from
|
||||
* the file-reading Tauri command — no trimming, no normalisation.
|
||||
*/
|
||||
export function readCsvFixture(name: CsvFixtureName): string {
|
||||
return readFileSync(resolve(import.meta.dirname, `${name}.csv`), "utf-8");
|
||||
}
|
||||
6
src/__fixtures__/csv/no-header.csv
Normal file
6
src/__fixtures__/csv/no-header.csv
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
05/01/2025;EPICERIE METRO SAINTE-FOY;-84,32
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;-142,18
|
||||
22/01/2025;RESTAURANT LE BISTRO;-56,75
|
||||
27/01/2025;VIREMENT RECU;300,00
|
||||
31/01/2025;FRAIS MENSUELS;-6,95
|
||||
|
10
src/__fixtures__/csv/preamble.csv
Normal file
10
src/__fixtures__/csv/preamble.csv
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
RELEVE DE COMPTE
|
||||
Compte cheques 12345-6789
|
||||
Periode couverte du 01 janvier 2025 au 31 janvier 2025
|
||||
Date;Description;Montant
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;-84,32
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;-142,18
|
||||
22/01/2025;RESTAURANT LE BISTRO;-56,75
|
||||
27/01/2025;VIREMENT RECU;300,00
|
||||
31/01/2025;FRAIS MENSUELS;-6,95
|
||||
|
7
src/__fixtures__/csv/signed-amount.csv
Normal file
7
src/__fixtures__/csv/signed-amount.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Montant
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;-84,32
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;1250,00
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;-142,18
|
||||
22/01/2025;RESTAURANT LE BISTRO;-56,75
|
||||
27/01/2025;VIREMENT RECU;300,00
|
||||
31/01/2025;FRAIS MENSUELS;-6,95
|
||||
|
7
src/__fixtures__/csv/unused-column-zero.csv
Normal file
7
src/__fixtures__/csv/unused-column-zero.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Date;Description;Debit;Credit
|
||||
05/01/2025;EPICERIE METRO SAINTE-FOY;84,32;0,00
|
||||
15/01/2025;DEPOT PAIE EMPLOYEUR;0,00;1250,00
|
||||
18/01/2025;HYDRO QUEBEC PREAUTORISE;142,18;0,00
|
||||
22/01/2025;RESTAURANT LE BISTRO;56,75;0,00
|
||||
27/01/2025;VIREMENT RECU;0,00;300,00
|
||||
31/01/2025;FRAIS MENSUELS;6,95;0,00
|
||||
|
262
src/__integration__/import-format-roundtrip.test.ts
Normal file
262
src/__integration__/import-format-roundtrip.test.ts
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
/**
|
||||
* Import format — save/reload integration (#324).
|
||||
*
|
||||
* The unit tests in `src/utils/importFormat.test.ts` prove the codec carries
|
||||
* every field. This file proves the SQL on either side of it does too: a format
|
||||
* written by `importSourceService` and read back is the same format, field by
|
||||
* field. That is the test that would have caught the root bug — before v17 the
|
||||
* INSERT simply had no `amount_mode` / `sign_convention` column to write to, so
|
||||
* a positive-expense source silently came back inverted on its second import.
|
||||
*
|
||||
* Like `balance-flow.test.ts`, real `tauri-plugin-sql` cannot be started outside
|
||||
* the Tauri WebView, so the services run against an in-memory FakeDb that
|
||||
* interprets the handful of statements they actually issue. It stores what it
|
||||
* is given, so `has_header` lands as the 0/1 integer SQLite stores and comes
|
||||
* back as one — the exact shape the codec has to absorb.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, vi } from "vitest";
|
||||
|
||||
vi.mock("../services/db", () => {
|
||||
const getDb = vi.fn();
|
||||
return {
|
||||
getDb,
|
||||
withTransaction: vi.fn(async (fn: (db: unknown) => unknown) => fn(await getDb())),
|
||||
};
|
||||
});
|
||||
|
||||
import { getDb } from "../services/db";
|
||||
import {
|
||||
createSource,
|
||||
getSourceByName,
|
||||
updateSource,
|
||||
} from "../services/importSourceService";
|
||||
import {
|
||||
createTemplate,
|
||||
getAllTemplates,
|
||||
updateTemplate,
|
||||
} from "../services/importConfigTemplateService";
|
||||
import { formatFromRow, formatToRow, FORMAT_FIELD_PAIRS } from "../utils/importFormat";
|
||||
import type { ImportFormat, SourceConfig } from "../shared/types";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// FakeDb — a tiny interpreter for the statements these two services issue.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type Row = Record<string, unknown>;
|
||||
|
||||
function makeFakeDb() {
|
||||
const tables: Record<string, Row[]> = {
|
||||
import_sources: [],
|
||||
import_config_templates: [],
|
||||
};
|
||||
let nextId = 1;
|
||||
|
||||
const insert = (sql: string, params: unknown[]) => {
|
||||
const [, table, cols] =
|
||||
/INSERT INTO (\w+) \(([^)]+)\)/.exec(sql) ?? [];
|
||||
const columns = cols.split(",").map((c) => c.trim());
|
||||
const row: Row = { id: nextId++ };
|
||||
columns.forEach((c, i) => (row[c] = params[i]));
|
||||
|
||||
if (/ON CONFLICT\(name\) DO UPDATE/.test(sql)) {
|
||||
const clash = tables[table].find((r) => r.name === row.name);
|
||||
if (clash) {
|
||||
// Mimic `excluded.*`: overwrite every listed column, keep the id.
|
||||
columns.forEach((c, i) => (clash[c] = params[i]));
|
||||
nextId--;
|
||||
return { lastInsertId: 0, rowsAffected: 1 };
|
||||
}
|
||||
}
|
||||
tables[table].push(row);
|
||||
return { lastInsertId: row.id as number, rowsAffected: 1 };
|
||||
};
|
||||
|
||||
const update = (sql: string, params: unknown[]) => {
|
||||
const [, table, assignments] =
|
||||
/UPDATE (\w+)\s+SET ([\s\S]+?)\s+WHERE id\s*=\s*\$(\d+)/.exec(sql) ?? [];
|
||||
const id = params[params.length - 1];
|
||||
const row = tables[table].find((r) => r.id === id);
|
||||
if (!row) return { rowsAffected: 0 };
|
||||
for (const [, column, index] of assignments.matchAll(
|
||||
/(\w+)\s*=\s*\$(\d+)/g
|
||||
)) {
|
||||
row[column] = params[Number(index) - 1];
|
||||
}
|
||||
return { rowsAffected: 1 };
|
||||
};
|
||||
|
||||
return {
|
||||
tables,
|
||||
execute: vi.fn(async (sql: string, params: unknown[] = []) => {
|
||||
if (sql.trimStart().startsWith("INSERT")) return insert(sql, params);
|
||||
if (sql.trimStart().startsWith("UPDATE")) return update(sql, params);
|
||||
throw new Error(`FakeDb: unsupported statement ${sql.slice(0, 40)}`);
|
||||
}),
|
||||
select: vi.fn(async (sql: string, params: unknown[] = []) => {
|
||||
const [, table] = /FROM (\w+)/.exec(sql) ?? [];
|
||||
const rows = tables[table];
|
||||
if (/WHERE name = \$1/.test(sql)) return rows.filter((r) => r.name === params[0]);
|
||||
if (/WHERE id = \$1/.test(sql)) return rows.filter((r) => r.id === params[0]);
|
||||
return [...rows].sort((a, b) => String(a.name).localeCompare(String(b.name)));
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
let db: ReturnType<typeof makeFakeDb>;
|
||||
|
||||
beforeEach(() => {
|
||||
db = makeFakeDb();
|
||||
vi.mocked(getDb).mockResolvedValue(db as never);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fixtures — a credit-card statement: expenses are POSITIVE. This is the
|
||||
// configuration the old code could not keep.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const CREDIT_CARD: SourceConfig = {
|
||||
name: "Visa Desjardins",
|
||||
delimiter: ",",
|
||||
encoding: "windows-1252",
|
||||
dateFormat: "YYYY-MM-DD",
|
||||
skipLines: 2,
|
||||
hasHeader: false,
|
||||
columnMapping: { date: 1, description: 3, amount: 4 },
|
||||
amountMode: "single",
|
||||
signConvention: "positive_expense",
|
||||
};
|
||||
|
||||
/** What `useImportWizard.selectSource` does on a stored source. */
|
||||
async function reload(name: string): Promise<SourceConfig> {
|
||||
const stored = await getSourceByName(name);
|
||||
if (!stored) throw new Error(`source ${name} not found`);
|
||||
return { name: stored.name, ...formatFromRow(stored) };
|
||||
}
|
||||
|
||||
function expectSameFormat(actual: ImportFormat, expected: ImportFormat) {
|
||||
for (const field of Object.keys(FORMAT_FIELD_PAIRS) as Array<keyof ImportFormat>) {
|
||||
expect(actual[field], `field ${field}`).toEqual(expected[field]);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("configure -> save -> reload (#324)", () => {
|
||||
it("returns the saved format field by field", async () => {
|
||||
await createSource({ name: CREDIT_CARD.name, ...formatToRow(CREDIT_CARD) });
|
||||
expectSameFormat(await reload(CREDIT_CARD.name), CREDIT_CARD);
|
||||
});
|
||||
|
||||
it("keeps positive_expense on the second import and every one after", async () => {
|
||||
await createSource({ name: CREDIT_CARD.name, ...formatToRow(CREDIT_CARD) });
|
||||
|
||||
// Three consecutive imports re-save what was reloaded, as the wizard does.
|
||||
let current = await reload(CREDIT_CARD.name);
|
||||
for (let i = 0; i < 3; i++) {
|
||||
const id = (await getSourceByName(CREDIT_CARD.name))!.id;
|
||||
await updateSource(id, { name: current.name, ...formatToRow(current) });
|
||||
current = await reload(CREDIT_CARD.name);
|
||||
}
|
||||
|
||||
expect(current.signConvention).toBe("positive_expense");
|
||||
expectSameFormat(current, CREDIT_CARD);
|
||||
});
|
||||
|
||||
it("stores has_header as the integer SQLite holds, not a JS boolean", async () => {
|
||||
await createSource({ name: CREDIT_CARD.name, ...formatToRow(CREDIT_CARD) });
|
||||
expect(db.tables.import_sources[0].has_header).toBe(0);
|
||||
expect((await reload(CREDIT_CARD.name)).hasHeader).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps the chosen mode when the mapping does not betray it", async () => {
|
||||
const debitCredit: SourceConfig = {
|
||||
...CREDIT_CARD,
|
||||
amountMode: "debit_credit",
|
||||
// No debitAmount: the old restore re-inferred "single" from this shape.
|
||||
columnMapping: { date: 1, description: 3, creditAmount: 5 },
|
||||
};
|
||||
await createSource({ name: debitCredit.name, ...formatToRow(debitCredit) });
|
||||
expect((await reload(debitCredit.name)).amountMode).toBe("debit_credit");
|
||||
});
|
||||
|
||||
it("updates the format in place when the source is re-configured", async () => {
|
||||
const id = await createSource({
|
||||
name: CREDIT_CARD.name,
|
||||
...formatToRow(CREDIT_CARD),
|
||||
});
|
||||
|
||||
const flipped: SourceConfig = {
|
||||
...CREDIT_CARD,
|
||||
signConvention: "negative_expense",
|
||||
amountMode: "debit_credit",
|
||||
columnMapping: { date: 1, description: 3, debitAmount: 4, creditAmount: 5 },
|
||||
};
|
||||
await updateSource(id, { name: flipped.name, ...formatToRow(flipped) });
|
||||
|
||||
expectSameFormat(await reload(CREDIT_CARD.name), flipped);
|
||||
expect(db.tables.import_sources).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("template_id is provenance, never format (#324)", () => {
|
||||
async function seedLinkedPair() {
|
||||
const templateId = await createTemplate({
|
||||
name: "Desjardins carte",
|
||||
...formatToRow(CREDIT_CARD),
|
||||
});
|
||||
await createSource({
|
||||
name: CREDIT_CARD.name,
|
||||
...formatToRow(CREDIT_CARD),
|
||||
template_id: templateId,
|
||||
});
|
||||
return templateId;
|
||||
}
|
||||
|
||||
it("records the template the source was configured from", async () => {
|
||||
const templateId = await seedLinkedPair();
|
||||
expect((await getSourceByName(CREDIT_CARD.name))!.template_id).toBe(templateId);
|
||||
});
|
||||
|
||||
// The acceptance criterion: the eight source columns are authoritative, so a
|
||||
// template edit cannot reach through the link and change how a source reads.
|
||||
it("editing the template alters no linked source", async () => {
|
||||
const templateId = await seedLinkedPair();
|
||||
|
||||
await updateTemplate(templateId, {
|
||||
name: "Desjardins carte",
|
||||
...formatToRow({
|
||||
...CREDIT_CARD,
|
||||
delimiter: "\t",
|
||||
encoding: "utf-8",
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
skipLines: 0,
|
||||
hasHeader: true,
|
||||
columnMapping: { date: 0, description: 1, debitAmount: 2, creditAmount: 3 },
|
||||
amountMode: "debit_credit",
|
||||
signConvention: "negative_expense",
|
||||
}),
|
||||
});
|
||||
|
||||
expectSameFormat(await reload(CREDIT_CARD.name), CREDIT_CARD);
|
||||
|
||||
// …and the template really did change, so the test is not vacuous.
|
||||
const [stored] = await getAllTemplates();
|
||||
expect(formatFromRow(stored).signConvention).toBe("negative_expense");
|
||||
expect(formatFromRow(stored).amountMode).toBe("debit_credit");
|
||||
});
|
||||
|
||||
it("survives a re-import that re-saves the format", async () => {
|
||||
const templateId = await seedLinkedPair();
|
||||
const id = (await getSourceByName(CREDIT_CARD.name))!.id;
|
||||
const current = await reload(CREDIT_CARD.name);
|
||||
|
||||
await updateSource(id, {
|
||||
name: current.name,
|
||||
...formatToRow(current),
|
||||
template_id: templateId,
|
||||
});
|
||||
|
||||
expect((await getSourceByName(CREDIT_CARD.name))!.template_id).toBe(templateId);
|
||||
});
|
||||
});
|
||||
|
|
@ -1,12 +1,19 @@
|
|||
import { useTranslation } from "react-i18next";
|
||||
import type { ColumnMapping, AmountMode } from "../../shared/types";
|
||||
import { clearMappingForMode } from "../../utils/importFormat";
|
||||
|
||||
interface ColumnMappingEditorProps {
|
||||
headers: string[];
|
||||
mapping: ColumnMapping;
|
||||
amountMode: AmountMode;
|
||||
onMappingChange: (mapping: ColumnMapping) => void;
|
||||
onAmountModeChange: (mode: AmountMode) => void;
|
||||
/**
|
||||
* The mode carries the pruned mapping with it. Both have to land in a single
|
||||
* state update: the parent's handlers each spread the same `config` prop, so
|
||||
* two consecutive calls would see the same stale value and the second would
|
||||
* overwrite the first.
|
||||
*/
|
||||
onAmountModeChange: (mode: AmountMode, mapping: ColumnMapping) => void;
|
||||
}
|
||||
|
||||
export default function ColumnMappingEditor({
|
||||
|
|
@ -18,6 +25,12 @@ export default function ColumnMappingEditor({
|
|||
}: ColumnMappingEditorProps) {
|
||||
const { t } = useTranslation();
|
||||
|
||||
// The mode is the source of truth for the mapping, not the reverse: leaving a
|
||||
// mode drops its columns so a stale key cannot keep deciding how amounts are
|
||||
// read (#324).
|
||||
const selectMode = (mode: AmountMode) =>
|
||||
onAmountModeChange(mode, clearMappingForMode(mapping, mode));
|
||||
|
||||
const columnOptions = headers.map((h, i) => (
|
||||
<option key={i} value={i}>
|
||||
{i}: {h}
|
||||
|
|
@ -79,7 +92,7 @@ export default function ColumnMappingEditor({
|
|||
name="amountMode"
|
||||
value="single"
|
||||
checked={amountMode === "single"}
|
||||
onChange={() => onAmountModeChange("single")}
|
||||
onChange={() => selectMode("single")}
|
||||
className="accent-[var(--primary)]"
|
||||
/>
|
||||
{t("import.config.singleAmount")}
|
||||
|
|
@ -90,7 +103,7 @@ export default function ColumnMappingEditor({
|
|||
name="amountMode"
|
||||
value="debit_credit"
|
||||
checked={amountMode === "debit_credit"}
|
||||
onChange={() => onAmountModeChange("debit_credit")}
|
||||
onChange={() => selectMode("debit_credit")}
|
||||
className="accent-[var(--primary)]"
|
||||
/>
|
||||
{t("import.config.debitCredit")}
|
||||
|
|
|
|||
|
|
@ -1,61 +0,0 @@
|
|||
import { useEffect } from "react";
|
||||
import { createPortal } from "react-dom";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { X } from "lucide-react";
|
||||
import FilePreviewTable from "./FilePreviewTable";
|
||||
import type { ParsedRow } from "../../shared/types";
|
||||
|
||||
interface FilePreviewModalProps {
|
||||
rows: ParsedRow[];
|
||||
totalCount: number;
|
||||
onClose: () => void;
|
||||
}
|
||||
|
||||
export default function FilePreviewModal({
|
||||
rows,
|
||||
totalCount,
|
||||
onClose,
|
||||
}: FilePreviewModalProps) {
|
||||
const { t } = useTranslation();
|
||||
|
||||
useEffect(() => {
|
||||
function handleEscape(e: KeyboardEvent) {
|
||||
if (e.key === "Escape") onClose();
|
||||
}
|
||||
document.addEventListener("keydown", handleEscape);
|
||||
return () => document.removeEventListener("keydown", handleEscape);
|
||||
}, [onClose]);
|
||||
|
||||
return createPortal(
|
||||
<div
|
||||
className="fixed inset-0 z-[200] flex items-center justify-center bg-black/50"
|
||||
onClick={(e) => { if (e.target === e.currentTarget) onClose(); }}
|
||||
>
|
||||
<div className="bg-[var(--card)] rounded-xl border border-[var(--border)] shadow-2xl w-full max-w-4xl max-h-[85vh] flex flex-col mx-4">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between px-6 py-4 border-b border-[var(--border)]">
|
||||
<h2 className="text-lg font-semibold">{t("import.preview.title")}</h2>
|
||||
<button
|
||||
onClick={onClose}
|
||||
className="p-1 rounded-lg hover:bg-[var(--muted)] transition-colors"
|
||||
>
|
||||
<X size={18} />
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{/* Body */}
|
||||
<div className="flex-1 overflow-auto p-6">
|
||||
<FilePreviewTable rows={rows} />
|
||||
{totalCount > rows.length && (
|
||||
<p className="text-sm text-[var(--muted-foreground)] text-center mt-4">
|
||||
{t("import.preview.moreRows", {
|
||||
count: totalCount - rows.length,
|
||||
})}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>,
|
||||
document.body
|
||||
);
|
||||
}
|
||||
|
|
@ -1,15 +1,25 @@
|
|||
import { useTranslation } from "react-i18next";
|
||||
import { AlertCircle } from "lucide-react";
|
||||
import { AlertCircle, ArrowDownLeft, ArrowUpRight, RefreshCw } from "lucide-react";
|
||||
import type { ParsedRow } from "../../shared/types";
|
||||
import { isRowErrorKey, summarizeParsedRows } from "../../utils/importFormat";
|
||||
import RepairPathNotice from "./RepairPathNotice";
|
||||
|
||||
/** Rows rendered in the table. The recap above it always covers the whole file. */
|
||||
const DISPLAYED_ROWS = 20;
|
||||
|
||||
interface FilePreviewTableProps {
|
||||
/** ALL parsed rows — the recap is meaningless on a truncated sample. */
|
||||
rows: ParsedRow[];
|
||||
onFlipSigns?: () => void;
|
||||
isFlipping?: boolean;
|
||||
}
|
||||
|
||||
export default function FilePreviewTable({
|
||||
rows,
|
||||
onFlipSigns,
|
||||
isFlipping = false,
|
||||
}: FilePreviewTableProps) {
|
||||
const { t } = useTranslation();
|
||||
const { t, i18n } = useTranslation();
|
||||
|
||||
if (rows.length === 0) {
|
||||
return (
|
||||
|
|
@ -19,7 +29,17 @@ export default function FilePreviewTable({
|
|||
);
|
||||
}
|
||||
|
||||
const errorCount = rows.filter((r) => r.error).length;
|
||||
const totals = summarizeParsedRows(rows);
|
||||
const displayedRows = rows.slice(0, DISPLAYED_ROWS);
|
||||
|
||||
const currency = new Intl.NumberFormat(
|
||||
i18n.language === "fr" ? "fr-CA" : "en-CA",
|
||||
{ style: "currency", currency: "CAD" }
|
||||
);
|
||||
|
||||
// Row errors are i18n keys (#325); anything else reaches us verbatim.
|
||||
const errorText = (error: string) =>
|
||||
isRowErrorKey(error) ? t(error) : error;
|
||||
|
||||
return (
|
||||
<div>
|
||||
|
|
@ -31,15 +51,87 @@ export default function FilePreviewTable({
|
|||
<span className="text-[var(--muted-foreground)]">
|
||||
{t("import.preview.rowCount", { count: rows.length })}
|
||||
</span>
|
||||
{errorCount > 0 && (
|
||||
{totals.errorCount > 0 && (
|
||||
<span className="flex items-center gap-1 text-[var(--negative)]">
|
||||
<AlertCircle size={14} />
|
||||
{t("import.preview.errorCount", { count: errorCount })}
|
||||
{t("import.preview.errorCount", { count: totals.errorCount })}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/*
|
||||
The signed recap (#329) — the last control before the database, and the
|
||||
only one that looks at what the amounts MEAN. A statement showing zero
|
||||
inflows next to a payroll line is wrong on its face, whatever produced
|
||||
it. Computed over every row, never over the twenty displayed below.
|
||||
*/}
|
||||
<div className="mb-4 p-4 rounded-xl bg-[var(--card)] border border-[var(--border)]">
|
||||
<div className="flex items-start justify-between gap-4 flex-wrap">
|
||||
<div className="grid grid-cols-1 sm:grid-cols-3 gap-4 flex-1 min-w-[16rem]">
|
||||
<div>
|
||||
<p className="flex items-center gap-1 text-xs text-[var(--muted-foreground)]">
|
||||
<ArrowDownLeft size={14} className="text-[var(--negative)]" />
|
||||
{t("import.preview.outflowCount", { count: totals.outflowCount })}
|
||||
</p>
|
||||
<p className="font-mono text-sm text-[var(--negative)]">
|
||||
{currency.format(totals.outflowTotal)}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="flex items-center gap-1 text-xs text-[var(--muted-foreground)]">
|
||||
<ArrowUpRight size={14} className="text-[var(--positive)]" />
|
||||
{t("import.preview.inflowCount", { count: totals.inflowCount })}
|
||||
</p>
|
||||
<p className="font-mono text-sm text-[var(--positive)]">
|
||||
{currency.format(totals.inflowTotal)}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="flex items-center gap-1 text-xs text-[var(--muted-foreground)]">
|
||||
<AlertCircle size={14} />
|
||||
{t("import.preview.errorRows")}
|
||||
</p>
|
||||
<p
|
||||
className={`font-mono text-sm ${
|
||||
totals.errorCount > 0
|
||||
? "text-[var(--negative)]"
|
||||
: "text-[var(--muted-foreground)]"
|
||||
}`}
|
||||
>
|
||||
{totals.errorCount}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{onFlipSigns && (
|
||||
<div className="text-right max-w-xs">
|
||||
<button
|
||||
onClick={onFlipSigns}
|
||||
disabled={isFlipping}
|
||||
className="flex items-center gap-1 px-3 py-2 text-sm rounded-lg border border-[var(--border)] text-[var(--foreground)] hover:bg-[var(--muted)] transition-colors disabled:opacity-50 disabled:cursor-not-allowed"
|
||||
>
|
||||
<RefreshCw size={14} />
|
||||
{t("import.preview.flipSigns")}
|
||||
</button>
|
||||
<p className="mt-1 text-xs text-[var(--muted-foreground)]">
|
||||
{t("import.preview.flipSignsHint")}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/*
|
||||
The repair path (#330), stated where the correction is offered. A user
|
||||
who flips the signs here has just learned that earlier imports of the
|
||||
same source read backwards; re-importing those files is the natural
|
||||
next move and it double-books every row instead of fixing them.
|
||||
*/}
|
||||
<div className="mt-4">
|
||||
<RepairPathNotice />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="overflow-x-auto rounded-xl border border-[var(--border)]">
|
||||
<table className="w-full text-sm">
|
||||
<thead>
|
||||
|
|
@ -62,7 +154,7 @@ export default function FilePreviewTable({
|
|||
</tr>
|
||||
</thead>
|
||||
<tbody className="divide-y divide-[var(--border)]">
|
||||
{rows.map((row) => (
|
||||
{displayedRows.map((row) => (
|
||||
<tr
|
||||
key={row.rowIndex}
|
||||
className={
|
||||
|
|
@ -77,7 +169,7 @@ export default function FilePreviewTable({
|
|||
<td className="px-3 py-2">
|
||||
{row.parsed?.date || (
|
||||
<span className="text-[var(--negative)] text-xs">
|
||||
{row.error || "—"}
|
||||
{row.error ? errorText(row.error) : "—"}
|
||||
</span>
|
||||
)}
|
||||
</td>
|
||||
|
|
@ -104,6 +196,14 @@ export default function FilePreviewTable({
|
|||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{rows.length > displayedRows.length && (
|
||||
<p className="text-sm text-[var(--muted-foreground)] text-center mt-4">
|
||||
{t("import.preview.moreRows", {
|
||||
count: rows.length - displayedRows.length,
|
||||
})}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
|
|||
133
src/components/import/FormatDriftPanel.tsx
Normal file
133
src/components/import/FormatDriftPanel.tsx
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
import { useTranslation } from "react-i18next";
|
||||
import { AlertTriangle, ArrowRight, Minus, Plus } from "lucide-react";
|
||||
import type { HeaderDriftEntry } from "../../utils/bankSignatures";
|
||||
import RepairPathNotice from "./RepairPathNotice";
|
||||
|
||||
/**
|
||||
* The header row of this source changed since its last successful import
|
||||
* (#330).
|
||||
*
|
||||
* This is the failure mode the whole chantier is most afraid of, because it is
|
||||
* the one nobody sees: a bank moves a column, the stored mapping keeps reading
|
||||
* position 3, and the import succeeds — with the balance in the amount column,
|
||||
* for as long as it takes someone to notice. There is no signal in the totals,
|
||||
* no error, no red row. The only signal is the header itself, which is why it
|
||||
* is recorded and compared.
|
||||
*
|
||||
* The panel does not decide. It shows what moved, column by column, and offers
|
||||
* the two answers that exist: read the file the way it looks now, or keep
|
||||
* reading it the way it used to. Nothing here writes to the database.
|
||||
*/
|
||||
interface FormatDriftPanelProps {
|
||||
entries: HeaderDriftEntry[];
|
||||
/** False when detection could not read the new shape — nothing to adopt. */
|
||||
canAdopt: boolean;
|
||||
onAdopt: () => void;
|
||||
onKeep: () => void;
|
||||
isBusy?: boolean;
|
||||
}
|
||||
|
||||
export default function FormatDriftPanel({
|
||||
entries,
|
||||
canAdopt,
|
||||
onAdopt,
|
||||
onKeep,
|
||||
isBusy = false,
|
||||
}: FormatDriftPanelProps) {
|
||||
const { t } = useTranslation();
|
||||
|
||||
const buttonClass =
|
||||
"px-3 py-2 text-sm rounded-lg transition-colors disabled:opacity-50 disabled:cursor-not-allowed";
|
||||
|
||||
return (
|
||||
<div className="p-4 rounded-xl bg-[var(--card)] border-2 border-[var(--accent)] space-y-4">
|
||||
<div className="flex items-start gap-2">
|
||||
<AlertTriangle
|
||||
size={18}
|
||||
className="text-[var(--accent)] shrink-0 mt-0.5"
|
||||
/>
|
||||
<div>
|
||||
<h3 className="text-sm font-semibold text-[var(--foreground)]">
|
||||
{t("import.drift.title")}
|
||||
</h3>
|
||||
<p className="text-sm text-[var(--muted-foreground)] mt-0.5">
|
||||
{t("import.drift.intro")}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/*
|
||||
Column by column, naming the columns. This is what the stored signature
|
||||
being a label list rather than a hash buys: a hash can say "something
|
||||
changed" and nothing more.
|
||||
*/}
|
||||
<ul className="space-y-1 text-sm font-mono">
|
||||
{entries.map((entry) => (
|
||||
<li
|
||||
key={`${entry.kind}-${entry.normalizedLabel}`}
|
||||
className="flex items-center gap-2 text-[var(--foreground)]"
|
||||
>
|
||||
{entry.kind === "moved" && (
|
||||
<ArrowRight size={14} className="text-[var(--accent)] shrink-0" />
|
||||
)}
|
||||
{entry.kind === "added" && (
|
||||
<Plus size={14} className="text-[var(--positive)] shrink-0" />
|
||||
)}
|
||||
{entry.kind === "removed" && (
|
||||
<Minus size={14} className="text-[var(--negative)] shrink-0" />
|
||||
)}
|
||||
<span>
|
||||
{entry.kind === "moved" &&
|
||||
t("import.drift.columnMoved", {
|
||||
label: entry.label,
|
||||
from: entry.previousIndex,
|
||||
to: entry.currentIndex,
|
||||
})}
|
||||
{entry.kind === "added" &&
|
||||
t("import.drift.columnAdded", {
|
||||
label: entry.label,
|
||||
to: entry.currentIndex,
|
||||
})}
|
||||
{entry.kind === "removed" &&
|
||||
t("import.drift.columnRemoved", {
|
||||
label: entry.label,
|
||||
from: entry.previousIndex,
|
||||
})}
|
||||
</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
|
||||
<RepairPathNotice />
|
||||
|
||||
<div className="flex flex-wrap items-start gap-4">
|
||||
<div>
|
||||
<button
|
||||
onClick={onAdopt}
|
||||
disabled={!canAdopt || isBusy}
|
||||
className={`${buttonClass} bg-[var(--primary)] text-white hover:opacity-90`}
|
||||
>
|
||||
{t("import.drift.adopt")}
|
||||
</button>
|
||||
<p className="mt-1 text-xs text-[var(--muted-foreground)] max-w-xs">
|
||||
{canAdopt
|
||||
? t("import.drift.adoptHint")
|
||||
: t("import.drift.adoptUnavailable")}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<button
|
||||
onClick={onKeep}
|
||||
disabled={isBusy}
|
||||
className={`${buttonClass} border border-[var(--border)] text-[var(--foreground)] hover:bg-[var(--muted)]`}
|
||||
>
|
||||
{t("import.drift.keep")}
|
||||
</button>
|
||||
<p className="mt-1 text-xs text-[var(--muted-foreground)] max-w-xs">
|
||||
{t("import.drift.keepHint")}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -5,6 +5,8 @@ import type { SourceConfig, ScannedFile, DuplicateCheckResult } from "../../shar
|
|||
interface ImportConfirmationProps {
|
||||
sourceName: string;
|
||||
config: SourceConfig;
|
||||
/** Header labels of the parsed file, used to name the mapped columns. */
|
||||
headers?: string[];
|
||||
selectedFiles: ScannedFile[];
|
||||
duplicateResult: DuplicateCheckResult;
|
||||
excludedCount: number;
|
||||
|
|
@ -13,6 +15,7 @@ interface ImportConfirmationProps {
|
|||
export default function ImportConfirmation({
|
||||
sourceName,
|
||||
config,
|
||||
headers = [],
|
||||
selectedFiles,
|
||||
duplicateResult,
|
||||
excludedCount,
|
||||
|
|
@ -22,6 +25,32 @@ export default function ImportConfirmation({
|
|||
const rowsToImport =
|
||||
duplicateResult.newRows.length + duplicateResult.duplicateRows.length - excludedCount;
|
||||
|
||||
/**
|
||||
* A mapped column, named by its header when the file has one. An index alone
|
||||
* ("2") tells the reader nothing about whether the right column was picked.
|
||||
*/
|
||||
const columnLabel = (index: number | undefined): string => {
|
||||
if (index === undefined) return t("import.confirm.columnUnmapped");
|
||||
const header = headers[index]?.trim();
|
||||
return header ? `${index} — ${header}` : String(index);
|
||||
};
|
||||
|
||||
// The mapping as the amount mode actually reads it: naming a debit column on
|
||||
// a single-amount format would describe an import that is not happening.
|
||||
const mappedColumns: Array<[string, number | undefined]> =
|
||||
config.amountMode === "debit_credit"
|
||||
? [
|
||||
["import.config.dateColumn", config.columnMapping.date],
|
||||
["import.config.descriptionColumn", config.columnMapping.description],
|
||||
["import.config.debitColumn", config.columnMapping.debitAmount],
|
||||
["import.config.creditColumn", config.columnMapping.creditAmount],
|
||||
]
|
||||
: [
|
||||
["import.config.dateColumn", config.columnMapping.date],
|
||||
["import.config.descriptionColumn", config.columnMapping.description],
|
||||
["import.config.amountColumn", config.columnMapping.amount],
|
||||
];
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
<h2 className="text-lg font-semibold">
|
||||
|
|
@ -73,6 +102,47 @@ export default function ImportConfirmation({
|
|||
<span className="font-medium">{t("import.config.skipLines")}:</span>{" "}
|
||||
{config.skipLines}
|
||||
</div>
|
||||
{/*
|
||||
The amount mode and the sign convention (#329). They decide what
|
||||
every amount MEANS, and they were the two settings this last
|
||||
checkpoint did not show — the most fragile pair, invisible.
|
||||
*/}
|
||||
<div>
|
||||
<span className="font-medium">{t("import.config.amountMode")}:</span>{" "}
|
||||
{config.amountMode === "debit_credit"
|
||||
? t("import.config.debitCredit")
|
||||
: t("import.config.singleAmount")}
|
||||
</div>
|
||||
{/*
|
||||
Single-amount mode only: `mapRow` computes `credit - debit` on
|
||||
magnitudes in debit/credit mode and never reads the convention
|
||||
there, so stating one would describe a rule that is not applied.
|
||||
*/}
|
||||
{config.amountMode === "single" && (
|
||||
<div>
|
||||
<span className="font-medium">
|
||||
{t("import.config.signConvention")}:
|
||||
</span>{" "}
|
||||
{config.signConvention === "positive_expense"
|
||||
? t("import.config.positiveExpense")
|
||||
: t("import.config.negativeExpense")}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Column mapping */}
|
||||
<div className="p-4">
|
||||
<p className="text-sm font-medium mb-2">
|
||||
{t("import.config.columnMapping")}
|
||||
</p>
|
||||
<div className="grid grid-cols-2 md:grid-cols-4 gap-2 text-xs text-[var(--muted-foreground)]">
|
||||
{mappedColumns.map(([labelKey, index]) => (
|
||||
<div key={labelKey}>
|
||||
<span className="font-medium">{t(labelKey)}:</span>{" "}
|
||||
{columnLabel(index)}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import {
|
|||
FileText,
|
||||
} from "lucide-react";
|
||||
import type { ImportReport } from "../../shared/types";
|
||||
import { isRowErrorKey } from "../../utils/importFormat";
|
||||
|
||||
interface ImportReportPanelProps {
|
||||
report: ImportReport;
|
||||
|
|
@ -104,7 +105,11 @@ export default function ImportReportPanel({
|
|||
{report.errors.map((err, i) => (
|
||||
<tr key={i}>
|
||||
<td className="px-3 py-2">{err.rowIndex + 1}</td>
|
||||
<td className="px-3 py-2 text-[var(--negative)]">{err.message}</td>
|
||||
<td className="px-3 py-2 text-[var(--negative)]">
|
||||
{/* Row errors are i18n keys (#325); an exception message
|
||||
that reached this list is NOT one and stays verbatim. */}
|
||||
{isRowErrorKey(err.message) ? t(err.message) : err.message}
|
||||
</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
|
|
|
|||
44
src/components/import/RepairPathNotice.tsx
Normal file
44
src/components/import/RepairPathNotice.tsx
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
import { useTranslation } from "react-i18next";
|
||||
import { LifeBuoy } from "lucide-react";
|
||||
|
||||
/**
|
||||
* The only safe way to repair an import that was already written wrong (#330).
|
||||
*
|
||||
* WHY THIS IS IN THE INTERFACE AND NOT ONLY IN A RISK TABLE. Every correction
|
||||
* this chantier delivers — the restored format (#324), the anchored amount
|
||||
* parser (#325), the sign flip and the drift panel — changes the amounts a
|
||||
* source PRODUCES. The natural next move is to re-import the file "now that it
|
||||
* reads right", and that move silently doubles the ledger:
|
||||
* `findDuplicates` (`transactionService.ts:164`) matches on
|
||||
* `date AND description AND amount`, so a corrected row does not match the
|
||||
* wrong row already stored. It is filed as new.
|
||||
*
|
||||
* A flipped sign is the worst shape of it: the wrong row and the corrected row
|
||||
* are mirror images, they net to about zero in every report, and nothing on any
|
||||
* screen looks off. The only path that leaves a correct ledger is deleting the
|
||||
* faulty import from the history first — `deleteImportWithTransactions` takes
|
||||
* its transactions with it — and replaying the file afterwards.
|
||||
*
|
||||
* Shown in both places a user can be about to do exactly that: the preview,
|
||||
* next to the sign flip, and the format-drift panel.
|
||||
*/
|
||||
export default function RepairPathNotice() {
|
||||
const { t } = useTranslation();
|
||||
|
||||
return (
|
||||
<div className="flex items-start gap-2 p-3 rounded-lg bg-[var(--muted)] border border-[var(--border)]">
|
||||
<LifeBuoy
|
||||
size={14}
|
||||
className="text-[var(--muted-foreground)] shrink-0 mt-0.5"
|
||||
/>
|
||||
<div>
|
||||
<p className="text-xs font-semibold text-[var(--foreground)]">
|
||||
{t("import.repairPath.title")}
|
||||
</p>
|
||||
<p className="text-xs text-[var(--muted-foreground)] mt-0.5">
|
||||
{t("import.repairPath.body")}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
import { useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { Wand2, Check, Save, X } from "lucide-react";
|
||||
import { Wand2, Check, Save, X, AlertTriangle } from "lucide-react";
|
||||
import type {
|
||||
ScannedSource,
|
||||
ScannedFile,
|
||||
|
|
@ -9,6 +9,11 @@ import type {
|
|||
ColumnMapping,
|
||||
ImportConfigTemplate,
|
||||
} from "../../shared/types";
|
||||
import type { DetectionScore } from "../../utils/csvAutoDetect";
|
||||
import {
|
||||
bankSignatureById,
|
||||
type BankSignatureId,
|
||||
} from "../../utils/bankSignatures";
|
||||
import ColumnMappingEditor from "./ColumnMappingEditor";
|
||||
|
||||
interface SourceConfigPanelProps {
|
||||
|
|
@ -27,6 +32,10 @@ interface SourceConfigPanelProps {
|
|||
onUpdateTemplate: () => void;
|
||||
onDeleteTemplate: (id: number) => void;
|
||||
selectedTemplateId: number | null;
|
||||
/** Result of the last detection run, or null when none ran on this source. */
|
||||
detectionScore?: DetectionScore | null;
|
||||
/** Bank recognised by signature during that same run, or null (#330). */
|
||||
detectedBank?: BankSignatureId | null;
|
||||
isLoading?: boolean;
|
||||
}
|
||||
|
||||
|
|
@ -46,12 +55,24 @@ export default function SourceConfigPanel({
|
|||
onUpdateTemplate,
|
||||
onDeleteTemplate,
|
||||
selectedTemplateId,
|
||||
detectionScore,
|
||||
detectedBank,
|
||||
isLoading,
|
||||
}: SourceConfigPanelProps) {
|
||||
const { t } = useTranslation();
|
||||
const [showSaveTemplate, setShowSaveTemplate] = useState(false);
|
||||
const [templateName, setTemplateName] = useState("");
|
||||
|
||||
/*
|
||||
A recognised bank replaces the generic wording, but ONLY on the confident
|
||||
arm (#330). "Format Desjardins reconnu" over a file two thirds of whose
|
||||
rows would not read is a claim the app cannot back: the label matched, the
|
||||
data did not follow, and the uncertain wording is the one that helps.
|
||||
*/
|
||||
const bank = detectionScore?.confident
|
||||
? bankSignatureById(detectedBank)
|
||||
: null;
|
||||
|
||||
const selectClass =
|
||||
"w-full px-3 py-2 text-sm rounded-lg border border-[var(--border)] bg-[var(--card)] text-[var(--foreground)] focus:outline-none focus:ring-2 focus:ring-[var(--primary)]";
|
||||
const inputClass = selectClass;
|
||||
|
|
@ -76,6 +97,60 @@ export default function SourceConfigPanel({
|
|||
</button>
|
||||
</div>
|
||||
|
||||
{/*
|
||||
Detection result. Present only when a detection actually ran on THIS
|
||||
source — a source opened on its stored format shows nothing, because
|
||||
that format was chosen once and is not being re-guessed.
|
||||
|
||||
The threshold colours the banner and nothing else: both arms leave every
|
||||
control editable and every button enabled. A low score is a reason to
|
||||
look at the preview, not a refusal to import.
|
||||
*/}
|
||||
{detectionScore && (
|
||||
<div
|
||||
className={`p-3 rounded-xl bg-[var(--card)] border-2 flex items-start gap-2 ${
|
||||
detectionScore.confident
|
||||
? "border-[var(--border)]"
|
||||
: "border-[var(--accent)]"
|
||||
}`}
|
||||
>
|
||||
{detectionScore.confident ? (
|
||||
<Check size={16} className="text-[var(--positive)] shrink-0 mt-0.5" />
|
||||
) : (
|
||||
<AlertTriangle
|
||||
size={16}
|
||||
className="text-[var(--accent)] shrink-0 mt-0.5"
|
||||
/>
|
||||
)}
|
||||
<div>
|
||||
<p className="text-sm text-[var(--foreground)]">
|
||||
{bank
|
||||
? t("import.config.detectionBank", {
|
||||
// A bank name is a proper noun: it is interpolated, never
|
||||
// translated.
|
||||
bank: bank.label,
|
||||
read: detectionScore.readRows,
|
||||
total: detectionScore.totalRows,
|
||||
})
|
||||
: t(
|
||||
detectionScore.confident
|
||||
? "import.config.detectionRecognized"
|
||||
: "import.config.detectionUncertain",
|
||||
{
|
||||
read: detectionScore.readRows,
|
||||
total: detectionScore.totalRows,
|
||||
}
|
||||
)}
|
||||
</p>
|
||||
{!detectionScore.confident && (
|
||||
<p className="text-xs text-[var(--muted-foreground)] mt-0.5">
|
||||
{t("import.config.detectionUncertainHint")}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Template row */}
|
||||
<div className="flex items-center gap-3 flex-wrap">
|
||||
<div className="flex items-center gap-2 flex-1 min-w-[200px]">
|
||||
|
|
@ -270,40 +345,50 @@ export default function SourceConfigPanel({
|
|||
</div>
|
||||
</div>
|
||||
|
||||
{/* Sign convention */}
|
||||
<div>
|
||||
<label className="block text-sm text-[var(--muted-foreground)] mb-1">
|
||||
{t("import.config.signConvention")}
|
||||
</label>
|
||||
<div className="flex gap-4">
|
||||
<label className="flex items-center gap-2 text-sm cursor-pointer">
|
||||
<input
|
||||
type="radio"
|
||||
name="signConvention"
|
||||
value="negative_expense"
|
||||
checked={config.signConvention === "negative_expense"}
|
||||
onChange={() =>
|
||||
updateConfig({ signConvention: "negative_expense" })
|
||||
}
|
||||
className="accent-[var(--primary)]"
|
||||
/>
|
||||
{t("import.config.negativeExpense")}
|
||||
</label>
|
||||
<label className="flex items-center gap-2 text-sm cursor-pointer">
|
||||
<input
|
||||
type="radio"
|
||||
name="signConvention"
|
||||
value="positive_expense"
|
||||
checked={config.signConvention === "positive_expense"}
|
||||
onChange={() =>
|
||||
updateConfig({ signConvention: "positive_expense" })
|
||||
}
|
||||
className="accent-[var(--primary)]"
|
||||
/>
|
||||
{t("import.config.positiveExpense")}
|
||||
{/*
|
||||
Sign convention — single-amount mode ONLY.
|
||||
|
||||
`mapRow` computes `credit - debit` on magnitudes in debit/credit mode and
|
||||
never reads `signConvention` there; the direction is carried by which
|
||||
column holds the value. Showing the selector anyway offered a control
|
||||
that changed nothing, which reads as "the app ignored my setting".
|
||||
The stored value is left untouched: hidden, not reset.
|
||||
*/}
|
||||
{config.amountMode === "single" && (
|
||||
<div>
|
||||
<label className="block text-sm text-[var(--muted-foreground)] mb-1">
|
||||
{t("import.config.signConvention")}
|
||||
</label>
|
||||
<div className="flex gap-4">
|
||||
<label className="flex items-center gap-2 text-sm cursor-pointer">
|
||||
<input
|
||||
type="radio"
|
||||
name="signConvention"
|
||||
value="negative_expense"
|
||||
checked={config.signConvention === "negative_expense"}
|
||||
onChange={() =>
|
||||
updateConfig({ signConvention: "negative_expense" })
|
||||
}
|
||||
className="accent-[var(--primary)]"
|
||||
/>
|
||||
{t("import.config.negativeExpense")}
|
||||
</label>
|
||||
<label className="flex items-center gap-2 text-sm cursor-pointer">
|
||||
<input
|
||||
type="radio"
|
||||
name="signConvention"
|
||||
value="positive_expense"
|
||||
checked={config.signConvention === "positive_expense"}
|
||||
onChange={() =>
|
||||
updateConfig({ signConvention: "positive_expense" })
|
||||
}
|
||||
className="accent-[var(--primary)]"
|
||||
/>
|
||||
{t("import.config.positiveExpense")}
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Column mapping */}
|
||||
{headers.length > 0 && (
|
||||
|
|
@ -314,8 +399,8 @@ export default function SourceConfigPanel({
|
|||
onMappingChange={(mapping: ColumnMapping) =>
|
||||
onConfigChange({ ...config, columnMapping: mapping })
|
||||
}
|
||||
onAmountModeChange={(mode: AmountMode) =>
|
||||
onConfigChange({ ...config, amountMode: mode })
|
||||
onAmountModeChange={(mode: AmountMode, mapping: ColumnMapping) =>
|
||||
onConfigChange({ ...config, amountMode: mode, columnMapping: mapping })
|
||||
}
|
||||
/>
|
||||
)}
|
||||
|
|
|
|||
|
|
@ -69,6 +69,8 @@ const SOURCE_CHEQUING: ImportSource = {
|
|||
column_mapping: "{}",
|
||||
skip_lines: 0,
|
||||
has_header: true,
|
||||
amount_mode: "single",
|
||||
sign_convention: "negative_expense",
|
||||
created_at: "2026-01-01",
|
||||
updated_at: "2026-01-01",
|
||||
};
|
||||
|
|
|
|||
|
|
@ -69,6 +69,24 @@ export default function ImportConfirmModal({
|
|||
);
|
||||
}
|
||||
|
||||
// The import configurations travel with both transaction modes (#331). They
|
||||
// are listed so the user sees that a restore now brings its sources back
|
||||
// rather than leaving every one of them to reconfigure.
|
||||
if (importType !== "categories_only") {
|
||||
if (summary.importSourcesCount > 0)
|
||||
willImport.push(
|
||||
t("settings.dataManagement.import.countImportSources", {
|
||||
count: summary.importSourcesCount,
|
||||
})
|
||||
);
|
||||
if (summary.importTemplatesCount > 0)
|
||||
willImport.push(
|
||||
t("settings.dataManagement.import.countImportTemplates", {
|
||||
count: summary.importTemplatesCount,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
const confirmWord = t("settings.dataManagement.import.confirmWord");
|
||||
const canConfirm = confirmText === confirmWord && !isImporting;
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,8 @@ import {
|
|||
getExportSuppliers,
|
||||
getExportKeywords,
|
||||
getExportTransactions,
|
||||
getExportImportSources,
|
||||
getExportImportTemplates,
|
||||
serializeToJson,
|
||||
serializeTransactionsToCsv,
|
||||
type ExportMode,
|
||||
|
|
@ -61,6 +63,11 @@ export function useDataExport() {
|
|||
}
|
||||
if (mode === "transactions_with_categories" || mode === "transactions_only") {
|
||||
data.transactions = await getExportTransactions();
|
||||
// The import configurations travel with the transaction modes — the
|
||||
// two the restore wipes `import_sources` for. A categories-only
|
||||
// backup neither carries them nor touches them (#331).
|
||||
data.import_sources = await getExportImportSources();
|
||||
data.import_config_templates = await getExportImportTemplates();
|
||||
}
|
||||
|
||||
// Serialize
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { useReducer, useCallback } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
import {
|
||||
parseImportedJson,
|
||||
|
|
@ -6,6 +7,7 @@ import {
|
|||
importCategoriesOnly,
|
||||
importTransactionsWithCategories,
|
||||
importTransactionsOnly,
|
||||
SrefValidationError,
|
||||
type ExportEnvelope,
|
||||
type ImportSummary,
|
||||
} from "../services/dataExportService";
|
||||
|
|
@ -108,6 +110,20 @@ function parseContent(
|
|||
|
||||
export function useDataImport() {
|
||||
const [state, dispatch] = useReducer(reducer, initialState);
|
||||
const { t } = useTranslation();
|
||||
|
||||
/**
|
||||
* A refusal at the configuration boundary carries an i18n key and the name of
|
||||
* the offending source; everything else is a raw exception message. The card
|
||||
* renders this string as it comes, so the translation happens here.
|
||||
*/
|
||||
const describeError = useCallback(
|
||||
(e: unknown): string => {
|
||||
if (e instanceof SrefValidationError) return t(e.i18nKey, e.params);
|
||||
return e instanceof Error ? e.message : String(e);
|
||||
},
|
||||
[t]
|
||||
);
|
||||
|
||||
const pickAndRead = useCallback(async () => {
|
||||
dispatch({ type: "READ_START" });
|
||||
|
|
@ -136,12 +152,9 @@ export function useDataImport() {
|
|||
const { summary, data, importType } = parseContent(content, filePath);
|
||||
dispatch({ type: "CONFIRMING", filePath, summary, data, importType });
|
||||
} catch (e) {
|
||||
dispatch({
|
||||
type: "IMPORT_ERROR",
|
||||
error: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
dispatch({ type: "IMPORT_ERROR", error: describeError(e) });
|
||||
}
|
||||
}, []);
|
||||
}, [describeError]);
|
||||
|
||||
const readWithPassword = useCallback(
|
||||
async (password: string) => {
|
||||
|
|
@ -156,13 +169,10 @@ export function useDataImport() {
|
|||
const { summary, data, importType } = parseContent(content, state.filePath);
|
||||
dispatch({ type: "CONFIRMING", filePath: state.filePath, summary, data, importType });
|
||||
} catch (e) {
|
||||
dispatch({
|
||||
type: "IMPORT_ERROR",
|
||||
error: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
dispatch({ type: "IMPORT_ERROR", error: describeError(e) });
|
||||
}
|
||||
},
|
||||
[state.filePath]
|
||||
[state.filePath, describeError]
|
||||
);
|
||||
|
||||
const executeImport = useCallback(async () => {
|
||||
|
|
@ -183,12 +193,9 @@ export function useDataImport() {
|
|||
}
|
||||
dispatch({ type: "IMPORT_SUCCESS" });
|
||||
} catch (e) {
|
||||
dispatch({
|
||||
type: "IMPORT_ERROR",
|
||||
error: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
dispatch({ type: "IMPORT_ERROR", error: describeError(e) });
|
||||
}
|
||||
}, [state.parsedData, state.importType, state.filePath]);
|
||||
}, [state.parsedData, state.importType, state.filePath, describeError]);
|
||||
|
||||
const reset = useCallback(() => dispatch({ type: "RESET" }), []);
|
||||
|
||||
|
|
|
|||
|
|
@ -11,7 +11,6 @@ import type {
|
|||
ImportReport,
|
||||
ImportSource,
|
||||
ImportConfigTemplate,
|
||||
ColumnMapping,
|
||||
} from "../shared/types";
|
||||
import {
|
||||
getImportFolder,
|
||||
|
|
@ -40,12 +39,32 @@ import {
|
|||
updateTemplate,
|
||||
deleteTemplate as deleteTemplateService,
|
||||
} from "../services/importConfigTemplateService";
|
||||
import { parseDate } from "../utils/dateParser";
|
||||
import { parseFrenchAmount } from "../utils/amountParser";
|
||||
import {
|
||||
preprocessQuotedCSV,
|
||||
autoDetectConfig as runAutoDetect,
|
||||
detectImportFormat as runAutoDetect,
|
||||
type DetectionScore,
|
||||
} from "../utils/csvAutoDetect";
|
||||
import {
|
||||
buildHeaderSignature,
|
||||
detectHeaderDrift,
|
||||
type BankSignatureId,
|
||||
type HeaderDriftEntry,
|
||||
} from "../utils/bankSignatures";
|
||||
import {
|
||||
detectAmountSeparators,
|
||||
flipSignFormat,
|
||||
formatFromRow,
|
||||
formatToRow,
|
||||
ImportFormatError,
|
||||
mapRow,
|
||||
ROW_ERROR_KEYS,
|
||||
} from "../utils/importFormat";
|
||||
|
||||
/** Error text for the banner: an i18n key when we have one, the raw message otherwise. */
|
||||
function errorMessage(e: unknown): string {
|
||||
if (e instanceof ImportFormatError) return e.i18nKey;
|
||||
return e instanceof Error ? e.message : String(e);
|
||||
}
|
||||
|
||||
interface WizardState {
|
||||
step: ImportWizardStep;
|
||||
|
|
@ -67,6 +86,29 @@ interface WizardState {
|
|||
importedFilesBySource: Map<string, Set<string>>;
|
||||
configTemplates: ImportConfigTemplate[];
|
||||
selectedTemplateId: number | null;
|
||||
/**
|
||||
* Result of the LAST detection run on the selected source, or null when none
|
||||
* ran — a source opened on its stored format, which is never re-detected.
|
||||
*/
|
||||
detectionScore: DetectionScore | null;
|
||||
/**
|
||||
* Bank whose documented layout the last detection recognised (#330). Tied to
|
||||
* `detectionScore`: it is set with it and cleared with it, so a badge naming
|
||||
* a bank can never outlive the detection that named it.
|
||||
*/
|
||||
detectedBank: BankSignatureId | null;
|
||||
/**
|
||||
* How the header row of the files being imported differs from the one the
|
||||
* source recorded at its last successful import — null when there is nothing
|
||||
* to report, which includes every source that has no stored signature.
|
||||
*/
|
||||
formatDrift: HeaderDriftEntry[] | null;
|
||||
/**
|
||||
* The configuration re-detected on the drifted file, offered by the panel as
|
||||
* "adopt". Null when detection could not read the new shape either: the drift
|
||||
* is still worth reporting, there is just nothing to adopt.
|
||||
*/
|
||||
driftConfig: SourceConfig | null;
|
||||
}
|
||||
|
||||
type WizardAction =
|
||||
|
|
@ -88,6 +130,12 @@ type WizardAction =
|
|||
| { type: "SET_CONFIGURED_SOURCES"; payload: { names: Set<string>; files: Map<string, Set<string>> } }
|
||||
| { type: "SET_CONFIG_TEMPLATES"; payload: ImportConfigTemplate[] }
|
||||
| { type: "SET_SELECTED_TEMPLATE_ID"; payload: number | null }
|
||||
| { type: "SET_DETECTION_SCORE"; payload: DetectionScore | null }
|
||||
| { type: "SET_DETECTED_BANK"; payload: BankSignatureId | null }
|
||||
| {
|
||||
type: "SET_FORMAT_DRIFT";
|
||||
payload: { entries: HeaderDriftEntry[]; config: SourceConfig | null } | null;
|
||||
}
|
||||
| { type: "RESET" };
|
||||
|
||||
const defaultConfig: SourceConfig = {
|
||||
|
|
@ -122,6 +170,10 @@ const initialState: WizardState = {
|
|||
importedFilesBySource: new Map(),
|
||||
configTemplates: [],
|
||||
selectedTemplateId: null,
|
||||
detectionScore: null,
|
||||
detectedBank: null,
|
||||
formatDrift: null,
|
||||
driftConfig: null,
|
||||
};
|
||||
|
||||
function reducer(state: WizardState, action: WizardAction): WizardState {
|
||||
|
|
@ -137,7 +189,19 @@ function reducer(state: WizardState, action: WizardAction): WizardState {
|
|||
case "SET_SCANNED_SOURCES":
|
||||
return { ...state, scannedSources: action.payload, isLoading: false };
|
||||
case "SET_SELECTED_SOURCE":
|
||||
return { ...state, selectedSource: action.payload };
|
||||
// The score belongs to the source it was measured on. Carrying it over
|
||||
// would let a source opened on its stored format inherit the confidence
|
||||
// of the previous one, which is worse than showing nothing. The bank
|
||||
// badge and the drift report are scoped the same way, for the same
|
||||
// reason — both describe one source's files.
|
||||
return {
|
||||
...state,
|
||||
selectedSource: action.payload,
|
||||
detectionScore: null,
|
||||
detectedBank: null,
|
||||
formatDrift: null,
|
||||
driftConfig: null,
|
||||
};
|
||||
case "SET_SELECTED_FILES":
|
||||
return { ...state, selectedFiles: action.payload };
|
||||
case "SET_SOURCE_CONFIG":
|
||||
|
|
@ -188,6 +252,25 @@ function reducer(state: WizardState, action: WizardAction): WizardState {
|
|||
return { ...state, configTemplates: action.payload };
|
||||
case "SET_SELECTED_TEMPLATE_ID":
|
||||
return { ...state, selectedTemplateId: action.payload };
|
||||
case "SET_DETECTION_SCORE":
|
||||
// Clearing the score clears the bank with it, in ONE place: the three
|
||||
// sites that drop the score (a hand edit, a template, a sign flip) then
|
||||
// cannot leave "Format Desjardins reconnu" standing over a format
|
||||
// detection never produced. A bank is only ever set by dispatching
|
||||
// `SET_DETECTED_BANK` right after a non-null score.
|
||||
return {
|
||||
...state,
|
||||
detectionScore: action.payload,
|
||||
detectedBank: action.payload === null ? null : state.detectedBank,
|
||||
};
|
||||
case "SET_DETECTED_BANK":
|
||||
return { ...state, detectedBank: action.payload };
|
||||
case "SET_FORMAT_DRIFT":
|
||||
return {
|
||||
...state,
|
||||
formatDrift: action.payload?.entries ?? null,
|
||||
driftConfig: action.payload?.config ?? null,
|
||||
};
|
||||
case "RESET":
|
||||
return {
|
||||
...initialState,
|
||||
|
|
@ -280,8 +363,62 @@ export function useImportWizard() {
|
|||
}
|
||||
}, [state.importFolder, scanFolderInternal]);
|
||||
|
||||
/**
|
||||
* Run detection on one file and report the outcome. Returns the detected
|
||||
* format merged onto `base`, or null when detection produced no usable
|
||||
* configuration (the reason is dispatched as the page error).
|
||||
*
|
||||
* THE single detection path, shared by the two things that start one: the
|
||||
* magic-wand button and the automatic run on a source that has never been
|
||||
* configured. Everything it needs is a parameter, so it never reads a piece of
|
||||
* `state` its caller has just dispatched but not yet re-rendered — which is
|
||||
* exactly why the automatic run could not simply call the button's callback.
|
||||
*
|
||||
* `base` keeps the fields detection does not decide (the source name, the
|
||||
* encoding), and the spread of `outcome.config` writes every field it does —
|
||||
* naming them one by one is how a field gets silently dropped (#324).
|
||||
*/
|
||||
const detectFormatForFile = useCallback(
|
||||
async (filePath: string, base: SourceConfig): Promise<SourceConfig | null> => {
|
||||
const content = await invoke<string>("read_file_content", {
|
||||
filePath,
|
||||
encoding: base.encoding,
|
||||
});
|
||||
|
||||
const outcome = runAutoDetect(content);
|
||||
|
||||
if (outcome.status !== "ok") {
|
||||
// A refused format states WHY (#327): "I cannot read this file" and
|
||||
// "this file is a shape this version would import backwards" are not
|
||||
// the same news, and only the second one is the app's own limitation.
|
||||
// `SET_ERROR` clears the loading flag on its own.
|
||||
dispatch({ type: "SET_DETECTION_SCORE", payload: null });
|
||||
dispatch({
|
||||
type: "SET_ERROR",
|
||||
payload:
|
||||
outcome.status === "rejected"
|
||||
? outcome.reason
|
||||
: "import.errors.autoDetectFailed",
|
||||
});
|
||||
return null;
|
||||
}
|
||||
|
||||
dispatch({ type: "SET_DETECTION_SCORE", payload: outcome.score });
|
||||
// Always dispatched, including the `null` of an unknown file: a bank left
|
||||
// over from the previously detected file would name the wrong bank.
|
||||
dispatch({ type: "SET_DETECTED_BANK", payload: outcome.bank });
|
||||
return { ...base, ...outcome.config };
|
||||
},
|
||||
[]
|
||||
);
|
||||
|
||||
const selectSource = useCallback(
|
||||
async (source: ScannedSource) => {
|
||||
// The banner is page-wide: an error left by the previous source must not
|
||||
// survive into this one. Cleared here, before anything that may set a new
|
||||
// one — a stored format that will not decode, a detection that refuses.
|
||||
dispatch({ type: "SET_ERROR", payload: null });
|
||||
|
||||
// Sort files: new files first, then already-imported
|
||||
const importedNames = state.importedFilesBySource.get(source.folder_name);
|
||||
const sorted = [...source.files].sort((a, b) => {
|
||||
|
|
@ -297,71 +434,106 @@ export function useImportWizard() {
|
|||
|
||||
dispatch({ type: "SET_SELECTED_SOURCE", payload: sortedSource });
|
||||
dispatch({ type: "SET_SELECTED_FILES", payload: newFiles });
|
||||
dispatch({ type: "SET_SELECTED_TEMPLATE_ID", payload: null });
|
||||
|
||||
// Check if this source already has config in DB
|
||||
const existing = await getSourceByName(source.folder_name);
|
||||
dispatch({ type: "SET_EXISTING_SOURCE", payload: existing });
|
||||
try {
|
||||
// Check if this source already has config in DB
|
||||
const existing = await getSourceByName(source.folder_name);
|
||||
dispatch({ type: "SET_EXISTING_SOURCE", payload: existing });
|
||||
|
||||
let activeDelimiter = defaultConfig.delimiter;
|
||||
let activeEncoding = "utf-8";
|
||||
let activeSkipLines = 0;
|
||||
let activeHasHeader = true;
|
||||
// Provenance of the stored format, not the format itself: the template
|
||||
// is never re-read, it is only shown as the source's origin. Blindly
|
||||
// resetting it to null used to make an already-linked source look
|
||||
// unconfigured.
|
||||
dispatch({
|
||||
type: "SET_SELECTED_TEMPLATE_ID",
|
||||
payload: existing?.template_id ?? null,
|
||||
});
|
||||
|
||||
if (existing) {
|
||||
// Restore config from DB
|
||||
const mapping = JSON.parse(existing.column_mapping) as ColumnMapping;
|
||||
const config: SourceConfig = {
|
||||
name: existing.name,
|
||||
delimiter: existing.delimiter,
|
||||
encoding: existing.encoding,
|
||||
dateFormat: existing.date_format,
|
||||
skipLines: existing.skip_lines,
|
||||
columnMapping: mapping,
|
||||
amountMode:
|
||||
mapping.debitAmount !== undefined ? "debit_credit" : "single",
|
||||
signConvention: "negative_expense",
|
||||
hasHeader: !!existing.has_header,
|
||||
};
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: config });
|
||||
activeDelimiter = existing.delimiter;
|
||||
activeEncoding = existing.encoding;
|
||||
activeSkipLines = existing.skip_lines;
|
||||
activeHasHeader = !!existing.has_header;
|
||||
} else {
|
||||
// Auto-detect encoding for first file
|
||||
if (source.files.length > 0) {
|
||||
let activeDelimiter = defaultConfig.delimiter;
|
||||
let activeEncoding = "utf-8";
|
||||
let activeSkipLines = 0;
|
||||
let activeHasHeader = true;
|
||||
|
||||
// Restore the format as it was SAVED. Nothing is re-inferred and
|
||||
// nothing is defaulted here — the amount mode and the sign convention
|
||||
// are columns now (v17), and reading them is the whole point of #324.
|
||||
let restored: SourceConfig | null = null;
|
||||
if (existing) {
|
||||
try {
|
||||
activeEncoding = await invoke<string>("detect_encoding", {
|
||||
filePath: source.files[0].file_path,
|
||||
});
|
||||
} catch {
|
||||
// fallback to utf-8
|
||||
restored = { name: existing.name, ...formatFromRow(existing) };
|
||||
} catch (e) {
|
||||
// A stored format we cannot decode is REPORTED, not quietly swapped
|
||||
// for a plausible default — that substitution is how amounts got
|
||||
// flipped. The wizard then opens on a fresh configuration, because
|
||||
// "reconfigure this source" has to be an action the user can take.
|
||||
dispatch({ type: "SET_ERROR", payload: errorMessage(e) });
|
||||
}
|
||||
}
|
||||
|
||||
dispatch({
|
||||
type: "SET_SOURCE_CONFIG",
|
||||
payload: {
|
||||
if (restored) {
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: restored });
|
||||
activeDelimiter = restored.delimiter;
|
||||
activeEncoding = restored.encoding;
|
||||
activeSkipLines = restored.skipLines;
|
||||
activeHasHeader = restored.hasHeader;
|
||||
} else {
|
||||
// Auto-detect encoding for first file
|
||||
if (source.files.length > 0) {
|
||||
try {
|
||||
activeEncoding = await invoke<string>("detect_encoding", {
|
||||
filePath: source.files[0].file_path,
|
||||
});
|
||||
} catch {
|
||||
// fallback to utf-8
|
||||
}
|
||||
}
|
||||
|
||||
const fresh: SourceConfig = {
|
||||
...defaultConfig,
|
||||
name: source.folder_name,
|
||||
encoding: activeEncoding,
|
||||
},
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
// Load preview headers from first file
|
||||
if (source.files.length > 0) {
|
||||
await loadHeadersWithConfig(
|
||||
source.files[0].file_path,
|
||||
activeDelimiter,
|
||||
activeEncoding,
|
||||
activeSkipLines,
|
||||
activeHasHeader
|
||||
);
|
||||
}
|
||||
// Detection fires on its own HERE and nowhere else: the arm reached
|
||||
// when the source has no stored format at all. `defaultConfig` is
|
||||
// `;` + `DD/MM/YYYY` + columns 0/1/2 — plausible enough to import a
|
||||
// whole file wrong rather than fail, so leaving a never-configured
|
||||
// source sitting on it is the defect (#328).
|
||||
//
|
||||
// The condition is `!existing`, not `!restored`: a source whose stored
|
||||
// format will not decode already carries an explicit error, and
|
||||
// re-detecting over it would replace that message with a silent guess.
|
||||
// Its wand button is still there.
|
||||
let active = fresh;
|
||||
if (!existing && source.files.length > 0) {
|
||||
const detected = await detectFormatForFile(
|
||||
source.files[0].file_path,
|
||||
fresh
|
||||
);
|
||||
if (detected) active = detected;
|
||||
}
|
||||
|
||||
dispatch({ type: "SET_STEP", payload: "source-config" });
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: active });
|
||||
activeDelimiter = active.delimiter;
|
||||
activeSkipLines = active.skipLines;
|
||||
activeHasHeader = active.hasHeader;
|
||||
}
|
||||
|
||||
// Load preview headers from first file
|
||||
if (source.files.length > 0) {
|
||||
await loadHeadersWithConfig(
|
||||
source.files[0].file_path,
|
||||
activeDelimiter,
|
||||
activeEncoding,
|
||||
activeSkipLines,
|
||||
activeHasHeader
|
||||
);
|
||||
}
|
||||
|
||||
dispatch({ type: "SET_STEP", payload: "source-config" });
|
||||
} catch (e) {
|
||||
dispatch({ type: "SET_ERROR", payload: errorMessage(e) });
|
||||
}
|
||||
},
|
||||
[state.importedFilesBySource] // eslint-disable-line react-hooks/exhaustive-deps
|
||||
);
|
||||
|
|
@ -408,6 +580,19 @@ export function useImportWizard() {
|
|||
(config: SourceConfig) => {
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: config });
|
||||
|
||||
// The score measures the format DETECTION produced. The moment any of
|
||||
// those eight fields is edited by hand, the number stops describing what
|
||||
// the wizard would read — a banner still claiming "147 of 150 rows" over
|
||||
// a mapping nobody measured is the same misinformation this chantier is
|
||||
// removing. Comparing through the codec keeps a rename (the ninth field,
|
||||
// which changes nothing about reading) from clearing it.
|
||||
if (
|
||||
JSON.stringify(formatToRow(config)) !==
|
||||
JSON.stringify(formatToRow(state.sourceConfig))
|
||||
) {
|
||||
dispatch({ type: "SET_DETECTION_SCORE", payload: null });
|
||||
}
|
||||
|
||||
// Reload headers when delimiter, encoding, skipLines, or hasHeader changes
|
||||
if (state.selectedFiles.length > 0) {
|
||||
loadHeadersWithConfig(
|
||||
|
|
@ -419,7 +604,7 @@ export function useImportWizard() {
|
|||
);
|
||||
}
|
||||
},
|
||||
[state.selectedFiles, loadHeadersWithConfig]
|
||||
[state.selectedFiles, state.sourceConfig, loadHeadersWithConfig]
|
||||
);
|
||||
|
||||
const toggleFile = useCallback(
|
||||
|
|
@ -457,9 +642,16 @@ export function useImportWizard() {
|
|||
}
|
||||
}, [state.selectedSource, state.importedFilesBySource]);
|
||||
|
||||
// Internal helper: parses selected files and returns rows + headers
|
||||
const parseFilesInternal = useCallback(async (): Promise<{ rows: ParsedRow[]; headers: string[] }> => {
|
||||
const config = state.sourceConfig;
|
||||
// Internal helper: parses selected files and returns rows + headers.
|
||||
//
|
||||
// `configOverride` exists for the sign flip: it re-parses under a format that
|
||||
// has just been dispatched and is therefore not yet readable in `state`.
|
||||
// Reading the stale one would redisplay the exact table the user asked to
|
||||
// correct, which reads as "the button did nothing".
|
||||
const parseFilesInternal = useCallback(async (
|
||||
configOverride?: SourceConfig
|
||||
): Promise<{ rows: ParsedRow[]; headers: string[] }> => {
|
||||
const config = configOverride ?? state.sourceConfig;
|
||||
const allRows: ParsedRow[] = [];
|
||||
let headers: string[] = [];
|
||||
|
||||
|
|
@ -486,66 +678,32 @@ export function useImportWizard() {
|
|||
headers = firstDataRow.map((_, i) => `Col ${i}`);
|
||||
}
|
||||
|
||||
// Data rows of THIS file, kept aside so the decimal separator of each
|
||||
// amount column is arbitrated over the whole column before any row is
|
||||
// read. `1.234` alone is ambiguous; the column is not.
|
||||
const dataRows: string[][] = [];
|
||||
for (let i = startIdx; i < data.length; i++) {
|
||||
const raw = data[i];
|
||||
if (raw.length <= 1 && raw[0]?.trim() === "") continue;
|
||||
dataRows.push(raw);
|
||||
}
|
||||
const decimalSeparators = detectAmountSeparators(dataRows, config);
|
||||
|
||||
for (const raw of dataRows) {
|
||||
try {
|
||||
const date = parseDate(
|
||||
raw[config.columnMapping.date]?.trim() || "",
|
||||
config.dateFormat
|
||||
allRows.push(
|
||||
mapRow(raw, config, {
|
||||
rowIndex: allRows.length,
|
||||
sourceFilename: file.filename,
|
||||
decimalSeparators,
|
||||
})
|
||||
);
|
||||
const description =
|
||||
raw[config.columnMapping.description]?.trim() || "";
|
||||
|
||||
let amount: number;
|
||||
if (config.amountMode === "debit_credit") {
|
||||
const debit = parseFrenchAmount(
|
||||
raw[config.columnMapping.debitAmount ?? 0] || ""
|
||||
);
|
||||
const credit = parseFrenchAmount(
|
||||
raw[config.columnMapping.creditAmount ?? 0] || ""
|
||||
);
|
||||
amount = isNaN(credit) ? -(isNaN(debit) ? 0 : debit) : credit;
|
||||
} else {
|
||||
amount = parseFrenchAmount(
|
||||
raw[config.columnMapping.amount ?? 0] || ""
|
||||
);
|
||||
if (config.signConvention === "positive_expense" && !isNaN(amount)) {
|
||||
amount = -amount;
|
||||
}
|
||||
}
|
||||
|
||||
if (!date) {
|
||||
allRows.push({
|
||||
rowIndex: allRows.length,
|
||||
raw,
|
||||
parsed: null,
|
||||
error: "Invalid date",
|
||||
sourceFilename: file.filename,
|
||||
});
|
||||
} else if (isNaN(amount)) {
|
||||
allRows.push({
|
||||
rowIndex: allRows.length,
|
||||
raw,
|
||||
parsed: null,
|
||||
error: "Invalid amount",
|
||||
sourceFilename: file.filename,
|
||||
});
|
||||
} else {
|
||||
allRows.push({
|
||||
rowIndex: allRows.length,
|
||||
raw,
|
||||
parsed: { date, description, amount },
|
||||
sourceFilename: file.filename,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
allRows.push({
|
||||
rowIndex: allRows.length,
|
||||
raw,
|
||||
parsed: null,
|
||||
error: "Parse error",
|
||||
error: ROW_ERROR_KEYS.parseError,
|
||||
sourceFilename: file.filename,
|
||||
});
|
||||
}
|
||||
|
|
@ -555,8 +713,15 @@ export function useImportWizard() {
|
|||
return { rows: allRows, headers };
|
||||
}, [state.selectedFiles, state.sourceConfig]);
|
||||
|
||||
// Parse files and store preview (does NOT change wizard step)
|
||||
const parsePreview = useCallback(async () => {
|
||||
// Parse the selected files and STOP at the preview.
|
||||
//
|
||||
// Every import goes through that step now (#329). It used to be an optional
|
||||
// modal nobody had to open, next to a button that went straight to the
|
||||
// duplicate check — so a file read under a wrong convention reached the
|
||||
// database without anyone ever seeing a total. The step is unconditional on
|
||||
// purpose: the detection score does not gate it, because a perfect score says
|
||||
// nothing about the sign of what was read.
|
||||
const parseAndPreview = useCallback(async () => {
|
||||
if (state.selectedFiles.length === 0) return;
|
||||
|
||||
dispatch({ type: "SET_LOADING", payload: true });
|
||||
|
|
@ -568,44 +733,69 @@ export function useImportWizard() {
|
|||
type: "SET_PARSED_PREVIEW",
|
||||
payload: result,
|
||||
});
|
||||
|
||||
// Format drift (#330). Compared HERE and not at the configuration step
|
||||
// because this is where the header of the files actually being imported
|
||||
// is known — `previewHeaders` at the configuration step is read from the
|
||||
// first file alone, under a delimiter the user may still be editing.
|
||||
//
|
||||
// `hasHeader` gates the whole thing: a headerless file has `Col 0`,
|
||||
// `Col 1` … for headers, which is not a signature and must never be
|
||||
// compared to one.
|
||||
const drift = state.sourceConfig.hasHeader
|
||||
? detectHeaderDrift(
|
||||
state.existingSource?.header_signature,
|
||||
result.headers
|
||||
)
|
||||
: null;
|
||||
|
||||
if (drift) {
|
||||
// Re-detect on the drifted file so the panel has something to offer.
|
||||
// This is the THIRD caller of the single detection path, and it goes
|
||||
// through it rather than around it on purpose: a second detector is the
|
||||
// divergence class this whole chantier removes. A file the re-detection
|
||||
// cannot read leaves `null` — the drift is still reported, with nothing
|
||||
// to adopt.
|
||||
const reDetected = await detectFormatForFile(
|
||||
state.selectedFiles[0].file_path,
|
||||
state.sourceConfig
|
||||
);
|
||||
// That run measured the RE-DETECTED format, which is not the one the
|
||||
// wizard is using — it is only what the panel offers. Leaving its score
|
||||
// and its bank badge behind would put "Format reconnu — 6 of 6 rows
|
||||
// read" next to the stored mapping the moment the user steps back to
|
||||
// the configuration. Same rule as the sign flip: a badge describes the
|
||||
// format in use or it describes nothing.
|
||||
dispatch({ type: "SET_DETECTION_SCORE", payload: null });
|
||||
dispatch({
|
||||
type: "SET_FORMAT_DRIFT",
|
||||
payload: { entries: drift, config: reDetected },
|
||||
});
|
||||
} else {
|
||||
dispatch({ type: "SET_FORMAT_DRIFT", payload: null });
|
||||
}
|
||||
|
||||
dispatch({ type: "SET_STEP", payload: "file-preview" });
|
||||
} catch (e) {
|
||||
dispatch({
|
||||
type: "SET_ERROR",
|
||||
payload: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
}
|
||||
}, [state.selectedFiles, parseFilesInternal]);
|
||||
}, [
|
||||
state.selectedFiles,
|
||||
state.sourceConfig,
|
||||
state.existingSource,
|
||||
parseFilesInternal,
|
||||
detectFormatForFile,
|
||||
]);
|
||||
|
||||
// Internal helper: runs duplicate checking against parsed rows
|
||||
// Internal helper: runs duplicate checking against parsed rows.
|
||||
//
|
||||
// Deliberately writes NOTHING. The source config used to be saved here, so an
|
||||
// import abandoned at the duplicate step still left a — possibly wrong —
|
||||
// configuration behind. It is written by `executeImport` now (#324).
|
||||
const checkDuplicatesInternal = useCallback(async (parsedRows: ParsedRow[]) => {
|
||||
// Save/update source config in DB
|
||||
const config = state.sourceConfig;
|
||||
const mappingJson = JSON.stringify(config.columnMapping);
|
||||
|
||||
let sourceId: number;
|
||||
if (state.existingSource) {
|
||||
sourceId = state.existingSource.id;
|
||||
await updateSource(sourceId, {
|
||||
name: config.name,
|
||||
delimiter: config.delimiter,
|
||||
encoding: config.encoding,
|
||||
date_format: config.dateFormat,
|
||||
column_mapping: mappingJson,
|
||||
skip_lines: config.skipLines,
|
||||
has_header: config.hasHeader,
|
||||
});
|
||||
} else {
|
||||
sourceId = await createSource({
|
||||
name: config.name,
|
||||
delimiter: config.delimiter,
|
||||
encoding: config.encoding,
|
||||
date_format: config.dateFormat,
|
||||
column_mapping: mappingJson,
|
||||
skip_lines: config.skipLines,
|
||||
has_header: config.hasHeader,
|
||||
});
|
||||
}
|
||||
|
||||
// Check file-level duplicates (check ALL selected files, not just the first)
|
||||
let fileAlreadyImported = false;
|
||||
let existingFileId: number | undefined;
|
||||
|
|
@ -679,9 +869,14 @@ export function useImportWizard() {
|
|||
},
|
||||
});
|
||||
dispatch({ type: "SET_STEP", payload: "duplicate-check" });
|
||||
}, [state.sourceConfig, state.existingSource, state.selectedFiles]);
|
||||
}, [state.selectedFiles]);
|
||||
|
||||
// Check duplicates using already-parsed preview data
|
||||
// Leave the preview for the duplicate check, on the rows it just displayed.
|
||||
//
|
||||
// Dead code until #329: the wizard jumped from the configuration straight to
|
||||
// the duplicates, re-parsing on the way. It is the preview step's "next"
|
||||
// button now, so the rows the user validated are the rows that get checked —
|
||||
// no second parse can quietly read the file differently in between.
|
||||
const checkDuplicates = useCallback(async () => {
|
||||
dispatch({ type: "SET_LOADING", payload: true });
|
||||
dispatch({ type: "SET_ERROR", payload: null });
|
||||
|
|
@ -696,27 +891,45 @@ export function useImportWizard() {
|
|||
}
|
||||
}, [state.parsedPreview, checkDuplicatesInternal]);
|
||||
|
||||
// Parse files then check duplicates in one step (skips preview step)
|
||||
const parseAndCheckDuplicates = useCallback(async () => {
|
||||
if (state.selectedFiles.length === 0) return;
|
||||
/**
|
||||
* Read the file the other way round, and remember it.
|
||||
*
|
||||
* The correction lands on the CONFIGURATION (`flipSignFormat`, which knows
|
||||
* that debit/credit flips by swapping columns rather than by toggling a
|
||||
* convention `mapRow` ignores there), so it is persisted with the source at
|
||||
* import time and the next file from that bank reads right on its own.
|
||||
*
|
||||
* Headers are deliberately NOT reloaded: a flip touches neither delimiter,
|
||||
* encoding, skipped lines nor header flag, and `loadHeadersWithConfig`
|
||||
* dispatches an empty row list — racing it against this re-parse is a coin
|
||||
* toss between the corrected table and a blank one.
|
||||
*/
|
||||
const flipSignConvention = useCallback(async () => {
|
||||
const flipped: SourceConfig = {
|
||||
...state.sourceConfig,
|
||||
...flipSignFormat(state.sourceConfig),
|
||||
};
|
||||
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: flipped });
|
||||
// Detection did not produce this format. Keeping its badge would vouch for
|
||||
// a configuration it never measured — the same rule as a manual edit.
|
||||
dispatch({ type: "SET_DETECTION_SCORE", payload: null });
|
||||
dispatch({ type: "SET_LOADING", payload: true });
|
||||
dispatch({ type: "SET_ERROR", payload: null });
|
||||
|
||||
try {
|
||||
const result = await parseFilesInternal();
|
||||
const result = await parseFilesInternal(flipped);
|
||||
dispatch({
|
||||
type: "SET_PARSED_PREVIEW",
|
||||
payload: result,
|
||||
});
|
||||
await checkDuplicatesInternal(result.rows);
|
||||
} catch (e) {
|
||||
dispatch({
|
||||
type: "SET_ERROR",
|
||||
payload: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
}
|
||||
}, [state.selectedFiles, parseFilesInternal, checkDuplicatesInternal]);
|
||||
}, [state.sourceConfig, parseFilesInternal]);
|
||||
|
||||
const executeImport = useCallback(async () => {
|
||||
if (!state.duplicateResult) return;
|
||||
|
|
@ -727,10 +940,39 @@ export function useImportWizard() {
|
|||
try {
|
||||
const config = state.sourceConfig;
|
||||
|
||||
// Get or create source ID
|
||||
const dbSource = await getSourceByName(config.name);
|
||||
if (!dbSource) throw new Error("Source not found in database");
|
||||
const sourceId = dbSource.id;
|
||||
// Persist the format — the ONLY write point. It happens here rather than
|
||||
// at the duplicate step so an abandoned import leaves no configuration
|
||||
// behind, and it goes through `formatToRow` so no field can be dropped.
|
||||
// It has to precede the file records, which carry a `source_id` FK.
|
||||
const formatRow = formatToRow(config);
|
||||
|
||||
// The header row this import actually read, recorded so the NEXT one can
|
||||
// be compared against it (#330). It is not a format field and does not go
|
||||
// through the codec: it says what the file looked like, not how to read
|
||||
// it. A headerless file writes null — `previewHeaders` holds `Col 0`,
|
||||
// `Col 1` … there, and storing that would make every later import look
|
||||
// like drift.
|
||||
const headerSignature = buildHeaderSignature(
|
||||
config.hasHeader ? state.previewHeaders : null
|
||||
);
|
||||
|
||||
let sourceId: number;
|
||||
if (state.existingSource) {
|
||||
sourceId = state.existingSource.id;
|
||||
await updateSource(sourceId, {
|
||||
name: config.name,
|
||||
...formatRow,
|
||||
header_signature: headerSignature,
|
||||
template_id: state.selectedTemplateId,
|
||||
});
|
||||
} else {
|
||||
sourceId = await createSource({
|
||||
name: config.name,
|
||||
...formatRow,
|
||||
header_signature: headerSignature,
|
||||
template_id: state.selectedTemplateId,
|
||||
});
|
||||
}
|
||||
|
||||
// Determine rows to import: new rows + non-excluded duplicates
|
||||
const includedDuplicates = state.duplicateResult.duplicateRows
|
||||
|
|
@ -826,7 +1068,10 @@ export function useImportWizard() {
|
|||
// Count errors from parsing
|
||||
const parseErrors = state.parsedPreview.filter((r) => r.error);
|
||||
for (const err of parseErrors) {
|
||||
errors.push({ rowIndex: err.rowIndex, message: err.error || "Parse error" });
|
||||
errors.push({
|
||||
rowIndex: err.rowIndex,
|
||||
message: err.error || ROW_ERROR_KEYS.parseError,
|
||||
});
|
||||
}
|
||||
|
||||
const report: ImportReport = {
|
||||
|
|
@ -854,6 +1099,8 @@ export function useImportWizard() {
|
|||
}, [
|
||||
state.duplicateResult,
|
||||
state.sourceConfig,
|
||||
state.existingSource,
|
||||
state.selectedTemplateId,
|
||||
state.excludedDuplicateIndices,
|
||||
state.parsedPreview,
|
||||
state.selectedFiles,
|
||||
|
|
@ -868,6 +1115,50 @@ export function useImportWizard() {
|
|||
dispatch({ type: "RESET" });
|
||||
}, []);
|
||||
|
||||
/**
|
||||
* Take the format re-detected on the drifted file, and re-read the preview
|
||||
* under it (#330).
|
||||
*
|
||||
* Same shape as the sign flip, and for the same reason: the adopted format is
|
||||
* PASSED to the parse instead of being read back from `state`, which has not
|
||||
* re-rendered yet. Reading the stale one would redisplay the table the user
|
||||
* just chose to replace.
|
||||
*
|
||||
* Nothing is written here. The adopted format reaches `import_sources` at
|
||||
* `executeImport` like every other configuration — an import abandoned on
|
||||
* this screen leaves the stored format exactly as it was.
|
||||
*/
|
||||
const adoptDriftFormat = useCallback(async () => {
|
||||
const adopted = state.driftConfig;
|
||||
if (!adopted) return;
|
||||
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: adopted });
|
||||
dispatch({ type: "SET_FORMAT_DRIFT", payload: null });
|
||||
dispatch({ type: "SET_LOADING", payload: true });
|
||||
dispatch({ type: "SET_ERROR", payload: null });
|
||||
|
||||
try {
|
||||
const result = await parseFilesInternal(adopted);
|
||||
dispatch({ type: "SET_PARSED_PREVIEW", payload: result });
|
||||
} catch (e) {
|
||||
dispatch({
|
||||
type: "SET_ERROR",
|
||||
payload: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
}
|
||||
}, [state.driftConfig, parseFilesInternal]);
|
||||
|
||||
/**
|
||||
* Keep the stored configuration and dismiss the panel.
|
||||
*
|
||||
* The rows on screen were already parsed under that configuration, so there
|
||||
* is nothing to re-read. The stored signature is still refreshed at import
|
||||
* time: the new header IS what this import read, whichever format read it.
|
||||
*/
|
||||
const keepCurrentFormat = useCallback(() => {
|
||||
dispatch({ type: "SET_FORMAT_DRIFT", payload: null });
|
||||
}, []);
|
||||
|
||||
const autoDetectConfig = useCallback(async () => {
|
||||
if (state.selectedFiles.length === 0) return;
|
||||
|
||||
|
|
@ -875,62 +1166,38 @@ export function useImportWizard() {
|
|||
dispatch({ type: "SET_ERROR", payload: null });
|
||||
|
||||
try {
|
||||
const content = await invoke<string>("read_file_content", {
|
||||
filePath: state.selectedFiles[0].file_path,
|
||||
encoding: state.sourceConfig.encoding,
|
||||
});
|
||||
const filePath = state.selectedFiles[0].file_path;
|
||||
// Same path the automatic run takes, so the button REPLAYS detection
|
||||
// rather than running a second, drifting variant of it.
|
||||
const newConfig = await detectFormatForFile(filePath, state.sourceConfig);
|
||||
if (!newConfig) return; // reason already dispatched, loading already off
|
||||
|
||||
const result = runAutoDetect(content);
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: newConfig });
|
||||
dispatch({ type: "SET_LOADING", payload: false });
|
||||
|
||||
if (result) {
|
||||
const newConfig = {
|
||||
...state.sourceConfig,
|
||||
delimiter: result.delimiter,
|
||||
hasHeader: result.hasHeader,
|
||||
skipLines: result.skipLines,
|
||||
dateFormat: result.dateFormat,
|
||||
columnMapping: result.columnMapping,
|
||||
amountMode: result.amountMode,
|
||||
signConvention: result.signConvention,
|
||||
};
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: newConfig });
|
||||
dispatch({ type: "SET_LOADING", payload: false });
|
||||
|
||||
// Refresh column headers with new config
|
||||
await loadHeadersWithConfig(
|
||||
state.selectedFiles[0].file_path,
|
||||
newConfig.delimiter,
|
||||
newConfig.encoding,
|
||||
newConfig.skipLines,
|
||||
newConfig.hasHeader
|
||||
);
|
||||
} else {
|
||||
dispatch({
|
||||
type: "SET_ERROR",
|
||||
payload: "Auto-detection failed. Please configure manually.",
|
||||
});
|
||||
}
|
||||
// Refresh column headers with new config
|
||||
await loadHeadersWithConfig(
|
||||
filePath,
|
||||
newConfig.delimiter,
|
||||
newConfig.encoding,
|
||||
newConfig.skipLines,
|
||||
newConfig.hasHeader
|
||||
);
|
||||
} catch (e) {
|
||||
dispatch({
|
||||
type: "SET_ERROR",
|
||||
payload: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
}
|
||||
}, [state.selectedFiles, state.sourceConfig, loadHeadersWithConfig]);
|
||||
}, [
|
||||
state.selectedFiles,
|
||||
state.sourceConfig,
|
||||
detectFormatForFile,
|
||||
loadHeadersWithConfig,
|
||||
]);
|
||||
|
||||
const saveConfigAsTemplate = useCallback(async (name: string) => {
|
||||
const config = state.sourceConfig;
|
||||
await createTemplate({
|
||||
name,
|
||||
delimiter: config.delimiter,
|
||||
encoding: config.encoding,
|
||||
date_format: config.dateFormat,
|
||||
skip_lines: config.skipLines,
|
||||
has_header: config.hasHeader ? 1 : 0,
|
||||
column_mapping: JSON.stringify(config.columnMapping),
|
||||
amount_mode: config.amountMode,
|
||||
sign_convention: config.signConvention,
|
||||
});
|
||||
await createTemplate({ name, ...formatToRow(state.sourceConfig) });
|
||||
const templates = await getAllTemplates();
|
||||
dispatch({ type: "SET_CONFIG_TEMPLATES", payload: templates });
|
||||
}, [state.sourceConfig]);
|
||||
|
|
@ -938,20 +1205,25 @@ export function useImportWizard() {
|
|||
const applyConfigTemplate = useCallback((templateId: number) => {
|
||||
const template = state.configTemplates.find((t) => t.id === templateId);
|
||||
if (!template) return;
|
||||
const mapping = JSON.parse(template.column_mapping) as ColumnMapping;
|
||||
const newConfig: SourceConfig = {
|
||||
name: state.sourceConfig.name,
|
||||
delimiter: template.delimiter,
|
||||
encoding: template.encoding,
|
||||
dateFormat: template.date_format,
|
||||
skipLines: template.skip_lines,
|
||||
columnMapping: mapping,
|
||||
amountMode: template.amount_mode,
|
||||
signConvention: template.sign_convention,
|
||||
hasHeader: !!template.has_header,
|
||||
};
|
||||
|
||||
let newConfig: SourceConfig;
|
||||
try {
|
||||
newConfig = {
|
||||
name: state.sourceConfig.name,
|
||||
...formatFromRow(template),
|
||||
};
|
||||
} catch (e) {
|
||||
dispatch({ type: "SET_ERROR", payload: errorMessage(e) });
|
||||
return;
|
||||
}
|
||||
|
||||
dispatch({ type: "SET_SOURCE_CONFIG", payload: newConfig });
|
||||
// Applying a template COPIES its format onto the source. The id recorded
|
||||
// here is provenance only — the copy is what the next import reads.
|
||||
dispatch({ type: "SET_SELECTED_TEMPLATE_ID", payload: templateId });
|
||||
// The format is the template's now, so a score measured on the detected
|
||||
// one would be vouching for a configuration it never saw.
|
||||
dispatch({ type: "SET_DETECTION_SCORE", payload: null });
|
||||
|
||||
// Reload headers with new config
|
||||
if (state.selectedFiles.length > 0) {
|
||||
|
|
@ -969,17 +1241,11 @@ export function useImportWizard() {
|
|||
if (!state.selectedTemplateId) return;
|
||||
const template = state.configTemplates.find((t) => t.id === state.selectedTemplateId);
|
||||
if (!template) return;
|
||||
const config = state.sourceConfig;
|
||||
// Writes to `import_config_templates` only: a source configured from this
|
||||
// template keeps its own eight columns and is untouched.
|
||||
await updateTemplate(state.selectedTemplateId, {
|
||||
name: template.name,
|
||||
delimiter: config.delimiter,
|
||||
encoding: config.encoding,
|
||||
date_format: config.dateFormat,
|
||||
skip_lines: config.skipLines,
|
||||
has_header: config.hasHeader ? 1 : 0,
|
||||
column_mapping: JSON.stringify(config.columnMapping),
|
||||
amount_mode: config.amountMode,
|
||||
sign_convention: config.signConvention,
|
||||
...formatToRow(state.sourceConfig),
|
||||
});
|
||||
const templates = await getAllTemplates();
|
||||
dispatch({ type: "SET_CONFIG_TEMPLATES", payload: templates });
|
||||
|
|
@ -1002,12 +1268,14 @@ export function useImportWizard() {
|
|||
updateConfig,
|
||||
toggleFile,
|
||||
selectAllFiles,
|
||||
parsePreview,
|
||||
parseAndPreview,
|
||||
checkDuplicates,
|
||||
parseAndCheckDuplicates,
|
||||
flipSignConvention,
|
||||
executeImport,
|
||||
goToStep,
|
||||
reset,
|
||||
adoptDriftFormat,
|
||||
keepCurrentFormat,
|
||||
autoDetectConfig,
|
||||
saveConfigAsTemplate,
|
||||
applyConfigTemplate,
|
||||
|
|
|
|||
|
|
@ -161,7 +161,9 @@ export function holdingsFromServiceHoldings(
|
|||
* column indices from `analyzeHoldingsCsv`. Behavior:
|
||||
* - Symbols are normalized (UPPER/TRIM) like manual entry (SecurityPicker) so
|
||||
* an imported title collapses onto the same `balance_securities` row.
|
||||
* - Numbers are parsed with `parseFrenchAmount` (handles `1 234,56`, `1,234.56`).
|
||||
* - Numbers are parsed with `parseFrenchAmount` (handles `1 234,56`, `1,234.56`,
|
||||
* `(140,10)`); a cell it cannot read is left EMPTY, or kept verbatim for the
|
||||
* quantity, so validation refuses the row instead of storing a wrong number.
|
||||
* - unit_price + book_cost are OPTIONAL: when the mapping's column is null (no
|
||||
* price/cost column) the field stays empty; the user fetches/types it later.
|
||||
* - Duplicate symbols WITHIN the CSV are merged into one draft to respect the
|
||||
|
|
@ -179,7 +181,13 @@ export function holdingsFromCsvRows(
|
|||
const order: string[] = [];
|
||||
const bySymbol = new Map<
|
||||
string,
|
||||
{ symbol: string; qty: number; book: number | null; price: string }
|
||||
{
|
||||
symbol: string;
|
||||
qty: number | null;
|
||||
qtyRaw: string;
|
||||
book: number | null;
|
||||
price: string;
|
||||
}
|
||||
>();
|
||||
|
||||
for (const row of rows) {
|
||||
|
|
@ -188,8 +196,14 @@ export function holdingsFromCsvRows(
|
|||
const symbol = normalizeSecuritySymbol(rawSymbol);
|
||||
if (!symbol) continue;
|
||||
|
||||
const qtyParsed = parseFrenchAmount((row[mapping.quantity] ?? "").trim());
|
||||
const qty = isNaN(qtyParsed) ? 0 : qtyParsed;
|
||||
// An unreadable quantity used to be coerced to 0, which SAVED a
|
||||
// zero-value position without a word (#325). It stays `null` now and the
|
||||
// draft keeps the offending text, so `buildDetailedLines` raises the
|
||||
// existing `snapshot_priced_quantity_required` and the user sees the cell
|
||||
// to correct.
|
||||
const qtyRaw = (row[mapping.quantity] ?? "").trim();
|
||||
const qtyParsed = parseFrenchAmount(qtyRaw);
|
||||
const qty = isNaN(qtyParsed) ? null : qtyParsed;
|
||||
|
||||
let price = "";
|
||||
if (mapping.unit_price !== null) {
|
||||
|
|
@ -205,12 +219,25 @@ export function holdingsFromCsvRows(
|
|||
|
||||
const existing = bySymbol.get(symbol);
|
||||
if (existing) {
|
||||
existing.qty += qty;
|
||||
// One unreadable lot taints the merged quantity: summing it as 0 would
|
||||
// hide the bad cell behind a plausible total.
|
||||
if (existing.qty === null || qty === null) {
|
||||
existing.qty = null;
|
||||
if (!existing.qtyRaw) existing.qtyRaw = qtyRaw;
|
||||
} else {
|
||||
existing.qty += qty;
|
||||
}
|
||||
if (book !== null) existing.book = (existing.book ?? 0) + book;
|
||||
if (!existing.price && price) existing.price = price; // first non-empty
|
||||
} else {
|
||||
order.push(symbol);
|
||||
bySymbol.set(symbol, { symbol, qty, book, price });
|
||||
bySymbol.set(symbol, {
|
||||
symbol,
|
||||
qty,
|
||||
qtyRaw: qty === null ? qtyRaw : "",
|
||||
book,
|
||||
price,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -219,7 +246,7 @@ export function holdingsFromCsvRows(
|
|||
return {
|
||||
...makeEmptyHolding(defaultAssetType),
|
||||
symbol: a.symbol,
|
||||
quantity: String(a.qty),
|
||||
quantity: a.qty !== null ? String(a.qty) : a.qtyRaw,
|
||||
unit_price: a.price,
|
||||
book_cost: a.book !== null ? String(a.book) : "",
|
||||
};
|
||||
|
|
|
|||
|
|
@ -113,7 +113,27 @@
|
|||
"templateSaved": "Template saved",
|
||||
"deleteTemplate": "Delete template",
|
||||
"noTemplates": "No templates saved",
|
||||
"updateTemplate": "Update template"
|
||||
"updateTemplate": "Update template",
|
||||
"detectionRecognized": "Format recognized — {{read}} of {{total}} rows read",
|
||||
"detectionUncertain": "Uncertain format — only {{read}} of {{total}} rows read",
|
||||
"detectionUncertainHint": "Check the column mapping and the date format, then look at the preview before importing.",
|
||||
"detectionBank": "{{bank}} format recognized — {{read}} of {{total}} rows read"
|
||||
},
|
||||
"drift": {
|
||||
"title": "This source's format has changed",
|
||||
"intro": "This file's header row no longer matches the one recorded at the last successful import. The stored columns may no longer point at the right data.",
|
||||
"columnMoved": "{{label}}: column {{from}} → {{to}}",
|
||||
"columnAdded": "{{label}}: new column, at position {{to}}",
|
||||
"columnRemoved": "{{label}}: column gone, it was at position {{from}}",
|
||||
"adopt": "Adopt the re-detected format",
|
||||
"adoptHint": "Reads the file the way it looks today. The new configuration is saved at import time.",
|
||||
"adoptUnavailable": "Detection could not read this new shape: fix the mapping at the previous step.",
|
||||
"keep": "Keep the current configuration",
|
||||
"keepHint": "Keeps the stored mapping. Use this when the header changed but the columns did not move."
|
||||
},
|
||||
"repairPath": {
|
||||
"title": "Repairing an import that was already written wrong",
|
||||
"body": "Do not re-import a file you have already imported in order to correct it: duplicates are matched on the date, the description AND the amount, so a row with a corrected amount does not match the faulty row — it is added next to it. A flipped sign also produces two mirror rows that cancel each other out in every report. Delete the faulty import from the import history first, then replay the file."
|
||||
},
|
||||
"preview": {
|
||||
"title": "Data Preview",
|
||||
|
|
@ -124,7 +144,12 @@
|
|||
"description": "Description",
|
||||
"amount": "Amount",
|
||||
"raw": "Raw data",
|
||||
"moreRows": "... and {{count}} more row(s)"
|
||||
"moreRows": "... and {{count}} more row(s)",
|
||||
"outflowCount": "{{count}} outflow(s)",
|
||||
"inflowCount": "{{count}} inflow(s)",
|
||||
"errorRows": "Rows in error",
|
||||
"flipSigns": "Flip the signs",
|
||||
"flipSignsHint": "Corrects the source configuration, not just this preview: the correction is remembered for the next imports."
|
||||
},
|
||||
"duplicates": {
|
||||
"title": "Duplicate Detection",
|
||||
|
|
@ -146,7 +171,8 @@
|
|||
"files": "Files",
|
||||
"settings": "Settings",
|
||||
"rowsToImport": "Rows to import",
|
||||
"rowsSummary": "{{count}} row(s) to import, {{skipped}} duplicate(s) skipped"
|
||||
"rowsSummary": "{{count}} row(s) to import, {{skipped}} duplicate(s) skipped",
|
||||
"columnUnmapped": "not mapped"
|
||||
},
|
||||
"progress": {
|
||||
"title": "Import in Progress",
|
||||
|
|
@ -186,6 +212,19 @@
|
|||
"confirm": "Confirm",
|
||||
"import": "Import"
|
||||
},
|
||||
"errors": {
|
||||
"unsupportedAmountMode": "The amount mode saved for this source is not recognized. Reconfigure the source before importing.",
|
||||
"unsupportedSignConvention": "The sign convention saved for this source is not recognized. Reconfigure the source before importing.",
|
||||
"invalidColumnMapping": "The column mapping saved for this source cannot be read. Reconfigure the source before importing.",
|
||||
"absoluteIndicatorFormat": "This file uses positive amounts with a separate column giving the direction (D for debit, C for credit). That format is not supported yet: importing it now would record every deposit as an expense. Export the statement with signed amounts, or with separate debit and credit columns.",
|
||||
"autoDetectFailed": "Auto-detection could not read this file. Configure the format manually."
|
||||
},
|
||||
"rowErrors": {
|
||||
"invalidDate": "Unreadable date",
|
||||
"invalidAmount": "Unreadable amount",
|
||||
"amountColumnNotMapped": "Amount column not mapped",
|
||||
"parseError": "Unreadable row"
|
||||
},
|
||||
"help": {
|
||||
"title": "How to import bank statements",
|
||||
"tips": [
|
||||
|
|
@ -581,12 +620,19 @@
|
|||
"countSuppliers": "{{count}} supplier(s)",
|
||||
"countKeywords": "{{count}} keyword(s)",
|
||||
"countTransactions": "{{count}} transaction(s)",
|
||||
"countImportSources": "{{count}} import source(s)",
|
||||
"countImportTemplates": "{{count}} import template(s)",
|
||||
"irreversibleWarning": "This action is irreversible. All existing data of the selected type will be permanently deleted and replaced.",
|
||||
"typeToConfirm": "Type \"{{word}}\" to confirm:",
|
||||
"confirmWord": "REPLACE",
|
||||
"replaceButton": "Replace Data",
|
||||
"success": "Import completed successfully",
|
||||
"tryAgain": "Try again"
|
||||
"tryAgain": "Try again",
|
||||
"errors": {
|
||||
"unsupportedAmountMode": "This file cannot be imported: the source \"{{name}}\" was saved with an amount mode this version does not support ({{value}}). Nothing has been modified.",
|
||||
"unsupportedSignConvention": "This file cannot be imported: the source \"{{name}}\" was saved with a sign convention this version does not support ({{value}}). Nothing has been modified.",
|
||||
"invalidFormatRow": "This file cannot be imported: the import configuration of \"{{name}}\" is incomplete or damaged. Nothing has been modified."
|
||||
}
|
||||
}
|
||||
},
|
||||
"userGuide": {
|
||||
|
|
@ -809,8 +855,12 @@
|
|||
"title": "Import",
|
||||
"overview": "Import bank statements from CSV files using a step-by-step wizard. Each bank account is represented as a source folder.",
|
||||
"features": [
|
||||
"Multi-step import wizard with data preview",
|
||||
"Configurable column mapping, delimiter, and date format",
|
||||
"Automatic format detection when you open a source that was never configured: delimiter, encoding, date format, and columns recognised by their header label (French and English)",
|
||||
"Bank layouts recognised by name — Desjardins, RBC, National Bank and Tangerine are read by their documented export format",
|
||||
"A confidence score shown after detection (“147 of 150 rows read”): it tells you what could be read, never whether the meaning is right",
|
||||
"A mandatory preview before every import, with a signed recap: outflows, inflows, totals and rows in error, plus a “Flip the signs” button",
|
||||
"A warning when a file's header no longer matches the one from the last successful import, column by column",
|
||||
"Configurable column mapping; delimiter, encoding and date format editable at any time",
|
||||
"Automatic duplicate detection (within batch and against existing data)",
|
||||
"Import templates to save and reuse source configurations",
|
||||
"Import history with the ability to delete past imports"
|
||||
|
|
@ -818,15 +868,23 @@
|
|||
"steps": [
|
||||
"Set your import folder via the folder picker at the top of the page",
|
||||
"Create a subfolder for each bank/source and place CSV files inside",
|
||||
"Click on a source to open the import wizard",
|
||||
"Configure the delimiter, encoding, date format, and column mapping",
|
||||
"Select which files to import and preview the parsed data",
|
||||
"Click on a source to open the import wizard — the first time, the format is detected automatically and a banner announces the result",
|
||||
"Check the proposed configuration (delimiter, encoding, date format, column mapping) and correct it if the banner reports an uncertain format",
|
||||
"Select which files to import",
|
||||
"Read the preview: the recap tells you how many outflows and inflows the file will produce, and for what amounts",
|
||||
"If inflows and outflows are swapped, click “Flip the signs” — the correction is saved with the source, so the next files from that bank read correctly on their own",
|
||||
"Check for duplicates, review the summary, then confirm the import"
|
||||
],
|
||||
"tips": [
|
||||
"Save your configuration as a template so you don't have to reconfigure each time",
|
||||
"The preview recap is the only check that sees what the amounts mean: a 100 % score means every row was read, not that your expenses are not being counted as income. Take two seconds to read it.",
|
||||
"Your configuration is stored in full on the source, including the amount mode and the sign convention: a source configured once is never guessed at again on later imports",
|
||||
"If your bank changes its column layout, a panel reports it before the import and offers two choices: adopt the newly detected format, or keep the stored one",
|
||||
"To repair an import that was already written the wrong way round, do not re-import the file on top of it: duplicates are matched on date, description AND amount, so a corrected row is added instead of replacing the faulty one. Delete the faulty import from the history first, then replay the file.",
|
||||
"A statement with positive amounts plus a “D / C” column is refused rather than imported backwards: re-export it from your bank with signed amounts, or with separate debit and credit columns",
|
||||
"Save your configuration as a template so you don't have to reconfigure each time. A template initialises a source; editing it afterwards changes no source you have already configured.",
|
||||
"Files already imported are marked with a badge — re-importing them will trigger duplicate detection",
|
||||
"You can delete an import from the history to remove all its transactions"
|
||||
"You can delete an import from the history to remove all its transactions",
|
||||
"Your import sources and templates are now part of the encrypted backup: restoring a backup no longer forces you to reconfigure everything"
|
||||
]
|
||||
},
|
||||
"transactions": {
|
||||
|
|
|
|||
|
|
@ -113,7 +113,27 @@
|
|||
"templateSaved": "Modèle sauvegardé",
|
||||
"deleteTemplate": "Supprimer le modèle",
|
||||
"noTemplates": "Aucun modèle sauvegardé",
|
||||
"updateTemplate": "Mettre à jour le modèle"
|
||||
"updateTemplate": "Mettre à jour le modèle",
|
||||
"detectionRecognized": "Format reconnu — {{read}} des {{total}} lignes lues",
|
||||
"detectionUncertain": "Format incertain — {{read}} des {{total}} lignes lues seulement",
|
||||
"detectionUncertainHint": "Vérifiez le mapping des colonnes et le format de date, puis regardez l'aperçu avant d'importer.",
|
||||
"detectionBank": "Format {{bank}} reconnu — {{read}} des {{total}} lignes lues"
|
||||
},
|
||||
"drift": {
|
||||
"title": "Le format de cette source a changé",
|
||||
"intro": "L'en-tête de ce fichier ne correspond plus à celui du dernier import réussi. Les colonnes mémorisées ne pointent peut-être plus sur les bonnes données.",
|
||||
"columnMoved": "{{label}} : colonne {{from}} → {{to}}",
|
||||
"columnAdded": "{{label}} : nouvelle colonne, en position {{to}}",
|
||||
"columnRemoved": "{{label}} : colonne disparue, elle était en position {{from}}",
|
||||
"adopt": "Adopter le format re-détecté",
|
||||
"adoptHint": "Relit le fichier tel qu'il se présente aujourd'hui. La nouvelle configuration est mémorisée à l'import.",
|
||||
"adoptUnavailable": "La détection n'a pas su lire cette nouvelle forme : corrigez le mapping à l'étape précédente.",
|
||||
"keep": "Conserver la configuration actuelle",
|
||||
"keepHint": "Garde le mapping mémorisé. À utiliser si l'en-tête a changé sans que les colonnes bougent."
|
||||
},
|
||||
"repairPath": {
|
||||
"title": "Réparer un import déjà écrit de travers",
|
||||
"body": "Ne réimportez pas un fichier déjà importé pour le corriger : les doublons sont repérés sur la date, la description ET le montant, donc une ligne au montant corrigé ne s'apparie pas à la ligne fautive — elle s'ajoute. Une inversion de signe crée en prime deux lignes miroir qui s'annulent dans les rapports. Supprimez d'abord l'import fautif dans l'historique des imports, puis rejouez le fichier."
|
||||
},
|
||||
"preview": {
|
||||
"title": "Aperçu des données",
|
||||
|
|
@ -124,7 +144,12 @@
|
|||
"description": "Description",
|
||||
"amount": "Montant",
|
||||
"raw": "Données brutes",
|
||||
"moreRows": "... et {{count}} ligne(s) supplémentaire(s)"
|
||||
"moreRows": "... et {{count}} ligne(s) supplémentaire(s)",
|
||||
"outflowCount": "{{count}} sortie(s)",
|
||||
"inflowCount": "{{count}} entrée(s)",
|
||||
"errorRows": "Lignes en erreur",
|
||||
"flipSigns": "Inverser les signes",
|
||||
"flipSignsHint": "Corrige la configuration de la source, pas seulement cet aperçu : la correction est mémorisée pour les prochains imports."
|
||||
},
|
||||
"duplicates": {
|
||||
"title": "Détection des doublons",
|
||||
|
|
@ -146,7 +171,8 @@
|
|||
"files": "Fichiers",
|
||||
"settings": "Paramètres",
|
||||
"rowsToImport": "Lignes à importer",
|
||||
"rowsSummary": "{{count}} ligne(s) à importer, {{skipped}} doublon(s) ignoré(s)"
|
||||
"rowsSummary": "{{count}} ligne(s) à importer, {{skipped}} doublon(s) ignoré(s)",
|
||||
"columnUnmapped": "non associée"
|
||||
},
|
||||
"progress": {
|
||||
"title": "Import en cours",
|
||||
|
|
@ -186,6 +212,19 @@
|
|||
"confirm": "Confirmer",
|
||||
"import": "Importer"
|
||||
},
|
||||
"errors": {
|
||||
"unsupportedAmountMode": "Le mode de montant enregistré pour cette source n'est pas reconnu. Reconfigurez la source avant d'importer.",
|
||||
"unsupportedSignConvention": "La convention de signe enregistrée pour cette source n'est pas reconnue. Reconfigurez la source avant d'importer.",
|
||||
"invalidColumnMapping": "Le mapping de colonnes enregistré pour cette source est illisible. Reconfigurez la source avant d'importer.",
|
||||
"absoluteIndicatorFormat": "Ce fichier utilise des montants positifs accompagnés d'une colonne indiquant le sens (D pour débit, C pour crédit). Ce format n'est pas encore pris en charge : l'importer maintenant enregistrerait chaque dépôt comme une dépense. Exportez le relevé avec des montants signés ou avec deux colonnes débit et crédit.",
|
||||
"autoDetectFailed": "La détection automatique n'a pas pu lire ce fichier. Configurez le format manuellement."
|
||||
},
|
||||
"rowErrors": {
|
||||
"invalidDate": "Date illisible",
|
||||
"invalidAmount": "Montant illisible",
|
||||
"amountColumnNotMapped": "Colonne de montant non mappée",
|
||||
"parseError": "Ligne illisible"
|
||||
},
|
||||
"help": {
|
||||
"title": "Comment importer des relevés bancaires",
|
||||
"tips": [
|
||||
|
|
@ -581,12 +620,19 @@
|
|||
"countSuppliers": "{{count}} fournisseur(s)",
|
||||
"countKeywords": "{{count}} mot(s)-clé(s)",
|
||||
"countTransactions": "{{count}} transaction(s)",
|
||||
"countImportSources": "{{count}} source(s) d'import",
|
||||
"countImportTemplates": "{{count}} modèle(s) d'import",
|
||||
"irreversibleWarning": "Cette action est irréversible. Toutes les données existantes du type sélectionné seront définitivement supprimées et remplacées.",
|
||||
"typeToConfirm": "Tapez « {{word}} » pour confirmer :",
|
||||
"confirmWord": "REMPLACER",
|
||||
"replaceButton": "Remplacer les données",
|
||||
"success": "Import terminé avec succès",
|
||||
"tryAgain": "Réessayer"
|
||||
"tryAgain": "Réessayer",
|
||||
"errors": {
|
||||
"unsupportedAmountMode": "Ce fichier ne peut pas être importé : la source « {{name}} » a été enregistrée avec un mode de montant que cette version ne prend pas en charge ({{value}}). Rien n'a été modifié.",
|
||||
"unsupportedSignConvention": "Ce fichier ne peut pas être importé : la source « {{name}} » a été enregistrée avec une convention de signe que cette version ne prend pas en charge ({{value}}). Rien n'a été modifié.",
|
||||
"invalidFormatRow": "Ce fichier ne peut pas être importé : la configuration d'import de « {{name}} » est incomplète ou endommagée. Rien n'a été modifié."
|
||||
}
|
||||
}
|
||||
},
|
||||
"userGuide": {
|
||||
|
|
@ -809,8 +855,12 @@
|
|||
"title": "Import",
|
||||
"overview": "Importez des relevés bancaires à partir de fichiers CSV à l'aide d'un assistant étape par étape. Chaque compte bancaire est représenté par un dossier source.",
|
||||
"features": [
|
||||
"Assistant d'import multi-étapes avec aperçu des données",
|
||||
"Mapping de colonnes configurable, délimiteur et format de date",
|
||||
"Détection automatique du format à l'ouverture d'une source jamais configurée : délimiteur, encodage, format de date et colonnes reconnues par leur libellé (français et anglais)",
|
||||
"Formats reconnus par banque — Desjardins, RBC, Banque Nationale et Tangerine sont lus par leur nom",
|
||||
"Score de confiance affiché après détection (« 147 des 150 lignes lues ») : il indique ce qui a pu être lu, jamais si le sens est bon",
|
||||
"Aperçu obligatoire avant chaque import, avec un récapitulatif signé : sorties, entrées, totaux et lignes en erreur, plus un bouton « Inverser les signes »",
|
||||
"Avertissement quand l'en-tête d'un fichier ne correspond plus à celui du dernier import réussi, colonne par colonne",
|
||||
"Mapping de colonnes configurable, délimiteur, encodage et format de date modifiables à tout moment",
|
||||
"Détection automatique des doublons (dans le lot et contre les données existantes)",
|
||||
"Modèles d'import pour sauvegarder et réutiliser les configurations",
|
||||
"Historique des imports avec possibilité de supprimer les imports précédents"
|
||||
|
|
@ -818,15 +868,23 @@
|
|||
"steps": [
|
||||
"Définissez votre dossier d'import via le sélecteur de dossier en haut de la page",
|
||||
"Créez un sous-dossier pour chaque banque/source et placez-y les fichiers CSV",
|
||||
"Cliquez sur une source pour ouvrir l'assistant d'import",
|
||||
"Configurez le délimiteur, l'encodage, le format de date et le mapping des colonnes",
|
||||
"Sélectionnez les fichiers à importer et prévisualisez les données analysées",
|
||||
"Cliquez sur une source pour ouvrir l'assistant d'import — la première fois, le format est détecté automatiquement et un bandeau annonce le résultat",
|
||||
"Vérifiez la configuration proposée (délimiteur, encodage, format de date, mapping des colonnes) et corrigez-la si le bandeau signale un format incertain",
|
||||
"Sélectionnez les fichiers à importer",
|
||||
"Examinez l'aperçu : le récapitulatif indique combien de sorties et d'entrées le fichier produira, et pour quels montants",
|
||||
"Si les entrées et les sorties sont inversées, cliquez sur « Inverser les signes » — la correction est enregistrée avec la source, les prochains fichiers de cette banque se liront correctement d'eux-mêmes",
|
||||
"Vérifiez les doublons, examinez le résumé, puis confirmez l'import"
|
||||
],
|
||||
"tips": [
|
||||
"Sauvegardez votre configuration comme modèle pour ne pas avoir à reconfigurer à chaque fois",
|
||||
"Le récapitulatif de l'aperçu est le seul contrôle qui voit le sens des montants : un score de 100 % signifie que chaque ligne a été lue, pas que vos dépenses ne sont pas comptées comme des revenus. Prenez deux secondes pour le lire.",
|
||||
"Votre configuration est mémorisée intégralement sur la source, y compris le mode de montant et la convention de signe : une source configurée une fois n'est plus jamais re-devinée aux imports suivants",
|
||||
"Si votre banque change la disposition de ses colonnes, un panneau le signale avant l'import et propose deux choix : adopter le nouveau format détecté, ou conserver celui enregistré",
|
||||
"Pour réparer un import déjà écrit de travers, ne ré-importez pas le fichier par-dessus : les doublons sont repérés sur la date, la description ET le montant, donc une ligne corrigée s'ajoute au lieu de remplacer la fautive. Supprimez d'abord l'import fautif dans l'historique, puis rejouez le fichier.",
|
||||
"Un relevé à montants positifs accompagné d'une colonne « D / C » est refusé plutôt qu'importé à l'envers : réexportez-le depuis votre banque avec des montants signés, ou avec deux colonnes débit et crédit séparées",
|
||||
"Sauvegardez votre configuration comme modèle pour ne pas avoir à reconfigurer à chaque fois. Un modèle sert à initialiser une source ; le modifier ensuite ne change aucune source déjà configurée.",
|
||||
"Les fichiers déjà importés sont marqués d'un badge — les ré-importer déclenchera la détection de doublons",
|
||||
"Vous pouvez supprimer un import de l'historique pour retirer toutes ses transactions"
|
||||
"Vous pouvez supprimer un import de l'historique pour retirer toutes ses transactions",
|
||||
"Vos sources et vos modèles d'import font désormais partie de la sauvegarde chiffrée : restaurer un backup ne vous oblige plus à tout reconfigurer"
|
||||
]
|
||||
},
|
||||
"transactions": {
|
||||
|
|
|
|||
|
|
@ -1,17 +1,17 @@
|
|||
import { useState, useCallback } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useImportWizard } from "../hooks/useImportWizard";
|
||||
import ImportFolderConfig from "../components/import/ImportFolderConfig";
|
||||
import SourceList from "../components/import/SourceList";
|
||||
import SourceConfigPanel from "../components/import/SourceConfigPanel";
|
||||
import FilePreviewTable from "../components/import/FilePreviewTable";
|
||||
import FormatDriftPanel from "../components/import/FormatDriftPanel";
|
||||
import DuplicateCheckPanel from "../components/import/DuplicateCheckPanel";
|
||||
import ImportConfirmation from "../components/import/ImportConfirmation";
|
||||
import ImportProgress from "../components/import/ImportProgress";
|
||||
import ImportReportPanel from "../components/import/ImportReportPanel";
|
||||
import WizardNavigation from "../components/import/WizardNavigation";
|
||||
import ImportHistoryPanel from "../components/import/ImportHistoryPanel";
|
||||
import FilePreviewModal from "../components/import/FilePreviewModal";
|
||||
import { AlertCircle, Eye, X, ChevronLeft } from "lucide-react";
|
||||
import { AlertCircle } from "lucide-react";
|
||||
import { PageHelp } from "../components/shared/PageHelp";
|
||||
|
||||
export default function ImportPage() {
|
||||
|
|
@ -24,11 +24,14 @@ export default function ImportPage() {
|
|||
updateConfig,
|
||||
toggleFile,
|
||||
selectAllFiles,
|
||||
parsePreview,
|
||||
parseAndCheckDuplicates,
|
||||
parseAndPreview,
|
||||
checkDuplicates,
|
||||
flipSignConvention,
|
||||
executeImport,
|
||||
goToStep,
|
||||
reset,
|
||||
adoptDriftFormat,
|
||||
keepCurrentFormat,
|
||||
autoDetectConfig,
|
||||
saveConfigAsTemplate,
|
||||
applyConfigTemplate,
|
||||
|
|
@ -38,13 +41,6 @@ export default function ImportPage() {
|
|||
setSkipAllDuplicates,
|
||||
} = useImportWizard();
|
||||
|
||||
const [showPreviewModal, setShowPreviewModal] = useState(false);
|
||||
|
||||
const handlePreview = useCallback(async () => {
|
||||
await parsePreview();
|
||||
setShowPreviewModal(true);
|
||||
}, [parsePreview]);
|
||||
|
||||
const nextDisabled = state.selectedFiles.length === 0 || !state.sourceConfig.name;
|
||||
|
||||
return (
|
||||
|
|
@ -58,8 +54,13 @@ export default function ImportPage() {
|
|||
{state.error && (
|
||||
<div className="mb-4 p-3 rounded-xl bg-[var(--card)] border-2 border-[var(--negative)] flex items-center gap-2">
|
||||
<AlertCircle size={16} className="text-[var(--negative)] shrink-0" />
|
||||
{/*
|
||||
The wizard reports a translation key when it has one (a stored
|
||||
format it cannot decode) and a raw message otherwise; `defaultValue`
|
||||
renders the latter unchanged.
|
||||
*/}
|
||||
<p className="text-sm text-[var(--foreground)]">
|
||||
{state.error}
|
||||
{t(state.error, { defaultValue: state.error })}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
|
@ -103,43 +104,54 @@ export default function ImportPage() {
|
|||
onUpdateTemplate={updateConfigTemplate}
|
||||
onDeleteTemplate={deleteConfigTemplate}
|
||||
selectedTemplateId={state.selectedTemplateId}
|
||||
detectionScore={state.detectionScore}
|
||||
detectedBank={state.detectedBank}
|
||||
isLoading={state.isLoading}
|
||||
/>
|
||||
<div className="flex items-center justify-between pt-6 border-t border-[var(--border)]">
|
||||
<div>
|
||||
<button
|
||||
onClick={reset}
|
||||
className="flex items-center gap-1 px-4 py-2 text-sm text-[var(--muted-foreground)] hover:text-[var(--foreground)] transition-colors"
|
||||
>
|
||||
<X size={16} />
|
||||
{t("common.cancel")}
|
||||
</button>
|
||||
</div>
|
||||
<div className="flex items-center gap-3">
|
||||
<button
|
||||
onClick={() => goToStep("source-list")}
|
||||
className="flex items-center gap-1 px-4 py-2 text-sm rounded-lg border border-[var(--border)] text-[var(--foreground)] hover:bg-[var(--muted)] transition-colors"
|
||||
>
|
||||
<ChevronLeft size={16} />
|
||||
{t("import.wizard.back")}
|
||||
</button>
|
||||
<button
|
||||
onClick={handlePreview}
|
||||
disabled={nextDisabled || state.isLoading}
|
||||
className="flex items-center gap-1 px-4 py-2 text-sm rounded-lg border border-[var(--border)] text-[var(--foreground)] hover:bg-[var(--muted)] transition-colors disabled:opacity-50 disabled:cursor-not-allowed"
|
||||
>
|
||||
<Eye size={16} />
|
||||
{t("import.wizard.preview")}
|
||||
</button>
|
||||
<button
|
||||
onClick={parseAndCheckDuplicates}
|
||||
disabled={nextDisabled || state.isLoading}
|
||||
className="flex items-center gap-1 px-4 py-2 text-sm rounded-lg bg-[var(--primary)] text-white hover:opacity-90 transition-opacity disabled:opacity-50 disabled:cursor-not-allowed"
|
||||
>
|
||||
{t("import.wizard.checkDuplicates")}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
{/*
|
||||
One way forward, and it goes through the preview (#329). The pair of
|
||||
buttons this replaces offered "Aperçu" as an optional detour and
|
||||
"Vérifier les doublons" as the real path, so the totals were the one
|
||||
screen an import never had to show.
|
||||
*/}
|
||||
<WizardNavigation
|
||||
onBack={() => goToStep("source-list")}
|
||||
onNext={parseAndPreview}
|
||||
onCancel={reset}
|
||||
nextLabel={t("import.wizard.preview")}
|
||||
nextDisabled={nextDisabled || state.isLoading}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{state.step === "file-preview" && (
|
||||
<div className="space-y-6">
|
||||
{/*
|
||||
Format drift (#330) — ABOVE the table, because it tells the user
|
||||
whether the table below is worth reading. Present only when the
|
||||
header row moved since the last successful import of this source.
|
||||
*/}
|
||||
{state.formatDrift && (
|
||||
<FormatDriftPanel
|
||||
entries={state.formatDrift}
|
||||
canAdopt={state.driftConfig !== null}
|
||||
onAdopt={adoptDriftFormat}
|
||||
onKeep={keepCurrentFormat}
|
||||
isBusy={state.isLoading}
|
||||
/>
|
||||
)}
|
||||
<FilePreviewTable
|
||||
rows={state.parsedPreview}
|
||||
onFlipSigns={flipSignConvention}
|
||||
isFlipping={state.isLoading}
|
||||
/>
|
||||
<WizardNavigation
|
||||
onBack={() => goToStep("source-config")}
|
||||
onNext={checkDuplicates}
|
||||
onCancel={reset}
|
||||
nextLabel={t("import.wizard.checkDuplicates")}
|
||||
nextDisabled={state.isLoading}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
|
|
@ -153,7 +165,7 @@ export default function ImportPage() {
|
|||
onIncludeAll={() => setSkipAllDuplicates(false)}
|
||||
/>
|
||||
<WizardNavigation
|
||||
onBack={() => goToStep("source-config")}
|
||||
onBack={() => goToStep("file-preview")}
|
||||
onNext={() => goToStep("confirm")}
|
||||
onCancel={reset}
|
||||
nextLabel={t("import.wizard.confirm")}
|
||||
|
|
@ -166,6 +178,7 @@ export default function ImportPage() {
|
|||
<ImportConfirmation
|
||||
sourceName={state.sourceConfig.name}
|
||||
config={state.sourceConfig}
|
||||
headers={state.previewHeaders}
|
||||
selectedFiles={state.selectedFiles}
|
||||
duplicateResult={state.duplicateResult}
|
||||
excludedCount={state.excludedDuplicateIndices.size}
|
||||
|
|
@ -191,15 +204,6 @@ export default function ImportPage() {
|
|||
{state.step === "report" && state.importReport && (
|
||||
<ImportReportPanel report={state.importReport} onDone={reset} />
|
||||
)}
|
||||
|
||||
{/* Preview modal */}
|
||||
{showPreviewModal && state.parsedPreview.length > 0 && (
|
||||
<FilePreviewModal
|
||||
rows={state.parsedPreview.slice(0, 20)}
|
||||
totalCount={state.parsedPreview.length}
|
||||
onClose={() => setShowPreviewModal(false)}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,8 @@ vi.mock("./dataExportService", async () => {
|
|||
getExportSuppliers: vi.fn(async () => []),
|
||||
getExportKeywords: vi.fn(async () => []),
|
||||
getExportTransactions: vi.fn(async () => []),
|
||||
getExportImportSources: vi.fn(async () => []),
|
||||
getExportImportTemplates: vi.fn(async () => []),
|
||||
};
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,8 @@ import {
|
|||
getExportSuppliers,
|
||||
getExportKeywords,
|
||||
getExportTransactions,
|
||||
getExportImportSources,
|
||||
getExportImportTemplates,
|
||||
serializeToJson,
|
||||
parseImportedJson,
|
||||
type ExportEnvelope,
|
||||
|
|
@ -254,17 +256,36 @@ export async function createPreMigrationBackup(
|
|||
}
|
||||
|
||||
// 1. Gather data — same mode as "transactions_with_categories" export.
|
||||
// Import sources and templates are part of that mode since #331: this
|
||||
// backup is a safety net, and one that came back without a single import
|
||||
// configuration would be a poor one.
|
||||
const appVersion = await getVersion();
|
||||
const [categories, suppliers, keywords, transactions] = await Promise.all([
|
||||
const [
|
||||
categories,
|
||||
suppliers,
|
||||
keywords,
|
||||
transactions,
|
||||
import_sources,
|
||||
import_config_templates,
|
||||
] = await Promise.all([
|
||||
getExportCategories(),
|
||||
getExportSuppliers(),
|
||||
getExportKeywords(),
|
||||
getExportTransactions(),
|
||||
getExportImportSources(),
|
||||
getExportImportTemplates(),
|
||||
]);
|
||||
|
||||
const content = serializeToJson(
|
||||
"transactions_with_categories",
|
||||
{ categories, suppliers, keywords, transactions },
|
||||
{
|
||||
categories,
|
||||
suppliers,
|
||||
keywords,
|
||||
transactions,
|
||||
import_sources,
|
||||
import_config_templates,
|
||||
},
|
||||
appVersion,
|
||||
);
|
||||
|
||||
|
|
|
|||
730
src/services/dataExportService.test.ts
Normal file
730
src/services/dataExportService.test.ts
Normal file
|
|
@ -0,0 +1,730 @@
|
|||
/**
|
||||
* The data export/restore cycle preserves import configurations (#331).
|
||||
*
|
||||
* Three properties are under test here, and each one is a way the format used
|
||||
* to be lost:
|
||||
* 1. What goes OUT — the file carries `import_sources` and
|
||||
* `import_config_templates`, which it never did.
|
||||
* 2. What comes back IN — templates before sources, `template_id` remapped
|
||||
* through resolved ids, and the synthetic "Data Import" source demoted to
|
||||
* what it always should have been: a host for transactions that describe
|
||||
* no folder of their own.
|
||||
* 3. What happens when it goes wrong — a restore that fails at row N leaves
|
||||
* the profile untouched. That one is the reason this file exists at all:
|
||||
* before #331 the service deleted six tables and then re-inserted them
|
||||
* with no transaction around any of it.
|
||||
*
|
||||
* Real `tauri-plugin-sql` cannot run outside the Tauri WebView, so the service
|
||||
* runs against an in-memory FakeDb that interprets the statements it issues —
|
||||
* the same approach as `import-format-roundtrip.test.ts`. This one additionally
|
||||
* models `UNIQUE(name)` and BEGIN/ROLLBACK, because those are the failures the
|
||||
* transaction exists to survive.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, vi } from "vitest";
|
||||
|
||||
vi.mock("./db", () => {
|
||||
const getDb = vi.fn();
|
||||
return {
|
||||
getDb,
|
||||
withTransaction: vi.fn(async (fn: (db: unknown) => unknown) => fn(await getDb())),
|
||||
};
|
||||
});
|
||||
|
||||
import { getDb } from "./db";
|
||||
import {
|
||||
getExportImportSources,
|
||||
getExportImportTemplates,
|
||||
importCategoriesOnly,
|
||||
importTransactionsOnly,
|
||||
importTransactionsWithCategories,
|
||||
parseImportedJson,
|
||||
serializeToJson,
|
||||
validateImportedFormatRows,
|
||||
LEGACY_SREF_FORMAT_VERSION,
|
||||
SREF_FORMAT_VERSION,
|
||||
SrefValidationError,
|
||||
type ExportEnvelope,
|
||||
type ExportImportSource,
|
||||
type ExportImportTemplate,
|
||||
} from "./dataExportService";
|
||||
import { formatFromRow, FORMAT_FIELD_PAIRS } from "../utils/importFormat";
|
||||
import type { Category } from "../shared/types";
|
||||
import fr from "../i18n/locales/fr.json";
|
||||
import en from "../i18n/locales/en.json";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// FakeDb — enough SQLite to make the failure modes real.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type Row = Record<string, unknown>;
|
||||
|
||||
const UNIQUE_NAME_TABLES = ["import_sources", "import_config_templates"];
|
||||
|
||||
function makeFakeDb(shouldFail?: (sql: string, params: unknown[]) => boolean) {
|
||||
const tables: Record<string, Row[]> = {
|
||||
categories: [],
|
||||
suppliers: [],
|
||||
keywords: [],
|
||||
transactions: [],
|
||||
imported_files: [],
|
||||
import_sources: [],
|
||||
import_config_templates: [],
|
||||
};
|
||||
const log: string[] = [];
|
||||
let snapshot: Record<string, Row[]> | null = null;
|
||||
let nextId = 1;
|
||||
|
||||
const clone = (source: Record<string, Row[]>) =>
|
||||
Object.fromEntries(
|
||||
Object.entries(source).map(([name, rows]) => [
|
||||
name,
|
||||
rows.map((row) => ({ ...row })),
|
||||
])
|
||||
);
|
||||
|
||||
const insert = (sql: string, params: unknown[]) => {
|
||||
const [, table, cols] = /INSERT INTO (\w+) \(([^)]+)\)/.exec(sql) ?? [];
|
||||
const columns = cols.split(",").map((c) => c.trim());
|
||||
const row: Row = { id: nextId++ };
|
||||
columns.forEach((c, i) => (row[c] = params[i]));
|
||||
|
||||
const clash = tables[table].find((r) => r.name === row.name);
|
||||
if (/ON CONFLICT\(name\) DO UPDATE/.test(sql)) {
|
||||
if (clash) {
|
||||
columns.forEach((c, i) => (clash[c] = params[i]));
|
||||
nextId--;
|
||||
return { lastInsertId: 0, rowsAffected: 1 };
|
||||
}
|
||||
} else if (clash && UNIQUE_NAME_TABLES.includes(table)) {
|
||||
throw new Error(`UNIQUE constraint failed: ${table}.name`);
|
||||
}
|
||||
|
||||
tables[table].push(row);
|
||||
return { lastInsertId: row.id as number, rowsAffected: 1 };
|
||||
};
|
||||
|
||||
return {
|
||||
tables,
|
||||
log,
|
||||
seed(table: string, rows: Row[]) {
|
||||
rows.forEach((row) => tables[table].push({ id: nextId++, ...row }));
|
||||
},
|
||||
execute: vi.fn(async (sql: string, params: unknown[] = []) => {
|
||||
const head = sql.trimStart().split(/\s+/).slice(0, 3).join(" ");
|
||||
log.push(head);
|
||||
if (shouldFail?.(sql, params)) throw new Error("boom");
|
||||
|
||||
const trimmed = sql.trimStart();
|
||||
if (trimmed === "BEGIN") {
|
||||
snapshot = clone(tables);
|
||||
return { rowsAffected: 0 };
|
||||
}
|
||||
if (trimmed === "COMMIT") {
|
||||
snapshot = null;
|
||||
return { rowsAffected: 0 };
|
||||
}
|
||||
if (trimmed === "ROLLBACK") {
|
||||
if (snapshot) {
|
||||
for (const [name, rows] of Object.entries(snapshot)) {
|
||||
tables[name] = rows;
|
||||
}
|
||||
snapshot = null;
|
||||
}
|
||||
return { rowsAffected: 0 };
|
||||
}
|
||||
if (trimmed.startsWith("DELETE FROM")) {
|
||||
const [, table] = /DELETE FROM (\w+)/.exec(trimmed) ?? [];
|
||||
tables[table] = [];
|
||||
return { rowsAffected: 0 };
|
||||
}
|
||||
if (trimmed.startsWith("UPDATE transactions SET")) {
|
||||
return { rowsAffected: 0 };
|
||||
}
|
||||
if (trimmed.startsWith("INSERT")) return insert(sql, params);
|
||||
throw new Error(`FakeDb: unsupported statement ${sql.slice(0, 40)}`);
|
||||
}),
|
||||
select: vi.fn(async (sql: string, params: unknown[] = []) => {
|
||||
const [, table] = /FROM (\w+)/.exec(sql) ?? [];
|
||||
const rows = tables[table] ?? [];
|
||||
if (/WHERE (?:\w+\.)?name = \$1/.test(sql))
|
||||
return rows.filter((r) => r.name === params[0]);
|
||||
// `getExportImportSources` resolves the template through a LEFT JOIN.
|
||||
if (table === "import_sources" && /LEFT JOIN/.test(sql)) {
|
||||
return rows.map((r) => ({
|
||||
...r,
|
||||
template_name:
|
||||
tables.import_config_templates.find((t) => t.id === r.template_id)
|
||||
?.name ?? null,
|
||||
}));
|
||||
}
|
||||
return [...rows];
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
let db: ReturnType<typeof makeFakeDb>;
|
||||
|
||||
function wire(instance: ReturnType<typeof makeFakeDb>) {
|
||||
db = instance;
|
||||
vi.mocked(getDb).mockResolvedValue(instance as never);
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.mocked(getDb).mockReset();
|
||||
wire(makeFakeDb());
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fixtures
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const TEMPLATE: ExportImportTemplate = {
|
||||
name: "Desjardins EOP",
|
||||
delimiter: ";",
|
||||
encoding: "windows-1252",
|
||||
date_format: "YYYY-MM-DD",
|
||||
skip_lines: 1,
|
||||
has_header: 1,
|
||||
column_mapping: JSON.stringify({ date: 0, description: 2, amount: 3 }),
|
||||
amount_mode: "single",
|
||||
sign_convention: "positive_expense",
|
||||
};
|
||||
|
||||
const SOURCE: ExportImportSource = {
|
||||
name: "Visa Desjardins",
|
||||
description: "Credit card",
|
||||
header_signature: "date|description|amount",
|
||||
template_name: TEMPLATE.name,
|
||||
delimiter: ",",
|
||||
encoding: "utf-8",
|
||||
date_format: "DD/MM/YYYY",
|
||||
skip_lines: 2,
|
||||
has_header: 0,
|
||||
column_mapping: JSON.stringify({ date: 1, description: 3, debitAmount: 4, creditAmount: 5 }),
|
||||
amount_mode: "debit_credit",
|
||||
sign_convention: "negative_expense",
|
||||
};
|
||||
|
||||
function envelopeData(
|
||||
overrides: Partial<ExportEnvelope["data"]> = {}
|
||||
): ExportEnvelope["data"] {
|
||||
return {
|
||||
categories: [],
|
||||
suppliers: [],
|
||||
keywords: [],
|
||||
transactions: [],
|
||||
import_sources: [SOURCE],
|
||||
import_config_templates: [TEMPLATE],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function transactionFixture(id: number) {
|
||||
return {
|
||||
id,
|
||||
date: "2026-03-10",
|
||||
description: `tx ${id}`,
|
||||
amount: -10,
|
||||
category_id: null,
|
||||
category_name: null,
|
||||
original_description: null,
|
||||
notes: null,
|
||||
is_manually_categorized: 0,
|
||||
is_split: 0,
|
||||
parent_transaction_id: null,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 1. What goes out
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("export carries the import configurations (#331)", () => {
|
||||
it("projects every format field of a source, plus its own columns", async () => {
|
||||
db.seed("import_config_templates", [{ name: TEMPLATE.name }]);
|
||||
const templateId = db.tables.import_config_templates[0].id;
|
||||
db.seed("import_sources", [
|
||||
{
|
||||
name: SOURCE.name,
|
||||
description: SOURCE.description,
|
||||
header_signature: SOURCE.header_signature,
|
||||
template_id: templateId,
|
||||
delimiter: SOURCE.delimiter,
|
||||
encoding: SOURCE.encoding,
|
||||
date_format: SOURCE.date_format,
|
||||
skip_lines: SOURCE.skip_lines,
|
||||
has_header: SOURCE.has_header,
|
||||
column_mapping: SOURCE.column_mapping,
|
||||
amount_mode: SOURCE.amount_mode,
|
||||
sign_convention: SOURCE.sign_convention,
|
||||
},
|
||||
]);
|
||||
|
||||
const [exported] = await getExportImportSources();
|
||||
for (const column of Object.values(FORMAT_FIELD_PAIRS)) {
|
||||
expect(exported[column], `column ${column}`).toEqual(SOURCE[column]);
|
||||
}
|
||||
expect(exported.name).toBe(SOURCE.name);
|
||||
expect(exported.description).toBe(SOURCE.description);
|
||||
// Drift metadata rides along beside the format, never through the codec.
|
||||
expect(exported.header_signature).toBe(SOURCE.header_signature);
|
||||
// The foreign key travels as a NAME — ids mean nothing in another profile.
|
||||
expect(exported.template_name).toBe(TEMPLATE.name);
|
||||
expect("id" in exported).toBe(false);
|
||||
expect("template_id" in exported).toBe(false);
|
||||
});
|
||||
|
||||
it("carries a source an older build left unreadable rather than aborting", async () => {
|
||||
// The `'{}'` mapping the restore itself used to write: `formatFromRow`
|
||||
// refuses it, so decoding on the way out would make the whole profile
|
||||
// impossible to back up.
|
||||
db.seed("import_sources", [
|
||||
{
|
||||
name: "Data Import",
|
||||
column_mapping: "{}",
|
||||
delimiter: ",",
|
||||
encoding: "utf-8",
|
||||
date_format: "%Y-%m-%d",
|
||||
skip_lines: 0,
|
||||
has_header: 1,
|
||||
amount_mode: "single",
|
||||
sign_convention: "negative_expense",
|
||||
},
|
||||
]);
|
||||
const [exported] = await getExportImportSources();
|
||||
expect(exported.column_mapping).toBe("{}");
|
||||
});
|
||||
|
||||
it("exports templates by name with their eight fields", async () => {
|
||||
db.seed("import_config_templates", [{ ...TEMPLATE }]);
|
||||
const [exported] = await getExportImportTemplates();
|
||||
expect(exported.name).toBe(TEMPLATE.name);
|
||||
for (const column of Object.values(FORMAT_FIELD_PAIRS)) {
|
||||
expect(exported[column], `column ${column}`).toEqual(TEMPLATE[column]);
|
||||
}
|
||||
});
|
||||
|
||||
it("stamps the envelope with an explicit format version", () => {
|
||||
const json = serializeToJson("transactions_with_categories", envelopeData(), "0.15.0");
|
||||
expect(JSON.parse(json).format_version).toBe(SREF_FORMAT_VERSION);
|
||||
});
|
||||
|
||||
it("round-trips the two arrays through serialize -> parse", () => {
|
||||
const json = serializeToJson("transactions_with_categories", envelopeData(), "0.15.0");
|
||||
const { envelope, summary } = parseImportedJson(json);
|
||||
expect(envelope.data.import_sources).toEqual([SOURCE]);
|
||||
expect(envelope.data.import_config_templates).toEqual([TEMPLATE]);
|
||||
expect(summary.importSourcesCount).toBe(1);
|
||||
expect(summary.importTemplatesCount).toBe(1);
|
||||
expect(summary.formatVersion).toBe(SREF_FORMAT_VERSION);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 2. Backward compatibility — a backup written before this change
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const LEGACY_BACKUP = JSON.stringify({
|
||||
export_type: "transactions_with_categories",
|
||||
app_version: "0.14.0",
|
||||
exported_at: "2026-07-01T00:00:00Z",
|
||||
data: {
|
||||
categories: [],
|
||||
suppliers: [],
|
||||
keywords: [],
|
||||
transactions: [transactionFixture(1)],
|
||||
},
|
||||
});
|
||||
|
||||
describe("a backup written before #331 still imports", () => {
|
||||
it("reads as format version 1 with both counts at zero", () => {
|
||||
const { envelope, summary } = parseImportedJson(LEGACY_BACKUP);
|
||||
expect(summary.formatVersion).toBe(LEGACY_SREF_FORMAT_VERSION);
|
||||
expect(summary.importSourcesCount).toBe(0);
|
||||
expect(summary.importTemplatesCount).toBe(0);
|
||||
expect(envelope.data.import_sources).toBeUndefined();
|
||||
});
|
||||
|
||||
it("restores exactly as it did before — wipe, then one host source", async () => {
|
||||
db.seed("import_sources", [{ name: "Old source", column_mapping: "{}" }]);
|
||||
const { envelope } = parseImportedJson(LEGACY_BACKUP);
|
||||
await importTransactionsWithCategories(envelope.data, "backup.json");
|
||||
|
||||
expect(db.tables.import_sources).toHaveLength(1);
|
||||
expect(db.tables.import_sources[0].name).toBe("Data Import");
|
||||
expect(db.tables.transactions).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 3. The whitelist at the import boundary
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("the import boundary refuses a format it cannot store", () => {
|
||||
it("accepts a file that carries no configuration at all", () => {
|
||||
expect(() => validateImportedFormatRows(undefined, undefined)).not.toThrow();
|
||||
});
|
||||
|
||||
it("rejects an amount mode outside the whitelist, naming the source", () => {
|
||||
// `absolute_indicator` passes the v17 CHECK but no code path can read it.
|
||||
const bad = { ...SOURCE, amount_mode: "absolute_indicator" as never };
|
||||
try {
|
||||
validateImportedFormatRows([bad], undefined);
|
||||
expect.unreachable("should have thrown");
|
||||
} catch (e) {
|
||||
expect(e).toBeInstanceOf(SrefValidationError);
|
||||
const err = e as SrefValidationError;
|
||||
expect(err.i18nKey).toBe(
|
||||
"settings.dataManagement.import.errors.unsupportedAmountMode"
|
||||
);
|
||||
expect(err.params).toEqual({
|
||||
name: SOURCE.name,
|
||||
value: "absolute_indicator",
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects an unknown sign convention", () => {
|
||||
const bad = { ...SOURCE, sign_convention: "inverted" as never };
|
||||
expect(() => validateImportedFormatRows([bad], undefined)).toThrow(
|
||||
SrefValidationError
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a row with no column mapping instead of letting NOT NULL fire", () => {
|
||||
const bad = { ...SOURCE, column_mapping: undefined as never };
|
||||
try {
|
||||
validateImportedFormatRows([bad], undefined);
|
||||
expect.unreachable("should have thrown");
|
||||
} catch (e) {
|
||||
expect((e as SrefValidationError).i18nKey).toBe(
|
||||
"settings.dataManagement.import.errors.invalidFormatRow"
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("checks templates too", () => {
|
||||
const bad = { ...TEMPLATE, amount_mode: "absolute_indicator" as never };
|
||||
expect(() => validateImportedFormatRows(undefined, [bad])).toThrow(
|
||||
SrefValidationError
|
||||
);
|
||||
});
|
||||
|
||||
it("refuses at PARSE time, so the confirmation never opens", () => {
|
||||
const json = serializeToJson(
|
||||
"transactions_with_categories",
|
||||
envelopeData({
|
||||
import_sources: [{ ...SOURCE, amount_mode: "absolute_indicator" as never }],
|
||||
}),
|
||||
"0.15.0"
|
||||
);
|
||||
expect(() => parseImportedJson(json)).toThrow(SrefValidationError);
|
||||
});
|
||||
|
||||
it("refuses again at the restore, before a single DELETE", async () => {
|
||||
db.seed("categories", [{ name: "Épicerie" }]);
|
||||
await expect(
|
||||
importTransactionsWithCategories(
|
||||
envelopeData({
|
||||
import_sources: [{ ...SOURCE, sign_convention: "inverted" as never }],
|
||||
}),
|
||||
"backup.json"
|
||||
)
|
||||
).rejects.toBeInstanceOf(SrefValidationError);
|
||||
expect(db.tables.categories).toHaveLength(1);
|
||||
expect(db.log).not.toContain("BEGIN");
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 4. The restore itself
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("restoring brings the configurations back (#331)", () => {
|
||||
it("restores a source field by field, template resolved to the new id", async () => {
|
||||
await importTransactionsWithCategories(envelopeData(), "backup.json");
|
||||
|
||||
expect(db.tables.import_config_templates).toHaveLength(1);
|
||||
const template = db.tables.import_config_templates[0];
|
||||
const [source] = db.tables.import_sources;
|
||||
|
||||
for (const column of Object.values(FORMAT_FIELD_PAIRS)) {
|
||||
expect(source[column], `column ${column}`).toEqual(SOURCE[column]);
|
||||
}
|
||||
expect(source.description).toBe(SOURCE.description);
|
||||
expect(source.header_signature).toBe(SOURCE.header_signature);
|
||||
expect(source.template_id).toBe(template.id);
|
||||
});
|
||||
|
||||
it("writes the templates BEFORE the sources — template_id is a foreign key", async () => {
|
||||
await importTransactionsWithCategories(envelopeData(), "backup.json");
|
||||
const inserts = db.log.filter((l) => l.startsWith("INSERT INTO"));
|
||||
expect(inserts.indexOf("INSERT INTO import_config_templates")).toBeLessThan(
|
||||
inserts.indexOf("INSERT INTO import_sources")
|
||||
);
|
||||
});
|
||||
|
||||
it("upserts a template that already exists by name and remaps to its id", async () => {
|
||||
// Restoring into a profile that already holds templates is the NORMAL path;
|
||||
// a plain INSERT would hit UNIQUE(name) and roll the whole restore back.
|
||||
db.seed("import_config_templates", [
|
||||
{ ...TEMPLATE, delimiter: "\t", skip_lines: 99 },
|
||||
]);
|
||||
const existingId = db.tables.import_config_templates[0].id;
|
||||
|
||||
await importTransactionsWithCategories(envelopeData(), "backup.json");
|
||||
|
||||
expect(db.tables.import_config_templates).toHaveLength(1);
|
||||
expect(db.tables.import_config_templates[0].id).toBe(existingId);
|
||||
expect(db.tables.import_config_templates[0].delimiter).toBe(TEMPLATE.delimiter);
|
||||
expect(db.tables.import_sources[0].template_id).toBe(existingId);
|
||||
});
|
||||
|
||||
it("leaves template_id null when the name resolves to nothing", async () => {
|
||||
await importTransactionsWithCategories(
|
||||
envelopeData({
|
||||
import_sources: [{ ...SOURCE, template_name: "Gone" }],
|
||||
import_config_templates: [],
|
||||
}),
|
||||
"backup.json"
|
||||
);
|
||||
expect(db.tables.import_sources[0].template_id).toBeNull();
|
||||
});
|
||||
|
||||
it("creates NO host source when the file carries no transactions", async () => {
|
||||
await importTransactionsWithCategories(envelopeData(), "backup.json");
|
||||
expect(db.tables.import_sources.map((s) => s.name)).toEqual([SOURCE.name]);
|
||||
expect(db.tables.imported_files).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("creates the host source only to carry transactions, with a readable mapping", async () => {
|
||||
await importTransactionsWithCategories(
|
||||
envelopeData({ transactions: [transactionFixture(1), transactionFixture(2)] }),
|
||||
"backup.json"
|
||||
);
|
||||
|
||||
const host = db.tables.import_sources.find((s) => s.name === "Data Import")!;
|
||||
expect(host).toBeDefined();
|
||||
// The mapping used to be `'{}'`, which the codec cannot decode: a source
|
||||
// the app wrote and then refused to read.
|
||||
const decoded = formatFromRow(host as never);
|
||||
expect(decoded.columnMapping.date).toBe(0);
|
||||
expect(decoded.columnMapping.description).toBe(1);
|
||||
expect(decoded.columnMapping.amount).toBe(2);
|
||||
|
||||
expect(db.tables.transactions).toHaveLength(2);
|
||||
expect(db.tables.transactions.every((t) => t.source_id === host.id)).toBe(true);
|
||||
});
|
||||
|
||||
it("reuses a restored source that already bears the host name", async () => {
|
||||
// A profile restored once already holds a "Data Import" source, which the
|
||||
// backup then carries — inserting a second one would break UNIQUE(name)
|
||||
// and roll back everything.
|
||||
await importTransactionsWithCategories(
|
||||
envelopeData({
|
||||
import_sources: [{ ...SOURCE, name: "Data Import" }],
|
||||
transactions: [transactionFixture(1)],
|
||||
}),
|
||||
"backup.json"
|
||||
);
|
||||
expect(db.tables.import_sources).toHaveLength(1);
|
||||
// The restored configuration wins; it is not overwritten by the host stub.
|
||||
expect(db.tables.import_sources[0].column_mapping).toBe(SOURCE.column_mapping);
|
||||
expect(db.tables.transactions[0].source_id).toBe(db.tables.import_sources[0].id);
|
||||
});
|
||||
|
||||
it("restores sources in transactions_only mode too", async () => {
|
||||
await importTransactionsOnly(envelopeData(), "backup.json");
|
||||
expect(db.tables.import_sources.map((s) => s.name)).toEqual([SOURCE.name]);
|
||||
expect(db.tables.import_config_templates).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("leaves import_sources alone in categories_only mode", async () => {
|
||||
db.seed("import_sources", [{ name: "Kept", column_mapping: "{}" }]);
|
||||
await importCategoriesOnly({ categories: [], suppliers: [], keywords: [] });
|
||||
expect(db.tables.import_sources.map((s) => s.name)).toEqual(["Kept"]);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 5. All or nothing — the property the whole restore rests on
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const CATEGORY_FIXTURES: Category[] = [1, 2, 3].map(
|
||||
(id) =>
|
||||
({
|
||||
id,
|
||||
name: `Cat ${id}`,
|
||||
parent_id: null,
|
||||
color: null,
|
||||
icon: null,
|
||||
type: "expense",
|
||||
is_active: 1,
|
||||
is_inputable: 1,
|
||||
sort_order: id,
|
||||
}) as unknown as Category
|
||||
);
|
||||
|
||||
describe("a restore that fails at row N leaves the profile untouched", () => {
|
||||
it("rolls the categories, sources and transactions back", async () => {
|
||||
let categoryInserts = 0;
|
||||
wire(
|
||||
makeFakeDb((sql) => {
|
||||
if (!/INSERT INTO categories/.test(sql)) return false;
|
||||
return ++categoryInserts === 3;
|
||||
})
|
||||
);
|
||||
db.seed("categories", [{ id: 900, name: "Existing" }]);
|
||||
db.seed("import_sources", [{ name: "Existing source", column_mapping: "{}" }]);
|
||||
db.seed("transactions", [{ description: "existing tx" }]);
|
||||
|
||||
await expect(
|
||||
importTransactionsWithCategories(
|
||||
envelopeData({
|
||||
categories: CATEGORY_FIXTURES,
|
||||
transactions: [transactionFixture(1)],
|
||||
}),
|
||||
"backup.json"
|
||||
)
|
||||
).rejects.toThrow("boom");
|
||||
|
||||
expect(db.log).toContain("ROLLBACK");
|
||||
expect(db.log).not.toContain("COMMIT");
|
||||
expect(db.tables.categories.map((c) => c.name)).toEqual(["Existing"]);
|
||||
expect(db.tables.import_sources.map((s) => s.name)).toEqual(["Existing source"]);
|
||||
expect(db.tables.transactions).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("rolls back a UNIQUE(name) collision between two restored sources", async () => {
|
||||
db.seed("categories", [{ id: 900, name: "Existing" }]);
|
||||
await expect(
|
||||
importTransactionsWithCategories(
|
||||
envelopeData({ import_sources: [SOURCE, { ...SOURCE }] }),
|
||||
"backup.json"
|
||||
)
|
||||
).rejects.toThrow(/UNIQUE constraint failed/);
|
||||
expect(db.tables.categories.map((c) => c.name)).toEqual(["Existing"]);
|
||||
expect(db.tables.import_sources).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("wraps categories_only as well", async () => {
|
||||
let keywordInserts = 0;
|
||||
wire(
|
||||
makeFakeDb((sql) => {
|
||||
if (!/INSERT INTO keywords/.test(sql)) return false;
|
||||
return ++keywordInserts === 1;
|
||||
})
|
||||
);
|
||||
db.seed("categories", [{ id: 900, name: "Existing" }]);
|
||||
|
||||
await expect(
|
||||
importCategoriesOnly({
|
||||
categories: CATEGORY_FIXTURES,
|
||||
suppliers: [],
|
||||
keywords: [
|
||||
{ id: 1, keyword: "epicerie", category_id: 1, priority: 1, is_active: 1 } as never,
|
||||
],
|
||||
})
|
||||
).rejects.toThrow("boom");
|
||||
|
||||
expect(db.log).toContain("ROLLBACK");
|
||||
expect(db.tables.categories.map((c) => c.name)).toEqual(["Existing"]);
|
||||
});
|
||||
|
||||
it("wraps transactions_only as well", async () => {
|
||||
let txInserts = 0;
|
||||
wire(
|
||||
makeFakeDb((sql) => {
|
||||
if (!/INSERT INTO transactions/.test(sql)) return false;
|
||||
return ++txInserts === 2;
|
||||
})
|
||||
);
|
||||
db.seed("transactions", [{ description: "existing tx" }]);
|
||||
|
||||
await expect(
|
||||
importTransactionsOnly(
|
||||
envelopeData({
|
||||
transactions: [transactionFixture(1), transactionFixture(2)],
|
||||
}),
|
||||
"backup.json"
|
||||
)
|
||||
).rejects.toThrow("boom");
|
||||
|
||||
expect(db.log).toContain("ROLLBACK");
|
||||
expect(db.tables.transactions.map((t) => t.description)).toEqual(["existing tx"]);
|
||||
expect(db.tables.import_sources).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 6. Every string the refusal can show exists in both languages
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("the refusal messages are translated (#331)", () => {
|
||||
const errorKeys = [
|
||||
"unsupportedAmountMode",
|
||||
"unsupportedSignConvention",
|
||||
"invalidFormatRow",
|
||||
] as const;
|
||||
|
||||
it("carries every refusal key in FR and EN", () => {
|
||||
for (const key of errorKeys) {
|
||||
expect(
|
||||
fr.settings.dataManagement.import.errors[key].length,
|
||||
`fr.${key}`
|
||||
).toBeGreaterThan(0);
|
||||
expect(
|
||||
en.settings.dataManagement.import.errors[key].length,
|
||||
`en.${key}`
|
||||
).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
it("names the offending source in both languages", () => {
|
||||
for (const key of errorKeys) {
|
||||
expect(fr.settings.dataManagement.import.errors[key], `fr.${key}`).toContain(
|
||||
"{{name}}"
|
||||
);
|
||||
expect(en.settings.dataManagement.import.errors[key], `en.${key}`).toContain(
|
||||
"{{name}}"
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("interpolates the two configuration counts", () => {
|
||||
for (const key of ["countImportSources", "countImportTemplates"] as const) {
|
||||
expect(fr.settings.dataManagement.import[key], `fr.${key}`).toContain(
|
||||
"{{count}}"
|
||||
);
|
||||
expect(en.settings.dataManagement.import[key], `en.${key}`).toContain(
|
||||
"{{count}}"
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("keys every SrefValidationError to a string that exists", () => {
|
||||
const cases: Array<[ExportImportSource, string]> = [
|
||||
[{ ...SOURCE, amount_mode: "absolute_indicator" as never }, "unsupportedAmountMode"],
|
||||
[{ ...SOURCE, sign_convention: "inverted" as never }, "unsupportedSignConvention"],
|
||||
[{ ...SOURCE, column_mapping: undefined as never }, "invalidFormatRow"],
|
||||
];
|
||||
for (const [row, suffix] of cases) {
|
||||
try {
|
||||
validateImportedFormatRows([row], undefined);
|
||||
expect.unreachable(`should have thrown for ${suffix}`);
|
||||
} catch (e) {
|
||||
const key = (e as SrefValidationError).i18nKey;
|
||||
expect(key).toBe(`settings.dataManagement.import.errors.${suffix}`);
|
||||
// The key must resolve in the bundle, not merely look plausible.
|
||||
const resolved = key
|
||||
.split(".")
|
||||
.reduce<unknown>(
|
||||
(node, part) => (node as Record<string, unknown>)[part],
|
||||
en as unknown
|
||||
);
|
||||
expect(typeof resolved).toBe("string");
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
|
@ -1,6 +1,19 @@
|
|||
import { getDb } from "./db";
|
||||
import type Database from "@tauri-apps/plugin-sql";
|
||||
import { getDb, withTransaction } from "./db";
|
||||
import Papa from "papaparse";
|
||||
import type { Category, Supplier, Keyword } from "../shared/types";
|
||||
import type {
|
||||
Category,
|
||||
ImportConfigTemplate,
|
||||
ImportFormatRow,
|
||||
ImportSource,
|
||||
Keyword,
|
||||
Supplier,
|
||||
} from "../shared/types";
|
||||
import {
|
||||
AMOUNT_MODES,
|
||||
SIGN_CONVENTIONS,
|
||||
pickFormatRow,
|
||||
} from "../utils/importFormat";
|
||||
|
||||
// --- Export types ---
|
||||
|
||||
|
|
@ -11,15 +24,34 @@ export type ExportMode =
|
|||
|
||||
export type ExportFormat = "json" | "csv";
|
||||
|
||||
/**
|
||||
* Version of the SREF envelope this build writes.
|
||||
*
|
||||
* Version 2 is the first to carry `import_sources` and
|
||||
* `import_config_templates`. A file with no `format_version` at all is a
|
||||
* version 1 file: it predates #331, so its missing arrays mean "this backup
|
||||
* never held any configuration", not "this profile had none". The restore then
|
||||
* behaves exactly as it did before — wipe the sources and hang the transactions
|
||||
* off a synthetic one — which is the only reading that cannot invent data.
|
||||
*/
|
||||
export const SREF_FORMAT_VERSION = 2;
|
||||
|
||||
/** What an envelope without an explicit `format_version` is. */
|
||||
export const LEGACY_SREF_FORMAT_VERSION = 1;
|
||||
|
||||
export interface ExportEnvelope {
|
||||
export_type: ExportMode;
|
||||
app_version: string;
|
||||
/** Absent on files written before #331 — see `SREF_FORMAT_VERSION`. */
|
||||
format_version?: number;
|
||||
exported_at: string;
|
||||
data: {
|
||||
categories?: Category[];
|
||||
suppliers?: Supplier[];
|
||||
keywords?: Keyword[];
|
||||
transactions?: ExportTransaction[];
|
||||
import_sources?: ExportImportSource[];
|
||||
import_config_templates?: ExportImportTemplate[];
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -37,6 +69,35 @@ export interface ExportTransaction {
|
|||
parent_transaction_id: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* An import source as it travels in the file: the eight format fields in their
|
||||
* persisted shape, plus the source's own columns.
|
||||
*
|
||||
* No `id`: nothing in the envelope points at a source by id (`ExportTransaction`
|
||||
* carries no `source_id`), so ids would only be noise that collides on restore.
|
||||
* The link to a template travels as `template_name` for the same reason — the
|
||||
* name is the table's natural key (`UNIQUE`), it survives renumbering, and it is
|
||||
* what the restore resolves the foreign key through.
|
||||
*/
|
||||
export interface ExportImportSource extends ImportFormatRow {
|
||||
name: string;
|
||||
description: string | null;
|
||||
/**
|
||||
* Drift metadata (#330), NOT one of the eight format fields — it travels
|
||||
* beside them, never through the codec. Round-tripping it keeps drift
|
||||
* detection armed across a restore; a source that comes back without one just
|
||||
* has detection off until its next successful import records it.
|
||||
*/
|
||||
header_signature: string | null;
|
||||
/** Name of the template this source was configured from, or null. */
|
||||
template_name: string | null;
|
||||
}
|
||||
|
||||
/** A template as it travels in the file: a named format, nothing more. */
|
||||
export interface ExportImportTemplate extends ImportFormatRow {
|
||||
name: string;
|
||||
}
|
||||
|
||||
// --- Import types ---
|
||||
|
||||
export interface ImportSummary {
|
||||
|
|
@ -45,6 +106,37 @@ export interface ImportSummary {
|
|||
suppliersCount: number;
|
||||
keywordsCount: number;
|
||||
transactionsCount: number;
|
||||
importSourcesCount: number;
|
||||
importTemplatesCount: number;
|
||||
/** `LEGACY_SREF_FORMAT_VERSION` when the file declares none. */
|
||||
formatVersion: number;
|
||||
}
|
||||
|
||||
/** i18n keys for the ways an imported configuration can be refused. */
|
||||
export type SrefValidationErrorKey =
|
||||
| "settings.dataManagement.import.errors.unsupportedAmountMode"
|
||||
| "settings.dataManagement.import.errors.unsupportedSignConvention"
|
||||
| "settings.dataManagement.import.errors.invalidFormatRow";
|
||||
|
||||
/**
|
||||
* Thrown when a backup carries a configuration this build cannot store.
|
||||
*
|
||||
* The `CHECK` added by migration v17 already refuses these values, but a SQLite
|
||||
* constraint error is not something a user can act on — and it would surface
|
||||
* only once the restore had already started deleting. This is raised at the
|
||||
* boundary, before anything is touched, and carries an i18n key plus the
|
||||
* offending source name so the message can name what to fix.
|
||||
*/
|
||||
export class SrefValidationError extends Error {
|
||||
readonly i18nKey: SrefValidationErrorKey;
|
||||
readonly params: Record<string, string>;
|
||||
|
||||
constructor(i18nKey: SrefValidationErrorKey, params: Record<string, string>) {
|
||||
super(`${i18nKey}: ${JSON.stringify(params)}`);
|
||||
this.name = "SrefValidationError";
|
||||
this.i18nKey = i18nKey;
|
||||
this.params = params;
|
||||
}
|
||||
}
|
||||
|
||||
// --- Data gathering ---
|
||||
|
|
@ -76,6 +168,45 @@ export async function getExportTransactions(): Promise<ExportTransaction[]> {
|
|||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Every configured import source, with its template resolved to a name.
|
||||
*
|
||||
* This is what the export was missing (#331): the file carried transactions but
|
||||
* not a single line of how they had been read, so restoring a backup meant
|
||||
* reconfiguring every source by hand.
|
||||
*/
|
||||
export async function getExportImportSources(): Promise<ExportImportSource[]> {
|
||||
const db = await getDb();
|
||||
const rows = await db.select<
|
||||
Array<ImportSource & { template_name: string | null }>
|
||||
>(
|
||||
`SELECT s.*, t.name AS template_name
|
||||
FROM import_sources s
|
||||
LEFT JOIN import_config_templates t ON s.template_id = t.id
|
||||
ORDER BY s.id`
|
||||
);
|
||||
return rows.map((row) => ({
|
||||
name: row.name,
|
||||
description: row.description ?? null,
|
||||
header_signature: row.header_signature ?? null,
|
||||
template_name: row.template_name ?? null,
|
||||
...pickFormatRow(row as unknown as Record<string, unknown>),
|
||||
}));
|
||||
}
|
||||
|
||||
export async function getExportImportTemplates(): Promise<
|
||||
ExportImportTemplate[]
|
||||
> {
|
||||
const db = await getDb();
|
||||
const rows = await db.select<ImportConfigTemplate[]>(
|
||||
"SELECT * FROM import_config_templates ORDER BY id"
|
||||
);
|
||||
return rows.map((row) => ({
|
||||
name: row.name,
|
||||
...pickFormatRow(row as unknown as Record<string, unknown>),
|
||||
}));
|
||||
}
|
||||
|
||||
// --- Serialization ---
|
||||
|
||||
export function serializeToJson(
|
||||
|
|
@ -86,6 +217,7 @@ export function serializeToJson(
|
|||
const envelope: ExportEnvelope = {
|
||||
export_type: exportType,
|
||||
app_version: appVersion,
|
||||
format_version: SREF_FORMAT_VERSION,
|
||||
exported_at: new Date().toISOString(),
|
||||
data,
|
||||
};
|
||||
|
|
@ -113,6 +245,49 @@ export function serializeTransactionsToCsv(
|
|||
|
||||
// --- Import parsing ---
|
||||
|
||||
/**
|
||||
* Refuse a configuration this build cannot store, BEFORE anything is deleted.
|
||||
*
|
||||
* Only the two enumerated fields are checked. `column_mapping` deliberately is
|
||||
* not decoded: a profile that already went through an old restore holds sources
|
||||
* with `'{}'` there, and refusing to restore a backup because of a row the app
|
||||
* itself wrote would strand the user with no way back in. It is carried through
|
||||
* as the opaque string the column stores, exactly as it was found — the wizard
|
||||
* validates it when it is actually about to read a file.
|
||||
*
|
||||
* PURE: no I/O, so it can be called from the parse step and again at the top of
|
||||
* each restore, which is where the guarantee has to hold.
|
||||
*/
|
||||
export function validateImportedFormatRows(
|
||||
sources: ExportImportSource[] | undefined,
|
||||
templates: ExportImportTemplate[] | undefined
|
||||
): void {
|
||||
const check = (row: ImportFormatRow, name: unknown) => {
|
||||
const label = typeof name === "string" && name.length > 0 ? name : "?";
|
||||
if (typeof row?.column_mapping !== "string") {
|
||||
throw new SrefValidationError(
|
||||
"settings.dataManagement.import.errors.invalidFormatRow",
|
||||
{ name: label }
|
||||
);
|
||||
}
|
||||
if (!AMOUNT_MODES.includes(row.amount_mode)) {
|
||||
throw new SrefValidationError(
|
||||
"settings.dataManagement.import.errors.unsupportedAmountMode",
|
||||
{ name: label, value: String(row.amount_mode) }
|
||||
);
|
||||
}
|
||||
if (!SIGN_CONVENTIONS.includes(row.sign_convention)) {
|
||||
throw new SrefValidationError(
|
||||
"settings.dataManagement.import.errors.unsupportedSignConvention",
|
||||
{ name: label, value: String(row.sign_convention) }
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
for (const template of templates ?? []) check(template, template?.name);
|
||||
for (const source of sources ?? []) check(source, source?.name);
|
||||
}
|
||||
|
||||
export function parseImportedJson(content: string): {
|
||||
envelope: ExportEnvelope;
|
||||
summary: ImportSummary;
|
||||
|
|
@ -141,6 +316,13 @@ export function parseImportedJson(content: string): {
|
|||
throw new Error(`Unknown export type: ${envelope.export_type}`);
|
||||
}
|
||||
|
||||
// Refuse an unreadable configuration while the file is only being LOOKED at:
|
||||
// the confirmation dialog never opens, so nothing is deleted.
|
||||
validateImportedFormatRows(
|
||||
envelope.data.import_sources,
|
||||
envelope.data.import_config_templates
|
||||
);
|
||||
|
||||
return {
|
||||
envelope,
|
||||
summary: {
|
||||
|
|
@ -149,6 +331,9 @@ export function parseImportedJson(content: string): {
|
|||
suppliersCount: envelope.data.suppliers?.length ?? 0,
|
||||
keywordsCount: envelope.data.keywords?.length ?? 0,
|
||||
transactionsCount: envelope.data.transactions?.length ?? 0,
|
||||
importSourcesCount: envelope.data.import_sources?.length ?? 0,
|
||||
importTemplatesCount: envelope.data.import_config_templates?.length ?? 0,
|
||||
formatVersion: envelope.format_version ?? LEGACY_SREF_FORMAT_VERSION,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -190,213 +375,347 @@ export function parseImportedCsv(content: string): {
|
|||
suppliersCount: 0,
|
||||
keywordsCount: 0,
|
||||
transactionsCount: transactions.length,
|
||||
// A flat CSV carries no configuration at all.
|
||||
importSourcesCount: 0,
|
||||
importTemplatesCount: 0,
|
||||
formatVersion: LEGACY_SREF_FORMAT_VERSION,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// --- Import execution ---
|
||||
|
||||
export async function importCategoriesOnly(data: ExportEnvelope["data"]): Promise<void> {
|
||||
const db = await getDb();
|
||||
/**
|
||||
* The source that hosts transactions restored from a file, which describe no
|
||||
* import folder of their own.
|
||||
*
|
||||
* Its mapping used to be `'{}'`, which `formatFromRow` cannot decode — a source
|
||||
* the app itself wrote and then refused to read. It describes the CSV this very
|
||||
* service exports instead: date, description, amount, comma-separated, ISO
|
||||
* dates, expenses negative. That is true of the file the rows came from, so the
|
||||
* wizard finds a usable format if a folder ever bears this name.
|
||||
*/
|
||||
const HOST_SOURCE_NAME = "Data Import";
|
||||
const HOST_SOURCE_MAPPING = JSON.stringify({
|
||||
date: 0,
|
||||
description: 1,
|
||||
amount: 2,
|
||||
});
|
||||
|
||||
// Wipe keywords, suppliers, categories
|
||||
await db.execute("DELETE FROM keywords");
|
||||
await db.execute("DELETE FROM suppliers");
|
||||
await db.execute("DELETE FROM categories");
|
||||
/**
|
||||
* Run `body` as one all-or-nothing restore.
|
||||
*
|
||||
* Every one of these functions starts by DELETING the profile's financial
|
||||
* history and then re-inserts it row by row, against tables carrying
|
||||
* `UNIQUE(name)` and a foreign key. Without a transaction, the first constraint
|
||||
* violation leaves the profile emptied at whatever row it reached, with nothing
|
||||
* to go back to. `withTransaction` holds the single DB lock for the whole
|
||||
* BEGIN..COMMIT so the statements cannot be split across pooled connections
|
||||
* (see `db.ts`); the BEGIN/COMMIT/ROLLBACK themselves are ours to issue.
|
||||
*/
|
||||
async function runRestore(
|
||||
body: (db: Database) => Promise<void>
|
||||
): Promise<void> {
|
||||
return withTransaction(async (db) => {
|
||||
await db.execute("BEGIN");
|
||||
let open = true;
|
||||
try {
|
||||
await body(db);
|
||||
await db.execute("COMMIT");
|
||||
open = false;
|
||||
} catch (e) {
|
||||
if (open) {
|
||||
try {
|
||||
await db.execute("ROLLBACK");
|
||||
} catch {
|
||||
// Preserve the original error — it is the one that explains the failure.
|
||||
}
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Nullify category/supplier references on transactions
|
||||
await db.execute(
|
||||
"UPDATE transactions SET category_id = NULL, supplier_id = NULL, is_manually_categorized = 0"
|
||||
async function restoreCategories(
|
||||
db: Database,
|
||||
categories: Category[] | undefined
|
||||
): Promise<void> {
|
||||
for (const cat of categories ?? []) {
|
||||
await db.execute(
|
||||
`INSERT INTO categories (id, name, parent_id, color, icon, type, is_active, is_inputable, sort_order)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
|
||||
[
|
||||
cat.id,
|
||||
cat.name,
|
||||
cat.parent_id ?? null,
|
||||
cat.color ?? null,
|
||||
cat.icon ?? null,
|
||||
cat.type,
|
||||
cat.is_active ? 1 : 0,
|
||||
cat.is_inputable ? 1 : 0,
|
||||
cat.sort_order,
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function restoreSuppliers(
|
||||
db: Database,
|
||||
suppliers: Supplier[] | undefined
|
||||
): Promise<void> {
|
||||
for (const sup of suppliers ?? []) {
|
||||
await db.execute(
|
||||
`INSERT INTO suppliers (id, name, normalized_name, category_id, is_active)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
[sup.id, sup.name, sup.normalized_name, sup.category_id ?? null, sup.is_active ? 1 : 0]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function restoreKeywords(
|
||||
db: Database,
|
||||
keywords: Keyword[] | undefined
|
||||
): Promise<void> {
|
||||
for (const kw of keywords ?? []) {
|
||||
await db.execute(
|
||||
`INSERT INTO keywords (id, keyword, category_id, supplier_id, priority, is_active)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)`,
|
||||
[kw.id, kw.keyword, kw.category_id, kw.supplier_id ?? null, kw.priority, kw.is_active ? 1 : 0]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore the templates and return their resolved ids, keyed by name.
|
||||
*
|
||||
* `import_config_templates` is in NEITHER wipe list, and restoring into a
|
||||
* profile that already has templates is the normal path, not an edge case — a
|
||||
* plain INSERT would hit `UNIQUE constraint failed: import_config_templates.name`
|
||||
* and (now) roll the whole restore back. Wiping the table instead was the other
|
||||
* option and was rejected: it would destroy templates the user never asked to
|
||||
* replace and that no confirmation dialog mentions. So: upsert by name, and give
|
||||
* the caller the map it needs to remap `import_sources.template_id`, since the
|
||||
* ids in the file mean nothing here.
|
||||
*
|
||||
* Templates go in BEFORE sources — `template_id` is a foreign key to this table.
|
||||
*/
|
||||
async function restoreImportTemplates(
|
||||
db: Database,
|
||||
templates: ExportImportTemplate[] | undefined
|
||||
): Promise<Map<string, number>> {
|
||||
const idsByName = new Map<string, number>();
|
||||
for (const template of templates ?? []) {
|
||||
await db.execute(
|
||||
`INSERT INTO import_config_templates (name, delimiter, encoding, date_format, skip_lines, has_header, column_mapping, amount_mode, sign_convention)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
|
||||
ON CONFLICT(name) DO UPDATE SET
|
||||
delimiter = excluded.delimiter,
|
||||
encoding = excluded.encoding,
|
||||
date_format = excluded.date_format,
|
||||
skip_lines = excluded.skip_lines,
|
||||
has_header = excluded.has_header,
|
||||
column_mapping = excluded.column_mapping,
|
||||
amount_mode = excluded.amount_mode,
|
||||
sign_convention = excluded.sign_convention`,
|
||||
[
|
||||
template.name,
|
||||
template.delimiter,
|
||||
template.encoding,
|
||||
template.date_format,
|
||||
template.skip_lines,
|
||||
template.has_header,
|
||||
template.column_mapping,
|
||||
template.amount_mode,
|
||||
template.sign_convention,
|
||||
]
|
||||
);
|
||||
// `lastInsertId` is 0 on the conflict branch, so the id is read back by
|
||||
// name — the same reason `importSourceService.createSource` does it.
|
||||
const rows = await db.select<Array<{ id: number }>>(
|
||||
"SELECT id FROM import_config_templates WHERE name = $1",
|
||||
[template.name]
|
||||
);
|
||||
if (rows.length > 0) idsByName.set(template.name, rows[0].id);
|
||||
}
|
||||
return idsByName;
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore the sources, resolving each `template_name` to the id the templates
|
||||
* pass just settled. A name with no match becomes NULL: `template_id` is a
|
||||
* provenance tag, never re-read as format, so losing it costs no configuration.
|
||||
*
|
||||
* Ids are not carried over — nothing in the envelope refers to a source by id.
|
||||
*/
|
||||
async function restoreImportSources(
|
||||
db: Database,
|
||||
sources: ExportImportSource[] | undefined,
|
||||
templateIds: Map<string, number>
|
||||
): Promise<void> {
|
||||
for (const source of sources ?? []) {
|
||||
await db.execute(
|
||||
`INSERT INTO import_sources (name, description, date_format, delimiter, encoding, column_mapping, skip_lines, has_header, amount_mode, sign_convention, header_signature, template_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)`,
|
||||
[
|
||||
source.name,
|
||||
source.description ?? null,
|
||||
source.date_format,
|
||||
source.delimiter,
|
||||
source.encoding,
|
||||
source.column_mapping,
|
||||
source.skip_lines,
|
||||
source.has_header,
|
||||
source.amount_mode,
|
||||
source.sign_convention,
|
||||
source.header_signature ?? null,
|
||||
source.template_name !== null
|
||||
? templateIds.get(source.template_name) ?? null
|
||||
: null,
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Attach the restored transactions to a source and one `imported_files` row.
|
||||
*
|
||||
* The host source is created ONLY here, and only when there are transactions to
|
||||
* hang off it — it exists to satisfy `transactions.source_id`, not to stand in
|
||||
* for the configurations the file now carries on its own. A restored source may
|
||||
* already bear the name (an earlier restore of this same profile wrote one), in
|
||||
* which case it is reused rather than re-inserted: `import_sources.name` is
|
||||
* UNIQUE, and a collision here would roll back the entire restore.
|
||||
*/
|
||||
async function attachTransactions(
|
||||
db: Database,
|
||||
transactions: ExportTransaction[] | undefined,
|
||||
filename: string
|
||||
): Promise<void> {
|
||||
if (!transactions || transactions.length === 0) return;
|
||||
|
||||
const existing = await db.select<Array<{ id: number }>>(
|
||||
"SELECT id FROM import_sources WHERE name = $1",
|
||||
[HOST_SOURCE_NAME]
|
||||
);
|
||||
|
||||
// Re-insert categories
|
||||
if (data.categories) {
|
||||
for (const cat of data.categories) {
|
||||
await db.execute(
|
||||
`INSERT INTO categories (id, name, parent_id, color, icon, type, is_active, is_inputable, sort_order)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
|
||||
[
|
||||
cat.id,
|
||||
cat.name,
|
||||
cat.parent_id ?? null,
|
||||
cat.color ?? null,
|
||||
cat.icon ?? null,
|
||||
cat.type,
|
||||
cat.is_active ? 1 : 0,
|
||||
cat.is_inputable ? 1 : 0,
|
||||
cat.sort_order,
|
||||
]
|
||||
);
|
||||
}
|
||||
let sourceId: number;
|
||||
if (existing.length > 0) {
|
||||
sourceId = existing[0].id;
|
||||
} else {
|
||||
const sourceResult = await db.execute(
|
||||
`INSERT INTO import_sources (name, description, date_format, delimiter, encoding, column_mapping, skip_lines, has_header, amount_mode, sign_convention)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
|
||||
[
|
||||
HOST_SOURCE_NAME,
|
||||
"Imported from settings",
|
||||
"%Y-%m-%d",
|
||||
",",
|
||||
"utf-8",
|
||||
HOST_SOURCE_MAPPING,
|
||||
0,
|
||||
1,
|
||||
"single",
|
||||
"negative_expense",
|
||||
]
|
||||
);
|
||||
sourceId = sourceResult.lastInsertId as number;
|
||||
}
|
||||
|
||||
// Re-insert suppliers
|
||||
if (data.suppliers) {
|
||||
for (const sup of data.suppliers) {
|
||||
await db.execute(
|
||||
`INSERT INTO suppliers (id, name, normalized_name, category_id, is_active)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
[sup.id, sup.name, sup.normalized_name, sup.category_id ?? null, sup.is_active ? 1 : 0]
|
||||
);
|
||||
}
|
||||
}
|
||||
const fileResult = await db.execute(
|
||||
`INSERT INTO imported_files (source_id, filename, file_hash, row_count, status)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
[sourceId, filename, `data-import-${Date.now()}`, transactions.length, "completed"]
|
||||
);
|
||||
const fileId = fileResult.lastInsertId;
|
||||
|
||||
// Re-insert keywords
|
||||
if (data.keywords) {
|
||||
for (const kw of data.keywords) {
|
||||
await db.execute(
|
||||
`INSERT INTO keywords (id, keyword, category_id, supplier_id, priority, is_active)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)`,
|
||||
[kw.id, kw.keyword, kw.category_id, kw.supplier_id ?? null, kw.priority, kw.is_active ? 1 : 0]
|
||||
);
|
||||
}
|
||||
for (const tx of transactions) {
|
||||
await db.execute(
|
||||
`INSERT INTO transactions (date, description, amount, category_id, original_description, notes, is_manually_categorized, is_split, parent_transaction_id, source_id, file_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
|
||||
[
|
||||
tx.date,
|
||||
tx.description,
|
||||
tx.amount,
|
||||
tx.category_id,
|
||||
tx.original_description,
|
||||
tx.notes,
|
||||
tx.is_manually_categorized,
|
||||
tx.is_split,
|
||||
tx.parent_transaction_id,
|
||||
sourceId,
|
||||
fileId,
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export async function importCategoriesOnly(
|
||||
data: ExportEnvelope["data"]
|
||||
): Promise<void> {
|
||||
return runRestore(async (db) => {
|
||||
// Wipe keywords, suppliers, categories
|
||||
await db.execute("DELETE FROM keywords");
|
||||
await db.execute("DELETE FROM suppliers");
|
||||
await db.execute("DELETE FROM categories");
|
||||
|
||||
// Nullify category/supplier references on transactions
|
||||
await db.execute(
|
||||
"UPDATE transactions SET category_id = NULL, supplier_id = NULL, is_manually_categorized = 0"
|
||||
);
|
||||
|
||||
await restoreCategories(db, data.categories);
|
||||
await restoreSuppliers(db, data.suppliers);
|
||||
await restoreKeywords(db, data.keywords);
|
||||
});
|
||||
}
|
||||
|
||||
export async function importTransactionsWithCategories(
|
||||
data: ExportEnvelope["data"],
|
||||
filename: string
|
||||
): Promise<void> {
|
||||
const db = await getDb();
|
||||
validateImportedFormatRows(data.import_sources, data.import_config_templates);
|
||||
|
||||
// Wipe everything
|
||||
await db.execute("DELETE FROM transactions");
|
||||
await db.execute("DELETE FROM imported_files");
|
||||
await db.execute("DELETE FROM import_sources");
|
||||
await db.execute("DELETE FROM keywords");
|
||||
await db.execute("DELETE FROM suppliers");
|
||||
await db.execute("DELETE FROM categories");
|
||||
return runRestore(async (db) => {
|
||||
// Wipe everything
|
||||
await db.execute("DELETE FROM transactions");
|
||||
await db.execute("DELETE FROM imported_files");
|
||||
await db.execute("DELETE FROM import_sources");
|
||||
await db.execute("DELETE FROM keywords");
|
||||
await db.execute("DELETE FROM suppliers");
|
||||
await db.execute("DELETE FROM categories");
|
||||
|
||||
// Re-insert categories
|
||||
if (data.categories) {
|
||||
for (const cat of data.categories) {
|
||||
await db.execute(
|
||||
`INSERT INTO categories (id, name, parent_id, color, icon, type, is_active, is_inputable, sort_order)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
|
||||
[
|
||||
cat.id,
|
||||
cat.name,
|
||||
cat.parent_id ?? null,
|
||||
cat.color ?? null,
|
||||
cat.icon ?? null,
|
||||
cat.type,
|
||||
cat.is_active ? 1 : 0,
|
||||
cat.is_inputable ? 1 : 0,
|
||||
cat.sort_order,
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
await restoreCategories(db, data.categories);
|
||||
await restoreSuppliers(db, data.suppliers);
|
||||
await restoreKeywords(db, data.keywords);
|
||||
|
||||
// Re-insert suppliers
|
||||
if (data.suppliers) {
|
||||
for (const sup of data.suppliers) {
|
||||
await db.execute(
|
||||
`INSERT INTO suppliers (id, name, normalized_name, category_id, is_active)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
[sup.id, sup.name, sup.normalized_name, sup.category_id ?? null, sup.is_active ? 1 : 0]
|
||||
);
|
||||
}
|
||||
}
|
||||
// Templates first: `import_sources.template_id` points at them.
|
||||
const templateIds = await restoreImportTemplates(
|
||||
db,
|
||||
data.import_config_templates
|
||||
);
|
||||
await restoreImportSources(db, data.import_sources, templateIds);
|
||||
|
||||
// Re-insert keywords
|
||||
if (data.keywords) {
|
||||
for (const kw of data.keywords) {
|
||||
await db.execute(
|
||||
`INSERT INTO keywords (id, keyword, category_id, supplier_id, priority, is_active)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)`,
|
||||
[kw.id, kw.keyword, kw.category_id, kw.supplier_id ?? null, kw.priority, kw.is_active ? 1 : 0]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Create tracking records for import history
|
||||
const sourceResult = await db.execute(
|
||||
`INSERT INTO import_sources (name, description, date_format, delimiter, encoding, column_mapping, skip_lines)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)`,
|
||||
["Data Import", "Imported from settings", "%Y-%m-%d", ",", "utf-8", "{}", 0]
|
||||
);
|
||||
const sourceId = sourceResult.lastInsertId;
|
||||
|
||||
const txCount = data.transactions?.length ?? 0;
|
||||
const fileResult = await db.execute(
|
||||
`INSERT INTO imported_files (source_id, filename, file_hash, row_count, status)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
[sourceId, filename, `data-import-${Date.now()}`, txCount, "completed"]
|
||||
);
|
||||
const fileId = fileResult.lastInsertId;
|
||||
|
||||
// Re-insert transactions linked to the import
|
||||
if (data.transactions) {
|
||||
for (const tx of data.transactions) {
|
||||
await db.execute(
|
||||
`INSERT INTO transactions (date, description, amount, category_id, original_description, notes, is_manually_categorized, is_split, parent_transaction_id, source_id, file_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
|
||||
[
|
||||
tx.date,
|
||||
tx.description,
|
||||
tx.amount,
|
||||
tx.category_id,
|
||||
tx.original_description,
|
||||
tx.notes,
|
||||
tx.is_manually_categorized,
|
||||
tx.is_split,
|
||||
tx.parent_transaction_id,
|
||||
sourceId,
|
||||
fileId,
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
await attachTransactions(db, data.transactions, filename);
|
||||
});
|
||||
}
|
||||
|
||||
export async function importTransactionsOnly(
|
||||
data: ExportEnvelope["data"],
|
||||
filename: string
|
||||
): Promise<void> {
|
||||
const db = await getDb();
|
||||
validateImportedFormatRows(data.import_sources, data.import_config_templates);
|
||||
|
||||
// Wipe transactions and import history
|
||||
await db.execute("DELETE FROM transactions");
|
||||
await db.execute("DELETE FROM imported_files");
|
||||
await db.execute("DELETE FROM import_sources");
|
||||
return runRestore(async (db) => {
|
||||
// Wipe transactions and import history
|
||||
await db.execute("DELETE FROM transactions");
|
||||
await db.execute("DELETE FROM imported_files");
|
||||
await db.execute("DELETE FROM import_sources");
|
||||
|
||||
// Create tracking records for import history
|
||||
const sourceResult = await db.execute(
|
||||
`INSERT INTO import_sources (name, description, date_format, delimiter, encoding, column_mapping, skip_lines)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)`,
|
||||
["Data Import", "Imported from settings", "%Y-%m-%d", ",", "utf-8", "{}", 0]
|
||||
);
|
||||
const sourceId = sourceResult.lastInsertId;
|
||||
const templateIds = await restoreImportTemplates(
|
||||
db,
|
||||
data.import_config_templates
|
||||
);
|
||||
await restoreImportSources(db, data.import_sources, templateIds);
|
||||
|
||||
const txCount = data.transactions?.length ?? 0;
|
||||
const fileResult = await db.execute(
|
||||
`INSERT INTO imported_files (source_id, filename, file_hash, row_count, status)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
[sourceId, filename, `data-import-${Date.now()}`, txCount, "completed"]
|
||||
);
|
||||
const fileId = fileResult.lastInsertId;
|
||||
|
||||
// Re-insert transactions linked to the import
|
||||
if (data.transactions) {
|
||||
for (const tx of data.transactions) {
|
||||
await db.execute(
|
||||
`INSERT INTO transactions (date, description, amount, category_id, original_description, notes, is_manually_categorized, is_split, parent_transaction_id, source_id, file_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
|
||||
[
|
||||
tx.date,
|
||||
tx.description,
|
||||
tx.amount,
|
||||
tx.category_id,
|
||||
tx.original_description,
|
||||
tx.notes,
|
||||
tx.is_manually_categorized,
|
||||
tx.is_split,
|
||||
tx.parent_transaction_id,
|
||||
sourceId,
|
||||
fileId,
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
await attachTransactions(db, data.transactions, filename);
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,15 @@
|
|||
import { getDb } from "./db";
|
||||
import type { ImportConfigTemplate } from "../shared/types";
|
||||
import type { ImportConfigTemplate, ImportFormatRow } from "../shared/types";
|
||||
|
||||
/**
|
||||
* A template is a named format, nothing more. Taking `ImportFormatRow` here —
|
||||
* the same shape `importSourceService` takes — is what keeps both writers on
|
||||
* the codec: a field added to the format cannot reach one table and miss the
|
||||
* other.
|
||||
*/
|
||||
export interface ImportTemplateInput extends ImportFormatRow {
|
||||
name: string;
|
||||
}
|
||||
|
||||
export async function getAllTemplates(): Promise<ImportConfigTemplate[]> {
|
||||
const db = await getDb();
|
||||
|
|
@ -9,7 +19,7 @@ export async function getAllTemplates(): Promise<ImportConfigTemplate[]> {
|
|||
}
|
||||
|
||||
export async function createTemplate(
|
||||
template: Omit<ImportConfigTemplate, "id" | "created_at">
|
||||
template: ImportTemplateInput
|
||||
): Promise<number> {
|
||||
const db = await getDb();
|
||||
const result = await db.execute(
|
||||
|
|
@ -30,9 +40,14 @@ export async function createTemplate(
|
|||
return result.lastInsertId as number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Editing a template touches `import_config_templates` and nothing else. A
|
||||
* source that was configured from this template keeps its own eight columns —
|
||||
* `import_sources.template_id` is a provenance tag, never re-read as format.
|
||||
*/
|
||||
export async function updateTemplate(
|
||||
id: number,
|
||||
template: Omit<ImportConfigTemplate, "id" | "created_at">
|
||||
template: ImportTemplateInput
|
||||
): Promise<void> {
|
||||
const db = await getDb();
|
||||
await db.execute(
|
||||
|
|
|
|||
|
|
@ -1,5 +1,24 @@
|
|||
import { getDb } from "./db";
|
||||
import type { ImportSource } from "../shared/types";
|
||||
import type { ImportFormatRow, ImportSource } from "../shared/types";
|
||||
|
||||
/**
|
||||
* What a writer hands to this service: the eight format fields in their
|
||||
* persisted shape (produced by `formatToRow` — never assembled by hand) plus
|
||||
* the source's own columns.
|
||||
*
|
||||
* Note it is NOT `Omit<ImportSource, "id" | ...>`: `ImportSource.has_header`
|
||||
* is declared boolean while the codec emits the 0/1 SQLite actually stores.
|
||||
* Taking `ImportFormatRow` here is what makes the codec the only place that
|
||||
* normalization happens.
|
||||
*/
|
||||
export interface ImportSourceInput extends ImportFormatRow {
|
||||
name: string;
|
||||
description?: string | null;
|
||||
/** Drift-detection metadata, written by #330. */
|
||||
header_signature?: string | null;
|
||||
/** Provenance tag — recorded and displayed, never re-read as format. */
|
||||
template_id?: number | null;
|
||||
}
|
||||
|
||||
export async function getAllSources(): Promise<ImportSource[]> {
|
||||
const db = await getDb();
|
||||
|
|
@ -29,12 +48,12 @@ export async function getSourceById(
|
|||
}
|
||||
|
||||
export async function createSource(
|
||||
source: Omit<ImportSource, "id" | "created_at" | "updated_at">
|
||||
source: ImportSourceInput
|
||||
): Promise<number> {
|
||||
const db = await getDb();
|
||||
const result = await db.execute(
|
||||
`INSERT INTO import_sources (name, description, date_format, delimiter, encoding, column_mapping, skip_lines, has_header)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
||||
`INSERT INTO import_sources (name, description, date_format, delimiter, encoding, column_mapping, skip_lines, has_header, amount_mode, sign_convention, header_signature, template_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
|
||||
ON CONFLICT(name) DO UPDATE SET
|
||||
description = excluded.description,
|
||||
date_format = excluded.date_format,
|
||||
|
|
@ -43,16 +62,27 @@ export async function createSource(
|
|||
column_mapping = excluded.column_mapping,
|
||||
skip_lines = excluded.skip_lines,
|
||||
has_header = excluded.has_header,
|
||||
amount_mode = excluded.amount_mode,
|
||||
sign_convention = excluded.sign_convention,
|
||||
header_signature = excluded.header_signature,
|
||||
template_id = excluded.template_id,
|
||||
updated_at = CURRENT_TIMESTAMP`,
|
||||
[
|
||||
source.name,
|
||||
source.description || null,
|
||||
source.description ?? null,
|
||||
source.date_format,
|
||||
source.delimiter,
|
||||
source.encoding,
|
||||
source.column_mapping,
|
||||
source.skip_lines,
|
||||
source.has_header ? 1 : 0,
|
||||
source.has_header,
|
||||
source.amount_mode,
|
||||
source.sign_convention,
|
||||
// Explicitly null for a headerless file: drift detection is inoperative
|
||||
// there and a stale signature from an earlier shape would be worse than
|
||||
// none (#330).
|
||||
source.header_signature ?? null,
|
||||
source.template_id ?? null,
|
||||
]
|
||||
);
|
||||
// On conflict, lastInsertId may be 0 — look up the existing row
|
||||
|
|
@ -63,45 +93,30 @@ export async function createSource(
|
|||
|
||||
export async function updateSource(
|
||||
id: number,
|
||||
source: Partial<Omit<ImportSource, "id" | "created_at" | "updated_at">>
|
||||
source: Partial<ImportSourceInput>
|
||||
): Promise<void> {
|
||||
const db = await getDb();
|
||||
const fields: string[] = [];
|
||||
const values: unknown[] = [];
|
||||
let paramIndex = 1;
|
||||
|
||||
if (source.name !== undefined) {
|
||||
fields.push(`name = $${paramIndex++}`);
|
||||
values.push(source.name);
|
||||
}
|
||||
if (source.description !== undefined) {
|
||||
fields.push(`description = $${paramIndex++}`);
|
||||
values.push(source.description);
|
||||
}
|
||||
if (source.date_format !== undefined) {
|
||||
fields.push(`date_format = $${paramIndex++}`);
|
||||
values.push(source.date_format);
|
||||
}
|
||||
if (source.delimiter !== undefined) {
|
||||
fields.push(`delimiter = $${paramIndex++}`);
|
||||
values.push(source.delimiter);
|
||||
}
|
||||
if (source.encoding !== undefined) {
|
||||
fields.push(`encoding = $${paramIndex++}`);
|
||||
values.push(source.encoding);
|
||||
}
|
||||
if (source.column_mapping !== undefined) {
|
||||
fields.push(`column_mapping = $${paramIndex++}`);
|
||||
values.push(source.column_mapping);
|
||||
}
|
||||
if (source.skip_lines !== undefined) {
|
||||
fields.push(`skip_lines = $${paramIndex++}`);
|
||||
values.push(source.skip_lines);
|
||||
}
|
||||
if (source.has_header !== undefined) {
|
||||
fields.push(`has_header = $${paramIndex++}`);
|
||||
values.push(source.has_header ? 1 : 0);
|
||||
}
|
||||
const setColumn = (column: string, value: unknown) => {
|
||||
fields.push(`${column} = $${paramIndex++}`);
|
||||
values.push(value);
|
||||
};
|
||||
|
||||
if (source.name !== undefined) setColumn("name", source.name);
|
||||
if (source.description !== undefined) setColumn("description", source.description);
|
||||
if (source.date_format !== undefined) setColumn("date_format", source.date_format);
|
||||
if (source.delimiter !== undefined) setColumn("delimiter", source.delimiter);
|
||||
if (source.encoding !== undefined) setColumn("encoding", source.encoding);
|
||||
if (source.column_mapping !== undefined) setColumn("column_mapping", source.column_mapping);
|
||||
if (source.skip_lines !== undefined) setColumn("skip_lines", source.skip_lines);
|
||||
if (source.has_header !== undefined) setColumn("has_header", source.has_header);
|
||||
if (source.amount_mode !== undefined) setColumn("amount_mode", source.amount_mode);
|
||||
if (source.sign_convention !== undefined) setColumn("sign_convention", source.sign_convention);
|
||||
if (source.header_signature !== undefined) setColumn("header_signature", source.header_signature);
|
||||
if (source.template_id !== undefined) setColumn("template_id", source.template_id);
|
||||
|
||||
if (fields.length === 0) return;
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,28 @@ export interface ImportSource {
|
|||
column_mapping: string;
|
||||
skip_lines: number;
|
||||
has_header: boolean;
|
||||
/**
|
||||
* The two fields that decide how an amount is READ, added by migration v17.
|
||||
* Before v17 they lived only on `import_config_templates`, so restoring a
|
||||
* source re-inferred the mode and hardcoded the convention (#324).
|
||||
*
|
||||
* They are never read directly: `formatFromRow` is the single conversion
|
||||
* point and validates them, because the column `CHECK` admits
|
||||
* `absolute_indicator` — a value the app cannot map today.
|
||||
*/
|
||||
amount_mode: AmountMode;
|
||||
sign_convention: SignConvention;
|
||||
/**
|
||||
* Normalized header labels seen at the last successful import, for drift
|
||||
* detection. NOT a format field: written by #330, not by the codec.
|
||||
*/
|
||||
header_signature?: string | null;
|
||||
/**
|
||||
* Provenance tag only — records which template this source was configured
|
||||
* from, and is NEVER re-read as format. The eight format fields above are
|
||||
* authoritative, so editing a template alters no linked source.
|
||||
*/
|
||||
template_id?: number | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
|
@ -154,17 +176,14 @@ export interface BudgetYearRow {
|
|||
previousYearTotal: number; // actual (transactions) total from the previous year
|
||||
}
|
||||
|
||||
export interface ImportConfigTemplate {
|
||||
/**
|
||||
* A named, reusable format. It carries exactly the eight persisted format
|
||||
* fields (via `ImportFormatRow`) plus its identity — the same eight an
|
||||
* `import_sources` row carries since v17.
|
||||
*/
|
||||
export interface ImportConfigTemplate extends ImportFormatRow {
|
||||
id: number;
|
||||
name: string;
|
||||
delimiter: string;
|
||||
encoding: string;
|
||||
date_format: string;
|
||||
skip_lines: number;
|
||||
has_header: number;
|
||||
column_mapping: string;
|
||||
amount_mode: AmountMode;
|
||||
sign_convention: SignConvention;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
|
|
@ -213,16 +232,57 @@ export interface ColumnMapping {
|
|||
export type AmountMode = "single" | "debit_credit";
|
||||
export type SignConvention = "negative_expense" | "positive_expense";
|
||||
|
||||
export interface SourceConfig {
|
||||
name: string;
|
||||
/**
|
||||
* --- The import format ------------------------------------------------------
|
||||
*
|
||||
* The eight fields that fully decide how a CSV file is read. They exist in two
|
||||
* shapes that CANNOT be a single composed type: the persisted rows are
|
||||
* snake_case with the mapping serialized to JSON and no boolean type in SQLite,
|
||||
* while the wizard works in camelCase on a parsed mapping.
|
||||
*
|
||||
* The guarantee that no field is ever half-persisted therefore does not come
|
||||
* from the type structure but from `src/utils/importFormat.ts`, the SINGLE
|
||||
* conversion point between the two shapes, and from its completeness test.
|
||||
*/
|
||||
|
||||
/** Persisted shape — snake_case, mapping as JSON, `has_header` normalized to 0/1. */
|
||||
export interface ImportFormatRow {
|
||||
delimiter: string;
|
||||
encoding: string;
|
||||
date_format: string;
|
||||
skip_lines: number;
|
||||
/** SQLite has no boolean: written as 0/1. Reads tolerate both, see `ImportFormatRowInput`. */
|
||||
has_header: number;
|
||||
column_mapping: string;
|
||||
amount_mode: AmountMode;
|
||||
sign_convention: SignConvention;
|
||||
}
|
||||
|
||||
/**
|
||||
* What `formatFromRow` accepts. `import_sources` rows declare `has_header` as a
|
||||
* boolean and `import_config_templates` rows as a number; both are the same 0/1
|
||||
* integer at runtime, and the codec normalizes it. Widening the field here is
|
||||
* what lets a row from either table be decoded without a cast.
|
||||
*/
|
||||
export type ImportFormatRowInput = Omit<ImportFormatRow, "has_header"> & {
|
||||
has_header: number | boolean;
|
||||
};
|
||||
|
||||
/** Domain shape — camelCase, mapping parsed. What the wizard manipulates. */
|
||||
export interface ImportFormat {
|
||||
delimiter: string;
|
||||
encoding: string;
|
||||
dateFormat: string;
|
||||
skipLines: number;
|
||||
hasHeader: boolean;
|
||||
columnMapping: ColumnMapping;
|
||||
amountMode: AmountMode;
|
||||
signConvention: SignConvention;
|
||||
hasHeader: boolean;
|
||||
}
|
||||
|
||||
/** A format plus the source it belongs to. */
|
||||
export interface SourceConfig extends ImportFormat {
|
||||
name: string;
|
||||
}
|
||||
|
||||
export interface ParsedRow {
|
||||
|
|
|
|||
253
src/utils/amountParser.test.ts
Normal file
253
src/utils/amountParser.test.ts
Normal file
|
|
@ -0,0 +1,253 @@
|
|||
// amountParser — characterization tests (#326), hardened (#325).
|
||||
//
|
||||
// `parseFrenchAmount` had no test at all, on a codebase of 871. It is called
|
||||
// from 11 sites (8 in `csvAutoDetect.ts`, 3 in `useSnapshotEditor.ts`) and sits
|
||||
// under every imported amount, so #326 pinned what it did before #325 touched
|
||||
// it. The `KNOWN DEFECT` blocks #326 left here have since flipped: the
|
||||
// expectations were updated in place and the markers dropped, per the standing
|
||||
// rule (update, never delete).
|
||||
//
|
||||
// The defect that motivated the whole chantier: `parseFrenchAmount` ended on
|
||||
// `parseFloat`, which stops at the first invalid character instead of rejecting
|
||||
// the string. A trailing unit or sign therefore yielded a magnitude off by a
|
||||
// factor of 100 — and it passed `isNaN`, so it counted as a VALID row
|
||||
// everywhere downstream. `"100,00 CAD"` did not fail; it imported as 10 000.
|
||||
// Validation is anchored now and such a cell is NaN, i.e. a visible row error.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { detectDecimalSeparator, parseFrenchAmount } from "./amountParser";
|
||||
import { autoDetectConfig } from "./csvAutoDetect";
|
||||
import {
|
||||
buildDetailedLines,
|
||||
holdingsFromCsvRows,
|
||||
} from "../hooks/useSnapshotEditor";
|
||||
import { BalanceServiceError } from "../services/balance.service";
|
||||
import { readCsvFixture } from "../__fixtures__/csv";
|
||||
|
||||
describe("parseFrenchAmount — separator contract (#326)", () => {
|
||||
it("reads the French decimal comma", () => {
|
||||
expect(parseFrenchAmount("1234,56")).toBe(1234.56);
|
||||
expect(parseFrenchAmount("12,5")).toBe(12.5);
|
||||
expect(parseFrenchAmount("0,00")).toBe(0);
|
||||
});
|
||||
|
||||
it("reads French thousand separators (space, non-breaking space, dot)", () => {
|
||||
expect(parseFrenchAmount("1 234,56")).toBe(1234.56);
|
||||
expect(parseFrenchAmount("1\u00A0234,56")).toBe(1234.56); // non-breaking space
|
||||
expect(parseFrenchAmount("1.234,56")).toBe(1234.56);
|
||||
});
|
||||
|
||||
it("reads English notation", () => {
|
||||
expect(parseFrenchAmount("1234.56")).toBe(1234.56);
|
||||
expect(parseFrenchAmount("1,234.56")).toBe(1234.56);
|
||||
});
|
||||
|
||||
it("keeps the sign of a leading minus", () => {
|
||||
expect(parseFrenchAmount("-84,32")).toBe(-84.32);
|
||||
expect(parseFrenchAmount(" -84,32 ")).toBe(-84.32);
|
||||
});
|
||||
|
||||
it("strips currency symbols on either side", () => {
|
||||
expect(parseFrenchAmount("1 250,00 $")).toBe(1250);
|
||||
expect(parseFrenchAmount("$1,250.00")).toBe(1250);
|
||||
expect(parseFrenchAmount("€84,32")).toBe(84.32);
|
||||
expect(parseFrenchAmount("£84,32")).toBe(84.32);
|
||||
});
|
||||
|
||||
it("rejects blank and non-numeric input", () => {
|
||||
expect(parseFrenchAmount("")).toBeNaN();
|
||||
expect(parseFrenchAmount(" ")).toBeNaN();
|
||||
expect(parseFrenchAmount("abc")).toBeNaN();
|
||||
expect(parseFrenchAmount("-")).toBeNaN();
|
||||
expect(parseFrenchAmount("--5")).toBeNaN();
|
||||
// Letter-leading labels are rejected — this is what keeps `detectHeader`
|
||||
// working on a header such as "Solde 2024" (see the twin defect below).
|
||||
expect(parseFrenchAmount("Solde 2024")).toBeNaN();
|
||||
// Non-string input is guarded before any parsing.
|
||||
expect(parseFrenchAmount(undefined as unknown as string)).toBeNaN();
|
||||
expect(parseFrenchAmount(42 as unknown as string)).toBeNaN();
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseFrenchAmount — anchored validation (#325, was a KNOWN DEFECT of #326)", () => {
|
||||
// FIXED. `parseFloat` used to return the longest valid PREFIX instead of
|
||||
// rejecting the string; validation is anchored over the whole normalized
|
||||
// value now, so any residual character yields NaN.
|
||||
//
|
||||
// Note on the two currency/indicator cases: link 1 wrote "should be 1234.56"
|
||||
// and "should be 100" in the titles, while the block header it wrote just
|
||||
// above said "all of these then become NaN, except the accounting-parenthesis
|
||||
// and trailing-sign forms". The header is what #325 implements, for two
|
||||
// reasons the titles missed. `CR`/`DB` carry a DIRECTION, so returning a
|
||||
// magnitude for both would replace a loud failure with a silent SIGN error —
|
||||
// and the D/C-indicator shape is refused upstream by design (#328). And
|
||||
// "100,00 CAD" is structurally identical to "2025 Montant": rescuing the
|
||||
// first by whitelisting a trailing word re-blinds `detectHeader` on the
|
||||
// second, which is the very defect measured below.
|
||||
|
||||
it("reads a trailing sign as a negative amount", () => {
|
||||
// "50,00-" is the trailing-minus convention of several bank exports.
|
||||
expect(parseFrenchAmount("50,00-")).toBe(-50);
|
||||
expect(parseFrenchAmount("1 234,56-")).toBe(-1234.56);
|
||||
expect(parseFrenchAmount("50,00+")).toBe(50);
|
||||
});
|
||||
|
||||
it("rejects a trailing direction indicator instead of guessing a sign", () => {
|
||||
expect(parseFrenchAmount("1 234,56 CR")).toBeNaN();
|
||||
expect(parseFrenchAmount("1 234,56 DB")).toBeNaN();
|
||||
});
|
||||
|
||||
it("rejects a trailing currency code", () => {
|
||||
expect(parseFrenchAmount("100,00 CAD")).toBeNaN();
|
||||
expect(parseFrenchAmount("84,32 USD")).toBeNaN();
|
||||
});
|
||||
|
||||
it("rejects the numeric prefix of a text label", () => {
|
||||
// This is what used to blind `detectHeader`: a header cell STARTING with
|
||||
// digits read as a number, so the header row was taken for data.
|
||||
expect(parseFrenchAmount("2024 Montant")).toBeNaN();
|
||||
expect(parseFrenchAmount("5%")).toBeNaN();
|
||||
});
|
||||
|
||||
it("rejects JavaScript number literals a bank never emits", () => {
|
||||
expect(parseFrenchAmount("1e3")).toBeNaN();
|
||||
expect(parseFrenchAmount("Infinity")).toBeNaN();
|
||||
expect(parseFrenchAmount("0x1F")).toBeNaN();
|
||||
});
|
||||
|
||||
it("rejects a malformed number instead of keeping its first two groups", () => {
|
||||
expect(parseFrenchAmount("1,2,3")).toBeNaN();
|
||||
expect(parseFrenchAmount("1.2.3")).toBeNaN();
|
||||
expect(parseFrenchAmount("1,23,456")).toBeNaN(); // groups must be 3 digits
|
||||
});
|
||||
|
||||
it("rejects two signs, wherever they sit", () => {
|
||||
expect(parseFrenchAmount("-50,00-")).toBeNaN();
|
||||
expect(parseFrenchAmount("(-50,00)")).toBeNaN();
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseFrenchAmount — accounting forms (#325, was a KNOWN DEFECT of #326)", () => {
|
||||
it("reads accounting parentheses as a negative amount", () => {
|
||||
expect(parseFrenchAmount("(50,00)")).toBe(-50);
|
||||
expect(parseFrenchAmount("(1 234,56)")).toBe(-1234.56);
|
||||
expect(parseFrenchAmount("(1,234.56)")).toBe(-1234.56);
|
||||
});
|
||||
|
||||
it("still rejects an unbalanced parenthesis", () => {
|
||||
expect(parseFrenchAmount("(50,00")).toBeNaN();
|
||||
expect(parseFrenchAmount("50,00)")).toBeNaN();
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseFrenchAmount — column-level arbitration (#325, was a KNOWN DEFECT of #326)", () => {
|
||||
// The French-vs-English decision used to be taken on each cell in isolation
|
||||
// and NOTHING else, so two cells of the same column could be read under two
|
||||
// different conventions. The isolated readings below are unchanged — they
|
||||
// are the best a lone cell allows — but a caller that knows the column now
|
||||
// passes its verdict and settles the ambiguity.
|
||||
|
||||
it("keeps the documented reading when no column context is given", () => {
|
||||
// "12,345" is 12.345 in a column of decimals, 12345 in a column of
|
||||
// thousands. Nothing in the cell ALONE can tell.
|
||||
expect(parseFrenchAmount("12,345")).toBe(12345);
|
||||
expect(parseFrenchAmount("1.234")).toBe(1.234);
|
||||
expect(parseFrenchAmount("1.234,56")).toBe(1234.56); // ...unless a comma follows
|
||||
});
|
||||
|
||||
it("obeys the column verdict when there is one", () => {
|
||||
expect(parseFrenchAmount("12,345", { decimalSeparator: "," })).toBe(12.345);
|
||||
expect(parseFrenchAmount("12,345", { decimalSeparator: "." })).toBe(12345);
|
||||
expect(parseFrenchAmount("1.234", { decimalSeparator: "," })).toBe(1234);
|
||||
expect(parseFrenchAmount("1.234", { decimalSeparator: "." })).toBe(1.234);
|
||||
});
|
||||
|
||||
it("still rejects a value that contradicts the column verdict", () => {
|
||||
// A grouping separator groups by three, always.
|
||||
expect(parseFrenchAmount("1.23", { decimalSeparator: "," })).toBeNaN();
|
||||
expect(parseFrenchAmount("1,23", { decimalSeparator: "." })).toBeNaN();
|
||||
});
|
||||
|
||||
it("arbitrates a column from a decisive sibling cell", () => {
|
||||
// "1.234" alone is ambiguous; "84,32" in the same column is not.
|
||||
expect(detectDecimalSeparator(["1.234", "84,32", "-6,95"])).toBe(",");
|
||||
expect(detectDecimalSeparator(["1,234", "84.32", "-6.95"])).toBe(".");
|
||||
// Mixed notation settles on the last separator of each decisive cell.
|
||||
expect(detectDecimalSeparator(["1.234,56"])).toBe(",");
|
||||
expect(detectDecimalSeparator(["1,234.56"])).toBe(".");
|
||||
});
|
||||
|
||||
it("returns no verdict when the column gives no evidence", () => {
|
||||
expect(detectDecimalSeparator([])).toBeUndefined();
|
||||
expect(detectDecimalSeparator(["1.234", "5.678"])).toBeUndefined();
|
||||
expect(detectDecimalSeparator(["", " ", "N/A"])).toBeUndefined();
|
||||
// One vote each way is a tie, not a majority.
|
||||
expect(detectDecimalSeparator(["84,32", "84.32"])).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseFrenchAmount — call-site fallout (#326, hardened by #325)", () => {
|
||||
// The /review-spec revision of #326 asks the corpus to reach the holdings
|
||||
// call sites too, because #325 hardens the parser GLOBALLY. These pin what
|
||||
// the shared parser does to its two most exposed consumers.
|
||||
|
||||
it("no longer blinds detectHeader when a header cell starts with digits", () => {
|
||||
// `detectHeader` (csvAutoDetect.ts:224) treats "parses as a number" as
|
||||
// proof of a data row. "2025 Montant" used to parse as 2025, so the header
|
||||
// was taken for data; anchoring makes it NaN and the header is recognised.
|
||||
const cfg = autoDetectConfig(readCsvFixture("header-numeric-label"))!;
|
||||
expect(cfg.hasHeader).toBe(true);
|
||||
});
|
||||
|
||||
it("no longer leaks a x100 magnitude into the holdings CSV import (#245)", () => {
|
||||
// `holdingsFromCsvRows` (useSnapshotEditor.ts:191-202) shares the parser.
|
||||
// A price column carrying its currency code used to multiply every
|
||||
// position by 100 — a $150.25 share stored at $15,025. It is refused now,
|
||||
// and an empty price is what `buildDetailedLines` rejects on save.
|
||||
const drafts = holdingsFromCsvRows(
|
||||
[
|
||||
["AAPL", "10", "150,25 CAD", "1 200,00"],
|
||||
["MSFT", "5", "300,50", "1 400,00 CAD"],
|
||||
],
|
||||
{ symbol: 0, quantity: 1, unit_price: 2, book_cost: 3 }
|
||||
);
|
||||
expect(drafts[0].unit_price).toBe(""); // refused, not 15025
|
||||
expect(drafts[0].book_cost).toBe("1200"); // clean cell
|
||||
expect(drafts[1].unit_price).toBe("300.5"); // clean cell
|
||||
expect(drafts[1].book_cost).toBe(""); // refused, not 140000
|
||||
});
|
||||
|
||||
it("reads an accounting-parenthesis price instead of dropping it", () => {
|
||||
const drafts = holdingsFromCsvRows(
|
||||
[["GOOG", "2", "(140,10)", "280,20"]],
|
||||
{ symbol: 0, quantity: 1, unit_price: 2, book_cost: 3 }
|
||||
);
|
||||
expect(drafts[0].unit_price).toBe("-140.1");
|
||||
expect(drafts[0].quantity).toBe("2");
|
||||
});
|
||||
|
||||
it("keeps an unreadable quantity verbatim so the save refuses it", () => {
|
||||
// It used to be coerced to 0, which SAVED a zero-value position in
|
||||
// silence. `buildDetailedLines` throws on the raw text instead.
|
||||
const drafts = holdingsFromCsvRows(
|
||||
[["GOOG", "2 parts", "140,10", "280,20"]],
|
||||
{ symbol: 0, quantity: 1, unit_price: 2, book_cost: 3 }
|
||||
);
|
||||
expect(drafts[0].quantity).toBe("2 parts");
|
||||
expect(() =>
|
||||
buildDetailedLines({ 7: drafts }, new Set([7]))
|
||||
).toThrowError(BalanceServiceError);
|
||||
});
|
||||
|
||||
it("taints the merged quantity when one lot of a symbol is unreadable", () => {
|
||||
const drafts = holdingsFromCsvRows(
|
||||
[
|
||||
["AAPL", "6", "150,00", "700"],
|
||||
["AAPL", "quatre", "151,00", "500"],
|
||||
],
|
||||
{ symbol: 0, quantity: 1, unit_price: 2, book_cost: 3 }
|
||||
);
|
||||
expect(drafts).toHaveLength(1);
|
||||
expect(drafts[0].quantity).toBe("quatre"); // NOT "6"
|
||||
});
|
||||
});
|
||||
|
|
@ -1,26 +1,192 @@
|
|||
/**
|
||||
* Parse a French-formatted amount string to a number.
|
||||
* Handles formats like: 1.234,56 / 1234,56 / -1 234.56 / 1 234,56
|
||||
* Amount parsing for imported files (#325).
|
||||
*
|
||||
* The function used to end on `parseFloat`, which returns the longest valid
|
||||
* PREFIX of a string instead of rejecting it. `"100,00 CAD"` therefore came out
|
||||
* as 10 000 and `"1 234,56 CR"` as 123 456 — a factor-100 error that passes
|
||||
* `isNaN`, so the row counted as VALID everywhere downstream. Validation is
|
||||
* ANCHORED now: after normalisation the whole remaining string must match a
|
||||
* numeric grammar, and any residual character yields `NaN`.
|
||||
*
|
||||
* `NaN` is the only safe answer for a cell we cannot read with certainty. A
|
||||
* trailing `CR` / `DB` carries a DIRECTION, not noise, so guessing a magnitude
|
||||
* for it would trade a loud failure for a silent sign error — the exact bug
|
||||
* class this chantier exists to remove. The absolute-amount + D/C-indicator
|
||||
* shape is refused, by design, upstream (#328).
|
||||
*
|
||||
* Two accounting forms ARE legitimate and supported: parentheses `(50,00)` and
|
||||
* a trailing sign `50,00-`, both meaning a negative amount.
|
||||
*
|
||||
* Separator arbitration: `1.234` means 1234 in a French column and 1.234 in an
|
||||
* English one, and nothing INSIDE the cell can tell. Callers that know the
|
||||
* column pass `decimalSeparator` (see `detectDecimalSeparator`); callers that
|
||||
* do not get the documented per-cell default.
|
||||
*/
|
||||
export function parseFrenchAmount(raw: string): number {
|
||||
|
||||
/** Which character separates the decimals in a given column. */
|
||||
export type DecimalSeparator = "," | ".";
|
||||
|
||||
export interface AmountParseOptions {
|
||||
/**
|
||||
* Decimal separator arbitrated at COLUMN level. When set, the other character
|
||||
* is read as a grouping separator, whatever a single cell looks like.
|
||||
*/
|
||||
decimalSeparator?: DecimalSeparator;
|
||||
}
|
||||
|
||||
/** Currency symbols and every flavour of space are noise, never data. */
|
||||
const NOISE = /[€$£\s\u00A0]/g;
|
||||
|
||||
/** Digits, optionally grouped in threes by `sep`. Anchored. */
|
||||
function groupedRe(sep: "," | "."): RegExp {
|
||||
const s = sep === "." ? "\\." : ",";
|
||||
return new RegExp(`^\\d{1,3}(?:${s}\\d{3})+$`);
|
||||
}
|
||||
|
||||
/** Digits, one `sep`, then decimals. Anchored. `limit` caps the decimal count. */
|
||||
function decimalRe(sep: "," | ".", limit?: number): RegExp {
|
||||
const s = sep === "." ? "\\." : ",";
|
||||
const tail = limit === undefined ? "+" : `{1,${limit}}`;
|
||||
return new RegExp(`^\\d+${s}\\d${tail}$`);
|
||||
}
|
||||
|
||||
/** Grouped digits then decimals, e.g. `1.234,56`. Anchored. */
|
||||
function groupedDecimalRe(group: "," | ".", dec: "," | "."): RegExp {
|
||||
const g = group === "." ? "\\." : ",";
|
||||
const d = dec === "." ? "\\." : ",";
|
||||
return new RegExp(`^\\d{1,3}(?:${g}\\d{3})+${d}\\d+$`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a fully normalised body (digits plus `.` and `,` only, no sign, no
|
||||
* spaces) under a known decimal separator. Returns `NaN` when the body does not
|
||||
* match the grammar exactly.
|
||||
*/
|
||||
function readWithSeparator(body: string, decimal: DecimalSeparator): number {
|
||||
const group: DecimalSeparator = decimal === "," ? "." : ",";
|
||||
const hasDecimal = body.includes(decimal);
|
||||
const hasGroup = body.includes(group);
|
||||
|
||||
let normalized: string | null = null;
|
||||
if (hasDecimal && hasGroup) {
|
||||
if (groupedDecimalRe(group, decimal).test(body)) normalized = body;
|
||||
} else if (hasDecimal) {
|
||||
if (decimalRe(decimal).test(body)) normalized = body;
|
||||
} else if (hasGroup) {
|
||||
if (groupedRe(group).test(body)) normalized = body;
|
||||
} else if (/^\d+$/.test(body)) {
|
||||
normalized = body;
|
||||
}
|
||||
if (normalized === null) return NaN;
|
||||
|
||||
const stripped = normalized.split(group).join("");
|
||||
return Number(decimal === "," ? stripped.replace(",", ".") : stripped);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a body with no column context. The rules reproduce the documented
|
||||
* per-cell behaviour, now anchored:
|
||||
* - both separators present -> the LAST one is the decimal;
|
||||
* - a single `,` followed by one or two digits -> French decimal;
|
||||
* - a single `.` -> English decimal (`1.234` reads as 1.234 alone);
|
||||
* - otherwise the separator must group digits in perfect threes.
|
||||
*/
|
||||
function readPerCell(body: string): number {
|
||||
const lastComma = body.lastIndexOf(",");
|
||||
const lastDot = body.lastIndexOf(".");
|
||||
|
||||
if (lastComma >= 0 && lastDot >= 0) {
|
||||
return readWithSeparator(body, lastComma > lastDot ? "," : ".");
|
||||
}
|
||||
if (lastComma >= 0) {
|
||||
if (decimalRe(",", 2).test(body)) return readWithSeparator(body, ",");
|
||||
return groupedRe(",").test(body) ? readWithSeparator(body, ".") : NaN;
|
||||
}
|
||||
if (lastDot >= 0) {
|
||||
if (decimalRe(".").test(body)) return readWithSeparator(body, ".");
|
||||
return groupedRe(".").test(body) ? readWithSeparator(body, ",") : NaN;
|
||||
}
|
||||
return /^\d+$/.test(body) ? Number(body) : NaN;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse an amount cell to a number, or `NaN` when it cannot be read with
|
||||
* certainty. Handles `1.234,56`, `1 234,56`, `1,234.56`, `-84,32`, `(50,00)`
|
||||
* and `50,00-`.
|
||||
*/
|
||||
export function parseFrenchAmount(
|
||||
raw: string,
|
||||
options: AmountParseOptions = {}
|
||||
): number {
|
||||
if (!raw || typeof raw !== "string") return NaN;
|
||||
|
||||
let cleaned = raw.trim();
|
||||
let negative = false;
|
||||
|
||||
// Remove currency symbols and whitespace
|
||||
cleaned = cleaned.replace(/[€$£\s\u00A0]/g, "");
|
||||
|
||||
// Detect if comma is decimal separator (French style)
|
||||
// Pattern: digits followed by comma followed by exactly 1-2 digits at end
|
||||
const frenchPattern = /,\d{1,2}$/;
|
||||
if (frenchPattern.test(cleaned)) {
|
||||
// French format: remove dots (thousand sep), replace comma with dot (decimal)
|
||||
cleaned = cleaned.replace(/\./g, "").replace(",", ".");
|
||||
} else {
|
||||
// English format or no decimal: remove commas (thousand sep)
|
||||
cleaned = cleaned.replace(/,/g, "");
|
||||
// Accounting parentheses. Only a balanced, outermost pair counts; `(50,00`
|
||||
// falls through and fails the anchored grammar below.
|
||||
const parens = /^\((.*)\)$/.exec(cleaned);
|
||||
if (parens) {
|
||||
negative = true;
|
||||
cleaned = parens[1].trim();
|
||||
}
|
||||
|
||||
const result = parseFloat(cleaned);
|
||||
return isNaN(result) ? NaN : result;
|
||||
cleaned = cleaned.replace(NOISE, "");
|
||||
if (!cleaned) return NaN;
|
||||
|
||||
// One sign, leading or trailing, never both, never inside parentheses.
|
||||
const leading = /^[+-]/.test(cleaned);
|
||||
const trailing = /[+-]$/.test(cleaned);
|
||||
if (leading && trailing) return NaN;
|
||||
if (leading || trailing) {
|
||||
if (negative) return NaN; // `(-50,00)` is not a form any bank emits
|
||||
negative = cleaned[leading ? 0 : cleaned.length - 1] === "-";
|
||||
cleaned = leading ? cleaned.slice(1) : cleaned.slice(0, -1);
|
||||
}
|
||||
|
||||
// Anchored gate: nothing but digits and separators may remain. This is what
|
||||
// rejects `2024Montant`, `1e3`, `Infinity`, `5%` and `100,00CAD`.
|
||||
if (!/^[\d.,]+$/.test(cleaned) || !/\d/.test(cleaned)) return NaN;
|
||||
|
||||
const value = options.decimalSeparator
|
||||
? readWithSeparator(cleaned, options.decimalSeparator)
|
||||
: readPerCell(cleaned);
|
||||
|
||||
if (!Number.isFinite(value)) return NaN;
|
||||
return negative ? -value : value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Arbitrate the decimal separator of a COLUMN from all of its cells.
|
||||
*
|
||||
* A cell is decisive when it carries both separators (the last one is the
|
||||
* decimal) or exactly one separator followed by one or two digits — a grouping
|
||||
* separator is always followed by three. Ambiguous cells such as `1.234` vote
|
||||
* for nothing. Returns `undefined` when the column gives no verdict, in which
|
||||
* case callers keep the per-cell default.
|
||||
*/
|
||||
export function detectDecimalSeparator(
|
||||
cells: Iterable<string>
|
||||
): DecimalSeparator | undefined {
|
||||
let comma = 0;
|
||||
let dot = 0;
|
||||
|
||||
for (const cell of cells) {
|
||||
if (!cell || typeof cell !== "string") continue;
|
||||
const body = cell.trim().replace(NOISE, "").replace(/^[+-]|[+-]$/g, "");
|
||||
if (!body) continue;
|
||||
|
||||
const lastComma = body.lastIndexOf(",");
|
||||
const lastDot = body.lastIndexOf(".");
|
||||
if (lastComma >= 0 && lastDot >= 0) {
|
||||
if (lastComma > lastDot) comma++;
|
||||
else dot++;
|
||||
continue;
|
||||
}
|
||||
if (lastComma >= 0 && decimalRe(",", 2).test(body)) comma++;
|
||||
else if (lastDot >= 0 && decimalRe(".", 2).test(body)) dot++;
|
||||
}
|
||||
|
||||
if (comma === dot) return undefined;
|
||||
return comma > dot ? "," : ".";
|
||||
}
|
||||
|
|
|
|||
764
src/utils/bankSignatures.test.ts
Normal file
764
src/utils/bankSignatures.test.ts
Normal file
|
|
@ -0,0 +1,764 @@
|
|||
// Bank signatures and format drift (#330).
|
||||
//
|
||||
// Two contracts are frozen here.
|
||||
//
|
||||
// THE SIGNATURES. Each of the four banks has a fixture carrying its documented
|
||||
// header layout. The test that matters is not "the signature matches" — it is
|
||||
// the pair: the same file WITH its bank header and WITHOUT it. Three of the
|
||||
// four layouts are read wrong by the generic dictionary, and the counterfactual
|
||||
// is what proves the signature earns its place instead of merely agreeing with
|
||||
// the heuristic. The other half of that pair is the fall-back: every fixture of
|
||||
// the #326 corpus must still be detected with no bank at all.
|
||||
//
|
||||
// THE DRIFT. `header_signature` records the labels of the last successful
|
||||
// import; a later file whose header normalizes differently is drift. The
|
||||
// interesting cases are the ones that must NOT fire: no stored signature, a
|
||||
// headerless file, a stored value that will not parse, and a cosmetic rename
|
||||
// (`Montant` -> `MONTANT ($)`) that normalizes to the same label.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { readFileSync } from "fs";
|
||||
import { resolve } from "path";
|
||||
import {
|
||||
BANK_SIGNATURES,
|
||||
MIN_SIGNATURE_LABELS,
|
||||
bankSignatureById,
|
||||
buildHeaderSignature,
|
||||
detectHeaderDrift,
|
||||
matchBankSignature,
|
||||
parseHeaderSignature,
|
||||
type BankSignatureId,
|
||||
} from "./bankSignatures";
|
||||
import { normalizeHeaderCell } from "./headerDictionary";
|
||||
import { detectImportFormat } from "./csvAutoDetect";
|
||||
import { CSV_FIXTURE_NAMES, readCsvFixture } from "../__fixtures__/csv";
|
||||
import fr from "../i18n/locales/fr.json";
|
||||
import en from "../i18n/locales/en.json";
|
||||
|
||||
/** The bank a file is recognised as, or null. Throws on a file detection refuses. */
|
||||
function bankOf(content: string): BankSignatureId | null {
|
||||
const outcome = detectImportFormat(content);
|
||||
if (outcome.status !== "ok") {
|
||||
throw new Error(`detection returned ${outcome.status}`);
|
||||
}
|
||||
return outcome.bank;
|
||||
}
|
||||
|
||||
/** The configuration detection settles on. Throws on a file it refuses. */
|
||||
function configOf(content: string) {
|
||||
const outcome = detectImportFormat(content);
|
||||
if (outcome.status !== "ok") {
|
||||
throw new Error(`detection returned ${outcome.status}`);
|
||||
}
|
||||
return outcome.config;
|
||||
}
|
||||
|
||||
/** The same file with its header row replaced — every data row untouched. */
|
||||
function withHeader(raw: string, header: string): string {
|
||||
const lines = raw.split("\n");
|
||||
lines[0] = header;
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
describe("the signature table is well formed (#330)", () => {
|
||||
it("gives every bank a distinct id", () => {
|
||||
const ids = BANK_SIGNATURES.map((s) => s.id);
|
||||
expect(new Set(ids).size).toBe(ids.length);
|
||||
});
|
||||
|
||||
it("declares every label already normalized", () => {
|
||||
// A label written `Montant` would never match: the header cell is
|
||||
// normalized before the comparison and the table's side is not.
|
||||
for (const signature of BANK_SIGNATURES) {
|
||||
for (const variant of signature.variants) {
|
||||
for (const label of variant.labels) {
|
||||
expect(normalizeHeaderCell(label), `${signature.id}: ${label}`).toBe(
|
||||
label
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps every role's label inside the fingerprint it projects", () => {
|
||||
// A role naming a label absent from `labels` would resolve to a column the
|
||||
// match never checked for.
|
||||
for (const signature of BANK_SIGNATURES) {
|
||||
for (const variant of signature.variants) {
|
||||
for (const [role, label] of Object.entries(variant.roles)) {
|
||||
expect(variant.labels, `${signature.id}.${role}`).toContain(label);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("refuses a fingerprint too small to identify anyone", () => {
|
||||
// `Date;Description;Montant` is the shape of half the corpus and of any
|
||||
// hand-made export. A three-label variant would put a bank's name over
|
||||
// files nobody can attribute to it.
|
||||
for (const signature of BANK_SIGNATURES) {
|
||||
for (const variant of signature.variants) {
|
||||
expect(
|
||||
variant.labels.length,
|
||||
`${signature.id}: ${variant.labels.join(",")}`
|
||||
).toBeGreaterThanOrEqual(MIN_SIGNATURE_LABELS);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("resolves a bank from its id, and nothing from an unknown one", () => {
|
||||
expect(bankSignatureById("desjardins")?.label).toBe("Desjardins");
|
||||
expect(bankSignatureById(null)).toBeNull();
|
||||
expect(bankSignatureById(undefined)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("each bank is recognised on its own fixture (#330)", () => {
|
||||
it("reads the Desjardins layout and names it", () => {
|
||||
expect(bankOf(readCsvFixture("bank-desjardins"))).toBe("desjardins");
|
||||
expect(configOf(readCsvFixture("bank-desjardins"))).toEqual({
|
||||
delimiter: ";",
|
||||
hasHeader: true,
|
||||
skipLines: 0,
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
// Solde (column 3) is the balance the signature names, and it stays out
|
||||
// of the amount candidates.
|
||||
columnMapping: { date: 0, description: 1, amount: 2 },
|
||||
amountMode: "single",
|
||||
signConvention: "negative_expense",
|
||||
});
|
||||
});
|
||||
|
||||
it("reads the English Desjardins header too", () => {
|
||||
const content =
|
||||
"Date;Description;Amount;Balance\n" +
|
||||
"05/01/2025;EPICERIE METRO;-84,32;915,68\n" +
|
||||
"15/01/2025;DEPOT PAIE;1250,00;2165,68\n" +
|
||||
"18/01/2025;HYDRO QUEBEC;-142,18;2023,50\n" +
|
||||
"22/01/2025;RESTAURANT;-56,75;1966,75\n";
|
||||
expect(bankOf(content)).toBe("desjardins");
|
||||
});
|
||||
|
||||
it("reads a whole-line-quoted Desjardins export", () => {
|
||||
// `preprocessQuotedCSV` unwraps it into a comma-separated file; the
|
||||
// signature declares the quirk and the comma, so the match survives it.
|
||||
const content =
|
||||
'"Date,""Description"",Montant,Solde"\n' +
|
||||
'"05/01/2025,""EPICERIE METRO"",-84.32,915.68"\n' +
|
||||
'"15/01/2025,""DEPOT PAIE"",1250.00,2165.68"\n' +
|
||||
'"18/01/2025,""HYDRO QUEBEC"",-142.18,2023.50"\n' +
|
||||
'"22/01/2025,""RESTAURANT"",-56.75,1966.75"\n';
|
||||
expect(bankOf(content)).toBe("desjardins");
|
||||
});
|
||||
|
||||
it("reads the RBC layout and maps CAD$ as the amount", () => {
|
||||
expect(bankOf(readCsvFixture("bank-rbc"))).toBe("rbc");
|
||||
expect(configOf(readCsvFixture("bank-rbc"))).toEqual({
|
||||
delimiter: ",",
|
||||
hasHeader: true,
|
||||
skipLines: 0,
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
columnMapping: { date: 2, description: 4, amount: 6 },
|
||||
amountMode: "single",
|
||||
signConvention: "negative_expense",
|
||||
});
|
||||
});
|
||||
|
||||
it("reads the Banque Nationale layout as a debit/credit pair", () => {
|
||||
expect(bankOf(readCsvFixture("bank-bnc"))).toBe("bnc");
|
||||
expect(configOf(readCsvFixture("bank-bnc"))).toEqual({
|
||||
delimiter: ";",
|
||||
hasHeader: true,
|
||||
skipLines: 0,
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
columnMapping: {
|
||||
date: 0,
|
||||
description: 1,
|
||||
debitAmount: 3,
|
||||
creditAmount: 4,
|
||||
},
|
||||
amountMode: "debit_credit",
|
||||
signConvention: "negative_expense",
|
||||
});
|
||||
});
|
||||
|
||||
it("reads the Tangerine layout and maps Name as the description", () => {
|
||||
expect(bankOf(readCsvFixture("bank-tangerine"))).toBe("tangerine");
|
||||
expect(configOf(readCsvFixture("bank-tangerine"))).toEqual({
|
||||
delimiter: ",",
|
||||
hasHeader: true,
|
||||
skipLines: 0,
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
columnMapping: { date: 0, description: 2, amount: 4 },
|
||||
amountMode: "single",
|
||||
signConvention: "negative_expense",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("what the signatures actually buy (#330)", () => {
|
||||
it("keeps RBC's cheque number out of the amounts", () => {
|
||||
// The SAME rows, under a header no signature knows. `CAD$` and
|
||||
// `Cheque Number` are sparse-complementary — one cheque number in six rows
|
||||
// — so the shape scan pairs them as debit/credit and the row carrying a
|
||||
// cheque number imports as -247.95 instead of -6.95. Nothing in the file
|
||||
// can tell that apart from a genuine debit/credit pair; a documented layout
|
||||
// can.
|
||||
const anonymised = withHeader(
|
||||
readCsvFixture("bank-rbc"),
|
||||
"Type,Numero,Date,Cheque,Libelle,Note,Montant,Devise"
|
||||
);
|
||||
expect(bankOf(anonymised)).toBeNull();
|
||||
expect(configOf(anonymised).amountMode).toBe("debit_credit");
|
||||
expect(configOf(anonymised).columnMapping).toEqual({
|
||||
date: 2,
|
||||
description: 4,
|
||||
debitAmount: 3,
|
||||
creditAmount: 6,
|
||||
});
|
||||
|
||||
// With the header the bank actually prints, the same rows read as one
|
||||
// signed column.
|
||||
expect(configOf(readCsvFixture("bank-rbc")).amountMode).toBe("single");
|
||||
});
|
||||
|
||||
it("keeps Tangerine's direction column out of the description, signature or not", () => {
|
||||
// `Transaction` is a description keyword of the generic dictionary, and
|
||||
// Tangerine's `Transaction` column holds DEBIT / CREDIT. Read as the
|
||||
// description, every transaction of the file is labelled `DEBIT`.
|
||||
//
|
||||
// This used to be rescued by the Tangerine signature alone, leaving any
|
||||
// unrecognised file with a `Transaction` column broken. The cardinality
|
||||
// veto in `detectDescriptionColumn` fixes the cause instead: a labelled
|
||||
// column that behaves like an enum is not the description, whether or not a
|
||||
// bank was identified.
|
||||
const anonymised = withHeader(
|
||||
readCsvFixture("bank-tangerine"),
|
||||
"Date,Transaction,Nom,Note,Montant"
|
||||
);
|
||||
expect(bankOf(anonymised)).toBeNull();
|
||||
expect(configOf(anonymised).columnMapping.description).toBe(2);
|
||||
|
||||
expect(configOf(readCsvFixture("bank-tangerine")).columnMapping.description).toBe(
|
||||
2
|
||||
);
|
||||
});
|
||||
|
||||
it("does not let a generic variant claim a richer header", () => {
|
||||
// Review finding on #340: Desjardins' fingerprint is four labels any
|
||||
// Canadian bank could emit, so matching them as a SUBSET announced
|
||||
// "Format Desjardins reconnu" over files that have nothing to do with it.
|
||||
// A variant made only of generic labels now has to describe the header
|
||||
// exactly.
|
||||
const richer =
|
||||
"Date;Description;Débit;Crédit;Montant;Solde\n" +
|
||||
"05/01/2025;EPICERIE;84,32;;-84,32;1000,00\n" +
|
||||
"15/01/2025;DEPOT PAIE;;1250,00;1250,00;2250,00\n" +
|
||||
"20/01/2025;LOYER;900,00;;-900,00;1350,00\n" +
|
||||
"25/01/2025;REMBOURSEMENT;;45,00;45,00;1395,00\n";
|
||||
expect(bankOf(richer)).toBeNull();
|
||||
|
||||
// The exact header still is Desjardins.
|
||||
const exact =
|
||||
"Date;Description;Montant;Solde\n" +
|
||||
"05/01/2025;EPICERIE;-84,32;1000,00\n" +
|
||||
"15/01/2025;DEPOT PAIE;1250,00;2250,00\n";
|
||||
expect(bankOf(exact)).toBe("desjardins");
|
||||
});
|
||||
|
||||
it("does not let a signature's amount column displace a pair it is not in", () => {
|
||||
// The measured regression: with the false Desjardins match above, the
|
||||
// signature's `Montant` short-circuited the sparse-complementary scan, so a
|
||||
// file that reads correctly as debit/credit became one unsigned column and
|
||||
// every deposit imported as an expense. The scan now runs first and the
|
||||
// signature only wins when the pair contains its column — which is what
|
||||
// RBC's genuine `Cheque Number` / `CAD$` case needs.
|
||||
const richer =
|
||||
"Date;Description;Débit;Crédit;Montant;Solde\n" +
|
||||
"05/01/2025;EPICERIE;84,32;;-84,32;1000,00\n" +
|
||||
"15/01/2025;DEPOT PAIE;;1250,00;1250,00;2250,00\n" +
|
||||
"20/01/2025;LOYER;900,00;;-900,00;1350,00\n" +
|
||||
"25/01/2025;REMBOURSEMENT;;45,00;45,00;1395,00\n";
|
||||
const config = configOf(richer);
|
||||
expect(config.amountMode).toBe("debit_credit");
|
||||
expect(config.columnMapping.debitAmount).toBe(2);
|
||||
expect(config.columnMapping.creditAmount).toBe(3);
|
||||
|
||||
// RBC keeps its override: there the declared amount column IS in the pair.
|
||||
expect(configOf(readCsvFixture("bank-rbc")).amountMode).toBe("single");
|
||||
});
|
||||
|
||||
it("holds even for a legitimately recognised bank whose file grew columns", () => {
|
||||
// The guard above is only reachable through a signature that really matches,
|
||||
// so it needs a discriminating one. Tangerine is identified by `memo`, so it
|
||||
// still matches as a subset when the export gains Débit/Crédit columns — and
|
||||
// then its declared `Amount` must not displace that genuine pair either.
|
||||
const grown =
|
||||
"Date,Transaction,Name,Memo,Amount,Débit,Crédit\n" +
|
||||
"05/01/2025,DEBIT,EPICERIE METRO,,-84.32,84.32,\n" +
|
||||
"15/01/2025,CREDIT,DEPOT PAIE,Paie,1250.00,,1250.00\n" +
|
||||
"20/01/2025,DEBIT,LOYER,,-900.00,900.00,\n" +
|
||||
"25/01/2025,CREDIT,REMBOURSEMENT,,45.00,,45.00\n";
|
||||
expect(bankOf(grown)).toBe("tangerine");
|
||||
|
||||
const config = configOf(grown);
|
||||
expect(config.amountMode).toBe("debit_credit");
|
||||
expect(config.columnMapping.debitAmount).toBe(5);
|
||||
expect(config.columnMapping.creditAmount).toBe(6);
|
||||
});
|
||||
});
|
||||
|
||||
describe("an unknown file falls back to the generic dictionary (#330)", () => {
|
||||
it("names no bank on any of the shape fixtures", () => {
|
||||
// The corpus frozen by #326 is synthetic and belongs to no bank. A single
|
||||
// false positive here is a banner claiming a bank the file is not from.
|
||||
for (const name of CSV_FIXTURE_NAMES) {
|
||||
if (name.startsWith("bank-")) continue;
|
||||
if (name === "absolute-indicator") continue; // refused before any bank
|
||||
expect(bankOf(readCsvFixture(name)), name).toBeNull();
|
||||
}
|
||||
});
|
||||
|
||||
it("leaves the pre-#330 configurations untouched", () => {
|
||||
// The signatures must not have moved a single mapping of the corpus.
|
||||
expect(configOf(readCsvFixture("signed-amount"))).toEqual({
|
||||
delimiter: ";",
|
||||
hasHeader: true,
|
||||
skipLines: 0,
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
columnMapping: { date: 0, description: 1, amount: 2 },
|
||||
amountMode: "single",
|
||||
signConvention: "negative_expense",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("the preconditions of a match (#330)", () => {
|
||||
const rows =
|
||||
"05/01/2025;EPICERIE METRO;-84,32;915,68\n" +
|
||||
"15/01/2025;DEPOT PAIE;1250,00;2165,68\n" +
|
||||
"18/01/2025;HYDRO QUEBEC;-142,18;2023,50\n" +
|
||||
"22/01/2025;RESTAURANT;-56,75;1966,75\n";
|
||||
|
||||
it("refuses a delimiter the bank does not use", () => {
|
||||
const tabbed = ("Date;Description;Montant;Solde\n" + rows).replace(
|
||||
/;/g,
|
||||
"\t"
|
||||
);
|
||||
expect(bankOf(tabbed)).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses an export carrying more preamble than the bank prints", () => {
|
||||
const content =
|
||||
"RELEVE DE COMPTE\nCompte 12345\nDate;Description;Montant;Solde\n" + rows;
|
||||
expect(bankOf(content)).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses a whole-line-quoted file for a bank that does not quote", () => {
|
||||
// Tangerine's labels, wrapped the way Desjardins wraps its exports. The
|
||||
// quirk is what says this is not a Tangerine file.
|
||||
const content =
|
||||
'"Date,""Transaction"",Name,Memo,Amount"\n' +
|
||||
'"05/01/2025,""DEBIT"",EPICERIE,,-84.32"\n' +
|
||||
'"15/01/2025,""CREDIT"",PAIE,,1250.00"\n' +
|
||||
'"18/01/2025,""DEBIT"",HYDRO,,-142.18"\n' +
|
||||
'"22/01/2025,""DEBIT"",RESTO,,-56.75"\n';
|
||||
expect(bankOf(content)).toBeNull();
|
||||
});
|
||||
|
||||
it("names no bank for a headerless file, whatever its shape", () => {
|
||||
// There is no label to read. This is the same boundary that leaves
|
||||
// `header_signature` null on those sources.
|
||||
expect(bankOf(readCsvFixture("no-header"))).toBeNull();
|
||||
});
|
||||
|
||||
it("matches on the whole label, never on a substring", () => {
|
||||
// `matchHeaderColumn` matches substrings — that is the generic dictionary's
|
||||
// rule and the reason it mis-reads these layouts. A fingerprint compared
|
||||
// loosely would inherit the same problem.
|
||||
expect(
|
||||
matchBankSignature({
|
||||
headerRow: ["Date", "Description", "Montant net", "Solde"],
|
||||
delimiter: ";",
|
||||
skipLines: 0,
|
||||
wholeLineQuoted: false,
|
||||
})
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it("resolves the roles to the columns of THIS file, not to the declared order", () => {
|
||||
const match = matchBankSignature({
|
||||
headerRow: ["Solde", "Montant", "Description", "Date"],
|
||||
delimiter: ";",
|
||||
skipLines: 0,
|
||||
wholeLineQuoted: false,
|
||||
});
|
||||
expect(match?.signature.id).toBe("desjardins");
|
||||
expect(match?.roles).toEqual({
|
||||
date: 3,
|
||||
description: 2,
|
||||
amount: 1,
|
||||
debit: null,
|
||||
credit: null,
|
||||
balance: 0,
|
||||
roleCount: 4,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("the stored signature (#330)", () => {
|
||||
it("stores the normalized labels, in order", () => {
|
||||
expect(buildHeaderSignature(["Date", "Description", "Montant", "Solde"])).toBe(
|
||||
'["date","description","montant","solde"]'
|
||||
);
|
||||
});
|
||||
|
||||
it("stores nothing for a file with no header row", () => {
|
||||
// The whole point of the boundary: `Col 0`, `Col 1` … is not a signature,
|
||||
// and inventing one from the data would fire on every change of content.
|
||||
expect(buildHeaderSignature(null)).toBeNull();
|
||||
expect(buildHeaderSignature([])).toBeNull();
|
||||
});
|
||||
|
||||
it("reads back what it wrote", () => {
|
||||
const stored = buildHeaderSignature(["Date", "Montant"]);
|
||||
expect(parseHeaderSignature(stored)).toEqual(["date", "montant"]);
|
||||
});
|
||||
|
||||
it("returns null rather than throwing on anything it did not write", () => {
|
||||
// A hash left by an older build, a truncated row, hand-edited SQL: drift
|
||||
// detection switches off, an import never blows up.
|
||||
expect(parseHeaderSignature(null)).toBeNull();
|
||||
expect(parseHeaderSignature("")).toBeNull();
|
||||
expect(parseHeaderSignature("d41d8cd98f00b204")).toBeNull();
|
||||
expect(parseHeaderSignature("{}")).toBeNull();
|
||||
expect(parseHeaderSignature("[1,2,3]")).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("drift detection stays silent when it should (#330)", () => {
|
||||
const stored = buildHeaderSignature(["Date", "Description", "Montant"]);
|
||||
|
||||
it("says nothing about a source that never recorded a signature", () => {
|
||||
expect(detectHeaderDrift(null, ["Date", "Solde"])).toBeNull();
|
||||
expect(detectHeaderDrift(undefined, ["Date", "Solde"])).toBeNull();
|
||||
});
|
||||
|
||||
it("says nothing about a headerless file", () => {
|
||||
expect(detectHeaderDrift(stored, null)).toBeNull();
|
||||
expect(detectHeaderDrift(stored, [])).toBeNull();
|
||||
});
|
||||
|
||||
it("says nothing about an identical header", () => {
|
||||
expect(
|
||||
detectHeaderDrift(stored, ["Date", "Description", "Montant"])
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it("says nothing about a cosmetic rename", () => {
|
||||
// Accents, case, punctuation and spacing are stripped before the
|
||||
// comparison: none of them changes which column holds what.
|
||||
expect(
|
||||
detectHeaderDrift(stored, ["DATE", "Description ", "MONTANT ($)"])
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it("says nothing when the stored value cannot be read", () => {
|
||||
expect(detectHeaderDrift("not json", ["Date"])).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("drift detection names the columns that moved (#330)", () => {
|
||||
it("reports a column that changed position", () => {
|
||||
const stored = buildHeaderSignature([
|
||||
"Date",
|
||||
"Description",
|
||||
"Montant",
|
||||
"Solde",
|
||||
]);
|
||||
expect(
|
||||
detectHeaderDrift(stored, ["Date", "Description", "Solde", "Montant"])
|
||||
).toEqual([
|
||||
{
|
||||
kind: "moved",
|
||||
normalizedLabel: "solde",
|
||||
label: "Solde",
|
||||
previousIndex: 3,
|
||||
currentIndex: 2,
|
||||
},
|
||||
{
|
||||
kind: "moved",
|
||||
normalizedLabel: "montant",
|
||||
label: "Montant",
|
||||
previousIndex: 2,
|
||||
currentIndex: 3,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports a new column and the shift it causes", () => {
|
||||
const stored = buildHeaderSignature(["Date", "Description", "Montant"]);
|
||||
expect(
|
||||
detectHeaderDrift(stored, ["Date", "Devise", "Description", "Montant"])
|
||||
).toEqual([
|
||||
{
|
||||
kind: "added",
|
||||
normalizedLabel: "devise",
|
||||
label: "Devise",
|
||||
previousIndex: null,
|
||||
currentIndex: 1,
|
||||
},
|
||||
{
|
||||
kind: "moved",
|
||||
normalizedLabel: "description",
|
||||
label: "Description",
|
||||
previousIndex: 1,
|
||||
currentIndex: 2,
|
||||
},
|
||||
{
|
||||
kind: "moved",
|
||||
normalizedLabel: "montant",
|
||||
label: "Montant",
|
||||
previousIndex: 2,
|
||||
currentIndex: 3,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports a column that disappeared, under its stored label", () => {
|
||||
// The raw spelling of a removed column is recorded nowhere — only the
|
||||
// normalized label survives, and that is what is shown.
|
||||
const stored = buildHeaderSignature(["Date", "Description", "Solde"]);
|
||||
expect(detectHeaderDrift(stored, ["Date", "Description"])).toEqual([
|
||||
{
|
||||
kind: "removed",
|
||||
normalizedLabel: "solde",
|
||||
label: "solde",
|
||||
previousIndex: 2,
|
||||
currentIndex: null,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("survives a bank swapping its amount column for a debit/credit pair", () => {
|
||||
const stored = buildHeaderSignature([
|
||||
"Date",
|
||||
"Description",
|
||||
"Montant",
|
||||
"Solde",
|
||||
]);
|
||||
const drift = detectHeaderDrift(stored, [
|
||||
"Date",
|
||||
"Description",
|
||||
"Débit",
|
||||
"Crédit",
|
||||
"Solde",
|
||||
])!;
|
||||
expect(drift.map((e) => [e.kind, e.normalizedLabel])).toEqual([
|
||||
["added", "debit"],
|
||||
["added", "credit"],
|
||||
["moved", "solde"],
|
||||
["removed", "montant"],
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Static guards. The wiring lives in a React hook and two components, and the
|
||||
// repository has no jsdom — same technique as the guards of #324 to #329.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const WIZARD_SRC = readFileSync(
|
||||
resolve(import.meta.dirname, "..", "hooks", "useImportWizard.ts"),
|
||||
"utf-8"
|
||||
);
|
||||
const PAGE_SRC = readFileSync(
|
||||
resolve(import.meta.dirname, "..", "pages", "ImportPage.tsx"),
|
||||
"utf-8"
|
||||
);
|
||||
const componentSrc = (name: string) =>
|
||||
readFileSync(
|
||||
resolve(import.meta.dirname, "..", "components", "import", name),
|
||||
"utf-8"
|
||||
);
|
||||
|
||||
describe("the signature is written at every successful import (#330)", () => {
|
||||
it("builds it from the header row the import actually read", () => {
|
||||
expect(WIZARD_SRC).toContain("buildHeaderSignature(");
|
||||
expect(WIZARD_SRC).toContain(
|
||||
"config.hasHeader ? state.previewHeaders : null"
|
||||
);
|
||||
});
|
||||
|
||||
it("writes it on both arms of the single write point", () => {
|
||||
const body = WIZARD_SRC.slice(
|
||||
WIZARD_SRC.indexOf("const executeImport ="),
|
||||
WIZARD_SRC.indexOf("const goToStep =")
|
||||
);
|
||||
expect(body.match(/header_signature: headerSignature/g)).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("keeps it out of the format codec", () => {
|
||||
// It is drift metadata, not one of the eight fields that decide how a row
|
||||
// is read. `formatToRow` naming it would put it back in the format.
|
||||
const CODEC = readFileSync(
|
||||
resolve(import.meta.dirname, "importFormat.ts"),
|
||||
"utf-8"
|
||||
);
|
||||
expect(CODEC).not.toContain("header_signature");
|
||||
expect(CODEC).not.toContain("headerSignature");
|
||||
});
|
||||
});
|
||||
|
||||
describe("drift is computed on the way to the preview (#330)", () => {
|
||||
const body = () =>
|
||||
WIZARD_SRC.slice(
|
||||
WIZARD_SRC.indexOf("const parseAndPreview ="),
|
||||
WIZARD_SRC.indexOf("const checkDuplicatesInternal =")
|
||||
);
|
||||
|
||||
it("compares the stored signature to the headers just parsed", () => {
|
||||
expect(body()).toContain("detectHeaderDrift(");
|
||||
expect(body()).toContain("state.existingSource?.header_signature");
|
||||
});
|
||||
|
||||
it("does not compare anything on a headerless file", () => {
|
||||
expect(body()).toContain("state.sourceConfig.hasHeader");
|
||||
});
|
||||
|
||||
it("still reaches the preview step", () => {
|
||||
expect(body()).toContain(
|
||||
'dispatch({ type: "SET_STEP", payload: "file-preview" })'
|
||||
);
|
||||
});
|
||||
|
||||
it("drops the score the re-detection produced, which describes another format", () => {
|
||||
// The re-detection measures the format the PANEL offers, not the one in
|
||||
// use. Keeping its score would show "Format reconnu — 6 of 6 rows read"
|
||||
// beside the stored mapping as soon as the user steps back.
|
||||
expect(body()).toContain(
|
||||
'dispatch({ type: "SET_DETECTION_SCORE", payload: null })'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the drift panel offers two outcomes and writes neither (#330)", () => {
|
||||
const PANEL = componentSrc("FormatDriftPanel.tsx");
|
||||
|
||||
it("is rendered on the preview step, above the table", () => {
|
||||
expect(PAGE_SRC).toContain("{state.formatDrift && (");
|
||||
expect(PAGE_SRC).toContain("<FormatDriftPanel");
|
||||
expect(PAGE_SRC).toContain("onAdopt={adoptDriftFormat}");
|
||||
expect(PAGE_SRC).toContain("onKeep={keepCurrentFormat}");
|
||||
expect(PAGE_SRC.indexOf("<FormatDriftPanel")).toBeLessThan(
|
||||
PAGE_SRC.indexOf("<FilePreviewTable")
|
||||
);
|
||||
});
|
||||
|
||||
it("re-parses under the adopted format instead of re-reading state", () => {
|
||||
// `state` has not re-rendered when the parse starts, exactly as for the
|
||||
// sign flip: reading it back would redisplay the table just replaced.
|
||||
const body = WIZARD_SRC.slice(
|
||||
WIZARD_SRC.indexOf("const adoptDriftFormat ="),
|
||||
WIZARD_SRC.indexOf("const keepCurrentFormat =")
|
||||
);
|
||||
expect(body).toContain("parseFilesInternal(adopted)");
|
||||
expect(body).not.toContain("createSource(");
|
||||
expect(body).not.toContain("updateSource(");
|
||||
});
|
||||
|
||||
it("renders the three kinds of change through i18n", () => {
|
||||
for (const key of ["columnMoved", "columnAdded", "columnRemoved"]) {
|
||||
expect(PANEL, key).toContain(`import.drift.${key}`);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("the repair path is stated in the interface (#330)", () => {
|
||||
const NOTICE = componentSrc("RepairPathNotice.tsx");
|
||||
|
||||
it("names the history deletion rather than a vague warning", () => {
|
||||
expect(NOTICE).toContain("import.repairPath.title");
|
||||
expect(NOTICE).toContain("import.repairPath.body");
|
||||
for (const locale of [fr, en]) {
|
||||
expect(locale.import.repairPath.body.length).toBeGreaterThan(0);
|
||||
}
|
||||
// The two facts a user needs: duplicates are matched on the amount, and the
|
||||
// faulty import has to go first.
|
||||
expect(fr.import.repairPath.body).toContain("historique");
|
||||
expect(en.import.repairPath.body).toContain("history");
|
||||
});
|
||||
|
||||
it("appears in the preview and in the drift panel, not in one of the two", () => {
|
||||
expect(componentSrc("FilePreviewTable.tsx")).toContain("<RepairPathNotice />");
|
||||
expect(componentSrc("FormatDriftPanel.tsx")).toContain("<RepairPathNotice />");
|
||||
});
|
||||
});
|
||||
|
||||
describe("the recognised-bank banner (#330)", () => {
|
||||
const PANEL = componentSrc("SourceConfigPanel.tsx");
|
||||
|
||||
it("names the bank only when the rows actually read", () => {
|
||||
// A bank label over a file two thirds of whose rows fail is a claim the app
|
||||
// cannot back; the uncertain wording is the one that helps there.
|
||||
expect(PANEL).toContain("detectionScore?.confident");
|
||||
expect(PANEL).toContain("import.config.detectionBank");
|
||||
});
|
||||
|
||||
it("interpolates the bank name instead of translating it", () => {
|
||||
// A bank is a proper noun. `Desjardins` is `Desjardins` in both languages.
|
||||
for (const locale of [fr, en]) {
|
||||
expect(locale.import.config.detectionBank).toContain("{{bank}}");
|
||||
expect(locale.import.config.detectionBank).toContain("{{read}}");
|
||||
expect(locale.import.config.detectionBank).toContain("{{total}}");
|
||||
}
|
||||
});
|
||||
|
||||
it("clears the bank with the score, in one place", () => {
|
||||
// Three sites drop the score (a hand edit, a template, a sign flip). The
|
||||
// reducer clearing the bank alongside it is what keeps a fourth from
|
||||
// forgetting.
|
||||
const reducerCase = WIZARD_SRC.slice(
|
||||
WIZARD_SRC.indexOf('case "SET_DETECTION_SCORE":'),
|
||||
WIZARD_SRC.indexOf('case "SET_DETECTED_BANK":')
|
||||
);
|
||||
expect(reducerCase).toContain("detectedBank: action.payload === null");
|
||||
});
|
||||
});
|
||||
|
||||
describe("every new string exists in both languages (#330)", () => {
|
||||
it("carries the drift panel", () => {
|
||||
for (const key of [
|
||||
"title",
|
||||
"intro",
|
||||
"columnMoved",
|
||||
"columnAdded",
|
||||
"columnRemoved",
|
||||
"adopt",
|
||||
"adoptHint",
|
||||
"adoptUnavailable",
|
||||
"keep",
|
||||
"keepHint",
|
||||
] as const) {
|
||||
expect(fr.import.drift[key].length, `fr.${key}`).toBeGreaterThan(0);
|
||||
expect(en.import.drift[key].length, `en.${key}`).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
it("interpolates the column positions in both languages", () => {
|
||||
for (const locale of [fr, en]) {
|
||||
expect(locale.import.drift.columnMoved).toContain("{{label}}");
|
||||
expect(locale.import.drift.columnMoved).toContain("{{from}}");
|
||||
expect(locale.import.drift.columnMoved).toContain("{{to}}");
|
||||
expect(locale.import.drift.columnAdded).toContain("{{to}}");
|
||||
expect(locale.import.drift.columnRemoved).toContain("{{from}}");
|
||||
}
|
||||
});
|
||||
|
||||
it("carries the repair path", () => {
|
||||
for (const key of ["title", "body"] as const) {
|
||||
expect(fr.import.repairPath[key].length, `fr.${key}`).toBeGreaterThan(0);
|
||||
expect(en.import.repairPath[key].length, `en.${key}`).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
477
src/utils/bankSignatures.ts
Normal file
477
src/utils/bankSignatures.ts
Normal file
|
|
@ -0,0 +1,477 @@
|
|||
/**
|
||||
* Bank signatures and header-signature drift (#330).
|
||||
*
|
||||
* TWO THINGS LIVE HERE, and they are the same thing seen twice: a header row
|
||||
* reduced to its normalized labels.
|
||||
*
|
||||
* - A BANK SIGNATURE is that label list written down in advance, per bank, so
|
||||
* a known export is recognised by name instead of being guessed at.
|
||||
* - A STORED SIGNATURE is that same label list recorded on the source at the
|
||||
* last successful import (`import_sources.header_signature`), so the next
|
||||
* file from the same bank can be compared against it column by column.
|
||||
*
|
||||
* WHY SIGNATURES RUN BEFORE THE GENERIC DICTIONARY. `headerDictionary.ts`
|
||||
* matches keywords as SUBSTRINGS, one role at a time, and has no notion of a
|
||||
* best match — the first column containing the keyword wins. That is the right
|
||||
* rule for an unknown file and the wrong one for a known layout:
|
||||
* - Tangerine writes `Date,Transaction,Name,Memo,Amount`. The dictionary reads
|
||||
* `Transaction` as the description (it is a description keyword) and maps
|
||||
* the transaction TYPE column — `DEBIT`/`CREDIT` — as the label of every
|
||||
* row. The signature names `Name`.
|
||||
* - RBC writes its amount column `CAD$`, which contains neither `montant` nor
|
||||
* `amount`, so the dictionary finds no amount column at all and the shape
|
||||
* heuristic pairs `CAD$` with the (usually empty) `USD$` as a debit/credit
|
||||
* pair. The signature names `CAD$` as a single signed amount.
|
||||
*
|
||||
* WHY A FAILING SIGNATURE CANNOT BREAK ANYTHING. These signatures are written
|
||||
* from documented export layouts, WITHOUT real statements — the app is
|
||||
* privacy-first and no real statement lands in this repository. So a signature
|
||||
* is only ever allowed to do what the generic dictionary already does: hand
|
||||
* `detectImportFormat` a set of PREFERENCES. Every one of them is dropped the
|
||||
* moment the data contradicts it (a date column nothing parses as a date, an
|
||||
* amount column the shape scan never proposed). A file no signature recognises
|
||||
* falls through to the dictionary, unchanged. Failing degrades; it never breaks.
|
||||
*
|
||||
* WHY THE STORED SIGNATURE IS A LABEL LIST AND NOT A HASH. The drift panel has
|
||||
* to name the columns that moved ("Montant : 3 → 4"), which a hash forbids.
|
||||
*/
|
||||
|
||||
import {
|
||||
normalizeHeaderCell,
|
||||
type LexicalHeaderMap,
|
||||
} from "./headerDictionary";
|
||||
|
||||
/** Every bank the table knows, and the i18n-free proper noun it is shown as. */
|
||||
export type BankSignatureId = "desjardins" | "rbc" | "bnc" | "tangerine";
|
||||
|
||||
/** The transaction roles a signature can name — `LexicalHeaderMap` minus its count. */
|
||||
export type HeaderRole = keyof Omit<LexicalHeaderMap, "roleCount">;
|
||||
|
||||
/**
|
||||
* One documented layout of one bank.
|
||||
*
|
||||
* `labels` is the FINGERPRINT: every one of them must appear in the header row,
|
||||
* compared on the whole normalized cell (not as a substring). `roles` is a
|
||||
* projection of that fingerprint onto the columns detection cares about —
|
||||
* labels carrying no role (`Catégorie`, `Memo`, `N° de chèque`) stay in
|
||||
* `labels` precisely because they are what makes the fingerprint distinctive.
|
||||
*/
|
||||
export interface BankHeaderVariant {
|
||||
readonly labels: readonly string[];
|
||||
readonly roles: Readonly<Partial<Record<HeaderRole, string>>>;
|
||||
}
|
||||
|
||||
export interface BankSignature {
|
||||
readonly id: BankSignatureId;
|
||||
/** Proper noun. Displayed as-is in every language, never translated. */
|
||||
readonly label: string;
|
||||
/** Delimiters this bank's exports use. A precondition of the match, never an override. */
|
||||
readonly delimiters: readonly string[];
|
||||
/** Preamble lines this export prints before its header row. */
|
||||
readonly maxPreambleLines: number;
|
||||
/** The export wraps whole lines in quotes — see `preprocessQuotedCSV`. */
|
||||
readonly wholeLineQuoted: boolean;
|
||||
readonly variants: readonly BankHeaderVariant[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Labels a variant must carry before it is allowed to claim a file.
|
||||
*
|
||||
* Three is not enough: `Date;Description;Montant` is the shape of half the
|
||||
* synthetic corpus and of any hand-made export, so a three-label Desjardins
|
||||
* variant would put "Format Desjardins reconnu" over files no one can attribute
|
||||
* to Desjardins. Announcing the wrong bank is worse than announcing none — the
|
||||
* generic path reads those files correctly already.
|
||||
*/
|
||||
export const MIN_SIGNATURE_LABELS = 4;
|
||||
|
||||
/**
|
||||
* Labels any Canadian bank could emit. A variant built only from these
|
||||
* identifies no bank in particular, so the count alone does not deliver the
|
||||
* property `MIN_SIGNATURE_LABELS` promises — such a variant has to match the
|
||||
* header exactly (see `matchBankSignature`).
|
||||
*/
|
||||
const GENERIC_LABELS: ReadonlySet<string> = new Set([
|
||||
"date",
|
||||
"description",
|
||||
"libelle",
|
||||
"detail",
|
||||
"transaction",
|
||||
"montant",
|
||||
"amount",
|
||||
"solde",
|
||||
"balance",
|
||||
"debit",
|
||||
"credit",
|
||||
"retrait",
|
||||
"depot",
|
||||
"withdrawal",
|
||||
"deposit",
|
||||
]);
|
||||
|
||||
/**
|
||||
* The table. Written from the banks' documented export layouts; none of it has
|
||||
* been verified against a real statement (see the file header). Adding a bank
|
||||
* is adding an entry — there is no code to write.
|
||||
*/
|
||||
export const BANK_SIGNATURES: readonly BankSignature[] = [
|
||||
{
|
||||
// AccèsD exports Date / Description / Montant / Solde, semicolon-separated,
|
||||
// with comma decimals, in French or in English. Some credit-card exports
|
||||
// wrap every line in quotes (handled upstream by `preprocessQuotedCSV`,
|
||||
// which leaves a comma-separated file behind) and some carry NO header row
|
||||
// at all — those cannot be signed, by construction.
|
||||
id: "desjardins",
|
||||
label: "Desjardins",
|
||||
delimiters: [";", ","],
|
||||
maxPreambleLines: 0,
|
||||
wholeLineQuoted: true,
|
||||
variants: [
|
||||
{
|
||||
labels: ["date", "description", "montant", "solde"],
|
||||
roles: {
|
||||
date: "date",
|
||||
description: "description",
|
||||
amount: "montant",
|
||||
balance: "solde",
|
||||
},
|
||||
},
|
||||
{
|
||||
labels: ["date", "description", "amount", "balance"],
|
||||
roles: {
|
||||
date: "date",
|
||||
description: "description",
|
||||
amount: "amount",
|
||||
balance: "balance",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
// RBC exports one comma-separated file for every account, amounts in a
|
||||
// currency-named column. `CAD$` normalizes to `cad`, which no generic
|
||||
// amount keyword matches — this variant is the only thing that maps it.
|
||||
id: "rbc",
|
||||
label: "RBC",
|
||||
delimiters: [","],
|
||||
maxPreambleLines: 0,
|
||||
wholeLineQuoted: false,
|
||||
variants: [
|
||||
{
|
||||
labels: [
|
||||
"accounttype",
|
||||
"accountnumber",
|
||||
"transactiondate",
|
||||
"chequenumber",
|
||||
"description1",
|
||||
"cad",
|
||||
],
|
||||
roles: {
|
||||
date: "transactiondate",
|
||||
description: "description1",
|
||||
amount: "cad",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
// Banque Nationale exports a debit/credit pair next to a category column,
|
||||
// semicolon-separated in French.
|
||||
id: "bnc",
|
||||
label: "Banque Nationale",
|
||||
delimiters: [";", ","],
|
||||
maxPreambleLines: 0,
|
||||
wholeLineQuoted: false,
|
||||
variants: [
|
||||
{
|
||||
labels: ["date", "description", "categorie", "debit", "credit", "solde"],
|
||||
roles: {
|
||||
date: "date",
|
||||
description: "description",
|
||||
debit: "debit",
|
||||
credit: "credit",
|
||||
balance: "solde",
|
||||
},
|
||||
},
|
||||
{
|
||||
labels: ["date", "description", "category", "debit", "credit", "balance"],
|
||||
roles: {
|
||||
date: "date",
|
||||
description: "description",
|
||||
debit: "debit",
|
||||
credit: "credit",
|
||||
balance: "balance",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
// Tangerine exports Date,Transaction,Name,Memo,Amount — comma-separated,
|
||||
// signed amounts. `Transaction` holds the direction word, NOT the label of
|
||||
// the operation; reading it as the description makes every row read
|
||||
// `DEBIT` or `CREDIT`.
|
||||
id: "tangerine",
|
||||
label: "Tangerine",
|
||||
delimiters: [","],
|
||||
maxPreambleLines: 0,
|
||||
wholeLineQuoted: false,
|
||||
variants: [
|
||||
{
|
||||
labels: ["date", "transaction", "name", "memo", "amount"],
|
||||
roles: { date: "date", description: "name", amount: "amount" },
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
/** What the caller needs to know before a signature may claim a file. */
|
||||
export interface BankSignatureInput {
|
||||
/** The header row as parsed, raw cells. */
|
||||
readonly headerRow: readonly string[];
|
||||
/** Delimiter detection settled on. */
|
||||
readonly delimiter: string;
|
||||
/** Preamble lines detection had to skip. */
|
||||
readonly skipLines: number;
|
||||
/** True when `preprocessQuotedCSV` actually unwrapped the file. */
|
||||
readonly wholeLineQuoted: boolean;
|
||||
}
|
||||
|
||||
export interface BankSignatureMatch {
|
||||
readonly signature: BankSignature;
|
||||
readonly variant: BankHeaderVariant;
|
||||
/** The variant's roles, resolved to column indices of THIS file. */
|
||||
readonly roles: LexicalHeaderMap;
|
||||
}
|
||||
|
||||
/**
|
||||
* First-occurrence index of every normalized label of a header row.
|
||||
*
|
||||
* First occurrence, not last: a file repeating a label (`Montant;…;Montant`)
|
||||
* is degenerate either way, and taking the leftmost keeps the result stable
|
||||
* between the matcher and the drift diff, which both read this map.
|
||||
*/
|
||||
function labelIndex(headerRow: readonly string[]): Map<string, number> {
|
||||
const index = new Map<string, number>();
|
||||
headerRow.forEach((cell, i) => {
|
||||
const label = normalizeHeaderCell(cell);
|
||||
if (label && !index.has(label)) index.set(label, i);
|
||||
});
|
||||
return index;
|
||||
}
|
||||
|
||||
function rolesOf(
|
||||
variant: BankHeaderVariant,
|
||||
index: ReadonlyMap<string, number>
|
||||
): LexicalHeaderMap {
|
||||
const columnOf = (role: HeaderRole): number | null => {
|
||||
const label = variant.roles[role];
|
||||
if (label === undefined) return null;
|
||||
return index.get(label) ?? null;
|
||||
};
|
||||
|
||||
const date = columnOf("date");
|
||||
const description = columnOf("description");
|
||||
const amount = columnOf("amount");
|
||||
const debit = columnOf("debit");
|
||||
const credit = columnOf("credit");
|
||||
const balance = columnOf("balance");
|
||||
|
||||
return {
|
||||
date,
|
||||
description,
|
||||
amount,
|
||||
debit,
|
||||
credit,
|
||||
balance,
|
||||
roleCount: [date, description, amount, debit, credit, balance].filter(
|
||||
(c) => c !== null
|
||||
).length,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the bank whose documented layout this header row is.
|
||||
*
|
||||
* Three preconditions guard every match, each one a way the file says it is not
|
||||
* this export: a delimiter the bank does not use, more preamble than the bank
|
||||
* prints, or a whole-line-quoted file claiming to be a bank that does not quote.
|
||||
* They can only ever turn a match into a fall-back to the generic dictionary.
|
||||
*
|
||||
* Returns null for an unknown file — which is the common case and not an error.
|
||||
*/
|
||||
export function matchBankSignature(
|
||||
input: BankSignatureInput
|
||||
): BankSignatureMatch | null {
|
||||
const index = labelIndex(input.headerRow);
|
||||
if (index.size === 0) return null;
|
||||
|
||||
for (const signature of BANK_SIGNATURES) {
|
||||
if (!signature.delimiters.includes(input.delimiter)) continue;
|
||||
if (input.skipLines > signature.maxPreambleLines) continue;
|
||||
if (input.wholeLineQuoted && !signature.wholeLineQuoted) continue;
|
||||
|
||||
for (const variant of signature.variants) {
|
||||
if (!variant.labels.every((label) => index.has(label))) continue;
|
||||
// A variant made only of generic labels must describe the header EXACTLY,
|
||||
// not merely be contained in it. Desjardins is
|
||||
// `date;description;montant;solde` — four labels any Canadian bank could
|
||||
// emit — so accepting them as a SUBSET let a
|
||||
// `Date;Description;Débit;Crédit;Montant;Solde` file claim to be
|
||||
// Desjardins. A variant carrying a discriminating label (`chequenumber`,
|
||||
// `categorie`, `memo`, …) keeps subset matching: extra columns are fine
|
||||
// once something actually identifies the bank.
|
||||
if (
|
||||
variant.labels.every((label) => GENERIC_LABELS.has(label)) &&
|
||||
index.size !== variant.labels.length
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
return { signature, variant, roles: rolesOf(variant, index) };
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/** The bank behind an id, for the banner. Null for an id the table lost. */
|
||||
export function bankSignatureById(
|
||||
id: BankSignatureId | null | undefined
|
||||
): BankSignature | null {
|
||||
if (!id) return null;
|
||||
return BANK_SIGNATURES.find((s) => s.id === id) ?? null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// The stored signature and the drift it detects
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* The header row of a successful import, as `import_sources.header_signature`
|
||||
* stores it: a JSON array of normalized labels.
|
||||
*
|
||||
* Returns null for a file with NO header row — and that null is written to the
|
||||
* column as-is. A headerless source has no signature to compare against and
|
||||
* drift detection is inoperative on it, deliberately: a signature invented from
|
||||
* the data would fire on every change of content, which is every import.
|
||||
*/
|
||||
export function buildHeaderSignature(
|
||||
headers: readonly string[] | null | undefined
|
||||
): string | null {
|
||||
if (!headers || headers.length === 0) return null;
|
||||
return JSON.stringify(headers.map(normalizeHeaderCell));
|
||||
}
|
||||
|
||||
/**
|
||||
* Read back a stored signature. Anything that is not an array of strings — a
|
||||
* hash written by an older build, a truncated row, hand-edited SQL — comes back
|
||||
* null, which switches drift detection off rather than throwing inside an
|
||||
* import. Same rule as everywhere else here: degrade, never break.
|
||||
*/
|
||||
export function parseHeaderSignature(
|
||||
stored: string | null | undefined
|
||||
): string[] | null {
|
||||
if (!stored) return null;
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(stored);
|
||||
if (!Array.isArray(parsed)) return null;
|
||||
if (!parsed.every((label) => typeof label === "string")) return null;
|
||||
return parsed as string[];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export type HeaderDriftKind = "moved" | "added" | "removed";
|
||||
|
||||
export interface HeaderDriftEntry {
|
||||
readonly kind: HeaderDriftKind;
|
||||
/** The normalized label — the key the two signatures are compared on. */
|
||||
readonly normalizedLabel: string;
|
||||
/**
|
||||
* What to show. The file's own spelling for a column that still exists
|
||||
* (`Montant`), the normalized label for one that disappeared — the raw
|
||||
* spelling of a removed column is not recorded anywhere.
|
||||
*/
|
||||
readonly label: string;
|
||||
/** Column index at the last successful import; null for a new column. */
|
||||
readonly previousIndex: number | null;
|
||||
/** Column index in the file being imported; null for a column that vanished. */
|
||||
readonly currentIndex: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare the header row of the file being imported against the one the last
|
||||
* successful import recorded.
|
||||
*
|
||||
* Returns null when there is nothing to say — no stored signature, a headerless
|
||||
* file, an unreadable stored value, or a header that normalizes to exactly the
|
||||
* same labels in the same order. `Montant` becoming `MONTANT ($)` is NOT drift:
|
||||
* it normalizes identically and changes nothing about how the file is read.
|
||||
*
|
||||
* A non-null result is always a non-empty list, so the caller can treat it as
|
||||
* "show the panel".
|
||||
*/
|
||||
export function detectHeaderDrift(
|
||||
stored: string | null | undefined,
|
||||
currentHeaders: readonly string[] | null | undefined
|
||||
): HeaderDriftEntry[] | null {
|
||||
const previous = parseHeaderSignature(stored);
|
||||
if (!previous || previous.length === 0) return null;
|
||||
if (!currentHeaders || currentHeaders.length === 0) return null;
|
||||
|
||||
const current = currentHeaders.map(normalizeHeaderCell);
|
||||
if (
|
||||
previous.length === current.length &&
|
||||
previous.every((label, i) => label === current[i])
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const previousIndex = new Map<string, number>();
|
||||
previous.forEach((label, i) => {
|
||||
if (label && !previousIndex.has(label)) previousIndex.set(label, i);
|
||||
});
|
||||
const currentIndex = labelIndex(currentHeaders);
|
||||
|
||||
const entries: HeaderDriftEntry[] = [];
|
||||
|
||||
// Walk the file being imported first, so the panel reads in its column order.
|
||||
current.forEach((label, i) => {
|
||||
if (!label) return;
|
||||
if (currentIndex.get(label) !== i) return; // duplicate label, already handled
|
||||
const before = previousIndex.get(label);
|
||||
if (before === undefined) {
|
||||
entries.push({
|
||||
kind: "added",
|
||||
normalizedLabel: label,
|
||||
label: (currentHeaders[i] ?? "").trim() || label,
|
||||
previousIndex: null,
|
||||
currentIndex: i,
|
||||
});
|
||||
} else if (before !== i) {
|
||||
entries.push({
|
||||
kind: "moved",
|
||||
normalizedLabel: label,
|
||||
label: (currentHeaders[i] ?? "").trim() || label,
|
||||
previousIndex: before,
|
||||
currentIndex: i,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// Then the columns that are simply gone, in the order they used to be in.
|
||||
previous.forEach((label, i) => {
|
||||
if (!label) return;
|
||||
if (previousIndex.get(label) !== i) return;
|
||||
if (currentIndex.has(label)) return;
|
||||
entries.push({
|
||||
kind: "removed",
|
||||
normalizedLabel: label,
|
||||
label,
|
||||
previousIndex: i,
|
||||
currentIndex: null,
|
||||
});
|
||||
});
|
||||
|
||||
return entries.length > 0 ? entries : null;
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,7 +1,27 @@
|
|||
import Papa from "papaparse";
|
||||
import { parseDate } from "./dateParser";
|
||||
import { parseFrenchAmount } from "./amountParser";
|
||||
import type { ColumnMapping, AmountMode, SignConvention } from "../shared/types";
|
||||
import {
|
||||
matchBankSignature,
|
||||
type BankSignatureId,
|
||||
type BankSignatureMatch,
|
||||
} from "./bankSignatures";
|
||||
import {
|
||||
CREDIT_INDICATOR_TOKENS,
|
||||
DEBIT_INDICATOR_TOKENS,
|
||||
matchHeaderColumn,
|
||||
matchTransactionHeaders,
|
||||
MIN_HEADER_ROLE_MATCHES,
|
||||
normalizeHeaderCell,
|
||||
type LexicalHeaderMap,
|
||||
} from "./headerDictionary";
|
||||
import { detectAmountSeparators, mapRow } from "./importFormat";
|
||||
import type {
|
||||
ColumnMapping,
|
||||
AmountMode,
|
||||
ImportFormat,
|
||||
SignConvention,
|
||||
} from "../shared/types";
|
||||
|
||||
export interface AutoDetectResult {
|
||||
delimiter: string;
|
||||
|
|
@ -13,6 +33,72 @@ export interface AutoDetectResult {
|
|||
signConvention: SignConvention;
|
||||
}
|
||||
|
||||
/**
|
||||
* i18n key of a file detection recognised but REFUSES to configure. Refusing is
|
||||
* a feature: the alternative for this shape is a config that imports every
|
||||
* debit as income (see `findDirectionIndicatorColumn`).
|
||||
*/
|
||||
export type AutoDetectRejectionKey = "import.errors.absoluteIndicatorFormat";
|
||||
|
||||
/**
|
||||
* How well the detected configuration reads the file it was detected from.
|
||||
*
|
||||
* Detection used to return a configuration without ever testing it: a plausible
|
||||
* delimiter, a plausible date column and a plausible amount column produced a
|
||||
* result whether or not a single row survived them. The score closes that hole
|
||||
* by REPLAYING the configuration — the confidence reported is measured, not
|
||||
* asserted.
|
||||
*
|
||||
* What it does NOT measure is whether the amounts carry the right SIGN. A file
|
||||
* of unsigned magnitudes (`all-positive` in the corpus) scores 100 % while every
|
||||
* expense imports as income, because every row parses. That is why the threshold
|
||||
* only colours the banner and blocks nothing: the preview step (#329) is the
|
||||
* real net, and it is traversed at every import whatever the score says.
|
||||
*/
|
||||
export interface DetectionScore {
|
||||
/** Rows whose date AND amount the configuration reads. */
|
||||
readRows: number;
|
||||
/** Data rows the replay ran on — header and skipped preamble excluded. */
|
||||
totalRows: number;
|
||||
/** `readRows / totalRows`, or 0 when the file carries no data row. */
|
||||
ratio: number;
|
||||
/** `ratio >= CONFIDENCE_THRESHOLD`. */
|
||||
confident: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Share of rows that must be read for the result to be announced as recognised
|
||||
* rather than doubtful (decided in planning). A statement mixing a few unusable
|
||||
* lines into an otherwise clean file is common enough that a stricter bar would
|
||||
* cry wolf; anything below this is a mapping worth a second look.
|
||||
*/
|
||||
export const CONFIDENCE_THRESHOLD = 0.9;
|
||||
|
||||
/**
|
||||
* The full-fidelity outcome of detection.
|
||||
*
|
||||
* `autoDetectConfig` keeps its `AutoDetectResult | null` shape for the callers
|
||||
* that only need the configuration, but null cannot say WHY: "I could not read
|
||||
* this file" and "I read this file and it is a format this version imports
|
||||
* backwards" call for different messages. `detectImportFormat` separates them.
|
||||
*/
|
||||
export type AutoDetectOutcome =
|
||||
| {
|
||||
status: "ok";
|
||||
config: AutoDetectResult;
|
||||
score: DetectionScore;
|
||||
/**
|
||||
* The bank whose documented layout this file matched, or null when the
|
||||
* generic dictionary read the header (#330). Deliberately NOT part of
|
||||
* `config`: it says where the configuration came from, it is not one of
|
||||
* the fields that decide how a row is read, and it is never persisted as
|
||||
* format.
|
||||
*/
|
||||
bank: BankSignatureId | null;
|
||||
}
|
||||
| { status: "rejected"; reason: AutoDetectRejectionKey }
|
||||
| { status: "failed" };
|
||||
|
||||
const DATE_FORMATS = [
|
||||
"DD/MM/YYYY",
|
||||
"MM/DD/YYYY",
|
||||
|
|
@ -25,6 +111,12 @@ const DATE_FORMATS = [
|
|||
|
||||
const DELIMITERS = [",", ";", "\t"];
|
||||
|
||||
/** Every token a direction-indicator column may hold, both directions. */
|
||||
const DIRECTION_INDICATOR_TOKENS = new Set([
|
||||
...DEBIT_INDICATOR_TOKENS,
|
||||
...CREDIT_INDICATOR_TOKENS,
|
||||
]);
|
||||
|
||||
/**
|
||||
* Detect and unwrap Desjardins-style CSVs where each entire line is
|
||||
* wrapped in quotes with "" escaping inside.
|
||||
|
|
@ -52,20 +144,48 @@ export function preprocessQuotedCSV(content: string): string {
|
|||
|
||||
/**
|
||||
* Analyze raw CSV content and return a suggested configuration,
|
||||
* or null if detection fails.
|
||||
* or null if detection fails OR the file is a refused format.
|
||||
*
|
||||
* Callers that must tell those two apart — the wizard, which owes the user a
|
||||
* message — use `detectImportFormat` instead.
|
||||
*/
|
||||
export function autoDetectConfig(rawContent: string): AutoDetectResult | null {
|
||||
const outcome = detectImportFormat(rawContent);
|
||||
return outcome.status === "ok" ? outcome.config : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Analyze raw CSV content and return a suggested configuration, the reason the
|
||||
* file is refused, or a plain failure.
|
||||
*
|
||||
* The pipeline is unchanged in its bones — delimiter, preamble, header, date,
|
||||
* numeric columns, amount mode — with a LEXICAL layer in front of it: when the
|
||||
* file has a header row, `matchTransactionHeaders` reads its labels and the
|
||||
* labels win. Every lexical hint is a preference, never a constraint: a label
|
||||
* naming a column the data contradicts (a "Date" column nothing parses as a
|
||||
* date, a "Solde" column that is the only amount candidate left) is dropped and
|
||||
* the shape heuristic decides, exactly as it did before #327.
|
||||
*
|
||||
* In FRONT of that lexical layer sits the bank-signature table (#330): a header
|
||||
* row matching a documented export layout gets its roles from that layout and
|
||||
* the generic dictionary is not consulted at all. The hints it produces are the
|
||||
* same kind of preference — the file is reported as recognised, not read
|
||||
* differently on trust. An unknown header falls straight through to the
|
||||
* dictionary, which is what every file did before.
|
||||
*/
|
||||
export function detectImportFormat(rawContent: string): AutoDetectOutcome {
|
||||
const failed: AutoDetectOutcome = { status: "failed" };
|
||||
const content = preprocessQuotedCSV(rawContent);
|
||||
const lines = content.split(/\r?\n/).filter((l) => l.trim());
|
||||
if (lines.length < 2) return null;
|
||||
if (lines.length < 2) return failed;
|
||||
|
||||
// Step 1: Detect delimiter
|
||||
const delimiter = detectDelimiter(lines.slice(0, 10));
|
||||
if (!delimiter) return null;
|
||||
if (!delimiter) return failed;
|
||||
|
||||
const parsed = Papa.parse(content, { delimiter, skipEmptyLines: true });
|
||||
const data = parsed.data as string[][];
|
||||
if (data.length < 2) return null;
|
||||
if (data.length < 2) return failed;
|
||||
|
||||
// Step 1b: Detect preamble lines to skip
|
||||
// Find the expected column count (most frequent count > 1)
|
||||
|
|
@ -92,20 +212,43 @@ export function autoDetectConfig(rawContent: string): AutoDetectResult | null {
|
|||
}
|
||||
|
||||
const effectiveData = data.slice(skipLines);
|
||||
if (effectiveData.length < 2) return null;
|
||||
if (effectiveData.length < 2) return failed;
|
||||
|
||||
// Step 2: Detect header
|
||||
const hasHeader = detectHeader(effectiveData[0]);
|
||||
|
||||
// Step 2b: Try the known banks first (#330). A signature is claimed on the
|
||||
// WHOLE normalized label, delimiter and preamble included, so an unknown file
|
||||
// simply does not match and the generic dictionary reads it as before. A
|
||||
// headerless file matches nothing by construction — there is no label to read.
|
||||
const signature = hasHeader
|
||||
? matchBankSignature({
|
||||
headerRow: effectiveData[0],
|
||||
delimiter,
|
||||
skipLines,
|
||||
// `preprocessQuotedCSV` returns its input untouched when the file is not
|
||||
// whole-line-quoted, so this comparison IS the quirk.
|
||||
wholeLineQuoted: content !== rawContent,
|
||||
})
|
||||
: null;
|
||||
|
||||
// Step 2c: Read the header labels. A signature that matched has already named
|
||||
// the roles; otherwise the generic dictionary does. A headerless file yields
|
||||
// no map at all, which is the explicit fall-back: every step below then runs
|
||||
// on shape only.
|
||||
const lexical = hasHeader
|
||||
? (signature?.roles ?? matchTransactionHeaders(effectiveData[0]))
|
||||
: null;
|
||||
|
||||
const dataStartIdx = hasHeader ? 1 : 0;
|
||||
const sampleRows = effectiveData.slice(dataStartIdx, dataStartIdx + 20);
|
||||
if (sampleRows.length === 0) return null;
|
||||
if (sampleRows.length === 0) return failed;
|
||||
|
||||
const colCount = Math.max(...effectiveData.slice(0, 10).map((r) => r.length));
|
||||
|
||||
// Step 3: Detect date column + format
|
||||
const dateResult = detectDateColumn(sampleRows, colCount);
|
||||
if (!dateResult) return null;
|
||||
const dateResult = detectDateColumn(sampleRows, colCount, lexical?.date);
|
||||
if (!dateResult) return failed;
|
||||
|
||||
// Step 3b: Find ALL date-like columns (to exclude from amount candidates)
|
||||
const dateLikeCols = new Set<number>();
|
||||
|
|
@ -129,23 +272,56 @@ export function autoDetectConfig(rawContent: string): AutoDetectResult | null {
|
|||
// Step 4: Detect numeric columns
|
||||
const numericCols = detectNumericColumns(sampleRows, colCount);
|
||||
|
||||
// Step 5: Detect balance columns and exclude them + date-like columns
|
||||
// Step 5: Detect balance columns and exclude them + date-like columns.
|
||||
// A column LABELLED "Solde"/"Balance" is excluded too — the arithmetic test
|
||||
// needs three consecutive rows and a matching amount column to fire, so a
|
||||
// short file or a statement whose balance does not reconcile keeps its
|
||||
// running balance in the running for the amount column. The exclusion is
|
||||
// dropped if it would leave nothing to map (see `narrowCandidates`).
|
||||
const balanceCols = detectBalanceColumns(sampleRows, numericCols);
|
||||
const amountCandidates = numericCols.filter(
|
||||
const shapeCandidates = numericCols.filter(
|
||||
(c) => !balanceCols.has(c) && !dateLikeCols.has(c)
|
||||
);
|
||||
const amountCandidates = narrowCandidates(
|
||||
shapeCandidates,
|
||||
(c) => c !== lexical?.balance
|
||||
);
|
||||
|
||||
// Step 6: Detect description column
|
||||
const descriptionCol = detectDescriptionColumn(
|
||||
sampleRows,
|
||||
colCount,
|
||||
dateResult.column,
|
||||
new Set([...numericCols, ...dateLikeCols])
|
||||
new Set([...numericCols, ...dateLikeCols]),
|
||||
lexical?.description
|
||||
);
|
||||
|
||||
// Step 7: Determine amount mode
|
||||
const amountResult = detectAmountMode(sampleRows, amountCandidates);
|
||||
if (!amountResult) return null;
|
||||
const amountResult = detectAmountMode(
|
||||
sampleRows,
|
||||
amountCandidates,
|
||||
lexical,
|
||||
signature
|
||||
);
|
||||
if (!amountResult) return failed;
|
||||
|
||||
// Step 7b: Refuse the third amount format — unsigned magnitudes plus a
|
||||
// neighbouring D/C column. Nothing here reads that column, so the file would
|
||||
// be configured as `positive_expense` and every credit imported as an
|
||||
// expense. Detected, refused, and left to a future `absolute_indicator` mode.
|
||||
if (amountResult.mode === "single") {
|
||||
const indicatorCol = findDirectionIndicatorColumn(
|
||||
sampleRows,
|
||||
amountResult.amountCol,
|
||||
colCount
|
||||
);
|
||||
if (indicatorCol !== null) {
|
||||
return {
|
||||
status: "rejected",
|
||||
reason: "import.errors.absoluteIndicatorFormat",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const mapping: ColumnMapping = {
|
||||
date: dateResult.column,
|
||||
|
|
@ -162,7 +338,7 @@ export function autoDetectConfig(rawContent: string): AutoDetectResult | null {
|
|||
signConvention = amountResult.signConvention;
|
||||
}
|
||||
|
||||
return {
|
||||
const config: AutoDetectResult = {
|
||||
delimiter,
|
||||
hasHeader,
|
||||
skipLines,
|
||||
|
|
@ -171,6 +347,83 @@ export function autoDetectConfig(rawContent: string): AutoDetectResult | null {
|
|||
amountMode: amountResult.mode,
|
||||
signConvention,
|
||||
};
|
||||
|
||||
// Step 8: Replay what we just decided, over the WHOLE file. The sample above
|
||||
// is 20 rows because that is enough to decide a shape; the score is what the
|
||||
// user is told, so it has to describe the actual file.
|
||||
return {
|
||||
status: "ok",
|
||||
config,
|
||||
score: scoreConfig(config, data),
|
||||
bank: signature?.signature.id ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Replay a detected configuration over the file's data rows and count how many
|
||||
* of them it reads.
|
||||
*
|
||||
* Deliberately NOT a rule of its own: the row is mapped by `mapRow`, the same
|
||||
* pure function `parseFilesInternal` runs at import time, under the same
|
||||
* column-level decimal arbitration. Two mappers would be the exact class of
|
||||
* divergence this chantier exists to remove — a score could then read 100 %
|
||||
* while the import wrote different amounts.
|
||||
*
|
||||
* The row selection mirrors `parseFilesInternal` line for line, including the
|
||||
* lone-empty-cell skip: a trailing blank line must not count as an unread row.
|
||||
*/
|
||||
function scoreConfig(
|
||||
config: AutoDetectResult,
|
||||
data: string[][]
|
||||
): DetectionScore {
|
||||
// `mapRow` reads no byte and therefore never looks at `encoding` — the
|
||||
// content reached us already decoded. Naming it here only satisfies the type.
|
||||
const format: ImportFormat = { ...config, encoding: "utf-8" };
|
||||
|
||||
const dataRows: string[][] = [];
|
||||
const startIdx = config.skipLines + (config.hasHeader ? 1 : 0);
|
||||
for (let i = startIdx; i < data.length; i++) {
|
||||
const raw = data[i];
|
||||
if (raw.length <= 1 && raw[0]?.trim() === "") continue;
|
||||
dataRows.push(raw);
|
||||
}
|
||||
|
||||
const decimalSeparators = detectAmountSeparators(dataRows, format);
|
||||
|
||||
let readRows = 0;
|
||||
for (const raw of dataRows) {
|
||||
try {
|
||||
if (mapRow(raw, format, { decimalSeparators }).parsed) readRows++;
|
||||
} catch {
|
||||
// `mapRow` is documented not to throw; if it ever did, that is one
|
||||
// unreadable row, not a detection that collapses on the whole file.
|
||||
}
|
||||
}
|
||||
|
||||
const totalRows = dataRows.length;
|
||||
const ratio = totalRows === 0 ? 0 : readRows / totalRows;
|
||||
return {
|
||||
readRows,
|
||||
totalRows,
|
||||
ratio,
|
||||
confident: totalRows > 0 && ratio >= CONFIDENCE_THRESHOLD,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply a lexical restriction to a candidate list, unless it would empty it.
|
||||
*
|
||||
* The lexical layer must never be able to turn "detected, possibly wrong" into
|
||||
* "detected nothing": a file whose only amount column happens to be labelled
|
||||
* `Solde` is still importable, and a user can fix a mapping far more easily
|
||||
* than a refusal to configure.
|
||||
*/
|
||||
function narrowCandidates(
|
||||
candidates: number[],
|
||||
keep: (col: number) => boolean
|
||||
): number[] {
|
||||
const narrowed = candidates.filter(keep);
|
||||
return narrowed.length > 0 ? narrowed : candidates;
|
||||
}
|
||||
|
||||
function detectDelimiter(lines: string[]): string | null {
|
||||
|
|
@ -211,6 +464,24 @@ function detectDelimiter(lines: string[]): string | null {
|
|||
return bestDelimiter;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether the first row is a header.
|
||||
*
|
||||
* The shape test — no parseable date, no parseable number — is kept and comes
|
||||
* first. It cannot classify a header whose cells carry bare numbers (a column
|
||||
* literally titled `2025`), and treating that row as data costs one lost
|
||||
* transaction plus a mapping computed from a sample that starts with a label
|
||||
* row. So a row the shape test rejects ONLY because of a number is put to the
|
||||
* lexical layer: two named roles in one row is a header.
|
||||
*
|
||||
* The date test stays absolute. A row carrying a parseable date is data, no
|
||||
* matter what its other cells spell — that is the guard that keeps a
|
||||
* transaction like `DEPOT PAIE EMPLOYEUR` (which contains the credit keyword
|
||||
* `depot`) from being swallowed as a header.
|
||||
*
|
||||
* The holdings flow calls this too. Its headers name no transaction role, so
|
||||
* the lexical clause never fires there.
|
||||
*/
|
||||
function detectHeader(firstRow: string[]): boolean {
|
||||
// A header row typically has no parseable dates and no parseable numbers
|
||||
let hasDate = false;
|
||||
|
|
@ -234,33 +505,59 @@ function detectHeader(firstRow: string[]): boolean {
|
|||
}
|
||||
}
|
||||
|
||||
return !hasDate && !hasNumber;
|
||||
if (hasDate) return false;
|
||||
if (!hasNumber) return true;
|
||||
|
||||
return matchTransactionHeaders(firstRow).roleCount >= MIN_HEADER_ROLE_MATCHES;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the date column and its format.
|
||||
*
|
||||
* `preferred` is the column the header labels name. It wins only if the data
|
||||
* agrees — same 0.8 parse rate the shape scan demands — so a file whose "Date
|
||||
* de production" column holds no date falls back to the best-parsing column
|
||||
* rather than mapping a label nothing supports. Its real job is arbitrating
|
||||
* between several equally parseable date columns (`Date de transaction` vs
|
||||
* `Date comptable`), which the rate alone cannot do.
|
||||
*/
|
||||
function detectDateColumn(
|
||||
rows: string[][],
|
||||
colCount: number
|
||||
colCount: number,
|
||||
preferred?: number | null
|
||||
): { column: number; format: string } | null {
|
||||
const rateOf = (col: number, fmt: string): number => {
|
||||
let success = 0;
|
||||
let total = 0;
|
||||
for (const row of rows) {
|
||||
const cell = row[col]?.trim();
|
||||
if (!cell) continue;
|
||||
total++;
|
||||
if (parseDate(cell, fmt)) success++;
|
||||
}
|
||||
return total === 0 ? 0 : success / total;
|
||||
};
|
||||
|
||||
if (preferred !== undefined && preferred !== null && preferred < colCount) {
|
||||
let bestFormat = "";
|
||||
let bestRate = 0;
|
||||
for (const fmt of DATE_FORMATS) {
|
||||
const rate = rateOf(preferred, fmt);
|
||||
if (rate > bestRate) {
|
||||
bestRate = rate;
|
||||
bestFormat = fmt;
|
||||
}
|
||||
}
|
||||
if (bestRate >= 0.8) return { column: preferred, format: bestFormat };
|
||||
}
|
||||
|
||||
let bestCol = -1;
|
||||
let bestFormat = "";
|
||||
let bestRate = 0;
|
||||
|
||||
for (let col = 0; col < colCount; col++) {
|
||||
for (const fmt of DATE_FORMATS) {
|
||||
let success = 0;
|
||||
let total = 0;
|
||||
|
||||
for (const row of rows) {
|
||||
const cell = row[col]?.trim();
|
||||
if (!cell) continue;
|
||||
total++;
|
||||
if (parseDate(cell, fmt)) {
|
||||
success++;
|
||||
}
|
||||
}
|
||||
|
||||
if (total === 0) continue;
|
||||
const rate = success / total;
|
||||
const rate = rateOf(col, fmt);
|
||||
if (rate > bestRate) {
|
||||
bestRate = rate;
|
||||
bestCol = col;
|
||||
|
|
@ -401,12 +698,64 @@ function detectBalanceColumns(
|
|||
return balanceCols;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the description column: the one the header labels name, else the
|
||||
* longest-on-average text column.
|
||||
*
|
||||
* The label is only honoured for a column the shape test would have been
|
||||
* allowed to pick at all — not the date column, not a numeric one. A file
|
||||
* labelling its amount column `Détail du montant` therefore cannot end up with
|
||||
* its amounts in the description.
|
||||
*/
|
||||
/**
|
||||
* Does this column behave like an enumeration rather than free text?
|
||||
*
|
||||
* A label alone is not enough to pick the description: Tangerine exports
|
||||
* `Date,Transaction,Name,Memo,Amount`, where `Transaction` holds DEBIT/CREDIT
|
||||
* and `Name` holds the merchant. Honouring the label there moves the merchant
|
||||
* out of the description and kills keyword categorisation.
|
||||
*
|
||||
* Cardinality separates the two — a description repeats almost nothing, an enum
|
||||
* repeats almost everything. Average length does NOT: `Note` and `Libellé` are
|
||||
* both short, and vetoing on length would reject legitimate columns.
|
||||
*/
|
||||
function looksLikeEnumColumn(rows: string[][], col: number): boolean {
|
||||
const distinct = new Set<string>();
|
||||
let filled = 0;
|
||||
|
||||
for (const row of rows) {
|
||||
const cell = row[col]?.trim();
|
||||
if (!cell) continue;
|
||||
filled++;
|
||||
distinct.add(cell.toLowerCase());
|
||||
}
|
||||
|
||||
// A labelled but entirely empty column is never the description either.
|
||||
if (filled === 0) return true;
|
||||
// Too few rows to read anything into the cardinality.
|
||||
if (filled < 4) return false;
|
||||
|
||||
return distinct.size <= Math.max(2, Math.floor(filled / 4));
|
||||
}
|
||||
|
||||
function detectDescriptionColumn(
|
||||
rows: string[][],
|
||||
colCount: number,
|
||||
dateCol: number,
|
||||
numericCols: Set<number>
|
||||
numericCols: Set<number>,
|
||||
preferred?: number | null
|
||||
): number {
|
||||
if (
|
||||
preferred !== undefined &&
|
||||
preferred !== null &&
|
||||
preferred < colCount &&
|
||||
preferred !== dateCol &&
|
||||
!numericCols.has(preferred) &&
|
||||
!looksLikeEnumColumn(rows, preferred)
|
||||
) {
|
||||
return preferred;
|
||||
}
|
||||
|
||||
let bestCol = 0;
|
||||
let bestAvgLen = 0;
|
||||
|
||||
|
|
@ -447,31 +796,184 @@ interface DebitCreditResult {
|
|||
|
||||
type AmountModeResult = SingleAmountResult | DebitCreditResult;
|
||||
|
||||
function detectAmountMode(
|
||||
/**
|
||||
* Decide the amount mode, and which column(s) carry it.
|
||||
*
|
||||
* `signature` is the one hint that outranks the sparse-complementary scan, and
|
||||
* only because that scan cannot be told apart from the truth by shape alone:
|
||||
* RBC's `CAD$` and `USD$` ARE complementary — the USD column is empty on a
|
||||
* Canadian account — so a file whose amounts are one signed column reads as a
|
||||
* debit/credit pair, with every credit imported as an expense. A bank that
|
||||
* documents its layout settles that; nothing else in the file can.
|
||||
*
|
||||
* It stays a preference all the same: the columns it names must be candidates
|
||||
* the shape scan itself proposed. A signature naming a column that parses as
|
||||
* nothing numeric is dropped here and the generic path decides, exactly like a
|
||||
* mismatched label.
|
||||
*/
|
||||
/**
|
||||
* The first sparse-complementary pair among the candidates, in column order, or
|
||||
* null. Extracted from `detectAmountMode` so a bank signature can be arbitrated
|
||||
* AGAINST the pair the shape scan would have found, instead of short-circuiting
|
||||
* a scan that never ran.
|
||||
*/
|
||||
function findSparseComplementaryPair(
|
||||
rows: string[][],
|
||||
amountCandidates: number[]
|
||||
): [number, number] | null {
|
||||
for (let a = 0; a < amountCandidates.length; a++) {
|
||||
for (let b = a + 1; b < amountCandidates.length; b++) {
|
||||
const colA = amountCandidates[a];
|
||||
const colB = amountCandidates[b];
|
||||
if (isSparseComplementary(rows, colA, colB)) return [colA, colB];
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function detectAmountMode(
|
||||
rows: string[][],
|
||||
amountCandidates: number[],
|
||||
lexical: LexicalHeaderMap | null,
|
||||
signature: BankSignatureMatch | null
|
||||
): AmountModeResult | null {
|
||||
if (amountCandidates.length === 0) return null;
|
||||
|
||||
const pair = findSparseComplementaryPair(rows, amountCandidates);
|
||||
|
||||
if (signature) {
|
||||
const { debit, credit, amount } = signature.roles;
|
||||
if (
|
||||
debit !== null &&
|
||||
credit !== null &&
|
||||
amountCandidates.includes(debit) &&
|
||||
amountCandidates.includes(credit)
|
||||
) {
|
||||
return { mode: "debit_credit", debitCol: debit, creditCol: credit };
|
||||
}
|
||||
// A signature's SINGLE amount column may not silently displace a genuine
|
||||
// debit/credit pair it has no part in. RBC needs the override — its
|
||||
// near-empty `Cheque Number` really is sparse-complementary with `CAD$` —
|
||||
// but there the pair contains the declared amount column. On a
|
||||
// `Date;Description;Débit;Crédit;Montant;Solde` file the pair does not, and
|
||||
// taking `Montant` unsigned imported every deposit as an expense.
|
||||
if (
|
||||
amount !== null &&
|
||||
amountCandidates.includes(amount) &&
|
||||
(!pair || pair.includes(amount))
|
||||
) {
|
||||
return detectSingleAmount(rows, amount);
|
||||
}
|
||||
}
|
||||
|
||||
if (amountCandidates.length === 1) {
|
||||
return detectSingleAmount(rows, amountCandidates[0]);
|
||||
}
|
||||
|
||||
// Check for sparse-complementary pair (debit/credit pattern)
|
||||
for (let a = 0; a < amountCandidates.length; a++) {
|
||||
for (let b = a + 1; b < amountCandidates.length; b++) {
|
||||
const colA = amountCandidates[a];
|
||||
const colB = amountCandidates[b];
|
||||
if (pair) {
|
||||
return orderDebitCredit(pair[0], pair[1], lexical);
|
||||
}
|
||||
|
||||
if (isSparseComplementary(rows, colA, colB)) {
|
||||
return { mode: "debit_credit", debitCol: colA, creditCol: colB };
|
||||
// No complementary pair found — the labelled amount column if there is one,
|
||||
// else the best single amount column by shape.
|
||||
const preferred = lexical?.amount;
|
||||
const bestCol =
|
||||
preferred !== undefined &&
|
||||
preferred !== null &&
|
||||
amountCandidates.includes(preferred)
|
||||
? preferred
|
||||
: pickBestAmountColumn(rows, amountCandidates);
|
||||
return detectSingleAmount(rows, bestCol);
|
||||
}
|
||||
|
||||
/**
|
||||
* Assign the two halves of a complementary pair to debit and credit.
|
||||
*
|
||||
* The loop above enumerates `a < b`, so before #327 the LEFTMOST column was the
|
||||
* debit, always: a file laid out `Date;Description;Crédit;Débit` was mapped
|
||||
* backwards and every sign of the import came out inverted — silently, since
|
||||
* the total is merely negated and no aggregate check notices.
|
||||
*
|
||||
* The labels decide when they name either half; position remains the fall-back
|
||||
* for a headerless file or labels the dictionary does not know. Knowing ONE of
|
||||
* the two is enough: the other column is the other role.
|
||||
*/
|
||||
function orderDebitCredit(
|
||||
colA: number,
|
||||
colB: number,
|
||||
lexical: LexicalHeaderMap | null
|
||||
): DebitCreditResult {
|
||||
const debitFirst = {
|
||||
mode: "debit_credit",
|
||||
debitCol: colA,
|
||||
creditCol: colB,
|
||||
} as const;
|
||||
const creditFirst = {
|
||||
mode: "debit_credit",
|
||||
debitCol: colB,
|
||||
creditCol: colA,
|
||||
} as const;
|
||||
|
||||
if (lexical?.debit === colA || lexical?.credit === colB) return debitFirst;
|
||||
if (lexical?.debit === colB || lexical?.credit === colA) return creditFirst;
|
||||
return debitFirst;
|
||||
}
|
||||
|
||||
/**
|
||||
* Find a direction-indicator column next to an all-positive amount column —
|
||||
* the third amount format, refused rather than imported.
|
||||
*
|
||||
* Three conditions, each one narrowing a way to raise a false alarm:
|
||||
* - the amount column carries no negative value, since a file that signs its
|
||||
* amounts needs no indicator and reads correctly today;
|
||||
* - the candidate is IMMEDIATELY next to it, as every export of this shape
|
||||
* writes the flag beside the magnitude. A file-wide scan would refuse
|
||||
* perfectly importable files over an unrelated one-letter flag column, and
|
||||
* a refusal the user cannot work around is worse than a mapping they can;
|
||||
* - it holds at least two DISTINCT tokens of the D/C alphabet. A column stuck
|
||||
* on a single value carries no direction, and a statement of pure expenses
|
||||
* imports correctly as `positive_expense`.
|
||||
*/
|
||||
function findDirectionIndicatorColumn(
|
||||
rows: string[][],
|
||||
amountCol: number,
|
||||
colCount: number
|
||||
): number | null {
|
||||
let seen = 0;
|
||||
for (const row of rows) {
|
||||
const cell = row[amountCol]?.trim();
|
||||
if (!cell) continue;
|
||||
const val = parseFrenchAmount(cell);
|
||||
if (isNaN(val)) continue;
|
||||
seen++;
|
||||
if (val < 0) return null;
|
||||
}
|
||||
if (seen === 0) return null;
|
||||
|
||||
for (const col of [amountCol - 1, amountCol + 1]) {
|
||||
if (col < 0 || col >= colCount) continue;
|
||||
|
||||
const distinct = new Set<string>();
|
||||
let nonEmpty = 0;
|
||||
let inAlphabet = 0;
|
||||
|
||||
for (const row of rows) {
|
||||
const cell = row[col]?.trim();
|
||||
if (!cell) continue;
|
||||
nonEmpty++;
|
||||
const token = cell.toLowerCase();
|
||||
if (DIRECTION_INDICATOR_TOKENS.has(token)) {
|
||||
inAlphabet++;
|
||||
distinct.add(token);
|
||||
}
|
||||
}
|
||||
|
||||
if (nonEmpty > 0 && inAlphabet === nonEmpty && distinct.size >= 2) {
|
||||
return col;
|
||||
}
|
||||
}
|
||||
|
||||
// No complementary pair found — pick best single amount column
|
||||
const bestCol = pickBestAmountColumn(rows, amountCandidates);
|
||||
return detectSingleAmount(rows, bestCol);
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Pick the best amount column: prefer columns with decimal values (cents). */
|
||||
|
|
@ -627,40 +1129,12 @@ const BOOKCOST_HEADER_KEYWORDS = [
|
|||
"cout",
|
||||
];
|
||||
// Value/market-value columns must never be auto-picked as price or book_cost.
|
||||
// NOTE `montant` is an EXCLUSION token here and the PRIMARY amount keyword of
|
||||
// the transaction dictionary (`headerDictionary.ts`) \u2014 which is why the two
|
||||
// tables are separate modules. `normalizeHeaderCell` and `matchHeaderColumn`
|
||||
// are the generic part, shared from there and unchanged.
|
||||
const VALUE_HEADER_KEYWORDS = ["value", "valeur", "montant", "marchande"];
|
||||
|
||||
/** Accent-strip + lowercase + keep alphanumerics only (for header matching). */
|
||||
function normalizeHeaderCell(s: string): string {
|
||||
return (s ?? "")
|
||||
.normalize("NFD")
|
||||
.replace(/[\u0300-\u036f]/g, "")
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]/g, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the first unused column whose normalized header contains one of the
|
||||
* keywords (keywords tried in priority order). `exclude` skips columns whose
|
||||
* header contains any excluded token (e.g. a "value" column for price).
|
||||
*/
|
||||
function matchHeaderColumn(
|
||||
normalizedHeaders: string[],
|
||||
keywords: string[],
|
||||
used: Set<number>,
|
||||
exclude: string[] = []
|
||||
): number | null {
|
||||
for (const kw of keywords) {
|
||||
for (let i = 0; i < normalizedHeaders.length; i++) {
|
||||
if (used.has(i)) continue;
|
||||
const h = normalizedHeaders[i];
|
||||
if (!h) continue;
|
||||
if (exclude.some((ex) => h.includes(ex))) continue;
|
||||
if (h.includes(kw)) return i;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Pick the shortest-average-length non-numeric, unused text column (symbols
|
||||
* are short tokens; a name/description column is longer). */
|
||||
function pickSymbolColumn(
|
||||
|
|
|
|||
160
src/utils/headerDictionary.test.ts
Normal file
160
src/utils/headerDictionary.test.ts
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
// headerDictionary — the lexical layer of transaction-CSV detection (#327).
|
||||
//
|
||||
// `csvAutoDetect.test.ts` covers what the dictionary DOES to a detected
|
||||
// configuration; this file covers the dictionary itself: every keyword it
|
||||
// claims to know, the exclusions that keep a label from claiming the wrong
|
||||
// column, and the mute row that hands control back to the shape heuristics.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
matchHeaderColumn,
|
||||
matchTransactionHeaders,
|
||||
normalizeHeaderCell,
|
||||
MIN_HEADER_ROLE_MATCHES,
|
||||
} from "./headerDictionary";
|
||||
|
||||
describe("normalizeHeaderCell (#245, shared by #327)", () => {
|
||||
it("strips accents, case and punctuation", () => {
|
||||
expect(normalizeHeaderCell("Débit")).toBe("debit");
|
||||
expect(normalizeHeaderCell("Libellé")).toBe("libelle");
|
||||
expect(normalizeHeaderCell("Date de l'opération")).toBe("datedeloperation");
|
||||
expect(normalizeHeaderCell("Montant ($)")).toBe("montant");
|
||||
});
|
||||
|
||||
it("keeps digits and tolerates an empty or missing cell", () => {
|
||||
expect(normalizeHeaderCell("Solde 2024")).toBe("solde2024");
|
||||
expect(normalizeHeaderCell("")).toBe("");
|
||||
expect(normalizeHeaderCell(undefined as unknown as string)).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("matchHeaderColumn (#245, shared by #327)", () => {
|
||||
const headers = ["date", "libelle", "montant", "solde"];
|
||||
|
||||
it("returns the first unused column containing a keyword", () => {
|
||||
expect(matchHeaderColumn(headers, ["montant"], new Set())).toBe(2);
|
||||
});
|
||||
|
||||
it("honours keyword priority order over column order", () => {
|
||||
expect(matchHeaderColumn(headers, ["solde", "date"], new Set())).toBe(3);
|
||||
});
|
||||
|
||||
it("skips used and excluded columns, and returns null on no match", () => {
|
||||
expect(matchHeaderColumn(headers, ["date"], new Set([0]))).toBeNull();
|
||||
expect(matchHeaderColumn(headers, ["mont"], new Set(), ["solde"])).toBe(2);
|
||||
expect(matchHeaderColumn(headers, ["quantite"], new Set())).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("matchTransactionHeaders — the FR/EN dictionary (#327)", () => {
|
||||
it("reads a French signed-amount header", () => {
|
||||
const m = matchTransactionHeaders(["Date", "Description", "Montant", "Solde"]);
|
||||
expect(m).toEqual({
|
||||
date: 0,
|
||||
description: 1,
|
||||
amount: 2,
|
||||
debit: null,
|
||||
credit: null,
|
||||
balance: 3,
|
||||
roleCount: 4,
|
||||
});
|
||||
});
|
||||
|
||||
it("reads an English signed-amount header", () => {
|
||||
const m = matchTransactionHeaders(["Date", "Description", "Amount", "Balance"]);
|
||||
expect(m.amount).toBe(2);
|
||||
expect(m.balance).toBe(3);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["Débit", "debit"],
|
||||
["Retrait", "debit"],
|
||||
["Déboursé", "debit"],
|
||||
["Withdrawal", "debit"],
|
||||
["Crédit", "credit"],
|
||||
["Dépôt", "credit"],
|
||||
["Encaissement", "credit"],
|
||||
["Deposit", "credit"],
|
||||
] as const)("knows %s as the %s column", (label, role) => {
|
||||
const m = matchTransactionHeaders(["Date", "Description", label]);
|
||||
expect(m[role]).toBe(2);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["Libellé", 1],
|
||||
["Détail", 1],
|
||||
["Transaction", 1],
|
||||
["Description", 1],
|
||||
] as const)("knows %s as the description column", (label, col) => {
|
||||
expect(matchTransactionHeaders(["Date", label, "Montant"]).description).toBe(
|
||||
col
|
||||
);
|
||||
});
|
||||
|
||||
it("resolves a reversed pair by label, not by position", () => {
|
||||
const m = matchTransactionHeaders(["Date", "Description", "Credit", "Debit"]);
|
||||
expect(m.debit).toBe(3);
|
||||
expect(m.credit).toBe(2);
|
||||
});
|
||||
|
||||
it("claims a column once, so debit/credit beat the amount keyword", () => {
|
||||
// Both columns contain `montant`. Resolving debit and credit first is what
|
||||
// stops `Montant débit` from being read as THE amount column.
|
||||
const m = matchTransactionHeaders([
|
||||
"Date",
|
||||
"Libelle",
|
||||
"Montant debit",
|
||||
"Montant credit",
|
||||
]);
|
||||
expect(m.debit).toBe(2);
|
||||
expect(m.credit).toBe(3);
|
||||
expect(m.amount).toBeNull();
|
||||
});
|
||||
|
||||
it("never reads a Solde column as the amount", () => {
|
||||
const m = matchTransactionHeaders(["Date", "Libelle", "Solde du compte"]);
|
||||
expect(m.balance).toBe(2);
|
||||
expect(m.amount).toBeNull();
|
||||
});
|
||||
|
||||
it("never reads a date column as the description", () => {
|
||||
// `Date de transaction` contains the description keyword `transaction`.
|
||||
const m = matchTransactionHeaders([
|
||||
"Date de transaction",
|
||||
"Date comptable",
|
||||
"Montant",
|
||||
]);
|
||||
expect(m.date).toBe(0);
|
||||
expect(m.description).toBeNull();
|
||||
});
|
||||
|
||||
it("returns a map of nulls for labels it does not know", () => {
|
||||
const m = matchTransactionHeaders(["Col A", "Col B", "Col C"]);
|
||||
expect(m).toEqual({
|
||||
date: null,
|
||||
description: null,
|
||||
amount: null,
|
||||
debit: null,
|
||||
credit: null,
|
||||
balance: null,
|
||||
roleCount: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("scores a data row below the header threshold", () => {
|
||||
// `DEPOT PAIE EMPLOYEUR` contains the credit keyword `depot` — one role,
|
||||
// which is exactly why one is not enough to call a row a header.
|
||||
const m = matchTransactionHeaders([
|
||||
"05/01/2025",
|
||||
"DEPOT PAIE EMPLOYEUR",
|
||||
"-84,32",
|
||||
]);
|
||||
expect(m.credit).toBe(1);
|
||||
expect(m.roleCount).toBeLessThan(MIN_HEADER_ROLE_MATCHES);
|
||||
});
|
||||
|
||||
it("tolerates empty cells and a ragged row", () => {
|
||||
expect(matchTransactionHeaders([]).roleCount).toBe(0);
|
||||
expect(matchTransactionHeaders(["", "Date", ""]).date).toBe(1);
|
||||
});
|
||||
});
|
||||
176
src/utils/headerDictionary.ts
Normal file
176
src/utils/headerDictionary.ts
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
/**
|
||||
* Header dictionary for the TRANSACTION import flow (#327).
|
||||
*
|
||||
* Detection used to reason on the SHAPE of the data alone — which column parses
|
||||
* as a date, which one carries the most characters, which pair is sparse and
|
||||
* complementary. Shape cannot tell a debit column from a credit column, so a
|
||||
* file laid out `Date;Description;Crédit;Débit` was mapped backwards and every
|
||||
* sign of the import was inverted. This module is the lexical layer that reads
|
||||
* the header labels; `csvAutoDetect.ts` puts it in FRONT of the shape
|
||||
* heuristics and falls back to them whenever the labels are mute (no header
|
||||
* row, unknown labels).
|
||||
*
|
||||
* WHY ITS OWN MODULE. The holdings flow (#245) already matches header labels,
|
||||
* with its own keyword tables living in `csvAutoDetect.ts`. Merging the two
|
||||
* tables is not possible: `montant` is an EXCLUSION token there
|
||||
* (`VALUE_HEADER_KEYWORDS` — a market-value column must never be read as a unit
|
||||
* price) and it is the PRIMARY amount keyword here. Two flows, two tables. Only
|
||||
* the two matching helpers are generic, so they live here and the holdings flow
|
||||
* imports them, unchanged — one direction of dependency, no cycle.
|
||||
*/
|
||||
|
||||
// Keyword tables, matched as SUBSTRINGS against accent-stripped,
|
||||
// alphanumeric-only header cells, so `Date de l'opération` matches `date` and
|
||||
// `Débit ($)` matches `debit`. Bilingual FR + EN, order inside each list is
|
||||
// priority order. Kept deliberately tight: a keyword that fires on the wrong
|
||||
// column is worse than a keyword that never fires, because the shape heuristic
|
||||
// behind it is the behaviour the corpus already froze.
|
||||
export const DATE_HEADER_KEYWORDS = ["date"];
|
||||
export const DESCRIPTION_HEADER_KEYWORDS = [
|
||||
"description",
|
||||
"libelle",
|
||||
"detail",
|
||||
"transaction",
|
||||
];
|
||||
export const AMOUNT_HEADER_KEYWORDS = ["montant", "amount"];
|
||||
export const DEBIT_HEADER_KEYWORDS = [
|
||||
"debit",
|
||||
"retrait",
|
||||
"debourse",
|
||||
"withdrawal",
|
||||
];
|
||||
export const CREDIT_HEADER_KEYWORDS = [
|
||||
"credit",
|
||||
"depot",
|
||||
"encaissement",
|
||||
"deposit",
|
||||
];
|
||||
export const BALANCE_HEADER_KEYWORDS = ["solde", "balance"];
|
||||
|
||||
/**
|
||||
* A running-balance column is never an amount column, and `Date de
|
||||
* transaction` is never the description column. Both are exclusion lists rather
|
||||
* than ordering tricks, because substring matching has no notion of "the best
|
||||
* match" — the first column containing the keyword wins.
|
||||
*/
|
||||
const AMOUNT_EXCLUDE = BALANCE_HEADER_KEYWORDS;
|
||||
const DESCRIPTION_EXCLUDE = DATE_HEADER_KEYWORDS;
|
||||
|
||||
/** Accent-strip + lowercase + keep alphanumerics only (for header matching). */
|
||||
export function normalizeHeaderCell(s: string): string {
|
||||
return (s ?? "")
|
||||
.normalize("NFD")
|
||||
.replace(/[\u0300-\u036f]/g, "")
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]/g, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the first unused column whose normalized header contains one of the
|
||||
* keywords (keywords tried in priority order). `exclude` skips columns whose
|
||||
* header contains any excluded token (e.g. a "value" column for price).
|
||||
*/
|
||||
export function matchHeaderColumn(
|
||||
normalizedHeaders: string[],
|
||||
keywords: string[],
|
||||
used: Set<number>,
|
||||
exclude: string[] = []
|
||||
): number | null {
|
||||
for (const kw of keywords) {
|
||||
for (let i = 0; i < normalizedHeaders.length; i++) {
|
||||
if (used.has(i)) continue;
|
||||
const h = normalizedHeaders[i];
|
||||
if (!h) continue;
|
||||
if (exclude.some((ex) => h.includes(ex))) continue;
|
||||
if (h.includes(kw)) return i;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** What the labels of one header row say, role by role. */
|
||||
export interface LexicalHeaderMap {
|
||||
date: number | null;
|
||||
description: number | null;
|
||||
amount: number | null;
|
||||
debit: number | null;
|
||||
credit: number | null;
|
||||
balance: number | null;
|
||||
/** Number of roles the row actually named — 0 when the labels are mute. */
|
||||
roleCount: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read one header row through the dictionary.
|
||||
*
|
||||
* Roles are resolved in the order balance -> date -> debit -> credit -> amount
|
||||
* -> description, each claiming its column so a later role cannot steal it.
|
||||
* That order is what makes `Montant débit` / `Montant crédit` resolve as a
|
||||
* debit/credit pair instead of the first one being read as THE amount column.
|
||||
*
|
||||
* Every field is `null` when no column names that role: a mute row yields a map
|
||||
* of nulls and `roleCount: 0`, which is the caller's signal to fall back to the
|
||||
* shape heuristics.
|
||||
*/
|
||||
export function matchTransactionHeaders(
|
||||
headerRow: readonly string[]
|
||||
): LexicalHeaderMap {
|
||||
const normalized = headerRow.map(normalizeHeaderCell);
|
||||
const used = new Set<number>();
|
||||
|
||||
const claim = (col: number | null): number | null => {
|
||||
if (col !== null) used.add(col);
|
||||
return col;
|
||||
};
|
||||
|
||||
const balance = claim(
|
||||
matchHeaderColumn(normalized, BALANCE_HEADER_KEYWORDS, used)
|
||||
);
|
||||
const date = claim(matchHeaderColumn(normalized, DATE_HEADER_KEYWORDS, used));
|
||||
const debit = claim(
|
||||
matchHeaderColumn(normalized, DEBIT_HEADER_KEYWORDS, used, AMOUNT_EXCLUDE)
|
||||
);
|
||||
const credit = claim(
|
||||
matchHeaderColumn(normalized, CREDIT_HEADER_KEYWORDS, used, AMOUNT_EXCLUDE)
|
||||
);
|
||||
const amount = claim(
|
||||
matchHeaderColumn(normalized, AMOUNT_HEADER_KEYWORDS, used, AMOUNT_EXCLUDE)
|
||||
);
|
||||
const description = claim(
|
||||
matchHeaderColumn(
|
||||
normalized,
|
||||
DESCRIPTION_HEADER_KEYWORDS,
|
||||
used,
|
||||
DESCRIPTION_EXCLUDE
|
||||
)
|
||||
);
|
||||
|
||||
const roleCount = [date, description, amount, debit, credit, balance].filter(
|
||||
(c) => c !== null
|
||||
).length;
|
||||
|
||||
return { date, description, amount, debit, credit, balance, roleCount };
|
||||
}
|
||||
|
||||
/**
|
||||
* How many named roles a row must carry before the lexical layer is allowed to
|
||||
* call it a header row on its own.
|
||||
*
|
||||
* Two, not one: a description cell like `DEPOT PAIE EMPLOYEUR` contains
|
||||
* `depot`, so a single match proves nothing. Two distinct roles in one row is a
|
||||
* shape a data row does not produce, and the caller keeps the date test in
|
||||
* front of this check anyway.
|
||||
*/
|
||||
export const MIN_HEADER_ROLE_MATCHES = 2;
|
||||
|
||||
/**
|
||||
* Tokens of a direction-indicator column: the third amount format, where the
|
||||
* amount column holds unsigned magnitudes and a neighbouring column says which
|
||||
* way each row goes. `D`/`C` in French exports, `DB`/`CR` in some English ones.
|
||||
*
|
||||
* The format is refused rather than imported (spec decision) — see
|
||||
* `csvAutoDetect.ts`. Reading it correctly is a separate feature; reading it
|
||||
* wrong imports every debit as income.
|
||||
*/
|
||||
export const DEBIT_INDICATOR_TOKENS = ["d", "db"];
|
||||
export const CREDIT_INDICATOR_TOKENS = ["c", "cr"];
|
||||
998
src/utils/importFormat.test.ts
Normal file
998
src/utils/importFormat.test.ts
Normal file
|
|
@ -0,0 +1,998 @@
|
|||
// importFormat — the format codec (#324).
|
||||
//
|
||||
// The chantier's root bug was a format field that never reached the database
|
||||
// and a restore that invented a value for it. These tests hold the two halves
|
||||
// of the repair: the codec carries EVERY field in both directions, and the
|
||||
// wizard restores what was saved instead of re-deriving it.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import type {
|
||||
ImportFormat,
|
||||
ImportFormatRow,
|
||||
ImportFormatRowInput,
|
||||
ParsedRow,
|
||||
} from "../shared/types";
|
||||
import {
|
||||
AMOUNT_MODES,
|
||||
FORMAT_FIELD_PAIRS,
|
||||
ImportFormatError,
|
||||
ROW_ERROR_KEYS,
|
||||
SIGN_CONVENTIONS,
|
||||
clearMappingForMode,
|
||||
detectAmountSeparators,
|
||||
flipSignFormat,
|
||||
formatFromRow,
|
||||
formatToRow,
|
||||
isRowErrorKey,
|
||||
mapRow,
|
||||
pickFormatRow,
|
||||
summarizeParsedRows,
|
||||
} from "./importFormat";
|
||||
import fr from "../i18n/locales/fr.json";
|
||||
import en from "../i18n/locales/en.json";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Samples. Every field deliberately differs from the wizard's default config,
|
||||
// so a dropped field surfaces as a mismatch rather than as an accidental pass.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const SAMPLE_FORMAT: ImportFormat = {
|
||||
delimiter: ",",
|
||||
encoding: "windows-1252",
|
||||
dateFormat: "YYYY-MM-DD",
|
||||
skipLines: 3,
|
||||
hasHeader: false,
|
||||
columnMapping: { date: 2, description: 5, amount: 7 },
|
||||
amountMode: "single",
|
||||
signConvention: "positive_expense",
|
||||
};
|
||||
|
||||
const SAMPLE_ROW: ImportFormatRow = {
|
||||
delimiter: ",",
|
||||
encoding: "windows-1252",
|
||||
date_format: "YYYY-MM-DD",
|
||||
skip_lines: 3,
|
||||
has_header: 0,
|
||||
column_mapping: '{"date":2,"description":5,"amount":7}',
|
||||
amount_mode: "single",
|
||||
sign_convention: "positive_expense",
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Completeness — the property the codec exists to hold
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("codec completeness (#324)", () => {
|
||||
// `FORMAT_FIELD_PAIRS` is typed `Record<keyof ImportFormat, keyof
|
||||
// ImportFormatRow>`, so a field added to `ImportFormat` fails to BUILD until
|
||||
// it is listed. These two assertions close the other half: a field listed but
|
||||
// not wired through a codec body fails the TEST.
|
||||
it("formatToRow emits exactly the persisted fields of the table", () => {
|
||||
expect(Object.keys(formatToRow(SAMPLE_FORMAT)).sort()).toEqual(
|
||||
Object.values(FORMAT_FIELD_PAIRS).sort()
|
||||
);
|
||||
});
|
||||
|
||||
it("formatFromRow emits exactly the domain fields of the table", () => {
|
||||
expect(Object.keys(formatFromRow(SAMPLE_ROW)).sort()).toEqual(
|
||||
Object.keys(FORMAT_FIELD_PAIRS).sort()
|
||||
);
|
||||
});
|
||||
|
||||
it("covers the eight format fields, no more and no fewer", () => {
|
||||
expect(Object.keys(FORMAT_FIELD_PAIRS)).toHaveLength(8);
|
||||
});
|
||||
|
||||
it("maps every domain field to a distinct persisted field", () => {
|
||||
const persisted = Object.values(FORMAT_FIELD_PAIRS);
|
||||
expect(new Set(persisted).size).toBe(persisted.length);
|
||||
});
|
||||
|
||||
it("pickFormatRow emits exactly the persisted fields of the table", () => {
|
||||
expect(Object.keys(pickFormatRow(SAMPLE_ROW)).sort()).toEqual(
|
||||
Object.values(FORMAT_FIELD_PAIRS).sort()
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Projection — the direction the SREF export needs (#331)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("pickFormatRow (#331)", () => {
|
||||
it("carries the eight columns of a row verbatim", () => {
|
||||
expect(pickFormatRow({ ...SAMPLE_ROW, id: 7, name: "Visa" })).toEqual(
|
||||
SAMPLE_ROW
|
||||
);
|
||||
});
|
||||
|
||||
it("normalizes has_header to the 0/1 the column stores", () => {
|
||||
// `ImportSource` declares it a boolean; SQLite has no such type.
|
||||
expect(pickFormatRow({ ...SAMPLE_ROW, has_header: true }).has_header).toBe(1);
|
||||
expect(pickFormatRow({ ...SAMPLE_ROW, has_header: false }).has_header).toBe(0);
|
||||
});
|
||||
|
||||
it("does NOT validate — an unreadable mapping still travels", () => {
|
||||
// `formatFromRow` refuses this row; a backup must still be able to carry a
|
||||
// source an older build left in that state.
|
||||
const row = { ...SAMPLE_ROW, column_mapping: "{}" };
|
||||
expect(() => formatFromRow(row)).toThrow(ImportFormatError);
|
||||
expect(pickFormatRow(row).column_mapping).toBe("{}");
|
||||
});
|
||||
|
||||
it("never names the drift metadata — it is not a format field", () => {
|
||||
const picked = pickFormatRow({
|
||||
...SAMPLE_ROW,
|
||||
header_signature: "date|desc",
|
||||
});
|
||||
expect("header_signature" in picked).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Round trip — configure -> save -> reload
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("round trip (#324)", () => {
|
||||
it("survives domain -> row -> domain field by field", () => {
|
||||
const restored = formatFromRow(formatToRow(SAMPLE_FORMAT));
|
||||
for (const field of Object.keys(FORMAT_FIELD_PAIRS) as Array<
|
||||
keyof ImportFormat
|
||||
>) {
|
||||
expect(restored[field]).toEqual(SAMPLE_FORMAT[field]);
|
||||
}
|
||||
expect(restored).toEqual(SAMPLE_FORMAT);
|
||||
});
|
||||
|
||||
it("survives row -> domain -> row field by field", () => {
|
||||
expect(formatToRow(formatFromRow(SAMPLE_ROW))).toEqual(SAMPLE_ROW);
|
||||
});
|
||||
|
||||
// THE regression: the credit-card case. Before v17 the convention was not a
|
||||
// column, so restoring one wrote "negative_expense" and every expense of the
|
||||
// next import landed as income.
|
||||
it("keeps positive_expense across a save and a reload", () => {
|
||||
const positive: ImportFormat = {
|
||||
...SAMPLE_FORMAT,
|
||||
signConvention: "positive_expense",
|
||||
};
|
||||
expect(formatFromRow(formatToRow(positive)).signConvention).toBe(
|
||||
"positive_expense"
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps negative_expense across a save and a reload", () => {
|
||||
const negative: ImportFormat = {
|
||||
...SAMPLE_FORMAT,
|
||||
signConvention: "negative_expense",
|
||||
};
|
||||
expect(formatFromRow(formatToRow(negative)).signConvention).toBe(
|
||||
"negative_expense"
|
||||
);
|
||||
});
|
||||
|
||||
// The mode used to be re-derived from `mapping.debitAmount !== undefined`. A
|
||||
// debit/credit source whose debit column is not mapped yet came back as
|
||||
// "single" — this is that shape.
|
||||
it("keeps debit_credit even when the mapping carries no debit column", () => {
|
||||
const partial: ImportFormat = {
|
||||
...SAMPLE_FORMAT,
|
||||
amountMode: "debit_credit",
|
||||
columnMapping: { date: 0, description: 1, creditAmount: 3 },
|
||||
};
|
||||
const restored = formatFromRow(formatToRow(partial));
|
||||
expect(restored.amountMode).toBe("debit_credit");
|
||||
expect(restored.columnMapping).toEqual({
|
||||
date: 0,
|
||||
description: 1,
|
||||
creditAmount: 3,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps single even when the mapping still carries debit/credit keys", () => {
|
||||
const stale: ImportFormat = {
|
||||
...SAMPLE_FORMAT,
|
||||
amountMode: "single",
|
||||
columnMapping: { date: 0, description: 1, amount: 2, debitAmount: 3 },
|
||||
};
|
||||
expect(formatFromRow(formatToRow(stale)).amountMode).toBe("single");
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// has_header normalization
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("has_header normalization", () => {
|
||||
it("writes 0/1, never a boolean — SQLite has no boolean type", () => {
|
||||
expect(formatToRow({ ...SAMPLE_FORMAT, hasHeader: true }).has_header).toBe(1);
|
||||
expect(formatToRow({ ...SAMPLE_FORMAT, hasHeader: false }).has_header).toBe(0);
|
||||
});
|
||||
|
||||
it("reads the integer an import_config_templates row carries", () => {
|
||||
expect(formatFromRow({ ...SAMPLE_ROW, has_header: 1 }).hasHeader).toBe(true);
|
||||
expect(formatFromRow({ ...SAMPLE_ROW, has_header: 0 }).hasHeader).toBe(false);
|
||||
});
|
||||
|
||||
it("reads the boolean an import_sources row is declared with", () => {
|
||||
const asBoolean: ImportFormatRowInput = { ...SAMPLE_ROW, has_header: true };
|
||||
expect(formatFromRow(asBoolean).hasHeader).toBe(true);
|
||||
expect(formatFromRow({ ...asBoolean, has_header: false }).hasHeader).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Validation — a value the app cannot map must raise, never fall back
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("formatFromRow validation", () => {
|
||||
it("accepts every mode and convention the app implements", () => {
|
||||
for (const amount_mode of AMOUNT_MODES) {
|
||||
expect(formatFromRow({ ...SAMPLE_ROW, amount_mode }).amountMode).toBe(
|
||||
amount_mode
|
||||
);
|
||||
}
|
||||
for (const sign_convention of SIGN_CONVENTIONS) {
|
||||
expect(
|
||||
formatFromRow({ ...SAMPLE_ROW, sign_convention }).signConvention
|
||||
).toBe(sign_convention);
|
||||
}
|
||||
});
|
||||
|
||||
// The v17 CHECK admits 'absolute_indicator' so the third mode ships without a
|
||||
// migration. Until it is implemented, reading one must raise — a fall-back to
|
||||
// the `single` branch would read the wrong column for every row.
|
||||
it("rejects an amount mode the app cannot map", () => {
|
||||
const row = {
|
||||
...SAMPLE_ROW,
|
||||
amount_mode: "absolute_indicator",
|
||||
} as unknown as ImportFormatRowInput;
|
||||
expect(() => formatFromRow(row)).toThrow(ImportFormatError);
|
||||
try {
|
||||
formatFromRow(row);
|
||||
} catch (e) {
|
||||
expect((e as ImportFormatError).i18nKey).toBe(
|
||||
"import.errors.unsupportedAmountMode"
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects an unknown sign convention rather than defaulting", () => {
|
||||
const row = {
|
||||
...SAMPLE_ROW,
|
||||
sign_convention: "whatever",
|
||||
} as unknown as ImportFormatRowInput;
|
||||
expect(() => formatFromRow(row)).toThrow(ImportFormatError);
|
||||
try {
|
||||
formatFromRow(row);
|
||||
} catch (e) {
|
||||
expect((e as ImportFormatError).i18nKey).toBe(
|
||||
"import.errors.unsupportedSignConvention"
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects a column mapping that is not valid JSON", () => {
|
||||
expect(() =>
|
||||
formatFromRow({ ...SAMPLE_ROW, column_mapping: "{not json" })
|
||||
).toThrow(ImportFormatError);
|
||||
});
|
||||
|
||||
it("rejects a column mapping without usable date/description columns", () => {
|
||||
for (const column_mapping of ["null", "[]", '{"date":"2"}', "{}"]) {
|
||||
expect(() => formatFromRow({ ...SAMPLE_ROW, column_mapping })).toThrow(
|
||||
ImportFormatError
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// clearMappingForMode — the mode owns the mapping
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("clearMappingForMode", () => {
|
||||
const full = {
|
||||
date: 0,
|
||||
description: 1,
|
||||
amount: 2,
|
||||
debitAmount: 3,
|
||||
creditAmount: 4,
|
||||
};
|
||||
|
||||
it("drops the debit/credit columns when switching to single", () => {
|
||||
expect(clearMappingForMode(full, "single")).toEqual({
|
||||
date: 0,
|
||||
description: 1,
|
||||
amount: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it("drops the amount column when switching to debit/credit", () => {
|
||||
expect(clearMappingForMode(full, "debit_credit")).toEqual({
|
||||
date: 0,
|
||||
description: 1,
|
||||
debitAmount: 3,
|
||||
creditAmount: 4,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps date and description untouched in both directions", () => {
|
||||
expect(clearMappingForMode({ date: 4, description: 6 }, "single")).toEqual({
|
||||
date: 4,
|
||||
description: 6,
|
||||
});
|
||||
expect(
|
||||
clearMappingForMode({ date: 4, description: 6 }, "debit_credit")
|
||||
).toEqual({ date: 4, description: 6 });
|
||||
});
|
||||
|
||||
// #325 replaces the `?? 0` fallbacks with an explicit "amount column not
|
||||
// mapped" row error. Materializing the column the <select> merely DISPLAYS
|
||||
// would make that error unreachable, so an absent column stays absent.
|
||||
it("does not invent the column the select displays by default", () => {
|
||||
const cleared = clearMappingForMode({ date: 0, description: 1 }, "single");
|
||||
expect("amount" in cleared).toBe(false);
|
||||
const pair = clearMappingForMode({ date: 0, description: 1 }, "debit_credit");
|
||||
expect("debitAmount" in pair).toBe(false);
|
||||
expect("creditAmount" in pair).toBe(false);
|
||||
});
|
||||
|
||||
it("emits no undefined-valued keys, so the persisted JSON stays clean", () => {
|
||||
const row = formatToRow({
|
||||
...SAMPLE_FORMAT,
|
||||
columnMapping: clearMappingForMode(full, "single"),
|
||||
});
|
||||
expect(row.column_mapping).toBe('{"date":0,"description":1,"amount":2}');
|
||||
});
|
||||
|
||||
it("is idempotent", () => {
|
||||
const once = clearMappingForMode(full, "debit_credit");
|
||||
expect(clearMappingForMode(once, "debit_credit")).toEqual(once);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// mapRow — the row rule, lifted out of the wizard (#325)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const DC_FORMAT: ImportFormat = {
|
||||
delimiter: ";",
|
||||
encoding: "utf-8",
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
skipLines: 0,
|
||||
hasHeader: true,
|
||||
columnMapping: { date: 0, description: 1, debitAmount: 2, creditAmount: 3 },
|
||||
amountMode: "debit_credit",
|
||||
signConvention: "negative_expense",
|
||||
};
|
||||
|
||||
const SINGLE_FORMAT: ImportFormat = {
|
||||
...DC_FORMAT,
|
||||
columnMapping: { date: 0, description: 1, amount: 2 },
|
||||
amountMode: "single",
|
||||
};
|
||||
|
||||
/** The amount of a row that parsed, or the error key of one that did not. */
|
||||
function outcome(row: ReturnType<typeof mapRow>): number | string {
|
||||
return row.parsed ? row.parsed.amount : row.error!;
|
||||
}
|
||||
|
||||
describe("mapRow — debit/credit is a subtraction, not a nullity test (#325)", () => {
|
||||
it("reads a debit as negative and a credit as positive", () => {
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "84,32", ""], DC_FORMAT))
|
||||
).toBe(-84.32);
|
||||
expect(
|
||||
outcome(mapRow(["15/01/2025", "PAIE", "", "1250,00"], DC_FORMAT))
|
||||
).toBe(1250);
|
||||
});
|
||||
|
||||
it("treats a 0,00 cell in the unused column as absent", () => {
|
||||
// The bug: `isNaN(credit)` was false for "0,00", so the credit won and
|
||||
// every debit imported as zero.
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "84,32", "0,00"], DC_FORMAT))
|
||||
).toBe(-84.32);
|
||||
expect(
|
||||
outcome(mapRow(["15/01/2025", "PAIE", "0,00", "1250,00"], DC_FORMAT))
|
||||
).toBe(1250);
|
||||
});
|
||||
|
||||
it("takes both columns as magnitudes, whatever sign they carry", () => {
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "-84,32", ""], DC_FORMAT))
|
||||
).toBe(-84.32);
|
||||
expect(
|
||||
outcome(mapRow(["15/01/2025", "PAIE", "", "-1250,00"], DC_FORMAT))
|
||||
).toBe(1250);
|
||||
});
|
||||
|
||||
it("nets a row that fills both columns", () => {
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "AJUST", "40,00", "100,00"], DC_FORMAT))
|
||||
).toBe(60);
|
||||
});
|
||||
|
||||
it("errors on an unreadable cell even when its sibling parses", () => {
|
||||
// The review finding on #336: testing `isNaN(debit) && isNaN(credit)` only
|
||||
// caught the case where BOTH sides fell. But the unused column carries
|
||||
// `0,00` in exactly the files this rule exists to fix, so an unreadable
|
||||
// debit beside a `0,00` credit computed 0 − 0 = 0 and imported silently —
|
||||
// the very bug, one cell over.
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "n/a", "0,00"], DC_FORMAT))
|
||||
).toBe(ROW_ERROR_KEYS.invalidAmount);
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "84,32 CAD", "0,00"], DC_FORMAT))
|
||||
).toBe(ROW_ERROR_KEYS.invalidAmount);
|
||||
// Symmetric: an unreadable credit beside a readable debit.
|
||||
expect(
|
||||
outcome(mapRow(["15/01/2025", "PAIE", "0,00", "1 250 $ CAD"], DC_FORMAT))
|
||||
).toBe(ROW_ERROR_KEYS.invalidAmount);
|
||||
});
|
||||
|
||||
it("still treats an EMPTY cell as absent, not as unreadable", () => {
|
||||
// The distinction the fix rests on: empty means "this column does not apply
|
||||
// to this row", which is the normal shape of a debit/credit file.
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "84,32", ""], DC_FORMAT))
|
||||
).toBe(-84.32);
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "84,32", " "], DC_FORMAT))
|
||||
).toBe(-84.32);
|
||||
});
|
||||
|
||||
it("errors when NEITHER column is readable, instead of importing 0", () => {
|
||||
expect(outcome(mapRow(["05/01/2025", "X", "", ""], DC_FORMAT))).toBe(
|
||||
ROW_ERROR_KEYS.invalidAmount
|
||||
);
|
||||
expect(outcome(mapRow(["05/01/2025", "X", "n/a", "-"], DC_FORMAT))).toBe(
|
||||
ROW_ERROR_KEYS.invalidAmount
|
||||
);
|
||||
});
|
||||
|
||||
it("ignores the sign convention, which only applies to a single column", () => {
|
||||
const flipped: ImportFormat = {
|
||||
...DC_FORMAT,
|
||||
signConvention: "positive_expense",
|
||||
};
|
||||
expect(
|
||||
outcome(mapRow(["05/01/2025", "EPICERIE", "84,32", ""], flipped))
|
||||
).toBe(-84.32);
|
||||
});
|
||||
});
|
||||
|
||||
describe("mapRow — an unmapped amount column is an error, not column 0 (#325)", () => {
|
||||
// The `?? 0` fallbacks read column 0 — usually the date — for every row.
|
||||
|
||||
it("refuses a single-amount format with no amount column", () => {
|
||||
const { amount: _drop, ...mapping } = SINGLE_FORMAT.columnMapping;
|
||||
const row = mapRow(["05/01/2025", "EPICERIE", "-84,32"], {
|
||||
...SINGLE_FORMAT,
|
||||
columnMapping: mapping,
|
||||
});
|
||||
expect(row.parsed).toBeNull();
|
||||
expect(row.error).toBe(ROW_ERROR_KEYS.amountColumnNotMapped);
|
||||
});
|
||||
|
||||
it("refuses a debit/credit format with neither column mapped", () => {
|
||||
const row = mapRow(["05/01/2025", "EPICERIE", "84,32", ""], {
|
||||
...DC_FORMAT,
|
||||
columnMapping: { date: 0, description: 1 },
|
||||
});
|
||||
expect(row.error).toBe(ROW_ERROR_KEYS.amountColumnNotMapped);
|
||||
});
|
||||
|
||||
it("accepts a debit/credit format with only ONE column mapped", () => {
|
||||
// Half a pair is a legitimate shape (a card statement with debits only).
|
||||
expect(
|
||||
outcome(
|
||||
mapRow(["05/01/2025", "EPICERIE", "84,32"], {
|
||||
...DC_FORMAT,
|
||||
columnMapping: { date: 0, description: 1, debitAmount: 2 },
|
||||
})
|
||||
)
|
||||
).toBe(-84.32);
|
||||
});
|
||||
|
||||
it("reports the format error before any per-row problem", () => {
|
||||
const row = mapRow(["not-a-date", "EPICERIE", "-84,32"], {
|
||||
...SINGLE_FORMAT,
|
||||
columnMapping: { date: 0, description: 1 },
|
||||
});
|
||||
expect(row.error).toBe(ROW_ERROR_KEYS.amountColumnNotMapped);
|
||||
});
|
||||
});
|
||||
|
||||
describe("mapRow — the rest of the contract (#325)", () => {
|
||||
it("keeps the date check ahead of the amount check", () => {
|
||||
expect(outcome(mapRow(["nope", "X", "abc"], SINGLE_FORMAT))).toBe(
|
||||
ROW_ERROR_KEYS.invalidDate
|
||||
);
|
||||
expect(outcome(mapRow(["05/01/2025", "X", "abc"], SINGLE_FORMAT))).toBe(
|
||||
ROW_ERROR_KEYS.invalidAmount
|
||||
);
|
||||
});
|
||||
|
||||
it("applies positive_expense to a single amount column", () => {
|
||||
expect(
|
||||
outcome(
|
||||
mapRow(["05/01/2025", "EPICERIE", "84,32"], {
|
||||
...SINGLE_FORMAT,
|
||||
signConvention: "positive_expense",
|
||||
})
|
||||
)
|
||||
).toBe(-84.32);
|
||||
});
|
||||
|
||||
it("carries rowIndex, raw and sourceFilename through", () => {
|
||||
const raw = ["05/01/2025", "EPICERIE", "-84,32"];
|
||||
const row = mapRow(raw, SINGLE_FORMAT, {
|
||||
rowIndex: 41,
|
||||
sourceFilename: "releve.csv",
|
||||
});
|
||||
expect(row.rowIndex).toBe(41);
|
||||
expect(row.raw).toBe(raw);
|
||||
expect(row.sourceFilename).toBe("releve.csv");
|
||||
expect(row.parsed).toEqual({
|
||||
date: "2025-01-05",
|
||||
description: "EPICERIE",
|
||||
amount: -84.32,
|
||||
});
|
||||
});
|
||||
|
||||
it("defaults rowIndex to 0 and omits sourceFilename when not given", () => {
|
||||
const row = mapRow(["05/01/2025", "X", "-1,00"], SINGLE_FORMAT);
|
||||
expect(row.rowIndex).toBe(0);
|
||||
expect("sourceFilename" in row).toBe(false);
|
||||
});
|
||||
|
||||
it("reports every failure as an i18n key", () => {
|
||||
for (const raw of [
|
||||
["nope", "X", "-1,00"],
|
||||
["05/01/2025", "X", "abc"],
|
||||
]) {
|
||||
const row = mapRow(raw, SINGLE_FORMAT);
|
||||
expect(isRowErrorKey(row.error!)).toBe(true);
|
||||
}
|
||||
expect(isRowErrorKey("boom: sqlite is locked")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("detectAmountSeparators — the column decides (#325)", () => {
|
||||
const rows = [
|
||||
["05/01/2025", "EPICERIE", "1.234", ""],
|
||||
["15/01/2025", "PAIE", "", "84,32"],
|
||||
];
|
||||
|
||||
it("arbitrates each amount column the format declares", () => {
|
||||
const seps = detectAmountSeparators(rows, DC_FORMAT);
|
||||
// Column 2 holds only the ambiguous "1.234" — no verdict of its own.
|
||||
expect(seps.get(2)).toBeUndefined();
|
||||
expect(seps.get(3)).toBe(",");
|
||||
});
|
||||
|
||||
it("turns an ambiguous cell into the column's reading", () => {
|
||||
const french = [
|
||||
["05/01/2025", "A", "1.234"],
|
||||
["15/01/2025", "B", "84,32"],
|
||||
];
|
||||
const seps = detectAmountSeparators(french, SINGLE_FORMAT);
|
||||
expect(seps.get(2)).toBe(",");
|
||||
expect(
|
||||
outcome(mapRow(french[0], SINGLE_FORMAT, { decimalSeparators: seps }))
|
||||
).toBe(1234);
|
||||
// Without the column verdict, the same cell reads as 1.234.
|
||||
expect(outcome(mapRow(french[0], SINGLE_FORMAT))).toBe(1.234);
|
||||
});
|
||||
|
||||
it("looks at no column the format does not use", () => {
|
||||
expect(detectAmountSeparators(rows, SINGLE_FORMAT).has(3)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Static guards on useImportWizard.
|
||||
//
|
||||
// The restore path and the config write point live inside `useCallback`s of a
|
||||
// React hook and the repository has no jsdom (see FilterPanel.test.tsx), so the
|
||||
// only way to assert on them is to read the source. Same technique as the guard
|
||||
// link 1 left on `parseFilesInternal` in csvAutoDetect.test.ts.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const WIZARD_SRC = readFileSync(
|
||||
resolve(import.meta.dirname, "..", "hooks", "useImportWizard.ts"),
|
||||
"utf-8"
|
||||
);
|
||||
|
||||
/** The body of a `const <name> = useCallback(...)` block, up to the next one. */
|
||||
function callbackBody(name: string, nextName: string): string {
|
||||
const start = WIZARD_SRC.indexOf(`const ${name} =`);
|
||||
const end = WIZARD_SRC.indexOf(`const ${nextName} =`, start);
|
||||
expect(start).toBeGreaterThan(-1);
|
||||
expect(end).toBeGreaterThan(start);
|
||||
return WIZARD_SRC.slice(start, end);
|
||||
}
|
||||
|
||||
/**
|
||||
* The block of `selectSource` that rebuilds a stored format. Scoped tightly on
|
||||
* purpose: the fresh-source arm below it legitimately names `encoding` while
|
||||
* auto-detecting one.
|
||||
*/
|
||||
function restoreBranch(): string {
|
||||
const body = callbackBody("selectSource", "loadHeadersWithConfig");
|
||||
const start = body.indexOf("let restored: SourceConfig | null = null;");
|
||||
const end = body.indexOf("if (restored) {", start);
|
||||
expect(start).toBeGreaterThan(-1);
|
||||
expect(end).toBeGreaterThan(start);
|
||||
return body.slice(start, end);
|
||||
}
|
||||
|
||||
describe("useImportWizard — restore reads, never re-derives (#324)", () => {
|
||||
it("routes the restore through the codec", () => {
|
||||
expect(restoreBranch()).toContain("...formatFromRow(existing)");
|
||||
});
|
||||
|
||||
it("no longer re-infers the amount mode from the mapping shape", () => {
|
||||
expect(WIZARD_SRC).not.toContain("mapping.debitAmount !== undefined");
|
||||
});
|
||||
|
||||
// The spread alone is not enough: a literal AFTER it silently wins, which is
|
||||
// how `signConvention: "negative_expense"` overrode the stored value. The
|
||||
// restore must name no format field of its own.
|
||||
it("names no format field of its own in the restore", () => {
|
||||
const branch = restoreBranch();
|
||||
for (const field of Object.keys(FORMAT_FIELD_PAIRS)) {
|
||||
expect(branch, `restore hardcodes ${field}`).not.toContain(`${field}:`);
|
||||
}
|
||||
});
|
||||
|
||||
it("restores the template provenance instead of blanking it", () => {
|
||||
expect(callbackBody("selectSource", "loadHeadersWithConfig")).toContain(
|
||||
"existing?.template_id ?? null"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("useImportWizard — the config write point (#324)", () => {
|
||||
it("writes nothing at the duplicate step", () => {
|
||||
const body = callbackBody("checkDuplicatesInternal", "checkDuplicates");
|
||||
expect(body).not.toContain("createSource(");
|
||||
expect(body).not.toContain("updateSource(");
|
||||
});
|
||||
|
||||
it("writes the format at import time, through the codec", () => {
|
||||
const body = callbackBody("executeImport", "goToStep");
|
||||
expect(body).toContain("formatToRow(config)");
|
||||
expect(body).toContain("createSource(");
|
||||
expect(body).toContain("updateSource(");
|
||||
});
|
||||
|
||||
it("keeps a single write point in the whole hook", () => {
|
||||
expect(WIZARD_SRC.match(/await createSource\(/g)).toHaveLength(1);
|
||||
expect(WIZARD_SRC.match(/await updateSource\(/g)).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// The signed recap and the sign flip (#329)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** A parsed row carrying an amount. */
|
||||
function row(rowIndex: number, amount: number): ParsedRow {
|
||||
return {
|
||||
rowIndex,
|
||||
raw: ["01/01/2025", "LABEL", String(amount)],
|
||||
parsed: { date: "2025-01-01", description: "LABEL", amount },
|
||||
};
|
||||
}
|
||||
|
||||
/** A row that produced no amount at all. */
|
||||
function errorRow(rowIndex: number): ParsedRow {
|
||||
return {
|
||||
rowIndex,
|
||||
raw: ["", "LABEL", "x"],
|
||||
parsed: null,
|
||||
error: ROW_ERROR_KEYS.invalidDate,
|
||||
};
|
||||
}
|
||||
|
||||
describe("summarizeParsedRows — the recap that reads MEANING (#329)", () => {
|
||||
it("counts and totals the two directions separately", () => {
|
||||
expect(
|
||||
summarizeParsedRows([row(0, -84.32), row(1, 1250), row(2, -56.75)])
|
||||
).toEqual({
|
||||
outflowCount: 2,
|
||||
outflowTotal: -141.07,
|
||||
inflowCount: 1,
|
||||
inflowTotal: 1250,
|
||||
errorCount: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps the totals SIGNED, as they will reach the ledger", () => {
|
||||
// Magnitudes would hide the one thing the recap exists to expose: a file
|
||||
// read backwards looks identical once the signs are dropped.
|
||||
const totals = summarizeParsedRows([row(0, -10), row(1, 40)]);
|
||||
expect(totals.outflowTotal).toBeLessThan(0);
|
||||
expect(totals.inflowTotal).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("counts rows with no parsed amount as errors, and nothing else", () => {
|
||||
const totals = summarizeParsedRows([errorRow(0), row(1, -10), errorRow(2)]);
|
||||
expect(totals.errorCount).toBe(2);
|
||||
expect(totals.outflowCount).toBe(1);
|
||||
expect(totals.inflowCount).toBe(0);
|
||||
});
|
||||
|
||||
it("files a zero amount under neither direction", () => {
|
||||
// Deliberate: a 0,00 row is its own anomaly. Filing it silently under
|
||||
// outflows or inflows would be a small lie of the same family as the big
|
||||
// one this recap exists to catch.
|
||||
const totals = summarizeParsedRows([row(0, 0), row(1, -10)]);
|
||||
expect(totals.outflowCount).toBe(1);
|
||||
expect(totals.inflowCount).toBe(0);
|
||||
expect(totals.errorCount).toBe(0);
|
||||
});
|
||||
|
||||
it("returns all zeroes on an empty file", () => {
|
||||
expect(summarizeParsedRows([])).toEqual({
|
||||
outflowCount: 0,
|
||||
outflowTotal: 0,
|
||||
inflowCount: 0,
|
||||
inflowTotal: 0,
|
||||
errorCount: 0,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("flipSignFormat — the correction lands on the FORMAT (#329)", () => {
|
||||
const SINGLE = {
|
||||
...SAMPLE_FORMAT,
|
||||
amountMode: "single" as const,
|
||||
signConvention: "negative_expense" as const,
|
||||
columnMapping: { date: 0, description: 1, amount: 2 },
|
||||
};
|
||||
const DEBIT_CREDIT = {
|
||||
...SAMPLE_FORMAT,
|
||||
amountMode: "debit_credit" as const,
|
||||
columnMapping: { date: 0, description: 1, debitAmount: 2, creditAmount: 3 },
|
||||
};
|
||||
|
||||
it("toggles the convention in single-amount mode", () => {
|
||||
expect(flipSignFormat(SINGLE).signConvention).toBe("positive_expense");
|
||||
expect(
|
||||
flipSignFormat({ ...SINGLE, signConvention: "positive_expense" })
|
||||
.signConvention
|
||||
).toBe("negative_expense");
|
||||
});
|
||||
|
||||
it("is its own inverse in single-amount mode", () => {
|
||||
const once = { ...SINGLE, ...flipSignFormat(SINGLE) };
|
||||
expect({ ...once, ...flipSignFormat(once) }).toEqual(SINGLE);
|
||||
});
|
||||
|
||||
it("leaves the mapping alone in single-amount mode", () => {
|
||||
expect(flipSignFormat(SINGLE).columnMapping).toEqual(SINGLE.columnMapping);
|
||||
});
|
||||
|
||||
it("swaps the two columns in debit/credit mode", () => {
|
||||
// `mapRow` ignores the sign convention there — toggling it would be inert,
|
||||
// which is the review finding this branch answers.
|
||||
expect(flipSignFormat(DEBIT_CREDIT).columnMapping).toEqual({
|
||||
date: 0,
|
||||
description: 1,
|
||||
debitAmount: 3,
|
||||
creditAmount: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it("does not touch the convention in debit/credit mode", () => {
|
||||
expect(flipSignFormat(DEBIT_CREDIT).signConvention).toBe(
|
||||
DEBIT_CREDIT.signConvention
|
||||
);
|
||||
});
|
||||
|
||||
it("moves a half-mapped column to the other role", () => {
|
||||
const half = {
|
||||
...DEBIT_CREDIT,
|
||||
columnMapping: { date: 0, description: 1, debitAmount: 2 },
|
||||
};
|
||||
expect(flipSignFormat(half).columnMapping).toEqual({
|
||||
date: 0,
|
||||
description: 1,
|
||||
creditAmount: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it("never materializes an unmapped column as a key", () => {
|
||||
// `mapRow` distinguishes "not mapped" from "mapped to column 0"; an
|
||||
// explicit `undefined` would make that distinction unreachable.
|
||||
const half = {
|
||||
...DEBIT_CREDIT,
|
||||
columnMapping: { date: 0, description: 1, creditAmount: 3 },
|
||||
};
|
||||
const mapping = flipSignFormat(half).columnMapping;
|
||||
expect(Object.keys(mapping)).not.toContain("creditAmount");
|
||||
expect(mapping.debitAmount).toBe(3);
|
||||
});
|
||||
|
||||
it("flips the sign of every row it is applied to (single)", () => {
|
||||
const raw = ["05/01/2025", "EPICERIE", "84.32"];
|
||||
const before = mapRow(raw, { ...SINGLE, dateFormat: "DD/MM/YYYY" });
|
||||
const after = mapRow(raw, {
|
||||
...SINGLE,
|
||||
dateFormat: "DD/MM/YYYY",
|
||||
...flipSignFormat(SINGLE),
|
||||
});
|
||||
expect(before.parsed!.amount).toBe(84.32);
|
||||
expect(after.parsed!.amount).toBe(-84.32);
|
||||
});
|
||||
|
||||
it("flips the sign of every row it is applied to (debit/credit)", () => {
|
||||
const raw = ["05/01/2025", "EPICERIE", "84.32", ""];
|
||||
const format = { ...DEBIT_CREDIT, dateFormat: "DD/MM/YYYY" };
|
||||
expect(mapRow(raw, format).parsed!.amount).toBe(-84.32);
|
||||
expect(
|
||||
mapRow(raw, { ...format, ...flipSignFormat(format) }).parsed!.amount
|
||||
).toBe(84.32);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Static guards: the preview step is traversed at EVERY import (#329).
|
||||
//
|
||||
// The step lives in a React page and the repository has no jsdom, so the wiring
|
||||
// is asserted on the source — same technique as the guards above.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const PAGE_SRC = readFileSync(
|
||||
resolve(import.meta.dirname, "..", "pages", "ImportPage.tsx"),
|
||||
"utf-8"
|
||||
);
|
||||
|
||||
describe("the preview step is mandatory (#329)", () => {
|
||||
it("has a transition that targets it", () => {
|
||||
// `file-preview` was declared in `ImportWizardStep` from the beginning and
|
||||
// no dispatch ever aimed at it — the optional modal had supplanted it.
|
||||
expect(callbackBody("parseAndPreview", "checkDuplicatesInternal")).toContain(
|
||||
'dispatch({ type: "SET_STEP", payload: "file-preview" })'
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves no callback that parses straight to the duplicate check", () => {
|
||||
// `parseAndCheckDuplicates` jumped from the configuration to the duplicates
|
||||
// ("skips preview step"). Keeping it exported would leave a live bypass.
|
||||
expect(WIZARD_SRC).not.toContain("parseAndCheckDuplicates");
|
||||
expect(WIZARD_SRC).not.toContain("skips preview step");
|
||||
});
|
||||
|
||||
it("renders the step and reaches it from the configuration", () => {
|
||||
expect(PAGE_SRC).toContain('state.step === "file-preview"');
|
||||
expect(PAGE_SRC).toContain("onNext={parseAndPreview}");
|
||||
expect(PAGE_SRC).toContain("onNext={checkDuplicates}");
|
||||
});
|
||||
|
||||
it("walks back through the preview from the duplicate step", () => {
|
||||
expect(PAGE_SRC).toContain('onBack={() => goToStep("file-preview")}');
|
||||
});
|
||||
|
||||
it("shows the recap over the WHOLE file, not the displayed sample", () => {
|
||||
// The table truncates its rows; a recap computed on the truncation would
|
||||
// state a total that is not the file's.
|
||||
expect(PAGE_SRC).toContain("rows={state.parsedPreview}");
|
||||
expect(PAGE_SRC).not.toContain("state.parsedPreview.slice(");
|
||||
});
|
||||
|
||||
it("no longer carries the optional preview modal", () => {
|
||||
expect(PAGE_SRC).not.toContain("FilePreviewModal");
|
||||
expect(
|
||||
existsSync(
|
||||
resolve(
|
||||
import.meta.dirname,
|
||||
"..",
|
||||
"components",
|
||||
"import",
|
||||
"FilePreviewModal.tsx"
|
||||
)
|
||||
)
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the sign flip is wired to the configuration (#329)", () => {
|
||||
const FLIP = () => callbackBody("flipSignConvention", "executeImport");
|
||||
|
||||
it("rewrites the config and re-parses under the new one", () => {
|
||||
const body = FLIP();
|
||||
expect(body).toContain("flipSignFormat(state.sourceConfig)");
|
||||
expect(body).toContain('dispatch({ type: "SET_SOURCE_CONFIG", payload: flipped })');
|
||||
// The flipped format is PASSED to the parse: `state` has not re-rendered
|
||||
// yet, so re-reading it would redisplay the table the user asked to fix.
|
||||
expect(body).toContain("parseFilesInternal(flipped)");
|
||||
});
|
||||
|
||||
it("drops the detection score, which never measured this format", () => {
|
||||
expect(FLIP()).toContain(
|
||||
'dispatch({ type: "SET_DETECTION_SCORE", payload: null })'
|
||||
);
|
||||
});
|
||||
|
||||
it("does not reload headers, which a flip cannot change", () => {
|
||||
// `loadHeadersWithConfig` dispatches an empty row list; racing it against
|
||||
// the re-parse is a coin toss between the corrected table and a blank one.
|
||||
expect(FLIP()).not.toContain("loadHeadersWithConfig");
|
||||
});
|
||||
});
|
||||
|
||||
describe("the confirmation states what decides the amounts (#329)", () => {
|
||||
const CONFIRM_SRC = readFileSync(
|
||||
resolve(
|
||||
import.meta.dirname,
|
||||
"..",
|
||||
"components",
|
||||
"import",
|
||||
"ImportConfirmation.tsx"
|
||||
),
|
||||
"utf-8"
|
||||
);
|
||||
|
||||
it("shows the amount mode and the column mapping", () => {
|
||||
expect(CONFIRM_SRC).toContain("import.config.amountMode");
|
||||
expect(CONFIRM_SRC).toContain("import.config.columnMapping");
|
||||
});
|
||||
|
||||
it("shows the sign convention, in the mode that applies it", () => {
|
||||
expect(CONFIRM_SRC).toContain("import.config.signConvention");
|
||||
expect(CONFIRM_SRC).toContain('config.amountMode === "single" && (');
|
||||
});
|
||||
|
||||
it("names the columns the amount mode actually reads", () => {
|
||||
// Naming a debit column on a single-amount format would describe an import
|
||||
// that is not happening, so the list is chosen by mode.
|
||||
const mapping = CONFIRM_SRC.slice(
|
||||
CONFIRM_SRC.indexOf("const mappedColumns"),
|
||||
CONFIRM_SRC.indexOf("return (")
|
||||
);
|
||||
expect(mapping).toContain('config.amountMode === "debit_credit"');
|
||||
for (const key of [
|
||||
"dateColumn",
|
||||
"descriptionColumn",
|
||||
"amountColumn",
|
||||
"debitColumn",
|
||||
"creditColumn",
|
||||
]) {
|
||||
expect(mapping, key).toContain(`import.config.${key}`);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("every new preview string exists in both languages (#329)", () => {
|
||||
it("carries the recap and the flip button", () => {
|
||||
for (const key of [
|
||||
"outflowCount",
|
||||
"inflowCount",
|
||||
"errorRows",
|
||||
"flipSigns",
|
||||
"flipSignsHint",
|
||||
] as const) {
|
||||
expect(fr.import.preview[key].length, `fr.${key}`).toBeGreaterThan(0);
|
||||
expect(en.import.preview[key].length, `en.${key}`).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
it("interpolates the two counts in both languages", () => {
|
||||
for (const key of ["outflowCount", "inflowCount"] as const) {
|
||||
expect(fr.import.preview[key], `fr.${key}`).toContain("{{count}}");
|
||||
expect(en.import.preview[key], `en.${key}`).toContain("{{count}}");
|
||||
}
|
||||
});
|
||||
|
||||
it("carries the unmapped-column label of the confirmation", () => {
|
||||
expect(fr.import.confirm.columnUnmapped.length).toBeGreaterThan(0);
|
||||
expect(en.import.confirm.columnUnmapped.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
526
src/utils/importFormat.ts
Normal file
526
src/utils/importFormat.ts
Normal file
|
|
@ -0,0 +1,526 @@
|
|||
/**
|
||||
* The import format codec — the SINGLE conversion point between the persisted
|
||||
* shape of a CSV format (`ImportFormatRow`, shared by `import_sources` and
|
||||
* `import_config_templates`) and its domain shape (`ImportFormat`).
|
||||
*
|
||||
* Why a codec and not a shared type: the two shapes cannot be unified. The
|
||||
* persisted rows are snake_case, store the column mapping as JSON, and have no
|
||||
* boolean type; the wizard works in camelCase on a parsed mapping. See the
|
||||
* comment block on `ImportFormatRow` in `shared/types`.
|
||||
*
|
||||
* The property this file exists to hold: a field cannot be added to the format
|
||||
* and then silently skipped on the way to or from the database. That is
|
||||
* enforced on two levels — `FORMAT_FIELD_PAIRS` is typed against
|
||||
* `keyof ImportFormat`, so the build breaks until a new field is listed, and
|
||||
* `importFormat.test.ts` compares the codec's actual output keys against that
|
||||
* table, so listing a field without wiring it fails the test. Losing
|
||||
* `sign_convention` on the way out is what this chantier is repairing (#324).
|
||||
*/
|
||||
|
||||
import type {
|
||||
AmountMode,
|
||||
ColumnMapping,
|
||||
ImportFormat,
|
||||
ImportFormatRow,
|
||||
ImportFormatRowInput,
|
||||
ParsedRow,
|
||||
SignConvention,
|
||||
} from "../shared/types";
|
||||
import {
|
||||
detectDecimalSeparator,
|
||||
parseFrenchAmount,
|
||||
type DecimalSeparator,
|
||||
} from "./amountParser";
|
||||
import { parseDate } from "./dateParser";
|
||||
|
||||
/**
|
||||
* Values the application can actually map. The database `CHECK` on
|
||||
* `import_sources.amount_mode` deliberately admits `absolute_indicator` so the
|
||||
* third mode ships without another migration — but until it is implemented,
|
||||
* reading one must be a visible error, never a silent fall-through to the
|
||||
* `single` branch (which would read the wrong column for every row).
|
||||
*/
|
||||
export const AMOUNT_MODES: readonly AmountMode[] = ["single", "debit_credit"];
|
||||
export const SIGN_CONVENTIONS: readonly SignConvention[] = [
|
||||
"negative_expense",
|
||||
"positive_expense",
|
||||
];
|
||||
|
||||
/**
|
||||
* The field correspondence table, domain key -> persisted key.
|
||||
*
|
||||
* Typed as `Record<keyof ImportFormat, keyof ImportFormatRow>`: adding a field
|
||||
* to `ImportFormat` makes this literal stop satisfying the mapped type and the
|
||||
* build fails until the field is listed here. The test then verifies that both
|
||||
* codec directions actually emit every listed field.
|
||||
*/
|
||||
export const FORMAT_FIELD_PAIRS: Record<keyof ImportFormat, keyof ImportFormatRow> = {
|
||||
delimiter: "delimiter",
|
||||
encoding: "encoding",
|
||||
dateFormat: "date_format",
|
||||
skipLines: "skip_lines",
|
||||
hasHeader: "has_header",
|
||||
columnMapping: "column_mapping",
|
||||
amountMode: "amount_mode",
|
||||
signConvention: "sign_convention",
|
||||
};
|
||||
|
||||
/** i18n keys for the ways a stored format can be unreadable. */
|
||||
export type ImportFormatErrorKey =
|
||||
| "import.errors.unsupportedAmountMode"
|
||||
| "import.errors.unsupportedSignConvention"
|
||||
| "import.errors.invalidColumnMapping";
|
||||
|
||||
/**
|
||||
* Thrown when a persisted format cannot be decoded. It carries an i18n key so
|
||||
* the wizard can surface it verbatim instead of falling back to a default
|
||||
* format — a wrong default is precisely how the original bug wrote reversed
|
||||
* amounts without an error.
|
||||
*/
|
||||
export class ImportFormatError extends Error {
|
||||
readonly i18nKey: ImportFormatErrorKey;
|
||||
|
||||
constructor(i18nKey: ImportFormatErrorKey, detail: string) {
|
||||
super(`${i18nKey}: ${detail}`);
|
||||
this.name = "ImportFormatError";
|
||||
this.i18nKey = i18nKey;
|
||||
}
|
||||
}
|
||||
|
||||
/** Domain format -> persisted row. `has_header` is normalized to 0/1 here. */
|
||||
export function formatToRow(format: ImportFormat): ImportFormatRow {
|
||||
return {
|
||||
delimiter: format.delimiter,
|
||||
encoding: format.encoding,
|
||||
date_format: format.dateFormat,
|
||||
skip_lines: format.skipLines,
|
||||
has_header: format.hasHeader ? 1 : 0,
|
||||
column_mapping: JSON.stringify(format.columnMapping),
|
||||
amount_mode: format.amountMode,
|
||||
sign_convention: format.signConvention,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Persisted row -> domain format.
|
||||
*
|
||||
* Every enumerated field is validated against the application's whitelist and
|
||||
* an unknown value raises rather than falls back, because both fall-backs are
|
||||
* silent data corruption: an unmapped `amount_mode` reads the wrong column, and
|
||||
* anything other than `positive_expense` would mean `negative_expense` and flip
|
||||
* every sign.
|
||||
*/
|
||||
export function formatFromRow(row: ImportFormatRowInput): ImportFormat {
|
||||
if (!AMOUNT_MODES.includes(row.amount_mode)) {
|
||||
throw new ImportFormatError(
|
||||
"import.errors.unsupportedAmountMode",
|
||||
String(row.amount_mode)
|
||||
);
|
||||
}
|
||||
if (!SIGN_CONVENTIONS.includes(row.sign_convention)) {
|
||||
throw new ImportFormatError(
|
||||
"import.errors.unsupportedSignConvention",
|
||||
String(row.sign_convention)
|
||||
);
|
||||
}
|
||||
|
||||
let columnMapping: ColumnMapping;
|
||||
try {
|
||||
columnMapping = JSON.parse(row.column_mapping) as ColumnMapping;
|
||||
} catch {
|
||||
throw new ImportFormatError(
|
||||
"import.errors.invalidColumnMapping",
|
||||
row.column_mapping
|
||||
);
|
||||
}
|
||||
if (
|
||||
columnMapping === null ||
|
||||
typeof columnMapping !== "object" ||
|
||||
typeof columnMapping.date !== "number" ||
|
||||
typeof columnMapping.description !== "number"
|
||||
) {
|
||||
throw new ImportFormatError(
|
||||
"import.errors.invalidColumnMapping",
|
||||
row.column_mapping
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
delimiter: row.delimiter,
|
||||
encoding: row.encoding,
|
||||
dateFormat: row.date_format,
|
||||
skipLines: row.skip_lines,
|
||||
hasHeader: !!row.has_header,
|
||||
columnMapping,
|
||||
amountMode: row.amount_mode,
|
||||
signConvention: row.sign_convention,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Project any persisted row onto EXACTLY the eight format columns.
|
||||
*
|
||||
* This is the direction the SREF export needs, and the reason it cannot simply
|
||||
* call `formatFromRow`: that one decodes AND validates, which is right when a
|
||||
* format is about to read a file but wrong when a whole profile is being
|
||||
* serialized. A single source left unreadable by an older build — the
|
||||
* `column_mapping: '{}'` that the data restore itself used to write — would
|
||||
* abort the entire backup. Projection carries the row out verbatim; the
|
||||
* whitelist runs on the way back IN (#331), where refusing costs nothing.
|
||||
*
|
||||
* Built from `FORMAT_FIELD_PAIRS` like both codec directions, so a field added
|
||||
* to the format is exported without this function being touched.
|
||||
*/
|
||||
export function pickFormatRow(row: object): ImportFormatRow {
|
||||
const source = row as Record<string, unknown>;
|
||||
const picked: Record<string, unknown> = {};
|
||||
for (const column of Object.values(FORMAT_FIELD_PAIRS)) {
|
||||
picked[column] = source[column];
|
||||
}
|
||||
// SQLite has no boolean, but `ImportSource` declares `has_header` as one, so
|
||||
// a row can reach here either way. Normalize exactly as `formatToRow` does:
|
||||
// the exported file always carries the 0/1 the column actually stores.
|
||||
picked[FORMAT_FIELD_PAIRS.hasHeader] = source[FORMAT_FIELD_PAIRS.hasHeader] ? 1 : 0;
|
||||
return picked as unknown as ImportFormatRow;
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop the columns belonging to the mode being left behind, so the amount mode
|
||||
* is the source of truth for the mapping rather than the reverse.
|
||||
*
|
||||
* Before v17 the mode was re-inferred from `mapping.debitAmount !== undefined`,
|
||||
* which made a stale key from an abandoned mode silently decide how amounts
|
||||
* were read. The mode is persisted now, but a mapping still carrying both
|
||||
* shapes would keep the two disagreeing — so switching mode prunes.
|
||||
*
|
||||
* The column the `<select>` merely DISPLAYS by default (`mapping.amount ?? 0`)
|
||||
* is deliberately not materialized: #325 turns an unmapped amount column into
|
||||
* an explicit row error, and writing a 0 here would make that error
|
||||
* unreachable.
|
||||
*/
|
||||
export function clearMappingForMode(
|
||||
mapping: ColumnMapping,
|
||||
mode: AmountMode
|
||||
): ColumnMapping {
|
||||
const base = { date: mapping.date, description: mapping.description };
|
||||
if (mode === "debit_credit") {
|
||||
return {
|
||||
...base,
|
||||
...(mapping.debitAmount !== undefined
|
||||
? { debitAmount: mapping.debitAmount }
|
||||
: {}),
|
||||
...(mapping.creditAmount !== undefined
|
||||
? { creditAmount: mapping.creditAmount }
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
return {
|
||||
...base,
|
||||
...(mapping.amount !== undefined ? { amount: mapping.amount } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Row mapping — one CSV row + one format -> one `ParsedRow` (#325)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* i18n keys for the ways a single row can fail. They used to be raw English
|
||||
* literals rendered straight into the preview table; they are keys now so the
|
||||
* strings live in the locale files like every other displayed text. Resolve
|
||||
* them with `isRowErrorKey` before calling `t()` — an `ImportReport` also
|
||||
* carries raw exception messages, which must never be fed to the translator.
|
||||
*/
|
||||
export const ROW_ERROR_KEYS = {
|
||||
invalidDate: "import.rowErrors.invalidDate",
|
||||
invalidAmount: "import.rowErrors.invalidAmount",
|
||||
amountColumnNotMapped: "import.rowErrors.amountColumnNotMapped",
|
||||
parseError: "import.rowErrors.parseError",
|
||||
} as const;
|
||||
|
||||
export type RowErrorKey = (typeof ROW_ERROR_KEYS)[keyof typeof ROW_ERROR_KEYS];
|
||||
|
||||
const ROW_ERROR_VALUES: readonly string[] = Object.values(ROW_ERROR_KEYS);
|
||||
|
||||
/** True when the string is one of ours and can safely be translated. */
|
||||
export function isRowErrorKey(value: string): value is RowErrorKey {
|
||||
return ROW_ERROR_VALUES.includes(value);
|
||||
}
|
||||
|
||||
export interface MapRowOptions {
|
||||
/** Position of the row in the whole import. Defaults to 0. */
|
||||
rowIndex?: number;
|
||||
/** File the row came from, carried through to the report. */
|
||||
sourceFilename?: string;
|
||||
/**
|
||||
* Decimal separator arbitrated at COLUMN level, keyed by column index — see
|
||||
* `detectAmountSeparators`. A column absent from the map keeps the per-cell
|
||||
* default.
|
||||
*/
|
||||
decimalSeparators?: ReadonlyMap<number, DecimalSeparator>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map one raw CSV row to a `ParsedRow` under a given format. PURE: no React, no
|
||||
* I/O, no throw — every failure comes back as `error` on the row.
|
||||
*
|
||||
* This rule used to live inside a `useCallback` of `useImportWizard`, which is
|
||||
* why the corpus tests had to keep a hand-written mirror of it and why the
|
||||
* detection score (#328) and the signed preview (#329) would each have had to
|
||||
* re-implement it. One implementation, three consumers.
|
||||
*
|
||||
* The two-column rule is `credit - debit` on MAGNITUDES. It used to be
|
||||
* `isNaN(credit) ? -debit : credit`, so a bank writing `0,00` in the unused
|
||||
* column — which is common — made every debit import as 0,00 and pass
|
||||
* validation, because `isNaN(0)` is false. Subtraction needs no special case
|
||||
* for a `0,00` cell: zero is the identity. `Math.abs` implements the documented
|
||||
* convention (both columns hold positive numbers) even when an export negates
|
||||
* its debits.
|
||||
*
|
||||
* A mapped cell that is NOT EMPTY but does not parse is an error, whatever its
|
||||
* sibling holds. Testing `isNaN(debit) && isNaN(credit)` was not enough: the
|
||||
* unused column carries `0,00` in exactly the files this rule exists to fix, so
|
||||
* an unreadable debit beside a `0,00` credit parsed as 0 − 0 = 0 and imported
|
||||
* silently — the very bug, one cell over. An EMPTY cell is different: it means
|
||||
* the column does not apply to this row, and contributes zero.
|
||||
*/
|
||||
export function mapRow(
|
||||
raw: string[],
|
||||
format: ImportFormat,
|
||||
options: MapRowOptions = {}
|
||||
): ParsedRow {
|
||||
const mapping = format.columnMapping;
|
||||
const base = {
|
||||
rowIndex: options.rowIndex ?? 0,
|
||||
raw,
|
||||
...(options.sourceFilename !== undefined
|
||||
? { sourceFilename: options.sourceFilename }
|
||||
: {}),
|
||||
};
|
||||
const fail = (error: RowErrorKey): ParsedRow => ({
|
||||
...base,
|
||||
parsed: null,
|
||||
error,
|
||||
});
|
||||
|
||||
const readAmount = (col: number): number =>
|
||||
parseFrenchAmount(raw[col]?.trim() ?? "", {
|
||||
decimalSeparator: options.decimalSeparators?.get(col),
|
||||
});
|
||||
|
||||
/**
|
||||
* Read one side of a debit/credit pair. `present` separates "the column does
|
||||
* not apply to this row" (empty cell, contributes zero) from "the column says
|
||||
* something we cannot read" (an error) — a distinction `isNaN` alone cannot
|
||||
* make once the other side parses.
|
||||
*/
|
||||
const readSide = (
|
||||
col: number | undefined
|
||||
): { present: boolean; readable: boolean; value: number } => {
|
||||
if (col === undefined) return { present: false, readable: true, value: 0 };
|
||||
const text = raw[col]?.trim() ?? "";
|
||||
if (!text) return { present: false, readable: true, value: 0 };
|
||||
const value = readAmount(col);
|
||||
return { present: true, readable: !isNaN(value), value };
|
||||
};
|
||||
|
||||
// 1. Configuration. An unmapped amount column used to fall back to `?? 0`,
|
||||
// reading column 0 — usually the date — for every row of the file. That is
|
||||
// a format error affecting the whole import, so it is reported before any
|
||||
// per-row problem.
|
||||
const debitMapped = mapping.debitAmount !== undefined;
|
||||
const creditMapped = mapping.creditAmount !== undefined;
|
||||
if (format.amountMode === "debit_credit") {
|
||||
if (!debitMapped && !creditMapped) {
|
||||
return fail(ROW_ERROR_KEYS.amountColumnNotMapped);
|
||||
}
|
||||
} else if (mapping.amount === undefined) {
|
||||
return fail(ROW_ERROR_KEYS.amountColumnNotMapped);
|
||||
}
|
||||
|
||||
// 2. Date. Kept ahead of the amount value, as it has always been.
|
||||
const date = parseDate(raw[mapping.date]?.trim() || "", format.dateFormat);
|
||||
if (!date) return fail(ROW_ERROR_KEYS.invalidDate);
|
||||
|
||||
// 3. Amount.
|
||||
let amount: number;
|
||||
if (format.amountMode === "debit_credit") {
|
||||
const debit = readSide(debitMapped ? mapping.debitAmount : undefined);
|
||||
const credit = readSide(creditMapped ? mapping.creditAmount : undefined);
|
||||
// A cell that holds something we cannot read fails the row even when its
|
||||
// sibling parses — otherwise the `0,00` filler silently answers for it.
|
||||
if (!debit.readable || !credit.readable) {
|
||||
return fail(ROW_ERROR_KEYS.invalidAmount);
|
||||
}
|
||||
// Both empty: the row states no amount at all.
|
||||
if (!debit.present && !credit.present) {
|
||||
return fail(ROW_ERROR_KEYS.invalidAmount);
|
||||
}
|
||||
amount =
|
||||
(credit.present ? Math.abs(credit.value) : 0) -
|
||||
(debit.present ? Math.abs(debit.value) : 0);
|
||||
} else {
|
||||
amount = readAmount(mapping.amount!);
|
||||
if (isNaN(amount)) return fail(ROW_ERROR_KEYS.invalidAmount);
|
||||
if (format.signConvention === "positive_expense") amount = -amount;
|
||||
}
|
||||
|
||||
return {
|
||||
...base,
|
||||
parsed: {
|
||||
date,
|
||||
description: raw[mapping.description]?.trim() || "",
|
||||
amount,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Arbitrate the decimal separator of every amount column the format declares,
|
||||
* over the whole set of data rows.
|
||||
*
|
||||
* `1.234` is 1234 in a French column and 1.234 in an English one, and no rule
|
||||
* applied to that cell ALONE can tell — but a sibling cell reading `84,32`
|
||||
* settles it for the column. Detection deliberately stays out of this (it runs
|
||||
* before a column is known to be an amount column at all); the verdict is
|
||||
* applied where the value actually becomes a transaction.
|
||||
*/
|
||||
export function detectAmountSeparators(
|
||||
rows: readonly string[][],
|
||||
format: ImportFormat
|
||||
): Map<number, DecimalSeparator> {
|
||||
const mapping = format.columnMapping;
|
||||
const columns =
|
||||
format.amountMode === "debit_credit"
|
||||
? [mapping.debitAmount, mapping.creditAmount]
|
||||
: [mapping.amount];
|
||||
|
||||
const result = new Map<number, DecimalSeparator>();
|
||||
for (const col of columns) {
|
||||
if (col === undefined) continue;
|
||||
const verdict = detectDecimalSeparator(
|
||||
rows.map((row) => row[col] ?? "")
|
||||
);
|
||||
if (verdict) result.set(col, verdict);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// The signed recap and the sign flip (#329)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* What a whole parsed file amounts to, before a single row is written.
|
||||
*
|
||||
* This is the chantier's last control, and the only one that looks at MEANING.
|
||||
* A confidence score reports how many rows were read, never what they say: the
|
||||
* `all-positive` fixture scores a perfect 100 % while every credit imports as
|
||||
* an expense (#328), because each of those rows is perfectly readable. A
|
||||
* statement whose recap shows six outflows and zero inflows is wrong on its
|
||||
* face, whatever produced it — a bad detection, a stale template, a bank that
|
||||
* changed its export.
|
||||
*
|
||||
* Totals are SIGNED, exactly as the amounts that would reach the ledger:
|
||||
* `outflowTotal` is negative or zero, `inflowTotal` positive or zero. Showing
|
||||
* magnitudes would hide the one thing the recap exists to expose.
|
||||
*
|
||||
* A row whose amount is exactly zero is neither an outflow nor an inflow, so
|
||||
* the two counts do not have to add up to the row count displayed beside them.
|
||||
* That is deliberate: a zero amount is its own anomaly and filing it silently
|
||||
* under one direction would be a small lie of the same family as the big one.
|
||||
*/
|
||||
export interface PreviewTotals {
|
||||
outflowCount: number;
|
||||
/** Negative or zero. */
|
||||
outflowTotal: number;
|
||||
inflowCount: number;
|
||||
/** Positive or zero. */
|
||||
inflowTotal: number;
|
||||
/** Rows that produced no amount at all — they will not be imported. */
|
||||
errorCount: number;
|
||||
}
|
||||
|
||||
/** Add up a parsed file into the recap the preview step displays. PURE. */
|
||||
export function summarizeParsedRows(
|
||||
rows: readonly ParsedRow[]
|
||||
): PreviewTotals {
|
||||
const totals: PreviewTotals = {
|
||||
outflowCount: 0,
|
||||
outflowTotal: 0,
|
||||
inflowCount: 0,
|
||||
inflowTotal: 0,
|
||||
errorCount: 0,
|
||||
};
|
||||
|
||||
for (const row of rows) {
|
||||
// No parsed amount is exactly the set of rows carrying an `error`; counting
|
||||
// the absence of a number rather than the presence of a message keeps this
|
||||
// count and the amounts it sits next to reading the same rows.
|
||||
if (!row.parsed) {
|
||||
totals.errorCount++;
|
||||
continue;
|
||||
}
|
||||
const amount = row.parsed.amount;
|
||||
if (amount < 0) {
|
||||
totals.outflowCount++;
|
||||
totals.outflowTotal += amount;
|
||||
} else if (amount > 0) {
|
||||
totals.inflowCount++;
|
||||
totals.inflowTotal += amount;
|
||||
}
|
||||
}
|
||||
|
||||
return totals;
|
||||
}
|
||||
|
||||
/**
|
||||
* Flip the direction a file's amounts are read in.
|
||||
*
|
||||
* It acts on the FORMAT, never on the parsed rows: the correction has to be
|
||||
* memorised with the source, so the next import of the same bank reads right on
|
||||
* its own. Flipping the rows alone would repair one import and let the next one
|
||||
* reintroduce the same reversal.
|
||||
*
|
||||
* The operation differs by mode, and getting that wrong makes the button inert:
|
||||
* - `single` — the sign convention is what decides, so it toggles.
|
||||
* - `debit_credit` — `mapRow` computes `credit - debit` on magnitudes and
|
||||
* never reads the sign convention there (#325); the direction is carried by
|
||||
* WHICH column holds which role. Toggling the convention would change
|
||||
* nothing at all, so the two column indices swap instead.
|
||||
*
|
||||
* A half-mapped debit/credit format swaps too: moving the single mapped column
|
||||
* to the other role is precisely the correction a file whose one amount column
|
||||
* was taken for the wrong side needs. Unmapped stays unmapped — `mapRow`
|
||||
* distinguishes "not mapped" from "mapped to column 0", so the key must not
|
||||
* materialize as `undefined`.
|
||||
*
|
||||
* It returns the two fields a flip decides rather than a whole format, so the
|
||||
* caller spreads it onto whatever carries them (`SourceConfig` in the wizard,
|
||||
* a detected configuration in the tests) and keeps its own extra fields. Both
|
||||
* fields always come back, so the pair can never be applied half way.
|
||||
*/
|
||||
export function flipSignFormat(
|
||||
format: Readonly<
|
||||
Pick<ImportFormat, "amountMode" | "signConvention" | "columnMapping">
|
||||
>
|
||||
): Pick<ImportFormat, "signConvention" | "columnMapping"> {
|
||||
if (format.amountMode === "debit_credit") {
|
||||
const { debitAmount, creditAmount, ...rest } = format.columnMapping;
|
||||
return {
|
||||
signConvention: format.signConvention,
|
||||
columnMapping: {
|
||||
...rest,
|
||||
...(creditAmount !== undefined ? { debitAmount: creditAmount } : {}),
|
||||
...(debitAmount !== undefined ? { creditAmount: debitAmount } : {}),
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
columnMapping: format.columnMapping,
|
||||
signConvention:
|
||||
format.signConvention === "negative_expense"
|
||||
? "positive_expense"
|
||||
: "negative_expense",
|
||||
};
|
||||
}
|
||||
Loading…
Reference in a new issue