photoanalyzer/docs/architecture.md

# Architecture

[← Documentation index](index.md)

For whoever has to change this code without breaking somebody's photo library. It
explains what the pieces are, which rules each one keeps, and where to look when a
stage refuses.

The product decisions behind all of it live in `INTEGRATED_PIPELINE_CONCEPT.md`; this
page describes what was built.

## Context

Five things outside the application, and what actually crosses each boundary.

```mermaid
flowchart LR
  operator([Operator]):::person -->|browser, one session| app
  app[Photo Pipeline]:::system -->|read, rename, EXIF write| library[(Photo library<br/>bind mount)]
  app -->|database, cache, journals, backups| data[(Data directory<br/>local filesystem)]
  app -->|confirmed-SFW images only| vision[Vision provider]:::ext
  app -->|immich-go, verified bytes| immich[Immich server]:::ext
  app -->|copy, verify, then remove| archive[(Archive medium)]
  classDef person fill:#1f6feb,stroke:#58a6ff,color:#fff
  classDef system fill:#238636,stroke:#3fb950,color:#fff
  classDef ext fill:#6e40c9,stroke:#a371f7,color:#fff
```

| boundary | leaves the machine? | carries |
|---|---|---|
| operator → app | no (loopback, or a proxy you configured) | commands against stable ids, never paths |
| app → library | no | reads, folder renames, EXIF merges |
| app → data directory | no | SQLite in WAL mode, thumbnails, journals, backups |
| app → vision provider | **yes** | image bytes of confirmed-SFW canonical assets only |
| app → Immich | **yes** | the exact verified bytes of an approved album |
| app → archive medium | no | copies, verified before the source is removed |

## Runtime

Two long-lived processes and a one-shot migration, sharing one database and one
library.

```mermaid
flowchart TD
  browser([Browser]) -->|JSON + SSE| api
  migrate["migrate<br/>backup, then upgrade"] -->|must exit 0| api
  migrate --> worker
  api["api · serve<br/>enqueues, serves, reads"] -->|jobs table| db[(SQLite WAL)]
  worker["worker<br/>claims and does the work"] -->|jobs table| db
  worker --> tools["exiftool · vision API · immich-go"]
  api -.->|api.lock.json| lock{{library lock}}
  worker -.->|worker.lock.json| lock
  api --> lib[(library)]
  worker --> lib
```

`serve` enqueues and renders; **it does not do the work**. Everything that scans,
scores, analyses, renames, uploads, or archives happens in the worker, which claims
a queued job atomically with a fencing token. One mutating job runs at a time, and
the lock file in the data directory is what makes a second worker impossible rather
than merely discouraged.

## The modules

| package | owns | must not |
|---|---|---|
| `api/` | HTTP surface, error envelope, security policy | contain SQL or business logic |
| `api/routes/` | one module per resource group, all under `/api/v1` | accept a filesystem path from the browser |
| `api/security.py` | host/origin/CSRF/session/size policy as one pure `evaluate`, plus the ASGI middleware | be bypassed per-route |
| `schemas/` | Pydantic request and response contracts | reach the database |
| `services/` | all domain logic; the only place a decision is made | be imported by the frozen CLI archive |
| `models/` | SQLAlchemy tables | hold behaviour |
| `jobs/` | durable job lifecycle, worker loop, lock ranks, handler registry | run work in the API process |
| `integrations/` | the real outside world: `exiftool`, vision, NSFW model, `immich-go` and its report grammars | be called without a version recorded |
| `path_policy.py` | the library boundary, `_IGNORE/` exclusion, symlink-escape refusal | be duplicated anywhere |
| `imaging.py` | the single bounded-decode door | let a caller open an image directly |
| `faults.py` | the crash barriers the tests fire | read anything but its one env var |
| `config.py` | typed settings, secrets as `SecretStr` | log or return a value |
| `db.py` | engine, sessions, WAL and foreign keys, migration entry | be used to bypass a repository |

Services worth knowing by name: `inventory`, `duplicates`, `safety`, `analysis`,
`albums`/`proposals`/`naming`, `renames`/`rename_apply`/`rename_journal`,
`uploads`/`upload_batches`/`upload_reports`/`upload_verification`,
`archives`/`archive_transfer`/`archive_journal`/`restores`/`availability`,
`thumbnails`, `hashing`, `exif_checkpoint`, `jobs`, `workflow`, `backup`,
`diagnostics`, `app_lock`, `release`, `benchmarks`, `library`, `legacy_import`.

## The state machines

Three journals decide what a restart is allowed to assume. All three are read from
the database plus the real world — never guessed from a missing file.

### Durable jobs

```mermaid
stateDiagram-v2
  [*] --> queued
  queued --> running: claimed with a fencing token
  queued --> cancelled
  queued --> cancelling
  running --> succeeded
  running --> failed
  running --> cancelling
  cancelling --> cancelled
  failed --> retry_queued
  retry_queued --> running
  succeeded --> [*]
  cancelled --> [*]
```

`succeeded` and `cancelled` are terminal, so a duplicate delivery cannot move a
finished job. Claiming is compare-and-set on the row version; a worker whose lease
expired cannot commit after another has taken over.

### The rename journal

The only state machine that moves somebody's folders.

```mermaid
stateDiagram-v2
  [*] --> planned
  planned --> moving
  moving --> moved
  moving --> planned: proven untouched
  moving --> rollback_required
  moved --> database_updated
  database_updated --> verified
  verified --> complete
  moved --> rollback_required
  database_updated --> rollback_required
  verified --> rollback_required
  planned --> failed
  failed --> planned
  rollback_required --> rolled_back
  complete --> [*]
  rolled_back --> [*]
```

Intent is written **before** the disk is touched, which is why `moving` can resolve
backwards: the evidence decides. `moving`, `moved`, `database_updated`, and
`rollback_required` are the unsafe states — while any operation sits in one, the
library may be half-renamed, so unrelated mutations are refused with
`409 rename_recovery_required` until a person resolves it.

### Upload batches

```mermaid
stateDiagram-v2
  [*] --> planned
  planned --> running
  running --> succeeded
  running --> failed
  running --> cancelling
  running --> unknown_requires_verification: process died after acceptance
  cancelling --> cancelled
  failed --> running: retry
  cancelled --> running: retry
```

`unknown_requires_verification` is deliberately **not** restartable. The server may
already hold the files; the answer is to ask Immich for the recorded SHA-1, not to
upload again and hope.

Archive transfers use the same shape — `planned → transferring → verified →
removing → complete` — and the source is removed only after the archived bytes are
verified.

## Identity and the data model

A path is metadata. The identity is `assets.id`, a UUID that never changes, and
`asset_paths` records every path an asset has ever had with the reason it changed.
That is what makes a rename cheap: nothing else in the database has to move.

Tables: `assets`, `asset_paths`, `thumbnails`, `jobs`, `job_items`, `job_events`,
`safety_reviews`, `analysis_results`, `exif_projections`, `duplicate_clusters`,
`duplicate_members`, `duplicate_negative_links`, `album_proposals`, `rename_plans`,
`rename_operations`, `upload_batches`, `upload_items`, `upload_verifications`,
`archive_locations`, `archive_plans`, `archive_operations`.

Availability is independent of workflow progress: `active`, `archiving`,
`archived_online`, `archived_offline`, `restoring`, `missing_unexpected`. An
unmounted archive disk is `archived_offline`, never `missing` — the scanner is not
allowed to conclude that a photo is gone because a disk is unplugged.

EXIF is a projection with its own verified state (`verified`, `divergent`,
`failed`): the desired values are written, read back, and compared, and anything the
stage does not own having changed makes the asset `divergent` and blocks the next
mutating stage.

## Where each invariant lives

| invariant | enforced in |
|---|---|
| `_IGNORE/` is never traversed, counted, or opened | `path_policy.is_excluded`, used by every discovery path |
| no path outside the library roots is reachable | `path_policy.resolve_in_roots` — it returns the resolved path, because validating one name and opening another is the symlink race |
| one writer per library | `services/app_lock.py` (an `flock` on a JSON lock file, per role) |
| one mutating job at a time | `services/jobs.py` lock keys plus `jobs/locks.py` rank ordering |
| only confirmed-SFW assets reach the vision provider | `services/analysis.py`, re-checked *after* the provider call so a decision that flipped mid-flight discards the result |
| EXIF is verified, and other fields preserved | `services/exif_checkpoint.py` |
| uploads carry the exact verified bytes | `services/uploads.py` preflight, re-proved immediately before the uploader runs |
| an uncertain upload is not a failure | `services/upload_batches.py`, `services/upload_verification.py` |
| the archive source outlives its copy until verified | `services/archive_transfer.py` |
| decoding is bounded | `imaging.py` |
| the trust boundary and CSRF | `api/security.py` |

## Concurrency

Locks are taken broad to narrow — `library → stage/job → album/folder → asset` — and
never the other way, which is what makes deadlock structural rather than lucky.
Ranks live in `jobs/locks.py`.

Exactly-once execution is not achievable across SQLite, a filesystem, subprocesses,
and a remote server. The design promises **at-least-once with idempotent recovery**:
every handler may run twice, and running twice must not produce two side effects.
That is why the rename journal records intent before moving, why EXIF writes merge
and verify, and why upload retries consult both local history and Immich.

## History

This application was extracted from two command-line tools rather than written from
nothing. Their sources are frozen in `legacy_cli_archive/` with the ledger mapping
each donated behaviour to the service that now owns it, the characterization tests
that pinned it, and every intentional difference. Production code must not import
them; they are provenance and rollback evidence.