photoanalyzer/docs/architecture.md

Architecture

← Documentation index

For whoever has to change this code without breaking somebody's photo library. It explains what the pieces are, which rules each one keeps, and where to look when a stage refuses.

The product decisions behind all of it live in INTEGRATED_PIPELINE_CONCEPT.md; this page describes what was built.

Context

Five things outside the application, and what actually crosses each boundary.

flowchart LR
  operator([Operator]):::person -->|browser, one session| app
  app[Photo Pipeline]:::system -->|read, rename, EXIF write| library[(Photo library<br/>bind mount)]
  app -->|database, cache, journals, backups| data[(Data directory<br/>local filesystem)]
  app -->|confirmed-SFW images only| vision[Vision provider]:::ext
  app -->|immich-go, verified bytes| immich[Immich server]:::ext
  app -->|copy, verify, then remove| archive[(Archive medium)]
  classDef person fill:#1f6feb,stroke:#58a6ff,color:#fff
  classDef system fill:#238636,stroke:#3fb950,color:#fff
  classDef ext fill:#6e40c9,stroke:#a371f7,color:#fff

boundary	leaves the machine?	carries
operator → app	no (loopback, or a proxy you configured)	commands against stable ids, never paths
app → library	no	reads, folder renames, EXIF merges
app → data directory	no	SQLite in WAL mode, thumbnails, journals, backups
app → vision provider	yes	image bytes of confirmed-SFW canonical assets only
app → Immich	yes	the exact verified bytes of an approved album
app → archive medium	no	copies, verified before the source is removed

Runtime

Two long-lived processes and a one-shot migration, sharing one database and one library.

flowchart TD
  browser([Browser]) -->|JSON + SSE| api
  migrate["migrate<br/>backup, then upgrade"] -->|must exit 0| api
  migrate --> worker
  api["api · serve<br/>enqueues, serves, reads"] -->|jobs table| db[(SQLite WAL)]
  worker["worker<br/>claims and does the work"] -->|jobs table| db
  worker --> tools["exiftool · vision API · immich-go"]
  api -.->|api.lock.json| lock{{library lock}}
  worker -.->|worker.lock.json| lock
  api --> lib[(library)]
  worker --> lib

serve enqueues and renders; it does not do the work. Everything that scans, scores, analyses, renames, uploads, or archives happens in the worker, which claims a queued job atomically with a fencing token. One mutating job runs at a time, and the lock file in the data directory is what makes a second worker impossible rather than merely discouraged.

The modules

package	owns	must not
api/	HTTP surface, error envelope, security policy	contain SQL or business logic
api/routes/	one module per resource group, all under /api/v1	accept a filesystem path from the browser
api/security.py	host/origin/CSRF/session/size policy as one pure evaluate, plus the ASGI middleware	be bypassed per-route
schemas/	Pydantic request and response contracts	reach the database
services/	all domain logic; the only place a decision is made	be imported by the frozen CLI archive
models/	SQLAlchemy tables	hold behaviour
jobs/	durable job lifecycle, worker loop, lock ranks, handler registry	run work in the API process
integrations/	the real outside world: exiftool, vision, NSFW model, immich-go and its report grammars	be called without a version recorded
path_policy.py	the library boundary, _IGNORE/ exclusion, symlink-escape refusal	be duplicated anywhere
imaging.py	the single bounded-decode door	let a caller open an image directly
faults.py	the crash barriers the tests fire	read anything but its one env var
config.py	typed settings, secrets as SecretStr	log or return a value
db.py	engine, sessions, WAL and foreign keys, migration entry	be used to bypass a repository

Services worth knowing by name: inventory, duplicates, safety, analysis, albums/proposals/naming, renames/rename_apply/rename_journal, uploads/upload_batches/upload_reports/upload_verification, archives/archive_transfer/archive_journal/restores/availability, thumbnails, hashing, exif_checkpoint, jobs, workflow, backup, diagnostics, app_lock, release, benchmarks, library, legacy_import.

The state machines

Three journals decide what a restart is allowed to assume. All three are read from the database plus the real world — never guessed from a missing file.

Durable jobs

stateDiagram-v2
  [*] --> queued
  queued --> running: claimed with a fencing token
  queued --> cancelled
  queued --> cancelling
  running --> succeeded
  running --> failed
  running --> cancelling
  cancelling --> cancelled
  failed --> retry_queued
  retry_queued --> running
  succeeded --> [*]
  cancelled --> [*]

succeeded and cancelled are terminal, so a duplicate delivery cannot move a finished job. Claiming is compare-and-set on the row version; a worker whose lease expired cannot commit after another has taken over.

The rename journal

The only state machine that moves somebody's folders.

stateDiagram-v2
  [*] --> planned
  planned --> moving
  moving --> moved
  moving --> planned: proven untouched
  moving --> rollback_required
  moved --> database_updated
  database_updated --> verified
  verified --> complete
  moved --> rollback_required
  database_updated --> rollback_required
  verified --> rollback_required
  planned --> failed
  failed --> planned
  rollback_required --> rolled_back
  complete --> [*]
  rolled_back --> [*]

Intent is written before the disk is touched, which is why moving can resolve backwards: the evidence decides. moving, moved, database_updated, and rollback_required are the unsafe states — while any operation sits in one, the library may be half-renamed, so unrelated mutations are refused with 409 rename_recovery_required until a person resolves it.

Upload batches

stateDiagram-v2
  [*] --> planned
  planned --> running
  running --> succeeded
  running --> failed
  running --> cancelling
  running --> unknown_requires_verification: process died after acceptance
  cancelling --> cancelled
  failed --> running: retry
  cancelled --> running: retry

unknown_requires_verification is deliberately not restartable. The server may already hold the files; the answer is to ask Immich for the recorded SHA-1, not to upload again and hope.

Archive transfers use the same shape — planned → transferring → verified → removing → complete — and the source is removed only after the archived bytes are verified.

Identity and the data model

A path is metadata. The identity is assets.id, a UUID that never changes, and asset_paths records every path an asset has ever had with the reason it changed. That is what makes a rename cheap: nothing else in the database has to move.

Tables: assets, asset_paths, thumbnails, jobs, job_items, job_events, safety_reviews, analysis_results, exif_projections, duplicate_clusters, duplicate_members, duplicate_negative_links, album_proposals, rename_plans, rename_operations, upload_batches, upload_items, upload_verifications, archive_locations, archive_plans, archive_operations.

Availability is independent of workflow progress: active, archiving, archived_online, archived_offline, restoring, missing_unexpected. An unmounted archive disk is archived_offline, never missing — the scanner is not allowed to conclude that a photo is gone because a disk is unplugged.

EXIF is a projection with its own verified state (verified, divergent, failed): the desired values are written, read back, and compared, and anything the stage does not own having changed makes the asset divergent and blocks the next mutating stage.

Where each invariant lives

invariant	enforced in
_IGNORE/ is never traversed, counted, or opened	path_policy.is_excluded, used by every discovery path
no path outside the library roots is reachable	path_policy.resolve_in_roots — it returns the resolved path, because validating one name and opening another is the symlink race
one writer per library	services/app_lock.py (an flock on a JSON lock file, per role)
one mutating job at a time	services/jobs.py lock keys plus jobs/locks.py rank ordering
only confirmed-SFW assets reach the vision provider	services/analysis.py, re-checked after the provider call so a decision that flipped mid-flight discards the result
EXIF is verified, and other fields preserved	services/exif_checkpoint.py
uploads carry the exact verified bytes	services/uploads.py preflight, re-proved immediately before the uploader runs
an uncertain upload is not a failure	services/upload_batches.py, services/upload_verification.py
the archive source outlives its copy until verified	services/archive_transfer.py
decoding is bounded	imaging.py
the trust boundary and CSRF	api/security.py

Concurrency

Locks are taken broad to narrow — library → stage/job → album/folder → asset — and never the other way, which is what makes deadlock structural rather than lucky. Ranks live in jobs/locks.py.

Exactly-once execution is not achievable across SQLite, a filesystem, subprocesses, and a remote server. The design promises at-least-once with idempotent recovery: every handler may run twice, and running twice must not produce two side effects. That is why the rename journal records intent before moving, why EXIF writes merge and verify, and why upload retries consult both local history and Immich.

History

This application was extracted from two command-line tools rather than written from nothing. Their sources are frozen in legacy_cli_archive/ with the ledger mapping each donated behaviour to the service that now owns it, the characterization tests that pinned it, and every intentional difference. Production code must not import them; they are provenance and rollback evidence.