photoanalyzer/README.md

# Photo Analyzer

Integrated, restart-safe photo analysis, duplicate review, metadata, upload, and
archive workflow. Planning lives in `INTEGRATED_PIPELINE_CONCEPT.md` and
`delivery_backlog/`.

## Application (`photo_pipeline`)

The target application lives in `photo_pipeline/` (FastAPI + SQLAlchemy + Alembic).
Run it with:

```bash
python -m photo_pipeline migrate   # apply database migrations
python -m photo_pipeline serve     # start the API + static review UI (127.0.0.1:8000)
```

Configuration comes from `PHOTO_PIPELINE_*` environment variables (see
`photo_pipeline/config.py`); secrets are referenced, never logged.

### API access (US07-02)

The app listens on loopback, so its attacker is another page in the same browser.
Every `/api/v1` route except `health/live`, `health/ready`, and `session` requires
the application session, and every mutation requires its CSRF token as well:

```bash
BASE=http://127.0.0.1:8000
TOKEN=$(curl -sc /tmp/pp.jar $BASE/api/v1/session | python -c 'import json,sys; print(json.load(sys.stdin)["csrf_token"])')
curl -sb /tmp/pp.jar -H "X-CSRF-Token: $TOKEN" -X POST $BASE/api/v1/albums/proposals -d '{}' -H 'Content-Type: application/json'
```

The session is per server process — restarting `serve` invalidates it, and the
browser client re-bootstraps by itself. Requests are also refused when the `Host` is
not a loopback name (DNS rebinding), when `Origin` is any other origin, when
`Sec-Fetch-Site` says the request came from another site (an `<img>` pointed at a
thumbnail), or when the body exceeds `PHOTO_PIPELINE_MAX_REQUEST_BYTES`. There is no
CORS middleware at all, so no other origin can read a response.

## Testing

One offline command runs the whole suite (unit, integration, and browser
end-to-end); it needs no network and uses only deterministic synthetic fixtures:

```bash
work_item/scripts/python -m pytest tests -q
```

Browser end-to-end tests require a one-time Playwright browser install:

```bash
python -m playwright install chromium
```

### Phase A acceptance gate

Phase A (Epic E01: shared identity, inventory, duplicates, thumbnails, review UI) is
gated by a reproducible end-to-end suite:

```bash
work_item/scripts/python -m pytest tests/e2e tests/integration -q
```

- `tests/e2e/test_phase_a_pipeline.py` launches the real API process against a fresh
  database and a deterministic fixture library, then drives scan, `_IGNORE/`
  exclusion, move reconciliation, exact/fuzzy duplicate review, thumbnail
  orientation, canonical selection, browser reload, and a full process restart —
  asserting durable API and database state after the restart.
- `tests/story_traceability.json` maps every delivered story to its tests;
  `tests/e2e/test_traceability.py` fails if a Phase A story loses coverage or a test
  file is left unexercised.

### Phase B acceptance gate

Phase B (Epic E02: durable jobs, workflow shell, safety/analysis views) is proven
through the real process and browser boundaries. One command runs the Phase B API,
worker-recovery, and browser (Playwright) suites:

```bash
work_item/scripts/python -m pytest tests/e2e -m phase_b -q
```

- `tests/e2e/test_phase_b_pipeline.py` launches the real server and durable worker as
  child processes and drives them only over HTTP/SSE: analysis start/progress,
  resumable SSE reconnect, the polling fallback, cancellation, per-asset error
  inspection, one-mutating-job rejection during read-only browsing, the NSFW→vision
  privacy gate, and durability across a full restart.
- `tests/e2e/test_worker_kill.py` kills a worker mid-item and proves a fresh worker
  resumes the fenced job (the "resume" journey).
- `tests/e2e/test_analysis_browser.py` starts a job from the Analyze view and watches
  live progress arrive over the browser's real SSE adapter.

The full Phase B regression, including the unchanged Phase A gate, is the whole
end-to-end suite:

```bash
work_item/scripts/python -m pytest tests/e2e -q
```

### Phase C acceptance gate

Phase C (Epic E03: album evidence, naming policy, versioned proposals, Albums view)
is proven through the real process and browser boundaries. One command runs the
Phase C API and browser (Playwright) suites with the deterministic naming provider:

```bash
work_item/scripts/python -m pytest tests/e2e -m phase_c -q
```

- `tests/e2e/test_phase_c_pipeline.py` drives a real server over HTTP: evidence
  aggregation, generation through the deterministic naming fake (asserting the exact
  provider inputs and that no file path or asset ID ever reaches it), provider
  failure and retry, invalid names, path-separator sanitization, editing with
  optimistic versions, stale-evidence approval refusal, valid approval, and
  durability across a full restart.
- `tests/e2e/test_albums_ui.py` covers the browser journeys: evidence display,
  editing, prompt validation, collision guidance, approval, stale conflict, and
  keyboard operation.
- Both suites assert that **no fixture path changes** — Phase C proposes names and
  never renames.

The deterministic naming provider is enabled only by test configuration
(`PHOTO_PIPELINE_FAKE_NAMING_LOG`); without it the application falls back to the
offline naming-policy name. Phase A and B suites remain green in the full run above.

### Phase D acceptance gate

Phase D (Epic E04: guarded renaming) is the first phase that changes the library on
disk, so its gate is the strictest. One command runs the rename API journeys, the
filesystem fault injection, and the browser suite:

```bash
work_item/scripts/python -m pytest tests/e2e -m phase_d -q
```

- `tests/e2e/test_phase_d_pipeline.py` drives a real server over HTTP: plan and
  export, confirmation with the plan version and checksum (a stale token is refused
  without touching disk), a valid apply, the case-only rename procedure, a collision
  whose occupant survives, a source that changed after planning, and durability
  across a full restart.
- **Fault injection is real.** `PHOTO_PIPELINE_FAULT_AFTER=<journal state>` kills the
  server process the instant that state is persisted. The suite crashes it at every
  journal transition in turn (`moving`, `moved`, `database_updated`, `verified`),
  starts a fresh process against the same database and library, and requires recovery
  to converge from journal and disk evidence alone — with the asset set, the stable
  IDs, and every content hash unchanged. Ambiguous evidence is never guessed: it stays
  classified `manual` and keeps blocking. An unresolved rename is the cancellation
  boundary — there is no cancel once a run starts, and unrelated mutations (album
  proposal generation and approval) are refused with 409 `rename_recovery_required`
  until it is resolved, while reads stay available.
- `tests/e2e/test_renames_ui.py` covers the browser journeys: preview of every
  affected path, confirmation carrying the server-issued token, apply with progress
  and terminal verification, stale confirmation, collision, interruption, recovery,
  rollback, keyboard confirmation, and the view still matching the journal after a
  server restart.

The fault barrier is test-only configuration; without `PHOTO_PIPELINE_FAULT_AFTER`
the apply path has no crash points. Phases A–C remain green in the full run above.

### Phase E acceptance gate

Phase E (Epic E05: Immich upload) is the one stage the application cannot take back,
so its gate runs the fake-uploader suites, the black-box upload API journeys, and the
browser suite as a single command:

```bash
work_item/scripts/python -m pytest -m phase_e -q
```

- `tests/integration/test_upload_*.py` drive a **real executable** standing in for
  `immich-go` through the real adapter and `subprocess` — argument construction,
  output bounding, report parsing, verification, and killing a running process.
- `tests/e2e/test_phase_e_pipeline.py` drives a real server and a real durable worker
  over HTTP: credential failure and an unreachable server, preflight blockers and the
  explicitly approved partial scope, a new album, an exact duplicate, an upgrade, a
  retryable failure and its successful retry, a lost acceptance response, verification
  against Immich, an inconclusive answer resolved by an operator with evidence, bytes
  edited after upload, cancellation and resume, and an interrupted attempt recovered
  across a restart.
- **EXIF precedes upload** is asserted, not assumed: an album without its verified
  safety and analysis checkpoints cannot be approved, and the uploader's own argv log
  proves it was never executed. Each finished upload re-hashes the files in the folder
  the uploader was handed and requires the persisted SHA-256/SHA-1 to match.
- **No secret is retained.** The API key is a sentinel string; after a full upload and
  verification it must appear in the uploader's argv and nowhere else — not in the
  database, the retained report, or any response the browser can read.
- `tests/e2e/test_uploads_ui.py` covers the browser journeys (preflight preview,
  confirmation, progress, stopping a run, verification, manual resolution, stale
  bytes, and recovery after a restart).

Phases A–D remain green in the full run above.

### Phase F acceptance gate

Phase F (Epic E06: archive lifecycle) is the only stage that *removes* originals
from the library, and the only one whose storage can walk away in someone's bag.
One command runs the archive fault-injection suites, the black-box archive and
restore API journeys, and the browser suite:

```bash
work_item/scripts/python -m pytest -m phase_f -q
```

- `tests/integration/test_archive_*.py` and `tests/integration/test_restore.py`
  drive real files on real filesystems: preflight against a mounted, missing,
  swapped, read-only, or full medium; copy-verify-remove and the same-filesystem
  move path; and a crash at **every** persisted journal transition in both transfer
  modes, asserting that no source is ever removed without a durable, byte-identical
  archive copy.
- `tests/e2e/test_phase_f_pipeline.py` drives a real server and a real durable
  worker over HTTP: preflight blockers (offline medium, wrong volume, insufficient
  capacity, bytes changed after upload), a verified archive whose manifest, hashes,
  and path history are checked on the medium itself, a worker killed at each of
  `transferring`, `verified`, `removing`, `source_removed`, and `complete`, the
  evidence-based recovery that follows, offline deduplication of an exact and a
  fuzzy copy while the medium is away, mount return, restore, and a collision that
  restores beside its occupant.
- **Archived is not missing.** An unmounted medium leaves its photos
  `archived_offline` — still hashed, still in the duplicate indexes, still
  previewable through their protected thumbnails — and a rescan neither prunes nor
  flags them.
- **Ambiguity is never guessed.** A journal state the medium contradicts stays
  `manual`, offers no automatic action, and keeps blocking further archiving until
  a human decides.
- `tests/e2e/test_archive_ui.py` covers the browser journeys (preview with
  destination identity and reclaimable bytes, blockers and mount instructions,
  progress split into transfer/verification/removal, interruption and recovery,
  offline browsing, restore, collision, keyboard confirmation, and reload).

Phases A–E remain green in the full run above.

## Media and metadata hardening (US07-03)

Every pixel the application reads goes through `photo_pipeline/imaging.py`: the
declared dimensions are checked before anything is decoded, Pillow's
decompression-bomb warning is treated as a refusal, JPEG decodes near the requested
size, and each decoder failure becomes one of two typed errors. A damaged file is a
per-item error with a persisted code, never a failed scan or a dead worker.

Every metadata stage ends with an EXIF checkpoint (`services/exif_checkpoint.py`):
snapshot, write the owned keywords, read back, prove the owned fields landed and that
nothing else moved, refresh the file hash. A field the stage does not own that
changed anyway makes the checkpoint `divergent` — recorded in `exif_projections`,
shown in the review queue, never repaired behind the user's back, and not counted as
verified, so upload stays blocked.

The golden corpus that proves all of it is generated, not committed:
`tests/fixtures/media_corpus.py` declares every format, orientation, profile,
damage, and metadata case with its expected outcome, and the suite regenerates it
twice to prove it does not drift.

```bash
work_item/scripts/python -m pytest tests/integration/test_media_hardening.py tests/integration/test_exif_checkpoints.py -q
```

## Legacy CLI archive

The command-line tools this application was extracted from are frozen in
`legacy_cli_archive/` (US07-01): the original sources, their docs, the dependency
lock they were last verified against, schema notes, a redacted sample
configuration, the donor ledger, and a checksum for every file.

```bash
cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256   # verify the archive
work_item/scripts/python -m pytest tests/unit/test_legacy_archive.py -q   # lint it
```

They are reference material and rollback evidence only. No module under
`photo_pipeline/` imports or executes them, the archive is not on the application's
import path, and `tests/unit/test_legacy_archive.py` enforces that along with the
checksums and the redaction. Only the two suites that compare *against* the donors —
`tests/characterization/` and `tests/integration/test_safety_parity.py` — put the
archived sources on `sys.path`.

The last path-keyed state they owned, `nsfw_scores.csv`, is imported once and then
left alone:

```bash
work_item/scripts/python -m photo_pipeline import-legacy-scores /path/to/nsfw_scores.csv --dry-run
```

The import writes scored-but-unreviewed `safety_reviews` rows onto stable asset ids,
never invents an asset for an unknown path, never overwrites a human decision, and
writes a reconciliation report to the data directory saying exactly what it did.
`legacy_cli_archive/donor_ledger.yaml` records every migrated behavior with its
target, the tests that pin the donor, the tests that prove the replacement, and each
intentional delta; rows still marked `pending` name the story that will resolve them.