Integrated, restart-safe photo analysis, duplicate review, metadata, upload, and archive workflow. Planning lives in INTEGRATED_PIPELINE_CONCEPT.md and delivery_backlog/.

Application (`photo_pipeline`)

The target application lives in photo_pipeline/ (FastAPI + SQLAlchemy + Alembic). Run it with:

python -m photo_pipeline migrate   # apply database migrations
python -m photo_pipeline serve     # start the API + static review UI (127.0.0.1:8000)

Configuration comes from PHOTO_PIPELINE_* environment variables (see photo_pipeline/config.py); secrets are referenced, never logged.

Testing

One offline command runs the whole suite (unit, integration, and browser end-to-end); it needs no network and uses only deterministic synthetic fixtures:

work_item/scripts/python -m pytest tests -q

Browser end-to-end tests require a one-time Playwright browser install:

python -m playwright install chromium

Phase A acceptance gate

Phase A (Epic E01: shared identity, inventory, duplicates, thumbnails, review UI) is gated by a reproducible end-to-end suite:

work_item/scripts/python -m pytest tests/e2e tests/integration -q

tests/e2e/test_phase_a_pipeline.py launches the real API process against a fresh database and a deterministic fixture library, then drives scan, _IGNORE/ exclusion, move reconciliation, exact/fuzzy duplicate review, thumbnail orientation, canonical selection, browser reload, and a full process restart — asserting durable API and database state after the restart.
tests/story_traceability.json maps every delivered story to its tests; tests/e2e/test_traceability.py fails if a Phase A story loses coverage or a test file is left unexercised.

Phase B acceptance gate

Phase B (Epic E02: durable jobs, workflow shell, safety/analysis views) is proven through the real process and browser boundaries. One command runs the Phase B API, worker-recovery, and browser (Playwright) suites:

work_item/scripts/python -m pytest tests/e2e -m phase_b -q

tests/e2e/test_phase_b_pipeline.py launches the real server and durable worker as child processes and drives them only over HTTP/SSE: analysis start/progress, resumable SSE reconnect, the polling fallback, cancellation, per-asset error inspection, one-mutating-job rejection during read-only browsing, the NSFW→vision privacy gate, and durability across a full restart.
tests/e2e/test_worker_kill.py kills a worker mid-item and proves a fresh worker resumes the fenced job (the "resume" journey).
tests/e2e/test_analysis_browser.py starts a job from the Analyze view and watches live progress arrive over the browser's real SSE adapter.

The full Phase B regression, including the unchanged Phase A gate, is the whole end-to-end suite:

work_item/scripts/python -m pytest tests/e2e -q

Phase C acceptance gate

Phase C (Epic E03: album evidence, naming policy, versioned proposals, Albums view) is proven through the real process and browser boundaries. One command runs the Phase C API and browser (Playwright) suites with the deterministic naming provider:

work_item/scripts/python -m pytest tests/e2e -m phase_c -q

tests/e2e/test_phase_c_pipeline.py drives a real server over HTTP: evidence aggregation, generation through the deterministic naming fake (asserting the exact provider inputs and that no file path or asset ID ever reaches it), provider failure and retry, invalid names, path-separator sanitization, editing with optimistic versions, stale-evidence approval refusal, valid approval, and durability across a full restart.
tests/e2e/test_albums_ui.py covers the browser journeys: evidence display, editing, prompt validation, collision guidance, approval, stale conflict, and keyboard operation.
Both suites assert that no fixture path changes — Phase C proposes names and never renames.

The deterministic naming provider is enabled only by test configuration (PHOTO_PIPELINE_FAKE_NAMING_LOG); without it the application falls back to the offline naming-policy name. Phase A and B suites remain green in the full run above.

Phase D acceptance gate

Phase D (Epic E04: guarded renaming) is the first phase that changes the library on disk, so its gate is the strictest. One command runs the rename API journeys, the filesystem fault injection, and the browser suite:

work_item/scripts/python -m pytest tests/e2e -m phase_d -q

tests/e2e/test_phase_d_pipeline.py drives a real server over HTTP: plan and export, confirmation with the plan version and checksum (a stale token is refused without touching disk), a valid apply, the case-only rename procedure, a collision whose occupant survives, a source that changed after planning, and durability across a full restart.
Fault injection is real. PHOTO_PIPELINE_FAULT_AFTER=<journal state> kills the server process the instant that state is persisted. The suite crashes it at every journal transition in turn (moving, moved, database_updated, verified), starts a fresh process against the same database and library, and requires recovery to converge from journal and disk evidence alone — with the asset set, the stable IDs, and every content hash unchanged. Ambiguous evidence is never guessed: it stays classified manual and keeps blocking. An unresolved rename is the cancellation boundary — there is no cancel once a run starts, and unrelated mutations (album proposal generation and approval) are refused with 409 rename_recovery_required until it is resolved, while reads stay available.
tests/e2e/test_renames_ui.py covers the browser journeys: preview of every affected path, confirmation carrying the server-issued token, apply with progress and terminal verification, stale confirmation, collision, interruption, recovery, rollback, keyboard confirmation, and the view still matching the journal after a server restart.

The fault barrier is test-only configuration; without PHOTO_PIPELINE_FAULT_AFTER the apply path has no crash points. Phases A–C remain green in the full run above.

Phase E acceptance gate

Phase E (Epic E05: Immich upload) is the one stage the application cannot take back, so its gate runs the fake-uploader suites, the black-box upload API journeys, and the browser suite as a single command:

work_item/scripts/python -m pytest -m phase_e -q

tests/integration/test_upload_*.py drive a real executable standing in for immich-go through the real adapter and subprocess — argument construction, output bounding, report parsing, verification, and killing a running process.
tests/e2e/test_phase_e_pipeline.py drives a real server and a real durable worker over HTTP: credential failure and an unreachable server, preflight blockers and the explicitly approved partial scope, a new album, an exact duplicate, an upgrade, a retryable failure and its successful retry, a lost acceptance response, verification against Immich, an inconclusive answer resolved by an operator with evidence, bytes edited after upload, cancellation and resume, and an interrupted attempt recovered across a restart.
EXIF precedes upload is asserted, not assumed: an album without its verified safety and analysis checkpoints cannot be approved, and the uploader's own argv log proves it was never executed. Each finished upload re-hashes the files in the folder the uploader was handed and requires the persisted SHA-256/SHA-1 to match.
No secret is retained. The API key is a sentinel string; after a full upload and verification it must appear in the uploader's argv and nowhere else — not in the database, the retained report, or any response the browser can read.
tests/e2e/test_uploads_ui.py covers the browser journeys (preflight preview, confirmation, progress, stopping a run, verification, manual resolution, stale bytes, and recovery after a restart).

Phases A–D remain green in the full run above.

Phase F acceptance gate

Phase F (Epic E06: archive lifecycle) is the only stage that removes originals from the library, and the only one whose storage can walk away in someone's bag. One command runs the archive fault-injection suites, the black-box archive and restore API journeys, and the browser suite:

work_item/scripts/python -m pytest -m phase_f -q

tests/integration/test_archive_*.py and tests/integration/test_restore.py drive real files on real filesystems: preflight against a mounted, missing, swapped, read-only, or full medium; copy-verify-remove and the same-filesystem move path; and a crash at every persisted journal transition in both transfer modes, asserting that no source is ever removed without a durable, byte-identical archive copy.
tests/e2e/test_phase_f_pipeline.py drives a real server and a real durable worker over HTTP: preflight blockers (offline medium, wrong volume, insufficient capacity, bytes changed after upload), a verified archive whose manifest, hashes, and path history are checked on the medium itself, a worker killed at each of transferring, verified, removing, source_removed, and complete, the evidence-based recovery that follows, offline deduplication of an exact and a fuzzy copy while the medium is away, mount return, restore, and a collision that restores beside its occupant.
Archived is not missing. An unmounted medium leaves its photos archived_offline — still hashed, still in the duplicate indexes, still previewable through their protected thumbnails — and a rescan neither prunes nor flags them.
Ambiguity is never guessed. A journal state the medium contradicts stays manual, offers no automatic action, and keeps blocking further archiving until a human decides.
tests/e2e/test_archive_ui.py covers the browser journeys (preview with destination identity and reclaimable bytes, blockers and mount instructions, progress split into transfer/verification/removal, interruption and recovery, offline browsing, restore, collision, keyboard confirmation, and reload).

Phases A–E remain green in the full run above.

Legacy CLI archive

The command-line tools this application was extracted from are frozen in legacy_cli_archive/ (US07-01): the original sources, their docs, the dependency lock they were last verified against, schema notes, a redacted sample configuration, the donor ledger, and a checksum for every file.

cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256   # verify the archive
work_item/scripts/python -m pytest tests/unit/test_legacy_archive.py -q   # lint it

They are reference material and rollback evidence only. No module under photo_pipeline/ imports or executes them, the archive is not on the application's import path, and tests/unit/test_legacy_archive.py enforces that along with the checksums and the redaction. Only the two suites that compare against the donors — tests/characterization/ and tests/integration/test_safety_parity.py — put the archived sources on sys.path.

The last path-keyed state they owned, nsfw_scores.csv, is imported once and then left alone:

work_item/scripts/python -m photo_pipeline import-legacy-scores /path/to/nsfw_scores.csv --dry-run

The import writes scored-but-unreviewed safety_reviews rows onto stable asset ids, never invents an asset for an unknown path, never overwrites a human decision, and writes a reconciliation report to the data directory saying exactly what it did. legacy_cli_archive/donor_ledger.yaml records every migrated behavior with its target, the tests that pin the donor, the tests that prove the replacement, and each intentional delta; rows still marked pending name the story that will resolve them.

README.md Unescape Escape

Photo Analyzer

Application (photo_pipeline)