Compare commits

...

5 Commits

Author SHA1 Message Date
e3ebd0b467 US09-03: Write the Architecture Overview
Some checks failed
Test / suites (pull_request) Failing after 2m58s
Test / container (pull_request) Has been skipped
2026-08-23 22:34:42 +02:00
0d4f9d1c28 US09-02: Write the Installation and Operations Manual (#108)
Some checks failed
Test / suites (push) Failing after 2m40s
Test / container (push) Failing after 5m10s
2026-08-23 21:46:00 +02:00
e508f9fd64 US09-01: Serve the Documentation Inside the Application (#107)
Some checks failed
Test / suites (push) Failing after 2m59s
Test / container (push) Failing after 3m56s
2026-08-23 21:09:43 +02:00
370f966d29 E09: Product documentation backlog (#101)
Some checks failed
Test / suites (push) Failing after 3m12s
Test / container (push) Failing after 4m23s
2026-08-23 15:04:45 +02:00
833bfa95bf US08-05: Automate Container Deployment Acceptance (#100)
Some checks failed
Test / suites (push) Failing after 2m19s
Test / container (push) Failing after 5m33s
2026-08-21 10:50:04 +02:00
38 changed files with 6864 additions and 162 deletions

View File

@@ -9,6 +9,7 @@
!photo_pipeline
!migrations
!frontend
!docs
!docker
# Nothing generated, even under an allowed directory.

9
.gitattributes vendored Normal file
View File

@@ -0,0 +1,9 @@
# Vendored third-party browser bundles (US09-01). Minified upstream files carry
# trailing whitespace and very long lines; they are not ours to reformat, and the
# submit gate's `git diff --check` would refuse them forever. Their integrity is
# controlled where it belongs instead: a pinned version and a recorded sha256 in
# frontend/js/vendor/VERSIONS.json, asserted by tests/integration/test_documentation.py.
#
# Only the whitespace check is turned off. They stay text, so a credential scan or a
# search still reads them.
frontend/js/vendor/*.js -whitespace linguist-vendored

View File

@@ -49,3 +49,55 @@ jobs:
- name: Application suite
run: work_item/scripts/python -m pytest tests -q
# The deployed container, verified the way the host application is (US08-05). It
# runs on `main` only — a pull request has nothing published to upgrade *from*, and
# building two images per push would pay for that on every commit. Deploy waits for
# this whole workflow, so a red container gate is a deploy that does not happen.
container:
if: gitea.event_name == 'push'
runs-on: ubuntu-latest
steps:
- name: Checkout
# The upgrade journey builds the previous commit's tree when no published
# image is named, so the history has to be there.
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install the application and its test dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e '.[test]'
python -m playwright install --with-deps chromium
- name: Container acceptance gate
# One command: it builds the image, provisions the composition against a
# temporary fixture library and an isolated volume, runs the phase_h journeys,
# destroys the stack, and writes the evidence. Any skipped check fails it.
env:
# The fixture libraries are bind-mounted into the containers, so this path
# has to be one the Docker daemon can see. On a runner that talks to a
# sibling daemon, point it at a shared host path instead of the workspace —
# an unshared path arrives as an empty mount and the journeys fail on the
# scan, which is the symptom to recognise.
PHOTO_PIPELINE_DATA_DIR: ${{ gitea.workspace }}/gate-data
PHOTO_PIPELINE_LIBRARY_ROOTS: ${{ gitea.workspace }}/gate-library
PHOTO_PIPELINE_TEST_MOUNT_BASE: ${{ gitea.workspace }}/gate-mounts
run: |
mkdir -p "$PHOTO_PIPELINE_LIBRARY_ROOTS" "$PHOTO_PIPELINE_TEST_MOUNT_BASE"
python -m photo_pipeline container-gate --output gate-evidence
- name: Keep the evidence
# Retained per run, and retained on failure especially: the logs are the only
# account of what the containers did.
if: always()
uses: actions/upload-artifact@v3
with:
name: container-gate-${{ gitea.sha }}
path: gate-evidence

View File

@@ -70,6 +70,8 @@ COPY pyproject.toml alembic.ini README.md ./
COPY photo_pipeline ./photo_pipeline
COPY migrations ./migrations
COPY frontend ./frontend
# The manuals are served by the application itself (US09-01), so they ship with it.
COPY docs ./docs
COPY docker/entrypoint.sh docker/healthcheck.sh /usr/local/bin/
# Runtime dependencies only: the `test` extra (pytest, playwright) and the `vision`

View File

@@ -241,6 +241,40 @@ image. The stack is managed by Portainer from git (`docker-compose.yml`), and th
values live in the Portainer stack's environment, so rotation is one place and a
repository read discloses nothing. A test asserts the workflows never name them.
## Container acceptance gate (US08-05)
The deployed container is verified the way the host application is, by one command:
```bash
work_item/scripts/python -m photo_pipeline container-gate --output gate-evidence
```
It builds the image, provisions the composition against a **temporary fixture library
on a bind mount and an isolated data volume**, runs the `phase_h` journeys against it,
destroys every stack afterwards, and writes `release-report.json`, `logs/container.log`,
and `CHECKSUMS.sha256` into the evidence directory. It exits non-zero when a journey
fails, when the story matrix has a hole, **or when any check skipped at all** — unlike
the release gate, this one accepts no environment excuse: a run that never reached the
containers proved nothing about them.
The journeys (`tests/e2e/test_phase_h_container.py`) are:
| journey | what it proves |
|---|---|
| browser | discovery, duplicate review, analysis, album proposal, rename, upload preflight, and archive views, driven through the containerized frontend — including a rename that really moves the operator's folder on the bind mount |
| upgrade | the previous version's image runs first, then this one against the same volume: schema at the new head, assets, analysis results, job progress, the unapplied rename plan, and the thumbnail cache all survive |
| restart | `docker kill` on both containers mid-job; the job resumes, every photo ends with exactly one stored result, and only the in-flight item ever reaches the provider twice |
| security | no session refused, a forged `X-Forwarded-Host` cannot smuggle an allowed hostname past the check, a symlink out of the mounted library is refused, an unmounted library root refuses startup, and no secret appears in `docker compose logs` |
The upgrade journey builds the previous commit's tree when no published image is named;
point it at the real one with `PHOTO_PIPELINE_PREVIOUS_IMAGE`. On a Docker VM (Colima,
Docker Desktop) the fixture library must live on a shared path — it defaults to
`~/.cache/photo-pipeline`, overridable with `PHOTO_PIPELINE_TEST_MOUNT_BASE`.
CI runs this gate as the `container` job of **Test** on pushes to `main` and keeps its
evidence as a run artefact. Deploy waits for the whole `Test` workflow, so a red
container gate is a publish that does not happen.
## Testing
One offline command runs the whole suite (unit, integration, and browser

View File

@@ -0,0 +1,77 @@
# E09 — Product Documentation
Concept phase: none. Like [E08](E08-container-deployment.md), this epic is a delivery
addition rather than a product-scope change: the same application, documented well
enough that somebody who did not build it can install it, understand it, and operate it
without reading the source.
It does not change the product scope in
[`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md). No safety
invariant moves, no schema changes, no new runtime capability. The one change to the
running application is a documentation view and the static assets it needs.
## Why the application serves its own documentation
The manuals describe an application that is reached over HTTP behind an access secret.
Documentation that lives only in the repository is unreachable from the deployment it
describes: an operator who has just been handed a URL and a secret has no Gitea account
in front of them. So the same markdown files are both the repository's documentation and
the deployment's `/docs` view, and neither is a copy of the other.
## Decisions made in this epic
**Markdown is the source.** Everything is written as markdown under `docs/`, so it is
reviewable in a diff, readable on Gitea, and renderable in the app. No documentation
format that only a tool can read.
**The renderer is vendored, not written and not fetched.** `marked` (MIT, no
dependencies, ships an ES module) is pinned and committed under
`frontend/js/vendor/`. A CDN is not an option: the application is deployed to a network
whose outbound access is not assumed, and `default-src 'self'` forbids it.
**Diagrams are mermaid, rendered client-side, with the script boundary intact.** The
claim that mermaid requires `'unsafe-eval'` was tested rather than believed: mermaid 11's
bundle contains no `eval(` and no `new Function` — only a lodash `Function("return this")`
global-detection fallback that short-circuits on `globalThis` and never executes.
Rendered under this application's exact CSP, a flowchart produced a 15.8 KB SVG and
raised **no `script-src` violation**. What it does raise is `style-src`: mermaid styles
its output with an injected `<style>` element and `style=` attributes.
So `script-src 'self'` stays exactly as US07-02 and US08-01 left it, and `style-src`
gains `'unsafe-inline'`. That relaxation is bounded on purpose: with `script-src` intact
no injected markup can execute, and with `img-src`, `connect-src`, and `font-src` all
still `'self'`, the CSS-based exfiltration channels stay closed. What remains is
defacement of a page the operator is already authenticated on.
The rejected alternative is recorded because it may become the better trade later:
pre-rendering each diagram to a committed SVG with the already-installed Playwright
(no Node toolchain needed) and serving that, which would leave CSP untouched entirely at
the cost of a generator and a freshness check. If `style-src 'unsafe-inline'` is ever
judged too much, that is the migration, and it does not change a single markdown file.
**Screenshots are produced, not pasted.** Every screenshot in the user manual is captured
by Playwright from the real application against a temporary fixture library, by the same
kind of code that already drives the browser suites. A screenshot nobody can regenerate
is a screenshot that silently stops being true.
**The documentation is held to the code by tests.** Configuration keys, CLI commands,
exit codes, API error codes, job and journal states, and module names all appear in both
the code and the manuals. Each of those is cross-checked, so the failure mode of stale
documentation is a red test rather than a misled operator.
## Stories
1. [US09-01 — Serve the documentation inside the application](stories/US09-01-docs-in-app.md)
2. [US09-02 — Write the installation and operations manual](stories/US09-02-installation-manual.md)
3. [US09-03 — Write the architecture overview](stories/US09-03-architecture-overview.md)
4. [US09-04 — Write the user manual with generated screenshots](stories/US09-04-user-manual.md)
5. [US09-05 — Automate documentation acceptance](stories/US09-05-docs-gate.md)
## Epic outcome
A deployed instance serves its own installation manual, architecture overview, and
illustrated user manual at `/app/#/docs`, rendered from the same markdown files that are
readable in the repository. Screenshots are regenerated from the running application,
every documented command, setting, state, and error code is cross-checked against the
code that implements it, and one gate fails the build when documentation and application
disagree.

View File

@@ -2,13 +2,14 @@
This backlog decomposes the phases in
[`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md) into seven
epics and small, independently verifiable user stories, plus one delivery-format
epic (E08) that packages the released application as a deployable container.
epics and small, independently verifiable user stories, plus two delivery-format
epics: E08 packages the released application as a deployable container, and E09
documents it for the people who install, operate, and use it.
## Numbering and file naming
- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` has no
concept phase and must not change product scope.
- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` and `E09`
have no concept phase and must not change product scope.
- Stories: `US<epic>-<sequence>`, for example `US03-02`.
- Epic files: `E01-<slug>.md`.
- Story files: `stories/US01-01-<slug>.md`.
@@ -39,6 +40,7 @@ epic (E08) that packages the released application as a deployable container.
6. [E06 — Archive lifecycle](E06-archive-lifecycle.md)
7. [E07 — Hardening and release](E07-hardening-release.md)
8. [E08 — Container deployment](E08-container-deployment.md)
9. [E09 — Product documentation](E09-documentation.md)
## Shared definition of done

View File

@@ -0,0 +1,49 @@
# US09-01 — Serve the Documentation Inside the Application
Epic: [E09](../E09-documentation.md)
As an operator who was handed a URL and an access secret, I want the manuals inside the
application I am looking at, so that understanding it does not require a repository
checkout or an internet connection.
## Acceptance criteria
- Documentation lives as markdown under `docs/`, with an index that names every page and
the order it is meant to be read in. The same files render on Gitea without
modification.
- The application serves a `/docs` view reachable from the main navigation, listing the
pages and rendering the selected one.
- Rendering uses a vendored, version-pinned markdown library committed under
`frontend/js/vendor/`. It is not fetched from a CDN, not bundled by a build step, and
not written by hand. The build records the version and a checksum of the vendored file.
- Diagrams written as ```mermaid``` fenced blocks render as diagrams. Mermaid is vendored
the same way.
- `script-src` in the Content-Security-Policy is unchanged: no `'unsafe-eval'`, no
`'unsafe-inline'`. Only `style-src` gains `'unsafe-inline'`, and the reason is recorded
where the policy is defined.
- Links between documents work in the app: a relative `../foo.md#section` link navigates
to that page and heading rather than downloading a file or leaving the application.
Every heading has a stable anchor, and a deep link to an anchor scrolls to it.
- An unknown page renders a documentation-specific not-found message with a link back to
the index, and never exposes a filesystem path.
- Documentation is readable without an access secret **or** behind the same session as
the rest of the application — whichever is chosen is stated explicitly in the story's
implementation notes and covered by a test, because an operator locked out by a
configuration mistake is exactly who needs the troubleshooting page.
- The view works with no outbound network access at all.
## Automated tests
- Unit: heading-anchor slugs (duplicates, punctuation, non-ASCII), and the rewriting of
relative markdown links into in-app routes.
- Integration: every markdown file under `docs/` is reachable from the index; every
internal link in every document resolves to a file and, where an anchor is given, to a
heading that exists; the vendored library files match their recorded checksums.
- Browser: the documentation view renders a page, follows an internal link, deep-links to
an anchor, renders a mermaid diagram to an SVG, and shows the not-found message for an
unknown page — all with **zero console errors and zero CSP violations**, asserted, not
eyeballed.
## Dependencies
- None beyond the delivered application.

View File

@@ -0,0 +1,46 @@
# US09-02 — Write the Installation and Operations Manual
Epic: [E09](../E09-documentation.md)
As somebody installing this application for the first time, I want one manual that takes
me from nothing to a running instance pointed at my photo library, so that I do not have
to reconstruct the procedure from the README, the compose file, and the test suite.
## Acceptance criteria
- A **host installation** path: prerequisites and their versions (Python, exiftool,
immich-go, Playwright's browser for the test suite), virtual environment, dependency
install, `.env`, database migration, starting `serve` and `worker`, and how to verify
the install succeeded.
- A **container installation** path: the published image, the compose file, the data
volume, the library bind mount and why it is mounted the way it is, published ports,
running behind a reverse proxy with `PHOTO_PIPELINE_ALLOWED_HOSTS` and
`PHOTO_PIPELINE_TRUSTED_PROXIES`, and the Portainer/webhook deployment already in use.
- A **configuration reference**: every `PHOTO_PIPELINE_*` setting with its meaning,
default, accepted values, and whether it is a secret. Secrets are described, never
exemplified with a real-looking value.
- A **first run checklist** that ends in a verified state: library discovered, worker
claiming jobs, readiness endpoint green, diagnostics clean.
- **Operations**: upgrading (including what the migration backup does), backup, verify,
restore, pruning, diagnostics, and the log locations for each role.
- **Troubleshooting**: each refusal an operator can hit at startup — every non-zero exit
code of the CLI, the trust-boundary refusal, the unmounted library root refusal, the
library lock being held, and a legacy CLI still running — with the cause and the fix.
- The manual states the safety invariants an installer must not work around: one worker
writes, `_IGNORE/` is never read, EXIF is verified before upload, and the library lock
is authoritative.
## Automated tests
- Every `PHOTO_PIPELINE_*` setting documented exists as a field on `Config`, and every
field on `Config` is documented — both directions, so a new setting cannot ship
undocumented.
- Every CLI command and flag named in the manual exists in the argument parser, and every
subcommand the parser accepts appears in the manual.
- Every exit code documented is one the CLI can actually return.
- No documented example value collides with a real secret pattern (the repository's own
credential scanner runs over `docs/`).
## Dependencies
- US09-01

View File

@@ -0,0 +1,47 @@
# US09-03 — Write the Architecture Overview
Epic: [E09](../E09-documentation.md)
As a developer or reviewer new to this repository, I want an architecture overview that
explains what the pieces are and which rules they enforce, so that I can find the right
module and not violate an invariant I never knew existed.
## Acceptance criteria
- A **context diagram**: the operator, the browser application, the API and worker, the
photo library, the Immich server, the vision provider, and the archive destination —
with the direction and nature of every interaction, including which ones leave the
machine.
- A **runtime diagram**: the migrate/api/worker composition, the data volume, the library
bind mount, the SQLite database, and the process lock — showing what is shared and what
is exclusive.
- A **module map**: every package under `photo_pipeline/` with its responsibility and the
boundary it must not cross (which modules may touch the filesystem, which may call an
external provider, which own schema).
- **Key flows**, each as a diagram plus prose: the durable job lifecycle from enqueue to
recovery; the rename journal state machine including the rollback and manual-resolution
paths; the upload lifecycle through preflight, run, report ingestion, and verification.
- **The invariants and where they are enforced**, each pointing at the module that owns
it: one writer at a time, the library process lock, the path policy and library roots,
`_IGNORE/` exclusion, verified EXIF before upload, safety decisions gating the vision
provider, and restart-safety of every mutation.
- **The data model**: the tables, what identity means for an asset, and how a moved file
keeps its identity.
- A short **history and donor** section: what was migrated out of the legacy CLIs, where
the archive and its ledger live, and why they are kept.
- Diagrams are ```mermaid``` blocks so the source is diffable and Gitea renders them
natively.
## Automated tests
- Every package and top-level module under `photo_pipeline/` appears in the module map,
and every module named in the map exists — a new service cannot appear on the map's
blind side.
- Every job state, journal state, and upload batch state named in the document exists in
the code, and every state the code defines is named in the document.
- Every mermaid block parses (rendered in the browser test without an error node).
- Every code path referenced by file name exists.
## Dependencies
- US09-01

View File

@@ -0,0 +1,52 @@
# US09-04 — Write the User Manual with Generated Screenshots
Epic: [E09](../E09-documentation.md)
As the person actually sorting a photo library, I want a manual that walks the workflow
screen by screen and tells me what each refusal means, so that I can use the application
confidently and know what it is about to do to my files before it does it.
## Acceptance criteria
- One page per workflow stage, in the order the application presents them: discovery and
inventory, duplicate review, safety review, analysis, album proposals, renames, upload,
archive, and diagnostics. Each page answers the same four questions: what this stage is
for, what I have to decide, **what it changes on disk or on the server**, and what it
refuses to do.
- A **guided first pass** that takes a new library from scan to a verified upload, naming
the point of no return in each stage and what is reversible after it.
- Every stage page carries at least one screenshot of the real application showing the
state being described.
- Screenshots are **generated** by a committed Playwright script that seeds a temporary
fixture library, drives the application, and writes the images. Regenerating them is one
documented command. No screenshot is captured by hand.
- Screenshots contain no real library paths, no personal photos, and no secrets — the
fixture library is synthetic, and this is asserted rather than assumed.
- An **error and refusal catalogue**: every `error.code` the API can return, with what
causes it, what the application did or refused to do, and what the operator should do
next. Refusals that protect data (`lock_held`, `rename_recovery_required`,
`stale_preflight`, `path_not_allowed`, `host_not_allowed`, `dry_run_not_approved`,
conflict codes) are explained as intentional, not as faults.
- A **recovery** page: an interrupted rename, an uncertain upload, a failed migration, a
restored backup — what the application does by itself and what needs a decision.
- The work-item safety allow list is extended to permit `docs/images/**`, since the
repository denies image files by default. This is an explicit, reviewed change, not a
quiet one, and it stays narrow enough that a real photo still cannot be committed.
## Automated tests
- Every `error.code` the application can emit is documented, and every code documented
exists in the code — both directions.
- Every image referenced by a documentation page exists, and every image under
`docs/images/` is referenced by a page.
- The screenshot generator runs end to end in the test environment and produces every
image the manual references, against a temporary fixture library that is destroyed
afterwards.
- Generated screenshots are checked for library paths outside the fixture root and for
the configured secrets.
- Browser: each stage page renders in the documentation view with its screenshot loaded
and no console error.
## Dependencies
- US09-01

View File

@@ -0,0 +1,38 @@
# US09-05 — Automate Documentation Acceptance
Epic: [E09](../E09-documentation.md)
As a release owner, I want one gate that proves the documentation still describes the
application, so that a change to the code cannot quietly make the manuals wrong.
## Acceptance criteria
- One documented command runs every documentation check and retains its evidence, in the
shape the release and container gates already use.
- The gate fails when: an internal link or anchor is dead; a page is unreachable from the
index; an image is referenced but missing or present but unreferenced; a documented
setting, command, exit code, error code, or state does not exist in the code; a code
path exists that the documentation is required to cover and does not.
- The gate regenerates the screenshots and fails when a regenerated image no longer
matches the committed one beyond a stated tolerance, so a UI change that invalidates the
manual is a red build rather than a discovery months later.
- The gate renders every documentation page in a real browser and fails on any console
error or CSP violation, and asserts that `script-src` contains neither `'unsafe-eval'`
nor `'unsafe-inline'`.
- The checks run on a `phase_i` marker; CI runs the gate, and the earlier epic suites keep
running unchanged.
- The README and the application's documentation index point at each other, so neither is
the forgotten copy.
- Documentation stories are mapped in the story traceability matrix like every other
story.
## Automated tests
- The gate's own contract is testable without a browser: a seeded broken link, a missing
image, an undocumented error code, and a stale screenshot each fail it, and a clean tree
passes.
- The full documentation suite runs on `phase_i` in CI.
## Dependencies
- US09-01 through US09-04

224
docs/architecture.md Normal file
View File

@@ -0,0 +1,224 @@
# Architecture
[← Documentation index](index.md)
For whoever has to change this code without breaking somebody's photo library. It
explains what the pieces are, which rules each one keeps, and where to look when a
stage refuses.
The product decisions behind all of it live in `INTEGRATED_PIPELINE_CONCEPT.md`; this
page describes what was built.
## Context
Five things outside the application, and what actually crosses each boundary.
```mermaid
flowchart LR
operator([Operator]):::person -->|browser, one session| app
app[Photo Pipeline]:::system -->|read, rename, EXIF write| library[(Photo library<br/>bind mount)]
app -->|database, cache, journals, backups| data[(Data directory<br/>local filesystem)]
app -->|confirmed-SFW images only| vision[Vision provider]:::ext
app -->|immich-go, verified bytes| immich[Immich server]:::ext
app -->|copy, verify, then remove| archive[(Archive medium)]
classDef person fill:#1f6feb,stroke:#58a6ff,color:#fff
classDef system fill:#238636,stroke:#3fb950,color:#fff
classDef ext fill:#6e40c9,stroke:#a371f7,color:#fff
```
| boundary | leaves the machine? | carries |
|---|---|---|
| operator → app | no (loopback, or a proxy you configured) | commands against stable ids, never paths |
| app → library | no | reads, folder renames, EXIF merges |
| app → data directory | no | SQLite in WAL mode, thumbnails, journals, backups |
| app → vision provider | **yes** | image bytes of confirmed-SFW canonical assets only |
| app → Immich | **yes** | the exact verified bytes of an approved album |
| app → archive medium | no | copies, verified before the source is removed |
## Runtime
Two long-lived processes and a one-shot migration, sharing one database and one
library.
```mermaid
flowchart TD
browser([Browser]) -->|JSON + SSE| api
migrate["migrate<br/>backup, then upgrade"] -->|must exit 0| api
migrate --> worker
api["api · serve<br/>enqueues, serves, reads"] -->|jobs table| db[(SQLite WAL)]
worker["worker<br/>claims and does the work"] -->|jobs table| db
worker --> tools["exiftool · vision API · immich-go"]
api -.->|api.lock.json| lock{{library lock}}
worker -.->|worker.lock.json| lock
api --> lib[(library)]
worker --> lib
```
`serve` enqueues and renders; **it does not do the work**. Everything that scans,
scores, analyses, renames, uploads, or archives happens in the worker, which claims
a queued job atomically with a fencing token. One mutating job runs at a time, and
the lock file in the data directory is what makes a second worker impossible rather
than merely discouraged.
## The modules
| package | owns | must not |
|---|---|---|
| `api/` | HTTP surface, error envelope, security policy | contain SQL or business logic |
| `api/routes/` | one module per resource group, all under `/api/v1` | accept a filesystem path from the browser |
| `api/security.py` | host/origin/CSRF/session/size policy as one pure `evaluate`, plus the ASGI middleware | be bypassed per-route |
| `schemas/` | Pydantic request and response contracts | reach the database |
| `services/` | all domain logic; the only place a decision is made | be imported by the frozen CLI archive |
| `models/` | SQLAlchemy tables | hold behaviour |
| `jobs/` | durable job lifecycle, worker loop, lock ranks, handler registry | run work in the API process |
| `integrations/` | the real outside world: `exiftool`, vision, NSFW model, `immich-go` and its report grammars | be called without a version recorded |
| `path_policy.py` | the library boundary, `_IGNORE/` exclusion, symlink-escape refusal | be duplicated anywhere |
| `imaging.py` | the single bounded-decode door | let a caller open an image directly |
| `faults.py` | the crash barriers the tests fire | read anything but its one env var |
| `config.py` | typed settings, secrets as `SecretStr` | log or return a value |
| `db.py` | engine, sessions, WAL and foreign keys, migration entry | be used to bypass a repository |
Services worth knowing by name: `inventory`, `duplicates`, `safety`, `analysis`,
`albums`/`proposals`/`naming`, `renames`/`rename_apply`/`rename_journal`,
`uploads`/`upload_batches`/`upload_reports`/`upload_verification`,
`archives`/`archive_transfer`/`archive_journal`/`restores`/`availability`,
`thumbnails`, `hashing`, `exif_checkpoint`, `jobs`, `workflow`, `backup`,
`diagnostics`, `app_lock`, `release`, `benchmarks`, `library`, `legacy_import`.
## The state machines
Three journals decide what a restart is allowed to assume. All three are read from
the database plus the real world — never guessed from a missing file.
### Durable jobs
```mermaid
stateDiagram-v2
[*] --> queued
queued --> running: claimed with a fencing token
queued --> cancelled
queued --> cancelling
running --> succeeded
running --> failed
running --> cancelling
cancelling --> cancelled
failed --> retry_queued
retry_queued --> running
succeeded --> [*]
cancelled --> [*]
```
`succeeded` and `cancelled` are terminal, so a duplicate delivery cannot move a
finished job. Claiming is compare-and-set on the row version; a worker whose lease
expired cannot commit after another has taken over.
### The rename journal
The only state machine that moves somebody's folders.
```mermaid
stateDiagram-v2
[*] --> planned
planned --> moving
moving --> moved
moving --> planned: proven untouched
moving --> rollback_required
moved --> database_updated
database_updated --> verified
verified --> complete
moved --> rollback_required
database_updated --> rollback_required
verified --> rollback_required
planned --> failed
failed --> planned
rollback_required --> rolled_back
complete --> [*]
rolled_back --> [*]
```
Intent is written **before** the disk is touched, which is why `moving` can resolve
backwards: the evidence decides. `moving`, `moved`, `database_updated`, and
`rollback_required` are the unsafe states — while any operation sits in one, the
library may be half-renamed, so unrelated mutations are refused with
`409 rename_recovery_required` until a person resolves it.
### Upload batches
```mermaid
stateDiagram-v2
[*] --> planned
planned --> running
running --> succeeded
running --> failed
running --> cancelling
running --> unknown_requires_verification: process died after acceptance
cancelling --> cancelled
failed --> running: retry
cancelled --> running: retry
```
`unknown_requires_verification` is deliberately **not** restartable. The server may
already hold the files; the answer is to ask Immich for the recorded SHA-1, not to
upload again and hope.
Archive transfers use the same shape — `planned → transferring → verified →
removing → complete` — and the source is removed only after the archived bytes are
verified.
## Identity and the data model
A path is metadata. The identity is `assets.id`, a UUID that never changes, and
`asset_paths` records every path an asset has ever had with the reason it changed.
That is what makes a rename cheap: nothing else in the database has to move.
Tables: `assets`, `asset_paths`, `thumbnails`, `jobs`, `job_items`, `job_events`,
`safety_reviews`, `analysis_results`, `exif_projections`, `duplicate_clusters`,
`duplicate_members`, `duplicate_negative_links`, `album_proposals`, `rename_plans`,
`rename_operations`, `upload_batches`, `upload_items`, `upload_verifications`,
`archive_locations`, `archive_plans`, `archive_operations`.
Availability is independent of workflow progress: `active`, `archiving`,
`archived_online`, `archived_offline`, `restoring`, `missing_unexpected`. An
unmounted archive disk is `archived_offline`, never `missing` — the scanner is not
allowed to conclude that a photo is gone because a disk is unplugged.
EXIF is a projection with its own verified state (`verified`, `divergent`,
`failed`): the desired values are written, read back, and compared, and anything the
stage does not own having changed makes the asset `divergent` and blocks the next
mutating stage.
## Where each invariant lives
| invariant | enforced in |
|---|---|
| `_IGNORE/` is never traversed, counted, or opened | `path_policy.is_excluded`, used by every discovery path |
| no path outside the library roots is reachable | `path_policy.resolve_in_roots` — it returns the resolved path, because validating one name and opening another is the symlink race |
| one writer per library | `services/app_lock.py` (an `flock` on a JSON lock file, per role) |
| one mutating job at a time | `services/jobs.py` lock keys plus `jobs/locks.py` rank ordering |
| only confirmed-SFW assets reach the vision provider | `services/analysis.py`, re-checked *after* the provider call so a decision that flipped mid-flight discards the result |
| EXIF is verified, and other fields preserved | `services/exif_checkpoint.py` |
| uploads carry the exact verified bytes | `services/uploads.py` preflight, re-proved immediately before the uploader runs |
| an uncertain upload is not a failure | `services/upload_batches.py`, `services/upload_verification.py` |
| the archive source outlives its copy until verified | `services/archive_transfer.py` |
| decoding is bounded | `imaging.py` |
| the trust boundary and CSRF | `api/security.py` |
## Concurrency
Locks are taken broad to narrow — `library → stage/job → album/folder → asset` — and
never the other way, which is what makes deadlock structural rather than lucky.
Ranks live in `jobs/locks.py`.
Exactly-once execution is not achievable across SQLite, a filesystem, subprocesses,
and a remote server. The design promises **at-least-once with idempotent recovery**:
every handler may run twice, and running twice must not produce two side effects.
That is why the rename journal records intent before moving, why EXIF writes merge
and verify, and why upload retries consult both local history and Immich.
## History
This application was extracted from two command-line tools rather than written from
nothing. Their sources are frozen in `legacy_cli_archive/` with the ledger mapping
each donated behaviour to the service that now owns it, the characterization tests
that pinned it, and every intentional difference. Production code must not import
them; they are provenance and rollback evidence.

35
docs/index.md Normal file
View File

@@ -0,0 +1,35 @@
# Photo Pipeline documentation
One local application that takes a photo library from discovery to a verified Immich
upload: duplicate detection, safety review, content analysis, album naming, guarded
renaming, upload, and archive — one visible, resume-safe workflow.
These pages are readable three ways, and they are the same files each time: in the
repository under `docs/`, on Gitea, and inside the running application under
**Docs**. There is no separate copy to fall out of date.
## Read in this order
1. [Overview](overview.md) — what the application does, the stages it moves a photo
through, and the rules it will not break.
2. [Installation and operations](installation.md) — host and container installation,
every setting, the first-run checklist, upgrades, backup and restore, and what
each refusal at startup means.
3. [Architecture](architecture.md) — the context and runtime diagrams, what each
module owns, the three journals a restart reads, and where every invariant is
enforced.
## Being written
The remaining manuals are accepted work, not aspiration; each is a story in
[E09](https://git.domverse-berlin.eu/domverse/photoanalyzer/src/branch/main/delivery_backlog/E09-documentation.md)
and will appear here as it lands.
- **User manual** — one page per workflow stage with screenshots of the real
application, and a catalogue of every error and refusal (US09-04).
## Conventions
A page tells you what a stage **changes on disk or on the server** before it tells
you how to run it. Refusals are documented as intentional: this application would
rather stop and explain than guess about somebody's photographs.

336
docs/installation.md Normal file
View File

@@ -0,0 +1,336 @@
# Installation and operations
[← Documentation index](index.md)
From nothing to a running instance pointed at your photo library, and everything you
need afterwards: upgrading, backing up, restoring, and understanding a refusal.
Two ways to install. **Container** is the deployment this project builds and ships;
**host** is what you want for development or a single machine you already manage.
Both run the same two processes against the same database.
> Read [what it will not do](overview.md#what-it-will-not-do) first. Several of the
> steps below only make sense once you know which rules the application is keeping.
## Before you start
| you need | version | why |
|---|---|---|
| Python | 3.12 or newer | the application |
| `exiftool` | 13.x (the image pins `13.25+dfsg-1`) | every EXIF read and write |
| `immich-go` | 0.32.0 (pinned in the image) | the upload stage only |
| Docker + the compose plugin | any current | the container installation only |
| a vision provider key | — | the analysis stage only |
You also need a photo library you can afford to be wrong about. Take a backup of it
before pointing anything at it for the first time, and consider running with
[`PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL`](#the-settings) on.
## Host installation
```bash
python3.12 -m venv .venv
.venv/bin/pip install -e ".[vision]" # drop [vision] for a review-only install
```
Configure it. Every setting is an environment variable; a `.env` file in the working
directory is read at startup:
```bash
cp .env.example .env && $EDITOR .env
```
`.env.example` lists every variable with its default and no values at all. For a
loopback installation you need exactly one:
```bash
PHOTO_PIPELINE_LIBRARY_ROOTS=/home/you/Pictures
```
Then migrate and start the two processes:
```bash
.venv/bin/python -m photo_pipeline migrate # create or upgrade the database
.venv/bin/python -m photo_pipeline serve # API and browser app on 127.0.0.1:8000
.venv/bin/python -m photo_pipeline worker # second terminal
```
**Both processes are required.** `serve` enqueues work and renders the application;
nothing is scanned, scored, analysed, renamed, uploaded, or archived without a
worker. Open <http://127.0.0.1:8000/app/> and confirm the workflow page loads.
### The configuration file
`.env`, or any path named by `PHOTO_PIPELINE_ENV_FILE`. It is parsed, never executed:
`KEY=value` lines, `#` comments, optional quotes, no interpolation and no `export`. A
configuration file that can run code is a configuration file that can be a
vulnerability.
**Anything already exported in the shell wins.** The file is your standing
configuration; the environment is the override for one run.
The archived CLI's names still work, so an existing `photo_analyzer.env` can be used
as it is:
| in the file | applied as |
|---|---|
| `LLM_API_KEY` / `GEMINI_API_KEY` | `OPENAI_API_KEY` |
| `LLM_BASE_URL` | `OPENAI_BASE_URL` |
| `LIBRARY` | `PHOTO_PIPELINE_LIBRARY_ROOTS` |
`.env` and `*.env` are gitignored and refused by the repository's safety checks. The
file holds a real key; it must never be committed.
## Container installation
```bash
cp .env.example .env && $EDITOR .env
docker compose up -d --build
```
The composition is one `serve` container, one `worker` container, a one-shot
`migrate` that both wait for, one bind-mounted library, and one named data volume.
Three variables it cannot start without:
| variable | is |
|---|---|
| `PHOTO_PIPELINE_LIBRARY_HOST_PATH` | the library on this host |
| `PHOTO_PIPELINE_LIBRARY_ROOTS` | where that library is mounted **inside** the container |
| `PHOTO_PIPELINE_ACCESS_SECRET` | required, because publishing a port means the app is reachable from outside the container |
Set `PHOTO_PIPELINE_UID` and `PHOTO_PIPELINE_GID` to the owner of the library: what
the containers rename and rewrite keeps that ownership.
### The data volume
The `data` volume holds the database, its write-ahead log, the thumbnail cache, the
operation journals, and the backups. **It must stay on a local filesystem.** SQLite
in WAL mode needs real local locking, so NFS, SMB, and network volume drivers do not
slow it down — they corrupt it. The library bind mount has no such restriction.
### Ports and reaching it
The API port is published to `127.0.0.1` unless `PHOTO_PIPELINE_PUBLISH_ADDRESS` says
otherwise. Being reachable *was* the authentication in earlier versions: whoever could
open the port owned the library. So the moment the application answers to anything but
loopback — a hostname, a proxy, `0.0.0.0` — the access secret becomes mandatory and
`serve` refuses to start without it rather than publishing your photographs.
Behind a reverse proxy:
```bash
PHOTO_PIPELINE_ALLOWED_HOSTS=photos.example.com
PHOTO_PIPELINE_ACCESS_SECRET=# python -c 'import secrets; print(secrets.token_urlsafe(32))'
PHOTO_PIPELINE_TRUSTED_PROXIES=10.0.0.2 # only the proxy's own address
```
`X-Forwarded-Proto` and `X-Forwarded-Host` are believed only from a trusted-proxy
address, so a client cannot declare its own origin. The browser asks for the secret
once per tab. Wrong secrets are rate-limited and logged with the caller's address
only. The health endpoints stay open so an orchestrator can restart the container;
nothing else is.
### Deployment
The stack is managed from git by Portainer, which redeploys on a webhook after CI
publishes an image built from `main`. Runtime secrets live in the Portainer stack's
environment rather than in the repository, so rotation happens in one place and
reading the repository discloses nothing.
## The settings
Every variable, its default, and whether it is a secret. `.env.example` is the same
list in copyable form.
### Library and data
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_LIBRARY_ROOTS` | none | the library boundary, `os.pathsep`-separated. No path outside these roots is ever read or written |
| `PHOTO_PIPELINE_DATA_DIR` | `data` | database, WAL, thumbnail cache, journals, backups. Never inside the library |
| `PHOTO_PIPELINE_DB_PATH` | `<data dir>/photo_pipeline.db` | the database file, if it must live elsewhere |
| `PHOTO_PIPELINE_ENV_FILE` | `.env` | where to read the configuration file from |
### Serving and the trust boundary
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_HOST` | `127.0.0.1` | bind address |
| `PHOTO_PIPELINE_PORT` | `8000` | port |
| `PHOTO_PIPELINE_ALLOWED_HOSTS` | none | comma-separated names the app answers to besides loopback. Naming one makes the access secret mandatory |
| `PHOTO_PIPELINE_ACCESS_SECRET` | none | **secret.** Traded for the session cookie at `GET /api/v1/session` |
| `PHOTO_PIPELINE_TRUSTED_PROXIES` | none | comma-separated peer addresses whose forwarded headers may be believed |
| `PHOTO_PIPELINE_MAX_REQUEST_BYTES` | `1048576` | largest request body accepted |
### Logging
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_LOG_LEVEL` | `INFO` | standard Python levels |
| `PHOTO_PIPELINE_LOG_FORMAT` | `json` | `json` or `text` |
### Limits
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_THUMBNAIL_CACHE_QUOTA_BYTES` | `500000000` | cache quota; over it is a diagnostics warning |
| `PHOTO_PIPELINE_THUMBNAIL_MAX_PIXELS` | `100000000` | refuse to decode anything larger. This is the decompression-bomb guard |
| `PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES` | `1000000000` | free space an archive destination must keep beyond the transfer itself |
### The safety gate
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL` | `false` | refuse every mutating request until a read-only dry run of this library has been produced and approved |
### External services
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_VISION_API_KEY` | none | **secret.** Without it the analysis stage cannot run |
| `PHOTO_PIPELINE_IMMICH_SERVER_URL` | none | the Immich server |
| `PHOTO_PIPELINE_IMMICH_API_KEY` | none | **secret.** |
| `PHOTO_PIPELINE_IMMICH_GO_BINARY` | `immich-go` | the uploader, found on `PATH` |
### Composition only
Read by `docker-compose.yml`, not by the application:
`PHOTO_PIPELINE_IMAGE`, `PHOTO_PIPELINE_LIBRARY_HOST_PATH`,
`PHOTO_PIPELINE_PUBLISH_ADDRESS`, `PHOTO_PIPELINE_UID`, `PHOTO_PIPELINE_GID`.
A secret is never written to a log, never returned by the API, never stored in the
database, and never recorded in a backup manifest — a manifest says `configured`, not
the value. Do not put a real key in an example, a ticket, or a screenshot.
## First run checklist
Not "it started" — verified.
1. **The database is at the current revision.**
`python -m photo_pipeline migrate` exits 0.
2. **The application answers.**
`curl -s localhost:8000/api/v1/health/ready` returns 200.
3. **The library was found.** Open the app, run a scan from the Inventory view, and
check the count against what you expect. Anything under `_IGNORE/` must be missing
from it — that is the exclusion working, not a bug.
4. **The worker is claiming.** The scan job reaches `succeeded`. If it stays
`queued`, no worker is running.
5. **Diagnostics are clean.**
`python -m photo_pipeline diagnostics` reports free space and an empty `warnings`.
6. **Nothing was modified.** The scan is read-only; your files' timestamps are
unchanged.
Only then point it at the whole library.
## Operations
Every operation is the same CLI, on a host or in the composition:
```bash
python -m photo_pipeline diagnostics
docker compose run --rm --no-deps api diagnostics
```
`--no-deps` keeps a one-off command from starting a second stack; `api` is only the
service it borrows the image and mounts from.
### Upgrading
1. `python -m photo_pipeline backup --reason before-upgrade`
2. Pull the new version (or `docker compose pull && docker compose up -d`).
3. Migrations run by themselves at startup, and a **pending schema change is
snapshotted first**. If the upgrade fails, the previous database and its
`pre-migration` backup are both intact, and the error log names the backup
directory.
An up-to-date database is not backed up again on every start.
### Backing up
```bash
python -m photo_pipeline backup --reason weekly --keep 7
python -m photo_pipeline verify-backup data/backups/<name>
```
Backups go through SQLite's online backup API, never a file copy: with WAL enabled
the `.db` file alone is missing every committed page still in the write-ahead log.
Each backup is a directory holding the snapshot and a `manifest.json` — schema
revision, SHA-256, row counts, the archive media the library depends on, and which
settings were configured.
`verify-backup` runs `PRAGMA integrity_check` **and** `PRAGMA foreign_key_check`,
compares the snapshot's SHA-256 against the manifest, and re-counts every table it
recorded. Bit rot, a truncated copy, and a "repaired" snapshot all fail it.
`--keep N` prunes the oldest and never the newest. The same is available at
`GET /api/v1/diagnostics`, `GET|POST /api/v1/backups`,
`GET /api/v1/backups/{name}/verify`, and `POST /api/v1/backups/prune`.
### Restoring
Restore is deliberately **not** an API call. It replaces the state of an
installation, so it belongs to a stopped one and a person at a terminal.
1. Stop `serve` and `worker`.
2. `python -m photo_pipeline verify-backup data/backups/<name>` — never restore an
unverified snapshot, and `restore` will refuse one anyway.
3. `python -m photo_pipeline restore data/backups/<name> --into /path/to/fresh-data`.
A target that already holds a database is refused; recovering in place means
moving the old data directory aside first.
4. Point `PHOTO_PIPELINE_DATA_DIR` at the restored directory and run `migrate`.
5. Run an inventory scan, so paths are reconciled against the real library.
6. Mount every archive location the manifest names before archiving again. The
database records where archived originals are; it does not contain them.
Practise this against a copy before you need it.
### Watching it
`diagnostics` reports the database, write-ahead log, thumbnail cache, uploader
reports, backups, and logs separately, with free space and warnings for low disk
(`disk_low`, `disk_critical`), a cache over its quota, a write-ahead log outgrowing
its database, and a legacy CLI writing the library.
Logs go to stdout in JSON by default — `docker compose logs -f worker`, or your
service manager's journal on a host. Uploader output is kept per attempt under the
data directory with the API key scrubbed line by line.
## When it refuses to start
Every refusal below is deliberate. The application would rather stop and explain than
guess about somebody's photographs.
| exit code | meaning | what to do |
|---|---|---|
| `1` | the operation failed and said why — an unverifiable backup, an occupied restore target, an unknown benchmark profile | read the message; nothing was changed |
| `2` | **the library lock is held.** Another `serve` or `worker` owns this data directory | stop the other process. The message names the holder; a lock whose process is gone is taken over automatically |
| `3` | **a legacy CLI looks active.** The frozen command-line tools are writing this library | stop them. `--allow-legacy` overrides and you own the outcome |
| `4` | **configuration refused.** The application is reachable beyond loopback and no access secret is set | set `PHOTO_PIPELINE_ACCESS_SECRET`, or bind to loopback only |
| `5` | **a library root is not mounted** (containers only) | fix the bind mount. Refusing is what stops the container writing into its own throwaway layer instead of your library |
Other things that look like faults and are not:
- **`403 host_not_allowed`** — the `Host` header is not a name in
`PHOTO_PIPELINE_ALLOWED_HOSTS`. Add the real hostname; do not add `*`.
- **`409 lock_held`** — a mutating job is already running. One at a time is what
makes a crash recoverable.
- **`409 rename_recovery_required`** — an interrupted rename is unresolved. Resolve
it in the Renames view; unrelated mutations stay blocked until then, on purpose.
- **A job stuck in `queued`** — no worker is running.
- **An empty scan** — check `PHOTO_PIPELINE_LIBRARY_ROOTS`, and in a container check
that the path is the *container-side* mount, not the host path.
## What an installer must not work around
- **One writer.** Do not run two workers, and do not run the frozen CLI beside the
application. Each is safe alone and destructive together — the lock is not
bureaucracy.
- **`_IGNORE/` is never read.** Do not "fix" the exclusion.
- **EXIF is verified before upload.** Do not skip the checkpoint to make a stage
finish; an unverified projection is exactly the case where the wrong bytes reach
Immich.
- **The data directory is not the library.** Keep them apart, and keep the data
directory on a local filesystem.
- **Secrets stay in the environment.** Not in the database, not in a log, not in a
commit.

77
docs/overview.md Normal file
View File

@@ -0,0 +1,77 @@
# Overview
[← Documentation index](index.md)
Photo Pipeline sorts a photo library. It finds duplicates before anything expensive
happens to them, asks a human which pictures may leave the machine, describes the
ones that may, proposes album names from what it found, renames folders under a
crash-safe journal, uploads through `immich-go`, and can archive a finished album off
active storage without ever forgetting it existed.
It runs locally. It talks to exactly three things outside itself: a vision provider,
an Immich server, and `exiftool` — and it will tell you before it uses any of them.
## The workflow
Each stage is a gate, not a tab. A later stage can always be looked at; its actions
stay disabled until what they depend on is true.
```mermaid
flowchart TD
I["0 · Inventory<br/>discover, hash, cluster duplicates"] --> S["1 · Safety<br/>score and human decision"]
S -->|sfw| A["2 · Analysis<br/>vision provider, EXIF checkpoint"]
S -->|nsfw| U
A --> B["3 · Albums<br/>proposal, then guarded rename"]
B --> U["4 · Upload<br/>immich-go, verified bytes"]
U --> R["5 · Archive<br/>copy, verify, then reclaim space"]
```
| Stage | What it decides | What it changes |
|---|---|---|
| Inventory | which file is the canonical copy of a picture | nothing — it only reads |
| Safety | whether a photo may be sent to a cloud provider | one `sfw`/`nsfw` EXIF keyword |
| Analysis | what a photo shows | a managed caption segment and additive keywords |
| Albums | what a folder should be called | folder names, through a journaled rename |
| Upload | which exact bytes reach Immich | nothing locally; assets appear in Immich |
| Archive | which album leaves active storage | files move to the archive, after verification |
## What it will not do
These are enforced in code, not by convention, and each one is why some action you
expected is sometimes refused.
- **Nothing under `_IGNORE/` is ever read.** Not scanned, not counted, not
thumbnailed, not sent anywhere.
- **A path is not an identity.** Every picture has a stable id, so moving or renaming
it loses no history.
- **Only a confirmed-SFW photo reaches the vision provider.** A photo marked NSFW is
still uploadable to Immich; it simply never leaves for analysis.
- **Metadata is verified, not hoped for.** Every stage that writes EXIF reads it back
and proves that what it did not own is unchanged.
- **One writer at a time.** A library lock, held by one process, is what makes a
crash recoverable instead of ambiguous.
- **Nothing irreversible happens without a preview and an explicit approval** that
names the exact count.
## Where things live
| | |
|---|---|
| the photo library | wherever you point `PHOTO_PIPELINE_LIBRARY_ROOTS`; mounted read-write |
| the database, cache, logs, backups | the data directory, never inside the library |
| secrets | the environment, never the database and never a log line |
| these documents | `docs/` in the repository, served at `/docs` by the application |
## Running it
The short version, for a host installation:
```bash
python -m photo_pipeline migrate
python -m photo_pipeline serve # the API and this browser application
python -m photo_pipeline worker # the process that does the long work
```
The full procedure, the container path, and every setting belong to the installation
manual, which is being written next — see
[the documentation index](index.md#being-written).

View File

@@ -170,6 +170,20 @@ a.link:hover { text-decoration: underline; }
.grid th, .grid td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); vertical-align: top; }
.path { font-family: ui-monospace, monospace; font-size: 0.85em; word-break: break-all; }
/* Docs: prose, so it gets a reading measure rather than the full window width. */
.doc { max-width: 72ch; }
.doc h1 { margin-top: 0; }
.doc h2, .doc h3 { margin-top: 1.6em; border-bottom: 1px solid var(--border); padding-bottom: 4px; }
.doc code { font-family: ui-monospace, monospace; font-size: 0.9em; background: var(--surface-2); padding: 1px 4px; border-radius: 4px; }
.doc pre { background: var(--surface-2); border: 1px solid var(--border); border-radius: var(--radius); padding: 12px; overflow: auto; }
.doc pre code { background: none; padding: 0; }
.doc table { border-collapse: collapse; width: 100%; }
.doc th, .doc td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); vertical-align: top; }
.doc blockquote { margin: 1em 0; padding-left: 12px; border-left: 3px solid var(--border); color: var(--muted); }
.doc img { max-width: 100%; }
.diagram { margin: 1.5em 0; padding: 0; overflow: auto; }
.diagram svg { max-width: 100%; height: auto; }
/* Narrow screens: stack the panes so blockers and actions stay reachable. */
@media (max-width: 720px) {
.two-pane { grid-template-columns: 1fr; }

View File

@@ -21,6 +21,7 @@
<a href="#/uploads" data-nav="uploads">Upload</a>
<a href="#/archive" data-nav="archive">Archive</a>
<a href="#/stats" data-nav="stats">Stats</a>
<a href="#/docs" data-nav="docs">Docs</a>
</nav>
</header>
<main id="app" aria-live="polite"><!-- views render here --></main>

View File

@@ -1,5 +1,6 @@
import { api } from "./api.js";
import { renderArchive, setArchiveRender } from "./archive.js";
import { renderDocs } from "./docs.js";
import { navigate, onRouteChange, parseHash } from "./router.js";
import { renderRenames, setRenamesRender } from "./renames.js";
import { renderUploads, setUploadsRender } from "./uploads.js";
@@ -385,6 +386,7 @@ function render() {
else if (path === "/uploads") renderUploads(root, params);
else if (path === "/archive") renderArchive(root, params);
else if (path === "/stats") renderStats(root, params);
else if (path === "/docs") renderDocs(root, params);
else show(errorBanner("Unknown view"));
}

263
frontend/js/docs.js Normal file
View File

@@ -0,0 +1,263 @@
// The documentation view (US09-01): the manuals in `docs/`, rendered in the app.
//
// The same markdown files are the repository's documentation and the deployment's
// documentation. An operator who was handed a URL and an access secret has no
// checkout in front of them, and the network the application runs on is not assumed
// to reach a CDN — so the renderer is vendored and everything here is same-origin.
//
// Nothing on this page is authenticated. It is served from the static mount beside
// the application shell, exactly like `index.html`: the troubleshooting page is
// needed most by whoever cannot get past the access secret, and no documentation
// file contains anything a session would protect.
//
// `marked` and `mermaid` are loaded lazily, on the first documentation page and the
// first diagram. They are large, and every other view does without them.
import { el, errorBanner, setActiveNav } from "./dom.js";
const DOCS_BASE = "/docs";
const VENDOR = "/app/js/vendor";
const INDEX_PAGE = "index";
// A page name comes from the hash, so it is caller input: no scheme, no traversal,
// no absolute path. The server would refuse those too; this refuses them earlier and
// without a request that looks like an attempt.
const PAGE_PATTERN = /^[\w-]+(\/[\w-]+)*$/;
let markedPromise;
let mermaidPromise;
function loadMarked() {
markedPromise ??= import(`${VENDOR}/marked.esm.js`).then((module) => module.marked);
return markedPromise;
}
// mermaid ships one large UMD bundle rather than a self-contained ES module, so it
// arrives through a script element. `script-src 'self'` allows it because it is ours
// and same-origin; nothing here relaxes that.
function loadMermaid() {
mermaidPromise ??= new Promise((resolve, reject) => {
const script = document.createElement("script");
script.src = `${VENDOR}/mermaid.min.js`;
script.onload = () => resolve(window.mermaid);
script.onerror = () => reject(new Error("the diagram renderer could not be loaded"));
document.head.appendChild(script);
});
return mermaidPromise;
}
// ── the pure parts, unit-tested in js/tests/unit.js ──────────────────────────
/** A stable, readable heading anchor. Letters and digits of any script survive. */
export function slug(text) {
const cleaned = String(text)
.toLowerCase()
.trim()
.replace(/[^\p{L}\p{N}\s-]/gu, "")
.replace(/[\s-]+/g, "-")
.replace(/^-|-$/g, "");
return cleaned || "section";
}
/** Assign every heading an id, disambiguating repeats the way a reader would
* expect: the first `Notes` keeps `#notes`, the second becomes `#notes-2`. */
export function assignHeadingIds(headings) {
const used = new Map();
for (const heading of headings) {
const base = slug(heading.textContent);
const seen = (used.get(base) || 0) + 1;
used.set(base, seen);
heading.id = seen === 1 ? base : `${base}-${seen}`;
}
return headings;
}
/**
* Where a link inside a documentation page should go.
*
* Markdown links between documents are relative file paths, which a browser would
* treat as downloads that leave the application. Returns the in-app target for a
* link to another document or to a heading, and `null` for anything else — external
* links stay exactly as the author wrote them.
*/
export function resolveDocLink(currentPage, href) {
if (!href) return null;
if (/^[a-z][a-z\d+.-]*:/i.test(href) || href.startsWith("//")) return null;
if (href.startsWith("#")) return { page: currentPage, anchor: href.slice(1) };
const [target, anchor = ""] = href.split("#");
if (!/\.md$/i.test(target)) return null;
// Resolved against the current document, so `../guides/x.md` means what it means
// in the repository. The origin is a placeholder; only the path is used.
const resolved = new URL(target, `https://docs.invalid/${currentPage}.md`);
const page = decodeURIComponent(resolved.pathname).replace(/^\//, "").replace(/\.md$/i, "");
return PAGE_PATTERN.test(page) ? { page, anchor } : null;
}
/**
* Point every in-app link at its route, and leave every other link alone.
*
* The resolved page is kept on the element, so the reading order can be read back
* from the rendered index rather than parsed out of the markdown a second time.
*/
export function rewriteLinks(article, page) {
for (const link of article.querySelectorAll("a[href]")) {
const href = link.getAttribute("href");
const target = resolveDocLink(page, href);
if (target) {
link.setAttribute("href", docHash(target.page, target.anchor));
link.dataset.docPage = target.page;
} else if (/^https?:/i.test(href)) {
link.setAttribute("rel", "noreferrer noopener");
link.setAttribute("target", "_blank");
}
}
return article;
}
/** The reading order, taken from the index document itself — one list, in one
* place, that is equally the index on Gitea and the navigation here. */
export function documentIndex(indexElement) {
const pages = [];
for (const link of indexElement.querySelectorAll("a[data-doc-page]")) {
const page = link.dataset.docPage;
if (page !== INDEX_PAGE && !pages.some((entry) => entry.page === page)) {
pages.push({ page, title: link.textContent.trim() });
}
}
return pages;
}
// ── the view ─────────────────────────────────────────────────────────────────
async function fetchPage(page) {
const response = await fetch(`${DOCS_BASE}/${page}.md`, { headers: { accept: "text/markdown" } });
if (!response.ok) throw new Error(`documentation page unavailable: ${response.status}`);
return response.text();
}
async function toArticle(markdown, page) {
const marked = await loadMarked();
const article = el("article", { class: "doc", "data-testid": "doc", "data-page": page });
// The only innerHTML in the application, and deliberate: rendering markdown *is*
// producing HTML. The input is a file from this repository, and the page's CSP
// (`script-src 'self'`, no `'unsafe-inline'`) means injected script and inline
// handlers do not run even if one ever were not.
article.innerHTML = marked.parse(markdown, { async: false });
assignHeadingIds(article.querySelectorAll("h1, h2, h3, h4, h5, h6"));
rewriteLinks(article, page);
return article;
}
function docHash(page, anchor = "") {
const query = new URLSearchParams(anchor ? { page, anchor } : { page });
return `#/docs?${query}`;
}
/** Diagrams are authored as ```mermaid blocks so the source is diffable and Gitea
* renders them natively. A failure here replaces the diagram, never the page. */
async function renderDiagrams(article) {
const blocks = [...article.querySelectorAll("pre > code.language-mermaid")];
if (!blocks.length) return;
let mermaid;
try {
mermaid = await loadMermaid();
mermaid.initialize({ startOnLoad: false, securityLevel: "strict", theme: "dark" });
} catch (error) {
blocks.forEach((block) => block.closest("pre").replaceWith(errorBanner(error.message)));
return;
}
for (const [index, block] of blocks.entries()) {
const figure = el("figure", { class: "diagram", "data-testid": "diagram" });
block.closest("pre").replaceWith(figure);
try {
const { svg } = await mermaid.render(`diagram-${index}-${Date.now()}`, block.textContent);
figure.innerHTML = svg;
} catch (error) {
figure.replaceWith(errorBanner(`Diagram could not be drawn: ${error.message}`));
}
}
}
function notFound() {
return el(
"div",
{ class: "alert", role: "alert", "data-testid": "doc-not-found" },
"That documentation page does not exist. ",
el("a", { class: "link", href: docHash(INDEX_PAGE) }, "Back to the documentation index")
);
}
function sidebar(pages, current) {
return el(
"nav",
{ class: "card", "aria-label": "Documentation" },
el("h2", {}, "Documentation"),
el(
"ul",
{ class: "album-list", "data-testid": "doc-pages" },
el(
"li",
{},
el(
"a",
{
href: docHash(INDEX_PAGE),
"aria-current": current === INDEX_PAGE ? "true" : false,
},
"Index"
)
),
...pages.map(({ page, title }) =>
el(
"li",
{},
el(
"a",
{
href: docHash(page),
"data-page": page,
"aria-current": page === current ? "true" : false,
},
title
)
)
)
)
);
}
export async function renderDocs(root, params = {}) {
setActiveNav("docs");
const requested = params.page || INDEX_PAGE;
const page = PAGE_PATTERN.test(requested) ? requested : "";
let indexArticle;
try {
indexArticle = await toArticle(await fetchPage(INDEX_PAGE), INDEX_PAGE);
} catch (error) {
root.replaceChildren(errorBanner(`Documentation is unavailable: ${error.message}`));
return;
}
const pages = documentIndex(indexArticle);
let article;
if (!page) article = notFound();
else if (page === INDEX_PAGE) article = indexArticle;
else {
try {
article = await toArticle(await fetchPage(page), page);
} catch {
article = notFound();
}
}
root.replaceChildren(el("div", { class: "two-pane" }, sidebar(pages, page), article));
await renderDiagrams(article);
scrollToAnchor(params.anchor);
}
// A documentation anchor cannot live in the hash — the hash is the route — so it
// travels as a parameter and is applied after the page renders.
function scrollToAnchor(anchor) {
if (!anchor) return;
const target = document.getElementById(anchor);
if (target) target.scrollIntoView({ block: "start" });
}

View File

@@ -7,6 +7,7 @@ import { createStore } from "../store.js";
import { parseHash, navigate } from "../router.js";
import { api, cancellable } from "../api.js";
import { subscribeJob } from "../events.js";
import { assignHeadingIds, documentIndex, resolveDocLink, rewriteLinks, slug } from "../docs.js";
const cases = [];
function ok(name, cond) {
@@ -142,6 +143,62 @@ async function run() {
window.EventSource = realES;
}
// ── documentation view (US09-01) ─────────────────────────────────────────
{
ok("slug lowercases and dashes a heading", slug("Key Flows") === "key-flows");
ok(
"slug drops punctuation but keeps words",
slug("What it *will not* do:") === "what-it-will-not-do"
);
ok("slug keeps non-ASCII letters", slug("Größe & Gewicht") === "größe-gewicht");
ok("slug never yields an empty anchor", slug("!!!") === "section");
const doc = document.createElement("div");
doc.innerHTML = "<h2>Notes</h2><h3>Notes</h3><h2>Notes</h2>";
const ids = [...assignHeadingIds(doc.querySelectorAll("h2, h3"))].map((h) => h.id);
ok("repeated headings get distinct anchors", ids.join(",") === "notes,notes-2,notes-3");
ok(
"a sibling document link becomes an in-app route",
resolveDocLink("index", "overview.md")?.page === "overview"
);
const nested = resolveDocLink("guides/install", "../overview.md#running-it");
ok("a relative link resolves against the current page", nested?.page === "overview");
ok("a link's anchor survives the rewrite", nested?.anchor === "running-it");
ok(
"a bare anchor stays on the current page",
resolveDocLink("overview", "#the-workflow")?.page === "overview"
);
ok("an external link is left alone", resolveDocLink("index", "https://example.test/x") === null);
ok("a non-markdown relative link is left alone", resolveDocLink("index", "images/a.png") === null);
// A climb cannot leave the documentation tree: it is clamped at the root, so the
// page it names is still fetched from under /docs and simply does not exist.
ok(
"a link that climbs above the docs root is clamped",
resolveDocLink("index", "../../etc/passwd.md")?.page === "etc/passwd"
);
ok(
"a page name that is not a page name is refused",
resolveDocLink("index", "..%2f..%2fetc%2fpasswd.md") === null
);
const index = document.createElement("div");
index.innerHTML =
'<a href="overview.md">Overview</a><a href="overview.md">again</a>' +
'<a href="index.md">itself</a><a href="https://example.test">out</a>';
const listed = documentIndex(rewriteLinks(index, "index"));
ok(
"an external link is marked safe to open away from the app",
index.querySelector('a[href^="https"]').rel === "noreferrer noopener"
);
ok(
"an in-app link points at the docs route",
index.querySelector("a").getAttribute("href") === "#/docs?page=overview"
);
ok("the index lists each page once, in order", listed.length === 1);
ok("the index takes its titles from the link text", listed[0].title === "Overview");
}
await tick();
const failed = cases.filter((c) => !c.ok);
window.__RESULTS__ = { passed: cases.length - failed.length, failed: failed.length, cases };

23
frontend/js/vendor/VERSIONS.json vendored Normal file
View File

@@ -0,0 +1,23 @@
{
"_comment": "Vendored browser libraries (US09-01). Committed rather than fetched: the application is deployed to a network whose outbound access is not assumed, and `default-src 'self'` forbids a CDN. Checksums are asserted by tests/integration/test_documentation.py, so replacing a file without recording it here fails the suite. Update by downloading the pinned URL and recording the new version and sha256 in the same commit.",
"libraries": [
{
"file": "marked.esm.js",
"name": "marked",
"version": "18.0.10",
"license": "MIT",
"url": "https://cdn.jsdelivr.net/npm/marked@18.0.10/lib/marked.esm.js",
"sha256": "4cf47dfebb7f614a08fc0a579ab0fe407ff0ed2b717bf953040c85b2f493a4f0",
"why": "Markdown to HTML for the documentation view. An ES module with no dependencies, imported lazily by js/docs.js."
},
{
"file": "mermaid.min.js",
"name": "mermaid",
"version": "11.17.0",
"license": "MIT",
"url": "https://cdn.jsdelivr.net/npm/mermaid@11.17.0/dist/mermaid.min.js",
"sha256": "8d8e0eec56d3a83b4b3c87f42050845546dee93ebe1875d2117c12e6947c0cb3",
"why": "Renders ```mermaid blocks so diagrams are diffable source that Gitea also renders. The single-file UMD build rather than the ES module entry, whose ~40 lazy chunks would each need vendoring and pinning."
}
]
}

78
frontend/js/vendor/marked.esm.js vendored Normal file

File diff suppressed because one or more lines are too long

3636
frontend/js/vendor/mermaid.min.js vendored Normal file

File diff suppressed because one or more lines are too long

View File

@@ -1,5 +1,5 @@
"""Application management CLI:
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics}``.
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,dry-run,approve-dry-run}``.
``serve`` and ``worker`` take the library process lock for their role (US07-05):
two workers, or the frozen CLI running beside the app, would each be safe on their
@@ -70,6 +70,14 @@ def main(argv: Sequence[str] | None = None) -> int:
"release-gate", help="Run every suite in an isolated stack and keep the evidence"
)
gate_cmd.add_argument("--output", help="Evidence directory (default: data/release/<stamp>)")
container_cmd = commands.add_parser(
"container-gate",
help="Provision the composition from the built image, run the phase_h "
"acceptance suite against it, destroy it, and keep the evidence (US08-05)",
)
container_cmd.add_argument(
"--output", help="Evidence directory (default: data/container-gate/<stamp>)"
)
dry_cmd = commands.add_parser(
"dry-run", help="Read-only reconciliation of the configured library (US07-07)"
)
@@ -155,6 +163,31 @@ def main(argv: Sequence[str] | None = None) -> int:
)
return 0 if report["ok"] else 1
if args.command == "container-gate":
from datetime import datetime, timezone
from photo_pipeline.services import release
# The suite provisions and destroys the composition itself; what this command
# adds is the single entry point and the retained evidence. No skip is an
# environment limit here: a gate that did not reach the containers proved
# nothing about them.
output = args.output or Path(config.data_dir) / "container-gate" / datetime.now(
timezone.utc
).strftime("%Y%m%dT%H%M%SZ")
report = release.run_gate(
config,
output=output,
stages=release.CONTAINER_STAGES,
allowed_skips=release.CONTAINER_ALLOWED_SKIP_REASONS,
)
print(
json.dumps(
{k: v for k, v in report.items() if k not in ("stages", "matrix")}, indent=2
)
)
return 0 if report["ok"] else 1
if args.command == "dry-run":
from photo_pipeline.services import release

View File

@@ -53,6 +53,7 @@ from photo_pipeline.services.thumbnails import ThumbnailService
from photo_pipeline.services.upload_batches import UploadBatchService
FRONTEND_DIR = Path(__file__).resolve().parents[2] / "frontend"
DOCS_DIR = Path(__file__).resolve().parents[2] / "docs"
log = logging.getLogger(__name__)
@@ -151,4 +152,10 @@ def create_app(config: Config | None = None) -> FastAPI:
# Static single-page app (hash-routed). Mounted last so /api/v1 wins.
if FRONTEND_DIR.is_dir():
app.mount("/app", StaticFiles(directory=FRONTEND_DIR, html=True), name="app")
# The manuals, as their own markdown files (US09-01). Unauthenticated for the
# same reason the application shell is: whoever cannot get past the access
# secret is exactly who needs the troubleshooting page, and no document holds
# anything a session would protect.
if DOCS_DIR.is_dir():
app.mount("/docs", StaticFiles(directory=DOCS_DIR), name="docs")
return app

View File

@@ -57,9 +57,21 @@ LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "::1", "[::1]"})
# backup a careful operator takes first, and its retention (US07-07).
MUTATION_EXEMPT_PATHS = frozenset({f"{API_PREFIX}/backups", f"{API_PREFIX}/backups/prune"})
# Applied to every response. No inline script/style is used by the frontend, so the
# policy can stay strict; `frame-ancestors 'none'` and CORP keep other pages from
# Applied to every response. `frame-ancestors 'none'` and CORP keep other pages from
# embedding the app or its thumbnails.
#
# `script-src 'self'` is the boundary that matters and it is unchanged: no
# `'unsafe-eval'`, no `'unsafe-inline'`, so nothing injected into the DOM can execute.
# Whether the vendored diagram renderer needed `'unsafe-eval'` was measured rather
# than assumed — mermaid's bundle contains no `eval(` and no `new Function`, and it
# renders under this exact policy without a single script-src violation.
#
# `style-src` does gain `'unsafe-inline'` (US09-01): mermaid styles the SVG it builds
# with an injected `<style>` element and `style=` attributes, and a diagram's CSS
# cannot be hashed in advance. The concession is bounded by the directives around it
# — with script execution still refused and `img-src`, `connect-src`, and `font-src`
# all `'self'`, the CSS exfiltration channels stay closed and what is left is
# defacement of a page its own operator is already looking at.
DEFAULT_HEADERS = {
"x-content-type-options": "nosniff",
"x-frame-options": "DENY",
@@ -67,8 +79,9 @@ DEFAULT_HEADERS = {
"cross-origin-resource-policy": "same-origin",
"cross-origin-opener-policy": "same-origin",
"content-security-policy": (
"default-src 'self'; img-src 'self' data:; style-src 'self'; script-src 'self'; "
"connect-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
"default-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; "
"script-src 'self'; connect-src 'self'; font-src 'self'; "
"frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
),
}

View File

@@ -60,6 +60,17 @@ ALLOWED_SKIP_REASONS = (
"bind-mount ownership is virtualised",
)
# The container acceptance gate (US08-05): the deployed application, verified the way
# the host application is. Deliberately one stage selected by marker, so adding a
# phase_h test is enough to put it in front of a deploy.
CONTAINER_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
("container", ("tests/e2e", "-m", "phase_h")),
)
# Nothing. This gate exists to prove the *deployed* runtime, and every reason a check
# would skip here — no daemon, no compose, no browser — means it was not proven.
CONTAINER_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
class ReleaseError(RuntimeError):
pass
@@ -175,13 +186,15 @@ def _run_stage(label: str, paths: tuple[str, ...], *, repo: Path, log_dir: Path)
return StageResult(label, command, result.returncode, elapsed, summary, skipped)
def unexpected_skips(results: list[StageResult]) -> list[str]:
def unexpected_skips(
results: list[StageResult], allowed: tuple[str, ...] = ALLOWED_SKIP_REASONS
) -> list[str]:
"""Skips the gate will not accept: everything but the documented environment ones."""
return [
line
for result in results
for line in result.skipped
if not any(reason in line for reason in ALLOWED_SKIP_REASONS)
if not any(reason in line for reason in allowed)
]
@@ -190,6 +203,7 @@ def run_gate(
*,
output: Path | str | None = None,
stages: tuple[tuple[str, tuple[str, ...]], ...] = STAGES,
allowed_skips: tuple[str, ...] = ALLOWED_SKIP_REASONS,
repo: Path | None = None,
) -> dict:
"""Run every suite in an isolated stack and retain checksummed evidence.
@@ -206,13 +220,14 @@ def run_gate(
logs = directory / "logs"
logs.mkdir(parents=True, exist_ok=True)
started_at = _now()
matrix = story_matrix(repo)
results = [_run_stage(label, paths, repo=repo, log_dir=logs) for label, paths in stages]
skips = unexpected_skips(results)
skips = unexpected_skips(results, allowed_skips)
report = {
"schema_version": SCHEMA_VERSION,
"started_at": _now().isoformat(),
"started_at": started_at.isoformat(),
"revision": revision(),
"python": sys.version.split()[0],
"platform": os.uname().sysname,

View File

@@ -53,4 +53,5 @@ markers = [
"phase_e: Phase E end-to-end acceptance (US05-06) — upload preflight, uploader, and browser journeys",
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
"container: builds and runs the container image and its composition (US08-02, US08-03) — needs a Docker daemon, the compose plugin, and the network",
"phase_h: Phase H container deployment acceptance (US08-05) — the browser, upgrade, restart, and security journeys against the composed stack",
]

View File

@@ -0,0 +1,260 @@
"""The composed stack, as a fixture: build, provision, drive, destroy (US08-03/US08-05).
Extracted from ``tests/e2e/test_compose_stack.py`` when the container acceptance gate
needed the same stack under a different image, project, and library. One
implementation, because two would drift on exactly the details that make a container
test worth anything — the mount, the volume, the ports, and the teardown.
Nothing here fakes anything below the process boundary: Docker builds the image,
Compose starts the real containers, and every helper talks to them over HTTP or the
Docker CLI.
"""
from __future__ import annotations
import os
import shutil
import socket
import subprocess
import tempfile
import time
from pathlib import Path
import httpx
import pytest
from PIL import Image
REPO = Path(__file__).resolve().parents[2]
COMPOSE_FILE = REPO / "docker-compose.yml"
CONTAINER_LIBRARY = "/library"
READY_TIMEOUT_SECONDS = 180
JOB_TIMEOUT_SECONDS = 300
UP_TIMEOUT_SECONDS = 30 * 60
def compose_available() -> bool:
try:
return (
subprocess.run(
["docker", "compose", "version"], capture_output=True, timeout=60
).returncode
== 0
)
except (OSError, subprocess.SubprocessError):
return False
def free_port() -> int:
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
def mount_base() -> Path:
"""A directory a Docker VM shares with the host.
Not pytest's ``tmp_path``: on macOS that is ``/var/folders/...``, which a Docker VM
(Colima, Docker Desktop) does not share, so the bind mount would arrive empty and
every assertion would be about nothing. ``$HOME`` is shared by every default
configuration.
"""
base = Path(
os.environ.get("PHOTO_PIPELINE_TEST_MOUNT_BASE", Path.home() / ".cache" / "photo-pipeline")
)
base.mkdir(parents=True, exist_ok=True)
return base
def temporary_library(albums: dict[str, int], *, prefix: str = "library-") -> Path:
"""A fixture library on a shareable host path, with the exclusion sentinel in it."""
root = Path(tempfile.mkdtemp(prefix=prefix, dir=mount_base()))
for position, (album, count) in enumerate(albums.items()):
(root / album).mkdir(parents=True, exist_ok=True)
for index in range(count):
# Distinct per album *and* index: two solid images of the same colour are
# byte-identical, which would make them a duplicate cluster by accident.
colour = (17 + 7 * index, 31 + 29 * position, 160 - 3 * index)
Image.new("RGB", (64, 48), colour).save(root / album / f"{album}_{index}.jpg")
# Never discovered, counted, analyzed, or uploaded — asserted from outside it.
(root / "_IGNORE").mkdir(exist_ok=True)
Image.new("RGB", (32, 32), (0, 0, 0)).save(root / "_IGNORE" / "sentinel.jpg")
return root
def remove_library(root: Path) -> None:
shutil.rmtree(root, ignore_errors=True)
def write_env_file(path: Path, secret: str, extra: dict[str, str] | None = None) -> Path:
"""Configuration and secrets come from the environment, so a test writes its own
file rather than borrowing the operator's ``.env``."""
lines = [
f"PHOTO_PIPELINE_ACCESS_SECRET={secret}",
"PHOTO_PIPELINE_LOG_FORMAT=text",
# The deterministic vision seam, in the data volume so both roles and the test
# can read it (concept §18).
"PHOTO_PIPELINE_FAKE_VISION_LOG=/data/vision.log",
*(f"{key}={value}" for key, value in (extra or {}).items()),
]
path.write_text("\n".join(lines) + "\n")
return path
class Stack:
"""The composition under test, plus the environment it was started with."""
def __init__(
self,
library: Path,
env_file: Path,
*,
project: str,
image: str,
secret: str,
) -> None:
self.library = library
self.project = project
self.secret = secret
self.port = free_port()
self.base = f"http://127.0.0.1:{self.port}"
# Compose reads the repository's own .env for substitution; the process
# environment wins over it, so the test's values are the ones that apply.
self.env = {
**os.environ,
"PHOTO_PIPELINE_IMAGE": image,
"PHOTO_PIPELINE_ENV_FILE": str(env_file),
"PHOTO_PIPELINE_LIBRARY_HOST_PATH": str(library),
"PHOTO_PIPELINE_LIBRARY_ROOTS": CONTAINER_LIBRARY,
"PHOTO_PIPELINE_PORT": str(self.port),
"PHOTO_PIPELINE_UID": str(os.getuid()),
"PHOTO_PIPELINE_GID": str(os.getgid()),
}
# ── the compose lifecycle ────────────────────────────────────────────────
def compose(self, *args: str, check: bool = True, timeout: int = 300):
result = subprocess.run(
["docker", "compose", "-p", self.project, "-f", str(COMPOSE_FILE), *args],
capture_output=True,
text=True,
env=self.env,
cwd=REPO,
timeout=timeout,
)
if check and result.returncode != 0:
raise AssertionError(
f"docker compose {' '.join(args)} failed:\n{result.stdout}\n{result.stderr}\n"
f"{self.compose('logs', '--tail', '80', check=False).stdout}"
)
return result
def up(self, *extra: str) -> None:
self.compose("up", "--detach", *extra, timeout=UP_TIMEOUT_SECONDS)
def down(self, *, volumes: bool = True) -> None:
self.compose(
"down",
*(("--volumes",) if volumes else ()),
"--remove-orphans",
check=False,
timeout=300,
)
def use_image(self, image: str) -> None:
"""Point the composition at another tag — the upgrade path (US08-05)."""
self.env["PHOTO_PIPELINE_IMAGE"] = image
def logs(self, *services: str) -> str:
result = self.compose("logs", *services, check=False)
return result.stdout + result.stderr
def wait_until_ready(self) -> None:
deadline = time.monotonic() + READY_TIMEOUT_SECONDS
while time.monotonic() < deadline:
try:
if httpx.get(f"{self.base}/api/v1/health/ready", timeout=5).status_code == 200:
return
except httpx.HTTPError:
pass
time.sleep(0.5)
raise AssertionError(f"the stack never became ready:\n{self.logs()}")
# ── talking to it ────────────────────────────────────────────────────────
def client(self) -> httpx.Client:
"""A browser that has loaded the app: session cookie in the jar, token in a
header. The session belongs to the API process, so it is re-bootstrapped after
every restart."""
client = httpx.Client(base_url=f"{self.base}/api/v1", timeout=60)
bootstrap = client.get("/session", headers={"X-Access-Secret": self.secret})
assert bootstrap.status_code == 200, bootstrap.text
client.headers["X-CSRF-Token"] = bootstrap.json()["csrf_token"]
return client
def await_job(client: httpx.Client, job_id: str, states=("succeeded",)) -> dict:
deadline = time.monotonic() + JOB_TIMEOUT_SECONDS
snapshot: dict = {}
while time.monotonic() < deadline:
response = client.get(f"/jobs/{job_id}")
if response.status_code == 200:
snapshot = response.json()
if snapshot["state"] in states:
return snapshot
time.sleep(0.5)
raise AssertionError(f"job {job_id} never reached {states}: {snapshot}")
def build_image(tag: str, *, revision: str | None = None) -> str:
"""Build the application image; from a git revision's tree when one is named.
``revision`` is how the upgrade journey gets the *previous* version without a
registry: the tree of that commit is the build context, so what it produces is the
image that commit would have published.
"""
if revision is None:
subprocess.run(
[
"docker",
"build",
"--build-arg",
f"UID={os.getuid()}",
"--build-arg",
f"GID={os.getgid()}",
"-t",
tag,
str(REPO),
],
check=True,
timeout=UP_TIMEOUT_SECONDS,
)
return tag
archive = subprocess.run(
["git", "archive", "--format=tar", revision],
cwd=REPO,
capture_output=True,
check=True,
timeout=300,
).stdout
subprocess.run(
[
"docker",
"build",
"--build-arg",
f"UID={os.getuid()}",
"--build-arg",
f"GID={os.getgid()}",
"-t",
tag,
"-",
],
input=archive,
check=True,
timeout=UP_TIMEOUT_SECONDS,
)
return tag
needs_compose = pytest.mark.skipif(
not compose_available(), reason="no Docker daemon with the compose plugin"
)

View File

@@ -10,188 +10,63 @@ what this story is about — the mount, the lock, the volume, and the restart ar
The file contract (one API, one worker, migrations first, no committed values) is
checked without a daemon in ``tests/integration/test_compose_runtime.py``; only the
running proof needs Docker, and CI is where it runs unskipped (US08-04).
The stack itself lives in ``tests/e2e/_container_harness.py``, shared with the
container acceptance gate (US08-05).
"""
from __future__ import annotations
import json
import os
import shutil
import socket
import subprocess
import sys
import tempfile
import time
from pathlib import Path
import httpx
import pytest
from PIL import Image
REPO = Path(__file__).resolve().parents[2]
from tests.e2e._container_harness import (
CONTAINER_LIBRARY,
Stack,
await_job,
compose_available,
needs_compose,
remove_library,
temporary_library,
write_env_file,
)
PROJECT = "photo-pipeline-us0803"
IMAGE = "photo-pipeline-test:us08-03"
SECRET = "compose-acceptance-secret"
CONTAINER_LIBRARY = "/library"
READY_TIMEOUT_SECONDS = 180
JOB_TIMEOUT_SECONDS = 180
UP_TIMEOUT_SECONDS = 30 * 60
pytestmark = pytest.mark.container
def compose_available() -> bool:
try:
return (
subprocess.run(
["docker", "compose", "version"], capture_output=True, timeout=60
).returncode
== 0
)
except (OSError, subprocess.SubprocessError):
return False
needs_compose = pytest.mark.skipif(
not compose_available(), reason="no Docker daemon with the compose plugin"
)
def free_port() -> int:
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
class Stack:
"""The composition under test, plus the environment it was started with."""
def __init__(self, library: Path, env_file: Path) -> None:
self.library = library
self.port = free_port()
self.base = f"http://127.0.0.1:{self.port}"
# Compose reads the repository's own .env for substitution; the process
# environment wins over it, so the test's values are the ones that apply.
self.env = {
**os.environ,
"PHOTO_PIPELINE_IMAGE": IMAGE,
"PHOTO_PIPELINE_ENV_FILE": str(env_file),
"PHOTO_PIPELINE_LIBRARY_HOST_PATH": str(library),
"PHOTO_PIPELINE_LIBRARY_ROOTS": CONTAINER_LIBRARY,
"PHOTO_PIPELINE_PORT": str(self.port),
"PHOTO_PIPELINE_UID": str(os.getuid()),
"PHOTO_PIPELINE_GID": str(os.getgid()),
}
def compose(self, *args: str, check: bool = True, timeout: int = 300):
result = subprocess.run(
["docker", "compose", "-p", PROJECT, "-f", str(REPO / "docker-compose.yml"), *args],
capture_output=True,
text=True,
env=self.env,
cwd=REPO,
timeout=timeout,
)
if check and result.returncode != 0:
raise AssertionError(
f"docker compose {' '.join(args)} failed:\n{result.stdout}\n{result.stderr}\n"
f"{self.compose('logs', '--tail', '80', check=False).stdout}"
)
return result
def wait_until_ready(self) -> None:
deadline = time.monotonic() + READY_TIMEOUT_SECONDS
while time.monotonic() < deadline:
try:
if httpx.get(f"{self.base}/api/v1/health/ready", timeout=5).status_code == 200:
return
except httpx.HTTPError:
pass
time.sleep(0.5)
logs = self.compose("logs", "--tail", "120", check=False)
raise AssertionError(f"the stack never became ready:\n{logs.stdout}\n{logs.stderr}")
def client(self) -> httpx.Client:
client = httpx.Client(base_url=f"{self.base}/api/v1", timeout=60)
bootstrap = client.get("/session", headers={"X-Access-Secret": SECRET})
assert bootstrap.status_code == 200, bootstrap.text
client.headers["X-CSRF-Token"] = bootstrap.json()["csrf_token"]
return client
@pytest.fixture(scope="module")
def library() -> Path:
"""A small fixture library on the host, mounted into both containers.
Not under pytest's ``tmp_path``: on macOS that is ``/var/folders/...``, which a
Docker VM (Colima, Docker Desktop) does not share, so the bind mount would arrive
empty and every assertion below would be about nothing. ``$HOME`` is shared by
every default configuration.
"""
base = Path(
os.environ.get("PHOTO_PIPELINE_TEST_MOUNT_BASE", Path.home() / ".cache" / "photo-pipeline")
)
base.mkdir(parents=True, exist_ok=True)
root = Path(tempfile.mkdtemp(prefix="library-", dir=base))
for album, count in (("01_day", 2), ("02_night", 1)):
(root / album).mkdir()
for index in range(count):
colour = (40 * (index + 1), 90, 160)
Image.new("RGB", (64, 48), colour).save(root / album / f"{album}_{index}.jpg")
# The exclusion sentinel: it must never be discovered, counted, or analyzed.
(root / "_IGNORE").mkdir()
Image.new("RGB", (32, 32), (0, 0, 0)).save(root / "_IGNORE" / "sentinel.jpg")
root = temporary_library({"01_day": 2, "02_night": 1})
try:
yield root
finally:
shutil.rmtree(root, ignore_errors=True)
remove_library(root)
@pytest.fixture(scope="module")
def env_file(tmp_path_factory) -> Path:
"""Configuration and secrets come from the environment, so the test writes its
own file rather than borrowing the operator's."""
path = tmp_path_factory.mktemp("config") / "compose.env"
path.write_text(
"\n".join(
[
f"PHOTO_PIPELINE_ACCESS_SECRET={SECRET}",
"PHOTO_PIPELINE_LOG_FORMAT=text",
# The deterministic vision seam, in the data volume so both roles and
# the test can see it (concept §18).
"PHOTO_PIPELINE_FAKE_VISION_LOG=/data/vision.log",
]
)
+ "\n"
)
return path
return write_env_file(tmp_path_factory.mktemp("config") / "compose.env", SECRET)
@pytest.fixture(scope="module")
def stack(library, env_file):
if not compose_available():
pytest.skip("no Docker daemon with the compose plugin")
running = Stack(library, env_file)
running.compose("down", "--volumes", "--remove-orphans", check=False)
running.compose("up", "--detach", "--build", timeout=UP_TIMEOUT_SECONDS)
running = Stack(library, env_file, project=PROJECT, image=IMAGE, secret=SECRET)
running.down()
running.up("--build")
try:
running.wait_until_ready()
yield running
finally:
running.compose("down", "--volumes", "--remove-orphans", check=False, timeout=300)
def await_job(client: httpx.Client, job_id: str, states=("succeeded",)) -> dict:
deadline = time.monotonic() + JOB_TIMEOUT_SECONDS
snapshot: dict = {}
while time.monotonic() < deadline:
response = client.get(f"/jobs/{job_id}")
if response.status_code == 200:
snapshot = response.json()
if snapshot["state"] in states:
return snapshot
time.sleep(0.5)
raise AssertionError(f"job {job_id} never reached {states}: {snapshot}")
running.down()
# ── the mounted library ──────────────────────────────────────────────────────
@@ -296,8 +171,8 @@ def test_the_data_volume_survives_recreating_the_containers(stack):
finally:
client.close()
stack.compose("down", "--remove-orphans", timeout=300)
stack.compose("up", "--detach", timeout=UP_TIMEOUT_SECONDS)
stack.down(volumes=False)
stack.up()
stack.wait_until_ready()
client = stack.client()

149
tests/e2e/test_docs_ui.py Normal file
View File

@@ -0,0 +1,149 @@
"""US09-01: the manuals, in the browser, under the application's real policy.
Everything here runs against a real ``photo_pipeline serve`` process serving the
committed ``docs/`` tree and the vendored renderer — no fixture markdown, no stubbed
fetch. What that buys is the assertion the story actually cares about: the pages
render, the links between them work, a diagram becomes a diagram, and the console
stays empty, including of CSP violations.
The offline contract — reachability, dead links, pinned checksums — is
``tests/integration/test_documentation.py``.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from tests.e2e._pipeline_harness import Server, seed_library
@pytest.fixture(scope="module")
def server(tmp_path_factory):
seeded = seed_library(tmp_path_factory.mktemp("docs"), {"a": 1}, {})
running = Server(seeded).start()
try:
yield running
finally:
running.stop()
@pytest.fixture
def quiet(page):
"""Any console error, page error, or failed request fails the test that caused
it. A CSP violation arrives as a console error, which is the point."""
problems: list[str] = []
page.on("console", lambda m: problems.append(m.text) if m.type == "error" else None)
page.on("pageerror", lambda error: problems.append(str(error)))
page.on("requestfailed", lambda request: problems.append(f"failed request: {request.url}"))
yield problems
def test_the_documentation_opens_from_the_navigation(page, server, quiet):
page.goto(f"{server.base}/app/#/workflow")
page.locator('nav a[data-nav="docs"]').click()
page.get_by_test_id("doc").wait_for()
assert page.get_by_test_id("doc").get_attribute("data-page") == "index"
# The sidebar is the index's own reading order, not a second list to maintain.
assert page.get_by_test_id("doc-pages").get_by_role("link", name="Overview").is_visible()
assert quiet == []
def test_a_link_between_documents_stays_inside_the_application(page, server, quiet):
page.goto(f"{server.base}/app/#/docs")
page.get_by_test_id("doc").wait_for()
page.get_by_test_id("doc").get_by_role("link", name="Overview").click()
page.wait_for_selector('[data-testid="doc"][data-page="overview"]')
assert "#/docs?page=overview" in page.url, "a relative .md link must not leave the app"
# And back again, by the link the document itself carries.
page.get_by_test_id("doc").get_by_role("link", name="← Documentation index").click()
page.wait_for_selector('[data-testid="doc"][data-page="index"]')
assert quiet == []
def test_a_deep_link_to_a_heading_lands_on_that_heading(page, server, quiet):
page.goto(f"{server.base}/app/#/docs?page=overview&anchor=the-workflow")
heading = page.locator("#the-workflow")
heading.wait_for()
assert heading.inner_text().strip() == "The workflow"
assert heading.evaluate("node => node.getBoundingClientRect().top < window.innerHeight")
assert quiet == []
def test_a_mermaid_block_becomes_a_diagram_under_the_unchanged_script_policy(page, server, quiet):
page.goto(f"{server.base}/app/#/docs?page=overview")
diagram = page.get_by_test_id("diagram").first
diagram.wait_for()
# A real drawing, not the source text and not an error node.
assert diagram.locator("svg").count() == 1
assert diagram.evaluate("node => node.querySelector('svg').getBBox().width") > 100
assert page.locator("pre code.language-mermaid").count() == 0
assert quiet == [], "rendering a diagram must not violate the policy"
policy = page.evaluate(
"async () => (await fetch('/app/')).headers.get('content-security-policy')"
)
assert "script-src 'self';" in policy
assert "unsafe-eval" not in policy
def test_every_diagram_in_the_architecture_overview_draws(page, server, quiet):
"""Five diagrams, three of them state machines (US09-03). A mermaid block with a
syntax error renders an error node instead of throwing, so a page that merely
loaded proves nothing — count the drawings and read the source for the states."""
source = (Path(__file__).resolve().parents[2] / "docs" / "architecture.md").read_text()
expected = source.count("```mermaid")
assert expected >= 5, "the overview lost its diagrams"
page.goto(f"{server.base}/app/#/docs?page=architecture")
page.get_by_test_id("diagram").first.wait_for()
page.wait_for_function(
"count => document.querySelectorAll('[data-testid=\"diagram\"] svg').length === count",
arg=expected,
)
drawn = page.locator('[data-testid="diagram"] svg')
assert drawn.count() == expected
for index in range(expected):
diagram = drawn.nth(index)
# Labels may be <text> or HTML inside a <foreignObject> depending on the
# diagram type, so ask for the rendered text rather than a specific element.
labels = diagram.evaluate("node => node.textContent.trim().length")
assert labels > 0, f"diagram {index} drew no labels"
assert diagram.locator(".error-icon, .error-text").count() == 0, f"diagram {index} errored"
assert page.locator("pre code.language-mermaid").count() == 0
assert quiet == []
def test_an_unknown_page_says_so_without_naming_a_path(page, server, quiet):
page.goto(f"{server.base}/app/#/docs?page=no-such-manual")
message = page.get_by_test_id("doc-not-found")
message.wait_for()
text = message.inner_text()
assert "does not exist" in text
assert "/" not in text.replace("Back to the documentation index", ""), text
message.get_by_role("link").click()
page.wait_for_selector('[data-testid="doc"][data-page="index"]')
# The missing page is a 404 and the browser says so; nothing else may go wrong,
# and in particular the view must not throw on the way to its own message.
assert all("404" in problem for problem in quiet), quiet
def test_the_documentation_is_readable_without_a_session(page, server, quiet):
"""The troubleshooting page is needed most by whoever is locked out."""
page.goto(f"{server.base}/app/#/docs")
page.get_by_test_id("doc").wait_for()
unauthenticated = page.evaluate(
"""async () => {
const response = await fetch('/docs/index.md', { credentials: 'omit' });
return { status: response.status, length: (await response.text()).length };
}"""
)
assert unauthenticated["status"] == 200
assert unauthenticated["length"] > 100

View File

@@ -0,0 +1,510 @@
"""Phase H — the container acceptance gate (US08-05).
The deployed application is verified the way the host application is: real image,
real composition, real browser, real restarts. Everything below runs against
containers that this suite provisions from the built image, against a temporary
fixture library on a bind mount and an isolated data volume, and destroys afterwards.
Four journeys, one per thing a deployment can get wrong:
* **the browser journey** — discovery, duplicate review, analysis, album proposal,
rename, upload preflight, and archive, driven through the containerized frontend;
* **the upgrade journey** — the previous version's image runs first, then this one,
and the database, its migrations, the rename journal, the job history, and the
thumbnail cache have to still be there;
* **the restart journey** — both containers are killed mid-job and the work resumes
without doing anything twice;
* **the security gates** — the refusals a loopback deployment made are still made
behind a published port: no session, forged forwarded headers, a path that leaves
the mounted library, and a secret in the logs.
Run it as one command, with evidence: ``python -m photo_pipeline container-gate``.
"""
from __future__ import annotations
import collections
from contextlib import closing
import json
import os
import shutil
import subprocess
import sys
import time
from pathlib import Path
import httpx
import pytest
from playwright.sync_api import expect
from photo_pipeline.db import head_revision
from tests.e2e._container_harness import (
Stack,
await_job,
build_image,
compose_available,
needs_compose,
remove_library,
temporary_library,
write_env_file,
)
pytestmark = [pytest.mark.phase_h, pytest.mark.container]
PROJECT = "photo-pipeline-us0805"
IMAGE = "photo-pipeline-test:us08-05"
# The version being upgraded *from*. CI passes the tag it last published; without one,
# the previous commit's tree is built, which is the same claim without a registry.
PREVIOUS_IMAGE = os.environ.get("PHOTO_PIPELINE_PREVIOUS_IMAGE")
PREVIOUS_TAG = "photo-pipeline-test:us08-05-previous"
SECRET = "container-gate-access-secret"
IMMICH_SENTINEL = "immich-sentinel-9f3a2b"
HOSTNAME = "photos.test"
ALBUM = "rome"
RENAMED = "2019 Rome"
BURST = 30
# ── the image and the stacks ─────────────────────────────────────────────────
@pytest.fixture(scope="session")
def image() -> str:
if not compose_available():
pytest.skip("no Docker daemon with the compose plugin")
return build_image(IMAGE)
@pytest.fixture(scope="module")
def journey(image, tmp_path_factory):
"""One album, one exact duplicate of a photo in it, and the exclusion sentinel."""
library = temporary_library({ALBUM: 3}, prefix="us0805-journey-")
shutil.copyfile(library / ALBUM / f"{ALBUM}_0.jpg", library / ALBUM / "copy.jpg")
env_file = write_env_file(
tmp_path_factory.mktemp("journey") / "gate.env",
SECRET,
{
"PHOTO_PIPELINE_ALLOWED_HOSTS": HOSTNAME,
# Configured but never reachable: the upload view's job here is to show the
# preflight blockers, and the key's job is to be absent from every log.
"PHOTO_PIPELINE_IMMICH_API_KEY": IMMICH_SENTINEL,
"PHOTO_PIPELINE_IMMICH_SERVER_URL": "http://127.0.0.1:1",
},
)
stack = Stack(library, env_file, project=f"{PROJECT}-journey", image=image, secret=SECRET)
try:
stack.down()
stack.up()
stack.wait_until_ready()
_seed_journey(stack)
yield stack
finally:
stack.down()
remove_library(library)
def _seed_journey(stack: Stack) -> None:
"""Everything the views need, established over the public API before they render."""
with closing(stack.client()) as client:
client.post("/inventory/scan").raise_for_status()
client.post("/duplicates/detect").raise_for_status()
queue = client.get("/safety/queue", params={"limit": 100}).json()["items"]
for item in queue:
decided = client.post(
"/safety/decisions", json={"asset_id": item["asset_id"], "decision": "sfw"}
)
assert decided.status_code == 200, decided.text
job = client.post("/analysis/jobs").json()
await_job(client, job["id"])
client.post("/albums/proposals", json={}).raise_for_status()
# An archive destination inside the data volume: a second mount would prove
# nothing more, and the view needs a location to have something to show.
stack.compose("exec", "-T", "api", "mkdir", "-p", "/data/archive")
client.post(
"/archive-locations", json={"name": "external", "root": "/data/archive"}
).raise_for_status()
# ── the browser journey ──────────────────────────────────────────────────────
@needs_compose
def test_the_browser_journey_covers_every_stage_view_of_the_deployed_app(page, journey):
"""One pass through the deployed frontend, in workflow order.
The access secret is supplied the way a person supplies it — the app asks, the
answer is kept for the tab — so what is proven is the authenticated deployment,
not a test-only bypass.
"""
page.on("dialog", lambda dialog: dialog.accept(SECRET))
errors: list[str] = []
page.on("pageerror", lambda error: errors.append(str(error)))
base = journey.base
with closing(journey.client()) as client:
cluster = client.get("/duplicates/clusters").json()["items"][0]
# ── discovery ────────────────────────────────────────────────────────────
page.goto(f"{base}/app/#/workflow")
page.get_by_test_id("stage-safety").wait_for()
expect(page.get_by_test_id("stage-analysis")).to_be_visible()
page.goto(f"{base}/app/#/inventory")
rows = page.get_by_test_id("asset-row")
rows.first.wait_for()
assert rows.count() == 4, "three photos and the duplicate copy; never the sentinel"
assert "sentinel" not in page.content() and "_IGNORE" not in page.content()
# ── duplicate review ─────────────────────────────────────────────────────
page.goto(f"{base}/app/#/duplicates/{cluster['id']}")
page.get_by_test_id("cluster-state").wait_for()
# Exact bytes: the cluster arrives decided, and the review surface has to show
# both members and the evidence the decision was made on.
expect(page.get_by_test_id("cluster-state")).to_contain_text("decided")
expect(page.get_by_test_id("member")).to_have_count(2)
expect(page.get_by_test_id("member").first).to_contain_text("/library/")
# ── analysis ─────────────────────────────────────────────────────────────
page.goto(f"{base}/app/#/analyze")
page.get_by_test_id("analyze-counts").wait_for()
expect(page.get_by_test_id("run-analysis")).to_be_visible()
# ── album proposal ───────────────────────────────────────────────────────
page.goto(f"{base}/app/#/albums?album={ALBUM}")
page.get_by_test_id("suggested-name").wait_for()
page.get_by_test_id("final-name").fill(RENAMED)
page.get_by_test_id("save-name").click()
page.get_by_test_id("approve").click()
expect(page.get_by_test_id("proposal-status")).to_contain_text("approved")
# ── rename, applied against the bind mount ───────────────────────────────
page.goto(f"{base}/app/#/renames")
page.get_by_test_id("build-plan").click()
page.get_by_test_id("operations").wait_for()
expect(page.get_by_test_id("op-destination").first).to_contain_text(RENAMED)
page.get_by_test_id("apply-plan").click()
expect(page.get_by_test_id("apply-result")).to_contain_text("Applied 1, failed 0")
# The mounted library is the host's directory: the container renamed the operator's
# folder, not a copy inside its own layer.
assert (journey.library / RENAMED).is_dir()
assert not (journey.library / ALBUM).exists()
# ── upload preflight ─────────────────────────────────────────────────────
page.goto(f"{base}/app/#/uploads")
page.get_by_test_id("upload-scope").wait_for()
expect(page.get_by_test_id("album-row").first).to_be_visible()
assert IMMICH_SENTINEL not in page.content(), "the API key never reaches the browser"
# ── archive ──────────────────────────────────────────────────────────────
page.goto(f"{base}/app/#/archive")
page.get_by_test_id("archive-locations").wait_for()
expect(page.get_by_test_id("location-row").first).to_contain_text("external")
assert errors == [], f"the deployed frontend raised page errors: {errors}"
# ── the upgrade journey ──────────────────────────────────────────────────────
@pytest.fixture(scope="module")
def previous_image(image) -> str:
"""The image the deployment is upgrading *from*.
The published tag when there is one. Before the first publish there is nothing to
pull, and refusing then would mean the gate could never let the first deploy
through — so the previous commit's tree is built instead, which is the same claim
without a registry.
"""
if PREVIOUS_IMAGE:
pulled = subprocess.run(
["docker", "pull", PREVIOUS_IMAGE], capture_output=True, timeout=1800
)
if pulled.returncode == 0:
return PREVIOUS_IMAGE
return build_image(PREVIOUS_TAG, revision="HEAD~1")
@needs_compose
def test_an_upgrade_keeps_the_database_the_journal_the_jobs_and_the_cache(
image, previous_image, tmp_path_factory
):
library = temporary_library({ALBUM: 2}, prefix="us0805-upgrade-")
env_file = write_env_file(tmp_path_factory.mktemp("upgrade") / "gate.env", SECRET)
stack = Stack(
library, env_file, project=f"{PROJECT}-upgrade", image=previous_image, secret=SECRET
)
try:
stack.down()
stack.up()
stack.wait_until_ready()
# ── what the previous version leaves behind ──────────────────────────
with closing(stack.client()) as client:
client.post("/inventory/scan").raise_for_status()
assets = client.get("/inventory/assets", params={"limit": 200}).json()["items"]
for item in client.get("/safety/queue", params={"limit": 100}).json()["items"]:
client.post(
"/safety/decisions", json={"asset_id": item["asset_id"], "decision": "sfw"}
).raise_for_status()
job = client.post("/analysis/jobs").json()
finished = await_job(client, job["id"])
for asset in assets: # populate the thumbnail cache
thumbnail = client.get(f"/assets/{asset['id']}/thumbnail", params={"size": 256})
assert thumbnail.status_code == 200, thumbnail.text
# A rename plan, left unapplied: the journal has to survive the upgrade
# exactly as it was, or a half-applied one could not be recovered.
client.post("/albums/proposals", json={}).raise_for_status()
_approve(client, RENAMED)
plan = client.post("/rename-plans").json()
before = {
"assets": sorted(asset["id"] for asset in assets),
"job": finished["progress"],
"plan": (plan["id"], plan["checksum"]),
"thumbnails": _cache_files(stack),
"revision": _revision(stack),
}
assert before["thumbnails"], "no thumbnail was cached, so nothing would be proven"
# ── the upgrade: same volume, new image ──────────────────────────────
stack.down(volumes=False)
stack.use_image(image)
stack.up()
stack.wait_until_ready()
after_revision = _revision(stack)
assert after_revision == head_revision(), "the new image did not migrate the volume"
if after_revision != before["revision"]:
# A schema change is snapshotted before it is applied (US07-05), so a
# failed upgrade is restorable rather than a lost library.
assert _ls(stack, "/data/backups"), "a migration ran without a backup"
with closing(stack.client()) as client:
assets = client.get("/inventory/assets", params={"limit": 200}).json()["items"]
assert sorted(asset["id"] for asset in assets) == before["assets"]
assert client.get(f"/jobs/{job['id']}").json()["progress"] == before["job"]
plans = client.get("/rename-plans").json()["items"]
assert (plans[0]["id"], plans[0]["checksum"]) == before["plan"]
for asset in assets:
assert (
client.get(f"/analysis/results/{asset['id']}").status_code == 200
), "an analysis result did not survive the upgrade"
# The cache is keyed by pixel hash and thumbnail version, so an upgrade that
# kept the volume must keep the files: regenerating them is work nobody asked
# for, and losing them silently is how a cache stops being one.
assert set(before["thumbnails"]) <= set(_cache_files(stack))
finally:
stack.down()
remove_library(library)
def _approve(client: httpx.Client, name: str, *, album: str = ALBUM) -> None:
for payload, route in (({"name": name}, "edit"), ({}, "approve")):
current = client.get(f"/albums/proposals/{album}").json()
client.post(
f"/albums/proposals/{album}/{route}",
json={**payload, "expected_version": current["version"]},
).raise_for_status()
def _exec(stack: Stack, *args: str) -> str:
return stack.compose("exec", "-T", "api", *args).stdout
def _ls(stack: Stack, directory: str) -> list[str]:
listing = stack.compose("exec", "-T", "api", "ls", directory, check=False)
return [line for line in listing.stdout.split() if line]
def _cache_files(stack: Stack) -> list[str]:
return sorted(
_exec(stack, "find", "/data/cache", "-type", "f", "-name", "*.webp").split()
)
def _revision(stack: Stack) -> str:
"""The schema revision the volume's database is actually at."""
return _exec(
stack,
"python",
"-c",
"import sqlite3;print(sqlite3.connect('/data/photo_pipeline.db')"
".execute('select version_num from alembic_version').fetchone()[0])",
).strip()
# ── the restart journey ──────────────────────────────────────────────────────
@needs_compose
def test_killing_both_containers_mid_job_resumes_without_doing_anything_twice(
image, tmp_path_factory
):
"""`docker kill` is the honest restart: no grace period, no orderly stop, no
chance for either process to write a tidy final state."""
library = temporary_library({"burst": BURST}, prefix="us0805-restart-")
env_file = write_env_file(tmp_path_factory.mktemp("restart") / "gate.env", SECRET)
stack = Stack(library, env_file, project=f"{PROJECT}-restart", image=image, secret=SECRET)
try:
stack.down()
stack.up()
stack.wait_until_ready()
with closing(stack.client()) as client:
client.post("/inventory/scan").raise_for_status()
assets = client.get("/inventory/assets", params={"limit": 200}).json()["items"]
assert len(assets) == BURST
for item in client.get("/safety/queue", params={"limit": 100}).json()["items"]:
client.post(
"/safety/decisions", json={"asset_id": item["asset_id"], "decision": "sfw"}
).raise_for_status()
job = client.post("/analysis/jobs").json()
progress = _wait_for_progress(client, job["id"])
assert 0 < progress["done"] < progress["total"], progress
stack.compose("kill", "api", "worker")
stack.up()
stack.wait_until_ready()
with closing(stack.client()) as client:
finished = await_job(client, job["id"])
assert finished["progress"]["done"] == BURST, finished
results = [
client.get(f"/analysis/results/{asset['id']}").json() for asset in assets
]
# Exactly one stored result per asset: at-least-once execution, idempotent
# recovery — a retried item overwrites its own attempt, it does not add one.
assert len(results) == BURST
assert all(result["description"] for result in results)
# And the side effect nobody can take back — the call to the provider — happened
# again only for whatever was in flight when the containers died.
analysed = collections.Counter(_exec(stack, "cat", "/data/vision.log").split())
assert len(analysed) == BURST, "every photo was analysed, and only the library's"
assert max(analysed.values()) <= 2, dict(analysed)
assert sum(1 for count in analysed.values() if count > 1) <= 1, dict(analysed)
finally:
stack.down()
remove_library(library)
def _wait_for_progress(client: httpx.Client, job_id: str, *, timeout: float = 120) -> dict:
"""Wait until the job is provably under way but provably unfinished."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
snapshot = client.get(f"/jobs/{job_id}").json()
progress = snapshot["progress"]
if progress["done"] and progress["done"] < progress["total"]:
return progress
if snapshot["state"] in ("succeeded", "failed"):
raise AssertionError(f"the job finished before it could be interrupted: {snapshot}")
time.sleep(0.05)
raise AssertionError(f"the job never started: {client.get(f'/jobs/{job_id}').json()}")
# ── the security gates ───────────────────────────────────────────────────────
@needs_compose
def test_the_deployed_instance_refuses_a_caller_without_a_session(journey):
with httpx.Client(base_url=f"{journey.base}/api/v1", timeout=30) as client:
for method, path in (("GET", "/workflow"), ("POST", "/inventory/scan")):
response = client.request(method, path)
assert response.status_code == 401, path
assert response.json()["error"]["code"] == "unauthenticated"
# A session still has to be paid for with the operator's secret.
assert client.get("/session", headers={"X-Access-Secret": "guessed"}).status_code == 401
# Readiness stays open: the orchestrator's health check holds no session.
assert client.get("/health/ready").status_code == 200
@needs_compose
def test_forged_forwarded_headers_cannot_smuggle_an_allowed_host_past_the_check(journey):
"""Behind a proxy the app believes ``X-Forwarded-*`` — but only from the proxy.
Nothing in this composition is a trusted proxy, so the claim is the client's."""
with httpx.Client(base_url=f"{journey.base}/api/v1", timeout=30) as client:
client.headers["X-CSRF-Token"] = (
client.get("/session", headers={"X-Access-Secret": SECRET}).json()["csrf_token"]
)
# The hostname this deployment is reached under is accepted.
assert client.get("/workflow", headers={"Host": HOSTNAME}).status_code == 200
forged = client.get(
"/workflow",
headers={"Host": "photos.evil.example", "X-Forwarded-Host": HOSTNAME},
)
assert forged.status_code == 403
assert forged.json()["error"]["code"] == "host_not_allowed"
# A forged protocol claim must not mark the session cookie as HTTPS-only
# either — that would strand the operator's real, plain-HTTP session.
bootstrap = httpx.get(
f"{journey.base}/api/v1/session",
headers={"X-Access-Secret": SECRET, "X-Forwarded-Proto": "https"},
timeout=30,
)
assert "secure" not in bootstrap.headers["set-cookie"].lower()
@needs_compose
def test_a_path_that_leaves_the_mounted_library_is_refused(journey):
"""The container's own filesystem is not the library. A symlink swapped under a
known asset is the sharpest version of the question, because the database still
points at a path inside the mount."""
with closing(journey.client()) as client:
asset = client.get("/inventory/assets", params={"limit": 200}).json()["items"][0]
original = journey.library / Path(asset["current_path"]).relative_to("/library")
kept = original.read_bytes()
original.unlink()
original.symlink_to("/etc/passwd")
try:
escaped = client.get(f"/assets/{asset['id']}/thumbnail", params={"size": 256})
finally:
original.unlink()
original.write_bytes(kept)
assert escaped.status_code == 403
assert escaped.json()["error"]["code"] == "path_not_allowed"
assert "root:" not in escaped.text
# And a root that names nothing mounted is refused before the process serves.
refused = journey.compose(
"run",
"--rm",
"--no-deps",
"--env",
"PHOTO_PIPELINE_LIBRARY_ROOTS=/srv/photos",
"api",
"serve",
check=False,
)
assert refused.returncode == 5, refused.stdout + refused.stderr
@needs_compose
def test_no_secret_reaches_the_container_logs(journey):
with httpx.Client(base_url=f"{journey.base}/api/v1", timeout=30) as client:
client.get("/session", headers={"X-Access-Secret": "wrong-secret-attempt"})
client.get("/session", headers={"X-Access-Secret": SECRET})
logs = journey.logs()
assert SECRET not in logs
assert IMMICH_SENTINEL not in logs
assert "wrong-secret-attempt" not in logs, "a rejected secret is still a secret"
# The refusal itself is logged, so an operator can see the attempt.
assert "access secret rejected" in logs
@needs_compose
def test_the_evidence_of_this_run_names_the_stack_it_was_produced_from(journey):
"""A gate that cannot say what it ran against is an opinion. `docker compose ps`
is the record: one API, one worker, one completed migration."""
listing = [
json.loads(line)
for line in journey.compose("ps", "--all", "--format", "json").stdout.splitlines()
if line.strip()
]
services = {entry["Service"]: entry["State"] for entry in listing}
assert services["api"] == "running" and services["worker"] == "running"
assert services["migrate"] == "exited"
if __name__ == "__main__": # a quick way to run just this file
raise SystemExit(pytest.main([__file__, "-v", *sys.argv[1:]]))

View File

@@ -0,0 +1,164 @@
"""US09-03: the architecture overview, checked against the architecture.
An architecture document is the one that rots most quietly: nothing breaks when it
describes a module that was renamed two epics ago, it just quietly misleads the next
person. So the parts of it that the code also knows — module names, state machines,
table names, the modules an invariant is claimed to live in — are compared with the
code, and a missing one fails the suite.
The diagrams' rendering is proven in ``tests/e2e/test_docs_ui.py``.
"""
from __future__ import annotations
import re
from pathlib import Path
import photo_pipeline.models # noqa: F401 (registers every table on Base.metadata)
from photo_pipeline.db import Base
from photo_pipeline.services import jobs, rename_journal, upload_batches
REPO = Path(__file__).resolve().parents[2]
PACKAGE = REPO / "photo_pipeline"
OVERVIEW = REPO / "docs" / "architecture.md"
TEXT = OVERVIEW.read_text()
# Names the map does not owe the reader individually: the package markers, the CLI
# entry point, and the two modules that exist to be small and obvious.
UNMAPPED = {"__init__", "__main__", "logging"}
def modules() -> set[str]:
"""Every module and package under photo_pipeline, by the name it is imported as."""
names = set()
for path in PACKAGE.rglob("*.py"):
relative = path.relative_to(PACKAGE)
names.add(relative.parts[0] if len(relative.parts) > 1 else relative.stem)
return {name.removesuffix(".py") for name in names} - UNMAPPED
def states(container: type) -> set[str]:
return {
value
for name, value in vars(container).items()
if name.isupper() and isinstance(value, str)
}
# ── the module map ───────────────────────────────────────────────────────────
def test_every_module_appears_in_the_map():
"""A service nobody documented is a service the next person re-implements."""
missing = sorted(name for name in modules() if name not in TEXT)
assert missing == [], f"absent from the architecture overview: {missing}"
def test_every_service_is_named_individually():
"""The package table says what `services/` is for; this is the list that actually
drifts, because a new service is added roughly every story."""
services = {path.stem for path in (PACKAGE / "services").glob("*.py")} - UNMAPPED
missing = sorted(name for name in services if not re.search(rf"\b{name}\b", TEXT))
assert missing == [], f"services missing from the overview: {missing}"
def test_the_map_names_no_module_that_does_not_exist():
"""Every `services/<name>` or backticked module path in the document resolves."""
referenced = set(re.findall(r"`(?:photo_pipeline/)?([a-z_]+)/([a-z_]+)\.py`", TEXT))
referenced |= {("", name) for name in re.findall(r"`([a-z_]+)\.py`", TEXT)}
missing = [
f"{package}/{module}.py" if package else f"{module}.py"
for package, module in referenced
if not (PACKAGE / package / f"{module}.py").is_file()
]
assert missing == [], f"documented but absent from the source: {missing}"
# ── the state machines ───────────────────────────────────────────────────────
def test_every_job_state_is_documented():
missing = sorted(state for state in states(jobs.JobState) if state not in TEXT)
assert missing == [], f"job states missing from the overview: {missing}"
def test_every_rename_journal_state_is_documented():
missing = sorted(state for state in states(rename_journal.JournalState) if state not in TEXT)
assert missing == [], f"journal states missing from the overview: {missing}"
def test_every_upload_batch_state_is_documented():
missing = sorted(state for state in states(upload_batches.BatchState) if state not in TEXT)
assert missing == [], f"batch states missing from the overview: {missing}"
def test_the_unsafe_journal_states_are_named_as_the_ones_that_block():
"""The reason an unrelated mutation is refused has to be findable."""
for state in rename_journal.UNSAFE_STATES:
assert state in TEXT
assert "rename_recovery_required" in TEXT
def test_an_uncertain_upload_is_documented_as_not_retryable():
assert upload_batches.BatchState.UNKNOWN not in upload_batches.RUNNABLE_STATES
assert "not** restartable" in TEXT or "not restartable" in TEXT
# ── the data model ───────────────────────────────────────────────────────────
def test_every_table_is_listed():
missing = sorted(name for name in Base.metadata.tables if name not in TEXT)
assert missing == [], f"tables missing from the overview: {missing}"
def test_the_document_lists_no_table_that_does_not_exist():
listed = set(re.findall(r"`([a-z_]+)`", TEXT))
plausible = {name for name in listed if name.endswith("s") and "_" in name}
invented = sorted(
name
for name in plausible
if name not in Base.metadata.tables
and not (PACKAGE / "services" / f"{name}.py").is_file()
and name not in {"library_roots", "trusted_proxies", "allowed_hosts", "asset_paths"}
)
assert invented == [], f"looks like a table but is not one: {invented}"
# ── the invariants ───────────────────────────────────────────────────────────
def test_each_invariant_points_at_a_module_that_enforces_it():
"""The point of the table is to answer 'where do I look'. A wrong answer there
costs more than no answer."""
claims = {
"path_policy.is_excluded": PACKAGE / "path_policy.py",
"path_policy.resolve_in_roots": PACKAGE / "path_policy.py",
"services/app_lock.py": PACKAGE / "services" / "app_lock.py",
"services/exif_checkpoint.py": PACKAGE / "services" / "exif_checkpoint.py",
"services/archive_transfer.py": PACKAGE / "services" / "archive_transfer.py",
"api/security.py": PACKAGE / "api" / "security.py",
"imaging.py": PACKAGE / "imaging.py",
}
for claim, path in claims.items():
assert claim in TEXT, f"the invariant table does not mention {claim}"
assert path.is_file(), f"{claim} does not exist"
function = claim.rpartition(".")[2] if "/" not in claim else ""
if function and function != "py":
assert f"def {function}" in path.read_text(), f"{claim} is not defined there"
def test_the_lock_order_matches_the_ranks_in_the_code():
from photo_pipeline.jobs import locks
order = [name for name, _ in sorted(locks.LOCK_RANK.items(), key=lambda item: item[1])]
assert order[0].startswith("library"), "the broadest lock is no longer the library"
assert "library → stage/job → album/folder → asset" in TEXT
def test_the_frozen_donor_archive_is_described_as_provenance_not_a_dependency():
"""That production never imports the frozen sources is proven by
``tests/unit/test_legacy_archive.py`` — which also forbids any other test from
naming that directory, so this one checks the claim by its consequence."""
assert "must not import" in TEXT
assert "provenance and rollback evidence" in TEXT

View File

@@ -0,0 +1,147 @@
"""US08-05: the container acceptance gate's own contract, checked without a daemon.
The gate itself needs Docker, a browser, and several minutes; running it from inside
the suite would be a fork bomb with better manners. What is checkable offline is what
makes it a *gate* rather than a long test run:
* it selects the container journeys by marker, so adding one is enough to put it in
front of a deploy;
* it accepts no skip at all — every reason a check would skip here (no daemon, no
compose plugin, no browser) means the deployed runtime was not proven;
* it retains checksummed evidence per run;
* and CI runs it on `main`, which is what the publish step waits for.
The running proof is ``tests/e2e/test_phase_h_container.py``.
"""
from __future__ import annotations
import json
import re
from pathlib import Path
import yaml
from photo_pipeline.config import Config
from photo_pipeline.services import release
REPO = Path(__file__).resolve().parents[2]
SUITE = REPO / "tests" / "e2e" / "test_phase_h_container.py"
TEST_WORKFLOW = yaml.safe_load((REPO / ".gitea" / "workflows" / "test.yml").read_text())
def _config(tmp_path) -> Config:
for name in ("data", "lib"):
(tmp_path / name).mkdir(exist_ok=True)
return Config.from_env(
{
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(tmp_path / "lib"),
}
)
# ── what the gate runs ───────────────────────────────────────────────────────
def test_the_gate_selects_the_container_journeys_by_their_marker():
assert release.CONTAINER_STAGES == (("container", ("tests/e2e", "-m", "phase_h")),)
declared = [line for line in (REPO / "pyproject.toml").read_text().splitlines()
if line.strip().startswith('"phase_h:')]
assert declared, "an unregistered marker selects nothing and fails no gate"
def test_every_journey_in_the_suite_carries_the_marker():
"""A test in that file without the marker is a check the gate never runs."""
source = SUITE.read_text()
assert "pytestmark = [pytest.mark.phase_h, pytest.mark.container]" in source
journeys = re.findall(r"^def (test_[a-z_]+)", source, re.MULTILINE)
assert len(journeys) >= 4, journeys
for required in ("browser", "upgrade", "kill", "secret"):
assert any(required in name for name in journeys), (required, journeys)
# ── no skip is an environment limit here ─────────────────────────────────────
def test_the_gate_accepts_no_skipped_check_at_all():
assert release.CONTAINER_ALLOWED_SKIP_REASONS == ()
docker_missing = release.StageResult(
"container", [], 0, 0.1, "1 skipped", ["SKIPPED [1] x.py:1: no Docker daemon available"]
)
# The release gate tolerates exactly this one, because the image definition is
# still checked offline. The container gate cannot: it is the deployment's proof.
assert release.unexpected_skips([docker_missing]) == []
assert release.unexpected_skips(
[docker_missing], release.CONTAINER_ALLOWED_SKIP_REASONS
) == ["SKIPPED [1] x.py:1: no Docker daemon available"]
def test_a_skipped_check_fails_the_run_and_the_evidence_says_so(tmp_path):
skipping = tmp_path / "test_skipping.py"
skipping.write_text(
"import pytest\n\n"
"def test_x():\n"
" pytest.skip('no Docker daemon available')\n"
)
evidence = tmp_path / "evidence"
report = release.run_gate(
_config(tmp_path),
output=evidence,
stages=(("container", (str(skipping),)),),
allowed_skips=release.CONTAINER_ALLOWED_SKIP_REASONS,
)
assert report["ok"] is False
assert report["failures"] == [], "the stage passed; the skip is what fails the gate"
assert report["unexpected_skips"], report
written = json.loads((evidence / release.REPORT_NAME).read_text())
assert written["ok"] is False
assert (evidence / "logs" / "container.log").exists()
for line in (evidence / release.CHECKSUMS_NAME).read_text().splitlines():
digest, name = line.split(" ", 1)
assert release.sha256_file(evidence / name) == digest
def test_the_same_run_passes_when_nothing_skips(tmp_path):
passing = tmp_path / "test_passing.py"
passing.write_text("def test_x():\n assert True\n")
report = release.run_gate(
_config(tmp_path),
output=tmp_path / "evidence",
stages=(("container", (str(passing),)),),
allowed_skips=release.CONTAINER_ALLOWED_SKIP_REASONS,
)
assert report["ok"] is True and report["unexpected_skips"] == []
assert report["revision"], "the evidence must say which commit it covers"
# ── the command, and CI ──────────────────────────────────────────────────────
def test_the_command_is_documented_and_wired():
from photo_pipeline.__main__ import main # noqa: F401 (import proves it loads)
assert "container-gate" in (REPO / "photo_pipeline" / "__main__.py").read_text()
assert "container-gate" in (REPO / "README.md").read_text()
def test_ci_runs_the_gate_on_main_and_keeps_its_evidence():
job = TEST_WORKFLOW["jobs"]["container"]
assert job["if"] == "gitea.event_name == 'push'", "pull requests have nothing to upgrade from"
script = "\n".join(step["run"] for step in job["steps"] if "run" in step)
assert "photo_pipeline container-gate" in script
evidence = next(step for step in job["steps"] if "upload-artifact" in str(step.get("uses")))
assert evidence["if"] == "always()", "a failed gate's logs are the ones worth keeping"
assert evidence["with"]["path"] == "gate-evidence"
def test_the_upgrade_journey_can_still_reach_the_previous_version():
"""Without depth, `git archive HEAD~1` has nothing to build."""
checkout = next(
step for step in TEST_WORKFLOW["jobs"]["container"]["steps"] if "checkout" in str(step.get("uses"))
)
assert checkout["with"]["fetch-depth"] >= 2

View File

@@ -0,0 +1,179 @@
"""US09-01: the documentation the application serves, checked without a browser.
What is checkable offline is everything that makes the manuals *navigable* rather
than merely present: every page reachable, every internal link and anchor landing
somewhere real, the vendored renderer being the file that was pinned, and the
application actually serving `docs/` in both the working copy and the image.
The rendering itself — marked, mermaid, anchors, link rewriting — is proven in a
real browser by ``tests/e2e/test_docs_ui.py``.
"""
from __future__ import annotations
import hashlib
import json
import re
from pathlib import Path
from starlette.testclient import TestClient
from photo_pipeline.api.app import create_app
from photo_pipeline.api.security import DEFAULT_HEADERS
from photo_pipeline.config import Config
REPO = Path(__file__).resolve().parents[2]
DOCS = REPO / "docs"
VENDOR = REPO / "frontend" / "js" / "vendor"
INDEX = DOCS / "index.md"
MARKDOWN_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)")
HEADING = re.compile(r"^#{1,6}\s+(.+?)\s*$", re.MULTILINE)
FENCE = re.compile(r"^```.*?^```", re.MULTILINE | re.DOTALL)
def pages() -> list[Path]:
return sorted(DOCS.rglob("*.md"))
def body(path: Path) -> str:
"""A document without its fenced code, so an example link in a shell snippet is
not mistaken for a link the reader can follow."""
return FENCE.sub("", path.read_text())
def slug(text: str) -> str:
"""The heading-anchor rule, mirroring ``frontend/js/docs.js``.
Deliberately duplicated: this check has to run without a browser. The browser
test is the authority on the real behaviour, and it asserts the same anchors.
"""
cleaned = re.sub(r"[^\w\s-]", "", text.lower(), flags=re.UNICODE).strip()
return re.sub(r"[\s-]+", "-", cleaned).strip("-") or "section"
def anchors(path: Path) -> set[str]:
found: dict[str, int] = {}
for heading in HEADING.findall(body(path)):
# Markdown emphasis and inline code are not part of the rendered text.
plain = re.sub(r"[*`_]", "", heading)
base = slug(plain)
found[base] = found.get(base, 0) + 1
return {name if index == 1 else f"{name}-{index}" for name, count in found.items() for index in range(1, count + 1)}
def internal_links(path: Path) -> list[str]:
return [
href
for href in MARKDOWN_LINK.findall(body(path))
if not re.match(r"^[a-z][a-z\d+.-]*:", href, re.IGNORECASE) and not href.startswith("//")
]
# ── the documentation tree ───────────────────────────────────────────────────
def test_the_index_exists_and_names_every_page():
"""A page nobody links to is a page nobody reads."""
assert INDEX.is_file()
listed = {
(INDEX.parent / href.split("#")[0]).resolve()
for href in internal_links(INDEX)
if href.split("#")[0].endswith(".md")
}
unreachable = [page.name for page in pages() if page != INDEX and page.resolve() not in listed]
assert unreachable == [], f"not linked from the index: {unreachable}"
def test_every_internal_link_and_anchor_resolves():
broken: list[str] = []
for page in pages():
for href in internal_links(page):
target, _, anchor = href.partition("#")
destination = page if not target else (page.parent / target).resolve()
if not destination.is_file():
broken.append(f"{page.name}{href} (no such file)")
continue
if anchor and anchor not in anchors(destination):
broken.append(f"{page.name}{href} (no such heading)")
assert broken == [], broken
def test_every_page_starts_with_one_title():
for page in pages():
titles = [line for line in page.read_text().splitlines() if line.startswith("# ")]
assert len(titles) == 1, f"{page.name} has {len(titles)} top-level titles"
# ── the vendored renderer ────────────────────────────────────────────────────
def test_the_vendored_libraries_are_the_files_that_were_pinned():
"""A vendored dependency without a recorded checksum is a dependency nobody is
reviewing. Replacing one has to be a visible change to this manifest."""
manifest = json.loads((VENDOR / "VERSIONS.json").read_text())
recorded = {entry["file"]: entry for entry in manifest["libraries"]}
on_disk = {path.name for path in VENDOR.iterdir() if path.suffix in (".js", ".mjs")}
assert on_disk == set(recorded), f"unrecorded vendored files: {on_disk ^ set(recorded)}"
for name, entry in recorded.items():
digest = hashlib.sha256((VENDOR / name).read_bytes()).hexdigest()
assert digest == entry["sha256"], f"{name} does not match the pinned checksum"
assert entry["version"] in entry["url"], f"{name}: the pinned URL and version disagree"
assert entry["license"], f"{name}: no license recorded"
def test_the_documentation_view_is_wired_into_the_shell():
shell = (REPO / "frontend" / "index.html").read_text()
assert 'data-nav="docs"' in shell, "the manuals are unreachable from the navigation"
assert '"/docs"' in (REPO / "frontend" / "js" / "app.js").read_text()
# ── the policy the view runs under ───────────────────────────────────────────
def test_the_script_boundary_is_unchanged_and_only_style_was_relaxed():
"""The measured cost of rendering diagrams in the browser, held to that cost.
mermaid needs to style the SVG it builds, which `style-src 'self'` refuses. It
does not need to execute generated code, so `script-src` must never acquire the
escape hatch that would let injected markup run.
"""
policy = DEFAULT_HEADERS["content-security-policy"]
directives = {
part.split(" ")[0]: part.split(" ")[1:]
for part in (piece.strip() for piece in policy.split(";"))
if part
}
assert directives["script-src"] == ["'self'"]
assert "'unsafe-inline'" in directives["style-src"]
for directive in ("default-src", "img-src", "connect-src", "font-src"):
assert "'self'" in directives[directive]
assert "'unsafe-inline'" not in directives[directive]
assert directives["frame-ancestors"] == ["'none'"]
# ── serving it ───────────────────────────────────────────────────────────────
def test_the_application_serves_the_markdown_without_a_session(tmp_path):
"""Whoever cannot get past the access secret is exactly who needs these pages."""
config = Config.from_env({"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data")})
(tmp_path / "data").mkdir()
with TestClient(create_app(config), raise_server_exceptions=False) as client:
response = client.get("/docs/index.md", headers={"cookie": ""})
assert response.status_code == 200
assert "Photo Pipeline documentation" in response.text
assert client.get("/docs/overview.md").status_code == 200
assert client.get("/docs/nothing-here.md").status_code == 404
# The mount is a directory, not a path parameter: nothing above it is reachable.
assert client.get("/docs/../pyproject.toml").status_code in (307, 404)
assert client.get("/docs/%2e%2e/pyproject.toml").status_code == 404
def test_the_image_ships_the_documentation_it_serves():
"""An image without `docs/` serves an empty manual — and the build context is
deny-by-default, so a new directory is excluded until it is named."""
assert "\n!docs\n" in (REPO / ".dockerignore").read_text()
assert "COPY docs ./docs" in (REPO / "Dockerfile").read_text()

View File

@@ -0,0 +1,178 @@
"""US09-02: the installation manual, checked against the thing it describes.
Documentation rots quietly. A setting is renamed, a command grows a flag, an exit
code changes meaning — and the manual keeps confidently saying the old thing until
somebody follows it into a bad afternoon. So every fact in it that the code also
knows is compared with the code, in both directions where both directions are
meaningful: a setting the manual invents fails, and a setting the manual forgot
fails too.
"""
from __future__ import annotations
import re
import subprocess
import sys
from functools import lru_cache
from pathlib import Path
from photo_pipeline.config import ENV_PREFIX, LEGACY_ALIASES, Config
REPO = Path(__file__).resolve().parents[2]
MANUAL = REPO / "docs" / "installation.md"
MAIN = REPO / "photo_pipeline" / "__main__.py"
TEXT = MANUAL.read_text()
ENV_NAMES = re.compile(rf"{ENV_PREFIX}[A-Z0-9_]+")
# Read by docker-compose.yml, not by Config. They are configuration an installer sets,
# so the manual documents them; they are simply not application settings.
COMPOSITION_ONLY = {
f"{ENV_PREFIX}IMAGE",
f"{ENV_PREFIX}LIBRARY_HOST_PATH",
f"{ENV_PREFIX}PUBLISH_ADDRESS",
f"{ENV_PREFIX}UID",
f"{ENV_PREFIX}GID",
f"{ENV_PREFIX}ENV_FILE", # read before Config exists, so it is not a field
}
def documented_env_names() -> set[str]:
return set(ENV_NAMES.findall(TEXT))
@lru_cache
def cli_help(*args: str) -> str:
"""What the CLI says about itself, asked the way an operator asks."""
result = subprocess.run(
[sys.executable, "-m", "photo_pipeline", *args, "--help"],
cwd=REPO,
capture_output=True,
text=True,
timeout=120,
check=False,
)
assert result.returncode == 0, result.stderr
return result.stdout
@lru_cache
def cli_commands() -> frozenset[str]:
listed = re.search(r"\{([a-z0-9,\-]+)\}", cli_help())
assert listed, f"the CLI listed no subcommands:\n{cli_help()}"
return frozenset(listed.group(1).split(","))
def cli_flags(command: str) -> frozenset[str]:
return frozenset(re.findall(r"--[a-z][a-z-]+", cli_help(command)))
# ── settings ─────────────────────────────────────────────────────────────────
def test_every_setting_is_documented():
"""A setting nobody documents is a setting nobody configures deliberately."""
expected = {f"{ENV_PREFIX}{field.upper()}" for field in Config.model_fields}
missing = sorted(expected - documented_env_names())
assert missing == [], f"undocumented settings: {missing}"
def test_the_manual_invents_no_settings():
unknown = sorted(
name
for name in documented_env_names()
if name.removeprefix(ENV_PREFIX).lower() not in Config.model_fields
and name not in COMPOSITION_ONLY
)
assert unknown == [], f"documented but not a real setting: {unknown}"
def test_the_documented_defaults_are_the_real_defaults():
"""Spot-checked where a wrong default is actively dangerous: the bind address,
the trust boundary, and the gate that protects an irreplaceable library."""
assert Config.model_fields["host"].default == "127.0.0.1"
assert "`PHOTO_PIPELINE_HOST` | `127.0.0.1`" in TEXT
assert Config.model_fields["allowed_hosts"].default == ()
assert Config.model_fields["require_dry_run_approval"].default is False
assert "`PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL` | `false`" in TEXT
def test_every_secret_is_marked_as_one():
for field in ("access_secret", "vision_api_key", "immich_api_key"):
name = f"{ENV_PREFIX}{field.upper()}"
row = next(line for line in TEXT.splitlines() if line.startswith(f"| `{name}`"))
assert "secret" in row.lower(), f"{name} is not marked as a secret"
def test_the_legacy_aliases_are_documented():
"""An operator with an existing photo_analyzer.env needs to know it still works."""
for alias in LEGACY_ALIASES:
assert alias in TEXT, f"legacy alias {alias} is undocumented"
# ── commands ─────────────────────────────────────────────────────────────────
# The manual is an installation and operations manual, not a command reference: these
# are the commands an installer actually runs, and every one of them must exist.
OPERATIONAL = ("migrate", "serve", "worker", "backup", "verify-backup", "restore", "diagnostics")
def test_every_command_the_manual_tells_you_to_run_exists():
missing = [name for name in OPERATIONAL if name not in cli_commands()]
assert missing == [], f"the manual names commands the CLI does not have: {missing}"
undocumented = [name for name in OPERATIONAL if not re.search(rf"\b{re.escape(name)}\b", TEXT)]
assert undocumented == [], f"operational commands the manual omits: {undocumented}"
def test_every_documented_flag_exists_on_the_command_it_is_shown_with():
for command, flag in (
("backup", "--reason"),
("backup", "--keep"),
("restore", "--into"),
("worker", "--allow-legacy"),
("serve", "--allow-legacy"),
):
assert flag in cli_flags(command), f"{command} has no {flag}"
assert flag in TEXT, f"{flag} is undocumented"
# ── exit codes ───────────────────────────────────────────────────────────────
def test_every_documented_exit_code_is_one_the_cli_can_return():
documented = {int(code) for code in re.findall(r"^\| `(\d)` \|", TEXT, re.MULTILINE)}
returned = {int(code) for code in re.findall(r"^\s+return (\d)$", MAIN.read_text(), re.MULTILINE)}
assert documented, "no exit codes are documented"
assert documented <= returned | {0}, f"documented but unreachable: {sorted(documented - returned)}"
def test_every_refusal_exit_code_is_documented():
"""0 is success and 1 is 'it said why'; every other code is a specific refusal an
operator will meet at startup, and meeting an undocumented one is the worst case."""
returned = {int(code) for code in re.findall(r"^\s+return (\d)$", MAIN.read_text(), re.MULTILINE)}
documented = {int(code) for code in re.findall(r"^\| `(\d)` \|", TEXT, re.MULTILINE)}
missing = sorted(code for code in returned if code not in documented and code != 0)
assert missing == [], f"undocumented exit codes: {missing}"
# ── the promises the manual makes ────────────────────────────────────────────
def test_the_manual_carries_no_credential_shaped_example():
"""The repository's own scanner refuses these in a commit; a manual is exactly
where a real-looking one gets copied from."""
suspicious = re.findall(
r"(?i)(api[_-]?key|secret|token|password)\s*[=:]\s*[\"']?([A-Za-z0-9_\-]{12,})",
TEXT,
)
assert suspicious == [], f"credential-shaped example: {suspicious}"
def test_the_manual_states_the_invariants_an_installer_must_not_break():
for invariant in ("_IGNORE/", "One writer", "EXIF is verified before upload"):
assert invariant in TEXT, f"the manual does not state: {invariant}"
def test_the_manual_is_reachable_from_the_index():
assert "installation.md" in (REPO / "docs" / "index.md").read_text()

View File

@@ -189,10 +189,26 @@
],
"US08-04": [
"tests/integration/test_deploy_workflows.py"
],
"US08-05": [
"tests/integration/test_container_gate.py",
"tests/e2e/test_phase_h_container.py"
],
"US09-01": [
"tests/integration/test_documentation.py",
"tests/e2e/test_docs_ui.py"
],
"US09-02": [
"tests/integration/test_installation_manual.py"
],
"US09-03": [
"tests/integration/test_architecture_overview.py",
"tests/e2e/test_docs_ui.py"
]
},
"planned": [
"US08-05"
"US09-04",
"US09-05"
],
"_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
}