From 792134c72dde02997eb0e997782bec36d9d790ab Mon Sep 17 00:00:00 2001
From: domverse <domverse@domverse-berlin.eu>
Date: Sun, 23 Aug 2026 15:03:55 +0200
Subject: [PATCH] docs: add E09, the product documentation epic

Adds the installation manual, architecture overview, and illustrated user
manual as a five-story epic, plus the acceptance gate that keeps them true.

The documentation is markdown under docs/, so the same files are readable in
the repository and rendered by the deployed application at /app/#/docs. An
operator who was handed a URL and an access secret has no repository checkout
in front of them, and the network the application is deployed to is not
assumed to reach a CDN.

Two decisions are recorded in the epic rather than left to implementation:

- The renderer is vendored (marked), not written and not fetched.
- Diagrams are mermaid, rendered client-side, with script-src untouched.
  Whether mermaid needs 'unsafe-eval' was measured rather than assumed: its
  bundle contains no eval( and no new Function, and rendered under this
  application's exact CSP it produced an SVG with no script-src violation.
  What it does violate is style-src, which gains 'unsafe-inline'. With
  script-src, img-src, connect-src, and font-src all unchanged, that leaves
  defacement rather than execution or exfiltration. The alternative --
  pre-rendering diagrams to committed SVG with the already-installed
  Playwright -- is recorded as the migration if that trade is ever refused.

Screenshots are generated from the running application, never pasted, and
every documented setting, command, exit code, error code, and state is
cross-checked against the code that implements it, so stale documentation
fails a test instead of misleading an operator.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015csGY8XV1M92fHfKnMmjtu
---
 delivery_backlog/E09-documentation.md         | 77 +++++++++++++++++++
 delivery_backlog/README.md                    | 10 ++-
 .../stories/US09-01-docs-in-app.md            | 49 ++++++++++++
 .../stories/US09-02-installation-manual.md    | 46 +++++++++++
 .../stories/US09-03-architecture-overview.md  | 47 +++++++++++
 .../stories/US09-04-user-manual.md            | 52 +++++++++++++
 delivery_backlog/stories/US09-05-docs-gate.md | 38 +++++++++
 tests/story_traceability.json                 |  8 +-
 8 files changed, 322 insertions(+), 5 deletions(-)
 create mode 100644 delivery_backlog/E09-documentation.md
 create mode 100644 delivery_backlog/stories/US09-01-docs-in-app.md
 create mode 100644 delivery_backlog/stories/US09-02-installation-manual.md
 create mode 100644 delivery_backlog/stories/US09-03-architecture-overview.md
 create mode 100644 delivery_backlog/stories/US09-04-user-manual.md
 create mode 100644 delivery_backlog/stories/US09-05-docs-gate.md

diff --git a/delivery_backlog/E09-documentation.md b/delivery_backlog/E09-documentation.md
new file mode 100644
index 0000000..d2c2cad
--- /dev/null
+++ b/delivery_backlog/E09-documentation.md
@@ -0,0 +1,77 @@
+# E09 — Product Documentation
+
+Concept phase: none. Like [E08](E08-container-deployment.md), this epic is a delivery
+addition rather than a product-scope change: the same application, documented well
+enough that somebody who did not build it can install it, understand it, and operate it
+without reading the source.
+
+It does not change the product scope in
+[`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md). No safety
+invariant moves, no schema changes, no new runtime capability. The one change to the
+running application is a documentation view and the static assets it needs.
+
+## Why the application serves its own documentation
+
+The manuals describe an application that is reached over HTTP behind an access secret.
+Documentation that lives only in the repository is unreachable from the deployment it
+describes: an operator who has just been handed a URL and a secret has no Gitea account
+in front of them. So the same markdown files are both the repository's documentation and
+the deployment's `/docs` view, and neither is a copy of the other.
+
+## Decisions made in this epic
+
+**Markdown is the source.** Everything is written as markdown under `docs/`, so it is
+reviewable in a diff, readable on Gitea, and renderable in the app. No documentation
+format that only a tool can read.
+
+**The renderer is vendored, not written and not fetched.** `marked` (MIT, no
+dependencies, ships an ES module) is pinned and committed under
+`frontend/js/vendor/`. A CDN is not an option: the application is deployed to a network
+whose outbound access is not assumed, and `default-src 'self'` forbids it.
+
+**Diagrams are mermaid, rendered client-side, with the script boundary intact.** The
+claim that mermaid requires `'unsafe-eval'` was tested rather than believed: mermaid 11's
+bundle contains no `eval(` and no `new Function` — only a lodash `Function("return this")`
+global-detection fallback that short-circuits on `globalThis` and never executes.
+Rendered under this application's exact CSP, a flowchart produced a 15.8 KB SVG and
+raised **no `script-src` violation**. What it does raise is `style-src`: mermaid styles
+its output with an injected `<style>` element and `style=` attributes.
+
+So `script-src 'self'` stays exactly as US07-02 and US08-01 left it, and `style-src`
+gains `'unsafe-inline'`. That relaxation is bounded on purpose: with `script-src` intact
+no injected markup can execute, and with `img-src`, `connect-src`, and `font-src` all
+still `'self'`, the CSS-based exfiltration channels stay closed. What remains is
+defacement of a page the operator is already authenticated on.
+
+The rejected alternative is recorded because it may become the better trade later:
+pre-rendering each diagram to a committed SVG with the already-installed Playwright
+(no Node toolchain needed) and serving that, which would leave CSP untouched entirely at
+the cost of a generator and a freshness check. If `style-src 'unsafe-inline'` is ever
+judged too much, that is the migration, and it does not change a single markdown file.
+
+**Screenshots are produced, not pasted.** Every screenshot in the user manual is captured
+by Playwright from the real application against a temporary fixture library, by the same
+kind of code that already drives the browser suites. A screenshot nobody can regenerate
+is a screenshot that silently stops being true.
+
+**The documentation is held to the code by tests.** Configuration keys, CLI commands,
+exit codes, API error codes, job and journal states, and module names all appear in both
+the code and the manuals. Each of those is cross-checked, so the failure mode of stale
+documentation is a red test rather than a misled operator.
+
+## Stories
+
+1. [US09-01 — Serve the documentation inside the application](stories/US09-01-docs-in-app.md)
+2. [US09-02 — Write the installation and operations manual](stories/US09-02-installation-manual.md)
+3. [US09-03 — Write the architecture overview](stories/US09-03-architecture-overview.md)
+4. [US09-04 — Write the user manual with generated screenshots](stories/US09-04-user-manual.md)
+5. [US09-05 — Automate documentation acceptance](stories/US09-05-docs-gate.md)
+
+## Epic outcome
+
+A deployed instance serves its own installation manual, architecture overview, and
+illustrated user manual at `/app/#/docs`, rendered from the same markdown files that are
+readable in the repository. Screenshots are regenerated from the running application,
+every documented command, setting, state, and error code is cross-checked against the
+code that implements it, and one gate fails the build when documentation and application
+disagree.
diff --git a/delivery_backlog/README.md b/delivery_backlog/README.md
index 5641fc1..fd69b62 100644
--- a/delivery_backlog/README.md
+++ b/delivery_backlog/README.md
@@ -2,13 +2,14 @@
 
 This backlog decomposes the phases in
 [`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md) into seven
-epics and small, independently verifiable user stories, plus one delivery-format
-epic (E08) that packages the released application as a deployable container.
+epics and small, independently verifiable user stories, plus two delivery-format
+epics: E08 packages the released application as a deployable container, and E09
+documents it for the people who install, operate, and use it.
 
 ## Numbering and file naming
 
-- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` has no
-  concept phase and must not change product scope.
+- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` and `E09`
+  have no concept phase and must not change product scope.
 - Stories: `US<epic>-<sequence>`, for example `US03-02`.
 - Epic files: `E01-<slug>.md`.
 - Story files: `stories/US01-01-<slug>.md`.
@@ -39,6 +40,7 @@ epic (E08) that packages the released application as a deployable container.
 6. [E06 — Archive lifecycle](E06-archive-lifecycle.md)
 7. [E07 — Hardening and release](E07-hardening-release.md)
 8. [E08 — Container deployment](E08-container-deployment.md)
+9. [E09 — Product documentation](E09-documentation.md)
 
 ## Shared definition of done
 
diff --git a/delivery_backlog/stories/US09-01-docs-in-app.md b/delivery_backlog/stories/US09-01-docs-in-app.md
new file mode 100644
index 0000000..6d0f166
--- /dev/null
+++ b/delivery_backlog/stories/US09-01-docs-in-app.md
@@ -0,0 +1,49 @@
+# US09-01 — Serve the Documentation Inside the Application
+
+Epic: [E09](../E09-documentation.md)
+
+As an operator who was handed a URL and an access secret, I want the manuals inside the
+application I am looking at, so that understanding it does not require a repository
+checkout or an internet connection.
+
+## Acceptance criteria
+
+- Documentation lives as markdown under `docs/`, with an index that names every page and
+  the order it is meant to be read in. The same files render on Gitea without
+  modification.
+- The application serves a `/docs` view reachable from the main navigation, listing the
+  pages and rendering the selected one.
+- Rendering uses a vendored, version-pinned markdown library committed under
+  `frontend/js/vendor/`. It is not fetched from a CDN, not bundled by a build step, and
+  not written by hand. The build records the version and a checksum of the vendored file.
+- Diagrams written as ```mermaid``` fenced blocks render as diagrams. Mermaid is vendored
+  the same way.
+- `script-src` in the Content-Security-Policy is unchanged: no `'unsafe-eval'`, no
+  `'unsafe-inline'`. Only `style-src` gains `'unsafe-inline'`, and the reason is recorded
+  where the policy is defined.
+- Links between documents work in the app: a relative `../foo.md#section` link navigates
+  to that page and heading rather than downloading a file or leaving the application.
+  Every heading has a stable anchor, and a deep link to an anchor scrolls to it.
+- An unknown page renders a documentation-specific not-found message with a link back to
+  the index, and never exposes a filesystem path.
+- Documentation is readable without an access secret **or** behind the same session as
+  the rest of the application — whichever is chosen is stated explicitly in the story's
+  implementation notes and covered by a test, because an operator locked out by a
+  configuration mistake is exactly who needs the troubleshooting page.
+- The view works with no outbound network access at all.
+
+## Automated tests
+
+- Unit: heading-anchor slugs (duplicates, punctuation, non-ASCII), and the rewriting of
+  relative markdown links into in-app routes.
+- Integration: every markdown file under `docs/` is reachable from the index; every
+  internal link in every document resolves to a file and, where an anchor is given, to a
+  heading that exists; the vendored library files match their recorded checksums.
+- Browser: the documentation view renders a page, follows an internal link, deep-links to
+  an anchor, renders a mermaid diagram to an SVG, and shows the not-found message for an
+  unknown page — all with **zero console errors and zero CSP violations**, asserted, not
+  eyeballed.
+
+## Dependencies
+
+- None beyond the delivered application.
diff --git a/delivery_backlog/stories/US09-02-installation-manual.md b/delivery_backlog/stories/US09-02-installation-manual.md
new file mode 100644
index 0000000..7e86b4f
--- /dev/null
+++ b/delivery_backlog/stories/US09-02-installation-manual.md
@@ -0,0 +1,46 @@
+# US09-02 — Write the Installation and Operations Manual
+
+Epic: [E09](../E09-documentation.md)
+
+As somebody installing this application for the first time, I want one manual that takes
+me from nothing to a running instance pointed at my photo library, so that I do not have
+to reconstruct the procedure from the README, the compose file, and the test suite.
+
+## Acceptance criteria
+
+- A **host installation** path: prerequisites and their versions (Python, exiftool,
+  immich-go, Playwright's browser for the test suite), virtual environment, dependency
+  install, `.env`, database migration, starting `serve` and `worker`, and how to verify
+  the install succeeded.
+- A **container installation** path: the published image, the compose file, the data
+  volume, the library bind mount and why it is mounted the way it is, published ports,
+  running behind a reverse proxy with `PHOTO_PIPELINE_ALLOWED_HOSTS` and
+  `PHOTO_PIPELINE_TRUSTED_PROXIES`, and the Portainer/webhook deployment already in use.
+- A **configuration reference**: every `PHOTO_PIPELINE_*` setting with its meaning,
+  default, accepted values, and whether it is a secret. Secrets are described, never
+  exemplified with a real-looking value.
+- A **first run checklist** that ends in a verified state: library discovered, worker
+  claiming jobs, readiness endpoint green, diagnostics clean.
+- **Operations**: upgrading (including what the migration backup does), backup, verify,
+  restore, pruning, diagnostics, and the log locations for each role.
+- **Troubleshooting**: each refusal an operator can hit at startup — every non-zero exit
+  code of the CLI, the trust-boundary refusal, the unmounted library root refusal, the
+  library lock being held, and a legacy CLI still running — with the cause and the fix.
+- The manual states the safety invariants an installer must not work around: one worker
+  writes, `_IGNORE/` is never read, EXIF is verified before upload, and the library lock
+  is authoritative.
+
+## Automated tests
+
+- Every `PHOTO_PIPELINE_*` setting documented exists as a field on `Config`, and every
+  field on `Config` is documented — both directions, so a new setting cannot ship
+  undocumented.
+- Every CLI command and flag named in the manual exists in the argument parser, and every
+  subcommand the parser accepts appears in the manual.
+- Every exit code documented is one the CLI can actually return.
+- No documented example value collides with a real secret pattern (the repository's own
+  credential scanner runs over `docs/`).
+
+## Dependencies
+
+- US09-01
diff --git a/delivery_backlog/stories/US09-03-architecture-overview.md b/delivery_backlog/stories/US09-03-architecture-overview.md
new file mode 100644
index 0000000..19f512c
--- /dev/null
+++ b/delivery_backlog/stories/US09-03-architecture-overview.md
@@ -0,0 +1,47 @@
+# US09-03 — Write the Architecture Overview
+
+Epic: [E09](../E09-documentation.md)
+
+As a developer or reviewer new to this repository, I want an architecture overview that
+explains what the pieces are and which rules they enforce, so that I can find the right
+module and not violate an invariant I never knew existed.
+
+## Acceptance criteria
+
+- A **context diagram**: the operator, the browser application, the API and worker, the
+  photo library, the Immich server, the vision provider, and the archive destination —
+  with the direction and nature of every interaction, including which ones leave the
+  machine.
+- A **runtime diagram**: the migrate/api/worker composition, the data volume, the library
+  bind mount, the SQLite database, and the process lock — showing what is shared and what
+  is exclusive.
+- A **module map**: every package under `photo_pipeline/` with its responsibility and the
+  boundary it must not cross (which modules may touch the filesystem, which may call an
+  external provider, which own schema).
+- **Key flows**, each as a diagram plus prose: the durable job lifecycle from enqueue to
+  recovery; the rename journal state machine including the rollback and manual-resolution
+  paths; the upload lifecycle through preflight, run, report ingestion, and verification.
+- **The invariants and where they are enforced**, each pointing at the module that owns
+  it: one writer at a time, the library process lock, the path policy and library roots,
+  `_IGNORE/` exclusion, verified EXIF before upload, safety decisions gating the vision
+  provider, and restart-safety of every mutation.
+- **The data model**: the tables, what identity means for an asset, and how a moved file
+  keeps its identity.
+- A short **history and donor** section: what was migrated out of the legacy CLIs, where
+  the archive and its ledger live, and why they are kept.
+- Diagrams are ```mermaid``` blocks so the source is diffable and Gitea renders them
+  natively.
+
+## Automated tests
+
+- Every package and top-level module under `photo_pipeline/` appears in the module map,
+  and every module named in the map exists — a new service cannot appear on the map's
+  blind side.
+- Every job state, journal state, and upload batch state named in the document exists in
+  the code, and every state the code defines is named in the document.
+- Every mermaid block parses (rendered in the browser test without an error node).
+- Every code path referenced by file name exists.
+
+## Dependencies
+
+- US09-01
diff --git a/delivery_backlog/stories/US09-04-user-manual.md b/delivery_backlog/stories/US09-04-user-manual.md
new file mode 100644
index 0000000..75b5550
--- /dev/null
+++ b/delivery_backlog/stories/US09-04-user-manual.md
@@ -0,0 +1,52 @@
+# US09-04 — Write the User Manual with Generated Screenshots
+
+Epic: [E09](../E09-documentation.md)
+
+As the person actually sorting a photo library, I want a manual that walks the workflow
+screen by screen and tells me what each refusal means, so that I can use the application
+confidently and know what it is about to do to my files before it does it.
+
+## Acceptance criteria
+
+- One page per workflow stage, in the order the application presents them: discovery and
+  inventory, duplicate review, safety review, analysis, album proposals, renames, upload,
+  archive, and diagnostics. Each page answers the same four questions: what this stage is
+  for, what I have to decide, **what it changes on disk or on the server**, and what it
+  refuses to do.
+- A **guided first pass** that takes a new library from scan to a verified upload, naming
+  the point of no return in each stage and what is reversible after it.
+- Every stage page carries at least one screenshot of the real application showing the
+  state being described.
+- Screenshots are **generated** by a committed Playwright script that seeds a temporary
+  fixture library, drives the application, and writes the images. Regenerating them is one
+  documented command. No screenshot is captured by hand.
+- Screenshots contain no real library paths, no personal photos, and no secrets — the
+  fixture library is synthetic, and this is asserted rather than assumed.
+- An **error and refusal catalogue**: every `error.code` the API can return, with what
+  causes it, what the application did or refused to do, and what the operator should do
+  next. Refusals that protect data (`lock_held`, `rename_recovery_required`,
+  `stale_preflight`, `path_not_allowed`, `host_not_allowed`, `dry_run_not_approved`,
+  conflict codes) are explained as intentional, not as faults.
+- A **recovery** page: an interrupted rename, an uncertain upload, a failed migration, a
+  restored backup — what the application does by itself and what needs a decision.
+- The work-item safety allow list is extended to permit `docs/images/**`, since the
+  repository denies image files by default. This is an explicit, reviewed change, not a
+  quiet one, and it stays narrow enough that a real photo still cannot be committed.
+
+## Automated tests
+
+- Every `error.code` the application can emit is documented, and every code documented
+  exists in the code — both directions.
+- Every image referenced by a documentation page exists, and every image under
+  `docs/images/` is referenced by a page.
+- The screenshot generator runs end to end in the test environment and produces every
+  image the manual references, against a temporary fixture library that is destroyed
+  afterwards.
+- Generated screenshots are checked for library paths outside the fixture root and for
+  the configured secrets.
+- Browser: each stage page renders in the documentation view with its screenshot loaded
+  and no console error.
+
+## Dependencies
+
+- US09-01
diff --git a/delivery_backlog/stories/US09-05-docs-gate.md b/delivery_backlog/stories/US09-05-docs-gate.md
new file mode 100644
index 0000000..3d9e337
--- /dev/null
+++ b/delivery_backlog/stories/US09-05-docs-gate.md
@@ -0,0 +1,38 @@
+# US09-05 — Automate Documentation Acceptance
+
+Epic: [E09](../E09-documentation.md)
+
+As a release owner, I want one gate that proves the documentation still describes the
+application, so that a change to the code cannot quietly make the manuals wrong.
+
+## Acceptance criteria
+
+- One documented command runs every documentation check and retains its evidence, in the
+  shape the release and container gates already use.
+- The gate fails when: an internal link or anchor is dead; a page is unreachable from the
+  index; an image is referenced but missing or present but unreferenced; a documented
+  setting, command, exit code, error code, or state does not exist in the code; a code
+  path exists that the documentation is required to cover and does not.
+- The gate regenerates the screenshots and fails when a regenerated image no longer
+  matches the committed one beyond a stated tolerance, so a UI change that invalidates the
+  manual is a red build rather than a discovery months later.
+- The gate renders every documentation page in a real browser and fails on any console
+  error or CSP violation, and asserts that `script-src` contains neither `'unsafe-eval'`
+  nor `'unsafe-inline'`.
+- The checks run on a `phase_i` marker; CI runs the gate, and the earlier epic suites keep
+  running unchanged.
+- The README and the application's documentation index point at each other, so neither is
+  the forgotten copy.
+- Documentation stories are mapped in the story traceability matrix like every other
+  story.
+
+## Automated tests
+
+- The gate's own contract is testable without a browser: a seeded broken link, a missing
+  image, an undocumented error code, and a stale screenshot each fail it, and a clean tree
+  passes.
+- The full documentation suite runs on `phase_i` in CI.
+
+## Dependencies
+
+- US09-01 through US09-04
diff --git a/tests/story_traceability.json b/tests/story_traceability.json
index b6a63fb..a7b6a12 100644
--- a/tests/story_traceability.json
+++ b/tests/story_traceability.json
@@ -195,6 +195,12 @@
       "tests/e2e/test_phase_h_container.py"
     ]
   },
-  "planned": [],
+  "planned": [
+    "US09-01",
+    "US09-02",
+    "US09-03",
+    "US09-04",
+    "US09-05"
+  ],
   "_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
 }
-- 
2.49.1