E09: Product documentation backlog (#101)

delivery_backlog/E09-documentation.md

@@ -0,0 +1,77 @@
+# E09 — Product Documentation
+
+Concept phase: none. Like [E08](E08-container-deployment.md), this epic is a delivery
+addition rather than a product-scope change: the same application, documented well
+enough that somebody who did not build it can install it, understand it, and operate it
+without reading the source.
+
+It does not change the product scope in
+[`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md). No safety
+invariant moves, no schema changes, no new runtime capability. The one change to the
+running application is a documentation view and the static assets it needs.
+
+## Why the application serves its own documentation
+
+The manuals describe an application that is reached over HTTP behind an access secret.
+Documentation that lives only in the repository is unreachable from the deployment it
+describes: an operator who has just been handed a URL and a secret has no Gitea account
+in front of them. So the same markdown files are both the repository's documentation and
+the deployment's `/docs` view, and neither is a copy of the other.
+
+## Decisions made in this epic
+
+**Markdown is the source.** Everything is written as markdown under `docs/`, so it is
+reviewable in a diff, readable on Gitea, and renderable in the app. No documentation
+format that only a tool can read.
+
+**The renderer is vendored, not written and not fetched.** `marked` (MIT, no
+dependencies, ships an ES module) is pinned and committed under
+`frontend/js/vendor/`. A CDN is not an option: the application is deployed to a network
+whose outbound access is not assumed, and `default-src 'self'` forbids it.
+
+**Diagrams are mermaid, rendered client-side, with the script boundary intact.** The
+claim that mermaid requires `'unsafe-eval'` was tested rather than believed: mermaid 11's
+bundle contains no `eval(` and no `new Function` — only a lodash `Function("return this")`
+global-detection fallback that short-circuits on `globalThis` and never executes.
+Rendered under this application's exact CSP, a flowchart produced a 15.8 KB SVG and
+raised **no `script-src` violation**. What it does raise is `style-src`: mermaid styles
+its output with an injected `<style>` element and `style=` attributes.
+
+So `script-src 'self'` stays exactly as US07-02 and US08-01 left it, and `style-src`
+gains `'unsafe-inline'`. That relaxation is bounded on purpose: with `script-src` intact
+no injected markup can execute, and with `img-src`, `connect-src`, and `font-src` all
+still `'self'`, the CSS-based exfiltration channels stay closed. What remains is
+defacement of a page the operator is already authenticated on.
+
+The rejected alternative is recorded because it may become the better trade later:
+pre-rendering each diagram to a committed SVG with the already-installed Playwright
+(no Node toolchain needed) and serving that, which would leave CSP untouched entirely at
+the cost of a generator and a freshness check. If `style-src 'unsafe-inline'` is ever
+judged too much, that is the migration, and it does not change a single markdown file.
+
+**Screenshots are produced, not pasted.** Every screenshot in the user manual is captured
+by Playwright from the real application against a temporary fixture library, by the same
+kind of code that already drives the browser suites. A screenshot nobody can regenerate
+is a screenshot that silently stops being true.
+
+**The documentation is held to the code by tests.** Configuration keys, CLI commands,
+exit codes, API error codes, job and journal states, and module names all appear in both
+the code and the manuals. Each of those is cross-checked, so the failure mode of stale
+documentation is a red test rather than a misled operator.
+
+## Stories
+
+1. [US09-01 — Serve the documentation inside the application](stories/US09-01-docs-in-app.md)
+2. [US09-02 — Write the installation and operations manual](stories/US09-02-installation-manual.md)
+3. [US09-03 — Write the architecture overview](stories/US09-03-architecture-overview.md)
+4. [US09-04 — Write the user manual with generated screenshots](stories/US09-04-user-manual.md)
+5. [US09-05 — Automate documentation acceptance](stories/US09-05-docs-gate.md)
+
+## Epic outcome
+
+A deployed instance serves its own installation manual, architecture overview, and
+illustrated user manual at `/app/#/docs`, rendered from the same markdown files that are
+readable in the repository. Screenshots are regenerated from the running application,
+every documented command, setting, state, and error code is cross-checked against the
+code that implements it, and one gate fails the build when documentation and application
+disagree.

delivery_backlog/README.md

@@ -2,13 +2,14 @@
 
 This backlog decomposes the phases in
 [`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md) into seven
-epics and small, independently verifiable user stories, plus one delivery-format
-epic (E08) that packages the released application as a deployable container.
+epics and small, independently verifiable user stories, plus two delivery-format
+epics: E08 packages the released application as a deployable container, and E09
+documents it for the people who install, operate, and use it.
 
 ## Numbering and file naming
 
-- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` has no
-  concept phase and must not change product scope.
+- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` and `E09`
+  have no concept phase and must not change product scope.
 - Stories: `US<epic>-<sequence>`, for example `US03-02`.
 - Epic files: `E01-<slug>.md`.
 - Story files: `stories/US01-01-<slug>.md`.
@@ -39,6 +40,7 @@ epic (E08) that packages the released application as a deployable container.
 6. [E06 — Archive lifecycle](E06-archive-lifecycle.md)
 7. [E07 — Hardening and release](E07-hardening-release.md)
 8. [E08 — Container deployment](E08-container-deployment.md)
+9. [E09 — Product documentation](E09-documentation.md)
 
 ## Shared definition of done
 

delivery_backlog/stories/US09-01-docs-in-app.md

@@ -0,0 +1,49 @@
+# US09-01 — Serve the Documentation Inside the Application
+
+Epic: [E09](../E09-documentation.md)
+
+As an operator who was handed a URL and an access secret, I want the manuals inside the
+application I am looking at, so that understanding it does not require a repository
+checkout or an internet connection.
+
+## Acceptance criteria
+
+- Documentation lives as markdown under `docs/`, with an index that names every page and
+  the order it is meant to be read in. The same files render on Gitea without
+  modification.
+- The application serves a `/docs` view reachable from the main navigation, listing the
+  pages and rendering the selected one.
+- Rendering uses a vendored, version-pinned markdown library committed under
+  `frontend/js/vendor/`. It is not fetched from a CDN, not bundled by a build step, and
+  not written by hand. The build records the version and a checksum of the vendored file.
+- Diagrams written as ```mermaid``` fenced blocks render as diagrams. Mermaid is vendored
+  the same way.
+- `script-src` in the Content-Security-Policy is unchanged: no `'unsafe-eval'`, no
+  `'unsafe-inline'`. Only `style-src` gains `'unsafe-inline'`, and the reason is recorded
+  where the policy is defined.
+- Links between documents work in the app: a relative `../foo.md#section` link navigates
+  to that page and heading rather than downloading a file or leaving the application.
+  Every heading has a stable anchor, and a deep link to an anchor scrolls to it.
+- An unknown page renders a documentation-specific not-found message with a link back to
+  the index, and never exposes a filesystem path.
+- Documentation is readable without an access secret **or** behind the same session as
+  the rest of the application — whichever is chosen is stated explicitly in the story's
+  implementation notes and covered by a test, because an operator locked out by a
+  configuration mistake is exactly who needs the troubleshooting page.
+- The view works with no outbound network access at all.
+
+## Automated tests
+
+- Unit: heading-anchor slugs (duplicates, punctuation, non-ASCII), and the rewriting of
+  relative markdown links into in-app routes.
+- Integration: every markdown file under `docs/` is reachable from the index; every
+  internal link in every document resolves to a file and, where an anchor is given, to a
+  heading that exists; the vendored library files match their recorded checksums.
+- Browser: the documentation view renders a page, follows an internal link, deep-links to
+  an anchor, renders a mermaid diagram to an SVG, and shows the not-found message for an
+  unknown page — all with **zero console errors and zero CSP violations**, asserted, not
+  eyeballed.
+
+## Dependencies
+
+- None beyond the delivered application.

delivery_backlog/stories/US09-02-installation-manual.md

@@ -0,0 +1,46 @@
+# US09-02 — Write the Installation and Operations Manual
+
+Epic: [E09](../E09-documentation.md)
+
+As somebody installing this application for the first time, I want one manual that takes
+me from nothing to a running instance pointed at my photo library, so that I do not have
+to reconstruct the procedure from the README, the compose file, and the test suite.
+
+## Acceptance criteria
+
+- A **host installation** path: prerequisites and their versions (Python, exiftool,
+  immich-go, Playwright's browser for the test suite), virtual environment, dependency
+  install, `.env`, database migration, starting `serve` and `worker`, and how to verify
+  the install succeeded.
+- A **container installation** path: the published image, the compose file, the data
+  volume, the library bind mount and why it is mounted the way it is, published ports,
+  running behind a reverse proxy with `PHOTO_PIPELINE_ALLOWED_HOSTS` and
+  `PHOTO_PIPELINE_TRUSTED_PROXIES`, and the Portainer/webhook deployment already in use.
+- A **configuration reference**: every `PHOTO_PIPELINE_*` setting with its meaning,
+  default, accepted values, and whether it is a secret. Secrets are described, never
+  exemplified with a real-looking value.
+- A **first run checklist** that ends in a verified state: library discovered, worker
+  claiming jobs, readiness endpoint green, diagnostics clean.
+- **Operations**: upgrading (including what the migration backup does), backup, verify,
+  restore, pruning, diagnostics, and the log locations for each role.
+- **Troubleshooting**: each refusal an operator can hit at startup — every non-zero exit
+  code of the CLI, the trust-boundary refusal, the unmounted library root refusal, the
+  library lock being held, and a legacy CLI still running — with the cause and the fix.
+- The manual states the safety invariants an installer must not work around: one worker
+  writes, `_IGNORE/` is never read, EXIF is verified before upload, and the library lock
+  is authoritative.
+
+## Automated tests
+
+- Every `PHOTO_PIPELINE_*` setting documented exists as a field on `Config`, and every
+  field on `Config` is documented — both directions, so a new setting cannot ship
+  undocumented.
+- Every CLI command and flag named in the manual exists in the argument parser, and every
+  subcommand the parser accepts appears in the manual.
+- Every exit code documented is one the CLI can actually return.
+- No documented example value collides with a real secret pattern (the repository's own
+  credential scanner runs over `docs/`).
+
+## Dependencies
+
+- US09-01

delivery_backlog/stories/US09-03-architecture-overview.md

@@ -0,0 +1,47 @@
+# US09-03 — Write the Architecture Overview
+
+Epic: [E09](../E09-documentation.md)
+
+As a developer or reviewer new to this repository, I want an architecture overview that
+explains what the pieces are and which rules they enforce, so that I can find the right
+module and not violate an invariant I never knew existed.
+
+## Acceptance criteria
+
+- A **context diagram**: the operator, the browser application, the API and worker, the
+  photo library, the Immich server, the vision provider, and the archive destination —
+  with the direction and nature of every interaction, including which ones leave the
+  machine.
+- A **runtime diagram**: the migrate/api/worker composition, the data volume, the library
+  bind mount, the SQLite database, and the process lock — showing what is shared and what
+  is exclusive.
+- A **module map**: every package under `photo_pipeline/` with its responsibility and the
+  boundary it must not cross (which modules may touch the filesystem, which may call an
+  external provider, which own schema).
+- **Key flows**, each as a diagram plus prose: the durable job lifecycle from enqueue to
+  recovery; the rename journal state machine including the rollback and manual-resolution
+  paths; the upload lifecycle through preflight, run, report ingestion, and verification.
+- **The invariants and where they are enforced**, each pointing at the module that owns
+  it: one writer at a time, the library process lock, the path policy and library roots,
+  `_IGNORE/` exclusion, verified EXIF before upload, safety decisions gating the vision
+  provider, and restart-safety of every mutation.
+- **The data model**: the tables, what identity means for an asset, and how a moved file
+  keeps its identity.
+- A short **history and donor** section: what was migrated out of the legacy CLIs, where
+  the archive and its ledger live, and why they are kept.
+- Diagrams are ```mermaid``` blocks so the source is diffable and Gitea renders them
+  natively.
+
+## Automated tests
+
+- Every package and top-level module under `photo_pipeline/` appears in the module map,
+  and every module named in the map exists — a new service cannot appear on the map's
+  blind side.
+- Every job state, journal state, and upload batch state named in the document exists in
+  the code, and every state the code defines is named in the document.
+- Every mermaid block parses (rendered in the browser test without an error node).
+- Every code path referenced by file name exists.
+
+## Dependencies
+
+- US09-01

delivery_backlog/stories/US09-04-user-manual.md

@@ -0,0 +1,52 @@
+# US09-04 — Write the User Manual with Generated Screenshots
+
+Epic: [E09](../E09-documentation.md)
+
+As the person actually sorting a photo library, I want a manual that walks the workflow
+screen by screen and tells me what each refusal means, so that I can use the application
+confidently and know what it is about to do to my files before it does it.
+
+## Acceptance criteria
+
+- One page per workflow stage, in the order the application presents them: discovery and
+  inventory, duplicate review, safety review, analysis, album proposals, renames, upload,
+  archive, and diagnostics. Each page answers the same four questions: what this stage is
+  for, what I have to decide, **what it changes on disk or on the server**, and what it
+  refuses to do.
+- A **guided first pass** that takes a new library from scan to a verified upload, naming
+  the point of no return in each stage and what is reversible after it.
+- Every stage page carries at least one screenshot of the real application showing the
+  state being described.
+- Screenshots are **generated** by a committed Playwright script that seeds a temporary
+  fixture library, drives the application, and writes the images. Regenerating them is one
+  documented command. No screenshot is captured by hand.
+- Screenshots contain no real library paths, no personal photos, and no secrets — the
+  fixture library is synthetic, and this is asserted rather than assumed.
+- An **error and refusal catalogue**: every `error.code` the API can return, with what
+  causes it, what the application did or refused to do, and what the operator should do
+  next. Refusals that protect data (`lock_held`, `rename_recovery_required`,
+  `stale_preflight`, `path_not_allowed`, `host_not_allowed`, `dry_run_not_approved`,
+  conflict codes) are explained as intentional, not as faults.
+- A **recovery** page: an interrupted rename, an uncertain upload, a failed migration, a
+  restored backup — what the application does by itself and what needs a decision.
+- The work-item safety allow list is extended to permit `docs/images/**`, since the
+  repository denies image files by default. This is an explicit, reviewed change, not a
+  quiet one, and it stays narrow enough that a real photo still cannot be committed.
+
+## Automated tests
+
+- Every `error.code` the application can emit is documented, and every code documented
+  exists in the code — both directions.
+- Every image referenced by a documentation page exists, and every image under
+  `docs/images/` is referenced by a page.
+- The screenshot generator runs end to end in the test environment and produces every
+  image the manual references, against a temporary fixture library that is destroyed
+  afterwards.
+- Generated screenshots are checked for library paths outside the fixture root and for
+  the configured secrets.
+- Browser: each stage page renders in the documentation view with its screenshot loaded
+  and no console error.
+
+## Dependencies
+
+- US09-01

delivery_backlog/stories/US09-05-docs-gate.md

@@ -0,0 +1,38 @@
+# US09-05 — Automate Documentation Acceptance
+
+Epic: [E09](../E09-documentation.md)
+
+As a release owner, I want one gate that proves the documentation still describes the
+application, so that a change to the code cannot quietly make the manuals wrong.
+
+## Acceptance criteria
+
+- One documented command runs every documentation check and retains its evidence, in the
+  shape the release and container gates already use.
+- The gate fails when: an internal link or anchor is dead; a page is unreachable from the
+  index; an image is referenced but missing or present but unreferenced; a documented
+  setting, command, exit code, error code, or state does not exist in the code; a code
+  path exists that the documentation is required to cover and does not.
+- The gate regenerates the screenshots and fails when a regenerated image no longer
+  matches the committed one beyond a stated tolerance, so a UI change that invalidates the
+  manual is a red build rather than a discovery months later.
+- The gate renders every documentation page in a real browser and fails on any console
+  error or CSP violation, and asserts that `script-src` contains neither `'unsafe-eval'`
+  nor `'unsafe-inline'`.
+- The checks run on a `phase_i` marker; CI runs the gate, and the earlier epic suites keep
+  running unchanged.
+- The README and the application's documentation index point at each other, so neither is
+  the forgotten copy.
+- Documentation stories are mapped in the story traceability matrix like every other
+  story.
+
+## Automated tests
+
+- The gate's own contract is testable without a browser: a seeded broken link, a missing
+  image, an undocumented error code, and a stale screenshot each fail it, and a clean tree
+  passes.
+- The full documentation suite runs on `phase_i` in CI.
+
+## Dependencies
+
+- US09-01 through US09-04

tests/story_traceability.json

@@ -195,6 +195,12 @@
       "tests/e2e/test_phase_h_container.py"
     ]
   },
-  "planned": [],
+  "planned": [
+    "US09-01",
+    "US09-02",
+    "US09-03",
+    "US09-04",
+    "US09-05"
+  ],
   "_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
 }