US09-02: Write the Installation and Operations Manual

docs/index.md

@@ -12,6 +12,9 @@ repository under `docs/`, on Gitea, and inside the running application under
 
 1. [Overview](overview.md) — what the application does, the stages it moves a photo
    through, and the rules it will not break.
+2. [Installation and operations](installation.md) — host and container installation,
+   every setting, the first-run checklist, upgrades, backup and restore, and what
+   each refusal at startup means.
 
 ## Being written
 
@@ -19,9 +22,6 @@ The remaining manuals are accepted work, not aspiration; each is a story in
 [E09](https://git.domverse-berlin.eu/domverse/photoanalyzer/src/branch/main/delivery_backlog/E09-documentation.md)
 and will appear here as it lands.
 
-- **Installation and operations** — host and container installation, every setting,
-  the first-run checklist, upgrades, backup and restore, and what each refusal at
-  startup means (US09-02).
 - **Architecture** — context and runtime diagrams, the module map, the job and
   journal state machines, and where each invariant is enforced (US09-03).
 - **User manual** — one page per workflow stage with screenshots of the real

docs/installation.md

@@ -0,0 +1,336 @@
+# Installation and operations
+
+[← Documentation index](index.md)
+
+From nothing to a running instance pointed at your photo library, and everything you
+need afterwards: upgrading, backing up, restoring, and understanding a refusal.
+
+Two ways to install. **Container** is the deployment this project builds and ships;
+**host** is what you want for development or a single machine you already manage.
+Both run the same two processes against the same database.
+
+> Read [what it will not do](overview.md#what-it-will-not-do) first. Several of the
+> steps below only make sense once you know which rules the application is keeping.
+
+## Before you start
+
+| you need | version | why |
+|---|---|---|
+| Python | 3.12 or newer | the application |
+| `exiftool` | 13.x (the image pins `13.25+dfsg-1`) | every EXIF read and write |
+| `immich-go` | 0.32.0 (pinned in the image) | the upload stage only |
+| Docker + the compose plugin | any current | the container installation only |
+| a vision provider key | — | the analysis stage only |
+
+You also need a photo library you can afford to be wrong about. Take a backup of it
+before pointing anything at it for the first time, and consider running with
+[`PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL`](#the-settings) on.
+
+## Host installation
+
+```bash
+python3.12 -m venv .venv
+.venv/bin/pip install -e ".[vision]"     # drop [vision] for a review-only install
+```
+
+Configure it. Every setting is an environment variable; a `.env` file in the working
+directory is read at startup:
+
+```bash
+cp .env.example .env && $EDITOR .env
+```
+
+`.env.example` lists every variable with its default and no values at all. For a
+loopback installation you need exactly one:
+
+```bash
+PHOTO_PIPELINE_LIBRARY_ROOTS=/home/you/Pictures
+```
+
+Then migrate and start the two processes:
+
+```bash
+.venv/bin/python -m photo_pipeline migrate    # create or upgrade the database
+.venv/bin/python -m photo_pipeline serve      # API and browser app on 127.0.0.1:8000
+.venv/bin/python -m photo_pipeline worker     # second terminal
+```
+
+**Both processes are required.** `serve` enqueues work and renders the application;
+nothing is scanned, scored, analysed, renamed, uploaded, or archived without a
+worker. Open <http://127.0.0.1:8000/app/> and confirm the workflow page loads.
+
+### The configuration file
+
+`.env`, or any path named by `PHOTO_PIPELINE_ENV_FILE`. It is parsed, never executed:
+`KEY=value` lines, `#` comments, optional quotes, no interpolation and no `export`. A
+configuration file that can run code is a configuration file that can be a
+vulnerability.
+
+**Anything already exported in the shell wins.** The file is your standing
+configuration; the environment is the override for one run.
+
+The archived CLI's names still work, so an existing `photo_analyzer.env` can be used
+as it is:
+
+| in the file | applied as |
+|---|---|
+| `LLM_API_KEY` / `GEMINI_API_KEY` | `OPENAI_API_KEY` |
+| `LLM_BASE_URL` | `OPENAI_BASE_URL` |
+| `LIBRARY` | `PHOTO_PIPELINE_LIBRARY_ROOTS` |
+
+`.env` and `*.env` are gitignored and refused by the repository's safety checks. The
+file holds a real key; it must never be committed.
+
+## Container installation
+
+```bash
+cp .env.example .env && $EDITOR .env
+docker compose up -d --build
+```
+
+The composition is one `serve` container, one `worker` container, a one-shot
+`migrate` that both wait for, one bind-mounted library, and one named data volume.
+
+Three variables it cannot start without:
+
+| variable | is |
+|---|---|
+| `PHOTO_PIPELINE_LIBRARY_HOST_PATH` | the library on this host |
+| `PHOTO_PIPELINE_LIBRARY_ROOTS` | where that library is mounted **inside** the container |
+| `PHOTO_PIPELINE_ACCESS_SECRET` | required, because publishing a port means the app is reachable from outside the container |
+
+Set `PHOTO_PIPELINE_UID` and `PHOTO_PIPELINE_GID` to the owner of the library: what
+the containers rename and rewrite keeps that ownership.
+
+### The data volume
+
+The `data` volume holds the database, its write-ahead log, the thumbnail cache, the
+operation journals, and the backups. **It must stay on a local filesystem.** SQLite
+in WAL mode needs real local locking, so NFS, SMB, and network volume drivers do not
+slow it down — they corrupt it. The library bind mount has no such restriction.
+
+### Ports and reaching it
+
+The API port is published to `127.0.0.1` unless `PHOTO_PIPELINE_PUBLISH_ADDRESS` says
+otherwise. Being reachable *was* the authentication in earlier versions: whoever could
+open the port owned the library. So the moment the application answers to anything but
+loopback — a hostname, a proxy, `0.0.0.0` — the access secret becomes mandatory and
+`serve` refuses to start without it rather than publishing your photographs.
+
+Behind a reverse proxy:
+
+```bash
+PHOTO_PIPELINE_ALLOWED_HOSTS=photos.example.com
+PHOTO_PIPELINE_ACCESS_SECRET=…                  # python -c 'import secrets; print(secrets.token_urlsafe(32))'
+PHOTO_PIPELINE_TRUSTED_PROXIES=10.0.0.2         # only the proxy's own address
+```
+
+`X-Forwarded-Proto` and `X-Forwarded-Host` are believed only from a trusted-proxy
+address, so a client cannot declare its own origin. The browser asks for the secret
+once per tab. Wrong secrets are rate-limited and logged with the caller's address
+only. The health endpoints stay open so an orchestrator can restart the container;
+nothing else is.
+
+### Deployment
+
+The stack is managed from git by Portainer, which redeploys on a webhook after CI
+publishes an image built from `main`. Runtime secrets live in the Portainer stack's
+environment rather than in the repository, so rotation happens in one place and
+reading the repository discloses nothing.
+
+## The settings
+
+Every variable, its default, and whether it is a secret. `.env.example` is the same
+list in copyable form.
+
+### Library and data
+
+| variable | default | meaning |
+|---|---|---|
+| `PHOTO_PIPELINE_LIBRARY_ROOTS` | none | the library boundary, `os.pathsep`-separated. No path outside these roots is ever read or written |
+| `PHOTO_PIPELINE_DATA_DIR` | `data` | database, WAL, thumbnail cache, journals, backups. Never inside the library |
+| `PHOTO_PIPELINE_DB_PATH` | `<data dir>/photo_pipeline.db` | the database file, if it must live elsewhere |
+| `PHOTO_PIPELINE_ENV_FILE` | `.env` | where to read the configuration file from |
+
+### Serving and the trust boundary
+
+| variable | default | meaning |
+|---|---|---|
+| `PHOTO_PIPELINE_HOST` | `127.0.0.1` | bind address |
+| `PHOTO_PIPELINE_PORT` | `8000` | port |
+| `PHOTO_PIPELINE_ALLOWED_HOSTS` | none | comma-separated names the app answers to besides loopback. Naming one makes the access secret mandatory |
+| `PHOTO_PIPELINE_ACCESS_SECRET` | none | **secret.** Traded for the session cookie at `GET /api/v1/session` |
+| `PHOTO_PIPELINE_TRUSTED_PROXIES` | none | comma-separated peer addresses whose forwarded headers may be believed |
+| `PHOTO_PIPELINE_MAX_REQUEST_BYTES` | `1048576` | largest request body accepted |
+
+### Logging
+
+| variable | default | meaning |
+|---|---|---|
+| `PHOTO_PIPELINE_LOG_LEVEL` | `INFO` | standard Python levels |
+| `PHOTO_PIPELINE_LOG_FORMAT` | `json` | `json` or `text` |
+
+### Limits
+
+| variable | default | meaning |
+|---|---|---|
+| `PHOTO_PIPELINE_THUMBNAIL_CACHE_QUOTA_BYTES` | `500000000` | cache quota; over it is a diagnostics warning |
+| `PHOTO_PIPELINE_THUMBNAIL_MAX_PIXELS` | `100000000` | refuse to decode anything larger. This is the decompression-bomb guard |
+| `PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES` | `1000000000` | free space an archive destination must keep beyond the transfer itself |
+
+### The safety gate
+
+| variable | default | meaning |
+|---|---|---|
+| `PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL` | `false` | refuse every mutating request until a read-only dry run of this library has been produced and approved |
+
+### External services
+
+| variable | default | meaning |
+|---|---|---|
+| `PHOTO_PIPELINE_VISION_API_KEY` | none | **secret.** Without it the analysis stage cannot run |
+| `PHOTO_PIPELINE_IMMICH_SERVER_URL` | none | the Immich server |
+| `PHOTO_PIPELINE_IMMICH_API_KEY` | none | **secret.** |
+| `PHOTO_PIPELINE_IMMICH_GO_BINARY` | `immich-go` | the uploader, found on `PATH` |
+
+### Composition only
+
+Read by `docker-compose.yml`, not by the application:
+`PHOTO_PIPELINE_IMAGE`, `PHOTO_PIPELINE_LIBRARY_HOST_PATH`,
+`PHOTO_PIPELINE_PUBLISH_ADDRESS`, `PHOTO_PIPELINE_UID`, `PHOTO_PIPELINE_GID`.
+
+A secret is never written to a log, never returned by the API, never stored in the
+database, and never recorded in a backup manifest — a manifest says `configured`, not
+the value. Do not put a real key in an example, a ticket, or a screenshot.
+
+## First run checklist
+
+Not "it started" — verified.
+
+1. **The database is at the current revision.**
+   `python -m photo_pipeline migrate` exits 0.
+2. **The application answers.**
+   `curl -s localhost:8000/api/v1/health/ready` returns 200.
+3. **The library was found.** Open the app, run a scan from the Inventory view, and
+   check the count against what you expect. Anything under `_IGNORE/` must be missing
+   from it — that is the exclusion working, not a bug.
+4. **The worker is claiming.** The scan job reaches `succeeded`. If it stays
+   `queued`, no worker is running.
+5. **Diagnostics are clean.**
+   `python -m photo_pipeline diagnostics` reports free space and an empty `warnings`.
+6. **Nothing was modified.** The scan is read-only; your files' timestamps are
+   unchanged.
+
+Only then point it at the whole library.
+
+## Operations
+
+Every operation is the same CLI, on a host or in the composition:
+
+```bash
+python -m photo_pipeline diagnostics
+docker compose run --rm --no-deps api diagnostics
+```
+
+`--no-deps` keeps a one-off command from starting a second stack; `api` is only the
+service it borrows the image and mounts from.
+
+### Upgrading
+
+1. `python -m photo_pipeline backup --reason before-upgrade`
+2. Pull the new version (or `docker compose pull && docker compose up -d`).
+3. Migrations run by themselves at startup, and a **pending schema change is
+   snapshotted first**. If the upgrade fails, the previous database and its
+   `pre-migration` backup are both intact, and the error log names the backup
+   directory.
+
+An up-to-date database is not backed up again on every start.
+
+### Backing up
+
+```bash
+python -m photo_pipeline backup --reason weekly --keep 7
+python -m photo_pipeline verify-backup data/backups/<name>
+```
+
+Backups go through SQLite's online backup API, never a file copy: with WAL enabled
+the `.db` file alone is missing every committed page still in the write-ahead log.
+Each backup is a directory holding the snapshot and a `manifest.json` — schema
+revision, SHA-256, row counts, the archive media the library depends on, and which
+settings were configured.
+
+`verify-backup` runs `PRAGMA integrity_check` **and** `PRAGMA foreign_key_check`,
+compares the snapshot's SHA-256 against the manifest, and re-counts every table it
+recorded. Bit rot, a truncated copy, and a "repaired" snapshot all fail it.
+
+`--keep N` prunes the oldest and never the newest. The same is available at
+`GET /api/v1/diagnostics`, `GET|POST /api/v1/backups`,
+`GET /api/v1/backups/{name}/verify`, and `POST /api/v1/backups/prune`.
+
+### Restoring
+
+Restore is deliberately **not** an API call. It replaces the state of an
+installation, so it belongs to a stopped one and a person at a terminal.
+
+1. Stop `serve` and `worker`.
+2. `python -m photo_pipeline verify-backup data/backups/<name>` — never restore an
+   unverified snapshot, and `restore` will refuse one anyway.
+3. `python -m photo_pipeline restore data/backups/<name> --into /path/to/fresh-data`.
+   A target that already holds a database is refused; recovering in place means
+   moving the old data directory aside first.
+4. Point `PHOTO_PIPELINE_DATA_DIR` at the restored directory and run `migrate`.
+5. Run an inventory scan, so paths are reconciled against the real library.
+6. Mount every archive location the manifest names before archiving again. The
+   database records where archived originals are; it does not contain them.
+
+Practise this against a copy before you need it.
+
+### Watching it
+
+`diagnostics` reports the database, write-ahead log, thumbnail cache, uploader
+reports, backups, and logs separately, with free space and warnings for low disk
+(`disk_low`, `disk_critical`), a cache over its quota, a write-ahead log outgrowing
+its database, and a legacy CLI writing the library.
+
+Logs go to stdout in JSON by default — `docker compose logs -f worker`, or your
+service manager's journal on a host. Uploader output is kept per attempt under the
+data directory with the API key scrubbed line by line.
+
+## When it refuses to start
+
+Every refusal below is deliberate. The application would rather stop and explain than
+guess about somebody's photographs.
+
+| exit code | meaning | what to do |
+|---|---|---|
+| `1` | the operation failed and said why — an unverifiable backup, an occupied restore target, an unknown benchmark profile | read the message; nothing was changed |
+| `2` | **the library lock is held.** Another `serve` or `worker` owns this data directory | stop the other process. The message names the holder; a lock whose process is gone is taken over automatically |
+| `3` | **a legacy CLI looks active.** The frozen command-line tools are writing this library | stop them. `--allow-legacy` overrides and you own the outcome |
+| `4` | **configuration refused.** The application is reachable beyond loopback and no access secret is set | set `PHOTO_PIPELINE_ACCESS_SECRET`, or bind to loopback only |
+| `5` | **a library root is not mounted** (containers only) | fix the bind mount. Refusing is what stops the container writing into its own throwaway layer instead of your library |
+
+Other things that look like faults and are not:
+
+- **`403 host_not_allowed`** — the `Host` header is not a name in
+  `PHOTO_PIPELINE_ALLOWED_HOSTS`. Add the real hostname; do not add `*`.
+- **`409 lock_held`** — a mutating job is already running. One at a time is what
+  makes a crash recoverable.
+- **`409 rename_recovery_required`** — an interrupted rename is unresolved. Resolve
+  it in the Renames view; unrelated mutations stay blocked until then, on purpose.
+- **A job stuck in `queued`** — no worker is running.
+- **An empty scan** — check `PHOTO_PIPELINE_LIBRARY_ROOTS`, and in a container check
+  that the path is the *container-side* mount, not the host path.
+
+## What an installer must not work around
+
+- **One writer.** Do not run two workers, and do not run the frozen CLI beside the
+  application. Each is safe alone and destructive together — the lock is not
+  bureaucracy.
+- **`_IGNORE/` is never read.** Do not "fix" the exclusion.
+- **EXIF is verified before upload.** Do not skip the checkpoint to make a stage
+  finish; an unverified projection is exactly the case where the wrong bytes reach
+  Immich.
+- **The data directory is not the library.** Keep them apart, and keep the data
+  directory on a local filesystem.
+- **Secrets stay in the environment.** Not in the database, not in a log, not in a
+  commit.

tests/integration/test_installation_manual.py

@@ -0,0 +1,178 @@
+"""US09-02: the installation manual, checked against the thing it describes.
+
+Documentation rots quietly. A setting is renamed, a command grows a flag, an exit
+code changes meaning — and the manual keeps confidently saying the old thing until
+somebody follows it into a bad afternoon. So every fact in it that the code also
+knows is compared with the code, in both directions where both directions are
+meaningful: a setting the manual invents fails, and a setting the manual forgot
+fails too.
+"""
+
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+from functools import lru_cache
+from pathlib import Path
+
+from photo_pipeline.config import ENV_PREFIX, LEGACY_ALIASES, Config
+
+REPO = Path(__file__).resolve().parents[2]
+MANUAL = REPO / "docs" / "installation.md"
+MAIN = REPO / "photo_pipeline" / "__main__.py"
+
+TEXT = MANUAL.read_text()
+ENV_NAMES = re.compile(rf"{ENV_PREFIX}[A-Z0-9_]+")
+
+# Read by docker-compose.yml, not by Config. They are configuration an installer sets,
+# so the manual documents them; they are simply not application settings.
+COMPOSITION_ONLY = {
+    f"{ENV_PREFIX}IMAGE",
+    f"{ENV_PREFIX}LIBRARY_HOST_PATH",
+    f"{ENV_PREFIX}PUBLISH_ADDRESS",
+    f"{ENV_PREFIX}UID",
+    f"{ENV_PREFIX}GID",
+    f"{ENV_PREFIX}ENV_FILE",  # read before Config exists, so it is not a field
+}
+
+
+def documented_env_names() -> set[str]:
+    return set(ENV_NAMES.findall(TEXT))
+
+
+@lru_cache
+def cli_help(*args: str) -> str:
+    """What the CLI says about itself, asked the way an operator asks."""
+    result = subprocess.run(
+        [sys.executable, "-m", "photo_pipeline", *args, "--help"],
+        cwd=REPO,
+        capture_output=True,
+        text=True,
+        timeout=120,
+        check=False,
+    )
+    assert result.returncode == 0, result.stderr
+    return result.stdout
+
+
+@lru_cache
+def cli_commands() -> frozenset[str]:
+    listed = re.search(r"\{([a-z0-9,\-]+)\}", cli_help())
+    assert listed, f"the CLI listed no subcommands:\n{cli_help()}"
+    return frozenset(listed.group(1).split(","))
+
+
+def cli_flags(command: str) -> frozenset[str]:
+    return frozenset(re.findall(r"--[a-z][a-z-]+", cli_help(command)))
+
+
+# ── settings ─────────────────────────────────────────────────────────────────
+
+
+def test_every_setting_is_documented():
+    """A setting nobody documents is a setting nobody configures deliberately."""
+    expected = {f"{ENV_PREFIX}{field.upper()}" for field in Config.model_fields}
+    missing = sorted(expected - documented_env_names())
+    assert missing == [], f"undocumented settings: {missing}"
+
+
+def test_the_manual_invents_no_settings():
+    unknown = sorted(
+        name
+        for name in documented_env_names()
+        if name.removeprefix(ENV_PREFIX).lower() not in Config.model_fields
+        and name not in COMPOSITION_ONLY
+    )
+    assert unknown == [], f"documented but not a real setting: {unknown}"
+
+
+def test_the_documented_defaults_are_the_real_defaults():
+    """Spot-checked where a wrong default is actively dangerous: the bind address,
+    the trust boundary, and the gate that protects an irreplaceable library."""
+    assert Config.model_fields["host"].default == "127.0.0.1"
+    assert "`PHOTO_PIPELINE_HOST` | `127.0.0.1`" in TEXT
+    assert Config.model_fields["allowed_hosts"].default == ()
+    assert Config.model_fields["require_dry_run_approval"].default is False
+    assert "`PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL` | `false`" in TEXT
+
+
+def test_every_secret_is_marked_as_one():
+    for field in ("access_secret", "vision_api_key", "immich_api_key"):
+        name = f"{ENV_PREFIX}{field.upper()}"
+        row = next(line for line in TEXT.splitlines() if line.startswith(f"| `{name}`"))
+        assert "secret" in row.lower(), f"{name} is not marked as a secret"
+
+
+def test_the_legacy_aliases_are_documented():
+    """An operator with an existing photo_analyzer.env needs to know it still works."""
+    for alias in LEGACY_ALIASES:
+        assert alias in TEXT, f"legacy alias {alias} is undocumented"
+
+
+# ── commands ─────────────────────────────────────────────────────────────────
+
+
+# The manual is an installation and operations manual, not a command reference: these
+# are the commands an installer actually runs, and every one of them must exist.
+OPERATIONAL = ("migrate", "serve", "worker", "backup", "verify-backup", "restore", "diagnostics")
+
+
+def test_every_command_the_manual_tells_you_to_run_exists():
+    missing = [name for name in OPERATIONAL if name not in cli_commands()]
+    assert missing == [], f"the manual names commands the CLI does not have: {missing}"
+    undocumented = [name for name in OPERATIONAL if not re.search(rf"\b{re.escape(name)}\b", TEXT)]
+    assert undocumented == [], f"operational commands the manual omits: {undocumented}"
+
+
+def test_every_documented_flag_exists_on_the_command_it_is_shown_with():
+    for command, flag in (
+        ("backup", "--reason"),
+        ("backup", "--keep"),
+        ("restore", "--into"),
+        ("worker", "--allow-legacy"),
+        ("serve", "--allow-legacy"),
+    ):
+        assert flag in cli_flags(command), f"{command} has no {flag}"
+        assert flag in TEXT, f"{flag} is undocumented"
+
+
+# ── exit codes ───────────────────────────────────────────────────────────────
+
+
+def test_every_documented_exit_code_is_one_the_cli_can_return():
+    documented = {int(code) for code in re.findall(r"^\| `(\d)` \|", TEXT, re.MULTILINE)}
+    returned = {int(code) for code in re.findall(r"^\s+return (\d)$", MAIN.read_text(), re.MULTILINE)}
+    assert documented, "no exit codes are documented"
+    assert documented <= returned | {0}, f"documented but unreachable: {sorted(documented - returned)}"
+
+
+def test_every_refusal_exit_code_is_documented():
+    """0 is success and 1 is 'it said why'; every other code is a specific refusal an
+    operator will meet at startup, and meeting an undocumented one is the worst case."""
+    returned = {int(code) for code in re.findall(r"^\s+return (\d)$", MAIN.read_text(), re.MULTILINE)}
+    documented = {int(code) for code in re.findall(r"^\| `(\d)` \|", TEXT, re.MULTILINE)}
+    missing = sorted(code for code in returned if code not in documented and code != 0)
+    assert missing == [], f"undocumented exit codes: {missing}"
+
+
+# ── the promises the manual makes ────────────────────────────────────────────
+
+
+def test_the_manual_carries_no_credential_shaped_example():
+    """The repository's own scanner refuses these in a commit; a manual is exactly
+    where a real-looking one gets copied from."""
+    suspicious = re.findall(
+        r"(?i)(api[_-]?key|secret|token|password)\s*[=:]\s*[\"']?([A-Za-z0-9_\-]{12,})",
+        TEXT,
+    )
+    assert suspicious == [], f"credential-shaped example: {suspicious}"
+
+
+def test_the_manual_states_the_invariants_an_installer_must_not_break():
+    for invariant in ("_IGNORE/", "One writer", "EXIF is verified before upload"):
+        assert invariant in TEXT, f"the manual does not state: {invariant}"
+
+
+def test_the_manual_is_reachable_from_the_index():
+    assert "installation.md" in (REPO / "docs" / "index.md").read_text()

tests/story_traceability.json

@@ -197,10 +197,12 @@
     "US09-01": [
       "tests/integration/test_documentation.py",
       "tests/e2e/test_docs_ui.py"
+    ],
+    "US09-02": [
+      "tests/integration/test_installation_manual.py"
     ]
   },
   "planned": [
-    "US09-02",
     "US09-03",
     "US09-04",
     "US09-05"