domverse/photoanalyzer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects 1 Releases Wiki Activity

US09-02: Write the Installation and Operations Manual (#108)
Some checks failed
Test / suites (push) Failing after 2m40s
Details
Test / container (push) Failing after 5m10s
Details

Browse Source
This commit was merged in pull request #108.
This commit is contained in:
domverse
2026-08-23 21:46:00 +02:00
parent e508f9fd64
commit 0d4f9d1c28
4 changed files with 520 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/index.md
View File

@@ -12,6 +12,9 @@ repository under `docs/`, on Gitea, and inside the running application under
1. [Overview](overview.md) — what the application does, the stages it moves a photo
through, and the rules it will not break.
2. [Installation and operations](installation.md) — host and container installation,
every setting, the first-run checklist, upgrades, backup and restore, and what
each refusal at startup means.
## Being written
@@ -19,9 +22,6 @@ The remaining manuals are accepted work, not aspiration; each is a story in
[E09](https://git.domverse-berlin.eu/domverse/photoanalyzer/src/branch/main/delivery_backlog/E09-documentation.md)
and will appear here as it lands.
- **Installation and operations** — host and container installation, every setting,
the first-run checklist, upgrades, backup and restore, and what each refusal at
startup means (US09-02).
- **Architecture** — context and runtime diagrams, the module map, the job and
journal state machines, and where each invariant is enforced (US09-03).
- **User manual** — one page per workflow stage with screenshots of the real

336
docs/installation.md Normal file
View File

@@ -0,0 +1,336 @@
# Installation and operations
[← Documentation index](index.md)
From nothing to a running instance pointed at your photo library, and everything you
need afterwards: upgrading, backing up, restoring, and understanding a refusal.
Two ways to install. **Container** is the deployment this project builds and ships;
**host** is what you want for development or a single machine you already manage.
Both run the same two processes against the same database.
> Read [what it will not do](overview.md#what-it-will-not-do) first. Several of the
> steps below only make sense once you know which rules the application is keeping.
## Before you start
| you need | version | why |
|---|---|---|
| Python | 3.12 or newer | the application |
| `exiftool` | 13.x (the image pins `13.25+dfsg-1`) | every EXIF read and write |
| `immich-go` | 0.32.0 (pinned in the image) | the upload stage only |
| Docker + the compose plugin | any current | the container installation only |
| a vision provider key | — | the analysis stage only |
You also need a photo library you can afford to be wrong about. Take a backup of it
before pointing anything at it for the first time, and consider running with
[`PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL`](#the-settings) on.
## Host installation
```bash
python3.12 -m venv .venv
.venv/bin/pip install -e ".[vision]" # drop [vision] for a review-only install
```
Configure it. Every setting is an environment variable; a `.env` file in the working
directory is read at startup:
```bash
cp .env.example .env && $EDITOR .env
```
`.env.example` lists every variable with its default and no values at all. For a
loopback installation you need exactly one:
```bash
PHOTO_PIPELINE_LIBRARY_ROOTS=/home/you/Pictures
```
Then migrate and start the two processes:
```bash
.venv/bin/python -m photo_pipeline migrate # create or upgrade the database
.venv/bin/python -m photo_pipeline serve # API and browser app on 127.0.0.1:8000
.venv/bin/python -m photo_pipeline worker # second terminal
```
**Both processes are required.** `serve` enqueues work and renders the application;
nothing is scanned, scored, analysed, renamed, uploaded, or archived without a
worker. Open <http://127.0.0.1:8000/app/> and confirm the workflow page loads.
### The configuration file
`.env`, or any path named by `PHOTO_PIPELINE_ENV_FILE`. It is parsed, never executed:
`KEY=value` lines, `#` comments, optional quotes, no interpolation and no `export`. A
configuration file that can run code is a configuration file that can be a
vulnerability.
**Anything already exported in the shell wins.** The file is your standing
configuration; the environment is the override for one run.
The archived CLI's names still work, so an existing `photo_analyzer.env` can be used
as it is:
| in the file | applied as |
|---|---|
| `LLM_API_KEY` / `GEMINI_API_KEY` | `OPENAI_API_KEY` |
| `LLM_BASE_URL` | `OPENAI_BASE_URL` |
| `LIBRARY` | `PHOTO_PIPELINE_LIBRARY_ROOTS` |
`.env` and `*.env` are gitignored and refused by the repository's safety checks. The
file holds a real key; it must never be committed.
## Container installation
```bash
cp .env.example .env && $EDITOR .env
docker compose up -d --build
```
The composition is one `serve` container, one `worker` container, a one-shot
`migrate` that both wait for, one bind-mounted library, and one named data volume.
Three variables it cannot start without:
| variable | is |
|---|---|
| `PHOTO_PIPELINE_LIBRARY_HOST_PATH` | the library on this host |
| `PHOTO_PIPELINE_LIBRARY_ROOTS` | where that library is mounted **inside** the container |
| `PHOTO_PIPELINE_ACCESS_SECRET` | required, because publishing a port means the app is reachable from outside the container |
Set `PHOTO_PIPELINE_UID` and `PHOTO_PIPELINE_GID` to the owner of the library: what
the containers rename and rewrite keeps that ownership.
### The data volume
The `data` volume holds the database, its write-ahead log, the thumbnail cache, the
operation journals, and the backups. **It must stay on a local filesystem.** SQLite
in WAL mode needs real local locking, so NFS, SMB, and network volume drivers do not
slow it down — they corrupt it. The library bind mount has no such restriction.
### Ports and reaching it
The API port is published to `127.0.0.1` unless `PHOTO_PIPELINE_PUBLISH_ADDRESS` says
otherwise. Being reachable *was* the authentication in earlier versions: whoever could
open the port owned the library. So the moment the application answers to anything but
loopback — a hostname, a proxy, `0.0.0.0` — the access secret becomes mandatory and
`serve` refuses to start without it rather than publishing your photographs.
Behind a reverse proxy:
```bash
PHOTO_PIPELINE_ALLOWED_HOSTS=photos.example.com
PHOTO_PIPELINE_ACCESS_SECRET=… # python -c 'import secrets; print(secrets.token_urlsafe(32))'
PHOTO_PIPELINE_TRUSTED_PROXIES=10.0.0.2 # only the proxy's own address
```
`X-Forwarded-Proto` and `X-Forwarded-Host` are believed only from a trusted-proxy
address, so a client cannot declare its own origin. The browser asks for the secret
once per tab. Wrong secrets are rate-limited and logged with the caller's address
only. The health endpoints stay open so an orchestrator can restart the container;
nothing else is.
### Deployment
The stack is managed from git by Portainer, which redeploys on a webhook after CI
publishes an image built from `main`. Runtime secrets live in the Portainer stack's
environment rather than in the repository, so rotation happens in one place and
reading the repository discloses nothing.
## The settings
Every variable, its default, and whether it is a secret. `.env.example` is the same
list in copyable form.
### Library and data
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_LIBRARY_ROOTS` | none | the library boundary, `os.pathsep`-separated. No path outside these roots is ever read or written |
| `PHOTO_PIPELINE_DATA_DIR` | `data` | database, WAL, thumbnail cache, journals, backups. Never inside the library |
| `PHOTO_PIPELINE_DB_PATH` | `<data dir>/photo_pipeline.db` | the database file, if it must live elsewhere |
| `PHOTO_PIPELINE_ENV_FILE` | `.env` | where to read the configuration file from |
### Serving and the trust boundary
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_HOST` | `127.0.0.1` | bind address |
| `PHOTO_PIPELINE_PORT` | `8000` | port |
| `PHOTO_PIPELINE_ALLOWED_HOSTS` | none | comma-separated names the app answers to besides loopback. Naming one makes the access secret mandatory |
| `PHOTO_PIPELINE_ACCESS_SECRET` | none | **secret.** Traded for the session cookie at `GET /api/v1/session` |
| `PHOTO_PIPELINE_TRUSTED_PROXIES` | none | comma-separated peer addresses whose forwarded headers may be believed |
| `PHOTO_PIPELINE_MAX_REQUEST_BYTES` | `1048576` | largest request body accepted |
### Logging
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_LOG_LEVEL` | `INFO` | standard Python levels |
| `PHOTO_PIPELINE_LOG_FORMAT` | `json` | `json` or `text` |
### Limits
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_THUMBNAIL_CACHE_QUOTA_BYTES` | `500000000` | cache quota; over it is a diagnostics warning |
| `PHOTO_PIPELINE_THUMBNAIL_MAX_PIXELS` | `100000000` | refuse to decode anything larger. This is the decompression-bomb guard |
| `PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES` | `1000000000` | free space an archive destination must keep beyond the transfer itself |
### The safety gate
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL` | `false` | refuse every mutating request until a read-only dry run of this library has been produced and approved |
### External services
| variable | default | meaning |
|---|---|---|
| `PHOTO_PIPELINE_VISION_API_KEY` | none | **secret.** Without it the analysis stage cannot run |
| `PHOTO_PIPELINE_IMMICH_SERVER_URL` | none | the Immich server |
| `PHOTO_PIPELINE_IMMICH_API_KEY` | none | **secret.** |
| `PHOTO_PIPELINE_IMMICH_GO_BINARY` | `immich-go` | the uploader, found on `PATH` |
### Composition only
Read by `docker-compose.yml`, not by the application:
`PHOTO_PIPELINE_IMAGE`, `PHOTO_PIPELINE_LIBRARY_HOST_PATH`,
`PHOTO_PIPELINE_PUBLISH_ADDRESS`, `PHOTO_PIPELINE_UID`, `PHOTO_PIPELINE_GID`.
A secret is never written to a log, never returned by the API, never stored in the
database, and never recorded in a backup manifest — a manifest says `configured`, not
the value. Do not put a real key in an example, a ticket, or a screenshot.
## First run checklist
Not "it started" — verified.
1. **The database is at the current revision.**
`python -m photo_pipeline migrate` exits 0.
2. **The application answers.**
`curl -s localhost:8000/api/v1/health/ready` returns 200.
3. **The library was found.** Open the app, run a scan from the Inventory view, and
check the count against what you expect. Anything under `_IGNORE/` must be missing
from it — that is the exclusion working, not a bug.
4. **The worker is claiming.** The scan job reaches `succeeded`. If it stays
`queued`, no worker is running.
5. **Diagnostics are clean.**
`python -m photo_pipeline diagnostics` reports free space and an empty `warnings`.
6. **Nothing was modified.** The scan is read-only; your files' timestamps are
unchanged.
Only then point it at the whole library.
## Operations
Every operation is the same CLI, on a host or in the composition:
```bash
python -m photo_pipeline diagnostics
docker compose run --rm --no-deps api diagnostics
```
`--no-deps` keeps a one-off command from starting a second stack; `api` is only the
service it borrows the image and mounts from.
### Upgrading
1. `python -m photo_pipeline backup --reason before-upgrade`
2. Pull the new version (or `docker compose pull && docker compose up -d`).
3. Migrations run by themselves at startup, and a **pending schema change is
snapshotted first**. If the upgrade fails, the previous database and its
`pre-migration` backup are both intact, and the error log names the backup
directory.
An up-to-date database is not backed up again on every start.
### Backing up
```bash
python -m photo_pipeline backup --reason weekly --keep 7
python -m photo_pipeline verify-backup data/backups/<name>
```
Backups go through SQLite's online backup API, never a file copy: with WAL enabled
the `.db` file alone is missing every committed page still in the write-ahead log.
Each backup is a directory holding the snapshot and a `manifest.json` — schema
revision, SHA-256, row counts, the archive media the library depends on, and which
settings were configured.
`verify-backup` runs `PRAGMA integrity_check` **and** `PRAGMA foreign_key_check`,
compares the snapshot's SHA-256 against the manifest, and re-counts every table it
recorded. Bit rot, a truncated copy, and a "repaired" snapshot all fail it.
`--keep N` prunes the oldest and never the newest. The same is available at
`GET /api/v1/diagnostics`, `GET|POST /api/v1/backups`,
`GET /api/v1/backups/{name}/verify`, and `POST /api/v1/backups/prune`.
### Restoring
Restore is deliberately **not** an API call. It replaces the state of an
installation, so it belongs to a stopped one and a person at a terminal.
1. Stop `serve` and `worker`.
2. `python -m photo_pipeline verify-backup data/backups/<name>` — never restore an
unverified snapshot, and `restore` will refuse one anyway.
3. `python -m photo_pipeline restore data/backups/<name> --into /path/to/fresh-data`.
A target that already holds a database is refused; recovering in place means
moving the old data directory aside first.
4. Point `PHOTO_PIPELINE_DATA_DIR` at the restored directory and run `migrate`.
5. Run an inventory scan, so paths are reconciled against the real library.
6. Mount every archive location the manifest names before archiving again. The
database records where archived originals are; it does not contain them.
Practise this against a copy before you need it.
### Watching it
`diagnostics` reports the database, write-ahead log, thumbnail cache, uploader
reports, backups, and logs separately, with free space and warnings for low disk
(`disk_low`, `disk_critical`), a cache over its quota, a write-ahead log outgrowing
its database, and a legacy CLI writing the library.
Logs go to stdout in JSON by default — `docker compose logs -f worker`, or your
service manager's journal on a host. Uploader output is kept per attempt under the
data directory with the API key scrubbed line by line.
## When it refuses to start
Every refusal below is deliberate. The application would rather stop and explain than
guess about somebody's photographs.
| exit code | meaning | what to do |
|---|---|---|
| `1` | the operation failed and said why — an unverifiable backup, an occupied restore target, an unknown benchmark profile | read the message; nothing was changed |
| `2` | **the library lock is held.** Another `serve` or `worker` owns this data directory | stop the other process. The message names the holder; a lock whose process is gone is taken over automatically |
| `3` | **a legacy CLI looks active.** The frozen command-line tools are writing this library | stop them. `--allow-legacy` overrides and you own the outcome |
| `4` | **configuration refused.** The application is reachable beyond loopback and no access secret is set | set `PHOTO_PIPELINE_ACCESS_SECRET`, or bind to loopback only |
| `5` | **a library root is not mounted** (containers only) | fix the bind mount. Refusing is what stops the container writing into its own throwaway layer instead of your library |
Other things that look like faults and are not:
- **`403 host_not_allowed`** — the `Host` header is not a name in
`PHOTO_PIPELINE_ALLOWED_HOSTS`. Add the real hostname; do not add `*`.
- **`409 lock_held`** — a mutating job is already running. One at a time is what
makes a crash recoverable.
- **`409 rename_recovery_required`** — an interrupted rename is unresolved. Resolve
it in the Renames view; unrelated mutations stay blocked until then, on purpose.
- **A job stuck in `queued`** — no worker is running.
- **An empty scan** — check `PHOTO_PIPELINE_LIBRARY_ROOTS`, and in a container check
that the path is the *container-side* mount, not the host path.
## What an installer must not work around
- **One writer.** Do not run two workers, and do not run the frozen CLI beside the
application. Each is safe alone and destructive together — the lock is not
bureaucracy.
- **`_IGNORE/` is never read.** Do not "fix" the exclusion.
- **EXIF is verified before upload.** Do not skip the checkpoint to make a stage
finish; an unverified projection is exactly the case where the wrong bytes reach
Immich.
- **The data directory is not the library.** Keep them apart, and keep the data
directory on a local filesystem.
- **Secrets stay in the environment.** Not in the database, not in a log, not in a
commit.