photoanalyzer/docs/installation.md

Installation and operations

← Documentation index

From nothing to a running instance pointed at your photo library, and everything you need afterwards: upgrading, backing up, restoring, and understanding a refusal.

Two ways to install. Container is the deployment this project builds and ships; host is what you want for development or a single machine you already manage. Both run the same two processes against the same database.

Read what it will not do first. Several of the steps below only make sense once you know which rules the application is keeping.

Before you start

you need	version	why
Python	3.12 or newer	the application
exiftool	13.x (the image pins 13.25+dfsg-1)	every EXIF read and write
immich-go	0.32.0 (pinned in the image)	the upload stage only
Docker + the compose plugin	any current	the container installation only
a vision provider key	—	the analysis stage only

You also need a photo library you can afford to be wrong about. Take a backup of it before pointing anything at it for the first time, and consider running with PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL on.

Host installation

python3.12 -m venv .venv
.venv/bin/pip install -e ".[vision]"     # drop [vision] for a review-only install

Configure it. Every setting is an environment variable; a .env file in the working directory is read at startup:

cp .env.example .env && $EDITOR .env

.env.example lists every variable with its default and no values at all. For a loopback installation you need exactly one:

PHOTO_PIPELINE_LIBRARY_ROOTS=/home/you/Pictures

Then migrate and start the two processes:

.venv/bin/python -m photo_pipeline migrate    # create or upgrade the database
.venv/bin/python -m photo_pipeline serve      # API and browser app on 127.0.0.1:8000
.venv/bin/python -m photo_pipeline worker     # second terminal

Both processes are required. serve enqueues work and renders the application; nothing is scanned, scored, analysed, renamed, uploaded, or archived without a worker. Open http://127.0.0.1:8000/app/ and confirm the workflow page loads.

The configuration file

.env, or any path named by PHOTO_PIPELINE_ENV_FILE. It is parsed, never executed: KEY=value lines, # comments, optional quotes, no interpolation and no export. A configuration file that can run code is a configuration file that can be a vulnerability.

Anything already exported in the shell wins. The file is your standing configuration; the environment is the override for one run.

The archived CLI's names still work, so an existing photo_analyzer.env can be used as it is:

in the file	applied as
LLM_API_KEY / GEMINI_API_KEY	OPENAI_API_KEY
LLM_BASE_URL	OPENAI_BASE_URL
LIBRARY	PHOTO_PIPELINE_LIBRARY_ROOTS

.env and *.env are gitignored and refused by the repository's safety checks. The file holds a real key; it must never be committed.

Container installation

cp .env.example .env && $EDITOR .env
docker compose up -d --build

The composition is one serve container, one worker container, a one-shot migrate that both wait for, one bind-mounted library, and one named data volume.

Three variables it cannot start without:

variable	is
PHOTO_PIPELINE_LIBRARY_HOST_PATH	the library on this host
PHOTO_PIPELINE_LIBRARY_ROOTS	where that library is mounted inside the container
PHOTO_PIPELINE_ACCESS_SECRET	required, because publishing a port means the app is reachable from outside the container

Set PHOTO_PIPELINE_UID and PHOTO_PIPELINE_GID to the owner of the library: what the containers rename and rewrite keeps that ownership.

The data volume

The data volume holds the database, its write-ahead log, the thumbnail cache, the operation journals, and the backups. It must stay on a local filesystem. SQLite in WAL mode needs real local locking, so NFS, SMB, and network volume drivers do not slow it down — they corrupt it. The library bind mount has no such restriction.

Ports and reaching it

The API port is published to 127.0.0.1 unless PHOTO_PIPELINE_PUBLISH_ADDRESS says otherwise. Being reachable was the authentication in earlier versions: whoever could open the port owned the library. So the moment the application answers to anything but loopback — a hostname, a proxy, 0.0.0.0 — the access secret becomes mandatory and serve refuses to start without it rather than publishing your photographs.

Behind a reverse proxy:

PHOTO_PIPELINE_ALLOWED_HOSTS=photos.example.com
PHOTO_PIPELINE_ACCESS_SECRET=…                  # python -c 'import secrets; print(secrets.token_urlsafe(32))'
PHOTO_PIPELINE_TRUSTED_PROXIES=10.0.0.2         # only the proxy's own address

X-Forwarded-Proto and X-Forwarded-Host are believed only from a trusted-proxy address, so a client cannot declare its own origin. The browser asks for the secret once per tab. Wrong secrets are rate-limited and logged with the caller's address only. The health endpoints stay open so an orchestrator can restart the container; nothing else is.

Deployment

The stack is managed from git by Portainer, which redeploys on a webhook after CI publishes an image built from main. Runtime secrets live in the Portainer stack's environment rather than in the repository, so rotation happens in one place and reading the repository discloses nothing.

The settings

Every variable, its default, and whether it is a secret. .env.example is the same list in copyable form.

Library and data

variable	default	meaning
PHOTO_PIPELINE_LIBRARY_ROOTS	none	the library boundary, os.pathsep-separated. No path outside these roots is ever read or written
PHOTO_PIPELINE_DATA_DIR	data	database, WAL, thumbnail cache, journals, backups. Never inside the library
PHOTO_PIPELINE_DB_PATH	<data dir>/photo_pipeline.db	the database file, if it must live elsewhere
PHOTO_PIPELINE_ENV_FILE	.env	where to read the configuration file from

Serving and the trust boundary

variable	default	meaning
PHOTO_PIPELINE_HOST	127.0.0.1	bind address
PHOTO_PIPELINE_PORT	8000	port
PHOTO_PIPELINE_ALLOWED_HOSTS	none	comma-separated names the app answers to besides loopback. Naming one makes the access secret mandatory
PHOTO_PIPELINE_ACCESS_SECRET	none	secret. Traded for the session cookie at GET /api/v1/session
PHOTO_PIPELINE_TRUSTED_PROXIES	none	comma-separated peer addresses whose forwarded headers may be believed
PHOTO_PIPELINE_MAX_REQUEST_BYTES	1048576	largest request body accepted

Logging

variable	default	meaning
PHOTO_PIPELINE_LOG_LEVEL	INFO	standard Python levels
PHOTO_PIPELINE_LOG_FORMAT	json	json or text

Limits

variable	default	meaning
PHOTO_PIPELINE_THUMBNAIL_CACHE_QUOTA_BYTES	500000000	cache quota; over it is a diagnostics warning
PHOTO_PIPELINE_THUMBNAIL_MAX_PIXELS	100000000	refuse to decode anything larger. This is the decompression-bomb guard
PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES	1000000000	free space an archive destination must keep beyond the transfer itself

The safety gate

variable	default	meaning
PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL	false	refuse every mutating request until a read-only dry run of this library has been produced and approved

External services

variable	default	meaning
PHOTO_PIPELINE_VISION_API_KEY	none	secret. Without it the analysis stage cannot run
PHOTO_PIPELINE_IMMICH_SERVER_URL	none	the Immich server
PHOTO_PIPELINE_IMMICH_API_KEY	none	secret.
PHOTO_PIPELINE_IMMICH_GO_BINARY	immich-go	the uploader, found on PATH

Composition only

Read by docker-compose.yml, not by the application: PHOTO_PIPELINE_IMAGE, PHOTO_PIPELINE_LIBRARY_HOST_PATH, PHOTO_PIPELINE_PUBLISH_ADDRESS, PHOTO_PIPELINE_UID, PHOTO_PIPELINE_GID.

A secret is never written to a log, never returned by the API, never stored in the database, and never recorded in a backup manifest — a manifest says configured, not the value. Do not put a real key in an example, a ticket, or a screenshot.

First run checklist

Not "it started" — verified.

1. The database is at the current revision. python -m photo_pipeline migrate exits 0.
2. The application answers. curl -s localhost:8000/api/v1/health/ready returns 200.
3. The library was found. Open the app, run a scan from the Inventory view, and check the count against what you expect. Anything under _IGNORE/ must be missing from it — that is the exclusion working, not a bug.
4. The worker is claiming. The scan job reaches succeeded. If it stays queued, no worker is running.
5. Diagnostics are clean. python -m photo_pipeline diagnostics reports free space and an empty warnings.
6. Nothing was modified. The scan is read-only; your files' timestamps are unchanged.

Only then point it at the whole library.

Operations

Every operation is the same CLI, on a host or in the composition:

python -m photo_pipeline diagnostics
docker compose run --rm --no-deps api diagnostics

--no-deps keeps a one-off command from starting a second stack; api is only the service it borrows the image and mounts from.

Upgrading

1. python -m photo_pipeline backup --reason before-upgrade
2. Pull the new version (or docker compose pull && docker compose up -d).
3. Migrations run by themselves at startup, and a pending schema change is snapshotted first. If the upgrade fails, the previous database and its pre-migration backup are both intact, and the error log names the backup directory.

An up-to-date database is not backed up again on every start.

Backing up

python -m photo_pipeline backup --reason weekly --keep 7
python -m photo_pipeline verify-backup data/backups/<name>

Backups go through SQLite's online backup API, never a file copy: with WAL enabled the .db file alone is missing every committed page still in the write-ahead log. Each backup is a directory holding the snapshot and a manifest.json — schema revision, SHA-256, row counts, the archive media the library depends on, and which settings were configured.

verify-backup runs PRAGMA integrity_check and PRAGMA foreign_key_check, compares the snapshot's SHA-256 against the manifest, and re-counts every table it recorded. Bit rot, a truncated copy, and a "repaired" snapshot all fail it.

--keep N prunes the oldest and never the newest. The same is available at GET /api/v1/diagnostics, GET|POST /api/v1/backups, GET /api/v1/backups/{name}/verify, and POST /api/v1/backups/prune.

Restoring

Restore is deliberately not an API call. It replaces the state of an installation, so it belongs to a stopped one and a person at a terminal.

1. Stop serve and worker.
2. python -m photo_pipeline verify-backup data/backups/<name> — never restore an unverified snapshot, and restore will refuse one anyway.
3. python -m photo_pipeline restore data/backups/<name> --into /path/to/fresh-data. A target that already holds a database is refused; recovering in place means moving the old data directory aside first.
4. Point PHOTO_PIPELINE_DATA_DIR at the restored directory and run migrate.
5. Run an inventory scan, so paths are reconciled against the real library.
6. Mount every archive location the manifest names before archiving again. The database records where archived originals are; it does not contain them.

Practise this against a copy before you need it.

Watching it

diagnostics reports the database, write-ahead log, thumbnail cache, uploader reports, backups, and logs separately, with free space and warnings for low disk (disk_low, disk_critical), a cache over its quota, a write-ahead log outgrowing its database, and a legacy CLI writing the library.

Logs go to stdout in JSON by default — docker compose logs -f worker, or your service manager's journal on a host. Uploader output is kept per attempt under the data directory with the API key scrubbed line by line.

When it refuses to start

Every refusal below is deliberate. The application would rather stop and explain than guess about somebody's photographs.

exit code	meaning	what to do
1	the operation failed and said why — an unverifiable backup, an occupied restore target, an unknown benchmark profile	read the message; nothing was changed
2	the library lock is held. Another serve or worker owns this data directory	stop the other process. The message names the holder; a lock whose process is gone is taken over automatically
3	a legacy CLI looks active. The frozen command-line tools are writing this library	stop them. --allow-legacy overrides and you own the outcome
4	configuration refused. The application is reachable beyond loopback and no access secret is set	set PHOTO_PIPELINE_ACCESS_SECRET, or bind to loopback only
5	a library root is not mounted (containers only)	fix the bind mount. Refusing is what stops the container writing into its own throwaway layer instead of your library

Other things that look like faults and are not:

• 403 host_not_allowed — the Host header is not a name in PHOTO_PIPELINE_ALLOWED_HOSTS. Add the real hostname; do not add *.
• 409 lock_held — a mutating job is already running. One at a time is what makes a crash recoverable.
• 409 rename_recovery_required — an interrupted rename is unresolved. Resolve it in the Renames view; unrelated mutations stay blocked until then, on purpose.
• A job stuck in queued — no worker is running.
• An empty scan — check PHOTO_PIPELINE_LIBRARY_ROOTS, and in a container check that the path is the container-side mount, not the host path.

What an installer must not work around

• One writer. Do not run two workers, and do not run the frozen CLI beside the application. Each is safe alone and destructive together — the lock is not bureaucracy.
• _IGNORE/ is never read. Do not "fix" the exclusion.
• EXIF is verified before upload. Do not skip the checkpoint to make a stage finish; an unverified projection is exactly the case where the wrong bytes reach Immich.
• The data directory is not the library. Keep them apart, and keep the data directory on a local filesystem.
• Secrets stay in the environment. Not in the database, not in a log, not in a commit.