Compare commits
1 Commits
main
...
chore/E09-
| Author | SHA1 | Date | |
|---|---|---|---|
| 792134c72d |
@@ -9,7 +9,6 @@
|
|||||||
!photo_pipeline
|
!photo_pipeline
|
||||||
!migrations
|
!migrations
|
||||||
!frontend
|
!frontend
|
||||||
!docs
|
|
||||||
!docker
|
!docker
|
||||||
|
|
||||||
# Nothing generated, even under an allowed directory.
|
# Nothing generated, even under an allowed directory.
|
||||||
|
|||||||
9
.gitattributes
vendored
@@ -1,9 +0,0 @@
|
|||||||
# Vendored third-party browser bundles (US09-01). Minified upstream files carry
|
|
||||||
# trailing whitespace and very long lines; they are not ours to reformat, and the
|
|
||||||
# submit gate's `git diff --check` would refuse them forever. Their integrity is
|
|
||||||
# controlled where it belongs instead: a pinned version and a recorded sha256 in
|
|
||||||
# frontend/js/vendor/VERSIONS.json, asserted by tests/integration/test_documentation.py.
|
|
||||||
#
|
|
||||||
# Only the whitespace check is turned off. They stay text, so a credential scan or a
|
|
||||||
# search still reads them.
|
|
||||||
frontend/js/vendor/*.js -whitespace linguist-vendored
|
|
||||||
@@ -101,39 +101,3 @@ jobs:
|
|||||||
with:
|
with:
|
||||||
name: container-gate-${{ gitea.sha }}
|
name: container-gate-${{ gitea.sha }}
|
||||||
path: gate-evidence
|
path: gate-evidence
|
||||||
|
|
||||||
# The manuals, checked against the application they describe (US09-05). Runs on
|
|
||||||
# every pull request as well as `main`: documentation drifts by the same commits
|
|
||||||
# that change behaviour, and telling the author while the change is still open is
|
|
||||||
# the only time the fix is cheap.
|
|
||||||
documentation:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Set up Python
|
|
||||||
uses: actions/setup-python@v5
|
|
||||||
with:
|
|
||||||
python-version: '3.12'
|
|
||||||
|
|
||||||
- name: Install the application and its test dependencies
|
|
||||||
run: |
|
|
||||||
python -m pip install --upgrade pip
|
|
||||||
python -m pip install -e '.[test]'
|
|
||||||
python -m playwright install --with-deps chromium
|
|
||||||
|
|
||||||
- name: Documentation gate
|
|
||||||
# One command: every phase_i check, offline and in the browser, with the
|
|
||||||
# evidence retained. It accepts no skip — a check that did not run is a page
|
|
||||||
# nobody compared with the code.
|
|
||||||
env:
|
|
||||||
PHOTO_PIPELINE_DATA_DIR: ${{ gitea.workspace }}/docs-gate-data
|
|
||||||
run: python -m photo_pipeline docs-gate --output docs-evidence
|
|
||||||
|
|
||||||
- name: Keep the evidence
|
|
||||||
if: always()
|
|
||||||
uses: actions/upload-artifact@v3
|
|
||||||
with:
|
|
||||||
name: docs-gate-${{ gitea.sha }}
|
|
||||||
path: docs-evidence
|
|
||||||
|
|||||||
@@ -70,8 +70,6 @@ COPY pyproject.toml alembic.ini README.md ./
|
|||||||
COPY photo_pipeline ./photo_pipeline
|
COPY photo_pipeline ./photo_pipeline
|
||||||
COPY migrations ./migrations
|
COPY migrations ./migrations
|
||||||
COPY frontend ./frontend
|
COPY frontend ./frontend
|
||||||
# The manuals are served by the application itself (US09-01), so they ship with it.
|
|
||||||
COPY docs ./docs
|
|
||||||
COPY docker/entrypoint.sh docker/healthcheck.sh /usr/local/bin/
|
COPY docker/entrypoint.sh docker/healthcheck.sh /usr/local/bin/
|
||||||
|
|
||||||
# Runtime dependencies only: the `test` extra (pytest, playwright) and the `vision`
|
# Runtime dependencies only: the `test` extra (pytest, playwright) and the `vision`
|
||||||
|
|||||||
18
README.md
@@ -4,24 +4,6 @@ Integrated, restart-safe photo analysis, duplicate review, metadata, upload, and
|
|||||||
archive workflow. Planning lives in `INTEGRATED_PIPELINE_CONCEPT.md` and
|
archive workflow. Planning lives in `INTEGRATED_PIPELINE_CONCEPT.md` and
|
||||||
`delivery_backlog/`.
|
`delivery_backlog/`.
|
||||||
|
|
||||||
## Documentation
|
|
||||||
|
|
||||||
The manuals live in [`docs/`](docs/index.md) and are the same files the running
|
|
||||||
application serves at `/app/#/docs`:
|
|
||||||
|
|
||||||
| | |
|
|
||||||
|---|---|
|
|
||||||
| [Overview](docs/overview.md) | what it does and the rules it will not break |
|
|
||||||
| [Installation and operations](docs/installation.md) | host and container install, every setting, backup and restore, every refusal |
|
|
||||||
| [A guided first pass](docs/first-pass.md) | one library from scan to verified upload |
|
|
||||||
| [The stages](docs/index.md) | one illustrated page each, from inventory to archive |
|
|
||||||
| [Errors and refusals](docs/errors.md) | every error code, cause, and remedy |
|
|
||||||
| [Architecture](docs/architecture.md) | diagrams, module map, journals, invariants |
|
|
||||||
|
|
||||||
This README stays the repository's own notes — how the code is built, tested, and
|
|
||||||
released. Anything an operator or a user needs belongs in `docs/`, and one gate
|
|
||||||
(`python -m photo_pipeline docs-gate`) keeps those pages honest against the code.
|
|
||||||
|
|
||||||
## Application (`photo_pipeline`)
|
## Application (`photo_pipeline`)
|
||||||
|
|
||||||
The target application lives in `photo_pipeline/` (FastAPI + SQLAlchemy + Alembic).
|
The target application lives in `photo_pipeline/` (FastAPI + SQLAlchemy + Alembic).
|
||||||
|
|||||||
@@ -1,224 +0,0 @@
|
|||||||
# Architecture
|
|
||||||
|
|
||||||
[← Documentation index](index.md)
|
|
||||||
|
|
||||||
For whoever has to change this code without breaking somebody's photo library. It
|
|
||||||
explains what the pieces are, which rules each one keeps, and where to look when a
|
|
||||||
stage refuses.
|
|
||||||
|
|
||||||
The product decisions behind all of it live in `INTEGRATED_PIPELINE_CONCEPT.md`; this
|
|
||||||
page describes what was built.
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
Five things outside the application, and what actually crosses each boundary.
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart LR
|
|
||||||
operator([Operator]):::person -->|browser, one session| app
|
|
||||||
app[Photo Pipeline]:::system -->|read, rename, EXIF write| library[(Photo library<br/>bind mount)]
|
|
||||||
app -->|database, cache, journals, backups| data[(Data directory<br/>local filesystem)]
|
|
||||||
app -->|confirmed-SFW images only| vision[Vision provider]:::ext
|
|
||||||
app -->|immich-go, verified bytes| immich[Immich server]:::ext
|
|
||||||
app -->|copy, verify, then remove| archive[(Archive medium)]
|
|
||||||
classDef person fill:#1f6feb,stroke:#58a6ff,color:#fff
|
|
||||||
classDef system fill:#238636,stroke:#3fb950,color:#fff
|
|
||||||
classDef ext fill:#6e40c9,stroke:#a371f7,color:#fff
|
|
||||||
```
|
|
||||||
|
|
||||||
| boundary | leaves the machine? | carries |
|
|
||||||
|---|---|---|
|
|
||||||
| operator → app | no (loopback, or a proxy you configured) | commands against stable ids, never paths |
|
|
||||||
| app → library | no | reads, folder renames, EXIF merges |
|
|
||||||
| app → data directory | no | SQLite in WAL mode, thumbnails, journals, backups |
|
|
||||||
| app → vision provider | **yes** | image bytes of confirmed-SFW canonical assets only |
|
|
||||||
| app → Immich | **yes** | the exact verified bytes of an approved album |
|
|
||||||
| app → archive medium | no | copies, verified before the source is removed |
|
|
||||||
|
|
||||||
## Runtime
|
|
||||||
|
|
||||||
Two long-lived processes and a one-shot migration, sharing one database and one
|
|
||||||
library.
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart TD
|
|
||||||
browser([Browser]) -->|JSON + SSE| api
|
|
||||||
migrate["migrate<br/>backup, then upgrade"] -->|must exit 0| api
|
|
||||||
migrate --> worker
|
|
||||||
api["api · serve<br/>enqueues, serves, reads"] -->|jobs table| db[(SQLite WAL)]
|
|
||||||
worker["worker<br/>claims and does the work"] -->|jobs table| db
|
|
||||||
worker --> tools["exiftool · vision API · immich-go"]
|
|
||||||
api -.->|api.lock.json| lock{{library lock}}
|
|
||||||
worker -.->|worker.lock.json| lock
|
|
||||||
api --> lib[(library)]
|
|
||||||
worker --> lib
|
|
||||||
```
|
|
||||||
|
|
||||||
`serve` enqueues and renders; **it does not do the work**. Everything that scans,
|
|
||||||
scores, analyses, renames, uploads, or archives happens in the worker, which claims
|
|
||||||
a queued job atomically with a fencing token. One mutating job runs at a time, and
|
|
||||||
the lock file in the data directory is what makes a second worker impossible rather
|
|
||||||
than merely discouraged.
|
|
||||||
|
|
||||||
## The modules
|
|
||||||
|
|
||||||
| package | owns | must not |
|
|
||||||
|---|---|---|
|
|
||||||
| `api/` | HTTP surface, error envelope, security policy | contain SQL or business logic |
|
|
||||||
| `api/routes/` | one module per resource group, all under `/api/v1` | accept a filesystem path from the browser |
|
|
||||||
| `api/security.py` | host/origin/CSRF/session/size policy as one pure `evaluate`, plus the ASGI middleware | be bypassed per-route |
|
|
||||||
| `schemas/` | Pydantic request and response contracts | reach the database |
|
|
||||||
| `services/` | all domain logic; the only place a decision is made | be imported by the frozen CLI archive |
|
|
||||||
| `models/` | SQLAlchemy tables | hold behaviour |
|
|
||||||
| `jobs/` | durable job lifecycle, worker loop, lock ranks, handler registry | run work in the API process |
|
|
||||||
| `integrations/` | the real outside world: `exiftool`, vision, NSFW model, `immich-go` and its report grammars | be called without a version recorded |
|
|
||||||
| `path_policy.py` | the library boundary, `_IGNORE/` exclusion, symlink-escape refusal | be duplicated anywhere |
|
|
||||||
| `imaging.py` | the single bounded-decode door | let a caller open an image directly |
|
|
||||||
| `faults.py` | the crash barriers the tests fire | read anything but its one env var |
|
|
||||||
| `config.py` | typed settings, secrets as `SecretStr` | log or return a value |
|
|
||||||
| `db.py` | engine, sessions, WAL and foreign keys, migration entry | be used to bypass a repository |
|
|
||||||
|
|
||||||
Services worth knowing by name: `inventory`, `duplicates`, `safety`, `analysis`,
|
|
||||||
`albums`/`proposals`/`naming`, `renames`/`rename_apply`/`rename_journal`,
|
|
||||||
`uploads`/`upload_batches`/`upload_reports`/`upload_verification`,
|
|
||||||
`archives`/`archive_transfer`/`archive_journal`/`restores`/`availability`,
|
|
||||||
`thumbnails`, `hashing`, `exif_checkpoint`, `jobs`, `workflow`, `backup`,
|
|
||||||
`diagnostics`, `app_lock`, `release`, `benchmarks`, `library`, `legacy_import`.
|
|
||||||
|
|
||||||
## The state machines
|
|
||||||
|
|
||||||
Three journals decide what a restart is allowed to assume. All three are read from
|
|
||||||
the database plus the real world — never guessed from a missing file.
|
|
||||||
|
|
||||||
### Durable jobs
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
stateDiagram-v2
|
|
||||||
[*] --> queued
|
|
||||||
queued --> running: claimed with a fencing token
|
|
||||||
queued --> cancelled
|
|
||||||
queued --> cancelling
|
|
||||||
running --> succeeded
|
|
||||||
running --> failed
|
|
||||||
running --> cancelling
|
|
||||||
cancelling --> cancelled
|
|
||||||
failed --> retry_queued
|
|
||||||
retry_queued --> running
|
|
||||||
succeeded --> [*]
|
|
||||||
cancelled --> [*]
|
|
||||||
```
|
|
||||||
|
|
||||||
`succeeded` and `cancelled` are terminal, so a duplicate delivery cannot move a
|
|
||||||
finished job. Claiming is compare-and-set on the row version; a worker whose lease
|
|
||||||
expired cannot commit after another has taken over.
|
|
||||||
|
|
||||||
### The rename journal
|
|
||||||
|
|
||||||
The only state machine that moves somebody's folders.
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
stateDiagram-v2
|
|
||||||
[*] --> planned
|
|
||||||
planned --> moving
|
|
||||||
moving --> moved
|
|
||||||
moving --> planned: proven untouched
|
|
||||||
moving --> rollback_required
|
|
||||||
moved --> database_updated
|
|
||||||
database_updated --> verified
|
|
||||||
verified --> complete
|
|
||||||
moved --> rollback_required
|
|
||||||
database_updated --> rollback_required
|
|
||||||
verified --> rollback_required
|
|
||||||
planned --> failed
|
|
||||||
failed --> planned
|
|
||||||
rollback_required --> rolled_back
|
|
||||||
complete --> [*]
|
|
||||||
rolled_back --> [*]
|
|
||||||
```
|
|
||||||
|
|
||||||
Intent is written **before** the disk is touched, which is why `moving` can resolve
|
|
||||||
backwards: the evidence decides. `moving`, `moved`, `database_updated`, and
|
|
||||||
`rollback_required` are the unsafe states — while any operation sits in one, the
|
|
||||||
library may be half-renamed, so unrelated mutations are refused with
|
|
||||||
`409 rename_recovery_required` until a person resolves it.
|
|
||||||
|
|
||||||
### Upload batches
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
stateDiagram-v2
|
|
||||||
[*] --> planned
|
|
||||||
planned --> running
|
|
||||||
running --> succeeded
|
|
||||||
running --> failed
|
|
||||||
running --> cancelling
|
|
||||||
running --> unknown_requires_verification: process died after acceptance
|
|
||||||
cancelling --> cancelled
|
|
||||||
failed --> running: retry
|
|
||||||
cancelled --> running: retry
|
|
||||||
```
|
|
||||||
|
|
||||||
`unknown_requires_verification` is deliberately **not** restartable. The server may
|
|
||||||
already hold the files; the answer is to ask Immich for the recorded SHA-1, not to
|
|
||||||
upload again and hope.
|
|
||||||
|
|
||||||
Archive transfers use the same shape — `planned → transferring → verified →
|
|
||||||
removing → complete` — and the source is removed only after the archived bytes are
|
|
||||||
verified.
|
|
||||||
|
|
||||||
## Identity and the data model
|
|
||||||
|
|
||||||
A path is metadata. The identity is `assets.id`, a UUID that never changes, and
|
|
||||||
`asset_paths` records every path an asset has ever had with the reason it changed.
|
|
||||||
That is what makes a rename cheap: nothing else in the database has to move.
|
|
||||||
|
|
||||||
Tables: `assets`, `asset_paths`, `thumbnails`, `jobs`, `job_items`, `job_events`,
|
|
||||||
`safety_reviews`, `analysis_results`, `exif_projections`, `duplicate_clusters`,
|
|
||||||
`duplicate_members`, `duplicate_negative_links`, `album_proposals`, `rename_plans`,
|
|
||||||
`rename_operations`, `upload_batches`, `upload_items`, `upload_verifications`,
|
|
||||||
`archive_locations`, `archive_plans`, `archive_operations`.
|
|
||||||
|
|
||||||
Availability is independent of workflow progress: `active`, `archiving`,
|
|
||||||
`archived_online`, `archived_offline`, `restoring`, `missing_unexpected`. An
|
|
||||||
unmounted archive disk is `archived_offline`, never `missing` — the scanner is not
|
|
||||||
allowed to conclude that a photo is gone because a disk is unplugged.
|
|
||||||
|
|
||||||
EXIF is a projection with its own verified state (`verified`, `divergent`,
|
|
||||||
`failed`): the desired values are written, read back, and compared, and anything the
|
|
||||||
stage does not own having changed makes the asset `divergent` and blocks the next
|
|
||||||
mutating stage.
|
|
||||||
|
|
||||||
## Where each invariant lives
|
|
||||||
|
|
||||||
| invariant | enforced in |
|
|
||||||
|---|---|
|
|
||||||
| `_IGNORE/` is never traversed, counted, or opened | `path_policy.is_excluded`, used by every discovery path |
|
|
||||||
| no path outside the library roots is reachable | `path_policy.resolve_in_roots` — it returns the resolved path, because validating one name and opening another is the symlink race |
|
|
||||||
| one writer per library | `services/app_lock.py` (an `flock` on a JSON lock file, per role) |
|
|
||||||
| one mutating job at a time | `services/jobs.py` lock keys plus `jobs/locks.py` rank ordering |
|
|
||||||
| only confirmed-SFW assets reach the vision provider | `services/analysis.py`, re-checked *after* the provider call so a decision that flipped mid-flight discards the result |
|
|
||||||
| EXIF is verified, and other fields preserved | `services/exif_checkpoint.py` |
|
|
||||||
| uploads carry the exact verified bytes | `services/uploads.py` preflight, re-proved immediately before the uploader runs |
|
|
||||||
| an uncertain upload is not a failure | `services/upload_batches.py`, `services/upload_verification.py` |
|
|
||||||
| the archive source outlives its copy until verified | `services/archive_transfer.py` |
|
|
||||||
| decoding is bounded | `imaging.py` |
|
|
||||||
| the trust boundary and CSRF | `api/security.py` |
|
|
||||||
|
|
||||||
## Concurrency
|
|
||||||
|
|
||||||
Locks are taken broad to narrow — `library → stage/job → album/folder → asset` — and
|
|
||||||
never the other way, which is what makes deadlock structural rather than lucky.
|
|
||||||
Ranks live in `jobs/locks.py`.
|
|
||||||
|
|
||||||
Exactly-once execution is not achievable across SQLite, a filesystem, subprocesses,
|
|
||||||
and a remote server. The design promises **at-least-once with idempotent recovery**:
|
|
||||||
every handler may run twice, and running twice must not produce two side effects.
|
|
||||||
That is why the rename journal records intent before moving, why EXIF writes merge
|
|
||||||
and verify, and why upload retries consult both local history and Immich.
|
|
||||||
|
|
||||||
## History
|
|
||||||
|
|
||||||
This application was extracted from two command-line tools rather than written from
|
|
||||||
nothing. Their sources are frozen in `legacy_cli_archive/` with the ledger mapping
|
|
||||||
each donated behaviour to the service that now owns it, the characterization tests
|
|
||||||
that pinned it, and every intentional difference. Production code must not import
|
|
||||||
them; they are provenance and rollback evidence.
|
|
||||||
@@ -1,95 +0,0 @@
|
|||||||
# Errors and refusals
|
|
||||||
|
|
||||||
[← Documentation index](index.md)
|
|
||||||
|
|
||||||
Every error the API returns carries a code:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"error": {"code": "lock_held", "message": "a library_write job is already running"}}
|
|
||||||
```
|
|
||||||
|
|
||||||
Most of them are **refusals, not faults**. This application would rather stop and
|
|
||||||
explain than guess about somebody's photographs, so a code below usually means it
|
|
||||||
protected something. Each row says what caused it and what to do.
|
|
||||||
|
|
||||||
## Access and the trust boundary
|
|
||||||
|
|
||||||
| code | cause | what to do |
|
|
||||||
|---|---|---|
|
|
||||||
| `unauthenticated` | no application session | reload the page; the browser bootstraps one |
|
|
||||||
| `access_denied` | wrong or missing access secret | check `PHOTO_PIPELINE_ACCESS_SECRET`. Attempts are logged with the caller's address only |
|
|
||||||
| `too_many_attempts` | more than five failed secret attempts in a minute | wait. This is the rate limit, not a lockout |
|
|
||||||
| `csrf_failed` | a mutation without the session's token | reload; a stale tab has an old token |
|
|
||||||
| `host_not_allowed` | the `Host` header is not a configured name | add the real hostname to `PHOTO_PIPELINE_ALLOWED_HOSTS` |
|
|
||||||
| `origin_not_allowed` | the request came from another origin | not something a browser tab of this app produces |
|
|
||||||
| `cross_site_blocked` | another site triggered the request | expected — this is the protection working |
|
|
||||||
| `payload_too_large` | the body exceeds `PHOTO_PIPELINE_MAX_REQUEST_BYTES` | send less; every endpoint takes small commands |
|
|
||||||
| `path_not_allowed` | a path resolved outside the library roots | usually a symlink. Nothing outside the roots is reachable, by design |
|
|
||||||
|
|
||||||
## The safety gate
|
|
||||||
|
|
||||||
| code | cause | what to do |
|
|
||||||
|---|---|---|
|
|
||||||
| `dry_run_not_approved` | mutation is gated until a dry run is approved | run `dry-run`, read it, then `approve-dry-run` |
|
|
||||||
| `approval_scope_mismatch` | the approval covers different library roots | approve a report for the roots actually configured |
|
|
||||||
| `approval_unreadable` | the approval record cannot be read | re-approve; do not edit it by hand |
|
|
||||||
|
|
||||||
## Concurrency and staleness
|
|
||||||
|
|
||||||
| code | cause | what to do |
|
|
||||||
|---|---|---|
|
|
||||||
| `lock_held` | a mutating job already holds the lane | wait for it. One at a time is what makes a crash recoverable |
|
|
||||||
| `version_conflict` | you acted on a version that changed underneath you | the view re-renders with the server's truth; decide again |
|
|
||||||
| `invalid_transition` | a state change that the machine does not allow | usually a stale tab; reload |
|
|
||||||
| `job_error` | the job service refused the request | the message says why |
|
|
||||||
| `rename_recovery_required` | an interrupted rename is unresolved | resolve it in [Renames](stages/renames.md). Unrelated mutations stay blocked on purpose |
|
|
||||||
|
|
||||||
## Stage refusals
|
|
||||||
|
|
||||||
| code | cause | what to do |
|
|
||||||
|---|---|---|
|
|
||||||
| `nothing_to_score` | no photo is eligible for safety scoring | the queue is already decided |
|
|
||||||
| `nothing_eligible` | no photo is eligible for analysis | resolve safety decisions first |
|
|
||||||
| `nothing_to_plan` | no approved album name to build a plan from | approve a proposal first |
|
|
||||||
| `invalid_decision` | the decision is not one this cluster accepts | reload the cluster |
|
|
||||||
| `invalid_proposal` | the name is empty, reserved, or has forbidden characters | fix the name; the rules are in [Album proposals](stages/albums.md) |
|
|
||||||
| `unknown_album` | the album is not a folder the library knows | rescan |
|
|
||||||
| `cannot_apply` | the plan is invalid, stale, or another plan is unresolved | read the blockers in the preview |
|
|
||||||
| `cannot_rollback` | the operation is complete or its preconditions no longer hold | rollback is recovery, not undo |
|
|
||||||
|
|
||||||
## Upload
|
|
||||||
|
|
||||||
| code | cause | what to do |
|
|
||||||
|---|---|---|
|
|
||||||
| `stale_preflight` | bytes changed between approval and running | re-run preflight; this is the check that stops the wrong bytes being uploaded |
|
|
||||||
| `not_runnable` | the batch is in a state that must not be re-run | an uncertain batch is verified, never retried |
|
|
||||||
| `requires_verification` | the outcome is unknown; the server may hold the files | verify against Immich, then resolve |
|
|
||||||
| `changed_after_upload` | the file changed after a successful upload | uploading again may create or upgrade an asset |
|
|
||||||
|
|
||||||
## Operations
|
|
||||||
|
|
||||||
| code | cause | what to do |
|
|
||||||
|---|---|---|
|
|
||||||
| `backup_failed` | the snapshot could not be taken or verified | check free space and the message |
|
|
||||||
| `invalid_retention` | a `--keep` value that is not a positive count | the newest backup is never pruned |
|
|
||||||
| `not_found` | no such id | usually a stale link |
|
|
||||||
|
|
||||||
## Generic
|
|
||||||
|
|
||||||
| code | cause |
|
|
||||||
|---|---|
|
|
||||||
| `invalid_request` | the request body failed validation. Field names only — never the values, which end up in logs and screenshots |
|
|
||||||
| `http_error` | a plain HTTP-level refusal, with its status |
|
|
||||||
| `internal_error` | an unhandled error. The code is all you get; the traceback is in the server log, because it can carry paths and credentials |
|
|
||||||
|
|
||||||
## Diagnostics warnings
|
|
||||||
|
|
||||||
Not errors — reported by `diagnostics` and worth acting on. They are listed with
|
|
||||||
their meanings in [Diagnostics](stages/diagnostics.md): `disk_low`, `disk_critical`,
|
|
||||||
`cache_over_quota`, `wal_growth`, `tool_version_drift`, `legacy_process_active`, and
|
|
||||||
`no_roots`.
|
|
||||||
|
|
||||||
## Startup refusals
|
|
||||||
|
|
||||||
Exit codes 2 to 5 are covered by the installation manual under
|
|
||||||
[when it refuses to start](installation.md#when-it-refuses-to-start).
|
|
||||||
@@ -1,66 +0,0 @@
|
|||||||
# A guided first pass
|
|
||||||
|
|
||||||
[← Documentation index](index.md)
|
|
||||||
|
|
||||||
One library, from nothing to a verified upload. Each stage links to its own page; this
|
|
||||||
is the order, and — more importantly — where the points of no return are.
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
The workflow page is the home view and the honest summary: each stage shows its state,
|
|
||||||
its counts, why it is blocked if it is, and the one action that moves it forward.
|
|
||||||
|
|
||||||
| # | stage | reversible afterwards? |
|
|
||||||
|---|---|---|
|
|
||||||
| 0 | [Inventory](stages/inventory.md) | nothing was changed |
|
|
||||||
| 1 | [Duplicate review](stages/duplicates.md) | yes — decisions can be changed, no file is deleted |
|
|
||||||
| 2 | [Safety review](stages/safety.md) | the decision, yes; the EXIF keyword is a file write |
|
|
||||||
| 3 | [Analysis](stages/analysis.md) | the result, yes; the caption and keywords are a file write |
|
|
||||||
| 4 | [Album proposals](stages/albums.md) | yes — approving renames nothing |
|
|
||||||
| 5 | [Renames](stages/renames.md) | **your folders move.** Recoverable, journaled, but real |
|
|
||||||
| 6 | [Upload](stages/uploads.md) | **assets exist on the Immich server.** Not undone from here |
|
|
||||||
| 7 | [Archive](stages/archive.md) | **originals leave active storage.** Restorable while the medium is reachable |
|
|
||||||
| — | [Diagnostics](stages/diagnostics.md) | read-only, any time |
|
|
||||||
|
|
||||||
## Before the first run
|
|
||||||
|
|
||||||
Take a backup of the photo library itself. This application is careful — it previews,
|
|
||||||
journals, and verifies — but it is the first time you are pointing it at your
|
|
||||||
photographs, and a backup is cheaper than confidence.
|
|
||||||
|
|
||||||
If the library is irreplaceable, turn on `PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL`
|
|
||||||
before anything else. Every mutating request is then refused until you have produced
|
|
||||||
a read-only reconciliation report and approved it by name.
|
|
||||||
|
|
||||||
## The pass
|
|
||||||
|
|
||||||
1. **Scan.** Inventory → *Rescan*. Reads only. Check the count, and check that
|
|
||||||
nothing under `_IGNORE/` appears.
|
|
||||||
2. **Resolve duplicates.** Exact byte matches can be accepted as recommended; anything
|
|
||||||
fuzzy gets looked at. Resolving first is what stops you paying to analyse the same
|
|
||||||
photo twice.
|
|
||||||
3. **Decide safety.** Every canonical photo becomes `sfw` or `nsfw`. Nothing reaches
|
|
||||||
the vision provider until it is confirmed SFW — this is the gate the whole design
|
|
||||||
exists around.
|
|
||||||
4. **Analyse.** Only confirmed-SFW photos. Each result is written to the database and
|
|
||||||
projected into EXIF, then read back and verified.
|
|
||||||
5. **Propose album names.** Evidence from what was analysed. Edit anything you disagree
|
|
||||||
with. Approving changes no file.
|
|
||||||
6. **Build the rename plan, read it, apply it.** The preview shows every old → new
|
|
||||||
path. This is where folders move.
|
|
||||||
7. **Rescan.** Paths are reconciled; identities are unchanged.
|
|
||||||
8. **Upload.** Preflight lists every blocker. The command is previewed without the API
|
|
||||||
key. One album at a time.
|
|
||||||
9. **Verify.** An uncertain upload is not a failure and not a success — ask the server.
|
|
||||||
10. **Archive** (optional). Copy, verify, and only then reclaim the space.
|
|
||||||
|
|
||||||
Stop anywhere. Every stage is resumable, and closing the browser cancels nothing.
|
|
||||||
|
|
||||||
## Two rules worth internalising
|
|
||||||
|
|
||||||
**The application refuses more than it warns.** A refusal is not a fault; it is the
|
|
||||||
design working. [Errors and refusals](errors.md) explains each one.
|
|
||||||
|
|
||||||
**One long job at a time.** Browsing stays available while a job runs, but a second
|
|
||||||
mutating job is refused — that is what makes a crash recoverable rather than
|
|
||||||
ambiguous.
|
|
||||||
|
Before Width: | Height: | Size: 64 KiB |
|
Before Width: | Height: | Size: 31 KiB |
|
Before Width: | Height: | Size: 128 KiB |
|
Before Width: | Height: | Size: 24 KiB |
|
Before Width: | Height: | Size: 55 KiB |
|
Before Width: | Height: | Size: 78 KiB |
|
Before Width: | Height: | Size: 31 KiB |
|
Before Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 101 KiB |
|
Before Width: | Height: | Size: 63 KiB |
@@ -1,68 +0,0 @@
|
|||||||
# Photo Pipeline documentation
|
|
||||||
|
|
||||||
One local application that takes a photo library from discovery to a verified Immich
|
|
||||||
upload: duplicate detection, safety review, content analysis, album naming, guarded
|
|
||||||
renaming, upload, and archive — one visible, resume-safe workflow.
|
|
||||||
|
|
||||||
These pages are readable three ways, and they are the same files each time: in the
|
|
||||||
repository under `docs/`, on Gitea, and inside the running application under
|
|
||||||
**Docs**. There is no separate copy to fall out of date.
|
|
||||||
|
|
||||||
## Read in this order
|
|
||||||
|
|
||||||
1. [Overview](overview.md) — what the application does, the stages it moves a photo
|
|
||||||
through, and the rules it will not break.
|
|
||||||
2. [Installation and operations](installation.md) — host and container installation,
|
|
||||||
every setting, the first-run checklist, upgrades, backup and restore, and what
|
|
||||||
each refusal at startup means.
|
|
||||||
3. [A guided first pass](first-pass.md) — one library from scan to verified upload,
|
|
||||||
with the point of no return named in each stage.
|
|
||||||
4. [Architecture](architecture.md) — the context and runtime diagrams, what each
|
|
||||||
module owns, the three journals a restart reads, and where every invariant is
|
|
||||||
enforced.
|
|
||||||
|
|
||||||
## The stages, one page each
|
|
||||||
|
|
||||||
Every page answers the same four questions: what the stage is for, what you decide,
|
|
||||||
what it changes on disk or on the server, and what it refuses.
|
|
||||||
|
|
||||||
1. [Inventory and discovery](stages/inventory.md)
|
|
||||||
2. [Duplicate review](stages/duplicates.md)
|
|
||||||
3. [Safety review](stages/safety.md)
|
|
||||||
4. [Analysis](stages/analysis.md)
|
|
||||||
5. [Album proposals](stages/albums.md)
|
|
||||||
6. [Renames](stages/renames.md)
|
|
||||||
7. [Upload](stages/uploads.md)
|
|
||||||
8. [Archive](stages/archive.md)
|
|
||||||
9. [Diagnostics and library statistics](stages/diagnostics.md)
|
|
||||||
|
|
||||||
## When something goes wrong
|
|
||||||
|
|
||||||
- [Errors and refusals](errors.md) — every error code, what caused it, what to do.
|
|
||||||
- [Recovery](recovery.md) — what the application resolves by itself, and what needs
|
|
||||||
you.
|
|
||||||
|
|
||||||
## Keeping these pages true
|
|
||||||
|
|
||||||
Documentation that disagrees with the application is worse than none, because it is
|
|
||||||
trusted. So one command compares them:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m photo_pipeline docs-gate
|
|
||||||
```
|
|
||||||
|
|
||||||
It runs every `phase_i` check — dead links and anchors, pages nobody links to, images
|
|
||||||
nobody shows, settings, commands, exit codes, error codes, and states that no longer
|
|
||||||
exist in the code, the pages rendering in a real browser without a console or policy
|
|
||||||
error, and the screenshots still showing what the application shows. It retains its
|
|
||||||
evidence and **accepts no skipped check**: a check that did not run is a page nobody
|
|
||||||
compared. CI runs it on every pull request.
|
|
||||||
|
|
||||||
The repository's own build, test, and release notes live in the
|
|
||||||
[README](../README.md).
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
A page tells you what a stage **changes on disk or on the server** before it tells
|
|
||||||
you how to run it. Refusals are documented as intentional: this application would
|
|
||||||
rather stop and explain than guess about somebody's photographs.
|
|
||||||
@@ -1,336 +0,0 @@
|
|||||||
# Installation and operations
|
|
||||||
|
|
||||||
[← Documentation index](index.md)
|
|
||||||
|
|
||||||
From nothing to a running instance pointed at your photo library, and everything you
|
|
||||||
need afterwards: upgrading, backing up, restoring, and understanding a refusal.
|
|
||||||
|
|
||||||
Two ways to install. **Container** is the deployment this project builds and ships;
|
|
||||||
**host** is what you want for development or a single machine you already manage.
|
|
||||||
Both run the same two processes against the same database.
|
|
||||||
|
|
||||||
> Read [what it will not do](overview.md#what-it-will-not-do) first. Several of the
|
|
||||||
> steps below only make sense once you know which rules the application is keeping.
|
|
||||||
|
|
||||||
## Before you start
|
|
||||||
|
|
||||||
| you need | version | why |
|
|
||||||
|---|---|---|
|
|
||||||
| Python | 3.12 or newer | the application |
|
|
||||||
| `exiftool` | 13.x (the image pins `13.25+dfsg-1`) | every EXIF read and write |
|
|
||||||
| `immich-go` | 0.32.0 (pinned in the image) | the upload stage only |
|
|
||||||
| Docker + the compose plugin | any current | the container installation only |
|
|
||||||
| a vision provider key | — | the analysis stage only |
|
|
||||||
|
|
||||||
You also need a photo library you can afford to be wrong about. Take a backup of it
|
|
||||||
before pointing anything at it for the first time, and consider running with
|
|
||||||
[`PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL`](#the-settings) on.
|
|
||||||
|
|
||||||
## Host installation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3.12 -m venv .venv
|
|
||||||
.venv/bin/pip install -e ".[vision]" # drop [vision] for a review-only install
|
|
||||||
```
|
|
||||||
|
|
||||||
Configure it. Every setting is an environment variable; a `.env` file in the working
|
|
||||||
directory is read at startup:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp .env.example .env && $EDITOR .env
|
|
||||||
```
|
|
||||||
|
|
||||||
`.env.example` lists every variable with its default and no values at all. For a
|
|
||||||
loopback installation you need exactly one:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PHOTO_PIPELINE_LIBRARY_ROOTS=/home/you/Pictures
|
|
||||||
```
|
|
||||||
|
|
||||||
Then migrate and start the two processes:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/python -m photo_pipeline migrate # create or upgrade the database
|
|
||||||
.venv/bin/python -m photo_pipeline serve # API and browser app on 127.0.0.1:8000
|
|
||||||
.venv/bin/python -m photo_pipeline worker # second terminal
|
|
||||||
```
|
|
||||||
|
|
||||||
**Both processes are required.** `serve` enqueues work and renders the application;
|
|
||||||
nothing is scanned, scored, analysed, renamed, uploaded, or archived without a
|
|
||||||
worker. Open <http://127.0.0.1:8000/app/> and confirm the workflow page loads.
|
|
||||||
|
|
||||||
### The configuration file
|
|
||||||
|
|
||||||
`.env`, or any path named by `PHOTO_PIPELINE_ENV_FILE`. It is parsed, never executed:
|
|
||||||
`KEY=value` lines, `#` comments, optional quotes, no interpolation and no `export`. A
|
|
||||||
configuration file that can run code is a configuration file that can be a
|
|
||||||
vulnerability.
|
|
||||||
|
|
||||||
**Anything already exported in the shell wins.** The file is your standing
|
|
||||||
configuration; the environment is the override for one run.
|
|
||||||
|
|
||||||
The archived CLI's names still work, so an existing `photo_analyzer.env` can be used
|
|
||||||
as it is:
|
|
||||||
|
|
||||||
| in the file | applied as |
|
|
||||||
|---|---|
|
|
||||||
| `LLM_API_KEY` / `GEMINI_API_KEY` | `OPENAI_API_KEY` |
|
|
||||||
| `LLM_BASE_URL` | `OPENAI_BASE_URL` |
|
|
||||||
| `LIBRARY` | `PHOTO_PIPELINE_LIBRARY_ROOTS` |
|
|
||||||
|
|
||||||
`.env` and `*.env` are gitignored and refused by the repository's safety checks. The
|
|
||||||
file holds a real key; it must never be committed.
|
|
||||||
|
|
||||||
## Container installation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp .env.example .env && $EDITOR .env
|
|
||||||
docker compose up -d --build
|
|
||||||
```
|
|
||||||
|
|
||||||
The composition is one `serve` container, one `worker` container, a one-shot
|
|
||||||
`migrate` that both wait for, one bind-mounted library, and one named data volume.
|
|
||||||
|
|
||||||
Three variables it cannot start without:
|
|
||||||
|
|
||||||
| variable | is |
|
|
||||||
|---|---|
|
|
||||||
| `PHOTO_PIPELINE_LIBRARY_HOST_PATH` | the library on this host |
|
|
||||||
| `PHOTO_PIPELINE_LIBRARY_ROOTS` | where that library is mounted **inside** the container |
|
|
||||||
| `PHOTO_PIPELINE_ACCESS_SECRET` | required, because publishing a port means the app is reachable from outside the container |
|
|
||||||
|
|
||||||
Set `PHOTO_PIPELINE_UID` and `PHOTO_PIPELINE_GID` to the owner of the library: what
|
|
||||||
the containers rename and rewrite keeps that ownership.
|
|
||||||
|
|
||||||
### The data volume
|
|
||||||
|
|
||||||
The `data` volume holds the database, its write-ahead log, the thumbnail cache, the
|
|
||||||
operation journals, and the backups. **It must stay on a local filesystem.** SQLite
|
|
||||||
in WAL mode needs real local locking, so NFS, SMB, and network volume drivers do not
|
|
||||||
slow it down — they corrupt it. The library bind mount has no such restriction.
|
|
||||||
|
|
||||||
### Ports and reaching it
|
|
||||||
|
|
||||||
The API port is published to `127.0.0.1` unless `PHOTO_PIPELINE_PUBLISH_ADDRESS` says
|
|
||||||
otherwise. Being reachable *was* the authentication in earlier versions: whoever could
|
|
||||||
open the port owned the library. So the moment the application answers to anything but
|
|
||||||
loopback — a hostname, a proxy, `0.0.0.0` — the access secret becomes mandatory and
|
|
||||||
`serve` refuses to start without it rather than publishing your photographs.
|
|
||||||
|
|
||||||
Behind a reverse proxy:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PHOTO_PIPELINE_ALLOWED_HOSTS=photos.example.com
|
|
||||||
PHOTO_PIPELINE_ACCESS_SECRET=… # python -c 'import secrets; print(secrets.token_urlsafe(32))'
|
|
||||||
PHOTO_PIPELINE_TRUSTED_PROXIES=10.0.0.2 # only the proxy's own address
|
|
||||||
```
|
|
||||||
|
|
||||||
`X-Forwarded-Proto` and `X-Forwarded-Host` are believed only from a trusted-proxy
|
|
||||||
address, so a client cannot declare its own origin. The browser asks for the secret
|
|
||||||
once per tab. Wrong secrets are rate-limited and logged with the caller's address
|
|
||||||
only. The health endpoints stay open so an orchestrator can restart the container;
|
|
||||||
nothing else is.
|
|
||||||
|
|
||||||
### Deployment
|
|
||||||
|
|
||||||
The stack is managed from git by Portainer, which redeploys on a webhook after CI
|
|
||||||
publishes an image built from `main`. Runtime secrets live in the Portainer stack's
|
|
||||||
environment rather than in the repository, so rotation happens in one place and
|
|
||||||
reading the repository discloses nothing.
|
|
||||||
|
|
||||||
## The settings
|
|
||||||
|
|
||||||
Every variable, its default, and whether it is a secret. `.env.example` is the same
|
|
||||||
list in copyable form.
|
|
||||||
|
|
||||||
### Library and data
|
|
||||||
|
|
||||||
| variable | default | meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `PHOTO_PIPELINE_LIBRARY_ROOTS` | none | the library boundary, `os.pathsep`-separated. No path outside these roots is ever read or written |
|
|
||||||
| `PHOTO_PIPELINE_DATA_DIR` | `data` | database, WAL, thumbnail cache, journals, backups. Never inside the library |
|
|
||||||
| `PHOTO_PIPELINE_DB_PATH` | `<data dir>/photo_pipeline.db` | the database file, if it must live elsewhere |
|
|
||||||
| `PHOTO_PIPELINE_ENV_FILE` | `.env` | where to read the configuration file from |
|
|
||||||
|
|
||||||
### Serving and the trust boundary
|
|
||||||
|
|
||||||
| variable | default | meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `PHOTO_PIPELINE_HOST` | `127.0.0.1` | bind address |
|
|
||||||
| `PHOTO_PIPELINE_PORT` | `8000` | port |
|
|
||||||
| `PHOTO_PIPELINE_ALLOWED_HOSTS` | none | comma-separated names the app answers to besides loopback. Naming one makes the access secret mandatory |
|
|
||||||
| `PHOTO_PIPELINE_ACCESS_SECRET` | none | **secret.** Traded for the session cookie at `GET /api/v1/session` |
|
|
||||||
| `PHOTO_PIPELINE_TRUSTED_PROXIES` | none | comma-separated peer addresses whose forwarded headers may be believed |
|
|
||||||
| `PHOTO_PIPELINE_MAX_REQUEST_BYTES` | `1048576` | largest request body accepted |
|
|
||||||
|
|
||||||
### Logging
|
|
||||||
|
|
||||||
| variable | default | meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `PHOTO_PIPELINE_LOG_LEVEL` | `INFO` | standard Python levels |
|
|
||||||
| `PHOTO_PIPELINE_LOG_FORMAT` | `json` | `json` or `text` |
|
|
||||||
|
|
||||||
### Limits
|
|
||||||
|
|
||||||
| variable | default | meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `PHOTO_PIPELINE_THUMBNAIL_CACHE_QUOTA_BYTES` | `500000000` | cache quota; over it is a diagnostics warning |
|
|
||||||
| `PHOTO_PIPELINE_THUMBNAIL_MAX_PIXELS` | `100000000` | refuse to decode anything larger. This is the decompression-bomb guard |
|
|
||||||
| `PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES` | `1000000000` | free space an archive destination must keep beyond the transfer itself |
|
|
||||||
|
|
||||||
### The safety gate
|
|
||||||
|
|
||||||
| variable | default | meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL` | `false` | refuse every mutating request until a read-only dry run of this library has been produced and approved |
|
|
||||||
|
|
||||||
### External services
|
|
||||||
|
|
||||||
| variable | default | meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `PHOTO_PIPELINE_VISION_API_KEY` | none | **secret.** Without it the analysis stage cannot run |
|
|
||||||
| `PHOTO_PIPELINE_IMMICH_SERVER_URL` | none | the Immich server |
|
|
||||||
| `PHOTO_PIPELINE_IMMICH_API_KEY` | none | **secret.** |
|
|
||||||
| `PHOTO_PIPELINE_IMMICH_GO_BINARY` | `immich-go` | the uploader, found on `PATH` |
|
|
||||||
|
|
||||||
### Composition only
|
|
||||||
|
|
||||||
Read by `docker-compose.yml`, not by the application:
|
|
||||||
`PHOTO_PIPELINE_IMAGE`, `PHOTO_PIPELINE_LIBRARY_HOST_PATH`,
|
|
||||||
`PHOTO_PIPELINE_PUBLISH_ADDRESS`, `PHOTO_PIPELINE_UID`, `PHOTO_PIPELINE_GID`.
|
|
||||||
|
|
||||||
A secret is never written to a log, never returned by the API, never stored in the
|
|
||||||
database, and never recorded in a backup manifest — a manifest says `configured`, not
|
|
||||||
the value. Do not put a real key in an example, a ticket, or a screenshot.
|
|
||||||
|
|
||||||
## First run checklist
|
|
||||||
|
|
||||||
Not "it started" — verified.
|
|
||||||
|
|
||||||
1. **The database is at the current revision.**
|
|
||||||
`python -m photo_pipeline migrate` exits 0.
|
|
||||||
2. **The application answers.**
|
|
||||||
`curl -s localhost:8000/api/v1/health/ready` returns 200.
|
|
||||||
3. **The library was found.** Open the app, run a scan from the Inventory view, and
|
|
||||||
check the count against what you expect. Anything under `_IGNORE/` must be missing
|
|
||||||
from it — that is the exclusion working, not a bug.
|
|
||||||
4. **The worker is claiming.** The scan job reaches `succeeded`. If it stays
|
|
||||||
`queued`, no worker is running.
|
|
||||||
5. **Diagnostics are clean.**
|
|
||||||
`python -m photo_pipeline diagnostics` reports free space and an empty `warnings`.
|
|
||||||
6. **Nothing was modified.** The scan is read-only; your files' timestamps are
|
|
||||||
unchanged.
|
|
||||||
|
|
||||||
Only then point it at the whole library.
|
|
||||||
|
|
||||||
## Operations
|
|
||||||
|
|
||||||
Every operation is the same CLI, on a host or in the composition:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m photo_pipeline diagnostics
|
|
||||||
docker compose run --rm --no-deps api diagnostics
|
|
||||||
```
|
|
||||||
|
|
||||||
`--no-deps` keeps a one-off command from starting a second stack; `api` is only the
|
|
||||||
service it borrows the image and mounts from.
|
|
||||||
|
|
||||||
### Upgrading
|
|
||||||
|
|
||||||
1. `python -m photo_pipeline backup --reason before-upgrade`
|
|
||||||
2. Pull the new version (or `docker compose pull && docker compose up -d`).
|
|
||||||
3. Migrations run by themselves at startup, and a **pending schema change is
|
|
||||||
snapshotted first**. If the upgrade fails, the previous database and its
|
|
||||||
`pre-migration` backup are both intact, and the error log names the backup
|
|
||||||
directory.
|
|
||||||
|
|
||||||
An up-to-date database is not backed up again on every start.
|
|
||||||
|
|
||||||
### Backing up
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m photo_pipeline backup --reason weekly --keep 7
|
|
||||||
python -m photo_pipeline verify-backup data/backups/<name>
|
|
||||||
```
|
|
||||||
|
|
||||||
Backups go through SQLite's online backup API, never a file copy: with WAL enabled
|
|
||||||
the `.db` file alone is missing every committed page still in the write-ahead log.
|
|
||||||
Each backup is a directory holding the snapshot and a `manifest.json` — schema
|
|
||||||
revision, SHA-256, row counts, the archive media the library depends on, and which
|
|
||||||
settings were configured.
|
|
||||||
|
|
||||||
`verify-backup` runs `PRAGMA integrity_check` **and** `PRAGMA foreign_key_check`,
|
|
||||||
compares the snapshot's SHA-256 against the manifest, and re-counts every table it
|
|
||||||
recorded. Bit rot, a truncated copy, and a "repaired" snapshot all fail it.
|
|
||||||
|
|
||||||
`--keep N` prunes the oldest and never the newest. The same is available at
|
|
||||||
`GET /api/v1/diagnostics`, `GET|POST /api/v1/backups`,
|
|
||||||
`GET /api/v1/backups/{name}/verify`, and `POST /api/v1/backups/prune`.
|
|
||||||
|
|
||||||
### Restoring
|
|
||||||
|
|
||||||
Restore is deliberately **not** an API call. It replaces the state of an
|
|
||||||
installation, so it belongs to a stopped one and a person at a terminal.
|
|
||||||
|
|
||||||
1. Stop `serve` and `worker`.
|
|
||||||
2. `python -m photo_pipeline verify-backup data/backups/<name>` — never restore an
|
|
||||||
unverified snapshot, and `restore` will refuse one anyway.
|
|
||||||
3. `python -m photo_pipeline restore data/backups/<name> --into /path/to/fresh-data`.
|
|
||||||
A target that already holds a database is refused; recovering in place means
|
|
||||||
moving the old data directory aside first.
|
|
||||||
4. Point `PHOTO_PIPELINE_DATA_DIR` at the restored directory and run `migrate`.
|
|
||||||
5. Run an inventory scan, so paths are reconciled against the real library.
|
|
||||||
6. Mount every archive location the manifest names before archiving again. The
|
|
||||||
database records where archived originals are; it does not contain them.
|
|
||||||
|
|
||||||
Practise this against a copy before you need it.
|
|
||||||
|
|
||||||
### Watching it
|
|
||||||
|
|
||||||
`diagnostics` reports the database, write-ahead log, thumbnail cache, uploader
|
|
||||||
reports, backups, and logs separately, with free space and warnings for low disk
|
|
||||||
(`disk_low`, `disk_critical`), a cache over its quota, a write-ahead log outgrowing
|
|
||||||
its database, and a legacy CLI writing the library.
|
|
||||||
|
|
||||||
Logs go to stdout in JSON by default — `docker compose logs -f worker`, or your
|
|
||||||
service manager's journal on a host. Uploader output is kept per attempt under the
|
|
||||||
data directory with the API key scrubbed line by line.
|
|
||||||
|
|
||||||
## When it refuses to start
|
|
||||||
|
|
||||||
Every refusal below is deliberate. The application would rather stop and explain than
|
|
||||||
guess about somebody's photographs.
|
|
||||||
|
|
||||||
| exit code | meaning | what to do |
|
|
||||||
|---|---|---|
|
|
||||||
| `1` | the operation failed and said why — an unverifiable backup, an occupied restore target, an unknown benchmark profile | read the message; nothing was changed |
|
|
||||||
| `2` | **the library lock is held.** Another `serve` or `worker` owns this data directory | stop the other process. The message names the holder; a lock whose process is gone is taken over automatically |
|
|
||||||
| `3` | **a legacy CLI looks active.** The frozen command-line tools are writing this library | stop them. `--allow-legacy` overrides and you own the outcome |
|
|
||||||
| `4` | **configuration refused.** The application is reachable beyond loopback and no access secret is set | set `PHOTO_PIPELINE_ACCESS_SECRET`, or bind to loopback only |
|
|
||||||
| `5` | **a library root is not mounted** (containers only) | fix the bind mount. Refusing is what stops the container writing into its own throwaway layer instead of your library |
|
|
||||||
|
|
||||||
Other things that look like faults and are not:
|
|
||||||
|
|
||||||
- **`403 host_not_allowed`** — the `Host` header is not a name in
|
|
||||||
`PHOTO_PIPELINE_ALLOWED_HOSTS`. Add the real hostname; do not add `*`.
|
|
||||||
- **`409 lock_held`** — a mutating job is already running. One at a time is what
|
|
||||||
makes a crash recoverable.
|
|
||||||
- **`409 rename_recovery_required`** — an interrupted rename is unresolved. Resolve
|
|
||||||
it in the Renames view; unrelated mutations stay blocked until then, on purpose.
|
|
||||||
- **A job stuck in `queued`** — no worker is running.
|
|
||||||
- **An empty scan** — check `PHOTO_PIPELINE_LIBRARY_ROOTS`, and in a container check
|
|
||||||
that the path is the *container-side* mount, not the host path.
|
|
||||||
|
|
||||||
## What an installer must not work around
|
|
||||||
|
|
||||||
- **One writer.** Do not run two workers, and do not run the frozen CLI beside the
|
|
||||||
application. Each is safe alone and destructive together — the lock is not
|
|
||||||
bureaucracy.
|
|
||||||
- **`_IGNORE/` is never read.** Do not "fix" the exclusion.
|
|
||||||
- **EXIF is verified before upload.** Do not skip the checkpoint to make a stage
|
|
||||||
finish; an unverified projection is exactly the case where the wrong bytes reach
|
|
||||||
Immich.
|
|
||||||
- **The data directory is not the library.** Keep them apart, and keep the data
|
|
||||||
directory on a local filesystem.
|
|
||||||
- **Secrets stay in the environment.** Not in the database, not in a log, not in a
|
|
||||||
commit.
|
|
||||||
@@ -1,77 +0,0 @@
|
|||||||
# Overview
|
|
||||||
|
|
||||||
[← Documentation index](index.md)
|
|
||||||
|
|
||||||
Photo Pipeline sorts a photo library. It finds duplicates before anything expensive
|
|
||||||
happens to them, asks a human which pictures may leave the machine, describes the
|
|
||||||
ones that may, proposes album names from what it found, renames folders under a
|
|
||||||
crash-safe journal, uploads through `immich-go`, and can archive a finished album off
|
|
||||||
active storage without ever forgetting it existed.
|
|
||||||
|
|
||||||
It runs locally. It talks to exactly three things outside itself: a vision provider,
|
|
||||||
an Immich server, and `exiftool` — and it will tell you before it uses any of them.
|
|
||||||
|
|
||||||
## The workflow
|
|
||||||
|
|
||||||
Each stage is a gate, not a tab. A later stage can always be looked at; its actions
|
|
||||||
stay disabled until what they depend on is true.
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart TD
|
|
||||||
I["0 · Inventory<br/>discover, hash, cluster duplicates"] --> S["1 · Safety<br/>score and human decision"]
|
|
||||||
S -->|sfw| A["2 · Analysis<br/>vision provider, EXIF checkpoint"]
|
|
||||||
S -->|nsfw| U
|
|
||||||
A --> B["3 · Albums<br/>proposal, then guarded rename"]
|
|
||||||
B --> U["4 · Upload<br/>immich-go, verified bytes"]
|
|
||||||
U --> R["5 · Archive<br/>copy, verify, then reclaim space"]
|
|
||||||
```
|
|
||||||
|
|
||||||
| Stage | What it decides | What it changes |
|
|
||||||
|---|---|---|
|
|
||||||
| Inventory | which file is the canonical copy of a picture | nothing — it only reads |
|
|
||||||
| Safety | whether a photo may be sent to a cloud provider | one `sfw`/`nsfw` EXIF keyword |
|
|
||||||
| Analysis | what a photo shows | a managed caption segment and additive keywords |
|
|
||||||
| Albums | what a folder should be called | folder names, through a journaled rename |
|
|
||||||
| Upload | which exact bytes reach Immich | nothing locally; assets appear in Immich |
|
|
||||||
| Archive | which album leaves active storage | files move to the archive, after verification |
|
|
||||||
|
|
||||||
## What it will not do
|
|
||||||
|
|
||||||
These are enforced in code, not by convention, and each one is why some action you
|
|
||||||
expected is sometimes refused.
|
|
||||||
|
|
||||||
- **Nothing under `_IGNORE/` is ever read.** Not scanned, not counted, not
|
|
||||||
thumbnailed, not sent anywhere.
|
|
||||||
- **A path is not an identity.** Every picture has a stable id, so moving or renaming
|
|
||||||
it loses no history.
|
|
||||||
- **Only a confirmed-SFW photo reaches the vision provider.** A photo marked NSFW is
|
|
||||||
still uploadable to Immich; it simply never leaves for analysis.
|
|
||||||
- **Metadata is verified, not hoped for.** Every stage that writes EXIF reads it back
|
|
||||||
and proves that what it did not own is unchanged.
|
|
||||||
- **One writer at a time.** A library lock, held by one process, is what makes a
|
|
||||||
crash recoverable instead of ambiguous.
|
|
||||||
- **Nothing irreversible happens without a preview and an explicit approval** that
|
|
||||||
names the exact count.
|
|
||||||
|
|
||||||
## Where things live
|
|
||||||
|
|
||||||
| | |
|
|
||||||
|---|---|
|
|
||||||
| the photo library | wherever you point `PHOTO_PIPELINE_LIBRARY_ROOTS`; mounted read-write |
|
|
||||||
| the database, cache, logs, backups | the data directory, never inside the library |
|
|
||||||
| secrets | the environment, never the database and never a log line |
|
|
||||||
| these documents | `docs/` in the repository, served at `/docs` by the application |
|
|
||||||
|
|
||||||
## Running it
|
|
||||||
|
|
||||||
The short version, for a host installation:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m photo_pipeline migrate
|
|
||||||
python -m photo_pipeline serve # the API and this browser application
|
|
||||||
python -m photo_pipeline worker # the process that does the long work
|
|
||||||
```
|
|
||||||
|
|
||||||
The full procedure, the container path, and every setting are in the
|
|
||||||
[installation manual](installation.md); if you would rather start using it, take
|
|
||||||
[the guided first pass](first-pass.md).
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# Recovery
|
|
||||||
|
|
||||||
[← Documentation index](index.md)
|
|
||||||
|
|
||||||
What the application resolves by itself, and what needs you. The dividing line is
|
|
||||||
simple: it resumes when the evidence is unambiguous, and it stops and asks when it is
|
|
||||||
not. It never guesses about your files.
|
|
||||||
|
|
||||||
## An interrupted rename
|
|
||||||
|
|
||||||
A rename records its intent **before** touching the disk, so after a crash the
|
|
||||||
journal plus the actual filesystem give a verdict per operation:
|
|
||||||
|
|
||||||
| verdict | means | what happens |
|
|
||||||
|---|---|---|
|
|
||||||
| resumable | the move provably did not happen | reset to planned and run again |
|
|
||||||
| rollback-safe | the move happened; the remaining steps can be completed or undone | drive it forward, or roll it back |
|
|
||||||
| manual | the evidence is contradictory | left alone, and it keeps blocking |
|
|
||||||
|
|
||||||
Open [Renames](stages/renames.md); the recovery panel lists every unresolved
|
|
||||||
operation and offers a resolve action only where one is safe. Until then, unrelated
|
|
||||||
mutations are refused with `rename_recovery_required` — building new work on a
|
|
||||||
half-applied move is how a library becomes unexplainable.
|
|
||||||
|
|
||||||
## An uncertain upload
|
|
||||||
|
|
||||||
`unknown_requires_verification` means the uploader died after Immich may have
|
|
||||||
accepted the files. It is **not** retryable. Verification asks the server whether it
|
|
||||||
holds the recorded SHA-1 and answers present, absent, or inconclusive. An
|
|
||||||
inconclusive answer is resolved by you, with your evidence and name recorded in the
|
|
||||||
batch's history.
|
|
||||||
|
|
||||||
## A failed migration
|
|
||||||
|
|
||||||
A pending schema change is snapshotted before it is applied. If the upgrade fails,
|
|
||||||
the previous database and its `pre-migration` backup are both intact and the error
|
|
||||||
log names the backup directory. Stop everything and run the
|
|
||||||
[restore drill](installation.md#restoring).
|
|
||||||
|
|
||||||
## A restored backup
|
|
||||||
|
|
||||||
After restoring, run a scan. Paths are reconciled against the real library and
|
|
||||||
identities are preserved where hashes match. Anything that does not match becomes a
|
|
||||||
visible `stale` or `divergent` state instead of being quietly accepted.
|
|
||||||
|
|
||||||
Mount every archive medium the manifest names before archiving again: the database
|
|
||||||
records where archived originals live, it does not contain them.
|
|
||||||
|
|
||||||
## A job that will not start
|
|
||||||
|
|
||||||
Check, in this order: is a worker running; is another mutating job holding the lane
|
|
||||||
(`lock_held`); is an interrupted rename blocking everything
|
|
||||||
(`rename_recovery_required`); is the library lock held by a process that is still
|
|
||||||
alive (exit code `2` names the holder).
|
|
||||||
|
|
||||||
## What is never automatic
|
|
||||||
|
|
||||||
No file is deleted, no folder is renamed, no upload is retried, and no archive source
|
|
||||||
is removed without either an explicit confirmation from you or a verification the
|
|
||||||
application performed itself. If you are reading this page because something happened
|
|
||||||
that you did not approve, that is a bug worth reporting — with the diagnostics output
|
|
||||||
and the relevant journal, both of which are safe to share: they carry no secrets.
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
# Album proposals
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Turning a folder of analysed photographs into a name a person
|
|
||||||
would have chosen, before any folder is touched.
|
|
||||||
|
|
||||||
**What you decide.** The final name. The proposal is a starting point with its
|
|
||||||
reasoning attached; you edit it, accept it, or ignore it.
|
|
||||||
|
|
||||||
**What it changes.** A row per album: proposed name, rationale, confidence, and your
|
|
||||||
final name, with a version. **Approving renames nothing.** Not one file moves at this
|
|
||||||
stage — approval only marks a name as agreed, and the [rename stage](renames.md) is
|
|
||||||
where it becomes a plan you have to confirm separately.
|
|
||||||
|
|
||||||
**What it refuses.** A name containing `/ \ : * ? " < > |`, a reserved device name, or
|
|
||||||
one that collides with an existing folder is rejected while you type and again on the
|
|
||||||
server. Approval is refused when the evidence changed since the proposal was
|
|
||||||
generated — the proposal is stale, and regenerating is the honest fix. Editing with a
|
|
||||||
stale version returns a conflict and shows you the server's truth rather than
|
|
||||||
overwriting it.
|
|
||||||
|
|
||||||
## Reading the view
|
|
||||||
|
|
||||||
The left pane lists albums with their state: `none`, `proposed`, `edited`,
|
|
||||||
`approved`, `error`, or `stale`. The right pane shows the evidence the name was built
|
|
||||||
from — date range, dominant tags, locations, counts — then the suggested name, the
|
|
||||||
rationale, the confidence, and an editable final name.
|
|
||||||
|
|
||||||
The default naming shape is predictable and sortable:
|
|
||||||
|
|
||||||
```text
|
|
||||||
YYYY-MM — Place — Event
|
|
||||||
YYYY — Event
|
|
||||||
Place — Event
|
|
||||||
```
|
|
||||||
|
|
||||||
## While a rename is unresolved
|
|
||||||
|
|
||||||
Generating and approving proposals is blocked while an interrupted rename is
|
|
||||||
outstanding, with `409 rename_recovery_required`. Resolve the rename first: building
|
|
||||||
new names on top of a half-applied move is how a library becomes hard to reason
|
|
||||||
about.
|
|
||||||
|
|
||||||
Next: [renames](renames.md).
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# Analysis
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Describing what a photograph shows, so albums can be named from
|
|
||||||
evidence and the library can be searched by content.
|
|
||||||
|
|
||||||
**What you decide.** When to run it, and over what scope. The descriptions themselves
|
|
||||||
come from the configured vision provider.
|
|
||||||
|
|
||||||
**What it changes.** For each eligible photo: a stored result — description, tags,
|
|
||||||
approximate year, location hint, model and prompt version — and then an EXIF
|
|
||||||
checkpoint that merges a **managed caption segment** and additive keywords into the
|
|
||||||
file, reads them back, and verifies that nothing else moved. The file's SHA-256 is
|
|
||||||
refreshed after the write.
|
|
||||||
|
|
||||||
**What it refuses.** Anything not confirmed SFW. Anything not canonical. A malformed
|
|
||||||
provider response is quarantined rather than stored. Keywords are only ever added,
|
|
||||||
never removed, because EXIF keywords carry no ownership and deleting a generated one
|
|
||||||
could delete yours.
|
|
||||||
|
|
||||||
## Running it
|
|
||||||
|
|
||||||
The view shows eligible, analysed, pending, and error counts, and the job's live
|
|
||||||
progress. Starting a second mutating job while it runs is refused — that is the
|
|
||||||
one-writer rule, not a queue.
|
|
||||||
|
|
||||||
Cancelling drains safely: completed items stay completed, and resuming continues
|
|
||||||
rather than restarting. A photo may be sent to the provider twice if a crash happens
|
|
||||||
mid-item, but its stored result and its EXIF are written once.
|
|
||||||
|
|
||||||
## Costs
|
|
||||||
|
|
||||||
Each analysed photo is a provider call. Resolving duplicates first is what keeps that
|
|
||||||
number honest, and the counts here are the ones to check before starting a large run.
|
|
||||||
|
|
||||||
Next: [album proposals](albums.md).
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# Archive
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Moving a finished album out of active storage onto another disk,
|
|
||||||
while keeping everything the library knows about it.
|
|
||||||
|
|
||||||
**What you decide.** Which album, which destination, and whether to accept the
|
|
||||||
reclaimed space in exchange for needing that medium mounted to see the originals
|
|
||||||
again.
|
|
||||||
|
|
||||||
**What it changes.** Files are copied to the archive location, verified there, and
|
|
||||||
only then removed from the library. The asset keeps its id, its hashes, its
|
|
||||||
decisions, its analysis, its upload history, and a durable preview thumbnail;
|
|
||||||
`current_path` becomes null and availability becomes `archived_online` or
|
|
||||||
`archived_offline`.
|
|
||||||
|
|
||||||
**What it refuses.** Preflight blocks on: an unverified upload, a checksum that no
|
|
||||||
longer matches the uploaded bytes, an unmounted or unwritable destination,
|
|
||||||
insufficient free space plus the configured reserve, a destination that already
|
|
||||||
holds unexpected paths, a missing durable thumbnail, and any conflicting job. The
|
|
||||||
source is **never** removed before the archived copy is verified — not because Immich
|
|
||||||
reported success, which is an ingest, not a backup.
|
|
||||||
|
|
||||||
## Offline is not missing
|
|
||||||
|
|
||||||
An archived photo whose medium is unplugged is `archived_offline`. It still appears
|
|
||||||
in search, still participates in duplicate detection through its retained hashes and
|
|
||||||
thumbnail, and is never reported as lost. If a full-resolution comparison is needed,
|
|
||||||
the application asks you to mount the named medium rather than guessing.
|
|
||||||
|
|
||||||
## Restore
|
|
||||||
|
|
||||||
Restore is planned and journaled like everything else: the medium must be online, the
|
|
||||||
bytes are copied back and verified into a collision-free destination, a new active
|
|
||||||
path is registered, and a reconciliation scan follows. Existing safety, analysis,
|
|
||||||
EXIF, and upload state stay valid when hashes match — and when they do not, the
|
|
||||||
result is a visible `divergent` state rather than silent acceptance of a different
|
|
||||||
file.
|
|
||||||
|
|
||||||
Next: [diagnostics](diagnostics.md).
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
# Diagnostics and library statistics
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Answering "is this installation healthy, and what is in this
|
|
||||||
library?" — the first thing to look at when a stage behaves unexpectedly.
|
|
||||||
|
|
||||||
**What you decide.** Nothing. Both are read-only.
|
|
||||||
|
|
||||||
**What it changes.** Nothing.
|
|
||||||
|
|
||||||
**What it refuses.** Nothing — both are reads, and both stay available while a job
|
|
||||||
holds the library. That is deliberate: the moment you most need to know what the
|
|
||||||
installation is doing is while it is busy.
|
|
||||||
|
|
||||||
## Statistics
|
|
||||||
|
|
||||||
The Stats view summarises the library itself: how many photos are analysed, the
|
|
||||||
common tags, the distribution across albums and years. It answers questions about
|
|
||||||
your photographs.
|
|
||||||
|
|
||||||
## Diagnostics
|
|
||||||
|
|
||||||
Diagnostics answers questions about the *installation*, and lives at
|
|
||||||
`GET /api/v1/diagnostics` and on the command line:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m photo_pipeline diagnostics
|
|
||||||
```
|
|
||||||
|
|
||||||
It reports the database, write-ahead log, thumbnail cache, uploader reports, backups,
|
|
||||||
and logs as separate sizes, plus free space, the tool versions it found against the
|
|
||||||
ones the image pinned, and which locks are held and by whom.
|
|
||||||
|
|
||||||
Its warnings are the ones worth acting on:
|
|
||||||
|
|
||||||
| warning | means |
|
|
||||||
|---|---|
|
|
||||||
| `disk_low` / `disk_critical` | free space is running out; mutating stages stop before the reserve |
|
|
||||||
| `cache_over_quota` | the thumbnail cache exceeds its configured quota |
|
|
||||||
| `wal_growth` | the write-ahead log is outgrowing its database — usually a long-running reader |
|
|
||||||
| `tool_version_drift` | the installed `exiftool` or `immich-go` is not the version the image pinned |
|
|
||||||
| `legacy_process_active` | the frozen command-line tools are writing this library. Stop them |
|
|
||||||
| `no_roots` | no library root is configured, so there is nothing to work on |
|
|
||||||
|
|
||||||
An empty `warnings` list on a fresh installation is what the
|
|
||||||
[first-run checklist](../installation.md#first-run-checklist) is looking for.
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
# Duplicate review
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Deciding which file is *the* copy of a picture, before anything
|
|
||||||
expensive or irreversible happens to the others.
|
|
||||||
|
|
||||||
**What you decide.** For each cluster: keep the recommended canonical, choose a
|
|
||||||
different one, keep everything as variants, declare it not a duplicate, or defer.
|
|
||||||
|
|
||||||
**What it changes.** Only the decision, recorded as an auditable event. **No file is
|
|
||||||
ever deleted.** Non-canonical copies stay exactly where they are; they are simply
|
|
||||||
excluded from analysis and upload.
|
|
||||||
|
|
||||||
**What it refuses.** Exact byte and identical-pixel matches may be accepted on the
|
|
||||||
recommendation. Anything fuzzy — a crop, a re-encode, a burst frame — requires an
|
|
||||||
explicit confirmation, because a perceptual hash is evidence, not proof.
|
|
||||||
|
|
||||||
## Reading a cluster
|
|
||||||
|
|
||||||
Each member shows its path, role, pHash distance, and size, with synchronised zoom
|
|
||||||
across the previews. The recommendation is visible but never pre-applied for fuzzy
|
|
||||||
matches. The confidence band tells you how much the machine is claiming.
|
|
||||||
|
|
||||||
Rejecting a pair records a negative link, so a later rescan does not keep proposing
|
|
||||||
the same wrong match.
|
|
||||||
|
|
||||||
## Why first
|
|
||||||
|
|
||||||
Resolving duplicates before safety and analysis is what stops you reviewing and
|
|
||||||
paying for the same photograph three times. It also means the album evidence is built
|
|
||||||
from one copy of each picture rather than a skewed count.
|
|
||||||
|
|
||||||
Next: [safety review](safety.md).
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# Inventory and discovery
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Finding every supported photo in the configured library roots and
|
|
||||||
giving each one a permanent identity, so that everything afterwards can refer to a
|
|
||||||
picture rather than a path.
|
|
||||||
|
|
||||||
**What you decide.** Nothing. This stage is the only one with no judgement in it.
|
|
||||||
|
|
||||||
**What it changes.** On disk, nothing at all — discovery reads. In the database it
|
|
||||||
records each photo's path, size, timestamps, full-file SHA-256, normalised pixel hash,
|
|
||||||
and perceptual hash, and it opens a path-history entry.
|
|
||||||
|
|
||||||
**What it refuses.** Anything under an `_IGNORE/` directory is never traversed,
|
|
||||||
counted, hashed, or previewed. Unsupported extensions and unreadable files are
|
|
||||||
reported rather than skipped silently. Nothing outside the configured roots is
|
|
||||||
reachable, including through a symlink.
|
|
||||||
|
|
||||||
## Reading the view
|
|
||||||
|
|
||||||
Each row is one asset: its current path, availability, whether the file is present,
|
|
||||||
and its size. The filter searches paths; the availability selector separates active
|
|
||||||
photos from archived ones.
|
|
||||||
|
|
||||||
A **rescan** after moving files around outside the application is the normal way to
|
|
||||||
reconcile: a file that moved keeps its identity, because identity is the hash and the
|
|
||||||
id, not the location.
|
|
||||||
|
|
||||||
## What good looks like
|
|
||||||
|
|
||||||
The count matches what you expect, `_IGNORE/` contents are absent, and nothing shows
|
|
||||||
as missing. A photo listed as missing means the file is no longer at its recorded
|
|
||||||
path and no new path matched its hashes — that is a question for you, not something
|
|
||||||
the scanner resolves by guessing.
|
|
||||||
|
|
||||||
Next: [duplicate review](duplicates.md).
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
# Renames
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Applying approved album names to the actual folders — the first
|
|
||||||
stage that changes your filesystem.
|
|
||||||
|
|
||||||
**What you decide.** Whether to apply the plan, after reading it. The confirmation
|
|
||||||
names the plan, its version, and its checksum, so what you approve is what runs.
|
|
||||||
|
|
||||||
**What it changes.** Folders move. For each operation: the intent is journaled
|
|
||||||
*before* the disk is touched, the move is made, `assets.current_path` is updated by
|
|
||||||
id in one transaction, the old path is closed and a new one opened in path history,
|
|
||||||
the result is verified, and only then is the operation complete.
|
|
||||||
|
|
||||||
**What it refuses.** A plan is invalid — not merely warned about — when a source is
|
|
||||||
missing or changed, two operations target the same destination, a destination already
|
|
||||||
exists, a path would leave the library or enter `_IGNORE/`, or a source is a symlink.
|
|
||||||
Case-only renames and cross-filesystem moves are allowed but flagged, because they
|
|
||||||
take a different, staged route. A plan whose checksum no longer matches is refused
|
|
||||||
rather than reinterpreted.
|
|
||||||
|
|
||||||
## Reading the preview
|
|
||||||
|
|
||||||
Every operation shows both full paths, the kind of move (`rename`, `case-only
|
|
||||||
rename`, `cross-filesystem move`), how many photos it affects, its journal state, and
|
|
||||||
any issue with its severity. The plan can be exported as JSON before you commit to it.
|
|
||||||
|
|
||||||
**A run cannot be stopped part-way.** It can be interrupted — by a crash, a power
|
|
||||||
cut, a killed container — and interruption is recoverable, but there is no cancel
|
|
||||||
button once it starts. That is stated on the confirmation, not hidden in a manual.
|
|
||||||
|
|
||||||
## After an interruption
|
|
||||||
|
|
||||||
The recovery panel lists each unresolved operation with a verdict read from the
|
|
||||||
journal *and the disk*: resumable, safely rollbackable, or needing a person. It never
|
|
||||||
guesses from a missing path alone. Until it is resolved, unrelated mutations are
|
|
||||||
refused with `409 rename_recovery_required`.
|
|
||||||
|
|
||||||
Rollback is recovery, not undo. A completed operation is terminal — the button only
|
|
||||||
appears for operations that are actually reversible. See
|
|
||||||
[recovery](../recovery.md).
|
|
||||||
|
|
||||||
Next: [upload](uploads.md).
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# Safety review
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Deciding which photographs may be sent to a cloud vision
|
|
||||||
provider. This is the privacy gate the rest of the design is built around.
|
|
||||||
|
|
||||||
**What you decide.** `sfw`, `nsfw`, or defer — per photo, or in bulk from the filters.
|
|
||||||
A local model can score first to order the queue, but the score is a suggestion; the
|
|
||||||
decision is yours.
|
|
||||||
|
|
||||||
**What it changes.** The decision is stored in the database *and* projected into the
|
|
||||||
file: one mutually exclusive `sfw` or `nsfw` keyword written into `Keywords` and
|
|
||||||
`Subject`. The write is merged with existing metadata, read back, and verified, and
|
|
||||||
the file's SHA-256 is refreshed afterwards.
|
|
||||||
|
|
||||||
**What it refuses.** An undecided or deferred photo does not reach the vision
|
|
||||||
provider, and does not reach Immich either. If the read-back shows that a field the
|
|
||||||
stage does not own has changed, the checkpoint is marked `divergent`, the asset is not
|
|
||||||
marked verified, and the next mutating stage is blocked rather than proceeding on
|
|
||||||
metadata nobody trusts.
|
|
||||||
|
|
||||||
## The rule that matters
|
|
||||||
|
|
||||||
> Only a confirmed `sfw` photo may enter cloud content analysis. A confirmed `nsfw`
|
|
||||||
> photo skips analysis entirely but remains eligible for upload to Immich once its
|
|
||||||
> keyword is verified.
|
|
||||||
|
|
||||||
That is why marking something NSFW is not a punishment: it routes the photo around an
|
|
||||||
external service while keeping it in your own library workflow.
|
|
||||||
|
|
||||||
The gate is re-checked *after* the provider call as well. If a decision flips to NSFW
|
|
||||||
while an analysis is in flight, the result is discarded rather than stored.
|
|
||||||
|
|
||||||
## Reading the view
|
|
||||||
|
|
||||||
The tabs filter by state. Each row shows the path, the model's score, its suggestion,
|
|
||||||
your decision, and an `✓ exif` badge once the keyword is verified on disk. A row
|
|
||||||
without that badge has a decision the file does not yet carry.
|
|
||||||
|
|
||||||
Next: [analysis](analysis.md).
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
# Upload
|
|
||||||
|
|
||||||
[← Documentation index](../index.md) · [Guided first pass](../first-pass.md)
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
**What it is for.** Sending an approved album to Immich through `immich-go`, with an
|
|
||||||
audit trail of exactly which bytes went.
|
|
||||||
|
|
||||||
**What you decide.** Which albums, and whether to proceed once the preflight has
|
|
||||||
listed its blockers. Upload is never an automatic consequence of renaming.
|
|
||||||
|
|
||||||
**What it changes.** Nothing locally. On the Immich server, assets appear. Locally a
|
|
||||||
batch is recorded: the album, the asset ids, the redacted command, the uploader
|
|
||||||
version, the pre-upload SHA-256 **and** the SHA-1 Immich matches on, the output log,
|
|
||||||
and the outcome counts.
|
|
||||||
|
|
||||||
**What it refuses.** Preflight blocks on: missing credentials, an unreachable server,
|
|
||||||
a missing `immich-go`, an unresolved rename, an undecided or deferred safety
|
|
||||||
decision, an unverified EXIF checkpoint, incomplete analysis for an SFW photo, a file
|
|
||||||
whose bytes changed since it was checkpointed, and a partial scope you have not
|
|
||||||
explicitly accepted. The approval is then **re-proved immediately before the uploader
|
|
||||||
runs** — bytes edited after you approved fail with `stale_preflight` and the uploader
|
|
||||||
never starts.
|
|
||||||
|
|
||||||
## Reading the view
|
|
||||||
|
|
||||||
The scope shows exactly which albums and how many photos. The command preview is the
|
|
||||||
real command with the API key removed; the key never reaches the browser and never
|
|
||||||
reaches a log.
|
|
||||||
|
|
||||||
## Uncertain is not failed
|
|
||||||
|
|
||||||
If the process dies after the server accepted files, the batch becomes
|
|
||||||
`unknown_requires_verification` — not "failed", and **not retryable**. The server may
|
|
||||||
already hold the photographs, and uploading again would create duplicates or
|
|
||||||
upgrades. Verification asks Immich about the recorded SHA-1 and answers present,
|
|
||||||
absent, or inconclusive; an inconclusive answer is resolved by you, with the evidence
|
|
||||||
recorded.
|
|
||||||
|
|
||||||
If a file's bytes change after a successful upload, it is marked
|
|
||||||
`changed_after_upload` and you are warned that uploading again may create or upgrade
|
|
||||||
an asset rather than doing nothing.
|
|
||||||
|
|
||||||
Next: [archive](archive.md).
|
|
||||||
@@ -170,20 +170,6 @@ a.link:hover { text-decoration: underline; }
|
|||||||
.grid th, .grid td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); vertical-align: top; }
|
.grid th, .grid td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); vertical-align: top; }
|
||||||
.path { font-family: ui-monospace, monospace; font-size: 0.85em; word-break: break-all; }
|
.path { font-family: ui-monospace, monospace; font-size: 0.85em; word-break: break-all; }
|
||||||
|
|
||||||
/* Docs: prose, so it gets a reading measure rather than the full window width. */
|
|
||||||
.doc { max-width: 72ch; }
|
|
||||||
.doc h1 { margin-top: 0; }
|
|
||||||
.doc h2, .doc h3 { margin-top: 1.6em; border-bottom: 1px solid var(--border); padding-bottom: 4px; }
|
|
||||||
.doc code { font-family: ui-monospace, monospace; font-size: 0.9em; background: var(--surface-2); padding: 1px 4px; border-radius: 4px; }
|
|
||||||
.doc pre { background: var(--surface-2); border: 1px solid var(--border); border-radius: var(--radius); padding: 12px; overflow: auto; }
|
|
||||||
.doc pre code { background: none; padding: 0; }
|
|
||||||
.doc table { border-collapse: collapse; width: 100%; }
|
|
||||||
.doc th, .doc td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); vertical-align: top; }
|
|
||||||
.doc blockquote { margin: 1em 0; padding-left: 12px; border-left: 3px solid var(--border); color: var(--muted); }
|
|
||||||
.doc img { max-width: 100%; }
|
|
||||||
.diagram { margin: 1.5em 0; padding: 0; overflow: auto; }
|
|
||||||
.diagram svg { max-width: 100%; height: auto; }
|
|
||||||
|
|
||||||
/* Narrow screens: stack the panes so blockers and actions stay reachable. */
|
/* Narrow screens: stack the panes so blockers and actions stay reachable. */
|
||||||
@media (max-width: 720px) {
|
@media (max-width: 720px) {
|
||||||
.two-pane { grid-template-columns: 1fr; }
|
.two-pane { grid-template-columns: 1fr; }
|
||||||
|
|||||||
@@ -21,7 +21,6 @@
|
|||||||
<a href="#/uploads" data-nav="uploads">Upload</a>
|
<a href="#/uploads" data-nav="uploads">Upload</a>
|
||||||
<a href="#/archive" data-nav="archive">Archive</a>
|
<a href="#/archive" data-nav="archive">Archive</a>
|
||||||
<a href="#/stats" data-nav="stats">Stats</a>
|
<a href="#/stats" data-nav="stats">Stats</a>
|
||||||
<a href="#/docs" data-nav="docs">Docs</a>
|
|
||||||
</nav>
|
</nav>
|
||||||
</header>
|
</header>
|
||||||
<main id="app" aria-live="polite"><!-- views render here --></main>
|
<main id="app" aria-live="polite"><!-- views render here --></main>
|
||||||
|
|||||||
@@ -1,7 +1,5 @@
|
|||||||
import { api } from "./api.js";
|
import { api } from "./api.js";
|
||||||
import { renderArchive, setArchiveRender } from "./archive.js";
|
import { renderArchive, setArchiveRender } from "./archive.js";
|
||||||
import { renderDocs } from "./docs.js";
|
|
||||||
import { show as showIn } from "./dom.js";
|
|
||||||
import { navigate, onRouteChange, parseHash } from "./router.js";
|
import { navigate, onRouteChange, parseHash } from "./router.js";
|
||||||
import { renderRenames, setRenamesRender } from "./renames.js";
|
import { renderRenames, setRenamesRender } from "./renames.js";
|
||||||
import { renderUploads, setUploadsRender } from "./uploads.js";
|
import { renderUploads, setUploadsRender } from "./uploads.js";
|
||||||
@@ -36,7 +34,7 @@ function el(tag, attrs = {}, ...children) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function show(...nodes) {
|
function show(...nodes) {
|
||||||
showIn(root, ...nodes);
|
root.replaceChildren(...nodes);
|
||||||
}
|
}
|
||||||
|
|
||||||
function setActiveNav(view) {
|
function setActiveNav(view) {
|
||||||
@@ -387,7 +385,6 @@ function render() {
|
|||||||
else if (path === "/uploads") renderUploads(root, params);
|
else if (path === "/uploads") renderUploads(root, params);
|
||||||
else if (path === "/archive") renderArchive(root, params);
|
else if (path === "/archive") renderArchive(root, params);
|
||||||
else if (path === "/stats") renderStats(root, params);
|
else if (path === "/stats") renderStats(root, params);
|
||||||
else if (path === "/docs") renderDocs(root, params);
|
|
||||||
else show(errorBanner("Unknown view"));
|
else show(errorBanner("Unknown view"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -9,7 +9,7 @@
|
|||||||
// is not mounted produces an instruction naming it rather than a disabled mystery,
|
// is not mounted produces an instruction naming it rather than a disabled mystery,
|
||||||
// and an operation whose evidence is ambiguous offers no button at all.
|
// and an operation whose evidence is ambiguous offers no button at all.
|
||||||
import { api } from "./api.js";
|
import { api } from "./api.js";
|
||||||
import { el, errorBanner, setActiveNav, show } from "./dom.js";
|
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||||
import { subscribeJob } from "./events.js";
|
import { subscribeJob } from "./events.js";
|
||||||
import { navigate } from "./router.js";
|
import { navigate } from "./router.js";
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@ export async function renderArchive(root, params = {}) {
|
|||||||
try {
|
try {
|
||||||
locations = (await api.archiveLocations()).locations;
|
locations = (await api.archiveLocations()).locations;
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load archive locations: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load archive locations: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
const location = locations.find((l) => l.id === params.location) || locations[0] || null;
|
const location = locations.find((l) => l.id === params.location) || locations[0] || null;
|
||||||
@@ -61,7 +61,7 @@ export async function renderArchive(root, params = {}) {
|
|||||||
),
|
),
|
||||||
outcomeBanner()
|
outcomeBanner()
|
||||||
);
|
);
|
||||||
show(root, ...nodes.filter(Boolean));
|
root.replaceChildren(...nodes.filter(Boolean));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -95,7 +95,7 @@ export async function renderArchive(root, params = {}) {
|
|||||||
archived ? archivedSection(archived.items, locations) : null,
|
archived ? archivedSection(archived.items, locations) : null,
|
||||||
restore ? restoreSection(restore, location) : null
|
restore ? restoreSection(restore, location) : null
|
||||||
);
|
);
|
||||||
show(root, ...nodes.filter(Boolean));
|
root.replaceChildren(...nodes.filter(Boolean));
|
||||||
}
|
}
|
||||||
|
|
||||||
function plans(listed) {
|
function plans(listed) {
|
||||||
|
|||||||
@@ -1,263 +0,0 @@
|
|||||||
// The documentation view (US09-01): the manuals in `docs/`, rendered in the app.
|
|
||||||
//
|
|
||||||
// The same markdown files are the repository's documentation and the deployment's
|
|
||||||
// documentation. An operator who was handed a URL and an access secret has no
|
|
||||||
// checkout in front of them, and the network the application runs on is not assumed
|
|
||||||
// to reach a CDN — so the renderer is vendored and everything here is same-origin.
|
|
||||||
//
|
|
||||||
// Nothing on this page is authenticated. It is served from the static mount beside
|
|
||||||
// the application shell, exactly like `index.html`: the troubleshooting page is
|
|
||||||
// needed most by whoever cannot get past the access secret, and no documentation
|
|
||||||
// file contains anything a session would protect.
|
|
||||||
//
|
|
||||||
// `marked` and `mermaid` are loaded lazily, on the first documentation page and the
|
|
||||||
// first diagram. They are large, and every other view does without them.
|
|
||||||
import { el, errorBanner, setActiveNav } from "./dom.js";
|
|
||||||
|
|
||||||
const DOCS_BASE = "/docs";
|
|
||||||
const VENDOR = "/app/js/vendor";
|
|
||||||
const INDEX_PAGE = "index";
|
|
||||||
// A page name comes from the hash, so it is caller input: no scheme, no traversal,
|
|
||||||
// no absolute path. The server would refuse those too; this refuses them earlier and
|
|
||||||
// without a request that looks like an attempt.
|
|
||||||
const PAGE_PATTERN = /^[\w-]+(\/[\w-]+)*$/;
|
|
||||||
|
|
||||||
let markedPromise;
|
|
||||||
let mermaidPromise;
|
|
||||||
|
|
||||||
function loadMarked() {
|
|
||||||
markedPromise ??= import(`${VENDOR}/marked.esm.js`).then((module) => module.marked);
|
|
||||||
return markedPromise;
|
|
||||||
}
|
|
||||||
|
|
||||||
// mermaid ships one large UMD bundle rather than a self-contained ES module, so it
|
|
||||||
// arrives through a script element. `script-src 'self'` allows it because it is ours
|
|
||||||
// and same-origin; nothing here relaxes that.
|
|
||||||
function loadMermaid() {
|
|
||||||
mermaidPromise ??= new Promise((resolve, reject) => {
|
|
||||||
const script = document.createElement("script");
|
|
||||||
script.src = `${VENDOR}/mermaid.min.js`;
|
|
||||||
script.onload = () => resolve(window.mermaid);
|
|
||||||
script.onerror = () => reject(new Error("the diagram renderer could not be loaded"));
|
|
||||||
document.head.appendChild(script);
|
|
||||||
});
|
|
||||||
return mermaidPromise;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── the pure parts, unit-tested in js/tests/unit.js ──────────────────────────
|
|
||||||
|
|
||||||
/** A stable, readable heading anchor. Letters and digits of any script survive. */
|
|
||||||
export function slug(text) {
|
|
||||||
const cleaned = String(text)
|
|
||||||
.toLowerCase()
|
|
||||||
.trim()
|
|
||||||
.replace(/[^\p{L}\p{N}\s-]/gu, "")
|
|
||||||
.replace(/[\s-]+/g, "-")
|
|
||||||
.replace(/^-|-$/g, "");
|
|
||||||
return cleaned || "section";
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Assign every heading an id, disambiguating repeats the way a reader would
|
|
||||||
* expect: the first `Notes` keeps `#notes`, the second becomes `#notes-2`. */
|
|
||||||
export function assignHeadingIds(headings) {
|
|
||||||
const used = new Map();
|
|
||||||
for (const heading of headings) {
|
|
||||||
const base = slug(heading.textContent);
|
|
||||||
const seen = (used.get(base) || 0) + 1;
|
|
||||||
used.set(base, seen);
|
|
||||||
heading.id = seen === 1 ? base : `${base}-${seen}`;
|
|
||||||
}
|
|
||||||
return headings;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Where a link inside a documentation page should go.
|
|
||||||
*
|
|
||||||
* Markdown links between documents are relative file paths, which a browser would
|
|
||||||
* treat as downloads that leave the application. Returns the in-app target for a
|
|
||||||
* link to another document or to a heading, and `null` for anything else — external
|
|
||||||
* links stay exactly as the author wrote them.
|
|
||||||
*/
|
|
||||||
export function resolveDocLink(currentPage, href) {
|
|
||||||
if (!href) return null;
|
|
||||||
if (/^[a-z][a-z\d+.-]*:/i.test(href) || href.startsWith("//")) return null;
|
|
||||||
if (href.startsWith("#")) return { page: currentPage, anchor: href.slice(1) };
|
|
||||||
const [target, anchor = ""] = href.split("#");
|
|
||||||
if (!/\.md$/i.test(target)) return null;
|
|
||||||
// Resolved against the current document, so `../guides/x.md` means what it means
|
|
||||||
// in the repository. The origin is a placeholder; only the path is used.
|
|
||||||
const resolved = new URL(target, `https://docs.invalid/${currentPage}.md`);
|
|
||||||
const page = decodeURIComponent(resolved.pathname).replace(/^\//, "").replace(/\.md$/i, "");
|
|
||||||
return PAGE_PATTERN.test(page) ? { page, anchor } : null;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Point every in-app link at its route, and leave every other link alone.
|
|
||||||
*
|
|
||||||
* The resolved page is kept on the element, so the reading order can be read back
|
|
||||||
* from the rendered index rather than parsed out of the markdown a second time.
|
|
||||||
*/
|
|
||||||
export function rewriteLinks(article, page) {
|
|
||||||
for (const link of article.querySelectorAll("a[href]")) {
|
|
||||||
const href = link.getAttribute("href");
|
|
||||||
const target = resolveDocLink(page, href);
|
|
||||||
if (target) {
|
|
||||||
link.setAttribute("href", docHash(target.page, target.anchor));
|
|
||||||
link.dataset.docPage = target.page;
|
|
||||||
} else if (/^https?:/i.test(href)) {
|
|
||||||
link.setAttribute("rel", "noreferrer noopener");
|
|
||||||
link.setAttribute("target", "_blank");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return article;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The reading order, taken from the index document itself — one list, in one
|
|
||||||
* place, that is equally the index on Gitea and the navigation here. */
|
|
||||||
export function documentIndex(indexElement) {
|
|
||||||
const pages = [];
|
|
||||||
for (const link of indexElement.querySelectorAll("a[data-doc-page]")) {
|
|
||||||
const page = link.dataset.docPage;
|
|
||||||
if (page !== INDEX_PAGE && !pages.some((entry) => entry.page === page)) {
|
|
||||||
pages.push({ page, title: link.textContent.trim() });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return pages;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── the view ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
async function fetchPage(page) {
|
|
||||||
const response = await fetch(`${DOCS_BASE}/${page}.md`, { headers: { accept: "text/markdown" } });
|
|
||||||
if (!response.ok) throw new Error(`documentation page unavailable: ${response.status}`);
|
|
||||||
return response.text();
|
|
||||||
}
|
|
||||||
|
|
||||||
async function toArticle(markdown, page) {
|
|
||||||
const marked = await loadMarked();
|
|
||||||
const article = el("article", { class: "doc", "data-testid": "doc", "data-page": page });
|
|
||||||
// The only innerHTML in the application, and deliberate: rendering markdown *is*
|
|
||||||
// producing HTML. The input is a file from this repository, and the page's CSP
|
|
||||||
// (`script-src 'self'`, no `'unsafe-inline'`) means injected script and inline
|
|
||||||
// handlers do not run even if one ever were not.
|
|
||||||
article.innerHTML = marked.parse(markdown, { async: false });
|
|
||||||
assignHeadingIds(article.querySelectorAll("h1, h2, h3, h4, h5, h6"));
|
|
||||||
rewriteLinks(article, page);
|
|
||||||
return article;
|
|
||||||
}
|
|
||||||
|
|
||||||
function docHash(page, anchor = "") {
|
|
||||||
const query = new URLSearchParams(anchor ? { page, anchor } : { page });
|
|
||||||
return `#/docs?${query}`;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Diagrams are authored as ```mermaid blocks so the source is diffable and Gitea
|
|
||||||
* renders them natively. A failure here replaces the diagram, never the page. */
|
|
||||||
async function renderDiagrams(article) {
|
|
||||||
const blocks = [...article.querySelectorAll("pre > code.language-mermaid")];
|
|
||||||
if (!blocks.length) return;
|
|
||||||
let mermaid;
|
|
||||||
try {
|
|
||||||
mermaid = await loadMermaid();
|
|
||||||
mermaid.initialize({ startOnLoad: false, securityLevel: "strict", theme: "dark" });
|
|
||||||
} catch (error) {
|
|
||||||
blocks.forEach((block) => block.closest("pre").replaceWith(errorBanner(error.message)));
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
for (const [index, block] of blocks.entries()) {
|
|
||||||
const figure = el("figure", { class: "diagram", "data-testid": "diagram" });
|
|
||||||
block.closest("pre").replaceWith(figure);
|
|
||||||
try {
|
|
||||||
const { svg } = await mermaid.render(`diagram-${index}-${Date.now()}`, block.textContent);
|
|
||||||
figure.innerHTML = svg;
|
|
||||||
} catch (error) {
|
|
||||||
figure.replaceWith(errorBanner(`Diagram could not be drawn: ${error.message}`));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function notFound() {
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "alert", role: "alert", "data-testid": "doc-not-found" },
|
|
||||||
"That documentation page does not exist. ",
|
|
||||||
el("a", { class: "link", href: docHash(INDEX_PAGE) }, "Back to the documentation index")
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function sidebar(pages, current) {
|
|
||||||
return el(
|
|
||||||
"nav",
|
|
||||||
{ class: "card", "aria-label": "Documentation" },
|
|
||||||
el("h2", {}, "Documentation"),
|
|
||||||
el(
|
|
||||||
"ul",
|
|
||||||
{ class: "album-list", "data-testid": "doc-pages" },
|
|
||||||
el(
|
|
||||||
"li",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"a",
|
|
||||||
{
|
|
||||||
href: docHash(INDEX_PAGE),
|
|
||||||
"aria-current": current === INDEX_PAGE ? "true" : false,
|
|
||||||
},
|
|
||||||
"Index"
|
|
||||||
)
|
|
||||||
),
|
|
||||||
...pages.map(({ page, title }) =>
|
|
||||||
el(
|
|
||||||
"li",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"a",
|
|
||||||
{
|
|
||||||
href: docHash(page),
|
|
||||||
"data-page": page,
|
|
||||||
"aria-current": page === current ? "true" : false,
|
|
||||||
},
|
|
||||||
title
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
export async function renderDocs(root, params = {}) {
|
|
||||||
setActiveNav("docs");
|
|
||||||
const requested = params.page || INDEX_PAGE;
|
|
||||||
const page = PAGE_PATTERN.test(requested) ? requested : "";
|
|
||||||
|
|
||||||
let indexArticle;
|
|
||||||
try {
|
|
||||||
indexArticle = await toArticle(await fetchPage(INDEX_PAGE), INDEX_PAGE);
|
|
||||||
} catch (error) {
|
|
||||||
root.replaceChildren(errorBanner(`Documentation is unavailable: ${error.message}`));
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const pages = documentIndex(indexArticle);
|
|
||||||
|
|
||||||
let article;
|
|
||||||
if (!page) article = notFound();
|
|
||||||
else if (page === INDEX_PAGE) article = indexArticle;
|
|
||||||
else {
|
|
||||||
try {
|
|
||||||
article = await toArticle(await fetchPage(page), page);
|
|
||||||
} catch {
|
|
||||||
article = notFound();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
root.replaceChildren(el("div", { class: "two-pane" }, sidebar(pages, page), article));
|
|
||||||
await renderDiagrams(article);
|
|
||||||
scrollToAnchor(params.anchor);
|
|
||||||
}
|
|
||||||
|
|
||||||
// A documentation anchor cannot live in the hash — the hash is the route — so it
|
|
||||||
// travels as a parameter and is applied after the page renders.
|
|
||||||
function scrollToAnchor(anchor) {
|
|
||||||
if (!anchor) return;
|
|
||||||
const target = document.getElementById(anchor);
|
|
||||||
if (target) target.scrollIntoView({ block: "start" });
|
|
||||||
}
|
|
||||||
@@ -21,19 +21,6 @@ export function errorBanner(message) {
|
|||||||
return el("div", { class: "alert", role: "alert" }, message);
|
return el("div", { class: "alert", role: "alert" }, message);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Replace a container's contents, dropping the absent ones.
|
|
||||||
*
|
|
||||||
* `el` already ignores null children, but `replaceChildren` does not: it stringifies
|
|
||||||
* them, so an optional node that is not there renders the word "null" on the page.
|
|
||||||
* Every view builds optional nodes — a banner only while a job runs, a conflict only
|
|
||||||
* after a 409 — so the filter belongs here rather than in each caller. It was missing
|
|
||||||
* from one of them, and a documentation screenshot is where that turned up (US09-04).
|
|
||||||
*/
|
|
||||||
export function show(root, ...nodes) {
|
|
||||||
root.replaceChildren(...nodes.flat().filter((node) => node != null && node !== false));
|
|
||||||
}
|
|
||||||
|
|
||||||
export function setActiveNav(view) {
|
export function setActiveNav(view) {
|
||||||
document.querySelectorAll("nav a[data-nav]").forEach((a) => {
|
document.querySelectorAll("nav a[data-nav]").forEach((a) => {
|
||||||
if (a.dataset.nav === view) a.setAttribute("aria-current", "page");
|
if (a.dataset.nav === view) a.setAttribute("aria-current", "page");
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
// recovery classifications all come from the server; the view only shows them and
|
// recovery classifications all come from the server; the view only shows them and
|
||||||
// refuses to offer an action the server would reject.
|
// refuses to offer an action the server would reject.
|
||||||
import { api } from "./api.js";
|
import { api } from "./api.js";
|
||||||
import { el, errorBanner, setActiveNav, show } from "./dom.js";
|
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||||
|
|
||||||
// What the last confirm did, for this tab only. Deliberately not persisted: after a
|
// What the last confirm did, for this tab only. Deliberately not persisted: after a
|
||||||
// reload the page must show what the journal says, not what this page remembers.
|
// reload the page must show what the journal says, not what this page remembers.
|
||||||
@@ -24,7 +24,7 @@ export async function renderRenames(root, params = {}) {
|
|||||||
try {
|
try {
|
||||||
[plans, recovery] = await Promise.all([api.listPlans(), api.renameRecovery()]);
|
[plans, recovery] = await Promise.all([api.listPlans(), api.renameRecovery()]);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load renames: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load renames: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -34,13 +34,13 @@ export async function renderRenames(root, params = {}) {
|
|||||||
try {
|
try {
|
||||||
plan = await api.getPlan(selectedId);
|
plan = await api.getPlan(selectedId);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load plan: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load plan: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const blocked = recovery.blocks_mutation;
|
const blocked = recovery.blocks_mutation;
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
el("h1", {}, "Renames"),
|
el("h1", {}, "Renames"),
|
||||||
recoveryPanel(recovery),
|
recoveryPanel(recovery),
|
||||||
el(
|
el(
|
||||||
|
|||||||
@@ -7,7 +7,6 @@ import { createStore } from "../store.js";
|
|||||||
import { parseHash, navigate } from "../router.js";
|
import { parseHash, navigate } from "../router.js";
|
||||||
import { api, cancellable } from "../api.js";
|
import { api, cancellable } from "../api.js";
|
||||||
import { subscribeJob } from "../events.js";
|
import { subscribeJob } from "../events.js";
|
||||||
import { assignHeadingIds, documentIndex, resolveDocLink, rewriteLinks, slug } from "../docs.js";
|
|
||||||
|
|
||||||
const cases = [];
|
const cases = [];
|
||||||
function ok(name, cond) {
|
function ok(name, cond) {
|
||||||
@@ -143,62 +142,6 @@ async function run() {
|
|||||||
window.EventSource = realES;
|
window.EventSource = realES;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── documentation view (US09-01) ─────────────────────────────────────────
|
|
||||||
{
|
|
||||||
ok("slug lowercases and dashes a heading", slug("Key Flows") === "key-flows");
|
|
||||||
ok(
|
|
||||||
"slug drops punctuation but keeps words",
|
|
||||||
slug("What it *will not* do:") === "what-it-will-not-do"
|
|
||||||
);
|
|
||||||
ok("slug keeps non-ASCII letters", slug("Größe & Gewicht") === "größe-gewicht");
|
|
||||||
ok("slug never yields an empty anchor", slug("!!!") === "section");
|
|
||||||
|
|
||||||
const doc = document.createElement("div");
|
|
||||||
doc.innerHTML = "<h2>Notes</h2><h3>Notes</h3><h2>Notes</h2>";
|
|
||||||
const ids = [...assignHeadingIds(doc.querySelectorAll("h2, h3"))].map((h) => h.id);
|
|
||||||
ok("repeated headings get distinct anchors", ids.join(",") === "notes,notes-2,notes-3");
|
|
||||||
|
|
||||||
ok(
|
|
||||||
"a sibling document link becomes an in-app route",
|
|
||||||
resolveDocLink("index", "overview.md")?.page === "overview"
|
|
||||||
);
|
|
||||||
const nested = resolveDocLink("guides/install", "../overview.md#running-it");
|
|
||||||
ok("a relative link resolves against the current page", nested?.page === "overview");
|
|
||||||
ok("a link's anchor survives the rewrite", nested?.anchor === "running-it");
|
|
||||||
ok(
|
|
||||||
"a bare anchor stays on the current page",
|
|
||||||
resolveDocLink("overview", "#the-workflow")?.page === "overview"
|
|
||||||
);
|
|
||||||
ok("an external link is left alone", resolveDocLink("index", "https://example.test/x") === null);
|
|
||||||
ok("a non-markdown relative link is left alone", resolveDocLink("index", "images/a.png") === null);
|
|
||||||
// A climb cannot leave the documentation tree: it is clamped at the root, so the
|
|
||||||
// page it names is still fetched from under /docs and simply does not exist.
|
|
||||||
ok(
|
|
||||||
"a link that climbs above the docs root is clamped",
|
|
||||||
resolveDocLink("index", "../../etc/passwd.md")?.page === "etc/passwd"
|
|
||||||
);
|
|
||||||
ok(
|
|
||||||
"a page name that is not a page name is refused",
|
|
||||||
resolveDocLink("index", "..%2f..%2fetc%2fpasswd.md") === null
|
|
||||||
);
|
|
||||||
|
|
||||||
const index = document.createElement("div");
|
|
||||||
index.innerHTML =
|
|
||||||
'<a href="overview.md">Overview</a><a href="overview.md">again</a>' +
|
|
||||||
'<a href="index.md">itself</a><a href="https://example.test">out</a>';
|
|
||||||
const listed = documentIndex(rewriteLinks(index, "index"));
|
|
||||||
ok(
|
|
||||||
"an external link is marked safe to open away from the app",
|
|
||||||
index.querySelector('a[href^="https"]').rel === "noreferrer noopener"
|
|
||||||
);
|
|
||||||
ok(
|
|
||||||
"an in-app link points at the docs route",
|
|
||||||
index.querySelector("a").getAttribute("href") === "#/docs?page=overview"
|
|
||||||
);
|
|
||||||
ok("the index lists each page once, in order", listed.length === 1);
|
|
||||||
ok("the index takes its titles from the link text", listed[0].title === "Overview");
|
|
||||||
}
|
|
||||||
|
|
||||||
await tick();
|
await tick();
|
||||||
const failed = cases.filter((c) => !c.ok);
|
const failed = cases.filter((c) => !c.ok);
|
||||||
window.__RESULTS__ = { passed: cases.length - failed.length, failed: failed.length, cases };
|
window.__RESULTS__ = { passed: cases.length - failed.length, failed: failed.length, cases };
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
// preview arrives redacted — so nothing in this view may reconstruct, store, or
|
// preview arrives redacted — so nothing in this view may reconstruct, store, or
|
||||||
// route a secret.
|
// route a secret.
|
||||||
import { api } from "./api.js";
|
import { api } from "./api.js";
|
||||||
import { el, errorBanner, setActiveNav, show } from "./dom.js";
|
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||||
import { subscribeJob } from "./events.js";
|
import { subscribeJob } from "./events.js";
|
||||||
import { navigate } from "./router.js";
|
import { navigate } from "./router.js";
|
||||||
|
|
||||||
@@ -45,7 +45,7 @@ export async function renderUploads(root, params = {}) {
|
|||||||
api.listUploadBatches(),
|
api.listUploadBatches(),
|
||||||
]);
|
]);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load uploads: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load uploads: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -62,12 +62,12 @@ export async function renderUploads(root, params = {}) {
|
|||||||
batch = detail;
|
batch = detail;
|
||||||
history = verifications.verifications;
|
history = verifications.verifications;
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load upload batch: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load upload batch: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
...[
|
...[
|
||||||
el("h1", {}, "Upload"),
|
el("h1", {}, "Upload"),
|
||||||
configurationCard(preflight),
|
configurationCard(preflight),
|
||||||
|
|||||||
23
frontend/js/vendor/VERSIONS.json
vendored
@@ -1,23 +0,0 @@
|
|||||||
{
|
|
||||||
"_comment": "Vendored browser libraries (US09-01). Committed rather than fetched: the application is deployed to a network whose outbound access is not assumed, and `default-src 'self'` forbids a CDN. Checksums are asserted by tests/integration/test_documentation.py, so replacing a file without recording it here fails the suite. Update by downloading the pinned URL and recording the new version and sha256 in the same commit.",
|
|
||||||
"libraries": [
|
|
||||||
{
|
|
||||||
"file": "marked.esm.js",
|
|
||||||
"name": "marked",
|
|
||||||
"version": "18.0.10",
|
|
||||||
"license": "MIT",
|
|
||||||
"url": "https://cdn.jsdelivr.net/npm/marked@18.0.10/lib/marked.esm.js",
|
|
||||||
"sha256": "4cf47dfebb7f614a08fc0a579ab0fe407ff0ed2b717bf953040c85b2f493a4f0",
|
|
||||||
"why": "Markdown to HTML for the documentation view. An ES module with no dependencies, imported lazily by js/docs.js."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"file": "mermaid.min.js",
|
|
||||||
"name": "mermaid",
|
|
||||||
"version": "11.17.0",
|
|
||||||
"license": "MIT",
|
|
||||||
"url": "https://cdn.jsdelivr.net/npm/mermaid@11.17.0/dist/mermaid.min.js",
|
|
||||||
"sha256": "8d8e0eec56d3a83b4b3c87f42050845546dee93ebe1875d2117c12e6947c0cb3",
|
|
||||||
"why": "Renders ```mermaid blocks so diagrams are diffable source that Gitea also renders. The single-file UMD build rather than the ES module entry, whose ~40 lazy chunks would each need vendoring and pinning."
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
78
frontend/js/vendor/marked.esm.js
vendored
3636
frontend/js/vendor/mermaid.min.js
vendored
@@ -3,7 +3,7 @@
|
|||||||
// HTML. Mutating actions are disabled while a job holds the library_write lock and
|
// HTML. Mutating actions are disabled while a job holds the library_write lock and
|
||||||
// say why (concept §shared interaction rules: read-only browsing stays available).
|
// say why (concept §shared interaction rules: read-only browsing stays available).
|
||||||
import { api } from "./api.js";
|
import { api } from "./api.js";
|
||||||
import { el, errorBanner, setActiveNav, show } from "./dom.js";
|
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||||
import { navigate } from "./router.js";
|
import { navigate } from "./router.js";
|
||||||
import { subscribeJob } from "./events.js";
|
import { subscribeJob } from "./events.js";
|
||||||
|
|
||||||
@@ -26,7 +26,7 @@ export async function renderWorkflow(root) {
|
|||||||
try {
|
try {
|
||||||
data = await api.workflow();
|
data = await api.workflow();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load workflow: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load workflow: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
const active = data.active_job;
|
const active = data.active_job;
|
||||||
@@ -67,7 +67,7 @@ export async function renderWorkflow(root) {
|
|||||||
)
|
)
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
el("h1", {}, "Workflow"),
|
el("h1", {}, "Workflow"),
|
||||||
jobBanner(active),
|
jobBanner(active),
|
||||||
el("div", { class: "stepper" }, ...cards)
|
el("div", { class: "stepper" }, ...cards)
|
||||||
@@ -105,7 +105,7 @@ export async function renderSafety(root, params) {
|
|||||||
try {
|
try {
|
||||||
data = await api.safetyQueue({ state });
|
data = await api.safetyQueue({ state });
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load safety queue: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load safety queue: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,7 +149,7 @@ export async function renderSafety(root, params) {
|
|||||||
)
|
)
|
||||||
);
|
);
|
||||||
|
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
el("h1", {}, "Safety review"),
|
el("h1", {}, "Safety review"),
|
||||||
tabs,
|
tabs,
|
||||||
el(
|
el(
|
||||||
@@ -197,7 +197,7 @@ export async function renderLibrary(root, params) {
|
|||||||
try {
|
try {
|
||||||
data = await api.libraryAssets({ q, offset, limit, sort: params.sort || "path" });
|
data = await api.libraryAssets({ q, offset, limit, sort: params.sort || "path" });
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load library: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load library: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -220,7 +220,7 @@ export async function renderLibrary(root, params) {
|
|||||||
)
|
)
|
||||||
);
|
);
|
||||||
|
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
el("h1", {}, "Library"),
|
el("h1", {}, "Library"),
|
||||||
el("div", { class: "toolbar" }, search, el("span", { class: "muted", "data-testid": "library-total" }, `${data.total} photos`)),
|
el("div", { class: "toolbar" }, search, el("span", { class: "muted", "data-testid": "library-total" }, `${data.total} photos`)),
|
||||||
data.rows.length ? el("div", { class: "cluster-grid" }, ...cards) : el("p", { class: "muted" }, "No matching photos."),
|
data.rows.length ? el("div", { class: "cluster-grid" }, ...cards) : el("p", { class: "muted" }, "No matching photos."),
|
||||||
@@ -235,7 +235,7 @@ export async function renderAnalyze(root) {
|
|||||||
try {
|
try {
|
||||||
[counts, workflow] = await Promise.all([api.analysisCounts(), api.workflow()]);
|
[counts, workflow] = await Promise.all([api.analysisCounts(), api.workflow()]);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load analysis: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load analysis: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
const active = workflow.active_job;
|
const active = workflow.active_job;
|
||||||
@@ -269,7 +269,7 @@ export async function renderAnalyze(root) {
|
|||||||
"Analyze eligible SFW assets"
|
"Analyze eligible SFW assets"
|
||||||
);
|
);
|
||||||
|
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
el("h1", {}, "Analyze"),
|
el("h1", {}, "Analyze"),
|
||||||
jobBanner(active),
|
jobBanner(active),
|
||||||
el(
|
el(
|
||||||
@@ -292,10 +292,10 @@ export async function renderStats(root) {
|
|||||||
try {
|
try {
|
||||||
data = await api.libraryStats();
|
data = await api.libraryStats();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load stats: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load stats: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
el("h1", {}, "Stats"),
|
el("h1", {}, "Stats"),
|
||||||
el("div", { class: "counts-grid", "data-testid": "stats-status" }, ...Object.entries(data.status).map(([k, v]) => stat(k, v))),
|
el("div", { class: "counts-grid", "data-testid": "stats-status" }, ...Object.entries(data.status).map(([k, v]) => stat(k, v))),
|
||||||
facetBlock("Top tags", data.top_tags),
|
facetBlock("Top tags", data.top_tags),
|
||||||
@@ -338,7 +338,7 @@ export async function renderAlbums(root, params = {}) {
|
|||||||
api.renameRecovery(),
|
api.renameRecovery(),
|
||||||
]);
|
]);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
show(root, errorBanner(`Failed to load albums: ${error.message}`));
|
root.replaceChildren(errorBanner(`Failed to load albums: ${error.message}`));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -385,7 +385,7 @@ export async function renderAlbums(root, params = {}) {
|
|||||||
)
|
)
|
||||||
: el("p", { class: "muted" }, "No albums with evidence yet.");
|
: el("p", { class: "muted" }, "No albums with evidence yet.");
|
||||||
|
|
||||||
show(root,
|
root.replaceChildren(
|
||||||
el("h1", {}, "Albums"),
|
el("h1", {}, "Albums"),
|
||||||
renameBlocked
|
renameBlocked
|
||||||
? el(
|
? el(
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
"""Application management CLI:
|
"""Application management CLI:
|
||||||
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,docs-gate,dry-run,approve-dry-run}``.
|
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,dry-run,approve-dry-run}``.
|
||||||
|
|
||||||
``serve`` and ``worker`` take the library process lock for their role (US07-05):
|
``serve`` and ``worker`` take the library process lock for their role (US07-05):
|
||||||
two workers, or the frozen CLI running beside the app, would each be safe on their
|
two workers, or the frozen CLI running beside the app, would each be safe on their
|
||||||
@@ -78,12 +78,6 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
container_cmd.add_argument(
|
container_cmd.add_argument(
|
||||||
"--output", help="Evidence directory (default: data/container-gate/<stamp>)"
|
"--output", help="Evidence directory (default: data/container-gate/<stamp>)"
|
||||||
)
|
)
|
||||||
docs_cmd = commands.add_parser(
|
|
||||||
"docs-gate",
|
|
||||||
help="Check the manuals against the application they describe and keep the "
|
|
||||||
"evidence (US09-05)",
|
|
||||||
)
|
|
||||||
docs_cmd.add_argument("--output", help="Evidence directory (default: data/docs-gate/<stamp>)")
|
|
||||||
dry_cmd = commands.add_parser(
|
dry_cmd = commands.add_parser(
|
||||||
"dry-run", help="Read-only reconciliation of the configured library (US07-07)"
|
"dry-run", help="Read-only reconciliation of the configured library (US07-07)"
|
||||||
)
|
)
|
||||||
@@ -194,30 +188,6 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
)
|
)
|
||||||
return 0 if report["ok"] else 1
|
return 0 if report["ok"] else 1
|
||||||
|
|
||||||
if args.command == "docs-gate":
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
from photo_pipeline.services import release
|
|
||||||
|
|
||||||
# Documentation that disagrees with the application is worse than none: it is
|
|
||||||
# trusted. So this gate accepts no skip either — a check that did not run is a
|
|
||||||
# page nobody compared with the code.
|
|
||||||
output = args.output or Path(config.data_dir) / "docs-gate" / datetime.now(
|
|
||||||
timezone.utc
|
|
||||||
).strftime("%Y%m%dT%H%M%SZ")
|
|
||||||
report = release.run_gate(
|
|
||||||
config,
|
|
||||||
output=output,
|
|
||||||
stages=release.DOCS_STAGES,
|
|
||||||
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
|
||||||
)
|
|
||||||
print(
|
|
||||||
json.dumps(
|
|
||||||
{k: v for k, v in report.items() if k not in ("stages", "matrix")}, indent=2
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return 0 if report["ok"] else 1
|
|
||||||
|
|
||||||
if args.command == "dry-run":
|
if args.command == "dry-run":
|
||||||
from photo_pipeline.services import release
|
from photo_pipeline.services import release
|
||||||
|
|
||||||
|
|||||||
@@ -53,7 +53,6 @@ from photo_pipeline.services.thumbnails import ThumbnailService
|
|||||||
from photo_pipeline.services.upload_batches import UploadBatchService
|
from photo_pipeline.services.upload_batches import UploadBatchService
|
||||||
|
|
||||||
FRONTEND_DIR = Path(__file__).resolve().parents[2] / "frontend"
|
FRONTEND_DIR = Path(__file__).resolve().parents[2] / "frontend"
|
||||||
DOCS_DIR = Path(__file__).resolve().parents[2] / "docs"
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -152,10 +151,4 @@ def create_app(config: Config | None = None) -> FastAPI:
|
|||||||
# Static single-page app (hash-routed). Mounted last so /api/v1 wins.
|
# Static single-page app (hash-routed). Mounted last so /api/v1 wins.
|
||||||
if FRONTEND_DIR.is_dir():
|
if FRONTEND_DIR.is_dir():
|
||||||
app.mount("/app", StaticFiles(directory=FRONTEND_DIR, html=True), name="app")
|
app.mount("/app", StaticFiles(directory=FRONTEND_DIR, html=True), name="app")
|
||||||
# The manuals, as their own markdown files (US09-01). Unauthenticated for the
|
|
||||||
# same reason the application shell is: whoever cannot get past the access
|
|
||||||
# secret is exactly who needs the troubleshooting page, and no document holds
|
|
||||||
# anything a session would protect.
|
|
||||||
if DOCS_DIR.is_dir():
|
|
||||||
app.mount("/docs", StaticFiles(directory=DOCS_DIR), name="docs")
|
|
||||||
return app
|
return app
|
||||||
|
|||||||
@@ -57,21 +57,9 @@ LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "::1", "[::1]"})
|
|||||||
# backup a careful operator takes first, and its retention (US07-07).
|
# backup a careful operator takes first, and its retention (US07-07).
|
||||||
MUTATION_EXEMPT_PATHS = frozenset({f"{API_PREFIX}/backups", f"{API_PREFIX}/backups/prune"})
|
MUTATION_EXEMPT_PATHS = frozenset({f"{API_PREFIX}/backups", f"{API_PREFIX}/backups/prune"})
|
||||||
|
|
||||||
# Applied to every response. `frame-ancestors 'none'` and CORP keep other pages from
|
# Applied to every response. No inline script/style is used by the frontend, so the
|
||||||
|
# policy can stay strict; `frame-ancestors 'none'` and CORP keep other pages from
|
||||||
# embedding the app or its thumbnails.
|
# embedding the app or its thumbnails.
|
||||||
#
|
|
||||||
# `script-src 'self'` is the boundary that matters and it is unchanged: no
|
|
||||||
# `'unsafe-eval'`, no `'unsafe-inline'`, so nothing injected into the DOM can execute.
|
|
||||||
# Whether the vendored diagram renderer needed `'unsafe-eval'` was measured rather
|
|
||||||
# than assumed — mermaid's bundle contains no `eval(` and no `new Function`, and it
|
|
||||||
# renders under this exact policy without a single script-src violation.
|
|
||||||
#
|
|
||||||
# `style-src` does gain `'unsafe-inline'` (US09-01): mermaid styles the SVG it builds
|
|
||||||
# with an injected `<style>` element and `style=` attributes, and a diagram's CSS
|
|
||||||
# cannot be hashed in advance. The concession is bounded by the directives around it
|
|
||||||
# — with script execution still refused and `img-src`, `connect-src`, and `font-src`
|
|
||||||
# all `'self'`, the CSS exfiltration channels stay closed and what is left is
|
|
||||||
# defacement of a page its own operator is already looking at.
|
|
||||||
DEFAULT_HEADERS = {
|
DEFAULT_HEADERS = {
|
||||||
"x-content-type-options": "nosniff",
|
"x-content-type-options": "nosniff",
|
||||||
"x-frame-options": "DENY",
|
"x-frame-options": "DENY",
|
||||||
@@ -79,9 +67,8 @@ DEFAULT_HEADERS = {
|
|||||||
"cross-origin-resource-policy": "same-origin",
|
"cross-origin-resource-policy": "same-origin",
|
||||||
"cross-origin-opener-policy": "same-origin",
|
"cross-origin-opener-policy": "same-origin",
|
||||||
"content-security-policy": (
|
"content-security-policy": (
|
||||||
"default-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; "
|
"default-src 'self'; img-src 'self' data:; style-src 'self'; script-src 'self'; "
|
||||||
"script-src 'self'; connect-src 'self'; font-src 'self'; "
|
"connect-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
|
||||||
"frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
|
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -71,18 +71,6 @@ CONTAINER_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
|||||||
# would skip here — no daemon, no compose, no browser — means it was not proven.
|
# would skip here — no daemon, no compose, no browser — means it was not proven.
|
||||||
CONTAINER_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
|
CONTAINER_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
|
||||||
|
|
||||||
# The documentation gate (US09-05): the manuals, held to the application. Selected by
|
|
||||||
# marker across the whole suite, because each check lives beside what it describes —
|
|
||||||
# the offline contracts under tests/integration, the browser and screenshot journeys
|
|
||||||
# under tests/e2e. Adding a phase_i test is enough to put it in front of a release.
|
|
||||||
DOCS_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
|
||||||
("documentation", ("tests", "-m", "phase_i")),
|
|
||||||
)
|
|
||||||
|
|
||||||
# Also nothing. Every check here is either offline or needs the browser the rest of
|
|
||||||
# the suite already needs, so a skip means the manuals were not compared with the code.
|
|
||||||
DOCS_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
|
|
||||||
|
|
||||||
|
|
||||||
class ReleaseError(RuntimeError):
|
class ReleaseError(RuntimeError):
|
||||||
pass
|
pass
|
||||||
|
|||||||
@@ -54,5 +54,4 @@ markers = [
|
|||||||
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
|
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
|
||||||
"container: builds and runs the container image and its composition (US08-02, US08-03) — needs a Docker daemon, the compose plugin, and the network",
|
"container: builds and runs the container image and its composition (US08-02, US08-03) — needs a Docker daemon, the compose plugin, and the network",
|
||||||
"phase_h: Phase H container deployment acceptance (US08-05) — the browser, upgrade, restart, and security journeys against the composed stack",
|
"phase_h: Phase H container deployment acceptance (US08-05) — the browser, upgrade, restart, and security journeys against the composed stack",
|
||||||
"phase_i: Phase I documentation acceptance (US09-05) — the manuals checked against the application, in the browser and offline",
|
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -1,152 +0,0 @@
|
|||||||
"""US09-01: the manuals, in the browser, under the application's real policy.
|
|
||||||
|
|
||||||
Everything here runs against a real ``photo_pipeline serve`` process serving the
|
|
||||||
committed ``docs/`` tree and the vendored renderer — no fixture markdown, no stubbed
|
|
||||||
fetch. What that buys is the assertion the story actually cares about: the pages
|
|
||||||
render, the links between them work, a diagram becomes a diagram, and the console
|
|
||||||
stays empty, including of CSP violations.
|
|
||||||
|
|
||||||
The offline contract — reachability, dead links, pinned checksums — is
|
|
||||||
``tests/integration/test_documentation.py``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from tests.e2e._pipeline_harness import Server, seed_library
|
|
||||||
|
|
||||||
# Every check here belongs to the documentation gate (US09-05).
|
|
||||||
pytestmark = pytest.mark.phase_i
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
|
||||||
def server(tmp_path_factory):
|
|
||||||
seeded = seed_library(tmp_path_factory.mktemp("docs"), {"a": 1}, {})
|
|
||||||
running = Server(seeded).start()
|
|
||||||
try:
|
|
||||||
yield running
|
|
||||||
finally:
|
|
||||||
running.stop()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def quiet(page):
|
|
||||||
"""Any console error, page error, or failed request fails the test that caused
|
|
||||||
it. A CSP violation arrives as a console error, which is the point."""
|
|
||||||
problems: list[str] = []
|
|
||||||
page.on("console", lambda m: problems.append(m.text) if m.type == "error" else None)
|
|
||||||
page.on("pageerror", lambda error: problems.append(str(error)))
|
|
||||||
page.on("requestfailed", lambda request: problems.append(f"failed request: {request.url}"))
|
|
||||||
yield problems
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_documentation_opens_from_the_navigation(page, server, quiet):
|
|
||||||
page.goto(f"{server.base}/app/#/workflow")
|
|
||||||
page.locator('nav a[data-nav="docs"]').click()
|
|
||||||
|
|
||||||
page.get_by_test_id("doc").wait_for()
|
|
||||||
assert page.get_by_test_id("doc").get_attribute("data-page") == "index"
|
|
||||||
# The sidebar is the index's own reading order, not a second list to maintain.
|
|
||||||
assert page.get_by_test_id("doc-pages").get_by_role("link", name="Overview").is_visible()
|
|
||||||
assert quiet == []
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_link_between_documents_stays_inside_the_application(page, server, quiet):
|
|
||||||
page.goto(f"{server.base}/app/#/docs")
|
|
||||||
page.get_by_test_id("doc").wait_for()
|
|
||||||
page.get_by_test_id("doc").get_by_role("link", name="Overview").click()
|
|
||||||
|
|
||||||
page.wait_for_selector('[data-testid="doc"][data-page="overview"]')
|
|
||||||
assert "#/docs?page=overview" in page.url, "a relative .md link must not leave the app"
|
|
||||||
# And back again, by the link the document itself carries.
|
|
||||||
page.get_by_test_id("doc").get_by_role("link", name="← Documentation index").click()
|
|
||||||
page.wait_for_selector('[data-testid="doc"][data-page="index"]')
|
|
||||||
assert quiet == []
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_deep_link_to_a_heading_lands_on_that_heading(page, server, quiet):
|
|
||||||
page.goto(f"{server.base}/app/#/docs?page=overview&anchor=the-workflow")
|
|
||||||
heading = page.locator("#the-workflow")
|
|
||||||
heading.wait_for()
|
|
||||||
|
|
||||||
assert heading.inner_text().strip() == "The workflow"
|
|
||||||
assert heading.evaluate("node => node.getBoundingClientRect().top < window.innerHeight")
|
|
||||||
assert quiet == []
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_mermaid_block_becomes_a_diagram_under_the_unchanged_script_policy(page, server, quiet):
|
|
||||||
page.goto(f"{server.base}/app/#/docs?page=overview")
|
|
||||||
diagram = page.get_by_test_id("diagram").first
|
|
||||||
diagram.wait_for()
|
|
||||||
|
|
||||||
# A real drawing, not the source text and not an error node.
|
|
||||||
assert diagram.locator("svg").count() == 1
|
|
||||||
assert diagram.evaluate("node => node.querySelector('svg').getBBox().width") > 100
|
|
||||||
assert page.locator("pre code.language-mermaid").count() == 0
|
|
||||||
assert quiet == [], "rendering a diagram must not violate the policy"
|
|
||||||
|
|
||||||
policy = page.evaluate(
|
|
||||||
"async () => (await fetch('/app/')).headers.get('content-security-policy')"
|
|
||||||
)
|
|
||||||
assert "script-src 'self';" in policy
|
|
||||||
assert "unsafe-eval" not in policy
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_diagram_in_the_architecture_overview_draws(page, server, quiet):
|
|
||||||
"""Five diagrams, three of them state machines (US09-03). A mermaid block with a
|
|
||||||
syntax error renders an error node instead of throwing, so a page that merely
|
|
||||||
loaded proves nothing — count the drawings and read the source for the states."""
|
|
||||||
source = (Path(__file__).resolve().parents[2] / "docs" / "architecture.md").read_text()
|
|
||||||
expected = source.count("```mermaid")
|
|
||||||
assert expected >= 5, "the overview lost its diagrams"
|
|
||||||
|
|
||||||
page.goto(f"{server.base}/app/#/docs?page=architecture")
|
|
||||||
page.get_by_test_id("diagram").first.wait_for()
|
|
||||||
page.wait_for_function(
|
|
||||||
"count => document.querySelectorAll('[data-testid=\"diagram\"] svg').length === count",
|
|
||||||
arg=expected,
|
|
||||||
)
|
|
||||||
|
|
||||||
drawn = page.locator('[data-testid="diagram"] svg')
|
|
||||||
assert drawn.count() == expected
|
|
||||||
for index in range(expected):
|
|
||||||
diagram = drawn.nth(index)
|
|
||||||
# Labels may be <text> or HTML inside a <foreignObject> depending on the
|
|
||||||
# diagram type, so ask for the rendered text rather than a specific element.
|
|
||||||
labels = diagram.evaluate("node => node.textContent.trim().length")
|
|
||||||
assert labels > 0, f"diagram {index} drew no labels"
|
|
||||||
assert diagram.locator(".error-icon, .error-text").count() == 0, f"diagram {index} errored"
|
|
||||||
assert page.locator("pre code.language-mermaid").count() == 0
|
|
||||||
assert quiet == []
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_unknown_page_says_so_without_naming_a_path(page, server, quiet):
|
|
||||||
page.goto(f"{server.base}/app/#/docs?page=no-such-manual")
|
|
||||||
message = page.get_by_test_id("doc-not-found")
|
|
||||||
message.wait_for()
|
|
||||||
|
|
||||||
text = message.inner_text()
|
|
||||||
assert "does not exist" in text
|
|
||||||
assert "/" not in text.replace("Back to the documentation index", ""), text
|
|
||||||
message.get_by_role("link").click()
|
|
||||||
page.wait_for_selector('[data-testid="doc"][data-page="index"]')
|
|
||||||
# The missing page is a 404 and the browser says so; nothing else may go wrong,
|
|
||||||
# and in particular the view must not throw on the way to its own message.
|
|
||||||
assert all("404" in problem for problem in quiet), quiet
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_documentation_is_readable_without_a_session(page, server, quiet):
|
|
||||||
"""The troubleshooting page is needed most by whoever is locked out."""
|
|
||||||
page.goto(f"{server.base}/app/#/docs")
|
|
||||||
page.get_by_test_id("doc").wait_for()
|
|
||||||
unauthenticated = page.evaluate(
|
|
||||||
"""async () => {
|
|
||||||
const response = await fetch('/docs/index.md', { credentials: 'omit' });
|
|
||||||
return { status: response.status, length: (await response.text()).length };
|
|
||||||
}"""
|
|
||||||
)
|
|
||||||
assert unauthenticated["status"] == 200
|
|
||||||
assert unauthenticated["length"] > 100
|
|
||||||
@@ -1,215 +0,0 @@
|
|||||||
"""US09-04: the user manual's screenshots, produced from the running application.
|
|
||||||
|
|
||||||
A screenshot nobody can regenerate is a screenshot that silently stops being true.
|
|
||||||
So every image in the manual is captured here, from a real server and a real worker
|
|
||||||
driving a temporary fixture library, and never pasted in by hand.
|
|
||||||
|
|
||||||
Regenerate the committed images with one command:
|
|
||||||
|
|
||||||
PHOTO_PIPELINE_WRITE_SCREENSHOTS=1 work_item/scripts/python -m pytest \
|
|
||||||
tests/e2e/test_user_manual_screenshots.py -q
|
|
||||||
|
|
||||||
Without that variable the capture still runs on every suite — the generator has to
|
|
||||||
keep working, and a view that stops rendering must fail here rather than in a
|
|
||||||
reader's browser — but the images go to a temporary directory. Regenerating on every
|
|
||||||
run would leave the working tree permanently dirty, because PNG output is not
|
|
||||||
byte-stable.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
from contextlib import closing
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from PIL import Image
|
|
||||||
from playwright.sync_api import TimeoutError as PlaywrightTimeout
|
|
||||||
|
|
||||||
from tests.conftest import session_client
|
|
||||||
from tests.e2e._pipeline_harness import (
|
|
||||||
Server,
|
|
||||||
approve_album,
|
|
||||||
seed_album,
|
|
||||||
start_worker,
|
|
||||||
wait_until,
|
|
||||||
)
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
|
||||||
DOCS = REPO / "docs"
|
|
||||||
IMAGES = DOCS / "images"
|
|
||||||
ALBUM = "rome"
|
|
||||||
VIEWPORT = {"width": 1280, "height": 900}
|
|
||||||
|
|
||||||
# One per stage page of the manual: the route, the heading that view renders, and the
|
|
||||||
# element whose presence means it has finished.
|
|
||||||
#
|
|
||||||
# The heading matters more than it looks. Every route replaces the same container, so
|
|
||||||
# waiting for "an h1" matches the *previous* view's heading and photographs the screen
|
|
||||||
# you just left — which is how the statistics page first shipped a picture of the
|
|
||||||
# archive view.
|
|
||||||
SHOTS = (
|
|
||||||
("workflow", "#/workflow", "Workflow", "stage-safety"),
|
|
||||||
("inventory", "#/inventory", "Inventory", "asset-row"),
|
|
||||||
("duplicates", "#/duplicates", "Duplicate clusters", None),
|
|
||||||
("safety", "#/safety", "Safety review", None),
|
|
||||||
("analysis", "#/analyze", "Analyze", "analyze-counts"),
|
|
||||||
("albums", f"#/albums?album={ALBUM}", "Albums", "suggested-name"),
|
|
||||||
("renames", "#/renames", "Renames", None),
|
|
||||||
("uploads", "#/uploads", "Upload", "upload-scope"),
|
|
||||||
("archive", "#/archive", "Archive", "archive-locations"),
|
|
||||||
("statistics", "#/stats", "Stats", None),
|
|
||||||
)
|
|
||||||
|
|
||||||
# Every check here belongs to the documentation gate (US09-05).
|
|
||||||
pytestmark = pytest.mark.phase_i
|
|
||||||
|
|
||||||
|
|
||||||
def writing() -> bool:
|
|
||||||
return os.environ.get("PHOTO_PIPELINE_WRITE_SCREENSHOTS") == "1"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
|
||||||
def stack(tmp_path_factory):
|
|
||||||
"""A library in the state the manual describes: one analysed album, a duplicate
|
|
||||||
to review, an archive destination, and a worker that claims jobs."""
|
|
||||||
tmp_path = tmp_path_factory.mktemp("manual")
|
|
||||||
seeded = seed_album(tmp_path, ALBUM, ("forum.jpg", "colosseum.jpg"))
|
|
||||||
worker = start_worker(seeded, fake_vision_log=tmp_path / "vision.log")
|
|
||||||
server = Server(seeded).start()
|
|
||||||
try:
|
|
||||||
_prepare(server.base, seeded)
|
|
||||||
yield server
|
|
||||||
finally:
|
|
||||||
server.stop()
|
|
||||||
worker.terminate()
|
|
||||||
worker.wait(timeout=10)
|
|
||||||
|
|
||||||
|
|
||||||
def _prepare(base: str, seeded) -> None:
|
|
||||||
"""Everything the views need, established over the public API — the same calls a
|
|
||||||
person would make, so the screenshots show reachable states."""
|
|
||||||
with closing(session_client(base, timeout=30)) as client:
|
|
||||||
client.post("/api/v1/duplicates/detect").raise_for_status()
|
|
||||||
client.post("/api/v1/albums/proposals", json={}).raise_for_status()
|
|
||||||
destination = seeded.data / "archive"
|
|
||||||
destination.mkdir(exist_ok=True)
|
|
||||||
client.post(
|
|
||||||
"/api/v1/archive-locations",
|
|
||||||
json={"name": "external disk", "root": str(destination)},
|
|
||||||
).raise_for_status()
|
|
||||||
wait_until(lambda: client.get("/api/v1/workflow").status_code == 200)
|
|
||||||
|
|
||||||
# An approved name and a built — deliberately unapplied — plan, so the renames
|
|
||||||
# screenshot shows the preview its page describes rather than an empty state.
|
|
||||||
# Building a plan moves nothing; applying it is what would, and nothing here does.
|
|
||||||
approve_album(base, album=ALBUM, name="2019 — Rome")
|
|
||||||
with closing(session_client(base, timeout=30)) as client:
|
|
||||||
client.post("/api/v1/rename-plans").raise_for_status()
|
|
||||||
|
|
||||||
|
|
||||||
def _capture(page, server, target: Path) -> list[str]:
|
|
||||||
target.mkdir(parents=True, exist_ok=True)
|
|
||||||
page.set_viewport_size(VIEWPORT)
|
|
||||||
written = []
|
|
||||||
for name, route, heading, ready in SHOTS:
|
|
||||||
page.goto(f"{server.base}/app/{route}")
|
|
||||||
page.locator("main h1", has_text=heading).first.wait_for(timeout=30_000)
|
|
||||||
if ready:
|
|
||||||
page.get_by_test_id(ready).first.wait_for(timeout=30_000)
|
|
||||||
page.screenshot(path=str(target / f"{name}.png"))
|
|
||||||
written.append(name)
|
|
||||||
return written
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_generator_produces_every_screenshot_the_manual_references(page, stack, tmp_path):
|
|
||||||
destination = IMAGES if writing() else tmp_path / "images"
|
|
||||||
written = _capture(page, stack, destination)
|
|
||||||
|
|
||||||
assert sorted(written) == sorted(name for name, *_ in SHOTS)
|
|
||||||
for name in written:
|
|
||||||
produced = destination / f"{name}.png"
|
|
||||||
assert produced.stat().st_size > 5_000, f"{name}.png is too small to be a view"
|
|
||||||
|
|
||||||
referenced = set(re.findall(r"images/([a-z-]+)\.png", "\n".join(
|
|
||||||
path.read_text() for path in DOCS.rglob("*.md")
|
|
||||||
)))
|
|
||||||
assert referenced <= set(written), f"the manual references images nobody generates: {referenced - set(written)}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_no_screenshot_shows_a_real_path_or_a_secret(page, stack, tmp_path):
|
|
||||||
"""The fixture library is synthetic and lives in a temporary directory, so the
|
|
||||||
pixels cannot carry someone's photographs — but the view can still print a path,
|
|
||||||
and that path must be the fixture's, never the operator's."""
|
|
||||||
with closing(session_client(stack.base, timeout=30)) as client:
|
|
||||||
rendered = client.get("/api/v1/inventory/assets", params={"limit": 200}).json()["items"]
|
|
||||||
paths = [item["current_path"] for item in rendered if item["current_path"]]
|
|
||||||
assert paths, "nothing was rendered, so nothing was proven"
|
|
||||||
assert all("/manual" in path or "pytest" in path or "/tmp" in path or "/var" in path for path in paths), paths
|
|
||||||
assert all(ALBUM in path or "images" in path for path in paths)
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_committed_screenshots_still_show_what_the_application_shows(page, stack, tmp_path):
|
|
||||||
"""A UI change that invalidates the manual should be a red build, not a discovery
|
|
||||||
months later by somebody following a picture of a screen that no longer exists.
|
|
||||||
|
|
||||||
The tolerance is on *content*, not pixels. Comparing the committed PNGs with a
|
|
||||||
fresh capture was tried first and rejected: PNG output is not reproducible, font
|
|
||||||
rasterisation differs between the machine that generated an image and the machine
|
|
||||||
running the gate, and several views legitimately print the fixture library's
|
|
||||||
absolute path, which is a fresh temporary directory every run. A pixel or
|
|
||||||
perceptual comparison therefore fails for reasons that have nothing to do with
|
|
||||||
the manual being wrong — and a check that cries wolf gets deleted.
|
|
||||||
|
|
||||||
What actually invalidates a screenshot is the view no longer showing what its
|
|
||||||
page says it shows. That is what is compared here: the heading, the elements the
|
|
||||||
page describes, and the fact that the image was captured at the same size.
|
|
||||||
|
|
||||||
ponytail: content comparison, not pixels. A visual-diff service with per-platform
|
|
||||||
baselines is the upgrade if cosmetic regressions ever need catching too.
|
|
||||||
"""
|
|
||||||
if not IMAGES.is_dir():
|
|
||||||
pytest.skip("no screenshots have been generated into the repository yet")
|
|
||||||
|
|
||||||
fresh = tmp_path / "fresh"
|
|
||||||
_capture(page, stack, fresh)
|
|
||||||
|
|
||||||
drifted = []
|
|
||||||
for name, route, heading, ready in SHOTS:
|
|
||||||
committed = IMAGES / f"{name}.png"
|
|
||||||
if not committed.is_file():
|
|
||||||
drifted.append(f"{name}: never committed")
|
|
||||||
continue
|
|
||||||
with Image.open(committed) as image:
|
|
||||||
if image.size != (VIEWPORT["width"], VIEWPORT["height"]):
|
|
||||||
drifted.append(f"{name}: committed at {image.size}, captured at {VIEWPORT}")
|
|
||||||
# The views render asynchronously, so each check waits rather than asking a
|
|
||||||
# question the page has not finished answering.
|
|
||||||
page.goto(f"{stack.base}/app/{route}")
|
|
||||||
try:
|
|
||||||
page.locator("main h1", has_text=heading).first.wait_for(timeout=15_000)
|
|
||||||
except PlaywrightTimeout:
|
|
||||||
drifted.append(f"{name}: the view no longer shows the heading '{heading}'")
|
|
||||||
continue
|
|
||||||
if ready:
|
|
||||||
try:
|
|
||||||
page.get_by_test_id(ready).first.wait_for(timeout=15_000)
|
|
||||||
except PlaywrightTimeout:
|
|
||||||
drifted.append(f"{name}: the view no longer renders '{ready}'")
|
|
||||||
|
|
||||||
assert drifted == [], (
|
|
||||||
f"the manual's screenshots no longer match the application: {drifted}. "
|
|
||||||
"Regenerate them with PHOTO_PIPELINE_WRITE_SCREENSHOTS=1 and review the result."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_committed_screenshot_is_referenced_by_a_page():
|
|
||||||
"""An image nobody shows is an image nobody updates."""
|
|
||||||
if not IMAGES.is_dir():
|
|
||||||
pytest.skip("no screenshots have been generated into the repository yet")
|
|
||||||
text = "\n".join(path.read_text() for path in DOCS.rglob("*.md"))
|
|
||||||
orphans = sorted(
|
|
||||||
image.name for image in IMAGES.glob("*.png") if f"images/{image.name}" not in text
|
|
||||||
)
|
|
||||||
assert orphans == [], f"committed but unreferenced: {orphans}"
|
|
||||||
@@ -1,169 +0,0 @@
|
|||||||
"""US09-03: the architecture overview, checked against the architecture.
|
|
||||||
|
|
||||||
An architecture document is the one that rots most quietly: nothing breaks when it
|
|
||||||
describes a module that was renamed two epics ago, it just quietly misleads the next
|
|
||||||
person. So the parts of it that the code also knows — module names, state machines,
|
|
||||||
table names, the modules an invariant is claimed to live in — are compared with the
|
|
||||||
code, and a missing one fails the suite.
|
|
||||||
|
|
||||||
The diagrams' rendering is proven in ``tests/e2e/test_docs_ui.py``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
import photo_pipeline.models # noqa: F401 (registers every table on Base.metadata)
|
|
||||||
from photo_pipeline.db import Base
|
|
||||||
from photo_pipeline.services import jobs, rename_journal, upload_batches
|
|
||||||
|
|
||||||
# Every check here belongs to the documentation gate (US09-05).
|
|
||||||
pytestmark = pytest.mark.phase_i
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
|
||||||
PACKAGE = REPO / "photo_pipeline"
|
|
||||||
OVERVIEW = REPO / "docs" / "architecture.md"
|
|
||||||
TEXT = OVERVIEW.read_text()
|
|
||||||
|
|
||||||
# Names the map does not owe the reader individually: the package markers, the CLI
|
|
||||||
# entry point, and the two modules that exist to be small and obvious.
|
|
||||||
UNMAPPED = {"__init__", "__main__", "logging"}
|
|
||||||
|
|
||||||
|
|
||||||
def modules() -> set[str]:
|
|
||||||
"""Every module and package under photo_pipeline, by the name it is imported as."""
|
|
||||||
names = set()
|
|
||||||
for path in PACKAGE.rglob("*.py"):
|
|
||||||
relative = path.relative_to(PACKAGE)
|
|
||||||
names.add(relative.parts[0] if len(relative.parts) > 1 else relative.stem)
|
|
||||||
return {name.removesuffix(".py") for name in names} - UNMAPPED
|
|
||||||
|
|
||||||
|
|
||||||
def states(container: type) -> set[str]:
|
|
||||||
return {
|
|
||||||
value
|
|
||||||
for name, value in vars(container).items()
|
|
||||||
if name.isupper() and isinstance(value, str)
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# ── the module map ───────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_module_appears_in_the_map():
|
|
||||||
"""A service nobody documented is a service the next person re-implements."""
|
|
||||||
missing = sorted(name for name in modules() if name not in TEXT)
|
|
||||||
assert missing == [], f"absent from the architecture overview: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_service_is_named_individually():
|
|
||||||
"""The package table says what `services/` is for; this is the list that actually
|
|
||||||
drifts, because a new service is added roughly every story."""
|
|
||||||
services = {path.stem for path in (PACKAGE / "services").glob("*.py")} - UNMAPPED
|
|
||||||
missing = sorted(name for name in services if not re.search(rf"\b{name}\b", TEXT))
|
|
||||||
assert missing == [], f"services missing from the overview: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_map_names_no_module_that_does_not_exist():
|
|
||||||
"""Every `services/<name>` or backticked module path in the document resolves."""
|
|
||||||
referenced = set(re.findall(r"`(?:photo_pipeline/)?([a-z_]+)/([a-z_]+)\.py`", TEXT))
|
|
||||||
referenced |= {("", name) for name in re.findall(r"`([a-z_]+)\.py`", TEXT)}
|
|
||||||
missing = [
|
|
||||||
f"{package}/{module}.py" if package else f"{module}.py"
|
|
||||||
for package, module in referenced
|
|
||||||
if not (PACKAGE / package / f"{module}.py").is_file()
|
|
||||||
]
|
|
||||||
assert missing == [], f"documented but absent from the source: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the state machines ───────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_job_state_is_documented():
|
|
||||||
missing = sorted(state for state in states(jobs.JobState) if state not in TEXT)
|
|
||||||
assert missing == [], f"job states missing from the overview: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_rename_journal_state_is_documented():
|
|
||||||
missing = sorted(state for state in states(rename_journal.JournalState) if state not in TEXT)
|
|
||||||
assert missing == [], f"journal states missing from the overview: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_upload_batch_state_is_documented():
|
|
||||||
missing = sorted(state for state in states(upload_batches.BatchState) if state not in TEXT)
|
|
||||||
assert missing == [], f"batch states missing from the overview: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_unsafe_journal_states_are_named_as_the_ones_that_block():
|
|
||||||
"""The reason an unrelated mutation is refused has to be findable."""
|
|
||||||
for state in rename_journal.UNSAFE_STATES:
|
|
||||||
assert state in TEXT
|
|
||||||
assert "rename_recovery_required" in TEXT
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_uncertain_upload_is_documented_as_not_retryable():
|
|
||||||
assert upload_batches.BatchState.UNKNOWN not in upload_batches.RUNNABLE_STATES
|
|
||||||
assert "not** restartable" in TEXT or "not restartable" in TEXT
|
|
||||||
|
|
||||||
|
|
||||||
# ── the data model ───────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_table_is_listed():
|
|
||||||
missing = sorted(name for name in Base.metadata.tables if name not in TEXT)
|
|
||||||
assert missing == [], f"tables missing from the overview: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_document_lists_no_table_that_does_not_exist():
|
|
||||||
listed = set(re.findall(r"`([a-z_]+)`", TEXT))
|
|
||||||
plausible = {name for name in listed if name.endswith("s") and "_" in name}
|
|
||||||
invented = sorted(
|
|
||||||
name
|
|
||||||
for name in plausible
|
|
||||||
if name not in Base.metadata.tables
|
|
||||||
and not (PACKAGE / "services" / f"{name}.py").is_file()
|
|
||||||
and name not in {"library_roots", "trusted_proxies", "allowed_hosts", "asset_paths"}
|
|
||||||
)
|
|
||||||
assert invented == [], f"looks like a table but is not one: {invented}"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the invariants ───────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_each_invariant_points_at_a_module_that_enforces_it():
|
|
||||||
"""The point of the table is to answer 'where do I look'. A wrong answer there
|
|
||||||
costs more than no answer."""
|
|
||||||
claims = {
|
|
||||||
"path_policy.is_excluded": PACKAGE / "path_policy.py",
|
|
||||||
"path_policy.resolve_in_roots": PACKAGE / "path_policy.py",
|
|
||||||
"services/app_lock.py": PACKAGE / "services" / "app_lock.py",
|
|
||||||
"services/exif_checkpoint.py": PACKAGE / "services" / "exif_checkpoint.py",
|
|
||||||
"services/archive_transfer.py": PACKAGE / "services" / "archive_transfer.py",
|
|
||||||
"api/security.py": PACKAGE / "api" / "security.py",
|
|
||||||
"imaging.py": PACKAGE / "imaging.py",
|
|
||||||
}
|
|
||||||
for claim, path in claims.items():
|
|
||||||
assert claim in TEXT, f"the invariant table does not mention {claim}"
|
|
||||||
assert path.is_file(), f"{claim} does not exist"
|
|
||||||
function = claim.rpartition(".")[2] if "/" not in claim else ""
|
|
||||||
if function and function != "py":
|
|
||||||
assert f"def {function}" in path.read_text(), f"{claim} is not defined there"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_lock_order_matches_the_ranks_in_the_code():
|
|
||||||
from photo_pipeline.jobs import locks
|
|
||||||
|
|
||||||
order = [name for name, _ in sorted(locks.LOCK_RANK.items(), key=lambda item: item[1])]
|
|
||||||
assert order[0].startswith("library"), "the broadest lock is no longer the library"
|
|
||||||
assert "library → stage/job → album/folder → asset" in TEXT
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_frozen_donor_archive_is_described_as_provenance_not_a_dependency():
|
|
||||||
"""That production never imports the frozen sources is proven by
|
|
||||||
``tests/unit/test_legacy_archive.py`` — which also forbids any other test from
|
|
||||||
naming that directory, so this one checks the claim by its consequence."""
|
|
||||||
assert "must not import" in TEXT
|
|
||||||
assert "provenance and rollback evidence" in TEXT
|
|
||||||
@@ -1,196 +0,0 @@
|
|||||||
"""US09-05: the documentation gate's own contract, checked without a browser.
|
|
||||||
|
|
||||||
The gate itself needs a browser and a server and takes minutes; what is checkable
|
|
||||||
offline is what makes it a *gate* rather than a long test run:
|
|
||||||
|
|
||||||
* it selects every documentation check by marker, so adding one is enough to put it
|
|
||||||
in front of a release;
|
|
||||||
* it accepts no skip at all — a check that did not run is a page nobody compared
|
|
||||||
with the code;
|
|
||||||
* it retains checksummed evidence per run;
|
|
||||||
* CI runs it on pull requests, which is when a fix is still cheap.
|
|
||||||
|
|
||||||
The checks it runs are the four suites written by US09-01 through US09-04.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
import yaml
|
|
||||||
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
from photo_pipeline.services import release
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
|
||||||
DOCS = REPO / "docs"
|
|
||||||
TEST_WORKFLOW = yaml.safe_load((REPO / ".gitea" / "workflows" / "test.yml").read_text())
|
|
||||||
|
|
||||||
pytestmark = pytest.mark.phase_i
|
|
||||||
|
|
||||||
# The suites the gate exists to run. Each is mapped to a story in the traceability
|
|
||||||
# matrix, so this list is also what stops one being quietly dropped.
|
|
||||||
DOCUMENTATION_SUITES = (
|
|
||||||
"tests/integration/test_documentation.py",
|
|
||||||
"tests/integration/test_installation_manual.py",
|
|
||||||
"tests/integration/test_architecture_overview.py",
|
|
||||||
"tests/integration/test_user_manual.py",
|
|
||||||
"tests/integration/test_docs_gate.py",
|
|
||||||
"tests/e2e/test_docs_ui.py",
|
|
||||||
"tests/e2e/test_user_manual_screenshots.py",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _config(tmp_path) -> Config:
|
|
||||||
(tmp_path / "data").mkdir(exist_ok=True)
|
|
||||||
return Config.from_env({"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data")})
|
|
||||||
|
|
||||||
|
|
||||||
# ── what the gate runs ───────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_gate_selects_every_documentation_check_by_marker():
|
|
||||||
assert release.DOCS_STAGES == (("documentation", ("tests", "-m", "phase_i")),)
|
|
||||||
declared = [
|
|
||||||
line
|
|
||||||
for line in (REPO / "pyproject.toml").read_text().splitlines()
|
|
||||||
if line.strip().startswith('"phase_i:')
|
|
||||||
]
|
|
||||||
assert declared, "an unregistered marker selects nothing and fails no gate"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_documentation_suite_carries_the_marker():
|
|
||||||
"""A documentation test without the marker is a check the gate never runs."""
|
|
||||||
unmarked = [
|
|
||||||
suite
|
|
||||||
for suite in DOCUMENTATION_SUITES
|
|
||||||
if "pytest.mark.phase_i" not in (REPO / suite).read_text()
|
|
||||||
]
|
|
||||||
assert unmarked == [], f"suites missing the phase_i marker: {unmarked}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_marker_selects_nothing_outside_the_documentation_suites():
|
|
||||||
"""Scope matters: the gate's promise is that it ran *the documentation checks*."""
|
|
||||||
marked = sorted(
|
|
||||||
str(path.relative_to(REPO))
|
|
||||||
for path in (REPO / "tests").rglob("test_*.py")
|
|
||||||
if "pytest.mark.phase_i" in path.read_text()
|
|
||||||
)
|
|
||||||
assert marked == sorted(DOCUMENTATION_SUITES)
|
|
||||||
|
|
||||||
|
|
||||||
# ── no skip is an environment limit here ─────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_gate_accepts_no_skipped_check_at_all():
|
|
||||||
assert release.DOCS_ALLOWED_SKIP_REASONS == ()
|
|
||||||
skipped = release.StageResult(
|
|
||||||
"documentation", [], 0, 0.1, "1 skipped", ["SKIPPED [1] x.py:1: exiftool not installed"]
|
|
||||||
)
|
|
||||||
# The release gate tolerates this one because it describes the machine. The
|
|
||||||
# documentation gate cannot: nothing here depends on the machine.
|
|
||||||
assert release.unexpected_skips([skipped]) == []
|
|
||||||
assert release.unexpected_skips([skipped], release.DOCS_ALLOWED_SKIP_REASONS) == [
|
|
||||||
"SKIPPED [1] x.py:1: exiftool not installed"
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_skipped_check_fails_the_run_and_the_evidence_says_so(tmp_path):
|
|
||||||
skipping = tmp_path / "test_skipping.py"
|
|
||||||
skipping.write_text(
|
|
||||||
"import pytest\n\ndef test_x():\n pytest.skip('exiftool not installed')\n"
|
|
||||||
)
|
|
||||||
evidence = tmp_path / "evidence"
|
|
||||||
|
|
||||||
report = release.run_gate(
|
|
||||||
_config(tmp_path),
|
|
||||||
output=evidence,
|
|
||||||
stages=(("documentation", (str(skipping),)),),
|
|
||||||
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert report["ok"] is False
|
|
||||||
assert report["failures"] == [], "the stage passed; the skip is what fails the gate"
|
|
||||||
assert report["unexpected_skips"], report
|
|
||||||
written = json.loads((evidence / release.REPORT_NAME).read_text())
|
|
||||||
assert written["ok"] is False
|
|
||||||
assert (evidence / "logs" / "documentation.log").exists()
|
|
||||||
for line in (evidence / release.CHECKSUMS_NAME).read_text().splitlines():
|
|
||||||
digest, name = line.split(" ", 1)
|
|
||||||
assert release.sha256_file(evidence / name) == digest
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_broken_documentation_check_fails_the_gate(tmp_path):
|
|
||||||
"""The failure mode that matters: a page and the code disagreeing."""
|
|
||||||
failing = tmp_path / "test_disagreement.py"
|
|
||||||
failing.write_text(
|
|
||||||
"def test_the_manual_matches_the_code():\n"
|
|
||||||
" documented = {'lock_held'}\n"
|
|
||||||
" emitted = {'lock_held', 'renamed_since'}\n"
|
|
||||||
" assert emitted <= documented\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
report = release.run_gate(
|
|
||||||
_config(tmp_path),
|
|
||||||
output=tmp_path / "evidence",
|
|
||||||
stages=(("documentation", (str(failing),)),),
|
|
||||||
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert report["ok"] is False
|
|
||||||
assert report["failures"], report
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_clean_tree_passes(tmp_path):
|
|
||||||
passing = tmp_path / "test_passing.py"
|
|
||||||
passing.write_text("def test_x():\n assert True\n")
|
|
||||||
|
|
||||||
report = release.run_gate(
|
|
||||||
_config(tmp_path),
|
|
||||||
output=tmp_path / "evidence",
|
|
||||||
stages=(("documentation", (str(passing),)),),
|
|
||||||
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert report["ok"] is True and report["unexpected_skips"] == []
|
|
||||||
assert report["revision"], "the evidence must say which commit it covers"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the command, and CI ──────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_command_is_documented_and_wired():
|
|
||||||
assert "docs-gate" in (REPO / "photo_pipeline" / "__main__.py").read_text()
|
|
||||||
assert "docs-gate" in (DOCS / "index.md").read_text()
|
|
||||||
|
|
||||||
|
|
||||||
def test_ci_runs_the_gate_on_pull_requests_and_keeps_its_evidence():
|
|
||||||
job = TEST_WORKFLOW["jobs"]["documentation"]
|
|
||||||
# No `if:` — documentation drifts on the same commits that change behaviour, and
|
|
||||||
# the pull request is when saying so is still cheap.
|
|
||||||
assert "if" not in job
|
|
||||||
assert "pull_request" in TEST_WORKFLOW[True] or "pull_request" in TEST_WORKFLOW.get("on", {})
|
|
||||||
script = "\n".join(step["run"] for step in job["steps"] if "run" in step)
|
|
||||||
assert "photo_pipeline docs-gate" in script
|
|
||||||
evidence = next(step for step in job["steps"] if "upload-artifact" in str(step.get("uses")))
|
|
||||||
assert evidence["if"] == "always()", "a failed gate's evidence is the one worth keeping"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the pages and the repository point at each other ─────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_readme_and_the_documentation_index_link_to_each_other():
|
|
||||||
"""Neither may become the forgotten copy."""
|
|
||||||
readme = (REPO / "README.md").read_text()
|
|
||||||
index = (DOCS / "index.md").read_text()
|
|
||||||
assert "docs/index.md" in readme
|
|
||||||
assert re.search(r"\.\./README\.md", index), "the index does not link back to the README"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_documentation_story_is_mapped_in_the_traceability_matrix():
|
|
||||||
matrix = json.loads((REPO / "tests" / "story_traceability.json").read_text())["stories"]
|
|
||||||
for story in ("US09-01", "US09-02", "US09-03", "US09-04", "US09-05"):
|
|
||||||
assert matrix.get(story), f"{story} is not mapped to any test"
|
|
||||||
@@ -1,184 +0,0 @@
|
|||||||
"""US09-01: the documentation the application serves, checked without a browser.
|
|
||||||
|
|
||||||
What is checkable offline is everything that makes the manuals *navigable* rather
|
|
||||||
than merely present: every page reachable, every internal link and anchor landing
|
|
||||||
somewhere real, the vendored renderer being the file that was pinned, and the
|
|
||||||
application actually serving `docs/` in both the working copy and the image.
|
|
||||||
|
|
||||||
The rendering itself — marked, mermaid, anchors, link rewriting — is proven in a
|
|
||||||
real browser by ``tests/e2e/test_docs_ui.py``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from starlette.testclient import TestClient
|
|
||||||
|
|
||||||
from photo_pipeline.api.app import create_app
|
|
||||||
from photo_pipeline.api.security import DEFAULT_HEADERS
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
|
|
||||||
# Every check here belongs to the documentation gate (US09-05).
|
|
||||||
pytestmark = pytest.mark.phase_i
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
|
||||||
DOCS = REPO / "docs"
|
|
||||||
VENDOR = REPO / "frontend" / "js" / "vendor"
|
|
||||||
INDEX = DOCS / "index.md"
|
|
||||||
|
|
||||||
MARKDOWN_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)")
|
|
||||||
HEADING = re.compile(r"^#{1,6}\s+(.+?)\s*$", re.MULTILINE)
|
|
||||||
FENCE = re.compile(r"^```.*?^```", re.MULTILINE | re.DOTALL)
|
|
||||||
|
|
||||||
|
|
||||||
def pages() -> list[Path]:
|
|
||||||
return sorted(DOCS.rglob("*.md"))
|
|
||||||
|
|
||||||
|
|
||||||
def body(path: Path) -> str:
|
|
||||||
"""A document without its fenced code, so an example link in a shell snippet is
|
|
||||||
not mistaken for a link the reader can follow."""
|
|
||||||
return FENCE.sub("", path.read_text())
|
|
||||||
|
|
||||||
|
|
||||||
def slug(text: str) -> str:
|
|
||||||
"""The heading-anchor rule, mirroring ``frontend/js/docs.js``.
|
|
||||||
|
|
||||||
Deliberately duplicated: this check has to run without a browser. The browser
|
|
||||||
test is the authority on the real behaviour, and it asserts the same anchors.
|
|
||||||
"""
|
|
||||||
cleaned = re.sub(r"[^\w\s-]", "", text.lower(), flags=re.UNICODE).strip()
|
|
||||||
return re.sub(r"[\s-]+", "-", cleaned).strip("-") or "section"
|
|
||||||
|
|
||||||
|
|
||||||
def anchors(path: Path) -> set[str]:
|
|
||||||
found: dict[str, int] = {}
|
|
||||||
for heading in HEADING.findall(body(path)):
|
|
||||||
# Markdown emphasis and inline code are not part of the rendered text.
|
|
||||||
plain = re.sub(r"[*`_]", "", heading)
|
|
||||||
base = slug(plain)
|
|
||||||
found[base] = found.get(base, 0) + 1
|
|
||||||
return {name if index == 1 else f"{name}-{index}" for name, count in found.items() for index in range(1, count + 1)}
|
|
||||||
|
|
||||||
|
|
||||||
def internal_links(path: Path) -> list[str]:
|
|
||||||
return [
|
|
||||||
href
|
|
||||||
for href in MARKDOWN_LINK.findall(body(path))
|
|
||||||
if not re.match(r"^[a-z][a-z\d+.-]*:", href, re.IGNORECASE) and not href.startswith("//")
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
# ── the documentation tree ───────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_index_exists_and_names_every_page():
|
|
||||||
"""A page nobody links to is a page nobody reads."""
|
|
||||||
assert INDEX.is_file()
|
|
||||||
listed = {
|
|
||||||
(INDEX.parent / href.split("#")[0]).resolve()
|
|
||||||
for href in internal_links(INDEX)
|
|
||||||
if href.split("#")[0].endswith(".md")
|
|
||||||
}
|
|
||||||
unreachable = [page.name for page in pages() if page != INDEX and page.resolve() not in listed]
|
|
||||||
assert unreachable == [], f"not linked from the index: {unreachable}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_internal_link_and_anchor_resolves():
|
|
||||||
broken: list[str] = []
|
|
||||||
for page in pages():
|
|
||||||
for href in internal_links(page):
|
|
||||||
target, _, anchor = href.partition("#")
|
|
||||||
destination = page if not target else (page.parent / target).resolve()
|
|
||||||
if not destination.is_file():
|
|
||||||
broken.append(f"{page.name} → {href} (no such file)")
|
|
||||||
continue
|
|
||||||
if anchor and anchor not in anchors(destination):
|
|
||||||
broken.append(f"{page.name} → {href} (no such heading)")
|
|
||||||
assert broken == [], broken
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_page_starts_with_one_title():
|
|
||||||
for page in pages():
|
|
||||||
titles = [line for line in page.read_text().splitlines() if line.startswith("# ")]
|
|
||||||
assert len(titles) == 1, f"{page.name} has {len(titles)} top-level titles"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the vendored renderer ────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_vendored_libraries_are_the_files_that_were_pinned():
|
|
||||||
"""A vendored dependency without a recorded checksum is a dependency nobody is
|
|
||||||
reviewing. Replacing one has to be a visible change to this manifest."""
|
|
||||||
manifest = json.loads((VENDOR / "VERSIONS.json").read_text())
|
|
||||||
recorded = {entry["file"]: entry for entry in manifest["libraries"]}
|
|
||||||
|
|
||||||
on_disk = {path.name for path in VENDOR.iterdir() if path.suffix in (".js", ".mjs")}
|
|
||||||
assert on_disk == set(recorded), f"unrecorded vendored files: {on_disk ^ set(recorded)}"
|
|
||||||
|
|
||||||
for name, entry in recorded.items():
|
|
||||||
digest = hashlib.sha256((VENDOR / name).read_bytes()).hexdigest()
|
|
||||||
assert digest == entry["sha256"], f"{name} does not match the pinned checksum"
|
|
||||||
assert entry["version"] in entry["url"], f"{name}: the pinned URL and version disagree"
|
|
||||||
assert entry["license"], f"{name}: no license recorded"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_documentation_view_is_wired_into_the_shell():
|
|
||||||
shell = (REPO / "frontend" / "index.html").read_text()
|
|
||||||
assert 'data-nav="docs"' in shell, "the manuals are unreachable from the navigation"
|
|
||||||
assert '"/docs"' in (REPO / "frontend" / "js" / "app.js").read_text()
|
|
||||||
|
|
||||||
|
|
||||||
# ── the policy the view runs under ───────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_script_boundary_is_unchanged_and_only_style_was_relaxed():
|
|
||||||
"""The measured cost of rendering diagrams in the browser, held to that cost.
|
|
||||||
|
|
||||||
mermaid needs to style the SVG it builds, which `style-src 'self'` refuses. It
|
|
||||||
does not need to execute generated code, so `script-src` must never acquire the
|
|
||||||
escape hatch that would let injected markup run.
|
|
||||||
"""
|
|
||||||
policy = DEFAULT_HEADERS["content-security-policy"]
|
|
||||||
directives = {
|
|
||||||
part.split(" ")[0]: part.split(" ")[1:]
|
|
||||||
for part in (piece.strip() for piece in policy.split(";"))
|
|
||||||
if part
|
|
||||||
}
|
|
||||||
assert directives["script-src"] == ["'self'"]
|
|
||||||
assert "'unsafe-inline'" in directives["style-src"]
|
|
||||||
for directive in ("default-src", "img-src", "connect-src", "font-src"):
|
|
||||||
assert "'self'" in directives[directive]
|
|
||||||
assert "'unsafe-inline'" not in directives[directive]
|
|
||||||
assert directives["frame-ancestors"] == ["'none'"]
|
|
||||||
|
|
||||||
|
|
||||||
# ── serving it ───────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_application_serves_the_markdown_without_a_session(tmp_path):
|
|
||||||
"""Whoever cannot get past the access secret is exactly who needs these pages."""
|
|
||||||
config = Config.from_env({"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data")})
|
|
||||||
(tmp_path / "data").mkdir()
|
|
||||||
with TestClient(create_app(config), raise_server_exceptions=False) as client:
|
|
||||||
response = client.get("/docs/index.md", headers={"cookie": ""})
|
|
||||||
assert response.status_code == 200
|
|
||||||
assert "Photo Pipeline documentation" in response.text
|
|
||||||
assert client.get("/docs/overview.md").status_code == 200
|
|
||||||
assert client.get("/docs/nothing-here.md").status_code == 404
|
|
||||||
# The mount is a directory, not a path parameter: nothing above it is reachable.
|
|
||||||
assert client.get("/docs/../pyproject.toml").status_code in (307, 404)
|
|
||||||
assert client.get("/docs/%2e%2e/pyproject.toml").status_code == 404
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_image_ships_the_documentation_it_serves():
|
|
||||||
"""An image without `docs/` serves an empty manual — and the build context is
|
|
||||||
deny-by-default, so a new directory is excluded until it is named."""
|
|
||||||
assert "\n!docs\n" in (REPO / ".dockerignore").read_text()
|
|
||||||
assert "COPY docs ./docs" in (REPO / "Dockerfile").read_text()
|
|
||||||
@@ -1,183 +0,0 @@
|
|||||||
"""US09-02: the installation manual, checked against the thing it describes.
|
|
||||||
|
|
||||||
Documentation rots quietly. A setting is renamed, a command grows a flag, an exit
|
|
||||||
code changes meaning — and the manual keeps confidently saying the old thing until
|
|
||||||
somebody follows it into a bad afternoon. So every fact in it that the code also
|
|
||||||
knows is compared with the code, in both directions where both directions are
|
|
||||||
meaningful: a setting the manual invents fails, and a setting the manual forgot
|
|
||||||
fails too.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from functools import lru_cache
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from photo_pipeline.config import ENV_PREFIX, LEGACY_ALIASES, Config
|
|
||||||
|
|
||||||
# Every check here belongs to the documentation gate (US09-05).
|
|
||||||
pytestmark = pytest.mark.phase_i
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
|
||||||
MANUAL = REPO / "docs" / "installation.md"
|
|
||||||
MAIN = REPO / "photo_pipeline" / "__main__.py"
|
|
||||||
|
|
||||||
TEXT = MANUAL.read_text()
|
|
||||||
ENV_NAMES = re.compile(rf"{ENV_PREFIX}[A-Z0-9_]+")
|
|
||||||
|
|
||||||
# Read by docker-compose.yml, not by Config. They are configuration an installer sets,
|
|
||||||
# so the manual documents them; they are simply not application settings.
|
|
||||||
COMPOSITION_ONLY = {
|
|
||||||
f"{ENV_PREFIX}IMAGE",
|
|
||||||
f"{ENV_PREFIX}LIBRARY_HOST_PATH",
|
|
||||||
f"{ENV_PREFIX}PUBLISH_ADDRESS",
|
|
||||||
f"{ENV_PREFIX}UID",
|
|
||||||
f"{ENV_PREFIX}GID",
|
|
||||||
f"{ENV_PREFIX}ENV_FILE", # read before Config exists, so it is not a field
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def documented_env_names() -> set[str]:
|
|
||||||
return set(ENV_NAMES.findall(TEXT))
|
|
||||||
|
|
||||||
|
|
||||||
@lru_cache
|
|
||||||
def cli_help(*args: str) -> str:
|
|
||||||
"""What the CLI says about itself, asked the way an operator asks."""
|
|
||||||
result = subprocess.run(
|
|
||||||
[sys.executable, "-m", "photo_pipeline", *args, "--help"],
|
|
||||||
cwd=REPO,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=120,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
assert result.returncode == 0, result.stderr
|
|
||||||
return result.stdout
|
|
||||||
|
|
||||||
|
|
||||||
@lru_cache
|
|
||||||
def cli_commands() -> frozenset[str]:
|
|
||||||
listed = re.search(r"\{([a-z0-9,\-]+)\}", cli_help())
|
|
||||||
assert listed, f"the CLI listed no subcommands:\n{cli_help()}"
|
|
||||||
return frozenset(listed.group(1).split(","))
|
|
||||||
|
|
||||||
|
|
||||||
def cli_flags(command: str) -> frozenset[str]:
|
|
||||||
return frozenset(re.findall(r"--[a-z][a-z-]+", cli_help(command)))
|
|
||||||
|
|
||||||
|
|
||||||
# ── settings ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_setting_is_documented():
|
|
||||||
"""A setting nobody documents is a setting nobody configures deliberately."""
|
|
||||||
expected = {f"{ENV_PREFIX}{field.upper()}" for field in Config.model_fields}
|
|
||||||
missing = sorted(expected - documented_env_names())
|
|
||||||
assert missing == [], f"undocumented settings: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_manual_invents_no_settings():
|
|
||||||
unknown = sorted(
|
|
||||||
name
|
|
||||||
for name in documented_env_names()
|
|
||||||
if name.removeprefix(ENV_PREFIX).lower() not in Config.model_fields
|
|
||||||
and name not in COMPOSITION_ONLY
|
|
||||||
)
|
|
||||||
assert unknown == [], f"documented but not a real setting: {unknown}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_documented_defaults_are_the_real_defaults():
|
|
||||||
"""Spot-checked where a wrong default is actively dangerous: the bind address,
|
|
||||||
the trust boundary, and the gate that protects an irreplaceable library."""
|
|
||||||
assert Config.model_fields["host"].default == "127.0.0.1"
|
|
||||||
assert "`PHOTO_PIPELINE_HOST` | `127.0.0.1`" in TEXT
|
|
||||||
assert Config.model_fields["allowed_hosts"].default == ()
|
|
||||||
assert Config.model_fields["require_dry_run_approval"].default is False
|
|
||||||
assert "`PHOTO_PIPELINE_REQUIRE_DRY_RUN_APPROVAL` | `false`" in TEXT
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_secret_is_marked_as_one():
|
|
||||||
for field in ("access_secret", "vision_api_key", "immich_api_key"):
|
|
||||||
name = f"{ENV_PREFIX}{field.upper()}"
|
|
||||||
row = next(line for line in TEXT.splitlines() if line.startswith(f"| `{name}`"))
|
|
||||||
assert "secret" in row.lower(), f"{name} is not marked as a secret"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_legacy_aliases_are_documented():
|
|
||||||
"""An operator with an existing photo_analyzer.env needs to know it still works."""
|
|
||||||
for alias in LEGACY_ALIASES:
|
|
||||||
assert alias in TEXT, f"legacy alias {alias} is undocumented"
|
|
||||||
|
|
||||||
|
|
||||||
# ── commands ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
# The manual is an installation and operations manual, not a command reference: these
|
|
||||||
# are the commands an installer actually runs, and every one of them must exist.
|
|
||||||
OPERATIONAL = ("migrate", "serve", "worker", "backup", "verify-backup", "restore", "diagnostics")
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_command_the_manual_tells_you_to_run_exists():
|
|
||||||
missing = [name for name in OPERATIONAL if name not in cli_commands()]
|
|
||||||
assert missing == [], f"the manual names commands the CLI does not have: {missing}"
|
|
||||||
undocumented = [name for name in OPERATIONAL if not re.search(rf"\b{re.escape(name)}\b", TEXT)]
|
|
||||||
assert undocumented == [], f"operational commands the manual omits: {undocumented}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_documented_flag_exists_on_the_command_it_is_shown_with():
|
|
||||||
for command, flag in (
|
|
||||||
("backup", "--reason"),
|
|
||||||
("backup", "--keep"),
|
|
||||||
("restore", "--into"),
|
|
||||||
("worker", "--allow-legacy"),
|
|
||||||
("serve", "--allow-legacy"),
|
|
||||||
):
|
|
||||||
assert flag in cli_flags(command), f"{command} has no {flag}"
|
|
||||||
assert flag in TEXT, f"{flag} is undocumented"
|
|
||||||
|
|
||||||
|
|
||||||
# ── exit codes ───────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_documented_exit_code_is_one_the_cli_can_return():
|
|
||||||
documented = {int(code) for code in re.findall(r"^\| `(\d)` \|", TEXT, re.MULTILINE)}
|
|
||||||
returned = {int(code) for code in re.findall(r"^\s+return (\d)$", MAIN.read_text(), re.MULTILINE)}
|
|
||||||
assert documented, "no exit codes are documented"
|
|
||||||
assert documented <= returned | {0}, f"documented but unreachable: {sorted(documented - returned)}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_refusal_exit_code_is_documented():
|
|
||||||
"""0 is success and 1 is 'it said why'; every other code is a specific refusal an
|
|
||||||
operator will meet at startup, and meeting an undocumented one is the worst case."""
|
|
||||||
returned = {int(code) for code in re.findall(r"^\s+return (\d)$", MAIN.read_text(), re.MULTILINE)}
|
|
||||||
documented = {int(code) for code in re.findall(r"^\| `(\d)` \|", TEXT, re.MULTILINE)}
|
|
||||||
missing = sorted(code for code in returned if code not in documented and code != 0)
|
|
||||||
assert missing == [], f"undocumented exit codes: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the promises the manual makes ────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_manual_carries_no_credential_shaped_example():
|
|
||||||
"""The repository's own scanner refuses these in a commit; a manual is exactly
|
|
||||||
where a real-looking one gets copied from."""
|
|
||||||
suspicious = re.findall(
|
|
||||||
r"(?i)(api[_-]?key|secret|token|password)\s*[=:]\s*[\"']?([A-Za-z0-9_\-]{12,})",
|
|
||||||
TEXT,
|
|
||||||
)
|
|
||||||
assert suspicious == [], f"credential-shaped example: {suspicious}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_manual_states_the_invariants_an_installer_must_not_break():
|
|
||||||
for invariant in ("_IGNORE/", "One writer", "EXIF is verified before upload"):
|
|
||||||
assert invariant in TEXT, f"the manual does not state: {invariant}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_manual_is_reachable_from_the_index():
|
|
||||||
assert "installation.md" in (REPO / "docs" / "index.md").read_text()
|
|
||||||
@@ -1,177 +0,0 @@
|
|||||||
"""US09-04: the user manual, checked against the application it describes.
|
|
||||||
|
|
||||||
The manual's most perishable claim is its error catalogue: a code renamed in a route
|
|
||||||
leaves a page confidently explaining something that can no longer happen, and the
|
|
||||||
operator meeting the new code finds nothing. So the catalogue is compared with the
|
|
||||||
codes the application can actually emit, in both directions.
|
|
||||||
|
|
||||||
The screenshots are generated and checked by
|
|
||||||
``tests/e2e/test_user_manual_screenshots.py``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
|
||||||
DOCS = REPO / "docs"
|
|
||||||
STAGES = DOCS / "stages"
|
|
||||||
PACKAGE = REPO / "photo_pipeline"
|
|
||||||
ERRORS = (DOCS / "errors.md").read_text()
|
|
||||||
ALL_DOCS = "\n".join(path.read_text() for path in DOCS.rglob("*.md"))
|
|
||||||
|
|
||||||
# The shapes an error code is emitted in. Everything the API returns goes through one
|
|
||||||
# of these, so the set they find is the set an operator can meet.
|
|
||||||
EMITTERS = (
|
|
||||||
re.compile(r'_error\(\s*\d+,\s*"([a-z_]+)"'),
|
|
||||||
# The status may be a literal or an exception's own code, so match either.
|
|
||||||
re.compile(r'_envelope\(\s*[\w.]+,\s*"([a-z_]+)"'),
|
|
||||||
re.compile(r'"code":\s*"([a-z_]+)"'),
|
|
||||||
re.compile(r'code="([a-z_]+)"'),
|
|
||||||
re.compile(r'Refusal\(\s*\d+,\s*"([a-z_]+)"'),
|
|
||||||
)
|
|
||||||
|
|
||||||
# Codes raised deep in a service and mapped to a response by its route; they are what
|
|
||||||
# the browser shows, so the manual owes the reader an entry.
|
|
||||||
SERVICE_CODES = {
|
|
||||||
"lock_held",
|
|
||||||
"path_not_allowed",
|
|
||||||
"rename_recovery_required",
|
|
||||||
"stale_preflight",
|
|
||||||
"access_denied",
|
|
||||||
"too_many_attempts",
|
|
||||||
}
|
|
||||||
|
|
||||||
STAGE_PAGES = (
|
|
||||||
"inventory",
|
|
||||||
"duplicates",
|
|
||||||
"safety",
|
|
||||||
"analysis",
|
|
||||||
"albums",
|
|
||||||
"renames",
|
|
||||||
"uploads",
|
|
||||||
"archive",
|
|
||||||
"diagnostics",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Every check here belongs to the documentation gate (US09-05).
|
|
||||||
pytestmark = pytest.mark.phase_i
|
|
||||||
|
|
||||||
|
|
||||||
def emitted_codes() -> set[str]:
|
|
||||||
found: set[str] = set()
|
|
||||||
for path in PACKAGE.rglob("*.py"):
|
|
||||||
text = path.read_text()
|
|
||||||
for pattern in EMITTERS:
|
|
||||||
found |= set(pattern.findall(text))
|
|
||||||
for code in SERVICE_CODES:
|
|
||||||
assert re.search(rf'"{code}"', "\n".join(p.read_text() for p in PACKAGE.rglob("*.py"))), (
|
|
||||||
f"{code} is documented as a service code but no longer exists"
|
|
||||||
)
|
|
||||||
return (found | SERVICE_CODES) - {"code"}
|
|
||||||
|
|
||||||
|
|
||||||
def documented_codes() -> set[str]:
|
|
||||||
return set(re.findall(r"`([a-z][a-z_]{3,})`", ERRORS))
|
|
||||||
|
|
||||||
|
|
||||||
# ── the error catalogue ──────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_error_the_application_can_emit_is_documented():
|
|
||||||
missing = sorted(emitted_codes() - documented_codes())
|
|
||||||
assert missing == [], f"undocumented error codes: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_catalogue_documents_no_code_that_cannot_happen():
|
|
||||||
"""Backticked snake_case in the catalogue is either a code, a setting, or one of
|
|
||||||
the handful of prose terms below."""
|
|
||||||
prose = {
|
|
||||||
"diagnostics",
|
|
||||||
"dry_run",
|
|
||||||
"approve_dry_run",
|
|
||||||
"current_path",
|
|
||||||
"archived_online",
|
|
||||||
"archived_offline",
|
|
||||||
}
|
|
||||||
settings = set(re.findall(r"PHOTO_PIPELINE_[A-Z_]+", ERRORS))
|
|
||||||
invented = sorted(
|
|
||||||
code
|
|
||||||
for code in documented_codes()
|
|
||||||
if code not in emitted_codes()
|
|
||||||
and code not in prose
|
|
||||||
and f"PHOTO_PIPELINE_{code.upper()}" not in settings
|
|
||||||
)
|
|
||||||
assert invented == [], f"documented but not emitted anywhere: {invented}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_protective_refusals_are_explained_as_intentional():
|
|
||||||
"""These are the ones a reader is most likely to mistake for a broken
|
|
||||||
application, which is exactly when they start working around them."""
|
|
||||||
for code in ("lock_held", "rename_recovery_required", "stale_preflight", "not_runnable"):
|
|
||||||
row = next(line for line in ERRORS.splitlines() if line.startswith(f"| `{code}`"))
|
|
||||||
assert len(row.split("|")) >= 4, f"{code} has no cause and remedy"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the stage pages ──────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_there_is_one_page_per_workflow_stage():
|
|
||||||
missing = [name for name in STAGE_PAGES if not (STAGES / f"{name}.md").is_file()]
|
|
||||||
assert missing == [], f"stages without a page: {missing}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_stage_page_answers_the_four_questions():
|
|
||||||
"""The shape is the point: a page that skips 'what it changes' is the page
|
|
||||||
somebody needed."""
|
|
||||||
for name in STAGE_PAGES:
|
|
||||||
text = (STAGES / f"{name}.md").read_text()
|
|
||||||
for question in ("What it is for.", "What you decide.", "What it changes.", "What it refuses."):
|
|
||||||
assert question in text, f"{name}.md does not answer: {question}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_stage_page_shows_a_screenshot():
|
|
||||||
for name in STAGE_PAGES:
|
|
||||||
text = (STAGES / f"{name}.md").read_text()
|
|
||||||
assert re.search(r"!\[[^\]]+\]\(\.\./images/[a-z-]+\.png\)", text), (
|
|
||||||
f"{name}.md carries no screenshot"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_guided_pass_names_the_irreversible_stages():
|
|
||||||
first_pass = (DOCS / "first-pass.md").read_text()
|
|
||||||
for stage in ("renames", "uploads", "archive"):
|
|
||||||
assert f"stages/{stage}.md" in first_pass
|
|
||||||
assert "point" in first_pass.lower() and "no return" in first_pass.lower()
|
|
||||||
|
|
||||||
|
|
||||||
def test_recovery_covers_each_interruption_the_application_can_survive():
|
|
||||||
recovery = (DOCS / "recovery.md").read_text()
|
|
||||||
for topic in (
|
|
||||||
"interrupted rename",
|
|
||||||
"uncertain upload",
|
|
||||||
"failed migration",
|
|
||||||
"restored backup",
|
|
||||||
):
|
|
||||||
assert topic in recovery.lower(), f"recovery does not cover: {topic}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_manual_is_reachable_from_the_index():
|
|
||||||
index = (DOCS / "index.md").read_text()
|
|
||||||
for page in [f"stages/{name}.md" for name in STAGE_PAGES] + [
|
|
||||||
"first-pass.md",
|
|
||||||
"errors.md",
|
|
||||||
"recovery.md",
|
|
||||||
]:
|
|
||||||
assert page in index, f"{page} is not linked from the index"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_screenshots_are_declared_as_generated():
|
|
||||||
"""The allow list had to be widened for them, so the reason has to be visible
|
|
||||||
where the widening is."""
|
|
||||||
config = (REPO / "work_item" / ".work-item.yml").read_text()
|
|
||||||
assert "docs/images/**" in config
|
|
||||||
assert "test_user_manual_screenshots.py" in config, "no note saying who generates them"
|
|
||||||
@@ -193,26 +193,14 @@
|
|||||||
"US08-05": [
|
"US08-05": [
|
||||||
"tests/integration/test_container_gate.py",
|
"tests/integration/test_container_gate.py",
|
||||||
"tests/e2e/test_phase_h_container.py"
|
"tests/e2e/test_phase_h_container.py"
|
||||||
],
|
|
||||||
"US09-01": [
|
|
||||||
"tests/integration/test_documentation.py",
|
|
||||||
"tests/e2e/test_docs_ui.py"
|
|
||||||
],
|
|
||||||
"US09-02": [
|
|
||||||
"tests/integration/test_installation_manual.py"
|
|
||||||
],
|
|
||||||
"US09-03": [
|
|
||||||
"tests/integration/test_architecture_overview.py",
|
|
||||||
"tests/e2e/test_docs_ui.py"
|
|
||||||
],
|
|
||||||
"US09-04": [
|
|
||||||
"tests/integration/test_user_manual.py",
|
|
||||||
"tests/e2e/test_user_manual_screenshots.py"
|
|
||||||
],
|
|
||||||
"US09-05": [
|
|
||||||
"tests/integration/test_docs_gate.py"
|
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"planned": [],
|
"planned": [
|
||||||
|
"US09-01",
|
||||||
|
"US09-02",
|
||||||
|
"US09-03",
|
||||||
|
"US09-04",
|
||||||
|
"US09-05"
|
||||||
|
],
|
||||||
"_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
|
"_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,9 +16,6 @@ safety:
|
|||||||
max_file_bytes: 5000000
|
max_file_bytes: 5000000
|
||||||
allow:
|
allow:
|
||||||
- tests/fixtures/**
|
- tests/fixtures/**
|
||||||
# Generated by tests/e2e/test_user_manual_screenshots.py from a synthetic fixture
|
|
||||||
# library, never a real photo. The deny list below blocks images by default.
|
|
||||||
- docs/images/**
|
|
||||||
deny:
|
deny:
|
||||||
- _IGNORE/**
|
- _IGNORE/**
|
||||||
- "**/_IGNORE/**"
|
- "**/_IGNORE/**"
|
||||||
|
|||||||