Compare commits
3 Commits
us/US08-05
...
us/US09-01
| Author | SHA1 | Date | |
|---|---|---|---|
| adfdf1dbdb | |||
| 370f966d29 | |||
| 833bfa95bf |
@@ -9,6 +9,7 @@
|
||||
!photo_pipeline
|
||||
!migrations
|
||||
!frontend
|
||||
!docs
|
||||
!docker
|
||||
|
||||
# Nothing generated, even under an allowed directory.
|
||||
|
||||
9
.gitattributes
vendored
Normal file
9
.gitattributes
vendored
Normal file
@@ -0,0 +1,9 @@
|
||||
# Vendored third-party browser bundles (US09-01). Minified upstream files carry
|
||||
# trailing whitespace and very long lines; they are not ours to reformat, and the
|
||||
# submit gate's `git diff --check` would refuse them forever. Their integrity is
|
||||
# controlled where it belongs instead: a pinned version and a recorded sha256 in
|
||||
# frontend/js/vendor/VERSIONS.json, asserted by tests/integration/test_documentation.py.
|
||||
#
|
||||
# Only the whitespace check is turned off. They stay text, so a credential scan or a
|
||||
# search still reads them.
|
||||
frontend/js/vendor/*.js -whitespace linguist-vendored
|
||||
@@ -49,3 +49,55 @@ jobs:
|
||||
|
||||
- name: Application suite
|
||||
run: work_item/scripts/python -m pytest tests -q
|
||||
|
||||
# The deployed container, verified the way the host application is (US08-05). It
|
||||
# runs on `main` only — a pull request has nothing published to upgrade *from*, and
|
||||
# building two images per push would pay for that on every commit. Deploy waits for
|
||||
# this whole workflow, so a red container gate is a deploy that does not happen.
|
||||
container:
|
||||
if: gitea.event_name == 'push'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
# The upgrade journey builds the previous commit's tree when no published
|
||||
# image is named, so the history has to be there.
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 2
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install the application and its test dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -e '.[test]'
|
||||
python -m playwright install --with-deps chromium
|
||||
|
||||
- name: Container acceptance gate
|
||||
# One command: it builds the image, provisions the composition against a
|
||||
# temporary fixture library and an isolated volume, runs the phase_h journeys,
|
||||
# destroys the stack, and writes the evidence. Any skipped check fails it.
|
||||
env:
|
||||
# The fixture libraries are bind-mounted into the containers, so this path
|
||||
# has to be one the Docker daemon can see. On a runner that talks to a
|
||||
# sibling daemon, point it at a shared host path instead of the workspace —
|
||||
# an unshared path arrives as an empty mount and the journeys fail on the
|
||||
# scan, which is the symptom to recognise.
|
||||
PHOTO_PIPELINE_DATA_DIR: ${{ gitea.workspace }}/gate-data
|
||||
PHOTO_PIPELINE_LIBRARY_ROOTS: ${{ gitea.workspace }}/gate-library
|
||||
PHOTO_PIPELINE_TEST_MOUNT_BASE: ${{ gitea.workspace }}/gate-mounts
|
||||
run: |
|
||||
mkdir -p "$PHOTO_PIPELINE_LIBRARY_ROOTS" "$PHOTO_PIPELINE_TEST_MOUNT_BASE"
|
||||
python -m photo_pipeline container-gate --output gate-evidence
|
||||
|
||||
- name: Keep the evidence
|
||||
# Retained per run, and retained on failure especially: the logs are the only
|
||||
# account of what the containers did.
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: container-gate-${{ gitea.sha }}
|
||||
path: gate-evidence
|
||||
|
||||
@@ -70,6 +70,8 @@ COPY pyproject.toml alembic.ini README.md ./
|
||||
COPY photo_pipeline ./photo_pipeline
|
||||
COPY migrations ./migrations
|
||||
COPY frontend ./frontend
|
||||
# The manuals are served by the application itself (US09-01), so they ship with it.
|
||||
COPY docs ./docs
|
||||
COPY docker/entrypoint.sh docker/healthcheck.sh /usr/local/bin/
|
||||
|
||||
# Runtime dependencies only: the `test` extra (pytest, playwright) and the `vision`
|
||||
|
||||
34
README.md
34
README.md
@@ -241,6 +241,40 @@ image. The stack is managed by Portainer from git (`docker-compose.yml`), and th
|
||||
values live in the Portainer stack's environment, so rotation is one place and a
|
||||
repository read discloses nothing. A test asserts the workflows never name them.
|
||||
|
||||
## Container acceptance gate (US08-05)
|
||||
|
||||
The deployed container is verified the way the host application is, by one command:
|
||||
|
||||
```bash
|
||||
work_item/scripts/python -m photo_pipeline container-gate --output gate-evidence
|
||||
```
|
||||
|
||||
It builds the image, provisions the composition against a **temporary fixture library
|
||||
on a bind mount and an isolated data volume**, runs the `phase_h` journeys against it,
|
||||
destroys every stack afterwards, and writes `release-report.json`, `logs/container.log`,
|
||||
and `CHECKSUMS.sha256` into the evidence directory. It exits non-zero when a journey
|
||||
fails, when the story matrix has a hole, **or when any check skipped at all** — unlike
|
||||
the release gate, this one accepts no environment excuse: a run that never reached the
|
||||
containers proved nothing about them.
|
||||
|
||||
The journeys (`tests/e2e/test_phase_h_container.py`) are:
|
||||
|
||||
| journey | what it proves |
|
||||
|---|---|
|
||||
| browser | discovery, duplicate review, analysis, album proposal, rename, upload preflight, and archive views, driven through the containerized frontend — including a rename that really moves the operator's folder on the bind mount |
|
||||
| upgrade | the previous version's image runs first, then this one against the same volume: schema at the new head, assets, analysis results, job progress, the unapplied rename plan, and the thumbnail cache all survive |
|
||||
| restart | `docker kill` on both containers mid-job; the job resumes, every photo ends with exactly one stored result, and only the in-flight item ever reaches the provider twice |
|
||||
| security | no session refused, a forged `X-Forwarded-Host` cannot smuggle an allowed hostname past the check, a symlink out of the mounted library is refused, an unmounted library root refuses startup, and no secret appears in `docker compose logs` |
|
||||
|
||||
The upgrade journey builds the previous commit's tree when no published image is named;
|
||||
point it at the real one with `PHOTO_PIPELINE_PREVIOUS_IMAGE`. On a Docker VM (Colima,
|
||||
Docker Desktop) the fixture library must live on a shared path — it defaults to
|
||||
`~/.cache/photo-pipeline`, overridable with `PHOTO_PIPELINE_TEST_MOUNT_BASE`.
|
||||
|
||||
CI runs this gate as the `container` job of **Test** on pushes to `main` and keeps its
|
||||
evidence as a run artefact. Deploy waits for the whole `Test` workflow, so a red
|
||||
container gate is a publish that does not happen.
|
||||
|
||||
## Testing
|
||||
|
||||
One offline command runs the whole suite (unit, integration, and browser
|
||||
|
||||
77
delivery_backlog/E09-documentation.md
Normal file
77
delivery_backlog/E09-documentation.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# E09 — Product Documentation
|
||||
|
||||
Concept phase: none. Like [E08](E08-container-deployment.md), this epic is a delivery
|
||||
addition rather than a product-scope change: the same application, documented well
|
||||
enough that somebody who did not build it can install it, understand it, and operate it
|
||||
without reading the source.
|
||||
|
||||
It does not change the product scope in
|
||||
[`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md). No safety
|
||||
invariant moves, no schema changes, no new runtime capability. The one change to the
|
||||
running application is a documentation view and the static assets it needs.
|
||||
|
||||
## Why the application serves its own documentation
|
||||
|
||||
The manuals describe an application that is reached over HTTP behind an access secret.
|
||||
Documentation that lives only in the repository is unreachable from the deployment it
|
||||
describes: an operator who has just been handed a URL and a secret has no Gitea account
|
||||
in front of them. So the same markdown files are both the repository's documentation and
|
||||
the deployment's `/docs` view, and neither is a copy of the other.
|
||||
|
||||
## Decisions made in this epic
|
||||
|
||||
**Markdown is the source.** Everything is written as markdown under `docs/`, so it is
|
||||
reviewable in a diff, readable on Gitea, and renderable in the app. No documentation
|
||||
format that only a tool can read.
|
||||
|
||||
**The renderer is vendored, not written and not fetched.** `marked` (MIT, no
|
||||
dependencies, ships an ES module) is pinned and committed under
|
||||
`frontend/js/vendor/`. A CDN is not an option: the application is deployed to a network
|
||||
whose outbound access is not assumed, and `default-src 'self'` forbids it.
|
||||
|
||||
**Diagrams are mermaid, rendered client-side, with the script boundary intact.** The
|
||||
claim that mermaid requires `'unsafe-eval'` was tested rather than believed: mermaid 11's
|
||||
bundle contains no `eval(` and no `new Function` — only a lodash `Function("return this")`
|
||||
global-detection fallback that short-circuits on `globalThis` and never executes.
|
||||
Rendered under this application's exact CSP, a flowchart produced a 15.8 KB SVG and
|
||||
raised **no `script-src` violation**. What it does raise is `style-src`: mermaid styles
|
||||
its output with an injected `<style>` element and `style=` attributes.
|
||||
|
||||
So `script-src 'self'` stays exactly as US07-02 and US08-01 left it, and `style-src`
|
||||
gains `'unsafe-inline'`. That relaxation is bounded on purpose: with `script-src` intact
|
||||
no injected markup can execute, and with `img-src`, `connect-src`, and `font-src` all
|
||||
still `'self'`, the CSS-based exfiltration channels stay closed. What remains is
|
||||
defacement of a page the operator is already authenticated on.
|
||||
|
||||
The rejected alternative is recorded because it may become the better trade later:
|
||||
pre-rendering each diagram to a committed SVG with the already-installed Playwright
|
||||
(no Node toolchain needed) and serving that, which would leave CSP untouched entirely at
|
||||
the cost of a generator and a freshness check. If `style-src 'unsafe-inline'` is ever
|
||||
judged too much, that is the migration, and it does not change a single markdown file.
|
||||
|
||||
**Screenshots are produced, not pasted.** Every screenshot in the user manual is captured
|
||||
by Playwright from the real application against a temporary fixture library, by the same
|
||||
kind of code that already drives the browser suites. A screenshot nobody can regenerate
|
||||
is a screenshot that silently stops being true.
|
||||
|
||||
**The documentation is held to the code by tests.** Configuration keys, CLI commands,
|
||||
exit codes, API error codes, job and journal states, and module names all appear in both
|
||||
the code and the manuals. Each of those is cross-checked, so the failure mode of stale
|
||||
documentation is a red test rather than a misled operator.
|
||||
|
||||
## Stories
|
||||
|
||||
1. [US09-01 — Serve the documentation inside the application](stories/US09-01-docs-in-app.md)
|
||||
2. [US09-02 — Write the installation and operations manual](stories/US09-02-installation-manual.md)
|
||||
3. [US09-03 — Write the architecture overview](stories/US09-03-architecture-overview.md)
|
||||
4. [US09-04 — Write the user manual with generated screenshots](stories/US09-04-user-manual.md)
|
||||
5. [US09-05 — Automate documentation acceptance](stories/US09-05-docs-gate.md)
|
||||
|
||||
## Epic outcome
|
||||
|
||||
A deployed instance serves its own installation manual, architecture overview, and
|
||||
illustrated user manual at `/app/#/docs`, rendered from the same markdown files that are
|
||||
readable in the repository. Screenshots are regenerated from the running application,
|
||||
every documented command, setting, state, and error code is cross-checked against the
|
||||
code that implements it, and one gate fails the build when documentation and application
|
||||
disagree.
|
||||
@@ -2,13 +2,14 @@
|
||||
|
||||
This backlog decomposes the phases in
|
||||
[`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md) into seven
|
||||
epics and small, independently verifiable user stories, plus one delivery-format
|
||||
epic (E08) that packages the released application as a deployable container.
|
||||
epics and small, independently verifiable user stories, plus two delivery-format
|
||||
epics: E08 packages the released application as a deployable container, and E09
|
||||
documents it for the people who install, operate, and use it.
|
||||
|
||||
## Numbering and file naming
|
||||
|
||||
- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` has no
|
||||
concept phase and must not change product scope.
|
||||
- Epics: `E01` through `E07`, matching concept Phases A through G; `E08` and `E09`
|
||||
have no concept phase and must not change product scope.
|
||||
- Stories: `US<epic>-<sequence>`, for example `US03-02`.
|
||||
- Epic files: `E01-<slug>.md`.
|
||||
- Story files: `stories/US01-01-<slug>.md`.
|
||||
@@ -39,6 +40,7 @@ epic (E08) that packages the released application as a deployable container.
|
||||
6. [E06 — Archive lifecycle](E06-archive-lifecycle.md)
|
||||
7. [E07 — Hardening and release](E07-hardening-release.md)
|
||||
8. [E08 — Container deployment](E08-container-deployment.md)
|
||||
9. [E09 — Product documentation](E09-documentation.md)
|
||||
|
||||
## Shared definition of done
|
||||
|
||||
|
||||
49
delivery_backlog/stories/US09-01-docs-in-app.md
Normal file
49
delivery_backlog/stories/US09-01-docs-in-app.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# US09-01 — Serve the Documentation Inside the Application
|
||||
|
||||
Epic: [E09](../E09-documentation.md)
|
||||
|
||||
As an operator who was handed a URL and an access secret, I want the manuals inside the
|
||||
application I am looking at, so that understanding it does not require a repository
|
||||
checkout or an internet connection.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Documentation lives as markdown under `docs/`, with an index that names every page and
|
||||
the order it is meant to be read in. The same files render on Gitea without
|
||||
modification.
|
||||
- The application serves a `/docs` view reachable from the main navigation, listing the
|
||||
pages and rendering the selected one.
|
||||
- Rendering uses a vendored, version-pinned markdown library committed under
|
||||
`frontend/js/vendor/`. It is not fetched from a CDN, not bundled by a build step, and
|
||||
not written by hand. The build records the version and a checksum of the vendored file.
|
||||
- Diagrams written as ```mermaid``` fenced blocks render as diagrams. Mermaid is vendored
|
||||
the same way.
|
||||
- `script-src` in the Content-Security-Policy is unchanged: no `'unsafe-eval'`, no
|
||||
`'unsafe-inline'`. Only `style-src` gains `'unsafe-inline'`, and the reason is recorded
|
||||
where the policy is defined.
|
||||
- Links between documents work in the app: a relative `../foo.md#section` link navigates
|
||||
to that page and heading rather than downloading a file or leaving the application.
|
||||
Every heading has a stable anchor, and a deep link to an anchor scrolls to it.
|
||||
- An unknown page renders a documentation-specific not-found message with a link back to
|
||||
the index, and never exposes a filesystem path.
|
||||
- Documentation is readable without an access secret **or** behind the same session as
|
||||
the rest of the application — whichever is chosen is stated explicitly in the story's
|
||||
implementation notes and covered by a test, because an operator locked out by a
|
||||
configuration mistake is exactly who needs the troubleshooting page.
|
||||
- The view works with no outbound network access at all.
|
||||
|
||||
## Automated tests
|
||||
|
||||
- Unit: heading-anchor slugs (duplicates, punctuation, non-ASCII), and the rewriting of
|
||||
relative markdown links into in-app routes.
|
||||
- Integration: every markdown file under `docs/` is reachable from the index; every
|
||||
internal link in every document resolves to a file and, where an anchor is given, to a
|
||||
heading that exists; the vendored library files match their recorded checksums.
|
||||
- Browser: the documentation view renders a page, follows an internal link, deep-links to
|
||||
an anchor, renders a mermaid diagram to an SVG, and shows the not-found message for an
|
||||
unknown page — all with **zero console errors and zero CSP violations**, asserted, not
|
||||
eyeballed.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- None beyond the delivered application.
|
||||
46
delivery_backlog/stories/US09-02-installation-manual.md
Normal file
46
delivery_backlog/stories/US09-02-installation-manual.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# US09-02 — Write the Installation and Operations Manual
|
||||
|
||||
Epic: [E09](../E09-documentation.md)
|
||||
|
||||
As somebody installing this application for the first time, I want one manual that takes
|
||||
me from nothing to a running instance pointed at my photo library, so that I do not have
|
||||
to reconstruct the procedure from the README, the compose file, and the test suite.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A **host installation** path: prerequisites and their versions (Python, exiftool,
|
||||
immich-go, Playwright's browser for the test suite), virtual environment, dependency
|
||||
install, `.env`, database migration, starting `serve` and `worker`, and how to verify
|
||||
the install succeeded.
|
||||
- A **container installation** path: the published image, the compose file, the data
|
||||
volume, the library bind mount and why it is mounted the way it is, published ports,
|
||||
running behind a reverse proxy with `PHOTO_PIPELINE_ALLOWED_HOSTS` and
|
||||
`PHOTO_PIPELINE_TRUSTED_PROXIES`, and the Portainer/webhook deployment already in use.
|
||||
- A **configuration reference**: every `PHOTO_PIPELINE_*` setting with its meaning,
|
||||
default, accepted values, and whether it is a secret. Secrets are described, never
|
||||
exemplified with a real-looking value.
|
||||
- A **first run checklist** that ends in a verified state: library discovered, worker
|
||||
claiming jobs, readiness endpoint green, diagnostics clean.
|
||||
- **Operations**: upgrading (including what the migration backup does), backup, verify,
|
||||
restore, pruning, diagnostics, and the log locations for each role.
|
||||
- **Troubleshooting**: each refusal an operator can hit at startup — every non-zero exit
|
||||
code of the CLI, the trust-boundary refusal, the unmounted library root refusal, the
|
||||
library lock being held, and a legacy CLI still running — with the cause and the fix.
|
||||
- The manual states the safety invariants an installer must not work around: one worker
|
||||
writes, `_IGNORE/` is never read, EXIF is verified before upload, and the library lock
|
||||
is authoritative.
|
||||
|
||||
## Automated tests
|
||||
|
||||
- Every `PHOTO_PIPELINE_*` setting documented exists as a field on `Config`, and every
|
||||
field on `Config` is documented — both directions, so a new setting cannot ship
|
||||
undocumented.
|
||||
- Every CLI command and flag named in the manual exists in the argument parser, and every
|
||||
subcommand the parser accepts appears in the manual.
|
||||
- Every exit code documented is one the CLI can actually return.
|
||||
- No documented example value collides with a real secret pattern (the repository's own
|
||||
credential scanner runs over `docs/`).
|
||||
|
||||
## Dependencies
|
||||
|
||||
- US09-01
|
||||
47
delivery_backlog/stories/US09-03-architecture-overview.md
Normal file
47
delivery_backlog/stories/US09-03-architecture-overview.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# US09-03 — Write the Architecture Overview
|
||||
|
||||
Epic: [E09](../E09-documentation.md)
|
||||
|
||||
As a developer or reviewer new to this repository, I want an architecture overview that
|
||||
explains what the pieces are and which rules they enforce, so that I can find the right
|
||||
module and not violate an invariant I never knew existed.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A **context diagram**: the operator, the browser application, the API and worker, the
|
||||
photo library, the Immich server, the vision provider, and the archive destination —
|
||||
with the direction and nature of every interaction, including which ones leave the
|
||||
machine.
|
||||
- A **runtime diagram**: the migrate/api/worker composition, the data volume, the library
|
||||
bind mount, the SQLite database, and the process lock — showing what is shared and what
|
||||
is exclusive.
|
||||
- A **module map**: every package under `photo_pipeline/` with its responsibility and the
|
||||
boundary it must not cross (which modules may touch the filesystem, which may call an
|
||||
external provider, which own schema).
|
||||
- **Key flows**, each as a diagram plus prose: the durable job lifecycle from enqueue to
|
||||
recovery; the rename journal state machine including the rollback and manual-resolution
|
||||
paths; the upload lifecycle through preflight, run, report ingestion, and verification.
|
||||
- **The invariants and where they are enforced**, each pointing at the module that owns
|
||||
it: one writer at a time, the library process lock, the path policy and library roots,
|
||||
`_IGNORE/` exclusion, verified EXIF before upload, safety decisions gating the vision
|
||||
provider, and restart-safety of every mutation.
|
||||
- **The data model**: the tables, what identity means for an asset, and how a moved file
|
||||
keeps its identity.
|
||||
- A short **history and donor** section: what was migrated out of the legacy CLIs, where
|
||||
the archive and its ledger live, and why they are kept.
|
||||
- Diagrams are ```mermaid``` blocks so the source is diffable and Gitea renders them
|
||||
natively.
|
||||
|
||||
## Automated tests
|
||||
|
||||
- Every package and top-level module under `photo_pipeline/` appears in the module map,
|
||||
and every module named in the map exists — a new service cannot appear on the map's
|
||||
blind side.
|
||||
- Every job state, journal state, and upload batch state named in the document exists in
|
||||
the code, and every state the code defines is named in the document.
|
||||
- Every mermaid block parses (rendered in the browser test without an error node).
|
||||
- Every code path referenced by file name exists.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- US09-01
|
||||
52
delivery_backlog/stories/US09-04-user-manual.md
Normal file
52
delivery_backlog/stories/US09-04-user-manual.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# US09-04 — Write the User Manual with Generated Screenshots
|
||||
|
||||
Epic: [E09](../E09-documentation.md)
|
||||
|
||||
As the person actually sorting a photo library, I want a manual that walks the workflow
|
||||
screen by screen and tells me what each refusal means, so that I can use the application
|
||||
confidently and know what it is about to do to my files before it does it.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- One page per workflow stage, in the order the application presents them: discovery and
|
||||
inventory, duplicate review, safety review, analysis, album proposals, renames, upload,
|
||||
archive, and diagnostics. Each page answers the same four questions: what this stage is
|
||||
for, what I have to decide, **what it changes on disk or on the server**, and what it
|
||||
refuses to do.
|
||||
- A **guided first pass** that takes a new library from scan to a verified upload, naming
|
||||
the point of no return in each stage and what is reversible after it.
|
||||
- Every stage page carries at least one screenshot of the real application showing the
|
||||
state being described.
|
||||
- Screenshots are **generated** by a committed Playwright script that seeds a temporary
|
||||
fixture library, drives the application, and writes the images. Regenerating them is one
|
||||
documented command. No screenshot is captured by hand.
|
||||
- Screenshots contain no real library paths, no personal photos, and no secrets — the
|
||||
fixture library is synthetic, and this is asserted rather than assumed.
|
||||
- An **error and refusal catalogue**: every `error.code` the API can return, with what
|
||||
causes it, what the application did or refused to do, and what the operator should do
|
||||
next. Refusals that protect data (`lock_held`, `rename_recovery_required`,
|
||||
`stale_preflight`, `path_not_allowed`, `host_not_allowed`, `dry_run_not_approved`,
|
||||
conflict codes) are explained as intentional, not as faults.
|
||||
- A **recovery** page: an interrupted rename, an uncertain upload, a failed migration, a
|
||||
restored backup — what the application does by itself and what needs a decision.
|
||||
- The work-item safety allow list is extended to permit `docs/images/**`, since the
|
||||
repository denies image files by default. This is an explicit, reviewed change, not a
|
||||
quiet one, and it stays narrow enough that a real photo still cannot be committed.
|
||||
|
||||
## Automated tests
|
||||
|
||||
- Every `error.code` the application can emit is documented, and every code documented
|
||||
exists in the code — both directions.
|
||||
- Every image referenced by a documentation page exists, and every image under
|
||||
`docs/images/` is referenced by a page.
|
||||
- The screenshot generator runs end to end in the test environment and produces every
|
||||
image the manual references, against a temporary fixture library that is destroyed
|
||||
afterwards.
|
||||
- Generated screenshots are checked for library paths outside the fixture root and for
|
||||
the configured secrets.
|
||||
- Browser: each stage page renders in the documentation view with its screenshot loaded
|
||||
and no console error.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- US09-01
|
||||
38
delivery_backlog/stories/US09-05-docs-gate.md
Normal file
38
delivery_backlog/stories/US09-05-docs-gate.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# US09-05 — Automate Documentation Acceptance
|
||||
|
||||
Epic: [E09](../E09-documentation.md)
|
||||
|
||||
As a release owner, I want one gate that proves the documentation still describes the
|
||||
application, so that a change to the code cannot quietly make the manuals wrong.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- One documented command runs every documentation check and retains its evidence, in the
|
||||
shape the release and container gates already use.
|
||||
- The gate fails when: an internal link or anchor is dead; a page is unreachable from the
|
||||
index; an image is referenced but missing or present but unreferenced; a documented
|
||||
setting, command, exit code, error code, or state does not exist in the code; a code
|
||||
path exists that the documentation is required to cover and does not.
|
||||
- The gate regenerates the screenshots and fails when a regenerated image no longer
|
||||
matches the committed one beyond a stated tolerance, so a UI change that invalidates the
|
||||
manual is a red build rather than a discovery months later.
|
||||
- The gate renders every documentation page in a real browser and fails on any console
|
||||
error or CSP violation, and asserts that `script-src` contains neither `'unsafe-eval'`
|
||||
nor `'unsafe-inline'`.
|
||||
- The checks run on a `phase_i` marker; CI runs the gate, and the earlier epic suites keep
|
||||
running unchanged.
|
||||
- The README and the application's documentation index point at each other, so neither is
|
||||
the forgotten copy.
|
||||
- Documentation stories are mapped in the story traceability matrix like every other
|
||||
story.
|
||||
|
||||
## Automated tests
|
||||
|
||||
- The gate's own contract is testable without a browser: a seeded broken link, a missing
|
||||
image, an undocumented error code, and a stale screenshot each fail it, and a clean tree
|
||||
passes.
|
||||
- The full documentation suite runs on `phase_i` in CI.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- US09-01 through US09-04
|
||||
34
docs/index.md
Normal file
34
docs/index.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Photo Pipeline documentation
|
||||
|
||||
One local application that takes a photo library from discovery to a verified Immich
|
||||
upload: duplicate detection, safety review, content analysis, album naming, guarded
|
||||
renaming, upload, and archive — one visible, resume-safe workflow.
|
||||
|
||||
These pages are readable three ways, and they are the same files each time: in the
|
||||
repository under `docs/`, on Gitea, and inside the running application under
|
||||
**Docs**. There is no separate copy to fall out of date.
|
||||
|
||||
## Read in this order
|
||||
|
||||
1. [Overview](overview.md) — what the application does, the stages it moves a photo
|
||||
through, and the rules it will not break.
|
||||
|
||||
## Being written
|
||||
|
||||
The remaining manuals are accepted work, not aspiration; each is a story in
|
||||
[E09](https://git.domverse-berlin.eu/domverse/photoanalyzer/src/branch/main/delivery_backlog/E09-documentation.md)
|
||||
and will appear here as it lands.
|
||||
|
||||
- **Installation and operations** — host and container installation, every setting,
|
||||
the first-run checklist, upgrades, backup and restore, and what each refusal at
|
||||
startup means (US09-02).
|
||||
- **Architecture** — context and runtime diagrams, the module map, the job and
|
||||
journal state machines, and where each invariant is enforced (US09-03).
|
||||
- **User manual** — one page per workflow stage with screenshots of the real
|
||||
application, and a catalogue of every error and refusal (US09-04).
|
||||
|
||||
## Conventions
|
||||
|
||||
A page tells you what a stage **changes on disk or on the server** before it tells
|
||||
you how to run it. Refusals are documented as intentional: this application would
|
||||
rather stop and explain than guess about somebody's photographs.
|
||||
77
docs/overview.md
Normal file
77
docs/overview.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Overview
|
||||
|
||||
[← Documentation index](index.md)
|
||||
|
||||
Photo Pipeline sorts a photo library. It finds duplicates before anything expensive
|
||||
happens to them, asks a human which pictures may leave the machine, describes the
|
||||
ones that may, proposes album names from what it found, renames folders under a
|
||||
crash-safe journal, uploads through `immich-go`, and can archive a finished album off
|
||||
active storage without ever forgetting it existed.
|
||||
|
||||
It runs locally. It talks to exactly three things outside itself: a vision provider,
|
||||
an Immich server, and `exiftool` — and it will tell you before it uses any of them.
|
||||
|
||||
## The workflow
|
||||
|
||||
Each stage is a gate, not a tab. A later stage can always be looked at; its actions
|
||||
stay disabled until what they depend on is true.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
I["0 · Inventory<br/>discover, hash, cluster duplicates"] --> S["1 · Safety<br/>score and human decision"]
|
||||
S -->|sfw| A["2 · Analysis<br/>vision provider, EXIF checkpoint"]
|
||||
S -->|nsfw| U
|
||||
A --> B["3 · Albums<br/>proposal, then guarded rename"]
|
||||
B --> U["4 · Upload<br/>immich-go, verified bytes"]
|
||||
U --> R["5 · Archive<br/>copy, verify, then reclaim space"]
|
||||
```
|
||||
|
||||
| Stage | What it decides | What it changes |
|
||||
|---|---|---|
|
||||
| Inventory | which file is the canonical copy of a picture | nothing — it only reads |
|
||||
| Safety | whether a photo may be sent to a cloud provider | one `sfw`/`nsfw` EXIF keyword |
|
||||
| Analysis | what a photo shows | a managed caption segment and additive keywords |
|
||||
| Albums | what a folder should be called | folder names, through a journaled rename |
|
||||
| Upload | which exact bytes reach Immich | nothing locally; assets appear in Immich |
|
||||
| Archive | which album leaves active storage | files move to the archive, after verification |
|
||||
|
||||
## What it will not do
|
||||
|
||||
These are enforced in code, not by convention, and each one is why some action you
|
||||
expected is sometimes refused.
|
||||
|
||||
- **Nothing under `_IGNORE/` is ever read.** Not scanned, not counted, not
|
||||
thumbnailed, not sent anywhere.
|
||||
- **A path is not an identity.** Every picture has a stable id, so moving or renaming
|
||||
it loses no history.
|
||||
- **Only a confirmed-SFW photo reaches the vision provider.** A photo marked NSFW is
|
||||
still uploadable to Immich; it simply never leaves for analysis.
|
||||
- **Metadata is verified, not hoped for.** Every stage that writes EXIF reads it back
|
||||
and proves that what it did not own is unchanged.
|
||||
- **One writer at a time.** A library lock, held by one process, is what makes a
|
||||
crash recoverable instead of ambiguous.
|
||||
- **Nothing irreversible happens without a preview and an explicit approval** that
|
||||
names the exact count.
|
||||
|
||||
## Where things live
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| the photo library | wherever you point `PHOTO_PIPELINE_LIBRARY_ROOTS`; mounted read-write |
|
||||
| the database, cache, logs, backups | the data directory, never inside the library |
|
||||
| secrets | the environment, never the database and never a log line |
|
||||
| these documents | `docs/` in the repository, served at `/docs` by the application |
|
||||
|
||||
## Running it
|
||||
|
||||
The short version, for a host installation:
|
||||
|
||||
```bash
|
||||
python -m photo_pipeline migrate
|
||||
python -m photo_pipeline serve # the API and this browser application
|
||||
python -m photo_pipeline worker # the process that does the long work
|
||||
```
|
||||
|
||||
The full procedure, the container path, and every setting belong to the installation
|
||||
manual, which is being written next — see
|
||||
[the documentation index](index.md#being-written).
|
||||
@@ -170,6 +170,20 @@ a.link:hover { text-decoration: underline; }
|
||||
.grid th, .grid td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); vertical-align: top; }
|
||||
.path { font-family: ui-monospace, monospace; font-size: 0.85em; word-break: break-all; }
|
||||
|
||||
/* Docs: prose, so it gets a reading measure rather than the full window width. */
|
||||
.doc { max-width: 72ch; }
|
||||
.doc h1 { margin-top: 0; }
|
||||
.doc h2, .doc h3 { margin-top: 1.6em; border-bottom: 1px solid var(--border); padding-bottom: 4px; }
|
||||
.doc code { font-family: ui-monospace, monospace; font-size: 0.9em; background: var(--surface-2); padding: 1px 4px; border-radius: 4px; }
|
||||
.doc pre { background: var(--surface-2); border: 1px solid var(--border); border-radius: var(--radius); padding: 12px; overflow: auto; }
|
||||
.doc pre code { background: none; padding: 0; }
|
||||
.doc table { border-collapse: collapse; width: 100%; }
|
||||
.doc th, .doc td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); vertical-align: top; }
|
||||
.doc blockquote { margin: 1em 0; padding-left: 12px; border-left: 3px solid var(--border); color: var(--muted); }
|
||||
.doc img { max-width: 100%; }
|
||||
.diagram { margin: 1.5em 0; padding: 0; overflow: auto; }
|
||||
.diagram svg { max-width: 100%; height: auto; }
|
||||
|
||||
/* Narrow screens: stack the panes so blockers and actions stay reachable. */
|
||||
@media (max-width: 720px) {
|
||||
.two-pane { grid-template-columns: 1fr; }
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
<a href="#/uploads" data-nav="uploads">Upload</a>
|
||||
<a href="#/archive" data-nav="archive">Archive</a>
|
||||
<a href="#/stats" data-nav="stats">Stats</a>
|
||||
<a href="#/docs" data-nav="docs">Docs</a>
|
||||
</nav>
|
||||
</header>
|
||||
<main id="app" aria-live="polite"><!-- views render here --></main>
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { api } from "./api.js";
|
||||
import { renderArchive, setArchiveRender } from "./archive.js";
|
||||
import { renderDocs } from "./docs.js";
|
||||
import { navigate, onRouteChange, parseHash } from "./router.js";
|
||||
import { renderRenames, setRenamesRender } from "./renames.js";
|
||||
import { renderUploads, setUploadsRender } from "./uploads.js";
|
||||
@@ -385,6 +386,7 @@ function render() {
|
||||
else if (path === "/uploads") renderUploads(root, params);
|
||||
else if (path === "/archive") renderArchive(root, params);
|
||||
else if (path === "/stats") renderStats(root, params);
|
||||
else if (path === "/docs") renderDocs(root, params);
|
||||
else show(errorBanner("Unknown view"));
|
||||
}
|
||||
|
||||
|
||||
263
frontend/js/docs.js
Normal file
263
frontend/js/docs.js
Normal file
@@ -0,0 +1,263 @@
|
||||
// The documentation view (US09-01): the manuals in `docs/`, rendered in the app.
|
||||
//
|
||||
// The same markdown files are the repository's documentation and the deployment's
|
||||
// documentation. An operator who was handed a URL and an access secret has no
|
||||
// checkout in front of them, and the network the application runs on is not assumed
|
||||
// to reach a CDN — so the renderer is vendored and everything here is same-origin.
|
||||
//
|
||||
// Nothing on this page is authenticated. It is served from the static mount beside
|
||||
// the application shell, exactly like `index.html`: the troubleshooting page is
|
||||
// needed most by whoever cannot get past the access secret, and no documentation
|
||||
// file contains anything a session would protect.
|
||||
//
|
||||
// `marked` and `mermaid` are loaded lazily, on the first documentation page and the
|
||||
// first diagram. They are large, and every other view does without them.
|
||||
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||
|
||||
const DOCS_BASE = "/docs";
|
||||
const VENDOR = "/app/js/vendor";
|
||||
const INDEX_PAGE = "index";
|
||||
// A page name comes from the hash, so it is caller input: no scheme, no traversal,
|
||||
// no absolute path. The server would refuse those too; this refuses them earlier and
|
||||
// without a request that looks like an attempt.
|
||||
const PAGE_PATTERN = /^[\w-]+(\/[\w-]+)*$/;
|
||||
|
||||
let markedPromise;
|
||||
let mermaidPromise;
|
||||
|
||||
function loadMarked() {
|
||||
markedPromise ??= import(`${VENDOR}/marked.esm.js`).then((module) => module.marked);
|
||||
return markedPromise;
|
||||
}
|
||||
|
||||
// mermaid ships one large UMD bundle rather than a self-contained ES module, so it
|
||||
// arrives through a script element. `script-src 'self'` allows it because it is ours
|
||||
// and same-origin; nothing here relaxes that.
|
||||
function loadMermaid() {
|
||||
mermaidPromise ??= new Promise((resolve, reject) => {
|
||||
const script = document.createElement("script");
|
||||
script.src = `${VENDOR}/mermaid.min.js`;
|
||||
script.onload = () => resolve(window.mermaid);
|
||||
script.onerror = () => reject(new Error("the diagram renderer could not be loaded"));
|
||||
document.head.appendChild(script);
|
||||
});
|
||||
return mermaidPromise;
|
||||
}
|
||||
|
||||
// ── the pure parts, unit-tested in js/tests/unit.js ──────────────────────────
|
||||
|
||||
/** A stable, readable heading anchor. Letters and digits of any script survive. */
|
||||
export function slug(text) {
|
||||
const cleaned = String(text)
|
||||
.toLowerCase()
|
||||
.trim()
|
||||
.replace(/[^\p{L}\p{N}\s-]/gu, "")
|
||||
.replace(/[\s-]+/g, "-")
|
||||
.replace(/^-|-$/g, "");
|
||||
return cleaned || "section";
|
||||
}
|
||||
|
||||
/** Assign every heading an id, disambiguating repeats the way a reader would
|
||||
* expect: the first `Notes` keeps `#notes`, the second becomes `#notes-2`. */
|
||||
export function assignHeadingIds(headings) {
|
||||
const used = new Map();
|
||||
for (const heading of headings) {
|
||||
const base = slug(heading.textContent);
|
||||
const seen = (used.get(base) || 0) + 1;
|
||||
used.set(base, seen);
|
||||
heading.id = seen === 1 ? base : `${base}-${seen}`;
|
||||
}
|
||||
return headings;
|
||||
}
|
||||
|
||||
/**
|
||||
* Where a link inside a documentation page should go.
|
||||
*
|
||||
* Markdown links between documents are relative file paths, which a browser would
|
||||
* treat as downloads that leave the application. Returns the in-app target for a
|
||||
* link to another document or to a heading, and `null` for anything else — external
|
||||
* links stay exactly as the author wrote them.
|
||||
*/
|
||||
export function resolveDocLink(currentPage, href) {
|
||||
if (!href) return null;
|
||||
if (/^[a-z][a-z\d+.-]*:/i.test(href) || href.startsWith("//")) return null;
|
||||
if (href.startsWith("#")) return { page: currentPage, anchor: href.slice(1) };
|
||||
const [target, anchor = ""] = href.split("#");
|
||||
if (!/\.md$/i.test(target)) return null;
|
||||
// Resolved against the current document, so `../guides/x.md` means what it means
|
||||
// in the repository. The origin is a placeholder; only the path is used.
|
||||
const resolved = new URL(target, `https://docs.invalid/${currentPage}.md`);
|
||||
const page = decodeURIComponent(resolved.pathname).replace(/^\//, "").replace(/\.md$/i, "");
|
||||
return PAGE_PATTERN.test(page) ? { page, anchor } : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Point every in-app link at its route, and leave every other link alone.
|
||||
*
|
||||
* The resolved page is kept on the element, so the reading order can be read back
|
||||
* from the rendered index rather than parsed out of the markdown a second time.
|
||||
*/
|
||||
export function rewriteLinks(article, page) {
|
||||
for (const link of article.querySelectorAll("a[href]")) {
|
||||
const href = link.getAttribute("href");
|
||||
const target = resolveDocLink(page, href);
|
||||
if (target) {
|
||||
link.setAttribute("href", docHash(target.page, target.anchor));
|
||||
link.dataset.docPage = target.page;
|
||||
} else if (/^https?:/i.test(href)) {
|
||||
link.setAttribute("rel", "noreferrer noopener");
|
||||
link.setAttribute("target", "_blank");
|
||||
}
|
||||
}
|
||||
return article;
|
||||
}
|
||||
|
||||
/** The reading order, taken from the index document itself — one list, in one
|
||||
* place, that is equally the index on Gitea and the navigation here. */
|
||||
export function documentIndex(indexElement) {
|
||||
const pages = [];
|
||||
for (const link of indexElement.querySelectorAll("a[data-doc-page]")) {
|
||||
const page = link.dataset.docPage;
|
||||
if (page !== INDEX_PAGE && !pages.some((entry) => entry.page === page)) {
|
||||
pages.push({ page, title: link.textContent.trim() });
|
||||
}
|
||||
}
|
||||
return pages;
|
||||
}
|
||||
|
||||
// ── the view ─────────────────────────────────────────────────────────────────
|
||||
|
||||
async function fetchPage(page) {
|
||||
const response = await fetch(`${DOCS_BASE}/${page}.md`, { headers: { accept: "text/markdown" } });
|
||||
if (!response.ok) throw new Error(`documentation page unavailable: ${response.status}`);
|
||||
return response.text();
|
||||
}
|
||||
|
||||
async function toArticle(markdown, page) {
|
||||
const marked = await loadMarked();
|
||||
const article = el("article", { class: "doc", "data-testid": "doc", "data-page": page });
|
||||
// The only innerHTML in the application, and deliberate: rendering markdown *is*
|
||||
// producing HTML. The input is a file from this repository, and the page's CSP
|
||||
// (`script-src 'self'`, no `'unsafe-inline'`) means injected script and inline
|
||||
// handlers do not run even if one ever were not.
|
||||
article.innerHTML = marked.parse(markdown, { async: false });
|
||||
assignHeadingIds(article.querySelectorAll("h1, h2, h3, h4, h5, h6"));
|
||||
rewriteLinks(article, page);
|
||||
return article;
|
||||
}
|
||||
|
||||
function docHash(page, anchor = "") {
|
||||
const query = new URLSearchParams(anchor ? { page, anchor } : { page });
|
||||
return `#/docs?${query}`;
|
||||
}
|
||||
|
||||
/** Diagrams are authored as ```mermaid blocks so the source is diffable and Gitea
|
||||
* renders them natively. A failure here replaces the diagram, never the page. */
|
||||
async function renderDiagrams(article) {
|
||||
const blocks = [...article.querySelectorAll("pre > code.language-mermaid")];
|
||||
if (!blocks.length) return;
|
||||
let mermaid;
|
||||
try {
|
||||
mermaid = await loadMermaid();
|
||||
mermaid.initialize({ startOnLoad: false, securityLevel: "strict", theme: "dark" });
|
||||
} catch (error) {
|
||||
blocks.forEach((block) => block.closest("pre").replaceWith(errorBanner(error.message)));
|
||||
return;
|
||||
}
|
||||
for (const [index, block] of blocks.entries()) {
|
||||
const figure = el("figure", { class: "diagram", "data-testid": "diagram" });
|
||||
block.closest("pre").replaceWith(figure);
|
||||
try {
|
||||
const { svg } = await mermaid.render(`diagram-${index}-${Date.now()}`, block.textContent);
|
||||
figure.innerHTML = svg;
|
||||
} catch (error) {
|
||||
figure.replaceWith(errorBanner(`Diagram could not be drawn: ${error.message}`));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function notFound() {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "doc-not-found" },
|
||||
"That documentation page does not exist. ",
|
||||
el("a", { class: "link", href: docHash(INDEX_PAGE) }, "Back to the documentation index")
|
||||
);
|
||||
}
|
||||
|
||||
function sidebar(pages, current) {
|
||||
return el(
|
||||
"nav",
|
||||
{ class: "card", "aria-label": "Documentation" },
|
||||
el("h2", {}, "Documentation"),
|
||||
el(
|
||||
"ul",
|
||||
{ class: "album-list", "data-testid": "doc-pages" },
|
||||
el(
|
||||
"li",
|
||||
{},
|
||||
el(
|
||||
"a",
|
||||
{
|
||||
href: docHash(INDEX_PAGE),
|
||||
"aria-current": current === INDEX_PAGE ? "true" : false,
|
||||
},
|
||||
"Index"
|
||||
)
|
||||
),
|
||||
...pages.map(({ page, title }) =>
|
||||
el(
|
||||
"li",
|
||||
{},
|
||||
el(
|
||||
"a",
|
||||
{
|
||||
href: docHash(page),
|
||||
"data-page": page,
|
||||
"aria-current": page === current ? "true" : false,
|
||||
},
|
||||
title
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
export async function renderDocs(root, params = {}) {
|
||||
setActiveNav("docs");
|
||||
const requested = params.page || INDEX_PAGE;
|
||||
const page = PAGE_PATTERN.test(requested) ? requested : "";
|
||||
|
||||
let indexArticle;
|
||||
try {
|
||||
indexArticle = await toArticle(await fetchPage(INDEX_PAGE), INDEX_PAGE);
|
||||
} catch (error) {
|
||||
root.replaceChildren(errorBanner(`Documentation is unavailable: ${error.message}`));
|
||||
return;
|
||||
}
|
||||
const pages = documentIndex(indexArticle);
|
||||
|
||||
let article;
|
||||
if (!page) article = notFound();
|
||||
else if (page === INDEX_PAGE) article = indexArticle;
|
||||
else {
|
||||
try {
|
||||
article = await toArticle(await fetchPage(page), page);
|
||||
} catch {
|
||||
article = notFound();
|
||||
}
|
||||
}
|
||||
|
||||
root.replaceChildren(el("div", { class: "two-pane" }, sidebar(pages, page), article));
|
||||
await renderDiagrams(article);
|
||||
scrollToAnchor(params.anchor);
|
||||
}
|
||||
|
||||
// A documentation anchor cannot live in the hash — the hash is the route — so it
|
||||
// travels as a parameter and is applied after the page renders.
|
||||
function scrollToAnchor(anchor) {
|
||||
if (!anchor) return;
|
||||
const target = document.getElementById(anchor);
|
||||
if (target) target.scrollIntoView({ block: "start" });
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import { createStore } from "../store.js";
|
||||
import { parseHash, navigate } from "../router.js";
|
||||
import { api, cancellable } from "../api.js";
|
||||
import { subscribeJob } from "../events.js";
|
||||
import { assignHeadingIds, documentIndex, resolveDocLink, rewriteLinks, slug } from "../docs.js";
|
||||
|
||||
const cases = [];
|
||||
function ok(name, cond) {
|
||||
@@ -142,6 +143,62 @@ async function run() {
|
||||
window.EventSource = realES;
|
||||
}
|
||||
|
||||
// ── documentation view (US09-01) ─────────────────────────────────────────
|
||||
{
|
||||
ok("slug lowercases and dashes a heading", slug("Key Flows") === "key-flows");
|
||||
ok(
|
||||
"slug drops punctuation but keeps words",
|
||||
slug("What it *will not* do:") === "what-it-will-not-do"
|
||||
);
|
||||
ok("slug keeps non-ASCII letters", slug("Größe & Gewicht") === "größe-gewicht");
|
||||
ok("slug never yields an empty anchor", slug("!!!") === "section");
|
||||
|
||||
const doc = document.createElement("div");
|
||||
doc.innerHTML = "<h2>Notes</h2><h3>Notes</h3><h2>Notes</h2>";
|
||||
const ids = [...assignHeadingIds(doc.querySelectorAll("h2, h3"))].map((h) => h.id);
|
||||
ok("repeated headings get distinct anchors", ids.join(",") === "notes,notes-2,notes-3");
|
||||
|
||||
ok(
|
||||
"a sibling document link becomes an in-app route",
|
||||
resolveDocLink("index", "overview.md")?.page === "overview"
|
||||
);
|
||||
const nested = resolveDocLink("guides/install", "../overview.md#running-it");
|
||||
ok("a relative link resolves against the current page", nested?.page === "overview");
|
||||
ok("a link's anchor survives the rewrite", nested?.anchor === "running-it");
|
||||
ok(
|
||||
"a bare anchor stays on the current page",
|
||||
resolveDocLink("overview", "#the-workflow")?.page === "overview"
|
||||
);
|
||||
ok("an external link is left alone", resolveDocLink("index", "https://example.test/x") === null);
|
||||
ok("a non-markdown relative link is left alone", resolveDocLink("index", "images/a.png") === null);
|
||||
// A climb cannot leave the documentation tree: it is clamped at the root, so the
|
||||
// page it names is still fetched from under /docs and simply does not exist.
|
||||
ok(
|
||||
"a link that climbs above the docs root is clamped",
|
||||
resolveDocLink("index", "../../etc/passwd.md")?.page === "etc/passwd"
|
||||
);
|
||||
ok(
|
||||
"a page name that is not a page name is refused",
|
||||
resolveDocLink("index", "..%2f..%2fetc%2fpasswd.md") === null
|
||||
);
|
||||
|
||||
const index = document.createElement("div");
|
||||
index.innerHTML =
|
||||
'<a href="overview.md">Overview</a><a href="overview.md">again</a>' +
|
||||
'<a href="index.md">itself</a><a href="https://example.test">out</a>';
|
||||
const listed = documentIndex(rewriteLinks(index, "index"));
|
||||
ok(
|
||||
"an external link is marked safe to open away from the app",
|
||||
index.querySelector('a[href^="https"]').rel === "noreferrer noopener"
|
||||
);
|
||||
ok(
|
||||
"an in-app link points at the docs route",
|
||||
index.querySelector("a").getAttribute("href") === "#/docs?page=overview"
|
||||
);
|
||||
ok("the index lists each page once, in order", listed.length === 1);
|
||||
ok("the index takes its titles from the link text", listed[0].title === "Overview");
|
||||
}
|
||||
|
||||
await tick();
|
||||
const failed = cases.filter((c) => !c.ok);
|
||||
window.__RESULTS__ = { passed: cases.length - failed.length, failed: failed.length, cases };
|
||||
|
||||
23
frontend/js/vendor/VERSIONS.json
vendored
Normal file
23
frontend/js/vendor/VERSIONS.json
vendored
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"_comment": "Vendored browser libraries (US09-01). Committed rather than fetched: the application is deployed to a network whose outbound access is not assumed, and `default-src 'self'` forbids a CDN. Checksums are asserted by tests/integration/test_documentation.py, so replacing a file without recording it here fails the suite. Update by downloading the pinned URL and recording the new version and sha256 in the same commit.",
|
||||
"libraries": [
|
||||
{
|
||||
"file": "marked.esm.js",
|
||||
"name": "marked",
|
||||
"version": "18.0.10",
|
||||
"license": "MIT",
|
||||
"url": "https://cdn.jsdelivr.net/npm/marked@18.0.10/lib/marked.esm.js",
|
||||
"sha256": "4cf47dfebb7f614a08fc0a579ab0fe407ff0ed2b717bf953040c85b2f493a4f0",
|
||||
"why": "Markdown to HTML for the documentation view. An ES module with no dependencies, imported lazily by js/docs.js."
|
||||
},
|
||||
{
|
||||
"file": "mermaid.min.js",
|
||||
"name": "mermaid",
|
||||
"version": "11.17.0",
|
||||
"license": "MIT",
|
||||
"url": "https://cdn.jsdelivr.net/npm/mermaid@11.17.0/dist/mermaid.min.js",
|
||||
"sha256": "8d8e0eec56d3a83b4b3c87f42050845546dee93ebe1875d2117c12e6947c0cb3",
|
||||
"why": "Renders ```mermaid blocks so diagrams are diffable source that Gitea also renders. The single-file UMD build rather than the ES module entry, whose ~40 lazy chunks would each need vendoring and pinning."
|
||||
}
|
||||
]
|
||||
}
|
||||
78
frontend/js/vendor/marked.esm.js
vendored
Normal file
78
frontend/js/vendor/marked.esm.js
vendored
Normal file
File diff suppressed because one or more lines are too long
3636
frontend/js/vendor/mermaid.min.js
vendored
Normal file
3636
frontend/js/vendor/mermaid.min.js
vendored
Normal file
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
|
||||
"""Application management CLI:
|
||||
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics}``.
|
||||
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,dry-run,approve-dry-run}``.
|
||||
|
||||
``serve`` and ``worker`` take the library process lock for their role (US07-05):
|
||||
two workers, or the frozen CLI running beside the app, would each be safe on their
|
||||
@@ -70,6 +70,14 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
"release-gate", help="Run every suite in an isolated stack and keep the evidence"
|
||||
)
|
||||
gate_cmd.add_argument("--output", help="Evidence directory (default: data/release/<stamp>)")
|
||||
container_cmd = commands.add_parser(
|
||||
"container-gate",
|
||||
help="Provision the composition from the built image, run the phase_h "
|
||||
"acceptance suite against it, destroy it, and keep the evidence (US08-05)",
|
||||
)
|
||||
container_cmd.add_argument(
|
||||
"--output", help="Evidence directory (default: data/container-gate/<stamp>)"
|
||||
)
|
||||
dry_cmd = commands.add_parser(
|
||||
"dry-run", help="Read-only reconciliation of the configured library (US07-07)"
|
||||
)
|
||||
@@ -155,6 +163,31 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
)
|
||||
return 0 if report["ok"] else 1
|
||||
|
||||
if args.command == "container-gate":
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from photo_pipeline.services import release
|
||||
|
||||
# The suite provisions and destroys the composition itself; what this command
|
||||
# adds is the single entry point and the retained evidence. No skip is an
|
||||
# environment limit here: a gate that did not reach the containers proved
|
||||
# nothing about them.
|
||||
output = args.output or Path(config.data_dir) / "container-gate" / datetime.now(
|
||||
timezone.utc
|
||||
).strftime("%Y%m%dT%H%M%SZ")
|
||||
report = release.run_gate(
|
||||
config,
|
||||
output=output,
|
||||
stages=release.CONTAINER_STAGES,
|
||||
allowed_skips=release.CONTAINER_ALLOWED_SKIP_REASONS,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{k: v for k, v in report.items() if k not in ("stages", "matrix")}, indent=2
|
||||
)
|
||||
)
|
||||
return 0 if report["ok"] else 1
|
||||
|
||||
if args.command == "dry-run":
|
||||
from photo_pipeline.services import release
|
||||
|
||||
|
||||
@@ -53,6 +53,7 @@ from photo_pipeline.services.thumbnails import ThumbnailService
|
||||
from photo_pipeline.services.upload_batches import UploadBatchService
|
||||
|
||||
FRONTEND_DIR = Path(__file__).resolve().parents[2] / "frontend"
|
||||
DOCS_DIR = Path(__file__).resolve().parents[2] / "docs"
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -151,4 +152,10 @@ def create_app(config: Config | None = None) -> FastAPI:
|
||||
# Static single-page app (hash-routed). Mounted last so /api/v1 wins.
|
||||
if FRONTEND_DIR.is_dir():
|
||||
app.mount("/app", StaticFiles(directory=FRONTEND_DIR, html=True), name="app")
|
||||
# The manuals, as their own markdown files (US09-01). Unauthenticated for the
|
||||
# same reason the application shell is: whoever cannot get past the access
|
||||
# secret is exactly who needs the troubleshooting page, and no document holds
|
||||
# anything a session would protect.
|
||||
if DOCS_DIR.is_dir():
|
||||
app.mount("/docs", StaticFiles(directory=DOCS_DIR), name="docs")
|
||||
return app
|
||||
|
||||
@@ -57,9 +57,21 @@ LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "::1", "[::1]"})
|
||||
# backup a careful operator takes first, and its retention (US07-07).
|
||||
MUTATION_EXEMPT_PATHS = frozenset({f"{API_PREFIX}/backups", f"{API_PREFIX}/backups/prune"})
|
||||
|
||||
# Applied to every response. No inline script/style is used by the frontend, so the
|
||||
# policy can stay strict; `frame-ancestors 'none'` and CORP keep other pages from
|
||||
# Applied to every response. `frame-ancestors 'none'` and CORP keep other pages from
|
||||
# embedding the app or its thumbnails.
|
||||
#
|
||||
# `script-src 'self'` is the boundary that matters and it is unchanged: no
|
||||
# `'unsafe-eval'`, no `'unsafe-inline'`, so nothing injected into the DOM can execute.
|
||||
# Whether the vendored diagram renderer needed `'unsafe-eval'` was measured rather
|
||||
# than assumed — mermaid's bundle contains no `eval(` and no `new Function`, and it
|
||||
# renders under this exact policy without a single script-src violation.
|
||||
#
|
||||
# `style-src` does gain `'unsafe-inline'` (US09-01): mermaid styles the SVG it builds
|
||||
# with an injected `<style>` element and `style=` attributes, and a diagram's CSS
|
||||
# cannot be hashed in advance. The concession is bounded by the directives around it
|
||||
# — with script execution still refused and `img-src`, `connect-src`, and `font-src`
|
||||
# all `'self'`, the CSS exfiltration channels stay closed and what is left is
|
||||
# defacement of a page its own operator is already looking at.
|
||||
DEFAULT_HEADERS = {
|
||||
"x-content-type-options": "nosniff",
|
||||
"x-frame-options": "DENY",
|
||||
@@ -67,8 +79,9 @@ DEFAULT_HEADERS = {
|
||||
"cross-origin-resource-policy": "same-origin",
|
||||
"cross-origin-opener-policy": "same-origin",
|
||||
"content-security-policy": (
|
||||
"default-src 'self'; img-src 'self' data:; style-src 'self'; script-src 'self'; "
|
||||
"connect-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
|
||||
"default-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; "
|
||||
"script-src 'self'; connect-src 'self'; font-src 'self'; "
|
||||
"frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -60,6 +60,17 @@ ALLOWED_SKIP_REASONS = (
|
||||
"bind-mount ownership is virtualised",
|
||||
)
|
||||
|
||||
# The container acceptance gate (US08-05): the deployed application, verified the way
|
||||
# the host application is. Deliberately one stage selected by marker, so adding a
|
||||
# phase_h test is enough to put it in front of a deploy.
|
||||
CONTAINER_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("container", ("tests/e2e", "-m", "phase_h")),
|
||||
)
|
||||
|
||||
# Nothing. This gate exists to prove the *deployed* runtime, and every reason a check
|
||||
# would skip here — no daemon, no compose, no browser — means it was not proven.
|
||||
CONTAINER_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
|
||||
|
||||
|
||||
class ReleaseError(RuntimeError):
|
||||
pass
|
||||
@@ -175,13 +186,15 @@ def _run_stage(label: str, paths: tuple[str, ...], *, repo: Path, log_dir: Path)
|
||||
return StageResult(label, command, result.returncode, elapsed, summary, skipped)
|
||||
|
||||
|
||||
def unexpected_skips(results: list[StageResult]) -> list[str]:
|
||||
def unexpected_skips(
|
||||
results: list[StageResult], allowed: tuple[str, ...] = ALLOWED_SKIP_REASONS
|
||||
) -> list[str]:
|
||||
"""Skips the gate will not accept: everything but the documented environment ones."""
|
||||
return [
|
||||
line
|
||||
for result in results
|
||||
for line in result.skipped
|
||||
if not any(reason in line for reason in ALLOWED_SKIP_REASONS)
|
||||
if not any(reason in line for reason in allowed)
|
||||
]
|
||||
|
||||
|
||||
@@ -190,6 +203,7 @@ def run_gate(
|
||||
*,
|
||||
output: Path | str | None = None,
|
||||
stages: tuple[tuple[str, tuple[str, ...]], ...] = STAGES,
|
||||
allowed_skips: tuple[str, ...] = ALLOWED_SKIP_REASONS,
|
||||
repo: Path | None = None,
|
||||
) -> dict:
|
||||
"""Run every suite in an isolated stack and retain checksummed evidence.
|
||||
@@ -206,13 +220,14 @@ def run_gate(
|
||||
logs = directory / "logs"
|
||||
logs.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
started_at = _now()
|
||||
matrix = story_matrix(repo)
|
||||
results = [_run_stage(label, paths, repo=repo, log_dir=logs) for label, paths in stages]
|
||||
skips = unexpected_skips(results)
|
||||
skips = unexpected_skips(results, allowed_skips)
|
||||
|
||||
report = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"started_at": _now().isoformat(),
|
||||
"started_at": started_at.isoformat(),
|
||||
"revision": revision(),
|
||||
"python": sys.version.split()[0],
|
||||
"platform": os.uname().sysname,
|
||||
|
||||
@@ -53,4 +53,5 @@ markers = [
|
||||
"phase_e: Phase E end-to-end acceptance (US05-06) — upload preflight, uploader, and browser journeys",
|
||||
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
|
||||
"container: builds and runs the container image and its composition (US08-02, US08-03) — needs a Docker daemon, the compose plugin, and the network",
|
||||
"phase_h: Phase H container deployment acceptance (US08-05) — the browser, upgrade, restart, and security journeys against the composed stack",
|
||||
]
|
||||
|
||||
260
tests/e2e/_container_harness.py
Normal file
260
tests/e2e/_container_harness.py
Normal file
@@ -0,0 +1,260 @@
|
||||
"""The composed stack, as a fixture: build, provision, drive, destroy (US08-03/US08-05).
|
||||
|
||||
Extracted from ``tests/e2e/test_compose_stack.py`` when the container acceptance gate
|
||||
needed the same stack under a different image, project, and library. One
|
||||
implementation, because two would drift on exactly the details that make a container
|
||||
test worth anything — the mount, the volume, the ports, and the teardown.
|
||||
|
||||
Nothing here fakes anything below the process boundary: Docker builds the image,
|
||||
Compose starts the real containers, and every helper talks to them over HTTP or the
|
||||
Docker CLI.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
COMPOSE_FILE = REPO / "docker-compose.yml"
|
||||
CONTAINER_LIBRARY = "/library"
|
||||
READY_TIMEOUT_SECONDS = 180
|
||||
JOB_TIMEOUT_SECONDS = 300
|
||||
UP_TIMEOUT_SECONDS = 30 * 60
|
||||
|
||||
|
||||
def compose_available() -> bool:
|
||||
try:
|
||||
return (
|
||||
subprocess.run(
|
||||
["docker", "compose", "version"], capture_output=True, timeout=60
|
||||
).returncode
|
||||
== 0
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return False
|
||||
|
||||
|
||||
def free_port() -> int:
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
return sock.getsockname()[1]
|
||||
|
||||
|
||||
def mount_base() -> Path:
|
||||
"""A directory a Docker VM shares with the host.
|
||||
|
||||
Not pytest's ``tmp_path``: on macOS that is ``/var/folders/...``, which a Docker VM
|
||||
(Colima, Docker Desktop) does not share, so the bind mount would arrive empty and
|
||||
every assertion would be about nothing. ``$HOME`` is shared by every default
|
||||
configuration.
|
||||
"""
|
||||
base = Path(
|
||||
os.environ.get("PHOTO_PIPELINE_TEST_MOUNT_BASE", Path.home() / ".cache" / "photo-pipeline")
|
||||
)
|
||||
base.mkdir(parents=True, exist_ok=True)
|
||||
return base
|
||||
|
||||
|
||||
def temporary_library(albums: dict[str, int], *, prefix: str = "library-") -> Path:
|
||||
"""A fixture library on a shareable host path, with the exclusion sentinel in it."""
|
||||
root = Path(tempfile.mkdtemp(prefix=prefix, dir=mount_base()))
|
||||
for position, (album, count) in enumerate(albums.items()):
|
||||
(root / album).mkdir(parents=True, exist_ok=True)
|
||||
for index in range(count):
|
||||
# Distinct per album *and* index: two solid images of the same colour are
|
||||
# byte-identical, which would make them a duplicate cluster by accident.
|
||||
colour = (17 + 7 * index, 31 + 29 * position, 160 - 3 * index)
|
||||
Image.new("RGB", (64, 48), colour).save(root / album / f"{album}_{index}.jpg")
|
||||
# Never discovered, counted, analyzed, or uploaded — asserted from outside it.
|
||||
(root / "_IGNORE").mkdir(exist_ok=True)
|
||||
Image.new("RGB", (32, 32), (0, 0, 0)).save(root / "_IGNORE" / "sentinel.jpg")
|
||||
return root
|
||||
|
||||
|
||||
def remove_library(root: Path) -> None:
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
|
||||
|
||||
def write_env_file(path: Path, secret: str, extra: dict[str, str] | None = None) -> Path:
|
||||
"""Configuration and secrets come from the environment, so a test writes its own
|
||||
file rather than borrowing the operator's ``.env``."""
|
||||
lines = [
|
||||
f"PHOTO_PIPELINE_ACCESS_SECRET={secret}",
|
||||
"PHOTO_PIPELINE_LOG_FORMAT=text",
|
||||
# The deterministic vision seam, in the data volume so both roles and the test
|
||||
# can read it (concept §18).
|
||||
"PHOTO_PIPELINE_FAKE_VISION_LOG=/data/vision.log",
|
||||
*(f"{key}={value}" for key, value in (extra or {}).items()),
|
||||
]
|
||||
path.write_text("\n".join(lines) + "\n")
|
||||
return path
|
||||
|
||||
|
||||
class Stack:
|
||||
"""The composition under test, plus the environment it was started with."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
library: Path,
|
||||
env_file: Path,
|
||||
*,
|
||||
project: str,
|
||||
image: str,
|
||||
secret: str,
|
||||
) -> None:
|
||||
self.library = library
|
||||
self.project = project
|
||||
self.secret = secret
|
||||
self.port = free_port()
|
||||
self.base = f"http://127.0.0.1:{self.port}"
|
||||
# Compose reads the repository's own .env for substitution; the process
|
||||
# environment wins over it, so the test's values are the ones that apply.
|
||||
self.env = {
|
||||
**os.environ,
|
||||
"PHOTO_PIPELINE_IMAGE": image,
|
||||
"PHOTO_PIPELINE_ENV_FILE": str(env_file),
|
||||
"PHOTO_PIPELINE_LIBRARY_HOST_PATH": str(library),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": CONTAINER_LIBRARY,
|
||||
"PHOTO_PIPELINE_PORT": str(self.port),
|
||||
"PHOTO_PIPELINE_UID": str(os.getuid()),
|
||||
"PHOTO_PIPELINE_GID": str(os.getgid()),
|
||||
}
|
||||
|
||||
# ── the compose lifecycle ────────────────────────────────────────────────
|
||||
|
||||
def compose(self, *args: str, check: bool = True, timeout: int = 300):
|
||||
result = subprocess.run(
|
||||
["docker", "compose", "-p", self.project, "-f", str(COMPOSE_FILE), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=self.env,
|
||||
cwd=REPO,
|
||||
timeout=timeout,
|
||||
)
|
||||
if check and result.returncode != 0:
|
||||
raise AssertionError(
|
||||
f"docker compose {' '.join(args)} failed:\n{result.stdout}\n{result.stderr}\n"
|
||||
f"{self.compose('logs', '--tail', '80', check=False).stdout}"
|
||||
)
|
||||
return result
|
||||
|
||||
def up(self, *extra: str) -> None:
|
||||
self.compose("up", "--detach", *extra, timeout=UP_TIMEOUT_SECONDS)
|
||||
|
||||
def down(self, *, volumes: bool = True) -> None:
|
||||
self.compose(
|
||||
"down",
|
||||
*(("--volumes",) if volumes else ()),
|
||||
"--remove-orphans",
|
||||
check=False,
|
||||
timeout=300,
|
||||
)
|
||||
|
||||
def use_image(self, image: str) -> None:
|
||||
"""Point the composition at another tag — the upgrade path (US08-05)."""
|
||||
self.env["PHOTO_PIPELINE_IMAGE"] = image
|
||||
|
||||
def logs(self, *services: str) -> str:
|
||||
result = self.compose("logs", *services, check=False)
|
||||
return result.stdout + result.stderr
|
||||
|
||||
def wait_until_ready(self) -> None:
|
||||
deadline = time.monotonic() + READY_TIMEOUT_SECONDS
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
if httpx.get(f"{self.base}/api/v1/health/ready", timeout=5).status_code == 200:
|
||||
return
|
||||
except httpx.HTTPError:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"the stack never became ready:\n{self.logs()}")
|
||||
|
||||
# ── talking to it ────────────────────────────────────────────────────────
|
||||
|
||||
def client(self) -> httpx.Client:
|
||||
"""A browser that has loaded the app: session cookie in the jar, token in a
|
||||
header. The session belongs to the API process, so it is re-bootstrapped after
|
||||
every restart."""
|
||||
client = httpx.Client(base_url=f"{self.base}/api/v1", timeout=60)
|
||||
bootstrap = client.get("/session", headers={"X-Access-Secret": self.secret})
|
||||
assert bootstrap.status_code == 200, bootstrap.text
|
||||
client.headers["X-CSRF-Token"] = bootstrap.json()["csrf_token"]
|
||||
return client
|
||||
|
||||
|
||||
def await_job(client: httpx.Client, job_id: str, states=("succeeded",)) -> dict:
|
||||
deadline = time.monotonic() + JOB_TIMEOUT_SECONDS
|
||||
snapshot: dict = {}
|
||||
while time.monotonic() < deadline:
|
||||
response = client.get(f"/jobs/{job_id}")
|
||||
if response.status_code == 200:
|
||||
snapshot = response.json()
|
||||
if snapshot["state"] in states:
|
||||
return snapshot
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"job {job_id} never reached {states}: {snapshot}")
|
||||
|
||||
|
||||
def build_image(tag: str, *, revision: str | None = None) -> str:
|
||||
"""Build the application image; from a git revision's tree when one is named.
|
||||
|
||||
``revision`` is how the upgrade journey gets the *previous* version without a
|
||||
registry: the tree of that commit is the build context, so what it produces is the
|
||||
image that commit would have published.
|
||||
"""
|
||||
if revision is None:
|
||||
subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"build",
|
||||
"--build-arg",
|
||||
f"UID={os.getuid()}",
|
||||
"--build-arg",
|
||||
f"GID={os.getgid()}",
|
||||
"-t",
|
||||
tag,
|
||||
str(REPO),
|
||||
],
|
||||
check=True,
|
||||
timeout=UP_TIMEOUT_SECONDS,
|
||||
)
|
||||
return tag
|
||||
archive = subprocess.run(
|
||||
["git", "archive", "--format=tar", revision],
|
||||
cwd=REPO,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
timeout=300,
|
||||
).stdout
|
||||
subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"build",
|
||||
"--build-arg",
|
||||
f"UID={os.getuid()}",
|
||||
"--build-arg",
|
||||
f"GID={os.getgid()}",
|
||||
"-t",
|
||||
tag,
|
||||
"-",
|
||||
],
|
||||
input=archive,
|
||||
check=True,
|
||||
timeout=UP_TIMEOUT_SECONDS,
|
||||
)
|
||||
return tag
|
||||
|
||||
|
||||
needs_compose = pytest.mark.skipif(
|
||||
not compose_available(), reason="no Docker daemon with the compose plugin"
|
||||
)
|
||||
@@ -10,188 +10,63 @@ what this story is about — the mount, the lock, the volume, and the restart ar
|
||||
The file contract (one API, one worker, migrations first, no committed values) is
|
||||
checked without a daemon in ``tests/integration/test_compose_runtime.py``; only the
|
||||
running proof needs Docker, and CI is where it runs unskipped (US08-04).
|
||||
|
||||
The stack itself lives in ``tests/e2e/_container_harness.py``, shared with the
|
||||
container acceptance gate (US08-05).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
from tests.e2e._container_harness import (
|
||||
CONTAINER_LIBRARY,
|
||||
Stack,
|
||||
await_job,
|
||||
compose_available,
|
||||
needs_compose,
|
||||
remove_library,
|
||||
temporary_library,
|
||||
write_env_file,
|
||||
)
|
||||
|
||||
PROJECT = "photo-pipeline-us0803"
|
||||
IMAGE = "photo-pipeline-test:us08-03"
|
||||
SECRET = "compose-acceptance-secret"
|
||||
CONTAINER_LIBRARY = "/library"
|
||||
READY_TIMEOUT_SECONDS = 180
|
||||
JOB_TIMEOUT_SECONDS = 180
|
||||
UP_TIMEOUT_SECONDS = 30 * 60
|
||||
|
||||
pytestmark = pytest.mark.container
|
||||
|
||||
|
||||
def compose_available() -> bool:
|
||||
try:
|
||||
return (
|
||||
subprocess.run(
|
||||
["docker", "compose", "version"], capture_output=True, timeout=60
|
||||
).returncode
|
||||
== 0
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return False
|
||||
|
||||
|
||||
needs_compose = pytest.mark.skipif(
|
||||
not compose_available(), reason="no Docker daemon with the compose plugin"
|
||||
)
|
||||
|
||||
|
||||
def free_port() -> int:
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
return sock.getsockname()[1]
|
||||
|
||||
|
||||
class Stack:
|
||||
"""The composition under test, plus the environment it was started with."""
|
||||
|
||||
def __init__(self, library: Path, env_file: Path) -> None:
|
||||
self.library = library
|
||||
self.port = free_port()
|
||||
self.base = f"http://127.0.0.1:{self.port}"
|
||||
# Compose reads the repository's own .env for substitution; the process
|
||||
# environment wins over it, so the test's values are the ones that apply.
|
||||
self.env = {
|
||||
**os.environ,
|
||||
"PHOTO_PIPELINE_IMAGE": IMAGE,
|
||||
"PHOTO_PIPELINE_ENV_FILE": str(env_file),
|
||||
"PHOTO_PIPELINE_LIBRARY_HOST_PATH": str(library),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": CONTAINER_LIBRARY,
|
||||
"PHOTO_PIPELINE_PORT": str(self.port),
|
||||
"PHOTO_PIPELINE_UID": str(os.getuid()),
|
||||
"PHOTO_PIPELINE_GID": str(os.getgid()),
|
||||
}
|
||||
|
||||
def compose(self, *args: str, check: bool = True, timeout: int = 300):
|
||||
result = subprocess.run(
|
||||
["docker", "compose", "-p", PROJECT, "-f", str(REPO / "docker-compose.yml"), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=self.env,
|
||||
cwd=REPO,
|
||||
timeout=timeout,
|
||||
)
|
||||
if check and result.returncode != 0:
|
||||
raise AssertionError(
|
||||
f"docker compose {' '.join(args)} failed:\n{result.stdout}\n{result.stderr}\n"
|
||||
f"{self.compose('logs', '--tail', '80', check=False).stdout}"
|
||||
)
|
||||
return result
|
||||
|
||||
def wait_until_ready(self) -> None:
|
||||
deadline = time.monotonic() + READY_TIMEOUT_SECONDS
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
if httpx.get(f"{self.base}/api/v1/health/ready", timeout=5).status_code == 200:
|
||||
return
|
||||
except httpx.HTTPError:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
logs = self.compose("logs", "--tail", "120", check=False)
|
||||
raise AssertionError(f"the stack never became ready:\n{logs.stdout}\n{logs.stderr}")
|
||||
|
||||
def client(self) -> httpx.Client:
|
||||
client = httpx.Client(base_url=f"{self.base}/api/v1", timeout=60)
|
||||
bootstrap = client.get("/session", headers={"X-Access-Secret": SECRET})
|
||||
assert bootstrap.status_code == 200, bootstrap.text
|
||||
client.headers["X-CSRF-Token"] = bootstrap.json()["csrf_token"]
|
||||
return client
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def library() -> Path:
|
||||
"""A small fixture library on the host, mounted into both containers.
|
||||
|
||||
Not under pytest's ``tmp_path``: on macOS that is ``/var/folders/...``, which a
|
||||
Docker VM (Colima, Docker Desktop) does not share, so the bind mount would arrive
|
||||
empty and every assertion below would be about nothing. ``$HOME`` is shared by
|
||||
every default configuration.
|
||||
"""
|
||||
base = Path(
|
||||
os.environ.get("PHOTO_PIPELINE_TEST_MOUNT_BASE", Path.home() / ".cache" / "photo-pipeline")
|
||||
)
|
||||
base.mkdir(parents=True, exist_ok=True)
|
||||
root = Path(tempfile.mkdtemp(prefix="library-", dir=base))
|
||||
for album, count in (("01_day", 2), ("02_night", 1)):
|
||||
(root / album).mkdir()
|
||||
for index in range(count):
|
||||
colour = (40 * (index + 1), 90, 160)
|
||||
Image.new("RGB", (64, 48), colour).save(root / album / f"{album}_{index}.jpg")
|
||||
# The exclusion sentinel: it must never be discovered, counted, or analyzed.
|
||||
(root / "_IGNORE").mkdir()
|
||||
Image.new("RGB", (32, 32), (0, 0, 0)).save(root / "_IGNORE" / "sentinel.jpg")
|
||||
root = temporary_library({"01_day": 2, "02_night": 1})
|
||||
try:
|
||||
yield root
|
||||
finally:
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
remove_library(root)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def env_file(tmp_path_factory) -> Path:
|
||||
"""Configuration and secrets come from the environment, so the test writes its
|
||||
own file rather than borrowing the operator's."""
|
||||
path = tmp_path_factory.mktemp("config") / "compose.env"
|
||||
path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
f"PHOTO_PIPELINE_ACCESS_SECRET={SECRET}",
|
||||
"PHOTO_PIPELINE_LOG_FORMAT=text",
|
||||
# The deterministic vision seam, in the data volume so both roles and
|
||||
# the test can see it (concept §18).
|
||||
"PHOTO_PIPELINE_FAKE_VISION_LOG=/data/vision.log",
|
||||
]
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
return path
|
||||
return write_env_file(tmp_path_factory.mktemp("config") / "compose.env", SECRET)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def stack(library, env_file):
|
||||
if not compose_available():
|
||||
pytest.skip("no Docker daemon with the compose plugin")
|
||||
running = Stack(library, env_file)
|
||||
running.compose("down", "--volumes", "--remove-orphans", check=False)
|
||||
running.compose("up", "--detach", "--build", timeout=UP_TIMEOUT_SECONDS)
|
||||
running = Stack(library, env_file, project=PROJECT, image=IMAGE, secret=SECRET)
|
||||
running.down()
|
||||
running.up("--build")
|
||||
try:
|
||||
running.wait_until_ready()
|
||||
yield running
|
||||
finally:
|
||||
running.compose("down", "--volumes", "--remove-orphans", check=False, timeout=300)
|
||||
|
||||
|
||||
def await_job(client: httpx.Client, job_id: str, states=("succeeded",)) -> dict:
|
||||
deadline = time.monotonic() + JOB_TIMEOUT_SECONDS
|
||||
snapshot: dict = {}
|
||||
while time.monotonic() < deadline:
|
||||
response = client.get(f"/jobs/{job_id}")
|
||||
if response.status_code == 200:
|
||||
snapshot = response.json()
|
||||
if snapshot["state"] in states:
|
||||
return snapshot
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"job {job_id} never reached {states}: {snapshot}")
|
||||
running.down()
|
||||
|
||||
|
||||
# ── the mounted library ──────────────────────────────────────────────────────
|
||||
@@ -296,8 +171,8 @@ def test_the_data_volume_survives_recreating_the_containers(stack):
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
stack.compose("down", "--remove-orphans", timeout=300)
|
||||
stack.compose("up", "--detach", timeout=UP_TIMEOUT_SECONDS)
|
||||
stack.down(volumes=False)
|
||||
stack.up()
|
||||
stack.wait_until_ready()
|
||||
|
||||
client = stack.client()
|
||||
|
||||
119
tests/e2e/test_docs_ui.py
Normal file
119
tests/e2e/test_docs_ui.py
Normal file
@@ -0,0 +1,119 @@
|
||||
"""US09-01: the manuals, in the browser, under the application's real policy.
|
||||
|
||||
Everything here runs against a real ``photo_pipeline serve`` process serving the
|
||||
committed ``docs/`` tree and the vendored renderer — no fixture markdown, no stubbed
|
||||
fetch. What that buys is the assertion the story actually cares about: the pages
|
||||
render, the links between them work, a diagram becomes a diagram, and the console
|
||||
stays empty, including of CSP violations.
|
||||
|
||||
The offline contract — reachability, dead links, pinned checksums — is
|
||||
``tests/integration/test_documentation.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.e2e._pipeline_harness import Server, seed_library
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def server(tmp_path_factory):
|
||||
seeded = seed_library(tmp_path_factory.mktemp("docs"), {"a": 1}, {})
|
||||
running = Server(seeded).start()
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def quiet(page):
|
||||
"""Any console error, page error, or failed request fails the test that caused
|
||||
it. A CSP violation arrives as a console error, which is the point."""
|
||||
problems: list[str] = []
|
||||
page.on("console", lambda m: problems.append(m.text) if m.type == "error" else None)
|
||||
page.on("pageerror", lambda error: problems.append(str(error)))
|
||||
page.on("requestfailed", lambda request: problems.append(f"failed request: {request.url}"))
|
||||
yield problems
|
||||
|
||||
|
||||
def test_the_documentation_opens_from_the_navigation(page, server, quiet):
|
||||
page.goto(f"{server.base}/app/#/workflow")
|
||||
page.locator('nav a[data-nav="docs"]').click()
|
||||
|
||||
page.get_by_test_id("doc").wait_for()
|
||||
assert page.get_by_test_id("doc").get_attribute("data-page") == "index"
|
||||
# The sidebar is the index's own reading order, not a second list to maintain.
|
||||
assert page.get_by_test_id("doc-pages").get_by_role("link", name="Overview").is_visible()
|
||||
assert quiet == []
|
||||
|
||||
|
||||
def test_a_link_between_documents_stays_inside_the_application(page, server, quiet):
|
||||
page.goto(f"{server.base}/app/#/docs")
|
||||
page.get_by_test_id("doc").wait_for()
|
||||
page.get_by_test_id("doc").get_by_role("link", name="Overview").click()
|
||||
|
||||
page.wait_for_selector('[data-testid="doc"][data-page="overview"]')
|
||||
assert "#/docs?page=overview" in page.url, "a relative .md link must not leave the app"
|
||||
# And back again, by the link the document itself carries.
|
||||
page.get_by_test_id("doc").get_by_role("link", name="← Documentation index").click()
|
||||
page.wait_for_selector('[data-testid="doc"][data-page="index"]')
|
||||
assert quiet == []
|
||||
|
||||
|
||||
def test_a_deep_link_to_a_heading_lands_on_that_heading(page, server, quiet):
|
||||
page.goto(f"{server.base}/app/#/docs?page=overview&anchor=the-workflow")
|
||||
heading = page.locator("#the-workflow")
|
||||
heading.wait_for()
|
||||
|
||||
assert heading.inner_text().strip() == "The workflow"
|
||||
assert heading.evaluate("node => node.getBoundingClientRect().top < window.innerHeight")
|
||||
assert quiet == []
|
||||
|
||||
|
||||
def test_a_mermaid_block_becomes_a_diagram_under_the_unchanged_script_policy(page, server, quiet):
|
||||
page.goto(f"{server.base}/app/#/docs?page=overview")
|
||||
diagram = page.get_by_test_id("diagram").first
|
||||
diagram.wait_for()
|
||||
|
||||
# A real drawing, not the source text and not an error node.
|
||||
assert diagram.locator("svg").count() == 1
|
||||
assert diagram.evaluate("node => node.querySelector('svg').getBBox().width") > 100
|
||||
assert page.locator("pre code.language-mermaid").count() == 0
|
||||
assert quiet == [], "rendering a diagram must not violate the policy"
|
||||
|
||||
policy = page.evaluate(
|
||||
"async () => (await fetch('/app/')).headers.get('content-security-policy')"
|
||||
)
|
||||
assert "script-src 'self';" in policy
|
||||
assert "unsafe-eval" not in policy
|
||||
|
||||
|
||||
def test_an_unknown_page_says_so_without_naming_a_path(page, server, quiet):
|
||||
page.goto(f"{server.base}/app/#/docs?page=no-such-manual")
|
||||
message = page.get_by_test_id("doc-not-found")
|
||||
message.wait_for()
|
||||
|
||||
text = message.inner_text()
|
||||
assert "does not exist" in text
|
||||
assert "/" not in text.replace("Back to the documentation index", ""), text
|
||||
message.get_by_role("link").click()
|
||||
page.wait_for_selector('[data-testid="doc"][data-page="index"]')
|
||||
# The missing page is a 404 and the browser says so; nothing else may go wrong,
|
||||
# and in particular the view must not throw on the way to its own message.
|
||||
assert all("404" in problem for problem in quiet), quiet
|
||||
|
||||
|
||||
def test_the_documentation_is_readable_without_a_session(page, server, quiet):
|
||||
"""The troubleshooting page is needed most by whoever is locked out."""
|
||||
page.goto(f"{server.base}/app/#/docs")
|
||||
page.get_by_test_id("doc").wait_for()
|
||||
unauthenticated = page.evaluate(
|
||||
"""async () => {
|
||||
const response = await fetch('/docs/index.md', { credentials: 'omit' });
|
||||
return { status: response.status, length: (await response.text()).length };
|
||||
}"""
|
||||
)
|
||||
assert unauthenticated["status"] == 200
|
||||
assert unauthenticated["length"] > 100
|
||||
510
tests/e2e/test_phase_h_container.py
Normal file
510
tests/e2e/test_phase_h_container.py
Normal file
@@ -0,0 +1,510 @@
|
||||
"""Phase H — the container acceptance gate (US08-05).
|
||||
|
||||
The deployed application is verified the way the host application is: real image,
|
||||
real composition, real browser, real restarts. Everything below runs against
|
||||
containers that this suite provisions from the built image, against a temporary
|
||||
fixture library on a bind mount and an isolated data volume, and destroys afterwards.
|
||||
|
||||
Four journeys, one per thing a deployment can get wrong:
|
||||
|
||||
* **the browser journey** — discovery, duplicate review, analysis, album proposal,
|
||||
rename, upload preflight, and archive, driven through the containerized frontend;
|
||||
* **the upgrade journey** — the previous version's image runs first, then this one,
|
||||
and the database, its migrations, the rename journal, the job history, and the
|
||||
thumbnail cache have to still be there;
|
||||
* **the restart journey** — both containers are killed mid-job and the work resumes
|
||||
without doing anything twice;
|
||||
* **the security gates** — the refusals a loopback deployment made are still made
|
||||
behind a published port: no session, forged forwarded headers, a path that leaves
|
||||
the mounted library, and a secret in the logs.
|
||||
|
||||
Run it as one command, with evidence: ``python -m photo_pipeline container-gate``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import collections
|
||||
from contextlib import closing
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import expect
|
||||
|
||||
from photo_pipeline.db import head_revision
|
||||
from tests.e2e._container_harness import (
|
||||
Stack,
|
||||
await_job,
|
||||
build_image,
|
||||
compose_available,
|
||||
needs_compose,
|
||||
remove_library,
|
||||
temporary_library,
|
||||
write_env_file,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.phase_h, pytest.mark.container]
|
||||
|
||||
PROJECT = "photo-pipeline-us0805"
|
||||
IMAGE = "photo-pipeline-test:us08-05"
|
||||
# The version being upgraded *from*. CI passes the tag it last published; without one,
|
||||
# the previous commit's tree is built, which is the same claim without a registry.
|
||||
PREVIOUS_IMAGE = os.environ.get("PHOTO_PIPELINE_PREVIOUS_IMAGE")
|
||||
PREVIOUS_TAG = "photo-pipeline-test:us08-05-previous"
|
||||
SECRET = "container-gate-access-secret"
|
||||
IMMICH_SENTINEL = "immich-sentinel-9f3a2b"
|
||||
HOSTNAME = "photos.test"
|
||||
ALBUM = "rome"
|
||||
RENAMED = "2019 Rome"
|
||||
BURST = 30
|
||||
|
||||
|
||||
# ── the image and the stacks ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def image() -> str:
|
||||
if not compose_available():
|
||||
pytest.skip("no Docker daemon with the compose plugin")
|
||||
return build_image(IMAGE)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def journey(image, tmp_path_factory):
|
||||
"""One album, one exact duplicate of a photo in it, and the exclusion sentinel."""
|
||||
library = temporary_library({ALBUM: 3}, prefix="us0805-journey-")
|
||||
shutil.copyfile(library / ALBUM / f"{ALBUM}_0.jpg", library / ALBUM / "copy.jpg")
|
||||
env_file = write_env_file(
|
||||
tmp_path_factory.mktemp("journey") / "gate.env",
|
||||
SECRET,
|
||||
{
|
||||
"PHOTO_PIPELINE_ALLOWED_HOSTS": HOSTNAME,
|
||||
# Configured but never reachable: the upload view's job here is to show the
|
||||
# preflight blockers, and the key's job is to be absent from every log.
|
||||
"PHOTO_PIPELINE_IMMICH_API_KEY": IMMICH_SENTINEL,
|
||||
"PHOTO_PIPELINE_IMMICH_SERVER_URL": "http://127.0.0.1:1",
|
||||
},
|
||||
)
|
||||
stack = Stack(library, env_file, project=f"{PROJECT}-journey", image=image, secret=SECRET)
|
||||
try:
|
||||
stack.down()
|
||||
stack.up()
|
||||
stack.wait_until_ready()
|
||||
_seed_journey(stack)
|
||||
yield stack
|
||||
finally:
|
||||
stack.down()
|
||||
remove_library(library)
|
||||
|
||||
|
||||
def _seed_journey(stack: Stack) -> None:
|
||||
"""Everything the views need, established over the public API before they render."""
|
||||
with closing(stack.client()) as client:
|
||||
client.post("/inventory/scan").raise_for_status()
|
||||
client.post("/duplicates/detect").raise_for_status()
|
||||
queue = client.get("/safety/queue", params={"limit": 100}).json()["items"]
|
||||
for item in queue:
|
||||
decided = client.post(
|
||||
"/safety/decisions", json={"asset_id": item["asset_id"], "decision": "sfw"}
|
||||
)
|
||||
assert decided.status_code == 200, decided.text
|
||||
job = client.post("/analysis/jobs").json()
|
||||
await_job(client, job["id"])
|
||||
client.post("/albums/proposals", json={}).raise_for_status()
|
||||
# An archive destination inside the data volume: a second mount would prove
|
||||
# nothing more, and the view needs a location to have something to show.
|
||||
stack.compose("exec", "-T", "api", "mkdir", "-p", "/data/archive")
|
||||
client.post(
|
||||
"/archive-locations", json={"name": "external", "root": "/data/archive"}
|
||||
).raise_for_status()
|
||||
|
||||
|
||||
# ── the browser journey ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_the_browser_journey_covers_every_stage_view_of_the_deployed_app(page, journey):
|
||||
"""One pass through the deployed frontend, in workflow order.
|
||||
|
||||
The access secret is supplied the way a person supplies it — the app asks, the
|
||||
answer is kept for the tab — so what is proven is the authenticated deployment,
|
||||
not a test-only bypass.
|
||||
"""
|
||||
page.on("dialog", lambda dialog: dialog.accept(SECRET))
|
||||
errors: list[str] = []
|
||||
page.on("pageerror", lambda error: errors.append(str(error)))
|
||||
base = journey.base
|
||||
with closing(journey.client()) as client:
|
||||
cluster = client.get("/duplicates/clusters").json()["items"][0]
|
||||
|
||||
# ── discovery ────────────────────────────────────────────────────────────
|
||||
page.goto(f"{base}/app/#/workflow")
|
||||
page.get_by_test_id("stage-safety").wait_for()
|
||||
expect(page.get_by_test_id("stage-analysis")).to_be_visible()
|
||||
|
||||
page.goto(f"{base}/app/#/inventory")
|
||||
rows = page.get_by_test_id("asset-row")
|
||||
rows.first.wait_for()
|
||||
assert rows.count() == 4, "three photos and the duplicate copy; never the sentinel"
|
||||
assert "sentinel" not in page.content() and "_IGNORE" not in page.content()
|
||||
|
||||
# ── duplicate review ─────────────────────────────────────────────────────
|
||||
page.goto(f"{base}/app/#/duplicates/{cluster['id']}")
|
||||
page.get_by_test_id("cluster-state").wait_for()
|
||||
# Exact bytes: the cluster arrives decided, and the review surface has to show
|
||||
# both members and the evidence the decision was made on.
|
||||
expect(page.get_by_test_id("cluster-state")).to_contain_text("decided")
|
||||
expect(page.get_by_test_id("member")).to_have_count(2)
|
||||
expect(page.get_by_test_id("member").first).to_contain_text("/library/")
|
||||
|
||||
# ── analysis ─────────────────────────────────────────────────────────────
|
||||
page.goto(f"{base}/app/#/analyze")
|
||||
page.get_by_test_id("analyze-counts").wait_for()
|
||||
expect(page.get_by_test_id("run-analysis")).to_be_visible()
|
||||
|
||||
# ── album proposal ───────────────────────────────────────────────────────
|
||||
page.goto(f"{base}/app/#/albums?album={ALBUM}")
|
||||
page.get_by_test_id("suggested-name").wait_for()
|
||||
page.get_by_test_id("final-name").fill(RENAMED)
|
||||
page.get_by_test_id("save-name").click()
|
||||
page.get_by_test_id("approve").click()
|
||||
expect(page.get_by_test_id("proposal-status")).to_contain_text("approved")
|
||||
|
||||
# ── rename, applied against the bind mount ───────────────────────────────
|
||||
page.goto(f"{base}/app/#/renames")
|
||||
page.get_by_test_id("build-plan").click()
|
||||
page.get_by_test_id("operations").wait_for()
|
||||
expect(page.get_by_test_id("op-destination").first).to_contain_text(RENAMED)
|
||||
page.get_by_test_id("apply-plan").click()
|
||||
expect(page.get_by_test_id("apply-result")).to_contain_text("Applied 1, failed 0")
|
||||
# The mounted library is the host's directory: the container renamed the operator's
|
||||
# folder, not a copy inside its own layer.
|
||||
assert (journey.library / RENAMED).is_dir()
|
||||
assert not (journey.library / ALBUM).exists()
|
||||
|
||||
# ── upload preflight ─────────────────────────────────────────────────────
|
||||
page.goto(f"{base}/app/#/uploads")
|
||||
page.get_by_test_id("upload-scope").wait_for()
|
||||
expect(page.get_by_test_id("album-row").first).to_be_visible()
|
||||
assert IMMICH_SENTINEL not in page.content(), "the API key never reaches the browser"
|
||||
|
||||
# ── archive ──────────────────────────────────────────────────────────────
|
||||
page.goto(f"{base}/app/#/archive")
|
||||
page.get_by_test_id("archive-locations").wait_for()
|
||||
expect(page.get_by_test_id("location-row").first).to_contain_text("external")
|
||||
|
||||
assert errors == [], f"the deployed frontend raised page errors: {errors}"
|
||||
|
||||
|
||||
# ── the upgrade journey ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def previous_image(image) -> str:
|
||||
"""The image the deployment is upgrading *from*.
|
||||
|
||||
The published tag when there is one. Before the first publish there is nothing to
|
||||
pull, and refusing then would mean the gate could never let the first deploy
|
||||
through — so the previous commit's tree is built instead, which is the same claim
|
||||
without a registry.
|
||||
"""
|
||||
if PREVIOUS_IMAGE:
|
||||
pulled = subprocess.run(
|
||||
["docker", "pull", PREVIOUS_IMAGE], capture_output=True, timeout=1800
|
||||
)
|
||||
if pulled.returncode == 0:
|
||||
return PREVIOUS_IMAGE
|
||||
return build_image(PREVIOUS_TAG, revision="HEAD~1")
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_an_upgrade_keeps_the_database_the_journal_the_jobs_and_the_cache(
|
||||
image, previous_image, tmp_path_factory
|
||||
):
|
||||
library = temporary_library({ALBUM: 2}, prefix="us0805-upgrade-")
|
||||
env_file = write_env_file(tmp_path_factory.mktemp("upgrade") / "gate.env", SECRET)
|
||||
stack = Stack(
|
||||
library, env_file, project=f"{PROJECT}-upgrade", image=previous_image, secret=SECRET
|
||||
)
|
||||
try:
|
||||
stack.down()
|
||||
stack.up()
|
||||
stack.wait_until_ready()
|
||||
|
||||
# ── what the previous version leaves behind ──────────────────────────
|
||||
with closing(stack.client()) as client:
|
||||
client.post("/inventory/scan").raise_for_status()
|
||||
assets = client.get("/inventory/assets", params={"limit": 200}).json()["items"]
|
||||
for item in client.get("/safety/queue", params={"limit": 100}).json()["items"]:
|
||||
client.post(
|
||||
"/safety/decisions", json={"asset_id": item["asset_id"], "decision": "sfw"}
|
||||
).raise_for_status()
|
||||
job = client.post("/analysis/jobs").json()
|
||||
finished = await_job(client, job["id"])
|
||||
for asset in assets: # populate the thumbnail cache
|
||||
thumbnail = client.get(f"/assets/{asset['id']}/thumbnail", params={"size": 256})
|
||||
assert thumbnail.status_code == 200, thumbnail.text
|
||||
# A rename plan, left unapplied: the journal has to survive the upgrade
|
||||
# exactly as it was, or a half-applied one could not be recovered.
|
||||
client.post("/albums/proposals", json={}).raise_for_status()
|
||||
_approve(client, RENAMED)
|
||||
plan = client.post("/rename-plans").json()
|
||||
|
||||
before = {
|
||||
"assets": sorted(asset["id"] for asset in assets),
|
||||
"job": finished["progress"],
|
||||
"plan": (plan["id"], plan["checksum"]),
|
||||
"thumbnails": _cache_files(stack),
|
||||
"revision": _revision(stack),
|
||||
}
|
||||
assert before["thumbnails"], "no thumbnail was cached, so nothing would be proven"
|
||||
|
||||
# ── the upgrade: same volume, new image ──────────────────────────────
|
||||
stack.down(volumes=False)
|
||||
stack.use_image(image)
|
||||
stack.up()
|
||||
stack.wait_until_ready()
|
||||
|
||||
after_revision = _revision(stack)
|
||||
assert after_revision == head_revision(), "the new image did not migrate the volume"
|
||||
if after_revision != before["revision"]:
|
||||
# A schema change is snapshotted before it is applied (US07-05), so a
|
||||
# failed upgrade is restorable rather than a lost library.
|
||||
assert _ls(stack, "/data/backups"), "a migration ran without a backup"
|
||||
|
||||
with closing(stack.client()) as client:
|
||||
assets = client.get("/inventory/assets", params={"limit": 200}).json()["items"]
|
||||
assert sorted(asset["id"] for asset in assets) == before["assets"]
|
||||
assert client.get(f"/jobs/{job['id']}").json()["progress"] == before["job"]
|
||||
plans = client.get("/rename-plans").json()["items"]
|
||||
assert (plans[0]["id"], plans[0]["checksum"]) == before["plan"]
|
||||
for asset in assets:
|
||||
assert (
|
||||
client.get(f"/analysis/results/{asset['id']}").status_code == 200
|
||||
), "an analysis result did not survive the upgrade"
|
||||
# The cache is keyed by pixel hash and thumbnail version, so an upgrade that
|
||||
# kept the volume must keep the files: regenerating them is work nobody asked
|
||||
# for, and losing them silently is how a cache stops being one.
|
||||
assert set(before["thumbnails"]) <= set(_cache_files(stack))
|
||||
finally:
|
||||
stack.down()
|
||||
remove_library(library)
|
||||
|
||||
|
||||
def _approve(client: httpx.Client, name: str, *, album: str = ALBUM) -> None:
|
||||
for payload, route in (({"name": name}, "edit"), ({}, "approve")):
|
||||
current = client.get(f"/albums/proposals/{album}").json()
|
||||
client.post(
|
||||
f"/albums/proposals/{album}/{route}",
|
||||
json={**payload, "expected_version": current["version"]},
|
||||
).raise_for_status()
|
||||
|
||||
|
||||
def _exec(stack: Stack, *args: str) -> str:
|
||||
return stack.compose("exec", "-T", "api", *args).stdout
|
||||
|
||||
|
||||
def _ls(stack: Stack, directory: str) -> list[str]:
|
||||
listing = stack.compose("exec", "-T", "api", "ls", directory, check=False)
|
||||
return [line for line in listing.stdout.split() if line]
|
||||
|
||||
|
||||
def _cache_files(stack: Stack) -> list[str]:
|
||||
return sorted(
|
||||
_exec(stack, "find", "/data/cache", "-type", "f", "-name", "*.webp").split()
|
||||
)
|
||||
|
||||
|
||||
def _revision(stack: Stack) -> str:
|
||||
"""The schema revision the volume's database is actually at."""
|
||||
return _exec(
|
||||
stack,
|
||||
"python",
|
||||
"-c",
|
||||
"import sqlite3;print(sqlite3.connect('/data/photo_pipeline.db')"
|
||||
".execute('select version_num from alembic_version').fetchone()[0])",
|
||||
).strip()
|
||||
|
||||
|
||||
# ── the restart journey ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_killing_both_containers_mid_job_resumes_without_doing_anything_twice(
|
||||
image, tmp_path_factory
|
||||
):
|
||||
"""`docker kill` is the honest restart: no grace period, no orderly stop, no
|
||||
chance for either process to write a tidy final state."""
|
||||
library = temporary_library({"burst": BURST}, prefix="us0805-restart-")
|
||||
env_file = write_env_file(tmp_path_factory.mktemp("restart") / "gate.env", SECRET)
|
||||
stack = Stack(library, env_file, project=f"{PROJECT}-restart", image=image, secret=SECRET)
|
||||
try:
|
||||
stack.down()
|
||||
stack.up()
|
||||
stack.wait_until_ready()
|
||||
|
||||
with closing(stack.client()) as client:
|
||||
client.post("/inventory/scan").raise_for_status()
|
||||
assets = client.get("/inventory/assets", params={"limit": 200}).json()["items"]
|
||||
assert len(assets) == BURST
|
||||
for item in client.get("/safety/queue", params={"limit": 100}).json()["items"]:
|
||||
client.post(
|
||||
"/safety/decisions", json={"asset_id": item["asset_id"], "decision": "sfw"}
|
||||
).raise_for_status()
|
||||
job = client.post("/analysis/jobs").json()
|
||||
progress = _wait_for_progress(client, job["id"])
|
||||
|
||||
assert 0 < progress["done"] < progress["total"], progress
|
||||
stack.compose("kill", "api", "worker")
|
||||
|
||||
stack.up()
|
||||
stack.wait_until_ready()
|
||||
with closing(stack.client()) as client:
|
||||
finished = await_job(client, job["id"])
|
||||
assert finished["progress"]["done"] == BURST, finished
|
||||
results = [
|
||||
client.get(f"/analysis/results/{asset['id']}").json() for asset in assets
|
||||
]
|
||||
|
||||
# Exactly one stored result per asset: at-least-once execution, idempotent
|
||||
# recovery — a retried item overwrites its own attempt, it does not add one.
|
||||
assert len(results) == BURST
|
||||
assert all(result["description"] for result in results)
|
||||
# And the side effect nobody can take back — the call to the provider — happened
|
||||
# again only for whatever was in flight when the containers died.
|
||||
analysed = collections.Counter(_exec(stack, "cat", "/data/vision.log").split())
|
||||
assert len(analysed) == BURST, "every photo was analysed, and only the library's"
|
||||
assert max(analysed.values()) <= 2, dict(analysed)
|
||||
assert sum(1 for count in analysed.values() if count > 1) <= 1, dict(analysed)
|
||||
finally:
|
||||
stack.down()
|
||||
remove_library(library)
|
||||
|
||||
|
||||
def _wait_for_progress(client: httpx.Client, job_id: str, *, timeout: float = 120) -> dict:
|
||||
"""Wait until the job is provably under way but provably unfinished."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
snapshot = client.get(f"/jobs/{job_id}").json()
|
||||
progress = snapshot["progress"]
|
||||
if progress["done"] and progress["done"] < progress["total"]:
|
||||
return progress
|
||||
if snapshot["state"] in ("succeeded", "failed"):
|
||||
raise AssertionError(f"the job finished before it could be interrupted: {snapshot}")
|
||||
time.sleep(0.05)
|
||||
raise AssertionError(f"the job never started: {client.get(f'/jobs/{job_id}').json()}")
|
||||
|
||||
|
||||
# ── the security gates ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_the_deployed_instance_refuses_a_caller_without_a_session(journey):
|
||||
with httpx.Client(base_url=f"{journey.base}/api/v1", timeout=30) as client:
|
||||
for method, path in (("GET", "/workflow"), ("POST", "/inventory/scan")):
|
||||
response = client.request(method, path)
|
||||
assert response.status_code == 401, path
|
||||
assert response.json()["error"]["code"] == "unauthenticated"
|
||||
# A session still has to be paid for with the operator's secret.
|
||||
assert client.get("/session", headers={"X-Access-Secret": "guessed"}).status_code == 401
|
||||
# Readiness stays open: the orchestrator's health check holds no session.
|
||||
assert client.get("/health/ready").status_code == 200
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_forged_forwarded_headers_cannot_smuggle_an_allowed_host_past_the_check(journey):
|
||||
"""Behind a proxy the app believes ``X-Forwarded-*`` — but only from the proxy.
|
||||
Nothing in this composition is a trusted proxy, so the claim is the client's."""
|
||||
with httpx.Client(base_url=f"{journey.base}/api/v1", timeout=30) as client:
|
||||
client.headers["X-CSRF-Token"] = (
|
||||
client.get("/session", headers={"X-Access-Secret": SECRET}).json()["csrf_token"]
|
||||
)
|
||||
# The hostname this deployment is reached under is accepted.
|
||||
assert client.get("/workflow", headers={"Host": HOSTNAME}).status_code == 200
|
||||
|
||||
forged = client.get(
|
||||
"/workflow",
|
||||
headers={"Host": "photos.evil.example", "X-Forwarded-Host": HOSTNAME},
|
||||
)
|
||||
assert forged.status_code == 403
|
||||
assert forged.json()["error"]["code"] == "host_not_allowed"
|
||||
# A forged protocol claim must not mark the session cookie as HTTPS-only
|
||||
# either — that would strand the operator's real, plain-HTTP session.
|
||||
bootstrap = httpx.get(
|
||||
f"{journey.base}/api/v1/session",
|
||||
headers={"X-Access-Secret": SECRET, "X-Forwarded-Proto": "https"},
|
||||
timeout=30,
|
||||
)
|
||||
assert "secure" not in bootstrap.headers["set-cookie"].lower()
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_a_path_that_leaves_the_mounted_library_is_refused(journey):
|
||||
"""The container's own filesystem is not the library. A symlink swapped under a
|
||||
known asset is the sharpest version of the question, because the database still
|
||||
points at a path inside the mount."""
|
||||
with closing(journey.client()) as client:
|
||||
asset = client.get("/inventory/assets", params={"limit": 200}).json()["items"][0]
|
||||
original = journey.library / Path(asset["current_path"]).relative_to("/library")
|
||||
kept = original.read_bytes()
|
||||
original.unlink()
|
||||
original.symlink_to("/etc/passwd")
|
||||
try:
|
||||
escaped = client.get(f"/assets/{asset['id']}/thumbnail", params={"size": 256})
|
||||
finally:
|
||||
original.unlink()
|
||||
original.write_bytes(kept)
|
||||
|
||||
assert escaped.status_code == 403
|
||||
assert escaped.json()["error"]["code"] == "path_not_allowed"
|
||||
assert "root:" not in escaped.text
|
||||
|
||||
# And a root that names nothing mounted is refused before the process serves.
|
||||
refused = journey.compose(
|
||||
"run",
|
||||
"--rm",
|
||||
"--no-deps",
|
||||
"--env",
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS=/srv/photos",
|
||||
"api",
|
||||
"serve",
|
||||
check=False,
|
||||
)
|
||||
assert refused.returncode == 5, refused.stdout + refused.stderr
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_no_secret_reaches_the_container_logs(journey):
|
||||
with httpx.Client(base_url=f"{journey.base}/api/v1", timeout=30) as client:
|
||||
client.get("/session", headers={"X-Access-Secret": "wrong-secret-attempt"})
|
||||
client.get("/session", headers={"X-Access-Secret": SECRET})
|
||||
logs = journey.logs()
|
||||
assert SECRET not in logs
|
||||
assert IMMICH_SENTINEL not in logs
|
||||
assert "wrong-secret-attempt" not in logs, "a rejected secret is still a secret"
|
||||
# The refusal itself is logged, so an operator can see the attempt.
|
||||
assert "access secret rejected" in logs
|
||||
|
||||
|
||||
@needs_compose
|
||||
def test_the_evidence_of_this_run_names_the_stack_it_was_produced_from(journey):
|
||||
"""A gate that cannot say what it ran against is an opinion. `docker compose ps`
|
||||
is the record: one API, one worker, one completed migration."""
|
||||
listing = [
|
||||
json.loads(line)
|
||||
for line in journey.compose("ps", "--all", "--format", "json").stdout.splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
services = {entry["Service"]: entry["State"] for entry in listing}
|
||||
assert services["api"] == "running" and services["worker"] == "running"
|
||||
assert services["migrate"] == "exited"
|
||||
|
||||
|
||||
if __name__ == "__main__": # a quick way to run just this file
|
||||
raise SystemExit(pytest.main([__file__, "-v", *sys.argv[1:]]))
|
||||
147
tests/integration/test_container_gate.py
Normal file
147
tests/integration/test_container_gate.py
Normal file
@@ -0,0 +1,147 @@
|
||||
"""US08-05: the container acceptance gate's own contract, checked without a daemon.
|
||||
|
||||
The gate itself needs Docker, a browser, and several minutes; running it from inside
|
||||
the suite would be a fork bomb with better manners. What is checkable offline is what
|
||||
makes it a *gate* rather than a long test run:
|
||||
|
||||
* it selects the container journeys by marker, so adding one is enough to put it in
|
||||
front of a deploy;
|
||||
* it accepts no skip at all — every reason a check would skip here (no daemon, no
|
||||
compose plugin, no browser) means the deployed runtime was not proven;
|
||||
* it retains checksummed evidence per run;
|
||||
* and CI runs it on `main`, which is what the publish step waits for.
|
||||
|
||||
The running proof is ``tests/e2e/test_phase_h_container.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.services import release
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
SUITE = REPO / "tests" / "e2e" / "test_phase_h_container.py"
|
||||
TEST_WORKFLOW = yaml.safe_load((REPO / ".gitea" / "workflows" / "test.yml").read_text())
|
||||
|
||||
|
||||
def _config(tmp_path) -> Config:
|
||||
for name in ("data", "lib"):
|
||||
(tmp_path / name).mkdir(exist_ok=True)
|
||||
return Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(tmp_path / "lib"),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ── what the gate runs ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_gate_selects_the_container_journeys_by_their_marker():
|
||||
assert release.CONTAINER_STAGES == (("container", ("tests/e2e", "-m", "phase_h")),)
|
||||
declared = [line for line in (REPO / "pyproject.toml").read_text().splitlines()
|
||||
if line.strip().startswith('"phase_h:')]
|
||||
assert declared, "an unregistered marker selects nothing and fails no gate"
|
||||
|
||||
|
||||
def test_every_journey_in_the_suite_carries_the_marker():
|
||||
"""A test in that file without the marker is a check the gate never runs."""
|
||||
source = SUITE.read_text()
|
||||
assert "pytestmark = [pytest.mark.phase_h, pytest.mark.container]" in source
|
||||
journeys = re.findall(r"^def (test_[a-z_]+)", source, re.MULTILINE)
|
||||
assert len(journeys) >= 4, journeys
|
||||
for required in ("browser", "upgrade", "kill", "secret"):
|
||||
assert any(required in name for name in journeys), (required, journeys)
|
||||
|
||||
|
||||
# ── no skip is an environment limit here ─────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_gate_accepts_no_skipped_check_at_all():
|
||||
assert release.CONTAINER_ALLOWED_SKIP_REASONS == ()
|
||||
docker_missing = release.StageResult(
|
||||
"container", [], 0, 0.1, "1 skipped", ["SKIPPED [1] x.py:1: no Docker daemon available"]
|
||||
)
|
||||
# The release gate tolerates exactly this one, because the image definition is
|
||||
# still checked offline. The container gate cannot: it is the deployment's proof.
|
||||
assert release.unexpected_skips([docker_missing]) == []
|
||||
assert release.unexpected_skips(
|
||||
[docker_missing], release.CONTAINER_ALLOWED_SKIP_REASONS
|
||||
) == ["SKIPPED [1] x.py:1: no Docker daemon available"]
|
||||
|
||||
|
||||
def test_a_skipped_check_fails_the_run_and_the_evidence_says_so(tmp_path):
|
||||
skipping = tmp_path / "test_skipping.py"
|
||||
skipping.write_text(
|
||||
"import pytest\n\n"
|
||||
"def test_x():\n"
|
||||
" pytest.skip('no Docker daemon available')\n"
|
||||
)
|
||||
evidence = tmp_path / "evidence"
|
||||
|
||||
report = release.run_gate(
|
||||
_config(tmp_path),
|
||||
output=evidence,
|
||||
stages=(("container", (str(skipping),)),),
|
||||
allowed_skips=release.CONTAINER_ALLOWED_SKIP_REASONS,
|
||||
)
|
||||
|
||||
assert report["ok"] is False
|
||||
assert report["failures"] == [], "the stage passed; the skip is what fails the gate"
|
||||
assert report["unexpected_skips"], report
|
||||
written = json.loads((evidence / release.REPORT_NAME).read_text())
|
||||
assert written["ok"] is False
|
||||
assert (evidence / "logs" / "container.log").exists()
|
||||
for line in (evidence / release.CHECKSUMS_NAME).read_text().splitlines():
|
||||
digest, name = line.split(" ", 1)
|
||||
assert release.sha256_file(evidence / name) == digest
|
||||
|
||||
|
||||
def test_the_same_run_passes_when_nothing_skips(tmp_path):
|
||||
passing = tmp_path / "test_passing.py"
|
||||
passing.write_text("def test_x():\n assert True\n")
|
||||
|
||||
report = release.run_gate(
|
||||
_config(tmp_path),
|
||||
output=tmp_path / "evidence",
|
||||
stages=(("container", (str(passing),)),),
|
||||
allowed_skips=release.CONTAINER_ALLOWED_SKIP_REASONS,
|
||||
)
|
||||
|
||||
assert report["ok"] is True and report["unexpected_skips"] == []
|
||||
assert report["revision"], "the evidence must say which commit it covers"
|
||||
|
||||
|
||||
# ── the command, and CI ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_command_is_documented_and_wired():
|
||||
from photo_pipeline.__main__ import main # noqa: F401 (import proves it loads)
|
||||
|
||||
assert "container-gate" in (REPO / "photo_pipeline" / "__main__.py").read_text()
|
||||
assert "container-gate" in (REPO / "README.md").read_text()
|
||||
|
||||
|
||||
def test_ci_runs_the_gate_on_main_and_keeps_its_evidence():
|
||||
job = TEST_WORKFLOW["jobs"]["container"]
|
||||
assert job["if"] == "gitea.event_name == 'push'", "pull requests have nothing to upgrade from"
|
||||
script = "\n".join(step["run"] for step in job["steps"] if "run" in step)
|
||||
assert "photo_pipeline container-gate" in script
|
||||
evidence = next(step for step in job["steps"] if "upload-artifact" in str(step.get("uses")))
|
||||
assert evidence["if"] == "always()", "a failed gate's logs are the ones worth keeping"
|
||||
assert evidence["with"]["path"] == "gate-evidence"
|
||||
|
||||
|
||||
def test_the_upgrade_journey_can_still_reach_the_previous_version():
|
||||
"""Without depth, `git archive HEAD~1` has nothing to build."""
|
||||
checkout = next(
|
||||
step for step in TEST_WORKFLOW["jobs"]["container"]["steps"] if "checkout" in str(step.get("uses"))
|
||||
)
|
||||
assert checkout["with"]["fetch-depth"] >= 2
|
||||
179
tests/integration/test_documentation.py
Normal file
179
tests/integration/test_documentation.py
Normal file
@@ -0,0 +1,179 @@
|
||||
"""US09-01: the documentation the application serves, checked without a browser.
|
||||
|
||||
What is checkable offline is everything that makes the manuals *navigable* rather
|
||||
than merely present: every page reachable, every internal link and anchor landing
|
||||
somewhere real, the vendored renderer being the file that was pinned, and the
|
||||
application actually serving `docs/` in both the working copy and the image.
|
||||
|
||||
The rendering itself — marked, mermaid, anchors, link rewriting — is proven in a
|
||||
real browser by ``tests/e2e/test_docs_ui.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.api.security import DEFAULT_HEADERS
|
||||
from photo_pipeline.config import Config
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DOCS = REPO / "docs"
|
||||
VENDOR = REPO / "frontend" / "js" / "vendor"
|
||||
INDEX = DOCS / "index.md"
|
||||
|
||||
MARKDOWN_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)")
|
||||
HEADING = re.compile(r"^#{1,6}\s+(.+?)\s*$", re.MULTILINE)
|
||||
FENCE = re.compile(r"^```.*?^```", re.MULTILINE | re.DOTALL)
|
||||
|
||||
|
||||
def pages() -> list[Path]:
|
||||
return sorted(DOCS.rglob("*.md"))
|
||||
|
||||
|
||||
def body(path: Path) -> str:
|
||||
"""A document without its fenced code, so an example link in a shell snippet is
|
||||
not mistaken for a link the reader can follow."""
|
||||
return FENCE.sub("", path.read_text())
|
||||
|
||||
|
||||
def slug(text: str) -> str:
|
||||
"""The heading-anchor rule, mirroring ``frontend/js/docs.js``.
|
||||
|
||||
Deliberately duplicated: this check has to run without a browser. The browser
|
||||
test is the authority on the real behaviour, and it asserts the same anchors.
|
||||
"""
|
||||
cleaned = re.sub(r"[^\w\s-]", "", text.lower(), flags=re.UNICODE).strip()
|
||||
return re.sub(r"[\s-]+", "-", cleaned).strip("-") or "section"
|
||||
|
||||
|
||||
def anchors(path: Path) -> set[str]:
|
||||
found: dict[str, int] = {}
|
||||
for heading in HEADING.findall(body(path)):
|
||||
# Markdown emphasis and inline code are not part of the rendered text.
|
||||
plain = re.sub(r"[*`_]", "", heading)
|
||||
base = slug(plain)
|
||||
found[base] = found.get(base, 0) + 1
|
||||
return {name if index == 1 else f"{name}-{index}" for name, count in found.items() for index in range(1, count + 1)}
|
||||
|
||||
|
||||
def internal_links(path: Path) -> list[str]:
|
||||
return [
|
||||
href
|
||||
for href in MARKDOWN_LINK.findall(body(path))
|
||||
if not re.match(r"^[a-z][a-z\d+.-]*:", href, re.IGNORECASE) and not href.startswith("//")
|
||||
]
|
||||
|
||||
|
||||
# ── the documentation tree ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_index_exists_and_names_every_page():
|
||||
"""A page nobody links to is a page nobody reads."""
|
||||
assert INDEX.is_file()
|
||||
listed = {
|
||||
(INDEX.parent / href.split("#")[0]).resolve()
|
||||
for href in internal_links(INDEX)
|
||||
if href.split("#")[0].endswith(".md")
|
||||
}
|
||||
unreachable = [page.name for page in pages() if page != INDEX and page.resolve() not in listed]
|
||||
assert unreachable == [], f"not linked from the index: {unreachable}"
|
||||
|
||||
|
||||
def test_every_internal_link_and_anchor_resolves():
|
||||
broken: list[str] = []
|
||||
for page in pages():
|
||||
for href in internal_links(page):
|
||||
target, _, anchor = href.partition("#")
|
||||
destination = page if not target else (page.parent / target).resolve()
|
||||
if not destination.is_file():
|
||||
broken.append(f"{page.name} → {href} (no such file)")
|
||||
continue
|
||||
if anchor and anchor not in anchors(destination):
|
||||
broken.append(f"{page.name} → {href} (no such heading)")
|
||||
assert broken == [], broken
|
||||
|
||||
|
||||
def test_every_page_starts_with_one_title():
|
||||
for page in pages():
|
||||
titles = [line for line in page.read_text().splitlines() if line.startswith("# ")]
|
||||
assert len(titles) == 1, f"{page.name} has {len(titles)} top-level titles"
|
||||
|
||||
|
||||
# ── the vendored renderer ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_vendored_libraries_are_the_files_that_were_pinned():
|
||||
"""A vendored dependency without a recorded checksum is a dependency nobody is
|
||||
reviewing. Replacing one has to be a visible change to this manifest."""
|
||||
manifest = json.loads((VENDOR / "VERSIONS.json").read_text())
|
||||
recorded = {entry["file"]: entry for entry in manifest["libraries"]}
|
||||
|
||||
on_disk = {path.name for path in VENDOR.iterdir() if path.suffix in (".js", ".mjs")}
|
||||
assert on_disk == set(recorded), f"unrecorded vendored files: {on_disk ^ set(recorded)}"
|
||||
|
||||
for name, entry in recorded.items():
|
||||
digest = hashlib.sha256((VENDOR / name).read_bytes()).hexdigest()
|
||||
assert digest == entry["sha256"], f"{name} does not match the pinned checksum"
|
||||
assert entry["version"] in entry["url"], f"{name}: the pinned URL and version disagree"
|
||||
assert entry["license"], f"{name}: no license recorded"
|
||||
|
||||
|
||||
def test_the_documentation_view_is_wired_into_the_shell():
|
||||
shell = (REPO / "frontend" / "index.html").read_text()
|
||||
assert 'data-nav="docs"' in shell, "the manuals are unreachable from the navigation"
|
||||
assert '"/docs"' in (REPO / "frontend" / "js" / "app.js").read_text()
|
||||
|
||||
|
||||
# ── the policy the view runs under ───────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_script_boundary_is_unchanged_and_only_style_was_relaxed():
|
||||
"""The measured cost of rendering diagrams in the browser, held to that cost.
|
||||
|
||||
mermaid needs to style the SVG it builds, which `style-src 'self'` refuses. It
|
||||
does not need to execute generated code, so `script-src` must never acquire the
|
||||
escape hatch that would let injected markup run.
|
||||
"""
|
||||
policy = DEFAULT_HEADERS["content-security-policy"]
|
||||
directives = {
|
||||
part.split(" ")[0]: part.split(" ")[1:]
|
||||
for part in (piece.strip() for piece in policy.split(";"))
|
||||
if part
|
||||
}
|
||||
assert directives["script-src"] == ["'self'"]
|
||||
assert "'unsafe-inline'" in directives["style-src"]
|
||||
for directive in ("default-src", "img-src", "connect-src", "font-src"):
|
||||
assert "'self'" in directives[directive]
|
||||
assert "'unsafe-inline'" not in directives[directive]
|
||||
assert directives["frame-ancestors"] == ["'none'"]
|
||||
|
||||
|
||||
# ── serving it ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_application_serves_the_markdown_without_a_session(tmp_path):
|
||||
"""Whoever cannot get past the access secret is exactly who needs these pages."""
|
||||
config = Config.from_env({"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data")})
|
||||
(tmp_path / "data").mkdir()
|
||||
with TestClient(create_app(config), raise_server_exceptions=False) as client:
|
||||
response = client.get("/docs/index.md", headers={"cookie": ""})
|
||||
assert response.status_code == 200
|
||||
assert "Photo Pipeline documentation" in response.text
|
||||
assert client.get("/docs/overview.md").status_code == 200
|
||||
assert client.get("/docs/nothing-here.md").status_code == 404
|
||||
# The mount is a directory, not a path parameter: nothing above it is reachable.
|
||||
assert client.get("/docs/../pyproject.toml").status_code in (307, 404)
|
||||
assert client.get("/docs/%2e%2e/pyproject.toml").status_code == 404
|
||||
|
||||
|
||||
def test_the_image_ships_the_documentation_it_serves():
|
||||
"""An image without `docs/` serves an empty manual — and the build context is
|
||||
deny-by-default, so a new directory is excluded until it is named."""
|
||||
assert "\n!docs\n" in (REPO / ".dockerignore").read_text()
|
||||
assert "COPY docs ./docs" in (REPO / "Dockerfile").read_text()
|
||||
@@ -189,10 +189,21 @@
|
||||
],
|
||||
"US08-04": [
|
||||
"tests/integration/test_deploy_workflows.py"
|
||||
],
|
||||
"US08-05": [
|
||||
"tests/integration/test_container_gate.py",
|
||||
"tests/e2e/test_phase_h_container.py"
|
||||
],
|
||||
"US09-01": [
|
||||
"tests/integration/test_documentation.py",
|
||||
"tests/e2e/test_docs_ui.py"
|
||||
]
|
||||
},
|
||||
"planned": [
|
||||
"US08-05"
|
||||
"US09-02",
|
||||
"US09-03",
|
||||
"US09-04",
|
||||
"US09-05"
|
||||
],
|
||||
"_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user