US09-05: Automate Documentation Acceptance #111

Merged
domverse merged 1 commits from us/US09-05-automate-documentation-acceptance into main 2026-08-24 00:23:00 +02:00
20 changed files with 427 additions and 20 deletions

View File

@@ -101,3 +101,39 @@ jobs:
with:
name: container-gate-${{ gitea.sha }}
path: gate-evidence
# The manuals, checked against the application they describe (US09-05). Runs on
# every pull request as well as `main`: documentation drifts by the same commits
# that change behaviour, and telling the author while the change is still open is
# the only time the fix is cheap.
documentation:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install the application and its test dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e '.[test]'
python -m playwright install --with-deps chromium
- name: Documentation gate
# One command: every phase_i check, offline and in the browser, with the
# evidence retained. It accepts no skip — a check that did not run is a page
# nobody compared with the code.
env:
PHOTO_PIPELINE_DATA_DIR: ${{ gitea.workspace }}/docs-gate-data
run: python -m photo_pipeline docs-gate --output docs-evidence
- name: Keep the evidence
if: always()
uses: actions/upload-artifact@v3
with:
name: docs-gate-${{ gitea.sha }}
path: docs-evidence

View File

@@ -4,6 +4,24 @@ Integrated, restart-safe photo analysis, duplicate review, metadata, upload, and
archive workflow. Planning lives in `INTEGRATED_PIPELINE_CONCEPT.md` and
`delivery_backlog/`.
## Documentation
The manuals live in [`docs/`](docs/index.md) and are the same files the running
application serves at `/app/#/docs`:
| | |
|---|---|
| [Overview](docs/overview.md) | what it does and the rules it will not break |
| [Installation and operations](docs/installation.md) | host and container install, every setting, backup and restore, every refusal |
| [A guided first pass](docs/first-pass.md) | one library from scan to verified upload |
| [The stages](docs/index.md) | one illustrated page each, from inventory to archive |
| [Errors and refusals](docs/errors.md) | every error code, cause, and remedy |
| [Architecture](docs/architecture.md) | diagrams, module map, journals, invariants |
This README stays the repository's own notes — how the code is built, tested, and
released. Anything an operator or a user needs belongs in `docs/`, and one gate
(`python -m photo_pipeline docs-gate`) keeps those pages honest against the code.
## Application (`photo_pipeline`)
The target application lives in `photo_pipeline/` (FastAPI + SQLAlchemy + Alembic).

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 127 KiB

After

Width:  |  Height:  |  Size: 128 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 56 KiB

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 127 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 101 KiB

After

Width:  |  Height:  |  Size: 101 KiB

View File

@@ -42,6 +42,25 @@ what it changes on disk or on the server, and what it refuses.
- [Recovery](recovery.md) — what the application resolves by itself, and what needs
you.
## Keeping these pages true
Documentation that disagrees with the application is worse than none, because it is
trusted. So one command compares them:
```bash
python -m photo_pipeline docs-gate
```
It runs every `phase_i` check — dead links and anchors, pages nobody links to, images
nobody shows, settings, commands, exit codes, error codes, and states that no longer
exist in the code, the pages rendering in a real browser without a console or policy
error, and the screenshots still showing what the application shows. It retains its
evidence and **accepts no skipped check**: a check that did not run is a page nobody
compared. CI runs it on every pull request.
The repository's own build, test, and release notes live in the
[README](../README.md).
## Conventions
A page tells you what a stage **changes on disk or on the server** before it tells

View File

@@ -1,5 +1,5 @@
"""Application management CLI:
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,dry-run,approve-dry-run}``.
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,docs-gate,dry-run,approve-dry-run}``.
``serve`` and ``worker`` take the library process lock for their role (US07-05):
two workers, or the frozen CLI running beside the app, would each be safe on their
@@ -78,6 +78,12 @@ def main(argv: Sequence[str] | None = None) -> int:
container_cmd.add_argument(
"--output", help="Evidence directory (default: data/container-gate/<stamp>)"
)
docs_cmd = commands.add_parser(
"docs-gate",
help="Check the manuals against the application they describe and keep the "
"evidence (US09-05)",
)
docs_cmd.add_argument("--output", help="Evidence directory (default: data/docs-gate/<stamp>)")
dry_cmd = commands.add_parser(
"dry-run", help="Read-only reconciliation of the configured library (US07-07)"
)
@@ -188,6 +194,30 @@ def main(argv: Sequence[str] | None = None) -> int:
)
return 0 if report["ok"] else 1
if args.command == "docs-gate":
from datetime import datetime, timezone
from photo_pipeline.services import release
# Documentation that disagrees with the application is worse than none: it is
# trusted. So this gate accepts no skip either — a check that did not run is a
# page nobody compared with the code.
output = args.output or Path(config.data_dir) / "docs-gate" / datetime.now(
timezone.utc
).strftime("%Y%m%dT%H%M%SZ")
report = release.run_gate(
config,
output=output,
stages=release.DOCS_STAGES,
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
)
print(
json.dumps(
{k: v for k, v in report.items() if k not in ("stages", "matrix")}, indent=2
)
)
return 0 if report["ok"] else 1
if args.command == "dry-run":
from photo_pipeline.services import release

View File

@@ -71,6 +71,18 @@ CONTAINER_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
# would skip here — no daemon, no compose, no browser — means it was not proven.
CONTAINER_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
# The documentation gate (US09-05): the manuals, held to the application. Selected by
# marker across the whole suite, because each check lives beside what it describes —
# the offline contracts under tests/integration, the browser and screenshot journeys
# under tests/e2e. Adding a phase_i test is enough to put it in front of a release.
DOCS_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
("documentation", ("tests", "-m", "phase_i")),
)
# Also nothing. Every check here is either offline or needs the browser the rest of
# the suite already needs, so a skip means the manuals were not compared with the code.
DOCS_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
class ReleaseError(RuntimeError):
pass

View File

@@ -54,4 +54,5 @@ markers = [
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
"container: builds and runs the container image and its composition (US08-02, US08-03) — needs a Docker daemon, the compose plugin, and the network",
"phase_h: Phase H container deployment acceptance (US08-05) — the browser, upgrade, restart, and security journeys against the composed stack",
"phase_i: Phase I documentation acceptance (US09-05) — the manuals checked against the application, in the browser and offline",
]

View File

@@ -18,6 +18,9 @@ import pytest
from tests.e2e._pipeline_harness import Server, seed_library
# Every check here belongs to the documentation gate (US09-05).
pytestmark = pytest.mark.phase_i
@pytest.fixture(scope="module")
def server(tmp_path_factory):

View File

@@ -24,10 +24,13 @@ from contextlib import closing
from pathlib import Path
import pytest
from PIL import Image
from playwright.sync_api import TimeoutError as PlaywrightTimeout
from tests.conftest import session_client
from tests.e2e._pipeline_harness import (
Server,
approve_album,
seed_album,
start_worker,
wait_until,
@@ -39,21 +42,29 @@ IMAGES = DOCS / "images"
ALBUM = "rome"
VIEWPORT = {"width": 1280, "height": 900}
# One per stage page of the manual: the route to visit, and the element whose
# presence means the view has actually finished rendering.
# One per stage page of the manual: the route, the heading that view renders, and the
# element whose presence means it has finished.
#
# The heading matters more than it looks. Every route replaces the same container, so
# waiting for "an h1" matches the *previous* view's heading and photographs the screen
# you just left — which is how the statistics page first shipped a picture of the
# archive view.
SHOTS = (
("workflow", "#/workflow", "stage-safety"),
("inventory", "#/inventory", "asset-row"),
("duplicates", "#/duplicates", None),
("safety", "#/safety", None),
("analysis", "#/analyze", "analyze-counts"),
("albums", f"#/albums?album={ALBUM}", "suggested-name"),
("renames", "#/renames", None),
("uploads", "#/uploads", "upload-scope"),
("archive", "#/archive", "archive-locations"),
("statistics", "#/stats", None),
("workflow", "#/workflow", "Workflow", "stage-safety"),
("inventory", "#/inventory", "Inventory", "asset-row"),
("duplicates", "#/duplicates", "Duplicate clusters", None),
("safety", "#/safety", "Safety review", None),
("analysis", "#/analyze", "Analyze", "analyze-counts"),
("albums", f"#/albums?album={ALBUM}", "Albums", "suggested-name"),
("renames", "#/renames", "Renames", None),
("uploads", "#/uploads", "Upload", "upload-scope"),
("archive", "#/archive", "Archive", "archive-locations"),
("statistics", "#/stats", "Stats", None),
)
# Every check here belongs to the documentation gate (US09-05).
pytestmark = pytest.mark.phase_i
def writing() -> bool:
return os.environ.get("PHOTO_PIPELINE_WRITE_SCREENSHOTS") == "1"
@@ -90,17 +101,23 @@ def _prepare(base: str, seeded) -> None:
).raise_for_status()
wait_until(lambda: client.get("/api/v1/workflow").status_code == 200)
# An approved name and a built — deliberately unapplied — plan, so the renames
# screenshot shows the preview its page describes rather than an empty state.
# Building a plan moves nothing; applying it is what would, and nothing here does.
approve_album(base, album=ALBUM, name="2019 — Rome")
with closing(session_client(base, timeout=30)) as client:
client.post("/api/v1/rename-plans").raise_for_status()
def _capture(page, server, target: Path) -> list[str]:
target.mkdir(parents=True, exist_ok=True)
page.set_viewport_size(VIEWPORT)
written = []
for name, route, ready in SHOTS:
for name, route, heading, ready in SHOTS:
page.goto(f"{server.base}/app/{route}")
page.locator("main h1", has_text=heading).first.wait_for(timeout=30_000)
if ready:
page.get_by_test_id(ready).first.wait_for(timeout=30_000)
else:
page.locator("main h1").first.wait_for(timeout=30_000)
page.screenshot(path=str(target / f"{name}.png"))
written.append(name)
return written
@@ -110,7 +127,7 @@ def test_the_generator_produces_every_screenshot_the_manual_references(page, sta
destination = IMAGES if writing() else tmp_path / "images"
written = _capture(page, stack, destination)
assert sorted(written) == sorted(name for name, _, _ in SHOTS)
assert sorted(written) == sorted(name for name, *_ in SHOTS)
for name in written:
produced = destination / f"{name}.png"
assert produced.stat().st_size > 5_000, f"{name}.png is too small to be a view"
@@ -133,6 +150,60 @@ def test_no_screenshot_shows_a_real_path_or_a_secret(page, stack, tmp_path):
assert all(ALBUM in path or "images" in path for path in paths)
def test_the_committed_screenshots_still_show_what_the_application_shows(page, stack, tmp_path):
"""A UI change that invalidates the manual should be a red build, not a discovery
months later by somebody following a picture of a screen that no longer exists.
The tolerance is on *content*, not pixels. Comparing the committed PNGs with a
fresh capture was tried first and rejected: PNG output is not reproducible, font
rasterisation differs between the machine that generated an image and the machine
running the gate, and several views legitimately print the fixture library's
absolute path, which is a fresh temporary directory every run. A pixel or
perceptual comparison therefore fails for reasons that have nothing to do with
the manual being wrong — and a check that cries wolf gets deleted.
What actually invalidates a screenshot is the view no longer showing what its
page says it shows. That is what is compared here: the heading, the elements the
page describes, and the fact that the image was captured at the same size.
ponytail: content comparison, not pixels. A visual-diff service with per-platform
baselines is the upgrade if cosmetic regressions ever need catching too.
"""
if not IMAGES.is_dir():
pytest.skip("no screenshots have been generated into the repository yet")
fresh = tmp_path / "fresh"
_capture(page, stack, fresh)
drifted = []
for name, route, heading, ready in SHOTS:
committed = IMAGES / f"{name}.png"
if not committed.is_file():
drifted.append(f"{name}: never committed")
continue
with Image.open(committed) as image:
if image.size != (VIEWPORT["width"], VIEWPORT["height"]):
drifted.append(f"{name}: committed at {image.size}, captured at {VIEWPORT}")
# The views render asynchronously, so each check waits rather than asking a
# question the page has not finished answering.
page.goto(f"{stack.base}/app/{route}")
try:
page.locator("main h1", has_text=heading).first.wait_for(timeout=15_000)
except PlaywrightTimeout:
drifted.append(f"{name}: the view no longer shows the heading '{heading}'")
continue
if ready:
try:
page.get_by_test_id(ready).first.wait_for(timeout=15_000)
except PlaywrightTimeout:
drifted.append(f"{name}: the view no longer renders '{ready}'")
assert drifted == [], (
f"the manual's screenshots no longer match the application: {drifted}. "
"Regenerate them with PHOTO_PIPELINE_WRITE_SCREENSHOTS=1 and review the result."
)
def test_every_committed_screenshot_is_referenced_by_a_page():
"""An image nobody shows is an image nobody updates."""
if not IMAGES.is_dir():

View File

@@ -14,10 +14,15 @@ from __future__ import annotations
import re
from pathlib import Path
import pytest
import photo_pipeline.models # noqa: F401 (registers every table on Base.metadata)
from photo_pipeline.db import Base
from photo_pipeline.services import jobs, rename_journal, upload_batches
# Every check here belongs to the documentation gate (US09-05).
pytestmark = pytest.mark.phase_i
REPO = Path(__file__).resolve().parents[2]
PACKAGE = REPO / "photo_pipeline"
OVERVIEW = REPO / "docs" / "architecture.md"

View File

@@ -0,0 +1,196 @@
"""US09-05: the documentation gate's own contract, checked without a browser.
The gate itself needs a browser and a server and takes minutes; what is checkable
offline is what makes it a *gate* rather than a long test run:
* it selects every documentation check by marker, so adding one is enough to put it
in front of a release;
* it accepts no skip at all — a check that did not run is a page nobody compared
with the code;
* it retains checksummed evidence per run;
* CI runs it on pull requests, which is when a fix is still cheap.
The checks it runs are the four suites written by US09-01 through US09-04.
"""
from __future__ import annotations
import json
import re
from pathlib import Path
import pytest
import yaml
from photo_pipeline.config import Config
from photo_pipeline.services import release
REPO = Path(__file__).resolve().parents[2]
DOCS = REPO / "docs"
TEST_WORKFLOW = yaml.safe_load((REPO / ".gitea" / "workflows" / "test.yml").read_text())
pytestmark = pytest.mark.phase_i
# The suites the gate exists to run. Each is mapped to a story in the traceability
# matrix, so this list is also what stops one being quietly dropped.
DOCUMENTATION_SUITES = (
"tests/integration/test_documentation.py",
"tests/integration/test_installation_manual.py",
"tests/integration/test_architecture_overview.py",
"tests/integration/test_user_manual.py",
"tests/integration/test_docs_gate.py",
"tests/e2e/test_docs_ui.py",
"tests/e2e/test_user_manual_screenshots.py",
)
def _config(tmp_path) -> Config:
(tmp_path / "data").mkdir(exist_ok=True)
return Config.from_env({"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data")})
# ── what the gate runs ───────────────────────────────────────────────────────
def test_the_gate_selects_every_documentation_check_by_marker():
assert release.DOCS_STAGES == (("documentation", ("tests", "-m", "phase_i")),)
declared = [
line
for line in (REPO / "pyproject.toml").read_text().splitlines()
if line.strip().startswith('"phase_i:')
]
assert declared, "an unregistered marker selects nothing and fails no gate"
def test_every_documentation_suite_carries_the_marker():
"""A documentation test without the marker is a check the gate never runs."""
unmarked = [
suite
for suite in DOCUMENTATION_SUITES
if "pytest.mark.phase_i" not in (REPO / suite).read_text()
]
assert unmarked == [], f"suites missing the phase_i marker: {unmarked}"
def test_the_marker_selects_nothing_outside_the_documentation_suites():
"""Scope matters: the gate's promise is that it ran *the documentation checks*."""
marked = sorted(
str(path.relative_to(REPO))
for path in (REPO / "tests").rglob("test_*.py")
if "pytest.mark.phase_i" in path.read_text()
)
assert marked == sorted(DOCUMENTATION_SUITES)
# ── no skip is an environment limit here ─────────────────────────────────────
def test_the_gate_accepts_no_skipped_check_at_all():
assert release.DOCS_ALLOWED_SKIP_REASONS == ()
skipped = release.StageResult(
"documentation", [], 0, 0.1, "1 skipped", ["SKIPPED [1] x.py:1: exiftool not installed"]
)
# The release gate tolerates this one because it describes the machine. The
# documentation gate cannot: nothing here depends on the machine.
assert release.unexpected_skips([skipped]) == []
assert release.unexpected_skips([skipped], release.DOCS_ALLOWED_SKIP_REASONS) == [
"SKIPPED [1] x.py:1: exiftool not installed"
]
def test_a_skipped_check_fails_the_run_and_the_evidence_says_so(tmp_path):
skipping = tmp_path / "test_skipping.py"
skipping.write_text(
"import pytest\n\ndef test_x():\n pytest.skip('exiftool not installed')\n"
)
evidence = tmp_path / "evidence"
report = release.run_gate(
_config(tmp_path),
output=evidence,
stages=(("documentation", (str(skipping),)),),
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
)
assert report["ok"] is False
assert report["failures"] == [], "the stage passed; the skip is what fails the gate"
assert report["unexpected_skips"], report
written = json.loads((evidence / release.REPORT_NAME).read_text())
assert written["ok"] is False
assert (evidence / "logs" / "documentation.log").exists()
for line in (evidence / release.CHECKSUMS_NAME).read_text().splitlines():
digest, name = line.split(" ", 1)
assert release.sha256_file(evidence / name) == digest
def test_a_broken_documentation_check_fails_the_gate(tmp_path):
"""The failure mode that matters: a page and the code disagreeing."""
failing = tmp_path / "test_disagreement.py"
failing.write_text(
"def test_the_manual_matches_the_code():\n"
" documented = {'lock_held'}\n"
" emitted = {'lock_held', 'renamed_since'}\n"
" assert emitted <= documented\n"
)
report = release.run_gate(
_config(tmp_path),
output=tmp_path / "evidence",
stages=(("documentation", (str(failing),)),),
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
)
assert report["ok"] is False
assert report["failures"], report
def test_a_clean_tree_passes(tmp_path):
passing = tmp_path / "test_passing.py"
passing.write_text("def test_x():\n assert True\n")
report = release.run_gate(
_config(tmp_path),
output=tmp_path / "evidence",
stages=(("documentation", (str(passing),)),),
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
)
assert report["ok"] is True and report["unexpected_skips"] == []
assert report["revision"], "the evidence must say which commit it covers"
# ── the command, and CI ──────────────────────────────────────────────────────
def test_the_command_is_documented_and_wired():
assert "docs-gate" in (REPO / "photo_pipeline" / "__main__.py").read_text()
assert "docs-gate" in (DOCS / "index.md").read_text()
def test_ci_runs_the_gate_on_pull_requests_and_keeps_its_evidence():
job = TEST_WORKFLOW["jobs"]["documentation"]
# No `if:` — documentation drifts on the same commits that change behaviour, and
# the pull request is when saying so is still cheap.
assert "if" not in job
assert "pull_request" in TEST_WORKFLOW[True] or "pull_request" in TEST_WORKFLOW.get("on", {})
script = "\n".join(step["run"] for step in job["steps"] if "run" in step)
assert "photo_pipeline docs-gate" in script
evidence = next(step for step in job["steps"] if "upload-artifact" in str(step.get("uses")))
assert evidence["if"] == "always()", "a failed gate's evidence is the one worth keeping"
# ── the pages and the repository point at each other ─────────────────────────
def test_the_readme_and_the_documentation_index_link_to_each_other():
"""Neither may become the forgotten copy."""
readme = (REPO / "README.md").read_text()
index = (DOCS / "index.md").read_text()
assert "docs/index.md" in readme
assert re.search(r"\.\./README\.md", index), "the index does not link back to the README"
def test_every_documentation_story_is_mapped_in_the_traceability_matrix():
matrix = json.loads((REPO / "tests" / "story_traceability.json").read_text())["stories"]
for story in ("US09-01", "US09-02", "US09-03", "US09-04", "US09-05"):
assert matrix.get(story), f"{story} is not mapped to any test"

View File

@@ -16,12 +16,17 @@ import json
import re
from pathlib import Path
import pytest
from starlette.testclient import TestClient
from photo_pipeline.api.app import create_app
from photo_pipeline.api.security import DEFAULT_HEADERS
from photo_pipeline.config import Config
# Every check here belongs to the documentation gate (US09-05).
pytestmark = pytest.mark.phase_i
REPO = Path(__file__).resolve().parents[2]
DOCS = REPO / "docs"
VENDOR = REPO / "frontend" / "js" / "vendor"

View File

@@ -16,8 +16,13 @@ import sys
from functools import lru_cache
from pathlib import Path
import pytest
from photo_pipeline.config import ENV_PREFIX, LEGACY_ALIASES, Config
# Every check here belongs to the documentation gate (US09-05).
pytestmark = pytest.mark.phase_i
REPO = Path(__file__).resolve().parents[2]
MANUAL = REPO / "docs" / "installation.md"
MAIN = REPO / "photo_pipeline" / "__main__.py"

View File

@@ -14,6 +14,8 @@ from __future__ import annotations
import re
from pathlib import Path
import pytest
REPO = Path(__file__).resolve().parents[2]
DOCS = REPO / "docs"
STAGES = DOCS / "stages"
@@ -55,6 +57,9 @@ STAGE_PAGES = (
"diagnostics",
)
# Every check here belongs to the documentation gate (US09-05).
pytestmark = pytest.mark.phase_i
def emitted_codes() -> set[str]:
found: set[str] = set()

View File

@@ -208,10 +208,11 @@
"US09-04": [
"tests/integration/test_user_manual.py",
"tests/e2e/test_user_manual_screenshots.py"
],
"US09-05": [
"tests/integration/test_docs_gate.py"
]
},
"planned": [
"US09-05"
],
"planned": [],
"_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
}