US09-05: Automate Documentation Acceptance
@@ -101,3 +101,39 @@ jobs:
|
|||||||
with:
|
with:
|
||||||
name: container-gate-${{ gitea.sha }}
|
name: container-gate-${{ gitea.sha }}
|
||||||
path: gate-evidence
|
path: gate-evidence
|
||||||
|
|
||||||
|
# The manuals, checked against the application they describe (US09-05). Runs on
|
||||||
|
# every pull request as well as `main`: documentation drifts by the same commits
|
||||||
|
# that change behaviour, and telling the author while the change is still open is
|
||||||
|
# the only time the fix is cheap.
|
||||||
|
documentation:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: '3.12'
|
||||||
|
|
||||||
|
- name: Install the application and its test dependencies
|
||||||
|
run: |
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
python -m pip install -e '.[test]'
|
||||||
|
python -m playwright install --with-deps chromium
|
||||||
|
|
||||||
|
- name: Documentation gate
|
||||||
|
# One command: every phase_i check, offline and in the browser, with the
|
||||||
|
# evidence retained. It accepts no skip — a check that did not run is a page
|
||||||
|
# nobody compared with the code.
|
||||||
|
env:
|
||||||
|
PHOTO_PIPELINE_DATA_DIR: ${{ gitea.workspace }}/docs-gate-data
|
||||||
|
run: python -m photo_pipeline docs-gate --output docs-evidence
|
||||||
|
|
||||||
|
- name: Keep the evidence
|
||||||
|
if: always()
|
||||||
|
uses: actions/upload-artifact@v3
|
||||||
|
with:
|
||||||
|
name: docs-gate-${{ gitea.sha }}
|
||||||
|
path: docs-evidence
|
||||||
|
|||||||
18
README.md
@@ -4,6 +4,24 @@ Integrated, restart-safe photo analysis, duplicate review, metadata, upload, and
|
|||||||
archive workflow. Planning lives in `INTEGRATED_PIPELINE_CONCEPT.md` and
|
archive workflow. Planning lives in `INTEGRATED_PIPELINE_CONCEPT.md` and
|
||||||
`delivery_backlog/`.
|
`delivery_backlog/`.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
The manuals live in [`docs/`](docs/index.md) and are the same files the running
|
||||||
|
application serves at `/app/#/docs`:
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| [Overview](docs/overview.md) | what it does and the rules it will not break |
|
||||||
|
| [Installation and operations](docs/installation.md) | host and container install, every setting, backup and restore, every refusal |
|
||||||
|
| [A guided first pass](docs/first-pass.md) | one library from scan to verified upload |
|
||||||
|
| [The stages](docs/index.md) | one illustrated page each, from inventory to archive |
|
||||||
|
| [Errors and refusals](docs/errors.md) | every error code, cause, and remedy |
|
||||||
|
| [Architecture](docs/architecture.md) | diagrams, module map, journals, invariants |
|
||||||
|
|
||||||
|
This README stays the repository's own notes — how the code is built, tested, and
|
||||||
|
released. Anything an operator or a user needs belongs in `docs/`, and one gate
|
||||||
|
(`python -m photo_pipeline docs-gate`) keeps those pages honest against the code.
|
||||||
|
|
||||||
## Application (`photo_pipeline`)
|
## Application (`photo_pipeline`)
|
||||||
|
|
||||||
The target application lives in `photo_pipeline/` (FastAPI + SQLAlchemy + Alembic).
|
The target application lives in `photo_pipeline/` (FastAPI + SQLAlchemy + Alembic).
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 64 KiB After Width: | Height: | Size: 64 KiB |
|
Before Width: | Height: | Size: 127 KiB After Width: | Height: | Size: 128 KiB |
|
Before Width: | Height: | Size: 56 KiB After Width: | Height: | Size: 55 KiB |
|
Before Width: | Height: | Size: 30 KiB After Width: | Height: | Size: 78 KiB |
|
Before Width: | Height: | Size: 127 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 101 KiB After Width: | Height: | Size: 101 KiB |
@@ -42,6 +42,25 @@ what it changes on disk or on the server, and what it refuses.
|
|||||||
- [Recovery](recovery.md) — what the application resolves by itself, and what needs
|
- [Recovery](recovery.md) — what the application resolves by itself, and what needs
|
||||||
you.
|
you.
|
||||||
|
|
||||||
|
## Keeping these pages true
|
||||||
|
|
||||||
|
Documentation that disagrees with the application is worse than none, because it is
|
||||||
|
trusted. So one command compares them:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m photo_pipeline docs-gate
|
||||||
|
```
|
||||||
|
|
||||||
|
It runs every `phase_i` check — dead links and anchors, pages nobody links to, images
|
||||||
|
nobody shows, settings, commands, exit codes, error codes, and states that no longer
|
||||||
|
exist in the code, the pages rendering in a real browser without a console or policy
|
||||||
|
error, and the screenshots still showing what the application shows. It retains its
|
||||||
|
evidence and **accepts no skipped check**: a check that did not run is a page nobody
|
||||||
|
compared. CI runs it on every pull request.
|
||||||
|
|
||||||
|
The repository's own build, test, and release notes live in the
|
||||||
|
[README](../README.md).
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
A page tells you what a stage **changes on disk or on the server** before it tells
|
A page tells you what a stage **changes on disk or on the server** before it tells
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
"""Application management CLI:
|
"""Application management CLI:
|
||||||
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,dry-run,approve-dry-run}``.
|
``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores,backup,verify-backup,restore,diagnostics,benchmark,release-gate,container-gate,docs-gate,dry-run,approve-dry-run}``.
|
||||||
|
|
||||||
``serve`` and ``worker`` take the library process lock for their role (US07-05):
|
``serve`` and ``worker`` take the library process lock for their role (US07-05):
|
||||||
two workers, or the frozen CLI running beside the app, would each be safe on their
|
two workers, or the frozen CLI running beside the app, would each be safe on their
|
||||||
@@ -78,6 +78,12 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
container_cmd.add_argument(
|
container_cmd.add_argument(
|
||||||
"--output", help="Evidence directory (default: data/container-gate/<stamp>)"
|
"--output", help="Evidence directory (default: data/container-gate/<stamp>)"
|
||||||
)
|
)
|
||||||
|
docs_cmd = commands.add_parser(
|
||||||
|
"docs-gate",
|
||||||
|
help="Check the manuals against the application they describe and keep the "
|
||||||
|
"evidence (US09-05)",
|
||||||
|
)
|
||||||
|
docs_cmd.add_argument("--output", help="Evidence directory (default: data/docs-gate/<stamp>)")
|
||||||
dry_cmd = commands.add_parser(
|
dry_cmd = commands.add_parser(
|
||||||
"dry-run", help="Read-only reconciliation of the configured library (US07-07)"
|
"dry-run", help="Read-only reconciliation of the configured library (US07-07)"
|
||||||
)
|
)
|
||||||
@@ -188,6 +194,30 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
)
|
)
|
||||||
return 0 if report["ok"] else 1
|
return 0 if report["ok"] else 1
|
||||||
|
|
||||||
|
if args.command == "docs-gate":
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from photo_pipeline.services import release
|
||||||
|
|
||||||
|
# Documentation that disagrees with the application is worse than none: it is
|
||||||
|
# trusted. So this gate accepts no skip either — a check that did not run is a
|
||||||
|
# page nobody compared with the code.
|
||||||
|
output = args.output or Path(config.data_dir) / "docs-gate" / datetime.now(
|
||||||
|
timezone.utc
|
||||||
|
).strftime("%Y%m%dT%H%M%SZ")
|
||||||
|
report = release.run_gate(
|
||||||
|
config,
|
||||||
|
output=output,
|
||||||
|
stages=release.DOCS_STAGES,
|
||||||
|
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{k: v for k, v in report.items() if k not in ("stages", "matrix")}, indent=2
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0 if report["ok"] else 1
|
||||||
|
|
||||||
if args.command == "dry-run":
|
if args.command == "dry-run":
|
||||||
from photo_pipeline.services import release
|
from photo_pipeline.services import release
|
||||||
|
|
||||||
|
|||||||
@@ -71,6 +71,18 @@ CONTAINER_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
|||||||
# would skip here — no daemon, no compose, no browser — means it was not proven.
|
# would skip here — no daemon, no compose, no browser — means it was not proven.
|
||||||
CONTAINER_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
|
CONTAINER_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
# The documentation gate (US09-05): the manuals, held to the application. Selected by
|
||||||
|
# marker across the whole suite, because each check lives beside what it describes —
|
||||||
|
# the offline contracts under tests/integration, the browser and screenshot journeys
|
||||||
|
# under tests/e2e. Adding a phase_i test is enough to put it in front of a release.
|
||||||
|
DOCS_STAGES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||||
|
("documentation", ("tests", "-m", "phase_i")),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Also nothing. Every check here is either offline or needs the browser the rest of
|
||||||
|
# the suite already needs, so a skip means the manuals were not compared with the code.
|
||||||
|
DOCS_ALLOWED_SKIP_REASONS: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
class ReleaseError(RuntimeError):
|
class ReleaseError(RuntimeError):
|
||||||
pass
|
pass
|
||||||
|
|||||||
@@ -54,4 +54,5 @@ markers = [
|
|||||||
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
|
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
|
||||||
"container: builds and runs the container image and its composition (US08-02, US08-03) — needs a Docker daemon, the compose plugin, and the network",
|
"container: builds and runs the container image and its composition (US08-02, US08-03) — needs a Docker daemon, the compose plugin, and the network",
|
||||||
"phase_h: Phase H container deployment acceptance (US08-05) — the browser, upgrade, restart, and security journeys against the composed stack",
|
"phase_h: Phase H container deployment acceptance (US08-05) — the browser, upgrade, restart, and security journeys against the composed stack",
|
||||||
|
"phase_i: Phase I documentation acceptance (US09-05) — the manuals checked against the application, in the browser and offline",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -18,6 +18,9 @@ import pytest
|
|||||||
|
|
||||||
from tests.e2e._pipeline_harness import Server, seed_library
|
from tests.e2e._pipeline_harness import Server, seed_library
|
||||||
|
|
||||||
|
# Every check here belongs to the documentation gate (US09-05).
|
||||||
|
pytestmark = pytest.mark.phase_i
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
@pytest.fixture(scope="module")
|
||||||
def server(tmp_path_factory):
|
def server(tmp_path_factory):
|
||||||
|
|||||||
@@ -24,10 +24,13 @@ from contextlib import closing
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
from PIL import Image
|
||||||
|
from playwright.sync_api import TimeoutError as PlaywrightTimeout
|
||||||
|
|
||||||
from tests.conftest import session_client
|
from tests.conftest import session_client
|
||||||
from tests.e2e._pipeline_harness import (
|
from tests.e2e._pipeline_harness import (
|
||||||
Server,
|
Server,
|
||||||
|
approve_album,
|
||||||
seed_album,
|
seed_album,
|
||||||
start_worker,
|
start_worker,
|
||||||
wait_until,
|
wait_until,
|
||||||
@@ -39,21 +42,29 @@ IMAGES = DOCS / "images"
|
|||||||
ALBUM = "rome"
|
ALBUM = "rome"
|
||||||
VIEWPORT = {"width": 1280, "height": 900}
|
VIEWPORT = {"width": 1280, "height": 900}
|
||||||
|
|
||||||
# One per stage page of the manual: the route to visit, and the element whose
|
# One per stage page of the manual: the route, the heading that view renders, and the
|
||||||
# presence means the view has actually finished rendering.
|
# element whose presence means it has finished.
|
||||||
|
#
|
||||||
|
# The heading matters more than it looks. Every route replaces the same container, so
|
||||||
|
# waiting for "an h1" matches the *previous* view's heading and photographs the screen
|
||||||
|
# you just left — which is how the statistics page first shipped a picture of the
|
||||||
|
# archive view.
|
||||||
SHOTS = (
|
SHOTS = (
|
||||||
("workflow", "#/workflow", "stage-safety"),
|
("workflow", "#/workflow", "Workflow", "stage-safety"),
|
||||||
("inventory", "#/inventory", "asset-row"),
|
("inventory", "#/inventory", "Inventory", "asset-row"),
|
||||||
("duplicates", "#/duplicates", None),
|
("duplicates", "#/duplicates", "Duplicate clusters", None),
|
||||||
("safety", "#/safety", None),
|
("safety", "#/safety", "Safety review", None),
|
||||||
("analysis", "#/analyze", "analyze-counts"),
|
("analysis", "#/analyze", "Analyze", "analyze-counts"),
|
||||||
("albums", f"#/albums?album={ALBUM}", "suggested-name"),
|
("albums", f"#/albums?album={ALBUM}", "Albums", "suggested-name"),
|
||||||
("renames", "#/renames", None),
|
("renames", "#/renames", "Renames", None),
|
||||||
("uploads", "#/uploads", "upload-scope"),
|
("uploads", "#/uploads", "Upload", "upload-scope"),
|
||||||
("archive", "#/archive", "archive-locations"),
|
("archive", "#/archive", "Archive", "archive-locations"),
|
||||||
("statistics", "#/stats", None),
|
("statistics", "#/stats", "Stats", None),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Every check here belongs to the documentation gate (US09-05).
|
||||||
|
pytestmark = pytest.mark.phase_i
|
||||||
|
|
||||||
|
|
||||||
def writing() -> bool:
|
def writing() -> bool:
|
||||||
return os.environ.get("PHOTO_PIPELINE_WRITE_SCREENSHOTS") == "1"
|
return os.environ.get("PHOTO_PIPELINE_WRITE_SCREENSHOTS") == "1"
|
||||||
@@ -90,17 +101,23 @@ def _prepare(base: str, seeded) -> None:
|
|||||||
).raise_for_status()
|
).raise_for_status()
|
||||||
wait_until(lambda: client.get("/api/v1/workflow").status_code == 200)
|
wait_until(lambda: client.get("/api/v1/workflow").status_code == 200)
|
||||||
|
|
||||||
|
# An approved name and a built — deliberately unapplied — plan, so the renames
|
||||||
|
# screenshot shows the preview its page describes rather than an empty state.
|
||||||
|
# Building a plan moves nothing; applying it is what would, and nothing here does.
|
||||||
|
approve_album(base, album=ALBUM, name="2019 — Rome")
|
||||||
|
with closing(session_client(base, timeout=30)) as client:
|
||||||
|
client.post("/api/v1/rename-plans").raise_for_status()
|
||||||
|
|
||||||
|
|
||||||
def _capture(page, server, target: Path) -> list[str]:
|
def _capture(page, server, target: Path) -> list[str]:
|
||||||
target.mkdir(parents=True, exist_ok=True)
|
target.mkdir(parents=True, exist_ok=True)
|
||||||
page.set_viewport_size(VIEWPORT)
|
page.set_viewport_size(VIEWPORT)
|
||||||
written = []
|
written = []
|
||||||
for name, route, ready in SHOTS:
|
for name, route, heading, ready in SHOTS:
|
||||||
page.goto(f"{server.base}/app/{route}")
|
page.goto(f"{server.base}/app/{route}")
|
||||||
|
page.locator("main h1", has_text=heading).first.wait_for(timeout=30_000)
|
||||||
if ready:
|
if ready:
|
||||||
page.get_by_test_id(ready).first.wait_for(timeout=30_000)
|
page.get_by_test_id(ready).first.wait_for(timeout=30_000)
|
||||||
else:
|
|
||||||
page.locator("main h1").first.wait_for(timeout=30_000)
|
|
||||||
page.screenshot(path=str(target / f"{name}.png"))
|
page.screenshot(path=str(target / f"{name}.png"))
|
||||||
written.append(name)
|
written.append(name)
|
||||||
return written
|
return written
|
||||||
@@ -110,7 +127,7 @@ def test_the_generator_produces_every_screenshot_the_manual_references(page, sta
|
|||||||
destination = IMAGES if writing() else tmp_path / "images"
|
destination = IMAGES if writing() else tmp_path / "images"
|
||||||
written = _capture(page, stack, destination)
|
written = _capture(page, stack, destination)
|
||||||
|
|
||||||
assert sorted(written) == sorted(name for name, _, _ in SHOTS)
|
assert sorted(written) == sorted(name for name, *_ in SHOTS)
|
||||||
for name in written:
|
for name in written:
|
||||||
produced = destination / f"{name}.png"
|
produced = destination / f"{name}.png"
|
||||||
assert produced.stat().st_size > 5_000, f"{name}.png is too small to be a view"
|
assert produced.stat().st_size > 5_000, f"{name}.png is too small to be a view"
|
||||||
@@ -133,6 +150,60 @@ def test_no_screenshot_shows_a_real_path_or_a_secret(page, stack, tmp_path):
|
|||||||
assert all(ALBUM in path or "images" in path for path in paths)
|
assert all(ALBUM in path or "images" in path for path in paths)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_committed_screenshots_still_show_what_the_application_shows(page, stack, tmp_path):
|
||||||
|
"""A UI change that invalidates the manual should be a red build, not a discovery
|
||||||
|
months later by somebody following a picture of a screen that no longer exists.
|
||||||
|
|
||||||
|
The tolerance is on *content*, not pixels. Comparing the committed PNGs with a
|
||||||
|
fresh capture was tried first and rejected: PNG output is not reproducible, font
|
||||||
|
rasterisation differs between the machine that generated an image and the machine
|
||||||
|
running the gate, and several views legitimately print the fixture library's
|
||||||
|
absolute path, which is a fresh temporary directory every run. A pixel or
|
||||||
|
perceptual comparison therefore fails for reasons that have nothing to do with
|
||||||
|
the manual being wrong — and a check that cries wolf gets deleted.
|
||||||
|
|
||||||
|
What actually invalidates a screenshot is the view no longer showing what its
|
||||||
|
page says it shows. That is what is compared here: the heading, the elements the
|
||||||
|
page describes, and the fact that the image was captured at the same size.
|
||||||
|
|
||||||
|
ponytail: content comparison, not pixels. A visual-diff service with per-platform
|
||||||
|
baselines is the upgrade if cosmetic regressions ever need catching too.
|
||||||
|
"""
|
||||||
|
if not IMAGES.is_dir():
|
||||||
|
pytest.skip("no screenshots have been generated into the repository yet")
|
||||||
|
|
||||||
|
fresh = tmp_path / "fresh"
|
||||||
|
_capture(page, stack, fresh)
|
||||||
|
|
||||||
|
drifted = []
|
||||||
|
for name, route, heading, ready in SHOTS:
|
||||||
|
committed = IMAGES / f"{name}.png"
|
||||||
|
if not committed.is_file():
|
||||||
|
drifted.append(f"{name}: never committed")
|
||||||
|
continue
|
||||||
|
with Image.open(committed) as image:
|
||||||
|
if image.size != (VIEWPORT["width"], VIEWPORT["height"]):
|
||||||
|
drifted.append(f"{name}: committed at {image.size}, captured at {VIEWPORT}")
|
||||||
|
# The views render asynchronously, so each check waits rather than asking a
|
||||||
|
# question the page has not finished answering.
|
||||||
|
page.goto(f"{stack.base}/app/{route}")
|
||||||
|
try:
|
||||||
|
page.locator("main h1", has_text=heading).first.wait_for(timeout=15_000)
|
||||||
|
except PlaywrightTimeout:
|
||||||
|
drifted.append(f"{name}: the view no longer shows the heading '{heading}'")
|
||||||
|
continue
|
||||||
|
if ready:
|
||||||
|
try:
|
||||||
|
page.get_by_test_id(ready).first.wait_for(timeout=15_000)
|
||||||
|
except PlaywrightTimeout:
|
||||||
|
drifted.append(f"{name}: the view no longer renders '{ready}'")
|
||||||
|
|
||||||
|
assert drifted == [], (
|
||||||
|
f"the manual's screenshots no longer match the application: {drifted}. "
|
||||||
|
"Regenerate them with PHOTO_PIPELINE_WRITE_SCREENSHOTS=1 and review the result."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_every_committed_screenshot_is_referenced_by_a_page():
|
def test_every_committed_screenshot_is_referenced_by_a_page():
|
||||||
"""An image nobody shows is an image nobody updates."""
|
"""An image nobody shows is an image nobody updates."""
|
||||||
if not IMAGES.is_dir():
|
if not IMAGES.is_dir():
|
||||||
|
|||||||
@@ -14,10 +14,15 @@ from __future__ import annotations
|
|||||||
import re
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
import photo_pipeline.models # noqa: F401 (registers every table on Base.metadata)
|
import photo_pipeline.models # noqa: F401 (registers every table on Base.metadata)
|
||||||
from photo_pipeline.db import Base
|
from photo_pipeline.db import Base
|
||||||
from photo_pipeline.services import jobs, rename_journal, upload_batches
|
from photo_pipeline.services import jobs, rename_journal, upload_batches
|
||||||
|
|
||||||
|
# Every check here belongs to the documentation gate (US09-05).
|
||||||
|
pytestmark = pytest.mark.phase_i
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
PACKAGE = REPO / "photo_pipeline"
|
PACKAGE = REPO / "photo_pipeline"
|
||||||
OVERVIEW = REPO / "docs" / "architecture.md"
|
OVERVIEW = REPO / "docs" / "architecture.md"
|
||||||
|
|||||||
196
tests/integration/test_docs_gate.py
Normal file
@@ -0,0 +1,196 @@
|
|||||||
|
"""US09-05: the documentation gate's own contract, checked without a browser.
|
||||||
|
|
||||||
|
The gate itself needs a browser and a server and takes minutes; what is checkable
|
||||||
|
offline is what makes it a *gate* rather than a long test run:
|
||||||
|
|
||||||
|
* it selects every documentation check by marker, so adding one is enough to put it
|
||||||
|
in front of a release;
|
||||||
|
* it accepts no skip at all — a check that did not run is a page nobody compared
|
||||||
|
with the code;
|
||||||
|
* it retains checksummed evidence per run;
|
||||||
|
* CI runs it on pull requests, which is when a fix is still cheap.
|
||||||
|
|
||||||
|
The checks it runs are the four suites written by US09-01 through US09-04.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
from photo_pipeline.config import Config
|
||||||
|
from photo_pipeline.services import release
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
DOCS = REPO / "docs"
|
||||||
|
TEST_WORKFLOW = yaml.safe_load((REPO / ".gitea" / "workflows" / "test.yml").read_text())
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.phase_i
|
||||||
|
|
||||||
|
# The suites the gate exists to run. Each is mapped to a story in the traceability
|
||||||
|
# matrix, so this list is also what stops one being quietly dropped.
|
||||||
|
DOCUMENTATION_SUITES = (
|
||||||
|
"tests/integration/test_documentation.py",
|
||||||
|
"tests/integration/test_installation_manual.py",
|
||||||
|
"tests/integration/test_architecture_overview.py",
|
||||||
|
"tests/integration/test_user_manual.py",
|
||||||
|
"tests/integration/test_docs_gate.py",
|
||||||
|
"tests/e2e/test_docs_ui.py",
|
||||||
|
"tests/e2e/test_user_manual_screenshots.py",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _config(tmp_path) -> Config:
|
||||||
|
(tmp_path / "data").mkdir(exist_ok=True)
|
||||||
|
return Config.from_env({"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data")})
|
||||||
|
|
||||||
|
|
||||||
|
# ── what the gate runs ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gate_selects_every_documentation_check_by_marker():
|
||||||
|
assert release.DOCS_STAGES == (("documentation", ("tests", "-m", "phase_i")),)
|
||||||
|
declared = [
|
||||||
|
line
|
||||||
|
for line in (REPO / "pyproject.toml").read_text().splitlines()
|
||||||
|
if line.strip().startswith('"phase_i:')
|
||||||
|
]
|
||||||
|
assert declared, "an unregistered marker selects nothing and fails no gate"
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_documentation_suite_carries_the_marker():
|
||||||
|
"""A documentation test without the marker is a check the gate never runs."""
|
||||||
|
unmarked = [
|
||||||
|
suite
|
||||||
|
for suite in DOCUMENTATION_SUITES
|
||||||
|
if "pytest.mark.phase_i" not in (REPO / suite).read_text()
|
||||||
|
]
|
||||||
|
assert unmarked == [], f"suites missing the phase_i marker: {unmarked}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_marker_selects_nothing_outside_the_documentation_suites():
|
||||||
|
"""Scope matters: the gate's promise is that it ran *the documentation checks*."""
|
||||||
|
marked = sorted(
|
||||||
|
str(path.relative_to(REPO))
|
||||||
|
for path in (REPO / "tests").rglob("test_*.py")
|
||||||
|
if "pytest.mark.phase_i" in path.read_text()
|
||||||
|
)
|
||||||
|
assert marked == sorted(DOCUMENTATION_SUITES)
|
||||||
|
|
||||||
|
|
||||||
|
# ── no skip is an environment limit here ─────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gate_accepts_no_skipped_check_at_all():
|
||||||
|
assert release.DOCS_ALLOWED_SKIP_REASONS == ()
|
||||||
|
skipped = release.StageResult(
|
||||||
|
"documentation", [], 0, 0.1, "1 skipped", ["SKIPPED [1] x.py:1: exiftool not installed"]
|
||||||
|
)
|
||||||
|
# The release gate tolerates this one because it describes the machine. The
|
||||||
|
# documentation gate cannot: nothing here depends on the machine.
|
||||||
|
assert release.unexpected_skips([skipped]) == []
|
||||||
|
assert release.unexpected_skips([skipped], release.DOCS_ALLOWED_SKIP_REASONS) == [
|
||||||
|
"SKIPPED [1] x.py:1: exiftool not installed"
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_skipped_check_fails_the_run_and_the_evidence_says_so(tmp_path):
|
||||||
|
skipping = tmp_path / "test_skipping.py"
|
||||||
|
skipping.write_text(
|
||||||
|
"import pytest\n\ndef test_x():\n pytest.skip('exiftool not installed')\n"
|
||||||
|
)
|
||||||
|
evidence = tmp_path / "evidence"
|
||||||
|
|
||||||
|
report = release.run_gate(
|
||||||
|
_config(tmp_path),
|
||||||
|
output=evidence,
|
||||||
|
stages=(("documentation", (str(skipping),)),),
|
||||||
|
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert report["ok"] is False
|
||||||
|
assert report["failures"] == [], "the stage passed; the skip is what fails the gate"
|
||||||
|
assert report["unexpected_skips"], report
|
||||||
|
written = json.loads((evidence / release.REPORT_NAME).read_text())
|
||||||
|
assert written["ok"] is False
|
||||||
|
assert (evidence / "logs" / "documentation.log").exists()
|
||||||
|
for line in (evidence / release.CHECKSUMS_NAME).read_text().splitlines():
|
||||||
|
digest, name = line.split(" ", 1)
|
||||||
|
assert release.sha256_file(evidence / name) == digest
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_broken_documentation_check_fails_the_gate(tmp_path):
|
||||||
|
"""The failure mode that matters: a page and the code disagreeing."""
|
||||||
|
failing = tmp_path / "test_disagreement.py"
|
||||||
|
failing.write_text(
|
||||||
|
"def test_the_manual_matches_the_code():\n"
|
||||||
|
" documented = {'lock_held'}\n"
|
||||||
|
" emitted = {'lock_held', 'renamed_since'}\n"
|
||||||
|
" assert emitted <= documented\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
report = release.run_gate(
|
||||||
|
_config(tmp_path),
|
||||||
|
output=tmp_path / "evidence",
|
||||||
|
stages=(("documentation", (str(failing),)),),
|
||||||
|
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert report["ok"] is False
|
||||||
|
assert report["failures"], report
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_clean_tree_passes(tmp_path):
|
||||||
|
passing = tmp_path / "test_passing.py"
|
||||||
|
passing.write_text("def test_x():\n assert True\n")
|
||||||
|
|
||||||
|
report = release.run_gate(
|
||||||
|
_config(tmp_path),
|
||||||
|
output=tmp_path / "evidence",
|
||||||
|
stages=(("documentation", (str(passing),)),),
|
||||||
|
allowed_skips=release.DOCS_ALLOWED_SKIP_REASONS,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert report["ok"] is True and report["unexpected_skips"] == []
|
||||||
|
assert report["revision"], "the evidence must say which commit it covers"
|
||||||
|
|
||||||
|
|
||||||
|
# ── the command, and CI ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_command_is_documented_and_wired():
|
||||||
|
assert "docs-gate" in (REPO / "photo_pipeline" / "__main__.py").read_text()
|
||||||
|
assert "docs-gate" in (DOCS / "index.md").read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_ci_runs_the_gate_on_pull_requests_and_keeps_its_evidence():
|
||||||
|
job = TEST_WORKFLOW["jobs"]["documentation"]
|
||||||
|
# No `if:` — documentation drifts on the same commits that change behaviour, and
|
||||||
|
# the pull request is when saying so is still cheap.
|
||||||
|
assert "if" not in job
|
||||||
|
assert "pull_request" in TEST_WORKFLOW[True] or "pull_request" in TEST_WORKFLOW.get("on", {})
|
||||||
|
script = "\n".join(step["run"] for step in job["steps"] if "run" in step)
|
||||||
|
assert "photo_pipeline docs-gate" in script
|
||||||
|
evidence = next(step for step in job["steps"] if "upload-artifact" in str(step.get("uses")))
|
||||||
|
assert evidence["if"] == "always()", "a failed gate's evidence is the one worth keeping"
|
||||||
|
|
||||||
|
|
||||||
|
# ── the pages and the repository point at each other ─────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_readme_and_the_documentation_index_link_to_each_other():
|
||||||
|
"""Neither may become the forgotten copy."""
|
||||||
|
readme = (REPO / "README.md").read_text()
|
||||||
|
index = (DOCS / "index.md").read_text()
|
||||||
|
assert "docs/index.md" in readme
|
||||||
|
assert re.search(r"\.\./README\.md", index), "the index does not link back to the README"
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_documentation_story_is_mapped_in_the_traceability_matrix():
|
||||||
|
matrix = json.loads((REPO / "tests" / "story_traceability.json").read_text())["stories"]
|
||||||
|
for story in ("US09-01", "US09-02", "US09-03", "US09-04", "US09-05"):
|
||||||
|
assert matrix.get(story), f"{story} is not mapped to any test"
|
||||||
@@ -16,12 +16,17 @@ import json
|
|||||||
import re
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
from starlette.testclient import TestClient
|
from starlette.testclient import TestClient
|
||||||
|
|
||||||
from photo_pipeline.api.app import create_app
|
from photo_pipeline.api.app import create_app
|
||||||
from photo_pipeline.api.security import DEFAULT_HEADERS
|
from photo_pipeline.api.security import DEFAULT_HEADERS
|
||||||
from photo_pipeline.config import Config
|
from photo_pipeline.config import Config
|
||||||
|
|
||||||
|
# Every check here belongs to the documentation gate (US09-05).
|
||||||
|
pytestmark = pytest.mark.phase_i
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
DOCS = REPO / "docs"
|
DOCS = REPO / "docs"
|
||||||
VENDOR = REPO / "frontend" / "js" / "vendor"
|
VENDOR = REPO / "frontend" / "js" / "vendor"
|
||||||
|
|||||||
@@ -16,8 +16,13 @@ import sys
|
|||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
from photo_pipeline.config import ENV_PREFIX, LEGACY_ALIASES, Config
|
from photo_pipeline.config import ENV_PREFIX, LEGACY_ALIASES, Config
|
||||||
|
|
||||||
|
# Every check here belongs to the documentation gate (US09-05).
|
||||||
|
pytestmark = pytest.mark.phase_i
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
MANUAL = REPO / "docs" / "installation.md"
|
MANUAL = REPO / "docs" / "installation.md"
|
||||||
MAIN = REPO / "photo_pipeline" / "__main__.py"
|
MAIN = REPO / "photo_pipeline" / "__main__.py"
|
||||||
|
|||||||
@@ -14,6 +14,8 @@ from __future__ import annotations
|
|||||||
import re
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
DOCS = REPO / "docs"
|
DOCS = REPO / "docs"
|
||||||
STAGES = DOCS / "stages"
|
STAGES = DOCS / "stages"
|
||||||
@@ -55,6 +57,9 @@ STAGE_PAGES = (
|
|||||||
"diagnostics",
|
"diagnostics",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Every check here belongs to the documentation gate (US09-05).
|
||||||
|
pytestmark = pytest.mark.phase_i
|
||||||
|
|
||||||
|
|
||||||
def emitted_codes() -> set[str]:
|
def emitted_codes() -> set[str]:
|
||||||
found: set[str] = set()
|
found: set[str] = set()
|
||||||
|
|||||||
@@ -208,10 +208,11 @@
|
|||||||
"US09-04": [
|
"US09-04": [
|
||||||
"tests/integration/test_user_manual.py",
|
"tests/integration/test_user_manual.py",
|
||||||
"tests/e2e/test_user_manual_screenshots.py"
|
"tests/e2e/test_user_manual_screenshots.py"
|
||||||
|
],
|
||||||
|
"US09-05": [
|
||||||
|
"tests/integration/test_docs_gate.py"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"planned": [
|
"planned": [],
|
||||||
"US09-05"
|
|
||||||
],
|
|
||||||
"_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
|
"_planned_comment": "Accepted backlog stories that are not implemented yet. The release gate (US07-07) requires every story file to be either mapped to tests or listed here, so an unimplemented story is a visible decision rather than a hole in the matrix."
|
||||||
}
|
}
|
||||||
|
|||||||