Compare commits
1 Commits
us/US07-01
...
us/US06-04
| Author | SHA1 | Date | |
|---|---|---|---|
| 0982ebda70 |
@@ -80,8 +80,7 @@ are the authoritative backlog.
|
||||
uncertain. Work only on the claimed story and its generated feature branch.
|
||||
4. Read the entire issue, linked specification, dependencies, and acceptance criteria.
|
||||
Reconcile them with the concept before designing or changing code.
|
||||
5. Inspect the legacy CLI donors — frozen in `legacy_cli_archive/` since US07-01,
|
||||
with their ledger — before replacing applicable behavior. Update the donor
|
||||
5. Inspect the legacy CLI donors before replacing applicable behavior. Update the donor
|
||||
ledger and characterization tests required by the story.
|
||||
6. Implement every acceptance criterion and its automated tests.
|
||||
7. Run story-specific tests and the accumulated regression suite required by the epic.
|
||||
|
||||
72
README.md
72
README.md
@@ -169,75 +169,3 @@ work_item/scripts/python -m pytest -m phase_e -q
|
||||
bytes, and recovery after a restart).
|
||||
|
||||
Phases A–D remain green in the full run above.
|
||||
|
||||
### Phase F acceptance gate
|
||||
|
||||
Phase F (Epic E06: archive lifecycle) is the only stage that *removes* originals
|
||||
from the library, and the only one whose storage can walk away in someone's bag.
|
||||
One command runs the archive fault-injection suites, the black-box archive and
|
||||
restore API journeys, and the browser suite:
|
||||
|
||||
```bash
|
||||
work_item/scripts/python -m pytest -m phase_f -q
|
||||
```
|
||||
|
||||
- `tests/integration/test_archive_*.py` and `tests/integration/test_restore.py`
|
||||
drive real files on real filesystems: preflight against a mounted, missing,
|
||||
swapped, read-only, or full medium; copy-verify-remove and the same-filesystem
|
||||
move path; and a crash at **every** persisted journal transition in both transfer
|
||||
modes, asserting that no source is ever removed without a durable, byte-identical
|
||||
archive copy.
|
||||
- `tests/e2e/test_phase_f_pipeline.py` drives a real server and a real durable
|
||||
worker over HTTP: preflight blockers (offline medium, wrong volume, insufficient
|
||||
capacity, bytes changed after upload), a verified archive whose manifest, hashes,
|
||||
and path history are checked on the medium itself, a worker killed at each of
|
||||
`transferring`, `verified`, `removing`, `source_removed`, and `complete`, the
|
||||
evidence-based recovery that follows, offline deduplication of an exact and a
|
||||
fuzzy copy while the medium is away, mount return, restore, and a collision that
|
||||
restores beside its occupant.
|
||||
- **Archived is not missing.** An unmounted medium leaves its photos
|
||||
`archived_offline` — still hashed, still in the duplicate indexes, still
|
||||
previewable through their protected thumbnails — and a rescan neither prunes nor
|
||||
flags them.
|
||||
- **Ambiguity is never guessed.** A journal state the medium contradicts stays
|
||||
`manual`, offers no automatic action, and keeps blocking further archiving until
|
||||
a human decides.
|
||||
- `tests/e2e/test_archive_ui.py` covers the browser journeys (preview with
|
||||
destination identity and reclaimable bytes, blockers and mount instructions,
|
||||
progress split into transfer/verification/removal, interruption and recovery,
|
||||
offline browsing, restore, collision, keyboard confirmation, and reload).
|
||||
|
||||
Phases A–E remain green in the full run above.
|
||||
|
||||
## Legacy CLI archive
|
||||
|
||||
The command-line tools this application was extracted from are frozen in
|
||||
`legacy_cli_archive/` (US07-01): the original sources, their docs, the dependency
|
||||
lock they were last verified against, schema notes, a redacted sample
|
||||
configuration, the donor ledger, and a checksum for every file.
|
||||
|
||||
```bash
|
||||
cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256 # verify the archive
|
||||
work_item/scripts/python -m pytest tests/unit/test_legacy_archive.py -q # lint it
|
||||
```
|
||||
|
||||
They are reference material and rollback evidence only. No module under
|
||||
`photo_pipeline/` imports or executes them, the archive is not on the application's
|
||||
import path, and `tests/unit/test_legacy_archive.py` enforces that along with the
|
||||
checksums and the redaction. Only the two suites that compare *against* the donors —
|
||||
`tests/characterization/` and `tests/integration/test_safety_parity.py` — put the
|
||||
archived sources on `sys.path`.
|
||||
|
||||
The last path-keyed state they owned, `nsfw_scores.csv`, is imported once and then
|
||||
left alone:
|
||||
|
||||
```bash
|
||||
work_item/scripts/python -m photo_pipeline import-legacy-scores /path/to/nsfw_scores.csv --dry-run
|
||||
```
|
||||
|
||||
The import writes scored-but-unreviewed `safety_reviews` rows onto stable asset ids,
|
||||
never invents an asset for an unknown path, never overwrites a human decision, and
|
||||
writes a reconciliation report to the data directory saying exactly what it did.
|
||||
`legacy_cli_archive/donor_ledger.yaml` records every migrated behavior with its
|
||||
target, the tests that pin the donor, the tests that prove the replacement, and each
|
||||
intentional delta; rows still marked `pending` name the story that will resolve them.
|
||||
|
||||
@@ -12,15 +12,7 @@
|
||||
# (if anything) carries over
|
||||
# Every row needs either `tests` (existing test IDs, module::function) or
|
||||
# `pending_story` (the backlog story that will characterize/deliver it).
|
||||
# status: characterized — donor behavior pinned by characterization tests
|
||||
# resolved — replacement shipped; `parity` names the tests that prove
|
||||
# it, and `delta` states every intentional difference
|
||||
# pending — not migrated yet; `pending_story` says which story will
|
||||
# parity: test ids (path::function) in any suite, proving the replacement
|
||||
# delta: what the replacement deliberately does differently, or not at all
|
||||
#
|
||||
# Archived by US07-01: the sources referenced below now live beside this file in
|
||||
# src/ and are frozen (see README.md). Nothing in photo_pipeline imports them.
|
||||
# status: characterized | pending
|
||||
|
||||
rows:
|
||||
# ── photo_analyzer.py ──────────────────────────────────────────────────────
|
||||
@@ -88,12 +80,8 @@ rows:
|
||||
Copying a primary's analysis into variant rows survives, but keyed by
|
||||
asset_id and recorded as stage state instead of raw row copies.
|
||||
target: photo_pipeline/services/duplicates.py
|
||||
parity:
|
||||
- tests/integration/test_duplicate_engine.py::test_exact_copies_form_auto_decided_cluster
|
||||
- tests/integration/test_duplicate_engine.py::test_perceptual_variant_is_review_only
|
||||
delta: >
|
||||
The donor propagated variant links implicitly while writing rows; the replacement makes the canonical link a reviewable, reversible cluster decision, so a propagated link can always be undone.
|
||||
status: resolved
|
||||
pending_story: US01-04
|
||||
status: pending
|
||||
|
||||
- id: pa-hashing
|
||||
area: hashing
|
||||
@@ -153,12 +141,8 @@ rows:
|
||||
classification: replace
|
||||
rationale: Console report; superseded by the duplicate-review API/UI (US01-06).
|
||||
target: photo_pipeline/api/routes + frontend duplicate review
|
||||
parity:
|
||||
- tests/integration/test_review_api.py::test_clusters_list_and_detail
|
||||
- tests/e2e/test_review_ui.py::test_fuzzy_decision_requires_confirmation
|
||||
delta: >
|
||||
The text listing became the paged cluster API and the comparison UI; no textual report is produced.
|
||||
status: resolved
|
||||
pending_story: US01-06
|
||||
status: pending
|
||||
|
||||
- id: pa-reconcile
|
||||
area: database
|
||||
@@ -218,7 +202,7 @@ rows:
|
||||
RGB-normalize (drops alpha, converts HEIC), LANCZOS resize to 2048px
|
||||
long-edge, JPEG q85 base64 — the provider-input contract. Truncated-image
|
||||
tolerance (ImageFile.LOAD_TRUNCATED_IMAGES) carries with it.
|
||||
target: photo_pipeline/services/analysis.py
|
||||
target: photo_pipeline/integrations/vision.py
|
||||
tests:
|
||||
- test_pa_imaging::test_prepare_image_small_passthrough_jpeg
|
||||
- test_pa_imaging::test_prepare_image_resizes_to_max_long_edge
|
||||
@@ -246,12 +230,9 @@ rows:
|
||||
response validation, 429/503 retry with exponential backoff. Prompt and
|
||||
model/config version must be persisted per analysis_runs. Characterized
|
||||
against a deterministic fake provider when the analysis service is ported.
|
||||
target: photo_pipeline/services/analysis.py
|
||||
parity:
|
||||
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
||||
delta: >
|
||||
The prompt and response schema carry over; the provider is an injected adapter so the privacy gate is testable, and results are keyed to asset ids rather than paths.
|
||||
status: resolved
|
||||
target: photo_pipeline/integrations/vision.py
|
||||
pending_story: US02-06
|
||||
status: pending
|
||||
|
||||
- id: pa-throttle
|
||||
area: logging
|
||||
@@ -260,8 +241,8 @@ rows:
|
||||
rationale: >
|
||||
Rolling throttle window + persistent throttle_events.jsonl + RPD day
|
||||
counter become job metrics/events on the durable job model.
|
||||
target: photo_pipeline/services/jobs.py + photo_pipeline/services/analysis.py
|
||||
pending_story: US07-04
|
||||
target: photo_pipeline/jobs/coordinator.py
|
||||
pending_story: US02-02
|
||||
status: pending
|
||||
|
||||
- id: pa-nsfw-filter
|
||||
@@ -331,12 +312,8 @@ rows:
|
||||
checked between items, double-SIGINT force quit — becomes the durable
|
||||
JobRunner worker loop with the same drain-and-resume semantics.
|
||||
target: photo_pipeline/jobs/worker.py
|
||||
parity:
|
||||
- tests/integration/test_worker.py::test_worker_processes_all_items
|
||||
- tests/integration/test_worker.py::test_cooperative_cancellation_leaves_items_resumable
|
||||
delta: >
|
||||
The in-process folder loop with SIGINT handling became durable jobs claimed by a worker: cancellation is a persisted request, not a signal, and an interrupted run resumes from the database instead of restarting.
|
||||
status: resolved
|
||||
pending_story: US02-02
|
||||
status: pending
|
||||
|
||||
- id: pa-ui-terminal
|
||||
area: ui
|
||||
@@ -394,12 +371,8 @@ rows:
|
||||
the JSONL history logger become structured JSON logging with job_id/
|
||||
asset_id and job_events rows; per-photo history maps to job events.
|
||||
target: photo_pipeline structured logging + jobs/job_events
|
||||
parity:
|
||||
- tests/integration/test_jobs.py::test_enqueue_persists_items_and_event
|
||||
- tests/integration/test_jobs_sse.py::test_sse_streams_all_events_then_closes
|
||||
delta: >
|
||||
The JSONL history file and rich console handler are replaced by structured JSON logs plus durable job_events; the browser reads events over SSE rather than tailing a file.
|
||||
status: resolved
|
||||
pending_story: US02-02
|
||||
status: pending
|
||||
|
||||
- id: pa-balance
|
||||
area: vision
|
||||
@@ -409,11 +382,8 @@ rows:
|
||||
Provider balance/quota probes (report 'unsupported' on providers without
|
||||
the endpoint). Network-bound; characterized against the fake provider.
|
||||
target: photo_pipeline/services/analysis.py
|
||||
parity:
|
||||
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
||||
delta: >
|
||||
Not carried over: balance/quota polling was provider-specific (Gemini/OpenAI billing endpoints) and key-scoped. Cost reporting, when a story asks for it, comes from the per-run usage recorded with each analysis result rather than from a vendor endpoint.
|
||||
status: resolved
|
||||
pending_story: US02-06
|
||||
status: pending
|
||||
|
||||
- id: pa-cli
|
||||
area: configuration
|
||||
@@ -423,13 +393,9 @@ rows:
|
||||
argparse surface is superseded by the API; flags map to job configs
|
||||
(documented in WEBAPP_CONCEPT.md §8 parity table). Transitional CLI calls
|
||||
the shared services until archival (E07).
|
||||
target: photo_pipeline/__main__.py (serve | worker | migrate | import-legacy-scores) + /api/v1
|
||||
parity:
|
||||
- tests/integration/test_app_lifecycle.py::test_restart_preserves_data_and_reruns_migrations
|
||||
- tests/unit/test_legacy_archive.py::test_production_code_never_imports_an_archived_module
|
||||
delta: >
|
||||
The argparse surface is not reproduced. Every flag that drove work became an API command or a job configuration; the CLI keeps only what an application needs to be operated (serve, worker, migrate) plus the one-off legacy CSV import.
|
||||
status: resolved
|
||||
target: photo_pipeline/api + transitional CLI
|
||||
pending_story: US07-01
|
||||
status: pending
|
||||
|
||||
# ── nsfwtag/ ───────────────────────────────────────────────────────────────
|
||||
- id: nt-discovery
|
||||
@@ -456,17 +422,9 @@ rows:
|
||||
nsfw_scores.csv stops being the source of truth (concept: DB state).
|
||||
Format characterized (4-decimal scores, bad rows dropped) because the
|
||||
existing CSV must migrate into assets.safety_score.
|
||||
target: photo_pipeline/services/legacy_import.py (one-off import into safety_reviews)
|
||||
target: photo_pipeline/repositories (safety), CSV import in US01-02 migration
|
||||
tests: [test_nsfwtag::test_score_cache_roundtrip_and_tolerance]
|
||||
parity:
|
||||
- tests/integration/test_legacy_import.py::test_scores_are_imported_onto_asset_identity
|
||||
- tests/integration/test_legacy_import.py::test_a_reviewed_asset_is_never_overwritten_by_the_csv
|
||||
delta: >
|
||||
The CSV is no longer read at runtime at all: it is imported once into
|
||||
scored-but-unreviewed safety_reviews rows and left on disk untouched. A path
|
||||
that matches nothing is reported, never turned into an asset, and a human
|
||||
decision always outranks an imported score.
|
||||
status: resolved
|
||||
status: characterized
|
||||
|
||||
- id: nt-score-model
|
||||
area: nsfw
|
||||
@@ -527,11 +485,8 @@ rows:
|
||||
Newline-list bulk tagging (nsfw_confirmed.txt flow) is superseded by DB
|
||||
review decisions; the existing list is a one-time migration input.
|
||||
target: photo_pipeline/services/safety.py (decision import in US01-02)
|
||||
parity:
|
||||
- tests/integration/test_safety_parity.py::test_extracted_marks_and_partition_match_donor
|
||||
delta: >
|
||||
Bulk keyword application from a file list is replaced by decisions against asset ids; the EXIF write itself is the extracted, read-back-verified one.
|
||||
status: resolved
|
||||
pending_story: US01-02
|
||||
status: pending
|
||||
|
||||
- id: nt-ui
|
||||
area: ui
|
||||
@@ -543,11 +498,8 @@ rows:
|
||||
threshold/score review flow, lightbox and keyboard model are the frontend
|
||||
donor for the Safety view (preserved per concept §10; ported in US02-01).
|
||||
target: photo_pipeline/api + frontend Safety view
|
||||
parity:
|
||||
- tests/e2e/test_workflow_views.py::test_safety_review_decide_persists_across_reload
|
||||
delta: >
|
||||
The stdlib review server is replaced by the API plus the Safety view; the donor's layout, thresholds, and keyboard flow carry over, its embedded HTML generation does not.
|
||||
status: resolved
|
||||
pending_story: US02-01
|
||||
status: pending
|
||||
|
||||
- id: nt-bench
|
||||
area: nsfw
|
||||
@@ -557,11 +509,8 @@ rows:
|
||||
Dev-only model benchmark; archived without webapp replacement (recorded
|
||||
basis of the AdamCodd model choice). No production caller.
|
||||
target: none (archive as reference)
|
||||
parity:
|
||||
- tests/unit/test_legacy_archive.py::test_every_archived_source_matches_its_checksum
|
||||
delta: >
|
||||
No replacement: a dev-only benchmark whose result (the AdamCodd model choice) is already recorded. Kept in the archive as the basis of that choice.
|
||||
status: resolved
|
||||
pending_story: US07-01
|
||||
status: pending
|
||||
|
||||
# ── webapp/ ────────────────────────────────────────────────────────────────
|
||||
- id: wa-query-search
|
||||
@@ -630,13 +579,9 @@ rows:
|
||||
Subprocess-driving-the-CLI job control is superseded by durable DB jobs
|
||||
with a worker process. Two ideas carry over: progress derived from DB
|
||||
counts (not job-private state) and single-mutating-job enforcement.
|
||||
target: photo_pipeline/services/jobs.py + photo_pipeline/jobs/worker.py
|
||||
parity:
|
||||
- tests/integration/test_jobs.py::test_idempotency_key_returns_same_job
|
||||
- tests/integration/test_worker.py::test_handler_failure_fails_the_job
|
||||
delta: >
|
||||
Subprocess supervision of a CLI is replaced by durable jobs in the same process family: there is no subprocess to supervise, and progress is persisted rather than scraped from stdout.
|
||||
status: resolved
|
||||
target: photo_pipeline/jobs/coordinator.py
|
||||
pending_story: US02-02
|
||||
status: pending
|
||||
|
||||
- id: wa-server
|
||||
area: ui
|
||||
@@ -648,9 +593,5 @@ rows:
|
||||
browser. analyzer.html + page.py design (dark OLED tokens, Library/
|
||||
Analyze/Stats views) is frontend donor material per concept §10.
|
||||
target: photo_pipeline/api/app.py + frontend
|
||||
parity:
|
||||
- tests/e2e/test_frontend_shell.py::test_shell_loads_assets_without_console_or_network_errors
|
||||
- tests/e2e/test_frontend_shell.py::test_deep_link_and_reload_restore_view_and_filters
|
||||
delta: >
|
||||
The stdlib HTTP routes become the versioned FastAPI API and the static frontend shell; Python no longer interpolates HTML.
|
||||
status: resolved
|
||||
pending_story: US02-05
|
||||
status: pending
|
||||
@@ -19,7 +19,6 @@
|
||||
<a href="#/albums" data-nav="albums">Albums</a>
|
||||
<a href="#/renames" data-nav="renames">Renames</a>
|
||||
<a href="#/uploads" data-nav="uploads">Upload</a>
|
||||
<a href="#/archive" data-nav="archive">Archive</a>
|
||||
<a href="#/stats" data-nav="stats">Stats</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
@@ -138,31 +138,4 @@ export const api = {
|
||||
}),
|
||||
uploadVerifications: (id, opts = {}) =>
|
||||
request(`/upload-batches/${encodeURIComponent(id)}/verifications`, opts),
|
||||
|
||||
// ── Archive and restore: destinations, preflight, plans, recovery ────────
|
||||
archiveLocations: (opts = {}) => request("/archive-locations", opts),
|
||||
registerArchiveLocation: (payload, opts = {}) =>
|
||||
request("/archive-locations", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
archivePreflight: (payload, opts = {}) =>
|
||||
request("/archive-preflight", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
createArchivePlan: (payload, opts = {}) =>
|
||||
request("/archive-plans", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
listArchivePlans: (opts = {}) => request("/archive-plans", opts),
|
||||
getArchivePlan: (id, opts = {}) => request(`/archive-plans/${encodeURIComponent(id)}`, opts),
|
||||
applyArchivePlan: (id, opts = {}) =>
|
||||
request(`/archive-plans/${encodeURIComponent(id)}/apply`, { method: "POST", ...opts }),
|
||||
archiveRecovery: (opts = {}) => request("/archive-recovery", opts),
|
||||
resolveArchiveRecovery: (opts = {}) =>
|
||||
request("/archive-recovery/resolve", { method: "POST", ...opts }),
|
||||
restorePreflight: (payload, opts = {}) =>
|
||||
request("/restore-preflight", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
createRestorePlan: (payload, opts = {}) =>
|
||||
request("/restore-plans", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
listRestorePlans: (opts = {}) => request("/restore-plans", opts),
|
||||
getRestorePlan: (id, opts = {}) => request(`/restore-plans/${encodeURIComponent(id)}`, opts),
|
||||
applyRestorePlan: (id, opts = {}) =>
|
||||
request(`/restore-plans/${encodeURIComponent(id)}/apply`, { method: "POST", ...opts }),
|
||||
restoreRecovery: (opts = {}) => request("/restore-recovery", opts),
|
||||
resolveRestoreRecovery: (opts = {}) =>
|
||||
request("/restore-recovery/resolve", { method: "POST", ...opts }),
|
||||
};
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
import { api } from "./api.js";
|
||||
import { renderArchive, setArchiveRender } from "./archive.js";
|
||||
import { navigate, onRouteChange, parseHash } from "./router.js";
|
||||
import { renderRenames, setRenamesRender } from "./renames.js";
|
||||
import { renderUploads, setUploadsRender } from "./uploads.js";
|
||||
@@ -367,7 +366,6 @@ function render() {
|
||||
else if (path === "/albums") renderAlbums(root, params);
|
||||
else if (path === "/renames") renderRenames(root, params);
|
||||
else if (path === "/uploads") renderUploads(root, params);
|
||||
else if (path === "/archive") renderArchive(root, params);
|
||||
else if (path === "/stats") renderStats(root, params);
|
||||
else show(errorBanner("Unknown view"));
|
||||
}
|
||||
@@ -376,6 +374,5 @@ function render() {
|
||||
setRender(render);
|
||||
setRenamesRender(render);
|
||||
setUploadsRender(render);
|
||||
setArchiveRender(render);
|
||||
onRouteChange(render);
|
||||
render();
|
||||
|
||||
@@ -1,870 +0,0 @@
|
||||
// Archive view (US06-05): preview what would leave active storage, confirm it
|
||||
// exactly, watch the transfer, recover an interrupted one, browse what is already
|
||||
// archived, and bring it back.
|
||||
//
|
||||
// Archiving is the only stage that removes originals, so this view never decides
|
||||
// anything itself: the destination's identity, every blocker, the confirmation
|
||||
// token, and what recovery may do all come from the server, and an action the
|
||||
// server would refuse is not offered. Two things follow from that. A medium that
|
||||
// is not mounted produces an instruction naming it rather than a disabled mystery,
|
||||
// and an operation whose evidence is ambiguous offers no button at all.
|
||||
import { api } from "./api.js";
|
||||
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||
import { subscribeJob } from "./events.js";
|
||||
import { navigate } from "./router.js";
|
||||
|
||||
let outcome = null;
|
||||
let activity = [];
|
||||
|
||||
let render = () => {};
|
||||
export function setArchiveRender(fn) {
|
||||
render = fn;
|
||||
}
|
||||
|
||||
// The per-file journal states, split into the three things an operator actually
|
||||
// wants told apart: bytes moving, bytes proven, original removed (concept §9).
|
||||
const PHASES = [
|
||||
["planned", "planned", "waiting"],
|
||||
["transferring", "transfer", "copying to the medium"],
|
||||
["verified", "verified", "archive copy hashed and manifested"],
|
||||
["removing", "removing", "removing the active original"],
|
||||
["complete", "complete", "archived and removed"],
|
||||
["failed", "failed", "left alone for a decision"],
|
||||
];
|
||||
|
||||
const AVAILABILITY_LABEL = {
|
||||
active: "in the library",
|
||||
archived_online: "archived · medium mounted",
|
||||
archived_offline: "archived · medium away",
|
||||
missing_unexpected: "missing — unexplained",
|
||||
};
|
||||
|
||||
export async function renderArchive(root, params = {}) {
|
||||
setActiveNav("archive");
|
||||
|
||||
let locations;
|
||||
try {
|
||||
locations = (await api.archiveLocations()).locations;
|
||||
} catch (error) {
|
||||
root.replaceChildren(errorBanner(`Failed to load archive locations: ${error.message}`));
|
||||
return;
|
||||
}
|
||||
const location = locations.find((l) => l.id === params.location) || locations[0] || null;
|
||||
|
||||
const nodes = [el("h1", {}, "Archive"), locationsCard(locations, location, params)];
|
||||
if (!location) {
|
||||
nodes.push(
|
||||
el(
|
||||
"p",
|
||||
{ class: "muted", "data-testid": "no-locations" },
|
||||
"Register the disk, NAS share, or removable medium that will hold archived originals."
|
||||
),
|
||||
outcomeBanner()
|
||||
);
|
||||
root.replaceChildren(...nodes.filter(Boolean));
|
||||
return;
|
||||
}
|
||||
|
||||
const [preflight, archivePlans, restorePlans, recovery, restoreRecovery, archived, restore] =
|
||||
await Promise.all([
|
||||
load(() => api.archivePreflight({ location_id: location.id })),
|
||||
load(() => api.listArchivePlans()),
|
||||
load(() => api.listRestorePlans()),
|
||||
load(() => api.archiveRecovery()),
|
||||
load(() => api.restoreRecovery()),
|
||||
load(() => api.listAssets({ limit: 200 })),
|
||||
load(() => api.restorePreflight({ location_id: location.id })),
|
||||
]);
|
||||
|
||||
// Both directions share the runs table: one lane moves these originals, so one
|
||||
// history is what an operator has to reason about.
|
||||
const runs = [...plans(archivePlans), ...plans(restorePlans)].sort((a, b) =>
|
||||
(a.created_at || "").localeCompare(b.created_at || "")
|
||||
);
|
||||
const chosen = runs.find((run) => run.id === params.plan) || runs[runs.length - 1] || null;
|
||||
const selectedPlan = chosen ? await load(() => planDetailOf(chosen)) : null;
|
||||
|
||||
nodes.push(
|
||||
preflight ? previewSection(preflight, location) : null,
|
||||
preflight ? confirmBlock(preflight, location) : null,
|
||||
outcomeBanner(),
|
||||
activityLog(),
|
||||
recoverySection(mergeRecovery(recovery, restoreRecovery)),
|
||||
planList(runs, chosen && chosen.id),
|
||||
selectedPlan ? planDetail(selectedPlan) : null,
|
||||
archived ? archivedSection(archived.items, locations) : null,
|
||||
restore ? restoreSection(restore, location) : null
|
||||
);
|
||||
root.replaceChildren(...nodes.filter(Boolean));
|
||||
}
|
||||
|
||||
function plans(listed) {
|
||||
return listed ? listed.plans : [];
|
||||
}
|
||||
|
||||
function planDetailOf(run) {
|
||||
return run.direction === "restore" ? api.getRestorePlan(run.id) : api.getArchivePlan(run.id);
|
||||
}
|
||||
|
||||
// Archive and restore recovery answer the same question about the same lane, so
|
||||
// they are one list; an unresolved item of either kind blocks the other.
|
||||
function mergeRecovery(archive, restore) {
|
||||
if (!archive && !restore) return null;
|
||||
return {
|
||||
operations: [...((archive || {}).operations || []), ...((restore || {}).operations || [])],
|
||||
manual: [...((archive || {}).manual || []), ...((restore || {}).manual || [])],
|
||||
};
|
||||
}
|
||||
|
||||
// A section whose data failed to load must not take the rest of the view with it:
|
||||
// the medium being away is exactly when the archived-asset list matters most.
|
||||
async function load(call) {
|
||||
try {
|
||||
return await call();
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ── locations ────────────────────────────────────────────────────────────────
|
||||
// A location is a medium, not a path: the marker's ``media_id`` is what proves the
|
||||
// right disk is mounted, so it is shown next to the state it produced.
|
||||
function locationsCard(locations, selected, params) {
|
||||
const name = el("input", {
|
||||
type: "text",
|
||||
"data-testid": "location-name",
|
||||
"aria-label": "Archive location name",
|
||||
placeholder: "External disk",
|
||||
});
|
||||
const root = el("input", {
|
||||
type: "text",
|
||||
"data-testid": "location-root",
|
||||
"aria-label": "Archive location path",
|
||||
placeholder: "/Volumes/archive",
|
||||
});
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "archive-locations" },
|
||||
el("h2", {}, "Destinations"),
|
||||
locations.length
|
||||
? el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "locations" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["", "Name", "Root", "Medium", "State", "Last seen"].map((label) =>
|
||||
el("th", { scope: "col" }, label)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...locations.map((location) =>
|
||||
el(
|
||||
"tr",
|
||||
{
|
||||
"data-testid": "location-row",
|
||||
"data-name": location.name,
|
||||
"aria-current": selected && location.id === selected.id ? "true" : false,
|
||||
},
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("input", {
|
||||
type: "radio",
|
||||
name: "archive-location",
|
||||
"data-testid": "select-location",
|
||||
"aria-label": `Use ${location.name}`,
|
||||
checked: selected && location.id === selected.id ? "checked" : false,
|
||||
onchange: () => navigate("/archive", { ...params, location: location.id }),
|
||||
})
|
||||
),
|
||||
el("td", { "data-testid": "location-label" }, location.name),
|
||||
el("td", { class: "path", "data-testid": "location-path" }, location.root),
|
||||
el("td", { class: "path", "data-testid": "location-media" }, location.media_id),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${location.state === "online" ? "complete" : "attention"}`,
|
||||
"data-testid": "location-state",
|
||||
},
|
||||
location.state
|
||||
)
|
||||
),
|
||||
el("td", { class: "muted" }, location.last_seen_at || "never")
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
: null,
|
||||
selected && selected.state !== "online" ? mountInstruction(selected) : null,
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
name,
|
||||
root,
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
"data-testid": "register-location",
|
||||
onclick: () =>
|
||||
run(() => api.registerArchiveLocation({ name: name.value, root: root.value })),
|
||||
},
|
||||
"Register destination"
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// The one thing the app cannot do for the user: name the medium to connect.
|
||||
function mountInstruction(location) {
|
||||
const detail =
|
||||
location.state === "wrong_volume"
|
||||
? `A different medium is mounted at ${location.root}.`
|
||||
: `Nothing is mounted at ${location.root}.`;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "confirm", role: "status", "data-testid": "mount-instruction" },
|
||||
`${detail} Connect “${location.name}” (medium ${location.media_id}) and mount it there, ` +
|
||||
"then reload this view. Archived photos stay listed and searchable meanwhile."
|
||||
);
|
||||
}
|
||||
|
||||
// ── preview ──────────────────────────────────────────────────────────────────
|
||||
function previewSection(preflight, location) {
|
||||
const totals = preflight.totals;
|
||||
const capacity = preflight.capacity;
|
||||
const rows = preflight.albums.map((album) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "archive-album-row", "data-album": album.album },
|
||||
el("td", { "data-testid": "album-name" }, album.album),
|
||||
el("td", { class: "path", "data-testid": "album-folder" }, album.folder),
|
||||
el("td", { class: "path", "data-testid": "album-destination" }, album.destination),
|
||||
el(
|
||||
"td",
|
||||
{ "data-testid": "album-method" },
|
||||
album.transfer_method === "move" ? "move (same filesystem)" : "copy · verify · remove"
|
||||
),
|
||||
el("td", { "data-testid": "album-assets" }, String(album.asset_count)),
|
||||
el("td", { "data-testid": "album-reclaim" }, bytes(album.reclaimable_bytes)),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("span", { class: `badge ${album.state}`, "data-testid": "album-state" }, album.state),
|
||||
...album.blockers.map((blocker) =>
|
||||
el(
|
||||
"div",
|
||||
{ class: "blocker", "data-testid": "album-blocker", "data-code": blocker.code },
|
||||
blocker.message
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "archive-preview" },
|
||||
el("h2", {}, "Preview"),
|
||||
el(
|
||||
"div",
|
||||
{ class: "decision-bar" },
|
||||
el(
|
||||
"span",
|
||||
{ class: "badge", "data-testid": "destination-identity" },
|
||||
`${location.name} · ${location.media_id}`
|
||||
),
|
||||
el("span", { class: "badge", "data-testid": "total-albums" }, `${totals.albums} album(s)`),
|
||||
el("span", { class: "badge", "data-testid": "total-assets" }, `${totals.assets} photo(s)`),
|
||||
el(
|
||||
"span",
|
||||
{ class: "badge", "data-testid": "total-reclaim" },
|
||||
`${bytes(totals.bytes)} reclaimable`
|
||||
),
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${capacity.sufficient ? "complete" : "blocked"}`,
|
||||
"data-testid": "capacity",
|
||||
},
|
||||
`free ${bytes(capacity.free_bytes)} · reserve ${bytes(capacity.reserve_bytes)}`
|
||||
)
|
||||
),
|
||||
blockerList(preflight.blockers, "preflight-blockers", "preflight-blocker", "This scope cannot be archived yet"),
|
||||
rows.length
|
||||
? el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "archive-albums" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Album", "Folder", "Destination", "Transfer", "Photos", "Reclaims", "State"].map(
|
||||
(label) => el("th", { scope: "col" }, label)
|
||||
)
|
||||
)
|
||||
),
|
||||
el("tbody", {}, ...rows)
|
||||
)
|
||||
: el(
|
||||
"p",
|
||||
{ class: "muted", "data-testid": "no-albums" },
|
||||
"No album has a verified upload whose bytes are still unchanged, so nothing may be archived."
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function blockerList(blockers, containerId, itemId, title) {
|
||||
if (!blockers || !blockers.length) return null;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": containerId },
|
||||
el("strong", {}, title),
|
||||
el(
|
||||
"ul",
|
||||
{},
|
||||
...blockers.map((blocker) =>
|
||||
el("li", { "data-testid": itemId, "data-code": blocker.code }, `${blocker.code}: ${blocker.message}`)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── confirmation ─────────────────────────────────────────────────────────────
|
||||
function confirmBlock(preflight, location) {
|
||||
const ready = preflight.state === "ready";
|
||||
const totals = preflight.totals;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "confirm" },
|
||||
el("h2", {}, "Confirm"),
|
||||
el(
|
||||
"p",
|
||||
{ class: "muted", "data-testid": "confirm-token" },
|
||||
`Preflight ${preflight.token.slice(0, 20)}… · ${location.name}`
|
||||
),
|
||||
el(
|
||||
"p",
|
||||
{ "data-testid": "archive-note" },
|
||||
"Archiving removes each original from the library — but only after its copy on " +
|
||||
"the medium has been written, hashed, and recorded in the manifest. The photos " +
|
||||
"stay searchable and deduplicable while the medium is away, and can be restored " +
|
||||
"from this view."
|
||||
),
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "start-archive",
|
||||
disabled: ready ? false : "disabled",
|
||||
title: ready ? false : "resolve the blockers above first",
|
||||
onclick: () =>
|
||||
run(async () => {
|
||||
const plan = await api.createArchivePlan({
|
||||
location_id: location.id,
|
||||
token: preflight.token,
|
||||
});
|
||||
const started = await api.applyArchivePlan(plan.id);
|
||||
watch(started.job.id, plan.id, "archive");
|
||||
return { archiving: plan.asset_count };
|
||||
}),
|
||||
},
|
||||
`Archive ${totals.ready_albums} album(s) · reclaim ${bytes(totals.bytes)}`
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── plans and progress ───────────────────────────────────────────────────────
|
||||
function planList(runs, selectedId) {
|
||||
if (!runs.length) {
|
||||
return el("p", { class: "muted", "data-testid": "no-plans" }, "Nothing has been archived yet.");
|
||||
}
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "archive-plans" },
|
||||
el("h2", {}, "Runs"),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "plans" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Created", "Direction", "State", "Photos", "Bytes"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...runs.map((plan) =>
|
||||
el(
|
||||
"tr",
|
||||
{
|
||||
"data-testid": "plan-row",
|
||||
"data-plan": plan.id,
|
||||
"data-direction": plan.direction,
|
||||
"aria-current": plan.id === selectedId ? "true" : false,
|
||||
},
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("a", { class: "link", href: `#/archive?plan=${encodeURIComponent(plan.id)}` }, plan.created_at || plan.id)
|
||||
),
|
||||
el("td", { "data-testid": "plan-direction" }, plan.direction),
|
||||
el("td", {}, el("span", { class: `badge ${plan.state}`, "data-testid": "plan-state" }, plan.state)),
|
||||
el("td", {}, String(plan.asset_count)),
|
||||
el("td", {}, bytes(plan.byte_size))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function planDetail(plan) {
|
||||
const operations = plan.operations || [];
|
||||
const counts = {};
|
||||
for (const operation of operations) {
|
||||
counts[operation.journal_state] = (counts[operation.journal_state] || 0) + 1;
|
||||
}
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "plan-detail", "data-plan": plan.id },
|
||||
el("h2", {}, `${plan.direction === "restore" ? "Restore" : "Archive"} run ${plan.created_at || plan.id}`),
|
||||
// Transfer, verification, and removal are separate answers to separate
|
||||
// questions: what has moved, what is proven, and what is already gone.
|
||||
el(
|
||||
"div",
|
||||
{ class: "decision-bar", "data-testid": "plan-progress" },
|
||||
el("span", { class: `badge ${plan.state}`, "data-testid": "detail-state" }, plan.state),
|
||||
...PHASES.map(([key, label, title]) =>
|
||||
el(
|
||||
"span",
|
||||
{ class: `badge ${key}`, "data-testid": `count-${label}`, title },
|
||||
`${label}: ${counts[key] ?? 0}`
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "operations" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Photo", "Destination", "Phase", "Attempts", "Problem"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...operations.map((operation) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "operation-row", "data-asset-id": operation.asset_id },
|
||||
el("td", { class: "path", "data-testid": "operation-source" }, operation.source_path),
|
||||
el("td", { class: "path", "data-testid": "operation-destination" }, operation.destination_path),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{ class: `badge ${operation.journal_state}`, "data-testid": "operation-phase" },
|
||||
phaseLabel(operation.journal_state)
|
||||
)
|
||||
),
|
||||
el("td", {}, String(operation.attempt_count)),
|
||||
el(
|
||||
"td",
|
||||
{ class: "muted", "data-testid": "operation-error", "data-code": operation.error_code || "" },
|
||||
operation.error_code ? `${operation.error_code}: ${operation.error_message || ""}` : "—"
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function phaseLabel(state) {
|
||||
const found = PHASES.find(([key]) => key === state);
|
||||
return found ? found[1] : state;
|
||||
}
|
||||
|
||||
// ── recovery ─────────────────────────────────────────────────────────────────
|
||||
// What an interrupted run left behind, straight from the journal plus the files on
|
||||
// disk. Only the operations the server itself classified as resolvable get an
|
||||
// action; ambiguous ones are shown with their evidence and no button.
|
||||
function recoverySection(recovery) {
|
||||
const operations = recovery ? recovery.operations : [];
|
||||
if (!operations.length) return null;
|
||||
const manual = recovery.manual || [];
|
||||
const resolvable = operations.length - manual.length;
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "archive-recovery" },
|
||||
el("h2", {}, "Interrupted work"),
|
||||
el(
|
||||
"p",
|
||||
{ "data-testid": "recovery-summary" },
|
||||
`${operations.length} operation(s) did not finish · ${resolvable} resolvable · ` +
|
||||
`${manual.length} need a decision`
|
||||
),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "recovery-operations" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Photo", "Phase", "Verdict", "Evidence"].map((l) => el("th", { scope: "col" }, l))
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...operations.map((verdict) =>
|
||||
el(
|
||||
"tr",
|
||||
{
|
||||
"data-testid": "recovery-row",
|
||||
"data-classification": verdict.classification,
|
||||
"data-direction": verdict.direction,
|
||||
},
|
||||
el("td", { class: "path", "data-testid": "recovery-source" }, verdict.source_path),
|
||||
el("td", {}, el("span", { class: "badge" }, phaseLabel(verdict.journal_state))),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${verdict.classification === "manual" ? "blocked" : "attention"}`,
|
||||
"data-testid": "recovery-verdict",
|
||||
},
|
||||
verdict.classification
|
||||
)
|
||||
),
|
||||
el(
|
||||
"td",
|
||||
{ class: "muted", "data-testid": "recovery-reason" },
|
||||
`${verdict.reason} (source ${verdict.source_exists ? "present" : "absent"}, ` +
|
||||
`archive copy ${verdict.destination_matches ? "verified" : verdict.destination_exists ? "different bytes" : "absent"})`
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
manual.length
|
||||
? el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "recovery-manual" },
|
||||
`${manual.length} operation(s) cannot be resolved from the evidence. Nothing will be ` +
|
||||
"removed or retried for them here: inspect the medium and the library, then decide."
|
||||
)
|
||||
: null,
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
resolvable
|
||||
? el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "resolve-recovery",
|
||||
onclick: () => run(() => api.resolveArchiveRecovery()),
|
||||
},
|
||||
`Finish ${resolvable} recoverable operation(s)`
|
||||
)
|
||||
: el(
|
||||
"span",
|
||||
{ class: "muted", "data-testid": "no-safe-recovery" },
|
||||
"No operation can be finished safely from here."
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── archived assets ──────────────────────────────────────────────────────────
|
||||
// Browsing what is already archived, including while the medium is away: the
|
||||
// retained protected preview and the recorded hashes are the evidence, so the row
|
||||
// stays complete and honest instead of turning into a missing file.
|
||||
function archivedSection(assets, locations) {
|
||||
const archived = assets.filter((asset) => asset.availability_state !== "active");
|
||||
if (!archived.length) return null;
|
||||
const names = Object.fromEntries(locations.map((location) => [location.id, location.name]));
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "archived-assets" },
|
||||
el("h2", {}, "Archived photos"),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "archived" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Preview", "Archived as", "Medium", "Availability", "Size"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...archived.map((asset) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "archived-row", "data-asset-id": asset.id },
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("img", {
|
||||
"data-testid": "archived-preview",
|
||||
width: 96,
|
||||
loading: "lazy",
|
||||
src: api.thumbnailUrl(asset.id, 256),
|
||||
alt: `Preview of ${asset.archive_path || asset.id}`,
|
||||
})
|
||||
),
|
||||
el("td", { class: "path", "data-testid": "archived-path" }, asset.archive_path || "—"),
|
||||
el(
|
||||
"td",
|
||||
{ "data-testid": "archived-medium" },
|
||||
names[asset.archive_location_id] || asset.archive_location_id || "—"
|
||||
),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${asset.availability_state === "archived_online" ? "complete" : "attention"}`,
|
||||
"data-testid": "archived-availability",
|
||||
},
|
||||
AVAILABILITY_LABEL[asset.availability_state] || asset.availability_state
|
||||
)
|
||||
),
|
||||
el("td", {}, bytes(asset.byte_size))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── restore ──────────────────────────────────────────────────────────────────
|
||||
function restoreSection(restore, location) {
|
||||
const ready = restore.state === "ready";
|
||||
const items = restore.items || [];
|
||||
if (!items.length && !restore.blockers.length) return null;
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "restore" },
|
||||
el("h2", {}, "Restore"),
|
||||
el(
|
||||
"p",
|
||||
{ "data-testid": "restore-note" },
|
||||
"Restoring copies the archived bytes back into the library and leaves the archive " +
|
||||
"copy where it is. A name that is already taken is never overwritten: the photo " +
|
||||
"comes back beside it under a visibly different name."
|
||||
),
|
||||
blockerList(restore.blockers, "restore-blockers", "restore-blocker", "This restore cannot run yet"),
|
||||
items.length
|
||||
? el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "restore-items" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Archived as", "Comes back as", "Size", "State"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...items.map((item) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "restore-row", "data-asset-id": item.asset_id },
|
||||
el("td", { class: "path", "data-testid": "restore-source" }, item.archive_path),
|
||||
el("td", { class: "path", "data-testid": "restore-destination" }, item.destination_path || "—"),
|
||||
el("td", {}, bytes(item.byte_size)),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
item.blockers.length
|
||||
? el(
|
||||
"span",
|
||||
{
|
||||
class: "badge blocked",
|
||||
"data-testid": "restore-item-blocker",
|
||||
"data-code": item.blockers[0].code,
|
||||
},
|
||||
item.blockers[0].code
|
||||
)
|
||||
: el("span", { class: "badge ready", "data-testid": "restore-item-state" }, "ready")
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
: null,
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "start-restore",
|
||||
disabled: ready ? false : "disabled",
|
||||
title: ready ? false : "the medium and every archived copy must check out first",
|
||||
onclick: () =>
|
||||
run(async () => {
|
||||
const plan = await api.createRestorePlan({
|
||||
location_id: location.id,
|
||||
token: restore.token,
|
||||
});
|
||||
const started = await api.applyRestorePlan(plan.id);
|
||||
watch(started.job.id, plan.id, "restore");
|
||||
return { restoring: plan.asset_count };
|
||||
}),
|
||||
},
|
||||
`Restore ${items.length} photo(s) from ${location.name}`
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── running commands ─────────────────────────────────────────────────────────
|
||||
async function run(action) {
|
||||
try {
|
||||
outcome = { kind: "ok", result: await action() };
|
||||
} catch (error) {
|
||||
outcome = error.status === 409 ? { kind: "conflict", error } : { kind: "error", error };
|
||||
}
|
||||
render();
|
||||
}
|
||||
|
||||
// Live job activity. The plan panel is refreshed on its own tick because the
|
||||
// journal advances per file, not per job event; a full re-render would re-run
|
||||
// preflight (which re-hashes the library), so that happens once when the job ends.
|
||||
const REFRESH_MS = 500;
|
||||
|
||||
function watch(jobId, planId, kind) {
|
||||
activity = [`Started ${kind} job ${jobId}`];
|
||||
const tick = setInterval(() => refreshPlan(planId, kind), REFRESH_MS);
|
||||
subscribeJob(jobId, {
|
||||
onEvent: (event) => {
|
||||
activity.push(`${event.type}${event.message ? ": " + event.message : ""}`);
|
||||
const log = document.querySelector('[data-testid="archive-activity"]');
|
||||
if (log) log.textContent = activity.join("\n");
|
||||
},
|
||||
onDone: () => {
|
||||
clearInterval(tick);
|
||||
activity.push("done");
|
||||
render();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function refreshPlan(planId, kind) {
|
||||
const node = document.querySelector(`[data-testid="plan-detail"][data-plan="${planId}"]`);
|
||||
if (!node) return; // the user navigated away from the running plan
|
||||
try {
|
||||
const plan = await (kind === "restore" ? api.getRestorePlan(planId) : api.getArchivePlan(planId));
|
||||
node.replaceWith(planDetail(plan));
|
||||
} catch (_) {
|
||||
// Transient; the next tick tries again and the job's end re-renders anyway.
|
||||
}
|
||||
}
|
||||
|
||||
function activityLog() {
|
||||
return el(
|
||||
"pre",
|
||||
{
|
||||
class: "activity-log",
|
||||
role: "status",
|
||||
"aria-live": "polite",
|
||||
"data-testid": "archive-activity",
|
||||
},
|
||||
activity.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
function outcomeBanner() {
|
||||
if (!outcome) return null;
|
||||
if (outcome.kind === "conflict") {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "conflict" },
|
||||
`The server refused this: ${outcome.error.message}. Nothing was moved or removed; ` +
|
||||
"the state below is the server's current one — review it and decide again."
|
||||
);
|
||||
}
|
||||
if (outcome.kind === "error") {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "archive-error" },
|
||||
`Failed: ${outcome.error.message}`
|
||||
);
|
||||
}
|
||||
const result = outcome.result || {};
|
||||
const message =
|
||||
result.archiving !== undefined
|
||||
? `Archiving ${result.archiving} photo(s). Originals are removed only after their copies verify.`
|
||||
: result.restoring !== undefined
|
||||
? `Restoring ${result.restoring} photo(s) into the library.`
|
||||
: "Done — the state below is the server's.";
|
||||
return el("div", { class: "alert", role: "status", "data-testid": "archive-result" }, message);
|
||||
}
|
||||
|
||||
// ── formatting ───────────────────────────────────────────────────────────────
|
||||
function bytes(value) {
|
||||
if (value == null) return "unknown";
|
||||
const units = ["B", "kB", "MB", "GB", "TB"];
|
||||
let size = value;
|
||||
let unit = 0;
|
||||
while (size >= 1000 && unit < units.length - 1) {
|
||||
size /= 1000;
|
||||
unit += 1;
|
||||
}
|
||||
return `${unit === 0 ? size : size.toFixed(1)} ${units[unit]}`;
|
||||
}
|
||||
@@ -1,22 +0,0 @@
|
||||
d098bbde13d2ebc872ca781e244cc1e48b97cc551fedaca3251d6a5278a15234 src/compare_models.py
|
||||
7d0f68cb95fbd6603e4c558620098b46929ebdb6fd91a598bbf84f26eea35e6c src/nsfw_tag.py
|
||||
35329c53570e215cef15f59429c1a5251a0448b90522b0910ab9d68efe1bc307 src/nsfwtag/__init__.py
|
||||
7f52b55e4f3b77eda3657d7cd2272c7cca8422100e241671de495603c1126ee2 src/nsfwtag/__main__.py
|
||||
6f5a97114e0d87d272ce22d065dc31fa0dd72ff8590d11ad14cb1c1f486d339c src/nsfwtag/bench.py
|
||||
29498fac1d73ba2b7420ffe1ffe49a87f684ec299e2c3b9c6e14e63e95643ba4 src/nsfwtag/exif.py
|
||||
934e82c402813ebf503e5a20eb03d5103f84df13117bc4dd1fd95deaa01ab263 src/nsfwtag/README.md
|
||||
68125e6184c7e4d2a5b0153f2155675753ab0b1329d1dbc4933d4769492ec4be src/nsfwtag/review.html
|
||||
a597eab74803dd31452b5d70abf7d6d6320eb17a61596c891b009da15d624e98 src/nsfwtag/scoring.py
|
||||
674969a18e58ee511a2abf574a875db92f2f524769b1b614d0d3662ab879a709 src/nsfwtag/server.py
|
||||
24a7c8d029da6e97d46b110a9fe9dbb3900127f8ae3c142f634fdd91d6c45243 src/nsfwtag/webapp.py
|
||||
2c2ea558f1b9095c1150f078ea29c8e0f180cc045a9d1097b83f61bfb145295b src/photo_analyzer.py
|
||||
5fb7ce2f977a17da1f501c45d7985328318ae14298a3f347eab46a6e3f02aab3 src/test_dedup.py
|
||||
1673dc76cc60aba56a5b065d9bc9f5dda00342cf2a79d000a3f74686fb7cffec src/test_nsfw_skip.py
|
||||
55aa82f348e0e90be0163ba6aa278e5fc8e445996e5335774917e5ef59ad3563 src/webapp/__init__.py
|
||||
ba4342bd0175591a2121f063f14a78a602d5679bcfcf24aa14c8d4b8cd5da1a2 src/webapp/__main__.py
|
||||
a60f24035a989909467778f888d62854c1afc2702f47b8d3cca32bf3cf04d5cf src/webapp/analyzer.html
|
||||
018bcc2f770d716444b456db58d6b4a41800138a98598b3c27ba503ab22d939e src/webapp/page.py
|
||||
5f599b107b2b117ca118b6fbdec5e5786a9ab4eea424aa9af1bfea7bf87cebab src/webapp/query.py
|
||||
875cce697caa02717c85a2707ba935c46dd9ace18428fd4abb13105b7a8e24d9 src/webapp/README.md
|
||||
af7d0d72d245b4bbb1e0e30f9708239697543212d4171defd1bfe26dd24a4f05 src/webapp/runner.py
|
||||
cddf0555b06fe7e4daa14309baaaf91130f8506922cb0ffb6507d76cbe39add4 src/webapp/server.py
|
||||
@@ -1,80 +0,0 @@
|
||||
# Legacy CLI archive (US07-01)
|
||||
|
||||
Frozen, read-only sources of the command-line tools this application was extracted
|
||||
from. They are **reference material and rollback evidence** — provenance for
|
||||
behavior that now lives in `photo_pipeline/`, and the only way to answer "what did
|
||||
the original actually do?" once the replacement has drifted.
|
||||
|
||||
> **Nothing here is production code.** No module under `photo_pipeline/` imports or
|
||||
> executes anything in this directory, and this directory is not on the application's
|
||||
> import path. `tests/unit/test_legacy_archive.py` enforces both, along with the
|
||||
> checksums and the redaction below.
|
||||
|
||||
## What is here
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `src/photo_analyzer.py` | the analysis CLI: discovery, hashing/dedup, vision analysis, EXIF writing, SQLite schema, album naming |
|
||||
| `src/nsfwtag/` | NSFW scoring, EXIF safety keywords, and the review server (`__init__` 1.1.0) |
|
||||
| `src/webapp/` | the stdlib review web app: FTS search, stats, subprocess runner, HTML shell (`__init__` 0.1.0) |
|
||||
| `src/nsfw_tag.py` | thin backwards-compatible entry point for `nsfwtag` |
|
||||
| `src/compare_models.py` | dev-only model comparison script |
|
||||
| `src/test_dedup.py`, `src/test_nsfw_skip.py` | the CLIs' own standalone self-checks (never pytest suites) |
|
||||
| `donor_ledger.yaml` | the donor ledger: every migrated behavior, its target, its tests, and every intentional delta |
|
||||
| `requirements-lock.txt` | the dependency versions the frozen sources were last verified against |
|
||||
| `photo_analyzer.env.sample` | the CLI's configuration surface, with every value replaced by a placeholder |
|
||||
| `CHECKSUMS.sha256` | SHA-256 of every archived source file |
|
||||
|
||||
`photo_analyzer.py` carries no `__version__`; its identity is its checksum, recorded
|
||||
in `CHECKSUMS.sha256` and taken at commit `9b7ee6b` (the merge of US06-06, the last
|
||||
commit before archival).
|
||||
|
||||
## Verifying the archive
|
||||
|
||||
```bash
|
||||
cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256
|
||||
```
|
||||
|
||||
Any edit to an archived source must be accompanied by a regenerated checksum file
|
||||
and a note here explaining why a *frozen* archive changed — the normal answer being
|
||||
that it should not.
|
||||
|
||||
## Schema notes
|
||||
|
||||
`photo_analyzer.py` owned a path-keyed SQLite database (`SCHEMA`, near the top of the
|
||||
file):
|
||||
|
||||
- `photos(id, path UNIQUE, status, phash, file_sha1, dup_of, description, tags,
|
||||
people_count, setting, time_of_day, season, mood, location_hint, approx_year,
|
||||
raw_response, error_message, analyzed_at, exif_written_at)`;
|
||||
- `photos_fts` — an FTS5 external-content index over `path, description, tags, mood,
|
||||
location_hint`, kept in sync by insert/update/delete triggers;
|
||||
- late columns (`phash`, `file_sha1`, `dup_of`) were added by an in-code
|
||||
`_migrate_schema()` rather than a migration tool, and their indexes are created
|
||||
only after the `ALTER`.
|
||||
|
||||
The replacement keeps the same analysis fields but re-keys everything to a stable
|
||||
`assets.id` (Alembic migrations `0001`…), because a path is not an identity: the
|
||||
donor's `path UNIQUE` is exactly what broke on every move and rename.
|
||||
|
||||
`nsfwtag` kept its safety scores outside the database in `nsfw_scores.csv`
|
||||
(`path,nsfw_score`, four decimals, unreadable rows dropped). That file is no longer a
|
||||
source of truth; `photo_pipeline/services/legacy_import.py` imports it into
|
||||
`assets.safety_score` and reports exactly what matched, what did not, and why.
|
||||
|
||||
## Redaction
|
||||
|
||||
The archive contains no credentials. `photo_analyzer.env.sample` documents the
|
||||
configuration surface (`LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL`, and the tuning
|
||||
variables) with placeholder values only; the CLI itself never contained a key, it
|
||||
read one from `photo_analyzer.env` or the environment. No `.env`, database, log, CSV,
|
||||
or photo from the author's library is archived.
|
||||
|
||||
## Why these tools were retired
|
||||
|
||||
Each behavior's fate is recorded per row in `donor_ledger.yaml`: `reuse`, `extract`,
|
||||
`refactor`, or `replace`, with the target module, the characterization tests that
|
||||
pinned the donor's behavior, the parity tests the replacement passes, and — where the
|
||||
replacement deliberately does something else or nothing at all — a `delta` saying so.
|
||||
Rows still marked `pending` name the backlog story that will resolve them; they are
|
||||
the honest list of what has *not* been carried over yet.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Sample configuration for the archived photo_analyzer.py CLI (US07-01).
|
||||
#
|
||||
# REDACTED: no value below is real. Each line names a variable the CLI read and
|
||||
# what belongs there; the placeholders are deliberately not key-shaped, so this
|
||||
# file can never be mistaken for — or scanned as — a credential.
|
||||
|
||||
LLM_API_KEY=<paste your provider key here>
|
||||
LLM_BASE_URL=<provider base url, e.g. the OpenAI-compatible Gemini endpoint>
|
||||
LLM_MODEL=<model id, e.g. a Gemini Flash release>
|
||||
|
||||
# Optional tuning the CLI read from the same file:
|
||||
PHASH_THRESHOLD=8
|
||||
MAX_WORKERS=4
|
||||
RETRY_ATTEMPTS=3
|
||||
RPD_LIMIT=0
|
||||
|
||||
# The CLI ignored its own shipped placeholder (a literal "sk-REPLACE..." string)
|
||||
# until it was replaced, and a shell variable always won over this file.
|
||||
@@ -1,20 +0,0 @@
|
||||
# Final dependency lock of the archived CLIs (US07-01).
|
||||
#
|
||||
# These are the versions present in the environment the archive was taken from —
|
||||
# what the frozen sources were last verified against by the characterization
|
||||
# suite. Restoring a donor for forensics means pinning these, not "latest".
|
||||
#
|
||||
# Python 3.14.6
|
||||
|
||||
openai==3.0.0 # photo_analyzer: OpenAI-compatible vision client
|
||||
numpy==2.4.6 # photo_analyzer, nsfwtag: pixel work
|
||||
Pillow==12.3.0 # photo_analyzer, nsfwtag: decode/resize
|
||||
rich==15.0.0 # photo_analyzer: console output
|
||||
scipy==1.18.0 # photo_analyzer: perceptual-hash DCT
|
||||
PyYAML==6.0.3 # tooling that reads the donor ledger
|
||||
|
||||
# NSFW inference (nsfwtag/scoring.py, nsfwtag/bench.py) was never installed in the
|
||||
# archiving environment; the model stack is recorded here from the sources so a
|
||||
# forensic run can reproduce it, not from a resolved lock:
|
||||
# torch, transformers, timm — AdamCodd/vit-base-nsfw-detector (see nsfwtag/README.md)
|
||||
# exiftool is an external binary, not a Python package.
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Application management CLI: ``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores}``."""
|
||||
"""Application management CLI: ``python -m photo_pipeline {serve,migrate}``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -16,17 +16,6 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
commands.add_parser("migrate", help="Upgrade the database to the latest revision")
|
||||
worker_cmd = commands.add_parser("worker", help="Run a durable-job worker")
|
||||
worker_cmd.add_argument("--id", default="worker-1", help="Worker id (lease owner)")
|
||||
import_cmd = commands.add_parser(
|
||||
"import-legacy-scores",
|
||||
help="Import the archived CLI's nsfw_scores.csv into the database (US07-01)",
|
||||
)
|
||||
import_cmd.add_argument("csv", help="Path to nsfw_scores.csv")
|
||||
import_cmd.add_argument(
|
||||
"--overwrite", action="store_true", help="Replace differing imported scores"
|
||||
)
|
||||
import_cmd.add_argument(
|
||||
"--dry-run", action="store_true", help="Report what would happen and change nothing"
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
config = Config.from_env()
|
||||
@@ -36,24 +25,6 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
run_migrations(config.database_url)
|
||||
return 0
|
||||
|
||||
if args.command == "import-legacy-scores":
|
||||
import json
|
||||
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory
|
||||
from photo_pipeline.services.legacy_import import LegacyImportService, write_report
|
||||
|
||||
run_migrations(config.database_url)
|
||||
engine = create_db_engine(config.database_url)
|
||||
service = LegacyImportService(create_session_factory(engine))
|
||||
report = service.import_nsfw_scores(
|
||||
args.csv, overwrite=args.overwrite, dry_run=args.dry_run
|
||||
)
|
||||
# The report is the point: an import nobody can audit is not a migration.
|
||||
if not args.dry_run:
|
||||
write_report(report, config.data_dir)
|
||||
print(json.dumps(report.counts, indent=2))
|
||||
return 0
|
||||
|
||||
if args.command == "worker":
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory
|
||||
|
||||
|
||||
@@ -208,12 +208,9 @@ class ArchiveService:
|
||||
report["blockers"].append(
|
||||
_issue(
|
||||
"insufficient_capacity",
|
||||
# The free-space number is deliberately left out: it drifts between
|
||||
# two identical preflights, and the token is a digest of this text,
|
||||
# so quoting it here would invalidate every approval instantly.
|
||||
f"{report['totals']['bytes']} B plus a "
|
||||
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit on "
|
||||
"the medium",
|
||||
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit in "
|
||||
f"{report['capacity']['free_bytes']} B of free space",
|
||||
)
|
||||
)
|
||||
report["backup"] = self._backup_probe()
|
||||
|
||||
@@ -1,220 +0,0 @@
|
||||
"""Import the last path-keyed CSV state into the database (US07-01).
|
||||
|
||||
``nsfwtag`` cached its safety scores in ``nsfw_scores.csv`` next to the library:
|
||||
one ``path,nsfw_score`` row per photo, four decimals, unreadable rows dropped. That
|
||||
file stops being a source of truth when the CLI is archived, so its scores are
|
||||
imported once — as scored-but-unreviewed ``safety_reviews`` rows on the stable
|
||||
``assets.id`` each path resolves to — and the CSV is left untouched on disk as its
|
||||
own backup.
|
||||
|
||||
The import is deliberately conservative, because a score is evidence about a photo
|
||||
and a path is not an identity:
|
||||
|
||||
- a row whose path matches no known asset is **unmatched**, never a new asset;
|
||||
- an asset a human already reviewed is **never** touched: a score is evidence, a
|
||||
decision is a judgement, and an import may not overwrite the second with the
|
||||
first — the difference is reported as a conflict instead;
|
||||
- a row for an asset that already carries a score is **skipped** unless
|
||||
``overwrite`` is asked for, and a differing score is reported either way;
|
||||
- an unparsable score or a duplicate path is **rejected** with its reason;
|
||||
- rerunning changes nothing that is already imported.
|
||||
|
||||
Everything it did — and everything it refused to do — comes back as a
|
||||
reconciliation report, which the caller can persist next to the database. The
|
||||
donor's own reader is the specification for the format (donor ledger:
|
||||
``nt-score-cache``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import uuid
|
||||
from collections import Counter
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview
|
||||
|
||||
REPORT_VERSION = 1
|
||||
# The donor wrote scores as four-decimal strings; anything outside 0..1 was never
|
||||
# something it produced, so it is data corruption rather than a score.
|
||||
SCORE_RANGE = (0.0, 1.0)
|
||||
# Recorded as the reviewer so an imported score is never mistaken for a judgement.
|
||||
REVIEWER = "legacy-csv-import"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ImportReport:
|
||||
source: str
|
||||
rows: int = 0
|
||||
imported: int = 0
|
||||
skipped_existing: int = 0
|
||||
unchanged: int = 0
|
||||
reviewed: int = 0
|
||||
unmatched: list[str] = field(default_factory=list)
|
||||
rejected: list[dict] = field(default_factory=list)
|
||||
conflicts: list[dict] = field(default_factory=list)
|
||||
generated_at: str = ""
|
||||
schema_version: int = REPORT_VERSION
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
@property
|
||||
def counts(self) -> dict[str, int]:
|
||||
return {
|
||||
"rows": self.rows,
|
||||
"imported": self.imported,
|
||||
"skipped_existing": self.skipped_existing,
|
||||
"unchanged": self.unchanged,
|
||||
"reviewed": self.reviewed,
|
||||
"unmatched": len(self.unmatched),
|
||||
"rejected": len(self.rejected),
|
||||
"conflicts": len(self.conflicts),
|
||||
}
|
||||
|
||||
|
||||
class LegacyImportService:
|
||||
def __init__(self, session_factory: sessionmaker) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
def import_nsfw_scores(
|
||||
self, csv_path: Path | str, *, overwrite: bool = False, dry_run: bool = False
|
||||
) -> ImportReport:
|
||||
"""Import ``nsfw_scores.csv`` onto asset identity and report what happened."""
|
||||
path = Path(csv_path)
|
||||
report = ImportReport(source=str(path))
|
||||
if not path.is_file():
|
||||
report.rejected.append({"path": str(path), "reason": "csv_missing"})
|
||||
return self._stamp(report)
|
||||
|
||||
with self._session_factory() as session:
|
||||
# Paths are matched against every occurrence an asset ever had, so a
|
||||
# photo scored before a rename is still recognised.
|
||||
by_path = self._path_index(session)
|
||||
latest = self._latest_reviews(session)
|
||||
seen: Counter[str] = Counter()
|
||||
|
||||
for row in self._rows(path, report):
|
||||
report.rows += 1
|
||||
raw_path, raw_score = row
|
||||
seen[raw_path] += 1
|
||||
if seen[raw_path] > 1:
|
||||
report.rejected.append({"path": raw_path, "reason": "duplicate_path"})
|
||||
continue
|
||||
score = _parse_score(raw_score)
|
||||
if score is None:
|
||||
report.rejected.append(
|
||||
{"path": raw_path, "reason": "unparsable_score", "value": raw_score}
|
||||
)
|
||||
continue
|
||||
asset_id = by_path.get(raw_path)
|
||||
if asset_id is None:
|
||||
report.unmatched.append(raw_path)
|
||||
continue
|
||||
|
||||
current = latest.get(asset_id)
|
||||
if current is not None and current.decision:
|
||||
# A human decided this one; the CSV is older evidence.
|
||||
report.reviewed += 1
|
||||
if current.score is None or abs(current.score - score) >= 1e-9:
|
||||
report.conflicts.append(
|
||||
{
|
||||
"path": raw_path,
|
||||
"asset_id": asset_id,
|
||||
"current": current.score,
|
||||
"decision": current.decision,
|
||||
"csv": score,
|
||||
}
|
||||
)
|
||||
continue
|
||||
if current is not None and current.score is not None:
|
||||
if abs(current.score - score) < 1e-9:
|
||||
report.unchanged += 1
|
||||
continue
|
||||
report.conflicts.append(
|
||||
{
|
||||
"path": raw_path,
|
||||
"asset_id": asset_id,
|
||||
"current": current.score,
|
||||
"decision": None,
|
||||
"csv": score,
|
||||
}
|
||||
)
|
||||
if not overwrite:
|
||||
report.skipped_existing += 1
|
||||
continue
|
||||
session.add(
|
||||
SafetyReview(
|
||||
id=str(uuid.uuid4()),
|
||||
asset_id=asset_id,
|
||||
score=score,
|
||||
reviewer=REVIEWER,
|
||||
)
|
||||
)
|
||||
latest[asset_id] = SafetyReview(asset_id=asset_id, score=score)
|
||||
report.imported += 1
|
||||
|
||||
if dry_run:
|
||||
session.rollback()
|
||||
else:
|
||||
session.commit()
|
||||
return self._stamp(report)
|
||||
|
||||
@staticmethod
|
||||
def _rows(path: Path, report: ImportReport):
|
||||
"""Yield ``(path, score)`` pairs, tolerating the donor's own sloppiness."""
|
||||
with open(path, newline="", encoding="utf-8", errors="replace") as handle:
|
||||
for row in csv.DictReader(handle):
|
||||
raw_path = (row.get("path") or "").strip()
|
||||
if not raw_path:
|
||||
report.rejected.append({"path": "", "reason": "missing_path"})
|
||||
continue
|
||||
yield raw_path, row.get("nsfw_score")
|
||||
|
||||
@staticmethod
|
||||
def _latest_reviews(session) -> dict[str, SafetyReview]:
|
||||
"""The current safety row per asset — latest wins, as everywhere else."""
|
||||
latest: dict[str, SafetyReview] = {}
|
||||
for review in session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)):
|
||||
latest[review.asset_id] = review
|
||||
return latest
|
||||
|
||||
@staticmethod
|
||||
def _path_index(session) -> dict[str, str]:
|
||||
"""Every path an asset is or was known by → its stable id."""
|
||||
index: dict[str, str] = {}
|
||||
for asset_id, path in session.execute(select(AssetPath.asset_id, AssetPath.path)):
|
||||
index.setdefault(path, asset_id)
|
||||
for asset_id, path in session.execute(select(Asset.id, Asset.current_path)):
|
||||
if path:
|
||||
index[path] = asset_id # the current path wins over a closed one
|
||||
return index
|
||||
|
||||
@staticmethod
|
||||
def _stamp(report: ImportReport) -> ImportReport:
|
||||
report.generated_at = datetime.now(timezone.utc).isoformat()
|
||||
return report
|
||||
|
||||
|
||||
def write_report(report: ImportReport, directory: Path) -> Path:
|
||||
"""Persist the reconciliation report; the import is not evidence until it is."""
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / "legacy-nsfw-import.json"
|
||||
path.write_text(json.dumps(report.to_dict(), indent=2, sort_keys=True), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _parse_score(value) -> float | None:
|
||||
try:
|
||||
score = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not SCORE_RANGE[0] <= score <= SCORE_RANGE[1]:
|
||||
return None
|
||||
return score
|
||||
@@ -140,11 +140,9 @@ class RestoreService:
|
||||
report["blockers"].append(
|
||||
_issue(
|
||||
"insufficient_capacity",
|
||||
# No free-space number here: it drifts between two identical
|
||||
# preflights and the token is a digest of this text (US06-06).
|
||||
f"{report['totals']['bytes']} B plus a "
|
||||
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit in "
|
||||
"the library",
|
||||
f"{report['capacity']['free_bytes']} B of free space",
|
||||
)
|
||||
)
|
||||
if not items:
|
||||
|
||||
@@ -24,8 +24,6 @@ test = [
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
# The frozen CLI archive is evidence, not code under maintenance (US07-01).
|
||||
extend-exclude = ["legacy_cli_archive"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
@@ -14,11 +14,6 @@ import pytest
|
||||
from PIL import Image
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
# The donors are frozen in the read-only archive (US07-01). Only this suite — and
|
||||
# the parity test that compares against them — puts that directory on sys.path;
|
||||
# production never does, which tests/unit/test_legacy_archive.py enforces.
|
||||
ARCHIVED_SOURCES = REPO / "legacy_cli_archive" / "src"
|
||||
sys.path.insert(0, str(ARCHIVED_SOURCES))
|
||||
sys.path.insert(0, str(REPO))
|
||||
|
||||
EXIFTOOL = shutil.which("exiftool")
|
||||
|
||||
@@ -1,20 +1,13 @@
|
||||
"""Ledger lint (US01-01, extended by US07-01).
|
||||
|
||||
Every donor-ledger row must carry a real source reference — now inside the frozen
|
||||
archive — a target location, and either existing characterization test IDs or a
|
||||
real pending backlog story. Since archival (US07-01) a row may also be ``resolved``:
|
||||
its replacement has shipped, ``parity`` names tests that exist and prove it, and
|
||||
``delta`` states every intentional difference. Nothing may quietly become
|
||||
"finished" without one of those two."""
|
||||
"""Ledger lint (US01-01): every donor-ledger row must carry a real source
|
||||
reference, a target location, and either existing characterization test IDs or
|
||||
a real pending backlog story."""
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
ARCHIVE = REPO / "legacy_cli_archive"
|
||||
LEDGER = ARCHIVE / "donor_ledger.yaml"
|
||||
ARCHIVED_SOURCES = ARCHIVE / "src"
|
||||
LEDGER = REPO / "donor_ledger.yaml"
|
||||
STORIES = REPO / "delivery_backlog" / "stories"
|
||||
TESTS_DIR = Path(__file__).resolve().parent
|
||||
|
||||
@@ -22,7 +15,7 @@ CLASSIFICATIONS = {"reuse", "extract", "refactor", "replace"}
|
||||
REQUIRED_AREAS = {"discovery", "hashing", "imaging", "nsfw", "vision", "exif",
|
||||
"database", "ui", "configuration", "logging", "cancellation",
|
||||
"error"}
|
||||
STATUSES = {"characterized", "resolved", "pending"}
|
||||
STATUSES = {"characterized", "pending"}
|
||||
|
||||
|
||||
def load_rows():
|
||||
@@ -56,11 +49,10 @@ def test_rows_have_required_fields_and_unique_ids():
|
||||
|
||||
|
||||
def test_source_references_resolve():
|
||||
"""Source paths are relative to the archive: the donors moved there, whole."""
|
||||
for r in load_rows():
|
||||
src = r["source"]
|
||||
f = ARCHIVED_SOURCES / src["file"]
|
||||
assert f.is_file(), f"{r['id']}: source file {src['file']} missing from the archive"
|
||||
f = REPO / src["file"]
|
||||
assert f.is_file(), f"{r['id']}: source file {src['file']} missing"
|
||||
text = f.read_text(encoding="utf-8")
|
||||
for sym in src["symbols"]:
|
||||
assert sym in text, f"{r['id']}: symbol {sym!r} not found in {src['file']}"
|
||||
@@ -71,9 +63,7 @@ def test_rows_have_tests_or_pending_story():
|
||||
for r in load_rows():
|
||||
tests = r.get("tests", [])
|
||||
pending = r.get("pending_story")
|
||||
parity = r.get("parity", [])
|
||||
assert tests or pending or parity, \
|
||||
f"{r['id']}: neither characterization tests, parity tests, nor a pending story"
|
||||
assert tests or pending, f"{r['id']}: neither tests nor pending_story"
|
||||
for t in tests:
|
||||
assert t in known_tests, f"{r['id']}: unknown test id {t}"
|
||||
if pending:
|
||||
@@ -83,58 +73,16 @@ def test_rows_have_tests_or_pending_story():
|
||||
assert tests, f"{r['id']}: characterized rows need test ids"
|
||||
|
||||
|
||||
def test_resolved_rows_name_their_parity_or_their_delta():
|
||||
"""A resolved row is a claim that the behavior is handled. It has to say how:
|
||||
tests that prove the replacement, or a stated difference — usually both."""
|
||||
for r in load_rows():
|
||||
if r["status"] != "resolved":
|
||||
assert "parity" not in r, f"{r['id']}: parity on a non-resolved row"
|
||||
continue
|
||||
parity = r.get("parity", [])
|
||||
delta = r.get("delta")
|
||||
assert parity or delta, f"{r['id']}: resolved without parity tests or a delta"
|
||||
if delta:
|
||||
assert len(str(delta).strip()) >= 20, f"{r['id']}: delta too thin to be a reason"
|
||||
for ref in parity:
|
||||
rel, _, func = ref.partition("::")
|
||||
path = REPO / rel
|
||||
assert path.is_file(), f"{r['id']}: parity test file {rel} missing"
|
||||
assert f"def {func}" in path.read_text(encoding="utf-8"), \
|
||||
f"{r['id']}: parity test {ref} not found"
|
||||
|
||||
|
||||
def test_pending_rows_are_the_only_unfinished_work():
|
||||
"""The ledger is the honest list of what has not been carried over: a pending
|
||||
row names the story that will, and that story must still be open work."""
|
||||
for r in load_rows():
|
||||
if r["status"] != "pending":
|
||||
continue
|
||||
story = r.get("pending_story")
|
||||
assert story, f"{r['id']}: pending without a story"
|
||||
assert list(STORIES.glob(f"{story}-*.md")), f"{r['id']}: unknown story {story}"
|
||||
|
||||
|
||||
def test_every_target_module_exists():
|
||||
"""A row is only finished if the thing it points at is really there."""
|
||||
for r in load_rows():
|
||||
if r["status"] == "pending":
|
||||
continue
|
||||
modules = re.findall(r"photo_pipeline/[\w/]+\.py", str(r["target"]))
|
||||
for module in modules:
|
||||
assert (REPO / module).is_file(), f"{r['id']}: target {module} does not exist"
|
||||
|
||||
|
||||
def test_all_required_areas_covered():
|
||||
covered = {r["area"] for r in load_rows()}
|
||||
assert REQUIRED_AREAS <= covered, f"uncovered areas: {REQUIRED_AREAS - covered}"
|
||||
assert covered <= REQUIRED_AREAS, f"unknown areas: {covered - REQUIRED_AREAS}"
|
||||
|
||||
|
||||
def test_every_donor_is_in_the_archive_and_nowhere_else():
|
||||
"""US01-01 forbade archiving before characterization; US07-01 requires it after.
|
||||
Each donor exists exactly once — frozen, in the archive."""
|
||||
def test_no_legacy_file_moved():
|
||||
# US01-01 explicitly forbids moving/archiving donors; the ledger's source
|
||||
# files must all still exist at their original locations.
|
||||
for donor in ("photo_analyzer.py", "nsfwtag/scoring.py", "nsfwtag/exif.py",
|
||||
"nsfwtag/server.py", "webapp/query.py", "webapp/runner.py",
|
||||
"webapp/server.py"):
|
||||
assert (ARCHIVED_SOURCES / donor).is_file(), f"donor missing from archive: {donor}"
|
||||
assert not (REPO / donor).exists(), f"donor still live at the repo root: {donor}"
|
||||
assert (REPO / donor).is_file(), f"donor moved: {donor}"
|
||||
|
||||
@@ -393,125 +393,6 @@ def mark_upload_ready(seeded: Seeded, *, unverified: tuple[str, ...] = ()) -> No
|
||||
session.commit()
|
||||
|
||||
|
||||
# ── Phase F: an archivable album and a mountable fake medium ─────────────────
|
||||
|
||||
|
||||
def mark_uploaded(seeded: Seeded, *, album: str = "rome") -> None:
|
||||
"""Give every seeded photo the verified upload evidence archiving requires.
|
||||
|
||||
Archiving refuses anything Immich is not proven to hold, and that proof is an
|
||||
upload batch — recorded here as fixture state so the archive journeys do not
|
||||
have to re-run an upload they are not testing.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.models import Asset, UploadBatch, UploadItem
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
|
||||
with session_factory(seeded) as sf:
|
||||
with sf() as session:
|
||||
batch_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
UploadBatch(
|
||||
id=batch_id,
|
||||
album=album,
|
||||
folder=str(seeded.lib / album),
|
||||
album_name=album,
|
||||
state="succeeded",
|
||||
preflight_token="v1:e2e",
|
||||
outcome_state="verified",
|
||||
created_at=NOW,
|
||||
)
|
||||
)
|
||||
for asset in session.scalars(select(Asset)):
|
||||
if not asset.current_path:
|
||||
continue
|
||||
session.add(
|
||||
UploadItem(
|
||||
batch_id=batch_id,
|
||||
asset_id=asset.id,
|
||||
path=asset.current_path,
|
||||
sha256=sha256_file(asset.current_path),
|
||||
sha1="0" * 40,
|
||||
state="sent",
|
||||
outcome="uploaded",
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
|
||||
class ArchiveStack:
|
||||
"""A seeded, archivable library plus the server, the worker, and a fake medium.
|
||||
|
||||
The medium is an ordinary directory whose marker file makes it identifiable;
|
||||
``unmount()`` takes that marker away, which is exactly what the application sees
|
||||
when an external disk is unplugged.
|
||||
"""
|
||||
|
||||
MARKER = ".photo-pipeline-archive.json"
|
||||
|
||||
def __init__(self, tmp_path: Path, seeded: Seeded) -> None:
|
||||
self.tmp_path = tmp_path
|
||||
self.seeded = seeded
|
||||
self.archive = tmp_path / "archive"
|
||||
self.archive.mkdir(exist_ok=True)
|
||||
self.server: Server | None = None
|
||||
self.worker: subprocess.Popen | None = None
|
||||
self.base = ""
|
||||
|
||||
def start(self, *, worker: bool = True, extra_env: dict[str, str] | None = None) -> "ArchiveStack":
|
||||
env = {"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0", **(extra_env or {})}
|
||||
self.server = Server(self.seeded, extra_env=env).start()
|
||||
self.base = self.server.base
|
||||
if worker:
|
||||
self.worker = start_worker(self.seeded, extra_env=env)
|
||||
return self
|
||||
|
||||
def register(self, name: str = "external") -> dict:
|
||||
response = httpx.post(
|
||||
f"{self.base}/api/v1/archive-locations",
|
||||
json={"name": name, "root": str(self.archive)},
|
||||
timeout=20,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
def unmount(self) -> None:
|
||||
(self.archive / self.MARKER).rename(self.archive / f"{self.MARKER}.away")
|
||||
|
||||
def remount(self) -> None:
|
||||
(self.archive / f"{self.MARKER}.away").rename(self.archive / self.MARKER)
|
||||
|
||||
def plans(self) -> list[dict]:
|
||||
return httpx.get(f"{self.base}/api/v1/archive-plans", timeout=20).json()["plans"]
|
||||
|
||||
def assets(self) -> list[dict]:
|
||||
return httpx.get(
|
||||
f"{self.base}/api/v1/inventory/assets", params={"limit": 200}, timeout=20
|
||||
).json()["items"]
|
||||
|
||||
def restart_server(self) -> None:
|
||||
self.server.stop()
|
||||
self.server.start()
|
||||
|
||||
def restart_worker(self, *, extra_env: dict[str, str] | None = None) -> None:
|
||||
"""Replace the worker — a healthy one after a crashed one, by default."""
|
||||
if self.worker is not None and self.worker.poll() is None:
|
||||
self.worker.kill()
|
||||
self.worker.wait(timeout=10)
|
||||
self.worker = start_worker(
|
||||
self.seeded,
|
||||
extra_env={"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0", **(extra_env or {})},
|
||||
)
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.worker is not None:
|
||||
self.worker.kill()
|
||||
self.worker.wait(timeout=10)
|
||||
if self.server is not None:
|
||||
self.server.stop()
|
||||
|
||||
|
||||
class UploadStack:
|
||||
"""A seeded, upload-ready library plus the server, worker, and fake Immich."""
|
||||
|
||||
|
||||
@@ -1,302 +0,0 @@
|
||||
"""Browser journeys for the archive view (US06-05).
|
||||
|
||||
Archiving is the only stage that removes originals, so these journeys check the
|
||||
two things a browser must never get wrong about it: that the preview names exactly
|
||||
what would leave and where it would go, and that nothing offers an action the
|
||||
server would refuse. The medium is a real directory whose marker makes it
|
||||
identifiable; unmounting it is what an unplugged disk looks like from here.
|
||||
|
||||
Nothing is mocked inside the browser: the transfer runs in the real worker process
|
||||
and the assertions read the filesystem afterwards.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from playwright.sync_api import expect
|
||||
|
||||
from tests.e2e._pipeline_harness import (
|
||||
ArchiveStack,
|
||||
mark_uploaded,
|
||||
seed_album,
|
||||
session_factory,
|
||||
wait_until,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
||||
|
||||
TIMEOUT = 10
|
||||
RUN_TIMEOUT = 30_000
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stack(tmp_path):
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_uploaded(seeded)
|
||||
running = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
def _open(page, stack) -> None:
|
||||
page.goto(f"{stack.base}/app/#/archive")
|
||||
page.get_by_test_id("archive-locations").wait_for()
|
||||
|
||||
|
||||
def _archive(page, stack) -> None:
|
||||
"""Confirm the archive and wait for the worker to finish the run."""
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-archive").click()
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete", timeout=RUN_TIMEOUT)
|
||||
|
||||
|
||||
def _archived_paths(stack) -> list[str]:
|
||||
return sorted(p.name for p in (stack.archive / "rome").glob("*.jpg"))
|
||||
|
||||
|
||||
# ── preview and confirmation ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_preview_names_the_scope_destination_and_reclaimable_bytes(page, stack):
|
||||
errors = []
|
||||
page.on("console", lambda m: errors.append(m.text) if m.type == "error" else None)
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
_open(page, stack)
|
||||
|
||||
row = page.get_by_test_id("location-row").first
|
||||
expect(row.get_by_test_id("location-label")).to_have_text("external")
|
||||
expect(row.get_by_test_id("location-media")).to_have_text(location["media_id"])
|
||||
expect(row.get_by_test_id("location-state")).to_have_text("online")
|
||||
|
||||
album = page.get_by_test_id("archive-album-row").first
|
||||
expect(album.get_by_test_id("album-name")).to_have_text("rome")
|
||||
expect(album.get_by_test_id("album-destination")).to_have_text(str(stack.archive / "rome"))
|
||||
expect(album.get_by_test_id("album-assets")).to_have_text("2")
|
||||
expect(album.get_by_test_id("album-state")).to_have_text("ready")
|
||||
# Same filesystem here, so the transfer method is the move path — and it is
|
||||
# named, because copy-verify-remove and move fail differently.
|
||||
expect(album.get_by_test_id("album-method")).to_contain_text("move")
|
||||
|
||||
expect(page.get_by_test_id("destination-identity")).to_contain_text(location["media_id"])
|
||||
expect(page.get_by_test_id("capacity")).to_contain_text("reserve")
|
||||
expect(page.get_by_test_id("start-archive")).to_contain_text("Archive 1 album(s) · reclaim")
|
||||
expect(page.get_by_test_id("start-archive")).to_be_enabled()
|
||||
assert stack.plans() == [], "previewing may not create anything"
|
||||
assert errors == [], f"console errors: {errors}"
|
||||
|
||||
|
||||
def test_an_offline_medium_blocks_the_confirmation_and_says_what_to_mount(page, stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
stack.unmount()
|
||||
_open(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("location-state")).to_have_text("offline")
|
||||
expect(page.get_by_test_id("preflight-blocker").first).to_have_attribute(
|
||||
"data-code", "location_offline"
|
||||
)
|
||||
instruction = page.get_by_test_id("mount-instruction")
|
||||
expect(instruction).to_contain_text("external")
|
||||
expect(instruction).to_contain_text(location["media_id"])
|
||||
expect(page.get_by_test_id("start-archive")).to_be_disabled()
|
||||
|
||||
|
||||
def test_an_unuploaded_album_is_blocked_with_its_reason(page, stack):
|
||||
# No upload evidence at all: archiving would remove the only copy.
|
||||
seeded = stack.seeded
|
||||
with session_factory(seeded) as sf:
|
||||
from sqlalchemy import delete
|
||||
|
||||
from photo_pipeline.models import UploadItem
|
||||
|
||||
with sf() as session:
|
||||
session.execute(delete(UploadItem))
|
||||
session.commit()
|
||||
stack.start(worker=False)
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("album-state")).to_have_text("blocked")
|
||||
expect(page.get_by_test_id("album-blocker").first).to_have_attribute(
|
||||
"data-code", "partial_scope"
|
||||
)
|
||||
expect(page.get_by_test_id("start-archive")).to_be_disabled()
|
||||
|
||||
|
||||
# ── running, progress, reload ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_confirmed_archive_runs_and_separates_transfer_verify_and_removal(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("count-complete")).to_have_text("complete: 2")
|
||||
expect(page.get_by_test_id("count-failed")).to_have_text("failed: 0")
|
||||
expect(page.get_by_test_id("count-transfer")).to_have_text("transfer: 0")
|
||||
expect(page.get_by_test_id("count-verified")).to_have_text("verified: 0")
|
||||
expect(page.get_by_test_id("count-removing")).to_have_text("removing: 0")
|
||||
expect(page.get_by_test_id("operation-phase").first).to_have_text("complete")
|
||||
|
||||
# The library really lost the originals and the medium really holds them.
|
||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"]
|
||||
assert not (stack.seeded.lib / "rome" / "a.jpg").exists()
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
||||
|
||||
|
||||
def test_the_run_survives_a_reload_because_the_state_is_the_servers(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("plan-detail").wait_for()
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete")
|
||||
expect(page.get_by_test_id("count-complete")).to_have_text("complete: 2")
|
||||
# And the run stays addressable by its own URL.
|
||||
plan_id = stack.plans()[0]["id"]
|
||||
page.goto(f"{stack.base}/app/#/archive?plan={plan_id}")
|
||||
expect(page.get_by_test_id("plan-detail")).to_have_attribute("data-plan", plan_id)
|
||||
|
||||
|
||||
# ── recovery ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_an_interrupted_transfer_is_shown_with_evidence_and_a_safe_action(page, stack):
|
||||
stack.start(worker=False)
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-archive").click()
|
||||
expect(page.get_by_test_id("archive-result")).to_be_visible()
|
||||
# No worker ran, so every item is still planned; leave one mid-transfer as a
|
||||
# crash would: intent recorded, nothing published, source intact.
|
||||
_interrupt(stack)
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("archive-recovery").wait_for()
|
||||
row = page.get_by_test_id("recovery-row").first
|
||||
expect(row).to_have_attribute("data-classification", "resumable")
|
||||
expect(row.get_by_test_id("recovery-reason")).to_contain_text("source present")
|
||||
page.get_by_test_id("resolve-recovery").click()
|
||||
expect(page.get_by_test_id("archive-recovery")).to_have_count(0)
|
||||
assert (stack.seeded.lib / "rome" / "a.jpg").exists(), "recovery may not move anything"
|
||||
|
||||
|
||||
def test_ambiguous_evidence_offers_no_action_at_all(page, stack):
|
||||
stack.start(worker=False)
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-archive").click()
|
||||
expect(page.get_by_test_id("archive-result")).to_be_visible()
|
||||
# Journal says the copy is verified, but the medium holds nothing: no evidence
|
||||
# supports either finishing or retrying this item.
|
||||
_interrupt(stack, state="verified")
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("archive-recovery").wait_for()
|
||||
expect(page.get_by_test_id("recovery-row").first).to_have_attribute(
|
||||
"data-classification", "manual"
|
||||
)
|
||||
expect(page.get_by_test_id("recovery-manual")).to_be_visible()
|
||||
expect(page.get_by_test_id("resolve-recovery")).to_have_count(0)
|
||||
expect(page.get_by_test_id("no-safe-recovery")).to_be_visible()
|
||||
|
||||
|
||||
def _interrupt(stack, *, state: str = "transferring") -> None:
|
||||
"""Leave the plan's first operation in a non-terminal journal state."""
|
||||
with session_factory(stack.seeded) as sf:
|
||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
||||
|
||||
journal = ArchiveJournal(sf)
|
||||
plan_id = stack.plans()[0]["id"]
|
||||
operation = journal.operations(plan_id)[0]
|
||||
journal.begin(operation["id"], worker_id="crashed", fencing_token=1)
|
||||
if state != "transferring":
|
||||
journal.transition(operation["id"], state, fencing_token=1)
|
||||
|
||||
|
||||
# ── offline browsing and restore ─────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_archived_photos_stay_browsable_while_the_medium_is_away(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
stack.unmount()
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("archived-assets").wait_for()
|
||||
row = page.get_by_test_id("archived-row").first
|
||||
expect(row.get_by_test_id("archived-availability")).to_have_text("archived · medium away")
|
||||
expect(row.get_by_test_id("archived-path")).to_contain_text("rome/")
|
||||
expect(row.get_by_test_id("archived-medium")).to_have_text("external")
|
||||
expect(page.get_by_test_id("mount-instruction")).to_contain_text("external")
|
||||
# The retained protected preview is served even though the original is gone.
|
||||
preview = row.get_by_test_id("archived-preview")
|
||||
assert preview.evaluate("img => img.complete && img.naturalWidth > 0")
|
||||
# Restoring is impossible right now, and says so instead of failing later.
|
||||
expect(page.get_by_test_id("start-restore")).to_be_disabled()
|
||||
|
||||
|
||||
def test_restoring_brings_the_photos_back_without_losing_identity(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
before = {asset["id"] for asset in stack.assets()}
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("restore").wait_for()
|
||||
expect(page.get_by_test_id("restore-row").first.get_by_test_id("restore-destination")).to_contain_text(
|
||||
str(stack.seeded.lib / "rome")
|
||||
)
|
||||
page.get_by_test_id("start-restore").click()
|
||||
wait_until(
|
||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()),
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
assert {asset["id"] for asset in stack.assets()} == before # same identities
|
||||
assert (stack.seeded.lib / "rome" / "a.jpg").exists()
|
||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"] # the archive copy stays
|
||||
|
||||
|
||||
def test_a_taken_name_is_restored_beside_it_never_over_it(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
squatter = stack.seeded.lib / "rome" / "a.jpg"
|
||||
squatter.parent.mkdir(parents=True, exist_ok=True)
|
||||
squatter.write_bytes(b"a different photo now lives here")
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("restore").wait_for()
|
||||
destinations = page.get_by_test_id("restore-destination").all_inner_texts()
|
||||
assert any("(restored)" in text for text in destinations), destinations
|
||||
|
||||
page.get_by_test_id("start-restore").click()
|
||||
wait_until(
|
||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()),
|
||||
timeout=30,
|
||||
)
|
||||
assert squatter.read_bytes() == b"a different photo now lives here"
|
||||
assert (stack.seeded.lib / "rome" / "a (restored).jpg").exists()
|
||||
|
||||
|
||||
# ── keyboard ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_whole_archive_can_be_confirmed_from_the_keyboard(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
|
||||
button = page.get_by_test_id("start-archive")
|
||||
button.focus()
|
||||
expect(button).to_be_focused()
|
||||
page.keyboard.press("Enter")
|
||||
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete", timeout=RUN_TIMEOUT)
|
||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"]
|
||||
@@ -1,505 +0,0 @@
|
||||
"""Phase F end-to-end acceptance (US06-06): archive lifecycle, black box.
|
||||
|
||||
Every journey drives a real ``photo_pipeline serve`` child process and a real
|
||||
durable worker over HTTP — register a medium, preflight it, archive an album,
|
||||
crash mid-transfer, recover, unmount the medium, rediscover the photos through
|
||||
their hashes while they are unreachable, mount it again, restore, and collide.
|
||||
Nothing is reached into: the medium is an ordinary directory whose marker file is
|
||||
its identity, and unmounting it means taking that marker away, which is what the
|
||||
application sees when a disk is unplugged.
|
||||
|
||||
Three invariants are asserted wherever they apply, because they are what make the
|
||||
only *removing* stage safe:
|
||||
|
||||
- **No source is removed before its archive copy is verified.** Every crash barrier
|
||||
is checked for the file still being somewhere: either in the library, or on the
|
||||
medium hashing exactly as recorded.
|
||||
- **Archived is not missing.** An unmounted medium leaves its photos
|
||||
``archived_offline``, still hashed, still deduplicable, still previewable.
|
||||
- **Restoring never overwrites.** A taken name comes back beside its occupant, and
|
||||
the archived copy stays on the medium.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from tests.e2e._pipeline_harness import (
|
||||
ArchiveStack,
|
||||
mark_uploaded,
|
||||
seed_album,
|
||||
session_factory,
|
||||
wait_until,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.phase_f
|
||||
|
||||
TIMEOUT = 20
|
||||
ALBUM = "rome"
|
||||
MANIFEST = "archive-manifest.jsonl"
|
||||
# Every persisted transition the transfer can die at, in the order it reaches them.
|
||||
BARRIERS = ["transferring", "verified", "removing", "source_removed", "complete"]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stack(tmp_path):
|
||||
"""An analysed album with verified upload evidence — archivable, nothing else."""
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_uploaded(seeded)
|
||||
running = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
# ── helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _preflight(stack, location_id: str, **body) -> dict:
|
||||
response = httpx.post(
|
||||
f"{stack.base}/api/v1/archive-preflight",
|
||||
json={"location_id": location_id, **body},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _plan(stack, location_id: str, **body) -> dict:
|
||||
report = _preflight(stack, location_id, **body)
|
||||
response = httpx.post(
|
||||
f"{stack.base}/api/v1/archive-plans",
|
||||
json={"location_id": location_id, "token": report["token"], **body},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _apply(stack, plan_id: str) -> httpx.Response:
|
||||
return httpx.post(f"{stack.base}/api/v1/archive-plans/{plan_id}/apply", timeout=TIMEOUT)
|
||||
|
||||
|
||||
def _plan_state(stack, plan_id: str) -> dict:
|
||||
return httpx.get(f"{stack.base}/api/v1/archive-plans/{plan_id}", timeout=TIMEOUT).json()
|
||||
|
||||
|
||||
def _await_plan(stack, plan_id: str, states=("complete", "failed"), *, timeout: float = 60) -> dict:
|
||||
return wait_until(
|
||||
lambda: (lambda p: p if p.get("state") in states else None)(_plan_state(stack, plan_id)),
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def _archive_album(stack) -> dict:
|
||||
"""Register, preflight, plan, apply, and wait for the run to finish."""
|
||||
location = stack.register()
|
||||
plan = _plan(stack, location["id"])
|
||||
_apply(stack, plan["id"]).raise_for_status()
|
||||
return {"location": location, "plan": _await_plan(stack, plan["id"])}
|
||||
|
||||
|
||||
def _assets(stack) -> dict[str, dict]:
|
||||
return {asset["id"]: asset for asset in stack.assets()}
|
||||
|
||||
|
||||
def _recovery(stack) -> dict:
|
||||
return httpx.get(f"{stack.base}/api/v1/archive-recovery", timeout=TIMEOUT).json()
|
||||
|
||||
|
||||
def _resolve(stack) -> dict:
|
||||
response = httpx.post(f"{stack.base}/api/v1/archive-recovery/resolve", timeout=TIMEOUT)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _restore(stack, location_id: str) -> dict:
|
||||
report = httpx.post(
|
||||
f"{stack.base}/api/v1/restore-preflight",
|
||||
json={"location_id": location_id},
|
||||
timeout=TIMEOUT,
|
||||
).json()
|
||||
plan = httpx.post(
|
||||
f"{stack.base}/api/v1/restore-plans",
|
||||
json={"location_id": location_id, "token": report["token"]},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
plan.raise_for_status()
|
||||
plan = plan.json()
|
||||
httpx.post(
|
||||
f"{stack.base}/api/v1/restore-plans/{plan['id']}/apply", timeout=TIMEOUT
|
||||
).raise_for_status()
|
||||
wait_until(
|
||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()), timeout=60
|
||||
)
|
||||
return {"preflight": report, "plan": plan}
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def _contents(*roots: Path) -> set[str]:
|
||||
"""Every photo byte-string reachable anywhere, keyed by hash."""
|
||||
return {
|
||||
_sha256(path)
|
||||
for root in roots
|
||||
for path in root.rglob("*.jpg")
|
||||
if path.is_file() and not path.name.startswith(".")
|
||||
}
|
||||
|
||||
|
||||
def _manifest(stack) -> list[dict]:
|
||||
path = stack.archive / ALBUM / MANIFEST
|
||||
if not path.exists():
|
||||
return []
|
||||
return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
|
||||
|
||||
|
||||
# ── preflight blockers ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_preflight_refuses_an_offline_medium_a_wrong_volume_and_a_full_disk(stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
|
||||
ready = _preflight(stack, location["id"])
|
||||
assert ready["state"] == "ready"
|
||||
assert ready["totals"]["assets"] == 2
|
||||
|
||||
stack.unmount()
|
||||
offline = _preflight(stack, location["id"])
|
||||
assert offline["state"] == "blocked"
|
||||
assert "location_offline" in {issue["code"] for issue in offline["blockers"]}
|
||||
|
||||
# A different disk mounted at the same place is not this medium.
|
||||
(stack.archive / ArchiveStack.MARKER).write_text('{"media_id": "another-disk"}')
|
||||
wrong = _preflight(stack, location["id"])
|
||||
assert "wrong_volume" in {issue["code"] for issue in wrong["blockers"]}
|
||||
|
||||
# Nothing has been created, moved, or removed by any of that.
|
||||
assert stack.plans() == []
|
||||
assert sorted(p.name for p in (stack.seeded.lib / ALBUM).glob("*.jpg")) == ["a.jpg", "b.jpg"]
|
||||
|
||||
|
||||
def test_preflight_refuses_capacity_it_cannot_prove_and_bytes_it_cannot_vouch_for(tmp_path):
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_uploaded(seeded)
|
||||
stack = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
# A reserve larger than any disk: the scope cannot be promised room.
|
||||
stack.start(
|
||||
worker=False,
|
||||
extra_env={"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": str(1 << 62)},
|
||||
)
|
||||
location = stack.register()
|
||||
report = _preflight(stack, location["id"])
|
||||
assert "insufficient_capacity" in {issue["code"] for issue in report["blockers"]}
|
||||
|
||||
# And a plan for a blocked scope is refused rather than half-created.
|
||||
refused = httpx.post(
|
||||
f"{stack.base}/api/v1/archive-plans",
|
||||
json={"location_id": location["id"], "token": report["token"]},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
assert refused.status_code == 422
|
||||
assert refused.json()["error"]["code"] == "blocked"
|
||||
assert stack.plans() == []
|
||||
finally:
|
||||
stack.stop()
|
||||
|
||||
|
||||
def test_an_album_whose_bytes_changed_after_upload_cannot_be_archived(stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
(stack.seeded.lib / ALBUM / "a.jpg").write_bytes(b"edited after the upload")
|
||||
|
||||
report = _preflight(stack, location["id"])
|
||||
album = report["albums"][0]
|
||||
assert album["state"] == "blocked"
|
||||
assert "partial_scope" in {issue["code"] for issue in album["blockers"]}
|
||||
assert "bytes_changed" in {
|
||||
issue["code"] for asset in album["assets"] for issue in asset["blockers"]
|
||||
}
|
||||
assert report["state"] == "blocked"
|
||||
|
||||
|
||||
# ── copy, verify, remove ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_an_archived_album_leaves_the_library_only_after_it_is_verified(stack):
|
||||
stack.start()
|
||||
before = _contents(stack.seeded.lib)
|
||||
hashes = {a["id"]: a["current_sha256"] for a in stack.assets()}
|
||||
|
||||
result = _archive_album(stack)
|
||||
plan = result["plan"]
|
||||
|
||||
assert plan["state"] == "complete"
|
||||
assert {op["journal_state"] for op in plan["operations"]} == {"complete"}
|
||||
# Same filesystem here, so the transfer took the atomic-move path — recorded as
|
||||
# what actually happened, not as the plan's guess.
|
||||
assert all(op["same_filesystem"] for op in plan["operations"])
|
||||
|
||||
# The bytes are on the medium, hashing exactly as recorded, and gone from the
|
||||
# library. Not one photo was lost in between.
|
||||
assert _contents(stack.archive) == before
|
||||
assert not list((stack.seeded.lib / ALBUM).glob("*.jpg"))
|
||||
for asset_id, sha256 in hashes.items():
|
||||
asset = _assets(stack)[asset_id]
|
||||
assert asset["current_path"] is None
|
||||
assert asset["availability_state"] == "archived_online"
|
||||
assert _sha256(stack.archive / asset["archive_path"]) == sha256
|
||||
|
||||
# The medium carries its own record of what it holds.
|
||||
manifest = _manifest(stack)
|
||||
assert {entry["asset_id"] for entry in manifest} == set(hashes)
|
||||
assert {entry["sha256"] for entry in manifest} == set(hashes.values())
|
||||
assert {entry["media_id"] for entry in manifest} == {result["location"]["media_id"]}
|
||||
|
||||
|
||||
def test_the_archived_state_survives_a_server_restart(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
before = _assets(stack)
|
||||
plan_before = _plan_state(stack, result["plan"]["id"])
|
||||
|
||||
stack.restart_server()
|
||||
|
||||
assert _assets(stack) == before
|
||||
assert _plan_state(stack, result["plan"]["id"]) == plan_before
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
||||
|
||||
|
||||
# ── interruption at every persisted transition ───────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("barrier", BARRIERS)
|
||||
def test_a_worker_killed_at_each_transition_loses_nothing_and_recovers(tmp_path, barrier):
|
||||
# One photo, so the crash lands on the only operation and the end state is the
|
||||
# recovery's own doing rather than a mixture with untouched work.
|
||||
seeded = seed_album(tmp_path, names=("a.jpg",))
|
||||
mark_uploaded(seeded)
|
||||
stack = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
# Only the worker carries the crash barrier: it dies the instant this journal
|
||||
# state is persisted, while the server stays up to be asked what happened.
|
||||
stack.start(worker=False)
|
||||
stack.restart_worker(extra_env={"PHOTO_PIPELINE_FAULT_AFTER": barrier})
|
||||
before = _contents(stack.seeded.lib)
|
||||
location = stack.register()
|
||||
plan = _plan(stack, location["id"])
|
||||
_apply(stack, plan["id"]).raise_for_status()
|
||||
|
||||
wait_until(lambda: stack.worker.poll() is not None, timeout=60)
|
||||
assert stack.worker.returncode in (9, -9), "the worker should have been killed"
|
||||
|
||||
# Whatever the crash interrupted, every photo is still readable somewhere.
|
||||
assert before <= _contents(stack.seeded.lib, stack.archive), f"content lost at {barrier}"
|
||||
# And nothing was removed that had not been verified first.
|
||||
for operation in _plan_state(stack, plan["id"])["operations"]:
|
||||
source = Path(operation["source_path"])
|
||||
if not source.exists():
|
||||
archived = Path(operation["destination_path"])
|
||||
assert archived.exists(), f"{barrier}: source removed without an archive copy"
|
||||
assert _sha256(archived) == operation["expected_sha256"]
|
||||
|
||||
# Recovery is offered from evidence, resolves without a worker, and converges.
|
||||
verdicts = _recovery(stack)
|
||||
assert all(v["classification"] != "manual" for v in verdicts["operations"]), verdicts
|
||||
_resolve(stack)
|
||||
assert _recovery(stack)["operations"] == []
|
||||
# Repeating it changes nothing.
|
||||
assert _resolve(stack) == {"resumed": 0, "completed": 0, "manual": 0}
|
||||
|
||||
# Recovery finishes what was durable and re-plans what was not. Before the
|
||||
# archive copy existed, that means putting the item back to `planned` with the
|
||||
# original untouched; from `verified` onwards it means completing it.
|
||||
stack.restart_server()
|
||||
operation = _plan_state(stack, plan["id"])["operations"][0]
|
||||
asset = next(iter(stack.assets()))
|
||||
if barrier == "transferring":
|
||||
assert operation["journal_state"] == "planned"
|
||||
assert Path(operation["source_path"]).exists()
|
||||
assert asset["availability_state"] == "active"
|
||||
assert _contents(stack.seeded.lib) == before
|
||||
else:
|
||||
assert operation["journal_state"] == "complete"
|
||||
assert not Path(operation["source_path"]).exists()
|
||||
assert _contents(stack.archive) == before
|
||||
assert asset["availability_state"] == "archived_online"
|
||||
assert _sha256(stack.archive / asset["archive_path"]) == asset["current_sha256"]
|
||||
assert {entry["sha256"] for entry in _manifest(stack)} == before
|
||||
finally:
|
||||
stack.stop()
|
||||
|
||||
|
||||
def test_evidence_that_contradicts_the_journal_is_never_guessed(stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
plan = _plan(stack, location["id"])
|
||||
# The journal claims a verified archive copy the medium does not have.
|
||||
with session_factory(stack.seeded) as sf:
|
||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
||||
|
||||
journal = ArchiveJournal(sf)
|
||||
operation = journal.operations(plan["id"])[0]
|
||||
journal.begin(operation["id"], worker_id="crashed", fencing_token=1)
|
||||
journal.transition(operation["id"], "verified", fencing_token=1)
|
||||
|
||||
verdicts = _recovery(stack)
|
||||
assert [v["classification"] for v in verdicts["operations"]] == ["manual"]
|
||||
assert _resolve(stack)["manual"] == 1
|
||||
# The source is untouched and the unresolved item keeps blocking a new archive.
|
||||
assert (stack.seeded.lib / ALBUM / "a.jpg").exists()
|
||||
blocked = _preflight(stack, location["id"])
|
||||
assert "archive_pending" in {issue["code"] for issue in blocked["blockers"]}
|
||||
|
||||
|
||||
# ── offline identity and deduplication ───────────────────────────────────────
|
||||
|
||||
|
||||
def test_offline_photos_stay_hashed_previewable_and_deduplicable(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
archived = _assets(stack)
|
||||
copy_source = stack.archive / next(iter(archived.values()))["archive_path"]
|
||||
exact = stack.seeded.lib / "inbox" / "again.jpg"
|
||||
exact.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(copy_source, exact)
|
||||
stack.unmount()
|
||||
|
||||
scanned = httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60)
|
||||
scanned.raise_for_status()
|
||||
# An unmounted medium is not a missing file.
|
||||
assert scanned.json()["counts"].get("missing") is None
|
||||
offline = {a["id"]: a for a in stack.assets() if a["id"] in archived}
|
||||
assert {a["availability_state"] for a in offline.values()} == {"archived_offline"}
|
||||
assert all(a["current_sha256"] for a in offline.values())
|
||||
|
||||
# The new active copy is recognised as the archived photo, not as a new one.
|
||||
httpx.post(f"{stack.base}/api/v1/duplicates/detect", timeout=60).raise_for_status()
|
||||
clusters = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters", params={"limit": 50}, timeout=TIMEOUT
|
||||
).json()["items"]
|
||||
exact_clusters = [c for c in clusters if c["method"] == "exact"]
|
||||
assert len(exact_clusters) == 1
|
||||
detail = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters/{exact_clusters[0]['id']}", timeout=TIMEOUT
|
||||
).json()
|
||||
canonical = next(m for m in detail["members"] if m["asset_id"] == detail["canonical_asset_id"])
|
||||
assert canonical["availability_state"] == "archived_offline"
|
||||
assert canonical["archive_location"] == result["location"]["name"]
|
||||
assert canonical["preview"]["state"] == "ready" and canonical["preview"]["protected"]
|
||||
|
||||
# And the retained preview really is served while the medium is away.
|
||||
thumbnail = httpx.get(
|
||||
f"{stack.base}/api/v1/assets/{canonical['asset_id']}/thumbnail",
|
||||
params={"size": 1280},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
assert thumbnail.status_code == 200
|
||||
assert thumbnail.headers["content-type"] == "image/webp"
|
||||
|
||||
|
||||
def test_a_fuzzy_offline_match_asks_for_the_medium_instead_of_guessing(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
archived = next(iter(_assets(stack).values()))
|
||||
variant = stack.seeded.lib / "inbox" / "resized.jpg"
|
||||
variant.parent.mkdir(parents=True, exist_ok=True)
|
||||
_resize(stack.archive / archived["archive_path"], variant)
|
||||
stack.unmount()
|
||||
|
||||
httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60).raise_for_status()
|
||||
httpx.post(f"{stack.base}/api/v1/duplicates/detect", timeout=60).raise_for_status()
|
||||
clusters = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters", params={"limit": 50}, timeout=TIMEOUT
|
||||
).json()["items"]
|
||||
perceptual = [c for c in clusters if c["method"] == "perceptual"]
|
||||
assert len(perceptual) == 1
|
||||
|
||||
detail = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters/{perceptual[0]['id']}", timeout=TIMEOUT
|
||||
).json()
|
||||
# A fuzzy match is never decided automatically, and full-resolution review names
|
||||
# the medium to mount rather than guessing from the preview.
|
||||
assert detail["state"] == "open" and detail["requires_confirmation"] is True
|
||||
assert detail["mount_required"] == [result["location"]["name"]]
|
||||
|
||||
|
||||
def test_mounting_the_medium_again_makes_the_originals_reachable(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
stack.unmount()
|
||||
httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60).raise_for_status()
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_offline"}
|
||||
# Restoring from a medium that is not there is refused, with its reason.
|
||||
away = httpx.post(
|
||||
f"{stack.base}/api/v1/restore-preflight",
|
||||
json={"location_id": result["location"]["id"]},
|
||||
timeout=TIMEOUT,
|
||||
).json()
|
||||
assert away["state"] == "blocked"
|
||||
assert "location_offline" in {issue["code"] for issue in away["blockers"]}
|
||||
|
||||
stack.remount()
|
||||
locations = httpx.get(f"{stack.base}/api/v1/archive-locations", timeout=TIMEOUT).json()
|
||||
assert locations["locations"][0]["state"] == "online"
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
||||
|
||||
|
||||
# ── restore ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_restoring_returns_the_bytes_without_losing_identity_or_the_archive(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
identities = set(_assets(stack))
|
||||
hashes = {a["id"]: a["current_sha256"] for a in stack.assets()}
|
||||
|
||||
_restore(stack, result["location"]["id"])
|
||||
|
||||
restored = _assets(stack)
|
||||
assert set(restored) == identities # the same photos, not new ones
|
||||
for asset_id, sha256 in hashes.items():
|
||||
asset = restored[asset_id]
|
||||
assert asset["availability_state"] == "active"
|
||||
assert _sha256(Path(asset["current_path"])) == sha256
|
||||
# The archive copy is a copy: restoring empties nothing.
|
||||
assert (stack.archive / asset["archive_path"]).exists()
|
||||
|
||||
stack.restart_server()
|
||||
assert _assets(stack) == restored
|
||||
|
||||
|
||||
def test_a_restore_comes_back_beside_an_occupant_never_over_it(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
squatter = stack.seeded.lib / ALBUM / "a.jpg"
|
||||
squatter.parent.mkdir(parents=True, exist_ok=True)
|
||||
squatter.write_bytes(b"a different photo lives here now")
|
||||
|
||||
report = _restore(stack, result["location"]["id"])["preflight"]
|
||||
destinations = {item["destination_path"] for item in report["items"]}
|
||||
assert any("(restored)" in destination for destination in destinations)
|
||||
|
||||
assert squatter.read_bytes() == b"a different photo lives here now"
|
||||
assert (stack.seeded.lib / ALBUM / "a (restored).jpg").exists()
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"active"}
|
||||
|
||||
|
||||
def _resize(source: Path, destination: Path, scale: float = 0.5) -> None:
|
||||
from PIL import Image
|
||||
|
||||
with Image.open(source) as opened:
|
||||
opened.resize(
|
||||
(int(opened.width * scale), int(opened.height * scale)), Image.LANCZOS
|
||||
).save(destination, quality=95)
|
||||
@@ -14,7 +14,6 @@ MAP = json.loads((REPO / "tests" / "story_traceability.json").read_text())["stor
|
||||
PHASE_A_STORIES = {f"US01-0{n}" for n in range(1, 8)}
|
||||
PHASE_D_STORIES = {f"US04-0{n}" for n in range(1, 7)}
|
||||
PHASE_E_STORIES = {f"US05-0{n}" for n in range(1, 7)}
|
||||
PHASE_F_STORIES = {f"US06-0{n}" for n in range(1, 7)}
|
||||
|
||||
|
||||
def test_all_phase_a_stories_are_mapped():
|
||||
@@ -33,13 +32,6 @@ def test_all_phase_e_stories_are_mapped():
|
||||
assert PHASE_E_STORIES <= set(MAP)
|
||||
|
||||
|
||||
def test_all_phase_f_stories_are_mapped():
|
||||
"""US06-06 acceptance: every archive-lifecycle story, destination through
|
||||
restore, is tied to automated tests — archiving is the only stage that removes
|
||||
an original."""
|
||||
assert PHASE_F_STORIES <= set(MAP)
|
||||
|
||||
|
||||
def test_every_mapped_test_file_exists_and_is_nonempty():
|
||||
for story, files in MAP.items():
|
||||
assert files, f"{story} maps to no tests"
|
||||
|
||||
@@ -1,238 +0,0 @@
|
||||
"""Importing the archived CLI's CSV state into the database (US07-01).
|
||||
|
||||
The donor's ``nsfw_scores.csv`` was keyed by path; the database is keyed by a
|
||||
stable asset id. Every case here is about that gap: a path that moved, a path that
|
||||
matches nothing, a score a human has already overruled, and a file that is simply
|
||||
malformed. The import may add evidence and must never invent an asset, overwrite a
|
||||
judgement, or fail silently — whatever it does ends up in the report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview
|
||||
from photo_pipeline.services.legacy_import import (
|
||||
REVIEWER,
|
||||
LegacyImportService,
|
||||
write_report,
|
||||
)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def factory(tmp_path):
|
||||
url = f"sqlite:///{tmp_path / 'legacy.db'}"
|
||||
run_migrations(url)
|
||||
engine = create_db_engine(url)
|
||||
yield create_session_factory(engine)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _asset(sf, path: Path, *, previous: Path | None = None) -> str:
|
||||
asset_id = str(uuid.uuid4())
|
||||
with sf() as session:
|
||||
session.add(
|
||||
Asset(
|
||||
id=asset_id,
|
||||
original_path=str(previous or path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
)
|
||||
)
|
||||
session.add(AssetPath(asset_id=asset_id, path=str(path), valid_from=NOW))
|
||||
if previous is not None:
|
||||
session.add(
|
||||
AssetPath(
|
||||
asset_id=asset_id, path=str(previous), valid_from=NOW, valid_until=NOW
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return asset_id
|
||||
|
||||
|
||||
def _review(sf, asset_id: str, *, score=None, decision=None, at=NOW) -> None:
|
||||
with sf() as session:
|
||||
session.add(
|
||||
SafetyReview(
|
||||
id=str(uuid.uuid4()),
|
||||
asset_id=asset_id,
|
||||
score=score,
|
||||
decision=decision,
|
||||
reviewer="dom",
|
||||
created_at=at,
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
|
||||
def _csv(tmp_path, rows: str) -> Path:
|
||||
path = tmp_path / "nsfw_scores.csv"
|
||||
path.write_text("path,nsfw_score\n" + rows, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _scores(sf) -> dict[str, float]:
|
||||
with sf() as session:
|
||||
return {
|
||||
review.asset_id: review.score
|
||||
for review in session.scalars(
|
||||
select(SafetyReview).order_by(SafetyReview.created_at)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def test_scores_are_imported_onto_asset_identity(tmp_path, factory):
|
||||
a = _asset(factory, tmp_path / "a.jpg")
|
||||
# This one was scored under its old path and has since moved.
|
||||
b = _asset(factory, tmp_path / "moved" / "b.jpg", previous=tmp_path / "b.jpg")
|
||||
csv_path = _csv(
|
||||
tmp_path, f"{tmp_path / 'a.jpg'},0.9123\n{tmp_path / 'b.jpg'},0.0100\n"
|
||||
)
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
assert report.counts == {
|
||||
"rows": 2,
|
||||
"imported": 2,
|
||||
"skipped_existing": 0,
|
||||
"unchanged": 0,
|
||||
"reviewed": 0,
|
||||
"unmatched": 0,
|
||||
"rejected": 0,
|
||||
"conflicts": 0,
|
||||
}
|
||||
assert _scores(factory) == {a: 0.9123, b: 0.0100}
|
||||
# The imported rows are evidence, not judgements: no decision is invented.
|
||||
with factory() as session:
|
||||
rows = list(session.scalars(select(SafetyReview)))
|
||||
assert {r.decision for r in rows} == {None}
|
||||
assert {r.reviewer for r in rows} == {REVIEWER}
|
||||
assert csv_path.exists(), "the CSV is left on disk as its own backup"
|
||||
|
||||
|
||||
def test_a_reviewed_asset_is_never_overwritten_by_the_csv(tmp_path, factory):
|
||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
||||
_review(factory, asset_id, score=0.2, decision="sfw")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.9999\n")
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path, overwrite=True)
|
||||
|
||||
assert (report.imported, report.reviewed) == (0, 1)
|
||||
assert report.conflicts[0]["decision"] == "sfw"
|
||||
assert report.conflicts[0]["csv"] == 0.9999
|
||||
with factory() as session:
|
||||
rows = list(session.scalars(select(SafetyReview)))
|
||||
assert len(rows) == 1 and rows[0].decision == "sfw"
|
||||
|
||||
|
||||
def test_an_unknown_path_is_reported_never_turned_into_an_asset(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(
|
||||
tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n{tmp_path / 'ghost.jpg'},0.5000\n"
|
||||
)
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
assert report.unmatched == [str(tmp_path / "ghost.jpg")]
|
||||
with factory() as session:
|
||||
assert session.scalar(select(Asset).where(Asset.current_path.like("%ghost%"))) is None
|
||||
|
||||
|
||||
def test_malformed_rows_are_rejected_with_their_reason(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(
|
||||
tmp_path,
|
||||
f"{tmp_path / 'a.jpg'},0.5000\n"
|
||||
f"{tmp_path / 'a.jpg'},0.5000\n" # the same path twice
|
||||
f"{tmp_path / 'b.jpg'},not-a-number\n"
|
||||
f"{tmp_path / 'c.jpg'},7.5\n" # outside 0..1: corruption, not a score
|
||||
",0.5\n", # no path at all
|
||||
)
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
reasons = sorted(r["reason"] for r in report.rejected)
|
||||
assert reasons == ["duplicate_path", "missing_path", "unparsable_score", "unparsable_score"]
|
||||
assert report.imported == 1
|
||||
|
||||
|
||||
def test_rerunning_changes_nothing_and_a_differing_score_needs_overwrite(tmp_path, factory):
|
||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
||||
service = LegacyImportService(factory)
|
||||
service.import_nsfw_scores(csv_path)
|
||||
|
||||
again = service.import_nsfw_scores(csv_path)
|
||||
assert (again.imported, again.unchanged) == (0, 1)
|
||||
assert len(_scores(factory)) == 1
|
||||
|
||||
changed = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.8000\n")
|
||||
refused = service.import_nsfw_scores(changed)
|
||||
assert (refused.imported, refused.skipped_existing) == (0, 1)
|
||||
assert refused.conflicts[0]["current"] == 0.5
|
||||
|
||||
forced = service.import_nsfw_scores(changed, overwrite=True)
|
||||
assert forced.imported == 1
|
||||
with factory() as session:
|
||||
latest = list(session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)))[-1]
|
||||
assert (latest.asset_id, latest.score) == (asset_id, 0.8)
|
||||
|
||||
|
||||
def test_a_dry_run_reports_without_writing(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path, dry_run=True)
|
||||
|
||||
assert report.imported == 1
|
||||
assert _scores(factory) == {}, "a dry run must leave the database alone"
|
||||
|
||||
|
||||
def test_a_missing_csv_is_a_reported_outcome_not_a_crash(tmp_path, factory):
|
||||
report = LegacyImportService(factory).import_nsfw_scores(tmp_path / "nothing.csv")
|
||||
assert report.rejected == [{"path": str(tmp_path / "nothing.csv"), "reason": "csv_missing"}]
|
||||
assert report.rows == 0
|
||||
|
||||
|
||||
def test_the_report_is_written_where_it_can_be_audited(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
written = write_report(report, tmp_path / "data")
|
||||
|
||||
payload = json.loads(written.read_text(encoding="utf-8"))
|
||||
assert payload["imported"] == 1
|
||||
assert payload["source"] == str(csv_path)
|
||||
assert payload["schema_version"] == 1
|
||||
assert payload["generated_at"]
|
||||
|
||||
|
||||
def test_a_later_review_still_wins_after_an_import(tmp_path, factory):
|
||||
"""The import is a floor, not a ceiling: a human decision made afterwards is
|
||||
the current state, exactly as everywhere else in the app."""
|
||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.9000\n")
|
||||
LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
# The import stamps itself with the wall clock, so "afterwards" has to be too.
|
||||
_review(
|
||||
factory,
|
||||
asset_id,
|
||||
score=0.9,
|
||||
decision="sfw",
|
||||
at=datetime.now(timezone.utc) + timedelta(hours=1),
|
||||
)
|
||||
|
||||
with factory() as session:
|
||||
latest = list(session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)))[-1]
|
||||
assert latest.decision == "sfw"
|
||||
@@ -3,8 +3,6 @@ same fixture files (real exiftool-written keywords)."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
@@ -13,10 +11,6 @@ from PIL import Image
|
||||
from photo_pipeline.integrations import exiftool
|
||||
from photo_pipeline.services import safety
|
||||
|
||||
# The donor is frozen in the read-only archive (US07-01); parity still compares
|
||||
# against it, so this suite is the second and last place that may import it.
|
||||
ARCHIVED_SOURCES = Path(__file__).resolve().parents[2] / "legacy_cli_archive" / "src"
|
||||
|
||||
EXIFTOOL = shutil.which("exiftool")
|
||||
pytestmark = pytest.mark.skipif(EXIFTOOL is None, reason="exiftool not installed")
|
||||
|
||||
@@ -37,7 +31,6 @@ def _tag(path, keyword):
|
||||
|
||||
|
||||
def test_extracted_marks_and_partition_match_donor(tmp_path):
|
||||
sys.path.insert(0, str(ARCHIVED_SOURCES)) # the donor lives in the archive now
|
||||
donor_exif = pytest.importorskip("nsfwtag.exif")
|
||||
|
||||
nsfw = _jpeg(tmp_path / "nsfw.jpg", 1)
|
||||
|
||||
@@ -137,16 +137,6 @@
|
||||
],
|
||||
"US06-04": [
|
||||
"tests/integration/test_restore.py"
|
||||
],
|
||||
"US06-05": [
|
||||
"tests/e2e/test_archive_ui.py"
|
||||
],
|
||||
"US06-06": [
|
||||
"tests/e2e/test_phase_f_pipeline.py"
|
||||
],
|
||||
"US07-01": [
|
||||
"tests/unit/test_legacy_archive.py",
|
||||
"tests/integration/test_legacy_import.py"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,173 +0,0 @@
|
||||
"""Archive lint (US07-01): the frozen CLI archive is complete, honest, and inert.
|
||||
|
||||
Freezing the donors is only worth something if three things stay true: the archive
|
||||
still holds exactly the bytes it claims, it holds no secret, and production cannot
|
||||
reach it. Each of those is a one-line mistake away — a helpful `sys.path` insert, a
|
||||
copied `.env`, an edited "just this once" source — so each is asserted here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
ARCHIVE = REPO / "legacy_cli_archive"
|
||||
SOURCES = ARCHIVE / "src"
|
||||
LEDGER = ARCHIVE / "donor_ledger.yaml"
|
||||
CHECKSUMS = ARCHIVE / "CHECKSUMS.sha256"
|
||||
|
||||
# Everything the concept requires an archive to carry (§3 "Donor-first CLI
|
||||
# migration and archival"): the sources, their docs, a dependency lock, schema
|
||||
# notes, a redacted sample configuration, the ledger, and recorded checksums.
|
||||
REQUIRED_ARTIFACTS = (
|
||||
"README.md",
|
||||
"donor_ledger.yaml",
|
||||
"CHECKSUMS.sha256",
|
||||
"requirements-lock.txt",
|
||||
"photo_analyzer.env.sample",
|
||||
)
|
||||
ARCHIVED_MODULES = ("photo_analyzer", "nsfwtag", "webapp", "nsfw_tag", "compare_models")
|
||||
# Only the two suites that compare against the donors may put the archive on the
|
||||
# import path; every other tree must not name it at all.
|
||||
IMPORT_ALLOWED = {
|
||||
REPO / "tests" / "characterization" / "conftest.py",
|
||||
REPO / "tests" / "characterization" / "test_donor_ledger.py", # lints the ledger there
|
||||
REPO / "tests" / "integration" / "test_safety_parity.py",
|
||||
Path(__file__),
|
||||
}
|
||||
SECRET_PATTERNS = (
|
||||
re.compile(r"sk-(?!REPLACE_WITH_YOUR_KEY)[A-Za-z0-9_\-]{16,}"),
|
||||
re.compile(r"AIza[0-9A-Za-z_\-]{20,}"), # Google API keys
|
||||
re.compile(r"(?i)api[_-]?key\s*[=:]\s*['\"][A-Za-z0-9_\-]{16,}['\"]"),
|
||||
)
|
||||
|
||||
|
||||
def _archived_files() -> list[Path]:
|
||||
return sorted(p for p in SOURCES.rglob("*") if p.is_file() and "__pycache__" not in p.parts)
|
||||
|
||||
|
||||
# ── completeness ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_archive_carries_every_required_artifact():
|
||||
for artifact in REQUIRED_ARTIFACTS:
|
||||
path = ARCHIVE / artifact
|
||||
assert path.is_file(), f"archive is missing {artifact}"
|
||||
assert path.stat().st_size > 0, f"archive artifact {artifact} is empty"
|
||||
assert (SOURCES / "photo_analyzer.py").is_file()
|
||||
assert (SOURCES / "nsfwtag" / "README.md").is_file(), "donor docs must be archived too"
|
||||
assert (SOURCES / "webapp" / "README.md").is_file()
|
||||
|
||||
|
||||
def test_every_ledger_source_is_present_in_the_archive():
|
||||
rows = yaml.safe_load(LEDGER.read_text(encoding="utf-8"))["rows"]
|
||||
for row in rows:
|
||||
source = SOURCES / row["source"]["file"]
|
||||
assert source.is_file(), f"{row['id']}: {row['source']['file']} is not archived"
|
||||
|
||||
|
||||
def test_the_readme_records_provenance_and_the_no_import_rule():
|
||||
readme = (ARCHIVE / "README.md").read_text(encoding="utf-8")
|
||||
for expected in ("CHECKSUMS.sha256", "requirements-lock.txt", "donor_ledger.yaml"):
|
||||
assert expected in readme, f"README does not point at {expected}"
|
||||
assert "Schema notes" in readme, "the donor's schema must be documented"
|
||||
assert "nsfw_scores.csv" in readme, "the CSV's fate must be documented"
|
||||
|
||||
|
||||
# ── integrity ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_every_archived_source_matches_its_checksum():
|
||||
"""A frozen archive that silently drifts is not evidence of anything."""
|
||||
recorded = {}
|
||||
for line in CHECKSUMS.read_text(encoding="utf-8").splitlines():
|
||||
digest, _, rel = line.partition(" ")
|
||||
if rel:
|
||||
recorded[rel.strip()] = digest
|
||||
actual = {
|
||||
str(path.relative_to(ARCHIVE)): hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
for path in _archived_files()
|
||||
}
|
||||
assert actual == recorded, "archived sources and CHECKSUMS.sha256 disagree"
|
||||
|
||||
|
||||
# ── redaction ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_archive_contains_no_credential():
|
||||
for path in [*_archived_files(), *(ARCHIVE / a for a in REQUIRED_ARTIFACTS)]:
|
||||
if path.suffix in (".png", ".jpg", ".webp"):
|
||||
continue
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
for pattern in SECRET_PATTERNS:
|
||||
assert not pattern.search(text), f"possible secret in {path.relative_to(REPO)}"
|
||||
|
||||
|
||||
def test_the_sample_configuration_is_a_placeholder_only():
|
||||
sample = (ARCHIVE / "photo_analyzer.env.sample").read_text(encoding="utf-8")
|
||||
assert "REDACTED" in sample
|
||||
# Placeholders are not key-shaped, so neither a scanner nor a reader can
|
||||
# mistake the sample for a credential.
|
||||
assert "LLM_API_KEY=<" in sample
|
||||
# No real env file, database, log, or CSV may ride along in the archive.
|
||||
strays = [
|
||||
p.name
|
||||
for p in ARCHIVE.rglob("*")
|
||||
if p.is_file() and p.suffix in (".env", ".db", ".sqlite", ".sqlite3", ".log", ".csv")
|
||||
]
|
||||
assert strays == [], f"unexpected runtime files archived: {strays}"
|
||||
|
||||
|
||||
# ── inertness ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_production_code_never_imports_an_archived_module():
|
||||
for path in (REPO / "photo_pipeline").rglob("*.py"):
|
||||
text = path.read_text(encoding="utf-8")
|
||||
for module in ARCHIVED_MODULES:
|
||||
assert not re.search(rf"^\s*(import|from)\s+{module}\b", text, re.MULTILINE), \
|
||||
f"{path.relative_to(REPO)} imports the archived {module}"
|
||||
assert "legacy_cli_archive" not in text, \
|
||||
f"{path.relative_to(REPO)} names the archive"
|
||||
|
||||
|
||||
def test_only_the_parity_suites_put_the_archive_on_the_import_path():
|
||||
for path in (REPO / "tests").rglob("*.py"):
|
||||
if path in IMPORT_ALLOWED or "__pycache__" in path.parts:
|
||||
continue
|
||||
assert "legacy_cli_archive" not in path.read_text(encoding="utf-8"), \
|
||||
f"{path.relative_to(REPO)} reaches into the archive"
|
||||
|
||||
|
||||
def test_the_archived_modules_are_unimportable_from_a_clean_interpreter():
|
||||
"""The real check: a fresh process with the repo on its path cannot load them."""
|
||||
script = (
|
||||
"import importlib.util, sys; "
|
||||
f"sys.path.insert(0, {str(REPO)!r}); "
|
||||
"print([m for m in "
|
||||
f"{list(ARCHIVED_MODULES)!r}"
|
||||
" if importlib.util.find_spec(m) is not None])"
|
||||
)
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", script], capture_output=True, text=True, cwd=str(REPO), check=True
|
||||
)
|
||||
assert result.stdout.strip() == "[]", f"still importable: {result.stdout.strip()}"
|
||||
|
||||
|
||||
def test_the_application_starts_without_the_archive(tmp_path):
|
||||
"""Nothing in the runtime path may need the frozen sources to exist."""
|
||||
pytest.importorskip("fastapi")
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
|
||||
config = Config.from_env(
|
||||
{"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"), "PHOTO_PIPELINE_LIBRARY_ROOTS": ""}
|
||||
)
|
||||
assert create_app(config) is not None
|
||||
Reference in New Issue
Block a user