Compare commits
1 Commits
us/US07-02
...
us/US06-02
| Author | SHA1 | Date | |
|---|---|---|---|
| 9503fd1cfc |
@@ -80,8 +80,7 @@ are the authoritative backlog.
|
|||||||
uncertain. Work only on the claimed story and its generated feature branch.
|
uncertain. Work only on the claimed story and its generated feature branch.
|
||||||
4. Read the entire issue, linked specification, dependencies, and acceptance criteria.
|
4. Read the entire issue, linked specification, dependencies, and acceptance criteria.
|
||||||
Reconcile them with the concept before designing or changing code.
|
Reconcile them with the concept before designing or changing code.
|
||||||
5. Inspect the legacy CLI donors — frozen in `legacy_cli_archive/` since US07-01,
|
5. Inspect the legacy CLI donors before replacing applicable behavior. Update the donor
|
||||||
with their ledger — before replacing applicable behavior. Update the donor
|
|
||||||
ledger and characterization tests required by the story.
|
ledger and characterization tests required by the story.
|
||||||
6. Implement every acceptance criterion and its automated tests.
|
6. Implement every acceptance criterion and its automated tests.
|
||||||
7. Run story-specific tests and the accumulated regression suite required by the epic.
|
7. Run story-specific tests and the accumulated regression suite required by the epic.
|
||||||
|
|||||||
91
README.md
91
README.md
@@ -17,25 +17,6 @@ python -m photo_pipeline serve # start the API + static review UI (127.0.0.1
|
|||||||
Configuration comes from `PHOTO_PIPELINE_*` environment variables (see
|
Configuration comes from `PHOTO_PIPELINE_*` environment variables (see
|
||||||
`photo_pipeline/config.py`); secrets are referenced, never logged.
|
`photo_pipeline/config.py`); secrets are referenced, never logged.
|
||||||
|
|
||||||
### API access (US07-02)
|
|
||||||
|
|
||||||
The app listens on loopback, so its attacker is another page in the same browser.
|
|
||||||
Every `/api/v1` route except `health/live`, `health/ready`, and `session` requires
|
|
||||||
the application session, and every mutation requires its CSRF token as well:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
BASE=http://127.0.0.1:8000
|
|
||||||
TOKEN=$(curl -sc /tmp/pp.jar $BASE/api/v1/session | python -c 'import json,sys; print(json.load(sys.stdin)["csrf_token"])')
|
|
||||||
curl -sb /tmp/pp.jar -H "X-CSRF-Token: $TOKEN" -X POST $BASE/api/v1/albums/proposals -d '{}' -H 'Content-Type: application/json'
|
|
||||||
```
|
|
||||||
|
|
||||||
The session is per server process — restarting `serve` invalidates it, and the
|
|
||||||
browser client re-bootstraps by itself. Requests are also refused when the `Host` is
|
|
||||||
not a loopback name (DNS rebinding), when `Origin` is any other origin, when
|
|
||||||
`Sec-Fetch-Site` says the request came from another site (an `<img>` pointed at a
|
|
||||||
thumbnail), or when the body exceeds `PHOTO_PIPELINE_MAX_REQUEST_BYTES`. There is no
|
|
||||||
CORS middleware at all, so no other origin can read a response.
|
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
One offline command runs the whole suite (unit, integration, and browser
|
One offline command runs the whole suite (unit, integration, and browser
|
||||||
@@ -188,75 +169,3 @@ work_item/scripts/python -m pytest -m phase_e -q
|
|||||||
bytes, and recovery after a restart).
|
bytes, and recovery after a restart).
|
||||||
|
|
||||||
Phases A–D remain green in the full run above.
|
Phases A–D remain green in the full run above.
|
||||||
|
|
||||||
### Phase F acceptance gate
|
|
||||||
|
|
||||||
Phase F (Epic E06: archive lifecycle) is the only stage that *removes* originals
|
|
||||||
from the library, and the only one whose storage can walk away in someone's bag.
|
|
||||||
One command runs the archive fault-injection suites, the black-box archive and
|
|
||||||
restore API journeys, and the browser suite:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
work_item/scripts/python -m pytest -m phase_f -q
|
|
||||||
```
|
|
||||||
|
|
||||||
- `tests/integration/test_archive_*.py` and `tests/integration/test_restore.py`
|
|
||||||
drive real files on real filesystems: preflight against a mounted, missing,
|
|
||||||
swapped, read-only, or full medium; copy-verify-remove and the same-filesystem
|
|
||||||
move path; and a crash at **every** persisted journal transition in both transfer
|
|
||||||
modes, asserting that no source is ever removed without a durable, byte-identical
|
|
||||||
archive copy.
|
|
||||||
- `tests/e2e/test_phase_f_pipeline.py` drives a real server and a real durable
|
|
||||||
worker over HTTP: preflight blockers (offline medium, wrong volume, insufficient
|
|
||||||
capacity, bytes changed after upload), a verified archive whose manifest, hashes,
|
|
||||||
and path history are checked on the medium itself, a worker killed at each of
|
|
||||||
`transferring`, `verified`, `removing`, `source_removed`, and `complete`, the
|
|
||||||
evidence-based recovery that follows, offline deduplication of an exact and a
|
|
||||||
fuzzy copy while the medium is away, mount return, restore, and a collision that
|
|
||||||
restores beside its occupant.
|
|
||||||
- **Archived is not missing.** An unmounted medium leaves its photos
|
|
||||||
`archived_offline` — still hashed, still in the duplicate indexes, still
|
|
||||||
previewable through their protected thumbnails — and a rescan neither prunes nor
|
|
||||||
flags them.
|
|
||||||
- **Ambiguity is never guessed.** A journal state the medium contradicts stays
|
|
||||||
`manual`, offers no automatic action, and keeps blocking further archiving until
|
|
||||||
a human decides.
|
|
||||||
- `tests/e2e/test_archive_ui.py` covers the browser journeys (preview with
|
|
||||||
destination identity and reclaimable bytes, blockers and mount instructions,
|
|
||||||
progress split into transfer/verification/removal, interruption and recovery,
|
|
||||||
offline browsing, restore, collision, keyboard confirmation, and reload).
|
|
||||||
|
|
||||||
Phases A–E remain green in the full run above.
|
|
||||||
|
|
||||||
## Legacy CLI archive
|
|
||||||
|
|
||||||
The command-line tools this application was extracted from are frozen in
|
|
||||||
`legacy_cli_archive/` (US07-01): the original sources, their docs, the dependency
|
|
||||||
lock they were last verified against, schema notes, a redacted sample
|
|
||||||
configuration, the donor ledger, and a checksum for every file.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256 # verify the archive
|
|
||||||
work_item/scripts/python -m pytest tests/unit/test_legacy_archive.py -q # lint it
|
|
||||||
```
|
|
||||||
|
|
||||||
They are reference material and rollback evidence only. No module under
|
|
||||||
`photo_pipeline/` imports or executes them, the archive is not on the application's
|
|
||||||
import path, and `tests/unit/test_legacy_archive.py` enforces that along with the
|
|
||||||
checksums and the redaction. Only the two suites that compare *against* the donors —
|
|
||||||
`tests/characterization/` and `tests/integration/test_safety_parity.py` — put the
|
|
||||||
archived sources on `sys.path`.
|
|
||||||
|
|
||||||
The last path-keyed state they owned, `nsfw_scores.csv`, is imported once and then
|
|
||||||
left alone:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
work_item/scripts/python -m photo_pipeline import-legacy-scores /path/to/nsfw_scores.csv --dry-run
|
|
||||||
```
|
|
||||||
|
|
||||||
The import writes scored-but-unreviewed `safety_reviews` rows onto stable asset ids,
|
|
||||||
never invents an asset for an unknown path, never overwrites a human decision, and
|
|
||||||
writes a reconciliation report to the data directory saying exactly what it did.
|
|
||||||
`legacy_cli_archive/donor_ledger.yaml` records every migrated behavior with its
|
|
||||||
target, the tests that pin the donor, the tests that prove the replacement, and each
|
|
||||||
intentional delta; rows still marked `pending` name the story that will resolve them.
|
|
||||||
|
|||||||
@@ -12,15 +12,7 @@
|
|||||||
# (if anything) carries over
|
# (if anything) carries over
|
||||||
# Every row needs either `tests` (existing test IDs, module::function) or
|
# Every row needs either `tests` (existing test IDs, module::function) or
|
||||||
# `pending_story` (the backlog story that will characterize/deliver it).
|
# `pending_story` (the backlog story that will characterize/deliver it).
|
||||||
# status: characterized — donor behavior pinned by characterization tests
|
# status: characterized | pending
|
||||||
# resolved — replacement shipped; `parity` names the tests that prove
|
|
||||||
# it, and `delta` states every intentional difference
|
|
||||||
# pending — not migrated yet; `pending_story` says which story will
|
|
||||||
# parity: test ids (path::function) in any suite, proving the replacement
|
|
||||||
# delta: what the replacement deliberately does differently, or not at all
|
|
||||||
#
|
|
||||||
# Archived by US07-01: the sources referenced below now live beside this file in
|
|
||||||
# src/ and are frozen (see README.md). Nothing in photo_pipeline imports them.
|
|
||||||
|
|
||||||
rows:
|
rows:
|
||||||
# ── photo_analyzer.py ──────────────────────────────────────────────────────
|
# ── photo_analyzer.py ──────────────────────────────────────────────────────
|
||||||
@@ -88,12 +80,8 @@ rows:
|
|||||||
Copying a primary's analysis into variant rows survives, but keyed by
|
Copying a primary's analysis into variant rows survives, but keyed by
|
||||||
asset_id and recorded as stage state instead of raw row copies.
|
asset_id and recorded as stage state instead of raw row copies.
|
||||||
target: photo_pipeline/services/duplicates.py
|
target: photo_pipeline/services/duplicates.py
|
||||||
parity:
|
pending_story: US01-04
|
||||||
- tests/integration/test_duplicate_engine.py::test_exact_copies_form_auto_decided_cluster
|
status: pending
|
||||||
- tests/integration/test_duplicate_engine.py::test_perceptual_variant_is_review_only
|
|
||||||
delta: >
|
|
||||||
The donor propagated variant links implicitly while writing rows; the replacement makes the canonical link a reviewable, reversible cluster decision, so a propagated link can always be undone.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: pa-hashing
|
- id: pa-hashing
|
||||||
area: hashing
|
area: hashing
|
||||||
@@ -153,12 +141,8 @@ rows:
|
|||||||
classification: replace
|
classification: replace
|
||||||
rationale: Console report; superseded by the duplicate-review API/UI (US01-06).
|
rationale: Console report; superseded by the duplicate-review API/UI (US01-06).
|
||||||
target: photo_pipeline/api/routes + frontend duplicate review
|
target: photo_pipeline/api/routes + frontend duplicate review
|
||||||
parity:
|
pending_story: US01-06
|
||||||
- tests/integration/test_review_api.py::test_clusters_list_and_detail
|
status: pending
|
||||||
- tests/e2e/test_review_ui.py::test_fuzzy_decision_requires_confirmation
|
|
||||||
delta: >
|
|
||||||
The text listing became the paged cluster API and the comparison UI; no textual report is produced.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: pa-reconcile
|
- id: pa-reconcile
|
||||||
area: database
|
area: database
|
||||||
@@ -218,7 +202,7 @@ rows:
|
|||||||
RGB-normalize (drops alpha, converts HEIC), LANCZOS resize to 2048px
|
RGB-normalize (drops alpha, converts HEIC), LANCZOS resize to 2048px
|
||||||
long-edge, JPEG q85 base64 — the provider-input contract. Truncated-image
|
long-edge, JPEG q85 base64 — the provider-input contract. Truncated-image
|
||||||
tolerance (ImageFile.LOAD_TRUNCATED_IMAGES) carries with it.
|
tolerance (ImageFile.LOAD_TRUNCATED_IMAGES) carries with it.
|
||||||
target: photo_pipeline/services/analysis.py
|
target: photo_pipeline/integrations/vision.py
|
||||||
tests:
|
tests:
|
||||||
- test_pa_imaging::test_prepare_image_small_passthrough_jpeg
|
- test_pa_imaging::test_prepare_image_small_passthrough_jpeg
|
||||||
- test_pa_imaging::test_prepare_image_resizes_to_max_long_edge
|
- test_pa_imaging::test_prepare_image_resizes_to_max_long_edge
|
||||||
@@ -246,12 +230,9 @@ rows:
|
|||||||
response validation, 429/503 retry with exponential backoff. Prompt and
|
response validation, 429/503 retry with exponential backoff. Prompt and
|
||||||
model/config version must be persisted per analysis_runs. Characterized
|
model/config version must be persisted per analysis_runs. Characterized
|
||||||
against a deterministic fake provider when the analysis service is ported.
|
against a deterministic fake provider when the analysis service is ported.
|
||||||
target: photo_pipeline/services/analysis.py
|
target: photo_pipeline/integrations/vision.py
|
||||||
parity:
|
pending_story: US02-06
|
||||||
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
status: pending
|
||||||
delta: >
|
|
||||||
The prompt and response schema carry over; the provider is an injected adapter so the privacy gate is testable, and results are keyed to asset ids rather than paths.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: pa-throttle
|
- id: pa-throttle
|
||||||
area: logging
|
area: logging
|
||||||
@@ -260,8 +241,8 @@ rows:
|
|||||||
rationale: >
|
rationale: >
|
||||||
Rolling throttle window + persistent throttle_events.jsonl + RPD day
|
Rolling throttle window + persistent throttle_events.jsonl + RPD day
|
||||||
counter become job metrics/events on the durable job model.
|
counter become job metrics/events on the durable job model.
|
||||||
target: photo_pipeline/services/jobs.py + photo_pipeline/services/analysis.py
|
target: photo_pipeline/jobs/coordinator.py
|
||||||
pending_story: US07-04
|
pending_story: US02-02
|
||||||
status: pending
|
status: pending
|
||||||
|
|
||||||
- id: pa-nsfw-filter
|
- id: pa-nsfw-filter
|
||||||
@@ -331,12 +312,8 @@ rows:
|
|||||||
checked between items, double-SIGINT force quit — becomes the durable
|
checked between items, double-SIGINT force quit — becomes the durable
|
||||||
JobRunner worker loop with the same drain-and-resume semantics.
|
JobRunner worker loop with the same drain-and-resume semantics.
|
||||||
target: photo_pipeline/jobs/worker.py
|
target: photo_pipeline/jobs/worker.py
|
||||||
parity:
|
pending_story: US02-02
|
||||||
- tests/integration/test_worker.py::test_worker_processes_all_items
|
status: pending
|
||||||
- tests/integration/test_worker.py::test_cooperative_cancellation_leaves_items_resumable
|
|
||||||
delta: >
|
|
||||||
The in-process folder loop with SIGINT handling became durable jobs claimed by a worker: cancellation is a persisted request, not a signal, and an interrupted run resumes from the database instead of restarting.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: pa-ui-terminal
|
- id: pa-ui-terminal
|
||||||
area: ui
|
area: ui
|
||||||
@@ -394,12 +371,8 @@ rows:
|
|||||||
the JSONL history logger become structured JSON logging with job_id/
|
the JSONL history logger become structured JSON logging with job_id/
|
||||||
asset_id and job_events rows; per-photo history maps to job events.
|
asset_id and job_events rows; per-photo history maps to job events.
|
||||||
target: photo_pipeline structured logging + jobs/job_events
|
target: photo_pipeline structured logging + jobs/job_events
|
||||||
parity:
|
pending_story: US02-02
|
||||||
- tests/integration/test_jobs.py::test_enqueue_persists_items_and_event
|
status: pending
|
||||||
- tests/integration/test_jobs_sse.py::test_sse_streams_all_events_then_closes
|
|
||||||
delta: >
|
|
||||||
The JSONL history file and rich console handler are replaced by structured JSON logs plus durable job_events; the browser reads events over SSE rather than tailing a file.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: pa-balance
|
- id: pa-balance
|
||||||
area: vision
|
area: vision
|
||||||
@@ -409,11 +382,8 @@ rows:
|
|||||||
Provider balance/quota probes (report 'unsupported' on providers without
|
Provider balance/quota probes (report 'unsupported' on providers without
|
||||||
the endpoint). Network-bound; characterized against the fake provider.
|
the endpoint). Network-bound; characterized against the fake provider.
|
||||||
target: photo_pipeline/services/analysis.py
|
target: photo_pipeline/services/analysis.py
|
||||||
parity:
|
pending_story: US02-06
|
||||||
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
status: pending
|
||||||
delta: >
|
|
||||||
Not carried over: balance/quota polling was provider-specific (Gemini/OpenAI billing endpoints) and key-scoped. Cost reporting, when a story asks for it, comes from the per-run usage recorded with each analysis result rather than from a vendor endpoint.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: pa-cli
|
- id: pa-cli
|
||||||
area: configuration
|
area: configuration
|
||||||
@@ -423,13 +393,9 @@ rows:
|
|||||||
argparse surface is superseded by the API; flags map to job configs
|
argparse surface is superseded by the API; flags map to job configs
|
||||||
(documented in WEBAPP_CONCEPT.md §8 parity table). Transitional CLI calls
|
(documented in WEBAPP_CONCEPT.md §8 parity table). Transitional CLI calls
|
||||||
the shared services until archival (E07).
|
the shared services until archival (E07).
|
||||||
target: photo_pipeline/__main__.py (serve | worker | migrate | import-legacy-scores) + /api/v1
|
target: photo_pipeline/api + transitional CLI
|
||||||
parity:
|
pending_story: US07-01
|
||||||
- tests/integration/test_app_lifecycle.py::test_restart_preserves_data_and_reruns_migrations
|
status: pending
|
||||||
- tests/unit/test_legacy_archive.py::test_production_code_never_imports_an_archived_module
|
|
||||||
delta: >
|
|
||||||
The argparse surface is not reproduced. Every flag that drove work became an API command or a job configuration; the CLI keeps only what an application needs to be operated (serve, worker, migrate) plus the one-off legacy CSV import.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
# ── nsfwtag/ ───────────────────────────────────────────────────────────────
|
# ── nsfwtag/ ───────────────────────────────────────────────────────────────
|
||||||
- id: nt-discovery
|
- id: nt-discovery
|
||||||
@@ -456,17 +422,9 @@ rows:
|
|||||||
nsfw_scores.csv stops being the source of truth (concept: DB state).
|
nsfw_scores.csv stops being the source of truth (concept: DB state).
|
||||||
Format characterized (4-decimal scores, bad rows dropped) because the
|
Format characterized (4-decimal scores, bad rows dropped) because the
|
||||||
existing CSV must migrate into assets.safety_score.
|
existing CSV must migrate into assets.safety_score.
|
||||||
target: photo_pipeline/services/legacy_import.py (one-off import into safety_reviews)
|
target: photo_pipeline/repositories (safety), CSV import in US01-02 migration
|
||||||
tests: [test_nsfwtag::test_score_cache_roundtrip_and_tolerance]
|
tests: [test_nsfwtag::test_score_cache_roundtrip_and_tolerance]
|
||||||
parity:
|
status: characterized
|
||||||
- tests/integration/test_legacy_import.py::test_scores_are_imported_onto_asset_identity
|
|
||||||
- tests/integration/test_legacy_import.py::test_a_reviewed_asset_is_never_overwritten_by_the_csv
|
|
||||||
delta: >
|
|
||||||
The CSV is no longer read at runtime at all: it is imported once into
|
|
||||||
scored-but-unreviewed safety_reviews rows and left on disk untouched. A path
|
|
||||||
that matches nothing is reported, never turned into an asset, and a human
|
|
||||||
decision always outranks an imported score.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: nt-score-model
|
- id: nt-score-model
|
||||||
area: nsfw
|
area: nsfw
|
||||||
@@ -527,11 +485,8 @@ rows:
|
|||||||
Newline-list bulk tagging (nsfw_confirmed.txt flow) is superseded by DB
|
Newline-list bulk tagging (nsfw_confirmed.txt flow) is superseded by DB
|
||||||
review decisions; the existing list is a one-time migration input.
|
review decisions; the existing list is a one-time migration input.
|
||||||
target: photo_pipeline/services/safety.py (decision import in US01-02)
|
target: photo_pipeline/services/safety.py (decision import in US01-02)
|
||||||
parity:
|
pending_story: US01-02
|
||||||
- tests/integration/test_safety_parity.py::test_extracted_marks_and_partition_match_donor
|
status: pending
|
||||||
delta: >
|
|
||||||
Bulk keyword application from a file list is replaced by decisions against asset ids; the EXIF write itself is the extracted, read-back-verified one.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: nt-ui
|
- id: nt-ui
|
||||||
area: ui
|
area: ui
|
||||||
@@ -543,11 +498,8 @@ rows:
|
|||||||
threshold/score review flow, lightbox and keyboard model are the frontend
|
threshold/score review flow, lightbox and keyboard model are the frontend
|
||||||
donor for the Safety view (preserved per concept §10; ported in US02-01).
|
donor for the Safety view (preserved per concept §10; ported in US02-01).
|
||||||
target: photo_pipeline/api + frontend Safety view
|
target: photo_pipeline/api + frontend Safety view
|
||||||
parity:
|
pending_story: US02-01
|
||||||
- tests/e2e/test_workflow_views.py::test_safety_review_decide_persists_across_reload
|
status: pending
|
||||||
delta: >
|
|
||||||
The stdlib review server is replaced by the API plus the Safety view; the donor's layout, thresholds, and keyboard flow carry over, its embedded HTML generation does not.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: nt-bench
|
- id: nt-bench
|
||||||
area: nsfw
|
area: nsfw
|
||||||
@@ -557,11 +509,8 @@ rows:
|
|||||||
Dev-only model benchmark; archived without webapp replacement (recorded
|
Dev-only model benchmark; archived without webapp replacement (recorded
|
||||||
basis of the AdamCodd model choice). No production caller.
|
basis of the AdamCodd model choice). No production caller.
|
||||||
target: none (archive as reference)
|
target: none (archive as reference)
|
||||||
parity:
|
pending_story: US07-01
|
||||||
- tests/unit/test_legacy_archive.py::test_every_archived_source_matches_its_checksum
|
status: pending
|
||||||
delta: >
|
|
||||||
No replacement: a dev-only benchmark whose result (the AdamCodd model choice) is already recorded. Kept in the archive as the basis of that choice.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
# ── webapp/ ────────────────────────────────────────────────────────────────
|
# ── webapp/ ────────────────────────────────────────────────────────────────
|
||||||
- id: wa-query-search
|
- id: wa-query-search
|
||||||
@@ -630,13 +579,9 @@ rows:
|
|||||||
Subprocess-driving-the-CLI job control is superseded by durable DB jobs
|
Subprocess-driving-the-CLI job control is superseded by durable DB jobs
|
||||||
with a worker process. Two ideas carry over: progress derived from DB
|
with a worker process. Two ideas carry over: progress derived from DB
|
||||||
counts (not job-private state) and single-mutating-job enforcement.
|
counts (not job-private state) and single-mutating-job enforcement.
|
||||||
target: photo_pipeline/services/jobs.py + photo_pipeline/jobs/worker.py
|
target: photo_pipeline/jobs/coordinator.py
|
||||||
parity:
|
pending_story: US02-02
|
||||||
- tests/integration/test_jobs.py::test_idempotency_key_returns_same_job
|
status: pending
|
||||||
- tests/integration/test_worker.py::test_handler_failure_fails_the_job
|
|
||||||
delta: >
|
|
||||||
Subprocess supervision of a CLI is replaced by durable jobs in the same process family: there is no subprocess to supervise, and progress is persisted rather than scraped from stdout.
|
|
||||||
status: resolved
|
|
||||||
|
|
||||||
- id: wa-server
|
- id: wa-server
|
||||||
area: ui
|
area: ui
|
||||||
@@ -648,9 +593,5 @@ rows:
|
|||||||
browser. analyzer.html + page.py design (dark OLED tokens, Library/
|
browser. analyzer.html + page.py design (dark OLED tokens, Library/
|
||||||
Analyze/Stats views) is frontend donor material per concept §10.
|
Analyze/Stats views) is frontend donor material per concept §10.
|
||||||
target: photo_pipeline/api/app.py + frontend
|
target: photo_pipeline/api/app.py + frontend
|
||||||
parity:
|
pending_story: US02-05
|
||||||
- tests/e2e/test_frontend_shell.py::test_shell_loads_assets_without_console_or_network_errors
|
status: pending
|
||||||
- tests/e2e/test_frontend_shell.py::test_deep_link_and_reload_restore_view_and_filters
|
|
||||||
delta: >
|
|
||||||
The stdlib HTTP routes become the versioned FastAPI API and the static frontend shell; Python no longer interpolates HTML.
|
|
||||||
status: resolved
|
|
||||||
@@ -19,7 +19,6 @@
|
|||||||
<a href="#/albums" data-nav="albums">Albums</a>
|
<a href="#/albums" data-nav="albums">Albums</a>
|
||||||
<a href="#/renames" data-nav="renames">Renames</a>
|
<a href="#/renames" data-nav="renames">Renames</a>
|
||||||
<a href="#/uploads" data-nav="uploads">Upload</a>
|
<a href="#/uploads" data-nav="uploads">Upload</a>
|
||||||
<a href="#/archive" data-nav="archive">Archive</a>
|
|
||||||
<a href="#/stats" data-nav="stats">Stats</a>
|
<a href="#/stats" data-nav="stats">Stats</a>
|
||||||
</nav>
|
</nav>
|
||||||
</header>
|
</header>
|
||||||
|
|||||||
@@ -2,43 +2,14 @@
|
|||||||
// cancellation. Every method accepts an optional { signal } from cancellable().
|
// cancellation. Every method accepts an optional { signal } from cancellable().
|
||||||
export const BASE = "/api/v1";
|
export const BASE = "/api/v1";
|
||||||
|
|
||||||
// The API refuses every request without the session cookie, and every mutation
|
|
||||||
// without this token echoed back. The token is readable only same-origin, which is
|
|
||||||
// what makes it proof that the caller is this app and not another page.
|
|
||||||
let csrfToken = null;
|
|
||||||
|
|
||||||
async function session() {
|
|
||||||
if (csrfToken === null) {
|
|
||||||
const response = await fetch(BASE + "/session", { credentials: "same-origin" });
|
|
||||||
const body = await response.json().catch(() => null);
|
|
||||||
csrfToken = (body && body.csrf_token) || null;
|
|
||||||
}
|
|
||||||
return csrfToken || "";
|
|
||||||
}
|
|
||||||
|
|
||||||
async function send(path, { signal, ...options }) {
|
|
||||||
return fetch(BASE + path, {
|
|
||||||
credentials: "same-origin",
|
|
||||||
signal,
|
|
||||||
...options,
|
|
||||||
headers: {
|
|
||||||
"Content-Type": "application/json",
|
|
||||||
"X-CSRF-Token": await session(),
|
|
||||||
...(options.headers || {}),
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
async function request(path, { signal, ...options } = {}) {
|
async function request(path, { signal, ...options } = {}) {
|
||||||
let response;
|
let response;
|
||||||
try {
|
try {
|
||||||
response = await send(path, { signal, ...options });
|
response = await fetch(BASE + path, {
|
||||||
// A restarted server issues a new session; re-bootstrap once rather than
|
headers: { "Content-Type": "application/json" },
|
||||||
// stranding an open tab on 401.
|
signal,
|
||||||
if (response.status === 401) {
|
...options,
|
||||||
csrfToken = null;
|
});
|
||||||
response = await send(path, { signal, ...options });
|
|
||||||
}
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
// A caller-cancelled fetch is not a failure; tag it so views can ignore it.
|
// A caller-cancelled fetch is not a failure; tag it so views can ignore it.
|
||||||
if (error.name === "AbortError") {
|
if (error.name === "AbortError") {
|
||||||
@@ -167,31 +138,4 @@ export const api = {
|
|||||||
}),
|
}),
|
||||||
uploadVerifications: (id, opts = {}) =>
|
uploadVerifications: (id, opts = {}) =>
|
||||||
request(`/upload-batches/${encodeURIComponent(id)}/verifications`, opts),
|
request(`/upload-batches/${encodeURIComponent(id)}/verifications`, opts),
|
||||||
|
|
||||||
// ── Archive and restore: destinations, preflight, plans, recovery ────────
|
|
||||||
archiveLocations: (opts = {}) => request("/archive-locations", opts),
|
|
||||||
registerArchiveLocation: (payload, opts = {}) =>
|
|
||||||
request("/archive-locations", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
|
||||||
archivePreflight: (payload, opts = {}) =>
|
|
||||||
request("/archive-preflight", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
|
||||||
createArchivePlan: (payload, opts = {}) =>
|
|
||||||
request("/archive-plans", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
|
||||||
listArchivePlans: (opts = {}) => request("/archive-plans", opts),
|
|
||||||
getArchivePlan: (id, opts = {}) => request(`/archive-plans/${encodeURIComponent(id)}`, opts),
|
|
||||||
applyArchivePlan: (id, opts = {}) =>
|
|
||||||
request(`/archive-plans/${encodeURIComponent(id)}/apply`, { method: "POST", ...opts }),
|
|
||||||
archiveRecovery: (opts = {}) => request("/archive-recovery", opts),
|
|
||||||
resolveArchiveRecovery: (opts = {}) =>
|
|
||||||
request("/archive-recovery/resolve", { method: "POST", ...opts }),
|
|
||||||
restorePreflight: (payload, opts = {}) =>
|
|
||||||
request("/restore-preflight", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
|
||||||
createRestorePlan: (payload, opts = {}) =>
|
|
||||||
request("/restore-plans", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
|
||||||
listRestorePlans: (opts = {}) => request("/restore-plans", opts),
|
|
||||||
getRestorePlan: (id, opts = {}) => request(`/restore-plans/${encodeURIComponent(id)}`, opts),
|
|
||||||
applyRestorePlan: (id, opts = {}) =>
|
|
||||||
request(`/restore-plans/${encodeURIComponent(id)}/apply`, { method: "POST", ...opts }),
|
|
||||||
restoreRecovery: (opts = {}) => request("/restore-recovery", opts),
|
|
||||||
resolveRestoreRecovery: (opts = {}) =>
|
|
||||||
request("/restore-recovery/resolve", { method: "POST", ...opts }),
|
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -1,5 +1,4 @@
|
|||||||
import { api } from "./api.js";
|
import { api } from "./api.js";
|
||||||
import { renderArchive, setArchiveRender } from "./archive.js";
|
|
||||||
import { navigate, onRouteChange, parseHash } from "./router.js";
|
import { navigate, onRouteChange, parseHash } from "./router.js";
|
||||||
import { renderRenames, setRenamesRender } from "./renames.js";
|
import { renderRenames, setRenamesRender } from "./renames.js";
|
||||||
import { renderUploads, setUploadsRender } from "./uploads.js";
|
import { renderUploads, setUploadsRender } from "./uploads.js";
|
||||||
@@ -367,7 +366,6 @@ function render() {
|
|||||||
else if (path === "/albums") renderAlbums(root, params);
|
else if (path === "/albums") renderAlbums(root, params);
|
||||||
else if (path === "/renames") renderRenames(root, params);
|
else if (path === "/renames") renderRenames(root, params);
|
||||||
else if (path === "/uploads") renderUploads(root, params);
|
else if (path === "/uploads") renderUploads(root, params);
|
||||||
else if (path === "/archive") renderArchive(root, params);
|
|
||||||
else if (path === "/stats") renderStats(root, params);
|
else if (path === "/stats") renderStats(root, params);
|
||||||
else show(errorBanner("Unknown view"));
|
else show(errorBanner("Unknown view"));
|
||||||
}
|
}
|
||||||
@@ -376,6 +374,5 @@ function render() {
|
|||||||
setRender(render);
|
setRender(render);
|
||||||
setRenamesRender(render);
|
setRenamesRender(render);
|
||||||
setUploadsRender(render);
|
setUploadsRender(render);
|
||||||
setArchiveRender(render);
|
|
||||||
onRouteChange(render);
|
onRouteChange(render);
|
||||||
render();
|
render();
|
||||||
|
|||||||
@@ -1,870 +0,0 @@
|
|||||||
// Archive view (US06-05): preview what would leave active storage, confirm it
|
|
||||||
// exactly, watch the transfer, recover an interrupted one, browse what is already
|
|
||||||
// archived, and bring it back.
|
|
||||||
//
|
|
||||||
// Archiving is the only stage that removes originals, so this view never decides
|
|
||||||
// anything itself: the destination's identity, every blocker, the confirmation
|
|
||||||
// token, and what recovery may do all come from the server, and an action the
|
|
||||||
// server would refuse is not offered. Two things follow from that. A medium that
|
|
||||||
// is not mounted produces an instruction naming it rather than a disabled mystery,
|
|
||||||
// and an operation whose evidence is ambiguous offers no button at all.
|
|
||||||
import { api } from "./api.js";
|
|
||||||
import { el, errorBanner, setActiveNav } from "./dom.js";
|
|
||||||
import { subscribeJob } from "./events.js";
|
|
||||||
import { navigate } from "./router.js";
|
|
||||||
|
|
||||||
let outcome = null;
|
|
||||||
let activity = [];
|
|
||||||
|
|
||||||
let render = () => {};
|
|
||||||
export function setArchiveRender(fn) {
|
|
||||||
render = fn;
|
|
||||||
}
|
|
||||||
|
|
||||||
// The per-file journal states, split into the three things an operator actually
|
|
||||||
// wants told apart: bytes moving, bytes proven, original removed (concept §9).
|
|
||||||
const PHASES = [
|
|
||||||
["planned", "planned", "waiting"],
|
|
||||||
["transferring", "transfer", "copying to the medium"],
|
|
||||||
["verified", "verified", "archive copy hashed and manifested"],
|
|
||||||
["removing", "removing", "removing the active original"],
|
|
||||||
["complete", "complete", "archived and removed"],
|
|
||||||
["failed", "failed", "left alone for a decision"],
|
|
||||||
];
|
|
||||||
|
|
||||||
const AVAILABILITY_LABEL = {
|
|
||||||
active: "in the library",
|
|
||||||
archived_online: "archived · medium mounted",
|
|
||||||
archived_offline: "archived · medium away",
|
|
||||||
missing_unexpected: "missing — unexplained",
|
|
||||||
};
|
|
||||||
|
|
||||||
export async function renderArchive(root, params = {}) {
|
|
||||||
setActiveNav("archive");
|
|
||||||
|
|
||||||
let locations;
|
|
||||||
try {
|
|
||||||
locations = (await api.archiveLocations()).locations;
|
|
||||||
} catch (error) {
|
|
||||||
root.replaceChildren(errorBanner(`Failed to load archive locations: ${error.message}`));
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const location = locations.find((l) => l.id === params.location) || locations[0] || null;
|
|
||||||
|
|
||||||
const nodes = [el("h1", {}, "Archive"), locationsCard(locations, location, params)];
|
|
||||||
if (!location) {
|
|
||||||
nodes.push(
|
|
||||||
el(
|
|
||||||
"p",
|
|
||||||
{ class: "muted", "data-testid": "no-locations" },
|
|
||||||
"Register the disk, NAS share, or removable medium that will hold archived originals."
|
|
||||||
),
|
|
||||||
outcomeBanner()
|
|
||||||
);
|
|
||||||
root.replaceChildren(...nodes.filter(Boolean));
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
const [preflight, archivePlans, restorePlans, recovery, restoreRecovery, archived, restore] =
|
|
||||||
await Promise.all([
|
|
||||||
load(() => api.archivePreflight({ location_id: location.id })),
|
|
||||||
load(() => api.listArchivePlans()),
|
|
||||||
load(() => api.listRestorePlans()),
|
|
||||||
load(() => api.archiveRecovery()),
|
|
||||||
load(() => api.restoreRecovery()),
|
|
||||||
load(() => api.listAssets({ limit: 200 })),
|
|
||||||
load(() => api.restorePreflight({ location_id: location.id })),
|
|
||||||
]);
|
|
||||||
|
|
||||||
// Both directions share the runs table: one lane moves these originals, so one
|
|
||||||
// history is what an operator has to reason about.
|
|
||||||
const runs = [...plans(archivePlans), ...plans(restorePlans)].sort((a, b) =>
|
|
||||||
(a.created_at || "").localeCompare(b.created_at || "")
|
|
||||||
);
|
|
||||||
const chosen = runs.find((run) => run.id === params.plan) || runs[runs.length - 1] || null;
|
|
||||||
const selectedPlan = chosen ? await load(() => planDetailOf(chosen)) : null;
|
|
||||||
|
|
||||||
nodes.push(
|
|
||||||
preflight ? previewSection(preflight, location) : null,
|
|
||||||
preflight ? confirmBlock(preflight, location) : null,
|
|
||||||
outcomeBanner(),
|
|
||||||
activityLog(),
|
|
||||||
recoverySection(mergeRecovery(recovery, restoreRecovery)),
|
|
||||||
planList(runs, chosen && chosen.id),
|
|
||||||
selectedPlan ? planDetail(selectedPlan) : null,
|
|
||||||
archived ? archivedSection(archived.items, locations) : null,
|
|
||||||
restore ? restoreSection(restore, location) : null
|
|
||||||
);
|
|
||||||
root.replaceChildren(...nodes.filter(Boolean));
|
|
||||||
}
|
|
||||||
|
|
||||||
function plans(listed) {
|
|
||||||
return listed ? listed.plans : [];
|
|
||||||
}
|
|
||||||
|
|
||||||
function planDetailOf(run) {
|
|
||||||
return run.direction === "restore" ? api.getRestorePlan(run.id) : api.getArchivePlan(run.id);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Archive and restore recovery answer the same question about the same lane, so
|
|
||||||
// they are one list; an unresolved item of either kind blocks the other.
|
|
||||||
function mergeRecovery(archive, restore) {
|
|
||||||
if (!archive && !restore) return null;
|
|
||||||
return {
|
|
||||||
operations: [...((archive || {}).operations || []), ...((restore || {}).operations || [])],
|
|
||||||
manual: [...((archive || {}).manual || []), ...((restore || {}).manual || [])],
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
// A section whose data failed to load must not take the rest of the view with it:
|
|
||||||
// the medium being away is exactly when the archived-asset list matters most.
|
|
||||||
async function load(call) {
|
|
||||||
try {
|
|
||||||
return await call();
|
|
||||||
} catch (_) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── locations ────────────────────────────────────────────────────────────────
|
|
||||||
// A location is a medium, not a path: the marker's ``media_id`` is what proves the
|
|
||||||
// right disk is mounted, so it is shown next to the state it produced.
|
|
||||||
function locationsCard(locations, selected, params) {
|
|
||||||
const name = el("input", {
|
|
||||||
type: "text",
|
|
||||||
"data-testid": "location-name",
|
|
||||||
"aria-label": "Archive location name",
|
|
||||||
placeholder: "External disk",
|
|
||||||
});
|
|
||||||
const root = el("input", {
|
|
||||||
type: "text",
|
|
||||||
"data-testid": "location-root",
|
|
||||||
"aria-label": "Archive location path",
|
|
||||||
placeholder: "/Volumes/archive",
|
|
||||||
});
|
|
||||||
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "card", "data-testid": "archive-locations" },
|
|
||||||
el("h2", {}, "Destinations"),
|
|
||||||
locations.length
|
|
||||||
? el(
|
|
||||||
"table",
|
|
||||||
{ class: "grid", "data-testid": "locations" },
|
|
||||||
el(
|
|
||||||
"thead",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{},
|
|
||||||
...["", "Name", "Root", "Medium", "State", "Last seen"].map((label) =>
|
|
||||||
el("th", { scope: "col" }, label)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"tbody",
|
|
||||||
{},
|
|
||||||
...locations.map((location) =>
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{
|
|
||||||
"data-testid": "location-row",
|
|
||||||
"data-name": location.name,
|
|
||||||
"aria-current": selected && location.id === selected.id ? "true" : false,
|
|
||||||
},
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el("input", {
|
|
||||||
type: "radio",
|
|
||||||
name: "archive-location",
|
|
||||||
"data-testid": "select-location",
|
|
||||||
"aria-label": `Use ${location.name}`,
|
|
||||||
checked: selected && location.id === selected.id ? "checked" : false,
|
|
||||||
onchange: () => navigate("/archive", { ...params, location: location.id }),
|
|
||||||
})
|
|
||||||
),
|
|
||||||
el("td", { "data-testid": "location-label" }, location.name),
|
|
||||||
el("td", { class: "path", "data-testid": "location-path" }, location.root),
|
|
||||||
el("td", { class: "path", "data-testid": "location-media" }, location.media_id),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{
|
|
||||||
class: `badge ${location.state === "online" ? "complete" : "attention"}`,
|
|
||||||
"data-testid": "location-state",
|
|
||||||
},
|
|
||||||
location.state
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el("td", { class: "muted" }, location.last_seen_at || "never")
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
: null,
|
|
||||||
selected && selected.state !== "online" ? mountInstruction(selected) : null,
|
|
||||||
el(
|
|
||||||
"div",
|
|
||||||
{ class: "toolbar" },
|
|
||||||
name,
|
|
||||||
root,
|
|
||||||
el(
|
|
||||||
"button",
|
|
||||||
{
|
|
||||||
"data-testid": "register-location",
|
|
||||||
onclick: () =>
|
|
||||||
run(() => api.registerArchiveLocation({ name: name.value, root: root.value })),
|
|
||||||
},
|
|
||||||
"Register destination"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// The one thing the app cannot do for the user: name the medium to connect.
|
|
||||||
function mountInstruction(location) {
|
|
||||||
const detail =
|
|
||||||
location.state === "wrong_volume"
|
|
||||||
? `A different medium is mounted at ${location.root}.`
|
|
||||||
: `Nothing is mounted at ${location.root}.`;
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "confirm", role: "status", "data-testid": "mount-instruction" },
|
|
||||||
`${detail} Connect “${location.name}” (medium ${location.media_id}) and mount it there, ` +
|
|
||||||
"then reload this view. Archived photos stay listed and searchable meanwhile."
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── preview ──────────────────────────────────────────────────────────────────
|
|
||||||
function previewSection(preflight, location) {
|
|
||||||
const totals = preflight.totals;
|
|
||||||
const capacity = preflight.capacity;
|
|
||||||
const rows = preflight.albums.map((album) =>
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{ "data-testid": "archive-album-row", "data-album": album.album },
|
|
||||||
el("td", { "data-testid": "album-name" }, album.album),
|
|
||||||
el("td", { class: "path", "data-testid": "album-folder" }, album.folder),
|
|
||||||
el("td", { class: "path", "data-testid": "album-destination" }, album.destination),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{ "data-testid": "album-method" },
|
|
||||||
album.transfer_method === "move" ? "move (same filesystem)" : "copy · verify · remove"
|
|
||||||
),
|
|
||||||
el("td", { "data-testid": "album-assets" }, String(album.asset_count)),
|
|
||||||
el("td", { "data-testid": "album-reclaim" }, bytes(album.reclaimable_bytes)),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el("span", { class: `badge ${album.state}`, "data-testid": "album-state" }, album.state),
|
|
||||||
...album.blockers.map((blocker) =>
|
|
||||||
el(
|
|
||||||
"div",
|
|
||||||
{ class: "blocker", "data-testid": "album-blocker", "data-code": blocker.code },
|
|
||||||
blocker.message
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ "data-testid": "archive-preview" },
|
|
||||||
el("h2", {}, "Preview"),
|
|
||||||
el(
|
|
||||||
"div",
|
|
||||||
{ class: "decision-bar" },
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{ class: "badge", "data-testid": "destination-identity" },
|
|
||||||
`${location.name} · ${location.media_id}`
|
|
||||||
),
|
|
||||||
el("span", { class: "badge", "data-testid": "total-albums" }, `${totals.albums} album(s)`),
|
|
||||||
el("span", { class: "badge", "data-testid": "total-assets" }, `${totals.assets} photo(s)`),
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{ class: "badge", "data-testid": "total-reclaim" },
|
|
||||||
`${bytes(totals.bytes)} reclaimable`
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{
|
|
||||||
class: `badge ${capacity.sufficient ? "complete" : "blocked"}`,
|
|
||||||
"data-testid": "capacity",
|
|
||||||
},
|
|
||||||
`free ${bytes(capacity.free_bytes)} · reserve ${bytes(capacity.reserve_bytes)}`
|
|
||||||
)
|
|
||||||
),
|
|
||||||
blockerList(preflight.blockers, "preflight-blockers", "preflight-blocker", "This scope cannot be archived yet"),
|
|
||||||
rows.length
|
|
||||||
? el(
|
|
||||||
"table",
|
|
||||||
{ class: "grid", "data-testid": "archive-albums" },
|
|
||||||
el(
|
|
||||||
"thead",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{},
|
|
||||||
...["Album", "Folder", "Destination", "Transfer", "Photos", "Reclaims", "State"].map(
|
|
||||||
(label) => el("th", { scope: "col" }, label)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el("tbody", {}, ...rows)
|
|
||||||
)
|
|
||||||
: el(
|
|
||||||
"p",
|
|
||||||
{ class: "muted", "data-testid": "no-albums" },
|
|
||||||
"No album has a verified upload whose bytes are still unchanged, so nothing may be archived."
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function blockerList(blockers, containerId, itemId, title) {
|
|
||||||
if (!blockers || !blockers.length) return null;
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "alert", role: "alert", "data-testid": containerId },
|
|
||||||
el("strong", {}, title),
|
|
||||||
el(
|
|
||||||
"ul",
|
|
||||||
{},
|
|
||||||
...blockers.map((blocker) =>
|
|
||||||
el("li", { "data-testid": itemId, "data-code": blocker.code }, `${blocker.code}: ${blocker.message}`)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── confirmation ─────────────────────────────────────────────────────────────
|
|
||||||
function confirmBlock(preflight, location) {
|
|
||||||
const ready = preflight.state === "ready";
|
|
||||||
const totals = preflight.totals;
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "card", "data-testid": "confirm" },
|
|
||||||
el("h2", {}, "Confirm"),
|
|
||||||
el(
|
|
||||||
"p",
|
|
||||||
{ class: "muted", "data-testid": "confirm-token" },
|
|
||||||
`Preflight ${preflight.token.slice(0, 20)}… · ${location.name}`
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"p",
|
|
||||||
{ "data-testid": "archive-note" },
|
|
||||||
"Archiving removes each original from the library — but only after its copy on " +
|
|
||||||
"the medium has been written, hashed, and recorded in the manifest. The photos " +
|
|
||||||
"stay searchable and deduplicable while the medium is away, and can be restored " +
|
|
||||||
"from this view."
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"div",
|
|
||||||
{ class: "toolbar" },
|
|
||||||
el(
|
|
||||||
"button",
|
|
||||||
{
|
|
||||||
class: "primary",
|
|
||||||
"data-testid": "start-archive",
|
|
||||||
disabled: ready ? false : "disabled",
|
|
||||||
title: ready ? false : "resolve the blockers above first",
|
|
||||||
onclick: () =>
|
|
||||||
run(async () => {
|
|
||||||
const plan = await api.createArchivePlan({
|
|
||||||
location_id: location.id,
|
|
||||||
token: preflight.token,
|
|
||||||
});
|
|
||||||
const started = await api.applyArchivePlan(plan.id);
|
|
||||||
watch(started.job.id, plan.id, "archive");
|
|
||||||
return { archiving: plan.asset_count };
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
`Archive ${totals.ready_albums} album(s) · reclaim ${bytes(totals.bytes)}`
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── plans and progress ───────────────────────────────────────────────────────
|
|
||||||
function planList(runs, selectedId) {
|
|
||||||
if (!runs.length) {
|
|
||||||
return el("p", { class: "muted", "data-testid": "no-plans" }, "Nothing has been archived yet.");
|
|
||||||
}
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ "data-testid": "archive-plans" },
|
|
||||||
el("h2", {}, "Runs"),
|
|
||||||
el(
|
|
||||||
"table",
|
|
||||||
{ class: "grid", "data-testid": "plans" },
|
|
||||||
el(
|
|
||||||
"thead",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{},
|
|
||||||
...["Created", "Direction", "State", "Photos", "Bytes"].map((l) =>
|
|
||||||
el("th", { scope: "col" }, l)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"tbody",
|
|
||||||
{},
|
|
||||||
...runs.map((plan) =>
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{
|
|
||||||
"data-testid": "plan-row",
|
|
||||||
"data-plan": plan.id,
|
|
||||||
"data-direction": plan.direction,
|
|
||||||
"aria-current": plan.id === selectedId ? "true" : false,
|
|
||||||
},
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el("a", { class: "link", href: `#/archive?plan=${encodeURIComponent(plan.id)}` }, plan.created_at || plan.id)
|
|
||||||
),
|
|
||||||
el("td", { "data-testid": "plan-direction" }, plan.direction),
|
|
||||||
el("td", {}, el("span", { class: `badge ${plan.state}`, "data-testid": "plan-state" }, plan.state)),
|
|
||||||
el("td", {}, String(plan.asset_count)),
|
|
||||||
el("td", {}, bytes(plan.byte_size))
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function planDetail(plan) {
|
|
||||||
const operations = plan.operations || [];
|
|
||||||
const counts = {};
|
|
||||||
for (const operation of operations) {
|
|
||||||
counts[operation.journal_state] = (counts[operation.journal_state] || 0) + 1;
|
|
||||||
}
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "card", "data-testid": "plan-detail", "data-plan": plan.id },
|
|
||||||
el("h2", {}, `${plan.direction === "restore" ? "Restore" : "Archive"} run ${plan.created_at || plan.id}`),
|
|
||||||
// Transfer, verification, and removal are separate answers to separate
|
|
||||||
// questions: what has moved, what is proven, and what is already gone.
|
|
||||||
el(
|
|
||||||
"div",
|
|
||||||
{ class: "decision-bar", "data-testid": "plan-progress" },
|
|
||||||
el("span", { class: `badge ${plan.state}`, "data-testid": "detail-state" }, plan.state),
|
|
||||||
...PHASES.map(([key, label, title]) =>
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{ class: `badge ${key}`, "data-testid": `count-${label}`, title },
|
|
||||||
`${label}: ${counts[key] ?? 0}`
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"table",
|
|
||||||
{ class: "grid", "data-testid": "operations" },
|
|
||||||
el(
|
|
||||||
"thead",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{},
|
|
||||||
...["Photo", "Destination", "Phase", "Attempts", "Problem"].map((l) =>
|
|
||||||
el("th", { scope: "col" }, l)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"tbody",
|
|
||||||
{},
|
|
||||||
...operations.map((operation) =>
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{ "data-testid": "operation-row", "data-asset-id": operation.asset_id },
|
|
||||||
el("td", { class: "path", "data-testid": "operation-source" }, operation.source_path),
|
|
||||||
el("td", { class: "path", "data-testid": "operation-destination" }, operation.destination_path),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{ class: `badge ${operation.journal_state}`, "data-testid": "operation-phase" },
|
|
||||||
phaseLabel(operation.journal_state)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el("td", {}, String(operation.attempt_count)),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{ class: "muted", "data-testid": "operation-error", "data-code": operation.error_code || "" },
|
|
||||||
operation.error_code ? `${operation.error_code}: ${operation.error_message || ""}` : "—"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function phaseLabel(state) {
|
|
||||||
const found = PHASES.find(([key]) => key === state);
|
|
||||||
return found ? found[1] : state;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── recovery ─────────────────────────────────────────────────────────────────
|
|
||||||
// What an interrupted run left behind, straight from the journal plus the files on
|
|
||||||
// disk. Only the operations the server itself classified as resolvable get an
|
|
||||||
// action; ambiguous ones are shown with their evidence and no button.
|
|
||||||
function recoverySection(recovery) {
|
|
||||||
const operations = recovery ? recovery.operations : [];
|
|
||||||
if (!operations.length) return null;
|
|
||||||
const manual = recovery.manual || [];
|
|
||||||
const resolvable = operations.length - manual.length;
|
|
||||||
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "card", "data-testid": "archive-recovery" },
|
|
||||||
el("h2", {}, "Interrupted work"),
|
|
||||||
el(
|
|
||||||
"p",
|
|
||||||
{ "data-testid": "recovery-summary" },
|
|
||||||
`${operations.length} operation(s) did not finish · ${resolvable} resolvable · ` +
|
|
||||||
`${manual.length} need a decision`
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"table",
|
|
||||||
{ class: "grid", "data-testid": "recovery-operations" },
|
|
||||||
el(
|
|
||||||
"thead",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{},
|
|
||||||
...["Photo", "Phase", "Verdict", "Evidence"].map((l) => el("th", { scope: "col" }, l))
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"tbody",
|
|
||||||
{},
|
|
||||||
...operations.map((verdict) =>
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{
|
|
||||||
"data-testid": "recovery-row",
|
|
||||||
"data-classification": verdict.classification,
|
|
||||||
"data-direction": verdict.direction,
|
|
||||||
},
|
|
||||||
el("td", { class: "path", "data-testid": "recovery-source" }, verdict.source_path),
|
|
||||||
el("td", {}, el("span", { class: "badge" }, phaseLabel(verdict.journal_state))),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{
|
|
||||||
class: `badge ${verdict.classification === "manual" ? "blocked" : "attention"}`,
|
|
||||||
"data-testid": "recovery-verdict",
|
|
||||||
},
|
|
||||||
verdict.classification
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{ class: "muted", "data-testid": "recovery-reason" },
|
|
||||||
`${verdict.reason} (source ${verdict.source_exists ? "present" : "absent"}, ` +
|
|
||||||
`archive copy ${verdict.destination_matches ? "verified" : verdict.destination_exists ? "different bytes" : "absent"})`
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
manual.length
|
|
||||||
? el(
|
|
||||||
"div",
|
|
||||||
{ class: "alert", role: "alert", "data-testid": "recovery-manual" },
|
|
||||||
`${manual.length} operation(s) cannot be resolved from the evidence. Nothing will be ` +
|
|
||||||
"removed or retried for them here: inspect the medium and the library, then decide."
|
|
||||||
)
|
|
||||||
: null,
|
|
||||||
el(
|
|
||||||
"div",
|
|
||||||
{ class: "toolbar" },
|
|
||||||
resolvable
|
|
||||||
? el(
|
|
||||||
"button",
|
|
||||||
{
|
|
||||||
class: "primary",
|
|
||||||
"data-testid": "resolve-recovery",
|
|
||||||
onclick: () => run(() => api.resolveArchiveRecovery()),
|
|
||||||
},
|
|
||||||
`Finish ${resolvable} recoverable operation(s)`
|
|
||||||
)
|
|
||||||
: el(
|
|
||||||
"span",
|
|
||||||
{ class: "muted", "data-testid": "no-safe-recovery" },
|
|
||||||
"No operation can be finished safely from here."
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── archived assets ──────────────────────────────────────────────────────────
|
|
||||||
// Browsing what is already archived, including while the medium is away: the
|
|
||||||
// retained protected preview and the recorded hashes are the evidence, so the row
|
|
||||||
// stays complete and honest instead of turning into a missing file.
|
|
||||||
function archivedSection(assets, locations) {
|
|
||||||
const archived = assets.filter((asset) => asset.availability_state !== "active");
|
|
||||||
if (!archived.length) return null;
|
|
||||||
const names = Object.fromEntries(locations.map((location) => [location.id, location.name]));
|
|
||||||
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ "data-testid": "archived-assets" },
|
|
||||||
el("h2", {}, "Archived photos"),
|
|
||||||
el(
|
|
||||||
"table",
|
|
||||||
{ class: "grid", "data-testid": "archived" },
|
|
||||||
el(
|
|
||||||
"thead",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{},
|
|
||||||
...["Preview", "Archived as", "Medium", "Availability", "Size"].map((l) =>
|
|
||||||
el("th", { scope: "col" }, l)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"tbody",
|
|
||||||
{},
|
|
||||||
...archived.map((asset) =>
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{ "data-testid": "archived-row", "data-asset-id": asset.id },
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el("img", {
|
|
||||||
"data-testid": "archived-preview",
|
|
||||||
width: 96,
|
|
||||||
loading: "lazy",
|
|
||||||
src: api.thumbnailUrl(asset.id, 256),
|
|
||||||
alt: `Preview of ${asset.archive_path || asset.id}`,
|
|
||||||
})
|
|
||||||
),
|
|
||||||
el("td", { class: "path", "data-testid": "archived-path" }, asset.archive_path || "—"),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{ "data-testid": "archived-medium" },
|
|
||||||
names[asset.archive_location_id] || asset.archive_location_id || "—"
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"span",
|
|
||||||
{
|
|
||||||
class: `badge ${asset.availability_state === "archived_online" ? "complete" : "attention"}`,
|
|
||||||
"data-testid": "archived-availability",
|
|
||||||
},
|
|
||||||
AVAILABILITY_LABEL[asset.availability_state] || asset.availability_state
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el("td", {}, bytes(asset.byte_size))
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── restore ──────────────────────────────────────────────────────────────────
|
|
||||||
function restoreSection(restore, location) {
|
|
||||||
const ready = restore.state === "ready";
|
|
||||||
const items = restore.items || [];
|
|
||||||
if (!items.length && !restore.blockers.length) return null;
|
|
||||||
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "card", "data-testid": "restore" },
|
|
||||||
el("h2", {}, "Restore"),
|
|
||||||
el(
|
|
||||||
"p",
|
|
||||||
{ "data-testid": "restore-note" },
|
|
||||||
"Restoring copies the archived bytes back into the library and leaves the archive " +
|
|
||||||
"copy where it is. A name that is already taken is never overwritten: the photo " +
|
|
||||||
"comes back beside it under a visibly different name."
|
|
||||||
),
|
|
||||||
blockerList(restore.blockers, "restore-blockers", "restore-blocker", "This restore cannot run yet"),
|
|
||||||
items.length
|
|
||||||
? el(
|
|
||||||
"table",
|
|
||||||
{ class: "grid", "data-testid": "restore-items" },
|
|
||||||
el(
|
|
||||||
"thead",
|
|
||||||
{},
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{},
|
|
||||||
...["Archived as", "Comes back as", "Size", "State"].map((l) =>
|
|
||||||
el("th", { scope: "col" }, l)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
),
|
|
||||||
el(
|
|
||||||
"tbody",
|
|
||||||
{},
|
|
||||||
...items.map((item) =>
|
|
||||||
el(
|
|
||||||
"tr",
|
|
||||||
{ "data-testid": "restore-row", "data-asset-id": item.asset_id },
|
|
||||||
el("td", { class: "path", "data-testid": "restore-source" }, item.archive_path),
|
|
||||||
el("td", { class: "path", "data-testid": "restore-destination" }, item.destination_path || "—"),
|
|
||||||
el("td", {}, bytes(item.byte_size)),
|
|
||||||
el(
|
|
||||||
"td",
|
|
||||||
{},
|
|
||||||
item.blockers.length
|
|
||||||
? el(
|
|
||||||
"span",
|
|
||||||
{
|
|
||||||
class: "badge blocked",
|
|
||||||
"data-testid": "restore-item-blocker",
|
|
||||||
"data-code": item.blockers[0].code,
|
|
||||||
},
|
|
||||||
item.blockers[0].code
|
|
||||||
)
|
|
||||||
: el("span", { class: "badge ready", "data-testid": "restore-item-state" }, "ready")
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
: null,
|
|
||||||
el(
|
|
||||||
"div",
|
|
||||||
{ class: "toolbar" },
|
|
||||||
el(
|
|
||||||
"button",
|
|
||||||
{
|
|
||||||
class: "primary",
|
|
||||||
"data-testid": "start-restore",
|
|
||||||
disabled: ready ? false : "disabled",
|
|
||||||
title: ready ? false : "the medium and every archived copy must check out first",
|
|
||||||
onclick: () =>
|
|
||||||
run(async () => {
|
|
||||||
const plan = await api.createRestorePlan({
|
|
||||||
location_id: location.id,
|
|
||||||
token: restore.token,
|
|
||||||
});
|
|
||||||
const started = await api.applyRestorePlan(plan.id);
|
|
||||||
watch(started.job.id, plan.id, "restore");
|
|
||||||
return { restoring: plan.asset_count };
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
`Restore ${items.length} photo(s) from ${location.name}`
|
|
||||||
)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── running commands ─────────────────────────────────────────────────────────
|
|
||||||
async function run(action) {
|
|
||||||
try {
|
|
||||||
outcome = { kind: "ok", result: await action() };
|
|
||||||
} catch (error) {
|
|
||||||
outcome = error.status === 409 ? { kind: "conflict", error } : { kind: "error", error };
|
|
||||||
}
|
|
||||||
render();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Live job activity. The plan panel is refreshed on its own tick because the
|
|
||||||
// journal advances per file, not per job event; a full re-render would re-run
|
|
||||||
// preflight (which re-hashes the library), so that happens once when the job ends.
|
|
||||||
const REFRESH_MS = 500;
|
|
||||||
|
|
||||||
function watch(jobId, planId, kind) {
|
|
||||||
activity = [`Started ${kind} job ${jobId}`];
|
|
||||||
const tick = setInterval(() => refreshPlan(planId, kind), REFRESH_MS);
|
|
||||||
subscribeJob(jobId, {
|
|
||||||
onEvent: (event) => {
|
|
||||||
activity.push(`${event.type}${event.message ? ": " + event.message : ""}`);
|
|
||||||
const log = document.querySelector('[data-testid="archive-activity"]');
|
|
||||||
if (log) log.textContent = activity.join("\n");
|
|
||||||
},
|
|
||||||
onDone: () => {
|
|
||||||
clearInterval(tick);
|
|
||||||
activity.push("done");
|
|
||||||
render();
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
async function refreshPlan(planId, kind) {
|
|
||||||
const node = document.querySelector(`[data-testid="plan-detail"][data-plan="${planId}"]`);
|
|
||||||
if (!node) return; // the user navigated away from the running plan
|
|
||||||
try {
|
|
||||||
const plan = await (kind === "restore" ? api.getRestorePlan(planId) : api.getArchivePlan(planId));
|
|
||||||
node.replaceWith(planDetail(plan));
|
|
||||||
} catch (_) {
|
|
||||||
// Transient; the next tick tries again and the job's end re-renders anyway.
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function activityLog() {
|
|
||||||
return el(
|
|
||||||
"pre",
|
|
||||||
{
|
|
||||||
class: "activity-log",
|
|
||||||
role: "status",
|
|
||||||
"aria-live": "polite",
|
|
||||||
"data-testid": "archive-activity",
|
|
||||||
},
|
|
||||||
activity.join("\n")
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function outcomeBanner() {
|
|
||||||
if (!outcome) return null;
|
|
||||||
if (outcome.kind === "conflict") {
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "alert", role: "alert", "data-testid": "conflict" },
|
|
||||||
`The server refused this: ${outcome.error.message}. Nothing was moved or removed; ` +
|
|
||||||
"the state below is the server's current one — review it and decide again."
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if (outcome.kind === "error") {
|
|
||||||
return el(
|
|
||||||
"div",
|
|
||||||
{ class: "alert", role: "alert", "data-testid": "archive-error" },
|
|
||||||
`Failed: ${outcome.error.message}`
|
|
||||||
);
|
|
||||||
}
|
|
||||||
const result = outcome.result || {};
|
|
||||||
const message =
|
|
||||||
result.archiving !== undefined
|
|
||||||
? `Archiving ${result.archiving} photo(s). Originals are removed only after their copies verify.`
|
|
||||||
: result.restoring !== undefined
|
|
||||||
? `Restoring ${result.restoring} photo(s) into the library.`
|
|
||||||
: "Done — the state below is the server's.";
|
|
||||||
return el("div", { class: "alert", role: "status", "data-testid": "archive-result" }, message);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── formatting ───────────────────────────────────────────────────────────────
|
|
||||||
function bytes(value) {
|
|
||||||
if (value == null) return "unknown";
|
|
||||||
const units = ["B", "kB", "MB", "GB", "TB"];
|
|
||||||
let size = value;
|
|
||||||
let unit = 0;
|
|
||||||
while (size >= 1000 && unit < units.length - 1) {
|
|
||||||
size /= 1000;
|
|
||||||
unit += 1;
|
|
||||||
}
|
|
||||||
return `${unit === 0 ? size : size.toFixed(1)} ${units[unit]}`;
|
|
||||||
}
|
|
||||||
@@ -29,10 +29,6 @@ function jsonResponse(status, body) {
|
|||||||
const tick = (ms = 0) => new Promise((r) => setTimeout(r, ms));
|
const tick = (ms = 0) => new Promise((r) => setTimeout(r, ms));
|
||||||
|
|
||||||
async function run() {
|
async function run() {
|
||||||
// The client fetches its CSRF token once, lazily (US07-02). Do that against the
|
|
||||||
// real server first, so the stubbed fetch below only ever sees the call under test.
|
|
||||||
await api.workflow().catch(() => {});
|
|
||||||
|
|
||||||
// ── store ────────────────────────────────────────────────────────────────
|
// ── store ────────────────────────────────────────────────────────────────
|
||||||
{
|
{
|
||||||
const store = createStore({ n: 0 });
|
const store = createStore({ n: 0 });
|
||||||
|
|||||||
@@ -1,22 +0,0 @@
|
|||||||
d098bbde13d2ebc872ca781e244cc1e48b97cc551fedaca3251d6a5278a15234 src/compare_models.py
|
|
||||||
7d0f68cb95fbd6603e4c558620098b46929ebdb6fd91a598bbf84f26eea35e6c src/nsfw_tag.py
|
|
||||||
35329c53570e215cef15f59429c1a5251a0448b90522b0910ab9d68efe1bc307 src/nsfwtag/__init__.py
|
|
||||||
7f52b55e4f3b77eda3657d7cd2272c7cca8422100e241671de495603c1126ee2 src/nsfwtag/__main__.py
|
|
||||||
6f5a97114e0d87d272ce22d065dc31fa0dd72ff8590d11ad14cb1c1f486d339c src/nsfwtag/bench.py
|
|
||||||
29498fac1d73ba2b7420ffe1ffe49a87f684ec299e2c3b9c6e14e63e95643ba4 src/nsfwtag/exif.py
|
|
||||||
934e82c402813ebf503e5a20eb03d5103f84df13117bc4dd1fd95deaa01ab263 src/nsfwtag/README.md
|
|
||||||
68125e6184c7e4d2a5b0153f2155675753ab0b1329d1dbc4933d4769492ec4be src/nsfwtag/review.html
|
|
||||||
a597eab74803dd31452b5d70abf7d6d6320eb17a61596c891b009da15d624e98 src/nsfwtag/scoring.py
|
|
||||||
674969a18e58ee511a2abf574a875db92f2f524769b1b614d0d3662ab879a709 src/nsfwtag/server.py
|
|
||||||
24a7c8d029da6e97d46b110a9fe9dbb3900127f8ae3c142f634fdd91d6c45243 src/nsfwtag/webapp.py
|
|
||||||
2c2ea558f1b9095c1150f078ea29c8e0f180cc045a9d1097b83f61bfb145295b src/photo_analyzer.py
|
|
||||||
5fb7ce2f977a17da1f501c45d7985328318ae14298a3f347eab46a6e3f02aab3 src/test_dedup.py
|
|
||||||
1673dc76cc60aba56a5b065d9bc9f5dda00342cf2a79d000a3f74686fb7cffec src/test_nsfw_skip.py
|
|
||||||
55aa82f348e0e90be0163ba6aa278e5fc8e445996e5335774917e5ef59ad3563 src/webapp/__init__.py
|
|
||||||
ba4342bd0175591a2121f063f14a78a602d5679bcfcf24aa14c8d4b8cd5da1a2 src/webapp/__main__.py
|
|
||||||
a60f24035a989909467778f888d62854c1afc2702f47b8d3cca32bf3cf04d5cf src/webapp/analyzer.html
|
|
||||||
018bcc2f770d716444b456db58d6b4a41800138a98598b3c27ba503ab22d939e src/webapp/page.py
|
|
||||||
5f599b107b2b117ca118b6fbdec5e5786a9ab4eea424aa9af1bfea7bf87cebab src/webapp/query.py
|
|
||||||
875cce697caa02717c85a2707ba935c46dd9ace18428fd4abb13105b7a8e24d9 src/webapp/README.md
|
|
||||||
af7d0d72d245b4bbb1e0e30f9708239697543212d4171defd1bfe26dd24a4f05 src/webapp/runner.py
|
|
||||||
cddf0555b06fe7e4daa14309baaaf91130f8506922cb0ffb6507d76cbe39add4 src/webapp/server.py
|
|
||||||
@@ -1,80 +0,0 @@
|
|||||||
# Legacy CLI archive (US07-01)
|
|
||||||
|
|
||||||
Frozen, read-only sources of the command-line tools this application was extracted
|
|
||||||
from. They are **reference material and rollback evidence** — provenance for
|
|
||||||
behavior that now lives in `photo_pipeline/`, and the only way to answer "what did
|
|
||||||
the original actually do?" once the replacement has drifted.
|
|
||||||
|
|
||||||
> **Nothing here is production code.** No module under `photo_pipeline/` imports or
|
|
||||||
> executes anything in this directory, and this directory is not on the application's
|
|
||||||
> import path. `tests/unit/test_legacy_archive.py` enforces both, along with the
|
|
||||||
> checksums and the redaction below.
|
|
||||||
|
|
||||||
## What is here
|
|
||||||
|
|
||||||
| Path | Role |
|
|
||||||
|---|---|
|
|
||||||
| `src/photo_analyzer.py` | the analysis CLI: discovery, hashing/dedup, vision analysis, EXIF writing, SQLite schema, album naming |
|
|
||||||
| `src/nsfwtag/` | NSFW scoring, EXIF safety keywords, and the review server (`__init__` 1.1.0) |
|
|
||||||
| `src/webapp/` | the stdlib review web app: FTS search, stats, subprocess runner, HTML shell (`__init__` 0.1.0) |
|
|
||||||
| `src/nsfw_tag.py` | thin backwards-compatible entry point for `nsfwtag` |
|
|
||||||
| `src/compare_models.py` | dev-only model comparison script |
|
|
||||||
| `src/test_dedup.py`, `src/test_nsfw_skip.py` | the CLIs' own standalone self-checks (never pytest suites) |
|
|
||||||
| `donor_ledger.yaml` | the donor ledger: every migrated behavior, its target, its tests, and every intentional delta |
|
|
||||||
| `requirements-lock.txt` | the dependency versions the frozen sources were last verified against |
|
|
||||||
| `photo_analyzer.env.sample` | the CLI's configuration surface, with every value replaced by a placeholder |
|
|
||||||
| `CHECKSUMS.sha256` | SHA-256 of every archived source file |
|
|
||||||
|
|
||||||
`photo_analyzer.py` carries no `__version__`; its identity is its checksum, recorded
|
|
||||||
in `CHECKSUMS.sha256` and taken at commit `9b7ee6b` (the merge of US06-06, the last
|
|
||||||
commit before archival).
|
|
||||||
|
|
||||||
## Verifying the archive
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256
|
|
||||||
```
|
|
||||||
|
|
||||||
Any edit to an archived source must be accompanied by a regenerated checksum file
|
|
||||||
and a note here explaining why a *frozen* archive changed — the normal answer being
|
|
||||||
that it should not.
|
|
||||||
|
|
||||||
## Schema notes
|
|
||||||
|
|
||||||
`photo_analyzer.py` owned a path-keyed SQLite database (`SCHEMA`, near the top of the
|
|
||||||
file):
|
|
||||||
|
|
||||||
- `photos(id, path UNIQUE, status, phash, file_sha1, dup_of, description, tags,
|
|
||||||
people_count, setting, time_of_day, season, mood, location_hint, approx_year,
|
|
||||||
raw_response, error_message, analyzed_at, exif_written_at)`;
|
|
||||||
- `photos_fts` — an FTS5 external-content index over `path, description, tags, mood,
|
|
||||||
location_hint`, kept in sync by insert/update/delete triggers;
|
|
||||||
- late columns (`phash`, `file_sha1`, `dup_of`) were added by an in-code
|
|
||||||
`_migrate_schema()` rather than a migration tool, and their indexes are created
|
|
||||||
only after the `ALTER`.
|
|
||||||
|
|
||||||
The replacement keeps the same analysis fields but re-keys everything to a stable
|
|
||||||
`assets.id` (Alembic migrations `0001`…), because a path is not an identity: the
|
|
||||||
donor's `path UNIQUE` is exactly what broke on every move and rename.
|
|
||||||
|
|
||||||
`nsfwtag` kept its safety scores outside the database in `nsfw_scores.csv`
|
|
||||||
(`path,nsfw_score`, four decimals, unreadable rows dropped). That file is no longer a
|
|
||||||
source of truth; `photo_pipeline/services/legacy_import.py` imports it into
|
|
||||||
`assets.safety_score` and reports exactly what matched, what did not, and why.
|
|
||||||
|
|
||||||
## Redaction
|
|
||||||
|
|
||||||
The archive contains no credentials. `photo_analyzer.env.sample` documents the
|
|
||||||
configuration surface (`LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL`, and the tuning
|
|
||||||
variables) with placeholder values only; the CLI itself never contained a key, it
|
|
||||||
read one from `photo_analyzer.env` or the environment. No `.env`, database, log, CSV,
|
|
||||||
or photo from the author's library is archived.
|
|
||||||
|
|
||||||
## Why these tools were retired
|
|
||||||
|
|
||||||
Each behavior's fate is recorded per row in `donor_ledger.yaml`: `reuse`, `extract`,
|
|
||||||
`refactor`, or `replace`, with the target module, the characterization tests that
|
|
||||||
pinned the donor's behavior, the parity tests the replacement passes, and — where the
|
|
||||||
replacement deliberately does something else or nothing at all — a `delta` saying so.
|
|
||||||
Rows still marked `pending` name the backlog story that will resolve them; they are
|
|
||||||
the honest list of what has *not* been carried over yet.
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
# Sample configuration for the archived photo_analyzer.py CLI (US07-01).
|
|
||||||
#
|
|
||||||
# REDACTED: no value below is real. Each line names a variable the CLI read and
|
|
||||||
# what belongs there; the placeholders are deliberately not key-shaped, so this
|
|
||||||
# file can never be mistaken for — or scanned as — a credential.
|
|
||||||
|
|
||||||
LLM_API_KEY=<paste your provider key here>
|
|
||||||
LLM_BASE_URL=<provider base url, e.g. the OpenAI-compatible Gemini endpoint>
|
|
||||||
LLM_MODEL=<model id, e.g. a Gemini Flash release>
|
|
||||||
|
|
||||||
# Optional tuning the CLI read from the same file:
|
|
||||||
PHASH_THRESHOLD=8
|
|
||||||
MAX_WORKERS=4
|
|
||||||
RETRY_ATTEMPTS=3
|
|
||||||
RPD_LIMIT=0
|
|
||||||
|
|
||||||
# The CLI ignored its own shipped placeholder (a literal "sk-REPLACE..." string)
|
|
||||||
# until it was replaced, and a shell variable always won over this file.
|
|
||||||
@@ -1,20 +0,0 @@
|
|||||||
# Final dependency lock of the archived CLIs (US07-01).
|
|
||||||
#
|
|
||||||
# These are the versions present in the environment the archive was taken from —
|
|
||||||
# what the frozen sources were last verified against by the characterization
|
|
||||||
# suite. Restoring a donor for forensics means pinning these, not "latest".
|
|
||||||
#
|
|
||||||
# Python 3.14.6
|
|
||||||
|
|
||||||
openai==3.0.0 # photo_analyzer: OpenAI-compatible vision client
|
|
||||||
numpy==2.4.6 # photo_analyzer, nsfwtag: pixel work
|
|
||||||
Pillow==12.3.0 # photo_analyzer, nsfwtag: decode/resize
|
|
||||||
rich==15.0.0 # photo_analyzer: console output
|
|
||||||
scipy==1.18.0 # photo_analyzer: perceptual-hash DCT
|
|
||||||
PyYAML==6.0.3 # tooling that reads the donor ledger
|
|
||||||
|
|
||||||
# NSFW inference (nsfwtag/scoring.py, nsfwtag/bench.py) was never installed in the
|
|
||||||
# archiving environment; the model stack is recorded here from the sources so a
|
|
||||||
# forensic run can reproduce it, not from a resolved lock:
|
|
||||||
# torch, transformers, timm — AdamCodd/vit-base-nsfw-detector (see nsfwtag/README.md)
|
|
||||||
# exiftool is an external binary, not a Python package.
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
"""Protected thumbnails (US06-03).
|
|
||||||
|
|
||||||
Revision ID: 0013_protected_thumbnails
|
|
||||||
Revises: 0012_archive_plans
|
|
||||||
Create Date: 2026-08-16
|
|
||||||
|
|
||||||
A protected thumbnail is the durable comparison preview of an asset whose
|
|
||||||
original has left active storage. It is evidence rather than cache, so the LRU
|
|
||||||
quota must not evict it: the archive medium may be offline when it is needed.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision = "0013_protected_thumbnails"
|
|
||||||
down_revision = "0012_archive_plans"
|
|
||||||
branch_labels = None
|
|
||||||
depends_on = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.add_column(
|
|
||||||
"thumbnails",
|
|
||||||
sa.Column("protected", sa.Boolean(), nullable=False, server_default=sa.false()),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_column("thumbnails", "protected")
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
"""Restore plans and archive divergence (US06-04).
|
|
||||||
|
|
||||||
Revision ID: 0014_restore_plans
|
|
||||||
Revises: 0013_protected_thumbnails
|
|
||||||
Create Date: 2026-08-16
|
|
||||||
|
|
||||||
Restore reuses the archive plan and journal tables: the crash-safe question is the
|
|
||||||
same one in the opposite direction (copy, verify, publish, register), so the rows
|
|
||||||
gain a ``direction`` instead of a parallel pair of tables. ``archive_divergent_at``
|
|
||||||
records the moment an archived copy was proven to hold bytes that are not the ones
|
|
||||||
the database recorded — a restore must never silently accept a different file.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision = "0014_restore_plans"
|
|
||||||
down_revision = "0013_protected_thumbnails"
|
|
||||||
branch_labels = None
|
|
||||||
depends_on = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
for table in ("archive_plans", "archive_operations"):
|
|
||||||
op.add_column(
|
|
||||||
table,
|
|
||||||
sa.Column("direction", sa.String(), nullable=False, server_default="archive"),
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"assets", sa.Column("archive_divergent_at", sa.DateTime(timezone=True), nullable=True)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_column("assets", "archive_divergent_at")
|
|
||||||
for table in ("archive_plans", "archive_operations"):
|
|
||||||
op.drop_column(table, "direction")
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
"""Application management CLI: ``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores}``."""
|
"""Application management CLI: ``python -m photo_pipeline {serve,migrate}``."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -16,17 +16,6 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
commands.add_parser("migrate", help="Upgrade the database to the latest revision")
|
commands.add_parser("migrate", help="Upgrade the database to the latest revision")
|
||||||
worker_cmd = commands.add_parser("worker", help="Run a durable-job worker")
|
worker_cmd = commands.add_parser("worker", help="Run a durable-job worker")
|
||||||
worker_cmd.add_argument("--id", default="worker-1", help="Worker id (lease owner)")
|
worker_cmd.add_argument("--id", default="worker-1", help="Worker id (lease owner)")
|
||||||
import_cmd = commands.add_parser(
|
|
||||||
"import-legacy-scores",
|
|
||||||
help="Import the archived CLI's nsfw_scores.csv into the database (US07-01)",
|
|
||||||
)
|
|
||||||
import_cmd.add_argument("csv", help="Path to nsfw_scores.csv")
|
|
||||||
import_cmd.add_argument(
|
|
||||||
"--overwrite", action="store_true", help="Replace differing imported scores"
|
|
||||||
)
|
|
||||||
import_cmd.add_argument(
|
|
||||||
"--dry-run", action="store_true", help="Report what would happen and change nothing"
|
|
||||||
)
|
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
config = Config.from_env()
|
config = Config.from_env()
|
||||||
@@ -36,24 +25,6 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
run_migrations(config.database_url)
|
run_migrations(config.database_url)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
if args.command == "import-legacy-scores":
|
|
||||||
import json
|
|
||||||
|
|
||||||
from photo_pipeline.db import create_db_engine, create_session_factory
|
|
||||||
from photo_pipeline.services.legacy_import import LegacyImportService, write_report
|
|
||||||
|
|
||||||
run_migrations(config.database_url)
|
|
||||||
engine = create_db_engine(config.database_url)
|
|
||||||
service = LegacyImportService(create_session_factory(engine))
|
|
||||||
report = service.import_nsfw_scores(
|
|
||||||
args.csv, overwrite=args.overwrite, dry_run=args.dry_run
|
|
||||||
)
|
|
||||||
# The report is the point: an import nobody can audit is not a migration.
|
|
||||||
if not args.dry_run:
|
|
||||||
write_report(report, config.data_dir)
|
|
||||||
print(json.dumps(report.counts, indent=2))
|
|
||||||
return 0
|
|
||||||
|
|
||||||
if args.command == "worker":
|
if args.command == "worker":
|
||||||
from photo_pipeline.db import create_db_engine, create_session_factory
|
from photo_pipeline.db import create_db_engine, create_session_factory
|
||||||
|
|
||||||
|
|||||||
@@ -8,15 +8,11 @@ and exposes the versioned ``/api/v1`` surface; US01-02 ships only health.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
|
||||||
from contextlib import asynccontextmanager
|
from contextlib import asynccontextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from fastapi import FastAPI, Request
|
from fastapi import FastAPI
|
||||||
from fastapi.exceptions import RequestValidationError
|
|
||||||
from fastapi.responses import JSONResponse
|
|
||||||
from fastapi.staticfiles import StaticFiles
|
from fastapi.staticfiles import StaticFiles
|
||||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
|
||||||
|
|
||||||
from photo_pipeline.api.routes import (
|
from photo_pipeline.api.routes import (
|
||||||
albums,
|
albums,
|
||||||
@@ -29,12 +25,10 @@ from photo_pipeline.api.routes import (
|
|||||||
library,
|
library,
|
||||||
renames,
|
renames,
|
||||||
safety,
|
safety,
|
||||||
session as session_routes,
|
|
||||||
thumbnails,
|
thumbnails,
|
||||||
uploads,
|
uploads,
|
||||||
workflow,
|
workflow,
|
||||||
)
|
)
|
||||||
from photo_pipeline.api.security import DEFAULT_HEADERS, SecurityMiddleware, Session
|
|
||||||
|
|
||||||
# Registers the safety_score / analysis job handlers on import.
|
# Registers the safety_score / analysis job handlers on import.
|
||||||
import photo_pipeline.jobs.domain_handlers # noqa: F401
|
import photo_pipeline.jobs.domain_handlers # noqa: F401
|
||||||
@@ -45,41 +39,6 @@ from photo_pipeline.services.upload_batches import UploadBatchService
|
|||||||
|
|
||||||
FRONTEND_DIR = Path(__file__).resolve().parents[2] / "frontend"
|
FRONTEND_DIR = Path(__file__).resolve().parents[2] / "frontend"
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def _envelope(status: int, code: str, message: str) -> JSONResponse:
|
|
||||||
return JSONResponse(
|
|
||||||
status_code=status,
|
|
||||||
content={"error": {"code": code, "message": message}},
|
|
||||||
headers=DEFAULT_HEADERS,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _install_error_handlers(app: FastAPI) -> None:
|
|
||||||
"""One JSON error envelope everywhere, and nothing behind it.
|
|
||||||
|
|
||||||
An unhandled exception carries the library's absolute paths, SQL, and sometimes
|
|
||||||
a credential in its text; the client gets a code, the operator gets the traceback
|
|
||||||
in the server log (US07-02).
|
|
||||||
"""
|
|
||||||
|
|
||||||
@app.exception_handler(StarletteHTTPException)
|
|
||||||
async def _http_error(request: Request, exc: StarletteHTTPException):
|
|
||||||
return _envelope(exc.status_code, "http_error", str(exc.detail))
|
|
||||||
|
|
||||||
@app.exception_handler(RequestValidationError)
|
|
||||||
async def _validation_error(request: Request, exc: RequestValidationError):
|
|
||||||
# Field locations only: the echoed input can be the caller's own data, but it
|
|
||||||
# is also what ends up in shared logs and screenshots.
|
|
||||||
fields = sorted(".".join(str(part) for part in error["loc"]) for error in exc.errors())
|
|
||||||
return _envelope(422, "invalid_request", f"invalid request fields: {', '.join(fields)}")
|
|
||||||
|
|
||||||
@app.exception_handler(Exception)
|
|
||||||
async def _unhandled(request: Request, exc: Exception):
|
|
||||||
log.exception("unhandled error serving %s", request.url.path)
|
|
||||||
return _envelope(500, "internal_error", "internal error")
|
|
||||||
|
|
||||||
|
|
||||||
def create_app(config: Config | None = None) -> FastAPI:
|
def create_app(config: Config | None = None) -> FastAPI:
|
||||||
config = config or Config.from_env()
|
config = config or Config.from_env()
|
||||||
@@ -103,12 +62,6 @@ def create_app(config: Config | None = None) -> FastAPI:
|
|||||||
app.state.engine = None
|
app.state.engine = None
|
||||||
|
|
||||||
app = FastAPI(title="Photo Pipeline", version="0.1.0", lifespan=lifespan)
|
app = FastAPI(title="Photo Pipeline", version="0.1.0", lifespan=lifespan)
|
||||||
# One session per process: the browser exchanges it for a cookie + CSRF token,
|
|
||||||
# and every other origin is refused before a route ever runs (US07-02).
|
|
||||||
app.state.session = Session.create()
|
|
||||||
app.add_middleware(SecurityMiddleware, session=app.state.session, config=config)
|
|
||||||
_install_error_handlers(app)
|
|
||||||
app.include_router(session_routes.router, prefix="/api/v1")
|
|
||||||
app.include_router(health.router, prefix="/api/v1")
|
app.include_router(health.router, prefix="/api/v1")
|
||||||
app.include_router(inventory.router, prefix="/api/v1")
|
app.include_router(inventory.router, prefix="/api/v1")
|
||||||
app.include_router(duplicates.router, prefix="/api/v1")
|
app.include_router(duplicates.router, prefix="/api/v1")
|
||||||
|
|||||||
@@ -18,10 +18,7 @@ router = APIRouter(tags=["analysis"])
|
|||||||
|
|
||||||
|
|
||||||
def _service(request: Request) -> AnalysisService:
|
def _service(request: Request) -> AnalysisService:
|
||||||
return AnalysisService(
|
return AnalysisService(request.app.state.session_factory)
|
||||||
request.app.state.session_factory,
|
|
||||||
library_roots=tuple(request.app.state.config.library_roots),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _error(status: int, code: str, message: str) -> JSONResponse:
|
def _error(status: int, code: str, message: str) -> JSONResponse:
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
"""Archive location, preflight, plan, and restore API (US06-01, US06-02, US06-04).
|
"""Archive location, preflight, and plan API (US06-01, US06-02).
|
||||||
|
|
||||||
Registering a location writes a marker onto the medium; preflight is a command
|
Registering a location writes a marker onto the medium; preflight is a command
|
||||||
rather than a read, because it probes the destination, hashes the scope, and issues
|
rather than a read, because it probes the destination, hashes the scope, and issues
|
||||||
@@ -13,11 +13,10 @@ from fastapi import APIRouter, Request
|
|||||||
from fastapi.responses import JSONResponse
|
from fastapi.responses import JSONResponse
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
|
|
||||||
from photo_pipeline.jobs.domain_handlers import ARCHIVE_LOCK, ARCHIVE_PLAN, RESTORE_PLAN
|
from photo_pipeline.jobs.domain_handlers import ARCHIVE_LOCK, ARCHIVE_PLAN
|
||||||
from photo_pipeline.services.archives import ArchiveError, ArchiveService
|
from photo_pipeline.services.archives import ArchiveError, ArchiveService
|
||||||
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
||||||
from photo_pipeline.services.jobs import JobBlocked, JobService
|
from photo_pipeline.services.jobs import JobBlocked, JobService
|
||||||
from photo_pipeline.services.restores import RestoreService
|
|
||||||
|
|
||||||
router = APIRouter(tags=["archives"])
|
router = APIRouter(tags=["archives"])
|
||||||
|
|
||||||
@@ -43,24 +42,10 @@ class CreatePlanRequest(PreflightRequest):
|
|||||||
token: str
|
token: str
|
||||||
|
|
||||||
|
|
||||||
class RestoreRequest(BaseModel):
|
|
||||||
location_id: str
|
|
||||||
# ``None`` means every asset archived at this location.
|
|
||||||
asset_ids: list[str] | None = None
|
|
||||||
|
|
||||||
|
|
||||||
class CreateRestoreRequest(RestoreRequest):
|
|
||||||
token: str
|
|
||||||
|
|
||||||
|
|
||||||
def _service(request: Request) -> ArchiveService:
|
def _service(request: Request) -> ArchiveService:
|
||||||
return ArchiveService(request.app.state.session_factory, config=request.app.state.config)
|
return ArchiveService(request.app.state.session_factory, config=request.app.state.config)
|
||||||
|
|
||||||
|
|
||||||
def _restores(request: Request) -> RestoreService:
|
|
||||||
return RestoreService(request.app.state.session_factory, config=request.app.state.config)
|
|
||||||
|
|
||||||
|
|
||||||
def _transfers(request: Request) -> ArchiveTransferService:
|
def _transfers(request: Request) -> ArchiveTransferService:
|
||||||
return ArchiveTransferService(
|
return ArchiveTransferService(
|
||||||
request.app.state.session_factory, config=request.app.state.config
|
request.app.state.session_factory, config=request.app.state.config
|
||||||
@@ -153,76 +138,6 @@ def apply_plan(plan_id: str, request: Request):
|
|||||||
return {"plan_id": plan_id, "job": job}
|
return {"plan_id": plan_id, "job": job}
|
||||||
|
|
||||||
|
|
||||||
@router.post("/restore-preflight")
|
|
||||||
def restore_preflight(body: RestoreRequest, request: Request):
|
|
||||||
"""Validate restoring archived assets back into the library. Nothing moves."""
|
|
||||||
try:
|
|
||||||
return _restores(request).preflight(body.location_id, body.asset_ids)
|
|
||||||
except ArchiveError as error:
|
|
||||||
return _error(error)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/restore-plans", status_code=201)
|
|
||||||
def create_restore_plan(body: CreateRestoreRequest, request: Request):
|
|
||||||
try:
|
|
||||||
return _restores(request).create(body.location_id, body.asset_ids, token=body.token)
|
|
||||||
except ArchiveError as error:
|
|
||||||
return _error(error)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/restore-plans")
|
|
||||||
def list_restore_plans(request: Request) -> dict:
|
|
||||||
return {"plans": _restores(request).list()}
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/restore-plans/{plan_id}")
|
|
||||||
def get_restore_plan(plan_id: str, request: Request):
|
|
||||||
plan = _restores(request).get(plan_id)
|
|
||||||
if plan is None:
|
|
||||||
return _error(ArchiveError("unknown_plan", f"unknown restore plan {plan_id}"))
|
|
||||||
return plan
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/restore-plans/{plan_id}/apply")
|
|
||||||
def apply_restore_plan(plan_id: str, request: Request):
|
|
||||||
"""Queue the restore on the archiver lane — the same single lane as archiving,
|
|
||||||
because both move the same originals."""
|
|
||||||
service = _restores(request)
|
|
||||||
plan = service.get(plan_id)
|
|
||||||
if plan is None:
|
|
||||||
return _error(ArchiveError("unknown_plan", f"unknown restore plan {plan_id}"))
|
|
||||||
unresolved = [row for row in service.journal.incomplete() if row["plan_id"] != plan_id]
|
|
||||||
if unresolved:
|
|
||||||
return _error(
|
|
||||||
ArchiveError(
|
|
||||||
"archive_pending",
|
|
||||||
f"an unresolved archive operation ({unresolved[0]['id']}) must be recovered",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
job = JobService(request.app.state.session_factory).enqueue(
|
|
||||||
RESTORE_PLAN,
|
|
||||||
lock=ARCHIVE_LOCK,
|
|
||||||
idempotency_key=f"restore:{plan_id}:{plan['version']}",
|
|
||||||
items=[plan_id],
|
|
||||||
)
|
|
||||||
except JobBlocked as error:
|
|
||||||
return JSONResponse(
|
|
||||||
status_code=409, content={"error": {"code": error.code, "message": str(error)}}
|
|
||||||
)
|
|
||||||
return {"plan_id": plan_id, "job": job}
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/restore-recovery")
|
|
||||||
def restore_recovery_status(request: Request) -> dict:
|
|
||||||
return _restores(request).recovery_status()
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/restore-recovery/resolve")
|
|
||||||
def resolve_restore_recovery(request: Request) -> dict:
|
|
||||||
return _restores(request).recover()
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/archive-recovery")
|
@router.get("/archive-recovery")
|
||||||
def recovery_status(request: Request) -> dict:
|
def recovery_status(request: Request) -> dict:
|
||||||
"""What an interrupted transfer left behind, straight from journal + disk."""
|
"""What an interrupted transfer left behind, straight from journal + disk."""
|
||||||
|
|||||||
@@ -1,30 +0,0 @@
|
|||||||
"""Session bootstrap: the one endpoint reachable without a session (US07-02).
|
|
||||||
|
|
||||||
It sets the ``HttpOnly``/``SameSite=Strict`` session cookie and returns the CSRF
|
|
||||||
token in the body. A foreign page can call this — it just cannot read the answer,
|
|
||||||
because the app sends no CORS headers — and the cookie it received is never attached
|
|
||||||
to a request that foreign page initiates.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Request
|
|
||||||
from fastapi.responses import JSONResponse
|
|
||||||
|
|
||||||
from photo_pipeline.api.security import SESSION_COOKIE
|
|
||||||
|
|
||||||
router = APIRouter(tags=["session"])
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/session")
|
|
||||||
def start_session(request: Request) -> JSONResponse:
|
|
||||||
session = request.app.state.session
|
|
||||||
response = JSONResponse({"csrf_token": session.csrf_token})
|
|
||||||
response.set_cookie(
|
|
||||||
SESSION_COOKIE,
|
|
||||||
session.id,
|
|
||||||
httponly=True,
|
|
||||||
samesite="strict",
|
|
||||||
path="/",
|
|
||||||
)
|
|
||||||
return response
|
|
||||||
@@ -31,7 +31,5 @@ def get_thumbnail(asset_id: str, request: Request, size: int = Query(512)):
|
|||||||
return FileResponse(
|
return FileResponse(
|
||||||
path,
|
path,
|
||||||
media_type="image/webp",
|
media_type="image/webp",
|
||||||
# private: the URL is versioned and immutable, but these bytes are the user's
|
headers={"Cache-Control": "public, max-age=31536000, immutable"},
|
||||||
# photos and must never sit in a shared cache (US07-02).
|
|
||||||
headers={"Cache-Control": "private, max-age=31536000, immutable"},
|
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,192 +0,0 @@
|
|||||||
"""Local-web attack surface: session, CSRF, Origin/Host checks, default headers.
|
|
||||||
|
|
||||||
The app binds to 127.0.0.1, so the attacker is not a remote client but another page
|
|
||||||
in the user's browser (concept §15, "Local web attack"): any site can issue requests
|
|
||||||
to ``http://127.0.0.1:8000`` and can embed ``<img src=...>`` against media endpoints.
|
|
||||||
The defenses stack, because each one alone has a hole:
|
|
||||||
|
|
||||||
* **Host** must be a loopback name — a DNS rebinding host that resolves to 127.0.0.1
|
|
||||||
passes the browser's origin rules but not this check.
|
|
||||||
* **Origin**, when the browser sends one, must be this exact origin (scheme, host,
|
|
||||||
port). There is no CORS middleware at all, so a foreign page can never *read* a
|
|
||||||
response even if it manages to send a request.
|
|
||||||
* **Sec-Fetch-Site** rejects cross-site loads that carry no Origin, which is what an
|
|
||||||
``<img>`` or ``<script>`` against a media endpoint looks like.
|
|
||||||
* A **session cookie** (``SameSite=Strict``, ``HttpOnly``) is required by every
|
|
||||||
``/api/v1`` route except liveness/readiness and the bootstrap itself. Strict means
|
|
||||||
the browser never attaches it to a request another site initiated.
|
|
||||||
* A **CSRF token** must be echoed in a header on every mutation. It is handed out
|
|
||||||
only in the bootstrap response body, which a foreign page cannot read (no CORS) —
|
|
||||||
so possessing it proves the caller is same-origin.
|
|
||||||
|
|
||||||
``evaluate`` is a pure function over the request metadata: the whole policy is one
|
|
||||||
table that a unit test can enumerate, and the middleware only applies its verdict.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import secrets
|
|
||||||
from collections.abc import Mapping
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from urllib.parse import urlsplit
|
|
||||||
|
|
||||||
from starlette.datastructures import Headers, MutableHeaders
|
|
||||||
from starlette.responses import JSONResponse
|
|
||||||
|
|
||||||
SESSION_COOKIE = "pp_session"
|
|
||||||
CSRF_HEADER = "x-csrf-token"
|
|
||||||
API_PREFIX = "/api/v1"
|
|
||||||
SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
|
|
||||||
# Reachable without a session: liveness/readiness (an orchestrator has no cookie)
|
|
||||||
# and the bootstrap that issues the session in the first place.
|
|
||||||
PUBLIC_PATHS = frozenset(
|
|
||||||
{f"{API_PREFIX}/health/live", f"{API_PREFIX}/health/ready", f"{API_PREFIX}/session"}
|
|
||||||
)
|
|
||||||
LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "::1", "[::1]"})
|
|
||||||
|
|
||||||
# Applied to every response. No inline script/style is used by the frontend, so the
|
|
||||||
# policy can stay strict; `frame-ancestors 'none'` and CORP keep other pages from
|
|
||||||
# embedding the app or its thumbnails.
|
|
||||||
DEFAULT_HEADERS = {
|
|
||||||
"x-content-type-options": "nosniff",
|
|
||||||
"x-frame-options": "DENY",
|
|
||||||
"referrer-policy": "no-referrer",
|
|
||||||
"cross-origin-resource-policy": "same-origin",
|
|
||||||
"cross-origin-opener-policy": "same-origin",
|
|
||||||
"content-security-policy": (
|
|
||||||
"default-src 'self'; img-src 'self' data:; style-src 'self'; script-src 'self'; "
|
|
||||||
"connect-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
|
|
||||||
),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class Session:
|
|
||||||
"""One process, one session. A local app has exactly one user; a session store
|
|
||||||
would be bookkeeping without a second subject to distinguish.
|
|
||||||
|
|
||||||
ponytail: per-session rows if the app ever serves more than one operator.
|
|
||||||
"""
|
|
||||||
|
|
||||||
id: str
|
|
||||||
csrf_token: str
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def create(cls) -> Session:
|
|
||||||
return cls(secrets.token_urlsafe(32), secrets.token_urlsafe(32))
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class Refusal:
|
|
||||||
status: int
|
|
||||||
code: str
|
|
||||||
message: str
|
|
||||||
|
|
||||||
|
|
||||||
def split_host(value: str) -> tuple[str, str]:
|
|
||||||
"""``"127.0.0.1:8000"`` -> ``("127.0.0.1", "8000")``; bracketed IPv6 aware."""
|
|
||||||
value = value.strip()
|
|
||||||
if value.startswith("["):
|
|
||||||
host, _, port = value.partition("]")
|
|
||||||
return host + "]", port.lstrip(":")
|
|
||||||
host, _, port = value.partition(":")
|
|
||||||
return host, port
|
|
||||||
|
|
||||||
|
|
||||||
def evaluate(
|
|
||||||
*,
|
|
||||||
method: str,
|
|
||||||
path: str,
|
|
||||||
headers: Mapping[str, str],
|
|
||||||
session: Session,
|
|
||||||
allowed_hosts: frozenset[str] = LOOPBACK_HOSTS,
|
|
||||||
max_request_bytes: int,
|
|
||||||
) -> Refusal | None:
|
|
||||||
"""Why this request must be refused, or ``None`` when it may proceed."""
|
|
||||||
host_header = headers.get("host", "")
|
|
||||||
host, port = split_host(host_header)
|
|
||||||
if host.lower() not in allowed_hosts:
|
|
||||||
return Refusal(403, "host_not_allowed", "request host is not a local address")
|
|
||||||
|
|
||||||
origin = headers.get("origin")
|
|
||||||
if origin is not None and origin != "":
|
|
||||||
parts = urlsplit(origin)
|
|
||||||
origin_host, origin_port = split_host(parts.netloc)
|
|
||||||
if (
|
|
||||||
parts.scheme not in ("http", "https")
|
|
||||||
or origin_host.lower() not in allowed_hosts
|
|
||||||
or origin_port != port
|
|
||||||
):
|
|
||||||
return Refusal(403, "origin_not_allowed", "request origin is not this application")
|
|
||||||
|
|
||||||
# Absent means a non-browser client; "none" is a user-initiated navigation.
|
|
||||||
fetch_site = headers.get("sec-fetch-site")
|
|
||||||
if fetch_site is not None and fetch_site not in ("same-origin", "none"):
|
|
||||||
return Refusal(403, "cross_site_blocked", "cross-site requests are not accepted")
|
|
||||||
|
|
||||||
length = headers.get("content-length")
|
|
||||||
if length and length.isdigit() and int(length) > max_request_bytes:
|
|
||||||
return Refusal(413, "payload_too_large", "request body exceeds the configured limit")
|
|
||||||
|
|
||||||
protected = path.startswith(API_PREFIX) and path not in PUBLIC_PATHS
|
|
||||||
if not protected:
|
|
||||||
return None
|
|
||||||
|
|
||||||
if headers.get("cookie-session") != session.id:
|
|
||||||
return Refusal(401, "unauthenticated", "a valid application session is required")
|
|
||||||
if method.upper() not in SAFE_METHODS and headers.get(CSRF_HEADER) != session.csrf_token:
|
|
||||||
return Refusal(403, "csrf_failed", "missing or invalid CSRF token")
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
class SecurityMiddleware:
|
|
||||||
"""Pure-ASGI so the SSE stream keeps streaming (BaseHTTPMiddleware buffers)."""
|
|
||||||
|
|
||||||
def __init__(self, app, *, session: Session, config) -> None:
|
|
||||||
self.app = app
|
|
||||||
self.session = session
|
|
||||||
self.max_request_bytes = config.max_request_bytes
|
|
||||||
self.allowed_hosts = frozenset(LOOPBACK_HOSTS | {str(config.host).lower()})
|
|
||||||
|
|
||||||
async def __call__(self, scope, receive, send) -> None:
|
|
||||||
if scope["type"] != "http":
|
|
||||||
await self.app(scope, receive, send)
|
|
||||||
return
|
|
||||||
headers = Headers(scope=scope)
|
|
||||||
# The cookie is read here and handed to the pure policy as one value, so the
|
|
||||||
# policy never has to parse a Cookie header.
|
|
||||||
lookup = dict(headers)
|
|
||||||
lookup["cookie-session"] = _cookie(headers.get("cookie", ""), SESSION_COOKIE)
|
|
||||||
refusal = evaluate(
|
|
||||||
method=scope.get("method", "GET"),
|
|
||||||
path=scope.get("path", "/"),
|
|
||||||
headers=lookup,
|
|
||||||
session=self.session,
|
|
||||||
allowed_hosts=self.allowed_hosts,
|
|
||||||
max_request_bytes=self.max_request_bytes,
|
|
||||||
)
|
|
||||||
if refusal is not None:
|
|
||||||
response = JSONResponse(
|
|
||||||
status_code=refusal.status,
|
|
||||||
content={"error": {"code": refusal.code, "message": refusal.message}},
|
|
||||||
headers=DEFAULT_HEADERS,
|
|
||||||
)
|
|
||||||
await response(scope, receive, send)
|
|
||||||
return
|
|
||||||
|
|
||||||
async def send_with_headers(message):
|
|
||||||
if message["type"] == "http.response.start":
|
|
||||||
out = MutableHeaders(scope=message)
|
|
||||||
for name, value in DEFAULT_HEADERS.items():
|
|
||||||
out.setdefault(name, value)
|
|
||||||
await send(message)
|
|
||||||
|
|
||||||
await self.app(scope, receive, send_with_headers)
|
|
||||||
|
|
||||||
|
|
||||||
def _cookie(header: str, name: str) -> str:
|
|
||||||
for part in header.split(";"):
|
|
||||||
key, _, value = part.strip().partition("=")
|
|
||||||
if key == name:
|
|
||||||
return value
|
|
||||||
return ""
|
|
||||||
@@ -30,10 +30,6 @@ class Config(BaseModel):
|
|||||||
log_level: str = "INFO"
|
log_level: str = "INFO"
|
||||||
log_format: str = "json" # "json" or "text"
|
log_format: str = "json" # "json" or "text"
|
||||||
|
|
||||||
# Largest request body the API accepts. Every endpoint takes small JSON commands;
|
|
||||||
# anything larger is a mistake or an attempt to exhaust memory (US07-02).
|
|
||||||
max_request_bytes: int = 1_048_576
|
|
||||||
|
|
||||||
# Library boundary for path validation (os.pathsep-separated in the env var).
|
# Library boundary for path validation (os.pathsep-separated in the env var).
|
||||||
library_roots: tuple[Path, ...] = ()
|
library_roots: tuple[Path, ...] = ()
|
||||||
thumbnail_cache_quota_bytes: int = 500_000_000
|
thumbnail_cache_quota_bytes: int = 500_000_000
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
"""Domain job handlers: safety scoring, content analysis, uploads, archive
|
"""Domain job handlers: safety scoring, content analysis, uploads, archive
|
||||||
transfers, restores (US02-06, US05-02, US06-02, US06-04).
|
transfers (US02-06, US05-02, US06-02).
|
||||||
|
|
||||||
Importing this module registers the ``safety_score``, ``analysis``,
|
Importing this module registers the ``safety_score``, ``analysis``,
|
||||||
``upload_batch``, ``archive_plan``, and ``restore_plan`` job types so the generic
|
``upload_batch``, and ``archive_plan`` job types so the generic worker can run them
|
||||||
worker can run them per item. Each handler delegates to its service, which owns the real work and the
|
per item. Each handler delegates to its service, which owns the real work and the
|
||||||
privacy gate. Handlers are idempotent: re-scoring or re-analyzing one asset is safe
|
privacy gate. Handlers are idempotent: re-scoring or re-analyzing one asset is safe
|
||||||
after an interrupted attempt, an upload batch refuses to re-run an attempt whose
|
after an interrupted attempt, an upload batch refuses to re-run an attempt whose
|
||||||
outcome is unknown, and an archive plan skips items it already completed.
|
outcome is unknown, and an archive plan skips items it already completed.
|
||||||
@@ -20,7 +20,6 @@ SAFETY_SCORE = "safety_score"
|
|||||||
ANALYSIS = "analysis"
|
ANALYSIS = "analysis"
|
||||||
UPLOAD_BATCH = "upload_batch"
|
UPLOAD_BATCH = "upload_batch"
|
||||||
ARCHIVE_PLAN = "archive_plan"
|
ARCHIVE_PLAN = "archive_plan"
|
||||||
RESTORE_PLAN = "restore_plan"
|
|
||||||
# Both mutate the library's metadata/derived state; one at a time (concept §one job).
|
# Both mutate the library's metadata/derived state; one at a time (concept §one job).
|
||||||
LIBRARY_WRITE_LOCK = "library_write"
|
LIBRARY_WRITE_LOCK = "library_write"
|
||||||
# The uploader lane: one album batch at a time (concept §16).
|
# The uploader lane: one album batch at a time (concept §16).
|
||||||
@@ -38,8 +37,7 @@ def _safety_score_item(asset_id: str, ctx: JobContext) -> None:
|
|||||||
def _analysis_item(asset_id: str, ctx: JobContext) -> None:
|
def _analysis_item(asset_id: str, ctx: JobContext) -> None:
|
||||||
from photo_pipeline.services.analysis import AnalysisService
|
from photo_pipeline.services.analysis import AnalysisService
|
||||||
|
|
||||||
roots = tuple(getattr(ctx.config, "library_roots", ()) or ())
|
AnalysisService(ctx.session_factory).run([asset_id])
|
||||||
AnalysisService(ctx.session_factory, library_roots=roots).run([asset_id])
|
|
||||||
|
|
||||||
|
|
||||||
def _upload_batch_item(batch_id: str, ctx: JobContext) -> None:
|
def _upload_batch_item(batch_id: str, ctx: JobContext) -> None:
|
||||||
@@ -72,22 +70,7 @@ def _archive_plan_item(plan_id: str, ctx: JobContext) -> None:
|
|||||||
raise RuntimeError(f"archive plan {plan_id}: {result['failed']} item(s) failed")
|
raise RuntimeError(f"archive plan {plan_id}: {result['failed']} item(s) failed")
|
||||||
|
|
||||||
|
|
||||||
def _restore_plan_item(plan_id: str, ctx: JobContext) -> None:
|
|
||||||
"""One item = one restore plan. A restore removes nothing, so an item failure
|
|
||||||
simply leaves that asset archived (US06-04)."""
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
from photo_pipeline.services.restores import RestoreService
|
|
||||||
|
|
||||||
config = ctx.config if ctx.config is not None else Config.from_env()
|
|
||||||
result = RestoreService(ctx.session_factory, config=config).apply(
|
|
||||||
plan_id, worker_id=ctx.worker_id
|
|
||||||
)
|
|
||||||
if result["failed"]:
|
|
||||||
raise RuntimeError(f"restore plan {plan_id}: {result['failed']} item(s) failed")
|
|
||||||
|
|
||||||
|
|
||||||
register(SAFETY_SCORE, _safety_score_item)
|
register(SAFETY_SCORE, _safety_score_item)
|
||||||
register(ANALYSIS, _analysis_item)
|
register(ANALYSIS, _analysis_item)
|
||||||
register(UPLOAD_BATCH, _upload_batch_item)
|
register(UPLOAD_BATCH, _upload_batch_item)
|
||||||
register(ARCHIVE_PLAN, _archive_plan_item)
|
register(ARCHIVE_PLAN, _archive_plan_item)
|
||||||
register(RESTORE_PLAN, _restore_plan_item)
|
|
||||||
|
|||||||
@@ -66,8 +66,6 @@ class ArchivePlan(Base):
|
|||||||
# The preflight token this plan was approved against; re-verified before apply.
|
# The preflight token this plan was approved against; re-verified before apply.
|
||||||
token: Mapped[str] = mapped_column(String, nullable=False)
|
token: Mapped[str] = mapped_column(String, nullable=False)
|
||||||
albums: Mapped[str | None] = mapped_column(String) # JSON array
|
albums: Mapped[str | None] = mapped_column(String) # JSON array
|
||||||
# archive | restore — the same journal read in the opposite direction (US06-04).
|
|
||||||
direction: Mapped[str] = mapped_column(String, nullable=False, default="archive")
|
|
||||||
|
|
||||||
# planned | applying | complete | failed
|
# planned | applying | complete | failed
|
||||||
state: Mapped[str] = mapped_column(String, nullable=False, default="planned")
|
state: Mapped[str] = mapped_column(String, nullable=False, default="planned")
|
||||||
@@ -102,9 +100,6 @@ class ArchiveOperation(Base):
|
|||||||
album: Mapped[str] = mapped_column(String, nullable=False)
|
album: Mapped[str] = mapped_column(String, nullable=False)
|
||||||
asset_id: Mapped[str] = mapped_column(ForeignKey("assets.id"), nullable=False, index=True)
|
asset_id: Mapped[str] = mapped_column(ForeignKey("assets.id"), nullable=False, index=True)
|
||||||
|
|
||||||
# archive: library → medium. restore: medium → library (US06-04). ``source_path``
|
|
||||||
# and ``destination_path`` always mean "from" and "to" for this direction.
|
|
||||||
direction: Mapped[str] = mapped_column(String, nullable=False, default="archive")
|
|
||||||
source_path: Mapped[str] = mapped_column(String, nullable=False)
|
source_path: Mapped[str] = mapped_column(String, nullable=False)
|
||||||
destination_path: Mapped[str] = mapped_column(String, nullable=False)
|
destination_path: Mapped[str] = mapped_column(String, nullable=False)
|
||||||
# Relative to the location root, because the medium can be mounted elsewhere.
|
# Relative to the location root, because the medium can be mounted elsewhere.
|
||||||
|
|||||||
@@ -43,10 +43,6 @@ class Asset(Base):
|
|||||||
# link is written and read by the archive service (US06-02).
|
# link is written and read by the archive service (US06-02).
|
||||||
archive_location_id: Mapped[str | None] = mapped_column(String)
|
archive_location_id: Mapped[str | None] = mapped_column(String)
|
||||||
archive_path: Mapped[str | None] = mapped_column(String)
|
archive_path: Mapped[str | None] = mapped_column(String)
|
||||||
# Set when the archived copy was proven to hold bytes other than the recorded
|
|
||||||
# ones (US06-04). Restore refuses such an asset instead of accepting a different
|
|
||||||
# file; cleared as soon as a verification matches again.
|
|
||||||
archive_divergent_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
|
||||||
# Duplicate canonical link: NULL when the asset is itself canonical or undecided.
|
# Duplicate canonical link: NULL when the asset is itself canonical or undecided.
|
||||||
canonical_asset_id: Mapped[str | None] = mapped_column(ForeignKey("assets.id"))
|
canonical_asset_id: Mapped[str | None] = mapped_column(ForeignKey("assets.id"))
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String, func
|
from sqlalchemy import DateTime, ForeignKey, Integer, String, func
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
from photo_pipeline.db import Base
|
from photo_pipeline.db import Base
|
||||||
@@ -29,9 +29,6 @@ class Thumbnail(Base):
|
|||||||
width: Mapped[int | None] = mapped_column(Integer)
|
width: Mapped[int | None] = mapped_column(Integer)
|
||||||
height: Mapped[int | None] = mapped_column(Integer)
|
height: Mapped[int | None] = mapped_column(Integer)
|
||||||
format: Mapped[str | None] = mapped_column(String)
|
format: Mapped[str | None] = mapped_column(String)
|
||||||
# Durable comparison evidence for an archived asset: never evicted by the LRU
|
|
||||||
# quota, because the original may be on a medium that is no longer reachable.
|
|
||||||
protected: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -49,29 +49,6 @@ def resolve_within(root: Path, path: os.PathLike | str) -> Path:
|
|||||||
return resolved
|
return resolved
|
||||||
|
|
||||||
|
|
||||||
def resolve_in_roots(roots: Iterable[os.PathLike | str], path: os.PathLike | str) -> Path:
|
|
||||||
"""The resolved path, proven to be inside one of ``roots`` and not excluded.
|
|
||||||
|
|
||||||
Callers must use the **returned** path for whatever they do next: validating one
|
|
||||||
name and then opening another is the symlink race this exists to close (US07-02).
|
|
||||||
The message names no path — it reaches API responses.
|
|
||||||
|
|
||||||
With no roots configured there is no boundary to check; that is a property of the
|
|
||||||
configuration, not permission granted to this call.
|
|
||||||
"""
|
|
||||||
if is_excluded(path):
|
|
||||||
raise PathPolicyError("path is inside an excluded (_IGNORE/) tree")
|
|
||||||
roots = list(roots)
|
|
||||||
if not roots:
|
|
||||||
return Path(path)
|
|
||||||
for root in roots:
|
|
||||||
try:
|
|
||||||
return resolve_within(Path(root), path)
|
|
||||||
except PathPolicyError:
|
|
||||||
continue
|
|
||||||
raise PathPolicyError("path is outside the configured library roots")
|
|
||||||
|
|
||||||
|
|
||||||
def iter_supported_files(root: os.PathLike | str) -> Iterator[Path]:
|
def iter_supported_files(root: os.PathLike | str) -> Iterator[Path]:
|
||||||
"""Yield supported, non-excluded files under ``root`` in deterministic order.
|
"""Yield supported, non-excluded files under ``root`` in deterministic order.
|
||||||
|
|
||||||
|
|||||||
@@ -25,7 +25,6 @@ from typing import Protocol
|
|||||||
from sqlalchemy import func, select
|
from sqlalchemy import func, select
|
||||||
from sqlalchemy.orm import sessionmaker
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
from photo_pipeline import path_policy
|
|
||||||
from photo_pipeline.integrations import exiftool
|
from photo_pipeline.integrations import exiftool
|
||||||
from photo_pipeline.models import AnalysisResult, Asset, SafetyReview
|
from photo_pipeline.models import AnalysisResult, Asset, SafetyReview
|
||||||
from photo_pipeline.services import hashing
|
from photo_pipeline.services import hashing
|
||||||
@@ -60,16 +59,9 @@ def _now() -> datetime:
|
|||||||
|
|
||||||
|
|
||||||
class AnalysisService:
|
class AnalysisService:
|
||||||
def __init__(
|
def __init__(self, session_factory: sessionmaker, *, provider: VisionProvider | None = None) -> None:
|
||||||
self,
|
|
||||||
session_factory: sessionmaker,
|
|
||||||
*,
|
|
||||||
provider: VisionProvider | None = None,
|
|
||||||
library_roots: tuple = (),
|
|
||||||
) -> None:
|
|
||||||
self._session_factory = session_factory
|
self._session_factory = session_factory
|
||||||
self._provider = provider
|
self._provider = provider
|
||||||
self._roots = tuple(library_roots)
|
|
||||||
|
|
||||||
def _sfw_asset_ids(self, session) -> set[str]:
|
def _sfw_asset_ids(self, session) -> set[str]:
|
||||||
"""Asset ids whose latest safety decision is ``sfw`` — the ONLY assets that
|
"""Asset ids whose latest safety decision is ``sfw`` — the ONLY assets that
|
||||||
@@ -144,18 +136,6 @@ class AnalysisService:
|
|||||||
if not path:
|
if not path:
|
||||||
skipped += 1
|
skipped += 1
|
||||||
continue
|
continue
|
||||||
# Second gate, at the moment of use: the database says where the file
|
|
||||||
# was, the filesystem decides what that name means now. A link swapped
|
|
||||||
# under an asset after the scan would otherwise send bytes from outside
|
|
||||||
# the library — the one place that leaves this machine (US07-02).
|
|
||||||
try:
|
|
||||||
path = str(path_policy.resolve_in_roots(self._roots, path))
|
|
||||||
except path_policy.PathPolicyError as error:
|
|
||||||
self._store(
|
|
||||||
asset_id, status="error", result=None, error=str(error), tokens=0, raw=""
|
|
||||||
)
|
|
||||||
errors += 1
|
|
||||||
continue
|
|
||||||
try:
|
try:
|
||||||
result = provider.analyze(path, album_hint=_album_hint(path))
|
result = provider.analyze(path, album_hint=_album_hint(path))
|
||||||
except Exception as error: # provider/validation failure is per-asset
|
except Exception as error: # provider/validation failure is per-asset
|
||||||
|
|||||||
@@ -15,11 +15,6 @@ planned → transferring → verified → removing → complete
|
|||||||
↘ ↘ ↘ failed
|
↘ ↘ ↘ failed
|
||||||
```
|
```
|
||||||
|
|
||||||
A restore (US06-04) uses the same rows with ``direction='restore'``: it copies from
|
|
||||||
the medium back into the library and removes nothing, so it goes ``verified →
|
|
||||||
complete`` directly. ``source_path``/``destination_path`` always mean "from"/"to",
|
|
||||||
which is why the evidence table below needs no direction of its own.
|
|
||||||
|
|
||||||
- ``transferring`` — intent recorded; a temporary copy may exist, the destination
|
- ``transferring`` — intent recorded; a temporary copy may exist, the destination
|
||||||
may or may not have been published. Nothing has been removed.
|
may or may not have been published. Nothing has been removed.
|
||||||
- ``verified`` — the archived bytes exist at their final path, hash exactly as
|
- ``verified`` — the archived bytes exist at their final path, hash exactly as
|
||||||
@@ -78,22 +73,10 @@ ALLOWED_TRANSITIONS = {
|
|||||||
ArchiveState.FAILED: {ArchiveState.PLANNED, ArchiveState.TRANSFERRING},
|
ArchiveState.FAILED: {ArchiveState.PLANNED, ArchiveState.TRANSFERRING},
|
||||||
}
|
}
|
||||||
|
|
||||||
# A restore removes nothing, so it has no ``removing`` step: a verified published
|
|
||||||
# copy is the whole job (US06-04). Keeping this as a separate table means the
|
|
||||||
# archive direction still cannot reach ``complete`` without going through removal.
|
|
||||||
RESTORE_TRANSITIONS = {
|
|
||||||
**ALLOWED_TRANSITIONS,
|
|
||||||
ArchiveState.VERIFIED: {ArchiveState.COMPLETE, ArchiveState.FAILED},
|
|
||||||
}
|
|
||||||
|
|
||||||
TERMINAL_STATES = frozenset({ArchiveState.COMPLETE})
|
TERMINAL_STATES = frozenset({ArchiveState.COMPLETE})
|
||||||
# States where this item may already have touched the filesystem.
|
# States where this item may already have touched the filesystem.
|
||||||
UNSAFE_STATES = frozenset({ArchiveState.TRANSFERRING, ArchiveState.VERIFIED, ArchiveState.REMOVING})
|
UNSAFE_STATES = frozenset({ArchiveState.TRANSFERRING, ArchiveState.VERIFIED, ArchiveState.REMOVING})
|
||||||
|
|
||||||
# Which way the bytes move. Same rows, same evidence table, opposite direction.
|
|
||||||
ARCHIVE = "archive"
|
|
||||||
RESTORE = "restore"
|
|
||||||
|
|
||||||
RESUMABLE = "resumable"
|
RESUMABLE = "resumable"
|
||||||
FORWARD = "forward"
|
FORWARD = "forward"
|
||||||
MANUAL = "manual"
|
MANUAL = "manual"
|
||||||
@@ -111,9 +94,8 @@ class JournalConflict(JournalError):
|
|||||||
"""Fencing check failed; a newer owner has taken over this operation."""
|
"""Fencing check failed; a newer owner has taken over this operation."""
|
||||||
|
|
||||||
|
|
||||||
def can_transition(current: str, target: str, direction: str = ARCHIVE) -> bool:
|
def can_transition(current: str, target: str) -> bool:
|
||||||
table = RESTORE_TRANSITIONS if direction == RESTORE else ALLOWED_TRANSITIONS
|
return target in ALLOWED_TRANSITIONS.get(current, set())
|
||||||
return target in table.get(current, set())
|
|
||||||
|
|
||||||
|
|
||||||
def _now() -> datetime:
|
def _now() -> datetime:
|
||||||
@@ -137,7 +119,7 @@ class ArchiveJournal:
|
|||||||
if row.journal_state in TERMINAL_STATES:
|
if row.journal_state in TERMINAL_STATES:
|
||||||
raise InvalidTransition(f"{row.journal_state} is terminal")
|
raise InvalidTransition(f"{row.journal_state} is terminal")
|
||||||
if row.journal_state != ArchiveState.TRANSFERRING and not can_transition(
|
if row.journal_state != ArchiveState.TRANSFERRING and not can_transition(
|
||||||
row.journal_state, ArchiveState.TRANSFERRING, row.direction
|
row.journal_state, ArchiveState.TRANSFERRING
|
||||||
):
|
):
|
||||||
raise InvalidTransition(f"{row.journal_state} -> {ArchiveState.TRANSFERRING}")
|
raise InvalidTransition(f"{row.journal_state} -> {ArchiveState.TRANSFERRING}")
|
||||||
if row.journal_state != ArchiveState.TRANSFERRING:
|
if row.journal_state != ArchiveState.TRANSFERRING:
|
||||||
@@ -178,7 +160,7 @@ class ArchiveJournal:
|
|||||||
if row.journal_state == target:
|
if row.journal_state == target:
|
||||||
session.commit()
|
session.commit()
|
||||||
return _operation_dict(row) # idempotent
|
return _operation_dict(row) # idempotent
|
||||||
if not can_transition(row.journal_state, target, row.direction):
|
if not can_transition(row.journal_state, target):
|
||||||
raise InvalidTransition(f"{row.journal_state} -> {target}")
|
raise InvalidTransition(f"{row.journal_state} -> {target}")
|
||||||
|
|
||||||
row.journal_state = target
|
row.journal_state = target
|
||||||
@@ -210,18 +192,16 @@ class ArchiveJournal:
|
|||||||
)
|
)
|
||||||
return [_operation_dict(row) for row in rows]
|
return [_operation_dict(row) for row in rows]
|
||||||
|
|
||||||
def incomplete(self, *, direction: str | None = None) -> list[dict]:
|
def incomplete(self) -> list[dict]:
|
||||||
"""Every operation left in a non-terminal, non-planned state — the work a
|
"""Every operation left in a non-terminal, non-planned state — the work a
|
||||||
restart has to reason about. Without ``direction`` this spans archives and
|
restart has to reason about."""
|
||||||
restores, because either one half-done blocks the other."""
|
|
||||||
with self._session_factory() as session:
|
with self._session_factory() as session:
|
||||||
stmt = select(ArchiveOperation).where(
|
rows = session.scalars(
|
||||||
|
select(ArchiveOperation)
|
||||||
|
.where(
|
||||||
ArchiveOperation.journal_state.not_in([*TERMINAL_STATES, ArchiveState.PLANNED])
|
ArchiveOperation.journal_state.not_in([*TERMINAL_STATES, ArchiveState.PLANNED])
|
||||||
)
|
)
|
||||||
if direction is not None:
|
.order_by(ArchiveOperation.plan_id, ArchiveOperation.sequence)
|
||||||
stmt = stmt.where(ArchiveOperation.direction == direction)
|
|
||||||
rows = session.scalars(
|
|
||||||
stmt.order_by(ArchiveOperation.plan_id, ArchiveOperation.sequence)
|
|
||||||
)
|
)
|
||||||
return [_operation_dict(row) for row in rows]
|
return [_operation_dict(row) for row in rows]
|
||||||
|
|
||||||
@@ -251,7 +231,6 @@ class ArchiveJournal:
|
|||||||
return {
|
return {
|
||||||
"operation_id": operation_id,
|
"operation_id": operation_id,
|
||||||
"plan_id": row["plan_id"],
|
"plan_id": row["plan_id"],
|
||||||
"direction": row["direction"],
|
|
||||||
"album": row["album"],
|
"album": row["album"],
|
||||||
"asset_id": row["asset_id"],
|
"asset_id": row["asset_id"],
|
||||||
"source_path": row["source_path"],
|
"source_path": row["source_path"],
|
||||||
@@ -264,8 +243,8 @@ class ArchiveJournal:
|
|||||||
"destination_matches": destination_matches,
|
"destination_matches": destination_matches,
|
||||||
}
|
}
|
||||||
|
|
||||||
def classify_all(self, *, direction: str | None = None) -> list[dict]:
|
def classify_all(self) -> list[dict]:
|
||||||
return [self.classify(row["id"]) for row in self.incomplete(direction=direction)]
|
return [self.classify(row["id"]) for row in self.incomplete()]
|
||||||
|
|
||||||
def blocks_mutation(self) -> bool:
|
def blocks_mutation(self) -> bool:
|
||||||
"""True when any item may have the library half-archived."""
|
"""True when any item may have the library half-archived."""
|
||||||
@@ -338,7 +317,6 @@ def _operation_dict(row: ArchiveOperation) -> dict:
|
|||||||
return {
|
return {
|
||||||
"id": row.id,
|
"id": row.id,
|
||||||
"plan_id": row.plan_id,
|
"plan_id": row.plan_id,
|
||||||
"direction": row.direction,
|
|
||||||
"sequence": row.sequence,
|
"sequence": row.sequence,
|
||||||
"album": row.album,
|
"album": row.album,
|
||||||
"asset_id": row.asset_id,
|
"asset_id": row.asset_id,
|
||||||
|
|||||||
@@ -54,17 +54,14 @@ from sqlalchemy.orm import sessionmaker
|
|||||||
from photo_pipeline.config import Config
|
from photo_pipeline.config import Config
|
||||||
from photo_pipeline.models import ArchiveLocation, ArchiveOperation, ArchivePlan, Asset, AssetPath
|
from photo_pipeline.models import ArchiveLocation, ArchiveOperation, ArchivePlan, Asset, AssetPath
|
||||||
from photo_pipeline.services.archive_journal import (
|
from photo_pipeline.services.archive_journal import (
|
||||||
ARCHIVE,
|
|
||||||
MANUAL,
|
MANUAL,
|
||||||
RESUMABLE,
|
RESUMABLE,
|
||||||
ArchiveJournal,
|
ArchiveJournal,
|
||||||
ArchiveState,
|
ArchiveState,
|
||||||
)
|
)
|
||||||
from photo_pipeline.services.archives import MARKER_NAME, ArchiveError, ArchiveService
|
from photo_pipeline.services.archives import MARKER_NAME, ArchiveError, ArchiveService
|
||||||
from photo_pipeline.services.duplicates import DuplicateService
|
|
||||||
from photo_pipeline.services.hashing import sha256_file
|
from photo_pipeline.services.hashing import sha256_file
|
||||||
from photo_pipeline.services.rename_apply import PreconditionFailed, maybe_fault
|
from photo_pipeline.services.rename_apply import PreconditionFailed, maybe_fault
|
||||||
from photo_pipeline.services.thumbnails import ThumbnailService
|
|
||||||
|
|
||||||
# The per-medium manifest: one JSON line per archived file, appended and fsynced
|
# The per-medium manifest: one JSON line per archived file, appended and fsynced
|
||||||
# before its source is removed. It lives with the bytes so the archive can still be
|
# before its source is removed. It lives with the bytes so the archive can still be
|
||||||
@@ -116,7 +113,6 @@ class ArchiveTransferService:
|
|||||||
location_id=location_id,
|
location_id=location_id,
|
||||||
token=token,
|
token=token,
|
||||||
albums=json.dumps(albums) if albums is not None else None,
|
albums=json.dumps(albums) if albums is not None else None,
|
||||||
direction=ARCHIVE,
|
|
||||||
state="planned",
|
state="planned",
|
||||||
schema_version=MANIFEST_VERSION,
|
schema_version=MANIFEST_VERSION,
|
||||||
asset_count=preflight["totals"]["assets"],
|
asset_count=preflight["totals"]["assets"],
|
||||||
@@ -134,7 +130,6 @@ class ArchiveTransferService:
|
|||||||
ArchiveOperation(
|
ArchiveOperation(
|
||||||
id=str(uuid.uuid4()),
|
id=str(uuid.uuid4()),
|
||||||
plan_id=plan_id,
|
plan_id=plan_id,
|
||||||
direction=ARCHIVE,
|
|
||||||
sequence=sequence,
|
sequence=sequence,
|
||||||
album=album["album"],
|
album=album["album"],
|
||||||
asset_id=asset["asset_id"],
|
asset_id=asset["asset_id"],
|
||||||
@@ -164,11 +159,7 @@ class ArchiveTransferService:
|
|||||||
|
|
||||||
def list(self) -> list[dict]:
|
def list(self) -> list[dict]:
|
||||||
with self._session_factory() as session:
|
with self._session_factory() as session:
|
||||||
rows = session.scalars(
|
rows = session.scalars(select(ArchivePlan).order_by(ArchivePlan.created_at))
|
||||||
select(ArchivePlan)
|
|
||||||
.where(ArchivePlan.direction == ARCHIVE)
|
|
||||||
.order_by(ArchivePlan.created_at)
|
|
||||||
)
|
|
||||||
return [_plan_dict(row) for row in rows]
|
return [_plan_dict(row) for row in rows]
|
||||||
|
|
||||||
# ── apply ─────────────────────────────────────────────────────────────────
|
# ── apply ─────────────────────────────────────────────────────────────────
|
||||||
@@ -269,7 +260,7 @@ class ArchiveTransferService:
|
|||||||
if same_filesystem:
|
if same_filesystem:
|
||||||
os.rename(source, destination)
|
os.rename(source, destination)
|
||||||
else:
|
else:
|
||||||
copy_verify_publish(source, destination, operation["expected_sha256"])
|
self._copy_and_publish(operation, source, destination)
|
||||||
_fsync_dir(destination.parent)
|
_fsync_dir(destination.parent)
|
||||||
|
|
||||||
# 4. The published file is the archive only once it hashes as recorded.
|
# 4. The published file is the archive only once it hashes as recorded.
|
||||||
@@ -289,6 +280,27 @@ class ArchiveTransferService:
|
|||||||
# 5. Only now may the active source go.
|
# 5. Only now may the active source go.
|
||||||
self._finish(self.journal.get(operation["id"]), location, token=token, worker_id=worker_id)
|
self._finish(self.journal.get(operation["id"]), location, token=token, worker_id=worker_id)
|
||||||
|
|
||||||
|
def _copy_and_publish(self, operation: dict, source: Path, destination: Path) -> None:
|
||||||
|
"""Cross-filesystem: copy to a temporary file beside the destination, prove
|
||||||
|
its bytes, then publish it atomically. The source is still untouched."""
|
||||||
|
temp = destination.with_name(f"{TEMP_PREFIX}{uuid.uuid4().hex}{TEMP_SUFFIX}")
|
||||||
|
try:
|
||||||
|
with open(source, "rb") as src, open(temp, "wb") as out:
|
||||||
|
shutil.copyfileobj(src, out, 1024 * 1024)
|
||||||
|
out.flush()
|
||||||
|
os.fsync(out.fileno())
|
||||||
|
if sha256_file(temp) != operation["expected_sha256"]:
|
||||||
|
raise PreconditionFailed("copy_mismatch", f"{source} copied with wrong bytes")
|
||||||
|
if destination.exists():
|
||||||
|
raise PreconditionFailed(
|
||||||
|
"destination_exists", f"{destination} appeared during the transfer"
|
||||||
|
)
|
||||||
|
# ponytail: rename after an exists() check. The archiver lane is single
|
||||||
|
# and local; use O_EXCL/link-based publish if a second writer ever exists.
|
||||||
|
os.rename(temp, destination)
|
||||||
|
finally:
|
||||||
|
temp.unlink(missing_ok=True)
|
||||||
|
|
||||||
def _finish(self, operation: dict, location: dict, *, token: int, worker_id: str) -> None:
|
def _finish(self, operation: dict, location: dict, *, token: int, worker_id: str) -> None:
|
||||||
"""Drive an item whose archive copy is durable through removal and
|
"""Drive an item whose archive copy is durable through removal and
|
||||||
bookkeeping. Every step is idempotent, so recovery may replay it."""
|
bookkeeping. Every step is idempotent, so recovery may replay it."""
|
||||||
@@ -323,7 +335,6 @@ class ArchiveTransferService:
|
|||||||
raise PreconditionFailed(
|
raise PreconditionFailed(
|
||||||
"archive_unverified", f"{destination} is not a verified archive copy"
|
"archive_unverified", f"{destination} is not a verified archive copy"
|
||||||
)
|
)
|
||||||
self._require_evidence(operation["asset_id"], destination)
|
|
||||||
if source.exists():
|
if source.exists():
|
||||||
if source.is_symlink():
|
if source.is_symlink():
|
||||||
raise PreconditionFailed("symlink", f"{source} became a symlink")
|
raise PreconditionFailed("symlink", f"{source} became a symlink")
|
||||||
@@ -366,28 +377,6 @@ class ArchiveTransferService:
|
|||||||
"asset_moved", f"asset {operation['asset_id']} is no longer at {source}"
|
"asset_moved", f"asset {operation['asset_id']} is no longer at {source}"
|
||||||
)
|
)
|
||||||
|
|
||||||
def _require_evidence(self, asset_id: str, source: Path) -> dict:
|
|
||||||
"""Review evidence must be durable before the original goes.
|
|
||||||
|
|
||||||
The perceptual hash keeps the asset in the fuzzy index once its bytes are
|
|
||||||
unreachable, and the protected preview is what duplicate review can still
|
|
||||||
look at. Both are read from the freshly verified archive copy, which holds
|
|
||||||
exactly the bytes being archived. A file that cannot be decoded has neither
|
|
||||||
— recorded, not fatal, since its exact hashes remain — but failing to
|
|
||||||
produce a preview from a decodable original stops the removal (concept §9).
|
|
||||||
"""
|
|
||||||
DuplicateService(self._session_factory).ensure_phash(asset_id, source=source)
|
|
||||||
preview = ThumbnailService(self._session_factory, self._config).ensure_protected(
|
|
||||||
asset_id, source=source
|
|
||||||
)
|
|
||||||
if preview["state"] == "unavailable":
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"preview_unavailable",
|
|
||||||
f"no durable comparison preview for asset {asset_id} "
|
|
||||||
f"({preview['error_code']})",
|
|
||||||
)
|
|
||||||
return preview
|
|
||||||
|
|
||||||
# ── database ──────────────────────────────────────────────────────────────
|
# ── database ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
def _record_archived(self, operation: dict, location: dict, destination: Path) -> None:
|
def _record_archived(self, operation: dict, location: dict, destination: Path) -> None:
|
||||||
@@ -443,7 +432,7 @@ class ArchiveTransferService:
|
|||||||
"""
|
"""
|
||||||
results = {"resumed": 0, "completed": 0, "manual": 0}
|
results = {"resumed": 0, "completed": 0, "manual": 0}
|
||||||
touched: set[str] = set()
|
touched: set[str] = set()
|
||||||
for verdict in self.journal.classify_all(direction=ARCHIVE):
|
for verdict in self.journal.classify_all():
|
||||||
operation = self.journal.get(verdict["operation_id"])
|
operation = self.journal.get(verdict["operation_id"])
|
||||||
touched.add(operation["plan_id"])
|
touched.add(operation["plan_id"])
|
||||||
token = (operation["fencing_token"] or 0) + 1
|
token = (operation["fencing_token"] or 0) + 1
|
||||||
@@ -470,7 +459,7 @@ class ArchiveTransferService:
|
|||||||
return results
|
return results
|
||||||
|
|
||||||
def recovery_status(self) -> dict:
|
def recovery_status(self) -> dict:
|
||||||
verdicts = self.journal.classify_all(direction=ARCHIVE)
|
verdicts = self.journal.classify_all()
|
||||||
return {
|
return {
|
||||||
"operations": verdicts,
|
"operations": verdicts,
|
||||||
"manual": [v for v in verdicts if v["classification"] == MANUAL],
|
"manual": [v for v in verdicts if v["classification"] == MANUAL],
|
||||||
@@ -514,33 +503,6 @@ class ArchiveTransferService:
|
|||||||
# ── module helpers ───────────────────────────────────────────────────────────
|
# ── module helpers ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
def copy_verify_publish(source: Path, destination: Path, expected_sha256: str) -> None:
|
|
||||||
"""Copy to a temporary file beside the destination, prove its bytes, then publish
|
|
||||||
it atomically. The source is never touched, so a failure costs nothing.
|
|
||||||
|
|
||||||
Shared by archiving (library → medium) and restoring (medium → library, US06-04):
|
|
||||||
both need the same promise that a published file is either complete and correct
|
|
||||||
or not there at all.
|
|
||||||
"""
|
|
||||||
temp = destination.with_name(f"{TEMP_PREFIX}{uuid.uuid4().hex}{TEMP_SUFFIX}")
|
|
||||||
try:
|
|
||||||
with open(source, "rb") as src, open(temp, "wb") as out:
|
|
||||||
shutil.copyfileobj(src, out, 1024 * 1024)
|
|
||||||
out.flush()
|
|
||||||
os.fsync(out.fileno())
|
|
||||||
if sha256_file(temp) != expected_sha256:
|
|
||||||
raise PreconditionFailed("copy_mismatch", f"{source} copied with wrong bytes")
|
|
||||||
if destination.exists():
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"destination_exists", f"{destination} appeared during the transfer"
|
|
||||||
)
|
|
||||||
# ponytail: rename after an exists() check. The archiver lane is single and
|
|
||||||
# local; use O_EXCL/link-based publish if a second writer ever exists.
|
|
||||||
os.rename(temp, destination)
|
|
||||||
finally:
|
|
||||||
temp.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
|
|
||||||
def _same_filesystem(source: Path, destination_dir: Path) -> bool:
|
def _same_filesystem(source: Path, destination_dir: Path) -> bool:
|
||||||
"""Proven at run time from the actual devices, never from the plan's preview."""
|
"""Proven at run time from the actual devices, never from the plan's preview."""
|
||||||
try:
|
try:
|
||||||
@@ -631,7 +593,6 @@ def _plan_dict(plan: ArchivePlan) -> dict:
|
|||||||
"id": plan.id,
|
"id": plan.id,
|
||||||
"location_id": plan.location_id,
|
"location_id": plan.location_id,
|
||||||
"token": plan.token,
|
"token": plan.token,
|
||||||
"direction": plan.direction,
|
|
||||||
"albums": json.loads(plan.albums) if plan.albums else None,
|
"albums": json.loads(plan.albums) if plan.albums else None,
|
||||||
"state": plan.state,
|
"state": plan.state,
|
||||||
"schema_version": plan.schema_version,
|
"schema_version": plan.schema_version,
|
||||||
|
|||||||
@@ -28,11 +28,7 @@ Blocker codes: ``no_library_root``, ``location_offline``, ``wrong_volume``,
|
|||||||
``unsafe_destination``, ``destination_not_writable``, ``manifest_unwritable``,
|
``unsafe_destination``, ``destination_not_writable``, ``manifest_unwritable``,
|
||||||
``insufficient_capacity``, ``backup_unavailable``, ``lock_conflict``,
|
``insufficient_capacity``, ``backup_unavailable``, ``lock_conflict``,
|
||||||
``rename_pending``, ``empty_scope``, ``destination_collision``,
|
``rename_pending``, ``empty_scope``, ``destination_collision``,
|
||||||
``upload_unverified``, ``bytes_changed``, ``file_missing``, ``preview_unavailable``.
|
``upload_unverified``, ``bytes_changed``, ``file_missing``.
|
||||||
|
|
||||||
Preflight also *creates* the durable comparison preview of every asset in scope
|
|
||||||
(US06-03): it is the evidence duplicate review falls back on once the original is
|
|
||||||
on a medium that may be offline, so it has to exist before the original leaves.
|
|
||||||
|
|
||||||
Like upload preflight, the confirmation token is *derived* from the report rather
|
Like upload preflight, the confirmation token is *derived* from the report rather
|
||||||
than stored: any change to the scope, the bytes, the destination, or the blockers
|
than stored: any change to the scope, the bytes, the destination, or the blockers
|
||||||
@@ -63,16 +59,14 @@ from photo_pipeline.models import ArchiveLocation, Asset, UploadBatch, UploadIte
|
|||||||
from photo_pipeline.path_policy import PathPolicyError, is_excluded, normalize_root, resolve_within
|
from photo_pipeline.path_policy import PathPolicyError, is_excluded, normalize_root, resolve_within
|
||||||
from photo_pipeline.services.albums import album_label
|
from photo_pipeline.services.albums import album_label
|
||||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
from photo_pipeline.services.archive_journal import ArchiveJournal
|
||||||
from photo_pipeline.services.availability import MARKER_NAME, read_marker as _read_marker
|
|
||||||
from photo_pipeline.services.availability import refresh as refresh_availability
|
|
||||||
from photo_pipeline.services.hashing import sha256_file
|
from photo_pipeline.services.hashing import sha256_file
|
||||||
from photo_pipeline.services.jobs import JobService
|
from photo_pipeline.services.jobs import JobService
|
||||||
from photo_pipeline.services.rename_journal import RenameJournal
|
from photo_pipeline.services.rename_journal import RenameJournal
|
||||||
from photo_pipeline.services.thumbnails import ThumbnailService
|
|
||||||
from photo_pipeline.services.upload_reports import VERIFIED
|
from photo_pipeline.services.upload_reports import VERIFIED
|
||||||
|
|
||||||
PREFLIGHT_VERSION = 1
|
PREFLIGHT_VERSION = 1
|
||||||
TOKEN_PREFIX = f"v{PREFLIGHT_VERSION}"
|
TOKEN_PREFIX = f"v{PREFLIGHT_VERSION}"
|
||||||
|
MARKER_NAME = ".photo-pipeline-archive.json"
|
||||||
MANIFEST_NAME = "archive-manifest.json"
|
MANIFEST_NAME = "archive-manifest.json"
|
||||||
|
|
||||||
# Upload outcomes that prove Immich holds these exact bytes. ``skipped``/``failed``/
|
# Upload outcomes that prove Immich holds these exact bytes. ``skipped``/``failed``/
|
||||||
@@ -169,9 +163,6 @@ class ArchiveService:
|
|||||||
location.capabilities = json.dumps(probe["capabilities"])
|
location.capabilities = json.dumps(probe["capabilities"])
|
||||||
reports.append(self._location_report(location, probe=probe))
|
reports.append(self._location_report(location, probe=probe))
|
||||||
session.commit()
|
session.commit()
|
||||||
# A medium that just appeared or vanished changes what is readable, so the
|
|
||||||
# archived assets are re-derived from the same probe (US06-03).
|
|
||||||
refresh_availability(self._session_factory)
|
|
||||||
return reports
|
return reports
|
||||||
|
|
||||||
# ── preflight ─────────────────────────────────────────────────────────────
|
# ── preflight ─────────────────────────────────────────────────────────────
|
||||||
@@ -208,12 +199,9 @@ class ArchiveService:
|
|||||||
report["blockers"].append(
|
report["blockers"].append(
|
||||||
_issue(
|
_issue(
|
||||||
"insufficient_capacity",
|
"insufficient_capacity",
|
||||||
# The free-space number is deliberately left out: it drifts between
|
|
||||||
# two identical preflights, and the token is a digest of this text,
|
|
||||||
# so quoting it here would invalidate every approval instantly.
|
|
||||||
f"{report['totals']['bytes']} B plus a "
|
f"{report['totals']['bytes']} B plus a "
|
||||||
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit on "
|
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit in "
|
||||||
"the medium",
|
f"{report['capacity']['free_bytes']} B of free space",
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
report["backup"] = self._backup_probe()
|
report["backup"] = self._backup_probe()
|
||||||
@@ -432,10 +420,7 @@ class ArchiveService:
|
|||||||
|
|
||||||
def _album(self, name: str, rows: list[dict], root: Path, *, reachable: bool) -> dict:
|
def _album(self, name: str, rows: list[dict], root: Path, *, reachable: bool) -> dict:
|
||||||
folder = Path(rows[0]["path"]).parent
|
folder = Path(rows[0]["path"]).parent
|
||||||
items = sorted(
|
items = sorted((_item(row) for row in rows), key=lambda item: item["current_path"])
|
||||||
(self._with_preview(_item(row)) for row in rows),
|
|
||||||
key=lambda item: item["current_path"],
|
|
||||||
)
|
|
||||||
blocked = [item for item in items if item["blockers"]]
|
blocked = [item for item in items if item["blockers"]]
|
||||||
blockers: list[dict] = []
|
blockers: list[dict] = []
|
||||||
|
|
||||||
@@ -471,30 +456,6 @@ class ArchiveService:
|
|||||||
"assets": items,
|
"assets": items,
|
||||||
}
|
}
|
||||||
|
|
||||||
def _with_preview(self, item: dict) -> dict:
|
|
||||||
"""Create the durable comparison preview while the original is still here.
|
|
||||||
|
|
||||||
This is the last moment it can be made: once the file is archived and the
|
|
||||||
medium leaves, only the retained preview can answer "is this new photo the
|
|
||||||
same picture?". An original that cannot be decoded at all has no preview to
|
|
||||||
keep — its hashes and metadata stay the evidence — but a preview that fails
|
|
||||||
for any other reason blocks the archive (concept §9).
|
|
||||||
"""
|
|
||||||
preview = self._previews().ensure_protected(item["asset_id"])
|
|
||||||
item["preview"] = preview
|
|
||||||
if preview["state"] == "unavailable" and not item["blockers"]:
|
|
||||||
item["blockers"].append(
|
|
||||||
_issue(
|
|
||||||
"preview_unavailable",
|
|
||||||
f"a durable comparison preview of {item['current_path']} could not be "
|
|
||||||
f"created ({preview['error_code']})",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return item
|
|
||||||
|
|
||||||
def _previews(self) -> ThumbnailService:
|
|
||||||
return ThumbnailService(self._session_factory, self._config)
|
|
||||||
|
|
||||||
|
|
||||||
# ── internals ────────────────────────────────────────────────────────────────
|
# ── internals ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
@@ -551,6 +512,13 @@ def _transfer_method(folder: Path, root: Path) -> str:
|
|||||||
return "copy_verify_remove"
|
return "copy_verify_remove"
|
||||||
|
|
||||||
|
|
||||||
|
def _read_marker(root: Path) -> dict | None:
|
||||||
|
try:
|
||||||
|
return json.loads((root / MARKER_NAME).read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _probe_write(path: Path, payload: bytes, *, keep: bool = False) -> str | None:
|
def _probe_write(path: Path, payload: bytes, *, keep: bool = False) -> str | None:
|
||||||
"""Write ``payload`` to ``path``; return the failure detail or ``None``."""
|
"""Write ``payload`` to ``path``; return the failure detail or ``None``."""
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -1,124 +0,0 @@
|
|||||||
"""Where an asset's bytes are right now (US06-03).
|
|
||||||
|
|
||||||
Archiving removes the original from the active library but never removes the
|
|
||||||
asset: its identity, hashes, decisions, and evidence stay. This module is the one
|
|
||||||
place that answers "can these bytes be read, and if not, why" so inventory,
|
|
||||||
duplicate review, thumbnails, and the archive service all give the same answer.
|
|
||||||
|
|
||||||
States (concept §9):
|
|
||||||
|
|
||||||
- ``active`` — the original is in the active library;
|
|
||||||
- ``archived_online`` — the recorded medium is mounted and holds the file;
|
|
||||||
- ``archived_offline`` — archived, but the medium is not available right now;
|
|
||||||
- ``missing_unexpected`` — neither an active path nor the recorded archive
|
|
||||||
location explains the absence. This is the state that must never be confused
|
|
||||||
with ``archived_offline``: an unmounted disk is normal, a mounted disk with a
|
|
||||||
hole in it is not.
|
|
||||||
|
|
||||||
A medium is identified by its marker file, never by its mountpoint, so a
|
|
||||||
different disk mounted at the recorded root is offline rather than accepted.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
from collections import Counter
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from sqlalchemy import select
|
|
||||||
from sqlalchemy.orm import Session, sessionmaker
|
|
||||||
|
|
||||||
from photo_pipeline.models import ArchiveLocation, Asset
|
|
||||||
|
|
||||||
ACTIVE = "active"
|
|
||||||
ARCHIVED_ONLINE = "archived_online"
|
|
||||||
ARCHIVED_OFFLINE = "archived_offline"
|
|
||||||
MISSING_UNEXPECTED = "missing_unexpected"
|
|
||||||
ARCHIVED = (ARCHIVED_ONLINE, ARCHIVED_OFFLINE)
|
|
||||||
|
|
||||||
MARKER_NAME = ".photo-pipeline-archive.json"
|
|
||||||
|
|
||||||
|
|
||||||
def read_marker(root: Path) -> dict | None:
|
|
||||||
"""The medium's identity marker, or ``None`` when it is not readable."""
|
|
||||||
try:
|
|
||||||
return json.loads((root / MARKER_NAME).read_text(encoding="utf-8"))
|
|
||||||
except (OSError, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def location_online(location: ArchiveLocation) -> bool:
|
|
||||||
"""True only when the *recorded* medium is mounted at its root."""
|
|
||||||
marker = read_marker(Path(location.root))
|
|
||||||
return bool(marker) and marker.get("media_id") == location.media_id
|
|
||||||
|
|
||||||
|
|
||||||
def archive_file(session: Session, asset: Asset) -> Path | None:
|
|
||||||
"""The archived file's absolute path, whether or not the medium is mounted."""
|
|
||||||
if not asset.archive_location_id or not asset.archive_path:
|
|
||||||
return None
|
|
||||||
location = session.get(ArchiveLocation, asset.archive_location_id)
|
|
||||||
if location is None:
|
|
||||||
return None
|
|
||||||
return Path(location.root) / asset.archive_path
|
|
||||||
|
|
||||||
|
|
||||||
def readable_path(session: Session, asset: Asset) -> Path | None:
|
|
||||||
"""A path whose bytes can be read now: the active file, else the archive copy."""
|
|
||||||
if asset.current_path and Path(asset.current_path).exists():
|
|
||||||
return Path(asset.current_path)
|
|
||||||
archived = archive_file(session, asset)
|
|
||||||
if archived is None:
|
|
||||||
return None
|
|
||||||
location = session.get(ArchiveLocation, asset.archive_location_id)
|
|
||||||
if not location_online(location) or not archived.exists():
|
|
||||||
return None
|
|
||||||
return archived
|
|
||||||
|
|
||||||
|
|
||||||
def state_of(session: Session, asset: Asset, *, online: dict[str, bool] | None = None) -> str:
|
|
||||||
"""The availability this asset's storage actually justifies right now."""
|
|
||||||
if asset.current_path:
|
|
||||||
return ACTIVE if Path(asset.current_path).exists() else MISSING_UNEXPECTED
|
|
||||||
if not asset.archive_location_id:
|
|
||||||
return MISSING_UNEXPECTED if asset.availability_state != ACTIVE else ACTIVE
|
|
||||||
location = session.get(ArchiveLocation, asset.archive_location_id)
|
|
||||||
if location is None:
|
|
||||||
return MISSING_UNEXPECTED
|
|
||||||
reachable = (
|
|
||||||
online[location.id] if online and location.id in online else location_online(location)
|
|
||||||
)
|
|
||||||
if not reachable:
|
|
||||||
return ARCHIVED_OFFLINE
|
|
||||||
archived = archive_file(session, asset)
|
|
||||||
# The medium is mounted and identified: the file is either there, or it is
|
|
||||||
# genuinely gone — that is not "offline", it needs a human.
|
|
||||||
return ARCHIVED_ONLINE if archived and archived.exists() else MISSING_UNEXPECTED
|
|
||||||
|
|
||||||
|
|
||||||
def refresh(session_factory: sessionmaker) -> dict[str, int]:
|
|
||||||
"""Re-derive availability for every archived asset from the media themselves.
|
|
||||||
|
|
||||||
Only archived assets are probed: whether an *active* file is present is the
|
|
||||||
inventory scan's job and costs one stat per library file. Each medium is
|
|
||||||
probed once, not once per asset.
|
|
||||||
"""
|
|
||||||
counts: Counter[str] = Counter()
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
with session_factory() as session:
|
|
||||||
online = {
|
|
||||||
location.id: location_online(location)
|
|
||||||
for location in session.scalars(select(ArchiveLocation))
|
|
||||||
}
|
|
||||||
for asset in session.scalars(
|
|
||||||
select(Asset).where(Asset.archive_location_id.is_not(None))
|
|
||||||
):
|
|
||||||
state = state_of(session, asset, online=online)
|
|
||||||
counts[state] += 1
|
|
||||||
if state != asset.availability_state:
|
|
||||||
asset.availability_state = state
|
|
||||||
asset.state_version += 1
|
|
||||||
asset.updated_at = now
|
|
||||||
session.commit()
|
|
||||||
return dict(counts)
|
|
||||||
@@ -11,12 +11,6 @@ Detection runs in two categories:
|
|||||||
band (NEAR/SIMILAR). These are review candidates: never decided automatically, and
|
band (NEAR/SIMILAR). These are review candidates: never decided automatically, and
|
||||||
negative-linked pairs are suppressed so a rejected pair is not re-suggested.
|
negative-linked pairs are suppressed so a rejected pair is not re-suggested.
|
||||||
|
|
||||||
Archived assets stay in both indexes (US06-03): a new active copy of an archived
|
|
||||||
original is recognised through its hashes even while the medium is offline, and
|
|
||||||
cluster review falls back to the retained protected preview plus hash evidence.
|
|
||||||
An exact/pixel match links straight to the archived canonical; a perceptual match
|
|
||||||
is a review candidate that names the medium to mount for a pixel-level decision.
|
|
||||||
|
|
||||||
Decisions (``canonical`` / ``not_duplicate`` / ``deferred``) persist with evidence,
|
Decisions (``canonical`` / ``not_duplicate`` / ``deferred``) persist with evidence,
|
||||||
use optimistic version checks, are reversible, and can never form a canonical cycle.
|
use optimistic version checks, are reversible, and can never form a canonical cycle.
|
||||||
A new content-identical member of an already-decided cluster inherits the established
|
A new content-identical member of an already-decided cluster inherits the established
|
||||||
@@ -40,14 +34,12 @@ from sqlalchemy import func, select
|
|||||||
from sqlalchemy.orm import sessionmaker
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
from photo_pipeline.models import (
|
from photo_pipeline.models import (
|
||||||
ArchiveLocation,
|
|
||||||
Asset,
|
Asset,
|
||||||
DuplicateCluster,
|
DuplicateCluster,
|
||||||
DuplicateMember,
|
DuplicateMember,
|
||||||
DuplicateNegativeLink,
|
DuplicateNegativeLink,
|
||||||
Thumbnail,
|
|
||||||
)
|
)
|
||||||
from photo_pipeline.services import availability, hashing
|
from photo_pipeline.services import hashing
|
||||||
|
|
||||||
NEAR_MAX = 5
|
NEAR_MAX = 5
|
||||||
SIMILAR_MAX = 10
|
SIMILAR_MAX = 10
|
||||||
@@ -135,18 +127,18 @@ class DuplicateService:
|
|||||||
|
|
||||||
# ── perceptual hash backfill ───────────────────────────────────────────
|
# ── perceptual hash backfill ───────────────────────────────────────────
|
||||||
def ensure_phashes(self) -> int:
|
def ensure_phashes(self) -> int:
|
||||||
"""Hash whatever is readable now — an archived asset keeps the hash it
|
|
||||||
already has, and gains one whenever its medium happens to be mounted."""
|
|
||||||
updated = 0
|
updated = 0
|
||||||
with self._session_factory() as session:
|
with self._session_factory() as session:
|
||||||
assets = session.execute(select(Asset)).scalars()
|
assets = session.execute(
|
||||||
|
select(Asset).where(
|
||||||
|
Asset.availability_state == "active",
|
||||||
|
Asset.current_path.isnot(None),
|
||||||
|
)
|
||||||
|
).scalars()
|
||||||
for asset in assets:
|
for asset in assets:
|
||||||
if asset.phash is not None and asset.phash_version == hashing.PHASH_VERSION:
|
if asset.phash is not None and asset.phash_version == hashing.PHASH_VERSION:
|
||||||
continue
|
continue
|
||||||
source = availability.readable_path(session, asset)
|
value = hashing.safe_phash(asset.current_path)
|
||||||
if source is None:
|
|
||||||
continue
|
|
||||||
value = hashing.safe_phash(str(source))
|
|
||||||
if value is not None:
|
if value is not None:
|
||||||
asset.phash = value
|
asset.phash = value
|
||||||
asset.phash_version = hashing.PHASH_VERSION
|
asset.phash_version = hashing.PHASH_VERSION
|
||||||
@@ -154,40 +146,20 @@ class DuplicateService:
|
|||||||
session.commit()
|
session.commit()
|
||||||
return updated
|
return updated
|
||||||
|
|
||||||
def ensure_phash(self, asset_id: str, *, source=None) -> str | None:
|
|
||||||
"""Backfill one asset's perceptual hash while its bytes are still readable.
|
|
||||||
|
|
||||||
Archiving calls this before the original leaves — passing the archive copy
|
|
||||||
as ``source``, since the database does not point at it yet — because an
|
|
||||||
asset without a pHash silently drops out of the fuzzy index the moment its
|
|
||||||
medium is away.
|
|
||||||
"""
|
|
||||||
with self._session_factory() as session:
|
|
||||||
asset = session.get(Asset, asset_id)
|
|
||||||
if asset is None:
|
|
||||||
return None
|
|
||||||
if asset.phash is not None and asset.phash_version == hashing.PHASH_VERSION:
|
|
||||||
return asset.phash
|
|
||||||
source = source or availability.readable_path(session, asset)
|
|
||||||
if source is None:
|
|
||||||
return None
|
|
||||||
value = hashing.safe_phash(str(source))
|
|
||||||
if value is not None:
|
|
||||||
asset.phash = value
|
|
||||||
asset.phash_version = hashing.PHASH_VERSION
|
|
||||||
session.commit()
|
|
||||||
return value
|
|
||||||
|
|
||||||
# ── detection ──────────────────────────────────────────────────────────
|
# ── detection ──────────────────────────────────────────────────────────
|
||||||
def detect(self) -> DetectionReport:
|
def detect(self) -> DetectionReport:
|
||||||
self.ensure_phashes()
|
self.ensure_phashes()
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
report = DetectionReport()
|
report = DetectionReport()
|
||||||
with self._session_factory() as session:
|
with self._session_factory() as session:
|
||||||
# Every known asset stays in the indexes, archived or not: a copy of an
|
assets = list(
|
||||||
# archived original must be recognised as a duplicate rather than
|
session.execute(
|
||||||
# treated as a new photo (concept §9, invariant 12).
|
select(Asset).where(
|
||||||
assets = list(session.execute(select(Asset)).scalars())
|
Asset.availability_state == "active",
|
||||||
|
Asset.current_path.isnot(None),
|
||||||
|
)
|
||||||
|
).scalars()
|
||||||
|
)
|
||||||
by_id = {a.id: a for a in assets}
|
by_id = {a.id: a for a in assets}
|
||||||
negatives = {
|
negatives = {
|
||||||
_pair(link.asset_a, link.asset_b)
|
_pair(link.asset_a, link.asset_b)
|
||||||
@@ -459,19 +431,10 @@ class DuplicateService:
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _recommend_canonical(ids, by_id) -> str:
|
def _recommend_canonical(ids, by_id) -> str:
|
||||||
# ponytail: largest file, then the archived copy, then path as a
|
# ponytail: largest file, path as deterministic tie-break. The concept's
|
||||||
# deterministic tie-break. Archived wins ties because it is the reviewed,
|
# richer policy (resolution, least recompression, metadata richness) lands
|
||||||
# uploaded original — a fresh active copy must not demote it to a variant.
|
# with the review UI story.
|
||||||
# The concept's richer policy (resolution, least recompression, metadata
|
return max(ids, key=lambda i: (by_id[i].byte_size or 0, by_id[i].current_path or ""))
|
||||||
# richness) lands with the review UI story.
|
|
||||||
return max(
|
|
||||||
ids,
|
|
||||||
key=lambda i: (
|
|
||||||
by_id[i].byte_size or 0,
|
|
||||||
by_id[i].availability_state in availability.ARCHIVED,
|
|
||||||
by_id[i].current_path or by_id[i].archive_path or "",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
def _apply_canonical(self, session, cluster, ids, canonical_id):
|
def _apply_canonical(self, session, cluster, ids, canonical_id):
|
||||||
for member in session.execute(
|
for member in session.execute(
|
||||||
@@ -544,15 +507,9 @@ class DuplicateService:
|
|||||||
"current_path": asset.current_path if asset else None,
|
"current_path": asset.current_path if asset else None,
|
||||||
"byte_size": asset.byte_size if asset else None,
|
"byte_size": asset.byte_size if asset else None,
|
||||||
"phash": asset.phash if asset else None,
|
"phash": asset.phash if asset else None,
|
||||||
**self._offline_evidence(session, asset),
|
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
members.sort(key=lambda m: m["asset_id"])
|
members.sort(key=lambda m: m["asset_id"])
|
||||||
# A full-resolution comparison of an offline original is impossible; the
|
|
||||||
# UI asks for that named medium instead of guessing (concept §9).
|
|
||||||
mount_required = sorted(
|
|
||||||
{m["archive_location"] for m in members if m["requires_mount"]}
|
|
||||||
)
|
|
||||||
return {
|
return {
|
||||||
"id": cluster.id,
|
"id": cluster.id,
|
||||||
"method": cluster.method,
|
"method": cluster.method,
|
||||||
@@ -562,54 +519,9 @@ class DuplicateService:
|
|||||||
"canonical_asset_id": cluster.canonical_asset_id,
|
"canonical_asset_id": cluster.canonical_asset_id,
|
||||||
"version": cluster.version,
|
"version": cluster.version,
|
||||||
"requires_confirmation": cluster.method == Method.PERCEPTUAL.value,
|
"requires_confirmation": cluster.method == Method.PERCEPTUAL.value,
|
||||||
"mount_required": mount_required,
|
|
||||||
"members": members,
|
"members": members,
|
||||||
}
|
}
|
||||||
|
|
||||||
def _offline_evidence(self, session, asset: Asset | None) -> dict:
|
|
||||||
"""What review can still rely on when a member's original is not readable."""
|
|
||||||
if asset is None:
|
|
||||||
return {
|
|
||||||
"availability_state": None,
|
|
||||||
"archive_location": None,
|
|
||||||
"archive_location_id": None,
|
|
||||||
"archive_path": None,
|
|
||||||
"preview": {"state": "missing", "protected": False},
|
|
||||||
"requires_mount": False,
|
|
||||||
}
|
|
||||||
location = (
|
|
||||||
session.get(ArchiveLocation, asset.archive_location_id)
|
|
||||||
if asset.archive_location_id
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
preview = self._preview_evidence(session, asset.id)
|
|
||||||
archived = asset.availability_state in availability.ARCHIVED
|
|
||||||
return {
|
|
||||||
"availability_state": asset.availability_state,
|
|
||||||
"archive_location": location.name if location else None,
|
|
||||||
"archive_location_id": asset.archive_location_id,
|
|
||||||
"archive_path": asset.archive_path,
|
|
||||||
"preview": preview,
|
|
||||||
# Offline archived members can still be compared through their retained
|
|
||||||
# preview and hash evidence; only pixel-level review needs the medium.
|
|
||||||
"requires_mount": archived
|
|
||||||
and asset.availability_state == availability.ARCHIVED_OFFLINE
|
|
||||||
and bool(location),
|
|
||||||
}
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _preview_evidence(session, asset_id: str) -> dict:
|
|
||||||
rows = list(
|
|
||||||
session.execute(select(Thumbnail).where(Thumbnail.asset_id == asset_id)).scalars()
|
|
||||||
)
|
|
||||||
ready = [r for r in rows if r.state == "ready" and r.path]
|
|
||||||
if ready:
|
|
||||||
best = max(ready, key=lambda r: (bool(r.protected), r.size or 0))
|
|
||||||
return {"state": "ready", "protected": bool(best.protected), "size": best.size}
|
|
||||||
if rows:
|
|
||||||
return {"state": "unsupported", "protected": False, "size": rows[0].size}
|
|
||||||
return {"state": "missing", "protected": False, "size": None}
|
|
||||||
|
|
||||||
# ── decisions ────────────────────────────────────────────────────────────
|
# ── decisions ────────────────────────────────────────────────────────────
|
||||||
def decide(
|
def decide(
|
||||||
self,
|
self,
|
||||||
|
|||||||
@@ -13,11 +13,8 @@ renames. Every discovered or absent path is classified as one occurrence:
|
|||||||
- ``missing`` — a known active asset whose file is gone (kept, flagged).
|
- ``missing`` — a known active asset whose file is gone (kept, flagged).
|
||||||
|
|
||||||
Missing files are never pruned (that would break identity); the asset is retained
|
Missing files are never pruned (that would break identity); the asset is retained
|
||||||
with ``missing_at`` set and its availability becomes ``missing_unexpected`` —
|
with ``missing_at`` set. Archived assets are left untouched. Rescanning unchanged
|
||||||
nothing explains where the bytes went. Archived assets are left untouched: their
|
input makes no durable change.
|
||||||
absence from the active roots is expected, and each scan re-derives whether their
|
|
||||||
medium is reachable (:mod:`photo_pipeline.services.availability`). Rescanning
|
|
||||||
unchanged input makes no durable change.
|
|
||||||
|
|
||||||
Extracted from photo_analyzer.discover_photos/reconcile_moved/prune_missing
|
Extracted from photo_analyzer.discover_photos/reconcile_moved/prune_missing
|
||||||
(see donor_ledger.yaml: pa-discovery, pa-prune-missing).
|
(see donor_ledger.yaml: pa-discovery, pa-prune-missing).
|
||||||
@@ -33,12 +30,12 @@ from enum import Enum
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Iterable
|
from typing import Iterable
|
||||||
|
|
||||||
from sqlalchemy import func, or_, select
|
from sqlalchemy import func, select
|
||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
from photo_pipeline import path_policy
|
from photo_pipeline import path_policy
|
||||||
from photo_pipeline.models import Asset, AssetPath
|
from photo_pipeline.models import Asset, AssetPath
|
||||||
from photo_pipeline.services import availability, hashing
|
from photo_pipeline.services import hashing
|
||||||
|
|
||||||
|
|
||||||
class Occurrence(str, Enum):
|
class Occurrence(str, Enum):
|
||||||
@@ -63,8 +60,6 @@ def _asset_dict(asset: Asset) -> dict:
|
|||||||
"id": asset.id,
|
"id": asset.id,
|
||||||
"current_path": asset.current_path,
|
"current_path": asset.current_path,
|
||||||
"availability_state": asset.availability_state,
|
"availability_state": asset.availability_state,
|
||||||
"archive_location_id": asset.archive_location_id,
|
|
||||||
"archive_path": asset.archive_path,
|
|
||||||
"byte_size": asset.byte_size,
|
"byte_size": asset.byte_size,
|
||||||
"current_sha256": asset.current_sha256,
|
"current_sha256": asset.current_sha256,
|
||||||
"pixel_sha256": asset.pixel_sha256,
|
"pixel_sha256": asset.pixel_sha256,
|
||||||
@@ -106,9 +101,7 @@ class InventoryService:
|
|||||||
result.asset_ids[str(path)] = asset.id
|
result.asset_ids[str(path)] = asset.id
|
||||||
|
|
||||||
for asset in assets:
|
for asset in assets:
|
||||||
# Archived assets are explained by their location, not by the active
|
if asset.availability_state != "active" or asset.id in seen_ids:
|
||||||
# roots: a scan must never prune or flag them (concept §9).
|
|
||||||
if asset.availability_state in availability.ARCHIVED or asset.id in seen_ids:
|
|
||||||
continue
|
continue
|
||||||
if asset.current_path and asset.current_path not in discovered_paths:
|
if asset.current_path and asset.current_path not in discovered_paths:
|
||||||
if not Path(asset.current_path).exists():
|
if not Path(asset.current_path).exists():
|
||||||
@@ -116,18 +109,10 @@ class InventoryService:
|
|||||||
asset.missing_at = now
|
asset.missing_at = now
|
||||||
asset.state_version += 1
|
asset.state_version += 1
|
||||||
asset.updated_at = now
|
asset.updated_at = now
|
||||||
# Nothing explains this absence — it is not an offline medium.
|
|
||||||
if asset.availability_state != availability.MISSING_UNEXPECTED:
|
|
||||||
asset.availability_state = availability.MISSING_UNEXPECTED
|
|
||||||
asset.state_version += 1
|
|
||||||
asset.updated_at = now
|
|
||||||
result.occurrences[asset.current_path] = Occurrence.MISSING.value
|
result.occurrences[asset.current_path] = Occurrence.MISSING.value
|
||||||
|
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
# Media may have been mounted or removed since the last scan.
|
|
||||||
availability.refresh(self._session_factory)
|
|
||||||
|
|
||||||
result.counts = dict(Counter(result.occurrences.values()))
|
result.counts = dict(Counter(result.occurrences.values()))
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@@ -147,11 +132,7 @@ class InventoryService:
|
|||||||
if availability:
|
if availability:
|
||||||
stmt = stmt.where(Asset.availability_state == availability)
|
stmt = stmt.where(Asset.availability_state == availability)
|
||||||
if query:
|
if query:
|
||||||
like = f"%{query}%"
|
stmt = stmt.where(Asset.current_path.like(f"%{query}%"))
|
||||||
# An archived asset has no active path; it is searched where it lives.
|
|
||||||
stmt = stmt.where(
|
|
||||||
or_(Asset.current_path.like(like), Asset.archive_path.like(like))
|
|
||||||
)
|
|
||||||
total = session.scalar(select(func.count()).select_from(stmt.subquery()))
|
total = session.scalar(select(func.count()).select_from(stmt.subquery()))
|
||||||
rows = session.execute(
|
rows = session.execute(
|
||||||
stmt.order_by(Asset.current_path).limit(limit).offset(offset)
|
stmt.order_by(Asset.current_path).limit(limit).offset(offset)
|
||||||
@@ -188,7 +169,6 @@ class InventoryService:
|
|||||||
self._open_path(session, existing.id, path_str, now, occ.value)
|
self._open_path(session, existing.id, path_str, now, occ.value)
|
||||||
if existing.missing_at is not None:
|
if existing.missing_at is not None:
|
||||||
existing.missing_at = None
|
existing.missing_at = None
|
||||||
existing.availability_state = availability.ACTIVE
|
|
||||||
existing.state_version += 1
|
existing.state_version += 1
|
||||||
existing.updated_at = now
|
existing.updated_at = now
|
||||||
return existing, occ
|
return existing, occ
|
||||||
@@ -209,7 +189,6 @@ class InventoryService:
|
|||||||
moved_from.current_path = path_str
|
moved_from.current_path = path_str
|
||||||
moved_from.byte_size = size
|
moved_from.byte_size = size
|
||||||
moved_from.missing_at = None
|
moved_from.missing_at = None
|
||||||
moved_from.availability_state = availability.ACTIVE
|
|
||||||
moved_from.state_version += 1
|
moved_from.state_version += 1
|
||||||
moved_from.updated_at = now
|
moved_from.updated_at = now
|
||||||
self._open_path(session, moved_from.id, path_str, now, Occurrence.MOVED.value)
|
self._open_path(session, moved_from.id, path_str, now, Occurrence.MOVED.value)
|
||||||
|
|||||||
@@ -1,220 +0,0 @@
|
|||||||
"""Import the last path-keyed CSV state into the database (US07-01).
|
|
||||||
|
|
||||||
``nsfwtag`` cached its safety scores in ``nsfw_scores.csv`` next to the library:
|
|
||||||
one ``path,nsfw_score`` row per photo, four decimals, unreadable rows dropped. That
|
|
||||||
file stops being a source of truth when the CLI is archived, so its scores are
|
|
||||||
imported once — as scored-but-unreviewed ``safety_reviews`` rows on the stable
|
|
||||||
``assets.id`` each path resolves to — and the CSV is left untouched on disk as its
|
|
||||||
own backup.
|
|
||||||
|
|
||||||
The import is deliberately conservative, because a score is evidence about a photo
|
|
||||||
and a path is not an identity:
|
|
||||||
|
|
||||||
- a row whose path matches no known asset is **unmatched**, never a new asset;
|
|
||||||
- an asset a human already reviewed is **never** touched: a score is evidence, a
|
|
||||||
decision is a judgement, and an import may not overwrite the second with the
|
|
||||||
first — the difference is reported as a conflict instead;
|
|
||||||
- a row for an asset that already carries a score is **skipped** unless
|
|
||||||
``overwrite`` is asked for, and a differing score is reported either way;
|
|
||||||
- an unparsable score or a duplicate path is **rejected** with its reason;
|
|
||||||
- rerunning changes nothing that is already imported.
|
|
||||||
|
|
||||||
Everything it did — and everything it refused to do — comes back as a
|
|
||||||
reconciliation report, which the caller can persist next to the database. The
|
|
||||||
donor's own reader is the specification for the format (donor ledger:
|
|
||||||
``nt-score-cache``).
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import csv
|
|
||||||
import json
|
|
||||||
import uuid
|
|
||||||
from collections import Counter
|
|
||||||
from dataclasses import asdict, dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from sqlalchemy import select
|
|
||||||
from sqlalchemy.orm import sessionmaker
|
|
||||||
|
|
||||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview
|
|
||||||
|
|
||||||
REPORT_VERSION = 1
|
|
||||||
# The donor wrote scores as four-decimal strings; anything outside 0..1 was never
|
|
||||||
# something it produced, so it is data corruption rather than a score.
|
|
||||||
SCORE_RANGE = (0.0, 1.0)
|
|
||||||
# Recorded as the reviewer so an imported score is never mistaken for a judgement.
|
|
||||||
REVIEWER = "legacy-csv-import"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class ImportReport:
|
|
||||||
source: str
|
|
||||||
rows: int = 0
|
|
||||||
imported: int = 0
|
|
||||||
skipped_existing: int = 0
|
|
||||||
unchanged: int = 0
|
|
||||||
reviewed: int = 0
|
|
||||||
unmatched: list[str] = field(default_factory=list)
|
|
||||||
rejected: list[dict] = field(default_factory=list)
|
|
||||||
conflicts: list[dict] = field(default_factory=list)
|
|
||||||
generated_at: str = ""
|
|
||||||
schema_version: int = REPORT_VERSION
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
return asdict(self)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def counts(self) -> dict[str, int]:
|
|
||||||
return {
|
|
||||||
"rows": self.rows,
|
|
||||||
"imported": self.imported,
|
|
||||||
"skipped_existing": self.skipped_existing,
|
|
||||||
"unchanged": self.unchanged,
|
|
||||||
"reviewed": self.reviewed,
|
|
||||||
"unmatched": len(self.unmatched),
|
|
||||||
"rejected": len(self.rejected),
|
|
||||||
"conflicts": len(self.conflicts),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class LegacyImportService:
|
|
||||||
def __init__(self, session_factory: sessionmaker) -> None:
|
|
||||||
self._session_factory = session_factory
|
|
||||||
|
|
||||||
def import_nsfw_scores(
|
|
||||||
self, csv_path: Path | str, *, overwrite: bool = False, dry_run: bool = False
|
|
||||||
) -> ImportReport:
|
|
||||||
"""Import ``nsfw_scores.csv`` onto asset identity and report what happened."""
|
|
||||||
path = Path(csv_path)
|
|
||||||
report = ImportReport(source=str(path))
|
|
||||||
if not path.is_file():
|
|
||||||
report.rejected.append({"path": str(path), "reason": "csv_missing"})
|
|
||||||
return self._stamp(report)
|
|
||||||
|
|
||||||
with self._session_factory() as session:
|
|
||||||
# Paths are matched against every occurrence an asset ever had, so a
|
|
||||||
# photo scored before a rename is still recognised.
|
|
||||||
by_path = self._path_index(session)
|
|
||||||
latest = self._latest_reviews(session)
|
|
||||||
seen: Counter[str] = Counter()
|
|
||||||
|
|
||||||
for row in self._rows(path, report):
|
|
||||||
report.rows += 1
|
|
||||||
raw_path, raw_score = row
|
|
||||||
seen[raw_path] += 1
|
|
||||||
if seen[raw_path] > 1:
|
|
||||||
report.rejected.append({"path": raw_path, "reason": "duplicate_path"})
|
|
||||||
continue
|
|
||||||
score = _parse_score(raw_score)
|
|
||||||
if score is None:
|
|
||||||
report.rejected.append(
|
|
||||||
{"path": raw_path, "reason": "unparsable_score", "value": raw_score}
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
asset_id = by_path.get(raw_path)
|
|
||||||
if asset_id is None:
|
|
||||||
report.unmatched.append(raw_path)
|
|
||||||
continue
|
|
||||||
|
|
||||||
current = latest.get(asset_id)
|
|
||||||
if current is not None and current.decision:
|
|
||||||
# A human decided this one; the CSV is older evidence.
|
|
||||||
report.reviewed += 1
|
|
||||||
if current.score is None or abs(current.score - score) >= 1e-9:
|
|
||||||
report.conflicts.append(
|
|
||||||
{
|
|
||||||
"path": raw_path,
|
|
||||||
"asset_id": asset_id,
|
|
||||||
"current": current.score,
|
|
||||||
"decision": current.decision,
|
|
||||||
"csv": score,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
if current is not None and current.score is not None:
|
|
||||||
if abs(current.score - score) < 1e-9:
|
|
||||||
report.unchanged += 1
|
|
||||||
continue
|
|
||||||
report.conflicts.append(
|
|
||||||
{
|
|
||||||
"path": raw_path,
|
|
||||||
"asset_id": asset_id,
|
|
||||||
"current": current.score,
|
|
||||||
"decision": None,
|
|
||||||
"csv": score,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
if not overwrite:
|
|
||||||
report.skipped_existing += 1
|
|
||||||
continue
|
|
||||||
session.add(
|
|
||||||
SafetyReview(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
asset_id=asset_id,
|
|
||||||
score=score,
|
|
||||||
reviewer=REVIEWER,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
latest[asset_id] = SafetyReview(asset_id=asset_id, score=score)
|
|
||||||
report.imported += 1
|
|
||||||
|
|
||||||
if dry_run:
|
|
||||||
session.rollback()
|
|
||||||
else:
|
|
||||||
session.commit()
|
|
||||||
return self._stamp(report)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _rows(path: Path, report: ImportReport):
|
|
||||||
"""Yield ``(path, score)`` pairs, tolerating the donor's own sloppiness."""
|
|
||||||
with open(path, newline="", encoding="utf-8", errors="replace") as handle:
|
|
||||||
for row in csv.DictReader(handle):
|
|
||||||
raw_path = (row.get("path") or "").strip()
|
|
||||||
if not raw_path:
|
|
||||||
report.rejected.append({"path": "", "reason": "missing_path"})
|
|
||||||
continue
|
|
||||||
yield raw_path, row.get("nsfw_score")
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _latest_reviews(session) -> dict[str, SafetyReview]:
|
|
||||||
"""The current safety row per asset — latest wins, as everywhere else."""
|
|
||||||
latest: dict[str, SafetyReview] = {}
|
|
||||||
for review in session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)):
|
|
||||||
latest[review.asset_id] = review
|
|
||||||
return latest
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _path_index(session) -> dict[str, str]:
|
|
||||||
"""Every path an asset is or was known by → its stable id."""
|
|
||||||
index: dict[str, str] = {}
|
|
||||||
for asset_id, path in session.execute(select(AssetPath.asset_id, AssetPath.path)):
|
|
||||||
index.setdefault(path, asset_id)
|
|
||||||
for asset_id, path in session.execute(select(Asset.id, Asset.current_path)):
|
|
||||||
if path:
|
|
||||||
index[path] = asset_id # the current path wins over a closed one
|
|
||||||
return index
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _stamp(report: ImportReport) -> ImportReport:
|
|
||||||
report.generated_at = datetime.now(timezone.utc).isoformat()
|
|
||||||
return report
|
|
||||||
|
|
||||||
|
|
||||||
def write_report(report: ImportReport, directory: Path) -> Path:
|
|
||||||
"""Persist the reconciliation report; the import is not evidence until it is."""
|
|
||||||
directory.mkdir(parents=True, exist_ok=True)
|
|
||||||
path = directory / "legacy-nsfw-import.json"
|
|
||||||
path.write_text(json.dumps(report.to_dict(), indent=2, sort_keys=True), encoding="utf-8")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_score(value) -> float | None:
|
|
||||||
try:
|
|
||||||
score = float(value)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
if not SCORE_RANGE[0] <= score <= SCORE_RANGE[1]:
|
|
||||||
return None
|
|
||||||
return score
|
|
||||||
@@ -1,650 +0,0 @@
|
|||||||
"""RestoreService — plan and execute safe restores (US06-04).
|
|
||||||
|
|
||||||
Restore is archiving read backwards, with one decisive difference: it removes
|
|
||||||
nothing. The archived copy stays on its medium, so every failure mode here costs
|
|
||||||
at most a discarded temporary file. What restore must never do is *lose identity*
|
|
||||||
— the asset that comes back is the same asset, with its duplicate decision, safety
|
|
||||||
review, analysis, and upload history intact — or *overwrite* something in the
|
|
||||||
active library.
|
|
||||||
|
|
||||||
Preflight proves, per concept §9 "Restore":
|
|
||||||
|
|
||||||
- the recorded medium is mounted and is the right one (marker ``media_id``);
|
|
||||||
- every selected asset is archived, its archive copy exists, and it hashes to
|
|
||||||
exactly the bytes the database recorded — a mismatch is ``divergent`` and is
|
|
||||||
refused, never silently accepted as "the file";
|
|
||||||
- the destination lies inside the library, outside ``_IGNORE/``, and is free; a
|
|
||||||
taken path is answered with a collision-free name, never an overwrite;
|
|
||||||
- the library filesystem has room for the scope plus the configured reserve;
|
|
||||||
- no rename, archive, or restore lease is holding the lane.
|
|
||||||
|
|
||||||
Blocker codes: ``no_library_root``, ``location_offline``, ``wrong_volume``,
|
|
||||||
``unsafe_destination``, ``library_not_writable``, ``insufficient_capacity``,
|
|
||||||
``lock_conflict``, ``rename_pending``, ``archive_pending``, ``empty_scope``,
|
|
||||||
``not_archived``, ``archive_missing``, ``bytes_changed``.
|
|
||||||
|
|
||||||
Per item the sequence is:
|
|
||||||
|
|
||||||
```
|
|
||||||
journal.begin (transferring) ← intent persisted BEFORE any disk change
|
|
||||||
recheck: medium, hash, free destination, asset still archived
|
|
||||||
copy to a temporary file beside the destination, fsync, hash it back
|
|
||||||
atomically publish into the library
|
|
||||||
journal → verified
|
|
||||||
current_path = destination, availability = active, path occurrence opened
|
|
||||||
journal → complete
|
|
||||||
```
|
|
||||||
|
|
||||||
Like archiving, the confirmation token is derived from the report, so a changed
|
|
||||||
scope, a swapped medium, or a destination that filled up invalidates it.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import shutil
|
|
||||||
import uuid
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from sqlalchemy import select
|
|
||||||
from sqlalchemy.orm import sessionmaker
|
|
||||||
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
from photo_pipeline.jobs.domain_handlers import ARCHIVE_LOCK, LIBRARY_WRITE_LOCK, UPLOAD_LOCK
|
|
||||||
from photo_pipeline.models import ArchiveLocation, ArchiveOperation, ArchivePlan, Asset, AssetPath
|
|
||||||
from photo_pipeline.path_policy import PathPolicyError, is_excluded, normalize_root, resolve_within
|
|
||||||
from photo_pipeline.services import availability
|
|
||||||
from photo_pipeline.services.archive_journal import (
|
|
||||||
MANUAL,
|
|
||||||
RESTORE,
|
|
||||||
RESUMABLE,
|
|
||||||
ArchiveJournal,
|
|
||||||
ArchiveState,
|
|
||||||
)
|
|
||||||
from photo_pipeline.services.archive_transfer import (
|
|
||||||
_clean_temp_files,
|
|
||||||
_fsync_dir,
|
|
||||||
_plan_dict,
|
|
||||||
copy_verify_publish,
|
|
||||||
)
|
|
||||||
from photo_pipeline.services.archives import ArchiveError
|
|
||||||
from photo_pipeline.services.hashing import sha256_file
|
|
||||||
from photo_pipeline.services.jobs import JobService
|
|
||||||
from photo_pipeline.services.rename_apply import PreconditionFailed, maybe_fault
|
|
||||||
from photo_pipeline.services.rename_journal import RenameJournal
|
|
||||||
|
|
||||||
PREFLIGHT_VERSION = 1
|
|
||||||
TOKEN_PREFIX = f"r{PREFLIGHT_VERSION}"
|
|
||||||
# What a restored file is called when its original name is taken. The suffix is
|
|
||||||
# visible on purpose: a restore that quietly reuses a name is indistinguishable
|
|
||||||
# from an overwrite.
|
|
||||||
RESTORED_SUFFIX = "restored"
|
|
||||||
|
|
||||||
LOCKS = (LIBRARY_WRITE_LOCK, UPLOAD_LOCK, ARCHIVE_LOCK)
|
|
||||||
|
|
||||||
APPLYABLE_PLAN_STATES = frozenset({"planned", "applying", "failed", "complete"})
|
|
||||||
|
|
||||||
|
|
||||||
def _now() -> datetime:
|
|
||||||
return datetime.now(timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
def _issue(code: str, message: str) -> dict:
|
|
||||||
return {"code": code, "message": message}
|
|
||||||
|
|
||||||
|
|
||||||
class RestoreService:
|
|
||||||
def __init__(self, session_factory: sessionmaker, *, config: Config) -> None:
|
|
||||||
self._session_factory = session_factory
|
|
||||||
self._config = config
|
|
||||||
self._roots = tuple(normalize_root(root) for root in config.library_roots)
|
|
||||||
self.journal = ArchiveJournal(session_factory)
|
|
||||||
|
|
||||||
# ── preflight ─────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def preflight(self, location_id: str, asset_ids: list[str] | None = None) -> dict:
|
|
||||||
"""Validate a restore scope and issue its token. Nothing is written."""
|
|
||||||
with self._session_factory() as session:
|
|
||||||
location = session.get(ArchiveLocation, location_id)
|
|
||||||
if location is None:
|
|
||||||
raise ArchiveError("unknown_location", f"unknown archive location {location_id!r}")
|
|
||||||
root = Path(location.root)
|
|
||||||
online = availability.location_online(location)
|
|
||||||
marker = availability.read_marker(root)
|
|
||||||
report = {
|
|
||||||
"schema_version": PREFLIGHT_VERSION,
|
|
||||||
"location": {
|
|
||||||
"id": location.id,
|
|
||||||
"name": location.name,
|
|
||||||
"root": str(root),
|
|
||||||
"media_id": location.media_id,
|
|
||||||
"state": _location_state(root, marker, location.media_id),
|
|
||||||
},
|
|
||||||
"blockers": [],
|
|
||||||
}
|
|
||||||
items = self._items(session, location, asset_ids, reachable=online)
|
|
||||||
|
|
||||||
report["blockers"] += self._destination_blockers(report["location"]["state"], root)
|
|
||||||
report["blockers"] += self._lock_blockers()
|
|
||||||
report["items"] = items
|
|
||||||
report["totals"] = {
|
|
||||||
"assets": len(items),
|
|
||||||
"blocked": sum(1 for item in items if item["blockers"]),
|
|
||||||
"bytes": sum(item["byte_size"] or 0 for item in items),
|
|
||||||
}
|
|
||||||
report["capacity"] = self._capacity(report["totals"]["bytes"])
|
|
||||||
if not report["capacity"]["sufficient"]:
|
|
||||||
report["blockers"].append(
|
|
||||||
_issue(
|
|
||||||
"insufficient_capacity",
|
|
||||||
# No free-space number here: it drifts between two identical
|
|
||||||
# preflights and the token is a digest of this text (US06-06).
|
|
||||||
f"{report['totals']['bytes']} B plus a "
|
|
||||||
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit in "
|
|
||||||
"the library",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if not items:
|
|
||||||
report["blockers"].append(
|
|
||||||
_issue("empty_scope", "no archived assets are in the selected scope")
|
|
||||||
)
|
|
||||||
report["state"] = (
|
|
||||||
"ready"
|
|
||||||
if not report["blockers"] and not report["totals"]["blocked"]
|
|
||||||
else "blocked"
|
|
||||||
)
|
|
||||||
report["token"] = _token(report)
|
|
||||||
report["generated_at"] = _now().isoformat()
|
|
||||||
return report
|
|
||||||
|
|
||||||
def verify_token(self, token: str, location_id: str, asset_ids: list[str] | None = None) -> bool:
|
|
||||||
return bool(token) and token == self.preflight(location_id, asset_ids)["token"]
|
|
||||||
|
|
||||||
def _items(
|
|
||||||
self, session, location: ArchiveLocation, asset_ids: list[str] | None, *, reachable: bool
|
|
||||||
) -> list[dict]:
|
|
||||||
stmt = select(Asset).where(Asset.archive_location_id == location.id)
|
|
||||||
if asset_ids is None:
|
|
||||||
# A restored asset keeps its archive link; the default scope is only what
|
|
||||||
# is still archived, so restoring twice is an empty scope, not a blocker.
|
|
||||||
stmt = stmt.where(Asset.availability_state.in_(availability.ARCHIVED))
|
|
||||||
else:
|
|
||||||
stmt = stmt.where(Asset.id.in_(asset_ids))
|
|
||||||
assets = list(session.scalars(stmt.order_by(Asset.archive_path)))
|
|
||||||
if asset_ids is not None:
|
|
||||||
unknown = sorted(set(asset_ids) - {asset.id for asset in assets})
|
|
||||||
if unknown:
|
|
||||||
raise ArchiveError(
|
|
||||||
"unknown_asset", f"not archived at this location: {', '.join(unknown)}"
|
|
||||||
)
|
|
||||||
taken: set[str] = set()
|
|
||||||
return [self._item(asset, location, reachable=reachable, taken=taken) for asset in assets]
|
|
||||||
|
|
||||||
def _item(self, asset: Asset, location: ArchiveLocation, *, reachable: bool, taken: set) -> dict:
|
|
||||||
source = Path(location.root) / (asset.archive_path or "")
|
|
||||||
blockers: list[dict] = []
|
|
||||||
archive_sha256 = None
|
|
||||||
|
|
||||||
if asset.availability_state not in availability.ARCHIVED:
|
|
||||||
blockers.append(
|
|
||||||
_issue("not_archived", f"asset {asset.id} is {asset.availability_state}")
|
|
||||||
)
|
|
||||||
if reachable:
|
|
||||||
if not source.exists():
|
|
||||||
blockers.append(_issue("archive_missing", f"{source} is not on the medium"))
|
|
||||||
else:
|
|
||||||
archive_sha256 = sha256_file(source)
|
|
||||||
if asset.current_sha256 and archive_sha256 != asset.current_sha256:
|
|
||||||
blockers.append(
|
|
||||||
_issue(
|
|
||||||
"bytes_changed",
|
|
||||||
f"{source} holds bytes that are not the recorded ones; "
|
|
||||||
"the archived copy is divergent",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
destination, destination_blockers = self._destination(asset, taken)
|
|
||||||
blockers += destination_blockers
|
|
||||||
if destination is not None:
|
|
||||||
taken.add(str(destination))
|
|
||||||
return {
|
|
||||||
"asset_id": asset.id,
|
|
||||||
"archive_path": asset.archive_path,
|
|
||||||
"source_path": str(source),
|
|
||||||
"destination_path": str(destination) if destination else None,
|
|
||||||
"expected_sha256": asset.current_sha256,
|
|
||||||
"archive_sha256": archive_sha256,
|
|
||||||
"byte_size": asset.byte_size,
|
|
||||||
"availability_state": asset.availability_state,
|
|
||||||
"blockers": blockers,
|
|
||||||
}
|
|
||||||
|
|
||||||
def _destination(self, asset: Asset, taken: set) -> tuple[Path | None, list[dict]]:
|
|
||||||
"""A free path inside the library that mirrors the archived layout.
|
|
||||||
|
|
||||||
Restoring onto an existing file is never an option, so a taken name is
|
|
||||||
answered with ``name (restored).ext`` — visible, ordinary, and impossible to
|
|
||||||
confuse with an overwrite.
|
|
||||||
"""
|
|
||||||
if not self._roots:
|
|
||||||
return None, [_issue("no_library_root", "no library root is configured")]
|
|
||||||
root = self._roots[0]
|
|
||||||
try:
|
|
||||||
candidate = resolve_within(root, root / (asset.archive_path or ""))
|
|
||||||
except PathPolicyError as error:
|
|
||||||
return None, [_issue("unsafe_destination", str(error))]
|
|
||||||
if is_excluded(candidate):
|
|
||||||
return None, [
|
|
||||||
_issue("unsafe_destination", f"{candidate} is inside an excluded (_IGNORE/) tree")
|
|
||||||
]
|
|
||||||
return _free_path(candidate, taken), []
|
|
||||||
|
|
||||||
def _destination_blockers(self, state: str, root: Path) -> list[dict]:
|
|
||||||
blockers: list[dict] = []
|
|
||||||
if not self._roots:
|
|
||||||
blockers.append(_issue("no_library_root", "no library root is configured"))
|
|
||||||
elif not os.access(self._roots[0], os.W_OK):
|
|
||||||
blockers.append(
|
|
||||||
_issue("library_not_writable", f"{self._roots[0]} is not writable")
|
|
||||||
)
|
|
||||||
if state == "offline":
|
|
||||||
blockers.append(
|
|
||||||
_issue("location_offline", f"the archive medium is not mounted at {root}")
|
|
||||||
)
|
|
||||||
elif state == "wrong_volume":
|
|
||||||
blockers.append(_issue("wrong_volume", f"{root} holds a different archive medium"))
|
|
||||||
return blockers
|
|
||||||
|
|
||||||
def _lock_blockers(self) -> list[dict]:
|
|
||||||
blockers: list[dict] = []
|
|
||||||
jobs = JobService(self._session_factory)
|
|
||||||
for lock in LOCKS:
|
|
||||||
held = jobs.blockers(lock)
|
|
||||||
if held:
|
|
||||||
blockers.append(
|
|
||||||
_issue("lock_conflict", f"the {lock} lane is busy: job {held[0]['id']}")
|
|
||||||
)
|
|
||||||
if RenameJournal(self._session_factory).blocks_mutation():
|
|
||||||
blockers.append(
|
|
||||||
_issue("rename_pending", "an unresolved rename must be recovered before restoring")
|
|
||||||
)
|
|
||||||
if self.journal.blocks_mutation():
|
|
||||||
blockers.append(
|
|
||||||
_issue(
|
|
||||||
"archive_pending",
|
|
||||||
"an unresolved archive or restore must be recovered before restoring",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return blockers
|
|
||||||
|
|
||||||
def _capacity(self, required: int) -> dict:
|
|
||||||
reserve = self._config.archive_free_space_reserve_bytes
|
|
||||||
free = shutil.disk_usage(self._roots[0]).free if self._roots else None
|
|
||||||
return {
|
|
||||||
"required_bytes": required,
|
|
||||||
"reserve_bytes": reserve,
|
|
||||||
"free_bytes": free,
|
|
||||||
"sufficient": free is not None and free >= required + reserve,
|
|
||||||
}
|
|
||||||
|
|
||||||
# ── plans ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def create(self, location_id: str, asset_ids: list[str] | None = None, *, token: str) -> dict:
|
|
||||||
preflight = self.preflight(location_id, asset_ids)
|
|
||||||
if not token or token != preflight["token"]:
|
|
||||||
raise ArchiveError("stale_token", "the restore preflight changed since it was approved")
|
|
||||||
if preflight["state"] != "ready":
|
|
||||||
codes = ", ".join(sorted({issue["code"] for issue in preflight["blockers"]})) or "-"
|
|
||||||
blocked = sorted(
|
|
||||||
{issue["code"] for item in preflight["items"] for issue in item["blockers"]}
|
|
||||||
)
|
|
||||||
raise ArchiveError(
|
|
||||||
"blocked", f"the restore scope is blocked: {', '.join(blocked) or codes}"
|
|
||||||
)
|
|
||||||
|
|
||||||
plan_id = str(uuid.uuid4())
|
|
||||||
with self._session_factory() as session:
|
|
||||||
session.add(
|
|
||||||
ArchivePlan(
|
|
||||||
id=plan_id,
|
|
||||||
location_id=location_id,
|
|
||||||
token=token,
|
|
||||||
albums=json.dumps(asset_ids) if asset_ids is not None else None,
|
|
||||||
direction=RESTORE,
|
|
||||||
state="planned",
|
|
||||||
schema_version=PREFLIGHT_VERSION,
|
|
||||||
asset_count=preflight["totals"]["assets"],
|
|
||||||
byte_size=preflight["totals"]["bytes"],
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.flush()
|
|
||||||
for sequence, item in enumerate(preflight["items"]):
|
|
||||||
session.add(
|
|
||||||
ArchiveOperation(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
plan_id=plan_id,
|
|
||||||
direction=RESTORE,
|
|
||||||
sequence=sequence,
|
|
||||||
album=Path(item["archive_path"]).parent.name or "(root)",
|
|
||||||
asset_id=item["asset_id"],
|
|
||||||
source_path=item["source_path"],
|
|
||||||
destination_path=item["destination_path"],
|
|
||||||
archive_path=item["archive_path"],
|
|
||||||
expected_sha256=item["expected_sha256"],
|
|
||||||
byte_size=item["byte_size"],
|
|
||||||
journal_state=ArchiveState.PLANNED,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return self.get(plan_id)
|
|
||||||
|
|
||||||
def get(self, plan_id: str) -> dict | None:
|
|
||||||
with self._session_factory() as session:
|
|
||||||
plan = session.get(ArchivePlan, plan_id)
|
|
||||||
if plan is None or plan.direction != RESTORE:
|
|
||||||
return None
|
|
||||||
report = _plan_dict(plan)
|
|
||||||
report["operations"] = self.journal.operations(plan_id)
|
|
||||||
return report
|
|
||||||
|
|
||||||
def list(self) -> list[dict]:
|
|
||||||
with self._session_factory() as session:
|
|
||||||
rows = session.scalars(
|
|
||||||
select(ArchivePlan)
|
|
||||||
.where(ArchivePlan.direction == RESTORE)
|
|
||||||
.order_by(ArchivePlan.created_at)
|
|
||||||
)
|
|
||||||
return [_plan_dict(row) for row in rows]
|
|
||||||
|
|
||||||
# ── apply ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def apply(
|
|
||||||
self, plan_id: str, *, expected_version: int | None = None, worker_id: str = "restore"
|
|
||||||
) -> dict:
|
|
||||||
plan = self._require_plan(plan_id)
|
|
||||||
if expected_version is not None and plan["version"] != expected_version:
|
|
||||||
raise ArchiveError(
|
|
||||||
"stale_plan",
|
|
||||||
f"plan {plan_id} is at version {plan['version']}, expected {expected_version}",
|
|
||||||
)
|
|
||||||
if plan["state"] not in APPLYABLE_PLAN_STATES:
|
|
||||||
raise ArchiveError("invalid_state", f"plan {plan_id} is {plan['state']}")
|
|
||||||
blocking = [row for row in self.journal.incomplete() if row["plan_id"] != plan_id]
|
|
||||||
if blocking:
|
|
||||||
raise ArchiveError(
|
|
||||||
"archive_pending",
|
|
||||||
f"another archive operation is unresolved ({blocking[0]['id']}); recover it first",
|
|
||||||
)
|
|
||||||
|
|
||||||
token = self._claim_plan(plan_id)
|
|
||||||
location = self._location(plan["location_id"])
|
|
||||||
restored = failed = skipped = 0
|
|
||||||
for operation in self.journal.operations(plan_id):
|
|
||||||
if operation["journal_state"] == ArchiveState.COMPLETE:
|
|
||||||
skipped += 1
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
if operation["journal_state"] == ArchiveState.VERIFIED:
|
|
||||||
self._finish(operation, token=token)
|
|
||||||
else:
|
|
||||||
self._restore_one(operation, location, token=token, worker_id=worker_id)
|
|
||||||
restored += 1
|
|
||||||
except PreconditionFailed as error:
|
|
||||||
self._fail(operation, token, error.code, str(error))
|
|
||||||
failed += 1
|
|
||||||
except Exception as error: # unexpected: record and stop touching disk
|
|
||||||
self._fail(operation, token, "restore_error", str(error))
|
|
||||||
failed += 1
|
|
||||||
state = self.journal.sync_plan_state(plan_id)
|
|
||||||
return {
|
|
||||||
"plan_id": plan_id,
|
|
||||||
"restored": restored,
|
|
||||||
"failed": failed,
|
|
||||||
"skipped": skipped,
|
|
||||||
"state": state,
|
|
||||||
}
|
|
||||||
|
|
||||||
def _restore_one(self, operation: dict, location: dict, *, token: int, worker_id: str) -> None:
|
|
||||||
source = Path(operation["source_path"])
|
|
||||||
destination = Path(operation["destination_path"])
|
|
||||||
|
|
||||||
# 1. Intent first; from here a crash is resolvable from journal + disk.
|
|
||||||
self.journal.begin(operation["id"], worker_id=worker_id, fencing_token=token)
|
|
||||||
maybe_fault(ArchiveState.TRANSFERRING)
|
|
||||||
|
|
||||||
# 2. Recheck against the medium and the library as they are right now.
|
|
||||||
self._recheck(operation, source, destination, location)
|
|
||||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
# 3. Always copy: the archived original stays on its medium.
|
|
||||||
copy_verify_publish(source, destination, operation["expected_sha256"])
|
|
||||||
_fsync_dir(destination.parent)
|
|
||||||
|
|
||||||
if sha256_file(destination) != operation["expected_sha256"]:
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"restore_mismatch", f"{destination} does not hold the expected bytes"
|
|
||||||
)
|
|
||||||
self.journal.transition(operation["id"], ArchiveState.VERIFIED, fencing_token=token)
|
|
||||||
maybe_fault(ArchiveState.VERIFIED)
|
|
||||||
|
|
||||||
self._finish(self.journal.get(operation["id"]), token=token)
|
|
||||||
|
|
||||||
def _finish(self, operation: dict, *, token: int) -> None:
|
|
||||||
"""Publish the restored file to the database. Idempotent, so recovery may
|
|
||||||
replay it after a crash between the copy and the bookkeeping."""
|
|
||||||
destination = Path(operation["destination_path"])
|
|
||||||
if not destination.exists() or sha256_file(destination) != operation["expected_sha256"]:
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"restore_unverified", f"{destination} is not a verified restored copy"
|
|
||||||
)
|
|
||||||
self._record_restored(operation, destination)
|
|
||||||
self.journal.transition(operation["id"], ArchiveState.COMPLETE, fencing_token=token)
|
|
||||||
maybe_fault(ArchiveState.COMPLETE)
|
|
||||||
|
|
||||||
def _recheck(self, operation: dict, source: Path, destination: Path, location: dict) -> None:
|
|
||||||
root = Path(location["root"])
|
|
||||||
if not root.is_dir() or not (root / availability.MARKER_NAME).exists():
|
|
||||||
raise PreconditionFailed("location_offline", f"{root} is not the archive medium")
|
|
||||||
if not source.exists():
|
|
||||||
raise PreconditionFailed("archive_missing", f"{source} is not on the medium")
|
|
||||||
if source.is_symlink() or destination.is_symlink():
|
|
||||||
raise PreconditionFailed("symlink", "refusing to restore through a symlink")
|
|
||||||
if destination.exists():
|
|
||||||
# Never overwrite: the plan's free path was taken since it was made.
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"destination_exists", f"destination {destination} is occupied"
|
|
||||||
)
|
|
||||||
if not self._inside_library(destination):
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"destination_escape", f"{destination} is outside the library roots"
|
|
||||||
)
|
|
||||||
if sha256_file(source) != operation["expected_sha256"]:
|
|
||||||
self._mark_divergent(operation["asset_id"])
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"bytes_changed", f"{source} changed since the plan was approved"
|
|
||||||
)
|
|
||||||
with self._session_factory() as session:
|
|
||||||
asset = session.get(Asset, operation["asset_id"])
|
|
||||||
if asset is None or asset.availability_state not in availability.ARCHIVED:
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"not_archived", f"asset {operation['asset_id']} is no longer archived"
|
|
||||||
)
|
|
||||||
|
|
||||||
def _inside_library(self, destination: Path) -> bool:
|
|
||||||
for root in self._roots:
|
|
||||||
try:
|
|
||||||
resolve_within(root, destination)
|
|
||||||
return True
|
|
||||||
except PathPolicyError:
|
|
||||||
continue
|
|
||||||
return False
|
|
||||||
|
|
||||||
# ── database ──────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def _record_restored(self, operation: dict, destination: Path) -> None:
|
|
||||||
"""The bytes are back in the library: open the new active occurrence and set
|
|
||||||
availability. Identity, decisions, and history are untouched — that is the
|
|
||||||
entire point of restoring rather than re-importing."""
|
|
||||||
now = _now()
|
|
||||||
with self._session_factory() as session:
|
|
||||||
asset = session.get(Asset, operation["asset_id"])
|
|
||||||
if asset is None:
|
|
||||||
raise PreconditionFailed(
|
|
||||||
"asset_missing", f"asset {operation['asset_id']} no longer exists"
|
|
||||||
)
|
|
||||||
# A restored asset may be returning to a path it once held, so only an
|
|
||||||
# *open* occurrence counts as already registered — that is what keeps
|
|
||||||
# recovery idempotent without collapsing the path history.
|
|
||||||
recorded = session.scalar(
|
|
||||||
select(AssetPath).where(
|
|
||||||
AssetPath.asset_id == asset.id,
|
|
||||||
AssetPath.path == str(destination),
|
|
||||||
AssetPath.valid_until.is_(None),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if recorded is None: # idempotent: recovery may replay this
|
|
||||||
session.add(
|
|
||||||
AssetPath(
|
|
||||||
asset_id=asset.id,
|
|
||||||
path=str(destination),
|
|
||||||
valid_from=now,
|
|
||||||
reason="restore",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
asset.current_path = str(destination)
|
|
||||||
asset.availability_state = availability.ACTIVE
|
|
||||||
asset.missing_at = None
|
|
||||||
# The archive copy stays where it is; keeping the link means a restored
|
|
||||||
# asset still knows which medium holds its archived bytes.
|
|
||||||
asset.archive_divergent_at = None
|
|
||||||
asset.state_version += 1
|
|
||||||
asset.updated_at = now
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
def _mark_divergent(self, asset_id: str) -> None:
|
|
||||||
"""Record that the archived copy is not the recorded file. Durable, because
|
|
||||||
the next restore attempt must not rediscover this from scratch."""
|
|
||||||
with self._session_factory() as session:
|
|
||||||
asset = session.get(Asset, asset_id)
|
|
||||||
if asset is None:
|
|
||||||
return
|
|
||||||
asset.archive_divergent_at = _now()
|
|
||||||
asset.state_version += 1
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
# ── recovery ──────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def recover(self, *, worker_id: str = "restore-recovery") -> dict:
|
|
||||||
"""Resolve every incomplete restore from journal + disk evidence.
|
|
||||||
|
|
||||||
A restore never removed anything, so ``resumable`` simply discards the
|
|
||||||
temporary debris and re-plans the item; ``forward`` finishes the bookkeeping
|
|
||||||
for a published file; ``manual`` is left untouched and keeps blocking.
|
|
||||||
"""
|
|
||||||
results = {"resumed": 0, "completed": 0, "manual": 0}
|
|
||||||
touched: set[str] = set()
|
|
||||||
for verdict in self.journal.classify_all(direction=RESTORE):
|
|
||||||
operation = self.journal.get(verdict["operation_id"])
|
|
||||||
touched.add(operation["plan_id"])
|
|
||||||
token = (operation["fencing_token"] or 0) + 1
|
|
||||||
if verdict["classification"] == MANUAL:
|
|
||||||
results["manual"] += 1
|
|
||||||
continue
|
|
||||||
if verdict["classification"] == RESUMABLE:
|
|
||||||
_clean_temp_files(Path(operation["destination_path"]).parent)
|
|
||||||
self.journal.transition(operation["id"], ArchiveState.PLANNED, fencing_token=token)
|
|
||||||
results["resumed"] += 1
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
self._finish(operation, token=token)
|
|
||||||
results["completed"] += 1
|
|
||||||
except PreconditionFailed as error:
|
|
||||||
self._fail(operation, token, error.code, str(error))
|
|
||||||
results["manual"] += 1
|
|
||||||
for plan_id in touched:
|
|
||||||
self.journal.sync_plan_state(plan_id)
|
|
||||||
return results
|
|
||||||
|
|
||||||
def recovery_status(self) -> dict:
|
|
||||||
verdicts = self.journal.classify_all(direction=RESTORE)
|
|
||||||
return {
|
|
||||||
"operations": verdicts,
|
|
||||||
"manual": [v for v in verdicts if v["classification"] == MANUAL],
|
|
||||||
"blocks_mutation": self.journal.blocks_mutation(),
|
|
||||||
}
|
|
||||||
|
|
||||||
# ── helpers ───────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def _fail(self, operation: dict, token: int, code: str, message: str) -> None:
|
|
||||||
self.journal.transition(
|
|
||||||
operation["id"], ArchiveState.FAILED, fencing_token=token, error=(code, message)
|
|
||||||
)
|
|
||||||
|
|
||||||
def _require_plan(self, plan_id: str) -> dict:
|
|
||||||
plan = self.get(plan_id)
|
|
||||||
if plan is None:
|
|
||||||
raise ArchiveError("unknown_plan", f"unknown restore plan {plan_id!r}")
|
|
||||||
return plan
|
|
||||||
|
|
||||||
def _location(self, location_id: str) -> dict:
|
|
||||||
with self._session_factory() as session:
|
|
||||||
location = session.get(ArchiveLocation, location_id)
|
|
||||||
if location is None:
|
|
||||||
raise ArchiveError("unknown_location", f"unknown archive location {location_id!r}")
|
|
||||||
return {"id": location.id, "root": location.root, "media_id": location.media_id}
|
|
||||||
|
|
||||||
def _claim_plan(self, plan_id: str) -> int:
|
|
||||||
with self._session_factory() as session:
|
|
||||||
plan = session.get(ArchivePlan, plan_id)
|
|
||||||
plan.version += 1
|
|
||||||
plan.state = "applying"
|
|
||||||
plan.updated_at = _now()
|
|
||||||
token = plan.version
|
|
||||||
session.commit()
|
|
||||||
return token
|
|
||||||
|
|
||||||
|
|
||||||
# ── module helpers ───────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def _location_state(root: Path, marker: dict | None, media_id: str) -> str:
|
|
||||||
if not root.is_dir() or marker is None:
|
|
||||||
return "offline"
|
|
||||||
return "online" if marker.get("media_id") == media_id else "wrong_volume"
|
|
||||||
|
|
||||||
|
|
||||||
def _free_path(candidate: Path, taken: set) -> Path:
|
|
||||||
"""``a.jpg`` → ``a (restored).jpg`` → ``a (restored 2).jpg`` …
|
|
||||||
|
|
||||||
``taken`` holds the destinations already claimed by earlier items of the same
|
|
||||||
plan, so two restores in one scope cannot plan the same path.
|
|
||||||
"""
|
|
||||||
if not candidate.exists() and str(candidate) not in taken:
|
|
||||||
return candidate
|
|
||||||
stem, suffix = candidate.stem, candidate.suffix
|
|
||||||
attempt = 1
|
|
||||||
while True:
|
|
||||||
label = RESTORED_SUFFIX if attempt == 1 else f"{RESTORED_SUFFIX} {attempt}"
|
|
||||||
alternative = candidate.with_name(f"{stem} ({label}){suffix}")
|
|
||||||
if not alternative.exists() and str(alternative) not in taken:
|
|
||||||
return alternative
|
|
||||||
attempt += 1
|
|
||||||
|
|
||||||
|
|
||||||
def _token(report: dict) -> str:
|
|
||||||
"""Digest of everything the report asserts about the scope and the medium.
|
|
||||||
|
|
||||||
Free space is excluded: it drifts constantly without changing what a restore
|
|
||||||
would do, and the capacity verdict itself is part of the digest.
|
|
||||||
"""
|
|
||||||
payload = {key: value for key, value in report.items() if key not in ("generated_at", "token")}
|
|
||||||
payload["capacity"] = {
|
|
||||||
key: value for key, value in payload["capacity"].items() if key != "free_bytes"
|
|
||||||
}
|
|
||||||
digest = hashlib.sha256(
|
|
||||||
json.dumps(payload, sort_keys=True, ensure_ascii=False, default=str).encode("utf-8")
|
|
||||||
).hexdigest()
|
|
||||||
return f"{TOKEN_PREFIX}:{digest}"
|
|
||||||
@@ -8,11 +8,6 @@ an EXIF-only edit reuses the file while a real pixel change invalidates it; writ
|
|||||||
are atomic and the cache is bounded by an LRU quota. Failures are persisted as typed
|
are atomic and the cache is bounded by an LRU quota. Failures are persisted as typed
|
||||||
errors so a broken original is not retried on every request.
|
errors so a broken original is not retried on every request.
|
||||||
|
|
||||||
An archived asset is served from its medium when that medium is mounted, and from
|
|
||||||
its *protected* preview when it is not (US06-03). Protected previews are evidence,
|
|
||||||
not cache: the quota never evicts them, because the original they describe may be
|
|
||||||
unreachable when duplicate review needs it.
|
|
||||||
|
|
||||||
Reuses photo_analyzer.prepare_image decode/resize/HEIC handling, adding the missing
|
Reuses photo_analyzer.prepare_image decode/resize/HEIC handling, adding the missing
|
||||||
EXIF-orientation step, WebP output, and a managed cache (donor_ledger.yaml:
|
EXIF-orientation step, WebP output, and a managed cache (donor_ledger.yaml:
|
||||||
pa-imaging).
|
pa-imaging).
|
||||||
@@ -31,7 +26,6 @@ from sqlalchemy.orm import sessionmaker
|
|||||||
from photo_pipeline import path_policy
|
from photo_pipeline import path_policy
|
||||||
from photo_pipeline.config import Config
|
from photo_pipeline.config import Config
|
||||||
from photo_pipeline.models import Asset, Thumbnail
|
from photo_pipeline.models import Asset, Thumbnail
|
||||||
from photo_pipeline.services import availability
|
|
||||||
|
|
||||||
# Best-effort HEIC support: registered only if the optional decoder is installed.
|
# Best-effort HEIC support: registered only if the optional decoder is installed.
|
||||||
try: # pragma: no cover - depends on an optional native dependency
|
try: # pragma: no cover - depends on an optional native dependency
|
||||||
@@ -44,8 +38,6 @@ except Exception: # pragma: no cover
|
|||||||
SIZES = (256, 512, 1280)
|
SIZES = (256, 512, 1280)
|
||||||
THUMB_VERSION = 1
|
THUMB_VERSION = 1
|
||||||
THUMB_FORMAT = "webp"
|
THUMB_FORMAT = "webp"
|
||||||
# The size kept as durable comparison evidence for archived assets (concept §9).
|
|
||||||
PROTECTED_SIZE = 1280
|
|
||||||
|
|
||||||
|
|
||||||
class ThumbnailError(RuntimeError):
|
class ThumbnailError(RuntimeError):
|
||||||
@@ -94,12 +86,7 @@ class ThumbnailService:
|
|||||||
self._config = config
|
self._config = config
|
||||||
self._cache_dir = config.thumbnail_cache_dir
|
self._cache_dir = config.thumbnail_cache_dir
|
||||||
|
|
||||||
def generate(
|
def generate(self, asset_id: str, size: int) -> Path:
|
||||||
self, asset_id: str, size: int, *, protected: bool = False, source: Path | None = None
|
|
||||||
) -> Path:
|
|
||||||
"""Render (or reuse) a preview. ``source`` overrides where the bytes are read
|
|
||||||
from — the archiver passes its verified archive copy, which the database does
|
|
||||||
not yet point at while the transfer is still in flight."""
|
|
||||||
if size not in SIZES:
|
if size not in SIZES:
|
||||||
raise InvalidSize(f"size must be one of {SIZES}")
|
raise InvalidSize(f"size must be one of {SIZES}")
|
||||||
|
|
||||||
@@ -107,7 +94,9 @@ class ThumbnailService:
|
|||||||
asset = session.get(Asset, asset_id)
|
asset = session.get(Asset, asset_id)
|
||||||
if asset is None:
|
if asset is None:
|
||||||
raise ThumbnailNotFound(f"unknown asset {asset_id}")
|
raise ThumbnailNotFound(f"unknown asset {asset_id}")
|
||||||
archived = asset.availability_state in availability.ARCHIVED
|
if asset.availability_state != "active" or not asset.current_path:
|
||||||
|
raise ThumbnailUnavailable(f"asset {asset_id} has no active file")
|
||||||
|
self._validate_path(asset.current_path)
|
||||||
cache_key = self._cache_key(asset, size)
|
cache_key = self._cache_key(asset, size)
|
||||||
|
|
||||||
row = session.get(Thumbnail, cache_key)
|
row = session.get(Thumbnail, cache_key)
|
||||||
@@ -118,21 +107,9 @@ class ThumbnailService:
|
|||||||
)
|
)
|
||||||
if row.path and Path(row.path).exists():
|
if row.path and Path(row.path).exists():
|
||||||
_touch(row.path)
|
_touch(row.path)
|
||||||
if protected and not row.protected:
|
|
||||||
self._protect(cache_key)
|
|
||||||
return Path(row.path)
|
return Path(row.path)
|
||||||
|
|
||||||
# An archived original is read from its medium; when that medium is not
|
source = asset.current_path
|
||||||
# mounted the retained preview above is the only evidence there is.
|
|
||||||
source = source or availability.readable_path(session, asset)
|
|
||||||
if source is None:
|
|
||||||
raise ThumbnailUnavailable(f"asset {asset_id} has no readable file")
|
|
||||||
source = str(source)
|
|
||||||
if source == asset.current_path:
|
|
||||||
# Render the *resolved* path the check approved: revalidating and then
|
|
||||||
# reopening the original name would let a symlink swapped in between
|
|
||||||
# the two steps decide which bytes are served (US07-02).
|
|
||||||
source = str(self._validate_path(source)) # archive roots lie outside
|
|
||||||
|
|
||||||
# Rendering happens outside the DB session (no transaction held during I/O).
|
# Rendering happens outside the DB session (no transaction held during I/O).
|
||||||
try:
|
try:
|
||||||
@@ -142,72 +119,25 @@ class ThumbnailService:
|
|||||||
self._record_error(cache_key, asset_id, size, error.code)
|
self._record_error(cache_key, asset_id, size, error.code)
|
||||||
raise
|
raise
|
||||||
|
|
||||||
# Archived assets keep their preview permanently: it is the comparison
|
self._record_ready(cache_key, asset_id, size, rendered)
|
||||||
# evidence that survives the original leaving active storage.
|
|
||||||
self._record_ready(cache_key, asset_id, size, rendered, protected=protected or archived)
|
|
||||||
self._enforce_quota(keep=rendered["path"])
|
self._enforce_quota(keep=rendered["path"])
|
||||||
return Path(rendered["path"])
|
return Path(rendered["path"])
|
||||||
|
|
||||||
def ensure_protected(self, asset_id: str, *, source: Path | None = None) -> dict:
|
|
||||||
"""Produce (or confirm) the durable comparison preview for an asset.
|
|
||||||
|
|
||||||
Returns evidence rather than raising, because the caller — archive
|
|
||||||
preflight and the transfer itself — decides what an unrenderable original
|
|
||||||
means. ``unsupported`` is a recorded property of the file, not a failure of
|
|
||||||
the policy: its hashes and metadata remain the comparison evidence.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
path = self.generate(asset_id, PROTECTED_SIZE, protected=True, source=source)
|
|
||||||
except tuple(_PERSISTED_ERRORS) as error:
|
|
||||||
return {"state": "unsupported", "error_code": error.code, "path": None}
|
|
||||||
except ThumbnailError as error:
|
|
||||||
return {"state": "unavailable", "error_code": error.code, "path": None}
|
|
||||||
return {"state": "ready", "error_code": None, "path": str(path)}
|
|
||||||
|
|
||||||
def evidence(self, asset_id: str) -> dict:
|
|
||||||
"""What durable preview this asset has right now, without rendering."""
|
|
||||||
with self._session_factory() as session:
|
|
||||||
rows = list(
|
|
||||||
session.execute(
|
|
||||||
select(Thumbnail).where(Thumbnail.asset_id == asset_id)
|
|
||||||
).scalars()
|
|
||||||
)
|
|
||||||
for row in rows:
|
|
||||||
if row.state == "ready" and row.path and Path(row.path).exists():
|
|
||||||
return {
|
|
||||||
"state": "ready",
|
|
||||||
"protected": bool(row.protected),
|
|
||||||
"size": row.size,
|
|
||||||
"error_code": None,
|
|
||||||
}
|
|
||||||
for row in rows:
|
|
||||||
if row.state == "error":
|
|
||||||
return {
|
|
||||||
"state": "unsupported",
|
|
||||||
"protected": False,
|
|
||||||
"size": row.size,
|
|
||||||
"error_code": row.error_code,
|
|
||||||
}
|
|
||||||
return {"state": "missing", "protected": False, "size": None, "error_code": None}
|
|
||||||
|
|
||||||
def _protect(self, cache_key: str) -> None:
|
|
||||||
with self._session_factory() as session:
|
|
||||||
row = session.get(Thumbnail, cache_key)
|
|
||||||
if row is not None:
|
|
||||||
row.protected = True
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
# ── path safety ──────────────────────────────────────────────────────────
|
# ── path safety ──────────────────────────────────────────────────────────
|
||||||
def _validate_path(self, current_path: str) -> Path:
|
def _validate_path(self, current_path: str) -> None:
|
||||||
"""The resolved path to read, or ``PathNotAllowed``.
|
path = Path(current_path)
|
||||||
|
if path_policy.is_excluded(path):
|
||||||
The message names no path: a refusal is returned to the browser, and where
|
raise PathNotAllowed(f"excluded path: {current_path}")
|
||||||
the library lives is not the caller's business (US07-02).
|
roots = self._config.library_roots
|
||||||
"""
|
if not roots:
|
||||||
|
return
|
||||||
|
for root in roots:
|
||||||
try:
|
try:
|
||||||
return path_policy.resolve_in_roots(self._config.library_roots, current_path)
|
path_policy.resolve_within(Path(root), path)
|
||||||
except path_policy.PathPolicyError as error:
|
return
|
||||||
raise PathNotAllowed(str(error)) from None
|
except path_policy.PathPolicyError:
|
||||||
|
continue
|
||||||
|
raise PathNotAllowed(f"path outside configured roots: {current_path}")
|
||||||
|
|
||||||
# ── cache key + rendering ──────────────────────────────────────────────────
|
# ── cache key + rendering ──────────────────────────────────────────────────
|
||||||
@staticmethod
|
@staticmethod
|
||||||
@@ -254,9 +184,7 @@ class ThumbnailService:
|
|||||||
}
|
}
|
||||||
|
|
||||||
# ── persistence ────────────────────────────────────────────────────────────
|
# ── persistence ────────────────────────────────────────────────────────────
|
||||||
def _record_ready(
|
def _record_ready(self, cache_key: str, asset_id: str, size: int, rendered: dict) -> None:
|
||||||
self, cache_key: str, asset_id: str, size: int, rendered: dict, *, protected: bool = False
|
|
||||||
) -> None:
|
|
||||||
with self._session_factory() as session:
|
with self._session_factory() as session:
|
||||||
session.merge(
|
session.merge(
|
||||||
Thumbnail(
|
Thumbnail(
|
||||||
@@ -269,7 +197,6 @@ class ThumbnailService:
|
|||||||
width=rendered["width"],
|
width=rendered["width"],
|
||||||
height=rendered["height"],
|
height=rendered["height"],
|
||||||
format=rendered["format"],
|
format=rendered["format"],
|
||||||
protected=protected,
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
@@ -303,10 +230,6 @@ class ThumbnailService:
|
|||||||
if total <= quota:
|
if total <= quota:
|
||||||
return
|
return
|
||||||
files.sort(key=lambda f: f.stat().st_mtime) # least-recently-used first
|
files.sort(key=lambda f: f.stat().st_mtime) # least-recently-used first
|
||||||
# Protected previews are evidence, not cache: an archived original cannot be
|
|
||||||
# re-rendered once its medium is away, so eviction never touches them.
|
|
||||||
protected = self._protected_paths()
|
|
||||||
files = [f for f in files if str(f) not in protected]
|
|
||||||
keep_path = str(Path(keep)) if keep else None
|
keep_path = str(Path(keep)) if keep else None
|
||||||
evicted: list[str] = []
|
evicted: list[str] = []
|
||||||
for f in files:
|
for f in files:
|
||||||
@@ -324,16 +247,6 @@ class ThumbnailService:
|
|||||||
if evicted:
|
if evicted:
|
||||||
self._forget(evicted)
|
self._forget(evicted)
|
||||||
|
|
||||||
def _protected_paths(self) -> set[str]:
|
|
||||||
with self._session_factory() as session:
|
|
||||||
return {
|
|
||||||
row.path
|
|
||||||
for row in session.execute(
|
|
||||||
select(Thumbnail).where(Thumbnail.protected.is_(True))
|
|
||||||
).scalars()
|
|
||||||
if row.path
|
|
||||||
}
|
|
||||||
|
|
||||||
def _forget(self, paths: list[str]) -> None:
|
def _forget(self, paths: list[str]) -> None:
|
||||||
with self._session_factory() as session:
|
with self._session_factory() as session:
|
||||||
rows = session.execute(
|
rows = session.execute(
|
||||||
|
|||||||
@@ -24,8 +24,6 @@ test = [
|
|||||||
|
|
||||||
[tool.ruff]
|
[tool.ruff]
|
||||||
line-length = 100
|
line-length = 100
|
||||||
# The frozen CLI archive is evidence, not code under maintenance (US07-01).
|
|
||||||
extend-exclude = ["legacy_cli_archive"]
|
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
testpaths = ["tests"]
|
testpaths = ["tests"]
|
||||||
|
|||||||
@@ -14,11 +14,6 @@ import pytest
|
|||||||
from PIL import Image
|
from PIL import Image
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
# The donors are frozen in the read-only archive (US07-01). Only this suite — and
|
|
||||||
# the parity test that compares against them — puts that directory on sys.path;
|
|
||||||
# production never does, which tests/unit/test_legacy_archive.py enforces.
|
|
||||||
ARCHIVED_SOURCES = REPO / "legacy_cli_archive" / "src"
|
|
||||||
sys.path.insert(0, str(ARCHIVED_SOURCES))
|
|
||||||
sys.path.insert(0, str(REPO))
|
sys.path.insert(0, str(REPO))
|
||||||
|
|
||||||
EXIFTOOL = shutil.which("exiftool")
|
EXIFTOOL = shutil.which("exiftool")
|
||||||
|
|||||||
@@ -1,20 +1,13 @@
|
|||||||
"""Ledger lint (US01-01, extended by US07-01).
|
"""Ledger lint (US01-01): every donor-ledger row must carry a real source
|
||||||
|
reference, a target location, and either existing characterization test IDs or
|
||||||
Every donor-ledger row must carry a real source reference — now inside the frozen
|
a real pending backlog story."""
|
||||||
archive — a target location, and either existing characterization test IDs or a
|
|
||||||
real pending backlog story. Since archival (US07-01) a row may also be ``resolved``:
|
|
||||||
its replacement has shipped, ``parity`` names tests that exist and prove it, and
|
|
||||||
``delta`` states every intentional difference. Nothing may quietly become
|
|
||||||
"finished" without one of those two."""
|
|
||||||
import re
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import yaml
|
import yaml
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
ARCHIVE = REPO / "legacy_cli_archive"
|
LEDGER = REPO / "donor_ledger.yaml"
|
||||||
LEDGER = ARCHIVE / "donor_ledger.yaml"
|
|
||||||
ARCHIVED_SOURCES = ARCHIVE / "src"
|
|
||||||
STORIES = REPO / "delivery_backlog" / "stories"
|
STORIES = REPO / "delivery_backlog" / "stories"
|
||||||
TESTS_DIR = Path(__file__).resolve().parent
|
TESTS_DIR = Path(__file__).resolve().parent
|
||||||
|
|
||||||
@@ -22,7 +15,7 @@ CLASSIFICATIONS = {"reuse", "extract", "refactor", "replace"}
|
|||||||
REQUIRED_AREAS = {"discovery", "hashing", "imaging", "nsfw", "vision", "exif",
|
REQUIRED_AREAS = {"discovery", "hashing", "imaging", "nsfw", "vision", "exif",
|
||||||
"database", "ui", "configuration", "logging", "cancellation",
|
"database", "ui", "configuration", "logging", "cancellation",
|
||||||
"error"}
|
"error"}
|
||||||
STATUSES = {"characterized", "resolved", "pending"}
|
STATUSES = {"characterized", "pending"}
|
||||||
|
|
||||||
|
|
||||||
def load_rows():
|
def load_rows():
|
||||||
@@ -56,11 +49,10 @@ def test_rows_have_required_fields_and_unique_ids():
|
|||||||
|
|
||||||
|
|
||||||
def test_source_references_resolve():
|
def test_source_references_resolve():
|
||||||
"""Source paths are relative to the archive: the donors moved there, whole."""
|
|
||||||
for r in load_rows():
|
for r in load_rows():
|
||||||
src = r["source"]
|
src = r["source"]
|
||||||
f = ARCHIVED_SOURCES / src["file"]
|
f = REPO / src["file"]
|
||||||
assert f.is_file(), f"{r['id']}: source file {src['file']} missing from the archive"
|
assert f.is_file(), f"{r['id']}: source file {src['file']} missing"
|
||||||
text = f.read_text(encoding="utf-8")
|
text = f.read_text(encoding="utf-8")
|
||||||
for sym in src["symbols"]:
|
for sym in src["symbols"]:
|
||||||
assert sym in text, f"{r['id']}: symbol {sym!r} not found in {src['file']}"
|
assert sym in text, f"{r['id']}: symbol {sym!r} not found in {src['file']}"
|
||||||
@@ -71,9 +63,7 @@ def test_rows_have_tests_or_pending_story():
|
|||||||
for r in load_rows():
|
for r in load_rows():
|
||||||
tests = r.get("tests", [])
|
tests = r.get("tests", [])
|
||||||
pending = r.get("pending_story")
|
pending = r.get("pending_story")
|
||||||
parity = r.get("parity", [])
|
assert tests or pending, f"{r['id']}: neither tests nor pending_story"
|
||||||
assert tests or pending or parity, \
|
|
||||||
f"{r['id']}: neither characterization tests, parity tests, nor a pending story"
|
|
||||||
for t in tests:
|
for t in tests:
|
||||||
assert t in known_tests, f"{r['id']}: unknown test id {t}"
|
assert t in known_tests, f"{r['id']}: unknown test id {t}"
|
||||||
if pending:
|
if pending:
|
||||||
@@ -83,58 +73,16 @@ def test_rows_have_tests_or_pending_story():
|
|||||||
assert tests, f"{r['id']}: characterized rows need test ids"
|
assert tests, f"{r['id']}: characterized rows need test ids"
|
||||||
|
|
||||||
|
|
||||||
def test_resolved_rows_name_their_parity_or_their_delta():
|
|
||||||
"""A resolved row is a claim that the behavior is handled. It has to say how:
|
|
||||||
tests that prove the replacement, or a stated difference — usually both."""
|
|
||||||
for r in load_rows():
|
|
||||||
if r["status"] != "resolved":
|
|
||||||
assert "parity" not in r, f"{r['id']}: parity on a non-resolved row"
|
|
||||||
continue
|
|
||||||
parity = r.get("parity", [])
|
|
||||||
delta = r.get("delta")
|
|
||||||
assert parity or delta, f"{r['id']}: resolved without parity tests or a delta"
|
|
||||||
if delta:
|
|
||||||
assert len(str(delta).strip()) >= 20, f"{r['id']}: delta too thin to be a reason"
|
|
||||||
for ref in parity:
|
|
||||||
rel, _, func = ref.partition("::")
|
|
||||||
path = REPO / rel
|
|
||||||
assert path.is_file(), f"{r['id']}: parity test file {rel} missing"
|
|
||||||
assert f"def {func}" in path.read_text(encoding="utf-8"), \
|
|
||||||
f"{r['id']}: parity test {ref} not found"
|
|
||||||
|
|
||||||
|
|
||||||
def test_pending_rows_are_the_only_unfinished_work():
|
|
||||||
"""The ledger is the honest list of what has not been carried over: a pending
|
|
||||||
row names the story that will, and that story must still be open work."""
|
|
||||||
for r in load_rows():
|
|
||||||
if r["status"] != "pending":
|
|
||||||
continue
|
|
||||||
story = r.get("pending_story")
|
|
||||||
assert story, f"{r['id']}: pending without a story"
|
|
||||||
assert list(STORIES.glob(f"{story}-*.md")), f"{r['id']}: unknown story {story}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_target_module_exists():
|
|
||||||
"""A row is only finished if the thing it points at is really there."""
|
|
||||||
for r in load_rows():
|
|
||||||
if r["status"] == "pending":
|
|
||||||
continue
|
|
||||||
modules = re.findall(r"photo_pipeline/[\w/]+\.py", str(r["target"]))
|
|
||||||
for module in modules:
|
|
||||||
assert (REPO / module).is_file(), f"{r['id']}: target {module} does not exist"
|
|
||||||
|
|
||||||
|
|
||||||
def test_all_required_areas_covered():
|
def test_all_required_areas_covered():
|
||||||
covered = {r["area"] for r in load_rows()}
|
covered = {r["area"] for r in load_rows()}
|
||||||
assert REQUIRED_AREAS <= covered, f"uncovered areas: {REQUIRED_AREAS - covered}"
|
assert REQUIRED_AREAS <= covered, f"uncovered areas: {REQUIRED_AREAS - covered}"
|
||||||
assert covered <= REQUIRED_AREAS, f"unknown areas: {covered - REQUIRED_AREAS}"
|
assert covered <= REQUIRED_AREAS, f"unknown areas: {covered - REQUIRED_AREAS}"
|
||||||
|
|
||||||
|
|
||||||
def test_every_donor_is_in_the_archive_and_nowhere_else():
|
def test_no_legacy_file_moved():
|
||||||
"""US01-01 forbade archiving before characterization; US07-01 requires it after.
|
# US01-01 explicitly forbids moving/archiving donors; the ledger's source
|
||||||
Each donor exists exactly once — frozen, in the archive."""
|
# files must all still exist at their original locations.
|
||||||
for donor in ("photo_analyzer.py", "nsfwtag/scoring.py", "nsfwtag/exif.py",
|
for donor in ("photo_analyzer.py", "nsfwtag/scoring.py", "nsfwtag/exif.py",
|
||||||
"nsfwtag/server.py", "webapp/query.py", "webapp/runner.py",
|
"nsfwtag/server.py", "webapp/query.py", "webapp/runner.py",
|
||||||
"webapp/server.py"):
|
"webapp/server.py"):
|
||||||
assert (ARCHIVED_SOURCES / donor).is_file(), f"donor missing from archive: {donor}"
|
assert (REPO / donor).is_file(), f"donor moved: {donor}"
|
||||||
assert not (REPO / donor).exists(), f"donor still live at the repo root: {donor}"
|
|
||||||
|
|||||||
@@ -1,15 +1,4 @@
|
|||||||
"""Make the repository root importable for the pipeline test suites, and give every
|
"""Make the repository root importable for the pipeline test suites."""
|
||||||
suite the application session the API requires since US07-02.
|
|
||||||
|
|
||||||
The suites drive the API the way the browser does — module-level ``httpx`` calls and
|
|
||||||
``TestClient`` — so instead of threading a cookie through several hundred call sites,
|
|
||||||
both clients bootstrap the session themselves exactly like ``frontend/js/api.js``:
|
|
||||||
fetch ``/api/v1/session`` once, then send the cookie plus the CSRF header, and
|
|
||||||
re-bootstrap once on 401 (a restarted server issues a new session).
|
|
||||||
|
|
||||||
Security tests deliberately bypass this by constructing their own ``httpx.Client``;
|
|
||||||
only the module-level helpers are wrapped.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -17,85 +6,3 @@ from pathlib import Path
|
|||||||
REPO = Path(__file__).resolve().parents[1]
|
REPO = Path(__file__).resolve().parents[1]
|
||||||
if str(REPO) not in sys.path:
|
if str(REPO) not in sys.path:
|
||||||
sys.path.insert(0, str(REPO))
|
sys.path.insert(0, str(REPO))
|
||||||
|
|
||||||
import httpx
|
|
||||||
import httpx._api # httpx.get/post resolve `request` in this module
|
|
||||||
import pytest
|
|
||||||
from starlette.testclient import TestClient
|
|
||||||
|
|
||||||
from photo_pipeline.api.security import CSRF_HEADER, SESSION_COOKIE
|
|
||||||
|
|
||||||
_SESSIONS: dict[str, tuple[str, str]] = {} # base url -> (session id, csrf token)
|
|
||||||
|
|
||||||
|
|
||||||
def _base(url) -> str:
|
|
||||||
parts = httpx.URL(str(url))
|
|
||||||
return f"{parts.scheme}://{parts.netloc.decode()}"
|
|
||||||
|
|
||||||
|
|
||||||
def _bootstrap(base: str) -> tuple[str, str]:
|
|
||||||
if base not in _SESSIONS:
|
|
||||||
response = httpx.Client(timeout=10).get(f"{base}/api/v1/session")
|
|
||||||
_SESSIONS[base] = (
|
|
||||||
response.cookies.get(SESSION_COOKIE, ""),
|
|
||||||
response.json().get("csrf_token", ""),
|
|
||||||
)
|
|
||||||
return _SESSIONS[base]
|
|
||||||
|
|
||||||
|
|
||||||
def session_client(base_url: str, **kwargs) -> httpx.Client:
|
|
||||||
"""An ``httpx.Client`` that has bootstrapped its own application session.
|
|
||||||
|
|
||||||
For suites that keep one client for a whole journey; it holds its own cookie, so
|
|
||||||
a client made after a server restart picks up the new session by construction.
|
|
||||||
"""
|
|
||||||
client = httpx.Client(base_url=base_url, **kwargs)
|
|
||||||
client.headers[CSRF_HEADER] = client.get("/api/v1/session").json()["csrf_token"]
|
|
||||||
return client
|
|
||||||
|
|
||||||
|
|
||||||
def _authorized(url, headers, cookies):
|
|
||||||
session_id, token = _bootstrap(_base(url))
|
|
||||||
headers = dict(headers or {})
|
|
||||||
headers.setdefault(CSRF_HEADER, token)
|
|
||||||
cookies = dict(cookies or {})
|
|
||||||
cookies.setdefault(SESSION_COOKIE, session_id)
|
|
||||||
return headers, cookies
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True, scope="session")
|
|
||||||
def _api_session():
|
|
||||||
real_request, real_stream = httpx._api.request, httpx._api.stream
|
|
||||||
real_client_request, real_client_init = TestClient.request, TestClient.__init__
|
|
||||||
|
|
||||||
def request(method, url, *, headers=None, cookies=None, **kwargs):
|
|
||||||
sent, jar = _authorized(url, headers, cookies)
|
|
||||||
response = real_request(method, url, headers=sent, cookies=jar, **kwargs)
|
|
||||||
if response.status_code == 401:
|
|
||||||
_SESSIONS.pop(_base(url), None)
|
|
||||||
sent, jar = _authorized(url, headers, cookies)
|
|
||||||
response = real_request(method, url, headers=sent, cookies=jar, **kwargs)
|
|
||||||
return response
|
|
||||||
|
|
||||||
def stream(method, url, *, headers=None, cookies=None, **kwargs):
|
|
||||||
sent, jar = _authorized(url, headers, cookies)
|
|
||||||
return real_stream(method, url, headers=sent, cookies=jar, **kwargs)
|
|
||||||
|
|
||||||
def client_init(self, app, *args, base_url="http://127.0.0.1", **kwargs):
|
|
||||||
# The default "http://testserver" is not a local host, which is exactly what
|
|
||||||
# the Host check refuses; in-process tests are still same-origin callers.
|
|
||||||
real_client_init(self, app, *args, base_url=base_url, **kwargs)
|
|
||||||
|
|
||||||
def client_request(self, method, url, *, headers=None, **kwargs):
|
|
||||||
if CSRF_HEADER not in self.headers:
|
|
||||||
response = real_client_request(self, "GET", "/api/v1/session")
|
|
||||||
self.headers[CSRF_HEADER] = response.json()["csrf_token"]
|
|
||||||
return real_client_request(self, method, url, headers=headers, **kwargs)
|
|
||||||
|
|
||||||
httpx._api.request, httpx.request = request, request
|
|
||||||
httpx._api.stream, httpx.stream = stream, stream
|
|
||||||
TestClient.request, TestClient.__init__ = client_request, client_init
|
|
||||||
yield
|
|
||||||
httpx._api.request, httpx.request = real_request, real_request
|
|
||||||
httpx._api.stream, httpx.stream = real_stream, real_stream
|
|
||||||
TestClient.request, TestClient.__init__ = real_client_request, real_client_init
|
|
||||||
|
|||||||
@@ -393,125 +393,6 @@ def mark_upload_ready(seeded: Seeded, *, unverified: tuple[str, ...] = ()) -> No
|
|||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
|
|
||||||
# ── Phase F: an archivable album and a mountable fake medium ─────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def mark_uploaded(seeded: Seeded, *, album: str = "rome") -> None:
|
|
||||||
"""Give every seeded photo the verified upload evidence archiving requires.
|
|
||||||
|
|
||||||
Archiving refuses anything Immich is not proven to hold, and that proof is an
|
|
||||||
upload batch — recorded here as fixture state so the archive journeys do not
|
|
||||||
have to re-run an upload they are not testing.
|
|
||||||
"""
|
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
from photo_pipeline.models import Asset, UploadBatch, UploadItem
|
|
||||||
from photo_pipeline.services.hashing import sha256_file
|
|
||||||
|
|
||||||
with session_factory(seeded) as sf:
|
|
||||||
with sf() as session:
|
|
||||||
batch_id = str(uuid.uuid4())
|
|
||||||
session.add(
|
|
||||||
UploadBatch(
|
|
||||||
id=batch_id,
|
|
||||||
album=album,
|
|
||||||
folder=str(seeded.lib / album),
|
|
||||||
album_name=album,
|
|
||||||
state="succeeded",
|
|
||||||
preflight_token="v1:e2e",
|
|
||||||
outcome_state="verified",
|
|
||||||
created_at=NOW,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
for asset in session.scalars(select(Asset)):
|
|
||||||
if not asset.current_path:
|
|
||||||
continue
|
|
||||||
session.add(
|
|
||||||
UploadItem(
|
|
||||||
batch_id=batch_id,
|
|
||||||
asset_id=asset.id,
|
|
||||||
path=asset.current_path,
|
|
||||||
sha256=sha256_file(asset.current_path),
|
|
||||||
sha1="0" * 40,
|
|
||||||
state="sent",
|
|
||||||
outcome="uploaded",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
|
|
||||||
class ArchiveStack:
|
|
||||||
"""A seeded, archivable library plus the server, the worker, and a fake medium.
|
|
||||||
|
|
||||||
The medium is an ordinary directory whose marker file makes it identifiable;
|
|
||||||
``unmount()`` takes that marker away, which is exactly what the application sees
|
|
||||||
when an external disk is unplugged.
|
|
||||||
"""
|
|
||||||
|
|
||||||
MARKER = ".photo-pipeline-archive.json"
|
|
||||||
|
|
||||||
def __init__(self, tmp_path: Path, seeded: Seeded) -> None:
|
|
||||||
self.tmp_path = tmp_path
|
|
||||||
self.seeded = seeded
|
|
||||||
self.archive = tmp_path / "archive"
|
|
||||||
self.archive.mkdir(exist_ok=True)
|
|
||||||
self.server: Server | None = None
|
|
||||||
self.worker: subprocess.Popen | None = None
|
|
||||||
self.base = ""
|
|
||||||
|
|
||||||
def start(self, *, worker: bool = True, extra_env: dict[str, str] | None = None) -> "ArchiveStack":
|
|
||||||
env = {"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0", **(extra_env or {})}
|
|
||||||
self.server = Server(self.seeded, extra_env=env).start()
|
|
||||||
self.base = self.server.base
|
|
||||||
if worker:
|
|
||||||
self.worker = start_worker(self.seeded, extra_env=env)
|
|
||||||
return self
|
|
||||||
|
|
||||||
def register(self, name: str = "external") -> dict:
|
|
||||||
response = httpx.post(
|
|
||||||
f"{self.base}/api/v1/archive-locations",
|
|
||||||
json={"name": name, "root": str(self.archive)},
|
|
||||||
timeout=20,
|
|
||||||
)
|
|
||||||
response.raise_for_status()
|
|
||||||
return response.json()
|
|
||||||
|
|
||||||
def unmount(self) -> None:
|
|
||||||
(self.archive / self.MARKER).rename(self.archive / f"{self.MARKER}.away")
|
|
||||||
|
|
||||||
def remount(self) -> None:
|
|
||||||
(self.archive / f"{self.MARKER}.away").rename(self.archive / self.MARKER)
|
|
||||||
|
|
||||||
def plans(self) -> list[dict]:
|
|
||||||
return httpx.get(f"{self.base}/api/v1/archive-plans", timeout=20).json()["plans"]
|
|
||||||
|
|
||||||
def assets(self) -> list[dict]:
|
|
||||||
return httpx.get(
|
|
||||||
f"{self.base}/api/v1/inventory/assets", params={"limit": 200}, timeout=20
|
|
||||||
).json()["items"]
|
|
||||||
|
|
||||||
def restart_server(self) -> None:
|
|
||||||
self.server.stop()
|
|
||||||
self.server.start()
|
|
||||||
|
|
||||||
def restart_worker(self, *, extra_env: dict[str, str] | None = None) -> None:
|
|
||||||
"""Replace the worker — a healthy one after a crashed one, by default."""
|
|
||||||
if self.worker is not None and self.worker.poll() is None:
|
|
||||||
self.worker.kill()
|
|
||||||
self.worker.wait(timeout=10)
|
|
||||||
self.worker = start_worker(
|
|
||||||
self.seeded,
|
|
||||||
extra_env={"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0", **(extra_env or {})},
|
|
||||||
)
|
|
||||||
|
|
||||||
def stop(self) -> None:
|
|
||||||
if self.worker is not None:
|
|
||||||
self.worker.kill()
|
|
||||||
self.worker.wait(timeout=10)
|
|
||||||
if self.server is not None:
|
|
||||||
self.server.stop()
|
|
||||||
|
|
||||||
|
|
||||||
class UploadStack:
|
class UploadStack:
|
||||||
"""A seeded, upload-ready library plus the server, worker, and fake Immich."""
|
"""A seeded, upload-ready library plus the server, worker, and fake Immich."""
|
||||||
|
|
||||||
|
|||||||
@@ -1,302 +0,0 @@
|
|||||||
"""Browser journeys for the archive view (US06-05).
|
|
||||||
|
|
||||||
Archiving is the only stage that removes originals, so these journeys check the
|
|
||||||
two things a browser must never get wrong about it: that the preview names exactly
|
|
||||||
what would leave and where it would go, and that nothing offers an action the
|
|
||||||
server would refuse. The medium is a real directory whose marker makes it
|
|
||||||
identifiable; unmounting it is what an unplugged disk looks like from here.
|
|
||||||
|
|
||||||
Nothing is mocked inside the browser: the transfer runs in the real worker process
|
|
||||||
and the assertions read the filesystem afterwards.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from playwright.sync_api import expect
|
|
||||||
|
|
||||||
from tests.e2e._pipeline_harness import (
|
|
||||||
ArchiveStack,
|
|
||||||
mark_uploaded,
|
|
||||||
seed_album,
|
|
||||||
session_factory,
|
|
||||||
wait_until,
|
|
||||||
)
|
|
||||||
|
|
||||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
|
||||||
|
|
||||||
TIMEOUT = 10
|
|
||||||
RUN_TIMEOUT = 30_000
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def stack(tmp_path):
|
|
||||||
seeded = seed_album(tmp_path)
|
|
||||||
mark_uploaded(seeded)
|
|
||||||
running = ArchiveStack(tmp_path, seeded)
|
|
||||||
try:
|
|
||||||
yield running
|
|
||||||
finally:
|
|
||||||
running.stop()
|
|
||||||
|
|
||||||
|
|
||||||
def _open(page, stack) -> None:
|
|
||||||
page.goto(f"{stack.base}/app/#/archive")
|
|
||||||
page.get_by_test_id("archive-locations").wait_for()
|
|
||||||
|
|
||||||
|
|
||||||
def _archive(page, stack) -> None:
|
|
||||||
"""Confirm the archive and wait for the worker to finish the run."""
|
|
||||||
_open(page, stack)
|
|
||||||
page.get_by_test_id("start-archive").click()
|
|
||||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete", timeout=RUN_TIMEOUT)
|
|
||||||
|
|
||||||
|
|
||||||
def _archived_paths(stack) -> list[str]:
|
|
||||||
return sorted(p.name for p in (stack.archive / "rome").glob("*.jpg"))
|
|
||||||
|
|
||||||
|
|
||||||
# ── preview and confirmation ─────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_preview_names_the_scope_destination_and_reclaimable_bytes(page, stack):
|
|
||||||
errors = []
|
|
||||||
page.on("console", lambda m: errors.append(m.text) if m.type == "error" else None)
|
|
||||||
stack.start(worker=False)
|
|
||||||
location = stack.register()
|
|
||||||
_open(page, stack)
|
|
||||||
|
|
||||||
row = page.get_by_test_id("location-row").first
|
|
||||||
expect(row.get_by_test_id("location-label")).to_have_text("external")
|
|
||||||
expect(row.get_by_test_id("location-media")).to_have_text(location["media_id"])
|
|
||||||
expect(row.get_by_test_id("location-state")).to_have_text("online")
|
|
||||||
|
|
||||||
album = page.get_by_test_id("archive-album-row").first
|
|
||||||
expect(album.get_by_test_id("album-name")).to_have_text("rome")
|
|
||||||
expect(album.get_by_test_id("album-destination")).to_have_text(str(stack.archive / "rome"))
|
|
||||||
expect(album.get_by_test_id("album-assets")).to_have_text("2")
|
|
||||||
expect(album.get_by_test_id("album-state")).to_have_text("ready")
|
|
||||||
# Same filesystem here, so the transfer method is the move path — and it is
|
|
||||||
# named, because copy-verify-remove and move fail differently.
|
|
||||||
expect(album.get_by_test_id("album-method")).to_contain_text("move")
|
|
||||||
|
|
||||||
expect(page.get_by_test_id("destination-identity")).to_contain_text(location["media_id"])
|
|
||||||
expect(page.get_by_test_id("capacity")).to_contain_text("reserve")
|
|
||||||
expect(page.get_by_test_id("start-archive")).to_contain_text("Archive 1 album(s) · reclaim")
|
|
||||||
expect(page.get_by_test_id("start-archive")).to_be_enabled()
|
|
||||||
assert stack.plans() == [], "previewing may not create anything"
|
|
||||||
assert errors == [], f"console errors: {errors}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_offline_medium_blocks_the_confirmation_and_says_what_to_mount(page, stack):
|
|
||||||
stack.start(worker=False)
|
|
||||||
location = stack.register()
|
|
||||||
stack.unmount()
|
|
||||||
_open(page, stack)
|
|
||||||
|
|
||||||
expect(page.get_by_test_id("location-state")).to_have_text("offline")
|
|
||||||
expect(page.get_by_test_id("preflight-blocker").first).to_have_attribute(
|
|
||||||
"data-code", "location_offline"
|
|
||||||
)
|
|
||||||
instruction = page.get_by_test_id("mount-instruction")
|
|
||||||
expect(instruction).to_contain_text("external")
|
|
||||||
expect(instruction).to_contain_text(location["media_id"])
|
|
||||||
expect(page.get_by_test_id("start-archive")).to_be_disabled()
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_unuploaded_album_is_blocked_with_its_reason(page, stack):
|
|
||||||
# No upload evidence at all: archiving would remove the only copy.
|
|
||||||
seeded = stack.seeded
|
|
||||||
with session_factory(seeded) as sf:
|
|
||||||
from sqlalchemy import delete
|
|
||||||
|
|
||||||
from photo_pipeline.models import UploadItem
|
|
||||||
|
|
||||||
with sf() as session:
|
|
||||||
session.execute(delete(UploadItem))
|
|
||||||
session.commit()
|
|
||||||
stack.start(worker=False)
|
|
||||||
stack.register()
|
|
||||||
_open(page, stack)
|
|
||||||
|
|
||||||
expect(page.get_by_test_id("album-state")).to_have_text("blocked")
|
|
||||||
expect(page.get_by_test_id("album-blocker").first).to_have_attribute(
|
|
||||||
"data-code", "partial_scope"
|
|
||||||
)
|
|
||||||
expect(page.get_by_test_id("start-archive")).to_be_disabled()
|
|
||||||
|
|
||||||
|
|
||||||
# ── running, progress, reload ────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_confirmed_archive_runs_and_separates_transfer_verify_and_removal(page, stack):
|
|
||||||
stack.start()
|
|
||||||
stack.register()
|
|
||||||
_archive(page, stack)
|
|
||||||
|
|
||||||
expect(page.get_by_test_id("count-complete")).to_have_text("complete: 2")
|
|
||||||
expect(page.get_by_test_id("count-failed")).to_have_text("failed: 0")
|
|
||||||
expect(page.get_by_test_id("count-transfer")).to_have_text("transfer: 0")
|
|
||||||
expect(page.get_by_test_id("count-verified")).to_have_text("verified: 0")
|
|
||||||
expect(page.get_by_test_id("count-removing")).to_have_text("removing: 0")
|
|
||||||
expect(page.get_by_test_id("operation-phase").first).to_have_text("complete")
|
|
||||||
|
|
||||||
# The library really lost the originals and the medium really holds them.
|
|
||||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"]
|
|
||||||
assert not (stack.seeded.lib / "rome" / "a.jpg").exists()
|
|
||||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_run_survives_a_reload_because_the_state_is_the_servers(page, stack):
|
|
||||||
stack.start()
|
|
||||||
stack.register()
|
|
||||||
_archive(page, stack)
|
|
||||||
|
|
||||||
page.reload()
|
|
||||||
page.get_by_test_id("plan-detail").wait_for()
|
|
||||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete")
|
|
||||||
expect(page.get_by_test_id("count-complete")).to_have_text("complete: 2")
|
|
||||||
# And the run stays addressable by its own URL.
|
|
||||||
plan_id = stack.plans()[0]["id"]
|
|
||||||
page.goto(f"{stack.base}/app/#/archive?plan={plan_id}")
|
|
||||||
expect(page.get_by_test_id("plan-detail")).to_have_attribute("data-plan", plan_id)
|
|
||||||
|
|
||||||
|
|
||||||
# ── recovery ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_interrupted_transfer_is_shown_with_evidence_and_a_safe_action(page, stack):
|
|
||||||
stack.start(worker=False)
|
|
||||||
stack.register()
|
|
||||||
_open(page, stack)
|
|
||||||
page.get_by_test_id("start-archive").click()
|
|
||||||
expect(page.get_by_test_id("archive-result")).to_be_visible()
|
|
||||||
# No worker ran, so every item is still planned; leave one mid-transfer as a
|
|
||||||
# crash would: intent recorded, nothing published, source intact.
|
|
||||||
_interrupt(stack)
|
|
||||||
|
|
||||||
page.reload()
|
|
||||||
page.get_by_test_id("archive-recovery").wait_for()
|
|
||||||
row = page.get_by_test_id("recovery-row").first
|
|
||||||
expect(row).to_have_attribute("data-classification", "resumable")
|
|
||||||
expect(row.get_by_test_id("recovery-reason")).to_contain_text("source present")
|
|
||||||
page.get_by_test_id("resolve-recovery").click()
|
|
||||||
expect(page.get_by_test_id("archive-recovery")).to_have_count(0)
|
|
||||||
assert (stack.seeded.lib / "rome" / "a.jpg").exists(), "recovery may not move anything"
|
|
||||||
|
|
||||||
|
|
||||||
def test_ambiguous_evidence_offers_no_action_at_all(page, stack):
|
|
||||||
stack.start(worker=False)
|
|
||||||
stack.register()
|
|
||||||
_open(page, stack)
|
|
||||||
page.get_by_test_id("start-archive").click()
|
|
||||||
expect(page.get_by_test_id("archive-result")).to_be_visible()
|
|
||||||
# Journal says the copy is verified, but the medium holds nothing: no evidence
|
|
||||||
# supports either finishing or retrying this item.
|
|
||||||
_interrupt(stack, state="verified")
|
|
||||||
|
|
||||||
page.reload()
|
|
||||||
page.get_by_test_id("archive-recovery").wait_for()
|
|
||||||
expect(page.get_by_test_id("recovery-row").first).to_have_attribute(
|
|
||||||
"data-classification", "manual"
|
|
||||||
)
|
|
||||||
expect(page.get_by_test_id("recovery-manual")).to_be_visible()
|
|
||||||
expect(page.get_by_test_id("resolve-recovery")).to_have_count(0)
|
|
||||||
expect(page.get_by_test_id("no-safe-recovery")).to_be_visible()
|
|
||||||
|
|
||||||
|
|
||||||
def _interrupt(stack, *, state: str = "transferring") -> None:
|
|
||||||
"""Leave the plan's first operation in a non-terminal journal state."""
|
|
||||||
with session_factory(stack.seeded) as sf:
|
|
||||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
|
||||||
|
|
||||||
journal = ArchiveJournal(sf)
|
|
||||||
plan_id = stack.plans()[0]["id"]
|
|
||||||
operation = journal.operations(plan_id)[0]
|
|
||||||
journal.begin(operation["id"], worker_id="crashed", fencing_token=1)
|
|
||||||
if state != "transferring":
|
|
||||||
journal.transition(operation["id"], state, fencing_token=1)
|
|
||||||
|
|
||||||
|
|
||||||
# ── offline browsing and restore ─────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_archived_photos_stay_browsable_while_the_medium_is_away(page, stack):
|
|
||||||
stack.start()
|
|
||||||
stack.register()
|
|
||||||
_archive(page, stack)
|
|
||||||
stack.unmount()
|
|
||||||
|
|
||||||
page.reload()
|
|
||||||
page.get_by_test_id("archived-assets").wait_for()
|
|
||||||
row = page.get_by_test_id("archived-row").first
|
|
||||||
expect(row.get_by_test_id("archived-availability")).to_have_text("archived · medium away")
|
|
||||||
expect(row.get_by_test_id("archived-path")).to_contain_text("rome/")
|
|
||||||
expect(row.get_by_test_id("archived-medium")).to_have_text("external")
|
|
||||||
expect(page.get_by_test_id("mount-instruction")).to_contain_text("external")
|
|
||||||
# The retained protected preview is served even though the original is gone.
|
|
||||||
preview = row.get_by_test_id("archived-preview")
|
|
||||||
assert preview.evaluate("img => img.complete && img.naturalWidth > 0")
|
|
||||||
# Restoring is impossible right now, and says so instead of failing later.
|
|
||||||
expect(page.get_by_test_id("start-restore")).to_be_disabled()
|
|
||||||
|
|
||||||
|
|
||||||
def test_restoring_brings_the_photos_back_without_losing_identity(page, stack):
|
|
||||||
stack.start()
|
|
||||||
stack.register()
|
|
||||||
_archive(page, stack)
|
|
||||||
before = {asset["id"] for asset in stack.assets()}
|
|
||||||
|
|
||||||
page.reload()
|
|
||||||
page.get_by_test_id("restore").wait_for()
|
|
||||||
expect(page.get_by_test_id("restore-row").first.get_by_test_id("restore-destination")).to_contain_text(
|
|
||||||
str(stack.seeded.lib / "rome")
|
|
||||||
)
|
|
||||||
page.get_by_test_id("start-restore").click()
|
|
||||||
wait_until(
|
|
||||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()),
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert {asset["id"] for asset in stack.assets()} == before # same identities
|
|
||||||
assert (stack.seeded.lib / "rome" / "a.jpg").exists()
|
|
||||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"] # the archive copy stays
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_taken_name_is_restored_beside_it_never_over_it(page, stack):
|
|
||||||
stack.start()
|
|
||||||
stack.register()
|
|
||||||
_archive(page, stack)
|
|
||||||
squatter = stack.seeded.lib / "rome" / "a.jpg"
|
|
||||||
squatter.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
squatter.write_bytes(b"a different photo now lives here")
|
|
||||||
|
|
||||||
page.reload()
|
|
||||||
page.get_by_test_id("restore").wait_for()
|
|
||||||
destinations = page.get_by_test_id("restore-destination").all_inner_texts()
|
|
||||||
assert any("(restored)" in text for text in destinations), destinations
|
|
||||||
|
|
||||||
page.get_by_test_id("start-restore").click()
|
|
||||||
wait_until(
|
|
||||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()),
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
assert squatter.read_bytes() == b"a different photo now lives here"
|
|
||||||
assert (stack.seeded.lib / "rome" / "a (restored).jpg").exists()
|
|
||||||
|
|
||||||
|
|
||||||
# ── keyboard ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_whole_archive_can_be_confirmed_from_the_keyboard(page, stack):
|
|
||||||
stack.start()
|
|
||||||
stack.register()
|
|
||||||
_open(page, stack)
|
|
||||||
|
|
||||||
button = page.get_by_test_id("start-archive")
|
|
||||||
button.focus()
|
|
||||||
expect(button).to_be_focused()
|
|
||||||
page.keyboard.press("Enter")
|
|
||||||
|
|
||||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete", timeout=RUN_TIMEOUT)
|
|
||||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"]
|
|
||||||
@@ -22,8 +22,6 @@ import numpy as np
|
|||||||
import pytest
|
import pytest
|
||||||
from PIL import Image
|
from PIL import Image
|
||||||
|
|
||||||
from tests.conftest import session_client
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
|
||||||
|
|
||||||
@@ -94,7 +92,7 @@ class ServerController:
|
|||||||
pytest.fail(f"server exited: {err.decode(errors='replace')}")
|
pytest.fail(f"server exited: {err.decode(errors='replace')}")
|
||||||
try:
|
try:
|
||||||
if httpx.get(f"{self.base}/api/v1/health/ready", timeout=1).status_code == 200:
|
if httpx.get(f"{self.base}/api/v1/health/ready", timeout=1).status_code == 200:
|
||||||
self.client = session_client(self.base, timeout=10)
|
self.client = httpx.Client(base_url=self.base, timeout=10)
|
||||||
return
|
return
|
||||||
except httpx.HTTPError:
|
except httpx.HTTPError:
|
||||||
time.sleep(0.2)
|
time.sleep(0.2)
|
||||||
|
|||||||
@@ -1,505 +0,0 @@
|
|||||||
"""Phase F end-to-end acceptance (US06-06): archive lifecycle, black box.
|
|
||||||
|
|
||||||
Every journey drives a real ``photo_pipeline serve`` child process and a real
|
|
||||||
durable worker over HTTP — register a medium, preflight it, archive an album,
|
|
||||||
crash mid-transfer, recover, unmount the medium, rediscover the photos through
|
|
||||||
their hashes while they are unreachable, mount it again, restore, and collide.
|
|
||||||
Nothing is reached into: the medium is an ordinary directory whose marker file is
|
|
||||||
its identity, and unmounting it means taking that marker away, which is what the
|
|
||||||
application sees when a disk is unplugged.
|
|
||||||
|
|
||||||
Three invariants are asserted wherever they apply, because they are what make the
|
|
||||||
only *removing* stage safe:
|
|
||||||
|
|
||||||
- **No source is removed before its archive copy is verified.** Every crash barrier
|
|
||||||
is checked for the file still being somewhere: either in the library, or on the
|
|
||||||
medium hashing exactly as recorded.
|
|
||||||
- **Archived is not missing.** An unmounted medium leaves its photos
|
|
||||||
``archived_offline``, still hashed, still deduplicable, still previewable.
|
|
||||||
- **Restoring never overwrites.** A taken name comes back beside its occupant, and
|
|
||||||
the archived copy stays on the medium.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import shutil
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from tests.e2e._pipeline_harness import (
|
|
||||||
ArchiveStack,
|
|
||||||
mark_uploaded,
|
|
||||||
seed_album,
|
|
||||||
session_factory,
|
|
||||||
wait_until,
|
|
||||||
)
|
|
||||||
|
|
||||||
pytestmark = pytest.mark.phase_f
|
|
||||||
|
|
||||||
TIMEOUT = 20
|
|
||||||
ALBUM = "rome"
|
|
||||||
MANIFEST = "archive-manifest.jsonl"
|
|
||||||
# Every persisted transition the transfer can die at, in the order it reaches them.
|
|
||||||
BARRIERS = ["transferring", "verified", "removing", "source_removed", "complete"]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def stack(tmp_path):
|
|
||||||
"""An analysed album with verified upload evidence — archivable, nothing else."""
|
|
||||||
seeded = seed_album(tmp_path)
|
|
||||||
mark_uploaded(seeded)
|
|
||||||
running = ArchiveStack(tmp_path, seeded)
|
|
||||||
try:
|
|
||||||
yield running
|
|
||||||
finally:
|
|
||||||
running.stop()
|
|
||||||
|
|
||||||
|
|
||||||
# ── helpers ──────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def _preflight(stack, location_id: str, **body) -> dict:
|
|
||||||
response = httpx.post(
|
|
||||||
f"{stack.base}/api/v1/archive-preflight",
|
|
||||||
json={"location_id": location_id, **body},
|
|
||||||
timeout=TIMEOUT,
|
|
||||||
)
|
|
||||||
response.raise_for_status()
|
|
||||||
return response.json()
|
|
||||||
|
|
||||||
|
|
||||||
def _plan(stack, location_id: str, **body) -> dict:
|
|
||||||
report = _preflight(stack, location_id, **body)
|
|
||||||
response = httpx.post(
|
|
||||||
f"{stack.base}/api/v1/archive-plans",
|
|
||||||
json={"location_id": location_id, "token": report["token"], **body},
|
|
||||||
timeout=TIMEOUT,
|
|
||||||
)
|
|
||||||
response.raise_for_status()
|
|
||||||
return response.json()
|
|
||||||
|
|
||||||
|
|
||||||
def _apply(stack, plan_id: str) -> httpx.Response:
|
|
||||||
return httpx.post(f"{stack.base}/api/v1/archive-plans/{plan_id}/apply", timeout=TIMEOUT)
|
|
||||||
|
|
||||||
|
|
||||||
def _plan_state(stack, plan_id: str) -> dict:
|
|
||||||
return httpx.get(f"{stack.base}/api/v1/archive-plans/{plan_id}", timeout=TIMEOUT).json()
|
|
||||||
|
|
||||||
|
|
||||||
def _await_plan(stack, plan_id: str, states=("complete", "failed"), *, timeout: float = 60) -> dict:
|
|
||||||
return wait_until(
|
|
||||||
lambda: (lambda p: p if p.get("state") in states else None)(_plan_state(stack, plan_id)),
|
|
||||||
timeout=timeout,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _archive_album(stack) -> dict:
|
|
||||||
"""Register, preflight, plan, apply, and wait for the run to finish."""
|
|
||||||
location = stack.register()
|
|
||||||
plan = _plan(stack, location["id"])
|
|
||||||
_apply(stack, plan["id"]).raise_for_status()
|
|
||||||
return {"location": location, "plan": _await_plan(stack, plan["id"])}
|
|
||||||
|
|
||||||
|
|
||||||
def _assets(stack) -> dict[str, dict]:
|
|
||||||
return {asset["id"]: asset for asset in stack.assets()}
|
|
||||||
|
|
||||||
|
|
||||||
def _recovery(stack) -> dict:
|
|
||||||
return httpx.get(f"{stack.base}/api/v1/archive-recovery", timeout=TIMEOUT).json()
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve(stack) -> dict:
|
|
||||||
response = httpx.post(f"{stack.base}/api/v1/archive-recovery/resolve", timeout=TIMEOUT)
|
|
||||||
response.raise_for_status()
|
|
||||||
return response.json()
|
|
||||||
|
|
||||||
|
|
||||||
def _restore(stack, location_id: str) -> dict:
|
|
||||||
report = httpx.post(
|
|
||||||
f"{stack.base}/api/v1/restore-preflight",
|
|
||||||
json={"location_id": location_id},
|
|
||||||
timeout=TIMEOUT,
|
|
||||||
).json()
|
|
||||||
plan = httpx.post(
|
|
||||||
f"{stack.base}/api/v1/restore-plans",
|
|
||||||
json={"location_id": location_id, "token": report["token"]},
|
|
||||||
timeout=TIMEOUT,
|
|
||||||
)
|
|
||||||
plan.raise_for_status()
|
|
||||||
plan = plan.json()
|
|
||||||
httpx.post(
|
|
||||||
f"{stack.base}/api/v1/restore-plans/{plan['id']}/apply", timeout=TIMEOUT
|
|
||||||
).raise_for_status()
|
|
||||||
wait_until(
|
|
||||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()), timeout=60
|
|
||||||
)
|
|
||||||
return {"preflight": report, "plan": plan}
|
|
||||||
|
|
||||||
|
|
||||||
def _sha256(path: Path) -> str:
|
|
||||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _contents(*roots: Path) -> set[str]:
|
|
||||||
"""Every photo byte-string reachable anywhere, keyed by hash."""
|
|
||||||
return {
|
|
||||||
_sha256(path)
|
|
||||||
for root in roots
|
|
||||||
for path in root.rglob("*.jpg")
|
|
||||||
if path.is_file() and not path.name.startswith(".")
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _manifest(stack) -> list[dict]:
|
|
||||||
path = stack.archive / ALBUM / MANIFEST
|
|
||||||
if not path.exists():
|
|
||||||
return []
|
|
||||||
return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
|
|
||||||
|
|
||||||
|
|
||||||
# ── preflight blockers ───────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_preflight_refuses_an_offline_medium_a_wrong_volume_and_a_full_disk(stack):
|
|
||||||
stack.start(worker=False)
|
|
||||||
location = stack.register()
|
|
||||||
|
|
||||||
ready = _preflight(stack, location["id"])
|
|
||||||
assert ready["state"] == "ready"
|
|
||||||
assert ready["totals"]["assets"] == 2
|
|
||||||
|
|
||||||
stack.unmount()
|
|
||||||
offline = _preflight(stack, location["id"])
|
|
||||||
assert offline["state"] == "blocked"
|
|
||||||
assert "location_offline" in {issue["code"] for issue in offline["blockers"]}
|
|
||||||
|
|
||||||
# A different disk mounted at the same place is not this medium.
|
|
||||||
(stack.archive / ArchiveStack.MARKER).write_text('{"media_id": "another-disk"}')
|
|
||||||
wrong = _preflight(stack, location["id"])
|
|
||||||
assert "wrong_volume" in {issue["code"] for issue in wrong["blockers"]}
|
|
||||||
|
|
||||||
# Nothing has been created, moved, or removed by any of that.
|
|
||||||
assert stack.plans() == []
|
|
||||||
assert sorted(p.name for p in (stack.seeded.lib / ALBUM).glob("*.jpg")) == ["a.jpg", "b.jpg"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_preflight_refuses_capacity_it_cannot_prove_and_bytes_it_cannot_vouch_for(tmp_path):
|
|
||||||
seeded = seed_album(tmp_path)
|
|
||||||
mark_uploaded(seeded)
|
|
||||||
stack = ArchiveStack(tmp_path, seeded)
|
|
||||||
try:
|
|
||||||
# A reserve larger than any disk: the scope cannot be promised room.
|
|
||||||
stack.start(
|
|
||||||
worker=False,
|
|
||||||
extra_env={"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": str(1 << 62)},
|
|
||||||
)
|
|
||||||
location = stack.register()
|
|
||||||
report = _preflight(stack, location["id"])
|
|
||||||
assert "insufficient_capacity" in {issue["code"] for issue in report["blockers"]}
|
|
||||||
|
|
||||||
# And a plan for a blocked scope is refused rather than half-created.
|
|
||||||
refused = httpx.post(
|
|
||||||
f"{stack.base}/api/v1/archive-plans",
|
|
||||||
json={"location_id": location["id"], "token": report["token"]},
|
|
||||||
timeout=TIMEOUT,
|
|
||||||
)
|
|
||||||
assert refused.status_code == 422
|
|
||||||
assert refused.json()["error"]["code"] == "blocked"
|
|
||||||
assert stack.plans() == []
|
|
||||||
finally:
|
|
||||||
stack.stop()
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_album_whose_bytes_changed_after_upload_cannot_be_archived(stack):
|
|
||||||
stack.start(worker=False)
|
|
||||||
location = stack.register()
|
|
||||||
(stack.seeded.lib / ALBUM / "a.jpg").write_bytes(b"edited after the upload")
|
|
||||||
|
|
||||||
report = _preflight(stack, location["id"])
|
|
||||||
album = report["albums"][0]
|
|
||||||
assert album["state"] == "blocked"
|
|
||||||
assert "partial_scope" in {issue["code"] for issue in album["blockers"]}
|
|
||||||
assert "bytes_changed" in {
|
|
||||||
issue["code"] for asset in album["assets"] for issue in asset["blockers"]
|
|
||||||
}
|
|
||||||
assert report["state"] == "blocked"
|
|
||||||
|
|
||||||
|
|
||||||
# ── copy, verify, remove ─────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_archived_album_leaves_the_library_only_after_it_is_verified(stack):
|
|
||||||
stack.start()
|
|
||||||
before = _contents(stack.seeded.lib)
|
|
||||||
hashes = {a["id"]: a["current_sha256"] for a in stack.assets()}
|
|
||||||
|
|
||||||
result = _archive_album(stack)
|
|
||||||
plan = result["plan"]
|
|
||||||
|
|
||||||
assert plan["state"] == "complete"
|
|
||||||
assert {op["journal_state"] for op in plan["operations"]} == {"complete"}
|
|
||||||
# Same filesystem here, so the transfer took the atomic-move path — recorded as
|
|
||||||
# what actually happened, not as the plan's guess.
|
|
||||||
assert all(op["same_filesystem"] for op in plan["operations"])
|
|
||||||
|
|
||||||
# The bytes are on the medium, hashing exactly as recorded, and gone from the
|
|
||||||
# library. Not one photo was lost in between.
|
|
||||||
assert _contents(stack.archive) == before
|
|
||||||
assert not list((stack.seeded.lib / ALBUM).glob("*.jpg"))
|
|
||||||
for asset_id, sha256 in hashes.items():
|
|
||||||
asset = _assets(stack)[asset_id]
|
|
||||||
assert asset["current_path"] is None
|
|
||||||
assert asset["availability_state"] == "archived_online"
|
|
||||||
assert _sha256(stack.archive / asset["archive_path"]) == sha256
|
|
||||||
|
|
||||||
# The medium carries its own record of what it holds.
|
|
||||||
manifest = _manifest(stack)
|
|
||||||
assert {entry["asset_id"] for entry in manifest} == set(hashes)
|
|
||||||
assert {entry["sha256"] for entry in manifest} == set(hashes.values())
|
|
||||||
assert {entry["media_id"] for entry in manifest} == {result["location"]["media_id"]}
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_archived_state_survives_a_server_restart(stack):
|
|
||||||
stack.start()
|
|
||||||
result = _archive_album(stack)
|
|
||||||
before = _assets(stack)
|
|
||||||
plan_before = _plan_state(stack, result["plan"]["id"])
|
|
||||||
|
|
||||||
stack.restart_server()
|
|
||||||
|
|
||||||
assert _assets(stack) == before
|
|
||||||
assert _plan_state(stack, result["plan"]["id"]) == plan_before
|
|
||||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
|
||||||
|
|
||||||
|
|
||||||
# ── interruption at every persisted transition ───────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("barrier", BARRIERS)
|
|
||||||
def test_a_worker_killed_at_each_transition_loses_nothing_and_recovers(tmp_path, barrier):
|
|
||||||
# One photo, so the crash lands on the only operation and the end state is the
|
|
||||||
# recovery's own doing rather than a mixture with untouched work.
|
|
||||||
seeded = seed_album(tmp_path, names=("a.jpg",))
|
|
||||||
mark_uploaded(seeded)
|
|
||||||
stack = ArchiveStack(tmp_path, seeded)
|
|
||||||
try:
|
|
||||||
# Only the worker carries the crash barrier: it dies the instant this journal
|
|
||||||
# state is persisted, while the server stays up to be asked what happened.
|
|
||||||
stack.start(worker=False)
|
|
||||||
stack.restart_worker(extra_env={"PHOTO_PIPELINE_FAULT_AFTER": barrier})
|
|
||||||
before = _contents(stack.seeded.lib)
|
|
||||||
location = stack.register()
|
|
||||||
plan = _plan(stack, location["id"])
|
|
||||||
_apply(stack, plan["id"]).raise_for_status()
|
|
||||||
|
|
||||||
wait_until(lambda: stack.worker.poll() is not None, timeout=60)
|
|
||||||
assert stack.worker.returncode in (9, -9), "the worker should have been killed"
|
|
||||||
|
|
||||||
# Whatever the crash interrupted, every photo is still readable somewhere.
|
|
||||||
assert before <= _contents(stack.seeded.lib, stack.archive), f"content lost at {barrier}"
|
|
||||||
# And nothing was removed that had not been verified first.
|
|
||||||
for operation in _plan_state(stack, plan["id"])["operations"]:
|
|
||||||
source = Path(operation["source_path"])
|
|
||||||
if not source.exists():
|
|
||||||
archived = Path(operation["destination_path"])
|
|
||||||
assert archived.exists(), f"{barrier}: source removed without an archive copy"
|
|
||||||
assert _sha256(archived) == operation["expected_sha256"]
|
|
||||||
|
|
||||||
# Recovery is offered from evidence, resolves without a worker, and converges.
|
|
||||||
verdicts = _recovery(stack)
|
|
||||||
assert all(v["classification"] != "manual" for v in verdicts["operations"]), verdicts
|
|
||||||
_resolve(stack)
|
|
||||||
assert _recovery(stack)["operations"] == []
|
|
||||||
# Repeating it changes nothing.
|
|
||||||
assert _resolve(stack) == {"resumed": 0, "completed": 0, "manual": 0}
|
|
||||||
|
|
||||||
# Recovery finishes what was durable and re-plans what was not. Before the
|
|
||||||
# archive copy existed, that means putting the item back to `planned` with the
|
|
||||||
# original untouched; from `verified` onwards it means completing it.
|
|
||||||
stack.restart_server()
|
|
||||||
operation = _plan_state(stack, plan["id"])["operations"][0]
|
|
||||||
asset = next(iter(stack.assets()))
|
|
||||||
if barrier == "transferring":
|
|
||||||
assert operation["journal_state"] == "planned"
|
|
||||||
assert Path(operation["source_path"]).exists()
|
|
||||||
assert asset["availability_state"] == "active"
|
|
||||||
assert _contents(stack.seeded.lib) == before
|
|
||||||
else:
|
|
||||||
assert operation["journal_state"] == "complete"
|
|
||||||
assert not Path(operation["source_path"]).exists()
|
|
||||||
assert _contents(stack.archive) == before
|
|
||||||
assert asset["availability_state"] == "archived_online"
|
|
||||||
assert _sha256(stack.archive / asset["archive_path"]) == asset["current_sha256"]
|
|
||||||
assert {entry["sha256"] for entry in _manifest(stack)} == before
|
|
||||||
finally:
|
|
||||||
stack.stop()
|
|
||||||
|
|
||||||
|
|
||||||
def test_evidence_that_contradicts_the_journal_is_never_guessed(stack):
|
|
||||||
stack.start(worker=False)
|
|
||||||
location = stack.register()
|
|
||||||
plan = _plan(stack, location["id"])
|
|
||||||
# The journal claims a verified archive copy the medium does not have.
|
|
||||||
with session_factory(stack.seeded) as sf:
|
|
||||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
|
||||||
|
|
||||||
journal = ArchiveJournal(sf)
|
|
||||||
operation = journal.operations(plan["id"])[0]
|
|
||||||
journal.begin(operation["id"], worker_id="crashed", fencing_token=1)
|
|
||||||
journal.transition(operation["id"], "verified", fencing_token=1)
|
|
||||||
|
|
||||||
verdicts = _recovery(stack)
|
|
||||||
assert [v["classification"] for v in verdicts["operations"]] == ["manual"]
|
|
||||||
assert _resolve(stack)["manual"] == 1
|
|
||||||
# The source is untouched and the unresolved item keeps blocking a new archive.
|
|
||||||
assert (stack.seeded.lib / ALBUM / "a.jpg").exists()
|
|
||||||
blocked = _preflight(stack, location["id"])
|
|
||||||
assert "archive_pending" in {issue["code"] for issue in blocked["blockers"]}
|
|
||||||
|
|
||||||
|
|
||||||
# ── offline identity and deduplication ───────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_offline_photos_stay_hashed_previewable_and_deduplicable(stack):
|
|
||||||
stack.start()
|
|
||||||
result = _archive_album(stack)
|
|
||||||
archived = _assets(stack)
|
|
||||||
copy_source = stack.archive / next(iter(archived.values()))["archive_path"]
|
|
||||||
exact = stack.seeded.lib / "inbox" / "again.jpg"
|
|
||||||
exact.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
shutil.copy2(copy_source, exact)
|
|
||||||
stack.unmount()
|
|
||||||
|
|
||||||
scanned = httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60)
|
|
||||||
scanned.raise_for_status()
|
|
||||||
# An unmounted medium is not a missing file.
|
|
||||||
assert scanned.json()["counts"].get("missing") is None
|
|
||||||
offline = {a["id"]: a for a in stack.assets() if a["id"] in archived}
|
|
||||||
assert {a["availability_state"] for a in offline.values()} == {"archived_offline"}
|
|
||||||
assert all(a["current_sha256"] for a in offline.values())
|
|
||||||
|
|
||||||
# The new active copy is recognised as the archived photo, not as a new one.
|
|
||||||
httpx.post(f"{stack.base}/api/v1/duplicates/detect", timeout=60).raise_for_status()
|
|
||||||
clusters = httpx.get(
|
|
||||||
f"{stack.base}/api/v1/duplicates/clusters", params={"limit": 50}, timeout=TIMEOUT
|
|
||||||
).json()["items"]
|
|
||||||
exact_clusters = [c for c in clusters if c["method"] == "exact"]
|
|
||||||
assert len(exact_clusters) == 1
|
|
||||||
detail = httpx.get(
|
|
||||||
f"{stack.base}/api/v1/duplicates/clusters/{exact_clusters[0]['id']}", timeout=TIMEOUT
|
|
||||||
).json()
|
|
||||||
canonical = next(m for m in detail["members"] if m["asset_id"] == detail["canonical_asset_id"])
|
|
||||||
assert canonical["availability_state"] == "archived_offline"
|
|
||||||
assert canonical["archive_location"] == result["location"]["name"]
|
|
||||||
assert canonical["preview"]["state"] == "ready" and canonical["preview"]["protected"]
|
|
||||||
|
|
||||||
# And the retained preview really is served while the medium is away.
|
|
||||||
thumbnail = httpx.get(
|
|
||||||
f"{stack.base}/api/v1/assets/{canonical['asset_id']}/thumbnail",
|
|
||||||
params={"size": 1280},
|
|
||||||
timeout=TIMEOUT,
|
|
||||||
)
|
|
||||||
assert thumbnail.status_code == 200
|
|
||||||
assert thumbnail.headers["content-type"] == "image/webp"
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_fuzzy_offline_match_asks_for_the_medium_instead_of_guessing(stack):
|
|
||||||
stack.start()
|
|
||||||
result = _archive_album(stack)
|
|
||||||
archived = next(iter(_assets(stack).values()))
|
|
||||||
variant = stack.seeded.lib / "inbox" / "resized.jpg"
|
|
||||||
variant.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
_resize(stack.archive / archived["archive_path"], variant)
|
|
||||||
stack.unmount()
|
|
||||||
|
|
||||||
httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60).raise_for_status()
|
|
||||||
httpx.post(f"{stack.base}/api/v1/duplicates/detect", timeout=60).raise_for_status()
|
|
||||||
clusters = httpx.get(
|
|
||||||
f"{stack.base}/api/v1/duplicates/clusters", params={"limit": 50}, timeout=TIMEOUT
|
|
||||||
).json()["items"]
|
|
||||||
perceptual = [c for c in clusters if c["method"] == "perceptual"]
|
|
||||||
assert len(perceptual) == 1
|
|
||||||
|
|
||||||
detail = httpx.get(
|
|
||||||
f"{stack.base}/api/v1/duplicates/clusters/{perceptual[0]['id']}", timeout=TIMEOUT
|
|
||||||
).json()
|
|
||||||
# A fuzzy match is never decided automatically, and full-resolution review names
|
|
||||||
# the medium to mount rather than guessing from the preview.
|
|
||||||
assert detail["state"] == "open" and detail["requires_confirmation"] is True
|
|
||||||
assert detail["mount_required"] == [result["location"]["name"]]
|
|
||||||
|
|
||||||
|
|
||||||
def test_mounting_the_medium_again_makes_the_originals_reachable(stack):
|
|
||||||
stack.start()
|
|
||||||
result = _archive_album(stack)
|
|
||||||
stack.unmount()
|
|
||||||
httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60).raise_for_status()
|
|
||||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_offline"}
|
|
||||||
# Restoring from a medium that is not there is refused, with its reason.
|
|
||||||
away = httpx.post(
|
|
||||||
f"{stack.base}/api/v1/restore-preflight",
|
|
||||||
json={"location_id": result["location"]["id"]},
|
|
||||||
timeout=TIMEOUT,
|
|
||||||
).json()
|
|
||||||
assert away["state"] == "blocked"
|
|
||||||
assert "location_offline" in {issue["code"] for issue in away["blockers"]}
|
|
||||||
|
|
||||||
stack.remount()
|
|
||||||
locations = httpx.get(f"{stack.base}/api/v1/archive-locations", timeout=TIMEOUT).json()
|
|
||||||
assert locations["locations"][0]["state"] == "online"
|
|
||||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
|
||||||
|
|
||||||
|
|
||||||
# ── restore ──────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_restoring_returns_the_bytes_without_losing_identity_or_the_archive(stack):
|
|
||||||
stack.start()
|
|
||||||
result = _archive_album(stack)
|
|
||||||
identities = set(_assets(stack))
|
|
||||||
hashes = {a["id"]: a["current_sha256"] for a in stack.assets()}
|
|
||||||
|
|
||||||
_restore(stack, result["location"]["id"])
|
|
||||||
|
|
||||||
restored = _assets(stack)
|
|
||||||
assert set(restored) == identities # the same photos, not new ones
|
|
||||||
for asset_id, sha256 in hashes.items():
|
|
||||||
asset = restored[asset_id]
|
|
||||||
assert asset["availability_state"] == "active"
|
|
||||||
assert _sha256(Path(asset["current_path"])) == sha256
|
|
||||||
# The archive copy is a copy: restoring empties nothing.
|
|
||||||
assert (stack.archive / asset["archive_path"]).exists()
|
|
||||||
|
|
||||||
stack.restart_server()
|
|
||||||
assert _assets(stack) == restored
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_restore_comes_back_beside_an_occupant_never_over_it(stack):
|
|
||||||
stack.start()
|
|
||||||
result = _archive_album(stack)
|
|
||||||
squatter = stack.seeded.lib / ALBUM / "a.jpg"
|
|
||||||
squatter.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
squatter.write_bytes(b"a different photo lives here now")
|
|
||||||
|
|
||||||
report = _restore(stack, result["location"]["id"])["preflight"]
|
|
||||||
destinations = {item["destination_path"] for item in report["items"]}
|
|
||||||
assert any("(restored)" in destination for destination in destinations)
|
|
||||||
|
|
||||||
assert squatter.read_bytes() == b"a different photo lives here now"
|
|
||||||
assert (stack.seeded.lib / ALBUM / "a (restored).jpg").exists()
|
|
||||||
assert {a["availability_state"] for a in stack.assets()} == {"active"}
|
|
||||||
|
|
||||||
|
|
||||||
def _resize(source: Path, destination: Path, scale: float = 0.5) -> None:
|
|
||||||
from PIL import Image
|
|
||||||
|
|
||||||
with Image.open(source) as opened:
|
|
||||||
opened.resize(
|
|
||||||
(int(opened.width * scale), int(opened.height * scale)), Image.LANCZOS
|
|
||||||
).save(destination, quality=95)
|
|
||||||
@@ -17,9 +17,6 @@ import httpx
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
import pytest
|
import pytest
|
||||||
from PIL import Image
|
from PIL import Image
|
||||||
from playwright.sync_api import expect
|
|
||||||
|
|
||||||
from tests.conftest import session_client
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
|
||||||
@@ -114,7 +111,7 @@ def server(tmp_path):
|
|||||||
proc.terminate()
|
proc.terminate()
|
||||||
pytest.fail("server never became ready")
|
pytest.fail("server never became ready")
|
||||||
|
|
||||||
client = session_client(base, timeout=5)
|
client = httpx.Client(base_url=base, timeout=5)
|
||||||
|
|
||||||
def cluster_by_method(method):
|
def cluster_by_method(method):
|
||||||
clusters = client.get("/api/v1/duplicates/clusters").json()["items"]
|
clusters = client.get("/api/v1/duplicates/clusters").json()["items"]
|
||||||
@@ -151,9 +148,9 @@ def test_fuzzy_decision_requires_confirmation(page, server):
|
|||||||
page.get_by_test_id("confirm").wait_for()
|
page.get_by_test_id("confirm").wait_for()
|
||||||
assert "open" in page.get_by_test_id("cluster-state").inner_text()
|
assert "open" in page.get_by_test_id("cluster-state").inner_text()
|
||||||
page.get_by_test_id("confirm-yes").click()
|
page.get_by_test_id("confirm-yes").click()
|
||||||
# A locator assertion, not wait_for_function: the app's CSP forbids eval, and
|
page.wait_for_function(
|
||||||
# Playwright's polling predicate is evaluated as a string in the page (US07-02).
|
"document.querySelector('[data-testid=cluster-state]').innerText.includes('dismissed')"
|
||||||
expect(page.get_by_test_id("cluster-state")).to_contain_text("dismissed")
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_stale_version_shows_conflict(page, server):
|
def test_stale_version_shows_conflict(page, server):
|
||||||
@@ -179,9 +176,9 @@ def test_decision_persists_after_reload(page, server):
|
|||||||
page.goto(f"{server.base}/app/#/duplicates/{cluster['id']}")
|
page.goto(f"{server.base}/app/#/duplicates/{cluster['id']}")
|
||||||
page.get_by_test_id("not-duplicate").click()
|
page.get_by_test_id("not-duplicate").click()
|
||||||
page.get_by_test_id("confirm-yes").click()
|
page.get_by_test_id("confirm-yes").click()
|
||||||
# A locator assertion, not wait_for_function: the app's CSP forbids eval, and
|
page.wait_for_function(
|
||||||
# Playwright's polling predicate is evaluated as a string in the page (US07-02).
|
"document.querySelector('[data-testid=cluster-state]').innerText.includes('dismissed')"
|
||||||
expect(page.get_by_test_id("cluster-state")).to_contain_text("dismissed")
|
)
|
||||||
page.reload()
|
page.reload()
|
||||||
page.get_by_test_id("cluster-state").wait_for()
|
page.get_by_test_id("cluster-state").wait_for()
|
||||||
assert "dismissed" in page.get_by_test_id("cluster-state").inner_text()
|
assert "dismissed" in page.get_by_test_id("cluster-state").inner_text()
|
||||||
|
|||||||
@@ -1,280 +0,0 @@
|
|||||||
"""US07-02 black-box security tests: authorization and path boundaries.
|
|
||||||
|
|
||||||
Everything here talks to a real ``photo_pipeline serve`` child process over HTTP with
|
|
||||||
its own ``httpx.Client``, deliberately outside the session helper the rest of the
|
|
||||||
suite uses — an attacker does not get a bootstrapped client.
|
|
||||||
|
|
||||||
The threat is a page in the user's browser, not a remote attacker: the app listens on
|
|
||||||
127.0.0.1, so any site the user visits can send requests to it and can point an
|
|
||||||
``<img>`` at its media endpoints. Each journey below is one of those attempts.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
from contextlib import contextmanager
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from tests.e2e._pipeline_harness import Server, image, seed_library
|
|
||||||
|
|
||||||
TIMEOUT = 20
|
|
||||||
SENTINEL_KEY = "immich-sentinel-9f3a2b"
|
|
||||||
SESSION_COOKIE = "pp_session"
|
|
||||||
CSRF_HEADER = "X-CSRF-Token"
|
|
||||||
|
|
||||||
# A GET, a mutation, and a media endpoint: the three shapes the policy must cover.
|
|
||||||
PROTECTED = [
|
|
||||||
("GET", "/api/v1/workflow", None),
|
|
||||||
("POST", "/api/v1/albums/proposals", {}),
|
|
||||||
("GET", "/api/v1/assets/{asset}/thumbnail?size=256", None),
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
|
||||||
def stack(tmp_path_factory):
|
|
||||||
"""One server, one album of two photos, and a credential sentinel in its config."""
|
|
||||||
tmp_path = tmp_path_factory.mktemp("security")
|
|
||||||
seeded = seed_library(tmp_path, {"a": 1, "b": 2}, {"a": "sfw", "b": "sfw"})
|
|
||||||
outside = tmp_path / "outside"
|
|
||||||
outside.mkdir()
|
|
||||||
image(outside / "secret.jpg", 99)
|
|
||||||
server = Server(
|
|
||||||
seeded,
|
|
||||||
extra_env={
|
|
||||||
"PHOTO_PIPELINE_IMMICH_API_KEY": SENTINEL_KEY,
|
|
||||||
"PHOTO_PIPELINE_IMMICH_SERVER_URL": "http://127.0.0.1:1",
|
|
||||||
},
|
|
||||||
).start()
|
|
||||||
yield server, seeded, outside
|
|
||||||
server.stop()
|
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
|
||||||
def anonymous(server):
|
|
||||||
with httpx.Client(base_url=server.base, timeout=TIMEOUT) as client:
|
|
||||||
yield client
|
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
|
||||||
def authenticated(server):
|
|
||||||
"""A browser that has loaded the app: session cookie in the jar, token in a header."""
|
|
||||||
with httpx.Client(base_url=server.base, timeout=TIMEOUT) as client:
|
|
||||||
client.headers[CSRF_HEADER] = client.get("/api/v1/session").json()["csrf_token"]
|
|
||||||
yield client
|
|
||||||
|
|
||||||
|
|
||||||
def call(client, method, path, body, asset):
|
|
||||||
return client.request(method, path.format(asset=asset), json=body)
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_api_refuses_every_caller_without_a_session(stack):
|
|
||||||
server, seeded, _ = stack
|
|
||||||
with anonymous(server) as client:
|
|
||||||
for method, path, body in PROTECTED:
|
|
||||||
response = call(client, method, path, body, seeded.asset_ids["a"])
|
|
||||||
assert response.status_code == 401, path
|
|
||||||
assert response.json()["error"]["code"] == "unauthenticated"
|
|
||||||
# Liveness and readiness stay open: an orchestrator holds no session.
|
|
||||||
assert client.get("/api/v1/health/ready").status_code == 200
|
|
||||||
assert client.get("/api/v1/health/live").status_code == 200
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_guessed_session_cookie_is_refused(stack):
|
|
||||||
server, _, _ = stack
|
|
||||||
with anonymous(server) as client:
|
|
||||||
client.cookies.set(SESSION_COOKIE, "guessed", domain="127.0.0.1")
|
|
||||||
response = client.get("/api/v1/workflow")
|
|
||||||
assert response.status_code == 401
|
|
||||||
assert response.json()["error"]["code"] == "unauthenticated"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_bootstrap_issues_a_strict_httponly_cookie(stack):
|
|
||||||
server, _, _ = stack
|
|
||||||
with anonymous(server) as client:
|
|
||||||
response = client.get("/api/v1/session")
|
|
||||||
cookie = response.headers["set-cookie"].lower()
|
|
||||||
assert "httponly" in cookie and "samesite=strict" in cookie and "path=/" in cookie
|
|
||||||
assert response.json()["csrf_token"]
|
|
||||||
# The token is in the body, which no other origin may read: no CORS header
|
|
||||||
# grants access to it.
|
|
||||||
assert "access-control-allow-origin" not in response.headers
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_session_without_the_csrf_token_may_read_but_not_mutate(stack):
|
|
||||||
server, _, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
del client.headers[CSRF_HEADER]
|
|
||||||
assert client.get("/api/v1/albums/proposals").status_code == 200
|
|
||||||
|
|
||||||
response = client.post("/api/v1/albums/proposals", json={})
|
|
||||||
assert response.status_code == 403
|
|
||||||
assert response.json()["error"]["code"] == "csrf_failed"
|
|
||||||
|
|
||||||
response = client.post(
|
|
||||||
"/api/v1/albums/proposals", json={}, headers={CSRF_HEADER: "guessed"}
|
|
||||||
)
|
|
||||||
assert response.status_code == 403
|
|
||||||
# And nothing was created behind the refusal.
|
|
||||||
with authenticated(server) as client:
|
|
||||||
assert client.get("/api/v1/albums/proposals").json()["items"] == []
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_foreign_origin_cannot_mutate_even_with_a_session(stack):
|
|
||||||
server, _, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
for origin in ("http://evil.example", "http://127.0.0.1:1", "null"):
|
|
||||||
response = client.post(
|
|
||||||
"/api/v1/albums/proposals", json={}, headers={"Origin": origin}
|
|
||||||
)
|
|
||||||
assert response.status_code == 403, origin
|
|
||||||
assert response.json()["error"]["code"] == "origin_not_allowed"
|
|
||||||
# This app's own origin is accepted, so the check is not simply refusing all.
|
|
||||||
allowed = client.post(
|
|
||||||
"/api/v1/albums/proposals", json={}, headers={"Origin": server.base}
|
|
||||||
)
|
|
||||||
assert allowed.status_code == 200
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_rebinding_host_is_refused(stack):
|
|
||||||
"""A name that resolves to 127.0.0.1 makes the browser treat the attacker's page
|
|
||||||
as same-origin. The Host header still carries that name, so it is checked."""
|
|
||||||
server, _, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
response = client.get("/api/v1/workflow", headers={"Host": "photos.evil.example"})
|
|
||||||
assert response.status_code == 403
|
|
||||||
assert response.json()["error"]["code"] == "host_not_allowed"
|
|
||||||
|
|
||||||
|
|
||||||
def test_media_cannot_be_embedded_by_another_page(stack):
|
|
||||||
server, seeded, _ = stack
|
|
||||||
url = f"/api/v1/assets/{seeded.asset_ids['a']}/thumbnail?size=256"
|
|
||||||
with authenticated(server) as client:
|
|
||||||
# What an <img> on another site produces: no Origin, but a cross-site marker.
|
|
||||||
blocked = client.get(url, headers={"Sec-Fetch-Site": "cross-site"})
|
|
||||||
assert blocked.status_code == 403
|
|
||||||
assert blocked.json()["error"]["code"] == "cross_site_blocked"
|
|
||||||
|
|
||||||
served = client.get(url, headers={"Sec-Fetch-Site": "same-origin"})
|
|
||||||
assert served.status_code == 200
|
|
||||||
assert served.headers["content-type"] == "image/webp"
|
|
||||||
assert served.headers["cross-origin-resource-policy"] == "same-origin"
|
|
||||||
# Photos must never land in a shared cache.
|
|
||||||
assert served.headers["cache-control"].startswith("private")
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_response_carries_the_default_headers_and_no_cors(stack):
|
|
||||||
server, seeded, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
responses = [
|
|
||||||
client.get("/api/v1/workflow"),
|
|
||||||
client.get(f"/api/v1/assets/{seeded.asset_ids['a']}/thumbnail?size=256"),
|
|
||||||
client.get("/app/"),
|
|
||||||
client.get("/api/v1/does-not-exist"),
|
|
||||||
]
|
|
||||||
for response in responses:
|
|
||||||
assert response.headers["x-content-type-options"] == "nosniff"
|
|
||||||
assert response.headers["x-frame-options"] == "DENY"
|
|
||||||
assert response.headers["referrer-policy"] == "no-referrer"
|
|
||||||
assert "frame-ancestors 'none'" in response.headers["content-security-policy"]
|
|
||||||
assert "access-control-allow-origin" not in response.headers
|
|
||||||
assert "access-control-allow-credentials" not in response.headers
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_traversal_attempt_addresses_nothing(stack):
|
|
||||||
"""Identifiers are database keys, not paths: traversal has nowhere to land."""
|
|
||||||
server, _, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
for path in (
|
|
||||||
"/api/v1/albums/proposals/..%2F..%2F..%2Fetc%2Fpasswd",
|
|
||||||
"/api/v1/albums/proposals/../../../etc/passwd",
|
|
||||||
"/api/v1/assets/..%2F..%2Fetc%2Fpasswd/thumbnail?size=256",
|
|
||||||
"/api/v1/assets/%2Fetc%2Fpasswd/thumbnail?size=256",
|
|
||||||
):
|
|
||||||
response = client.get(path)
|
|
||||||
assert response.status_code in (404, 422), path
|
|
||||||
assert "root:" not in response.text, path
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_oversized_request_is_refused_before_it_is_parsed(stack):
|
|
||||||
server, _, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
response = client.post(
|
|
||||||
"/api/v1/albums/proposals",
|
|
||||||
content=json.dumps({"albums": ["x" * 2_000_000]}),
|
|
||||||
headers={"Content-Type": "application/json"},
|
|
||||||
)
|
|
||||||
assert response.status_code == 413
|
|
||||||
assert response.json()["error"]["code"] == "payload_too_large"
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_malformed_request_reports_the_field_and_nothing_else(stack):
|
|
||||||
server, _, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
for content in ("{", '{"albums": 5}', ""):
|
|
||||||
response = client.post(
|
|
||||||
"/api/v1/albums/proposals",
|
|
||||||
content=content,
|
|
||||||
headers={"Content-Type": "application/json"},
|
|
||||||
)
|
|
||||||
assert response.status_code == 422, content
|
|
||||||
error = response.json()["error"]
|
|
||||||
assert error["code"] == "invalid_request"
|
|
||||||
assert "Traceback" not in response.text and "photo_pipeline/" not in response.text
|
|
||||||
|
|
||||||
|
|
||||||
def test_errors_reveal_neither_the_credential_nor_an_internal(stack):
|
|
||||||
"""The API key is configured but must appear nowhere; failures additionally say
|
|
||||||
nothing about where the library lives or how the server is built.
|
|
||||||
|
|
||||||
Successful responses are a different matter: path previews are the point of the
|
|
||||||
rename and archive views, and the operator is the one who owns those paths.
|
|
||||||
"""
|
|
||||||
server, seeded, _ = stack
|
|
||||||
with authenticated(server) as client:
|
|
||||||
responses = [
|
|
||||||
client.post("/api/v1/upload-preflight", json={}),
|
|
||||||
client.get("/api/v1/workflow"),
|
|
||||||
client.get("/api/v1/upload-batches"),
|
|
||||||
client.get("/api/v1/assets/unknown-asset/thumbnail?size=256"),
|
|
||||||
client.get("/api/v1/archive-plans/unknown-plan"),
|
|
||||||
client.post("/api/v1/archive-locations", json={"name": "x", "root": "/nope"}),
|
|
||||||
client.get("/api/v1/does-not-exist"),
|
|
||||||
]
|
|
||||||
for response in responses:
|
|
||||||
assert SENTINEL_KEY not in response.text
|
|
||||||
assert "Traceback" not in response.text
|
|
||||||
if response.status_code >= 400:
|
|
||||||
assert str(seeded.lib) not in response.text
|
|
||||||
assert "photo_pipeline/" not in response.text
|
|
||||||
assert "sqlite" not in response.text.lower()
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_symlink_swapped_under_an_asset_cannot_be_served(stack):
|
|
||||||
"""TOCTOU on the media path: the file the database points at is replaced by a
|
|
||||||
link to something outside the library between the scan and the request."""
|
|
||||||
server, seeded, outside = stack
|
|
||||||
secret = outside / "secret.jpg"
|
|
||||||
original = seeded.lib / "b.jpg"
|
|
||||||
original.unlink()
|
|
||||||
original.symlink_to(secret)
|
|
||||||
|
|
||||||
with authenticated(server) as client:
|
|
||||||
response = client.get(f"/api/v1/assets/{seeded.asset_ids['b']}/thumbnail?size=256")
|
|
||||||
assert response.status_code == 403
|
|
||||||
assert response.json()["error"]["code"] == "path_not_allowed"
|
|
||||||
# The refusal names no filesystem location, and no bytes escaped with it.
|
|
||||||
assert str(outside) not in response.text and str(seeded.lib) not in response.text
|
|
||||||
assert secret.read_bytes()[:16] not in response.content
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_frontend_shell_stays_reachable_without_a_session(stack):
|
|
||||||
"""It must load before any JavaScript can ask for a session."""
|
|
||||||
server, _, _ = stack
|
|
||||||
with anonymous(server) as client:
|
|
||||||
response = client.get("/app/", headers={"Sec-Fetch-Site": "none"})
|
|
||||||
assert response.status_code == 200
|
|
||||||
assert response.headers["content-type"].startswith("text/html")
|
|
||||||
assert Path("frontend/index.html").exists()
|
|
||||||
@@ -14,7 +14,6 @@ MAP = json.loads((REPO / "tests" / "story_traceability.json").read_text())["stor
|
|||||||
PHASE_A_STORIES = {f"US01-0{n}" for n in range(1, 8)}
|
PHASE_A_STORIES = {f"US01-0{n}" for n in range(1, 8)}
|
||||||
PHASE_D_STORIES = {f"US04-0{n}" for n in range(1, 7)}
|
PHASE_D_STORIES = {f"US04-0{n}" for n in range(1, 7)}
|
||||||
PHASE_E_STORIES = {f"US05-0{n}" for n in range(1, 7)}
|
PHASE_E_STORIES = {f"US05-0{n}" for n in range(1, 7)}
|
||||||
PHASE_F_STORIES = {f"US06-0{n}" for n in range(1, 7)}
|
|
||||||
|
|
||||||
|
|
||||||
def test_all_phase_a_stories_are_mapped():
|
def test_all_phase_a_stories_are_mapped():
|
||||||
@@ -33,13 +32,6 @@ def test_all_phase_e_stories_are_mapped():
|
|||||||
assert PHASE_E_STORIES <= set(MAP)
|
assert PHASE_E_STORIES <= set(MAP)
|
||||||
|
|
||||||
|
|
||||||
def test_all_phase_f_stories_are_mapped():
|
|
||||||
"""US06-06 acceptance: every archive-lifecycle story, destination through
|
|
||||||
restore, is tied to automated tests — archiving is the only stage that removes
|
|
||||||
an original."""
|
|
||||||
assert PHASE_F_STORIES <= set(MAP)
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_mapped_test_file_exists_and_is_nonempty():
|
def test_every_mapped_test_file_exists_and_is_nonempty():
|
||||||
for story, files in MAP.items():
|
for story, files in MAP.items():
|
||||||
assert files, f"{story} maps to no tests"
|
assert files, f"{story} maps to no tests"
|
||||||
|
|||||||
@@ -167,8 +167,7 @@ def test_missing_file_is_flagged_not_deleted(
|
|||||||
assets = assets_by_id(make_factory(db_url))
|
assets = assets_by_id(make_factory(db_url))
|
||||||
assert missing_id in assets # not pruned
|
assert missing_id in assets # not pruned
|
||||||
assert assets[missing_id].missing_at is not None
|
assert assets[missing_id].missing_at is not None
|
||||||
# Nothing explains the absence: this is not an offline archive medium (US06-03).
|
assert assets[missing_id].availability_state == "active"
|
||||||
assert assets[missing_id].availability_state == "missing_unexpected"
|
|
||||||
|
|
||||||
|
|
||||||
def test_reappearing_file_clears_missing(
|
def test_reappearing_file_clears_missing(
|
||||||
@@ -187,7 +186,6 @@ def test_reappearing_file_clears_missing(
|
|||||||
inventory.scan(lib)
|
inventory.scan(lib)
|
||||||
assets = assets_by_id(make_factory(db_url))
|
assets = assets_by_id(make_factory(db_url))
|
||||||
assert assets[asset_id].missing_at is None
|
assert assets[asset_id].missing_at is None
|
||||||
assert assets[asset_id].availability_state == "active"
|
|
||||||
|
|
||||||
|
|
||||||
def test_identity_and_state_durable_across_restart(
|
def test_identity_and_state_durable_across_restart(
|
||||||
|
|||||||
@@ -1,238 +0,0 @@
|
|||||||
"""Importing the archived CLI's CSV state into the database (US07-01).
|
|
||||||
|
|
||||||
The donor's ``nsfw_scores.csv`` was keyed by path; the database is keyed by a
|
|
||||||
stable asset id. Every case here is about that gap: a path that moved, a path that
|
|
||||||
matches nothing, a score a human has already overruled, and a file that is simply
|
|
||||||
malformed. The import may add evidence and must never invent an asset, overwrite a
|
|
||||||
judgement, or fail silently — whatever it does ends up in the report.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import uuid
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
|
||||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview
|
|
||||||
from photo_pipeline.services.legacy_import import (
|
|
||||||
REVIEWER,
|
|
||||||
LegacyImportService,
|
|
||||||
write_report,
|
|
||||||
)
|
|
||||||
|
|
||||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def factory(tmp_path):
|
|
||||||
url = f"sqlite:///{tmp_path / 'legacy.db'}"
|
|
||||||
run_migrations(url)
|
|
||||||
engine = create_db_engine(url)
|
|
||||||
yield create_session_factory(engine)
|
|
||||||
engine.dispose()
|
|
||||||
|
|
||||||
|
|
||||||
def _asset(sf, path: Path, *, previous: Path | None = None) -> str:
|
|
||||||
asset_id = str(uuid.uuid4())
|
|
||||||
with sf() as session:
|
|
||||||
session.add(
|
|
||||||
Asset(
|
|
||||||
id=asset_id,
|
|
||||||
original_path=str(previous or path),
|
|
||||||
current_path=str(path),
|
|
||||||
discovered_at=NOW,
|
|
||||||
hash_version=1,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.add(AssetPath(asset_id=asset_id, path=str(path), valid_from=NOW))
|
|
||||||
if previous is not None:
|
|
||||||
session.add(
|
|
||||||
AssetPath(
|
|
||||||
asset_id=asset_id, path=str(previous), valid_from=NOW, valid_until=NOW
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return asset_id
|
|
||||||
|
|
||||||
|
|
||||||
def _review(sf, asset_id: str, *, score=None, decision=None, at=NOW) -> None:
|
|
||||||
with sf() as session:
|
|
||||||
session.add(
|
|
||||||
SafetyReview(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
asset_id=asset_id,
|
|
||||||
score=score,
|
|
||||||
decision=decision,
|
|
||||||
reviewer="dom",
|
|
||||||
created_at=at,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def _csv(tmp_path, rows: str) -> Path:
|
|
||||||
path = tmp_path / "nsfw_scores.csv"
|
|
||||||
path.write_text("path,nsfw_score\n" + rows, encoding="utf-8")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def _scores(sf) -> dict[str, float]:
|
|
||||||
with sf() as session:
|
|
||||||
return {
|
|
||||||
review.asset_id: review.score
|
|
||||||
for review in session.scalars(
|
|
||||||
select(SafetyReview).order_by(SafetyReview.created_at)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def test_scores_are_imported_onto_asset_identity(tmp_path, factory):
|
|
||||||
a = _asset(factory, tmp_path / "a.jpg")
|
|
||||||
# This one was scored under its old path and has since moved.
|
|
||||||
b = _asset(factory, tmp_path / "moved" / "b.jpg", previous=tmp_path / "b.jpg")
|
|
||||||
csv_path = _csv(
|
|
||||||
tmp_path, f"{tmp_path / 'a.jpg'},0.9123\n{tmp_path / 'b.jpg'},0.0100\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
|
||||||
|
|
||||||
assert report.counts == {
|
|
||||||
"rows": 2,
|
|
||||||
"imported": 2,
|
|
||||||
"skipped_existing": 0,
|
|
||||||
"unchanged": 0,
|
|
||||||
"reviewed": 0,
|
|
||||||
"unmatched": 0,
|
|
||||||
"rejected": 0,
|
|
||||||
"conflicts": 0,
|
|
||||||
}
|
|
||||||
assert _scores(factory) == {a: 0.9123, b: 0.0100}
|
|
||||||
# The imported rows are evidence, not judgements: no decision is invented.
|
|
||||||
with factory() as session:
|
|
||||||
rows = list(session.scalars(select(SafetyReview)))
|
|
||||||
assert {r.decision for r in rows} == {None}
|
|
||||||
assert {r.reviewer for r in rows} == {REVIEWER}
|
|
||||||
assert csv_path.exists(), "the CSV is left on disk as its own backup"
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_reviewed_asset_is_never_overwritten_by_the_csv(tmp_path, factory):
|
|
||||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
|
||||||
_review(factory, asset_id, score=0.2, decision="sfw")
|
|
||||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.9999\n")
|
|
||||||
|
|
||||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path, overwrite=True)
|
|
||||||
|
|
||||||
assert (report.imported, report.reviewed) == (0, 1)
|
|
||||||
assert report.conflicts[0]["decision"] == "sfw"
|
|
||||||
assert report.conflicts[0]["csv"] == 0.9999
|
|
||||||
with factory() as session:
|
|
||||||
rows = list(session.scalars(select(SafetyReview)))
|
|
||||||
assert len(rows) == 1 and rows[0].decision == "sfw"
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_unknown_path_is_reported_never_turned_into_an_asset(tmp_path, factory):
|
|
||||||
_asset(factory, tmp_path / "a.jpg")
|
|
||||||
csv_path = _csv(
|
|
||||||
tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n{tmp_path / 'ghost.jpg'},0.5000\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
|
||||||
|
|
||||||
assert report.unmatched == [str(tmp_path / "ghost.jpg")]
|
|
||||||
with factory() as session:
|
|
||||||
assert session.scalar(select(Asset).where(Asset.current_path.like("%ghost%"))) is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_malformed_rows_are_rejected_with_their_reason(tmp_path, factory):
|
|
||||||
_asset(factory, tmp_path / "a.jpg")
|
|
||||||
csv_path = _csv(
|
|
||||||
tmp_path,
|
|
||||||
f"{tmp_path / 'a.jpg'},0.5000\n"
|
|
||||||
f"{tmp_path / 'a.jpg'},0.5000\n" # the same path twice
|
|
||||||
f"{tmp_path / 'b.jpg'},not-a-number\n"
|
|
||||||
f"{tmp_path / 'c.jpg'},7.5\n" # outside 0..1: corruption, not a score
|
|
||||||
",0.5\n", # no path at all
|
|
||||||
)
|
|
||||||
|
|
||||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
|
||||||
|
|
||||||
reasons = sorted(r["reason"] for r in report.rejected)
|
|
||||||
assert reasons == ["duplicate_path", "missing_path", "unparsable_score", "unparsable_score"]
|
|
||||||
assert report.imported == 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_rerunning_changes_nothing_and_a_differing_score_needs_overwrite(tmp_path, factory):
|
|
||||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
|
||||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
|
||||||
service = LegacyImportService(factory)
|
|
||||||
service.import_nsfw_scores(csv_path)
|
|
||||||
|
|
||||||
again = service.import_nsfw_scores(csv_path)
|
|
||||||
assert (again.imported, again.unchanged) == (0, 1)
|
|
||||||
assert len(_scores(factory)) == 1
|
|
||||||
|
|
||||||
changed = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.8000\n")
|
|
||||||
refused = service.import_nsfw_scores(changed)
|
|
||||||
assert (refused.imported, refused.skipped_existing) == (0, 1)
|
|
||||||
assert refused.conflicts[0]["current"] == 0.5
|
|
||||||
|
|
||||||
forced = service.import_nsfw_scores(changed, overwrite=True)
|
|
||||||
assert forced.imported == 1
|
|
||||||
with factory() as session:
|
|
||||||
latest = list(session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)))[-1]
|
|
||||||
assert (latest.asset_id, latest.score) == (asset_id, 0.8)
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_dry_run_reports_without_writing(tmp_path, factory):
|
|
||||||
_asset(factory, tmp_path / "a.jpg")
|
|
||||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
|
||||||
|
|
||||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path, dry_run=True)
|
|
||||||
|
|
||||||
assert report.imported == 1
|
|
||||||
assert _scores(factory) == {}, "a dry run must leave the database alone"
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_missing_csv_is_a_reported_outcome_not_a_crash(tmp_path, factory):
|
|
||||||
report = LegacyImportService(factory).import_nsfw_scores(tmp_path / "nothing.csv")
|
|
||||||
assert report.rejected == [{"path": str(tmp_path / "nothing.csv"), "reason": "csv_missing"}]
|
|
||||||
assert report.rows == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_report_is_written_where_it_can_be_audited(tmp_path, factory):
|
|
||||||
_asset(factory, tmp_path / "a.jpg")
|
|
||||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
|
||||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
|
||||||
|
|
||||||
written = write_report(report, tmp_path / "data")
|
|
||||||
|
|
||||||
payload = json.loads(written.read_text(encoding="utf-8"))
|
|
||||||
assert payload["imported"] == 1
|
|
||||||
assert payload["source"] == str(csv_path)
|
|
||||||
assert payload["schema_version"] == 1
|
|
||||||
assert payload["generated_at"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_later_review_still_wins_after_an_import(tmp_path, factory):
|
|
||||||
"""The import is a floor, not a ceiling: a human decision made afterwards is
|
|
||||||
the current state, exactly as everywhere else in the app."""
|
|
||||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
|
||||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.9000\n")
|
|
||||||
LegacyImportService(factory).import_nsfw_scores(csv_path)
|
|
||||||
# The import stamps itself with the wall clock, so "afterwards" has to be too.
|
|
||||||
_review(
|
|
||||||
factory,
|
|
||||||
asset_id,
|
|
||||||
score=0.9,
|
|
||||||
decision="sfw",
|
|
||||||
at=datetime.now(timezone.utc) + timedelta(hours=1),
|
|
||||||
)
|
|
||||||
|
|
||||||
with factory() as session:
|
|
||||||
latest = list(session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)))[-1]
|
|
||||||
assert latest.decision == "sfw"
|
|
||||||
@@ -1,377 +0,0 @@
|
|||||||
"""Offline identity and review evidence (US06-03).
|
|
||||||
|
|
||||||
An archived photo is not gone: it keeps its identity, its hashes, and enough
|
|
||||||
evidence to be recognised in a duplicate cluster while its medium sits in a
|
|
||||||
drawer. Every case here archives a *real* album through the real transfer, then
|
|
||||||
takes the medium away by removing its marker — the same thing the service sees
|
|
||||||
when an external disk is unplugged — and asks whether the application still tells
|
|
||||||
the truth about where the bytes are.
|
|
||||||
|
|
||||||
The distinction that matters throughout: an unmounted medium is
|
|
||||||
``archived_offline`` (expected, harmless), a mounted medium with a hole in it is
|
|
||||||
``missing_unexpected`` (needs a human). Confusing the two is how an archive
|
|
||||||
quietly loses a photo.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
|
||||||
import uuid
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from PIL import Image
|
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
from photo_pipeline.api.app import create_app
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
|
||||||
from photo_pipeline.models import Asset, Thumbnail, UploadBatch, UploadItem
|
|
||||||
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
|
||||||
from photo_pipeline.services.archives import MARKER_NAME, ArchiveService
|
|
||||||
from photo_pipeline.services.availability import (
|
|
||||||
ACTIVE,
|
|
||||||
ARCHIVED_OFFLINE,
|
|
||||||
ARCHIVED_ONLINE,
|
|
||||||
MISSING_UNEXPECTED,
|
|
||||||
)
|
|
||||||
from photo_pipeline.services.duplicates import ClusterState, DuplicateService, Method
|
|
||||||
from photo_pipeline.services.hashing import sha256_file
|
|
||||||
from photo_pipeline.services.inventory import InventoryService
|
|
||||||
from photo_pipeline.services.thumbnails import PROTECTED_SIZE, ThumbnailService, ThumbnailUnavailable
|
|
||||||
|
|
||||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
|
||||||
|
|
||||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
# ── environment ──────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def _env(tmp_path):
|
|
||||||
(tmp_path / "data").mkdir(exist_ok=True)
|
|
||||||
lib = tmp_path / "lib"
|
|
||||||
lib.mkdir(exist_ok=True)
|
|
||||||
archive = tmp_path / "archive"
|
|
||||||
archive.mkdir(exist_ok=True)
|
|
||||||
config = Config.from_env(
|
|
||||||
{
|
|
||||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
|
||||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
|
||||||
"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
run_migrations(config.database_url)
|
|
||||||
return config, create_session_factory(create_db_engine(config.database_url)), lib, archive
|
|
||||||
|
|
||||||
|
|
||||||
def structured(path, seed, size=(256, 192)):
|
|
||||||
"""A deterministic, decodable photo — previews and pHashes must be real."""
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
rng = np.random.default_rng(seed)
|
|
||||||
w, h = size
|
|
||||||
base = np.zeros((h, w, 3), dtype=np.uint8)
|
|
||||||
for _ in range(6):
|
|
||||||
x0 = int(rng.integers(0, w - 60))
|
|
||||||
y0 = int(rng.integers(0, h - 60))
|
|
||||||
base[y0 : y0 + 60, x0 : x0 + 60] = rng.integers(0, 256, 3)
|
|
||||||
grad = np.linspace(0, 120, w, dtype=np.uint8)
|
|
||||||
base[:, :, 0] = np.clip(base[:, :, 0].astype(int) + grad[None, :], 0, 255)
|
|
||||||
Image.fromarray(base).save(path, quality=95)
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def resized_copy(src, dst, scale=0.5):
|
|
||||||
with Image.open(src) as image:
|
|
||||||
image.resize(
|
|
||||||
(int(image.width * scale), int(image.height * scale)), Image.LANCZOS
|
|
||||||
).save(dst, quality=95)
|
|
||||||
return dst
|
|
||||||
|
|
||||||
|
|
||||||
def _uploaded(sf, lib, album="rome", seeds=(1, 2)):
|
|
||||||
"""A scanned album carrying the verified upload evidence archiving requires."""
|
|
||||||
folder = lib / album
|
|
||||||
for index, seed in enumerate(seeds):
|
|
||||||
structured(folder / f"{index}.jpg", seed)
|
|
||||||
result = InventoryService(sf).scan(lib)
|
|
||||||
with sf() as session:
|
|
||||||
batch_id = str(uuid.uuid4())
|
|
||||||
session.add(
|
|
||||||
UploadBatch(
|
|
||||||
id=batch_id,
|
|
||||||
album=album,
|
|
||||||
folder=str(folder),
|
|
||||||
album_name=album,
|
|
||||||
state="succeeded",
|
|
||||||
preflight_token="v1:test",
|
|
||||||
outcome_state="verified",
|
|
||||||
created_at=NOW,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
for path, asset_id in result.asset_ids.items():
|
|
||||||
session.add(
|
|
||||||
UploadItem(
|
|
||||||
batch_id=batch_id,
|
|
||||||
asset_id=asset_id,
|
|
||||||
path=path,
|
|
||||||
sha256=sha256_file(path),
|
|
||||||
sha1="0" * 40,
|
|
||||||
state="sent",
|
|
||||||
outcome="uploaded",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return folder, result.asset_ids
|
|
||||||
|
|
||||||
|
|
||||||
def _archive(sf, config, archive, albums=None):
|
|
||||||
service = ArchiveService(sf, config=config)
|
|
||||||
location = service.register("external", str(archive))
|
|
||||||
token = service.preflight(location["id"], albums)["token"]
|
|
||||||
transfers = ArchiveTransferService(sf, config=config)
|
|
||||||
plan = transfers.create(location["id"], albums, token=token)
|
|
||||||
transfers.apply(plan["id"])
|
|
||||||
return location
|
|
||||||
|
|
||||||
|
|
||||||
def _unmount(archive):
|
|
||||||
"""Take the medium away the way a real one goes: its marker stops answering."""
|
|
||||||
(archive / MARKER_NAME).rename(archive / f"{MARKER_NAME}.away")
|
|
||||||
|
|
||||||
|
|
||||||
def _remount(archive):
|
|
||||||
(archive / f"{MARKER_NAME}.away").rename(archive / MARKER_NAME)
|
|
||||||
|
|
||||||
|
|
||||||
def _assets(sf):
|
|
||||||
with sf() as session:
|
|
||||||
return {asset.id: asset for asset in session.scalars(select(Asset))}
|
|
||||||
|
|
||||||
|
|
||||||
# ── availability ─────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_archived_assets_report_online_offline_and_missing(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
_, ids = _uploaded(sf, lib)
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
inventory = InventoryService(sf)
|
|
||||||
|
|
||||||
states = {a.availability_state for a in _assets(sf).values()}
|
|
||||||
assert states == {ARCHIVED_ONLINE}
|
|
||||||
|
|
||||||
_unmount(archive)
|
|
||||||
inventory.scan(lib) # the album folder is gone from the active roots
|
|
||||||
assert {a.availability_state for a in _assets(sf).values()} == {ARCHIVED_OFFLINE}
|
|
||||||
assert all(a.missing_at is None for a in _assets(sf).values()) # not "missing"
|
|
||||||
|
|
||||||
_remount(archive)
|
|
||||||
inventory.scan(lib)
|
|
||||||
assert {a.availability_state for a in _assets(sf).values()} == {ARCHIVED_ONLINE}
|
|
||||||
|
|
||||||
# Mounted medium, absent file: that is not an offline archive, it needs a human.
|
|
||||||
victim = sorted(ids.values())[0]
|
|
||||||
with sf() as session:
|
|
||||||
asset = session.get(Asset, victim)
|
|
||||||
(archive / asset.archive_path).unlink()
|
|
||||||
inventory.scan(lib)
|
|
||||||
assert _assets(sf)[victim].availability_state == MISSING_UNEXPECTED
|
|
||||||
|
|
||||||
|
|
||||||
def test_scan_never_prunes_or_flags_offline_assets(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
_, ids = _uploaded(sf, lib)
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
_unmount(archive)
|
|
||||||
|
|
||||||
before = _assets(sf)
|
|
||||||
result = InventoryService(sf).scan(lib)
|
|
||||||
|
|
||||||
assert result.counts.get("missing") is None
|
|
||||||
after = _assets(sf)
|
|
||||||
assert set(after) == set(before) == set(ids.values())
|
|
||||||
for asset in after.values():
|
|
||||||
assert asset.availability_state == ARCHIVED_OFFLINE
|
|
||||||
assert asset.current_sha256 and asset.pixel_sha256 # hashes retained
|
|
||||||
assert asset.archive_location_id and asset.archive_path
|
|
||||||
|
|
||||||
|
|
||||||
def test_active_missing_file_is_missing_unexpected_not_offline(tmp_path):
|
|
||||||
config, sf, lib, _archive_root = _env(tmp_path)
|
|
||||||
structured(lib / "loose" / "a.jpg", 7)
|
|
||||||
ids = InventoryService(sf).scan(lib).asset_ids
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
|
|
||||||
(lib / "loose" / "a.jpg").unlink()
|
|
||||||
InventoryService(sf).scan(lib)
|
|
||||||
|
|
||||||
asset = _assets(sf)[asset_id]
|
|
||||||
assert asset.availability_state == MISSING_UNEXPECTED
|
|
||||||
assert asset.missing_at is not None
|
|
||||||
|
|
||||||
|
|
||||||
# ── deduplication against archived originals ─────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_exact_copy_of_offline_asset_links_to_archived_canonical(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
folder, ids = _uploaded(sf, lib, seeds=(1,))
|
|
||||||
archived_id = next(iter(ids.values()))
|
|
||||||
original = next(iter(ids))
|
|
||||||
kept = tmp_path / "kept.jpg"
|
|
||||||
shutil.copy2(original, kept)
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
_unmount(archive)
|
|
||||||
|
|
||||||
# The same photo turns up again in the active library while the disk is away.
|
|
||||||
(lib / "inbox").mkdir(parents=True, exist_ok=True)
|
|
||||||
shutil.copy2(kept, lib / "inbox" / "again.jpg")
|
|
||||||
scan = InventoryService(sf).scan(lib)
|
|
||||||
new_id = scan.asset_ids[str(lib / "inbox" / "again.jpg")]
|
|
||||||
assert scan.occurrences[str(lib / "inbox" / "again.jpg")] == "copied"
|
|
||||||
|
|
||||||
clusters = DuplicateService(sf).detect().clusters
|
|
||||||
exact = [c for c in clusters if c["method"] == Method.EXACT.value]
|
|
||||||
assert len(exact) == 1
|
|
||||||
cluster = exact[0]
|
|
||||||
# Byte-identical: linked directly, and the archived original stays canonical.
|
|
||||||
assert cluster["state"] == ClusterState.DECIDED.value
|
|
||||||
assert cluster["canonical_asset_id"] == archived_id
|
|
||||||
assert _assets(sf)[new_id].canonical_asset_id == archived_id
|
|
||||||
|
|
||||||
|
|
||||||
def test_fuzzy_copy_of_offline_asset_requires_review_and_names_the_medium(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
folder, ids = _uploaded(sf, lib, seeds=(1,))
|
|
||||||
archived_id = next(iter(ids.values()))
|
|
||||||
original = next(iter(ids))
|
|
||||||
variant_source = resized_copy(original, tmp_path / "small.jpg")
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
_unmount(archive)
|
|
||||||
|
|
||||||
(lib / "inbox").mkdir(parents=True, exist_ok=True)
|
|
||||||
shutil.copy2(variant_source, lib / "inbox" / "small.jpg")
|
|
||||||
InventoryService(sf).scan(lib)
|
|
||||||
|
|
||||||
duplicates = DuplicateService(sf)
|
|
||||||
clusters = duplicates.detect().clusters
|
|
||||||
perceptual = [c for c in clusters if c["method"] == Method.PERCEPTUAL.value]
|
|
||||||
assert len(perceptual) == 1
|
|
||||||
|
|
||||||
detail = duplicates.get_cluster(perceptual[0]["id"])
|
|
||||||
assert detail["state"] == ClusterState.OPEN.value # never auto-decided
|
|
||||||
assert detail["requires_confirmation"] is True
|
|
||||||
assert detail["mount_required"] == ["external"] # full-resolution needs the disk
|
|
||||||
archived = next(m for m in detail["members"] if m["asset_id"] == archived_id)
|
|
||||||
assert archived["availability_state"] == ARCHIVED_OFFLINE
|
|
||||||
assert archived["current_path"] is None
|
|
||||||
assert archived["archive_path"] and archived["evidence"]["phash"]
|
|
||||||
# The retained preview is what makes the offline member reviewable at all.
|
|
||||||
assert archived["preview"] == {"state": "ready", "protected": True, "size": PROTECTED_SIZE}
|
|
||||||
|
|
||||||
|
|
||||||
# ── protected review evidence ────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_protected_preview_survives_quota_and_serves_while_offline(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
_, ids = _uploaded(sf, lib, seeds=(1,))
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
_unmount(archive)
|
|
||||||
|
|
||||||
thumbnails = ThumbnailService(sf, config)
|
|
||||||
served = thumbnails.generate(asset_id, PROTECTED_SIZE)
|
|
||||||
assert served.exists() # rendered before the original left, not from the medium
|
|
||||||
|
|
||||||
# An aggressive quota may empty the cache, but not this evidence.
|
|
||||||
tight = ThumbnailService(sf, config.model_copy(update={"thumbnail_cache_quota_bytes": 1}))
|
|
||||||
tight._enforce_quota()
|
|
||||||
assert served.exists()
|
|
||||||
with sf() as session:
|
|
||||||
row = session.scalar(select(Thumbnail).where(Thumbnail.asset_id == asset_id))
|
|
||||||
assert row.protected is True
|
|
||||||
assert thumbnails.evidence(asset_id)["state"] == "ready"
|
|
||||||
|
|
||||||
|
|
||||||
def test_offline_asset_without_preview_reports_unavailable(tmp_path):
|
|
||||||
"""No preview and no medium is an honest 409, never a wrong picture."""
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
_, ids = _uploaded(sf, lib, seeds=(1,))
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
with sf() as session:
|
|
||||||
for row in session.scalars(select(Thumbnail).where(Thumbnail.asset_id == asset_id)):
|
|
||||||
session.delete(row)
|
|
||||||
session.commit()
|
|
||||||
_unmount(archive)
|
|
||||||
|
|
||||||
with pytest.raises(ThumbnailUnavailable):
|
|
||||||
ThumbnailService(sf, config).generate(asset_id, 256)
|
|
||||||
|
|
||||||
_remount(archive) # mounted again: the archived original is readable
|
|
||||||
assert ThumbnailService(sf, config).generate(asset_id, 256).exists()
|
|
||||||
|
|
||||||
|
|
||||||
def test_offline_asset_is_browsable_through_the_api(tmp_path):
|
|
||||||
"""The browser sees an archived asset, its medium, and its preview — offline."""
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
_, ids = _uploaded(sf, lib, seeds=(1,))
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
_unmount(archive)
|
|
||||||
|
|
||||||
with TestClient(create_app(config)) as client:
|
|
||||||
client.post("/api/v1/inventory/scan")
|
|
||||||
listed = client.get("/api/v1/inventory/assets", params={"availability": ARCHIVED_OFFLINE})
|
|
||||||
assert listed.status_code == 200
|
|
||||||
item = next(row for row in listed.json()["items"] if row["id"] == asset_id)
|
|
||||||
assert item["current_path"] is None
|
|
||||||
assert item["archive_path"] == "rome/0.jpg"
|
|
||||||
assert item["missing"] is False
|
|
||||||
|
|
||||||
# Searching by the archived path still finds it.
|
|
||||||
found = client.get("/api/v1/inventory/assets", params={"q": "rome"}).json()
|
|
||||||
assert [row["id"] for row in found["items"]] == [asset_id]
|
|
||||||
|
|
||||||
preview = client.get(f"/api/v1/assets/{asset_id}/thumbnail", params={"size": 1280})
|
|
||||||
assert preview.status_code == 200
|
|
||||||
assert preview.headers["content-type"] == "image/webp"
|
|
||||||
|
|
||||||
|
|
||||||
def test_offline_state_is_stable_across_restart(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
_, ids = _uploaded(sf, lib, seeds=(1, 2))
|
|
||||||
_archive(sf, config, archive)
|
|
||||||
_unmount(archive)
|
|
||||||
InventoryService(sf).scan(lib)
|
|
||||||
before = {
|
|
||||||
asset_id: (
|
|
||||||
asset.availability_state,
|
|
||||||
asset.archive_path,
|
|
||||||
asset.current_sha256,
|
|
||||||
asset.phash,
|
|
||||||
)
|
|
||||||
for asset_id, asset in _assets(sf).items()
|
|
||||||
}
|
|
||||||
|
|
||||||
# Restart: a fresh engine and session factory against the same database.
|
|
||||||
restarted = create_session_factory(create_db_engine(config.database_url))
|
|
||||||
after = {
|
|
||||||
asset_id: (
|
|
||||||
asset.availability_state,
|
|
||||||
asset.archive_path,
|
|
||||||
asset.current_sha256,
|
|
||||||
asset.phash,
|
|
||||||
)
|
|
||||||
for asset_id, asset in _assets(restarted).items()
|
|
||||||
}
|
|
||||||
assert after == before
|
|
||||||
assert set(after) == set(ids.values())
|
|
||||||
|
|
||||||
# And the medium coming back is picked up by the restarted process.
|
|
||||||
_remount(archive)
|
|
||||||
assert ArchiveService(restarted, config=config).locations()[0]["state"] == "online"
|
|
||||||
assert {a.availability_state for a in _assets(restarted).values()} == {ARCHIVED_ONLINE}
|
|
||||||
assert ACTIVE not in {a.availability_state for a in _assets(restarted).values()}
|
|
||||||
@@ -1,417 +0,0 @@
|
|||||||
"""Planning and executing safe restores (US06-04).
|
|
||||||
|
|
||||||
Restoring is the one archive operation that can *add* a file to the library, so
|
|
||||||
every case here asks two questions: did the right bytes come back under the right
|
|
||||||
identity, and did anything already in the library get touched? The media are real
|
|
||||||
directories, the hashes are real, and the failure paths assert that the archived
|
|
||||||
copy is still exactly where it was — a restore that fails must cost nothing.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
|
||||||
import uuid
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from PIL import Image
|
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
from photo_pipeline.api.app import create_app
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
|
||||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview, UploadBatch, UploadItem
|
|
||||||
from photo_pipeline.services import availability
|
|
||||||
from photo_pipeline.services.archive_journal import ArchiveState
|
|
||||||
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
|
||||||
from photo_pipeline.services.archives import MARKER_NAME, ArchiveError, ArchiveService
|
|
||||||
from photo_pipeline.services.hashing import sha256_file
|
|
||||||
from photo_pipeline.services.inventory import InventoryService
|
|
||||||
from photo_pipeline.services.restores import RestoreService
|
|
||||||
|
|
||||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
|
||||||
|
|
||||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
# ── environment ──────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def _env(tmp_path):
|
|
||||||
(tmp_path / "data").mkdir(exist_ok=True)
|
|
||||||
lib = tmp_path / "lib"
|
|
||||||
lib.mkdir(exist_ok=True)
|
|
||||||
archive = tmp_path / "archive"
|
|
||||||
archive.mkdir(exist_ok=True)
|
|
||||||
config = Config.from_env(
|
|
||||||
{
|
|
||||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
|
||||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
|
||||||
"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
run_migrations(config.database_url)
|
|
||||||
return config, create_session_factory(create_db_engine(config.database_url)), lib, archive
|
|
||||||
|
|
||||||
|
|
||||||
def structured(path, seed, size=(192, 144)):
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
rng = np.random.default_rng(seed)
|
|
||||||
w, h = size
|
|
||||||
base = np.zeros((h, w, 3), dtype=np.uint8)
|
|
||||||
for _ in range(5):
|
|
||||||
x0 = int(rng.integers(0, w - 40))
|
|
||||||
y0 = int(rng.integers(0, h - 40))
|
|
||||||
base[y0 : y0 + 40, x0 : x0 + 40] = rng.integers(0, 256, 3)
|
|
||||||
Image.fromarray(base).save(path, quality=95)
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def _archived(sf, config, lib, archive, album="rome", seeds=(1, 2)):
|
|
||||||
"""A real album taken all the way through archiving, ready to be restored."""
|
|
||||||
folder = lib / album
|
|
||||||
for index, seed in enumerate(seeds):
|
|
||||||
structured(folder / f"{index}.jpg", seed)
|
|
||||||
scan = InventoryService(sf).scan(lib)
|
|
||||||
with sf() as session:
|
|
||||||
batch_id = str(uuid.uuid4())
|
|
||||||
session.add(
|
|
||||||
UploadBatch(
|
|
||||||
id=batch_id,
|
|
||||||
album=album,
|
|
||||||
folder=str(folder),
|
|
||||||
album_name=album,
|
|
||||||
state="succeeded",
|
|
||||||
preflight_token="v1:test",
|
|
||||||
outcome_state="verified",
|
|
||||||
created_at=NOW,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
for path, asset_id in scan.asset_ids.items():
|
|
||||||
session.add(
|
|
||||||
UploadItem(
|
|
||||||
batch_id=batch_id,
|
|
||||||
asset_id=asset_id,
|
|
||||||
path=path,
|
|
||||||
sha256=sha256_file(path),
|
|
||||||
sha1="0" * 40,
|
|
||||||
state="sent",
|
|
||||||
outcome="uploaded",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
# A decision that must survive the whole round trip.
|
|
||||||
session.add(
|
|
||||||
SafetyReview(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
asset_id=asset_id,
|
|
||||||
decision="sfw",
|
|
||||||
score=0.01,
|
|
||||||
reviewer="test",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
service = ArchiveService(sf, config=config)
|
|
||||||
location = service.register("external", str(archive))
|
|
||||||
token = service.preflight(location["id"])["token"]
|
|
||||||
transfers = ArchiveTransferService(sf, config=config)
|
|
||||||
plan = transfers.create(location["id"], None, token=token)
|
|
||||||
transfers.apply(plan["id"])
|
|
||||||
return location, scan.asset_ids
|
|
||||||
|
|
||||||
|
|
||||||
def _restore(sf, config, location_id, asset_ids=None):
|
|
||||||
service = RestoreService(sf, config=config)
|
|
||||||
token = service.preflight(location_id, asset_ids)["token"]
|
|
||||||
plan = service.create(location_id, asset_ids, token=token)
|
|
||||||
return service, plan, service.apply(plan["id"])
|
|
||||||
|
|
||||||
|
|
||||||
def _unmount(archive):
|
|
||||||
(archive / MARKER_NAME).rename(archive / f"{MARKER_NAME}.away")
|
|
||||||
|
|
||||||
|
|
||||||
def _assets(sf):
|
|
||||||
with sf() as session:
|
|
||||||
return {asset.id: asset for asset in session.scalars(select(Asset))}
|
|
||||||
|
|
||||||
|
|
||||||
def _codes(report):
|
|
||||||
return {issue["code"] for issue in report["blockers"]} | {
|
|
||||||
issue["code"] for item in report["items"] for issue in item["blockers"]
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# ── preflight ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_preflight_blocks_offline_medium(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, _ = _archived(sf, config, lib, archive)
|
|
||||||
_unmount(archive)
|
|
||||||
|
|
||||||
report = RestoreService(sf, config=config).preflight(location["id"])
|
|
||||||
assert report["state"] == "blocked"
|
|
||||||
assert "location_offline" in _codes(report)
|
|
||||||
|
|
||||||
|
|
||||||
def test_preflight_blocks_wrong_volume(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, _ = _archived(sf, config, lib, archive)
|
|
||||||
(archive / MARKER_NAME).write_text('{"media_id": "someone-elses-disk"}', encoding="utf-8")
|
|
||||||
|
|
||||||
report = RestoreService(sf, config=config).preflight(location["id"])
|
|
||||||
assert report["state"] == "blocked"
|
|
||||||
assert "wrong_volume" in _codes(report)
|
|
||||||
|
|
||||||
|
|
||||||
def test_preflight_blocks_changed_archive_bytes(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
with sf() as session:
|
|
||||||
archived_file = archive / session.get(Asset, asset_id).archive_path
|
|
||||||
archived_file.write_bytes(b"not the photo that was archived")
|
|
||||||
|
|
||||||
report = RestoreService(sf, config=config).preflight(location["id"])
|
|
||||||
assert report["state"] == "blocked"
|
|
||||||
assert "bytes_changed" in _codes(report)
|
|
||||||
with pytest.raises(ArchiveError) as error:
|
|
||||||
_restore(sf, config, location["id"])
|
|
||||||
assert error.value.code == "blocked"
|
|
||||||
|
|
||||||
|
|
||||||
def test_preflight_blocks_insufficient_capacity(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, _ = _archived(sf, config, lib, archive)
|
|
||||||
greedy = config.model_copy(
|
|
||||||
update={"archive_free_space_reserve_bytes": 1 << 62} # more than any disk has
|
|
||||||
)
|
|
||||||
|
|
||||||
report = RestoreService(sf, config=greedy).preflight(location["id"])
|
|
||||||
assert report["state"] == "blocked"
|
|
||||||
assert "insufficient_capacity" in _codes(report)
|
|
||||||
|
|
||||||
|
|
||||||
def test_token_changes_with_the_scope(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive)
|
|
||||||
service = RestoreService(sf, config=config)
|
|
||||||
|
|
||||||
whole = service.preflight(location["id"])["token"]
|
|
||||||
partial = service.preflight(location["id"], [sorted(ids.values())[0]])["token"]
|
|
||||||
assert whole != partial
|
|
||||||
assert service.verify_token(whole, location["id"])
|
|
||||||
assert not service.verify_token(partial, location["id"])
|
|
||||||
|
|
||||||
|
|
||||||
# ── restore ──────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_returns_bytes_identity_and_decisions(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive)
|
|
||||||
archived_hashes = {
|
|
||||||
asset_id: asset.current_sha256 for asset_id, asset in _assets(sf).items()
|
|
||||||
}
|
|
||||||
|
|
||||||
service, plan, result = _restore(sf, config, location["id"])
|
|
||||||
assert (result["restored"], result["failed"], result["state"]) == (2, 0, "complete")
|
|
||||||
|
|
||||||
for asset_id, asset in _assets(sf).items():
|
|
||||||
assert asset.availability_state == availability.ACTIVE
|
|
||||||
assert asset.current_path == str(lib / asset.archive_path)
|
|
||||||
assert sha256_file(asset.current_path) == archived_hashes[asset_id]
|
|
||||||
# The archived copy is a copy: restoring never empties the medium.
|
|
||||||
assert (archive / asset.archive_path).exists()
|
|
||||||
assert asset.archive_location_id == location["id"]
|
|
||||||
|
|
||||||
with sf() as session:
|
|
||||||
# Identity and decisions survived: same ids, same reviews, new occurrence.
|
|
||||||
assert set(ids.values()) == {a.id for a in session.scalars(select(Asset))}
|
|
||||||
assert {r.decision for r in session.scalars(select(SafetyReview))} == {"sfw"}
|
|
||||||
occurrences = [
|
|
||||||
row.reason
|
|
||||||
for row in session.scalars(
|
|
||||||
select(AssetPath).where(AssetPath.asset_id == sorted(ids.values())[0])
|
|
||||||
)
|
|
||||||
]
|
|
||||||
assert "restore" in occurrences
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_never_overwrites_a_collision(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
with sf() as session:
|
|
||||||
archive_path = session.get(Asset, asset_id).archive_path
|
|
||||||
occupied = lib / archive_path
|
|
||||||
occupied.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
occupied.write_bytes(b"a different photo already lives here")
|
|
||||||
before = occupied.read_bytes()
|
|
||||||
|
|
||||||
service, plan, result = _restore(sf, config, location["id"])
|
|
||||||
assert result["failed"] == 0
|
|
||||||
|
|
||||||
assert occupied.read_bytes() == before # untouched
|
|
||||||
restored = _assets(sf)[asset_id].current_path
|
|
||||||
assert restored != str(occupied)
|
|
||||||
assert "(restored)" in restored
|
|
||||||
assert sha256_file(restored) == sha256_file(archive / archive_path)
|
|
||||||
|
|
||||||
|
|
||||||
def test_apply_refuses_a_destination_taken_after_planning(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
|
||||||
service = RestoreService(sf, config=config)
|
|
||||||
token = service.preflight(location["id"])["token"]
|
|
||||||
plan = service.create(location["id"], None, token=token)
|
|
||||||
|
|
||||||
# Someone drops a file exactly where the plan intends to publish.
|
|
||||||
destination = plan["operations"][0]["destination_path"]
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
Path(destination).parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
Path(destination).write_bytes(b"squatter")
|
|
||||||
|
|
||||||
result = service.apply(plan["id"])
|
|
||||||
assert result["failed"] == 1
|
|
||||||
operation = service.journal.operations(plan["id"])[0]
|
|
||||||
assert operation["journal_state"] == ArchiveState.FAILED
|
|
||||||
assert operation["error_code"] == "destination_exists"
|
|
||||||
assert Path(destination).read_bytes() == b"squatter"
|
|
||||||
assert _assets(sf)[next(iter(ids.values()))].availability_state == availability.ARCHIVED_ONLINE
|
|
||||||
|
|
||||||
|
|
||||||
def test_changed_archive_bytes_mark_the_asset_divergent(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
service = RestoreService(sf, config=config)
|
|
||||||
token = service.preflight(location["id"])["token"]
|
|
||||||
plan = service.create(location["id"], None, token=token)
|
|
||||||
|
|
||||||
# The medium's copy is edited after the plan was approved.
|
|
||||||
with sf() as session:
|
|
||||||
archived_file = archive / session.get(Asset, asset_id).archive_path
|
|
||||||
archived_file.write_bytes(b"edited on the shelf")
|
|
||||||
|
|
||||||
result = service.apply(plan["id"])
|
|
||||||
assert result["failed"] == 1
|
|
||||||
operation = service.journal.operations(plan["id"])[0]
|
|
||||||
assert operation["error_code"] == "bytes_changed"
|
|
||||||
asset = _assets(sf)[asset_id]
|
|
||||||
assert asset.archive_divergent_at is not None # durable divergence
|
|
||||||
assert asset.availability_state == availability.ARCHIVED_ONLINE
|
|
||||||
assert asset.current_path is None # nothing was published
|
|
||||||
|
|
||||||
|
|
||||||
# ── interruption and idempotency ─────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_interrupted_before_publishing_is_resumable(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
|
||||||
service = RestoreService(sf, config=config)
|
|
||||||
token = service.preflight(location["id"])["token"]
|
|
||||||
plan = service.create(location["id"], None, token=token)
|
|
||||||
operation = service.journal.operations(plan["id"])[0]
|
|
||||||
|
|
||||||
# Model a kill right after the intent was written: nothing published yet.
|
|
||||||
service.journal.begin(operation["id"], worker_id="killed", fencing_token=1)
|
|
||||||
|
|
||||||
status = service.recovery_status()
|
|
||||||
assert status["operations"][0]["classification"] == "resumable"
|
|
||||||
assert service.recover() == {"resumed": 1, "completed": 0, "manual": 0}
|
|
||||||
assert service.journal.operations(plan["id"])[0]["journal_state"] == ArchiveState.PLANNED
|
|
||||||
|
|
||||||
result = service.apply(plan["id"])
|
|
||||||
assert result["failed"] == 0
|
|
||||||
assert _assets(sf)[next(iter(ids.values()))].availability_state == availability.ACTIVE
|
|
||||||
|
|
||||||
|
|
||||||
def test_interrupted_after_publishing_is_finished_by_recovery(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
|
||||||
asset_id = next(iter(ids.values()))
|
|
||||||
service = RestoreService(sf, config=config)
|
|
||||||
token = service.preflight(location["id"])["token"]
|
|
||||||
plan = service.create(location["id"], None, token=token)
|
|
||||||
operation = service.journal.operations(plan["id"])[0]
|
|
||||||
|
|
||||||
# Model a kill between the published copy and the database update.
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
destination = Path(operation["destination_path"])
|
|
||||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
shutil.copy2(operation["source_path"], destination)
|
|
||||||
service.journal.begin(operation["id"], worker_id="killed", fencing_token=1)
|
|
||||||
service.journal.transition(operation["id"], ArchiveState.VERIFIED, fencing_token=1)
|
|
||||||
|
|
||||||
assert service.recovery_status()["operations"][0]["classification"] == "forward"
|
|
||||||
assert service.recover()["completed"] == 1
|
|
||||||
asset = _assets(sf)[asset_id]
|
|
||||||
assert asset.availability_state == availability.ACTIVE
|
|
||||||
assert asset.current_path == str(destination)
|
|
||||||
|
|
||||||
# Repeated recovery and a repeated apply converge on the same state.
|
|
||||||
assert service.recover() == {"resumed": 0, "completed": 0, "manual": 0}
|
|
||||||
again = service.apply(plan["id"])
|
|
||||||
assert (again["skipped"], again["failed"]) == (1, 0)
|
|
||||||
assert _assets(sf)[asset_id].current_path == str(destination)
|
|
||||||
|
|
||||||
|
|
||||||
def test_restored_state_survives_restart_and_rescan(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive)
|
|
||||||
_restore(sf, config, location["id"])
|
|
||||||
|
|
||||||
restarted = create_session_factory(create_db_engine(config.database_url))
|
|
||||||
InventoryService(restarted).scan(lib)
|
|
||||||
|
|
||||||
assets = _assets(restarted)
|
|
||||||
assert set(assets) == set(ids.values()) # no new identities from the rescan
|
|
||||||
for asset in assets.values():
|
|
||||||
assert asset.availability_state == availability.ACTIVE
|
|
||||||
assert asset.missing_at is None
|
|
||||||
|
|
||||||
# Nothing is archived at that location any more, so there is nothing to restore.
|
|
||||||
again = RestoreService(restarted, config=config).preflight(location["id"])
|
|
||||||
assert _codes(again) == {"empty_scope"}
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_api_round_trip(tmp_path):
|
|
||||||
config, sf, lib, archive = _env(tmp_path)
|
|
||||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
|
||||||
|
|
||||||
with TestClient(create_app(config)) as client:
|
|
||||||
report = client.post(
|
|
||||||
"/api/v1/restore-preflight", json={"location_id": location["id"]}
|
|
||||||
).json()
|
|
||||||
assert report["state"] == "ready"
|
|
||||||
|
|
||||||
stale = client.post(
|
|
||||||
"/api/v1/restore-plans",
|
|
||||||
json={"location_id": location["id"], "token": "r1:not-the-token"},
|
|
||||||
)
|
|
||||||
assert stale.status_code == 409
|
|
||||||
|
|
||||||
created = client.post(
|
|
||||||
"/api/v1/restore-plans",
|
|
||||||
json={"location_id": location["id"], "token": report["token"]},
|
|
||||||
)
|
|
||||||
assert created.status_code == 201
|
|
||||||
plan_id = created.json()["id"]
|
|
||||||
assert created.json()["direction"] == "restore"
|
|
||||||
|
|
||||||
# The plan is visible and applying it queues work on the archiver lane.
|
|
||||||
assert client.get(f"/api/v1/restore-plans/{plan_id}").status_code == 200
|
|
||||||
queued = client.post(f"/api/v1/restore-plans/{plan_id}/apply")
|
|
||||||
assert queued.status_code == 200
|
|
||||||
assert queued.json()["job"]["job_type"] == "restore_plan"
|
|
||||||
assert queued.json()["job"]["lock_key"] == "archive"
|
|
||||||
assert client.get("/api/v1/restore-recovery").json()["manual"] == []
|
|
||||||
|
|
||||||
assert _assets(sf)[next(iter(ids.values()))].availability_state == (
|
|
||||||
availability.ARCHIVED_ONLINE # the worker, not the request, does the work
|
|
||||||
)
|
|
||||||
@@ -3,8 +3,6 @@ same fixture files (real exiftool-written keywords)."""
|
|||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pytest
|
import pytest
|
||||||
@@ -13,10 +11,6 @@ from PIL import Image
|
|||||||
from photo_pipeline.integrations import exiftool
|
from photo_pipeline.integrations import exiftool
|
||||||
from photo_pipeline.services import safety
|
from photo_pipeline.services import safety
|
||||||
|
|
||||||
# The donor is frozen in the read-only archive (US07-01); parity still compares
|
|
||||||
# against it, so this suite is the second and last place that may import it.
|
|
||||||
ARCHIVED_SOURCES = Path(__file__).resolve().parents[2] / "legacy_cli_archive" / "src"
|
|
||||||
|
|
||||||
EXIFTOOL = shutil.which("exiftool")
|
EXIFTOOL = shutil.which("exiftool")
|
||||||
pytestmark = pytest.mark.skipif(EXIFTOOL is None, reason="exiftool not installed")
|
pytestmark = pytest.mark.skipif(EXIFTOOL is None, reason="exiftool not installed")
|
||||||
|
|
||||||
@@ -37,7 +31,6 @@ def _tag(path, keyword):
|
|||||||
|
|
||||||
|
|
||||||
def test_extracted_marks_and_partition_match_donor(tmp_path):
|
def test_extracted_marks_and_partition_match_donor(tmp_path):
|
||||||
sys.path.insert(0, str(ARCHIVED_SOURCES)) # the donor lives in the archive now
|
|
||||||
donor_exif = pytest.importorskip("nsfwtag.exif")
|
donor_exif = pytest.importorskip("nsfwtag.exif")
|
||||||
|
|
||||||
nsfw = _jpeg(tmp_path / "nsfw.jpg", 1)
|
nsfw = _jpeg(tmp_path / "nsfw.jpg", 1)
|
||||||
|
|||||||
@@ -1,136 +0,0 @@
|
|||||||
"""US07-02: what an unhandled failure says, and where the library ends.
|
|
||||||
|
|
||||||
Two boundaries that only show up below the HTTP surface:
|
|
||||||
|
|
||||||
* Exception text is where internals leak — absolute paths, SQL, and occasionally a
|
|
||||||
credential passed to the call that blew up. The first tests drive the real
|
|
||||||
application with a route that raises such an exception (no production route does)
|
|
||||||
and assert the client sees only a code.
|
|
||||||
* The path the database recorded is not the path the filesystem will open a moment
|
|
||||||
later. Analysis is the one stage whose bytes leave this machine, so it resolves
|
|
||||||
the source against the library roots immediately before the provider call.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import uuid
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from PIL import Image
|
|
||||||
from starlette.testclient import TestClient
|
|
||||||
|
|
||||||
from photo_pipeline.api.app import create_app
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
|
||||||
from photo_pipeline.models import AnalysisResult, Asset
|
|
||||||
from photo_pipeline.services.analysis import AnalysisService
|
|
||||||
from photo_pipeline.services.safety import SafetyService
|
|
||||||
|
|
||||||
BOOM = "sqlite:///Users/someone/Pictures/private.db failed with key sk-secret-123"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client(tmp_path):
|
|
||||||
app = create_app(Config(data_dir=tmp_path / "data"))
|
|
||||||
|
|
||||||
@app.get("/api/v1/boom")
|
|
||||||
def boom():
|
|
||||||
raise RuntimeError(BOOM)
|
|
||||||
|
|
||||||
with TestClient(app, raise_server_exceptions=False) as test_client:
|
|
||||||
yield test_client
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_unhandled_error_returns_a_bare_envelope(client, caplog):
|
|
||||||
with caplog.at_level(logging.ERROR):
|
|
||||||
response = client.get("/api/v1/boom")
|
|
||||||
|
|
||||||
assert response.status_code == 500
|
|
||||||
assert response.json() == {"error": {"code": "internal_error", "message": "internal error"}}
|
|
||||||
assert BOOM not in response.text and "Traceback" not in response.text
|
|
||||||
# The operator still gets the whole story, on the server side.
|
|
||||||
assert BOOM in caplog.text
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_refusal_response_still_carries_the_default_headers(client):
|
|
||||||
"""A 500 escaping the middleware's response path would also escape its headers."""
|
|
||||||
response = client.get("/api/v1/boom")
|
|
||||||
assert response.headers["x-content-type-options"] == "nosniff"
|
|
||||||
assert response.headers["x-frame-options"] == "DENY"
|
|
||||||
|
|
||||||
|
|
||||||
# ── the library boundary, revalidated at the moment of use ───────────────────
|
|
||||||
|
|
||||||
|
|
||||||
class RecordingProvider:
|
|
||||||
def __init__(self):
|
|
||||||
self.calls = []
|
|
||||||
|
|
||||||
def analyze(self, path, *, album_hint):
|
|
||||||
self.calls.append(path)
|
|
||||||
return {"description": "a photo", "tags": []}
|
|
||||||
|
|
||||||
|
|
||||||
def _library(tmp_path):
|
|
||||||
lib = tmp_path / "lib"
|
|
||||||
lib.mkdir()
|
|
||||||
outside = tmp_path / "outside"
|
|
||||||
outside.mkdir()
|
|
||||||
for path in (lib / "inside.jpg", outside / "private.jpg"):
|
|
||||||
Image.new("RGB", (8, 8), "blue").save(path)
|
|
||||||
(tmp_path / "data").mkdir()
|
|
||||||
config = Config.from_env({"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data")})
|
|
||||||
run_migrations(config.database_url)
|
|
||||||
return lib, outside, create_session_factory(create_db_engine(config.database_url))
|
|
||||||
|
|
||||||
|
|
||||||
def _sfw_asset(sf, path):
|
|
||||||
asset = Asset(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
original_path=str(path),
|
|
||||||
current_path=str(path),
|
|
||||||
discovered_at=datetime.now(timezone.utc),
|
|
||||||
hash_version=1,
|
|
||||||
)
|
|
||||||
with sf() as session:
|
|
||||||
session.add(asset)
|
|
||||||
session.commit()
|
|
||||||
SafetyService(sf).decide(asset.id, "sfw", write_exif=False)
|
|
||||||
return asset.id
|
|
||||||
|
|
||||||
|
|
||||||
def test_analysis_will_not_send_a_file_that_left_the_library(tmp_path):
|
|
||||||
"""A link swapped under an asset after the scan points at something the user
|
|
||||||
never put in the library. Those bytes must not reach the vision provider — it is
|
|
||||||
the one place in the pipeline where content leaves this machine."""
|
|
||||||
lib, outside, sf = _library(tmp_path)
|
|
||||||
inside = lib / "inside.jpg"
|
|
||||||
asset_id = _sfw_asset(sf, inside)
|
|
||||||
inside.unlink()
|
|
||||||
inside.symlink_to(outside / "private.jpg")
|
|
||||||
|
|
||||||
provider = RecordingProvider()
|
|
||||||
result = AnalysisService(sf, provider=provider, library_roots=(lib,)).run([asset_id])
|
|
||||||
|
|
||||||
assert provider.calls == [], "the provider must never have been constructed a request"
|
|
||||||
assert result == {"analyzed": 0, "skipped": 0, "errors": 1}
|
|
||||||
with sf() as session:
|
|
||||||
row = session.get(AnalysisResult, asset_id)
|
|
||||||
# The failure is visible and names no path.
|
|
||||||
assert row.status == "error"
|
|
||||||
assert "outside the configured library roots" in row.error_message
|
|
||||||
assert str(outside) not in row.error_message
|
|
||||||
|
|
||||||
|
|
||||||
def test_analysis_still_reads_a_file_that_stayed_inside(tmp_path):
|
|
||||||
"""The guard must resolve real paths, not refuse everything."""
|
|
||||||
lib, _, sf = _library(tmp_path)
|
|
||||||
asset_id = _sfw_asset(sf, lib / "inside.jpg")
|
|
||||||
|
|
||||||
provider = RecordingProvider()
|
|
||||||
result = AnalysisService(sf, provider=provider, library_roots=(lib,)).run([asset_id])
|
|
||||||
|
|
||||||
assert result["analyzed"] == 1
|
|
||||||
assert provider.calls == [str((lib / "inside.jpg").resolve())]
|
|
||||||
@@ -131,27 +131,6 @@
|
|||||||
"tests/unit/test_archive_journal_states.py",
|
"tests/unit/test_archive_journal_states.py",
|
||||||
"tests/integration/test_archive_transfer.py",
|
"tests/integration/test_archive_transfer.py",
|
||||||
"tests/integration/test_archive_recovery.py"
|
"tests/integration/test_archive_recovery.py"
|
||||||
],
|
|
||||||
"US06-03": [
|
|
||||||
"tests/integration/test_offline_assets.py"
|
|
||||||
],
|
|
||||||
"US06-04": [
|
|
||||||
"tests/integration/test_restore.py"
|
|
||||||
],
|
|
||||||
"US06-05": [
|
|
||||||
"tests/e2e/test_archive_ui.py"
|
|
||||||
],
|
|
||||||
"US06-06": [
|
|
||||||
"tests/e2e/test_phase_f_pipeline.py"
|
|
||||||
],
|
|
||||||
"US07-01": [
|
|
||||||
"tests/unit/test_legacy_archive.py",
|
|
||||||
"tests/integration/test_legacy_import.py"
|
|
||||||
],
|
|
||||||
"US07-02": [
|
|
||||||
"tests/unit/test_security_policy.py",
|
|
||||||
"tests/integration/test_security_boundaries.py",
|
|
||||||
"tests/e2e/test_security.py"
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,173 +0,0 @@
|
|||||||
"""Archive lint (US07-01): the frozen CLI archive is complete, honest, and inert.
|
|
||||||
|
|
||||||
Freezing the donors is only worth something if three things stay true: the archive
|
|
||||||
still holds exactly the bytes it claims, it holds no secret, and production cannot
|
|
||||||
reach it. Each of those is a one-line mistake away — a helpful `sys.path` insert, a
|
|
||||||
copied `.env`, an edited "just this once" source — so each is asserted here.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
import yaml
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[2]
|
|
||||||
ARCHIVE = REPO / "legacy_cli_archive"
|
|
||||||
SOURCES = ARCHIVE / "src"
|
|
||||||
LEDGER = ARCHIVE / "donor_ledger.yaml"
|
|
||||||
CHECKSUMS = ARCHIVE / "CHECKSUMS.sha256"
|
|
||||||
|
|
||||||
# Everything the concept requires an archive to carry (§3 "Donor-first CLI
|
|
||||||
# migration and archival"): the sources, their docs, a dependency lock, schema
|
|
||||||
# notes, a redacted sample configuration, the ledger, and recorded checksums.
|
|
||||||
REQUIRED_ARTIFACTS = (
|
|
||||||
"README.md",
|
|
||||||
"donor_ledger.yaml",
|
|
||||||
"CHECKSUMS.sha256",
|
|
||||||
"requirements-lock.txt",
|
|
||||||
"photo_analyzer.env.sample",
|
|
||||||
)
|
|
||||||
ARCHIVED_MODULES = ("photo_analyzer", "nsfwtag", "webapp", "nsfw_tag", "compare_models")
|
|
||||||
# Only the two suites that compare against the donors may put the archive on the
|
|
||||||
# import path; every other tree must not name it at all.
|
|
||||||
IMPORT_ALLOWED = {
|
|
||||||
REPO / "tests" / "characterization" / "conftest.py",
|
|
||||||
REPO / "tests" / "characterization" / "test_donor_ledger.py", # lints the ledger there
|
|
||||||
REPO / "tests" / "integration" / "test_safety_parity.py",
|
|
||||||
Path(__file__),
|
|
||||||
}
|
|
||||||
SECRET_PATTERNS = (
|
|
||||||
re.compile(r"sk-(?!REPLACE_WITH_YOUR_KEY)[A-Za-z0-9_\-]{16,}"),
|
|
||||||
re.compile(r"AIza[0-9A-Za-z_\-]{20,}"), # Google API keys
|
|
||||||
re.compile(r"(?i)api[_-]?key\s*[=:]\s*['\"][A-Za-z0-9_\-]{16,}['\"]"),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _archived_files() -> list[Path]:
|
|
||||||
return sorted(p for p in SOURCES.rglob("*") if p.is_file() and "__pycache__" not in p.parts)
|
|
||||||
|
|
||||||
|
|
||||||
# ── completeness ─────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_archive_carries_every_required_artifact():
|
|
||||||
for artifact in REQUIRED_ARTIFACTS:
|
|
||||||
path = ARCHIVE / artifact
|
|
||||||
assert path.is_file(), f"archive is missing {artifact}"
|
|
||||||
assert path.stat().st_size > 0, f"archive artifact {artifact} is empty"
|
|
||||||
assert (SOURCES / "photo_analyzer.py").is_file()
|
|
||||||
assert (SOURCES / "nsfwtag" / "README.md").is_file(), "donor docs must be archived too"
|
|
||||||
assert (SOURCES / "webapp" / "README.md").is_file()
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_ledger_source_is_present_in_the_archive():
|
|
||||||
rows = yaml.safe_load(LEDGER.read_text(encoding="utf-8"))["rows"]
|
|
||||||
for row in rows:
|
|
||||||
source = SOURCES / row["source"]["file"]
|
|
||||||
assert source.is_file(), f"{row['id']}: {row['source']['file']} is not archived"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_readme_records_provenance_and_the_no_import_rule():
|
|
||||||
readme = (ARCHIVE / "README.md").read_text(encoding="utf-8")
|
|
||||||
for expected in ("CHECKSUMS.sha256", "requirements-lock.txt", "donor_ledger.yaml"):
|
|
||||||
assert expected in readme, f"README does not point at {expected}"
|
|
||||||
assert "Schema notes" in readme, "the donor's schema must be documented"
|
|
||||||
assert "nsfw_scores.csv" in readme, "the CSV's fate must be documented"
|
|
||||||
|
|
||||||
|
|
||||||
# ── integrity ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_archived_source_matches_its_checksum():
|
|
||||||
"""A frozen archive that silently drifts is not evidence of anything."""
|
|
||||||
recorded = {}
|
|
||||||
for line in CHECKSUMS.read_text(encoding="utf-8").splitlines():
|
|
||||||
digest, _, rel = line.partition(" ")
|
|
||||||
if rel:
|
|
||||||
recorded[rel.strip()] = digest
|
|
||||||
actual = {
|
|
||||||
str(path.relative_to(ARCHIVE)): hashlib.sha256(path.read_bytes()).hexdigest()
|
|
||||||
for path in _archived_files()
|
|
||||||
}
|
|
||||||
assert actual == recorded, "archived sources and CHECKSUMS.sha256 disagree"
|
|
||||||
|
|
||||||
|
|
||||||
# ── redaction ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_archive_contains_no_credential():
|
|
||||||
for path in [*_archived_files(), *(ARCHIVE / a for a in REQUIRED_ARTIFACTS)]:
|
|
||||||
if path.suffix in (".png", ".jpg", ".webp"):
|
|
||||||
continue
|
|
||||||
text = path.read_text(encoding="utf-8", errors="replace")
|
|
||||||
for pattern in SECRET_PATTERNS:
|
|
||||||
assert not pattern.search(text), f"possible secret in {path.relative_to(REPO)}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_sample_configuration_is_a_placeholder_only():
|
|
||||||
sample = (ARCHIVE / "photo_analyzer.env.sample").read_text(encoding="utf-8")
|
|
||||||
assert "REDACTED" in sample
|
|
||||||
# Placeholders are not key-shaped, so neither a scanner nor a reader can
|
|
||||||
# mistake the sample for a credential.
|
|
||||||
assert "LLM_API_KEY=<" in sample
|
|
||||||
# No real env file, database, log, or CSV may ride along in the archive.
|
|
||||||
strays = [
|
|
||||||
p.name
|
|
||||||
for p in ARCHIVE.rglob("*")
|
|
||||||
if p.is_file() and p.suffix in (".env", ".db", ".sqlite", ".sqlite3", ".log", ".csv")
|
|
||||||
]
|
|
||||||
assert strays == [], f"unexpected runtime files archived: {strays}"
|
|
||||||
|
|
||||||
|
|
||||||
# ── inertness ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def test_production_code_never_imports_an_archived_module():
|
|
||||||
for path in (REPO / "photo_pipeline").rglob("*.py"):
|
|
||||||
text = path.read_text(encoding="utf-8")
|
|
||||||
for module in ARCHIVED_MODULES:
|
|
||||||
assert not re.search(rf"^\s*(import|from)\s+{module}\b", text, re.MULTILINE), \
|
|
||||||
f"{path.relative_to(REPO)} imports the archived {module}"
|
|
||||||
assert "legacy_cli_archive" not in text, \
|
|
||||||
f"{path.relative_to(REPO)} names the archive"
|
|
||||||
|
|
||||||
|
|
||||||
def test_only_the_parity_suites_put_the_archive_on_the_import_path():
|
|
||||||
for path in (REPO / "tests").rglob("*.py"):
|
|
||||||
if path in IMPORT_ALLOWED or "__pycache__" in path.parts:
|
|
||||||
continue
|
|
||||||
assert "legacy_cli_archive" not in path.read_text(encoding="utf-8"), \
|
|
||||||
f"{path.relative_to(REPO)} reaches into the archive"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_archived_modules_are_unimportable_from_a_clean_interpreter():
|
|
||||||
"""The real check: a fresh process with the repo on its path cannot load them."""
|
|
||||||
script = (
|
|
||||||
"import importlib.util, sys; "
|
|
||||||
f"sys.path.insert(0, {str(REPO)!r}); "
|
|
||||||
"print([m for m in "
|
|
||||||
f"{list(ARCHIVED_MODULES)!r}"
|
|
||||||
" if importlib.util.find_spec(m) is not None])"
|
|
||||||
)
|
|
||||||
result = subprocess.run(
|
|
||||||
[sys.executable, "-c", script], capture_output=True, text=True, cwd=str(REPO), check=True
|
|
||||||
)
|
|
||||||
assert result.stdout.strip() == "[]", f"still importable: {result.stdout.strip()}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_application_starts_without_the_archive(tmp_path):
|
|
||||||
"""Nothing in the runtime path may need the frozen sources to exist."""
|
|
||||||
pytest.importorskip("fastapi")
|
|
||||||
from photo_pipeline.api.app import create_app
|
|
||||||
from photo_pipeline.config import Config
|
|
||||||
|
|
||||||
config = Config.from_env(
|
|
||||||
{"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"), "PHOTO_PIPELINE_LIBRARY_ROOTS": ""}
|
|
||||||
)
|
|
||||||
assert create_app(config) is not None
|
|
||||||
@@ -81,42 +81,3 @@ def test_symlink_within_root_is_allowed(tmp_path):
|
|||||||
pytest.skip("cannot create symlink on this platform")
|
pytest.skip("cannot create symlink on this platform")
|
||||||
found = path_policy.discover([root])
|
found = path_policy.discover([root])
|
||||||
assert target in found and link in found
|
assert target in found and link in found
|
||||||
|
|
||||||
|
|
||||||
def test_resolve_in_roots_returns_the_path_the_caller_must_use(tmp_path):
|
|
||||||
"""The resolved path is the answer, not a yes/no: a caller that revalidates one
|
|
||||||
name and then opens another has an open symlink race (US07-02)."""
|
|
||||||
root = tmp_path / "lib"
|
|
||||||
(root / "sub").mkdir(parents=True)
|
|
||||||
target = root / "sub" / "real.jpg"
|
|
||||||
target.write_bytes(b"x")
|
|
||||||
link = root / "alias.jpg"
|
|
||||||
os.symlink(target, link)
|
|
||||||
|
|
||||||
assert path_policy.resolve_in_roots([root], link) == target.resolve()
|
|
||||||
assert path_policy.resolve_in_roots([root], target) == target.resolve()
|
|
||||||
|
|
||||||
|
|
||||||
def test_resolve_in_roots_refuses_escapes_without_naming_them(tmp_path):
|
|
||||||
root = tmp_path / "lib"
|
|
||||||
root.mkdir()
|
|
||||||
outside = tmp_path / "outside.jpg"
|
|
||||||
outside.write_bytes(b"x")
|
|
||||||
link = root / "alias.jpg"
|
|
||||||
os.symlink(outside, link)
|
|
||||||
|
|
||||||
for candidate in (link, outside, root / "_IGNORE" / "a.jpg"):
|
|
||||||
with pytest.raises(path_policy.PathPolicyError) as raised:
|
|
||||||
path_policy.resolve_in_roots([root], candidate)
|
|
||||||
assert str(outside) not in str(raised.value)
|
|
||||||
|
|
||||||
|
|
||||||
def test_resolve_in_roots_checks_every_configured_root(tmp_path):
|
|
||||||
first, second = tmp_path / "one", tmp_path / "two"
|
|
||||||
first.mkdir()
|
|
||||||
second.mkdir()
|
|
||||||
photo = second / "b.jpg"
|
|
||||||
photo.write_bytes(b"x")
|
|
||||||
assert path_policy.resolve_in_roots([first, second], photo) == photo.resolve()
|
|
||||||
# No configured boundary means nothing to check against.
|
|
||||||
assert path_policy.resolve_in_roots([], photo) == photo
|
|
||||||
|
|||||||
@@ -1,169 +0,0 @@
|
|||||||
"""US07-02: the request-admission policy, enumerated.
|
|
||||||
|
|
||||||
``evaluate`` decides every refusal the API can make before a route runs, so the
|
|
||||||
whole local-web threat model is one table here: who may call, from where, with what
|
|
||||||
proof. The middleware and the endpoints are covered black box in
|
|
||||||
``tests/e2e/test_security.py``; this file pins the rules themselves, including the
|
|
||||||
combinations a browser can produce but a test client rarely does.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from photo_pipeline.api.security import (
|
|
||||||
CSRF_HEADER,
|
|
||||||
PUBLIC_PATHS,
|
|
||||||
Session,
|
|
||||||
evaluate,
|
|
||||||
split_host,
|
|
||||||
)
|
|
||||||
|
|
||||||
SESSION = Session(id="session-id", csrf_token="csrf-token")
|
|
||||||
HOST = "127.0.0.1:8000"
|
|
||||||
LIMIT = 1024
|
|
||||||
|
|
||||||
|
|
||||||
def check(method="GET", path="/api/v1/workflow", **headers):
|
|
||||||
"""Evaluate a request that is authenticated and same-origin unless overridden."""
|
|
||||||
sent = {
|
|
||||||
"host": HOST,
|
|
||||||
"cookie-session": SESSION.id,
|
|
||||||
CSRF_HEADER: SESSION.csrf_token,
|
|
||||||
}
|
|
||||||
sent.update({name.replace("_", "-"): value for name, value in headers.items()})
|
|
||||||
sent = {name: value for name, value in sent.items() if value is not None}
|
|
||||||
return evaluate(
|
|
||||||
method=method,
|
|
||||||
path=path,
|
|
||||||
headers=sent,
|
|
||||||
session=SESSION,
|
|
||||||
max_request_bytes=LIMIT,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_authenticated_same_origin_request_is_admitted():
|
|
||||||
assert check() is None
|
|
||||||
assert check(method="POST", origin="http://127.0.0.1:8000") is None
|
|
||||||
assert check(sec_fetch_site="same-origin") is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("host", ["evil.example", "evil.example:8000", "192.168.1.10:8000", ""])
|
|
||||||
def test_a_non_loopback_host_is_refused(host):
|
|
||||||
"""DNS rebinding: the browser thinks it is talking to the attacker's name, which
|
|
||||||
resolves to 127.0.0.1. The name is the evidence, so the name is checked."""
|
|
||||||
refusal = check(host=host)
|
|
||||||
assert (refusal.status, refusal.code) == (403, "host_not_allowed")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("host", ["127.0.0.1:8000", "localhost:8000", "[::1]:8000", "localhost"])
|
|
||||||
def test_loopback_hosts_are_accepted(host):
|
|
||||||
assert check(host=host) is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"origin",
|
|
||||||
[
|
|
||||||
"http://evil.example",
|
|
||||||
"https://evil.example:8000",
|
|
||||||
"http://127.0.0.1:9999", # another local app is still another origin
|
|
||||||
"http://localhost.evil.example:8000",
|
|
||||||
"null",
|
|
||||||
"file://",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_a_foreign_origin_is_refused(origin):
|
|
||||||
refusal = check(method="POST", origin=origin)
|
|
||||||
assert (refusal.status, refusal.code) == (403, "origin_not_allowed")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("origin", ["http://127.0.0.1:8000", "http://localhost:8000"])
|
|
||||||
def test_this_applications_origin_is_accepted(origin):
|
|
||||||
assert check(method="POST", origin=origin) is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("site", ["cross-site", "same-site"])
|
|
||||||
def test_a_cross_site_fetch_is_refused_even_without_an_origin(site):
|
|
||||||
"""What ``<img src="http://127.0.0.1:8000/...">`` on another page looks like."""
|
|
||||||
refusal = check(path="/api/v1/assets/a1/thumbnail", sec_fetch_site=site)
|
|
||||||
assert (refusal.status, refusal.code) == (403, "cross_site_blocked")
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_user_initiated_navigation_is_accepted():
|
|
||||||
assert check(sec_fetch_site="none") is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_request_without_a_session_is_unauthenticated():
|
|
||||||
for method, path in [
|
|
||||||
("GET", "/api/v1/workflow"),
|
|
||||||
("POST", "/api/v1/jobs"),
|
|
||||||
("GET", "/api/v1/assets/a1/thumbnail"),
|
|
||||||
]:
|
|
||||||
refusal = check(method=method, path=path, cookie_session=None)
|
|
||||||
assert (refusal.status, refusal.code) == (401, "unauthenticated"), path
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_forged_session_is_unauthenticated():
|
|
||||||
refusal = check(cookie_session="guessed")
|
|
||||||
assert (refusal.status, refusal.code) == (401, "unauthenticated")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("path", sorted(PUBLIC_PATHS))
|
|
||||||
def test_health_and_the_bootstrap_stay_reachable_without_a_session(path):
|
|
||||||
assert check(path=path, cookie_session=None) is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_static_shell_needs_no_session():
|
|
||||||
"""It has to load before any JavaScript can ask for one."""
|
|
||||||
assert check(path="/app/index.html", cookie_session=None) is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("method", ["POST", "PUT", "PATCH", "DELETE"])
|
|
||||||
def test_a_mutation_without_a_valid_csrf_token_is_refused(method):
|
|
||||||
for token in (None, "guessed"):
|
|
||||||
refusal = check(method=method, path="/api/v1/jobs", **{CSRF_HEADER: token})
|
|
||||||
assert (refusal.status, refusal.code) == (403, "csrf_failed")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("method", ["GET", "HEAD", "OPTIONS"])
|
|
||||||
def test_reads_need_no_csrf_token(method):
|
|
||||||
assert check(method=method, **{CSRF_HEADER: None}) is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_oversized_body_is_refused_before_it_is_read():
|
|
||||||
refusal = check(method="POST", path="/api/v1/jobs", content_length=str(LIMIT + 1))
|
|
||||||
assert (refusal.status, refusal.code) == (413, "payload_too_large")
|
|
||||||
assert check(method="POST", path="/api/v1/jobs", content_length=str(LIMIT)) is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_host_check_precedes_authentication():
|
|
||||||
"""A refusal must not tell a foreign caller whether its session guess was right."""
|
|
||||||
refusal = check(host="evil.example", cookie_session="guessed")
|
|
||||||
assert refusal.code == "host_not_allowed"
|
|
||||||
|
|
||||||
|
|
||||||
def test_refusals_name_no_path_secret_or_internal():
|
|
||||||
refusals = [
|
|
||||||
check(host="evil.example"),
|
|
||||||
check(method="POST", origin="http://evil.example"),
|
|
||||||
check(cookie_session=None),
|
|
||||||
check(method="POST", **{CSRF_HEADER: None}),
|
|
||||||
]
|
|
||||||
for refusal in refusals:
|
|
||||||
assert SESSION.id not in refusal.message
|
|
||||||
assert SESSION.csrf_token not in refusal.message
|
|
||||||
assert "/" not in refusal.message
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"value,expected",
|
|
||||||
[
|
|
||||||
("127.0.0.1:8000", ("127.0.0.1", "8000")),
|
|
||||||
("localhost", ("localhost", "")),
|
|
||||||
("[::1]:8000", ("[::1]", "8000")),
|
|
||||||
("[::1]", ("[::1]", "")),
|
|
||||||
("", ("", "")),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_split_host(value, expected):
|
|
||||||
assert split_host(value) == expected
|
|
||||||
Reference in New Issue
Block a user