Compare commits
11 Commits
us/US05-03
...
us/US07-01
| Author | SHA1 | Date | |
|---|---|---|---|
| dc1a64203b | |||
| 9b7ee6b560 | |||
| ea65bb764c | |||
| fb8567284d | |||
| 439eb9971e | |||
| b90d1718be | |||
| b4b316acf5 | |||
| 3ccef58796 | |||
| ac9943884d | |||
| 3c440f3d43 | |||
| 675876cad9 |
@@ -80,7 +80,8 @@ are the authoritative backlog.
|
||||
uncertain. Work only on the claimed story and its generated feature branch.
|
||||
4. Read the entire issue, linked specification, dependencies, and acceptance criteria.
|
||||
Reconcile them with the concept before designing or changing code.
|
||||
5. Inspect the legacy CLI donors before replacing applicable behavior. Update the donor
|
||||
5. Inspect the legacy CLI donors — frozen in `legacy_cli_archive/` since US07-01,
|
||||
with their ledger — before replacing applicable behavior. Update the donor
|
||||
ledger and characterization tests required by the story.
|
||||
6. Implement every acceptance criterion and its automated tests.
|
||||
7. Run story-specific tests and the accumulated regression suite required by the epic.
|
||||
|
||||
105
README.md
105
README.md
@@ -136,3 +136,108 @@ work_item/scripts/python -m pytest tests/e2e -m phase_d -q
|
||||
|
||||
The fault barrier is test-only configuration; without `PHOTO_PIPELINE_FAULT_AFTER`
|
||||
the apply path has no crash points. Phases A–C remain green in the full run above.
|
||||
|
||||
### Phase E acceptance gate
|
||||
|
||||
Phase E (Epic E05: Immich upload) is the one stage the application cannot take back,
|
||||
so its gate runs the fake-uploader suites, the black-box upload API journeys, and the
|
||||
browser suite as a single command:
|
||||
|
||||
```bash
|
||||
work_item/scripts/python -m pytest -m phase_e -q
|
||||
```
|
||||
|
||||
- `tests/integration/test_upload_*.py` drive a **real executable** standing in for
|
||||
`immich-go` through the real adapter and `subprocess` — argument construction,
|
||||
output bounding, report parsing, verification, and killing a running process.
|
||||
- `tests/e2e/test_phase_e_pipeline.py` drives a real server and a real durable worker
|
||||
over HTTP: credential failure and an unreachable server, preflight blockers and the
|
||||
explicitly approved partial scope, a new album, an exact duplicate, an upgrade, a
|
||||
retryable failure and its successful retry, a lost acceptance response, verification
|
||||
against Immich, an inconclusive answer resolved by an operator with evidence, bytes
|
||||
edited after upload, cancellation and resume, and an interrupted attempt recovered
|
||||
across a restart.
|
||||
- **EXIF precedes upload** is asserted, not assumed: an album without its verified
|
||||
safety and analysis checkpoints cannot be approved, and the uploader's own argv log
|
||||
proves it was never executed. Each finished upload re-hashes the files in the folder
|
||||
the uploader was handed and requires the persisted SHA-256/SHA-1 to match.
|
||||
- **No secret is retained.** The API key is a sentinel string; after a full upload and
|
||||
verification it must appear in the uploader's argv and nowhere else — not in the
|
||||
database, the retained report, or any response the browser can read.
|
||||
- `tests/e2e/test_uploads_ui.py` covers the browser journeys (preflight preview,
|
||||
confirmation, progress, stopping a run, verification, manual resolution, stale
|
||||
bytes, and recovery after a restart).
|
||||
|
||||
Phases A–D remain green in the full run above.
|
||||
|
||||
### Phase F acceptance gate
|
||||
|
||||
Phase F (Epic E06: archive lifecycle) is the only stage that *removes* originals
|
||||
from the library, and the only one whose storage can walk away in someone's bag.
|
||||
One command runs the archive fault-injection suites, the black-box archive and
|
||||
restore API journeys, and the browser suite:
|
||||
|
||||
```bash
|
||||
work_item/scripts/python -m pytest -m phase_f -q
|
||||
```
|
||||
|
||||
- `tests/integration/test_archive_*.py` and `tests/integration/test_restore.py`
|
||||
drive real files on real filesystems: preflight against a mounted, missing,
|
||||
swapped, read-only, or full medium; copy-verify-remove and the same-filesystem
|
||||
move path; and a crash at **every** persisted journal transition in both transfer
|
||||
modes, asserting that no source is ever removed without a durable, byte-identical
|
||||
archive copy.
|
||||
- `tests/e2e/test_phase_f_pipeline.py` drives a real server and a real durable
|
||||
worker over HTTP: preflight blockers (offline medium, wrong volume, insufficient
|
||||
capacity, bytes changed after upload), a verified archive whose manifest, hashes,
|
||||
and path history are checked on the medium itself, a worker killed at each of
|
||||
`transferring`, `verified`, `removing`, `source_removed`, and `complete`, the
|
||||
evidence-based recovery that follows, offline deduplication of an exact and a
|
||||
fuzzy copy while the medium is away, mount return, restore, and a collision that
|
||||
restores beside its occupant.
|
||||
- **Archived is not missing.** An unmounted medium leaves its photos
|
||||
`archived_offline` — still hashed, still in the duplicate indexes, still
|
||||
previewable through their protected thumbnails — and a rescan neither prunes nor
|
||||
flags them.
|
||||
- **Ambiguity is never guessed.** A journal state the medium contradicts stays
|
||||
`manual`, offers no automatic action, and keeps blocking further archiving until
|
||||
a human decides.
|
||||
- `tests/e2e/test_archive_ui.py` covers the browser journeys (preview with
|
||||
destination identity and reclaimable bytes, blockers and mount instructions,
|
||||
progress split into transfer/verification/removal, interruption and recovery,
|
||||
offline browsing, restore, collision, keyboard confirmation, and reload).
|
||||
|
||||
Phases A–E remain green in the full run above.
|
||||
|
||||
## Legacy CLI archive
|
||||
|
||||
The command-line tools this application was extracted from are frozen in
|
||||
`legacy_cli_archive/` (US07-01): the original sources, their docs, the dependency
|
||||
lock they were last verified against, schema notes, a redacted sample
|
||||
configuration, the donor ledger, and a checksum for every file.
|
||||
|
||||
```bash
|
||||
cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256 # verify the archive
|
||||
work_item/scripts/python -m pytest tests/unit/test_legacy_archive.py -q # lint it
|
||||
```
|
||||
|
||||
They are reference material and rollback evidence only. No module under
|
||||
`photo_pipeline/` imports or executes them, the archive is not on the application's
|
||||
import path, and `tests/unit/test_legacy_archive.py` enforces that along with the
|
||||
checksums and the redaction. Only the two suites that compare *against* the donors —
|
||||
`tests/characterization/` and `tests/integration/test_safety_parity.py` — put the
|
||||
archived sources on `sys.path`.
|
||||
|
||||
The last path-keyed state they owned, `nsfw_scores.csv`, is imported once and then
|
||||
left alone:
|
||||
|
||||
```bash
|
||||
work_item/scripts/python -m photo_pipeline import-legacy-scores /path/to/nsfw_scores.csv --dry-run
|
||||
```
|
||||
|
||||
The import writes scored-but-unreviewed `safety_reviews` rows onto stable asset ids,
|
||||
never invents an asset for an unknown path, never overwrites a human decision, and
|
||||
writes a reconciliation report to the data directory saying exactly what it did.
|
||||
`legacy_cli_archive/donor_ledger.yaml` records every migrated behavior with its
|
||||
target, the tests that pin the donor, the tests that prove the replacement, and each
|
||||
intentional delta; rows still marked `pending` name the story that will resolve them.
|
||||
|
||||
@@ -18,6 +18,8 @@
|
||||
<a href="#/analyze" data-nav="analyze">Analyze</a>
|
||||
<a href="#/albums" data-nav="albums">Albums</a>
|
||||
<a href="#/renames" data-nav="renames">Renames</a>
|
||||
<a href="#/uploads" data-nav="uploads">Upload</a>
|
||||
<a href="#/archive" data-nav="archive">Archive</a>
|
||||
<a href="#/stats" data-nav="stats">Stats</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
@@ -114,4 +114,55 @@ export const api = {
|
||||
request(`/rename-plans/${encodeURIComponent(id)}/rollback`, { method: "POST", ...opts }),
|
||||
renameRecovery: (opts = {}) => request("/rename-recovery", opts),
|
||||
resolveRecovery: (opts = {}) => request("/rename-recovery/resolve", { method: "POST", ...opts }),
|
||||
|
||||
// ── Uploads: preflight, batches, verification ───────────────────────────
|
||||
// The API key never travels through here: preflight reports only whether one is
|
||||
// configured, and every command preview arrives already redacted.
|
||||
uploadPreflight: (payload = {}, opts = {}) =>
|
||||
request("/upload-preflight", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
createUploadBatches: (payload, opts = {}) =>
|
||||
request("/upload-batches", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
listUploadBatches: (opts = {}) => request("/upload-batches", opts),
|
||||
getUploadBatch: (id, opts = {}) => request(`/upload-batches/${encodeURIComponent(id)}`, opts),
|
||||
startUploadBatch: (id, opts = {}) =>
|
||||
request(`/upload-batches/${encodeURIComponent(id)}/start`, { method: "POST", ...opts }),
|
||||
cancelUploadBatch: (id, opts = {}) =>
|
||||
request(`/upload-batches/${encodeURIComponent(id)}/cancel`, { method: "POST", ...opts }),
|
||||
verifyUploadBatch: (id, opts = {}) =>
|
||||
request(`/upload-batches/${encodeURIComponent(id)}/verify`, { method: "POST", ...opts }),
|
||||
resolveUploadItem: (id, payload, opts = {}) =>
|
||||
request(`/upload-batches/${encodeURIComponent(id)}/resolve`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify(payload),
|
||||
...opts,
|
||||
}),
|
||||
uploadVerifications: (id, opts = {}) =>
|
||||
request(`/upload-batches/${encodeURIComponent(id)}/verifications`, opts),
|
||||
|
||||
// ── Archive and restore: destinations, preflight, plans, recovery ────────
|
||||
archiveLocations: (opts = {}) => request("/archive-locations", opts),
|
||||
registerArchiveLocation: (payload, opts = {}) =>
|
||||
request("/archive-locations", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
archivePreflight: (payload, opts = {}) =>
|
||||
request("/archive-preflight", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
createArchivePlan: (payload, opts = {}) =>
|
||||
request("/archive-plans", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
listArchivePlans: (opts = {}) => request("/archive-plans", opts),
|
||||
getArchivePlan: (id, opts = {}) => request(`/archive-plans/${encodeURIComponent(id)}`, opts),
|
||||
applyArchivePlan: (id, opts = {}) =>
|
||||
request(`/archive-plans/${encodeURIComponent(id)}/apply`, { method: "POST", ...opts }),
|
||||
archiveRecovery: (opts = {}) => request("/archive-recovery", opts),
|
||||
resolveArchiveRecovery: (opts = {}) =>
|
||||
request("/archive-recovery/resolve", { method: "POST", ...opts }),
|
||||
restorePreflight: (payload, opts = {}) =>
|
||||
request("/restore-preflight", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
createRestorePlan: (payload, opts = {}) =>
|
||||
request("/restore-plans", { method: "POST", body: JSON.stringify(payload), ...opts }),
|
||||
listRestorePlans: (opts = {}) => request("/restore-plans", opts),
|
||||
getRestorePlan: (id, opts = {}) => request(`/restore-plans/${encodeURIComponent(id)}`, opts),
|
||||
applyRestorePlan: (id, opts = {}) =>
|
||||
request(`/restore-plans/${encodeURIComponent(id)}/apply`, { method: "POST", ...opts }),
|
||||
restoreRecovery: (opts = {}) => request("/restore-recovery", opts),
|
||||
resolveRestoreRecovery: (opts = {}) =>
|
||||
request("/restore-recovery/resolve", { method: "POST", ...opts }),
|
||||
};
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { api } from "./api.js";
|
||||
import { renderArchive, setArchiveRender } from "./archive.js";
|
||||
import { navigate, onRouteChange, parseHash } from "./router.js";
|
||||
import { renderRenames, setRenamesRender } from "./renames.js";
|
||||
import { renderUploads, setUploadsRender } from "./uploads.js";
|
||||
import {
|
||||
renderAlbums,
|
||||
renderAnalyze,
|
||||
@@ -364,6 +366,8 @@ function render() {
|
||||
else if (path === "/analyze") renderAnalyze(root, params);
|
||||
else if (path === "/albums") renderAlbums(root, params);
|
||||
else if (path === "/renames") renderRenames(root, params);
|
||||
else if (path === "/uploads") renderUploads(root, params);
|
||||
else if (path === "/archive") renderArchive(root, params);
|
||||
else if (path === "/stats") renderStats(root, params);
|
||||
else show(errorBanner("Unknown view"));
|
||||
}
|
||||
@@ -371,5 +375,7 @@ function render() {
|
||||
// Let views re-render the current route after a mutation.
|
||||
setRender(render);
|
||||
setRenamesRender(render);
|
||||
setUploadsRender(render);
|
||||
setArchiveRender(render);
|
||||
onRouteChange(render);
|
||||
render();
|
||||
|
||||
870
frontend/js/archive.js
Normal file
870
frontend/js/archive.js
Normal file
@@ -0,0 +1,870 @@
|
||||
// Archive view (US06-05): preview what would leave active storage, confirm it
|
||||
// exactly, watch the transfer, recover an interrupted one, browse what is already
|
||||
// archived, and bring it back.
|
||||
//
|
||||
// Archiving is the only stage that removes originals, so this view never decides
|
||||
// anything itself: the destination's identity, every blocker, the confirmation
|
||||
// token, and what recovery may do all come from the server, and an action the
|
||||
// server would refuse is not offered. Two things follow from that. A medium that
|
||||
// is not mounted produces an instruction naming it rather than a disabled mystery,
|
||||
// and an operation whose evidence is ambiguous offers no button at all.
|
||||
import { api } from "./api.js";
|
||||
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||
import { subscribeJob } from "./events.js";
|
||||
import { navigate } from "./router.js";
|
||||
|
||||
let outcome = null;
|
||||
let activity = [];
|
||||
|
||||
let render = () => {};
|
||||
export function setArchiveRender(fn) {
|
||||
render = fn;
|
||||
}
|
||||
|
||||
// The per-file journal states, split into the three things an operator actually
|
||||
// wants told apart: bytes moving, bytes proven, original removed (concept §9).
|
||||
const PHASES = [
|
||||
["planned", "planned", "waiting"],
|
||||
["transferring", "transfer", "copying to the medium"],
|
||||
["verified", "verified", "archive copy hashed and manifested"],
|
||||
["removing", "removing", "removing the active original"],
|
||||
["complete", "complete", "archived and removed"],
|
||||
["failed", "failed", "left alone for a decision"],
|
||||
];
|
||||
|
||||
const AVAILABILITY_LABEL = {
|
||||
active: "in the library",
|
||||
archived_online: "archived · medium mounted",
|
||||
archived_offline: "archived · medium away",
|
||||
missing_unexpected: "missing — unexplained",
|
||||
};
|
||||
|
||||
export async function renderArchive(root, params = {}) {
|
||||
setActiveNav("archive");
|
||||
|
||||
let locations;
|
||||
try {
|
||||
locations = (await api.archiveLocations()).locations;
|
||||
} catch (error) {
|
||||
root.replaceChildren(errorBanner(`Failed to load archive locations: ${error.message}`));
|
||||
return;
|
||||
}
|
||||
const location = locations.find((l) => l.id === params.location) || locations[0] || null;
|
||||
|
||||
const nodes = [el("h1", {}, "Archive"), locationsCard(locations, location, params)];
|
||||
if (!location) {
|
||||
nodes.push(
|
||||
el(
|
||||
"p",
|
||||
{ class: "muted", "data-testid": "no-locations" },
|
||||
"Register the disk, NAS share, or removable medium that will hold archived originals."
|
||||
),
|
||||
outcomeBanner()
|
||||
);
|
||||
root.replaceChildren(...nodes.filter(Boolean));
|
||||
return;
|
||||
}
|
||||
|
||||
const [preflight, archivePlans, restorePlans, recovery, restoreRecovery, archived, restore] =
|
||||
await Promise.all([
|
||||
load(() => api.archivePreflight({ location_id: location.id })),
|
||||
load(() => api.listArchivePlans()),
|
||||
load(() => api.listRestorePlans()),
|
||||
load(() => api.archiveRecovery()),
|
||||
load(() => api.restoreRecovery()),
|
||||
load(() => api.listAssets({ limit: 200 })),
|
||||
load(() => api.restorePreflight({ location_id: location.id })),
|
||||
]);
|
||||
|
||||
// Both directions share the runs table: one lane moves these originals, so one
|
||||
// history is what an operator has to reason about.
|
||||
const runs = [...plans(archivePlans), ...plans(restorePlans)].sort((a, b) =>
|
||||
(a.created_at || "").localeCompare(b.created_at || "")
|
||||
);
|
||||
const chosen = runs.find((run) => run.id === params.plan) || runs[runs.length - 1] || null;
|
||||
const selectedPlan = chosen ? await load(() => planDetailOf(chosen)) : null;
|
||||
|
||||
nodes.push(
|
||||
preflight ? previewSection(preflight, location) : null,
|
||||
preflight ? confirmBlock(preflight, location) : null,
|
||||
outcomeBanner(),
|
||||
activityLog(),
|
||||
recoverySection(mergeRecovery(recovery, restoreRecovery)),
|
||||
planList(runs, chosen && chosen.id),
|
||||
selectedPlan ? planDetail(selectedPlan) : null,
|
||||
archived ? archivedSection(archived.items, locations) : null,
|
||||
restore ? restoreSection(restore, location) : null
|
||||
);
|
||||
root.replaceChildren(...nodes.filter(Boolean));
|
||||
}
|
||||
|
||||
function plans(listed) {
|
||||
return listed ? listed.plans : [];
|
||||
}
|
||||
|
||||
function planDetailOf(run) {
|
||||
return run.direction === "restore" ? api.getRestorePlan(run.id) : api.getArchivePlan(run.id);
|
||||
}
|
||||
|
||||
// Archive and restore recovery answer the same question about the same lane, so
|
||||
// they are one list; an unresolved item of either kind blocks the other.
|
||||
function mergeRecovery(archive, restore) {
|
||||
if (!archive && !restore) return null;
|
||||
return {
|
||||
operations: [...((archive || {}).operations || []), ...((restore || {}).operations || [])],
|
||||
manual: [...((archive || {}).manual || []), ...((restore || {}).manual || [])],
|
||||
};
|
||||
}
|
||||
|
||||
// A section whose data failed to load must not take the rest of the view with it:
|
||||
// the medium being away is exactly when the archived-asset list matters most.
|
||||
async function load(call) {
|
||||
try {
|
||||
return await call();
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ── locations ────────────────────────────────────────────────────────────────
|
||||
// A location is a medium, not a path: the marker's ``media_id`` is what proves the
|
||||
// right disk is mounted, so it is shown next to the state it produced.
|
||||
function locationsCard(locations, selected, params) {
|
||||
const name = el("input", {
|
||||
type: "text",
|
||||
"data-testid": "location-name",
|
||||
"aria-label": "Archive location name",
|
||||
placeholder: "External disk",
|
||||
});
|
||||
const root = el("input", {
|
||||
type: "text",
|
||||
"data-testid": "location-root",
|
||||
"aria-label": "Archive location path",
|
||||
placeholder: "/Volumes/archive",
|
||||
});
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "archive-locations" },
|
||||
el("h2", {}, "Destinations"),
|
||||
locations.length
|
||||
? el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "locations" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["", "Name", "Root", "Medium", "State", "Last seen"].map((label) =>
|
||||
el("th", { scope: "col" }, label)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...locations.map((location) =>
|
||||
el(
|
||||
"tr",
|
||||
{
|
||||
"data-testid": "location-row",
|
||||
"data-name": location.name,
|
||||
"aria-current": selected && location.id === selected.id ? "true" : false,
|
||||
},
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("input", {
|
||||
type: "radio",
|
||||
name: "archive-location",
|
||||
"data-testid": "select-location",
|
||||
"aria-label": `Use ${location.name}`,
|
||||
checked: selected && location.id === selected.id ? "checked" : false,
|
||||
onchange: () => navigate("/archive", { ...params, location: location.id }),
|
||||
})
|
||||
),
|
||||
el("td", { "data-testid": "location-label" }, location.name),
|
||||
el("td", { class: "path", "data-testid": "location-path" }, location.root),
|
||||
el("td", { class: "path", "data-testid": "location-media" }, location.media_id),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${location.state === "online" ? "complete" : "attention"}`,
|
||||
"data-testid": "location-state",
|
||||
},
|
||||
location.state
|
||||
)
|
||||
),
|
||||
el("td", { class: "muted" }, location.last_seen_at || "never")
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
: null,
|
||||
selected && selected.state !== "online" ? mountInstruction(selected) : null,
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
name,
|
||||
root,
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
"data-testid": "register-location",
|
||||
onclick: () =>
|
||||
run(() => api.registerArchiveLocation({ name: name.value, root: root.value })),
|
||||
},
|
||||
"Register destination"
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// The one thing the app cannot do for the user: name the medium to connect.
|
||||
function mountInstruction(location) {
|
||||
const detail =
|
||||
location.state === "wrong_volume"
|
||||
? `A different medium is mounted at ${location.root}.`
|
||||
: `Nothing is mounted at ${location.root}.`;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "confirm", role: "status", "data-testid": "mount-instruction" },
|
||||
`${detail} Connect “${location.name}” (medium ${location.media_id}) and mount it there, ` +
|
||||
"then reload this view. Archived photos stay listed and searchable meanwhile."
|
||||
);
|
||||
}
|
||||
|
||||
// ── preview ──────────────────────────────────────────────────────────────────
|
||||
function previewSection(preflight, location) {
|
||||
const totals = preflight.totals;
|
||||
const capacity = preflight.capacity;
|
||||
const rows = preflight.albums.map((album) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "archive-album-row", "data-album": album.album },
|
||||
el("td", { "data-testid": "album-name" }, album.album),
|
||||
el("td", { class: "path", "data-testid": "album-folder" }, album.folder),
|
||||
el("td", { class: "path", "data-testid": "album-destination" }, album.destination),
|
||||
el(
|
||||
"td",
|
||||
{ "data-testid": "album-method" },
|
||||
album.transfer_method === "move" ? "move (same filesystem)" : "copy · verify · remove"
|
||||
),
|
||||
el("td", { "data-testid": "album-assets" }, String(album.asset_count)),
|
||||
el("td", { "data-testid": "album-reclaim" }, bytes(album.reclaimable_bytes)),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("span", { class: `badge ${album.state}`, "data-testid": "album-state" }, album.state),
|
||||
...album.blockers.map((blocker) =>
|
||||
el(
|
||||
"div",
|
||||
{ class: "blocker", "data-testid": "album-blocker", "data-code": blocker.code },
|
||||
blocker.message
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "archive-preview" },
|
||||
el("h2", {}, "Preview"),
|
||||
el(
|
||||
"div",
|
||||
{ class: "decision-bar" },
|
||||
el(
|
||||
"span",
|
||||
{ class: "badge", "data-testid": "destination-identity" },
|
||||
`${location.name} · ${location.media_id}`
|
||||
),
|
||||
el("span", { class: "badge", "data-testid": "total-albums" }, `${totals.albums} album(s)`),
|
||||
el("span", { class: "badge", "data-testid": "total-assets" }, `${totals.assets} photo(s)`),
|
||||
el(
|
||||
"span",
|
||||
{ class: "badge", "data-testid": "total-reclaim" },
|
||||
`${bytes(totals.bytes)} reclaimable`
|
||||
),
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${capacity.sufficient ? "complete" : "blocked"}`,
|
||||
"data-testid": "capacity",
|
||||
},
|
||||
`free ${bytes(capacity.free_bytes)} · reserve ${bytes(capacity.reserve_bytes)}`
|
||||
)
|
||||
),
|
||||
blockerList(preflight.blockers, "preflight-blockers", "preflight-blocker", "This scope cannot be archived yet"),
|
||||
rows.length
|
||||
? el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "archive-albums" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Album", "Folder", "Destination", "Transfer", "Photos", "Reclaims", "State"].map(
|
||||
(label) => el("th", { scope: "col" }, label)
|
||||
)
|
||||
)
|
||||
),
|
||||
el("tbody", {}, ...rows)
|
||||
)
|
||||
: el(
|
||||
"p",
|
||||
{ class: "muted", "data-testid": "no-albums" },
|
||||
"No album has a verified upload whose bytes are still unchanged, so nothing may be archived."
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function blockerList(blockers, containerId, itemId, title) {
|
||||
if (!blockers || !blockers.length) return null;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": containerId },
|
||||
el("strong", {}, title),
|
||||
el(
|
||||
"ul",
|
||||
{},
|
||||
...blockers.map((blocker) =>
|
||||
el("li", { "data-testid": itemId, "data-code": blocker.code }, `${blocker.code}: ${blocker.message}`)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── confirmation ─────────────────────────────────────────────────────────────
|
||||
function confirmBlock(preflight, location) {
|
||||
const ready = preflight.state === "ready";
|
||||
const totals = preflight.totals;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "confirm" },
|
||||
el("h2", {}, "Confirm"),
|
||||
el(
|
||||
"p",
|
||||
{ class: "muted", "data-testid": "confirm-token" },
|
||||
`Preflight ${preflight.token.slice(0, 20)}… · ${location.name}`
|
||||
),
|
||||
el(
|
||||
"p",
|
||||
{ "data-testid": "archive-note" },
|
||||
"Archiving removes each original from the library — but only after its copy on " +
|
||||
"the medium has been written, hashed, and recorded in the manifest. The photos " +
|
||||
"stay searchable and deduplicable while the medium is away, and can be restored " +
|
||||
"from this view."
|
||||
),
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "start-archive",
|
||||
disabled: ready ? false : "disabled",
|
||||
title: ready ? false : "resolve the blockers above first",
|
||||
onclick: () =>
|
||||
run(async () => {
|
||||
const plan = await api.createArchivePlan({
|
||||
location_id: location.id,
|
||||
token: preflight.token,
|
||||
});
|
||||
const started = await api.applyArchivePlan(plan.id);
|
||||
watch(started.job.id, plan.id, "archive");
|
||||
return { archiving: plan.asset_count };
|
||||
}),
|
||||
},
|
||||
`Archive ${totals.ready_albums} album(s) · reclaim ${bytes(totals.bytes)}`
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── plans and progress ───────────────────────────────────────────────────────
|
||||
function planList(runs, selectedId) {
|
||||
if (!runs.length) {
|
||||
return el("p", { class: "muted", "data-testid": "no-plans" }, "Nothing has been archived yet.");
|
||||
}
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "archive-plans" },
|
||||
el("h2", {}, "Runs"),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "plans" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Created", "Direction", "State", "Photos", "Bytes"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...runs.map((plan) =>
|
||||
el(
|
||||
"tr",
|
||||
{
|
||||
"data-testid": "plan-row",
|
||||
"data-plan": plan.id,
|
||||
"data-direction": plan.direction,
|
||||
"aria-current": plan.id === selectedId ? "true" : false,
|
||||
},
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("a", { class: "link", href: `#/archive?plan=${encodeURIComponent(plan.id)}` }, plan.created_at || plan.id)
|
||||
),
|
||||
el("td", { "data-testid": "plan-direction" }, plan.direction),
|
||||
el("td", {}, el("span", { class: `badge ${plan.state}`, "data-testid": "plan-state" }, plan.state)),
|
||||
el("td", {}, String(plan.asset_count)),
|
||||
el("td", {}, bytes(plan.byte_size))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function planDetail(plan) {
|
||||
const operations = plan.operations || [];
|
||||
const counts = {};
|
||||
for (const operation of operations) {
|
||||
counts[operation.journal_state] = (counts[operation.journal_state] || 0) + 1;
|
||||
}
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "plan-detail", "data-plan": plan.id },
|
||||
el("h2", {}, `${plan.direction === "restore" ? "Restore" : "Archive"} run ${plan.created_at || plan.id}`),
|
||||
// Transfer, verification, and removal are separate answers to separate
|
||||
// questions: what has moved, what is proven, and what is already gone.
|
||||
el(
|
||||
"div",
|
||||
{ class: "decision-bar", "data-testid": "plan-progress" },
|
||||
el("span", { class: `badge ${plan.state}`, "data-testid": "detail-state" }, plan.state),
|
||||
...PHASES.map(([key, label, title]) =>
|
||||
el(
|
||||
"span",
|
||||
{ class: `badge ${key}`, "data-testid": `count-${label}`, title },
|
||||
`${label}: ${counts[key] ?? 0}`
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "operations" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Photo", "Destination", "Phase", "Attempts", "Problem"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...operations.map((operation) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "operation-row", "data-asset-id": operation.asset_id },
|
||||
el("td", { class: "path", "data-testid": "operation-source" }, operation.source_path),
|
||||
el("td", { class: "path", "data-testid": "operation-destination" }, operation.destination_path),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{ class: `badge ${operation.journal_state}`, "data-testid": "operation-phase" },
|
||||
phaseLabel(operation.journal_state)
|
||||
)
|
||||
),
|
||||
el("td", {}, String(operation.attempt_count)),
|
||||
el(
|
||||
"td",
|
||||
{ class: "muted", "data-testid": "operation-error", "data-code": operation.error_code || "" },
|
||||
operation.error_code ? `${operation.error_code}: ${operation.error_message || ""}` : "—"
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function phaseLabel(state) {
|
||||
const found = PHASES.find(([key]) => key === state);
|
||||
return found ? found[1] : state;
|
||||
}
|
||||
|
||||
// ── recovery ─────────────────────────────────────────────────────────────────
|
||||
// What an interrupted run left behind, straight from the journal plus the files on
|
||||
// disk. Only the operations the server itself classified as resolvable get an
|
||||
// action; ambiguous ones are shown with their evidence and no button.
|
||||
function recoverySection(recovery) {
|
||||
const operations = recovery ? recovery.operations : [];
|
||||
if (!operations.length) return null;
|
||||
const manual = recovery.manual || [];
|
||||
const resolvable = operations.length - manual.length;
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "archive-recovery" },
|
||||
el("h2", {}, "Interrupted work"),
|
||||
el(
|
||||
"p",
|
||||
{ "data-testid": "recovery-summary" },
|
||||
`${operations.length} operation(s) did not finish · ${resolvable} resolvable · ` +
|
||||
`${manual.length} need a decision`
|
||||
),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "recovery-operations" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Photo", "Phase", "Verdict", "Evidence"].map((l) => el("th", { scope: "col" }, l))
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...operations.map((verdict) =>
|
||||
el(
|
||||
"tr",
|
||||
{
|
||||
"data-testid": "recovery-row",
|
||||
"data-classification": verdict.classification,
|
||||
"data-direction": verdict.direction,
|
||||
},
|
||||
el("td", { class: "path", "data-testid": "recovery-source" }, verdict.source_path),
|
||||
el("td", {}, el("span", { class: "badge" }, phaseLabel(verdict.journal_state))),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${verdict.classification === "manual" ? "blocked" : "attention"}`,
|
||||
"data-testid": "recovery-verdict",
|
||||
},
|
||||
verdict.classification
|
||||
)
|
||||
),
|
||||
el(
|
||||
"td",
|
||||
{ class: "muted", "data-testid": "recovery-reason" },
|
||||
`${verdict.reason} (source ${verdict.source_exists ? "present" : "absent"}, ` +
|
||||
`archive copy ${verdict.destination_matches ? "verified" : verdict.destination_exists ? "different bytes" : "absent"})`
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
),
|
||||
manual.length
|
||||
? el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "recovery-manual" },
|
||||
`${manual.length} operation(s) cannot be resolved from the evidence. Nothing will be ` +
|
||||
"removed or retried for them here: inspect the medium and the library, then decide."
|
||||
)
|
||||
: null,
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
resolvable
|
||||
? el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "resolve-recovery",
|
||||
onclick: () => run(() => api.resolveArchiveRecovery()),
|
||||
},
|
||||
`Finish ${resolvable} recoverable operation(s)`
|
||||
)
|
||||
: el(
|
||||
"span",
|
||||
{ class: "muted", "data-testid": "no-safe-recovery" },
|
||||
"No operation can be finished safely from here."
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── archived assets ──────────────────────────────────────────────────────────
|
||||
// Browsing what is already archived, including while the medium is away: the
|
||||
// retained protected preview and the recorded hashes are the evidence, so the row
|
||||
// stays complete and honest instead of turning into a missing file.
|
||||
function archivedSection(assets, locations) {
|
||||
const archived = assets.filter((asset) => asset.availability_state !== "active");
|
||||
if (!archived.length) return null;
|
||||
const names = Object.fromEntries(locations.map((location) => [location.id, location.name]));
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "archived-assets" },
|
||||
el("h2", {}, "Archived photos"),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "archived" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Preview", "Archived as", "Medium", "Availability", "Size"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...archived.map((asset) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "archived-row", "data-asset-id": asset.id },
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("img", {
|
||||
"data-testid": "archived-preview",
|
||||
width: 96,
|
||||
loading: "lazy",
|
||||
src: api.thumbnailUrl(asset.id, 256),
|
||||
alt: `Preview of ${asset.archive_path || asset.id}`,
|
||||
})
|
||||
),
|
||||
el("td", { class: "path", "data-testid": "archived-path" }, asset.archive_path || "—"),
|
||||
el(
|
||||
"td",
|
||||
{ "data-testid": "archived-medium" },
|
||||
names[asset.archive_location_id] || asset.archive_location_id || "—"
|
||||
),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{
|
||||
class: `badge ${asset.availability_state === "archived_online" ? "complete" : "attention"}`,
|
||||
"data-testid": "archived-availability",
|
||||
},
|
||||
AVAILABILITY_LABEL[asset.availability_state] || asset.availability_state
|
||||
)
|
||||
),
|
||||
el("td", {}, bytes(asset.byte_size))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── restore ──────────────────────────────────────────────────────────────────
|
||||
function restoreSection(restore, location) {
|
||||
const ready = restore.state === "ready";
|
||||
const items = restore.items || [];
|
||||
if (!items.length && !restore.blockers.length) return null;
|
||||
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "restore" },
|
||||
el("h2", {}, "Restore"),
|
||||
el(
|
||||
"p",
|
||||
{ "data-testid": "restore-note" },
|
||||
"Restoring copies the archived bytes back into the library and leaves the archive " +
|
||||
"copy where it is. A name that is already taken is never overwritten: the photo " +
|
||||
"comes back beside it under a visibly different name."
|
||||
),
|
||||
blockerList(restore.blockers, "restore-blockers", "restore-blocker", "This restore cannot run yet"),
|
||||
items.length
|
||||
? el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "restore-items" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Archived as", "Comes back as", "Size", "State"].map((l) =>
|
||||
el("th", { scope: "col" }, l)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...items.map((item) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "restore-row", "data-asset-id": item.asset_id },
|
||||
el("td", { class: "path", "data-testid": "restore-source" }, item.archive_path),
|
||||
el("td", { class: "path", "data-testid": "restore-destination" }, item.destination_path || "—"),
|
||||
el("td", {}, bytes(item.byte_size)),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
item.blockers.length
|
||||
? el(
|
||||
"span",
|
||||
{
|
||||
class: "badge blocked",
|
||||
"data-testid": "restore-item-blocker",
|
||||
"data-code": item.blockers[0].code,
|
||||
},
|
||||
item.blockers[0].code
|
||||
)
|
||||
: el("span", { class: "badge ready", "data-testid": "restore-item-state" }, "ready")
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
: null,
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "start-restore",
|
||||
disabled: ready ? false : "disabled",
|
||||
title: ready ? false : "the medium and every archived copy must check out first",
|
||||
onclick: () =>
|
||||
run(async () => {
|
||||
const plan = await api.createRestorePlan({
|
||||
location_id: location.id,
|
||||
token: restore.token,
|
||||
});
|
||||
const started = await api.applyRestorePlan(plan.id);
|
||||
watch(started.job.id, plan.id, "restore");
|
||||
return { restoring: plan.asset_count };
|
||||
}),
|
||||
},
|
||||
`Restore ${items.length} photo(s) from ${location.name}`
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── running commands ─────────────────────────────────────────────────────────
|
||||
async function run(action) {
|
||||
try {
|
||||
outcome = { kind: "ok", result: await action() };
|
||||
} catch (error) {
|
||||
outcome = error.status === 409 ? { kind: "conflict", error } : { kind: "error", error };
|
||||
}
|
||||
render();
|
||||
}
|
||||
|
||||
// Live job activity. The plan panel is refreshed on its own tick because the
|
||||
// journal advances per file, not per job event; a full re-render would re-run
|
||||
// preflight (which re-hashes the library), so that happens once when the job ends.
|
||||
const REFRESH_MS = 500;
|
||||
|
||||
function watch(jobId, planId, kind) {
|
||||
activity = [`Started ${kind} job ${jobId}`];
|
||||
const tick = setInterval(() => refreshPlan(planId, kind), REFRESH_MS);
|
||||
subscribeJob(jobId, {
|
||||
onEvent: (event) => {
|
||||
activity.push(`${event.type}${event.message ? ": " + event.message : ""}`);
|
||||
const log = document.querySelector('[data-testid="archive-activity"]');
|
||||
if (log) log.textContent = activity.join("\n");
|
||||
},
|
||||
onDone: () => {
|
||||
clearInterval(tick);
|
||||
activity.push("done");
|
||||
render();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function refreshPlan(planId, kind) {
|
||||
const node = document.querySelector(`[data-testid="plan-detail"][data-plan="${planId}"]`);
|
||||
if (!node) return; // the user navigated away from the running plan
|
||||
try {
|
||||
const plan = await (kind === "restore" ? api.getRestorePlan(planId) : api.getArchivePlan(planId));
|
||||
node.replaceWith(planDetail(plan));
|
||||
} catch (_) {
|
||||
// Transient; the next tick tries again and the job's end re-renders anyway.
|
||||
}
|
||||
}
|
||||
|
||||
function activityLog() {
|
||||
return el(
|
||||
"pre",
|
||||
{
|
||||
class: "activity-log",
|
||||
role: "status",
|
||||
"aria-live": "polite",
|
||||
"data-testid": "archive-activity",
|
||||
},
|
||||
activity.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
function outcomeBanner() {
|
||||
if (!outcome) return null;
|
||||
if (outcome.kind === "conflict") {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "conflict" },
|
||||
`The server refused this: ${outcome.error.message}. Nothing was moved or removed; ` +
|
||||
"the state below is the server's current one — review it and decide again."
|
||||
);
|
||||
}
|
||||
if (outcome.kind === "error") {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "archive-error" },
|
||||
`Failed: ${outcome.error.message}`
|
||||
);
|
||||
}
|
||||
const result = outcome.result || {};
|
||||
const message =
|
||||
result.archiving !== undefined
|
||||
? `Archiving ${result.archiving} photo(s). Originals are removed only after their copies verify.`
|
||||
: result.restoring !== undefined
|
||||
? `Restoring ${result.restoring} photo(s) into the library.`
|
||||
: "Done — the state below is the server's.";
|
||||
return el("div", { class: "alert", role: "status", "data-testid": "archive-result" }, message);
|
||||
}
|
||||
|
||||
// ── formatting ───────────────────────────────────────────────────────────────
|
||||
function bytes(value) {
|
||||
if (value == null) return "unknown";
|
||||
const units = ["B", "kB", "MB", "GB", "TB"];
|
||||
let size = value;
|
||||
let unit = 0;
|
||||
while (size >= 1000 && unit < units.length - 1) {
|
||||
size /= 1000;
|
||||
unit += 1;
|
||||
}
|
||||
return `${unit === 0 ? size : size.toFixed(1)} ${units[unit]}`;
|
||||
}
|
||||
704
frontend/js/uploads.js
Normal file
704
frontend/js/uploads.js
Normal file
@@ -0,0 +1,704 @@
|
||||
// Upload view (US05-05): preflight a scope, confirm exactly what will be sent,
|
||||
// watch the batch run, and resolve whatever the uploader left uncertain.
|
||||
//
|
||||
// Two rules shape this file. First, nothing here decides what is safe: blockers,
|
||||
// the preflight token, and the retry policy all come from the server, and an action
|
||||
// the server would refuse is not offered at all. Second, the API key never reaches
|
||||
// the browser — preflight reports only that one is configured, and every command
|
||||
// preview arrives redacted — so nothing in this view may reconstruct, store, or
|
||||
// route a secret.
|
||||
import { api } from "./api.js";
|
||||
import { el, errorBanner, setActiveNav } from "./dom.js";
|
||||
import { subscribeJob } from "./events.js";
|
||||
import { navigate } from "./router.js";
|
||||
|
||||
// Result of the last command issued from this tab, and the activity of the last
|
||||
// upload job. Deliberately not persisted: after a reload the page must show what
|
||||
// the server says happened, not what this page remembers.
|
||||
let outcome = null;
|
||||
let activity = [];
|
||||
|
||||
let render = () => {};
|
||||
export function setUploadsRender(fn) {
|
||||
render = fn;
|
||||
}
|
||||
|
||||
// Report outcomes (services/upload_reports.py) in the words the operator uses.
|
||||
const OUTCOME_LABEL = [
|
||||
["uploaded", "new"],
|
||||
["upgraded", "upgraded"],
|
||||
["duplicate", "duplicate"],
|
||||
["skipped", "skipped"],
|
||||
["failed", "failed"],
|
||||
["unknown", "uncertain"],
|
||||
];
|
||||
|
||||
export async function renderUploads(root, params = {}) {
|
||||
setActiveNav("uploads");
|
||||
const albums = parseAlbums(params.albums);
|
||||
const allowPartial = params.partial === "1";
|
||||
|
||||
let preflight, listed;
|
||||
try {
|
||||
[preflight, listed] = await Promise.all([
|
||||
api.uploadPreflight({ albums, allow_partial: allowPartial }),
|
||||
api.listUploadBatches(),
|
||||
]);
|
||||
} catch (error) {
|
||||
root.replaceChildren(errorBanner(`Failed to load uploads: ${error.message}`));
|
||||
return;
|
||||
}
|
||||
|
||||
const batches = listed.batches;
|
||||
const selectedId = params.batch || (batches.length ? batches[batches.length - 1].id : null);
|
||||
let batch = null;
|
||||
let history = [];
|
||||
if (selectedId) {
|
||||
try {
|
||||
const [detail, verifications] = await Promise.all([
|
||||
api.getUploadBatch(selectedId),
|
||||
api.uploadVerifications(selectedId),
|
||||
]);
|
||||
batch = detail;
|
||||
history = verifications.verifications;
|
||||
} catch (error) {
|
||||
root.replaceChildren(errorBanner(`Failed to load upload batch: ${error.message}`));
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
root.replaceChildren(
|
||||
...[
|
||||
el("h1", {}, "Upload"),
|
||||
configurationCard(preflight),
|
||||
preflightBlockers(preflight),
|
||||
scopeSection(preflight, params, albums, allowPartial),
|
||||
confirmBlock(preflight, params, allowPartial),
|
||||
outcomeBanner(),
|
||||
activityLog(),
|
||||
batchList(batches, selectedId),
|
||||
batch ? batchDetail(batch, history) : null,
|
||||
].filter(Boolean)
|
||||
);
|
||||
}
|
||||
|
||||
// Album names can contain commas, so each one is escaped before the URL joins them.
|
||||
function parseAlbums(value) {
|
||||
if (!value) return null;
|
||||
const names = value.split(",").filter(Boolean).map(decodeURIComponent);
|
||||
return names.length ? names : null;
|
||||
}
|
||||
|
||||
function albumsParam(names) {
|
||||
return names.map(encodeURIComponent).join(",");
|
||||
}
|
||||
|
||||
// ── configuration ────────────────────────────────────────────────────────────
|
||||
// What the upload is aimed at, in the only form the browser is ever given: the
|
||||
// server URL, whether a key exists, and the uploader's version.
|
||||
function configurationCard(preflight) {
|
||||
const credentials = preflight.credentials;
|
||||
const uploader = preflight.uploader;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "upload-config" },
|
||||
el("h2", {}, "Configuration"),
|
||||
el(
|
||||
"dl",
|
||||
{},
|
||||
el("dt", {}, "Immich server"),
|
||||
el("dd", { "data-testid": "config-server" }, credentials.server_url || "not configured"),
|
||||
el("dt", {}, "API key"),
|
||||
// Presence, never the value — and never a length or prefix either.
|
||||
el(
|
||||
"dd",
|
||||
{ "data-testid": "config-key" },
|
||||
credentials.api_key_configured ? "configured (never shown)" : "missing"
|
||||
),
|
||||
el("dt", {}, "Reachable"),
|
||||
el(
|
||||
"dd",
|
||||
{ "data-testid": "config-reachable" },
|
||||
preflight.server.reachable ? "yes" : `no — ${preflight.server.detail || "unknown"}`
|
||||
),
|
||||
el("dt", {}, "Uploader"),
|
||||
el(
|
||||
"dd",
|
||||
{ "data-testid": "config-uploader" },
|
||||
uploader.installed ? uploader.version || "installed" : `${uploader.binary} is not installed`
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function preflightBlockers(preflight) {
|
||||
if (!preflight.blockers.length) return null;
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "preflight-blockers" },
|
||||
el("strong", {}, "This scope cannot be uploaded yet"),
|
||||
el(
|
||||
"ul",
|
||||
{},
|
||||
...preflight.blockers.map((blocker) =>
|
||||
el(
|
||||
"li",
|
||||
{ "data-testid": "preflight-blocker", "data-code": blocker.code },
|
||||
`${blocker.code}: ${blocker.message}`
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── scope ────────────────────────────────────────────────────────────────────
|
||||
function scopeSection(preflight, params, albums, allowPartial) {
|
||||
const selected = new Set(albums || preflight.albums.map((album) => album.album));
|
||||
|
||||
function toggle(name, checked) {
|
||||
const next = new Set(selected);
|
||||
if (checked) next.add(name);
|
||||
else next.delete(name);
|
||||
// An empty selection means "everything" again, which is also what an absent
|
||||
// parameter means — there is no way to preflight nothing.
|
||||
navigate("/uploads", { ...params, albums: albumsParam([...next]), batch: params.batch });
|
||||
}
|
||||
|
||||
const rows = preflight.albums.map((album) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "album-row", "data-album": album.album },
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("input", {
|
||||
type: "checkbox",
|
||||
"data-testid": "album-selected",
|
||||
"aria-label": `Include ${album.album}`,
|
||||
checked: selected.has(album.album) ? "checked" : false,
|
||||
onchange: (event) => toggle(album.album, event.target.checked),
|
||||
})
|
||||
),
|
||||
el("td", { "data-testid": "album-name" }, album.album),
|
||||
// What Immich will call it, which is not always what the folder is called here.
|
||||
el("td", { "data-testid": "album-immich-name" }, album.album_name),
|
||||
el("td", { class: "path", "data-testid": "album-folder" }, album.folder),
|
||||
el("td", { "data-testid": "album-eligible" }, String(album.eligible_count)),
|
||||
el("td", { "data-testid": "album-blocked" }, String(album.blocked_count)),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("span", { class: `badge ${album.state}`, "data-testid": "album-state" }, album.state),
|
||||
...album.blockers.map((blocker) =>
|
||||
el(
|
||||
"div",
|
||||
{ class: "blocker", "data-testid": "album-blocker", "data-code": blocker.code },
|
||||
blocker.message
|
||||
)
|
||||
)
|
||||
),
|
||||
// The exact invocation, as the server built it. Shown so the upload holds no
|
||||
// surprises; the key is masked at the source, not here.
|
||||
el("td", { class: "path", "data-testid": "album-command" }, album.command_preview.join(" "))
|
||||
)
|
||||
);
|
||||
|
||||
const totals = preflight.totals;
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "upload-scope" },
|
||||
el("h2", {}, "Scope"),
|
||||
el(
|
||||
"div",
|
||||
{ class: "decision-bar" },
|
||||
el("span", { class: "badge", "data-testid": "total-albums" }, `${totals.albums} album(s)`),
|
||||
el("span", { class: "badge", "data-testid": "total-eligible" }, `${totals.eligible} ready`),
|
||||
totals.blocked
|
||||
? el(
|
||||
"span",
|
||||
{ class: "badge attention", "data-testid": "total-blocked" },
|
||||
`${totals.blocked} blocked`
|
||||
)
|
||||
: null,
|
||||
el(
|
||||
"label",
|
||||
{},
|
||||
el("input", {
|
||||
type: "checkbox",
|
||||
"data-testid": "allow-partial",
|
||||
checked: allowPartial ? "checked" : false,
|
||||
onchange: (event) =>
|
||||
navigate("/uploads", { ...params, partial: event.target.checked ? "1" : "" }),
|
||||
}),
|
||||
" Upload ready photos and leave the blocked ones behind"
|
||||
)
|
||||
),
|
||||
rows.length
|
||||
? el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "albums" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["", "Album", "Immich album", "Folder", "Ready", "Blocked", "State", "Command"].map(
|
||||
(label) => el("th", { scope: "col" }, label)
|
||||
)
|
||||
)
|
||||
),
|
||||
el("tbody", {}, ...rows)
|
||||
)
|
||||
: el("p", { class: "muted", "data-testid": "no-albums" }, "No album is ready to upload.")
|
||||
);
|
||||
}
|
||||
|
||||
// ── confirmation ─────────────────────────────────────────────────────────────
|
||||
function confirmBlock(preflight, params, allowPartial) {
|
||||
const ready = preflight.state === "ready";
|
||||
const totals = preflight.totals;
|
||||
const albums = preflight.albums.map((album) => album.album);
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "confirm" },
|
||||
el("h2", {}, "Confirm"),
|
||||
// The token is shown, not merely sent: a confirmation the user cannot see is a
|
||||
// confirmation they cannot check against the preview above.
|
||||
el(
|
||||
"p",
|
||||
{ class: "muted", "data-testid": "confirm-token" },
|
||||
`Preflight ${preflight.token.slice(0, 20)}… · ${allowPartial ? "partial" : "complete"} scope`
|
||||
),
|
||||
el(
|
||||
"p",
|
||||
{ "data-testid": "upload-note" },
|
||||
"Uploading is not reversible from here: Immich decides what to do with each " +
|
||||
"file, and the app can only record what it reports. One album is sent at a " +
|
||||
"time and a running album can be stopped."
|
||||
),
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "start-upload",
|
||||
disabled: ready ? false : "disabled",
|
||||
title: ready ? false : "resolve the blockers above first",
|
||||
onclick: () =>
|
||||
run(async () => {
|
||||
const created = await api.createUploadBatches({
|
||||
albums,
|
||||
token: preflight.token,
|
||||
allow_partial: allowPartial,
|
||||
});
|
||||
// One lane: the first batch starts now, the rest wait with their own
|
||||
// start buttons rather than queueing behind a lock that would reject
|
||||
// them.
|
||||
const first = created.batches.find((batch) => !batch.retry_blockers.length);
|
||||
if (!first) return { batches: created.batches.length, started: null };
|
||||
const started = await api.startUploadBatch(first.id);
|
||||
watch(started.job.id, first.id);
|
||||
return { batches: created.batches.length, started: first.album };
|
||||
}),
|
||||
},
|
||||
`Upload ${totals.albums} album(s) · ${totals.eligible} photo(s)`
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── batches ──────────────────────────────────────────────────────────────────
|
||||
function batchList(batches, selectedId) {
|
||||
if (!batches.length) {
|
||||
return el("p", { class: "muted", "data-testid": "no-batches" }, "No upload has been started yet.");
|
||||
}
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "upload-batches" },
|
||||
el("h2", {}, "Batches"),
|
||||
el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "batches" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Album", "State", "Evidence", "Attempts", "Photos"].map((label) =>
|
||||
el("th", { scope: "col" }, label)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...batches.map((batch) =>
|
||||
el(
|
||||
"tr",
|
||||
{
|
||||
"data-testid": "batch-row",
|
||||
"data-album": batch.album,
|
||||
"aria-current": batch.id === selectedId ? "true" : false,
|
||||
},
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("a", { class: "link", href: `#/uploads?batch=${encodeURIComponent(batch.id)}` }, batch.album)
|
||||
),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el("span", { class: `badge ${batch.state}`, "data-testid": "batch-state" }, batch.state)
|
||||
),
|
||||
el("td", { "data-testid": "batch-outcome-state" }, batch.outcome_state || "not parsed"),
|
||||
el("td", {}, String(batch.attempt_count)),
|
||||
el("td", {}, String(batch.asset_count))
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function batchDetail(batch, history) {
|
||||
// Derived from the items, not from the batch's parsed-report summary: verifying
|
||||
// or resolving an item changes what is true without re-parsing a report, and the
|
||||
// progress line must show the current answer rather than the uploader's old one.
|
||||
const counts = {};
|
||||
for (const item of batch.items) {
|
||||
const key = item.outcome || "unknown";
|
||||
counts[key] = (counts[key] || 0) + 1;
|
||||
}
|
||||
const blockers = batch.retry_blockers;
|
||||
const uncertain = batch.outcome_state === "requires_verification" || counts.unknown > 0;
|
||||
|
||||
const nodes = [
|
||||
el("h2", {}, `${batch.album} — attempt ${batch.attempt_count}`),
|
||||
el(
|
||||
"div",
|
||||
{ class: "decision-bar", "data-testid": "batch-progress" },
|
||||
el("span", { class: `badge ${batch.state}`, "data-testid": "detail-state" }, batch.state),
|
||||
...OUTCOME_LABEL.map(([key, label]) =>
|
||||
el(
|
||||
"span",
|
||||
{ class: `badge ${key}`, "data-testid": `count-${label}` },
|
||||
`${label}: ${counts[key] ?? 0}`
|
||||
)
|
||||
)
|
||||
),
|
||||
];
|
||||
|
||||
if (batch.stale_bytes) {
|
||||
nodes.push(
|
||||
el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "stale-bytes" },
|
||||
"Files in this batch changed after they were uploaded. Immich still holds the " +
|
||||
"bytes that were sent; re-approve the album through a fresh preflight rather " +
|
||||
"than uploading the new bytes over it."
|
||||
)
|
||||
);
|
||||
}
|
||||
if (batch.error_code) {
|
||||
nodes.push(
|
||||
el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "batch-error" },
|
||||
`${batch.error_code}: ${batch.error_message || ""}`
|
||||
)
|
||||
);
|
||||
}
|
||||
if (uncertain) {
|
||||
nodes.push(
|
||||
el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "uncertain" },
|
||||
el("strong", {}, "This upload's outcome is not fully known"),
|
||||
el(
|
||||
"p",
|
||||
{},
|
||||
"The uploader's report does not account for every file. Retrying could create " +
|
||||
"a second copy of something Immich already accepted, so verify it first: the " +
|
||||
"check asks Immich whether it holds the exact bytes that were sent."
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
nodes.push(
|
||||
el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
// A start button exists only when the server would accept one. An uncertain
|
||||
// outcome and changed bytes therefore offer verification, never a retry.
|
||||
blockers.length
|
||||
? el(
|
||||
"div",
|
||||
{ class: "blocker", "data-testid": "retry-blocked" },
|
||||
blockers.map((blocker) => `${blocker.code}: ${blocker.message}`).join("; ")
|
||||
)
|
||||
: el(
|
||||
"button",
|
||||
{
|
||||
class: "primary",
|
||||
"data-testid": "start-batch",
|
||||
onclick: () =>
|
||||
run(async () => {
|
||||
const started = await api.startUploadBatch(batch.id);
|
||||
watch(started.job.id, batch.id);
|
||||
return { started: batch.album };
|
||||
}),
|
||||
},
|
||||
batch.attempt_count ? "Run this album again" : "Upload this album"
|
||||
),
|
||||
["planned", "running"].includes(batch.state)
|
||||
? el(
|
||||
"button",
|
||||
{
|
||||
"data-testid": "cancel-batch",
|
||||
onclick: () => run(() => api.cancelUploadBatch(batch.id)),
|
||||
},
|
||||
batch.state === "running" ? "Stop after the current file" : "Cancel this album"
|
||||
)
|
||||
: null,
|
||||
// Offered for anything that has run, not only for uncertain outcomes:
|
||||
// re-checking is read-only and idempotent, and it is how a file edited after
|
||||
// its upload is discovered.
|
||||
batch.attempt_count
|
||||
? el(
|
||||
"button",
|
||||
{
|
||||
"data-testid": "verify-batch",
|
||||
onclick: () => run(() => api.verifyUploadBatch(batch.id)),
|
||||
},
|
||||
"Verify against Immich"
|
||||
)
|
||||
: null
|
||||
),
|
||||
itemsTable(batch)
|
||||
);
|
||||
if (history.length) nodes.push(historyList(history));
|
||||
return el(
|
||||
"div",
|
||||
{ class: "card", "data-testid": "batch-detail", "data-batch": batch.id },
|
||||
...nodes
|
||||
);
|
||||
}
|
||||
|
||||
function itemsTable(batch) {
|
||||
if (!batch.items.length) {
|
||||
return el("p", { class: "muted", "data-testid": "no-items" }, "This batch has no photos.");
|
||||
}
|
||||
return el(
|
||||
"table",
|
||||
{ class: "grid", "data-testid": "items" },
|
||||
el(
|
||||
"thead",
|
||||
{},
|
||||
el(
|
||||
"tr",
|
||||
{},
|
||||
...["Photo", "Outcome", "Evidence", "Verification", "Bytes now", "Resolve"].map((label) =>
|
||||
el("th", { scope: "col" }, label)
|
||||
)
|
||||
)
|
||||
),
|
||||
el(
|
||||
"tbody",
|
||||
{},
|
||||
...batch.items.map((item) =>
|
||||
el(
|
||||
"tr",
|
||||
{ "data-testid": "item-row", "data-asset-id": item.asset_id },
|
||||
el("td", { class: "path", "data-testid": "item-path" }, item.path),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
el(
|
||||
"span",
|
||||
{ class: `badge ${item.outcome || ""}`, "data-testid": "item-outcome" },
|
||||
item.outcome === "uploaded" ? "new" : item.outcome || "pending"
|
||||
)
|
||||
),
|
||||
el("td", { class: "muted", "data-testid": "item-evidence" }, item.evidence || "—"),
|
||||
el("td", { "data-testid": "item-verification" }, item.verification || "—"),
|
||||
el(
|
||||
"td",
|
||||
{},
|
||||
item.changed_after_upload
|
||||
? el("span", { class: "badge attention", "data-testid": "item-changed" }, "changed")
|
||||
: el("span", { class: "muted" }, "unchanged")
|
||||
),
|
||||
el("td", {}, resolveForm(batch, item))
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// Manual resolution is evidence, not permission: the note and the author are
|
||||
// required by the server, so the form collects both and offers no default.
|
||||
function resolveForm(batch, item) {
|
||||
const unresolved = !item.outcome || item.outcome === "unknown" || item.verification === "inconclusive";
|
||||
if (!unresolved) return el("span", { class: "muted" }, "—");
|
||||
|
||||
const outcomeSelect = el(
|
||||
"select",
|
||||
{ "data-testid": "resolve-outcome", "aria-label": `Outcome for ${item.path}` },
|
||||
...OUTCOME_LABEL.map(([key, label]) => el("option", { value: key }, label))
|
||||
);
|
||||
const evidence = el("input", {
|
||||
type: "text",
|
||||
"data-testid": "resolve-evidence",
|
||||
"aria-label": `What you checked for ${item.path}`,
|
||||
placeholder: "What did you check?",
|
||||
});
|
||||
const actor = el("input", {
|
||||
type: "text",
|
||||
"data-testid": "resolve-actor",
|
||||
"aria-label": `Who checked ${item.path}`,
|
||||
placeholder: "Who are you?",
|
||||
});
|
||||
return el(
|
||||
"div",
|
||||
{ class: "toolbar" },
|
||||
outcomeSelect,
|
||||
evidence,
|
||||
actor,
|
||||
el(
|
||||
"button",
|
||||
{
|
||||
"data-testid": "resolve-item",
|
||||
onclick: () =>
|
||||
run(() =>
|
||||
api.resolveUploadItem(batch.id, {
|
||||
asset_id: item.asset_id,
|
||||
outcome: outcomeSelect.value,
|
||||
evidence: evidence.value,
|
||||
actor: actor.value,
|
||||
})
|
||||
),
|
||||
},
|
||||
"Record"
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function historyList(history) {
|
||||
return el(
|
||||
"div",
|
||||
{ "data-testid": "verification-history" },
|
||||
el("h3", {}, "Verification history"),
|
||||
el(
|
||||
"ul",
|
||||
{},
|
||||
...history.map((entry) =>
|
||||
el(
|
||||
"li",
|
||||
{ "data-testid": "history-entry", "data-source": entry.source },
|
||||
`${entry.created_at || ""} · ${entry.action} · ${entry.source} · ${entry.result} → ` +
|
||||
`${entry.outcome || "unresolved"} — ${entry.evidence}` +
|
||||
(entry.actor ? ` (${entry.actor})` : "")
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// ── running commands ─────────────────────────────────────────────────────────
|
||||
async function run(action) {
|
||||
try {
|
||||
outcome = { kind: "ok", result: await action() };
|
||||
} catch (error) {
|
||||
outcome = error.status === 409 ? { kind: "conflict", error } : { kind: "error", error };
|
||||
}
|
||||
render();
|
||||
}
|
||||
|
||||
// Live job activity. Events are appended to the log node and the running batch's
|
||||
// panel is refreshed on its own tick — the uploader reports per file, not per job
|
||||
// event, so waiting for the next event would leave the panel behind. Only that
|
||||
// panel is rebuilt: a full re-render would re-run preflight, which re-hashes the
|
||||
// library, so that happens once when the job ends.
|
||||
const REFRESH_MS = 1000;
|
||||
|
||||
function watch(jobId, batchId) {
|
||||
activity = [`Started upload job ${jobId}`];
|
||||
const tick = setInterval(() => refreshBatch(batchId), REFRESH_MS);
|
||||
subscribeJob(jobId, {
|
||||
onEvent: (event) => {
|
||||
activity.push(`${event.type}${event.message ? ": " + event.message : ""}`);
|
||||
const log = document.querySelector('[data-testid="upload-activity"]');
|
||||
if (log) log.textContent = activity.join("\n");
|
||||
},
|
||||
onDone: () => {
|
||||
clearInterval(tick);
|
||||
activity.push("done");
|
||||
render();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function refreshBatch(batchId) {
|
||||
const node = document.querySelector(`[data-testid="batch-detail"][data-batch="${batchId}"]`);
|
||||
if (!node) return; // the user navigated away from the running batch
|
||||
try {
|
||||
const [batch, verifications] = await Promise.all([
|
||||
api.getUploadBatch(batchId),
|
||||
api.uploadVerifications(batchId),
|
||||
]);
|
||||
node.replaceWith(batchDetail(batch, verifications.verifications));
|
||||
} catch (_) {
|
||||
// Transient: the next tick tries again, and the job's end re-renders anyway.
|
||||
}
|
||||
}
|
||||
|
||||
function activityLog() {
|
||||
return el(
|
||||
"pre",
|
||||
{
|
||||
class: "activity-log",
|
||||
role: "status",
|
||||
"aria-live": "polite",
|
||||
"data-testid": "upload-activity",
|
||||
},
|
||||
activity.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
function outcomeBanner() {
|
||||
if (!outcome) return null;
|
||||
if (outcome.kind === "conflict") {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "conflict" },
|
||||
`The server refused this: ${outcome.error.message}. Nothing was uploaded; the ` +
|
||||
"state below is the server's current one — review it and decide again."
|
||||
);
|
||||
}
|
||||
if (outcome.kind === "error") {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "alert", "data-testid": "upload-error" },
|
||||
`Failed: ${outcome.error.message}`
|
||||
);
|
||||
}
|
||||
const result = outcome.result;
|
||||
if (result && result.batches !== undefined) {
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "status", "data-testid": "upload-result" },
|
||||
`Approved ${result.batches} album batch(es).` +
|
||||
(result.started ? ` Uploading ${result.started} now.` : " Nothing could be started yet.")
|
||||
);
|
||||
}
|
||||
return el(
|
||||
"div",
|
||||
{ class: "alert", role: "status", "data-testid": "upload-result" },
|
||||
"Done — the state below is the server's."
|
||||
);
|
||||
}
|
||||
22
legacy_cli_archive/CHECKSUMS.sha256
Normal file
22
legacy_cli_archive/CHECKSUMS.sha256
Normal file
@@ -0,0 +1,22 @@
|
||||
d098bbde13d2ebc872ca781e244cc1e48b97cc551fedaca3251d6a5278a15234 src/compare_models.py
|
||||
7d0f68cb95fbd6603e4c558620098b46929ebdb6fd91a598bbf84f26eea35e6c src/nsfw_tag.py
|
||||
35329c53570e215cef15f59429c1a5251a0448b90522b0910ab9d68efe1bc307 src/nsfwtag/__init__.py
|
||||
7f52b55e4f3b77eda3657d7cd2272c7cca8422100e241671de495603c1126ee2 src/nsfwtag/__main__.py
|
||||
6f5a97114e0d87d272ce22d065dc31fa0dd72ff8590d11ad14cb1c1f486d339c src/nsfwtag/bench.py
|
||||
29498fac1d73ba2b7420ffe1ffe49a87f684ec299e2c3b9c6e14e63e95643ba4 src/nsfwtag/exif.py
|
||||
934e82c402813ebf503e5a20eb03d5103f84df13117bc4dd1fd95deaa01ab263 src/nsfwtag/README.md
|
||||
68125e6184c7e4d2a5b0153f2155675753ab0b1329d1dbc4933d4769492ec4be src/nsfwtag/review.html
|
||||
a597eab74803dd31452b5d70abf7d6d6320eb17a61596c891b009da15d624e98 src/nsfwtag/scoring.py
|
||||
674969a18e58ee511a2abf574a875db92f2f524769b1b614d0d3662ab879a709 src/nsfwtag/server.py
|
||||
24a7c8d029da6e97d46b110a9fe9dbb3900127f8ae3c142f634fdd91d6c45243 src/nsfwtag/webapp.py
|
||||
2c2ea558f1b9095c1150f078ea29c8e0f180cc045a9d1097b83f61bfb145295b src/photo_analyzer.py
|
||||
5fb7ce2f977a17da1f501c45d7985328318ae14298a3f347eab46a6e3f02aab3 src/test_dedup.py
|
||||
1673dc76cc60aba56a5b065d9bc9f5dda00342cf2a79d000a3f74686fb7cffec src/test_nsfw_skip.py
|
||||
55aa82f348e0e90be0163ba6aa278e5fc8e445996e5335774917e5ef59ad3563 src/webapp/__init__.py
|
||||
ba4342bd0175591a2121f063f14a78a602d5679bcfcf24aa14c8d4b8cd5da1a2 src/webapp/__main__.py
|
||||
a60f24035a989909467778f888d62854c1afc2702f47b8d3cca32bf3cf04d5cf src/webapp/analyzer.html
|
||||
018bcc2f770d716444b456db58d6b4a41800138a98598b3c27ba503ab22d939e src/webapp/page.py
|
||||
5f599b107b2b117ca118b6fbdec5e5786a9ab4eea424aa9af1bfea7bf87cebab src/webapp/query.py
|
||||
875cce697caa02717c85a2707ba935c46dd9ace18428fd4abb13105b7a8e24d9 src/webapp/README.md
|
||||
af7d0d72d245b4bbb1e0e30f9708239697543212d4171defd1bfe26dd24a4f05 src/webapp/runner.py
|
||||
cddf0555b06fe7e4daa14309baaaf91130f8506922cb0ffb6507d76cbe39add4 src/webapp/server.py
|
||||
80
legacy_cli_archive/README.md
Normal file
80
legacy_cli_archive/README.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# Legacy CLI archive (US07-01)
|
||||
|
||||
Frozen, read-only sources of the command-line tools this application was extracted
|
||||
from. They are **reference material and rollback evidence** — provenance for
|
||||
behavior that now lives in `photo_pipeline/`, and the only way to answer "what did
|
||||
the original actually do?" once the replacement has drifted.
|
||||
|
||||
> **Nothing here is production code.** No module under `photo_pipeline/` imports or
|
||||
> executes anything in this directory, and this directory is not on the application's
|
||||
> import path. `tests/unit/test_legacy_archive.py` enforces both, along with the
|
||||
> checksums and the redaction below.
|
||||
|
||||
## What is here
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `src/photo_analyzer.py` | the analysis CLI: discovery, hashing/dedup, vision analysis, EXIF writing, SQLite schema, album naming |
|
||||
| `src/nsfwtag/` | NSFW scoring, EXIF safety keywords, and the review server (`__init__` 1.1.0) |
|
||||
| `src/webapp/` | the stdlib review web app: FTS search, stats, subprocess runner, HTML shell (`__init__` 0.1.0) |
|
||||
| `src/nsfw_tag.py` | thin backwards-compatible entry point for `nsfwtag` |
|
||||
| `src/compare_models.py` | dev-only model comparison script |
|
||||
| `src/test_dedup.py`, `src/test_nsfw_skip.py` | the CLIs' own standalone self-checks (never pytest suites) |
|
||||
| `donor_ledger.yaml` | the donor ledger: every migrated behavior, its target, its tests, and every intentional delta |
|
||||
| `requirements-lock.txt` | the dependency versions the frozen sources were last verified against |
|
||||
| `photo_analyzer.env.sample` | the CLI's configuration surface, with every value replaced by a placeholder |
|
||||
| `CHECKSUMS.sha256` | SHA-256 of every archived source file |
|
||||
|
||||
`photo_analyzer.py` carries no `__version__`; its identity is its checksum, recorded
|
||||
in `CHECKSUMS.sha256` and taken at commit `9b7ee6b` (the merge of US06-06, the last
|
||||
commit before archival).
|
||||
|
||||
## Verifying the archive
|
||||
|
||||
```bash
|
||||
cd legacy_cli_archive && shasum -a 256 -c CHECKSUMS.sha256
|
||||
```
|
||||
|
||||
Any edit to an archived source must be accompanied by a regenerated checksum file
|
||||
and a note here explaining why a *frozen* archive changed — the normal answer being
|
||||
that it should not.
|
||||
|
||||
## Schema notes
|
||||
|
||||
`photo_analyzer.py` owned a path-keyed SQLite database (`SCHEMA`, near the top of the
|
||||
file):
|
||||
|
||||
- `photos(id, path UNIQUE, status, phash, file_sha1, dup_of, description, tags,
|
||||
people_count, setting, time_of_day, season, mood, location_hint, approx_year,
|
||||
raw_response, error_message, analyzed_at, exif_written_at)`;
|
||||
- `photos_fts` — an FTS5 external-content index over `path, description, tags, mood,
|
||||
location_hint`, kept in sync by insert/update/delete triggers;
|
||||
- late columns (`phash`, `file_sha1`, `dup_of`) were added by an in-code
|
||||
`_migrate_schema()` rather than a migration tool, and their indexes are created
|
||||
only after the `ALTER`.
|
||||
|
||||
The replacement keeps the same analysis fields but re-keys everything to a stable
|
||||
`assets.id` (Alembic migrations `0001`…), because a path is not an identity: the
|
||||
donor's `path UNIQUE` is exactly what broke on every move and rename.
|
||||
|
||||
`nsfwtag` kept its safety scores outside the database in `nsfw_scores.csv`
|
||||
(`path,nsfw_score`, four decimals, unreadable rows dropped). That file is no longer a
|
||||
source of truth; `photo_pipeline/services/legacy_import.py` imports it into
|
||||
`assets.safety_score` and reports exactly what matched, what did not, and why.
|
||||
|
||||
## Redaction
|
||||
|
||||
The archive contains no credentials. `photo_analyzer.env.sample` documents the
|
||||
configuration surface (`LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL`, and the tuning
|
||||
variables) with placeholder values only; the CLI itself never contained a key, it
|
||||
read one from `photo_analyzer.env` or the environment. No `.env`, database, log, CSV,
|
||||
or photo from the author's library is archived.
|
||||
|
||||
## Why these tools were retired
|
||||
|
||||
Each behavior's fate is recorded per row in `donor_ledger.yaml`: `reuse`, `extract`,
|
||||
`refactor`, or `replace`, with the target module, the characterization tests that
|
||||
pinned the donor's behavior, the parity tests the replacement passes, and — where the
|
||||
replacement deliberately does something else or nothing at all — a `delta` saying so.
|
||||
Rows still marked `pending` name the backlog story that will resolve them; they are
|
||||
the honest list of what has *not* been carried over yet.
|
||||
@@ -12,7 +12,15 @@
|
||||
# (if anything) carries over
|
||||
# Every row needs either `tests` (existing test IDs, module::function) or
|
||||
# `pending_story` (the backlog story that will characterize/deliver it).
|
||||
# status: characterized | pending
|
||||
# status: characterized — donor behavior pinned by characterization tests
|
||||
# resolved — replacement shipped; `parity` names the tests that prove
|
||||
# it, and `delta` states every intentional difference
|
||||
# pending — not migrated yet; `pending_story` says which story will
|
||||
# parity: test ids (path::function) in any suite, proving the replacement
|
||||
# delta: what the replacement deliberately does differently, or not at all
|
||||
#
|
||||
# Archived by US07-01: the sources referenced below now live beside this file in
|
||||
# src/ and are frozen (see README.md). Nothing in photo_pipeline imports them.
|
||||
|
||||
rows:
|
||||
# ── photo_analyzer.py ──────────────────────────────────────────────────────
|
||||
@@ -80,8 +88,12 @@ rows:
|
||||
Copying a primary's analysis into variant rows survives, but keyed by
|
||||
asset_id and recorded as stage state instead of raw row copies.
|
||||
target: photo_pipeline/services/duplicates.py
|
||||
pending_story: US01-04
|
||||
status: pending
|
||||
parity:
|
||||
- tests/integration/test_duplicate_engine.py::test_exact_copies_form_auto_decided_cluster
|
||||
- tests/integration/test_duplicate_engine.py::test_perceptual_variant_is_review_only
|
||||
delta: >
|
||||
The donor propagated variant links implicitly while writing rows; the replacement makes the canonical link a reviewable, reversible cluster decision, so a propagated link can always be undone.
|
||||
status: resolved
|
||||
|
||||
- id: pa-hashing
|
||||
area: hashing
|
||||
@@ -141,8 +153,12 @@ rows:
|
||||
classification: replace
|
||||
rationale: Console report; superseded by the duplicate-review API/UI (US01-06).
|
||||
target: photo_pipeline/api/routes + frontend duplicate review
|
||||
pending_story: US01-06
|
||||
status: pending
|
||||
parity:
|
||||
- tests/integration/test_review_api.py::test_clusters_list_and_detail
|
||||
- tests/e2e/test_review_ui.py::test_fuzzy_decision_requires_confirmation
|
||||
delta: >
|
||||
The text listing became the paged cluster API and the comparison UI; no textual report is produced.
|
||||
status: resolved
|
||||
|
||||
- id: pa-reconcile
|
||||
area: database
|
||||
@@ -202,7 +218,7 @@ rows:
|
||||
RGB-normalize (drops alpha, converts HEIC), LANCZOS resize to 2048px
|
||||
long-edge, JPEG q85 base64 — the provider-input contract. Truncated-image
|
||||
tolerance (ImageFile.LOAD_TRUNCATED_IMAGES) carries with it.
|
||||
target: photo_pipeline/integrations/vision.py
|
||||
target: photo_pipeline/services/analysis.py
|
||||
tests:
|
||||
- test_pa_imaging::test_prepare_image_small_passthrough_jpeg
|
||||
- test_pa_imaging::test_prepare_image_resizes_to_max_long_edge
|
||||
@@ -230,9 +246,12 @@ rows:
|
||||
response validation, 429/503 retry with exponential backoff. Prompt and
|
||||
model/config version must be persisted per analysis_runs. Characterized
|
||||
against a deterministic fake provider when the analysis service is ported.
|
||||
target: photo_pipeline/integrations/vision.py
|
||||
pending_story: US02-06
|
||||
status: pending
|
||||
target: photo_pipeline/services/analysis.py
|
||||
parity:
|
||||
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
||||
delta: >
|
||||
The prompt and response schema carry over; the provider is an injected adapter so the privacy gate is testable, and results are keyed to asset ids rather than paths.
|
||||
status: resolved
|
||||
|
||||
- id: pa-throttle
|
||||
area: logging
|
||||
@@ -241,8 +260,8 @@ rows:
|
||||
rationale: >
|
||||
Rolling throttle window + persistent throttle_events.jsonl + RPD day
|
||||
counter become job metrics/events on the durable job model.
|
||||
target: photo_pipeline/jobs/coordinator.py
|
||||
pending_story: US02-02
|
||||
target: photo_pipeline/services/jobs.py + photo_pipeline/services/analysis.py
|
||||
pending_story: US07-04
|
||||
status: pending
|
||||
|
||||
- id: pa-nsfw-filter
|
||||
@@ -312,8 +331,12 @@ rows:
|
||||
checked between items, double-SIGINT force quit — becomes the durable
|
||||
JobRunner worker loop with the same drain-and-resume semantics.
|
||||
target: photo_pipeline/jobs/worker.py
|
||||
pending_story: US02-02
|
||||
status: pending
|
||||
parity:
|
||||
- tests/integration/test_worker.py::test_worker_processes_all_items
|
||||
- tests/integration/test_worker.py::test_cooperative_cancellation_leaves_items_resumable
|
||||
delta: >
|
||||
The in-process folder loop with SIGINT handling became durable jobs claimed by a worker: cancellation is a persisted request, not a signal, and an interrupted run resumes from the database instead of restarting.
|
||||
status: resolved
|
||||
|
||||
- id: pa-ui-terminal
|
||||
area: ui
|
||||
@@ -371,8 +394,12 @@ rows:
|
||||
the JSONL history logger become structured JSON logging with job_id/
|
||||
asset_id and job_events rows; per-photo history maps to job events.
|
||||
target: photo_pipeline structured logging + jobs/job_events
|
||||
pending_story: US02-02
|
||||
status: pending
|
||||
parity:
|
||||
- tests/integration/test_jobs.py::test_enqueue_persists_items_and_event
|
||||
- tests/integration/test_jobs_sse.py::test_sse_streams_all_events_then_closes
|
||||
delta: >
|
||||
The JSONL history file and rich console handler are replaced by structured JSON logs plus durable job_events; the browser reads events over SSE rather than tailing a file.
|
||||
status: resolved
|
||||
|
||||
- id: pa-balance
|
||||
area: vision
|
||||
@@ -382,8 +409,11 @@ rows:
|
||||
Provider balance/quota probes (report 'unsupported' on providers without
|
||||
the endpoint). Network-bound; characterized against the fake provider.
|
||||
target: photo_pipeline/services/analysis.py
|
||||
pending_story: US02-06
|
||||
status: pending
|
||||
parity:
|
||||
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
||||
delta: >
|
||||
Not carried over: balance/quota polling was provider-specific (Gemini/OpenAI billing endpoints) and key-scoped. Cost reporting, when a story asks for it, comes from the per-run usage recorded with each analysis result rather than from a vendor endpoint.
|
||||
status: resolved
|
||||
|
||||
- id: pa-cli
|
||||
area: configuration
|
||||
@@ -393,9 +423,13 @@ rows:
|
||||
argparse surface is superseded by the API; flags map to job configs
|
||||
(documented in WEBAPP_CONCEPT.md §8 parity table). Transitional CLI calls
|
||||
the shared services until archival (E07).
|
||||
target: photo_pipeline/api + transitional CLI
|
||||
pending_story: US07-01
|
||||
status: pending
|
||||
target: photo_pipeline/__main__.py (serve | worker | migrate | import-legacy-scores) + /api/v1
|
||||
parity:
|
||||
- tests/integration/test_app_lifecycle.py::test_restart_preserves_data_and_reruns_migrations
|
||||
- tests/unit/test_legacy_archive.py::test_production_code_never_imports_an_archived_module
|
||||
delta: >
|
||||
The argparse surface is not reproduced. Every flag that drove work became an API command or a job configuration; the CLI keeps only what an application needs to be operated (serve, worker, migrate) plus the one-off legacy CSV import.
|
||||
status: resolved
|
||||
|
||||
# ── nsfwtag/ ───────────────────────────────────────────────────────────────
|
||||
- id: nt-discovery
|
||||
@@ -422,9 +456,17 @@ rows:
|
||||
nsfw_scores.csv stops being the source of truth (concept: DB state).
|
||||
Format characterized (4-decimal scores, bad rows dropped) because the
|
||||
existing CSV must migrate into assets.safety_score.
|
||||
target: photo_pipeline/repositories (safety), CSV import in US01-02 migration
|
||||
target: photo_pipeline/services/legacy_import.py (one-off import into safety_reviews)
|
||||
tests: [test_nsfwtag::test_score_cache_roundtrip_and_tolerance]
|
||||
status: characterized
|
||||
parity:
|
||||
- tests/integration/test_legacy_import.py::test_scores_are_imported_onto_asset_identity
|
||||
- tests/integration/test_legacy_import.py::test_a_reviewed_asset_is_never_overwritten_by_the_csv
|
||||
delta: >
|
||||
The CSV is no longer read at runtime at all: it is imported once into
|
||||
scored-but-unreviewed safety_reviews rows and left on disk untouched. A path
|
||||
that matches nothing is reported, never turned into an asset, and a human
|
||||
decision always outranks an imported score.
|
||||
status: resolved
|
||||
|
||||
- id: nt-score-model
|
||||
area: nsfw
|
||||
@@ -485,8 +527,11 @@ rows:
|
||||
Newline-list bulk tagging (nsfw_confirmed.txt flow) is superseded by DB
|
||||
review decisions; the existing list is a one-time migration input.
|
||||
target: photo_pipeline/services/safety.py (decision import in US01-02)
|
||||
pending_story: US01-02
|
||||
status: pending
|
||||
parity:
|
||||
- tests/integration/test_safety_parity.py::test_extracted_marks_and_partition_match_donor
|
||||
delta: >
|
||||
Bulk keyword application from a file list is replaced by decisions against asset ids; the EXIF write itself is the extracted, read-back-verified one.
|
||||
status: resolved
|
||||
|
||||
- id: nt-ui
|
||||
area: ui
|
||||
@@ -498,8 +543,11 @@ rows:
|
||||
threshold/score review flow, lightbox and keyboard model are the frontend
|
||||
donor for the Safety view (preserved per concept §10; ported in US02-01).
|
||||
target: photo_pipeline/api + frontend Safety view
|
||||
pending_story: US02-01
|
||||
status: pending
|
||||
parity:
|
||||
- tests/e2e/test_workflow_views.py::test_safety_review_decide_persists_across_reload
|
||||
delta: >
|
||||
The stdlib review server is replaced by the API plus the Safety view; the donor's layout, thresholds, and keyboard flow carry over, its embedded HTML generation does not.
|
||||
status: resolved
|
||||
|
||||
- id: nt-bench
|
||||
area: nsfw
|
||||
@@ -509,8 +557,11 @@ rows:
|
||||
Dev-only model benchmark; archived without webapp replacement (recorded
|
||||
basis of the AdamCodd model choice). No production caller.
|
||||
target: none (archive as reference)
|
||||
pending_story: US07-01
|
||||
status: pending
|
||||
parity:
|
||||
- tests/unit/test_legacy_archive.py::test_every_archived_source_matches_its_checksum
|
||||
delta: >
|
||||
No replacement: a dev-only benchmark whose result (the AdamCodd model choice) is already recorded. Kept in the archive as the basis of that choice.
|
||||
status: resolved
|
||||
|
||||
# ── webapp/ ────────────────────────────────────────────────────────────────
|
||||
- id: wa-query-search
|
||||
@@ -579,9 +630,13 @@ rows:
|
||||
Subprocess-driving-the-CLI job control is superseded by durable DB jobs
|
||||
with a worker process. Two ideas carry over: progress derived from DB
|
||||
counts (not job-private state) and single-mutating-job enforcement.
|
||||
target: photo_pipeline/jobs/coordinator.py
|
||||
pending_story: US02-02
|
||||
status: pending
|
||||
target: photo_pipeline/services/jobs.py + photo_pipeline/jobs/worker.py
|
||||
parity:
|
||||
- tests/integration/test_jobs.py::test_idempotency_key_returns_same_job
|
||||
- tests/integration/test_worker.py::test_handler_failure_fails_the_job
|
||||
delta: >
|
||||
Subprocess supervision of a CLI is replaced by durable jobs in the same process family: there is no subprocess to supervise, and progress is persisted rather than scraped from stdout.
|
||||
status: resolved
|
||||
|
||||
- id: wa-server
|
||||
area: ui
|
||||
@@ -593,5 +648,9 @@ rows:
|
||||
browser. analyzer.html + page.py design (dark OLED tokens, Library/
|
||||
Analyze/Stats views) is frontend donor material per concept §10.
|
||||
target: photo_pipeline/api/app.py + frontend
|
||||
pending_story: US02-05
|
||||
status: pending
|
||||
parity:
|
||||
- tests/e2e/test_frontend_shell.py::test_shell_loads_assets_without_console_or_network_errors
|
||||
- tests/e2e/test_frontend_shell.py::test_deep_link_and_reload_restore_view_and_filters
|
||||
delta: >
|
||||
The stdlib HTTP routes become the versioned FastAPI API and the static frontend shell; Python no longer interpolates HTML.
|
||||
status: resolved
|
||||
18
legacy_cli_archive/photo_analyzer.env.sample
Normal file
18
legacy_cli_archive/photo_analyzer.env.sample
Normal file
@@ -0,0 +1,18 @@
|
||||
# Sample configuration for the archived photo_analyzer.py CLI (US07-01).
|
||||
#
|
||||
# REDACTED: no value below is real. Each line names a variable the CLI read and
|
||||
# what belongs there; the placeholders are deliberately not key-shaped, so this
|
||||
# file can never be mistaken for — or scanned as — a credential.
|
||||
|
||||
LLM_API_KEY=<paste your provider key here>
|
||||
LLM_BASE_URL=<provider base url, e.g. the OpenAI-compatible Gemini endpoint>
|
||||
LLM_MODEL=<model id, e.g. a Gemini Flash release>
|
||||
|
||||
# Optional tuning the CLI read from the same file:
|
||||
PHASH_THRESHOLD=8
|
||||
MAX_WORKERS=4
|
||||
RETRY_ATTEMPTS=3
|
||||
RPD_LIMIT=0
|
||||
|
||||
# The CLI ignored its own shipped placeholder (a literal "sk-REPLACE..." string)
|
||||
# until it was replaced, and a shell variable always won over this file.
|
||||
20
legacy_cli_archive/requirements-lock.txt
Normal file
20
legacy_cli_archive/requirements-lock.txt
Normal file
@@ -0,0 +1,20 @@
|
||||
# Final dependency lock of the archived CLIs (US07-01).
|
||||
#
|
||||
# These are the versions present in the environment the archive was taken from —
|
||||
# what the frozen sources were last verified against by the characterization
|
||||
# suite. Restoring a donor for forensics means pinning these, not "latest".
|
||||
#
|
||||
# Python 3.14.6
|
||||
|
||||
openai==3.0.0 # photo_analyzer: OpenAI-compatible vision client
|
||||
numpy==2.4.6 # photo_analyzer, nsfwtag: pixel work
|
||||
Pillow==12.3.0 # photo_analyzer, nsfwtag: decode/resize
|
||||
rich==15.0.0 # photo_analyzer: console output
|
||||
scipy==1.18.0 # photo_analyzer: perceptual-hash DCT
|
||||
PyYAML==6.0.3 # tooling that reads the donor ledger
|
||||
|
||||
# NSFW inference (nsfwtag/scoring.py, nsfwtag/bench.py) was never installed in the
|
||||
# archiving environment; the model stack is recorded here from the sources so a
|
||||
# forensic run can reproduce it, not from a resolved lock:
|
||||
# torch, transformers, timm — AdamCodd/vit-base-nsfw-detector (see nsfwtag/README.md)
|
||||
# exiftool is an external binary, not a Python package.
|
||||
52
migrations/versions/0009_upload_report_outcomes.py
Normal file
52
migrations/versions/0009_upload_report_outcomes.py
Normal file
@@ -0,0 +1,52 @@
|
||||
"""Parsed uploader outcomes (US05-03).
|
||||
|
||||
Revision ID: 0009_upload_report_outcomes
|
||||
Revises: 0008_upload_batches
|
||||
Create Date: 2026-08-16
|
||||
|
||||
Per-item outcomes parsed from the immich-go report plus the parser evidence that
|
||||
produced them, so "what did Immich do with this file?" survives a restart.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0009_upload_report_outcomes"
|
||||
down_revision = "0008_upload_batches"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# NULL parser = the uploader version has no pinned grammar; the batch is then
|
||||
# requires_verification however the process exited.
|
||||
op.add_column("upload_batches", sa.Column("parser", sa.String(), nullable=True))
|
||||
op.add_column("upload_batches", sa.Column("parser_version", sa.Integer(), nullable=True))
|
||||
op.add_column(
|
||||
"upload_batches", sa.Column("parsed_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
# verified | requires_verification
|
||||
op.add_column("upload_batches", sa.Column("outcome_state", sa.String(), nullable=True))
|
||||
op.add_column("upload_batches", sa.Column("outcome_counts", sa.String(), nullable=True))
|
||||
op.add_column("upload_batches", sa.Column("report_counts", sa.String(), nullable=True))
|
||||
|
||||
# uploaded | upgraded | duplicate | skipped | failed | unknown
|
||||
op.add_column("upload_items", sa.Column("outcome", sa.String(), nullable=True))
|
||||
op.add_column("upload_items", sa.Column("evidence", sa.String(), nullable=True))
|
||||
op.add_column(
|
||||
"upload_items", sa.Column("outcome_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for column in ("outcome_at", "evidence", "outcome"):
|
||||
op.drop_column("upload_items", column)
|
||||
for column in (
|
||||
"report_counts",
|
||||
"outcome_counts",
|
||||
"outcome_state",
|
||||
"parsed_at",
|
||||
"parser_version",
|
||||
"parser",
|
||||
):
|
||||
op.drop_column("upload_batches", column)
|
||||
70
migrations/versions/0010_upload_verification.py
Normal file
70
migrations/versions/0010_upload_verification.py
Normal file
@@ -0,0 +1,70 @@
|
||||
"""Upload verification and manual resolution (US05-04).
|
||||
|
||||
Revision ID: 0010_upload_verification
|
||||
Revises: 0009_upload_report_outcomes
|
||||
Create Date: 2026-08-16
|
||||
|
||||
Per-item server evidence plus the append-only history that produced it, so an
|
||||
uncertain upload can be resolved without ever guessing that it succeeded.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0010_upload_verification"
|
||||
down_revision = "0009_upload_report_outcomes"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"upload_batches", sa.Column("verified_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
# At least one uploaded file was edited afterwards: a visible warning that also
|
||||
# blocks re-running the batch.
|
||||
op.add_column(
|
||||
"upload_batches",
|
||||
sa.Column("stale_bytes", sa.Boolean(), nullable=False, server_default=sa.false()),
|
||||
)
|
||||
|
||||
# present | absent | inconclusive | manual
|
||||
op.add_column("upload_items", sa.Column("verification", sa.String(), nullable=True))
|
||||
op.add_column(
|
||||
"upload_items", sa.Column("verified_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
op.add_column("upload_items", sa.Column("observed_sha256", sa.String(), nullable=True))
|
||||
op.add_column(
|
||||
"upload_items",
|
||||
sa.Column("changed_after_upload", sa.Boolean(), nullable=False, server_default=sa.false()),
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"upload_verifications",
|
||||
sa.Column("id", sa.String(), primary_key=True),
|
||||
sa.Column(
|
||||
"batch_id",
|
||||
sa.String(),
|
||||
sa.ForeignKey("upload_batches.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
),
|
||||
sa.Column("asset_id", sa.String(), nullable=False),
|
||||
sa.Column("action", sa.String(), nullable=False),
|
||||
sa.Column("source", sa.String(), nullable=False),
|
||||
sa.Column("result", sa.String(), nullable=False),
|
||||
sa.Column("outcome", sa.String(), nullable=True),
|
||||
sa.Column("evidence", sa.String(), nullable=False),
|
||||
sa.Column("actor", sa.String(), nullable=True),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("upload_verifications")
|
||||
for column in ("changed_after_upload", "observed_sha256", "verified_at", "verification"):
|
||||
op.drop_column("upload_items", column)
|
||||
for column in ("stale_bytes", "verified_at"):
|
||||
op.drop_column("upload_batches", column)
|
||||
40
migrations/versions/0011_archive_locations.py
Normal file
40
migrations/versions/0011_archive_locations.py
Normal file
@@ -0,0 +1,40 @@
|
||||
"""Archive locations (US06-01).
|
||||
|
||||
Revision ID: 0011_archive_locations
|
||||
Revises: 0010_upload_verification
|
||||
Create Date: 2026-08-16
|
||||
|
||||
Configured archive destinations with their stable media identity, last probed
|
||||
capabilities, and state.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0011_archive_locations"
|
||||
down_revision = "0010_upload_verification"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"archive_locations",
|
||||
sa.Column("id", sa.String(), primary_key=True),
|
||||
sa.Column("name", sa.String(), nullable=False, unique=True),
|
||||
sa.Column("root", sa.String(), nullable=False),
|
||||
sa.Column("media_id", sa.String(), nullable=False, unique=True),
|
||||
sa.Column("capabilities", sa.String(), nullable=True),
|
||||
sa.Column("state", sa.String(), nullable=False, server_default="offline"),
|
||||
sa.Column("last_seen_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("archive_locations")
|
||||
96
migrations/versions/0012_archive_plans.py
Normal file
96
migrations/versions/0012_archive_plans.py
Normal file
@@ -0,0 +1,96 @@
|
||||
"""Archive plans, per-item transfer journal, and archived asset location (US06-02).
|
||||
|
||||
Revision ID: 0012_archive_plans
|
||||
Revises: 0011_archive_locations
|
||||
Create Date: 2026-08-16
|
||||
|
||||
The journal is what makes removing an original recoverable: every item records its
|
||||
source, destination, expected hash, and the state it had reached before the process
|
||||
died. Assets gain the archive location and relative path so an offline original is
|
||||
still explained rather than looking missing.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0012_archive_plans"
|
||||
down_revision = "0011_archive_locations"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"archive_plans",
|
||||
sa.Column("id", sa.String(), primary_key=True),
|
||||
sa.Column(
|
||||
"location_id",
|
||||
sa.String(),
|
||||
sa.ForeignKey("archive_locations.id"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
),
|
||||
# The preflight token this plan was approved against.
|
||||
sa.Column("token", sa.String(), nullable=False),
|
||||
sa.Column("albums", sa.String(), nullable=True), # JSON array
|
||||
# planned | applying | complete | failed
|
||||
sa.Column("state", sa.String(), nullable=False, server_default="planned"),
|
||||
sa.Column("schema_version", sa.Integer(), nullable=False, server_default="1"),
|
||||
sa.Column("asset_count", sa.Integer(), nullable=False, server_default="0"),
|
||||
sa.Column("byte_size", sa.Integer(), nullable=False, server_default="0"),
|
||||
# Bumped on every claim and used as the fencing token.
|
||||
sa.Column("version", sa.Integer(), nullable=False, server_default="1"),
|
||||
sa.Column("worker_id", sa.String(), nullable=True),
|
||||
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()
|
||||
),
|
||||
)
|
||||
op.create_table(
|
||||
"archive_operations",
|
||||
sa.Column("id", sa.String(), primary_key=True),
|
||||
sa.Column(
|
||||
"plan_id",
|
||||
sa.String(),
|
||||
sa.ForeignKey("archive_plans.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
),
|
||||
sa.Column("sequence", sa.Integer(), nullable=False),
|
||||
sa.Column("album", sa.String(), nullable=False),
|
||||
sa.Column("asset_id", sa.String(), sa.ForeignKey("assets.id"), nullable=False, index=True),
|
||||
sa.Column("source_path", sa.String(), nullable=False),
|
||||
sa.Column("destination_path", sa.String(), nullable=False),
|
||||
# Relative to the location root: the medium can be mounted anywhere later.
|
||||
sa.Column("archive_path", sa.String(), nullable=False),
|
||||
sa.Column("expected_sha256", sa.String(), nullable=False),
|
||||
sa.Column("byte_size", sa.Integer(), nullable=True),
|
||||
sa.Column("same_filesystem", sa.Boolean(), nullable=True),
|
||||
# planned | transferring | verified | removing | complete | failed
|
||||
sa.Column("journal_state", sa.String(), nullable=False, server_default="planned"),
|
||||
sa.Column("attempt_count", sa.Integer(), nullable=False, server_default="0"),
|
||||
sa.Column("fencing_token", sa.Integer(), nullable=True),
|
||||
sa.Column("worker_id", sa.String(), nullable=True),
|
||||
sa.Column("verified_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("removed_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("error_code", sa.String(), nullable=True),
|
||||
sa.Column("error_message", sa.String(), nullable=True),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()
|
||||
),
|
||||
sa.UniqueConstraint("plan_id", "sequence", name="uq_archive_operations_plan_sequence"),
|
||||
)
|
||||
# Plain columns: SQLite cannot ALTER a table to add a foreign key, and the
|
||||
# relationship is enforced by the service that writes them.
|
||||
op.add_column("assets", sa.Column("archive_location_id", sa.String(), nullable=True))
|
||||
op.add_column("assets", sa.Column("archive_path", sa.String(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("assets", "archive_path")
|
||||
op.drop_column("assets", "archive_location_id")
|
||||
op.drop_table("archive_operations")
|
||||
op.drop_table("archive_plans")
|
||||
29
migrations/versions/0013_protected_thumbnails.py
Normal file
29
migrations/versions/0013_protected_thumbnails.py
Normal file
@@ -0,0 +1,29 @@
|
||||
"""Protected thumbnails (US06-03).
|
||||
|
||||
Revision ID: 0013_protected_thumbnails
|
||||
Revises: 0012_archive_plans
|
||||
Create Date: 2026-08-16
|
||||
|
||||
A protected thumbnail is the durable comparison preview of an asset whose
|
||||
original has left active storage. It is evidence rather than cache, so the LRU
|
||||
quota must not evict it: the archive medium may be offline when it is needed.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0013_protected_thumbnails"
|
||||
down_revision = "0012_archive_plans"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"thumbnails",
|
||||
sa.Column("protected", sa.Boolean(), nullable=False, server_default=sa.false()),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("thumbnails", "protected")
|
||||
37
migrations/versions/0014_restore_plans.py
Normal file
37
migrations/versions/0014_restore_plans.py
Normal file
@@ -0,0 +1,37 @@
|
||||
"""Restore plans and archive divergence (US06-04).
|
||||
|
||||
Revision ID: 0014_restore_plans
|
||||
Revises: 0013_protected_thumbnails
|
||||
Create Date: 2026-08-16
|
||||
|
||||
Restore reuses the archive plan and journal tables: the crash-safe question is the
|
||||
same one in the opposite direction (copy, verify, publish, register), so the rows
|
||||
gain a ``direction`` instead of a parallel pair of tables. ``archive_divergent_at``
|
||||
records the moment an archived copy was proven to hold bytes that are not the ones
|
||||
the database recorded — a restore must never silently accept a different file.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0014_restore_plans"
|
||||
down_revision = "0013_protected_thumbnails"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for table in ("archive_plans", "archive_operations"):
|
||||
op.add_column(
|
||||
table,
|
||||
sa.Column("direction", sa.String(), nullable=False, server_default="archive"),
|
||||
)
|
||||
op.add_column(
|
||||
"assets", sa.Column("archive_divergent_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("assets", "archive_divergent_at")
|
||||
for table in ("archive_plans", "archive_operations"):
|
||||
op.drop_column(table, "direction")
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Application management CLI: ``python -m photo_pipeline {serve,migrate}``."""
|
||||
"""Application management CLI: ``python -m photo_pipeline {serve,migrate,worker,import-legacy-scores}``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -16,6 +16,17 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
commands.add_parser("migrate", help="Upgrade the database to the latest revision")
|
||||
worker_cmd = commands.add_parser("worker", help="Run a durable-job worker")
|
||||
worker_cmd.add_argument("--id", default="worker-1", help="Worker id (lease owner)")
|
||||
import_cmd = commands.add_parser(
|
||||
"import-legacy-scores",
|
||||
help="Import the archived CLI's nsfw_scores.csv into the database (US07-01)",
|
||||
)
|
||||
import_cmd.add_argument("csv", help="Path to nsfw_scores.csv")
|
||||
import_cmd.add_argument(
|
||||
"--overwrite", action="store_true", help="Replace differing imported scores"
|
||||
)
|
||||
import_cmd.add_argument(
|
||||
"--dry-run", action="store_true", help="Report what would happen and change nothing"
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
config = Config.from_env()
|
||||
@@ -25,6 +36,24 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
run_migrations(config.database_url)
|
||||
return 0
|
||||
|
||||
if args.command == "import-legacy-scores":
|
||||
import json
|
||||
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory
|
||||
from photo_pipeline.services.legacy_import import LegacyImportService, write_report
|
||||
|
||||
run_migrations(config.database_url)
|
||||
engine = create_db_engine(config.database_url)
|
||||
service = LegacyImportService(create_session_factory(engine))
|
||||
report = service.import_nsfw_scores(
|
||||
args.csv, overwrite=args.overwrite, dry_run=args.dry_run
|
||||
)
|
||||
# The report is the point: an import nobody can audit is not a migration.
|
||||
if not args.dry_run:
|
||||
write_report(report, config.data_dir)
|
||||
print(json.dumps(report.counts, indent=2))
|
||||
return 0
|
||||
|
||||
if args.command == "worker":
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ from fastapi.staticfiles import StaticFiles
|
||||
from photo_pipeline.api.routes import (
|
||||
albums,
|
||||
analysis,
|
||||
archives,
|
||||
duplicates,
|
||||
health,
|
||||
inventory,
|
||||
@@ -73,6 +74,7 @@ def create_app(config: Config | None = None) -> FastAPI:
|
||||
app.include_router(albums.router, prefix="/api/v1")
|
||||
app.include_router(renames.router, prefix="/api/v1")
|
||||
app.include_router(uploads.router, prefix="/api/v1")
|
||||
app.include_router(archives.router, prefix="/api/v1")
|
||||
# Static single-page app (hash-routed). Mounted last so /api/v1 wins.
|
||||
if FRONTEND_DIR.is_dir():
|
||||
app.mount("/app", StaticFiles(directory=FRONTEND_DIR, html=True), name="app")
|
||||
|
||||
234
photo_pipeline/api/routes/archives.py
Normal file
234
photo_pipeline/api/routes/archives.py
Normal file
@@ -0,0 +1,234 @@
|
||||
"""Archive location, preflight, plan, and restore API (US06-01, US06-02, US06-04).
|
||||
|
||||
Registering a location writes a marker onto the medium; preflight is a command
|
||||
rather than a read, because it probes the destination, hashes the scope, and issues
|
||||
the token an archive plan must present. Creating a plan writes only database rows —
|
||||
the transfer itself runs on the durable ``archive`` lane, never in the request
|
||||
thread, because it removes originals.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from photo_pipeline.jobs.domain_handlers import ARCHIVE_LOCK, ARCHIVE_PLAN, RESTORE_PLAN
|
||||
from photo_pipeline.services.archives import ArchiveError, ArchiveService
|
||||
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
||||
from photo_pipeline.services.jobs import JobBlocked, JobService
|
||||
from photo_pipeline.services.restores import RestoreService
|
||||
|
||||
router = APIRouter(tags=["archives"])
|
||||
|
||||
# Which failures are the caller's request (422), a missing thing (404), or state
|
||||
# that changed under the caller (409).
|
||||
NOT_FOUND_CODES = {"unknown_location", "unknown_plan"}
|
||||
CONFLICT_CODES = {"stale_token", "stale_plan", "archive_pending"}
|
||||
|
||||
|
||||
class RegisterLocationRequest(BaseModel):
|
||||
name: str
|
||||
root: str
|
||||
|
||||
|
||||
class PreflightRequest(BaseModel):
|
||||
location_id: str
|
||||
# ``None`` means every album; an explicit list scopes the check.
|
||||
albums: list[str] | None = None
|
||||
|
||||
|
||||
class CreatePlanRequest(PreflightRequest):
|
||||
# The token of the preflight the user approved; a stale one is refused.
|
||||
token: str
|
||||
|
||||
|
||||
class RestoreRequest(BaseModel):
|
||||
location_id: str
|
||||
# ``None`` means every asset archived at this location.
|
||||
asset_ids: list[str] | None = None
|
||||
|
||||
|
||||
class CreateRestoreRequest(RestoreRequest):
|
||||
token: str
|
||||
|
||||
|
||||
def _service(request: Request) -> ArchiveService:
|
||||
return ArchiveService(request.app.state.session_factory, config=request.app.state.config)
|
||||
|
||||
|
||||
def _restores(request: Request) -> RestoreService:
|
||||
return RestoreService(request.app.state.session_factory, config=request.app.state.config)
|
||||
|
||||
|
||||
def _transfers(request: Request) -> ArchiveTransferService:
|
||||
return ArchiveTransferService(
|
||||
request.app.state.session_factory, config=request.app.state.config
|
||||
)
|
||||
|
||||
|
||||
def _error(error: ArchiveError) -> JSONResponse:
|
||||
if error.code in NOT_FOUND_CODES:
|
||||
status = 404
|
||||
elif error.code in CONFLICT_CODES:
|
||||
status = 409
|
||||
else:
|
||||
status = 422
|
||||
return JSONResponse(
|
||||
status_code=status, content={"error": {"code": error.code, "message": str(error)}}
|
||||
)
|
||||
|
||||
|
||||
@router.post("/archive-locations", status_code=201)
|
||||
def register_location(body: RegisterLocationRequest, request: Request):
|
||||
try:
|
||||
return _service(request).register(body.name, body.root)
|
||||
except ArchiveError as error:
|
||||
return _error(error)
|
||||
|
||||
|
||||
@router.get("/archive-locations")
|
||||
def list_locations(request: Request) -> dict:
|
||||
return {"locations": _service(request).locations()}
|
||||
|
||||
|
||||
@router.post("/archive-preflight")
|
||||
def preflight(body: PreflightRequest, request: Request):
|
||||
try:
|
||||
return _service(request).preflight(body.location_id, body.albums)
|
||||
except ArchiveError as error:
|
||||
return _error(error)
|
||||
|
||||
|
||||
@router.post("/archive-plans", status_code=201)
|
||||
def create_plan(body: CreatePlanRequest, request: Request):
|
||||
"""Turn an approved preflight into a durable, journaled plan. Nothing moves."""
|
||||
try:
|
||||
return _transfers(request).create(body.location_id, body.albums, token=body.token)
|
||||
except ArchiveError as error:
|
||||
return _error(error)
|
||||
|
||||
|
||||
@router.get("/archive-plans")
|
||||
def list_plans(request: Request) -> dict:
|
||||
return {"plans": _transfers(request).list()}
|
||||
|
||||
|
||||
@router.get("/archive-plans/{plan_id}")
|
||||
def get_plan(plan_id: str, request: Request):
|
||||
plan = _transfers(request).get(plan_id)
|
||||
if plan is None:
|
||||
return _error(ArchiveError("unknown_plan", f"unknown archive plan {plan_id}"))
|
||||
return plan
|
||||
|
||||
|
||||
@router.post("/archive-plans/{plan_id}/apply")
|
||||
def apply_plan(plan_id: str, request: Request):
|
||||
"""Queue the transfer on the archiver lane. The worker removes the sources."""
|
||||
service = _transfers(request)
|
||||
plan = service.get(plan_id)
|
||||
if plan is None:
|
||||
return _error(ArchiveError("unknown_plan", f"unknown archive plan {plan_id}"))
|
||||
if service.journal.blocks_mutation():
|
||||
unresolved = [row for row in service.journal.incomplete() if row["plan_id"] != plan_id]
|
||||
if unresolved:
|
||||
return _error(
|
||||
ArchiveError(
|
||||
"archive_pending",
|
||||
f"an unresolved archive operation ({unresolved[0]['id']}) must be recovered",
|
||||
)
|
||||
)
|
||||
try:
|
||||
job = JobService(request.app.state.session_factory).enqueue(
|
||||
ARCHIVE_PLAN,
|
||||
lock=ARCHIVE_LOCK,
|
||||
# One queued attempt per plan version: a double-clicked apply reuses it.
|
||||
idempotency_key=f"archive:{plan_id}:{plan['version']}",
|
||||
items=[plan_id],
|
||||
)
|
||||
except JobBlocked as error:
|
||||
return JSONResponse(
|
||||
status_code=409, content={"error": {"code": error.code, "message": str(error)}}
|
||||
)
|
||||
return {"plan_id": plan_id, "job": job}
|
||||
|
||||
|
||||
@router.post("/restore-preflight")
|
||||
def restore_preflight(body: RestoreRequest, request: Request):
|
||||
"""Validate restoring archived assets back into the library. Nothing moves."""
|
||||
try:
|
||||
return _restores(request).preflight(body.location_id, body.asset_ids)
|
||||
except ArchiveError as error:
|
||||
return _error(error)
|
||||
|
||||
|
||||
@router.post("/restore-plans", status_code=201)
|
||||
def create_restore_plan(body: CreateRestoreRequest, request: Request):
|
||||
try:
|
||||
return _restores(request).create(body.location_id, body.asset_ids, token=body.token)
|
||||
except ArchiveError as error:
|
||||
return _error(error)
|
||||
|
||||
|
||||
@router.get("/restore-plans")
|
||||
def list_restore_plans(request: Request) -> dict:
|
||||
return {"plans": _restores(request).list()}
|
||||
|
||||
|
||||
@router.get("/restore-plans/{plan_id}")
|
||||
def get_restore_plan(plan_id: str, request: Request):
|
||||
plan = _restores(request).get(plan_id)
|
||||
if plan is None:
|
||||
return _error(ArchiveError("unknown_plan", f"unknown restore plan {plan_id}"))
|
||||
return plan
|
||||
|
||||
|
||||
@router.post("/restore-plans/{plan_id}/apply")
|
||||
def apply_restore_plan(plan_id: str, request: Request):
|
||||
"""Queue the restore on the archiver lane — the same single lane as archiving,
|
||||
because both move the same originals."""
|
||||
service = _restores(request)
|
||||
plan = service.get(plan_id)
|
||||
if plan is None:
|
||||
return _error(ArchiveError("unknown_plan", f"unknown restore plan {plan_id}"))
|
||||
unresolved = [row for row in service.journal.incomplete() if row["plan_id"] != plan_id]
|
||||
if unresolved:
|
||||
return _error(
|
||||
ArchiveError(
|
||||
"archive_pending",
|
||||
f"an unresolved archive operation ({unresolved[0]['id']}) must be recovered",
|
||||
)
|
||||
)
|
||||
try:
|
||||
job = JobService(request.app.state.session_factory).enqueue(
|
||||
RESTORE_PLAN,
|
||||
lock=ARCHIVE_LOCK,
|
||||
idempotency_key=f"restore:{plan_id}:{plan['version']}",
|
||||
items=[plan_id],
|
||||
)
|
||||
except JobBlocked as error:
|
||||
return JSONResponse(
|
||||
status_code=409, content={"error": {"code": error.code, "message": str(error)}}
|
||||
)
|
||||
return {"plan_id": plan_id, "job": job}
|
||||
|
||||
|
||||
@router.get("/restore-recovery")
|
||||
def restore_recovery_status(request: Request) -> dict:
|
||||
return _restores(request).recovery_status()
|
||||
|
||||
|
||||
@router.post("/restore-recovery/resolve")
|
||||
def resolve_restore_recovery(request: Request) -> dict:
|
||||
return _restores(request).recover()
|
||||
|
||||
|
||||
@router.get("/archive-recovery")
|
||||
def recovery_status(request: Request) -> dict:
|
||||
"""What an interrupted transfer left behind, straight from journal + disk."""
|
||||
return _transfers(request).recovery_status()
|
||||
|
||||
|
||||
@router.post("/archive-recovery/resolve")
|
||||
def resolve_recovery(request: Request) -> dict:
|
||||
return _transfers(request).recover()
|
||||
@@ -1,9 +1,12 @@
|
||||
"""Upload preflight and batch API (US05-01, US05-02).
|
||||
"""Upload preflight, batch, and verification API (US05-01, US05-02, US05-04).
|
||||
|
||||
Preflight is a command, not a resource read: it contacts the Immich server, hashes
|
||||
the current bytes, and issues a token. Creating a batch requires that token, and
|
||||
starting one enqueues a durable job on the single ``upload`` lane — the API never
|
||||
runs the uploader in the request thread.
|
||||
|
||||
``verify`` and ``resolve`` are the way out of an uncertain outcome; ``start``
|
||||
refuses one with ``409`` rather than letting the browser retry it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -19,6 +22,11 @@ from photo_pipeline.services.upload_batches import (
|
||||
BatchError,
|
||||
UploadBatchService,
|
||||
)
|
||||
from photo_pipeline.services.upload_verification import (
|
||||
UploadVerificationService,
|
||||
VerificationError,
|
||||
retry_blockers,
|
||||
)
|
||||
from photo_pipeline.services.uploads import UploadError, UploadService
|
||||
|
||||
router = APIRouter(tags=["uploads"])
|
||||
@@ -36,6 +44,16 @@ class CreateBatchRequest(PreflightRequest):
|
||||
token: str
|
||||
|
||||
|
||||
class ResolveRequest(BaseModel):
|
||||
asset_id: str
|
||||
# uploaded | upgraded | duplicate | skipped | failed | unknown
|
||||
outcome: str
|
||||
# What the operator actually checked, and who they are — both mandatory so a
|
||||
# manual resolution can never look like server evidence.
|
||||
evidence: str
|
||||
actor: str
|
||||
|
||||
|
||||
def _service(request: Request) -> UploadService:
|
||||
return UploadService(request.app.state.session_factory, config=request.app.state.config)
|
||||
|
||||
@@ -44,6 +62,12 @@ def _batches(request: Request) -> UploadBatchService:
|
||||
return UploadBatchService(request.app.state.session_factory, config=request.app.state.config)
|
||||
|
||||
|
||||
def _verification(request: Request) -> UploadVerificationService:
|
||||
return UploadVerificationService(
|
||||
request.app.state.session_factory, config=request.app.state.config
|
||||
)
|
||||
|
||||
|
||||
def _error(status: int, code: str, message: str) -> JSONResponse:
|
||||
return JSONResponse(status_code=status, content={"error": {"code": code, "message": message}})
|
||||
|
||||
@@ -61,9 +85,11 @@ def preflight(request: Request, body: PreflightRequest | None = None):
|
||||
def create_batches(body: CreateBatchRequest, request: Request):
|
||||
"""Turn an approved preflight into one durable batch per album."""
|
||||
try:
|
||||
return {"batches": _batches(request).create(
|
||||
body.albums, token=body.token, allow_partial=body.allow_partial
|
||||
)}
|
||||
return {
|
||||
"batches": _batches(request).create(
|
||||
body.albums, token=body.token, allow_partial=body.allow_partial
|
||||
)
|
||||
}
|
||||
except UploadError as error:
|
||||
return _error(422, "unknown_album", str(error))
|
||||
except BatchConflict as error:
|
||||
@@ -92,6 +118,11 @@ def start_batch(batch_id: str, request: Request):
|
||||
batch = service.get(batch_id)
|
||||
if batch is None:
|
||||
return _error(404, "not_found", f"unknown upload batch {batch_id}")
|
||||
# An uncertain outcome or bytes changed after upload must be resolved first
|
||||
# (US05-04); the worker refuses them too, but the user is told here.
|
||||
blocked = retry_blockers(batch)
|
||||
if blocked:
|
||||
return _error(409, blocked[0]["code"], blocked[0]["message"])
|
||||
try:
|
||||
job = JobService(request.app.state.session_factory).enqueue(
|
||||
UPLOAD_BATCH,
|
||||
@@ -105,6 +136,38 @@ def start_batch(batch_id: str, request: Request):
|
||||
return {"batch_id": batch_id, "job": job}
|
||||
|
||||
|
||||
@router.post("/upload-batches/{batch_id}/verify")
|
||||
def verify_batch(batch_id: str, request: Request):
|
||||
"""Check the batch against Immich and the bytes on disk (US05-04)."""
|
||||
try:
|
||||
return _verification(request).verify(batch_id)
|
||||
except VerificationError as error:
|
||||
return _error(404 if error.code == "not_found" else 422, error.code, str(error))
|
||||
|
||||
|
||||
@router.post("/upload-batches/{batch_id}/resolve")
|
||||
def resolve_item(batch_id: str, body: ResolveRequest, request: Request):
|
||||
"""Record an operator's own verification of one item. Evidence is mandatory."""
|
||||
try:
|
||||
return _verification(request).resolve(
|
||||
batch_id,
|
||||
body.asset_id,
|
||||
outcome=body.outcome,
|
||||
evidence=body.evidence,
|
||||
actor=body.actor,
|
||||
)
|
||||
except VerificationError as error:
|
||||
return _error(404 if error.code == "not_found" else 422, error.code, str(error))
|
||||
|
||||
|
||||
@router.get("/upload-batches/{batch_id}/verifications")
|
||||
def list_verifications(batch_id: str, request: Request):
|
||||
batch = _batches(request).get(batch_id)
|
||||
if batch is None:
|
||||
return _error(404, "not_found", f"unknown upload batch {batch_id}")
|
||||
return {"verifications": _verification(request).history(batch_id)}
|
||||
|
||||
|
||||
@router.post("/upload-batches/{batch_id}/cancel")
|
||||
def cancel_batch(batch_id: str, request: Request):
|
||||
try:
|
||||
|
||||
@@ -35,6 +35,9 @@ class Config(BaseModel):
|
||||
thumbnail_cache_quota_bytes: int = 500_000_000
|
||||
thumbnail_max_pixels: int = 100_000_000
|
||||
|
||||
# Free space an archive destination must keep beyond the transfer itself.
|
||||
archive_free_space_reserve_bytes: int = 1_000_000_000
|
||||
|
||||
vision_api_key: SecretStr | None = None
|
||||
immich_api_key: SecretStr | None = None
|
||||
immich_server_url: str = ""
|
||||
|
||||
@@ -8,7 +8,9 @@ activity log. Both come from the same builder so the preview can never drift fro
|
||||
the command that would actually run.
|
||||
|
||||
Server reachability uses ``/api/server/ping`` through stdlib ``urllib`` — the app
|
||||
has no HTTP client dependency and this is one request.
|
||||
has no HTTP client dependency and this is one request. :func:`bulk_upload_check`
|
||||
uses the same client to ask Immich which uploaded bytes it already holds, which is
|
||||
the authoritative evidence behind upload verification (US05-04).
|
||||
|
||||
:func:`run_upload` is the only place the uploader is actually executed. It never
|
||||
uses a shell (the argument list goes straight to ``execve``, so no path or album
|
||||
@@ -35,6 +37,11 @@ from pathlib import Path
|
||||
REDACTED = "***"
|
||||
PING_PATH = "/api/server/ping"
|
||||
PING_TIMEOUT_SECONDS = 5.0
|
||||
# Immich's own deduplication endpoint: the authoritative answer to "do you already
|
||||
# have these exact bytes?" used to verify uncertain uploads (US05-04).
|
||||
BULK_CHECK_PATH = "/api/assets/bulk-upload-check"
|
||||
CHECK_TIMEOUT_SECONDS = 30.0
|
||||
CHECK_BATCH_SIZE = 500
|
||||
# Reports are kept in full up to this size; beyond it the tail is dropped and the
|
||||
# result is flagged truncated rather than growing without bound (concept §17).
|
||||
MAX_REPORT_BYTES = 4_000_000
|
||||
@@ -89,6 +96,68 @@ def ping(server_url: str, *, timeout: float = PING_TIMEOUT_SECONDS) -> tuple[boo
|
||||
return False, "server did not answer with pong"
|
||||
|
||||
|
||||
def bulk_upload_check(
|
||||
server_url: str,
|
||||
api_key: str | None,
|
||||
checksums: dict[str, str],
|
||||
*,
|
||||
timeout: float = CHECK_TIMEOUT_SECONDS,
|
||||
) -> dict:
|
||||
"""Ask Immich which of these exact bytes it already holds (US05-04).
|
||||
|
||||
``checksums`` maps an application key (the asset id) to the SHA-1 of the bytes
|
||||
that were uploaded — the digest Immich itself deduplicates on. The answer is
|
||||
``{"reachable", "detail", "present"}`` where ``present`` maps each key to
|
||||
``True`` (the server rejected it as a duplicate, so it holds those bytes),
|
||||
``False`` (the server would accept it, so it does not), or ``None`` (the server
|
||||
answered something this adapter will not interpret).
|
||||
|
||||
An unreachable or unparsable server is reported, never guessed at: the caller
|
||||
must treat it as uncertainty rather than absence.
|
||||
"""
|
||||
if not server_url or not api_key:
|
||||
return {"reachable": False, "detail": "no Immich credentials configured", "present": {}}
|
||||
keys = list(checksums)
|
||||
present: dict[str, bool | None] = {}
|
||||
for start in range(0, len(keys), CHECK_BATCH_SIZE):
|
||||
# ponytail: fixed chunk size; make it configurable if a server ever rejects it.
|
||||
chunk = keys[start : start + CHECK_BATCH_SIZE]
|
||||
payload = {"assets": [{"id": key, "checksum": checksums[key]} for key in chunk]}
|
||||
request = urllib.request.Request( # noqa: S310 — http(s) URL from configuration
|
||||
server_url.rstrip("/") + BULK_CHECK_PATH,
|
||||
data=json.dumps(payload).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json", "x-api-key": api_key},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310
|
||||
body = json.loads(response.read().decode("utf-8") or "{}")
|
||||
except (urllib.error.URLError, OSError, ValueError, TimeoutError) as error:
|
||||
return {"reachable": False, "detail": f"{type(error).__name__}: {error}", "present": {}}
|
||||
results = body.get("results")
|
||||
if not isinstance(results, list):
|
||||
return {
|
||||
"reachable": False,
|
||||
"detail": "unrecognised bulk-upload-check response",
|
||||
"present": {},
|
||||
}
|
||||
for result in results:
|
||||
if not isinstance(result, dict) or result.get("id") not in checksums:
|
||||
continue
|
||||
present[result["id"]] = _holds_bytes(result)
|
||||
return {"reachable": True, "detail": None, "present": present}
|
||||
|
||||
|
||||
def _holds_bytes(result: dict) -> bool | None:
|
||||
"""Whether one bulk-upload-check result means the server already has the file."""
|
||||
action, reason = result.get("action"), result.get("reason")
|
||||
if action == "reject":
|
||||
# Only a duplicate proves possession; "unsupported-format" and friends say
|
||||
# nothing about whether the bytes are there.
|
||||
return True if reason == "duplicate" else None
|
||||
return False if action == "accept" else None
|
||||
|
||||
|
||||
def build_command(
|
||||
*,
|
||||
binary: str,
|
||||
|
||||
204
photo_pipeline/integrations/immich_go_report.py
Normal file
204
photo_pipeline/integrations/immich_go_report.py
Normal file
@@ -0,0 +1,204 @@
|
||||
"""immich-go report parsing (US05-03).
|
||||
|
||||
The uploader's output is the only local evidence of what Immich did with each file,
|
||||
and its wording changes between releases (concept §15 "external integration
|
||||
risks"). So parsing is deliberately conservative:
|
||||
|
||||
- **the parser is chosen by uploader version, not by guessing the format.** Each
|
||||
supported version family maps to one adapter with a pinned line grammar. An
|
||||
unrecognised version yields no adapter at all, which makes every item uncertain
|
||||
rather than optimistically successful.
|
||||
- **an unmatched line is never success.** Lines the adapter does not recognise are
|
||||
counted (``unparsed``) and kept as evidence; they never classify an item.
|
||||
- **evidence is bounded.** Each entry keeps a trimmed copy of the line that
|
||||
classified it, and the number of entries is capped, so a looping uploader cannot
|
||||
turn the report into unbounded database rows.
|
||||
|
||||
Two grammars are supported. ``text-v1`` is the default human-readable log of the
|
||||
0.21/0.22 line, ``json-v1`` the structured log (``--log-type=json``) of 0.23/0.24.
|
||||
Both classify through one shared phrase table, so the vocabulary cannot drift
|
||||
between them:
|
||||
|
||||
```text
|
||||
text-v1 INFO uploaded /lib/rome/a.jpg
|
||||
INFO server has the same file /lib/rome/b.jpg
|
||||
ERROR error uploading /lib/rome/e.jpg: connection reset
|
||||
Uploaded 1, duplicates 1, errors 1
|
||||
|
||||
json-v1 {"level":"INFO","msg":"uploaded","file":"/lib/rome/a.jpg"}
|
||||
{"level":"INFO","msg":"report","counts":{"uploaded":1}}
|
||||
```
|
||||
|
||||
Nothing here touches the database; :mod:`photo_pipeline.services.upload_reports`
|
||||
turns a parse result into durable per-item outcomes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
PARSER_VERSION = 1
|
||||
|
||||
# Per-item outcomes. ``UNKNOWN`` is the safe default everywhere: it means the app
|
||||
# does not know what happened to that file and US05-04 must verify it.
|
||||
UPLOADED = "uploaded"
|
||||
UPGRADED = "upgraded"
|
||||
DUPLICATE = "duplicate"
|
||||
SKIPPED = "skipped"
|
||||
FAILED = "failed"
|
||||
UNKNOWN = "unknown"
|
||||
|
||||
OUTCOMES = (UPLOADED, UPGRADED, DUPLICATE, SKIPPED, FAILED, UNKNOWN)
|
||||
|
||||
# Longest/most specific phrases first: "server has an older file" also contains
|
||||
# "server has", and an error line about uploading also contains "upload".
|
||||
_PHRASES: tuple[tuple[str, str], ...] = (
|
||||
("server has the same file", DUPLICATE),
|
||||
("server has an older file", UPGRADED),
|
||||
("upgraded", UPGRADED),
|
||||
("duplicate", DUPLICATE),
|
||||
("discarded", SKIPPED),
|
||||
("skipped", SKIPPED),
|
||||
("error", FAILED),
|
||||
("failed", FAILED),
|
||||
("uploaded", UPLOADED),
|
||||
("upload", UPLOADED),
|
||||
)
|
||||
|
||||
# Summary line of the text grammar: "Uploaded 3, duplicates 1, errors 2".
|
||||
_SUMMARY_WORDS = {
|
||||
"uploaded": UPLOADED,
|
||||
"upgraded": UPGRADED,
|
||||
"duplicates": DUPLICATE,
|
||||
"duplicate": DUPLICATE,
|
||||
"skipped": SKIPPED,
|
||||
"discarded": SKIPPED,
|
||||
"errors": FAILED,
|
||||
"error": FAILED,
|
||||
}
|
||||
_SUMMARY_PAIR = re.compile(r"([A-Za-z]+)\s+(\d+)")
|
||||
# A path is anything that looks absolute, up to an explanatory ": reason" tail.
|
||||
_PATH = re.compile(r"(/[^\s:][^:]*?)(?=:|$)")
|
||||
|
||||
MAX_EVIDENCE_CHARS = 300
|
||||
# The report file is already byte-capped; this caps the rows it can produce.
|
||||
MAX_ENTRIES = 20_000
|
||||
|
||||
|
||||
def parser_for(uploader_version: str | None) -> str | None:
|
||||
"""Adapter name for a recorded uploader version, or ``None`` when unsupported.
|
||||
|
||||
Unsupported is not an error — it is the honest answer that this build's output
|
||||
format was never pinned, and it makes the whole report uncertain.
|
||||
"""
|
||||
match = re.search(r"(\d+)\.(\d+)", uploader_version or "")
|
||||
if match is None:
|
||||
return None
|
||||
return _SUPPORTED.get(f"{match.group(1)}.{match.group(2)}")
|
||||
|
||||
|
||||
def parse(report: str, uploader_version: str | None) -> dict:
|
||||
"""Classify a raw report.
|
||||
|
||||
Returns ``{"parser", "parser_version", "supported", "entries", "counts",
|
||||
"unparsed", "entries_truncated"}`` where ``entries`` is a list of
|
||||
``{"path", "outcome", "evidence"}`` and ``counts`` is the uploader's own
|
||||
summary when it printed one (``None`` otherwise, never invented).
|
||||
"""
|
||||
parser = parser_for(uploader_version)
|
||||
result = {
|
||||
"parser": parser,
|
||||
"parser_version": PARSER_VERSION,
|
||||
"supported": parser is not None,
|
||||
"entries": [],
|
||||
"counts": None,
|
||||
"unparsed": 0,
|
||||
"entries_truncated": False,
|
||||
}
|
||||
if parser is None:
|
||||
return result
|
||||
read_line = _text_line if parser == "text-v1" else _json_line
|
||||
for raw in report.splitlines():
|
||||
line = raw.strip()
|
||||
if not line:
|
||||
continue
|
||||
path, outcome, counts = read_line(line)
|
||||
if counts is not None:
|
||||
# Later summaries win: the uploader prints its totals once, at the end.
|
||||
result["counts"] = counts
|
||||
continue
|
||||
if path is None or outcome is None:
|
||||
result["unparsed"] += 1
|
||||
continue
|
||||
if len(result["entries"]) >= MAX_ENTRIES:
|
||||
result["entries_truncated"] = True
|
||||
continue
|
||||
result["entries"].append(
|
||||
{"path": path, "outcome": outcome, "evidence": line[:MAX_EVIDENCE_CHARS]}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _classify(text: str) -> str | None:
|
||||
lowered = text.lower()
|
||||
for phrase, outcome in _PHRASES:
|
||||
if phrase in lowered:
|
||||
return outcome
|
||||
return None
|
||||
|
||||
|
||||
def _summary(text: str) -> dict[str, int] | None:
|
||||
"""Totals from a summary line, or ``None`` when the line is not one."""
|
||||
counts: dict[str, int] = {}
|
||||
for word, number in _SUMMARY_PAIR.findall(text):
|
||||
outcome = _SUMMARY_WORDS.get(word.lower())
|
||||
if outcome is None:
|
||||
return None # an unknown noun means this is not the summary grammar
|
||||
counts[outcome] = counts.get(outcome, 0) + int(number)
|
||||
return counts or None
|
||||
|
||||
|
||||
def _text_line(line: str) -> tuple[str | None, str | None, dict | None]:
|
||||
path_match = _PATH.search(line)
|
||||
if path_match is None:
|
||||
return None, None, _summary(line)
|
||||
path = path_match.group(1).strip()
|
||||
# Classify from the words around the path, never from the path itself: an
|
||||
# album called "errors" must not turn an upload into a failure.
|
||||
context = line.replace(path, " ")
|
||||
return path, _classify(context), None
|
||||
|
||||
|
||||
def _json_line(line: str) -> tuple[str | None, str | None, dict | None]:
|
||||
try:
|
||||
record = json.loads(line)
|
||||
except ValueError:
|
||||
return None, None, None
|
||||
if not isinstance(record, dict):
|
||||
return None, None, None
|
||||
counts = record.get("counts")
|
||||
if isinstance(counts, dict):
|
||||
totals = {
|
||||
_SUMMARY_WORDS[key.lower()]: int(value)
|
||||
for key, value in counts.items()
|
||||
if key.lower() in _SUMMARY_WORDS and isinstance(value, int)
|
||||
}
|
||||
return None, None, totals or None
|
||||
path = record.get("file")
|
||||
message = record.get("msg")
|
||||
if not isinstance(path, str) or not isinstance(message, str):
|
||||
return None, None, None
|
||||
return path, _classify(message), None
|
||||
|
||||
|
||||
# Version family → adapter. Pinning this is the point: a build outside the list is
|
||||
# uncertain by construction (concept §15 "pin and record supported immich-go
|
||||
# versions").
|
||||
_SUPPORTED = {
|
||||
"0.21": "text-v1",
|
||||
"0.22": "text-v1",
|
||||
"0.23": "json-v1",
|
||||
"0.24": "json-v1",
|
||||
}
|
||||
SUPPORTED_VERSIONS = tuple(sorted(_SUPPORTED))
|
||||
@@ -1,10 +1,12 @@
|
||||
"""Domain job handlers: safety scoring, content analysis, uploads (US02-06, US05-02).
|
||||
"""Domain job handlers: safety scoring, content analysis, uploads, archive
|
||||
transfers, restores (US02-06, US05-02, US06-02, US06-04).
|
||||
|
||||
Importing this module registers the ``safety_score``, ``analysis``, and
|
||||
``upload_batch`` job types so the generic worker can run them per item. Each handler
|
||||
delegates to its service, which owns the real work and the privacy gate. Handlers
|
||||
are idempotent: re-scoring or re-analyzing one asset is safe after an interrupted
|
||||
attempt, and an upload batch refuses to re-run an attempt whose outcome is unknown.
|
||||
Importing this module registers the ``safety_score``, ``analysis``,
|
||||
``upload_batch``, ``archive_plan``, and ``restore_plan`` job types so the generic
|
||||
worker can run them per item. Each handler delegates to its service, which owns the real work and the
|
||||
privacy gate. Handlers are idempotent: re-scoring or re-analyzing one asset is safe
|
||||
after an interrupted attempt, an upload batch refuses to re-run an attempt whose
|
||||
outcome is unknown, and an archive plan skips items it already completed.
|
||||
|
||||
Providers/models are the service defaults here (real NsfwModel / vision provider);
|
||||
tests exercise the services directly with injected fakes rather than the worker.
|
||||
@@ -17,10 +19,14 @@ from photo_pipeline.jobs.handlers import Cancelled, JobContext, register
|
||||
SAFETY_SCORE = "safety_score"
|
||||
ANALYSIS = "analysis"
|
||||
UPLOAD_BATCH = "upload_batch"
|
||||
ARCHIVE_PLAN = "archive_plan"
|
||||
RESTORE_PLAN = "restore_plan"
|
||||
# Both mutate the library's metadata/derived state; one at a time (concept §one job).
|
||||
LIBRARY_WRITE_LOCK = "library_write"
|
||||
# The uploader lane: one album batch at a time (concept §16).
|
||||
UPLOAD_LOCK = "upload"
|
||||
# The archiver lane: one archive/restore plan at a time (concept §16).
|
||||
ARCHIVE_LOCK = "archive"
|
||||
|
||||
|
||||
def _safety_score_item(asset_id: str, ctx: JobContext) -> None:
|
||||
@@ -51,6 +57,36 @@ def _upload_batch_item(batch_id: str, ctx: JobContext) -> None:
|
||||
raise RuntimeError(f"upload batch {batch_id} is {batch['state']}: {batch['error_code']}")
|
||||
|
||||
|
||||
def _archive_plan_item(plan_id: str, ctx: JobContext) -> None:
|
||||
"""One item = one archive plan. Item-level failures stay in the journal (the
|
||||
source is then still there); only an unusable plan fails the job."""
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
||||
|
||||
config = ctx.config if ctx.config is not None else Config.from_env()
|
||||
result = ArchiveTransferService(ctx.session_factory, config=config).apply(
|
||||
plan_id, worker_id=ctx.worker_id
|
||||
)
|
||||
if result["failed"]:
|
||||
raise RuntimeError(f"archive plan {plan_id}: {result['failed']} item(s) failed")
|
||||
|
||||
|
||||
def _restore_plan_item(plan_id: str, ctx: JobContext) -> None:
|
||||
"""One item = one restore plan. A restore removes nothing, so an item failure
|
||||
simply leaves that asset archived (US06-04)."""
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.services.restores import RestoreService
|
||||
|
||||
config = ctx.config if ctx.config is not None else Config.from_env()
|
||||
result = RestoreService(ctx.session_factory, config=config).apply(
|
||||
plan_id, worker_id=ctx.worker_id
|
||||
)
|
||||
if result["failed"]:
|
||||
raise RuntimeError(f"restore plan {plan_id}: {result['failed']} item(s) failed")
|
||||
|
||||
|
||||
register(SAFETY_SCORE, _safety_score_item)
|
||||
register(ANALYSIS, _analysis_item)
|
||||
register(UPLOAD_BATCH, _upload_batch_item)
|
||||
register(ARCHIVE_PLAN, _archive_plan_item)
|
||||
register(RESTORE_PLAN, _restore_plan_item)
|
||||
|
||||
@@ -96,6 +96,16 @@ class Worker:
|
||||
return
|
||||
try:
|
||||
if cancelled or snapshot["state"] == JobState.CANCELLING:
|
||||
# A handler may stop on its own — an upload batch cancelled through
|
||||
# its own API never touches the job — so the job can still be
|
||||
# ``running`` here. Record the request before the outcome: a stop is
|
||||
# always observable as cancelling → cancelled, and ``running ->
|
||||
# cancelled`` is not a legal jump. Without this hop the transition
|
||||
# is rejected and the job keeps its lock forever.
|
||||
if snapshot["state"] == JobState.RUNNING:
|
||||
self.service.transition(
|
||||
job_id, JobState.CANCELLING, worker_id=self.worker_id, fencing_token=token
|
||||
)
|
||||
self.service.transition(
|
||||
job_id, JobState.CANCELLED, worker_id=self.worker_id, fencing_token=token
|
||||
)
|
||||
|
||||
@@ -5,6 +5,7 @@ Alembic environment relies on.
|
||||
"""
|
||||
|
||||
from photo_pipeline.models.albums import AlbumProposal
|
||||
from photo_pipeline.models.archives import ArchiveLocation, ArchiveOperation, ArchivePlan
|
||||
from photo_pipeline.models.assets import Asset, AssetPath
|
||||
from photo_pipeline.models.duplicates import (
|
||||
DuplicateCluster,
|
||||
@@ -14,11 +15,14 @@ from photo_pipeline.models.duplicates import (
|
||||
from photo_pipeline.models.jobs import Job, JobEvent, JobItem
|
||||
from photo_pipeline.models.renames import RenameOperation, RenamePlan
|
||||
from photo_pipeline.models.thumbnails import Thumbnail
|
||||
from photo_pipeline.models.uploads import UploadBatch, UploadItem
|
||||
from photo_pipeline.models.uploads import UploadBatch, UploadItem, UploadVerification
|
||||
from photo_pipeline.models.workflow import AnalysisResult, SafetyReview
|
||||
|
||||
__all__ = [
|
||||
"AlbumProposal",
|
||||
"ArchiveLocation",
|
||||
"ArchiveOperation",
|
||||
"ArchivePlan",
|
||||
"Asset",
|
||||
"AssetPath",
|
||||
"DuplicateCluster",
|
||||
@@ -32,6 +36,7 @@ __all__ = [
|
||||
"Thumbnail",
|
||||
"UploadBatch",
|
||||
"UploadItem",
|
||||
"UploadVerification",
|
||||
"SafetyReview",
|
||||
"AnalysisResult",
|
||||
]
|
||||
|
||||
127
photo_pipeline/models/archives.py
Normal file
127
photo_pipeline/models/archives.py
Normal file
@@ -0,0 +1,127 @@
|
||||
"""Archive location, plan, and transfer-journal persistence (US06-01, US06-02).
|
||||
|
||||
An archive location is a *medium*, not a path. External disks get mounted at
|
||||
different mountpoints, and a different disk can be mounted at the same one, so a
|
||||
recorded root alone can never prove "these bytes went to that volume". Each
|
||||
location therefore owns a marker file written onto the medium itself; its
|
||||
``media_id`` is the stable identity, and the root is only where it was last seen.
|
||||
|
||||
``capabilities`` and ``state`` are the last probe result, kept so the UI can list
|
||||
locations without touching a sleeping disk. Preflight always re-probes — a stored
|
||||
state is a hint, never evidence.
|
||||
|
||||
An ``ArchivePlan`` is one approved preflight turned into durable work, and each
|
||||
``ArchiveOperation`` is one file's crash-safe journal row (US06-02). The row records
|
||||
what the transfer *intends* to do before it does it — source, destination, expected
|
||||
hash — because after a crash that intent plus the files on disk is the only evidence
|
||||
available for deciding whether an original may be removed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from photo_pipeline.db import Base
|
||||
|
||||
|
||||
class ArchiveLocation(Base):
|
||||
__tablename__ = "archive_locations"
|
||||
|
||||
id: Mapped[str] = mapped_column(String, primary_key=True)
|
||||
name: Mapped[str] = mapped_column(String, nullable=False, unique=True)
|
||||
root: Mapped[str] = mapped_column(String, nullable=False)
|
||||
# Written into the marker file on the medium; proves the right volume is mounted.
|
||||
media_id: Mapped[str] = mapped_column(String, nullable=False, unique=True)
|
||||
capabilities: Mapped[str | None] = mapped_column(String) # JSON, last probe
|
||||
# online | offline | wrong_volume | unwritable — the last probe's verdict.
|
||||
state: Mapped[str] = mapped_column(String, nullable=False, default="offline")
|
||||
last_seen_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
|
||||
)
|
||||
|
||||
|
||||
class ArchivePlan(Base):
|
||||
__tablename__ = "archive_plans"
|
||||
|
||||
id: Mapped[str] = mapped_column(String, primary_key=True)
|
||||
location_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("archive_locations.id"), nullable=False, index=True
|
||||
)
|
||||
# The preflight token this plan was approved against; re-verified before apply.
|
||||
token: Mapped[str] = mapped_column(String, nullable=False)
|
||||
albums: Mapped[str | None] = mapped_column(String) # JSON array
|
||||
# archive | restore — the same journal read in the opposite direction (US06-04).
|
||||
direction: Mapped[str] = mapped_column(String, nullable=False, default="archive")
|
||||
|
||||
# planned | applying | complete | failed
|
||||
state: Mapped[str] = mapped_column(String, nullable=False, default="planned")
|
||||
schema_version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
asset_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
byte_size: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
# Bumped on every claim and used as the fencing token, so a superseded attempt
|
||||
# cannot commit.
|
||||
version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
worker_id: Mapped[str | None] = mapped_column(String)
|
||||
completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
|
||||
)
|
||||
|
||||
|
||||
class ArchiveOperation(Base):
|
||||
__tablename__ = "archive_operations"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("plan_id", "sequence", name="uq_archive_operations_plan_sequence"),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String, primary_key=True)
|
||||
plan_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("archive_plans.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
sequence: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
album: Mapped[str] = mapped_column(String, nullable=False)
|
||||
asset_id: Mapped[str] = mapped_column(ForeignKey("assets.id"), nullable=False, index=True)
|
||||
|
||||
# archive: library → medium. restore: medium → library (US06-04). ``source_path``
|
||||
# and ``destination_path`` always mean "from" and "to" for this direction.
|
||||
direction: Mapped[str] = mapped_column(String, nullable=False, default="archive")
|
||||
source_path: Mapped[str] = mapped_column(String, nullable=False)
|
||||
destination_path: Mapped[str] = mapped_column(String, nullable=False)
|
||||
# Relative to the location root, because the medium can be mounted elsewhere.
|
||||
archive_path: Mapped[str] = mapped_column(String, nullable=False)
|
||||
expected_sha256: Mapped[str] = mapped_column(String, nullable=False)
|
||||
byte_size: Mapped[int | None] = mapped_column(Integer)
|
||||
same_filesystem: Mapped[bool | None] = mapped_column(Boolean)
|
||||
|
||||
# planned | transferring | verified | removing | complete | failed
|
||||
journal_state: Mapped[str] = mapped_column(String, nullable=False, default="planned")
|
||||
attempt_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
fencing_token: Mapped[int | None] = mapped_column(Integer)
|
||||
worker_id: Mapped[str | None] = mapped_column(String)
|
||||
verified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
removed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
error_code: Mapped[str | None] = mapped_column(String)
|
||||
error_message: Mapped[str | None] = mapped_column(String)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
|
||||
)
|
||||
@@ -35,7 +35,18 @@ class Asset(Base):
|
||||
|
||||
discovered_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
missing_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# active | archiving | archived_online | archived_offline | restoring |
|
||||
# missing_unexpected (concept §9). ``current_path`` is NULL once archived; the
|
||||
# original is then explained by the location plus its relative archive path.
|
||||
availability_state: Mapped[str] = mapped_column(String, nullable=False, default="active")
|
||||
# Not a declared foreign key: SQLite cannot add one to an existing table, so the
|
||||
# link is written and read by the archive service (US06-02).
|
||||
archive_location_id: Mapped[str | None] = mapped_column(String)
|
||||
archive_path: Mapped[str | None] = mapped_column(String)
|
||||
# Set when the archived copy was proven to hold bytes other than the recorded
|
||||
# ones (US06-04). Restore refuses such an asset instead of accepting a different
|
||||
# file; cleared as soon as a verification matches again.
|
||||
archive_divergent_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# Duplicate canonical link: NULL when the asset is itself canonical or undecided.
|
||||
canonical_asset_id: Mapped[str | None] = mapped_column(ForeignKey("assets.id"))
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, Integer, String, func
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from photo_pipeline.db import Base
|
||||
@@ -29,6 +29,9 @@ class Thumbnail(Base):
|
||||
width: Mapped[int | None] = mapped_column(Integer)
|
||||
height: Mapped[int | None] = mapped_column(Integer)
|
||||
format: Mapped[str | None] = mapped_column(String)
|
||||
# Durable comparison evidence for an archived asset: never evicted by the LRU
|
||||
# quota, because the original may be on a medium that is no longer reachable.
|
||||
protected: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
|
||||
@@ -57,6 +57,23 @@ class UploadBatch(Base):
|
||||
report_bytes: Mapped[int | None] = mapped_column(Integer)
|
||||
report_truncated: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
exit_code: Mapped[int | None] = mapped_column(Integer)
|
||||
|
||||
# Parsed report evidence (US05-03). ``parser`` is NULL when the uploader's
|
||||
# version has no pinned grammar; ``outcome_state`` is then
|
||||
# ``requires_verification`` regardless of how the process exited.
|
||||
parser: Mapped[str | None] = mapped_column(String)
|
||||
parser_version: Mapped[int | None] = mapped_column(Integer)
|
||||
parsed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# verified | requires_verification; NULL until a report has been parsed.
|
||||
outcome_state: Mapped[str | None] = mapped_column(String)
|
||||
outcome_counts: Mapped[str | None] = mapped_column(String) # JSON, from the items
|
||||
report_counts: Mapped[str | None] = mapped_column(String) # JSON, uploader's own
|
||||
|
||||
# Verification (US05-04). ``stale_bytes`` means at least one uploaded file has
|
||||
# been edited since: the batch carries a visible warning and cannot be re-run.
|
||||
verified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
stale_bytes: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
|
||||
error_code: Mapped[str | None] = mapped_column(String)
|
||||
error_message: Mapped[str | None] = mapped_column(String)
|
||||
|
||||
@@ -81,10 +98,54 @@ class UploadItem(Base):
|
||||
# Hashes of the bytes as they were when the batch was created.
|
||||
sha256: Mapped[str | None] = mapped_column(String)
|
||||
sha1: Mapped[str | None] = mapped_column(String)
|
||||
# pending | sent | failed — the per-asset upload *result* is parsed from the
|
||||
# report in US05-03 and verified in US05-04; ``sent`` only means the batch
|
||||
# process exited successfully.
|
||||
# pending | sent | failed — what the batch *process* did with this item;
|
||||
# ``sent`` only means the uploader exited successfully.
|
||||
state: Mapped[str] = mapped_column(String, nullable=False, default="pending")
|
||||
# What the uploader's report says happened (US05-03): uploaded | upgraded |
|
||||
# duplicate | skipped | failed | unknown. NULL before the report is parsed;
|
||||
# ``unknown`` whenever the report does not classify this file — never success.
|
||||
outcome: Mapped[str | None] = mapped_column(String)
|
||||
evidence: Mapped[str | None] = mapped_column(String) # the bounded report line
|
||||
outcome_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
# Verification against the server (US05-04): present | absent | inconclusive |
|
||||
# manual. NULL until the item has been verified; ``inconclusive`` whenever the
|
||||
# server could not answer — which is never treated as success.
|
||||
verification: Mapped[str | None] = mapped_column(String)
|
||||
verified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# The bytes on disk at verification time, and whether they still are the bytes
|
||||
# this batch uploaded.
|
||||
observed_sha256: Mapped[str | None] = mapped_column(String)
|
||||
changed_after_upload: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
|
||||
)
|
||||
|
||||
|
||||
class UploadVerification(Base):
|
||||
"""Append-only evidence for every verification and manual resolution (US05-04).
|
||||
|
||||
The item row is a projection of the latest answer; this table is the history
|
||||
that answers "who decided this, on what evidence, and when?". Rows are never
|
||||
updated or deleted, so a manual resolution can always be told apart from
|
||||
server evidence.
|
||||
"""
|
||||
|
||||
__tablename__ = "upload_verifications"
|
||||
|
||||
id: Mapped[str] = mapped_column(String, primary_key=True)
|
||||
batch_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("upload_batches.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
asset_id: Mapped[str] = mapped_column(String, nullable=False)
|
||||
action: Mapped[str] = mapped_column(String, nullable=False) # verify | resolve
|
||||
source: Mapped[str] = mapped_column(String, nullable=False) # immich_api | operator
|
||||
# present | absent | inconclusive for a verify; the recorded outcome for a resolve.
|
||||
result: Mapped[str] = mapped_column(String, nullable=False)
|
||||
outcome: Mapped[str | None] = mapped_column(String)
|
||||
evidence: Mapped[str] = mapped_column(String, nullable=False)
|
||||
actor: Mapped[str | None] = mapped_column(String)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
|
||||
359
photo_pipeline/services/archive_journal.py
Normal file
359
photo_pipeline/services/archive_journal.py
Normal file
@@ -0,0 +1,359 @@
|
||||
"""Archive transfer journal — the durable record of every per-file transition
|
||||
(US06-02).
|
||||
|
||||
Archiving is the only stage that deletes an original, so the journal exists to make
|
||||
one question answerable after any crash: *may this source file be removed?* Intent
|
||||
is written before the mutation it describes, and the recorded state plus the real
|
||||
files on disk are the sole basis for answering it later. This module owns the state
|
||||
machine, the durable writes, and the evidence table; it never touches a photo
|
||||
(:mod:`photo_pipeline.services.archive_transfer` does).
|
||||
|
||||
Per-item state machine (concept §9 "Transfer and removal semantics"):
|
||||
|
||||
```
|
||||
planned → transferring → verified → removing → complete
|
||||
↘ ↘ ↘ failed
|
||||
```
|
||||
|
||||
A restore (US06-04) uses the same rows with ``direction='restore'``: it copies from
|
||||
the medium back into the library and removes nothing, so it goes ``verified →
|
||||
complete`` directly. ``source_path``/``destination_path`` always mean "from"/"to",
|
||||
which is why the evidence table below needs no direction of its own.
|
||||
|
||||
- ``transferring`` — intent recorded; a temporary copy may exist, the destination
|
||||
may or may not have been published. Nothing has been removed.
|
||||
- ``verified`` — the archived bytes exist at their final path, hash exactly as
|
||||
recorded, and the manifest entry is durable. Only from here may a source go.
|
||||
- ``removing`` — the source removal is committed to; the source may already be
|
||||
gone while the database still points at it.
|
||||
- ``complete`` — source absent, database updated, availability recorded.
|
||||
|
||||
``classify`` labels each incomplete item from the journal plus disk evidence:
|
||||
|
||||
- ``resumable`` — nothing was published; the source is intact, so applying again is
|
||||
safe.
|
||||
- ``forward`` — the archived copy exists and matches its recorded hash, so the
|
||||
remaining steps (manifest, removal, bookkeeping) can be finished deterministically.
|
||||
- ``manual`` — the evidence contradicts the journal (missing archive copy, wrong
|
||||
bytes, source and archive both gone). Nothing is guessed and nothing is removed;
|
||||
the item blocks unrelated mutations until a human decides.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.models import ArchiveOperation, ArchivePlan
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
|
||||
|
||||
class ArchiveState:
|
||||
PLANNED = "planned"
|
||||
TRANSFERRING = "transferring"
|
||||
VERIFIED = "verified"
|
||||
REMOVING = "removing"
|
||||
COMPLETE = "complete"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
ALLOWED_TRANSITIONS = {
|
||||
ArchiveState.PLANNED: {ArchiveState.TRANSFERRING, ArchiveState.FAILED},
|
||||
# From `transferring` the outcome is unknown until evidence is gathered, so it
|
||||
# may resolve forward, back to planned (proven nothing was published), or fail.
|
||||
ArchiveState.TRANSFERRING: {
|
||||
ArchiveState.VERIFIED,
|
||||
ArchiveState.PLANNED,
|
||||
ArchiveState.FAILED,
|
||||
},
|
||||
ArchiveState.VERIFIED: {ArchiveState.REMOVING, ArchiveState.FAILED},
|
||||
# No path back: once the source may be gone, only finishing is safe.
|
||||
ArchiveState.REMOVING: {ArchiveState.COMPLETE, ArchiveState.FAILED},
|
||||
ArchiveState.COMPLETE: set(),
|
||||
# A retry re-enters `transferring`, which rechecks every precondition from
|
||||
# scratch; recovery may also reset a failed item to `planned`.
|
||||
ArchiveState.FAILED: {ArchiveState.PLANNED, ArchiveState.TRANSFERRING},
|
||||
}
|
||||
|
||||
# A restore removes nothing, so it has no ``removing`` step: a verified published
|
||||
# copy is the whole job (US06-04). Keeping this as a separate table means the
|
||||
# archive direction still cannot reach ``complete`` without going through removal.
|
||||
RESTORE_TRANSITIONS = {
|
||||
**ALLOWED_TRANSITIONS,
|
||||
ArchiveState.VERIFIED: {ArchiveState.COMPLETE, ArchiveState.FAILED},
|
||||
}
|
||||
|
||||
TERMINAL_STATES = frozenset({ArchiveState.COMPLETE})
|
||||
# States where this item may already have touched the filesystem.
|
||||
UNSAFE_STATES = frozenset({ArchiveState.TRANSFERRING, ArchiveState.VERIFIED, ArchiveState.REMOVING})
|
||||
|
||||
# Which way the bytes move. Same rows, same evidence table, opposite direction.
|
||||
ARCHIVE = "archive"
|
||||
RESTORE = "restore"
|
||||
|
||||
RESUMABLE = "resumable"
|
||||
FORWARD = "forward"
|
||||
MANUAL = "manual"
|
||||
|
||||
|
||||
class JournalError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class InvalidTransition(JournalError):
|
||||
pass
|
||||
|
||||
|
||||
class JournalConflict(JournalError):
|
||||
"""Fencing check failed; a newer owner has taken over this operation."""
|
||||
|
||||
|
||||
def can_transition(current: str, target: str, direction: str = ARCHIVE) -> bool:
|
||||
table = RESTORE_TRANSITIONS if direction == RESTORE else ALLOWED_TRANSITIONS
|
||||
return target in table.get(current, set())
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class ArchiveJournal:
|
||||
def __init__(self, session_factory: sessionmaker) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
# ── intent ────────────────────────────────────────────────────────────────
|
||||
|
||||
def begin(self, operation_id: str, *, worker_id: str, fencing_token: int) -> dict:
|
||||
"""Record the intent to transfer **before** touching the filesystem."""
|
||||
with self._session_factory() as session:
|
||||
row = self._require(session, operation_id)
|
||||
if row.fencing_token is not None and fencing_token < row.fencing_token:
|
||||
raise JournalConflict(
|
||||
f"stale fencing token {fencing_token} (current {row.fencing_token})"
|
||||
)
|
||||
if row.journal_state in TERMINAL_STATES:
|
||||
raise InvalidTransition(f"{row.journal_state} is terminal")
|
||||
if row.journal_state != ArchiveState.TRANSFERRING and not can_transition(
|
||||
row.journal_state, ArchiveState.TRANSFERRING, row.direction
|
||||
):
|
||||
raise InvalidTransition(f"{row.journal_state} -> {ArchiveState.TRANSFERRING}")
|
||||
if row.journal_state != ArchiveState.TRANSFERRING:
|
||||
row.attempt_count += 1
|
||||
row.journal_state = ArchiveState.TRANSFERRING
|
||||
row.worker_id = worker_id
|
||||
row.fencing_token = fencing_token
|
||||
row.error_code = row.error_message = None
|
||||
row.updated_at = _now()
|
||||
session.commit()
|
||||
return _operation_dict(row)
|
||||
|
||||
# ── transitions ───────────────────────────────────────────────────────────
|
||||
|
||||
def transition(
|
||||
self,
|
||||
operation_id: str,
|
||||
target: str,
|
||||
*,
|
||||
fencing_token: int | None = None,
|
||||
error: tuple[str, str] | None = None,
|
||||
same_filesystem: bool | None = None,
|
||||
) -> dict:
|
||||
"""Move one operation to ``target``, enforcing the state machine.
|
||||
|
||||
Re-entering the state an operation already holds is a no-op, which is what
|
||||
makes recovery idempotent across repeated restarts.
|
||||
"""
|
||||
with self._session_factory() as session:
|
||||
row = self._require(session, operation_id)
|
||||
if fencing_token is not None and row.fencing_token is not None:
|
||||
if fencing_token < row.fencing_token:
|
||||
raise JournalConflict(
|
||||
f"stale fencing token {fencing_token} (current {row.fencing_token})"
|
||||
)
|
||||
if same_filesystem is not None:
|
||||
row.same_filesystem = same_filesystem
|
||||
if row.journal_state == target:
|
||||
session.commit()
|
||||
return _operation_dict(row) # idempotent
|
||||
if not can_transition(row.journal_state, target, row.direction):
|
||||
raise InvalidTransition(f"{row.journal_state} -> {target}")
|
||||
|
||||
row.journal_state = target
|
||||
row.updated_at = _now()
|
||||
if target == ArchiveState.VERIFIED:
|
||||
row.verified_at = _now()
|
||||
if target == ArchiveState.COMPLETE:
|
||||
row.removed_at = _now()
|
||||
if error:
|
||||
row.error_code, row.error_message = error[0], error[1][:500]
|
||||
elif target != ArchiveState.FAILED:
|
||||
row.error_code = row.error_message = None
|
||||
session.commit()
|
||||
return _operation_dict(row)
|
||||
|
||||
# ── reads ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def get(self, operation_id: str) -> dict | None:
|
||||
with self._session_factory() as session:
|
||||
row = session.get(ArchiveOperation, operation_id)
|
||||
return _operation_dict(row) if row else None
|
||||
|
||||
def operations(self, plan_id: str) -> list[dict]:
|
||||
with self._session_factory() as session:
|
||||
rows = session.scalars(
|
||||
select(ArchiveOperation)
|
||||
.where(ArchiveOperation.plan_id == plan_id)
|
||||
.order_by(ArchiveOperation.sequence)
|
||||
)
|
||||
return [_operation_dict(row) for row in rows]
|
||||
|
||||
def incomplete(self, *, direction: str | None = None) -> list[dict]:
|
||||
"""Every operation left in a non-terminal, non-planned state — the work a
|
||||
restart has to reason about. Without ``direction`` this spans archives and
|
||||
restores, because either one half-done blocks the other."""
|
||||
with self._session_factory() as session:
|
||||
stmt = select(ArchiveOperation).where(
|
||||
ArchiveOperation.journal_state.not_in([*TERMINAL_STATES, ArchiveState.PLANNED])
|
||||
)
|
||||
if direction is not None:
|
||||
stmt = stmt.where(ArchiveOperation.direction == direction)
|
||||
rows = session.scalars(
|
||||
stmt.order_by(ArchiveOperation.plan_id, ArchiveOperation.sequence)
|
||||
)
|
||||
return [_operation_dict(row) for row in rows]
|
||||
|
||||
# ── startup classification ────────────────────────────────────────────────
|
||||
|
||||
def classify(self, operation_id: str) -> dict:
|
||||
"""Classify one incomplete operation from the journal plus disk evidence.
|
||||
|
||||
Hashes the archived copy when one exists: "a file is at the destination" is
|
||||
not evidence that the *right* bytes are, and only the right bytes justify
|
||||
removing an original. Never mutates anything.
|
||||
"""
|
||||
row = self.get(operation_id)
|
||||
if row is None:
|
||||
raise JournalError(f"unknown archive operation {operation_id!r}")
|
||||
source = Path(row["source_path"])
|
||||
destination = Path(row["destination_path"])
|
||||
source_exists = source.exists()
|
||||
destination_exists = destination.exists()
|
||||
destination_matches = (
|
||||
destination_exists and sha256_file(destination) == row["expected_sha256"]
|
||||
)
|
||||
|
||||
classification, reason = _classify(
|
||||
row["journal_state"], source_exists, destination_exists, destination_matches
|
||||
)
|
||||
return {
|
||||
"operation_id": operation_id,
|
||||
"plan_id": row["plan_id"],
|
||||
"direction": row["direction"],
|
||||
"album": row["album"],
|
||||
"asset_id": row["asset_id"],
|
||||
"source_path": row["source_path"],
|
||||
"destination_path": row["destination_path"],
|
||||
"journal_state": row["journal_state"],
|
||||
"classification": classification,
|
||||
"reason": reason,
|
||||
"source_exists": source_exists,
|
||||
"destination_exists": destination_exists,
|
||||
"destination_matches": destination_matches,
|
||||
}
|
||||
|
||||
def classify_all(self, *, direction: str | None = None) -> list[dict]:
|
||||
return [self.classify(row["id"]) for row in self.incomplete(direction=direction)]
|
||||
|
||||
def blocks_mutation(self) -> bool:
|
||||
"""True when any item may have the library half-archived."""
|
||||
return any(row["journal_state"] in UNSAFE_STATES for row in self.incomplete())
|
||||
|
||||
# ── plan-level ────────────────────────────────────────────────────────────
|
||||
|
||||
def plan_state(self, plan_id: str) -> str:
|
||||
"""Derive the plan's state from its items, so the summary can never disagree
|
||||
with the journal."""
|
||||
states = {row["journal_state"] for row in self.operations(plan_id)}
|
||||
if not states:
|
||||
return "planned"
|
||||
if states <= {ArchiveState.COMPLETE}:
|
||||
return "complete"
|
||||
if states & {ArchiveState.FAILED}:
|
||||
return "failed"
|
||||
if states & UNSAFE_STATES:
|
||||
return "applying"
|
||||
return "planned"
|
||||
|
||||
def sync_plan_state(self, plan_id: str) -> str:
|
||||
state = self.plan_state(plan_id)
|
||||
with self._session_factory() as session:
|
||||
plan = session.get(ArchivePlan, plan_id)
|
||||
if plan is None:
|
||||
raise JournalError(f"unknown archive plan {plan_id!r}")
|
||||
if plan.state != state:
|
||||
plan.state = state
|
||||
plan.version += 1
|
||||
plan.updated_at = _now()
|
||||
if state == "complete" and plan.completed_at is None:
|
||||
plan.completed_at = _now()
|
||||
session.commit()
|
||||
return state
|
||||
|
||||
@staticmethod
|
||||
def _require(session, operation_id: str) -> ArchiveOperation:
|
||||
row = session.get(ArchiveOperation, operation_id)
|
||||
if row is None:
|
||||
raise JournalError(f"unknown archive operation {operation_id!r}")
|
||||
return row
|
||||
|
||||
|
||||
def _classify(
|
||||
state: str, source_exists: bool, destination_exists: bool, destination_matches: bool
|
||||
) -> tuple[str, str]:
|
||||
"""The evidence table. Kept a pure function so every combination is testable."""
|
||||
if destination_exists and not destination_matches and state != ArchiveState.PLANNED:
|
||||
# Someone else's file, or a partial/edited copy: never overwrite it, and
|
||||
# never treat it as the durable archive that justifies a deletion.
|
||||
return MANUAL, "the archived path holds bytes that are not the recorded ones"
|
||||
|
||||
if state in (ArchiveState.TRANSFERRING, ArchiveState.FAILED):
|
||||
if destination_matches:
|
||||
return FORWARD, "the archived copy is durable; finish the remaining steps"
|
||||
if source_exists:
|
||||
return RESUMABLE, "nothing was published; the source is intact"
|
||||
return MANUAL, "neither the source nor a verified archive copy is present"
|
||||
|
||||
if state in (ArchiveState.VERIFIED, ArchiveState.REMOVING):
|
||||
if destination_matches:
|
||||
return FORWARD, "the archived copy is durable; finish the remaining steps"
|
||||
return MANUAL, f"journal says {state} but the archived copy is missing"
|
||||
|
||||
return MANUAL, f"unhandled journal state {state}"
|
||||
|
||||
|
||||
def _operation_dict(row: ArchiveOperation) -> dict:
|
||||
return {
|
||||
"id": row.id,
|
||||
"plan_id": row.plan_id,
|
||||
"direction": row.direction,
|
||||
"sequence": row.sequence,
|
||||
"album": row.album,
|
||||
"asset_id": row.asset_id,
|
||||
"source_path": row.source_path,
|
||||
"destination_path": row.destination_path,
|
||||
"archive_path": row.archive_path,
|
||||
"expected_sha256": row.expected_sha256,
|
||||
"byte_size": row.byte_size,
|
||||
"same_filesystem": row.same_filesystem,
|
||||
"journal_state": row.journal_state,
|
||||
"attempt_count": row.attempt_count,
|
||||
"fencing_token": row.fencing_token,
|
||||
"worker_id": row.worker_id,
|
||||
"verified_at": row.verified_at.isoformat() if row.verified_at else None,
|
||||
"removed_at": row.removed_at.isoformat() if row.removed_at else None,
|
||||
"error_code": row.error_code,
|
||||
"error_message": row.error_message,
|
||||
}
|
||||
644
photo_pipeline/services/archive_transfer.py
Normal file
644
photo_pipeline/services/archive_transfer.py
Normal file
@@ -0,0 +1,644 @@
|
||||
"""Transfer an approved archive plan, verify it, and remove the active sources
|
||||
(US06-02).
|
||||
|
||||
This is the only module that deletes originals from the photo library, so every
|
||||
step exists to make one promise keepable: **a source is removed only after the
|
||||
archived bytes are durable and proven identical.** The journal (US06-02,
|
||||
:mod:`photo_pipeline.services.archive_journal`) records intent before each mutation;
|
||||
this module performs the mutations and the recovery that reads that intent back.
|
||||
|
||||
Per file the sequence is:
|
||||
|
||||
```
|
||||
journal.begin (transferring) ← intent persisted BEFORE any disk change
|
||||
recheck preconditions ← source hash, free destination, no symlink
|
||||
copy to a temporary file ← same directory, so the publish is atomic
|
||||
fsync, close, hash it back ← read from disk; the write is not the evidence
|
||||
atomically publish ← rename onto the final archive path
|
||||
append the manifest entry ← durable on the medium itself, fsynced
|
||||
journal → verified
|
||||
journal → removing ← intent to delete, persisted first
|
||||
re-verify the archive copy, remove the source, verify its absence
|
||||
current_path=NULL, close the path occurrence, availability + location recorded
|
||||
journal → complete
|
||||
```
|
||||
|
||||
Same-filesystem albums may skip the copy and use an atomic ``rename`` instead
|
||||
(concept §9), but only when the shared device is proven at run time — never from the
|
||||
plan's stored guess — and the published file is hashed afterwards exactly as in the
|
||||
copy path.
|
||||
|
||||
Rules that are never relaxed:
|
||||
|
||||
- An occupied destination is never overwritten; the item fails with the source
|
||||
untouched.
|
||||
- A source whose bytes no longer match the plan is never archived and never removed.
|
||||
- A crash resolves from journal + disk evidence only: an archive copy that is
|
||||
missing or hashes differently blocks the item for a human instead of being
|
||||
retried or, worse, treated as a successful archive.
|
||||
- Recovery is idempotent — repeated passes converge on the same state.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.models import ArchiveLocation, ArchiveOperation, ArchivePlan, Asset, AssetPath
|
||||
from photo_pipeline.services.archive_journal import (
|
||||
ARCHIVE,
|
||||
MANUAL,
|
||||
RESUMABLE,
|
||||
ArchiveJournal,
|
||||
ArchiveState,
|
||||
)
|
||||
from photo_pipeline.services.archives import MARKER_NAME, ArchiveError, ArchiveService
|
||||
from photo_pipeline.services.duplicates import DuplicateService
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.rename_apply import PreconditionFailed, maybe_fault
|
||||
from photo_pipeline.services.thumbnails import ThumbnailService
|
||||
|
||||
# The per-medium manifest: one JSON line per archived file, appended and fsynced
|
||||
# before its source is removed. It lives with the bytes so the archive can still be
|
||||
# read back if the database is lost.
|
||||
MANIFEST_NAME = "archive-manifest.jsonl"
|
||||
MANIFEST_VERSION = 1
|
||||
TEMP_SUFFIX = ".part"
|
||||
TEMP_PREFIX = ".archive-"
|
||||
# Fault barrier between "the source is gone" and "the database knows it" — not a
|
||||
# journal state, but the transition crash tests care about most.
|
||||
SOURCE_REMOVED = "source_removed"
|
||||
|
||||
APPLYABLE_PLAN_STATES = frozenset({"planned", "applying", "failed", "complete"})
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class ArchiveTransferService:
|
||||
def __init__(self, session_factory: sessionmaker, *, config: Config) -> None:
|
||||
self._session_factory = session_factory
|
||||
self._config = config
|
||||
self.journal = ArchiveJournal(session_factory)
|
||||
|
||||
# ── plans ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def create(self, location_id: str, albums: list[str] | None = None, *, token: str) -> dict:
|
||||
"""Turn an approved preflight into a durable plan.
|
||||
|
||||
The token is re-derived from a fresh preflight, so a plan can only be
|
||||
created for the exact scope, bytes, and destination the user approved.
|
||||
"""
|
||||
preflight = ArchiveService(self._session_factory, config=self._config).preflight(
|
||||
location_id, albums
|
||||
)
|
||||
if not token or token != preflight["token"]:
|
||||
raise ArchiveError("stale_token", "the archive preflight changed since it was approved")
|
||||
if preflight["state"] != "ready":
|
||||
codes = ", ".join(sorted({issue["code"] for issue in preflight["blockers"]})) or "-"
|
||||
raise ArchiveError("blocked", f"the archive scope is blocked: {codes}")
|
||||
|
||||
root = Path(preflight["location"]["root"])
|
||||
plan_id = str(uuid.uuid4())
|
||||
with self._session_factory() as session:
|
||||
session.add(
|
||||
ArchivePlan(
|
||||
id=plan_id,
|
||||
location_id=location_id,
|
||||
token=token,
|
||||
albums=json.dumps(albums) if albums is not None else None,
|
||||
direction=ARCHIVE,
|
||||
state="planned",
|
||||
schema_version=MANIFEST_VERSION,
|
||||
asset_count=preflight["totals"]["assets"],
|
||||
byte_size=preflight["totals"]["bytes"],
|
||||
)
|
||||
)
|
||||
session.flush() # the plan row must exist before its items reference it
|
||||
sequence = 0
|
||||
for album in preflight["albums"]:
|
||||
destination_dir = Path(album["destination"])
|
||||
for asset in album["assets"]:
|
||||
source = Path(asset["current_path"])
|
||||
destination = destination_dir / source.name
|
||||
session.add(
|
||||
ArchiveOperation(
|
||||
id=str(uuid.uuid4()),
|
||||
plan_id=plan_id,
|
||||
direction=ARCHIVE,
|
||||
sequence=sequence,
|
||||
album=album["album"],
|
||||
asset_id=asset["asset_id"],
|
||||
source_path=str(source),
|
||||
destination_path=str(destination),
|
||||
archive_path=str(destination.relative_to(root)),
|
||||
expected_sha256=asset["current_sha256"],
|
||||
byte_size=asset["byte_size"],
|
||||
# Recorded as a preview only; the real decision is made
|
||||
# against the devices at apply time.
|
||||
same_filesystem=album["transfer_method"] == "move",
|
||||
journal_state=ArchiveState.PLANNED,
|
||||
)
|
||||
)
|
||||
sequence += 1
|
||||
session.commit()
|
||||
return self.get(plan_id)
|
||||
|
||||
def get(self, plan_id: str) -> dict | None:
|
||||
with self._session_factory() as session:
|
||||
plan = session.get(ArchivePlan, plan_id)
|
||||
if plan is None:
|
||||
return None
|
||||
report = _plan_dict(plan)
|
||||
report["operations"] = self.journal.operations(plan_id)
|
||||
return report
|
||||
|
||||
def list(self) -> list[dict]:
|
||||
with self._session_factory() as session:
|
||||
rows = session.scalars(
|
||||
select(ArchivePlan)
|
||||
.where(ArchivePlan.direction == ARCHIVE)
|
||||
.order_by(ArchivePlan.created_at)
|
||||
)
|
||||
return [_plan_dict(row) for row in rows]
|
||||
|
||||
# ── apply ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def apply(
|
||||
self,
|
||||
plan_id: str,
|
||||
*,
|
||||
expected_version: int | None = None,
|
||||
worker_id: str = "archive",
|
||||
) -> dict:
|
||||
"""Archive every item of a plan, then report what happened.
|
||||
|
||||
Items are independent: one failure records its reason and leaves that
|
||||
source in place; the rest of the album continues.
|
||||
"""
|
||||
plan = self._require_plan(plan_id)
|
||||
if expected_version is not None and plan["version"] != expected_version:
|
||||
raise ArchiveError(
|
||||
"stale_plan",
|
||||
f"plan {plan_id} is at version {plan['version']}, expected {expected_version}",
|
||||
)
|
||||
if plan["state"] not in APPLYABLE_PLAN_STATES:
|
||||
raise ArchiveError("invalid_state", f"plan {plan_id} is {plan['state']}")
|
||||
|
||||
# One archiver lane: never start while another plan may be half-archived
|
||||
# (concept §16 lock hierarchy).
|
||||
blocking = [row for row in self.journal.incomplete() if row["plan_id"] != plan_id]
|
||||
if blocking:
|
||||
raise ArchiveError(
|
||||
"archive_pending",
|
||||
f"another archive operation is unresolved ({blocking[0]['id']}); recover it first",
|
||||
)
|
||||
|
||||
token = self._claim_plan(plan_id)
|
||||
location = self._location(plan["location_id"])
|
||||
archived = failed = skipped = 0
|
||||
for operation in self.journal.operations(plan_id):
|
||||
if operation["journal_state"] == ArchiveState.COMPLETE:
|
||||
skipped += 1 # repeated apply is a no-op for finished work
|
||||
continue
|
||||
try:
|
||||
if operation["journal_state"] in (ArchiveState.VERIFIED, ArchiveState.REMOVING):
|
||||
# The bytes are already archived; never transfer them twice.
|
||||
self._finish(operation, location, token=token, worker_id=worker_id)
|
||||
else:
|
||||
self._archive_one(operation, location, token=token, worker_id=worker_id)
|
||||
archived += 1
|
||||
except PreconditionFailed as error:
|
||||
self._fail(operation, token, error.code, str(error))
|
||||
failed += 1
|
||||
except Exception as error: # unexpected: record and stop touching disk
|
||||
self._fail(operation, token, "archive_error", str(error))
|
||||
failed += 1
|
||||
self._prune_empty_sources(plan_id)
|
||||
state = self.journal.sync_plan_state(plan_id)
|
||||
return {
|
||||
"plan_id": plan_id,
|
||||
"archived": archived,
|
||||
"failed": failed,
|
||||
"skipped": skipped,
|
||||
"state": state,
|
||||
}
|
||||
|
||||
def _prune_empty_sources(self, plan_id: str) -> None:
|
||||
"""Drop an album folder once every one of its files is archived.
|
||||
|
||||
``rmdir`` only: a folder that still holds anything at all — an unarchived
|
||||
file, someone else's file, a subfolder — is left exactly as it is.
|
||||
"""
|
||||
folders: dict[Path, set[str]] = {}
|
||||
for operation in self.journal.operations(plan_id):
|
||||
folders.setdefault(Path(operation["source_path"]).parent, set()).add(
|
||||
operation["journal_state"]
|
||||
)
|
||||
for folder, states in folders.items():
|
||||
if states == {ArchiveState.COMPLETE}:
|
||||
try:
|
||||
folder.rmdir()
|
||||
except OSError:
|
||||
pass # not empty, or gone already; either way, leave it alone
|
||||
|
||||
def _archive_one(self, operation: dict, location: dict, *, token: int, worker_id: str) -> None:
|
||||
source = Path(operation["source_path"])
|
||||
destination = Path(operation["destination_path"])
|
||||
|
||||
# 1. Intent first — after this point a crash is recoverable from evidence.
|
||||
self.journal.begin(operation["id"], worker_id=worker_id, fencing_token=token)
|
||||
maybe_fault(ArchiveState.TRANSFERRING)
|
||||
|
||||
# 2. Recheck immediately before mutating; the plan's snapshot is not trusted.
|
||||
self._recheck(operation, source, destination, location)
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 3. Transfer. Same-filesystem is an optimisation, so it has to be proven
|
||||
# here rather than believed from the plan.
|
||||
same_filesystem = _same_filesystem(source, destination.parent)
|
||||
if same_filesystem:
|
||||
os.rename(source, destination)
|
||||
else:
|
||||
copy_verify_publish(source, destination, operation["expected_sha256"])
|
||||
_fsync_dir(destination.parent)
|
||||
|
||||
# 4. The published file is the archive only once it hashes as recorded.
|
||||
if sha256_file(destination) != operation["expected_sha256"]:
|
||||
raise PreconditionFailed(
|
||||
"archive_mismatch", f"{destination} does not hold the expected bytes"
|
||||
)
|
||||
_append_manifest(destination.parent, _manifest_entry(operation, location))
|
||||
self.journal.transition(
|
||||
operation["id"],
|
||||
ArchiveState.VERIFIED,
|
||||
fencing_token=token,
|
||||
same_filesystem=same_filesystem,
|
||||
)
|
||||
maybe_fault(ArchiveState.VERIFIED)
|
||||
|
||||
# 5. Only now may the active source go.
|
||||
self._finish(self.journal.get(operation["id"]), location, token=token, worker_id=worker_id)
|
||||
|
||||
def _finish(self, operation: dict, location: dict, *, token: int, worker_id: str) -> None:
|
||||
"""Drive an item whose archive copy is durable through removal and
|
||||
bookkeeping. Every step is idempotent, so recovery may replay it."""
|
||||
destination = Path(operation["destination_path"])
|
||||
source = Path(operation["source_path"])
|
||||
state = operation["journal_state"]
|
||||
|
||||
if state == ArchiveState.FAILED:
|
||||
# The archive copy is durable even though the attempt ended badly:
|
||||
# re-enter the transfer state so the remaining steps can run.
|
||||
self.journal.transition(operation["id"], ArchiveState.TRANSFERRING, fencing_token=token)
|
||||
state = ArchiveState.TRANSFERRING
|
||||
|
||||
if state == ArchiveState.TRANSFERRING:
|
||||
if sha256_file(destination) != operation["expected_sha256"]:
|
||||
raise PreconditionFailed(
|
||||
"archive_mismatch", f"{destination} does not hold the expected bytes"
|
||||
)
|
||||
_append_manifest(destination.parent, _manifest_entry(operation, location))
|
||||
self.journal.transition(operation["id"], ArchiveState.VERIFIED, fencing_token=token)
|
||||
state = ArchiveState.VERIFIED
|
||||
|
||||
if state == ArchiveState.VERIFIED:
|
||||
self.journal.transition(operation["id"], ArchiveState.REMOVING, fencing_token=token)
|
||||
maybe_fault(ArchiveState.REMOVING)
|
||||
state = ArchiveState.REMOVING
|
||||
|
||||
if state == ArchiveState.REMOVING:
|
||||
# Re-verify the archived bytes immediately before deleting the original:
|
||||
# this check is the entire justification for the removal.
|
||||
if not destination.exists() or sha256_file(destination) != operation["expected_sha256"]:
|
||||
raise PreconditionFailed(
|
||||
"archive_unverified", f"{destination} is not a verified archive copy"
|
||||
)
|
||||
self._require_evidence(operation["asset_id"], destination)
|
||||
if source.exists():
|
||||
if source.is_symlink():
|
||||
raise PreconditionFailed("symlink", f"{source} became a symlink")
|
||||
if sha256_file(source) != operation["expected_sha256"]:
|
||||
raise PreconditionFailed(
|
||||
"source_changed", f"{source} changed; it is not ours to remove"
|
||||
)
|
||||
source.unlink()
|
||||
if source.exists():
|
||||
raise PreconditionFailed("removal_failed", f"{source} is still present")
|
||||
# The dangerous window: active storage no longer holds the file while the
|
||||
# database still points at it.
|
||||
maybe_fault(SOURCE_REMOVED)
|
||||
self._record_archived(operation, location, destination)
|
||||
self.journal.transition(operation["id"], ArchiveState.COMPLETE, fencing_token=token)
|
||||
maybe_fault(ArchiveState.COMPLETE)
|
||||
|
||||
def _recheck(self, operation: dict, source: Path, destination: Path, location: dict) -> None:
|
||||
if not source.exists():
|
||||
raise PreconditionFailed("source_missing", f"source {source} disappeared")
|
||||
if source.is_symlink() or destination.is_symlink():
|
||||
raise PreconditionFailed("symlink", "refusing to archive through a symlink")
|
||||
if destination.exists():
|
||||
raise PreconditionFailed("destination_exists", f"destination {destination} is occupied")
|
||||
root = Path(location["root"])
|
||||
if root not in destination.parents:
|
||||
raise PreconditionFailed(
|
||||
"destination_escape", f"{destination} is outside the archive location {root}"
|
||||
)
|
||||
if not root.is_dir() or not (root / MARKER_NAME).exists():
|
||||
raise PreconditionFailed("location_offline", f"{root} is not the archive medium")
|
||||
if sha256_file(source) != operation["expected_sha256"]:
|
||||
raise PreconditionFailed(
|
||||
"source_changed", f"{source} changed since the plan was approved"
|
||||
)
|
||||
with self._session_factory() as session:
|
||||
asset = session.get(Asset, operation["asset_id"])
|
||||
if asset is None or asset.current_path != str(source):
|
||||
raise PreconditionFailed(
|
||||
"asset_moved", f"asset {operation['asset_id']} is no longer at {source}"
|
||||
)
|
||||
|
||||
def _require_evidence(self, asset_id: str, source: Path) -> dict:
|
||||
"""Review evidence must be durable before the original goes.
|
||||
|
||||
The perceptual hash keeps the asset in the fuzzy index once its bytes are
|
||||
unreachable, and the protected preview is what duplicate review can still
|
||||
look at. Both are read from the freshly verified archive copy, which holds
|
||||
exactly the bytes being archived. A file that cannot be decoded has neither
|
||||
— recorded, not fatal, since its exact hashes remain — but failing to
|
||||
produce a preview from a decodable original stops the removal (concept §9).
|
||||
"""
|
||||
DuplicateService(self._session_factory).ensure_phash(asset_id, source=source)
|
||||
preview = ThumbnailService(self._session_factory, self._config).ensure_protected(
|
||||
asset_id, source=source
|
||||
)
|
||||
if preview["state"] == "unavailable":
|
||||
raise PreconditionFailed(
|
||||
"preview_unavailable",
|
||||
f"no durable comparison preview for asset {asset_id} "
|
||||
f"({preview['error_code']})",
|
||||
)
|
||||
return preview
|
||||
|
||||
# ── database ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _record_archived(self, operation: dict, location: dict, destination: Path) -> None:
|
||||
"""The original is gone from active storage: drop ``current_path``, close its
|
||||
occurrence, record where the bytes now live, and set availability."""
|
||||
now = _now()
|
||||
with self._session_factory() as session:
|
||||
asset = session.get(Asset, operation["asset_id"])
|
||||
if asset is None:
|
||||
raise PreconditionFailed(
|
||||
"asset_missing", f"asset {operation['asset_id']} no longer exists"
|
||||
)
|
||||
if asset.current_path:
|
||||
for row in session.scalars(
|
||||
select(AssetPath).where(
|
||||
AssetPath.asset_id == asset.id,
|
||||
AssetPath.path == asset.current_path,
|
||||
AssetPath.valid_until.is_(None),
|
||||
)
|
||||
):
|
||||
row.valid_until = now
|
||||
recorded = session.scalar(
|
||||
select(AssetPath).where(
|
||||
AssetPath.asset_id == asset.id, AssetPath.path == str(destination)
|
||||
)
|
||||
)
|
||||
if recorded is None: # idempotent: recovery may replay this
|
||||
session.add(
|
||||
AssetPath(
|
||||
asset_id=asset.id,
|
||||
path=str(destination),
|
||||
valid_from=now,
|
||||
reason="archive",
|
||||
)
|
||||
)
|
||||
asset.current_path = None
|
||||
asset.availability_state = (
|
||||
"archived_online" if destination.exists() else "archived_offline"
|
||||
)
|
||||
asset.archive_location_id = location["id"]
|
||||
asset.archive_path = operation["archive_path"]
|
||||
asset.state_version += 1
|
||||
asset.updated_at = now
|
||||
session.commit()
|
||||
|
||||
# ── recovery ──────────────────────────────────────────────────────────────
|
||||
|
||||
def recover(self, *, worker_id: str = "archive-recovery") -> dict:
|
||||
"""Resolve every incomplete item from journal + disk evidence.
|
||||
|
||||
Idempotent: running it repeatedly converges. Ambiguous (``manual``) work is
|
||||
left exactly as found and keeps blocking unrelated mutations.
|
||||
"""
|
||||
results = {"resumed": 0, "completed": 0, "manual": 0}
|
||||
touched: set[str] = set()
|
||||
for verdict in self.journal.classify_all(direction=ARCHIVE):
|
||||
operation = self.journal.get(verdict["operation_id"])
|
||||
touched.add(operation["plan_id"])
|
||||
token = (operation["fencing_token"] or 0) + 1
|
||||
if verdict["classification"] == MANUAL:
|
||||
results["manual"] += 1
|
||||
continue
|
||||
if verdict["classification"] == RESUMABLE:
|
||||
# Nothing was published: discard the debris and let a later apply
|
||||
# retry the item cleanly.
|
||||
_clean_temp_files(Path(operation["destination_path"]).parent)
|
||||
self.journal.transition(operation["id"], ArchiveState.PLANNED, fencing_token=token)
|
||||
results["resumed"] += 1
|
||||
continue
|
||||
location = self._location(self._require_plan(operation["plan_id"])["location_id"])
|
||||
try:
|
||||
self._finish(operation, location, token=token, worker_id=worker_id)
|
||||
results["completed"] += 1
|
||||
except PreconditionFailed as error:
|
||||
self._fail(operation, token, error.code, str(error))
|
||||
results["manual"] += 1
|
||||
for plan_id in touched:
|
||||
self._prune_empty_sources(plan_id)
|
||||
self.journal.sync_plan_state(plan_id)
|
||||
return results
|
||||
|
||||
def recovery_status(self) -> dict:
|
||||
verdicts = self.journal.classify_all(direction=ARCHIVE)
|
||||
return {
|
||||
"operations": verdicts,
|
||||
"manual": [v for v in verdicts if v["classification"] == MANUAL],
|
||||
"blocks_mutation": self.journal.blocks_mutation(),
|
||||
}
|
||||
|
||||
# ── helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
def _fail(self, operation: dict, token: int, code: str, message: str) -> None:
|
||||
self.journal.transition(
|
||||
operation["id"], ArchiveState.FAILED, fencing_token=token, error=(code, message)
|
||||
)
|
||||
|
||||
def _require_plan(self, plan_id: str) -> dict:
|
||||
with self._session_factory() as session:
|
||||
plan = session.get(ArchivePlan, plan_id)
|
||||
if plan is None:
|
||||
raise ArchiveError("unknown_plan", f"unknown archive plan {plan_id!r}")
|
||||
return _plan_dict(plan)
|
||||
|
||||
def _location(self, location_id: str) -> dict:
|
||||
with self._session_factory() as session:
|
||||
location = session.get(ArchiveLocation, location_id)
|
||||
if location is None:
|
||||
raise ArchiveError("unknown_location", f"unknown archive location {location_id!r}")
|
||||
return {"id": location.id, "root": location.root, "media_id": location.media_id}
|
||||
|
||||
def _claim_plan(self, plan_id: str) -> int:
|
||||
"""Bump the plan version and use it as this attempt's fencing token, so a
|
||||
worker from a superseded attempt cannot commit."""
|
||||
with self._session_factory() as session:
|
||||
plan = session.get(ArchivePlan, plan_id)
|
||||
plan.version += 1
|
||||
plan.state = "applying"
|
||||
plan.updated_at = _now()
|
||||
token = plan.version
|
||||
session.commit()
|
||||
return token
|
||||
|
||||
|
||||
# ── module helpers ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def copy_verify_publish(source: Path, destination: Path, expected_sha256: str) -> None:
|
||||
"""Copy to a temporary file beside the destination, prove its bytes, then publish
|
||||
it atomically. The source is never touched, so a failure costs nothing.
|
||||
|
||||
Shared by archiving (library → medium) and restoring (medium → library, US06-04):
|
||||
both need the same promise that a published file is either complete and correct
|
||||
or not there at all.
|
||||
"""
|
||||
temp = destination.with_name(f"{TEMP_PREFIX}{uuid.uuid4().hex}{TEMP_SUFFIX}")
|
||||
try:
|
||||
with open(source, "rb") as src, open(temp, "wb") as out:
|
||||
shutil.copyfileobj(src, out, 1024 * 1024)
|
||||
out.flush()
|
||||
os.fsync(out.fileno())
|
||||
if sha256_file(temp) != expected_sha256:
|
||||
raise PreconditionFailed("copy_mismatch", f"{source} copied with wrong bytes")
|
||||
if destination.exists():
|
||||
raise PreconditionFailed(
|
||||
"destination_exists", f"{destination} appeared during the transfer"
|
||||
)
|
||||
# ponytail: rename after an exists() check. The archiver lane is single and
|
||||
# local; use O_EXCL/link-based publish if a second writer ever exists.
|
||||
os.rename(temp, destination)
|
||||
finally:
|
||||
temp.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def _same_filesystem(source: Path, destination_dir: Path) -> bool:
|
||||
"""Proven at run time from the actual devices, never from the plan's preview."""
|
||||
try:
|
||||
return source.stat().st_dev == destination_dir.stat().st_dev
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _fsync_dir(path: Path) -> None:
|
||||
"""Make the directory entry itself durable, so the published name survives a
|
||||
power loss and not just the file's data."""
|
||||
fd = os.open(path, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(fd)
|
||||
except OSError:
|
||||
pass # some filesystems refuse directory fsync; the data is already synced
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def _manifest_entry(operation: dict, location: dict) -> dict:
|
||||
return {
|
||||
"schema_version": MANIFEST_VERSION,
|
||||
"plan_id": operation["plan_id"],
|
||||
"asset_id": operation["asset_id"],
|
||||
"album": operation["album"],
|
||||
"archive_path": operation["archive_path"],
|
||||
"source_path": operation["source_path"],
|
||||
"sha256": operation["expected_sha256"],
|
||||
"byte_size": operation["byte_size"],
|
||||
"media_id": location["media_id"],
|
||||
"archived_at": _now().isoformat(),
|
||||
}
|
||||
|
||||
|
||||
def _append_manifest(directory: Path, entry: dict) -> None:
|
||||
"""Append one durable manifest line, skipping an entry that is already there.
|
||||
|
||||
The manifest is written before the source is removed, so it is the medium's own
|
||||
record of what it holds even if the database is lost.
|
||||
"""
|
||||
path = directory / MANIFEST_NAME
|
||||
# ponytail: rereads the album manifest per file (O(n²) lines for one album).
|
||||
# Keep an in-memory index per plan if an album ever holds enough files to matter.
|
||||
if path.exists():
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
existing = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if (existing.get("asset_id"), existing.get("sha256")) == (
|
||||
entry["asset_id"],
|
||||
entry["sha256"],
|
||||
):
|
||||
return
|
||||
with open(path, "a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(entry, sort_keys=True) + "\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
_fsync_dir(directory)
|
||||
|
||||
|
||||
def read_manifest(directory: Path) -> list[dict]:
|
||||
"""Every manifest entry an archived album directory holds."""
|
||||
path = directory / MANIFEST_NAME
|
||||
if not path.exists():
|
||||
return []
|
||||
entries = []
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
entries.append(json.loads(line))
|
||||
except ValueError:
|
||||
continue
|
||||
return entries
|
||||
|
||||
|
||||
def _clean_temp_files(directory: Path) -> None:
|
||||
"""Remove this application's own abandoned transfer temporaries — never any
|
||||
other file (concept §17: startup cleans only recognised stale temporaries)."""
|
||||
if not directory.is_dir():
|
||||
return
|
||||
for temp in directory.glob(f"{TEMP_PREFIX}*{TEMP_SUFFIX}"):
|
||||
temp.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def _plan_dict(plan: ArchivePlan) -> dict:
|
||||
return {
|
||||
"id": plan.id,
|
||||
"location_id": plan.location_id,
|
||||
"token": plan.token,
|
||||
"direction": plan.direction,
|
||||
"albums": json.loads(plan.albums) if plan.albums else None,
|
||||
"state": plan.state,
|
||||
"schema_version": plan.schema_version,
|
||||
"asset_count": plan.asset_count,
|
||||
"byte_size": plan.byte_size,
|
||||
"version": plan.version,
|
||||
"worker_id": plan.worker_id,
|
||||
"completed_at": plan.completed_at.isoformat() if plan.completed_at else None,
|
||||
"created_at": plan.created_at.isoformat() if plan.created_at else None,
|
||||
}
|
||||
640
photo_pipeline/services/archives.py
Normal file
640
photo_pipeline/services/archives.py
Normal file
@@ -0,0 +1,640 @@
|
||||
"""ArchiveService — destinations and archive preflight (US06-01).
|
||||
|
||||
Archive is the only stage that *removes* originals from the active library, so
|
||||
this service does the opposite of removing anything: it registers destinations and
|
||||
proves, before a single byte moves, that an album could be archived safely. The
|
||||
transfer itself is US06-02.
|
||||
|
||||
An archive location is a medium, not a path (see :class:`ArchiveLocation`). A
|
||||
marker file on the medium carries its ``media_id``, so a disk mounted at the
|
||||
recorded root but holding a different marker is ``wrong_volume`` rather than
|
||||
silently accepted — the classic "the external disk came back at the same
|
||||
mountpoint" data-loss path.
|
||||
|
||||
Preflight proves, per concept §9 "Archive preflight":
|
||||
|
||||
- the album's upload is *verified*, not merely process-successful, and its bytes on
|
||||
disk still hash to exactly what was uploaded;
|
||||
- the destination medium is mounted, is the right one, is writable, lies outside
|
||||
every library root and every ``_IGNORE/`` tree, and has room for the scope plus a
|
||||
configured reserve;
|
||||
- nothing already occupies the destination;
|
||||
- no rename/upload/archive lease is held, and no rename is half-applied;
|
||||
- a database backup and the archive manifest can really be written — both are
|
||||
probed by writing them, not assumed.
|
||||
|
||||
Blocker codes: ``no_library_root``, ``location_offline``, ``wrong_volume``,
|
||||
``archive_pending``,
|
||||
``unsafe_destination``, ``destination_not_writable``, ``manifest_unwritable``,
|
||||
``insufficient_capacity``, ``backup_unavailable``, ``lock_conflict``,
|
||||
``rename_pending``, ``empty_scope``, ``destination_collision``,
|
||||
``upload_unverified``, ``bytes_changed``, ``file_missing``, ``preview_unavailable``.
|
||||
|
||||
Preflight also *creates* the durable comparison preview of every asset in scope
|
||||
(US06-03): it is the evidence duplicate review falls back on once the original is
|
||||
on a medium that may be offline, so it has to exist before the original leaves.
|
||||
|
||||
Like upload preflight, the confirmation token is *derived* from the report rather
|
||||
than stored: any change to the scope, the bytes, the destination, or the blockers
|
||||
produces a different token, so a stale browser confirmation can never apply. Values
|
||||
that drift without meaning anything (free space, backup size, timestamps) are left
|
||||
out of the digest.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import uuid
|
||||
from contextlib import closing
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.integrations import immich_go_report as report_parser
|
||||
from photo_pipeline.jobs.domain_handlers import ARCHIVE_LOCK, LIBRARY_WRITE_LOCK, UPLOAD_LOCK
|
||||
from photo_pipeline.models import ArchiveLocation, Asset, UploadBatch, UploadItem
|
||||
from photo_pipeline.path_policy import PathPolicyError, is_excluded, normalize_root, resolve_within
|
||||
from photo_pipeline.services.albums import album_label
|
||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
||||
from photo_pipeline.services.availability import MARKER_NAME, read_marker as _read_marker
|
||||
from photo_pipeline.services.availability import refresh as refresh_availability
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.jobs import JobService
|
||||
from photo_pipeline.services.rename_journal import RenameJournal
|
||||
from photo_pipeline.services.thumbnails import ThumbnailService
|
||||
from photo_pipeline.services.upload_reports import VERIFIED
|
||||
|
||||
PREFLIGHT_VERSION = 1
|
||||
TOKEN_PREFIX = f"v{PREFLIGHT_VERSION}"
|
||||
MANIFEST_NAME = "archive-manifest.json"
|
||||
|
||||
# Upload outcomes that prove Immich holds these exact bytes. ``skipped``/``failed``/
|
||||
# ``unknown`` never qualify: archiving on them would remove the only copy.
|
||||
ARCHIVED_OUTCOMES = frozenset(
|
||||
{report_parser.UPLOADED, report_parser.UPGRADED, report_parser.DUPLICATE}
|
||||
)
|
||||
|
||||
LOCKS = (LIBRARY_WRITE_LOCK, UPLOAD_LOCK, ARCHIVE_LOCK)
|
||||
|
||||
|
||||
class ArchiveError(RuntimeError):
|
||||
"""The request cannot be carried out (unknown location/album, unsafe root)."""
|
||||
|
||||
def __init__(self, code: str, message: str) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _issue(code: str, message: str) -> dict:
|
||||
return {"code": code, "message": message}
|
||||
|
||||
|
||||
class ArchiveService:
|
||||
def __init__(self, session_factory: sessionmaker, *, config: Config) -> None:
|
||||
self._session_factory = session_factory
|
||||
self._config = config
|
||||
self._roots = tuple(normalize_root(root) for root in config.library_roots)
|
||||
|
||||
# ── locations ─────────────────────────────────────────────────────────────
|
||||
|
||||
def register(self, name: str, root: str) -> dict:
|
||||
"""Register an archive destination and stamp its medium with a marker.
|
||||
|
||||
The marker is what makes the location identifiable later, so registering is
|
||||
the one archive operation that writes to the destination up front.
|
||||
"""
|
||||
name = (name or "").strip()
|
||||
if not name:
|
||||
raise ArchiveError("name_required", "an archive location needs a name")
|
||||
path = Path(root).expanduser()
|
||||
if not path.is_dir():
|
||||
raise ArchiveError("root_missing", f"{path} is not an existing directory")
|
||||
path = normalize_root(path)
|
||||
unsafe = self._unsafe_destination(path)
|
||||
if unsafe:
|
||||
raise ArchiveError("unsafe_destination", unsafe)
|
||||
|
||||
marker = _read_marker(path)
|
||||
with self._session_factory() as session:
|
||||
if session.scalar(select(ArchiveLocation).where(ArchiveLocation.name == name)):
|
||||
raise ArchiveError("duplicate_name", f"an archive location named {name!r} exists")
|
||||
if marker and session.scalar(
|
||||
select(ArchiveLocation).where(ArchiveLocation.media_id == marker.get("media_id"))
|
||||
):
|
||||
raise ArchiveError(
|
||||
"already_registered", f"{path} already belongs to another archive location"
|
||||
)
|
||||
media_id = marker.get("media_id") if marker else str(uuid.uuid4())
|
||||
error = _probe_write(
|
||||
path / MARKER_NAME,
|
||||
json.dumps({"media_id": media_id, "name": name}, indent=2).encode("utf-8"),
|
||||
keep=True,
|
||||
)
|
||||
if error:
|
||||
raise ArchiveError("destination_not_writable", error)
|
||||
location = ArchiveLocation(
|
||||
id=str(uuid.uuid4()),
|
||||
name=name,
|
||||
root=str(path),
|
||||
media_id=media_id,
|
||||
state="online",
|
||||
last_seen_at=_now(),
|
||||
)
|
||||
location.capabilities = json.dumps(_capabilities(path))
|
||||
session.add(location)
|
||||
session.commit()
|
||||
return self._location_report(location, probe=_probe_location(location))
|
||||
|
||||
def locations(self) -> list[dict]:
|
||||
"""Every configured location with a fresh probe of its medium."""
|
||||
with self._session_factory() as session:
|
||||
rows = list(session.scalars(select(ArchiveLocation).order_by(ArchiveLocation.name)))
|
||||
reports = []
|
||||
for location in rows:
|
||||
probe = _probe_location(location)
|
||||
location.state = probe["state"]
|
||||
if probe["state"] == "online":
|
||||
location.last_seen_at = _now()
|
||||
location.capabilities = json.dumps(probe["capabilities"])
|
||||
reports.append(self._location_report(location, probe=probe))
|
||||
session.commit()
|
||||
# A medium that just appeared or vanished changes what is readable, so the
|
||||
# archived assets are re-derived from the same probe (US06-03).
|
||||
refresh_availability(self._session_factory)
|
||||
return reports
|
||||
|
||||
# ── preflight ─────────────────────────────────────────────────────────────
|
||||
|
||||
def preflight(self, location_id: str, albums: list[str] | None = None) -> dict:
|
||||
"""Validate an archive scope against a destination and issue its token.
|
||||
|
||||
Read-only with respect to the library: it hashes files, probes the
|
||||
destination with its own temporary files, and writes nothing else.
|
||||
"""
|
||||
with self._session_factory() as session:
|
||||
location = session.get(ArchiveLocation, location_id)
|
||||
if location is None:
|
||||
raise ArchiveError("unknown_location", f"unknown archive location {location_id!r}")
|
||||
probe = _probe_location(location)
|
||||
location.state = probe["state"]
|
||||
if probe["state"] == "online":
|
||||
location.last_seen_at = _now()
|
||||
location.capabilities = json.dumps(probe["capabilities"])
|
||||
report = {
|
||||
"schema_version": PREFLIGHT_VERSION,
|
||||
"location": self._location_report(location, probe=probe),
|
||||
"blockers": [],
|
||||
}
|
||||
root = Path(location.root)
|
||||
session.commit()
|
||||
|
||||
report["blockers"] += self._destination_blockers(root, probe)
|
||||
report["blockers"] += self._lock_blockers()
|
||||
report["albums"] = self._albums(albums, root, reachable=probe["state"] == "online")
|
||||
report["totals"] = _totals(report["albums"])
|
||||
report["capacity"] = self._capacity(report["totals"]["bytes"], probe)
|
||||
if not report["capacity"]["sufficient"]:
|
||||
report["blockers"].append(
|
||||
_issue(
|
||||
"insufficient_capacity",
|
||||
# The free-space number is deliberately left out: it drifts between
|
||||
# two identical preflights, and the token is a digest of this text,
|
||||
# so quoting it here would invalidate every approval instantly.
|
||||
f"{report['totals']['bytes']} B plus a "
|
||||
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit on "
|
||||
"the medium",
|
||||
)
|
||||
)
|
||||
report["backup"] = self._backup_probe()
|
||||
if not report["backup"]["ok"]:
|
||||
report["blockers"].append(
|
||||
_issue(
|
||||
"backup_unavailable",
|
||||
f"a database backup could not be written: {report['backup']['detail']}",
|
||||
)
|
||||
)
|
||||
report["manifest"] = self._manifest_probe(
|
||||
root, report["albums"], writable=probe["writable"]
|
||||
)
|
||||
if not report["manifest"]["ok"]:
|
||||
report["blockers"].append(
|
||||
_issue(
|
||||
"manifest_unwritable",
|
||||
f"the archive manifest could not be written: {report['manifest']['detail']}",
|
||||
)
|
||||
)
|
||||
if not report["albums"]:
|
||||
report["blockers"].append(
|
||||
_issue("empty_scope", "no canonical, active assets are in the selected scope")
|
||||
)
|
||||
report["state"] = (
|
||||
"ready"
|
||||
if not report["blockers"] and all(a["state"] == "ready" for a in report["albums"])
|
||||
else "blocked"
|
||||
)
|
||||
report["token"] = _token(report)
|
||||
report["generated_at"] = _now().isoformat()
|
||||
return report
|
||||
|
||||
def verify_token(self, token: str, location_id: str, albums: list[str] | None = None) -> bool:
|
||||
"""True when ``token`` still describes this scope and this destination.
|
||||
|
||||
Recomputed, never looked up: an edited source file, a swapped medium, or a
|
||||
newly occupied destination invalidates it without anything writing to the
|
||||
database.
|
||||
"""
|
||||
return bool(token) and token == self.preflight(location_id, albums)["token"]
|
||||
|
||||
# ── destination ───────────────────────────────────────────────────────────
|
||||
|
||||
def _unsafe_destination(self, root: Path) -> str | None:
|
||||
"""Why this root may never hold archived originals, or ``None``."""
|
||||
if is_excluded(root):
|
||||
return f"{root} is inside an excluded (_IGNORE/) tree"
|
||||
for library in self._roots:
|
||||
if root == library or library in root.parents or root in library.parents:
|
||||
return f"{root} overlaps the active library root {library}"
|
||||
return None
|
||||
|
||||
def _destination_blockers(self, root: Path, probe: dict) -> list[dict]:
|
||||
blockers: list[dict] = []
|
||||
if not self._roots:
|
||||
blockers.append(_issue("no_library_root", "no library root is configured"))
|
||||
if probe["state"] == "offline":
|
||||
blockers.append(
|
||||
_issue("location_offline", f"the archive medium is not mounted at {root}")
|
||||
)
|
||||
elif probe["state"] == "wrong_volume":
|
||||
blockers.append(
|
||||
_issue(
|
||||
"wrong_volume",
|
||||
f"{root} holds a different archive medium ({probe['detail']})",
|
||||
)
|
||||
)
|
||||
unsafe = self._unsafe_destination(root)
|
||||
if unsafe:
|
||||
blockers.append(_issue("unsafe_destination", unsafe))
|
||||
if probe["state"] == "unwritable":
|
||||
blockers.append(
|
||||
_issue("destination_not_writable", f"{root} is not writable: {probe['detail']}")
|
||||
)
|
||||
return blockers
|
||||
|
||||
def _lock_blockers(self) -> list[dict]:
|
||||
"""Archive is blocked by any lease that may still be moving bytes or metadata."""
|
||||
blockers: list[dict] = []
|
||||
jobs = JobService(self._session_factory)
|
||||
for lock in LOCKS:
|
||||
held = jobs.blockers(lock)
|
||||
if held:
|
||||
blockers.append(
|
||||
_issue("lock_conflict", f"the {lock} lane is busy: job {held[0]['id']}")
|
||||
)
|
||||
if RenameJournal(self._session_factory).blocks_mutation():
|
||||
blockers.append(
|
||||
_issue("rename_pending", "an unresolved rename must be recovered before archiving")
|
||||
)
|
||||
if ArchiveJournal(self._session_factory).blocks_mutation():
|
||||
blockers.append(
|
||||
_issue(
|
||||
"archive_pending",
|
||||
"an unresolved archive transfer must be recovered before archiving again",
|
||||
)
|
||||
)
|
||||
return blockers
|
||||
|
||||
def _capacity(self, required: int, probe: dict) -> dict:
|
||||
reserve = self._config.archive_free_space_reserve_bytes
|
||||
free = probe["free_bytes"]
|
||||
return {
|
||||
"required_bytes": required,
|
||||
"reserve_bytes": reserve,
|
||||
"free_bytes": free,
|
||||
"sufficient": free is not None and free >= required + reserve,
|
||||
}
|
||||
|
||||
def _backup_probe(self) -> dict:
|
||||
"""Write a real online backup of the database, then discard it.
|
||||
|
||||
A backup that is merely assumed to be possible is worth nothing on the day
|
||||
the archive removes the originals, so this actually runs SQLite's backup API.
|
||||
"""
|
||||
source = self._config.database_path
|
||||
target = source.parent / f".archive-preflight-backup-{uuid.uuid4()}.db"
|
||||
try:
|
||||
with closing(sqlite3.connect(source)) as src, closing(sqlite3.connect(target)) as dst:
|
||||
src.backup(dst)
|
||||
size = target.stat().st_size
|
||||
except (sqlite3.Error, OSError) as error:
|
||||
return {"ok": False, "bytes": None, "detail": str(error)}
|
||||
finally:
|
||||
target.unlink(missing_ok=True)
|
||||
return {"ok": True, "bytes": size, "detail": None}
|
||||
|
||||
def _manifest_probe(self, root: Path, albums: list[dict], *, writable: bool) -> dict:
|
||||
"""Prove the manifest can be created by writing this exact content and
|
||||
removing it again. The real manifest is written by the transfer (US06-02)."""
|
||||
manifest = {
|
||||
"schema_version": PREFLIGHT_VERSION,
|
||||
"albums": [
|
||||
{
|
||||
"album": album["album"],
|
||||
"destination": album["destination"],
|
||||
"files": [
|
||||
{
|
||||
"asset_id": asset["asset_id"],
|
||||
"source": asset["current_path"],
|
||||
"sha256": asset["current_sha256"],
|
||||
"byte_size": asset["byte_size"],
|
||||
}
|
||||
for asset in album["assets"]
|
||||
],
|
||||
}
|
||||
for album in albums
|
||||
],
|
||||
}
|
||||
payload = json.dumps(manifest, indent=2, sort_keys=True).encode("utf-8")
|
||||
if not writable:
|
||||
return {"ok": False, "bytes": len(payload), "detail": "the destination is unavailable"}
|
||||
error = _probe_write(root / f".{MANIFEST_NAME}.probe-{uuid.uuid4()}", payload)
|
||||
return {"ok": error is None, "bytes": len(payload), "detail": error}
|
||||
|
||||
def _location_report(self, location: ArchiveLocation, *, probe: dict) -> dict:
|
||||
return {
|
||||
"id": location.id,
|
||||
"name": location.name,
|
||||
"root": location.root,
|
||||
"media_id": location.media_id,
|
||||
"state": probe["state"],
|
||||
"writable": probe["writable"],
|
||||
"device_id": probe["device_id"],
|
||||
"detail": probe["detail"],
|
||||
"last_seen_at": location.last_seen_at.isoformat() if location.last_seen_at else None,
|
||||
}
|
||||
|
||||
# ── scope ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def _albums(self, requested: list[str] | None, root: Path, *, reachable: bool) -> list[dict]:
|
||||
by_album = self._scope()
|
||||
if requested is not None:
|
||||
unknown = sorted(set(requested) - set(by_album))
|
||||
if unknown:
|
||||
raise ArchiveError("unknown_album", f"unknown album(s): {', '.join(unknown)}")
|
||||
by_album = {name: by_album[name] for name in sorted(set(requested))}
|
||||
return [
|
||||
self._album(name, rows, root, reachable=reachable)
|
||||
for name, rows in sorted(by_album.items())
|
||||
]
|
||||
|
||||
def _scope(self) -> dict[str, list[dict]]:
|
||||
"""Canonical, active assets grouped by album, each with its upload evidence."""
|
||||
with self._session_factory() as session:
|
||||
assets = list(
|
||||
session.scalars(
|
||||
select(Asset).where(
|
||||
Asset.canonical_asset_id.is_(None),
|
||||
Asset.availability_state == "active",
|
||||
Asset.current_path.is_not(None),
|
||||
)
|
||||
)
|
||||
)
|
||||
uploads: dict[str, UploadItem] = {}
|
||||
for item, batch in session.execute(
|
||||
select(UploadItem, UploadBatch)
|
||||
.join(UploadBatch, UploadBatch.id == UploadItem.batch_id)
|
||||
.order_by(UploadBatch.created_at)
|
||||
):
|
||||
if _proves_upload(item, batch):
|
||||
uploads[item.asset_id] = item # the latest verified batch wins
|
||||
|
||||
by_album: dict[str, list[dict]] = {}
|
||||
for asset in assets:
|
||||
by_album.setdefault(album_label(asset.current_path, self._roots), []).append(
|
||||
{
|
||||
"asset_id": asset.id,
|
||||
"path": asset.current_path,
|
||||
"byte_size": asset.byte_size,
|
||||
"upload": uploads.get(asset.id),
|
||||
}
|
||||
)
|
||||
return by_album
|
||||
|
||||
def _album(self, name: str, rows: list[dict], root: Path, *, reachable: bool) -> dict:
|
||||
folder = Path(rows[0]["path"]).parent
|
||||
items = sorted(
|
||||
(self._with_preview(_item(row)) for row in rows),
|
||||
key=lambda item: item["current_path"],
|
||||
)
|
||||
blocked = [item for item in items if item["blockers"]]
|
||||
blockers: list[dict] = []
|
||||
|
||||
destination = root / name
|
||||
try:
|
||||
resolve_within(root, destination)
|
||||
except PathPolicyError as error:
|
||||
blockers.append(_issue("unsafe_destination", str(error)))
|
||||
if reachable and destination.exists() and any(destination.iterdir()):
|
||||
blockers.append(
|
||||
_issue("destination_collision", f"{destination} already exists and is not empty")
|
||||
)
|
||||
if blocked:
|
||||
blockers.append(
|
||||
_issue(
|
||||
"partial_scope",
|
||||
f"{len(blocked)} of {len(items)} asset(s) are not archivable; an album is "
|
||||
"archived whole or not at all",
|
||||
)
|
||||
)
|
||||
return {
|
||||
"album": name,
|
||||
"folder": str(folder),
|
||||
"destination": str(destination),
|
||||
# Same filesystem means the transfer can be an atomic move; anything else
|
||||
# is copy-verify-remove (concept §9).
|
||||
"transfer_method": _transfer_method(folder, root),
|
||||
"asset_count": len(items),
|
||||
"blocked_count": len(blocked),
|
||||
"reclaimable_bytes": sum(item["byte_size"] or 0 for item in items),
|
||||
"state": "blocked" if blockers else "ready",
|
||||
"blockers": blockers,
|
||||
"assets": items,
|
||||
}
|
||||
|
||||
def _with_preview(self, item: dict) -> dict:
|
||||
"""Create the durable comparison preview while the original is still here.
|
||||
|
||||
This is the last moment it can be made: once the file is archived and the
|
||||
medium leaves, only the retained preview can answer "is this new photo the
|
||||
same picture?". An original that cannot be decoded at all has no preview to
|
||||
keep — its hashes and metadata stay the evidence — but a preview that fails
|
||||
for any other reason blocks the archive (concept §9).
|
||||
"""
|
||||
preview = self._previews().ensure_protected(item["asset_id"])
|
||||
item["preview"] = preview
|
||||
if preview["state"] == "unavailable" and not item["blockers"]:
|
||||
item["blockers"].append(
|
||||
_issue(
|
||||
"preview_unavailable",
|
||||
f"a durable comparison preview of {item['current_path']} could not be "
|
||||
f"created ({preview['error_code']})",
|
||||
)
|
||||
)
|
||||
return item
|
||||
|
||||
def _previews(self) -> ThumbnailService:
|
||||
return ThumbnailService(self._session_factory, self._config)
|
||||
|
||||
|
||||
# ── internals ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _item(row: dict) -> dict:
|
||||
"""One asset's archivability: verified upload plus the bytes on disk right now."""
|
||||
path = Path(row["path"])
|
||||
upload: UploadItem | None = row["upload"]
|
||||
blockers: list[dict] = []
|
||||
current_sha256 = None
|
||||
|
||||
if not path.exists():
|
||||
blockers.append(_issue("file_missing", f"{path} is missing"))
|
||||
else:
|
||||
# ponytail: full re-hash of the scope. Gate on (size, mtime_ns) first if a
|
||||
# large album makes this slow — the hash stays the authority.
|
||||
current_sha256 = sha256_file(path)
|
||||
|
||||
if upload is None:
|
||||
blockers.append(
|
||||
_issue("upload_unverified", "a verified Immich upload of these bytes is required")
|
||||
)
|
||||
elif current_sha256 is not None and upload.sha256 and current_sha256 != upload.sha256:
|
||||
blockers.append(
|
||||
_issue("bytes_changed", f"{path} changed since it was uploaded; re-upload it first")
|
||||
)
|
||||
|
||||
return {
|
||||
"asset_id": row["asset_id"],
|
||||
"current_path": str(path),
|
||||
"byte_size": row["byte_size"],
|
||||
"current_sha256": current_sha256,
|
||||
"uploaded_sha256": upload.sha256 if upload else None,
|
||||
"blockers": blockers,
|
||||
}
|
||||
|
||||
|
||||
def _proves_upload(item: UploadItem, batch: UploadBatch) -> bool:
|
||||
"""Whether this upload item is evidence that Immich holds these exact bytes."""
|
||||
return (
|
||||
batch.outcome_state == VERIFIED
|
||||
and not batch.stale_bytes
|
||||
and not item.changed_after_upload
|
||||
and item.outcome in ARCHIVED_OUTCOMES
|
||||
)
|
||||
|
||||
|
||||
def _transfer_method(folder: Path, root: Path) -> str:
|
||||
try:
|
||||
if folder.stat().st_dev == root.stat().st_dev:
|
||||
return "move"
|
||||
except OSError:
|
||||
pass
|
||||
return "copy_verify_remove"
|
||||
|
||||
|
||||
def _probe_write(path: Path, payload: bytes, *, keep: bool = False) -> str | None:
|
||||
"""Write ``payload`` to ``path``; return the failure detail or ``None``."""
|
||||
try:
|
||||
path.write_bytes(payload)
|
||||
except OSError as error:
|
||||
return str(error)
|
||||
if not keep:
|
||||
try:
|
||||
path.unlink()
|
||||
except OSError as error:
|
||||
return str(error)
|
||||
return None
|
||||
|
||||
|
||||
def _capabilities(root: Path) -> dict:
|
||||
usage = shutil.disk_usage(root)
|
||||
return {
|
||||
"device_id": root.stat().st_dev,
|
||||
"total_bytes": usage.total,
|
||||
"writable": os.access(root, os.W_OK),
|
||||
}
|
||||
|
||||
|
||||
def _probe_location(location: ArchiveLocation) -> dict:
|
||||
"""Is the right medium mounted, and can it take bytes right now?"""
|
||||
root = Path(location.root)
|
||||
blank = {"device_id": None, "free_bytes": None, "total_bytes": None, "capabilities": {}}
|
||||
if not root.is_dir():
|
||||
return {"state": "offline", "writable": False, "detail": f"{root} is not mounted", **blank}
|
||||
marker = _read_marker(root)
|
||||
if marker is None:
|
||||
return {
|
||||
"state": "offline",
|
||||
"writable": False,
|
||||
"detail": f"no archive marker found at {root}",
|
||||
**blank,
|
||||
}
|
||||
if marker.get("media_id") != location.media_id:
|
||||
return {
|
||||
"state": "wrong_volume",
|
||||
"writable": False,
|
||||
"detail": f"marker media_id {marker.get('media_id')!r}",
|
||||
**blank,
|
||||
}
|
||||
capabilities = _capabilities(root)
|
||||
usage = shutil.disk_usage(root)
|
||||
# os.access lies on some filesystems; a real write is the only proof.
|
||||
detail = _probe_write(root / f".archive-write-probe-{uuid.uuid4()}", b"")
|
||||
return {
|
||||
"state": "online" if detail is None else "unwritable",
|
||||
"writable": detail is None,
|
||||
"detail": detail,
|
||||
"device_id": capabilities["device_id"],
|
||||
"free_bytes": usage.free,
|
||||
"total_bytes": usage.total,
|
||||
"capabilities": capabilities,
|
||||
}
|
||||
|
||||
|
||||
def _totals(albums: list[dict]) -> dict:
|
||||
return {
|
||||
"albums": len(albums),
|
||||
"ready_albums": sum(1 for album in albums if album["state"] == "ready"),
|
||||
"assets": sum(album["asset_count"] for album in albums),
|
||||
"blocked": sum(album["blocked_count"] for album in albums),
|
||||
"bytes": sum(album["reclaimable_bytes"] for album in albums),
|
||||
}
|
||||
|
||||
|
||||
def _token(report: dict) -> str:
|
||||
"""Digest of everything the report asserts about the scope and the destination.
|
||||
|
||||
Values that drift without changing what would happen — free space, backup size,
|
||||
timestamps — are excluded so the same situation always yields the same token.
|
||||
"""
|
||||
payload = {key: value for key, value in report.items() if key not in ("generated_at", "token")}
|
||||
payload["location"] = {
|
||||
key: value for key, value in payload["location"].items() if key != "last_seen_at"
|
||||
}
|
||||
payload["capacity"] = {
|
||||
key: value for key, value in payload["capacity"].items() if key != "free_bytes"
|
||||
}
|
||||
payload["backup"] = {key: value for key, value in payload["backup"].items() if key != "bytes"}
|
||||
digest = hashlib.sha256(
|
||||
json.dumps(payload, sort_keys=True, ensure_ascii=False, default=str).encode("utf-8")
|
||||
).hexdigest()
|
||||
return f"{TOKEN_PREFIX}:{digest}"
|
||||
124
photo_pipeline/services/availability.py
Normal file
124
photo_pipeline/services/availability.py
Normal file
@@ -0,0 +1,124 @@
|
||||
"""Where an asset's bytes are right now (US06-03).
|
||||
|
||||
Archiving removes the original from the active library but never removes the
|
||||
asset: its identity, hashes, decisions, and evidence stay. This module is the one
|
||||
place that answers "can these bytes be read, and if not, why" so inventory,
|
||||
duplicate review, thumbnails, and the archive service all give the same answer.
|
||||
|
||||
States (concept §9):
|
||||
|
||||
- ``active`` — the original is in the active library;
|
||||
- ``archived_online`` — the recorded medium is mounted and holds the file;
|
||||
- ``archived_offline`` — archived, but the medium is not available right now;
|
||||
- ``missing_unexpected`` — neither an active path nor the recorded archive
|
||||
location explains the absence. This is the state that must never be confused
|
||||
with ``archived_offline``: an unmounted disk is normal, a mounted disk with a
|
||||
hole in it is not.
|
||||
|
||||
A medium is identified by its marker file, never by its mountpoint, so a
|
||||
different disk mounted at the recorded root is offline rather than accepted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from photo_pipeline.models import ArchiveLocation, Asset
|
||||
|
||||
ACTIVE = "active"
|
||||
ARCHIVED_ONLINE = "archived_online"
|
||||
ARCHIVED_OFFLINE = "archived_offline"
|
||||
MISSING_UNEXPECTED = "missing_unexpected"
|
||||
ARCHIVED = (ARCHIVED_ONLINE, ARCHIVED_OFFLINE)
|
||||
|
||||
MARKER_NAME = ".photo-pipeline-archive.json"
|
||||
|
||||
|
||||
def read_marker(root: Path) -> dict | None:
|
||||
"""The medium's identity marker, or ``None`` when it is not readable."""
|
||||
try:
|
||||
return json.loads((root / MARKER_NAME).read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def location_online(location: ArchiveLocation) -> bool:
|
||||
"""True only when the *recorded* medium is mounted at its root."""
|
||||
marker = read_marker(Path(location.root))
|
||||
return bool(marker) and marker.get("media_id") == location.media_id
|
||||
|
||||
|
||||
def archive_file(session: Session, asset: Asset) -> Path | None:
|
||||
"""The archived file's absolute path, whether or not the medium is mounted."""
|
||||
if not asset.archive_location_id or not asset.archive_path:
|
||||
return None
|
||||
location = session.get(ArchiveLocation, asset.archive_location_id)
|
||||
if location is None:
|
||||
return None
|
||||
return Path(location.root) / asset.archive_path
|
||||
|
||||
|
||||
def readable_path(session: Session, asset: Asset) -> Path | None:
|
||||
"""A path whose bytes can be read now: the active file, else the archive copy."""
|
||||
if asset.current_path and Path(asset.current_path).exists():
|
||||
return Path(asset.current_path)
|
||||
archived = archive_file(session, asset)
|
||||
if archived is None:
|
||||
return None
|
||||
location = session.get(ArchiveLocation, asset.archive_location_id)
|
||||
if not location_online(location) or not archived.exists():
|
||||
return None
|
||||
return archived
|
||||
|
||||
|
||||
def state_of(session: Session, asset: Asset, *, online: dict[str, bool] | None = None) -> str:
|
||||
"""The availability this asset's storage actually justifies right now."""
|
||||
if asset.current_path:
|
||||
return ACTIVE if Path(asset.current_path).exists() else MISSING_UNEXPECTED
|
||||
if not asset.archive_location_id:
|
||||
return MISSING_UNEXPECTED if asset.availability_state != ACTIVE else ACTIVE
|
||||
location = session.get(ArchiveLocation, asset.archive_location_id)
|
||||
if location is None:
|
||||
return MISSING_UNEXPECTED
|
||||
reachable = (
|
||||
online[location.id] if online and location.id in online else location_online(location)
|
||||
)
|
||||
if not reachable:
|
||||
return ARCHIVED_OFFLINE
|
||||
archived = archive_file(session, asset)
|
||||
# The medium is mounted and identified: the file is either there, or it is
|
||||
# genuinely gone — that is not "offline", it needs a human.
|
||||
return ARCHIVED_ONLINE if archived and archived.exists() else MISSING_UNEXPECTED
|
||||
|
||||
|
||||
def refresh(session_factory: sessionmaker) -> dict[str, int]:
|
||||
"""Re-derive availability for every archived asset from the media themselves.
|
||||
|
||||
Only archived assets are probed: whether an *active* file is present is the
|
||||
inventory scan's job and costs one stat per library file. Each medium is
|
||||
probed once, not once per asset.
|
||||
"""
|
||||
counts: Counter[str] = Counter()
|
||||
now = datetime.now(timezone.utc)
|
||||
with session_factory() as session:
|
||||
online = {
|
||||
location.id: location_online(location)
|
||||
for location in session.scalars(select(ArchiveLocation))
|
||||
}
|
||||
for asset in session.scalars(
|
||||
select(Asset).where(Asset.archive_location_id.is_not(None))
|
||||
):
|
||||
state = state_of(session, asset, online=online)
|
||||
counts[state] += 1
|
||||
if state != asset.availability_state:
|
||||
asset.availability_state = state
|
||||
asset.state_version += 1
|
||||
asset.updated_at = now
|
||||
session.commit()
|
||||
return dict(counts)
|
||||
@@ -11,6 +11,12 @@ Detection runs in two categories:
|
||||
band (NEAR/SIMILAR). These are review candidates: never decided automatically, and
|
||||
negative-linked pairs are suppressed so a rejected pair is not re-suggested.
|
||||
|
||||
Archived assets stay in both indexes (US06-03): a new active copy of an archived
|
||||
original is recognised through its hashes even while the medium is offline, and
|
||||
cluster review falls back to the retained protected preview plus hash evidence.
|
||||
An exact/pixel match links straight to the archived canonical; a perceptual match
|
||||
is a review candidate that names the medium to mount for a pixel-level decision.
|
||||
|
||||
Decisions (``canonical`` / ``not_duplicate`` / ``deferred``) persist with evidence,
|
||||
use optimistic version checks, are reversible, and can never form a canonical cycle.
|
||||
A new content-identical member of an already-decided cluster inherits the established
|
||||
@@ -34,12 +40,14 @@ from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.models import (
|
||||
ArchiveLocation,
|
||||
Asset,
|
||||
DuplicateCluster,
|
||||
DuplicateMember,
|
||||
DuplicateNegativeLink,
|
||||
Thumbnail,
|
||||
)
|
||||
from photo_pipeline.services import hashing
|
||||
from photo_pipeline.services import availability, hashing
|
||||
|
||||
NEAR_MAX = 5
|
||||
SIMILAR_MAX = 10
|
||||
@@ -127,18 +135,18 @@ class DuplicateService:
|
||||
|
||||
# ── perceptual hash backfill ───────────────────────────────────────────
|
||||
def ensure_phashes(self) -> int:
|
||||
"""Hash whatever is readable now — an archived asset keeps the hash it
|
||||
already has, and gains one whenever its medium happens to be mounted."""
|
||||
updated = 0
|
||||
with self._session_factory() as session:
|
||||
assets = session.execute(
|
||||
select(Asset).where(
|
||||
Asset.availability_state == "active",
|
||||
Asset.current_path.isnot(None),
|
||||
)
|
||||
).scalars()
|
||||
assets = session.execute(select(Asset)).scalars()
|
||||
for asset in assets:
|
||||
if asset.phash is not None and asset.phash_version == hashing.PHASH_VERSION:
|
||||
continue
|
||||
value = hashing.safe_phash(asset.current_path)
|
||||
source = availability.readable_path(session, asset)
|
||||
if source is None:
|
||||
continue
|
||||
value = hashing.safe_phash(str(source))
|
||||
if value is not None:
|
||||
asset.phash = value
|
||||
asset.phash_version = hashing.PHASH_VERSION
|
||||
@@ -146,20 +154,40 @@ class DuplicateService:
|
||||
session.commit()
|
||||
return updated
|
||||
|
||||
def ensure_phash(self, asset_id: str, *, source=None) -> str | None:
|
||||
"""Backfill one asset's perceptual hash while its bytes are still readable.
|
||||
|
||||
Archiving calls this before the original leaves — passing the archive copy
|
||||
as ``source``, since the database does not point at it yet — because an
|
||||
asset without a pHash silently drops out of the fuzzy index the moment its
|
||||
medium is away.
|
||||
"""
|
||||
with self._session_factory() as session:
|
||||
asset = session.get(Asset, asset_id)
|
||||
if asset is None:
|
||||
return None
|
||||
if asset.phash is not None and asset.phash_version == hashing.PHASH_VERSION:
|
||||
return asset.phash
|
||||
source = source or availability.readable_path(session, asset)
|
||||
if source is None:
|
||||
return None
|
||||
value = hashing.safe_phash(str(source))
|
||||
if value is not None:
|
||||
asset.phash = value
|
||||
asset.phash_version = hashing.PHASH_VERSION
|
||||
session.commit()
|
||||
return value
|
||||
|
||||
# ── detection ──────────────────────────────────────────────────────────
|
||||
def detect(self) -> DetectionReport:
|
||||
self.ensure_phashes()
|
||||
now = datetime.now(timezone.utc)
|
||||
report = DetectionReport()
|
||||
with self._session_factory() as session:
|
||||
assets = list(
|
||||
session.execute(
|
||||
select(Asset).where(
|
||||
Asset.availability_state == "active",
|
||||
Asset.current_path.isnot(None),
|
||||
)
|
||||
).scalars()
|
||||
)
|
||||
# Every known asset stays in the indexes, archived or not: a copy of an
|
||||
# archived original must be recognised as a duplicate rather than
|
||||
# treated as a new photo (concept §9, invariant 12).
|
||||
assets = list(session.execute(select(Asset)).scalars())
|
||||
by_id = {a.id: a for a in assets}
|
||||
negatives = {
|
||||
_pair(link.asset_a, link.asset_b)
|
||||
@@ -431,10 +459,19 @@ class DuplicateService:
|
||||
|
||||
@staticmethod
|
||||
def _recommend_canonical(ids, by_id) -> str:
|
||||
# ponytail: largest file, path as deterministic tie-break. The concept's
|
||||
# richer policy (resolution, least recompression, metadata richness) lands
|
||||
# with the review UI story.
|
||||
return max(ids, key=lambda i: (by_id[i].byte_size or 0, by_id[i].current_path or ""))
|
||||
# ponytail: largest file, then the archived copy, then path as a
|
||||
# deterministic tie-break. Archived wins ties because it is the reviewed,
|
||||
# uploaded original — a fresh active copy must not demote it to a variant.
|
||||
# The concept's richer policy (resolution, least recompression, metadata
|
||||
# richness) lands with the review UI story.
|
||||
return max(
|
||||
ids,
|
||||
key=lambda i: (
|
||||
by_id[i].byte_size or 0,
|
||||
by_id[i].availability_state in availability.ARCHIVED,
|
||||
by_id[i].current_path or by_id[i].archive_path or "",
|
||||
),
|
||||
)
|
||||
|
||||
def _apply_canonical(self, session, cluster, ids, canonical_id):
|
||||
for member in session.execute(
|
||||
@@ -507,9 +544,15 @@ class DuplicateService:
|
||||
"current_path": asset.current_path if asset else None,
|
||||
"byte_size": asset.byte_size if asset else None,
|
||||
"phash": asset.phash if asset else None,
|
||||
**self._offline_evidence(session, asset),
|
||||
}
|
||||
)
|
||||
members.sort(key=lambda m: m["asset_id"])
|
||||
# A full-resolution comparison of an offline original is impossible; the
|
||||
# UI asks for that named medium instead of guessing (concept §9).
|
||||
mount_required = sorted(
|
||||
{m["archive_location"] for m in members if m["requires_mount"]}
|
||||
)
|
||||
return {
|
||||
"id": cluster.id,
|
||||
"method": cluster.method,
|
||||
@@ -519,9 +562,54 @@ class DuplicateService:
|
||||
"canonical_asset_id": cluster.canonical_asset_id,
|
||||
"version": cluster.version,
|
||||
"requires_confirmation": cluster.method == Method.PERCEPTUAL.value,
|
||||
"mount_required": mount_required,
|
||||
"members": members,
|
||||
}
|
||||
|
||||
def _offline_evidence(self, session, asset: Asset | None) -> dict:
|
||||
"""What review can still rely on when a member's original is not readable."""
|
||||
if asset is None:
|
||||
return {
|
||||
"availability_state": None,
|
||||
"archive_location": None,
|
||||
"archive_location_id": None,
|
||||
"archive_path": None,
|
||||
"preview": {"state": "missing", "protected": False},
|
||||
"requires_mount": False,
|
||||
}
|
||||
location = (
|
||||
session.get(ArchiveLocation, asset.archive_location_id)
|
||||
if asset.archive_location_id
|
||||
else None
|
||||
)
|
||||
preview = self._preview_evidence(session, asset.id)
|
||||
archived = asset.availability_state in availability.ARCHIVED
|
||||
return {
|
||||
"availability_state": asset.availability_state,
|
||||
"archive_location": location.name if location else None,
|
||||
"archive_location_id": asset.archive_location_id,
|
||||
"archive_path": asset.archive_path,
|
||||
"preview": preview,
|
||||
# Offline archived members can still be compared through their retained
|
||||
# preview and hash evidence; only pixel-level review needs the medium.
|
||||
"requires_mount": archived
|
||||
and asset.availability_state == availability.ARCHIVED_OFFLINE
|
||||
and bool(location),
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _preview_evidence(session, asset_id: str) -> dict:
|
||||
rows = list(
|
||||
session.execute(select(Thumbnail).where(Thumbnail.asset_id == asset_id)).scalars()
|
||||
)
|
||||
ready = [r for r in rows if r.state == "ready" and r.path]
|
||||
if ready:
|
||||
best = max(ready, key=lambda r: (bool(r.protected), r.size or 0))
|
||||
return {"state": "ready", "protected": bool(best.protected), "size": best.size}
|
||||
if rows:
|
||||
return {"state": "unsupported", "protected": False, "size": rows[0].size}
|
||||
return {"state": "missing", "protected": False, "size": None}
|
||||
|
||||
# ── decisions ────────────────────────────────────────────────────────────
|
||||
def decide(
|
||||
self,
|
||||
|
||||
@@ -13,8 +13,11 @@ renames. Every discovered or absent path is classified as one occurrence:
|
||||
- ``missing`` — a known active asset whose file is gone (kept, flagged).
|
||||
|
||||
Missing files are never pruned (that would break identity); the asset is retained
|
||||
with ``missing_at`` set. Archived assets are left untouched. Rescanning unchanged
|
||||
input makes no durable change.
|
||||
with ``missing_at`` set and its availability becomes ``missing_unexpected`` —
|
||||
nothing explains where the bytes went. Archived assets are left untouched: their
|
||||
absence from the active roots is expected, and each scan re-derives whether their
|
||||
medium is reachable (:mod:`photo_pipeline.services.availability`). Rescanning
|
||||
unchanged input makes no durable change.
|
||||
|
||||
Extracted from photo_analyzer.discover_photos/reconcile_moved/prune_missing
|
||||
(see donor_ledger.yaml: pa-discovery, pa-prune-missing).
|
||||
@@ -30,12 +33,12 @@ from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from photo_pipeline import path_policy
|
||||
from photo_pipeline.models import Asset, AssetPath
|
||||
from photo_pipeline.services import hashing
|
||||
from photo_pipeline.services import availability, hashing
|
||||
|
||||
|
||||
class Occurrence(str, Enum):
|
||||
@@ -60,6 +63,8 @@ def _asset_dict(asset: Asset) -> dict:
|
||||
"id": asset.id,
|
||||
"current_path": asset.current_path,
|
||||
"availability_state": asset.availability_state,
|
||||
"archive_location_id": asset.archive_location_id,
|
||||
"archive_path": asset.archive_path,
|
||||
"byte_size": asset.byte_size,
|
||||
"current_sha256": asset.current_sha256,
|
||||
"pixel_sha256": asset.pixel_sha256,
|
||||
@@ -101,7 +106,9 @@ class InventoryService:
|
||||
result.asset_ids[str(path)] = asset.id
|
||||
|
||||
for asset in assets:
|
||||
if asset.availability_state != "active" or asset.id in seen_ids:
|
||||
# Archived assets are explained by their location, not by the active
|
||||
# roots: a scan must never prune or flag them (concept §9).
|
||||
if asset.availability_state in availability.ARCHIVED or asset.id in seen_ids:
|
||||
continue
|
||||
if asset.current_path and asset.current_path not in discovered_paths:
|
||||
if not Path(asset.current_path).exists():
|
||||
@@ -109,10 +116,18 @@ class InventoryService:
|
||||
asset.missing_at = now
|
||||
asset.state_version += 1
|
||||
asset.updated_at = now
|
||||
# Nothing explains this absence — it is not an offline medium.
|
||||
if asset.availability_state != availability.MISSING_UNEXPECTED:
|
||||
asset.availability_state = availability.MISSING_UNEXPECTED
|
||||
asset.state_version += 1
|
||||
asset.updated_at = now
|
||||
result.occurrences[asset.current_path] = Occurrence.MISSING.value
|
||||
|
||||
session.commit()
|
||||
|
||||
# Media may have been mounted or removed since the last scan.
|
||||
availability.refresh(self._session_factory)
|
||||
|
||||
result.counts = dict(Counter(result.occurrences.values()))
|
||||
return result
|
||||
|
||||
@@ -132,7 +147,11 @@ class InventoryService:
|
||||
if availability:
|
||||
stmt = stmt.where(Asset.availability_state == availability)
|
||||
if query:
|
||||
stmt = stmt.where(Asset.current_path.like(f"%{query}%"))
|
||||
like = f"%{query}%"
|
||||
# An archived asset has no active path; it is searched where it lives.
|
||||
stmt = stmt.where(
|
||||
or_(Asset.current_path.like(like), Asset.archive_path.like(like))
|
||||
)
|
||||
total = session.scalar(select(func.count()).select_from(stmt.subquery()))
|
||||
rows = session.execute(
|
||||
stmt.order_by(Asset.current_path).limit(limit).offset(offset)
|
||||
@@ -169,6 +188,7 @@ class InventoryService:
|
||||
self._open_path(session, existing.id, path_str, now, occ.value)
|
||||
if existing.missing_at is not None:
|
||||
existing.missing_at = None
|
||||
existing.availability_state = availability.ACTIVE
|
||||
existing.state_version += 1
|
||||
existing.updated_at = now
|
||||
return existing, occ
|
||||
@@ -189,6 +209,7 @@ class InventoryService:
|
||||
moved_from.current_path = path_str
|
||||
moved_from.byte_size = size
|
||||
moved_from.missing_at = None
|
||||
moved_from.availability_state = availability.ACTIVE
|
||||
moved_from.state_version += 1
|
||||
moved_from.updated_at = now
|
||||
self._open_path(session, moved_from.id, path_str, now, Occurrence.MOVED.value)
|
||||
|
||||
220
photo_pipeline/services/legacy_import.py
Normal file
220
photo_pipeline/services/legacy_import.py
Normal file
@@ -0,0 +1,220 @@
|
||||
"""Import the last path-keyed CSV state into the database (US07-01).
|
||||
|
||||
``nsfwtag`` cached its safety scores in ``nsfw_scores.csv`` next to the library:
|
||||
one ``path,nsfw_score`` row per photo, four decimals, unreadable rows dropped. That
|
||||
file stops being a source of truth when the CLI is archived, so its scores are
|
||||
imported once — as scored-but-unreviewed ``safety_reviews`` rows on the stable
|
||||
``assets.id`` each path resolves to — and the CSV is left untouched on disk as its
|
||||
own backup.
|
||||
|
||||
The import is deliberately conservative, because a score is evidence about a photo
|
||||
and a path is not an identity:
|
||||
|
||||
- a row whose path matches no known asset is **unmatched**, never a new asset;
|
||||
- an asset a human already reviewed is **never** touched: a score is evidence, a
|
||||
decision is a judgement, and an import may not overwrite the second with the
|
||||
first — the difference is reported as a conflict instead;
|
||||
- a row for an asset that already carries a score is **skipped** unless
|
||||
``overwrite`` is asked for, and a differing score is reported either way;
|
||||
- an unparsable score or a duplicate path is **rejected** with its reason;
|
||||
- rerunning changes nothing that is already imported.
|
||||
|
||||
Everything it did — and everything it refused to do — comes back as a
|
||||
reconciliation report, which the caller can persist next to the database. The
|
||||
donor's own reader is the specification for the format (donor ledger:
|
||||
``nt-score-cache``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import uuid
|
||||
from collections import Counter
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview
|
||||
|
||||
REPORT_VERSION = 1
|
||||
# The donor wrote scores as four-decimal strings; anything outside 0..1 was never
|
||||
# something it produced, so it is data corruption rather than a score.
|
||||
SCORE_RANGE = (0.0, 1.0)
|
||||
# Recorded as the reviewer so an imported score is never mistaken for a judgement.
|
||||
REVIEWER = "legacy-csv-import"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ImportReport:
|
||||
source: str
|
||||
rows: int = 0
|
||||
imported: int = 0
|
||||
skipped_existing: int = 0
|
||||
unchanged: int = 0
|
||||
reviewed: int = 0
|
||||
unmatched: list[str] = field(default_factory=list)
|
||||
rejected: list[dict] = field(default_factory=list)
|
||||
conflicts: list[dict] = field(default_factory=list)
|
||||
generated_at: str = ""
|
||||
schema_version: int = REPORT_VERSION
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
@property
|
||||
def counts(self) -> dict[str, int]:
|
||||
return {
|
||||
"rows": self.rows,
|
||||
"imported": self.imported,
|
||||
"skipped_existing": self.skipped_existing,
|
||||
"unchanged": self.unchanged,
|
||||
"reviewed": self.reviewed,
|
||||
"unmatched": len(self.unmatched),
|
||||
"rejected": len(self.rejected),
|
||||
"conflicts": len(self.conflicts),
|
||||
}
|
||||
|
||||
|
||||
class LegacyImportService:
|
||||
def __init__(self, session_factory: sessionmaker) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
def import_nsfw_scores(
|
||||
self, csv_path: Path | str, *, overwrite: bool = False, dry_run: bool = False
|
||||
) -> ImportReport:
|
||||
"""Import ``nsfw_scores.csv`` onto asset identity and report what happened."""
|
||||
path = Path(csv_path)
|
||||
report = ImportReport(source=str(path))
|
||||
if not path.is_file():
|
||||
report.rejected.append({"path": str(path), "reason": "csv_missing"})
|
||||
return self._stamp(report)
|
||||
|
||||
with self._session_factory() as session:
|
||||
# Paths are matched against every occurrence an asset ever had, so a
|
||||
# photo scored before a rename is still recognised.
|
||||
by_path = self._path_index(session)
|
||||
latest = self._latest_reviews(session)
|
||||
seen: Counter[str] = Counter()
|
||||
|
||||
for row in self._rows(path, report):
|
||||
report.rows += 1
|
||||
raw_path, raw_score = row
|
||||
seen[raw_path] += 1
|
||||
if seen[raw_path] > 1:
|
||||
report.rejected.append({"path": raw_path, "reason": "duplicate_path"})
|
||||
continue
|
||||
score = _parse_score(raw_score)
|
||||
if score is None:
|
||||
report.rejected.append(
|
||||
{"path": raw_path, "reason": "unparsable_score", "value": raw_score}
|
||||
)
|
||||
continue
|
||||
asset_id = by_path.get(raw_path)
|
||||
if asset_id is None:
|
||||
report.unmatched.append(raw_path)
|
||||
continue
|
||||
|
||||
current = latest.get(asset_id)
|
||||
if current is not None and current.decision:
|
||||
# A human decided this one; the CSV is older evidence.
|
||||
report.reviewed += 1
|
||||
if current.score is None or abs(current.score - score) >= 1e-9:
|
||||
report.conflicts.append(
|
||||
{
|
||||
"path": raw_path,
|
||||
"asset_id": asset_id,
|
||||
"current": current.score,
|
||||
"decision": current.decision,
|
||||
"csv": score,
|
||||
}
|
||||
)
|
||||
continue
|
||||
if current is not None and current.score is not None:
|
||||
if abs(current.score - score) < 1e-9:
|
||||
report.unchanged += 1
|
||||
continue
|
||||
report.conflicts.append(
|
||||
{
|
||||
"path": raw_path,
|
||||
"asset_id": asset_id,
|
||||
"current": current.score,
|
||||
"decision": None,
|
||||
"csv": score,
|
||||
}
|
||||
)
|
||||
if not overwrite:
|
||||
report.skipped_existing += 1
|
||||
continue
|
||||
session.add(
|
||||
SafetyReview(
|
||||
id=str(uuid.uuid4()),
|
||||
asset_id=asset_id,
|
||||
score=score,
|
||||
reviewer=REVIEWER,
|
||||
)
|
||||
)
|
||||
latest[asset_id] = SafetyReview(asset_id=asset_id, score=score)
|
||||
report.imported += 1
|
||||
|
||||
if dry_run:
|
||||
session.rollback()
|
||||
else:
|
||||
session.commit()
|
||||
return self._stamp(report)
|
||||
|
||||
@staticmethod
|
||||
def _rows(path: Path, report: ImportReport):
|
||||
"""Yield ``(path, score)`` pairs, tolerating the donor's own sloppiness."""
|
||||
with open(path, newline="", encoding="utf-8", errors="replace") as handle:
|
||||
for row in csv.DictReader(handle):
|
||||
raw_path = (row.get("path") or "").strip()
|
||||
if not raw_path:
|
||||
report.rejected.append({"path": "", "reason": "missing_path"})
|
||||
continue
|
||||
yield raw_path, row.get("nsfw_score")
|
||||
|
||||
@staticmethod
|
||||
def _latest_reviews(session) -> dict[str, SafetyReview]:
|
||||
"""The current safety row per asset — latest wins, as everywhere else."""
|
||||
latest: dict[str, SafetyReview] = {}
|
||||
for review in session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)):
|
||||
latest[review.asset_id] = review
|
||||
return latest
|
||||
|
||||
@staticmethod
|
||||
def _path_index(session) -> dict[str, str]:
|
||||
"""Every path an asset is or was known by → its stable id."""
|
||||
index: dict[str, str] = {}
|
||||
for asset_id, path in session.execute(select(AssetPath.asset_id, AssetPath.path)):
|
||||
index.setdefault(path, asset_id)
|
||||
for asset_id, path in session.execute(select(Asset.id, Asset.current_path)):
|
||||
if path:
|
||||
index[path] = asset_id # the current path wins over a closed one
|
||||
return index
|
||||
|
||||
@staticmethod
|
||||
def _stamp(report: ImportReport) -> ImportReport:
|
||||
report.generated_at = datetime.now(timezone.utc).isoformat()
|
||||
return report
|
||||
|
||||
|
||||
def write_report(report: ImportReport, directory: Path) -> Path:
|
||||
"""Persist the reconciliation report; the import is not evidence until it is."""
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / "legacy-nsfw-import.json"
|
||||
path.write_text(json.dumps(report.to_dict(), indent=2, sort_keys=True), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _parse_score(value) -> float | None:
|
||||
try:
|
||||
score = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not SCORE_RANGE[0] <= score <= SCORE_RANGE[1]:
|
||||
return None
|
||||
return score
|
||||
@@ -83,12 +83,13 @@ def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _maybe_fault(state: str) -> None:
|
||||
def maybe_fault(state: str) -> None:
|
||||
"""Test-only crash barrier (concept §18 fault injection).
|
||||
|
||||
When ``PHOTO_PIPELINE_FAULT_AFTER`` names a journal state, the process dies
|
||||
abruptly the moment that state has been persisted — modelling a real kill at
|
||||
exactly that transition. Never set outside tests.
|
||||
exactly that transition. Never set outside tests. Shared with the archive
|
||||
transfer journal (US06-02), which uses the same env var and its own state names.
|
||||
"""
|
||||
if os.environ.get("PHOTO_PIPELINE_FAULT_AFTER") == state:
|
||||
os._exit(9)
|
||||
@@ -172,7 +173,7 @@ class RenameApplyService:
|
||||
|
||||
# 1. Intent first — after this point a crash is recoverable from evidence.
|
||||
self.journal.begin(operation["id"], worker_id=worker_id, fencing_token=token)
|
||||
_maybe_fault(JournalState.MOVING)
|
||||
maybe_fault(JournalState.MOVING)
|
||||
|
||||
# 2. Recheck preconditions immediately before mutating, never trusting the
|
||||
# plan's snapshot: files can change between preview and confirmation.
|
||||
@@ -187,21 +188,21 @@ class RenameApplyService:
|
||||
os.rename(source, destination)
|
||||
|
||||
self.journal.transition(operation["id"], JournalState.MOVED, fencing_token=token)
|
||||
_maybe_fault(JournalState.MOVED)
|
||||
maybe_fault(JournalState.MOVED)
|
||||
|
||||
# 4. Database: stable IDs keep their identity, paths are re-pointed and the
|
||||
# old occurrence is closed — all in one transaction.
|
||||
self._reconcile_paths(operation, source, destination)
|
||||
self.journal.transition(operation["id"], JournalState.DATABASE_UPDATED, fencing_token=token)
|
||||
_maybe_fault(JournalState.DATABASE_UPDATED)
|
||||
maybe_fault(JournalState.DATABASE_UPDATED)
|
||||
|
||||
# 5. Postconditions: the bytes really are at the new paths.
|
||||
self._verify(operation, destination)
|
||||
self.journal.transition(operation["id"], JournalState.VERIFIED, fencing_token=token)
|
||||
_maybe_fault(JournalState.VERIFIED)
|
||||
maybe_fault(JournalState.VERIFIED)
|
||||
|
||||
self.journal.transition(operation["id"], JournalState.COMPLETE, fencing_token=token)
|
||||
_maybe_fault(JournalState.COMPLETE)
|
||||
maybe_fault(JournalState.COMPLETE)
|
||||
|
||||
def _recheck(self, operation: dict, source: Path, destination: Path) -> None:
|
||||
if not source.exists():
|
||||
|
||||
650
photo_pipeline/services/restores.py
Normal file
650
photo_pipeline/services/restores.py
Normal file
@@ -0,0 +1,650 @@
|
||||
"""RestoreService — plan and execute safe restores (US06-04).
|
||||
|
||||
Restore is archiving read backwards, with one decisive difference: it removes
|
||||
nothing. The archived copy stays on its medium, so every failure mode here costs
|
||||
at most a discarded temporary file. What restore must never do is *lose identity*
|
||||
— the asset that comes back is the same asset, with its duplicate decision, safety
|
||||
review, analysis, and upload history intact — or *overwrite* something in the
|
||||
active library.
|
||||
|
||||
Preflight proves, per concept §9 "Restore":
|
||||
|
||||
- the recorded medium is mounted and is the right one (marker ``media_id``);
|
||||
- every selected asset is archived, its archive copy exists, and it hashes to
|
||||
exactly the bytes the database recorded — a mismatch is ``divergent`` and is
|
||||
refused, never silently accepted as "the file";
|
||||
- the destination lies inside the library, outside ``_IGNORE/``, and is free; a
|
||||
taken path is answered with a collision-free name, never an overwrite;
|
||||
- the library filesystem has room for the scope plus the configured reserve;
|
||||
- no rename, archive, or restore lease is holding the lane.
|
||||
|
||||
Blocker codes: ``no_library_root``, ``location_offline``, ``wrong_volume``,
|
||||
``unsafe_destination``, ``library_not_writable``, ``insufficient_capacity``,
|
||||
``lock_conflict``, ``rename_pending``, ``archive_pending``, ``empty_scope``,
|
||||
``not_archived``, ``archive_missing``, ``bytes_changed``.
|
||||
|
||||
Per item the sequence is:
|
||||
|
||||
```
|
||||
journal.begin (transferring) ← intent persisted BEFORE any disk change
|
||||
recheck: medium, hash, free destination, asset still archived
|
||||
copy to a temporary file beside the destination, fsync, hash it back
|
||||
atomically publish into the library
|
||||
journal → verified
|
||||
current_path = destination, availability = active, path occurrence opened
|
||||
journal → complete
|
||||
```
|
||||
|
||||
Like archiving, the confirmation token is derived from the report, so a changed
|
||||
scope, a swapped medium, or a destination that filled up invalidates it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.jobs.domain_handlers import ARCHIVE_LOCK, LIBRARY_WRITE_LOCK, UPLOAD_LOCK
|
||||
from photo_pipeline.models import ArchiveLocation, ArchiveOperation, ArchivePlan, Asset, AssetPath
|
||||
from photo_pipeline.path_policy import PathPolicyError, is_excluded, normalize_root, resolve_within
|
||||
from photo_pipeline.services import availability
|
||||
from photo_pipeline.services.archive_journal import (
|
||||
MANUAL,
|
||||
RESTORE,
|
||||
RESUMABLE,
|
||||
ArchiveJournal,
|
||||
ArchiveState,
|
||||
)
|
||||
from photo_pipeline.services.archive_transfer import (
|
||||
_clean_temp_files,
|
||||
_fsync_dir,
|
||||
_plan_dict,
|
||||
copy_verify_publish,
|
||||
)
|
||||
from photo_pipeline.services.archives import ArchiveError
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.jobs import JobService
|
||||
from photo_pipeline.services.rename_apply import PreconditionFailed, maybe_fault
|
||||
from photo_pipeline.services.rename_journal import RenameJournal
|
||||
|
||||
PREFLIGHT_VERSION = 1
|
||||
TOKEN_PREFIX = f"r{PREFLIGHT_VERSION}"
|
||||
# What a restored file is called when its original name is taken. The suffix is
|
||||
# visible on purpose: a restore that quietly reuses a name is indistinguishable
|
||||
# from an overwrite.
|
||||
RESTORED_SUFFIX = "restored"
|
||||
|
||||
LOCKS = (LIBRARY_WRITE_LOCK, UPLOAD_LOCK, ARCHIVE_LOCK)
|
||||
|
||||
APPLYABLE_PLAN_STATES = frozenset({"planned", "applying", "failed", "complete"})
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _issue(code: str, message: str) -> dict:
|
||||
return {"code": code, "message": message}
|
||||
|
||||
|
||||
class RestoreService:
|
||||
def __init__(self, session_factory: sessionmaker, *, config: Config) -> None:
|
||||
self._session_factory = session_factory
|
||||
self._config = config
|
||||
self._roots = tuple(normalize_root(root) for root in config.library_roots)
|
||||
self.journal = ArchiveJournal(session_factory)
|
||||
|
||||
# ── preflight ─────────────────────────────────────────────────────────────
|
||||
|
||||
def preflight(self, location_id: str, asset_ids: list[str] | None = None) -> dict:
|
||||
"""Validate a restore scope and issue its token. Nothing is written."""
|
||||
with self._session_factory() as session:
|
||||
location = session.get(ArchiveLocation, location_id)
|
||||
if location is None:
|
||||
raise ArchiveError("unknown_location", f"unknown archive location {location_id!r}")
|
||||
root = Path(location.root)
|
||||
online = availability.location_online(location)
|
||||
marker = availability.read_marker(root)
|
||||
report = {
|
||||
"schema_version": PREFLIGHT_VERSION,
|
||||
"location": {
|
||||
"id": location.id,
|
||||
"name": location.name,
|
||||
"root": str(root),
|
||||
"media_id": location.media_id,
|
||||
"state": _location_state(root, marker, location.media_id),
|
||||
},
|
||||
"blockers": [],
|
||||
}
|
||||
items = self._items(session, location, asset_ids, reachable=online)
|
||||
|
||||
report["blockers"] += self._destination_blockers(report["location"]["state"], root)
|
||||
report["blockers"] += self._lock_blockers()
|
||||
report["items"] = items
|
||||
report["totals"] = {
|
||||
"assets": len(items),
|
||||
"blocked": sum(1 for item in items if item["blockers"]),
|
||||
"bytes": sum(item["byte_size"] or 0 for item in items),
|
||||
}
|
||||
report["capacity"] = self._capacity(report["totals"]["bytes"])
|
||||
if not report["capacity"]["sufficient"]:
|
||||
report["blockers"].append(
|
||||
_issue(
|
||||
"insufficient_capacity",
|
||||
# No free-space number here: it drifts between two identical
|
||||
# preflights and the token is a digest of this text (US06-06).
|
||||
f"{report['totals']['bytes']} B plus a "
|
||||
f"{self._config.archive_free_space_reserve_bytes} B reserve do not fit in "
|
||||
"the library",
|
||||
)
|
||||
)
|
||||
if not items:
|
||||
report["blockers"].append(
|
||||
_issue("empty_scope", "no archived assets are in the selected scope")
|
||||
)
|
||||
report["state"] = (
|
||||
"ready"
|
||||
if not report["blockers"] and not report["totals"]["blocked"]
|
||||
else "blocked"
|
||||
)
|
||||
report["token"] = _token(report)
|
||||
report["generated_at"] = _now().isoformat()
|
||||
return report
|
||||
|
||||
def verify_token(self, token: str, location_id: str, asset_ids: list[str] | None = None) -> bool:
|
||||
return bool(token) and token == self.preflight(location_id, asset_ids)["token"]
|
||||
|
||||
def _items(
|
||||
self, session, location: ArchiveLocation, asset_ids: list[str] | None, *, reachable: bool
|
||||
) -> list[dict]:
|
||||
stmt = select(Asset).where(Asset.archive_location_id == location.id)
|
||||
if asset_ids is None:
|
||||
# A restored asset keeps its archive link; the default scope is only what
|
||||
# is still archived, so restoring twice is an empty scope, not a blocker.
|
||||
stmt = stmt.where(Asset.availability_state.in_(availability.ARCHIVED))
|
||||
else:
|
||||
stmt = stmt.where(Asset.id.in_(asset_ids))
|
||||
assets = list(session.scalars(stmt.order_by(Asset.archive_path)))
|
||||
if asset_ids is not None:
|
||||
unknown = sorted(set(asset_ids) - {asset.id for asset in assets})
|
||||
if unknown:
|
||||
raise ArchiveError(
|
||||
"unknown_asset", f"not archived at this location: {', '.join(unknown)}"
|
||||
)
|
||||
taken: set[str] = set()
|
||||
return [self._item(asset, location, reachable=reachable, taken=taken) for asset in assets]
|
||||
|
||||
def _item(self, asset: Asset, location: ArchiveLocation, *, reachable: bool, taken: set) -> dict:
|
||||
source = Path(location.root) / (asset.archive_path or "")
|
||||
blockers: list[dict] = []
|
||||
archive_sha256 = None
|
||||
|
||||
if asset.availability_state not in availability.ARCHIVED:
|
||||
blockers.append(
|
||||
_issue("not_archived", f"asset {asset.id} is {asset.availability_state}")
|
||||
)
|
||||
if reachable:
|
||||
if not source.exists():
|
||||
blockers.append(_issue("archive_missing", f"{source} is not on the medium"))
|
||||
else:
|
||||
archive_sha256 = sha256_file(source)
|
||||
if asset.current_sha256 and archive_sha256 != asset.current_sha256:
|
||||
blockers.append(
|
||||
_issue(
|
||||
"bytes_changed",
|
||||
f"{source} holds bytes that are not the recorded ones; "
|
||||
"the archived copy is divergent",
|
||||
)
|
||||
)
|
||||
|
||||
destination, destination_blockers = self._destination(asset, taken)
|
||||
blockers += destination_blockers
|
||||
if destination is not None:
|
||||
taken.add(str(destination))
|
||||
return {
|
||||
"asset_id": asset.id,
|
||||
"archive_path": asset.archive_path,
|
||||
"source_path": str(source),
|
||||
"destination_path": str(destination) if destination else None,
|
||||
"expected_sha256": asset.current_sha256,
|
||||
"archive_sha256": archive_sha256,
|
||||
"byte_size": asset.byte_size,
|
||||
"availability_state": asset.availability_state,
|
||||
"blockers": blockers,
|
||||
}
|
||||
|
||||
def _destination(self, asset: Asset, taken: set) -> tuple[Path | None, list[dict]]:
|
||||
"""A free path inside the library that mirrors the archived layout.
|
||||
|
||||
Restoring onto an existing file is never an option, so a taken name is
|
||||
answered with ``name (restored).ext`` — visible, ordinary, and impossible to
|
||||
confuse with an overwrite.
|
||||
"""
|
||||
if not self._roots:
|
||||
return None, [_issue("no_library_root", "no library root is configured")]
|
||||
root = self._roots[0]
|
||||
try:
|
||||
candidate = resolve_within(root, root / (asset.archive_path or ""))
|
||||
except PathPolicyError as error:
|
||||
return None, [_issue("unsafe_destination", str(error))]
|
||||
if is_excluded(candidate):
|
||||
return None, [
|
||||
_issue("unsafe_destination", f"{candidate} is inside an excluded (_IGNORE/) tree")
|
||||
]
|
||||
return _free_path(candidate, taken), []
|
||||
|
||||
def _destination_blockers(self, state: str, root: Path) -> list[dict]:
|
||||
blockers: list[dict] = []
|
||||
if not self._roots:
|
||||
blockers.append(_issue("no_library_root", "no library root is configured"))
|
||||
elif not os.access(self._roots[0], os.W_OK):
|
||||
blockers.append(
|
||||
_issue("library_not_writable", f"{self._roots[0]} is not writable")
|
||||
)
|
||||
if state == "offline":
|
||||
blockers.append(
|
||||
_issue("location_offline", f"the archive medium is not mounted at {root}")
|
||||
)
|
||||
elif state == "wrong_volume":
|
||||
blockers.append(_issue("wrong_volume", f"{root} holds a different archive medium"))
|
||||
return blockers
|
||||
|
||||
def _lock_blockers(self) -> list[dict]:
|
||||
blockers: list[dict] = []
|
||||
jobs = JobService(self._session_factory)
|
||||
for lock in LOCKS:
|
||||
held = jobs.blockers(lock)
|
||||
if held:
|
||||
blockers.append(
|
||||
_issue("lock_conflict", f"the {lock} lane is busy: job {held[0]['id']}")
|
||||
)
|
||||
if RenameJournal(self._session_factory).blocks_mutation():
|
||||
blockers.append(
|
||||
_issue("rename_pending", "an unresolved rename must be recovered before restoring")
|
||||
)
|
||||
if self.journal.blocks_mutation():
|
||||
blockers.append(
|
||||
_issue(
|
||||
"archive_pending",
|
||||
"an unresolved archive or restore must be recovered before restoring",
|
||||
)
|
||||
)
|
||||
return blockers
|
||||
|
||||
def _capacity(self, required: int) -> dict:
|
||||
reserve = self._config.archive_free_space_reserve_bytes
|
||||
free = shutil.disk_usage(self._roots[0]).free if self._roots else None
|
||||
return {
|
||||
"required_bytes": required,
|
||||
"reserve_bytes": reserve,
|
||||
"free_bytes": free,
|
||||
"sufficient": free is not None and free >= required + reserve,
|
||||
}
|
||||
|
||||
# ── plans ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def create(self, location_id: str, asset_ids: list[str] | None = None, *, token: str) -> dict:
|
||||
preflight = self.preflight(location_id, asset_ids)
|
||||
if not token or token != preflight["token"]:
|
||||
raise ArchiveError("stale_token", "the restore preflight changed since it was approved")
|
||||
if preflight["state"] != "ready":
|
||||
codes = ", ".join(sorted({issue["code"] for issue in preflight["blockers"]})) or "-"
|
||||
blocked = sorted(
|
||||
{issue["code"] for item in preflight["items"] for issue in item["blockers"]}
|
||||
)
|
||||
raise ArchiveError(
|
||||
"blocked", f"the restore scope is blocked: {', '.join(blocked) or codes}"
|
||||
)
|
||||
|
||||
plan_id = str(uuid.uuid4())
|
||||
with self._session_factory() as session:
|
||||
session.add(
|
||||
ArchivePlan(
|
||||
id=plan_id,
|
||||
location_id=location_id,
|
||||
token=token,
|
||||
albums=json.dumps(asset_ids) if asset_ids is not None else None,
|
||||
direction=RESTORE,
|
||||
state="planned",
|
||||
schema_version=PREFLIGHT_VERSION,
|
||||
asset_count=preflight["totals"]["assets"],
|
||||
byte_size=preflight["totals"]["bytes"],
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
for sequence, item in enumerate(preflight["items"]):
|
||||
session.add(
|
||||
ArchiveOperation(
|
||||
id=str(uuid.uuid4()),
|
||||
plan_id=plan_id,
|
||||
direction=RESTORE,
|
||||
sequence=sequence,
|
||||
album=Path(item["archive_path"]).parent.name or "(root)",
|
||||
asset_id=item["asset_id"],
|
||||
source_path=item["source_path"],
|
||||
destination_path=item["destination_path"],
|
||||
archive_path=item["archive_path"],
|
||||
expected_sha256=item["expected_sha256"],
|
||||
byte_size=item["byte_size"],
|
||||
journal_state=ArchiveState.PLANNED,
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return self.get(plan_id)
|
||||
|
||||
def get(self, plan_id: str) -> dict | None:
|
||||
with self._session_factory() as session:
|
||||
plan = session.get(ArchivePlan, plan_id)
|
||||
if plan is None or plan.direction != RESTORE:
|
||||
return None
|
||||
report = _plan_dict(plan)
|
||||
report["operations"] = self.journal.operations(plan_id)
|
||||
return report
|
||||
|
||||
def list(self) -> list[dict]:
|
||||
with self._session_factory() as session:
|
||||
rows = session.scalars(
|
||||
select(ArchivePlan)
|
||||
.where(ArchivePlan.direction == RESTORE)
|
||||
.order_by(ArchivePlan.created_at)
|
||||
)
|
||||
return [_plan_dict(row) for row in rows]
|
||||
|
||||
# ── apply ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def apply(
|
||||
self, plan_id: str, *, expected_version: int | None = None, worker_id: str = "restore"
|
||||
) -> dict:
|
||||
plan = self._require_plan(plan_id)
|
||||
if expected_version is not None and plan["version"] != expected_version:
|
||||
raise ArchiveError(
|
||||
"stale_plan",
|
||||
f"plan {plan_id} is at version {plan['version']}, expected {expected_version}",
|
||||
)
|
||||
if plan["state"] not in APPLYABLE_PLAN_STATES:
|
||||
raise ArchiveError("invalid_state", f"plan {plan_id} is {plan['state']}")
|
||||
blocking = [row for row in self.journal.incomplete() if row["plan_id"] != plan_id]
|
||||
if blocking:
|
||||
raise ArchiveError(
|
||||
"archive_pending",
|
||||
f"another archive operation is unresolved ({blocking[0]['id']}); recover it first",
|
||||
)
|
||||
|
||||
token = self._claim_plan(plan_id)
|
||||
location = self._location(plan["location_id"])
|
||||
restored = failed = skipped = 0
|
||||
for operation in self.journal.operations(plan_id):
|
||||
if operation["journal_state"] == ArchiveState.COMPLETE:
|
||||
skipped += 1
|
||||
continue
|
||||
try:
|
||||
if operation["journal_state"] == ArchiveState.VERIFIED:
|
||||
self._finish(operation, token=token)
|
||||
else:
|
||||
self._restore_one(operation, location, token=token, worker_id=worker_id)
|
||||
restored += 1
|
||||
except PreconditionFailed as error:
|
||||
self._fail(operation, token, error.code, str(error))
|
||||
failed += 1
|
||||
except Exception as error: # unexpected: record and stop touching disk
|
||||
self._fail(operation, token, "restore_error", str(error))
|
||||
failed += 1
|
||||
state = self.journal.sync_plan_state(plan_id)
|
||||
return {
|
||||
"plan_id": plan_id,
|
||||
"restored": restored,
|
||||
"failed": failed,
|
||||
"skipped": skipped,
|
||||
"state": state,
|
||||
}
|
||||
|
||||
def _restore_one(self, operation: dict, location: dict, *, token: int, worker_id: str) -> None:
|
||||
source = Path(operation["source_path"])
|
||||
destination = Path(operation["destination_path"])
|
||||
|
||||
# 1. Intent first; from here a crash is resolvable from journal + disk.
|
||||
self.journal.begin(operation["id"], worker_id=worker_id, fencing_token=token)
|
||||
maybe_fault(ArchiveState.TRANSFERRING)
|
||||
|
||||
# 2. Recheck against the medium and the library as they are right now.
|
||||
self._recheck(operation, source, destination, location)
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 3. Always copy: the archived original stays on its medium.
|
||||
copy_verify_publish(source, destination, operation["expected_sha256"])
|
||||
_fsync_dir(destination.parent)
|
||||
|
||||
if sha256_file(destination) != operation["expected_sha256"]:
|
||||
raise PreconditionFailed(
|
||||
"restore_mismatch", f"{destination} does not hold the expected bytes"
|
||||
)
|
||||
self.journal.transition(operation["id"], ArchiveState.VERIFIED, fencing_token=token)
|
||||
maybe_fault(ArchiveState.VERIFIED)
|
||||
|
||||
self._finish(self.journal.get(operation["id"]), token=token)
|
||||
|
||||
def _finish(self, operation: dict, *, token: int) -> None:
|
||||
"""Publish the restored file to the database. Idempotent, so recovery may
|
||||
replay it after a crash between the copy and the bookkeeping."""
|
||||
destination = Path(operation["destination_path"])
|
||||
if not destination.exists() or sha256_file(destination) != operation["expected_sha256"]:
|
||||
raise PreconditionFailed(
|
||||
"restore_unverified", f"{destination} is not a verified restored copy"
|
||||
)
|
||||
self._record_restored(operation, destination)
|
||||
self.journal.transition(operation["id"], ArchiveState.COMPLETE, fencing_token=token)
|
||||
maybe_fault(ArchiveState.COMPLETE)
|
||||
|
||||
def _recheck(self, operation: dict, source: Path, destination: Path, location: dict) -> None:
|
||||
root = Path(location["root"])
|
||||
if not root.is_dir() or not (root / availability.MARKER_NAME).exists():
|
||||
raise PreconditionFailed("location_offline", f"{root} is not the archive medium")
|
||||
if not source.exists():
|
||||
raise PreconditionFailed("archive_missing", f"{source} is not on the medium")
|
||||
if source.is_symlink() or destination.is_symlink():
|
||||
raise PreconditionFailed("symlink", "refusing to restore through a symlink")
|
||||
if destination.exists():
|
||||
# Never overwrite: the plan's free path was taken since it was made.
|
||||
raise PreconditionFailed(
|
||||
"destination_exists", f"destination {destination} is occupied"
|
||||
)
|
||||
if not self._inside_library(destination):
|
||||
raise PreconditionFailed(
|
||||
"destination_escape", f"{destination} is outside the library roots"
|
||||
)
|
||||
if sha256_file(source) != operation["expected_sha256"]:
|
||||
self._mark_divergent(operation["asset_id"])
|
||||
raise PreconditionFailed(
|
||||
"bytes_changed", f"{source} changed since the plan was approved"
|
||||
)
|
||||
with self._session_factory() as session:
|
||||
asset = session.get(Asset, operation["asset_id"])
|
||||
if asset is None or asset.availability_state not in availability.ARCHIVED:
|
||||
raise PreconditionFailed(
|
||||
"not_archived", f"asset {operation['asset_id']} is no longer archived"
|
||||
)
|
||||
|
||||
def _inside_library(self, destination: Path) -> bool:
|
||||
for root in self._roots:
|
||||
try:
|
||||
resolve_within(root, destination)
|
||||
return True
|
||||
except PathPolicyError:
|
||||
continue
|
||||
return False
|
||||
|
||||
# ── database ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _record_restored(self, operation: dict, destination: Path) -> None:
|
||||
"""The bytes are back in the library: open the new active occurrence and set
|
||||
availability. Identity, decisions, and history are untouched — that is the
|
||||
entire point of restoring rather than re-importing."""
|
||||
now = _now()
|
||||
with self._session_factory() as session:
|
||||
asset = session.get(Asset, operation["asset_id"])
|
||||
if asset is None:
|
||||
raise PreconditionFailed(
|
||||
"asset_missing", f"asset {operation['asset_id']} no longer exists"
|
||||
)
|
||||
# A restored asset may be returning to a path it once held, so only an
|
||||
# *open* occurrence counts as already registered — that is what keeps
|
||||
# recovery idempotent without collapsing the path history.
|
||||
recorded = session.scalar(
|
||||
select(AssetPath).where(
|
||||
AssetPath.asset_id == asset.id,
|
||||
AssetPath.path == str(destination),
|
||||
AssetPath.valid_until.is_(None),
|
||||
)
|
||||
)
|
||||
if recorded is None: # idempotent: recovery may replay this
|
||||
session.add(
|
||||
AssetPath(
|
||||
asset_id=asset.id,
|
||||
path=str(destination),
|
||||
valid_from=now,
|
||||
reason="restore",
|
||||
)
|
||||
)
|
||||
asset.current_path = str(destination)
|
||||
asset.availability_state = availability.ACTIVE
|
||||
asset.missing_at = None
|
||||
# The archive copy stays where it is; keeping the link means a restored
|
||||
# asset still knows which medium holds its archived bytes.
|
||||
asset.archive_divergent_at = None
|
||||
asset.state_version += 1
|
||||
asset.updated_at = now
|
||||
session.commit()
|
||||
|
||||
def _mark_divergent(self, asset_id: str) -> None:
|
||||
"""Record that the archived copy is not the recorded file. Durable, because
|
||||
the next restore attempt must not rediscover this from scratch."""
|
||||
with self._session_factory() as session:
|
||||
asset = session.get(Asset, asset_id)
|
||||
if asset is None:
|
||||
return
|
||||
asset.archive_divergent_at = _now()
|
||||
asset.state_version += 1
|
||||
session.commit()
|
||||
|
||||
# ── recovery ──────────────────────────────────────────────────────────────
|
||||
|
||||
def recover(self, *, worker_id: str = "restore-recovery") -> dict:
|
||||
"""Resolve every incomplete restore from journal + disk evidence.
|
||||
|
||||
A restore never removed anything, so ``resumable`` simply discards the
|
||||
temporary debris and re-plans the item; ``forward`` finishes the bookkeeping
|
||||
for a published file; ``manual`` is left untouched and keeps blocking.
|
||||
"""
|
||||
results = {"resumed": 0, "completed": 0, "manual": 0}
|
||||
touched: set[str] = set()
|
||||
for verdict in self.journal.classify_all(direction=RESTORE):
|
||||
operation = self.journal.get(verdict["operation_id"])
|
||||
touched.add(operation["plan_id"])
|
||||
token = (operation["fencing_token"] or 0) + 1
|
||||
if verdict["classification"] == MANUAL:
|
||||
results["manual"] += 1
|
||||
continue
|
||||
if verdict["classification"] == RESUMABLE:
|
||||
_clean_temp_files(Path(operation["destination_path"]).parent)
|
||||
self.journal.transition(operation["id"], ArchiveState.PLANNED, fencing_token=token)
|
||||
results["resumed"] += 1
|
||||
continue
|
||||
try:
|
||||
self._finish(operation, token=token)
|
||||
results["completed"] += 1
|
||||
except PreconditionFailed as error:
|
||||
self._fail(operation, token, error.code, str(error))
|
||||
results["manual"] += 1
|
||||
for plan_id in touched:
|
||||
self.journal.sync_plan_state(plan_id)
|
||||
return results
|
||||
|
||||
def recovery_status(self) -> dict:
|
||||
verdicts = self.journal.classify_all(direction=RESTORE)
|
||||
return {
|
||||
"operations": verdicts,
|
||||
"manual": [v for v in verdicts if v["classification"] == MANUAL],
|
||||
"blocks_mutation": self.journal.blocks_mutation(),
|
||||
}
|
||||
|
||||
# ── helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
def _fail(self, operation: dict, token: int, code: str, message: str) -> None:
|
||||
self.journal.transition(
|
||||
operation["id"], ArchiveState.FAILED, fencing_token=token, error=(code, message)
|
||||
)
|
||||
|
||||
def _require_plan(self, plan_id: str) -> dict:
|
||||
plan = self.get(plan_id)
|
||||
if plan is None:
|
||||
raise ArchiveError("unknown_plan", f"unknown restore plan {plan_id!r}")
|
||||
return plan
|
||||
|
||||
def _location(self, location_id: str) -> dict:
|
||||
with self._session_factory() as session:
|
||||
location = session.get(ArchiveLocation, location_id)
|
||||
if location is None:
|
||||
raise ArchiveError("unknown_location", f"unknown archive location {location_id!r}")
|
||||
return {"id": location.id, "root": location.root, "media_id": location.media_id}
|
||||
|
||||
def _claim_plan(self, plan_id: str) -> int:
|
||||
with self._session_factory() as session:
|
||||
plan = session.get(ArchivePlan, plan_id)
|
||||
plan.version += 1
|
||||
plan.state = "applying"
|
||||
plan.updated_at = _now()
|
||||
token = plan.version
|
||||
session.commit()
|
||||
return token
|
||||
|
||||
|
||||
# ── module helpers ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _location_state(root: Path, marker: dict | None, media_id: str) -> str:
|
||||
if not root.is_dir() or marker is None:
|
||||
return "offline"
|
||||
return "online" if marker.get("media_id") == media_id else "wrong_volume"
|
||||
|
||||
|
||||
def _free_path(candidate: Path, taken: set) -> Path:
|
||||
"""``a.jpg`` → ``a (restored).jpg`` → ``a (restored 2).jpg`` …
|
||||
|
||||
``taken`` holds the destinations already claimed by earlier items of the same
|
||||
plan, so two restores in one scope cannot plan the same path.
|
||||
"""
|
||||
if not candidate.exists() and str(candidate) not in taken:
|
||||
return candidate
|
||||
stem, suffix = candidate.stem, candidate.suffix
|
||||
attempt = 1
|
||||
while True:
|
||||
label = RESTORED_SUFFIX if attempt == 1 else f"{RESTORED_SUFFIX} {attempt}"
|
||||
alternative = candidate.with_name(f"{stem} ({label}){suffix}")
|
||||
if not alternative.exists() and str(alternative) not in taken:
|
||||
return alternative
|
||||
attempt += 1
|
||||
|
||||
|
||||
def _token(report: dict) -> str:
|
||||
"""Digest of everything the report asserts about the scope and the medium.
|
||||
|
||||
Free space is excluded: it drifts constantly without changing what a restore
|
||||
would do, and the capacity verdict itself is part of the digest.
|
||||
"""
|
||||
payload = {key: value for key, value in report.items() if key not in ("generated_at", "token")}
|
||||
payload["capacity"] = {
|
||||
key: value for key, value in payload["capacity"].items() if key != "free_bytes"
|
||||
}
|
||||
digest = hashlib.sha256(
|
||||
json.dumps(payload, sort_keys=True, ensure_ascii=False, default=str).encode("utf-8")
|
||||
).hexdigest()
|
||||
return f"{TOKEN_PREFIX}:{digest}"
|
||||
@@ -8,6 +8,11 @@ an EXIF-only edit reuses the file while a real pixel change invalidates it; writ
|
||||
are atomic and the cache is bounded by an LRU quota. Failures are persisted as typed
|
||||
errors so a broken original is not retried on every request.
|
||||
|
||||
An archived asset is served from its medium when that medium is mounted, and from
|
||||
its *protected* preview when it is not (US06-03). Protected previews are evidence,
|
||||
not cache: the quota never evicts them, because the original they describe may be
|
||||
unreachable when duplicate review needs it.
|
||||
|
||||
Reuses photo_analyzer.prepare_image decode/resize/HEIC handling, adding the missing
|
||||
EXIF-orientation step, WebP output, and a managed cache (donor_ledger.yaml:
|
||||
pa-imaging).
|
||||
@@ -26,6 +31,7 @@ from sqlalchemy.orm import sessionmaker
|
||||
from photo_pipeline import path_policy
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.models import Asset, Thumbnail
|
||||
from photo_pipeline.services import availability
|
||||
|
||||
# Best-effort HEIC support: registered only if the optional decoder is installed.
|
||||
try: # pragma: no cover - depends on an optional native dependency
|
||||
@@ -38,6 +44,8 @@ except Exception: # pragma: no cover
|
||||
SIZES = (256, 512, 1280)
|
||||
THUMB_VERSION = 1
|
||||
THUMB_FORMAT = "webp"
|
||||
# The size kept as durable comparison evidence for archived assets (concept §9).
|
||||
PROTECTED_SIZE = 1280
|
||||
|
||||
|
||||
class ThumbnailError(RuntimeError):
|
||||
@@ -86,7 +94,12 @@ class ThumbnailService:
|
||||
self._config = config
|
||||
self._cache_dir = config.thumbnail_cache_dir
|
||||
|
||||
def generate(self, asset_id: str, size: int) -> Path:
|
||||
def generate(
|
||||
self, asset_id: str, size: int, *, protected: bool = False, source: Path | None = None
|
||||
) -> Path:
|
||||
"""Render (or reuse) a preview. ``source`` overrides where the bytes are read
|
||||
from — the archiver passes its verified archive copy, which the database does
|
||||
not yet point at while the transfer is still in flight."""
|
||||
if size not in SIZES:
|
||||
raise InvalidSize(f"size must be one of {SIZES}")
|
||||
|
||||
@@ -94,9 +107,7 @@ class ThumbnailService:
|
||||
asset = session.get(Asset, asset_id)
|
||||
if asset is None:
|
||||
raise ThumbnailNotFound(f"unknown asset {asset_id}")
|
||||
if asset.availability_state != "active" or not asset.current_path:
|
||||
raise ThumbnailUnavailable(f"asset {asset_id} has no active file")
|
||||
self._validate_path(asset.current_path)
|
||||
archived = asset.availability_state in availability.ARCHIVED
|
||||
cache_key = self._cache_key(asset, size)
|
||||
|
||||
row = session.get(Thumbnail, cache_key)
|
||||
@@ -107,9 +118,18 @@ class ThumbnailService:
|
||||
)
|
||||
if row.path and Path(row.path).exists():
|
||||
_touch(row.path)
|
||||
if protected and not row.protected:
|
||||
self._protect(cache_key)
|
||||
return Path(row.path)
|
||||
|
||||
source = asset.current_path
|
||||
# An archived original is read from its medium; when that medium is not
|
||||
# mounted the retained preview above is the only evidence there is.
|
||||
source = source or availability.readable_path(session, asset)
|
||||
if source is None:
|
||||
raise ThumbnailUnavailable(f"asset {asset_id} has no readable file")
|
||||
source = str(source)
|
||||
if source == asset.current_path:
|
||||
self._validate_path(source) # archive roots lie outside the library
|
||||
|
||||
# Rendering happens outside the DB session (no transaction held during I/O).
|
||||
try:
|
||||
@@ -119,10 +139,61 @@ class ThumbnailService:
|
||||
self._record_error(cache_key, asset_id, size, error.code)
|
||||
raise
|
||||
|
||||
self._record_ready(cache_key, asset_id, size, rendered)
|
||||
# Archived assets keep their preview permanently: it is the comparison
|
||||
# evidence that survives the original leaving active storage.
|
||||
self._record_ready(cache_key, asset_id, size, rendered, protected=protected or archived)
|
||||
self._enforce_quota(keep=rendered["path"])
|
||||
return Path(rendered["path"])
|
||||
|
||||
def ensure_protected(self, asset_id: str, *, source: Path | None = None) -> dict:
|
||||
"""Produce (or confirm) the durable comparison preview for an asset.
|
||||
|
||||
Returns evidence rather than raising, because the caller — archive
|
||||
preflight and the transfer itself — decides what an unrenderable original
|
||||
means. ``unsupported`` is a recorded property of the file, not a failure of
|
||||
the policy: its hashes and metadata remain the comparison evidence.
|
||||
"""
|
||||
try:
|
||||
path = self.generate(asset_id, PROTECTED_SIZE, protected=True, source=source)
|
||||
except tuple(_PERSISTED_ERRORS) as error:
|
||||
return {"state": "unsupported", "error_code": error.code, "path": None}
|
||||
except ThumbnailError as error:
|
||||
return {"state": "unavailable", "error_code": error.code, "path": None}
|
||||
return {"state": "ready", "error_code": None, "path": str(path)}
|
||||
|
||||
def evidence(self, asset_id: str) -> dict:
|
||||
"""What durable preview this asset has right now, without rendering."""
|
||||
with self._session_factory() as session:
|
||||
rows = list(
|
||||
session.execute(
|
||||
select(Thumbnail).where(Thumbnail.asset_id == asset_id)
|
||||
).scalars()
|
||||
)
|
||||
for row in rows:
|
||||
if row.state == "ready" and row.path and Path(row.path).exists():
|
||||
return {
|
||||
"state": "ready",
|
||||
"protected": bool(row.protected),
|
||||
"size": row.size,
|
||||
"error_code": None,
|
||||
}
|
||||
for row in rows:
|
||||
if row.state == "error":
|
||||
return {
|
||||
"state": "unsupported",
|
||||
"protected": False,
|
||||
"size": row.size,
|
||||
"error_code": row.error_code,
|
||||
}
|
||||
return {"state": "missing", "protected": False, "size": None, "error_code": None}
|
||||
|
||||
def _protect(self, cache_key: str) -> None:
|
||||
with self._session_factory() as session:
|
||||
row = session.get(Thumbnail, cache_key)
|
||||
if row is not None:
|
||||
row.protected = True
|
||||
session.commit()
|
||||
|
||||
# ── path safety ──────────────────────────────────────────────────────────
|
||||
def _validate_path(self, current_path: str) -> None:
|
||||
path = Path(current_path)
|
||||
@@ -184,7 +255,9 @@ class ThumbnailService:
|
||||
}
|
||||
|
||||
# ── persistence ────────────────────────────────────────────────────────────
|
||||
def _record_ready(self, cache_key: str, asset_id: str, size: int, rendered: dict) -> None:
|
||||
def _record_ready(
|
||||
self, cache_key: str, asset_id: str, size: int, rendered: dict, *, protected: bool = False
|
||||
) -> None:
|
||||
with self._session_factory() as session:
|
||||
session.merge(
|
||||
Thumbnail(
|
||||
@@ -197,6 +270,7 @@ class ThumbnailService:
|
||||
width=rendered["width"],
|
||||
height=rendered["height"],
|
||||
format=rendered["format"],
|
||||
protected=protected,
|
||||
)
|
||||
)
|
||||
try:
|
||||
@@ -230,6 +304,10 @@ class ThumbnailService:
|
||||
if total <= quota:
|
||||
return
|
||||
files.sort(key=lambda f: f.stat().st_mtime) # least-recently-used first
|
||||
# Protected previews are evidence, not cache: an archived original cannot be
|
||||
# re-rendered once its medium is away, so eviction never touches them.
|
||||
protected = self._protected_paths()
|
||||
files = [f for f in files if str(f) not in protected]
|
||||
keep_path = str(Path(keep)) if keep else None
|
||||
evicted: list[str] = []
|
||||
for f in files:
|
||||
@@ -247,6 +325,16 @@ class ThumbnailService:
|
||||
if evicted:
|
||||
self._forget(evicted)
|
||||
|
||||
def _protected_paths(self) -> set[str]:
|
||||
with self._session_factory() as session:
|
||||
return {
|
||||
row.path
|
||||
for row in session.execute(
|
||||
select(Thumbnail).where(Thumbnail.protected.is_(True))
|
||||
).scalars()
|
||||
if row.path
|
||||
}
|
||||
|
||||
def _forget(self, paths: list[str]) -> None:
|
||||
with self._session_factory() as session:
|
||||
rows = session.execute(
|
||||
|
||||
@@ -16,12 +16,14 @@ cannot undo, so the discipline is:
|
||||
- **an interrupted attempt is uncertain, not failed.** Immich may have accepted
|
||||
files the app never saw a report for, so ``recover`` marks a batch whose worker
|
||||
vanished ``unknown_requires_verification`` (concept §15) instead of retrying it
|
||||
blindly. Resolving that is US05-04.
|
||||
blindly. :func:`~photo_pipeline.services.upload_verification.retry_blockers`
|
||||
(US05-04) is what decides whether an attempt may start at all.
|
||||
|
||||
Per-asset upload *results* are deliberately not interpreted here — parsing the
|
||||
report is US05-03. What this story guarantees is that the report exists, is bounded,
|
||||
carries no credential, and that the batch's state is an honest description of what
|
||||
the process did.
|
||||
Per-asset upload *results* are not interpreted here: after the attempt ends the
|
||||
report is handed to :class:`~photo_pipeline.services.upload_reports.
|
||||
UploadReportService` (US05-03), which classifies every file. The batch ``state``
|
||||
stays an honest description of what the *process* did; ``outcome_state`` says
|
||||
whether the report's evidence is complete enough to trust.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -38,6 +40,8 @@ from photo_pipeline.config import Config
|
||||
from photo_pipeline.integrations import immich_go
|
||||
from photo_pipeline.models import UploadBatch, UploadItem
|
||||
from photo_pipeline.services.hashing import sha1_file
|
||||
from photo_pipeline.services.upload_reports import UploadReportService
|
||||
from photo_pipeline.services.upload_verification import retry_blockers
|
||||
from photo_pipeline.services.uploads import UploadService
|
||||
|
||||
|
||||
@@ -60,6 +64,7 @@ RUNNABLE_STATES = frozenset({BatchState.PLANNED, BatchState.FAILED, BatchState.C
|
||||
# States whose batch is still the live one for its album.
|
||||
OPEN_STATES = frozenset({BatchState.PLANNED, BatchState.RUNNING, BatchState.CANCELLING})
|
||||
|
||||
|
||||
class ItemState:
|
||||
PENDING = "pending"
|
||||
# ``sent`` means the batch process exited cleanly, not that Immich confirmed the
|
||||
@@ -160,13 +165,11 @@ class UploadBatchService:
|
||||
def run(self, batch_id: str, *, worker_id: str = "uploader", cancelled=None) -> dict:
|
||||
"""Run one batch to completion. Blocks for the duration of the upload."""
|
||||
batch = self._require(batch_id)
|
||||
if batch["state"] not in RUNNABLE_STATES:
|
||||
code = (
|
||||
"requires_verification"
|
||||
if batch["state"] == BatchState.UNKNOWN
|
||||
else "not_runnable"
|
||||
)
|
||||
raise BatchError(code, f"batch {batch_id} is {batch['state']}")
|
||||
# Retry policy (US05-04): a safe failure may run again, an uncertain outcome
|
||||
# or bytes edited after upload may not.
|
||||
blocked = retry_blockers(batch)
|
||||
if blocked:
|
||||
raise BatchError(blocked[0]["code"], blocked[0]["message"])
|
||||
busy = [row for row in self.list() if row["id"] != batch_id and row["state"] in LANE_STATES]
|
||||
if busy:
|
||||
raise BatchConflict("lane_busy", f"upload batch {busy[0]['id']} is still running")
|
||||
@@ -187,9 +190,7 @@ class UploadBatchService:
|
||||
|
||||
token = self._claim(batch_id, worker_id=worker_id)
|
||||
attempt = self.get(batch_id)["attempt_count"]
|
||||
report_path = (
|
||||
Path(self._config.data_dir) / "uploads" / f"{batch_id}-attempt-{attempt}.log"
|
||||
)
|
||||
report_path = Path(self._config.data_dir) / "uploads" / f"{batch_id}-attempt-{attempt}.log"
|
||||
key = self._config.immich_api_key.get_secret_value() if self._config.immich_api_key else ""
|
||||
command = immich_go.build_command(
|
||||
binary=self._config.immich_go_binary,
|
||||
@@ -238,6 +239,13 @@ class UploadBatchService:
|
||||
self._finish(batch_id, token=token, state=state, error=error, result=result)
|
||||
if item_state:
|
||||
self._set_items(batch_id, item_state)
|
||||
# The report is the only evidence of what happened per file, so it is read
|
||||
# while it is fresh (US05-03). A parse failure must not lose the batch
|
||||
# outcome that was just recorded; the items simply stay unknown.
|
||||
try:
|
||||
UploadReportService(self._session_factory).ingest(batch_id)
|
||||
except OSError:
|
||||
pass
|
||||
return self.get(batch_id)
|
||||
|
||||
def cancel(self, batch_id: str) -> dict:
|
||||
@@ -389,7 +397,7 @@ class UploadBatchService:
|
||||
|
||||
|
||||
def _batch_dict(row: UploadBatch, items: list[UploadItem]) -> dict:
|
||||
return {
|
||||
batch = {
|
||||
"id": row.id,
|
||||
"album": row.album,
|
||||
"folder": row.folder,
|
||||
@@ -409,6 +417,18 @@ def _batch_dict(row: UploadBatch, items: list[UploadItem]) -> dict:
|
||||
"exit_code": row.exit_code,
|
||||
"error_code": row.error_code,
|
||||
"error_message": row.error_message,
|
||||
# Parsed report evidence (US05-03): what the uploader said per file, and
|
||||
# whether that evidence is complete enough to be trusted.
|
||||
"parser": row.parser,
|
||||
"parser_version": row.parser_version,
|
||||
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
|
||||
"outcome_state": row.outcome_state,
|
||||
"outcome_counts": json.loads(row.outcome_counts) if row.outcome_counts else None,
|
||||
"report_counts": json.loads(row.report_counts) if row.report_counts else None,
|
||||
# Verification evidence (US05-04). ``stale_bytes`` is the visible warning
|
||||
# that an uploaded file has since been edited.
|
||||
"verified_at": row.verified_at.isoformat() if row.verified_at else None,
|
||||
"stale_bytes": row.stale_bytes,
|
||||
"started_at": row.started_at.isoformat() if row.started_at else None,
|
||||
"finished_at": row.finished_at.isoformat() if row.finished_at else None,
|
||||
"items": [
|
||||
@@ -418,7 +438,19 @@ def _batch_dict(row: UploadBatch, items: list[UploadItem]) -> dict:
|
||||
"sha256": item.sha256,
|
||||
"sha1": item.sha1,
|
||||
"state": item.state,
|
||||
"outcome": item.outcome,
|
||||
"evidence": item.evidence,
|
||||
"outcome_at": item.outcome_at.isoformat() if item.outcome_at else None,
|
||||
"verification": item.verification,
|
||||
"verified_at": item.verified_at.isoformat() if item.verified_at else None,
|
||||
"observed_sha256": item.observed_sha256,
|
||||
"changed_after_upload": item.changed_after_upload,
|
||||
}
|
||||
for item in items
|
||||
],
|
||||
}
|
||||
# Why this batch may not be (re)started, from the one place that decides it
|
||||
# (US05-04). Carried in the record so the browser can hide an action the server
|
||||
# would refuse instead of re-implementing the policy (US05-05).
|
||||
batch["retry_blockers"] = retry_blockers(batch)
|
||||
return batch
|
||||
|
||||
140
photo_pipeline/services/upload_reports.py
Normal file
140
photo_pipeline/services/upload_reports.py
Normal file
@@ -0,0 +1,140 @@
|
||||
"""UploadReportService — durable per-item upload outcomes (US05-03).
|
||||
|
||||
US05-02 proves the uploader ran and kept a bounded, credential-free report. This
|
||||
service is the step that reads that report and answers the operator's actual
|
||||
question — *what happened to each photo?* — without ever guessing in the app's
|
||||
favour:
|
||||
|
||||
- **an item is ``unknown`` until the report says otherwise.** A file the report
|
||||
never mentions, a report from an unpinned uploader version, and a line the
|
||||
grammar does not recognise all leave the item ``unknown``. Exit code 0 is
|
||||
evidence the *process* ended well, never evidence that a particular file reached
|
||||
Immich (concept §15 "wrong upload state").
|
||||
- **counts are reconciled, not trusted.** The uploader's own summary is stored
|
||||
next to the totals derived from the items. A disagreement makes the batch
|
||||
``requires_verification`` even when every line parsed.
|
||||
- **reprocessing is idempotent.** Outcomes are keyed by ``(batch_id, asset_id)``
|
||||
and rewritten in place, so parsing the same report again — after a restart, or
|
||||
because the operator asked — converges on the same rows and the same counts.
|
||||
|
||||
The batch's own ``state`` still describes the process (US05-02's contract);
|
||||
``outcome_state`` describes the evidence. Turning ``requires_verification`` into
|
||||
verification, retry, and manual resolution is US05-04.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.integrations import immich_go_report as report_parser
|
||||
from photo_pipeline.models import UploadBatch, UploadItem
|
||||
|
||||
VERIFIED = "verified"
|
||||
REQUIRES_VERIFICATION = "requires_verification"
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _key(path: str) -> str:
|
||||
return os.path.normpath(path)
|
||||
|
||||
|
||||
class UploadReportService:
|
||||
def __init__(self, session_factory: sessionmaker) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
def ingest(self, batch_id: str) -> dict:
|
||||
"""Parse this batch's report and persist an outcome for every item.
|
||||
|
||||
Returns ``{"parser", "parser_version", "outcome_state", "counts",
|
||||
"report_counts", "unmatched", "unparsed"}``. Safe to call repeatedly.
|
||||
"""
|
||||
with self._session_factory() as session:
|
||||
batch = session.get(UploadBatch, batch_id)
|
||||
if batch is None:
|
||||
raise KeyError(f"unknown upload batch {batch_id!r}")
|
||||
items = list(session.scalars(select(UploadItem).where(UploadItem.batch_id == batch_id)))
|
||||
parsed = report_parser.parse(_read_report(batch.report_path), batch.uploader_version)
|
||||
|
||||
by_path = {_key(entry["path"]): entry for entry in parsed["entries"]}
|
||||
# Basenames are the fallback: an uploader may log a path relative to the
|
||||
# folder it was given. Ambiguous basenames are dropped rather than
|
||||
# guessed at.
|
||||
by_name: dict[str, dict] = {}
|
||||
for key, entry in by_path.items():
|
||||
name = os.path.basename(key)
|
||||
by_name[name] = None if name in by_name else entry
|
||||
|
||||
matched: set[str] = set()
|
||||
counts = {outcome: 0 for outcome in report_parser.OUTCOMES}
|
||||
for item in items:
|
||||
key = _key(item.path)
|
||||
entry = by_path.get(key) or by_name.get(os.path.basename(key))
|
||||
if entry is not None:
|
||||
matched.add(_key(entry["path"]))
|
||||
item.outcome = entry["outcome"] if entry else report_parser.UNKNOWN
|
||||
item.evidence = entry["evidence"] if entry else None
|
||||
item.outcome_at = _now()
|
||||
counts[item.outcome] += 1
|
||||
|
||||
unmatched = sorted(set(by_path) - matched)
|
||||
reconciled = _reconciles(parsed["counts"], counts)
|
||||
state = (
|
||||
VERIFIED
|
||||
if (
|
||||
parsed["supported"]
|
||||
and items
|
||||
and not counts[report_parser.UNKNOWN]
|
||||
and not unmatched
|
||||
and not parsed["entries_truncated"]
|
||||
and reconciled
|
||||
)
|
||||
else REQUIRES_VERIFICATION
|
||||
)
|
||||
|
||||
batch.parser = parsed["parser"]
|
||||
batch.parser_version = parsed["parser_version"]
|
||||
batch.parsed_at = _now()
|
||||
batch.outcome_state = state
|
||||
batch.outcome_counts = json.dumps(counts, sort_keys=True)
|
||||
batch.report_counts = (
|
||||
json.dumps(parsed["counts"], sort_keys=True) if parsed["counts"] else None
|
||||
)
|
||||
session.commit()
|
||||
|
||||
return {
|
||||
"parser": parsed["parser"],
|
||||
"parser_version": parsed["parser_version"],
|
||||
"outcome_state": state,
|
||||
"counts": counts,
|
||||
"report_counts": parsed["counts"],
|
||||
"unmatched": unmatched,
|
||||
"unparsed": parsed["unparsed"],
|
||||
}
|
||||
|
||||
|
||||
def _read_report(report_path: str | None) -> str:
|
||||
"""The raw report, or ``""`` when the attempt never produced one."""
|
||||
if not report_path:
|
||||
return ""
|
||||
path = Path(report_path)
|
||||
return path.read_text(errors="replace") if path.exists() else ""
|
||||
|
||||
|
||||
def _reconciles(report_counts: dict | None, derived: dict) -> bool:
|
||||
"""Whether the uploader's own totals agree with the per-item outcomes.
|
||||
|
||||
No summary is not a disagreement — most of the uncertainty it would catch is
|
||||
already caught by unmatched entries and unknown items.
|
||||
"""
|
||||
if not report_counts:
|
||||
return True
|
||||
return all(derived.get(outcome) == total for outcome, total in report_counts.items())
|
||||
349
photo_pipeline/services/upload_verification.py
Normal file
349
photo_pipeline/services/upload_verification.py
Normal file
@@ -0,0 +1,349 @@
|
||||
"""UploadVerificationService — resolving uncertain uploads (US05-04).
|
||||
|
||||
US05-03 leaves a batch honest but sometimes uncertain: a killed uploader, an
|
||||
unpinned report grammar, or a file the report never mentioned all end as
|
||||
``unknown``. Retrying such a batch is the dangerous move — Immich may already hold
|
||||
the files, and a blind retry is how a lost response turns into a second server
|
||||
asset. So this service resolves uncertainty *before* anything is re-run:
|
||||
|
||||
- **evidence beats the report.** Verification asks Immich itself whether it holds
|
||||
the exact SHA-1 the batch recorded before uploading. That answer is authoritative
|
||||
over the uploader's text, in both directions: present makes an unknown item
|
||||
``uploaded``, absent makes it ``failed`` and therefore safe to retry.
|
||||
- **no answer is never "no".** An unreachable server, missing credentials, or a
|
||||
response this adapter will not interpret leave the item ``inconclusive``. The
|
||||
batch stays ``unknown_requires_verification`` and stays un-runnable.
|
||||
- **changed bytes are a stale warning, not a silent re-upload.** Verification
|
||||
re-hashes what is on disk. A file edited after its upload is flagged, the batch
|
||||
is marked ``stale_bytes``, and re-running it is refused: uploading again would
|
||||
create or upgrade a server asset the user never approved (concept §8).
|
||||
- **manual resolution is evidence, not permission.** An operator may record what
|
||||
they checked in Immich, but only with a non-empty note and their identity, and
|
||||
every decision is appended to an immutable history alongside the server answers.
|
||||
|
||||
Retry policy lives in :func:`retry_blockers`, which :class:`UploadBatchService`
|
||||
enforces before every attempt and the API surfaces as ``409``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.integrations import immich_go
|
||||
from photo_pipeline.integrations import immich_go_report as report_parser
|
||||
from photo_pipeline.models import UploadBatch, UploadItem, UploadVerification
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.upload_reports import REQUIRES_VERIFICATION, VERIFIED
|
||||
|
||||
PRESENT = "present"
|
||||
ABSENT = "absent"
|
||||
INCONCLUSIVE = "inconclusive"
|
||||
MANUAL = "manual"
|
||||
|
||||
MAX_EVIDENCE_CHARS = 500
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class VerificationError(RuntimeError):
|
||||
"""The request cannot be carried out (unknown batch/item, missing evidence)."""
|
||||
|
||||
def __init__(self, code: str, message: str) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
|
||||
|
||||
class UploadVerificationService:
|
||||
def __init__(self, session_factory: sessionmaker, *, config: Config) -> None:
|
||||
self._session_factory = session_factory
|
||||
self._config = config
|
||||
|
||||
# ── verification ──────────────────────────────────────────────────────────
|
||||
|
||||
def verify(self, batch_id: str) -> dict:
|
||||
"""Check every item of a batch against the server and the bytes on disk.
|
||||
|
||||
Returns ``{"batch_id", "state", "outcome_state", "stale_bytes",
|
||||
"server_reachable", "detail", "counts", "items"}``. Safe to call
|
||||
repeatedly: the same evidence produces the same rows, and each run appends
|
||||
its own history entries.
|
||||
"""
|
||||
with self._session_factory() as session:
|
||||
batch = session.get(UploadBatch, batch_id)
|
||||
if batch is None:
|
||||
raise VerificationError("not_found", f"unknown upload batch {batch_id!r}")
|
||||
items = list(
|
||||
session.scalars(
|
||||
select(UploadItem)
|
||||
.where(UploadItem.batch_id == batch_id)
|
||||
.order_by(UploadItem.path)
|
||||
)
|
||||
)
|
||||
answer = immich_go.bulk_upload_check(
|
||||
self._config.immich_server_url,
|
||||
self._config.immich_api_key.get_secret_value()
|
||||
if self._config.immich_api_key
|
||||
else None,
|
||||
{item.asset_id: item.sha1 for item in items if item.sha1},
|
||||
)
|
||||
detail = answer["detail"]
|
||||
|
||||
for item in items:
|
||||
observed, changed = _current_bytes(item)
|
||||
held = answer["present"].get(item.asset_id)
|
||||
if item.sha1 is None:
|
||||
result, evidence = INCONCLUSIVE, "no upload hash was recorded for this file"
|
||||
elif held is True:
|
||||
result, evidence = PRESENT, f"Immich holds sha1 {item.sha1}"
|
||||
elif held is False:
|
||||
result, evidence = ABSENT, f"Immich does not hold sha1 {item.sha1}"
|
||||
else:
|
||||
result = INCONCLUSIVE
|
||||
evidence = detail or "the server did not classify these bytes"
|
||||
|
||||
item.verification = result
|
||||
item.verified_at = _now()
|
||||
item.observed_sha256 = observed
|
||||
item.changed_after_upload = changed
|
||||
# The server is authoritative over the report text — but only when
|
||||
# it actually answered.
|
||||
if result == PRESENT:
|
||||
item.outcome = report_parser.UPLOADED
|
||||
item.evidence = evidence
|
||||
item.outcome_at = _now()
|
||||
elif result == ABSENT:
|
||||
item.outcome = report_parser.FAILED
|
||||
item.evidence = evidence
|
||||
item.outcome_at = _now()
|
||||
session.add(
|
||||
_event(
|
||||
batch_id,
|
||||
item.asset_id,
|
||||
action="verify",
|
||||
source="immich_api",
|
||||
result=result,
|
||||
outcome=item.outcome,
|
||||
evidence=evidence,
|
||||
)
|
||||
)
|
||||
|
||||
_resolve_batch(batch, items)
|
||||
session.commit()
|
||||
return _report(batch, items, server_reachable=answer["reachable"], detail=detail)
|
||||
|
||||
# ── manual resolution ─────────────────────────────────────────────────────
|
||||
|
||||
def resolve(
|
||||
self, batch_id: str, asset_id: str, *, outcome: str, evidence: str, actor: str
|
||||
) -> dict:
|
||||
"""Record an operator's own verification of one item.
|
||||
|
||||
``evidence`` and ``actor`` are mandatory: a manual resolution is only worth
|
||||
keeping if it says what was checked and who checked it.
|
||||
"""
|
||||
if outcome not in report_parser.OUTCOMES:
|
||||
raise VerificationError("invalid_outcome", f"unknown upload outcome {outcome!r}")
|
||||
evidence = (evidence or "").strip()
|
||||
actor = (actor or "").strip()
|
||||
if not evidence:
|
||||
raise VerificationError("evidence_required", "a manual resolution must record evidence")
|
||||
if not actor:
|
||||
raise VerificationError("actor_required", "a manual resolution must record its author")
|
||||
|
||||
with self._session_factory() as session:
|
||||
batch = session.get(UploadBatch, batch_id)
|
||||
if batch is None:
|
||||
raise VerificationError("not_found", f"unknown upload batch {batch_id!r}")
|
||||
item = session.get(UploadItem, {"batch_id": batch_id, "asset_id": asset_id})
|
||||
if item is None:
|
||||
raise VerificationError("not_found", f"{asset_id!r} is not part of this batch")
|
||||
|
||||
item.outcome = outcome
|
||||
item.evidence = evidence[:MAX_EVIDENCE_CHARS]
|
||||
item.outcome_at = _now()
|
||||
item.verification = MANUAL
|
||||
item.verified_at = _now()
|
||||
session.add(
|
||||
_event(
|
||||
batch_id,
|
||||
asset_id,
|
||||
action="resolve",
|
||||
source="operator",
|
||||
result=MANUAL,
|
||||
outcome=outcome,
|
||||
evidence=evidence,
|
||||
actor=actor,
|
||||
)
|
||||
)
|
||||
|
||||
items = list(
|
||||
session.scalars(
|
||||
select(UploadItem)
|
||||
.where(UploadItem.batch_id == batch_id)
|
||||
.order_by(UploadItem.path)
|
||||
)
|
||||
)
|
||||
_resolve_batch(batch, items)
|
||||
session.commit()
|
||||
return _report(batch, items, server_reachable=None, detail=None)
|
||||
|
||||
# ── history ───────────────────────────────────────────────────────────────
|
||||
|
||||
def history(self, batch_id: str) -> list[dict]:
|
||||
"""Every verification and resolution recorded for this batch, oldest first."""
|
||||
with self._session_factory() as session:
|
||||
rows = session.scalars(
|
||||
select(UploadVerification)
|
||||
.where(UploadVerification.batch_id == batch_id)
|
||||
.order_by(UploadVerification.created_at, UploadVerification.id)
|
||||
)
|
||||
return [
|
||||
{
|
||||
"id": row.id,
|
||||
"asset_id": row.asset_id,
|
||||
"action": row.action,
|
||||
"source": row.source,
|
||||
"result": row.result,
|
||||
"outcome": row.outcome,
|
||||
"evidence": row.evidence,
|
||||
"actor": row.actor,
|
||||
"created_at": row.created_at.isoformat() if row.created_at else None,
|
||||
}
|
||||
for row in rows
|
||||
]
|
||||
|
||||
|
||||
# ── retry policy ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def retry_blockers(batch: dict) -> list[dict]:
|
||||
"""Why this batch may not be (re)run, in the order the user should fix them.
|
||||
|
||||
A plain uploader failure is a *safe* failure: nothing uncertain happened, so it
|
||||
is retryable. An uncertain outcome and edited bytes are not.
|
||||
"""
|
||||
from photo_pipeline.services.upload_batches import BatchState, RUNNABLE_STATES
|
||||
|
||||
blockers: list[dict] = []
|
||||
if batch["state"] == BatchState.UNKNOWN:
|
||||
blockers.append(
|
||||
{
|
||||
"code": "requires_verification",
|
||||
"message": "this upload's outcome is uncertain; verify it before retrying",
|
||||
}
|
||||
)
|
||||
if batch.get("stale_bytes"):
|
||||
blockers.append(
|
||||
{
|
||||
"code": "changed_after_upload",
|
||||
"message": "files in this batch changed after they were uploaded; "
|
||||
"re-approve them through a fresh preflight",
|
||||
}
|
||||
)
|
||||
if not blockers and batch["state"] not in RUNNABLE_STATES:
|
||||
blockers.append({"code": "not_runnable", "message": f"batch is {batch['state']}"})
|
||||
return blockers
|
||||
|
||||
|
||||
# ── internals ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _current_bytes(item: UploadItem) -> tuple[str | None, bool]:
|
||||
"""``(hash on disk now, changed since upload)``. A missing file counts as changed."""
|
||||
path = Path(item.path)
|
||||
if not path.exists():
|
||||
return None, item.sha256 is not None
|
||||
observed = sha256_file(path)
|
||||
return observed, bool(item.sha256 and observed != item.sha256)
|
||||
|
||||
|
||||
def _event(
|
||||
batch_id: str,
|
||||
asset_id: str,
|
||||
*,
|
||||
action: str,
|
||||
source: str,
|
||||
result: str,
|
||||
outcome: str | None,
|
||||
evidence: str,
|
||||
actor: str | None = None,
|
||||
) -> UploadVerification:
|
||||
return UploadVerification(
|
||||
id=str(uuid.uuid4()),
|
||||
batch_id=batch_id,
|
||||
asset_id=asset_id,
|
||||
action=action,
|
||||
source=source,
|
||||
result=result,
|
||||
outcome=outcome,
|
||||
evidence=evidence[:MAX_EVIDENCE_CHARS],
|
||||
actor=actor,
|
||||
created_at=_now(),
|
||||
)
|
||||
|
||||
|
||||
def _resolve_batch(batch: UploadBatch, items: list[UploadItem]) -> None:
|
||||
"""Fold the item evidence back into the batch's own state.
|
||||
|
||||
An uncertain batch only leaves that state once every item is accounted for:
|
||||
all present makes it succeeded, any absent makes it a safe failure to retry,
|
||||
and a single inconclusive item keeps it uncertain.
|
||||
"""
|
||||
from photo_pipeline.services.upload_batches import BatchState
|
||||
|
||||
batch.stale_bytes = any(item.changed_after_upload for item in items)
|
||||
batch.verified_at = _now()
|
||||
unresolved = [item for item in items if item.outcome in (None, report_parser.UNKNOWN)]
|
||||
batch.outcome_state = VERIFIED if items and not unresolved else REQUIRES_VERIFICATION
|
||||
|
||||
if batch.state != BatchState.UNKNOWN or unresolved:
|
||||
return
|
||||
if any(item.outcome == report_parser.FAILED for item in items):
|
||||
batch.state = BatchState.FAILED
|
||||
batch.error_code = "verified_incomplete"
|
||||
batch.error_message = "verification proved some files never reached Immich"
|
||||
else:
|
||||
batch.state = BatchState.SUCCEEDED
|
||||
batch.error_code = batch.error_message = None
|
||||
|
||||
|
||||
def _report(
|
||||
batch: UploadBatch,
|
||||
items: list[UploadItem],
|
||||
*,
|
||||
server_reachable: bool | None,
|
||||
detail: str | None,
|
||||
) -> dict:
|
||||
counts: dict[str, int] = {}
|
||||
for item in items:
|
||||
key = item.verification or "unverified"
|
||||
counts[key] = counts.get(key, 0) + 1
|
||||
return {
|
||||
"batch_id": batch.id,
|
||||
"state": batch.state,
|
||||
"outcome_state": batch.outcome_state,
|
||||
"stale_bytes": batch.stale_bytes,
|
||||
"server_reachable": server_reachable,
|
||||
"detail": detail,
|
||||
"counts": counts,
|
||||
"items": [
|
||||
{
|
||||
"asset_id": item.asset_id,
|
||||
"path": item.path,
|
||||
"verification": item.verification,
|
||||
"outcome": item.outcome,
|
||||
"evidence": item.evidence,
|
||||
"changed_after_upload": item.changed_after_upload,
|
||||
"verified_at": item.verified_at.isoformat() if item.verified_at else None,
|
||||
}
|
||||
for item in items
|
||||
],
|
||||
}
|
||||
@@ -24,6 +24,8 @@ test = [
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
# The frozen CLI archive is evidence, not code under maintenance (US07-01).
|
||||
extend-exclude = ["legacy_cli_archive"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
@@ -31,4 +33,6 @@ markers = [
|
||||
"phase_b: Phase B end-to-end acceptance (US02-07) — API, worker-recovery, and browser journeys",
|
||||
"phase_c: Phase C end-to-end acceptance (US03-05) — album proposal API and browser journeys",
|
||||
"phase_d: Phase D end-to-end acceptance (US04-06) — guarded rename API, fault, and browser journeys",
|
||||
"phase_e: Phase E end-to-end acceptance (US05-06) — upload preflight, uploader, and browser journeys",
|
||||
"phase_f: Phase F end-to-end acceptance (US06-06) — archive destination, transfer, and restore journeys",
|
||||
]
|
||||
|
||||
@@ -14,6 +14,11 @@ import pytest
|
||||
from PIL import Image
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
# The donors are frozen in the read-only archive (US07-01). Only this suite — and
|
||||
# the parity test that compares against them — puts that directory on sys.path;
|
||||
# production never does, which tests/unit/test_legacy_archive.py enforces.
|
||||
ARCHIVED_SOURCES = REPO / "legacy_cli_archive" / "src"
|
||||
sys.path.insert(0, str(ARCHIVED_SOURCES))
|
||||
sys.path.insert(0, str(REPO))
|
||||
|
||||
EXIFTOOL = shutil.which("exiftool")
|
||||
|
||||
@@ -1,13 +1,20 @@
|
||||
"""Ledger lint (US01-01): every donor-ledger row must carry a real source
|
||||
reference, a target location, and either existing characterization test IDs or
|
||||
a real pending backlog story."""
|
||||
"""Ledger lint (US01-01, extended by US07-01).
|
||||
|
||||
Every donor-ledger row must carry a real source reference — now inside the frozen
|
||||
archive — a target location, and either existing characterization test IDs or a
|
||||
real pending backlog story. Since archival (US07-01) a row may also be ``resolved``:
|
||||
its replacement has shipped, ``parity`` names tests that exist and prove it, and
|
||||
``delta`` states every intentional difference. Nothing may quietly become
|
||||
"finished" without one of those two."""
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
LEDGER = REPO / "donor_ledger.yaml"
|
||||
ARCHIVE = REPO / "legacy_cli_archive"
|
||||
LEDGER = ARCHIVE / "donor_ledger.yaml"
|
||||
ARCHIVED_SOURCES = ARCHIVE / "src"
|
||||
STORIES = REPO / "delivery_backlog" / "stories"
|
||||
TESTS_DIR = Path(__file__).resolve().parent
|
||||
|
||||
@@ -15,7 +22,7 @@ CLASSIFICATIONS = {"reuse", "extract", "refactor", "replace"}
|
||||
REQUIRED_AREAS = {"discovery", "hashing", "imaging", "nsfw", "vision", "exif",
|
||||
"database", "ui", "configuration", "logging", "cancellation",
|
||||
"error"}
|
||||
STATUSES = {"characterized", "pending"}
|
||||
STATUSES = {"characterized", "resolved", "pending"}
|
||||
|
||||
|
||||
def load_rows():
|
||||
@@ -49,10 +56,11 @@ def test_rows_have_required_fields_and_unique_ids():
|
||||
|
||||
|
||||
def test_source_references_resolve():
|
||||
"""Source paths are relative to the archive: the donors moved there, whole."""
|
||||
for r in load_rows():
|
||||
src = r["source"]
|
||||
f = REPO / src["file"]
|
||||
assert f.is_file(), f"{r['id']}: source file {src['file']} missing"
|
||||
f = ARCHIVED_SOURCES / src["file"]
|
||||
assert f.is_file(), f"{r['id']}: source file {src['file']} missing from the archive"
|
||||
text = f.read_text(encoding="utf-8")
|
||||
for sym in src["symbols"]:
|
||||
assert sym in text, f"{r['id']}: symbol {sym!r} not found in {src['file']}"
|
||||
@@ -63,7 +71,9 @@ def test_rows_have_tests_or_pending_story():
|
||||
for r in load_rows():
|
||||
tests = r.get("tests", [])
|
||||
pending = r.get("pending_story")
|
||||
assert tests or pending, f"{r['id']}: neither tests nor pending_story"
|
||||
parity = r.get("parity", [])
|
||||
assert tests or pending or parity, \
|
||||
f"{r['id']}: neither characterization tests, parity tests, nor a pending story"
|
||||
for t in tests:
|
||||
assert t in known_tests, f"{r['id']}: unknown test id {t}"
|
||||
if pending:
|
||||
@@ -73,16 +83,58 @@ def test_rows_have_tests_or_pending_story():
|
||||
assert tests, f"{r['id']}: characterized rows need test ids"
|
||||
|
||||
|
||||
def test_resolved_rows_name_their_parity_or_their_delta():
|
||||
"""A resolved row is a claim that the behavior is handled. It has to say how:
|
||||
tests that prove the replacement, or a stated difference — usually both."""
|
||||
for r in load_rows():
|
||||
if r["status"] != "resolved":
|
||||
assert "parity" not in r, f"{r['id']}: parity on a non-resolved row"
|
||||
continue
|
||||
parity = r.get("parity", [])
|
||||
delta = r.get("delta")
|
||||
assert parity or delta, f"{r['id']}: resolved without parity tests or a delta"
|
||||
if delta:
|
||||
assert len(str(delta).strip()) >= 20, f"{r['id']}: delta too thin to be a reason"
|
||||
for ref in parity:
|
||||
rel, _, func = ref.partition("::")
|
||||
path = REPO / rel
|
||||
assert path.is_file(), f"{r['id']}: parity test file {rel} missing"
|
||||
assert f"def {func}" in path.read_text(encoding="utf-8"), \
|
||||
f"{r['id']}: parity test {ref} not found"
|
||||
|
||||
|
||||
def test_pending_rows_are_the_only_unfinished_work():
|
||||
"""The ledger is the honest list of what has not been carried over: a pending
|
||||
row names the story that will, and that story must still be open work."""
|
||||
for r in load_rows():
|
||||
if r["status"] != "pending":
|
||||
continue
|
||||
story = r.get("pending_story")
|
||||
assert story, f"{r['id']}: pending without a story"
|
||||
assert list(STORIES.glob(f"{story}-*.md")), f"{r['id']}: unknown story {story}"
|
||||
|
||||
|
||||
def test_every_target_module_exists():
|
||||
"""A row is only finished if the thing it points at is really there."""
|
||||
for r in load_rows():
|
||||
if r["status"] == "pending":
|
||||
continue
|
||||
modules = re.findall(r"photo_pipeline/[\w/]+\.py", str(r["target"]))
|
||||
for module in modules:
|
||||
assert (REPO / module).is_file(), f"{r['id']}: target {module} does not exist"
|
||||
|
||||
|
||||
def test_all_required_areas_covered():
|
||||
covered = {r["area"] for r in load_rows()}
|
||||
assert REQUIRED_AREAS <= covered, f"uncovered areas: {REQUIRED_AREAS - covered}"
|
||||
assert covered <= REQUIRED_AREAS, f"unknown areas: {covered - REQUIRED_AREAS}"
|
||||
|
||||
|
||||
def test_no_legacy_file_moved():
|
||||
# US01-01 explicitly forbids moving/archiving donors; the ledger's source
|
||||
# files must all still exist at their original locations.
|
||||
def test_every_donor_is_in_the_archive_and_nowhere_else():
|
||||
"""US01-01 forbade archiving before characterization; US07-01 requires it after.
|
||||
Each donor exists exactly once — frozen, in the archive."""
|
||||
for donor in ("photo_analyzer.py", "nsfwtag/scoring.py", "nsfwtag/exif.py",
|
||||
"nsfwtag/server.py", "webapp/query.py", "webapp/runner.py",
|
||||
"webapp/server.py"):
|
||||
assert (REPO / donor).is_file(), f"donor moved: {donor}"
|
||||
assert (ARCHIVED_SOURCES / donor).is_file(), f"donor missing from archive: {donor}"
|
||||
assert not (REPO / donor).exists(), f"donor still live at the repo root: {donor}"
|
||||
|
||||
@@ -9,14 +9,18 @@ and worker, never mocked inside a test.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
@@ -135,15 +139,27 @@ class Server:
|
||||
self.proc = None
|
||||
|
||||
|
||||
def start_worker(seeded: Seeded, *, fake_vision_log: Path) -> subprocess.Popen:
|
||||
"""Launch a real durable worker wired to the recording vision fake."""
|
||||
def start_worker(
|
||||
seeded: Seeded,
|
||||
*,
|
||||
fake_vision_log: Path | None = None,
|
||||
extra_env: dict[str, str] | None = None,
|
||||
) -> subprocess.Popen:
|
||||
"""Launch a real durable worker wired to the recording vision fake.
|
||||
|
||||
``extra_env`` carries whatever else the job under test needs — the Immich
|
||||
credentials and uploader path, for the upload lane.
|
||||
"""
|
||||
extra = dict(extra_env or {})
|
||||
if fake_vision_log is not None:
|
||||
extra["PHOTO_PIPELINE_FAKE_VISION_LOG"] = str(fake_vision_log)
|
||||
return subprocess.Popen(
|
||||
[sys.executable, "-m", "photo_pipeline", "worker", "--id", "e2e-worker"],
|
||||
cwd=str(REPO),
|
||||
env=_env(
|
||||
seeded,
|
||||
free_port(), # unused by the worker, but keeps the env shape uniform
|
||||
extra={"PHOTO_PIPELINE_FAKE_VISION_LOG": str(fake_vision_log)},
|
||||
extra=extra,
|
||||
),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
@@ -246,3 +262,301 @@ def approve_album(base: str, *, album: str = "rome", name: str) -> None:
|
||||
json={**payload, "expected_version": current["version"]},
|
||||
timeout=10,
|
||||
).raise_for_status()
|
||||
|
||||
|
||||
# ── Phase E: an upload-ready album, a fake Immich, and a real fake uploader ───
|
||||
|
||||
SENTINEL_KEY = "immich-sentinel-9f3a2b"
|
||||
UPLOADER_VERSION = "immich-go 0.21.0" # a pinned family, so reports are parsable
|
||||
|
||||
# Uploader bodies for the pinned ``text-v1`` grammar. ``$6`` is the folder argument
|
||||
# of ``upload from-folder``.
|
||||
REPORTING_UPLOADER = (
|
||||
'echo "INFO uploaded $6/a.jpg"\n'
|
||||
'echo "INFO server has the same file $6/b.jpg"\n'
|
||||
'echo "Uploaded 1, duplicates 1"\n'
|
||||
"exit 0\n"
|
||||
)
|
||||
# Exits cleanly but says nothing about any file: the process succeeded, the
|
||||
# per-file outcome is unknown.
|
||||
SILENT_UPLOADER = "exit 0\n"
|
||||
|
||||
|
||||
def _immich_handler(state: dict):
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self): # noqa: N802 (BaseHTTPRequestHandler API)
|
||||
self._json(200, {"res": "pong"})
|
||||
|
||||
def do_POST(self): # noqa: N802
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
payload = json.loads(self.rfile.read(length) or b"{}")
|
||||
if state["mode"] == "broken":
|
||||
self.send_error(500, "bulk-upload-check is unavailable")
|
||||
return
|
||||
reject = state["mode"] == "present"
|
||||
self._json(
|
||||
200,
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"id": asset["id"],
|
||||
"action": "reject" if reject else "accept",
|
||||
"reason": "duplicate" if reject else None,
|
||||
}
|
||||
for asset in payload.get("assets", [])
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
def _json(self, code: int, body: dict) -> None:
|
||||
raw = json.dumps(body).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(raw)))
|
||||
self.end_headers()
|
||||
self.wfile.write(raw)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
return Handler
|
||||
|
||||
|
||||
class FakeImmich:
|
||||
"""An Immich that answers ping, and says whether it holds the exact bytes.
|
||||
|
||||
``mode`` is what the next verification will find: ``present`` (the server
|
||||
deduplicates them, so it has them), ``absent`` (it would accept them, so it does
|
||||
not), or ``broken`` (no usable answer at all).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.state = {"mode": "present"}
|
||||
self._server = HTTPServer(("127.0.0.1", 0), _immich_handler(self.state))
|
||||
threading.Thread(target=self._server.serve_forever, daemon=True).start()
|
||||
self.url = f"http://127.0.0.1:{self._server.server_port}"
|
||||
self._running = True
|
||||
|
||||
def mode(self, mode: str) -> None:
|
||||
self.state["mode"] = mode
|
||||
|
||||
def stop(self) -> None:
|
||||
"""Idempotent, so a test may take Immich away mid-journey."""
|
||||
if not self._running:
|
||||
return
|
||||
self._running = False
|
||||
self._server.shutdown()
|
||||
self._server.server_close()
|
||||
|
||||
|
||||
def fake_uploader(tmp_path: Path, body: str) -> Path:
|
||||
"""A real executable standing in for immich-go.
|
||||
|
||||
``--version`` answers like the real tool; any other invocation appends its
|
||||
complete argv to ``immich-go.argv`` — which is how a test proves the uploader
|
||||
ran, what folder it was handed, or that it never ran at all.
|
||||
"""
|
||||
path = tmp_path / "immich-go"
|
||||
path.write_text(
|
||||
"#!/bin/sh\n"
|
||||
f'if [ "$1" = "--version" ]; then echo "{UPLOADER_VERSION}"; exit 0; fi\n'
|
||||
f'printf "%s\\n" "$*" >> "{tmp_path / "immich-go.argv"}"\n'
|
||||
f"{body}"
|
||||
)
|
||||
path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
|
||||
return path
|
||||
|
||||
|
||||
def uploader_argv(tmp_path: Path) -> list[str]:
|
||||
"""Every upload invocation the fake uploader saw, oldest first."""
|
||||
log = tmp_path / "immich-go.argv"
|
||||
return log.read_text().splitlines() if log.exists() else []
|
||||
|
||||
|
||||
def mark_upload_ready(seeded: Seeded, *, unverified: tuple[str, ...] = ()) -> None:
|
||||
"""Give every seeded photo the verified EXIF checkpoints upload requires.
|
||||
|
||||
``unverified`` names stems whose analysis checkpoint stays incomplete, which is
|
||||
what makes an album partially blocked.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.models import AnalysisResult, SafetyReview
|
||||
|
||||
blocked = {seeded.asset_ids[stem] for stem in unverified}
|
||||
with session_factory(seeded) as sf:
|
||||
with sf() as session:
|
||||
for review in session.scalars(select(SafetyReview)):
|
||||
review.exif_verified_at = NOW
|
||||
for analysis in session.scalars(select(AnalysisResult)):
|
||||
analysis.exif_written_at = None if analysis.asset_id in blocked else NOW
|
||||
session.commit()
|
||||
|
||||
|
||||
# ── Phase F: an archivable album and a mountable fake medium ─────────────────
|
||||
|
||||
|
||||
def mark_uploaded(seeded: Seeded, *, album: str = "rome") -> None:
|
||||
"""Give every seeded photo the verified upload evidence archiving requires.
|
||||
|
||||
Archiving refuses anything Immich is not proven to hold, and that proof is an
|
||||
upload batch — recorded here as fixture state so the archive journeys do not
|
||||
have to re-run an upload they are not testing.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.models import Asset, UploadBatch, UploadItem
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
|
||||
with session_factory(seeded) as sf:
|
||||
with sf() as session:
|
||||
batch_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
UploadBatch(
|
||||
id=batch_id,
|
||||
album=album,
|
||||
folder=str(seeded.lib / album),
|
||||
album_name=album,
|
||||
state="succeeded",
|
||||
preflight_token="v1:e2e",
|
||||
outcome_state="verified",
|
||||
created_at=NOW,
|
||||
)
|
||||
)
|
||||
for asset in session.scalars(select(Asset)):
|
||||
if not asset.current_path:
|
||||
continue
|
||||
session.add(
|
||||
UploadItem(
|
||||
batch_id=batch_id,
|
||||
asset_id=asset.id,
|
||||
path=asset.current_path,
|
||||
sha256=sha256_file(asset.current_path),
|
||||
sha1="0" * 40,
|
||||
state="sent",
|
||||
outcome="uploaded",
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
|
||||
class ArchiveStack:
|
||||
"""A seeded, archivable library plus the server, the worker, and a fake medium.
|
||||
|
||||
The medium is an ordinary directory whose marker file makes it identifiable;
|
||||
``unmount()`` takes that marker away, which is exactly what the application sees
|
||||
when an external disk is unplugged.
|
||||
"""
|
||||
|
||||
MARKER = ".photo-pipeline-archive.json"
|
||||
|
||||
def __init__(self, tmp_path: Path, seeded: Seeded) -> None:
|
||||
self.tmp_path = tmp_path
|
||||
self.seeded = seeded
|
||||
self.archive = tmp_path / "archive"
|
||||
self.archive.mkdir(exist_ok=True)
|
||||
self.server: Server | None = None
|
||||
self.worker: subprocess.Popen | None = None
|
||||
self.base = ""
|
||||
|
||||
def start(self, *, worker: bool = True, extra_env: dict[str, str] | None = None) -> "ArchiveStack":
|
||||
env = {"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0", **(extra_env or {})}
|
||||
self.server = Server(self.seeded, extra_env=env).start()
|
||||
self.base = self.server.base
|
||||
if worker:
|
||||
self.worker = start_worker(self.seeded, extra_env=env)
|
||||
return self
|
||||
|
||||
def register(self, name: str = "external") -> dict:
|
||||
response = httpx.post(
|
||||
f"{self.base}/api/v1/archive-locations",
|
||||
json={"name": name, "root": str(self.archive)},
|
||||
timeout=20,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
def unmount(self) -> None:
|
||||
(self.archive / self.MARKER).rename(self.archive / f"{self.MARKER}.away")
|
||||
|
||||
def remount(self) -> None:
|
||||
(self.archive / f"{self.MARKER}.away").rename(self.archive / self.MARKER)
|
||||
|
||||
def plans(self) -> list[dict]:
|
||||
return httpx.get(f"{self.base}/api/v1/archive-plans", timeout=20).json()["plans"]
|
||||
|
||||
def assets(self) -> list[dict]:
|
||||
return httpx.get(
|
||||
f"{self.base}/api/v1/inventory/assets", params={"limit": 200}, timeout=20
|
||||
).json()["items"]
|
||||
|
||||
def restart_server(self) -> None:
|
||||
self.server.stop()
|
||||
self.server.start()
|
||||
|
||||
def restart_worker(self, *, extra_env: dict[str, str] | None = None) -> None:
|
||||
"""Replace the worker — a healthy one after a crashed one, by default."""
|
||||
if self.worker is not None and self.worker.poll() is None:
|
||||
self.worker.kill()
|
||||
self.worker.wait(timeout=10)
|
||||
self.worker = start_worker(
|
||||
self.seeded,
|
||||
extra_env={"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0", **(extra_env or {})},
|
||||
)
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.worker is not None:
|
||||
self.worker.kill()
|
||||
self.worker.wait(timeout=10)
|
||||
if self.server is not None:
|
||||
self.server.stop()
|
||||
|
||||
|
||||
class UploadStack:
|
||||
"""A seeded, upload-ready library plus the server, worker, and fake Immich."""
|
||||
|
||||
def __init__(self, tmp_path: Path, seeded: Seeded) -> None:
|
||||
self.tmp_path = tmp_path
|
||||
self.seeded = seeded
|
||||
self.immich = FakeImmich()
|
||||
self.server: Server | None = None
|
||||
self.worker: subprocess.Popen | None = None
|
||||
self.base = ""
|
||||
|
||||
def start(
|
||||
self,
|
||||
*,
|
||||
uploader: str = REPORTING_UPLOADER,
|
||||
worker: bool = True,
|
||||
credentials: bool = True,
|
||||
) -> "UploadStack":
|
||||
env = {
|
||||
"PHOTO_PIPELINE_IMMICH_SERVER_URL": self.immich.url if credentials else "",
|
||||
"PHOTO_PIPELINE_IMMICH_GO_BINARY": str(fake_uploader(self.tmp_path, uploader)),
|
||||
}
|
||||
if credentials:
|
||||
env["PHOTO_PIPELINE_IMMICH_API_KEY"] = SENTINEL_KEY
|
||||
self.server = Server(self.seeded, extra_env=env).start()
|
||||
self.base = self.server.base
|
||||
if worker:
|
||||
self.worker = start_worker(self.seeded, extra_env=env)
|
||||
return self
|
||||
|
||||
def restart_server(self) -> None:
|
||||
"""A genuinely fresh process against the same database and library."""
|
||||
self.server.stop()
|
||||
self.server.start()
|
||||
|
||||
def batches(self) -> list[dict]:
|
||||
return httpx.get(f"{self.base}/api/v1/upload-batches", timeout=20).json()["batches"]
|
||||
|
||||
def argv(self) -> list[str]:
|
||||
return uploader_argv(self.tmp_path)
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.worker is not None:
|
||||
self.worker.kill()
|
||||
self.worker.wait(timeout=10)
|
||||
if self.server is not None:
|
||||
self.server.stop()
|
||||
self.immich.stop()
|
||||
|
||||
302
tests/e2e/test_archive_ui.py
Normal file
302
tests/e2e/test_archive_ui.py
Normal file
@@ -0,0 +1,302 @@
|
||||
"""Browser journeys for the archive view (US06-05).
|
||||
|
||||
Archiving is the only stage that removes originals, so these journeys check the
|
||||
two things a browser must never get wrong about it: that the preview names exactly
|
||||
what would leave and where it would go, and that nothing offers an action the
|
||||
server would refuse. The medium is a real directory whose marker makes it
|
||||
identifiable; unmounting it is what an unplugged disk looks like from here.
|
||||
|
||||
Nothing is mocked inside the browser: the transfer runs in the real worker process
|
||||
and the assertions read the filesystem afterwards.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from playwright.sync_api import expect
|
||||
|
||||
from tests.e2e._pipeline_harness import (
|
||||
ArchiveStack,
|
||||
mark_uploaded,
|
||||
seed_album,
|
||||
session_factory,
|
||||
wait_until,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
||||
|
||||
TIMEOUT = 10
|
||||
RUN_TIMEOUT = 30_000
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stack(tmp_path):
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_uploaded(seeded)
|
||||
running = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
def _open(page, stack) -> None:
|
||||
page.goto(f"{stack.base}/app/#/archive")
|
||||
page.get_by_test_id("archive-locations").wait_for()
|
||||
|
||||
|
||||
def _archive(page, stack) -> None:
|
||||
"""Confirm the archive and wait for the worker to finish the run."""
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-archive").click()
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete", timeout=RUN_TIMEOUT)
|
||||
|
||||
|
||||
def _archived_paths(stack) -> list[str]:
|
||||
return sorted(p.name for p in (stack.archive / "rome").glob("*.jpg"))
|
||||
|
||||
|
||||
# ── preview and confirmation ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_preview_names_the_scope_destination_and_reclaimable_bytes(page, stack):
|
||||
errors = []
|
||||
page.on("console", lambda m: errors.append(m.text) if m.type == "error" else None)
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
_open(page, stack)
|
||||
|
||||
row = page.get_by_test_id("location-row").first
|
||||
expect(row.get_by_test_id("location-label")).to_have_text("external")
|
||||
expect(row.get_by_test_id("location-media")).to_have_text(location["media_id"])
|
||||
expect(row.get_by_test_id("location-state")).to_have_text("online")
|
||||
|
||||
album = page.get_by_test_id("archive-album-row").first
|
||||
expect(album.get_by_test_id("album-name")).to_have_text("rome")
|
||||
expect(album.get_by_test_id("album-destination")).to_have_text(str(stack.archive / "rome"))
|
||||
expect(album.get_by_test_id("album-assets")).to_have_text("2")
|
||||
expect(album.get_by_test_id("album-state")).to_have_text("ready")
|
||||
# Same filesystem here, so the transfer method is the move path — and it is
|
||||
# named, because copy-verify-remove and move fail differently.
|
||||
expect(album.get_by_test_id("album-method")).to_contain_text("move")
|
||||
|
||||
expect(page.get_by_test_id("destination-identity")).to_contain_text(location["media_id"])
|
||||
expect(page.get_by_test_id("capacity")).to_contain_text("reserve")
|
||||
expect(page.get_by_test_id("start-archive")).to_contain_text("Archive 1 album(s) · reclaim")
|
||||
expect(page.get_by_test_id("start-archive")).to_be_enabled()
|
||||
assert stack.plans() == [], "previewing may not create anything"
|
||||
assert errors == [], f"console errors: {errors}"
|
||||
|
||||
|
||||
def test_an_offline_medium_blocks_the_confirmation_and_says_what_to_mount(page, stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
stack.unmount()
|
||||
_open(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("location-state")).to_have_text("offline")
|
||||
expect(page.get_by_test_id("preflight-blocker").first).to_have_attribute(
|
||||
"data-code", "location_offline"
|
||||
)
|
||||
instruction = page.get_by_test_id("mount-instruction")
|
||||
expect(instruction).to_contain_text("external")
|
||||
expect(instruction).to_contain_text(location["media_id"])
|
||||
expect(page.get_by_test_id("start-archive")).to_be_disabled()
|
||||
|
||||
|
||||
def test_an_unuploaded_album_is_blocked_with_its_reason(page, stack):
|
||||
# No upload evidence at all: archiving would remove the only copy.
|
||||
seeded = stack.seeded
|
||||
with session_factory(seeded) as sf:
|
||||
from sqlalchemy import delete
|
||||
|
||||
from photo_pipeline.models import UploadItem
|
||||
|
||||
with sf() as session:
|
||||
session.execute(delete(UploadItem))
|
||||
session.commit()
|
||||
stack.start(worker=False)
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("album-state")).to_have_text("blocked")
|
||||
expect(page.get_by_test_id("album-blocker").first).to_have_attribute(
|
||||
"data-code", "partial_scope"
|
||||
)
|
||||
expect(page.get_by_test_id("start-archive")).to_be_disabled()
|
||||
|
||||
|
||||
# ── running, progress, reload ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_confirmed_archive_runs_and_separates_transfer_verify_and_removal(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("count-complete")).to_have_text("complete: 2")
|
||||
expect(page.get_by_test_id("count-failed")).to_have_text("failed: 0")
|
||||
expect(page.get_by_test_id("count-transfer")).to_have_text("transfer: 0")
|
||||
expect(page.get_by_test_id("count-verified")).to_have_text("verified: 0")
|
||||
expect(page.get_by_test_id("count-removing")).to_have_text("removing: 0")
|
||||
expect(page.get_by_test_id("operation-phase").first).to_have_text("complete")
|
||||
|
||||
# The library really lost the originals and the medium really holds them.
|
||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"]
|
||||
assert not (stack.seeded.lib / "rome" / "a.jpg").exists()
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
||||
|
||||
|
||||
def test_the_run_survives_a_reload_because_the_state_is_the_servers(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("plan-detail").wait_for()
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete")
|
||||
expect(page.get_by_test_id("count-complete")).to_have_text("complete: 2")
|
||||
# And the run stays addressable by its own URL.
|
||||
plan_id = stack.plans()[0]["id"]
|
||||
page.goto(f"{stack.base}/app/#/archive?plan={plan_id}")
|
||||
expect(page.get_by_test_id("plan-detail")).to_have_attribute("data-plan", plan_id)
|
||||
|
||||
|
||||
# ── recovery ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_an_interrupted_transfer_is_shown_with_evidence_and_a_safe_action(page, stack):
|
||||
stack.start(worker=False)
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-archive").click()
|
||||
expect(page.get_by_test_id("archive-result")).to_be_visible()
|
||||
# No worker ran, so every item is still planned; leave one mid-transfer as a
|
||||
# crash would: intent recorded, nothing published, source intact.
|
||||
_interrupt(stack)
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("archive-recovery").wait_for()
|
||||
row = page.get_by_test_id("recovery-row").first
|
||||
expect(row).to_have_attribute("data-classification", "resumable")
|
||||
expect(row.get_by_test_id("recovery-reason")).to_contain_text("source present")
|
||||
page.get_by_test_id("resolve-recovery").click()
|
||||
expect(page.get_by_test_id("archive-recovery")).to_have_count(0)
|
||||
assert (stack.seeded.lib / "rome" / "a.jpg").exists(), "recovery may not move anything"
|
||||
|
||||
|
||||
def test_ambiguous_evidence_offers_no_action_at_all(page, stack):
|
||||
stack.start(worker=False)
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-archive").click()
|
||||
expect(page.get_by_test_id("archive-result")).to_be_visible()
|
||||
# Journal says the copy is verified, but the medium holds nothing: no evidence
|
||||
# supports either finishing or retrying this item.
|
||||
_interrupt(stack, state="verified")
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("archive-recovery").wait_for()
|
||||
expect(page.get_by_test_id("recovery-row").first).to_have_attribute(
|
||||
"data-classification", "manual"
|
||||
)
|
||||
expect(page.get_by_test_id("recovery-manual")).to_be_visible()
|
||||
expect(page.get_by_test_id("resolve-recovery")).to_have_count(0)
|
||||
expect(page.get_by_test_id("no-safe-recovery")).to_be_visible()
|
||||
|
||||
|
||||
def _interrupt(stack, *, state: str = "transferring") -> None:
|
||||
"""Leave the plan's first operation in a non-terminal journal state."""
|
||||
with session_factory(stack.seeded) as sf:
|
||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
||||
|
||||
journal = ArchiveJournal(sf)
|
||||
plan_id = stack.plans()[0]["id"]
|
||||
operation = journal.operations(plan_id)[0]
|
||||
journal.begin(operation["id"], worker_id="crashed", fencing_token=1)
|
||||
if state != "transferring":
|
||||
journal.transition(operation["id"], state, fencing_token=1)
|
||||
|
||||
|
||||
# ── offline browsing and restore ─────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_archived_photos_stay_browsable_while_the_medium_is_away(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
stack.unmount()
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("archived-assets").wait_for()
|
||||
row = page.get_by_test_id("archived-row").first
|
||||
expect(row.get_by_test_id("archived-availability")).to_have_text("archived · medium away")
|
||||
expect(row.get_by_test_id("archived-path")).to_contain_text("rome/")
|
||||
expect(row.get_by_test_id("archived-medium")).to_have_text("external")
|
||||
expect(page.get_by_test_id("mount-instruction")).to_contain_text("external")
|
||||
# The retained protected preview is served even though the original is gone.
|
||||
preview = row.get_by_test_id("archived-preview")
|
||||
assert preview.evaluate("img => img.complete && img.naturalWidth > 0")
|
||||
# Restoring is impossible right now, and says so instead of failing later.
|
||||
expect(page.get_by_test_id("start-restore")).to_be_disabled()
|
||||
|
||||
|
||||
def test_restoring_brings_the_photos_back_without_losing_identity(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
before = {asset["id"] for asset in stack.assets()}
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("restore").wait_for()
|
||||
expect(page.get_by_test_id("restore-row").first.get_by_test_id("restore-destination")).to_contain_text(
|
||||
str(stack.seeded.lib / "rome")
|
||||
)
|
||||
page.get_by_test_id("start-restore").click()
|
||||
wait_until(
|
||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()),
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
assert {asset["id"] for asset in stack.assets()} == before # same identities
|
||||
assert (stack.seeded.lib / "rome" / "a.jpg").exists()
|
||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"] # the archive copy stays
|
||||
|
||||
|
||||
def test_a_taken_name_is_restored_beside_it_never_over_it(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_archive(page, stack)
|
||||
squatter = stack.seeded.lib / "rome" / "a.jpg"
|
||||
squatter.parent.mkdir(parents=True, exist_ok=True)
|
||||
squatter.write_bytes(b"a different photo now lives here")
|
||||
|
||||
page.reload()
|
||||
page.get_by_test_id("restore").wait_for()
|
||||
destinations = page.get_by_test_id("restore-destination").all_inner_texts()
|
||||
assert any("(restored)" in text for text in destinations), destinations
|
||||
|
||||
page.get_by_test_id("start-restore").click()
|
||||
wait_until(
|
||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()),
|
||||
timeout=30,
|
||||
)
|
||||
assert squatter.read_bytes() == b"a different photo now lives here"
|
||||
assert (stack.seeded.lib / "rome" / "a (restored).jpg").exists()
|
||||
|
||||
|
||||
# ── keyboard ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_whole_archive_can_be_confirmed_from_the_keyboard(page, stack):
|
||||
stack.start()
|
||||
stack.register()
|
||||
_open(page, stack)
|
||||
|
||||
button = page.get_by_test_id("start-archive")
|
||||
button.focus()
|
||||
expect(button).to_be_focused()
|
||||
page.keyboard.press("Enter")
|
||||
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("complete", timeout=RUN_TIMEOUT)
|
||||
assert _archived_paths(stack) == ["a.jpg", "b.jpg"]
|
||||
568
tests/e2e/test_phase_e_pipeline.py
Normal file
568
tests/e2e/test_phase_e_pipeline.py
Normal file
@@ -0,0 +1,568 @@
|
||||
"""Phase E end-to-end acceptance (US05-06): Immich upload, black box.
|
||||
|
||||
Every journey drives a real ``photo_pipeline serve`` child process and a real durable
|
||||
worker over HTTP — preflight, approve, upload, duplicate, upgrade, fail, retry, lose
|
||||
the acceptance response, verify, cancel, crash, restart. Nothing external is mocked
|
||||
inside the application: ``immich-go`` is a real executable that records the argv it
|
||||
was handed, and Immich is a real HTTP server answering the same ``ping`` and
|
||||
``bulk-upload-check`` endpoints the adapter calls in production.
|
||||
|
||||
Two invariants are asserted in every relevant journey, because they are what make an
|
||||
irreversible stage safe:
|
||||
|
||||
- **EXIF precedes upload.** An album without its verified safety and analysis
|
||||
checkpoints cannot be approved, and the uploader's argv log proves it was never
|
||||
even executed.
|
||||
- **The persisted hashes are the submitted bytes.** After each upload the recorded
|
||||
SHA-256/SHA-1 of every item is recomputed from the files in the folder the uploader
|
||||
was actually given.
|
||||
|
||||
The API key is a sentinel string, so the last journey can prove it reached the
|
||||
uploader and nothing else that was retained.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from tests.e2e._pipeline_harness import (
|
||||
SENTINEL_KEY,
|
||||
SILENT_UPLOADER,
|
||||
UploadStack,
|
||||
mark_upload_ready,
|
||||
seed_album,
|
||||
session_factory,
|
||||
wait_until,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.phase_e
|
||||
|
||||
TIMEOUT = 20
|
||||
ALBUM = "rome"
|
||||
TERMINAL = {"succeeded", "failed", "cancelled", "unknown_requires_verification"}
|
||||
|
||||
# Uploader bodies in the pinned ``text-v1`` grammar. ``$6`` is the folder argument of
|
||||
# ``upload from-folder``, so each line names the real path of a real file.
|
||||
ALL_NEW = (
|
||||
'echo "INFO uploaded $6/a.jpg"\n'
|
||||
'echo "INFO uploaded $6/b.jpg"\n'
|
||||
'echo "Uploaded 2"\n'
|
||||
"exit 0\n"
|
||||
)
|
||||
EXACT_DUPLICATES = (
|
||||
'echo "INFO server has the same file $6/a.jpg"\n'
|
||||
'echo "INFO server has the same file $6/b.jpg"\n'
|
||||
'echo "Duplicates 2"\n'
|
||||
"exit 0\n"
|
||||
)
|
||||
UPGRADES = (
|
||||
'echo "INFO server has an older file $6/a.jpg"\n'
|
||||
'echo "INFO server has an older file $6/b.jpg"\n'
|
||||
'echo "Upgraded 2"\n'
|
||||
"exit 0\n"
|
||||
)
|
||||
|
||||
|
||||
def _once_then(tmp_path: Path, first: str, rest: str) -> str:
|
||||
"""An uploader that behaves one way on its first attempt and another afterwards.
|
||||
|
||||
The flag file is the attempt counter, so retry and resume journeys are
|
||||
deterministic without any test reaching into the running application.
|
||||
"""
|
||||
flag = tmp_path / "first-attempt.flag"
|
||||
return f'if [ ! -f "{flag}" ]; then\n touch "{flag}"\n{first}fi\n{rest}'
|
||||
|
||||
|
||||
FAILING_FIRST = (' echo "ERROR error uploading $6/a.jpg: connection reset"\n exit 1\n', ALL_NEW)
|
||||
SLOW_FIRST = (' echo "INFO starting"\n sleep 30\n exit 0\n', ALL_NEW)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stack(tmp_path):
|
||||
"""An analysed album whose EXIF checkpoints are already verified."""
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_upload_ready(seeded)
|
||||
running = UploadStack(tmp_path, seeded)
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def unfinished(tmp_path):
|
||||
"""The same album *before* its EXIF checkpoints were written."""
|
||||
seeded = seed_album(tmp_path)
|
||||
running = UploadStack(tmp_path, seeded)
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
# ── helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _preflight(base: str, **body) -> dict:
|
||||
response = httpx.post(f"{base}/api/v1/upload-preflight", json=body, timeout=TIMEOUT)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _create(base: str, report: dict, **body) -> httpx.Response:
|
||||
return httpx.post(
|
||||
f"{base}/api/v1/upload-batches",
|
||||
json={"token": report["token"], **body},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
|
||||
|
||||
def _get(base: str, batch_id: str) -> dict:
|
||||
return httpx.get(f"{base}/api/v1/upload-batches/{batch_id}", timeout=TIMEOUT).json()
|
||||
|
||||
|
||||
def _start(base: str, batch_id: str) -> httpx.Response:
|
||||
return httpx.post(f"{base}/api/v1/upload-batches/{batch_id}/start", timeout=TIMEOUT)
|
||||
|
||||
|
||||
def _start_accepted(base: str, batch_id: str) -> httpx.Response:
|
||||
"""Start, waiting out the uploader lane the previous attempt still holds.
|
||||
|
||||
A stopped attempt releases its job a moment after the batch itself reaches
|
||||
``cancelled``; ``lock_held`` is that gap, not a refusal of this batch.
|
||||
"""
|
||||
|
||||
def _attempt():
|
||||
response = _start(base, batch_id)
|
||||
if response.status_code == 409 and response.json()["error"]["code"] == "lock_held":
|
||||
return None
|
||||
response.raise_for_status()
|
||||
return response
|
||||
|
||||
return wait_until(_attempt)
|
||||
|
||||
|
||||
def _verify(base: str, batch_id: str) -> dict:
|
||||
response = httpx.post(f"{base}/api/v1/upload-batches/{batch_id}/verify", timeout=TIMEOUT)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _await_state(base: str, batch_id: str, states: set[str], *, timeout: float = 60) -> dict:
|
||||
return wait_until(
|
||||
lambda: (lambda b: b if b.get("state") in states else None)(_get(base, batch_id)),
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def _await_report(base: str, batch_id: str, states: set[str] = TERMINAL) -> dict:
|
||||
"""Wait for a finished attempt *and* the report that explains it.
|
||||
|
||||
The batch state is recorded a moment before its report is parsed — the outcome of
|
||||
the process and the outcome of each file are deliberately separate facts — so a
|
||||
journey that reads per-item evidence must wait for the second one too.
|
||||
"""
|
||||
return wait_until(
|
||||
lambda: (lambda b: b if b.get("state") in states and b.get("parsed_at") else None)(
|
||||
_get(base, batch_id)
|
||||
),
|
||||
timeout=60,
|
||||
)
|
||||
|
||||
|
||||
def _approve(stack, **body) -> dict:
|
||||
"""Preflight, approve exactly that report, and return the created batch."""
|
||||
report = _preflight(stack.base, **body)
|
||||
response = _create(stack.base, report, **body)
|
||||
response.raise_for_status()
|
||||
return response.json()["batches"][0]
|
||||
|
||||
|
||||
def _upload(stack, **body) -> dict:
|
||||
"""The whole approved journey, up to whatever terminal state it reaches."""
|
||||
batch = _approve(stack, **body)
|
||||
_start(stack.base, batch["id"]).raise_for_status()
|
||||
return _await_report(stack.base, batch["id"])
|
||||
|
||||
|
||||
def _outcomes(batch: dict) -> dict[str, str]:
|
||||
return {Path(item["path"]).name: item["outcome"] for item in batch["items"]}
|
||||
|
||||
|
||||
def _uploaded_folder(stack) -> Path:
|
||||
"""The folder the uploader was actually handed, from its own argv log."""
|
||||
invocations = stack.argv()
|
||||
assert invocations, "the uploader was never executed"
|
||||
return Path(invocations[-1].split()[-1])
|
||||
|
||||
|
||||
def _assert_hashes_match_submitted_bytes(stack, batch: dict) -> None:
|
||||
folder = _uploaded_folder(stack)
|
||||
for item in batch["items"]:
|
||||
submitted = folder / Path(item["path"]).name
|
||||
raw = submitted.read_bytes()
|
||||
assert item["sha256"] == hashlib.sha256(raw).hexdigest(), submitted
|
||||
assert item["sha1"] == hashlib.sha1(raw).hexdigest(), submitted # noqa: S324 — Immich's
|
||||
|
||||
|
||||
# ── credentials ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_missing_credentials_block_the_preflight_and_no_upload_is_attempted(stack):
|
||||
stack.start(worker=False, credentials=False)
|
||||
|
||||
report = _preflight(stack.base)
|
||||
|
||||
assert report["state"] == "blocked"
|
||||
assert [issue["code"] for issue in report["blockers"]] == ["credentials_missing"]
|
||||
assert report["credentials"]["api_key_configured"] is False
|
||||
# A blocked scope still issues a token; approving it is what is refused.
|
||||
refused = _create(stack.base, report)
|
||||
assert refused.status_code == 422
|
||||
assert refused.json()["error"]["code"] == "not_ready"
|
||||
assert stack.batches() == []
|
||||
assert stack.argv() == [], "the uploader must not run without credentials"
|
||||
|
||||
|
||||
def test_a_server_that_stops_answering_blocks_the_preflight(stack):
|
||||
stack.start(worker=False)
|
||||
assert _preflight(stack.base)["state"] == "ready"
|
||||
|
||||
stack.immich.stop() # Immich goes away between one preview and the next
|
||||
|
||||
report = _preflight(stack.base)
|
||||
assert report["state"] == "blocked"
|
||||
assert [issue["code"] for issue in report["blockers"]] == ["server_unreachable"]
|
||||
assert report["server"]["reachable"] is False
|
||||
assert _create(stack.base, report).status_code == 422
|
||||
assert stack.argv() == []
|
||||
|
||||
|
||||
# ── EXIF precedes upload ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_exif_checkpoints_must_be_written_before_anything_is_uploaded(unfinished):
|
||||
stack = unfinished
|
||||
stack.start()
|
||||
|
||||
report = _preflight(stack.base)
|
||||
assert report["state"] == "blocked"
|
||||
codes = {
|
||||
issue["code"]
|
||||
for album in report["albums"]
|
||||
for asset in album["assets"]
|
||||
for issue in asset["blockers"]
|
||||
}
|
||||
assert codes == {"safety_exif_unverified"}
|
||||
assert _create(stack.base, report).status_code == 422
|
||||
assert stack.argv() == [], "the uploader must not run before the EXIF checkpoints"
|
||||
|
||||
mark_upload_ready(stack.seeded) # the checkpoints are written
|
||||
|
||||
batch = _upload(stack)
|
||||
assert batch["state"] == "succeeded"
|
||||
assert stack.argv(), "the same scope uploads once its checkpoints exist"
|
||||
# The recorded checkpoints predate the attempt that was allowed to run.
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.models import AnalysisResult, SafetyReview
|
||||
|
||||
with session_factory(stack.seeded) as sf, sf() as session:
|
||||
checkpoints = [
|
||||
*[row.exif_verified_at for row in session.scalars(select(SafetyReview))],
|
||||
*[row.exif_written_at for row in session.scalars(select(AnalysisResult))],
|
||||
]
|
||||
started = batch["started_at"]
|
||||
assert checkpoints and all(stamp.isoformat() < started for stamp in checkpoints)
|
||||
|
||||
|
||||
def test_an_unfinished_photo_blocks_its_album_until_a_partial_upload_is_approved(tmp_path):
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_upload_ready(seeded, unverified=("b",))
|
||||
stack = UploadStack(tmp_path, seeded)
|
||||
try:
|
||||
stack.start()
|
||||
report = _preflight(stack.base)
|
||||
assert report["state"] == "blocked"
|
||||
assert [issue["code"] for issue in report["albums"][0]["blockers"]] == ["partial_scope"]
|
||||
assert report["totals"] == {
|
||||
"albums": 1,
|
||||
"ready_albums": 0,
|
||||
"assets": 2,
|
||||
"eligible": 1,
|
||||
"blocked": 1,
|
||||
}
|
||||
assert _create(stack.base, report).status_code == 422
|
||||
|
||||
partial = _preflight(stack.base, allow_partial=True)
|
||||
assert partial["state"] == "ready"
|
||||
assert partial["token"] != report["token"]
|
||||
# A full-scope approval can never be replayed as a partial one.
|
||||
replayed = _create(stack.base, report, allow_partial=True)
|
||||
assert replayed.status_code == 409
|
||||
assert replayed.json()["error"]["code"] == "stale_preflight"
|
||||
|
||||
batch = _upload(stack, allow_partial=True)
|
||||
assert batch["state"] == "succeeded"
|
||||
assert batch["allow_partial"] is True
|
||||
assert [Path(item["path"]).name for item in batch["items"]] == ["a.jpg"]
|
||||
finally:
|
||||
stack.stop()
|
||||
|
||||
|
||||
# ── outcomes ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_new_album_uploads_and_persists_the_hashes_that_were_submitted(stack):
|
||||
stack.start(uploader=ALL_NEW)
|
||||
|
||||
batch = _upload(stack)
|
||||
|
||||
assert batch["state"] == "succeeded"
|
||||
assert batch["outcome_state"] == "verified"
|
||||
assert _outcomes(batch) == {"a.jpg": "uploaded", "b.jpg": "uploaded"}
|
||||
assert batch["outcome_counts"]["uploaded"] == 2
|
||||
assert batch["report_counts"] == {"uploaded": 2}
|
||||
assert batch["parser"] == "text-v1"
|
||||
_assert_hashes_match_submitted_bytes(stack, batch)
|
||||
# One album, one invocation, scoped to that album's own folder.
|
||||
assert len(stack.argv()) == 1
|
||||
assert f"--album-name={ALBUM}" in stack.argv()[0]
|
||||
assert _uploaded_folder(stack) == stack.seeded.lib / ALBUM
|
||||
|
||||
|
||||
def test_an_exact_duplicate_is_recorded_as_a_duplicate_not_a_new_asset(stack):
|
||||
stack.start(uploader=EXACT_DUPLICATES)
|
||||
|
||||
batch = _upload(stack)
|
||||
|
||||
assert batch["state"] == "succeeded"
|
||||
assert batch["outcome_state"] == "verified"
|
||||
assert set(_outcomes(batch).values()) == {"duplicate"}
|
||||
assert batch["outcome_counts"]["uploaded"] == 0
|
||||
_assert_hashes_match_submitted_bytes(stack, batch)
|
||||
|
||||
|
||||
def test_a_better_copy_is_recorded_as_an_upgrade(stack):
|
||||
stack.start(uploader=UPGRADES)
|
||||
|
||||
batch = _upload(stack)
|
||||
|
||||
assert batch["state"] == "succeeded"
|
||||
assert batch["outcome_state"] == "verified"
|
||||
assert set(_outcomes(batch).values()) == {"upgraded"}
|
||||
assert batch["report_counts"] == {"upgraded": 2}
|
||||
|
||||
|
||||
# ── failure and retry ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_plain_uploader_failure_is_retryable_and_the_retry_succeeds(stack, tmp_path):
|
||||
stack.start(uploader=_once_then(tmp_path, *FAILING_FIRST))
|
||||
|
||||
failed = _upload(stack)
|
||||
|
||||
assert failed["state"] == "failed"
|
||||
assert failed["error_code"] == "uploader_failed"
|
||||
assert failed["exit_code"] == 1
|
||||
assert _outcomes(failed)["a.jpg"] == "failed"
|
||||
# Nothing uncertain happened, so the batch is offered again rather than blocked.
|
||||
assert failed["retry_blockers"] == []
|
||||
|
||||
_start_accepted(stack.base, failed["id"])
|
||||
retried = _await_report(stack.base, failed["id"], {"succeeded"})
|
||||
|
||||
assert retried["attempt_count"] == 2
|
||||
assert _outcomes(retried) == {"a.jpg": "uploaded", "b.jpg": "uploaded"}
|
||||
assert len(stack.argv()) == 2
|
||||
|
||||
|
||||
# ── uncertainty ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_lost_acceptance_response_stays_uncertain_and_is_never_retried_blindly(stack):
|
||||
stack.start(uploader=SILENT_UPLOADER)
|
||||
|
||||
batch = _upload(stack)
|
||||
|
||||
# The process succeeded; what happened to each file is simply not known.
|
||||
assert batch["state"] == "succeeded"
|
||||
assert batch["outcome_state"] == "requires_verification"
|
||||
assert set(_outcomes(batch).values()) == {"unknown"}
|
||||
refused = _start(stack.base, batch["id"])
|
||||
assert refused.status_code == 409
|
||||
assert refused.json()["error"]["code"] == "not_runnable"
|
||||
assert len(stack.argv()) == 1, "a blind retry must not reach the uploader"
|
||||
|
||||
|
||||
def test_verification_asks_immich_and_resolves_every_uncertain_item(stack):
|
||||
stack.start(uploader=SILENT_UPLOADER)
|
||||
stack.immich.mode("present") # Immich holds exactly the bytes that were sent
|
||||
batch = _upload(stack)
|
||||
|
||||
result = _verify(stack.base, batch["id"])
|
||||
|
||||
assert result["counts"] == {"present": 2}
|
||||
assert result["outcome_state"] == "verified"
|
||||
verified = _get(stack.base, batch["id"])
|
||||
assert set(_outcomes(verified).values()) == {"uploaded"}
|
||||
history = httpx.get(
|
||||
f"{stack.base}/api/v1/upload-batches/{batch['id']}/verifications", timeout=TIMEOUT
|
||||
).json()["verifications"]
|
||||
assert {entry["source"] for entry in history} == {"immich_api"}
|
||||
|
||||
|
||||
def test_an_unusable_answer_stays_uncertain_until_an_operator_records_evidence(stack):
|
||||
stack.start(uploader=SILENT_UPLOADER)
|
||||
stack.immich.mode("broken") # answers, but nothing this adapter will interpret
|
||||
batch = _upload(stack)
|
||||
|
||||
result = _verify(stack.base, batch["id"])
|
||||
|
||||
# No answer is never "no": the items stay uncertain rather than being called failed.
|
||||
assert result["counts"] == {"inconclusive": 2}
|
||||
assert result["outcome_state"] == "requires_verification"
|
||||
assert set(_outcomes(_get(stack.base, batch["id"])).values()) == {"unknown"}
|
||||
|
||||
asset_id = batch["items"][0]["asset_id"]
|
||||
incomplete = httpx.post(
|
||||
f"{stack.base}/api/v1/upload-batches/{batch['id']}/resolve",
|
||||
json={"asset_id": asset_id, "outcome": "uploaded", "evidence": "", "actor": "dom"},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
assert incomplete.status_code == 422
|
||||
assert incomplete.json()["error"]["code"] == "evidence_required"
|
||||
|
||||
resolved = httpx.post(
|
||||
f"{stack.base}/api/v1/upload-batches/{batch['id']}/resolve",
|
||||
json={
|
||||
"asset_id": asset_id,
|
||||
"outcome": "uploaded",
|
||||
"evidence": "found it in Immich by checksum",
|
||||
"actor": "dom",
|
||||
},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
resolved.raise_for_status()
|
||||
assert _outcomes(_get(stack.base, batch["id"]))["a.jpg"] == "uploaded"
|
||||
manual = httpx.get(
|
||||
f"{stack.base}/api/v1/upload-batches/{batch['id']}/verifications", timeout=TIMEOUT
|
||||
).json()["verifications"][-1]
|
||||
assert manual["source"] == "operator"
|
||||
assert manual["actor"] == "dom"
|
||||
|
||||
|
||||
def test_bytes_edited_after_the_upload_are_flagged_and_block_another_run(stack):
|
||||
stack.start(uploader=ALL_NEW)
|
||||
batch = _upload(stack)
|
||||
assert batch["state"] == "succeeded"
|
||||
|
||||
(stack.seeded.lib / ALBUM / "a.jpg").write_bytes(b"edited after the upload")
|
||||
|
||||
result = _verify(stack.base, batch["id"])
|
||||
assert result["stale_bytes"] is True
|
||||
changed = [item for item in result["items"] if item["changed_after_upload"]]
|
||||
assert [Path(item["path"]).name for item in changed] == ["a.jpg"]
|
||||
refused = _start(stack.base, batch["id"])
|
||||
assert refused.status_code == 409
|
||||
assert refused.json()["error"]["code"] == "changed_after_upload"
|
||||
# The preflight agrees: those bytes are no longer approved for any new upload.
|
||||
report = _preflight(stack.base)
|
||||
assert report["state"] == "blocked"
|
||||
assert "bytes_changed" in {
|
||||
issue["code"]
|
||||
for album in report["albums"]
|
||||
for asset in album["assets"]
|
||||
for issue in asset["blockers"]
|
||||
}
|
||||
|
||||
|
||||
# ── cancellation and resume ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_running_album_can_be_stopped_and_run_again_from_that_boundary(stack, tmp_path):
|
||||
stack.start(uploader=_once_then(tmp_path, *SLOW_FIRST))
|
||||
batch = _approve(stack)
|
||||
_start(stack.base, batch["id"]).raise_for_status()
|
||||
_await_state(stack.base, batch["id"], {"running"})
|
||||
|
||||
httpx.post(
|
||||
f"{stack.base}/api/v1/upload-batches/{batch['id']}/cancel", timeout=TIMEOUT
|
||||
).raise_for_status()
|
||||
cancelled = _await_state(stack.base, batch["id"], {"cancelled"})
|
||||
|
||||
# A stopped album is a clean boundary, not an uncertain one.
|
||||
assert cancelled["retry_blockers"] == []
|
||||
_start_accepted(stack.base, batch["id"])
|
||||
resumed = _await_report(stack.base, batch["id"], {"succeeded"})
|
||||
assert resumed["attempt_count"] == 2
|
||||
assert _outcomes(resumed) == {"a.jpg": "uploaded", "b.jpg": "uploaded"}
|
||||
|
||||
|
||||
def test_an_interrupted_attempt_is_uncertain_after_a_restart_and_stays_blocked(stack, tmp_path):
|
||||
"""The worker vanishes mid-upload: Immich may hold the files, so the outcome is
|
||||
unknown. Startup recovery must say so, refuse a retry, and survive the restart."""
|
||||
stack.start(uploader=_once_then(tmp_path, *SLOW_FIRST))
|
||||
batch = _approve(stack)
|
||||
_start(stack.base, batch["id"]).raise_for_status()
|
||||
_await_state(stack.base, batch["id"], {"running"})
|
||||
|
||||
stack.worker.kill() # no chance to record any outcome
|
||||
stack.worker.wait(timeout=10)
|
||||
stack.restart_server()
|
||||
|
||||
recovered = _get(stack.base, batch["id"])
|
||||
assert recovered["state"] == "unknown_requires_verification"
|
||||
assert recovered["error_code"] == "interrupted"
|
||||
assert recovered["attempt_count"] == 1
|
||||
refused = _start(stack.base, batch["id"])
|
||||
assert refused.status_code == 409
|
||||
assert refused.json()["error"]["code"] == "requires_verification"
|
||||
|
||||
# Verification is the only way out, and it is what makes the batch certain again.
|
||||
stack.immich.mode("present")
|
||||
result = _verify(stack.base, batch["id"])
|
||||
assert result["state"] == "succeeded"
|
||||
assert result["outcome_state"] == "verified"
|
||||
|
||||
stack.restart_server() # the resolution is durable, not in-process memory
|
||||
after = _get(stack.base, batch["id"])
|
||||
assert after["state"] == "succeeded"
|
||||
assert set(_outcomes(after).values()) == {"uploaded"}
|
||||
|
||||
|
||||
# ── privacy ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_no_secret_appears_in_any_retained_artifact(stack):
|
||||
stack.start(uploader=ALL_NEW)
|
||||
planned = _approve(stack)
|
||||
job = _start(stack.base, planned["id"]).json()["job"]
|
||||
batch = _await_report(stack.base, planned["id"])
|
||||
stack.immich.mode("present")
|
||||
_verify(stack.base, batch["id"])
|
||||
|
||||
# The uploader really was given the key…
|
||||
assert f"--api-key={SENTINEL_KEY}" in stack.argv()[0]
|
||||
# …and it is in nothing that was kept: not the database, not the retained report,
|
||||
# not any response the browser can read.
|
||||
retained = [path for path in stack.seeded.data.rglob("*") if path.is_file()]
|
||||
assert any(path.suffix == ".log" for path in retained), "the report was not retained"
|
||||
for path in retained:
|
||||
assert SENTINEL_KEY.encode() not in path.read_bytes(), path
|
||||
for url in (
|
||||
f"{stack.base}/api/v1/upload-batches",
|
||||
f"{stack.base}/api/v1/upload-batches/{batch['id']}",
|
||||
f"{stack.base}/api/v1/upload-batches/{batch['id']}/verifications",
|
||||
f"{stack.base}/api/v1/jobs/{job['id']}",
|
||||
f"{stack.base}/api/v1/jobs/{job['id']}/events",
|
||||
):
|
||||
assert SENTINEL_KEY not in httpx.get(url, timeout=TIMEOUT).text, url
|
||||
assert SENTINEL_KEY not in httpx.post(
|
||||
f"{stack.base}/api/v1/upload-preflight", json={}, timeout=TIMEOUT
|
||||
).text
|
||||
assert "--api-key=***" in " ".join(_get(stack.base, batch["id"])["command"])
|
||||
505
tests/e2e/test_phase_f_pipeline.py
Normal file
505
tests/e2e/test_phase_f_pipeline.py
Normal file
@@ -0,0 +1,505 @@
|
||||
"""Phase F end-to-end acceptance (US06-06): archive lifecycle, black box.
|
||||
|
||||
Every journey drives a real ``photo_pipeline serve`` child process and a real
|
||||
durable worker over HTTP — register a medium, preflight it, archive an album,
|
||||
crash mid-transfer, recover, unmount the medium, rediscover the photos through
|
||||
their hashes while they are unreachable, mount it again, restore, and collide.
|
||||
Nothing is reached into: the medium is an ordinary directory whose marker file is
|
||||
its identity, and unmounting it means taking that marker away, which is what the
|
||||
application sees when a disk is unplugged.
|
||||
|
||||
Three invariants are asserted wherever they apply, because they are what make the
|
||||
only *removing* stage safe:
|
||||
|
||||
- **No source is removed before its archive copy is verified.** Every crash barrier
|
||||
is checked for the file still being somewhere: either in the library, or on the
|
||||
medium hashing exactly as recorded.
|
||||
- **Archived is not missing.** An unmounted medium leaves its photos
|
||||
``archived_offline``, still hashed, still deduplicable, still previewable.
|
||||
- **Restoring never overwrites.** A taken name comes back beside its occupant, and
|
||||
the archived copy stays on the medium.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from tests.e2e._pipeline_harness import (
|
||||
ArchiveStack,
|
||||
mark_uploaded,
|
||||
seed_album,
|
||||
session_factory,
|
||||
wait_until,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.phase_f
|
||||
|
||||
TIMEOUT = 20
|
||||
ALBUM = "rome"
|
||||
MANIFEST = "archive-manifest.jsonl"
|
||||
# Every persisted transition the transfer can die at, in the order it reaches them.
|
||||
BARRIERS = ["transferring", "verified", "removing", "source_removed", "complete"]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stack(tmp_path):
|
||||
"""An analysed album with verified upload evidence — archivable, nothing else."""
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_uploaded(seeded)
|
||||
running = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
# ── helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _preflight(stack, location_id: str, **body) -> dict:
|
||||
response = httpx.post(
|
||||
f"{stack.base}/api/v1/archive-preflight",
|
||||
json={"location_id": location_id, **body},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _plan(stack, location_id: str, **body) -> dict:
|
||||
report = _preflight(stack, location_id, **body)
|
||||
response = httpx.post(
|
||||
f"{stack.base}/api/v1/archive-plans",
|
||||
json={"location_id": location_id, "token": report["token"], **body},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _apply(stack, plan_id: str) -> httpx.Response:
|
||||
return httpx.post(f"{stack.base}/api/v1/archive-plans/{plan_id}/apply", timeout=TIMEOUT)
|
||||
|
||||
|
||||
def _plan_state(stack, plan_id: str) -> dict:
|
||||
return httpx.get(f"{stack.base}/api/v1/archive-plans/{plan_id}", timeout=TIMEOUT).json()
|
||||
|
||||
|
||||
def _await_plan(stack, plan_id: str, states=("complete", "failed"), *, timeout: float = 60) -> dict:
|
||||
return wait_until(
|
||||
lambda: (lambda p: p if p.get("state") in states else None)(_plan_state(stack, plan_id)),
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def _archive_album(stack) -> dict:
|
||||
"""Register, preflight, plan, apply, and wait for the run to finish."""
|
||||
location = stack.register()
|
||||
plan = _plan(stack, location["id"])
|
||||
_apply(stack, plan["id"]).raise_for_status()
|
||||
return {"location": location, "plan": _await_plan(stack, plan["id"])}
|
||||
|
||||
|
||||
def _assets(stack) -> dict[str, dict]:
|
||||
return {asset["id"]: asset for asset in stack.assets()}
|
||||
|
||||
|
||||
def _recovery(stack) -> dict:
|
||||
return httpx.get(f"{stack.base}/api/v1/archive-recovery", timeout=TIMEOUT).json()
|
||||
|
||||
|
||||
def _resolve(stack) -> dict:
|
||||
response = httpx.post(f"{stack.base}/api/v1/archive-recovery/resolve", timeout=TIMEOUT)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def _restore(stack, location_id: str) -> dict:
|
||||
report = httpx.post(
|
||||
f"{stack.base}/api/v1/restore-preflight",
|
||||
json={"location_id": location_id},
|
||||
timeout=TIMEOUT,
|
||||
).json()
|
||||
plan = httpx.post(
|
||||
f"{stack.base}/api/v1/restore-plans",
|
||||
json={"location_id": location_id, "token": report["token"]},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
plan.raise_for_status()
|
||||
plan = plan.json()
|
||||
httpx.post(
|
||||
f"{stack.base}/api/v1/restore-plans/{plan['id']}/apply", timeout=TIMEOUT
|
||||
).raise_for_status()
|
||||
wait_until(
|
||||
lambda: all(a["availability_state"] == "active" for a in stack.assets()), timeout=60
|
||||
)
|
||||
return {"preflight": report, "plan": plan}
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def _contents(*roots: Path) -> set[str]:
|
||||
"""Every photo byte-string reachable anywhere, keyed by hash."""
|
||||
return {
|
||||
_sha256(path)
|
||||
for root in roots
|
||||
for path in root.rglob("*.jpg")
|
||||
if path.is_file() and not path.name.startswith(".")
|
||||
}
|
||||
|
||||
|
||||
def _manifest(stack) -> list[dict]:
|
||||
path = stack.archive / ALBUM / MANIFEST
|
||||
if not path.exists():
|
||||
return []
|
||||
return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
|
||||
|
||||
|
||||
# ── preflight blockers ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_preflight_refuses_an_offline_medium_a_wrong_volume_and_a_full_disk(stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
|
||||
ready = _preflight(stack, location["id"])
|
||||
assert ready["state"] == "ready"
|
||||
assert ready["totals"]["assets"] == 2
|
||||
|
||||
stack.unmount()
|
||||
offline = _preflight(stack, location["id"])
|
||||
assert offline["state"] == "blocked"
|
||||
assert "location_offline" in {issue["code"] for issue in offline["blockers"]}
|
||||
|
||||
# A different disk mounted at the same place is not this medium.
|
||||
(stack.archive / ArchiveStack.MARKER).write_text('{"media_id": "another-disk"}')
|
||||
wrong = _preflight(stack, location["id"])
|
||||
assert "wrong_volume" in {issue["code"] for issue in wrong["blockers"]}
|
||||
|
||||
# Nothing has been created, moved, or removed by any of that.
|
||||
assert stack.plans() == []
|
||||
assert sorted(p.name for p in (stack.seeded.lib / ALBUM).glob("*.jpg")) == ["a.jpg", "b.jpg"]
|
||||
|
||||
|
||||
def test_preflight_refuses_capacity_it_cannot_prove_and_bytes_it_cannot_vouch_for(tmp_path):
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_uploaded(seeded)
|
||||
stack = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
# A reserve larger than any disk: the scope cannot be promised room.
|
||||
stack.start(
|
||||
worker=False,
|
||||
extra_env={"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": str(1 << 62)},
|
||||
)
|
||||
location = stack.register()
|
||||
report = _preflight(stack, location["id"])
|
||||
assert "insufficient_capacity" in {issue["code"] for issue in report["blockers"]}
|
||||
|
||||
# And a plan for a blocked scope is refused rather than half-created.
|
||||
refused = httpx.post(
|
||||
f"{stack.base}/api/v1/archive-plans",
|
||||
json={"location_id": location["id"], "token": report["token"]},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
assert refused.status_code == 422
|
||||
assert refused.json()["error"]["code"] == "blocked"
|
||||
assert stack.plans() == []
|
||||
finally:
|
||||
stack.stop()
|
||||
|
||||
|
||||
def test_an_album_whose_bytes_changed_after_upload_cannot_be_archived(stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
(stack.seeded.lib / ALBUM / "a.jpg").write_bytes(b"edited after the upload")
|
||||
|
||||
report = _preflight(stack, location["id"])
|
||||
album = report["albums"][0]
|
||||
assert album["state"] == "blocked"
|
||||
assert "partial_scope" in {issue["code"] for issue in album["blockers"]}
|
||||
assert "bytes_changed" in {
|
||||
issue["code"] for asset in album["assets"] for issue in asset["blockers"]
|
||||
}
|
||||
assert report["state"] == "blocked"
|
||||
|
||||
|
||||
# ── copy, verify, remove ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_an_archived_album_leaves_the_library_only_after_it_is_verified(stack):
|
||||
stack.start()
|
||||
before = _contents(stack.seeded.lib)
|
||||
hashes = {a["id"]: a["current_sha256"] for a in stack.assets()}
|
||||
|
||||
result = _archive_album(stack)
|
||||
plan = result["plan"]
|
||||
|
||||
assert plan["state"] == "complete"
|
||||
assert {op["journal_state"] for op in plan["operations"]} == {"complete"}
|
||||
# Same filesystem here, so the transfer took the atomic-move path — recorded as
|
||||
# what actually happened, not as the plan's guess.
|
||||
assert all(op["same_filesystem"] for op in plan["operations"])
|
||||
|
||||
# The bytes are on the medium, hashing exactly as recorded, and gone from the
|
||||
# library. Not one photo was lost in between.
|
||||
assert _contents(stack.archive) == before
|
||||
assert not list((stack.seeded.lib / ALBUM).glob("*.jpg"))
|
||||
for asset_id, sha256 in hashes.items():
|
||||
asset = _assets(stack)[asset_id]
|
||||
assert asset["current_path"] is None
|
||||
assert asset["availability_state"] == "archived_online"
|
||||
assert _sha256(stack.archive / asset["archive_path"]) == sha256
|
||||
|
||||
# The medium carries its own record of what it holds.
|
||||
manifest = _manifest(stack)
|
||||
assert {entry["asset_id"] for entry in manifest} == set(hashes)
|
||||
assert {entry["sha256"] for entry in manifest} == set(hashes.values())
|
||||
assert {entry["media_id"] for entry in manifest} == {result["location"]["media_id"]}
|
||||
|
||||
|
||||
def test_the_archived_state_survives_a_server_restart(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
before = _assets(stack)
|
||||
plan_before = _plan_state(stack, result["plan"]["id"])
|
||||
|
||||
stack.restart_server()
|
||||
|
||||
assert _assets(stack) == before
|
||||
assert _plan_state(stack, result["plan"]["id"]) == plan_before
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
||||
|
||||
|
||||
# ── interruption at every persisted transition ───────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("barrier", BARRIERS)
|
||||
def test_a_worker_killed_at_each_transition_loses_nothing_and_recovers(tmp_path, barrier):
|
||||
# One photo, so the crash lands on the only operation and the end state is the
|
||||
# recovery's own doing rather than a mixture with untouched work.
|
||||
seeded = seed_album(tmp_path, names=("a.jpg",))
|
||||
mark_uploaded(seeded)
|
||||
stack = ArchiveStack(tmp_path, seeded)
|
||||
try:
|
||||
# Only the worker carries the crash barrier: it dies the instant this journal
|
||||
# state is persisted, while the server stays up to be asked what happened.
|
||||
stack.start(worker=False)
|
||||
stack.restart_worker(extra_env={"PHOTO_PIPELINE_FAULT_AFTER": barrier})
|
||||
before = _contents(stack.seeded.lib)
|
||||
location = stack.register()
|
||||
plan = _plan(stack, location["id"])
|
||||
_apply(stack, plan["id"]).raise_for_status()
|
||||
|
||||
wait_until(lambda: stack.worker.poll() is not None, timeout=60)
|
||||
assert stack.worker.returncode in (9, -9), "the worker should have been killed"
|
||||
|
||||
# Whatever the crash interrupted, every photo is still readable somewhere.
|
||||
assert before <= _contents(stack.seeded.lib, stack.archive), f"content lost at {barrier}"
|
||||
# And nothing was removed that had not been verified first.
|
||||
for operation in _plan_state(stack, plan["id"])["operations"]:
|
||||
source = Path(operation["source_path"])
|
||||
if not source.exists():
|
||||
archived = Path(operation["destination_path"])
|
||||
assert archived.exists(), f"{barrier}: source removed without an archive copy"
|
||||
assert _sha256(archived) == operation["expected_sha256"]
|
||||
|
||||
# Recovery is offered from evidence, resolves without a worker, and converges.
|
||||
verdicts = _recovery(stack)
|
||||
assert all(v["classification"] != "manual" for v in verdicts["operations"]), verdicts
|
||||
_resolve(stack)
|
||||
assert _recovery(stack)["operations"] == []
|
||||
# Repeating it changes nothing.
|
||||
assert _resolve(stack) == {"resumed": 0, "completed": 0, "manual": 0}
|
||||
|
||||
# Recovery finishes what was durable and re-plans what was not. Before the
|
||||
# archive copy existed, that means putting the item back to `planned` with the
|
||||
# original untouched; from `verified` onwards it means completing it.
|
||||
stack.restart_server()
|
||||
operation = _plan_state(stack, plan["id"])["operations"][0]
|
||||
asset = next(iter(stack.assets()))
|
||||
if barrier == "transferring":
|
||||
assert operation["journal_state"] == "planned"
|
||||
assert Path(operation["source_path"]).exists()
|
||||
assert asset["availability_state"] == "active"
|
||||
assert _contents(stack.seeded.lib) == before
|
||||
else:
|
||||
assert operation["journal_state"] == "complete"
|
||||
assert not Path(operation["source_path"]).exists()
|
||||
assert _contents(stack.archive) == before
|
||||
assert asset["availability_state"] == "archived_online"
|
||||
assert _sha256(stack.archive / asset["archive_path"]) == asset["current_sha256"]
|
||||
assert {entry["sha256"] for entry in _manifest(stack)} == before
|
||||
finally:
|
||||
stack.stop()
|
||||
|
||||
|
||||
def test_evidence_that_contradicts_the_journal_is_never_guessed(stack):
|
||||
stack.start(worker=False)
|
||||
location = stack.register()
|
||||
plan = _plan(stack, location["id"])
|
||||
# The journal claims a verified archive copy the medium does not have.
|
||||
with session_factory(stack.seeded) as sf:
|
||||
from photo_pipeline.services.archive_journal import ArchiveJournal
|
||||
|
||||
journal = ArchiveJournal(sf)
|
||||
operation = journal.operations(plan["id"])[0]
|
||||
journal.begin(operation["id"], worker_id="crashed", fencing_token=1)
|
||||
journal.transition(operation["id"], "verified", fencing_token=1)
|
||||
|
||||
verdicts = _recovery(stack)
|
||||
assert [v["classification"] for v in verdicts["operations"]] == ["manual"]
|
||||
assert _resolve(stack)["manual"] == 1
|
||||
# The source is untouched and the unresolved item keeps blocking a new archive.
|
||||
assert (stack.seeded.lib / ALBUM / "a.jpg").exists()
|
||||
blocked = _preflight(stack, location["id"])
|
||||
assert "archive_pending" in {issue["code"] for issue in blocked["blockers"]}
|
||||
|
||||
|
||||
# ── offline identity and deduplication ───────────────────────────────────────
|
||||
|
||||
|
||||
def test_offline_photos_stay_hashed_previewable_and_deduplicable(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
archived = _assets(stack)
|
||||
copy_source = stack.archive / next(iter(archived.values()))["archive_path"]
|
||||
exact = stack.seeded.lib / "inbox" / "again.jpg"
|
||||
exact.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(copy_source, exact)
|
||||
stack.unmount()
|
||||
|
||||
scanned = httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60)
|
||||
scanned.raise_for_status()
|
||||
# An unmounted medium is not a missing file.
|
||||
assert scanned.json()["counts"].get("missing") is None
|
||||
offline = {a["id"]: a for a in stack.assets() if a["id"] in archived}
|
||||
assert {a["availability_state"] for a in offline.values()} == {"archived_offline"}
|
||||
assert all(a["current_sha256"] for a in offline.values())
|
||||
|
||||
# The new active copy is recognised as the archived photo, not as a new one.
|
||||
httpx.post(f"{stack.base}/api/v1/duplicates/detect", timeout=60).raise_for_status()
|
||||
clusters = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters", params={"limit": 50}, timeout=TIMEOUT
|
||||
).json()["items"]
|
||||
exact_clusters = [c for c in clusters if c["method"] == "exact"]
|
||||
assert len(exact_clusters) == 1
|
||||
detail = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters/{exact_clusters[0]['id']}", timeout=TIMEOUT
|
||||
).json()
|
||||
canonical = next(m for m in detail["members"] if m["asset_id"] == detail["canonical_asset_id"])
|
||||
assert canonical["availability_state"] == "archived_offline"
|
||||
assert canonical["archive_location"] == result["location"]["name"]
|
||||
assert canonical["preview"]["state"] == "ready" and canonical["preview"]["protected"]
|
||||
|
||||
# And the retained preview really is served while the medium is away.
|
||||
thumbnail = httpx.get(
|
||||
f"{stack.base}/api/v1/assets/{canonical['asset_id']}/thumbnail",
|
||||
params={"size": 1280},
|
||||
timeout=TIMEOUT,
|
||||
)
|
||||
assert thumbnail.status_code == 200
|
||||
assert thumbnail.headers["content-type"] == "image/webp"
|
||||
|
||||
|
||||
def test_a_fuzzy_offline_match_asks_for_the_medium_instead_of_guessing(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
archived = next(iter(_assets(stack).values()))
|
||||
variant = stack.seeded.lib / "inbox" / "resized.jpg"
|
||||
variant.parent.mkdir(parents=True, exist_ok=True)
|
||||
_resize(stack.archive / archived["archive_path"], variant)
|
||||
stack.unmount()
|
||||
|
||||
httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60).raise_for_status()
|
||||
httpx.post(f"{stack.base}/api/v1/duplicates/detect", timeout=60).raise_for_status()
|
||||
clusters = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters", params={"limit": 50}, timeout=TIMEOUT
|
||||
).json()["items"]
|
||||
perceptual = [c for c in clusters if c["method"] == "perceptual"]
|
||||
assert len(perceptual) == 1
|
||||
|
||||
detail = httpx.get(
|
||||
f"{stack.base}/api/v1/duplicates/clusters/{perceptual[0]['id']}", timeout=TIMEOUT
|
||||
).json()
|
||||
# A fuzzy match is never decided automatically, and full-resolution review names
|
||||
# the medium to mount rather than guessing from the preview.
|
||||
assert detail["state"] == "open" and detail["requires_confirmation"] is True
|
||||
assert detail["mount_required"] == [result["location"]["name"]]
|
||||
|
||||
|
||||
def test_mounting_the_medium_again_makes_the_originals_reachable(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
stack.unmount()
|
||||
httpx.post(f"{stack.base}/api/v1/inventory/scan", timeout=60).raise_for_status()
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_offline"}
|
||||
# Restoring from a medium that is not there is refused, with its reason.
|
||||
away = httpx.post(
|
||||
f"{stack.base}/api/v1/restore-preflight",
|
||||
json={"location_id": result["location"]["id"]},
|
||||
timeout=TIMEOUT,
|
||||
).json()
|
||||
assert away["state"] == "blocked"
|
||||
assert "location_offline" in {issue["code"] for issue in away["blockers"]}
|
||||
|
||||
stack.remount()
|
||||
locations = httpx.get(f"{stack.base}/api/v1/archive-locations", timeout=TIMEOUT).json()
|
||||
assert locations["locations"][0]["state"] == "online"
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"archived_online"}
|
||||
|
||||
|
||||
# ── restore ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_restoring_returns_the_bytes_without_losing_identity_or_the_archive(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
identities = set(_assets(stack))
|
||||
hashes = {a["id"]: a["current_sha256"] for a in stack.assets()}
|
||||
|
||||
_restore(stack, result["location"]["id"])
|
||||
|
||||
restored = _assets(stack)
|
||||
assert set(restored) == identities # the same photos, not new ones
|
||||
for asset_id, sha256 in hashes.items():
|
||||
asset = restored[asset_id]
|
||||
assert asset["availability_state"] == "active"
|
||||
assert _sha256(Path(asset["current_path"])) == sha256
|
||||
# The archive copy is a copy: restoring empties nothing.
|
||||
assert (stack.archive / asset["archive_path"]).exists()
|
||||
|
||||
stack.restart_server()
|
||||
assert _assets(stack) == restored
|
||||
|
||||
|
||||
def test_a_restore_comes_back_beside_an_occupant_never_over_it(stack):
|
||||
stack.start()
|
||||
result = _archive_album(stack)
|
||||
squatter = stack.seeded.lib / ALBUM / "a.jpg"
|
||||
squatter.parent.mkdir(parents=True, exist_ok=True)
|
||||
squatter.write_bytes(b"a different photo lives here now")
|
||||
|
||||
report = _restore(stack, result["location"]["id"])["preflight"]
|
||||
destinations = {item["destination_path"] for item in report["items"]}
|
||||
assert any("(restored)" in destination for destination in destinations)
|
||||
|
||||
assert squatter.read_bytes() == b"a different photo lives here now"
|
||||
assert (stack.seeded.lib / ALBUM / "a (restored).jpg").exists()
|
||||
assert {a["availability_state"] for a in stack.assets()} == {"active"}
|
||||
|
||||
|
||||
def _resize(source: Path, destination: Path, scale: float = 0.5) -> None:
|
||||
from PIL import Image
|
||||
|
||||
with Image.open(source) as opened:
|
||||
opened.resize(
|
||||
(int(opened.width * scale), int(opened.height * scale)), Image.LANCZOS
|
||||
).save(destination, quality=95)
|
||||
@@ -13,6 +13,8 @@ REPO = Path(__file__).resolve().parents[2]
|
||||
MAP = json.loads((REPO / "tests" / "story_traceability.json").read_text())["stories"]
|
||||
PHASE_A_STORIES = {f"US01-0{n}" for n in range(1, 8)}
|
||||
PHASE_D_STORIES = {f"US04-0{n}" for n in range(1, 7)}
|
||||
PHASE_E_STORIES = {f"US05-0{n}" for n in range(1, 7)}
|
||||
PHASE_F_STORIES = {f"US06-0{n}" for n in range(1, 7)}
|
||||
|
||||
|
||||
def test_all_phase_a_stories_are_mapped():
|
||||
@@ -25,6 +27,19 @@ def test_all_phase_d_stories_are_mapped():
|
||||
assert PHASE_D_STORIES <= set(MAP)
|
||||
|
||||
|
||||
def test_all_phase_e_stories_are_mapped():
|
||||
"""US05-06 acceptance: every upload story, preflight through browser, is tied to
|
||||
automated tests — upload is the one stage the app cannot take back."""
|
||||
assert PHASE_E_STORIES <= set(MAP)
|
||||
|
||||
|
||||
def test_all_phase_f_stories_are_mapped():
|
||||
"""US06-06 acceptance: every archive-lifecycle story, destination through
|
||||
restore, is tied to automated tests — archiving is the only stage that removes
|
||||
an original."""
|
||||
assert PHASE_F_STORIES <= set(MAP)
|
||||
|
||||
|
||||
def test_every_mapped_test_file_exists_and_is_nonempty():
|
||||
for story, files in MAP.items():
|
||||
assert files, f"{story} maps to no tests"
|
||||
|
||||
283
tests/e2e/test_uploads_ui.py
Normal file
283
tests/e2e/test_uploads_ui.py
Normal file
@@ -0,0 +1,283 @@
|
||||
"""Browser journeys for the upload view (US05-05).
|
||||
|
||||
Covers the preflight preview (scope, redacted configuration, blockers, exact
|
||||
confirmation), a real upload through the real worker with per-outcome progress,
|
||||
stopping a running album, and the two ways out of an uncertain outcome —
|
||||
verification against Immich and a manual resolution that records its evidence.
|
||||
|
||||
Nothing external is mocked inside the browser: the uploader is a real executable
|
||||
driven by the real worker process, and Immich is a real HTTP server answering the
|
||||
same ``ping``/``bulk-upload-check`` endpoints the adapter calls in production. The
|
||||
API key is a sentinel string, so the last test can prove it never reached the page.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import expect
|
||||
|
||||
from tests.e2e._pipeline_harness import (
|
||||
SENTINEL_KEY,
|
||||
SILENT_UPLOADER,
|
||||
UPLOADER_VERSION,
|
||||
UploadStack,
|
||||
mark_upload_ready,
|
||||
seed_album,
|
||||
session_factory,
|
||||
wait_until,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.phase_e
|
||||
|
||||
TIMEOUT = 10
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stack(tmp_path):
|
||||
seeded = seed_album(tmp_path)
|
||||
mark_upload_ready(seeded)
|
||||
running = UploadStack(tmp_path, seeded)
|
||||
try:
|
||||
yield running
|
||||
finally:
|
||||
running.stop()
|
||||
|
||||
|
||||
def _open(page, stack) -> None:
|
||||
page.goto(f"{stack.base}/app/#/uploads")
|
||||
page.get_by_test_id("upload-scope").wait_for()
|
||||
|
||||
|
||||
def _upload(page, stack) -> None:
|
||||
"""Confirm the upload and wait for the worker to finish the album."""
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-upload").click()
|
||||
expect(page.get_by_test_id("detail-state")).not_to_have_text("planned", timeout=30_000)
|
||||
|
||||
|
||||
# ── preflight ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_preview_shows_scope_configuration_and_an_exact_confirmation(page, stack):
|
||||
errors = []
|
||||
page.on("console", lambda m: errors.append(m.text) if m.type == "error" else None)
|
||||
stack.start(worker=False)
|
||||
_open(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("config-server")).to_have_text(stack.immich.url)
|
||||
expect(page.get_by_test_id("config-key")).to_have_text("configured (never shown)")
|
||||
expect(page.get_by_test_id("config-reachable")).to_have_text("yes")
|
||||
expect(page.get_by_test_id("config-uploader")).to_have_text(UPLOADER_VERSION)
|
||||
|
||||
row = page.get_by_test_id("album-row").first
|
||||
expect(row.get_by_test_id("album-name")).to_have_text("rome")
|
||||
expect(row.get_by_test_id("album-immich-name")).to_have_text("rome")
|
||||
expect(row.get_by_test_id("album-eligible")).to_have_text("2")
|
||||
expect(row.get_by_test_id("album-state")).to_have_text("ready")
|
||||
# The exact invocation is previewed, with the key masked at the source.
|
||||
command = row.get_by_test_id("album-command").inner_text()
|
||||
assert "upload from-folder" in command and "--album-name=rome" in command
|
||||
assert "--api-key=***" in command
|
||||
|
||||
# The confirmation names the scope it is about to send, not just "Upload".
|
||||
expect(page.get_by_test_id("start-upload")).to_have_text("Upload 1 album(s) · 2 photo(s)")
|
||||
expect(page.get_by_test_id("start-upload")).to_be_enabled()
|
||||
assert errors == [], f"console errors: {errors}"
|
||||
|
||||
|
||||
def test_an_unfinished_photo_blocks_its_album_and_the_confirmation(page, stack):
|
||||
mark_upload_ready(stack.seeded, unverified=("b",))
|
||||
stack.start(worker=False)
|
||||
_open(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("album-state")).to_have_text("blocked")
|
||||
expect(page.get_by_test_id("album-blocker")).to_have_attribute("data-code", "partial_scope")
|
||||
expect(page.get_by_test_id("album-eligible")).to_have_text("1")
|
||||
expect(page.get_by_test_id("album-blocked")).to_have_text("1")
|
||||
expect(page.get_by_test_id("start-upload")).to_be_disabled()
|
||||
|
||||
# Partial upload exists, but only as a deliberate act: ticking it re-runs the
|
||||
# preflight under that policy and the confirmation then names the smaller scope.
|
||||
page.get_by_test_id("allow-partial").check()
|
||||
expect(page.get_by_test_id("album-state")).to_have_text("ready")
|
||||
expect(page.get_by_test_id("start-upload")).to_have_text("Upload 1 album(s) · 1 photo(s)")
|
||||
assert stack.batches() == [], "nothing may be created by previewing"
|
||||
|
||||
|
||||
# ── uploading ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_confirmed_upload_runs_and_reports_each_outcome(page, stack):
|
||||
stack.start()
|
||||
_upload(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("succeeded")
|
||||
expect(page.get_by_test_id("count-new")).to_have_text("new: 1")
|
||||
expect(page.get_by_test_id("count-duplicate")).to_have_text("duplicate: 1")
|
||||
expect(page.get_by_test_id("count-uncertain")).to_have_text("uncertain: 0")
|
||||
expect(page.get_by_test_id("count-failed")).to_have_text("failed: 0")
|
||||
expect(page.get_by_test_id("batch-outcome-state")).to_have_text("verified")
|
||||
|
||||
outcomes = sorted(page.get_by_test_id("item-outcome").all_inner_texts())
|
||||
assert outcomes == ["duplicate", "new"]
|
||||
# A finished album offers no restart: the server would refuse one.
|
||||
expect(page.get_by_test_id("retry-blocked")).to_contain_text("not_runnable")
|
||||
expect(page.get_by_test_id("start-batch")).to_have_count(0)
|
||||
|
||||
|
||||
def test_a_running_album_can_be_stopped(page, stack):
|
||||
stack.start(uploader='echo "INFO starting"; sleep 20; exit 0\n')
|
||||
_open(page, stack)
|
||||
page.get_by_test_id("start-upload").click()
|
||||
|
||||
stop = page.get_by_test_id("cancel-batch")
|
||||
expect(stop).to_have_text("Stop after the current file", timeout=30_000)
|
||||
stop.click()
|
||||
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("cancelled", timeout=30_000)
|
||||
# A stopped album is a clean boundary, not an uncertain one: it can run again.
|
||||
expect(page.get_by_test_id("start-batch")).to_be_visible()
|
||||
|
||||
|
||||
def test_the_finished_upload_survives_a_reload(page, stack):
|
||||
stack.start()
|
||||
_upload(page, stack)
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("succeeded")
|
||||
|
||||
page.reload()
|
||||
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("succeeded")
|
||||
expect(page.get_by_test_id("count-new")).to_have_text("new: 1")
|
||||
# The result banner is this tab's memory, not server state, so it stays gone.
|
||||
expect(page.get_by_test_id("upload-result")).to_have_count(0)
|
||||
|
||||
|
||||
# ── uncertainty ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_an_uncertain_outcome_offers_verification_and_no_retry(page, stack):
|
||||
stack.start(uploader=SILENT_UPLOADER)
|
||||
_upload(page, stack)
|
||||
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("succeeded")
|
||||
expect(page.get_by_test_id("batch-outcome-state")).to_have_text("requires_verification")
|
||||
expect(page.get_by_test_id("count-uncertain")).to_have_text("uncertain: 2")
|
||||
expect(page.get_by_test_id("uncertain")).to_be_visible()
|
||||
# The point of the story: verification is offered, a retry is not.
|
||||
expect(page.get_by_test_id("verify-batch")).to_be_visible()
|
||||
expect(page.get_by_test_id("start-batch")).to_have_count(0)
|
||||
|
||||
|
||||
def test_verification_asks_immich_and_resolves_the_uncertain_items(page, stack):
|
||||
stack.start(uploader=SILENT_UPLOADER)
|
||||
stack.immich.mode("present") # Immich holds exactly the bytes that were sent
|
||||
_upload(page, stack)
|
||||
|
||||
page.get_by_test_id("verify-batch").click()
|
||||
|
||||
expect(page.get_by_test_id("batch-outcome-state")).to_have_text("verified")
|
||||
expect(page.get_by_test_id("count-new")).to_have_text("new: 2")
|
||||
expect(page.get_by_test_id("count-uncertain")).to_have_text("uncertain: 0")
|
||||
expect(page.get_by_test_id("item-verification").first).to_have_text("present")
|
||||
expect(page.get_by_test_id("history-entry").first).to_have_attribute(
|
||||
"data-source", "immich_api"
|
||||
)
|
||||
expect(page.get_by_test_id("uncertain")).to_have_count(0)
|
||||
|
||||
|
||||
def test_an_unusable_answer_stays_uncertain_until_someone_records_evidence(page, stack):
|
||||
stack.start(uploader=SILENT_UPLOADER)
|
||||
stack.immich.mode("broken") # answers, but nothing this adapter will interpret
|
||||
_upload(page, stack)
|
||||
|
||||
page.get_by_test_id("verify-batch").click()
|
||||
|
||||
# No answer is never "no": the items stay uncertain rather than being called failed.
|
||||
expect(page.get_by_test_id("item-verification").first).to_have_text("inconclusive")
|
||||
expect(page.get_by_test_id("uncertain")).to_be_visible()
|
||||
expect(page.get_by_test_id("start-batch")).to_have_count(0)
|
||||
|
||||
row = page.get_by_test_id("item-row").first
|
||||
row.get_by_test_id("resolve-outcome").select_option("uploaded")
|
||||
row.get_by_test_id("resolve-evidence").fill("found it in Immich by checksum")
|
||||
row.get_by_test_id("resolve-actor").fill("dom")
|
||||
row.get_by_test_id("resolve-item").click()
|
||||
|
||||
expect(page.get_by_test_id("item-row").first.get_by_test_id("item-outcome")).to_have_text("new")
|
||||
manual = page.get_by_test_id("history-entry").last
|
||||
expect(manual).to_have_attribute("data-source", "operator")
|
||||
expect(manual).to_contain_text("found it in Immich by checksum")
|
||||
expect(manual).to_contain_text("dom")
|
||||
|
||||
|
||||
def test_bytes_changed_after_upload_are_flagged_and_block_another_run(page, stack):
|
||||
stack.start()
|
||||
_upload(page, stack)
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("succeeded")
|
||||
# The user edits a photo after it was uploaded; Immich still holds the old bytes.
|
||||
(stack.seeded.lib / "rome" / "a.jpg").write_bytes(b"edited after the upload")
|
||||
|
||||
page.get_by_test_id("verify-batch").click()
|
||||
|
||||
expect(page.get_by_test_id("stale-bytes")).to_be_visible()
|
||||
expect(page.get_by_test_id("item-changed")).to_have_count(1)
|
||||
expect(page.get_by_test_id("retry-blocked")).to_contain_text("changed_after_upload")
|
||||
expect(page.get_by_test_id("start-batch")).to_have_count(0)
|
||||
# The preflight agrees: those bytes are no longer approved for any new upload.
|
||||
expect(page.get_by_test_id("album-state")).to_have_text("blocked")
|
||||
expect(page.get_by_test_id("album-blocked")).to_have_text("1")
|
||||
|
||||
|
||||
# ── privacy ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_api_key_never_reaches_the_browser(page, stack):
|
||||
logs = []
|
||||
page.on("console", lambda message: logs.append(message.text))
|
||||
stack.start()
|
||||
_upload(page, stack)
|
||||
page.get_by_test_id("verify-batch").click()
|
||||
expect(page.get_by_test_id("item-verification").first).to_have_text("present")
|
||||
|
||||
storage = page.evaluate(
|
||||
"() => JSON.stringify([{...localStorage}, {...sessionStorage}, document.cookie])"
|
||||
)
|
||||
assert SENTINEL_KEY not in page.content()
|
||||
assert SENTINEL_KEY not in page.url
|
||||
assert SENTINEL_KEY not in storage
|
||||
assert SENTINEL_KEY not in "\n".join(logs)
|
||||
# The uploader was given the real key even though nothing on the page shows it.
|
||||
report = wait_until(lambda: sorted((stack.seeded.data / "uploads").glob("*.log")))[0]
|
||||
assert SENTINEL_KEY not in report.read_text()
|
||||
|
||||
|
||||
# ── recovery ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_an_interrupted_attempt_is_shown_as_uncertain_after_a_restart(page, stack):
|
||||
"""A worker that vanished mid-upload leaves a batch whose outcome nobody knows.
|
||||
Startup recovery marks it uncertain, and the view must not offer to retry it."""
|
||||
from photo_pipeline.models import UploadBatch
|
||||
|
||||
stack.start(worker=False)
|
||||
_open(page, stack)
|
||||
token = httpx.post(f"{stack.base}/api/v1/upload-preflight", json={}, timeout=TIMEOUT).json()[
|
||||
"token"
|
||||
]
|
||||
created = httpx.post(
|
||||
f"{stack.base}/api/v1/upload-batches", json={"token": token}, timeout=TIMEOUT
|
||||
).json()["batches"][0]
|
||||
with session_factory(stack.seeded) as sf: # what a killed worker leaves behind
|
||||
with sf() as session:
|
||||
session.get(UploadBatch, created["id"]).state = "running"
|
||||
session.commit()
|
||||
|
||||
stack.restart_server() # the same port, so recovery runs in a genuinely fresh process
|
||||
page.reload()
|
||||
|
||||
expect(page.get_by_test_id("detail-state")).to_have_text("unknown_requires_verification")
|
||||
expect(page.get_by_test_id("batch-error")).to_contain_text("interrupted")
|
||||
expect(page.get_by_test_id("uncertain")).to_be_visible()
|
||||
expect(page.get_by_test_id("start-batch")).to_have_count(0)
|
||||
expect(page.get_by_test_id("retry-blocked")).to_contain_text("requires_verification")
|
||||
592
tests/integration/test_archive_preflight.py
Normal file
592
tests/integration/test_archive_preflight.py
Normal file
@@ -0,0 +1,592 @@
|
||||
"""Archive destinations and preflight (US06-01).
|
||||
|
||||
Archive is the only stage that removes originals, so every case here asks the same
|
||||
question: would this preflight let an album leave active storage when it should
|
||||
not? The destinations are real directories on real filesystems — mounted, missing,
|
||||
swapped for another medium, read-only, or full — and preflight itself must stay
|
||||
non-destructive: the library snapshot is asserted unchanged.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.jobs.domain_handlers import ARCHIVE_LOCK, UPLOAD_LOCK
|
||||
from photo_pipeline.models import (
|
||||
Asset,
|
||||
RenameOperation,
|
||||
RenamePlan,
|
||||
UploadBatch,
|
||||
UploadItem,
|
||||
)
|
||||
from photo_pipeline.services.archives import MARKER_NAME, ArchiveError, ArchiveService
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.jobs import JobService
|
||||
|
||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
# ── environment ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _env(tmp_path, *, reserve=0):
|
||||
(tmp_path / "data").mkdir(exist_ok=True)
|
||||
lib = tmp_path / "lib"
|
||||
lib.mkdir(exist_ok=True)
|
||||
archive = tmp_path / "archive"
|
||||
archive.mkdir(exist_ok=True)
|
||||
config = Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
||||
"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": str(reserve),
|
||||
}
|
||||
)
|
||||
run_migrations(config.database_url)
|
||||
return config, create_session_factory(create_db_engine(config.database_url)), lib, archive
|
||||
|
||||
|
||||
def _album(
|
||||
sf,
|
||||
lib,
|
||||
album="rome",
|
||||
names=("a.jpg", "b.jpg"),
|
||||
*,
|
||||
uploaded=True,
|
||||
outcome="uploaded",
|
||||
outcome_state="verified",
|
||||
stale_bytes=False,
|
||||
):
|
||||
"""A real album folder whose assets carry their upload evidence."""
|
||||
folder = lib / album
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
ids = []
|
||||
with sf() as session:
|
||||
batch_id = str(uuid.uuid4())
|
||||
if uploaded:
|
||||
session.add(
|
||||
UploadBatch(
|
||||
id=batch_id,
|
||||
album=album,
|
||||
folder=str(folder),
|
||||
album_name=album,
|
||||
state="succeeded",
|
||||
preflight_token="v1:test",
|
||||
outcome_state=outcome_state,
|
||||
stale_bytes=stale_bytes,
|
||||
created_at=NOW,
|
||||
)
|
||||
)
|
||||
for name in names:
|
||||
path = folder / name
|
||||
path.write_bytes(name.encode() * 16)
|
||||
asset_id = str(uuid.uuid4())
|
||||
ids.append(asset_id)
|
||||
session.add(
|
||||
Asset(
|
||||
id=asset_id,
|
||||
original_path=str(path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
byte_size=path.stat().st_size,
|
||||
current_sha256=sha256_file(path),
|
||||
)
|
||||
)
|
||||
if uploaded:
|
||||
session.add(
|
||||
UploadItem(
|
||||
batch_id=batch_id,
|
||||
asset_id=asset_id,
|
||||
path=str(path),
|
||||
sha256=sha256_file(path),
|
||||
sha1="0" * 40,
|
||||
state="sent",
|
||||
outcome=outcome,
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return folder, ids
|
||||
|
||||
|
||||
def _service(sf, config):
|
||||
return ArchiveService(sf, config=config)
|
||||
|
||||
|
||||
def _location(sf, config, archive, name="external"):
|
||||
return _service(sf, config).register(name, str(archive))
|
||||
|
||||
|
||||
def _snapshot(lib):
|
||||
return {
|
||||
str(p.relative_to(lib)): (p.read_bytes() if p.is_file() else None)
|
||||
for p in sorted(lib.rglob("*"))
|
||||
}
|
||||
|
||||
|
||||
def _codes(report):
|
||||
return (
|
||||
{issue["code"] for issue in report["blockers"]}
|
||||
| {issue["code"] for album in report["albums"] for issue in album["blockers"]}
|
||||
| {
|
||||
issue["code"]
|
||||
for album in report["albums"]
|
||||
for asset in album["assets"]
|
||||
for issue in asset["blockers"]
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ── locations ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_registering_a_location_stamps_the_medium_with_its_identity(tmp_path):
|
||||
config, sf, _, archive = _env(tmp_path)
|
||||
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
marker = json.loads((archive / MARKER_NAME).read_text())
|
||||
assert marker["media_id"] == location["media_id"]
|
||||
assert location["state"] == "online" and location["writable"] is True
|
||||
assert location["root"] == str(archive.resolve())
|
||||
listed = _service(sf, config).locations()
|
||||
assert [(row["id"], row["media_id"], row["state"]) for row in listed] == [
|
||||
(location["id"], location["media_id"], "online")
|
||||
]
|
||||
|
||||
|
||||
def test_a_second_location_cannot_claim_the_same_medium(tmp_path):
|
||||
config, sf, _, archive = _env(tmp_path)
|
||||
_location(sf, config, archive)
|
||||
|
||||
with pytest.raises(ArchiveError) as error:
|
||||
_location(sf, config, archive, name="second")
|
||||
|
||||
assert error.value.code == "already_registered"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("inside", ["", "sub"])
|
||||
def test_a_destination_inside_the_library_is_refused(tmp_path, inside):
|
||||
"""The library may never archive into itself: the 'reclaimed' bytes would still
|
||||
be in the active tree, and a later scan would rediscover them."""
|
||||
config, sf, lib, _ = _env(tmp_path)
|
||||
root = lib / inside if inside else lib
|
||||
root.mkdir(exist_ok=True)
|
||||
|
||||
with pytest.raises(ArchiveError) as error:
|
||||
_service(sf, config).register("bad", str(root))
|
||||
|
||||
assert error.value.code == "unsafe_destination"
|
||||
|
||||
|
||||
def test_an_ignored_destination_is_refused(tmp_path):
|
||||
config, sf, _, _ = _env(tmp_path)
|
||||
root = tmp_path / "_IGNORE" / "archive"
|
||||
root.mkdir(parents=True)
|
||||
|
||||
with pytest.raises(ArchiveError) as error:
|
||||
_service(sf, config).register("ignored", str(root))
|
||||
|
||||
assert error.value.code == "unsafe_destination"
|
||||
|
||||
|
||||
def test_listing_reports_an_unmounted_medium_as_offline(tmp_path):
|
||||
config, sf, _, archive = _env(tmp_path)
|
||||
_location(sf, config, archive)
|
||||
(archive / MARKER_NAME).unlink()
|
||||
|
||||
assert [row["state"] for row in _service(sf, config).locations()] == ["offline"]
|
||||
|
||||
|
||||
# ── happy path ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_ready_preflight_previews_scope_method_and_reclaimable_bytes(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, ids = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
before = _snapshot(lib)
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert report["state"] == "ready" and report["blockers"] == []
|
||||
album = report["albums"][0]
|
||||
assert album["album"] == "rome" and album["folder"] == str(folder)
|
||||
assert album["destination"] == str(archive.resolve() / "rome")
|
||||
assert album["transfer_method"] in ("move", "copy_verify_remove")
|
||||
assert album["reclaimable_bytes"] == sum(p.stat().st_size for p in folder.iterdir())
|
||||
assert sorted(a["asset_id"] for a in album["assets"]) == sorted(ids)
|
||||
assert report["totals"]["bytes"] == album["reclaimable_bytes"]
|
||||
assert report["capacity"]["sufficient"] is True
|
||||
# Both must be proven by writing, not assumed.
|
||||
assert report["backup"]["ok"] is True and report["backup"]["bytes"] > 0
|
||||
assert report["manifest"]["ok"] is True
|
||||
assert report["token"].startswith("v1:")
|
||||
assert _snapshot(lib) == before, "preflight must not touch the library"
|
||||
assert not list(archive.glob("*probe*")), "probe files must be cleaned up"
|
||||
|
||||
|
||||
def test_same_filesystem_destination_is_previewed_as_a_move(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
# tmp_path is one filesystem, so this is the same-filesystem case by construction.
|
||||
assert report["albums"][0]["transfer_method"] == "move"
|
||||
|
||||
|
||||
def test_scoping_to_one_album_excludes_the_others(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib, "rome")
|
||||
_album(sf, lib, "paris", names=("c.jpg",))
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
report = _service(sf, config).preflight(location["id"], ["paris"])
|
||||
|
||||
assert [album["album"] for album in report["albums"]] == ["paris"]
|
||||
assert report["totals"]["assets"] == 1
|
||||
|
||||
|
||||
def test_unknown_album_and_unknown_location_are_refused(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service = _service(sf, config)
|
||||
|
||||
with pytest.raises(ArchiveError) as unknown_album:
|
||||
service.preflight(location["id"], ["atlantis"])
|
||||
with pytest.raises(ArchiveError) as unknown_location:
|
||||
service.preflight("nope")
|
||||
|
||||
assert unknown_album.value.code == "unknown_album"
|
||||
assert unknown_location.value.code == "unknown_location"
|
||||
|
||||
|
||||
def test_empty_scope_is_a_blocker(tmp_path):
|
||||
config, sf, _, archive = _env(tmp_path)
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert report["state"] == "blocked" and "empty_scope" in _codes(report)
|
||||
|
||||
|
||||
# ── destination ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_offline_medium_blocks(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
(archive / MARKER_NAME).unlink() # the disk went away
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert report["state"] == "blocked" and "location_offline" in _codes(report)
|
||||
assert report["location"]["state"] == "offline"
|
||||
|
||||
|
||||
def test_a_different_medium_at_the_same_mountpoint_blocks(tmp_path):
|
||||
"""The mountpoint is right, the disk is not — never write the archive here."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
(archive / MARKER_NAME).write_text(json.dumps({"media_id": "some-other-disk"}))
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert "wrong_volume" in _codes(report)
|
||||
assert report["location"]["state"] == "wrong_volume"
|
||||
|
||||
|
||||
def test_read_only_destination_blocks_and_cannot_write_the_manifest(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
mode = archive.stat().st_mode
|
||||
archive.chmod(mode & ~stat.S_IWUSR & ~stat.S_IWGRP & ~stat.S_IWOTH)
|
||||
try:
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
finally:
|
||||
archive.chmod(mode)
|
||||
|
||||
assert {"destination_not_writable", "manifest_unwritable"} <= _codes(report)
|
||||
assert report["manifest"]["ok"] is False
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.geteuid() == 0, reason="root ignores directory permissions")
|
||||
def test_read_only_destination_is_detected_by_a_real_write(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
mode = archive.stat().st_mode
|
||||
archive.chmod(stat.S_IRUSR | stat.S_IXUSR)
|
||||
try:
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
finally:
|
||||
archive.chmod(mode)
|
||||
|
||||
assert report["location"]["writable"] is False
|
||||
|
||||
|
||||
def test_insufficient_capacity_blocks(tmp_path):
|
||||
"""The reserve is what stops an archive from filling its own destination."""
|
||||
config, sf, lib, archive = _env(tmp_path, reserve=10**15)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert report["state"] == "blocked" and "insufficient_capacity" in _codes(report)
|
||||
assert report["capacity"]["sufficient"] is False
|
||||
assert report["capacity"]["reserve_bytes"] == 10**15
|
||||
|
||||
|
||||
def test_an_occupied_destination_blocks_that_album(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
(archive / "rome").mkdir()
|
||||
(archive / "rome" / "a.jpg").write_bytes(b"something already here")
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert "destination_collision" in _codes(report)
|
||||
assert report["albums"][0]["state"] == "blocked"
|
||||
|
||||
|
||||
def test_a_destination_moved_into_the_library_blocks_even_though_it_registered(tmp_path):
|
||||
"""Registration validated the root once; preflight validates it again, because a
|
||||
mountpoint can be moved after the fact."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
with sf() as session:
|
||||
from photo_pipeline.models import ArchiveLocation
|
||||
|
||||
session.get(ArchiveLocation, location["id"]).root = str(lib / "inside")
|
||||
session.commit()
|
||||
(lib / "inside").mkdir()
|
||||
(lib / "inside" / MARKER_NAME).write_text(json.dumps({"media_id": location["media_id"]}))
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert "unsafe_destination" in _codes(report)
|
||||
|
||||
|
||||
# ── source readiness ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"kwargs,code",
|
||||
[
|
||||
({"uploaded": False}, "upload_unverified"),
|
||||
({"outcome_state": "requires_verification"}, "upload_unverified"),
|
||||
({"outcome": "failed"}, "upload_unverified"),
|
||||
({"outcome": "skipped"}, "upload_unverified"),
|
||||
({"stale_bytes": True}, "upload_unverified"),
|
||||
],
|
||||
)
|
||||
def test_an_unverified_upload_blocks_the_album(tmp_path, kwargs, code):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib, **kwargs)
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert report["state"] == "blocked" and code in _codes(report)
|
||||
assert report["albums"][0]["blocked_count"] == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("outcome", ["upgraded", "duplicate"])
|
||||
def test_upgraded_and_duplicate_uploads_are_evidence_enough(tmp_path, outcome):
|
||||
"""Immich already holds these exact bytes; that is what archiving requires."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib, outcome=outcome)
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
assert _service(sf, config).preflight(location["id"])["state"] == "ready"
|
||||
|
||||
|
||||
def test_bytes_changed_since_upload_block_the_album(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
(folder / "a.jpg").write_bytes(b"edited after the upload")
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert {"bytes_changed", "partial_scope"} <= _codes(report)
|
||||
assert report["albums"][0]["blocked_count"] == 1
|
||||
|
||||
|
||||
def test_a_missing_source_file_blocks_the_album(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
(folder / "a.jpg").unlink()
|
||||
|
||||
assert "file_missing" in _codes(_service(sf, config).preflight(location["id"]))
|
||||
|
||||
|
||||
# ── leases ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("lock", [UPLOAD_LOCK, ARCHIVE_LOCK])
|
||||
def test_a_held_lease_blocks_archiving(tmp_path, lock):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
JobService(sf).enqueue("upload_batch", lock=lock, items=["x"])
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert report["state"] == "blocked" and "lock_conflict" in _codes(report)
|
||||
|
||||
|
||||
def test_a_half_applied_rename_blocks_archiving(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
with sf() as session:
|
||||
plan_id = str(uuid.uuid4())
|
||||
session.add(RenamePlan(id=plan_id, state="applying", operation_count=1))
|
||||
session.flush()
|
||||
session.add(
|
||||
RenameOperation(
|
||||
id=str(uuid.uuid4()),
|
||||
plan_id=plan_id,
|
||||
sequence=0,
|
||||
operation="move_folder",
|
||||
source_path=str(folder),
|
||||
destination_path=str(lib / "2019 Rome"),
|
||||
journal_state="moving",
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
report = _service(sf, config).preflight(location["id"])
|
||||
|
||||
assert report["state"] == "blocked" and "rename_pending" in _codes(report)
|
||||
|
||||
|
||||
# ── token ────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_token_is_stable_while_nothing_relevant_changes(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service = _service(sf, config)
|
||||
|
||||
first = service.preflight(location["id"])["token"]
|
||||
|
||||
assert service.preflight(location["id"])["token"] == first
|
||||
assert service.verify_token(first, location["id"]) is True
|
||||
|
||||
|
||||
def test_an_edited_source_makes_the_token_stale(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service = _service(sf, config)
|
||||
token = service.preflight(location["id"])["token"]
|
||||
|
||||
(folder / "b.jpg").write_bytes(b"edited outside the app")
|
||||
|
||||
assert service.verify_token(token, location["id"]) is False
|
||||
|
||||
|
||||
def test_a_changed_destination_makes_the_token_stale(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service = _service(sf, config)
|
||||
token = service.preflight(location["id"])["token"]
|
||||
|
||||
(archive / "rome").mkdir()
|
||||
(archive / "rome" / "a.jpg").write_bytes(b"appeared after approval")
|
||||
|
||||
assert service.verify_token(token, location["id"]) is False
|
||||
|
||||
|
||||
def test_a_token_from_another_scope_or_medium_is_rejected(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib, "rome")
|
||||
_album(sf, lib, "paris", names=("c.jpg",))
|
||||
other = tmp_path / "archive2"
|
||||
other.mkdir()
|
||||
location = _location(sf, config, archive)
|
||||
second = _location(sf, config, other, name="second")
|
||||
service = _service(sf, config)
|
||||
|
||||
rome = service.preflight(location["id"], ["rome"])["token"]
|
||||
|
||||
assert service.verify_token(rome, location["id"], ["paris"]) is False
|
||||
assert service.verify_token(rome, second["id"], ["rome"]) is False
|
||||
assert service.verify_token("v1:not-a-real-token", location["id"]) is False
|
||||
assert service.verify_token("", location["id"]) is False
|
||||
|
||||
|
||||
def test_the_token_survives_free_space_and_timestamp_drift(tmp_path):
|
||||
"""Free space changes constantly on a live disk; a token that expired on every
|
||||
byte written elsewhere would train users to ignore it."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service = _service(sf, config)
|
||||
token = service.preflight(location["id"])["token"]
|
||||
|
||||
(tmp_path / "unrelated.bin").write_bytes(b"0" * 100_000)
|
||||
with sf() as session:
|
||||
from photo_pipeline.models import ArchiveLocation
|
||||
|
||||
session.get(ArchiveLocation, location["id"]).last_seen_at = NOW - timedelta(days=5)
|
||||
session.commit()
|
||||
|
||||
assert service.verify_token(token, location["id"]) is True
|
||||
|
||||
|
||||
# ── API surface ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_api_registers_a_location_and_returns_a_preflight_report(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
created = client.post(
|
||||
"/api/v1/archive-locations", json={"name": "external", "root": str(archive)}
|
||||
)
|
||||
listed = client.get("/api/v1/archive-locations")
|
||||
report = client.post(
|
||||
"/api/v1/archive-preflight", json={"location_id": created.json()["id"]}
|
||||
)
|
||||
|
||||
assert created.status_code == 201
|
||||
assert [row["name"] for row in listed.json()["locations"]] == ["external"]
|
||||
assert report.status_code == 200
|
||||
assert report.json()["state"] == "ready" and report.json()["token"].startswith("v1:")
|
||||
|
||||
|
||||
def test_api_rejects_an_unknown_location_and_an_unsafe_root(tmp_path):
|
||||
config, sf, lib, _ = _env(tmp_path)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
unknown = client.post("/api/v1/archive-preflight", json={"location_id": "nope"})
|
||||
unsafe = client.post("/api/v1/archive-locations", json={"name": "bad", "root": str(lib)})
|
||||
|
||||
assert unknown.status_code == 404 and unknown.json()["error"]["code"] == "unknown_location"
|
||||
assert unsafe.status_code == 422 and unsafe.json()["error"]["code"] == "unsafe_destination"
|
||||
313
tests/integration/test_archive_recovery.py
Normal file
313
tests/integration/test_archive_recovery.py
Normal file
@@ -0,0 +1,313 @@
|
||||
"""Recovering interrupted archive transfers (US06-02).
|
||||
|
||||
The crash tests are real: a child process applies an archive plan and is killed by
|
||||
the ``PHOTO_PIPELINE_FAULT_AFTER`` barrier at each persisted transition in turn —
|
||||
including the moment immediately after an active source has been unlinked. The
|
||||
parent then reopens the database and asserts the one invariant archiving exists to
|
||||
uphold: **no verified file is ever lost, and no source is removed without a durable,
|
||||
byte-identical archive copy.**
|
||||
|
||||
Both transfer paths are exercised: the same-filesystem atomic move and (with the
|
||||
child forcing the device comparison) the cross-filesystem copy/verify/publish.
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.models import Asset, UploadBatch, UploadItem
|
||||
from photo_pipeline.services.archive_journal import FORWARD, MANUAL, RESUMABLE, ArchiveState
|
||||
from photo_pipeline.services.archive_transfer import (
|
||||
MANIFEST_NAME,
|
||||
ArchiveTransferService,
|
||||
read_manifest,
|
||||
)
|
||||
from photo_pipeline.services.archives import ArchiveService
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
|
||||
pytestmark = pytest.mark.phase_f
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
# Barriers, in the order the transfer persists them. ``source_removed`` is the one
|
||||
# that matters most: the original is already gone at that point.
|
||||
BARRIERS = [
|
||||
ArchiveState.TRANSFERRING,
|
||||
ArchiveState.VERIFIED,
|
||||
ArchiveState.REMOVING,
|
||||
"source_removed",
|
||||
ArchiveState.COMPLETE,
|
||||
]
|
||||
|
||||
# A child that applies the plan and dies at the configured barrier. ``force_copy``
|
||||
# makes it take the cross-filesystem path without a second real volume.
|
||||
APPLY_SCRIPT = """
|
||||
import sys
|
||||
sys.path.insert(0, {repo!r})
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory
|
||||
from photo_pipeline.services import archive_transfer
|
||||
|
||||
db_url, data_dir, lib, plan_id, force_copy = sys.argv[1:6]
|
||||
if force_copy == "1":
|
||||
archive_transfer._same_filesystem = lambda *args: False
|
||||
config = Config.from_env(
|
||||
{{"PHOTO_PIPELINE_DATA_DIR": data_dir, "PHOTO_PIPELINE_LIBRARY_ROOTS": lib}}
|
||||
)
|
||||
sf = create_session_factory(create_db_engine(db_url))
|
||||
archive_transfer.ArchiveTransferService(sf, config=config).apply(plan_id)
|
||||
"""
|
||||
|
||||
|
||||
# ── environment ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _env(tmp_path):
|
||||
(tmp_path / "data").mkdir(exist_ok=True)
|
||||
lib = tmp_path / "lib"
|
||||
lib.mkdir(exist_ok=True)
|
||||
archive = tmp_path / "archive"
|
||||
archive.mkdir(exist_ok=True)
|
||||
config = Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
||||
"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0",
|
||||
}
|
||||
)
|
||||
run_migrations(config.database_url)
|
||||
return config, create_session_factory(create_db_engine(config.database_url)), lib, archive
|
||||
|
||||
|
||||
def _album(sf, lib, album="rome", names=("a.jpg", "b.jpg")):
|
||||
folder = lib / album
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
with sf() as session:
|
||||
batch_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
UploadBatch(
|
||||
id=batch_id,
|
||||
album=album,
|
||||
folder=str(folder),
|
||||
album_name=album,
|
||||
state="succeeded",
|
||||
preflight_token="v1:test",
|
||||
outcome_state="verified",
|
||||
created_at=NOW,
|
||||
)
|
||||
)
|
||||
for name in names:
|
||||
path = folder / name
|
||||
path.write_bytes(f"{album}/{name} content".encode() * 8)
|
||||
asset_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
Asset(
|
||||
id=asset_id,
|
||||
original_path=str(path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
byte_size=path.stat().st_size,
|
||||
current_sha256=sha256_file(path),
|
||||
)
|
||||
)
|
||||
session.add(
|
||||
UploadItem(
|
||||
batch_id=batch_id,
|
||||
asset_id=asset_id,
|
||||
path=str(path),
|
||||
sha256=sha256_file(path),
|
||||
sha1="0" * 40,
|
||||
state="sent",
|
||||
outcome="uploaded",
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return folder
|
||||
|
||||
|
||||
def _plan(sf, config, archive, albums=None):
|
||||
location = ArchiveService(sf, config=config).register("external", str(archive))
|
||||
token = ArchiveService(sf, config=config).preflight(location["id"], albums)["token"]
|
||||
service = ArchiveTransferService(sf, config=config)
|
||||
return service, service.create(location["id"], albums, token=token)
|
||||
|
||||
|
||||
def _crash_during_apply(config, tmp_path, lib, plan, barrier, *, force_copy=False):
|
||||
script = tmp_path / f"apply_{barrier}.py"
|
||||
script.write_text(APPLY_SCRIPT.format(repo=str(REPO)))
|
||||
env = dict(os.environ)
|
||||
env["PHOTO_PIPELINE_FAULT_AFTER"] = barrier
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(script),
|
||||
config.database_url,
|
||||
str(tmp_path / "data"),
|
||||
str(lib),
|
||||
plan["id"],
|
||||
"1" if force_copy else "0",
|
||||
],
|
||||
env=env,
|
||||
capture_output=True,
|
||||
)
|
||||
assert result.returncode in (9, -9), (
|
||||
f"child should have been killed at {barrier}, got {result.returncode}: "
|
||||
f"{result.stderr.decode(errors='replace')[-400:]}"
|
||||
)
|
||||
|
||||
|
||||
def _contents(*roots):
|
||||
return sorted(
|
||||
path.read_bytes()
|
||||
for root in roots
|
||||
for path in root.rglob("*")
|
||||
if path.is_file() and path.name != MANIFEST_NAME and not path.name.startswith(".")
|
||||
)
|
||||
|
||||
|
||||
def _reopen(config):
|
||||
return create_session_factory(create_db_engine(config.database_url))
|
||||
|
||||
|
||||
# ── fault injection at every persisted transition ────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("barrier", BARRIERS)
|
||||
@pytest.mark.parametrize("force_copy", [False, True], ids=["move", "copy"])
|
||||
def test_a_crash_at_every_transition_loses_nothing_and_recovers(tmp_path, barrier, force_copy):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
service, plan = _plan(sf, config, archive)
|
||||
before = _contents(lib)
|
||||
|
||||
_crash_during_apply(config, tmp_path, lib, plan, barrier, force_copy=force_copy)
|
||||
|
||||
# Every file still exists somewhere: the crash may not have cost a single byte.
|
||||
assert set(before) <= set(_contents(lib, archive)), f"content lost at {barrier}"
|
||||
|
||||
reopened = _reopen(config)
|
||||
recovery = ArchiveTransferService(reopened, config=config)
|
||||
# Whatever the crash interrupted is recoverable from evidence — never ambiguous.
|
||||
# (A crash on the last barrier can land on a terminal item and leave none.)
|
||||
verdicts = recovery.journal.classify_all()
|
||||
assert all(v["classification"] in {RESUMABLE, FORWARD} for v in verdicts), verdicts
|
||||
|
||||
recovery.recover()
|
||||
recovery.apply(plan["id"]) # finish whatever the crash never started
|
||||
|
||||
assert _contents(archive) == before
|
||||
assert _contents(lib) == []
|
||||
assert recovery.journal.blocks_mutation() is False
|
||||
assert recovery.journal.plan_state(plan["id"]) == "complete"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("barrier", [ArchiveState.VERIFIED, "source_removed"])
|
||||
def test_recovery_is_idempotent_across_repeated_restarts(tmp_path, barrier):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
service, plan = _plan(sf, config, archive)
|
||||
_crash_during_apply(config, tmp_path, lib, plan, barrier)
|
||||
|
||||
reopened = _reopen(config)
|
||||
recovery = ArchiveTransferService(reopened, config=config)
|
||||
recovery.recover()
|
||||
settled = (_contents(lib), _contents(archive), read_manifest(archive / "rome"))
|
||||
|
||||
for _ in range(2):
|
||||
recovery.recover()
|
||||
assert (_contents(lib), _contents(archive), read_manifest(archive / "rome")) == settled
|
||||
|
||||
|
||||
def test_a_crash_before_anything_was_published_leaves_the_source_intact(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder = _album(sf, lib, names=("a.jpg",))
|
||||
service, plan = _plan(sf, config, archive)
|
||||
|
||||
_crash_during_apply(config, tmp_path, lib, plan, ArchiveState.TRANSFERRING)
|
||||
|
||||
recovery = ArchiveTransferService(_reopen(config), config=config)
|
||||
verdict = recovery.journal.classify_all()[0]
|
||||
assert verdict["classification"] == RESUMABLE
|
||||
assert (folder / "a.jpg").exists()
|
||||
assert not (archive / "rome" / "a.jpg").exists()
|
||||
|
||||
assert recovery.recover() == {"resumed": 1, "completed": 0, "manual": 0}
|
||||
assert recovery.journal.operations(plan["id"])[0]["journal_state"] == ArchiveState.PLANNED
|
||||
|
||||
|
||||
def test_a_crash_after_the_source_was_removed_finishes_the_bookkeeping(tmp_path):
|
||||
"""The dangerous window: the original is gone and the database still points at
|
||||
it. Recovery must complete the record, never re-transfer or report loss."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder = _album(sf, lib, names=("a.jpg",))
|
||||
service, plan = _plan(sf, config, archive)
|
||||
expected = service.journal.operations(plan["id"])[0]["expected_sha256"]
|
||||
|
||||
_crash_during_apply(config, tmp_path, lib, plan, "source_removed")
|
||||
|
||||
reopened = _reopen(config)
|
||||
recovery = ArchiveTransferService(reopened, config=config)
|
||||
verdict = recovery.journal.classify_all()[0]
|
||||
assert verdict["classification"] == FORWARD
|
||||
assert not (folder / "a.jpg").exists()
|
||||
assert sha256_file(archive / "rome" / "a.jpg") == expected
|
||||
with reopened() as session:
|
||||
stranded = session.scalar(select(Asset))
|
||||
assert stranded.current_path is not None, "the crash happened before the DB update"
|
||||
|
||||
assert recovery.recover() == {"resumed": 0, "completed": 1, "manual": 0}
|
||||
|
||||
with reopened() as session:
|
||||
asset = session.scalar(select(Asset))
|
||||
assert asset.current_path is None
|
||||
assert asset.availability_state == "archived_online"
|
||||
assert asset.archive_path == "rome/a.jpg"
|
||||
assert recovery.journal.operations(plan["id"])[0]["journal_state"] == ArchiveState.COMPLETE
|
||||
assert len(read_manifest(archive / "rome")) == 1, "the manifest is not duplicated"
|
||||
|
||||
|
||||
def test_an_archive_copy_that_changed_after_the_crash_blocks_for_a_human(tmp_path):
|
||||
"""Wrong bytes at the destination can never justify deleting the original, and
|
||||
are never overwritten either."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder = _album(sf, lib, names=("a.jpg",))
|
||||
service, plan = _plan(sf, config, archive)
|
||||
_crash_during_apply(config, tmp_path, lib, plan, ArchiveState.VERIFIED, force_copy=True)
|
||||
(archive / "rome" / "a.jpg").write_bytes(b"tampered with while the app was down")
|
||||
|
||||
recovery = ArchiveTransferService(_reopen(config), config=config)
|
||||
verdict = recovery.journal.classify_all()[0]
|
||||
|
||||
assert verdict["classification"] == MANUAL
|
||||
assert recovery.recover() == {"resumed": 0, "completed": 0, "manual": 1}
|
||||
assert (folder / "a.jpg").exists(), "the source is kept while the archive is unproven"
|
||||
assert (archive / "rome" / "a.jpg").read_bytes() == b"tampered with while the app was down"
|
||||
# And the unresolved item keeps blocking further archiving.
|
||||
assert recovery.journal.blocks_mutation() is True
|
||||
report = ArchiveService(_reopen(config), config=config).preflight(plan["location_id"])
|
||||
assert "archive_pending" in {issue["code"] for issue in report["blockers"]}
|
||||
|
||||
|
||||
def test_a_verified_archive_whose_copy_vanished_is_never_reported_as_archived(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder = _album(sf, lib, names=("a.jpg",))
|
||||
service, plan = _plan(sf, config, archive)
|
||||
_crash_during_apply(config, tmp_path, lib, plan, ArchiveState.REMOVING, force_copy=True)
|
||||
(archive / "rome" / "a.jpg").unlink() # the medium lost it
|
||||
|
||||
recovery = ArchiveTransferService(_reopen(config), config=config)
|
||||
|
||||
assert recovery.journal.classify_all()[0]["classification"] == MANUAL
|
||||
assert recovery.recover()["manual"] == 1
|
||||
assert (folder / "a.jpg").exists()
|
||||
assert recovery.journal.operations(plan["id"])[0]["journal_state"] != ArchiveState.COMPLETE
|
||||
480
tests/integration/test_archive_transfer.py
Normal file
480
tests/integration/test_archive_transfer.py
Normal file
@@ -0,0 +1,480 @@
|
||||
"""Transferring, verifying, and removing active sources (US06-02).
|
||||
|
||||
Every case here asks the same question the service exists to answer: could an
|
||||
original leave active storage without a durable, byte-identical archive copy? The
|
||||
files are real, the hashes are real, and each failure path asserts that the source
|
||||
is still exactly where it was.
|
||||
|
||||
Cross-filesystem behaviour is forced by patching the device comparison rather than
|
||||
by requiring a second real filesystem in CI — the copy/verify/publish code that runs
|
||||
is the production one.
|
||||
"""
|
||||
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.models import Asset, AssetPath, UploadBatch, UploadItem
|
||||
from photo_pipeline.services import archive_transfer as transfer_module
|
||||
from photo_pipeline.services.archive_journal import ArchiveState
|
||||
from photo_pipeline.services.archive_transfer import (
|
||||
MANIFEST_NAME,
|
||||
ArchiveTransferService,
|
||||
read_manifest,
|
||||
)
|
||||
from photo_pipeline.services.archives import ArchiveError, ArchiveService
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
|
||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
# ── environment ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _env(tmp_path):
|
||||
(tmp_path / "data").mkdir(exist_ok=True)
|
||||
lib = tmp_path / "lib"
|
||||
lib.mkdir(exist_ok=True)
|
||||
archive = tmp_path / "archive"
|
||||
archive.mkdir(exist_ok=True)
|
||||
config = Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
||||
"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0",
|
||||
}
|
||||
)
|
||||
run_migrations(config.database_url)
|
||||
return config, create_session_factory(create_db_engine(config.database_url)), lib, archive
|
||||
|
||||
|
||||
def _album(sf, lib, album="rome", names=("a.jpg", "b.jpg")):
|
||||
"""A real album whose assets carry the verified upload evidence archiving needs."""
|
||||
folder = lib / album
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
ids = []
|
||||
with sf() as session:
|
||||
batch_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
UploadBatch(
|
||||
id=batch_id,
|
||||
album=album,
|
||||
folder=str(folder),
|
||||
album_name=album,
|
||||
state="succeeded",
|
||||
preflight_token="v1:test",
|
||||
outcome_state="verified",
|
||||
created_at=NOW,
|
||||
)
|
||||
)
|
||||
for name in names:
|
||||
path = folder / name
|
||||
path.write_bytes(f"{album}/{name} bytes".encode() * 8)
|
||||
asset_id = str(uuid.uuid4())
|
||||
ids.append(asset_id)
|
||||
session.add(
|
||||
Asset(
|
||||
id=asset_id,
|
||||
original_path=str(path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
byte_size=path.stat().st_size,
|
||||
current_sha256=sha256_file(path),
|
||||
)
|
||||
)
|
||||
session.add(AssetPath(asset_id=asset_id, path=str(path), valid_from=NOW))
|
||||
session.add(
|
||||
UploadItem(
|
||||
batch_id=batch_id,
|
||||
asset_id=asset_id,
|
||||
path=str(path),
|
||||
sha256=sha256_file(path),
|
||||
sha1="0" * 40,
|
||||
state="sent",
|
||||
outcome="uploaded",
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return folder, ids
|
||||
|
||||
|
||||
def _location(sf, config, archive, name="external"):
|
||||
return ArchiveService(sf, config=config).register(name, str(archive))
|
||||
|
||||
|
||||
def _plan(sf, config, location_id, albums=None):
|
||||
service = ArchiveTransferService(sf, config=config)
|
||||
token = ArchiveService(sf, config=config).preflight(location_id, albums)["token"]
|
||||
return service, service.create(location_id, albums, token=token)
|
||||
|
||||
|
||||
def _contents(*roots):
|
||||
"""Every file's bytes under the given roots, ignoring our own bookkeeping."""
|
||||
return sorted(
|
||||
path.read_bytes()
|
||||
for root in roots
|
||||
for path in root.rglob("*")
|
||||
if path.is_file() and path.name != MANIFEST_NAME and not path.name.startswith(".")
|
||||
)
|
||||
|
||||
|
||||
def _states(service, plan_id):
|
||||
return [row["journal_state"] for row in service.journal.operations(plan_id)]
|
||||
|
||||
|
||||
def _asset(sf, asset_id):
|
||||
with sf() as session:
|
||||
return session.get(Asset, asset_id)
|
||||
|
||||
|
||||
# ── planning ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_plan_records_every_file_with_its_expected_hash_and_moves_nothing(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, ids = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
before = _contents(lib)
|
||||
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
|
||||
assert plan["state"] == "planned" and plan["asset_count"] == 2
|
||||
assert sorted(op["asset_id"] for op in plan["operations"]) == sorted(ids)
|
||||
for operation in plan["operations"]:
|
||||
source = folder / operation["archive_path"].split("/")[-1]
|
||||
assert operation["expected_sha256"] == sha256_file(source)
|
||||
assert operation["archive_path"].startswith("rome/")
|
||||
assert operation["journal_state"] == ArchiveState.PLANNED
|
||||
assert _contents(lib) == before
|
||||
assert _contents(archive) == []
|
||||
|
||||
|
||||
def test_a_stale_token_cannot_create_a_plan(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
token = ArchiveService(sf, config=config).preflight(location["id"])["token"]
|
||||
|
||||
(folder / "a.jpg").write_bytes(b"edited after approval")
|
||||
|
||||
with pytest.raises(ArchiveError) as error:
|
||||
ArchiveTransferService(sf, config=config).create(location["id"], token=token)
|
||||
assert error.value.code == "stale_token"
|
||||
|
||||
|
||||
def test_a_blocked_scope_cannot_create_a_plan(tmp_path):
|
||||
"""No verified upload means Immich may not hold these bytes; archiving would
|
||||
remove the only copy."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder = lib / "rome"
|
||||
folder.mkdir()
|
||||
path = folder / "a.jpg"
|
||||
path.write_bytes(b"never uploaded")
|
||||
with sf() as session:
|
||||
session.add(
|
||||
Asset(
|
||||
id=str(uuid.uuid4()),
|
||||
original_path=str(path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
byte_size=path.stat().st_size,
|
||||
current_sha256=sha256_file(path),
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
location = _location(sf, config, archive)
|
||||
token = ArchiveService(sf, config=config).preflight(location["id"])["token"]
|
||||
|
||||
with pytest.raises(ArchiveError) as error:
|
||||
ArchiveTransferService(sf, config=config).create(location["id"], token=token)
|
||||
|
||||
assert error.value.code == "blocked"
|
||||
assert path.exists()
|
||||
|
||||
|
||||
# ── the transfer ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("same_filesystem", [True, False])
|
||||
def test_archiving_verifies_the_copy_before_the_source_is_removed(
|
||||
tmp_path, monkeypatch, same_filesystem
|
||||
):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, ids = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
expected = {op["asset_id"]: op["expected_sha256"] for op in plan["operations"]}
|
||||
before = _contents(lib)
|
||||
if not same_filesystem:
|
||||
# Force the copy-verify-publish path without needing a second real volume.
|
||||
monkeypatch.setattr(transfer_module, "_same_filesystem", lambda *_: False)
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
|
||||
assert result == {
|
||||
"plan_id": plan["id"],
|
||||
"archived": 2,
|
||||
"failed": 0,
|
||||
"skipped": 0,
|
||||
"state": "complete",
|
||||
}
|
||||
# The bytes moved: nothing is left in the library, everything is in the archive.
|
||||
assert _contents(archive) == before
|
||||
assert _contents(lib) == []
|
||||
assert not folder.exists(), "an emptied album folder is not left behind"
|
||||
for asset_id, digest in expected.items():
|
||||
archived = archive / "rome" / _asset(sf, asset_id).archive_path.split("/")[-1]
|
||||
assert sha256_file(archived) == digest
|
||||
assert _states(service, plan["id"]) == [ArchiveState.COMPLETE] * 2
|
||||
# No transfer temporaries survive either path.
|
||||
assert not list((archive / "rome").glob(".archive-*"))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("same_filesystem", [True, False])
|
||||
def test_the_manifest_on_the_medium_matches_the_archived_bytes(
|
||||
tmp_path, monkeypatch, same_filesystem
|
||||
):
|
||||
"""The manifest is the medium's own record: it must be usable to verify the
|
||||
archive with no database at all."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
if not same_filesystem:
|
||||
monkeypatch.setattr(transfer_module, "_same_filesystem", lambda *_: False)
|
||||
|
||||
service.apply(plan["id"])
|
||||
|
||||
entries = read_manifest(archive / "rome")
|
||||
assert len(entries) == 2
|
||||
for entry in entries:
|
||||
archived = archive / entry["archive_path"]
|
||||
assert sha256_file(archived) == entry["sha256"]
|
||||
assert entry["byte_size"] == archived.stat().st_size
|
||||
assert entry["media_id"] == location["media_id"]
|
||||
assert entry["plan_id"] == plan["id"] and entry["album"] == "rome"
|
||||
|
||||
|
||||
def test_archived_assets_keep_their_identity_and_gain_their_new_location(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, ids = _album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
|
||||
service.apply(plan["id"])
|
||||
|
||||
with sf() as session:
|
||||
for asset_id in ids:
|
||||
asset = session.get(Asset, asset_id)
|
||||
assert asset is not None, "archiving never deletes the record"
|
||||
assert asset.current_path is None
|
||||
assert asset.availability_state == "archived_online"
|
||||
assert asset.archive_location_id == location["id"]
|
||||
assert (archive / asset.archive_path).exists()
|
||||
occurrences = session.scalars(
|
||||
select(AssetPath).where(AssetPath.asset_id == asset_id)
|
||||
).all()
|
||||
active = [row for row in occurrences if row.valid_until is None]
|
||||
assert [row.path for row in active] == [str(archive / asset.archive_path)]
|
||||
closed = [row for row in occurrences if row.valid_until is not None]
|
||||
assert [row.path for row in closed] == [str(folder / asset.archive_path.split("/")[-1])]
|
||||
|
||||
|
||||
def test_applying_the_same_plan_again_changes_nothing(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
service.apply(plan["id"])
|
||||
archived = _contents(archive)
|
||||
manifest = read_manifest(archive / "rome")
|
||||
|
||||
again = service.apply(plan["id"])
|
||||
|
||||
assert again["skipped"] == 2 and again["archived"] == 0 and again["failed"] == 0
|
||||
assert _contents(archive) == archived
|
||||
assert read_manifest(archive / "rome") == manifest
|
||||
|
||||
|
||||
# ── unexpected changes stop the item ─────────────────────────────────────────
|
||||
|
||||
|
||||
def test_an_occupied_destination_is_never_overwritten(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib, names=("a.jpg",))
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
(archive / "rome").mkdir()
|
||||
(archive / "rome" / "a.jpg").write_bytes(b"someone else's file")
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
|
||||
assert result["failed"] == 1 and result["state"] == "failed"
|
||||
assert (archive / "rome" / "a.jpg").read_bytes() == b"someone else's file"
|
||||
assert (folder / "a.jpg").exists(), "the source must survive a refused transfer"
|
||||
assert service.journal.operations(plan["id"])[0]["error_code"] == "destination_exists"
|
||||
|
||||
|
||||
def test_a_source_edited_after_planning_is_neither_archived_nor_removed(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib, names=("a.jpg",))
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
|
||||
(folder / "a.jpg").write_bytes(b"edited between approval and apply")
|
||||
result = service.apply(plan["id"])
|
||||
|
||||
assert result["failed"] == 1
|
||||
assert (folder / "a.jpg").read_bytes() == b"edited between approval and apply"
|
||||
assert not (archive / "rome" / "a.jpg").exists()
|
||||
assert service.journal.operations(plan["id"])[0]["error_code"] == "source_changed"
|
||||
|
||||
|
||||
def test_a_copy_that_lands_with_the_wrong_bytes_is_not_published(tmp_path, monkeypatch):
|
||||
"""The read-back hash — not the fact that a write returned — is what proves the
|
||||
archive copy."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib, names=("a.jpg",))
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
monkeypatch.setattr(transfer_module, "_same_filesystem", lambda *_: False)
|
||||
monkeypatch.setattr(
|
||||
transfer_module.shutil,
|
||||
"copyfileobj",
|
||||
lambda src, dst, length=0: dst.write(b"corrupted in flight"),
|
||||
)
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
|
||||
assert result["failed"] == 1
|
||||
assert service.journal.operations(plan["id"])[0]["error_code"] == "copy_mismatch"
|
||||
assert (folder / "a.jpg").exists()
|
||||
assert not (archive / "rome" / "a.jpg").exists()
|
||||
assert not list((archive / "rome").glob(".archive-*")), "the failed copy is cleaned up"
|
||||
|
||||
|
||||
def test_a_source_replaced_by_a_symlink_is_refused(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib, names=("a.jpg",))
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
elsewhere = tmp_path / "elsewhere.jpg"
|
||||
elsewhere.write_bytes(b"not a library file")
|
||||
(folder / "a.jpg").unlink()
|
||||
(folder / "a.jpg").symlink_to(elsewhere)
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
|
||||
assert result["failed"] == 1
|
||||
assert elsewhere.exists() and (folder / "a.jpg").is_symlink()
|
||||
assert not (archive / "rome" / "a.jpg").exists()
|
||||
|
||||
|
||||
def test_one_failed_file_does_not_stop_the_rest_of_the_album(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, _ = _album(sf, lib, names=("a.jpg", "b.jpg"))
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
(folder / "a.jpg").write_bytes(b"edited after approval")
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
|
||||
assert (result["archived"], result["failed"]) == (1, 1)
|
||||
assert (folder / "a.jpg").exists() and not (folder / "b.jpg").exists()
|
||||
assert (archive / "rome" / "b.jpg").exists()
|
||||
assert folder.exists(), "a folder that still holds an unarchived file stays"
|
||||
|
||||
|
||||
def test_an_unresolved_transfer_blocks_the_next_preflight(tmp_path, monkeypatch):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib, "rome", names=("a.jpg",))
|
||||
_album(sf, lib, "paris", names=("c.jpg",))
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"], ["rome"])
|
||||
# Leave one item stuck mid-transfer, as a killed process would.
|
||||
service.journal.begin(
|
||||
service.journal.operations(plan["id"])[0]["id"], worker_id="w", fencing_token=1
|
||||
)
|
||||
|
||||
report = ArchiveService(sf, config=config).preflight(location["id"], ["paris"])
|
||||
|
||||
assert report["state"] == "blocked"
|
||||
assert "archive_pending" in {issue["code"] for issue in report["blockers"]}
|
||||
|
||||
|
||||
# ── API surface ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_api_creates_a_plan_and_queues_it_on_the_archiver_lane(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
location = client.post(
|
||||
"/api/v1/archive-locations", json={"name": "external", "root": str(archive)}
|
||||
).json()
|
||||
token = client.post(
|
||||
"/api/v1/archive-preflight", json={"location_id": location["id"]}
|
||||
).json()["token"]
|
||||
created = client.post(
|
||||
"/api/v1/archive-plans", json={"location_id": location["id"], "token": token}
|
||||
)
|
||||
plan_id = created.json()["id"]
|
||||
fetched = client.get(f"/api/v1/archive-plans/{plan_id}")
|
||||
applied = client.post(f"/api/v1/archive-plans/{plan_id}/apply")
|
||||
listed = client.get("/api/v1/archive-plans")
|
||||
|
||||
assert created.status_code == 201 and created.json()["asset_count"] == 2
|
||||
assert fetched.status_code == 200 and len(fetched.json()["operations"]) == 2
|
||||
assert applied.status_code == 200 and applied.json()["job"]["state"] == "queued"
|
||||
assert applied.json()["job"]["lock_key"] == "archive"
|
||||
assert [row["id"] for row in listed.json()["plans"]] == [plan_id]
|
||||
# Queuing alone must not have touched a single file.
|
||||
assert _contents(lib) and _contents(archive) == []
|
||||
|
||||
|
||||
def test_api_refuses_a_stale_token_and_an_unknown_plan(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib)
|
||||
location = _location(sf, config, archive)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
stale = client.post(
|
||||
"/api/v1/archive-plans",
|
||||
json={"location_id": location["id"], "token": "v1:not-the-real-token"},
|
||||
)
|
||||
unknown = client.get("/api/v1/archive-plans/nope")
|
||||
unknown_apply = client.post("/api/v1/archive-plans/nope/apply")
|
||||
|
||||
assert stale.status_code == 409 and stale.json()["error"]["code"] == "stale_token"
|
||||
assert unknown.status_code == 404 and unknown_apply.status_code == 404
|
||||
|
||||
|
||||
def test_api_reports_recovery_state_for_an_interrupted_transfer(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_album(sf, lib, names=("a.jpg",))
|
||||
location = _location(sf, config, archive)
|
||||
service, plan = _plan(sf, config, location["id"])
|
||||
service.journal.begin(
|
||||
service.journal.operations(plan["id"])[0]["id"], worker_id="w", fencing_token=1
|
||||
)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
status = client.get("/api/v1/archive-recovery").json()
|
||||
resolved = client.post("/api/v1/archive-recovery/resolve").json()
|
||||
|
||||
assert status["blocks_mutation"] is True
|
||||
assert status["operations"][0]["classification"] == "resumable"
|
||||
assert resolved == {"resumed": 1, "completed": 0, "manual": 0}
|
||||
assert json.loads(json.dumps(resolved)) # plain JSON, nothing exotic
|
||||
@@ -167,7 +167,8 @@ def test_missing_file_is_flagged_not_deleted(
|
||||
assets = assets_by_id(make_factory(db_url))
|
||||
assert missing_id in assets # not pruned
|
||||
assert assets[missing_id].missing_at is not None
|
||||
assert assets[missing_id].availability_state == "active"
|
||||
# Nothing explains the absence: this is not an offline archive medium (US06-03).
|
||||
assert assets[missing_id].availability_state == "missing_unexpected"
|
||||
|
||||
|
||||
def test_reappearing_file_clears_missing(
|
||||
@@ -186,6 +187,7 @@ def test_reappearing_file_clears_missing(
|
||||
inventory.scan(lib)
|
||||
assets = assets_by_id(make_factory(db_url))
|
||||
assert assets[asset_id].missing_at is None
|
||||
assert assets[asset_id].availability_state == "active"
|
||||
|
||||
|
||||
def test_identity_and_state_durable_across_restart(
|
||||
|
||||
238
tests/integration/test_legacy_import.py
Normal file
238
tests/integration/test_legacy_import.py
Normal file
@@ -0,0 +1,238 @@
|
||||
"""Importing the archived CLI's CSV state into the database (US07-01).
|
||||
|
||||
The donor's ``nsfw_scores.csv`` was keyed by path; the database is keyed by a
|
||||
stable asset id. Every case here is about that gap: a path that moved, a path that
|
||||
matches nothing, a score a human has already overruled, and a file that is simply
|
||||
malformed. The import may add evidence and must never invent an asset, overwrite a
|
||||
judgement, or fail silently — whatever it does ends up in the report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview
|
||||
from photo_pipeline.services.legacy_import import (
|
||||
REVIEWER,
|
||||
LegacyImportService,
|
||||
write_report,
|
||||
)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def factory(tmp_path):
|
||||
url = f"sqlite:///{tmp_path / 'legacy.db'}"
|
||||
run_migrations(url)
|
||||
engine = create_db_engine(url)
|
||||
yield create_session_factory(engine)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _asset(sf, path: Path, *, previous: Path | None = None) -> str:
|
||||
asset_id = str(uuid.uuid4())
|
||||
with sf() as session:
|
||||
session.add(
|
||||
Asset(
|
||||
id=asset_id,
|
||||
original_path=str(previous or path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
)
|
||||
)
|
||||
session.add(AssetPath(asset_id=asset_id, path=str(path), valid_from=NOW))
|
||||
if previous is not None:
|
||||
session.add(
|
||||
AssetPath(
|
||||
asset_id=asset_id, path=str(previous), valid_from=NOW, valid_until=NOW
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return asset_id
|
||||
|
||||
|
||||
def _review(sf, asset_id: str, *, score=None, decision=None, at=NOW) -> None:
|
||||
with sf() as session:
|
||||
session.add(
|
||||
SafetyReview(
|
||||
id=str(uuid.uuid4()),
|
||||
asset_id=asset_id,
|
||||
score=score,
|
||||
decision=decision,
|
||||
reviewer="dom",
|
||||
created_at=at,
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
|
||||
def _csv(tmp_path, rows: str) -> Path:
|
||||
path = tmp_path / "nsfw_scores.csv"
|
||||
path.write_text("path,nsfw_score\n" + rows, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _scores(sf) -> dict[str, float]:
|
||||
with sf() as session:
|
||||
return {
|
||||
review.asset_id: review.score
|
||||
for review in session.scalars(
|
||||
select(SafetyReview).order_by(SafetyReview.created_at)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def test_scores_are_imported_onto_asset_identity(tmp_path, factory):
|
||||
a = _asset(factory, tmp_path / "a.jpg")
|
||||
# This one was scored under its old path and has since moved.
|
||||
b = _asset(factory, tmp_path / "moved" / "b.jpg", previous=tmp_path / "b.jpg")
|
||||
csv_path = _csv(
|
||||
tmp_path, f"{tmp_path / 'a.jpg'},0.9123\n{tmp_path / 'b.jpg'},0.0100\n"
|
||||
)
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
assert report.counts == {
|
||||
"rows": 2,
|
||||
"imported": 2,
|
||||
"skipped_existing": 0,
|
||||
"unchanged": 0,
|
||||
"reviewed": 0,
|
||||
"unmatched": 0,
|
||||
"rejected": 0,
|
||||
"conflicts": 0,
|
||||
}
|
||||
assert _scores(factory) == {a: 0.9123, b: 0.0100}
|
||||
# The imported rows are evidence, not judgements: no decision is invented.
|
||||
with factory() as session:
|
||||
rows = list(session.scalars(select(SafetyReview)))
|
||||
assert {r.decision for r in rows} == {None}
|
||||
assert {r.reviewer for r in rows} == {REVIEWER}
|
||||
assert csv_path.exists(), "the CSV is left on disk as its own backup"
|
||||
|
||||
|
||||
def test_a_reviewed_asset_is_never_overwritten_by_the_csv(tmp_path, factory):
|
||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
||||
_review(factory, asset_id, score=0.2, decision="sfw")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.9999\n")
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path, overwrite=True)
|
||||
|
||||
assert (report.imported, report.reviewed) == (0, 1)
|
||||
assert report.conflicts[0]["decision"] == "sfw"
|
||||
assert report.conflicts[0]["csv"] == 0.9999
|
||||
with factory() as session:
|
||||
rows = list(session.scalars(select(SafetyReview)))
|
||||
assert len(rows) == 1 and rows[0].decision == "sfw"
|
||||
|
||||
|
||||
def test_an_unknown_path_is_reported_never_turned_into_an_asset(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(
|
||||
tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n{tmp_path / 'ghost.jpg'},0.5000\n"
|
||||
)
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
assert report.unmatched == [str(tmp_path / "ghost.jpg")]
|
||||
with factory() as session:
|
||||
assert session.scalar(select(Asset).where(Asset.current_path.like("%ghost%"))) is None
|
||||
|
||||
|
||||
def test_malformed_rows_are_rejected_with_their_reason(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(
|
||||
tmp_path,
|
||||
f"{tmp_path / 'a.jpg'},0.5000\n"
|
||||
f"{tmp_path / 'a.jpg'},0.5000\n" # the same path twice
|
||||
f"{tmp_path / 'b.jpg'},not-a-number\n"
|
||||
f"{tmp_path / 'c.jpg'},7.5\n" # outside 0..1: corruption, not a score
|
||||
",0.5\n", # no path at all
|
||||
)
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
reasons = sorted(r["reason"] for r in report.rejected)
|
||||
assert reasons == ["duplicate_path", "missing_path", "unparsable_score", "unparsable_score"]
|
||||
assert report.imported == 1
|
||||
|
||||
|
||||
def test_rerunning_changes_nothing_and_a_differing_score_needs_overwrite(tmp_path, factory):
|
||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
||||
service = LegacyImportService(factory)
|
||||
service.import_nsfw_scores(csv_path)
|
||||
|
||||
again = service.import_nsfw_scores(csv_path)
|
||||
assert (again.imported, again.unchanged) == (0, 1)
|
||||
assert len(_scores(factory)) == 1
|
||||
|
||||
changed = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.8000\n")
|
||||
refused = service.import_nsfw_scores(changed)
|
||||
assert (refused.imported, refused.skipped_existing) == (0, 1)
|
||||
assert refused.conflicts[0]["current"] == 0.5
|
||||
|
||||
forced = service.import_nsfw_scores(changed, overwrite=True)
|
||||
assert forced.imported == 1
|
||||
with factory() as session:
|
||||
latest = list(session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)))[-1]
|
||||
assert (latest.asset_id, latest.score) == (asset_id, 0.8)
|
||||
|
||||
|
||||
def test_a_dry_run_reports_without_writing(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
||||
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path, dry_run=True)
|
||||
|
||||
assert report.imported == 1
|
||||
assert _scores(factory) == {}, "a dry run must leave the database alone"
|
||||
|
||||
|
||||
def test_a_missing_csv_is_a_reported_outcome_not_a_crash(tmp_path, factory):
|
||||
report = LegacyImportService(factory).import_nsfw_scores(tmp_path / "nothing.csv")
|
||||
assert report.rejected == [{"path": str(tmp_path / "nothing.csv"), "reason": "csv_missing"}]
|
||||
assert report.rows == 0
|
||||
|
||||
|
||||
def test_the_report_is_written_where_it_can_be_audited(tmp_path, factory):
|
||||
_asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.5000\n")
|
||||
report = LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
|
||||
written = write_report(report, tmp_path / "data")
|
||||
|
||||
payload = json.loads(written.read_text(encoding="utf-8"))
|
||||
assert payload["imported"] == 1
|
||||
assert payload["source"] == str(csv_path)
|
||||
assert payload["schema_version"] == 1
|
||||
assert payload["generated_at"]
|
||||
|
||||
|
||||
def test_a_later_review_still_wins_after_an_import(tmp_path, factory):
|
||||
"""The import is a floor, not a ceiling: a human decision made afterwards is
|
||||
the current state, exactly as everywhere else in the app."""
|
||||
asset_id = _asset(factory, tmp_path / "a.jpg")
|
||||
csv_path = _csv(tmp_path, f"{tmp_path / 'a.jpg'},0.9000\n")
|
||||
LegacyImportService(factory).import_nsfw_scores(csv_path)
|
||||
# The import stamps itself with the wall clock, so "afterwards" has to be too.
|
||||
_review(
|
||||
factory,
|
||||
asset_id,
|
||||
score=0.9,
|
||||
decision="sfw",
|
||||
at=datetime.now(timezone.utc) + timedelta(hours=1),
|
||||
)
|
||||
|
||||
with factory() as session:
|
||||
latest = list(session.scalars(select(SafetyReview).order_by(SafetyReview.created_at)))[-1]
|
||||
assert latest.decision == "sfw"
|
||||
377
tests/integration/test_offline_assets.py
Normal file
377
tests/integration/test_offline_assets.py
Normal file
@@ -0,0 +1,377 @@
|
||||
"""Offline identity and review evidence (US06-03).
|
||||
|
||||
An archived photo is not gone: it keeps its identity, its hashes, and enough
|
||||
evidence to be recognised in a duplicate cluster while its medium sits in a
|
||||
drawer. Every case here archives a *real* album through the real transfer, then
|
||||
takes the medium away by removing its marker — the same thing the service sees
|
||||
when an external disk is unplugged — and asks whether the application still tells
|
||||
the truth about where the bytes are.
|
||||
|
||||
The distinction that matters throughout: an unmounted medium is
|
||||
``archived_offline`` (expected, harmless), a mounted medium with a hole in it is
|
||||
``missing_unexpected`` (needs a human). Confusing the two is how an archive
|
||||
quietly loses a photo.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from PIL import Image
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.models import Asset, Thumbnail, UploadBatch, UploadItem
|
||||
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
||||
from photo_pipeline.services.archives import MARKER_NAME, ArchiveService
|
||||
from photo_pipeline.services.availability import (
|
||||
ACTIVE,
|
||||
ARCHIVED_OFFLINE,
|
||||
ARCHIVED_ONLINE,
|
||||
MISSING_UNEXPECTED,
|
||||
)
|
||||
from photo_pipeline.services.duplicates import ClusterState, DuplicateService, Method
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.inventory import InventoryService
|
||||
from photo_pipeline.services.thumbnails import PROTECTED_SIZE, ThumbnailService, ThumbnailUnavailable
|
||||
|
||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
# ── environment ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _env(tmp_path):
|
||||
(tmp_path / "data").mkdir(exist_ok=True)
|
||||
lib = tmp_path / "lib"
|
||||
lib.mkdir(exist_ok=True)
|
||||
archive = tmp_path / "archive"
|
||||
archive.mkdir(exist_ok=True)
|
||||
config = Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
||||
"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0",
|
||||
}
|
||||
)
|
||||
run_migrations(config.database_url)
|
||||
return config, create_session_factory(create_db_engine(config.database_url)), lib, archive
|
||||
|
||||
|
||||
def structured(path, seed, size=(256, 192)):
|
||||
"""A deterministic, decodable photo — previews and pHashes must be real."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
rng = np.random.default_rng(seed)
|
||||
w, h = size
|
||||
base = np.zeros((h, w, 3), dtype=np.uint8)
|
||||
for _ in range(6):
|
||||
x0 = int(rng.integers(0, w - 60))
|
||||
y0 = int(rng.integers(0, h - 60))
|
||||
base[y0 : y0 + 60, x0 : x0 + 60] = rng.integers(0, 256, 3)
|
||||
grad = np.linspace(0, 120, w, dtype=np.uint8)
|
||||
base[:, :, 0] = np.clip(base[:, :, 0].astype(int) + grad[None, :], 0, 255)
|
||||
Image.fromarray(base).save(path, quality=95)
|
||||
return path
|
||||
|
||||
|
||||
def resized_copy(src, dst, scale=0.5):
|
||||
with Image.open(src) as image:
|
||||
image.resize(
|
||||
(int(image.width * scale), int(image.height * scale)), Image.LANCZOS
|
||||
).save(dst, quality=95)
|
||||
return dst
|
||||
|
||||
|
||||
def _uploaded(sf, lib, album="rome", seeds=(1, 2)):
|
||||
"""A scanned album carrying the verified upload evidence archiving requires."""
|
||||
folder = lib / album
|
||||
for index, seed in enumerate(seeds):
|
||||
structured(folder / f"{index}.jpg", seed)
|
||||
result = InventoryService(sf).scan(lib)
|
||||
with sf() as session:
|
||||
batch_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
UploadBatch(
|
||||
id=batch_id,
|
||||
album=album,
|
||||
folder=str(folder),
|
||||
album_name=album,
|
||||
state="succeeded",
|
||||
preflight_token="v1:test",
|
||||
outcome_state="verified",
|
||||
created_at=NOW,
|
||||
)
|
||||
)
|
||||
for path, asset_id in result.asset_ids.items():
|
||||
session.add(
|
||||
UploadItem(
|
||||
batch_id=batch_id,
|
||||
asset_id=asset_id,
|
||||
path=path,
|
||||
sha256=sha256_file(path),
|
||||
sha1="0" * 40,
|
||||
state="sent",
|
||||
outcome="uploaded",
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return folder, result.asset_ids
|
||||
|
||||
|
||||
def _archive(sf, config, archive, albums=None):
|
||||
service = ArchiveService(sf, config=config)
|
||||
location = service.register("external", str(archive))
|
||||
token = service.preflight(location["id"], albums)["token"]
|
||||
transfers = ArchiveTransferService(sf, config=config)
|
||||
plan = transfers.create(location["id"], albums, token=token)
|
||||
transfers.apply(plan["id"])
|
||||
return location
|
||||
|
||||
|
||||
def _unmount(archive):
|
||||
"""Take the medium away the way a real one goes: its marker stops answering."""
|
||||
(archive / MARKER_NAME).rename(archive / f"{MARKER_NAME}.away")
|
||||
|
||||
|
||||
def _remount(archive):
|
||||
(archive / f"{MARKER_NAME}.away").rename(archive / MARKER_NAME)
|
||||
|
||||
|
||||
def _assets(sf):
|
||||
with sf() as session:
|
||||
return {asset.id: asset for asset in session.scalars(select(Asset))}
|
||||
|
||||
|
||||
# ── availability ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_archived_assets_report_online_offline_and_missing(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_, ids = _uploaded(sf, lib)
|
||||
_archive(sf, config, archive)
|
||||
inventory = InventoryService(sf)
|
||||
|
||||
states = {a.availability_state for a in _assets(sf).values()}
|
||||
assert states == {ARCHIVED_ONLINE}
|
||||
|
||||
_unmount(archive)
|
||||
inventory.scan(lib) # the album folder is gone from the active roots
|
||||
assert {a.availability_state for a in _assets(sf).values()} == {ARCHIVED_OFFLINE}
|
||||
assert all(a.missing_at is None for a in _assets(sf).values()) # not "missing"
|
||||
|
||||
_remount(archive)
|
||||
inventory.scan(lib)
|
||||
assert {a.availability_state for a in _assets(sf).values()} == {ARCHIVED_ONLINE}
|
||||
|
||||
# Mounted medium, absent file: that is not an offline archive, it needs a human.
|
||||
victim = sorted(ids.values())[0]
|
||||
with sf() as session:
|
||||
asset = session.get(Asset, victim)
|
||||
(archive / asset.archive_path).unlink()
|
||||
inventory.scan(lib)
|
||||
assert _assets(sf)[victim].availability_state == MISSING_UNEXPECTED
|
||||
|
||||
|
||||
def test_scan_never_prunes_or_flags_offline_assets(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_, ids = _uploaded(sf, lib)
|
||||
_archive(sf, config, archive)
|
||||
_unmount(archive)
|
||||
|
||||
before = _assets(sf)
|
||||
result = InventoryService(sf).scan(lib)
|
||||
|
||||
assert result.counts.get("missing") is None
|
||||
after = _assets(sf)
|
||||
assert set(after) == set(before) == set(ids.values())
|
||||
for asset in after.values():
|
||||
assert asset.availability_state == ARCHIVED_OFFLINE
|
||||
assert asset.current_sha256 and asset.pixel_sha256 # hashes retained
|
||||
assert asset.archive_location_id and asset.archive_path
|
||||
|
||||
|
||||
def test_active_missing_file_is_missing_unexpected_not_offline(tmp_path):
|
||||
config, sf, lib, _archive_root = _env(tmp_path)
|
||||
structured(lib / "loose" / "a.jpg", 7)
|
||||
ids = InventoryService(sf).scan(lib).asset_ids
|
||||
asset_id = next(iter(ids.values()))
|
||||
|
||||
(lib / "loose" / "a.jpg").unlink()
|
||||
InventoryService(sf).scan(lib)
|
||||
|
||||
asset = _assets(sf)[asset_id]
|
||||
assert asset.availability_state == MISSING_UNEXPECTED
|
||||
assert asset.missing_at is not None
|
||||
|
||||
|
||||
# ── deduplication against archived originals ─────────────────────────────────
|
||||
|
||||
|
||||
def test_exact_copy_of_offline_asset_links_to_archived_canonical(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, ids = _uploaded(sf, lib, seeds=(1,))
|
||||
archived_id = next(iter(ids.values()))
|
||||
original = next(iter(ids))
|
||||
kept = tmp_path / "kept.jpg"
|
||||
shutil.copy2(original, kept)
|
||||
_archive(sf, config, archive)
|
||||
_unmount(archive)
|
||||
|
||||
# The same photo turns up again in the active library while the disk is away.
|
||||
(lib / "inbox").mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(kept, lib / "inbox" / "again.jpg")
|
||||
scan = InventoryService(sf).scan(lib)
|
||||
new_id = scan.asset_ids[str(lib / "inbox" / "again.jpg")]
|
||||
assert scan.occurrences[str(lib / "inbox" / "again.jpg")] == "copied"
|
||||
|
||||
clusters = DuplicateService(sf).detect().clusters
|
||||
exact = [c for c in clusters if c["method"] == Method.EXACT.value]
|
||||
assert len(exact) == 1
|
||||
cluster = exact[0]
|
||||
# Byte-identical: linked directly, and the archived original stays canonical.
|
||||
assert cluster["state"] == ClusterState.DECIDED.value
|
||||
assert cluster["canonical_asset_id"] == archived_id
|
||||
assert _assets(sf)[new_id].canonical_asset_id == archived_id
|
||||
|
||||
|
||||
def test_fuzzy_copy_of_offline_asset_requires_review_and_names_the_medium(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
folder, ids = _uploaded(sf, lib, seeds=(1,))
|
||||
archived_id = next(iter(ids.values()))
|
||||
original = next(iter(ids))
|
||||
variant_source = resized_copy(original, tmp_path / "small.jpg")
|
||||
_archive(sf, config, archive)
|
||||
_unmount(archive)
|
||||
|
||||
(lib / "inbox").mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(variant_source, lib / "inbox" / "small.jpg")
|
||||
InventoryService(sf).scan(lib)
|
||||
|
||||
duplicates = DuplicateService(sf)
|
||||
clusters = duplicates.detect().clusters
|
||||
perceptual = [c for c in clusters if c["method"] == Method.PERCEPTUAL.value]
|
||||
assert len(perceptual) == 1
|
||||
|
||||
detail = duplicates.get_cluster(perceptual[0]["id"])
|
||||
assert detail["state"] == ClusterState.OPEN.value # never auto-decided
|
||||
assert detail["requires_confirmation"] is True
|
||||
assert detail["mount_required"] == ["external"] # full-resolution needs the disk
|
||||
archived = next(m for m in detail["members"] if m["asset_id"] == archived_id)
|
||||
assert archived["availability_state"] == ARCHIVED_OFFLINE
|
||||
assert archived["current_path"] is None
|
||||
assert archived["archive_path"] and archived["evidence"]["phash"]
|
||||
# The retained preview is what makes the offline member reviewable at all.
|
||||
assert archived["preview"] == {"state": "ready", "protected": True, "size": PROTECTED_SIZE}
|
||||
|
||||
|
||||
# ── protected review evidence ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_protected_preview_survives_quota_and_serves_while_offline(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_, ids = _uploaded(sf, lib, seeds=(1,))
|
||||
asset_id = next(iter(ids.values()))
|
||||
_archive(sf, config, archive)
|
||||
_unmount(archive)
|
||||
|
||||
thumbnails = ThumbnailService(sf, config)
|
||||
served = thumbnails.generate(asset_id, PROTECTED_SIZE)
|
||||
assert served.exists() # rendered before the original left, not from the medium
|
||||
|
||||
# An aggressive quota may empty the cache, but not this evidence.
|
||||
tight = ThumbnailService(sf, config.model_copy(update={"thumbnail_cache_quota_bytes": 1}))
|
||||
tight._enforce_quota()
|
||||
assert served.exists()
|
||||
with sf() as session:
|
||||
row = session.scalar(select(Thumbnail).where(Thumbnail.asset_id == asset_id))
|
||||
assert row.protected is True
|
||||
assert thumbnails.evidence(asset_id)["state"] == "ready"
|
||||
|
||||
|
||||
def test_offline_asset_without_preview_reports_unavailable(tmp_path):
|
||||
"""No preview and no medium is an honest 409, never a wrong picture."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_, ids = _uploaded(sf, lib, seeds=(1,))
|
||||
asset_id = next(iter(ids.values()))
|
||||
_archive(sf, config, archive)
|
||||
with sf() as session:
|
||||
for row in session.scalars(select(Thumbnail).where(Thumbnail.asset_id == asset_id)):
|
||||
session.delete(row)
|
||||
session.commit()
|
||||
_unmount(archive)
|
||||
|
||||
with pytest.raises(ThumbnailUnavailable):
|
||||
ThumbnailService(sf, config).generate(asset_id, 256)
|
||||
|
||||
_remount(archive) # mounted again: the archived original is readable
|
||||
assert ThumbnailService(sf, config).generate(asset_id, 256).exists()
|
||||
|
||||
|
||||
def test_offline_asset_is_browsable_through_the_api(tmp_path):
|
||||
"""The browser sees an archived asset, its medium, and its preview — offline."""
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_, ids = _uploaded(sf, lib, seeds=(1,))
|
||||
asset_id = next(iter(ids.values()))
|
||||
_archive(sf, config, archive)
|
||||
_unmount(archive)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
client.post("/api/v1/inventory/scan")
|
||||
listed = client.get("/api/v1/inventory/assets", params={"availability": ARCHIVED_OFFLINE})
|
||||
assert listed.status_code == 200
|
||||
item = next(row for row in listed.json()["items"] if row["id"] == asset_id)
|
||||
assert item["current_path"] is None
|
||||
assert item["archive_path"] == "rome/0.jpg"
|
||||
assert item["missing"] is False
|
||||
|
||||
# Searching by the archived path still finds it.
|
||||
found = client.get("/api/v1/inventory/assets", params={"q": "rome"}).json()
|
||||
assert [row["id"] for row in found["items"]] == [asset_id]
|
||||
|
||||
preview = client.get(f"/api/v1/assets/{asset_id}/thumbnail", params={"size": 1280})
|
||||
assert preview.status_code == 200
|
||||
assert preview.headers["content-type"] == "image/webp"
|
||||
|
||||
|
||||
def test_offline_state_is_stable_across_restart(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
_, ids = _uploaded(sf, lib, seeds=(1, 2))
|
||||
_archive(sf, config, archive)
|
||||
_unmount(archive)
|
||||
InventoryService(sf).scan(lib)
|
||||
before = {
|
||||
asset_id: (
|
||||
asset.availability_state,
|
||||
asset.archive_path,
|
||||
asset.current_sha256,
|
||||
asset.phash,
|
||||
)
|
||||
for asset_id, asset in _assets(sf).items()
|
||||
}
|
||||
|
||||
# Restart: a fresh engine and session factory against the same database.
|
||||
restarted = create_session_factory(create_db_engine(config.database_url))
|
||||
after = {
|
||||
asset_id: (
|
||||
asset.availability_state,
|
||||
asset.archive_path,
|
||||
asset.current_sha256,
|
||||
asset.phash,
|
||||
)
|
||||
for asset_id, asset in _assets(restarted).items()
|
||||
}
|
||||
assert after == before
|
||||
assert set(after) == set(ids.values())
|
||||
|
||||
# And the medium coming back is picked up by the restarted process.
|
||||
_remount(archive)
|
||||
assert ArchiveService(restarted, config=config).locations()[0]["state"] == "online"
|
||||
assert {a.availability_state for a in _assets(restarted).values()} == {ARCHIVED_ONLINE}
|
||||
assert ACTIVE not in {a.availability_state for a in _assets(restarted).values()}
|
||||
417
tests/integration/test_restore.py
Normal file
417
tests/integration/test_restore.py
Normal file
@@ -0,0 +1,417 @@
|
||||
"""Planning and executing safe restores (US06-04).
|
||||
|
||||
Restoring is the one archive operation that can *add* a file to the library, so
|
||||
every case here asks two questions: did the right bytes come back under the right
|
||||
identity, and did anything already in the library get touched? The media are real
|
||||
directories, the hashes are real, and the failure paths assert that the archived
|
||||
copy is still exactly where it was — a restore that fails must cost nothing.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from PIL import Image
|
||||
from sqlalchemy import select
|
||||
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.models import Asset, AssetPath, SafetyReview, UploadBatch, UploadItem
|
||||
from photo_pipeline.services import availability
|
||||
from photo_pipeline.services.archive_journal import ArchiveState
|
||||
from photo_pipeline.services.archive_transfer import ArchiveTransferService
|
||||
from photo_pipeline.services.archives import MARKER_NAME, ArchiveError, ArchiveService
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.inventory import InventoryService
|
||||
from photo_pipeline.services.restores import RestoreService
|
||||
|
||||
pytestmark = pytest.mark.phase_f # part of the Phase F acceptance gate (US06-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
# ── environment ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _env(tmp_path):
|
||||
(tmp_path / "data").mkdir(exist_ok=True)
|
||||
lib = tmp_path / "lib"
|
||||
lib.mkdir(exist_ok=True)
|
||||
archive = tmp_path / "archive"
|
||||
archive.mkdir(exist_ok=True)
|
||||
config = Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
||||
"PHOTO_PIPELINE_ARCHIVE_FREE_SPACE_RESERVE_BYTES": "0",
|
||||
}
|
||||
)
|
||||
run_migrations(config.database_url)
|
||||
return config, create_session_factory(create_db_engine(config.database_url)), lib, archive
|
||||
|
||||
|
||||
def structured(path, seed, size=(192, 144)):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
rng = np.random.default_rng(seed)
|
||||
w, h = size
|
||||
base = np.zeros((h, w, 3), dtype=np.uint8)
|
||||
for _ in range(5):
|
||||
x0 = int(rng.integers(0, w - 40))
|
||||
y0 = int(rng.integers(0, h - 40))
|
||||
base[y0 : y0 + 40, x0 : x0 + 40] = rng.integers(0, 256, 3)
|
||||
Image.fromarray(base).save(path, quality=95)
|
||||
return path
|
||||
|
||||
|
||||
def _archived(sf, config, lib, archive, album="rome", seeds=(1, 2)):
|
||||
"""A real album taken all the way through archiving, ready to be restored."""
|
||||
folder = lib / album
|
||||
for index, seed in enumerate(seeds):
|
||||
structured(folder / f"{index}.jpg", seed)
|
||||
scan = InventoryService(sf).scan(lib)
|
||||
with sf() as session:
|
||||
batch_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
UploadBatch(
|
||||
id=batch_id,
|
||||
album=album,
|
||||
folder=str(folder),
|
||||
album_name=album,
|
||||
state="succeeded",
|
||||
preflight_token="v1:test",
|
||||
outcome_state="verified",
|
||||
created_at=NOW,
|
||||
)
|
||||
)
|
||||
for path, asset_id in scan.asset_ids.items():
|
||||
session.add(
|
||||
UploadItem(
|
||||
batch_id=batch_id,
|
||||
asset_id=asset_id,
|
||||
path=path,
|
||||
sha256=sha256_file(path),
|
||||
sha1="0" * 40,
|
||||
state="sent",
|
||||
outcome="uploaded",
|
||||
)
|
||||
)
|
||||
# A decision that must survive the whole round trip.
|
||||
session.add(
|
||||
SafetyReview(
|
||||
id=str(uuid.uuid4()),
|
||||
asset_id=asset_id,
|
||||
decision="sfw",
|
||||
score=0.01,
|
||||
reviewer="test",
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
service = ArchiveService(sf, config=config)
|
||||
location = service.register("external", str(archive))
|
||||
token = service.preflight(location["id"])["token"]
|
||||
transfers = ArchiveTransferService(sf, config=config)
|
||||
plan = transfers.create(location["id"], None, token=token)
|
||||
transfers.apply(plan["id"])
|
||||
return location, scan.asset_ids
|
||||
|
||||
|
||||
def _restore(sf, config, location_id, asset_ids=None):
|
||||
service = RestoreService(sf, config=config)
|
||||
token = service.preflight(location_id, asset_ids)["token"]
|
||||
plan = service.create(location_id, asset_ids, token=token)
|
||||
return service, plan, service.apply(plan["id"])
|
||||
|
||||
|
||||
def _unmount(archive):
|
||||
(archive / MARKER_NAME).rename(archive / f"{MARKER_NAME}.away")
|
||||
|
||||
|
||||
def _assets(sf):
|
||||
with sf() as session:
|
||||
return {asset.id: asset for asset in session.scalars(select(Asset))}
|
||||
|
||||
|
||||
def _codes(report):
|
||||
return {issue["code"] for issue in report["blockers"]} | {
|
||||
issue["code"] for item in report["items"] for issue in item["blockers"]
|
||||
}
|
||||
|
||||
|
||||
# ── preflight ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_preflight_blocks_offline_medium(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, _ = _archived(sf, config, lib, archive)
|
||||
_unmount(archive)
|
||||
|
||||
report = RestoreService(sf, config=config).preflight(location["id"])
|
||||
assert report["state"] == "blocked"
|
||||
assert "location_offline" in _codes(report)
|
||||
|
||||
|
||||
def test_preflight_blocks_wrong_volume(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, _ = _archived(sf, config, lib, archive)
|
||||
(archive / MARKER_NAME).write_text('{"media_id": "someone-elses-disk"}', encoding="utf-8")
|
||||
|
||||
report = RestoreService(sf, config=config).preflight(location["id"])
|
||||
assert report["state"] == "blocked"
|
||||
assert "wrong_volume" in _codes(report)
|
||||
|
||||
|
||||
def test_preflight_blocks_changed_archive_bytes(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
||||
asset_id = next(iter(ids.values()))
|
||||
with sf() as session:
|
||||
archived_file = archive / session.get(Asset, asset_id).archive_path
|
||||
archived_file.write_bytes(b"not the photo that was archived")
|
||||
|
||||
report = RestoreService(sf, config=config).preflight(location["id"])
|
||||
assert report["state"] == "blocked"
|
||||
assert "bytes_changed" in _codes(report)
|
||||
with pytest.raises(ArchiveError) as error:
|
||||
_restore(sf, config, location["id"])
|
||||
assert error.value.code == "blocked"
|
||||
|
||||
|
||||
def test_preflight_blocks_insufficient_capacity(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, _ = _archived(sf, config, lib, archive)
|
||||
greedy = config.model_copy(
|
||||
update={"archive_free_space_reserve_bytes": 1 << 62} # more than any disk has
|
||||
)
|
||||
|
||||
report = RestoreService(sf, config=greedy).preflight(location["id"])
|
||||
assert report["state"] == "blocked"
|
||||
assert "insufficient_capacity" in _codes(report)
|
||||
|
||||
|
||||
def test_token_changes_with_the_scope(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive)
|
||||
service = RestoreService(sf, config=config)
|
||||
|
||||
whole = service.preflight(location["id"])["token"]
|
||||
partial = service.preflight(location["id"], [sorted(ids.values())[0]])["token"]
|
||||
assert whole != partial
|
||||
assert service.verify_token(whole, location["id"])
|
||||
assert not service.verify_token(partial, location["id"])
|
||||
|
||||
|
||||
# ── restore ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_restore_returns_bytes_identity_and_decisions(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive)
|
||||
archived_hashes = {
|
||||
asset_id: asset.current_sha256 for asset_id, asset in _assets(sf).items()
|
||||
}
|
||||
|
||||
service, plan, result = _restore(sf, config, location["id"])
|
||||
assert (result["restored"], result["failed"], result["state"]) == (2, 0, "complete")
|
||||
|
||||
for asset_id, asset in _assets(sf).items():
|
||||
assert asset.availability_state == availability.ACTIVE
|
||||
assert asset.current_path == str(lib / asset.archive_path)
|
||||
assert sha256_file(asset.current_path) == archived_hashes[asset_id]
|
||||
# The archived copy is a copy: restoring never empties the medium.
|
||||
assert (archive / asset.archive_path).exists()
|
||||
assert asset.archive_location_id == location["id"]
|
||||
|
||||
with sf() as session:
|
||||
# Identity and decisions survived: same ids, same reviews, new occurrence.
|
||||
assert set(ids.values()) == {a.id for a in session.scalars(select(Asset))}
|
||||
assert {r.decision for r in session.scalars(select(SafetyReview))} == {"sfw"}
|
||||
occurrences = [
|
||||
row.reason
|
||||
for row in session.scalars(
|
||||
select(AssetPath).where(AssetPath.asset_id == sorted(ids.values())[0])
|
||||
)
|
||||
]
|
||||
assert "restore" in occurrences
|
||||
|
||||
|
||||
def test_restore_never_overwrites_a_collision(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
||||
asset_id = next(iter(ids.values()))
|
||||
with sf() as session:
|
||||
archive_path = session.get(Asset, asset_id).archive_path
|
||||
occupied = lib / archive_path
|
||||
occupied.parent.mkdir(parents=True, exist_ok=True)
|
||||
occupied.write_bytes(b"a different photo already lives here")
|
||||
before = occupied.read_bytes()
|
||||
|
||||
service, plan, result = _restore(sf, config, location["id"])
|
||||
assert result["failed"] == 0
|
||||
|
||||
assert occupied.read_bytes() == before # untouched
|
||||
restored = _assets(sf)[asset_id].current_path
|
||||
assert restored != str(occupied)
|
||||
assert "(restored)" in restored
|
||||
assert sha256_file(restored) == sha256_file(archive / archive_path)
|
||||
|
||||
|
||||
def test_apply_refuses_a_destination_taken_after_planning(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
||||
service = RestoreService(sf, config=config)
|
||||
token = service.preflight(location["id"])["token"]
|
||||
plan = service.create(location["id"], None, token=token)
|
||||
|
||||
# Someone drops a file exactly where the plan intends to publish.
|
||||
destination = plan["operations"][0]["destination_path"]
|
||||
from pathlib import Path
|
||||
|
||||
Path(destination).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(destination).write_bytes(b"squatter")
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
assert result["failed"] == 1
|
||||
operation = service.journal.operations(plan["id"])[0]
|
||||
assert operation["journal_state"] == ArchiveState.FAILED
|
||||
assert operation["error_code"] == "destination_exists"
|
||||
assert Path(destination).read_bytes() == b"squatter"
|
||||
assert _assets(sf)[next(iter(ids.values()))].availability_state == availability.ARCHIVED_ONLINE
|
||||
|
||||
|
||||
def test_changed_archive_bytes_mark_the_asset_divergent(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
||||
asset_id = next(iter(ids.values()))
|
||||
service = RestoreService(sf, config=config)
|
||||
token = service.preflight(location["id"])["token"]
|
||||
plan = service.create(location["id"], None, token=token)
|
||||
|
||||
# The medium's copy is edited after the plan was approved.
|
||||
with sf() as session:
|
||||
archived_file = archive / session.get(Asset, asset_id).archive_path
|
||||
archived_file.write_bytes(b"edited on the shelf")
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
assert result["failed"] == 1
|
||||
operation = service.journal.operations(plan["id"])[0]
|
||||
assert operation["error_code"] == "bytes_changed"
|
||||
asset = _assets(sf)[asset_id]
|
||||
assert asset.archive_divergent_at is not None # durable divergence
|
||||
assert asset.availability_state == availability.ARCHIVED_ONLINE
|
||||
assert asset.current_path is None # nothing was published
|
||||
|
||||
|
||||
# ── interruption and idempotency ─────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_interrupted_before_publishing_is_resumable(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
||||
service = RestoreService(sf, config=config)
|
||||
token = service.preflight(location["id"])["token"]
|
||||
plan = service.create(location["id"], None, token=token)
|
||||
operation = service.journal.operations(plan["id"])[0]
|
||||
|
||||
# Model a kill right after the intent was written: nothing published yet.
|
||||
service.journal.begin(operation["id"], worker_id="killed", fencing_token=1)
|
||||
|
||||
status = service.recovery_status()
|
||||
assert status["operations"][0]["classification"] == "resumable"
|
||||
assert service.recover() == {"resumed": 1, "completed": 0, "manual": 0}
|
||||
assert service.journal.operations(plan["id"])[0]["journal_state"] == ArchiveState.PLANNED
|
||||
|
||||
result = service.apply(plan["id"])
|
||||
assert result["failed"] == 0
|
||||
assert _assets(sf)[next(iter(ids.values()))].availability_state == availability.ACTIVE
|
||||
|
||||
|
||||
def test_interrupted_after_publishing_is_finished_by_recovery(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
||||
asset_id = next(iter(ids.values()))
|
||||
service = RestoreService(sf, config=config)
|
||||
token = service.preflight(location["id"])["token"]
|
||||
plan = service.create(location["id"], None, token=token)
|
||||
operation = service.journal.operations(plan["id"])[0]
|
||||
|
||||
# Model a kill between the published copy and the database update.
|
||||
from pathlib import Path
|
||||
|
||||
destination = Path(operation["destination_path"])
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(operation["source_path"], destination)
|
||||
service.journal.begin(operation["id"], worker_id="killed", fencing_token=1)
|
||||
service.journal.transition(operation["id"], ArchiveState.VERIFIED, fencing_token=1)
|
||||
|
||||
assert service.recovery_status()["operations"][0]["classification"] == "forward"
|
||||
assert service.recover()["completed"] == 1
|
||||
asset = _assets(sf)[asset_id]
|
||||
assert asset.availability_state == availability.ACTIVE
|
||||
assert asset.current_path == str(destination)
|
||||
|
||||
# Repeated recovery and a repeated apply converge on the same state.
|
||||
assert service.recover() == {"resumed": 0, "completed": 0, "manual": 0}
|
||||
again = service.apply(plan["id"])
|
||||
assert (again["skipped"], again["failed"]) == (1, 0)
|
||||
assert _assets(sf)[asset_id].current_path == str(destination)
|
||||
|
||||
|
||||
def test_restored_state_survives_restart_and_rescan(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive)
|
||||
_restore(sf, config, location["id"])
|
||||
|
||||
restarted = create_session_factory(create_db_engine(config.database_url))
|
||||
InventoryService(restarted).scan(lib)
|
||||
|
||||
assets = _assets(restarted)
|
||||
assert set(assets) == set(ids.values()) # no new identities from the rescan
|
||||
for asset in assets.values():
|
||||
assert asset.availability_state == availability.ACTIVE
|
||||
assert asset.missing_at is None
|
||||
|
||||
# Nothing is archived at that location any more, so there is nothing to restore.
|
||||
again = RestoreService(restarted, config=config).preflight(location["id"])
|
||||
assert _codes(again) == {"empty_scope"}
|
||||
|
||||
|
||||
def test_restore_api_round_trip(tmp_path):
|
||||
config, sf, lib, archive = _env(tmp_path)
|
||||
location, ids = _archived(sf, config, lib, archive, seeds=(1,))
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
report = client.post(
|
||||
"/api/v1/restore-preflight", json={"location_id": location["id"]}
|
||||
).json()
|
||||
assert report["state"] == "ready"
|
||||
|
||||
stale = client.post(
|
||||
"/api/v1/restore-plans",
|
||||
json={"location_id": location["id"], "token": "r1:not-the-token"},
|
||||
)
|
||||
assert stale.status_code == 409
|
||||
|
||||
created = client.post(
|
||||
"/api/v1/restore-plans",
|
||||
json={"location_id": location["id"], "token": report["token"]},
|
||||
)
|
||||
assert created.status_code == 201
|
||||
plan_id = created.json()["id"]
|
||||
assert created.json()["direction"] == "restore"
|
||||
|
||||
# The plan is visible and applying it queues work on the archiver lane.
|
||||
assert client.get(f"/api/v1/restore-plans/{plan_id}").status_code == 200
|
||||
queued = client.post(f"/api/v1/restore-plans/{plan_id}/apply")
|
||||
assert queued.status_code == 200
|
||||
assert queued.json()["job"]["job_type"] == "restore_plan"
|
||||
assert queued.json()["job"]["lock_key"] == "archive"
|
||||
assert client.get("/api/v1/restore-recovery").json()["manual"] == []
|
||||
|
||||
assert _assets(sf)[next(iter(ids.values()))].availability_state == (
|
||||
availability.ARCHIVED_ONLINE # the worker, not the request, does the work
|
||||
)
|
||||
@@ -3,6 +3,8 @@ same fixture files (real exiftool-written keywords)."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
@@ -11,6 +13,10 @@ from PIL import Image
|
||||
from photo_pipeline.integrations import exiftool
|
||||
from photo_pipeline.services import safety
|
||||
|
||||
# The donor is frozen in the read-only archive (US07-01); parity still compares
|
||||
# against it, so this suite is the second and last place that may import it.
|
||||
ARCHIVED_SOURCES = Path(__file__).resolve().parents[2] / "legacy_cli_archive" / "src"
|
||||
|
||||
EXIFTOOL = shutil.which("exiftool")
|
||||
pytestmark = pytest.mark.skipif(EXIFTOOL is None, reason="exiftool not installed")
|
||||
|
||||
@@ -31,6 +37,7 @@ def _tag(path, keyword):
|
||||
|
||||
|
||||
def test_extracted_marks_and_partition_match_donor(tmp_path):
|
||||
sys.path.insert(0, str(ARCHIVED_SOURCES)) # the donor lives in the archive now
|
||||
donor_exif = pytest.importorskip("nsfwtag.exif")
|
||||
|
||||
nsfw = _jpeg(tmp_path / "nsfw.jpg", 1)
|
||||
|
||||
@@ -38,6 +38,8 @@ from photo_pipeline.services.upload_batches import (
|
||||
)
|
||||
from photo_pipeline.services.uploads import UploadService
|
||||
|
||||
pytestmark = pytest.mark.phase_e # part of the Phase E acceptance gate (US05-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
SENTINEL_KEY = "immich-sentinel-9f3a2b"
|
||||
UPLOADER_VERSION = "immich-go 0.21.0"
|
||||
|
||||
@@ -29,6 +29,8 @@ from photo_pipeline.models import (
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.uploads import UploadError, UploadService
|
||||
|
||||
pytestmark = pytest.mark.phase_e # part of the Phase E acceptance gate (US05-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
# A sentinel credential: every assertion below proves it never leaves configuration.
|
||||
SENTINEL_KEY = "immich-sentinel-9f3a2b"
|
||||
|
||||
335
tests/integration/test_upload_reports.py
Normal file
335
tests/integration/test_upload_reports.py
Normal file
@@ -0,0 +1,335 @@
|
||||
"""Durable per-item upload outcomes (US05-03).
|
||||
|
||||
The uploader is a real executable that prints a real report, driven through the
|
||||
real batch service, so the classification path exercised here is the one
|
||||
production uses. What matters is that the stored outcome is never more optimistic
|
||||
than the evidence: an unmentioned file, an unpinned uploader version, or counts
|
||||
that disagree all leave the batch requiring verification.
|
||||
"""
|
||||
|
||||
import json
|
||||
import stat
|
||||
import threading
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.integrations import immich_go_report as report_parser
|
||||
from photo_pipeline.models import AnalysisResult, Asset, SafetyReview
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.upload_batches import BatchState, UploadBatchService
|
||||
from photo_pipeline.services.upload_reports import (
|
||||
REQUIRES_VERIFICATION,
|
||||
VERIFIED,
|
||||
UploadReportService,
|
||||
)
|
||||
from photo_pipeline.services.uploads import UploadService
|
||||
|
||||
pytestmark = pytest.mark.phase_e # part of the Phase E acceptance gate (US05-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
SUPPORTED_VERSION = "immich-go 0.21.0"
|
||||
|
||||
|
||||
# ── fake external boundary ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
class _PingHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self): # noqa: N802 (BaseHTTPRequestHandler API)
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"res":"pong"}')
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def immich_server():
|
||||
server = HTTPServer(("127.0.0.1", 0), _PingHandler)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
yield f"http://127.0.0.1:{server.server_port}"
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
|
||||
def _uploader(tmp_path, report_body: str, *, version=SUPPORTED_VERSION):
|
||||
"""An uploader that answers ``--version`` and prints ``report_body``."""
|
||||
path = tmp_path / "immich-go"
|
||||
path.write_text(
|
||||
"#!/bin/sh\n"
|
||||
f'if [ "$1" = "--version" ]; then echo "{version}"; exit 0; fi\n'
|
||||
f"cat <<'REPORT'\n{report_body}\nREPORT\n"
|
||||
"exit 0\n"
|
||||
)
|
||||
path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
|
||||
return path
|
||||
|
||||
|
||||
# ── environment ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _env(tmp_path, server_url, uploader):
|
||||
(tmp_path / "data").mkdir(exist_ok=True)
|
||||
lib = tmp_path / "lib"
|
||||
lib.mkdir(exist_ok=True)
|
||||
config = Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
||||
"PHOTO_PIPELINE_IMMICH_SERVER_URL": server_url,
|
||||
"PHOTO_PIPELINE_IMMICH_API_KEY": "immich-sentinel-9f3a2b",
|
||||
"PHOTO_PIPELINE_IMMICH_GO_BINARY": str(uploader),
|
||||
}
|
||||
)
|
||||
run_migrations(config.database_url)
|
||||
return config, create_session_factory(create_db_engine(config.database_url)), lib
|
||||
|
||||
|
||||
def _album(sf, lib, album="rome", names=("new.jpg", "dup.jpg")):
|
||||
folder = lib / album
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
with sf() as session:
|
||||
for name in names:
|
||||
path = folder / name
|
||||
path.write_bytes(f"{album}/{name}".encode() * 16)
|
||||
asset_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
Asset(
|
||||
id=asset_id,
|
||||
original_path=str(path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
byte_size=path.stat().st_size,
|
||||
current_sha256=sha256_file(path),
|
||||
)
|
||||
)
|
||||
session.add(
|
||||
SafetyReview(
|
||||
id=str(uuid.uuid4()),
|
||||
asset_id=asset_id,
|
||||
decision="sfw",
|
||||
exif_verified_at=NOW,
|
||||
)
|
||||
)
|
||||
session.add(AnalysisResult(asset_id=asset_id, status="analyzed", exif_written_at=NOW))
|
||||
session.commit()
|
||||
return folder
|
||||
|
||||
|
||||
def _run(sf, config, albums=None):
|
||||
"""Create a batch from a fresh preflight and run it to completion."""
|
||||
report = UploadService(sf, config=config).preflight(albums)
|
||||
assert report["state"] == "ready", report["blockers"]
|
||||
service = UploadBatchService(sf, config=config)
|
||||
(batch,) = service.create(albums, token=report["token"])
|
||||
return service.run(batch["id"])
|
||||
|
||||
|
||||
def _outcomes(batch) -> dict:
|
||||
return {Path(item["path"]).name: item["outcome"] for item in batch["items"]}
|
||||
|
||||
|
||||
def _report_for(folder, **outcomes) -> str:
|
||||
"""A text-v1 report plus a matching summary line."""
|
||||
lines = {
|
||||
"uploaded": "INFO uploaded {path}",
|
||||
"duplicate": "INFO server has the same file {path}",
|
||||
"upgraded": "INFO server has an older file, upgrading {path}",
|
||||
"failed": "ERROR error uploading {path}: connection reset",
|
||||
}
|
||||
body = [lines[outcome].format(path=folder / name) for name, outcome in outcomes.items()]
|
||||
totals: dict[str, int] = {}
|
||||
for outcome in outcomes.values():
|
||||
totals[outcome] = totals.get(outcome, 0) + 1
|
||||
words = {
|
||||
"uploaded": "Uploaded",
|
||||
"duplicate": "duplicates",
|
||||
"upgraded": "upgraded",
|
||||
"failed": "errors",
|
||||
}
|
||||
body.append(", ".join(f"{words[k]} {v}" for k, v in totals.items()))
|
||||
return "\n".join(body)
|
||||
|
||||
|
||||
# ── classification ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_every_file_gets_its_reported_outcome_with_evidence(tmp_path, immich_server):
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "duplicate"})
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
|
||||
batch = _run(sf, config)
|
||||
|
||||
assert batch["state"] == BatchState.SUCCEEDED
|
||||
assert batch["outcome_state"] == VERIFIED
|
||||
assert _outcomes(batch) == {"new.jpg": "uploaded", "dup.jpg": "duplicate"}
|
||||
assert batch["outcome_counts"]["uploaded"] == 1
|
||||
assert batch["outcome_counts"]["duplicate"] == 1
|
||||
assert batch["parser"] == "text-v1"
|
||||
assert batch["parser_version"] == report_parser.PARSER_VERSION
|
||||
assert batch["parsed_at"]
|
||||
for item in batch["items"]:
|
||||
assert item["sha1"] and item["sha256"], "the uploaded bytes stay identifiable"
|
||||
assert Path(item["path"]).name in item["evidence"]
|
||||
assert item["outcome_at"]
|
||||
|
||||
|
||||
def test_a_file_the_report_never_mentions_is_unknown_not_sent(tmp_path, immich_server):
|
||||
"""Exit code 0 says the process ended well, not that this photo reached Immich."""
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(lib / "rome", **{"new.jpg": "uploaded"})
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
|
||||
batch = _run(sf, config)
|
||||
|
||||
assert batch["state"] == BatchState.SUCCEEDED, "the process itself succeeded"
|
||||
assert _outcomes(batch)["dup.jpg"] == report_parser.UNKNOWN
|
||||
assert batch["outcome_state"] == REQUIRES_VERIFICATION
|
||||
|
||||
|
||||
def test_an_unsupported_uploader_version_makes_every_item_unknown(tmp_path, immich_server):
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "uploaded"})
|
||||
uploader = _uploader(tmp_path, report, version="immich-go 9.99.0")
|
||||
config, sf, lib = _env(tmp_path, immich_server, uploader)
|
||||
_album(sf, lib)
|
||||
|
||||
batch = _run(sf, config)
|
||||
|
||||
assert batch["parser"] is None
|
||||
assert set(_outcomes(batch).values()) == {report_parser.UNKNOWN}
|
||||
assert batch["outcome_state"] == REQUIRES_VERIFICATION
|
||||
|
||||
|
||||
def test_an_empty_report_leaves_everything_unknown(tmp_path, immich_server):
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, ""))
|
||||
_album(sf, lib)
|
||||
|
||||
batch = _run(sf, config)
|
||||
|
||||
assert set(_outcomes(batch).values()) == {report_parser.UNKNOWN}
|
||||
assert batch["outcome_state"] == REQUIRES_VERIFICATION
|
||||
|
||||
|
||||
def test_a_failed_upload_is_classified_from_its_own_line(tmp_path, immich_server):
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "failed"})
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
|
||||
batch = _run(sf, config)
|
||||
|
||||
assert _outcomes(batch) == {"new.jpg": "uploaded", "dup.jpg": "failed"}
|
||||
# Every file is accounted for, so the evidence is complete even though one
|
||||
# upload failed — resolving the failure is US05-04's job, not a re-parse.
|
||||
assert batch["outcome_state"] == VERIFIED
|
||||
|
||||
|
||||
# ── reconciliation ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_summary_that_disagrees_with_the_lines_requires_verification(tmp_path, immich_server):
|
||||
lib = tmp_path / "lib"
|
||||
report = (
|
||||
_report_for(lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "uploaded"}).rsplit("\n", 1)[
|
||||
0
|
||||
]
|
||||
+ "\nUploaded 5"
|
||||
)
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
|
||||
batch = _run(sf, config)
|
||||
|
||||
assert batch["report_counts"] == {"uploaded": 5}
|
||||
assert batch["outcome_counts"]["uploaded"] == 2
|
||||
assert batch["outcome_state"] == REQUIRES_VERIFICATION
|
||||
|
||||
|
||||
def test_a_reported_file_outside_the_batch_requires_verification(tmp_path, immich_server):
|
||||
"""The uploader touched something the batch never approved."""
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(
|
||||
lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "uploaded", "stranger.jpg": "uploaded"}
|
||||
)
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
|
||||
batch = _run(sf, config)
|
||||
|
||||
assert set(_outcomes(batch).values()) == {"uploaded"}
|
||||
assert batch["outcome_state"] == REQUIRES_VERIFICATION
|
||||
|
||||
|
||||
# ── idempotency and restart ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_reprocessing_the_same_report_is_idempotent(tmp_path, immich_server):
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "duplicate"})
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
batch = _run(sf, config)
|
||||
|
||||
first = UploadReportService(sf).ingest(batch["id"])
|
||||
second = UploadReportService(sf).ingest(batch["id"])
|
||||
|
||||
assert first == second
|
||||
reread = UploadBatchService(sf, config=config).get(batch["id"])
|
||||
assert _outcomes(reread) == _outcomes(batch)
|
||||
assert len(reread["items"]) == 2, "re-import must not duplicate item rows"
|
||||
assert reread["outcome_counts"] == batch["outcome_counts"]
|
||||
|
||||
|
||||
def test_outcomes_survive_a_restart_and_reach_the_api(tmp_path, immich_server):
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "duplicate"})
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
batch = _run(sf, config)
|
||||
|
||||
with TestClient(create_app(config)) as client: # a fresh application process
|
||||
fetched = client.get(f"/api/v1/upload-batches/{batch['id']}").json()
|
||||
|
||||
assert fetched["outcome_state"] == VERIFIED
|
||||
assert {Path(i["path"]).name: i["outcome"] for i in fetched["items"]} == {
|
||||
"new.jpg": "uploaded",
|
||||
"dup.jpg": "duplicate",
|
||||
}
|
||||
assert json.dumps(fetched) # the record stays JSON-serialisable for the UI
|
||||
|
||||
|
||||
def test_a_missing_report_file_does_not_lose_the_batch(tmp_path, immich_server):
|
||||
lib = tmp_path / "lib"
|
||||
report = _report_for(lib / "rome", **{"new.jpg": "uploaded", "dup.jpg": "duplicate"})
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, report))
|
||||
_album(sf, lib)
|
||||
batch = _run(sf, config)
|
||||
Path(batch["report_path"]).unlink()
|
||||
|
||||
result = UploadReportService(sf).ingest(batch["id"])
|
||||
|
||||
assert result["outcome_state"] == REQUIRES_VERIFICATION
|
||||
assert set(_outcomes(UploadBatchService(sf, config=config).get(batch["id"])).values()) == {
|
||||
report_parser.UNKNOWN
|
||||
}
|
||||
|
||||
|
||||
def test_ingesting_an_unknown_batch_is_an_error(tmp_path, immich_server):
|
||||
config, sf, lib = _env(tmp_path, immich_server, _uploader(tmp_path, ""))
|
||||
|
||||
with pytest.raises(KeyError):
|
||||
UploadReportService(sf).ingest("does-not-exist")
|
||||
537
tests/integration/test_upload_verification.py
Normal file
537
tests/integration/test_upload_verification.py
Normal file
@@ -0,0 +1,537 @@
|
||||
"""Verifying, retrying, and resolving uncertain uploads (US05-04).
|
||||
|
||||
The Immich boundary is a real HTTP server here: it answers ``/api/server/ping``
|
||||
and ``/api/assets/bulk-upload-check`` exactly as the app's own adapter parses
|
||||
them, and the set of checksums it "holds" is what each fault scenario controls.
|
||||
The uploader stays a real executable driven through the real batch service, so
|
||||
every state under test is reached the way production reaches it.
|
||||
|
||||
The invariant these tests defend is one-directional: uncertainty may only become
|
||||
success when something authoritative said so — the server, or an operator who
|
||||
recorded what they checked.
|
||||
"""
|
||||
|
||||
import json
|
||||
import stat
|
||||
import threading
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
from photo_pipeline.db import create_db_engine, create_session_factory, run_migrations
|
||||
from photo_pipeline.models import AnalysisResult, Asset, SafetyReview, UploadBatch
|
||||
from photo_pipeline.services.hashing import sha256_file
|
||||
from photo_pipeline.services.upload_batches import BatchError, BatchState, UploadBatchService
|
||||
from photo_pipeline.services.upload_reports import REQUIRES_VERIFICATION, VERIFIED
|
||||
from photo_pipeline.services.upload_verification import (
|
||||
ABSENT,
|
||||
INCONCLUSIVE,
|
||||
MANUAL,
|
||||
PRESENT,
|
||||
UploadVerificationService,
|
||||
VerificationError,
|
||||
retry_blockers,
|
||||
)
|
||||
from photo_pipeline.services.uploads import UploadService
|
||||
|
||||
pytestmark = pytest.mark.phase_e # part of the Phase E acceptance gate (US05-06)
|
||||
|
||||
NOW = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
SENTINEL_KEY = "immich-sentinel-9f3a2b"
|
||||
UPLOADER_VERSION = "immich-go 0.21.0"
|
||||
|
||||
|
||||
# ── fake Immich ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class _FakeImmich:
|
||||
"""The server's view of the world: which checksums it holds, and whether it
|
||||
is willing to answer at all."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.held: set[str] = set()
|
||||
self.available = True
|
||||
self.checked: list[str] = []
|
||||
|
||||
|
||||
def _handler(state: _FakeImmich):
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self): # noqa: N802 (BaseHTTPRequestHandler API)
|
||||
self._json(200, {"res": "pong"})
|
||||
|
||||
def do_POST(self): # noqa: N802
|
||||
body = json.loads(self.rfile.read(int(self.headers["Content-Length"] or 0)) or "{}")
|
||||
if not state.available:
|
||||
self._json(503, {"error": "unavailable"})
|
||||
return
|
||||
if self.headers.get("x-api-key") != SENTINEL_KEY:
|
||||
self._json(401, {"error": "unauthorized"})
|
||||
return
|
||||
results = []
|
||||
for asset in body.get("assets", []):
|
||||
state.checked.append(asset["checksum"])
|
||||
results.append(
|
||||
{"id": asset["id"], "action": "reject", "reason": "duplicate"}
|
||||
if asset["checksum"] in state.held
|
||||
else {"id": asset["id"], "action": "accept"}
|
||||
)
|
||||
self._json(200, {"results": results})
|
||||
|
||||
def _json(self, status: int, payload: dict) -> None:
|
||||
body = json.dumps(payload).encode()
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
return Handler
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def immich():
|
||||
state = _FakeImmich()
|
||||
server = HTTPServer(("127.0.0.1", 0), _handler(state))
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
state.url = f"http://127.0.0.1:{server.server_port}"
|
||||
yield state
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
|
||||
# ── environment ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _uploader(tmp_path, report_body: str = "", *, exit_code: int = 0, version=UPLOADER_VERSION):
|
||||
path = tmp_path / "immich-go"
|
||||
path.write_text(
|
||||
"#!/bin/sh\n"
|
||||
f'if [ "$1" = "--version" ]; then echo "{version}"; exit 0; fi\n'
|
||||
f"cat <<'REPORT'\n{report_body}\nREPORT\n"
|
||||
f"exit {exit_code}\n"
|
||||
)
|
||||
path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
|
||||
return path
|
||||
|
||||
|
||||
def _env(tmp_path, immich, uploader=None):
|
||||
(tmp_path / "data").mkdir(exist_ok=True)
|
||||
lib = tmp_path / "lib"
|
||||
lib.mkdir(exist_ok=True)
|
||||
config = Config.from_env(
|
||||
{
|
||||
"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"),
|
||||
"PHOTO_PIPELINE_LIBRARY_ROOTS": str(lib),
|
||||
"PHOTO_PIPELINE_IMMICH_SERVER_URL": immich.url,
|
||||
"PHOTO_PIPELINE_IMMICH_API_KEY": SENTINEL_KEY,
|
||||
"PHOTO_PIPELINE_IMMICH_GO_BINARY": str(
|
||||
uploader if uploader is not None else _uploader(tmp_path)
|
||||
),
|
||||
}
|
||||
)
|
||||
run_migrations(config.database_url)
|
||||
return config, create_session_factory(create_db_engine(config.database_url)), lib
|
||||
|
||||
|
||||
def _album(sf, lib, album="rome", names=("a.jpg", "b.jpg")):
|
||||
folder = lib / album
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
with sf() as session:
|
||||
for name in names:
|
||||
path = folder / name
|
||||
path.write_bytes(f"{album}/{name}".encode() * 16)
|
||||
asset_id = str(uuid.uuid4())
|
||||
session.add(
|
||||
Asset(
|
||||
id=asset_id,
|
||||
original_path=str(path),
|
||||
current_path=str(path),
|
||||
discovered_at=NOW,
|
||||
hash_version=1,
|
||||
byte_size=path.stat().st_size,
|
||||
current_sha256=sha256_file(path),
|
||||
)
|
||||
)
|
||||
session.add(
|
||||
SafetyReview(
|
||||
id=str(uuid.uuid4()), asset_id=asset_id, decision="sfw", exif_verified_at=NOW
|
||||
)
|
||||
)
|
||||
session.add(AnalysisResult(asset_id=asset_id, status="analyzed", exif_written_at=NOW))
|
||||
session.commit()
|
||||
return folder
|
||||
|
||||
|
||||
def _batch(sf, config, albums=None):
|
||||
report = UploadService(sf, config=config).preflight(albums)
|
||||
assert report["state"] == "ready", report["blockers"]
|
||||
(batch,) = UploadBatchService(sf, config=config).create(albums, token=report["token"])
|
||||
return batch
|
||||
|
||||
|
||||
def _interrupt(sf, config, batch_id):
|
||||
"""Leave behind exactly what a killed worker leaves: a mid-flight attempt."""
|
||||
with sf() as session:
|
||||
session.get(UploadBatch, batch_id).state = BatchState.RUNNING
|
||||
session.commit()
|
||||
UploadBatchService(sf, config=config).recover()
|
||||
|
||||
|
||||
def _server_holds(immich, batch):
|
||||
immich.held.update(item["sha1"] for item in batch["items"])
|
||||
|
||||
|
||||
def _by_name(batch) -> dict:
|
||||
return {Path(item["path"]).name: item for item in batch["items"]}
|
||||
|
||||
|
||||
def _uncertain_batch(tmp_path, immich, *, hold: bool):
|
||||
"""A batch whose attempt died mid-upload, with the server holding the bytes or not."""
|
||||
config, sf, lib = _env(tmp_path, immich)
|
||||
_album(sf, lib)
|
||||
batch = _batch(sf, config)
|
||||
if hold:
|
||||
_server_holds(immich, batch)
|
||||
_interrupt(sf, config, batch["id"])
|
||||
return config, sf, lib, batch
|
||||
|
||||
|
||||
# ── accepted-but-unknown ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_acceptance_then_lost_response_is_confirmed_by_the_server(tmp_path, immich):
|
||||
"""The classic lost response: Immich took the files, the app never saw a report."""
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
|
||||
result = UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
assert result["state"] == BatchState.SUCCEEDED, "server evidence resolves the uncertainty"
|
||||
assert result["outcome_state"] == VERIFIED
|
||||
assert result["counts"] == {PRESENT: 2}
|
||||
verified = UploadBatchService(sf, config=config).get(batch["id"])
|
||||
for item in verified["items"]:
|
||||
assert item["outcome"] == "uploaded"
|
||||
assert item["verification"] == PRESENT
|
||||
assert item["sha1"] in item["evidence"], "the exact bytes are named in the evidence"
|
||||
assert immich.checked, "verification actually asked the server"
|
||||
|
||||
|
||||
def test_timeout_before_acceptance_leaves_a_safe_retry(tmp_path, immich):
|
||||
"""Nothing arrived, so the batch becomes a plain failure and may run again."""
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=False)
|
||||
service = UploadBatchService(sf, config=config)
|
||||
|
||||
result = UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
assert result["state"] == BatchState.FAILED
|
||||
assert result["counts"] == {ABSENT: 2}
|
||||
assert {item["outcome"] for item in service.get(batch["id"])["items"]} == {"failed"}
|
||||
assert retry_blockers(service.get(batch["id"])) == []
|
||||
assert service.run(batch["id"])["state"] == BatchState.SUCCEEDED
|
||||
|
||||
|
||||
def test_an_uncertain_batch_cannot_be_retried_before_verification(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
service = UploadBatchService(sf, config=config)
|
||||
|
||||
with pytest.raises(BatchError) as error:
|
||||
service.run(batch["id"])
|
||||
|
||||
assert error.value.code == "requires_verification"
|
||||
with TestClient(create_app(config)) as client:
|
||||
response = client.post(f"/api/v1/upload-batches/{batch['id']}/start")
|
||||
assert response.status_code == 409
|
||||
assert response.json()["error"]["code"] == "requires_verification"
|
||||
|
||||
|
||||
def test_a_partly_arrived_batch_is_a_failure_not_a_success(tmp_path, immich):
|
||||
config, sf, lib = _env(tmp_path, immich)
|
||||
_album(sf, lib)
|
||||
batch = _batch(sf, config)
|
||||
immich.held.add(batch["items"][0]["sha1"]) # only one file made it
|
||||
_interrupt(sf, config, batch["id"])
|
||||
|
||||
result = UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
assert result["state"] == BatchState.FAILED
|
||||
assert result["counts"] == {PRESENT: 1, ABSENT: 1}
|
||||
assert result["outcome_state"] == VERIFIED, "every file is accounted for"
|
||||
|
||||
|
||||
# ── parser uncertainty ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_parser_uncertainty_is_settled_by_server_evidence(tmp_path, immich):
|
||||
"""An unpinned uploader version leaves every item unknown; the server decides."""
|
||||
config, sf, lib = _env(tmp_path, immich, _uploader(tmp_path, "done", version="immich-go 9.9.9"))
|
||||
_album(sf, lib)
|
||||
batch = _batch(sf, config)
|
||||
service = UploadBatchService(sf, config=config)
|
||||
ran = service.run(batch["id"])
|
||||
assert ran["state"] == BatchState.SUCCEEDED and ran["outcome_state"] == REQUIRES_VERIFICATION
|
||||
_server_holds(immich, ran)
|
||||
|
||||
result = UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
assert result["outcome_state"] == VERIFIED
|
||||
assert {item["outcome"] for item in service.get(batch["id"])["items"]} == {"uploaded"}
|
||||
|
||||
|
||||
def test_an_unreachable_server_never_turns_uncertainty_into_success(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
immich.available = False
|
||||
service = UploadBatchService(sf, config=config)
|
||||
|
||||
result = UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
assert result["server_reachable"] is False
|
||||
assert result["detail"], "the reason the server could not answer is reported"
|
||||
assert result["counts"] == {INCONCLUSIVE: 2}
|
||||
assert result["state"] == BatchState.UNKNOWN, "still uncertain, not succeeded"
|
||||
assert result["outcome_state"] == REQUIRES_VERIFICATION
|
||||
assert [b["code"] for b in retry_blockers(service.get(batch["id"]))] == [
|
||||
"requires_verification"
|
||||
]
|
||||
|
||||
|
||||
# ── safe failures ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_plain_uploader_failure_is_retryable_without_verification(tmp_path, immich):
|
||||
"""Nothing uncertain happened: the process failed before/while reporting an error."""
|
||||
config, sf, lib = _env(tmp_path, immich, _uploader(tmp_path, "boom", exit_code=1))
|
||||
_album(sf, lib)
|
||||
batch = _batch(sf, config)
|
||||
service = UploadBatchService(sf, config=config)
|
||||
assert service.run(batch["id"])["state"] == BatchState.FAILED
|
||||
|
||||
assert retry_blockers(service.get(batch["id"])) == []
|
||||
with TestClient(create_app(config)) as client:
|
||||
assert client.post(f"/api/v1/upload-batches/{batch['id']}/start").status_code == 200
|
||||
|
||||
|
||||
# ── changed bytes ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_bytes_changed_after_upload_warn_and_block_a_rerun(tmp_path, immich):
|
||||
config, sf, lib = _env(tmp_path, immich)
|
||||
folder = _album(sf, lib)
|
||||
batch = _batch(sf, config)
|
||||
service = UploadBatchService(sf, config=config)
|
||||
ran = service.run(batch["id"])
|
||||
_server_holds(immich, ran)
|
||||
(folder / "a.jpg").write_bytes(b"edited after the upload")
|
||||
|
||||
result = UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
assert result["stale_bytes"] is True
|
||||
changed = _by_name(result)["a.jpg"]
|
||||
assert changed["changed_after_upload"] is True
|
||||
assert _by_name(result)["b.jpg"]["changed_after_upload"] is False
|
||||
stored = service.get(batch["id"])
|
||||
assert stored["stale_bytes"] is True, "the warning is durable, not only in the response"
|
||||
assert _by_name(stored)["a.jpg"]["observed_sha256"] != _by_name(stored)["a.jpg"]["sha256"]
|
||||
with pytest.raises(BatchError) as error:
|
||||
service.run(batch["id"])
|
||||
assert error.value.code == "changed_after_upload"
|
||||
with TestClient(create_app(config)) as client:
|
||||
response = client.post(f"/api/v1/upload-batches/{batch['id']}/start")
|
||||
assert response.status_code == 409
|
||||
assert response.json()["error"]["code"] == "changed_after_upload"
|
||||
|
||||
|
||||
def test_a_deleted_file_counts_as_changed_after_upload(tmp_path, immich):
|
||||
config, sf, lib = _env(tmp_path, immich)
|
||||
folder = _album(sf, lib)
|
||||
batch = _batch(sf, config)
|
||||
ran = UploadBatchService(sf, config=config).run(batch["id"])
|
||||
_server_holds(immich, ran)
|
||||
(folder / "a.jpg").unlink()
|
||||
|
||||
result = UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
assert _by_name(result)["a.jpg"]["changed_after_upload"] is True
|
||||
assert result["stale_bytes"] is True
|
||||
# The bytes are still on the server: verification is about the upload, not the
|
||||
# local file's continued existence.
|
||||
assert _by_name(result)["a.jpg"]["verification"] == PRESENT
|
||||
|
||||
|
||||
# ── repeated verification and history ────────────────────────────────────────
|
||||
|
||||
|
||||
def test_repeated_verification_converges_and_keeps_every_answer(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
service = UploadVerificationService(sf, config=config)
|
||||
|
||||
first = service.verify(batch["id"])
|
||||
second = service.verify(batch["id"])
|
||||
|
||||
assert first["counts"] == second["counts"] == {PRESENT: 2}
|
||||
assert first["state"] == second["state"] == BatchState.SUCCEEDED
|
||||
history = service.history(batch["id"])
|
||||
assert len(history) == 4, "the audit trail appends, it never overwrites"
|
||||
assert {entry["source"] for entry in history} == {"immich_api"}
|
||||
assert all(entry["action"] == "verify" for entry in history)
|
||||
|
||||
|
||||
def test_verification_that_changes_its_mind_keeps_both_answers(tmp_path, immich):
|
||||
"""A file that was absent and later present must show both, in order."""
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=False)
|
||||
service = UploadVerificationService(sf, config=config)
|
||||
service.verify(batch["id"])
|
||||
_server_holds(immich, batch)
|
||||
|
||||
service.verify(batch["id"])
|
||||
|
||||
asset_id = batch["items"][0]["asset_id"]
|
||||
results = [e["result"] for e in service.history(batch["id"]) if e["asset_id"] == asset_id]
|
||||
assert results == [ABSENT, PRESENT]
|
||||
|
||||
|
||||
# ── manual resolution ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_manual_resolution_requires_evidence_and_an_author(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
service = UploadVerificationService(sf, config=config)
|
||||
asset_id = batch["items"][0]["asset_id"]
|
||||
|
||||
for kwargs, code in (
|
||||
({"evidence": " ", "actor": "dom"}, "evidence_required"),
|
||||
({"evidence": "checked in Immich", "actor": ""}, "actor_required"),
|
||||
({"evidence": "checked", "actor": "dom", "outcome": "definitely-fine"}, "invalid_outcome"),
|
||||
):
|
||||
with pytest.raises(VerificationError) as error:
|
||||
service.resolve(batch["id"], asset_id, **{"outcome": "uploaded", **kwargs})
|
||||
assert error.value.code == code
|
||||
|
||||
assert service.history(batch["id"]) == [], "a refused resolution records nothing"
|
||||
assert UploadBatchService(sf, config=config).get(batch["id"])["state"] == BatchState.UNKNOWN
|
||||
|
||||
|
||||
def test_manual_resolution_is_recorded_as_operator_evidence(tmp_path, immich):
|
||||
"""An operator may settle what the server cannot — but never anonymously."""
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
immich.available = False
|
||||
service = UploadVerificationService(sf, config=config)
|
||||
service.verify(batch["id"]) # inconclusive: the server is down
|
||||
|
||||
for item in batch["items"]:
|
||||
result = service.resolve(
|
||||
batch["id"],
|
||||
item["asset_id"],
|
||||
outcome="uploaded",
|
||||
evidence="found in Immich by checksum in the web UI",
|
||||
actor="dom",
|
||||
)
|
||||
|
||||
assert result["state"] == BatchState.SUCCEEDED
|
||||
assert result["outcome_state"] == VERIFIED
|
||||
resolutions = [e for e in service.history(batch["id"]) if e["action"] == "resolve"]
|
||||
assert len(resolutions) == 2
|
||||
assert {e["actor"] for e in resolutions} == {"dom"}
|
||||
assert {e["source"] for e in resolutions} == {"operator"}
|
||||
assert all("web UI" in e["evidence"] for e in resolutions)
|
||||
stored = UploadBatchService(sf, config=config).get(batch["id"])
|
||||
assert {item["verification"] for item in stored["items"]} == {MANUAL}, (
|
||||
"a manual answer stays distinguishable from server evidence"
|
||||
)
|
||||
|
||||
|
||||
def test_resolving_one_item_leaves_the_batch_uncertain(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
immich.available = False
|
||||
service = UploadVerificationService(sf, config=config)
|
||||
service.verify(batch["id"])
|
||||
|
||||
result = service.resolve(
|
||||
batch["id"],
|
||||
batch["items"][0]["asset_id"],
|
||||
outcome="uploaded",
|
||||
evidence="visible in Immich",
|
||||
actor="dom",
|
||||
)
|
||||
|
||||
assert result["state"] == BatchState.UNKNOWN
|
||||
assert result["outcome_state"] == REQUIRES_VERIFICATION
|
||||
|
||||
|
||||
def test_resolving_an_unknown_item_or_batch_is_refused(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
service = UploadVerificationService(sf, config=config)
|
||||
|
||||
for batch_id, asset_id in ((batch["id"], "not-in-this-batch"), ("no-such-batch", "x")):
|
||||
with pytest.raises(VerificationError) as error:
|
||||
service.resolve(batch_id, asset_id, outcome="uploaded", evidence="checked", actor="dom")
|
||||
assert error.value.code == "not_found"
|
||||
|
||||
|
||||
def test_verifying_an_unknown_batch_is_refused(tmp_path, immich):
|
||||
config, sf, lib = _env(tmp_path, immich)
|
||||
|
||||
with pytest.raises(VerificationError) as error:
|
||||
UploadVerificationService(sf, config=config).verify("does-not-exist")
|
||||
|
||||
assert error.value.code == "not_found"
|
||||
|
||||
|
||||
# ── API surface and durability ───────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_api_verifies_resolves_and_lists_history_without_secrets(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=False)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
verified = client.post(f"/api/v1/upload-batches/{batch['id']}/verify")
|
||||
resolved = client.post(
|
||||
f"/api/v1/upload-batches/{batch['id']}/resolve",
|
||||
json={
|
||||
"asset_id": batch["items"][0]["asset_id"],
|
||||
"outcome": "skipped",
|
||||
"evidence": "the file was withdrawn from the album",
|
||||
"actor": "dom",
|
||||
},
|
||||
)
|
||||
refused = client.post(
|
||||
f"/api/v1/upload-batches/{batch['id']}/resolve",
|
||||
json={
|
||||
"asset_id": batch["items"][0]["asset_id"],
|
||||
"outcome": "uploaded",
|
||||
"evidence": "",
|
||||
"actor": "dom",
|
||||
},
|
||||
)
|
||||
history = client.get(f"/api/v1/upload-batches/{batch['id']}/verifications")
|
||||
missing = client.post("/api/v1/upload-batches/does-not-exist/verify")
|
||||
|
||||
assert verified.status_code == 200 and verified.json()["counts"] == {ABSENT: 2}
|
||||
assert resolved.status_code == 200
|
||||
assert refused.status_code == 422 and refused.json()["error"]["code"] == "evidence_required"
|
||||
assert len(history.json()["verifications"]) == 3
|
||||
assert missing.status_code == 404
|
||||
assert SENTINEL_KEY not in verified.text + resolved.text + history.text
|
||||
|
||||
|
||||
def test_verification_survives_a_restart(tmp_path, immich):
|
||||
config, sf, lib, batch = _uncertain_batch(tmp_path, immich, hold=True)
|
||||
UploadVerificationService(sf, config=config).verify(batch["id"])
|
||||
|
||||
with TestClient(create_app(config)) as client: # a fresh application process
|
||||
fetched = client.get(f"/api/v1/upload-batches/{batch['id']}").json()
|
||||
history = client.get(f"/api/v1/upload-batches/{batch['id']}/verifications").json()
|
||||
|
||||
assert fetched["state"] == BatchState.SUCCEEDED
|
||||
assert fetched["verified_at"]
|
||||
assert {item["verification"] for item in fetched["items"]} == {PRESENT}
|
||||
assert len(history["verifications"]) == 2
|
||||
assert json.dumps(fetched) # the record stays JSON-serialisable for the UI
|
||||
@@ -90,6 +90,26 @@ def test_cooperative_cancellation_leaves_items_resumable(sf, jobs):
|
||||
assert by_state.get(ItemState.QUEUED) == 1 # "b" left resumable
|
||||
|
||||
|
||||
def test_a_handler_that_stops_itself_releases_the_lock(sf, jobs):
|
||||
"""A handler may stop without anyone cancelling the *job* — an upload batch
|
||||
cancelled through its own API does exactly that. The job is still ``running``
|
||||
when it raises, so it has to reach ``cancelled`` through ``cancelling``; if that
|
||||
hop is skipped the transition is rejected and the lock is held forever."""
|
||||
from photo_pipeline.jobs.handlers import Cancelled
|
||||
|
||||
def handler(item, ctx):
|
||||
raise Cancelled("the work this job wraps was stopped elsewhere")
|
||||
|
||||
worker = Worker(sf, {"scan": handler}, "w1")
|
||||
job = jobs.enqueue("scan", lock="library_write", items=["a"])
|
||||
worker.run_once()
|
||||
|
||||
assert jobs.get(job["id"])["state"] == JobState.CANCELLED
|
||||
assert jobs.progress(job["id"])["by_state"] == {ItemState.QUEUED: 1} # resumable
|
||||
# The lane is free: the next job may be enqueued under the same lock.
|
||||
assert jobs.enqueue("scan", lock="library_write", items=["b"])["state"] == JobState.QUEUED
|
||||
|
||||
|
||||
def test_fencing_rejects_superseded_worker(sf, jobs):
|
||||
job = jobs.enqueue("scan", items=["a"])
|
||||
stale = jobs.claim(["scan"], "old")
|
||||
|
||||
@@ -109,6 +109,44 @@
|
||||
],
|
||||
"US05-02": [
|
||||
"tests/integration/test_upload_batches.py"
|
||||
],
|
||||
"US05-03": [
|
||||
"tests/unit/test_immich_go_report.py",
|
||||
"tests/integration/test_upload_reports.py"
|
||||
],
|
||||
"US05-04": [
|
||||
"tests/unit/test_immich_bulk_check.py",
|
||||
"tests/integration/test_upload_verification.py"
|
||||
],
|
||||
"US05-05": [
|
||||
"tests/e2e/test_uploads_ui.py"
|
||||
],
|
||||
"US05-06": [
|
||||
"tests/e2e/test_phase_e_pipeline.py"
|
||||
],
|
||||
"US06-01": [
|
||||
"tests/integration/test_archive_preflight.py"
|
||||
],
|
||||
"US06-02": [
|
||||
"tests/unit/test_archive_journal_states.py",
|
||||
"tests/integration/test_archive_transfer.py",
|
||||
"tests/integration/test_archive_recovery.py"
|
||||
],
|
||||
"US06-03": [
|
||||
"tests/integration/test_offline_assets.py"
|
||||
],
|
||||
"US06-04": [
|
||||
"tests/integration/test_restore.py"
|
||||
],
|
||||
"US06-05": [
|
||||
"tests/e2e/test_archive_ui.py"
|
||||
],
|
||||
"US06-06": [
|
||||
"tests/e2e/test_phase_f_pipeline.py"
|
||||
],
|
||||
"US07-01": [
|
||||
"tests/unit/test_legacy_archive.py",
|
||||
"tests/integration/test_legacy_import.py"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
105
tests/unit/test_archive_journal_states.py
Normal file
105
tests/unit/test_archive_journal_states.py
Normal file
@@ -0,0 +1,105 @@
|
||||
"""Archive journal state machine and evidence table (US06-02).
|
||||
|
||||
The evidence table decides whether an original may be deleted, so every
|
||||
combination of journal state and disk reality is asserted here as a pure function —
|
||||
no database, no files. A wrong cell in this table is data loss.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from photo_pipeline.services.archive_journal import (
|
||||
ALLOWED_TRANSITIONS,
|
||||
FORWARD,
|
||||
MANUAL,
|
||||
RESUMABLE,
|
||||
TERMINAL_STATES,
|
||||
UNSAFE_STATES,
|
||||
ArchiveState,
|
||||
_classify,
|
||||
can_transition,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.phase_f
|
||||
|
||||
|
||||
# ── state machine ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_happy_path_is_the_only_way_forward():
|
||||
assert can_transition(ArchiveState.PLANNED, ArchiveState.TRANSFERRING)
|
||||
assert can_transition(ArchiveState.TRANSFERRING, ArchiveState.VERIFIED)
|
||||
assert can_transition(ArchiveState.VERIFIED, ArchiveState.REMOVING)
|
||||
assert can_transition(ArchiveState.REMOVING, ArchiveState.COMPLETE)
|
||||
# No shortcut may skip verification before a source is removed.
|
||||
assert not can_transition(ArchiveState.TRANSFERRING, ArchiveState.REMOVING)
|
||||
assert not can_transition(ArchiveState.PLANNED, ArchiveState.VERIFIED)
|
||||
assert not can_transition(ArchiveState.VERIFIED, ArchiveState.COMPLETE)
|
||||
|
||||
|
||||
def test_removal_never_goes_backwards():
|
||||
"""Once the source may be gone, retrying the transfer would archive nothing and
|
||||
could overwrite the copy that is now the only one."""
|
||||
assert ALLOWED_TRANSITIONS[ArchiveState.REMOVING] == {
|
||||
ArchiveState.COMPLETE,
|
||||
ArchiveState.FAILED,
|
||||
}
|
||||
assert not can_transition(ArchiveState.REMOVING, ArchiveState.TRANSFERRING)
|
||||
assert not can_transition(ArchiveState.REMOVING, ArchiveState.PLANNED)
|
||||
|
||||
|
||||
def test_complete_is_terminal_and_failed_can_be_retried():
|
||||
assert ALLOWED_TRANSITIONS[ArchiveState.COMPLETE] == set()
|
||||
assert TERMINAL_STATES == {ArchiveState.COMPLETE}
|
||||
assert can_transition(ArchiveState.FAILED, ArchiveState.TRANSFERRING)
|
||||
assert can_transition(ArchiveState.FAILED, ArchiveState.PLANNED)
|
||||
|
||||
|
||||
def test_every_state_that_can_touch_the_disk_is_marked_unsafe():
|
||||
assert UNSAFE_STATES == {
|
||||
ArchiveState.TRANSFERRING,
|
||||
ArchiveState.VERIFIED,
|
||||
ArchiveState.REMOVING,
|
||||
}
|
||||
assert ArchiveState.PLANNED not in UNSAFE_STATES
|
||||
|
||||
|
||||
# ── evidence table ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"state,source,destination,matches,expected",
|
||||
[
|
||||
# Nothing published yet: the source is still the only copy.
|
||||
(ArchiveState.TRANSFERRING, True, False, False, RESUMABLE),
|
||||
(ArchiveState.FAILED, True, False, False, RESUMABLE),
|
||||
# The archive copy is durable and correct: finish the remaining steps.
|
||||
(ArchiveState.TRANSFERRING, True, True, True, FORWARD),
|
||||
(ArchiveState.TRANSFERRING, False, True, True, FORWARD),
|
||||
(ArchiveState.VERIFIED, True, True, True, FORWARD),
|
||||
(ArchiveState.REMOVING, False, True, True, FORWARD),
|
||||
(ArchiveState.FAILED, True, True, True, FORWARD),
|
||||
# Wrong bytes at the destination: never overwrite, never remove.
|
||||
(ArchiveState.TRANSFERRING, True, True, False, MANUAL),
|
||||
(ArchiveState.VERIFIED, True, True, False, MANUAL),
|
||||
(ArchiveState.REMOVING, False, True, False, MANUAL),
|
||||
# The journal claims an archived copy that is not there.
|
||||
(ArchiveState.VERIFIED, True, False, False, MANUAL),
|
||||
(ArchiveState.REMOVING, False, False, False, MANUAL),
|
||||
# Neither copy exists — never silently accepted as success.
|
||||
(ArchiveState.TRANSFERRING, False, False, False, MANUAL),
|
||||
(ArchiveState.FAILED, False, False, False, MANUAL),
|
||||
],
|
||||
)
|
||||
def test_classification_of_every_evidence_combination(
|
||||
state, source, destination, matches, expected
|
||||
):
|
||||
classification, reason = _classify(state, source, destination, matches)
|
||||
|
||||
assert classification == expected, reason
|
||||
|
||||
|
||||
def test_a_source_that_is_gone_without_an_archive_copy_is_never_called_recoverable():
|
||||
"""The one combination that must always stop: the original left active storage
|
||||
and nothing verifiable took its place."""
|
||||
for state in (ArchiveState.TRANSFERRING, ArchiveState.VERIFIED, ArchiveState.REMOVING):
|
||||
assert _classify(state, False, False, False)[0] == MANUAL
|
||||
27
tests/unit/test_immich_bulk_check.py
Normal file
27
tests/unit/test_immich_bulk_check.py
Normal file
@@ -0,0 +1,27 @@
|
||||
"""How one bulk-upload-check result is read (US05-04).
|
||||
|
||||
Only a duplicate rejection proves Immich holds the bytes. Every other answer — a
|
||||
rejection for another reason, an action this adapter does not know — must stay
|
||||
uncertain, because "the server did not say yes" is not "the file is not there".
|
||||
"""
|
||||
|
||||
from photo_pipeline.integrations.immich_go import _holds_bytes, bulk_upload_check
|
||||
|
||||
|
||||
def test_a_duplicate_rejection_is_the_only_proof_of_possession():
|
||||
assert _holds_bytes({"action": "reject", "reason": "duplicate"}) is True
|
||||
assert _holds_bytes({"action": "accept"}) is False
|
||||
|
||||
|
||||
def test_any_other_answer_is_uncertain_never_absent():
|
||||
assert _holds_bytes({"action": "reject", "reason": "unsupported-format"}) is None
|
||||
assert _holds_bytes({"action": "quarantine"}) is None
|
||||
assert _holds_bytes({}) is None
|
||||
|
||||
|
||||
def test_missing_credentials_are_reported_not_silently_treated_as_absence():
|
||||
result = bulk_upload_check("", None, {"asset": "abc"})
|
||||
|
||||
assert result["reachable"] is False
|
||||
assert result["present"] == {}
|
||||
assert "credentials" in result["detail"]
|
||||
139
tests/unit/test_immich_go_report.py
Normal file
139
tests/unit/test_immich_go_report.py
Normal file
@@ -0,0 +1,139 @@
|
||||
"""Golden parser fixtures for immich-go reports (US05-03).
|
||||
|
||||
The reports below are the pinned grammar for each supported uploader version. If a
|
||||
future immich-go changes its wording, the fix is a new adapter and a new golden
|
||||
report — never a looser pattern here, because a loose pattern is how an unread line
|
||||
becomes a false success.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from photo_pipeline.integrations import immich_go_report as parser
|
||||
|
||||
TEXT_REPORT = """
|
||||
Scanning /lib/rome
|
||||
INFO uploaded /lib/rome/new.jpg
|
||||
INFO server has the same file /lib/rome/dup.jpg
|
||||
INFO server has an older file, upgrading /lib/rome/old.jpg
|
||||
WARN discarded /lib/rome/notes.txt: unsupported file type
|
||||
ERROR error uploading /lib/rome/broken.jpg: connection reset by peer
|
||||
Uploaded 1, upgraded 1, duplicates 1, skipped 1, errors 1
|
||||
"""
|
||||
|
||||
JSON_REPORT = """
|
||||
{"time":"2026-01-01T10:00:00Z","level":"INFO","msg":"uploaded","file":"/lib/rome/new.jpg"}
|
||||
{"level":"INFO","msg":"server has the same file","file":"/lib/rome/dup.jpg"}
|
||||
{"level":"INFO","msg":"server has an older file, upgrading","file":"/lib/rome/old.jpg"}
|
||||
{"level":"WARN","msg":"discarded: unsupported file type","file":"/lib/rome/notes.txt"}
|
||||
{"level":"ERROR","msg":"error uploading","file":"/lib/rome/broken.jpg"}
|
||||
{"level":"INFO","msg":"report","counts":{"uploaded":1,"upgraded":1,"duplicates":1,\
|
||||
"skipped":1,"errors":1}}
|
||||
"""
|
||||
|
||||
EXPECTED = {
|
||||
"/lib/rome/new.jpg": parser.UPLOADED,
|
||||
"/lib/rome/dup.jpg": parser.DUPLICATE,
|
||||
"/lib/rome/old.jpg": parser.UPGRADED,
|
||||
"/lib/rome/notes.txt": parser.SKIPPED,
|
||||
"/lib/rome/broken.jpg": parser.FAILED,
|
||||
}
|
||||
EXPECTED_COUNTS = {
|
||||
parser.UPLOADED: 1,
|
||||
parser.UPGRADED: 1,
|
||||
parser.DUPLICATE: 1,
|
||||
parser.SKIPPED: 1,
|
||||
parser.FAILED: 1,
|
||||
}
|
||||
|
||||
|
||||
def _outcomes(parsed) -> dict:
|
||||
return {entry["path"]: entry["outcome"] for entry in parsed["entries"]}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("version", "report", "expected_parser"),
|
||||
[
|
||||
("immich-go 0.21.0", TEXT_REPORT, "text-v1"),
|
||||
("immich-go 0.22.3", TEXT_REPORT, "text-v1"),
|
||||
("immich-go 0.23.1", JSON_REPORT, "json-v1"),
|
||||
("immich-go 0.24.0", JSON_REPORT, "json-v1"),
|
||||
],
|
||||
)
|
||||
def test_every_supported_version_classifies_every_outcome(version, report, expected_parser):
|
||||
parsed = parser.parse(report, version)
|
||||
|
||||
assert parsed["parser"] == expected_parser and parsed["supported"]
|
||||
assert parsed["parser_version"] == parser.PARSER_VERSION
|
||||
assert _outcomes(parsed) == EXPECTED
|
||||
assert parsed["counts"] == EXPECTED_COUNTS
|
||||
|
||||
|
||||
def test_an_unsupported_version_parses_nothing_at_all():
|
||||
"""A build whose output was never pinned must not be read optimistically."""
|
||||
parsed = parser.parse(TEXT_REPORT, "immich-go 9.99.0")
|
||||
|
||||
assert parsed["parser"] is None and parsed["supported"] is False
|
||||
assert parsed["entries"] == [] and parsed["counts"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version", [None, "", "immich-go dev", "unknown"])
|
||||
def test_a_missing_or_unreadable_version_is_unsupported(version):
|
||||
assert parser.parser_for(version) is None
|
||||
|
||||
|
||||
def test_malformed_lines_are_counted_never_classified():
|
||||
report = (
|
||||
"{not json at all\n"
|
||||
'{"level":"INFO","msg":"uploaded"}\n' # no file
|
||||
'["not","an","object"]\n'
|
||||
'{"level":"INFO","msg":"uploaded","file":"/lib/rome/new.jpg"}\n'
|
||||
)
|
||||
|
||||
parsed = parser.parse(report, "immich-go 0.23.1")
|
||||
|
||||
assert _outcomes(parsed) == {"/lib/rome/new.jpg": parser.UPLOADED}
|
||||
assert parsed["unparsed"] == 3
|
||||
|
||||
|
||||
def test_a_line_with_a_path_but_no_known_verb_is_not_an_outcome():
|
||||
parsed = parser.parse("INFO considering /lib/rome/a.jpg\n", "immich-go 0.21.0")
|
||||
|
||||
assert parsed["entries"] == [] and parsed["unparsed"] == 1
|
||||
|
||||
|
||||
def test_the_album_name_never_classifies_the_line():
|
||||
"""A folder called "errors" must not turn a successful upload into a failure."""
|
||||
parsed = parser.parse("INFO uploaded /lib/errors/a.jpg\n", "immich-go 0.21.0")
|
||||
|
||||
assert _outcomes(parsed) == {"/lib/errors/a.jpg": parser.UPLOADED}
|
||||
|
||||
|
||||
def test_paths_with_spaces_and_reasons_are_read_whole():
|
||||
report = "ERROR error uploading /lib/summer holiday/a b.jpg: connection reset\n"
|
||||
|
||||
parsed = parser.parse(report, "immich-go 0.21.0")
|
||||
|
||||
assert _outcomes(parsed) == {"/lib/summer holiday/a b.jpg": parser.FAILED}
|
||||
|
||||
|
||||
def test_evidence_is_bounded_per_entry():
|
||||
line = "INFO uploaded /lib/rome/a.jpg " + "x" * 5_000
|
||||
|
||||
(entry,) = parser.parse(line, "immich-go 0.21.0")["entries"]
|
||||
|
||||
assert len(entry["evidence"]) == parser.MAX_EVIDENCE_CHARS
|
||||
|
||||
|
||||
def test_a_looping_uploader_cannot_produce_unbounded_entries(monkeypatch):
|
||||
monkeypatch.setattr(parser, "MAX_ENTRIES", 10)
|
||||
report = "".join(f"INFO uploaded /lib/rome/{n}.jpg\n" for n in range(50))
|
||||
|
||||
parsed = parser.parse(report, "immich-go 0.21.0")
|
||||
|
||||
assert len(parsed["entries"]) == 10 and parsed["entries_truncated"] is True
|
||||
|
||||
|
||||
def test_a_report_without_a_summary_reports_no_counts():
|
||||
parsed = parser.parse("INFO uploaded /lib/rome/a.jpg\n", "immich-go 0.21.0")
|
||||
|
||||
assert parsed["counts"] is None, "counts are read, never invented"
|
||||
173
tests/unit/test_legacy_archive.py
Normal file
173
tests/unit/test_legacy_archive.py
Normal file
@@ -0,0 +1,173 @@
|
||||
"""Archive lint (US07-01): the frozen CLI archive is complete, honest, and inert.
|
||||
|
||||
Freezing the donors is only worth something if three things stay true: the archive
|
||||
still holds exactly the bytes it claims, it holds no secret, and production cannot
|
||||
reach it. Each of those is a one-line mistake away — a helpful `sys.path` insert, a
|
||||
copied `.env`, an edited "just this once" source — so each is asserted here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
ARCHIVE = REPO / "legacy_cli_archive"
|
||||
SOURCES = ARCHIVE / "src"
|
||||
LEDGER = ARCHIVE / "donor_ledger.yaml"
|
||||
CHECKSUMS = ARCHIVE / "CHECKSUMS.sha256"
|
||||
|
||||
# Everything the concept requires an archive to carry (§3 "Donor-first CLI
|
||||
# migration and archival"): the sources, their docs, a dependency lock, schema
|
||||
# notes, a redacted sample configuration, the ledger, and recorded checksums.
|
||||
REQUIRED_ARTIFACTS = (
|
||||
"README.md",
|
||||
"donor_ledger.yaml",
|
||||
"CHECKSUMS.sha256",
|
||||
"requirements-lock.txt",
|
||||
"photo_analyzer.env.sample",
|
||||
)
|
||||
ARCHIVED_MODULES = ("photo_analyzer", "nsfwtag", "webapp", "nsfw_tag", "compare_models")
|
||||
# Only the two suites that compare against the donors may put the archive on the
|
||||
# import path; every other tree must not name it at all.
|
||||
IMPORT_ALLOWED = {
|
||||
REPO / "tests" / "characterization" / "conftest.py",
|
||||
REPO / "tests" / "characterization" / "test_donor_ledger.py", # lints the ledger there
|
||||
REPO / "tests" / "integration" / "test_safety_parity.py",
|
||||
Path(__file__),
|
||||
}
|
||||
SECRET_PATTERNS = (
|
||||
re.compile(r"sk-(?!REPLACE_WITH_YOUR_KEY)[A-Za-z0-9_\-]{16,}"),
|
||||
re.compile(r"AIza[0-9A-Za-z_\-]{20,}"), # Google API keys
|
||||
re.compile(r"(?i)api[_-]?key\s*[=:]\s*['\"][A-Za-z0-9_\-]{16,}['\"]"),
|
||||
)
|
||||
|
||||
|
||||
def _archived_files() -> list[Path]:
|
||||
return sorted(p for p in SOURCES.rglob("*") if p.is_file() and "__pycache__" not in p.parts)
|
||||
|
||||
|
||||
# ── completeness ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_archive_carries_every_required_artifact():
|
||||
for artifact in REQUIRED_ARTIFACTS:
|
||||
path = ARCHIVE / artifact
|
||||
assert path.is_file(), f"archive is missing {artifact}"
|
||||
assert path.stat().st_size > 0, f"archive artifact {artifact} is empty"
|
||||
assert (SOURCES / "photo_analyzer.py").is_file()
|
||||
assert (SOURCES / "nsfwtag" / "README.md").is_file(), "donor docs must be archived too"
|
||||
assert (SOURCES / "webapp" / "README.md").is_file()
|
||||
|
||||
|
||||
def test_every_ledger_source_is_present_in_the_archive():
|
||||
rows = yaml.safe_load(LEDGER.read_text(encoding="utf-8"))["rows"]
|
||||
for row in rows:
|
||||
source = SOURCES / row["source"]["file"]
|
||||
assert source.is_file(), f"{row['id']}: {row['source']['file']} is not archived"
|
||||
|
||||
|
||||
def test_the_readme_records_provenance_and_the_no_import_rule():
|
||||
readme = (ARCHIVE / "README.md").read_text(encoding="utf-8")
|
||||
for expected in ("CHECKSUMS.sha256", "requirements-lock.txt", "donor_ledger.yaml"):
|
||||
assert expected in readme, f"README does not point at {expected}"
|
||||
assert "Schema notes" in readme, "the donor's schema must be documented"
|
||||
assert "nsfw_scores.csv" in readme, "the CSV's fate must be documented"
|
||||
|
||||
|
||||
# ── integrity ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_every_archived_source_matches_its_checksum():
|
||||
"""A frozen archive that silently drifts is not evidence of anything."""
|
||||
recorded = {}
|
||||
for line in CHECKSUMS.read_text(encoding="utf-8").splitlines():
|
||||
digest, _, rel = line.partition(" ")
|
||||
if rel:
|
||||
recorded[rel.strip()] = digest
|
||||
actual = {
|
||||
str(path.relative_to(ARCHIVE)): hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
for path in _archived_files()
|
||||
}
|
||||
assert actual == recorded, "archived sources and CHECKSUMS.sha256 disagree"
|
||||
|
||||
|
||||
# ── redaction ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_the_archive_contains_no_credential():
|
||||
for path in [*_archived_files(), *(ARCHIVE / a for a in REQUIRED_ARTIFACTS)]:
|
||||
if path.suffix in (".png", ".jpg", ".webp"):
|
||||
continue
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
for pattern in SECRET_PATTERNS:
|
||||
assert not pattern.search(text), f"possible secret in {path.relative_to(REPO)}"
|
||||
|
||||
|
||||
def test_the_sample_configuration_is_a_placeholder_only():
|
||||
sample = (ARCHIVE / "photo_analyzer.env.sample").read_text(encoding="utf-8")
|
||||
assert "REDACTED" in sample
|
||||
# Placeholders are not key-shaped, so neither a scanner nor a reader can
|
||||
# mistake the sample for a credential.
|
||||
assert "LLM_API_KEY=<" in sample
|
||||
# No real env file, database, log, or CSV may ride along in the archive.
|
||||
strays = [
|
||||
p.name
|
||||
for p in ARCHIVE.rglob("*")
|
||||
if p.is_file() and p.suffix in (".env", ".db", ".sqlite", ".sqlite3", ".log", ".csv")
|
||||
]
|
||||
assert strays == [], f"unexpected runtime files archived: {strays}"
|
||||
|
||||
|
||||
# ── inertness ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_production_code_never_imports_an_archived_module():
|
||||
for path in (REPO / "photo_pipeline").rglob("*.py"):
|
||||
text = path.read_text(encoding="utf-8")
|
||||
for module in ARCHIVED_MODULES:
|
||||
assert not re.search(rf"^\s*(import|from)\s+{module}\b", text, re.MULTILINE), \
|
||||
f"{path.relative_to(REPO)} imports the archived {module}"
|
||||
assert "legacy_cli_archive" not in text, \
|
||||
f"{path.relative_to(REPO)} names the archive"
|
||||
|
||||
|
||||
def test_only_the_parity_suites_put_the_archive_on_the_import_path():
|
||||
for path in (REPO / "tests").rglob("*.py"):
|
||||
if path in IMPORT_ALLOWED or "__pycache__" in path.parts:
|
||||
continue
|
||||
assert "legacy_cli_archive" not in path.read_text(encoding="utf-8"), \
|
||||
f"{path.relative_to(REPO)} reaches into the archive"
|
||||
|
||||
|
||||
def test_the_archived_modules_are_unimportable_from_a_clean_interpreter():
|
||||
"""The real check: a fresh process with the repo on its path cannot load them."""
|
||||
script = (
|
||||
"import importlib.util, sys; "
|
||||
f"sys.path.insert(0, {str(REPO)!r}); "
|
||||
"print([m for m in "
|
||||
f"{list(ARCHIVED_MODULES)!r}"
|
||||
" if importlib.util.find_spec(m) is not None])"
|
||||
)
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", script], capture_output=True, text=True, cwd=str(REPO), check=True
|
||||
)
|
||||
assert result.stdout.strip() == "[]", f"still importable: {result.stdout.strip()}"
|
||||
|
||||
|
||||
def test_the_application_starts_without_the_archive(tmp_path):
|
||||
"""Nothing in the runtime path may need the frozen sources to exist."""
|
||||
pytest.importorskip("fastapi")
|
||||
from photo_pipeline.api.app import create_app
|
||||
from photo_pipeline.config import Config
|
||||
|
||||
config = Config.from_env(
|
||||
{"PHOTO_PIPELINE_DATA_DIR": str(tmp_path / "data"), "PHOTO_PIPELINE_LIBRARY_ROOTS": ""}
|
||||
)
|
||||
assert create_app(config) is not None
|
||||
Reference in New Issue
Block a user