Files

domverse 80447092e9 Bootstrap project and safe work-item workflow

2026-07-13 13:51:41 +02:00

93 KiB

Raw Permalink Blame History

Photo Library Pipeline — Integrated App Concept

Unify duplicate detection, safety review, content analysis, album naming, guarded renaming, and Immich upload into one local web application.

North star: every photo moves through one visible, resume-safe workflow. Destructive or irreversible actions always require a preview and explicit approval. A file path is mutable metadata, never the identity of a photo.

1. Product goals

The app should:

guide the user through the stages in the correct order;
preserve the existing NSFW review and Photo Analyzer UI/UX;
prevent NSFW photos from reaching the cloud vision provider while still allowing reviewed, EXIF-tagged NSFW photos to be uploaded to Immich;
detect duplicates before spending API quota or changing metadata;
keep database records valid when files or folders are renamed;
create a verified DB↔EXIF consistency checkpoint after every metadata-producing stage, always merging with and preserving existing metadata;
upload each approved album through immich-go with a visible audit trail;
optionally archive successfully uploaded albums out of active storage while retaining their complete identity, metadata, thumbnails, and deduplication evidence;
remain safe to stop and resume at every stage;
exclude every _IGNORE/ directory from discovery, counts, previews, and actions.

The application remains local and binds only to 127.0.0.1 by default. It preserves the existing visual design while replacing the ad-hoc stdlib servers with a typed, versioned application API.

2. Canonical workflow

0. Inventory & duplicates
          ↓ approved canonical set
1. Safety score & human review
          ├── SFW assets → content analysis
          └── NSFW assets → skip analysis, retain safety EXIF
2. Content analysis & analysis EXIF for SFW assets
          ↓ reviewed, metadata-ready assets
3. Album suggestions & guarded rename
          ↓ approved final folder layout
4. Immich upload
          ↓ verified managed assets
5. Archive
          ↓ removed from active storage, retained in the permanent ledger

Stages are sequential gates, not merely navigation tabs. Later stages may be viewed at any time, but actions are disabled until their prerequisites are met.

Gate rules

Stage	Ready when	Blocks next stage when
Inventory	discovery and hashes are complete	unclassified duplicate clusters or missing files remain
Safety	every canonical asset is `sfw`, `nsfw`, or explicitly deferred	undecided assets remain unless the user accepts a partial run
Analysis	every eligible SFW asset has DB data and verified analysis EXIF	pending/error or EXIF-divergent records remain unless explicitly deferred
Albums	every selected album proposal is applied or dismissed	rename plan is pending, partially applied, or inconsistent
Upload	final file hashes are current and credentials validate	metadata changed after upload preparation
Archive	selected album has a verified upload and validated archive destination	upload is uncertain, files changed, archive target unavailable, or transfer verification failed

The app can support a partial pipeline for selected folders, but it must show that the library is only partially complete.

3. Core architecture

Real application, not CLI scripts with attached pages

The target is a cohesive Python application with a documented API, database-backed services, persistent jobs, and one frontend. The current CLIs are the primary donor implementations for the new services: reuse their proven code, algorithms, prompts, format handling, subprocess adapters, and edge-case behavior wherever compatible with the target architecture. They are not the final application boundary and will be archived only after their donated behavior has automated parity coverage.

Browser
   │ JSON API + Server-Sent Events
   ▼
FastAPI application
   ├── inventory / duplicate service
   ├── thumbnail service
   ├── safety service
   ├── analysis service
   ├── album and rename service
   ├── upload service
   └── job coordinator
          │ claims durable jobs
          ▼
Python worker process ── exiftool / local model / vision API / immich-go
          │
          ▼
SQLite database in WAL mode + managed application cache

Recommended backend foundation:

FastAPI for typed HTTP APIs, validation, OpenAPI documentation, and streaming;
Pydantic request/response models so browser input never reaches services raw;
SQLAlchemy 2.x for explicit transactions and repository boundaries;
Alembic for versioned, testable database migrations;
SQLite WAL mode initially, with short write transactions and one job writer;
a separate Python worker process that claims durable jobs from SQLite;
Server-Sent Events for live job/activity updates, with polling as fallback;
structured JSON logging carrying job_id, asset_id, and operation_id.

SQLite is appropriate for a single-user local application. Repository and service interfaces must avoid SQLite-specific business logic so PostgreSQL remains a possible future deployment option, not an immediate dependency.

Domain services

Current code should be progressively extracted into importable services:

InventoryService — discovery, path reconciliation, hashes, duplicate candidates;
ThumbnailService — safe, cached previews for every supported image format;
DuplicateService — clusters, recommendations, decisions, canonical membership;
SafetyService — scoring, review decisions, EXIF safety keywords;
AnalysisService — vision requests, result validation, DB and EXIF writes;
AlbumService — folder summaries, AI proposals, naming policy;
RenameService — plans, validation, journaled apply/recovery;
UploadService — preflight, batches, immich-go, verification;
ArchiveService — archive plans, transfer verification, offline state, restore;
JobService — durable job lifecycle, locks, progress, cancellation, events.

The CLI entry points may call these same services for backwards compatibility. The web API never shells out to photo_analyzer.py or nsfw_tag.py; only genuine external tools such as exiftool and immich-go remain subprocesses.

Donor-first CLI migration and archival

This is an extraction and consolidation project, not a greenfield rewrite. Before implementing a replacement service, inspect the corresponding CLI implementation and classify each relevant function or behavior as reuse unchanged, extract with a thin adapter, refactor while preserving behavior, or replace with documented reason. Default to extraction; replacement requires a concrete incompatibility such as unsafe concurrency, path-keyed identity, or inability to support durable jobs.

Create and maintain a donor ledger mapping legacy modules/functions to their target service location, characterization tests, migration status, intentional behavior changes, and final parity test. Important donor material includes:

file discovery, extension handling, folder grouping, and path exclusion policy;
exact/perceptual hashing, duplicate clustering, reconciliation, and resume behavior;
image decode, orientation, resize, HEIC conversion, and thumbnail behavior;
vision prompts, request construction, JSON validation, retries, and rate limiting;
NSFW inference, scoring thresholds, review decisions, and model integration;
EXIF field projection, preservation rules, subprocess invocation, and read-back;
SQLite schema knowledge, migrations, FTS behavior, status transitions, and stats;
existing frontend design tokens, layouts, interactions, keyboard behavior, and useful browser-side components;
operational logging, cancellation, error reporting, configuration, and CLI flags.

For each donor area, first add characterization tests against the current CLI code and record representative outputs using the deterministic fixture corpus. Extract the smallest coherent implementation into the new service layer, then run the same fixture expectations against both implementations. Preserve known behavior unless the concept explicitly changes it; every intentional difference needs a rationale and a new acceptance test. Avoid copying code into parallel implementations—after extraction, the legacy entry point should temporarily import the shared service where practical.

Archive a CLI script only when all of its in-scope donor rows are resolved, its replacement passes unit, characterization, integration, and end-to-end parity tests, database migration/reconciliation is verified, and no supported workflow still depends on the old entry point. Archival means moving the original sources and their documentation into a read-only legacy_cli_archive/ area with their final dependency lock, schema notes, sample configuration with secrets removed, donor ledger, and a recorded final version/checksum. Archived scripts are reference material and rollback evidence; production code must not import or execute them.

API boundaries

All mutations are commands against stable IDs, never raw filesystem paths. Example API groups:

Method	Endpoint	Purpose
GET	`/api/v1/workflow`	stage readiness, blockers, totals, active job
POST	`/api/v1/inventory/scan`	enqueue discovery/reconciliation job
GET	`/api/v1/duplicates/clusters`	paged clusters by confidence/state
GET	`/api/v1/duplicates/clusters/{id}`	members, thumbnails, evidence
POST	`/api/v1/duplicates/clusters/{id}/decision`	canonical/variants/not-duplicate/defer
GET	`/api/v1/assets/{id}`	stable asset record and stage history
GET	`/api/v1/assets/{id}/thumbnail?size=512`	cached, oriented preview
POST	`/api/v1/safety/jobs`	enqueue scoring for eligible assets
POST	`/api/v1/safety/decisions`	persist reviewed decisions
POST	`/api/v1/analysis/jobs`	enqueue content analysis
POST	`/api/v1/albums/proposals`	generate album suggestions
POST	`/api/v1/rename-plans/{id}/apply`	apply a validated plan
POST	`/api/v1/upload-batches/{id}/start`	start approved Immich upload
POST	`/api/v1/archive-plans`	create and validate an archive plan
POST	`/api/v1/archive-plans/{id}/apply`	transfer/remove active folder after verification
POST	`/api/v1/assets/{id}/restore`	restore an archived asset/album to active storage
GET	`/api/v1/jobs/{id}`	durable status and progress
POST	`/api/v1/jobs/{id}/cancel`	cooperative cancellation request
GET	`/api/v1/events`	SSE activity stream

API responses use explicit enums and error codes. Conflicts such as stale hashes, running jobs, or changed rename plans return 409 Conflict; validation errors return 422; unknown IDs return 404. OpenAPI becomes the contract between backend and UI.

Persistent job model

Long operations are database jobs, not in-memory threads:

queued → running → cancelling → cancelled
                 ↘ succeeded
                 ↘ failed → retry_queued

The worker claims a queued job atomically, records a heartbeat, persists incremental progress and events, and checks cancellation between items. On application restart, stale running jobs are detected from their heartbeat and offered for safe resume. Mutating job types declare locks such as library_write, rename, or upload, which the coordinator uses to prevent unsafe concurrency.

One durable database

Extend the current SQLite database rather than introduce separate score and state files. CSV export may remain available, but it is not authoritative.

Most importantly, replace path identity with a stable internal asset identifier.

CREATE TABLE assets (
    id                  TEXT PRIMARY KEY,   -- generated UUID, never changes
    current_path        TEXT UNIQUE,           -- active-library path; NULL when offline
    original_path       TEXT NOT NULL,
    current_sha256      TEXT,               -- current file bytes; refreshed after EXIF
    pixel_sha256        TEXT,               -- normalized decoded pixels; ignores EXIF
    phash               TEXT,               -- resilient to resize/recompression
    hash_version        INTEGER NOT NULL,    -- decoder/normalization algorithm version
    byte_size           INTEGER,
    discovered_at       TEXT NOT NULL,
    missing_at          TEXT,

    canonical_asset_id  TEXT REFERENCES assets(id),
    safety_score        REAL,
    availability_state  TEXT NOT NULL DEFAULT 'active',
    archive_location_id TEXT,
    state_version       INTEGER NOT NULL DEFAULT 1
);

CREATE TABLE asset_paths (
    asset_id            TEXT NOT NULL REFERENCES assets(id),
    path                TEXT NOT NULL,
    valid_from          TEXT NOT NULL,
    valid_until         TEXT,
    reason              TEXT,
    PRIMARY KEY (asset_id, path, valid_from)
);

CREATE TABLE asset_stage_states (
    asset_id            TEXT NOT NULL REFERENCES assets(id),
    stage               TEXT NOT NULL,
    state               TEXT NOT NULL,
    substate            TEXT,
    input_version       TEXT,
    attempt_count       INTEGER NOT NULL DEFAULT 0,
    active_attempt_id   TEXT,
    blocker_code        TEXT,
    error_code          TEXT,
    error_message       TEXT,
    updated_at          TEXT NOT NULL,
    completed_at        TEXT,
    version             INTEGER NOT NULL DEFAULT 1,
    PRIMARY KEY (asset_id, stage)
);

CREATE TABLE asset_stage_attempts (
    id                  TEXT PRIMARY KEY,
    asset_id            TEXT NOT NULL REFERENCES assets(id),
    stage               TEXT NOT NULL,
    job_id              TEXT NOT NULL REFERENCES jobs(id),
    attempt_number      INTEGER NOT NULL,
    input_version       TEXT NOT NULL,
    started_at          TEXT NOT NULL,
    heartbeat_at        TEXT,
    finished_at         TEXT,
    outcome             TEXT,
    last_substate       TEXT,
    error_code          TEXT,
    error_message       TEXT,
    worker_id           TEXT,
    fencing_token       INTEGER NOT NULL,
    UNIQUE (asset_id, stage, attempt_number)
);

Analysis fields can remain in photos during migration, but photos should gain an asset_id foreign key and eventually use it as its stable relationship. All FTS, EXIF, duplicate, rename, and upload records refer to asset_id, not path.

Per-asset workflow state

Every asset has one durable state row per stage. This is the authoritative answer to “what happened to this picture?” The broad workflow totals are derived from these rows, not maintained independently.

Stages include:

inventory → duplicate_review → safety → analysis → album → upload → archive

The common state vocabulary is:

State	Meaning
`blocked`	prerequisite is incomplete; `blocker_code` explains why
`ready`	all prerequisites satisfied
`queued`	assigned to a durable job but not started
`running`	worker owns a valid lease and attempt
`complete`	stage output and required verification are durable
`deferred`	user intentionally postponed the stage
`skipped`	stage does not apply, with a recorded reason
`failed`	attempt ended with a retryable or terminal error
`cancelled`	user/system cancelled before completion
`interrupted`	worker disappeared or application stopped mid-attempt
`stale`	previously complete output no longer matches current inputs

substate identifies the precise point within a running/interrupted stage, for example:

safety:   decoding | scoring | awaiting_review | writing_exif | verifying_exif
analysis: preparing | calling_provider | validating | storing_result |
          writing_exif | verifying_exif
album:    summarizing | generating_name | awaiting_approval | renaming | verifying_paths
upload:   preflight | hashing | uploading | verifying_remote
archive:  preflight | transferring | verifying_archive | removing_active |
          verifying_absence | complete_offline

An interruption is not guessed from a browser disconnect. A stage becomes interrupted only when its active attempt was running, the worker heartbeat/lease expired or the process shut down uncleanly, and no valid newer attempt owns it. The recovery service records the interrupted attempt, inspects its last substate and real file/external state, then chooses one of:

resume safely from a verified checkpoint;
restart the stage idempotently;
mark unknown_requires_verification and request user action.

The immutable asset_stage_attempts table preserves every start, retry, interruption, cancellation, error, and successful completion. The current state row is a projection for fast UI queries; history is never overwritten.

Dependency and invalidation rules

Stage completion is versioned against its inputs. Examples:

a changed file invalidates inventory hashes and may make duplicate review stale;
changing the duplicate canonical can skip later stages for one asset and ready them for another;
changing SFW→NSFW makes analysis skipped and invalidates any in-flight result, but does not invalidate upload eligibility after safety EXIF verification;
changing NSFW→SFW makes analysis ready and upload blocked until analysis completes;
external EXIF changes make the relevant EXIF projection stale or divergent;
a rename changes paths but does not invalidate content analysis or safety state;
any byte-changing EXIF checkpoint makes upload stale until its exact hashes are refreshed.
archiving changes availability and path state but does not invalidate duplicate, safety, analysis, EXIF, or upload history;
restoring an archived asset creates a new active path occurrence and requires a reconciliation scan before mutation.

State transitions use optimistic version checks and are committed with their audit event. Workers may only complete the attempt they own with the current fencing token. There is no generic “set status” endpoint; services expose valid domain transitions.

Additional tables

duplicate_clusters / duplicate_members — reviewable cluster decisions.
safety_reviews — score, final decision, reviewer, timestamp, prior decision.
analysis_runs — model/config/prompt version, token usage, status, timestamps.
album_proposals — source folder, proposed name, rationale, confidence, status.
rename_plans / rename_operations — complete preview and crash-safe journal.
upload_batches / upload_items — command, album, verified content checksums, result, log.
archive_locations / archive_plans / archive_operations — target configuration, manifests, journaled transfers, verification, removal, and restore history.
jobs / job_events — shared progress and activity history across all stages.
thumbnails — cache key, dimensions, format, path, generation state, error.

Database ownership and transactions

API routes call application services; they do not contain SQL.
Services use repositories; repositories are the only database access layer.
Every state change and corresponding audit event is written in one transaction.
Filesystem operations use persisted journals because they cannot participate in a database transaction.
SQLite foreign keys are enabled, WAL mode is required, and migrations run once at startup under an application lock.
Database backups use SQLite's online backup API rather than copying a live DB file.
Secrets live in environment/keychain-backed configuration, never database events.

EXIF ownership and consistency checkpoints

EXIF is not a final export performed only before upload. It is a durable projection of completed workflow decisions. Every stage that produces metadata ends with an EXIF checkpoint for each affected asset:

persist desired stage result in DB
→ read current EXIF
→ merge only the stage-owned fields into current metadata
→ write through exiftool
→ read EXIF back
→ verify desired values and preserved values
→ refresh current full-file SHA-256
→ mark checkpoint verified

The database stores both the desired projection and the last verified projection. Suggested table:

CREATE TABLE exif_projections (
    asset_id              TEXT NOT NULL REFERENCES assets(id),
    stage                 TEXT NOT NULL,        -- safety | analysis | album
    projection_version    INTEGER NOT NULL,
    desired_json          TEXT NOT NULL,
    verified_json         TEXT,
    source_file_sha256    TEXT NOT NULL,
    result_file_sha256    TEXT,
    state                 TEXT NOT NULL,        -- pending | writing | verified | divergent | error
    verified_at           TEXT,
    PRIMARY KEY (asset_id, stage)
);

Field ownership is intentionally narrow:

Owner	May modify	Must preserve
User/pre-existing metadata	nothing automatically owns it	all dates, GPS, camera, ratings, people, captions, and keywords
Safety stage	only mutually exclusive `nsfw` / `sfw` entries in `Keywords` and `Subject`	all non-safety keywords and all other fields
Analysis stage	managed AI caption segment plus additive AI keywords	user caption text, safety keywords, existing keywords, and unrelated metadata
Album stage	no EXIF by default; optional managed album field only when enabled	every existing field

Non-destructive means semantic preservation, not that the file bytes remain unchanged: exiftool may rewrite the container. Before writing, retain the relevant EXIF snapshot in the operation journal; afterward, compare all fields outside the stage's ownership. Any unexpected change marks the asset divergent, blocks the next mutating stage, and exposes a recovery action.

Analysis captions should use a recognizable managed segment so re-analysis can replace only its own text, for example an application-owned AI: portion, without overwriting a user's description. Keywords are merged as a set. Because standard EXIF keywords do not record ownership, removing an old AI keyword could also remove a user keyword with the same value. The conservative default is therefore additive: never automatically remove non-safety keywords during re-analysis. Potentially stale generated keywords are reported for optional review rather than silently deleted.

Stage consistency rules:

Inventory/duplicate stage: read-only; records an EXIF fingerprint and proves it did not modify the file.
Safety stage: complete only after the chosen sfw or nsfw keyword is written, the opposite safety keyword is absent, all unrelated metadata is preserved, and the new file SHA-256 is stored.
Analysis stage: applies only to SFW assets and completes per asset only after its DB result, managed caption segment, additive keywords, preservation checks, and new file SHA-256 are verified.
Album/rename stage: renaming does not itself require an EXIF rewrite. If optional album metadata is enabled, it receives its own projection/checkpoint after rename.
Upload stage: writes no metadata. It verifies every required prior projection, recalculates the current file SHA-256, and uploads exactly those verified bytes.

If a user edits EXIF externally after a checkpoint, the next scan compares the current projection/hash with the verified state. User additions are preserved and incorporated; missing or conflicting managed values create a visible divergent checkpoint requiring repair or explicit acceptance. Upload never silently repairs metadata as a side effect.

4. Stage 0 — Inventory and duplicate review

Discovery

Discovery registers supported photos without modifying them. It records current path, size, dates, SHA-256, normalized-pixel hash, and perceptual hash. It always skips _IGNORE/ and generated thumbnail/cache directories.

The inventory screen shows:

total files and folders;
exact duplicates by SHA-256;
perceptual duplicate clusters by phash distance;
likely variants such as edited, resized, or motion-photo derivatives;
unreadable and unsupported files;
estimated API calls saved by resolving duplicates.

Duplicate review

Do not immediately mark the largest file canonical without review. Exact byte or normalized-pixel matches can be recommended with very high confidence; every fuzzy match receives a visual decision surface.

Viable thumbnail pipeline

Duplicate review must never depend on the browser being able to render the original. The backend generates managed thumbnails for JPEG, PNG, WebP, TIFF, HEIC/HEIF, and other supported image types.

Thumbnail generation rules:

read the original through Pillow plus the configured HEIC decoder;
apply EXIF orientation before resizing;
convert safely to sRGB/RGB without modifying the original;
generate at least 256px, 512px, and 1280px long-edge variants;
encode browser-compatible WebP, with JPEG fallback;
write to a temporary file and atomically rename into the cache;
persist success/error state so broken originals do not create retry loops.

The cache lives in application data outside the photo library, for example:

data/cache/thumbs/{pixel_hash_prefix}/{pixel_hash}-512-v2.webp

The key contains the normalized pixel hash, requested size, and thumbnail algorithm version. Therefore:

EXIF-only changes reuse the same thumbnail;
folder moves and renames reuse the same thumbnail;
visually different files never collide merely because filenames match;
changing decoder/color/orientation behavior invalidates cache through the version;
cache files can be deleted and regenerated without losing application state.

Thumbnail endpoints accept only an asset_id and bounded size enum. They resolve the current path through the database, set immutable-cache headers for versioned keys, and never accept an arbitrary path from the browser.

Fuzzy duplicate comparison UI

Show uncertain clusters in a dedicated comparison workspace rather than a small grid:

synchronized side-by-side thumbnails, initially fit-to-window;
click/keyboard toggle between candidates;
linked zoom and pan using the 1280px preview;
optional blink/overlay mode to reveal framing and crop differences;
pixel dimensions, aspect ratio, file size, format, dates, folder, camera, EXIF richness, file hash, pixel hash, pHash distance, and confidence explanation;
badges for exact bytes, metadata-only difference, resized/recompressed copy, probable crop, edited variant, or uncertain similarity;
recommended canonical clearly marked, but never silently selected for fuzzy matches;
prefetch the next cluster's thumbnails for fast keyboard review.

The decision applies to a complete cluster and is stored as an auditable event. If a new copy appears later, the cluster is reopened only when it changes the confidence or conflicts with the previous decision; otherwise the new member inherits the established canonical relationship.

The app recommends a canonical item using a visible scoring policy:

original/highest resolution;
least recompressed format;
richest reliable metadata;
preferred folder rules;
largest file only as a final tie-breaker.

Actions:

Keep recommended;
Choose another canonical;
Keep all as variants;
Not duplicates;
Defer.

No file is deleted automatically. Non-canonical items are excluded from later stages and upload, but remain on disk unless a separate future cleanup feature is approved.

5. Stage 1 — Safety identification and EXIF tagging

Preserve the current NSFW review experience: folder tree, threshold, score ordering, bulk selection, per-image decision control, lightbox, EXIF panel, and activity log.

Changes for integration:

run only on canonical, non-ignored assets;
store score and decisions in SQLite as well as EXIF;
write mutually exclusive nsfw or sfw keywords to Keywords and Subject;
merge safety keywords with all existing EXIF and verify unrelated metadata remains;
refresh current_sha256 after the verified EXIF checkpoint;
replace nsfw_scores.csv as the source of truth with database state;
make _IGNORE/ exclusion mandatory in the shared discovery service;
offer filters for undecided, SFW, NSFW, deferred, and write errors.

Privacy invariant:

Only assets with a confirmed sfw decision may enter cloud content analysis. Confirmed nsfw assets skip that analyzer but remain eligible for Immich upload.

An optional setting may treat unreviewed low-score images as SFW, but it should be off by default and clearly marked as reducing safety.

6. Stage 2 — Content analysis and analysis EXIF

Preserve the current Library, Analyze, and Stats views, but scope the engine to confirmed SFW canonical assets.

For each eligible asset:

prepare a resized in-memory image;
call the configured OpenAI-compatible vision model;
validate and store the response by asset_id;
read the latest EXIF and merge the managed AI caption segment and additive tags;
write and read back EXIF without removing safety or user metadata;
compare non-owned fields with the pre-write snapshot;
refresh the full-file SHA-256 after verification;
mark the analysis EXIF checkpoint verified and the asset metadata-complete.

The UI shows model and prompt version, progress, errors, retries, tokens, estimated remaining cost/quota, and EXIF verification state.

The stage is complete only when every included database result and on-disk EXIF projection agree. Verification is per asset, not merely a sample; the UI may additionally offer a human-readable sample review before proceeding.

7. Stage 3 — AI album suggestions and guarded renaming

This stage operates primarily on folders, not individual filenames. The default goal is to create useful Immich albums while minimizing filesystem changes.

Inputs for an album proposal

For each leaf folder, build a compact summary from:

current folder name and parent path;
capture-date range and dominant year;
number of photos and videos;
common analysis tags, settings, places, and descriptions;
GPS/location hints where present;
detected events or themes;
neighboring folder names and date ranges;
existing user-provided naming conventions.

Prefer using already-stored analysis data. Do not resend every image. If visual sampling is useful, make it an explicit option using a few representative SFW images.

Suggested naming format

The default recommendation should be predictable and sortable:

YYYY-MM — Place — Event
YYYY — Event
Place — Event

Examples:

2019-07 — Rome — Summer Holiday
2016 — Capgemini Team Event
Family Portraits — Selected

Names should be concise, human-readable, filesystem-safe, and suitable for Immich's folder-as-album behavior. The app shows why each name was proposed and flags uncertain date/place inferences.

Proposal review UI

Use a two-pane Albums view:

left: folder tree with proposal state and confidence;
right: current name, proposed name, representative thumbnails, date range, common tags, rationale, and editable final name.

Bulk actions may accept high-confidence proposals, but never apply them immediately. Accepted edits first become a rename plan.

Guarded rename plan

Before any filesystem operation, validate the entire plan:

source exists and is still the registered folder;
destination is inside the configured library;
neither source nor destination is inside _IGNORE/;
destination names are unique on the target filesystem;
case-only renames are handled safely on case-insensitive filesystems;
no asset is currently being analyzed, EXIF-written, or uploaded;
final paths do not exceed configured portability limits;
all affected assets are present and hashes match the latest inventory;
database and filesystem are writable;
no previously uploaded asset would be silently changed.

The preview displays every old → new path and supports exporting the plan as JSON. Applying requires an explicit confirmation such as Apply 12 folder renames.

Crash-safe rename transaction

Filesystem and SQLite cannot share a true transaction, so use a journaled state machine:

planned → validated → moving → moved → database_updated → verified → complete
                                  ↘ failed / rollback_required

For each operation:

persist the operation and expected hashes;
rename on the same filesystem using an atomic move where possible;
update assets.current_path by asset_id in one SQLite transaction;
append old/new entries to asset_paths;
verify all files at their new paths and refresh folder summaries;
mark the operation complete.

On restart, the app inspects the journal and filesystem to resume or offer rollback. It never guesses based only on a missing old path.

Because records use stable asset IDs, renaming does not invalidate analysis, safety, duplicate, or upload history.

8. Stage 4 — Immich upload via immich-go

Upload is an explicit final stage, never an automatic consequence of rename.

Safety and upload eligibility

Safety classification controls analysis routing, not whether a reviewed photo may exist in Immich:

Safety state	Cloud content analysis	Immich upload
`sfw`	eligible/required by normal workflow	eligible after verified safety and analysis EXIF checkpoints
`nsfw`	forbidden	eligible after verified `nsfw` EXIF tag
`unreviewed`	forbidden	blocked until reviewed
`deferred`	forbidden	blocked unless an explicit future policy permits it

Immich therefore receives both reviewed SFW and reviewed NSFW canonical assets. The nsfw keyword is written before upload so those assets can be found, filtered, or managed in Immich without exposing them to the external content-analysis provider.

Preflight

Before enabling upload, verify:

asset is canonical and has a confirmed sfw or nsfw safety decision;
SFW assets have verified safety and analysis EXIF checkpoints, unless explicitly waived with a recorded exception;
NSFW assets carry their verified nsfw EXIF keyword and have not been sent to the cloud content analyzer;
rename journal is empty and album paths are final;
current SHA-256 matches the latest verified EXIF checkpoint and post-rename inventory;
Immich server and API key validate;
immich-go is installed;
no External Library workflow is configured for the same source;
the album name preview matches the folder-as-album result.

Upload batches

Create one batch per selected leaf folder. The UI previews the exact command without printing the API key. A batch stores:

album/folder and asset IDs;
redacted command and immich-go version;
pre-upload SHA-256 plus the SHA-1 used by Immich/immich-go for every item, both calculated from the latest verified bytes;
stdout/stderr activity stream;
new, upgraded, metadata-updated, duplicate, skipped, and failed counts;
start/end timestamps and exit status.

The Upload view provides Validate, Dry run where supported, Upload selected, Stop after current album, and Retry failed actions.

After a successful upload, an asset becomes uploaded only when its result is parsed or otherwise verified. If the file bytes later change, the app marks it changed_after_upload and warns that uploading again may create or upgrade an asset.

9. Stage 5 — Archive and active-storage reclamation

Archive is an optional final lifecycle stage. It moves a completed album out of the active library branch to a configured archive location, then marks its assets archived and unavailable from active storage while preserving their permanent database records.

“Archived” means no longer present in the active photo library. The archive may be another local disk, external disk, NAS-backed archive workflow, or removable medium. When that destination is not mounted, the asset is archived_offline, not missing.

Archive is not deduplication and never deletes database history. It preserves:

stable asset ID and complete path history;
exact file SHA-256 and Immich SHA-1 from the archived bytes;
normalized pixel hash and perceptual hashes;
duplicate cluster and canonical decision;
safety decision and verified EXIF projections;
content analysis, FTS data, album proposal, rename history, and upload history;
a durable browser-compatible review thumbnail, preferably the 1280px variant;
archive location, relative archive path, media/volume identity, and transfer manifest.

Availability states

Availability is independent from workflow completion:

State	Meaning
`active`	original is present in the active library
`archiving`	journaled transfer/removal is in progress
`archived_online`	original exists at a currently reachable archive location
`archived_offline`	original is archived but its medium/location is unavailable
`restoring`	copy back to active storage is in progress
`missing_unexpected`	neither active nor recorded archive location explains absence

The ordinary inventory scanner must not prune archived assets. It scans active roots and currently mounted archive roots separately and interprets absence using availability_state and archive-location identity.

Archive preflight

Before allowing an album to be archived:

every selected canonical asset has a confirmed safety decision;
every SFW asset has verified analysis EXIF and every NSFW asset has verified safety EXIF;
the selected upload batch is verified, not merely process-successful;
current file checksums match the exact uploaded/checkpointed bytes;
no rename, EXIF, analysis, upload, restore, or other archive job holds a conflicting lease;
the archive destination is configured, mounted, writable, outside _IGNORE/, and has sufficient free space plus a reserve;
the destination does not already contain unexpected paths;
the durable comparison thumbnail exists and is verified before the original leaves active storage;
a database backup and archive manifest can be written successfully.

The UI previews source, destination, number of files, total bytes, expected reclaimed active space, transfer method, archive medium, and every blocker. Archive requires an explicit confirmation such as Archive 1 album · reclaim 84.3 GB.

Transfer and removal semantics

Archive uses a journaled plan similar to rename, but cross-filesystem movement is expected:

planned → validated → transferring → archive_verified → removing_active
        → active_absence_verified → database_updated → complete

For each file:

persist source, archive destination, expected size and hashes;
copy to a temporary destination file;
flush/close it and calculate its SHA-256;
atomically rename the verified temporary file to its archive destination;
record the verified archive occurrence and manifest entry;
only then remove the active source;
verify the active path is absent and archive path still matches;
set current_path=NULL, close the active path-history entry, and set availability to archived_online or archived_offline according to location reachability.

If source and archive destination are demonstrably on the same filesystem, an atomic move may replace copy-verify-delete, but the operation still verifies the resulting file and persists the journal before updating database state.

The source is never removed merely because Immich reports success. Immich managed storage is an ingest target, not automatically the only backup. A future explicit “uploaded-only retention” policy may exist, but it must be a separate high-risk option and is not the default Archive behavior.

Future deduplication against archived assets

Archived assets remain in every hash index. When a new active photo is discovered:

exact SHA-256 or normalized-pixel matches can link directly to the archived canonical;
perceptual matches create/reopen a fuzzy duplicate cluster;
the retained thumbnail, metadata, dimensions, and hash evidence appear in duplicate review even when the archive disk is offline;
if full-resolution comparison is required, the UI requests that the named archive location/medium be mounted rather than guessing;
choosing the archived asset as canonical keeps the new active copy excluded from analysis/upload unless the user deliberately restores or changes the canonical.

This is why durable comparison thumbnails are part of archival preflight rather than an evictable cache entry. They may use a separate protected archive-preview store with its own backup policy.

Restore

Restore is also planned and journaled. It requires the archive location online, copies and verifies the archived bytes into a collision-free active destination, registers a new active path occurrence, changes availability to active, and runs reconciliation. Existing safety, analysis, EXIF, duplicate, and upload states remain valid when hashes match. Any mismatch produces stale/divergent state instead of silently accepting a different file.

10. Information architecture and UX

Retain the dark OLED design, cards, folder sidebar, lightbox, segmented controls, progress bars, status colors, activity drawer, and keyboard navigation. The visual language remains familiar even though the frontend is now an API client rather than HTML generated by Python.

Frontend architecture

Keep the first implementation deliberately lightweight and dependency-free: semantic HTML, CSS, and modular browser JavaScript served as separate static assets by FastAPI. index.html contains only the persistent application shell and references the stylesheets and JavaScript entry point; it does not embed the application's CSS, JavaScript, cards, or operational data. Organize JavaScript by API client, state/store, views, and reusable components rather than putting everything in one HTML file or one large script. If the duplicate comparison and workflow state outgrow this approach, the same versioned API can support a later TypeScript framework without changing the backend.

Frontend rules:

no Python string interpolation of cards or operational state;
index.html, CSS, and JavaScript are separate source files;
the initial HTML response is only the application shell; JavaScript fetches all operational data from /api/v1 after the shell loads;
/api/v1 endpoints return JSON (and SSE only for the documented event stream), never HTML fragments or the application shell; API errors use a consistent JSON error envelope and appropriate HTTP status codes;
HTML navigation routes may fall back to index.html, but requests under /api/v1 must never use that fallback;
one shared API client handles errors, cancellation, and optimistic concurrency;
SSE updates the job/activity store; failed streams fall back to polling;
URL routes encode the current view, cluster, asset, album, and filters;
destructive confirmations include the server-issued plan/version token so stale browser state cannot apply changed operations.

Global header

[Photo Pipeline]  Library: pictures/   [Workflow] [Review] [Library] [Albums] [Upload] [Archive]
                   25,318 files · 21 warnings · no job running              [Activity]

Workflow home

The default page is a five-step vertical or horizontal stepper:

✓ 0 Inventory       24,901 canonical · 417 duplicates
! 1 Safety          24,620 SFW · 83 NSFW · 198 undecided
● 2 Analysis        20,110 complete · 4,510 pending
○ 3 Albums          blocked by analysis · 43 suggestions ready
○ 4 Upload          not ready
○ 5 Archive         blocked by upload · 0 B reclaimable

Each stage card contains status, prerequisite warnings, last-run time, primary action, and a link to details. Status is communicated with text/icons as well as color.

Shared interaction rules

one long-running mutating job at a time;
read-only browsing remains available during jobs;
every bulk mutation has preview → validate → confirm → execute → verify;
stop requests drain safely and leave a resumable job record;
every error links to the affected asset, album, or operation;
all actions are scoped to the visible selection and show exact counts;
credentials are never returned to the browser or written to logs;
all paths served by image/EXIF endpoints are validated by asset ID and current path.

11. Consistency rules

The integrated app enforces these invariants centrally:

_IGNORE/ is excluded by one shared path-policy function used everywhere.
Duplicate resolution precedes safety scoring and paid analysis.
NSFW assets never reach the external vision API, but remain eligible for Immich upload after their safety EXIF keyword is verified.
Safety keywords survive content EXIF writes.
Every metadata stage ends with a non-destructive, read-back-verified EXIF checkpoint; upload uses the SHA-256 of the latest verified bytes.
Rename plans cannot run concurrently with analysis, EXIF, or upload jobs.
Asset identity is stable across moves and renames.
Upload history is tied to asset ID plus the exact uploaded SHA-256 and Immich SHA-1.
Changing bytes after upload produces a visible stale-upload warning.
No automatic deletion, rename, or upload occurs without explicit approval.
Archiving never removes an active source until the archive bytes and durable manifest are verified.
Archived assets remain in deduplication indexes even when their original storage is offline.

12. Proposed project layout

pipeline_app/
├── pyproject.toml
├── alembic.ini
├── migrations/
├── src/photo_pipeline/
│   ├── __main__.py             # API/worker management CLI
│   ├── config.py               # typed configuration and secret references
│   ├── db.py                   # engine, sessions, WAL/foreign-key setup
│   ├── models/                 # SQLAlchemy persistence models
│   ├── schemas/                # Pydantic API contracts
│   ├── repositories/           # database access only
│   ├── services/
│   │   ├── inventory.py
│   │   ├── thumbnails.py
│   │   ├── duplicates.py
│   │   ├── safety.py
│   │   ├── analysis.py
│   │   ├── albums.py
│   │   ├── renames.py
│   │   ├── uploads.py
│   │   ├── archives.py
│   │   └── workflow.py
│   ├── jobs/
│   │   ├── coordinator.py      # enqueue, locks, cancellation, recovery
│   │   ├── worker.py           # durable worker loop
│   │   └── handlers.py         # job-type dispatch
│   ├── api/
│   │   ├── app.py              # FastAPI factory and lifecycle
│   │   ├── dependencies.py
│   │   ├── errors.py
│   │   └── routes/             # versioned /api/v1 route modules
│   ├── integrations/
│   │   ├── exiftool.py
│   │   ├── vision.py
│   │   ├── nsfw_model.py
│   │   └── immich_go.py
│   └── path_policy.py          # library boundary and _IGNORE rules
├── frontend/
│   ├── index.html              # markup-only application shell
│   ├── css/
│   │   └── app.css             # shared design tokens, layout, components
│   └── js/
│       ├── app.js              # module entry point and application bootstrap
│       ├── api.js
│       ├── store.js
│       ├── router.js
│       ├── components/
│       └── views/
├── legacy_cli_archive/         # frozen donor sources after verified parity
│   ├── README.md               # provenance, final versions, and archive rationale
│   └── donor-ledger.yaml       # source → service/test mapping and intentional deltas
├── tests/
│   ├── unit/
│   ├── integration/
│   └── e2e/
└── data/                       # runtime DB/cache; ignored by source control
    └── cache/thumbs/

Existing CLIs remain available during migration. Their proven internals should be extracted into the new service layer, after which the CLI entry points should call those shared services until the archival gates above pass. Business logic must have one implementation shared by API, worker, and transitional CLI. This avoids both a needless rewrite and two subtly different discovery, path-exclusion, EXIF, or state models surviving in parallel.

13. Recommended implementation phases

Automated acceptance policy for every phase

Automated tests are part of implementation, not deferred hardening. Every user story must have at least one automated acceptance test that exercises its observable behavior and failure conditions. Maintain a traceable mapping from story identifier to test name/path; a story is incomplete while its mapped tests are missing, skipped, or failing.

Every phase below must add and pass an end-to-end suite covering the user journeys introduced or changed by that phase through the real browser, HTTP/SSE API, worker, isolated database, and temporary photo library. External services and executables may use controlled fakes at their integration boundaries, but browser-visible behavior and backend orchestration must not be mocked. The full end-to-end regression suite for all completed phases must also remain green. A phase cannot be declared complete or merged merely because its unit or API tests pass.

Phase A — Shared identity and inventory

inventory both CLI codebases into the donor ledger and add characterization tests before replacing or relocating any in-scope behavior;
extract reusable discovery, hashing, reconciliation, image/EXIF, model, database, configuration, and UI donor components into stable modules with provenance noted;
scaffold FastAPI, SQLAlchemy, Alembic, configuration, and application lifecycle;
add stable asset IDs and path history;
centralize discovery and _IGNORE/ policy;
migrate current photo and NSFW state;
implement normalized pixel hashes plus exact and perceptual indexes;
implement managed thumbnail generation and its asset-ID API;
build inventory and visual duplicate-review UI.

End-to-end gate: automated journeys cover scan/reconciliation, _IGNORE/ exclusion, stable identity across a move, thumbnail loading/orientation, exact and fuzzy duplicate clusters, canonical selection, and persistence after reload. Donor characterization and parity suites pass for every CLI behavior migrated in this phase.

This is the foundation. Do not implement renaming before it is complete.

Phase B — Unified workflow shell

implement durable jobs, worker heartbeats, recovery, locks, and SSE events;
create Workflow home and shared job/activity model;
port the existing safety review to the shared APIs;
port the existing Library/Analyze/Stats views to the shared APIs;
enforce gates and one-mutating-job policy.

End-to-end gate: automated journeys cover the workflow shell loading data from JSON APIs, job start/progress/reconnect/cancel/resume, activity SSE with polling fallback, safety decisions, Library/Analyze/Stats behavior, prerequisite blockers, and rejection of a second concurrent mutating job.

Phase C — Album proposals

aggregate folder metadata from existing DB results;
define configurable naming templates and forbidden characters;
add AI proposal generation, rationale, confidence, editing, and approval.

End-to-end gate: automated journeys cover proposal generation, editing, approval, invalid names, collisions, provider failures, stale proposal versions, and persistence after reload.

Phase D — Guarded renaming

implement plan validation and JSON export;
implement journaled rename/apply/verify;
test interruption at every state transition;
add recovery and rollback UI.

End-to-end gate: automated journeys cover preview/validate/confirm/apply/verify, stale confirmation rejection, collisions and case-only renames, interruption at each journal state, restart recovery, rollback, and refusal to overwrite unexpected files.

Phase E — Immich upload

add credential preflight and redacted command preview;
run one album at a time through immich-go;
parse and persist reports;
add retry, verification, and changed-after-upload warnings.

End-to-end gate: automated journeys cover credential preflight without secret leakage, album scope and command preview, successful upload, exact duplicate, upgrade, retryable failure, uncertain outcome verification, cancellation/resume, and stale-byte blocking after EXIF changes.

Phase F — Archive lifecycle

configure archive locations and stable volume/media identities;
implement archive preview, capacity checks, and protected thumbnails;
implement journaled copy-verify-remove and same-filesystem move paths;
retain offline assets in hash indexes and duplicate review;
implement mount detection, restore plans, and recovery UI;
test interruption before and after source removal.

End-to-end gate: automated journeys cover archive preview and capacity blockers, copy/hash verification, same- and cross-filesystem paths, offline browsing through protected thumbnails, interruption before and after source removal, mount changes, restart recovery, restore, and collision-safe restoration.

Phase G — Hardening

resolve every donor-ledger row, document intentional deltas, and archive the CLI sources only after their replacement parity and end-to-end gates pass;
migrate remaining CSV state into SQLite;
add end-to-end tests with a temporary library and fake uploader;
test API authorization assumptions, path validation, stale versions, and malformed requests;
test thumbnail orientation, HEIC decoding, corruption, regeneration, and cache invalidation;
test case-only renames, collisions, interruptions, EXIF failures, and stale hashes;
document backup and recovery procedures.

End-to-end gate: the complete automated story matrix passes as one reproducible command against a fresh temporary environment, including fault injection, restart recovery, authorization/path attacks, malformed inputs, supported image formats, and the full discovery-to-upload-and-archive smoke journey.

14. Definition of done

The integrated application is successful when a new folder can be taken from discovery to verified Immich upload and optional archive without using separate commands, while still allowing every stage to be stopped, reviewed, resumed, restored, or rerun safely. At every point the UI can answer:

What stage is this asset or album in?
Why is it blocked?
What will the next action change?
Can that change be reversed?
Which exact bytes were uploaded to Immich?
Where is the original now, and how can it be restored?

In addition, every user story is linked to a passing automated acceptance test, every phase's end-to-end gate passes, and the accumulated end-to-end regression suite runs successfully from a clean environment with one documented command. Manual testing is exploratory evidence only and never substitutes for these automated checks. Every legacy CLI donor-ledger row is resolved, all intentional behavior changes are tested and documented, and the frozen CLI archive is sufficient to trace the origin of reused behavior without being a runtime dependency.

15. Architecture challenge and risk register

The design is viable, but it can fail in several predictable ways. These risks should be treated as design inputs, not deferred hardening.

Critical risks

Risk	Failure mode	Required mitigation
False duplicate decision	distinct burst/cropped photos are grouped and excluded	auto-resolve only exact byte/pixel matches; fuzzy matches require review; decisions reversible
Identity ambiguity	an old path disappears and several matching new paths appear	preserve the old asset; create occurrences; ask which path is canonical; never infer deletion
Files change during work	user/editor modifies or moves a file after validation	stat/hash before and after; optimistic asset version; fail stale operations instead of continuing
SQLite writer contention	many worker threads produce `database is locked` or long stalls	one serialized DB-write channel, short transactions, bounded queue, busy timeout, backpressure
Crash between filesystem and DB	rename succeeds but DB update does not, or EXIF partly completes	durable operation journal, idempotent recovery, pre/postconditions, never hold DB transaction during tool call
Duplicate job execution	stale heartbeat causes the same item to run twice	leases, idempotency keys, compare-and-set transitions, idempotent handlers; assume at-least-once delivery
Wrong upload state	`immich-go` output changes or process dies after server accepted files	store exact pre-upload hash, raw report, versioned parser; use `unknown_requires_verification`, not success
Archive data loss	active source is removed before archive bytes are durable	copy, close, hash-verify, manifest, then remove; journal every item; never overwrite unexpected destinations
Offline canonical uncertainty	fuzzy match targets an archived original that cannot be opened	retain protected preview/evidence; require archive mount for full-resolution decisions
Resource exhaustion	large images/high concurrency exhaust RAM, file descriptors, GPU, or disk	bounded pools and queues, pixel limits, cache quota, admission control, metrics
External path access	symlink or race escapes the configured library	resolve and validate at operation time, reject symlink escapes, use asset IDs, repeat validation before writes
Local web attack	another page triggers mutations through localhost	random session token, strict Origin/Host checks, SameSite cookie, CSRF protection, no permissive CORS

Deduplication risks

Perceptual hashes have false positives, particularly for screenshots, blank images, graphics, bursts, and similarly framed portraits.
They have false negatives for crops, rotations, overlays, strong color edits, or decoder differences.
Normalized pixel hashes can change when color-management or decoder versions change.
Automatically attaching a newly discovered file to an already-reviewed cluster can propagate an old mistake.
Selecting one canonical can discard useful metadata present only on another copy.

Mitigations:

store hash algorithm and decoder versions;
keep exact, normalized-pixel, and perceptual evidence separately;
define confidence bands rather than a single pHash threshold;
require visual review for fuzzy matches and for cluster merges;
preserve all files and make canonical decisions reversible;
show metadata differences and optionally merge approved metadata before exclusion;
reopen a cluster when a new member contradicts its existing dimensions/date/evidence;
keep not_duplicate negative links so rescans do not repeatedly suggest rejected pairs.

Filesystem and EXIF risks

A file can be moved, replaced, truncated, or edited between discovery and action.
exiftool can rewrite the whole container, fail after creating a temporary file, or behave differently by format.
Filesystems may be case-insensitive, normalization-sensitive, remote, or lack atomic directory rename guarantees.
Symlinks can redirect a previously validated path.
File timestamps can change due to metadata writes and cannot identify content.
Rename rollback may be impossible if another process creates the old destination.

Required behavior:

store a lightweight snapshot (device, inode where useful, size, mtime_ns, hash, asset_version) when planning;
revalidate immediately before mutation and again afterward;
perform EXIF writes one asset at a time with a durable item state;
verify safety keywords and analysis fields by reading EXIF back;
treat cross-filesystem moves as copy-verify-delete and require stronger confirmation;
refuse rename rollback when either path no longer matches recorded preconditions;
never overwrite an unexpected destination;
mark ambiguous outcomes for manual recovery instead of guessing.

External integration risks

Vision providers rate-limit, return malformed JSON, change model behavior, or block content despite an SFW review.
A model or prompt upgrade makes old and new descriptions inconsistent.
NSFW model downloads or GPU libraries may be unavailable.
immich-go flags and report text can change between versions.
Immich may accept an upload even if the local process loses its response.

Mitigations:

persist provider/model/prompt/schema versions with every result;
validate structured responses and quarantine invalid results;
use bounded retries with jitter and a circuit breaker, never infinite retry;
pin and record supported immich-go versions;
keep raw subprocess output and parse through version-specific adapters;
represent uncertain external outcomes explicitly and provide verification/retry tools;
make every external step restartable at item level.

Migration and operational risks

Importing the current path-keyed DB can accidentally merge records or lose FTS data.
Running old CLIs while the new app is active can bypass locks and mutate files.
Database corruption or an incorrect migration can strand the whole library state.
Logs, raw model responses, and thumbnails can expose private information.

Mitigations:

migration begins with a verified online backup and produces a reconciliation report;
run migrations transactionally where SQLite permits and test downgrade/recovery;
add a library-level application lock understood by both new and migrated CLI paths;
warn or refuse when an incompatible legacy process is detected;
keep application data owner-readable only and support log/thumbnail retention limits;
provide database integrity checks and backup restore drills.

16. Concurrency, locking, and race-condition model

High internal concurrency is allowed for image decoding, hashing, inference, and API calls. It must not translate into uncontrolled database or filesystem concurrency.

Concurrency lanes

Scanner/read pool       bounded threads, read-only filesystem work
CPU image pool          bounded processes for decode/hash/thumbnail work
GPU safety lane         normally one process/bounded batch queue
Vision API pool         bounded async requests with rate limiter
Database writer         one serialized writer queue
Filesystem mutator      one library-wide lane for EXIF/rename operations
Uploader                one active album batch by default
Archiver                one active archive plan by default

Defaults must derive from resources, not a single global MAX_WORKERS. For example, thumbnail decoding may allow four workers while GPU inference allows one and network analysis allows twenty. Every lane has a bounded queue; when full, producers block or pause rather than allocating indefinitely.

Database rules

SQLAlchemy sessions are never shared across threads or tasks.
Worker threads produce result messages; a dedicated writer persists them in small batches or individual short transactions.
No transaction remains open while hashing, decoding, calling a model, running exiftool, moving files, or uploading.
Reads use separate short-lived sessions. Long UI queries are paged and cannot hold a WAL read transaction indefinitely.
Job claiming is an atomic compare-and-set update from queued to running with a lease owner and expiry.
Asset updates include WHERE id=? AND version=?; zero updated rows means stale data and produces a conflict/retry decision.
The write queue has a maximum depth and exposes pressure metrics.
WAL checkpoints are monitored; repeated checkpoint starvation is an operational warning, not silently ignored.

SQLite still has one writer. If sustained write pressure or UI latency exceeds agreed limits after batching and backpressure, PostgreSQL becomes a planned scale transition. Adding more SQLite writer threads is not the solution.

Lock hierarchy

Locks have a fixed acquisition order to prevent deadlocks:

library lease → stage/job lease → album/folder lease → asset lease

Never acquire a broader lock while holding a narrower one.
Database row/lease ownership is authoritative; in-process mutexes are optimizations.
Leases have owner IDs, heartbeats, expiry, and fencing tokens.
A recovered worker with an old fencing token cannot commit after a new worker has taken ownership.
Read-only thumbnail generation may run with analysis but not for an asset currently being renamed unless it already has a cache hit.
Rename takes an exclusive library mutation lease.
EXIF writes take an asset mutation lease and are blocked during rename.
Upload takes album leases and validates that no EXIF/rename lease is active.
Archive takes an exclusive album mutation lease plus archive-location lease and is blocked by analysis, EXIF, rename, upload, restore, or another archive operation.

TOCTOU protection

Every mutating operation follows:

load asset + version
→ acquire lease/fencing token
→ resolve and validate current path
→ verify expected snapshot/hash
→ perform one external/filesystem action
→ verify result
→ commit state with version + fencing-token check
→ release lease

If the commit loses its version race, the result is not discarded. It is recorded as an ambiguous operation event and reconciled from the actual file state.

Idempotency

Exactly-once execution is not achievable across SQLite, filesystem, subprocesses, and Immich. The design promises at-least-once execution with idempotent recovery.

API mutation requests accept or generate idempotency keys.
Job items have unique (job_type, asset_id, input_version) constraints.
EXIF writes merge desired fields and verify the final value.
Thumbnail writes target content/version keys and use atomic replacement.
Rename operations persist source, destination, and expected hashes before moving.
Upload retries consult both local upload history and Immich/immich-go results.

17. Resource allocation and backpressure

Memory

Decoded images, not file sizes, determine memory pressure. A compressed 100 MB image can require far more RAM when decoded.

reject or specially queue images exceeding configurable pixel/dimension limits;
use Pillow decompression-bomb protection and treat warnings as reviewable errors;
decode thumbnails directly to bounded target sizes where decoder support permits;
avoid retaining full pixel arrays after hashing/thumbnail generation;
limit concurrent decoded-pixel budget, not only worker count;
isolate crash-prone/large decoders in worker processes so memory is reclaimed.

Disk

thumbnail cache has a configurable quota and LRU eviction;
active thumbnails are pinned while a review page references them;
temporary files live beside their destination when atomic rename is required;
startup cleans only recognized stale temporary files, never arbitrary files;
database, WAL, logs, raw responses, backups, and thumbnails expose separate size metrics and low-disk warnings;
mutating stages stop before the disk reaches a safety reserve.

CPU, GPU, and network

CPU pools default to a conservative fraction of available cores;
GPU inference has an explicit single-owner semaphore unless proven safe otherwise;
network concurrency and requests-per-minute are separate controls;
retry traffic consumes the same rate-limit budget as first attempts;
file descriptor use is bounded by closing images/subprocess pipes promptly;
subprocess stdout is streamed with a size cap while the complete raw report may be stored in a rotated file when required.

18. Testing strategy

Testing must prove crash safety and concurrency behavior, not only successful API responses. Tests use synthetic images and temporary directories; they never access the real photo library or _IGNORE/ contents.

Deterministic fixture photo library

Maintain a small, generated, non-private, source-controlled fixture specification from which the test harness creates a fresh photo library for each test. Do not depend on the developer's photo library, network downloads, current time, random model output, filesystem enumeration order, or mutable shared fixture directories. If binary fixtures are committed, they must be redistributable and accompanied by their source and license; prefer reproducible generation where format support permits it.

The fixture library must include enough deliberately distinct files and folders to exercise every workflow state:

root-level photos, nested albums, empty folders, similarly named folders, Unicode and whitespace in names, long names, mixed-case extensions, and names that collide on case-insensitive or Unicode-normalizing filesystems;
an exclusion-policy sentinel whose identifier is asserted to be absent from scan results, counts, logs, provider calls, thumbnails, and upload requests; tests must never open or inspect excluded content to prove exclusion;
unique synthetic scenes with known visible properties, including zero, one, and multiple people; indoor/outdoor; day/night; readable synthetic signs; portrait and landscape orientation; transparency; and simple SFW/NSFW-classification fixtures;
exact byte copies, renamed copies, same pixels with different metadata, EXIF-only changes, resized/recompressed variants, orientation variants, crops, bursts, watermarks, color edits, blank images, and intentionally similar non-duplicates;
JPEG, PNG, WebP, TIFF, and HEIC/HEIF where the pinned CI decoder supports them, plus uppercase extensions, unsupported extensions, corrupt/truncated images, zero-byte files, and decompression-bomb/huge-dimension headers that remain safe to test;
files with no EXIF, representative existing user EXIF, previous safety keywords, previous analysis metadata, conflicting managed fields, malformed metadata, known timestamps, and timezone/offset edge cases;
read-only, missing, externally moved, externally copied, replaced-after-scan, and symlink-escape cases created by the harness only on platforms that support them;
album/upload/archive cases for valid names, invalid names, collisions, offline archive media, insufficient capacity, already-uploaded bytes, changed-after-upload bytes, and restore destination collisions.

A versioned machine-readable manifest is the authority for the corpus. For every fixture it records a stable logical ID (never an absolute path), relative path, role, generation recipe or source/license, byte SHA-256, decoded dimensions/orientation, relevant metadata, duplicate-family membership and confidence class, expected stage eligibility, and expected fake-provider outcomes. Expected results are keyed by logical ID so tests remain valid after copying the corpus into a random temporary directory. Hash and thumbnail goldens also record algorithm, decoder, encoder, and fixture-schema versions; intentional changes require an explicit golden update and review.

Fixture generation must be deterministic: pin image-library/codec versions in CI; use fixed seeds for pixels and generated names; fix clock, timezone, and locale; write explicit timestamps instead of using creation time; sort all discovered paths; avoid lossy re-encoding unless its exact output is committed and checksummed; and skip a format with an explicit platform marker when byte-identical generation cannot be guaranteed. Tests compare semantic pixels/metadata where cross-platform encoders are not byte-stable.

Deterministic fakes map stable fixture IDs to fixed vision JSON, safety scores, exiftool behavior, and immich-go reports. The manifest includes success, delay, rate-limit, malformed-response, retry, cancellation, and uncertain-outcome scripts. Timing-sensitive tests synchronize on observable events or fault-controller barriers, never sleeps. Each test clones or regenerates the minimal named fixture subset, and the harness verifies its checksums before startup, resets the database/cache/archive, and fails on undeclared files or unexpected provider/uploader calls.

Keep the default corpus small enough for every change while retaining a separate extended corpus for format/platform and soak coverage. CI must provide one fixture validation command that regenerates the corpus twice, compares manifests/checksums, validates all expected relationships, and proves that a complete test run leaves the source corpus unchanged.

Concrete fixture-corpus TODO checklist

Build the default corpus below as the first implementation target. Counts describe logical source images; explicitly listed copies/variants are additional files. All people and sensitive-content fixtures must be synthetic, illustrated, generated, or properly licensed adult-only material—never personal photos and never minors in any sexualized or ambiguous context.

Folder topology

Create 00_root/ behavior with 3 supported images placed directly at the library root: 1 JPEG, 1 PNG with transparency, and 1 uppercase .JPG.
Create 3 ordinary album folders with distinct conditions: 01_family_day/ (outdoor/day/people), 02_city_night/ (outdoor/night/readable sign), and 03_home_indoor/ (indoor/mixed lighting/no location hint), with 5 unique source images each.
Create 3 nested leaf albums under one parent, with 2 unique images per leaf, to prove leaf-album counts are not collapsed into their parent.
Create 3 naming-edge folders containing 1 image each: one with spaces, one with Unicode characters, and one mixed-case name. Add a pair whose names differ only by case and a pair using composed/decomposed Unicode for collision simulation.
Create 1 empty album folder and verify it is handled consistently and never invents an asset.
Create the exclusion-policy sentinel directory and assert only from outside the boundary that it is never traversed, counted, logged, thumbnailed, sent to a provider, or uploaded.

Known visual content

Create 5 deterministic NSFW-class fixtures spanning the fake classifier's score bands: 2 clearly NSFW, 1 boundary/review-required, and 2 clearly SFW controls that are visually similar enough to catch simplistic color/skin heuristics.
Create 5 additional SFW fixtures: 1 landscape with no people, 1 single adult, 1 group of 3 adults, 1 indoor pet, and 1 document/screenshot with readable synthetic text.
Across the ordinary albums, include at least 3 portrait, 3 landscape, and 2 square images; 3 indoor and 3 outdoor scenes; and morning, afternoon, evening, night, and unknown time-of-day expectations.
Include exactly 3 synthetic location-hint cases: 1 readable street sign, 1 unmistakable generated landmark, and 1 ambiguous sign that must yield no confident location.

Duplicate and similarity families

Family A: 1 canonical JPEG plus 2 byte-identical copies with different paths.
Family B: 1 canonical image plus 2 files with identical decoded pixels but different EXIF/timestamps.
Family C: 1 canonical plus 3 near-duplicates: resized, JPEG-recompressed, and WebP-converted.
Family D: 1 canonical plus 2 orientation variants: EXIF rotation and physically rotated pixels.
Family E: 1 canonical plus 3 review-required variants: crop, watermark, and color edit.
Create 3 intentionally similar non-duplicate pairs: blank/near-blank images, burst-like frames with a meaningful subject change, and two screenshots sharing the same layout. Expected decisions must be not_duplicate or review_required, never automatic merge.

Formats and damaged inputs

Include at least 3 JPEG, 3 PNG, 2 WebP, 2 TIFF, and 2 HEIC/HEIF files in the extended corpus; keep one supported example of each CI-stable format in the default corpus.
Add 1 unsupported-extension file, 1 zero-byte file, 1 truncated JPEG, 1 corrupt PNG, and 1 safe synthetic huge-dimension/decompression-bomb header.
Add 1 grayscale, 1 transparent RGBA, 1 wide-gamut/profile-bearing, and 1 very small image to exercise decoding and thumbnail normalization.

Metadata and time cases

Create 2 files without EXIF, 2 with ordinary user EXIF that must survive, 2 with existing safety keywords, 2 with existing analysis fields, 1 with conflicting managed fields, and 1 with malformed metadata.
Create 5 fixed capture-time cases: UTC, positive offset, negative offset, daylight-saving boundary, and missing timezone. Freeze filesystem mtimes separately so capture-time fallback behavior is explicit.
Create 1 file whose EXIF write changes its byte hash without changing decoded pixels, then assert upload uses the post-checkpoint hash.

Runtime mutations produced by the harness

From clean source fixtures, create one moved file, one copied file, one renamed folder, one file replaced after scan, one file modified after upload, and one file removed after scan.
Where supported, create one read-only file, one read-only destination folder, one safe in-library symlink, and one symlink escape attempt.
Create 3 rename-plan cases: valid multi-file rename, destination collision, and case-only rename; add one stale plan by modifying a source after validation.
Create 4 fake-upload results: new upload, exact duplicate, server upgrade, and accepted-but-response-lost uncertainty.
Create 3 archive cases: successful copy/verify, insufficient capacity, and interruption after verification but before source removal; add one offline restore and one restore-destination collision.

Manifest and acceptance totals

Give every file a stable logical ID and manifest entry; declare its expected discovery status, safety class, analysis response, duplicate family/decision, EXIF projection, album membership, and upload/archive eligibility.
Record the exact expected counts after each workflow stage, including canonical, duplicate, review-required, excluded, error, analyzed, EXIF-verified, uploaded, and archived totals. Tests must compare the complete count object, not selected fields.
Add a corpus-lint test that fails on missing IDs, duplicate IDs, undeclared files, checksum drift, inconsistent family membership, missing expected outcomes, or a fixture not exercised by at least one automated test.
Add a reproducibility test that generates the corpus twice in separate temporary roots and compares manifests, relative paths, semantic metadata, and checksums where byte stability is promised.

Unit tests

path-policy normalization, symlink escape rejection, and _IGNORE/ exclusion;
normalized pixel hashing across EXIF-only changes and orientation variants;
pHash distance/confidence classification and negative-link behavior;
canonical recommendation scoring;
stage-gate evaluation and blocker explanations;
state-machine transitions for jobs, renames, uploads, and duplicate decisions;
idempotency-key and optimistic-version handling;
naming sanitization across macOS/Linux and case-insensitive collision simulation;
immich-go parser fixtures for every supported version and unknown output.

Use property-based tests for path transformations, rename plans, hash grouping, state machines, and arbitrary API payloads. Important properties include: paths never escape the library, canonical clusters never contain cycles, retries do not duplicate events, and applying a valid rename plan preserves the asset set.

Golden-image tests

Maintain a small generated/non-private corpus containing:

exact copies with different names;
identical pixels with different EXIF;
resized and recompressed copies;
rotations represented by EXIF and by transformed pixels;
crops, watermarks, color edits, screenshots, blank images, and burst-like images;
JPEG, PNG, WebP, TIFF, HEIC fixtures where licensing permits;
corrupt, truncated, huge-dimension, and unsupported files.

Assert hashes, confidence bands, thumbnail orientation/dimensions, and which cases must require human review. Do not assert that fuzzy hashing is universally correct.

Repository and API integration tests

run every migration from an empty DB and from anonymized snapshots of each supported historical schema;
verify foreign keys, FTS rebuilds, WAL settings, online backup, and integrity checks;
test API status codes, schemas, pagination, idempotency, stale versions, CSRF/Origin, path validation, and event-stream reconnect behavior;
replace vision, NSFW, exiftool, and immich-go integrations with deterministic fakes that can succeed, delay, emit malformed output, or fail at chosen points.
snapshot representative EXIF before and after every stage; assert only owned fields change, previous-stage values survive, read-back matches the desired projection, and the stored post-write hashes match the actual file bytes.

Black-box API end-to-end tests

API end-to-end tests must not import service or repository internals. They launch the real FastAPI server and worker as child processes, create a temporary application-data directory and photo library, and communicate only over HTTP/SSE. This validates the same process boundaries used in production.

Recommended harness:

pytest owns a session-scoped temporary directory and allocates free ports;
Alembic creates a fresh SQLite database through the production startup path;
the API and worker receive an isolated test configuration through environment vars;
deterministic fake executables/providers replace exiftool, the vision API, NSFW inference, and immich-go, but are invoked through the real integration layer;
readiness waits on /api/v1/health/ready, never an arbitrary sleep;
tests use an HTTP client and consume SSE events with reconnect support;
processes are terminated and checked for leaked children after each test session.

Do not add a powerful test-only mutation API to production. Fixtures are prepared before process startup through files, migrations, and fake integrations. When runtime fault control is required, use a separate fake-provider control server bound to its own random localhost port and enabled only by test configuration.

Core API scenarios:

Fresh workflow — scan, wait for job completion, review duplicate cluster, mark safety decisions, verify the safety EXIF checkpoint, analyze, verify the analysis EXIF checkpoint and preservation snapshot, generate/apply rename plan, upload, archive, restart, and verify the offline asset remains searchable/deduplicable.
Move and copy outside the app — stop jobs, manually move A→B and copy B→C, rescan, verify the original asset ID survives and C becomes a duplicate candidate.
Idempotent commands — submit the same idempotency key concurrently and assert one job/decision plus identical responses.
Optimistic conflict — submit a decision with an old asset/cluster version and require 409 Conflict without mutation.
Cancellation and resume — cancel each job type mid-item, restart processes, and verify a valid resumable/terminal state.
Lease recovery — kill the worker, let its lease expire, start another worker, then prove the old worker cannot commit with its stale fencing token.
Safety race — change an asset to NSFW while an analysis fake is delayed; the eventual analysis result must be discarded and must not write analysis EXIF. The asset remains upload-eligible after its NSFW EXIF keyword is verified.
Rename uncertainty — change a source after plan validation and assert apply is rejected without moving any unaffected paths.
Unknown upload result — fake server acceptance followed by process failure; require unknown_requires_verification, never automatic retry/success.
Privacy boundary — verify _IGNORE/ identifiers never appear in provider or uploader logs; NSFW identifiers never appear in analysis-provider logs but do appear in uploader logs when their album is explicitly uploaded.
External EXIF edit — add user metadata after a verified checkpoint, rescan, and verify the next stage preserves the addition; remove or conflict with a managed value and require divergent instead of silent repair/upload.
Archive interruption — terminate during transfer, after archive verification, and after active-source removal; restart and assert no verified source is lost or removed twice.
Offline deduplication — unmount the fake archive, discover an exact and a fuzzy active copy, and verify hash matching plus protected-thumbnail review without treating the archived canonical as unexpectedly missing.

For every scenario, assert both API state and durable state after a full server/worker restart. An HTTP 200 before restart is not sufficient evidence of durability.

API contract tests also validate the generated OpenAPI schema. Breaking response or enum changes require an explicit API version/migration decision rather than silently breaking the frontend.

Playwright browser end-to-end tests

Playwright tests run against the same real API/worker test stack. They validate user journeys and visible safety controls, while API tests remain responsible for exhaustive state combinations. Browser tests should not duplicate every backend assertion.

Use stable accessible locators first (getByRole, labels, names) and add data-testid only for dynamic structures without reliable semantics. Never locate by CSS styling classes or incidental text generated by AI fixtures.

Critical browser journeys:

Workflow overview — stage counts render, blockers are understandable, and disabled actions explain their prerequisite.
Duplicate comparison — thumbnails load in the correct orientation; keyboard navigation, synchronized zoom, blink/overlay, metadata evidence, and canonical selection work; fuzzy matches require explicit confirmation.
Safety review — individual and bulk SFW/NSFW decisions persist after reload; filters/counts update; an NSFW asset disappears from analysis-eligible totals while remaining visible in upload-eligible totals.
Analysis job — start, live progress via SSE, reconnect after page reload, cancel, resume, and inspect an error without losing completed work.
Album proposal — edit a suggestion, detect collision/invalid name, preview the complete path plan, and verify stale confirmation is rejected visibly.
Rename recovery — simulate a controlled interrupted rename, reopen the UI, and confirm the recovery screen prevents unrelated mutations.
Upload preflight — blocked reasons render, secrets never appear in DOM/logs, exact album scope is shown, and uncertain results require verification.
Move/copy reconciliation — after the harness changes files externally, rescan from the UI and resolve the newly formed duplicate cluster.
Archive and restore — preview reclaimable bytes, inspect blockers, confirm an archive plan, observe progress/recovery, browse the offline asset from its retained thumbnail, and restore it when the fake archive location is mounted again.
Accessibility and keyboard — complete duplicate and safety decisions without a mouse; focus remains visible; dialogs trap/restore focus; status is not color-only.
Responsive layout — desktop is primary, but tablet/narrow widths retain access to blockers, decisions, logs, and confirmations without horizontal action loss.

Playwright execution policy:

each test gets a fresh browser context and deterministic seeded library state;
tests wait for semantic UI state or API responses, never fixed timeouts;
external provider calls are observed at the fake integration boundary, not mocked inside the browser, so frontend-to-backend behavior remains real;
use Playwright's API request context only for setup when the action is also available through the public API and does not bypass the behavior under test;
run Chromium on every change; run Firefox/WebKit on scheduled or release builds;
collect trace, console, network log, screenshot, and video on first retry/failure;
fail on uncaught page errors, unexpected console errors, failed API responses, and unhandled dialogs;
visual snapshots cover only stable layouts/components, with deterministic fonts, viewport, timezone, locale, animation disabling, and generated thumbnails;
accessibility scans complement—not replace—keyboard journey assertions.

Avoid large all-in-one browser tests. Each journey should establish state through the API, perform the behavior under test through the UI, and assert the visible result plus one authoritative API outcome. Keep one longer smoke journey for the complete pipeline.

Test-stack topology

pytest / Playwright
      │
      ├── browser ───────────────► real FastAPI + static frontend
      │                                  │
      ├── HTTP/SSE client ───────────────┤
      │                                  ▼
      │                            isolated SQLite DB
      │                                  ▲
      └── fixture/fault controller       │
              │                    real worker process
              └── fake vision / NSFW / exiftool / immich-go

The harness must expose a single command for local and CI use, for example:

pytest tests/e2e/api
playwright test tests/e2e/web
pytest tests/e2e/full_pipeline

Exact commands depend on the final frontend tooling, but both suites must provision the same stack through one reusable test-harness module.

Concurrency and race tests

Run repeatedly with randomized timing:

many hash/thumbnail workers completing simultaneously into the DB writer queue;
two workers attempting to claim the same job;
lease expiry followed by a late commit from the old worker;
user moves/replaces a file between validation and EXIF write;
thumbnail request races with rename and cache eviction;
analysis result races with a safety decision changing to NSFW;
upload preparation races with an EXIF update;
two rename plans target the same destination;
long UI reads create WAL checkpoint pressure;
cancellation arrives during decoding, API retry, EXIF write, rename, and upload.

Assertions include no deadlocks, bounded queue depth, no duplicate terminal state, no lost audit events, no writes from stale fencing tokens, and eventual database integrity.

Crash/fault-injection tests

Use real child processes and terminate them at every persisted transition:

after journal write but before rename;
after filesystem rename but before DB update;
during EXIF subprocess execution;
after upload acceptance but before report persistence;
while a DB writer batch is committing;
while thumbnail temporary files exist.

Restart the application and assert it either resumes safely or presents a precise manual-recovery state. Never accept silent guessing as a passing result.

Fault injection should also cover disk full, read-only directories, permission changes, database busy timeouts, corrupt DB copies, network timeouts, malformed model JSON, GPU out-of-memory, subprocess hangs, and unavailable binaries.

End-to-end workflow tests

Build a temporary library, including an _IGNORE/ sentinel, then run:

scan → duplicate decisions → safety review → analysis → EXIF verification
→ album proposal → rename → rescan/reconciliation → fake Immich upload
→ archive → offline deduplication → restore

Repeat after manually moving and copying files between stages. Assert stable asset IDs, preserved decisions, duplicate exclusion, verified EXIF checkpoint order, upload hashes, archive manifests, offline hash-index membership, successful restore, and that the sentinel is never discovered, counted, opened, or modified.

Load and soak tests

synthetic databases at 25k, 100k, and 500k asset rows;
duplicate clusters with thousands of members;
sustained thumbnail browsing while analysis writes complete;
multi-hour worker soak with cancellation/retry and periodic backups;
cache eviction under quota pressure;
memory/RSS, open-file, DB latency, WAL size, queue depth, and event throughput limits.

Define acceptance budgets before implementation, for example:

API list/search p95 under 250 ms on 100k rows;
no unbounded queue or cache growth;
stable RSS during a multi-hour run;
clean restart with no lost completed item after forced termination;
zero automatic fuzzy-duplicate exclusions without recorded user approval.

Release gates

Unit/property/golden suites pass.
Migration test succeeds from a copied current database.
Concurrency suite passes repeatedly with randomized scheduling.
Every rename crash point passes recovery tests.
Black-box API workflow passes through real server and worker processes, including a restart after every completed stage.
Critical Playwright journeys pass in Chromium with no unexpected browser console, network, accessibility, or page errors.
Scheduled cross-browser Playwright suite passes before a release.
Full end-to-end pipeline passes with fake external services and retained artifacts.
Archive crash/recovery suite proves that active sources are removed only after verified durable copies and remain deduplicable while offline.
A read-only dry run against the real library produces an approved reconciliation report before any mutation is enabled.
The donor ledger has no unresolved in-scope rows; characterization/parity tests pass, intentional deltas are approved, and archived CLI sources are not imported or executed by the production application.

93 KiB Raw Permalink Blame History

Photo Library Pipeline — Integrated App Concept

1. Product goals

2. Canonical workflow

Gate rules

3. Core architecture

Real application, not CLI scripts with attached pages

Domain services

Donor-first CLI migration and archival

API boundaries

Persistent job model

One durable database

Per-asset workflow state

Dependency and invalidation rules

Additional tables

Database ownership and transactions

EXIF ownership and consistency checkpoints

4. Stage 0 — Inventory and duplicate review

Discovery

Duplicate review

Viable thumbnail pipeline

Fuzzy duplicate comparison UI

5. Stage 1 — Safety identification and EXIF tagging

6. Stage 2 — Content analysis and analysis EXIF

7. Stage 3 — AI album suggestions and guarded renaming

Inputs for an album proposal

Suggested naming format

Proposal review UI

Guarded rename plan

Crash-safe rename transaction

8. Stage 4 — Immich upload via immich-go

Safety and upload eligibility

Preflight

Upload batches

9. Stage 5 — Archive and active-storage reclamation

Availability states

Archive preflight

Transfer and removal semantics

Future deduplication against archived assets

Restore

10. Information architecture and UX

Frontend architecture

Global header

Workflow home

Shared interaction rules

11. Consistency rules

12. Proposed project layout

13. Recommended implementation phases

Automated acceptance policy for every phase

Phase A — Shared identity and inventory

Phase B — Unified workflow shell

Phase C — Album proposals

Phase D — Guarded renaming

Phase E — Immich upload

Phase F — Archive lifecycle

Phase G — Hardening

14. Definition of done

15. Architecture challenge and risk register

Critical risks

Deduplication risks

Filesystem and EXIF risks

External integration risks

Migration and operational risks

16. Concurrency, locking, and race-condition model

Concurrency lanes

Database rules

Lock hierarchy

TOCTOU protection

Idempotency

17. Resource allocation and backpressure

Memory

Disk

CPU, GPU, and network

18. Testing strategy

Deterministic fixture photo library

Concrete fixture-corpus TODO checklist

Unit tests

Golden-image tests

Repository and API integration tests

Black-box API end-to-end tests

93 KiB

Raw Permalink Blame History