# Donor ledger — US01-01 (INTEGRATED_PIPELINE_CONCEPT.md §3 "Donor-first CLI migration") # # Maps every relevant legacy module/function to its classification, rationale, # target module in the future photo_pipeline package, and characterization # test IDs. Linted by tests/characterization/test_donor_ledger.py. # # classification: reuse — take as-is # extract — move behind a thin adapter, behavior unchanged # refactor — extract while changing structure, behavior preserved # unless an intentional delta is stated # replace — new implementation; rationale documents why and what # (if anything) carries over # Every row needs either `tests` (existing test IDs, module::function) or # `pending_story` (the backlog story that will characterize/deliver it). # status: characterized | pending rows: # ── photo_analyzer.py ────────────────────────────────────────────────────── - id: pa-discovery area: discovery source: {file: photo_analyzer.py, symbols: [discover_photos, SUPPORTED_EXTENSIONS]} classification: extract rationale: > Proven recursive discovery with _IGNORE/ and .@__thumb exclusion, sorted + deduplicated. Becomes the one shared discovery used by every stage; the exclusion predicate moves to path_policy.py. target: photo_pipeline/services/inventory.py + photo_pipeline/path_policy.py tests: - test_pa_discovery::test_discover_excludes_ignore_and_thumbs - test_pa_discovery::test_discover_finds_supported_including_uppercase - test_pa_discovery::test_discover_sorted_and_deduplicated - test_pa_discovery::test_supported_extensions_contract status: characterized - id: pa-purge-excluded area: discovery source: {file: photo_analyzer.py, symbols: [purge_excluded]} classification: refactor rationale: > SQL LIKE patterns duplicate the path policy; refactor to call the shared path_policy predicate so exclusion has exactly one definition. target: photo_pipeline/services/inventory.py tests: [test_pa_db::test_purge_excluded_removes_ignore_and_thumb_rows] status: characterized - id: pa-prune-missing area: database source: {file: photo_analyzer.py, symbols: [prune_missing]} classification: replace rationale: > Deletes rows for missing files — incompatible with stable asset identity; the concept keeps the asset and sets missing_at/availability_state. The unmounted-library guard (never prune when the root is absent) carries over as behavior. target: photo_pipeline/services/inventory.py tests: [test_pa_db::test_prune_missing_guards_unmounted_library] status: characterized - id: pa-variants area: discovery source: {file: photo_analyzer.py, symbols: [_strip_variant_markers, _is_variant, _variant_key, build_variant_groups]} classification: extract rationale: > Filename-based variant grouping (pixel-size suffix, 'bearbeitet' edit marker) with unmarked-largest primary selection; feeds duplicate/variant clustering evidence. target: photo_pipeline/services/duplicates.py tests: - test_pa_variants::test_strip_variant_markers_goldens - test_pa_variants::test_is_variant - test_pa_variants::test_variant_key_folder_scoped_case_insensitive - test_pa_variants::test_build_variant_groups_primary_is_largest_unmarked status: characterized - id: pa-variant-propagate area: database source: {file: photo_analyzer.py, symbols: [propagate_variants, _row_to_result]} classification: refactor rationale: > Copying a primary's analysis into variant rows survives, but keyed by asset_id and recorded as stage state instead of raw row copies. target: photo_pipeline/services/duplicates.py pending_story: US01-04 status: pending - id: pa-hashing area: hashing source: {file: photo_analyzer.py, symbols: [_sha1_file, _phash_image]} classification: extract rationale: > Streamed SHA-1 plus 64-bit DCT phash (imagehash.phash recipe on numpy/scipy, no extra dependency). Hash outputs are goldens — a change means the decoder/algorithm changed and hash_version must bump. target: photo_pipeline/services/inventory.py tests: - test_pa_hashing_dedup::test_phash_goldens - test_pa_hashing_dedup::test_phash_survives_resize_and_recompress - test_pa_hashing_dedup::test_phash_format_16_hex_chars - test_pa_hashing_dedup::test_sha1_matches_hashlib - test_pa_hashing_dedup::test_sha1_unreadable_returns_none status: characterized - id: pa-ensure-hashes area: hashing source: {file: photo_analyzer.py, symbols: [ensure_hashes]} classification: extract rationale: > Resume-safe incremental hashing (skip already-hashed, COALESCE so a failed hash never clobbers a stored one, periodic commits, honors stop event). target: photo_pipeline/services/inventory.py tests: [test_pa_hashing_dedup::test_ensure_hashes_skips_hashed_and_is_resume_safe] status: characterized - id: pa-cluster area: hashing source: {file: photo_analyzer.py, symbols: [cluster_duplicates]} classification: extract rationale: > Union-find over the vectorized Hamming graph; largest-first ordering. Known ceiling (documented in-source): O(n²) scan, BK-tree past ~100k. target: photo_pipeline/services/duplicates.py tests: [test_pa_hashing_dedup::test_cluster_and_mark_duplicates_largest_is_canonical] status: characterized - id: pa-mark-duplicates area: database source: {file: photo_analyzer.py, symbols: [mark_duplicates]} classification: refactor rationale: > Intentional delta — the concept forbids silently marking fuzzy matches: largest-file canonical becomes a *recommendation*; the decision lands in reviewable duplicate_clusters, auto-resolve only for exact matches. The status='duplicate' exclusion from analysis/upload carries over. target: photo_pipeline/services/duplicates.py tests: [test_pa_hashing_dedup::test_cluster_and_mark_duplicates_largest_is_canonical] status: characterized - id: pa-list-duplicates area: ui source: {file: photo_analyzer.py, symbols: [list_duplicates]} classification: replace rationale: Console report; superseded by the duplicate-review API/UI (US01-06). target: photo_pipeline/api/routes + frontend duplicate review pending_story: US01-06 status: pending - id: pa-reconcile area: database source: {file: photo_analyzer.py, symbols: [reconcile_moved]} classification: refactor rationale: > SHA-1-keyed move/rename reconciliation preserving analysis + EXIF state. Refactor to update assets.current_path by asset_id and append an asset_paths history row instead of rewriting the path in place. target: photo_pipeline/services/inventory.py tests: [test_pa_hashing_dedup::test_reconcile_moved_preserves_row_by_sha1] status: characterized - id: pa-db-schema area: database source: {file: photo_analyzer.py, symbols: [SCHEMA, get_db, _migrate_schema]} classification: refactor rationale: > photos table + FTS5 + idempotent ALTER migration become the migration baseline; new schema adds assets/asset_paths/stage state per concept. Existing data must migrate losslessly (US01-02). target: photo_pipeline migrations (Alembic) tests: [test_pa_db::test_schema_and_migration_idempotent] status: characterized - id: pa-db-status area: error source: {file: photo_analyzer.py, symbols: [upsert_pending, mark_analyzed, mark_error, mark_exif_written, get_pending, get_analyzed_no_exif]} classification: refactor rationale: > Status lifecycle (pending→analyzed→exif_written; error rows auto-retried on the next run; INSERT OR IGNORE never downgrades; duplicates never queued) is the resume-safety contract — preserved as asset_stage_states transitions keyed by asset_id. target: photo_pipeline/repositories + services/analysis.py tests: - test_pa_db::test_status_lifecycle_pending_analyzed_exif_written - test_pa_db::test_mark_error_and_retry_via_get_pending - test_pa_db::test_mark_analyzed_clears_error - test_pa_db::test_upsert_pending_never_downgrades status: characterized - id: pa-fts area: database source: {file: photo_analyzer.py, symbols: [photos_fts]} classification: reuse rationale: FTS5 table + sync triggers already work; carried into the new schema. target: photo_pipeline migrations (Alembic) tests: [test_pa_db::test_fts_kept_in_sync_by_triggers] status: characterized - id: pa-imaging area: imaging source: {file: photo_analyzer.py, symbols: [prepare_image, MAX_LONG_EDGE]} classification: extract rationale: > RGB-normalize (drops alpha, converts HEIC), LANCZOS resize to 2048px long-edge, JPEG q85 base64 — the provider-input contract. Truncated-image tolerance (ImageFile.LOAD_TRUNCATED_IMAGES) carries with it. target: photo_pipeline/integrations/vision.py tests: - test_pa_imaging::test_prepare_image_small_passthrough_jpeg - test_pa_imaging::test_prepare_image_resizes_to_max_long_edge - test_pa_imaging::test_prepare_image_converts_png_alpha_to_rgb_jpeg status: characterized - id: pa-repair area: error source: {file: photo_analyzer.py, symbols: [repair_image, _REPAIRABLE_ERRORS, _log_repair]} classification: refactor rationale: > Re-encode structurally broken images via sips (macOS-only — portability note), .orig backup, atomic replace, repairs.jsonl audit. Needs corrupt fixtures to characterize; lands with media hardening. target: photo_pipeline/integrations/exiftool.py pending_story: US07-03 status: pending - id: pa-vision area: vision source: {file: photo_analyzer.py, symbols: [analyze_image, ANALYSIS_PROMPT, LLM_BASE_URL, LLM_MODEL]} classification: extract rationale: > OpenAI-compatible request construction, base64 image_url payload, JSON response validation, 429/503 retry with exponential backoff. Prompt and model/config version must be persisted per analysis_runs. Characterized against a deterministic fake provider when the analysis service is ported. target: photo_pipeline/integrations/vision.py pending_story: US02-06 status: pending - id: pa-throttle area: logging source: {file: photo_analyzer.py, symbols: [_record_ratelimit, _ratelimit_recent, write_throttle_summary, _rpd_load, _rpd_increment, _rpd_count]} classification: refactor rationale: > Rolling throttle window + persistent throttle_events.jsonl + RPD day counter become job metrics/events on the durable job model. target: photo_pipeline/jobs/coordinator.py pending_story: US02-02 status: pending - id: pa-nsfw-filter area: nsfw source: {file: photo_analyzer.py, symbols: [_rec_has_nsfw, filter_nsfw_tagged]} classification: replace rationale: > Intentional delta — donor fails OPEN (exiftool hiccup → analyze everything), which violates the concept privacy invariant; the new gate is the DB safety decision and fails CLOSED (only confirmed sfw reaches the provider). EXIF keyword reading survives as evidence/verification, not as the gate. Current behavior captured for migration. target: photo_pipeline/services/safety.py tests: - test_pa_exif::test_rec_has_nsfw_list_scalar_case - test_pa_exif::test_filter_nsfw_tagged_splits_and_normalizes - test_pa_exif::test_filter_nsfw_tagged_empty_list_noop status: characterized - id: pa-exif-caption area: exif source: {file: photo_analyzer.py, symbols: [build_exif_caption, build_exif_caption_from_result]} classification: refactor rationale: > Caption format ('desc | Tags: … | Mood: … | Location: … | ~year') is what 25k+ photos already carry — preserved for compatibility. Intentional delta: new writes go into a managed 'AI:' segment so re-analysis can replace only its own text (concept EXIF ownership). target: photo_pipeline/services/exif projection tests: - test_pa_exif::test_caption_golden_full - test_pa_exif::test_caption_golden_sparse status: characterized - id: pa-exif-read area: exif source: {file: photo_analyzer.py, symbols: [read_existing_exif]} classification: extract rationale: Targeted exiftool JSON read of the four owned fields, 30s timeout, fail-empty. target: photo_pipeline/integrations/exiftool.py tests: [test_pa_exif::test_write_exif_roundtrip_and_idempotent] status: characterized - id: pa-exif-write area: exif source: {file: photo_analyzer.py, symbols: [write_exif]} classification: refactor rationale: > Merge-don't-overwrite semantics are the crown jewels: caption prepended to existing text (idempotent — never appended twice), keywords merged + order-preserving dedup, -m -overwrite_original, repair-and-retry on structurally broken files. Refactored to add the concept's read-back verification + non-owned-field snapshot comparison. sys.exit on missing exiftool becomes a service error. target: photo_pipeline/integrations/exiftool.py + services/exif checkpoints tests: - test_pa_exif::test_write_exif_roundtrip_and_idempotent - test_pa_exif::test_write_exif_preserves_existing_metadata status: characterized - id: pa-run-loop area: cancellation source: {file: photo_analyzer.py, symbols: [run_analysis, _run_folder_loop, run_exif_write, _stop, _handle_sigint]} classification: refactor rationale: > Folder-grouped worker pool, per-item DB commits, cooperative stop event checked between items, double-SIGINT force quit — becomes the durable JobRunner worker loop with the same drain-and-resume semantics. target: photo_pipeline/jobs/worker.py pending_story: US02-02 status: pending - id: pa-ui-terminal area: ui source: {file: photo_analyzer.py, symbols: [_Dashboard, _LiveRenderer, _PlainRenderer, _make_progress, _stats_panel, _feed_panel, _tokens_panel, _menu_panel, _read_key, _key_listener, print_stats, fit_label]} classification: replace rationale: > Rich terminal dashboard (Live layout, hotkey menu, termios key reader) is superseded by the web Workflow/Analyze views. The *data* it renders (album progress, feed, tokens, ETA) survives via webapp.runner.progress (see wa-runner). fit_label/print_stats die with it. target: frontend Analyze view (Phase B) tests: [test_pa_album::test_fit_label_left_truncates] status: characterized - id: pa-album-label area: ui source: {file: photo_analyzer.py, symbols: [album_label]} classification: extract rationale: > Album = leaf folder relative to library, '(root)' at root, bare parent outside the library. The single album-identity rule for stats, proposals and folder-as-album upload. webapp.query.album_of is a copy — consolidate to one implementation (see wa-album-of). Extracted in US03-01 as photo_pipeline.services.albums.album_label (parity test below); the album evidence aggregator uses it. LibraryService._album_of still keeps its own leaf-name grouping — fold it into this rule when the Library view is next touched. target: photo_pipeline/services/albums.py tests: - test_pa_album::test_album_label_leaf_folder_relative - test_pa_album::test_webapp_album_of_mirrors_album_label - test_pa_album::test_albums_service_album_label_matches_donor status: characterized - id: pa-config area: configuration source: {file: photo_analyzer.py, symbols: [load_env_file, _env_str, _env_int, _env_bool, ENV_FILE]} classification: refactor rationale: > photo_analyzer.env parsing (shell env wins, export prefix, quoted values, inline-comment rules) feeds the typed config module; same file keeps working so the transition needs no re-setup. target: photo_pipeline/config.py tests: - test_pa_config::test_load_env_file_parsing - test_pa_config::test_env_helpers status: characterized - id: pa-logging area: logging source: {file: photo_analyzer.py, symbols: [log_history, _history_handler, _rich_handler]} classification: replace rationale: > Module-level triple-handler logging (console + info log + debug log) and the JSONL history logger become structured JSON logging with job_id/ asset_id and job_events rows; per-photo history maps to job events. target: photo_pipeline structured logging + jobs/job_events pending_story: US02-02 status: pending - id: pa-balance area: vision source: {file: photo_analyzer.py, symbols: [fetch_balance, query_balance, check_quota]} classification: extract rationale: > Provider balance/quota probes (report 'unsupported' on providers without the endpoint). Network-bound; characterized against the fake provider. target: photo_pipeline/services/analysis.py pending_story: US02-06 status: pending - id: pa-cli area: configuration source: {file: photo_analyzer.py, symbols: [main]} classification: replace rationale: > argparse surface is superseded by the API; flags map to job configs (documented in WEBAPP_CONCEPT.md §8 parity table). Transitional CLI calls the shared services until archival (E07). target: photo_pipeline/api + transitional CLI pending_story: US07-01 status: pending # ── nsfwtag/ ─────────────────────────────────────────────────────────────── - id: nt-discovery area: discovery source: {file: nsfwtag/scoring.py, symbols: [discover_images]} classification: refactor rationale: > Merged into the shared discovery. Known divergence captured: nsfwtag EXTS lacks .tiff/.tif, is non-recursive by default, and dedupes by resolved path (symlink guard) — the shared service adopts the superset + symlink dedup, always recursive, plus _IGNORE policy (donor has none). target: photo_pipeline/services/inventory.py tests: - test_nsfwtag::test_exts_contract_differs_from_photo_analyzer - test_nsfwtag::test_discover_images_flat_recursive_limit - test_nsfwtag::test_discover_images_dedupes_symlinked_paths status: characterized - id: nt-score-cache area: nsfw source: {file: nsfwtag/scoring.py, symbols: [load_cache, _save_cache, CSV_NAME]} classification: replace rationale: > nsfw_scores.csv stops being the source of truth (concept: DB state). Format characterized (4-decimal scores, bad rows dropped) because the existing CSV must migrate into assets.safety_score. target: photo_pipeline/repositories (safety), CSV import in US01-02 migration tests: [test_nsfwtag::test_score_cache_roundtrip_and_tolerance] status: characterized - id: nt-score-model area: nsfw source: {file: nsfwtag/scoring.py, symbols: [score_images, MODEL_ID, BATCH]} classification: extract rationale: > On-device ViT scoring: MPS/CPU pick, model-defined input size (not hardcoded — survives model swap), batch inference where one bad batch never discards prior scores, periodic checkpoint every 500. Cache-hit short-circuit (model never loads when nothing is new) characterized now; inference path needs the local model + deterministic fake. target: photo_pipeline/integrations/nsfw_model.py tests: [test_nsfwtag::test_score_images_cache_hit_skips_model] status: characterized - id: nt-exif-keyword area: exif source: {file: nsfwtag/exif.py, symbols: [write_keyword, remove_keyword]} classification: extract rationale: > Idempotent add/remove via '-Keywords-=x -Keywords+=x' (no duplicates on re-run), touching only Keywords+Subject. Basis of the safety EXIF checkpoint; the mutual-exclusion write (sfw removed when nsfw added and vice versa) is the concept's addition on top. target: photo_pipeline/services/safety.py + integrations/exiftool.py tests: [test_nsfwtag::test_write_remove_keyword_idempotent] status: characterized - id: nt-exif-marks area: nsfw source: {file: nsfwtag/exif.py, symbols: [_read_keywords, read_tagged, read_marks]} classification: extract rationale: > Batched exiftool read via stdin, lowercase normalization, and the safety rule that a file carrying both keywords reads as nsfw (never as safe). target: photo_pipeline/services/safety.py tests: - test_nsfwtag::test_write_remove_keyword_idempotent - test_nsfwtag::test_read_marks_nsfw_wins_over_sfw status: characterized - id: nt-exif-view area: exif source: {file: nsfwtag/exif.py, symbols: [read_exif, _EXIF_NOISE]} classification: extract rationale: > Lightbox EXIF read: filesystem-noise filter, binary/oversize skip, signed decimal GPS, Google-Maps link prepended. Backs the asset EXIF endpoint. target: photo_pipeline/api (asset EXIF endpoint) tests: [test_nsfwtag::test_read_exif_filters_noise_and_prepends_map] status: characterized - id: nt-apply-list area: nsfw source: {file: nsfwtag/exif.py, symbols: [apply_list]} classification: replace rationale: > Newline-list bulk tagging (nsfw_confirmed.txt flow) is superseded by DB review decisions; the existing list is a one-time migration input. target: photo_pipeline/services/safety.py (decision import in US01-02) pending_story: US01-02 status: pending - id: nt-ui area: ui source: {file: nsfwtag/server.py, symbols: [serve_review]} classification: replace rationale: > stdlib ThreadingHTTPServer + token-injected review.html backend is superseded by FastAPI. review.html's design tokens, folder tree, threshold/score review flow, lightbox and keyboard model are the frontend donor for the Safety view (preserved per concept §10; ported in US02-01). target: photo_pipeline/api + frontend Safety view pending_story: US02-01 status: pending - id: nt-bench area: nsfw source: {file: nsfwtag/bench.py, symbols: [main]} classification: replace rationale: > Dev-only model benchmark; archived without webapp replacement (recorded basis of the AdamCodd model choice). No production caller. target: none (archive as reference) pending_story: US07-01 status: pending # ── webapp/ ──────────────────────────────────────────────────────────────── - id: wa-query-search area: database source: {file: webapp/query.py, symbols: [_fts_query, _where, search, _row_to_card, photo, _SORTS]} classification: extract rationale: > Safe FTS5 MATCH building (tokenize → quoted prefix terms; punctuation can't crash MATCH), facet WHERE composition, paged search with relevance/sort modes, raw_response kept out of list payloads. target: photo_pipeline/repositories (search) + api/routes tests: - test_webapp_query::test_fts_query_sanitization_goldens - test_webapp_query::test_where_clause_goldens - test_webapp_query::test_search_fts_prefix_match - test_webapp_query::test_search_filters_and_browse - test_webapp_query::test_search_paging - test_webapp_query::test_photo_omits_raw_response status: characterized - id: wa-query-aggregates area: database source: {file: webapp/query.py, symbols: [facets, stats, _col_counts, _people_buckets, _top_tags]} classification: extract rationale: > Facet counts, year range, album tree, status/album progress (done = analyzed + exif_written) and error listing — the Stats/Workflow data model. target: photo_pipeline/repositories + services/workflow.py tests: - test_webapp_query::test_facets_and_albums - test_webapp_query::test_stats_album_progress_and_errors status: characterized - id: wa-album-of area: ui source: {file: webapp/query.py, symbols: [album_of]} classification: replace rationale: > Verbatim copy of photo_analyzer.album_label (equivalence characterized); consolidated into the single albums-service implementation (pa-album-label), now photo_pipeline.services.albums.album_label (US03-01, parity test below). target: photo_pipeline/services/albums.py tests: - test_pa_album::test_webapp_album_of_mirrors_album_label - test_pa_album::test_albums_service_album_label_matches_donor status: characterized - id: wa-allowlist area: ui source: {file: webapp/query.py, symbols: [all_paths]} classification: replace rationale: > Path-set allowlist for /img and /exif is superseded by asset-ID-addressed endpoints that never accept a browser-supplied path (concept thumbnail rules). The validate-before-serving intent carries over. target: photo_pipeline/api (asset_id thumbnail/EXIF endpoints) tests: [test_webapp_query::test_all_paths_is_the_image_endpoint_allowlist] status: characterized - id: wa-runner area: cancellation source: {file: webapp/runner.py, symbols: [Runner, progress]} classification: replace rationale: > Subprocess-driving-the-CLI job control is superseded by durable DB jobs with a worker process. Two ideas carry over: progress derived from DB counts (not job-private state) and single-mutating-job enforcement. target: photo_pipeline/jobs/coordinator.py pending_story: US02-02 status: pending - id: wa-server area: ui source: {file: webapp/server.py, symbols: [serve]} classification: replace rationale: > stdlib HTTP route ladder → FastAPI typed routes. Carried intents: 127.0.0.1 bind, path validation against the DB, no secrets to the browser. analyzer.html + page.py design (dark OLED tokens, Library/ Analyze/Stats views) is frontend donor material per concept §10. target: photo_pipeline/api/app.py + frontend pending_story: US02-05 status: pending