664 lines
32 KiB
YAML
664 lines
32 KiB
YAML
# Donor ledger — US01-01 (INTEGRATED_PIPELINE_CONCEPT.md §3 "Donor-first CLI migration")
|
|
#
|
|
# Maps every relevant legacy module/function to its classification, rationale,
|
|
# target module in the future photo_pipeline package, and characterization
|
|
# test IDs. Linted by tests/characterization/test_donor_ledger.py.
|
|
#
|
|
# classification: reuse — take as-is
|
|
# extract — move behind a thin adapter, behavior unchanged
|
|
# refactor — extract while changing structure, behavior preserved
|
|
# unless an intentional delta is stated
|
|
# replace — new implementation; rationale documents why and what
|
|
# (if anything) carries over
|
|
# Every row needs either `tests` (existing test IDs, module::function) or
|
|
# `pending_story` (the backlog story that will characterize/deliver it).
|
|
# status: characterized — donor behavior pinned by characterization tests
|
|
# resolved — replacement shipped; `parity` names the tests that prove
|
|
# it, and `delta` states every intentional difference
|
|
# pending — not migrated yet; `pending_story` says which story will
|
|
# parity: test ids (path::function) in any suite, proving the replacement
|
|
# delta: what the replacement deliberately does differently, or not at all
|
|
#
|
|
# Archived by US07-01: the sources referenced below now live beside this file in
|
|
# src/ and are frozen (see README.md). Nothing in photo_pipeline imports them.
|
|
|
|
rows:
|
|
# ── photo_analyzer.py ──────────────────────────────────────────────────────
|
|
- id: pa-discovery
|
|
area: discovery
|
|
source: {file: photo_analyzer.py, symbols: [discover_photos, SUPPORTED_EXTENSIONS]}
|
|
classification: extract
|
|
rationale: >
|
|
Proven recursive discovery with _IGNORE/ and .@__thumb exclusion, sorted +
|
|
deduplicated. Becomes the one shared discovery used by every stage; the
|
|
exclusion predicate moves to path_policy.py.
|
|
target: photo_pipeline/services/inventory.py + photo_pipeline/path_policy.py
|
|
tests:
|
|
- test_pa_discovery::test_discover_excludes_ignore_and_thumbs
|
|
- test_pa_discovery::test_discover_finds_supported_including_uppercase
|
|
- test_pa_discovery::test_discover_sorted_and_deduplicated
|
|
- test_pa_discovery::test_supported_extensions_contract
|
|
status: characterized
|
|
|
|
- id: pa-purge-excluded
|
|
area: discovery
|
|
source: {file: photo_analyzer.py, symbols: [purge_excluded]}
|
|
classification: refactor
|
|
rationale: >
|
|
SQL LIKE patterns duplicate the path policy; refactor to call the shared
|
|
path_policy predicate so exclusion has exactly one definition.
|
|
target: photo_pipeline/services/inventory.py
|
|
tests: [test_pa_db::test_purge_excluded_removes_ignore_and_thumb_rows]
|
|
status: characterized
|
|
|
|
- id: pa-prune-missing
|
|
area: database
|
|
source: {file: photo_analyzer.py, symbols: [prune_missing]}
|
|
classification: replace
|
|
rationale: >
|
|
Deletes rows for missing files — incompatible with stable asset identity;
|
|
the concept keeps the asset and sets missing_at/availability_state.
|
|
The unmounted-library guard (never prune when the root is absent) carries
|
|
over as behavior.
|
|
target: photo_pipeline/services/inventory.py
|
|
tests: [test_pa_db::test_prune_missing_guards_unmounted_library]
|
|
status: characterized
|
|
|
|
- id: pa-variants
|
|
area: discovery
|
|
source: {file: photo_analyzer.py, symbols: [_strip_variant_markers, _is_variant, _variant_key, build_variant_groups]}
|
|
classification: extract
|
|
rationale: >
|
|
Filename-based variant grouping (pixel-size suffix, 'bearbeitet' edit
|
|
marker) with unmarked-largest primary selection; feeds duplicate/variant
|
|
clustering evidence.
|
|
target: photo_pipeline/services/duplicates.py
|
|
tests:
|
|
- test_pa_variants::test_strip_variant_markers_goldens
|
|
- test_pa_variants::test_is_variant
|
|
- test_pa_variants::test_variant_key_folder_scoped_case_insensitive
|
|
- test_pa_variants::test_build_variant_groups_primary_is_largest_unmarked
|
|
status: characterized
|
|
|
|
- id: pa-variant-propagate
|
|
area: database
|
|
source: {file: photo_analyzer.py, symbols: [propagate_variants, _row_to_result]}
|
|
classification: refactor
|
|
rationale: >
|
|
Copying a primary's analysis into variant rows survives, but keyed by
|
|
asset_id and recorded as stage state instead of raw row copies.
|
|
target: photo_pipeline/services/duplicates.py
|
|
parity:
|
|
- tests/integration/test_duplicate_engine.py::test_exact_copies_form_auto_decided_cluster
|
|
- tests/integration/test_duplicate_engine.py::test_perceptual_variant_is_review_only
|
|
delta: >
|
|
The donor propagated variant links implicitly while writing rows; the replacement makes the canonical link a reviewable, reversible cluster decision, so a propagated link can always be undone.
|
|
status: resolved
|
|
|
|
- id: pa-hashing
|
|
area: hashing
|
|
source: {file: photo_analyzer.py, symbols: [_sha1_file, _phash_image]}
|
|
classification: extract
|
|
rationale: >
|
|
Streamed SHA-1 plus 64-bit DCT phash (imagehash.phash recipe on
|
|
numpy/scipy, no extra dependency). Hash outputs are goldens — a change
|
|
means the decoder/algorithm changed and hash_version must bump.
|
|
target: photo_pipeline/services/inventory.py
|
|
tests:
|
|
- test_pa_hashing_dedup::test_phash_goldens
|
|
- test_pa_hashing_dedup::test_phash_survives_resize_and_recompress
|
|
- test_pa_hashing_dedup::test_phash_format_16_hex_chars
|
|
- test_pa_hashing_dedup::test_sha1_matches_hashlib
|
|
- test_pa_hashing_dedup::test_sha1_unreadable_returns_none
|
|
status: characterized
|
|
|
|
- id: pa-ensure-hashes
|
|
area: hashing
|
|
source: {file: photo_analyzer.py, symbols: [ensure_hashes]}
|
|
classification: extract
|
|
rationale: >
|
|
Resume-safe incremental hashing (skip already-hashed, COALESCE so a failed
|
|
hash never clobbers a stored one, periodic commits, honors stop event).
|
|
target: photo_pipeline/services/inventory.py
|
|
tests: [test_pa_hashing_dedup::test_ensure_hashes_skips_hashed_and_is_resume_safe]
|
|
status: characterized
|
|
|
|
- id: pa-cluster
|
|
area: hashing
|
|
source: {file: photo_analyzer.py, symbols: [cluster_duplicates]}
|
|
classification: extract
|
|
rationale: >
|
|
Union-find over the vectorized Hamming graph; largest-first ordering.
|
|
Known ceiling (documented in-source): O(n²) scan, BK-tree past ~100k.
|
|
target: photo_pipeline/services/duplicates.py
|
|
tests: [test_pa_hashing_dedup::test_cluster_and_mark_duplicates_largest_is_canonical]
|
|
status: characterized
|
|
|
|
- id: pa-mark-duplicates
|
|
area: database
|
|
source: {file: photo_analyzer.py, symbols: [mark_duplicates]}
|
|
classification: refactor
|
|
rationale: >
|
|
Intentional delta — the concept forbids silently marking fuzzy matches:
|
|
largest-file canonical becomes a *recommendation*; the decision lands in
|
|
reviewable duplicate_clusters, auto-resolve only for exact matches.
|
|
The status='duplicate' exclusion from analysis/upload carries over.
|
|
target: photo_pipeline/services/duplicates.py
|
|
tests: [test_pa_hashing_dedup::test_cluster_and_mark_duplicates_largest_is_canonical]
|
|
status: characterized
|
|
|
|
- id: pa-list-duplicates
|
|
area: ui
|
|
source: {file: photo_analyzer.py, symbols: [list_duplicates]}
|
|
classification: replace
|
|
rationale: Console report; superseded by the duplicate-review API/UI (US01-06).
|
|
target: photo_pipeline/api/routes + frontend duplicate review
|
|
parity:
|
|
- tests/integration/test_review_api.py::test_clusters_list_and_detail
|
|
- tests/e2e/test_review_ui.py::test_fuzzy_decision_requires_confirmation
|
|
delta: >
|
|
The text listing became the paged cluster API and the comparison UI; no textual report is produced.
|
|
status: resolved
|
|
|
|
- id: pa-reconcile
|
|
area: database
|
|
source: {file: photo_analyzer.py, symbols: [reconcile_moved]}
|
|
classification: refactor
|
|
rationale: >
|
|
SHA-1-keyed move/rename reconciliation preserving analysis + EXIF state.
|
|
Refactor to update assets.current_path by asset_id and append an
|
|
asset_paths history row instead of rewriting the path in place.
|
|
target: photo_pipeline/services/inventory.py
|
|
tests: [test_pa_hashing_dedup::test_reconcile_moved_preserves_row_by_sha1]
|
|
status: characterized
|
|
|
|
- id: pa-db-schema
|
|
area: database
|
|
source: {file: photo_analyzer.py, symbols: [SCHEMA, get_db, _migrate_schema]}
|
|
classification: refactor
|
|
rationale: >
|
|
photos table + FTS5 + idempotent ALTER migration become the migration
|
|
baseline; new schema adds assets/asset_paths/stage state per concept.
|
|
Existing data must migrate losslessly (US01-02).
|
|
target: photo_pipeline migrations (Alembic)
|
|
tests: [test_pa_db::test_schema_and_migration_idempotent]
|
|
status: characterized
|
|
|
|
- id: pa-db-status
|
|
area: error
|
|
source: {file: photo_analyzer.py, symbols: [upsert_pending, mark_analyzed, mark_error, mark_exif_written, get_pending, get_analyzed_no_exif]}
|
|
classification: refactor
|
|
rationale: >
|
|
Status lifecycle (pending→analyzed→exif_written; error rows auto-retried
|
|
on the next run; INSERT OR IGNORE never downgrades; duplicates never
|
|
queued) is the resume-safety contract — preserved as asset_stage_states
|
|
transitions keyed by asset_id.
|
|
target: photo_pipeline/repositories + services/analysis.py
|
|
tests:
|
|
- test_pa_db::test_status_lifecycle_pending_analyzed_exif_written
|
|
- test_pa_db::test_mark_error_and_retry_via_get_pending
|
|
- test_pa_db::test_mark_analyzed_clears_error
|
|
- test_pa_db::test_upsert_pending_never_downgrades
|
|
status: characterized
|
|
|
|
- id: pa-fts
|
|
area: database
|
|
source: {file: photo_analyzer.py, symbols: [photos_fts]}
|
|
classification: reuse
|
|
rationale: FTS5 table + sync triggers already work; carried into the new schema.
|
|
target: photo_pipeline migrations (Alembic)
|
|
tests: [test_pa_db::test_fts_kept_in_sync_by_triggers]
|
|
status: characterized
|
|
|
|
- id: pa-imaging
|
|
area: imaging
|
|
source: {file: photo_analyzer.py, symbols: [prepare_image, MAX_LONG_EDGE]}
|
|
classification: extract
|
|
rationale: >
|
|
RGB-normalize (drops alpha, converts HEIC), LANCZOS resize to 2048px
|
|
long-edge, JPEG q85 base64 — the provider-input contract. Truncated-image
|
|
tolerance (ImageFile.LOAD_TRUNCATED_IMAGES) carries with it.
|
|
target: photo_pipeline/services/analysis.py
|
|
tests:
|
|
- test_pa_imaging::test_prepare_image_small_passthrough_jpeg
|
|
- test_pa_imaging::test_prepare_image_resizes_to_max_long_edge
|
|
- test_pa_imaging::test_prepare_image_converts_png_alpha_to_rgb_jpeg
|
|
status: characterized
|
|
|
|
- id: pa-repair
|
|
area: error
|
|
source: {file: photo_analyzer.py, symbols: [repair_image, _REPAIRABLE_ERRORS, _log_repair]}
|
|
classification: refactor
|
|
rationale: >
|
|
Re-encode structurally broken images via sips (macOS-only — portability
|
|
note), .orig backup, atomic replace, repairs.jsonl audit. Needs corrupt
|
|
fixtures to characterize; lands with media hardening.
|
|
target: photo_pipeline/integrations/exiftool.py
|
|
pending_story: US07-03
|
|
status: pending
|
|
|
|
- id: pa-vision
|
|
area: vision
|
|
source: {file: photo_analyzer.py, symbols: [analyze_image, ANALYSIS_PROMPT, LLM_BASE_URL, LLM_MODEL]}
|
|
classification: extract
|
|
rationale: >
|
|
OpenAI-compatible request construction, base64 image_url payload, JSON
|
|
response validation, 429/503 retry with exponential backoff. Prompt and
|
|
model/config version must be persisted per analysis_runs. Characterized
|
|
against a deterministic fake provider when the analysis service is ported.
|
|
target: photo_pipeline/services/analysis.py
|
|
parity:
|
|
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
|
delta: >
|
|
The prompt and response schema carry over; the provider is an injected adapter so the privacy gate is testable, and results are keyed to asset ids rather than paths.
|
|
status: resolved
|
|
|
|
- id: pa-throttle
|
|
area: logging
|
|
source: {file: photo_analyzer.py, symbols: [_record_ratelimit, _ratelimit_recent, write_throttle_summary, _rpd_load, _rpd_increment, _rpd_count]}
|
|
classification: refactor
|
|
rationale: >
|
|
Rolling throttle window + persistent throttle_events.jsonl + RPD day
|
|
counter become job metrics/events on the durable job model.
|
|
target: photo_pipeline/services/jobs.py + photo_pipeline/services/analysis.py
|
|
pending_story: US07-04
|
|
status: pending
|
|
|
|
- id: pa-nsfw-filter
|
|
area: nsfw
|
|
source: {file: photo_analyzer.py, symbols: [_rec_has_nsfw, filter_nsfw_tagged]}
|
|
classification: replace
|
|
rationale: >
|
|
Intentional delta — donor fails OPEN (exiftool hiccup → analyze
|
|
everything), which violates the concept privacy invariant; the new gate
|
|
is the DB safety decision and fails CLOSED (only confirmed sfw reaches
|
|
the provider). EXIF keyword reading survives as evidence/verification,
|
|
not as the gate. Current behavior captured for migration.
|
|
target: photo_pipeline/services/safety.py
|
|
tests:
|
|
- test_pa_exif::test_rec_has_nsfw_list_scalar_case
|
|
- test_pa_exif::test_filter_nsfw_tagged_splits_and_normalizes
|
|
- test_pa_exif::test_filter_nsfw_tagged_empty_list_noop
|
|
status: characterized
|
|
|
|
- id: pa-exif-caption
|
|
area: exif
|
|
source: {file: photo_analyzer.py, symbols: [build_exif_caption, build_exif_caption_from_result]}
|
|
classification: refactor
|
|
rationale: >
|
|
Caption format ('desc | Tags: … | Mood: … | Location: … | ~year') is
|
|
what 25k+ photos already carry — preserved for compatibility. Intentional
|
|
delta: new writes go into a managed 'AI:' segment so re-analysis can
|
|
replace only its own text (concept EXIF ownership).
|
|
target: photo_pipeline/services/exif projection
|
|
tests:
|
|
- test_pa_exif::test_caption_golden_full
|
|
- test_pa_exif::test_caption_golden_sparse
|
|
status: characterized
|
|
|
|
- id: pa-exif-read
|
|
area: exif
|
|
source: {file: photo_analyzer.py, symbols: [read_existing_exif]}
|
|
classification: extract
|
|
rationale: Targeted exiftool JSON read of the four owned fields, 30s timeout, fail-empty.
|
|
target: photo_pipeline/integrations/exiftool.py
|
|
tests: [test_pa_exif::test_write_exif_roundtrip_and_idempotent]
|
|
status: characterized
|
|
|
|
- id: pa-exif-write
|
|
area: exif
|
|
source: {file: photo_analyzer.py, symbols: [write_exif]}
|
|
classification: refactor
|
|
rationale: >
|
|
Merge-don't-overwrite semantics are the crown jewels: caption prepended to
|
|
existing text (idempotent — never appended twice), keywords merged +
|
|
order-preserving dedup, -m -overwrite_original, repair-and-retry on
|
|
structurally broken files. Refactored to add the concept's read-back
|
|
verification + non-owned-field snapshot comparison. sys.exit on missing
|
|
exiftool becomes a service error.
|
|
target: photo_pipeline/integrations/exiftool.py + services/exif checkpoints
|
|
tests:
|
|
- test_pa_exif::test_write_exif_roundtrip_and_idempotent
|
|
- test_pa_exif::test_write_exif_preserves_existing_metadata
|
|
status: characterized
|
|
|
|
- id: pa-run-loop
|
|
area: cancellation
|
|
source: {file: photo_analyzer.py, symbols: [run_analysis, _run_folder_loop, run_exif_write, _stop, _handle_sigint]}
|
|
classification: refactor
|
|
rationale: >
|
|
Folder-grouped worker pool, per-item DB commits, cooperative stop event
|
|
checked between items, double-SIGINT force quit — becomes the durable
|
|
JobRunner worker loop with the same drain-and-resume semantics.
|
|
target: photo_pipeline/jobs/worker.py
|
|
parity:
|
|
- tests/integration/test_worker.py::test_worker_processes_all_items
|
|
- tests/integration/test_worker.py::test_cooperative_cancellation_leaves_items_resumable
|
|
delta: >
|
|
The in-process folder loop with SIGINT handling became durable jobs claimed by a worker: cancellation is a persisted request, not a signal, and an interrupted run resumes from the database instead of restarting.
|
|
status: resolved
|
|
|
|
- id: pa-ui-terminal
|
|
area: ui
|
|
source: {file: photo_analyzer.py, symbols: [_Dashboard, _LiveRenderer, _PlainRenderer, _make_progress, _stats_panel, _feed_panel, _tokens_panel, _menu_panel, _read_key, _key_listener, print_stats, fit_label]}
|
|
classification: replace
|
|
rationale: >
|
|
Rich terminal dashboard (Live layout, hotkey menu, termios key reader) is
|
|
superseded by the web Workflow/Analyze views. The *data* it renders
|
|
(album progress, feed, tokens, ETA) survives via webapp.runner.progress
|
|
(see wa-runner). fit_label/print_stats die with it.
|
|
target: frontend Analyze view (Phase B)
|
|
tests: [test_pa_album::test_fit_label_left_truncates]
|
|
status: characterized
|
|
|
|
- id: pa-album-label
|
|
area: ui
|
|
source: {file: photo_analyzer.py, symbols: [album_label]}
|
|
classification: extract
|
|
rationale: >
|
|
Album = leaf folder relative to library, '(root)' at root, bare parent
|
|
outside the library. The single album-identity rule for stats, proposals
|
|
and folder-as-album upload. webapp.query.album_of is a copy — consolidate
|
|
to one implementation (see wa-album-of). Extracted in US03-01 as
|
|
photo_pipeline.services.albums.album_label (parity test below); the album
|
|
evidence aggregator uses it. LibraryService._album_of still keeps its own
|
|
leaf-name grouping — fold it into this rule when the Library view is next
|
|
touched.
|
|
target: photo_pipeline/services/albums.py
|
|
tests:
|
|
- test_pa_album::test_album_label_leaf_folder_relative
|
|
- test_pa_album::test_webapp_album_of_mirrors_album_label
|
|
- test_pa_album::test_albums_service_album_label_matches_donor
|
|
status: characterized
|
|
|
|
- id: pa-config
|
|
area: configuration
|
|
source: {file: photo_analyzer.py, symbols: [load_env_file, _env_str, _env_int, _env_bool, ENV_FILE]}
|
|
classification: refactor
|
|
rationale: >
|
|
photo_analyzer.env parsing (shell env wins, export prefix, quoted values,
|
|
inline-comment rules) feeds the typed config module; same file keeps
|
|
working so the transition needs no re-setup.
|
|
target: photo_pipeline/config.py
|
|
tests:
|
|
- test_pa_config::test_load_env_file_parsing
|
|
- test_pa_config::test_env_helpers
|
|
status: characterized
|
|
|
|
- id: pa-logging
|
|
area: logging
|
|
source: {file: photo_analyzer.py, symbols: [log_history, _history_handler, _rich_handler]}
|
|
classification: replace
|
|
rationale: >
|
|
Module-level triple-handler logging (console + info log + debug log) and
|
|
the JSONL history logger become structured JSON logging with job_id/
|
|
asset_id and job_events rows; per-photo history maps to job events.
|
|
target: photo_pipeline structured logging + jobs/job_events
|
|
parity:
|
|
- tests/integration/test_jobs.py::test_enqueue_persists_items_and_event
|
|
- tests/integration/test_jobs_sse.py::test_sse_streams_all_events_then_closes
|
|
delta: >
|
|
The JSONL history file and rich console handler are replaced by structured JSON logs plus durable job_events; the browser reads events over SSE rather than tailing a file.
|
|
status: resolved
|
|
|
|
- id: pa-balance
|
|
area: vision
|
|
source: {file: photo_analyzer.py, symbols: [fetch_balance, query_balance, check_quota]}
|
|
classification: extract
|
|
rationale: >
|
|
Provider balance/quota probes (report 'unsupported' on providers without
|
|
the endpoint). Network-bound; characterized against the fake provider.
|
|
target: photo_pipeline/services/analysis.py
|
|
parity:
|
|
- tests/integration/test_safety_analysis.py::test_provider_called_only_for_confirmed_sfw
|
|
delta: >
|
|
Not carried over: balance/quota polling was provider-specific (Gemini/OpenAI billing endpoints) and key-scoped. Cost reporting, when a story asks for it, comes from the per-run usage recorded with each analysis result rather than from a vendor endpoint.
|
|
status: resolved
|
|
|
|
- id: pa-cli
|
|
area: configuration
|
|
source: {file: photo_analyzer.py, symbols: [main]}
|
|
classification: replace
|
|
rationale: >
|
|
argparse surface is superseded by the API; flags map to job configs
|
|
(documented in WEBAPP_CONCEPT.md §8 parity table). Transitional CLI calls
|
|
the shared services until archival (E07).
|
|
target: photo_pipeline/__main__.py (serve | worker | migrate | import-legacy-scores) + /api/v1
|
|
parity:
|
|
- tests/integration/test_app_lifecycle.py::test_restart_preserves_data_and_reruns_migrations
|
|
- tests/unit/test_legacy_archive.py::test_production_code_never_imports_an_archived_module
|
|
delta: >
|
|
The argparse surface is not reproduced. Every flag that drove work became an API command or a job configuration; the CLI keeps only what an application needs to be operated (serve, worker, migrate) plus the one-off legacy CSV import.
|
|
status: resolved
|
|
|
|
# ── nsfwtag/ ───────────────────────────────────────────────────────────────
|
|
- id: nt-discovery
|
|
area: discovery
|
|
source: {file: nsfwtag/scoring.py, symbols: [discover_images]}
|
|
classification: refactor
|
|
rationale: >
|
|
Merged into the shared discovery. Known divergence captured: nsfwtag EXTS
|
|
lacks .tiff/.tif, is non-recursive by default, and dedupes by resolved
|
|
path (symlink guard) — the shared service adopts the superset + symlink
|
|
dedup, always recursive, plus _IGNORE policy (donor has none).
|
|
target: photo_pipeline/services/inventory.py
|
|
tests:
|
|
- test_nsfwtag::test_exts_contract_differs_from_photo_analyzer
|
|
- test_nsfwtag::test_discover_images_flat_recursive_limit
|
|
- test_nsfwtag::test_discover_images_dedupes_symlinked_paths
|
|
status: characterized
|
|
|
|
- id: nt-score-cache
|
|
area: nsfw
|
|
source: {file: nsfwtag/scoring.py, symbols: [load_cache, _save_cache, CSV_NAME]}
|
|
classification: replace
|
|
rationale: >
|
|
nsfw_scores.csv stops being the source of truth (concept: DB state).
|
|
Format characterized (4-decimal scores, bad rows dropped) because the
|
|
existing CSV must migrate into assets.safety_score.
|
|
target: photo_pipeline/services/legacy_import.py (one-off import into safety_reviews)
|
|
tests: [test_nsfwtag::test_score_cache_roundtrip_and_tolerance]
|
|
parity:
|
|
- tests/integration/test_legacy_import.py::test_scores_are_imported_onto_asset_identity
|
|
- tests/integration/test_legacy_import.py::test_a_reviewed_asset_is_never_overwritten_by_the_csv
|
|
delta: >
|
|
The CSV is no longer read at runtime at all: it is imported once into
|
|
scored-but-unreviewed safety_reviews rows and left on disk untouched. A path
|
|
that matches nothing is reported, never turned into an asset, and a human
|
|
decision always outranks an imported score.
|
|
status: resolved
|
|
|
|
- id: nt-score-model
|
|
area: nsfw
|
|
source: {file: nsfwtag/scoring.py, symbols: [score_images, MODEL_ID, BATCH]}
|
|
classification: extract
|
|
rationale: >
|
|
On-device ViT scoring: MPS/CPU pick, model-defined input size (not
|
|
hardcoded — survives model swap), batch inference where one bad batch
|
|
never discards prior scores, periodic checkpoint every 500. Cache-hit
|
|
short-circuit (model never loads when nothing is new) characterized now;
|
|
inference path needs the local model + deterministic fake.
|
|
target: photo_pipeline/integrations/nsfw_model.py
|
|
tests: [test_nsfwtag::test_score_images_cache_hit_skips_model]
|
|
delta: >
|
|
The donor set Pillow's process-global ImageFile.LOAD_TRUNCATED_IMAGES so a
|
|
partially downloaded file still scored. Here the same process also hashes
|
|
files and renders previews, and those must keep refusing a truncated file
|
|
rather than silently working on half of one; scoring opens images through
|
|
the bounded photo_pipeline.imaging door instead and skips the ones it cannot
|
|
read, leaving them unscored and visibly undecided (US07-03).
|
|
status: characterized
|
|
|
|
- id: nt-exif-keyword
|
|
area: exif
|
|
source: {file: nsfwtag/exif.py, symbols: [write_keyword, remove_keyword]}
|
|
classification: extract
|
|
rationale: >
|
|
Idempotent add/remove via '-Keywords-=x -Keywords+=x' (no duplicates on
|
|
re-run), touching only Keywords+Subject. Basis of the safety EXIF
|
|
checkpoint; the mutual-exclusion write (sfw removed when nsfw added and
|
|
vice versa) is the concept's addition on top.
|
|
target: photo_pipeline/services/safety.py + integrations/exiftool.py
|
|
tests: [test_nsfwtag::test_write_remove_keyword_idempotent]
|
|
status: characterized
|
|
|
|
- id: nt-exif-marks
|
|
area: nsfw
|
|
source: {file: nsfwtag/exif.py, symbols: [_read_keywords, read_tagged, read_marks]}
|
|
classification: extract
|
|
rationale: >
|
|
Batched exiftool read via stdin, lowercase normalization, and the safety
|
|
rule that a file carrying both keywords reads as nsfw (never as safe).
|
|
target: photo_pipeline/services/safety.py
|
|
tests:
|
|
- test_nsfwtag::test_write_remove_keyword_idempotent
|
|
- test_nsfwtag::test_read_marks_nsfw_wins_over_sfw
|
|
status: characterized
|
|
|
|
- id: nt-exif-view
|
|
area: exif
|
|
source: {file: nsfwtag/exif.py, symbols: [read_exif, _EXIF_NOISE]}
|
|
classification: extract
|
|
rationale: >
|
|
Lightbox EXIF read: filesystem-noise filter, binary/oversize skip, signed
|
|
decimal GPS, Google-Maps link prepended. Backs the asset EXIF endpoint.
|
|
target: photo_pipeline/api (asset EXIF endpoint)
|
|
tests: [test_nsfwtag::test_read_exif_filters_noise_and_prepends_map]
|
|
status: characterized
|
|
|
|
- id: nt-apply-list
|
|
area: nsfw
|
|
source: {file: nsfwtag/exif.py, symbols: [apply_list]}
|
|
classification: replace
|
|
rationale: >
|
|
Newline-list bulk tagging (nsfw_confirmed.txt flow) is superseded by DB
|
|
review decisions; the existing list is a one-time migration input.
|
|
target: photo_pipeline/services/safety.py (decision import in US01-02)
|
|
parity:
|
|
- tests/integration/test_safety_parity.py::test_extracted_marks_and_partition_match_donor
|
|
delta: >
|
|
Bulk keyword application from a file list is replaced by decisions against asset ids; the EXIF write itself is the extracted, read-back-verified one.
|
|
status: resolved
|
|
|
|
- id: nt-ui
|
|
area: ui
|
|
source: {file: nsfwtag/server.py, symbols: [serve_review]}
|
|
classification: replace
|
|
rationale: >
|
|
stdlib ThreadingHTTPServer + token-injected review.html backend is
|
|
superseded by FastAPI. review.html's design tokens, folder tree,
|
|
threshold/score review flow, lightbox and keyboard model are the frontend
|
|
donor for the Safety view (preserved per concept §10; ported in US02-01).
|
|
target: photo_pipeline/api + frontend Safety view
|
|
parity:
|
|
- tests/e2e/test_workflow_views.py::test_safety_review_decide_persists_across_reload
|
|
delta: >
|
|
The stdlib review server is replaced by the API plus the Safety view; the donor's layout, thresholds, and keyboard flow carry over, its embedded HTML generation does not.
|
|
status: resolved
|
|
|
|
- id: nt-bench
|
|
area: nsfw
|
|
source: {file: nsfwtag/bench.py, symbols: [main]}
|
|
classification: replace
|
|
rationale: >
|
|
Dev-only model benchmark; archived without webapp replacement (recorded
|
|
basis of the AdamCodd model choice). No production caller.
|
|
target: none (archive as reference)
|
|
parity:
|
|
- tests/unit/test_legacy_archive.py::test_every_archived_source_matches_its_checksum
|
|
delta: >
|
|
No replacement: a dev-only benchmark whose result (the AdamCodd model choice) is already recorded. Kept in the archive as the basis of that choice.
|
|
status: resolved
|
|
|
|
# ── webapp/ ────────────────────────────────────────────────────────────────
|
|
- id: wa-query-search
|
|
area: database
|
|
source: {file: webapp/query.py, symbols: [_fts_query, _where, search, _row_to_card, photo, _SORTS]}
|
|
classification: extract
|
|
rationale: >
|
|
Safe FTS5 MATCH building (tokenize → quoted prefix terms; punctuation
|
|
can't crash MATCH), facet WHERE composition, paged search with
|
|
relevance/sort modes, raw_response kept out of list payloads.
|
|
target: photo_pipeline/repositories (search) + api/routes
|
|
tests:
|
|
- test_webapp_query::test_fts_query_sanitization_goldens
|
|
- test_webapp_query::test_where_clause_goldens
|
|
- test_webapp_query::test_search_fts_prefix_match
|
|
- test_webapp_query::test_search_filters_and_browse
|
|
- test_webapp_query::test_search_paging
|
|
- test_webapp_query::test_photo_omits_raw_response
|
|
status: characterized
|
|
|
|
- id: wa-query-aggregates
|
|
area: database
|
|
source: {file: webapp/query.py, symbols: [facets, stats, _col_counts, _people_buckets, _top_tags]}
|
|
classification: extract
|
|
rationale: >
|
|
Facet counts, year range, album tree, status/album progress (done =
|
|
analyzed + exif_written) and error listing — the Stats/Workflow data
|
|
model.
|
|
target: photo_pipeline/repositories + services/workflow.py
|
|
tests:
|
|
- test_webapp_query::test_facets_and_albums
|
|
- test_webapp_query::test_stats_album_progress_and_errors
|
|
status: characterized
|
|
|
|
- id: wa-album-of
|
|
area: ui
|
|
source: {file: webapp/query.py, symbols: [album_of]}
|
|
classification: replace
|
|
rationale: >
|
|
Verbatim copy of photo_analyzer.album_label (equivalence characterized);
|
|
consolidated into the single albums-service implementation (pa-album-label),
|
|
now photo_pipeline.services.albums.album_label (US03-01, parity test below).
|
|
target: photo_pipeline/services/albums.py
|
|
tests:
|
|
- test_pa_album::test_webapp_album_of_mirrors_album_label
|
|
- test_pa_album::test_albums_service_album_label_matches_donor
|
|
status: characterized
|
|
|
|
- id: wa-allowlist
|
|
area: ui
|
|
source: {file: webapp/query.py, symbols: [all_paths]}
|
|
classification: replace
|
|
rationale: >
|
|
Path-set allowlist for /img and /exif is superseded by asset-ID-addressed
|
|
endpoints that never accept a browser-supplied path (concept thumbnail
|
|
rules). The validate-before-serving intent carries over.
|
|
target: photo_pipeline/api (asset_id thumbnail/EXIF endpoints)
|
|
tests: [test_webapp_query::test_all_paths_is_the_image_endpoint_allowlist]
|
|
status: characterized
|
|
|
|
- id: wa-runner
|
|
area: cancellation
|
|
source: {file: webapp/runner.py, symbols: [Runner, progress]}
|
|
classification: replace
|
|
rationale: >
|
|
Subprocess-driving-the-CLI job control is superseded by durable DB jobs
|
|
with a worker process. Two ideas carry over: progress derived from DB
|
|
counts (not job-private state) and single-mutating-job enforcement.
|
|
target: photo_pipeline/services/jobs.py + photo_pipeline/jobs/worker.py
|
|
parity:
|
|
- tests/integration/test_jobs.py::test_idempotency_key_returns_same_job
|
|
- tests/integration/test_worker.py::test_handler_failure_fails_the_job
|
|
delta: >
|
|
Subprocess supervision of a CLI is replaced by durable jobs in the same process family: there is no subprocess to supervise, and progress is persisted rather than scraped from stdout.
|
|
status: resolved
|
|
|
|
- id: wa-server
|
|
area: ui
|
|
source: {file: webapp/server.py, symbols: [serve]}
|
|
classification: replace
|
|
rationale: >
|
|
stdlib HTTP route ladder → FastAPI typed routes. Carried intents:
|
|
127.0.0.1 bind, path validation against the DB, no secrets to the
|
|
browser. analyzer.html + page.py design (dark OLED tokens, Library/
|
|
Analyze/Stats views) is frontend donor material per concept §10.
|
|
target: photo_pipeline/api/app.py + frontend
|
|
parity:
|
|
- tests/e2e/test_frontend_shell.py::test_shell_loads_assets_without_console_or_network_errors
|
|
- tests/e2e/test_frontend_shell.py::test_deep_link_and_reload_restore_view_and_filters
|
|
delta: >
|
|
The stdlib HTTP routes become the versioned FastAPI API and the static frontend shell; Python no longer interpolates HTML.
|
|
status: resolved
|