domverse/photoanalyzer
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

Bootstrap project and safe work-item workflow

Browse Source
This commit is contained in:
domverse
2026-07-13 13:51:41 +02:00
commit 80447092e9
90 changed files with 10562 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
.gitignore vendored Normal file
View File

@@ -0,0 +1,18 @@
__pycache__/
*.py[cod]
.pytest_cache/
.coverage
htmlcov/
.DS_Store
.env
*.db
*.db-*
*.sqlite*
*.log
data/
downloads/
_archive/
_todo/
pictures/
photos/
_IGNORE/

46
.work-item.toml Normal file
View File

@@ -0,0 +1,46 @@
[repository]
slug = "domverse/photoanalyzer"
login = "domverse"
assignee = "domverse"
remote = "origin"
main_branch = "main"
[workflow]
branch_prefix = "us"
require_ci = false
required_tests = ["python3 -m unittest discover -s tests -v"]
[safety]
max_file_bytes = 5000000
allow = ["tests/fixtures/**"]
deny = [
"_IGNORE/**",
"**/_IGNORE/**",
"pictures/**",
"photos/**",
"_archive/**",
"_todo/**",
"data/**",
"downloads/**",
"*.db",
"*.db-*",
"*.sqlite",
"*.sqlite3",
"*.log",
"*.env",
"*.pem",
"*.key",
"*.jpg",
"*.jpeg",
"*.png",
"*.gif",
"*.webp",
"*.heic",
"*.heif",
"*.tif",
"*.tiff",
"*.mp4",
"*.mov",
"*.avi",
"*.mkv",
]

50
AGENTS.md Normal file
View File

@@ -0,0 +1,50 @@
# Agent Instructions — Photo Analyzer
Read `CLAUDE.md` and `INTEGRATED_PIPELINE_CONCEPT.md` before implementation. Preserve
the established project invariants: never inspect or process content beneath any
`_IGNORE/` directory; write verified EXIF before Immich upload; use managed
`immich-go` uploads rather than External Libraries; keep operations resume-safe; and
treat the existing CLI implementations as primary donors rather than rewriting them.
## Mandatory implementation workflow
When asked to start, resume, or continue implementation, work from this isolated Git
repository and use `scripts/work-item`. Gitea issues and their dependencies are the
authoritative backlog.
1. Run `scripts/work-item claim`.
2. Work only on the claimed story and its generated feature branch.
3. Read the entire issue, linked specification, dependencies, and acceptance criteria.
4. Inspect the legacy CLI donors before replacing applicable behavior. Update the donor
ledger and characterization tests required by the story.
5. Implement every acceptance criterion and its automated tests.
6. Run story-specific tests and the accumulated regression suite required by the epic.
7. Run `scripts/work-item submit --test "<story-specific command>"` without `--yes` and
review the reported diff and file list.
8. If the changes are correct and safe, rerun with `--yes`. This commits, pushes, opens
a pull request, records test evidence, and moves the issue to review.
9. After review and successful CI, run `scripts/work-item complete --merge`. This
verifies the merge, closes the issue, returns to updated `main`, and removes the
local feature branch.
10. Continue with the next eligible story by returning to step 1.
If genuinely blocked, run `scripts/work-item block --reason "<precise reason>"` and
stop. Never skip to a later story whose dependencies are open. Never mark work done
when tests are missing, skipped, or failing.
## Git and privacy safety
- Never run automated Git operations from the personal photo-library directory.
- Never stage or commit real photos, videos, `_IGNORE/` content, databases, logs,
credentials, environment files, caches, generated thumbnails, or runtime data.
- Never use force-push, destructive reset, untracked-file deletion, or automatic merge
conflict resolution.
- Do not bypass `scripts/work-item` safety checks or manually close implementation
issues to conceal a failed transition.
- Preserve unrelated user changes and stop if the working tree is unexpectedly dirty.
## Completion standard
A story is complete only when its pull request is merged, required automated tests and
CI pass, test evidence is recorded on the issue, the issue is closed with
`status/done`, and the local repository is back on a clean, current `main` branch.

3
CLAUDE.md Normal file
View File

@@ -0,0 +1,3 @@
# CLAUDE.md
Project guidance lives in [AGENTS.md](AGENTS.md). Read it before making any changes.

257
DASHBOARD_PLAN.md Normal file
View File

@@ -0,0 +1,257 @@
# Rich Live Dashboard — Upgrade Plan
Replace the scrolling `Progress` context in `photo_analyzer.py` with a persistent
`Live + Layout` that shows **album progress**, a **photo feed**, and the **progress
bar** simultaneously — without touching the DB logic or threading model.
> **Focus: robustness.** The dashboard is a *view layer only* — it reads state,
> never owns it. SQLite stays the source of truth at all times.
---
## Target layout
```
┌─ python photo_analyzer.py --library pictures/ ─────────────────┐
│ ┌─ Album Progress — leaf folder = album ───────────────────┐ │
│ │ Camera Roll 238 / 550 ████████░░░░░░░░░░░ 43% │ │
│ │ Urlaub/Rom 45 / 45 ██████████████████ ✓ │ │
│ │ Urlaub/Venedig 12 / 60 ███░░░░░░░░░░░░░░░░ 20% │ │
│ │ WhatsApp Images 0 / 120 ░░░░░░░░░░░░░░░░░░ 0% │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ┌─ Recent ─────────────────────────────────────────────────┐ │
│ │ ✓ IMG_4821.jpg │ │
│ │ A golden retriever playing fetch on a sunny beach │ │
│ │ ✗ DSC_0042.jpg — Exhausted 4 retries: connection timeout │ │
│ │ ✓ IMG_4820.jpg │ │
│ │ Two children building a sandcastle at sunset │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ┌─ Analyzing — Camera Roll ────────────────────────────────┐ │
│ │ ████████░░░░░░░░░░ 238/550 43% ok=235 err=3 │ │
│ │ tokens=284,192 est. remaining ~340,000 ETA 14:23 │ │
│ └──────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
```
---
## Core rule: album = leaf folder
An **album is the leaf folder directly containing the photos**, not a top-level
parent. `Urlaub/Rom` and `Urlaub/Venedig` are two separate albums, never collapsed
into `Urlaub`. Key every stats dict on `album_label(path, library)` (already added
to the script for `--stats`):
```python
def album_label(photo: Path, library: Path = None) -> str:
"""Leaf folder relative to library: Urlaub/Rom/IMG.jpg -> 'Urlaub/Rom'."""
parent = photo.parent
if library:
try:
rel = parent.relative_to(library)
return "(root)" if str(rel) == "." else str(rel)
except ValueError:
pass
return parent.name or str(parent)
```
The analysis loop already groups on `Path(p).parent` (the leaf), so each sub-album
completes fully before the next starts. Only the *display* side needs this fix.
---
## Phases
### Phase 01 — Scaffold Live + Layout *(structural, ~30 min)*
Replace `with Progress(...) as progress:` with a `Live` context wrapping a
`Layout`. **Progress becomes a renderable, not a context manager** — it lives
inside the layout rather than owning the terminal. All DB logic, threading, and
retry logic stays unchanged.
```python
from rich.layout import Layout
from rich.live import Live
layout = Layout()
layout.split_column(
Layout(name="stats", ratio=2),
Layout(name="feed", ratio=2),
Layout(name="progress", size=5),
)
progress = Progress(...) # same columns as today
task_id = progress.add_task(...)
layout["progress"].update(progress)
with Live(layout, console=console, refresh_per_second=4, transient=False):
# ... existing folder loop unchanged ...
```
- **Test:** script starts, layout renders with placeholder panels, progress bar
advances, Ctrl-C stops cleanly.
- Do not move on until this is stable — later phases build on this scaffold.
---
### Phase 02 — Thread-safe log buffer *(concurrency, ~20 min)*
Workers must **never write to the terminal directly**. Replace every
`progress.console.print()` call with an append to a shared `deque`. The Live
renderer reads from the deque on each refresh cycle and renders it as the feed
panel. Threads write, the renderer reads — no other direction.
```python
from collections import deque
log_buffer = deque(maxlen=10) # newest first
log_lock = threading.Lock()
# in process() — replace progress.console.print(...) with:
with log_lock:
log_buffer.appendleft(entry)
# in the Live render function:
def render_feed() -> Table:
with log_lock:
rows = list(log_buffer) # snapshot
# build Rich Table from rows ...
```
- **Never** call `live.console.print()` from a worker thread — always via the buffer.
- `deque(maxlen=10)` auto-evicts old entries; no memory growth.
- The lock protects the *snapshot* read, not the render.
---
### Phase 03 — In-memory album stats *(performance, ~20 min)*
The stats panel refreshes at 4 Hz — **no DB queries in the render loop**. Key
every dict on `album_label(path, library)`. Compute `folder_total` from the sorted
`pending` list, seed `folder_done` from one upfront DB query, and have workers
increment it in memory after each success.
```python
# album = leaf folder relative to library — reuse the existing helper
folder_total: dict[str, int] = {}
folder_done: dict[str, int] = {}
for folder, paths in folder_groups: # folder = abs leaf dir
album = album_label(Path(paths[0]), library)
folder_total[album] = len(paths)
folder_done[album] = 0
# Seed already-done counts from DB (single query):
for path, status in conn.execute(
"SELECT path, status FROM photos WHERE status IN ('analyzed','exif_written')"
):
album = album_label(Path(path), library)
if album in folder_done:
folder_done[album] += 1
# In the worker result handler (album computed once per folder):
with token_lock: # reuse existing lock
folder_done[album] = folder_done.get(album, 0) + 1
```
- **Album = leaf folder, not parent.** Nested trees like `Urlaub/*` become one
row per sub-album.
- Render reads the dicts under the existing `token_lock` — no new lock needed.
- Shows **all** albums (future ones at 0%); long paths left-truncated via `fit_label()`.
- Errors do *not* increment `folder_done` — an album shows ✓ only when every
photo succeeded.
---
### Phase 04 — Non-TTY fallback *(safety, ~10 min)*
Rich's `Live` behaves unexpectedly when stdout is piped or redirected (log
capture, CI, `nohup`). **Gate the Live path on `console.is_terminal`.** Extract
the current Progress-based loop into `_run_analysis_plain()` unchanged; the new
Live implementation goes in `_run_analysis_live()`.
```python
def run_analysis(args, conn, client):
... # shared setup (discover, pending, folder_groups)
if console.is_terminal:
_run_analysis_live(args, conn, client, pending, folder_groups, ...)
else:
_run_analysis_plain(args, conn, client, pending, folder_groups, ...)
```
- Plain mode keeps every existing behavior — no regression for scripted use.
- File logs always write regardless of mode.
- No flag needed — detection is automatic.
---
### Phase 05 — Error containment *(robustness, ~15 min)*
The Live layer must never take down the pipeline. **DB writes happen in worker
threads, outside the render path** — a renderer crash cannot corrupt the
database. Wrap Live construction in try/except: if Rich fails to start, fall back
to plain mode. SIGINT already works (`_stop` set, executor drains, `with` exits).
```python
try:
with Live(layout, console=console, refresh_per_second=4):
_run_folder_loop(executor, folder_groups, ...)
except Exception as e:
log.warning(f"Live render failed ({e}), falling back to plain mode")
_run_analysis_plain(args, conn, client, pending, folder_groups, ...)
```
- **Worker exception path:** `mark_error()` runs before anything is appended to
`log_buffer` — DB is always consistent even if the render crashes mid-frame.
- **Render exceptions:** catch inside the render function, return a plain
`Text("render error")` panel — never let a bad render kill the loop.
- **Resume safety:** unchanged — `--stats` and restart behavior identical to today.
---
## Robustness constraints
| Constraint | Detail |
|---|---|
| No DB queries in render path | Renderer fires 4×/s. Album counts kept in in-memory dicts, seeded once at startup. SQLite never touched during animation. |
| Workers never touch the terminal | Only interface between workers and display is `log_buffer` (deque + lock). Rich's Live is single-threaded by design. |
| Render errors isolated | Each panel's render function wraps its body in try/except, returns a fallback `Text` on failure. A bad render is cosmetic, not fatal. |
| Ctrl-C still works | `_stop` event and SIGINT handler unchanged. Live context exits when the `with` block ends. In-flight requests complete; queued ones cancel. |
| Non-TTY produces no garbage | Piped output, `nohup`, log capture → plain mode via `console.is_terminal`. No ANSI escapes leak into log files. |
| Resume unchanged | All state lives in SQLite. Kill at any point; restart picks up where it left off. Dashboard reads state, never owns it. |
---
## Status: implemented ✓ (2026-06-22)
All five phases shipped. Old script archived as `photo_analyzer.py.bak-20260622-091208`.
Structure: `run_analysis()` does shared setup, then dispatches to
`_run_analysis_live()` (TTY) or `_run_analysis_plain()` (piped/redirected), both
driving the shared `_run_folder_loop()` engine. Live uses a `_Dashboard.__rich__`
that rebuilds stats/feed/progress panels each tick from in-memory dicts + a deque.
**Extra hardening found during testing (not in original plan):** the shared
`sqlite3.Connection` was committed from multiple worker threads, racing with
"cannot commit - no transaction is active". Real API latency hid it; instant
dry-run exposed it. Fixed with a module-level `_db_lock` serialising all writes
(`mark_analyzed` / `mark_error` / `mark_exif_written` / `upsert_pending`).
Verified: 200 photos × 8 workers, zero spurious errors.
Tested: plain mode (piped), live mode (PTY, 6 and 200 photos), Live-failure
fallback, render-error isolation, album windowing (>14 albums summarised),
leaf-folder album grouping.
---
## Implementation order
1. **Phase 1** — scaffold Live+Layout. Test: layout renders, progress bar
advances, Ctrl-C exits. *(must pass before continuing)*
2. **Phase 3** — in-memory album stats panel. Test: album rows appear with correct
counts, numbers increment live as photos complete.
3. **Phase 2** — log buffer + feed panel. Test: photo results appear in the feed,
errors show in red, old entries scroll off.
4. **Phase 4** — non-TTY fallback. Test: `python photo_analyzer.py ... | cat`
produces clean plain output.
5. **Phase 5** — error containment. Test: force a render exception, confirm
pipeline continues; interrupt mid-album, confirm DB is consistent on restart.

1923
INTEGRATED_PIPELINE_CONCEPT.md Normal file
View File

File diff suppressed because it is too large Load Diff

32
README.md Normal file
View File

@@ -0,0 +1,32 @@
# Photo Analyzer
Integrated, restart-safe photo analysis, duplicate review, metadata, upload, and
archive workflow. Planning lives in `INTEGRATED_PIPELINE_CONCEPT.md` and
`delivery_backlog/`.
## Agent workflow helper
Requirements: Python 3.11+, Git, and an authenticated `tea` login named `domverse`.
```bash
scripts/work-item next
scripts/work-item claim
scripts/work-item status
scripts/work-item submit --test "python3 -m unittest discover -s tests -v"
scripts/work-item submit --test "python3 -m unittest discover -s tests -v" --yes
scripts/work-item complete --merge
scripts/work-item block --reason "precise blocking condition"
```
The helper selects the lowest numbered open story whose dependencies are closed,
creates a story branch, runs required tests, rejects private or unsafe files, creates a
pull request, and closes the issue only after the merge is verified.
Workflow state is stored inside `.git/` and is never committed. Status labels are the
machine-readable source of truth; the Gitea Project board is the visual view.
## Tests
```bash
python3 -m unittest discover -s tests -v
```

285
WEBAPP_CONCEPT.md Normal file
View File

@@ -0,0 +1,285 @@
# Photo Analyzer — Web App UX Concept
Transform `photo_analyzer.py` from a terminal pipeline into a **local web app** that
reuses the `nsfwtag` review-app design language and architecture. Concept first —
implementation follows after review.
> **North star:** the analyzed data (descriptions, tags, mood, people, year, …) is
> currently invisible — it lives in a SQLite DB nobody can see. The web app's *primary*
> job is to surface it (browse + FTS5 search); the analysis pipeline becomes a
> monitored background job instead of a foreground terminal takeover.
Design validated with the `ui-ux-pro-max` skill: **Data-Dense Dashboard** pattern
(KPI cards, status colors green/amber/red, filtering-first, hover tooltips, row
highlight), debounced search with a real "no results" state, list virtualization for
25k+ photos, and — to stay dependency-free — **CSS progress bars with values-always-
visible** for stats rather than a chart library (the skill's AAA-accessible option).
---
## 1. Architecture (reuse nsfwtag's, unchanged in spirit)
- One **stdlib `ThreadingHTTPServer`** on `127.0.0.1:<random-free-port>`, opened in the
browser. No framework, no build step, no external assets (matches nsfwtag).
- One **token-injected HTML page** (`analyzer.html`, `%%TOKEN%%` replacement) + JSON
endpoints. Same "real HTML file, not a Python string" rule.
- **SQLite stays the single source of truth.** The server never owns pipeline state;
it reads the DB and an in-memory run-state object (exactly like the terminal
`_Dashboard` reads dicts + a deque). A render/HTTP crash can never corrupt the DB.
- **Path args validated against the DB** on every request (nsfwtag validates against
the scanned candidate set) — the server can't be pointed at arbitrary files.
- The existing analysis engine (`_run_folder_loop`, workers, retry/backoff, `_db_lock`,
variant grouping, nsfw-skip, `_IGNORE` exclusion) is **reused verbatim**, driven from
a background thread. The web layer is a new view + control surface, not a rewrite.
### Endpoints
| Method | Route | Purpose |
|--------|-------|---------|
| GET | `/` | The app page |
| GET | `/img?path=` | Thumbnail bytes *(reuse nsfwtag)* |
| GET | `/exif?path=` | Raw EXIF + decimal GPS + Maps URL *(reuse nsfwtag)* |
| GET | `/photo?path=` | Full analyzed record for one photo (all DB fields) |
| GET | `/search?q=&setting=&tod=&season=&people=&year_min=&year_max=&status=&album=&sort=&offset=` | Paged result set (FTS5 when `q` present) → `{rows:[…], total, offset}` |
| GET | `/facets` | Distinct filter values + counts (for the toolbar) |
| GET | `/stats` | Aggregates for the Stats view |
| GET | `/progress` | Live run state: `{running, totals, tokens, eta, albums:[…], feed:[…]}` |
| POST | `/run` | Start analysis with a config body (the CLI flags) |
| POST | `/stop` | Signal the current run to stop (reuses the existing `_stop` event) |
| GET | `/balance` · `/quota-check` | Proxy the existing functions → JSON |
| GET | `/log` | Activity log *(reuse nsfwtag)* |
Live updates use **polling** (`/progress` ~1.5 s, `/log` ~2 s) — same pattern as
nsfwtag's log panel. No SSE/WebSocket dependency.
---
## 2. Information architecture
Flat, three top-level views behind a header **segmented control** (deep-linked via
`#library` / `#analyze` / `#stats`; active state highlighted per the skill's
`nav-state-active`). An **Activity-log** slide-in panel is available from any view.
```
Header (sticky): [◧ logo] Photo Analyzer ⟨ Library · Analyze · Stats ⟩ [ total 25,318 · analyzed 25,316 · errors 8 ] [☰ log]
^ segmented, active = --text on --elev
```
- **Library** (default) — browse + search the analyzed photos. The payoff.
- **Analyze** — run controls + the live dashboard (album progress, feed, tokens/ETA).
- **Stats** — aggregates, breakdowns, error list.
---
## 3. View: Library *(browse + FTS5 search)*
```
┌ Header ─ view: Library ──────────────────────────────────────────────────────────┐
├ Toolbar (sticky) ─────────────────────────────────────────────────────────────────┤
│ 🔍 Search description, tags, mood, place… Setting▾ Time▾ Season▾ People▾ │
│ Year 1998 ●──────● 2019 ☐ has location Status▾ Sort: Relevance▾ │
├──────────────┬────────────────────────────────────────────────────────────────────┤
│ Folders │ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ← virtualized / infinite │
│ ▸ Urlaub │ │ [img] │ │ [img] │ │ [img] │ │ [img] │ scroll, server-paged │
│ 2015 USA │ │ '15 👤2│ │ '04 │ │ '11 👤5│ │ ⚠err │ │
│ ▸ job │ └───────┘ └───────┘ └───────┘ └───────┘ │
│ ▸ selfies │ beach sunset… cathedral… wedding… (error: timeout) │
│ … │ … 25,316 results … │
└──────────────┴────────────────────────────────────────────────────────────────────┘
```
- **Search** (`.field`): debounced 250 ms → `/search` FTS5 over `description, tags,
mood, location_hint`. Autocomplete-style live results (skill `Search/Autocomplete`).
Empty query = browse-all (paged). **No-results state** shows a message + suggestions
("try a broader term, or clear filters"), never a blank grid.
- **Facet filters** map every analyzed field:
- Setting (indoor/outdoor) · Time of day · Season → `select` or `.chip` toggles.
- People → buckets `0 / 1 / 2 / 3+`.
- Year → dual-handle range slider (`approx_year` min–max), tabular numbers.
- Has-location → toggle (`location_hint` not null).
- Status → analyzed / exif_written / error / pending.
- Filters AND with the search; each updates the live result count.
- **Folder sidebar** (reuse nsfwtag `.sidebar`/`.fitem`): album = **leaf folder**
(reuse `album_label`), with counts; filter to a folder + its subfolders.
- **Card** (reuse `.card`/`.thumb`): lazy `/img` thumbnail; overlays — **year badge**
+ **people badge** (reuse `.score` badge styling), a **status dot** (error=red,
pending=faint), and the first line of the description under the thumbnail. Tag chips
on hover.
- **Grid scale:** 25k+ photos → **server-paged infinite scroll** (append on scroll),
not load-all (skill `virtualize-lists`, `content-jumping`). Reserve card height to
avoid layout shift.
- **Lightbox** (reuse nsfwtag lightbox + keyboard + prev/next within the current result
set): left = full image; right = a two-section panel:
1. **Analysis** — description, tags, mood, setting, time-of-day, season, people,
year, location. (The EXIF caption Immich will index, made human-readable.)
2. **EXIF** — raw fields + GPS **Google Maps** link (reuse `/exif`).
Per-photo action: **Re-analyze this one** (POST `/run` scoped to one path).
---
## 4. View: Analyze *(run controls + live dashboard)*
Mirrors the terminal `_Dashboard` (see `DASHBOARD_PLAN.md`) as a web surface.
```
┌ Header ─ view: Analyze ───────────────────────────────────────────────────────────┐
├ Run bar ──────────────────────────────────────────────────────────────────────────┤
│ Library [ /…/pictures ▾ ] ☐ Dry-run ☐ Re-analyze ☐ No-EXIF ☐ EXIF-only ☐ Variants │
│ [ Balance ] [ Quota check ] [▶ Start] │
├ Overall ──────────────────────────────────────────────────────────────────────────┤
│ ████████████████░░░░░░░░ 238 / 550 43 % ok 235 · err 3 │
│ tokens 284,192 · est. remaining ~340,000 · ETA 14:23 [■ Stop] (danger) │
├ Albums (data-dense) ──────────────────┬ Recent feed ───────────────────────────────┤
│ Camera Roll 238/550 ███████░░ 43% │ ✓ IMG_4821.jpg │
│ Urlaub/Rom 45/ 45 ████████ ✓ │ A golden retriever on a sunny beach │
│ Urlaub/Venedig 12/ 60 ██░░░░░░ 20% │ ✗ DSC_0042.jpg — 4 retries: timeout │
│ WhatsApp Images 0/120 ░░░░░░░░ 0% │ ✓ IMG_4820.jpg │
│ … │ Two children building a sandcastle │
└───────────────────────────────────────┴─────────────────────────────────────────────┘
```
- **Run bar** = the CLI flags as controls:
Library path (default from `LIBRARY` env) · **Dry-run · Re-analyze · No-EXIF ·
EXIF-only · Group-variants** toggles · **Start** (primary) / **Stop** (danger).
One run at a time (guarded); Start on an interrupted library simply **resumes**
(DB is truth) — resume-safety is automatic, no separate control.
- **Overall panel**: big progress bar, ok/err counts, tokens used + estimated
remaining + ETA (all tabular). Status colors: green done, amber running, red error.
- **Albums panel**: one row per **leaf-folder album** with a mini progress bar + `✓`
when fully done (the DASHBOARD_PLAN target, now clickable → jumps to that album in
Library). Future albums shown at 0 %.
- **Recent feed**: newest-first list — thumbnail + filename + one-line description;
errors in red with the message. Reuses the activity-log line styling.
- **Balance / Quota-check**: small buttons → `/balance`, `/quota-check` → toast/inline
result. **Debug log**: available via the Activity-log panel.
- Everything here is driven by polling `/progress`; the panel degrades to "Idle — no
run in progress" with a Start hint when nothing is running (empty state with action).
---
## 5. View: Stats *(aggregates + errors, no chart lib)*
```
┌ KPI cards ────────────────────────────────────────────────────────────────────────┐
│ 25,318 total 25,316 analyzed 25,316 EXIF-written 8 errors 2 pending │
├ Breakdowns (CSS bars, counts always visible) ───────────────────────────────────────┤
│ Setting outdoor ██████████████ 14,902 indoor ████████ 9,120 │
│ Season summer ████████ · autumn ██████ · spring ████ · winter ███ · unknown ██ │
│ People 0 ████████ · 1 ██████ · 2 ████ · 3+ ███ │
│ By year 1998▁ 2003▃ 2004▅ 2010▇ 2011█ 2015█ … (histogram of bars) │
│ Top tags beach 1,204 · wedding 980 · city 900 · food 610 … │
├ Errors ─────────────────────────────────────────────────────────────────────────────┤
│ DSC_0042.jpg — connection timeout (4 retries) [Retry] │
│ … │
└──────────────────────────────────────────────────────────────────────────────────────┘
```
- KPI cards reuse `.metric`. Breakdowns are **horizontal CSS bars with the count
printed** (skill: values-always-visible, AAA; no dependency, no CSP issue).
- Error table lists `status='error'` rows with `error_message` + a **Retry** action
(re-queues that path for the next run — matches "re-running retries error records").
---
## 6. Component inventory (reuse existing tokens)
Everything below already exists in `nsfwtag/review.html`'s `:root` token set and class
library — the web app inherits it wholesale (same dark OLED palette, radii, focus
rings, reduced-motion guard).
| Component | Source | New/changed |
|---|---|---|
| Header `.app`/`.brand`/`.logo`/`.titles` | reuse | title copy |
| KPI `.metrics`/`.metric` | reuse | used for header counts + Stats cards |
| View switcher | reuse `.seg` | active-state = current view |
| Toolbar `.toolbar` | reuse | — |
| Search `.field` | reuse | — |
| Facet `.chip` / `select` / `.seg` | reuse | People/Setting/Season/Status |
| Range slider `input[type=range]` | reuse | **dual-handle** year range (small JS) |
| Sidebar `.sidebar`/`.fitem` | reuse | album tree |
| Grid `.grid`/`.card`/`.thumb` | reuse | + year/people badges, status dot |
| Tag chips | reuse `.chip` (mini) | — |
| Lightbox + EXIF panel | reuse | + Analysis section |
| Buttons `.btn`/`.primary`/`.danger`/`.ghost` | reuse | Start/Stop/Balance |
| Toast `.toast` | reuse | — |
| Activity log panel | reuse | run events too |
| **Progress bar** | **new** `.bar` (fill = `--primary`) | overall + per-album + stat bars |
| **Run toggles** | **new** switch styled from tokens | the 5 flags |
**One new token:** `--warn:#f59e0b` (amber) for "running / partial" states — the only
gap in the current green/red/faint status set.
---
## 7. Interaction model
- **View switching**: header segmented control; state deep-linked in the URL hash and
preserved on back/forward (skill `deep-linking`, `state-preservation`).
- **Search + filters**: debounced; combine as AND; the URL hash encodes `q` + active
filters so a search is shareable and survives reload. Result count updates live.
- **Browse at scale**: infinite scroll appends server-paged results; card heights
reserved (no CLS); thumbnails lazy-load.
- **Lightbox**: `←/→` navigate the current result set, `Esc`/backdrop close; EXIF and
full record fetched lazily on open.
- **Analyze**: Start → POST `/run` (config) → server spawns the run on a background
thread → dashboard polls `/progress`. Stop → POST `/stop` → sets the existing
`_stop` event; in-flight requests drain, DB stays consistent, restart resumes.
- **Accessibility/motion**: keyboard nav + visible focus rings (inherited), status
never by color alone (badges carry text/icons), `prefers-reduced-motion` respected
(token media query already present), all touch targets ≥ 36–44 px.
---
## 8. CLI → Web feature parity
| CLI feature | Web surface | Preserved how |
|---|---|---|
| `--library` | Analyze › Run bar path (default `LIBRARY` env) | same discovery |
| `--dry-run` | Run toggle | passed to engine |
| `--reanalyze` | Run toggle | passed to engine |
| `--no-exif` | Run toggle | passed to engine |
| `--exif-only` | Run toggle | passed to engine |
| `--group-variants` | Run toggle | reuses variant grouping |
| `--stats` | **Stats view** | `/stats` aggregates |
| `--balance` | Balance button | `/balance` |
| `--quota-check` | Quota-check button | `/quota-check` |
| `--debug` | Activity-log panel | server log |
| resume-safety | automatic (Start resumes) | DB source of truth |
| `_IGNORE/` exclusion | server-side discovery | `discover_photos` unchanged |
| nsfw-skip | automatic in every run | `filter_nsfw_tagged` unchanged |
| FTS5 search *(CLI had none)* | **Library search** | new surface for existing FTS5 table |
| DB (all fields) | Library cards + lightbox + facets | new surface for existing schema |
Nothing is dropped. The only genuinely new capability is **surfacing** what the DB
already stores (search/browse/stats) — everything else is the same engine behind a
web control panel instead of a terminal.
---
## 9. Proposed file layout (mirrors `nsfwtag/`)
```
webapp/ (or analyzer_web/)
├── __main__.py # CLI: python -m webapp → starts server, opens browser
├── server.py # ThreadingHTTPServer + routes (reuse nsfwtag/server.py shape)
├── page.py # render analyzer.html with %%TOKENS%% (reuse webapp.py shape)
├── analyzer.html # the single-page frontend (inherits nsfwtag tokens/classes)
├── query.py # DB read helpers: search (FTS5), facets, stats, one-photo
└── runner.py # drive photo_analyzer's engine on a background thread + run-state
```
`photo_analyzer.py` stays the pipeline library; `runner.py` imports and drives it. No
duplication of analysis logic.
---
## 10. Build order (for after approval)
1. **Read-only Library first** — server + page + `/search` + `/facets` + grid +
lightbox over the *existing* DB. Immediately useful, zero risk to the pipeline.
2. **Stats view** — `/stats` + CSS breakdown bars.
3. **Analyze view** — `/run` + `/stop` + `/progress` polling + run-state object,
driving the existing engine on a thread.
4. **Balance/quota/log** wiring + polish (empty states, deep-link hash, reduced-motion).
Each step ships independently; the pipeline keeps working from the CLI throughout.

204
album_naming.md Normal file
View File

@@ -0,0 +1,204 @@
# Immich album naming scheme
Goal: turn the messy on-disk album folders into clean, consistent Immich albums.
## The scheme
**Pattern:** `YYYY[-MM[-DD]] Name` for anything datable; `Category – Detail` for
undated evergreen sets.
Rules:
1. **Year leads, ISO format.** `2015 USA`, not `USA 2015`. Add month/day only when
you actually know it: weddings get `YYYY-MM-DD`, month-tagged trips get `YYYY-MM`.
A leading 4-digit year sorts every dated album chronologically in Immich's
(alphabetical) album list; lettered evergreen albums fall below them.
2. **One name, one album.** Keep German, keep umlauts (Immich is UTF-8). Title Case.
3. **Separator:** a single space after the date; ` – ` (en dash) or `( )` for a
qualifier. No underscores, no double spaces, no cryptic tokens.
4. **Decode export artifacts.** Embedded month numbers (`Brindisi 9 2010`,
`Kuba 1 2011`) become `YYYY-MM`. Drop `alle` / `_MobileMe` / `MobileMe` — those
are old Apple/full-export copies, not real albums (see *Duplicates* below).
5. **Flatten the fake `Urlaub/` tree.** `Urlaub` (holiday) currently holds weddings,
job photos, and applications too. Drop it as a parent; the year prefix does the
grouping.
## How to apply it (no disk renames)
immich-go derives the album from the folder, but you can override per upload:
```bash
immich-go upload \
--server https://immich.domverse-berlin.eu --api-key <key> --no-ui \
from-folder \
--album "2015 USA" \
"/Users/domverse/Documents/Bilder/pictures/Urlaub/2015 USA/"
```
One `--album` per folder, using the *Proposed* name from the table below. This
keeps folder paths (and content hashes) untouched — no re-upload risk. Multiple
folders can map to the *same* `--album` name to merge them (that's how the
duplicates collapse).
> Verify the exact flag against your immich-go version — recent builds use
> `--album "<name>"` on `from-folder`; older ones used `--create-album-name`.
---
## Mapping
`[m?]` = month inferred from an embedded number, confirm before merging.
`[y?]` = year is a guess, confirm.
### Travel (chronological)
| Current folder | Proposed album |
|---|---|
| Urlaub/Indien 2003 | `2003 Indien` |
| Urlaub/Irina München 2003 | `2003 München (Irina)` |
| Urlaub/Mallorca 2003 | `2003 Mallorca` |
| Urlaub/Italien 2004 | `2004 Italien` |
| Urlaub/Irina 2004 | `2004 Irina` |
| Urlaub/Melania 30.Geb 2004 | `2004 Melania 30. Geburtstag` |
| Urlaub/Spanien 2007 | `2007 Spanien` |
| Urlaub/Tauchsafari 2008 | `2008 Tauchsafari` |
| Urlaub/Marokko 2009 + Urlaub/Marokko | `2009 Marokko` *(merge)* |
| Urlaub/Tauchen 2009 | `2009 Tauchen` |
| Urlaub/Ägypten 2010 | `2010 Ägypten` |
| Urlaub/Karwendel 2010 | `2010 Karwendel` |
| Urlaub/Maui 2010 | `2010 Maui` |
| Urlaub/Bootsausflug 2010 | `2010 Bootsausflug` |
| Urlaub/Kuba 2010 | `2010 Kuba` |
| Urlaub/Brindisi 9 2010 | `2010-09 Brindisi` `[m?]` |
| Urlaub/Wanderurlaub 6 2010 (+ …alle) | `2010-06 Wanderurlaub` `[m?]` *(merge)* |
| Urlaub/Italien 2011 | `2011 Italien` |
| Urlaub/Italien 7 2011 (+ …alle) | `2011-07 Italien` `[m?]` *(merge)* |
| Urlaub/Kuba 1 2011 (+ …alle, …_MobileMe, Kuba alle) | `2011-01 Kuba` `[m?]` *(merge)* |
| 2012 Dienstreise Zürich | `2012 Dienstreise Zürich` |
| 2012 Safari Hochzeitsreise | `2012 Hochzeitsreise (Safari)` |
| Urlaub/2013 Paris | `2013 Paris` |
| Urlaub/Schwedenurlaub (Edit) 07-2013 | `2013-07 Schweden (Edit)` |
| Urlaub/Schwedenurlaub (Tanja) 07-2013 | `2013-07 Schweden (Tanja)` |
| Urlaub/2014 Graz Nürnberg | `2014 Graz & Nürnberg` |
| Urlaub/2014 London | `2014 London` |
| Urlaub/2015 USA + Urlaub/USA 2015 | `2015 USA` *(merge)* |
| Urlaub/2015 XMas Paris | `2015-12 Paris (Weihnachten)` |
| Urlaub/2016 Tauchen | `2016 Tauchen` |
| Urlaub/2017 Paris | `2017 Paris` |
| Urlaub/2017 Ägypten | `2017 Ägypten` |
| Urlaub/2017 Mama in HH | `2017 Mama in Hamburg` |
| Urlaub/2017 Mammutmarsch + Mammutmarsch | `2017 Mammutmarsch` *(merge)* |
| Urlaub/2018 Italien | `2018 Italien` |
| Urlaub/2018 Ausflug Sylvester | `2018 Ausflug Silvester` |
| Urlaub/2019 Tauchen | `2019 Tauchen` |
| Urlaub/Berlin | `Berlin` `[y?]` |
| Urlaub/Uckermark | `Uckermark` `[y?]` |
| Urlaub/Malerweg | `Malerweg` `[y?]` |
| Urlaub/Dresden_im_Winter | `Dresden im Winter` `[y?]` |
| Sächsische Schweiz | `Sächsische Schweiz` `[y?]` |
| Urlaub/Kurzurlaube | `Kurzurlaube` |
| Urlaub/foodlog | `Foodlog` |
### Weddings
| Current folder | Proposed album |
|---|---|
| Urlaub/Hochzeit Alex _ Tanja | `2012-04-05 Hochzeit Alex & Tanja` *(your wedding)* |
| Urlaub/Hochzeit 05.04.2012 - Dia | `2012-04-05 Hochzeit Alex & Tanja (Dia-Scan)` |
| Urlaub/Hochzeit 05.04.2012 - Farbe | `2012-04-05 Hochzeit Alex & Tanja (Farbscan)` |
| Urlaub/Hochzeit 05.04.2012 - SW | `2012-04-05 Hochzeit Alex & Tanja (SW-Scan)` |
| 2012 Hochzeit | → fold into `2012-04-05 Hochzeit Alex & Tanja` `[?]` |
| Urlaub/Hochzeit (934) + Urlaub/Hochzeit_Gesamt (577) | `2012-04-05 Hochzeit Alex & Tanja (Gesamt)` `[?]` *(confirm same wedding)* |
| Urlaub/Hochzeit_Astrid_Paul | `Hochzeit Astrid & Paul` `[y?]` |
| Urlaub/Hochzeit_Carlos | `Hochzeit Carlos` `[y?]` |
| Urlaub/Hochzeit_Carlos Freunde | `Hochzeit Carlos (Freunde)` `[y?]` |
| Urlaub/Hochzeit_Cosimo | `Hochzeit Cosimo` `[y?]` |
| Urlaub/Hochezeit_Bjoern | `Hochzeit Björn` `[y?]` *(typo fixed)* |
| Urlaub/Sardinien & Hochzeit Alberto Kerstin | `Sardinien – Hochzeit Alberto & Kerstin` `[y?]` |
| Urlaub/Hochzeitessen Italien | `Hochzeitsessen Italien` `[y?]` |
### Applications / Job
| Current folder | Proposed album |
|---|---|
| Urlaub/Bewerbung 2006 | `2006 Bewerbung` |
| 2008_erste Bewerbungsbilder | `2008 Bewerbungsbilder (erste)` |
| Urlaub/Bewerbungsfotos 2011 | `2011 Bewerbungsfotos` |
| Urlaub/Bewerbungsfotos (Auswahl) | `Bewerbungsfotos (Auswahl)` |
| bewerbungsbilder | `Bewerbungsbilder` |
| bewerbungsbilder/foto_karoline_wolf | `Bewerbungsbilder (Karoline Wolf)` |
| Urlaub/FOKUS SE 2007 + Urlaub/Fokus 2007 | `2007 Fokus SE` *(merge)* |
| job/Capgemini | `Job – Capgemini` |
| job/Capgemini/jugendhackt | `Job – Capgemini (Jugend Hackt)` |
| job/adesso | `Job – adesso` |
| job/lucanet | `Job – LucaNet` |
| job/TS MMS | `Job – TS MMS` |
| Urlaub/2014 Jugend Hackt | `2014 Jugend Hackt` |
| Urlaub/2016_GTA | `2016 GTA` `[?]` |
### People / Family / Personal
| Current folder | Proposed album |
|---|---|
| Urlaub/Fotos Tanja _ Alex 3 2008 | `2008-03 Tanja & Alex` `[m?]` |
| Teenie Jahre | `Teenie Jahre` |
| Teenie Jahre/[Originals] | `Teenie Jahre (Originals)` |
| eltern | `Familie – Eltern` |
| eltern_tanja | `Familie – Eltern (Tanja)` |
| tanja | `Tanja` |
| Camera_Tanja + 0_Open/Camera_Tanja | `Kamera – Tanja` *(merge, same 243)* |
| 0_Open/Tanja Best Of Egypt | `Ägypten – Best Of (Tanja)` `[y?]` |
| selfies | `Selfies` |
| selfies/alex | `Selfies – Alex` |
| selfies/tanja | `Selfies – Tanja` |
| selfies/wir | `Selfies – Wir` |
| profile | `Profilbilder` |
| workout | `Workout` |
### Evergreen / camera roll / misc
| Current folder | Proposed album |
|---|---|
| Camera Roll | `Camera Roll` |
| Urlaub/iphone | `iPhone` |
| Urlaub/Handyvideos | `Handyvideos` |
| Wohnung | `Wohnung` |
| 2012_Wohnung DD | `2012 Wohnung Dresden` |
| Urlaub/Zu Hause | `Zu Hause` |
| fun_pics | `Fun Pics` |
| Urlaub/Auswahl Abzug Bild | `Abzüge (Auswahl)` |
| Urlaub/Unterschrift | `Unterschrift` `[?]` |
| Wallpaper/UW_Bilder | `Wallpaper – Unterwasser` |
| Wallpaper/wallpaper_sfw | `Wallpaper` |
| Wallpaper/wallpaper_ww | `Wallpaper – WW` `[?]` |
---
## Duplicates to merge (point several folders at one `--album`)
- **Marokko** — `Marokko 2009` + `Marokko` → `2009 Marokko`
- **Kuba** — `Kuba 1 2011` + `…alle` + `…_MobileMe` + `Kuba alle` → `2011-01 Kuba`; keep `Kuba 2010` separate as `2010 Kuba`
- **Italien 7 2011** + `…alle` → `2011-07 Italien`
- **Wanderurlaub 6 2010** + `…alle` → `2010-06 Wanderurlaub`
- **USA 2015** + `2015 USA` → `2015 USA`
- **Mammutmarsch** (top-level) + `Urlaub/2017 Mammutmarsch` → `2017 Mammutmarsch`
- **Camera_Tanja** (top-level) + `0_Open/Camera_Tanja` → `Kamera – Tanja`
- **Fokus** — `FOKUS SE 2007` + `Fokus 2007` → `2007 Fokus SE`
immich-go dedupes on content hash, so uploading two folders into the same album
never creates duplicate assets — identical files land once, the album just gains
both source sets.
## Needs your call (I guessed or couldn't tell)
- **Years** for: Berlin, Uckermark, Malerweg, Dresden im Winter, Sächsische Schweiz,
and all the friends' weddings (`Astrid & Paul`, `Carlos`, `Cosimo`, `Björn`,
`Sardinien`). Add `YYYY ` if you know them.
- **`2016_GTA`** (620) — game screenshots? Greater Toronto? a person? → real name.
- **`Unterschrift`** (378, "signature"), **`SiS`** (24), **`ZGIDC`** (18),
**`Horka`** (1), **`Neues Album`** (4), **`Tanja`** (4) — cryptic/tiny; rename or
skip.
- **Big `Hochzeit` (934) / `Hochzeit_Gesamt` (577)** — confirm these are your own
2012 wedding before merging into that album.
- **Month inferences** `[m?]` — verify `9 2010`, `1 2011`, `7 2011`, `6 2010`,
`3 2008` are months, not sequence numbers, before locking the `-MM`.

179
compare_models.py Normal file
View File

@@ -0,0 +1,179 @@
#!/usr/bin/env python3
"""
Side-by-side model comparison for photo_analyzer.
Reads already-analyzed photos from the DB (the stored result — no re-run, no
extra cost on the model that produced them) and runs a second model (default:
Gemini 2.5 Flash via its OpenAI-compatible endpoint) live on the same images,
then prints the two results next to each other plus measured token cost.
Reuses photo_analyzer's prepare_image, ANALYSIS_PROMPT, get_db and env loader —
no logic is duplicated.
Usage:
GEMINI_API_KEY=... python compare_models.py -n 10
python compare_models.py --selftest # offline check, no API call
Put GEMINI_API_KEY in photo_analyzer.env (the same file the main pipeline reads).
"""
import argparse
import json
import os
import random
import sys
from collections import defaultdict
from pathlib import Path
from openai import OpenAI
import photo_analyzer as pa
# Gemini's OpenAI-compatible endpoint — the existing OpenAI client just works.
GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai/"
DEFAULT_MODEL = "gemini-2.5-flash"
# $/million tokens, for the live model's measured cost. Update if you change model.
PRICES = { # (input, output)
"gemini-2.5-flash": (0.30, 2.50),
"gemini-2.5-flash-lite": (0.10, 0.40),
"gemini-3.5-flash": (1.50, 9.00),
}
def _strip_json_fences(raw: str) -> str:
"""Same fence-stripping photo_analyzer uses — models often wrap JSON in ```."""
raw = raw.strip()
if raw.startswith("```"):
raw = raw.split("```")[1]
if raw.startswith("json"):
raw = raw[4:]
raw = raw.strip()
return raw
def analyze_with(client: OpenAI, model: str, path: Path) -> tuple[dict, dict]:
"""Run `model` on one image. Returns (parsed_result, usage_dict)."""
b64, mime = pa.prepare_image(path)
resp = client.chat.completions.create(
model=model,
max_tokens=4096,
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": f"data:{mime};base64,{b64}"}},
{"type": "text", "text": pa.ANALYSIS_PROMPT},
],
}],
)
result = json.loads(_strip_json_fences(resp.choices[0].message.content))
u = resp.usage
usage = {"prompt": u.prompt_tokens if u else 0,
"completion": u.completion_tokens if u else 0}
return result, usage
def fetch_analyzed(conn, limit: int) -> list:
"""
Randomly sample already-analyzed photos, spread across folders. Groups rows by
parent directory, shuffles, then round-robins one per folder so the sample
covers as many different folders as possible. Only picked files are checked on
disk (cheap), not the whole library.
"""
# "analyzed already" = has a description, regardless of status — once EXIF is
# written the status flips from 'analyzed' to 'exif_written', but the result stays.
rows = conn.execute(
"""SELECT path, description, tags, people_count, mood, setting, approx_year
FROM photos WHERE description IS NOT NULL AND description != ''""").fetchall()
buckets = defaultdict(list)
for r in rows:
buckets[str(Path(r["path"]).parent)].append(r)
folders = list(buckets.keys())
random.shuffle(folders)
for f in folders:
random.shuffle(buckets[f])
out, exhausted = [], False
while len(out) < limit and not exhausted:
exhausted = True
for f in folders: # one per folder per pass → spread
if not buckets[f]:
continue
exhausted = False
r = buckets[f].pop()
if Path(r["path"]).exists():
out.append(r)
if len(out) >= limit:
break
return out
def _fmt(label: str, desc, tags, people, mood) -> str:
tags_s = ", ".join(tags) if isinstance(tags, list) else (tags or "")
return (f" {label}\n"
f" desc: {desc}\n"
f" tags: {tags_s}\n"
f" people: {people} mood: {mood}")
def selftest():
assert _strip_json_fences('```json\n{"a":1}\n```') == '{"a":1}'
assert _strip_json_fences('{"a":1}') == '{"a":1}'
assert _strip_json_fences('```\n{"b":2}\n```') == '{"b":2}'
print("selftest OK")
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("-n", "--num", type=int, default=10, help="photos to compare")
ap.add_argument("--model", default=DEFAULT_MODEL, help="live model to compare against the stored results")
ap.add_argument("--db", default="photo_analysis.db")
ap.add_argument("--selftest", action="store_true", help="run offline check and exit")
args = ap.parse_args()
if args.selftest:
selftest()
return
pa.load_env_file() # picks up GEMINI_API_KEY from photo_analyzer.env
key = os.environ.get("GEMINI_API_KEY")
if not key:
sys.exit("GEMINI_API_KEY not set (add it to photo_analyzer.env or export it).")
conn = pa.get_db(args.db)
rows = fetch_analyzed(conn, args.num)
if not rows:
sys.exit("No analyzed photos found in the DB to compare against.")
client = OpenAI(api_key=key, base_url=GEMINI_BASE_URL)
p_in, p_out = PRICES.get(args.model, (0.0, 0.0))
tot_in = tot_out = 0
ok = 0
for r in rows:
path = Path(r["path"])
print(f"\n=== {path.parent.name}/{path.name} ===")
# stored result (already in the DB — no re-run)
try:
stored_tags = json.loads(r["tags"]) if r["tags"] else []
except Exception:
stored_tags = r["tags"]
print(_fmt("stored (DB)", r["description"], stored_tags,
r["people_count"], r["mood"]))
# live call to the comparison model
try:
res, usage = analyze_with(client, args.model, path)
tot_in += usage["prompt"]; tot_out += usage["completion"]; ok += 1
print(_fmt(f"{args.model} (live)", res.get("description"),
res.get("tags", []), res.get("people_count"), res.get("mood")))
except Exception as e:
print(f" {args.model} (live)\n ERROR: {e}")
if ok:
cost = tot_in / 1e6 * p_in + tot_out / 1e6 * p_out
print(f"\n— {args.model}: {ok} photos, {tot_in:,} in + {tot_out:,} out tokens, "
f"measured cost ${cost:.4f} (${cost / ok * 1000:.2f}/1k photos)")
if __name__ == "__main__":
main()

21
delivery_backlog/E01-shared-identity-inventory.md Normal file
View File

@@ -0,0 +1,21 @@
# E01 — Shared Identity and Inventory
Concept phase: A. Establish the application foundation, stable asset identity,
inventory, duplicate review, and managed thumbnails by extracting proven CLI donor
code wherever possible.
## Stories
1. [US01-01 — Inventory and characterize donor code](stories/US01-01-donor-characterization.md)
2. [US01-02 — Establish application and database foundation](stories/US01-02-app-database-foundation.md)
3. [US01-03 — Discover and reconcile stable assets](stories/US01-03-inventory-identity.md)
4. [US01-04 — Detect and decide duplicates](stories/US01-04-duplicate-engine.md)
5. [US01-05 — Generate and serve managed thumbnails](stories/US01-05-thumbnail-service.md)
6. [US01-06 — Review inventory and duplicates in the browser](stories/US01-06-inventory-review-ui.md)
7. [US01-07 — Automate Phase A end-to-end acceptance](stories/US01-07-phase-a-e2e.md)
## Epic outcome
The system discovers the fixture library without violating path policy, preserves
asset identity across moves, creates explainable duplicate clusters, serves oriented
thumbnails, and persists review decisions through restart.

20
delivery_backlog/E02-unified-workflow-shell.md Normal file
View File

@@ -0,0 +1,20 @@
# E02 — Unified Workflow Shell
Concept phase: B. Add durable jobs and the shared browser application, then migrate
safety review and the existing Library, Analyze, and Stats experiences onto the common
API and service layer.
## Stories
1. [US02-01 — Extract safety and existing web UI donors](stories/US02-01-safety-ui-donors.md)
2. [US02-02 — Persist and coordinate durable jobs](stories/US02-02-durable-jobs.md)
3. [US02-03 — Execute jobs with leases, locks, and recovery](stories/US02-03-worker-recovery.md)
4. [US02-04 — Expose job APIs and activity events](stories/US02-04-job-api-events.md)
5. [US02-05 — Build the static application shell](stories/US02-05-frontend-shell.md)
6. [US02-06 — Deliver workflow, safety, library, analysis, and stats views](stories/US02-06-workflow-views.md)
7. [US02-07 — Automate Phase B end-to-end acceptance](stories/US02-07-phase-b-e2e.md)
## Epic outcome
Users can operate and observe the shared workflow through one browser application;
mutating jobs are durable, cancellable, recoverable, and limited to safe concurrency.

17
delivery_backlog/E03-album-proposals.md Normal file
View File

@@ -0,0 +1,17 @@
# E03 — Album Proposals
Concept phase: C. Turn analyzed folder metadata into editable, explainable, versioned
album naming proposals without mutating files.
## Stories
1. [US03-01 — Aggregate album evidence](stories/US03-01-album-evidence.md)
2. [US03-02 — Define album naming policy](stories/US03-02-naming-policy.md)
3. [US03-03 — Generate and persist versioned proposals](stories/US03-03-proposal-generation.md)
4. [US03-04 — Review and approve proposals in the browser](stories/US03-04-proposal-ui.md)
5. [US03-05 — Automate Phase C end-to-end acceptance](stories/US03-05-phase-c-e2e.md)
## Epic outcome
Users receive deterministic, evidence-backed album suggestions, can edit them safely,
and can approve a validated proposal without any rename occurring yet.

18
delivery_backlog/E04-guarded-renaming.md Normal file
View File

@@ -0,0 +1,18 @@
# E04 — Guarded Renaming
Concept phase: D. Convert approved proposals into validated, journaled, recoverable
filesystem rename operations.
## Stories
1. [US04-01 — Build and export rename plans](stories/US04-01-rename-plan.md)
2. [US04-02 — Journal rename state and preconditions](stories/US04-02-rename-journal.md)
3. [US04-03 — Apply and verify guarded renames](stories/US04-03-rename-apply.md)
4. [US04-04 — Recover or roll back interrupted renames](stories/US04-04-rename-recovery.md)
5. [US04-05 — Operate rename plans in the browser](stories/US04-05-rename-ui.md)
6. [US04-06 — Automate Phase D end-to-end acceptance](stories/US04-06-phase-d-e2e.md)
## Epic outcome
No rename occurs without a current validated plan and explicit confirmation; crashes
produce deterministic recovery or a precise manual-recovery state.

18
delivery_backlog/E05-immich-upload.md Normal file
View File

@@ -0,0 +1,18 @@
# E05 — Immich Upload
Concept phase: E. Upload verified album bytes to Immich through controlled,
observable, restart-safe `immich-go` batches.
## Stories
1. [US05-01 — Validate credentials and upload readiness](stories/US05-01-upload-preflight.md)
2. [US05-02 — Orchestrate album upload batches](stories/US05-02-upload-batches.md)
3. [US05-03 — Parse and persist uploader outcomes](stories/US05-03-upload-reports.md)
4. [US05-04 — Verify, retry, and resolve uncertain uploads](stories/US05-04-upload-verification.md)
5. [US05-05 — Operate uploads in the browser](stories/US05-05-upload-ui.md)
6. [US05-06 — Automate Phase E end-to-end acceptance](stories/US05-06-phase-e-e2e.md)
## Epic outcome
Users can upload an explicitly scoped album only after metadata checkpoints pass and
can distinguish new, duplicate, upgraded, failed, and uncertain outcomes.

18
delivery_backlog/E06-archive-lifecycle.md Normal file
View File

@@ -0,0 +1,18 @@
# E06 — Archive Lifecycle
Concept phase: F. Reclaim active storage only after verified archival, while retaining
identity, duplicate evidence, browsing, recovery, and restoration.
## Stories
1. [US06-01 — Configure and preflight archive destinations](stories/US06-01-archive-preflight.md)
2. [US06-02 — Transfer, verify, and remove active sources](stories/US06-02-archive-transfer.md)
3. [US06-03 — Preserve offline identity and review evidence](stories/US06-03-offline-assets.md)
4. [US06-04 — Plan and execute safe restores](stories/US06-04-restore.md)
5. [US06-05 — Operate archive and restore in the browser](stories/US06-05-archive-ui.md)
6. [US06-06 — Automate Phase F end-to-end acceptance](stories/US06-06-phase-f-e2e.md)
## Epic outcome
Uploaded albums can be archived without data loss and later browsed, deduplicated,
recovered, and restored even when archive media is temporarily offline.

19
delivery_backlog/E07-hardening-release.md Normal file
View File

@@ -0,0 +1,19 @@
# E07 — Hardening and Release
Concept phase: G. Resolve migration debt, harden hostile and failure conditions,
validate performance, document operations, and prove release readiness.
## Stories
1. [US07-01 — Complete donor migration and freeze the CLI archive](stories/US07-01-donor-archive.md)
2. [US07-02 — Harden API authorization and path boundaries](stories/US07-02-security-hardening.md)
3. [US07-03 — Harden media and metadata edge cases](stories/US07-03-media-hardening.md)
4. [US07-04 — Prove concurrency and crash recovery](stories/US07-04-fault-concurrency.md)
5. [US07-05 — Deliver backup and operational recovery](stories/US07-05-backup-operations.md)
6. [US07-06 — Validate performance and resource bounds](stories/US07-06-performance.md)
7. [US07-07 — Automate full release acceptance](stories/US07-07-release-e2e.md)
## Epic outcome
The integrated application meets security, durability, recovery, performance, donor
parity, and full workflow release gates with reproducible automated evidence.

49
delivery_backlog/README.md Normal file
View File

@@ -0,0 +1,49 @@
# Integrated Photo Pipeline Delivery Backlog
This backlog decomposes the phases in
[`INTEGRATED_PIPELINE_CONCEPT.md`](../INTEGRATED_PIPELINE_CONCEPT.md) into seven
epics and small, independently verifiable user stories.
## Numbering and file naming
- Epics: `E01` through `E07`, matching concept Phases A through G.
- Stories: `US<epic>-<sequence>`, for example `US03-02`.
- Epic files: `E01-<slug>.md`.
- Story files: `stories/US01-01-<slug>.md`.
- IDs are permanent. Renaming or reordering a story must not reuse its ID.
## Delivery rules
- Implement epics in numeric order unless an epic explicitly permits overlap.
- A story should produce one coherent, reviewable outcome and be small enough for one
focused implementation change.
- Donor-first migration is mandatory: characterize and extract useful CLI behavior
before replacing it. Record provenance in the donor ledger.
- Every implementation story includes automated unit, property, golden, contract, or
integration tests appropriate to its risk.
- Every epic ends with a dedicated automated end-to-end story that exercises the real
browser, API/SSE, worker, isolated database, and temporary fixture library.
- A story is complete only when all acceptance criteria and automated tests pass with
no unexplained skips. An epic is complete only when its cumulative regression suite
and end-to-end gate pass.
## Epics
1. [E01 — Shared identity and inventory](E01-shared-identity-inventory.md)
2. [E02 — Unified workflow shell](E02-unified-workflow-shell.md)
3. [E03 — Album proposals](E03-album-proposals.md)
4. [E04 — Guarded renaming](E04-guarded-renaming.md)
5. [E05 — Immich upload](E05-immich-upload.md)
6. [E06 — Archive lifecycle](E06-archive-lifecycle.md)
7. [E07 — Hardening and release](E07-hardening-release.md)
## Shared definition of done
- Acceptance criteria are demonstrably satisfied.
- Public API/schema changes are documented and contract-tested.
- Database and filesystem mutations are restart-safe and idempotent where required.
- Paths are constrained by the shared library policy.
- No credentials or private image data appear in logs or browser responses.
- Relevant donor-ledger rows and story-to-test traceability entries are current.
- Automated tests pass locally and in CI; changed user behavior has an automated
acceptance test.

20
delivery_backlog/stories/US01-01-donor-characterization.md Normal file
View File

@@ -0,0 +1,20 @@
# US01-01 — Inventory and Characterize Donor Code
Epic: [E01](../E01-shared-identity-inventory.md)
As a maintainer, I want a donor ledger and characterization suite for the current CLI
implementations so proven behavior is reused deliberately instead of rewritten.
## Acceptance criteria
- Relevant functions in both CLI codebases are classified as reuse, extract, refactor,
or replace, with rationale and target module.
- The ledger covers discovery, hashing, imaging, NSFW, vision, EXIF, database, UI,
configuration, logging, cancellation, and error behavior.
- Representative current outputs are captured against stable fixture IDs.
- No legacy file is moved or archived in this story.
## Automated tests
- Characterization tests run against current donor entry points and core functions.
- A ledger-lint test rejects missing source references, target locations, or test IDs.

24
delivery_backlog/stories/US01-02-app-database-foundation.md Normal file
View File

@@ -0,0 +1,24 @@
# US01-02 — Establish Application and Database Foundation
Epic: [E01](../E01-shared-identity-inventory.md)
As an operator, I want the API, worker configuration, and database lifecycle to start
consistently so later services share one durable foundation.
## Acceptance criteria
- FastAPI factory, typed configuration, SQLAlchemy sessions, Alembic, WAL, foreign
keys, structured logging, and readiness are configured.
- Stable asset IDs, path occurrences, versions, and audit timestamps have migrations.
- Secrets are referenced through configuration and never returned or logged.
- Startup and shutdown leave no leaked worker or database resources.
## Automated tests
- Migrations pass from an empty database and supported legacy schema snapshots.
- Process-level integration tests verify readiness, restart, WAL, foreign keys, and
clean shutdown.
## Dependencies
- US01-01

23
delivery_backlog/stories/US01-03-inventory-identity.md Normal file
View File

@@ -0,0 +1,23 @@
# US01-03 — Discover and Reconcile Stable Assets
Epic: [E01](../E01-shared-identity-inventory.md)
As a user, I want scans to discover eligible photos and retain identity across moves
so reorganizing folders does not lose prior work.
## Acceptance criteria
- Donor discovery and reconciliation logic is extracted behind `InventoryService`.
- All supported roots use one boundary and exclusion policy; unsafe symlink paths fail.
- New, moved, copied, replaced, missing, active, and archived occurrences are distinct.
- Enumeration is deterministic and rescanning unchanged input is idempotent.
## Automated tests
- Unit/property tests cover path normalization, boundaries, and reconciliation.
- Integration tests move, copy, replace, remove, and rescan fixture assets, asserting
stable IDs and exact durable states after restart.
## Dependencies
- US01-02

24
delivery_backlog/stories/US01-04-duplicate-engine.md Normal file
View File

@@ -0,0 +1,24 @@
# US01-04 — Detect and Decide Duplicates
Epic: [E01](../E01-shared-identity-inventory.md)
As a user, I want explainable duplicate clusters and reversible decisions so redundant
copies are excluded without collapsing distinct photos automatically.
## Acceptance criteria
- Donor exact, normalized-pixel, and perceptual hash behavior is extracted and
versioned.
- Confidence bands distinguish automatic exact matches from fuzzy review candidates.
- Canonical, variant, not-duplicate, and deferred decisions persist with evidence.
- New contradictory members reopen reviewed clusters; canonical links cannot cycle.
## Automated tests
- Golden corpus tests assert hash relationships and confidence classes.
- Property/integration tests cover clustering, negative links, reversals, new members,
and persistence after restart.
## Dependencies
- US01-03

24
delivery_backlog/stories/US01-05-thumbnail-service.md Normal file
View File

@@ -0,0 +1,24 @@
# US01-05 — Generate and Serve Managed Thumbnails
Epic: [E01](../E01-shared-identity-inventory.md)
As a reviewer, I want safe, correctly oriented previews so I can compare photos without
exposing arbitrary filesystem paths.
## Acceptance criteria
- Donor decode, orientation, resize, and HEIC behavior is reused where compatible.
- Thumbnails are requested by asset ID, cached atomically, quota-managed, and
invalidated when source identity changes.
- Corrupt, oversized, missing, and unsupported inputs produce bounded typed errors.
- Requests cannot escape configured roots or access excluded paths.
## Automated tests
- Golden tests assert orientation, dimensions, transparency handling, and cache keys.
- Integration tests cover concurrent generation, corruption, invalidation, eviction,
and path attacks.
## Dependencies
- US01-03

25
delivery_backlog/stories/US01-06-inventory-review-ui.md Normal file
View File

@@ -0,0 +1,25 @@
# US01-06 — Review Inventory and Duplicates in the Browser
Epic: [E01](../E01-shared-identity-inventory.md)
As a user, I want to browse inventory and compare duplicate candidates visually so I
can make informed canonical decisions.
## Acceptance criteria
- Inventory lists are paged, filterable, keyboard accessible, and use stable URLs.
- Duplicate comparison shows oriented previews, metadata evidence, confidence, and
synchronized comparison controls.
- Fuzzy decisions require explicit confirmation; stale versions receive a visible
conflict and never mutate state.
- Reloading restores the selected view and persisted decisions.
## Automated tests
- Component tests cover state, routing, and API error presentation.
- Playwright tests cover keyboard review, fuzzy confirmation, stale conflicts, and
persistence after reload.
## Dependencies
- US01-04, US01-05

24
delivery_backlog/stories/US01-07-phase-a-e2e.md Normal file
View File

@@ -0,0 +1,24 @@
# US01-07 — Automate Phase A End-to-End Acceptance
Epic: [E01](../E01-shared-identity-inventory.md)
As a delivery owner, I want reproducible Phase A end-to-end tests so identity,
inventory, duplicates, and thumbnails cannot regress unnoticed.
## Acceptance criteria
- The harness launches the real API and worker with a fresh database and deterministic
temporary fixture library.
- Journeys cover scan, exclusion, move reconciliation, exact/fuzzy duplicate review,
thumbnail orientation, canonical selection, reload, and full process restart.
- API and durable database state are asserted after restart.
- Story IDs US01-01 through US01-06 map to passing automated tests.
## Automated tests
- One documented command runs Phase A API and Playwright suites without network access.
- CI retains diagnostics on failure and rejects skipped, flaky, or unexercised fixtures.
## Dependencies
- US01-01 through US01-06

22
delivery_backlog/stories/US02-01-safety-ui-donors.md Normal file
View File

@@ -0,0 +1,22 @@
# US02-01 — Extract Safety and Existing Web UI Donors
Epic: [E02](../E02-unified-workflow-shell.md)
As a maintainer, I want safety logic and proven web interactions extracted from the
existing tools so the unified workflow retains working behavior and visual language.
## Acceptance criteria
- The donor ledger maps scoring, thresholds, decisions, EXIF keywords, filters,
layouts, tokens, keyboard controls, and error states to new modules.
- Behavior changes required by stable identity or shared jobs are documented.
- Shared modules have no dependency on archived entry points.
## Automated tests
- Characterization tests compare donor and extracted behavior on the same fixture IDs.
- Snapshot/interaction tests cover reused design tokens and critical controls.
## Dependencies
- E01 complete

22
delivery_backlog/stories/US02-02-durable-jobs.md Normal file
View File

@@ -0,0 +1,22 @@
# US02-02 — Persist and Coordinate Durable Jobs
Epic: [E02](../E02-unified-workflow-shell.md)
As an operator, I want jobs and item progress stored durably so work can resume after
browser, API, or worker interruption.
## Acceptance criteria
- Job, item, event, attempt, progress, cancellation, and terminal states are persisted.
- Commands support idempotency keys and atomic state transitions.
- Prerequisites and one-mutating-job policy produce explicit blockers.
- Progress derives from durable item state rather than browser memory.
## Automated tests
- State-machine/property tests reject invalid transitions and duplicate terminal state.
- Repository tests cover concurrent idempotent submission and restart persistence.
## Dependencies
- US01-02

22
delivery_backlog/stories/US02-03-worker-recovery.md Normal file
View File

@@ -0,0 +1,22 @@
# US02-03 — Execute Jobs with Leases, Locks, and Recovery
Epic: [E02](../E02-unified-workflow-shell.md)
As an operator, I want workers to claim, cancel, and recover jobs safely so crashes or
duplicate workers cannot corrupt state.
## Acceptance criteria
- Workers use leases, heartbeats, fencing tokens, bounded queues, and typed lanes.
- Cancellation is cooperative and leaves resumable or terminal item states.
- Lock ordering prevents conflicting analysis, EXIF, rename, upload, and archive work.
- A recovered job rejects late commits from its former worker.
## Automated tests
- Process tests kill workers, expire leases, recover jobs, and attempt stale commits.
- Randomized concurrency tests assert no deadlocks, duplicate completion, or lost events.
## Dependencies
- US02-02

23
delivery_backlog/stories/US02-04-job-api-events.md Normal file
View File

@@ -0,0 +1,23 @@
# US02-04 — Expose Job APIs and Activity Events
Epic: [E02](../E02-unified-workflow-shell.md)
As a frontend client, I want versioned job commands, status APIs, and activity events
so the browser can operate workflows without HTML responses or internal imports.
## Acceptance criteria
- `/api/v1` exposes typed job start, status, cancel, blocker, and event contracts.
- Operational endpoints return JSON; the event endpoint returns SSE with resumable IDs.
- SSE reconnects without duplicate/lost durable events and polling is sufficient as a
fallback.
- Errors use the shared JSON envelope and correct status codes, including `409`.
## Automated tests
- OpenAPI/contract tests validate schemas, status codes, errors, and content types.
- Black-box tests cover SSE disconnect/reconnect, polling, idempotency, and cancellation.
## Dependencies
- US02-02, US02-03

24
delivery_backlog/stories/US02-05-frontend-shell.md Normal file
View File

@@ -0,0 +1,24 @@
# US02-05 — Build the Static Application Shell
Epic: [E02](../E02-unified-workflow-shell.md)
As a user, I want a fast, navigable application shell so all workflow views share one
consistent interface without server-generated operational HTML.
## Acceptance criteria
- `index.html`, CSS, and modular JavaScript are separate static files.
- Shared API client, store, router, error handling, cancellation, and SSE/polling
adapters are implemented.
- Navigation routes restore current view and filters after reload.
- HTML fallback never intercepts `/api/v1` requests.
## Automated tests
- JavaScript tests cover API errors, store transitions, routing, and event fallback.
- Browser tests assert asset loading, deep links, reload, JSON-only APIs, and no
unexpected console/network errors.
## Dependencies
- US02-04

25
delivery_backlog/stories/US02-06-workflow-views.md Normal file
View File

@@ -0,0 +1,25 @@
# US02-06 — Deliver Workflow, Safety, Library, Analysis, and Stats Views
Epic: [E02](../E02-unified-workflow-shell.md)
As a user, I want one workflow home and migrated operational views so I can understand
readiness, review safety, browse assets, run analysis, and inspect results.
## Acceptance criteria
- Workflow cards show counts, blockers, last run, action, and details without relying
on color alone.
- Safety decisions persist and prevent NSFW assets from reaching vision analysis while
retaining verified upload eligibility.
- Library, Analyze, and Stats preserve useful donor features on shared APIs.
- Read-only browsing remains available during jobs; conflicting actions explain why
they are disabled.
## Automated tests
- View tests cover counts, filters, blockers, decisions, progress, errors, and reload.
- Integration tests assert provider-call privacy and verified safety EXIF behavior.
## Dependencies
- US02-01, US02-05

23
delivery_backlog/stories/US02-07-phase-b-e2e.md Normal file
View File

@@ -0,0 +1,23 @@
# US02-07 — Automate Phase B End-to-End Acceptance
Epic: [E02](../E02-unified-workflow-shell.md)
As a delivery owner, I want Phase B workflow automation so durable jobs and migrated
views are proven through real process and browser boundaries.
## Acceptance criteria
- Journeys cover shell loading, workflow blockers, safety review, analysis start,
progress, SSE reconnect, polling fallback, cancellation, resume, and error inspection.
- A second conflicting mutation is rejected while read-only browsing continues.
- NSFW fixture IDs never reach the fake vision provider and decisions survive restart.
- Story IDs US02-01 through US02-06 map to passing automated tests.
## Automated tests
- One documented command runs Phase B API, worker-recovery, and Playwright suites.
- The Phase A suite runs unchanged as a required regression gate.
## Dependencies
- US02-01 through US02-06

23
delivery_backlog/stories/US03-01-album-evidence.md Normal file
View File

@@ -0,0 +1,23 @@
# US03-01 — Aggregate Album Evidence
Epic: [E03](../E03-album-proposals.md)
As a user, I want album suggestions based on summarized photo evidence so names reflect
the folder without exposing unnecessary raw responses.
## Acceptance criteria
- The service aggregates canonical, eligible asset descriptions, tags, dates,
locations, people counts, and current folder identity.
- Missing, conflicting, deferred, duplicate, and NSFW evidence follows documented rules.
- Aggregation is versioned, deterministic, paged where needed, and invalidated by
relevant asset changes.
## Automated tests
- Unit/property tests cover aggregation, exclusions, conflicts, ordering, and versions.
- Repository tests verify results before and after relevant asset changes.
## Dependencies
- E02 complete

22
delivery_backlog/stories/US03-02-naming-policy.md Normal file
View File

@@ -0,0 +1,22 @@
# US03-02 — Define Album Naming Policy
Epic: [E03](../E03-album-proposals.md)
As a user, I want configurable naming templates and validation so proposed album names
are readable and safe on supported filesystems.
## Acceptance criteria
- Templates define allowed evidence, formatting, length, separators, and fallbacks.
- Validation handles forbidden characters, reserved names, whitespace, Unicode,
case-insensitive collisions, existing destinations, and empty results.
- Validation returns structured issues and suggestions without mutating files.
## Automated tests
- Property tests generate names across platforms and prove paths stay inside the root.
- Golden cases cover valid names, collisions, normalization, and template fallbacks.
## Dependencies
- US03-01

25
delivery_backlog/stories/US03-03-proposal-generation.md Normal file
View File

@@ -0,0 +1,25 @@
# US03-03 — Generate and Persist Versioned Proposals
Epic: [E03](../E03-album-proposals.md)
As a user, I want AI-assisted album proposals with rationale and confidence so I can
evaluate suggestions before approving them.
## Acceptance criteria
- A provider adapter receives only the minimum aggregated evidence and returns a
validated structured proposal.
- Proposal, evidence version, model/prompt version, rationale, confidence, edits, and
approval state are durable.
- Malformed, rate-limited, failed, duplicate, and stale responses are explicit and
safely retryable.
- Approval validates the latest proposal version and performs no rename.
## Automated tests
- Deterministic fake-provider tests cover success, malformed output, retry, and failure.
- API tests cover creation, edits, optimistic conflicts, approval, and restart.
## Dependencies
- US03-01, US03-02

24
delivery_backlog/stories/US03-04-proposal-ui.md Normal file
View File

@@ -0,0 +1,24 @@
# US03-04 — Review and Approve Proposals in the Browser
Epic: [E03](../E03-album-proposals.md)
As a user, I want to inspect evidence, edit a proposal, and approve it so I retain
control over album naming.
## Acceptance criteria
- The view shows source album, suggested name, evidence summary, rationale, confidence,
validation issues, and affected count.
- Edits validate promptly; collisions and invalid names are actionable.
- Approval requires explicit action and rejects stale state visibly.
- Reload preserves edits and approval state through the API.
## Automated tests
- UI tests cover editing, errors, collision guidance, approval, stale conflict, and
keyboard operation.
- Playwright asserts no file path changes occur during proposal approval.
## Dependencies
- US03-03

22
delivery_backlog/stories/US03-05-phase-c-e2e.md Normal file
View File

@@ -0,0 +1,22 @@
# US03-05 — Automate Phase C End-to-End Acceptance
Epic: [E03](../E03-album-proposals.md)
As a delivery owner, I want automated proposal journeys so generation, editing, and
approval remain deterministic and non-mutating.
## Acceptance criteria
- Journeys cover evidence aggregation, success, provider failure, invalid name,
collision, editing, stale approval, valid approval, reload, and restart.
- Tests assert provider inputs, persisted versions, and unchanged fixture paths.
- Story IDs US03-01 through US03-04 map to passing automated tests.
## Automated tests
- One command runs Phase C API and Playwright suites with the deterministic provider.
- Phase A–B end-to-end suites remain green.
## Dependencies
- US03-01 through US03-04

23
delivery_backlog/stories/US04-01-rename-plan.md Normal file
View File

@@ -0,0 +1,23 @@
# US04-01 — Build and Export Rename Plans
Epic: [E04](../E04-guarded-renaming.md)
As a user, I want a complete validated rename plan before mutation so I can understand
every source, destination, collision, and blocker.
## Acceptance criteria
- Approved album proposals produce itemized plans with asset IDs, source/destination,
expected hashes, versions, and required operations.
- Validation detects missing/changed sources, duplicate targets, case-only and Unicode
collisions, root escapes, symlinks, and cross-filesystem behavior.
- A portable JSON export contains no credentials and has a schema version/checksum.
## Automated tests
- Property tests validate collision-free plans and asset-set preservation.
- API/golden tests cover valid, invalid, stale, case-only, Unicode, and exported plans.
## Dependencies
- E03 complete

23
delivery_backlog/stories/US04-02-rename-journal.md Normal file
View File

@@ -0,0 +1,23 @@
# US04-02 — Journal Rename State and Preconditions
Epic: [E04](../E04-guarded-renaming.md)
As an operator, I want every rename transition journaled durably so interrupted work
can be diagnosed and recovered without guessing.
## Acceptance criteria
- Plan and item state machines record preconditions, intent, attempts, outcomes,
timestamps, fencing tokens, and verification evidence.
- Journal writes occur before filesystem mutations and terminal transitions are
idempotent.
- Startup classifies incomplete transitions as resumable, rollback-safe, or manual.
## Automated tests
- State-machine tests cover all valid/invalid transitions and repeated recovery.
- Migration/repository tests prove durable state after process restart.
## Dependencies
- US04-01, US02-03

24
delivery_backlog/stories/US04-03-rename-apply.md Normal file
View File

@@ -0,0 +1,24 @@
# US04-03 — Apply and Verify Guarded Renames
Epic: [E04](../E04-guarded-renaming.md)
As a user, I want confirmed plans applied without overwrites so reorganizing the
library preserves identity and bytes.
## Acceptance criteria
- Apply requires current plan/version confirmation and an exclusive mutation lease.
- Preconditions are rechecked immediately before each mutation; unexpected targets
are never overwritten.
- Case-only and cross-filesystem operations use safe staged procedures.
- Postconditions verify bytes and paths, update occurrences, and reconcile stable IDs.
## Automated tests
- Filesystem integration tests cover normal, case-only, collision, cross-filesystem,
stale source, and repeated apply.
- Tests assert byte preservation and database/path consistency after restart.
## Dependencies
- US04-02

23
delivery_backlog/stories/US04-04-rename-recovery.md Normal file
View File

@@ -0,0 +1,23 @@
# US04-04 — Recover or Roll Back Interrupted Renames
Epic: [E04](../E04-guarded-renaming.md)
As an operator, I want safe recovery choices after interruption so partially renamed
albums never trigger blind retries or data loss.
## Acceptance criteria
- Recovery derives action from journal plus current source/destination evidence.
- Resume and rollback revalidate recorded hashes and refuse changed paths.
- Ambiguous states block unrelated mutations and explain manual recovery precisely.
- Recovery and rollback are idempotent across repeated restarts.
## Automated tests
- Fault injection terminates the process at every persisted rename transition.
- Tests cover resume, rollback, changed paths, unexpected destinations, and repeated
restart with no overwrite or asset loss.
## Dependencies
- US04-03

22
delivery_backlog/stories/US04-05-rename-ui.md Normal file
View File

@@ -0,0 +1,22 @@
# US04-05 — Operate Rename Plans in the Browser
Epic: [E04](../E04-guarded-renaming.md)
As a user, I want to preview, confirm, monitor, and recover renames in the browser so
filesystem changes remain understandable and deliberate.
## Acceptance criteria
- Preview shows every affected path, validation issue, operation type, and count.
- Confirmation includes the current server-issued plan/version token.
- Progress, cancellation limits, terminal verification, and errors are visible.
- Recovery blocks unrelated mutations and offers only evidence-safe actions.
## Automated tests
- UI/Playwright tests cover preview, stale confirmation, apply, progress, collision,
interruption, recovery, rollback, focus, and reload.
## Dependencies
- US04-03, US04-04

22
delivery_backlog/stories/US04-06-phase-d-e2e.md Normal file
View File

@@ -0,0 +1,22 @@
# US04-06 — Automate Phase D End-to-End Acceptance
Epic: [E04](../E04-guarded-renaming.md)
As a delivery owner, I want exhaustive automated rename journeys so every crash point
and stale condition is proven safe.
## Acceptance criteria
- Journeys cover plan/export, confirmation, valid apply, case-only rename, collision,
stale source, cancellation boundary, every journal crash point, recovery, and rollback.
- Stable IDs, hashes, asset set, journal, and UI state are asserted after restart.
- Story IDs US04-01 through US04-05 map to passing automated tests.
## Automated tests
- One command runs API, filesystem fault-injection, and Playwright rename suites.
- Phases A–C remain green and the source fixture corpus remains unchanged.
## Dependencies
- US04-01 through US04-05

23
delivery_backlog/stories/US05-01-upload-preflight.md Normal file
View File

@@ -0,0 +1,23 @@
# US05-01 — Validate Credentials and Upload Readiness
Epic: [E05](../E05-immich-upload.md)
As a user, I want upload blockers and scope validated before execution so unverified or
changed files never reach Immich accidentally.
## Acceptance criteria
- Preflight checks credentials without exposing them, server reachability, selected
album scope, canonical/safety/analysis/EXIF readiness, and current byte hashes.
- The redacted command preview shows exact folders, album mapping, and asset counts.
- Partial scope and every blocker require explicit resolution or approved policy.
- Preflight produces a versioned token invalidated by relevant changes.
## Automated tests
- Integration tests cover valid, missing/invalid credential, blocked stage, changed
bytes, partial album, stale token, and secret redaction.
## Dependencies
- E04 complete

23
delivery_backlog/stories/US05-02-upload-batches.md Normal file
View File

@@ -0,0 +1,23 @@
# US05-02 — Orchestrate Album Upload Batches
Epic: [E05](../E05-immich-upload.md)
As a user, I want one approved album uploaded at a time so scope, progress, and retries
remain controlled.
## Acceptance criteria
- A durable upload batch records album, asset IDs, pre-upload hashes, command version,
attempts, progress, cancellation, and raw report location.
- `immich-go` is invoked through the integration adapter with bounded output and no
shell interpolation or secret logging.
- Only one configured uploader lane runs; restart resumes at safe item/batch boundaries.
## Automated tests
- Fake-executable integration tests verify arguments, album isolation, concurrency,
cancellation, restart, output limits, and credential privacy.
## Dependencies
- US05-01, US02-03

22
delivery_backlog/stories/US05-03-upload-reports.md Normal file
View File

@@ -0,0 +1,22 @@
# US05-03 — Parse and Persist Uploader Outcomes
Epic: [E05](../E05-immich-upload.md)
As an operator, I want uploader reports converted into durable typed outcomes so I can
distinguish new, upgraded, duplicate, failed, and unknown assets.
## Acceptance criteria
- Version-specific parsers preserve bounded raw evidence and classify every item.
- Unknown versions or lines never become success; they require verification.
- Uploaded SHA-1/content evidence, timestamps, counts, and parser version are stored.
- Reprocessing the same report is idempotent.
## Automated tests
- Golden parser fixtures cover every supported outcome/version and malformed output.
- Repository tests cover idempotent import, count reconciliation, and restart.
## Dependencies
- US05-02

22
delivery_backlog/stories/US05-04-upload-verification.md Normal file
View File

@@ -0,0 +1,22 @@
# US05-04 — Verify, Retry, and Resolve Uncertain Uploads
Epic: [E05](../E05-immich-upload.md)
As a user, I want uncertain uploads verified before retry so a lost response cannot
create duplicate server assets.
## Acceptance criteria
- Retry policy distinguishes safe failures from accepted-but-unknown outcomes.
- Verification compares recorded local bytes with authoritative available evidence.
- Changed-after-upload bytes create a visible stale warning and block unsafe actions.
- Manual resolution records evidence and audit history; it never silently assumes success.
## Automated tests
- Fault tests cover timeout before acceptance, acceptance then lost response, parser
uncertainty, safe retry, changed bytes, and repeated verification.
## Dependencies
- US05-03

23
delivery_backlog/stories/US05-05-upload-ui.md Normal file
View File

@@ -0,0 +1,23 @@
# US05-05 — Operate Uploads in the Browser
Epic: [E05](../E05-immich-upload.md)
As a user, I want to preflight, confirm, monitor, and verify uploads so I know exactly
what reached Immich and what remains uncertain.
## Acceptance criteria
- The view shows album scope, blockers, redacted configuration, counts, and exact
confirmation before start.
- Progress distinguishes new, upgraded, duplicate, failed, cancelled, and uncertain.
- Uncertain outcomes expose verification actions, not an automatic retry button.
- Secrets never appear in DOM, logs, URLs, traces, or browser storage.
## Automated tests
- Playwright covers blocked/valid preflight, start, progress, cancellation, report,
uncertainty, verification, stale bytes, reload, and secret scanning.
## Dependencies
- US05-04

23
delivery_backlog/stories/US05-06-phase-e-e2e.md Normal file
View File

@@ -0,0 +1,23 @@
# US05-06 — Automate Phase E End-to-End Acceptance
Epic: [E05](../E05-immich-upload.md)
As a delivery owner, I want deterministic upload journeys so metadata ordering,
deduplication outcomes, and uncertainty handling cannot regress.
## Acceptance criteria
- Journeys cover credential failure, preflight blockers, new, exact duplicate, upgrade,
retryable failure, acceptance-response loss, verification, cancellation, and resume.
- Tests prove EXIF precedes upload and persisted hashes match submitted bytes.
- No secret appears in retained artifacts; state is asserted after restart.
- Story IDs US05-01 through US05-05 map to passing automated tests.
## Automated tests
- One command runs fake uploader, black-box API, and Playwright upload suites.
- Phases A–D remain green.
## Dependencies
- US05-01 through US05-05

23
delivery_backlog/stories/US06-01-archive-preflight.md Normal file
View File

@@ -0,0 +1,23 @@
# US06-01 — Configure and Preflight Archive Destinations
Epic: [E06](../E06-archive-lifecycle.md)
As a user, I want archive destinations identified and validated so an album is never
moved to the wrong, unavailable, or undersized storage.
## Acceptance criteria
- Archive locations record stable media/volume identity, root, capabilities, and state.
- Preflight validates verified upload, current bytes, destination identity, capacity,
writability, boundaries, collisions, locks, backup, and manifest creation.
- Preview shows transfer method, exact scope, reclaimable bytes, and every blocker.
- Confirmation uses a versioned token invalidated by source/destination changes.
## Automated tests
- Integration tests cover valid, offline, wrong-volume, low-space, read-only, collision,
stale source, lock conflict, and unsafe path cases.
## Dependencies
- E05 complete

23
delivery_backlog/stories/US06-02-archive-transfer.md Normal file
View File

@@ -0,0 +1,23 @@
# US06-02 — Transfer, Verify, and Remove Active Sources
Epic: [E06](../E06-archive-lifecycle.md)
As a user, I want source files removed only after durable archive verification so
reclaiming space cannot cause data loss.
## Acceptance criteria
- A journal records planned, transferring, verified, removing, and terminal item state.
- Cross-filesystem flow copies to a temporary destination, closes, hashes, atomically
publishes, records manifest, then removes the matching source.
- Same-filesystem optimization is used only when proven safe and still verified.
- Unexpected source/destination changes stop the item without overwrite or deletion.
## Automated tests
- Filesystem/process tests interrupt every transition before and after source removal.
- Tests assert manifest/hash correctness, idempotent recovery, and zero verified loss.
## Dependencies
- US06-01, US02-03

22
delivery_backlog/stories/US06-03-offline-assets.md Normal file
View File

@@ -0,0 +1,22 @@
# US06-03 — Preserve Offline Identity and Review Evidence
Epic: [E06](../E06-archive-lifecycle.md)
As a user, I want archived assets searchable and usable for deduplication while their
media is offline so new copies are not mistaken for unrelated photos.
## Acceptance criteria
- Availability distinguishes active, archived online/offline, and unexpectedly missing.
- Archived hashes remain in indexes and inventory scans do not prune offline assets.
- Protected comparison previews and evidence remain available under a durable policy.
- Exact and fuzzy active matches link to archived canonicals with appropriate review.
## Automated tests
- Integration tests toggle fake mounts, rescan, discover exact/fuzzy copies, review
previews, and assert stable state across restart.
## Dependencies
- US06-02

24
delivery_backlog/stories/US06-04-restore.md Normal file
View File

@@ -0,0 +1,24 @@
# US06-04 — Plan and Execute Safe Restores
Epic: [E06](../E06-archive-lifecycle.md)
As a user, I want archived assets restored to collision-free active paths so I can
recover originals without losing identity or previous decisions.
## Acceptance criteria
- Restore preflight requires the correct archive identity, matching bytes, capacity,
safe destination, and no conflicting lease.
- Restore copies, closes, verifies, atomically publishes, and registers an active
occurrence while preserving asset identity and stage history.
- Collisions never overwrite; mismatches become stale/divergent states.
- Restore is journaled, resumable, and idempotent.
## Automated tests
- Tests cover offline media, wrong volume, normal restore, collision, changed archive
bytes, interruption, repeated restart, and preserved decisions.
## Dependencies
- US06-03

23
delivery_backlog/stories/US06-05-archive-ui.md Normal file
View File

@@ -0,0 +1,23 @@
# US06-05 — Operate Archive and Restore in the Browser
Epic: [E06](../E06-archive-lifecycle.md)
As a user, I want to preview, confirm, monitor, recover, and restore archives so storage
reclamation remains understandable and reversible.
## Acceptance criteria
- Archive preview shows exact scope, destination identity, transfer method, capacity,
blockers, and reclaimable bytes.
- Progress separates transfer, verification, and source-removal states.
- Offline assets remain browsable with clear availability and mount instructions.
- Recovery and restore views expose only safe evidence-backed actions.
## Automated tests
- Playwright covers preflight blockers, confirmation, progress, interruption/recovery,
offline browsing, restore, collision, keyboard operation, and reload.
## Dependencies
- US06-02, US06-03, US06-04

23
delivery_backlog/stories/US06-06-phase-f-e2e.md Normal file
View File

@@ -0,0 +1,23 @@
# US06-06 — Automate Phase F End-to-End Acceptance
Epic: [E06](../E06-archive-lifecycle.md)
As a delivery owner, I want archive lifecycle automation so removal, offline behavior,
recovery, and restore are proven without risking real files.
## Acceptance criteria
- Journeys cover preflight blockers, copy/verify/remove, same-filesystem behavior,
every interruption point, offline deduplication, mount return, restore, and collision.
- API, journal, manifest, paths, hashes, previews, and durable state are asserted after
restart; no source is removed before archive verification.
- Story IDs US06-01 through US06-05 map to passing automated tests.
## Automated tests
- One command runs archive fault-injection, black-box API, and Playwright suites.
- Phases A–E remain green.
## Dependencies
- US06-01 through US06-05

24
delivery_backlog/stories/US07-01-donor-archive.md Normal file
View File

@@ -0,0 +1,24 @@
# US07-01 — Complete Donor Migration and Freeze the CLI Archive
Epic: [E07](../E07-hardening-release.md)
As a maintainer, I want all useful legacy behavior resolved and original CLIs frozen
with provenance so production has one implementation without losing historical evidence.
## Acceptance criteria
- Every in-scope donor-ledger row has a target, characterization test, parity result,
and documented intentional delta where applicable.
- Remaining CSV state is migrated and reconciled into SQLite with a report.
- Original sources, docs, dependency lock, schema/config notes, version, and checksums
are placed in the read-only legacy archive with secrets removed.
- Production imports and runtime paths cannot load or execute archived scripts.
## Automated tests
- Ledger/archive lint verifies completeness, checksums, redaction, and non-importability.
- Full parity and database migration/reconciliation suites pass.
## Dependencies
- E01–E06 complete

23
delivery_backlog/stories/US07-02-security-hardening.md Normal file
View File

@@ -0,0 +1,23 @@
# US07-02 — Harden API Authorization and Path Boundaries
Epic: [E07](../E07-hardening-release.md)
As an operator, I want the local application resistant to cross-origin and path attacks
so another process or webpage cannot access photos or trigger mutations.
## Acceptance criteria
- Session authentication, Origin/Host checks, SameSite cookies, CSRF protection, and
restrictive CORS/default headers cover all mutation and media endpoints.
- Asset IDs, repeated root validation, symlink defense, upload limits, and schema
validation prevent raw-path and race escapes.
- Errors reveal no secrets, filesystem internals, or private metadata unnecessarily.
## Automated tests
- Black-box security tests cover missing/invalid auth, CSRF, hostile origins/hosts,
traversal, symlink races, malformed/oversized requests, and information leakage.
## Dependencies
- US07-01

24
delivery_backlog/stories/US07-03-media-hardening.md Normal file
View File

@@ -0,0 +1,24 @@
# US07-03 — Harden Media and Metadata Edge Cases
Epic: [E07](../E07-hardening-release.md)
As a user, I want unusual or damaged media handled safely so one file cannot corrupt
metadata, exhaust resources, or stop the library workflow.
## Acceptance criteria
- Supported format/orientation/profile combinations have bounded decode behavior.
- Corrupt, truncated, huge-dimension, unsupported, and malformed-metadata files become
precise item errors without worker failure.
- EXIF checkpoints preserve user fields and previous-stage fields, read back owned
fields, and mark conflicts divergent rather than silently repairing them.
- Cache regeneration/invalidation and temporary-file cleanup are safe and scoped.
## Automated tests
- Extended golden corpus covers all declared format, orientation, corruption, and EXIF
cases with memory/time bounds and before/after metadata snapshots.
## Dependencies
- US07-01

25
delivery_backlog/stories/US07-04-fault-concurrency.md Normal file
View File

@@ -0,0 +1,25 @@
# US07-04 — Prove Concurrency and Crash Recovery
Epic: [E07](../E07-hardening-release.md)
As an operator, I want randomized race and fault tests so concurrency cannot produce
deadlocks, stale commits, duplicate terminal states, or data loss.
## Acceptance criteria
- Test control points exist at persisted transitions without exposing production
mutation APIs.
- Tests cover database pressure, worker claim races, file changes, thumbnail races,
safety/analysis races, EXIF/upload races, rename/archive conflicts, and cancellation.
- Faults cover process death, disk full, read-only paths, DB busy/corruption, network
failures, malformed providers, GPU exhaustion, subprocess hangs, and missing tools.
- Recovery is deterministic or enters an explicit manual-recovery state.
## Automated tests
- Randomized suites repeat with recorded seeds and assert invariants after restart.
- CI retains journals, logs, DB, seed, and filesystem manifest for any failure.
## Dependencies
- US07-02, US07-03

24
delivery_backlog/stories/US07-05-backup-operations.md Normal file
View File

@@ -0,0 +1,24 @@
# US07-05 — Deliver Backup and Operational Recovery
Epic: [E07](../E07-hardening-release.md)
As an operator, I want documented and tested backup/recovery procedures so application
state can be restored after migration failure, database damage, or storage loss.
## Acceptance criteria
- Online backup includes database consistency, required manifests, configuration
references, and retention guidance without copying secrets into logs.
- Integrity checks, migration rollback/recovery, archive media handling, and restore
drills have step-by-step documentation.
- Operational diagnostics report DB/WAL/cache/log/thumbnail sizes and low-disk risks.
- Legacy-process locking prevents incompatible simultaneous mutations.
## Automated tests
- Recovery drills restore backups into fresh roots and run integrity/reconciliation.
- Tests cover failed migration, corrupt copy detection, retention, and process locking.
## Dependencies
- US07-01, US07-04

24
delivery_backlog/stories/US07-06-performance.md Normal file
View File

@@ -0,0 +1,24 @@
# US07-06 — Validate Performance and Resource Bounds
Epic: [E07](../E07-hardening-release.md)
As an operator, I want measured performance and bounded resource use so large libraries
remain responsive and do not exhaust memory, disk, handles, or queues.
## Acceptance criteria
- Agreed budgets cover API latency, throughput, RSS, open files, WAL, queues, cache,
disk reserve, and event delivery at 25k, 100k, and 500k synthetic assets.
- Duplicate clusters with thousands of members remain usable and paged.
- Multi-hour work plus browsing, cancellation, retry, backup, and cache eviction shows
no unbounded growth or starvation.
- Budget exceptions are measured, documented, and approved before release.
## Automated tests
- Repeatable load/soak commands export machine-readable metrics and fail budget breaches.
- CI runs a short profile; scheduled infrastructure runs the full soak matrix.
## Dependencies
- US07-04, US07-05

29
delivery_backlog/stories/US07-07-release-e2e.md Normal file
View File

@@ -0,0 +1,29 @@
# US07-07 — Automate Full Release Acceptance
Epic: [E07](../E07-hardening-release.md)
As a release owner, I want one reproducible automated release gate so every user story
and complete workflow is proven before real-library mutation is enabled.
## Acceptance criteria
- A story-to-test matrix contains every backlog story with no missing, skipped, or
failing required test.
- One fresh-environment journey runs discovery, duplicate review, safety, analysis,
EXIF verification, album proposal, rename, rescan, upload, archive, offline
deduplication, restore, and full process restarts.
- Security, migration, donor parity, crash recovery, accessibility, responsive UI,
browser console/network, fixture reproducibility, and archive safety gates pass.
- A read-only real-library dry run produces an explicitly approved reconciliation
report before mutation can be configured.
## Automated tests
- One documented release command provisions and destroys the full isolated stack and
retains signed/versioned evidence.
- Chromium passes every change; supported cross-browser and full soak suites pass for
release; all earlier epic suites run unchanged.
## Dependencies
- US07-01 through US07-06

11
nsfw_tag.py Normal file
View File

@@ -0,0 +1,11 @@
#!/usr/bin/env python3
"""Backwards-compatible shim — the code now lives in the nsfwtag/ package.
Run either of these (identical behaviour):
python nsfw_tag.py "<folder>" -r
python -m nsfwtag "<folder>" -r
"""
from nsfwtag.__main__ import main
if __name__ == "__main__":
main()

282
nsfwtag/README.md Normal file
View File

@@ -0,0 +1,282 @@
# nsfwtag
Local, on-device NSFW image review and tagging for large photo libraries — built to feed [Immich](https://immich.app) search and to complement the `photo_analyzer.py` captioning pipeline.
It scores a photo library with a **local** vision model (Apple GPU via MPS), opens a fast **browser review app**, and writes an `nsfw` (or `sfw`) keyword into each photo's EXIF for the ones you confirm — **preserving every other tag**. Nothing ever leaves the machine: no API calls, no uploads, no telemetry.
```
python -m nsfwtag "pictures/" -r
```
---
## Table of contents
- [The idea](#the-idea)
- [Features](#features)
- [How it works](#how-it-works)
- [Model choice](#model-choice)
- [Requirements & install](#requirements--install)
- [CLI usage](#cli-usage)
- [The review app](#the-review-app)
- [EXIF / review-state model](#exif--review-state-model)
- [HTTP API](#http-api)
- [Data & caching](#data--caching)
- [Project layout](#project-layout)
- [Design decisions](#design-decisions)
- [Immich integration](#immich-integration)
---
## The idea
A large personal photo library has a handful of sensitive images scattered through
thousands of holiday snaps. To filter them in [Immich](https://immich.app) you need a
searchable marker on each one — but going through every photo by hand is infeasible, and
sending a private library to a cloud moderation API is a non-starter.
**The model does recall; you do precision.** A local classifier is good at narrowing
thousands of photos down to a few dozen candidates, but it is *not* trustworthy as the
final word — it over-flags swimwear, art, medical shots. So this is a **human-in-the-loop**
tool: the model proposes, you confirm in a fast visual review app, and only your confirmed
decisions are written to disk. Nothing ever leaves the machine — no API, no upload, no
telemetry.
The decision lives **in the file's own EXIF** as an `nsfw`/`sfw` keyword, not in a
side database, so it survives across tools, backups and restarts, and Immich indexes it
for free at ingest.
---
## Features
- **On-device scoring** with `AdamCodd/vit-base-nsfw-detector` (ViT, ~350 MB, downloaded once then fully offline). Chosen over Falconsai after benchmarking against the library — better recall, fewer false positives (see `bench.py`).
- **Resume-safe**: scores are cached to a CSV; a rerun reuses them and only scores newly-added files — skipping the model load entirely when nothing is new.
- **Browser review app** (dark OLED UI, zero web dependencies — served by a stdlib HTTP server):
- Per-image **checkmark** to mark NSFW / SFW; frame colors show state at a glance.
- **Folder browser** sidebar, live search, a single **Show** filter (All / To-handle / NSFW / SFW), sort, NSFW-score threshold, and a contextual action group that appears only when images are selected.
- **Lightbox** with prev/next, keyboard control, and an **EXIF panel** (incl. a clickable **Google Maps** link for geotagged photos).
- **Activity log** tab with a per-file audit trail of every EXIF write.
- Lazy, per-folder tag scan with a **progress ring**.
- **Reversible & non-destructive**: tag, untag, or toggle at any time; only `Keywords`/`Subject` are modified, so Caption, dates, GPS, camera info, etc. are always preserved.
- **Persistent decisions**: marking an image safe writes an `sfw` keyword, so your review survives restarts.
---
## How it works
```
discover images score (cached) review in browser write EXIF
folder ─────────────────▶ [*.jpg …] ───────────────▶ nsfw_scores.csv ─────────────────▶ exiftool: Keywords/Subject += nsfw|sfw
(recursive, de-duped) supported local ViT on MPS (path,score) local HTTP server + review.html (all other tags preserved)
```
1. **Discover** — walk the folder (optionally recursive), keep supported image types, de-duplicate by resolved path.
2. **Score** — load cached scores; run the model only on files not in the cache; each image is resized to 224×224 and scored 0..1 for the `nsfw` class. Merge and persist.
3. **Review** — candidates (score ≥ `--review-min`) are served as a single-page app. A lazy per-folder scan reads existing `nsfw`/`sfw` keywords and colors each card.
4. **Tag** — your actions POST to the local server, which calls `exiftool` to add/remove the keyword. EXIF is written **before** the file is later uploaded to Immich.
---
## Model choice
The tool ships with **`AdamCodd/vit-base-nsfw-detector`** — a ~350 MB ViT binary
classifier that runs offline on Apple MPS. It was picked by benchmarking candidates
against **your own library**, using the photos you'd already hand-tagged `nsfw`/`sfw`
as ground truth (`bench.py` reads those EXIF marks). A generic public benchmark wouldn't
predict how a model behaves on *these* photos, so the comparison is run on the real data.
Candidates compared (`nsfwtag/bench.py`):
| Model | Notes |
|-------|-------|
| **AdamCodd/vit-base-nsfw-detector** ✅ | Chosen — best recall with the fewest false positives on this library |
| Falconsai/nsfw_image_detection | Previous default; missed more true positives and needed a high `0.85` cutoff |
| Marqo/nsfw-image-detection-384 | Tiny/fast, but noisier on borderline shots |
| OwenElliott/image-safety-classifier-xs | 3-class (SFW/NSFW/NSFL); useful signal, weaker overall here |
**Selection criterion — recall first.** Because a human confirms every candidate, a false
*positive* costs one extra glance, but a false *negative* means a sensitive photo silently
slips through untagged. So the model is judged on catching everything at a threshold low
enough to be safe without burying you in noise. AdamCodd's sweet spot is **`0.6`** (vs
Falconsai's `0.85`) — that is the shipped default.
**Reproduce it** on any sample folder:
```bash
python -m nsfwtag.bench "pictures/SomeFolder" -r
# writes bench_scores.csv + bench_report.html (thumbnails, per-model scores,
# disagreements highlighted) and prints precision/recall vs your manual tags.
```
To switch models, change `MODEL_ID` in `nsfwtag/__init__.py` (any HF image-classification
id) and re-tune `DEFAULT_THRESHOLD`.
---
## Requirements & install
- **Python 3.10+**
- **exiftool** on `PATH` — `brew install exiftool` (macOS) / `apt install libimage-exiftool-perl` (Debian/Ubuntu)
- Python packages: `pip install torch transformers pillow numpy`
- A GPU is optional — uses Apple **MPS** if available, else CPU.
The model downloads automatically on first run (~350 MB) and is cached in `~/.cache/huggingface`; every run after that is offline (`local_files_only=True`, which also skips the slow HF Hub update check).
---
## CLI usage
Run from the directory that contains the `nsfwtag/` package:
```bash
# Score + open the review app
python -m nsfwtag "pictures/" # this folder only
python -m nsfwtag "pictures/" -r # recurse into subfolders
python nsfw_tag.py "pictures/" -r # identical (backwards-compat shim)
# Tag directly by score, no review UI
python -m nsfwtag "pictures/" -r --apply
# Tag every path listed in a text file (one path per line)
python -m nsfwtag confirmed.txt [more.txt ...]
```
| Flag | Default | Description |
|------|---------|-------------|
| `-r, --recursive` | off | Descend into every subfolder |
| `--threshold` | `0.6` | Score ≥ this is treated as "NSFW to review" (and tagged directly in `--apply`) |
| `--review-min` | `0.20` | Lowest score shown in the review app |
| `--limit N` | `0` | Only the first N images (calibration) |
| `--apply` | off | Tag score ≥ threshold directly, skip the review UI |
| `--rescore` | off | Ignore the score cache and re-run the model on everything |
The argument is auto-detected: a **folder** opens the review app; one or more **`.txt`** files tag the listed paths directly.
---
## The review app
Served on `http://127.0.0.1:<random-free-port>/` and opened in your browser automatically. Press `Ctrl-C` in the terminal when done.
There are **two independent channels** per card: the **decision** (the EXIF tag,
shown as the frame + tick color) and the **selection** (a blue ring, for bulk
actions). A card can be both selected *and* decided.
### Frame colors (the decision)
| Color | Meaning | EXIF |
|-------|---------|------|
| 🩷 **pink** | tagged NSFW | `nsfw` keyword |
| 🟢 **green** | tagged safe | `sfw` keyword |
| 🔴 **red** | undecided **and** selected — a bulk candidate, nothing written yet | none |
| ⚪ neutral | undecided, not selected | none |
| 🔵 blue ring | selected for a bulk action (independent of the frame color) | — |
Each card has a **tick** (bottom-right circle) that makes a per-image decision and
**writes EXIF on every click**, cycling: undecided → **NSFW** (pink) → **SFW**
(green) → undecided. One click is enough — it no longer depends on the selection
state. Clicking elsewhere on a card does nothing (no accidental toggles).
### Controls
The toolbar separates **viewing** (always visible) from **acting** (contextual). It reads left-to-right:
```
🔍 Search… Show ▾ Sort ▾ NSFW ≥ 0.6 ●───── Select flagged Select all ⟨ N selected · Deselect · Remove · Mark safe · Mark NSFW ⟩
```
- **Folder browser** (left sidebar, toggleable): filter to a folder + its subfolders; counts per folder.
- **Search**: filter by filename or folder.
- **Show** (single filter): **All** / **To-handle** (no keyword yet — what's left to decide) / **NSFW** (tagged nsfw) / **SFW** (tagged sfw).
- **Sort**: score / name / folder.
- **NSFW ≥ slider**: score threshold used by *Select flagged* and the red "identified" coloring.
- **Select flagged**: selects visible, untagged images at/above the threshold.
- **Select all**: selects every currently-visible image (respects the folder/filter). Pair with **Deselect** to clear. Use this to build a selection after a filter where *Select flagged* matches nothing.
- **Per-image tick**: one click writes EXIF and cycles undecided → NSFW → SFW → undecided; the tick and frame colors show the current decision.
- **Action group** (appears only when ≥1 image is selected): **Mark NSFW** / **Mark safe** / **Remove** (clears either keyword) / **Deselect**. Counts and actions are scoped to the current view (folder/filter), so the count matches what you see.
- **Lightbox** (⤢ on a thumbnail): full-res image, EXIF side panel (incl. Google Maps link), `←`/`→` navigate, `Space` toggle selection, `Esc`/backdrop close.
- **Activity log** (header button): timestamped, per-file record of every EXIF write.
---
## EXIF / review-state model
The review decision is stored **in the file's EXIF** as one of two mutually-exclusive keywords, written to both `Keywords` (IPTC) and `Subject` (XMP):
| Keyword | Meaning |
|---------|---------|
| `nsfw` | confirmed NSFW |
| `sfw` | confirmed safe |
| *(neither)* | not yet reviewed |
Rules:
- Writing `nsfw` removes any `sfw`, and vice-versa — a file can hold **at most one**.
- On read, **`nsfw` takes precedence**: a stray file with both is reported as NSFW, never as safe.
- **Only `Keywords` and `Subject` are ever modified.** All other metadata — `ImageDescription`/Caption, GPS, capture dates, camera make/model, ratings — is left byte-for-byte intact in both directions (add and remove).
- `exiftool` runs with `-overwrite_original` (no sidecar `_original` files) and `-m` (ignore minor warnings so writes still apply to slightly-nonconforming files).
Immich indexes `Keywords`, so tagged photos become searchable there (search `nsfw`).
---
## HTTP API
Stdlib `ThreadingHTTPServer` bound to `127.0.0.1` on an OS-assigned free port. **Every path argument is validated against the scanned candidate set** — the server can't be pointed at arbitrary files.
| Method | Route | Purpose |
|--------|-------|---------|
| GET | `/` | The review page |
| GET | `/img?path=` | Thumbnail bytes for a candidate image |
| GET | `/exif?path=` | Filtered EXIF for the lightbox panel (decimal GPS + Google Maps URL) |
| GET | `/log` | Activity-log entries `[{t, m}, …]` |
| POST | `/tagged` | Body: `[path,…]` → `{nsfw:[…], sfw:[…]}` (batched keyword read) |
| POST | `/apply` | Body: `[path,…]` → add `nsfw` (strips `sfw`) → `{tagged:n}` — "Mark NSFW" |
| POST | `/mark?state=sfw` | Body: `[path,…]` → add `sfw` (strips `nsfw`) — "Mark safe" / checkmark→green |
| POST | `/untag` | Body: `[path,…]` → remove both keywords (un-review) → `{untagged:n}` — "Remove" |
| POST | `/mark?state=clear` | Body: `[path,…]` → remove `sfw` only (checkmark→red) |
| POST | `/toggle` | Body: `[path,…]` → flip `nsfw` per file → `{tagged:[…]}` (endpoint retained; not used by the current UI) |
---
## Data & caching
- **`nsfw_scores.csv`** (written in the scanned folder): `path,nsfw_score`. This is the resume cache — reruns reuse it and only score new files. Delete it or pass `--rescore` to force a full re-score.
- **Review state** is **not** in the CSV; it lives in each file's EXIF (see above), so it persists independently of the score cache and survives restarts.
- The activity log is in-memory (last 2000 events) and resets when the server stops; messages are also echoed to stderr.
---
## Project layout
```
nsfwtag/
├── __init__.py # constants (model id, extensions, defaults), warning suppression
├── exif.py # read/write/remove nsfw|sfw via exiftool; read_marks, read_exif (+GPS)
├── scoring.py # discover images, cached model scoring
├── webapp.py # render the page: fill review.html tokens (cards, folder tree, totals)
├── review.html # the frontend — HTML/CSS/JS (no build step, no dependencies)
├── server.py # local HTTP server + routes + activity log
├── __main__.py # CLI entry point (python -m nsfwtag)
└── README.md
nsfw_tag.py # thin shim: `python nsfw_tag.py` == `python -m nsfwtag`
```
The frontend lives in a real `review.html` (dynamic parts injected via `%%TOKEN%%` replacement) rather than a Python string, so it edits with normal HTML/CSS/JS tooling.
---
## Design decisions
- **No pre-selection at render.** Cards render unselected; the **lazy per-folder scan** decides each card's state from its EXIF keywords (pink/green) or score (red). This is deliberate: it guarantees a photo you marked safe never reappears pre-selected-red after a restart, and keeps startup instant (the browser opens before any `exiftool` scan runs).
- **Lazy, chunked, per-folder scanning** with a progress ring, instead of one blocking scan at startup — so switching into a folder shows real progress and huge libraries stay responsive.
- **Score cache first, model second** — the model (and its load) is skipped entirely on reruns with no new files.
- **View-scoped counters/actions** — Apply/Toggle/Remove and their counts act on the folder/filter you're viewing, so the number always matches what's on screen.
- **Stdlib-only server + dependency-free frontend** — nothing to build or install for the UI; the whole app is one HTML file plus a ~120-line Python server.
---
## Immich integration
Order matters: **write EXIF first, then upload.** immich-go deduplicates on the file's SHA-1 content hash, so writing EXIF after upload would create duplicates. This tool only writes metadata in place; ingest into Immich is a separate, deliberate `immich-go upload` step. Once uploaded, `Keywords` (including `nsfw`) are searchable in Immich alongside its own ML pipeline. **Do not** also point Immich at the folder as an External Library.

25
nsfwtag/__init__.py Normal file
View File

@@ -0,0 +1,25 @@
"""nsfwtag — local, on-device NSFW image tagging for Immich.
Score a photo library with a local ViT model (Apple GPU via MPS), review the
flagged images in a browser, and write an 'nsfw' EXIF keyword into the ones you
confirm — preserving every other tag. Nothing leaves the machine.
Entry point: `python -m nsfwtag <folder> [-r]` (or the nsfw_tag.py shim).
"""
import os
import warnings
os.environ.setdefault("PYTHONWARNINGS", "ignore") # quiet numpy-2/-1 ABI noise from optional libs
warnings.filterwarnings("ignore")
# AdamCodd/vit-base-nsfw-detector — chosen after benchmarking against the user's own
# library (much better recall + fewer false positives than Falconsai). See bench.py.
MODEL_ID = "AdamCodd/vit-base-nsfw-detector"
EXTS = {".jpg", ".jpeg", ".png", ".heic", ".heif", ".webp"}
CSV_NAME = "nsfw_scores.csv"
DEFAULT_THRESHOLD = 0.6 # this model's sweet spot; Falconsai used 0.85
DEFAULT_REVIEW_MIN = 0.2
BATCH = 16
KEYWORD = "nsfw"
__version__ = "1.1.0"

65
nsfwtag/__main__.py Normal file
View File

@@ -0,0 +1,65 @@
"""CLI entry point.
python -m nsfwtag "<folder>" # score + open the review app
python -m nsfwtag "<folder>" -r # recurse into subfolders
python -m nsfwtag list.txt [list2.txt] # tag every path in the .txt list(s)
The argument is auto-detected: a folder opens the review app; .txt file(s) tag
directly. A tiny stdlib HTTP server serves the page and tags on Apply — no data
leaves the machine.
"""
import argparse
import sys
from pathlib import Path
from . import CSV_NAME, DEFAULT_REVIEW_MIN, DEFAULT_THRESHOLD
from .exif import apply_list, write_keyword
from .scoring import discover_images, score_images
from .server import serve_review
def main():
ap = argparse.ArgumentParser(prog="nsfwtag", description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("target", nargs="+", help="a FOLDER to scan (opens the review app) OR one/more .txt lists to tag")
ap.add_argument("-r", "--recursive", action="store_true", help="descend into every subfolder")
ap.add_argument("--apply", action="store_true", help="folder mode: tag score>=threshold directly, skip review")
ap.add_argument("--threshold", type=float, default=DEFAULT_THRESHOLD, help=f"score >= this is pre-selected / tagged (default {DEFAULT_THRESHOLD})")
ap.add_argument("--review-min", type=float, default=DEFAULT_REVIEW_MIN, help=f"lowest score shown in the review app (default {DEFAULT_REVIEW_MIN})")
ap.add_argument("--limit", type=int, default=0, help="only the first N images (calibration)")
ap.add_argument("--rescore", action="store_true", help="ignore cached scores and re-run the model on everything")
args = ap.parse_args()
# auto-detected: .txt list(s) = apply mode; a single folder = review mode.
targets = [Path(t).expanduser() for t in args.target]
if targets and all(t.is_file() for t in targets):
for t in targets:
apply_list(str(t))
return
if len(targets) != 1 or not targets[0].is_dir():
sys.exit("Pass ONE folder (review), or one or more .txt lists (apply).")
target = targets[0]
imgs = discover_images(target, recursive=args.recursive, limit=args.limit)
if not imgs:
sys.exit("No images found.")
csv_path = target / CSV_NAME
scored, errors = score_images(imgs, csv_path, rescore=args.rescore)
print(f"\n{len(scored)} images ready, {errors} errors (cache -> {csv_path})")
if args.apply:
tagged = sum(1 for p, s in scored if s >= args.threshold and write_keyword(str(p)))
print(f" tagged 'nsfw' in EXIF: {tagged} (score >= {args.threshold})")
return
# live review app: one page for ALL candidates across the tree, tag on click
cands = sorted([(p, s) for p, s in scored if s >= args.review_min], key=lambda x: -x[1])
if not cands:
print(f" no candidates >= {args.review_min} — nothing to review.")
return
serve_review(cands, args.threshold) # tag-state is checked lazily in-browser, per folder
if __name__ == "__main__":
main()

426
nsfwtag/bench.py Normal file
View File

@@ -0,0 +1,426 @@
"""Benchmark / compare NSFW models on YOUR OWN photos.
python -m nsfwtag.bench "<sample_folder>" [-r] [--models a,b,c] [--limit N] [--threshold 0.85]
Scores every image with each selected model and writes, into the sample folder:
* bench_scores.csv — path + one nsfw-score column per model
* bench_report.html — thumbnails with per-model scores, sortable, disagreements
highlighted (open it to eyeball which model over-flags)
…plus a console summary (flag counts, agreement, biggest disagreements).
Every score is normalized to nsfw = 1 - P(safe-class) so binary and multi-class
models are directly comparable. Nothing leaves the machine.
Models (need `pip install timm` for the timm ones; each downloads once):
falconsai Falconsai/nsfw_image_detection (current, transformers, binary)
adamcodd AdamCodd/vit-base-nsfw-detector (transformers, binary)
marqo Marqo/nsfw-image-detection-384 (timm, binary, tiny)
owen-xs OwenElliott/image-safety-classifier-xs (timm, 3-class SFW/NSFW/NSFL)
"""
import argparse
import csv
import gc
import html
import json
import os
import sys
import webbrowser
from pathlib import Path
from urllib.parse import quote
from .exif import read_marks
from .scoring import discover_images
BATCH = 16
# label (lowercased) treated as "safe"; nsfw score = 1 - sum(P over these)
SAFE_LABELS = {"sfw", "safe", "normal", "neutral"}
MODELS = { # name -> (kind, hf_id)
"falconsai": ("hf", "Falconsai/nsfw_image_detection"),
"adamcodd": ("hf", "AdamCodd/vit-base-nsfw-detector"),
"marqo": ("timm", "Marqo/nsfw-image-detection-384"),
"owen-xs": ("timm", "OwenElliott/image-safety-classifier-xs"),
}
def _device():
import torch
return "mps" if torch.backends.mps.is_available() else "cpu"
def _unsafe_indices(labels):
"""Given lowercased class labels, return (safe_idx, use_safe). If any safe
label is present we score 1 - P(safe); otherwise P(labels that look unsafe)."""
safe = [i for i, l in enumerate(labels) if l in SAFE_LABELS]
if safe:
return safe, True
unsafe = [i for i, l in enumerate(labels)
if any(k in l for k in ("nsfw", "nsfl", "porn", "sexy", "hentai", "explicit"))]
return unsafe, False
def load_hf(model_id):
import torch
from transformers import AutoImageProcessor, AutoModelForImageClassification
dev = _device()
proc = AutoImageProcessor.from_pretrained(model_id)
model = AutoModelForImageClassification.from_pretrained(model_id).to(dev).eval()
labels = [model.config.id2label[i].lower() for i in range(len(model.config.id2label))]
idx, use_safe = _unsafe_indices(labels)
def score(imgs):
inputs = proc(images=imgs, return_tensors="pt").to(dev)
with torch.no_grad():
p = model(**inputs).logits.softmax(-1)
part = p[:, idx].sum(-1) if idx else p[:, :1].sum(-1) * 0
return (1 - part if use_safe else part).cpu().tolist()
return score, model
def load_timm(model_id):
import timm
import torch
dev = _device()
model = timm.create_model(f"hf_hub:{model_id}", pretrained=True).to(dev).eval()
cfg = timm.data.resolve_model_data_config(model)
tf = timm.data.create_transform(**cfg, is_training=False)
labels = [l.lower() for l in model.pretrained_cfg["label_names"]]
idx, use_safe = _unsafe_indices(labels)
def score(imgs):
batch = torch.stack([tf(im) for im in imgs]).to(dev)
with torch.no_grad():
p = model(batch).softmax(-1)
part = p[:, idx].sum(-1) if idx else p[:, :1].sum(-1) * 0
return (1 - part if use_safe else part).cpu().tolist()
return score, model
def score_all(names, imgs):
"""Return {model_name: {path: nsfw_score}}; loads one model at a time."""
from PIL import Image, ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True
dev = _device()
out = {}
for name in names:
kind, mid = MODELS[name]
print(f"\n[{name}] loading {mid} on {dev} …", file=sys.stderr)
try:
score, model = (load_hf if kind == "hf" else load_timm)(mid)
except Exception as e:
print(f" SKIP {name}: {e}", file=sys.stderr)
continue
res = {}
for i in range(0, len(imgs), BATCH):
pil, paths = [], []
for p in imgs[i:i + BATCH]:
try:
pil.append(Image.open(p).convert("RGB")); paths.append(p)
except Exception as e:
print(f"\n error {p}: {e}", file=sys.stderr)
if not pil:
continue
for p, s in zip(paths, score(pil)):
res[str(p)] = float(s)
print(f"\r {min(i + BATCH, len(imgs))}/{len(imgs)}", end="", file=sys.stderr, flush=True)
print(file=sys.stderr)
out[name] = res
del score, model
gc.collect()
return out
def _chip_color(s):
hue = round((1 - s) * 130) # green (safe) -> red (unsafe)
return f"hsl({hue} 70% 40%)"
def write_report(rows, names, target, threshold, gt):
"""rows: list of (path, {name: score}). gt: {path: 1(nsfw)/0(sfw)}. Writes
bench_report.html in target. Chips that disagree with your manual tag get a
red ring; sortable by error count."""
target = Path(target)
cards = []
for p, scores in rows:
rel = quote(os.path.relpath(p, target))
vals = [scores.get(n) for n in names if scores.get(n) is not None]
spread = (max(vals) - min(vals)) if len(vals) > 1 else 0.0
gtv = gt.get(p) # 1 nsfw / 0 sfw / None
errors, chips = 0, []
for n in names:
s = scores.get(n)
if s is None:
chips.append(f'<span class="chip na">{n} —</span>'); continue
hot = s >= threshold
err = gtv is not None and (hot != (gtv == 1))
errors += err
cls = "chip" + (" hot" if hot else "") + (" err" if err else "")
chips.append(f'<span class="{cls}" style="background:{_chip_color(s)}">{n} {s:.2f}</span>')
gtbadge = (f'<span class="gt gt-{"nsfw" if gtv==1 else "sfw"}">you: {"nsfw" if gtv==1 else "sfw"}</span>'
if gtv is not None else "")
data = html.escape(json.dumps({n: scores.get(n) for n in names}), quote=True)
cards.append(
f'<div class="card" data-scores="{data}" data-spread="{spread:.4f}" data-errors="{errors}" '
f'data-name="{html.escape(Path(p).name.lower(), quote=True)}">'
f'{gtbadge}<img loading="lazy" src="{rel}" alt="">'
f'<div class="chips">{"".join(chips)}</div>'
f'<div class="nm" title="{html.escape(str(p), quote=True)}">{html.escape(Path(p).name)}</div></div>')
opts = []
if gt:
opts.append('<option value="errors" selected>Errors vs. your tags ↓</option>')
opts.append('<option value="spread"' + ('' if gt else ' selected') + '>Model disagreement ↓</option>')
opts += [f'<option value="{n}">{n} ↓</option>' for n in names]
opts.append('<option value="name">Name</option>')
doc = _REPORT.replace("%%CARDS%%", "".join(cards)).replace("%%SORTOPTS%%", "".join(opts)) \
.replace("%%N%%", str(len(rows))).replace("%%THR%%", f"{threshold:.2f}")
out = target / "bench_report.html"
out.write_text(doc, encoding="utf-8")
return out
_REPORT = """<!doctype html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1"><title>NSFW model bench (%%N%%)</title>
<style>
*{box-sizing:border-box}body{margin:0;background:#0b0d10;color:#e8eaee;font-size:14px;
font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif}
.bar{position:sticky;top:0;z-index:5;display:flex;flex-wrap:wrap;gap:14px;align-items:center;
padding:12px 16px;background:rgba(11,13,16,.92);backdrop-filter:blur(8px);border-bottom:1px solid #262b34}
.bar b{font-size:15px}.bar .sum{color:#9aa1ae;font-size:12.5px}
select{background:#14171c;color:#e8eaee;border:1px solid #262b34;border-radius:8px;height:32px;padding:0 8px}
.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(200px,1fr));gap:12px;padding:14px}
.card{position:relative;background:#14171c;border:1px solid #262b34;border-radius:10px;overflow:hidden}
.card img{width:100%;height:190px;object-fit:cover;display:block;background:#000}
.gt{position:absolute;top:6px;left:6px;z-index:2;font-size:10px;font-weight:700;padding:2px 6px;border-radius:5px}
.gt-nsfw{background:#e05fa8;color:#2a0a1c}.gt-sfw{background:#10b981;color:#04130c}
.chips{display:flex;flex-wrap:wrap;gap:4px;padding:7px}
.chip{font-size:11px;font-weight:700;color:#fff;border-radius:5px;padding:2px 6px;border:2px solid transparent;
font-variant-numeric:tabular-nums;text-shadow:0 1px 2px rgba(0,0,0,.5)}
.chip.hot{border-color:#fff}.chip.err{border-color:#ef4444}.chip.na{background:#20252e!important;color:#6b727f}
.nm{font-size:11px;color:#8b90a0;padding:0 8px 8px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
</style></head><body>
<div class="bar"><b>NSFW model bench</b><span class="sum">%%N%% images · white ring = ≥ %%THR%% · red ring = disagrees with your tag</span>
<label style="margin-left:auto;color:#9aa1ae;font-size:12.5px">Sort
<select id="sort">%%SORTOPTS%%</select></label></div>
<div class="grid" id="grid">%%CARDS%%</div>
<script>
const grid=document.getElementById('grid'),cards=[...grid.children],sel=document.getElementById('sort');
sel.addEventListener('change',e=>{
const k=e.target.value;
cards.sort((a,b)=>{
if(k==='name')return a.dataset.name.localeCompare(b.dataset.name);
if(k==='spread')return b.dataset.spread-a.dataset.spread;
if(k==='errors')return b.dataset.errors-a.dataset.errors;
const av=JSON.parse(a.dataset.scores)[k],bv=JSON.parse(b.dataset.scores)[k];
return (bv==null?-1:bv)-(av==null?-1:av);});
cards.forEach(c=>grid.appendChild(c));});
sel.dispatchEvent(new Event('change')); // apply default sort on load
</script></body></html>"""
def summarize(rows, names, threshold):
print(f"\n=== {len(rows)} images · flagged = score ≥ {threshold} ===")
print(f"{'model':<12}{'flagged':>10}{'mean':>8}{'median':>8}")
for n in names:
vals = sorted(s[n] for _, s in rows if s.get(n) is not None)
if not vals:
print(f"{n:<12}{'(skipped)':>10}"); continue
flagged = sum(v >= threshold for v in vals)
mean = sum(vals) / len(vals)
med = vals[len(vals) // 2]
print(f"{n:<12}{f'{flagged}/{len(vals)}':>10}{mean:>8.3f}{med:>8.3f}")
ok = [n for n in names if any(s.get(n) is not None for _, s in rows)]
if len(ok) > 1:
print("\npairwise agreement (same side of threshold):")
for i in range(len(ok)):
for j in range(i + 1, len(ok)):
a, b = ok[i], ok[j]
both = [(s[a], s[b]) for _, s in rows if s.get(a) is not None and s.get(b) is not None]
agree = sum((x >= threshold) == (y >= threshold) for x, y in both)
print(f" {a:<10} vs {b:<10} {agree}/{len(both)} ({100*agree/max(1,len(both)):.0f}%)")
if len(ok) > 1:
print("\nbiggest disagreements (open the HTML to judge):")
spread = sorted(rows, key=lambda r: -(max(v for v in (r[1].get(n) for n in ok) if v is not None)
- min(v for v in (r[1].get(n) for n in ok) if v is not None)))
for p, s in spread[:8]:
print(" " + Path(p).name + " " + " ".join(f"{n}={s[n]:.2f}" for n in ok if s.get(n) is not None))
def metrics(rows, names, gt, threshold):
"""Per-model precision/recall/F1 vs. your manual tags, at the given threshold,
plus each model's F1-optimal threshold."""
labs = [gt[p] for p, _ in rows if p in gt]
npos, nneg = sum(labs), len(labs) - sum(labs)
single = npos == 0 or nneg == 0 # a known single-class folder (--assume)
print(f"\n=== vs. ground truth · {npos} nsfw + {nneg} sfw · flag = score ≥ {threshold} ===")
print(f"{'model':<11}{'acc':>7}{'prec':>7}{'recall':>8}{'F1':>6}{'FP':>5}{'FN':>5}{' best-thr':>11}{'F1@best':>8}")
for n in names:
pairs = [(gt[p], s[n]) for p, s in rows if p in gt and s.get(n) is not None]
if not pairs:
print(f"{n:<11}{'(skipped)':>7}"); continue
def conf(t):
tp = fp = fn = tn = 0
for y, sc in pairs:
pred = sc >= t
if y == 1 and pred: tp += 1
elif y == 0 and pred: fp += 1
elif y == 1: fn += 1
else: tn += 1
return tp, fp, fn, tn
def f1(t):
tp, fp, fn, _ = conf(t)
pr = tp / (tp + fp) if tp + fp else 0
rc = tp / (tp + fn) if tp + fn else 0
return 2 * pr * rc / (pr + rc) if pr + rc else 0
tp, fp, fn, tn = conf(threshold)
pr = tp / (tp + fp) if tp + fp else 0
rc = tp / (tp + fn) if tp + fn else 0
f = 2 * pr * rc / (pr + rc) if pr + rc else 0
acc = (tp + tn) / len(pairs)
if single:
bt_col = f"{'—':>11}{'—':>8}" # threshold tuning needs both classes
else:
cand = sorted({round(sc, 3) for _, sc in pairs} | {threshold})
bt = max(cand, key=f1)
bt_col = f"{bt:>11.2f}{f1(bt):>8.2f}"
print(f"{n:<11}{acc:>7.2f}{pr:>7.2f}{rc:>8.2f}{f:>6.2f}{fp:>5}{fn:>5}{bt_col}")
if single and npos:
print("single-class NSFW folder: recall = catch rate, FN = missed. (precision/FP need safe images too — "
"run the bench on a mixed/SFW folder for those.)")
elif single:
print("single-class SFW folder: FP = false alarms, acc = correct-reject rate. (recall needs NSFW images.)")
else:
print("prec = of what it flagged, how much you'd tagged nsfw (fewer false alarms) · "
"recall = of your nsfw, how much it caught · FP/FN at the current threshold")
def combine(csv_paths, threshold):
"""Pool the labeled rows from several bench_scores.csv files and print a
precision/recall/F1 threshold sweep per model — the real cross-dataset verdict."""
per_model, seen = {}, set()
pos = neg = 0
for cp in csv_paths:
with open(cp, newline="", encoding="utf-8") as fh:
r = csv.DictReader(fh)
models = [c for c in r.fieldnames if c not in ("path", "manual")]
for row in r:
man = row.get("manual", "")
if man not in ("nsfw", "sfw") or row["path"] in seen:
continue
seen.add(row["path"])
y = 1 if man == "nsfw" else 0
pos += y; neg += (1 - y)
for m in models:
v = row.get(m, "")
if v != "":
per_model.setdefault(m, []).append((y, float(v)))
if not pos or not neg:
sys.exit("Need both nsfw and sfw labels pooled — include a mixed/SFW run and an NSFW run.")
print(f"pooled {pos} nsfw + {neg} sfw from {len(csv_paths)} file(s)")
def prf(pairs, t):
tp = sum(y == 1 and s >= t for y, s in pairs)
fp = sum(y == 0 and s >= t for y, s in pairs)
fn = sum(y == 1 and s < t for y, s in pairs)
pr = tp / (tp + fp) if tp + fp else 0
rc = tp / (tp + fn) if tp + fn else 0
return pr, rc, (2 * pr * rc / (pr + rc) if pr + rc else 0), fp, fn
ranking = []
for m, pairs in per_model.items():
cand = sorted({round(s, 3) for _, s in pairs})
best = max(cand, key=lambda t: prf(pairs, t)[2])
ranking.append((prf(pairs, best)[2], best, m))
print(f"\n[{m}] best-F1 threshold = {best:.2f}")
print(f" {'thr':>5}{'prec':>7}{'recall':>8}{'F1':>6}{'FP':>6}{'FN':>6}")
for t in sorted(set([0.5, 0.7, 0.85, 0.9, 0.95, 0.99, round(best, 2)])):
pr, rc, f, fp, fn = prf(pairs, t)
print(f" {t:>5.2f}{pr:>7.2f}{rc:>8.2f}{f:>6.2f}{fp:>6}{fn:>6}{' <- best F1' if abs(t - best) < 5e-3 else ''}")
ranking.sort(reverse=True)
print("\nranking by best F1: " + " ".join(f"{m}={f:.2f}@{t:.2f}" for f, t, m in ranking))
def main():
ap = argparse.ArgumentParser(prog="nsfwtag.bench", description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("target", nargs="?", help="folder of photos to score with each model")
ap.add_argument("--combine", nargs="+", metavar="CSV", help="pool these bench_scores.csv files (using their manual labels) and print a precision/recall/threshold sweep per model — no scoring")
ap.add_argument("-r", "--recursive", action="store_true", help="descend into subfolders")
ap.add_argument("--models", default=",".join(MODELS), help=f"comma list (default: all) — {', '.join(MODELS)}")
ap.add_argument("--limit", type=int, default=0, help="cap number of images scored (quick pass)")
ap.add_argument("--threshold", type=float, default=0.85, help="flag cutoff for the report (default 0.85)")
ap.add_argument("--all", action="store_true", help="score ALL images, not just your manually-tagged ones")
ap.add_argument("--assume", choices=["nsfw", "sfw"], help="treat EVERY image in the folder as this label (a known folder) instead of reading EXIF tags")
args = ap.parse_args()
if args.combine:
combine(args.combine, args.threshold)
return
if not args.target:
ap.error("a target folder is required (or use --combine)")
target = Path(args.target).expanduser()
if not target.is_dir():
sys.exit(f"Not a folder: {target}\n Check the path/spelling (that folder does not exist).")
names = [n.strip() for n in args.models.split(",") if n.strip()]
bad = [n for n in names if n not in MODELS]
if bad:
sys.exit(f"Unknown model(s): {bad}. Available: {list(MODELS)}")
imgs = discover_images(target, recursive=args.recursive)
if not imgs:
has_subdirs = any(p.is_dir() for p in target.iterdir())
hint = " Images look to be in subfolders — add -r to scan recursively." if (has_subdirs and not args.recursive) else ""
sys.exit(f"No images found directly in {target}.\n{hint}".rstrip())
if args.assume: # whole folder is a known class
if args.limit:
imgs = imgs[:args.limit]
gt = {str(p): (1 if args.assume == "nsfw" else 0) for p in imgs}
print(f"assuming all {len(imgs)} images are {args.assume.upper()} (folder is ground truth).", file=sys.stderr)
else:
# ground truth from your manual nsfw/sfw EXIF tags (one batched exiftool read)
print(f"found {len(imgs)} images — reading your manual tags …", file=sys.stderr)
marks = read_marks([str(p) for p in imgs])
gt = {p: 1 for p in marks["nsfw"]}
gt.update({p: 0 for p in marks["sfw"] if p not in marks["nsfw"]})
if gt and not args.all:
imgs = [p for p in imgs if str(p) in gt] # evaluate only what you've tagged
print(f"evaluating on {len(imgs)} tagged images "
f"({sum(gt.values())} nsfw / {len(gt) - sum(gt.values())} sfw). Use --all to score everything.", file=sys.stderr)
elif not gt:
print("no manual nsfw/sfw tags found — falling back to a relative model comparison.", file=sys.stderr)
if args.limit:
imgs = imgs[:args.limit]
print(f"scoring {len(imgs)} images with {len(names)} model(s): {names}", file=sys.stderr)
scores = score_all(names, imgs)
names = [n for n in names if n in scores] # keep only models that loaded
if not names:
sys.exit("No models loaded (need: pip install torch transformers timm).")
rows = [(str(p), {n: scores[n].get(str(p)) for n in names}) for p in imgs]
csv_path = target / "bench_scores.csv"
with open(csv_path, "w", newline="", encoding="utf-8") as fh:
w = csv.writer(fh); w.writerow(["path", "manual", *names])
for p, s in rows:
man = {1: "nsfw", 0: "sfw"}.get(gt.get(p), "")
w.writerow([p, man, *[("" if s[n] is None else f"{s[n]:.4f}") for n in names]])
if any(p in gt for p, _ in rows):
metrics(rows, names, gt, args.threshold)
else:
summarize(rows, names, args.threshold)
report = write_report(rows, names, target, args.threshold, gt)
print(f"\nscores -> {csv_path}\nreport -> {report}", file=sys.stderr)
webbrowser.open(report.resolve().as_uri())
if __name__ == "__main__":
main()

123
nsfwtag/exif.py Normal file
View File

@@ -0,0 +1,123 @@
"""EXIF keyword read/write via exiftool.
Only the Keywords + Subject fields are ever touched, so every other tag
(Caption, ImageDescription, dates, GPS, Artist, …) is preserved on tag/untag.
"""
import json
import os
import subprocess
import sys
from pathlib import Path
from . import KEYWORD
def write_keyword(path: str, kw: str = KEYWORD) -> bool:
"""Idempotently add a keyword to EXIF Keywords + Subject (no duplicates on re-run)."""
return subprocess.run(
["exiftool", "-m", "-overwrite_original",
f"-Keywords-={kw}", f"-Keywords+={kw}",
f"-Subject-={kw}", f"-Subject+={kw}", path],
capture_output=True, text=True,
).returncode == 0
def remove_keyword(path: str, kw: str = KEYWORD) -> bool:
"""Reverse of write_keyword: strip the keyword from EXIF Keywords + Subject."""
return subprocess.run(
["exiftool", "-m", "-overwrite_original",
f"-Keywords-={kw}", f"-Subject-={kw}", path],
capture_output=True, text=True,
).returncode == 0
def _read_keywords(paths) -> dict:
"""{path: set(lowercased Keywords+Subject values)} via one batched exiftool
call (paths fed via stdin). Empty dict if exiftool isn't available."""
paths = list(paths)
if not paths:
return {}
want = {os.path.normpath(p): p for p in paths}
try:
r = subprocess.run(
["exiftool", "-m", "-j", "-Keywords", "-Subject", "-@", "-"],
input="\n".join(paths), capture_output=True, text=True,
)
except FileNotFoundError:
return {}
out = {}
try:
for obj in json.loads(r.stdout or "[]"):
vals = []
for f in ("Keywords", "Subject"):
v = obj.get(f)
if isinstance(v, list):
vals += v
elif v is not None:
vals.append(v)
key = want.get(os.path.normpath(obj.get("SourceFile", "")))
if key is not None:
out[key] = {str(x).strip().lower() for x in vals}
except (ValueError, TypeError):
pass
return out
def read_tagged(paths, kw: str = KEYWORD) -> set:
"""Subset of `paths` whose EXIF already contains keyword `kw`."""
kw = kw.lower()
return {p for p, kws in _read_keywords(paths).items() if kw in kws}
def read_marks(paths) -> dict:
"""Which paths carry the 'nsfw' vs the 'sfw' review keyword, in one read:
{'nsfw': set, 'sfw': set}. They should be mutually exclusive; if a stray file
somehow has both, nsfw wins so it never reads as safe."""
got = _read_keywords(paths)
nsfw = {p for p, k in got.items() if "nsfw" in k}
sfw = {p for p, k in got.items() if "sfw" in k and p not in nsfw}
return {"nsfw": nsfw, "sfw": sfw}
_EXIF_NOISE = {
"SourceFile", "ExifToolVersion", "Directory", "FilePermissions",
"FileModifyDate", "FileAccessDate", "FileInodeChangeDate", "FileTypeExtension",
}
def read_exif(path: str) -> dict:
"""Human-relevant EXIF tags for one file (for the lightbox overlay), in
exiftool's order. Skips file-system noise and binary/oversized values. When
the photo is geotagged, a "Map" entry with a Google Maps URL is prepended.
`-c %+.6f` prints GPS coordinates as signed decimals (other tags unaffected)."""
r = subprocess.run(["exiftool", "-m", "-j", "-c", "%+.6f", path], capture_output=True, text=True)
try:
obj = json.loads(r.stdout or "[]")[0]
except (ValueError, IndexError):
return {}
out = {}
for k, v in obj.items():
if k in _EXIF_NOISE:
continue
s = ", ".join(str(x) for x in v) if isinstance(v, list) else str(v)
if not s or s.startswith("(Binary data") or len(s) > 300:
continue
out[k] = s
lat, lon = out.get("GPSLatitude"), out.get("GPSLongitude")
if lat and lon:
try:
out = {"Map": f"https://www.google.com/maps?q={float(lat):.6f},{float(lon):.6f}", **out}
except ValueError:
pass
return out
def apply_list(list_file: str):
"""Write the nsfw keyword to every path in a newline-delimited file."""
paths = [l.strip() for l in Path(list_file).read_text(encoding="utf-8").splitlines() if l.strip()]
ok = miss = 0
for p in paths:
if not Path(p).exists():
miss += 1; print(f" missing: {p}", file=sys.stderr); continue
ok += 1 if write_keyword(p) else 0
print(f"tagged 'nsfw' in EXIF: {ok}/{len(paths)}" + (f" ({miss} missing)" if miss else ""))

426
nsfwtag/review.html Normal file
View File

@@ -0,0 +1,426 @@
<!doctype html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>NSFW Review · %%TOTAL%% images</title>
<style>
*{box-sizing:border-box}
:root{
--bg:#0b0d10;--surface:#14171c;--surface-2:#1b1f26;--elev:#20252e;
--border:#262b34;--border-2:#323947;--text:#e8eaee;--muted:#9aa1ae;--faint:#6b727f;
--primary:#10b981;--primary-2:#0ea372;--primary-ink:#04130c;--danger:#ef4444;--ring:#3b82f6;--pink:#e05fa8;
--r:12px;--r-sm:8px}
@media (prefers-reduced-motion:reduce){*{transition:none!important;animation:none!important}}
html,body{margin:0}
body{background:var(--bg);color:var(--text);font-size:14px;line-height:1.5;-webkit-font-smoothing:antialiased;
font-family:'Inter',-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Helvetica,Arial,sans-serif}
svg{width:1em;height:1em;display:block}
button{font:inherit;color:inherit;cursor:pointer}
:focus-visible{outline:2px solid var(--ring);outline-offset:2px}
.app{position:sticky;top:0;z-index:30;display:flex;align-items:center;gap:14px;min-height:58px;
padding:11px 18px;background:rgba(11,13,16,.86);backdrop-filter:blur(12px);border-bottom:1px solid var(--border)}
.brand{display:flex;align-items:center;gap:11px}
.logo{display:grid;place-items:center;width:34px;height:34px;border-radius:10px;font-size:18px;
background:linear-gradient(135deg,#1e3a5f,#10b981);color:#fff}
.titles h1{margin:0;font-size:15px;font-weight:600;letter-spacing:-.01em}
.titles p{margin:0;font-size:11px;color:var(--faint)}
.grow{flex:1}
.metrics{display:flex;gap:8px}
.metric{display:flex;flex-direction:column;align-items:center;min-width:66px;padding:5px 12px;
background:var(--surface);border:1px solid var(--border);border-radius:var(--r-sm)}
.metric .num{font-size:16px;font-weight:600;font-variant-numeric:tabular-nums}
.metric .lbl{font-size:10px;color:var(--muted);text-transform:uppercase;letter-spacing:.05em}
.toolbar{position:sticky;top:58px;z-index:20;display:flex;flex-wrap:wrap;align-items:center;gap:10px;
padding:10px 18px;background:rgba(16,19,24,.92);backdrop-filter:blur(8px);border-bottom:1px solid var(--border)}
.field{display:flex;align-items:center;gap:8px;height:36px;padding:0 10px;background:var(--surface);
border:1px solid var(--border);border-radius:var(--r-sm)}
.field svg{color:var(--faint);font-size:15px}
.field input{background:none;border:0;outline:none;color:var(--text);height:100%;width:200px}
.field input::placeholder{color:var(--faint)}
.field:focus-within{border-color:var(--border-2);box-shadow:0 0 0 3px rgba(59,130,246,.22)}
.seg{display:inline-flex;padding:3px;background:var(--surface);border:1px solid var(--border);border-radius:var(--r-sm)}
.seg button{border:0;background:none;color:var(--muted);padding:5px 11px;border-radius:6px;font-size:12.5px;font-weight:500}
.seg button.on{background:var(--elev);color:var(--text)}
.seg button:hover:not(.on){color:var(--text)}
.legend{display:flex;gap:8px;align-items:center}
.chip{display:inline-flex;align-items:center;gap:6px;padding:4px 11px;border-radius:16px;font:inherit;
font-size:12px;font-weight:500;border:1px solid var(--border);background:var(--surface);color:var(--muted);
cursor:pointer;transition:opacity .15s,color .15s,border-color .15s}
.chip .sw{width:10px;height:10px;border-radius:3px}
.sw-nsfw{background:var(--pink)}
.sw-sfw{background:var(--primary)}
.chip.on{color:var(--text);border-color:var(--border-2)}
.chip:not(.on){opacity:.4;text-decoration:line-through}
.chip .cn{color:var(--faint);font-variant-numeric:tabular-nums}
.thr,.sortf{display:flex;align-items:center;gap:8px;color:var(--muted);font-size:12.5px;white-space:nowrap}
.thr output{min-width:2.4em;color:var(--text);font-weight:600;font-variant-numeric:tabular-nums}
input[type=range]{-webkit-appearance:none;appearance:none;width:110px;height:4px;border-radius:3px;background:var(--border-2);outline:none}
input[type=range]::-webkit-slider-thumb{-webkit-appearance:none;width:16px;height:16px;border-radius:50%;
background:var(--primary);cursor:pointer;border:2px solid var(--bg);box-shadow:0 0 0 1px var(--primary)}
select{height:36px;padding:0 8px;background:var(--surface);color:var(--text);border:1px solid var(--border);border-radius:var(--r-sm);outline:none}
select:focus{border-color:var(--border-2);box-shadow:0 0 0 3px rgba(59,130,246,.22)}
.btn{display:inline-flex;align-items:center;gap:7px;height:36px;padding:0 14px;border-radius:var(--r-sm);
border:1px solid var(--border);background:var(--surface);color:var(--text);font-weight:500;
transition:background .15s,border-color .15s,transform .05s}
.btn svg{font-size:15px}
.btn.ghost:hover{background:var(--elev);border-color:var(--border-2)}
.btn:active{transform:translateY(1px)}
.btn.primary{background:var(--primary);border-color:transparent;color:var(--primary-ink);font-weight:600}
.btn.primary:hover{background:var(--primary-2)}
.btn.primary:disabled{opacity:.45;cursor:not-allowed}
.btn.danger{border-color:var(--danger);color:#fca5a5}
.btn.danger:hover{background:rgba(239,68,68,.14)}
.btn:disabled{opacity:.45;cursor:not-allowed}
.btn.act-nsfw{background:var(--pink);border-color:transparent;color:#2a0a1c;font-weight:600}
.btn.act-nsfw:hover{filter:brightness(1.08)}
.btn.act-safe{background:var(--primary);border-color:transparent;color:var(--primary-ink);font-weight:600}
.btn.act-safe:hover{background:var(--primary-2)}
.ctl{display:flex;align-items:center;gap:6px;color:var(--muted);font-size:12.5px;white-space:nowrap}
.actions{display:flex;align-items:center;gap:8px;padding-left:10px;border-left:1px solid var(--border)}
.actions[hidden]{display:none}
.selinfo{color:var(--muted);font-size:12.5px;white-space:nowrap}
.selinfo b{color:var(--text);font-variant-numeric:tabular-nums}
.btn .pill{min-width:20px;height:19px;padding:0 6px;display:grid;place-items:center;border-radius:10px;
background:rgba(0,0,0,.22);font-size:11.5px;font-weight:700;font-variant-numeric:tabular-nums}
.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(190px,1fr));gap:12px;padding:16px 18px 64px}
.card{position:relative;overflow:hidden;background:var(--surface);border:2px solid var(--border);border-radius:var(--r);
cursor:pointer;transition:border-color .15s,transform .12s,box-shadow .15s}
.card:hover{transform:translateY(-2px);box-shadow:0 8px 24px rgba(0,0,0,.35)}
.card:focus-within{outline:2px solid var(--ring);outline-offset:2px}
.card.c-safe{border-color:var(--primary)}
.card.c-tag{border-color:var(--pink)}
.card.c-sel{border-color:var(--danger);box-shadow:0 0 0 1px var(--danger)}
.card.selected{outline:3px solid var(--ring);outline-offset:-3px}
.thumb{position:relative;aspect-ratio:1/1;background:#000}
.thumb img{width:100%;height:100%;object-fit:cover;display:block}
.nsfw{position:absolute;top:8px;left:8px;width:1px;height:1px;opacity:0;pointer-events:none}
.score{position:absolute;left:8px;top:8px;padding:2px 7px;border-radius:6px;color:#fff;font-size:11.5px;
font-weight:700;font-variant-numeric:tabular-nums;text-shadow:0 1px 2px rgba(0,0,0,.5)}
.icon{display:grid;place-items:center;width:30px;height:30px;font-size:15px;border:0;border-radius:8px;
color:#fff;background:rgba(6,8,12,.62);backdrop-filter:blur(3px);transition:background .15s}
.icon:hover{background:rgba(6,8,12,.9)}
.zoom{position:absolute;right:8px;top:8px}
.tick{position:absolute;right:8px;bottom:8px;width:28px;height:28px;display:grid;place-items:center;border-radius:50%;
border:2px solid rgba(255,255,255,.55);background:rgba(6,8,12,.5);color:transparent;font-size:15px;
cursor:pointer;backdrop-filter:blur(3px);transition:background .15s,border-color .15s,color .15s,transform .1s}
.tick:hover{border-color:#fff;transform:scale(1.08)}
.card.t-nsfw .tick{background:var(--pink);border-color:var(--pink);color:#2a0a1c}
.card.t-sfw .tick{background:var(--primary);border-color:var(--primary);color:#04121f}
.tagged{position:absolute;left:8px;bottom:8px;display:inline-flex;align-items:center;gap:4px;
padding:2px 8px 2px 5px;border-radius:6px;background:var(--pink);color:#2a0a1c;font-size:11px;font-weight:700}
.tagged[hidden]{display:none}
.tagged svg{font-size:13px}
.safe{position:absolute;left:8px;bottom:8px;display:inline-flex;align-items:center;gap:4px;
padding:2px 8px 2px 5px;border-radius:6px;background:var(--primary);color:var(--primary-ink);font-size:11px;font-weight:700}
.safe[hidden]{display:none}
.safe svg{font-size:13px}
.info{padding:8px 10px 10px}
.name{font-size:12px;font-weight:500;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.path{margin-top:2px;font-size:10.5px;color:var(--faint);white-space:nowrap;overflow:hidden;text-overflow:ellipsis;direction:rtl;text-align:left}
.card.hidden{display:none}
.empty{padding:64px 20px;text-align:center;color:var(--muted)}
.navtoggle{display:grid;place-items:center;width:36px;height:36px;font-size:18px;border-radius:var(--r-sm);
background:var(--surface);border:1px solid var(--border);color:var(--muted);transition:background .15s,color .15s}
.navtoggle:hover{background:var(--elev);color:var(--text)}
.navtoggle.on{color:var(--primary);border-color:var(--border-2)}
.shell{display:flex;align-items:flex-start}
.sidebar{position:sticky;top:58px;align-self:flex-start;flex:0 0 252px;width:252px;height:calc(100dvh - 58px);
overflow:auto;padding:8px;background:var(--surface);border-right:1px solid var(--border)}
.shead{padding:8px 10px 4px;font-size:10px;font-weight:600;text-transform:uppercase;letter-spacing:.06em;color:var(--faint)}
.content{flex:1;min-width:0}
.shell.collapsed .sidebar{display:none}
.fitem{display:flex;align-items:center;gap:8px;width:100%;padding:6px 10px;border:0;border-radius:6px;
background:none;color:var(--muted);font-size:12.5px;text-align:left;transition:background .12s,color .12s}
.fitem:hover{background:var(--elev);color:var(--text)}
.fitem.on{background:var(--elev);color:var(--text);font-weight:600}
.fname{flex:1;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.fcount{min-width:22px;padding:0 7px;text-align:center;font-size:11px;font-variant-numeric:tabular-nums;
color:var(--faint);background:var(--surface-2);border:1px solid var(--border);border-radius:20px}
.fitem.on .fcount{color:var(--text);border-color:var(--border-2)}
@media (max-width:820px){.sidebar{position:fixed;left:0;top:58px;bottom:0;z-index:25;box-shadow:12px 0 32px rgba(0,0,0,.5)}}
.lb{position:fixed;inset:0;z-index:50;display:flex;gap:16px;padding:20px;
background:rgba(4,5,7,.94);backdrop-filter:blur(6px)}
.lb[hidden]{display:none}
.lb-fig{margin:0;flex:1;min-width:0;display:flex;flex-direction:column;align-items:center;justify-content:center;gap:14px}
.lb-fig img{max-width:100%;max-height:82vh;object-fit:contain;border-radius:8px;box-shadow:0 20px 60px rgba(0,0,0,.6)}
.lb-bar{display:flex;align-items:center;gap:14px;max-width:100%}
.lb-bar figcaption{color:var(--muted);font-size:13px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.lb-bar figcaption b{color:var(--text)}
.lb .icon{width:40px;height:40px;font-size:18px;background:rgba(20,23,28,.82);border:1px solid var(--border)}
.lb .icon:hover{background:var(--elev)}
.lb-close{position:absolute;top:20px;left:20px;z-index:2}
.lb-meta{flex:0 0 300px;width:300px;overflow:auto;background:rgba(20,23,28,.72);border:1px solid var(--border);
border-radius:10px;padding:12px 14px;font-size:11px;line-height:1.5}
.mrow{display:flex;gap:8px;padding:3px 0;border-bottom:1px solid rgba(255,255,255,.05)}
.mk{flex:0 0 42%;color:var(--faint);word-break:break-word}
.mv{flex:1;min-width:0;color:var(--text);word-break:break-word}
.mv a{color:#6ea8ff;text-decoration:none}
.mv a:hover{text-decoration:underline}
.mload{color:var(--muted);padding:6px 0}
@media (max-width:720px){.lb-meta{display:none}}
.toast{position:fixed;left:50%;bottom:26px;z-index:60;transform:translateX(-50%) translateY(12px);
padding:11px 18px;border-radius:10px;background:var(--elev);border:1px solid var(--border-2);color:var(--text);
font-size:13px;font-weight:500;box-shadow:0 12px 32px rgba(0,0,0,.5);opacity:0;pointer-events:none;
transition:opacity .2s,transform .2s}
.toast.show{opacity:1;transform:translateX(-50%) translateY(0)}
.toast.ok{border-color:var(--primary)}
.toast.err{border-color:var(--danger);color:#fecaca}
.prog{position:fixed;right:20px;bottom:20px;z-index:55;display:flex;align-items:center;gap:12px;
padding:10px 16px 10px 10px;background:var(--elev);border:1px solid var(--border-2);border-radius:12px;
box-shadow:0 12px 32px rgba(0,0,0,.5)}
.prog[hidden]{display:none}
.pring{position:relative;width:40px;height:40px;display:grid;place-items:center}
.pring svg{position:absolute;inset:0;transform:rotate(-90deg)}
.pring .pbg{fill:none;stroke:var(--border-2);stroke-width:3.5}
.pring .pfg{fill:none;stroke:var(--primary);stroke-width:3.5;stroke-linecap:round;
stroke-dasharray:100.53;stroke-dashoffset:100.53;transition:stroke-dashoffset .18s}
.pring #ppct{font-size:10.5px;font-weight:700;font-variant-numeric:tabular-nums}
.plbl{font-size:12px;color:var(--muted)}
.logpanel{position:fixed;left:0;right:0;bottom:0;z-index:45;height:240px;display:flex;flex-direction:column;
background:rgba(11,13,16,.98);border-top:1px solid var(--border-2);box-shadow:0 -12px 32px rgba(0,0,0,.5)}
.logpanel[hidden]{display:none}
.loghead{display:flex;align-items:center;gap:10px;padding:8px 14px;border-bottom:1px solid var(--border)}
.loghead b{font-size:12px}
.logbody{flex:1;overflow:auto;padding:8px 14px;line-height:1.6;
font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:11.5px}
.logline{display:flex;gap:10px;white-space:pre-wrap;word-break:break-word}
.logline .lt{flex:0 0 auto;color:var(--faint);font-variant-numeric:tabular-nums}
.logempty{color:var(--muted)}
@media (max-width:640px){
.metrics{display:none}.field input{width:120px}
.grid{grid-template-columns:repeat(auto-fill,minmax(140px,1fr));gap:8px;padding:12px}}
</style></head><body>
<header class="app">
<button id="navtoggle" class="navtoggle on" aria-label="Toggle folder browser" aria-pressed="true" onclick="toggleSidebar()"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="3" y="3" width="18" height="18" rx="2"/><path d="M9 3v18"/></svg></button>
<div class="brand">
<span class="logo"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10Z"/><path d="m9 12 2 2 4-4"/></svg></span>
<div class="titles"><h1>NSFW Review</h1><p>On-device · nothing leaves this Mac</p></div>
</div>
<div class="grow"></div>
<div class="metrics">
<div class="metric"><span class="num">%%TOTAL%%</span><span class="lbl">images</span></div>
</div>
<button id="logbtn" class="navtoggle" aria-label="Toggle activity log" onclick="toggleLog()"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M8 6h11M8 12h11M8 18h11"/><path d="M4 6h.01M4 12h.01M4 18h.01"/></svg></button>
</header>
<div class="shell" id="shell">
<aside class="sidebar" aria-label="Folder browser"><div class="shead">Folders</div>%%TREE%%</aside>
<div class="content">
<div class="toolbar">
<div class="field"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="11" cy="11" r="7"/><path d="m21 21-4.3-4.3"/></svg>
<input id="q" type="search" placeholder="Search name or folder…" aria-label="Search images"></div>
<label class="ctl">Show
<select id="showSel" aria-label="Show which images"><option value="all">All</option><option value="todo">To-handle</option><option value="nsfw">NSFW</option><option value="sfw">SFW</option></select></label>
<label class="ctl">Sort
<select id="sort" aria-label="Sort order"><option value="score">Score ↓</option><option value="name">Name</option><option value="path">Folder</option></select></label>
<label class="thr">NSFW ≥ <output id="thrOut">%%THRESHOLD%%</output>
<input id="thr" type="range" min="0" max="1" step="0.01" value="%%THRESHOLD%%" aria-label="NSFW score threshold"></label>
<button class="btn ghost" onclick="selectFlagged()">Select flagged</button>
<button class="btn ghost" onclick="setAll(true)">Select all</button>
<div class="grow"></div>
<div class="actions" id="actions" hidden>
<span class="selinfo"><b id="selCount">0</b> selected</span>
<button class="btn ghost" onclick="setAll(false)">Deselect</button>
<button id="untagb" class="btn ghost" onclick="untag()">Remove tag</button>
<button id="markSafeb" class="btn act-safe" onclick="markSafe()">Mark safe</button>
<button id="applyb" class="btn act-nsfw" onclick="apply()">Mark NSFW</button>
</div>
</div>
<main id="grid" class="grid">%%CARDS%%</main>
<p id="empty" class="empty" hidden>No images match your filter.</p>
</div></div>
<div id="lb" class="lb" hidden>
<button class="icon lb-close" aria-label="Close" onclick="closeLb()"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M18 6 6 18M6 6l12 12"/></svg></button>
<figure class="lb-fig">
<img id="lbImg" alt="">
<div class="lb-bar">
<button class="icon" aria-label="Previous" onclick="stepLb(-1)"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m15 18-6-6 6-6"/></svg></button>
<figcaption id="lbCap"></figcaption>
<button class="icon" aria-label="Next" onclick="stepLb(1)"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m9 18 6-6-6-6"/></svg></button>
</div>
</figure>
<aside id="lbMeta" class="lb-meta" aria-label="EXIF metadata"></aside>
</div>
<div id="prog" class="prog" role="status" aria-live="polite" hidden>
<div class="pring"><svg viewBox="0 0 36 36"><circle class="pbg" cx="18" cy="18" r="16"/><circle id="pring" class="pfg" cx="18" cy="18" r="16"/></svg><span id="ppct">0%</span></div>
<span class="plbl">Checking existing tags…</span>
</div>
<div id="logpanel" class="logpanel" hidden>
<div class="loghead"><b>Activity log</b><span id="logcount" class="plbl"></span><div class="grow"></div>
<button class="btn ghost" onclick="toggleLog()">Close</button></div>
<div id="logbody" class="logbody"><div class="logempty">No activity yet.</div></div>
</div>
<div id="toast" class="toast" role="status" aria-live="polite" hidden></div>
<script>
const $=s=>document.querySelector(s);
const grid=$('#grid'),q=$('#q'),thr=$('#thr'),thrOut=$('#thrOut'),sortSel=$('#sort'),showSel=$('#showSel'),
selCount=$('#selCount'),applyb=$('#applyb'),untagb=$('#untagb'),markSafeb=$('#markSafeb'),
actions=$('#actions'),empty=$('#empty'),toastEl=$('#toast');
const cards=[...document.querySelectorAll('.card')];
let folderFilter='',show='all';
function refreshTagN(){} // (chip counts removed with the toolbar redesign)
function pickFolder(btn){
folderFilter=btn.dataset.folder||'';
document.querySelectorAll('.fitem').forEach(x=>x.classList.toggle('on',x===btn));
applyView();scanFolder();}
// Lazily ask the server which images in the current folder already carry the
// 'nsfw' tag, in chunks, updating the progress ring and filtering on the fly.
const prog=$('#prog'),pring=$('#pring'),ppct=$('#ppct'),PC=100.53;
function setProg(f){ppct.textContent=Math.round(f*100)+'%';pring.style.strokeDashoffset=PC*(1-f);}
let scanning=false;
async function scanFolder(){
if(scanning)return;
const scope=cards.filter(c=>c.dataset.tagged==='?'&&inFolder(c));
if(!scope.length)return;
scanning=true;prog.hidden=false;setProg(0);
const CH=300;let done=0;
for(let i=0;i<scope.length;i+=CH){
const chunk=scope.slice(i,i+CH);let res={nsfw:[],sfw:[]};
try{res=await(await fetch('/tagged',{method:'POST',body:JSON.stringify(chunk.map(c=>box(c).value))})).json();}
catch(e){}
const nset=new Set(res.nsfw),sset=new Set(res.sfw);
chunk.forEach(c=>{const v=box(c).value,isN=nset.has(v),isS=sset.has(v);
c.dataset.tagged=isN?'1':'0';c.dataset.sfw=isS?'1':'0';
if(isN||isS)box(c).checked=false; // decided: nsfw->pink / sfw->green
else box(c).checked=parseFloat(c.dataset.score)>=curThr(); // undecided: flag by score -> red
const b=c.querySelector('.tagged');if(b)b.hidden=!isN;paint(c);});
done+=chunk.length;setProg(done/scope.length);
refreshTagN();updateCount();applyView();
}
scanning=false;prog.hidden=true;}
function inFolder(c){if(!folderFilter)return true;const f=c.dataset.folder;return f===folderFilter||f.startsWith(folderFilter+'/');}
function toggleSidebar(){
const sh=$('#shell'),tb=$('#navtoggle'),off=sh.classList.toggle('collapsed');
tb.classList.toggle('on',!off);tb.setAttribute('aria-pressed',String(!off));
if(off){folderFilter='';document.querySelectorAll('.fitem').forEach((x,i)=>x.classList.toggle('on',i===0));applyView();}}
document.querySelectorAll('.score').forEach(el=>{
const s=parseFloat(el.dataset.s),hue=Math.round((1-s)*130);
el.style.background='hsl('+hue+' 70% 40% / .9)';});
const box=c=>c.querySelector('.nsfw');
const visible=c=>!c.classList.contains('hidden');
// Counts + actions operate on the CURRENT view (folder/search/filter) so the
// counter tracks what you see, and Clear/Invert change it as expected.
const selected=()=>cards.filter(c=>box(c).checked&&visible(c));
const curThr=()=>parseFloat(thr.value);
// Two independent channels: DECISION (the EXIF tag) colors the frame + tick;
// SELECTION (the checkbox, for bulk actions) shows as a ring — a card can be both.
function paint(c){const sel=box(c).checked,tag=c.dataset.tagged==='1',safe=c.dataset.sfw==='1';
c.classList.toggle('c-tag',tag); // pink frame: nsfw in EXIF
c.classList.toggle('c-safe',!tag&&safe); // green frame: sfw in EXIF
c.classList.toggle('c-sel',!tag&&!safe&&sel); // red frame: undecided & selected (bulk candidate)
c.classList.toggle('t-nsfw',tag); // tick fill mirrors the decision
c.classList.toggle('t-sfw',safe);
c.classList.toggle('selected',sel); // ring: selected (visible even on tagged cards)
const tb=c.querySelector('.tagged');if(tb)tb.hidden=!tag; // pink "tagged" badge (nsfw in EXIF)
const sb=c.querySelector('.safe');if(sb)sb.hidden=!safe;} // green "tagged" badge (sfw in EXIF)
const paintAll=()=>cards.forEach(paint);
// Per-image decision. One click writes EXIF and cycles: undecided -> NSFW -> SFW -> undecided.
// Independent of selection, so it never needs a second click and always reaches every state.
async function toggleCard(btn){
const c=btn.closest('.card');
const cur=c.dataset.tagged==='1'?'nsfw':(c.dataset.sfw==='1'?'sfw':'none');
const next=cur==='none'?'nsfw':(cur==='nsfw'?'sfw':'none');
const url=next==='nsfw'?'/apply':next==='sfw'?'/mark?state=sfw':'/untag';
c.dataset.tagged=next==='nsfw'?'1':'0';c.dataset.sfw=next==='sfw'?'1':'0';
paint(c);updateCount(); // optimistic
try{await fetch(url,{method:'POST',body:JSON.stringify([box(c).value])});}
catch(e){toast('Could not save decision — '+e,'err');}
refreshTagN();updateCount();}
function selectFlagged(){cards.forEach(c=>{if(!c.classList.contains('hidden')&&c.dataset.tagged!=='1'&&parseFloat(c.dataset.score)>=curThr())box(c).checked=true;});updateCount();paintAll();}
function updateCount(){const sel=selected(),n=sel.length,nt=sel.filter(c=>c.dataset.tagged==='1'||c.dataset.sfw==='1').length;
selCount.textContent=n;actions.hidden=n===0;
applyb.disabled=n===0;markSafeb.disabled=n===0;untagb.disabled=nt===0;}
function setAll(v){cards.forEach(c=>{if(!c.classList.contains('hidden'))box(c).checked=v;});updateCount();paintAll();}
grid.addEventListener('change',e=>{if(e.target.classList.contains('nsfw')){updateCount();paint(e.target.closest('.card'));}});
thr.addEventListener('input',()=>{thrOut.textContent=curThr().toFixed(2);paintAll();});
showSel.addEventListener('change',()=>{show=showSel.value;applyView();});
q.addEventListener('input',applyView);
sortSel.addEventListener('change',applyView);
function applyView(){
const term=q.value.trim().toLowerCase();
let vis=cards.filter(c=>{
if(term&&!c.dataset.hay.includes(term))return false;
if(folderFilter){const f=c.dataset.folder;if(f!==folderFilter&&!f.startsWith(folderFilter+'/'))return false;}
if(show==='todo'&&(c.dataset.tagged==='1'||c.dataset.sfw==='1'))return false; // only un-decided
if(show==='nsfw'&&c.dataset.tagged!=='1')return false; // only NSFW-tagged
if(show==='sfw'&&c.dataset.sfw!=='1')return false; // only SFW-tagged
return true;});
const key=sortSel.value;
vis.sort((a,b)=>{
if(key==='name')return a.dataset.name.localeCompare(b.dataset.name);
if(key==='path')return a.dataset.folder.localeCompare(b.dataset.folder);
return parseFloat(b.dataset.score)-parseFloat(a.dataset.score);});
cards.forEach(c=>c.classList.add('hidden'));
const frag=document.createDocumentFragment();
vis.forEach(c=>{c.classList.remove('hidden');frag.appendChild(c);});
grid.appendChild(frag);
empty.hidden=vis.length>0;
updateCount();} // counter reflects the folder/filter you're viewing
const lb=$('#lb'),lbImg=$('#lbImg'),lbCap=$('#lbCap'),lbMeta=$('#lbMeta');
const esc=s=>String(s).replace(/[&<>"]/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
let lbList=[],lbIdx=-1,metaTok=0;
function openLb(btn){
lbList=cards.filter(c=>!c.classList.contains('hidden'));
lbIdx=lbList.indexOf(btn.closest('.card'));showLb();lb.hidden=false;}
function metaVal(v){
if(/^https?:\/\//.test(v)){const t=v.includes('google.com/maps')?'Open in Google Maps ↗':esc(v);
return '<a href="'+esc(v)+'" target="_blank" rel="noopener">'+t+'</a>';}
return esc(v);}
async function loadMeta(path){
const tok=++metaTok;lbMeta.innerHTML='<div class="mload">Loading EXIF…</div>';
try{
const d=await(await fetch('/exif?path='+encodeURIComponent(path))).json();
if(tok!==metaTok)return;
const rows=Object.entries(d).map(([k,v])=>'<div class="mrow"><span class="mk">'+esc(k)+'</span><span class="mv">'+metaVal(v)+'</span></div>').join('');
lbMeta.innerHTML=rows||'<div class="mload">No EXIF.</div>';
}catch(e){if(tok===metaTok)lbMeta.innerHTML='<div class="mload">EXIF unavailable.</div>';}}
function showLb(){
const c=lbList[lbIdx];if(!c)return;const b=c.querySelector('.zoom');
lbImg.src=b.dataset.src;
lbCap.innerHTML='<b>'+b.dataset.name+'</b> · score '+b.dataset.score+' · '+(lbIdx+1)+' / '+lbList.length;
loadMeta(box(c).value);}
function stepLb(d){lbIdx=(lbIdx+d+lbList.length)%lbList.length;showLb();}
function closeLb(){lb.hidden=true;lbImg.src='';metaTok++;lbMeta.innerHTML='';}
lb.addEventListener('click',e=>{if(e.target===lb)closeLb();});
addEventListener('keydown',e=>{
if(lb.hidden)return;
if(e.key==='Escape')closeLb();
else if(e.key==='ArrowLeft')stepLb(-1);
else if(e.key==='ArrowRight')stepLb(1);
else if(e.key===' '){e.preventDefault();const c=lbList[lbIdx];box(c).checked=!box(c).checked;updateCount();paint(c);}});
let toastT;
function toast(msg,cls){
toastEl.textContent=msg;toastEl.hidden=false;toastEl.className='toast show '+(cls||'');
clearTimeout(toastT);toastT=setTimeout(()=>toastEl.classList.remove('show'),3500);}
// batch action helper: run `fn` on selected paths, apply `after(card)` to each, toast, restore label
async function runAction(btn,label,url,after,msg){
const sel=selected(),paths=sel.map(c=>box(c).value);
if(!paths.length)return;
btn.disabled=true;const lbl=btn.textContent;btn.textContent=label;
try{
const j=await(await fetch(url,{method:'POST',body:JSON.stringify(paths)})).json();
sel.forEach(c=>{after(c);box(c).checked=false;paint(c);});
applyView();toast(msg(paths.length,j),'ok');
}catch(e){toast('Error: '+e,'err');}
btn.textContent=lbl;updateCount();}
const apply=()=>runAction(applyb,'Marking…','/apply',
c=>{c.dataset.tagged='1';c.dataset.sfw='0';},(n,j)=>'Tagged '+n+' NSFW');
const markSafe=()=>runAction(markSafeb,'Marking…','/mark?state=sfw',
c=>{c.dataset.sfw='1';c.dataset.tagged='0';},(n)=>'Marked '+n+' safe');
const untag=()=>runAction(untagb,'Removing…','/untag',
c=>{c.dataset.tagged='0';c.dataset.sfw='0';},(n,j)=>'Removed tag from '+(j.untagged??n));
const logpanel=$('#logpanel'),logbody=$('#logbody'),logcount=$('#logcount');
let logTimer=null,logLen=-1;
function toggleLog(){
const open=logpanel.hidden;logpanel.hidden=!open;$('#logbtn').classList.toggle('on',open);
if(open){logLen=-1;refreshLog();logTimer=setInterval(refreshLog,2000);}
else{clearInterval(logTimer);logTimer=null;}}
async function refreshLog(){
let ents;try{ents=await(await fetch('/log')).json();}catch(e){return;}
logcount.textContent=ents.length+' event'+(ents.length===1?'':'s');
if(ents.length===logLen)return;logLen=ents.length;
const atBottom=logbody.scrollHeight-logbody.scrollTop-logbody.clientHeight<40;
logbody.innerHTML=ents.length
? ents.map(e=>'<div class="logline"><span class="lt">'+esc(e.t)+'</span><span>'+esc(e.m)+'</span></div>').join('')
: '<div class="logempty">No activity yet.</div>';
if(atBottom)logbody.scrollTop=logbody.scrollHeight;}
refreshTagN();paintAll();updateCount();applyView();
scanFolder(); // scan the whole "All folders" view on load so tag state is defined at startup
</script></body></html>

119
nsfwtag/scoring.py Normal file
View File

@@ -0,0 +1,119 @@
"""Discover images and score them with the local NSFW model.
Scores are cached to a CSV next to the library, so a rerun reuses them and only
scores newly-added files — skipping the model (and its load) entirely when
nothing is new.
"""
import csv
import sys
from pathlib import Path
from . import BATCH, CSV_NAME, EXTS, MODEL_ID
def discover_images(target: Path, recursive: bool = False, limit: int = 0):
"""All supported image files under `target`, de-duplicated by resolved path
(guards against symlinks / duplicate paths). Sorted for stable ordering."""
walk = target.rglob("*") if recursive else target.iterdir()
seen, imgs = set(), []
for p in sorted(walk):
if not (p.is_file() and p.suffix.lower() in EXTS):
continue
rp = p.resolve()
if rp in seen:
continue
seen.add(rp)
imgs.append(p)
return imgs[:limit] if limit else imgs
def load_cache(csv_path: Path) -> dict:
"""Read a scores CSV into {path: score}. Missing file -> empty dict."""
cache = {}
if csv_path.exists():
with open(csv_path, newline="", encoding="utf-8") as fh:
for row in csv.DictReader(fh):
try:
cache[row["path"]] = float(row["nsfw_score"])
except (KeyError, ValueError):
pass
return cache
def _save_cache(csv_path: Path, cache: dict):
with open(csv_path, "w", newline="", encoding="utf-8") as fh:
w = csv.writer(fh)
w.writerow(["path", "nsfw_score"])
w.writerows([(pth, f"{sc:.4f}") for pth, sc in cache.items()])
def score_images(imgs, csv_path: Path, rescore: bool = False, batch: int = BATCH):
"""Return (scored, errors) where scored is [(Path, score)] for every image in
`imgs` that has a score. Reuses the CSV cache; only `imgs` missing from it are
run through the model, then the merged cache is persisted."""
cache = {} if rescore else load_cache(csv_path)
todo = [p for p in imgs if str(p) not in cache]
if cache:
print(f"cache: {len(cache)} scores from {csv_path} — {len(todo)} new to score", file=sys.stderr)
errors = 0
if todo:
import numpy as np
import torch
from PIL import Image, ImageFile
from transformers import AutoModelForImageClassification
ImageFile.LOAD_TRUNCATED_IMAGES = True
device = "mps" if torch.backends.mps.is_available() else "cpu"
# Load from local cache without hitting the HF Hub (skips the slow per-run
# update check). First run falls through to a one-time download, then cached.
try:
model = AutoModelForImageClassification.from_pretrained(MODEL_ID, local_files_only=True)
print(f"Loaded {MODEL_ID} from cache on {device}", file=sys.stderr)
except Exception:
print(f"Downloading {MODEL_ID} (one-time, ~350 MB) …", file=sys.stderr)
model = AutoModelForImageClassification.from_pretrained(MODEL_ID)
model = model.to(device).eval()
nsfw_idx = next(i for i, l in model.config.id2label.items() if l.lower() == "nsfw")
# Input size comes from the model, not a constant — AdamCodd wants 384,
# Falconsai wanted 224. Hardcoding it silently breaks on a model swap.
size = getattr(model.config, "image_size", 224)
def preprocess(img):
img = img.convert("RGB").resize((size, size), Image.BILINEAR)
a = (np.asarray(img, dtype="float32") / 255.0 - 0.5) / 0.5
return torch.from_numpy(a).permute(2, 0, 1)
print(f"Scoring {len(todo)} images …", file=sys.stderr)
done_since_save = 0
for i in range(0, len(todo), batch):
tens, paths = [], []
for p in todo[i:i + batch]:
try:
tens.append(preprocess(Image.open(p))); paths.append(p)
except Exception as e:
errors += 1; print(f"\n error {p}: {e}", file=sys.stderr)
if not tens:
continue
try:
with torch.no_grad():
probs = model(pixel_values=torch.stack(tens).to(device)).logits.softmax(-1)
for p, pr in zip(paths, probs):
cache[str(p)] = pr[nsfw_idx].item()
except Exception as e:
# One bad batch (e.g. MPS OOM / Metal error, common when two GPU
# jobs run at once) must not discard everything scored so far.
errors += len(paths)
print(f"\n batch {i}-{i+len(paths)} failed: {e}", file=sys.stderr)
done_since_save += len(paths)
if done_since_save >= 500: # periodic checkpoint → crash-resumable
_save_cache(csv_path, cache); done_since_save = 0
print(f"\r {min(i + batch, len(todo))}/{len(todo)}", end="", file=sys.stderr, flush=True)
print(file=sys.stderr)
_save_cache(csv_path, cache)
else:
print("all images already scored — skipping model load", file=sys.stderr)
scored = [(p, cache[str(p)]) for p in imgs if str(p) in cache]
return scored, errors

126
nsfwtag/server.py Normal file
View File

@@ -0,0 +1,126 @@
"""Local HTTP server for the review app (stdlib only).
Routes:
GET / the review page
GET /img?path= a thumbnail (restricted to the scanned candidate set)
POST /tagged -> which of the posted paths already carry the nsfw keyword
POST /apply -> add the nsfw keyword to the posted paths
POST /untag -> remove the nsfw keyword from the posted paths
All POST routes are restricted to the allowed candidate set.
"""
import sys
from pathlib import Path
from .exif import read_exif, read_marks, read_tagged, remove_keyword, write_keyword
from .webapp import render_page
def serve_review(cands, threshold, host="127.0.0.1"):
"""Serve the review page and tag/untag confirmed images directly on request."""
import http.server
import json
import mimetypes
import time
import webbrowser
from collections import deque
from urllib.parse import parse_qs, unquote, urlparse
allowed = {str(p) for p, _ in cands} # only these paths may be served or tagged
page = render_page(cands, threshold).encode("utf-8")
log_entries = deque(maxlen=2000) # shown in the browser's Activity log tab
def log(msg):
log_entries.append({"t": time.strftime("%H:%M:%S"), "m": msg})
print(msg, file=sys.stderr)
def label(p): # folder + picture name for the log
pp = Path(p)
return f"{pp.parent.name}/{pp.name}" if pp.parent.name else pp.name
log(f"review ready — {len(cands)} candidate images")
class H(http.server.BaseHTTPRequestHandler):
def log_message(self, *a): # quiet
pass
def _send(self, code, ctype, body):
self.send_response(code)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
u = urlparse(self.path)
if u.path == "/":
return self._send(200, "text/html; charset=utf-8", page)
if u.path == "/img":
fp = unquote(parse_qs(u.query).get("path", [""])[0])
# ponytail: browsers won't render HEIC bytes; those thumbs show broken — score/tag still work.
if fp in allowed and Path(fp).exists():
ctype = mimetypes.guess_type(fp)[0] or "application/octet-stream"
return self._send(200, ctype, Path(fp).read_bytes())
if u.path == "/exif": # metadata panel for the lightbox
fp = unquote(parse_qs(u.query).get("path", [""])[0])
if fp in allowed and Path(fp).exists():
return self._send(200, "application/json", json.dumps(read_exif(fp)).encode())
if u.path == "/log": # activity log tab
return self._send(200, "application/json", json.dumps(list(log_entries)).encode())
return self._send(404, "text/plain", b"not found")
def do_POST(self):
u = urlparse(self.path)
route = u.path
n = int(self.headers.get("Content-Length", 0))
paths = [p for p in json.loads(self.rfile.read(n) or b"[]") if p in allowed]
if route == "/tagged": # review state of these paths (lazy, chunked): nsfw + sfw
m = read_marks(paths)
return self._send(200, "application/json",
json.dumps({"nsfw": list(m["nsfw"]), "sfw": list(m["sfw"])}).encode())
if route == "/mark": # persist the manual SFW decision (opposite of nsfw)
state = parse_qs(u.query).get("state", [""])[0]
if state == "sfw": # mark safe: write 'sfw', drop 'nsfw'
for p in paths:
write_keyword(p, "sfw"); remove_keyword(p, "nsfw")
log(f"marked sfw — {label(p)}")
elif state == "clear": # un-mark safe: drop 'sfw'
for p in paths:
remove_keyword(p, "sfw")
log(f"cleared sfw — {label(p)}")
return self._send(200, "application/json", json.dumps({"ok": len(paths)}).encode())
if route == "/apply":
ok = 0
for p in paths:
if write_keyword(p):
remove_keyword(p, "sfw") # nsfw and sfw are mutually exclusive
ok += 1; log(f"tagged nsfw — {label(p)}")
return self._send(200, "application/json", json.dumps({"tagged": ok}).encode())
if route == "/untag": # clear the review keyword (nsfw or sfw) — back to un-decided
ok = 0
for p in paths:
remove_keyword(p, "nsfw"); remove_keyword(p, "sfw")
ok += 1; log(f"removed tag — {label(p)}")
return self._send(200, "application/json", json.dumps({"untagged": ok}).encode())
if route == "/toggle": # flip each path's tag both ways; other EXIF preserved
cur = read_tagged(paths) # one batched read, then add/remove per file
now_tagged = []
for p in paths:
if p in cur:
remove_keyword(p); log(f"removed nsfw — {label(p)}")
elif write_keyword(p):
remove_keyword(p, "sfw") # mutually exclusive
now_tagged.append(p); log(f"tagged nsfw — {label(p)}")
return self._send(200, "application/json", json.dumps({"tagged": now_tagged}).encode())
return self._send(404, "text/plain", b"not found")
srv = http.server.ThreadingHTTPServer((host, 0), H) # port 0 = OS picks a free one
url = f"http://{host}:{srv.server_address[1]}/"
print(f"\nReview app: {url}\n Tick images, click 'Apply' — writes the 'nsfw' EXIF tag directly (idempotent, safe to re-apply).\n Press Ctrl-C here when done.", file=sys.stderr)
webbrowser.open(url)
try:
srv.serve_forever()
except KeyboardInterrupt:
print("\nstopped.", file=sys.stderr)
finally:
srv.server_close()

91
nsfwtag/webapp.py Normal file
View File

@@ -0,0 +1,91 @@
"""Render the review page.
The HTML/CSS/JS lives in review.html (loaded once at import). This module fills
in the %%TOKENS%%: the folder-tree sidebar, the image cards, totals. The server
contract is preserved: each .nsfw checkbox's value is the image's absolute path,
thumbnails load via GET /img?path=, actions POST to /apply /untag /tagged.
"""
import html
import os
from pathlib import Path
from urllib.parse import quote
# Inline Lucide-style SVG icons (no emoji, theme-able via currentColor).
_SVG_EXPAND = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M8 3H5a2 2 0 0 0-2 2v3m18 0V5a2 2 0 0 0-2-2h-3M8 21H5a2 2 0 0 1-2-2v-3m18 0v3a2 2 0 0 1-2 2h-3"/></svg>'
_SVG_CHECK = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6 9 17l-5-5"/></svg>'
_PAGE = (Path(__file__).parent / "review.html").read_text(encoding="utf-8")
def folder_tree_html(cands):
"""Indented folder list for the sidebar file browser, built from the candidate
paths. Each row filters the grid to that folder + its subfolders (prefix match);
'All folders' resets. Counts are images directly in that folder."""
folders = {}
for p, _ in cands:
folders[str(p.parent)] = folders.get(str(p.parent), 0) + 1
if not folders:
return ""
paths = sorted(folders)
root = os.path.commonpath(paths) if len(paths) > 1 else paths[0]
rows = [
f'<button type="button" class="fitem on" data-folder="" onclick="pickFolder(this)">'
f'<span class="fname">All folders</span><span class="fcount">{sum(folders.values())}</span></button>'
]
for folder in paths:
rel = os.path.relpath(folder, root)
depth = 0 if rel == "." else rel.count(os.sep) + 1
name = os.path.basename(folder) or folder
rows.append(
f'<button type="button" class="fitem" data-folder="{html.escape(folder.lower(), quote=True)}" '
f'style="padding-left:{depth * 14 + 12}px" title="{html.escape(folder, quote=True)}" '
f'onclick="pickFolder(this)"><span class="fname">{html.escape(name)}</span>'
f'<span class="fcount">{folders[folder]}</span></button>'
)
return "".join(rows)
def render_page(cands, threshold):
"""Build the review page. cands: list of (Path, score) sorted desc."""
cards, seen = [], set()
for p, score in cands:
sp = str(p)
if sp in seen: # never list the same path twice
continue
seen.add(sp)
# No pre-selection at render. The per-folder scan decides state so persisted
# marks win: nsfw -> pink, sfw -> green, undecided high-score -> red.
# (A dismissed/sfw image must never come back pre-selected red on reload.)
checked = ""
src = f"/img?path={quote(sp)}"
esc = html.escape(sp, quote=True)
name = html.escape(p.name)
folder = html.escape(str(p.parent))
folder_attr = html.escape(str(p.parent), quote=True)
name_low = html.escape(p.name.lower(), quote=True)
folder_low = html.escape(str(p.parent).lower(), quote=True)
cards.append(
f'<div class="card" data-score="{score:.4f}" data-name="{name_low}" '
f'data-folder="{folder_low}" data-hay="{name_low} {folder_low}" '
f'data-tagged="?" data-sfw="?">'
f'<div class="thumb">'
f'<img loading="lazy" src="{src}" alt="">'
f'<span class="score" data-s="{score:.4f}">{score:.2f}</span>'
f'<button type="button" class="icon zoom" aria-label="View full size" '
f'data-src="{src}" data-name="{name}" data-score="{score:.2f}" '
f'onclick="openLb(this)">{_SVG_EXPAND}</button>'
f'<span class="tagged" hidden>{_SVG_CHECK}<span>tagged</span></span>'
f'<span class="safe" hidden>{_SVG_CHECK}<span>tagged</span></span>'
f'<button type="button" class="tick" aria-label="Mark {name} NSFW or SFW" '
f'onclick="toggleCard(this)">{_SVG_CHECK}</button>'
f'<input type="checkbox" class="nsfw" value="{esc}" {checked} tabindex="-1" aria-hidden="true">'
f'</div>'
f'<div class="info"><div class="name" title="{esc}">{name}</div>'
f'<div class="path" title="{folder_attr}">{folder}</div></div>'
f'</div>'
)
return (_PAGE
.replace("%%TREE%%", folder_tree_html(cands))
.replace("%%CARDS%%", "".join(cards))
.replace("%%TOTAL%%", str(len(cards)))
.replace("%%THRESHOLD%%", f"{threshold:.2f}"))

2454
photo_analyzer.py Normal file
View File

File diff suppressed because it is too large Load Diff

7
pyproject.toml Normal file
View File

@@ -0,0 +1,7 @@
[project]
name = "photoanalyzer"
version = "0.1.0"
requires-python = ">=3.11"
[tool.ruff]
line-length = 100

11
scripts/work-item Executable file
View File

@@ -0,0 +1,11 @@
#!/bin/sh
set -eu
for python in python3.13 python3.12 python3.11; do
if command -v "$python" >/dev/null 2>&1; then
exec "$python" -m work_item "$@"
fi
done
echo "work-item: Python 3.11 or newer is required" >&2
exit 2

122
test_dedup.py Normal file
View File

@@ -0,0 +1,122 @@
#!/usr/bin/env python3
"""Self-check for the phash/dedup/reconcile additions. Runnable, no framework:
python test_dedup.py
Creates synthetic images in a temp dir + a temp DB, asserts behaviour, cleans up.
"""
import io
import os
import sqlite3
import tempfile
from pathlib import Path
import numpy as np
from PIL import Image
import photo_analyzer as pa
def _make_photo(path: Path, seed: int, size=(800, 600)):
"""Deterministic, structured (not pure-noise) image so phash is stable."""
rng = np.random.default_rng(seed)
# low-frequency structure: a few coloured blocks + gradient — survives resize
base = np.zeros((size[1], size[0], 3), dtype=np.uint8)
for _ in range(6):
x0, y0 = rng.integers(0, size[0] - 100), rng.integers(0, size[1] - 100)
col = rng.integers(0, 256, 3)
base[y0:y0 + 150, x0:x0 + 150] = col
grad = np.linspace(0, 120, size[0], dtype=np.uint8)
base[:, :, 0] = np.clip(base[:, :, 0].astype(int) + grad[None, :], 0, 255)
Image.fromarray(base).save(path, quality=95)
def main():
tmp = Path(tempfile.mkdtemp(prefix="dedup_test_"))
db_path = tmp / "test.db"
try:
# ── synthetic library ────────────────────────────────────────────────
orig = tmp / "A.jpg"
_make_photo(orig, seed=1)
# exact byte copy (backup-of-backup)
exact = tmp / "A_backup.jpg"
exact.write_bytes(orig.read_bytes())
# near-dupe: same picture, resized + recompressed (different phone)
near = tmp / "A_phone.jpg"
with Image.open(orig) as im:
im.resize((400, 300)).save(near, quality=60)
# a genuinely different photo
other = tmp / "B.jpg"
_make_photo(other, seed=999)
# ── phash: dupes close, different far ────────────────────────────────
def dist(p1, p2):
h1, h2 = int(pa._phash_image(p1), 16), int(pa._phash_image(p2), 16)
return bin(h1 ^ h2).count("1")
d_exact = dist(orig, exact)
d_near = dist(orig, near)
d_other = dist(orig, other)
print(f"phash Hamming: exact={d_exact} near={d_near} other={d_other}")
assert d_exact == 0, f"exact copy should have phash distance 0, got {d_exact}"
assert d_near <= pa.PHASH_THRESHOLD, f"near-dupe {d_near} > threshold {pa.PHASH_THRESHOLD}"
assert d_other > pa.PHASH_THRESHOLD, f"different photo {d_other} <= threshold — false match"
# ── sha1: identical bytes match, different bytes don't ───────────────
assert pa._sha1_file(orig) == pa._sha1_file(exact), "byte-identical files must share sha1"
assert pa._sha1_file(orig) != pa._sha1_file(near), "recompressed file must differ in sha1"
# ── DB: register, hash, cluster, mark ────────────────────────────────
conn = pa.get_db(str(db_path))
photos = [orig, exact, near, other]
for p in photos:
pa.upsert_pending(conn, str(p))
n = pa.ensure_hashes(conn, photos)
assert n == 4, f"expected 4 hashed, got {n}"
# all four should carry phash + sha1 now
rows = conn.execute("SELECT path, phash, file_sha1 FROM photos").fetchall()
assert all(r["phash"] and r["file_sha1"] for r in rows), "some hashes missing"
clusters = pa.cluster_duplicates(conn, pa.PHASH_THRESHOLD)
# exactly one cluster: {orig, exact, near}; 'other' stands alone
assert len(clusters) == 1, f"expected 1 cluster, got {len(clusters)}"
cluster_paths = {it["path"] for it in clusters[0]}
assert cluster_paths == {str(orig), str(exact), str(near)}, cluster_paths
# canonical = largest file (orig, saved at q95 full-res)
assert clusters[0][0]["path"] == str(orig), "canonical should be the largest file"
marked, ncl = pa.mark_duplicates(conn, pa.PHASH_THRESHOLD)
assert (marked, ncl) == (2, 1), f"expected (2 marked, 1 cluster), got ({marked}, {ncl})"
dupes = conn.execute(
"SELECT path, dup_of FROM photos WHERE status='duplicate'"
).fetchall()
assert {r["path"] for r in dupes} == {str(exact), str(near)}
assert all(r["dup_of"] == str(orig) for r in dupes), "dup_of must point at canonical"
# get_pending must exclude duplicates
pending = pa.get_pending(conn, reanalyze=True)
assert str(exact) not in pending and str(near) not in pending, "dupes leaked into pending"
assert str(orig) in pending and str(other) in pending
# ── reconcile_moved: rename a file, path follows by sha1 ─────────────
moved_dst = tmp / "B_renamed.jpg"
os.rename(other, moved_dst) # 'other' now missing at old path
n_moved = pa.reconcile_moved(conn, tmp)
assert n_moved == 1, f"expected 1 reconciled, got {n_moved}"
row = conn.execute("SELECT path FROM photos WHERE path=?", (str(moved_dst),)).fetchone()
assert row is not None, "moved file's row was not repathed"
gone = conn.execute("SELECT 1 FROM photos WHERE path=?", (str(other),)).fetchone()
assert gone is None, "old path should no longer exist in DB"
conn.close()
print("ALL DEDUP/RECONCILE CHECKS PASSED")
finally:
import shutil
shutil.rmtree(tmp, ignore_errors=True)
if __name__ == "__main__":
main()

15
test_nsfw_skip.py Normal file
View File

@@ -0,0 +1,15 @@
"""Self-check for photo_analyzer's nsfw-skip parsing. Run: python test_nsfw_skip.py"""
from photo_analyzer import _rec_has_nsfw
def test():
assert _rec_has_nsfw({"Keywords": ["beach", "nsfw"]}) # list, present
assert _rec_has_nsfw({"Subject": "nsfw"}) # scalar in Subject
assert _rec_has_nsfw({"Keywords": "NSFW"}) # case-insensitive
assert _rec_has_nsfw({"Keywords": [" nsfw "]}) # whitespace
assert not _rec_has_nsfw({"Keywords": ["sfw", "beach"]}) # sfw only
assert not _rec_has_nsfw({"Keywords": ["nsfw-ish"]}) # substring != tag
assert not _rec_has_nsfw({}) # no fields
print("ok")
if __name__ == "__main__":
test()

221
tests/test_cli_e2e.py Normal file
View File

@@ -0,0 +1,221 @@
from __future__ import annotations
import json
import os
import shutil
import subprocess
import sys
import tempfile
import textwrap
import unittest
from pathlib import Path
SOURCE_ROOT = Path(__file__).resolve().parents[1]
def run(args, *, cwd: Path, env=None, check=True):
result = subprocess.run(args, cwd=cwd, env=env, text=True, capture_output=True)
if check and result.returncode:
raise AssertionError(f"{args}\nstdout={result.stdout}\nstderr={result.stderr}")
return result
FAKE_TEA = r'''#!/usr/bin/env python3
import json, os, subprocess, sys
from pathlib import Path
state_path = Path(os.environ["FAKE_TEA_STATE"])
state = json.loads(state_path.read_text())
args = sys.argv[1:]
def save():
state_path.write_text(json.dumps(state, indent=2, sort_keys=True) + "\n")
def out(value):
print(json.dumps(value))
if args[0] == "api":
endpoint = args[1]
if endpoint.endswith("/issues?state=open&type=issues&limit=100"):
out([issue for issue in state["issues"] if issue["state"] == "open"])
elif "/dependencies" in endpoint:
issue = int(endpoint.split("/issues/")[1].split("/")[0])
blockers = state["dependencies"].get(str(issue), [])
out([next(x for x in state["issues"] if x["number"] == number) for number in blockers])
elif "/pulls/" in endpoint:
number = int(endpoint.rsplit("/", 1)[1])
out(state["prs"][str(number)])
elif "/commits/" in endpoint and endpoint.endswith("/status"):
out({"state": "success"})
else:
raise SystemExit(f"unsupported api: {endpoint}")
elif args[:2] == ["issues", "edit"]:
issue = next(x for x in state["issues"] if x["number"] == int(args[2]))
if "--add-assignees" in args:
issue["assignees"] = [{"login": args[args.index("--add-assignees") + 1]}]
labels = {x["name"] for x in issue.get("labels", [])}
if "--remove-labels" in args:
labels -= set(args[args.index("--remove-labels") + 1].split(","))
if "--add-labels" in args:
labels |= set(args[args.index("--add-labels") + 1].split(","))
issue["labels"] = [{"name": x} for x in sorted(labels)]
save(); out(issue)
elif args[:2] == ["comments", "add"]:
state["comments"].append({"issue": int(args[2]), "body": args[3]})
save(); out(state["comments"][-1])
elif args[:2] == ["pulls", "create"]:
number = len(state["prs"]) + 1
branch = args[args.index("--head") + 1]
sha = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
pr = {"number": number, "index": number, "url": f"https://example.test/pr/{number}",
"merged": False, "head": {"sha": sha, "ref": branch}}
state["prs"][str(number)] = pr
save(); out(pr)
elif args[:2] == ["pulls", "merge"]:
number = int(args[2]); pr = state["prs"][str(number)]
subprocess.check_call(["git", "push", "origin", f"{pr['head']['ref']}:main"])
pr["merged"] = True
save(); out(pr)
elif args[:2] == ["issues", "close"]:
issue = next(x for x in state["issues"] if x["number"] == int(args[2]))
issue["state"] = "closed"; save(); out(issue)
else:
raise SystemExit("unsupported tea command: " + repr(args))
'''
class CliEndToEndTests(unittest.TestCase):
def setUp(self) -> None:
self.temp = tempfile.TemporaryDirectory()
self.base = Path(self.temp.name)
remote_parent = self.base / "domverse"
remote_parent.mkdir()
seed = self.base / "seed"
seed.mkdir()
run(["git", "init", "-b", "main"], cwd=seed)
run(["git", "config", "user.name", "Test Agent"], cwd=seed)
run(["git", "config", "user.email", "agent@example.test"], cwd=seed)
(seed / "README.md").write_text("seed\n")
run(["git", "add", "README.md"], cwd=seed)
run(["git", "commit", "-m", "seed"], cwd=seed)
self.remote = remote_parent / "photoanalyzer.git"
run(["git", "clone", "--bare", str(seed), str(self.remote)], cwd=self.base)
self.repo = self.base / "work"
run(["git", "clone", str(self.remote), str(self.repo)], cwd=self.base)
run(["git", "config", "user.name", "Test Agent"], cwd=self.repo)
run(["git", "config", "user.email", "agent@example.test"], cwd=self.repo)
config = textwrap.dedent(f'''\
[repository]
slug = "domverse/photoanalyzer"
login = "fake"
assignee = "agent"
remote = "origin"
main_branch = "main"
[workflow]
branch_prefix = "us"
require_ci = true
required_tests = []
[safety]
max_file_bytes = 100000
allow = ["tests/fixtures/**"]
deny = ["_IGNORE/**", "**/_IGNORE/**", "pictures/**", "*.env", "*.jpg"]
''')
(self.repo / ".work-item.toml").write_text(config)
run(["git", "add", ".work-item.toml"], cwd=self.repo)
run(["git", "commit", "-m", "config"], cwd=self.repo)
run(["git", "push", "origin", "main"], cwd=self.repo)
bin_dir = self.base / "bin"
bin_dir.mkdir()
tea = bin_dir / "tea"
tea.write_text(FAKE_TEA)
tea.chmod(0o755)
self.state_path = self.base / "tea-state.json"
self.state_path.write_text(json.dumps({
"issues": [{
"number": 1,
"title": "US01-01 — Implement Safe Workflow",
"state": "open",
"labels": [{"name": "status/backlog"}],
"assignees": [],
}],
"dependencies": {"1": []},
"comments": [],
"prs": {},
}))
self.env = os.environ.copy()
self.env["PATH"] = f"{bin_dir}{os.pathsep}{self.env['PATH']}"
self.env["FAKE_TEA_STATE"] = str(self.state_path)
self.env["PYTHONPATH"] = str(SOURCE_ROOT)
def tearDown(self) -> None:
self.temp.cleanup()
def cli(self, *args, check=True):
return run([sys.executable, "-m", "work_item", *args], cwd=self.repo, env=self.env, check=check)
def state(self):
return json.loads(self.state_path.read_text())
def test_claim_submit_merge_complete_lifecycle(self) -> None:
next_result = self.cli("next")
self.assertEqual(json.loads(next_result.stdout)["story_id"], "US01-01")
claim = json.loads(self.cli("claim").stdout)
self.assertEqual(claim["status"], "in-progress")
self.assertEqual(run(["git", "branch", "--show-current"], cwd=self.repo).stdout.strip(), claim["branch"])
issue = self.state()["issues"][0]
self.assertEqual(issue["assignees"], [{"login": "agent"}])
self.assertIn({"name": "status/in-progress"}, issue["labels"])
(self.repo / "private.env").write_text("not-a-real-secret\n")
unsafe = self.cli("submit", "--test", "true", "--yes", check=False)
self.assertEqual(unsafe.returncode, 2)
self.assertIn("denied path", unsafe.stderr)
(self.repo / "private.env").unlink()
(self.repo / "feature.txt").write_text("implemented\n")
review_only = self.cli("submit", "--test", "true", check=False)
self.assertEqual(review_only.returncode, 2)
self.assertIn("rerun with --yes", review_only.stderr)
self.assertEqual(run(["git", "status", "--porcelain"], cwd=self.repo).stdout.strip(), "?? feature.txt")
submitted = json.loads(self.cli("submit", "--test", "true", "--yes").stdout)
self.assertEqual(submitted["status"], "review")
self.assertEqual(submitted["pr"], 1)
self.assertIn("status/review", {x["name"] for x in self.state()["issues"][0]["labels"]})
not_merged = self.cli("complete", check=False)
self.assertEqual(not_merged.returncode, 2)
self.assertIn("not merged", not_merged.stderr)
completed = json.loads(self.cli("complete", "--merge").stdout)
self.assertEqual(completed["status"], "done")
self.assertEqual(run(["git", "branch", "--show-current"], cwd=self.repo).stdout.strip(), "main")
self.assertEqual(run(["git", "status", "--porcelain"], cwd=self.repo).stdout.strip(), "")
issue = self.state()["issues"][0]
self.assertEqual(issue["state"], "closed")
self.assertIn("status/done", {x["name"] for x in issue["labels"]})
self.assertFalse((self.repo / ".git" / "work-item-state.json").exists())
def test_dirty_tree_prevents_claim(self) -> None:
(self.repo / "unexpected.txt").write_text("dirty\n")
result = self.cli("claim", check=False)
self.assertEqual(result.returncode, 2)
self.assertIn("clean", result.stderr)
self.assertEqual(run(["git", "branch", "--show-current"], cwd=self.repo).stdout.strip(), "main")
def test_block_requires_reason_and_records_state(self) -> None:
self.cli("claim")
result = self.cli("block", "--reason", "Provider fixture unavailable")
payload = json.loads(result.stdout)
self.assertEqual(payload["status"], "blocked")
issue = self.state()["issues"][0]
self.assertIn("status/blocked", {x["name"] for x in issue["labels"]})
self.assertIn("Provider fixture unavailable", self.state()["comments"][-1]["body"])
if __name__ == "__main__":
unittest.main()

178
tests/test_core.py Normal file
View File

@@ -0,0 +1,178 @@
from __future__ import annotations
import json
import subprocess
import tempfile
import unittest
from pathlib import Path
from unittest.mock import Mock
from work_item.core import Config, GitRepo, Story, WorkItemError, Workflow
def run(*args: str, cwd: Path) -> str:
result = subprocess.run(args, cwd=cwd, text=True, capture_output=True, check=True)
return result.stdout.strip()
def config_for(slug: str, **changes) -> Config:
values = dict(
repo_slug=slug,
login="test",
assignee="agent",
remote="origin",
main_branch="main",
branch_prefix="us",
require_ci=False,
max_file_bytes=64,
required_tests=(),
denied_patterns=("_IGNORE/**", "**/_IGNORE/**", "pictures/**", "*.env", "*.jpg"),
allowed_patterns=("tests/fixtures/**",),
)
values.update(changes)
return Config(**values)
class RepositoryFixture(unittest.TestCase):
def setUp(self) -> None:
self.temp = tempfile.TemporaryDirectory()
self.base = Path(self.temp.name)
remote_parent = self.base / "domverse"
remote_parent.mkdir()
self.seed = self.base / "seed"
self.seed.mkdir()
run("git", "init", "-b", "main", cwd=self.seed)
run("git", "config", "user.name", "Test Agent", cwd=self.seed)
run("git", "config", "user.email", "agent@example.test", cwd=self.seed)
(self.seed / "README.md").write_text("seed\n", encoding="utf-8")
run("git", "add", "README.md", cwd=self.seed)
run("git", "commit", "-m", "seed", cwd=self.seed)
self.remote = remote_parent / "photoanalyzer.git"
run("git", "clone", "--bare", str(self.seed), str(self.remote), cwd=self.base)
self.repo = self.base / "work"
run("git", "clone", str(self.remote), str(self.repo), cwd=self.base)
run("git", "config", "user.name", "Test Agent", cwd=self.repo)
run("git", "config", "user.email", "agent@example.test", cwd=self.repo)
self.config = config_for("domverse/photoanalyzer")
def tearDown(self) -> None:
self.temp.cleanup()
class StoryTests(unittest.TestCase):
def test_parse_and_branch_slug(self) -> None:
story = Story.from_issue({
"number": 12,
"title": "US02-05 — Build the Static Application Shell",
"labels": [{"name": "type/feature"}],
"assignees": [{"login": "agent"}],
})
self.assertIsNotNone(story)
assert story is not None
self.assertEqual((story.epic, story.sequence), (2, 5))
self.assertEqual(story.branch_slug, "build-the-static-application-shell")
self.assertEqual(story.assignees, ("agent",))
def test_non_story_issue_is_ignored(self) -> None:
self.assertIsNone(Story.from_issue({"number": 1, "title": "Maintenance"}))
class GitSafetyTests(RepositoryFixture):
def test_verify_remote_and_clean_tree(self) -> None:
git = GitRepo(self.repo, self.config, Mock(wraps=None))
# Use the real runner; assigning after construction keeps the test explicit.
from work_item.core import Runner
git.runner = Runner()
git.verify()
git.ensure_clean()
(self.repo / "change.txt").write_text("change\n", encoding="utf-8")
with self.assertRaisesRegex(WorkItemError, "clean"):
git.ensure_clean()
def test_rejects_private_paths_large_files_and_secrets(self) -> None:
from work_item.core import Runner
git = GitRepo(self.repo, self.config, Runner())
(self.repo / "pictures").mkdir()
(self.repo / "pictures" / "private.jpg").write_bytes(b"x")
with self.assertRaisesRegex(WorkItemError, "denied path"):
git.assert_safe_changes()
(self.repo / "pictures" / "private.jpg").unlink()
(self.repo / "pictures").rmdir()
(self.repo / "large.txt").write_text("x" * 65, encoding="utf-8")
with self.assertRaisesRegex(WorkItemError, "exceeds"):
git.assert_safe_changes()
(self.repo / "large.txt").unlink()
(self.repo / "secret.txt").write_text("api_key = abcdefghijklmnopqrstuvwxyz\n", encoding="utf-8")
with self.assertRaisesRegex(WorkItemError, "possible secret"):
git.assert_safe_changes()
def test_fixture_allowlist_overrides_image_deny(self) -> None:
from work_item.core import Runner
git = GitRepo(self.repo, self.config, Runner())
fixture = self.repo / "tests" / "fixtures" / "synthetic.jpg"
fixture.parent.mkdir(parents=True)
fixture.write_bytes(b"synthetic")
self.assertEqual(git.assert_safe_changes(), [Path("tests/fixtures/synthetic.jpg")])
def test_remote_mismatch_is_rejected(self) -> None:
from work_item.core import Runner
git = GitRepo(self.repo, config_for("someone/else"), Runner())
with self.assertRaisesRegex(WorkItemError, "does not match"):
git.verify()
class SelectionTests(unittest.TestCase):
def make_workflow(self) -> Workflow:
workflow = object.__new__(Workflow)
workflow.gitea = Mock()
return workflow
def test_next_story_uses_numeric_story_order(self) -> None:
from work_item.core import Gitea
gitea = object.__new__(Gitea)
gitea.open_stories = Mock(return_value=[
Story(2, "US01-02", "Second", 1, 2, frozenset(), (), {}),
Story(1, "US01-01", "First", 1, 1, frozenset(), (), {}),
])
gitea.dependencies_closed = Mock(return_value=True)
# open_stories normally sorts; make the contract explicit here too.
gitea.open_stories.return_value.sort(key=lambda x: (x.epic, x.sequence))
self.assertEqual(gitea.next_story().story_id, "US01-01")
def test_next_story_skips_claimed_blocked_and_open_dependencies(self) -> None:
from work_item.core import Gitea
gitea = object.__new__(Gitea)
gitea.open_stories = Mock(return_value=[
Story(1, "US01-01", "Busy", 1, 1, frozenset({"status/in-progress"}), (), {}),
Story(2, "US01-02", "Blocked", 1, 2, frozenset({"status/blocked"}), (), {}),
Story(3, "US01-03", "Dependency", 1, 3, frozenset(), (), {}),
Story(4, "US01-04", "Ready", 1, 4, frozenset(), (), {}),
])
gitea.dependencies_closed = Mock(side_effect=lambda issue: issue == 4)
self.assertEqual(gitea.next_story().number, 4)
def test_no_eligible_story_fails(self) -> None:
from work_item.core import Gitea
gitea = object.__new__(Gitea)
gitea.open_stories = Mock(return_value=[])
with self.assertRaisesRegex(WorkItemError, "No eligible"):
gitea.next_story()
class StateTests(RepositoryFixture):
def test_state_is_stored_inside_git_directory(self) -> None:
workflow = Workflow(self.repo, self.config)
workflow.save_state({"issue": 1})
self.assertTrue(workflow.state_path.is_file())
self.assertIn(".git", workflow.state_path.parts)
self.assertEqual(json.loads(workflow.state_path.read_text()), {"issue": 1})
self.assertEqual(run("git", "status", "--porcelain", cwd=self.repo), "")
workflow.clear_state()
self.assertFalse(workflow.state_path.exists())
if __name__ == "__main__":
unittest.main()

62
webapp/README.md Normal file
View File

@@ -0,0 +1,62 @@
# webapp — Photo Analyzer web UI
A local, browser-based UI for the `photo_analyzer.py` pipeline. Browse and
full-text-search the analyzed library, watch analysis runs live, and start/stop
runs — reusing the `nsfwtag` review-app design (dark OLED, stdlib HTTP server, one
token-injected HTML page). Nothing leaves the machine.
```bash
python -m webapp # DB + library default from photo_analyzer.env
python -m webapp --db photo_analysis.db --library pictures/
python -m webapp --no-open # don't auto-open a browser
```
See `../WEBAPP_CONCEPT.md` for the full UX concept and rationale.
## Views
- **Library** — FTS5 search over description/tags/mood/location + facet filters
(setting, time of day, season, people, year range, has-place, status), a folder
sidebar (album = leaf folder), an infinite-scroll grid, and a lightbox with an
**Analysis** panel (all DB fields) and a raw **EXIF** panel (incl. Google Maps
link for geotagged photos).
- **Analyze** — run controls mapping every CLI flag (dry-run / reanalyze / no-exif /
exif-only / group-variants), a live overall progress bar, per-album progress, and a
recent feed. Start/Stop drive `photo_analyzer.py` as a subprocess; Stop is the
analyzer's own SIGINT drain. Balance / quota-check buttons. A **Duplicates &
maintenance** row runs the content-hash actions (backfill hashes / find duplicates /
mark duplicates) — no API calls; output streams to the Activity log and marked
duplicates appear under the `duplicate` status filter. (Move/rename reconciliation
and perceptual hashing also run automatically at the start of every normal run.)
- **Stats** — KPI cards + CSS breakdown bars (setting/time/season/people, by-year
histogram, top tags) + an error list. No chart library.
## Architecture
- `server.py` — stdlib `ThreadingHTTPServer` on `127.0.0.1:<free-port>`; routes below.
- `query.py` — read-only DB access (FTS5 search, facets, stats, one-photo).
- `runner.py` — drives `photo_analyzer.py` as a subprocess; DB-derived progress.
- `page.py` / `analyzer.html` — the single frontend page (inherits nsfwtag's tokens).
- SQLite stays the source of truth; the server only reads it (+ a `busy_timeout` so a
concurrent analysis write never errors a read). `/img` and `/exif` paths are
validated against the DB — the server can't be pointed at arbitrary files.
| Route | Purpose |
|-------|---------|
| GET `/` | the page |
| GET `/search?q=&setting=&tod=&season=&people=&year_min=&year_max=&has_location=&status=&album=&sort=&offset=` | paged results (FTS5 when `q`) |
| GET `/facets` · `/stats` | filter options · aggregates |
| GET `/photo?path=` · `/exif?path=` · `/img?path=` | one record · raw EXIF · thumbnail |
| GET `/progress` · `/log` | live run state · activity log |
| POST `/run` · `/stop` | start / stop a run — body `{library, ...flags}` or `{library, action}` where action ∈ backfill-phash \| list-dupes \| dedupe |
| GET `/balance` · `/quota-check` | provider balance · one-shot quota probe |
## Known ceilings (ponytail)
- `/img` serves **full-resolution originals**, not resized thumbnails — a 120-card
page can move a lot of bytes. Fine locally; add a Pillow thumbnail+cache endpoint if
the grid feels slow. (Same limitation as nsfwtag.)
- HEIC files won't render in the browser (broken thumb); metadata/search still work.
- The Analyze **tokens/ETA** readout is omitted — those live only in the terminal
dashboard's memory; the subprocess model surfaces DB-derived progress instead. Parse
run stdout for token counts if you want them back.

11
webapp/__init__.py Normal file
View File

@@ -0,0 +1,11 @@
"""webapp — a local web UI for photo_analyzer.
Browse and full-text-search the analyzed photo library, watch analysis runs live,
and start/stop runs — all in the browser, reusing the nsfwtag review-app design.
Nothing leaves the machine (stdlib HTTP server bound to 127.0.0.1).
Entry point: python -m webapp [--db photo_analysis.db] [--library pictures/]
"""
__version__ = "0.1.0"
PAGE_SIZE = 120 # results per /search page (infinite scroll appends)

42
webapp/__main__.py Normal file
View File

@@ -0,0 +1,42 @@
"""python -m webapp — launch the Photo Analyzer web UI.
Defaults come from photo_analyzer's env (photo_analyzer.env: DB, LIBRARY), so in
the common case you can just run `python -m webapp`.
"""
import argparse
import os
import sys
from pathlib import Path
from .server import serve
DEFAULT_DB = "photo_analysis.db"
def _env(name, fallback=None):
v = os.environ.get(name)
return v if v not in (None, "") else fallback
def main():
# Reuse photo_analyzer's .env loader so DB/LIBRARY defaults match the CLI.
try:
import photo_analyzer as pa
pa.load_env_file()
except Exception:
pass
ap = argparse.ArgumentParser(description="Web UI for the photo_analyzer pipeline.")
ap.add_argument("--db", default=_env("DB", DEFAULT_DB), help=f"SQLite DB (default {DEFAULT_DB})")
ap.add_argument("--library", default=_env("LIBRARY"), help="Photo library root (for album grouping + run default)")
ap.add_argument("--no-open", action="store_true", help="Don't open a browser")
args = ap.parse_args()
if not Path(args.db).exists():
sys.exit(f"DB not found: {args.db} — run photo_analyzer first, or pass --db.")
serve(args.db, args.library, open_browser=not args.no_open)
if __name__ == "__main__":
main()

490
webapp/analyzer.html Normal file
View File

@@ -0,0 +1,490 @@
<!doctype html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Photo Analyzer · %%TOTAL%% photos</title>
<style>
*{box-sizing:border-box}
:root{
--bg:#0b0d10;--surface:#14171c;--surface-2:#1b1f26;--elev:#20252e;
--border:#262b34;--border-2:#323947;--text:#e8eaee;--muted:#9aa1ae;--faint:#6b727f;
--primary:#10b981;--primary-2:#0ea372;--primary-ink:#04130c;--danger:#ef4444;
--warn:#f59e0b;--ring:#3b82f6;--pink:#e05fa8;
--r:12px;--r-sm:8px}
@media (prefers-reduced-motion:reduce){*{transition:none!important;animation:none!important}}
html,body{margin:0}
body{background:var(--bg);color:var(--text);font-size:14px;line-height:1.5;-webkit-font-smoothing:antialiased;
font-family:'Inter',-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Helvetica,Arial,sans-serif}
svg{width:1em;height:1em;display:block}
button{font:inherit;color:inherit;cursor:pointer}
a{color:var(--ring)}
:focus-visible{outline:2px solid var(--ring);outline-offset:2px}
.tabnum{font-variant-numeric:tabular-nums}
/* ── header ── */
.app{position:sticky;top:0;z-index:30;display:flex;align-items:center;gap:14px;min-height:58px;
padding:11px 18px;background:rgba(11,13,16,.86);backdrop-filter:blur(12px);border-bottom:1px solid var(--border)}
.brand{display:flex;align-items:center;gap:11px}
.logo{display:grid;place-items:center;width:34px;height:34px;border-radius:10px;font-size:18px;
background:linear-gradient(135deg,#1e3a5f,#10b981);color:#fff}
.titles h1{margin:0;font-size:15px;font-weight:600;letter-spacing:-.01em}
.titles p{margin:0;font-size:11px;color:var(--faint)}
.grow{flex:1}
.metrics{display:flex;gap:8px}
.metric{display:flex;flex-direction:column;align-items:center;min-width:70px;padding:5px 12px;
background:var(--surface);border:1px solid var(--border);border-radius:var(--r-sm)}
.metric .num{font-size:16px;font-weight:600;font-variant-numeric:tabular-nums}
.metric .lbl{font-size:10px;color:var(--muted);text-transform:uppercase;letter-spacing:.05em}
.metric.err .num{color:var(--danger)}
/* ── segmented view switcher ── */
.seg{display:inline-flex;padding:3px;background:var(--surface);border:1px solid var(--border);border-radius:var(--r-sm)}
.seg button{border:0;background:none;color:var(--muted);padding:6px 14px;border-radius:6px;font-size:13px;font-weight:500}
.seg button.on{background:var(--elev);color:var(--text)}
.seg button:hover:not(.on){color:var(--text)}
.navtoggle{display:grid;place-items:center;width:36px;height:36px;font-size:18px;border-radius:var(--r-sm);
background:var(--surface);border:1px solid var(--border);color:var(--muted)}
.navtoggle:hover{background:var(--elev);color:var(--text)}
.navtoggle.on{color:var(--primary);border-color:var(--border-2)}
/* ── toolbar ── */
.toolbar{position:sticky;top:58px;z-index:20;display:flex;flex-wrap:wrap;align-items:center;gap:10px;
padding:10px 18px;background:rgba(16,19,24,.92);backdrop-filter:blur(8px);border-bottom:1px solid var(--border)}
.field{display:flex;align-items:center;gap:8px;height:36px;padding:0 10px;background:var(--surface);
border:1px solid var(--border);border-radius:var(--r-sm)}
.field svg{color:var(--faint);font-size:15px}
.field input{background:none;border:0;outline:none;color:var(--text);height:100%;width:230px}
.field input::placeholder{color:var(--faint)}
.field:focus-within{border-color:var(--border-2);box-shadow:0 0 0 3px rgba(59,130,246,.22)}
.ctl{display:flex;align-items:center;gap:6px;color:var(--muted);font-size:12.5px;white-space:nowrap}
select{height:36px;padding:0 8px;background:var(--surface);color:var(--text);border:1px solid var(--border);border-radius:var(--r-sm);outline:none}
select:focus{border-color:var(--border-2);box-shadow:0 0 0 3px rgba(59,130,246,.22)}
input[type=number]{width:64px;height:36px;padding:0 8px;background:var(--surface);color:var(--text);
border:1px solid var(--border);border-radius:var(--r-sm);outline:none;font-variant-numeric:tabular-nums}
input[type=checkbox]{accent-color:var(--primary);width:16px;height:16px}
.btn{display:inline-flex;align-items:center;gap:7px;height:36px;padding:0 14px;border-radius:var(--r-sm);
border:1px solid var(--border);background:var(--surface);color:var(--text);font-weight:500;
transition:background .15s,border-color .15s,transform .05s}
.btn svg{font-size:15px}
.btn.ghost:hover{background:var(--elev);border-color:var(--border-2)}
.btn:active{transform:translateY(1px)}
.btn.primary{background:var(--primary);border-color:transparent;color:var(--primary-ink);font-weight:600}
.btn.primary:hover{background:var(--primary-2)}
.btn.danger{border-color:var(--danger);color:#fca5a5}
.btn.danger:hover{background:rgba(239,68,68,.14)}
.btn:disabled{opacity:.45;cursor:not-allowed}
.count{color:var(--faint);font-size:12.5px;font-variant-numeric:tabular-nums;white-space:nowrap}
/* ── layout shell ── */
.view{display:none}
.view.on{display:block}
.shell{display:flex;align-items:flex-start}
.sidebar{position:sticky;top:104px;align-self:flex-start;flex:0 0 250px;width:250px;
height:calc(100dvh - 104px);overflow:auto;border-right:1px solid var(--border);padding:12px 8px}
.shell.collapsed .sidebar{display:none}
.fitem{display:flex;justify-content:space-between;gap:8px;width:100%;text-align:left;border:0;background:none;
color:var(--muted);padding:7px 10px;border-radius:var(--r-sm);font-size:12.5px}
.fitem:hover{background:var(--surface);color:var(--text)}
.fitem.on{background:var(--elev);color:var(--text)}
.fitem .cn{color:var(--faint);font-variant-numeric:tabular-nums}
.main{flex:1;min-width:0}
/* ── grid + cards ── */
.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(190px,1fr));gap:12px;padding:16px 18px 80px}
.card{position:relative;overflow:hidden;background:var(--surface);border:2px solid var(--border);border-radius:var(--r);
cursor:pointer;transition:border-color .15s,transform .12s,box-shadow .15s}
.card:hover{transform:translateY(-2px);box-shadow:0 8px 24px rgba(0,0,0,.35)}
.card.err{border-color:var(--danger)}
.thumb{position:relative;aspect-ratio:1/1;background:#000}
.thumb img{width:100%;height:100%;object-fit:cover;display:block}
.badges{position:absolute;left:8px;top:8px;display:flex;gap:6px}
.badge{padding:2px 7px;border-radius:6px;background:rgba(6,8,12,.72);color:#fff;font-size:11.5px;
font-weight:700;font-variant-numeric:tabular-nums;backdrop-filter:blur(3px)}
.badge.warnb{background:var(--warn);color:#241a02}
.dot{position:absolute;right:8px;top:8px;width:10px;height:10px;border-radius:50%;box-shadow:0 0 0 2px rgba(0,0,0,.4)}
.dot.err{background:var(--danger)}.dot.pending{background:var(--faint)}
.info{padding:8px 10px}
.desc{font-size:12px;color:var(--text);display:-webkit-box;-webkit-line-clamp:2;-webkit-box-orient:vertical;overflow:hidden}
.pathline{font-size:11px;color:var(--faint);margin-top:3px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.empty{padding:80px 20px;text-align:center;color:var(--muted)}
.empty h3{color:var(--text);margin:0 0 6px}
/* ── progress bar ── */
.bar{position:relative;height:8px;border-radius:5px;background:var(--border-2);overflow:hidden}
.bar>span{position:absolute;inset:0 auto 0 0;background:var(--primary);border-radius:5px;transition:width .3s}
.bar.run>span{background:var(--warn)}
/* ── analyze view ── */
.panel{margin:14px 18px;background:var(--surface);border:1px solid var(--border);border-radius:var(--r);padding:16px}
.panel h2{margin:0 0 12px;font-size:13px;font-weight:600;color:var(--muted);text-transform:uppercase;letter-spacing:.05em}
.runbar{display:flex;flex-wrap:wrap;align-items:center;gap:14px}
.runbar label{display:flex;align-items:center;gap:6px;color:var(--muted);font-size:12.5px}
.overall .big{display:flex;align-items:baseline;gap:12px;margin-bottom:8px}
.overall .pct{font-size:26px;font-weight:700;font-variant-numeric:tabular-nums}
.overall .sub{color:var(--muted);font-size:12.5px;font-variant-numeric:tabular-nums}
.cols{display:grid;grid-template-columns:1fr 1fr;gap:14px}
@media(max-width:820px){.cols{grid-template-columns:1fr}}
.albrow{display:grid;grid-template-columns:1fr auto 70px;align-items:center;gap:10px;padding:5px 0;font-size:12.5px}
.albrow .nm{white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.albrow .ct{color:var(--faint);font-variant-numeric:tabular-nums}
.feed{max-height:420px;overflow:auto}
.feedline{display:flex;gap:9px;padding:7px 0;border-bottom:1px solid var(--border);font-size:12.5px}
.feedline .ic{flex:0 0 auto}.feedline .ok{color:var(--primary)}.feedline .bad{color:var(--danger)}
.feedline .fn{font-weight:600}.feedline .dz{color:var(--muted)}
/* ── stats view ── */
.kpis{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;padding:16px 18px 4px}
.kpi{background:var(--surface);border:1px solid var(--border);border-radius:var(--r);padding:14px 16px}
.kpi .num{font-size:24px;font-weight:700;font-variant-numeric:tabular-nums}
.kpi .lbl{color:var(--muted);font-size:12px;margin-top:2px}
.statbar{display:grid;grid-template-columns:120px 1fr 64px;align-items:center;gap:10px;padding:4px 0;font-size:12.5px}
.statbar .lab{color:var(--muted);white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.statbar .val{text-align:right;color:var(--faint);font-variant-numeric:tabular-nums}
.statbar .track{height:14px;border-radius:4px;background:var(--surface-2);overflow:hidden}
.statbar .track>span{display:block;height:100%;background:var(--primary);opacity:.85}
.years{display:flex;align-items:flex-end;gap:3px;height:90px;padding-top:8px}
.years .yb{flex:1;min-width:4px;background:var(--primary);opacity:.8;border-radius:2px 2px 0 0}
.years .yb:hover{opacity:1}
.tags{display:flex;flex-wrap:wrap;gap:7px}
.tag{padding:4px 10px;border-radius:16px;border:1px solid var(--border);background:var(--surface);
color:var(--muted);font-size:12px;cursor:pointer}
.tag:hover{color:var(--text);border-color:var(--border-2)}
.tag .cn{color:var(--faint)}
.errline{display:flex;justify-content:space-between;gap:10px;padding:7px 0;border-bottom:1px solid var(--border);font-size:12.5px}
.errline .msg{color:#fca5a5}
/* ── lightbox ── */
.lb{position:fixed;inset:0;z-index:50;display:none;background:rgba(6,8,11,.9);backdrop-filter:blur(4px)}
.lb.on{display:flex}
.lb .stage{flex:1;display:flex;align-items:center;justify-content:center;padding:24px;min-width:0}
.lb img{max-width:100%;max-height:100%;object-fit:contain;border-radius:var(--r)}
.lb .side{flex:0 0 340px;max-width:44vw;background:var(--surface);border-left:1px solid var(--border);
padding:18px;overflow:auto}
.lb .side h3{margin:0 0 4px;font-size:14px}
.lb .side .fn2{color:var(--faint);font-size:12px;margin-bottom:14px;word-break:break-all}
.kv{margin:0 0 14px}
.kv dt{color:var(--faint);font-size:10.5px;text-transform:uppercase;letter-spacing:.05em;margin-top:8px}
.kv dd{margin:1px 0 0;font-size:13px}
.lbbtns{position:absolute;top:16px;right:16px;display:flex;gap:8px}
.icon{display:grid;place-items:center;width:36px;height:36px;border:0;border-radius:8px;color:#fff;
background:rgba(6,8,12,.62);backdrop-filter:blur(3px)}
.icon:hover{background:rgba(6,8,12,.9)}
.lbnav{position:absolute;top:50%;transform:translateY(-50%)}
.lbnav.prev{left:16px}.lbnav.next{right:calc(340px + 16px)}
@media(max-width:700px){.lb .side{display:none}.lbnav.next{right:16px}}
/* ── log panel + toast ── */
.logpanel{position:fixed;top:0;right:0;bottom:0;width:380px;max-width:92vw;z-index:60;background:var(--surface);
border-left:1px solid var(--border);display:flex;flex-direction:column;transform:translateX(100%);transition:transform .2s}
.logpanel:not([hidden]){transform:none}
.loghead{display:flex;align-items:center;justify-content:space-between;padding:14px 16px;border-bottom:1px solid var(--border)}
.logbody{flex:1;overflow:auto;padding:8px 16px;font-size:12.5px}
.logline{display:flex;gap:10px;padding:5px 0;border-bottom:1px solid var(--border)}
.logline .lt{color:var(--faint);font-variant-numeric:tabular-nums;flex:0 0 auto}
.toast{position:fixed;left:50%;bottom:24px;transform:translateX(-50%) translateY(20px);z-index:70;opacity:0;
padding:10px 16px;background:var(--elev);border:1px solid var(--border-2);border-radius:var(--r-sm);
pointer-events:none;transition:opacity .2s,transform .2s}
.toast.show{opacity:1;transform:translateX(-50%)}
.toast.err{border-color:var(--danger)}.toast.ok{border-color:var(--primary)}
</style></head>
<body>
<header class="app">
<div class="brand"><div class="logo">◧</div>
<div class="titles"><h1>Photo Analyzer</h1><p>%%LIBRARY%%</p></div></div>
<div class="seg" id="viewseg" role="tablist">
<button data-view="library" class="on">Library</button>
<button data-view="analyze">Analyze</button>
<button data-view="stats">Stats</button></div>
<div class="grow"></div>
<div class="metrics">
<div class="metric"><span class="num tabnum" id="mTotal">%%TOTAL%%</span><span class="lbl">Total</span></div>
<div class="metric"><span class="num tabnum" id="mDone">–</span><span class="lbl">Analyzed</span></div>
<div class="metric err"><span class="num tabnum" id="mErr">–</span><span class="lbl">Errors</span></div>
</div>
<button class="navtoggle" id="logbtn" aria-label="Activity log" onclick="toggleLog()">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M8 6h11M8 12h11M8 18h11"/><path d="M4 6h.01M4 12h.01M4 18h.01"/></svg></button>
</header>
<!-- ══ LIBRARY ══ -->
<section id="view-library" class="view on">
<div class="toolbar">
<button class="navtoggle on" id="sidetoggle" aria-label="Toggle folders" onclick="toggleSidebar()">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><rect x="3" y="3" width="18" height="18" rx="2"/><path d="M9 3v18"/></svg></button>
<label class="field"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><circle cx="11" cy="11" r="7"/><path d="m21 21-4.3-4.3"/></svg>
<input id="q" type="search" placeholder="Search description, tags, mood, place…" autocomplete="off"></label>
<label class="ctl">Setting <select id="fSetting"><option value="">any</option></select></label>
<label class="ctl">Time <select id="fTod"><option value="">any</option></select></label>
<label class="ctl">Season <select id="fSeason"><option value="">any</option></select></label>
<label class="ctl">People <select id="fPeople"><option value="">any</option><option value="0">0</option><option value="1">1</option><option value="2">2</option><option value="3+">3+</option></select></label>
<label class="ctl">Year <input id="fYmin" type="number" placeholder="min"> – <input id="fYmax" type="number" placeholder="max"></label>
<label class="ctl"><input id="fLoc" type="checkbox"> has place</label>
<label class="ctl">Status <select id="fStatus"><option value="">any</option></select></label>
<div class="grow"></div>
<label class="ctl">Sort <select id="sort">
<option value="relevance">Relevance</option><option value="year">Year ↓</option>
<option value="people">People ↓</option><option value="recent">Recent</option><option value="path">Folder</option></select></label>
<button class="btn ghost" onclick="resetFilters()">Clear</button>
<span class="count" id="resCount"></span>
</div>
<div class="shell" id="shell">
<aside class="sidebar" id="folders"></aside>
<div class="main">
<div class="grid" id="grid"></div>
<div id="sentinel" style="height:1px"></div>
<div class="empty" id="emptyLib" hidden><h3>No matches</h3><p>Try a broader term or clear the filters.</p></div>
</div>
</div>
</section>
<!-- ══ ANALYZE ══ -->
<section id="view-analyze" class="view">
<div class="panel"><h2>Run</h2>
<div class="runbar">
<label class="ctl">Library <input id="runLib" type="text" value="%%LIBRARY%%" style="width:260px"></label>
<label><input type="checkbox" id="opt-dry-run"> Dry-run</label>
<label><input type="checkbox" id="opt-reanalyze"> Re-analyze</label>
<label><input type="checkbox" id="opt-no-exif"> No-EXIF</label>
<label><input type="checkbox" id="opt-exif-only"> EXIF-only</label>
<label><input type="checkbox" id="opt-group-variants"> Variants</label>
<div class="grow"></div>
<button class="btn ghost" onclick="apiBalance()">Balance</button>
<button class="btn ghost" onclick="apiQuota()">Quota check</button>
<button class="btn primary" id="startBtn" onclick="startRun()">▶ Start</button>
<button class="btn danger" id="stopBtn" onclick="stopRun()" hidden>■ Stop</button>
</div>
</div>
<div class="panel"><h2>Duplicates &amp; maintenance</h2>
<div class="runbar">
<span class="ctl" style="max-width:560px;white-space:normal">Content-hash tools — no API calls.
Output appears in the Activity log; photos marked as duplicates show up under the
<b>duplicate</b> status filter in Library.</span>
<div class="grow"></div>
<button class="btn ghost" id="act-backfill-phash" onclick="runAction('backfill-phash')">⧉ Backfill hashes</button>
<button class="btn ghost" id="act-list-dupes" onclick="runAction('list-dupes')">🔍 Find duplicates</button>
<button class="btn ghost" id="act-dedupe" onclick="runAction('dedupe')">✓ Mark duplicates</button>
</div>
</div>
<div class="panel overall"><h2>Overall</h2>
<div class="big"><span class="pct" id="ovPct">Idle</span>
<span class="sub" id="ovCount"></span></div>
<div class="bar" id="ovBar"><span style="width:0"></span></div>
<div class="sub" id="ovTokens" style="margin-top:8px"></div>
</div>
<div class="cols">
<div class="panel"><h2>Albums</h2><div id="albums"><p class="count">No run in progress.</p></div></div>
<div class="panel"><h2>Recent</h2><div class="feed" id="feed"><p class="count">Analyzed photos will appear here.</p></div></div>
</div>
</section>
<!-- ══ STATS ══ -->
<section id="view-stats" class="view">
<div class="kpis" id="kpis"></div>
<div class="cols">
<div class="panel"><h2>Setting</h2><div id="stSetting"></div></div>
<div class="panel"><h2>Time of day</h2><div id="stTod"></div></div>
<div class="panel"><h2>Season</h2><div id="stSeason"></div></div>
<div class="panel"><h2>People</h2><div id="stPeople"></div></div>
</div>
<div class="panel"><h2>By year</h2><div class="years" id="stYears"></div></div>
<div class="panel"><h2>Top tags</h2><div class="tags" id="stTags"></div></div>
<div class="panel"><h2>Errors</h2><div id="stErrors"></div></div>
</section>
<!-- lightbox -->
<div class="lb" id="lb">
<div class="lbbtns"><button class="icon" onclick="closeLb()" aria-label="Close">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M18 6 6 18M6 6l12 12"/></svg></button></div>
<button class="icon lbnav prev" onclick="stepLb(-1)" aria-label="Previous">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="m15 18-6-6 6-6"/></svg></button>
<button class="icon lbnav next" onclick="stepLb(1)" aria-label="Next">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="m9 18 6-6-6-6"/></svg></button>
<div class="stage"><img id="lbImg" alt=""></div>
<div class="side"><h3 id="lbName"></h3><div class="fn2" id="lbPath"></div>
<dl class="kv" id="lbAnalysis"></dl><dl class="kv" id="lbExif"></dl></div>
</div>
<!-- activity log -->
<aside class="logpanel" id="logpanel" hidden>
<div class="loghead"><b>Activity</b><button class="btn ghost" onclick="toggleLog()">Close</button></div>
<div class="logbody" id="logbody"></div>
</aside>
<div class="toast" id="toast" hidden></div>
<script>
const $=s=>document.querySelector(s), $$=s=>[...document.querySelectorAll(s)];
const esc=s=>String(s??'').replace(/[&<>"]/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
const enc=encodeURIComponent;
function toast(m,c){const t=$('#toast');t.textContent=m;t.hidden=false;t.className='toast show '+(c||'');
clearTimeout(t._t);t._t=setTimeout(()=>t.classList.remove('show'),3200);}
/* ── view switching (deep-linked via hash) ── */
function setView(v){
if(!['library','analyze','stats'].includes(v))v='library';
$$('.view').forEach(s=>s.classList.toggle('on',s.id==='view-'+v));
$$('#viewseg button').forEach(b=>b.classList.toggle('on',b.dataset.view===v));
if(location.hash.slice(1).split('?')[0]!==v)history.replaceState(null,'','#'+v);
if(v==='stats')loadStats();
if(v==='analyze')pollProgress();
}
$$('#viewseg button').forEach(b=>b.onclick=()=>setView(b.dataset.view));
addEventListener('hashchange',()=>setView(location.hash.slice(1).split('?')[0]));
/* ── LIBRARY ── */
let offset=0,total=0,loading=false,done=true,curAlbum='';
const filt=()=>({q:$('#q').value.trim(),setting:$('#fSetting').value,tod:$('#fTod').value,
season:$('#fSeason').value,people:$('#fPeople').value,year_min:$('#fYmin').value,
year_max:$('#fYmax').value,has_location:$('#fLoc').checked?'1':'',status:$('#fStatus').value,
album:curAlbum,sort:$('#sort').value});
function qs(o){return Object.entries(o).filter(([,v])=>v!==''&&v!=null).map(([k,v])=>k+'='+enc(v)).join('&');}
async function runSearch(reset=true){
if(loading)return; loading=true;
if(reset){offset=0;$('#grid').innerHTML='';done=false;}
const r=await fetch('/search?'+qs({...filt(),offset})).then(r=>r.json());
total=r.total; $('#resCount').textContent=total.toLocaleString()+' photo'+(total===1?'':'s');
$('#emptyLib').hidden=total>0;
$('#grid').insertAdjacentHTML('beforeend',r.rows.map(card).join(''));
offset+=r.rows.length; done=offset>=total || r.rows.length===0;
loading=false;
}
function card(p){
const yr=p.approx_year?`<span class="badge">${p.approx_year}</span>`:'';
const pe=(p.people_count>0)?`<span class="badge">👤 ${p.people_count}</span>`:'';
const du=p.status==='duplicate'?`<span class="badge warnb" title="perceptual duplicate">dup</span>`:'';
const dot=p.status==='error'?'<span class="dot err" title="error"></span>'
:(p.status==='pending'?'<span class="dot pending" title="not analyzed"></span>':'');
return `<div class="card ${p.status==='error'?'err':''}" data-path="${esc(p.path)}" onclick="openLb(this)">
<div class="thumb"><img loading="lazy" src="/img?path=${enc(p.path)}" alt="">
<div class="badges">${yr}${pe}${du}</div>${dot}</div>
<div class="info"><div class="desc">${esc(p.description||(p.status==='error'?'—':'(not analyzed)'))}</div>
<div class="pathline">${esc(p.path.split('/').slice(-2).join('/'))}</div></div></div>`;
}
let searchT;
function debouncedSearch(){clearTimeout(searchT);searchT=setTimeout(()=>runSearch(true),250);}
['#q','#fYmin','#fYmax'].forEach(s=>$(s).addEventListener('input',debouncedSearch));
['#fSetting','#fTod','#fSeason','#fPeople','#fStatus','#sort'].forEach(s=>$(s).addEventListener('change',()=>runSearch(true)));
$('#fLoc').addEventListener('change',()=>runSearch(true));
function resetFilters(){['#q','#fYmin','#fYmax'].forEach(s=>$(s).value='');
['#fSetting','#fTod','#fSeason','#fPeople','#fStatus'].forEach(s=>$(s).value='');
$('#fLoc').checked=false;$('#sort').value='relevance';curAlbum='';
$$('.fitem').forEach(x=>x.classList.remove('on'));runSearch(true);}
new IntersectionObserver(es=>{if(es[0].isIntersecting&&!done&&!loading)runSearch(false);})
.observe($('#sentinel'));
function toggleSidebar(){const c=$('#shell').classList.toggle('collapsed');
$('#sidetoggle').classList.toggle('on',!c);}
function opt(sel,items,none='any'){
$(sel).innerHTML=`<option value="">${none}</option>`+items.map(i=>
`<option value="${esc(i.value)}">${esc(i.value)} (${i.count.toLocaleString()})</option>`).join('');}
async function loadFacets(){
const f=await fetch('/facets').then(r=>r.json());
opt('#fSetting',f.setting);opt('#fTod',f.tod);opt('#fSeason',f.season);opt('#fStatus',f.status);
if(f.year_min){$('#fYmin').placeholder=f.year_min;$('#fYmax').placeholder=f.year_max;}
$('#folders').innerHTML=`<button class="fitem on" onclick="pickAlbum(this,'')"><span>All folders</span></button>`+
f.albums.map(a=>`<button class="fitem" onclick="pickAlbum(this,'${esc(a.album)}')">
<span>${esc(a.album)}</span><span class="cn">${a.count.toLocaleString()}</span></button>`).join('');
// album buttons carry the folder name; map to absolute path server-side via album= param
}
function pickAlbum(btn,album){curAlbum=album?(LIBRARY.replace(/\/$/,'')+'/'+album):'';
$$('.fitem').forEach(x=>x.classList.toggle('on',x===btn));runSearch(true);}
const LIBRARY='%%LIBRARY%%';
/* ── LIGHTBOX ── */
let lbCards=[],lbIdx=0;
function openLb(el){lbCards=$$('#grid .card');lbIdx=lbCards.indexOf(el);showLb();$('#lb').classList.add('on');}
function closeLb(){$('#lb').classList.remove('on');}
function stepLb(d){lbIdx=(lbIdx+d+lbCards.length)%lbCards.length;showLb();}
async function showLb(){
const el=lbCards[lbIdx],path=el.dataset.path;
$('#lbImg').src='/img?path='+enc(path);
$('#lbName').textContent=path.split('/').pop();$('#lbPath').textContent=path;
$('#lbAnalysis').innerHTML='<dt>Analysis</dt><dd class="count">loading…</dd>';
$('#lbExif').innerHTML='';
const p=await fetch('/photo?path='+enc(path)).then(r=>r.json()).catch(()=>null);
$('#lbAnalysis').innerHTML=p?analysisDl(p):'<dt>Analysis</dt><dd>—</dd>';
const ex=await fetch('/exif?path='+enc(path)).then(r=>r.json()).catch(()=>({}));
$('#lbExif').innerHTML='<dt style="margin-top:16px">EXIF</dt>'+Object.entries(ex).map(([k,v])=>
k==='Map'?`<dd><a href="${esc(v)}" target="_blank" rel="noopener">Open in Google Maps ↗</a></dd>`
:`<dt>${esc(k)}</dt><dd>${esc(v)}</dd>`).join('');
}
function row(k,v){return v==null||v===''?'':`<dt>${k}</dt><dd>${esc(v)}</dd>`;}
function analysisDl(p){
return row('Description',p.description)+
(p.tags&&p.tags.length?`<dt>Tags</dt><dd>${p.tags.map(esc).join(', ')}</dd>`:'')+
row('Mood',p.mood)+row('Setting',p.setting)+row('Time of day',p.time_of_day)+
row('Season',p.season)+row('People',p.people_count)+row('Approx. year',p.approx_year)+
row('Location',p.location_hint)+row('Status',p.status)+row('Duplicate of',p.dup_of)+row('Error',p.error_message);
}
addEventListener('keydown',e=>{if(!$('#lb').classList.contains('on'))return;
if(e.key==='Escape')closeLb();else if(e.key==='ArrowLeft')stepLb(-1);else if(e.key==='ArrowRight')stepLb(1);});
$('#lb').addEventListener('click',e=>{if(e.target.id==='lb')closeLb();});
/* ── STATS ── */
function bars(el,items,max){const m=max||Math.max(1,...items.map(i=>i.count));
$(el).innerHTML=items.map(i=>`<div class="statbar"><span class="lab">${esc(i.value)}</span>
<span class="track"><span style="width:${i.count/m*100}%"></span></span>
<span class="val">${i.count.toLocaleString()}</span></div>`).join('')||'<p class="count">no data</p>';}
async function loadStats(){
const s=await fetch('/stats').then(r=>r.json());
const st=s.status||{};
const done=(st.analyzed||0)+(st.exif_written||0);
$('#mDone').textContent=done.toLocaleString();$('#mErr').textContent=(st.error||0).toLocaleString();
const kpi=[['Total',s.total],['Analyzed',done],['EXIF written',st.exif_written||0],
['Duplicates',st.duplicate||0],['Errors',st.error||0],['Pending',st.pending||0]];
$('#kpis').innerHTML=kpi.map(([l,n])=>`<div class="kpi"><div class="num tabnum">${(n||0).toLocaleString()}</div><div class="lbl">${l}</div></div>`).join('');
bars('#stSetting',s.setting);bars('#stTod',s.time_of_day);bars('#stSeason',s.season);bars('#stPeople',s.people);
const ym=Math.max(1,...s.years.map(y=>y.count));
$('#stYears').innerHTML=s.years.map(y=>`<div class="yb" style="height:${y.count/ym*100}%" title="${y.value}: ${y.count}"></div>`).join('')||'<p class="count">no dated photos</p>';
$('#stTags').innerHTML=s.top_tags.map(t=>`<button class="tag" onclick="searchTag('${esc(t.value)}')">${esc(t.value)} <span class="cn">${t.count}</span></button>`).join('');
$('#stErrors').innerHTML=s.errors.length?s.errors.map(e=>`<div class="errline"><span>${esc(e.path.split('/').pop())}</span><span class="msg">${esc(e.error||'')}</span></div>`).join(''):'<p class="count">No errors 🎉</p>';
}
function searchTag(t){$('#q').value=t;setView('library');runSearch(true);}
/* ── ANALYZE (progress polling; run controls wired to server) ── */
let progT=null;
async function pollProgress(){
clearTimeout(progT);
let s;try{s=await fetch('/progress').then(r=>r.json());}catch(e){progT=setTimeout(pollProgress,2000);return;}
renderProgress(s);
if($('#view-analyze').classList.contains('on'))progT=setTimeout(pollProgress,s.running?1500:4000);
}
function renderProgress(s){
$('#startBtn').hidden=!!s.running;$('#stopBtn').hidden=!s.running;
['act-backfill-phash','act-list-dupes','act-dedupe'].forEach(id=>{const b=$('#'+id);if(b)b.disabled=!!s.running;});
if(!s.running&&!s.totals){$('#ovPct').textContent='Idle';$('#ovCount').textContent='';$('#ovBar').firstElementChild.style.width='0';return;}
const t=s.totals||{done:0,total:0,ok:0,err:0};
const pct=t.total?Math.round(t.done/t.total*100):0;
$('#ovPct').textContent=pct+'%';
$('#ovCount').textContent=`${(t.done||0).toLocaleString()} / ${(t.total||0).toLocaleString()} · ok ${t.ok||0} · err ${t.err||0}`+(t.dup?` · dup ${t.dup.toLocaleString()}`:'');
const bar=$('#ovBar');bar.classList.toggle('run',!!s.running);bar.firstElementChild.style.width=pct+'%';
if(s.tokens!=null)$('#ovTokens').textContent=`tokens ${(s.tokens||0).toLocaleString()}`+(s.eta?` · ETA ${s.eta}`:'');
$('#albums').innerHTML=(s.albums||[]).map(a=>{const p=a.total?Math.round(a.done/a.total*100):0;
return `<div class="albrow"><span class="nm">${esc(a.album)}</span>
<span class="bar" style="width:120px"><span style="width:${p}%"></span></span>
<span class="ct">${a.done===a.total?'✓':p+'%'}</span></div>`;}).join('')||'<p class="count">No run in progress.</p>';
$('#feed').innerHTML=(s.feed||[]).map(f=>`<div class="feedline"><span class="ic ${f.ok?'ok':'bad'}">${f.ok?'✓':'✗'}</span>
<span><span class="fn">${esc(f.name)}</span><div class="dz">${esc(f.msg||'')}</div></span></div>`).join('')||'<p class="count">Analyzed photos will appear here.</p>';
}
async function startRun(){
const opts={};['dry-run','reanalyze','no-exif','exif-only','group-variants'].forEach(k=>opts[k]=$('#opt-'+k).checked);
opts.library=$('#runLib').value;
const r=await fetch('/run',{method:'POST',body:JSON.stringify(opts)}).then(r=>r.json()).catch(e=>({error:''+e}));
if(r.error)toast(r.error,'err');else{toast('Run started','ok');pollProgress();}
}
async function stopRun(){await fetch('/stop',{method:'POST'});toast('Stopping…');pollProgress();}
const ACTION_LABELS={'backfill-phash':'Backfilling hashes','list-dupes':'Finding duplicates','dedupe':'Marking duplicates'};
function ensureLogOpen(){if($('#logpanel').hidden)toggleLog();}
async function runAction(action){
const r=await fetch('/run',{method:'POST',body:JSON.stringify({action,library:$('#runLib').value})})
.then(r=>r.json()).catch(e=>({error:''+e}));
if(r.error){toast(r.error,'err');return;}
toast(ACTION_LABELS[action]+'…','ok');ensureLogOpen();pollProgress(); // output streams to Activity log
}
async function apiBalance(){const r=await fetch('/balance').then(r=>r.json()).catch(()=>({}));
toast(r.text||'Balance unavailable',r.text?'ok':'err');}
async function apiQuota(){toast('Checking quota…');const r=await fetch('/quota-check').then(r=>r.json()).catch(()=>({ok:false}));
toast(r.ok?'Quota available ✓':'Quota check failed',r.ok?'ok':'err');}
/* ── activity log ── */
let logT=null;
function toggleLog(){const p=$('#logpanel'),open=p.hidden;p.hidden=!open;$('#logbtn').classList.toggle('on',open);
if(open){refreshLog();logT=setInterval(refreshLog,2500);}else clearInterval(logT);}
async function refreshLog(){const es=await fetch('/log').then(r=>r.json()).catch(()=>[]);
$('#logbody').innerHTML=es.length?es.slice().reverse().map(e=>`<div class="logline"><span class="lt">${esc(e.t)}</span><span>${esc(e.m)}</span></div>`).join(''):'<p class="count">No activity yet.</p>';}
/* ── boot ── */
loadFacets().then(()=>runSearch(true));
loadStats();
setView(location.hash.slice(1).split('?')[0]||'library');
</script>
</body></html>

12
webapp/page.py Normal file
View File

@@ -0,0 +1,12 @@
"""Render analyzer.html with %%TOKENS%% filled in. The page is otherwise static
and pulls its data from the JSON endpoints, so this only injects boot values."""
from pathlib import Path
_HTML = (Path(__file__).with_name("analyzer.html")).read_text(encoding="utf-8")
def render_page(total: int, library: str, db: str) -> str:
return (_HTML
.replace("%%TOTAL%%", f"{total:,}")
.replace("%%LIBRARY%%", library)
.replace("%%DB%%", db))

214
webapp/query.py Normal file
View File

@@ -0,0 +1,214 @@
"""Read-only DB access for the web UI: FTS5 search, facets, stats, one-photo.
The analysis pipeline (photo_analyzer.py) owns writes; this module only reads.
Album = leaf folder relative to the library (same rule as photo_analyzer.album_label).
"""
import json
import os
import re
import sqlite3
from pathlib import Path
from . import PAGE_SIZE
# Columns sent to the browser for a card / lightbox. raw_response is intentionally
# omitted from lists (big); the lightbox fetches it via /photo if ever needed.
CARD_COLS = ("path", "status", "description", "tags", "people_count", "setting",
"time_of_day", "season", "mood", "location_hint", "approx_year")
DONE = ("analyzed", "exif_written")
def connect(db_path: str) -> sqlite3.Connection:
conn = sqlite3.connect(db_path, check_same_thread=False) # server is multi-threaded
conn.row_factory = sqlite3.Row
return conn
def album_of(path: str, library: Path | None) -> str:
"""Leaf folder relative to the library — mirrors photo_analyzer.album_label."""
parent = Path(path).parent
if library:
try:
rel = parent.relative_to(library)
return "(root)" if str(rel) == "." else str(rel)
except ValueError:
pass
return parent.name or str(parent)
def _fts_query(q: str) -> str | None:
"""Turn free text into a safe FTS5 MATCH string. Each word becomes a prefix
term (AND-ed), so 'beach sun' matches 'beachy sunset'. Returns None if empty
(→ caller browses instead of matching), so raw punctuation can't crash MATCH."""
tokens = re.findall(r"\w+", q, re.UNICODE)
return " ".join(f'"{t}"*' for t in tokens) if tokens else None
def _where(filters: dict) -> tuple[list[str], list]:
"""Build a WHERE clause fragment list + params from the facet filters."""
clauses, params = [], []
if v := filters.get("setting"):
clauses.append("p.setting = ?"); params.append(v)
if v := filters.get("tod"):
clauses.append("p.time_of_day = ?"); params.append(v)
if v := filters.get("season"):
clauses.append("p.season = ?"); params.append(v)
if v := filters.get("status"):
clauses.append("p.status = ?"); params.append(v)
people = filters.get("people")
if people == "3+":
clauses.append("p.people_count >= 3")
elif people in ("0", "1", "2"):
clauses.append("p.people_count = ?"); params.append(int(people))
if filters.get("year_min"):
clauses.append("p.approx_year >= ?"); params.append(int(filters["year_min"]))
if filters.get("year_max"):
clauses.append("p.approx_year <= ?"); params.append(int(filters["year_max"]))
if filters.get("has_location"):
clauses.append("p.location_hint IS NOT NULL AND TRIM(p.location_hint) != '' "
"AND LOWER(p.location_hint) != 'null'")
if album := filters.get("album"):
# album is an absolute leaf-folder path; match it and its subfolders
clauses.append("(p.path LIKE ? OR p.path LIKE ?)")
params += [f"{album}/%", f"{album}%/%"]
return clauses, params
_SORTS = {
"year": "p.approx_year DESC NULLS LAST, p.path",
"people": "p.people_count DESC NULLS LAST, p.path",
"recent": "p.analyzed_at DESC NULLS LAST, p.path",
"path": "p.path",
}
def _row_to_card(r: sqlite3.Row) -> dict:
d = {k: r[k] for k in CARD_COLS}
try:
d["tags"] = json.loads(r["tags"]) if r["tags"] else []
except (ValueError, TypeError):
d["tags"] = []
return d
def search(conn, q="", filters=None, sort="relevance", offset=0, limit=PAGE_SIZE):
"""Paged result set. FTS5 when `q` is present, plain filter/browse otherwise.
Returns {rows, total, offset, limit}."""
filters = filters or {}
clauses, params = _where(filters)
match = _fts_query(q) if q else None
if match:
base = ("FROM photos p JOIN photos_fts f ON p.id = f.rowid "
"WHERE photos_fts MATCH ?")
params = [match] + params
order = "f.rank" if sort in ("relevance", "") else _SORTS.get(sort, "f.rank")
else:
base = "FROM photos p WHERE 1=1"
order = _SORTS.get(sort if sort != "relevance" else "path", "p.path")
if clauses:
base += " AND " + " AND ".join(clauses)
total = conn.execute(f"SELECT COUNT(*) {base}", params).fetchone()[0]
rows = conn.execute(
f"SELECT p.* {base} ORDER BY {order} LIMIT ? OFFSET ?",
params + [limit, offset],
).fetchall()
return {"rows": [_row_to_card(r) for r in rows], "total": total,
"offset": offset, "limit": limit}
def photo(conn, path: str) -> dict | None:
r = conn.execute("SELECT * FROM photos WHERE path = ?", (path,)).fetchone()
if not r:
return None
d = {k: r[k] for k in r.keys() if k != "raw_response"}
try:
d["tags"] = json.loads(r["tags"]) if r["tags"] else []
except (ValueError, TypeError):
d["tags"] = []
return d
def _col_counts(conn, col: str) -> list[dict]:
rows = conn.execute(
f"SELECT {col} AS v, COUNT(*) AS n FROM photos "
f"WHERE {col} IS NOT NULL AND TRIM({col}) != '' GROUP BY {col} ORDER BY n DESC"
).fetchall()
return [{"value": r["v"], "count": r["n"]} for r in rows]
def facets(conn, library: Path | None) -> dict:
"""Distinct filter values + counts, plus the album tree, for the toolbar."""
year = conn.execute(
"SELECT MIN(approx_year), MAX(approx_year) FROM photos WHERE approx_year IS NOT NULL"
).fetchone()
albums: dict[str, int] = {}
for (p,) in conn.execute("SELECT path FROM photos"):
a = album_of(p, library)
albums[a] = albums.get(a, 0) + 1
return {
"setting": _col_counts(conn, "setting"),
"tod": _col_counts(conn, "time_of_day"),
"season": _col_counts(conn, "season"),
"status": _col_counts(conn, "status"),
"year_min": year[0], "year_max": year[1],
"albums": [{"album": a, "count": n} for a, n in sorted(albums.items())],
}
def _people_buckets(conn) -> list[dict]:
rows = conn.execute(
"SELECT CASE WHEN people_count >= 3 THEN '3+' ELSE CAST(people_count AS TEXT) END AS b, "
"COUNT(*) AS n FROM photos WHERE people_count IS NOT NULL GROUP BY b"
).fetchall()
return [{"value": r["b"], "count": r["n"]} for r in rows]
def _top_tags(conn, limit=40) -> list[dict]:
counts: dict[str, int] = {}
for (t,) in conn.execute("SELECT tags FROM photos WHERE tags IS NOT NULL"):
try:
for tag in json.loads(t):
tag = str(tag).strip()
if tag:
counts[tag] = counts.get(tag, 0) + 1
except (ValueError, TypeError):
continue
top = sorted(counts.items(), key=lambda kv: kv[1], reverse=True)[:limit]
return [{"value": t, "count": n} for t, n in top]
def stats(conn, library: Path | None) -> dict:
status = {r["status"]: r["n"] for r in conn.execute(
"SELECT status, COUNT(*) AS n FROM photos GROUP BY status")}
total = sum(status.values())
years = [{"value": r["y"], "count": r["n"]} for r in conn.execute(
"SELECT approx_year AS y, COUNT(*) AS n FROM photos "
"WHERE approx_year IS NOT NULL GROUP BY approx_year ORDER BY approx_year")]
albums: dict[str, dict] = {}
for r in conn.execute("SELECT path, status FROM photos"):
a = album_of(r["path"], library)
d = albums.setdefault(a, {"album": a, "done": 0, "total": 0})
d["total"] += 1
if r["status"] in DONE:
d["done"] += 1
errors = [{"path": r["path"], "error": r["error_message"]} for r in conn.execute(
"SELECT path, error_message FROM photos WHERE status = 'error' ORDER BY path")]
return {
"total": total, "status": status,
"setting": _col_counts(conn, "setting"),
"time_of_day": _col_counts(conn, "time_of_day"),
"season": _col_counts(conn, "season"),
"people": _people_buckets(conn),
"years": years,
"top_tags": _top_tags(conn),
"albums": sorted(albums.values(), key=lambda d: d["album"]),
"errors": errors,
}
def all_paths(conn) -> set:
"""Every registered path — the allow-set for /img and /exif (path validation)."""
return {r[0] for r in conn.execute("SELECT path FROM photos")}

110
webapp/runner.py Normal file
View File

@@ -0,0 +1,110 @@
"""Drive photo_analyzer.py as a subprocess and expose live run-state to the web UI.
Why a subprocess and not an in-process thread: the pipeline already is a complete,
resume-safe CLI that owns SQLite as the source of truth. Running it as `python
photo_analyzer.py …` keeps the two fully decoupled — the web server only *reads* the
same DB for progress, Stop is the analyzer's own tested SIGINT drain, and a web crash
can't corrupt a run (or vice-versa). Progress is derived from the DB, so it survives
a browser reload and matches `--stats` exactly.
"""
import signal
import subprocess
import sys
import threading
from pathlib import Path
from . import query
REPO = Path(__file__).resolve().parent.parent # dir holding photo_analyzer.py
_FLAGS = ("dry-run", "reanalyze", "no-exif", "exif-only", "group-variants")
# Exit-after-action modes: each runs one maintenance task and quits (no analysis).
# Passed alone — the boolean _FLAGS are ignored when an action is chosen.
_ACTIONS = ("backfill-phash", "list-dupes", "dedupe")
class Runner:
def __init__(self, db_path: str, log):
self.db = db_path
self.log = log
self.proc: subprocess.Popen | None = None
self.lock = threading.Lock()
def running(self) -> bool:
return self.proc is not None and self.proc.poll() is None
def start(self, opts: dict) -> dict:
with self.lock:
if self.running():
return {"error": "A run is already in progress."}
lib = (opts.get("library") or "").strip()
if not lib:
return {"error": "Library path is required."}
argv = [sys.executable, str(REPO / "photo_analyzer.py"),
"--library", lib, "--db", self.db]
action = opts.get("action")
if action in _ACTIONS:
argv.append(f"--{action}") # maintenance action, runs alone
else:
argv += [f"--{f}" for f in _FLAGS if opts.get(f)]
try:
self.proc = subprocess.Popen(
argv, cwd=str(REPO), stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, bufsize=1)
except Exception as e:
return {"error": f"Could not start: {e}"}
threading.Thread(target=self._reader, args=(self.proc,), daemon=True).start()
self.log("run started: " + " ".join(argv[2:]))
return {"ok": True}
def _reader(self, proc):
# Surface the analyzer's own log lines in the browser's Activity panel.
for line in proc.stdout:
line = line.rstrip()
if line:
self.log(line)
self.log(f"run finished (exit {proc.poll()})")
def stop(self) -> dict:
with self.lock:
if self.running():
self.proc.send_signal(signal.SIGINT) # reuse the tested drain path
self.log("stop requested")
return {"ok": True}
def progress(conn, library: Path | None, running: bool) -> dict:
"""DB-derived run state for /progress. Cheap full scan (~25k rows, a few ms);
ponytail: if the library grows past ~200k, cache album counts between polls."""
st = {r["status"]: r["n"] for r in conn.execute(
"SELECT status, COUNT(*) AS n FROM photos GROUP BY status")}
done = st.get("analyzed", 0) + st.get("exif_written", 0)
dup = st.get("duplicate", 0)
# Duplicates are deliberately skipped, not pending work — exclude them from the
# denominator so the bar reflects real progress rather than dragging to <100%.
work_total = sum(st.values()) - dup
albums: dict[str, dict] = {}
for r in conn.execute("SELECT path, status FROM photos"):
if r["status"] == "duplicate":
continue # not work; keep albums reaching 100%
a = query.album_of(r["path"], library)
d = albums.setdefault(a, {"album": a, "done": 0, "total": 0})
d["total"] += 1
if r["status"] in query.DONE:
d["done"] += 1
# incomplete albums first — that's what a running job is working on
alist = sorted(albums.values(), key=lambda d: (d["done"] >= d["total"], d["album"]))
feed = []
for r in conn.execute(
"SELECT path, status, description, error_message FROM photos "
"WHERE analyzed_at IS NOT NULL ORDER BY analyzed_at DESC LIMIT 14"
):
ok = r["status"] != "error"
feed.append({"ok": ok, "name": Path(r["path"]).name,
"msg": (r["description"] if ok else r["error_message"]) or ""})
return {"running": running,
"totals": {"done": done, "total": work_total, "ok": done,
"err": st.get("error", 0), "dup": dup},
"albums": alist[:60], "feed": feed, "tokens": None}

162
webapp/server.py Normal file
View File

@@ -0,0 +1,162 @@
"""Local, stdlib-only HTTP server for the Photo Analyzer web UI.
Serves one page + JSON endpoints, all bound to 127.0.0.1. Reads the analysis DB
(via query.py), serves thumbnails/EXIF for the lightbox (reusing nsfwtag's readers),
and drives analysis runs (runner.py). Path args on /img and /exif are validated
against the DB, so the server can't be pointed at arbitrary files.
"""
import http.server
import json
import mimetypes
import sys
import time
import webbrowser
from collections import deque
from pathlib import Path
from urllib.parse import parse_qs, unquote, urlparse
from . import PAGE_SIZE, page, query
from .runner import Runner, progress as run_progress
try:
from nsfwtag.exif import read_exif # reuse the lightbox EXIF reader
except Exception: # nsfwtag not importable → degrade
def read_exif(_p): return {}
def serve(db_path: str, library: str | None, host: str = "127.0.0.1", open_browser=True):
conn = query.connect(db_path)
conn.execute("PRAGMA busy_timeout=3000") # tolerate the analyzer writing concurrently
lib = Path(library).expanduser().resolve() if library else None
log_entries = deque(maxlen=2000)
def log(m):
log_entries.append({"t": time.strftime("%H:%M:%S"), "m": m})
print(m, file=sys.stderr)
runner = Runner(db_path, log)
def allowed(fp: str) -> bool:
return conn.execute("SELECT 1 FROM photos WHERE path=? LIMIT 1", (fp,)).fetchone() is not None
def filters_from(qd: dict) -> dict:
g = lambda k: qd.get(k, [""])[0]
return {k: g(k) for k in ("q", "setting", "tod", "season", "people",
"year_min", "year_max", "has_location", "status",
"album", "sort")}
total0 = conn.execute("SELECT COUNT(*) FROM photos").fetchone()[0]
page_bytes = page.render_page(total0, str(lib) if lib else (library or ""), db_path).encode("utf-8")
log(f"web ui ready — {total0:,} photos indexed")
class H(http.server.BaseHTTPRequestHandler):
def log_message(self, *a):
pass
def _send(self, code, ctype, body):
self.send_response(code)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def _json(self, obj, code=200):
self._send(code, "application/json", json.dumps(obj).encode("utf-8"))
def do_GET(self):
u = urlparse(self.path)
qd = parse_qs(u.query)
p = u.path
if p == "/":
return self._send(200, "text/html; charset=utf-8", page_bytes)
if p == "/img":
fp = unquote(qd.get("path", [""])[0])
if allowed(fp) and Path(fp).exists():
ctype = mimetypes.guess_type(fp)[0] or "application/octet-stream"
return self._send(200, ctype, Path(fp).read_bytes())
return self._send(404, "text/plain", b"no image")
if p == "/exif":
fp = unquote(qd.get("path", [""])[0])
if allowed(fp) and Path(fp).exists():
return self._json(read_exif(fp))
return self._json({})
if p == "/photo":
fp = unquote(qd.get("path", [""])[0])
return self._json(query.photo(conn, fp) or {})
if p == "/search":
f = filters_from(qd)
try:
offset = int(qd.get("offset", ["0"])[0])
except ValueError:
offset = 0
return self._json(query.search(
conn, q=f.pop("q"), filters=f, sort=f.get("sort") or "relevance",
offset=offset, limit=PAGE_SIZE))
if p == "/facets":
return self._json(query.facets(conn, lib))
if p == "/stats":
return self._json(query.stats(conn, lib))
if p == "/progress":
return self._json(run_progress(conn, lib, runner.running()))
if p == "/log":
return self._json(list(log_entries))
if p == "/balance":
return self._json(_balance())
if p == "/quota-check":
return self._json({"ok": _quota()})
return self._send(404, "text/plain", b"not found")
def do_POST(self):
u = urlparse(self.path)
n = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(n) if n else b"{}"
if u.path == "/run":
try:
opts = json.loads(body or b"{}")
except ValueError:
return self._json({"error": "bad request"}, 400)
return self._json(runner.start(opts))
if u.path == "/stop":
return self._json(runner.stop())
return self._send(404, "text/plain", b"not found")
def _balance():
try:
import photo_analyzer as pa
pa.load_env_file()
import os
key = os.environ.get("LLM_API_KEY") or os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
d = pa.fetch_balance(key)
if not d:
return {"text": ""}
return {"text": f"available {d.get('available_balance')} · "
f"cash {d.get('cash_balance')} · voucher {d.get('voucher_balance')}"}
except Exception as e:
log(f"balance error: {e}")
return {"text": ""}
def _quota():
try:
import os
import photo_analyzer as pa
pa.load_env_file()
from openai import OpenAI
key = os.environ.get("LLM_API_KEY") or os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
client = OpenAI(api_key=key or "x", base_url=pa.LLM_BASE_URL)
return pa.check_quota(client)
except Exception as e:
log(f"quota error: {e}")
return False
srv = http.server.ThreadingHTTPServer((host, 0), H)
url = f"http://{host}:{srv.server_address[1]}/"
print(f"\nPhoto Analyzer web UI: {url}\n Ctrl-C here to stop the server.\n", file=sys.stderr)
if open_browser:
webbrowser.open(url)
try:
srv.serve_forever()
except KeyboardInterrupt:
print("\nstopped.", file=sys.stderr)
finally:
srv.server_close()

5
work_item/__init__.py Normal file
View File

@@ -0,0 +1,5 @@
"""Safe Gitea-backed work-item workflow helpers."""
from .core import WorkItemError, Workflow
__all__ = ["WorkItemError", "Workflow"]

3
work_item/__main__.py Normal file
View File

@@ -0,0 +1,3 @@
from .core import main
raise SystemExit(main())

547
work_item/core.py Normal file
View File

@@ -0,0 +1,547 @@
from __future__ import annotations
import argparse
import fnmatch
import json
import os
import re
import shlex
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterable, Sequence
try:
import tomllib
except ModuleNotFoundError: # pragma: no cover - Python < 3.11 is unsupported
tomllib = None
STORY_RE = re.compile(r"^(US(?P<epic>\d{2})-(?P<story>\d{2}))\s+—\s+(?P<title>.+)$")
SECRET_PATTERNS = (
re.compile(r"(?i)(api[_-]?key|access[_-]?token|client[_-]?secret|password)\s*[:=]\s*['\"]?[A-Za-z0-9_./+\-=]{12,}"),
re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"),
re.compile(r"(?i)authorization:\s*(?:bearer|token)\s+[A-Za-z0-9._~+\-/=]{12,}"),
)
class WorkItemError(RuntimeError):
pass
class Runner:
def run(
self,
args: Sequence[str],
*,
cwd: Path,
check: bool = True,
input_text: str | None = None,
) -> subprocess.CompletedProcess[str]:
result = subprocess.run(
list(args),
cwd=cwd,
text=True,
input=input_text,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
if check and result.returncode:
command = " ".join(shlex.quote(part) for part in args)
detail = (result.stderr or result.stdout).strip()
raise WorkItemError(f"Command failed ({result.returncode}): {command}\n{detail}")
return result
def run_shell(self, command: str, *, cwd: Path) -> subprocess.CompletedProcess[str]:
result = subprocess.run(command, cwd=cwd, text=True, shell=True)
if result.returncode:
raise WorkItemError(f"Test command failed ({result.returncode}): {command}")
return result
@dataclass(frozen=True)
class Config:
repo_slug: str
login: str
assignee: str
remote: str = "origin"
main_branch: str = "main"
branch_prefix: str = "us"
require_ci: bool = False
max_file_bytes: int = 5_000_000
required_tests: tuple[str, ...] = ()
denied_patterns: tuple[str, ...] = ()
allowed_patterns: tuple[str, ...] = ()
@classmethod
def load(cls, path: Path) -> "Config":
if tomllib is None:
raise WorkItemError("Python 3.11 or newer is required")
if not path.is_file():
raise WorkItemError(f"Missing workflow configuration: {path}")
with path.open("rb") as handle:
raw = tomllib.load(handle)
repo = raw.get("repository", {})
workflow = raw.get("workflow", {})
safety = raw.get("safety", {})
required = ("slug", "login", "assignee")
missing = [key for key in required if not repo.get(key)]
if missing:
raise WorkItemError(f"Missing repository configuration: {', '.join(missing)}")
return cls(
repo_slug=repo["slug"],
login=repo["login"],
assignee=repo["assignee"],
remote=repo.get("remote", "origin"),
main_branch=repo.get("main_branch", "main"),
branch_prefix=workflow.get("branch_prefix", "us"),
require_ci=bool(workflow.get("require_ci", False)),
max_file_bytes=int(safety.get("max_file_bytes", 5_000_000)),
required_tests=tuple(workflow.get("required_tests", ())),
denied_patterns=tuple(safety.get("deny", ())),
allowed_patterns=tuple(safety.get("allow", ())),
)
@dataclass(frozen=True)
class Story:
number: int
story_id: str
title: str
epic: int
sequence: int
labels: frozenset[str]
assignees: tuple[str, ...]
raw: dict[str, Any]
@classmethod
def from_issue(cls, issue: dict[str, Any]) -> "Story | None":
match = STORY_RE.match(issue.get("title", ""))
if not match:
return None
labels = frozenset(label["name"] for label in issue.get("labels") or ())
assignees = tuple(user["login"] for user in issue.get("assignees") or ())
return cls(
number=int(issue["number"]),
story_id=match.group(1),
title=match.group("title"),
epic=int(match.group("epic")),
sequence=int(match.group("story")),
labels=labels,
assignees=assignees,
raw=issue,
)
@property
def branch_slug(self) -> str:
value = re.sub(r"[^a-z0-9]+", "-", self.title.lower()).strip("-")
return value[:48].rstrip("-")
class GitRepo:
def __init__(self, root: Path, config: Config, runner: Runner) -> None:
self.root = root
self.config = config
self.runner = runner
def git(self, *args: str, check: bool = True) -> str:
return self.runner.run(("git", *args), cwd=self.root, check=check).stdout.strip()
def verify(self) -> None:
actual = Path(self.git("rev-parse", "--show-toplevel")).resolve()
if actual != self.root.resolve():
raise WorkItemError(f"Run from repository root {actual}; current root is {self.root}")
remote_url = self.git("remote", "get-url", self.config.remote)
normalized = remote_url.removesuffix(".git").replace(":", "/")
if self.config.repo_slug not in normalized:
raise WorkItemError(
f"Remote {self.config.remote!r} does not match {self.config.repo_slug!r}: {remote_url}"
)
def ensure_clean(self) -> None:
if self.git("status", "--porcelain=v1"):
raise WorkItemError("Working tree must be clean before claiming a story")
def update_main(self) -> None:
self.git("fetch", "--prune", self.config.remote)
self.git("switch", self.config.main_branch)
self.git("pull", "--ff-only", self.config.remote, self.config.main_branch)
def current_branch(self) -> str:
return self.git("branch", "--show-current")
def create_story_branch(self, story: Story) -> str:
branch = f"{self.config.branch_prefix}/{story.story_id}-{story.branch_slug}"
self.git("switch", "-c", branch)
return branch
def changed_paths(self) -> list[Path]:
output = self.git("status", "--porcelain=v1", "--untracked-files=all", "-z")
if not output:
return []
entries = output.split("\0")
paths: list[Path] = []
skip_rename_source = False
for entry in entries:
if not entry:
continue
if skip_rename_source:
skip_rename_source = False
continue
if len(entry) < 4:
continue
status, name = entry[:2], entry[3:]
paths.append(Path(name))
if "R" in status or "C" in status:
skip_rename_source = True
return paths
def assert_safe_changes(self) -> list[Path]:
paths = self.changed_paths()
if not paths:
raise WorkItemError("No changes to submit")
violations: list[str] = []
for relative in paths:
portable = relative.as_posix()
allowed = any(fnmatch.fnmatch(portable, pattern) for pattern in self.config.allowed_patterns)
denied = any(fnmatch.fnmatch(portable, pattern) for pattern in self.config.denied_patterns)
if denied and not allowed:
violations.append(f"denied path: {portable}")
continue
candidate = self.root / relative
if candidate.is_file() and candidate.stat().st_size > self.config.max_file_bytes:
violations.append(f"file exceeds {self.config.max_file_bytes} bytes: {portable}")
continue
if candidate.is_file() and candidate.stat().st_size <= self.config.max_file_bytes:
try:
content = candidate.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError):
content = ""
for pattern in SECRET_PATTERNS:
if pattern.search(content):
violations.append(f"possible secret in: {portable}")
break
if violations:
raise WorkItemError("Unsafe changes detected:\n- " + "\n- ".join(violations))
self.git("diff", "--check")
return paths
def stage_commit_push(self, story: Story, branch: str) -> str:
self.git("add", "--all")
staged = self.git("diff", "--cached", "--name-only")
if not staged:
raise WorkItemError("Nothing was staged")
subject = f"{story.story_id}: {story.title}"
self.git("commit", "-m", subject)
self.git("push", "--set-upstream", self.config.remote, branch)
return self.git("rev-parse", "HEAD")
class Gitea:
def __init__(self, root: Path, config: Config, runner: Runner) -> None:
self.root = root
self.config = config
self.runner = runner
def tea(self, *args: str) -> str:
command = ("tea", *args, "--login", self.config.login)
return self.runner.run(command, cwd=self.root).stdout.strip()
def api(self, endpoint: str, *, method: str = "GET", data: Any = None) -> Any:
args = ["api", endpoint, "--method", method]
if data is not None:
args.extend(("--data", json.dumps(data)))
output = self.tea(*args)
return json.loads(output) if output else None
@property
def base(self) -> str:
return f"/repos/{self.config.repo_slug}"
def open_stories(self) -> list[Story]:
issues = self.api(f"{self.base}/issues?state=open&type=issues&limit=100")
stories = [story for issue in issues if (story := Story.from_issue(issue))]
return sorted(stories, key=lambda story: (story.epic, story.sequence, story.number))
def dependencies_closed(self, issue_number: int) -> bool:
dependencies = self.api(f"{self.base}/issues/{issue_number}/dependencies?limit=100")
return all(issue.get("state") == "closed" for issue in dependencies)
def next_story(self) -> Story:
excluded = {"status/in-progress", "status/review", "status/blocked", "status/done"}
for story in self.open_stories():
if story.labels & excluded or story.assignees:
continue
if self.dependencies_closed(story.number):
return story
raise WorkItemError("No eligible open user story is available")
def edit_labels(self, issue: int, *, add: Iterable[str], remove: Iterable[str]) -> None:
args = ["issues", "edit", str(issue), "--repo", self.config.repo_slug]
add_value = ",".join(add)
remove_value = ",".join(remove)
if add_value:
args.extend(("--add-labels", add_value))
if remove_value:
args.extend(("--remove-labels", remove_value))
self.tea(*args)
def claim(self, story: Story) -> None:
self.tea(
"issues", "edit", str(story.number),
"--repo", self.config.repo_slug,
"--add-assignees", self.config.assignee,
"--add-labels", "status/in-progress",
"--remove-labels", "status/backlog,status/ready,status/blocked,status/review,status/done",
)
def comment(self, issue: int, body: str) -> None:
self.tea("comments", "add", str(issue), body, "--repo", self.config.repo_slug)
def create_pr(self, story: Story, branch: str, tests: Sequence[str], commit: str) -> dict[str, Any]:
body = "\n".join((
f"Implements #{story.number}",
"",
"Tests:",
*(f"- `{command}`" for command in tests),
"",
f"Commit: `{commit}`",
))
output = self.tea(
"pulls", "create",
"--repo", self.config.repo_slug,
"--head", branch,
"--base", self.config.main_branch,
"--title", f"{story.story_id}: {story.title}",
"--description", body,
"--output", "json",
)
try:
return json.loads(output)
except json.JSONDecodeError as error:
raise WorkItemError(f"Could not parse created pull request: {output}") from error
def pr(self, number: int) -> dict[str, Any]:
return self.api(f"{self.base}/pulls/{number}")
def merge_pr(self, number: int) -> None:
self.tea("pulls", "merge", str(number), "--repo", self.config.repo_slug, "--style", "squash")
def require_ci_success(self, pr: dict[str, Any]) -> None:
if not self.config.require_ci:
return
sha = pr["head"]["sha"]
combined = self.api(f"{self.base}/commits/{sha}/status")
if combined.get("state") != "success":
raise WorkItemError(f"Required CI is not successful: {combined.get('state', 'unknown')}")
def mark_review(self, issue: int, pr: dict[str, Any], tests: Sequence[str]) -> None:
self.edit_labels(
issue,
add=("status/review",),
remove=("status/in-progress", "status/backlog", "status/ready", "status/blocked", "status/done"),
)
number = pr.get("index") or pr.get("number")
url = pr.get("url") or pr.get("html_url") or f"PR #{number}"
self.comment(issue, f"Submitted for review: {url}\n\nTests passed:\n" + "\n".join(f"- `{x}`" for x in tests))
def mark_done(self, issue: int, pr_number: int) -> None:
self.edit_labels(
issue,
add=("status/done",),
remove=("status/in-progress", "status/backlog", "status/ready", "status/blocked", "status/review"),
)
self.comment(issue, f"Completed and merged via PR #{pr_number}.")
self.tea("issues", "close", str(issue), "--repo", self.config.repo_slug)
class Workflow:
STATE_NAME = "work-item-state.json"
def __init__(self, root: Path, config: Config, runner: Runner | None = None) -> None:
self.root = root.resolve()
self.config = config
self.runner = runner or Runner()
self.git = GitRepo(self.root, config, self.runner)
self.gitea = Gitea(self.root, config, self.runner)
@property
def state_path(self) -> Path:
git_dir = self.git.git("rev-parse", "--git-dir")
path = Path(git_dir)
if not path.is_absolute():
path = self.root / path
return path / self.STATE_NAME
def load_state(self) -> dict[str, Any]:
if not self.state_path.is_file():
raise WorkItemError("No story is currently claimed")
return json.loads(self.state_path.read_text(encoding="utf-8"))
def save_state(self, state: dict[str, Any]) -> None:
temporary = self.state_path.with_suffix(".tmp")
temporary.write_text(json.dumps(state, indent=2, sort_keys=True) + "\n", encoding="utf-8")
temporary.replace(self.state_path)
def clear_state(self) -> None:
self.state_path.unlink(missing_ok=True)
def next(self) -> Story:
self.git.verify()
return self.gitea.next_story()
def claim(self) -> dict[str, Any]:
self.git.verify()
self.git.ensure_clean()
if self.state_path.exists():
raise WorkItemError("A story is already claimed; complete or block it first")
self.git.update_main()
story = self.gitea.next_story()
branch = self.git.create_story_branch(story)
self.gitea.claim(story)
state = {
"issue": story.number,
"story_id": story.story_id,
"title": story.title,
"branch": branch,
"status": "in-progress",
}
self.save_state(state)
self.gitea.comment(story.number, f"Claimed for implementation on branch `{branch}`.")
return state
def status(self) -> dict[str, Any]:
self.git.verify()
state = self.load_state()
return {**state, "current_branch": self.git.current_branch(), "changes": [p.as_posix() for p in self.git.changed_paths()]}
def submit(self, tests: Sequence[str], *, confirmed: bool) -> dict[str, Any]:
self.git.verify()
state = self.load_state()
if self.git.current_branch() != state["branch"]:
raise WorkItemError(f"Expected branch {state['branch']}, found {self.git.current_branch()}")
commands = tuple(dict.fromkeys((*self.config.required_tests, *tests)))
if not commands:
raise WorkItemError("At least one automated test command is required")
self.git.assert_safe_changes()
for command in commands:
self.runner.run_shell(command, cwd=self.root)
paths = self.git.assert_safe_changes()
if not confirmed:
summary = self.git.git("diff", "--stat")
raise WorkItemError(
"Review the changes and rerun with --yes to commit:\n"
f"{summary}\nFiles: {', '.join(path.as_posix() for path in paths)}"
)
story = Story(
number=int(state["issue"]), story_id=state["story_id"], title=state["title"],
epic=int(state["story_id"][2:4]), sequence=int(state["story_id"][5:7]),
labels=frozenset(), assignees=(), raw={},
)
commit = self.git.stage_commit_push(story, state["branch"])
pr = self.gitea.create_pr(story, state["branch"], commands, commit)
pr_number = int(pr.get("index") or pr.get("number"))
state.update({"status": "review", "pr": pr_number, "commit": commit, "tests": list(commands)})
self.save_state(state)
self.gitea.mark_review(story.number, pr, commands)
return state
def complete(self, *, merge: bool) -> dict[str, Any]:
self.git.verify()
state = self.load_state()
if state.get("status") != "review" or not state.get("pr"):
raise WorkItemError("The claimed story has not been submitted for review")
pr = self.gitea.pr(int(state["pr"]))
self.gitea.require_ci_success(pr)
if not pr.get("merged"):
if not merge:
raise WorkItemError("Pull request is not merged; rerun with --merge after review")
self.gitea.merge_pr(int(state["pr"]))
pr = self.gitea.pr(int(state["pr"]))
if not pr.get("merged"):
raise WorkItemError("Pull request merge could not be verified")
self.gitea.mark_done(int(state["issue"]), int(state["pr"]))
branch = state["branch"]
self.git.git("switch", self.config.main_branch)
self.git.git("pull", "--ff-only", self.config.remote, self.config.main_branch)
# A squash-merged PR is verified by Gitea but its feature commit is not an
# ancestor of main, so ordinary `-d` would incorrectly reject safe cleanup.
self.git.git("branch", "-D", branch)
result = {**state, "status": "done"}
self.clear_state()
return result
def block(self, reason: str) -> dict[str, Any]:
self.git.verify()
if not reason.strip():
raise WorkItemError("A non-empty blocking reason is required")
state = self.load_state()
self.gitea.edit_labels(
int(state["issue"]),
add=("status/blocked",),
remove=("status/in-progress", "status/ready", "status/review", "status/done"),
)
self.gitea.comment(int(state["issue"]), f"Blocked: {reason.strip()}")
state.update({"status": "blocked", "reason": reason.strip()})
self.save_state(state)
return state
def find_root(start: Path) -> Path:
result = subprocess.run(
("git", "rev-parse", "--show-toplevel"), cwd=start, text=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
)
if result.returncode:
raise WorkItemError("This command must run inside the photoanalyzer Git repository")
return Path(result.stdout.strip()).resolve()
def parser() -> argparse.ArgumentParser:
result = argparse.ArgumentParser(prog="work-item", description="Safe issue/Git workflow helper")
result.add_argument("--config", default=".work-item.toml")
commands = result.add_subparsers(dest="command", required=True)
commands.add_parser("next", help="Show the top eligible user story")
commands.add_parser("claim", help="Claim the top story and create its branch")
commands.add_parser("status", help="Show the current claim and Git state")
submit = commands.add_parser("submit", help="Test, commit, push, and open a pull request")
submit.add_argument("--test", action="append", default=[], help="Additional automated test command; repeatable")
submit.add_argument("--yes", action="store_true", help="Confirm the reviewed diff may be committed")
complete = commands.add_parser("complete", help="Verify merge, close the issue, and return to main")
complete.add_argument("--merge", action="store_true", help="Merge the reviewed PR before completing")
block = commands.add_parser("block", help="Mark the current story blocked")
block.add_argument("--reason", required=True)
return result
def main(argv: Sequence[str] | None = None) -> int:
args = parser().parse_args(argv)
try:
root = find_root(Path.cwd())
config_path = Path(args.config)
if not config_path.is_absolute():
config_path = root / config_path
workflow = Workflow(root, Config.load(config_path))
if args.command == "next":
story = workflow.next()
payload: Any = {"issue": story.number, "story_id": story.story_id, "title": story.title}
elif args.command == "claim":
payload = workflow.claim()
elif args.command == "status":
payload = workflow.status()
elif args.command == "submit":
payload = workflow.submit(args.test, confirmed=args.yes)
elif args.command == "complete":
payload = workflow.complete(merge=args.merge)
else:
payload = workflow.block(args.reason)
print(json.dumps(payload, indent=2, sort_keys=True))
return 0
except WorkItemError as error:
print(f"work-item: {error}", file=sys.stderr)
return 2
if __name__ == "__main__":
raise SystemExit(main())