Bootstrap project and safe work-item workflow
This commit is contained in:
257
DASHBOARD_PLAN.md
Normal file
257
DASHBOARD_PLAN.md
Normal file
@@ -0,0 +1,257 @@
|
||||
# Rich Live Dashboard — Upgrade Plan
|
||||
|
||||
Replace the scrolling `Progress` context in `photo_analyzer.py` with a persistent
|
||||
`Live + Layout` that shows **album progress**, a **photo feed**, and the **progress
|
||||
bar** simultaneously — without touching the DB logic or threading model.
|
||||
|
||||
> **Focus: robustness.** The dashboard is a *view layer only* — it reads state,
|
||||
> never owns it. SQLite stays the source of truth at all times.
|
||||
|
||||
---
|
||||
|
||||
## Target layout
|
||||
|
||||
```
|
||||
┌─ python photo_analyzer.py --library pictures/ ─────────────────┐
|
||||
│ ┌─ Album Progress — leaf folder = album ───────────────────┐ │
|
||||
│ │ Camera Roll 238 / 550 ████████░░░░░░░░░░░ 43% │ │
|
||||
│ │ Urlaub/Rom 45 / 45 ██████████████████ ✓ │ │
|
||||
│ │ Urlaub/Venedig 12 / 60 ███░░░░░░░░░░░░░░░░ 20% │ │
|
||||
│ │ WhatsApp Images 0 / 120 ░░░░░░░░░░░░░░░░░░ 0% │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ ┌─ Recent ─────────────────────────────────────────────────┐ │
|
||||
│ │ ✓ IMG_4821.jpg │ │
|
||||
│ │ A golden retriever playing fetch on a sunny beach │ │
|
||||
│ │ ✗ DSC_0042.jpg — Exhausted 4 retries: connection timeout │ │
|
||||
│ │ ✓ IMG_4820.jpg │ │
|
||||
│ │ Two children building a sandcastle at sunset │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ ┌─ Analyzing — Camera Roll ────────────────────────────────┐ │
|
||||
│ │ ████████░░░░░░░░░░ 238/550 43% ok=235 err=3 │ │
|
||||
│ │ tokens=284,192 est. remaining ~340,000 ETA 14:23 │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core rule: album = leaf folder
|
||||
|
||||
An **album is the leaf folder directly containing the photos**, not a top-level
|
||||
parent. `Urlaub/Rom` and `Urlaub/Venedig` are two separate albums, never collapsed
|
||||
into `Urlaub`. Key every stats dict on `album_label(path, library)` (already added
|
||||
to the script for `--stats`):
|
||||
|
||||
```python
|
||||
def album_label(photo: Path, library: Path = None) -> str:
|
||||
"""Leaf folder relative to library: Urlaub/Rom/IMG.jpg -> 'Urlaub/Rom'."""
|
||||
parent = photo.parent
|
||||
if library:
|
||||
try:
|
||||
rel = parent.relative_to(library)
|
||||
return "(root)" if str(rel) == "." else str(rel)
|
||||
except ValueError:
|
||||
pass
|
||||
return parent.name or str(parent)
|
||||
```
|
||||
|
||||
The analysis loop already groups on `Path(p).parent` (the leaf), so each sub-album
|
||||
completes fully before the next starts. Only the *display* side needs this fix.
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 01 — Scaffold Live + Layout *(structural, ~30 min)*
|
||||
|
||||
Replace `with Progress(...) as progress:` with a `Live` context wrapping a
|
||||
`Layout`. **Progress becomes a renderable, not a context manager** — it lives
|
||||
inside the layout rather than owning the terminal. All DB logic, threading, and
|
||||
retry logic stays unchanged.
|
||||
|
||||
```python
|
||||
from rich.layout import Layout
|
||||
from rich.live import Live
|
||||
|
||||
layout = Layout()
|
||||
layout.split_column(
|
||||
Layout(name="stats", ratio=2),
|
||||
Layout(name="feed", ratio=2),
|
||||
Layout(name="progress", size=5),
|
||||
)
|
||||
progress = Progress(...) # same columns as today
|
||||
task_id = progress.add_task(...)
|
||||
layout["progress"].update(progress)
|
||||
|
||||
with Live(layout, console=console, refresh_per_second=4, transient=False):
|
||||
# ... existing folder loop unchanged ...
|
||||
```
|
||||
|
||||
- **Test:** script starts, layout renders with placeholder panels, progress bar
|
||||
advances, Ctrl-C stops cleanly.
|
||||
- Do not move on until this is stable — later phases build on this scaffold.
|
||||
|
||||
---
|
||||
|
||||
### Phase 02 — Thread-safe log buffer *(concurrency, ~20 min)*
|
||||
|
||||
Workers must **never write to the terminal directly**. Replace every
|
||||
`progress.console.print()` call with an append to a shared `deque`. The Live
|
||||
renderer reads from the deque on each refresh cycle and renders it as the feed
|
||||
panel. Threads write, the renderer reads — no other direction.
|
||||
|
||||
```python
|
||||
from collections import deque
|
||||
|
||||
log_buffer = deque(maxlen=10) # newest first
|
||||
log_lock = threading.Lock()
|
||||
|
||||
# in process() — replace progress.console.print(...) with:
|
||||
with log_lock:
|
||||
log_buffer.appendleft(entry)
|
||||
|
||||
# in the Live render function:
|
||||
def render_feed() -> Table:
|
||||
with log_lock:
|
||||
rows = list(log_buffer) # snapshot
|
||||
# build Rich Table from rows ...
|
||||
```
|
||||
|
||||
- **Never** call `live.console.print()` from a worker thread — always via the buffer.
|
||||
- `deque(maxlen=10)` auto-evicts old entries; no memory growth.
|
||||
- The lock protects the *snapshot* read, not the render.
|
||||
|
||||
---
|
||||
|
||||
### Phase 03 — In-memory album stats *(performance, ~20 min)*
|
||||
|
||||
The stats panel refreshes at 4 Hz — **no DB queries in the render loop**. Key
|
||||
every dict on `album_label(path, library)`. Compute `folder_total` from the sorted
|
||||
`pending` list, seed `folder_done` from one upfront DB query, and have workers
|
||||
increment it in memory after each success.
|
||||
|
||||
```python
|
||||
# album = leaf folder relative to library — reuse the existing helper
|
||||
folder_total: dict[str, int] = {}
|
||||
folder_done: dict[str, int] = {}
|
||||
|
||||
for folder, paths in folder_groups: # folder = abs leaf dir
|
||||
album = album_label(Path(paths[0]), library)
|
||||
folder_total[album] = len(paths)
|
||||
folder_done[album] = 0
|
||||
|
||||
# Seed already-done counts from DB (single query):
|
||||
for path, status in conn.execute(
|
||||
"SELECT path, status FROM photos WHERE status IN ('analyzed','exif_written')"
|
||||
):
|
||||
album = album_label(Path(path), library)
|
||||
if album in folder_done:
|
||||
folder_done[album] += 1
|
||||
|
||||
# In the worker result handler (album computed once per folder):
|
||||
with token_lock: # reuse existing lock
|
||||
folder_done[album] = folder_done.get(album, 0) + 1
|
||||
```
|
||||
|
||||
- **Album = leaf folder, not parent.** Nested trees like `Urlaub/*` become one
|
||||
row per sub-album.
|
||||
- Render reads the dicts under the existing `token_lock` — no new lock needed.
|
||||
- Shows **all** albums (future ones at 0%); long paths left-truncated via `fit_label()`.
|
||||
- Errors do *not* increment `folder_done` — an album shows ✓ only when every
|
||||
photo succeeded.
|
||||
|
||||
---
|
||||
|
||||
### Phase 04 — Non-TTY fallback *(safety, ~10 min)*
|
||||
|
||||
Rich's `Live` behaves unexpectedly when stdout is piped or redirected (log
|
||||
capture, CI, `nohup`). **Gate the Live path on `console.is_terminal`.** Extract
|
||||
the current Progress-based loop into `_run_analysis_plain()` unchanged; the new
|
||||
Live implementation goes in `_run_analysis_live()`.
|
||||
|
||||
```python
|
||||
def run_analysis(args, conn, client):
|
||||
... # shared setup (discover, pending, folder_groups)
|
||||
if console.is_terminal:
|
||||
_run_analysis_live(args, conn, client, pending, folder_groups, ...)
|
||||
else:
|
||||
_run_analysis_plain(args, conn, client, pending, folder_groups, ...)
|
||||
```
|
||||
|
||||
- Plain mode keeps every existing behavior — no regression for scripted use.
|
||||
- File logs always write regardless of mode.
|
||||
- No flag needed — detection is automatic.
|
||||
|
||||
---
|
||||
|
||||
### Phase 05 — Error containment *(robustness, ~15 min)*
|
||||
|
||||
The Live layer must never take down the pipeline. **DB writes happen in worker
|
||||
threads, outside the render path** — a renderer crash cannot corrupt the
|
||||
database. Wrap Live construction in try/except: if Rich fails to start, fall back
|
||||
to plain mode. SIGINT already works (`_stop` set, executor drains, `with` exits).
|
||||
|
||||
```python
|
||||
try:
|
||||
with Live(layout, console=console, refresh_per_second=4):
|
||||
_run_folder_loop(executor, folder_groups, ...)
|
||||
except Exception as e:
|
||||
log.warning(f"Live render failed ({e}), falling back to plain mode")
|
||||
_run_analysis_plain(args, conn, client, pending, folder_groups, ...)
|
||||
```
|
||||
|
||||
- **Worker exception path:** `mark_error()` runs before anything is appended to
|
||||
`log_buffer` — DB is always consistent even if the render crashes mid-frame.
|
||||
- **Render exceptions:** catch inside the render function, return a plain
|
||||
`Text("render error")` panel — never let a bad render kill the loop.
|
||||
- **Resume safety:** unchanged — `--stats` and restart behavior identical to today.
|
||||
|
||||
---
|
||||
|
||||
## Robustness constraints
|
||||
|
||||
| Constraint | Detail |
|
||||
|---|---|
|
||||
| No DB queries in render path | Renderer fires 4×/s. Album counts kept in in-memory dicts, seeded once at startup. SQLite never touched during animation. |
|
||||
| Workers never touch the terminal | Only interface between workers and display is `log_buffer` (deque + lock). Rich's Live is single-threaded by design. |
|
||||
| Render errors isolated | Each panel's render function wraps its body in try/except, returns a fallback `Text` on failure. A bad render is cosmetic, not fatal. |
|
||||
| Ctrl-C still works | `_stop` event and SIGINT handler unchanged. Live context exits when the `with` block ends. In-flight requests complete; queued ones cancel. |
|
||||
| Non-TTY produces no garbage | Piped output, `nohup`, log capture → plain mode via `console.is_terminal`. No ANSI escapes leak into log files. |
|
||||
| Resume unchanged | All state lives in SQLite. Kill at any point; restart picks up where it left off. Dashboard reads state, never owns it. |
|
||||
|
||||
---
|
||||
|
||||
## Status: implemented ✓ (2026-06-22)
|
||||
|
||||
All five phases shipped. Old script archived as `photo_analyzer.py.bak-20260622-091208`.
|
||||
|
||||
Structure: `run_analysis()` does shared setup, then dispatches to
|
||||
`_run_analysis_live()` (TTY) or `_run_analysis_plain()` (piped/redirected), both
|
||||
driving the shared `_run_folder_loop()` engine. Live uses a `_Dashboard.__rich__`
|
||||
that rebuilds stats/feed/progress panels each tick from in-memory dicts + a deque.
|
||||
|
||||
**Extra hardening found during testing (not in original plan):** the shared
|
||||
`sqlite3.Connection` was committed from multiple worker threads, racing with
|
||||
"cannot commit - no transaction is active". Real API latency hid it; instant
|
||||
dry-run exposed it. Fixed with a module-level `_db_lock` serialising all writes
|
||||
(`mark_analyzed` / `mark_error` / `mark_exif_written` / `upsert_pending`).
|
||||
Verified: 200 photos × 8 workers, zero spurious errors.
|
||||
|
||||
Tested: plain mode (piped), live mode (PTY, 6 and 200 photos), Live-failure
|
||||
fallback, render-error isolation, album windowing (>14 albums summarised),
|
||||
leaf-folder album grouping.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. **Phase 1** — scaffold Live+Layout. Test: layout renders, progress bar
|
||||
advances, Ctrl-C exits. *(must pass before continuing)*
|
||||
2. **Phase 3** — in-memory album stats panel. Test: album rows appear with correct
|
||||
counts, numbers increment live as photos complete.
|
||||
3. **Phase 2** — log buffer + feed panel. Test: photo results appear in the feed,
|
||||
errors show in red, old entries scroll off.
|
||||
4. **Phase 4** — non-TTY fallback. Test: `python photo_analyzer.py ... | cat`
|
||||
produces clean plain output.
|
||||
5. **Phase 5** — error containment. Test: force a render exception, confirm
|
||||
pipeline continues; interrupt mid-album, confirm DB is consistent on restart.
|
||||
Reference in New Issue
Block a user