258 lines
12 KiB
Markdown
258 lines
12 KiB
Markdown
# Rich Live Dashboard — Upgrade Plan
|
||
|
||
Replace the scrolling `Progress` context in `photo_analyzer.py` with a persistent
|
||
`Live + Layout` that shows **album progress**, a **photo feed**, and the **progress
|
||
bar** simultaneously — without touching the DB logic or threading model.
|
||
|
||
> **Focus: robustness.** The dashboard is a *view layer only* — it reads state,
|
||
> never owns it. SQLite stays the source of truth at all times.
|
||
|
||
---
|
||
|
||
## Target layout
|
||
|
||
```
|
||
┌─ python photo_analyzer.py --library pictures/ ─────────────────┐
|
||
│ ┌─ Album Progress — leaf folder = album ───────────────────┐ │
|
||
│ │ Camera Roll 238 / 550 ████████░░░░░░░░░░░ 43% │ │
|
||
│ │ Urlaub/Rom 45 / 45 ██████████████████ ✓ │ │
|
||
│ │ Urlaub/Venedig 12 / 60 ███░░░░░░░░░░░░░░░░ 20% │ │
|
||
│ │ WhatsApp Images 0 / 120 ░░░░░░░░░░░░░░░░░░ 0% │ │
|
||
│ └──────────────────────────────────────────────────────────┘ │
|
||
│ ┌─ Recent ─────────────────────────────────────────────────┐ │
|
||
│ │ ✓ IMG_4821.jpg │ │
|
||
│ │ A golden retriever playing fetch on a sunny beach │ │
|
||
│ │ ✗ DSC_0042.jpg — Exhausted 4 retries: connection timeout │ │
|
||
│ │ ✓ IMG_4820.jpg │ │
|
||
│ │ Two children building a sandcastle at sunset │ │
|
||
│ └──────────────────────────────────────────────────────────┘ │
|
||
│ ┌─ Analyzing — Camera Roll ────────────────────────────────┐ │
|
||
│ │ ████████░░░░░░░░░░ 238/550 43% ok=235 err=3 │ │
|
||
│ │ tokens=284,192 est. remaining ~340,000 ETA 14:23 │ │
|
||
│ └──────────────────────────────────────────────────────────┘ │
|
||
└────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Core rule: album = leaf folder
|
||
|
||
An **album is the leaf folder directly containing the photos**, not a top-level
|
||
parent. `Urlaub/Rom` and `Urlaub/Venedig` are two separate albums, never collapsed
|
||
into `Urlaub`. Key every stats dict on `album_label(path, library)` (already added
|
||
to the script for `--stats`):
|
||
|
||
```python
|
||
def album_label(photo: Path, library: Path = None) -> str:
|
||
"""Leaf folder relative to library: Urlaub/Rom/IMG.jpg -> 'Urlaub/Rom'."""
|
||
parent = photo.parent
|
||
if library:
|
||
try:
|
||
rel = parent.relative_to(library)
|
||
return "(root)" if str(rel) == "." else str(rel)
|
||
except ValueError:
|
||
pass
|
||
return parent.name or str(parent)
|
||
```
|
||
|
||
The analysis loop already groups on `Path(p).parent` (the leaf), so each sub-album
|
||
completes fully before the next starts. Only the *display* side needs this fix.
|
||
|
||
---
|
||
|
||
## Phases
|
||
|
||
### Phase 01 — Scaffold Live + Layout *(structural, ~30 min)*
|
||
|
||
Replace `with Progress(...) as progress:` with a `Live` context wrapping a
|
||
`Layout`. **Progress becomes a renderable, not a context manager** — it lives
|
||
inside the layout rather than owning the terminal. All DB logic, threading, and
|
||
retry logic stays unchanged.
|
||
|
||
```python
|
||
from rich.layout import Layout
|
||
from rich.live import Live
|
||
|
||
layout = Layout()
|
||
layout.split_column(
|
||
Layout(name="stats", ratio=2),
|
||
Layout(name="feed", ratio=2),
|
||
Layout(name="progress", size=5),
|
||
)
|
||
progress = Progress(...) # same columns as today
|
||
task_id = progress.add_task(...)
|
||
layout["progress"].update(progress)
|
||
|
||
with Live(layout, console=console, refresh_per_second=4, transient=False):
|
||
# ... existing folder loop unchanged ...
|
||
```
|
||
|
||
- **Test:** script starts, layout renders with placeholder panels, progress bar
|
||
advances, Ctrl-C stops cleanly.
|
||
- Do not move on until this is stable — later phases build on this scaffold.
|
||
|
||
---
|
||
|
||
### Phase 02 — Thread-safe log buffer *(concurrency, ~20 min)*
|
||
|
||
Workers must **never write to the terminal directly**. Replace every
|
||
`progress.console.print()` call with an append to a shared `deque`. The Live
|
||
renderer reads from the deque on each refresh cycle and renders it as the feed
|
||
panel. Threads write, the renderer reads — no other direction.
|
||
|
||
```python
|
||
from collections import deque
|
||
|
||
log_buffer = deque(maxlen=10) # newest first
|
||
log_lock = threading.Lock()
|
||
|
||
# in process() — replace progress.console.print(...) with:
|
||
with log_lock:
|
||
log_buffer.appendleft(entry)
|
||
|
||
# in the Live render function:
|
||
def render_feed() -> Table:
|
||
with log_lock:
|
||
rows = list(log_buffer) # snapshot
|
||
# build Rich Table from rows ...
|
||
```
|
||
|
||
- **Never** call `live.console.print()` from a worker thread — always via the buffer.
|
||
- `deque(maxlen=10)` auto-evicts old entries; no memory growth.
|
||
- The lock protects the *snapshot* read, not the render.
|
||
|
||
---
|
||
|
||
### Phase 03 — In-memory album stats *(performance, ~20 min)*
|
||
|
||
The stats panel refreshes at 4 Hz — **no DB queries in the render loop**. Key
|
||
every dict on `album_label(path, library)`. Compute `folder_total` from the sorted
|
||
`pending` list, seed `folder_done` from one upfront DB query, and have workers
|
||
increment it in memory after each success.
|
||
|
||
```python
|
||
# album = leaf folder relative to library — reuse the existing helper
|
||
folder_total: dict[str, int] = {}
|
||
folder_done: dict[str, int] = {}
|
||
|
||
for folder, paths in folder_groups: # folder = abs leaf dir
|
||
album = album_label(Path(paths[0]), library)
|
||
folder_total[album] = len(paths)
|
||
folder_done[album] = 0
|
||
|
||
# Seed already-done counts from DB (single query):
|
||
for path, status in conn.execute(
|
||
"SELECT path, status FROM photos WHERE status IN ('analyzed','exif_written')"
|
||
):
|
||
album = album_label(Path(path), library)
|
||
if album in folder_done:
|
||
folder_done[album] += 1
|
||
|
||
# In the worker result handler (album computed once per folder):
|
||
with token_lock: # reuse existing lock
|
||
folder_done[album] = folder_done.get(album, 0) + 1
|
||
```
|
||
|
||
- **Album = leaf folder, not parent.** Nested trees like `Urlaub/*` become one
|
||
row per sub-album.
|
||
- Render reads the dicts under the existing `token_lock` — no new lock needed.
|
||
- Shows **all** albums (future ones at 0%); long paths left-truncated via `fit_label()`.
|
||
- Errors do *not* increment `folder_done` — an album shows ✓ only when every
|
||
photo succeeded.
|
||
|
||
---
|
||
|
||
### Phase 04 — Non-TTY fallback *(safety, ~10 min)*
|
||
|
||
Rich's `Live` behaves unexpectedly when stdout is piped or redirected (log
|
||
capture, CI, `nohup`). **Gate the Live path on `console.is_terminal`.** Extract
|
||
the current Progress-based loop into `_run_analysis_plain()` unchanged; the new
|
||
Live implementation goes in `_run_analysis_live()`.
|
||
|
||
```python
|
||
def run_analysis(args, conn, client):
|
||
... # shared setup (discover, pending, folder_groups)
|
||
if console.is_terminal:
|
||
_run_analysis_live(args, conn, client, pending, folder_groups, ...)
|
||
else:
|
||
_run_analysis_plain(args, conn, client, pending, folder_groups, ...)
|
||
```
|
||
|
||
- Plain mode keeps every existing behavior — no regression for scripted use.
|
||
- File logs always write regardless of mode.
|
||
- No flag needed — detection is automatic.
|
||
|
||
---
|
||
|
||
### Phase 05 — Error containment *(robustness, ~15 min)*
|
||
|
||
The Live layer must never take down the pipeline. **DB writes happen in worker
|
||
threads, outside the render path** — a renderer crash cannot corrupt the
|
||
database. Wrap Live construction in try/except: if Rich fails to start, fall back
|
||
to plain mode. SIGINT already works (`_stop` set, executor drains, `with` exits).
|
||
|
||
```python
|
||
try:
|
||
with Live(layout, console=console, refresh_per_second=4):
|
||
_run_folder_loop(executor, folder_groups, ...)
|
||
except Exception as e:
|
||
log.warning(f"Live render failed ({e}), falling back to plain mode")
|
||
_run_analysis_plain(args, conn, client, pending, folder_groups, ...)
|
||
```
|
||
|
||
- **Worker exception path:** `mark_error()` runs before anything is appended to
|
||
`log_buffer` — DB is always consistent even if the render crashes mid-frame.
|
||
- **Render exceptions:** catch inside the render function, return a plain
|
||
`Text("render error")` panel — never let a bad render kill the loop.
|
||
- **Resume safety:** unchanged — `--stats` and restart behavior identical to today.
|
||
|
||
---
|
||
|
||
## Robustness constraints
|
||
|
||
| Constraint | Detail |
|
||
|---|---|
|
||
| No DB queries in render path | Renderer fires 4×/s. Album counts kept in in-memory dicts, seeded once at startup. SQLite never touched during animation. |
|
||
| Workers never touch the terminal | Only interface between workers and display is `log_buffer` (deque + lock). Rich's Live is single-threaded by design. |
|
||
| Render errors isolated | Each panel's render function wraps its body in try/except, returns a fallback `Text` on failure. A bad render is cosmetic, not fatal. |
|
||
| Ctrl-C still works | `_stop` event and SIGINT handler unchanged. Live context exits when the `with` block ends. In-flight requests complete; queued ones cancel. |
|
||
| Non-TTY produces no garbage | Piped output, `nohup`, log capture → plain mode via `console.is_terminal`. No ANSI escapes leak into log files. |
|
||
| Resume unchanged | All state lives in SQLite. Kill at any point; restart picks up where it left off. Dashboard reads state, never owns it. |
|
||
|
||
---
|
||
|
||
## Status: implemented ✓ (2026-06-22)
|
||
|
||
All five phases shipped. Old script archived as `photo_analyzer.py.bak-20260622-091208`.
|
||
|
||
Structure: `run_analysis()` does shared setup, then dispatches to
|
||
`_run_analysis_live()` (TTY) or `_run_analysis_plain()` (piped/redirected), both
|
||
driving the shared `_run_folder_loop()` engine. Live uses a `_Dashboard.__rich__`
|
||
that rebuilds stats/feed/progress panels each tick from in-memory dicts + a deque.
|
||
|
||
**Extra hardening found during testing (not in original plan):** the shared
|
||
`sqlite3.Connection` was committed from multiple worker threads, racing with
|
||
"cannot commit - no transaction is active". Real API latency hid it; instant
|
||
dry-run exposed it. Fixed with a module-level `_db_lock` serialising all writes
|
||
(`mark_analyzed` / `mark_error` / `mark_exif_written` / `upsert_pending`).
|
||
Verified: 200 photos × 8 workers, zero spurious errors.
|
||
|
||
Tested: plain mode (piped), live mode (PTY, 6 and 200 photos), Live-failure
|
||
fallback, render-error isolation, album windowing (>14 albums summarised),
|
||
leaf-folder album grouping.
|
||
|
||
---
|
||
|
||
## Implementation order
|
||
|
||
1. **Phase 1** — scaffold Live+Layout. Test: layout renders, progress bar
|
||
advances, Ctrl-C exits. *(must pass before continuing)*
|
||
2. **Phase 3** — in-memory album stats panel. Test: album rows appear with correct
|
||
counts, numbers increment live as photos complete.
|
||
3. **Phase 2** — log buffer + feed panel. Test: photo results appear in the feed,
|
||
errors show in red, old entries scroll off.
|
||
4. **Phase 4** — non-TTY fallback. Test: `python photo_analyzer.py ... | cat`
|
||
produces clean plain output.
|
||
5. **Phase 5** — error containment. Test: force a render exception, confirm
|
||
pipeline continues; interrupt mid-album, confirm DB is consistent on restart.
|