Rich Live Dashboard — Upgrade Plan

Replace the scrolling Progress context in photo_analyzer.py with a persistent Live + Layout that shows album progress, a photo feed, and the progress bar simultaneously — without touching the DB logic or threading model.

Focus: robustness. The dashboard is a view layer only — it reads state, never owns it. SQLite stays the source of truth at all times.

Target layout

┌─ python photo_analyzer.py --library pictures/ ─────────────────┐
│ ┌─ Album Progress — leaf folder = album ───────────────────┐   │
│ │ Camera Roll        238 / 550  ████████░░░░░░░░░░░  43%    │   │
│ │ Urlaub/Rom          45 /  45  ██████████████████   ✓      │   │
│ │ Urlaub/Venedig      12 /  60  ███░░░░░░░░░░░░░░░░   20%    │   │
│ │ WhatsApp Images      0 / 120  ░░░░░░░░░░░░░░░░░░    0%     │   │
│ └──────────────────────────────────────────────────────────┘   │
│ ┌─ Recent ─────────────────────────────────────────────────┐   │
│ │ ✓  IMG_4821.jpg                                           │   │
│ │      A golden retriever playing fetch on a sunny beach    │   │
│ │ ✗  DSC_0042.jpg — Exhausted 4 retries: connection timeout │   │
│ │ ✓  IMG_4820.jpg                                           │   │
│ │      Two children building a sandcastle at sunset         │   │
│ └──────────────────────────────────────────────────────────┘   │
│ ┌─ Analyzing — Camera Roll ────────────────────────────────┐   │
│ │ ████████░░░░░░░░░░  238/550  43%  ok=235 err=3            │   │
│ │ tokens=284,192  est. remaining ~340,000  ETA 14:23        │   │
│ └──────────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────────┘

Core rule: album = leaf folder

An album is the leaf folder directly containing the photos, not a top-level parent. Urlaub/Rom and Urlaub/Venedig are two separate albums, never collapsed into Urlaub. Key every stats dict on album_label(path, library) (already added to the script for --stats):

def album_label(photo: Path, library: Path = None) -> str:
    """Leaf folder relative to library: Urlaub/Rom/IMG.jpg -> 'Urlaub/Rom'."""
    parent = photo.parent
    if library:
        try:
            rel = parent.relative_to(library)
            return "(root)" if str(rel) == "." else str(rel)
        except ValueError:
            pass
    return parent.name or str(parent)

The analysis loop already groups on Path(p).parent (the leaf), so each sub-album completes fully before the next starts. Only the display side needs this fix.

Phases

Phase 01 — Scaffold Live + Layout (structural, ~30 min)

Replace with Progress(...) as progress: with a Live context wrapping a Layout. Progress becomes a renderable, not a context manager — it lives inside the layout rather than owning the terminal. All DB logic, threading, and retry logic stays unchanged.

from rich.layout import Layout
from rich.live   import Live

layout = Layout()
layout.split_column(
    Layout(name="stats",    ratio=2),
    Layout(name="feed",     ratio=2),
    Layout(name="progress", size=5),
)
progress = Progress(...)        # same columns as today
task_id  = progress.add_task(...)
layout["progress"].update(progress)

with Live(layout, console=console, refresh_per_second=4, transient=False):
    # ... existing folder loop unchanged ...

Test: script starts, layout renders with placeholder panels, progress bar advances, Ctrl-C stops cleanly.
Do not move on until this is stable — later phases build on this scaffold.

Phase 02 — Thread-safe log buffer (concurrency, ~20 min)

Workers must never write to the terminal directly. Replace every progress.console.print() call with an append to a shared deque. The Live renderer reads from the deque on each refresh cycle and renders it as the feed panel. Threads write, the renderer reads — no other direction.

from collections import deque

log_buffer = deque(maxlen=10)   # newest first
log_lock   = threading.Lock()

# in process() — replace progress.console.print(...) with:
with log_lock:
    log_buffer.appendleft(entry)

# in the Live render function:
def render_feed() -> Table:
    with log_lock:
        rows = list(log_buffer)   # snapshot
    # build Rich Table from rows ...

Never call live.console.print() from a worker thread — always via the buffer.
deque(maxlen=10) auto-evicts old entries; no memory growth.
The lock protects the snapshot read, not the render.

Phase 03 — In-memory album stats (performance, ~20 min)

The stats panel refreshes at 4 Hz — no DB queries in the render loop. Key every dict on album_label(path, library). Compute folder_total from the sorted pending list, seed folder_done from one upfront DB query, and have workers increment it in memory after each success.

# album = leaf folder relative to library — reuse the existing helper
folder_total: dict[str, int] = {}
folder_done:  dict[str, int] = {}

for folder, paths in folder_groups:     # folder = abs leaf dir
    album = album_label(Path(paths[0]), library)
    folder_total[album] = len(paths)
    folder_done[album]  = 0

# Seed already-done counts from DB (single query):
for path, status in conn.execute(
    "SELECT path, status FROM photos WHERE status IN ('analyzed','exif_written')"
):
    album = album_label(Path(path), library)
    if album in folder_done:
        folder_done[album] += 1

# In the worker result handler (album computed once per folder):
with token_lock:                        # reuse existing lock
    folder_done[album] = folder_done.get(album, 0) + 1

Album = leaf folder, not parent. Nested trees like Urlaub/* become one row per sub-album.
Render reads the dicts under the existing token_lock — no new lock needed.
Shows all albums (future ones at 0%); long paths left-truncated via fit_label().
Errors do not increment folder_done — an album shows ✓ only when every photo succeeded.

Phase 04 — Non-TTY fallback (safety, ~10 min)

Rich's Live behaves unexpectedly when stdout is piped or redirected (log capture, CI, nohup). Gate the Live path on console.is_terminal. Extract the current Progress-based loop into _run_analysis_plain() unchanged; the new Live implementation goes in _run_analysis_live().

def run_analysis(args, conn, client):
    ...  # shared setup (discover, pending, folder_groups)
    if console.is_terminal:
        _run_analysis_live(args, conn, client, pending, folder_groups, ...)
    else:
        _run_analysis_plain(args, conn, client, pending, folder_groups, ...)

Plain mode keeps every existing behavior — no regression for scripted use.
File logs always write regardless of mode.
No flag needed — detection is automatic.

Phase 05 — Error containment (robustness, ~15 min)

The Live layer must never take down the pipeline. DB writes happen in worker threads, outside the render path — a renderer crash cannot corrupt the database. Wrap Live construction in try/except: if Rich fails to start, fall back to plain mode. SIGINT already works (_stop set, executor drains, with exits).

try:
    with Live(layout, console=console, refresh_per_second=4):
        _run_folder_loop(executor, folder_groups, ...)
except Exception as e:
    log.warning(f"Live render failed ({e}), falling back to plain mode")
    _run_analysis_plain(args, conn, client, pending, folder_groups, ...)

Worker exception path: mark_error() runs before anything is appended to log_buffer — DB is always consistent even if the render crashes mid-frame.
Render exceptions: catch inside the render function, return a plain Text("render error") panel — never let a bad render kill the loop.
Resume safety: unchanged — --stats and restart behavior identical to today.

Robustness constraints

Constraint	Detail
No DB queries in render path	Renderer fires 4×/s. Album counts kept in in-memory dicts, seeded once at startup. SQLite never touched during animation.
Workers never touch the terminal	Only interface between workers and display is `log_buffer` (deque + lock). Rich's Live is single-threaded by design.
Render errors isolated	Each panel's render function wraps its body in try/except, returns a fallback `Text` on failure. A bad render is cosmetic, not fatal.
Ctrl-C still works	`_stop` event and SIGINT handler unchanged. Live context exits when the `with` block ends. In-flight requests complete; queued ones cancel.
Non-TTY produces no garbage	Piped output, `nohup`, log capture → plain mode via `console.is_terminal`. No ANSI escapes leak into log files.
Resume unchanged	All state lives in SQLite. Kill at any point; restart picks up where it left off. Dashboard reads state, never owns it.

Status: implemented ✓ (2026-06-22)

All five phases shipped. Old script archived as photo_analyzer.py.bak-20260622-091208.

Structure: run_analysis() does shared setup, then dispatches to _run_analysis_live() (TTY) or _run_analysis_plain() (piped/redirected), both driving the shared _run_folder_loop() engine. Live uses a _Dashboard.__rich__ that rebuilds stats/feed/progress panels each tick from in-memory dicts + a deque.

Extra hardening found during testing (not in original plan): the shared sqlite3.Connection was committed from multiple worker threads, racing with "cannot commit - no transaction is active". Real API latency hid it; instant dry-run exposed it. Fixed with a module-level _db_lock serialising all writes (mark_analyzed / mark_error / mark_exif_written / upsert_pending). Verified: 200 photos × 8 workers, zero spurious errors.

Tested: plain mode (piped), live mode (PTY, 6 and 200 photos), Live-failure fallback, render-error isolation, album windowing (>14 albums summarised), leaf-folder album grouping.

Implementation order

Phase 1 — scaffold Live+Layout. Test: layout renders, progress bar advances, Ctrl-C exits. (must pass before continuing)
Phase 3 — in-memory album stats panel. Test: album rows appear with correct counts, numbers increment live as photos complete.
Phase 2 — log buffer + feed panel. Test: photo results appear in the feed, errors show in red, old entries scroll off.
Phase 4 — non-TTY fallback. Test: python photo_analyzer.py ... | cat produces clean plain output.
Phase 5 — error containment. Test: force a render exception, confirm pipeline continues; interrupt mid-album, confirm DB is consistent on restart.

12 KiB Raw Permalink Blame History Unescape Escape