photoanalyzer/delivery_backlog/stories/US02-03-worker-recovery.md

US02-03 — Execute Jobs with Leases, Locks, and Recovery

Epic: E02

As an operator, I want workers to claim, cancel, and recover jobs safely so crashes or duplicate workers cannot corrupt state.

Acceptance criteria

• Workers use leases, heartbeats, fencing tokens, bounded queues, and typed lanes.
• Cancellation is cooperative and leaves resumable or terminal item states.
• Lock ordering prevents conflicting analysis, EXIF, rename, upload, and archive work.
• A recovered job rejects late commits from its former worker.

Automated tests

• Process tests kill workers, expire leases, recover jobs, and attempt stale commits.
• Randomized concurrency tests assert no deadlocks, duplicate completion, or lost events.

Dependencies

• US02-02