Compare commits

...

14 Commits

Author SHA1 Message Date
7c125dbaeb fix: add domverse network to backend service
Some checks failed
Deploy / deploy (push) Failing after 37s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 12:07:54 +01:00
65b0d48f9d feat: add Traefik + Authentik integration to docker-compose
- Route https://flights.domverse-berlin.eu via Traefik on the domverse network
- Protect with Authentik (authentik@docker ForwardAuth middleware)
- Remove host port bindings (80, 8000) — Traefik handles all ingress
- Frontend joins both default compose network (nginx→backend) and domverse (Traefik)
- Backend stays internal-only, no external exposure

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 12:07:31 +01:00
cdb8c20e82 ci: switch from GitLab CI to Gitea Actions, fix Dockerfile.backend
- Replace .gitlab-ci.yml with .gitea/workflows/deploy.yml
- Fix Dockerfile.backend: add scan_processor.py and searcher_v3.py to
  COPY command (they were missing, would cause runtime ImportError)
- Update docker-compose.yml comment to reference Gitea workflow

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 11:42:02 +01:00
717b976293 ci: add GitLab CI/CD pipeline for Docker deploy
On every push to main: builds both Docker images on the server via
docker compose up --build -d, prunes dangling images, and prints the
running container list. No registry required — shell executor runner
on the deployment server is all that's needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 11:32:18 +01:00
836c8474eb feat: add scheduled scans (cron-like recurring scans)
- New `scheduled_scans` table with daily/weekly/monthly frequencies
- asyncio background scheduler loop checks for due schedules every 60s
- 6 REST endpoints: CRUD + toggle enabled + run-now
- `scheduled_scan_id` FK added to scans table; migrated automatically
- Frontend: Schedules page (list + create form), Schedules nav link,
  "Scheduled" badge on ScanDetails when scan was triggered by a schedule

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 10:48:43 +01:00
ef5a27097d fix: enrich route destination names from airport DB when not stored
Specific-airports mode scans never resolved full airport names — they
stored the IATA code as destination_name. Fixed in two places:

- airports.py: add lookup_airport(iata) cached helper
- api_server.py: enrich destination_name/city on the fly in the routes
  endpoint when the stored value equals the IATA code (fixes all past scans)
- scan_processor.py: resolve airport names at scan time in specific-airports
  mode using lookup_airport (fixes future scans at the DB level)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 21:04:46 +01:00
0a2fed7465 feat: add info icon tooltip for airport names in routes table
Replaces the non-functional title attribute with a small Info icon
next to the IATA code badge. Hovering shows a dark tooltip with the
full airport name and city. Only rendered when useful name data exists.
Clicking the icon stops propagation so it doesn't expand the flights row.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 20:59:37 +01:00
ce1cf667d2 feat: write routes live during scan instead of bulk-insert at completion
Routes and individual flights are now written to the database as each
query result arrives, rather than after all queries finish. The frontend
already polls /scans/:id/routes while status=running, so routes appear
progressively with no frontend changes needed.

Changes:
- database/schema.sql: UNIQUE INDEX uq_routes_scan_dest(scan_id, destination)
- database/init_db.py: _migrate_add_routes_unique_index() migration
- scan_processor.py: _write_route_incremental() helper; progress_callback
  now writes routes/flights immediately; Phase 2 bulk-write replaced with
  a lightweight totals query
- searcher_v3.py: pass flights= kwarg to progress_callback on cache_hit
  and api_success paths

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 20:53:04 +01:00
4926e89e46 feat: re-run and delete scan from detail page
Backend:
- DELETE /api/v1/scans/{id} — 204 on success, 404 if missing,
  409 if pending/running; CASCADE removes routes and flights

Frontend (api.ts):
- scanApi.delete(id)

Frontend (ScanDetails.tsx):
- Re-run button: derives window_months from stored dates, detects
  country vs airports mode via comma in scan.country, creates new
  scan and navigates to it; disabled while scan is active
- Delete button: inline two-step confirm (no modal), navigates to
  dashboard on success; disabled while scan is active

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 16:33:45 +01:00
f9411edd3c remove: docker-compose.dev.yml — develop locally, deploy with docker compose
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 16:24:12 +01:00
06e6ae700f feat: add docker-compose.dev.yml for local development
- docker-compose.dev.yml: backend on 8000, frontend (Vite) on 5173
- Backend mounts source files + uvicorn --reload for hot reload
- Frontend uses node:20-alpine, mounts ./frontend, runs npm run dev --host
- vite.config.ts: proxy target reads from API_TARGET env var
  (defaults to localhost:8000 for plain npm run dev,
   set to http://backend:8000 by docker-compose.dev.yml)

Usage: docker compose -f docker-compose.dev.yml up

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 16:08:53 +01:00
6d168652d4 refactor: split back into two containers (backend + frontend)
Single-container supervisord approach added unnecessary complexity.
Two containers is simpler and more standard:

- Dockerfile.backend: python:3.11-slim, uvicorn on port 8000
- Dockerfile.frontend: node build → nginx:alpine on port 80
- nginx.conf: proxy_pass restored to http://backend:8000
- docker-compose.yml: two services with depends_on
- Removed combined Dockerfile and supervisord.conf

Each container does one thing; logs are separate; either can be
restarted independently.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 16:06:26 +01:00
8bd47ac43a fix: improve airport search ranking and add missing modern airports
- Rewrite airport search to use priority buckets instead of simple
  append: exact IATA → IATA prefix → city prefix → city contains →
  name prefix → name contains → country match. This ensures BER
  appears before Berlin-Schönefeld when typing "BER".
- Add _MISSING_AIRPORTS patch list to get_airport_data() so airports
  absent from the OpenFlights dataset (e.g. BER opened Nov 2020,
  IST new Istanbul airport) are included at runtime.
- Deduplicate results via seen-set to avoid duplicate entries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 15:48:43 +01:00
260f3aa196 fix: add web app deps and git to Docker build
- Added fastapi, uvicorn, pydantic, requests to requirements.txt
  (were missing — only CLI deps were present)
- Changed fast-flights entry to git+GitHub URL (v3 not on PyPI)
- Added git to apt-get install in Dockerfile (needed for pip git+ URL)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 15:42:11 +01:00
22 changed files with 2659 additions and 226 deletions

View File

@@ -0,0 +1,53 @@
# ──────────────────────────────────────────────────────────────────────────────
# Flight Radar — Gitea Actions CI/CD
#
# PREREQUISITES (one-time setup — see README for full instructions):
#
# 1. Add the act_runner service to your Gitea Portainer stack.
#
# 2. Pre-create the runner config file on the host:
# /srv/docker/traefik/stacks/gitea/volumes/act_runner/config.yaml
# (see content in the README / deployment docs)
#
# 3. Start the runner, then grab the registration token from:
# Gitea → Site Administration → Runners → Create Runner
# Add ACT_RUNNER_TOKEN to Portainer stack environment variables.
#
# 4. Give the runner access to Docker (socket mounted via config.yaml).
#
# PIPELINE BEHAVIOUR:
# • Triggers on every push to the default branch (main).
# • Builds both Docker images on the server (no registry needed).
# • Brings the app up with docker compose; only changed services restart.
# • If the build fails the old containers keep running — no downtime.
# • Prunes dangling images after a successful deploy.
# ──────────────────────────────────────────────────────────────────────────────
name: Deploy
on:
push:
branches:
- main
env:
COMPOSE_PROJECT: flight-radar
COMPOSE_FILE: flight-comparator/docker-compose.yml
jobs:
deploy:
runs-on: ubuntu-latest # resolved to catthehacker/ubuntu:act-22.04 by runner config
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy with docker compose
run: |
echo "=== Deploying commit ${{ gitea.sha }} to ${{ gitea.ref_name }} ==="
docker compose -f "$COMPOSE_FILE" -p "$COMPOSE_PROJECT" up --build -d --remove-orphans
- name: Prune dangling images
run: docker image prune -f
- name: Show running containers
run: docker compose -f "$COMPOSE_FILE" -p "$COMPOSE_PROJECT" ps

View File

@@ -1,54 +0,0 @@
# ── Stage 1: Build React frontend ─────────────────────────────────────────
FROM node:20-alpine AS frontend-builder
WORKDIR /app
COPY frontend/package*.json ./
RUN npm ci
COPY frontend/ .
RUN npm run build
# ── Stage 2: Single runtime image ─────────────────────────────────────────
FROM python:3.11-slim
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=1
# Install nginx, supervisor, and gcc (for some pip packages)
RUN apt-get update && apt-get install -y --no-install-recommends \
nginx \
supervisor \
gcc \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Backend source
COPY api_server.py airports.py cache.py ./
COPY database/ ./database/
# Frontend build output
COPY --from=frontend-builder /app/dist /usr/share/nginx/html
# Config files
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY supervisord.conf /etc/supervisor/conf.d/app.conf
# Remove the default nginx site
RUN rm -f /etc/nginx/sites-enabled/default
# Pre-fetch airport data and initialise the database at build time
RUN mkdir -p data && \
python -c "from airports import download_and_build_airport_data; download_and_build_airport_data()" && \
python database/init_db.py
EXPOSE 80
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD wget -q --spider http://localhost/ || exit 1
CMD ["/usr/bin/supervisord", "-n", "-c", "/etc/supervisor/conf.d/app.conf"]

View File

@@ -0,0 +1,30 @@
FROM python:3.11-slim
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=1
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY api_server.py airports.py cache.py scan_processor.py searcher_v3.py ./
COPY database/ ./database/
# Pre-fetch airport data and initialise the database at build time
RUN mkdir -p data && \
python -c "from airports import download_and_build_airport_data; download_and_build_airport_data()" && \
python database/init_db.py
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
CMD ["python", "api_server.py"]

View File

@@ -0,0 +1,19 @@
# ── Stage 1: Build React frontend ─────────────────────────────────────────
FROM node:20-alpine AS builder
WORKDIR /app
COPY frontend/package*.json ./
RUN npm ci
COPY frontend/ .
RUN npm run build
# ── Stage 2: Serve with nginx ──────────────────────────────────────────────
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -q --spider http://localhost/ || exit 1

View File

@@ -0,0 +1,427 @@
# PRD: Live Routes During Active Scan
**Status:** Draft
**Date:** 2026-02-27
**Scope:** Backend only — no frontend or API contract changes required
---
## 1. Problem
Routes and flights are only visible in the UI **after a scan fully completes**. For large scans (e.g., 30 destinations × 14 days = 420 queries), users stare at a progress bar with zero actionable data for potentially many minutes.
The root cause is a two-phase waterfall in `scan_processor.py`:
- **Phase 1 (line 170):** `await search_multiple_routes(...)``asyncio.gather()` waits for **all** queries to finish before returning anything.
- **Phase 2 (line 197):** Bulk `INSERT INTO routes` and `INSERT INTO flights` only after Phase 1 completes.
---
## 2. Goal
Write each destination's route row and its individual flight rows to the database **as soon as that destination's results arrive**, rather than after all queries finish.
The frontend already polls `/scans/:id/routes` on an interval while status is `running`. No frontend changes are needed — routes will simply appear progressively.
---
## 3. Desired Behavior
| Moment | Before fix | After fix |
|--------|-----------|-----------|
| Query 1 completes (BER → MUC) | Nothing visible | MUC route + flights appear in UI |
| 50% through scan | 0 routes in UI | ~50% of routes visible |
| Scan completes | All routes appear at once | No change (already visible) |
---
## 4. Architecture Analysis
### 4.1 Event Loop Threading
`progress_callback` is called from within `search_direct_flights` (`searcher_v3.py`):
- **Cache hit** (line 121): called directly in the async coroutine → **event loop thread**
- **API success** (line 143): called after `await asyncio.to_thread()` returns → **event loop thread**
- **Error** (line 159): same → **event loop thread**
All callback invocations happen on the **single asyncio event loop thread**. No locking is needed. The existing `progress_callback` in `scan_processor.py` already opens and closes a fresh `get_connection()` per call — we reuse the same pattern.
### 4.2 Why Multiple Callbacks Fire Per Destination
For a scan covering 14 dates, a single destination (e.g., MUC) gets 14 separate queries:
- `(BER, MUC, 2026-03-01)`, `(BER, MUC, 2026-03-02)`, ... `(BER, MUC, 2026-03-14)`
Each query completes independently and fires `progress_callback`. The incremental write logic must **merge** successive results for the same destination into a single route row.
### 4.3 Required Schema Change
`routes` has no UNIQUE constraint on `(scan_id, destination)`. We need one to support upsert semantics (detect "route already started" vs "first result for this destination").
Current schema (no uniqueness):
```sql
CREATE INDEX IF NOT EXISTS idx_routes_scan_id ON routes(scan_id);
```
Required addition:
```sql
CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_dest ON routes(scan_id, destination);
```
---
## 5. Changes Required
### 5.1 `database/schema.sql`
Add a unique index at the end of the routes indexes block:
```sql
-- Unique constraint: one route row per (scan, destination)
CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_dest
ON routes(scan_id, destination);
```
### 5.2 `database/init_db.py`
Add a migration function to create the index on existing databases where it doesn't yet exist. Called before `executescript(schema_sql)`.
```python
def _migrate_add_routes_unique_index(conn, verbose=True):
"""
Migration: Add UNIQUE index on routes(scan_id, destination).
Required for incremental route writes during active scans.
The index enables upsert (INSERT + UPDATE on conflict) semantics.
Safe to run on existing data: if any (scan_id, destination) duplicates
exist, we collapse them first (keep the row with more flights).
"""
cursor = conn.execute("""
SELECT name FROM sqlite_master
WHERE type='index' AND name='uq_routes_scan_dest'
""")
if cursor.fetchone():
return # Already migrated
if verbose:
print(" 🔄 Migrating routes table: adding unique index on (scan_id, destination)...")
# Collapse any existing duplicates (completed scans may have none,
# but guard against edge cases).
conn.execute("""
DELETE FROM routes
WHERE id NOT IN (
SELECT MIN(id)
FROM routes
GROUP BY scan_id, destination
)
""")
# Create the unique index
conn.execute("""
CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_dest
ON routes(scan_id, destination)
""")
conn.commit()
if verbose:
print(" ✅ Migration complete: uq_routes_scan_dest index created")
```
Wire it into `initialize_database()` before `executescript`:
```python
# Apply migrations before running schema
_migrate_relax_country_constraint(conn, verbose)
_migrate_add_routes_unique_index(conn, verbose) # ← ADD THIS
# Load and execute schema
schema_sql = load_schema()
```
### 5.3 `scan_processor.py` — Extend `progress_callback`
#### Current signature (line 138):
```python
def progress_callback(origin: str, destination: str, date: str,
status: str, count: int, error: str = None):
```
#### New signature:
```python
def progress_callback(origin: str, destination: str, date: str,
status: str, count: int, error: str = None,
flights: list = None):
```
#### New helper function (add before `process_scan`):
```python
def _write_route_incremental(scan_id: int, destination: str,
dest_name: str, dest_city: str,
new_flights: list):
"""
Write or update a route row and its flight rows incrementally.
Called from progress_callback each time a query for (scan_id, destination)
returns results. Merges into the existing route row if one already exists.
Uses a read-then-write pattern so airlines JSON arrays can be merged in
Python rather than in SQLite (SQLite has no native JSON array merge).
Safe to call from the event loop thread — opens its own connection.
"""
if not new_flights:
return
prices = [f.get('price') for f in new_flights if f.get('price')]
if not prices:
return # No priced flights — nothing to aggregate
new_airlines = list({f.get('airline') for f in new_flights if f.get('airline')})
new_count = len(new_flights)
new_min = min(prices)
new_max = max(prices)
new_avg = sum(prices) / len(prices)
try:
conn = get_connection()
cursor = conn.cursor()
# Check if a route row already exists for this destination
cursor.execute("""
SELECT id, flight_count, min_price, max_price, avg_price, airlines
FROM routes
WHERE scan_id = ? AND destination = ?
""", (scan_id, destination))
existing = cursor.fetchone()
if existing is None:
# First result for this destination — INSERT
cursor.execute("""
INSERT INTO routes (
scan_id, destination, destination_name, destination_city,
flight_count, airlines, min_price, max_price, avg_price
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
scan_id, destination, dest_name, dest_city,
new_count, json.dumps(new_airlines),
new_min, new_max, new_avg,
))
else:
# Subsequent result — merge stats
old_count = existing['flight_count'] or 0
old_min = existing['min_price']
old_max = existing['max_price']
old_avg = existing['avg_price'] or 0.0
old_airlines = json.loads(existing['airlines']) if existing['airlines'] else []
merged_count = old_count + new_count
merged_min = min(old_min, new_min) if old_min is not None else new_min
merged_max = max(old_max, new_max) if old_max is not None else new_max
# Running weighted average
merged_avg = (old_avg * old_count + new_avg * new_count) / merged_count
merged_airlines = json.dumps(list(set(old_airlines) | set(new_airlines)))
cursor.execute("""
UPDATE routes
SET flight_count = ?,
min_price = ?,
max_price = ?,
avg_price = ?,
airlines = ?
WHERE scan_id = ? AND destination = ?
""", (
merged_count, merged_min, merged_max, merged_avg, merged_airlines,
scan_id, destination,
))
# INSERT individual flight rows (always new rows; duplicates not expected
# because each (scan_id, destination, date) query fires exactly once)
for flight in new_flights:
if not flight.get('price'):
continue
cursor.execute("""
INSERT INTO flights (
scan_id, destination, date, airline,
departure_time, arrival_time, price, stops
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""", (
scan_id,
destination,
flight.get('date', ''), # date passed from caller
flight.get('airline'),
flight.get('departure_time'),
flight.get('arrival_time'),
flight.get('price'),
flight.get('stops', 0),
))
conn.commit()
conn.close()
except Exception as e:
logger.error(f"[Scan {scan_id}] Failed to write incremental route {destination}: {e}")
```
**Note on flight date:** Individual flights must carry their `date` when passed to `_write_route_incremental`. The callback receives `date` as a parameter, so attach it to each flight dict before passing: `f['date'] = date`.
#### Updated `progress_callback` inside `process_scan`:
```python
def progress_callback(origin: str, destination: str, date: str,
status: str, count: int, error: str = None,
flights: list = None):
nonlocal routes_scanned_count
if status in ('cache_hit', 'api_success', 'error'):
routes_scanned_count += 1
# Write route + flights immediately if we have results
if flights and status in ('cache_hit', 'api_success'):
# Annotate each flight with its query date
for f in flights:
f['date'] = date
dest_info = next((d for d in destinations if d['iata'] == destination), None)
dest_name = dest_info.get('name', destination) if dest_info else destination
dest_city = dest_info.get('city', '') if dest_info else ''
_write_route_incremental(scan_id, destination, dest_name, dest_city, flights)
# Update progress counter
try:
progress_conn = get_connection()
progress_cursor = progress_conn.cursor()
progress_cursor.execute("""
UPDATE scans
SET routes_scanned = routes_scanned + 1,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (scan_id,))
progress_conn.commit()
progress_conn.close()
if routes_scanned_count % 10 == 0:
logger.info(f"[Scan {scan_id}] Progress: {routes_scanned_count}/{len(routes_to_scan)} routes ({status}: {origin}{destination})")
except Exception as e:
logger.error(f"[Scan {scan_id}] Failed to update progress: {str(e)}")
```
#### Replace Phase 2 (lines 182262 in `process_scan`):
Remove the existing bulk-write block. Replace with a lightweight totals-only block:
```python
# Wait for all queries to complete
results = await search_multiple_routes(
routes=routes_to_scan,
seat_class=seat_class or 'economy',
adults=adults or 1,
use_cache=True,
cache_threshold_hours=24,
max_workers=3,
progress_callback=progress_callback
)
logger.info(f"[Scan {scan_id}] All queries complete. Finalizing scan...")
# Count total flights (routes already written by callback)
total_flights = sum(len(flights) for flights in results.values())
routes_saved = cursor.execute(
"SELECT COUNT(*) FROM routes WHERE scan_id = ?", (scan_id,)
).fetchone()[0]
logger.info(f"[Scan {scan_id}] ✅ {routes_saved} routes, {total_flights} flights")
```
### 5.4 `searcher_v3.py` — Pass `flights` to callback
In `search_direct_flights`, update both callback calls to pass `flights=`:
**Cache hit (line 121):**
```python
if progress_callback:
progress_callback(origin, destination, date, "cache_hit", len(cached), flights=cached)
return cached
```
**API success (line 143):**
```python
if progress_callback:
progress_callback(origin, destination, date, "api_success", len(result), flights=result)
```
The `error` callback (line 159) does not need `flights=` since there are no results.
---
## 6. Concurrency & Safety
| Concern | Analysis | Verdict |
|---------|----------|---------|
| Simultaneous writes to same route row | Cannot happen: the event loop is single-threaded; callbacks are not concurrent. `asyncio.gather` runs tasks concurrently but yields at `await` points, and callbacks fire synchronously after each `await to_thread` returns. | ✅ Safe |
| Two callbacks for same destination arriving "simultaneously" | Impossible in single-threaded event loop. Second callback blocks until first completes (no `await` in callback). | ✅ Safe |
| Duplicate flight rows | Each `(scan_id, destination, date)` query fires exactly once; its flights are written exactly once. | ✅ No duplicates |
| `total_flights` trigger still fires correctly | SQLite triggers on `INSERT INTO routes` and `UPDATE OF flight_count ON routes` fire for each incremental write — counts stay accurate. | ✅ Works |
| Scan completion `total_flights` update | Still set explicitly at completion from `results` dict count — redundant but harmless. | ✅ OK |
---
## 7. Edge Cases
| Case | Handling |
|------|----------|
| Destination returns 0 flights | `_write_route_incremental` returns early — no row created. Route only appears if at least one priced flight found. |
| Scan is deleted mid-run | `DELETE CASCADE` on `scans` removes routes/flights automatically. Progress callback write will fail with FK error, caught by `except` block and logged. |
| Scan fails mid-run | Routes written so far remain in DB. Status set to `failed`. UI will show partial results with `failed` badge — acceptable. |
| DB write error in callback | Logged, does not crash the scan. Query continues, flight data lost for that callback. |
| Existing scans (pre-feature) | No impact. Migration adds index but doesn't change old data (all complete scans already have 1 row per destination). |
---
## 8. Migration Plan
1. **`database/schema.sql`**: Add `CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_dest`.
2. **`database/init_db.py`**: Add `_migrate_add_routes_unique_index()` + call it in `initialize_database()`.
3. **`scan_processor.py`**: Add `_write_route_incremental()` helper; update `progress_callback` closure; remove bulk-write Phase 2.
4. **`searcher_v3.py`**: Pass `flights=` kwarg to both successful callback invocations.
**Migration is backward-safe:** The UNIQUE index is added with `IF NOT EXISTS`. Existing `completed` scans already have at most 1 route row per destination — the index creation will succeed without errors.
**No API changes:** `/scans/:id/routes` endpoint already returns live data from the `routes` table. The frontend polling already works.
---
## 9. Rollback
To revert: remove the `flights=` kwarg from `searcher_v3.py` callbacks, restore the bulk-write Phase 2 in `scan_processor.py`, and remove `_write_route_incremental`. The UNIQUE index can remain — it only adds a constraint that is naturally satisfied by the bulk-write approach anyway.
---
## 10. Testing Plan
### Unit Tests (new)
1. `test_write_route_incremental_new` — first call creates route row
2. `test_write_route_incremental_merge` — second call updates stats correctly
3. `test_write_route_incremental_no_prices` — empty-price flights produce no row
4. `test_write_route_incremental_airlines_merge` — duplicate airlines deduplicated
5. `test_weighted_average_formula` — verify avg formula with known numbers
### Integration Tests (extend existing)
6. Extend `test_scan_lifecycle` — poll routes every 0.1s during mock scan, verify routes appear before completion
7. `test_incremental_writes_idempotent` — simulate same callback called twice for same destination
8. `test_unique_index_exists` — verify migration creates index
9. `test_migration_collapses_duplicates` — seed duplicate route rows, run migration, verify collapsed
---
## 11. Out of Scope
- WebSocket or SSE for push-based updates (polling already works)
- Frontend changes (none needed)
- Real-time price charts
- Partial scan resume after crash
- `total_flights` trigger removal (keep for consistency)

View File

@@ -0,0 +1,409 @@
# PRD: Scheduled Scans
**Status:** Draft
**Date:** 2026-02-27
**Verdict:** Fully feasible — no new dependencies required
---
## 1. Problem
Every scan is triggered manually. If you want to track prices for a route over time (e.g. BDS → Germany every Monday) you have to remember to click "Re-run" yourself. Price trends are only discoverable by comparing scan history manually.
---
## 2. Goal
Let users define a recurring schedule for any scan configuration. The server runs the scan automatically at the defined cadence, building a historical record of price data over time.
---
## 3. User Stories
- **As a user**, I want to schedule a weekly scan of BDS → Germany so I can see how prices change without manually re-running it.
- **As a user**, I want to enable/disable a schedule without deleting it.
- **As a user**, I want to see which scans were created by a schedule and navigate to that schedule from a scan.
- **As a user**, I want to trigger a scheduled scan immediately without waiting for the next interval.
---
## 4. Scheduling Options
Three frequencies are sufficient for flight price tracking:
| Frequency | Parameters | Example |
|-----------|-----------|---------|
| `daily` | hour, minute | Every day at 06:00 |
| `weekly` | day_of_week (0=Mon6=Sun), hour, minute | Every Monday at 06:00 |
| `monthly` | day_of_month (128), hour, minute | 1st of every month at 06:00 |
Day of month capped at 28 to avoid Feb 29/30/31 edge cases. All times stored and executed in UTC.
---
## 5. Architecture
### 5.1 Scheduler Design
No new dependencies. A simple asyncio background task wakes every 60 seconds, queries the DB for due schedules, and fires a scan for each.
```
lifespan startup
└── asyncio.create_task(_scheduler_loop())
└── while True:
_check_and_run_due_schedules() # queries DB
await asyncio.sleep(60)
```
`_check_and_run_due_schedules()`:
1. `SELECT * FROM scheduled_scans WHERE enabled=1 AND next_run_at <= NOW()`
2. For each result, skip if previous scan for this schedule is still `pending` or `running`
3. Create a new scan row (same INSERT as `POST /scans`)
4. Call `start_scan_processor(scan_id)`
5. Update `last_run_at = NOW()` and compute + store `next_run_at`
### 5.2 `next_run_at` Computation
Precomputed in Python after every run (and on create/update). Stored as a TIMESTAMP column with an index — scheduler lookup is a single indexed range query.
```python
def compute_next_run(frequency, hour, minute,
day_of_week=None, day_of_month=None,
after=None) -> datetime:
now = after or datetime.utcnow()
base = now.replace(hour=hour, minute=minute, second=0, microsecond=0)
if frequency == 'daily':
return base if base > now else base + timedelta(days=1)
elif frequency == 'weekly':
days_ahead = (day_of_week - now.weekday()) % 7
if days_ahead == 0 and base <= now:
days_ahead = 7
return (now + timedelta(days=days_ahead)).replace(
hour=hour, minute=minute, second=0, microsecond=0)
elif frequency == 'monthly':
candidate = now.replace(day=day_of_month, hour=hour, minute=minute, second=0, microsecond=0)
if candidate <= now:
m, y = (now.month % 12) + 1, now.year + (1 if now.month == 12 else 0)
candidate = candidate.replace(year=y, month=m)
return candidate
```
---
## 6. Schema Changes
### 6.1 New table: `scheduled_scans`
```sql
CREATE TABLE IF NOT EXISTS scheduled_scans (
id INTEGER PRIMARY KEY AUTOINCREMENT,
-- Scan parameters
origin TEXT NOT NULL CHECK(length(origin) = 3),
country TEXT NOT NULL CHECK(length(country) >= 2),
window_months INTEGER NOT NULL DEFAULT 1
CHECK(window_months >= 1 AND window_months <= 12),
seat_class TEXT NOT NULL DEFAULT 'economy',
adults INTEGER NOT NULL DEFAULT 1
CHECK(adults > 0 AND adults <= 9),
-- Schedule definition
frequency TEXT NOT NULL
CHECK(frequency IN ('daily', 'weekly', 'monthly')),
hour INTEGER NOT NULL DEFAULT 6
CHECK(hour >= 0 AND hour <= 23),
minute INTEGER NOT NULL DEFAULT 0
CHECK(minute >= 0 AND minute <= 59),
day_of_week INTEGER CHECK(day_of_week >= 0 AND day_of_week <= 6),
day_of_month INTEGER CHECK(day_of_month >= 1 AND day_of_month <= 28),
-- State
enabled INTEGER NOT NULL DEFAULT 1,
label TEXT,
last_run_at TIMESTAMP,
next_run_at TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- Frequency-specific constraints
CHECK(
(frequency = 'weekly' AND day_of_week IS NOT NULL) OR
(frequency = 'monthly' AND day_of_month IS NOT NULL) OR
(frequency = 'daily')
)
);
-- Fast lookup of due schedules
CREATE UNIQUE INDEX IF NOT EXISTS uq_scheduled_scans_id
ON scheduled_scans(id);
CREATE INDEX IF NOT EXISTS idx_scheduled_scans_next_run
ON scheduled_scans(next_run_at)
WHERE enabled = 1;
-- Auto-update updated_at
CREATE TRIGGER IF NOT EXISTS update_scheduled_scans_timestamp
AFTER UPDATE ON scheduled_scans
FOR EACH ROW BEGIN
UPDATE scheduled_scans SET updated_at = CURRENT_TIMESTAMP WHERE id = NEW.id;
END;
-- Insert schema version bump
INSERT OR IGNORE INTO schema_version (version, description)
VALUES (2, 'Add scheduled_scans table');
```
### 6.2 Add FK column to `scans`
```sql
-- Migration: add scheduled_scan_id to scans
ALTER TABLE scans ADD COLUMN scheduled_scan_id INTEGER
REFERENCES scheduled_scans(id) ON DELETE SET NULL;
CREATE INDEX IF NOT EXISTS idx_scans_scheduled_scan_id
ON scans(scheduled_scan_id)
WHERE scheduled_scan_id IS NOT NULL;
```
---
## 7. Migration (`database/init_db.py`)
Add two migration functions, called before `executescript(schema_sql)`:
```python
def _migrate_add_scheduled_scans(conn, verbose=True):
"""Migration: create scheduled_scans table and add FK to scans."""
cursor = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name='scheduled_scans'"
)
if cursor.fetchone():
return # Already exists
if verbose:
print(" 🔄 Migrating: adding scheduled_scans table...")
conn.execute("""
CREATE TABLE scheduled_scans (
id INTEGER PRIMARY KEY AUTOINCREMENT, ...
)
""")
# Add scheduled_scan_id to existing scans table
try:
conn.execute("ALTER TABLE scans ADD COLUMN scheduled_scan_id INTEGER REFERENCES scheduled_scans(id) ON DELETE SET NULL")
except sqlite3.OperationalError:
pass # Column already exists
conn.execute("CREATE INDEX IF NOT EXISTS idx_scans_scheduled_scan_id ON scans(scheduled_scan_id) WHERE scheduled_scan_id IS NOT NULL")
conn.commit()
if verbose:
print(" ✅ Migration complete: scheduled_scans table created")
```
---
## 8. API Endpoints
All under `/api/v1/schedules`. Rate limit: 30 req/min per IP (same as scans list).
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/schedules` | List all schedules (paginated) |
| `POST` | `/schedules` | Create a schedule |
| `GET` | `/schedules/{id}` | Schedule details + last 5 scan IDs |
| `PATCH` | `/schedules/{id}` | Update (enable/disable, change frequency/params) |
| `DELETE` | `/schedules/{id}` | Delete schedule (scans are kept, FK set to NULL) |
| `POST` | `/schedules/{id}/run-now` | Trigger immediately (ignores next_run_at) |
### Request model: `CreateScheduleRequest`
```python
class CreateScheduleRequest(BaseModel):
origin: str # 3-char IATA
country: Optional[str] # 2-letter ISO country code
destinations: Optional[List[str]] # Alternative: list of IATA codes
window_months: int = 1 # Weeks of data per scan run
seat_class: str = 'economy'
adults: int = 1
label: Optional[str] # Human-readable name
frequency: str # 'daily' | 'weekly' | 'monthly'
hour: int = 6 # UTC hour (023)
minute: int = 0 # UTC minute (059)
day_of_week: Optional[int] # Required when frequency='weekly' (0=Mon)
day_of_month: Optional[int] # Required when frequency='monthly' (128)
```
### Response model: `Schedule`
```python
class Schedule(BaseModel):
id: int
origin: str
country: str
window_months: int
seat_class: str
adults: int
label: Optional[str]
frequency: str
hour: int
minute: int
day_of_week: Optional[int]
day_of_month: Optional[int]
enabled: bool
last_run_at: Optional[str]
next_run_at: str
created_at: str
recent_scan_ids: List[int] # Last 5 scans created by this schedule
```
---
## 9. Scheduler Lifecycle (`api_server.py`)
### 9.1 Startup
In the existing `lifespan()` context manager, after existing startup code:
```python
scheduler_task = asyncio.create_task(_scheduler_loop())
logger.info("Scheduled scan background task started")
yield
scheduler_task.cancel()
try:
await scheduler_task
except asyncio.CancelledError:
pass
```
### 9.2 Missed runs on restart
When the server starts, `_check_and_run_due_schedules()` fires immediately (before the 60-second sleep), catching any schedules that were due while the server was down. Each overdue schedule runs exactly once — `next_run_at` is then advanced to the next future interval. Multiple missed intervals are not caught up.
### 9.3 Concurrency guard
Before firing a scan for a schedule, check:
```python
running = conn.execute("""
SELECT id FROM scans
WHERE scheduled_scan_id = ? AND status IN ('pending', 'running')
""", (schedule_id,)).fetchone()
if running:
logger.info(f"Schedule {schedule_id}: previous scan {running[0]} still active, skipping this run")
# Still advance next_run_at so we try again next interval
continue
```
---
## 10. Frontend Changes
### 10.1 New page: `Schedules.tsx`
**List view:**
- Table of all schedules: label, origin → country, frequency, next run (local time), last run, enabled toggle
- "New Schedule" button opens create form (same airport search component as Scans)
- Inline enable/disable toggle (PATCH request, optimistic update)
- "Run now" button per row
**Create form fields (below existing scan form fields):**
- Frequency selector: Daily / Weekly / Monthly (segmented button)
- Time of day: hour:minute picker (UTC, with note)
- Day of week (shown only for Weekly): MonSun selector
- Day of month (shown only for Monthly): 128 number input
- Optional label field
### 10.2 Modified: `ScanDetails.tsx`
When a scan has `scheduled_scan_id`, show a small "Scheduled" chip in the header with a link to `/schedules/{scheduled_scan_id}`.
### 10.3 Navigation (`Layout.tsx`)
Add "Schedules" link to sidebar between Scans and Airports.
### 10.4 API client (`api.ts`)
```typescript
export interface Schedule {
id: number;
origin: string;
country: string;
window_months: number;
seat_class: string;
adults: number;
label?: string;
frequency: 'daily' | 'weekly' | 'monthly';
hour: number;
minute: number;
day_of_week?: number;
day_of_month?: number;
enabled: boolean;
last_run_at?: string;
next_run_at: string;
created_at: string;
recent_scan_ids: number[];
}
export const scheduleApi = {
list: (page = 1, limit = 20) =>
api.get<PaginatedResponse<Schedule>>('/schedules', { params: { page, limit } }),
get: (id: number) =>
api.get<Schedule>(`/schedules/${id}`),
create: (data: CreateScheduleRequest) =>
api.post<Schedule>('/schedules', data),
update: (id: number, data: Partial<CreateScheduleRequest> & { enabled?: boolean }) =>
api.patch<Schedule>(`/schedules/${id}`, data),
delete: (id: number) =>
api.delete(`/schedules/${id}`),
runNow: (id: number) =>
api.post<{ scan_id: number }>(`/schedules/${id}/run-now`),
};
```
---
## 11. Edge Cases
| Case | Handling |
|------|----------|
| Previous scan still running at next interval | Skip this interval's run, advance `next_run_at`, log warning |
| Server down when schedule is due | On startup, runs any overdue schedule once; does not catch up multiple missed intervals |
| Schedule deleted while scan is running | `ON DELETE SET NULL` on FK — scan continues, `scheduled_scan_id` becomes NULL |
| `window_months` covers past dates | Scan start date is always "tomorrow" at creation time, same as manual scans |
| Monthly with day_of_month=29..31 | Capped at 28 in validation — avoids invalid dates in all months |
| Simultaneous due schedules | Each creates an independent asyncio task; existing `max_workers=3` semaphore in scan_processor limits total API concurrency across all running scans |
| Schedule created at 05:59, fires at 06:00 UTC | `next_run_at` is computed at creation time — if 06:00 today already passed, fires tomorrow |
---
## 12. Files Changed
| File | Change |
|------|--------|
| `database/schema.sql` | Add `scheduled_scans` table, trigger, indexes, schema_version bump |
| `database/init_db.py` | `_migrate_add_scheduled_scans()` + call in `initialize_database()` |
| `api_server.py` | `compute_next_run()`, `_scheduler_loop()`, `_check_and_run_due_schedules()`, 6 new endpoints, lifespan update, new Pydantic models |
| `frontend/src/api.ts` | `Schedule` type, `CreateScheduleRequest` type, `scheduleApi` object |
| `frontend/src/pages/Schedules.tsx` | New page (list + inline create form) |
| `frontend/src/pages/ScanDetails.tsx` | "Scheduled" badge + link when `scheduled_scan_id` present |
| `frontend/src/components/Layout.tsx` | Schedules nav link |
Total: 7 files. Estimated ~500 new lines (backend ~250, frontend ~250).
---
## 13. Out of Scope
- Notifications / alerts when a scheduled scan completes (email, webhook)
- Per-schedule price change detection / diffing between runs
- Timezone-aware scheduling (all times UTC for now)
- Pause/resume of scheduled scans (separate PRD)
- Rate limiting across simultaneous scheduled scans (existing semaphore provides soft protection)
- Dashboard widgets for upcoming scheduled runs

View File

@@ -6,6 +6,7 @@ Handles loading and filtering airport data from OpenFlights dataset.
import json import json
import csv import csv
from functools import lru_cache
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
import urllib.request import urllib.request
@@ -225,6 +226,25 @@ def resolve_airport_list(country: Optional[str], from_airports: Optional[str]) -
raise ValueError("Either --country or --from must be provided") raise ValueError("Either --country or --from must be provided")
@lru_cache(maxsize=1)
def _all_airports_by_iata() -> dict:
"""Return {iata: airport_dict} for every airport. Cached after first load."""
if not AIRPORTS_JSON_PATH.exists():
download_and_build_airport_data()
with open(AIRPORTS_JSON_PATH, 'r', encoding='utf-8') as f:
airports_by_country = json.load(f)
return {
a['iata']: a
for airports in airports_by_country.values()
for a in airports
}
def lookup_airport(iata: str) -> dict | None:
"""Look up a single airport by IATA code. Returns None if not found."""
return _all_airports_by_iata().get(iata.upper())
if __name__ == "__main__": if __name__ == "__main__":
# Build the dataset if run directly # Build the dataset if run directly
download_and_build_airport_data(force_rebuild=True) download_and_build_airport_data(force_rebuild=True)

View File

@@ -22,6 +22,7 @@ from pydantic import BaseModel, Field, validator, ValidationError
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from functools import lru_cache from functools import lru_cache
from datetime import datetime, date, timedelta from datetime import datetime, date, timedelta
import asyncio
import json import json
import os import os
import re import re
@@ -224,6 +225,7 @@ RATE_LIMITS = {
'scans': (50, 60), # 50 scan creations per minute 'scans': (50, 60), # 50 scan creations per minute
'logs': (100, 60), # 100 log requests per minute 'logs': (100, 60), # 100 log requests per minute
'airports': (500, 60), # 500 airport searches per minute 'airports': (500, 60), # 500 airport searches per minute
'schedules': (30, 60), # 30 schedule requests per minute
} }
@@ -240,10 +242,127 @@ def get_rate_limit_for_path(path: str) -> tuple[str, int, int]:
return 'logs', *RATE_LIMITS['logs'] return 'logs', *RATE_LIMITS['logs']
elif '/airports' in path: elif '/airports' in path:
return 'airports', *RATE_LIMITS['airports'] return 'airports', *RATE_LIMITS['airports']
elif '/schedules' in path:
return 'schedules', *RATE_LIMITS['schedules']
else: else:
return 'default', *RATE_LIMITS['default'] return 'default', *RATE_LIMITS['default']
# =============================================================================
# Scheduler
# =============================================================================
def compute_next_run(frequency: str, hour: int, minute: int,
day_of_week: int = None, day_of_month: int = None,
after: datetime = None) -> datetime:
"""Compute the next UTC run time for a scheduled scan."""
now = after or datetime.utcnow()
base = now.replace(hour=hour, minute=minute, second=0, microsecond=0)
if frequency == 'daily':
return base if base > now else base + timedelta(days=1)
elif frequency == 'weekly':
days_ahead = (day_of_week - now.weekday()) % 7
if days_ahead == 0 and base <= now:
days_ahead = 7
return (now + timedelta(days=days_ahead)).replace(
hour=hour, minute=minute, second=0, microsecond=0)
elif frequency == 'monthly':
candidate = now.replace(day=day_of_month, hour=hour, minute=minute,
second=0, microsecond=0)
if candidate <= now:
m, y = (now.month % 12) + 1, now.year + (1 if now.month == 12 else 0)
candidate = candidate.replace(year=y, month=m)
return candidate
raise ValueError(f"Unknown frequency: {frequency}")
def _check_and_run_due_schedules():
"""Query DB for due schedules and fire a scan for each."""
try:
conn = get_connection()
cursor = conn.cursor()
now_str = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')
cursor.execute("""
SELECT id, origin, country, window_months, seat_class, adults,
frequency, hour, minute, day_of_week, day_of_month
FROM scheduled_scans
WHERE enabled = 1 AND next_run_at <= ?
""", (now_str,))
due = cursor.fetchall()
for row in due:
(sched_id, origin, country, window_months, seat_class, adults,
frequency, hour, minute, day_of_week, day_of_month) = row
# Concurrency guard: skip if a scan for this schedule is still active
running = conn.execute("""
SELECT id FROM scans
WHERE scheduled_scan_id = ? AND status IN ('pending', 'running')
""", (sched_id,)).fetchone()
if running:
logging.info(
f"Schedule {sched_id}: previous scan {running[0]} still active, skipping"
)
else:
# Compute date window
start_date = (date.today() + timedelta(days=1)).isoformat()
end_dt = date.today() + timedelta(days=1) + timedelta(days=30 * window_months)
end_date = end_dt.isoformat()
conn.execute("""
INSERT INTO scans (
origin, country, start_date, end_date,
status, seat_class, adults, scheduled_scan_id
) VALUES (?, ?, ?, ?, 'pending', ?, ?, ?)
""", (origin, country, start_date, end_date,
seat_class, adults, sched_id))
conn.commit()
scan_id = conn.execute("SELECT last_insert_rowid()").fetchone()[0]
try:
start_scan_processor(scan_id)
logging.info(
f"Schedule {sched_id}: fired scan {scan_id} "
f"({origin}{country})"
)
except Exception as e:
logging.error(
f"Schedule {sched_id}: failed to start scan {scan_id}: {e}"
)
# Advance next_run_at regardless of whether we fired
next_run = compute_next_run(
frequency, hour, minute, day_of_week, day_of_month
)
conn.execute("""
UPDATE scheduled_scans
SET last_run_at = ?, next_run_at = ?
WHERE id = ?
""", (now_str, next_run.strftime('%Y-%m-%d %H:%M:%S'), sched_id))
conn.commit()
conn.close()
except Exception as e:
logging.error(f"Scheduler error: {e}", exc_info=True)
async def _scheduler_loop():
"""Background task: check for due schedules every 60 seconds."""
logging.info("Scheduler loop started")
# Run immediately on startup to catch any missed schedules
_check_and_run_due_schedules()
while True:
await asyncio.sleep(60)
_check_and_run_due_schedules()
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): async def lifespan(app: FastAPI):
"""Initialize airport data and database on server start.""" """Initialize airport data and database on server start."""
@@ -308,7 +427,18 @@ async def lifespan(app: FastAPI):
print(f"⚠️ Scan cleanup warning: {e}") print(f"⚠️ Scan cleanup warning: {e}")
logging.info("Flight Radar API v2.0 startup complete") logging.info("Flight Radar API v2.0 startup complete")
# Start scheduled scan background task
scheduler_task = asyncio.create_task(_scheduler_loop())
logging.info("Scheduled scan background task started")
yield yield
scheduler_task.cancel()
try:
await scheduler_task
except asyncio.CancelledError:
pass
logging.info("Flight Radar API v2.0 shutting down") logging.info("Flight Radar API v2.0 shutting down")
@@ -799,6 +929,7 @@ class Scan(BaseModel):
error_message: Optional[str] = Field(None, description="Error message if scan failed") error_message: Optional[str] = Field(None, description="Error message if scan failed")
seat_class: str = Field(..., description="Seat class") seat_class: str = Field(..., description="Seat class")
adults: int = Field(..., ge=1, le=9, description="Number of adults") adults: int = Field(..., ge=1, le=9, description="Number of adults")
scheduled_scan_id: Optional[int] = Field(None, description="ID of the schedule that created this scan")
class ScanCreateResponse(BaseModel): class ScanCreateResponse(BaseModel):
@@ -880,40 +1011,56 @@ async def search_airports(
raise HTTPException(status_code=500, detail=f"Failed to load airport data: {e}") raise HTTPException(status_code=500, detail=f"Failed to load airport data: {e}")
query = q.lower().strip() query = q.lower().strip()
results = []
# Search all airports # Priority buckets — higher bucket = shown first
p0_exact_iata: list[Airport] = [] # IATA == query exactly (e.g. "BER")
p1_iata_prefix: list[Airport] = [] # IATA starts with query (e.g. "BE" → BER)
p2_city_prefix: list[Airport] = [] # city starts with query (e.g. "ber" → Berlin)
p3_city_contains: list[Airport] = [] # city contains query
p4_name_prefix: list[Airport] = [] # name starts with query
p5_name_contains: list[Airport] = [] # name contains query
p6_country: list[Airport] = [] # country code contains query
seen: set[str] = set()
for airport in airports_data: for airport in airports_data:
# Skip invalid airport data (data quality issues in OpenFlights dataset)
try: try:
# Search in IATA code (exact match prioritized) iata_l = airport['iata'].lower()
if airport['iata'].lower() == query: city_l = airport.get('city', '').lower()
results.insert(0, Airport(**airport)) # Exact match at top name_l = airport['name'].lower()
country_l = airport.get('country', '').lower()
if iata_l in seen:
continue continue
# Search in IATA code (partial match) obj = Airport(**airport)
if query in airport['iata'].lower():
results.append(Airport(**airport)) if iata_l == query:
p0_exact_iata.append(obj)
elif iata_l.startswith(query):
p1_iata_prefix.append(obj)
elif city_l.startswith(query):
p2_city_prefix.append(obj)
elif query in city_l:
p3_city_contains.append(obj)
elif name_l.startswith(query):
p4_name_prefix.append(obj)
elif query in name_l:
p5_name_contains.append(obj)
elif query in country_l:
p6_country.append(obj)
else:
continue continue
# Search in city name seen.add(iata_l)
if query in airport.get('city', '').lower():
results.append(Airport(**airport))
continue
# Search in airport name
if query in airport['name'].lower():
results.append(Airport(**airport))
continue
# Search in country code
if query in airport['country'].lower():
results.append(Airport(**airport))
continue
except Exception: except Exception:
# Skip airports with invalid data (e.g., invalid IATA codes like 'DU9') # Skip airports with invalid data (e.g., invalid IATA codes like 'DU9')
continue continue
results = (
p0_exact_iata + p1_iata_prefix + p2_city_prefix +
p3_city_contains + p4_name_prefix + p5_name_contains + p6_country
)
# Calculate pagination # Calculate pagination
total = len(results) total = len(results)
total_pages = math.ceil(total / limit) if total > 0 else 0 total_pages = math.ceil(total / limit) if total > 0 else 0
@@ -1107,7 +1254,7 @@ async def create_scan(request: ScanRequest):
SELECT id, origin, country, start_date, end_date, SELECT id, origin, country, start_date, end_date,
created_at, updated_at, status, total_routes, created_at, updated_at, status, total_routes,
routes_scanned, total_flights, error_message, routes_scanned, total_flights, error_message,
seat_class, adults seat_class, adults, scheduled_scan_id
FROM scans FROM scans
WHERE id = ? WHERE id = ?
""", (scan_id,)) """, (scan_id,))
@@ -1132,7 +1279,8 @@ async def create_scan(request: ScanRequest):
total_flights=row[10], total_flights=row[10],
error_message=row[11], error_message=row[11],
seat_class=row[12], seat_class=row[12],
adults=row[13] adults=row[13],
scheduled_scan_id=row[14] if len(row) > 14 else None
) )
logging.info(f"Scan created: ID={scan_id}, origin={scan.origin}, country={scan.country}, dates={scan.start_date} to {scan.end_date}") logging.info(f"Scan created: ID={scan_id}, origin={scan.origin}, country={scan.country}, dates={scan.start_date} to {scan.end_date}")
@@ -1211,7 +1359,7 @@ async def list_scans(
SELECT id, origin, country, start_date, end_date, SELECT id, origin, country, start_date, end_date,
created_at, updated_at, status, total_routes, created_at, updated_at, status, total_routes,
routes_scanned, total_flights, error_message, routes_scanned, total_flights, error_message,
seat_class, adults seat_class, adults, scheduled_scan_id
FROM scans FROM scans
{where_clause} {where_clause}
ORDER BY created_at DESC ORDER BY created_at DESC
@@ -1238,7 +1386,8 @@ async def list_scans(
total_flights=row[10], total_flights=row[10],
error_message=row[11], error_message=row[11],
seat_class=row[12], seat_class=row[12],
adults=row[13] adults=row[13],
scheduled_scan_id=row[14] if len(row) > 14 else None
)) ))
# Build pagination metadata # Build pagination metadata
@@ -1279,7 +1428,7 @@ async def get_scan_status(scan_id: int):
SELECT id, origin, country, start_date, end_date, SELECT id, origin, country, start_date, end_date,
created_at, updated_at, status, total_routes, created_at, updated_at, status, total_routes,
routes_scanned, total_flights, error_message, routes_scanned, total_flights, error_message,
seat_class, adults seat_class, adults, scheduled_scan_id
FROM scans FROM scans
WHERE id = ? WHERE id = ?
""", (scan_id,)) """, (scan_id,))
@@ -1307,7 +1456,8 @@ async def get_scan_status(scan_id: int):
total_flights=row[10], total_flights=row[10],
error_message=row[11], error_message=row[11],
seat_class=row[12], seat_class=row[12],
adults=row[13] adults=row[13],
scheduled_scan_id=row[14] if len(row) > 14 else None
) )
except HTTPException: except HTTPException:
@@ -1321,6 +1471,42 @@ async def get_scan_status(scan_id: int):
) )
@router_v1.delete("/scans/{scan_id}", status_code=204)
async def delete_scan(scan_id: int):
"""
Delete a scan and all its associated routes and flights (CASCADE).
Returns 409 if the scan is currently running or pending.
"""
try:
conn = get_connection()
cursor = conn.cursor()
cursor.execute("SELECT status FROM scans WHERE id = ?", (scan_id,))
row = cursor.fetchone()
if not row:
conn.close()
raise HTTPException(status_code=404, detail=f"Scan not found: {scan_id}")
if row[0] in ('pending', 'running'):
conn.close()
raise HTTPException(
status_code=409,
detail="Cannot delete a scan that is currently pending or running."
)
cursor.execute("DELETE FROM scans WHERE id = ?", (scan_id,))
conn.commit()
conn.close()
logging.info(f"Scan {scan_id} deleted")
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to delete scan: {str(e)}")
@router_v1.get("/scans/{scan_id}/routes", response_model=PaginatedResponse[Route]) @router_v1.get("/scans/{scan_id}/routes", response_model=PaginatedResponse[Route])
async def get_scan_routes( async def get_scan_routes(
scan_id: int, scan_id: int,
@@ -1378,7 +1564,8 @@ async def get_scan_routes(
rows = cursor.fetchall() rows = cursor.fetchall()
conn.close() conn.close()
# Convert to Route models # Convert to Route models, enriching name/city from airport DB when missing
lookup = _iata_lookup()
routes = [] routes = []
for row in rows: for row in rows:
# Parse airlines JSON # Parse airlines JSON
@@ -1387,12 +1574,22 @@ async def get_scan_routes(
except: except:
airlines = [] airlines = []
dest = row[2]
dest_name = row[3] or dest
dest_city = row[4] or ''
# If name was never resolved (stored as IATA code), look it up now
if dest_name == dest:
airport = lookup.get(dest, {})
dest_name = airport.get('name', dest)
dest_city = airport.get('city', dest_city)
routes.append(Route( routes.append(Route(
id=row[0], id=row[0],
scan_id=row[1], scan_id=row[1],
destination=row[2], destination=dest,
destination_name=row[3], destination_name=dest_name,
destination_city=row[4], destination_city=dest_city,
flight_count=row[5], flight_count=row[5],
airlines=airlines, airlines=airlines,
min_price=row[7], min_price=row[7],
@@ -1586,7 +1783,7 @@ async def get_logs(
@router_v1.get("/flights/{route_id}") @router_v1.get("/flights/{route_id}")
async def get_flights(route_id: str): async def get_flights_stub(route_id: str):
""" """
Get all flights for a specific route. Get all flights for a specific route.
@@ -1596,6 +1793,348 @@ async def get_flights(route_id: str):
raise HTTPException(status_code=501, detail="Flights endpoint not yet implemented") raise HTTPException(status_code=501, detail="Flights endpoint not yet implemented")
# =============================================================================
# Schedules
# =============================================================================
class CreateScheduleRequest(BaseModel):
"""Request body for creating or updating a scheduled scan."""
origin: str = Field(..., description="Origin airport IATA code (3 letters)")
country: str = Field(..., description="Destination country ISO code (2 letters) or comma-separated IATA codes")
window_months: int = Field(1, ge=1, le=12, description="Months of data per scan run")
seat_class: str = Field('economy', description="Seat class")
adults: int = Field(1, ge=1, le=9, description="Number of adults")
label: Optional[str] = Field(None, description="Human-readable name for this schedule")
frequency: str = Field(..., description="Recurrence: daily | weekly | monthly")
hour: int = Field(6, ge=0, le=23, description="UTC hour (023)")
minute: int = Field(0, ge=0, le=59, description="UTC minute (059)")
day_of_week: Optional[int] = Field(None, ge=0, le=6, description="Required for weekly (0=Mon)")
day_of_month: Optional[int] = Field(None, ge=1, le=28, description="Required for monthly (128)")
@validator('origin', pre=True)
def uppercase_origin(cls, v):
return v.strip().upper() if v else v
@validator('country', pre=True)
def uppercase_country(cls, v):
return v.strip().upper() if v else v
@validator('frequency')
def validate_frequency(cls, v):
if v not in ('daily', 'weekly', 'monthly'):
raise ValueError("frequency must be daily, weekly, or monthly")
return v
@validator('day_of_week', always=True)
def validate_day_of_week(cls, v, values):
if values.get('frequency') == 'weekly' and v is None:
raise ValueError("day_of_week is required when frequency is weekly")
return v
@validator('day_of_month', always=True)
def validate_day_of_month(cls, v, values):
if values.get('frequency') == 'monthly' and v is None:
raise ValueError("day_of_month is required when frequency is monthly")
return v
class UpdateScheduleRequest(BaseModel):
"""Request body for PATCH /schedules/{id}."""
enabled: Optional[bool] = None
label: Optional[str] = None
frequency: Optional[str] = None
hour: Optional[int] = Field(None, ge=0, le=23)
minute: Optional[int] = Field(None, ge=0, le=59)
day_of_week: Optional[int] = Field(None, ge=0, le=6)
day_of_month: Optional[int] = Field(None, ge=1, le=28)
window_months: Optional[int] = Field(None, ge=1, le=12)
seat_class: Optional[str] = None
adults: Optional[int] = Field(None, ge=1, le=9)
@validator('frequency')
def validate_frequency(cls, v):
if v is not None and v not in ('daily', 'weekly', 'monthly'):
raise ValueError("frequency must be daily, weekly, or monthly")
return v
class Schedule(BaseModel):
"""A recurring scheduled scan."""
id: int
origin: str
country: str
window_months: int
seat_class: str
adults: int
label: Optional[str]
frequency: str
hour: int
minute: int
day_of_week: Optional[int]
day_of_month: Optional[int]
enabled: bool
last_run_at: Optional[str]
next_run_at: str
created_at: str
recent_scan_ids: List[int]
def _row_to_schedule(row, recent_scan_ids: list) -> Schedule:
"""Convert a DB row (sqlite3.Row or tuple) to a Schedule model."""
return Schedule(
id=row['id'],
origin=row['origin'],
country=row['country'],
window_months=row['window_months'],
seat_class=row['seat_class'],
adults=row['adults'],
label=row['label'],
frequency=row['frequency'],
hour=row['hour'],
minute=row['minute'],
day_of_week=row['day_of_week'],
day_of_month=row['day_of_month'],
enabled=bool(row['enabled']),
last_run_at=row['last_run_at'],
next_run_at=row['next_run_at'],
created_at=row['created_at'],
recent_scan_ids=recent_scan_ids,
)
def _get_recent_scan_ids(conn, schedule_id: int, limit: int = 5) -> list:
rows = conn.execute("""
SELECT id FROM scans
WHERE scheduled_scan_id = ?
ORDER BY created_at DESC
LIMIT ?
""", (schedule_id, limit)).fetchall()
return [r[0] for r in rows]
@router_v1.get("/schedules", response_model=PaginatedResponse[Schedule])
async def list_schedules(
page: int = Query(1, ge=1),
limit: int = Query(20, ge=1, le=100),
):
"""List all scheduled scans with pagination."""
try:
conn = get_connection()
total = conn.execute("SELECT COUNT(*) FROM scheduled_scans").fetchone()[0]
total_pages = math.ceil(total / limit) if total > 0 else 0
offset = (page - 1) * limit
rows = conn.execute("""
SELECT * FROM scheduled_scans
ORDER BY created_at DESC
LIMIT ? OFFSET ?
""", (limit, offset)).fetchall()
items = [
_row_to_schedule(r, _get_recent_scan_ids(conn, r['id']))
for r in rows
]
conn.close()
pagination = PaginationMetadata(
page=page, limit=limit, total=total, pages=total_pages,
has_next=page < total_pages, has_prev=page > 1,
)
return PaginatedResponse(data=items, pagination=pagination)
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to list schedules: {e}")
@router_v1.post("/schedules", response_model=Schedule, status_code=201)
async def create_schedule(request: CreateScheduleRequest):
"""Create a new scheduled scan."""
try:
next_run = compute_next_run(
request.frequency, request.hour, request.minute,
request.day_of_week, request.day_of_month,
)
next_run_str = next_run.strftime('%Y-%m-%d %H:%M:%S')
conn = get_connection()
conn.execute("""
INSERT INTO scheduled_scans (
origin, country, window_months, seat_class, adults,
label, frequency, hour, minute, day_of_week, day_of_month,
enabled, next_run_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)
""", (
request.origin, request.country, request.window_months,
request.seat_class, request.adults, request.label,
request.frequency, request.hour, request.minute,
request.day_of_week, request.day_of_month, next_run_str,
))
conn.commit()
sched_id = conn.execute("SELECT last_insert_rowid()").fetchone()[0]
row = conn.execute(
"SELECT * FROM scheduled_scans WHERE id = ?", (sched_id,)
).fetchone()
result = _row_to_schedule(row, [])
conn.close()
return result
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to create schedule: {e}")
@router_v1.get("/schedules/{schedule_id}", response_model=Schedule)
async def get_schedule(schedule_id: int):
"""Get a single schedule by ID, including its last 5 scan IDs."""
try:
conn = get_connection()
row = conn.execute(
"SELECT * FROM scheduled_scans WHERE id = ?", (schedule_id,)
).fetchone()
if not row:
conn.close()
raise HTTPException(status_code=404, detail=f"Schedule not found: {schedule_id}")
recent = _get_recent_scan_ids(conn, schedule_id)
result = _row_to_schedule(row, recent)
conn.close()
return result
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to get schedule: {e}")
@router_v1.patch("/schedules/{schedule_id}", response_model=Schedule)
async def update_schedule(schedule_id: int, request: UpdateScheduleRequest):
"""Update schedule fields. Recomputes next_run_at if schedule params change."""
try:
conn = get_connection()
row = conn.execute(
"SELECT * FROM scheduled_scans WHERE id = ?", (schedule_id,)
).fetchone()
if not row:
conn.close()
raise HTTPException(status_code=404, detail=f"Schedule not found: {schedule_id}")
# Merge updates on top of existing values
frequency = request.frequency if request.frequency is not None else row['frequency']
hour = request.hour if request.hour is not None else row['hour']
minute = request.minute if request.minute is not None else row['minute']
day_of_week = request.day_of_week if request.day_of_week is not None else row['day_of_week']
day_of_month = request.day_of_month if request.day_of_month is not None else row['day_of_month']
next_run = compute_next_run(frequency, hour, minute, day_of_week, day_of_month)
next_run_str = next_run.strftime('%Y-%m-%d %H:%M:%S')
enabled_val = int(request.enabled) if request.enabled is not None else row['enabled']
label_val = request.label if request.label is not None else row['label']
wm_val = request.window_months if request.window_months is not None else row['window_months']
sc_val = request.seat_class if request.seat_class is not None else row['seat_class']
adults_val = request.adults if request.adults is not None else row['adults']
conn.execute("""
UPDATE scheduled_scans
SET enabled = ?, label = ?, frequency = ?, hour = ?, minute = ?,
day_of_week = ?, day_of_month = ?, window_months = ?,
seat_class = ?, adults = ?, next_run_at = ?
WHERE id = ?
""", (
enabled_val, label_val, frequency, hour, minute,
day_of_week, day_of_month, wm_val, sc_val, adults_val,
next_run_str, schedule_id,
))
conn.commit()
updated_row = conn.execute(
"SELECT * FROM scheduled_scans WHERE id = ?", (schedule_id,)
).fetchone()
recent = _get_recent_scan_ids(conn, schedule_id)
result = _row_to_schedule(updated_row, recent)
conn.close()
return result
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to update schedule: {e}")
@router_v1.delete("/schedules/{schedule_id}", status_code=204)
async def delete_schedule(schedule_id: int):
"""Delete a schedule. Associated scans are kept with scheduled_scan_id set to NULL."""
try:
conn = get_connection()
row = conn.execute(
"SELECT id FROM scheduled_scans WHERE id = ?", (schedule_id,)
).fetchone()
if not row:
conn.close()
raise HTTPException(status_code=404, detail=f"Schedule not found: {schedule_id}")
# Nullify FK in scans before deleting (SQLite FK cascade may not be set)
conn.execute(
"UPDATE scans SET scheduled_scan_id = NULL WHERE scheduled_scan_id = ?",
(schedule_id,)
)
conn.execute("DELETE FROM scheduled_scans WHERE id = ?", (schedule_id,))
conn.commit()
conn.close()
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to delete schedule: {e}")
@router_v1.post("/schedules/{schedule_id}/run-now")
async def run_schedule_now(schedule_id: int):
"""Trigger a scheduled scan immediately, ignoring next_run_at."""
try:
conn = get_connection()
row = conn.execute(
"SELECT * FROM scheduled_scans WHERE id = ?", (schedule_id,)
).fetchone()
if not row:
conn.close()
raise HTTPException(status_code=404, detail=f"Schedule not found: {schedule_id}")
start_date = (date.today() + timedelta(days=1)).isoformat()
end_dt = date.today() + timedelta(days=1) + timedelta(days=30 * row['window_months'])
end_date = end_dt.isoformat()
conn.execute("""
INSERT INTO scans (
origin, country, start_date, end_date,
status, seat_class, adults, scheduled_scan_id
) VALUES (?, ?, ?, ?, 'pending', ?, ?, ?)
""", (
row['origin'], row['country'], start_date, end_date,
row['seat_class'], row['adults'], schedule_id,
))
conn.commit()
scan_id = conn.execute("SELECT last_insert_rowid()").fetchone()[0]
conn.close()
start_scan_processor(scan_id)
logging.info(f"Schedule {schedule_id}: manual run-now fired scan {scan_id}")
return {"scan_id": scan_id}
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to run schedule: {e}")
# ============================================================================= # =============================================================================
# Include Router (IMPORTANT!) # Include Router (IMPORTANT!)
# ============================================================================= # =============================================================================
@@ -1607,6 +2146,13 @@ app.include_router(router_v1)
# Helper Functions # Helper Functions
# ============================================================================= # =============================================================================
# Airports missing from the OpenFlights dataset (opened/renamed after dataset freeze)
_MISSING_AIRPORTS = [
{'iata': 'BER', 'name': 'Berlin Brandenburg Airport', 'city': 'Berlin', 'country': 'DE'},
{'iata': 'IST', 'name': 'Istanbul Airport', 'city': 'Istanbul', 'country': 'TR'},
]
@lru_cache(maxsize=1) @lru_cache(maxsize=1)
def get_airport_data(): def get_airport_data():
""" """
@@ -1637,9 +2183,28 @@ def get_airport_data():
'longitude': airport.get('lon', 0.0), 'longitude': airport.get('lon', 0.0),
}) })
# Patch in modern airports missing from the OpenFlights dataset
existing_iatas = {a['iata'] for a in airports}
for extra in _MISSING_AIRPORTS:
if extra['iata'] not in existing_iatas:
airports.append({
'iata': extra['iata'],
'name': extra['name'],
'city': extra['city'],
'country': extra['country'],
'latitude': 0.0,
'longitude': 0.0,
})
return airports return airports
@lru_cache(maxsize=1)
def _iata_lookup() -> dict:
"""Return {iata: airport_dict} built from get_airport_data(). Cached."""
return {a['iata']: a for a in get_airport_data()}
if __name__ == "__main__": if __name__ == "__main__":
import uvicorn import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000) uvicorn.run(app, host="0.0.0.0", port=8000)

View File

@@ -130,6 +130,66 @@ def _migrate_relax_country_constraint(conn, verbose=True):
print(" ✅ Migration complete: country column now accepts >= 2 chars") print(" ✅ Migration complete: country column now accepts >= 2 chars")
def _migrate_add_routes_unique_index(conn, verbose=True):
"""
Migration: Add UNIQUE index on routes(scan_id, destination).
Required for incremental route writes during active scans.
Collapses any pre-existing duplicate (scan_id, destination) rows first
(keeps the row with the lowest id) before creating the index.
"""
cursor = conn.execute(
"SELECT name FROM sqlite_master WHERE type='index' AND name='uq_routes_scan_dest'"
)
if cursor.fetchone():
return # Already migrated
if verbose:
print(" 🔄 Migrating routes table: adding UNIQUE index on (scan_id, destination)...")
# Collapse any existing duplicates (guard against edge cases)
conn.execute("""
DELETE FROM routes
WHERE id NOT IN (
SELECT MIN(id)
FROM routes
GROUP BY scan_id, destination
)
""")
conn.execute("""
CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_dest
ON routes(scan_id, destination)
""")
conn.commit()
if verbose:
print(" ✅ Migration complete: uq_routes_scan_dest index created")
def _migrate_add_scheduled_scan_id_to_scans(conn, verbose=True):
"""
Migration: add scheduled_scan_id column to scans table.
Existing rows get NULL (manual scans). New column has no inline FK
declaration because SQLite's ALTER TABLE ADD COLUMN doesn't support it;
the relationship is enforced at the application level.
"""
cursor = conn.execute("PRAGMA table_info(scans)")
columns = [row[1] for row in cursor.fetchall()]
if 'scheduled_scan_id' in columns:
return # Already migrated
if verbose:
print(" 🔄 Migrating scans table: adding scheduled_scan_id column...")
conn.execute("ALTER TABLE scans ADD COLUMN scheduled_scan_id INTEGER")
conn.commit()
if verbose:
print(" ✅ Migration complete: scheduled_scan_id column added to scans")
def initialize_database(db_path=None, verbose=True): def initialize_database(db_path=None, verbose=True):
""" """
Initialize or migrate the database. Initialize or migrate the database.
@@ -174,6 +234,8 @@ def initialize_database(db_path=None, verbose=True):
# Apply migrations before running schema # Apply migrations before running schema
_migrate_relax_country_constraint(conn, verbose) _migrate_relax_country_constraint(conn, verbose)
_migrate_add_routes_unique_index(conn, verbose)
_migrate_add_scheduled_scan_id_to_scans(conn, verbose)
# Load and execute schema # Load and execute schema
schema_sql = load_schema() schema_sql = load_schema()

View File

@@ -45,6 +45,9 @@ CREATE TABLE IF NOT EXISTS scans (
seat_class TEXT DEFAULT 'economy', seat_class TEXT DEFAULT 'economy',
adults INTEGER DEFAULT 1 CHECK(adults > 0 AND adults <= 9), adults INTEGER DEFAULT 1 CHECK(adults > 0 AND adults <= 9),
-- FK to scheduled_scans (NULL for manual scans)
scheduled_scan_id INTEGER,
-- Constraints across columns -- Constraints across columns
CHECK(end_date >= start_date), CHECK(end_date >= start_date),
CHECK(routes_scanned <= total_routes OR total_routes = 0) CHECK(routes_scanned <= total_routes OR total_routes = 0)
@@ -61,6 +64,10 @@ CREATE INDEX IF NOT EXISTS idx_scans_status
CREATE INDEX IF NOT EXISTS idx_scans_created_at CREATE INDEX IF NOT EXISTS idx_scans_created_at
ON scans(created_at DESC); -- For recent scans query ON scans(created_at DESC); -- For recent scans query
CREATE INDEX IF NOT EXISTS idx_scans_scheduled_scan_id
ON scans(scheduled_scan_id)
WHERE scheduled_scan_id IS NOT NULL;
-- ============================================================================ -- ============================================================================
-- Table: routes -- Table: routes
-- Purpose: Store discovered routes with flight statistics -- Purpose: Store discovered routes with flight statistics
@@ -111,6 +118,10 @@ CREATE INDEX IF NOT EXISTS idx_routes_min_price
ON routes(min_price) ON routes(min_price)
WHERE min_price IS NOT NULL; -- Partial index for routes with prices WHERE min_price IS NOT NULL; -- Partial index for routes with prices
-- One route row per (scan, destination) — enables incremental upsert writes
CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_dest
ON routes(scan_id, destination);
-- ============================================================================ -- ============================================================================
-- Triggers: Auto-update timestamps and aggregates -- Triggers: Auto-update timestamps and aggregates
-- ============================================================================ -- ============================================================================
@@ -240,7 +251,9 @@ ORDER BY created_at ASC;
-- Initial Data: None (tables start empty) -- Initial Data: None (tables start empty)
-- ============================================================================ -- ============================================================================
-- ============================================================================
-- Schema version tracking (for future migrations) -- Schema version tracking (for future migrations)
-- ============================================================================
CREATE TABLE IF NOT EXISTS schema_version ( CREATE TABLE IF NOT EXISTS schema_version (
version INTEGER PRIMARY KEY, version INTEGER PRIMARY KEY,
applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
@@ -250,6 +263,64 @@ CREATE TABLE IF NOT EXISTS schema_version (
INSERT OR IGNORE INTO schema_version (version, description) INSERT OR IGNORE INTO schema_version (version, description)
VALUES (1, 'Initial web app schema with scans and routes tables'); VALUES (1, 'Initial web app schema with scans and routes tables');
-- ============================================================================
-- Table: scheduled_scans
-- Purpose: Define recurring scan schedules (daily / weekly / monthly)
-- ============================================================================
CREATE TABLE IF NOT EXISTS scheduled_scans (
id INTEGER PRIMARY KEY AUTOINCREMENT,
-- Scan parameters (same as scans table)
origin TEXT NOT NULL CHECK(length(origin) = 3),
country TEXT NOT NULL CHECK(length(country) >= 2),
window_months INTEGER NOT NULL DEFAULT 1
CHECK(window_months >= 1 AND window_months <= 12),
seat_class TEXT NOT NULL DEFAULT 'economy',
adults INTEGER NOT NULL DEFAULT 1
CHECK(adults > 0 AND adults <= 9),
-- Schedule definition
frequency TEXT NOT NULL
CHECK(frequency IN ('daily', 'weekly', 'monthly')),
hour INTEGER NOT NULL DEFAULT 6
CHECK(hour >= 0 AND hour <= 23),
minute INTEGER NOT NULL DEFAULT 0
CHECK(minute >= 0 AND minute <= 59),
day_of_week INTEGER CHECK(day_of_week >= 0 AND day_of_week <= 6),
day_of_month INTEGER CHECK(day_of_month >= 1 AND day_of_month <= 28),
-- State
enabled INTEGER NOT NULL DEFAULT 1,
label TEXT,
last_run_at TIMESTAMP,
next_run_at TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- Frequency-specific field requirements
CHECK(
(frequency = 'weekly' AND day_of_week IS NOT NULL) OR
(frequency = 'monthly' AND day_of_month IS NOT NULL) OR
(frequency = 'daily')
)
);
-- Fast lookup of due schedules (partial index on enabled rows only)
CREATE INDEX IF NOT EXISTS idx_scheduled_scans_next_run
ON scheduled_scans(next_run_at)
WHERE enabled = 1;
-- Auto-update updated_at on every PATCH
CREATE TRIGGER IF NOT EXISTS update_scheduled_scans_timestamp
AFTER UPDATE ON scheduled_scans
FOR EACH ROW BEGIN
UPDATE scheduled_scans SET updated_at = CURRENT_TIMESTAMP WHERE id = NEW.id;
END;
INSERT OR IGNORE INTO schema_version (version, description)
VALUES (2, 'Add scheduled_scans table');
-- ============================================================================ -- ============================================================================
-- Verification Queries (for testing) -- Verification Queries (for testing)
-- ============================================================================ -- ============================================================================

View File

@@ -1,15 +1,45 @@
name: flight-radar # pins the project name — must match COMPOSE_PROJECT in .gitea/workflows/deploy.yml
services: services:
app: backend:
build: . build:
container_name: flight-radar context: .
dockerfile: Dockerfile.backend
container_name: flight-radar-backend
restart: unless-stopped restart: unless-stopped
ports:
- "80:80"
environment: environment:
- DATABASE_PATH=/app/data/cache.db - DATABASE_PATH=/app/data/cache.db
volumes: volumes:
- flight-radar-data:/app/data - flight-radar-data:/app/data
networks:
- default
- domverse
# No ports exposed — only reachable by the frontend via nginx proxy
frontend:
build:
context: .
dockerfile: Dockerfile.frontend
container_name: flight-radar-frontend
restart: unless-stopped
depends_on:
- backend
networks:
- default # shares default compose network with backend (nginx → http://backend:8000)
- domverse # Traefik discovers the container on this network
labels:
- "traefik.enable=true"
- "traefik.http.routers.flight-radar.rule=Host(`flights.domverse-berlin.eu`)"
- "traefik.http.routers.flight-radar.entrypoints=https"
- "traefik.http.routers.flight-radar.tls.certresolver=http"
- "traefik.http.routers.flight-radar.middlewares=authentik@docker"
- "traefik.http.services.flight-radar.loadbalancer.server.port=80"
volumes: volumes:
flight-radar-data: flight-radar-data:
driver: local driver: local
networks:
default: {} # explicit declaration required when any service has a custom networks block
domverse:
external: true

View File

@@ -3,6 +3,7 @@ import Layout from './components/Layout';
import Dashboard from './pages/Dashboard'; import Dashboard from './pages/Dashboard';
import Scans from './pages/Scans'; import Scans from './pages/Scans';
import ScanDetails from './pages/ScanDetails'; import ScanDetails from './pages/ScanDetails';
import Schedules from './pages/Schedules';
import Airports from './pages/Airports'; import Airports from './pages/Airports';
import Logs from './pages/Logs'; import Logs from './pages/Logs';
import ErrorBoundary from './components/ErrorBoundary'; import ErrorBoundary from './components/ErrorBoundary';
@@ -16,6 +17,7 @@ function App() {
<Route index element={<Dashboard />} /> <Route index element={<Dashboard />} />
<Route path="scans" element={<Scans />} /> <Route path="scans" element={<Scans />} />
<Route path="scans/:id" element={<ScanDetails />} /> <Route path="scans/:id" element={<ScanDetails />} />
<Route path="schedules" element={<Schedules />} />
<Route path="airports" element={<Airports />} /> <Route path="airports" element={<Airports />} />
<Route path="logs" element={<Logs />} /> <Route path="logs" element={<Logs />} />
</Route> </Route>

View File

@@ -23,6 +23,41 @@ export interface Scan {
error_message?: string; error_message?: string;
seat_class: string; seat_class: string;
adults: number; adults: number;
scheduled_scan_id?: number;
}
export interface Schedule {
id: number;
origin: string;
country: string;
window_months: number;
seat_class: string;
adults: number;
label?: string;
frequency: 'daily' | 'weekly' | 'monthly';
hour: number;
minute: number;
day_of_week?: number;
day_of_month?: number;
enabled: boolean;
last_run_at?: string;
next_run_at: string;
created_at: string;
recent_scan_ids: number[];
}
export interface CreateScheduleRequest {
origin: string;
country: string;
window_months?: number;
seat_class?: string;
adults?: number;
label?: string;
frequency: 'daily' | 'weekly' | 'monthly';
hour?: number;
minute?: number;
day_of_week?: number;
day_of_month?: number;
} }
export interface Route { export interface Route {
@@ -123,6 +158,8 @@ export const scanApi = {
if (destination) params.destination = destination; if (destination) params.destination = destination;
return api.get<PaginatedResponse<Flight>>(`/scans/${id}/flights`, { params }); return api.get<PaginatedResponse<Flight>>(`/scans/${id}/flights`, { params });
}, },
delete: (id: number) => api.delete(`/scans/${id}`),
}; };
export const airportApi = { export const airportApi = {
@@ -133,6 +170,26 @@ export const airportApi = {
}, },
}; };
export const scheduleApi = {
list: (page = 1, limit = 20) =>
api.get<PaginatedResponse<Schedule>>('/schedules', { params: { page, limit } }),
get: (id: number) =>
api.get<Schedule>(`/schedules/${id}`),
create: (data: CreateScheduleRequest) =>
api.post<Schedule>('/schedules', data),
update: (id: number, data: Partial<CreateScheduleRequest> & { enabled?: boolean }) =>
api.patch<Schedule>(`/schedules/${id}`, data),
delete: (id: number) =>
api.delete(`/schedules/${id}`),
runNow: (id: number) =>
api.post<{ scan_id: number }>(`/schedules/${id}/run-now`),
};
export const logApi = { export const logApi = {
list: (page = 1, limit = 50, level?: string, search?: string) => { list: (page = 1, limit = 50, level?: string, search?: string) => {
const params: any = { page, limit }; const params: any = { page, limit };

View File

@@ -7,6 +7,7 @@ import {
ScrollText, ScrollText,
PlaneTakeoff, PlaneTakeoff,
Plus, Plus,
CalendarClock,
} from 'lucide-react'; } from 'lucide-react';
import { cn } from '../lib/utils'; import { cn } from '../lib/utils';
@@ -18,8 +19,9 @@ type NavItem = {
const PRIMARY_NAV: NavItem[] = [ const PRIMARY_NAV: NavItem[] = [
{ icon: LayoutDashboard, label: 'Dashboard', path: '/' }, { icon: LayoutDashboard, label: 'Dashboard', path: '/' },
{ icon: ScanSearch, label: 'Scans', path: '/scans' }, { icon: ScanSearch, label: 'Scans', path: '/scans' },
{ icon: MapPin, label: 'Airports', path: '/airports' }, { icon: CalendarClock, label: 'Schedules', path: '/schedules' },
{ icon: MapPin, label: 'Airports', path: '/airports' },
]; ];
const SECONDARY_NAV: NavItem[] = [ const SECONDARY_NAV: NavItem[] = [
@@ -32,6 +34,7 @@ function getPageTitle(pathname: string): string {
if (pathname === '/') return 'Dashboard'; if (pathname === '/') return 'Dashboard';
if (pathname.startsWith('/scans/')) return 'Scan Details'; if (pathname.startsWith('/scans/')) return 'Scan Details';
if (pathname === '/scans') return 'New Scan'; if (pathname === '/scans') return 'New Scan';
if (pathname === '/schedules') return 'Schedules';
if (pathname === '/airports') return 'Airports'; if (pathname === '/airports') return 'Airports';
if (pathname === '/logs') return 'Logs'; if (pathname === '/logs') return 'Logs';
return 'Flight Radar'; return 'Flight Radar';

View File

@@ -1,9 +1,10 @@
import { Fragment, useEffect, useState } from 'react'; import { Fragment, useEffect, useState } from 'react';
import { useParams, useNavigate } from 'react-router-dom'; import { useParams, useNavigate, Link } from 'react-router-dom';
import { import {
ArrowLeft, ArrowLeft,
PlaneTakeoff, PlaneTakeoff,
Calendar, Calendar,
CalendarClock,
Users, Users,
Armchair, Armchair,
Clock, Clock,
@@ -13,6 +14,9 @@ import {
MapPin, MapPin,
AlertCircle, AlertCircle,
Loader2, Loader2,
RotateCcw,
Trash2,
Info,
} from 'lucide-react'; } from 'lucide-react';
import { scanApi } from '../api'; import { scanApi } from '../api';
import type { Scan, Route, Flight } from '../api'; import type { Scan, Route, Flight } from '../api';
@@ -45,6 +49,9 @@ export default function ScanDetails() {
const [expandedRoute, setExpandedRoute] = useState<string | null>(null); const [expandedRoute, setExpandedRoute] = useState<string | null>(null);
const [flightsByDest, setFlightsByDest] = useState<Record<string, Flight[]>>({}); const [flightsByDest, setFlightsByDest] = useState<Record<string, Flight[]>>({});
const [loadingFlights, setLoadingFlights] = useState<string | null>(null); const [loadingFlights, setLoadingFlights] = useState<string | null>(null);
const [rerunning, setRerunning] = useState(false);
const [confirmDelete, setConfirmDelete] = useState(false);
const [deleting, setDeleting] = useState(false);
useEffect(() => { useEffect(() => {
if (id) loadScanDetails(); if (id) loadScanDetails();
@@ -110,6 +117,45 @@ export default function ScanDetails() {
} }
}; };
const handleRerun = async () => {
if (!scan) return;
setRerunning(true);
try {
// Compute window from stored dates so the new scan covers the same span
const ms = new Date(scan.end_date).getTime() - new Date(scan.start_date).getTime();
const window_months = Math.max(1, Math.round(ms / (1000 * 60 * 60 * 24 * 30)));
// country column holds either "IT" or "BRI,BDS"
const isAirports = scan.country.includes(',');
const resp = await scanApi.create({
origin: scan.origin,
window_months,
seat_class: scan.seat_class as 'economy' | 'premium' | 'business' | 'first',
adults: scan.adults,
...(isAirports
? { destinations: scan.country.split(',') }
: { country: scan.country }),
});
navigate(`/scans/${resp.data.id}`);
} catch {
// silently fall through — the navigate won't happen
} finally {
setRerunning(false);
}
};
const handleDelete = async () => {
if (!scan) return;
setDeleting(true);
try {
await scanApi.delete(scan.id);
navigate('/');
} catch {
setDeleting(false);
setConfirmDelete(false);
}
};
const SortIcon = ({ field }: { field: typeof sortField }) => { const SortIcon = ({ field }: { field: typeof sortField }) => {
if (sortField !== field) return <ChevronUp size={14} className="opacity-30" />; if (sortField !== field) return <ChevronUp size={14} className="opacity-30" />;
return sortDirection === 'asc' return sortDirection === 'asc'
@@ -176,6 +222,16 @@ export default function ScanDetails() {
<h1 className="text-xl font-semibold text-on-surface"> <h1 className="text-xl font-semibold text-on-surface">
{scan.origin} {scan.country} {scan.origin} {scan.country}
</h1> </h1>
{scan.scheduled_scan_id != null && (
<Link
to={`/schedules`}
className="inline-flex items-center gap-1 px-2 py-0.5 rounded-full text-xs font-medium bg-primary-container text-on-primary-container hover:opacity-80 transition-opacity"
title={`Scheduled scan #${scan.scheduled_scan_id}`}
>
<CalendarClock size={11} aria-hidden="true" />
Scheduled
</Link>
)}
</div> </div>
<StatusChip status={scan.status as ScanStatus} /> <StatusChip status={scan.status as ScanStatus} />
</div> </div>
@@ -203,6 +259,49 @@ export default function ScanDetails() {
Created {formatDate(scan.created_at)} Created {formatDate(scan.created_at)}
</p> </p>
)} )}
{/* Row 4: actions */}
<div className="mt-4 pt-4 border-t border-outline flex items-center justify-end gap-2">
{/* Re-run */}
<button
onClick={handleRerun}
disabled={rerunning || isActive}
className="inline-flex items-center gap-1.5 px-3 py-1.5 text-sm font-medium rounded-xs border border-outline text-on-surface hover:bg-surface-2 disabled:opacity-40 disabled:cursor-not-allowed transition-colors"
>
<RotateCcw size={14} className={rerunning ? 'animate-spin' : ''} />
{rerunning ? 'Starting…' : 'Re-run'}
</button>
{/* Delete — inline confirm */}
{confirmDelete ? (
<div className="inline-flex items-center gap-1.5">
<span className="text-sm text-on-surface-variant">Delete this scan?</span>
<button
onClick={handleDelete}
disabled={deleting}
className="inline-flex items-center gap-1 px-3 py-1.5 text-sm font-medium rounded-xs bg-error text-white hover:bg-error/90 disabled:opacity-60 transition-colors"
>
{deleting ? 'Deleting…' : 'Yes, delete'}
</button>
<button
onClick={() => setConfirmDelete(false)}
disabled={deleting}
className="px-3 py-1.5 text-sm font-medium rounded-xs border border-outline text-on-surface hover:bg-surface-2 transition-colors"
>
Cancel
</button>
</div>
) : (
<button
onClick={() => setConfirmDelete(true)}
disabled={isActive}
className="inline-flex items-center gap-1.5 px-3 py-1.5 text-sm font-medium rounded-xs border border-error/40 text-error hover:bg-error/5 disabled:opacity-40 disabled:cursor-not-allowed transition-colors"
>
<Trash2 size={14} />
Delete
</button>
)}
</div>
</div> </div>
{/* ── Stat cards ────────────────────────────────────────────── */} {/* ── Stat cards ────────────────────────────────────────────── */}
@@ -332,6 +431,19 @@ export default function ScanDetails() {
<span className="text-sm text-on-surface-variant truncate max-w-[180px]"> <span className="text-sm text-on-surface-variant truncate max-w-[180px]">
{route.destination_name || route.destination_city || ''} {route.destination_name || route.destination_city || ''}
</span> </span>
{/* Info icon + tooltip — only when useful name data exists */}
{(route.destination_name && route.destination_name !== route.destination) || route.destination_city ? (
<span
className="relative group/tip inline-flex shrink-0"
onClick={e => e.stopPropagation()}
>
<Info size={12} className="text-on-surface-variant/40 hover:text-on-surface-variant/70 cursor-help transition-colors" />
<span className="pointer-events-none absolute bottom-full left-1/2 -translate-x-1/2 mb-2 z-50 invisible group-hover/tip:visible bg-gray-900 text-white text-xs rounded px-2 py-1 whitespace-nowrap shadow-lg">
{[route.destination_name, route.destination_city].filter(s => s && s !== route.destination).join(', ')}
<span className="absolute top-full left-1/2 -translate-x-1/2 border-4 border-transparent border-t-gray-900" />
</span>
</span>
) : null}
</div> </div>
</td> </td>
{/* Flights */} {/* Flights */}

View File

@@ -0,0 +1,609 @@
import { useEffect, useState } from 'react';
import {
Globe,
PlaneTakeoff,
Minus,
Plus,
Play,
Trash2,
CalendarClock,
} from 'lucide-react';
import { scheduleApi } from '../api';
import type { Schedule, CreateScheduleRequest } from '../api';
import AirportSearch from '../components/AirportSearch';
import SegmentedButton from '../components/SegmentedButton';
import AirportChip from '../components/AirportChip';
import Button from '../components/Button';
import Toast from '../components/Toast';
import EmptyState from '../components/EmptyState';
import { cn } from '../lib/utils';
// ── Helpers ──────────────────────────────────────────────────────────────────
const DAYS = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'];
function formatNextRun(utcStr: string): string {
// utcStr is like "2026-03-01 06:00:00" (no Z suffix from SQLite)
const d = new Date(utcStr.replace(' ', 'T') + 'Z');
if (isNaN(d.getTime())) return utcStr;
return d.toLocaleString(undefined, {
month: 'short', day: 'numeric',
hour: '2-digit', minute: '2-digit',
});
}
function formatLastRun(utcStr?: string): string {
if (!utcStr) return '—';
const d = new Date(utcStr.replace(' ', 'T') + 'Z');
if (isNaN(d.getTime())) return utcStr;
return d.toLocaleString(undefined, {
month: 'short', day: 'numeric',
hour: '2-digit', minute: '2-digit',
});
}
function describeSchedule(s: Schedule): string {
const pad = (n: number) => String(n).padStart(2, '0');
const time = `${pad(s.hour)}:${pad(s.minute)} UTC`;
if (s.frequency === 'daily') return `Every day at ${time}`;
if (s.frequency === 'weekly') return `Every ${DAYS[s.day_of_week ?? 0]} at ${time}`;
if (s.frequency === 'monthly') return `${s.day_of_month}th of month at ${time}`;
return s.frequency;
}
// ── Form state ────────────────────────────────────────────────────────────────
interface FormState {
origin: string;
country: string;
window_months: number;
seat_class: string;
adults: number;
label: string;
frequency: 'daily' | 'weekly' | 'monthly';
hour: number;
minute: number;
day_of_week: number;
day_of_month: number;
}
const defaultForm = (): FormState => ({
origin: '',
country: '',
window_months: 1,
seat_class: 'economy',
adults: 1,
label: '',
frequency: 'weekly',
hour: 6,
minute: 0,
day_of_week: 0,
day_of_month: 1,
});
interface FormErrors {
origin?: string;
country?: string;
airports?: string;
hour?: string;
minute?: string;
}
// ── Component ─────────────────────────────────────────────────────────────────
export default function Schedules() {
const [schedules, setSchedules] = useState<Schedule[]>([]);
const [loading, setLoading] = useState(true);
const [showForm, setShowForm] = useState(false);
const [destinationMode, setDestinationMode] = useState<'country' | 'airports'>('country');
const [selectedAirports, setSelectedAirports] = useState<string[]>([]);
const [form, setForm] = useState<FormState>(defaultForm);
const [errors, setErrors] = useState<FormErrors>({});
const [saving, setSaving] = useState(false);
const [runningId, setRunningId] = useState<number | null>(null);
const [deletingId, setDeletingId] = useState<number | null>(null);
const [toast, setToast] = useState<{ message: string; type: 'success' | 'error' } | null>(null);
useEffect(() => { loadSchedules(); }, []);
const loadSchedules = async () => {
try {
setLoading(true);
const res = await scheduleApi.list(1, 100);
setSchedules(res.data.data);
} catch {
setToast({ message: 'Failed to load schedules', type: 'error' });
} finally {
setLoading(false);
}
};
const validate = (): boolean => {
const next: FormErrors = {};
if (!form.origin || form.origin.length !== 3)
next.origin = 'Enter a valid 3-letter IATA code';
if (destinationMode === 'country' && (!form.country || form.country.length < 2))
next.country = 'Enter a valid 2-letter country code';
if (destinationMode === 'airports' && selectedAirports.length === 0)
next.airports = 'Add at least one destination airport';
if (form.hour < 0 || form.hour > 23)
next.hour = '023';
if (form.minute < 0 || form.minute > 59)
next.minute = '059';
setErrors(next);
return Object.keys(next).length === 0;
};
const handleCreate = async (e: React.FormEvent) => {
e.preventDefault();
if (!validate()) return;
setSaving(true);
try {
const req: CreateScheduleRequest = {
origin: form.origin,
country: destinationMode === 'country'
? form.country
: selectedAirports.join(','),
window_months: form.window_months,
seat_class: form.seat_class,
adults: form.adults,
label: form.label || undefined,
frequency: form.frequency,
hour: form.hour,
minute: form.minute,
...(form.frequency === 'weekly' ? { day_of_week: form.day_of_week } : {}),
...(form.frequency === 'monthly' ? { day_of_month: form.day_of_month } : {}),
};
await scheduleApi.create(req);
setToast({ message: 'Schedule created', type: 'success' });
setShowForm(false);
setForm(defaultForm());
setSelectedAirports([]);
loadSchedules();
} catch (err: any) {
const msg = err.response?.data?.detail || 'Failed to create schedule';
setToast({ message: typeof msg === 'string' ? msg : JSON.stringify(msg), type: 'error' });
} finally {
setSaving(false);
}
};
const toggleEnabled = async (s: Schedule) => {
try {
const updated = await scheduleApi.update(s.id, { enabled: !s.enabled });
setSchedules(prev => prev.map(x => x.id === s.id ? updated.data : x));
} catch {
setToast({ message: 'Failed to update schedule', type: 'error' });
}
};
const handleRunNow = async (s: Schedule) => {
setRunningId(s.id);
try {
const res = await scheduleApi.runNow(s.id);
setToast({ message: `Scan #${res.data.scan_id} started`, type: 'success' });
loadSchedules();
} catch {
setToast({ message: 'Failed to trigger scan', type: 'error' });
} finally {
setRunningId(null);
}
};
const handleDelete = async (s: Schedule) => {
if (!confirm(`Delete schedule "${s.label || `${s.origin}${s.country}`}"?`)) return;
setDeletingId(s.id);
try {
await scheduleApi.delete(s.id);
setSchedules(prev => prev.filter(x => x.id !== s.id));
setToast({ message: 'Schedule deleted', type: 'success' });
} catch {
setToast({ message: 'Failed to delete schedule', type: 'error' });
} finally {
setDeletingId(null);
}
};
const adjustNumber = (field: 'window_months' | 'adults', delta: number) => {
const limits: Record<string, [number, number]> = { window_months: [1, 12], adults: [1, 9] };
const [min, max] = limits[field];
setForm(prev => ({ ...prev, [field]: Math.min(max, Math.max(min, prev[field] + delta)) }));
};
const inputCls = (hasError?: boolean) =>
`w-full h-12 px-3 border rounded-xs bg-surface text-on-surface text-sm outline-none transition-colors ` +
(hasError
? 'border-error focus:border-error focus:ring-2 focus:ring-error/20'
: 'border-outline focus:border-primary focus:ring-2 focus:ring-primary/20');
// ── Render ────────────────────────────────────────────────────────────────
return (
<>
<div className="space-y-4 max-w-4xl">
{/* Header actions */}
<div className="flex items-center justify-between">
<p className="text-sm text-on-surface-variant">
{loading ? 'Loading…' : `${schedules.length} schedule${schedules.length !== 1 ? 's' : ''}`}
</p>
{!showForm && (
<Button variant="filled" onClick={() => setShowForm(true)}>
New Schedule
</Button>
)}
</div>
{/* ── Create Form ─────────────────────────────────────────────── */}
{showForm && (
<form onSubmit={handleCreate} className="space-y-4">
{/* Origin */}
<div className="bg-surface rounded-lg shadow-level-1 p-6">
<p className="text-xs font-semibold uppercase tracking-wider text-on-surface-variant mb-4">Origin</p>
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Origin Airport
</label>
<AirportSearch
value={form.origin}
onChange={(v) => {
setForm(prev => ({ ...prev, origin: v }));
if (errors.origin) setErrors(prev => ({ ...prev, origin: undefined }));
}}
placeholder="e.g. BDS, MUC, FRA"
hasError={!!errors.origin}
/>
{errors.origin
? <p className="mt-1 text-xs text-error">{errors.origin}</p>
: <p className="mt-1 text-xs text-on-surface-variant">3-letter IATA code</p>}
</div>
</div>
{/* Destination */}
<div className="bg-surface rounded-lg shadow-level-1 p-6">
<p className="text-xs font-semibold uppercase tracking-wider text-on-surface-variant mb-4">Destination</p>
<SegmentedButton
options={[
{ value: 'country', label: 'By Country', icon: Globe },
{ value: 'airports', label: 'By Airports', icon: PlaneTakeoff },
]}
value={destinationMode}
onChange={(v) => {
setDestinationMode(v as 'country' | 'airports');
setErrors(prev => ({ ...prev, country: undefined, airports: undefined }));
}}
className="mb-4"
/>
{destinationMode === 'country' ? (
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Destination Country
</label>
<input
type="text"
value={form.country}
onChange={(e) => {
setForm(prev => ({ ...prev, country: e.target.value.toUpperCase() }));
if (errors.country) setErrors(prev => ({ ...prev, country: undefined }));
}}
maxLength={2}
placeholder="e.g. DE, IT, ES"
className={inputCls(!!errors.country)}
/>
{errors.country
? <p className="mt-1 text-xs text-error">{errors.country}</p>
: <p className="mt-1 text-xs text-on-surface-variant">ISO 2-letter country code</p>}
</div>
) : (
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Destination Airports
</label>
<AirportSearch
value=""
onChange={(code) => {
if (code && code.length === 3 && !selectedAirports.includes(code)) {
setSelectedAirports(prev => [...prev, code]);
if (errors.airports) setErrors(prev => ({ ...prev, airports: undefined }));
}
}}
clearAfterSelect
placeholder="Search and add airports…"
hasError={!!errors.airports}
/>
{selectedAirports.length > 0 && (
<div className="flex flex-wrap gap-2 mt-3">
{selectedAirports.map(code => (
<AirportChip
key={code}
code={code}
onRemove={() => setSelectedAirports(prev => prev.filter(c => c !== code))}
/>
))}
</div>
)}
{errors.airports
? <p className="mt-1 text-xs text-error">{errors.airports}</p>
: <p className="mt-1 text-xs text-on-surface-variant">
{selectedAirports.length === 0
? 'Search and add destination airports'
: `${selectedAirports.length} airport${selectedAirports.length !== 1 ? 's' : ''} selected`}
</p>}
</div>
)}
</div>
{/* Parameters */}
<div className="bg-surface rounded-lg shadow-level-1 p-6">
<p className="text-xs font-semibold uppercase tracking-wider text-on-surface-variant mb-4">Parameters</p>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4">
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">Search Window</label>
<div className="flex items-center gap-2">
<button type="button" onClick={() => adjustNumber('window_months', -1)}
className="w-9 h-9 flex items-center justify-center rounded-full border border-outline text-on-surface-variant hover:bg-surface-2 transition-colors">
<Minus size={14} />
</button>
<div className="flex-1 h-12 flex items-center justify-center border border-outline rounded-xs bg-surface text-on-surface text-sm font-medium">
{form.window_months} {form.window_months === 1 ? 'month' : 'months'}
</div>
<button type="button" onClick={() => adjustNumber('window_months', 1)}
className="w-9 h-9 flex items-center justify-center rounded-full border border-outline text-on-surface-variant hover:bg-surface-2 transition-colors">
<Plus size={14} />
</button>
</div>
</div>
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">Seat Class</label>
<select value={form.seat_class}
onChange={(e) => setForm(prev => ({ ...prev, seat_class: e.target.value }))}
className={inputCls()}>
<option value="economy">Economy</option>
<option value="premium">Premium Economy</option>
<option value="business">Business</option>
<option value="first">First Class</option>
</select>
</div>
</div>
<div className="mt-4">
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">Passengers</label>
<div className="flex items-center gap-2">
<button type="button" onClick={() => adjustNumber('adults', -1)}
className="w-9 h-9 flex items-center justify-center rounded-full border border-outline text-on-surface-variant hover:bg-surface-2 transition-colors">
<Minus size={14} />
</button>
<div className="w-32 h-12 flex items-center justify-center border border-outline rounded-xs bg-surface text-on-surface text-sm font-medium">
{form.adults} {form.adults === 1 ? 'adult' : 'adults'}
</div>
<button type="button" onClick={() => adjustNumber('adults', 1)}
className="w-9 h-9 flex items-center justify-center rounded-full border border-outline text-on-surface-variant hover:bg-surface-2 transition-colors">
<Plus size={14} />
</button>
</div>
</div>
</div>
{/* Schedule */}
<div className="bg-surface rounded-lg shadow-level-1 p-6">
<p className="text-xs font-semibold uppercase tracking-wider text-on-surface-variant mb-4">Schedule</p>
{/* Optional label */}
<div className="mb-4">
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Label <span className="font-normal opacity-60">(optional)</span>
</label>
<input
type="text"
value={form.label}
onChange={(e) => setForm(prev => ({ ...prev, label: e.target.value }))}
placeholder="e.g. Weekly BDS → Germany"
className={inputCls()}
maxLength={100}
/>
</div>
{/* Frequency */}
<div className="mb-4">
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">Frequency</label>
<div className="flex gap-2">
{(['daily', 'weekly', 'monthly'] as const).map(f => (
<button
key={f}
type="button"
onClick={() => setForm(prev => ({ ...prev, frequency: f }))}
className={cn(
'flex-1 h-10 rounded-xs border text-sm font-medium transition-colors',
form.frequency === f
? 'bg-primary text-on-primary border-primary'
: 'border-outline text-on-surface hover:bg-surface-2',
)}
>
{f.charAt(0).toUpperCase() + f.slice(1)}
</button>
))}
</div>
</div>
{/* Day of week (weekly only) */}
{form.frequency === 'weekly' && (
<div className="mb-4">
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">Day of week</label>
<div className="flex gap-1.5 flex-wrap">
{DAYS.map((d, i) => (
<button
key={d}
type="button"
onClick={() => setForm(prev => ({ ...prev, day_of_week: i }))}
className={cn(
'w-12 h-10 rounded-xs border text-sm font-medium transition-colors',
form.day_of_week === i
? 'bg-primary text-on-primary border-primary'
: 'border-outline text-on-surface hover:bg-surface-2',
)}
>
{d}
</button>
))}
</div>
</div>
)}
{/* Day of month (monthly only) */}
{form.frequency === 'monthly' && (
<div className="mb-4">
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Day of month <span className="font-normal opacity-60">(128)</span>
</label>
<input
type="number"
value={form.day_of_month}
onChange={(e) => setForm(prev => ({ ...prev, day_of_month: Number(e.target.value) }))}
min={1} max={28}
className={cn(inputCls(), 'w-28')}
/>
</div>
)}
{/* Time */}
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Time <span className="font-normal opacity-60">(UTC)</span>
</label>
<div className="flex items-center gap-2">
<input
type="number"
value={form.hour}
onChange={(e) => setForm(prev => ({ ...prev, hour: Number(e.target.value) }))}
min={0} max={23}
className={cn(inputCls(!!errors.hour), 'w-20 text-center')}
placeholder="HH"
/>
<span className="text-on-surface-variant font-bold">:</span>
<input
type="number"
value={form.minute}
onChange={(e) => setForm(prev => ({ ...prev, minute: Number(e.target.value) }))}
min={0} max={59}
className={cn(inputCls(!!errors.minute), 'w-20 text-center')}
placeholder="MM"
/>
</div>
{(errors.hour || errors.minute) && (
<p className="mt-1 text-xs text-error">Hour: 023, Minute: 059</p>
)}
</div>
</div>
{/* Actions */}
<div className="flex justify-end gap-3 pb-4">
<Button variant="outlined" type="button"
onClick={() => { setShowForm(false); setForm(defaultForm()); setErrors({}); }}>
Cancel
</Button>
<Button variant="filled" type="submit" loading={saving}>
Create Schedule
</Button>
</div>
</form>
)}
{/* ── Schedules list ───────────────────────────────────────────── */}
{!loading && schedules.length === 0 && !showForm && (
<EmptyState
icon={CalendarClock}
title="No schedules yet"
description="Create a schedule to automatically run scans at a regular interval."
action={{ label: 'New Schedule', onClick: () => setShowForm(true) }}
/>
)}
{schedules.length > 0 && (
<div className="bg-surface rounded-lg shadow-level-1 overflow-hidden">
<table className="w-full text-sm">
<thead className="border-b border-outline">
<tr>
<th className="px-4 py-3 text-left text-xs font-semibold uppercase tracking-wider text-on-surface-variant">Route</th>
<th className="px-4 py-3 text-left text-xs font-semibold uppercase tracking-wider text-on-surface-variant">Cadence</th>
<th className="px-4 py-3 text-left text-xs font-semibold uppercase tracking-wider text-on-surface-variant hidden md:table-cell">Next Run</th>
<th className="px-4 py-3 text-left text-xs font-semibold uppercase tracking-wider text-on-surface-variant hidden lg:table-cell">Last Run</th>
<th className="px-4 py-3 text-center text-xs font-semibold uppercase tracking-wider text-on-surface-variant">Active</th>
<th className="px-4 py-3 text-right text-xs font-semibold uppercase tracking-wider text-on-surface-variant">Actions</th>
</tr>
</thead>
<tbody className="divide-y divide-outline">
{schedules.map(s => (
<tr key={s.id} className="hover:bg-surface-2 transition-colors">
<td className="px-4 py-3">
<div className="font-medium text-on-surface">
{s.label || `${s.origin}${s.country}`}
</div>
{s.label && (
<div className="text-xs text-on-surface-variant mt-0.5">
{s.origin} {s.country}
</div>
)}
</td>
<td className="px-4 py-3 text-on-surface-variant">
{describeSchedule(s)}
</td>
<td className="px-4 py-3 text-on-surface-variant hidden md:table-cell">
{formatNextRun(s.next_run_at)}
</td>
<td className="px-4 py-3 text-on-surface-variant hidden lg:table-cell">
{formatLastRun(s.last_run_at)}
</td>
<td className="px-4 py-3 text-center">
{/* Toggle switch */}
<button
type="button"
onClick={() => toggleEnabled(s)}
className={cn(
'relative inline-flex h-6 w-11 items-center rounded-full transition-colors focus:outline-none',
s.enabled ? 'bg-primary' : 'bg-outline',
)}
aria-label={s.enabled ? 'Disable schedule' : 'Enable schedule'}
>
<span className={cn(
'inline-block h-4 w-4 transform rounded-full bg-white transition-transform',
s.enabled ? 'translate-x-6' : 'translate-x-1',
)} />
</button>
</td>
<td className="px-4 py-3">
<div className="flex items-center justify-end gap-1">
<button
type="button"
onClick={() => handleRunNow(s)}
disabled={runningId === s.id}
className="p-2 rounded-xs text-on-surface-variant hover:bg-surface-2 hover:text-primary disabled:opacity-40 disabled:cursor-not-allowed transition-colors"
title="Run now"
>
<Play size={15} />
</button>
<button
type="button"
onClick={() => handleDelete(s)}
disabled={deletingId === s.id}
className="p-2 rounded-xs text-on-surface-variant hover:bg-surface-2 hover:text-error disabled:opacity-40 disabled:cursor-not-allowed transition-colors"
title="Delete schedule"
>
<Trash2 size={15} />
</button>
</div>
</td>
</tr>
))}
</tbody>
</table>
</div>
)}
</div>
{toast && (
<Toast message={toast.message} type={toast.type} onClose={() => setToast(null)} />
)}
</>
);
}

View File

@@ -1,20 +1,18 @@
import { defineConfig } from 'vite' import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react' import react from '@vitejs/plugin-react'
// When running inside Docker the backend is reachable via its service name.
// Outside Docker (plain `npm run dev`) it falls back to localhost.
const apiTarget = process.env.API_TARGET ?? 'http://localhost:8000'
// https://vite.dev/config/ // https://vite.dev/config/
export default defineConfig({ export default defineConfig({
plugins: [react()], plugins: [react()],
server: { server: {
port: 5173, port: 5173,
proxy: { proxy: {
'/api': { '/api': { target: apiTarget, changeOrigin: true },
target: 'http://localhost:8000', '/health': { target: apiTarget, changeOrigin: true },
changeOrigin: true,
},
'/health': {
target: 'http://localhost:8000',
changeOrigin: true,
}
} }
} }
}) })

View File

@@ -17,7 +17,7 @@ server {
# API proxy # API proxy
location /api/ { location /api/ {
proxy_pass http://localhost:8000; proxy_pass http://backend:8000;
proxy_http_version 1.1; proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade; proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade'; proxy_set_header Connection 'upgrade';
@@ -30,7 +30,7 @@ server {
# Health check endpoint proxy # Health check endpoint proxy
location /health { location /health {
proxy_pass http://localhost:8000; proxy_pass http://backend:8000;
proxy_http_version 1.1; proxy_http_version 1.1;
proxy_set_header Host $host; proxy_set_header Host $host;
} }

View File

@@ -1,4 +1,15 @@
# Web API
fastapi>=0.104.0
uvicorn[standard]>=0.24.0
pydantic>=2.0.0
# CLI tool
click>=8.0.0 click>=8.0.0
python-dateutil>=2.8.0 python-dateutil>=2.8.0
rich>=13.0.0 rich>=13.0.0
fast-flights>=3.0.0
# Shared utilities
requests>=2.31.0
# Flight search (GitHub only — not on PyPI as a stable release)
git+https://github.com/AWeirdDev/flights.git

View File

@@ -12,17 +12,112 @@ Runs as async background tasks within the FastAPI application.
import asyncio import asyncio
import logging import logging
from datetime import datetime, date, timedelta from datetime import datetime, date, timedelta
from typing import Dict, List, Optional
import json import json
from database import get_connection from database import get_connection
from airports import get_airports_for_country from airports import get_airports_for_country, lookup_airport
from searcher_v3 import search_multiple_routes from searcher_v3 import search_multiple_routes
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
def _write_route_incremental(scan_id: int, destination: str,
dest_name: str, dest_city: str,
new_flights: list):
"""
Write or update a route row and its individual flight rows immediately.
Called from progress_callback each time a (scan_id, destination, date)
query returns results. Merges into the existing route row if one already
exists, using a running weighted average for avg_price.
Opens its own DB connection — safe to call from the event loop thread.
"""
prices = [f.get('price') for f in new_flights if f.get('price')]
if not prices:
return
new_airlines = list({f.get('airline') for f in new_flights if f.get('airline')})
new_count = len(prices)
new_min = min(prices)
new_max = max(prices)
new_avg = sum(prices) / new_count
try:
conn = get_connection()
cursor = conn.cursor()
cursor.execute("""
SELECT id, flight_count, min_price, max_price, avg_price, airlines
FROM routes
WHERE scan_id = ? AND destination = ?
""", (scan_id, destination))
existing = cursor.fetchone()
if existing is None:
cursor.execute("""
INSERT INTO routes (
scan_id, destination, destination_name, destination_city,
flight_count, airlines, min_price, max_price, avg_price
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
scan_id, destination, dest_name, dest_city,
new_count, json.dumps(new_airlines),
new_min, new_max, new_avg,
))
else:
old_count = existing['flight_count'] or 0
old_min = existing['min_price']
old_max = existing['max_price']
old_avg = existing['avg_price'] or 0.0
old_airlines = json.loads(existing['airlines']) if existing['airlines'] else []
merged_count = old_count + new_count
merged_min = min(old_min, new_min) if old_min is not None else new_min
merged_max = max(old_max, new_max) if old_max is not None else new_max
merged_avg = (old_avg * old_count + new_avg * new_count) / merged_count
merged_airlines = json.dumps(list(set(old_airlines) | set(new_airlines)))
cursor.execute("""
UPDATE routes
SET flight_count = ?,
min_price = ?,
max_price = ?,
avg_price = ?,
airlines = ?
WHERE scan_id = ? AND destination = ?
""", (
merged_count, merged_min, merged_max, merged_avg, merged_airlines,
scan_id, destination,
))
for flight in new_flights:
if not flight.get('price'):
continue
cursor.execute("""
INSERT INTO flights (
scan_id, destination, date, airline,
departure_time, arrival_time, price, stops
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""", (
scan_id,
destination,
flight.get('date', ''),
flight.get('airline'),
flight.get('departure_time'),
flight.get('arrival_time'),
flight.get('price'),
flight.get('stops', 0),
))
conn.commit()
conn.close()
except Exception as e:
logger.error(f"[Scan {scan_id}] Failed to write incremental route {destination}: {e}")
async def process_scan(scan_id: int): async def process_scan(scan_id: int):
""" """
Process a pending scan by querying flights and saving routes. Process a pending scan by querying flights and saving routes.
@@ -85,7 +180,10 @@ async def process_scan(scan_id: int):
else: else:
# Specific airports mode: parse comma-separated list # Specific airports mode: parse comma-separated list
destination_codes = [code.strip() for code in country_or_airports.split(',')] destination_codes = [code.strip() for code in country_or_airports.split(',')]
destinations = [] # No pre-fetched airport details; fallback to IATA code as name destinations = [
lookup_airport(code) or {'iata': code, 'name': code, 'city': ''}
for code in destination_codes
]
logger.info(f"[Scan {scan_id}] Mode: Specific airports ({len(destination_codes)} destinations: {destination_codes})") logger.info(f"[Scan {scan_id}] Mode: Specific airports ({len(destination_codes)} destinations: {destination_codes})")
except Exception as e: except Exception as e:
@@ -131,19 +229,28 @@ async def process_scan(scan_id: int):
""", (len(routes_to_scan), scan_id)) """, (len(routes_to_scan), scan_id))
conn.commit() conn.commit()
# Progress callback to update database # Progress callback updates DB progress counter and writes routes live
# Signature: callback(origin, destination, date, status, count, error=None) # Signature: callback(origin, destination, date, status, count, error=None, flights=None)
routes_scanned_count = 0 routes_scanned_count = 0
def progress_callback(origin: str, destination: str, date: str, def progress_callback(origin: str, destination: str, date: str,
status: str, count: int, error: str = None): status: str, count: int, error: str = None,
flights: list = None):
nonlocal routes_scanned_count nonlocal routes_scanned_count
# Increment counter for each route query (cache hit or API call)
if status in ('cache_hit', 'api_success', 'error'): if status in ('cache_hit', 'api_success', 'error'):
routes_scanned_count += 1 routes_scanned_count += 1
# Update progress in database # Write route + flights to DB immediately if results available
if flights and status in ('cache_hit', 'api_success'):
for f in flights:
f['date'] = date
dest_info = next((d for d in destinations if d['iata'] == destination), None)
dest_name = dest_info.get('name', destination) if dest_info else destination
dest_city = dest_info.get('city', '') if dest_info else ''
_write_route_incremental(scan_id, destination, dest_name, dest_city, flights)
# Update progress counter
try: try:
progress_conn = get_connection() progress_conn = get_connection()
progress_cursor = progress_conn.cursor() progress_cursor = progress_conn.cursor()
@@ -158,7 +265,7 @@ async def process_scan(scan_id: int):
progress_conn.commit() progress_conn.commit()
progress_conn.close() progress_conn.close()
if routes_scanned_count % 10 == 0: # Log every 10 routes if routes_scanned_count % 10 == 0:
logger.info(f"[Scan {scan_id}] Progress: {routes_scanned_count}/{len(routes_to_scan)} routes ({status}: {origin}{destination})") logger.info(f"[Scan {scan_id}] Progress: {routes_scanned_count}/{len(routes_to_scan)} routes ({status}: {origin}{destination})")
except Exception as e: except Exception as e:
@@ -177,89 +284,15 @@ async def process_scan(scan_id: int):
progress_callback=progress_callback progress_callback=progress_callback
) )
logger.info(f"[Scan {scan_id}] Flight queries complete. Processing results...") logger.info(f"[Scan {scan_id}] Flight queries complete.")
# Group results by destination, preserving date per flight # Routes and flights were written incrementally by progress_callback.
# Structure: {dest: [(flight_dict, date), ...]} routes_saved = cursor.execute(
routes_by_destination: Dict[str, List] = {} "SELECT COUNT(*) FROM routes WHERE scan_id = ?", (scan_id,)
total_flights = 0 ).fetchone()[0]
total_flights_saved = cursor.execute(
for (orig, dest, scan_date), flights in results.items(): "SELECT COALESCE(SUM(flight_count), 0) FROM routes WHERE scan_id = ?", (scan_id,)
if dest not in routes_by_destination: ).fetchone()[0]
routes_by_destination[dest] = []
for flight in flights:
routes_by_destination[dest].append((flight, scan_date))
total_flights += len(flights)
logger.info(f"[Scan {scan_id}] Found {total_flights} total flights across {len(routes_by_destination)} destinations")
# Save routes and individual flights to database
routes_saved = 0
for destination, flight_date_pairs in routes_by_destination.items():
if not flight_date_pairs:
continue # Skip destinations with no flights
flights = [f for f, _ in flight_date_pairs]
# Get destination details (fall back to IATA code if not in DB)
dest_info = next((d for d in destinations if d['iata'] == destination), None)
dest_name = dest_info.get('name', destination) if dest_info else destination
dest_city = dest_info.get('city', '') if dest_info else ''
# Calculate statistics
prices = [f.get('price') for f in flights if f.get('price')]
airlines = list(set(f.get('airline') for f in flights if f.get('airline')))
if not prices:
logger.info(f"[Scan {scan_id}] Skipping {destination} - no prices available")
continue
min_price = min(prices)
max_price = max(prices)
avg_price = sum(prices) / len(prices)
# Insert route summary
cursor.execute("""
INSERT INTO routes (
scan_id, destination, destination_name, destination_city,
min_price, max_price, avg_price, flight_count, airlines
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
scan_id,
destination,
dest_name,
dest_city,
min_price,
max_price,
avg_price,
len(flights),
json.dumps(airlines)
))
# Insert individual flights
for flight, flight_date in flight_date_pairs:
if not flight.get('price'):
continue
cursor.execute("""
INSERT INTO flights (
scan_id, destination, date, airline,
departure_time, arrival_time, price, stops
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""", (
scan_id,
destination,
flight_date,
flight.get('airline'),
flight.get('departure_time'),
flight.get('arrival_time'),
flight.get('price'),
flight.get('stops', 0),
))
routes_saved += 1
conn.commit()
# Update scan to completed # Update scan to completed
cursor.execute(""" cursor.execute("""
@@ -268,10 +301,10 @@ async def process_scan(scan_id: int):
total_flights = ?, total_flights = ?,
updated_at = CURRENT_TIMESTAMP updated_at = CURRENT_TIMESTAMP
WHERE id = ? WHERE id = ?
""", (total_flights, scan_id)) """, (total_flights_saved, scan_id))
conn.commit() conn.commit()
logger.info(f"[Scan {scan_id}] ✅ Scan completed successfully! {routes_saved} routes saved with {total_flights} flights") logger.info(f"[Scan {scan_id}] ✅ Scan completed successfully! {routes_saved} routes saved with {total_flights_saved} flights")
except Exception as e: except Exception as e:
logger.error(f"[Scan {scan_id}] ❌ Scan failed with error: {str(e)}", exc_info=True) logger.error(f"[Scan {scan_id}] ❌ Scan failed with error: {str(e)}", exc_info=True)

View File

@@ -118,7 +118,7 @@ async def search_direct_flights(
) )
if cached is not None: if cached is not None:
if progress_callback: if progress_callback:
progress_callback(origin, destination, date, "cache_hit", len(cached)) progress_callback(origin, destination, date, "cache_hit", len(cached), flights=cached)
return cached return cached
# Add random delay to avoid rate limiting # Add random delay to avoid rate limiting
@@ -140,7 +140,7 @@ async def search_direct_flights(
# Report progress # Report progress
if progress_callback: if progress_callback:
progress_callback(origin, destination, date, "api_success", len(result)) progress_callback(origin, destination, date, "api_success", len(result), flights=result)
return result return result

View File

@@ -1,24 +0,0 @@
[supervisord]
nodaemon=true
logfile=/dev/null
logfile_maxbytes=0
pidfile=/tmp/supervisord.pid
[program:api]
command=python /app/api_server.py
directory=/app
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:nginx]
command=nginx -g "daemon off;"
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0