domverse/ciaovolo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement reverse scan (country → specific airports)
All checks were successful
Deploy / deploy (push) Successful in 30s
Details

Browse Source 
- DB schema: relaxed origin CHECK to >=2 chars, added scan_mode column to
  scans and scheduled_scans, added origin_airport to routes and flights,
  updated unique index to (scan_id, COALESCE(origin_airport,''), destination)
- Migrations: init_db.py recreates tables and adds columns via guarded ALTERs
- API: scan_mode field on ScanRequest/Scan; Route/Flight expose origin_airport;
  GET /scans/{id}/flights accepts origin_airport filter; CreateScheduleRequest
  and Schedule carry scan_mode; scheduler and run-now pass scan_mode through
- scan_processor: _write_route_incremental accepts origin_airport; process_scan
  branches on scan_mode=reverse (country → airports × destinations × dates)
- Frontend: new CountrySelect component (populated from GET /api/v1/countries);
  Scans page adds Direction toggle + CountrySelect for both modes; ScanDetails
  shows Origin column for reverse scans and uses composite route keys; Re-run
  preserves scan_mode

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
domverse
2026-03-01 17:58:55 +01:00
parent 7ece1f9b45
commit 77d2a46264
9 changed files with 1070 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files

321
PRD-reverse-scan.md Normal file
View File

@@ -0,0 +1,321 @@
# PRD: Reverse Scan — Country → Specific Airport(s)
**Status:** Ready for implementation
**Date:** 2026-03-01
**Target:** Web App (api_server.py + scan_processor.py + database + frontend)
---
## 1. Problem
All scans today are fixed-origin, variable-destination:
> *"From BDS — which German airports can I fly to directly?"*
The opposite question is equally common but unsupported:
> *"I want to fly to BRI or BDS — which German airport should I depart from?"*
This is the natural perspective of a traveller based in Germany looking for the cheapest access point to Puglia, not a traveller already in Puglia wondering where to go.
---
## 2. Goal
Let users create a scan where:
- **Origin** = all airports in a country (e.g. DE)
- **Destination** = one or more specific airports (e.g. BRI, BDS)
The results table for a reverse scan shows one row per **origin airport × destination** pair, so the user can directly compare which German airport is cheapest for each destination.
---
## 3. User Stories
- **As a user**, I want to search DE → BRI,BDS so I can see which German airport has the cheapest direct flight to Puglia.
- **As a user**, I want to schedule a reverse scan to run weekly.
- **As a user**, I want to mix both directions in my scan history.
- **As a user**, I want the destination country in forward scans to be a dropdown, not free text.
---
## 4. Scope
### In scope
- `scan_mode` field on scans and scheduled scans: `"forward"` (default) or `"reverse"`
- Reverse scan: origin = ISO country (dropdown), destinations = specific airport(s)
- Forward scan: destination country changed from text input to dropdown
- New `GET /api/v1/countries` endpoint (powers both dropdowns)
- `routes` table: add `origin_airport` column (nullable; populated for reverse scans)
- `flights` table: add `origin_airport` column (nullable; populated for reverse scans)
- ScanDetails routes table: show Origin column for reverse scans
- Scheduled scans: support `scan_mode`
### Out of scope
- Multi-country origin (e.g. DE + IT → BRI)
- Round-trip reverse scans
---
## 5. Data Model Changes
### 5.1 `scans` table
```sql
ALTER TABLE scans ADD COLUMN scan_mode TEXT NOT NULL DEFAULT 'forward'
CHECK (scan_mode IN ('forward', 'reverse'));
```
Column semantics by mode:
| Column | Forward (existing) | Reverse (new) |
|--------|--------------------|---------------|
| `origin` | Single IATA (e.g. `BDS`) | ISO country code (e.g. `DE`) |
| `country` | ISO destination country OR comma-separated destination IATAs | Comma-separated destination IATAs (e.g. `BRI,BDS`) |
| `scan_mode` | `forward` | `reverse` |
All existing rows default to `forward`. No data migration needed.
### 5.2 `routes` table
```sql
ALTER TABLE routes ADD COLUMN origin_airport TEXT;
```
- **Forward scans**: `origin_airport` = NULL (origin is always the scan-level `origin` IATA, already known)
- **Reverse scans**: `origin_airport` = the specific origin IATA for this route (e.g. `BER`, `FRA`)
The unique constraint on routes changes from `(scan_id, destination)` to `(scan_id, COALESCE(origin_airport, ''), destination)`.
**Reverse scan routes example** for `DE → BRI, BDS`:
| origin_airport | destination | flight_count | min_price |
|----------------|-------------|--------------|-----------|
| BER | BRI | 14 | €15 |
| FMM | BRI | 8 | €17 |
| DUS | BRI | 6 | €22 |
| BER | BDS | 3 | €28 |
| MUC | BDS | 2 | €34 |
### 5.3 `flights` table
```sql
ALTER TABLE flights ADD COLUMN origin_airport TEXT;
```
- **Forward scans**: `origin_airport` = NULL (origin is the scan-level `origin` IATA)
- **Reverse scans**: `origin_airport` = the specific IATA the flight departs from
### 5.4 `scheduled_scans` table
```sql
ALTER TABLE scheduled_scans ADD COLUMN scan_mode TEXT NOT NULL DEFAULT 'forward'
CHECK (scan_mode IN ('forward', 'reverse'));
```
Same column semantics as `scans`. When the scheduler fires a reverse scheduled scan, it creates a reverse scan child.
---
## 6. New API Endpoint
### `GET /api/v1/countries`
Returns the list of countries that have at least one airport in the database. Powers both the forward-mode destination country dropdown and the reverse-mode origin country dropdown.
**Response:**
```json
{
"data": [
{ "code": "DE", "name": "Germany", "airport_count": 23 },
{ "code": "IT", "name": "Italy", "airport_count": 41 },
...
]
}
```
Sorted alphabetically by name. Derived from `data/airports_by_country.json` at startup (same source as airport resolution). Rate limit: 100/min.
Country names are resolved from a static ISO → name mapping in `api_server.py` (same `COUNTRY_NAME_TO_ISO` dict already in `airports.py`, inverted).
---
## 7. Backend Changes
### 7.1 `ScanRequest` model
Add `scan_mode: Literal['forward', 'reverse'] = 'forward'`.
Root validator enforces:
**Forward mode** (unchanged):
- `origin`: required, 3-char IATA
- `country` OR `destinations`: required
**Reverse mode**:
- `origin`: required, 2-char ISO country code
- `destinations`: required, 1–10 IATAs
- `country` must be absent or null
### 7.2 `scan_processor.py` — `process_scan()`
**Forward** (unchanged):
```
fixed origin IATA × [airports resolved from destination country / destination IATAs]
```
**Reverse** (new):
```
[airports resolved from origin country] × fixed destination IATA(s)
```
Steps for reverse:
1. Read `scan.origin` as ISO country code → resolve airport list via `airports.py`
2. Read `scan.country` as comma-separated destination IATAs
3. Build pairs: `[(orig, dest) for orig in origin_airports for dest in destination_iatas]`
4. For each pair: call `search_multiple_routes(orig, dest)` as usual
5. Save route with `origin_airport=orig`, `destination=dest`
6. Save each flight with `origin_airport=orig`
`total_routes` = `len(origin_airports) × len(destination_iatas)`.
### 7.3 `GET /scans/{id}/routes` response
Add `origin_airport: str | null` to the `Route` response model. Frontend uses this to show/hide the Origin column.
### 7.4 `GET /scans/{id}/flights` response
Add `origin_airport: str | null` to the `Flight` response model.
### 7.5 Scheduled scans
`ScheduleRequest` and `Schedule` models gain `scan_mode`. The scheduler's `_fire_due_scans()` function passes `scan_mode` when inserting child scans into the `scans` table.
---
## 8. Frontend Changes
### 8.1 New `countriesApi` client (`api.ts`)
```typescript
export interface Country { code: string; name: string; airport_count: number; }
export const countriesApi = {
list: () => api.get<{ data: Country[] }>('/countries'),
};
```
### 8.2 New `CountrySelect` component
A standard `<select>` dropdown, populated from `GET /api/v1/countries` on mount. Shows `"Germany (DE)"` style labels, emits the ISO code. Reused in two places:
- Forward scan form: destination country field (replaces current text input)
- Reverse scan form: origin country field
### 8.3 Scan creation form (`/scans` page)
Add a **Direction toggle** at the top of the form (same segmented button style as "By Country / By Airports"):
```
[ → Forward ] [ ← Reverse ]
```
**Forward mode** (current layout, destination country now uses `CountrySelect`):
```
Origin airport: [AirportSearch]
Destination: [By Country ▼] [By Airports]
→ Country: [CountrySelect dropdown]
→ Airports: [comma-sep IATA input]
```
**Reverse mode** (new):
```
Origin country: [CountrySelect dropdown]
Destination airports: [AirportSearch / comma-sep IATA input]
```
### 8.4 ScanDetails — routes table
For reverse scans (`scan_mode === 'reverse'`), prepend an **Origin** column to the routes table:
| Origin | Destination | Flights | Airlines | Min | Avg | Max |
|--------|-------------|---------|----------|-----|-----|-----|
| BER Berlin | BRI Bari | 14 | Ryanair | €15 | €22 | €45 |
| FMM Memmingen | BRI Bari | 8 | Ryanair | €17 | €25 | €50 |
The existing click-to-expand flights sub-table still works — shows individual flight dates/times/prices for that specific origin→destination pair.
For forward scans: routes table unchanged (no Origin column).
### 8.5 ScanDetails — header
```
Forward: BDS → DE
Reverse: DE → BRI, BDS
```
### 8.6 `CreateScanRequest` type update (`api.ts`)
```typescript
export interface CreateScanRequest {
scan_mode?: 'forward' | 'reverse';
origin: string; // IATA (forward) or ISO country code (reverse)
country?: string; // forward only
destinations?: string[]; // forward (by airports) or reverse
window_months?: number;
seat_class?: 'economy' | 'premium' | 'business' | 'first';
adults?: number;
}
```
---
## 9. API Contract
### `POST /api/v1/scans`
**Forward — by country (existing, unchanged):**
```json
{ "origin": "BDS", "country": "DE", "window_months": 6, "seat_class": "economy", "adults": 1 }
```
**Forward — by airports (existing, unchanged):**
```json
{ "origin": "BDS", "destinations": ["FRA", "MUC"], "window_months": 6, "seat_class": "economy", "adults": 1 }
```
**Reverse (new):**
```json
{ "scan_mode": "reverse", "origin": "DE", "destinations": ["BRI", "BDS"], "window_months": 6, "seat_class": "economy", "adults": 1 }
```
---
## 10. Migration (`init_db.py`)
Four `ALTER TABLE … ADD COLUMN` statements, each guarded by `try/except OperationalError`:
```python
migrations = [
"ALTER TABLE scans ADD COLUMN scan_mode TEXT NOT NULL DEFAULT 'forward' CHECK (scan_mode IN ('forward', 'reverse'))",
"ALTER TABLE routes ADD COLUMN origin_airport TEXT",
"ALTER TABLE flights ADD COLUMN origin_airport TEXT",
"ALTER TABLE scheduled_scans ADD COLUMN scan_mode TEXT NOT NULL DEFAULT 'forward' CHECK (scan_mode IN ('forward', 'reverse'))",
]
for sql in migrations:
try:
conn.execute(sql)
except sqlite3.OperationalError:
pass # column already exists
conn.commit()
```
---
## 11. Acceptance Criteria
1. `GET /api/v1/countries` returns a sorted list of countries with airport counts.
2. Forward scan destination country field is a dropdown populated from that endpoint.
3. A reverse scan `DE → BRI, BDS` can be created via the form and the API.
4. The processor iterates all German airports × [BRI, BDS], storing `origin_airport` on each route and flight row.
5. ScanDetails for a reverse scan shows the Origin column in the routes table and `DE → BRI, BDS` in the header.
6. Scheduled scans accept `scan_mode`; the scheduler passes it through to child scans.
7. All existing forward scans continue to work — no regressions.
8. All four DB columns default correctly after migration with no data loss.

250
flight-comparator/api_server.py
View File

@@ -38,7 +38,10 @@ from threading import Lock
T = TypeVar('T') T = TypeVar('T')
# Import existing modules # Import existing modules
from airports import download_and_build_airport_data from airports import download_and_build_airport_data, COUNTRY_NAME_TO_ISO
# Inverted mapping: ISO code → country name (for /countries endpoint)
_ISO_TO_COUNTRY_NAME = {v: k for k, v in COUNTRY_NAME_TO_ISO.items()}
from database import get_connection from database import get_connection
from scan_processor import start_scan_processor, start_resume_processor, pause_scan_task, stop_scan_task from scan_processor import start_scan_processor, start_resume_processor, pause_scan_task, stop_scan_task
@@ -294,7 +297,7 @@ def _check_and_run_due_schedules():
now_str = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S') now_str = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')
cursor.execute(""" cursor.execute("""
SELECT id, origin, country, window_months, seat_class, adults, SELECT id, origin, country, scan_mode, window_months, seat_class, adults,
frequency, hour, minute, day_of_week, day_of_month frequency, hour, minute, day_of_week, day_of_month
FROM scheduled_scans FROM scheduled_scans
WHERE enabled = 1 AND next_run_at <= ? WHERE enabled = 1 AND next_run_at <= ?
@@ -302,7 +305,7 @@ def _check_and_run_due_schedules():
due = cursor.fetchall() due = cursor.fetchall()
for row in due: for row in due:
(sched_id, origin, country, window_months, seat_class, adults, (sched_id, origin, country, scan_mode, window_months, seat_class, adults,
frequency, hour, minute, day_of_week, day_of_month) = row frequency, hour, minute, day_of_week, day_of_month) = row
# Concurrency guard: skip if a scan for this schedule is still active # Concurrency guard: skip if a scan for this schedule is still active
@@ -323,10 +326,10 @@ def _check_and_run_due_schedules():
conn.execute(""" conn.execute("""
INSERT INTO scans ( INSERT INTO scans (
origin, country, start_date, end_date, origin, country, scan_mode, start_date, end_date,
status, seat_class, adults, scheduled_scan_id status, seat_class, adults, scheduled_scan_id
) VALUES (?, ?, ?, ?, 'pending', ?, ?, ?) ) VALUES (?, ?, ?, ?, ?, 'pending', ?, ?, ?)
""", (origin, country, start_date, end_date, """, (origin, country, scan_mode, start_date, end_date,
seat_class, adults, sched_id)) seat_class, adults, sched_id))
conn.commit() conn.commit()
scan_id = conn.execute("SELECT last_insert_rowid()").fetchone()[0] scan_id = conn.execute("SELECT last_insert_rowid()").fetchone()[0]
@@ -705,11 +708,15 @@ class Country(BaseModel):
class ScanRequest(BaseModel): class ScanRequest(BaseModel):
"""Flight scan request model with comprehensive validation.""" """Flight scan request model with comprehensive validation."""
scan_mode: str = Field(
'forward',
description="Scan direction: 'forward' (IATA → country) or 'reverse' (country → IATAs)"
)
origin: str = Field( origin: str = Field(
..., ...,
min_length=3, min_length=2,
max_length=3, max_length=3,
description="Origin airport IATA code (3 uppercase letters)" description="Origin airport IATA code (forward) or ISO country code (reverse)"
) )
destination_country: Optional[str] = Field( destination_country: Optional[str] = Field(
None, None,
@@ -747,9 +754,20 @@ class ScanRequest(BaseModel):
description="Number of adults (1-9)" description="Number of adults (1-9)"
) )
@validator('scan_mode')
def validate_scan_mode(cls, v):
if v not in ('forward', 'reverse'):
raise ValueError("scan_mode must be 'forward' or 'reverse'")
return v
@validator('origin') @validator('origin')
def validate_origin(cls, v): def validate_origin(cls, v, values):
v = v.upper() # Normalize to uppercase v = v.strip().upper()
mode = values.get('scan_mode', 'forward')
if mode == 'reverse':
if not re.match(r'^[A-Z]{2}$', v):
raise ValueError('For reverse scans, origin must be a 2-letter ISO country code (e.g., DE, IT)')
else:
if not re.match(r'^[A-Z]{3}$', v): if not re.match(r'^[A-Z]{3}$', v):
raise ValueError('Origin must be a 3-letter IATA code (e.g., BDS, MUC)') raise ValueError('Origin must be a 3-letter IATA code (e.g., BDS, MUC)')
return v return v
@@ -791,15 +809,19 @@ class ScanRequest(BaseModel):
@validator('destinations', pre=False, always=True) @validator('destinations', pre=False, always=True)
def check_destination_mode(cls, v, values): def check_destination_mode(cls, v, values):
"""Ensure either country or destinations is provided, but not both.""" """Ensure correct destination fields for the chosen scan_mode."""
country = values.get('destination_country') country = values.get('destination_country')
mode = values.get('scan_mode', 'forward')
if mode == 'reverse':
if not v:
raise ValueError('Reverse scans require destinations (list of destination airport IATA codes)')
return v
else:
if country and v: if country and v:
raise ValueError('Provide either country OR destinations, not both') raise ValueError('Provide either country OR destinations, not both')
if not country and not v: if not country and not v:
raise ValueError('Must provide either country or destinations') raise ValueError('Must provide either country or destinations')
return v return v
@validator('start_date') @validator('start_date')
@@ -895,6 +917,7 @@ class Route(BaseModel):
"""Route model - represents a discovered flight route.""" """Route model - represents a discovered flight route."""
id: int = Field(..., description="Route ID") id: int = Field(..., description="Route ID")
scan_id: int = Field(..., description="Parent scan ID") scan_id: int = Field(..., description="Parent scan ID")
origin_airport: Optional[str] = Field(None, description="Origin airport IATA code (reverse scans only)")
destination: str = Field(..., description="Destination airport IATA code") destination: str = Field(..., description="Destination airport IATA code")
destination_name: str = Field(..., description="Destination airport name") destination_name: str = Field(..., description="Destination airport name")
destination_city: Optional[str] = Field(None, description="Destination city") destination_city: Optional[str] = Field(None, description="Destination city")
@@ -910,6 +933,7 @@ class Flight(BaseModel):
"""Individual flight discovered by a scan.""" """Individual flight discovered by a scan."""
id: int = Field(..., description="Flight ID") id: int = Field(..., description="Flight ID")
scan_id: int = Field(..., description="Parent scan ID") scan_id: int = Field(..., description="Parent scan ID")
origin_airport: Optional[str] = Field(None, description="Origin airport IATA code (reverse scans only)")
destination: str = Field(..., description="Destination airport IATA code") destination: str = Field(..., description="Destination airport IATA code")
date: str = Field(..., description="Flight date (YYYY-MM-DD)") date: str = Field(..., description="Flight date (YYYY-MM-DD)")
airline: Optional[str] = Field(None, description="Operating airline") airline: Optional[str] = Field(None, description="Operating airline")
@@ -922,8 +946,9 @@ class Flight(BaseModel):
class Scan(BaseModel): class Scan(BaseModel):
"""Scan model - represents a flight scan with full details.""" """Scan model - represents a flight scan with full details."""
id: int = Field(..., description="Scan ID") id: int = Field(..., description="Scan ID")
origin: str = Field(..., description="Origin airport IATA code") scan_mode: str = Field('forward', description="Scan direction: forward or reverse")
country: str = Field(..., description="Destination country code") origin: str = Field(..., description="Origin airport IATA code (forward) or ISO country code (reverse)")
country: str = Field(..., description="Destination country code or comma-separated destination IATAs")
start_date: str = Field(..., description="Start date (YYYY-MM-DD)") start_date: str = Field(..., description="Start date (YYYY-MM-DD)")
end_date: str = Field(..., description="End date (YYYY-MM-DD)") end_date: str = Field(..., description="End date (YYYY-MM-DD)")
created_at: str = Field(..., description="ISO timestamp when scan was created") created_at: str = Field(..., description="ISO timestamp when scan was created")
@@ -1183,16 +1208,14 @@ async def get_countries():
country = airport['country'] country = airport['country']
country_counts[country] = country_counts.get(country, 0) + 1 country_counts[country] = country_counts.get(country, 0) + 1
# Get country names (we'll need a mapping file for this) countries = sorted([
# For now, just return codes
countries = [
Country( Country(
code=code, code=code,
name=code, # TODO: Add country name mapping name=_ISO_TO_COUNTRY_NAME.get(code, code),
airport_count=count airport_count=count
) )
for code, count in sorted(country_counts.items()) for code, count in country_counts.items()
] ], key=lambda c: c.name)
return countries return countries
@@ -1241,12 +1264,13 @@ async def create_scan(request: ScanRequest):
cursor.execute(""" cursor.execute("""
INSERT INTO scans ( INSERT INTO scans (
origin, country, start_date, end_date, origin, country, scan_mode, start_date, end_date,
status, seat_class, adults status, seat_class, adults
) VALUES (?, ?, ?, ?, ?, ?, ?) ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""", ( """, (
request.origin, request.origin,
country_or_airports, country_or_airports,
request.scan_mode,
start_date, start_date,
end_date, end_date,
'pending', 'pending',
@@ -1259,7 +1283,7 @@ async def create_scan(request: ScanRequest):
# Fetch the created scan # Fetch the created scan
cursor.execute(""" cursor.execute("""
SELECT id, origin, country, start_date, end_date, SELECT id, origin, country, scan_mode, start_date, end_date,
created_at, updated_at, status, total_routes, created_at, updated_at, status, total_routes,
routes_scanned, total_flights, error_message, routes_scanned, total_flights, error_message,
seat_class, adults, scheduled_scan_id, seat_class, adults, scheduled_scan_id,
@@ -1278,20 +1302,21 @@ async def create_scan(request: ScanRequest):
id=row[0], id=row[0],
origin=row[1], origin=row[1],
country=row[2], country=row[2],
start_date=row[3], scan_mode=row[3],
end_date=row[4], start_date=row[4],
created_at=row[5], end_date=row[5],
updated_at=row[6], created_at=row[6],
status=row[7], updated_at=row[7],
total_routes=row[8], status=row[8],
routes_scanned=row[9], total_routes=row[9],
total_flights=row[10], routes_scanned=row[10],
error_message=row[11], total_flights=row[11],
seat_class=row[12], error_message=row[12],
adults=row[13], seat_class=row[13],
scheduled_scan_id=row[14] if len(row) > 14 else None, adults=row[14],
started_at=row[15] if len(row) > 15 else None, scheduled_scan_id=row[15] if len(row) > 15 else None,
completed_at=row[16] if len(row) > 16 else None, started_at=row[16] if len(row) > 16 else None,
completed_at=row[17] if len(row) > 17 else None,
) )
logging.info(f"Scan created: ID={scan_id}, origin={scan.origin}, country={scan.country}, dates={scan.start_date} to {scan.end_date}") logging.info(f"Scan created: ID={scan_id}, origin={scan.origin}, country={scan.country}, dates={scan.start_date} to {scan.end_date}")
@@ -1367,7 +1392,7 @@ async def list_scans(
# Get paginated results # Get paginated results
offset = (page - 1) * limit offset = (page - 1) * limit
query = f""" query = f"""
SELECT id, origin, country, start_date, end_date, SELECT id, origin, country, scan_mode, start_date, end_date,
created_at, updated_at, status, total_routes, created_at, updated_at, status, total_routes,
routes_scanned, total_flights, error_message, routes_scanned, total_flights, error_message,
seat_class, adults, scheduled_scan_id, seat_class, adults, scheduled_scan_id,
@@ -1388,20 +1413,21 @@ async def list_scans(
id=row[0], id=row[0],
origin=row[1], origin=row[1],
country=row[2], country=row[2],
start_date=row[3], scan_mode=row[3],
end_date=row[4], start_date=row[4],
created_at=row[5], end_date=row[5],
updated_at=row[6], created_at=row[6],
status=row[7], updated_at=row[7],
total_routes=row[8], status=row[8],
routes_scanned=row[9], total_routes=row[9],
total_flights=row[10], routes_scanned=row[10],
error_message=row[11], total_flights=row[11],
seat_class=row[12], error_message=row[12],
adults=row[13], seat_class=row[13],
scheduled_scan_id=row[14] if len(row) > 14 else None, adults=row[14],
started_at=row[15] if len(row) > 15 else None, scheduled_scan_id=row[15] if len(row) > 15 else None,
completed_at=row[16] if len(row) > 16 else None, started_at=row[16] if len(row) > 16 else None,
completed_at=row[17] if len(row) > 17 else None,
)) ))
# Build pagination metadata # Build pagination metadata
@@ -1439,7 +1465,7 @@ async def get_scan_status(scan_id: int):
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute(""" cursor.execute("""
SELECT id, origin, country, start_date, end_date, SELECT id, origin, country, scan_mode, start_date, end_date,
created_at, updated_at, status, total_routes, created_at, updated_at, status, total_routes,
routes_scanned, total_flights, error_message, routes_scanned, total_flights, error_message,
seat_class, adults, scheduled_scan_id, seat_class, adults, scheduled_scan_id,
@@ -1461,20 +1487,21 @@ async def get_scan_status(scan_id: int):
id=row[0], id=row[0],
origin=row[1], origin=row[1],
country=row[2], country=row[2],
start_date=row[3], scan_mode=row[3],
end_date=row[4], start_date=row[4],
created_at=row[5], end_date=row[5],
updated_at=row[6], created_at=row[6],
status=row[7], updated_at=row[7],
total_routes=row[8], status=row[8],
routes_scanned=row[9], total_routes=row[9],
total_flights=row[10], routes_scanned=row[10],
error_message=row[11], total_flights=row[11],
seat_class=row[12], error_message=row[12],
adults=row[13], seat_class=row[13],
scheduled_scan_id=row[14] if len(row) > 14 else None, adults=row[14],
started_at=row[15] if len(row) > 15 else None, scheduled_scan_id=row[15] if len(row) > 15 else None,
completed_at=row[16] if len(row) > 16 else None, started_at=row[16] if len(row) > 16 else None,
completed_at=row[17] if len(row) > 17 else None,
) )
except HTTPException: except HTTPException:
@@ -1716,7 +1743,7 @@ async def get_scan_routes(
# Get paginated results # Get paginated results
offset = (page - 1) * limit offset = (page - 1) * limit
cursor.execute(""" cursor.execute("""
SELECT id, scan_id, destination, destination_name, destination_city, SELECT id, scan_id, origin_airport, destination, destination_name, destination_city,
flight_count, airlines, min_price, max_price, avg_price, created_at flight_count, airlines, min_price, max_price, avg_price, created_at
FROM routes FROM routes
WHERE scan_id = ? WHERE scan_id = ?
@@ -1736,13 +1763,13 @@ async def get_scan_routes(
for row in rows: for row in rows:
# Parse airlines JSON # Parse airlines JSON
try: try:
airlines = json.loads(row[6]) if row[6] else [] airlines = json.loads(row[7]) if row[7] else []
except: except:
airlines = [] airlines = []
dest = row[2] dest = row[3]
dest_name = row[3] or dest dest_name = row[4] or dest
dest_city = row[4] or '' dest_city = row[5] or ''
# If name was never resolved (stored as IATA code), look it up now # If name was never resolved (stored as IATA code), look it up now
if dest_name == dest: if dest_name == dest:
@@ -1753,15 +1780,16 @@ async def get_scan_routes(
routes.append(Route( routes.append(Route(
id=row[0], id=row[0],
scan_id=row[1], scan_id=row[1],
origin_airport=row[2],
destination=dest, destination=dest,
destination_name=dest_name, destination_name=dest_name,
destination_city=dest_city, destination_city=dest_city,
flight_count=row[5], flight_count=row[6],
airlines=airlines, airlines=airlines,
min_price=row[7], min_price=row[8],
max_price=row[8], max_price=row[9],
avg_price=row[9], avg_price=row[10],
created_at=row[10] created_at=row[11]
)) ))
# Build pagination metadata # Build pagination metadata
@@ -1791,14 +1819,15 @@ async def get_scan_routes(
async def get_scan_flights( async def get_scan_flights(
scan_id: int, scan_id: int,
destination: Optional[str] = Query(None, min_length=3, max_length=3, description="Filter by destination IATA code"), destination: Optional[str] = Query(None, min_length=3, max_length=3, description="Filter by destination IATA code"),
origin_airport: Optional[str] = Query(None, min_length=3, max_length=3, description="Filter by origin airport IATA code (reverse scans)"),
page: int = Query(1, ge=1, description="Page number"), page: int = Query(1, ge=1, description="Page number"),
limit: int = Query(50, ge=1, le=200, description="Items per page") limit: int = Query(50, ge=1, le=200, description="Items per page")
): ):
""" """
Get individual flights discovered by a specific scan. Get individual flights discovered by a specific scan.
Optionally filter by destination airport code. Optionally filter by destination and/or origin airport code.
Results are ordered by price ascending. Results are ordered by date then price ascending.
""" """
try: try:
conn = get_connection() conn = get_connection()
@@ -1809,45 +1838,41 @@ async def get_scan_flights(
conn.close() conn.close()
raise HTTPException(status_code=404, detail=f"Scan not found: {scan_id}") raise HTTPException(status_code=404, detail=f"Scan not found: {scan_id}")
# Build dynamic WHERE clause
conditions = ["scan_id = ?"]
params: list = [scan_id]
if destination: if destination:
cursor.execute( conditions.append("destination = ?")
"SELECT COUNT(*) FROM flights WHERE scan_id = ? AND destination = ?", params.append(destination.upper())
(scan_id, destination.upper()) if origin_airport:
) conditions.append("origin_airport = ?")
else: params.append(origin_airport.upper())
cursor.execute("SELECT COUNT(*) FROM flights WHERE scan_id = ?", (scan_id,)) where = " AND ".join(conditions)
cursor.execute(f"SELECT COUNT(*) FROM flights WHERE {where}", params)
total = cursor.fetchone()[0] total = cursor.fetchone()[0]
total_pages = math.ceil(total / limit) if total > 0 else 0 total_pages = math.ceil(total / limit) if total > 0 else 0
offset = (page - 1) * limit offset = (page - 1) * limit
if destination: cursor.execute(f"""
cursor.execute(""" SELECT id, scan_id, origin_airport, destination, date, airline,
SELECT id, scan_id, destination, date, airline,
departure_time, arrival_time, price, stops departure_time, arrival_time, price, stops
FROM flights FROM flights
WHERE scan_id = ? AND destination = ? WHERE {where}
ORDER BY date ASC, price ASC ORDER BY date ASC, price ASC
LIMIT ? OFFSET ? LIMIT ? OFFSET ?
""", (scan_id, destination.upper(), limit, offset)) """, params + [limit, offset])
else:
cursor.execute("""
SELECT id, scan_id, destination, date, airline,
departure_time, arrival_time, price, stops
FROM flights
WHERE scan_id = ?
ORDER BY date ASC, price ASC
LIMIT ? OFFSET ?
""", (scan_id, limit, offset))
rows = cursor.fetchall() rows = cursor.fetchall()
conn.close() conn.close()
flights = [ flights = [
Flight( Flight(
id=row[0], scan_id=row[1], destination=row[2], date=row[3], id=row[0], scan_id=row[1], origin_airport=row[2],
airline=row[4], departure_time=row[5], arrival_time=row[6], destination=row[3], date=row[4], airline=row[5],
price=row[7], stops=row[8] departure_time=row[6], arrival_time=row[7],
price=row[8], stops=row[9]
) )
for row in rows for row in rows
] ]
@@ -1965,7 +1990,8 @@ async def get_flights_stub(route_id: str):
class CreateScheduleRequest(BaseModel): class CreateScheduleRequest(BaseModel):
"""Request body for creating or updating a scheduled scan.""" """Request body for creating or updating a scheduled scan."""
origin: str = Field(..., description="Origin airport IATA code (3 letters)") scan_mode: str = Field('forward', description="Scan direction: 'forward' or 'reverse'")
origin: str = Field(..., description="Origin airport IATA code (forward) or ISO country code (reverse)")
country: str = Field(..., description="Destination country ISO code (2 letters) or comma-separated IATA codes") country: str = Field(..., description="Destination country ISO code (2 letters) or comma-separated IATA codes")
window_months: int = Field(1, ge=1, le=12, description="Months of data per scan run") window_months: int = Field(1, ge=1, le=12, description="Months of data per scan run")
seat_class: str = Field('economy', description="Seat class") seat_class: str = Field('economy', description="Seat class")
@@ -2027,6 +2053,7 @@ class UpdateScheduleRequest(BaseModel):
class Schedule(BaseModel): class Schedule(BaseModel):
"""A recurring scheduled scan.""" """A recurring scheduled scan."""
id: int id: int
scan_mode: str
origin: str origin: str
country: str country: str
window_months: int window_months: int
@@ -2049,6 +2076,7 @@ def _row_to_schedule(row, recent_scan_ids: list) -> Schedule:
"""Convert a DB row (sqlite3.Row or tuple) to a Schedule model.""" """Convert a DB row (sqlite3.Row or tuple) to a Schedule model."""
return Schedule( return Schedule(
id=row['id'], id=row['id'],
scan_mode=row['scan_mode'] if 'scan_mode' in row.keys() else 'forward',
origin=row['origin'], origin=row['origin'],
country=row['country'], country=row['country'],
window_months=row['window_months'], window_months=row['window_months'],
@@ -2126,12 +2154,12 @@ async def create_schedule(request: CreateScheduleRequest):
conn = get_connection() conn = get_connection()
conn.execute(""" conn.execute("""
INSERT INTO scheduled_scans ( INSERT INTO scheduled_scans (
origin, country, window_months, seat_class, adults, scan_mode, origin, country, window_months, seat_class, adults,
label, frequency, hour, minute, day_of_week, day_of_month, label, frequency, hour, minute, day_of_week, day_of_month,
enabled, next_run_at enabled, next_run_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?) ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)
""", ( """, (
request.origin, request.country, request.window_months, request.scan_mode, request.origin, request.country, request.window_months,
request.seat_class, request.adults, request.label, request.seat_class, request.adults, request.label,
request.frequency, request.hour, request.minute, request.frequency, request.hour, request.minute,
request.day_of_week, request.day_of_month, next_run_str, request.day_of_week, request.day_of_month, next_run_str,
@@ -2279,11 +2307,13 @@ async def run_schedule_now(schedule_id: int):
conn.execute(""" conn.execute("""
INSERT INTO scans ( INSERT INTO scans (
origin, country, start_date, end_date, origin, country, scan_mode, start_date, end_date,
status, seat_class, adults, scheduled_scan_id status, seat_class, adults, scheduled_scan_id
) VALUES (?, ?, ?, ?, 'pending', ?, ?, ?) ) VALUES (?, ?, ?, ?, ?, 'pending', ?, ?, ?)
""", ( """, (
row['origin'], row['country'], start_date, end_date, row['origin'], row['country'],
row['scan_mode'] if 'scan_mode' in row.keys() else 'forward',
start_date, end_date,
row['seat_class'], row['adults'], schedule_id, row['seat_class'], row['adults'], schedule_id,
)) ))
conn.commit() conn.commit()

166
flight-comparator/database/init_db.py
View File

@@ -301,6 +301,171 @@ def _migrate_add_pause_cancel_status(conn, verbose=True):
print(" ✅ Migration complete: status now accepts 'paused' and 'cancelled'") print(" ✅ Migration complete: status now accepts 'paused' and 'cancelled'")
def _migrate_add_reverse_scan_support(conn, verbose=True):
"""
Migration: Add reverse scan support across all affected tables.
Changes:
- scans: relax origin CHECK (3→>=2), add scan_mode column
- routes: add origin_airport column, replace unique index
- flights: add origin_airport column
- scheduled_scans: relax origin CHECK (3→>=2), add scan_mode column
"""
# ── scans table ──────────────────────────────────────────────────────────
cursor = conn.execute("PRAGMA table_info(scans)")
scans_cols = [row[1] for row in cursor.fetchall()]
if scans_cols and 'scan_mode' not in scans_cols:
if verbose:
print(" 🔄 Migrating scans table: relaxing origin constraint, adding scan_mode…")
conn.execute("PRAGMA foreign_keys = OFF")
conn.execute("DROP TRIGGER IF EXISTS update_scans_timestamp")
conn.execute("DROP TRIGGER IF EXISTS update_scan_flight_count_insert")
conn.execute("DROP TRIGGER IF EXISTS update_scan_flight_count_update")
conn.execute("DROP TRIGGER IF EXISTS update_scan_flight_count_delete")
conn.execute("""
CREATE TABLE scans_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
origin TEXT NOT NULL CHECK(length(origin) >= 2),
country TEXT NOT NULL CHECK(length(country) >= 2),
scan_mode TEXT NOT NULL DEFAULT 'forward'
CHECK(scan_mode IN ('forward', 'reverse')),
start_date TEXT NOT NULL,
end_date TEXT NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
started_at TIMESTAMP,
completed_at TIMESTAMP,
status TEXT NOT NULL DEFAULT 'pending'
CHECK(status IN ('pending', 'running', 'completed', 'failed', 'cancelled', 'paused')),
total_routes INTEGER NOT NULL DEFAULT 0 CHECK(total_routes >= 0),
routes_scanned INTEGER NOT NULL DEFAULT 0 CHECK(routes_scanned >= 0),
total_flights INTEGER NOT NULL DEFAULT 0 CHECK(total_flights >= 0),
error_message TEXT,
seat_class TEXT DEFAULT 'economy',
adults INTEGER DEFAULT 1 CHECK(adults > 0 AND adults <= 9),
scheduled_scan_id INTEGER,
CHECK(end_date >= start_date),
CHECK(routes_scanned <= total_routes OR total_routes = 0)
)
""")
conn.execute("""
INSERT INTO scans_new (
id, origin, country, scan_mode, start_date, end_date,
created_at, updated_at, started_at, completed_at,
status, total_routes, routes_scanned, total_flights,
error_message, seat_class, adults, scheduled_scan_id
)
SELECT
id, origin, country, 'forward', start_date, end_date,
created_at, updated_at, started_at, completed_at,
status, total_routes, routes_scanned, total_flights,
error_message, seat_class, adults, scheduled_scan_id
FROM scans
""")
conn.execute("DROP TABLE scans")
conn.execute("ALTER TABLE scans_new RENAME TO scans")
conn.execute("PRAGMA foreign_keys = ON")
conn.commit()
if verbose:
print(" ✅ scans table migrated")
# ── routes: add origin_airport column ────────────────────────────────────
cursor = conn.execute("PRAGMA table_info(routes)")
routes_cols = [row[1] for row in cursor.fetchall()]
if routes_cols and 'origin_airport' not in routes_cols:
if verbose:
print(" 🔄 Migrating routes table: adding origin_airport column…")
conn.execute("ALTER TABLE routes ADD COLUMN origin_airport TEXT")
conn.commit()
if verbose:
print(" ✅ routes.origin_airport column added")
# ── routes: replace unique index ─────────────────────────────────────────
cursor = conn.execute(
"SELECT name FROM sqlite_master WHERE type='index' AND name='uq_routes_scan_dest'"
)
if cursor.fetchone():
if verbose:
print(" 🔄 Replacing routes unique index…")
conn.execute("DROP INDEX IF EXISTS uq_routes_scan_dest")
conn.execute("""
CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_origin_dest
ON routes(scan_id, COALESCE(origin_airport, ''), destination)
""")
conn.commit()
if verbose:
print(" ✅ Routes unique index replaced")
# ── flights: add origin_airport column ───────────────────────────────────
cursor = conn.execute("PRAGMA table_info(flights)")
flights_cols = [row[1] for row in cursor.fetchall()]
if flights_cols and 'origin_airport' not in flights_cols:
if verbose:
print(" 🔄 Migrating flights table: adding origin_airport column…")
conn.execute("ALTER TABLE flights ADD COLUMN origin_airport TEXT")
conn.commit()
if verbose:
print(" ✅ flights.origin_airport column added")
# ── scheduled_scans: relax origin + add scan_mode ────────────────────────
cursor = conn.execute("PRAGMA table_info(scheduled_scans)")
sched_cols = [row[1] for row in cursor.fetchall()]
if sched_cols and 'scan_mode' not in sched_cols:
if verbose:
print(" 🔄 Migrating scheduled_scans table: relaxing origin constraint, adding scan_mode…")
conn.execute("DROP TRIGGER IF EXISTS update_scheduled_scans_timestamp")
conn.execute("""
CREATE TABLE scheduled_scans_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
origin TEXT NOT NULL CHECK(length(origin) >= 2),
country TEXT NOT NULL CHECK(length(country) >= 2),
scan_mode TEXT NOT NULL DEFAULT 'forward'
CHECK(scan_mode IN ('forward', 'reverse')),
window_months INTEGER NOT NULL DEFAULT 1
CHECK(window_months >= 1 AND window_months <= 12),
seat_class TEXT NOT NULL DEFAULT 'economy',
adults INTEGER NOT NULL DEFAULT 1
CHECK(adults > 0 AND adults <= 9),
frequency TEXT NOT NULL
CHECK(frequency IN ('daily', 'weekly', 'monthly')),
hour INTEGER NOT NULL DEFAULT 6
CHECK(hour >= 0 AND hour <= 23),
minute INTEGER NOT NULL DEFAULT 0
CHECK(minute >= 0 AND minute <= 59),
day_of_week INTEGER CHECK(day_of_week >= 0 AND day_of_week <= 6),
day_of_month INTEGER CHECK(day_of_month >= 1 AND day_of_month <= 28),
enabled INTEGER NOT NULL DEFAULT 1,
label TEXT,
last_run_at TIMESTAMP,
next_run_at TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
CHECK(
(frequency = 'weekly' AND day_of_week IS NOT NULL) OR
(frequency = 'monthly' AND day_of_month IS NOT NULL) OR
(frequency = 'daily')
)
)
""")
conn.execute("""
INSERT INTO scheduled_scans_new (
id, origin, country, scan_mode, window_months, seat_class, adults,
frequency, hour, minute, day_of_week, day_of_month,
enabled, label, last_run_at, next_run_at, created_at, updated_at
)
SELECT
id, origin, country, 'forward', window_months, seat_class, adults,
frequency, hour, minute, day_of_week, day_of_month,
enabled, label, last_run_at, next_run_at, created_at, updated_at
FROM scheduled_scans
""")
conn.execute("DROP TABLE scheduled_scans")
conn.execute("ALTER TABLE scheduled_scans_new RENAME TO scheduled_scans")
conn.commit()
if verbose:
print(" ✅ scheduled_scans table migrated")
def initialize_database(db_path=None, verbose=True): def initialize_database(db_path=None, verbose=True):
""" """
Initialize or migrate the database. Initialize or migrate the database.
@@ -349,6 +514,7 @@ def initialize_database(db_path=None, verbose=True):
_migrate_add_scheduled_scan_id_to_scans(conn, verbose) _migrate_add_scheduled_scan_id_to_scans(conn, verbose)
_migrate_add_timing_columns_to_scans(conn, verbose) _migrate_add_timing_columns_to_scans(conn, verbose)
_migrate_add_pause_cancel_status(conn, verbose) _migrate_add_pause_cancel_status(conn, verbose)
_migrate_add_reverse_scan_support(conn, verbose)
# Load and execute schema # Load and execute schema
schema_sql = load_schema() schema_sql = load_schema()

22
flight-comparator/database/schema.sql
View File

@@ -20,8 +20,11 @@ CREATE TABLE IF NOT EXISTS scans (
id INTEGER PRIMARY KEY AUTOINCREMENT, id INTEGER PRIMARY KEY AUTOINCREMENT,
-- Search parameters (validated by CHECK constraints) -- Search parameters (validated by CHECK constraints)
origin TEXT NOT NULL CHECK(length(origin) = 3), -- origin stores IATA code (forward scans) or ISO country code (reverse scans)
origin TEXT NOT NULL CHECK(length(origin) >= 2),
country TEXT NOT NULL CHECK(length(country) >= 2), country TEXT NOT NULL CHECK(length(country) >= 2),
scan_mode TEXT NOT NULL DEFAULT 'forward'
CHECK(scan_mode IN ('forward', 'reverse')),
start_date TEXT NOT NULL, -- ISO 8601: YYYY-MM-DD start_date TEXT NOT NULL, -- ISO 8601: YYYY-MM-DD
end_date TEXT NOT NULL, end_date TEXT NOT NULL,
@@ -81,7 +84,10 @@ CREATE TABLE IF NOT EXISTS routes (
-- Foreign key to scans (cascade delete) -- Foreign key to scans (cascade delete)
scan_id INTEGER NOT NULL, scan_id INTEGER NOT NULL,
-- Destination airport -- Route airports
-- For forward scans: origin_airport is NULL (implicit from scan.origin)
-- For reverse scans: origin_airport is the variable origin IATA
origin_airport TEXT,
destination TEXT NOT NULL CHECK(length(destination) = 3), destination TEXT NOT NULL CHECK(length(destination) = 3),
destination_name TEXT NOT NULL, destination_name TEXT NOT NULL,
destination_city TEXT, destination_city TEXT,
@@ -120,9 +126,9 @@ CREATE INDEX IF NOT EXISTS idx_routes_min_price
ON routes(min_price) ON routes(min_price)
WHERE min_price IS NOT NULL; -- Partial index for routes with prices WHERE min_price IS NOT NULL; -- Partial index for routes with prices
-- One route row per (scan, destination) — enables incremental upsert writes -- One route row per (scan, origin_airport, destination) — supports both forward and reverse scans
CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_dest CREATE UNIQUE INDEX IF NOT EXISTS uq_routes_scan_origin_dest
ON routes(scan_id, destination); ON routes(scan_id, COALESCE(origin_airport, ''), destination);
-- ============================================================================ -- ============================================================================
-- Triggers: Auto-update timestamps and aggregates -- Triggers: Auto-update timestamps and aggregates
@@ -191,6 +197,8 @@ CREATE TABLE IF NOT EXISTS flights (
scan_id INTEGER NOT NULL, scan_id INTEGER NOT NULL,
-- Route -- Route
-- origin_airport: NULL for forward scans, specific IATA for reverse scans
origin_airport TEXT,
destination TEXT NOT NULL CHECK(length(destination) = 3), destination TEXT NOT NULL CHECK(length(destination) = 3),
date TEXT NOT NULL, -- ISO 8601: YYYY-MM-DD date TEXT NOT NULL, -- ISO 8601: YYYY-MM-DD
@@ -273,8 +281,10 @@ CREATE TABLE IF NOT EXISTS scheduled_scans (
id INTEGER PRIMARY KEY AUTOINCREMENT, id INTEGER PRIMARY KEY AUTOINCREMENT,
-- Scan parameters (same as scans table) -- Scan parameters (same as scans table)
origin TEXT NOT NULL CHECK(length(origin) = 3), origin TEXT NOT NULL CHECK(length(origin) >= 2),
country TEXT NOT NULL CHECK(length(country) >= 2), country TEXT NOT NULL CHECK(length(country) >= 2),
scan_mode TEXT NOT NULL DEFAULT 'forward'
CHECK(scan_mode IN ('forward', 'reverse')),
window_months INTEGER NOT NULL DEFAULT 1 window_months INTEGER NOT NULL DEFAULT 1
CHECK(window_months >= 1 AND window_months <= 12), CHECK(window_months >= 1 AND window_months <= 12),
seat_class TEXT NOT NULL DEFAULT 'economy', seat_class TEXT NOT NULL DEFAULT 'economy',

19
flight-comparator/frontend/src/api.ts
View File

@@ -10,6 +10,7 @@ const api = axios.create({
// Types // Types
export interface Scan { export interface Scan {
id: number; id: number;
scan_mode: 'forward' | 'reverse';
origin: string; origin: string;
country: string; country: string;
start_date: string; start_date: string;
@@ -30,6 +31,7 @@ export interface Scan {
export interface Schedule { export interface Schedule {
id: number; id: number;
scan_mode: 'forward' | 'reverse';
origin: string; origin: string;
country: string; country: string;
window_months: number; window_months: number;
@@ -49,6 +51,7 @@ export interface Schedule {
} }
export interface CreateScheduleRequest { export interface CreateScheduleRequest {
scan_mode?: 'forward' | 'reverse';
origin: string; origin: string;
country: string; country: string;
window_months?: number; window_months?: number;
@@ -65,6 +68,7 @@ export interface CreateScheduleRequest {
export interface Route { export interface Route {
id: number; id: number;
scan_id: number; scan_id: number;
origin_airport?: string;
destination: string; destination: string;
destination_name: string; destination_name: string;
destination_city?: string; destination_city?: string;
@@ -79,6 +83,7 @@ export interface Route {
export interface Flight { export interface Flight {
id: number; id: number;
scan_id: number; scan_id: number;
origin_airport?: string;
destination: string; destination: string;
date: string; date: string;
airline?: string; airline?: string;
@@ -116,7 +121,14 @@ export interface PaginatedResponse<T> {
}; };
} }
export interface Country {
code: string;
name: string;
airport_count: number;
}
export interface CreateScanRequest { export interface CreateScanRequest {
scan_mode?: 'forward' | 'reverse';
origin: string; origin: string;
country?: string; // Optional: provide either country or destinations country?: string; // Optional: provide either country or destinations
destinations?: string[]; // Optional: provide either country or destinations destinations?: string[]; // Optional: provide either country or destinations
@@ -155,9 +167,10 @@ export const scanApi = {
}); });
}, },
getFlights: (id: number, destination?: string, page = 1, limit = 50) => { getFlights: (id: number, destination?: string, originAirport?: string, page = 1, limit = 50) => {
const params: Record<string, unknown> = { page, limit }; const params: Record<string, unknown> = { page, limit };
if (destination) params.destination = destination; if (destination) params.destination = destination;
if (originAirport) params.origin_airport = originAirport;
return api.get<PaginatedResponse<Flight>>(`/scans/${id}/flights`, { params }); return api.get<PaginatedResponse<Flight>>(`/scans/${id}/flights`, { params });
}, },
@@ -176,6 +189,10 @@ export const airportApi = {
}, },
}; };
export const countriesApi = {
list: () => api.get<Country[]>('/countries'),
};
export const scheduleApi = { export const scheduleApi = {
list: (page = 1, limit = 20) => list: (page = 1, limit = 20) =>
api.get<PaginatedResponse<Schedule>>('/schedules', { params: { page, limit } }), api.get<PaginatedResponse<Schedule>>('/schedules', { params: { page, limit } }),

51
flight-comparator/frontend/src/components/CountrySelect.tsx Normal file
View File

@@ -0,0 +1,51 @@
import { useEffect, useState } from 'react';
import { countriesApi } from '../api';
import type { Country } from '../api';
interface CountrySelectProps {
value: string;
onChange: (code: string) => void;
placeholder?: string;
hasError?: boolean;
className?: string;
}
export default function CountrySelect({
value,
onChange,
placeholder = 'Select a country…',
hasError = false,
className,
}: CountrySelectProps) {
const [countries, setCountries] = useState<Country[]>([]);
const [loading, setLoading] = useState(true);
useEffect(() => {
countriesApi.list()
.then(resp => setCountries(resp.data))
.catch(() => setCountries([]))
.finally(() => setLoading(false));
}, []);
const baseCls =
'w-full h-12 px-3 border rounded-xs bg-surface text-on-surface text-sm outline-none transition-colors ' +
(hasError
? 'border-error focus:border-error focus:ring-2 focus:ring-error/20'
: 'border-outline focus:border-primary focus:ring-2 focus:ring-primary/20');
return (
<select
value={value}
onChange={e => onChange(e.target.value)}
disabled={loading}
className={`${baseCls} ${className ?? ''}`}
>
<option value="">{loading ? 'Loading countries…' : placeholder}</option>
{countries.map(c => (
<option key={c.code} value={c.code}>
{c.name} ({c.code}) — {c.airport_count} airport{c.airport_count !== 1 ? 's' : ''}
</option>
))}
</select>
);
}

73
flight-comparator/frontend/src/pages/ScanDetails.tsx
View File

@@ -136,16 +136,21 @@ export default function ScanDetails() {
return 0; return 0;
}); });
const toggleFlights = async (destination: string) => { // For reverse scans, route key = "ORIG:DEST"; for forward scans = "DEST"
if (expandedRoute === destination) { setExpandedRoute(null); return; } const routeKey = (route: Route) =>
setExpandedRoute(destination); route.origin_airport ? `${route.origin_airport}:${route.destination}` : route.destination;
if (flightsByDest[destination]) return;
setLoadingFlights(destination); const toggleFlights = async (route: Route) => {
const key = routeKey(route);
if (expandedRoute === key) { setExpandedRoute(null); return; }
setExpandedRoute(key);
if (flightsByDest[key]) return;
setLoadingFlights(key);
try { try {
const resp = await scanApi.getFlights(Number(id), destination, 1, 200); const resp = await scanApi.getFlights(Number(id), route.destination, route.origin_airport, 1, 200);
setFlightsByDest(prev => ({ ...prev, [destination]: resp.data.data })); setFlightsByDest(prev => ({ ...prev, [key]: resp.data.data }));
} catch { } catch {
setFlightsByDest(prev => ({ ...prev, [destination]: [] })); setFlightsByDest(prev => ({ ...prev, [key]: [] }));
} finally { } finally {
setLoadingFlights(null); setLoadingFlights(null);
} }
@@ -155,21 +160,30 @@ export default function ScanDetails() {
if (!scan) return; if (!scan) return;
setRerunning(true); setRerunning(true);
try { try {
// Compute window from stored dates so the new scan covers the same span
const ms = new Date(scan.end_date).getTime() - new Date(scan.start_date).getTime(); const ms = new Date(scan.end_date).getTime() - new Date(scan.start_date).getTime();
const window_months = Math.max(1, Math.round(ms / (1000 * 60 * 60 * 24 * 30))); const window_months = Math.max(1, Math.round(ms / (1000 * 60 * 60 * 24 * 30)));
// country column holds either "IT" or "BRI,BDS" const base = {
const isAirports = scan.country.includes(','); scan_mode: (scan.scan_mode ?? 'forward') as 'forward' | 'reverse',
const resp = await scanApi.create({
origin: scan.origin, origin: scan.origin,
window_months, window_months,
seat_class: scan.seat_class as 'economy' | 'premium' | 'business' | 'first', seat_class: scan.seat_class as 'economy' | 'premium' | 'business' | 'first',
adults: scan.adults, adults: scan.adults,
...(isAirports };
let extra: Record<string, unknown>;
if (scan.scan_mode === 'reverse') {
// For reverse: country column holds comma-separated dest IATAs
extra = { destinations: scan.country.split(',') };
} else {
// For forward: country column holds ISO code or comma-separated IATAs
const isAirports = scan.country.includes(',');
extra = isAirports
? { destinations: scan.country.split(',') } ? { destinations: scan.country.split(',') }
: { country: scan.country }), : { country: scan.country };
}); }
const resp = await scanApi.create({ ...base, ...extra });
navigate(`/scans/${resp.data.id}`); navigate(`/scans/${resp.data.id}`);
} catch { } catch {
// silently fall through — the navigate won't happen // silently fall through — the navigate won't happen
@@ -302,7 +316,9 @@ export default function ScanDetails() {
<div className="flex items-center gap-2 flex-wrap min-w-0"> <div className="flex items-center gap-2 flex-wrap min-w-0">
<PlaneTakeoff size={20} className="text-primary shrink-0" aria-hidden="true" /> <PlaneTakeoff size={20} className="text-primary shrink-0" aria-hidden="true" />
<h1 className="text-xl font-semibold text-on-surface"> <h1 className="text-xl font-semibold text-on-surface">
{scan.origin} → {scan.country} {scan.scan_mode === 'reverse'
? `${scan.origin} → ${scan.country.split(',').join(', ')}`
: `${scan.origin} → ${scan.country}`}
</h1> </h1>
{scan.scheduled_scan_id != null && ( {scan.scheduled_scan_id != null && (
<Link <Link
@@ -586,6 +602,9 @@ export default function ScanDetails() {
<table className="w-full"> <table className="w-full">
<thead className="bg-surface-2 border-b border-outline"> <thead className="bg-surface-2 border-b border-outline">
<tr> <tr>
{scan.scan_mode === 'reverse' && (
<th className={thCls()}>Origin</th>
)}
<th <th
className={thCls('destination')} className={thCls('destination')}
onClick={() => handleSort('destination')} onClick={() => handleSort('destination')}
@@ -617,13 +636,23 @@ export default function ScanDetails() {
</thead> </thead>
<tbody className="divide-y divide-outline"> <tbody className="divide-y divide-outline">
{routes.map((route) => { {routes.map((route) => {
const isExpanded = expandedRoute === route.destination; const key = routeKey(route);
const isExpanded = expandedRoute === key;
const colSpan = scan.scan_mode === 'reverse' ? 7 : 6;
return ( return (
<Fragment key={route.id}> <Fragment key={route.id}>
<tr <tr
className="hover:bg-surface-2 cursor-pointer transition-colors duration-150" className="hover:bg-surface-2 cursor-pointer transition-colors duration-150"
onClick={() => toggleFlights(route.destination)} onClick={() => toggleFlights(route)}
> >
{/* Origin (reverse scans only) */}
{scan.scan_mode === 'reverse' && (
<td className="px-4 py-4">
<span className="font-mono text-secondary bg-surface-2 px-2 py-0.5 rounded-sm text-sm font-medium">
{route.origin_airport ?? '—'}
</span>
</td>
)}
{/* Destination */} {/* Destination */}
<td className="px-4 py-4"> <td className="px-4 py-4">
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
@@ -680,13 +709,13 @@ export default function ScanDetails() {
{/* Expanded flights sub-row */} {/* Expanded flights sub-row */}
<tr key={`${route.id}-flights`}> <tr key={`${route.id}-flights`}>
<td colSpan={6} className="p-0"> <td colSpan={colSpan} className="p-0">
<div <div
className="overflow-hidden transition-all duration-250 ease-in-out" className="overflow-hidden transition-all duration-250 ease-in-out"
style={{ maxHeight: isExpanded ? '600px' : '0' }} style={{ maxHeight: isExpanded ? '600px' : '0' }}
> >
<div className="bg-[#F8FDF9]"> <div className="bg-[#F8FDF9]">
{loadingFlights === route.destination ? ( {loadingFlights === key ? (
<table className="w-full"> <table className="w-full">
<tbody> <tbody>
<SkeletonTableRow /> <SkeletonTableRow />
@@ -720,7 +749,7 @@ export default function ScanDetails() {
</tr> </tr>
</thead> </thead>
<tbody className="divide-y divide-[#D4EDDA]"> <tbody className="divide-y divide-[#D4EDDA]">
{sortedFlights(flightsByDest[route.destination] || []).map((f) => ( {sortedFlights(flightsByDest[key] || []).map((f) => (
<tr key={f.id} className="hover:bg-[#EEF7F0] transition-colors"> <tr key={f.id} className="hover:bg-[#EEF7F0] transition-colors">
<td className="pl-12 pr-4 py-2.5 text-sm text-on-surface"> <td className="pl-12 pr-4 py-2.5 text-sm text-on-surface">
<span className="font-mono text-xs font-semibold text-on-surface-variant mr-2">{weekday(f.date)}</span> <span className="font-mono text-xs font-semibold text-on-surface-variant mr-2">{weekday(f.date)}</span>
@@ -734,7 +763,7 @@ export default function ScanDetails() {
</td> </td>
</tr> </tr>
))} ))}
{(flightsByDest[route.destination] || []).length === 0 && ( {(flightsByDest[key] || []).length === 0 && (
<tr> <tr>
<td colSpan={5} className="pl-12 py-4 text-sm text-on-surface-variant"> <td colSpan={5} className="pl-12 py-4 text-sm text-on-surface-variant">
No flight details available No flight details available

157
flight-comparator/frontend/src/pages/Scans.tsx
View File

@@ -1,11 +1,12 @@
import { useState } from 'react'; import { useState } from 'react';
import { useNavigate } from 'react-router-dom'; import { useNavigate } from 'react-router-dom';
import { Globe, PlaneTakeoff, Minus, Plus } from 'lucide-react'; import { Globe, PlaneTakeoff, Minus, Plus, ArrowRight, ArrowLeft } from 'lucide-react';
import { scanApi } from '../api'; import { scanApi } from '../api';
import type { CreateScanRequest } from '../api'; import type { CreateScanRequest } from '../api';
import AirportSearch from '../components/AirportSearch'; import AirportSearch from '../components/AirportSearch';
import SegmentedButton from '../components/SegmentedButton'; import SegmentedButton from '../components/SegmentedButton';
import AirportChip from '../components/AirportChip'; import AirportChip from '../components/AirportChip';
import CountrySelect from '../components/CountrySelect';
import Button from '../components/Button'; import Button from '../components/Button';
import Toast from '../components/Toast'; import Toast from '../components/Toast';
@@ -19,6 +20,11 @@ interface FormErrors {
export default function Scans() { export default function Scans() {
const navigate = useNavigate(); const navigate = useNavigate();
// Direction: forward (fixed origin → variable destinations) or reverse (variable origins → fixed destinations)
const [scanMode, setScanMode] = useState<'forward' | 'reverse'>('forward');
// Forward mode state
const [destinationMode, setDestinationMode] = useState<'country' | 'airports'>('country'); const [destinationMode, setDestinationMode] = useState<'country' | 'airports'>('country');
const [formData, setFormData] = useState<CreateScanRequest>({ const [formData, setFormData] = useState<CreateScanRequest>({
origin: '', origin: '',
@@ -27,22 +33,37 @@ export default function Scans() {
seat_class: 'economy', seat_class: 'economy',
adults: 1, adults: 1,
}); });
// Shared state
const [selectedAirports, setSelectedAirports] = useState<string[]>([]); const [selectedAirports, setSelectedAirports] = useState<string[]>([]);
const [selectedOriginCountry, setSelectedOriginCountry] = useState('');
const [loading, setLoading] = useState(false); const [loading, setLoading] = useState(false);
const [errors, setErrors] = useState<FormErrors>({}); const [errors, setErrors] = useState<FormErrors>({});
const [toast, setToast] = useState<{ message: string; type: 'success' | 'error' } | null>(null); const [toast, setToast] = useState<{ message: string; type: 'success' | 'error' } | null>(null);
const validate = (): boolean => { const validate = (): boolean => {
const next: FormErrors = {}; const next: FormErrors = {};
if (scanMode === 'reverse') {
if (!selectedOriginCountry) {
next.country = 'Select an origin country';
}
if (selectedAirports.length === 0) {
next.airports = 'Add at least one destination airport';
}
} else {
if (!formData.origin || formData.origin.length !== 3) { if (!formData.origin || formData.origin.length !== 3) {
next.origin = 'Enter a valid 3-letter IATA code'; next.origin = 'Enter a valid 3-letter IATA code';
} }
if (destinationMode === 'country' && (!formData.country || formData.country.length !== 2)) { if (destinationMode === 'country' && !formData.country) {
next.country = 'Enter a valid 2-letter country code'; next.country = 'Select a destination country';
} }
if (destinationMode === 'airports' && selectedAirports.length === 0) { if (destinationMode === 'airports' && selectedAirports.length === 0) {
next.airports = 'Add at least one destination airport'; next.airports = 'Add at least one destination airport';
} }
}
setErrors(next); setErrors(next);
return Object.keys(next).length === 0; return Object.keys(next).length === 0;
}; };
@@ -53,17 +74,28 @@ export default function Scans() {
setLoading(true); setLoading(true);
try { try {
const requestData: any = { let requestData: CreateScanRequest;
origin: formData.origin,
if (scanMode === 'reverse') {
requestData = {
scan_mode: 'reverse',
origin: selectedOriginCountry,
destinations: selectedAirports,
window_months: formData.window_months, window_months: formData.window_months,
seat_class: formData.seat_class, seat_class: formData.seat_class,
adults: formData.adults, adults: formData.adults,
}; };
if (destinationMode === 'country') {
requestData.country = formData.country;
} else { } else {
requestData.destinations = selectedAirports; requestData = {
scan_mode: 'forward',
origin: formData.origin,
window_months: formData.window_months,
seat_class: formData.seat_class,
adults: formData.adults,
...(destinationMode === 'country'
? { country: formData.country }
: { destinations: selectedAirports }),
};
} }
const response = await scanApi.create(requestData); const response = await scanApi.create(requestData);
@@ -86,7 +118,6 @@ export default function Scans() {
})); }));
}; };
// Shared input class
const inputCls = (hasError?: boolean) => const inputCls = (hasError?: boolean) =>
`w-full h-12 px-3 border rounded-xs bg-surface text-on-surface text-sm outline-none transition-colors ` + `w-full h-12 px-3 border rounded-xs bg-surface text-on-surface text-sm outline-none transition-colors ` +
(hasError (hasError
@@ -97,12 +128,57 @@ export default function Scans() {
<> <>
<form onSubmit={handleSubmit} className="space-y-4 max-w-2xl"> <form onSubmit={handleSubmit} className="space-y-4 max-w-2xl">
{/* ── Section: Direction ───────────────────────────────────── */}
<div className="bg-surface rounded-lg shadow-level-1 p-6">
<p className="text-xs font-semibold uppercase tracking-wider text-on-surface-variant mb-4">
Direction
</p>
<SegmentedButton
options={[
{ value: 'forward', label: 'Forward', icon: ArrowRight },
{ value: 'reverse', label: 'Reverse', icon: ArrowLeft },
]}
value={scanMode}
onChange={(v) => {
setScanMode(v as 'forward' | 'reverse');
setErrors({});
setSelectedAirports([]);
}}
/>
<p className="mt-2 text-xs text-on-surface-variant">
{scanMode === 'forward'
? 'Fixed origin airport → all airports in a country (or specific airports)'
: 'All airports in a country → specific destination airport(s)'}
</p>
</div>
{/* ── Section: Origin ─────────────────────────────────────── */} {/* ── Section: Origin ─────────────────────────────────────── */}
<div className="bg-surface rounded-lg shadow-level-1 p-6"> <div className="bg-surface rounded-lg shadow-level-1 p-6">
<p className="text-xs font-semibold uppercase tracking-wider text-on-surface-variant mb-4"> <p className="text-xs font-semibold uppercase tracking-wider text-on-surface-variant mb-4">
Origin Origin
</p> </p>
{scanMode === 'reverse' ? (
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Origin Country
</label>
<CountrySelect
value={selectedOriginCountry}
onChange={(code) => {
setSelectedOriginCountry(code);
if (errors.country) setErrors(prev => ({ ...prev, country: undefined }));
}}
placeholder="Select origin country…"
hasError={!!errors.country}
/>
{errors.country ? (
<p className="mt-1 text-xs text-error">{errors.country}</p>
) : (
<p className="mt-1 text-xs text-on-surface-variant">All airports in this country will be checked</p>
)}
</div>
) : (
<div> <div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5"> <label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Origin Airport Origin Airport
@@ -122,6 +198,7 @@ export default function Scans() {
<p className="mt-1 text-xs text-on-surface-variant">3-letter IATA code (e.g. BDS for Brindisi)</p> <p className="mt-1 text-xs text-on-surface-variant">3-letter IATA code (e.g. BDS for Brindisi)</p>
)} )}
</div> </div>
)}
</div> </div>
{/* ── Section: Destination ────────────────────────────────── */} {/* ── Section: Destination ────────────────────────────────── */}
@@ -130,6 +207,48 @@ export default function Scans() {
Destination Destination
</p> </p>
{scanMode === 'reverse' ? (
/* Reverse: specific destination airports */
<div>
<label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Destination Airports
</label>
<AirportSearch
value=""
onChange={(code) => {
if (code && code.length === 3 && !selectedAirports.includes(code)) {
setSelectedAirports(prev => [...prev, code]);
if (errors.airports) setErrors(prev => ({ ...prev, airports: undefined }));
}
}}
clearAfterSelect
placeholder="Search and add airports…"
hasError={!!errors.airports}
/>
{selectedAirports.length > 0 && (
<div className="flex flex-wrap gap-2 mt-3">
{selectedAirports.map((code) => (
<AirportChip
key={code}
code={code}
onRemove={() => setSelectedAirports(prev => prev.filter(c => c !== code))}
/>
))}
</div>
)}
{errors.airports ? (
<p className="mt-1 text-xs text-error">{errors.airports}</p>
) : (
<p className="mt-1 text-xs text-on-surface-variant">
{selectedAirports.length === 0
? 'Search and add destination airports (up to 50)'
: `${selectedAirports.length} airport${selectedAirports.length !== 1 ? 's' : ''} selected`}
</p>
)}
</div>
) : (
/* Forward: by country or by specific airports */
<>
<SegmentedButton <SegmentedButton
options={[ options={[
{ value: 'country', label: 'By Country', icon: Globe }, { value: 'country', label: 'By Country', icon: Globe },
@@ -148,21 +267,19 @@ export default function Scans() {
<label className="block text-sm font-medium text-on-surface-variant mb-1.5"> <label className="block text-sm font-medium text-on-surface-variant mb-1.5">
Destination Country Destination Country
</label> </label>
<input <CountrySelect
type="text" value={formData.country ?? ''}
value={formData.country} onChange={(code) => {
onChange={(e) => { setFormData(prev => ({ ...prev, country: code }));
setFormData(prev => ({ ...prev, country: e.target.value.toUpperCase() }));
if (errors.country) setErrors(prev => ({ ...prev, country: undefined })); if (errors.country) setErrors(prev => ({ ...prev, country: undefined }));
}} }}
maxLength={2} placeholder="Select destination country…"
placeholder="e.g. DE, IT, ES" hasError={!!errors.country}
className={inputCls(!!errors.country)}
/> />
{errors.country ? ( {errors.country ? (
<p className="mt-1 text-xs text-error">{errors.country}</p> <p className="mt-1 text-xs text-error">{errors.country}</p>
) : ( ) : (
<p className="mt-1 text-xs text-on-surface-variant">ISO 2-letter country code (e.g. DE for Germany)</p> <p className="mt-1 text-xs text-on-surface-variant">All airports in this country will be searched</p>
)} )}
</div> </div>
) : ( ) : (
@@ -204,6 +321,8 @@ export default function Scans() {
)} )}
</div> </div>
)} )}
</>
)}
</div> </div>
{/* ── Section: Parameters ─────────────────────────────────── */} {/* ── Section: Parameters ─────────────────────────────────── */}

122
flight-comparator/scan_processor.py
View File

@@ -52,7 +52,7 @@ def stop_scan_task(scan_id: int) -> bool:
def _write_route_incremental(scan_id: int, destination: str, def _write_route_incremental(scan_id: int, destination: str,
dest_name: str, dest_city: str, dest_name: str, dest_city: str,
new_flights: list): new_flights: list, origin_airport: str = None):
""" """
Write or update a route row and its individual flight rows immediately. Write or update a route row and its individual flight rows immediately.
@@ -60,6 +60,9 @@ def _write_route_incremental(scan_id: int, destination: str,
query returns results. Merges into the existing route row if one already query returns results. Merges into the existing route row if one already
exists, using a running weighted average for avg_price. exists, using a running weighted average for avg_price.
For reverse scans, origin_airport is the variable origin IATA code.
For forward scans, origin_airport is None.
Opens its own DB connection — safe to call from the event loop thread. Opens its own DB connection — safe to call from the event loop thread.
""" """
prices = [f.get('price') for f in new_flights if f.get('price')] prices = [f.get('price') for f in new_flights if f.get('price')]
@@ -76,21 +79,29 @@ def _write_route_incremental(scan_id: int, destination: str,
conn = get_connection() conn = get_connection()
cursor = conn.cursor() cursor = conn.cursor()
# Fetch existing route row (key: scan_id + origin_airport + destination)
if origin_airport is None:
cursor.execute(""" cursor.execute("""
SELECT id, flight_count, min_price, max_price, avg_price, airlines SELECT id, flight_count, min_price, max_price, avg_price, airlines
FROM routes FROM routes
WHERE scan_id = ? AND destination = ? WHERE scan_id = ? AND origin_airport IS NULL AND destination = ?
""", (scan_id, destination)) """, (scan_id, destination))
else:
cursor.execute("""
SELECT id, flight_count, min_price, max_price, avg_price, airlines
FROM routes
WHERE scan_id = ? AND origin_airport = ? AND destination = ?
""", (scan_id, origin_airport, destination))
existing = cursor.fetchone() existing = cursor.fetchone()
if existing is None: if existing is None:
cursor.execute(""" cursor.execute("""
INSERT INTO routes ( INSERT INTO routes (
scan_id, destination, destination_name, destination_city, scan_id, origin_airport, destination, destination_name, destination_city,
flight_count, airlines, min_price, max_price, avg_price flight_count, airlines, min_price, max_price, avg_price
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", ( """, (
scan_id, destination, dest_name, dest_city, scan_id, origin_airport, destination, dest_name, dest_city,
new_count, json.dumps(new_airlines), new_count, json.dumps(new_airlines),
new_min, new_max, new_avg, new_min, new_max, new_avg,
)) ))
@@ -107,6 +118,7 @@ def _write_route_incremental(scan_id: int, destination: str,
merged_avg = (old_avg * old_count + new_avg * new_count) / merged_count merged_avg = (old_avg * old_count + new_avg * new_count) / merged_count
merged_airlines = json.dumps(list(set(old_airlines) | set(new_airlines))) merged_airlines = json.dumps(list(set(old_airlines) | set(new_airlines)))
if origin_airport is None:
cursor.execute(""" cursor.execute("""
UPDATE routes UPDATE routes
SET flight_count = ?, SET flight_count = ?,
@@ -114,22 +126,36 @@ def _write_route_incremental(scan_id: int, destination: str,
max_price = ?, max_price = ?,
avg_price = ?, avg_price = ?,
airlines = ? airlines = ?
WHERE scan_id = ? AND destination = ? WHERE scan_id = ? AND origin_airport IS NULL AND destination = ?
""", ( """, (
merged_count, merged_min, merged_max, merged_avg, merged_airlines, merged_count, merged_min, merged_max, merged_avg, merged_airlines,
scan_id, destination, scan_id, destination,
)) ))
else:
cursor.execute("""
UPDATE routes
SET flight_count = ?,
min_price = ?,
max_price = ?,
avg_price = ?,
airlines = ?
WHERE scan_id = ? AND origin_airport = ? AND destination = ?
""", (
merged_count, merged_min, merged_max, merged_avg, merged_airlines,
scan_id, origin_airport, destination,
))
for flight in new_flights: for flight in new_flights:
if not flight.get('price'): if not flight.get('price'):
continue continue
cursor.execute(""" cursor.execute("""
INSERT INTO flights ( INSERT INTO flights (
scan_id, destination, date, airline, scan_id, origin_airport, destination, date, airline,
departure_time, arrival_time, price, stops departure_time, arrival_time, price, stops
) VALUES (?, ?, ?, ?, ?, ?, ?, ?) ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""", ( """, (
scan_id, scan_id,
origin_airport,
destination, destination,
flight.get('date', ''), flight.get('date', ''),
flight.get('airline'), flight.get('airline'),
@@ -170,7 +196,7 @@ async def process_scan(scan_id: int):
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute(""" cursor.execute("""
SELECT origin, country, start_date, end_date, seat_class, adults SELECT origin, country, scan_mode, start_date, end_date, seat_class, adults
FROM scans FROM scans
WHERE id = ? WHERE id = ?
""", (scan_id,)) """, (scan_id,))
@@ -180,9 +206,10 @@ async def process_scan(scan_id: int):
logger.error(f"[Scan {scan_id}] Scan not found in database") logger.error(f"[Scan {scan_id}] Scan not found in database")
return return
origin, country_or_airports, start_date_str, end_date_str, seat_class, adults = row origin, country_or_airports, scan_mode, start_date_str, end_date_str, seat_class, adults = row
scan_mode = scan_mode or 'forward'
logger.info(f"[Scan {scan_id}] Scan details: {origin} -> {country_or_airports}, {start_date_str} to {end_date_str}") logger.info(f"[Scan {scan_id}] Scan details: mode={scan_mode}, {origin} -> {country_or_airports}, {start_date_str} to {end_date_str}")
# Update status to 'running' and record when processing started # Update status to 'running' and record when processing started
cursor.execute(""" cursor.execute("""
@@ -192,27 +219,39 @@ async def process_scan(scan_id: int):
""", (scan_id,)) """, (scan_id,))
conn.commit() conn.commit()
# Determine mode: country (2 letters) or specific airports (comma-separated) # Resolve airports based on scan_mode
try: try:
if len(country_or_airports) == 2 and country_or_airports.isalpha(): if scan_mode == 'reverse':
# Country mode: resolve airports from country code # Reverse scan: origin = ISO country, country_or_airports = comma-separated dest IATAs
logger.info(f"[Scan {scan_id}] Mode: Country search ({country_or_airports})") logger.info(f"[Scan {scan_id}] Mode: Reverse scan ({origin} country → {country_or_airports})")
destinations = get_airports_for_country(country_or_airports) origin_airports = get_airports_for_country(origin)
if not destinations: if not origin_airports:
raise ValueError(f"No airports found for country: {country_or_airports}") raise ValueError(f"No airports found for origin country: {origin}")
origin_iatas = [a['iata'] for a in origin_airports]
destination_codes = [d['iata'] for d in destinations] destination_codes = [code.strip() for code in country_or_airports.split(',')]
dest_infos = {
logger.info(f"[Scan {scan_id}] Found {len(destination_codes)} destination airports: {destination_codes}") code: lookup_airport(code) or {'iata': code, 'name': code, 'city': ''}
for code in destination_codes
}
logger.info(f"[Scan {scan_id}] {len(origin_iatas)} origins × {len(destination_codes)} destinations")
else: else:
# Specific airports mode: parse comma-separated list # Forward scan: origin = fixed IATA, country_or_airports = country code or dest IATAs
if len(country_or_airports) == 2 and country_or_airports.isalpha():
logger.info(f"[Scan {scan_id}] Mode: Forward country search ({country_or_airports})")
dest_list = get_airports_for_country(country_or_airports)
if not dest_list:
raise ValueError(f"No airports found for country: {country_or_airports}")
destination_codes = [d['iata'] for d in dest_list]
dest_infos = {d['iata']: d for d in dest_list}
else:
destination_codes = [code.strip() for code in country_or_airports.split(',')] destination_codes = [code.strip() for code in country_or_airports.split(',')]
destinations = [ dest_infos = {
lookup_airport(code) or {'iata': code, 'name': code, 'city': ''} code: lookup_airport(code) or {'iata': code, 'name': code, 'city': ''}
for code in destination_codes for code in destination_codes
] }
logger.info(f"[Scan {scan_id}] Mode: Specific airports ({len(destination_codes)} destinations: {destination_codes})") logger.info(f"[Scan {scan_id}] Mode: Forward specific airports ({destination_codes})")
except Exception as e: except Exception as e:
logger.error(f"[Scan {scan_id}] Failed to resolve airports: {str(e)}") logger.error(f"[Scan {scan_id}] Failed to resolve airports: {str(e)}")
@@ -227,8 +266,6 @@ async def process_scan(scan_id: int):
conn.commit() conn.commit()
return return
# Note: Don't update total_routes yet - we'll set it after we know the actual number of route queries
# Generate dates to scan — every day in the window # Generate dates to scan — every day in the window
start_date = datetime.strptime(start_date_str, '%Y-%m-%d').date() start_date = datetime.strptime(start_date_str, '%Y-%m-%d').date()
end_date = datetime.strptime(end_date_str, '%Y-%m-%d').date() end_date = datetime.strptime(end_date_str, '%Y-%m-%d').date()
@@ -241,11 +278,17 @@ async def process_scan(scan_id: int):
logger.info(f"[Scan {scan_id}] Will scan {len(dates)} dates: {dates}") logger.info(f"[Scan {scan_id}] Will scan {len(dates)} dates: {dates}")
# Build routes list: [(origin, destination, date), ...] # Build routes list: [(origin_iata, destination_iata, date), ...]
routes_to_scan = [] routes_to_scan = []
for dest in destination_codes: if scan_mode == 'reverse':
for orig_iata in origin_iatas:
for dest_code in destination_codes:
for scan_date in dates: for scan_date in dates:
routes_to_scan.append((origin, dest, scan_date)) routes_to_scan.append((orig_iata, dest_code, scan_date))
else:
for dest_code in destination_codes:
for scan_date in dates:
routes_to_scan.append((origin, dest_code, scan_date))
logger.info(f"[Scan {scan_id}] Total route queries: {len(routes_to_scan)}") logger.info(f"[Scan {scan_id}] Total route queries: {len(routes_to_scan)}")
@@ -262,7 +305,7 @@ async def process_scan(scan_id: int):
# Signature: callback(origin, destination, date, status, count, error=None, flights=None) # Signature: callback(origin, destination, date, status, count, error=None, flights=None)
routes_scanned_count = 0 routes_scanned_count = 0
def progress_callback(origin: str, destination: str, date: str, def progress_callback(cb_origin: str, destination: str, date: str,
status: str, count: int, error: str = None, status: str, count: int, error: str = None,
flights: list = None): flights: list = None):
nonlocal routes_scanned_count nonlocal routes_scanned_count
@@ -274,10 +317,15 @@ async def process_scan(scan_id: int):
if flights and status in ('cache_hit', 'api_success'): if flights and status in ('cache_hit', 'api_success'):
for f in flights: for f in flights:
f['date'] = date f['date'] = date
dest_info = next((d for d in destinations if d['iata'] == destination), None) dest_info = dest_infos.get(destination) or {'iata': destination, 'name': destination, 'city': ''}
dest_name = dest_info.get('name', destination) if dest_info else destination dest_name = dest_info.get('name', destination)
dest_city = dest_info.get('city', '') if dest_info else '' dest_city = dest_info.get('city', '')
_write_route_incremental(scan_id, destination, dest_name, dest_city, flights) # For reverse scans, cb_origin is the variable origin airport IATA
route_origin = cb_origin if scan_mode == 'reverse' else None
_write_route_incremental(
scan_id, destination, dest_name, dest_city, flights,
origin_airport=route_origin
)
# Update progress counter # Update progress counter
try: try:
@@ -295,7 +343,7 @@ async def process_scan(scan_id: int):
progress_conn.close() progress_conn.close()
if routes_scanned_count % 10 == 0: if routes_scanned_count % 10 == 0:
logger.info(f"[Scan {scan_id}] Progress: {routes_scanned_count}/{len(routes_to_scan)} routes ({status}: {origin}→{destination})") logger.info(f"[Scan {scan_id}] Progress: {routes_scanned_count}/{len(routes_to_scan)} routes ({status}: {cb_origin}→{destination})")
except Exception as e: except Exception as e:
logger.error(f"[Scan {scan_id}] Failed to update progress: {str(e)}") logger.error(f"[Scan {scan_id}] Failed to update progress: {str(e)}")