domverse/outline-sync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add sync engine, web UI, Docker setup, and tests

Browse Source 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-03-07 20:54:59 +01:00
parent e4c69efd12
commit b3137a8af3
27 changed files with 7133 additions and 278 deletions
Download Patch File Download Diff File Expand all files Collapse all files

484
WEBUI_PRD.md Normal file
View File

@@ -0,0 +1,484 @@
# PRD: Outline Sync Web UI
**Version:** 2.0
**Date:** 2026-03-06
**Status:** Draft — WebDAV Architecture
**Depends on:** SYNC_PRD.md (sync engine)
---
## 1. Problem Statement
The sync engine (`sync.sh`) works but requires terminal access on the VPS. The user edits notes locally in Obsidian on their own machine. Changes need to flow bidirectionally between local Obsidian and the remote Outline wiki without terminal interaction.
The key constraint: **Obsidian runs locally, the vault git repo lives on the VPS.**
---
## 2. Architecture Options Evaluated
| Option | Mechanism | Verdict |
|---|---|---|
| **WebDAV + Remotely Save** | WebDAV server on VPS serves vault dir; Obsidian plugin syncs automatically | ✅ Recommended |
| **Self-hosted LiveSync** | CouchDB on VPS; Obsidian plugin syncs in real-time | ❌ Adds CouchDB; no direct file access for sync engine |
| **Local REST API** | Obsidian exposes REST on localhost:27124 | ❌ VPS can't reach local machine |
| **Manual zip download/upload** | Browser download/upload of vault archives | ❌ Eliminated by WebDAV option |
**Decision: WebDAV.** A WebDAV Docker container on the VPS serves the vault directory directly. The Obsidian plugin `remotely-save` syncs to it automatically. No tunnel, no extra database, no build step.
---
## 3. System Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Your local machine │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Obsidian │ │
│ │ Plugin: remotely-save │ │
│ │ Auto-sync every N minutes (or on open/close) │ │
│ └──────────────────────┬──────────────────────────┘ │
└─────────────────────────│───────────────────────────────┘
│ WebDAV over Tailscale (encrypted tunnel)
│ (basic auth + Tailscale network isolation)
┌─────────────────────────▼───────────────────────────────┐
│ VPS (domverse.de) │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ WebDAV container (obsidian-webdav) │ │
│ │ Serves: /outline-vault/ (read-write) │ │
│ │ URL: http://100.x.x.x (Tailscale only) │ │
│ └────────────────────┬─────────────────────────────┘ │
│ │ shared volume │
│ ┌────────────────────▼─────────────────────────────┐ │
│ │ /outline-vault/ (git repo) │ │
│ │ ├── .git/ │ │
│ │ │ ├── outline branch (last Outline state) │ │
│ │ │ └── main branch (current vault state) │ │
│ │ ├── Bewerbungen/ │ │
│ │ ├── Projekte/ │ │
│ │ └── _sync_log.md │ │
│ └────────────────────┬─────────────────────────────┘ │
│ │ shared volume │
│ ┌────────────────────▼─────────────────────────────┐ │
│ │ outline-sync-ui (FastAPI, port 8080) │ │
│ │ URL: https://sync.domverse.de/ │ │
│ │ Auth: Authentik │ │
│ └────────────────────┬─────────────────────────────┘ │
│ │ │
│ ┌────────────────────▼─────────────────────────────┐ │
│ │ outline_sync.py (sync engine) │ │
│ │ network: domnet │ │
│ │ API: http://outline:3000 │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
### Data Flow: Obsidian → Outline
1. User edits notes locally in Obsidian
2. `remotely-save` plugin syncs changed files to WebDAV server on VPS (automatic)
3. WebDAV writes files directly into `/outline-vault/`
4. User clicks "Send to Outline" in web UI (or cron runs push)
5. Sync engine: `git diff outline..main` → calls Outline API per changed file
6. Sync engine advances `outline` branch
### Data Flow: Outline → Obsidian
1. User clicks "Get from Outline" in web UI (or cron runs pull)
2. Sync engine exports Outline → commits to `outline` branch → merges into `main`
3. Files updated in `/outline-vault/`
4. WebDAV immediately serves updated files
5. `remotely-save` plugin picks up changes on next sync interval
---
## 4. User-Facing Mental Model
Two systems, one bridge:
| What the user sees | What actually happens |
|---|---|
| Obsidian auto-syncs | `remotely-save` talks WebDAV to VPS |
| "Get from Outline" | sync engine pulls Outline API → git merge → files update in vault |
| "Send to Outline" | sync engine diffs vault → calls Outline API |
| "Version conflict" | `git merge` produced conflict markers |
| "Keep mine / Keep Outline's" | `sync resolve --accept local/remote` |
Git is invisible. WebDAV is invisible. Obsidian just sees files that stay in sync.
---
## 5. Goals
| # | Goal |
|---|------|
| G1 | Obsidian syncs to VPS automatically — no manual file transfer |
| G2 | Pulling from Outline and pushing to Outline are single button clicks in a browser |
| G3 | Live output visible while sync runs |
| G4 | Conflicts resolvable in browser with side-by-side diff |
| G5 | New files created in Obsidian appear in Outline after push |
| G6 | New documents created in Outline appear in Obsidian after next WebDAV sync |
| G7 | Web UI protected by Authentik SSO |
| G8 | WebDAV endpoint protected by authentication |
| G9 | Zero terminal interaction for normal workflow |
## 6. Non-Goals
| # | Non-Goal | Reason |
|---|----------|--------|
| NG1 | In-browser markdown editor | Obsidian is the editor |
| NG2 | Real-time Outline → Obsidian sync | WebDAV poll interval (e.g. 5 min) is sufficient |
| NG3 | Syncing `.obsidian/` config, templates, daily notes | Outline-synced content only |
| NG4 | Multi-user | Single-user system |
| NG5 | Mobile Obsidian support | Remotely Save does support mobile, but not a primary target |
---
## 6. Component Inventory
### 6.1 WebDAV Server (new container)
Simple nginx-WebDAV container. Serves `/outline-vault/` as a WebDAV share.
**Not exposed via Traefik.** Bound exclusively to the VPS Tailscale interface — invisible from the public internet.
```yaml
obsidian-webdav:
image: dgraziotin/nginx-webdav-nononsense:latest
container_name: obsidian-webdav
networks:
- domnet
ports:
- "100.x.x.x:80:80" # Tailscale IP only — replace with: tailscale ip -4
volumes:
- /home/crabix/docker_authentik/outline-vault:/data
environment:
- WEBDAV_USERNAME=obsidian
- WEBDAV_PASSWORD=${WEBDAV_PASSWORD} # from .env
restart: unless-stopped
# No Traefik labels — not publicly routed
```
**Auth note:** WebDAV cannot use Authentik forward auth (Obsidian plugin can't handle SSO redirect). Network isolation via Tailscale is the primary security layer — the endpoint is unreachable from the public internet. Basic auth (htpasswd) via nginx provides a second layer. Tailscale encrypts the tunnel, so plain HTTP is safe for this leg.
### 6.2 Web UI (new container)
FastAPI + HTMX. Control plane for sync operations.
```yaml
outline-sync-ui:
image: python:3.11-slim
container_name: outline-sync-ui
networks:
- domnet
volumes:
- /home/crabix/docker_authentik/outline-tools:/app:ro # scripts, read-only
- /home/crabix/docker_authentik/outline-vault:/vault # vault, read-write
working_dir: /app
command: >
bash -c "pip install -qqq fastapi uvicorn requests &&
uvicorn webui:app --host 0.0.0.0 --port 8080"
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.outline-sync.rule=Host(`sync.domverse.de`)"
- "traefik.http.routers.outline-sync.entrypoints=https"
- "traefik.http.routers.outline-sync.tls=true"
- "traefik.http.routers.outline-sync.middlewares=secHeaders@file,middlewares-authentik-new@file"
- "traefik.http.services.outline-sync.loadbalancer.server.port=8080"
```
### 6.3 Obsidian Setup (one-time, on local machine)
1. Install plugin `remotely-save` from Obsidian community plugins
2. Ensure Tailscale is running on the local machine and connected to the VPS
3. Configure:
- **Remote service:** WebDAV
- **Server URL:** `http://100.x.x.x` (VPS Tailscale IP — plain HTTP, tunnel is encrypted)
- **Username / Password:** WebDAV credentials
- **Sync on startup:** yes
- **Sync interval:** every 5 minutes (or on vault open/close)
- **Sync direction:** bidirectional (default)
4. Exclude `.git/` from sync (configure in plugin's ignore list)
---
## 7. New File Handling
This is the key correctness concern: when the user creates a new `.md` file in Obsidian, it must reach Outline correctly.
### Flow for new files
1. User creates `Projekte/NewNote.md` in Obsidian (no frontmatter)
2. `remotely-save` syncs it to VPS via WebDAV → file appears in `/outline-vault/Projekte/NewNote.md`
3. User clicks "Send to Outline" in web UI
4. Sync engine: `git diff outline..main` shows `A Projekte/NewNote.md`
5. Sync engine determines parent: `Projekte/` folder exists → look up `Projekte.md` for its `outline_id` → use as `parentDocumentId`
6. `documents.create` called → new document created in Outline under correct collection/parent
7. `outline_id` written back into frontmatter of `NewNote.md`
8. File committed → WebDAV serves updated file with frontmatter → Obsidian picks it up on next sync
### New file in new folder
1. User creates `NewCollection/FirstDoc.md`
2. Sync engine: top-level folder `NewCollection/` not in any known collection → `collections.create("NewCollection")`
3. `FirstDoc` created inside new collection
4. Frontmatter written back to both files
### Ordering guarantee
Sync engine creates documents in topological order (parents before children), regardless of which order files were synced via WebDAV.
---
## 8. Conflict Model
A conflict occurs when:
- User edited `File.md` in Obsidian (synced via WebDAV to `main` branch)
- Someone also edited the same document in Outline
- `sync pull` → `git merge outline` → conflict markers in `File.md`
After conflict:
- `File.md` contains `<<<<<<<` markers → WebDAV immediately serves this broken file → Obsidian shows it with markers
- Conflict is resolved in the **web UI** (not Obsidian)
- After resolution, WebDAV serves the clean file → Obsidian's next sync picks it up
**This is the key reason the web UI exists:** conflict resolution in a browser with a diff view is far better than editing git conflict markers in a text editor.
---
## 9. Web UI Screens
### 9.1 Dashboard (`/`)
```
┌─────────────────────────────────────────────────────────┐
│ Outline Sync sync.domverse.de │
├─────────────────────────────────────────────────────────┤
│ │
│ Vault status ● Clean │
│ Last pull 2026-03-06 14:32 (from Outline) │
│ Last push 2026-03-05 09:10 (to Outline) │
│ Pending local 5 changes (from Obsidian via WebDAV) │
│ │
│ ┌───────────────────┐ ┌───────────────────┐ │
│ │ Get from Outline │ │ Send to Outline │ │
│ │ (pull) │ │ (5 pending) │ │
│ └───────────────────┘ └───────────────────┘ │
│ │
│ ⚠ 2 version conflicts need resolution [Resolve →] │
│ │
└─────────────────────────────────────────────────────────┘
```
"Pending local" count = `git diff outline..main --name-only | wc -l`
(these are files Obsidian wrote via WebDAV, not yet pushed to Outline)
### 9.2 Live Sync Output (inline SSE)
Triggered by either button. Live output streamed line by line via Server-Sent Events. HTMX replaces the button area with output panel.
```
┌─────────────────────────────────────────────────────────┐
│ Sending to Outline... │
│ ───────────────────────────────────────────────────── │
│ ✓ 5 local changes detected │
│ ✓ Projekte/NewNote.md → created (id: 4f2a...) │
│ ✓ Bewerbungen/CV.md → updated │
│ ✓ Infra/HomeLab.md → updated │
│ ✓ Infra/OldDoc.md → deleted (deletions=off, skip) │
│ ✓ Projekte/Renamed.md → title updated │
│ ───────────────────────────────────────────────────── │
│ Done. 3 updated, 1 created, 1 skipped. │
│ [Back to Dashboard] │
└─────────────────────────────────────────────────────────┘
```
### 9.3 Pending Changes (`/changes`)
Before pushing, shows what will happen.
```
┌─────────────────────────────────────────────────────────┐
│ Pending changes (5) [Send to Outline] │
├─────────────────────────────────────────────────────────┤
│ │
│ Modified │
│ ● Bewerbungen/CV.md [preview diff] │
│ ● Infra/HomeLab.md [preview diff] │
│ │
│ New — will be created in Outline │
│ + Projekte/NewNote.md │
│ + NewCollection/FirstDoc.md (new collection too) │
│ │
│ Renamed │
│ → Projekte/OldName.md → Projekte/NewName.md │
│ │
│ Deleted (skipped — deletions are off in settings) │
│ ✗ Infra/OldDoc.md │
│ │
└─────────────────────────────────────────────────────────┘
```
### 9.4 Conflict Resolution (`/conflicts`)
One card per conflict. Expand for side-by-side diff.
```
┌─────────────────────────────────────────────────────────┐
│ Version conflicts (2) │
│ Same document was edited in Obsidian and in Outline. │
├─────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ Bewerbungen/CV.md │ │
│ │ Your edit (via Obsidian): 2026-03-06 09:14 │ │
│ │ Outline's edit: 2026-03-06 11:03 │ │
│ │ │ │
│ │ ┌─────────────────┬─────────────────────────┐ │ │
│ │ │ Your version │ Outline's version │ │ │
│ │ │─────────────────│─────────────────────────│ │ │
│ │ │ # CV │ # CV │ │ │
│ │ │ │ │ │ │
│ │ │[+ New section] │ │ │ │
│ │ │ │[+ Contact info updated] │ │ │
│ │ └─────────────────┴─────────────────────────┘ │ │
│ │ │ │
│ │ [Keep mine] [Keep Outline's] │ │
│ └───────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────┘
```
After resolving, card collapses. If all resolved, redirect to dashboard showing "Push now available".
### 9.5 Sync History (`/history`)
`_sync_log.md` rendered as a table, most recent first.
---
## 10. API Endpoints (internal)
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/` | Dashboard |
| `GET` | `/status` | JSON vault state (counts, last sync, conflicts) |
| `GET` | `/changes` | Pending changes list |
| `GET` | `/conflicts` | Conflict list |
| `GET` | `/history` | Sync log view |
| `POST` | `/pull` | Start pull; streams SSE |
| `POST` | `/push` | Start push; streams SSE |
| `GET` | `/stream/{job_id}` | SSE stream for running job |
| `POST` | `/resolve` | Body: `{file, accept: local|remote}` |
| `GET` | `/diff/{encoded_path}` | HTML fragment: side-by-side diff |
---
## 11. Security
### WebDAV endpoint
- **Network isolation:** bound to Tailscale interface only — not reachable from public internet
- **Basic auth** (username + password from `.env`) — second layer of protection
- **Encryption:** Tailscale encrypts the tunnel end-to-end; plain HTTP between Obsidian and VPS is safe
- No Authentik forward auth — Obsidian plugin can't handle SSO redirects; Tailscale isolation makes this a non-issue
- No `vault.domverse.de` subdomain — no DNS exposure, no Traefik involvement
### Web UI
- Authentik forward auth via Traefik
- No additional app-level auth needed
- Token in `settings.json` never exposed to browser
### Subprocess safety
- Sync commands invoked with fixed argument lists
- File paths from `/resolve` validated against known conflict list before shell use
- `outline-tools/` mounted read-only in UI container
---
## 12. Implementation Phases
### Phase A — WebDAV Container (infrastructure)
Deploy `obsidian-webdav` container bound to the VPS Tailscale IP (`tailscale ip -4`). No Traefik config needed.
Test WebDAV access from local machine via Tailscale using a WebDAV client (e.g. `curl --user obsidian:$PASS http://100.x.x.x/`).
Configure `remotely-save` plugin in Obsidian with the Tailscale URL. Verify files sync bidirectionally.
**Done when:** editing a file in Obsidian and running sync → file appears on VPS.
---
### Phase B — Read-Only Dashboard
`webui.py`: FastAPI app. `GET /status` reads git status, parses into JSON. Dashboard template shows status badge, pending count, last sync times. No write operations yet.
**Done when:** `https://sync.domverse.de` shows current vault state.
---
### Phase C — Pull with Live Output
`POST /pull` spawns `outline_sync.py pull` as async subprocess. `GET /stream/{job_id}` streams stdout via SSE. HTMX wires button → POST → SSE panel → auto-refresh dashboard.
**Done when:** "Get from Outline" button works with live output; new Outline docs appear in Obsidian on next WebDAV sync.
---
### Phase D — Pending Changes View
`GET /changes`: parses `git diff outline..main --name-status` into structured list. Shows modified / new / renamed / deleted with explanatory labels. Inline diff preview for modified files.
**Done when:** changes page accurately shows what Obsidian wrote via WebDAV.
---
### Phase E — Push with Live Output
`POST /push` spawns `outline_sync.py push`. Same SSE pattern. Button disabled if conflicts exist. New-file creation flow (frontmatter written back → WebDAV serves updated file → Obsidian picks up IDs).
**Done when:** new Obsidian files appear in Outline; modified files update; frontmatter IDs land back in Obsidian vault.
---
### Phase F — Conflict Resolution
`GET /conflicts`: lists conflict files. `GET /diff/{path}`: renders side-by-side HTML diff using `difflib`. `POST /resolve`: calls `sync resolve`. Card UX with per-conflict actions.
**Done when:** all conflicts resolvable via browser; resolved files served cleanly by WebDAV to Obsidian.
---
### Phase G — History View
Render `_sync_log.md` as reverse-chronological HTML table.
---
## 13. Risks
| Risk | Mitigation |
|---|---|
| WebDAV sync and web UI push run simultaneously | Job lock in web UI: only one sync job at a time |
| `remotely-save` overwrites conflict markers in Obsidian | Pull always creates `.conflict.md` sidecar; original file left clean (conflict is in sidecar, not the working file) |
| WebDAV serving `.git/` directory | Configure nginx WebDAV to deny access to `.git/` path |
| New file created in Obsidian without a matching collection folder | Sync engine warns and skips; status page shows "unknown collection" |
| User edits `_sync_log.md` in Obsidian | `.gitattributes` union merge prevents conflicts; worst case: duplicate log entries |
| WebDAV password brute force | Endpoint bound to Tailscale interface only — not reachable from public internet |
| `remotely-save` syncs `.obsidian/` config files | Configure plugin ignore list: `.obsidian/`, `*.conflict.md` |
---
## 14. Open Questions
1. **Push trigger:** Manual-only (button) or also auto-trigger after WebDAV sync completes? Manual is safer; auto-push requires detecting that `remotely-save` finished syncing (no event available).
2. **WebDAV auth:** Basic auth (htpasswd) with Tailscale network isolation. Tailscale is the primary security boundary; basic auth is a fallback. Acceptable for single-user. ✅ Resolved.
3. **Cron pull:** Ofelia can run `sync pull --auto` hourly so Outline changes appear in Obsidian without clicking "Get from Outline". Recommended to enable.
4. **`.git/` in WebDAV:** The vault root contains `.git/`. The WebDAV server must not serve it (security + Obsidian confusion). Verify nginx config denies `/.git/` access.
---
## 15. Summary
| Component | Technology | Purpose |
|---|---|---|
| `100.x.x.x` (Tailscale) | nginx-WebDAV container | Obsidian ↔ VPS file sync (VPN-only) |
| `sync.domverse.de` | FastAPI + HTMX | Control plane: pull/push/resolve |
| `/outline-vault/` | git repo (shared volume) | Merge layer + history |
| `remotely-save` | Obsidian plugin | Local → WebDAV sync |
| `outline_sync.py` | existing sync engine | WebDAV vault ↔ Outline API |
| Ofelia | existing cron scheduler | Scheduled pull from Outline |