domverse/outline-sync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a1ec0e482bd80bbd8b6447b043595b299c7cff4d
outline-sync/WEBUI_PRD.md
Claude b3137a8af3 Add sync engine, web UI, Docker setup, and tests 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 20:54:59 +01:00

26 KiB
Raw Blame History

PRD: Outline Sync Web UI

Version: 2.0 Date: 2026-03-06 Status: Draft — WebDAV Architecture Depends on: SYNC_PRD.md (sync engine)

1. Problem Statement

The sync engine (sync.sh) works but requires terminal access on the VPS. The user edits notes locally in Obsidian on their own machine. Changes need to flow bidirectionally between local Obsidian and the remote Outline wiki without terminal interaction.

The key constraint: Obsidian runs locally, the vault git repo lives on the VPS.

2. Architecture Options Evaluated

Option Mechanism Verdict
WebDAV + Remotely Save WebDAV server on VPS serves vault dir; Obsidian plugin syncs automatically ✅ Recommended
Self-hosted LiveSync CouchDB on VPS; Obsidian plugin syncs in real-time ❌ Adds CouchDB; no direct file access for sync engine
Local REST API Obsidian exposes REST on localhost:27124 ❌ VPS can't reach local machine
Manual zip download/upload Browser download/upload of vault archives ❌ Eliminated by WebDAV option

Decision: WebDAV. A WebDAV Docker container on the VPS serves the vault directory directly. The Obsidian plugin remotely-save syncs to it automatically. No tunnel, no extra database, no build step.

3. System Architecture

┌─────────────────────────────────────────────────────────┐
│  Your local machine                                     │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Obsidian                                       │   │
│  │  Plugin: remotely-save                          │   │
│  │  Auto-sync every N minutes (or on open/close)   │   │
│  └──────────────────────┬──────────────────────────┘   │
└─────────────────────────│───────────────────────────────┘
                          │ WebDAV over Tailscale (encrypted tunnel)
                          │ (basic auth + Tailscale network isolation)
┌─────────────────────────▼───────────────────────────────┐
│  VPS (domverse.de)                                      │
│                                                         │
│  ┌──────────────────────────────────────────────────┐  │
│  │  WebDAV container  (obsidian-webdav)             │  │
│  │  Serves: /outline-vault/  (read-write)           │  │
│  │  URL: http://100.x.x.x  (Tailscale only)        │  │
│  └────────────────────┬─────────────────────────────┘  │
│                       │ shared volume                   │
│  ┌────────────────────▼─────────────────────────────┐  │
│  │  /outline-vault/   (git repo)                    │  │
│  │  ├── .git/                                       │  │
│  │  │   ├── outline branch  (last Outline state)    │  │
│  │  │   └── main branch     (current vault state)   │  │
│  │  ├── Bewerbungen/                                │  │
│  │  ├── Projekte/                                   │  │
│  │  └── _sync_log.md                               │  │
│  └────────────────────┬─────────────────────────────┘  │
│                       │ shared volume                   │
│  ┌────────────────────▼─────────────────────────────┐  │
│  │  outline-sync-ui  (FastAPI, port 8080)            │  │
│  │  URL: https://sync.domverse.de/                  │  │
│  │  Auth: Authentik                                 │  │
│  └────────────────────┬─────────────────────────────┘  │
│                       │                                 │
│  ┌────────────────────▼─────────────────────────────┐  │
│  │  outline_sync.py  (sync engine)                  │  │
│  │  network: domnet                                 │  │
│  │  API: http://outline:3000                        │  │
│  └──────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

Data Flow: Obsidian → Outline

  1. User edits notes locally in Obsidian
  2. remotely-save plugin syncs changed files to WebDAV server on VPS (automatic)
  3. WebDAV writes files directly into /outline-vault/
  4. User clicks "Send to Outline" in web UI (or cron runs push)
  5. Sync engine: git diff outline..main → calls Outline API per changed file
  6. Sync engine advances outline branch

Data Flow: Outline → Obsidian

  1. User clicks "Get from Outline" in web UI (or cron runs pull)
  2. Sync engine exports Outline → commits to outline branch → merges into main
  3. Files updated in /outline-vault/
  4. WebDAV immediately serves updated files
  5. remotely-save plugin picks up changes on next sync interval

4. User-Facing Mental Model

Two systems, one bridge:

What the user sees What actually happens
Obsidian auto-syncs remotely-save talks WebDAV to VPS
"Get from Outline" sync engine pulls Outline API → git merge → files update in vault
"Send to Outline" sync engine diffs vault → calls Outline API
"Version conflict" git merge produced conflict markers
"Keep mine / Keep Outline's" sync resolve --accept local/remote

Git is invisible. WebDAV is invisible. Obsidian just sees files that stay in sync.

5. Goals

# Goal
G1 Obsidian syncs to VPS automatically — no manual file transfer
G2 Pulling from Outline and pushing to Outline are single button clicks in a browser
G3 Live output visible while sync runs
G4 Conflicts resolvable in browser with side-by-side diff
G5 New files created in Obsidian appear in Outline after push
G6 New documents created in Outline appear in Obsidian after next WebDAV sync
G7 Web UI protected by Authentik SSO
G8 WebDAV endpoint protected by authentication
G9 Zero terminal interaction for normal workflow

6. Non-Goals

# Non-Goal Reason
NG1 In-browser markdown editor Obsidian is the editor
NG2 Real-time Outline → Obsidian sync WebDAV poll interval (e.g. 5 min) is sufficient
NG3 Syncing .obsidian/ config, templates, daily notes Outline-synced content only
NG4 Multi-user Single-user system
NG5 Mobile Obsidian support Remotely Save does support mobile, but not a primary target

6. Component Inventory

6.1 WebDAV Server (new container)

Simple nginx-WebDAV container. Serves /outline-vault/ as a WebDAV share. Not exposed via Traefik. Bound exclusively to the VPS Tailscale interface — invisible from the public internet.

obsidian-webdav:
  image: dgraziotin/nginx-webdav-nononsense:latest
  container_name: obsidian-webdav
  networks:
    - domnet
  ports:
    - "100.x.x.x:80:80"   # Tailscale IP only — replace with: tailscale ip -4
  volumes:
    - /home/crabix/docker_authentik/outline-vault:/data
  environment:
    - WEBDAV_USERNAME=obsidian
    - WEBDAV_PASSWORD=${WEBDAV_PASSWORD}   # from .env
  restart: unless-stopped
  # No Traefik labels — not publicly routed

Auth note: WebDAV cannot use Authentik forward auth (Obsidian plugin can't handle SSO redirect). Network isolation via Tailscale is the primary security layer — the endpoint is unreachable from the public internet. Basic auth (htpasswd) via nginx provides a second layer. Tailscale encrypts the tunnel, so plain HTTP is safe for this leg.

6.2 Web UI (new container)

FastAPI + HTMX. Control plane for sync operations.

outline-sync-ui:
  image: python:3.11-slim
  container_name: outline-sync-ui
  networks:
    - domnet
  volumes:
    - /home/crabix/docker_authentik/outline-tools:/app:ro    # scripts, read-only
    - /home/crabix/docker_authentik/outline-vault:/vault     # vault, read-write
  working_dir: /app
  command: >
    bash -c "pip install -qqq fastapi uvicorn requests &&
             uvicorn webui:app --host 0.0.0.0 --port 8080"
  restart: unless-stopped
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.outline-sync.rule=Host(`sync.domverse.de`)"
    - "traefik.http.routers.outline-sync.entrypoints=https"
    - "traefik.http.routers.outline-sync.tls=true"
    - "traefik.http.routers.outline-sync.middlewares=secHeaders@file,middlewares-authentik-new@file"
    - "traefik.http.services.outline-sync.loadbalancer.server.port=8080"

6.3 Obsidian Setup (one-time, on local machine)

  1. Install plugin remotely-save from Obsidian community plugins
  2. Ensure Tailscale is running on the local machine and connected to the VPS
  3. Configure:
    • Remote service: WebDAV
    • Server URL: http://100.x.x.x (VPS Tailscale IP — plain HTTP, tunnel is encrypted)
    • Username / Password: WebDAV credentials
    • Sync on startup: yes
    • Sync interval: every 5 minutes (or on vault open/close)
    • Sync direction: bidirectional (default)
  4. Exclude .git/ from sync (configure in plugin's ignore list)

7. New File Handling

This is the key correctness concern: when the user creates a new .md file in Obsidian, it must reach Outline correctly.

Flow for new files

  1. User creates Projekte/NewNote.md in Obsidian (no frontmatter)
  2. remotely-save syncs it to VPS via WebDAV → file appears in /outline-vault/Projekte/NewNote.md
  3. User clicks "Send to Outline" in web UI
  4. Sync engine: git diff outline..main shows A Projekte/NewNote.md
  5. Sync engine determines parent: Projekte/ folder exists → look up Projekte.md for its outline_id → use as parentDocumentId
  6. documents.create called → new document created in Outline under correct collection/parent
  7. outline_id written back into frontmatter of NewNote.md
  8. File committed → WebDAV serves updated file with frontmatter → Obsidian picks it up on next sync

New file in new folder

  1. User creates NewCollection/FirstDoc.md
  2. Sync engine: top-level folder NewCollection/ not in any known collection → collections.create("NewCollection")
  3. FirstDoc created inside new collection
  4. Frontmatter written back to both files

Ordering guarantee

Sync engine creates documents in topological order (parents before children), regardless of which order files were synced via WebDAV.

8. Conflict Model

A conflict occurs when:

  • User edited File.md in Obsidian (synced via WebDAV to main branch)
  • Someone also edited the same document in Outline
  • sync pull → git merge outline → conflict markers in File.md

After conflict:

  • File.md contains <<<<<<< markers → WebDAV immediately serves this broken file → Obsidian shows it with markers
  • Conflict is resolved in the web UI (not Obsidian)
  • After resolution, WebDAV serves the clean file → Obsidian's next sync picks it up

This is the key reason the web UI exists: conflict resolution in a browser with a diff view is far better than editing git conflict markers in a text editor.

9. Web UI Screens

9.1 Dashboard (/)

┌─────────────────────────────────────────────────────────┐
│  Outline Sync                         sync.domverse.de  │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Vault status    ● Clean                                │
│  Last pull       2026-03-06 14:32  (from Outline)       │
│  Last push       2026-03-05 09:10  (to Outline)         │
│  Pending local   5 changes (from Obsidian via WebDAV)   │
│                                                         │
│  ┌───────────────────┐   ┌───────────────────┐         │
│  │  Get from Outline │   │  Send to Outline  │         │
│  │  (pull)           │   │  (5 pending)      │         │
│  └───────────────────┘   └───────────────────┘         │
│                                                         │
│  ⚠ 2 version conflicts need resolution  [Resolve →]    │
│                                                         │
└─────────────────────────────────────────────────────────┘

"Pending local" count = git diff outline..main --name-only | wc -l (these are files Obsidian wrote via WebDAV, not yet pushed to Outline)

9.2 Live Sync Output (inline SSE)

Triggered by either button. Live output streamed line by line via Server-Sent Events. HTMX replaces the button area with output panel.

┌─────────────────────────────────────────────────────────┐
│  Sending to Outline...                                  │
│  ─────────────────────────────────────────────────────  │
│  ✓  5 local changes detected                            │
│  ✓  Projekte/NewNote.md → created (id: 4f2a...)         │
│  ✓  Bewerbungen/CV.md   → updated                       │
│  ✓  Infra/HomeLab.md    → updated                       │
│  ✓  Infra/OldDoc.md     → deleted (deletions=off, skip) │
│  ✓  Projekte/Renamed.md → title updated                 │
│  ─────────────────────────────────────────────────────  │
│  Done. 3 updated, 1 created, 1 skipped.                 │
│                                [Back to Dashboard]      │
└─────────────────────────────────────────────────────────┘

9.3 Pending Changes (/changes)

Before pushing, shows what will happen.

┌─────────────────────────────────────────────────────────┐
│  Pending changes (5)                  [Send to Outline] │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Modified                                               │
│  ● Bewerbungen/CV.md              [preview diff]        │
│  ● Infra/HomeLab.md               [preview diff]        │
│                                                         │
│  New — will be created in Outline                       │
│  + Projekte/NewNote.md                                  │
│  + NewCollection/FirstDoc.md      (new collection too)  │
│                                                         │
│  Renamed                                                │
│  → Projekte/OldName.md → Projekte/NewName.md           │
│                                                         │
│  Deleted (skipped — deletions are off in settings)      │
│  ✗ Infra/OldDoc.md                                      │
│                                                         │
└─────────────────────────────────────────────────────────┘

9.4 Conflict Resolution (/conflicts)

One card per conflict. Expand for side-by-side diff.

┌─────────────────────────────────────────────────────────┐
│  Version conflicts (2)                                  │
│  Same document was edited in Obsidian and in Outline.   │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  ┌───────────────────────────────────────────────────┐  │
│  │  Bewerbungen/CV.md                                │  │
│  │  Your edit (via Obsidian): 2026-03-06 09:14       │  │
│  │  Outline's edit:           2026-03-06 11:03       │  │
│  │                                                   │  │
│  │  ┌─────────────────┬─────────────────────────┐   │  │
│  │  │ Your version    │ Outline's version        │   │  │
│  │  │─────────────────│─────────────────────────│   │  │
│  │  │ # CV            │ # CV                    │   │  │
│  │  │                 │                         │   │  │
│  │  │[+ New section]  │                         │   │  │
│  │  │                 │[+ Contact info updated] │   │  │
│  │  └─────────────────┴─────────────────────────┘   │  │
│  │                                                   │  │
│  │  [Keep mine]                  [Keep Outline's]    │  │
│  └───────────────────────────────────────────────────┘  │
│                                                         │
└─────────────────────────────────────────────────────────┘

After resolving, card collapses. If all resolved, redirect to dashboard showing "Push now available".

9.5 Sync History (/history)

_sync_log.md rendered as a table, most recent first.

10. API Endpoints (internal)

Method Path Purpose
GET / Dashboard
GET /status JSON vault state (counts, last sync, conflicts)
GET /changes Pending changes list
GET /conflicts Conflict list
GET /history Sync log view
POST /pull Start pull; streams SSE
POST /push Start push; streams SSE
GET /stream/{job_id} SSE stream for running job
POST /resolve Body: `{file, accept: local
GET /diff/{encoded_path} HTML fragment: side-by-side diff

11. Security

WebDAV endpoint

  • Network isolation: bound to Tailscale interface only — not reachable from public internet
  • Basic auth (username + password from .env) — second layer of protection
  • Encryption: Tailscale encrypts the tunnel end-to-end; plain HTTP between Obsidian and VPS is safe
  • No Authentik forward auth — Obsidian plugin can't handle SSO redirects; Tailscale isolation makes this a non-issue
  • No vault.domverse.de subdomain — no DNS exposure, no Traefik involvement

Web UI

  • Authentik forward auth via Traefik
  • No additional app-level auth needed
  • Token in settings.json never exposed to browser

Subprocess safety

  • Sync commands invoked with fixed argument lists
  • File paths from /resolve validated against known conflict list before shell use
  • outline-tools/ mounted read-only in UI container

12. Implementation Phases

Phase A — WebDAV Container (infrastructure)

Deploy obsidian-webdav container bound to the VPS Tailscale IP (tailscale ip -4). No Traefik config needed. Test WebDAV access from local machine via Tailscale using a WebDAV client (e.g. curl --user obsidian:$PASS http://100.x.x.x/). Configure remotely-save plugin in Obsidian with the Tailscale URL. Verify files sync bidirectionally.

Done when: editing a file in Obsidian and running sync → file appears on VPS.

Phase B — Read-Only Dashboard

webui.py: FastAPI app. GET /status reads git status, parses into JSON. Dashboard template shows status badge, pending count, last sync times. No write operations yet.

Done when: https://sync.domverse.de shows current vault state.

Phase C — Pull with Live Output

POST /pull spawns outline_sync.py pull as async subprocess. GET /stream/{job_id} streams stdout via SSE. HTMX wires button → POST → SSE panel → auto-refresh dashboard.

Done when: "Get from Outline" button works with live output; new Outline docs appear in Obsidian on next WebDAV sync.

Phase D — Pending Changes View

GET /changes: parses git diff outline..main --name-status into structured list. Shows modified / new / renamed / deleted with explanatory labels. Inline diff preview for modified files.

Done when: changes page accurately shows what Obsidian wrote via WebDAV.

Phase E — Push with Live Output

POST /push spawns outline_sync.py push. Same SSE pattern. Button disabled if conflicts exist. New-file creation flow (frontmatter written back → WebDAV serves updated file → Obsidian picks up IDs).

Done when: new Obsidian files appear in Outline; modified files update; frontmatter IDs land back in Obsidian vault.

Phase F — Conflict Resolution

GET /conflicts: lists conflict files. GET /diff/{path}: renders side-by-side HTML diff using difflib. POST /resolve: calls sync resolve. Card UX with per-conflict actions.

Done when: all conflicts resolvable via browser; resolved files served cleanly by WebDAV to Obsidian.

Phase G — History View

Render _sync_log.md as reverse-chronological HTML table.

13. Risks

Risk Mitigation
WebDAV sync and web UI push run simultaneously Job lock in web UI: only one sync job at a time
remotely-save overwrites conflict markers in Obsidian Pull always creates .conflict.md sidecar; original file left clean (conflict is in sidecar, not the working file)
WebDAV serving .git/ directory Configure nginx WebDAV to deny access to .git/ path
New file created in Obsidian without a matching collection folder Sync engine warns and skips; status page shows "unknown collection"
User edits _sync_log.md in Obsidian .gitattributes union merge prevents conflicts; worst case: duplicate log entries
WebDAV password brute force Endpoint bound to Tailscale interface only — not reachable from public internet
remotely-save syncs .obsidian/ config files Configure plugin ignore list: .obsidian/, *.conflict.md

14. Open Questions

  1. Push trigger: Manual-only (button) or also auto-trigger after WebDAV sync completes? Manual is safer; auto-push requires detecting that remotely-save finished syncing (no event available).

  2. WebDAV auth: Basic auth (htpasswd) with Tailscale network isolation. Tailscale is the primary security boundary; basic auth is a fallback. Acceptable for single-user. ✅ Resolved.

  3. Cron pull: Ofelia can run sync pull --auto hourly so Outline changes appear in Obsidian without clicking "Get from Outline". Recommended to enable.

  4. .git/ in WebDAV: The vault root contains .git/. The WebDAV server must not serve it (security + Obsidian confusion). Verify nginx config denies /.git/ access.

15. Summary

Component Technology Purpose
100.x.x.x (Tailscale) nginx-WebDAV container Obsidian ↔ VPS file sync (VPN-only)
sync.domverse.de FastAPI + HTMX Control plane: pull/push/resolve
/outline-vault/ git repo (shared volume) Merge layer + history
remotely-save Obsidian plugin Local → WebDAV sync
outline_sync.py existing sync engine WebDAV vault ↔ Outline API
Ofelia existing cron scheduler Scheduled pull from Outline
Reference in New Issue View Git Blame Copy Permalink