Sanitize codebase, reorganize docs, and add missing deploy files

Remove dead code identified in docs/SANITIZATION-REVIEW.md: - app/blueprints/content_old.py and app/blueprints/playlist.py - app/models/group.py, app/utils/nginx_config_reader.py - orphaned templates (content_list, edit_content, upload_content, player_page) and the related group/Template references Result: 6 blueprints, 82 routes, no dead modules or orphan templates. Add files that deploy.sh and docker-entrypoint.sh already require but which were never tracked: - https_manager.py (referenced by deploy.sh, migrate_network.sh, docker-entrypoint.sh) - Caddyfile.example (seeded by deploy.sh; its absence aborts deploy) Relocate generated Graphify artifacts from graphify-out/ to docs/graphify-out/ (110 files, no content change) and archive the superseded docs under docs/. Ignore hygiene: - ignore ad-hoc .env backups (.env.bak*) — they contain live secrets - keep the pre-sanitization snapshots (docs/legacy code/, docs/old_code_documentation/) on disk but out of the repo Fix .env.example: drop a duplicated config block, genericize the hardcoded host IP, and document HOSTNAME_INTERNAL.
2026-09-11 12:18:34 +03:00
parent 1c5186463a
commit 46602f1933
226 changed files with 3999 additions and 15737 deletions
@@ -63,40 +63,232 @@ flowchart LR

 ## 4. `docker-entrypoint.sh`

-1. Create `/app/instance` and `/app/app/static/uploads`.
-2. If `dashboard.db` is missing: create app + `db.create_all()`, then create/update the admin user from `ADMIN_USERNAME` / `ADMIN_PASSWORD`.
-3. Start **Gunicorn**: `--bind 0.0.0.0:5000 --workers 4 --timeout 120 app.app:create_app('production')`.
+1. **Pin the database**: export `DATABASE_URL` (default `sqlite:////app/instance/dashboard.db`) so migrations and the app resolve to the *same* file. Required because the config classes default to different files (`dev.db` vs `dashboard.db`) and most migration scripts call `create_app()` without an argument (→ development config), while the app runs `create_app('production')`.
+2. Create `/app/instance` and `/app/app/static/uploads`.
+3. Ensure schema + admin user — runs on **every** start (idempotent): `db.create_all()`, then create the admin from `ADMIN_USERNAME` / `ADMIN_PASSWORD` or refresh its password.
+4. Run the **migration chain** (idempotent, ordered — table-creating migrations run before those that alter them):
+
+```
+add_https_config_table.py
+add_player_user_table.py
+add_email_to_https_config.py
+migrate_player_user_global.py
+add_url_to_content.py
+add_original_filename_to_content.py
+add_deployment_fields_to_player.py
+```
+
+   Migrations are **non-fatal**: failures are logged as `⚠️ WARNING` and startup continues, so one bad migration can't strand the container in a restart loop.
+
+5. Start **Gunicorn**: `--bind 0.0.0.0:5000 --workers 4 --timeout 120 app.app:create_app('production')`.
+
+> Because migrations now run automatically on startup, `deploy.sh` step 4 is redundant (harmless — the scripts are idempotent).
+
+### Data layout (bind mounts)
+
+| Host path | Container path | Contents |
+|---|---|---|
+| `data/instance` | `/app/instance` | SQLite DB (`dashboard.db`), `player_build.json` |
+| `data/uploads` | `/app/app/static/uploads` | Media files + `edited_media/` |
+| `data/Caddyfile` | `/etc/caddy/Caddyfile` | Reverse-proxy config (**a file, not a directory**) |
+| `data/caddy-data` | `/data` | Caddy state (instance UUID, certificates) |
+| `data/caddy-config` | `/config` | Caddy autosave |
+| `data/caddy-logs` | `/var/log/caddy` | Access logs |
+
+> ⚠️ **`data/Caddyfile` must exist before `docker compose up`.** It is bind-mounted as a *file*;
+> if it is missing, Docker creates a **directory** in its place and Caddy fails to start.
+> `deploy.sh` seeds it from the version-controlled `Caddyfile.example`. For a manual start:
+> ```
+> mkdir -p data/instance data/uploads data/caddy-data data/caddy-config data/caddy-logs
+> cp Caddyfile.example data/Caddyfile
+> docker compose up -d --build
+> ```
+
+> ℹ️ The legacy `data/nginx-*` and `data/certbot` folders are **obsolete** — the reverse proxy is
+> Caddy. They are no longer created by `deploy.sh`.
+
+### Clean start (wipe all runtime data)
+
+`data/` is **gitignored**, so wiping it is irreversible. To reset to a pristine deployment:
+
+```
+docker compose down
+docker rmi digiserver-v2-digiserver-app:latest        # drop stale image
+docker image prune -f && docker builder prune -a -f   # reclaim build cache
+rm -rf data                                            # WIPES db, uploads, certs
+mkdir -p data/instance data/uploads data/caddy-data data/caddy-config data/caddy-logs
+cp Caddyfile.example data/Caddyfile
+./deploy.sh
+```
+
+> ⚠️ Files under `data/caddy-*` are created **root-owned** by the Caddy container, so a plain
+> `rm -rf data` may fail with *Permission denied*. Remove them via a helper container:
+> ```
+> docker run --rm -v "$PWD/data:/data" caddy:2-alpine sh -c 'rm -rf /data/caddy-config /data/caddy-data'
+> ```
+> Avoid `docker system prune --volumes` — this host also holds volumes for **other** projects.

 ---

 ## 5. `deploy.sh` (One-Shot Deployment)

 ```
-1. Validate compose + project; create data/ subdirs; copy nginx configs
-2. docker compose up -d + verify containers "Up"
-3. Run migration scripts (add_https_config_table, add_player_user_table,
+1. Detect compose: `docker compose` (plugin) or `docker-compose` (v1 fallback)
+   — stored in $COMPOSE and used for every subsequent call
+2. Create data/ subdirs (instance, uploads, caddy-data, caddy-config, caddy-logs)
+   and seed data/Caddyfile from Caddyfile.example
+3. $COMPOSE up -d + verify containers "Up"
+4. Run migration scripts (add_https_config_table, add_player_user_table,
   add_email_to_https_config, migrate_player_user_global,
   add_original_filename_to_content)
-4. Run /app/https_manager.py enable <hostname> <domain> <email> <ip> <port>
-   ⚠  https_manager.py is NOT in the current repo — this step needs attention
-5. Verify DB tables via SQLAlchemy inspector
-6. caddy validate + https_manager.py status; print access URLs + default creds
+   ↳ NOTE: the container entrypoint already applies all seven on startup.
+     This step is idempotent and therefore redundant.
+5. /app/https_manager.py enable <hostname> <domain> <email> <ip> <port>
+   ↳ Exit code 2 ("config applied, Caddy not reloaded") is non-fatal.
+6. Verify DB tables via SQLAlchemy inspector; caddy validate;
+   https_manager.py status; print access URLs + default creds
 ```

+### Configuration variables
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `HOSTNAME` | `digiserver` | Display hostname |
+| `HTTPS_MODE` | `internal` | `internal` \| `acme` \| `off` |
+| `DOMAIN` | *(empty)* | Required only when `HTTPS_MODE=acme` |
+| `IP_ADDRESS` | **auto-detected** | Primary LAN IP (override if needed) |
+| `EMAIL` | `admin@example.com` | ACME account email (unused by internal CA) |
+| `PORT` | `8443` | Externally published HTTPS port |
+
+> If `IP_ADDRESS` is unset, `deploy.sh` auto-detects it
+> (`ip -4 route get 1.1.1.1` → `src` address, falling back to `hostname -I`).
+> The old hard-coded defaults (`10.76.152.164`, a `.intra` domain) were wrong for
+> most hosts and have been removed.
+
 ---

 ## 6. HTTPS Setup (Caddy)

-HTTPS is configured through the **Admin → HTTPS Configuration** page, which:
+Three equivalent entry points drive the **same** code path
+(`HTTPSConfig` + `CaddyConfigGenerator`) so CLI, env bootstrap and UI cannot diverge:
+
+1. **Env bootstrap (deploy time)** — the container entrypoint runs
+   `python /app/https_manager.py bootstrap`, which reads `HOSTNAME_INTERNAL` and
+   `HOST_IP` from the environment and configures Caddy for HTTPS automatically.
+2. **CLI** — `python /app/https_manager.py enable … | verify | status | disable`
+3. **UI** — *Admin → HTTPS Configuration* (reloads Caddy live; the ongoing source of truth)
+
+### Addressing model — one HTTP endpoint, one HTTPS endpoint
+
+| Port | What it does |
+|---|---|
+| **80** | Always answers. A catch-all `:80` block serves **any** Host header, plus explicit blocks for the IP and hostname so both work. |
+| **443** | HTTPS for the same names, using the internal CA (or ACME for a public domain). |
+
+If HTTPS is disabled or never configured, port 80 simply serves the app — there is
+no separate "HTTP mode" to set.
+
+> ⚠️ **`default_sni` is required for IP access.** Browsers send **no SNI** when the
+> URL is an IP address (an IP is not a valid SNI hostname). Caddy then identifies the
+> connection by the container's own internal IP and aborts the handshake with
+> `no certificate available for '<container-ip>'`. To prevent this, the generator emits
+> `default_sni <ip>` whenever internal-CA mode is used, so `https://<ip>` works in a
+> plain browser. This was found by end-to-end testing — see
+> `docs/tools/test_http_https_runtime.sh`.
+
+### Mode selection
+
+| Condition | Result |
+|---|---|
+| HTTPS off, or no IP/domain | Plain HTTP on port 80 |
+| HTTPS on + `ip_address` / `hostname` | `tls internal` per name (no DNS, no ACME) |
+| HTTPS on + `domain` set | Let's Encrypt for that name |
+
+### Automatic fallback if HTTPS does not work
+
+After applying a config, `https_manager.py` probes the HTTPS endpoint
+(`/api/health`, certificate validation deliberately disabled). If the TLS listener
+does not come up, the configuration is **automatically reverted to plain HTTP** so a
+failed certificate can never make the site unreachable:
+
+```
+enable HTTPS → reload Caddy → probe https://<host>:<port>/api/health
+                               ├─ OK   → keep HTTPS
+                               └─ FAIL → revert to HTTP-only, log a warning
+```
+
+Disable the probe with `HTTPS_VERIFY=false` (or `enable --no-verify`).
+Re-check at any time with `python /app/https_manager.py verify`.
+
+### Deploy-time bootstrap via `.env`
+
+Copy `.env.example` → `.env` and set the host address. `docker-compose.yml`
+forwards these to the app container:
+
+| Variable | Effect |
+|---|---|
+| `HOSTNAME_INTERNAL` | Hostname served (both HTTP and HTTPS) |
+| `HOST_IP` | IP served and certified |
+| `DOMAIN` | **Leave empty for an intranet name** → internal CA. Set only for Let's Encrypt |
+| `SSL_EMAIL` | ACME contact (ignored by internal CA) |
+| `HTTP_PORT` / `HTTPS_PORT` | Host ports mapped to Caddy's 80/443 (default `80`/`443`) |
+| `HTTPS_HTTP_FALLBACK` | `true` (default) also serves plain HTTP; `false` redirects instead |
+| `HTTPS_VERIFY` | `true` (default) probe HTTPS and auto-fall back on failure |
+
+```bash
+cp .env.example .env
+# set HOSTNAME_INTERNAL and HOST_IP
+docker compose up -d --build
+```
+
+> **If either `HOSTNAME_INTERNAL` or `HOST_IP` is missing the bootstrap is a no-op** —
+> the app starts on plain HTTP and stays reachable. HTTPS can then be enabled from
+> *Admin → HTTPS Configuration*, which regenerates the Caddyfile and reloads Caddy
+> **without a restart**.
+
+### Who owns the config (env vs admin UI)
+
+The admin UI is the **ongoing source of truth**. The bootstrap runs on every container
+start, so it guards against silently overwriting an admin's change by tracking
+provenance in `HTTPSConfig.updated_by`:
+
+| Current config | Bootstrap behaviour |
+|---|---|
+| *(none — first deploy)* | Apply from env ✅ |
+| Written by env/CLI (`updated_by='deploy.sh'`) | Apply from env ✅ — so editing `HOST_IP` and redeploying works |
+| Written by a user (`updated_by='<username>'`) | **Skip** — the admin's setting is preserved |
+
+So an admin change made in the UI survives restarts even while the env vars remain set.
+
+### Why internal CA (and not Let's Encrypt) for `.intra`
+
+An intranet name such as `digiserver.sibiusb.harting.intra` is **not resolvable from the
+public internet**, so Let's Encrypt's HTTP-01/TLS-ALPN challenge cannot succeed. Setting
+`DOMAIN=` empty makes Caddy sign the certificate itself with its local CA — no external
+dependency at all.
+
+**Trust caveat:** the internal CA is not in any client trust store, so browsers show a
+warning and a Kivy player with `verify_ssl: true` will **reject the connection**. Options:
+
+1. **Use HTTP** — port 80 always serves the app, so players need no trust configuration.
+2. **Install the root CA** on each device:
+   ```
+   docker compose cp caddy:/data/caddy/pki/authorities/local/root.crt ./caddy-root.crt
+   ```
+
+### Ports
+
+| Host → Container | Purpose |
+|---|---|
+| `80 → 80` | HTTP (always available) |
+| `443 → 443` | HTTPS |
+| `5000 → 5000` | Direct app access (bypasses Caddy; dev/testing) |
+
+> Ports are configurable via `HTTP_PORT`/`HTTPS_PORT` so the stack also works where
+> 80/443 are already taken (e.g. `HTTP_PORT=8080 HTTPS_PORT=8443`).
+

-1. Saves `HTTPSConfig` (hostname, domain, IP, email, port, enabled).
-2. `CaddyConfigGenerator.generate_caddyfile(config)` picks a template:
-   - **HTTP-only** → `:80` reverse proxy
-   - **Domain** → automatic Let's Encrypt
-   - **IP** → internal CA self-signed
-3. Writes `/etc/caddy/Caddyfile` and reloads Caddy via `POST http://caddy:2019/load`.

-The current `data/Caddyfile` (HTTP mode) includes: admin API on `0.0.0.0:2019`, `:80` block → `digiserver-app:5000`, 2 GB body limit, gzip, security headers, access log.

 ---

@@ -105,17 +297,19 @@ The current `data/Caddyfile` (HTTP mode) includes: admin API on `0.0.0.0:2019`,
 Sections checked (pass/fail/warn counters):
 - git status
 - `.env` / `.env.example`
- Docker + Compose versions + `compose config` syntax
+- Docker + Compose versions (plugin **or** v1) + `compose config` syntax
 - Dockerfile best practices (HEALTHCHECK, non-root, slim base)
 - `requirements.txt` critical packages + versions
 - migrations directory
- **SSL cert expiry** (openssl)
+- **TLS certificate** — Caddy internal CA expiry (`data/caddy-data/caddy/pki/authorities/local/root.crt`)
 - Flask config (`ProductionConfig`, `SESSION_COOKIE_SECURE`)
- nginx.conf checks
- runtime container health
+- `data/Caddyfile` checks (reverse_proxy, admin API, TLS mode)
+- runtime container health + live HTTP/HTTPS endpoint probes
 - security best practices

-> ⚠ Note: the script still references `docker-compose` (v1) and `digiserver-nginx` — the current stack uses Compose v2 + Caddy.
+> The script now detects `docker compose` (plugin) or `docker-compose` (v1) and warns when
+> buildx is too old for compose v1 builds — matching `deploy.sh`, which then falls back to
+> `docker build`.

 ---