updates to digiserver app server

This commit is contained in:
2026-09-10 21:03:16 +03:00
parent 046f5e5efd
commit 1c5186463a
129 changed files with 33014 additions and 17 deletions
+198
View File
@@ -0,0 +1,198 @@
# 01 · System Architecture
> Generated from the Graphify knowledge graph (41 communities) and source analysis of `digiserver-v2`.
---
## 1. System Overview
DigiServer v2 is a Flask application serving **three audiences**:
1. **Humans (admins)** — manage content, playlists, players, users, HTTPS via the web UI.
2. **Players (physical signage)** — poll the REST API for playlists, send status feedback, upload edited media.
3. **Operators** — deploy/update player software remotely over SSH.
```mermaid
flowchart LR
subgraph Browser["Web UI (Admins)"]
A[Login]
B[Dashboard / Admin / Content / Players]
end
subgraph App["digiserver-app (Flask :5000)"]
C[Gunicorn x4]
D[7 Blueprints]
E[SQLAlchemy + SQLite]
F[Cache / Background tasks]
end
subgraph Players["Signage Players"]
G[Player 1]
H[Player N]
end
subgraph Proxy["Reverse Proxy"]
I[Caddy :80/:443]
J[Umbrella nginx gateway - optional SSO]
end
I -->|reverse_proxy digiserver-app:5000| C
J -.X-Auth-Username .-> C
D --> E
G <-->|"/api/*"| D
H <-->|"/api/*"| D
Browser --> I
C -.SSH deploy/rsync.-> G
C -.SSH deploy/rsync.-> H
```
---
## 2. Layered Architecture
Graphify assigns each node a **level** (0 = entry/global → 3 = utility):
| Layer | Level | Components |
|---|---|---|
| **L0 — Entry / Global** | 0 | `create_app()`, `app.py`, config classes, error handlers, CLI commands |
| **L1 — Strategic / Core** | 1 | Blueprint route handlers (players, content, api, admin), model classes |
| **L2 — Implementation** | 2 | Playlist/group management helpers, processing helpers, model methods |
| **L3 — Utility** | 3 | `logger.py`, `ssh_deploy.py`, `caddy_manager.py`, `uploads.py`, `pptx_converter.py`, migrations |
```mermaid
flowchart TD
subgraph L0["L0 · Entry"]
create_app["create_app()"]
ProxyFix
ScriptNameFix
portal_sso["init_portal_sso()"]
end
subgraph L1["L1 · Blueprints + Models"]
bp["main · auth · admin · players · content · playlist · api"]
models["User Player Content Playlist PlayerEdit PlayerFeedback PlayerUser HTTPSConfig ServerLog"]
end
subgraph L2["L2 · Implementation"]
bg["run_background_task()"]
dep["deploy_player_to_host()"]
caddy["CaddyConfigGenerator"]
pb["build_player_files() / write_base_config()"]
end
subgraph L3["L3 · Utility"]
log["log_action()"]
up["uploads.py"]
pptx["pptx_converter.py"]
nginx["NginxConfigReader"]
end
create_app --> bp
create_app --> models
bp --> log
bp --> bg
bp --> dep
bp --> caddy
bp --> pb
bp --> up
bp --> pptx
```
---
## 3. Component Map (files → communities)
Graphify clustered the code into **41 communities**. The 12 meaningful ones are shown below (communities 1340 are mostly migrations, isolated helper functions, and archived code).
| Community | Domain (derived) | Files | Role |
|---|---|---|---|
| **C0** (90) | **Authentication + legacy groups** | `blueprints/auth.py`, `blueprints/content_old.py`, `models/server_log.py`, `utils/logger.py`, `utils/group_player_management.py`, `old_code_documentation/blueprint_groups.py` | Auth flows + audit logging + (legacy) group features |
| **C1** (80) | **Admin + HTTPS + player users** | `blueprints/admin.py`, `models/https_config.py`, `models/player_user.py`, `utils/caddy_manager.py`, `migrations/add_player_user_table.py` | Admin panel, Caddy HTTPS generation, editing-user registry |
| **C2** (68) | **Playlist & content workflows** | `blueprints/content.py`, `blueprints/playlist.py`, `models/playlist.py` | Modern playlist-centric content management + legacy redirects |
| **C3** (62) | **Player API + edits** | `blueprints/api.py`, `models/player_edit.py` | Player-facing REST, edited-media pipeline |
| **C4** (55) | **Application core** | `app.py`, `config.py`, `models/user.py`, `utils/portal_sso.py`, `utils/script_name_fix.py` | App factory, config, auth identity, middleware |
| **C5** (54) | **Content & player models** | `models/content.py`, `models/player.py`, `models/player_feedback.py`, `blueprints/main.py` | Media model, player model, feedback, dashboard |
| **C6** (51) | **Player management UI** | `blueprints/players.py`, `migrations/add_https_config_table.py` | Player CRUD, manage page, deployment polling |
| **C7** (42) | **Groups (legacy)** | `models/group.py`, `utils/nginx_config_reader.py` | Archived groups feature + legacy nginx reader |
| **C8** (24) | **Dev tooling** | `old_code_documentation/test_edit_media_api.py`, `Colors`, integrity checker | Test/analysis utilities |
| **C9** (21) | **Upload processing** | `utils/uploads.py` | Upload progress, video/image processing |
| **C10** (20) | **Deployment** | `utils/background_tasks.py`, `utils/ssh_deploy.py` | Background SSH deployment engine |
| **C11** (19) | **Player build/staging** | `utils/player_build.py` | Stage player source code on server |
| **C12** (8) | **PPTX conversion** | `utils/pptx_converter.py` | LibreOffice PPTX → PDF → PNG |
```mermaid
flowchart TD
C0["C0 · Auth & logging"]
C1["C1 · Admin & HTTPS"]
C2["C2 · Playlists & content"]
C3["C3 · Player API & edits"]
C4["C4 · App core"]
C5["C5 · Content/player models"]
C6["C6 · Player management UI"]
C9["C9 · Upload processing"]
C10["C10 · Deployment"]
C11["C11 · Player build"]
C12["C12 · PPTX conversion"]
C1 -->|27 edges| C3
C2 -->|25 edges| C3
C2 -->|25 edges| C0
C3 -->|25 edges| C1
C1 -->|21 edges| C0
C6 -->|16 edges| C1
C6 -->|16 edges| C3
C3 -->|14 edges| C0
C6 -->|14 edges| C0
C5 -->|9 edges| C3
C9 -->|5 edges| C0
C1 -->|3 edges| C11
C10 --> C6
C11 --> C10
C12 --> C9
```
*(edge weights from `graphify-out/metadata.json` communityLinks)*
---
## 4. Middleware & Request Pipeline
Every request flows through three layers before reaching a blueprint:
```mermaid
flowchart LR
R["Raw request"] --> PF["werkzeug ProxyFix"]
PF --> SF["ScriptNameFix"]
SF --> SSO["portal_sso (before_request)"]
SSO --> BP["Blueprint route"]
BP --> DB[("SQLAlchemy / SQLite")]
```
1. **ProxyFix** — trusts `X-Forwarded-For/Proto/Host/Port` from Caddy/nginx (1 hop).
2. **ScriptNameFix** — reads `X-Script-Name` (e.g. `/digiserver`) so `url_for()` produces correct prefixed URLs behind an umbrella nginx gateway.
3. **Portal SSO** (`portal_sso.py`) — if the umbrella nginx sends `X-Auth-Username` / `X-Auth-Role`, automatically create + log in the local user.
---
## 5. Key Architectural Decisions
| Decision | Implementation |
|---|---|
| **Application factory** | `create_app(config_name)` — clean testability, per-env config |
| **Blueprint isolation** | 7 blueprints with clear URL prefixes; API separate from UI |
| **Playlist versioning** | `Playlist.version` incremented on every mutation → players poll `/api/playlist-version/<id>` for refresh |
| **In-memory caching** | Flask-Caching `@cache.memoize(300)` on playlist builders; `cache.clear()` on mutations |
| **DB-backed audit log** | `log_action()` writes `ServerLog` rows; surfaced on dashboard & admin |
| **Two auth systems** | Humans: Flask-Login (`User`). Players: `auth_code` Bearer + `quickconnect_code` (bcrypt) |
| **Remote provisioning** | SSH + rsync/git staged code → write `app_config.json` → run install/start scripts |
| **HTTPS automation** | `CaddyConfigGenerator` writes `Caddyfile`; reloads Caddy via admin API on `:2019` |
---
## 6. Runtime Dependencies (requirements.txt)
**Runtime:** Flask 3.1, Werkzeug 3.1, SQLAlchemy 2.0.37, Flask-SQLAlchemy 3.1.1, Flask-Migrate, Flask-Bcrypt, Flask-Login, Flask-Caching, Flask-Cors, Flask-Talisman, pdf2image, Pillow, ffmpeg-python, python-magic, bcrypt, cryptography, gunicorn 23, psutil, python-dotenv.
**System binaries (Docker):** `poppler-utils`, `ffmpeg`, `libmagic1`, `LibreOffice` (core/impress/writer), `fonts-noto-color-emoji`, `sshpass`, `openssh-client`, `rsync`, `git`.
**Dev:** black, flake8, pytest, pytest-cov. *(`gevent` is commented out — incompatible with Python 3.13.)*
---
> Next: [02 · Knowledge Graph](02-knowledge-graph.md)
+148
View File
@@ -0,0 +1,148 @@
# 02 · The Graphify Knowledge Graph
This document explains the **interactive knowledge graph** generated by the [Graphify](https://marketplace.visualstudio.com/items?itemName=anytechiestudio.graphify-vscode) extension, and how to use it to explore and maintain DigiServer v2.
---
## 1. What Was Generated
Running `Graphify: Build Knowledge Graph` on the project root produced `graphify-out/`:
| Artifact | Description |
|---|---|
| `graph.html` | **Interactive visualizer** — zoom/pan/filter nodes, click to jump to source |
| `graph.json` | Raw graph data (nodes, links, communities, heat) |
| `GRAPH_REPORT.md` | God nodes, surprising connections, community list, knowledge gaps |
| `COMPASS.md` | Token-optimized architecture summary (god nodes, layers) |
| `DOMAINS.md` | Community ID → domain table |
| `intelligence.json` | AI-detected intelligence (god nodes, surprising links) |
| `graph.compact.txt` | Compact graph dump, ideal for LLM context |
| `wiki/` | Markdown articles per community + per god-node, with Mermaid diagrams |
**Graph statistics:**
```
631 nodes · 1162 edges · 41 communities
Extraction: 56% EXTRACTED · 44% INFERRED (avg confidence 0.62)
Node types: 313 FUNCTION · 300 FILE · 18 CLASS
Link types: 311 uses · 290 calls · 275 rationale_for · 243 contains · 39 method · 4 inherits
```
---
## 2. Opening the Visualizer
1. In VS Code, click the **Graphify** activity-bar icon (sidebar).
2. Click **Build Knowledge Graph** (or right-click any folder → *Graphify: Build Knowledge Graph*).
3. Once built, click **Open Interactive Visualizer** — or open `graphify-out/graph.html` directly in a browser.
> ⚠️ The visualizer opens best inside the VS Code webview (activity bar). Opening `graph.html` via `file://` may show a minor console error but still renders the graph data (nodes/edges/communities listed in the sidebar).
**View modes:** Deep Dive · Full Detail · Heatmap · Simulation (component-removal impact).
---
## 3. God Nodes — The Core Abstractions
These are the most-connected nodes (graph centrality). They are the architectural "load-bearing" abstractions:
| Rank | Node | Degree | File | Why it matters |
|---|---|---|---|---|
| 1 | `log_action()` | 116 | `app/utils/logger.py` | Every important action across all blueprints logs here |
| 2 | `PlayerEdit` | 99 | `app/models/player_edit.py` | Central record of on-player media edits |
| 3 | `PlayerUser` | 74 | `app/models/player_user.py` | Maps player edit user codes to names |
| 4 | `Playlist` | 33 | `app/models/playlist.py` | Core content organization + version sync |
| 5 | `CaddyConfigGenerator` | 31 | `app/utils/caddy_manager.py` | HTTPS Caddyfile generation/reload |
| 6 | `HTTPSConfig` | 30 | `app/models/https_config.py` | Persisted HTTPS settings |
| 7 | `Content` | 24 | `app/models/content.py` | Media model used by everything |
| 8 | `User` | 19 | `app/models/user.py` | Human accounts (admin/user/viewer) |
| 9 | `create_app()` | 13 | `app/app.py` | Application factory |
| 10 | `models/__init__.py` | 13 | `app/models/__init__.py` | Package export hub |
---
## 4. Heatmap — Hottest Code
`heat` (01) reflects how central/complex a node is in the graph. The hottest functions drive most of the system's behaviour:
| Heat | Function | File |
|---|---|---|
| 0.620 | `receive_edited_media()` | `app/blueprints/api.py` |
| 0.614 | `deploy_player_to_host()` | `app/utils/ssh_deploy.py` |
| 0.592 | `add_player()` | `app/blueprints/players.py` |
| 0.537 | `upload_media()` | `app/blueprints/content.py` |
| 0.429 | `manage_player()` | `app/blueprints/players.py` |
| 0.425 | `receive_player_feedback()` | `app/blueprints/api.py` |
| 0.407 | `update_https_config()` | `app/blueprints/admin.py` |
| 0.376 | `process_file_in_background()` | `app/blueprints/content.py` |
| 0.365 | `CaddyConfigGenerator` | `app/utils/caddy_manager.py` |
| 0.354 | `get_playlist_by_quickconnect()` | `app/blueprints/api.py` |
---
## 5. Communities (Clusters)
Graphify groups related code into communities. See [01-architecture.md §3](01-architecture.md#3-component-map-files--communities) for the full mapping. The largest:
- **C0 · Auth & logging** (90 nodes) — login/logout/register + audit logging + legacy groups
- **C1 · Admin & HTTPS** (80 nodes) — admin panel, Caddy generation, editing users
- **C2 · Playlists & content** (68 nodes) — the modern content/playlist workflow
- **C3 · Player API & edits** (62 nodes) — player-facing REST + edited-media pipeline
- **C4 · Application core** (55 nodes) — `create_app`, config, user, middleware
- **C5 · Content/player models** (54 nodes) — `Content`, `Player`, feedback, dashboard
Each community has a wiki article with a **Mermaid class diagram**, key concepts, source files, and an audit trail (extracted vs inferred edges). Example: `graphify-out/wiki/Community_4.md`.
---
## 6. Surprising Connections (AI-detected)
Graphify flags links that are non-obvious. Examples from `GRAPH_REPORT.md`:
- `reset_user_password()``log_action()` — admin password resets are audited
- `upload_header_logo()``log_action()` — logo changes are audited
- `delete_editing_user()``log_action()` — editing-user deletion is audited
- `delete_playlist()``log_action()` — playlist deletion is audited
- `Main playlist management page` uses `PlayerEdit` — the content UI surfaces edit counts
**Pattern:** virtually every destructive/admin action flows through `log_action()`, which is why it is the #1 god node.
---
## 7. Known Gaps in the Graph
- **178 isolated nodes** (≤1 connection) — mostly standalone migration scripts and model helper methods.
- **Thin communities 1340** — single-file migrations, isolated utility functions, and archived `old_code_documentation/` scripts. These are not "broken" — they are intentionally decoupled.
- The graph does **not** index templates or static assets (Python AST extraction only).
---
## 8. Querying the Graph (CLI)
The `anytechie-graphify` engine (installed in `.venv`) ships a CLI for graph-aware questions:
```bash
# Shortest path between two nodes
python -m graphify path "Content" "Playlist"
# Explain a node and its neighbours
python -m graphify explain "deploy_player_to_host()"
# BFS traversal to answer a question (no LLM)
python -m graphify query "how is a playlist synced to a player?"
# Grounded chat over the graph (requires an LLM API key)
python -m graphify ask "what happens when a player uploads edited media?"
```
---
## 9. Keeping the Graph Fresh
- **After code changes:** `Graphify: Build Knowledge Graph` again (or run `python -m graphify update <path>`).
- **Git hooks:** `python -m graphify hook install` auto-rebuilds on `post-commit`/`post-checkout`.
- The graph is stored in `graphify-out/` (not committed to git by default).
---
> Next: [03 · Data Model](03-data-model.md)
+198
View File
@@ -0,0 +1,198 @@
# 03 · Data Model
DigiServer v2 uses **SQLAlchemy 2.0** with **SQLite** (production: `instance/dashboard.db`, dev: `instance/dev.db`, tests: in-memory). All models live in `app/models/` and are re-exported from `app/models/__init__.py`.
---
## 1. Entity-Relationship Overview
```mermaid
erDiagram
user ||--o{ server_log : "writes"
player ||--o{ player_feedback : "sends (cascade)"
player ||--o{ player_edit : "edits (cascade)"
player }o--o| playlist : "assigned_to"
content ||--o{ player_edit : "edited (cascade)"
content ||--o{ player_feedback : "playing"
playlist ||--o{ content : "playlist_content M2M (position,duration,muted,edit_on_player_enabled)"
content }o--o{ group : "group_content M2M (legacy)"
player_user ||--o{ player_edit : "user_code"
https_config ||--o| https_config : "singleton row"
```
---
## 2. Tables & Columns
### `user` — human accounts
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `username` | String(80) | unique, NOT NULL, indexed |
| `password` | String(120) | bcrypt hash, NOT NULL |
| `role` | String(20) | default `'user'`; also `'admin'`, `'viewer'`; indexed |
| `theme` | String(20) | default `'light'` |
| `created_at` | DateTime | NOT NULL |
| `last_login` | DateTime | nullable |
Key methods: `is_admin` (property), `update_last_login()`. Mixin: `UserMixin` (Flask-Login).
### `player` — signage devices
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `name` | String(255) | NOT NULL |
| `hostname` | String(255) | unique, NOT NULL, indexed |
| `location` | String(255) | nullable |
| `auth_code` | String(255) | unique, NOT NULL, indexed (legacy Bearer auth) |
| `password_hash` | String(255) | NOT NULL (bcrypt) |
| `quickconnect_code` | String(255) | nullable, bcrypt-hashed |
| `orientation` | String(16) | default `'Landscape'` |
| `status` | String(50) | default `'offline'`, indexed |
| `last_seen` | DateTime | indexed |
| `last_heartbeat` | DateTime | indexed |
| `created_at` | DateTime | NOT NULL |
| `playlist_id` | Integer | FK → `playlist.id` ON DELETE SET NULL, indexed |
| `deployment_status` | String(50) | default `'pending'`; pending/deploying/deployed/failed |
| `last_deployment_at` | DateTime | nullable |
| `last_deployment_status` | String(50) | nullable |
| `last_deployment_message` | Text | nullable |
Relationships: `playlist`, `feedback` (→ PlayerFeedback, cascade delete-orphan).
Methods: `is_online` (5-min window), `update_status()`, `set_password()/check_password()`, `set_quickconnect_code()/check_quickconnect_code()`, static `authenticate(hostname, password, quickconnect_code)`.
### `content` — media items (also weblinks)
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `filename` | String(255) | unique, NOT NULL, indexed; may point to `edited_media/<id>/...` after a player edit |
| `original_filename` | String(255) | nullable, indexed — pristine upload name |
| `content_type` | String(50) | NOT NULL, indexed; image/video/pdf/pptx/weblink/other |
| `duration` | Integer | default 10, nullable |
| `file_size` | BigInteger | nullable |
| `url` | String(2048) | nullable — target URL for `weblink` content |
| `description` | Text | nullable |
| `uploaded_at` | DateTime | NOT NULL, indexed |
Relationships: `playlists` (M2M via `playlist_content`), `groups` (M2M via `group_content`).
Properties/methods: `file_size_mb`, `group_count`, `original_display_name`, `original_media_path`, `current_media_path`, `is_image()/is_video()/is_pdf()/is_weblink()`, `has_player_edits`.
### `playlist` — ordered collections of content
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `name` | String(100) | unique, NOT NULL, indexed |
| `description` | Text | nullable |
| `orientation` | String(20) | default `'Landscape'` |
| `version` | Integer | default 1, NOT NULL — **sync detection** |
| `is_active` | Boolean | default True |
| `created_at` | DateTime | NOT NULL |
| `updated_at` | DateTime | NOT NULL, onupdate |
Methods: `player_count`, `content_count`, `total_duration` (properties), `increment_version()`, `get_content_ordered()`.
### `playlist_content` — association table (M2M playlist ↔ content)
| Column | Type | Notes |
|---|---|---|
| `playlist_id` | Integer | FK → `playlist.id` CASCADE, composite PK |
| `content_id` | Integer | FK → `content.id` CASCADE, composite PK |
| `position` | Integer | default 0 — ordering |
| `duration` | Integer | default 10 — per-playlist override |
| `muted` | Boolean | default True |
| `edit_on_player_enabled` | Boolean | default False — whether player-side editing allowed |
### `player_edit` — on-player media edits
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `player_id` | Integer | FK → `player.id` CASCADE, indexed |
| `content_id` | Integer | FK → `content.id` CASCADE, indexed |
| `original_name` | String(255) | NOT NULL |
| `new_name` | String(255) | NOT NULL |
| `version` | Integer | default 1 |
| `user` | String(255) | nullable (user code) |
| `time_of_modification` | DateTime | nullable |
| `metadata_path` | String(512) | nullable |
| `edited_file_path` | String(512) | NOT NULL |
| `created_at` | DateTime | NOT NULL, indexed |
Relationships: `player`, `content`. Method: `to_dict()`.
### `player_feedback` — status messages from players
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `player_id` | Integer | FK → `player.id`, NOT NULL, indexed |
| `status` | String(50) | default `'unknown'` |
| `current_content_id` | Integer | FK → `content.id`, nullable |
| `message` | Text | nullable |
| `error` | Text | nullable |
| `timestamp` | DateTime | NOT NULL, indexed |
Properties/classmethod: `is_error`, `age_seconds`, `get_latest_for_player()`.
### `player_user` — mapping of player edit user codes
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `user_code` | String(255) | **globally unique**, NOT NULL, indexed |
| `user_name` | String(255) | nullable |
| `created_at` | DateTime | NOT NULL |
| `updated_at` | DateTime | NOT NULL, onupdate |
Method: `to_dict()`. *(Migrated from a per-player table — see `migrate_player_user_global.py`.)*
### `server_log` — DB-backed audit log
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `level` | String(20) | NOT NULL, indexed, default `'info'` |
| `message` | Text | NOT NULL |
| `timestamp` | DateTime | NOT NULL, indexed |
Classmethods: `log_info`, `log_warning`, `log_error`.
### `https_config` — HTTPS settings (singleton row)
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `https_enabled` | Boolean | default False |
| `hostname` | String(255) | nullable |
| `domain` | String(255) | nullable |
| `ip_address` | String(45) | nullable (IPv6-capable) |
| `email` | String(255) | nullable (Let's Encrypt contact) |
| `port` | Integer | default 443 |
| `created_at` | DateTime | NOT NULL |
| `updated_at` | DateTime | NOT NULL, onupdate |
| `updated_by` | String(255) | nullable |
Classmethods: `get_config()` (first row), `create_or_update(...)`. Method: `to_dict()`.
### `group` + `group_content` — **ARCHIVED / LEGACY**
| Column | Type | Notes |
|---|---|---|
| `id` | Integer | PK |
| `name` | String(100) | unique, NOT NULL, indexed |
| `description` | Text | nullable |
| `created_at` / `updated_at` | DateTime | NOT NULL |
`group_content(group_id, content_id)` composite-PK M2M. The current `Player` model has **no** `group_id` column — group features are archived (see [09 · Legacy](09-legacy-and-migrations.md)).
---
## 3. Relationship Summary
| Relationship | Cardinality | FK / Mechanism |
|---|---|---|
| `playlist``content` | M:N | `playlist_content` (positioned, with extras) |
| `group``content` | M:N | `group_content` (legacy) |
| `player``playlist` | N:1 | `player.playlist_id` (ON DELETE SET NULL) |
| `player``player_feedback` | 1:N | `player_feedback.player_id` (cascade) |
| `player``player_edit` | 1:N | `player_edit.player_id` (cascade) |
| `content``player_edit` | 1:N | `player_edit.content_id` (cascade) |
| `player_user``player_edit` | 1:N | `player_edit.user` = `player_user.user_code` (logical) |
| `https_config` | 1 row | singleton via `get_config()` |
---
> Next: [04 · Application Core](04-application-core.md)
+178
View File
@@ -0,0 +1,178 @@
# 04 · Application Core
Covers the application factory, configuration, extensions, middleware, CLI commands, context processors, and template layer. This is **Community 4** in the Graphify knowledge graph.
---
## 1. Application Factory — `app/app.py`
The entire app is constructed by `create_app(config_name=None)`:
```mermaid
flowchart TD
A["create_app()"] --> B["Set instance_path"]
B --> C["Load config: Dev / Prod / Test"]
C --> D["ProxyFix middleware"]
D --> E["ScriptNameFix middleware"]
E --> F["init extensions: db bcrypt login migrate cache cors"]
F --> G["configure_login_manager()"]
G --> H["register_blueprints()"]
H --> I["register_error_handlers()"]
I --> J["register_commands()"]
J --> K["register_context_processors()"]
K --> L["register_template_filters()"]
L --> M["init_portal_sso(app)"]
M --> N["db.create_all() (idempotent)"]
N --> O["return app"]
```
### Blueprint registration
```python
def register_blueprints(app):
from app.blueprints.main import main_bp
from app.blueprints.auth import auth_bp
from app.blueprints.admin import admin_bp
from app.blueprints.players import players_bp
from app.blueprints.content import content_bp
from app.blueprints.playlist import playlist_bp
from app.blueprints.api import api_bp
...
```
> Note: `app.blueprints.content_old` is **not** imported — it is dead/legacy code.
---
## 2. Configuration — `app/config.py`
Four classes: `Config` (base) → `DevelopmentConfig`, `ProductionConfig`, `TestingConfig`.
| Setting | Value | Notes |
|---|---|---|
| `MAX_CONTENT_LENGTH` | 2 GB | upload cap |
| `UPLOAD_FOLDER` | `app/static/uploads` | media |
| `UPLOAD_FOLDERLOGO` | `app/static/resurse` | logos |
| `ALLOWED_EXTENSIONS` | png jpg jpeg gif bmp mp4 avi mkv mov webm pdf ppt pptx | |
| `PERMANENT_SESSION_LIFETIME` | 30 min | |
| `SESSION_COOKIE_SECURE` | False (True in prod) | |
| `ITEMS_PER_PAGE` | 20 | |
| `SERVER_VERSION` | 2.0.0 | shown in UI footer |
| `PLAYER_CODE_DIR` | `/app/data/player` | staged player source |
| `PLAYER_REPO_URL` | env `PLAYER_REPO_URL` | Kiwy-Signage repo |
| DB (dev) | `instance/dev.db` | |
| DB (prod) | `instance/dashboard.db` | |
| DB (test) | in-memory | |
---
## 3. Extensions — `app/extensions.py`
Centralizes shared singletons:
```python
db = SQLAlchemy()
bcrypt = Bcrypt()
login_manager = LoginManager() # login_view='auth.login'
migrate = Migrate()
cache = Cache()
cors = CORS()
```
CORS is configured in `create_app` for `/api/*`: all origins, GET/POST/OPTIONS/PUT/DELETE, supports credentials.
---
## 4. Middleware
### ProxyFix (Werkzeug)
`app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1, x_port=1)` — trusts one reverse-proxy hop (Caddy / umbrella nginx).
### ScriptNameFix — `app/utils/script_name_fix.py`
`ScriptNameFix` WSGI middleware sets `SCRIPT_NAME` from `HTTP_X_SCRIPT_NAME` so `url_for()` generates correct paths when the app is mounted under a sub-path (e.g. `/digiserver`).
---
## 5. Login Manager
```python
login_manager.login_view = 'auth.login'
login_manager.login_message = 'Please log in to access this page.'
@login_manager.user_loader
def load_user(user_id):
return User.query.get(int(user_id))
```
---
## 6. Error Handlers
| Code | Template | Behaviour |
|---|---|---|
| 404 | `errors/404.html` | Not found |
| 403 | `errors/403.html` | Forbidden |
| 500 | `errors/500.html` | Rolls back DB session |
| 413 | `errors/413.html` | Payload too large |
| 408 | `errors/408.html` | Request timeout |
The `/api` blueprint registers its own **JSON** 404/405/500 handlers.
---
## 7. CLI Commands
| Command | Purpose |
|---|---|
| `flask init-db` | `db.create_all()` |
| `flask create-admin --username X` | Create admin user (prompts for password) |
| `flask seed-db` | Seed sample data (blocked in production) |
---
## 8. Context Processors
- **`inject_config`** — exposes `server_version`, `build_date`, `logo_exists` to all templates.
- **`inject_user_theme`** — exposes the authenticated user's `theme` (`light`/`dark`) for the UI toggle.
---
## 9. Template Filters
- **`localtime`** — converts naive/UTC datetimes to local time with a configurable `strftime` format (default `%Y-%m-%d %H:%M`).
---
## 10. Template & Static Layout
```
app/templates/
├── base.html ← main layout (theme, logos, nav)
├── dashboard.html
├── auth/ login.html, register.html, change_password.html
├── admin/ admin.html, user_management.html, leftover_media.html,
│ dependencies.html, customize_logos.html, editing_users.html,
│ https_config.html, build_player.html
├── content/ content_list.html (legacy), content_list_new.html (modern),
│ media_library.html, upload_content.html (legacy),
│ upload_media.html, manage_playlist_content.html, edit_content.html
├── players/ players_list.html, add_player.html, edit_player.html,
│ manage_player.html, player_page.html, player_fullscreen.html,
│ edited_media.html, edited_media_report.html, _deploy_badge.html
└── errors/ 403.html, 404.html, 500.html (+413/408)
app/static/
├── icons/ edit, home, info, monitor, moon, playlist, sun, trash, upload, warning (SVG)
├── uploads/ uploaded media + edited_media/<content_id>/ (versioned edits)
└── (resurse/ logo storage — referenced by config)
```
---
## 11. Portal SSO — `app/utils/portal_sso.py`
`init_portal_sso(app)` registers a `before_request` hook that reads `X-Auth-Username` / `X-Auth-Role` headers (set by the umbrella nginx gateway). If present, it auto-creates and logs in the local `User` (`_get_or_create_user`). This lets the app sit behind an existing corporate SSO portal.
---
> Next: [05 · Blueprints & API](05-blueprints-api.md)
+280
View File
@@ -0,0 +1,280 @@
# 05 · Blueprints & REST API
DigiServer v2 registers **7 active blueprints** (`content_old.py` is legacy dead code and is not registered).
---
## 1. Blueprint Overview
| Blueprint | URL prefix | Module | Focus |
|---|---|---|---|
| `main_bp` | `/` | `blueprints/main.py` | Dashboard, health |
| `auth_bp` | `/` | `blueprints/auth.py` | Login/logout/register/password |
| `admin_bp` | `/admin` | `blueprints/admin.py` | Admin panel, HTTPS, build, users, logos |
| `players_bp` | `/players` | `blueprints/players.py` | Player management |
| `content_bp` | `/content` | `blueprints/content.py` | Media library + modern playlist management |
| `playlist_bp` | `/playlist` | `blueprints/playlist.py` | Legacy per-player playlist (redirects) |
| `api_bp` | `/api` | `blueprints/api.py` | Player-facing REST + deployment API |
---
## 2. `main` — Dashboard & Health (`/`)
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/` | GET | `dashboard` | Counts (players/playlists/content), storage MB, 20 recent logs (cached 60s except `viewer` role) |
| `/health` | GET | `health` | Health JSON: `SELECT 1` DB ping + free disk space |
---
## 3. `auth` — Authentication (`/`)
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/login` | GET/POST | `login` | bcrypt login, `remember`, `next` redirect, failed-attempt logging |
| `/logout` | GET | `logout` | Logs out + logs action |
| `/register` | GET/POST | `register` | Self-registration → default `viewer` role |
| `/change-password` | GET/POST | `change_password` | Change own password (verifies current) |
---
## 4. `admin` — Admin Panel (`/admin`)
**Security:** `@login_required` on all routes; `@admin_required` (custom decorator, `role == 'admin'`) on user-management.
### Users
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/` | GET | `admin_panel` | Stats, recent logs, users, storage |
| `/user/create` | POST | `create_user` | Create user (username ≥3, password ≥6) |
| `/user/<id>/role` | POST | `change_user_role` | Change role (not own) |
| `/user/<id>/delete` | POST | `delete_user` | Delete user (not own) |
| `/user/<id>/password` | POST | `reset_user_password` | Admin reset password |
| `/users` | GET | `user_management` | User list page |
| `/theme` | POST | `change_theme` | light/dark theme |
### Media housekeeping
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/leftover-media` | GET | `leftover_media` | Content not assigned to any playlist, by type + size |
| `/delete-leftover-images` | POST | — | Bulk-delete leftover images (+ archive + PlayerEdit) |
| `/delete-leftover-videos` | POST | — | Bulk-delete leftover videos |
| `/delete-single-leftover/<id>` | POST | — | Delete one leftover item |
### System / dependencies
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/system/info` | GET | `system_info` | JSON: platform + psutil (CPU/mem/disk) |
| `/dependencies` | GET | `dependencies` | Checks LibreOffice, Poppler, FFmpeg, emoji fonts |
| `/install-libreoffice` | POST | — | `sudo -n install_libreoffice.sh` |
| `/install-emoji-fonts` | POST | — | `sudo -n install_emoji_fonts.sh` |
| `/logo/upload` | POST | `upload_logo` | Upload logo.png |
| `/logs/clear` | POST | `clear_logs` | Delete all `ServerLog` rows |
| `/customize-logos` | GET | `customize_logos` | Logo customization page |
| `/upload-header-logo` | POST | — | Save header_logo.png |
| `/upload-login-logo` | POST | — | Save login_logo.png |
### Editing users
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/editing-users` | GET | `manage_editing_users` | List `PlayerUser` + per-user edit counts |
| `/editing-users/<id>/update` | POST | — | Rename editing user |
| `/editing-users/<id>/delete` | POST | — | Delete editing user |
### HTTPS (Caddy)
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/https-config` | GET | `https_config` | HTTPS config page (auto-corrects DB flag if request is HTTPS) |
| `/https-config/update` | POST | `update_https_config` | Validate + save config → regenerate Caddyfile → reload Caddy |
| `/https-config/status` | GET | `https_config_status` | Config as JSON |
### Player build & deployment
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/build-player` | GET | `build_player` | Build page + staged-code status |
| `/build-player` | POST | `build_player_action` | `build_files` / `save_config` / `build_and_config` → clone/refresh code, write `app_config.json`, persist `instance/player_build.json` |
---
## 5. `players` — Player Management (`/players`)
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/` and `/list` | GET | `list` | Player list + live deployment-status polling data |
| `/add` | GET/POST | `add_player` | Create player; optional **background SSH deployment** |
| `/bulk/delete` | POST | — | JSON bulk delete |
| `/bulk/assign-playlist` | POST | — | JSON bulk assign to playlist |
| `/deployment-status` | GET | — | JSON deployment status for all players (polling) |
| `/<int:player_id>` | GET | — | Redirect to `manage_player` |
| `/<id>/edit` | GET/POST | — | Edit name/location (legacy) |
| `/<id>/delete` | POST | — | Delete player + feedback |
| `/<id>/regenerate-auth` | POST | — | New auth code |
| `/<id>/manage` | GET/POST | `manage_player` | Manage page: `update_credentials` / `assign_playlist`; feedback, edits, status |
| `/<id>/edited-media` | GET | `edited_media` | All edited media + user mapping |
| `/<id>/edited-media-report` | GET | `edited_media_report` | Tabular report |
| `/<id>/fullscreen` | GET | — | Player fullscreen (NO auth; optional `?auth=` check), cached playlist |
| `/<id>/reorder` | POST | — | Legacy stub → 400 "use Playlists page" |
| `/<id>/playlist/reorder` | POST | — | Legacy per-player reorder (JSON up/down) |
| `/<id>/playlist/remove` | POST | — | Legacy removal (bumps `playlist_version`) |
Helper: `get_player_playlist(player_id)``@cache.memoize(300)` builds playlist dicts (`url`, `type`, `duration`, `position`, `muted`, `audio`).
---
## 6. `content` — Media Library & Modern Playlist Management (`/content`)
> This is the **primary workflow** in v2 (1,519 lines).
### Library & uploads
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/` | GET | `content_list` | Main page `content_list_new.html`: playlists, last-3 media, counts, players |
| `/media-library` | GET | `media_library` | All media by type + edit counts |
| `/media/<id>/delete` | POST | `delete_media` | Delete file + archive + edits; removes from playlists + bumps versions |
| `/upload-media-page` | GET | `upload_media_page` | Upload page (playlist selector) |
| `/upload-media` | POST | `upload_media` | Core upload: images optimized, videos validated, **PDF/PPTX → Full-HD PNGs**, large files → background threads |
### Web links
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/add-weblink` | POST | `add_weblink` | Create weblink Content (http/https validated), optional playlist add |
| `/playlist/<id>/add-weblink` | POST | — | Create weblink + append to playlist |
### Playlist management
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/playlist/create` | POST | — | Create playlist (name, description, orientation) |
| `/playlist/<id>/delete` | POST | — | Delete + unassign players |
| `/playlist/<id>/manage` | GET | `manage_playlist_content` | Ordered content + available library (no weblinks) |
| `/playlist/<id>/add-content` | POST | — | Add library content at next position |
| `/playlist/<id>/remove-content/<content_id>` | POST | — | Remove content (deletes orphan weblinks) |
| `/playlist/<id>/bulk-remove` | POST | — | JSON bulk remove |
| `/playlist/<id>/reorder` | POST | — | JSON reorder (position = index) |
| `/playlist/<id>/update-muted/<content_id>` | POST | — | Toggle muted |
| `/playlist/<id>/update-edit-enabled/<content_id>` | POST | — | Toggle `edit_on_player_enabled` |
| `/playlist/<id>/update-duration/<content_id>` | POST | — | Per-playlist duration (≥1s) |
| `/player/<id>/assign-playlist` | POST | — | Assign/unassign player to playlist |
**All playlist mutations** call `Playlist.increment_version()` and `cache.clear()`.
### Processing helpers (module-level)
`process_image_file`, `process_video_file_extended` (ffprobe), `process_pdf_file` (pdf2image @300 DPI → `_pageNNN.png`), `process_presentation_file` (LibreOffice → PDF → pdftoppm → Full-HD PNG), `create_fullhd_image`, `resize_image_to_fullhd`, `optimize_image_to_fullhd`, `process_file_in_background` (thread wrapper).
---
## 7. `playlist` — Legacy Per-Player Routes (`/playlist`)
Redirects/legacy — kept for compatibility:
| Route | Methods | Purpose |
|---|---|---|
| `/<int:player_id>` | GET | Redirect to modern manage page |
| `/<id>/add` | POST | Add content to player's playlist |
| `/<id>/remove/<content_id>` | POST | Remove + renumber |
| `/<id>/reorder` | POST | JSON reorder from `content_ids` |
| `/<id>/update-duration/<content_id>` | POST | Update duration |
| `/<id>/update-muted/<content_id>` | POST | Update muted |
| `/<id>/clear` | POST | Clear all items |
---
## 8. `api` — Player-Facing REST + Deployment API (`/api`)
**Custom decorators:** `rate_limit(max_requests, window)` (in-memory, IP/Bearer keyed) and `verify_player_auth` (validates Bearer `auth_code` → sets `request.player`).
### Health & auth
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/health` | GET | `health_check` | API health (`version: 2.0.0`) |
| `/certificate` | GET | — | TLS certificate test |
| `/auth/player` | POST | `authenticate_player` | hostname + password/quickconnect → auth_code, playlist_id, orientation; sets online |
| `/auth/verify` | POST | `verify_auth_code` | Verify auth code → player info |
### Playlist delivery
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/playlists` | GET | `get_playlist_by_quickconnect` | Kivy-compatible: `?hostname=&quickconnect_code=` → playlist + version + hashed quickconnect |
| `/playlists/<player_id>` | GET | `get_player_playlist` | Bearer-authed fetch (cached) |
| `/playlist-version/<player_id>` | GET | `get_playlist_version` | Lightweight refresh-poll version check |
**Payload keys the players expect** (from `get_cached_playlist`): `file_name`, `type`, `duration`, `position`, `url`, `description`, `edit_on_player_enabled`, `muted`, `audio` (`"on"`/`"off"`).
### Feedback & monitoring
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/player-feedback` | POST | `receive_player_feedback` | Status (playing/paused/error/restarting); infers player; **auto-marks deployment `deployed`** |
| `/player-status/<player_id>` | GET | `get_player_status` | Online (5-min), latest feedback |
| `/system-info` | GET | — | Counts: players online/total, groups, content, 24h logs |
| `/content` | GET | — | List content with counts |
| `/logs` | GET | `get_logs` | Query logs (`limit`/`level`/`since`) |
### Edited media
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/player-edit-media` | POST | `receive_edited_media` | Bearer-authed multipart upload of edited image + metadata; versionizes to `edited_media/<content_id>/`; records `PlayerEdit`; auto-creates `PlayerUser`; repoints `Content.filename`; bumps playlist version; clears cache |
### SSH deployment
| Route | Methods | Endpoint | Purpose |
|---|---|---|---|
| `/deploy/test-ssh` | POST | `test_ssh_connection` | Test SSH (sshpass/ssh) |
| `/deploy/player` | POST | `deploy_player` | Full remote deploy (`deploy_player_to_host`); API key = `sha256(name:hostname)[:32]` |
**Error handlers:** JSON 404 / 405 / 500.
---
## 9. API Call Flows (Mermaid)
### Player bootstrap (auth → playlist)
```mermaid
sequenceDiagram
participant P as Player
participant A as /api
P->>A: POST /api/auth/player {hostname, password|quickconnect}
A->>A: Player.authenticate()
A-->>P: {auth_code, playlist_id, orientation}
P->>A: GET /api/playlists/<id> (Bearer auth_code)
A-->>P: playlist items + version
loop refresh poll
P->>A: GET /api/playlist-version/<id>
A-->>P: {version}
end
```
### Edited media upload
```mermaid
sequenceDiagram
participant P as Player
participant A as /api
participant DB as SQLite
P->>A: POST /api/player-edit-media (Bearer, multipart image + metadata)
A->>DB: locate Content (filename / edited_media/<id>/ regex / last PlayerEdit)
A->>DB: move original → edited_media/<id>/original_*
A->>DB: save new version + side-car JSON
A->>DB: get_or_create PlayerUser (user_code)
A->>DB: create PlayerEdit
A->>DB: repoint Content.filename → latest edit (preserve original_filename)
A->>DB: Playlist.increment_version()
A-->>P: 200 OK
```
### Deployment
```mermaid
sequenceDiagram
participant Admin as Admin UI
participant S as Server (app)
participant T as Background task
participant H as Player host
Admin->>S: add player / deploy
S->>T: background_player_deployment()
T->>H: sshpass test_ssh_connection()
T->>H: rsync staged code (or git clone/pull)
T->>H: write config/app_config.json
T->>H: temp passwordless sudo → install.sh → start.sh
T->>H: cleanup sudoers
T-->>S: update player.deployment_status = deployed|failed
Admin->>S: GET /players/deployment-status (poll)
```
---
> Next: [06 · Utils & Services](06-utils-services.md)
+140
View File
@@ -0,0 +1,140 @@
# 06 · Utilities & Services
All shared services live in `app/utils/`. This document details each module, its main functions, and how they fit together.
---
## Overview Table
| Module | Community | Responsibility | Key symbols |
|---|---|---|---|
| `logger.py` | C0 | DB-backed audit logging | `log_action()`, `get_recent_logs()`, `clear_old_logs()` |
| `group_player_management.py` | C0 | Group/player stats (legacy) | `get_player_status_info()`, `assign_player_to_group()`, `get_online_players_count()` |
| `caddy_manager.py` | C1 | HTTPS Caddyfile generation | `CaddyConfigGenerator`, `write_caddyfile()`, `reload_caddy()` |
| `background_tasks.py` | C10 | Async task execution | `run_background_task()`, `background_player_deployment()` |
| `ssh_deploy.py` | C10 | Remote player provisioning | `deploy_player_to_host()`, `test_ssh_connection()`, `generate_player_config()` |
| `player_build.py` | C11 | Stage player source code | `build_player_files()`, `write_base_config()`, `load/save_build_settings()` |
| `pptx_converter.py` | C12 | PPTX → PDF → PNG | `pptx_to_pdf_libreoffice()`, `validate_pptx_file()`, `cleanup_libreoffice_processes()` |
| `uploads.py` | C9 | Upload progress + file ops | `get/set/clear_upload_progress()`, `save_uploaded_file()`, `process_video_file()` |
| `portal_sso.py` | C4 | SSO auto-login | `init_portal_sso()`, `_get_or_create_user()` |
| `script_name_fix.py` | C4 | WSGI sub-path middleware | `ScriptNameFix` |
| `nginx_config_reader.py` | C7 | Legacy nginx status parsing | `NginxConfigReader`, `get_nginx_status()` |
---
## 1. `logger.py` — Audit Logging ⭐ God Node (116 edges)
The most-connected module in the system. Every meaningful action across all blueprints is recorded here.
| Function | Purpose |
|---|---|
| `log_action(level, message)` | Create + commit a `ServerLog` row |
| `log_info(message)` / `log_warning(message)` / `log_error(message)` | Convenience wrappers |
| `get_recent_logs(limit, level)` | Query recent logs (used by dashboard/admin) |
| `clear_old_logs(days)` | Housekeeping |
**Usage pattern:** admin actions (`reset_user_password`, `upload_header_logo`, `delete_editing_user`, `delete_playlist`, login failures, etc.) all funnel through `log_action()`.
---
## 2. `background_tasks.py` — Async Execution
| Function | Purpose |
|---|---|
| `run_background_task(func, *args)` | Spawns a **daemon thread** with a Flask app context pushed, then runs `func` |
| `background_player_deployment(...)` | Runs SSH deployment in background; updates `player.deployment_status` (`deploying → deployed|failed`) |
Used by `players.add_player` to avoid blocking the HTTP request during long remote installs.
---
## 3. `ssh_deploy.py` — Remote Player Provisioning ⭐ (heat 0.614)
The deployment engine. Full pipeline implemented by `deploy_player_to_host(...)`:
```mermaid
flowchart TD
A["deploy_player_to_host()"] --> B["test_ssh_connection() (sshpass)"]
B --> C["mkdir remote dir"]
C --> D{"staged code present?"}
D -- yes --> E["rsync pre-staged code"]
D -- no --> F["git clone / pull"]
E --> G["write config/app_config.json"]
F --> G
G --> H["temp passwordless sudo"]
H --> I["run install.sh"]
I --> J["run start.sh"]
J --> K["cleanup sudoers"]
K --> L["return steps[]"]
```
Other helpers:
- `get_local_player_code_status()` — inspect staged code state.
- `detect_server_ip()` / `parse_server_address()` — resolve the server address players should reach.
- `generate_player_config()` / `generate_app_config()` — write player's `config/app_config.json` with `server_ip`, `port`, `screen_name`, `quickconnect_key`, `orientation`, `use_https`, `verify_ssl`.
---
## 4. `caddy_manager.py` — HTTPS Automation ⭐ God Node
`CaddyConfigGenerator` produces the Caddyfile for the reverse proxy and reloads Caddy without restart.
| Method | Purpose |
|---|---|
| `generate_caddyfile(config)` | Pick template by mode: **HTTP-only** (`:80`), **domain** (Let's Encrypt), or **IP** (internal CA self-signed). Includes `reverse_proxy digiserver-app:5000`, 2 GB body limit, gzip, security headers |
| `write_caddyfile(content, path=/etc/caddy/Caddyfile)` | Write to disk |
| `reload_caddy()` | POST to Caddy admin API `http://caddy:2019/load` |
Triggered from `admin.update_https_config` after saving `HTTPSConfig`.
---
## 5. `player_build.py` — Staging Player Code
| Function | Purpose |
|---|---|
| `build_player_files(dir, repo_url, branch)` | Clone or `fetch + reset` the player repository into `PLAYER_CODE_DIR` |
| `write_base_config(...)` | Write `config/app_config.json` (blank screen / quickconnect) |
| `load_build_settings()` / `save_build_settings()` | JSON at `instance/player_build.json` |
| `get_player_server_settings` / `make_build_record` / `get_short_head` | Build metadata helpers |
---
## 6. `uploads.py` — Upload & Media Processing
| Function | Purpose |
|---|---|
| `get_upload_progress` / `set_upload_progress` / `clear_upload_progress` | In-memory per-file progress (for the upload page) |
| `save_uploaded_file` | Save a multipart upload |
| `process_video_file` | FFmpeg → H.264 main, 30 fps, ≤1080p, faststart |
| `process_pdf_file` | PDF processing (stub — real logic in content.py) |
| `get_file_size` / `delete_file` | FS helpers |
---
## 7. `pptx_converter.py` — LibreOffice Integration
| Function | Purpose |
|---|---|
| `pptx_to_pdf_libreoffice(pptx_path, output_dir)` | Headless LibreOffice → PDF (300 s timeout) |
| `validate_pptx_file()` | Validate a file is a real PPTX |
| `cleanup_libreoffice_processes()` | `pkill soffice` — clean hanging processes |
Used by the upload pipeline: **PPTX → PDF → PNG slides (Full HD)**.
---
## 8. `portal_sso.py` & `script_name_fix.py` — Gateway Integration
- **`portal_sso.py`** — `before_request` reads `X-Auth-Username`/`X-Auth-Role` from the umbrella nginx and auto-logs-in the local user (creating it on first arrival). See [04 · Application Core §11](04-application-core.md#11-portal-sso--apputilsportal_sso_py).
- **`script_name_fix.py`** — WSGI middleware mapping `X-Script-Name``SCRIPT_NAME` so `url_for()` is correct behind a path-prefixed gateway.
---
## 9. `nginx_config_reader.py` — Legacy (informational)
`NginxConfigReader` parses an `nginx.conf` and reports `ssl_enabled`, ports, upstreams, `server_names`, `ssl_protocols`, `client_max_body_size`, `gzip`. **Legacy** — the current reverse proxy is Caddy; retained for reference and the old deployment stack.
---
> Next: [07 · Deployment](07-deployment.md)
+162
View File
@@ -0,0 +1,162 @@
# 07 · Deployment
DigiServer v2 runs as a **Docker Compose** stack (app + Caddy) with optional remote SSH provisioning of players.
---
## 1. Architecture
```mermaid
flowchart LR
subgraph Host["Docker Host"]
subgraph Net["digiserver-network"]
APP["digiserver-app\nFlask + Gunicorn :5000"]
CAD["caddy:2-alpine\n:80 / :443"]
end
VOL1["./data/instance → /app/instance\n(SQLite DBs)"]
VOL2["./data/uploads → /app/app/static/uploads"]
VOL3["./data/Caddyfile → /etc/caddy/Caddyfile"]
VOL4["./data/caddy-data → /data\n./data/caddy-config → /config"]
end
Browser["Browser"] --> CAD
CAD --> APP
APP --> VOL1
APP --> VOL2
CAD --> VOL3
CAD --> VOL4
APP -.SSH/rsync.-> Players["Remote player hosts"]
```
---
## 2. `docker-compose.yml`
### Service `digiserver-app`
- **Build:** `.` (Dockerfile, `python:3.13-slim`)
- **Ports:** `5000:5000` (direct dev access; also `expose: 5000`)
- **Volumes:** `./data/instance:/app/instance`, `./data/uploads:/app/app/static/uploads`
- **Env:** `FLASK_ENV=production`, `SECRET_KEY`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`
- **Healthcheck:** HTTP GET `http://localhost:5000/` (30s interval, 40s start)
- **Restart:** `unless-stopped`
### Service `caddy`
- **Image:** `caddy:2-alpine`
- **Ports:** `8080:80`, `8443:443`
- **Volumes:** `./data/Caddyfile:/etc/caddy/Caddyfile:rw`, `./data/caddy-data:/data`, `./data/caddy-config:/config`, `./data/caddy-logs:/var/log/caddy`
- **Depends on:** app (service started)
- **Healthcheck:** `wget` on port 80
---
## 3. Dockerfile
| Stage | Content |
|---|---|
| Base | `python:3.13-slim` |
| System deps | `poppler-utils`, `ffmpeg`, `libmagic1`, `sudo`, `fonts-noto-color-emoji`, `LibreOffice` (core/impress/writer), `sshpass`, `openssh-client`, `rsync`, `git` |
| Python | `COPY requirements.txt``pip install` (cached layer), then `COPY . .` |
| App config | `FLASK_APP=app.app:create_app`, `FLASK_ENV=production`, `EXPOSE 5000` |
| User | Non-root `appuser` (UID 1000) with **passwordless sudo** limited to `apt-get`, `install_libreoffice.sh`, `install_emoji_fonts.sh` |
| Runtime | `HEALTHCHECK` (HTTP 5000), `ENTRYPOINT /docker-entrypoint.sh` |
---
## 4. `docker-entrypoint.sh`
1. Create `/app/instance` and `/app/app/static/uploads`.
2. If `dashboard.db` is missing: create app + `db.create_all()`, then create/update the admin user from `ADMIN_USERNAME` / `ADMIN_PASSWORD`.
3. Start **Gunicorn**: `--bind 0.0.0.0:5000 --workers 4 --timeout 120 app.app:create_app('production')`.
---
## 5. `deploy.sh` (One-Shot Deployment)
```
1. Validate compose + project; create data/ subdirs; copy nginx configs
2. docker compose up -d + verify containers "Up"
3. Run migration scripts (add_https_config_table, add_player_user_table,
add_email_to_https_config, migrate_player_user_global,
add_original_filename_to_content)
4. Run /app/https_manager.py enable <hostname> <domain> <email> <ip> <port>
⚠ https_manager.py is NOT in the current repo — this step needs attention
5. Verify DB tables via SQLAlchemy inspector
6. caddy validate + https_manager.py status; print access URLs + default creds
```
---
## 6. HTTPS Setup (Caddy)
HTTPS is configured through the **Admin → HTTPS Configuration** page, which:
1. Saves `HTTPSConfig` (hostname, domain, IP, email, port, enabled).
2. `CaddyConfigGenerator.generate_caddyfile(config)` picks a template:
- **HTTP-only** → `:80` reverse proxy
- **Domain** → automatic Let's Encrypt
- **IP** → internal CA self-signed
3. Writes `/etc/caddy/Caddyfile` and reloads Caddy via `POST http://caddy:2019/load`.
The current `data/Caddyfile` (HTTP mode) includes: admin API on `0.0.0.0:2019`, `:80` block → `digiserver-app:5000`, 2 GB body limit, gzip, security headers, access log.
---
## 7. `verify-deployment.sh` — Pre/Post-Deployment Checks
Sections checked (pass/fail/warn counters):
- git status
- `.env` / `.env.example`
- Docker + Compose versions + `compose config` syntax
- Dockerfile best practices (HEALTHCHECK, non-root, slim base)
- `requirements.txt` critical packages + versions
- migrations directory
- **SSL cert expiry** (openssl)
- Flask config (`ProductionConfig`, `SESSION_COOKIE_SECURE`)
- nginx.conf checks
- runtime container health
- security best practices
> ⚠ Note: the script still references `docker-compose` (v1) and `digiserver-nginx` — the current stack uses Compose v2 + Caddy.
---
## 8. Player Deployment Pipeline
```mermaid
sequenceDiagram
participant U as Admin
participant A as App
participant B as Background
participant H as Player host
U->>A: Add player (name, hostname, password/quickconnect)
A->>B: background_player_deployment()
B->>B: build/refresh staged code (PLAYER_CODE_DIR)
B->>H: sshpass + ssh test
B->>H: rsync code (or git clone/pull fallback)
B->>H: write config/app_config.json (server_ip, quickconnect, https...)
B->>H: temporary passwordless sudo
B->>H: ./install.sh && ./start.sh
B->>H: remove temp sudoers
B-->>A: player.deployment_status = 'deployed' | 'failed'
U->>A: poll /players/deployment-status
```
Statuses: `pending → deploying → deployed | failed`, with `last_deployment_at/status/message` persisted on the player row. Receiving player feedback also auto-marks deployment `deployed`.
---
## 9. Ports & Volumes Summary
| Item | Value |
|---|---|
| App port (container) | 5000 |
| App port (host, dev) | 5000 |
| Caddy HTTP | 8080 → 80 |
| Caddy HTTPS | 8443 → 443 |
| Caddy admin API | 2019 (container) |
| DB volume | `./data/instance` |
| Uploads volume | `./data/uploads` |
| Caddy data/config/logs | `./data/caddy-*` |
---
> Next: [08 · Workflows](08-workflows.md)
+150
View File
@@ -0,0 +1,150 @@
# 08 · End-to-End Workflows
This document traces the main user journeys through the system, tying together blueprints, models, utils, and the Graphify graph.
---
## 1. Content Upload → Playlist → Player
```mermaid
flowchart LR
U["Admin uploads media"] --> UF["/content/upload-media"]
UF -->|image| IMG["optimize_image_to_fullhd"]
UF -->|video| VID["validate via ffprobe"]
UF -->|pdf| PDF["pdf2image @300dpi → page PNGs"]
UF -->|pptx| PPT["LibreOffice → PDF → PNG slides"]
UF -->|large file| BG["process_file_in_background (thread)"]
UF --> DB[("Content row")]
DB --> PLA["/content/playlist/<id>/add-content"]
PLA --> PLC["playlist_content (position, duration, muted, edit_on_player_enabled)"]
PLC --> VER["Playlist.increment_version() + cache.clear()"]
VER --> API["/api/playlists/<id>"]
API --> P["Player fetches new playlist"]
```
**Key files:** `app/blueprints/content.py`, `app/utils/uploads.py`, `app/utils/pptx_converter.py`, `app/models/content.py`, `app/models/playlist.py`.
---
## 2. Playlist Synchronization (the "version" mechanism)
Every playlist mutation bumps `Playlist.version`. Players detect changes by polling:
```mermaid
sequenceDiagram
participant U as Admin UI
participant App
participant DB as SQLite
participant P as Player
U->>App: reorder / mute / duration / add / remove
App->>DB: mutate playlist_content
App->>DB: playlist.increment_version()
App->>App: cache.clear()
loop while player runs
P->>App: GET /api/playlist-version/<player_id>
App-->>P: {version: N}
alt version changed
P->>App: GET /api/playlists/<player_id>
App-->>P: new playlist payload
end
end
```
**Playlist payload items** include: `file_name`, `type`, `duration`, `position`, `url` (weblink or static file), `description`, `edit_on_player_enabled`, `muted`, `audio`.
---
## 3. On-Player Media Editing Pipeline ⭐ (heat 0.620)
When a signage operator edits an image directly on the player device:
```mermaid
flowchart TD
A["Player posts edited image"] --> B["POST /api/player-edit-media (Bearer + multipart)"]
B --> C{"Locate Content"}
C -->|by filename| D["match filename"]
C -->|edited_media regex| E["match edited_media/<id>/"]
C -->|fallback| F["last PlayerEdit"]
D/E/F --> G["Move original → edited_media/<content_id>/original_*"]
G --> H["Save new version + side-car metadata JSON"]
H --> I{"PlayerUser exists for user_code?"}
I -- no --> J["auto-create PlayerUser"]
I -- yes --> K["reuse"]
J/K --> L["Create PlayerEdit (version, original_name, new_name...)"]
L --> M["Repoint Content.filename → latest edit"]
M --> N["Preserve original_filename"]
N --> O["Playlist.increment_version() + cache.clear()"]
O --> P["Player sees updated media on refresh"]
```
**Why this is a god node:** `PlayerEdit` (99 edges) and `PlayerUser` (74 edges) make this the most-connected data flow in the system. See `app/models/player_edit.py`, `app/models/player_user.py`, `app/blueprints/api.py`.
---
## 4. HTTPS Configuration (Caddy)
```mermaid
flowchart TD
A["Admin → /admin/https-config"] --> B["POST /admin/https-config/update"]
B --> C["Validate + save HTTPSConfig (email, domain, ip, port)"]
C --> D["CaddyConfigGenerator.generate_caddyfile()"]
D --> E{"Mode"}
E -->|HTTP| F[":80 reverse proxy"]
E -->|domain| G["Let's Encrypt automatic HTTPS"]
E -->|IP| H["Internal CA self-signed"]
F/G/H --> I["write_caddyfile(/etc/caddy/Caddyfile)"]
I --> J["POST http://caddy:2019/load (hot reload)"]
J --> K["HTTPS live without restart"]
```
---
## 5. User Management & Auditing
```mermaid
flowchart TD
A["Admin panel"] --> B["/admin/user/create"]
A --> C["/admin/user/<id>/role"]
A --> D["/admin/user/<id>/password"]
A --> E["/admin/user/<id>/delete"]
B/C/D/E --> L["log_action(level, message)"]
L --> S[("server_log")]
S --> F["dashboard / admin recent logs"]
S --> G["/api/logs (limit, level, since)"]
```
**Roles:** `admin` (full), `user` (manage content/players), `viewer` (read-only dashboard).
---
## 6. Portal SSO (optional corporate gateway)
```mermaid
sequenceDiagram
participant B as Browser
participant G as Umbrella nginx gateway
participant A as DigiServer
B->>G: request (authenticated by portal)
G->>A: proxy + headers X-Auth-Username, X-Auth-Role
A->>A: init_portal_sso before_request
A->>A: _get_or_create_user(username, role)
A-->>B: dashboard (auto logged-in)
```
---
## 7. Media Housekeeping (Leftovers)
Content not assigned to any playlist is flagged as "leftover":
```mermaid
flowchart LR
L["/admin/leftover-media"] --> T["grouped by type: image/video/pdf/pptx + sizes"]
T --> D["/admin/delete-leftover-images|videos"]
D --> X["delete file + edited_media archive + PlayerEdit rows"]
```
---
> Next: [09 · Legacy & Migrations](09-legacy-and-migrations.md)
+64
View File
@@ -0,0 +1,64 @@
# 09 · Migrations & Legacy Components
---
## 1. Migrations
Migrations are **standalone Python scripts** (run via `docker compose exec ... python`, **not** Alembic revisions — there is no `migrations/versions/`).
| Script | Table(s) | What it does |
|---|---|---|
| `add_https_config_table.py` | `https_config` | Creates table via `db.create_all()` |
| `add_player_user_table.py` | `player_user` | Creates table via `db.create_all()` |
| `add_email_to_https_config.py` | `https_config` | `ALTER TABLE ... ADD COLUMN email VARCHAR(255)` (idempotent) |
| `add_deployment_fields_to_player.py` | `player` | Adds `deployment_status`, `last_deployment_at`, `last_deployment_status`, `last_deployment_message` (idempotent) |
| `add_url_to_content.py` | `content` | `ALTER TABLE ... ADD COLUMN url VARCHAR(2048)` (weblink support) |
| `add_original_filename_to_content.py` | `content` | Adds `original_filename` then **backfills**: from first (v1) `PlayerEdit.original_name` for edited content, else `filename` |
| `migrate_player_user_global.py` | `player_user` | **Drop & recreate**: removes old `player_id` FK, makes `user_code` globally unique, `user_name` nullable |
**Order (from `deploy.sh`):**
```
add_https_config_table.py
add_player_user_table.py
add_email_to_https_config.py
migrate_player_user_global.py
add_original_filename_to_content.py
```
> ⚠ `deploy.sh` also calls `/app/https_manager.py` — **that file is not present in this repo**, so that deployment step needs attention.
---
## 2. Legacy / Archived Components
| Component | Status | Notes |
|---|---|---|
| `app/blueprints/content_old.py` | **Dead code** | Legacy per-player content routes (`/`, `/upload`, `/<id>/edit`, `/bulk/delete`, `/upload-progress`, `/preview`, `/statistics`, `/check-duplicates`, `/<id>/groups`). Not imported by `create_app`. |
| `app/blueprints/playlist.py` | **Active but legacy** | Per-player playlist routes kept as redirects to the modern content workflow. |
| `Group` model + group routes | **Archived** | `Player` no longer has `group_id`; group routes commented out; `group_player_management.py` and `group_content` association remain for reference. |
| `nginx` stack | **Replaced by Caddy** | `data/nginx.conf`, `data/nginx-custom-domains.conf`, `data/nginx-logs/`, `data/nginx-ssl/` retained. `utils/nginx_config_reader.py` still parses it. |
| `https_manager.py` | **Missing** | Referenced by `deploy.sh` but not in repo — likely merged into `CaddyConfigGenerator`. |
| `old_code_documentation/` | **Archive** | Full legacy docs, old scripts (`blueprint_groups.py`, `add_muted_column.py`, `fix_player_user_schema.py`, `test_edit_media_*.py`, `check_fix_player.py`, `migrate_add_edit_enabled.py`), deployment guides, HTTPS analysis, player analysis. |
| `QUICK_DEPLOYMENT.md`, `deployment-commands-reference.sh` | **Active reference** | Manual deployment notes. |
---
## 3. Recommended Cleanup (optional)
- Remove `content_old.py` and `old_code_documentation/*.py` scripts that are no longer needed (keep the `.md` docs).
- Resolve the missing `https_manager.py` in `deploy.sh` (use `CaddyConfigGenerator` equivalents).
- Update `verify-deployment.sh` to reference Compose v2 and Caddy instead of `docker-compose`/nginx.
---
## 📚 Back to Index
- [README · Documentation Home](README.md)
- [01 · Architecture](01-architecture.md)
- [02 · Knowledge Graph](02-knowledge-graph.md)
- [03 · Data Model](03-data-model.md)
- [04 · Application Core](04-application-core.md)
- [05 · Blueprints & API](05-blueprints-api.md)
- [06 · Utils & Services](06-utils-services.md)
- [07 · Deployment](07-deployment.md)
- [08 · Workflows](08-workflows.md)
+108
View File
@@ -0,0 +1,108 @@
# DigiServer v2 — Project Documentation
> **Comprehensive technical documentation generated with [Graphify](https://marketplace.visualstudio.com/items?itemName=anytechiestudio.graphify-vscode)** — an interactive knowledge-graph engine for AI-assisted coding.
**Version:** 2.0.0 · **Build date:** 2025-11-12 · **Generated:** 2026-08-16
DigiServer v2 is a **digital signage management server** built with **Flask**. It manages media content (images, videos, PDFs, presentations), organizes it into **playlists**, assigns **players** (physical signage displays), and remotely **deploys and monitors** those players over SSH — including HTTPS/SSL provisioning via Caddy and an on-player photo-editing pipeline.
---
## 🗂 Documentation Index
| Document | Purpose |
|---|---|
| [**01-architecture.md**](01-architecture.md) | System overview, layers, component map, architecture diagrams |
| [**02-knowledge-graph.md**](02-knowledge-graph.md) | Graphify graph: 41 communities, god nodes, and how to use the visualizer |
| [**03-data-model.md**](03-data-model.md) | Database schema: all 10 tables, relationships, ER diagrams |
| [**04-application-core.md**](04-application-core.md) | App factory, config, extensions, middleware, CLI, templates |
| [**05-blueprints-api.md**](05-blueprints-api.md) | All 7 blueprints + full REST API reference |
| [**06-utils-services.md**](06-utils-services.md) | Background tasks, SSH deploy, Caddy manager, player build, SSO, uploads |
| [**07-deployment.md**](07-deployment.md) | Docker, Caddy, deployment pipeline, HTTPS setup, verify script |
| [**08-workflows.md**](08-workflows.md) | End-to-end flows: upload, playlists, player edits, deployment, sync |
| [**09-legacy-and-migrations.md**](09-legacy-and-migrations.md) | Migration scripts, deprecated/archived components |
**Auto-generated Graphify artifacts** (regenerate anytime — see [02-knowledge-graph.md](02-knowledge-graph.md)):
```
graphify-out/
├── graph.html ← Interactive visualizer (open in browser)
├── graph.json ← Raw graph data (631 nodes / 1162 edges)
├── GRAPH_REPORT.md ← God nodes, communities, knowledge gaps
├── COMPASS.md ← Token-optimized architecture summary
├── DOMAINS.md ← Community → domain mapping
├── intelligence.json ← AI-detected insights (god nodes, surprises)
├── graph.compact.txt ← Compact graph dump for LLM context
└── wiki/ ← Per-community + per-god-node Markdown articles
├── index.md
├── Community_0.md … Community_40.md
├── CaddyConfigGenerator.md, Content.md, HTTPSConfig.md,
├── PlayerEdit.md, PlayerUser.md, Playlist.md, User.md,
├── log_action().md, create_app().md …
```
---
## ⚡ Quick Facts
| Metric | Value |
|---|---|
| Framework | Flask 3.1 + SQLAlchemy 2.0 (SQLite) + Gunicorn |
| Application type | Digital signage content/playlist/player management |
| Graph size | **631 nodes · 1162 edges · 41 communities** |
| Blueprints | 7 active (`main`, `auth`, `admin`, `players`, `content`, `playlist`, `api`) |
| Database tables | 10 (`user`, `player`, `player_edit`, `player_feedback`, `player_user`, `content`, `group`, `playlist`, `server_log`, `https_config`) |
| Reverse proxy | Caddy 2 (automatic HTTPS / Let's Encrypt) |
| Deployment | Docker Compose (app + Caddy) + remote SSH player provisioning |
| Key externals | LibreOffice, Poppler (pdf2image), FFmpeg, sshpass/rsync |
---
## 🔑 Core Abstractions (Graphify "God Nodes")
Detected automatically from graph centrality:
1. `log_action()` — 116 edges · DB-backed audit logging (`app/utils/logger.py`)
2. `PlayerEdit` — 99 edges · on-player edit records (`app/models/player_edit.py`)
3. `PlayerUser` — 74 edges · player edit user mapping (`app/models/player_user.py`)
4. `Playlist` — 33 edges · playlist + versioning (`app/models/playlist.py`)
5. `CaddyConfigGenerator` — 31 edges · HTTPS Caddyfile generation (`app/utils/caddy_manager.py`)
6. `HTTPSConfig` — 30 edges · HTTPS settings model (`app/models/https_config.py`)
7. `Content` — 24 edges · media content model (`app/models/content.py`)
8. `User` — 19 edges · admin/user/viewer accounts (`app/models/user.py`)
9. `create_app()` — 13 edges · application factory (`app/app.py`)
10. `app/models/__init__.py` — 13 edges · models package export
---
## 🧭 Quick Navigation
### The 7 Active Blueprints
| Blueprint | Prefix | Responsibility |
|---|---|---|
| `main` | `/` | Dashboard + health check |
| `auth` | `/` | Login / logout / register / change password |
| `admin` | `/admin` | Users, HTTPS config, player build, logos, logs, leftovers |
| `players` | `/players` | Player CRUD, manage, edited media, deployment status |
| `content` | `/content` | Media library, uploads, playlist management (modern) |
| `playlist` | `/playlist` | Legacy per-player playlist routes (redirects) |
| `api` | `/api` | Player-facing REST API (auth, playlists, feedback, edits, deploy) |
### Data Model at a Glance
```mermaid
erDiagram
user ||--o{ server_log : ""
player ||--o{ player_feedback : "cascade"
player ||--o{ player_edit : "cascade"
player }o--o| playlist : "assigned"
content ||--o{ player_edit : "cascade"
content ||--o{ player_feedback : ""
playlist ||--o{ content : "playlist_content (M2M)"
content }o--o{ group : "group_content (M2M)"
player_user ||--o{ player_edit : "user_code"
https_config ||--|| https_config : "single row config"
```
> **Proceed to [01-architecture.md](01-architecture.md)** for the full system overview and diagrams.