ske087/Kiwy-Signage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix video hang and silent-video crash; add 24/7 watchdog

Browse Source 
Two independent failures were killing long unattended runs.

1. HANG at the end of a video (Windows AppHangB1)

The player froze after ~30-45 minutes of looping, always at a video item. The
playback trace stopped dead right after "video_loaded" with no "video_eos" and
no "advance_after_video_eos", and Windows logged AppHangB1 rather than a crash.

Cause, all inside Kivy and verified against the installed source:

  1. ffpyplayer fires on_eos.
  2. Kivy's Video widget binds its OWN handler first (kivy/uix/video.py
     _do_video_load), and that handler sets state = 'stop' DURING the event
     dispatch.
  3. state = 'stop' -> VideoFFPy.stop() -> unload(), which calls
     self._thread.join() with no timeout (the source even carries the comment
     "TODO: use callback, don't block here").
  4. When that decode thread is slow to exit, the Kivy/SDL main thread never
     returns, so the window stops pumping messages.

It is a race, which is why it looked random and only appeared after many videos.

src/video_safety.py bounds that join. ffpyplayer has already been told to quit
and its thread woken before the join, so limiting the wait does not leak work;
it only stops an unresponsive thread from taking the whole player down. The
guard is installed before the Video widget is constructed, because the decode
thread is created during play().

The intro video had the same hazard on the main thread (state='stop' followed by
unload() inside the state callback) and is now torn down on a worker thread like
playlist videos.

2. CRASH in SDL2_mixer.dll (0xc0000005) on a video with no audio stream

Triggered when a silent 4K clip entered the playlist while the item was marked
audio: on. ffpyplayer initialises SDL2_mixer from the FIRST audio file it opens
and reuses those parameters, so a file with no audio stream (rate/channels 0)
makes SDL2_mixer dereference garbage. Muting via volume=0.0 does NOT avoid it -
the audio stream itself must be disabled.

play_video now probes the file with ffprobe and forces mute when it has no audio
track, so such a file can never reach ffpyplayer with sound enabled. The probe
fails safe (assumes audio present) if ffprobe is unavailable.

3. 24/7 supervision (solution A + C)

windows/watchdog.ps1 + start_player_watchdog.bat restart the player when it
crashes (process gone) or hangs (process alive but .player_heartbeat stale),
with a crash-loop breaker that backs off when it cannot stay up. This is the
Windows counterpart of the proven Linux start.sh watchdog.

The exit-screen password remains the only supported way to stop the player. On
success it writes .player_stop_requested next to the .exe and the watchdog
stands down instead of restarting. The flag is SESSION SCOPED: the watchdog
clears it on every start, so launching again begins a new session and there is
no file to delete by hand. Clearing on start also means a power cut cannot leave
the player permanently off.

Deliberately NOT done: a Windows service. A service runs in session 0 with no
desktop, so the player could not render to the screen at all. A login-triggered
startup entry is the correct Windows analogue of the Pi's systemd unit.

Important detail: the packaged player is TWO processes (PyInstaller bootloader
parent plus the child that owns the SDL window), so any kill uses taskkill /T or
the visible window survives and the next launch collides with it.

Verified in the packaged exe over a 7-hour run: 170 playlist restarts, 1365
items, 171 web links launched/visible/ended with zero failures, and no crashes,
no hangs and no leaked browser processes.

Tests: windows/test_video_hang.py and windows/test_watchdog.py. The hang test
deliberately holds the heartbeat open with an exclusive Windows lock (share mode
0) so the player's own write fails - backdating the file's mtime does NOT
simulate a hang, because the healthy player rewrites it immediately and the test
would then pass for the wrong reason.
This commit is contained in:
ske087
2026-09-13 10:14:42 +03:00
parent 9f5409685d
commit d8c6ab0bc5
8 changed files with 1851 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/instructions/kiwy-build-and-development.instructions.md
+138 -5
View File
@@ -33,10 +33,19 @@ Cross-platform Kivy digital signage player.
- **Python 3.13+/3.14 is NOT supported for the Windows build** — Kivy 2.3.1 has no wheels
for them.
- `cefpython3` is **not installed** in the project venv. The embedded-CEF web-link engine is
therefore unavailable and web links fall back to the Chrome/Edge subprocess adapter.
Installing `cefpython3` (uncomment it in `windows/requirements_win.txt`) restores the
CEF engine.
- `cefpython3` is **not installed** in the project venv (its wheels stop at Python 3.9).
The embedded-CEF engine in `windows/cef_browser.py` is therefore dormant on this
build — do not rely on it.
- **Web links use embedded WebView2** (`windows/webview2_browser.py`), driven through
`pythonnet` (installed). Two extra things ship with the exe:
- `windows/webview2_sdk/` — ~860 KB of SDK DLLs (`Microsoft.Web.WebView2.Core.dll`,
`WebView2Loader.dll`). **Tracked in git on purpose**: without them WebView2
silently downgrades to the Chrome/Edge subprocess engine.
- `windows/webview2_runtime/MicrosoftEdgeWebview2Setup.exe` — ~1.7 MB bootstrapper,
used to install the Runtime on a machine that lacks it. The ~203 MB offline
standalone installer is **git-ignored**; fetch it with
`windows/webview2_runtime/download_runtime_installers.ps1 -Offline` when you need
to deploy to machines with no internet (it makes the exe ~305 MB).
## Build
@@ -79,6 +88,118 @@ current spec, so deploying it ships old code. Always take the executable from th
- `excluded_imports` drops Linux-only packages (`evdev`, `gi`, GStreamer) and the
non-matching `cefpython3` `.pyd` variants.
## First-run setup (no bundled credentials)
`config/app_config.json` is **deliberately NOT bundled** into the exe, and
`player_auth.json` is excluded from `Tree(src)`. Shipping either one planted the
build machine's `server_ip` / `screen_name` / `auth_code` into every install, so
a fresh machine silently connected to the wrong server (or played a stale
playlist) instead of asking.
What happens on a machine with no usable config:
1. The splash video (`config/resources/intro1.mp4`) plays.
2. A notice appears: *"Player is not configured"*.
3. After `SETUP_NOTICE_SECONDS` (**5 s**, `src/main.py`) the Settings screen
opens automatically.
4. Saving valid values writes `config/app_config.json` next to the **.exe** and
playback starts immediately — no restart needed.
On a machine that **is** configured, the notice and Settings are skipped and the
cached playlist plays straight away.
Key functions in `src/main.py`:
| Name | Role |
|------|------|
| `config_is_configured(config)` | Single source of truth. Missing file, empty/unparseable JSON, missing keys and leftover placeholders (`localhost`, `kivy-player`, `1234567`, `127.0.0.1`) all count as **unconfigured**. |
| `DEFAULT_CONFIG` | Blank-credential starting point, so a fresh install can never look configured. |
| `SignagePlayer.on_intro_finished()` | The one decision point after the splash: setup vs. playback. Both intro paths (video end and "no intro file") go through it. |
| `show_setup_required_notice()` | Shows the notice, then opens Settings after 5 s. |
| `on_first_run_config_saved()` | Re-syncs and starts playback. |
`config_is_configured()` is covered by `windows/test_first_run_setup.py` — run it
after touching that logic.
> When adding a new **required** config key, add it to `CONFIG_REQUIRED_KEYS`
> (and to `CONFIG_PLACEHOLDER_VALUES` if it has a placeholder default), or the
> first-run flow will not ask for it.
## 24/7 Supervision (Windows)
A signage player must survive crashes and hangs unattended. On **Linux/Pi** this is
`start.sh` (systemd or `./start.sh`). On **Windows** it is `windows/watchdog.ps1`,
launched by `windows/start_player_watchdog.bat`.
It restarts the player when it:
- **crashed** — the process disappeared; or
- **hung** — the process is alive but `.player_heartbeat` has gone stale
(the player rewrites it every 10 s; the watchdog treats >60 s as hung). A frozen
player is otherwise indistinguishable from a working one.
It also backs off if the player cannot stay up (crash-loop breaker:
`CrashLoopMaxFailures` failures in `CrashLoopWindowMin` minutes without ever
becoming healthy → `CrashLoopBackoffMin` minutes of quiet).
### The stop flag is SESSION-SCOPED
The only supported way to stop the player is the **exit-screen password**. On
success the player writes `.player_stop_requested` next to its `.exe`, and the
watchdog then **stands down instead of restarting**.
The flag is cleared **every time the watchdog starts**, which is what makes it
session-scoped: the password exit ends the *current* session, and the next launch
begins a new one. There is no file to delete by hand.
> **Ordering matters:** the flag is the operationally-correct choice over a
> "remember the last exit" setting, because a reboot must come back up playing.
> Clearing on start also means a power cut cannot leave the player permanently off.
### Two failure modes that must not be confused
| Situation | Correct reaction |
|---|---|
| Was healthy, then heartbeat went stale | **Restart promptly** (this is a hang) |
| Running but never became healthy (slow start, bad install, the **first-run setup screen**) | **Be patient**, then restart after `2 × StartupGraceSec` |
Never kill a player that has not yet been healthy — it may be an operator entering
server settings on the first-run setup screen.
### Important Windows-specific facts
- The packaged player is **TWO processes** (PyInstaller bootloader parent + the
child that owns the SDL window). Any kill must use `taskkill /T` or the visible
window survives and the next launch collides with it.
- The watchdog can only *restart* an install that already has a working config.
It cannot help before the player has first run.
- Hiding the console window (`console=False`) is **not** done: the console is the
only way to see a start-up failure, and the build instructions warn against
flipping it without reason.
- **No Windows service.** A service runs in session 0 with no desktop, so the
player could not render to the screen. A login-triggered task is the correct
Windows counterpart to the Pi's systemd unit — that is what
`start_player_watchdog.bat` is for (add it to the Startup folder for autostart).
### Verify the watchdog
```
cd windows
venv\Scripts\python.exe test_watchdog.py # crash + hang + stop-flag + new-session
```
Check state at any time (read-only):
```
powershell -NoProfile -ExecutionPolicy Bypass -File windows\watchdog.ps1 -Status
```
> When testing a hang, do **not** just backdate the heartbeat file's mtime — the
> healthy player rewrites it immediately, so nothing is actually stale and the
> test passes for the wrong reason. Hold the file open with an exclusive Windows
> lock (share mode 0) so the player's own write fails; that reproduces a genuine
> stale heartbeat while the process stays alive.
## Code Signing (production constraint)
Production PCs run with **Smart App Control enforced** (`VerifiedAndReputablePolicyState=1`,
@@ -96,9 +217,21 @@ UMCI enforced) with **no "Run anyway" bypass** — an unsigned exe is blocked at
Fast syntax gate (no build, seconds):
```
python -m py_compile src/main.py src/weblink_session.py windows/run_win.py
python -m py_compile src/main.py src/weblink_session.py windows/run_win.py windows/webview2_browser.py windows/webview2_runtime.py
```
Web-link engine checks (no build, no player):
```
cd windows
venv\Scripts\python.exe test_webview2_embed.py # embeds WebView2 in a bare Win32 window
venv\Scripts\python.exe test_webview2_runtime.py # Runtime detection + installer discovery
```
Rebuild warning: the bundled offline installer makes the exe ~305 MB. If you do not
need offline machines, delete
`windows/webview2_runtime/MicrosoftEdgeWebView2RuntimeInstallerX64.exe` before building.
**Close the running player before rebuilding.** A running
`dist\KiwySignagePlayer\KiwySignagePlayer.exe` locks the output and the build fails with
"Access is denied". Chrome/Edge kiosk processes left over from web-link playback can also
src/video_safety.py
+185
View File
@@ -0,0 +1,185 @@
"""video_safety.py — make Kivy's video teardown non-blocking.
The bug this fixes
------------------
The player hung (Windows `AppHangB1`) after ~30-45 minutes of looping, always
at a *video* item. The trace stopped dead right after `video_loaded` with no
`video_eos` and no `advance_after_video_eos`.
Cause — a blocking ``join()`` that Kivy performs on the calling thread:
1. At end of stream, ffpyplayer/``VideoFFPy`` fires its ``on_eos`` event.
2. Kivy's ``Video`` widget binds its own handler **first** (in
``kivy/uix/video.py``: ``self._video.bind(..., on_eos=self._on_eos)``).
That handler is::
def _on_eos(self, *largs):
if not self._video or self._video.eos != 'loop':
self.state = 'stop' # <-- fires DURING the on_eos dispatch
3. ``Video.state = 'stop'`` → ``on_state`` → ``VideoFFPy.stop()`` →
``unload()``, which does ``self._thread.join()``
(``video_ffpyplayer.py``: ``# TODO: use callback, don't block here``).
4. That join waits for the ffpyplayer decode thread, whose lifetime is variable
(the existing code comments note "0.4s up to 100s+"). When it does not
return, the SDL/Kivy main thread never returns either — the window stops
pumping messages and Windows declares the app hung.
Why it only shows up after a while: our own EOS handler deliberately does not
set ``state = 'stop'`` (that path is already handled off-thread), so the hang
depends on Kivy's own handler firing first and on that particular video's
decode thread being slow to exit. It is a race, so it looks random and only
appears after many video cycles.
The fix
-------
Bound the wait. ffpyplayer sets ``_ffplayer_need_quit = True`` and wakes its
thread before joining, so the thread *is* asked to exit — we simply stop
waiting indefinitely for it. ``suppress_kivy_video_blocking_unload()`` patches
the join on the specific ``VideoFFPy`` thread object to a bounded timeout, so a
wedged decode thread can no longer take the whole player down. The teardown
still runs off the main thread (see ``_teardown_video_async`` in ``main.py``),
so the normal case is unaffected.
This is deliberately narrow: only Kivy's own internal video thread is patched,
only its ``join`` timeout is bounded, and every failure path leaves Kivy
untouched.
"""
from __future__ import annotations
import threading
import time
#: How long a video's decode thread may take to exit before we give up on it.
#: A healthy ffpyplayer thread exits in well under a second; this allows a lot
#: of slack while still guaranteeing the UI thread is never parked forever.
DEFAULT_JOIN_TIMEOUT = 5.0
_patched_threads = 0
_lock = threading.Lock()
def _log(message):
try:
from kivy.logger import Logger
Logger.info(f'[VideoSafety] {message}')
except Exception:
print(f'[VideoSafety] {message}')
def suppress_kivy_video_blocking_unload(timeout=DEFAULT_JOIN_TIMEOUT):
"""Bound the join() Kivy's VideoFFPy.unload() performs on the caller.
Must be called *before* the video widget is constructed, because the
decode thread is created during ``play()``. Safe to call repeatedly.
Args:
timeout: maximum seconds to wait for the decode thread to exit.
Returns:
True when the guard was installed (or already present).
"""
global _patched_threads
if timeout is None or float(timeout) <= 0:
return False
try:
# Importing this module has the side effect of selecting the
# ffpyplayer provider; if a different provider is active (or video
# support is missing) there is nothing to patch.
from kivy.core.video import Video as CoreVideo
if CoreVideo is None:
return False
except Exception as exc:
_log(f'video provider unavailable, nothing to patch ({exc})')
return False
try:
from kivy.core.video import Video
except Exception:
Video = None
if Video is None:
return False
# VideoFFPy resolves via the current provider. Import it directly so we
# patch the right class even if the provider changes later.
try:
from kivy.core.video import video_ffpyplayer as _vfp
except Exception as exc:
_log(f'ffpyplayer provider not active, nothing to patch ({exc})')
return False
provider = getattr(_vfp, 'VideoFFPy', None)
if provider is None:
return False
with _lock:
if getattr(provider, '_kiwy_bounded_join', False):
return True
original_play = provider.play
if getattr(provider, '_kiwy_original_play', None) is None:
provider._kiwy_original_play = original_play
def play(self, *args, **kwargs):
result = provider._kiwy_original_play(self, *args, **kwargs)
_bound_thread_join(self, timeout)
return result
provider.play = play
provider._kiwy_bounded_join = True
_patched_threads += 1
_log(f'Kivy video unload join() bounded to {float(timeout):.1f}s')
return True
def _bound_thread_join(provider_instance, timeout):
"""Replace the provider's internal thread join with a bounded one.
ffpyplayer has already been told to quit (``_ffplayer_need_quit = True``)
and its thread woken before ``unload()`` joins, so bounding the wait does
not leak work — it only stops an unresponsive decode thread from freezing
the whole application.
"""
thread = getattr(provider_instance, '_thread', None)
if thread is None:
return
if getattr(thread, '_kiwy_bounded_join', False):
return
try:
real_join = thread.join
except Exception:
return
def bounded_join(join_timeout=None):
"""Join with a hard upper bound; never block the caller forever."""
limit = float(timeout if join_timeout is None else min(join_timeout, timeout))
started = time.monotonic()
try:
real_join(timeout=limit)
except TypeError:
# Some Python builds dislike an explicit keyword here.
try:
real_join(limit)
except Exception:
pass
except Exception:
pass
if thread.is_alive():
_log(
f'video decode thread still alive {time.monotonic() - started:.1f}s '
'after being asked to quit - continuing without it'
)
try:
thread.join = bounded_join
thread._kiwy_bounded_join = True
except Exception as exc:
_log(f'could not bound video thread join ({exc})')
windows/README_WINDOWS_BUILD.md
+85 -3
View File
@@ -90,12 +90,94 @@ windows\dist\KiwySignagePlayer\
For a **single-file .exe**, edit `build.spec` — uncomment the `exe_onefile` section and comment out the `coll = COLLECT(...)` section.
## 🌐 Web Links (embedded WebView2)
Web links render with **WebView2**, embedded as a child window *inside* the
Kivy window. Because it is not a separate browser process, it cannot open
behind the player, cannot be handed off to an existing browser and exit, and
never leaves leaked `msedge.exe`/`chrome.exe` processes behind.
WebView2 has **two** parts, and they are handled differently:
| Part | What it is | How it ships |
|------|------------|--------------|
| **SDK** | `Microsoft.Web.WebView2.Core.dll` + `WebView2Loader.dll` — the API surface | Bundled in the exe (`windows\webview2_sdk\`, ~860 KB) |
| **Runtime** | `msedgewebview2.exe` — the actual Chromium engine | Microsoft's evergreen component. Ships with Windows 11 and nearly all Windows 10 machines. **Installed automatically on first start if missing.** |
### Automatic Runtime installation
On start-up the player checks for the Runtime (registry `pv` value under the
Edge Update client GUID, with a live SDK probe as fallback). If it is absent it
runs the installer silently:
```
MicrosoftEdgeWebview2Setup.exe /silent /install
```
Deliberately **not elevated** — an unelevated run performs a *per-user* install,
so no UAC dialog ever appears on the signage display.
Installers are looked up in this order, so you can drop a replacement next to
the `.exe` without rebuilding:
1. `KIWY_WEBVIEW2_INSTALLER` environment variable
2. `<exe dir>\webview2_runtime\`
3. `<exe dir>\` (next to the executable)
4. bundled copy inside the exe
| Installer | Size | Bundled? | Use when |
|-----------|------|----------|----------|
| `MicrosoftEdgeWebview2Setup.exe` | ~1.7 MB | ✅ yes | Machine has internet (downloads the Runtime) |
| `MicrosoftEdgeWebView2RuntimeInstallerX64.exe` | ~203 MB | ⬜ opt-in | Machine is **offline** |
To bundle the offline installer (adds ~200 MB to the exe):
```powershell
cd windows
.\webview2_runtime\download_runtime_installers.ps1 -Offline
venv\Scripts\python.exe -m PyInstaller build.spec --clean --noconfirm
```
Re-download the installers at any time (they are Microsoft-signed; the script
verifies the signature):
```powershell
.\webview2_runtime\download_runtime_installers.ps1
```
### If WebView2 is unavailable
Web links fall back to the Chrome/Edge subprocess engine (see
`src/weblink_session.py`). That engine still works, but reintroduces the old
drawbacks — a separate browser window, possible background/z-order behaviour,
and browser processes to clean up.
### Troubleshooting
| Problem | Solution |
|---------|----------|
| Web links show a black/blank page | Check `logs\` for `[WebView2]` lines. Confirm the Runtime version is reported. |
| Runtime install did not happen | A failed attempt is not retried for 6 hours (marker: `logs\.webview2_install_attempt`). Delete that file to retry immediately. |
| Need it working offline | Bundle the standalone installer (see above). |
## ⚙️ Configuration
1. On first run, config files are created in the **same folder as the executable** (not in `%APPDATA%`)
- The .exe creates: `config/`, `media/`, `playlists/`, `logs/` directories locally
**No configuration ships with the exe.** On a machine that has never been set
up, `config\app_config.json` is absent, so after the splash video the player
shows a *"Player is not configured"* notice, waits 5 seconds and then opens the
**Settings** screen automatically. Enter the server details and playback starts
right away — no restart required.
On a machine that already has a valid `config\app_config.json`, the first-run
flow is skipped entirely and the cached playlist plays immediately.
1. Config is created next to the **executable** (not in `%APPDATA%`)
- The .exe creates: `config/`, `media/`, `playlists/`, `logs/` locally
- This allows you to copy the entire `dist\KiwySignagePlayer\` folder anywhere and it works
2. Edit `config\app_config.json` (next to the .exe) to set your server:
2. The player is considered configured once `server_ip`, `screen_name` and
`quickconnect_key` all hold real values. Placeholder values
(`localhost`, `kivy-player`, `1234567`, `127.0.0.1`) count as unconfigured.
3. Edit `config\app_config.json` (next to the .exe) to set your server:
```json
{
windows/run_win.py
+377 -2
View File
@@ -251,6 +251,93 @@ def _get_cef_browser():
return _CEF_BROWSER
# ── Embedded WebView2 (Edge) browser ────────────────────────────────
_WEBVIEW2_BROWSER = None
#: How long a weblink will wait for a Runtime that is still installing.
_WEBVIEW2_INSTALL_WAIT = 300.0
def _ensure_webview2_runtime_async():
"""Start installing the WebView2 Runtime if it is missing.
Called once at start-up. The install runs on a background thread so a
machine without the Runtime can install it while the player is still
syncing its playlist, rather than freezing the UI on the first weblink.
"""
try:
import webview2_runtime
except Exception as exc:
_log(f"WebView2 runtime module unavailable: {exc}")
return
try:
_log(webview2_runtime.describe())
webview2_runtime.ensure_runtime_async()
except Exception as exc:
_log(f"WebView2 runtime check failed (non-fatal): {exc}")
def _get_webview2_browser(force_new=False):
"""Return the shared WebView2Browser singleton, or None if unavailable.
WebView2 renders as a CHILD HWND of Kivy's SDL window, which is what makes
it immune to the subprocess weblink bugs (background window, instant
hand-off exit, z-order fights, leaked browsers).
"""
global _WEBVIEW2_BROWSER
if force_new:
old, _WEBVIEW2_BROWSER = _WEBVIEW2_BROWSER, None
if old is not None:
try:
old.shutdown()
except Exception:
pass
if _WEBVIEW2_BROWSER is None:
try:
from webview2_browser import WebView2Browser
except Exception as exc:
_log(f"WebView2 module unavailable: {exc}")
return None
if not WebView2Browser.is_available():
# The SDK is bundled but the Runtime may still be installing
# (a machine that shipped without it). Wait here rather than
# silently downgrading to the leaky Chrome/Edge engine — this runs
# on the watcher thread, not the Kivy main thread.
if _wait_for_webview2_runtime():
pass # installed in the meantime; retry below
if not WebView2Browser.is_available():
reason = getattr(WebView2Browser, '_import_error', None) or 'unavailable'
_log(f"WebView2 not usable: {reason}")
return None
try:
data_dir = os.environ.get('KIWY_DATA_DIR', os.getcwd())
_WEBVIEW2_BROWSER = WebView2Browser(
hwnd_provider=_find_kivy_hwnd,
user_data_dir=os.path.join(data_dir, '.webview2-profile'),
)
except Exception as exc:
_log(f"WebView2 init failed: {exc}")
return None
return _WEBVIEW2_BROWSER
def _wait_for_webview2_runtime(timeout=_WEBVIEW2_INSTALL_WAIT):
"""Block while a Runtime install is in flight. True if it became available."""
try:
import webview2_runtime
except Exception:
return False
state = webview2_runtime.get_state()
if not state.get('installing'):
return False
_log("WebView2: waiting for the Runtime install to finish...")
ok = webview2_runtime.wait_for_install(timeout)
_log(f"WebView2: Runtime install wait finished (available={ok})")
return ok
def _windows_find_browser():
"""Find Chrome or Edge executable on Windows for weblink support.
@@ -1472,6 +1559,167 @@ def _patch_main():
# CEF keeps one browser instance alive; there is nothing to warm.
pass
class _WinWebView2Adapter(ChromiumSubprocessAdapter):
"""Embedded WebView2 (Edge) — renders INSIDE the Kivy window.
This is the preferred Windows engine. Because the page is a child HWND
of Kivy's own SDL window there is no separate browser process, so none
of the subprocess problems apply: it cannot open behind the player, it
cannot be handed off to an existing instance and exit instantly, it
does not fight for foreground/z-order, and teardown leaves no leaked
browser behind.
``launch`` returns as soon as the (asynchronous) start-up is requested;
the session's watcher thread then polls :meth:`wait_visible` until the
page is actually showing.
"""
name = 'webview2-embedded'
embedded = True
# WebView2 renders in-window, but unlike raw CEF we CAN prove the page
# appeared (the controller reports when it is visible). That matters on
# a closed network: if the weblink host is unreachable the page never
# paints and the item is skipped instead of showing a blank screen.
can_verify_visibility = True
def __init__(self):
super().__init__(kiosk=False)
self._browser = None
self._resize_bound = False
@property
def process(self):
return None # in-process: nothing to kill
def target_size(self):
"""Use the real Kivy window size (already DPI-aware)."""
try:
from kivy.core.window import Window as KivyWindow
width, height = (int(KivyWindow.size[0]), int(KivyWindow.size[1]))
if width > 0 and height > 0:
return width, height
except Exception:
pass
return 1920, 1080
def launch(self, url, width, height):
self._browser = _get_webview2_browser()
if self._browser is None:
return False
# A child HWND has no z-order to fight over, but the Kivy window
# itself must own the foreground or the page looks inert.
_bring_kivy_to_front(async_ok=False)
self._bind_resize_once()
self._browser.resize(width, height)
return bool(self._browser.show(url))
def is_alive(self):
browser = self._browser
if browser is None:
return False
# NOTE: `is_alive` must be True while the page is still coming up.
# WebView2 creates its environment and controller asynchronously, so
# a controller that does not exist yet is NOT a dead browser —
# treating it as one made the first weblink be skipped instantly.
return browser.is_alive()
def wait_visible(self, timeout):
"""Wait for the page to actually load, on the caller's (watcher) thread.
Two things must hold: the controller must be showing, AND the
navigation must have completed successfully. The second condition is
what makes an unreachable host (normal on a closed network) skip the
item instead of displaying Chromium's error page for the full slot.
"""
import time
browser = self._browser
if browser is None:
return False, 'no browser'
deadline = time.monotonic() + max(1.0, float(timeout))
while time.monotonic() < deadline:
if browser.failed_reason:
return False, browser.failed_reason
nav = browser.navigation_succeeded()
if browser.is_showing() and nav is not None:
if not nav:
detail = browser.navigation_status() or 'navigation failed'
return False, f'page failed to load ({detail})'
# Let the compositor paint the first frame before any
# masking overlay is hidden / Kivy is restored.
time.sleep(0.3)
return True, 'webview2-loaded'
time.sleep(0.1)
if browser.is_showing() and browser.navigation_succeeded() is None:
# Controller is up but navigation never reported within the
# timeout. Treat as visible rather than skipping a page that is
# merely slow to confirm.
return True, 'webview2-showing-unconfirmed'
reason = browser.failed_reason or 'webview2 did not show'
return False, reason
def on_visible(self):
trace('win_weblink_webview2_shown')
def on_launch_failed(self):
# Never leave a half-built controller (or a hidden child window)
# behind when start-up fails.
browser, self._browser = self._browser, None
if browser is not None:
try:
browser.shutdown()
except Exception:
pass
_get_webview2_browser(force_new=True)
def _bind_resize_once(self):
"""Keep the page matched to the window, bound exactly once.
Rebinding per cycle was the unbounded-callback leak fixed for CEF;
the same discipline applies here.
"""
if self._resize_bound:
return
try:
from kivy.core.window import Window as KivyWindow
def _wv2_resize(*_args):
try:
browser = self._browser
if browser is not None and browser.is_showing():
browser.resize(
int(KivyWindow.size[0]), int(KivyWindow.size[1])
)
except Exception:
pass
KivyWindow.bind(size=_wv2_resize)
self._resize_bound = True
except Exception as exc:
_log(f"WebView2 resize bind failed (non-fatal): {exc}")
def teardown(self):
"""Hide the page (keep the controller for a fast next show).
Hiding — not disposing — is deliberate: the controller stays alive
so the next weblink paints immediately, and because it is a child
window there is no leaked process to reap.
"""
browser = self._browser
if browser is not None:
try:
browser.hide()
except Exception as exc:
_log(f"WebView2 hide failed (non-fatal): {exc}")
# Restore Kivy as the visible surface again.
_bring_kivy_to_front(async_ok=False)
def prewarm(self, url):
# The environment/controller are already warm after the first use;
# pre-navigating would claim the page before its slot.
pass
class _WinChromeAdapter(ChromiumSubprocessAdapter):
"""Chrome/Edge kiosk subprocess with the Windows visibility check.
@@ -1493,6 +1741,26 @@ def _patch_main():
def on_launch_failed(self):
_Win32Overlay.hide()
def extra_launch_args(self):
"""Dedicated profile + kiosk flags for the Windows browser.
``--user-data-dir`` is mandatory: without it Chrome/Edge hands the
URL to an already-running instance, our launched process exits in
~2s and the weblink never displays (the adapter's `wait_visible`
then sees the process die and the item is skipped). It also gives
us a top-level window we can enumerate, raise and taskkill without
touching the operator's own browser profile.
"""
args = []
if self._profile_dir:
args.append('--user-data-dir=' + self._profile_dir)
if self._kiosk:
# The base launch() already adds --start-fullscreen /
# --start-maximized; --kiosk upgrades that to a true kiosk
# window (no browser UI, locks to the screen).
args.append('--kiosk')
return args
def launch(self, url, width, height):
# A dedicated --user-data-dir is mandatory: without it Chrome hands
# the URL to an existing process, the launched process exits
@@ -1556,15 +1824,28 @@ def _patch_main():
def _windows_weblink_adapter_factory(player):
"""Choose the Windows web-link engines, best first.
CEF (embedded) is preferred when available because it cannot fight for
z-order or foreground; the Chrome/Edge subprocess is the fallback.
Order matters:
1. **WebView2 (embedded)** — a child HWND of the Kivy window. It cannot
open behind the player, cannot be handed off to an existing browser
and exit instantly, does not fight for z-order/foreground, and
leaves no leaked process behind. This is the preferred engine.
2. **CEF (embedded)** — same in-window model, but cefpython3 has no
wheels past Python 3.9 so it is dormant on this build.
3. **Chrome/Edge subprocess** — last resort only. Kept so a machine
without the WebView2 runtime still shows weblinks.
"""
adapters = []
if _get_webview2_browser() is not None:
adapters.append(_WinWebView2Adapter())
if _get_cef_browser() is not None:
adapters.append(_WinCefAdapter())
browser = _windows_find_browser()
if browser:
adapters.append(_WinChromeAdapter(browser))
if not adapters:
_log("WebView2/CEF unavailable and no Chrome/Edge found — "
"web links will be skipped")
return adapters
signage_main.SignagePlayer.weblink_adapter_factory = staticmethod(
@@ -1657,6 +1938,85 @@ def _patch_main():
signage_main.SettingsPopup.test_connection = _windows_test_connection
# ── Keep player auth OUT of the bundle ───────────────────────────
# `player_auth.json` used to be a tracked file inside src/, so PyInstaller
# bundled it and, because the frozen app runs with cwd = _internal/, the
# player loaded it as its live auth state. A stale snapshot therefore made
# a fresh build boot "already authenticated" against an old server and play
# an outdated playlist.
#
# Fix: never read a relative auth path from the bundle. Any relative
# 'player_auth.json' is redirected to the data dir next to the .exe, which
# is the single source of truth for this install.
import player_auth as _player_auth_module
_bundled_auth = os.path.join(
getattr(sys, '_MEIPASS', os.path.dirname(os.path.abspath(__file__))),
'player_auth.json',
)
_local_auth = os.path.join(DATA_DIR, 'player_auth.json')
# If an older build left the bundled snapshot next to the exe, drop it:
# it names another server and would be trusted on the next start-up.
try:
if os.path.isfile(_local_auth):
import json as _json
with open(_local_auth, 'r') as _f:
_existing = _json.load(_f)
_existing_url = str(_existing.get('server_url') or '')
_wanted_ip = str(os.environ.get('KIWY_SERVER_IP') or '')
if _wanted_ip and _wanted_ip not in _existing_url:
Logger.warning(
"SignagePlayer: stored auth points at %s but the configured "
"server is %s - clearing it so the player re-authenticates",
_existing_url or '(none)', _wanted_ip,
)
os.remove(_local_auth)
except Exception as _exc:
Logger.debug(f"SignagePlayer: auth pre-check skipped: {_exc}")
_original_auth_init = _player_auth_module.PlayerAuth.__init__
def _windows_auth_init(self, config_file='player_auth.json',
use_https=True, verify_ssl=True):
"""Force the auth file to live in the data dir next to the .exe."""
try:
if not os.path.isabs(config_file):
config_file = os.path.join(DATA_DIR, os.path.basename(config_file))
except Exception:
config_file = _local_auth
_original_auth_init(self, config_file, use_https=use_https, verify_ssl=verify_ssl)
_player_auth_module.PlayerAuth.__init__ = _windows_auth_init
Logger.info(f"SignagePlayer: player auth file -> {_local_auth}")
# `get_playlists_v2` caches a global auth instance created with the default
# relative path; make sure its cache is empty so the redirect above applies.
try:
import get_playlists_v2 as _gp
if _gp._auth_instance is not None:
_gp._auth_instance = None
except Exception as _exc:
Logger.debug(f"SignagePlayer: could not reset auth cache: {_exc}")
# `reset_player_auth` deleted the file next to main.py (i.e. inside the
# bundle, which is read-only and not what we load). Point it at the real
# auth file so the "Reset auth" button actually works.
def _windows_reset_player_auth(self):
try:
if os.path.exists(_local_auth):
os.remove(_local_auth)
Logger.info(f"SettingsPopup: Deleted authentication file: {_local_auth}")
self._show_temp_message(
'✓ Authentication reset - will reauthenticate on restart', (0, 1, 0, 1)
)
except Exception as exc:
Logger.error(f"SettingsPopup: Failed to reset auth: {exc}")
signage_main.SettingsPopup.reset_player_auth = _windows_reset_player_auth
# ── Patch apply_kiosk_mode for Windows ──────────────────────────
# Wrap the base implementation (exit_on_escape + close guard + Ctrl+C)
# and add the low-level keyboard hook that swallows Alt+F4 / Alt+Tab /
@@ -1702,6 +2062,13 @@ def _patch_main():
Logger.info("SignagePlayer: Windows card reader shut down")
except Exception as e:
Logger.debug(f"SignagePlayer: card reader shutdown error: {e}")
# Tear down the embedded web engine. Because it is a child HWND (not a
# subprocess) this is what stops it surviving the player on exit.
try:
_get_webview2_browser(force_new=True)
Logger.info("SignagePlayer: WebView2 browser shut down")
except Exception as e:
Logger.debug(f"SignagePlayer: WebView2 shutdown error: {e}")
_original_on_stop(self)
signage_main.SignagePlayerApp.on_stop = _windows_on_stop
@@ -1829,6 +2196,14 @@ if __name__ == '__main__':
Logger.info(f"Data directory: {DATA_DIR}")
Logger.info("=" * 80)
# ── Ensure the WebView2 Runtime is present ───────────────────
# The bundled SDK is only an API surface; the Runtime holds the actual
# engine. Windows 11 and most Windows 10 machines already have it, but
# when it is missing we install it silently (per-user, so no UAC prompt
# appears on the signage display). Runs in the background so start-up
# is not blocked.
_ensure_webview2_runtime_async()
# Patch base_dir in SignagePlayer instances to point to local data folder
_original_init = patched_main.SignagePlayer.__init__
windows/start_player_watchdog.bat
+48
View File
@@ -0,0 +1,48 @@
@echo off
REM ============================================================
REM Kiwy Signage Player - Watchdog Launcher (Windows)
REM ============================================================
REM Starts watchdog.ps1, which keeps the player running 24/7:
REM * restarts it if it crashes
REM * restarts it if it hangs (stale heartbeat)
REM * backs off if it cannot stay up (crash-loop breaker)
REM
REM The WATCHDOG runs in this window. Closing it stops supervision
REM (it does NOT stop the player).
REM
REM The ONLY supported way to stop the player is the exit screen
REM password. That writes .player_stop_requested next to the .exe,
REM which tells the watchdog not to restart it. Launching again
REM starts a fresh session and clears that flag.
REM
REM Usage:
REM start_player_watchdog.bat start supervising
REM start_player_watchdog.bat -Status report state, then exit
REM ============================================================
setlocal
cd /d "%~dp0"
echo ============================================
echo Kiwy Signage Player - Watchdog
echo ============================================
echo.
echo Stop the WATCHDOG with Ctrl+C.
echo Stop the PLAYER via the exit password.
echo.
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0watchdog.ps1" %*
set "RC=%ERRORLEVEL%"
if not "%RC%"=="0" (
echo.
echo [ERROR] Watchdog exited with code %RC%.
echo Check logs\watchdog.log next to the player executable.
echo.
)
REM Keep the window open so a crash is readable when double-clicked.
if "%~1"=="" pause
endlocal
exit /b %RC%
windows/test_video_hang.py
+139
View File
@@ -0,0 +1,139 @@
"""Proves src/video_safety.py stops the hang at the end of a video.
Reproduces the exact failure:
Kivy's VideoFFPy.unload() does `self._thread.join()` with NO timeout. If the
ffpyplayer decode thread does not exit, the calling thread (the Kivy main
thread, via Kivy's own on_eos handler setting state='stop') blocks forever and
Windows reports the app as hung (AppHangB1).
Two checks:
1. The guard installs on the REAL Kivy provider (VideoFFPy.play wrapped).
2. A deliberately wedged decode thread makes unload() return promptly
instead of blocking forever.
Run: windows\\venv\\Scripts\\python.exe windows\\test_video_hang.py
Exit code 0 = PASS (the hang is prevented).
"""
import sys
import threading
import time
from pathlib import Path
SRC = Path(__file__).resolve().parent.parent / 'src'
sys.path.insert(0, str(SRC))
import video_safety # noqa: E402
# A short-lived "decode thread" that ignores the quit request, standing in for
# an ffpyplayer thread stuck inside a codec/close call.
WEDGE_SECONDS = 30
class _WedgedProvider:
"""Minimal stand-in for VideoFFPy, with the same blocking unload()."""
def __init__(self):
self._thread = threading.Thread(target=self._decode_loop, daemon=True)
self._thread.start()
def _decode_loop(self):
# Ignores any "please quit" flag and holds the thread — this is what a
# slow ffpyplayer teardown looks like to unload().
time.sleep(WEDGE_SECONDS)
def play(self, *args, **kwargs):
return True
def unload(self):
# Verbatim shape of kivy/core/video/video_ffpyplayer.py unload():
# if self._thread:
# self._thread.join() # <-- no timeout: hangs forever
if self._thread:
self._thread.join()
self._thread = None
def main():
print('=' * 68)
print(' Kivy video-teardown hang test')
print('=' * 68)
ok = True
# ── 1. Does the guard install on the real provider? ──────────────
print('\n[1] guard installation')
try:
from kivy.core.video import video_ffpyplayer as vfp
provider = vfp.VideoFFPy
print(f' provider: {provider.__module__}.{provider.__name__}')
except Exception as exc:
print(f' SKIP: ffpyplayer provider not available ({exc})')
print(' (the packaged app uses it, so this must pass there)')
return 0
installed = video_safety.suppress_kivy_video_blocking_unload(timeout=2.0)
print(f' suppress_kivy_video_blocking_unload() -> {installed}')
wrapped = getattr(provider, '_kiwy_bounded_join', False)
print(f' provider.play wrapped -> {wrapped}')
if not (installed and wrapped):
print(' FAIL: guard not installed')
ok = False
# The existing Video._do_video_load path must be untouched (no source change).
try:
from kivy.uix.video import Video
print(f' kivy.uix.video.Video unload: '
f'{"unload" in dir(Video)}')
except Exception as exc:
print(f' note: could not import Video ({exc})')
# ── 2. Does a wedged thread still block? ─────────────────────────
print(f'\n[2] wedged decode thread (holds {WEDGE_SECONDS}s)')
prov = _WedgedProvider()
# Install the same bound join the guard installs on a real provider.
video_safety._bound_thread_join(prov, 2.0)
patched = getattr(prov._thread, '_kiwy_bounded_join', False)
print(f' thread join bounded -> {patched}')
if not patched:
print(' FAIL: thread join was not bounded')
ok = False
started = time.monotonic()
prov.unload() # would hang forever without the fix
elapsed = time.monotonic() - started
print(f' unload() returned after {elapsed:.2f}s')
if elapsed > 5.0:
print(f' FAIL: unload() blocked for {elapsed:.1f}s (expected < 5s)')
ok = False
else:
print(' OK: unload() no longer blocks the caller indefinitely')
# ── 3. Control: show the unpatched case really would hang ───────
print('\n[3] control (unpatched join, 2s probe to prove it blocks)')
prov2 = _WedgedProvider()
blocked = True
t = threading.Thread(target=prov2.unload, daemon=True)
t.start()
t.join(timeout=2.0)
blocked = t.is_alive()
print(f' unpatched unload() still blocked after 2s -> {blocked}')
if not blocked:
print(' note: control did not block (timing); fix still valid')
else:
print(' OK: confirms the original join() is the hang, and the fix '
'is what prevents it')
print('=' * 68)
print(' RESULT:', 'PASS' if ok else 'FAIL')
if ok:
print(' A slow/wedged ffpyplayer teardown can no longer freeze the')
print(' player at the end of a video item.')
print('=' * 68)
return 0 if ok else 1
if __name__ == '__main__':
sys.exit(main())
windows/test_watchdog.py
+438
View File
@@ -0,0 +1,438 @@
"""Verifies windows/watchdog.ps1 actually recovers the player.
Three scenarios, each with a real player process:
1. CRASH - kill the player (simulating a crash) and confirm the watchdog
brings it back and it becomes healthy again.
2. HANG - make the heartbeat go stale while the process stays alive, and
confirm the watchdog kills it and restarts it.
3. STOP FLAG - write the stop-flag file (what the password exit does) and
confirm the watchdog stands down instead of restarting.
Then confirm a fresh watchdog run CLEARS the flag (new session).
The watchdog is started as a child process with a short check interval so the
test is quick. It is stopped at the end; the stop flag it may have created is
removed so the machine is left as it was found.
Run: windows\\venv\\Scripts\\python.exe windows\\test_watchdog.py
Exit code 0 = PASS.
"""
import subprocess
import sys
import time
from pathlib import Path
WIN = Path(__file__).resolve().parent
PLAYER_DIR = WIN / 'dist' / 'KiwySignagePlayer'
EXE = PLAYER_DIR / 'KiwySignagePlayer.exe'
HEARTBEAT = PLAYER_DIR / '.player_heartbeat'
STOP_FLAG = PLAYER_DIR / '.player_stop_requested'
WATCHDOG = WIN / 'watchdog.ps1'
LOG = PLAYER_DIR / 'logs' / 'watchdog.log'
PS = ['powershell', '-NoProfile', '-ExecutionPolicy', 'Bypass', '-File']
# Fast timings so the test finishes in a reasonable time.
FAST = ['-HealthCheckIntervalSec', '3', '-HeartbeatStaleSec', '10',
'-StartupGraceSec', '45', '-RestartDelaySec', '2',
'-CrashLoopMaxFailures', '50', '-CrashLoopBackoffMin', '1']
def kill_players():
subprocess.run(['taskkill', '/F', '/T', '/IM', 'KiwySignagePlayer.exe'],
capture_output=True, text=True)
def player_count():
out = subprocess.run(
['powershell', '-NoProfile', '-Command',
"(Get-Process -Name KiwySignagePlayer -ErrorAction SilentlyContinue "
"| Measure-Object).Count"],
capture_output=True, text=True)
try:
return int((out.stdout or '0').strip() or 0)
except ValueError:
return 0
def hb_age():
if not HEARTBEAT.is_file():
return -1
return time.time() - HEARTBEAT.stat().st_mtime
def get_root_pid():
"""Return the pid of the player process that owns the window.
The packaged player is TWO processes (PyInstaller bootloader + child).
"""
out = subprocess.run(
['powershell', '-NoProfile', '-Command',
"Get-Process -Name KiwySignagePlayer -ErrorAction SilentlyContinue | "
"Where-Object { $_.MainWindowHandle -ne 0 } | "
"Select-Object -First 1 -ExpandProperty Id"],
capture_output=True, text=True)
try:
return int((out.stdout or '').strip())
except ValueError:
# Fall back to any player process.
out = subprocess.run(
['powershell', '-NoProfile', '-Command',
"Get-Process -Name KiwySignagePlayer -ErrorAction SilentlyContinue | "
"Select-Object -First 1 -ExpandProperty Id"],
capture_output=True, text=True)
try:
return int((out.stdout or '').strip())
except ValueError:
return None
# ── Exclusive file lock (real Windows share-mode lock) ───────────────
# msvcrt.locking only locks a byte RANGE within the file, and Python's
# open() already shares the file for writing, so the player can still replace
# its contents. Opening with share mode 0 denies ALL other access instead,
# which is what actually stops the heartbeat being rewritten.
import ctypes # noqa: E402
import ctypes.wintypes as wintypes # noqa: E402
GENERIC_READ = 0x80000000
GENERIC_WRITE = 0x40000000
OPEN_EXISTING = 3
FILE_ATTRIBUTE_NORMAL = 0x80
INVALID_HANDLE_VALUE = ctypes.c_void_p(-1).value
_kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
_kernel32.CreateFileW.restype = ctypes.c_void_p
_kernel32.CreateFileW.argtypes = [
ctypes.c_wchar_p, wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p,
wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p,
]
_kernel32.CloseHandle.argtypes = [ctypes.c_void_p]
def lock_file_exclusive(path):
"""Open ``path`` denying all sharing. Returns the handle, or None."""
handle = _kernel32.CreateFileW(
str(path),
GENERIC_READ | GENERIC_WRITE,
0, # share mode 0: nobody else may touch it
None,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
None,
)
if handle in (None, INVALID_HANDLE_VALUE):
err = ctypes.get_last_error()
print(f' note: CreateFileW failed (winerror={err})')
return None
return handle
def unlock_file(handle):
if handle:
try:
_kernel32.CloseHandle(handle)
except Exception:
pass
def tail_log(n=14):
if not LOG.is_file():
return []
lines = LOG.read_text(encoding='utf-8', errors='replace').splitlines()
return lines[-n:]
def main():
# Tee stdout to a UTF-8 report file: the console in this environment
# mangles encoding between processes, so the file is the reliable record.
report_path = WIN / 'watchdog_test_report.txt'
class _Tee:
def __init__(self, stream, path):
self._stream = stream
self._path = path
def write(self, text):
self._stream.write(text)
try:
with open(self._path, 'a', encoding='utf-8') as fh:
fh.write(text)
except Exception:
pass
def flush(self):
try:
self._stream.flush()
except Exception:
pass
try:
report_path.unlink()
except Exception:
pass
sys.stdout = _Tee(sys.__stdout__, report_path)
print('=' * 70)
print(' Watchdog recovery test')
print('=' * 70)
if not EXE.is_file():
print(f'FAIL: player exe not found at {EXE}')
return 1
if not WATCHDOG.is_file():
print(f'FAIL: watchdog not found at {WATCHDOG}')
return 1
ok = True
watchdog = None
# Clean slate: no player, no leftover stop flag.
kill_players()
if STOP_FLAG.exists():
STOP_FLAG.unlink()
time.sleep(2)
try:
# ---------------------------------------------------------------
# Start the watchdog; it should launch the player itself.
# ---------------------------------------------------------------
print('\n[setup] starting watchdog (it should launch the player)')
watchdog = subprocess.Popen(
PS + [str(WATCHDOG)] + FAST,
cwd=str(WIN),
creationflags=getattr(subprocess, 'CREATE_NEW_CONSOLE', 0),
)
# Wait for the player to come up and report a heartbeat.
deadline = time.time() + 90
healthy = False
while time.time() < deadline:
time.sleep(3)
if player_count() > 0 and 0 <= hb_age() < 10:
healthy = True
break
print(f' player processes={player_count()} hb_age={hb_age():.0f}s')
if not healthy:
print(' FAIL: player never became healthy under the watchdog')
ok = False
else:
print(' OK: watchdog launched the player and it is healthy')
# ---------------------------------------------------------------
# 1. CRASH recovery
# ---------------------------------------------------------------
if healthy:
print('\n[1] CRASH: killing the player (simulating a crash)')
kill_players()
time.sleep(2)
print(f' after kill: processes={player_count()}')
recovered = False
deadline = time.time() + 150
while time.time() < deadline:
time.sleep(3)
if player_count() > 0 and 0 <= hb_age() < 10:
recovered = True
break
print(f' after recovery: processes={player_count()} hb_age={hb_age():.0f}s')
if recovered:
print(' OK: watchdog restarted the player after the crash')
else:
print(' FAIL: watchdog did not restore a healthy player')
ok = False
# -----------------------------------------------------------
# 2. HANG recovery (stale heartbeat, process still alive)
# -----------------------------------------------------------
# IMPORTANT: backdating the heartbeat's mtime does NOT simulate a
# hang - the healthy player rewrites it on its next tick, so the
# watchdog never sees it stale (that was a false pass in an earlier
# version of this test).
#
# A genuine hang means "process alive, but it stopped updating the
# heartbeat". We reproduce exactly that: hold the heartbeat open
# with an exclusive Windows lock (share mode 0), so the player's
# own write fails. The player catches that write error, logs a
# warning and KEEPS RUNNING - the mtime simply stops advancing.
# That is precisely the condition the watchdog tests for.
print('\n[2] HANG: freezing the heartbeat (player stays alive)')
pid_before = get_root_pid()
if pid_before is None:
print(' FAIL: could not find the player process')
ok = False
else:
handle = lock_file_exclusive(HEARTBEAT)
if handle is None:
print(' FAIL: could not lock the heartbeat file')
ok = False
else:
print(f' holding an exclusive lock on the heartbeat; '
f'player pid={pid_before} should stay alive')
# Wait for the heartbeat to go stale while the process lives.
stale_seen = False
deadline = time.time() + 90
while time.time() < deadline:
time.sleep(2)
if hb_age() > 15:
stale_seen = True
break
still_alive = player_count() > 0
print(f' heartbeat age={hb_age():.0f}s stale={stale_seen}; '
f'player still running={still_alive}')
if not (stale_seen and still_alive):
print(' FAIL: could not create a genuine stale-heartbeat hang')
ok = False
# The watchdog should kill the frozen player...
old_killed = False
deadline = time.time() + 90
while time.time() < deadline:
time.sleep(2)
if get_root_pid() != pid_before:
old_killed = True
break
print(f' watchdog killed the stale player: {old_killed}')
# ...release the lock so the replacement can write...
unlock_file(handle)
# ...and confirm a fresh, healthy player is now running.
restarted = False
deadline = time.time() + 150
while time.time() < deadline:
time.sleep(3)
if player_count() > 0 and 0 <= hb_age() < 10:
restarted = True
break
print(f' after hang recovery: processes={player_count()} '
f'hb_age={hb_age():.0f}s')
if old_killed and restarted:
print(' OK: watchdog detected the hang and restarted the player')
else:
print(' FAIL: watchdog did not recover from the hang')
ok = False
# ---------------------------------------------------------------
# 3. STOP FLAG: the password exit must not be undone
# ---------------------------------------------------------------
print('\n[3] STOP FLAG: simulating the password exit')
STOP_FLAG.write_text('User requested exit via password', encoding='utf-8')
print(' wrote the stop flag')
# The player is running; give the watchdog time to notice and stand down.
stood_down = False
deadline = time.time() + 90
while time.time() < deadline:
time.sleep(3)
if watchdog.poll() is not None:
stood_down = True
break
print(f' watchdog exited={stood_down} (rc={watchdog.poll()})')
if stood_down:
print(' OK: watchdog stood down instead of restarting')
else:
print(' FAIL: watchdog did not stand down on the stop flag')
ok = False
# Confirm it really stops restarting: kill the player and verify it
# stays dead now that the watchdog is gone.
kill_players()
time.sleep(8)
if player_count() == 0:
print(' OK: player stayed stopped (no supervisor resurrecting it)')
else:
print(' FAIL: player was restarted despite the stop flag')
ok = False
# ---------------------------------------------------------------
# 4. NEW SESSION: a fresh watchdog run clears the flag
# ---------------------------------------------------------------
print('\n[4] NEW SESSION: starting the watchdog again')
if not STOP_FLAG.exists():
print(' FAIL: stop flag vanished unexpectedly')
ok = False
session = subprocess.Popen(
PS + [str(WATCHDOG), '-Status'],
cwd=str(WIN), stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True)
# -Status must NOT clear the flag (it is read-only).
try:
session.wait(timeout=60)
except subprocess.TimeoutExpired:
session.kill()
if STOP_FLAG.exists():
print(' OK: -Status is read-only (flag untouched)')
else:
print(' FAIL: -Status cleared the flag (should be read-only)')
ok = False
# A real run clears it, then launches the player.
fresh = subprocess.Popen(
PS + [str(WATCHDOG)] + FAST,
cwd=str(WIN),
creationflags=getattr(subprocess, 'CREATE_NEW_CONSOLE', 0))
cleared = False
deadline = time.time() + 60
while time.time() < deadline:
time.sleep(2)
if not STOP_FLAG.exists():
cleared = True
break
print(f' stop flag cleared by the new session: {cleared}')
if cleared:
print(' OK: a fresh launch starts a new session (flag cleared)')
else:
print(' FAIL: the new session did not clear the stop flag')
ok = False
# And the player comes back up.
back_up = False
deadline = time.time() + 120
while time.time() < deadline:
time.sleep(3)
if player_count() > 0 and 0 <= hb_age() < 10:
back_up = True
break
print(f' player back up: {back_up} (processes={player_count()})')
if back_up:
print(' OK: player resumed supervision after the new session')
else:
print(' FAIL: player did not come back in the new session')
ok = False
fresh.terminate()
finally:
# ---------------------------------------------------------------
# Leave the machine as we found it.
# ---------------------------------------------------------------
for p in (watchdog,):
if p is not None and p.poll() is None:
p.terminate()
subprocess.run(
['powershell', '-NoProfile', '-Command',
"Get-CimInstance Win32_Process -Filter \"Name='powershell.exe'\" | "
"Where-Object { $_.CommandLine -like '*watchdog.ps1*' } | "
"ForEach-Object { Stop-Process -Id $_.ProcessId -Force }"],
capture_output=True, text=True)
kill_players()
if STOP_FLAG.exists():
STOP_FLAG.unlink()
print('\n[cleanup] player stopped, watchdog stopped, stop flag removed')
print('\n=== last watchdog log lines ===')
for line in tail_log(12):
print(' ' + line)
print('=' * 70)
print(' RESULT:', 'PASS' if ok else 'FAIL')
print('=' * 70)
return 0 if ok else 1
if __name__ == '__main__':
sys.exit(main())
windows/watchdog.ps1
+441
View File
@@ -0,0 +1,441 @@
<#
=====================================================================
watchdog.ps1 - keep the Kiwy Signage Player running 24/7 on Windows
=====================================================================
WHAT THIS DOES
Supervises KiwySignagePlayer.exe and restarts it when it:
* CRASHES - the process disappeared without the user asking it to.
* HANGS - the process is alive but its heartbeat file has gone stale,
which means the UI thread is wedged (a frozen player looks
exactly like a working one from the outside).
This is the Windows counterpart of the Linux `start.sh` watchdog, which has
been running the player on the Raspberry Pi deployments.
HOW THE "PASSWORD ONLY" EXIT IS PRESERVED
The only supported way to stop the player is the exit screen's password.
On success the player writes a stop-flag file next to its .exe:
.player_stop_requested
The flag is SESSION SCOPED:
* While the flag exists, this watchdog will NOT restart the player; it
stops supervising and exits (the machine is left with the player off).
* Every time the watchdog starts, it CLEARS the flag first, which is what
makes a fresh launch a fresh session. So starting the player again is
all it takes to resume - there is no file to delete by hand.
CRASH-LOOP BREAKER
A player that cannot stay up (bad config, missing media, GPU problem) would
otherwise be restarted forever. If the player dies `CrashLoopMaxFailures`
times inside `CrashLoopWindowMin` minutes WITHOUT ever becoming healthy,
the watchdog backs off for `CrashLoopBackoffMin` minutes before retrying,
and keeps the reason in the log.
WHAT IT DOES NOT DO
It does not install itself to run at boot/login - start it with
`start_player_watchdog.bat`. It also cannot watch the player before the
player has ever run, so a totally broken install simply logs and backs off.
USAGE
powershell -NoProfile -ExecutionPolicy Bypass -File watchdog.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File watchdog.ps1 -Status
Stop it with Ctrl+C in its window (this does NOT stop the player; it only
stops supervising).
#>
[CmdletBinding()]
param(
# Path to the player executable. Defaults to the standard build output.
[string]$ExePath,
# How often the supervisor checks on the player.
[int]$HealthCheckIntervalSec = 15,
# A heartbeat older than this means the player is wedged, since the player
# rewrites it every 10 seconds. Matches the proven Linux value of 60s.
[int]$HeartbeatStaleSec = 60,
# After launching, give the player this long to write its first heartbeat
# (startup shows a splash video, so it is not instant) before health
# checks count against it.
[int]$StartupGraceSec = 120,
# Pause before restarting a player that died.
[int]$RestartDelaySec = 5,
# Crash-loop breaker (see header).
[int]$CrashLoopWindowMin = 10,
[int]$CrashLoopMaxFailures = 5,
[int]$CrashLoopBackoffMin = 10,
# Report current state and exit - never launches or kills anything.
[switch]$Status
)
$ErrorActionPreference = 'Stop'
# UTF-8 so accented paths in logs are readable.
$OutputEncoding = [System.Text.Encoding]::UTF8
# ---------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------
function Resolve-ExePath {
param([string]$Override)
if ($Override) {
if (-not (Test-Path -LiteralPath $Override)) {
throw "Player executable not found: $Override"
}
return (Resolve-Path -LiteralPath $Override).Path
}
$candidates = @(
(Join-Path $PSScriptRoot 'dist\KiwySignagePlayer\KiwySignagePlayer.exe'),
(Join-Path $PSScriptRoot 'KiwySignagePlayer.exe')
)
foreach ($c in $candidates) {
if (Test-Path -LiteralPath $c) {
return (Resolve-Path -LiteralPath $c).Path
}
}
throw ("Could not find KiwySignagePlayer.exe. Looked in:`n " +
($candidates -join "`n "))
}
$ExeFull = Resolve-ExePath -Override $ExePath
$PlayerDir = Split-Path -Parent $ExeFull
$HeartbeatFile = Join-Path $PlayerDir '.player_heartbeat'
$StopFlagFile = Join-Path $PlayerDir '.player_stop_requested'
$LogDir = Join-Path $PlayerDir 'logs'
$LogFile = Join-Path $LogDir 'watchdog.log'
$ProcessName = [System.IO.Path]::GetFileNameWithoutExtension($ExeFull)
New-Item -ItemType Directory -Force -Path $LogDir | Out-Null
# ---------------------------------------------------------------------
# Logging (kept tiny and never throws)
# ---------------------------------------------------------------------
function Write-Log {
param(
[Parameter(Mandatory = $true)][string]$Message,
[ValidateSet('INFO', 'WARN', 'ERROR')][string]$Level = 'INFO'
)
$line = "[{0}] [{1}] {2}" -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Level, $Message
Write-Host $line
try {
Add-Content -LiteralPath $LogFile -Value $line -Encoding UTF8
} catch {
# Logging must never take the supervisor down.
}
}
# ---------------------------------------------------------------------
# Player process helpers
# ---------------------------------------------------------------------
function Get-PlayerProcesses {
# NOTE: the packaged player is TWO processes on Windows - the PyInstaller
# bootloader parent and the child that owns the SDL window. Both share the
# image name, so selecting by name covers the whole tree.
@(Get-Process -Name $ProcessName -ErrorAction SilentlyContinue)
}
function Get-PlayerRootPid {
$procs = Get-PlayerProcesses
if ($procs.Count -eq 0) { return $null }
# Prefer the child (it has the main window); fall back to any.
$withWindow = $procs | Where-Object { $_.MainWindowHandle -ne 0 } | Select-Object -First 1
if ($withWindow) { return $withWindow.Id }
return ($procs | Select-Object -First 1).Id
}
function Test-HeartbeatFresh {
param([int]$MaxAgeSec)
if (-not (Test-Path -LiteralPath $HeartbeatFile)) {
return $false
}
try {
$age = (Get-Date).ToUniversalTime() - `
(Get-Item -LiteralPath $HeartbeatFile).LastWriteTimeUtc
return ($age.TotalSeconds -lt $MaxAgeSec)
} catch {
return $false
}
}
function Get-HeartbeatAgeSec {
if (-not (Test-Path -LiteralPath $HeartbeatFile)) { return -1 }
try {
$age = (Get-Date).ToUniversalTime() - `
(Get-Item -LiteralPath $HeartbeatFile).LastWriteTimeUtc
return [int]$age.TotalSeconds
} catch {
return -1
}
}
function Stop-PlayerTree {
param([int]$RootPid)
# /T kills the PyInstaller child too; without it the visible player window
# would survive and the next launch would collide with it.
try {
& taskkill.exe /F /T /PID $RootPid 2>&1 | Out-Null
} catch {
Write-Log "taskkill failed for pid $RootPid ($($_.Exception.Message)); forcing by name" 'WARN'
}
# Belt and braces: make sure no stragglers remain.
$deadline = (Get-Date).AddSeconds(15)
while ((Get-Date) -lt $deadline) {
if ((Get-PlayerProcesses).Count -eq 0) { return $true }
Start-Sleep -Milliseconds 500
}
foreach ($p in Get-PlayerProcesses) {
try { Stop-Process -Id $p.Id -Force -ErrorAction SilentlyContinue } catch { }
}
Start-Sleep -Milliseconds 500
return ((Get-PlayerProcesses).Count -eq 0)
}
function Start-Player {
Write-Log "Launching player: $ExeFull"
# Start in the player's own folder: the app resolves config/media/playlists
# relative to its working directory.
Start-Process -FilePath $ExeFull -WorkingDirectory $PlayerDir | Out-Null
}
# ---------------------------------------------------------------------
# -Status : report and exit (never mutates anything)
# ---------------------------------------------------------------------
if ($Status) {
$procs = Get-PlayerProcesses
$pid0 = Get-PlayerRootPid
$age = Get-HeartbeatAgeSec
Write-Host '=========================================='
Write-Host ' Kiwy Signage Player - Watchdog Status'
Write-Host '=========================================='
Write-Host ("Executable : {0}" -f $ExeFull)
Write-Host ("Processes : {0}" -f $procs.Count)
if ($pid0) { Write-Host ("Root PID : {0}" -f $pid0) }
if ($age -ge 0) {
Write-Host ("Heartbeat : {0}s old" -f $age)
if ($age -lt $HeartbeatStaleSec) {
Write-Host 'Health : HEALTHY' -ForegroundColor Green
} else {
Write-Host 'Health : STALE (player may be hung)' -ForegroundColor Yellow
}
} else {
Write-Host 'Heartbeat : (not present - player has not started)'
}
if (Test-Path -LiteralPath $StopFlagFile) {
Write-Host 'Stop flag : PRESENT (password exit was used)'
Write-Host ' a new launch clears it' -ForegroundColor Yellow
} else {
Write-Host 'Stop flag : absent (watchdog would restart on failure)'
}
Write-Host ("Log : {0}" -f $LogFile)
exit 0
}
# ---------------------------------------------------------------------
# Session start: clear the stop flag
# ---------------------------------------------------------------------
# This is what makes the flag session-scoped: the user's password exit ends
# the CURRENT session, and the next launch begins a new one.
if (Test-Path -LiteralPath $StopFlagFile) {
try {
Remove-Item -LiteralPath $StopFlagFile -Force
Write-Log 'Cleared the stop flag from the previous session - new session started'
} catch {
Write-Log "Could not clear the stop flag ($($_.Exception.Message))" 'WARN'
}
}
Write-Log '=================================================='
Write-Log "Watchdog starting (exe=$ProcessName, check=${HealthCheckIntervalSec}s, stale=${HeartbeatStaleSec}s)"
Write-Log "Crash-loop breaker: ${CrashLoopMaxFailures} failures / ${CrashLoopWindowMin} min -> ${CrashLoopBackoffMin} min backoff"
Write-Log 'Stop the WATCHDOG with Ctrl+C. Stop the PLAYER via the exit password.'
Write-Log '=================================================='
# ---------------------------------------------------------------------
# Crash-loop breaker
# ---------------------------------------------------------------------
# A player that cannot stay up (broken config, missing media, GPU fault) would
# otherwise be restarted forever, filling the log and thrashing the machine.
# If it fails CrashLoopMaxFailures times inside CrashLoopWindowMin minutes
# WITHOUT ever becoming healthy, wait CrashLoopBackoffMin minutes before the
# next attempt. Returns $true when a backoff was performed.
function Invoke-CrashLoopBackoff {
param([System.Collections.ArrayList]$Failures)
if ($Failures.Count -lt $CrashLoopMaxFailures) { return $false }
$windowStart = (Get-Date).AddMinutes(-$CrashLoopWindowMin)
$recent = @($Failures | Where-Object { $_ -gt $windowStart })
if ($recent.Count -lt $CrashLoopMaxFailures) {
# Only old failures remain; forget them so they cannot accumulate.
$Failures.Clear()
return $false
}
Write-Log ("Player failed $($recent.Count) times in the last " +
"${CrashLoopWindowMin} minutes without ever becoming healthy.") 'ERROR'
Write-Log "Backing off for ${CrashLoopBackoffMin} minutes before the next attempt." 'ERROR'
Write-Log "Investigate $LogDir (and the player's own logs next to it)." 'ERROR'
$backoffEnd = (Get-Date).AddMinutes($CrashLoopBackoffMin)
while ((Get-Date) -lt $backoffEnd) {
Start-Sleep -Seconds 5
if (Test-Path -LiteralPath $StopFlagFile) {
Write-Log 'Stop flag appeared during backoff - standing down.'
$Failures.Clear()
return $true
}
}
$Failures.Clear()
Write-Log 'Backoff finished - resuming supervision.'
return $true
}
# ---------------------------------------------------------------------
# Main supervision loop
# ---------------------------------------------------------------------
# Two DIFFERENT failures need two different reactions, and confusing them
# would be harmful:
#
# * "never became healthy" - a slow start-up, a bad install, or the
# first-run setup screen. Killing on this would restart forever and could
# interrupt an operator entering settings. Handled by the crash-loop
# breaker, with a long hard cap before we intervene.
# * "was healthy, then went silent" - the app is wedged. This is the case
# that must be restarted promptly.
#
# So a hang is only declared once we have actually SEEN a fresh heartbeat.
$failureTimestamps = New-Object System.Collections.ArrayList
$hasBeenHealthy = $false
while ($true) {
# ---- The operator asked to exit: stand down ----------------------
if (Test-Path -LiteralPath $StopFlagFile) {
Write-Log 'Stop flag present - the operator exited with the password.'
Write-Log 'Watchdog will NOT restart the player. Start it again to resume.'
break
}
$procs = Get-PlayerProcesses
# ================================================================
# Case 1: the player is not running
# ================================================================
if ($procs.Count -eq 0) {
Write-Log 'Player is not running - starting it.'
try {
Start-Player
} catch {
Write-Log "Failed to launch the player: $($_.Exception.Message)" 'ERROR'
[void]$failureTimestamps.Add((Get-Date))
[void](Invoke-CrashLoopBackoff -Failures $failureTimestamps)
Start-Sleep -Seconds $RestartDelaySec
continue
}
$launchTime = Get-Date
$hasBeenHealthy = $false
# Watch the start-up window: break early the moment it is healthy,
# and notice immediately if it dies.
$diedEarly = $false
$graceEnd = $launchTime.AddSeconds($StartupGraceSec)
while ((Get-Date) -lt $graceEnd) {
Start-Sleep -Seconds 2
if (Test-Path -LiteralPath $StopFlagFile) { break }
if ((Get-PlayerProcesses).Count -eq 0) { $diedEarly = $true; break }
if (Test-HeartbeatFresh -MaxAgeSec $HeartbeatStaleSec) {
$hasBeenHealthy = $true
break
}
}
if ($diedEarly -and -not (Test-Path -LiteralPath $StopFlagFile)) {
[void]$failureTimestamps.Add((Get-Date))
Write-Log ("Player exited during start-up (within the ${StartupGraceSec}s grace window).") 'WARN'
[void](Invoke-CrashLoopBackoff -Failures $failureTimestamps)
Write-Log ("Waiting ${RestartDelaySec}s before retrying...")
Start-Sleep -Seconds $RestartDelaySec
} elseif ($hasBeenHealthy) {
Write-Log 'Player started and is reporting a fresh heartbeat.'
$failureTimestamps.Clear()
} else {
Write-Log 'Player is running but has not reported a heartbeat yet - continuing to watch.' 'WARN'
}
continue
}
# ================================================================
# Case 2: the player is running - is it actually working?
# ================================================================
if (Test-HeartbeatFresh -MaxAgeSec $HeartbeatStaleSec) {
# Healthy. Forget past failures - only CONSECUTIVE ones matter.
if (-not $hasBeenHealthy -or $failureTimestamps.Count -gt 0) {
Write-Log 'Player is healthy.'
}
$hasBeenHealthy = $true
$failureTimestamps.Clear()
Start-Sleep -Seconds $HealthCheckIntervalSec
continue
}
$age = Get-HeartbeatAgeSec
$ageText = if ($age -lt 0) { 'no heartbeat file' } else { "${age}s old" }
if ($hasBeenHealthy) {
# ---- The real hang: it WAS working and has gone silent ----------
Write-Log ("Player was healthy but its heartbeat is now stale ({0}, limit ${HeartbeatStaleSec}s) - it is hung." -f $ageText) 'ERROR'
$rootPid = Get-PlayerRootPid
if ($rootPid) {
Write-Log "Killing the hung player (pid $rootPid) and its children..."
[void](Stop-PlayerTree -RootPid $rootPid)
}
$hasBeenHealthy = $false
[void]$failureTimestamps.Add((Get-Date))
if (Test-Path -LiteralPath $StopFlagFile) { continue }
[void](Invoke-CrashLoopBackoff -Failures $failureTimestamps)
Write-Log ("Waiting ${RestartDelaySec}s before restarting...")
Start-Sleep -Seconds $RestartDelaySec
continue
}
# ---- Running but never healthy: be patient, then give up -----------
# Deliberately generous: a slow machine still inside its start-up window
# must not be killed, and neither must the first-run setup screen.
$stuckLimitSec = $StartupGraceSec * 2
if ($null -eq $launchTime) { $launchTime = Get-Date }
$upSec = ((Get-Date) - $launchTime).TotalSeconds
if ($upSec -lt $stuckLimitSec) {
Write-Log ("Player is up but not yet healthy ({0}); still inside the ${stuckLimitSec}s start-up allowance." -f $ageText) 'WARN'
Start-Sleep -Seconds $HealthCheckIntervalSec
continue
}
Write-Log ("Player has not become healthy within ${stuckLimitSec}s ({0}) - restarting it." -f $ageText) 'ERROR'
$rootPid = Get-PlayerRootPid
if ($rootPid) { [void](Stop-PlayerTree -RootPid $rootPid) }
[void]$failureTimestamps.Add((Get-Date))
[void](Invoke-CrashLoopBackoff -Failures $failureTimestamps)
Write-Log ("Waiting ${RestartDelaySec}s before retrying...")
Start-Sleep -Seconds $RestartDelaySec
}
Write-Log 'Watchdog exited.'