From d8c6ab0bc586ca6244c9869e25f4dd9a68eb73ee Mon Sep 17 00:00:00 2001 From: ske087 Date: Sun, 13 Sep 2026 10:14:42 +0300 Subject: [PATCH] Fix video hang and silent-video crash; add 24/7 watchdog Two independent failures were killing long unattended runs. 1. HANG at the end of a video (Windows AppHangB1) The player froze after ~30-45 minutes of looping, always at a video item. The playback trace stopped dead right after "video_loaded" with no "video_eos" and no "advance_after_video_eos", and Windows logged AppHangB1 rather than a crash. Cause, all inside Kivy and verified against the installed source: 1. ffpyplayer fires on_eos. 2. Kivy's Video widget binds its OWN handler first (kivy/uix/video.py _do_video_load), and that handler sets state = 'stop' DURING the event dispatch. 3. state = 'stop' -> VideoFFPy.stop() -> unload(), which calls self._thread.join() with no timeout (the source even carries the comment "TODO: use callback, don't block here"). 4. When that decode thread is slow to exit, the Kivy/SDL main thread never returns, so the window stops pumping messages. It is a race, which is why it looked random and only appeared after many videos. src/video_safety.py bounds that join. ffpyplayer has already been told to quit and its thread woken before the join, so limiting the wait does not leak work; it only stops an unresponsive thread from taking the whole player down. The guard is installed before the Video widget is constructed, because the decode thread is created during play(). The intro video had the same hazard on the main thread (state='stop' followed by unload() inside the state callback) and is now torn down on a worker thread like playlist videos. 2. CRASH in SDL2_mixer.dll (0xc0000005) on a video with no audio stream Triggered when a silent 4K clip entered the playlist while the item was marked audio: on. ffpyplayer initialises SDL2_mixer from the FIRST audio file it opens and reuses those parameters, so a file with no audio stream (rate/channels 0) makes SDL2_mixer dereference garbage. Muting via volume=0.0 does NOT avoid it - the audio stream itself must be disabled. play_video now probes the file with ffprobe and forces mute when it has no audio track, so such a file can never reach ffpyplayer with sound enabled. The probe fails safe (assumes audio present) if ffprobe is unavailable. 3. 24/7 supervision (solution A + C) windows/watchdog.ps1 + start_player_watchdog.bat restart the player when it crashes (process gone) or hangs (process alive but .player_heartbeat stale), with a crash-loop breaker that backs off when it cannot stay up. This is the Windows counterpart of the proven Linux start.sh watchdog. The exit-screen password remains the only supported way to stop the player. On success it writes .player_stop_requested next to the .exe and the watchdog stands down instead of restarting. The flag is SESSION SCOPED: the watchdog clears it on every start, so launching again begins a new session and there is no file to delete by hand. Clearing on start also means a power cut cannot leave the player permanently off. Deliberately NOT done: a Windows service. A service runs in session 0 with no desktop, so the player could not render to the screen at all. A login-triggered startup entry is the correct Windows analogue of the Pi's systemd unit. Important detail: the packaged player is TWO processes (PyInstaller bootloader parent plus the child that owns the SDL window), so any kill uses taskkill /T or the visible window survives and the next launch collides with it. Verified in the packaged exe over a 7-hour run: 170 playlist restarts, 1365 items, 171 web links launched/visible/ended with zero failures, and no crashes, no hangs and no leaked browser processes. Tests: windows/test_video_hang.py and windows/test_watchdog.py. The hang test deliberately holds the heartbeat open with an exclusive Windows lock (share mode 0) so the player's own write fails - backdating the file's mtime does NOT simulate a hang, because the healthy player rewrites it immediately and the test would then pass for the wrong reason. --- ...kiwy-build-and-development.instructions.md | 143 +++++- src/video_safety.py | 185 ++++++++ windows/README_WINDOWS_BUILD.md | 88 +++- windows/run_win.py | 379 ++++++++++++++- windows/start_player_watchdog.bat | 48 ++ windows/test_video_hang.py | 139 ++++++ windows/test_watchdog.py | 438 +++++++++++++++++ windows/watchdog.ps1 | 441 ++++++++++++++++++ 8 files changed, 1851 insertions(+), 10 deletions(-) create mode 100644 src/video_safety.py create mode 100644 windows/start_player_watchdog.bat create mode 100644 windows/test_video_hang.py create mode 100644 windows/test_watchdog.py create mode 100644 windows/watchdog.ps1 diff --git a/.github/instructions/kiwy-build-and-development.instructions.md b/.github/instructions/kiwy-build-and-development.instructions.md index 0e8c4cf..4fb8a03 100644 --- a/.github/instructions/kiwy-build-and-development.instructions.md +++ b/.github/instructions/kiwy-build-and-development.instructions.md @@ -33,10 +33,19 @@ Cross-platform Kivy digital signage player. - **Python 3.13+/3.14 is NOT supported for the Windows build** — Kivy 2.3.1 has no wheels for them. -- `cefpython3` is **not installed** in the project venv. The embedded-CEF web-link engine is - therefore unavailable and web links fall back to the Chrome/Edge subprocess adapter. - Installing `cefpython3` (uncomment it in `windows/requirements_win.txt`) restores the - CEF engine. +- `cefpython3` is **not installed** in the project venv (its wheels stop at Python 3.9). + The embedded-CEF engine in `windows/cef_browser.py` is therefore dormant on this + build — do not rely on it. +- **Web links use embedded WebView2** (`windows/webview2_browser.py`), driven through + `pythonnet` (installed). Two extra things ship with the exe: + - `windows/webview2_sdk/` — ~860 KB of SDK DLLs (`Microsoft.Web.WebView2.Core.dll`, + `WebView2Loader.dll`). **Tracked in git on purpose**: without them WebView2 + silently downgrades to the Chrome/Edge subprocess engine. + - `windows/webview2_runtime/MicrosoftEdgeWebview2Setup.exe` — ~1.7 MB bootstrapper, + used to install the Runtime on a machine that lacks it. The ~203 MB offline + standalone installer is **git-ignored**; fetch it with + `windows/webview2_runtime/download_runtime_installers.ps1 -Offline` when you need + to deploy to machines with no internet (it makes the exe ~305 MB). ## Build @@ -79,6 +88,118 @@ current spec, so deploying it ships old code. Always take the executable from th - `excluded_imports` drops Linux-only packages (`evdev`, `gi`, GStreamer) and the non-matching `cefpython3` `.pyd` variants. +## First-run setup (no bundled credentials) + +`config/app_config.json` is **deliberately NOT bundled** into the exe, and +`player_auth.json` is excluded from `Tree(src)`. Shipping either one planted the +build machine's `server_ip` / `screen_name` / `auth_code` into every install, so +a fresh machine silently connected to the wrong server (or played a stale +playlist) instead of asking. + +What happens on a machine with no usable config: + +1. The splash video (`config/resources/intro1.mp4`) plays. +2. A notice appears: *"Player is not configured"*. +3. After `SETUP_NOTICE_SECONDS` (**5 s**, `src/main.py`) the Settings screen + opens automatically. +4. Saving valid values writes `config/app_config.json` next to the **.exe** and + playback starts immediately — no restart needed. + +On a machine that **is** configured, the notice and Settings are skipped and the +cached playlist plays straight away. + +Key functions in `src/main.py`: + +| Name | Role | +|------|------| +| `config_is_configured(config)` | Single source of truth. Missing file, empty/unparseable JSON, missing keys and leftover placeholders (`localhost`, `kivy-player`, `1234567`, `127.0.0.1`) all count as **unconfigured**. | +| `DEFAULT_CONFIG` | Blank-credential starting point, so a fresh install can never look configured. | +| `SignagePlayer.on_intro_finished()` | The one decision point after the splash: setup vs. playback. Both intro paths (video end and "no intro file") go through it. | +| `show_setup_required_notice()` | Shows the notice, then opens Settings after 5 s. | +| `on_first_run_config_saved()` | Re-syncs and starts playback. | + +`config_is_configured()` is covered by `windows/test_first_run_setup.py` — run it +after touching that logic. + +> When adding a new **required** config key, add it to `CONFIG_REQUIRED_KEYS` +> (and to `CONFIG_PLACEHOLDER_VALUES` if it has a placeholder default), or the +> first-run flow will not ask for it. + +## 24/7 Supervision (Windows) + +A signage player must survive crashes and hangs unattended. On **Linux/Pi** this is +`start.sh` (systemd or `./start.sh`). On **Windows** it is `windows/watchdog.ps1`, +launched by `windows/start_player_watchdog.bat`. + +It restarts the player when it: + +- **crashed** — the process disappeared; or +- **hung** — the process is alive but `.player_heartbeat` has gone stale + (the player rewrites it every 10 s; the watchdog treats >60 s as hung). A frozen + player is otherwise indistinguishable from a working one. + +It also backs off if the player cannot stay up (crash-loop breaker: +`CrashLoopMaxFailures` failures in `CrashLoopWindowMin` minutes without ever +becoming healthy → `CrashLoopBackoffMin` minutes of quiet). + +### The stop flag is SESSION-SCOPED + +The only supported way to stop the player is the **exit-screen password**. On +success the player writes `.player_stop_requested` next to its `.exe`, and the +watchdog then **stands down instead of restarting**. + +The flag is cleared **every time the watchdog starts**, which is what makes it +session-scoped: the password exit ends the *current* session, and the next launch +begins a new one. There is no file to delete by hand. + +> **Ordering matters:** the flag is the operationally-correct choice over a +> "remember the last exit" setting, because a reboot must come back up playing. +> Clearing on start also means a power cut cannot leave the player permanently off. + +### Two failure modes that must not be confused + +| Situation | Correct reaction | +|---|---| +| Was healthy, then heartbeat went stale | **Restart promptly** (this is a hang) | +| Running but never became healthy (slow start, bad install, the **first-run setup screen**) | **Be patient**, then restart after `2 × StartupGraceSec` | + +Never kill a player that has not yet been healthy — it may be an operator entering +server settings on the first-run setup screen. + +### Important Windows-specific facts + +- The packaged player is **TWO processes** (PyInstaller bootloader parent + the + child that owns the SDL window). Any kill must use `taskkill /T` or the visible + window survives and the next launch collides with it. +- The watchdog can only *restart* an install that already has a working config. + It cannot help before the player has first run. +- Hiding the console window (`console=False`) is **not** done: the console is the + only way to see a start-up failure, and the build instructions warn against + flipping it without reason. +- **No Windows service.** A service runs in session 0 with no desktop, so the + player could not render to the screen. A login-triggered task is the correct + Windows counterpart to the Pi's systemd unit — that is what + `start_player_watchdog.bat` is for (add it to the Startup folder for autostart). + +### Verify the watchdog + +``` +cd windows +venv\Scripts\python.exe test_watchdog.py # crash + hang + stop-flag + new-session +``` + +Check state at any time (read-only): + +``` +powershell -NoProfile -ExecutionPolicy Bypass -File windows\watchdog.ps1 -Status +``` + +> When testing a hang, do **not** just backdate the heartbeat file's mtime — the +> healthy player rewrites it immediately, so nothing is actually stale and the +> test passes for the wrong reason. Hold the file open with an exclusive Windows +> lock (share mode 0) so the player's own write fails; that reproduces a genuine +> stale heartbeat while the process stays alive. + ## Code Signing (production constraint) Production PCs run with **Smart App Control enforced** (`VerifiedAndReputablePolicyState=1`, @@ -96,9 +217,21 @@ UMCI enforced) with **no "Run anyway" bypass** — an unsigned exe is blocked at Fast syntax gate (no build, seconds): ``` -python -m py_compile src/main.py src/weblink_session.py windows/run_win.py +python -m py_compile src/main.py src/weblink_session.py windows/run_win.py windows/webview2_browser.py windows/webview2_runtime.py ``` +Web-link engine checks (no build, no player): + +``` +cd windows +venv\Scripts\python.exe test_webview2_embed.py # embeds WebView2 in a bare Win32 window +venv\Scripts\python.exe test_webview2_runtime.py # Runtime detection + installer discovery +``` + +Rebuild warning: the bundled offline installer makes the exe ~305 MB. If you do not +need offline machines, delete +`windows/webview2_runtime/MicrosoftEdgeWebView2RuntimeInstallerX64.exe` before building. + **Close the running player before rebuilding.** A running `dist\KiwySignagePlayer\KiwySignagePlayer.exe` locks the output and the build fails with "Access is denied". Chrome/Edge kiosk processes left over from web-link playback can also diff --git a/src/video_safety.py b/src/video_safety.py new file mode 100644 index 0000000..12d3302 --- /dev/null +++ b/src/video_safety.py @@ -0,0 +1,185 @@ +"""video_safety.py — make Kivy's video teardown non-blocking. + +The bug this fixes +------------------ +The player hung (Windows `AppHangB1`) after ~30-45 minutes of looping, always +at a *video* item. The trace stopped dead right after `video_loaded` with no +`video_eos` and no `advance_after_video_eos`. + +Cause — a blocking ``join()`` that Kivy performs on the calling thread: + +1. At end of stream, ffpyplayer/``VideoFFPy`` fires its ``on_eos`` event. +2. Kivy's ``Video`` widget binds its own handler **first** (in + ``kivy/uix/video.py``: ``self._video.bind(..., on_eos=self._on_eos)``). + That handler is:: + + def _on_eos(self, *largs): + if not self._video or self._video.eos != 'loop': + self.state = 'stop' # <-- fires DURING the on_eos dispatch + +3. ``Video.state = 'stop'`` → ``on_state`` → ``VideoFFPy.stop()`` → + ``unload()``, which does ``self._thread.join()`` + (``video_ffpyplayer.py``: ``# TODO: use callback, don't block here``). +4. That join waits for the ffpyplayer decode thread, whose lifetime is variable + (the existing code comments note "0.4s up to 100s+"). When it does not + return, the SDL/Kivy main thread never returns either — the window stops + pumping messages and Windows declares the app hung. + +Why it only shows up after a while: our own EOS handler deliberately does not +set ``state = 'stop'`` (that path is already handled off-thread), so the hang +depends on Kivy's own handler firing first and on that particular video's +decode thread being slow to exit. It is a race, so it looks random and only +appears after many video cycles. + +The fix +------- +Bound the wait. ffpyplayer sets ``_ffplayer_need_quit = True`` and wakes its +thread before joining, so the thread *is* asked to exit — we simply stop +waiting indefinitely for it. ``suppress_kivy_video_blocking_unload()`` patches +the join on the specific ``VideoFFPy`` thread object to a bounded timeout, so a +wedged decode thread can no longer take the whole player down. The teardown +still runs off the main thread (see ``_teardown_video_async`` in ``main.py``), +so the normal case is unaffected. + +This is deliberately narrow: only Kivy's own internal video thread is patched, +only its ``join`` timeout is bounded, and every failure path leaves Kivy +untouched. +""" + +from __future__ import annotations + +import threading +import time + +#: How long a video's decode thread may take to exit before we give up on it. +#: A healthy ffpyplayer thread exits in well under a second; this allows a lot +#: of slack while still guaranteeing the UI thread is never parked forever. +DEFAULT_JOIN_TIMEOUT = 5.0 + +_patched_threads = 0 +_lock = threading.Lock() + + +def _log(message): + try: + from kivy.logger import Logger + + Logger.info(f'[VideoSafety] {message}') + except Exception: + print(f'[VideoSafety] {message}') + + +def suppress_kivy_video_blocking_unload(timeout=DEFAULT_JOIN_TIMEOUT): + """Bound the join() Kivy's VideoFFPy.unload() performs on the caller. + + Must be called *before* the video widget is constructed, because the + decode thread is created during ``play()``. Safe to call repeatedly. + + Args: + timeout: maximum seconds to wait for the decode thread to exit. + + Returns: + True when the guard was installed (or already present). + """ + global _patched_threads + + if timeout is None or float(timeout) <= 0: + return False + + try: + # Importing this module has the side effect of selecting the + # ffpyplayer provider; if a different provider is active (or video + # support is missing) there is nothing to patch. + from kivy.core.video import Video as CoreVideo + + if CoreVideo is None: + return False + except Exception as exc: + _log(f'video provider unavailable, nothing to patch ({exc})') + return False + + try: + from kivy.core.video import Video + except Exception: + Video = None + + if Video is None: + return False + + # VideoFFPy resolves via the current provider. Import it directly so we + # patch the right class even if the provider changes later. + try: + from kivy.core.video import video_ffpyplayer as _vfp + except Exception as exc: + _log(f'ffpyplayer provider not active, nothing to patch ({exc})') + return False + + provider = getattr(_vfp, 'VideoFFPy', None) + if provider is None: + return False + + with _lock: + if getattr(provider, '_kiwy_bounded_join', False): + return True + + original_play = provider.play + if getattr(provider, '_kiwy_original_play', None) is None: + provider._kiwy_original_play = original_play + + def play(self, *args, **kwargs): + result = provider._kiwy_original_play(self, *args, **kwargs) + _bound_thread_join(self, timeout) + return result + + provider.play = play + provider._kiwy_bounded_join = True + _patched_threads += 1 + + _log(f'Kivy video unload join() bounded to {float(timeout):.1f}s') + return True + + +def _bound_thread_join(provider_instance, timeout): + """Replace the provider's internal thread join with a bounded one. + + ffpyplayer has already been told to quit (``_ffplayer_need_quit = True``) + and its thread woken before ``unload()`` joins, so bounding the wait does + not leak work — it only stops an unresponsive decode thread from freezing + the whole application. + """ + thread = getattr(provider_instance, '_thread', None) + if thread is None: + return + if getattr(thread, '_kiwy_bounded_join', False): + return + + try: + real_join = thread.join + except Exception: + return + + def bounded_join(join_timeout=None): + """Join with a hard upper bound; never block the caller forever.""" + limit = float(timeout if join_timeout is None else min(join_timeout, timeout)) + started = time.monotonic() + try: + real_join(timeout=limit) + except TypeError: + # Some Python builds dislike an explicit keyword here. + try: + real_join(limit) + except Exception: + pass + except Exception: + pass + if thread.is_alive(): + _log( + f'video decode thread still alive {time.monotonic() - started:.1f}s ' + 'after being asked to quit - continuing without it' + ) + + try: + thread.join = bounded_join + thread._kiwy_bounded_join = True + except Exception as exc: + _log(f'could not bound video thread join ({exc})') diff --git a/windows/README_WINDOWS_BUILD.md b/windows/README_WINDOWS_BUILD.md index e017895..72aa5d6 100644 --- a/windows/README_WINDOWS_BUILD.md +++ b/windows/README_WINDOWS_BUILD.md @@ -90,12 +90,94 @@ windows\dist\KiwySignagePlayer\ For a **single-file .exe**, edit `build.spec` — uncomment the `exe_onefile` section and comment out the `coll = COLLECT(...)` section. +## 🌐 Web Links (embedded WebView2) + +Web links render with **WebView2**, embedded as a child window *inside* the +Kivy window. Because it is not a separate browser process, it cannot open +behind the player, cannot be handed off to an existing browser and exit, and +never leaves leaked `msedge.exe`/`chrome.exe` processes behind. + +WebView2 has **two** parts, and they are handled differently: + +| Part | What it is | How it ships | +|------|------------|--------------| +| **SDK** | `Microsoft.Web.WebView2.Core.dll` + `WebView2Loader.dll` — the API surface | Bundled in the exe (`windows\webview2_sdk\`, ~860 KB) | +| **Runtime** | `msedgewebview2.exe` — the actual Chromium engine | Microsoft's evergreen component. Ships with Windows 11 and nearly all Windows 10 machines. **Installed automatically on first start if missing.** | + +### Automatic Runtime installation + +On start-up the player checks for the Runtime (registry `pv` value under the +Edge Update client GUID, with a live SDK probe as fallback). If it is absent it +runs the installer silently: + +``` +MicrosoftEdgeWebview2Setup.exe /silent /install +``` + +Deliberately **not elevated** — an unelevated run performs a *per-user* install, +so no UAC dialog ever appears on the signage display. + +Installers are looked up in this order, so you can drop a replacement next to +the `.exe` without rebuilding: + +1. `KIWY_WEBVIEW2_INSTALLER` environment variable +2. `\webview2_runtime\` +3. `\` (next to the executable) +4. bundled copy inside the exe + +| Installer | Size | Bundled? | Use when | +|-----------|------|----------|----------| +| `MicrosoftEdgeWebview2Setup.exe` | ~1.7 MB | ✅ yes | Machine has internet (downloads the Runtime) | +| `MicrosoftEdgeWebView2RuntimeInstallerX64.exe` | ~203 MB | ⬜ opt-in | Machine is **offline** | + +To bundle the offline installer (adds ~200 MB to the exe): + +```powershell +cd windows +.\webview2_runtime\download_runtime_installers.ps1 -Offline +venv\Scripts\python.exe -m PyInstaller build.spec --clean --noconfirm +``` + +Re-download the installers at any time (they are Microsoft-signed; the script +verifies the signature): + +```powershell +.\webview2_runtime\download_runtime_installers.ps1 +``` + +### If WebView2 is unavailable + +Web links fall back to the Chrome/Edge subprocess engine (see +`src/weblink_session.py`). That engine still works, but reintroduces the old +drawbacks — a separate browser window, possible background/z-order behaviour, +and browser processes to clean up. + +### Troubleshooting + +| Problem | Solution | +|---------|----------| +| Web links show a black/blank page | Check `logs\` for `[WebView2]` lines. Confirm the Runtime version is reported. | +| Runtime install did not happen | A failed attempt is not retried for 6 hours (marker: `logs\.webview2_install_attempt`). Delete that file to retry immediately. | +| Need it working offline | Bundle the standalone installer (see above). | + ## ⚙️ Configuration -1. On first run, config files are created in the **same folder as the executable** (not in `%APPDATA%`) - - The .exe creates: `config/`, `media/`, `playlists/`, `logs/` directories locally +**No configuration ships with the exe.** On a machine that has never been set +up, `config\app_config.json` is absent, so after the splash video the player +shows a *"Player is not configured"* notice, waits 5 seconds and then opens the +**Settings** screen automatically. Enter the server details and playback starts +right away — no restart required. + +On a machine that already has a valid `config\app_config.json`, the first-run +flow is skipped entirely and the cached playlist plays immediately. + +1. Config is created next to the **executable** (not in `%APPDATA%`) + - The .exe creates: `config/`, `media/`, `playlists/`, `logs/` locally - This allows you to copy the entire `dist\KiwySignagePlayer\` folder anywhere and it works -2. Edit `config\app_config.json` (next to the .exe) to set your server: +2. The player is considered configured once `server_ip`, `screen_name` and + `quickconnect_key` all hold real values. Placeholder values + (`localhost`, `kivy-player`, `1234567`, `127.0.0.1`) count as unconfigured. +3. Edit `config\app_config.json` (next to the .exe) to set your server: ```json { diff --git a/windows/run_win.py b/windows/run_win.py index b4c671b..77af6ed 100644 --- a/windows/run_win.py +++ b/windows/run_win.py @@ -251,6 +251,93 @@ def _get_cef_browser(): return _CEF_BROWSER +# ── Embedded WebView2 (Edge) browser ──────────────────────────────── +_WEBVIEW2_BROWSER = None + +#: How long a weblink will wait for a Runtime that is still installing. +_WEBVIEW2_INSTALL_WAIT = 300.0 + + +def _ensure_webview2_runtime_async(): + """Start installing the WebView2 Runtime if it is missing. + + Called once at start-up. The install runs on a background thread so a + machine without the Runtime can install it while the player is still + syncing its playlist, rather than freezing the UI on the first weblink. + """ + try: + import webview2_runtime + except Exception as exc: + _log(f"WebView2 runtime module unavailable: {exc}") + return + try: + _log(webview2_runtime.describe()) + webview2_runtime.ensure_runtime_async() + except Exception as exc: + _log(f"WebView2 runtime check failed (non-fatal): {exc}") + + +def _get_webview2_browser(force_new=False): + """Return the shared WebView2Browser singleton, or None if unavailable. + + WebView2 renders as a CHILD HWND of Kivy's SDL window, which is what makes + it immune to the subprocess weblink bugs (background window, instant + hand-off exit, z-order fights, leaked browsers). + """ + global _WEBVIEW2_BROWSER + if force_new: + old, _WEBVIEW2_BROWSER = _WEBVIEW2_BROWSER, None + if old is not None: + try: + old.shutdown() + except Exception: + pass + if _WEBVIEW2_BROWSER is None: + try: + from webview2_browser import WebView2Browser + except Exception as exc: + _log(f"WebView2 module unavailable: {exc}") + return None + + if not WebView2Browser.is_available(): + # The SDK is bundled but the Runtime may still be installing + # (a machine that shipped without it). Wait here rather than + # silently downgrading to the leaky Chrome/Edge engine — this runs + # on the watcher thread, not the Kivy main thread. + if _wait_for_webview2_runtime(): + pass # installed in the meantime; retry below + if not WebView2Browser.is_available(): + reason = getattr(WebView2Browser, '_import_error', None) or 'unavailable' + _log(f"WebView2 not usable: {reason}") + return None + + try: + data_dir = os.environ.get('KIWY_DATA_DIR', os.getcwd()) + _WEBVIEW2_BROWSER = WebView2Browser( + hwnd_provider=_find_kivy_hwnd, + user_data_dir=os.path.join(data_dir, '.webview2-profile'), + ) + except Exception as exc: + _log(f"WebView2 init failed: {exc}") + return None + return _WEBVIEW2_BROWSER + + +def _wait_for_webview2_runtime(timeout=_WEBVIEW2_INSTALL_WAIT): + """Block while a Runtime install is in flight. True if it became available.""" + try: + import webview2_runtime + except Exception: + return False + state = webview2_runtime.get_state() + if not state.get('installing'): + return False + _log("WebView2: waiting for the Runtime install to finish...") + ok = webview2_runtime.wait_for_install(timeout) + _log(f"WebView2: Runtime install wait finished (available={ok})") + return ok + + def _windows_find_browser(): """Find Chrome or Edge executable on Windows for weblink support. @@ -1472,6 +1559,167 @@ def _patch_main(): # CEF keeps one browser instance alive; there is nothing to warm. pass + class _WinWebView2Adapter(ChromiumSubprocessAdapter): + """Embedded WebView2 (Edge) — renders INSIDE the Kivy window. + + This is the preferred Windows engine. Because the page is a child HWND + of Kivy's own SDL window there is no separate browser process, so none + of the subprocess problems apply: it cannot open behind the player, it + cannot be handed off to an existing instance and exit instantly, it + does not fight for foreground/z-order, and teardown leaves no leaked + browser behind. + + ``launch`` returns as soon as the (asynchronous) start-up is requested; + the session's watcher thread then polls :meth:`wait_visible` until the + page is actually showing. + """ + + name = 'webview2-embedded' + embedded = True + # WebView2 renders in-window, but unlike raw CEF we CAN prove the page + # appeared (the controller reports when it is visible). That matters on + # a closed network: if the weblink host is unreachable the page never + # paints and the item is skipped instead of showing a blank screen. + can_verify_visibility = True + + def __init__(self): + super().__init__(kiosk=False) + self._browser = None + self._resize_bound = False + + @property + def process(self): + return None # in-process: nothing to kill + + def target_size(self): + """Use the real Kivy window size (already DPI-aware).""" + try: + from kivy.core.window import Window as KivyWindow + + width, height = (int(KivyWindow.size[0]), int(KivyWindow.size[1])) + if width > 0 and height > 0: + return width, height + except Exception: + pass + return 1920, 1080 + + def launch(self, url, width, height): + self._browser = _get_webview2_browser() + if self._browser is None: + return False + # A child HWND has no z-order to fight over, but the Kivy window + # itself must own the foreground or the page looks inert. + _bring_kivy_to_front(async_ok=False) + self._bind_resize_once() + self._browser.resize(width, height) + return bool(self._browser.show(url)) + + def is_alive(self): + browser = self._browser + if browser is None: + return False + # NOTE: `is_alive` must be True while the page is still coming up. + # WebView2 creates its environment and controller asynchronously, so + # a controller that does not exist yet is NOT a dead browser — + # treating it as one made the first weblink be skipped instantly. + return browser.is_alive() + + def wait_visible(self, timeout): + """Wait for the page to actually load, on the caller's (watcher) thread. + + Two things must hold: the controller must be showing, AND the + navigation must have completed successfully. The second condition is + what makes an unreachable host (normal on a closed network) skip the + item instead of displaying Chromium's error page for the full slot. + """ + import time + + browser = self._browser + if browser is None: + return False, 'no browser' + deadline = time.monotonic() + max(1.0, float(timeout)) + while time.monotonic() < deadline: + if browser.failed_reason: + return False, browser.failed_reason + nav = browser.navigation_succeeded() + if browser.is_showing() and nav is not None: + if not nav: + detail = browser.navigation_status() or 'navigation failed' + return False, f'page failed to load ({detail})' + # Let the compositor paint the first frame before any + # masking overlay is hidden / Kivy is restored. + time.sleep(0.3) + return True, 'webview2-loaded' + time.sleep(0.1) + if browser.is_showing() and browser.navigation_succeeded() is None: + # Controller is up but navigation never reported within the + # timeout. Treat as visible rather than skipping a page that is + # merely slow to confirm. + return True, 'webview2-showing-unconfirmed' + reason = browser.failed_reason or 'webview2 did not show' + return False, reason + + def on_visible(self): + trace('win_weblink_webview2_shown') + + def on_launch_failed(self): + # Never leave a half-built controller (or a hidden child window) + # behind when start-up fails. + browser, self._browser = self._browser, None + if browser is not None: + try: + browser.shutdown() + except Exception: + pass + _get_webview2_browser(force_new=True) + + def _bind_resize_once(self): + """Keep the page matched to the window, bound exactly once. + + Rebinding per cycle was the unbounded-callback leak fixed for CEF; + the same discipline applies here. + """ + if self._resize_bound: + return + try: + from kivy.core.window import Window as KivyWindow + + def _wv2_resize(*_args): + try: + browser = self._browser + if browser is not None and browser.is_showing(): + browser.resize( + int(KivyWindow.size[0]), int(KivyWindow.size[1]) + ) + except Exception: + pass + + KivyWindow.bind(size=_wv2_resize) + self._resize_bound = True + except Exception as exc: + _log(f"WebView2 resize bind failed (non-fatal): {exc}") + + def teardown(self): + """Hide the page (keep the controller for a fast next show). + + Hiding — not disposing — is deliberate: the controller stays alive + so the next weblink paints immediately, and because it is a child + window there is no leaked process to reap. + """ + browser = self._browser + if browser is not None: + try: + browser.hide() + except Exception as exc: + _log(f"WebView2 hide failed (non-fatal): {exc}") + # Restore Kivy as the visible surface again. + _bring_kivy_to_front(async_ok=False) + + def prewarm(self, url): + # The environment/controller are already warm after the first use; + # pre-navigating would claim the page before its slot. + pass + class _WinChromeAdapter(ChromiumSubprocessAdapter): """Chrome/Edge kiosk subprocess with the Windows visibility check. @@ -1493,6 +1741,26 @@ def _patch_main(): def on_launch_failed(self): _Win32Overlay.hide() + def extra_launch_args(self): + """Dedicated profile + kiosk flags for the Windows browser. + + ``--user-data-dir`` is mandatory: without it Chrome/Edge hands the + URL to an already-running instance, our launched process exits in + ~2s and the weblink never displays (the adapter's `wait_visible` + then sees the process die and the item is skipped). It also gives + us a top-level window we can enumerate, raise and taskkill without + touching the operator's own browser profile. + """ + args = [] + if self._profile_dir: + args.append('--user-data-dir=' + self._profile_dir) + if self._kiosk: + # The base launch() already adds --start-fullscreen / + # --start-maximized; --kiosk upgrades that to a true kiosk + # window (no browser UI, locks to the screen). + args.append('--kiosk') + return args + def launch(self, url, width, height): # A dedicated --user-data-dir is mandatory: without it Chrome hands # the URL to an existing process, the launched process exits @@ -1556,15 +1824,28 @@ def _patch_main(): def _windows_weblink_adapter_factory(player): """Choose the Windows web-link engines, best first. - CEF (embedded) is preferred when available because it cannot fight for - z-order or foreground; the Chrome/Edge subprocess is the fallback. + Order matters: + + 1. **WebView2 (embedded)** — a child HWND of the Kivy window. It cannot + open behind the player, cannot be handed off to an existing browser + and exit instantly, does not fight for z-order/foreground, and + leaves no leaked process behind. This is the preferred engine. + 2. **CEF (embedded)** — same in-window model, but cefpython3 has no + wheels past Python 3.9 so it is dormant on this build. + 3. **Chrome/Edge subprocess** — last resort only. Kept so a machine + without the WebView2 runtime still shows weblinks. """ adapters = [] + if _get_webview2_browser() is not None: + adapters.append(_WinWebView2Adapter()) if _get_cef_browser() is not None: adapters.append(_WinCefAdapter()) browser = _windows_find_browser() if browser: adapters.append(_WinChromeAdapter(browser)) + if not adapters: + _log("WebView2/CEF unavailable and no Chrome/Edge found — " + "web links will be skipped") return adapters signage_main.SignagePlayer.weblink_adapter_factory = staticmethod( @@ -1657,6 +1938,85 @@ def _patch_main(): signage_main.SettingsPopup.test_connection = _windows_test_connection + # ── Keep player auth OUT of the bundle ─────────────────────────── + # `player_auth.json` used to be a tracked file inside src/, so PyInstaller + # bundled it and, because the frozen app runs with cwd = _internal/, the + # player loaded it as its live auth state. A stale snapshot therefore made + # a fresh build boot "already authenticated" against an old server and play + # an outdated playlist. + # + # Fix: never read a relative auth path from the bundle. Any relative + # 'player_auth.json' is redirected to the data dir next to the .exe, which + # is the single source of truth for this install. + import player_auth as _player_auth_module + + _bundled_auth = os.path.join( + getattr(sys, '_MEIPASS', os.path.dirname(os.path.abspath(__file__))), + 'player_auth.json', + ) + _local_auth = os.path.join(DATA_DIR, 'player_auth.json') + + # If an older build left the bundled snapshot next to the exe, drop it: + # it names another server and would be trusted on the next start-up. + try: + if os.path.isfile(_local_auth): + import json as _json + + with open(_local_auth, 'r') as _f: + _existing = _json.load(_f) + _existing_url = str(_existing.get('server_url') or '') + _wanted_ip = str(os.environ.get('KIWY_SERVER_IP') or '') + if _wanted_ip and _wanted_ip not in _existing_url: + Logger.warning( + "SignagePlayer: stored auth points at %s but the configured " + "server is %s - clearing it so the player re-authenticates", + _existing_url or '(none)', _wanted_ip, + ) + os.remove(_local_auth) + except Exception as _exc: + Logger.debug(f"SignagePlayer: auth pre-check skipped: {_exc}") + + _original_auth_init = _player_auth_module.PlayerAuth.__init__ + + def _windows_auth_init(self, config_file='player_auth.json', + use_https=True, verify_ssl=True): + """Force the auth file to live in the data dir next to the .exe.""" + try: + if not os.path.isabs(config_file): + config_file = os.path.join(DATA_DIR, os.path.basename(config_file)) + except Exception: + config_file = _local_auth + _original_auth_init(self, config_file, use_https=use_https, verify_ssl=verify_ssl) + + _player_auth_module.PlayerAuth.__init__ = _windows_auth_init + Logger.info(f"SignagePlayer: player auth file -> {_local_auth}") + + # `get_playlists_v2` caches a global auth instance created with the default + # relative path; make sure its cache is empty so the redirect above applies. + try: + import get_playlists_v2 as _gp + + if _gp._auth_instance is not None: + _gp._auth_instance = None + except Exception as _exc: + Logger.debug(f"SignagePlayer: could not reset auth cache: {_exc}") + + # `reset_player_auth` deleted the file next to main.py (i.e. inside the + # bundle, which is read-only and not what we load). Point it at the real + # auth file so the "Reset auth" button actually works. + def _windows_reset_player_auth(self): + try: + if os.path.exists(_local_auth): + os.remove(_local_auth) + Logger.info(f"SettingsPopup: Deleted authentication file: {_local_auth}") + self._show_temp_message( + '✓ Authentication reset - will reauthenticate on restart', (0, 1, 0, 1) + ) + except Exception as exc: + Logger.error(f"SettingsPopup: Failed to reset auth: {exc}") + + signage_main.SettingsPopup.reset_player_auth = _windows_reset_player_auth + # ── Patch apply_kiosk_mode for Windows ────────────────────────── # Wrap the base implementation (exit_on_escape + close guard + Ctrl+C) # and add the low-level keyboard hook that swallows Alt+F4 / Alt+Tab / @@ -1702,6 +2062,13 @@ def _patch_main(): Logger.info("SignagePlayer: Windows card reader shut down") except Exception as e: Logger.debug(f"SignagePlayer: card reader shutdown error: {e}") + # Tear down the embedded web engine. Because it is a child HWND (not a + # subprocess) this is what stops it surviving the player on exit. + try: + _get_webview2_browser(force_new=True) + Logger.info("SignagePlayer: WebView2 browser shut down") + except Exception as e: + Logger.debug(f"SignagePlayer: WebView2 shutdown error: {e}") _original_on_stop(self) signage_main.SignagePlayerApp.on_stop = _windows_on_stop @@ -1829,6 +2196,14 @@ if __name__ == '__main__': Logger.info(f"Data directory: {DATA_DIR}") Logger.info("=" * 80) + # ── Ensure the WebView2 Runtime is present ─────────────────── + # The bundled SDK is only an API surface; the Runtime holds the actual + # engine. Windows 11 and most Windows 10 machines already have it, but + # when it is missing we install it silently (per-user, so no UAC prompt + # appears on the signage display). Runs in the background so start-up + # is not blocked. + _ensure_webview2_runtime_async() + # Patch base_dir in SignagePlayer instances to point to local data folder _original_init = patched_main.SignagePlayer.__init__ diff --git a/windows/start_player_watchdog.bat b/windows/start_player_watchdog.bat new file mode 100644 index 0000000..63b7f64 --- /dev/null +++ b/windows/start_player_watchdog.bat @@ -0,0 +1,48 @@ +@echo off +REM ============================================================ +REM Kiwy Signage Player - Watchdog Launcher (Windows) +REM ============================================================ +REM Starts watchdog.ps1, which keeps the player running 24/7: +REM * restarts it if it crashes +REM * restarts it if it hangs (stale heartbeat) +REM * backs off if it cannot stay up (crash-loop breaker) +REM +REM The WATCHDOG runs in this window. Closing it stops supervision +REM (it does NOT stop the player). +REM +REM The ONLY supported way to stop the player is the exit screen +REM password. That writes .player_stop_requested next to the .exe, +REM which tells the watchdog not to restart it. Launching again +REM starts a fresh session and clears that flag. +REM +REM Usage: +REM start_player_watchdog.bat start supervising +REM start_player_watchdog.bat -Status report state, then exit +REM ============================================================ + +setlocal +cd /d "%~dp0" + +echo ============================================ +echo Kiwy Signage Player - Watchdog +echo ============================================ +echo. +echo Stop the WATCHDOG with Ctrl+C. +echo Stop the PLAYER via the exit password. +echo. + +powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0watchdog.ps1" %* + +set "RC=%ERRORLEVEL%" + +if not "%RC%"=="0" ( + echo. + echo [ERROR] Watchdog exited with code %RC%. + echo Check logs\watchdog.log next to the player executable. + echo. +) + +REM Keep the window open so a crash is readable when double-clicked. +if "%~1"=="" pause +endlocal +exit /b %RC% diff --git a/windows/test_video_hang.py b/windows/test_video_hang.py new file mode 100644 index 0000000..b580cf3 --- /dev/null +++ b/windows/test_video_hang.py @@ -0,0 +1,139 @@ +"""Proves src/video_safety.py stops the hang at the end of a video. + +Reproduces the exact failure: + Kivy's VideoFFPy.unload() does `self._thread.join()` with NO timeout. If the + ffpyplayer decode thread does not exit, the calling thread (the Kivy main + thread, via Kivy's own on_eos handler setting state='stop') blocks forever and + Windows reports the app as hung (AppHangB1). + +Two checks: + 1. The guard installs on the REAL Kivy provider (VideoFFPy.play wrapped). + 2. A deliberately wedged decode thread makes unload() return promptly + instead of blocking forever. + +Run: windows\\venv\\Scripts\\python.exe windows\\test_video_hang.py +Exit code 0 = PASS (the hang is prevented). +""" + +import sys +import threading +import time +from pathlib import Path + +SRC = Path(__file__).resolve().parent.parent / 'src' +sys.path.insert(0, str(SRC)) + +import video_safety # noqa: E402 + +# A short-lived "decode thread" that ignores the quit request, standing in for +# an ffpyplayer thread stuck inside a codec/close call. +WEDGE_SECONDS = 30 + + +class _WedgedProvider: + """Minimal stand-in for VideoFFPy, with the same blocking unload().""" + + def __init__(self): + self._thread = threading.Thread(target=self._decode_loop, daemon=True) + self._thread.start() + + def _decode_loop(self): + # Ignores any "please quit" flag and holds the thread — this is what a + # slow ffpyplayer teardown looks like to unload(). + time.sleep(WEDGE_SECONDS) + + def play(self, *args, **kwargs): + return True + + def unload(self): + # Verbatim shape of kivy/core/video/video_ffpyplayer.py unload(): + # if self._thread: + # self._thread.join() # <-- no timeout: hangs forever + if self._thread: + self._thread.join() + self._thread = None + + +def main(): + print('=' * 68) + print(' Kivy video-teardown hang test') + print('=' * 68) + + ok = True + + # ── 1. Does the guard install on the real provider? ────────────── + print('\n[1] guard installation') + try: + from kivy.core.video import video_ffpyplayer as vfp + + provider = vfp.VideoFFPy + print(f' provider: {provider.__module__}.{provider.__name__}') + except Exception as exc: + print(f' SKIP: ffpyplayer provider not available ({exc})') + print(' (the packaged app uses it, so this must pass there)') + return 0 + + installed = video_safety.suppress_kivy_video_blocking_unload(timeout=2.0) + print(f' suppress_kivy_video_blocking_unload() -> {installed}') + wrapped = getattr(provider, '_kiwy_bounded_join', False) + print(f' provider.play wrapped -> {wrapped}') + if not (installed and wrapped): + print(' FAIL: guard not installed') + ok = False + + # The existing Video._do_video_load path must be untouched (no source change). + try: + from kivy.uix.video import Video + + print(f' kivy.uix.video.Video unload: ' + f'{"unload" in dir(Video)}') + except Exception as exc: + print(f' note: could not import Video ({exc})') + + # ── 2. Does a wedged thread still block? ───────────────────────── + print(f'\n[2] wedged decode thread (holds {WEDGE_SECONDS}s)') + prov = _WedgedProvider() + # Install the same bound join the guard installs on a real provider. + video_safety._bound_thread_join(prov, 2.0) + patched = getattr(prov._thread, '_kiwy_bounded_join', False) + print(f' thread join bounded -> {patched}') + if not patched: + print(' FAIL: thread join was not bounded') + ok = False + + started = time.monotonic() + prov.unload() # would hang forever without the fix + elapsed = time.monotonic() - started + print(f' unload() returned after {elapsed:.2f}s') + if elapsed > 5.0: + print(f' FAIL: unload() blocked for {elapsed:.1f}s (expected < 5s)') + ok = False + else: + print(' OK: unload() no longer blocks the caller indefinitely') + + # ── 3. Control: show the unpatched case really would hang ─────── + print('\n[3] control (unpatched join, 2s probe to prove it blocks)') + prov2 = _WedgedProvider() + blocked = True + t = threading.Thread(target=prov2.unload, daemon=True) + t.start() + t.join(timeout=2.0) + blocked = t.is_alive() + print(f' unpatched unload() still blocked after 2s -> {blocked}') + if not blocked: + print(' note: control did not block (timing); fix still valid') + else: + print(' OK: confirms the original join() is the hang, and the fix ' + 'is what prevents it') + + print('=' * 68) + print(' RESULT:', 'PASS' if ok else 'FAIL') + if ok: + print(' A slow/wedged ffpyplayer teardown can no longer freeze the') + print(' player at the end of a video item.') + print('=' * 68) + return 0 if ok else 1 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/windows/test_watchdog.py b/windows/test_watchdog.py new file mode 100644 index 0000000..2534e01 --- /dev/null +++ b/windows/test_watchdog.py @@ -0,0 +1,438 @@ +"""Verifies windows/watchdog.ps1 actually recovers the player. + +Three scenarios, each with a real player process: + + 1. CRASH - kill the player (simulating a crash) and confirm the watchdog + brings it back and it becomes healthy again. + 2. HANG - make the heartbeat go stale while the process stays alive, and + confirm the watchdog kills it and restarts it. + 3. STOP FLAG - write the stop-flag file (what the password exit does) and + confirm the watchdog stands down instead of restarting. + Then confirm a fresh watchdog run CLEARS the flag (new session). + +The watchdog is started as a child process with a short check interval so the +test is quick. It is stopped at the end; the stop flag it may have created is +removed so the machine is left as it was found. + +Run: windows\\venv\\Scripts\\python.exe windows\\test_watchdog.py +Exit code 0 = PASS. +""" + +import subprocess +import sys +import time +from pathlib import Path + +WIN = Path(__file__).resolve().parent +PLAYER_DIR = WIN / 'dist' / 'KiwySignagePlayer' +EXE = PLAYER_DIR / 'KiwySignagePlayer.exe' +HEARTBEAT = PLAYER_DIR / '.player_heartbeat' +STOP_FLAG = PLAYER_DIR / '.player_stop_requested' +WATCHDOG = WIN / 'watchdog.ps1' +LOG = PLAYER_DIR / 'logs' / 'watchdog.log' + +PS = ['powershell', '-NoProfile', '-ExecutionPolicy', 'Bypass', '-File'] + +# Fast timings so the test finishes in a reasonable time. +FAST = ['-HealthCheckIntervalSec', '3', '-HeartbeatStaleSec', '10', + '-StartupGraceSec', '45', '-RestartDelaySec', '2', + '-CrashLoopMaxFailures', '50', '-CrashLoopBackoffMin', '1'] + + +def kill_players(): + subprocess.run(['taskkill', '/F', '/T', '/IM', 'KiwySignagePlayer.exe'], + capture_output=True, text=True) + + +def player_count(): + out = subprocess.run( + ['powershell', '-NoProfile', '-Command', + "(Get-Process -Name KiwySignagePlayer -ErrorAction SilentlyContinue " + "| Measure-Object).Count"], + capture_output=True, text=True) + try: + return int((out.stdout or '0').strip() or 0) + except ValueError: + return 0 + + +def hb_age(): + if not HEARTBEAT.is_file(): + return -1 + return time.time() - HEARTBEAT.stat().st_mtime + + +def get_root_pid(): + """Return the pid of the player process that owns the window. + + The packaged player is TWO processes (PyInstaller bootloader + child). + """ + out = subprocess.run( + ['powershell', '-NoProfile', '-Command', + "Get-Process -Name KiwySignagePlayer -ErrorAction SilentlyContinue | " + "Where-Object { $_.MainWindowHandle -ne 0 } | " + "Select-Object -First 1 -ExpandProperty Id"], + capture_output=True, text=True) + try: + return int((out.stdout or '').strip()) + except ValueError: + # Fall back to any player process. + out = subprocess.run( + ['powershell', '-NoProfile', '-Command', + "Get-Process -Name KiwySignagePlayer -ErrorAction SilentlyContinue | " + "Select-Object -First 1 -ExpandProperty Id"], + capture_output=True, text=True) + try: + return int((out.stdout or '').strip()) + except ValueError: + return None + + +# ── Exclusive file lock (real Windows share-mode lock) ─────────────── +# msvcrt.locking only locks a byte RANGE within the file, and Python's +# open() already shares the file for writing, so the player can still replace +# its contents. Opening with share mode 0 denies ALL other access instead, +# which is what actually stops the heartbeat being rewritten. +import ctypes # noqa: E402 +import ctypes.wintypes as wintypes # noqa: E402 + +GENERIC_READ = 0x80000000 +GENERIC_WRITE = 0x40000000 +OPEN_EXISTING = 3 +FILE_ATTRIBUTE_NORMAL = 0x80 +INVALID_HANDLE_VALUE = ctypes.c_void_p(-1).value + +_kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) +_kernel32.CreateFileW.restype = ctypes.c_void_p +_kernel32.CreateFileW.argtypes = [ + ctypes.c_wchar_p, wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p, + wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p, +] +_kernel32.CloseHandle.argtypes = [ctypes.c_void_p] + + +def lock_file_exclusive(path): + """Open ``path`` denying all sharing. Returns the handle, or None.""" + handle = _kernel32.CreateFileW( + str(path), + GENERIC_READ | GENERIC_WRITE, + 0, # share mode 0: nobody else may touch it + None, + OPEN_EXISTING, + FILE_ATTRIBUTE_NORMAL, + None, + ) + if handle in (None, INVALID_HANDLE_VALUE): + err = ctypes.get_last_error() + print(f' note: CreateFileW failed (winerror={err})') + return None + return handle + + +def unlock_file(handle): + if handle: + try: + _kernel32.CloseHandle(handle) + except Exception: + pass + + +def tail_log(n=14): + if not LOG.is_file(): + return [] + lines = LOG.read_text(encoding='utf-8', errors='replace').splitlines() + return lines[-n:] + + +def main(): + # Tee stdout to a UTF-8 report file: the console in this environment + # mangles encoding between processes, so the file is the reliable record. + report_path = WIN / 'watchdog_test_report.txt' + + class _Tee: + def __init__(self, stream, path): + self._stream = stream + self._path = path + + def write(self, text): + self._stream.write(text) + try: + with open(self._path, 'a', encoding='utf-8') as fh: + fh.write(text) + except Exception: + pass + + def flush(self): + try: + self._stream.flush() + except Exception: + pass + + try: + report_path.unlink() + except Exception: + pass + sys.stdout = _Tee(sys.__stdout__, report_path) + + print('=' * 70) + print(' Watchdog recovery test') + print('=' * 70) + + if not EXE.is_file(): + print(f'FAIL: player exe not found at {EXE}') + return 1 + if not WATCHDOG.is_file(): + print(f'FAIL: watchdog not found at {WATCHDOG}') + return 1 + + ok = True + watchdog = None + + # Clean slate: no player, no leftover stop flag. + kill_players() + if STOP_FLAG.exists(): + STOP_FLAG.unlink() + time.sleep(2) + + try: + # --------------------------------------------------------------- + # Start the watchdog; it should launch the player itself. + # --------------------------------------------------------------- + print('\n[setup] starting watchdog (it should launch the player)') + watchdog = subprocess.Popen( + PS + [str(WATCHDOG)] + FAST, + cwd=str(WIN), + creationflags=getattr(subprocess, 'CREATE_NEW_CONSOLE', 0), + ) + + # Wait for the player to come up and report a heartbeat. + deadline = time.time() + 90 + healthy = False + while time.time() < deadline: + time.sleep(3) + if player_count() > 0 and 0 <= hb_age() < 10: + healthy = True + break + print(f' player processes={player_count()} hb_age={hb_age():.0f}s') + if not healthy: + print(' FAIL: player never became healthy under the watchdog') + ok = False + else: + print(' OK: watchdog launched the player and it is healthy') + + # --------------------------------------------------------------- + # 1. CRASH recovery + # --------------------------------------------------------------- + if healthy: + print('\n[1] CRASH: killing the player (simulating a crash)') + kill_players() + time.sleep(2) + print(f' after kill: processes={player_count()}') + + recovered = False + deadline = time.time() + 150 + while time.time() < deadline: + time.sleep(3) + if player_count() > 0 and 0 <= hb_age() < 10: + recovered = True + break + print(f' after recovery: processes={player_count()} hb_age={hb_age():.0f}s') + if recovered: + print(' OK: watchdog restarted the player after the crash') + else: + print(' FAIL: watchdog did not restore a healthy player') + ok = False + + # ----------------------------------------------------------- + # 2. HANG recovery (stale heartbeat, process still alive) + # ----------------------------------------------------------- + # IMPORTANT: backdating the heartbeat's mtime does NOT simulate a + # hang - the healthy player rewrites it on its next tick, so the + # watchdog never sees it stale (that was a false pass in an earlier + # version of this test). + # + # A genuine hang means "process alive, but it stopped updating the + # heartbeat". We reproduce exactly that: hold the heartbeat open + # with an exclusive Windows lock (share mode 0), so the player's + # own write fails. The player catches that write error, logs a + # warning and KEEPS RUNNING - the mtime simply stops advancing. + # That is precisely the condition the watchdog tests for. + print('\n[2] HANG: freezing the heartbeat (player stays alive)') + pid_before = get_root_pid() + if pid_before is None: + print(' FAIL: could not find the player process') + ok = False + else: + handle = lock_file_exclusive(HEARTBEAT) + if handle is None: + print(' FAIL: could not lock the heartbeat file') + ok = False + else: + print(f' holding an exclusive lock on the heartbeat; ' + f'player pid={pid_before} should stay alive') + + # Wait for the heartbeat to go stale while the process lives. + stale_seen = False + deadline = time.time() + 90 + while time.time() < deadline: + time.sleep(2) + if hb_age() > 15: + stale_seen = True + break + still_alive = player_count() > 0 + print(f' heartbeat age={hb_age():.0f}s stale={stale_seen}; ' + f'player still running={still_alive}') + if not (stale_seen and still_alive): + print(' FAIL: could not create a genuine stale-heartbeat hang') + ok = False + + # The watchdog should kill the frozen player... + old_killed = False + deadline = time.time() + 90 + while time.time() < deadline: + time.sleep(2) + if get_root_pid() != pid_before: + old_killed = True + break + print(f' watchdog killed the stale player: {old_killed}') + + # ...release the lock so the replacement can write... + unlock_file(handle) + + # ...and confirm a fresh, healthy player is now running. + restarted = False + deadline = time.time() + 150 + while time.time() < deadline: + time.sleep(3) + if player_count() > 0 and 0 <= hb_age() < 10: + restarted = True + break + print(f' after hang recovery: processes={player_count()} ' + f'hb_age={hb_age():.0f}s') + if old_killed and restarted: + print(' OK: watchdog detected the hang and restarted the player') + else: + print(' FAIL: watchdog did not recover from the hang') + ok = False + + # --------------------------------------------------------------- + # 3. STOP FLAG: the password exit must not be undone + # --------------------------------------------------------------- + print('\n[3] STOP FLAG: simulating the password exit') + STOP_FLAG.write_text('User requested exit via password', encoding='utf-8') + print(' wrote the stop flag') + + # The player is running; give the watchdog time to notice and stand down. + stood_down = False + deadline = time.time() + 90 + while time.time() < deadline: + time.sleep(3) + if watchdog.poll() is not None: + stood_down = True + break + print(f' watchdog exited={stood_down} (rc={watchdog.poll()})') + if stood_down: + print(' OK: watchdog stood down instead of restarting') + else: + print(' FAIL: watchdog did not stand down on the stop flag') + ok = False + + # Confirm it really stops restarting: kill the player and verify it + # stays dead now that the watchdog is gone. + kill_players() + time.sleep(8) + if player_count() == 0: + print(' OK: player stayed stopped (no supervisor resurrecting it)') + else: + print(' FAIL: player was restarted despite the stop flag') + ok = False + + # --------------------------------------------------------------- + # 4. NEW SESSION: a fresh watchdog run clears the flag + # --------------------------------------------------------------- + print('\n[4] NEW SESSION: starting the watchdog again') + if not STOP_FLAG.exists(): + print(' FAIL: stop flag vanished unexpectedly') + ok = False + session = subprocess.Popen( + PS + [str(WATCHDOG), '-Status'], + cwd=str(WIN), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + text=True) + # -Status must NOT clear the flag (it is read-only). + try: + session.wait(timeout=60) + except subprocess.TimeoutExpired: + session.kill() + if STOP_FLAG.exists(): + print(' OK: -Status is read-only (flag untouched)') + else: + print(' FAIL: -Status cleared the flag (should be read-only)') + ok = False + + # A real run clears it, then launches the player. + fresh = subprocess.Popen( + PS + [str(WATCHDOG)] + FAST, + cwd=str(WIN), + creationflags=getattr(subprocess, 'CREATE_NEW_CONSOLE', 0)) + cleared = False + deadline = time.time() + 60 + while time.time() < deadline: + time.sleep(2) + if not STOP_FLAG.exists(): + cleared = True + break + print(f' stop flag cleared by the new session: {cleared}') + if cleared: + print(' OK: a fresh launch starts a new session (flag cleared)') + else: + print(' FAIL: the new session did not clear the stop flag') + ok = False + + # And the player comes back up. + back_up = False + deadline = time.time() + 120 + while time.time() < deadline: + time.sleep(3) + if player_count() > 0 and 0 <= hb_age() < 10: + back_up = True + break + print(f' player back up: {back_up} (processes={player_count()})') + if back_up: + print(' OK: player resumed supervision after the new session') + else: + print(' FAIL: player did not come back in the new session') + ok = False + + fresh.terminate() + + finally: + # --------------------------------------------------------------- + # Leave the machine as we found it. + # --------------------------------------------------------------- + for p in (watchdog,): + if p is not None and p.poll() is None: + p.terminate() + subprocess.run( + ['powershell', '-NoProfile', '-Command', + "Get-CimInstance Win32_Process -Filter \"Name='powershell.exe'\" | " + "Where-Object { $_.CommandLine -like '*watchdog.ps1*' } | " + "ForEach-Object { Stop-Process -Id $_.ProcessId -Force }"], + capture_output=True, text=True) + kill_players() + if STOP_FLAG.exists(): + STOP_FLAG.unlink() + print('\n[cleanup] player stopped, watchdog stopped, stop flag removed') + + print('\n=== last watchdog log lines ===') + for line in tail_log(12): + print(' ' + line) + + print('=' * 70) + print(' RESULT:', 'PASS' if ok else 'FAIL') + print('=' * 70) + return 0 if ok else 1 + + +if __name__ == '__main__': + sys.exit(main()) + diff --git a/windows/watchdog.ps1 b/windows/watchdog.ps1 new file mode 100644 index 0000000..7a76ff7 --- /dev/null +++ b/windows/watchdog.ps1 @@ -0,0 +1,441 @@ +<# +===================================================================== + watchdog.ps1 - keep the Kiwy Signage Player running 24/7 on Windows +===================================================================== + +WHAT THIS DOES + Supervises KiwySignagePlayer.exe and restarts it when it: + + * CRASHES - the process disappeared without the user asking it to. + * HANGS - the process is alive but its heartbeat file has gone stale, + which means the UI thread is wedged (a frozen player looks + exactly like a working one from the outside). + + This is the Windows counterpart of the Linux `start.sh` watchdog, which has + been running the player on the Raspberry Pi deployments. + +HOW THE "PASSWORD ONLY" EXIT IS PRESERVED + The only supported way to stop the player is the exit screen's password. + On success the player writes a stop-flag file next to its .exe: + + .player_stop_requested + + The flag is SESSION SCOPED: + + * While the flag exists, this watchdog will NOT restart the player; it + stops supervising and exits (the machine is left with the player off). + * Every time the watchdog starts, it CLEARS the flag first, which is what + makes a fresh launch a fresh session. So starting the player again is + all it takes to resume - there is no file to delete by hand. + +CRASH-LOOP BREAKER + A player that cannot stay up (bad config, missing media, GPU problem) would + otherwise be restarted forever. If the player dies `CrashLoopMaxFailures` + times inside `CrashLoopWindowMin` minutes WITHOUT ever becoming healthy, + the watchdog backs off for `CrashLoopBackoffMin` minutes before retrying, + and keeps the reason in the log. + +WHAT IT DOES NOT DO + It does not install itself to run at boot/login - start it with + `start_player_watchdog.bat`. It also cannot watch the player before the + player has ever run, so a totally broken install simply logs and backs off. + +USAGE + powershell -NoProfile -ExecutionPolicy Bypass -File watchdog.ps1 + powershell -NoProfile -ExecutionPolicy Bypass -File watchdog.ps1 -Status + + Stop it with Ctrl+C in its window (this does NOT stop the player; it only + stops supervising). +#> + +[CmdletBinding()] +param( + # Path to the player executable. Defaults to the standard build output. + [string]$ExePath, + + # How often the supervisor checks on the player. + [int]$HealthCheckIntervalSec = 15, + + # A heartbeat older than this means the player is wedged, since the player + # rewrites it every 10 seconds. Matches the proven Linux value of 60s. + [int]$HeartbeatStaleSec = 60, + + # After launching, give the player this long to write its first heartbeat + # (startup shows a splash video, so it is not instant) before health + # checks count against it. + [int]$StartupGraceSec = 120, + + # Pause before restarting a player that died. + [int]$RestartDelaySec = 5, + + # Crash-loop breaker (see header). + [int]$CrashLoopWindowMin = 10, + [int]$CrashLoopMaxFailures = 5, + [int]$CrashLoopBackoffMin = 10, + + # Report current state and exit - never launches or kills anything. + [switch]$Status +) + +$ErrorActionPreference = 'Stop' + +# UTF-8 so accented paths in logs are readable. +$OutputEncoding = [System.Text.Encoding]::UTF8 + +# --------------------------------------------------------------------- +# Paths +# --------------------------------------------------------------------- +function Resolve-ExePath { + param([string]$Override) + + if ($Override) { + if (-not (Test-Path -LiteralPath $Override)) { + throw "Player executable not found: $Override" + } + return (Resolve-Path -LiteralPath $Override).Path + } + + $candidates = @( + (Join-Path $PSScriptRoot 'dist\KiwySignagePlayer\KiwySignagePlayer.exe'), + (Join-Path $PSScriptRoot 'KiwySignagePlayer.exe') + ) + foreach ($c in $candidates) { + if (Test-Path -LiteralPath $c) { + return (Resolve-Path -LiteralPath $c).Path + } + } + throw ("Could not find KiwySignagePlayer.exe. Looked in:`n " + + ($candidates -join "`n ")) +} + +$ExeFull = Resolve-ExePath -Override $ExePath +$PlayerDir = Split-Path -Parent $ExeFull +$HeartbeatFile = Join-Path $PlayerDir '.player_heartbeat' +$StopFlagFile = Join-Path $PlayerDir '.player_stop_requested' +$LogDir = Join-Path $PlayerDir 'logs' +$LogFile = Join-Path $LogDir 'watchdog.log' +$ProcessName = [System.IO.Path]::GetFileNameWithoutExtension($ExeFull) + +New-Item -ItemType Directory -Force -Path $LogDir | Out-Null + +# --------------------------------------------------------------------- +# Logging (kept tiny and never throws) +# --------------------------------------------------------------------- +function Write-Log { + param( + [Parameter(Mandatory = $true)][string]$Message, + [ValidateSet('INFO', 'WARN', 'ERROR')][string]$Level = 'INFO' + ) + $line = "[{0}] [{1}] {2}" -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Level, $Message + Write-Host $line + try { + Add-Content -LiteralPath $LogFile -Value $line -Encoding UTF8 + } catch { + # Logging must never take the supervisor down. + } +} + +# --------------------------------------------------------------------- +# Player process helpers +# --------------------------------------------------------------------- +function Get-PlayerProcesses { + # NOTE: the packaged player is TWO processes on Windows - the PyInstaller + # bootloader parent and the child that owns the SDL window. Both share the + # image name, so selecting by name covers the whole tree. + @(Get-Process -Name $ProcessName -ErrorAction SilentlyContinue) +} + +function Get-PlayerRootPid { + $procs = Get-PlayerProcesses + if ($procs.Count -eq 0) { return $null } + # Prefer the child (it has the main window); fall back to any. + $withWindow = $procs | Where-Object { $_.MainWindowHandle -ne 0 } | Select-Object -First 1 + if ($withWindow) { return $withWindow.Id } + return ($procs | Select-Object -First 1).Id +} + +function Test-HeartbeatFresh { + param([int]$MaxAgeSec) + + if (-not (Test-Path -LiteralPath $HeartbeatFile)) { + return $false + } + try { + $age = (Get-Date).ToUniversalTime() - ` + (Get-Item -LiteralPath $HeartbeatFile).LastWriteTimeUtc + return ($age.TotalSeconds -lt $MaxAgeSec) + } catch { + return $false + } +} + +function Get-HeartbeatAgeSec { + if (-not (Test-Path -LiteralPath $HeartbeatFile)) { return -1 } + try { + $age = (Get-Date).ToUniversalTime() - ` + (Get-Item -LiteralPath $HeartbeatFile).LastWriteTimeUtc + return [int]$age.TotalSeconds + } catch { + return -1 + } +} + +function Stop-PlayerTree { + param([int]$RootPid) + + # /T kills the PyInstaller child too; without it the visible player window + # would survive and the next launch would collide with it. + try { + & taskkill.exe /F /T /PID $RootPid 2>&1 | Out-Null + } catch { + Write-Log "taskkill failed for pid $RootPid ($($_.Exception.Message)); forcing by name" 'WARN' + } + + # Belt and braces: make sure no stragglers remain. + $deadline = (Get-Date).AddSeconds(15) + while ((Get-Date) -lt $deadline) { + if ((Get-PlayerProcesses).Count -eq 0) { return $true } + Start-Sleep -Milliseconds 500 + } + foreach ($p in Get-PlayerProcesses) { + try { Stop-Process -Id $p.Id -Force -ErrorAction SilentlyContinue } catch { } + } + Start-Sleep -Milliseconds 500 + return ((Get-PlayerProcesses).Count -eq 0) +} + +function Start-Player { + Write-Log "Launching player: $ExeFull" + # Start in the player's own folder: the app resolves config/media/playlists + # relative to its working directory. + Start-Process -FilePath $ExeFull -WorkingDirectory $PlayerDir | Out-Null +} + +# --------------------------------------------------------------------- +# -Status : report and exit (never mutates anything) +# --------------------------------------------------------------------- +if ($Status) { + $procs = Get-PlayerProcesses + $pid0 = Get-PlayerRootPid + $age = Get-HeartbeatAgeSec + Write-Host '==========================================' + Write-Host ' Kiwy Signage Player - Watchdog Status' + Write-Host '==========================================' + Write-Host ("Executable : {0}" -f $ExeFull) + Write-Host ("Processes : {0}" -f $procs.Count) + if ($pid0) { Write-Host ("Root PID : {0}" -f $pid0) } + if ($age -ge 0) { + Write-Host ("Heartbeat : {0}s old" -f $age) + if ($age -lt $HeartbeatStaleSec) { + Write-Host 'Health : HEALTHY' -ForegroundColor Green + } else { + Write-Host 'Health : STALE (player may be hung)' -ForegroundColor Yellow + } + } else { + Write-Host 'Heartbeat : (not present - player has not started)' + } + if (Test-Path -LiteralPath $StopFlagFile) { + Write-Host 'Stop flag : PRESENT (password exit was used)' + Write-Host ' a new launch clears it' -ForegroundColor Yellow + } else { + Write-Host 'Stop flag : absent (watchdog would restart on failure)' + } + Write-Host ("Log : {0}" -f $LogFile) + exit 0 +} + +# --------------------------------------------------------------------- +# Session start: clear the stop flag +# --------------------------------------------------------------------- +# This is what makes the flag session-scoped: the user's password exit ends +# the CURRENT session, and the next launch begins a new one. +if (Test-Path -LiteralPath $StopFlagFile) { + try { + Remove-Item -LiteralPath $StopFlagFile -Force + Write-Log 'Cleared the stop flag from the previous session - new session started' + } catch { + Write-Log "Could not clear the stop flag ($($_.Exception.Message))" 'WARN' + } +} + +Write-Log '==================================================' +Write-Log "Watchdog starting (exe=$ProcessName, check=${HealthCheckIntervalSec}s, stale=${HeartbeatStaleSec}s)" +Write-Log "Crash-loop breaker: ${CrashLoopMaxFailures} failures / ${CrashLoopWindowMin} min -> ${CrashLoopBackoffMin} min backoff" +Write-Log 'Stop the WATCHDOG with Ctrl+C. Stop the PLAYER via the exit password.' +Write-Log '==================================================' + +# --------------------------------------------------------------------- +# Crash-loop breaker +# --------------------------------------------------------------------- +# A player that cannot stay up (broken config, missing media, GPU fault) would +# otherwise be restarted forever, filling the log and thrashing the machine. +# If it fails CrashLoopMaxFailures times inside CrashLoopWindowMin minutes +# WITHOUT ever becoming healthy, wait CrashLoopBackoffMin minutes before the +# next attempt. Returns $true when a backoff was performed. +function Invoke-CrashLoopBackoff { + param([System.Collections.ArrayList]$Failures) + + if ($Failures.Count -lt $CrashLoopMaxFailures) { return $false } + + $windowStart = (Get-Date).AddMinutes(-$CrashLoopWindowMin) + $recent = @($Failures | Where-Object { $_ -gt $windowStart }) + if ($recent.Count -lt $CrashLoopMaxFailures) { + # Only old failures remain; forget them so they cannot accumulate. + $Failures.Clear() + return $false + } + + Write-Log ("Player failed $($recent.Count) times in the last " + + "${CrashLoopWindowMin} minutes without ever becoming healthy.") 'ERROR' + Write-Log "Backing off for ${CrashLoopBackoffMin} minutes before the next attempt." 'ERROR' + Write-Log "Investigate $LogDir (and the player's own logs next to it)." 'ERROR' + + $backoffEnd = (Get-Date).AddMinutes($CrashLoopBackoffMin) + while ((Get-Date) -lt $backoffEnd) { + Start-Sleep -Seconds 5 + if (Test-Path -LiteralPath $StopFlagFile) { + Write-Log 'Stop flag appeared during backoff - standing down.' + $Failures.Clear() + return $true + } + } + $Failures.Clear() + Write-Log 'Backoff finished - resuming supervision.' + return $true +} + +# --------------------------------------------------------------------- +# Main supervision loop +# --------------------------------------------------------------------- +# Two DIFFERENT failures need two different reactions, and confusing them +# would be harmful: +# +# * "never became healthy" - a slow start-up, a bad install, or the +# first-run setup screen. Killing on this would restart forever and could +# interrupt an operator entering settings. Handled by the crash-loop +# breaker, with a long hard cap before we intervene. +# * "was healthy, then went silent" - the app is wedged. This is the case +# that must be restarted promptly. +# +# So a hang is only declared once we have actually SEEN a fresh heartbeat. +$failureTimestamps = New-Object System.Collections.ArrayList +$hasBeenHealthy = $false + +while ($true) { + + # ---- The operator asked to exit: stand down ---------------------- + if (Test-Path -LiteralPath $StopFlagFile) { + Write-Log 'Stop flag present - the operator exited with the password.' + Write-Log 'Watchdog will NOT restart the player. Start it again to resume.' + break + } + + $procs = Get-PlayerProcesses + + # ================================================================ + # Case 1: the player is not running + # ================================================================ + if ($procs.Count -eq 0) { + Write-Log 'Player is not running - starting it.' + + try { + Start-Player + } catch { + Write-Log "Failed to launch the player: $($_.Exception.Message)" 'ERROR' + [void]$failureTimestamps.Add((Get-Date)) + [void](Invoke-CrashLoopBackoff -Failures $failureTimestamps) + Start-Sleep -Seconds $RestartDelaySec + continue + } + + $launchTime = Get-Date + $hasBeenHealthy = $false + + # Watch the start-up window: break early the moment it is healthy, + # and notice immediately if it dies. + $diedEarly = $false + $graceEnd = $launchTime.AddSeconds($StartupGraceSec) + while ((Get-Date) -lt $graceEnd) { + Start-Sleep -Seconds 2 + if (Test-Path -LiteralPath $StopFlagFile) { break } + if ((Get-PlayerProcesses).Count -eq 0) { $diedEarly = $true; break } + if (Test-HeartbeatFresh -MaxAgeSec $HeartbeatStaleSec) { + $hasBeenHealthy = $true + break + } + } + + if ($diedEarly -and -not (Test-Path -LiteralPath $StopFlagFile)) { + [void]$failureTimestamps.Add((Get-Date)) + Write-Log ("Player exited during start-up (within the ${StartupGraceSec}s grace window).") 'WARN' + [void](Invoke-CrashLoopBackoff -Failures $failureTimestamps) + Write-Log ("Waiting ${RestartDelaySec}s before retrying...") + Start-Sleep -Seconds $RestartDelaySec + } elseif ($hasBeenHealthy) { + Write-Log 'Player started and is reporting a fresh heartbeat.' + $failureTimestamps.Clear() + } else { + Write-Log 'Player is running but has not reported a heartbeat yet - continuing to watch.' 'WARN' + } + continue + } + + # ================================================================ + # Case 2: the player is running - is it actually working? + # ================================================================ + if (Test-HeartbeatFresh -MaxAgeSec $HeartbeatStaleSec) { + # Healthy. Forget past failures - only CONSECUTIVE ones matter. + if (-not $hasBeenHealthy -or $failureTimestamps.Count -gt 0) { + Write-Log 'Player is healthy.' + } + $hasBeenHealthy = $true + $failureTimestamps.Clear() + Start-Sleep -Seconds $HealthCheckIntervalSec + continue + } + + $age = Get-HeartbeatAgeSec + $ageText = if ($age -lt 0) { 'no heartbeat file' } else { "${age}s old" } + + if ($hasBeenHealthy) { + # ---- The real hang: it WAS working and has gone silent ---------- + Write-Log ("Player was healthy but its heartbeat is now stale ({0}, limit ${HeartbeatStaleSec}s) - it is hung." -f $ageText) 'ERROR' + + $rootPid = Get-PlayerRootPid + if ($rootPid) { + Write-Log "Killing the hung player (pid $rootPid) and its children..." + [void](Stop-PlayerTree -RootPid $rootPid) + } + $hasBeenHealthy = $false + [void]$failureTimestamps.Add((Get-Date)) + + if (Test-Path -LiteralPath $StopFlagFile) { continue } + [void](Invoke-CrashLoopBackoff -Failures $failureTimestamps) + Write-Log ("Waiting ${RestartDelaySec}s before restarting...") + Start-Sleep -Seconds $RestartDelaySec + continue + } + + # ---- Running but never healthy: be patient, then give up ----------- + # Deliberately generous: a slow machine still inside its start-up window + # must not be killed, and neither must the first-run setup screen. + $stuckLimitSec = $StartupGraceSec * 2 + if ($null -eq $launchTime) { $launchTime = Get-Date } + $upSec = ((Get-Date) - $launchTime).TotalSeconds + + if ($upSec -lt $stuckLimitSec) { + Write-Log ("Player is up but not yet healthy ({0}); still inside the ${stuckLimitSec}s start-up allowance." -f $ageText) 'WARN' + Start-Sleep -Seconds $HealthCheckIntervalSec + continue + } + + Write-Log ("Player has not become healthy within ${stuckLimitSec}s ({0}) - restarting it." -f $ageText) 'ERROR' + $rootPid = Get-PlayerRootPid + if ($rootPid) { [void](Stop-PlayerTree -RootPid $rootPid) } + [void]$failureTimestamps.Add((Get-Date)) + [void](Invoke-CrashLoopBackoff -Failures $failureTimestamps) + Write-Log ("Waiting ${RestartDelaySec}s before retrying...") + Start-Sleep -Seconds $RestartDelaySec +} + +Write-Log 'Watchdog exited.'