From f8c3e9de60fbb36f9d1680fdb2c6128b7e60d867 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Tue, 17 Nov 2026 00:32:46 +0000 Subject: [PATCH] feat(cluster): restart a Colima VM whose runtime stops answering between runs A VZ-backed Colima VM can stop without a crash report while colima list still says Running. tasks/runtime_wedged.yml recovers it, but only when the playbook runs, so the node and its runner pods stay down until then. The cluster role now installs a launchd job (a daemon for the profile started at boot, an agent for any other) that runs files/colima-watchdog.sh every minute. The script acts only on the evidence the converge-time check asks for: docker info failing on consecutive runs, a failing guest command, a host agent older than the boot grace and a Docker context that points at the profile's socket. Restarts are capped per hour with a doubling gap, a failed restart counts, and each decision is one log line. github_runner_cluster_runtime_watchdog turns it off. --- roles/github_runner_cluster/README.md | 11 + roles/github_runner_cluster/defaults/main.yml | 15 ++ .../files/colima-watchdog.sh | 189 ++++++++++++++++ roles/github_runner_cluster/tasks/runtime.yml | 4 + .../tasks/runtime_watchdog.yml | 114 ++++++++++ .../colima-watchdog-launchd.plist.j2 | 37 +++ tests/runtime_autostart/render_plist.yml | 6 +- tests/unit/test_colima_watchdog.py | 213 ++++++++++++++++++ tests/unit/test_runtime_autostart.py | 106 ++++++++- 9 files changed, 691 insertions(+), 4 deletions(-) create mode 100755 roles/github_runner_cluster/files/colima-watchdog.sh create mode 100644 roles/github_runner_cluster/tasks/runtime_watchdog.yml create mode 100644 roles/github_runner_cluster/templates/colima-watchdog-launchd.plist.j2 create mode 100644 tests/unit/test_colima_watchdog.py diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index b2e351c8..4aacf65a 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -181,6 +181,16 @@ A Mac still has to log in for the agent to run: a launch agent starts with the u A host that got the agent from release 2.19.0 moves over on its next run: the role unloads the node profile's agent from the user's login session when there is one, deletes its property list (also when nobody has logged in, so it cannot load at the next login and start the profile a second time) and loads the daemon. Unloading either kind does not stop the VM, which the job started detached, and loading one runs `colima start`, which does nothing to a running profile. A rerun with nothing changed makes no `launchctl` call that changes anything. With `github_runner_cluster_runtime_autostart` off the role installs, loads and removes none of this. +**Watchdog.** `tasks/runtime_wedged.yml` stops and starts a wedged VM only while the playbook runs, so a VM that wedges between runs stays down, with its node and runner pods, until someone converges the host. With `github_runner_cluster_runtime_watchdog` on (the default, under `github_runner_cluster_runtime_autostart`) the role installs a launchd job beside the start job, `com.exadev.github-runner.colima-watchdog.`, that runs `files/colima-watchdog.sh` every `github_runner_cluster_runtime_watchdog_interval` seconds (a daemon for the profile started at boot, an agent for any other, as for the start job). The script, `run_bounded.pl` and the script's state live under `~/.local/share/exadev-github-runner/colima-watchdog-`, and its log is `~/Library/Logs/