Skip to content

feat(cluster): restart a Colima VM whose runtime stops answering between runs - #260

Merged
Mearman merged 1 commit into
mainfrom
feat/colima-watchdog
Nov 17, 2026
Merged

Mearman merged 1 commit into
mainfrom
feat/colima-watchdog

Conversation

@Mearman

@Mearman Mearman commented Nov 17, 2026

Copy link
Copy Markdown
Member

The cluster role's wedge handling (tasks/runtime_wedged.yml) only runs with the playbook. A VZ-backed Colima VM that stops without a crash report while colima list still says Running stays down, with the node and its runner pods, until someone converges the host.

This adds a launchd job beside the existing start job (a daemon for the boot profile, an agent for any other) that runs files/colima-watchdog.sh every 60 seconds. A run where docker info answers prints nothing. It restarts the VM (colima stop, --force if that fails, then colima start --activate=false) only when all of these hold:

  • docker info failed, each try bounded by the probe timeout, on 3 consecutive runs
  • Colima lists the profile Running, a command in the guest fails, the VM's host agent is older than the boot grace, and the Docker context points at the profile's socket
  • fewer than 3 restarts in the last hour and the gap since the previous one is at least 600 seconds, doubled for each further restart in that hour; a failed restart counts

Each decision is one line in ~/Library/Logs/<label>.log. github_runner_cluster_runtime_watchdog: false removes the job.

Tests: tests/unit/test_colima_watchdog.py runs the script against stand-ins for docker, colima and ps (when it acts, when it leaves the VM alone, cap and backoff); tests/unit/test_runtime_autostart.py covers the template, defaults and the role's install, reload and removal of the job. The stand-in launchctl now tracks each job's loaded state separately.

Not run against a real VM or host.

…een runs

A VZ-backed Colima VM can stop without a crash report while colima list
still says Running. tasks/runtime_wedged.yml recovers it, but only when
the playbook runs, so the node and its runner pods stay down until then.

The cluster role now installs a launchd job (a daemon for the profile
started at boot, an agent for any other) that runs files/colima-watchdog.sh
every minute. The script acts only on the evidence the converge-time check
asks for: docker info failing on consecutive runs, a failing guest command,
a host agent older than the boot grace and a Docker context that points at
the profile's socket. Restarts are capped per hour with a doubling gap, a
failed restart counts, and each decision is one log line.

github_runner_cluster_runtime_watchdog turns it off.
@Mearman
Mearman marked this pull request as ready for review November 17, 2026 05:03
@Mearman
Mearman merged commit d99991d into main Nov 17, 2026
57 of 59 checks passed
@Mearman
Mearman deleted the feat/colima-watchdog branch November 17, 2026 05:03
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 3.4.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant