Repository navigation
feat(cluster): restart a Colima VM whose runtime stops answering between runs - #260
Merged
Merged
Conversation
…een runs A VZ-backed Colima VM can stop without a crash report while colima list still says Running. tasks/runtime_wedged.yml recovers it, but only when the playbook runs, so the node and its runner pods stay down until then. The cluster role now installs a launchd job (a daemon for the profile started at boot, an agent for any other) that runs files/colima-watchdog.sh every minute. The script acts only on the evidence the converge-time check asks for: docker info failing on consecutive runs, a failing guest command, a host agent older than the boot grace and a Docker context that points at the profile's socket. Restarts are capped per hour with a doubling gap, a failed restart counts, and each decision is one log line. github_runner_cluster_runtime_watchdog turns it off.
|
🎉 This PR is included in version 3.4.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The cluster role's wedge handling (
tasks/runtime_wedged.yml) only runs with the playbook. A VZ-backed Colima VM that stops without a crash report whilecolima liststill says Running stays down, with the node and its runner pods, until someone converges the host.This adds a launchd job beside the existing start job (a daemon for the boot profile, an agent for any other) that runs
files/colima-watchdog.shevery 60 seconds. A run wheredocker infoanswers prints nothing. It restarts the VM (colima stop,--forceif that fails, thencolima start --activate=false) only when all of these hold:docker infofailed, each try bounded by the probe timeout, on 3 consecutive runsEach decision is one line in
~/Library/Logs/<label>.log.github_runner_cluster_runtime_watchdog: falseremoves the job.Tests:
tests/unit/test_colima_watchdog.pyruns the script against stand-ins for docker, colima and ps (when it acts, when it leaves the VM alone, cap and backoff);tests/unit/test_runtime_autostart.pycovers the template, defaults and the role's install, reload and removal of the job. The stand-in launchctl now tracks each job's loaded state separately.Not run against a real VM or host.