Skip to content
23 changes: 21 additions & 2 deletions plugins/filter/container_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,28 @@ def cluster_colima_home(endpoint: Any, profile: Any, user_home: Any) -> str:
return colima_home


SUPERVISOR_FAILURE_STATUSES = (127, 255)
SUPERVISOR_FAILURE_MESSAGE = "Error executing command"


def cluster_supervisor_failed(result: Any) -> bool:
"""Say whether the registered result of a command run under ``files/run_bounded.pl`` shows that the supervisor itself could not run, as opposed to the supervised command failing or timing out.

The supervisor exits 127 when it cannot start the command and 255 when it is misused (and the command module's own exit 127 is the same case for a missing interpreter). When the command module cannot start ``perl`` at all (not on the PATH, not executable) it reports the failure with the message ``Error executing command`` and an ``rc`` that is the OS error number (2 for a missing file, 13 for a permission error) or 257, so the message is what identifies it, whatever the ``rc``.

:param result: The registered result of the command task: a mapping with ``rc`` and, when the module could not start the command, ``msg``.
:returns: True when the run says nothing about the command, so it must fail the play and never count as the command failing.
:raises AnsibleFilterError: the value is not a mapping.
"""
if not isinstance(result, dict):
raise AnsibleFilterError(f"The result of a supervised command must be a mapping, not {type(result).__name__}")
message = result.get("msg")
return result.get("rc") in SUPERVISOR_FAILURE_STATUSES or (isinstance(message, str) and SUPERVISOR_FAILURE_MESSAGE in message)


class FilterModule:
"""Registers the container runtime filter with Ansible."""
"""Registers the container runtime filters with Ansible."""

def filters(self) -> dict[str, Any]:
"""Return the filters this plugin provides."""
return {"cluster_container_runtime": cluster_container_runtime, "cluster_colima_home": cluster_colima_home}
return {"cluster_container_runtime": cluster_container_runtime, "cluster_colima_home": cluster_colima_home, "cluster_supervisor_failed": cluster_supervisor_failed}
3 changes: 2 additions & 1 deletion roles/github_runner_cluster/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ On a Mac the container runtime is a Linux VM (Colima, or Docker Desktop) that st

Which runtime a Mac runs is taken from the Docker context the role drives (see [Docker contexts](#docker-contexts)): `colima`, or `colima-<profile>` for another profile, is Colima, and `desktop-linux` is Docker Desktop. `github_runner_cluster_runtime` says so explicitly for a host whose context does not tell, for example one that uses `DOCKER_HOST`, and `github_runner_cluster_colima_profile` names the profile to start when the context does not.

**Colima.** With `github_runner_cluster_runtime_autostart` on (the default, which acts on Colima hosts only), the role starts the profile if it is stopped, with `colima start --profile <profile> --activate=false`, so the profile does not make its context the host's current one, and installs the launchd job that starts it again after a reboot. There are two kinds, chosen by `github_runner_cluster_runtime_autostart_at` (`login`, `boot` or `auto`, the default). `auto` is `boot` for the profile named by `github_runner_cluster_runtime_boot_profile` (default `node`, the profile the bridged k3s node runs in) and `login` for any other, such as a `default` profile a host keeps for other workloads. A profile has one kind at a time: installing either retires the other's property list, unloading it first, so the profile is not started twice.
**Colima.** With `github_runner_cluster_runtime_autostart` on (the default, which acts on Colima hosts only), the role starts the profile if it is stopped, with `colima start --profile <profile> --activate=false`, so the profile does not make its context the host's current one. A profile that `colima list` shows as running while its Docker runtime does not answer is not started again, because a start finds it running and does nothing. The role stops it, with `colima stop` and then `colima stop --force` only if that fails, and starts it again from the same disk, but only when `github_runner_cluster_runtime_force_stop_wedged` is on (the default) and all of these hold: `docker info` fails on `github_runner_cluster_runtime_probe_attempts` further tries (default 5), each given `github_runner_cluster_runtime_probe_timeout` seconds (default 20) and `github_runner_cluster_runtime_probe_delay` seconds apart (default 10); a command in the guest (`colima ssh -- true`) fails within the same timeout; the VM's host agent has been running for more than `github_runner_cluster_runtime_boot_grace` seconds (default 600), read from the elapsed time of the live process that the pid file in the profile's Lima instance directory names, and only when the program that process runs is Lima's host agent (named limactl, with the argument hostagent; a missing file, a dead pid, another program or an unreadable time leaves the age unknown, which never counts as old, whatever the file's date); and the Docker context the probes used can be read and points at that profile's own socket. A host that names its Colima home with `github_runner_cluster_colima_home` because it has no context that says (it uses `DOCKER_HOST`, say) runs as before, but its stop is never proven and the run fails with the signals it saw instead of stopping anything. Every probe, the guest command and the clean stop run under `files/run_bounded.pl`, which owns the deadline: it sends TERM to the command's process group when the time is up and KILL `github_runner_cluster_runtime_kill_grace` seconds later (default 5), so a program that ignores signals or never replies (docker and colima ignore `SIGALRM`) ends the try as failed and cannot hold the run; the first probe of the runtime is bounded the same way. If the supervisor itself is signalled (TERM, INT or HUP) it ends the command's process group the same way and exits 143; KILL of the supervisor cannot be handled and leaves the command running. Only the exit status 124 (deadline) or the tool's own non-zero status counts as a failed try: a supervisor that cannot start the command (127), is misused (255) or is missing (no `perl` on the PATH the role uses) fails the run and names `files/run_bounded.pl`, and is never evidence about the VM. A host that is only loaded, a Docker daemon that is restarting, a VM that is still booting and a context that points at another runtime each fail one of these, so nothing is stopped. A stop does not shut the guest down gracefully when the clean stop fails or times out (`github_runner_cluster_runtime_clean_stop_timeout`, default 120 seconds): every process in the guest, k3s and etcd included, ends, and the VM's disk, containers and volumes are kept, so on an etcd voter the member is out until the start completes. Switched off, or when a condition does not hold, the run fails with the signals it saw and the commands to recover by hand. With all of this in force the stop never follows a single failed or slow probe, never touches a VM whose guest answers or whose agent process is young or unknown, and never ends in a hang; that is why the switch is on by default. Under `--check` the reads are made, nothing is stopped and the report says what a real run would do. It installs the launchd job that starts it again after a reboot. There are two kinds, chosen by `github_runner_cluster_runtime_autostart_at` (`login`, `boot` or `auto`, the default). `auto` is `boot` for the profile named by `github_runner_cluster_runtime_boot_profile` (default `node`, the profile the bridged k3s node runs in) and `login` for any other, such as a `default` profile a host keeps for other workloads. A profile has one kind at a time: installing either retires the other's property list, unloading it first, so the profile is not started twice.

**At login (an agent).** The role:

Expand Down Expand Up @@ -491,6 +491,7 @@ The role controller holds the regrade Lease (`kube-system/fleet-regrade`, the Le
- `github_runner_cluster_docker_context` (default empty): pins the Docker context this host's node runs in (see [Docker contexts](#docker-contexts)).
- `github_runner_cluster_network_mode` (default `bridge`, or `host`): the network the node's container runs on, and with `host`, `github_runner_cluster_lan_cidr` (required: the LAN the nodes share) and `github_runner_cluster_network_probe_image` (default `busybox:1.37`, for reading the host's addresses) (see [Networking](#networking)).
- `github_runner_cluster_runtime_autostart_at` (default `auto`; `login` or `boot`) and `github_runner_cluster_runtime_boot_profile` (default `node`): whether the Colima profile is started by a launchd agent at login or by a launchd daemon at boot, and which profile `auto` starts at boot (see [Container runtime after a reboot](#container-runtime-after-a-reboot-macos)).
- `github_runner_cluster_runtime_force_stop_wedged` (default true), `github_runner_cluster_runtime_probe_attempts` (default 5), `github_runner_cluster_runtime_probe_delay` (default 10), `github_runner_cluster_runtime_probe_timeout` (default 20), `github_runner_cluster_runtime_boot_grace` (default 600), `github_runner_cluster_runtime_kill_grace` (default 5) and `github_runner_cluster_runtime_clean_stop_timeout` (default 120): when a Colima profile listed as running with a silent runtime is stopped, see the Colima paragraph above.
- `github_runner_cluster_runtime_autostart` (default true), `github_runner_cluster_runtime` (default `auto`; `colima`, `docker_desktop` or `other`), `github_runner_cluster_colima_profile` (default empty, taken from the Docker context), `github_runner_cluster_colima_home` (default empty: the `COLIMA_HOME` is read from the endpoint of the profile's Docker context, and is unset for Colima's own `~/.colima`; set it only for a context that does not say, and the role passes it to colima and to the launchd agent), `github_runner_cluster_runtime_path` (default both Homebrew prefixes and the system directories): on a macOS host, start Colima at login through a launchd agent and check the runtime (see [Container runtime at login](#container-runtime-after-a-reboot-macos)).
- `github_runner_cluster_colima_memory_gib` and `github_runner_cluster_colima_cpus` (default empty, keeping the profile's own): a Colima host's VM memory in GiB and its CPUs, each a positive whole number, applied one node at a time through the quorum guard when the VM differs (see Colima VM size under [Container runtime after a reboot](#container-runtime-after-a-reboot-macos)). `github_runner_cluster_colima_memory_reserved_gib` (default empty) derives the memory instead, or bounds one set explicitly, from the host's memory less it and `github_runner_cluster_colima_memory_host_reserve_gib` (default 10, macOS's own), never below `github_runner_cluster_colima_guest_baseline_gib` (default 3.4) plus `github_runner_cluster_colima_runner_memory_limit_gib` (default empty, required by a derivation) rounded up. `github_runner_cluster_colima_resize_lease_seconds`, `github_runner_cluster_colima_resize_lease_holder`, `github_runner_cluster_regrade_lease_name` and `github_runner_cluster_regrade_lease_namespace` are derived, never set directly. `github_runner_cluster_native_runner_wait_timeout` and `github_runner_cluster_native_ready_timeout` bound the wait for the node's runner pods and for the node to come back, as they do for a native node.
- `github_runner_cluster_container_name` (default `github-runner-k3s`) and `github_runner_cluster_agent_container_name` (default `github-runner-k3s-agent`): the names of the node's server and agent containers, which a second cluster on the same Docker hosts sets to names of its own (see [Several clusters on one host](#several-clusters-on-one-host)).
Expand Down
15 changes: 14 additions & 1 deletion roles/github_runner_cluster/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,20 @@ github_runner_cluster_server_ready_timeout: 600
github_runner_cluster_runtime_autostart: true
# When the role starts a Colima profile: login installs a per-user launchd agent (a property list in the connecting user's ~/Library/LaunchAgents, loaded into the user's login session), which runs only once that user has logged in; boot installs a launchd daemon (/Library/LaunchDaemons, owned by root, loaded into the system domain, run as the connecting user through UserName), which runs when the Mac starts, with nobody logged in, so a node recovers from a power loss or a reboot at the login window; auto is boot for the profile named by github_runner_cluster_runtime_boot_profile and login for any other. Only one of the two exists for a profile at a time: installing either removes the other's property list, and unloads it, so the profile is not started twice. A daemon takes root (become) to write and to load, which the agent never does.
github_runner_cluster_runtime_autostart_at: auto
# The Colima profile that runs this collection's k3s node, which must come back with no one logged in: the one github_runner_cluster_runtime_autostart_at auto starts at boot. Any other profile on the host (the mini's default profile for its own workloads, say) keeps the login agent.
# Whether the role may stop a Colima profile that Colima lists as running while its container runtime does not answer, so that the start after it can boot it again. It does so only when tasks/runtime_wedged.yml has confirmed the failure (repeated probes, a failing guest command, a host agent older than the boot grace and a Docker context that points at that profile's socket). Off, such a profile fails the run with what was seen and the commands to recover by hand. Separate from github_runner_cluster_runtime_autostart, which starts a stopped profile and installs the launchd job.
github_runner_cluster_runtime_force_stop_wedged: true
# How many times docker info is tried, and the seconds between tries, before a profile listed as running with a silent runtime is looked into further. Together with the timeout they set the window in which a loaded host or a restarting Docker daemon can still answer.
github_runner_cluster_runtime_probe_attempts: 5
github_runner_cluster_runtime_probe_delay: 10
# The seconds each docker info, and the command run in the guest, is given before it counts as failed (a run that does not end by then is ended by files/run_bounded.pl).
github_runner_cluster_runtime_probe_timeout: 20
# The seconds a VM's host agent process must have been running before a silent runtime counts as wedged rather than booting.
github_runner_cluster_runtime_boot_grace: 600
# The seconds a probe, a guest command or a clean stop that has passed its deadline is given to end after TERM before it is killed.
github_runner_cluster_runtime_kill_grace: 5
# The seconds colima stop is given before colima stop --force follows it.
github_runner_cluster_runtime_clean_stop_timeout: 120
# The Colima profile that runs this collection's k3s node, which must come back with no one logged in: the one github_runner_cluster_runtime_autostart_at auto starts at boot. Any other profile on the host (a `default` profile a host keeps for other workloads, say) keeps the login agent.
github_runner_cluster_runtime_boot_profile: node
# Which container runtime a macOS host runs: auto takes it from the Docker context the role drives (colima or colima-<profile> is Colima, desktop-linux is Docker Desktop, anything else is other), colima, docker_desktop and other say so for a host whose context does not tell (DOCKER_HOST set, a context with another name).
github_runner_cluster_runtime: auto
Expand Down
65 changes: 65 additions & 0 deletions roles/github_runner_cluster/files/run_bounded.pl
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
#!/usr/bin/perl
# Runs a command with a deadline that binds whatever the command does with signals. Usage: run_bounded.pl DEADLINE GRACE COMMAND [ARGUMENT...] (both in seconds).
#
# The command runs in a process group of its own. When DEADLINE passes, the group gets TERM; if any member is still there GRACE seconds later it gets KILL. A timer set with alarm before exec does not do this: a Go program such as docker or colima does not exit on SIGALRM, and a program may ignore TERM too. The deadline is measured on the monotonic clock, so a change of the system time does not move it.
#
# The exit status is the command's own, 128 plus the signal number when a signal ended it, 124 when the deadline passed (as timeout(1) reports it), 127 when the command could not be started and 255 when this script was misused. 143 means this supervisor itself was sent TERM, INT or HUP: the command's group is then ended the same way (TERM, then KILL after GRACE) so that nothing is left running on its own. KILL of the supervisor cannot be handled and leaves the command running. The command's standard input, output and error are the caller's.
use strict;
use warnings;
use POSIX qw(WNOHANG setsid);
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC sleep);

my ($deadline, $grace, @command) = @ARGV;
die "usage: run_bounded.pl DEADLINE GRACE COMMAND [ARGUMENT...]\n" unless @command && defined $grace;

sub now { return clock_gettime(CLOCK_MONOTONIC); }

my $child = fork();
die "fork: $!\n" unless defined $child;
if ($child == 0) {
setsid();
exec { $command[0] } @command or exit 127;
}

# The status of the child once it has ended, in the shell's convention.
sub status_of {
my ($status) = @_;
return $status & 127 ? 128 + ($status & 127) : $status >> 8;
}

# True once the child has ended, with its status in $status.
our $status;
sub ended {
return 1 if defined $status;
my $done = waitpid($child, WNOHANG);
return 0 if $done == 0;
$status = status_of($?);
return 1;
}

# Ends the child's group: TERM, then KILL when GRACE has passed or the leader is already gone. The group is killed whether or not its leader ended, so that nothing it started outlives it.
sub end_group {
kill 'TERM', -$child;
my $kill_at = now() + $grace;
while (now() < $kill_at) {
last if ended();
sleep 0.05;
}
kill 'KILL', -$child;
waitpid($child, 0) unless defined $status;
}

for my $name (qw(TERM INT HUP)) {
$SIG{$name} = sub { end_group(); exit 143; };
}

my $end = now() + $deadline;
while (now() < $end) {
exit $status if ended();
sleep 0.05;
}
# One last look, so that a command that ended in the final poll interval is not reported as timed out.
exit $status if ended();

end_group();
exit 124;
Loading
Loading