Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions roles/github_runner_cluster/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,16 @@ A Mac still has to log in for the agent to run: a launch agent starts with the u

A host that got the agent from release 2.19.0 moves over on its next run: the role unloads the node profile's agent from the user's login session when there is one, deletes its property list (also when nobody has logged in, so it cannot load at the next login and start the profile a second time) and loads the daemon. Unloading either kind does not stop the VM, which the job started detached, and loading one runs `colima start`, which does nothing to a running profile. A rerun with nothing changed makes no `launchctl` call that changes anything. With `github_runner_cluster_runtime_autostart` off the role installs, loads and removes none of this.

**Watchdog.** `tasks/runtime_wedged.yml` stops and starts a wedged VM only while the playbook runs, so a VM that wedges between runs stays down, with its node and runner pods, until someone converges the host. With `github_runner_cluster_runtime_watchdog` on (the default, under `github_runner_cluster_runtime_autostart`) the role installs a launchd job beside the start job, `com.exadev.github-runner.colima-watchdog.<profile>`, that runs `files/colima-watchdog.sh` every `github_runner_cluster_runtime_watchdog_interval` seconds (a daemon for the profile started at boot, an agent for any other, as for the start job). The script, `run_bounded.pl` and the script's state live under `~/.local/share/exadev-github-runner/colima-watchdog-<profile>`, and its log is `~/Library/Logs/<label>.log`, which holds only decisions: a run that finds `docker info` answering prints nothing.

It restarts the VM (`colima stop`, then `colima stop --force` if that fails or passes `github_runner_cluster_runtime_clean_stop_timeout`, then `colima start --profile <profile> --activate=false`) only when all of these hold, the same evidence the converge-time check asks for:

- `docker info` failed, each try bounded by `github_runner_cluster_runtime_probe_timeout` seconds, on `github_runner_cluster_runtime_watchdog_failures` consecutive runs. Any answer resets the count, so a loaded host or a restarting daemon never adds up to a restart.
- Colima lists the profile as `Running` (a profile it lists as `Stopped` is left to the start job, since the role's own resize and an operator's `colima stop` stop it on purpose), a command in the guest (`colima ssh -- true`) fails, the VM's host agent has been running for more than `github_runner_cluster_runtime_boot_grace` seconds (which also keeps a VM that was just restarted from being restarted again), and the host's Docker context points at this profile's socket.
- The restart log allows it: at most `github_runner_cluster_runtime_watchdog_max_restarts` in the last hour, and at least `github_runner_cluster_runtime_watchdog_min_gap` seconds since the previous one, doubled for every further restart in that hour. A restart that fails counts. At the cap the script logs the commands to recover by hand and exits 1 on every run until an hour has passed.

A restart stops every container on the host until the VM is back, which the node's `restart: unless-stopped` containers then follow. Off, the role unloads the job and removes its property list, and leaves its directory in place.

**Docker Desktop.** The role cannot start Docker Desktop. Turn on Settings > General > Start Docker Desktop when you sign in to your computer on each such host (https://docs.docker.com/desktop/settings-and-maintenance/settings/), and give it the same automatic login. If it is not running when the role runs, the run fails with that message.

**End checks.** On every macOS host the role checks, before anything reads the daemon and again after it has started the node's Compose project, that the runtime answers (`docker info` through the host's Docker context), and after starting the project that the node's k3s container (`github-runner-k3s`, or `github-runner-k3s-agent` on an agent) is running. A converge of a host that rebooted therefore repairs a stopped Colima or fails with a message that says what to do, and never reports success over a node that is not up. A `--check` run does not fail on a stopped runtime or container it has not started.
Expand Down Expand Up @@ -508,6 +518,7 @@ The role controller holds the regrade Lease (`kube-system/fleet-regrade`, the Le
- `github_runner_cluster_etcd_lan_address` (per server, default empty) and `github_runner_cluster_etcd_peers_over_lan` (for the group, default false): a server's LAN address for etcd, and whether etcd's members use them, one member moved per run; `github_runner_cluster_etcd_lan_quiet_minutes` (default 10) and `github_runner_cluster_etcd_lan_settle_seconds` (default 60) bound the checks before and after each move (see [etcd peers over the LAN](#etcd-peers-over-the-lan)).
- `github_runner_cluster_network_mode` (default `bridge`, or `host`): the network the node's container runs on, and with `host`, `github_runner_cluster_lan_cidr` (required: the LAN the nodes share) and `github_runner_cluster_network_probe_image` (default `busybox:1.37`, for reading the host's addresses) (see [Networking](#networking)).
- `github_runner_cluster_runtime_autostart_at` (default `auto`; `login` or `boot`) and `github_runner_cluster_runtime_boot_profile` (default `node`): whether the Colima profile is started by a launchd agent at login or by a launchd daemon at boot, and which profile `auto` starts at boot (see [Container runtime after a reboot](#container-runtime-after-a-reboot-macos)).
- `github_runner_cluster_runtime_watchdog` (default true), `github_runner_cluster_runtime_watchdog_interval` (default 60), `github_runner_cluster_runtime_watchdog_failures` (default 3), `github_runner_cluster_runtime_watchdog_max_restarts` (default 3), `github_runner_cluster_runtime_watchdog_min_gap` (default 600) and `github_runner_cluster_runtime_watchdog_start_timeout` (default 900): the launchd job that restarts a Colima VM whose runtime stopped answering between runs (see Watchdog under [Container runtime after a reboot](#container-runtime-after-a-reboot-macos)).
- `github_runner_cluster_runtime_force_stop_wedged` (default true), `github_runner_cluster_runtime_probe_attempts` (default 5), `github_runner_cluster_runtime_probe_delay` (default 10), `github_runner_cluster_runtime_probe_timeout` (default 20), `github_runner_cluster_runtime_boot_grace` (default 600), `github_runner_cluster_runtime_kill_grace` (default 5) and `github_runner_cluster_runtime_clean_stop_timeout` (default 120): when a Colima profile listed as running with a silent runtime is stopped, see the Colima paragraph above.
- `github_runner_cluster_runtime_autostart` (default true), `github_runner_cluster_runtime` (default `auto`; `colima`, `docker_desktop` or `other`), `github_runner_cluster_colima_profile` (default empty, taken from the Docker context), `github_runner_cluster_colima_home` (default empty: the `COLIMA_HOME` is read from the endpoint of the profile's Docker context, and is unset for Colima's own `~/.colima`; set it only for a context that does not say, and the role passes it to colima and to the launchd agent), `github_runner_cluster_runtime_path` (default both Homebrew prefixes and the system directories): on a macOS host, start Colima at login through a launchd agent and check the runtime (see [Container runtime at login](#container-runtime-after-a-reboot-macos)).
- `github_runner_cluster_colima_memory_gib` and `github_runner_cluster_colima_cpus` (default empty, keeping the profile's own): a Colima host's VM memory in GiB and its CPUs, each a positive whole number, applied one node at a time through the quorum guard when the VM differs (see Colima VM size under [Container runtime after a reboot](#container-runtime-after-a-reboot-macos)). `github_runner_cluster_colima_memory_reserved_gib` (default empty) derives the memory instead, or bounds one set explicitly, from the host's memory less it and `github_runner_cluster_colima_memory_host_reserve_gib` (default 10, macOS's own), never below `github_runner_cluster_colima_guest_baseline_gib` (default 3.4) plus `github_runner_cluster_colima_runner_memory_limit_gib` (default empty, required by a derivation) rounded up. `github_runner_cluster_colima_resize_lease_seconds`, `github_runner_cluster_colima_resize_lease_holder`, `github_runner_cluster_regrade_lease_name` and `github_runner_cluster_regrade_lease_namespace` are derived, never set directly. `github_runner_cluster_native_runner_wait_timeout` and `github_runner_cluster_native_ready_timeout` bound the wait for the node's runner pods and for the node to come back, as they do for a native node.
Expand Down
15 changes: 15 additions & 0 deletions roles/github_runner_cluster/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,21 @@ github_runner_cluster_runtime_boot_grace: 600
github_runner_cluster_runtime_kill_grace: 5
# The seconds colima stop is given before colima stop --force follows it.
github_runner_cluster_runtime_clean_stop_timeout: 120
# Whether the role installs a launchd job that restarts a Colima profile whose container runtime stops answering between runs (files/colima-watchdog.sh, run every github_runner_cluster_runtime_watchdog_interval seconds, as a daemon for a profile started at boot and as an agent for any other). Only for a host with github_runner_cluster_runtime_autostart on. Off removes the job. A restart stops every container on the host until the VM is back, so it acts only on the evidence tasks/runtime_wedged.yml asks for and is bounded by a cap and a backoff (below).
github_runner_cluster_runtime_watchdog: true
# The seconds between the watchdog's runs, and how many consecutive runs must find docker info failing before it looks further. Together with github_runner_cluster_runtime_probe_timeout they set how long a runtime can stay silent before anything happens.
github_runner_cluster_runtime_watchdog_interval: 60
github_runner_cluster_runtime_watchdog_failures: 3
# The most restarts the watchdog makes in an hour, and the least seconds between two of them (doubled for every further restart in that hour, so a VM that wedges again soon after a restart waits longer). A restart that fails counts. The boot grace (github_runner_cluster_runtime_boot_grace) also keeps a VM that was just restarted from being restarted again until its host agent is older than that.
github_runner_cluster_runtime_watchdog_max_restarts: 3
github_runner_cluster_runtime_watchdog_min_gap: 600
# The seconds a restart's start may take before the watchdog ends it.
github_runner_cluster_runtime_watchdog_start_timeout: 900
# Derived, never set directly: the watchdog's launchd label (one per profile), and the directory under the connecting user's home that holds its script, the supervisor beside it and its state (the consecutive failures, the restart log).
github_runner_cluster_runtime_watchdog_label: "com.exadev.github-runner.colima-watchdog.{{ github_runner_cluster_effective_runtime.colima_profile }}"
github_runner_cluster_runtime_watchdog_dir: "{{ ansible_facts['env']['HOME'] }}/.local/share/exadev-github-runner/colima-watchdog-{{ github_runner_cluster_effective_runtime.colima_profile }}"
github_runner_cluster_runtime_watchdog_agent_plist: "{{ ansible_facts['env']['HOME'] }}/Library/LaunchAgents/{{ github_runner_cluster_runtime_watchdog_label }}.plist"
github_runner_cluster_runtime_watchdog_daemon_plist: "/Library/LaunchDaemons/{{ github_runner_cluster_runtime_watchdog_label }}.plist"
# The Colima profile that runs this collection's k3s node, which must come back with no one logged in: the one github_runner_cluster_runtime_autostart_at auto starts at boot. Any other profile on the host (a `default` profile a host keeps for other workloads, say) keeps the login agent.
github_runner_cluster_runtime_boot_profile: node
# Which container runtime a macOS host runs: auto takes it from the Docker context the role drives (colima or colima-<profile> is Colima, desktop-linux is Docker Desktop, anything else is other), colima, docker_desktop and other say so for a host whose context does not tell (DOCKER_HOST set, a context with another name).
Expand Down
Loading
Loading