Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .env.bootstrap.example
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
# Arbitrary shared secret for node-join auth inside the k3s cluster, unrelated to GitHub (e.g. `openssl rand -hex 32`). The same value on every server/agent host.
K3S_TOKEN=

# A reusable, pre-authorized Tailscale auth key: k3s's own --vpn-auth integration runs Tailscale inside each node's k3s container with it. See the README's Tailscale section for how it's provisioned.
# A reusable, pre-authorized Tailscale auth key: k3s's own --vpn-auth integration runs Tailscale inside each node's k3s container with it. See the Tailscale section of docs/architecture.md for how it's provisioned.
K3S_VPN_AUTH_JOIN_KEY=

# Optional: a Tailscale API token. When set, the role also ensures the tailnet's ACL grants cross-node pod traffic (see roles/github_runner_cluster/tasks/mesh/tailscale_api.yml). Without it the ACL is left as it is.
TAILSCALE_API_TOKEN=

# Every server's own Tailscale hostname, space-separated, including this one's own. Leave empty on a single-server setup, which then uses this host's node name. A multi-server cluster needs every server's cert to cover every server's name; see the README's Architecture section.
# Every server's own Tailscale hostname, space-separated, including this one's own. Leave empty on a single-server setup, which then uses this host's node name. A multi-server cluster needs every server's cert to cover every server's name; see the Architecture section of docs/architecture.md.
K3S_TLS_SAN_LIST=

# This host's k3s node name, which is also its Tailscale device hostname. Leave empty to use this machine's own short hostname.
Expand Down Expand Up @@ -60,7 +60,7 @@ GHCR_PULL_TOKEN=
HEARTBEAT_GH_TOKEN=
HEARTBEAT_GIST_ID=

# The usage-driven autoscaler (see scripts/autoscaler.sh and the README's Autoscaler section) manages the scale set only when AUTOSCALER_USABLE_BUDGET_GI is set, and then also needs AUTOSCALER_FLOOR: the memory runner pods may use across every node in GiB, and the maxRunners in the scale set's values, which every Helm upgrade reverts to. They depend on the machines, so they have no defaults. It works out the highest maxRunners it may set from the nodes on every poll; AUTOSCALER_MAX_CEILING, optional, bounds that. The rest default to the role's own values (roles/github_runner_arc/defaults/main.yml).
# The usage-driven autoscaler (see scripts/autoscaler.sh and docs/autoscaler.md) manages the scale set only when AUTOSCALER_USABLE_BUDGET_GI is set, and then also needs AUTOSCALER_FLOOR: the memory runner pods may use across every node in GiB, and the maxRunners in the scale set's values, which every Helm upgrade reverts to. They depend on the machines, so they have no defaults. It works out the highest maxRunners it may set from the nodes on every poll; AUTOSCALER_MAX_CEILING, optional, bounds that. The rest default to the role's own values (roles/github_runner_arc/defaults/main.yml).
# AUTOSCALER_USABLE_BUDGET_GI=
# AUTOSCALER_MAX_CEILING=
# AUTOSCALER_FLOOR=
Expand Down
2 changes: 1 addition & 1 deletion .github/actionlint.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
self-hosted-runner:
labels:
# The ARC runner scale set label the fleet smoke tests (test-arc-runner.yml, test-autoscaler.yml) target - see the project README's Architecture section.
# The ARC runner scale set label the fleet smoke tests (test-arc-runner.yml, test-autoscaler.yml) target - see docs/architecture.md.
- exadev-runners

# Configuration variables in array of strings defined in your repository or
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: CI

# Lints every shell script, workflow file, YAML file, and Dockerfile in the repo. There is no application test suite here (this repo is fleet infrastructure config, not application code) - these checks are its actual verification surface, matching the README's Build/test/lint section.
# Lints every shell script, workflow file, YAML file, and Dockerfile in the repo. There is no application test suite here (this repo is fleet infrastructure config, not application code) - these checks are its actual verification surface, matching docs/build-and-test.md.
#
# Runs on GitHub-hosted runners, which are free for a public repository, so the checks of the fleet's own configuration don't depend on the fleet being up. The fleet's own behaviour is tested by test-arc-runner.yml and test-autoscaler.yml, which target it directly.
#
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-autoscaler.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: test-autoscaler

# Manual traffic generator for the usage-driven autoscaler: dispatches several jobs onto exadev-runners at once so there is real, concurrent memory usage to watch the autoscaler's dry-run output/status react to (see the project README's Autoscaler section). This workflow does not assert anything about the autoscaler itself - it only creates the load; watch `kubectl logs -n github-runner-platform deploy/autoscaler -f` and `kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.status\.json}'` while this runs.
# Manual traffic generator for the usage-driven autoscaler: dispatches several jobs onto exadev-runners at once so there is real, concurrent memory usage to watch the autoscaler's dry-run output/status react to (see docs/autoscaler.md). This workflow does not assert anything about the autoscaler itself - it only creates the load; watch `kubectl logs -n github-runner-platform deploy/autoscaler -f` and `kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.status\.json}'` while this runs.
on:
workflow_dispatch:
inputs:
Expand Down
254 changes: 33 additions & 221 deletions README.md

Large diffs are not rendered by default.

42 changes: 42 additions & 0 deletions docs/ansible.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Ansible collection layout, roles and host prerequisites

## Ansible

The repository is laid out as an Ansible collection, `exadev.github_runner` (see `galaxy.yml`): its roles live in `roles/` and its playbooks in `playbooks/`: `site.yml` brings up the whole fleet, `arc.yml` installs only ARC into a cluster that already exists (from the control machine, with a kubeconfig), and `github_app_setup.yml` creates the runners' GitHub App through GitHub's manifest flow, and `recover_in_cluster_mesh.yml` recovers the in-cluster Headscale provider's bootstrap server. A fleet keeps its inventory, `host_vars` and Helm values in a repository of its own, installs the collection there, and runs the playbooks by name (`exadev.github_runner.site`); `examples/` has inventories to start from. `ansible/` holds a generic `ansible.cfg` and the collections the roles require, for running the playbooks from a checkout of this repository as `bootstrap.sh` and CI do; `playbooks/collections/` links back to the checkout, so a playbook run from it finds `exadev.github_runner` in place without installing it. Verified end to end against real infrastructure (three hosts, a real single-node-to-3-node etcd HA cutover, real jobs scheduling across all three). `bootstrap.sh` (see [Getting started](../README.md#getting-started)) is a thin wrapper that runs this same playbook locally.

```bash
# in the fleet's own repository, next to its inventory
ansible-galaxy collection install git+https://github.com/ExaDev/github-runner.git
ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1 --check # dry run
ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1 # real run
```

The playbook runs these roles, each with its own README: `github_runner_cluster` brings the host up as a k3s node, in a container under Docker Compose or, with `github_runner_cluster_node_mode: native`, directly under systemd on a Linux host, `github_runner_arc` installs ARC and the fleet-health platform, `github_runner_k8s_client` (included by the other two) gives a host the kubernetes Python client, and `github_runner_secrets_onepassword` reads the secrets from 1Password when `github_runner_secrets_source` is `onepassword`. The cluster and ARC roles read every secret (`K3S_TOKEN`, the mesh keys, GHCR credentials, the heartbeat PAT, each org's App private key) as a plain variable, so any source Ansible reads works: ansible-vault, SOPS, an env or file lookup, HashiCorp Vault or AWS SSM (see `examples/ansible-vault/`, `examples/env/` and `examples/onepassword/`). With `github_runner_secrets_source: onepassword`, every secret is read live from one 1Password item at task time; `roles/github_runner_secrets_onepassword/README.md` lists the item's fields and the `op://` references. `bootstrap.sh` sets the same variables directly from `.env.bootstrap`. The cluster role ships the node's `docker-compose.yml` and k3s image build context in `roles/github_runner_cluster/files/`, copies them on every run into each host's `github_runner_cluster_dir` (`~/.github-runner` by default), and templates the `.env` for their `k3s`/`k3s-agent` services there, so a host needs no checkout of this repository. The Compose project name is pinned to `github-runner` (`github_runner_cluster_compose_project`) rather than taken from the directory's name, because the named volumes holding the k3s datastore and each node's mesh identity are prefixed with it; the directory can move without losing them.

Which hosts are k3s servers is worked out from the inventory, not set per host. The `github_runner_cluster` inventory group lists the cluster's hosts in order: the first N become servers (control-plane/etcd members running `k3s server`, genuine voting members of the raft quorum), where N is the largest odd number no greater than both the number of hosts and `github_runner_cluster_max_servers` (5 by default), so 1 or 2 hosts give 1 server, 3 or 4 give 3, and 5 or more give 5. The first server bootstraps the cluster (`--cluster-init`, embedded etcd) and every other host joins it, the rest as agents (`k3s agent`, schedulable capacity with no control-plane role). Join URLs and every server's `--tls-san` list are derived from each server's node name and `github_runner_cluster_tailnet_domain`. A host can still set `github_runner_cluster_node_role`, `github_runner_cluster_bootstrap`, `github_runner_cluster_server_url` or `github_runner_cluster_tls_sans` to override its part, and the run fails with an explanation if the result has an even number of servers, none at all, or more than one bootstrap host; `roles/github_runner_cluster/defaults/main.yml` describes each. Installing the ARC controller, each org's runner scale set, and the fleet-health platform is not tied to a node role at all: each is driven purely by whether the installing host's own `github_runner_arc_orgs`/`github_runner_arc_heartbeat_gist_id` is set, so any server host can be the one Ansible installs against, and moving which host does the installing is a config change (which host's `host_vars` sets it), not a role change. See the [Architecture](architecture.md#architecture) section for why 3 servers is the minimum sane count for genuine HA (2 is worse than 1 for quorum math, not better). To add a server or agent, add the host to the `github_runner_cluster` group in the fleet's inventory and give it its own `host_vars/<host>.yml`; keep the bootstrap host first.

A `github_runner_arc_orgs` entry registers its scale sets in an organisation, or with `repository` in one repository of a user or an organisation (`githubConfigUrl` `https://github.com/<name>/<repository>`); a repository's scale sets take only that repository's jobs, so each repository needs its own entry (see Repository scope in `roles/github_runner_arc/README.md`). `github_runner_arc_orgs[].scale_set_profiles` is a list of `{suffix, values_file?, node_selector?, runs_on_label?, capabilities?, autoscale?, sizing?, burst?}` entries (see `roles/github_runner_arc/README.md` for every key) - one Helm release and one namespace per entry, both removed by the next run once the entry leaves the inventory (see Profiles that leave the inventory in `roles/github_runner_arc/README.md`), pooled under the org's own shared `runs-on` label via `scaleSetLabels` (not `runnerScaleSetName`, which is a scale set's own unique GitHub-side registration identity, not just a label - two releases sharing it collide outright) unless a profile sets its own `runs_on_label` to deliberately opt out of that pool - e.g. a dedicated image-building profile ordinary CI jobs have no business landing on. `node_selector` is optional per profile, not mandatory: omitting it leaves that profile's pods genuinely unpinned, letting the Kubernetes scheduler place them on whichever node in the fleet actually has room - real failover if one machine goes down, not just capacity pooling. Values files are rendered on the control node from `github_runner_arc_values_dir`, a directory in the fleet's own repository, so no fleet host needs them. ExaDev's own fleet uses two profiles: an ordinary CI profile unpinned across the whole fleet (sized with Guaranteed QoS for the smallest machine's real capacity), and a second, separately-labelled, Docker-in-Docker profile pinned to one host for building the runner image - a deliberate isolation choice, not a capacity constraint.

### Prerequisites

The collection brings up the k3s nodes and what runs on them; it does not set up the hosts underneath. Each host must already have:

- Python 3, or `github_runner_standalone_python_install: true` for a host that has none (see [Unraid hosts](#unraid-hosts)).
- A connecting user that is not root and can become root without a password, or a play run with `--ask-become-pass`: the tasks of the cluster role that change the host, and those of the macOS Actions runner role, use `become`.
- Docker with the Compose plugin for a node under Docker Compose (see the Prerequisites under [Getting started](../README.md#getting-started)). On macOS that includes Homebrew: the playbook installs `helm`, `kubectl`, Docker and Compose through it, and the macOS Actions runner role's `PATH` names both of its prefixes.
- A host that stays awake while it carries a node. A Mac needs its AC power settings set so that it never sleeps (and, with the lid closed, `pmset disablesleep`), and automatic restart after a power failure if it should come back by itself. A Linux laptop needs its suspend targets masked and its lid switch ignored, and on Wi-Fi its radio's power save turned off. The cluster role sets none of these.
- For a native node, nothing more where `github_runner_cluster_native_tailscale_install` is `true` (the default), since the role installs the pinned Tailscale package. With `false`, a Tailscale client already installed and logged in to the cluster's tailnet. A node under Docker Compose has its own Tailscale device inside the container and needs no client on the host.

`github_runner_cluster_k3s_prerequisites` and `github_runner_cluster_containerd_apparmor_stacking_fix` are the two host steps the cluster role does offer, because they exist only for k3s (see Host preparation in `roles/github_runner_cluster/README.md`).

### Unraid hosts

An Unraid host runs the same cluster role as any other Docker host (it ships Docker and the Compose plugin), with three differences. It has no Python, so `github_runner_standalone_python_install: true` in its `host_vars` makes `site.yml` install a pinned one first (`roles/github_runner_standalone_python/README.md`), at `github_runner_standalone_python_install_dir`, with `ansible_python_interpreter` pointing at it. Its root filesystem and home directory live in RAM, so `github_runner_cluster_dir` and that install directory must be on persistent storage (under `/mnt/user/appdata`), never the default `~/.github-runner`. And Docker only runs once the array is started, so the node comes back after a reboot only when the array does. A host that is a k3s server also needs a k3s data volume on storage with fast synchronous writes, because etcd waits for the disk on every write: Docker's data directory on a spinning array disk is not enough, and a server there makes the whole control plane slow. Such a host belongs in the inventory after the servers, as an agent, until Docker's data directory sits on an SSD pool.

### Capabilities

A profile can also add tools, libraries and setup to its runner pods when they start, rather than baking them into a runner image someone has to build, publish and patch: `capabilities` names entries of `github_runner_arc_capabilities`. A tool-cache capability is a public, pinned image holding a tool in the Actions tool-cache layout; an init container copies it into the runner's `RUNNER_TOOL_CACHE`, where `actions/setup-node`, `setup-tflint` and other setup actions that consult the cache find it and skip their download, and it can also be put on `PATH` for a tool no setup action looks up. A system-library capability is a public, pinned image holding a relocatable install prefix (shared libraries, headers, pkg-config files, programs) for what cannot live in the tool cache, such as an MPI implementation, a solver library or a database client library; an init container copies it into a sysroot (`/opt/sysroot` by default), and the job-started hook puts the sysroot on `PATH`, `LD_LIBRARY_PATH`, `LIBRARY_PATH`, `CPATH`, `PKG_CONFIG_PATH` and `CMAKE_PREFIX_PATH` and sets any relocation variable the library needs. A hook capability is a script run before each job's first step, through a job-started hook mounted from a ConfigMap. A node-cache capability is a directory of the node shared by the profile's runner pods there, for a package manager's store, which a pruner keeps under a size limit. A profile with its own `runs_on_label` then works as a named runner, for example one with Terraform, TFLint and the AWS CLI, or one with Open MPI and a solver library, that workflows select with `runs-on`, all from one values file and the stock `actions-runner` image. `tool-images/` has Dockerfiles for reference tool images (Node.js, Terraform, TFLint, the AWS CLI) reference system-library images (Open MPI, HiGHS, GLPK, libpq, MariaDB Connector/C) and a reference build runner image (the stock runner image with a C toolchain, for jobs that compile against those libraries), which `.github/workflows/publish-tool-images.yml` publishes when started by hand or by a `tool-images-*` tag; `roles/github_runner_arc/README.md`'s Capabilities section has the tool-image and payload contracts, how each reference library is declared and proven to work from a relocated prefix, how to build an image of your own, and where capabilities apply.

### Registry cache

Each Docker-in-Docker runner pod starts with an empty Docker daemon, so every job pulls its base images, service containers and container actions from the registries again. `github_runner_arc_registry_cache_enabled: true` installs a pull-through cache (CNCF Distribution in proxy mode, one container per registry behind one Service, on persistent or ephemeral storage) and points every dind profile's daemon at it through a `hosts.toml` mirror per registry, and the `docker-container` builders `docker/setup-buildx-action` creates at it through buildx's default `buildkitd.default.toml`, so nothing in a workflow changes. Registries are a list (Docker Hub, ghcr.io and quay.io by default), each optionally pulling with a read-only credential from a Secret, and pulls fall back to the registry whenever the cache is down or failing. It is off by default, and off renders exactly what the role rendered without it. `roles/github_runner_arc/README.md`'s Registry cache section says which pull paths it covers, which bypass it, and what a credential on it exposes.
Loading
Loading