Skip to content

ci: test host audio on Windows and macOS through virtual loopback devices - #666

Open
jamesbraid wants to merge 3 commits into
ci/audio-real-devicefrom
ci/audio-ci-phase2
Open

ci: test host audio on Windows and macOS through virtual loopback devices#666
jamesbraid wants to merge 3 commits into
ci/audio-real-devicefrom
ci/audio-ci-phase2

Conversation

@jamesbraid

@jamesbraid jamesbraid commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

The real-device audio tests (a card plays through cpal into the host's default output, and the test records it back) run in CI on Linux only. A break in cpal's WASAPI or CoreAudio path, such as the wrong device or rate, would only show up on a desk. This adds Windows and macOS: the runner is given a virtual sound device, the test records it in-process through cpal, and the same tests run.

Neither new step has run on a hosted runner. They are non-blocking (continue-on-error: true) until one run of each has passed. Linux stays blocking, as in the PR it builds on. Windows now gates on main, but the new Windows step has its own continue-on-error, so it does not.

What is in it

  1. audio: record the real-device tests' loopback through cpal on Windows and macOS. A second recorder behind the existing Backend::RealDevice, with IRIS_AUDIO_LOOPBACK=default:default-render on Windows (cpal builds a WASAPI loopback when an input stream is opened on a render device) and default:BlackHole 2ch on macOS (an ordinary input at 48 kHz). No new crate. cpal's macOS process-tap loopback is not used: it needs a system-audio-recording permission a runner cannot grant, and it delivers silence without it, which would look like the bug under test.
  2. scripts: set up a virtual loopback device on Windows and macOS runners. audio-loopback-windows.ps1 (VB-CABLE pack 43 and devcon, both pinned by SHA-256, then waits for the endpoint) and audio-loopback-macos.sh (Homebrew blackhole-2ch and switchaudio-osx, restart coreaudiod, set defaults).
  3. ci: run the real-device audio tests on Windows and macOS through virtual devices. Two steps in the Test job, plus TESTING.md.

What was run

  • Linux, on top of the audio PR's current head: cargo check --tests -p iris, default and --features jitv2, and cargo test -p iris --lib host_audio (8 passed; the real-device test skips with the variable unset). The diff touches only host_audio among source modules.
  • The new recorder compiles for x86_64-pc-windows-gnu, x86_64-pc-windows-msvc and aarch64-apple-darwin, checked in a stand-in crate holding only that module. The full crate does not cross-compile on the Linux host because its C dependencies need a target toolchain.
  • Not run: anything on Windows or macOS, the PowerShell script, the macOS script. A Mac was reachable, but BlackHole is not installed there and installing a driver on it was out of bounds, so the macOS recorder was not run either.

What the hosted runners offer

  • The runner images list no audio software: Windows2025-Readme, macos-26-arm64-Readme. GitHub will not add a virtual device: "We do not pre-install any virtual hardware since it can affect end-users" (runner-images#6983).
  • Both drivers can be installed at run time, by other projects' CI that passes today: Mozilla's cubeb installs VB-CABLE on windows-2025 and BlackHole on macos-15 and macos-15-intel, then runs its device tests (build.yml, setup_windows_audio.ps1). In run 39754653738 (2026-10-22) every install and test step on those platforms succeeded.
  • Unknown: whether the same works on macos-26 and windows-latest, which are the images iris uses, and, on macOS, whether the recording returns samples or silence under the runner's microphone permission (the image pre-grants that permission to its own provisioner binaries, configure-tccdb-macos.sh, not to arbitrary processes). cubeb does not wait for the VB-CABLE endpoint to appear; this script does, for up to 60 s.
  • The Windows recorder pads silence from the wall clock when a loopback buffer arrives late, because WASAPI delivers nothing while nothing renders and the tests look for the gap.

Licences

  • VB-CABLE is donationware. The vendor's page says it is free to use but a licence is expected for professional use, "any company, institution or organization where users are not expected to pay/donate", and that distributing or reselling needs a purchased licence. The script downloads the pack from the vendor on every run and commits nothing of it, so there is no redistribution. Whether an open-source project's CI counts as professional use is the vendor's reading, not ours. The project owner has decided to use it with no licence purchase.
  • BlackHole's source is GPL-3.0; its official binaries are copyright Existential Audio (LICENSE). CI runs it as a separate program, nothing is linked or shipped.

First run to watch

The first push-to-main Test job (macOS only runs on pushes; Windows also runs on ready pull requests). On Windows the step "Test audio through VB-CABLE", on macOS "Test audio through BlackHole". If only a_tone_survives_an_underrun_on_the_real_device (the bare host output, no card) fails, the setup is broken, not a card.

@jamesbraid
jamesbraid added this pull request to stack #667 November 12, 2026 07:17
@jamesbraid
jamesbraid marked this pull request as draft November 12, 2026 08:21
@jamesbraid
jamesbraid removed this pull request from stack #667 November 12, 2026 09:12
@jamesbraid
jamesbraid added this pull request to stack #675 November 12, 2026 09:13
@jamesbraid
jamesbraid marked this pull request as ready for review November 12, 2026 09:37
@jamesbraid
jamesbraid force-pushed the ci/audio-ci-phase2 branch 2 times, most recently from 63528ae to abcf2be Compare November 12, 2026 14:45
@jamesbraid
audio: record the real-device tests' loopback through cpal on Windows…
1a0420d 
… and macOS

The real-device audio tests could only record on Linux; elsewhere
IRIS_AUDIO_LOOPBACK made them panic, so cpal's WASAPI and CoreAudio
output paths had no test. Both platforms can loop the default output back
inside the test process.

Windows: `default-render` records the default output through WASAPI
loopback, which cpal enables for an input stream on a render device. The
mix format comes from default_output_config, because cpal's
default_input_config refuses render devices. A loopback delivers nothing
while nothing renders, so a late buffer has the silence before it filled
in from the wall clock; the tests look for the pause.

macOS: the source is an input device's name, such as `BlackHole 2ch`,
recorded at 48 kHz. cpal's process-tap loopback is not used: it needs a
system-audio-recording grant a runner cannot give.

Unproven: the recorder compiles for x86_64-pc-windows-gnu,
x86_64-pc-windows-msvc and aarch64-apple-darwin in a stand-in crate
holding only this module, and has not recorded on either platform.
@jamesbraid
scripts: set up a virtual loopback device on Windows and macOS runners
8840409 
Hosted Windows and macOS runners have no sound device, so the real-device
audio tests have nothing to play into, and GitHub will not preinstall one
(actions/runner-images#6983). Mozilla's cubeb installs VB-CABLE on
windows-2025 and BlackHole on macos-15 in CI and its run 39754653738
passed with both.

audio-loopback-windows.ps1 downloads VB-CABLE pack 43 and devcon, checks
both against pinned SHA-256 sums, trusts the driver's publisher, installs
the cable and waits up to 60 s for its render endpoint. Windows names the
endpoints after the driver model, not the cable's ends: a run on
windows-latest listed "Speakers (VB-Audio Virtual Cable)" and "CABLE Output
(VB-Audio Virtual Cable)". The script therefore picks the render side by
its instance ID and the cable's name, prints every endpoint it found, and
fails if another render endpoint exists, since the tests play into the
default output and the cable is the default only while it is the sole one.
audio-loopback-macos.sh installs BlackHole 2ch from Homebrew, restarts
coreaudiod and makes it the default output and input. VB-CABLE is
donationware and is downloaded from the vendor on every run, never kept in
the repository.

A setup failure fails the script loudly, but the workflow step is
continue-on-error, so the job stays green and the real-device tests do
not run. Unproven: the macOS script has not run on a runner (it runs only
on pushes to main); on Windows the install is shown to work on
windows-latest, but the endpoint selection and the recording after it
have not run.
@jamesbraid
ci: run the real-device audio tests on Windows and macOS through virt…
465ac0f 
…ual devices

The real-device audio tests ran in CI on Linux only, so a break in cpal's
WASAPI or CoreAudio path would first show on someone's desk. The Test job
now installs VB-CABLE on Windows and BlackHole on macOS with the new
scripts and runs the tests recorded back through them.

Both steps have continue-on-error, so a failure does not turn the job
red even where the platform gates. Neither install has run on these
images; they become blocking once one run of each has passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jamesbraid