Skip to content

Select configs among the configurations of the current tasks, and give the taskgraph their labels - #7598

Draft
marco-c wants to merge 20 commits into
mozilla:masterfrom
marco-c:config-selection-task-configs
Draft

marco-c wants to merge 20 commits into
mozilla:masterfrom
marco-c:config-selection-task-configs

Conversation

@marco-c

@marco-c marco-c commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

The config selection models are trained with the configurations of the push data, as mozci names them from the task definitions (renamed by rename_task, e.g. shippable configurations are the ones of their non-shippable equivalents). But the candidates were obtained by matching the labels of the tasks in a decision task's manifests-by-task.json with the configs seen in training, and the taskgraph matches the selected configs with the labels of its tasks. Those aren't the same:

  • the configs of tasks whose label doesn't have their Treeherder platform (e.g. Windows ASan: test-windows11-64-25h2-asan/opt-… vs test-windows11-64-25h2/asan-*) and of shippable tasks were unknown to the models, and the taskgraph couldn't run the Windows ASan ones when selected;
  • suites and variants named alike were confused (e.g. mochitest-webgpu and the webgpu variant), and the taskgraph also ran all the variants of a selected config (test-linux2404-64/opt-* matches the -swr, -standalone… tasks);
  • tasks which test selection doesn't optimize (e.g. standalone and repeat, which only run on backstop pushes) were candidates.

On a backstop push, about 8–12% of the target test tasks got the wrong config this way, and 36–64 candidate configs were unknown to the model. With this change, the configs of all the current tasks are known to the model, except those of brand-new variants.

What changes

  • The HTTP service retrieves the test tasks from the task graph of the latest backstop push (which runs all of them, with all their manifests), and computes their configurations with mozci's get_configuration, like the push data. Configs are selected among them.
  • config_groups in the replies now contains the exact labels (without chunk number) of the tasks of the selected configs, and config_group_configs contains the selected configs. The taskgraph matches labels with fnmatch, so exact labels work without any Gecko change. Selecting test-linux2404-64/opt-* no longer runs its variants, so the config selection strategies schedule fewer tasks.
  • Shippable tasks are never selected (they're too costly); their non-shippable equivalents are.
  • Because the labels must be the ones of the push's tasks, the tasks are updated every 10 minutes in the refresh process: from a new backstop push as soon as there is one, and with the tasks of the pushes after it that aren't known yet (e.g. renamed). Those are found among the pushes' target tasks (before optimization), with their definitions from the full task graph, which is only fetched (and only partly parsed) when there are unknown test tasks.
  • The configurations are computed with each push's own variants.yml, since mozci memoizes mozilla-central's, which lacks variants added on autoland until they're merged. Tasks whose configuration can't be computed are retried later rather than ignored.
  • Requires mozci 2.6.1 (Bump mozci to 2.6.1 #7599), whose get_configuration reads the test variants from test-setting. Adds pyyaml to the HTTP service's dependencies.

Checks

  • On live autoland data: boot retrieval takes about 40 s and peaks at about 700 MB. A 10-minute update with no new pushes makes no requests. A full-graph fetch (rare) takes about 1 minute and peaks at about 1 GB.
  • Offline evaluation (risk-weighted harness, production setting): training with the configs test selection can't control, while selecting only among the others, is better than training without them (−1.7 points without them, 95% CI −3.2 to −0.2, in each of three windows), so training still uses them. Keeping shippable configs separate instead of renamed made no measurable difference, so the rename is kept.
  • On backstop pushes (complete labels), the model catches 78.8 / 86.0 / 89.3% of failing pushes at 60 / 100 / 150 config/groups per push, against 71.4 / 80.2 / 84.8% for pooled statistics and 62.7 / 77.4 / 80.4% for a design with nothing learned.

Known limitation

A push that renames its own tasks (and pushes racing with it) can still lose the groups that only ran on the old labels, until its decision task finishes and the next update picks up the new labels. Fixing that needs the taskgraph to leave a group unrestricted when none of its labels match, which needs the global view discussed in bug 1927227.

Reading order

The commits are meant to be reviewed one by one. For the larger ones:

  • Select configs among the configs of the latest backstop push's tasks: start with _eligible_tasks and get_config_selection_tasks in http_service/bugbug_http/boot.py (which tasks are candidates, and how their configs are computed), then update_config_selection_tasks (which backstop push they come from), then the index in bugbug/models/testconfigselect.py (LiveTask, live_tasks, live_candidates) and its use in http_service/bugbug_http/models.py.
  • Select configs on the tasks of the pushes after the backstop push too: start with the second half of update_config_selection_tasks (the scan of the later pushes), then _classify_new_tasks, then the ignore list in the first half.
  • Keep the config selection tasks between updates: the known backstop push at the start of update_config_selection_tasks, then the processed pushes in the scan of the later pushes.
  • Try again to compute the configurations of the config selection tasks which failed: _eligible_tasks' failed argument, then _resolve_unresolved and where update_config_selection_tasks calls it.

@marco-c
marco-c force-pushed the config-selection-task-configs branch 4 times, most recently from 3d6a2d2 to 74129e1 Compare November 19, 2026 23:53
… as the models name them

The config selection models are trained with the configurations of the push data, as mozci names
them from the task definitions (renamed, e.g. shippable configurations are the ones of their
non-shippable equivalents). The candidates were instead obtained by matching the labels of the
current tasks with the configs seen in training, which isn't the same:
- configs of tasks whose label doesn't have their Treeherder platform (e.g. the Windows ASan ones)
  were unknown to the models;
- suites and variants named alike were confused (e.g. mochitest-webgpu and the webgpu variant).

The HTTP service now retrieves the test tasks from the task graph of the latest backstop push
(which runs all of them, with all their manifests), with their configs computed with mozci's
get_configuration, like the ones of the push data. Configs are selected among them.
E.g. the standalone tasks only run on backstop pushes, whatever is selected.
They are too costly to run: the config selection models are trained with the configs of their
non-shippable equivalents, which are selected instead.
So that a stalled request fails (and is retried) instead of blocking the boot or the refresh
forever.
…available or has none

The decision task of the latest backstop push may not have published its graph yet (or ever, if it
failed), and its graph may have none of the tasks among which configs are selected.
The taskgraph matched the selected configs with the labels of its tasks, so it couldn't run the
configs of tasks whose label doesn't have their Treeherder platform (e.g. the Windows ASan ones),
and it also ran the variants of a selected config (e.g. test-linux2404-64/opt-* matching the swr
tasks). The replies now give it the labels (without chunk number) of the tasks of the selected
configs, and the selected configs themselves in "config_group_configs".
…ection models

They were only used to name the tasks of the current task graph, which are now named after their
configurations.
… parsing all of it

A decision task's full-task-graph.json is hundreds of MB (and much more once parsed), so only the
definitions of the given tasks are parsed, where they are in the text. If one isn't found this way
(e.g. if the format of the file changed), the whole graph is parsed.
… their configs

Tasks can be renamed after the task graph the config selection tasks come from. The tasks whose
manifests aren't known, with their configs, are given for all the groups selected on their configs,
as the taskgraph only runs a task for a group if the task runs it.
As the taskgraph matches the labels, they must be the ones of the push's tasks: the test tasks of
the pushes after the backstop push which weren't in its task graph (e.g. renamed since then) are
stored too. Those are found among the target tasks of the pushes (before optimization, which could
remove them as they match none of the labels in the replies), with their definitions from the full
task graph, which is only retrieved when there are unknown test tasks: the test tasks of the
backstop push which aren't eligible are remembered.
… graph

All the tasks that test selection optimizes run on backstop pushes, so the target tasks which
aren't in the task graph are optimized otherwise (e.g. jittest when no JS engine file changed), and
the other pushes targeting them don't need their full task graph to know it.
…op push

Some tasks are only targeted by some pushes (e.g. test-verify when tests are modified), so the
newer backstop push doesn't necessarily target them: the ones found before are kept, as long as the
recent pushes target them.
The tasks of the pushes after the backstop push (e.g. renamed since then) are only useful if they
are found soon after they appear, so the test tasks among which configs are selected are updated
every 10 minutes in the refresh process, rather than once per day.
The updates now start from the previous one: the tasks of the backstop push are kept until there's
a newer backstop push, and the pushes after it which were already processed aren't processed again.
The updates look for the latest backstop push every 10 minutes: the pushes after it which aren't
backstop pushes are remembered, so that their parameters aren't retrieved again.
Most updates don't change anything (no new backstop push, no new tasks).
…t variants of their push

mozci memoizes the test variants of mozilla-central for the lifetime of the process (the refresh runs
for weeks), and those don't have the variants added on autoland until they are merged, so the
configurations of their tasks couldn't be computed. The HTTP service gives mozci the test variants
of each push it reads tasks from (from the push's taskcluster/test_configs/variants.yml), or makes it
retrieve the current ones of mozilla-central again if they can't be retrieved.
… which failed

Tasks whose configuration can't be computed (e.g. if retrieving the push's test variants failed)
were ignored, or skipped and left behind once their push was processed. They are now kept pending,
with the parts of their definitions needed to try again, and resolved by a later update: the ones
of the backstop push become candidates (with their manifests), the ones of the later pushes new
tasks.
…nfig selection tasks

The tasks whose configuration couldn't be computed are retried from the definitions kept for them,
so the other pushes targeting them don't need their full task graph.
@marco-c
marco-c force-pushed the config-selection-task-configs branch from 74129e1 to 0a2bcd4 Compare November 20, 2026 00:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant