Repository navigation
Select configs among the configurations of the current tasks, and give the taskgraph their labels - #7598
Draft
marco-c wants to merge 20 commits into
Draft
Select configs among the configurations of the current tasks, and give the taskgraph their labels#7598marco-c wants to merge 20 commits into
marco-c wants to merge 20 commits into
Conversation
marco-c
force-pushed
the
config-selection-task-configs
branch
4 times, most recently
from
November 19, 2026 23:53
3d6a2d2 to
74129e1
Compare
… as the models name them The config selection models are trained with the configurations of the push data, as mozci names them from the task definitions (renamed, e.g. shippable configurations are the ones of their non-shippable equivalents). The candidates were instead obtained by matching the labels of the current tasks with the configs seen in training, which isn't the same: - configs of tasks whose label doesn't have their Treeherder platform (e.g. the Windows ASan ones) were unknown to the models; - suites and variants named alike were confused (e.g. mochitest-webgpu and the webgpu variant). The HTTP service now retrieves the test tasks from the task graph of the latest backstop push (which runs all of them, with all their manifests), with their configs computed with mozci's get_configuration, like the ones of the push data. Configs are selected among them.
E.g. the standalone tasks only run on backstop pushes, whatever is selected.
They are too costly to run: the config selection models are trained with the configs of their non-shippable equivalents, which are selected instead.
So that a stalled request fails (and is retried) instead of blocking the boot or the refresh forever.
…available or has none The decision task of the latest backstop push may not have published its graph yet (or ever, if it failed), and its graph may have none of the tasks among which configs are selected.
The taskgraph matched the selected configs with the labels of its tasks, so it couldn't run the configs of tasks whose label doesn't have their Treeherder platform (e.g. the Windows ASan ones), and it also ran the variants of a selected config (e.g. test-linux2404-64/opt-* matching the swr tasks). The replies now give it the labels (without chunk number) of the tasks of the selected configs, and the selected configs themselves in "config_group_configs".
…ection models They were only used to name the tasks of the current task graph, which are now named after their configurations.
… parsing all of it A decision task's full-task-graph.json is hundreds of MB (and much more once parsed), so only the definitions of the given tasks are parsed, where they are in the text. If one isn't found this way (e.g. if the format of the file changed), the whole graph is parsed.
… their configs Tasks can be renamed after the task graph the config selection tasks come from. The tasks whose manifests aren't known, with their configs, are given for all the groups selected on their configs, as the taskgraph only runs a task for a group if the task runs it.
As the taskgraph matches the labels, they must be the ones of the push's tasks: the test tasks of the pushes after the backstop push which weren't in its task graph (e.g. renamed since then) are stored too. Those are found among the target tasks of the pushes (before optimization, which could remove them as they match none of the labels in the replies), with their definitions from the full task graph, which is only retrieved when there are unknown test tasks: the test tasks of the backstop push which aren't eligible are remembered.
… graph All the tasks that test selection optimizes run on backstop pushes, so the target tasks which aren't in the task graph are optimized otherwise (e.g. jittest when no JS engine file changed), and the other pushes targeting them don't need their full task graph to know it.
…op push Some tasks are only targeted by some pushes (e.g. test-verify when tests are modified), so the newer backstop push doesn't necessarily target them: the ones found before are kept, as long as the recent pushes target them.
The tasks of the pushes after the backstop push (e.g. renamed since then) are only useful if they are found soon after they appear, so the test tasks among which configs are selected are updated every 10 minutes in the refresh process, rather than once per day.
The updates now start from the previous one: the tasks of the backstop push are kept until there's a newer backstop push, and the pushes after it which were already processed aren't processed again.
The updates look for the latest backstop push every 10 minutes: the pushes after it which aren't backstop pushes are remembered, so that their parameters aren't retrieved again.
Most updates don't change anything (no new backstop push, no new tasks).
…t variants of their push mozci memoizes the test variants of mozilla-central for the lifetime of the process (the refresh runs for weeks), and those don't have the variants added on autoland until they are merged, so the configurations of their tasks couldn't be computed. The HTTP service gives mozci the test variants of each push it reads tasks from (from the push's taskcluster/test_configs/variants.yml), or makes it retrieve the current ones of mozilla-central again if they can't be retrieved.
… which failed Tasks whose configuration can't be computed (e.g. if retrieving the push's test variants failed) were ignored, or skipped and left behind once their push was processed. They are now kept pending, with the parts of their definitions needed to try again, and resolved by a later update: the ones of the backstop push become candidates (with their manifests), the ones of the later pushes new tasks.
…nfig selection tasks The tasks whose configuration couldn't be computed are retried from the definitions kept for them, so the other pushes targeting them don't need their full task graph.
marco-c
force-pushed
the
config-selection-task-configs
branch
from
November 20, 2026 00:11
74129e1 to
0a2bcd4
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The config selection models are trained with the configurations of the push data, as mozci names them from the task definitions (renamed by
rename_task, e.g. shippable configurations are the ones of their non-shippable equivalents). But the candidates were obtained by matching the labels of the tasks in a decision task'smanifests-by-task.jsonwith the configs seen in training, and the taskgraph matches the selected configs with the labels of its tasks. Those aren't the same:test-windows11-64-25h2-asan/opt-…vstest-windows11-64-25h2/asan-*) and of shippable tasks were unknown to the models, and the taskgraph couldn't run the Windows ASan ones when selected;mochitest-webgpuand thewebgpuvariant), and the taskgraph also ran all the variants of a selected config (test-linux2404-64/opt-*matches the-swr,-standalone… tasks);standaloneandrepeat, which only run on backstop pushes) were candidates.On a backstop push, about 8–12% of the target test tasks got the wrong config this way, and 36–64 candidate configs were unknown to the model. With this change, the configs of all the current tasks are known to the model, except those of brand-new variants.
What changes
get_configuration, like the push data. Configs are selected among them.config_groupsin the replies now contains the exact labels (without chunk number) of the tasks of the selected configs, andconfig_group_configscontains the selected configs. The taskgraph matches labels withfnmatch, so exact labels work without any Gecko change. Selectingtest-linux2404-64/opt-*no longer runs its variants, so the config selection strategies schedule fewer tasks.variants.yml, since mozci memoizes mozilla-central's, which lacks variants added on autoland until they're merged. Tasks whose configuration can't be computed are retried later rather than ignored.get_configurationreads the test variants fromtest-setting. Addspyyamlto the HTTP service's dependencies.Checks
Known limitation
A push that renames its own tasks (and pushes racing with it) can still lose the groups that only ran on the old labels, until its decision task finishes and the next update picks up the new labels. Fixing that needs the taskgraph to leave a group unrestricted when none of its labels match, which needs the global view discussed in bug 1927227.
Reading order
The commits are meant to be reviewed one by one. For the larger ones:
_eligible_tasksandget_config_selection_tasksinhttp_service/bugbug_http/boot.py(which tasks are candidates, and how their configs are computed), thenupdate_config_selection_tasks(which backstop push they come from), then the index inbugbug/models/testconfigselect.py(LiveTask,live_tasks,live_candidates) and its use inhttp_service/bugbug_http/models.py.update_config_selection_tasks(the scan of the later pushes), then_classify_new_tasks, then the ignore list in the first half.update_config_selection_tasks, then the processed pushes in the scan of the later pushes._eligible_tasks'failedargument, then_resolve_unresolvedand whereupdate_config_selection_taskscalls it.