Repository navigation
FEAT: Save converters and scorers created through the API - #4051
Open
varunj-msft wants to merge 160 commits into
Open
varunj-msft wants to merge 160 commits into
varunj-msft wants to merge 160 commits into
Conversation
…API input External construction now rejects a parameter whose type includes Path unless the calling service passes the name in allowed_path_params after making the value safe. The converter service allows its upload parameters once they are stored, and the target service allows the server-set upload folder and the GitHubCopilotTarget working directory. The scorer-only check is removed, and the "Valid parameters" list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a target that needs a disallowed path is left out instead of listed without the field, matching the scorer catalog. allowed_path_params is typed frozenset[str] so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts: # tests/unit/backend/test_converter_service.py
…benchmark's recipes folder
# Conflicts: # pyrit/backend/services/runtime_lifecycle.py # tests/unit/backend/test_runtime_lifecycle.py
# Conflicts: # pyrit/setup/initializers/tools.py # tests/unit/backend/test_scenario_run_service.py # tests/unit/backend/test_scenario_suite_execution_service.py # tests/unit/registry/test_attack_technique_registry.py # tests/unit/registry/test_instance_registry.py # tests/unit/registry/test_registry_recipe_restore.py
create_named_instance checks the name before it builds, and the technique registry checks names case-insensitively before it registers. Both raised the duplicate-name error without recording the clash, so a launch script that caught either one could still leave a saved entry in place of its own. The name check now records the clash itself, as register already did, and the technique check records it too.
# Conflicts: # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/models/targets.py # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_mappers.py # tests/unit/backend/test_runtime_lifecycle.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_instance_registry.py
# Conflicts: # frontend/src/components/Config/CreateTargetDialog.tsx # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_registry_recipe_rules.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/routes/targets.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_mappers.py # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # tests/unit/registry/test_registry_recipe_rules.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/models/scorers.py # pyrit/backend/routes/converters.py # pyrit/backend/routes/targets.py # pyrit/backend/services/converter_service.py # pyrit/backend/services/scorer_service.py # pyrit/backend/services/target_service.py # pyrit/registry/instance_registry.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_converter_service.py # tests/unit/backend/test_mappers.py # tests/unit/backend/test_runtime_lifecycle.py # tests/unit/backend/test_scorer_service.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_instance_registry.py
main now declares that TextTarget takes no credentials, with an empty supported_auth_modes. The saved-entry check read that as supporting no mode at all, so it refused saved TextTargets and left the saveability check for the target types with no mode to report. A class that declares no auth modes is now checked as the default api_key mode, which it has no key for.
…d to OpenAIChatTarget
# Conflicts: # pyrit/registry/registry_recipe_rules.py
A reference left to a configured default can be a list, such as a composite scorer's scorers. Each entry in the list is now named in the saved recipe, in order, and the list is refused if any of them is Temporary or not registered, instead of the whole list being refused as one unregistered entry.
varunj-msft
requested review from
hannahwestra25 and
Richard Lundeen (richlundeen)
November 8, 2026 02:55
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. It builds on #4050, which refuses saved converters and scorers that would use this server's credentials. Until the earlier PRs merge, the diff here also shows their changes.
A converter or scorer created through the API is gone when the backend restarts. With
"persist": true,POST /api/convertersandPOST /api/scorersnow save it the wayPOST /api/targetsdoes (#4026), so restore brings it back at every start.target_service.pyto a sharedservices/saved_entries.py: turn off the server values an entry would read, refuse what it can't hold or a Temporary entry it uses, write the recipe create-only, register it, and delete the recipe again if registering fails. The owner and not-saveable helpers move toroutes/common.py, andTargetOrigin(new in FEAT: Save targets created through the API #4026) becomesRegistryEntryOrigin. Targets behave as before.persiston converters and scorers follows the target rules. It needsregisterand a name matching^[a-z][a-z0-9_]{0,63}$(422 otherwise). A refusal is a typed 400 (/errors/not-saveable), a taken name is 409, and unreadable storage is 503. The owner is recorded as for targets.@apply_defaults) is written into the recipe by name, each entry of a list included, so restore uses the same entries even if nothing sets that default at the next start. A default that is Temporary, or isn't a registered entry, is refused. Other configured defaults apply again from the configuration at every start, as they do for startup entries.sourcecan't be saved yet (422).origin(saved,configurationortemporary) on list, get and create, and the startup entries of all three kinds are marked as configuration.DELETE /api/converters/{id}returns 409 for a saved converter, since it would come back from its file at the next start. Temporary and configuration converters are removed as before.typesis now reserved, as it is for targets and converters, sinceGET /api/scorers/typeswould hide a scorer with that name.pyrit_conf.mddescribe saving converters and scorers.Not covered:
GET /api/scorers/typesdoesn't say yet why a scorer type can't be saved; saving one refuses it with the reason.