Skip to content

FEAT: Save converters and scorers created through the API - #4051

Open
varunj-msft wants to merge 160 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Converters-Scorers
Open

varunj-msft wants to merge 160 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Converters-Scorers

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Stacked on #4050. This PR's own changes: varunj-msft/PyRIT@varunj-msft/13178-Refuse-Saved-Converter-Scorer-Credentials...varunj-msft/13178-Save-Converters-Scorers

This is one of the changes for saving Registry entries across restarts. It builds on #4050, which refuses saved converters and scorers that would use this server's credentials. Until the earlier PRs merge, the diff here also shows their changes.

A converter or scorer created through the API is gone when the backend restarts. With "persist": true, POST /api/converters and POST /api/scorers now save it the way POST /api/targets does (#4026), so restore brings it back at every start.

  • The first commit only moves code. The target save steps go from target_service.py to a shared services/saved_entries.py: turn off the server values an entry would read, refuse what it can't hold or a Temporary entry it uses, write the recipe create-only, register it, and delete the recipe again if registering fails. The owner and not-saveable helpers move to routes/common.py, and TargetOrigin (new in FEAT: Save targets created through the API #4026) becomes RegistryEntryOrigin. Targets behave as before.
  • persist on converters and scorers follows the target rules. It needs register and a name matching ^[a-z][a-z0-9_]{0,63}$ (422 otherwise). A refusal is a typed 400 (/errors/not-saveable), a taken name is 409, and unreadable storage is 503. The owner is recorded as for targets.
  • A saved converter or scorer may only use saved or configuration entries. A reference left out of the request but filled from the configured default values (@apply_defaults) is written into the recipe by name, each entry of a list included, so restore uses the same entries even if nothing sets that default at the next start. A default that is Temporary, or isn't a registered entry, is refused. Other configured defaults apply again from the configuration at every start, as they do for startup entries.
  • A converter to save can't take an uploaded file yet, and is built without file parameters, as restore builds it. A type that needs a file says so. A copy made with source can't be saved yet (422).
  • Saving a scorer finds its class and builds it off the event loop, since finding the class can take over a second.
  • Converters and scorers now have origin (saved, configuration or temporary) on list, get and create, and the startup entries of all three kinds are marked as configuration.
  • DELETE /api/converters/{id} returns 409 for a saved converter, since it would come back from its file at the next start. Temporary and configuration converters are removed as before.
  • The scorer name types is now reserved, as it is for targets and converters, since GET /api/scorers/types would hide a scorer with that name.
  • The backend README and pyrit_conf.md describe saving converters and scorers.

Not covered:

  • Converters that pick randomly without a seed, such as the bijection and policy puppetry ones, come back with a new mapping after a restart, as startup converters do.
  • GET /api/scorers/types doesn't say yet why a scorer type can't be saved; saving one refuses it with the reason.
  • Saving converters from the GUI is the next PR. Showing who saved a converter or scorer, saving copies, deleting or editing saved entries, and keeping uploaded or Blob files on saved converters come later.

…API input

External construction now rejects a parameter whose type includes Path unless the
calling service passes the name in allowed_path_params after making the value safe.
The converter service allows its upload parameters once they are stored, and the
target service allows the server-set upload folder and the GitHubCopilotTarget
working directory. The scorer-only check is removed, and the "Valid parameters"
list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a
target that needs a disallowed path is left out instead of listed without the
field, matching the scorer catalog. allowed_path_params is typed frozenset[str]
so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its
server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts:
#	tests/unit/backend/test_converter_service.py
# Conflicts:
#	pyrit/backend/services/runtime_lifecycle.py
#	tests/unit/backend/test_runtime_lifecycle.py
# Conflicts:
#	pyrit/setup/initializers/tools.py
#	tests/unit/backend/test_scenario_run_service.py
#	tests/unit/backend/test_scenario_suite_execution_service.py
#	tests/unit/registry/test_attack_technique_registry.py
#	tests/unit/registry/test_instance_registry.py
#	tests/unit/registry/test_registry_recipe_restore.py
create_named_instance checks the name before it builds, and the technique
registry checks names case-insensitively before it registers. Both raised the
duplicate-name error without recording the clash, so a launch script that
caught either one could still leave a saved entry in place of its own. The name
check now records the clash itself, as register already did, and the technique
check records it too.
# Conflicts:
#	tests/unit/registry/test_registry_recipe_restore.py
# Conflicts:
#	pyrit/backend/mappers/target_mappers.py
#	pyrit/backend/models/targets.py
#	pyrit/backend/routes/targets.py
#	pyrit/backend/services/target_service.py
#	tests/unit/backend/test_api_routes.py
#	tests/unit/backend/test_mappers.py
#	tests/unit/backend/test_runtime_lifecycle.py
#	tests/unit/backend/test_target_service.py
#	tests/unit/registry/test_instance_registry.py
# Conflicts:
#	frontend/src/components/Config/CreateTargetDialog.tsx
#	pyrit/backend/routes/targets.py
#	pyrit/backend/services/target_service.py
#	tests/unit/backend/test_target_service.py
#	tests/unit/registry/test_registry_recipe_rules.py
# Conflicts:
#	pyrit/backend/mappers/target_mappers.py
#	pyrit/backend/routes/targets.py
#	tests/unit/backend/test_api_routes.py
#	tests/unit/backend/test_mappers.py
#	tests/unit/registry/test_registry_recipe_restore.py
# Conflicts:
#	tests/unit/registry/test_registry_recipe_rules.py
# Conflicts:
#	pyrit/backend/mappers/target_mappers.py
#	pyrit/backend/models/scorers.py
#	pyrit/backend/routes/converters.py
#	pyrit/backend/routes/targets.py
#	pyrit/backend/services/converter_service.py
#	pyrit/backend/services/scorer_service.py
#	pyrit/backend/services/target_service.py
#	pyrit/registry/instance_registry.py
#	tests/unit/backend/test_api_routes.py
#	tests/unit/backend/test_converter_service.py
#	tests/unit/backend/test_mappers.py
#	tests/unit/backend/test_runtime_lifecycle.py
#	tests/unit/backend/test_scorer_service.py
#	tests/unit/backend/test_target_service.py
#	tests/unit/registry/test_instance_registry.py
main now declares that TextTarget takes no credentials, with an empty
supported_auth_modes. The saved-entry check read that as supporting no mode at
all, so it refused saved TextTargets and left the saveability check for the
target types with no mode to report. A class that declares no auth modes is now
checked as the default api_key mode, which it has no key for.
# Conflicts:
#	pyrit/registry/registry_recipe_rules.py
A reference left to a configured default can be a list, such as a composite
scorer's scorers. Each entry in the list is now named in the saved recipe, in
order, and the list is refused if any of them is Temporary or not registered,
instead of the whole list being refused as one unregistered entry.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants