[camielvs]

What

When an admission webhook denies a launch, the execution now ends FAILED with the webhook's message, not SYSTEM_ERROR. It isn't counted in execution.system_errors or sent to Bugsnag. Its downstream executions are SKIPPED, as for any other failure.

create_namespaced_job / create_namespaced_pod fails with	Before	After
400 or 403, message admission webhook "…" denied the request…	SYSTEM_ERROR	FAILED
403 RBAC (… is forbidden: User … cannot create …)	SYSTEM_ERROR	SYSTEM_ERROR
500 failed calling webhook (webhook unreachable)	SYSTEM_ERROR	SYSTEM_ERROR
Any other status, or a timeout	SYSTEM_ERROR	SYSTEM_ERROR

• launchers/interfaces.py: new LaunchRejectedError(LauncherError), with message (shown to the user) and reason (a short code for metrics).
• launchers/kubernetes_launchers.py: on a denial, the Job and Pod launchers raise LaunchRejectedError(reason="admission_denied") with the webhook's message. They log the submitted workload at WARNING, with command lines left out, because the user's message doesn't include it.
• orchestrator_sql.py: a LaunchRejectedError makes the execution FAILED, with the message in orchestration_error_message. It logs a WARNING and adds to a new counter, execution.launch_rejections, with the attribute execution.launch_rejection.reason.
• api_server_sql.py: get_container_execution_log returns the error for a FAILED execution with no container execution, as it already did for SYSTEM_ERROR. Without this, the UI shows nothing for a denied task.

Why

A cluster policy started denying the launcher's Jobs. A pipeline that fanned out to 80 tasks produced 80 SYSTEM_ERRORs, which set off system-error alerts. Nothing in the system was broken, and the webhook's message says what the user needs to change.

Dev Notes

• Matching: the API server reports a webhook denial as a Status whose message reads admission webhook "<name>" denied the request. The status code is the webhook's choice: 400 or 403 in practice. The classification needs both the code and that text, so an RBAC 403 or an unreachable webhook (500) stays a system error.
• Not covered:
  • ValidatingAdmissionPolicy denials use different wording and stay SYSTEM_ERROR.
  • A policy that matches Pods admits a Job, and then denies the pods the Job controller creates for it. That shows up only as an event on the Job, so the task stays PENDING as it does today.
• Monitoring: denials no longer reach system-error alerts. A policy that wrongly denies every workload shows up in execution.launch_rejections.

Tests

• tests/test_kubernetes_launcher_error_classification.py:
  • Kyverno- and Gatekeeper-shaped denials match.
  • These don't match: an RBAC 403, an unreachable webhook, denial text under a 500, a body that isn't JSON, and an exception without a response.
  • For both the Job and Pod launchers:
    • A denial raises LaunchRejectedError with the webhook's message.
    • The denied workload is logged without its command line.
    • Other failures stay plain LauncherErrors.
• tests/test_orchestrator_sql.py:
  • A rejected launch is FAILED with its message, and its downstream execution is SKIPPED. No system error or Bugsnag report is recorded, and the counter is incremented.
  • A plain LauncherError is still SYSTEM_ERROR.
• tests/test_execution_nodes_api_service.py:
  • The log endpoint serves the error of a FAILED execution with no container execution.
  • A PENDING one still raises ContainerExecutionNotReadyError.
• The tests depend on the change: against master's source, 17 of the 19 new tests fail. The other two check behavior this PR keeps.
• Full suite: 625 tests pass on Python 3.10, 3.12 and 3.13.
• Formatting: black --check . passes with 26.5.1 (the version in uv.lock) and 26.10.1.
• Not run against a live cluster: the tests use ApiExceptions with Status bodies shaped like the API server's.

🤖 Generated with Claude Code

[camielvs]

🤖 This is an AI-generated code review comment.

4 findings, inline below: two Medium on how broad the rejection scope is, two Low on the metric and the fast-fail message.

Cross-repo note, not for this PR: the API puts orchestration_error_message into system_error_exception_full, and tangle-ui shows that field as a "System error" banner (SystemErrorSection.tsx) and a "System Error Logs" viewer (TaskOverview/logs.tsx). So a rejected task shows FAILED, but its error still reads as a system error. Missing-outputs failures already look like this. The fix belongs in tangle-ui.

[camielvs]

🤖 This is an AI-generated code review comment.

Second pass, on 27b1146. The four comments from the first review are addressed: the kind is in the scope, there's a spec digest, the fast-fail message names the task, and the metric has an attempted attribute.

13 findings inline:

• Medium: The scope is still wider than some policies (what the digest covers, spot VMs, wrapping launchers). The one-hour cache treats state-dependent denials as permanent. Platform-wide denials no longer reach system-error alerting. Pod-level denials under a Job still hang. The state and stream endpoints still return 409 for rejected tasks.
• Low: transaction ordering vs. the cache, a duplicate traceback for invalid retries, pickling, the denied spec not being logged, ValidatingAdmissionPolicy, and the test helper's shared session.

The "System Error Logs" labelling from the first review is unchanged. The new test at tests/test_execution_nodes_api_service.py:626 asserts system_error_exception_full == message, so that test will need updating when tangle-ui stops reading that field.