Skip to content

ci: add on-demand Codex routing evals - #65

Draft
teallarson wants to merge 1 commit into
codex/claude-routing-evals-cifrom
codex/openai-routing-evals-ci
Draft

teallarson wants to merge 1 commit into
codex/claude-routing-evals-cifrom
codex/openai-routing-evals-ci

Conversation

@teallarson

Copy link
Copy Markdown
Contributor

Run routing evals through the pinned Codex CLI with synthetic Arcade and direct MCP tools. The run compares Arcade alone with Arcade plus a competing MCP server and uploads a summary.

After merge, restrict the routing-evals-openai environment to main, require an approver, and add an environment secret named OPENAI_API_KEY. Dispatch Codex routing eval from main with the PR number and exact head SHA.

Full verification and both keyless setup checks passed. The actual CLI routed mock calls through both MCP servers with a scripted local model endpoint. Paid inference still needs the environment key. The eval measures Codex CLI routing with the skill description visible. It does not cover the ChatGPT UI or skill-file reads.

— Strider 🐦‍⬛

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant