Repository navigation
feat(benchmarks): filter the task list with --iterative - #553
Inference1 wants to merge 1 commit into
Conversation
Share the packaged protocol-file check with comparison configuration so both commands discover the same tasks without loading task code. Agent-assisted-by: OpenAI Codex (implementation, tests and documentation) Signed-off-by: Inference_ <3617237115@qq.com>
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff, diffComplete: true). This is Dong-bot (authorized delegate) acting as one reviewer. No prior formal reviews and no outstanding CHANGES_REQUESTED are seeded for this head.
Verified from the diff:
registry.has_iterative_protocol(name)is a file-existence check only:task_resources(name).joinpath("optimization.py").is_file(). It never imports or executes task code, which matches the docs claim. Unknown names still go throughtask_resourcesvalidation.benchmark_compare._validate_confignow calls this helper instead of an inlinePath(str(task_resources(task))) / "optimization.py"check. The condition and error message are unchanged, so there is one source of truth for "iterative" across the CLI and compare.benchmarks list --iterativeinapps/cli/__init__.pyfilters the already-sortedbuiltin_tasks()result. Plainlistis unchanged. The flag isstore_trueon thelistsubparser with help text.- Tests cover the behaviour:
- A directory named
optimization.pyis excluded. - Output order is stable (
a-task,z-task). - An
optimization.pythat raises on execution does not break listing. - The packaged list is
[moe-forward-npu, qwen21-denoise-npu]. list --helpmentions--iterative.- The subprocess device-library import guard now also runs
main(["benchmarks","list","--iterative"]).
- A directory named
- Docs (
engineering-benchmarks.md,operations.md,optimization-benchmarks.md) match the implementation. The TTS single-candidate note is kept, and the hard-coded task names in the prose are replaced by the command, with the current list still stated.
Not verified here (I can't run commands):
- I did not execute the test suite or see CI results. Please confirm CI is green, especially
test_discovery_does_not_import_device_libraries, since it now importsinferrsi.apps.cliin the subprocess. - I could not see other tests that might monkeypatch
benchmark_compare.task_resources. Such a patch would no longer affect the validation path now that the helper lives inregistry. This is a non-blocking check for the author. - No governing issue was linked in the seed, so I did not check accepted scope.
The merge ruleset still requires two eligible approvals. This Approve counts as one vote.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
PASS
首轮独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30 → 9a4b5871ac6220f86207e6be8a5b5335e7fb2952。实现符合 #549 的约定范围:CLI 与 comparison 共用 registry 的文件存在判断,默认列表仍按 builtin_tasks() 排序,session start 不变。文件存在并不代表协议或设备可用。
独立执行了当前提交的 registry、参数定义和列表分支,用临时资源验证默认排序、空结果、排除同名目录、包含导入即报错的文件(未执行该文件)和帮助说明。核对了已打包的 MoE、Qwen denoise 和 TTS 入口。没有独立重建 wheel、运行完整测试集或查看 CI。此批准针对代码改动,不替代 #549 的最终验收。
|
I checked the two points from the review on Could a maintainer approve the Ready CI run for this head? GitHub currently shows @zengchuang-hw, thank you for checking the resource edge cases. If your approval still stands, could you resubmit it with |
|
The code approval still stands for |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff).
Verified against the accepted checklist in #549:
has_iterative_protocolinsrc/inferrsi/benchmark/registry.pyistask_resources(name).joinpath("optimization.py").is_file()and does not import task code.benchmarks list --iterativefilters the already-sortedbuiltin_tasks()result; the unfiltered list is unchanged.session startis not touched.benchmark_compare._validate_configuses the same helper instead of an inlinePath(str(task_resources(...)))check; the error message is unchanged.task_resourcesremains used later in the file.- Tests cover the packaged inventory
["moe-forward-npu", "qwen21-denoise-npu"], help text, empty match, excluding a directory namedoptimization.py, including an import-raising file without executing it, and the device-library import guard now also runslist --iterative. - Docs in
engineering-benchmarks.md,operations.md, andoptimization-benchmarks.mdmatch the implementation.
Accuracy and performance comparison evidence is not applicable (CLI discovery filter, docs, and tests; no model output or runtime path change). No redundant architecture, code, or test concerns.
Note: Ready lane CI (run 44891405026) is action_required with no jobs executed yet, so I have not independently confirmed lint/tests in CI. Merge should wait until that lane actually runs and is green.
|
@Dong1017, the two first-line |
|
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5). I can't approve the Ready run for Review state on
Diff check
Next steps
|
|
@jiangkuaixue123, Dong-bot asked me to route the fork workflow approval. Could you approve Ready CI for |
Governing Issue and delivery slice
Refs #549. Dong1017’s delegate accepted this scope and assigned implementation to Inference1.
inferrsi benchmarks list --iterativelists the installed tasks that shipoptimization.pyas a file at their package root. Both the CLI and comparison configuration use the same check in the Benchmark registry. Discovery reads the package entries without loading task code or probing devices.Changed artifacts and evidence
The CLI preserves the ordering from
builtin_tasks(). I built the wheel, installed it in a fresh environment with no optional dependencies, and ran both commands outside the checkout. Their JSON output is shown here with whitespace condensed.CPU tests cover the existing list, filtered order, help text, and an empty match (
[], exit 0). A temporary registry excludes a directory namedoptimization.pyand includes a file that raises on import. The device-library discovery test also exercises the new flag.Checks and dependencies
pytest tests/test_engineering_benchmarks.py tests/test_benchmark_operations.py tests/test_benchmark_compare.py— 73 passed.ruff check src tests toolsandgit diff --check— passed.pyteston macOS arm64, Python 3.12: base9466bc8dhad 5,776 passed, 118 failed and 33 skipped. The feature tree had 5,780 passed, 117 failed and 33 skipped. Every failing test also failed on the base. Most failures occur in test-service process cleanup on this host.Knowledge authorship declaration
The change covers benchmark discovery, its CLI, tests and user guides. The protected knowledge-content declaration is inapplicable. OpenAI Codex assisted with implementation, tests and documentation.
Remaining work and Issue closure
Acceptance remains with Dong1017’s queue. The commands above and the empty-result regression provide the checks requested in #549.