Skip to content

feat(benchmarks): filter the task list with --iterative - #553

Open
Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:feat/benchmarks-list-iterative
Open

Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:feat/benchmarks-list-iterative

Conversation

@Inference1

Copy link
Copy Markdown

Governing Issue and delivery slice

Refs #549. Dong1017’s delegate accepted this scope and assigned implementation to Inference1.

inferrsi benchmarks list --iterative lists the installed tasks that ship optimization.py as a file at their package root. Both the CLI and comparison configuration use the same check in the Benchmark registry. Discovery reads the package entries without loading task code or probing devices.

Changed artifacts and evidence

The CLI preserves the ordering from builtin_tasks(). I built the wheel, installed it in a fresh environment with no optional dependencies, and ran both commands outside the checkout. Their JSON output is shown here with whitespace condensed.

$ inferrsi benchmarks list
["moe-forward-npu", "qwen21-denoise-npu", "qwen3-tts-npu"]
$ inferrsi benchmarks list --iterative
["moe-forward-npu", "qwen21-denoise-npu"]

CPU tests cover the existing list, filtered order, help text, and an empty match ([], exit 0). A temporary registry excludes a directory named optimization.py and includes a file that raises on import. The device-library discovery test also exercises the new flag.

Checks and dependencies

  • pytest tests/test_engineering_benchmarks.py tests/test_benchmark_operations.py tests/test_benchmark_compare.py — 73 passed.
  • ruff check src tests tools and git diff --check — passed.
  • Full pytest on macOS arm64, Python 3.12: base 9466bc8d had 5,776 passed, 118 failed and 33 skipped. The feature tree had 5,780 passed, 117 failed and 33 skipped. Every failing test also failed on the base. Most failures occur in test-service process cleanup on this host.

Knowledge authorship declaration

The change covers benchmark discovery, its CLI, tests and user guides. The protected knowledge-content declaration is inapplicable. OpenAI Codex assisted with implementation, tests and documentation.

Remaining work and Issue closure

Acceptance remains with Dong1017’s queue. The commands above and the empty-result regression provide the checks requested in #549.

Share the packaged protocol-file check with comparison configuration so both commands discover the same tasks without loading task code.

Agent-assisted-by: OpenAI Codex (implementation, tests and documentation)
Signed-off-by: Inference_ <3617237115@qq.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff, diffComplete: true). This is Dong-bot (authorized delegate) acting as one reviewer. No prior formal reviews and no outstanding CHANGES_REQUESTED are seeded for this head.

Verified from the diff:

  • registry.has_iterative_protocol(name) is a file-existence check only: task_resources(name).joinpath("optimization.py").is_file(). It never imports or executes task code, which matches the docs claim. Unknown names still go through task_resources validation.
  • benchmark_compare._validate_config now calls this helper instead of an inline Path(str(task_resources(task))) / "optimization.py" check. The condition and error message are unchanged, so there is one source of truth for "iterative" across the CLI and compare.
  • benchmarks list --iterative in apps/cli/__init__.py filters the already-sorted builtin_tasks() result. Plain list is unchanged. The flag is store_true on the list subparser with help text.
  • Tests cover the behaviour:
    • A directory named optimization.py is excluded.
    • Output order is stable (a-task, z-task).
    • An optimization.py that raises on execution does not break listing.
    • The packaged list is [moe-forward-npu, qwen21-denoise-npu].
    • list --help mentions --iterative.
    • The subprocess device-library import guard now also runs main(["benchmarks","list","--iterative"]).
  • Docs (engineering-benchmarks.md, operations.md, optimization-benchmarks.md) match the implementation. The TTS single-candidate note is kept, and the hard-coded task names in the prose are replaced by the command, with the current list still stated.

Not verified here (I can't run commands):

  • I did not execute the test suite or see CI results. Please confirm CI is green, especially test_discovery_does_not_import_device_libraries, since it now imports inferrsi.apps.cli in the subprocess.
  • I could not see other tests that might monkeypatch benchmark_compare.task_resources. Such a patch would no longer affect the validation path now that the helper lives in registry. This is a non-blocking check for the author.
  • No governing issue was linked in the seed, so I did not check accepted scope.

The merge ruleset still requires two eligible approvals. This Approve counts as one vote.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

首轮独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30 → 9a4b5871ac6220f86207e6be8a5b5335e7fb2952。实现符合 #549 的约定范围:CLI 与 comparison 共用 registry 的文件存在判断,默认列表仍按 builtin_tasks() 排序,session start 不变。文件存在并不代表协议或设备可用。

独立执行了当前提交的 registry、参数定义和列表分支,用临时资源验证默认排序、空结果、排除同名目录、包含导入即报错的文件(未执行该文件)和帮助说明。核对了已打包的 MoE、Qwen denoise 和 TTS 入口。没有独立重建 wheel、运行完整测试集或查看 CI。此批准针对代码改动,不替代 #549 的最终验收。

@Inference1

Copy link
Copy Markdown
Author

I checked the two points from the review on 9a4b5871. The device-library test runs benchmarks list --iterative in a fresh Python process, then checks that the five listed runtime libraries stayed unloaded. The comparison fixture patches both compare.task_resources and registry.task_resources to the same synthetic package. I reran those two tests and the import-raising protocol fixture together on CPU. All three passed. Discovery test · Comparison fixture

Could a maintainer approve the Ready CI run for this head? GitHub currently shows action_required and zero jobs. I’ll check the results as soon as it runs.

@zengchuang-hw, thank you for checking the resource edge cases. If your approval still stands, could you resubmit it with LGTM alone on the first line, as requested by the repository review guide?

Copy link
Copy Markdown
Contributor

The code approval still stands for 9a4b5871. The request to reformat it is separate from code review: this approval starts with PASS, so it does not meet the repository’s exact first-line LGTM requirement for merge. I haven’t resubmitted or approved the Ready workflow.

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 21, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff).

Verified against the accepted checklist in #549:

  • has_iterative_protocol in src/inferrsi/benchmark/registry.py is task_resources(name).joinpath("optimization.py").is_file() and does not import task code.
  • benchmarks list --iterative filters the already-sorted builtin_tasks() result; the unfiltered list is unchanged. session start is not touched.
  • benchmark_compare._validate_config uses the same helper instead of an inline Path(str(task_resources(...))) check; the error message is unchanged. task_resources remains used later in the file.
  • Tests cover the packaged inventory ["moe-forward-npu", "qwen21-denoise-npu"], help text, empty match, excluding a directory named optimization.py, including an import-raising file without executing it, and the device-library import guard now also runs list --iterative.
  • Docs in engineering-benchmarks.md, operations.md, and optimization-benchmarks.md match the implementation.

Accuracy and performance comparison evidence is not applicable (CLI discovery filter, docs, and tests; no model output or runtime path change). No redundant architecture, code, or test concerns.

Note: Ready lane CI (run 44891405026) is action_required with no jobs executed yet, so I have not independently confirmed lint/tests in CI. Merge should wait until that lane actually runs and is green.

@Inference1

Copy link
Copy Markdown
Author

@Dong1017, the two first-line LGTM approvals are now in for 9a4b5871. Could you approve the Ready CI run for this fork PR, or route it to the maintainer who handles Actions approvals? It still shows action_required with zero jobs. I’ll check the Linux results when they’re available.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

I can't approve the Ready run for 9a4b5871. Dong-bot (authorized delegate) can only post reviews and comments, and has no access to Actions or workflow approval. A maintainer with Actions approval rights needs to click "Approve and run" on run 44891405026. It shows action_required with zero jobs because this is a fork PR. Please route it to whoever handles those approvals.

Review state on 9a4b5871 (base 9466bc8d)

  • There are three formal APPROVED reviews on the current head: Dong1017 (posted by Dong-bot), @amy-why-3459 and @zengchuang-hw. No CHANGES_REQUESTED is outstanding.
  • Only two of them open with an exact LGTM line: the Dong1017/Dong-bot review and @amy-why-3459's. @zengchuang-hw's review still opens with PASS, and they declined to resubmit it. So the two first-line LGTM approvals are Dong-bot's and @amy-why-3459's, not @zengchuang-hw's. Check that against the merge rules rather than assume it.

Diff check

  • I re-read the full diff (diffComplete: true) and found no blockers.
  • has_iterative_protocol is task_resources(name).joinpath("optimization.py").is_file(). It never imports task code.
  • benchmark_compare._validate_config now calls that helper, and its error message is unchanged. The CLI and comparison share one check.
  • benchmarks list --iterative filters the already-sorted builtin_tasks() result. Plain list and session start are unchanged.
  • The tests cover the empty match, the directory named optimization.py, the import-raising file, help text, the packaged list, and the device-library import guard.
  • I did not run the tests or CI. The pytest and ruff results come from the PR description.

Next steps

@Inference1

Copy link
Copy Markdown
Author

@jiangkuaixue123, Dong-bot asked me to route the fork workflow approval. Could you approve Ready CI for 9a4b5871 when you have a chance? The run is waiting to start. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants