Repository navigation
Conversation
…iction) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 9466bc8 and head 7aa1112 (full diff, complete). There is no formal review and no outstanding CHANGES_REQUESTED on this head, so this Approve is the first independent review. I read this as a Dong-bot (authorized delegate) review.
Scope. This is a docs and examples change: README link row, a new "Bringing up a new model" section in the Apple silicon playbook, two new docs (inkling2-mini-apple-silicon.md, scheduling-rsi-lessons.md), and examples/inkling2_mini_mlx/ (scripts and round records). It touches no inferrsi package code.
Claims policy. Both new pages say up front that they are maintainer measurements with no acceptance issue and are not an InferRSI PASS. Rows are labelled measured / public / ledger / inferred, and the unpublished-log limits are stated. I found nothing that over-claims against the contributing-evidence rules.
What I checked against the committed records
- Round 1. I recomputed the adjacent-paired ratios from
r1/log-evolve.jsonl, using the README's rule that default run1+3k+iprecedes repetitioniof candidatek. They are gates 1.031, MoE 1.015, gates+MoE 1.048 and residual 1.012. These match the doc. The doc's 1.053 all-runs-median figure matchesscore-evolve.json. The drift from 47.4 to 48.9 tok/s matchesdec_all(47.45 → 48.87). - Round 2. The medians of the
pairedarrays match every table cell, for example winner evolve 1.0677 (doc 1.068) and holdout 1.064 with (1.068, 1.064, 1.060). Noise values of 2.3% (evolve) and 3.9% (holdout) matchnoise_range. All candidates showsame_output: true. - Mechanism. The two-process averages in
mechanism.txtmatch the doc: 22.07 → 20.94 ms per step and build time 3.89 → 3.34 ms. The doc's "about half" attribution to CPU graph construction is right (0.55 of 1.13 ms).mechanism.pyuses 60 steps with a 10-step warm-up, which gives the "median of 50 steps". - Playbook rows. The new rows are consistent with the same data: per-module compiles at 1–3%, the 6.4% held-out figure, and the 1.048 → 1.053 inflation.
- Gates. The 5% no-conclusion rule, evolve/holdout split, token-identity gate and adjacent pairing in
rsi_mlx.pymatch the gates the doc says were frozen. The doc also lists what was not established (one machine, single stream, short prompts, no upstream or server test). - Candidates. The winning candidate path named in the doc,
rounds/r2/cands/r2d_fused_half_gates.py, exists in the diff.
Non-blocking follow-ups
r1/score-evolve.jsonstill carries the old-scorer verdict forc3_compile_gates_moe("留" at 1.053), while the doc says every round-1 candidate was below 5% and "no conclusion". The README explains that the round-1 records predate adjacent pairing. One more sentence would help: the committed verdict strings and speedup there are from the superseded scorer. The Chinese verdict strings in the records also differ from the English onesrsi_mlx.pynow prints.- The README now links
docs/h3-metal-apple-silicon-lessons.md. It is not in this diff, so please confirm it exists onmainand the link does not 404. - The route-comparison numbers (MLX 47.0 / 1,677 tok/s against llama.cpp 41.7 / 998) and the Campaign 1/2 ledger figures are not reproducible from the repository. The docs label this honestly, but I cannot verify them.
Not verified here. I did not run any script, MLX, llama.cpp or the simulators. I did not check CI status, the external links (mlx-lm#2086, the HF repos, vllm-evolve#1), the model facts, or the macOS build string. Delegation and two-approval merge eligibility are for the integrator to confirm.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
首次独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30…7aa11120db6e9bec7ae4ab3464b6fbddf03a4d8a。结论针对本 PR 新增的评测脚本和公开文档,不是对 Inkling 加速结果作验收。无上一轮意见。
- 不完整的对照会被判为
keep。examples/inkling2_mini_mlx/rsi_mlx.py:69–90捕获异常并保存err,但最终只检查有没有速度数据,不检查err或配对数。三轮中若第一轮 token 一致且快 6%,第二轮中断,现有逻辑仍能用第一轮给出keep。请保留原始样本和错误,但只有无错误、达到--reps、完成回滚且逐样本质量门通过的候选才可进入打分判定;不完整样本应标为INCONCLUSIVE。建议加一个「第一对成功、第二对抛错」的 CPU 回归用例。 - 区分脚本的速度筛选和文档宣称的结论。
docs/inkling2-mini-apple-silicon.md把机制证据列为「better」的必要条件,但rsi_mlx.py仅看 token 和速比就写keep,不读取mechanism.py的证据。请将自动输出改为明确的「速度筛选通过,待机制验证」,或把机制证据纳入最终判定。两篇文档的公开数字也应分别链接能支撑对应范围的验收/证据(尤其是只见未公开 ledger 的调度数字),或明确标为未验证的探索记录而非已建立的性能收益。这里只验证了提交内可复算的部分。
已核对: 阅读 27 个变更文件的差异及仓库证据规则;从提交内记录独立重算了两轮邻接配对中位数。R1 最佳约 1.048,R2 胜出候选的 holdout 约 1.064,与正文一致;这不能独立核对 token 原始序列或硬件行为。没有重跑 M5 Max、llama.cpp、Ascend NPU 或调度模拟。
|
首次完整检视: 已从提交内记录独立复算,与正文一致:
同意 @zengchuang-hw 在 #544 (review) 的两点,因此本轮不 Approve。
非阻塞: CI 观察(不归因为本 PR 的 diff):Ready lane |
…n-identical Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
增量:7aa11120db6e9bec7ae4ab3464b6fbddf03a4d8a..fc17889e1adf1f065813b9f77bca6786ecf801ec,2 个提交、24 个文件。上一轮检视。
- 原有判定缺陷未修复。 R3/R4 增加了候选、记录和单步机制测量,但没有改动共用的
examples/inkling2_mini_mlx/rsi_mlx.py。它仍能在配对未达到--reps或记录了err时输出keep;它也不读取机制记录。批准前请让未完成的试验输出INCONCLUSIVE,并区分「速度筛选通过」与满足机制门后的最终判定。 - 新增的 1.491× 需要明确证据边界。 三组相邻配对可从
rounds/r4/score-holdout.json复算为 1.462 / 1.491 / 1.515,机制记录也与正文的单步耗时相符;这支持对已提交样本的描述,不等于验收。对公开性能结论,请按仓库规则关联验收 Issue 和可复核证据(固定版本、配置及原始产物),或把量化结论从公开标题中移除,保留明确标为未验证的探索经验。调度仿真的旧结论也仍只有未公开账本。 - 非阻塞建议: 写明最终候选是在查看留出集之前,基于
evolve的结果和预先约定的 5% 规则选定的;若这一点没有记录,就把两组结果都描述为探索数据。批量门控在 evolve 的记录里数值略高,但差距不到 5%,不应称任一版本为确定最快。
本轮独立阅读了新增路径、记录和文档,复算了配对结果,核对了已提交的单步计时;未运行 Metal/MLX 硬件复测、逐 token 复验或调度仿真。
|
增量检视: 已从本轮提交记录独立复算,与正文一致:
上一轮两点仍未解决:本轮未改
非阻塞:
CI:Ready lane |
…nly a screen - A candidate with an error, fewer than --reps pairs or a failed revert gets verdict INCONCLUSIVE; its error and samples stay in the record. - Token gate compares length as well as ids (zip compared a prefix only). - A complete, token-identical win >=5% now prints "speed screen passed, pending mechanism check" instead of "keep"; the mechanism gate (mechanism.py, count_ops2.py) is separate. - Scoring moved into functions with MLX imported inside main(), so it is testable on CPU; tests/test_inkling2_mlx_round_scoring.py covers first pair ok / second pair raises, a complete win, and a shorter output with an equal prefix.
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
增量:fc17889e1adf1f065813b9f77bca6786ecf801ec..e29c2cfdc15b23fb290fd760f79109449272e628,2 个文件。上一轮检视。部分配对不再给 keep,完整配对也只输出速度筛选结果,机制验证单列;这两点已修复。
仍可污染后续基线。 rsi_mlx.py 在 mod.apply(model) 返回后才把 reverted 设为 False。如果 apply() 修改模型后抛错,清理会被跳过,结果还会写成 reverted=True。实际 R4 _fast.apply() 就是在替换 DecoderLayer.__call__ 后才执行 _fuse_msg()。此外,真正的 revert() 失败后,主循环仍会把下一次测量当成默认路径。请在调用 apply() 前标记状态可能已变,在部分失败时也尝试恢复;不能确认恢复时,保留失败记录并停止本轮,或重建干净基线后再继续。请覆盖这两种异常路径。
文档证据边界未改。 R4 留出组的 1.491× 是逐对比值的中位数;已提交的两臂绝对中位数约为 45.0 → 66.6 tok/s,不是标题的 47 → 70。请分开表述,或明确后者只是估算;公开量化结论仍需可复核证据和对应验收 Issue,否则保留为明确未验证的探索记录。
在隔离环境执行了变更中的 3 个 CPU 测试,均通过;另外用两个最小替身独立复现部分 apply 失败和 rollback 失败。未重跑 MLX/Metal 或调度模拟。
Per the public performance claim rules in docs/contributing.md: - Inkling page: drop the quantitative claims from title and lead; state that it is an exploratory record without an acceptance issue; link raw route comparison outputs (examples/inkling2_mini_mlx/route/); describe the scorer's speed screen vs the separate mechanism gate; note the committed verdict strings (keep / 留, round-1 1.053) are from superseded scorers; report evolve and holdout as exploratory data (no record that the candidate was fixed before holdout was read). - Scheduling lessons: remove the 4.61x headline; ledger figures unverified. - Playbook and README rows labelled exploratory / unverified.
|
@zengchuang-hw @amy-why-3459 感谢两轮检视。按意见改了两个提交: 1. 不完整对照会被判为
2. 脚本
3. 公开数字的证据边界(两位的第 2 点后半;1.491×、路线对比、调度 4.61×) ——
4. 非阻塞
CI:
|
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
增量:e29c2cfdc15b23fb290fd760f79109449272e628..f229ed596617402f2c151b74fcc3f9e9625fe27b,1 个提交、10 个文档及记录文件。上一轮检视。
文档已移除把 47 → 70 tok/s 等同于留出组结果的标题,明确两组提示都是探索数据,并补了路线对比的原始记录。正文的证据范围比上一版清楚。
批准前仍须修复评分器的回滚路径。本提交没有改动 rsi_mlx.py:apply() 部分成功后抛错仍会跳过恢复,revert() 失败后仍会继续测后续候选。上轮两项 CPU 复现仍适用于未变的实现。请先确保部分应用也能清理;无法恢复时停止或重建基线,避免把被修改的路径当作默认对照。
另请把 route/mlx-chat.jsonl 的原始模型文本与前三行计时 JSON 分开,或将混合文件改成 .log 并更新说明;逐行校验显示第 4 行起不是有效 JSONL。已核对新增路线计时记录和文档口径,未重跑 MLX/Metal 或调度模拟。
|
增量检视: 上一轮两点已修好:
同意 @zengchuang-hw 在 #544 (review) 的判断,因此本轮不 Approve。
非阻塞:README 的 Field lessons 整行(含已有的 Apple silicon playbook 与 h3-metal lessons)都被标成 exploratory, unverified。苤只想限定本 PR 新增的两篇,建议把标签写到那两篇链接上,避免扩大到既有文档。 CI:当前 head 的 Ready lane |
…; stop the round on an unconfirmed revert - pair_candidate() sets reverted=False before calling apply(), so an apply() that changes the model and then raises is still reverted. - run_round() stops the round when a revert is not confirmed, so later default runs are not taken from a possibly changed model. - Two CPU tests for these paths. - route/mlx-chat.jsonl keeps only the timing records; the model output moves to route/mlx-chat-output.txt. Docs updated. - README: the exploratory label is on the two new pages only.
|
@amy-why-3459 @zengchuang-hw 按 #544 (comment) 改完,新提交 e6a6c03:
非阻塞:README 的 Field lessons 去掉整行标签,只在本 PR 新增的两篇(Inkling 2 Mini on Apple silicon、Scheduling RSI lessons)链接后标 CI:上一个 head f229ed5 的 ready lane(https://github.com/InferMatrixLab/InferRSI/actions/runs/44885328359)`full-check (3.10) |
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
增量:f229ed596617402f2c151b74fcc3f9e9625fe27b..e6a6c03eba9b3838e3a2e7223cdac68ba6e4d543,7 个文件。上一轮检视的两项已修:部分 apply() 失败会尝试恢复,无法确认恢复会在下一次基线测量前停止;MLX 计时 JSONL 和模型输出已拆开。未发现新的代码阻塞。
剩余是公开文档的证据要求。新版已经标为探索性,这是正确的;但 docs/scheduling-rsi-lessons.md 导语仍公开给出仅来自未公开账本的 4.61×,playbook 也仍发布数值总结。按 AGENTS.md 的数字发布规则,审批前请关联相应验收 Issue 和可核证据,或将没有验收依据的公开数字总结删去,保留方法、失败模式等定性经验。这个要求不是让本 PR 重跑硬件。
我在隔离目录执行了新增文件中的 5 个 CPU 测试,全部通过,mlx-chat.jsonl 的 3 条计时记录也均可解析。未重跑 MLX/Metal 或调度模拟。
…test line length AGENTS.md (Issues and claims, Research lock): no published numbers without an acceptance issue. Removed every speed, latency, memory, hit-rate, capacity and speedup figure from the Inkling page, the scheduling lessons and the playbook rows; kept method and failure lessons. Raw records under examples/inkling2_mini_mlx/rounds and route stay as unverified artifacts. Wrap a test line over the 100-column ruff limit.
|
增量检视: 上一轮两点已修好:
非阻塞:README 的 Field lessons 已把 本轮不 Approve。
未重跑 MLX/Metal 或调度模拟。因 ruff 失败,本 head 的 Ready lane 也尚未跑到测试。 |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PASS
增量检视:e6a6c03eba9b3838e3a2e7223cdac68ba6e4d543 → 655c3dc8268278d115585494dc7e4279b50457c8(1 个提交、4 个文件)。上轮检视的剩余阻塞已解决:三份文档移除了未经验收的量化性能总结,只保留注明未验证的定性经验和实验方法;原始探索记录和旧评分并未作为已验收结果对外宣称。此前的回滚保护和 JSONL 分离未被改动。
独立核对了三份文档与测试的差异,并执行了当前提交的 5 个 CPU 用例,全部通过。未运行 MLX/Metal 或调度实验,未看 CI。本批准只针对文档和样例改动,不是性能结论或合入判断。
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base e6a6c03 and head 655c3dc (PR base 9466bc8). Incremental review of 1 commit, 4 files. This Approve covers the docs and examples changes, not a performance acceptance or merge decision.
上一轮评论 的两点已修好:
tests/test_inkling2_mlx_round_scoring.py原先超长的run_round(...)调用已折行,最长行 93 列(限制 100)。Ready lane https://github.com/InferMatrixLab/InferRSI/actions/runs/44889531592 的full-check (3.10)与(3.11)的 Ruff 步骤均已 success,正在跑 Tests。docs/scheduling-rsi-lessons.md已撤下未公开 ledger 中的4.61×等容量/时延/命中率数字,只保留方法与失败模式;docs/inkling2-mini-apple-silicon.md与 playbook 新增的「Bringing up a new model」行也不再发布速度、时延、显存或加速比总结,原始记录仍留在examples/inkling2_mini_mlx/rounds与route/。此前的回滚路径与 JSONL 拆分未被改动。
当前 head 上 @zengchuang-hw 的 Approve 已覆盖其先前 NO PASS;未发现未解决的 CHANGES_REQUESTED 或针对本 head 的阻塞意见。未重跑 MLX/Metal 或调度实验。
|
@zengchuang-hw @amy-why-3459 按 #544 (review) 这一轮(剩余:没有验收依据的公开数字),新提交 1. 删掉所有没有验收依据的公开数字(依据
2. CI 上的 Ruff 失败(上一个 head
|
|
补上
|
|
关于 ready lane 上唯一剩下的失败(
这条在 |
Adds an exploratory on-device record for a looped MoE model on Apple silicon, four InferRSI-style search rounds on it, and the method lessons of two earlier scheduling campaigns of the same RSI line. Nothing here is a performance claim: there is no acceptance issue, nothing claims PASS, and the speed figures are labelled exploratory / unverified per the public performance claim rules in
docs/contributing.md.1.
docs/inkling2-mini-apple-silicon.md(new): Inkling 2 Mini on one M5 Max (128 GB), exploratory record.examples/inkling2_mini_mlx/route/.rounds/r*/records; evolve and holdout are both reported as exploratory data (no record that the candidate was fixed before holdout was read). Mechanism (step time, graph building, op count) is measured separately from the speed screen.2.
docs/scheduling-rsi-lessons.md(new): inherited method lessons. Campaign figures come from an unpublished ledger and are marked unverified; no headline number.3.
docs/apple-silicon-playbook.md: new "Bringing up a new model" rows (labelled exploratory / unverified).README.md: "Field lessons" link line.4.
examples/inkling2_mini_mlx/(new): benchmark scripts, the round runner, the mechanism probe, the op counter, all candidates of rounds 1–4, the records they wrote, and the raw route comparison outputs.rsi_mlx.pymarks incomplete runs (error, fewer pairs than--reps, failed revert) INCONCLUSIVE, requires equal token length and ids, and prints "speed screen passed, pending mechanism check" instead of a final verdict. CPU regression test:tests/test_inkling2_mlx_round_scoring.py.External package. The round-4 candidate is also packaged outside this repository: https://github.com/zuiho-kai/inkling2-mlx-fastpath
No knowledge-model-policy protected, policy or generated path is touched (
tools/knowledge_declaration_paths.py: all lists empty except other paths).AI authorship: drafted by Claude Code (
claude-opus-5-5); search candidates written by the same agent from the model source; not yet reviewed by a human.🤖 Generated with Claude Code