diff --git a/.changeset/report-rule-evidence.md b/.changeset/report-rule-evidence.md new file mode 100644 index 00000000..e85016cb --- /dev/null +++ b/.changeset/report-rule-evidence.md @@ -0,0 +1,5 @@ +--- +"@taskless/cli": patch +--- + +`taskless report` now credits a rule's evidence beyond review comments. The facts file takes optional `sources` and `evidence` on every rule: the kinds of source the rule came from (`agent-files`, `conventions`, `docs`, `tests`, `reviews`, `commits`, `issues`) and how many times each one called for it. The CLI passes them to tskl.es unchanged, and refuses by rule and field what tskl.es would refuse. The publish summary gives each rule its totals, "N times in reviews" and "N times in conventions", as the report page does. Onboarding now tallies these while it mines, so a rule chosen from repeated fix-up commits, docs or tests no longer publishes as raised in zero reviews. Facts without the new fields publish exactly as before. diff --git a/openspec/changes/archive/2026-11-17-report-rule-evidence/.openspec.yaml b/openspec/changes/archive/2026-11-17-report-rule-evidence/.openspec.yaml new file mode 100644 index 00000000..bbe8b393 --- /dev/null +++ b/openspec/changes/archive/2026-11-17-report-rule-evidence/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-17 diff --git a/openspec/changes/archive/2026-11-17-report-rule-evidence/design.md b/openspec/changes/archive/2026-11-17-report-rule-evidence/design.md new file mode 100644 index 00000000..cd7d427e --- /dev/null +++ b/openspec/changes/archive/2026-11-17-report-rule-evidence/design.md @@ -0,0 +1,79 @@ +## Context + +The service half shipped as `report-rule-evidence` in taskless/taskless (archived 2026-11-17). Its design fixes the contract this change follows and does not reopen: + +| Kind | Group | A call-out is | +| ------------- | ----------- | ---------------------------------------------------------- | +| `agent-files` | conventions | a line in AGENTS.md, CLAUDE.md or similar | +| `conventions` | conventions | a code comment (TODO, FIXME, XXX, HACK) asking for it | +| `docs` | conventions | a passage in the docs, `@deprecated` and the like included | +| `tests` | conventions | a test that pins the behaviour | +| `reviews` | reviews | a review comment, counted as `surfaced.reviewComments` | +| `commits` | reviews | a commit in `scope.window` that made the fix | +| `issues` | reviews | an issue, ticket, recorded decision, or a comment on one | + +Counts are of explicit call-outs, never of code that follows a pattern. `evidence` does not require `history`, because commits are read with `git` alone. The issue (#681) says `trackers`; the service named the kind `issues`, and TODO/FIXME comments are `conventions`, not a kind of their own. + +The CLI builds the payload from a facts file the agent writes (`schemas/report-facts.ts`), validates the payload against the vendored JSON Schema, and repeats a service check itself where the agent needs the field named (`checkWindow`). The JSON Schema cannot express the cross-field rules on `sources` and `evidence`; only the service's `checkReport` does. + +## Goals / Non-Goals + +**Goals:** carry `sources` and `evidence` from the facts to the payload; refuse what tskl.es would refuse, by field, before publishing; show the two totals in the summary; get agents counting each kind while they mine. + +**Non-Goals:** changing what `surfaced` counts; any report-level total for the new kinds; making `sources` required in the facts; ranking one kind above another. + +## Decisions + +### D1: Kinds are a CLI table, held equal to the vendored schema by a test + +`report/evidence.ts` lists the seven kinds with the group of each, and which kinds must carry a count; the facts schema builds its `sources` enum and `evidence` keys from that table. A test asserts the table's kinds equal the vendored schema's `sources` enum and that its `evidence` keys equal the schema's, so a refresh that adds a kind fails a test until the kind is given a group. + +Alternative: build the facts enum from the vendored JSON at runtime, as `REPORT_FIXTURE_MAX_LENGTH` is read. Rejected during implementation: a JSON import types its enum as `string`, so the facts, `ReportRule` and the group table would lose the kind as a type, and a refresh that adds a kind still needs a hand edit to give it a group. One typed list, pinned to the schema by a test, is the same guarantee with the types intact. + +### D2: Facts mirror the payload shape for these two fields + +`sources` and `evidence` go on the shared `outcome` fields, so `rules`, `legacy` and `cloud` all take them, under the same names and shape as ReportV1. `assemble` copies them as given, adding no key when absent. They are agent-only facts (the CLI cannot see where a rule came from), so there is no CLI value to win a conflict with. + +Alternative: one `evidence` object holding kinds and counts together, with `sources` derived. Rejected: `agent-files` and `conventions` may be listed without a count, so the list is not derivable from the counts, and a facts shape unlike the payload is one more thing the recipe has to explain. + +### D3: The CLI repeats the service's cross-field checks + +`assemble` refuses, under `REPORT_INVALID`, each condition the service's `checkSources`/`checkEvidence` refuses, naming the rule and the field: + +- `sources` given on some rules but not all +- a kind listed twice +- `reviews` listed with no `surfaced.reviewComments` above zero +- `commits`, `issues`, `docs` or `tests` listed with no `evidence` count for it +- an `evidence` count whose kind `sources` does not list, or `evidence` with no `sources` + +A refusal from tskl.es arrives after redaction and assembly, as a service error the agent has to map back to a fact; a refusal here names the fact. The checks run against the facts before redaction, like `checkWindow`. + +Alternative: rely on the service. Rejected for the reason `checkWindow` exists. The risk is drift if the service tightens its checks; the service stays authoritative, so drift costs a later and less specific error, not a bad publish. + +### D4: Summary totals per rule, beside the title + +Each rule line gains its group totals, using the page's sums and words: reviews = `surfaced.reviewComments` + `evidence.commits` + `evidence.issues`; conventions = `evidence["agent-files"]` + `conventions` + `docs` + `tests`. Each is shown only when above zero, larger first, reviews first on a tie, "1 time" for one: + +``` + - No direct fetch calls (Taskless ast-grep rule): 12 times in reviews, 3 times in conventions +``` + +The window-level "Raised in review" line stays as it is: it is about review comments and PRs only, which is still true. + +Alternative: one "strongest evidence" phrase per rule, as #681 first suggested. Rejected, as the service rejected it: ranking kinds against each other has no basis, and the summary should read like the page it links to. + +### D5: The recipes count while mining, one procedure per kind + +`onboard`'s "Choosing candidates" already tells the agent to note PRs and comments under each candidate while mining. It now notes every call-out under the candidate with its kind, so `sources` and `evidence` come from the same list as `surfaced`, without a second read. "Evidence from commit history ... never counts" becomes: commits count as `commits` evidence, never as `surfaced`. + +`report`'s "Counting surfaced" section becomes "Counting evidence", keeping the `surfaced` procedure unchanged and adding the call-out definition per kind from the table above, the window rule (commits inside `window`, other kinds as found at publish), and the privacy rule (kinds, never paths, file names, SHAs or issue titles). Section budgets apply: the long recipes are served as sections that fit one tool call, so the new text replaces rather than adds where it can. + +## Risks / Trade-offs + +- [The service tightens a check the CLI does not repeat] → the payload is still validated by tskl.es; the agent sees the service error instead of a field-named one. The checks are short and live in one file. +- [Agents count loosely, e.g. every file following a pattern] → the recipe defines a call-out per kind and says what does not count; counts stay self-reported like every count in a report. +- [Summing commits and issues into "times in reviews" overstates review activity] → the service's choice, mirrored so the terminal and the page agree; the line says "times", not "review comments". + +## Migration Plan + +None. The service shipped first; facts without the new fields keep producing the same payload. diff --git a/openspec/changes/archive/2026-11-17-report-rule-evidence/proposal.md b/openspec/changes/archive/2026-11-17-report-rule-evidence/proposal.md new file mode 100644 index 00000000..77c4c5e5 --- /dev/null +++ b/openspec/changes/archive/2026-11-17-report-rule-evidence/proposal.md @@ -0,0 +1,30 @@ +## Why + +Onboarding mines five kinds of source for rule candidates, but the report credits only review comments: the recipe says commit evidence "never counts toward these numbers". A repository that pushes to `main`, or has little PR history, publishes every rule as raised in zero reviews even when the evidence was strong. About five runs in the CLI agent survey (Oct 12 to Nov 6) reported it. tskl.es now accepts the missing evidence (taskless/taskless#527, which closed taskless/taskless#516, deployed 2026-11-17), so the CLI can start sending it. + +## What Changes + +- Regenerate the vendored ReportV1 schema. It gains per-rule `sources`, seven kinds (`agent-files`, `conventions`, `docs`, `tests`, `reviews`, `commits`, `issues`), and `evidence`, a count of explicit call-outs for every kind but `reviews` (whose count stays `surfaced.reviewComments`). Both are optional, and `schemaVersion` stays `"1"`. +- The facts file accepts `sources` and `evidence` on every rule (`rules`, `legacy`, `cloud`), and the CLI passes them through to the payload unchanged. Their kinds are read from the vendored schema, so a schema refresh that adds a kind needs no edit to the facts. +- The CLI refuses, by field name and before any network call, the facts tskl.es would refuse on these fields: `sources` on some rules but not all, a kind listed twice, `reviews` without a review comment in `surfaced`, `commits`/`issues`/`docs`/`tests` listed without an `evidence` count, and an `evidence` count for a kind `sources` does not list. +- The publish summary gives each rule its two totals, the way the page does: "N times in reviews" (review comments, commits, issues) and "N times in conventions" (agent files, code comments, docs, tests), larger first, each only when above zero. +- The `onboard` recipe tallies evidence per candidate and per source while mining, replacing "commit history never counts" with a count of the commits that made the fix. The `report` recipe defines what a call-out is for each kind and asks for `sources` and `evidence` on every rule. Kinds are recorded, never paths. +- Facts without `sources` or `evidence` assemble and publish exactly as they do today. + +## Capabilities + +### New Capabilities + +None. + +### Modified Capabilities + +- `cli-report`: the facts carry `sources` and `evidence`, which the CLI checks and passes through; the summary shows each rule's two evidence totals; the report recipe defines how to count each kind. +- `cli-onboard`: the scan step tallies evidence for every source while mining, and commit history is counted as `commits` evidence instead of not at all. + +## Impact + +- `packages/cli/src/generated/report-v1.schema.json` (regenerated), `src/schemas/report-facts.ts`, `src/report/types.ts`, `src/report/assemble.ts`, `src/report/summary.ts`, a new `src/report/evidence.ts` for the kind groups and checks, `src/agent/onboard.md`, `src/agent/report.md`, and their tests. +- No change to the wire contract beyond what tskl.es already accepts. A CLI without this change keeps publishing valid reports. +- Delivery shape: **single PR**. One contract the service already ships, its checks, one summary line per rule, and the two recipes that ask for it, with tests; well under the review budget. +- Changeset: `patch` (pre-1.0, additive). diff --git a/openspec/changes/archive/2026-11-17-report-rule-evidence/specs/cli-onboard/spec.md b/openspec/changes/archive/2026-11-17-report-rule-evidence/specs/cli-onboard/spec.md new file mode 100644 index 00000000..05d7d786 --- /dev/null +++ b/openspec/changes/archive/2026-11-17-report-rule-evidence/specs/cli-onboard/spec.md @@ -0,0 +1,151 @@ +## MODIFIED Requirements + +### Requirement: Onboard recipe follows the canonical recipe template and asks one question + +The `onboard.md` file SHALL follow the canonical recipe template defined in the `cli-agent` capability (header with CLI version + topic version, `## Goal`, `## Preconditions`, `## Steps`, `## Errors`, `## See Also`). The spec SHALL NOT pin the topic version: it changes with every revision of the recipe, and the recipe header is where it is recorded. + +The recipe SHALL instruct the agent to: + +1. Read `.taskless/taskless.json` and respect the `install.onboarded` field. +2. Ask the user exactly one question, "Do you want me to set up Taskless for you?", preceded by a scripted introduction the agent SHALL repeat word for word and SHALL NOT reword or extend, set apart from surrounding text by blank lines, and through the host's structured question tool when the host has one. The introduction SHALL name the sources in general terms (the repository, agent files, documentation, PRs, bug tracking), state that rules are written locally with no login, and state that a report is generated. The recipe SHALL state that a yes covers mining the repository, writing rules, publishing the report, and sending onboarding feedback. On anything but a yes the agent SHALL stop. +3. Establish the routing surface before proposing any candidate, by fetching the `route` topic for the destination criterion and running `taskless detect --json` for the repository's linters, languages, rule styles, host tools, and `agent` hints. +4. Mine every reachable source without asking which: codebase TODOs/FIXMEs, agent-memory files (CLAUDE.md, AGENTS.md, .cursorrules, and similar), review comments through whichever host tool `detect` reports present and applicable (pull requests through `gh`, merge requests through `glab`), the commit history of the window being mined, read for fixes that recur, and any MCP the agent has that reaches a bug tracker, an engineering-context tool, or architectural decisions. Whether to run a hinted command such as `gh auth status`, and which MCPs to use, SHALL be left to the agent. Wherever the review-comment source is offered, the recipe SHALL direct the agent to cover the whole window rather than the most recent pull or merge requests, SHALL give commands that rely on the host CLI's own pagination so the agent writes no pagination loop of its own, and, when the window holds more than the agent can fetch review bodies for, SHALL direct it to sample evenly across the whole window and to report counts from a sample as a floor. For GitHub it SHALL name the 1000-result cap of `gh pr list` and `gh search`, give a command that counts the window's merged pull requests and one that lists them by date range, and give the repository-wide review-comments endpoint filtered by `since` for the window's inline comments. +5. Filter for high-signal candidates: repeated patterns across multiple PRs/files/comments, comments that cite a doc or style guide, and merge-blocking review feedback; drop one-off nits and pure formatting feedback. The recipe SHALL state that a rule's `surfaced` counts review comments alone, and that every other source the agent mines is credited as that kind's evidence for the report instead: commits that made the fix as `commits`, tracker and engineering-context items as `issues`, TODO/FIXME-style comments as `conventions`, agent-memory files as `agent-files`, documentation as `docs`, and tests as `tests`. The recipe SHALL direct the agent to record, while mining, each counted pull request and comment and each call-out from every other source under its candidate, with its kind, scoped to the rule it expects to write, so `surfaced`, `sources`, and `evidence` all come from that record rather than a second read. The recipe SHALL direct the agent to keep only kinds and counts for the report, never paths, file names, commits, or issue titles. +6. Author every candidate the routing surface sends to a local destination (`legacy`, `sg`, or `vale`) by following the `route` topic, without asking per candidate. +7. Record, and not author, every candidate the agent judges better solved by a runtime rule, for the report's `cloud` tier, noting how many instances of its pattern the agent saw in the code, which the report takes as that candidate's estimate. +8. Once the rules are written, run `taskless onboard --mark-complete` without asking. +9. Hand off to the `report` topic to publish the report, and, in parallel, send onboarding feedback by following the `rule-feedback` topic as the respondent. +10. Close by showing the report summary, then, as the last thing the agent does, offer to wire `taskless check` into the repository's CI by following the `ci` topic, and strongly encourage it: Taskless checks in CI are what keep agents from reintroducing the patterns these rules catch. The offer SHALL name the CI systems `detect --json` found. On a yes the agent SHALL follow the `ci` topic; on a no it SHALL name the `hooks` topic in one line as the local alternative. Either way, the agent's final message SHALL end with the report link on its own line, as a plain URL or a Markdown link and not inside a code block, and the agent SHALL then stop. The link SHALL be the one the report summary showed, never the contents of the record in `.taskless/reports/`. + +The recipe SHALL operate logged out throughout. It SHALL NOT direct the agent to `auth login`, and SHALL NOT author a runtime rule. + +Beyond the opening question and the closing CI offer, the recipe SHALL permit a question only when the agent is unsure what rules to write AND the `report` topic's gate has refused for too few Taskless rules. It SHALL limit such a question to what would let the agent write the missing rules. + +The recipe SHALL describe a detected tool as present, never as verified, and SHALL NOT instruct the agent to run `command -v` or any equivalent probe for a tool `detect` reports on. + +The recipe SHALL NOT name one bug tracker as the expected one. It MAY name trackers such as Jira and Linear as examples of the class. + +When a review-comment source is unavailable, the recipe SHALL say in one line why, stating the reason that cannot be remedied first: a repository whose `origin` is on neither GitHub nor GitLab SHALL be told there are no pull or merge requests to mine, not that a CLI is missing. + +The recipe SHALL NOT restate the destination criterion itself; it SHALL reference the `route` topic. + +#### Scenario: Recipe header includes CLI and topic version + +- **WHEN** `onboard.md` is read +- **THEN** the first line SHALL match the canonical header format `# Topic: onboard (CLI v / topic v)` + +#### Scenario: The opening question is the only scripted question + +- **WHEN** the recipe `## Steps` section is read +- **THEN** it SHALL contain the question "Do you want me to set up Taskless for you?" +- **AND** SHALL precede it with the scripted introduction, and direct the agent not to reword or extend it +- **AND** SHALL direct the agent to use the host's structured question tool when one exists +- **AND** no other step SHALL direct the agent to ask the user a question, except the step handling a refused report gate and the closing CI offer + +#### Scenario: Recipe establishes the routing surface before proposing candidates + +- **WHEN** the recipe `## Steps` section is read +- **THEN** a step instructing the agent to fetch the `route` topic and to run `detect --json` SHALL appear before the step that authors rules + +#### Scenario: Recipe does not duplicate the destination criterion + +- **WHEN** `onboard.md` is read +- **THEN** it SHALL NOT contain a table or enumeration comparing the rule destinations against one another +- **AND** it SHALL direct the agent to the `route` topic for that comparison + +#### Scenario: Sources are mined without a menu + +- **WHEN** the recipe `## Steps` section is read +- **THEN** it SHALL name codebase TODOs/FIXMEs, agent-memory files, review comments, commit history, and MCP-reachable trackers and engineering context as sources +- **AND** SHALL NOT direct the agent to ask the user which sources to include + +#### Scenario: A busy review window is covered without hand-written pagination + +- **WHEN** the recipe is rendered for a GitHub repository on a host where `gh` is on `PATH`, or with `gh` unmeasured +- **THEN** the review-comment source SHALL state that `gh pr list` and `gh search` stop at 1000 results +- **AND** SHALL give a command counting the window's merged pull requests, a command listing them by date range, and a `gh api --paginate` command for the repository's review comments `since` the window's first day +- **AND** SHALL direct the agent to write no pagination loop of its own +- **AND** SHALL direct any sample to span the whole window, never the most recent pull requests, and state that counts from a sample are a floor + +#### Scenario: Surfaced is counted while mining + +- **WHEN** the recipe's scan step is read +- **THEN** it SHALL direct the agent to record each counted pull request and comment under its candidate as it mines, at the scope of the rule it expects to write + +#### Scenario: Every source is tallied while mining + +- **WHEN** the recipe's scan step is read +- **THEN** it SHALL direct the agent to record each call-out from commits, trackers, code comments, agent-memory files, documentation, and tests under its candidate, with its kind, as it mines +- **AND** SHALL direct the agent to keep kinds and counts, never paths or titles + +#### Scenario: A present and applicable tool is offered as present + +- **WHEN** the recipe is rendered for a GitHub repository on a host where `gh` is on `PATH` +- **THEN** the review-comment source SHALL be included +- **AND** the text SHALL state that `gh` is on the PATH and that Taskless did not run it + +#### Scenario: An absent tool is omitted with its reason + +- **WHEN** the recipe is rendered for a GitHub repository on a host where `gh` is not on `PATH` +- **THEN** the review-comment source SHALL NOT be included +- **AND** the recipe SHALL state in one line that `gh` was not found on the PATH + +#### Scenario: A repository on neither host is told the source does not apply + +- **WHEN** the recipe is rendered for a repository whose `origin` is on neither GitHub nor GitLab, whatever tools are on `PATH` +- **THEN** the review-comment source SHALL NOT be included +- **AND** the stated reason SHALL be that there are no pull or merge requests to mine + +#### Scenario: Onboarding stays logged out + +- **WHEN** `onboard.md` is read +- **THEN** it SHALL NOT direct the agent to `auth login` +- **AND** SHALL direct runtime-shaped candidates to the report's `cloud` tier rather than to authoring +- **AND** SHALL direct the agent to note how many instances of each such candidate's pattern it saw, for the report's estimate + +#### Scenario: Onboarding completes without a confirmation question + +- **WHEN** the recipe `## Steps` section is read +- **THEN** it SHALL direct the agent to run `taskless onboard --mark-complete` once the rules are written +- **AND** SHALL NOT direct the agent to ask before doing so + +#### Scenario: Onboarding hands off to the report and sends feedback + +- **WHEN** the recipe `## Steps` section is read +- **THEN** a step after mark-complete SHALL direct the agent to the `report` topic +- **AND** SHALL direct it to follow the `rule-feedback` topic as the respondent + +#### Scenario: CI is offered last and strongly encouraged + +- **WHEN** the recipe's closing step is read +- **THEN** it SHALL appear after the report summary is shown and be the last step +- **AND** it SHALL direct the agent to offer wiring `check` into CI via the `ci` topic, naming the CI systems `detect --json` found +- **AND** it SHALL direct the agent to strongly encourage a yes, stating that checks in CI keep agents from reintroducing what the rules catch + +#### Scenario: Declining CI points at hooks + +- **WHEN** the user declines the CI offer +- **THEN** the recipe SHALL direct the agent to name the `hooks` topic in one line, end with the report link, and stop + +#### Scenario: The final message ends with the report link + +- **WHEN** the recipe's closing step is read +- **THEN** it SHALL direct the agent to end its final message with the report link on its own line, whether the user accepted or declined CI +- **AND** SHALL direct that the link be a plain URL or a Markdown link, not inside a code block +- **AND** SHALL direct the agent not to print the record in `.taskless/reports/`, which holds the delete URL + +#### Scenario: Recipe references its neighbors in See Also + +- **WHEN** the `## See Also` section is read +- **THEN** it SHALL include references to `taskless agent route`, `taskless agent report`, `taskless agent rule-feedback`, `taskless agent ci`, and `taskless agent hooks` + +#### Scenario: Commit history is mined for recurring fixes + +- **WHEN** the recipe is rendered, in any host-tool state +- **THEN** the sources SHALL include the commit history of the window, read for fixes that recur +- **AND** the recipe SHALL state that such fixes can be the only signal in a repository reviewed before its pull requests open, or written mostly by agents + +#### Scenario: Commit evidence does not count as surfaced + +- **WHEN** the recipe's scan step is read +- **THEN** it SHALL state that commits that made a rule's fix count as `commits` evidence and never toward `surfaced` diff --git a/openspec/changes/archive/2026-11-17-report-rule-evidence/specs/cli-report/spec.md b/openspec/changes/archive/2026-11-17-report-rule-evidence/specs/cli-report/spec.md new file mode 100644 index 00000000..11556624 --- /dev/null +++ b/openspec/changes/archive/2026-11-17-report-rule-evidence/specs/cli-report/spec.md @@ -0,0 +1,203 @@ +## MODIFIED Requirements + +### Requirement: The agent supplies session facts, not a report + +`taskless report --from ` SHALL read a facts file, not a ReportV1. The facts file SHALL carry only what the agent knows from its session: the history counts and the window they were mined over, per-rule titles, an `explain` for every rule (the agent's take, in one or two sentences, on why the rule is worth holding in this codebase), and `surfaced` counts (the merged pull requests in the window whose review raised the rule's pattern, and the review comments that asked for it), optionally per-rule `sources` (the kinds of source the rule came from) and `evidence` (a count of explicit call-outs per kind), rules authored into an existing linter (`legacy` tier), runtime-shaped candidates the agent did not author (`cloud` tier) each with `estimatedViolations`, the agent's estimate of how many instances of the candidate's pattern are in the code today, the agent's own redaction counts, and the fixtures it elects to withhold. The facts file SHALL NOT accept any value input: no token, call, price, time, or rate estimate. The CLI SHALL merge the facts with what it establishes itself, and SHALL treat its own values as authoritative for every field it can establish. + +Every published rule SHALL carry a `state`. Rules under `.taskless/rules/` and `legacy` rules SHALL be `written`; `cloud` candidates from the facts file SHALL be `candidate`. + +The facts file SHALL be validated against a published facts schema before anything else happens. The `report` recipe SHALL embed that schema. + +Every rule in the facts, in `rules`, `legacy`, or `cloud`, MAY carry `sources` and `evidence` in the shape ReportV1 gives them, and the CLI SHALL take their kinds from the vendored ReportV1 schema rather than from a list of its own. The CLI SHALL carry each rule's `sources` and `evidence` to the published rule unchanged, and SHALL add neither field to a rule whose facts omit it. Before any network call, the CLI SHALL refuse under `REPORT_INVALID`, naming the rule and the field, facts in which: `sources` is given on some rules but not all; a rule lists a kind twice; a rule lists `reviews` without `surfaced.reviewComments` above zero; a rule lists `commits`, `issues`, `docs`, or `tests` without an `evidence` count for that kind; or a rule carries an `evidence` count for a kind its `sources` does not list, or `evidence` without `sources`. A rule MAY list `agent-files` or `conventions` without a count, and MAY carry `surfaced` without listing `reviews`. + +#### Scenario: Facts cannot set what the CLI knows + +- **WHEN** the facts file gives a rule under `.taskless/rules/` an engine, a language, a test status, or a state +- **THEN** the command SHALL refuse the field as unknown under `REPORT_INVALID` +- **AND** a report that publishes SHALL carry the engine, language, test status, and state the CLI established + +#### Scenario: Malformed facts are refused before any other work + +- **WHEN** the facts file is not valid JSON or fails the facts schema +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` naming each failing field +- **AND** SHALL NOT perform any network call + +#### Scenario: Cloud candidates are carried as cloud tier + +- **WHEN** the facts file lists a runtime-shaped candidate +- **THEN** the published report SHALL include it with `tier: "cloud"`, `engine: "runtime"`, and `state: "candidate"` +- **AND** its tests status SHALL be `not-run` +- **AND** it SHALL carry the agent's `estimatedViolations` + +#### Scenario: A candidate without an estimate is refused + +- **WHEN** the facts file lists a `cloud` candidate with no `estimatedViolations` +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` + +#### Scenario: Every rule carries the agent's explanation + +- **WHEN** the facts give a local rule, a `legacy` rule, and a `cloud` candidate each an `explain` +- **THEN** the published report SHALL carry each one's `explain` on that rule + +#### Scenario: A rule without an explanation is refused + +- **WHEN** the facts give any rule, in `rules`, `legacy`, or `cloud`, no `explain`, or one that is blank or longer than 1000 characters +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` + +#### Scenario: Value inputs are not accepted + +- **WHEN** the facts file includes `estimates`, `contextTokens`, `callsPerCatch`, or `complexity` +- **THEN** the command SHALL refuse the field as unknown under `REPORT_INVALID` + +#### Scenario: Fix counts are not accepted + +- **WHEN** the facts file gives a rule `fixed`, `fixedBasis`, or `origin` +- **THEN** the command SHALL refuse the field as unknown under `REPORT_INVALID` + +#### Scenario: Sources and evidence are carried to the report + +- **WHEN** the facts give a local rule `sources: ["commits", "docs"]` and `evidence: { "commits": 4, "docs": 1 }` +- **THEN** the published rule SHALL carry exactly that `sources` and that `evidence` + +#### Scenario: Facts without sources publish as before + +- **WHEN** no rule in the facts carries `sources` or `evidence` +- **THEN** no published rule SHALL carry `sources` or `evidence` +- **AND** the payload SHALL otherwise be the one the same facts produced before this change + +#### Scenario: Sources must be given on every rule or on none + +- **WHEN** the facts give one rule `sources` and another rule none +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` +- **AND** SHALL NOT perform any network call + +#### Scenario: A counted kind needs its count + +- **WHEN** a rule lists `commits` in `sources` and carries no `evidence.commits` +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` naming that rule and `evidence.commits` + +#### Scenario: A count needs its kind + +- **WHEN** a rule carries `evidence.tests` and its `sources` does not list `tests` +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` naming that rule and `evidence.tests` + +#### Scenario: Reviews need a review comment + +- **WHEN** a rule lists `reviews` and its `surfaced.reviewComments` is 0 +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` naming that rule + +#### Scenario: Original kinds may be listed without a count + +- **WHEN** a rule lists `agent-files` and `conventions` and carries no `evidence` +- **THEN** the command SHALL NOT refuse it for that + +### Requirement: A published report is recorded and summarized + +On a 201 response the CLI SHALL write `.taskless/reports/.json` (UTC) holding the report URL, the delete credential exactly as the service returned it, and the expiry. When a record by that name already exists, the CLI SHALL instead write `_.json`, with the smallest `n` from 2 upward that is free, and SHALL NOT overwrite any existing record. When the record cannot be written at all, the report is live and the record was its only copy of the delete credential, so the CLI SHALL exit non-zero with `REPORT_RECORD_FAILED`, with a message stating that the report is live and giving its URL and its delete credential; with `--json` the error envelope SHALL also carry them as `url` and `deleteUrl`, with the expiry as `expiresAt`. Otherwise it SHALL then print a brief summary rendered from a CLI template: one line per rule giving its title and tier, followed by the rule's evidence totals; when the report carries `findings`, a line stating how many problems the rules found in how many files in the code today; a line stating how many review comments across how many pull requests raised the rules' patterns over the window; the report URL; the expiry; and one sentence stating the report can be deleted after review. A rule's evidence totals SHALL be two, worded as the report page words them: "N times in reviews", the sum of `surfaced.reviewComments`, `evidence.commits`, and `evidence.issues`; and "N times in conventions", the sum of `evidence["agent-files"]`, `evidence.conventions`, `evidence.docs`, and `evidence.tests`. Each SHALL be shown only when above zero, the larger first and reviews first on a tie, with "1 time" for one; a rule with neither shows its title and tier alone. The summary SHALL NOT present a monthly rate. With `--json` it SHALL instead print a JSON object carrying the URL, the expiry, the record's path, and the rendered summary. When the record is written, the delete credential SHALL NOT appear on stdout in either form. + +#### Scenario: The record is written + +- **WHEN** a publish succeeds +- **THEN** exactly one new file SHALL exist under `.taskless/reports/` holding the URL, the delete credential, and the expiry + +#### Scenario: A taken record name moves to a suffix + +- **WHEN** a publish succeeds and `.taskless/reports/2026-10-13_142200.json` already exists for the same second +- **THEN** the record SHALL be written as `2026-10-13_142200_2.json` +- **AND** the existing record SHALL be unchanged +- **AND** `taskless report --delete 2026-10-13_142200_2.json` SHALL accept it + +#### Scenario: A record that cannot be written keeps the credential + +- **WHEN** a publish succeeds and `.taskless/reports/` cannot be written +- **THEN** the command SHALL exit non-zero with `REPORT_RECORD_FAILED` +- **AND** its output SHALL contain the report URL and the delete credential, with or without `--json` + +#### Scenario: The summary names the link and the way back + +- **WHEN** a publish succeeds without `--json` +- **THEN** stdout SHALL list each published rule's title and tier +- **AND** SHALL print the report URL +- **AND** SHALL state that the report can be deleted after review + +#### Scenario: The summary leads with what the rules found + +- **WHEN** a publish succeeds and the report carries `findings` of 342 violations in 118 files +- **THEN** the summary SHALL state that the rules found 342 problems in 118 files in the code today + +#### Scenario: The summary omits findings it does not have + +- **WHEN** a publish succeeds and the report carries no `findings` +- **THEN** the summary SHALL NOT state a count of problems found + +#### Scenario: The summary gives each rule its evidence totals + +- **WHEN** a publish succeeds and a rule carries `surfaced.reviewComments` of 2, `evidence.commits` of 4, and `evidence.docs` of 1 +- **THEN** that rule's line SHALL state "6 times in reviews, 1 time in conventions" + +#### Scenario: Conventions lead when they are larger + +- **WHEN** a publish succeeds and a rule carries no `surfaced` and `evidence` of `{ "agent-files": 2, "tests": 3 }` +- **THEN** that rule's line SHALL state "5 times in conventions" and no reviews total + +#### Scenario: A rule without evidence shows no totals + +- **WHEN** a publish succeeds and a rule carries no `evidence` and no review comments +- **THEN** that rule's line SHALL give its title and tier and no totals + +#### Scenario: The delete credential stays on disk + +- **WHEN** a publish succeeds and its record is written, with or without `--json` +- **THEN** stdout SHALL NOT contain the delete credential + +### Requirement: The report recipe drives publishing and deletion + +The CLI SHALL embed a `report` recipe following the canonical recipe template. It SHALL instruct the agent to: run `taskless report` and read `missing`; supply each missing fact from its session, writing the facts to `.taskless/.tmp-report.json`; give every rule an `explain` of one or two sentences saying what goes wrong without the rule and what holding it buys, grounded in what the session saw and carrying no counts; count each rule's `surfaced` by the method the recipe states; give every rule `sources` and `evidence`, counted by the method the recipe states; give each cloud candidate an `estimatedViolations` from what it saw while mining; run `taskless report --from` that path; on `REPORT_INSUFFICIENT`, ask the user only what would let it author the rules or establish the facts the refusal names, then retry; on success, show the CLI's summary as printed and tell the user the report can be deleted after review if they ask; delete the facts file; and on a request to delete, run `taskless report --delete` with the record. The recipe SHALL embed the facts schema and SHALL state that tskl.es does not redact, so anything the agent cannot vouch for is to be withheld. + +The counting method SHALL be a single procedure: a pull request counts toward a rule's `surfaced.prs` when it merged inside the window and at least one review comment on it asked for the change the rule enforces; a comment counts toward `surfaced.reviewComments` when a reader of that comment would say it asks for that change. Only review comments count toward `surfaced`: the recipe SHALL state that a fix-up commit, a `git blame` date, or a finding in the code today is not a review comment and SHALL NOT be counted toward either field. Every author counts: a comment from a bot account or from the pull request's own author SHALL count the same as a reviewer's, since each one spends cycles, tokens, time, or money. The recipe SHALL direct the agent to judge each comment by reading it, not by matching keywords. + +The recipe SHALL define, for every kind `evidence` counts, what one call-out is: a line in an agent file (`agent-files`); a code comment such as TODO, FIXME, XXX or HACK asking for the pattern (`conventions`); a passage in the documentation, doc annotations such as `@deprecated` included (`docs`); a test that pins the behaviour (`tests`); a commit inside the window that made the fix (`commits`); and an issue, ticket, recorded decision, or comment on one that asked for it (`issues`). It SHALL state that code which merely follows a pattern is not a call-out, that commits are counted inside the window while every other kind is counted as found at publish, that `reviews` has no `evidence` count because `surfaced.reviewComments` is its count, that a listed `commits`, `issues`, `docs`, or `tests` needs its count, and that `sources` and `evidence` name kinds only, never a path, file name, commit, or issue. + +Whenever the agent restates the report's numbers in its own words, the recipe SHALL require it to keep counts the CLI measured in the code today, counts mined from review history over the window, and its own estimates for cloud candidates apart, and to say which is which. + +The recipe SHALL NOT ask the agent for any value input: tskl.es estimates reviewer time, agent tokens, and dollars itself. + +#### Scenario: The recipe asks questions only on refusal + +- **WHEN** the `report` recipe's steps are read +- **THEN** the only step that permits a question to the user SHALL be the one handling `REPORT_INSUFFICIENT` + +#### Scenario: The recipe embeds the facts schema + +- **WHEN** an agent runs `taskless agent report` +- **THEN** stdout SHALL open with `# Topic: report` and contain the JSON Schema for the facts file + +#### Scenario: The recipe states one counting method + +- **WHEN** an agent runs `taskless agent report` +- **THEN** stdout SHALL state how to count `surfaced.prs` and `surfaced.reviewComments`, including that bot and author comments count +- **AND** SHALL NOT ask for a count of fixes + +#### Scenario: Only review comments count as surfaced + +- **WHEN** an agent runs `taskless agent report` +- **THEN** the `## Counting evidence` section SHALL state that a fix-up commit, a `git blame` date, and a finding in the code today are not review comments and do not count toward `surfaced` + +#### Scenario: The recipe defines a call-out for every counted kind + +- **WHEN** an agent runs `taskless agent report` +- **THEN** the `## Counting evidence` section SHALL state what one call-out is for `agent-files`, `conventions`, `docs`, `tests`, `commits`, and `issues` +- **AND** SHALL state that code which merely follows a pattern is not counted +- **AND** SHALL state that `sources` and `evidence` name kinds, never paths, files, commits, or issues + +#### Scenario: The recipe asks for an explanation of every rule + +- **WHEN** an agent runs `taskless agent report` +- **THEN** stdout SHALL contain an `## Explaining a rule` section asking for one or two sentences on why each rule, written or candidate, is worth holding in this codebase +- **AND** that section SHALL tell the agent to leave counts out of `explain` + +#### Scenario: The recipe asks for a candidate estimate and no value input + +- **WHEN** an agent runs `taskless agent report` +- **THEN** the step that supplies facts SHALL direct the agent to give each cloud candidate an `estimatedViolations` +- **AND** SHALL NOT mention `contextTokens` or `callsPerCatch` diff --git a/openspec/changes/archive/2026-11-17-report-rule-evidence/tasks.md b/openspec/changes/archive/2026-11-17-report-rule-evidence/tasks.md new file mode 100644 index 00000000..10623721 --- /dev/null +++ b/openspec/changes/archive/2026-11-17-report-rule-evidence/tasks.md @@ -0,0 +1,24 @@ +## 1. Contract and facts + +- [x] 1.1 Commit the regenerated `src/generated/report-v1.schema.json` (fetched from `https://tskl.es/api/v1/schema` after taskless/taskless#527 deployed) and verify `pnpm --filter @taskless/cli test` still passes against it unchanged +- [x] 1.2 Add `src/report/evidence.ts`: the kinds with the group of each, held equal to the vendored schema's `sources` enum and `evidence` keys by a test, and the kinds that must carry a count; verify with a test that every kind in the vendored schema has a group and that `reviews` is the only kind without an `evidence` key +- [x] 1.3 Add optional `sources` and `evidence` to the shared rule facts in `src/schemas/report-facts.ts`, built from 1.2's kinds, and to `ReportRule` in `src/report/types.ts`; verify the facts schema refuses an unknown kind, a zero count, and an unknown evidence key, and that the `report` recipe's embedded facts schema shows both fields + +## 2. Assemble and check + +- [x] 2.1 Carry `sources` and `evidence` from `rules`, `legacy` and `cloud` facts to the payload in `assemble.ts`, adding no key when absent; verify with tests that a payload carries them unchanged and validates against the vendored schema, and that facts without them produce the same payload as before +- [x] 2.2 Add the cross-field checks (every rule or none, duplicate kind, `reviews` without review comments, counted kind without its count, count without its kind, `evidence` without `sources`) as `REPORT_INVALID` naming rule and field; verify one test per check fails when that check is disabled, and that `agent-files`/`conventions` without counts pass + +## 3. Summary + +- [x] 3.1 Add each rule's evidence totals to its summary line in `summary.ts` (sums, ordering, tie, "1 time", omitted at zero) per design D4; verify with tests for the three summary scenarios in the `cli-report` delta and that a report without evidence renders as before apart from rules with review comments + +## 4. Recipes + +- [x] 4.1 Rewrite `onboard.md` "Choosing candidates" to tally every source's call-outs per candidate with its kind while mining, crediting commits as `commits` evidence and never as `surfaced`; bump the topic version; verify the recipe tests and the section budget pass +- [x] 4.2 Rename `report.md` "Counting surfaced" to "Counting evidence", add the per-kind call-out definitions, window rule and kinds-not-paths rule, and ask for `sources`/`evidence` in step 2; bump the topic version; verify with `pnpm build && pnpm cli agent report` that the section reads as the delta specifies and the recipe tests and section budget pass + +## 5. Ship + +- [x] 5.1 Add a `patch` changeset for `@taskless/cli` describing the new facts fields and summary totals; verify `pnpm typecheck`, `pnpm lint` and `pnpm test` pass +- [x] 5.2 Run `pnpm openspec validate report-rule-evidence --strict`, then the pre-archive check from AGENTS.md (archive, confirm every prior scenario of `cli-report` and `cli-onboard` survives, reset); archive the change and verify `openspec/changes/` holds only `archive/` diff --git a/openspec/specs/cli-onboard/spec.md b/openspec/specs/cli-onboard/spec.md index d74cb74d..274310da 100644 --- a/openspec/specs/cli-onboard/spec.md +++ b/openspec/specs/cli-onboard/spec.md @@ -189,7 +189,7 @@ The recipe SHALL instruct the agent to: 2. Ask the user exactly one question, "Do you want me to set up Taskless for you?", preceded by a scripted introduction the agent SHALL repeat word for word and SHALL NOT reword or extend, set apart from surrounding text by blank lines, and through the host's structured question tool when the host has one. The introduction SHALL name the sources in general terms (the repository, agent files, documentation, PRs, bug tracking), state that rules are written locally with no login, and state that a report is generated. The recipe SHALL state that a yes covers mining the repository, writing rules, publishing the report, and sending onboarding feedback. On anything but a yes the agent SHALL stop. 3. Establish the routing surface before proposing any candidate, by fetching the `route` topic for the destination criterion and running `taskless detect --json` for the repository's linters, languages, rule styles, host tools, and `agent` hints. 4. Mine every reachable source without asking which: codebase TODOs/FIXMEs, agent-memory files (CLAUDE.md, AGENTS.md, .cursorrules, and similar), review comments through whichever host tool `detect` reports present and applicable (pull requests through `gh`, merge requests through `glab`), the commit history of the window being mined, read for fixes that recur, and any MCP the agent has that reaches a bug tracker, an engineering-context tool, or architectural decisions. Whether to run a hinted command such as `gh auth status`, and which MCPs to use, SHALL be left to the agent. Wherever the review-comment source is offered, the recipe SHALL direct the agent to cover the whole window rather than the most recent pull or merge requests, SHALL give commands that rely on the host CLI's own pagination so the agent writes no pagination loop of its own, and, when the window holds more than the agent can fetch review bodies for, SHALL direct it to sample evenly across the whole window and to report counts from a sample as a floor. For GitHub it SHALL name the 1000-result cap of `gh pr list` and `gh search`, give a command that counts the window's merged pull requests and one that lists them by date range, and give the repository-wide review-comments endpoint filtered by `since` for the window's inline comments. -5. Filter for high-signal candidates: repeated patterns across multiple PRs/files/comments, comments that cite a doc or style guide, and merge-blocking review feedback; drop one-off nits and pure formatting feedback. Commit-history evidence SHALL choose candidates only; the recipe SHALL state that it never counts toward a rule's `surfaced`, which counts review comments alone. The recipe SHALL direct the agent to record each counted pull request and comment under its candidate while mining, scoped to the rule it expects to write, so the counts come from that record rather than a second read. +5. Filter for high-signal candidates: repeated patterns across multiple PRs/files/comments, comments that cite a doc or style guide, and merge-blocking review feedback; drop one-off nits and pure formatting feedback. The recipe SHALL state that a rule's `surfaced` counts review comments alone, and that every other source the agent mines is credited as that kind's evidence for the report instead: commits that made the fix as `commits`, tracker and engineering-context items as `issues`, TODO/FIXME-style comments as `conventions`, agent-memory files as `agent-files`, documentation as `docs`, and tests as `tests`. The recipe SHALL direct the agent to record, while mining, each counted pull request and comment and each call-out from every other source under its candidate, with its kind, scoped to the rule it expects to write, so `surfaced`, `sources`, and `evidence` all come from that record rather than a second read. The recipe SHALL direct the agent to keep only kinds and counts for the report, never paths, file names, commits, or issue titles. 6. Author every candidate the routing surface sends to a local destination (`legacy`, `sg`, or `vale`) by following the `route` topic, without asking per candidate. 7. Record, and not author, every candidate the agent judges better solved by a runtime rule, for the report's `cloud` tier, noting how many instances of its pattern the agent saw in the code, which the report takes as that candidate's estimate. 8. Once the rules are written, run `taskless onboard --mark-complete` without asking. @@ -251,6 +251,12 @@ The recipe SHALL NOT restate the destination criterion itself; it SHALL referenc - **WHEN** the recipe's scan step is read - **THEN** it SHALL direct the agent to record each counted pull request and comment under its candidate as it mines, at the scope of the rule it expects to write +#### Scenario: Every source is tallied while mining + +- **WHEN** the recipe's scan step is read +- **THEN** it SHALL direct the agent to record each call-out from commits, trackers, code comments, agent-memory files, documentation, and tests under its candidate, with its kind, as it mines +- **AND** SHALL direct the agent to keep kinds and counts, never paths or titles + #### Scenario: A present and applicable tool is offered as present - **WHEN** the recipe is rendered for a GitHub repository on a host where `gh` is on `PATH` @@ -321,7 +327,7 @@ The recipe SHALL NOT restate the destination criterion itself; it SHALL referenc #### Scenario: Commit evidence does not count as surfaced - **WHEN** the recipe's scan step is read -- **THEN** it SHALL state that evidence from commit history chooses candidates and does not count toward `surfaced` +- **THEN** it SHALL state that commits that made a rule's fix count as `commits` evidence and never toward `surfaced` ### Requirement: Onboard recipe first checks a worktree has Taskless committed diff --git a/openspec/specs/cli-report/spec.md b/openspec/specs/cli-report/spec.md index 5c16a5b2..36091ce8 100644 --- a/openspec/specs/cli-report/spec.md +++ b/openspec/specs/cli-report/spec.md @@ -65,12 +65,14 @@ The draft SHALL NOT perform any network call. ### Requirement: The agent supplies session facts, not a report -`taskless report --from ` SHALL read a facts file, not a ReportV1. The facts file SHALL carry only what the agent knows from its session: the history counts and the window they were mined over, per-rule titles, an `explain` for every rule (the agent's take, in one or two sentences, on why the rule is worth holding in this codebase), and `surfaced` counts (the merged pull requests in the window whose review raised the rule's pattern, and the review comments that asked for it), rules authored into an existing linter (`legacy` tier), runtime-shaped candidates the agent did not author (`cloud` tier) each with `estimatedViolations`, the agent's estimate of how many instances of the candidate's pattern are in the code today, the agent's own redaction counts, and the fixtures it elects to withhold. The facts file SHALL NOT accept any value input: no token, call, price, time, or rate estimate. The CLI SHALL merge the facts with what it establishes itself, and SHALL treat its own values as authoritative for every field it can establish. +`taskless report --from ` SHALL read a facts file, not a ReportV1. The facts file SHALL carry only what the agent knows from its session: the history counts and the window they were mined over, per-rule titles, an `explain` for every rule (the agent's take, in one or two sentences, on why the rule is worth holding in this codebase), and `surfaced` counts (the merged pull requests in the window whose review raised the rule's pattern, and the review comments that asked for it), optionally per-rule `sources` (the kinds of source the rule came from) and `evidence` (a count of explicit call-outs per kind), rules authored into an existing linter (`legacy` tier), runtime-shaped candidates the agent did not author (`cloud` tier) each with `estimatedViolations`, the agent's estimate of how many instances of the candidate's pattern are in the code today, the agent's own redaction counts, and the fixtures it elects to withhold. The facts file SHALL NOT accept any value input: no token, call, price, time, or rate estimate. The CLI SHALL merge the facts with what it establishes itself, and SHALL treat its own values as authoritative for every field it can establish. Every published rule SHALL carry a `state`. Rules under `.taskless/rules/` and `legacy` rules SHALL be `written`; `cloud` candidates from the facts file SHALL be `candidate`. The facts file SHALL be validated against a published facts schema before anything else happens. The `report` recipe SHALL embed that schema. +Every rule in the facts, in `rules`, `legacy`, or `cloud`, MAY carry `sources` and `evidence` in the shape ReportV1 gives them, and the CLI SHALL take their kinds from the vendored ReportV1 schema rather than from a list of its own. The CLI SHALL carry each rule's `sources` and `evidence` to the published rule unchanged, and SHALL add neither field to a rule whose facts omit it. Before any network call, the CLI SHALL refuse under `REPORT_INVALID`, naming the rule and the field, facts in which: `sources` is given on some rules but not all; a rule lists a kind twice; a rule lists `reviews` without `surfaced.reviewComments` above zero; a rule lists `commits`, `issues`, `docs`, or `tests` without an `evidence` count for that kind; or a rule carries an `evidence` count for a kind its `sources` does not list, or `evidence` without `sources`. A rule MAY list `agent-files` or `conventions` without a count, and MAY carry `surfaced` without listing `reviews`. + #### Scenario: Facts cannot set what the CLI knows - **WHEN** the facts file gives a rule under `.taskless/rules/` an engine, a language, a test status, or a state @@ -115,6 +117,43 @@ The facts file SHALL be validated against a published facts schema before anythi - **WHEN** the facts file gives a rule `fixed`, `fixedBasis`, or `origin` - **THEN** the command SHALL refuse the field as unknown under `REPORT_INVALID` +#### Scenario: Sources and evidence are carried to the report + +- **WHEN** the facts give a local rule `sources: ["commits", "docs"]` and `evidence: { "commits": 4, "docs": 1 }` +- **THEN** the published rule SHALL carry exactly that `sources` and that `evidence` + +#### Scenario: Facts without sources publish as before + +- **WHEN** no rule in the facts carries `sources` or `evidence` +- **THEN** no published rule SHALL carry `sources` or `evidence` +- **AND** the payload SHALL otherwise be the one the same facts produced before this change + +#### Scenario: Sources must be given on every rule or on none + +- **WHEN** the facts give one rule `sources` and another rule none +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` +- **AND** SHALL NOT perform any network call + +#### Scenario: A counted kind needs its count + +- **WHEN** a rule lists `commits` in `sources` and carries no `evidence.commits` +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` naming that rule and `evidence.commits` + +#### Scenario: A count needs its kind + +- **WHEN** a rule carries `evidence.tests` and its `sources` does not list `tests` +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` naming that rule and `evidence.tests` + +#### Scenario: Reviews need a review comment + +- **WHEN** a rule lists `reviews` and its `surfaced.reviewComments` is 0 +- **THEN** the command SHALL exit non-zero with `REPORT_INVALID` naming that rule + +#### Scenario: Original kinds may be listed without a count + +- **WHEN** a rule lists `agent-files` and `conventions` and carries no `evidence` +- **THEN** the command SHALL NOT refuse it for that + ### Requirement: The CLI redacts and withholds before publishing tskl.es performs no redaction of its own. Before publishing, the CLI SHALL scan every fixture and every string supplied by the facts file for secrets, email addresses, host names, IP addresses, and phone numbers. A fixture containing a match SHALL be withheld in full, never partially masked: its code SHALL be omitted, its side SHALL be recorded in that rule's `withheld`, and `redaction.withheldFixtures` SHALL count it. A match in any other agent-supplied string SHALL refuse the publish with `REPORT_REDACTION`, naming the field, so the agent can rewrite it. A fixture whose code exceeds the schema's length limit SHALL be withheld the same way. @@ -182,7 +221,7 @@ The CLI SHALL publish with `POST https://tskl.es/api/v1/reports` and SHALL alway ### Requirement: A published report is recorded and summarized -On a 201 response the CLI SHALL write `.taskless/reports/.json` (UTC) holding the report URL, the delete credential exactly as the service returned it, and the expiry. When a record by that name already exists, the CLI SHALL instead write `_.json`, with the smallest `n` from 2 upward that is free, and SHALL NOT overwrite any existing record. When the record cannot be written at all, the report is live and the record was its only copy of the delete credential, so the CLI SHALL exit non-zero with `REPORT_RECORD_FAILED`, with a message stating that the report is live and giving its URL and its delete credential; with `--json` the error envelope SHALL also carry them as `url` and `deleteUrl`, with the expiry as `expiresAt`. Otherwise it SHALL then print a brief summary rendered from a CLI template: one line per rule giving its title and tier; when the report carries `findings`, a line stating how many problems the rules found in how many files in the code today; a line stating how many review comments across how many pull requests raised the rules' patterns over the window; the report URL; the expiry; and one sentence stating the report can be deleted after review. The summary SHALL NOT present a monthly rate. With `--json` it SHALL instead print a JSON object carrying the URL, the expiry, the record's path, and the rendered summary. When the record is written, the delete credential SHALL NOT appear on stdout in either form. +On a 201 response the CLI SHALL write `.taskless/reports/.json` (UTC) holding the report URL, the delete credential exactly as the service returned it, and the expiry. When a record by that name already exists, the CLI SHALL instead write `_.json`, with the smallest `n` from 2 upward that is free, and SHALL NOT overwrite any existing record. When the record cannot be written at all, the report is live and the record was its only copy of the delete credential, so the CLI SHALL exit non-zero with `REPORT_RECORD_FAILED`, with a message stating that the report is live and giving its URL and its delete credential; with `--json` the error envelope SHALL also carry them as `url` and `deleteUrl`, with the expiry as `expiresAt`. Otherwise it SHALL then print a brief summary rendered from a CLI template: one line per rule giving its title and tier, followed by the rule's evidence totals; when the report carries `findings`, a line stating how many problems the rules found in how many files in the code today; a line stating how many review comments across how many pull requests raised the rules' patterns over the window; the report URL; the expiry; and one sentence stating the report can be deleted after review. A rule's evidence totals SHALL be two, worded as the report page words them: "N times in reviews", the sum of `surfaced.reviewComments`, `evidence.commits`, and `evidence.issues`; and "N times in conventions", the sum of `evidence["agent-files"]`, `evidence.conventions`, `evidence.docs`, and `evidence.tests`. Each SHALL be shown only when above zero, the larger first and reviews first on a tie, with "1 time" for one; a rule with neither shows its title and tier alone. The summary SHALL NOT present a monthly rate. With `--json` it SHALL instead print a JSON object carrying the URL, the expiry, the record's path, and the rendered summary. When the record is written, the delete credential SHALL NOT appear on stdout in either form. #### Scenario: The record is written @@ -219,6 +258,21 @@ On a 201 response the CLI SHALL write `.taskless/reports/.jso - **WHEN** a publish succeeds and the report carries no `findings` - **THEN** the summary SHALL NOT state a count of problems found +#### Scenario: The summary gives each rule its evidence totals + +- **WHEN** a publish succeeds and a rule carries `surfaced.reviewComments` of 2, `evidence.commits` of 4, and `evidence.docs` of 1 +- **THEN** that rule's line SHALL state "6 times in reviews, 1 time in conventions" + +#### Scenario: Conventions lead when they are larger + +- **WHEN** a publish succeeds and a rule carries no `surfaced` and `evidence` of `{ "agent-files": 2, "tests": 3 }` +- **THEN** that rule's line SHALL state "5 times in conventions" and no reviews total + +#### Scenario: A rule without evidence shows no totals + +- **WHEN** a publish succeeds and a rule carries no `evidence` and no review comments +- **THEN** that rule's line SHALL give its title and tier and no totals + #### Scenario: The delete credential stays on disk - **WHEN** a publish succeeds and its record is written, with or without `--json` @@ -344,9 +398,11 @@ The counts SHALL be all-or-nothing. The CLI SHALL record them only when every wr ### Requirement: The report recipe drives publishing and deletion -The CLI SHALL embed a `report` recipe following the canonical recipe template. It SHALL instruct the agent to: run `taskless report` and read `missing`; supply each missing fact from its session, writing the facts to `.taskless/.tmp-report.json`; give every rule an `explain` of one or two sentences saying what goes wrong without the rule and what holding it buys, grounded in what the session saw and carrying no counts; count each rule's `surfaced` by the method the recipe states; give each cloud candidate an `estimatedViolations` from what it saw while mining; run `taskless report --from` that path; on `REPORT_INSUFFICIENT`, ask the user only what would let it author the rules or establish the facts the refusal names, then retry; on success, show the CLI's summary as printed and tell the user the report can be deleted after review if they ask; delete the facts file; and on a request to delete, run `taskless report --delete` with the record. The recipe SHALL embed the facts schema and SHALL state that tskl.es does not redact, so anything the agent cannot vouch for is to be withheld. +The CLI SHALL embed a `report` recipe following the canonical recipe template. It SHALL instruct the agent to: run `taskless report` and read `missing`; supply each missing fact from its session, writing the facts to `.taskless/.tmp-report.json`; give every rule an `explain` of one or two sentences saying what goes wrong without the rule and what holding it buys, grounded in what the session saw and carrying no counts; count each rule's `surfaced` by the method the recipe states; give every rule `sources` and `evidence`, counted by the method the recipe states; give each cloud candidate an `estimatedViolations` from what it saw while mining; run `taskless report --from` that path; on `REPORT_INSUFFICIENT`, ask the user only what would let it author the rules or establish the facts the refusal names, then retry; on success, show the CLI's summary as printed and tell the user the report can be deleted after review if they ask; delete the facts file; and on a request to delete, run `taskless report --delete` with the record. The recipe SHALL embed the facts schema and SHALL state that tskl.es does not redact, so anything the agent cannot vouch for is to be withheld. + +The counting method SHALL be a single procedure: a pull request counts toward a rule's `surfaced.prs` when it merged inside the window and at least one review comment on it asked for the change the rule enforces; a comment counts toward `surfaced.reviewComments` when a reader of that comment would say it asks for that change. Only review comments count toward `surfaced`: the recipe SHALL state that a fix-up commit, a `git blame` date, or a finding in the code today is not a review comment and SHALL NOT be counted toward either field. Every author counts: a comment from a bot account or from the pull request's own author SHALL count the same as a reviewer's, since each one spends cycles, tokens, time, or money. The recipe SHALL direct the agent to judge each comment by reading it, not by matching keywords. -The counting method SHALL be a single procedure: a pull request counts toward a rule's `surfaced.prs` when it merged inside the window and at least one review comment on it asked for the change the rule enforces; a comment counts toward `surfaced.reviewComments` when a reader of that comment would say it asks for that change. Only review comments count: the recipe SHALL state that a fix-up commit, a `git blame` date, or a finding in the code today is not a review comment and SHALL NOT be counted toward either field, since the facts have no field that would label it honestly. Every author counts: a comment from a bot account or from the pull request's own author SHALL count the same as a reviewer's, since each one spends cycles, tokens, time, or money. The recipe SHALL direct the agent to judge each comment by reading it, not by matching keywords. +The recipe SHALL define, for every kind `evidence` counts, what one call-out is: a line in an agent file (`agent-files`); a code comment such as TODO, FIXME, XXX or HACK asking for the pattern (`conventions`); a passage in the documentation, doc annotations such as `@deprecated` included (`docs`); a test that pins the behaviour (`tests`); a commit inside the window that made the fix (`commits`); and an issue, ticket, recorded decision, or comment on one that asked for it (`issues`). It SHALL state that code which merely follows a pattern is not a call-out, that commits are counted inside the window while every other kind is counted as found at publish, that `reviews` has no `evidence` count because `surfaced.reviewComments` is its count, that a listed `commits`, `issues`, `docs`, or `tests` needs its count, and that `sources` and `evidence` name kinds only, never a path, file name, commit, or issue. Whenever the agent restates the report's numbers in its own words, the recipe SHALL require it to keep counts the CLI measured in the code today, counts mined from review history over the window, and its own estimates for cloud candidates apart, and to say which is which. @@ -371,7 +427,14 @@ The recipe SHALL NOT ask the agent for any value input: tskl.es estimates review #### Scenario: Only review comments count as surfaced - **WHEN** an agent runs `taskless agent report` -- **THEN** the `## Counting surfaced` section SHALL state that a fix-up commit, a `git blame` date, and a finding in the code today are not review comments and do not count +- **THEN** the `## Counting evidence` section SHALL state that a fix-up commit, a `git blame` date, and a finding in the code today are not review comments and do not count toward `surfaced` + +#### Scenario: The recipe defines a call-out for every counted kind + +- **WHEN** an agent runs `taskless agent report` +- **THEN** the `## Counting evidence` section SHALL state what one call-out is for `agent-files`, `conventions`, `docs`, `tests`, `commits`, and `issues` +- **AND** SHALL state that code which merely follows a pattern is not counted +- **AND** SHALL state that `sources` and `evidence` name kinds, never paths, files, commits, or issues #### Scenario: The recipe asks for an explanation of every rule diff --git a/packages/cli/src/agent/onboard.md b/packages/cli/src/agent/onboard.md index acb6f73c..db8e5651 100644 --- a/packages/cli/src/agent/onboard.md +++ b/packages/cli/src/agent/onboard.md @@ -1,4 +1,4 @@ -# Topic: onboard (CLI v%(CLI_VERSION)s / topic v14) +# Topic: onboard (CLI v%(CLI_VERSION)s / topic v15) %(ONE_COMMAND_PER_CALL)s @@ -221,15 +221,31 @@ Note, for the report, how many pull requests and review comments raised each one in the window you mined: a pull request counts when it merged in the window and a review comment on it asked for the change, and each such comment counts. Bot and author comments count too. Judge -each comment by reading it, never by matching keywords. Evidence from -commit history chooses candidates; it never counts toward these -numbers, which count review comments alone. - -Count while you mine, not afterwards. As each comment joins a -candidate, note its pull request and comment under that candidate, -scoped to the rule you expect to write rather than a broader theme, -so the counts come from that list instead of a second read. When -`route` narrows a candidate, recount from the list. +each comment by reading it, never by matching keywords. These numbers +are `surfaced`, and they count review comments alone. + +Every other source is credited too, as its own kind of evidence, so a +rule chosen from commit history does not publish as raised in zero +reviews. Note each call-out, a place that states or asks for the +pattern, under its kind: + +- `commits`: a commit in the window that made the fix +- `issues`: an issue, ticket, recorded decision, or a comment on one +- `conventions`: a TODO, FIXME, XXX, or HACK comment asking for it +- `agent-files`: a line in CLAUDE.md, AGENTS.md, or similar +- `docs`: a passage in the docs, `@deprecated` and the like included +- `tests`: a test that pins the behaviour + +Code that merely follows a pattern is not a call-out. A commit never +counts toward `surfaced`, however clearly it shows the rule's worth. +Keep kinds and counts only: no paths, file names, commits, or issue +titles reach the report. + +Count while you mine, not afterwards. As each comment or call-out +joins a candidate, note it under that candidate with its kind, scoped +to the rule you expect to write rather than a broader theme, so the +counts come from that list instead of a second read. When `route` +narrows a candidate, recount from the list. ## Legacy and runtime candidates diff --git a/packages/cli/src/agent/report.md b/packages/cli/src/agent/report.md index 53332917..88f98eab 100644 --- a/packages/cli/src/agent/report.md +++ b/packages/cli/src/agent/report.md @@ -1,4 +1,4 @@ -# Topic: report (CLI v%(CLI_VERSION)s / topic v9) +# Topic: report (CLI v%(CLI_VERSION)s / topic v10) %(ONE_COMMAND_PER_CALL)s @@ -53,13 +53,15 @@ later and deletes one on request. for one you did not read. - `rules`: one entry per rule the draft listed, with a title a reader understands, an `explain` written as - [Explaining a rule](#explaining-a-rule) says, and its `surfaced` - counts, counted as [Counting surfaced](#counting-surfaced) says. + [Explaining a rule](#explaining-a-rule) says, its `surfaced` + counts, and its `sources` and `evidence`, all counted as + [Counting evidence](#counting-evidence) says. - `legacy`: rules you wrote into the repository's own linter, which the CLI cannot see. Each one needs an `explain` too. - `cloud`: runtime-shaped candidates you judged better solved by a - runtime rule and did not author. Count their `surfaced` the same - way, and give each one an `explain`. Give each one + runtime rule and did not author. Count their `surfaced`, + `sources` and `evidence` the same way, and give each one an + `explain`. Give each one `estimatedViolations`: how many instances of its pattern you estimate are in the code today, made as [Estimating violations](#estimating-violations) says. No rule has @@ -83,15 +85,15 @@ later and deletes one on request. asking. 4. **Show the result.** On success the CLI prints a short summary: the - rules, what they found in the code today, what review raised over - the window, the link, and when it expires. Relay it to the user as - ordinary chat text, NOT inside a code block or quote: a fenced block - makes the link unclickable and buries the report the whole session - led up to. Keep its content and order; do not add, drop, or reword - facts. Put the link on its own line as a plain URL or a Markdown - link, so the user can open it directly, and render the rules as a - list. The summary already says the report can be deleted on - request, so do not repeat it. + rules with each one's evidence totals, what they found in the code + today, what review raised over the window, the link, and when it + expires. Relay it to the user as ordinary chat text, NOT inside a + code block or quote: a fenced block makes the link unclickable and + buries the report the whole session led up to. Keep its content + and order; do not add, drop, or reword facts. Put the link on its + own line as a plain URL or a Markdown link, so the user can open it + directly, and render the rules as a list. The summary already says + the report can be deleted on request, so do not repeat it. If you restate any of the report's numbers in your own words, keep the kinds apart and label each: counts the CLI found **in the code @@ -130,17 +132,19 @@ who never saw the session understands what the rule protects. both. - Ground it in what you saw: the review comments, the code, the bug the pattern caused. Do not restate the title. -- No counts. The report already carries `violations`, `surfaced` and - `estimatedViolations`, labelled apart; a number in prose is a third - copy that can contradict them. +- No counts. The report already carries `violations`, `surfaced`, + `evidence` and `estimatedViolations`, labelled apart; a number in + prose is another copy that can contradict them. - No names, hosts, or contact details. The CLI screens `explain` like `title` and refuses the publish if its patterns match. -## Counting surfaced +## Counting evidence -Every agent counts `surfaced` the same way, so reports from different -sessions compare: +Every agent counts the same way, so reports from different sessions +compare. Use the list you kept while mining; do not re-read. + +`surfaced` counts review comments, and only those: - **`surfaced.prs`**: pull or merge requests merged inside `window` with at least one review comment asking for the change the rule @@ -156,10 +160,31 @@ sessions compare: keywords: "fixed" or "removed" in a reply says nothing about which rule it concerns. - A comment that asks for two rules' changes counts once for each. -- **Only review comments count.** A fix-up commit, a `git blame` - date, or a finding in the code today is not a review comment, - however well it shows what the rule is worth. The facts have no - field for it; leave it out rather than count it as review. +- **Only review comments count toward `surfaced`.** A fix-up commit, a + `git blame` date, or a finding in the code today is not a review + comment, however well it shows what the rule is worth. + +`sources` lists the kinds a rule came from, each once, and `evidence` +counts its call-outs per kind: places that state or ask for the +pattern. Code that merely follows a pattern is not a call-out. + +| Kind | One call-out is | +| ------------- | ------------------------------------------------------- | +| `reviews` | a review comment; its count is `surfaced`, not evidence | +| `commits` | a commit inside `window` that made the fix | +| `issues` | an issue, ticket, recorded decision, or comment on one | +| `agent-files` | a line in CLAUDE.md, AGENTS.md, or similar | +| `conventions` | a TODO, FIXME, XXX, or HACK comment asking for it | +| `docs` | a docs passage, `@deprecated` and the like included | +| `tests` | a test that pins the behaviour | + +- Commits count inside `window`; every other kind as you found it. +- A listed `commits`, `issues`, `docs`, or `tests` needs its count. + `agent-files` and `conventions` may be listed without one. +- Listing `reviews` needs at least one counted review comment. +- Give `sources` on every rule or on none. +- Kinds and counts only. No path, file name, commit, or issue title + goes in the facts. ## Estimating violations diff --git a/packages/cli/src/generated/report-v1.schema.json b/packages/cli/src/generated/report-v1.schema.json index d3cca1fc..86827510 100644 --- a/packages/cli/src/generated/report-v1.schema.json +++ b/packages/cli/src/generated/report-v1.schema.json @@ -83,7 +83,7 @@ }, "required": ["from", "to"], "additionalProperties": false, - "description": "The period the PR history was mined over." + "description": "The period the history (merged PRs and commits) was mined over." } }, "required": ["provider", "owner", "repo", "url", "window"], @@ -294,6 +294,67 @@ "additionalProperties": false, "description": "How often reviewers raised this rule's pattern in `scope.window`: what the rule saves reviewers. Requires `history`." }, + "sources": { + "minItems": 1, + "maxItems": 7, + "type": "array", + "items": { + "type": "string", + "enum": [ + "agent-files", + "conventions", + "reviews", + "commits", + "issues", + "docs", + "tests" + ] + }, + "description": "Where the rule came from, as kinds, never file names or paths. Each kind belongs to one of two groups. The conventions group, what the repository writes down: `agent-files`, the instructions the repository keeps for coding agents (such as AGENTS.md or CLAUDE.md); `conventions`, a pattern the repository's code already follows, or one a code comment such as TODO, FIXME, XXX or HACK asks for; `docs`, the repository's documentation, including doc annotations such as `@deprecated`, other than its agent files; `tests`, tests the repository already has that pin the behaviour. The reviews group, where a person or an agent pointed it out: `reviews`, review comments on merged PRs in `scope.window`, from the mined PR history; `commits`, a fix that recurs in the repository's commit history in `scope.window`; `issues`, issues, tickets or recorded decisions in a bug tracker or engineering-context tool, and their comments. List every kind that applies, each once. On every tier and state, Cloud candidates and legacy rules included. Give it on every rule or on none. A rule that lists `reviews` must carry `surfaced` with at least one review comment; a rule may carry `surfaced` without listing `reviews`, because reviewers can ask for a rule that came from elsewhere. A rule that lists `commits`, `issues`, `docs` or `tests` must carry its count in `evidence`; `agent-files` and `conventions` may be listed without one." + }, + "evidence": { + "type": "object", + "properties": { + "agent-files": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 1000000000, + "description": "Lines in the repository's agent files that state or ask for this rule's pattern, as the agent found them at publish." + }, + "conventions": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 1000000000, + "description": "Code comments (TODO, FIXME, XXX, HACK and the like) in the repository's code that ask for this rule's pattern, as the agent found them at publish. Code that merely follows the pattern is not counted." + }, + "commits": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 1000000000, + "description": "Commits in `scope.window` that made this rule's fix, from the repository's git history." + }, + "issues": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 1000000000, + "description": "Issues, tickets, recorded decisions or comments on them that asked for this rule's pattern, as the agent found them at publish." + }, + "docs": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 1000000000, + "description": "Passages in the repository's documentation, doc annotations such as `@deprecated` included, that state or ask for this rule's pattern, as the agent found them at publish." + }, + "tests": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 1000000000, + "description": "Tests in the repository that pin the behaviour this rule enforces, as the agent found them at publish." + } + }, + "additionalProperties": false, + "description": "How many explicit call-outs of this rule's pattern each of its `sources` holds: places that state or ask for it, never code that merely follows it. Keyed by kind, each a count of at least 1; a kind left out has none. Every kind but `reviews`, whose count is `surfaced.reviewComments`. Each key's kind must be listed in `sources`. Does not require `history`." + }, "tests": { "type": "object", "properties": { @@ -509,6 +570,11 @@ "prs": 96, "reviewComments": 142 }, + "sources": ["conventions", "reviews", "commits"], + "evidence": { + "conventions": 3, + "commits": 12 + }, "tests": { "status": "pass", "fail": { @@ -532,6 +598,11 @@ "prs": 63, "reviewComments": 88 }, + "sources": ["agent-files", "reviews", "issues"], + "evidence": { + "agent-files": 1, + "issues": 4 + }, "tests": { "status": "not-run", "fail": { diff --git a/packages/cli/src/report/assemble.ts b/packages/cli/src/report/assemble.ts index 5d6cf07d..fce95924 100644 --- a/packages/cli/src/report/assemble.ts +++ b/packages/cli/src/report/assemble.ts @@ -1,6 +1,7 @@ import { CLIError } from "../util/cli-error"; import type { ReportDraft } from "./draft"; import type { ReportFacts } from "../schemas/report-facts"; +import { evidenceIssues } from "./evidence"; import { REDACTION_POLICY_VERSION, hasHits, @@ -75,6 +76,33 @@ function measuredFindings( return draft.findings; } +/** + * A rule's `sources` and `evidence`, exactly as the agent gave them, with no + * key for a field it left out: a report without them is the older format, + * and must stay byte-for-byte the payload it was. + */ +function evidenceOf( + facts: Pick +): Pick { + return { + ...(facts.sources === undefined ? {} : { sources: facts.sources }), + ...(facts.evidence === undefined ? {} : { evidence: facts.evidence }), + }; +} + +/** + * Refuse what tskl.es refuses across `sources`, `evidence` and `surfaced`, + * naming the rule and the field. The service checks the same, after the + * fixtures are redacted and the payload sent; here the agent hears which fact + * to fix while it still has the session that produced it. + */ +function checkEvidence(candidates: Candidate[]): void { + const issues = evidenceIssues(candidates.map(({ rule }) => rule)); + if (issues.length > 0) { + throw invalid(`rule evidence does not add up: ${issues.join("; ")}`); + } +} + /** Pair each draft rule with its facts, and add the legacy and cloud rules. */ function mergeRules( draft: ReportDraft, @@ -117,6 +145,7 @@ function mergeRules( ? { violations: local.violations, files: local.files ?? 0 } : {}), surfaced: ruleFacts.surfaced, + ...evidenceOf(ruleFacts), }, status: local.tests.status, fail: local.tests.fail, @@ -135,6 +164,7 @@ function mergeRules( language: legacy.language, state: "written", surfaced: legacy.surfaced, + ...evidenceOf(legacy), }, // The CLI cannot run another linter's rule, so it does not claim it // passed. Agent-written legacy rules still count toward the gate. @@ -158,6 +188,7 @@ function mergeRules( state: "candidate", estimatedViolations: cloud.estimatedViolations, surfaced: cloud.surfaced, + ...evidenceOf(cloud), }, status: "not-run", fail: cloud.fail ?? null, @@ -320,6 +351,7 @@ export function assembleReport( ): ReportV1 { const findings = measuredFindings(draft, facts); const candidates = mergeRules(draft, facts, findings !== undefined); + checkEvidence(candidates); enforceGate(candidates); const scope = buildScope(draft, facts); diff --git a/packages/cli/src/report/evidence.ts b/packages/cli/src/report/evidence.ts new file mode 100644 index 00000000..187febfc --- /dev/null +++ b/packages/cli/src/report/evidence.ts @@ -0,0 +1,133 @@ +/** + * Where a rule came from, and how often each source called for it. + * + * ReportV1 names seven kinds of source and counts explicit call-outs for six + * of them (`reviews` is counted by `surfaced.reviewComments`). The kinds are + * the vendored schema's; what this file adds is what the schema cannot say: + * the group a reader sees each kind under, which kinds must carry a count, and + * the checks tskl.es applies across `sources`, `evidence` and `surfaced`, + * repeated here so the agent hears which fact to fix rather than a service + * error after publish. + * + * A test holds the kinds below equal to the vendored schema's, so a schema + * refresh that adds a kind fails there until it is given a group. + */ + +/** `conventions`: what the repository writes down. `reviews`: where someone pointed it out. */ +export type SourceGroup = "conventions" | "reviews"; + +export const SOURCE_GROUPS = { + "agent-files": "conventions", + conventions: "conventions", + docs: "conventions", + tests: "conventions", + reviews: "reviews", + commits: "reviews", + issues: "reviews", +} as const satisfies Record; + +export type RuleSource = keyof typeof SOURCE_GROUPS; + +/** Every kind, in the schema's order. */ +export const RULE_SOURCES = [ + "agent-files", + "conventions", + "reviews", + "commits", + "issues", + "docs", + "tests", +] as const satisfies readonly RuleSource[]; + +/** The kinds `evidence` counts: every kind but `reviews`. */ +export type EvidenceSource = Exclude; + +export const EVIDENCE_SOURCES = RULE_SOURCES.filter( + (source): source is EvidenceSource => source !== "reviews" +); + +/** + * Kinds that exist only to be counted. `agent-files` and `conventions` are + * exempt, since reports from before `evidence` list them without counts. + */ +const COUNTED_SOURCES: readonly EvidenceSource[] = [ + "commits", + "issues", + "docs", + "tests", +]; + +export type Evidence = Partial>; + +interface RuleEvidence { + id: string; + sources?: RuleSource[]; + evidence?: Evidence; + surfaced?: { reviewComments: number }; +} + +/** + * Every way a set of rules breaks tskl.es's rules for `sources` and + * `evidence`, one message per failure, naming the rule and the field. + */ +export function evidenceIssues(rules: readonly RuleEvidence[]): string[] { + const issues: string[] = []; + + const withSources = rules.filter((rule) => rule.sources !== undefined); + if (withSources.length > 0 && withSources.length < rules.length) { + const without = rules + .filter((rule) => rule.sources === undefined) + .map((rule) => rule.id); + issues.push( + `sources must be given on every rule or on none; these rules have none: ${without.join(", ")}` + ); + } + + for (const { id, sources, evidence, surfaced } of rules) { + if (sources === undefined) { + if (evidence !== undefined) { + issues.push(`${id}: evidence requires sources`); + } + continue; + } + const duplicated = sources.filter( + (source, index) => sources.indexOf(source) !== index + ); + if (duplicated.length > 0) { + issues.push(`${id}.sources lists ${duplicated.join(", ")} twice`); + } + if (sources.includes("reviews") && !surfaced?.reviewComments) { + issues.push( + `${id}.sources lists reviews, so surfaced.reviewComments must be at least 1` + ); + } + for (const source of COUNTED_SOURCES) { + if (sources.includes(source) && evidence?.[source] === undefined) { + issues.push( + `${id}.sources lists ${source}, so it needs evidence.${source}` + ); + } + } + for (const source of EVIDENCE_SOURCES) { + if (evidence?.[source] !== undefined && !sources.includes(source)) { + issues.push(`${id}.evidence.${source} needs sources to list ${source}`); + } + } + } + return issues; +} + +/** A rule's two totals, as the report page shows them. */ +export function groupTotals(rule: { + surfaced?: { reviewComments: number }; + evidence?: Evidence; +}): Record { + const totals: Record = { + reviews: rule.surfaced?.reviewComments ?? 0, + conventions: 0, + }; + for (const source of EVIDENCE_SOURCES) { + totals[SOURCE_GROUPS[source]] += rule.evidence?.[source] ?? 0; + } + return totals; +} diff --git a/packages/cli/src/report/summary.ts b/packages/cli/src/report/summary.ts index cff01a6c..6c07468e 100644 --- a/packages/cli/src/report/summary.ts +++ b/packages/cli/src/report/summary.ts @@ -1,3 +1,4 @@ +import { groupTotals, type SourceGroup } from "./evidence"; import type { ReportRule, ReportV1 } from "./types"; /** @@ -14,6 +15,10 @@ import type { ReportRule, ReportV1 } from "./types"; * formula here is a number that drifts from the page without anyone noticing. * The summary states the two kinds of count it has, labelled apart: what the * CLI found in the code today, and what review raised over the window. + * + * Each rule also carries its two evidence totals, summed and worded the way + * the report page sums and words them, so the terminal and the page it links + * to never disagree about a rule. */ function tierLabel(rule: ReportRule): string { @@ -34,6 +39,22 @@ function plural(count: number, one: string, many: string): string { return `${String(count)} ${count === 1 ? one : many}`; } +/** + * "6 times in reviews, 1 time in conventions": each group above zero, the + * larger first, reviews first on a tie. Empty when the rule has neither. + */ +function evidenceSuffix(rule: ReportRule): string { + const totals = groupTotals(rule); + const order: SourceGroup[] = + totals.conventions > totals.reviews + ? ["conventions", "reviews"] + : ["reviews", "conventions"]; + const parts = order + .filter((group) => totals[group] > 0) + .map((group) => `${plural(totals[group], "time", "times")} in ${group}`); + return parts.length === 0 ? "" : `: ${parts.join(", ")}`; +} + /** What review raised over the window, summed across every rule. */ function surfacedLine(report: ReportV1): string { let prs = 0; @@ -59,7 +80,9 @@ export function renderSummary(options: { "Taskless report published", "", `Rules (${String(report.rules.length)}):`, - ...report.rules.map((rule) => ` - ${rule.title} (${tierLabel(rule)})`), + ...report.rules.map( + (rule) => ` - ${rule.title} (${tierLabel(rule)})${evidenceSuffix(rule)}` + ), "", ...(findings === undefined ? [] diff --git a/packages/cli/src/report/types.ts b/packages/cli/src/report/types.ts index 0942183b..1df40091 100644 --- a/packages/cli/src/report/types.ts +++ b/packages/cli/src/report/types.ts @@ -1,3 +1,5 @@ +import type { Evidence, RuleSource } from "./evidence"; + /** * The ReportV1 payload, as this CLI builds it. * @@ -53,6 +55,10 @@ export interface ReportRule { */ estimatedViolations?: number; surfaced?: Surfaced; + /** The kinds of source the rule came from, on every rule or on none. */ + sources?: RuleSource[]; + /** Explicit call-outs per listed kind; `reviews` is counted by `surfaced`. */ + evidence?: Evidence; tests: ReportTests; } diff --git a/packages/cli/src/schemas/report-facts.ts b/packages/cli/src/schemas/report-facts.ts index 11fc19e9..b035428d 100644 --- a/packages/cli/src/schemas/report-facts.ts +++ b/packages/cli/src/schemas/report-facts.ts @@ -1,5 +1,7 @@ import { z } from "zod"; +import { EVIDENCE_SOURCES, RULE_SOURCES } from "../report/evidence"; + /** * What the agent knows that the CLI cannot: the facts `taskless report --from` * reads. @@ -38,6 +40,27 @@ const surfaced = z .strict() .describe("How often review raised this rule's pattern over the window"); +const sources = z + .array(z.enum(RULE_SOURCES)) + .min(1) + .describe( + "The kinds of source this rule came from, each once, never paths or file names. Conventions: `agent-files` (AGENTS.md, CLAUDE.md and the like), `conventions` (a pattern the code follows, or a TODO/FIXME-style comment asking for it), `docs` (documentation, `@deprecated` included), `tests` (tests that pin the behaviour). Reviews: `reviews` (review comments, counted in `surfaced`), `commits` (commits in the window that made the fix), `issues` (issues, tickets, recorded decisions, and comments on them). Give it on every rule or on none" + ); + +const evidence = z + .object( + Object.fromEntries( + EVIDENCE_SOURCES.map((source) => [ + source, + z.number().int().min(1).optional(), + ]) + ) as Record<(typeof EVIDENCE_SOURCES)[number], z.ZodOptional> + ) + .strict() + .describe( + "How many explicit call-outs of this rule's pattern each listed kind holds: places that state or ask for it, never code that merely follows it. Each at least 1; leave a kind out for none. `reviews` has no key: `surfaced.reviewComments` is its count. Every key must be listed in `sources`, and `commits`, `issues`, `docs` and `tests` need a count when listed" + ); + const outcome = { title: z .string() @@ -58,6 +81,8 @@ const outcome = { "Why this rule is worth holding in this codebase, in one or two sentences: what goes wrong without it, and what holding it buys" ), surfaced, + sources: sources.optional(), + evidence: evidence.optional(), }; const fixture = z diff --git a/packages/cli/test/onboard.test.ts b/packages/cli/test/onboard.test.ts index 94ee649d..cf2105a7 100644 --- a/packages/cli/test/onboard.test.ts +++ b/packages/cli/test/onboard.test.ts @@ -422,8 +422,8 @@ describe("onboard recipe asks one question and finishes on its own", () => { : stdout.slice(start, start + title.length + next); } - it("is topic v14", () => { - expect(stdout).toMatch(/^# Topic: onboard {5}\(CLI v.+ \/ topic v14\)/m); + it("is topic v15", () => { + expect(stdout).toMatch(/^# Topic: onboard {5}\(CLI v.+ \/ topic v15\)/m); }); // v10 (#554): a worktree holds only commits, and `init` commits nothing, so @@ -469,7 +469,7 @@ describe("onboard recipe asks one question and finishes on its own", () => { expect(scan).toContain("recount from the list"); }); - it("mines commit history for recurring fixes, but never counts it as surfaced", async () => { + it("mines commit history for recurring fixes, counted as commits and never as surfaced", async () => { const mine = step("Mine every source you can reach"); expect(mine).toContain("**Commit history**"); expect(mine.replaceAll(/\s+/g, " ")).toContain( @@ -479,8 +479,31 @@ describe("onboard recipe asks one question and finishes on its own", () => { const choosing = await section("choosing-candidates"); const scan = choosing.replaceAll(/\s+/g, " "); expect(scan).toContain( - "Evidence from commit history chooses candidates; it never counts toward these numbers" + "- `commits`: a commit in the window that made the fix" ); + expect(scan).toContain("A commit never counts toward `surfaced`"); + }); + + // taskless/cli#681: rules chosen from commits, docs, or tests published as + // raised in zero reviews, because only review comments had a field. + it("tallies every source's call-outs by kind while mining, never paths", async () => { + const choosing = await section("choosing-candidates"); + const scan = choosing.replaceAll(/\s+/g, " "); + for (const kind of [ + "commits", + "issues", + "conventions", + "agent-files", + "docs", + "tests", + ]) { + expect(scan).toContain(`- \`${kind}\`:`); + } + expect(scan).toContain( + "Code that merely follows a pattern is not a call-out" + ); + expect(scan).toContain("no paths, file names, commits, or issue titles"); + expect(scan).toContain("note it under that candidate with its kind"); }); it("asks the one question, set apart, through a question tool when there is one", () => { diff --git a/packages/cli/test/report.test.ts b/packages/cli/test/report.test.ts index 1ee065f2..4c07e421 100644 --- a/packages/cli/test/report.test.ts +++ b/packages/cli/test/report.test.ts @@ -19,6 +19,11 @@ import { stringify } from "yaml"; import reportSchemaDocument from "../src/generated/report-v1.schema.json"; import { assembleReport } from "../src/report/assemble"; +import { + EVIDENCE_SOURCES, + RULE_SOURCES, + SOURCE_GROUPS, +} from "../src/report/evidence"; import type { LocalRule, ReportDraft } from "../src/report/draft"; import { reportFactsSchema, @@ -93,6 +98,19 @@ function factsFor( }); } +/** The message of the `REPORT_INVALID` a run throws, or empty when it does not. */ +function messageOf(run: () => unknown): string { + try { + run(); + } catch (error) { + if (error instanceof CLIError && error.code === "REPORT_INVALID") { + return error.message; + } + throw error; + } + return ""; +} + function codeOf(run: () => unknown): string | undefined { try { run(); @@ -114,6 +132,68 @@ describe("the vendored ReportV1 schema", () => { }); }); +/** Whether the facts accept one rule carrying `extra`. */ +function factsAcceptRule(extra: Record): boolean { + return reportFactsSchema.safeParse({ + ...factsFor([]), + rules: [ + { + id: "a", + title: "Rule a", + explain: "Holding it keeps review on design.", + surfaced: { prs: 0, reviewComments: 0 }, + ...extra, + }, + ], + }).success; +} + +describe("the kinds of rule evidence", () => { + const ruleSchema = ( + reportSchemaDocument as unknown as { + properties: { + rules: { + items: { + properties: { + sources: { items: { enum: string[] } }; + evidence: { properties: Record }; + }; + }; + }; + }; + } + ).properties.rules.items.properties; + + // A schema refresh that adds a kind fails here until the kind has a group: + // otherwise its counts would be summed into neither total. + it("are the vendored schema's, each with a group", () => { + expect([...RULE_SOURCES]).toEqual(ruleSchema.sources.items.enum); + expect(Object.keys(SOURCE_GROUPS).toSorted()).toEqual( + [...RULE_SOURCES].toSorted() + ); + expect(Object.keys(ruleSchema.evidence.properties).toSorted()).toEqual( + [...EVIDENCE_SOURCES].toSorted() + ); + expect( + RULE_SOURCES.filter((s) => !EVIDENCE_SOURCES.includes(s as never)) + ).toEqual(["reviews"]); + }); + + it("are the only ones the facts accept, each counted at least once", () => { + expect( + factsAcceptRule({ sources: ["commits"], evidence: { commits: 2 } }) + ).toBe(true); + expect(factsAcceptRule({ sources: ["trackers"] })).toBe(false); + expect(factsAcceptRule({ sources: [] })).toBe(false); + expect(factsAcceptRule({ sources: ["docs"], evidence: { docs: 0 } })).toBe( + false + ); + expect( + factsAcceptRule({ sources: ["reviews"], evidence: { reviews: 3 } }) + ).toBe(false); + }); +}); + describe("the vendored delete contract", () => { // `deletePublishedReport` is written against this contract. A refresh that // changes it must fail here rather than leave the client calling a method or @@ -179,6 +259,157 @@ describe("assembleReport", () => { }); }); + it("carries sources and evidence unchanged, on every tier", () => { + const evidence = { + sources: ["commits", "docs"], + evidence: { commits: 4, docs: 1 }, + }; + const facts = factsFor(three, { + rules: three.map((id) => ({ + id, + title: `Rule ${id}`, + explain: "Holding it keeps review on design instead of style.", + surfaced: { prs: 0, reviewComments: 0 }, + ...evidence, + })), + legacy: [ + { + id: "no-moment", + title: "No moment", + explain: "Moment is frozen and heavy.", + surfaced: { prs: 1, reviewComments: 2 }, + tool: "eslint", + language: "TypeScript", + sources: ["reviews", "agent-files"], + }, + ], + cloud: [ + { + id: "idempotent-jobs", + title: "Jobs are idempotent", + explain: "A retried job charges twice.", + surfaced: { prs: 0, reviewComments: 0 }, + language: "TypeScript", + estimatedViolations: 3, + sources: ["issues"], + evidence: { issues: 2 }, + }, + ], + }); + const report = assembleReport( + draftOf(three.map((id) => localRule(id))), + facts + ); + expect(reportSchemaIssues(report)).toEqual([]); + expect(report.rules[0]).toMatchObject(evidence); + const legacy = report.rules.find((rule) => rule.id === "no-moment"); + expect(legacy?.sources).toEqual(["reviews", "agent-files"]); + expect(legacy).not.toHaveProperty("evidence"); + expect( + report.rules.find((rule) => rule.id === "idempotent-jobs") + ).toMatchObject({ sources: ["issues"], evidence: { issues: 2 } }); + }); + + it("adds neither field when the facts give neither", () => { + const report = assembleReport( + draftOf(three.map((id) => localRule(id))), + factsFor(three) + ); + for (const rule of report.rules) { + expect(rule).not.toHaveProperty("sources"); + expect(rule).not.toHaveProperty("evidence"); + } + }); + + describe("refuses evidence tskl.es would refuse, naming rule and field", () => { + type Extra = { sources?: string[]; evidence?: Record }; + function withEvidence(perRule: Extra[], reviewComments = 0): () => unknown { + return () => + assembleReport( + draftOf(three.map((id) => localRule(id))), + factsFor(three, { + rules: three.map((id, index) => ({ + id, + title: `Rule ${id}`, + explain: "Holding it keeps review on design instead of style.", + surfaced: { prs: reviewComments, reviewComments }, + ...perRule[index], + })), + }) + ); + } + const conventions: Extra = { sources: ["conventions"] }; + + it("sources on some rules but not all", () => { + expect(messageOf(withEvidence([conventions, conventions, {}]))).toContain( + "every rule or on none; these rules have none: c" + ); + }); + + it("a kind listed twice", () => { + expect( + messageOf( + withEvidence([ + { sources: ["conventions", "conventions"] }, + conventions, + conventions, + ]) + ) + ).toContain("a.sources lists conventions twice"); + }); + + it("reviews without a review comment", () => { + expect( + messageOf( + withEvidence([{ sources: ["reviews"] }, conventions, conventions]) + ) + ).toContain("a.sources lists reviews, so surfaced.reviewComments"); + expect( + codeOf( + withEvidence([{ sources: ["reviews"] }, conventions, conventions], 2) + ) + ).toBeUndefined(); + }); + + it("a counted kind without its count", () => { + expect( + messageOf( + withEvidence([{ sources: ["commits"] }, conventions, conventions]) + ) + ).toContain("a.sources lists commits, so it needs evidence.commits"); + }); + + it("a count without its kind", () => { + expect( + messageOf( + withEvidence([ + { sources: ["conventions"], evidence: { tests: 2 } }, + conventions, + conventions, + ]) + ) + ).toContain("a.evidence.tests needs sources to list tests"); + }); + + it("evidence without sources", () => { + expect( + messageOf(withEvidence([{ evidence: { docs: 1 } }, {}, {}])) + ).toContain("a: evidence requires sources"); + }); + + it("but not the original kinds listed without counts", () => { + expect( + codeOf( + withEvidence([ + { sources: ["agent-files", "conventions"] }, + conventions, + { sources: ["agent-files"] }, + ]) + ) + ).toBeUndefined(); + }); + }); + it("carries every rule's explanation, on every tier", () => { const facts = factsFor(["a"], { legacy: ["l1", "l2"].map((id) => ({ @@ -594,6 +825,21 @@ describe("scanText", () => { }); }); +/** One rule's facts, carrying `extra` beside its review counts. */ +function ruleWithEvidence( + id: string, + reviewComments: number, + extra: Record +) { + return { + id, + title: `Rule ${id}`, + explain: "Holding it keeps review on design instead of style.", + surfaced: { prs: reviewComments, reviewComments }, + ...extra, + }; +} + function summaryOf(draft: ReportDraft): string { return renderSummary({ report: assembleReport(draft, factsFor(["a", "b", "c"])), @@ -617,7 +863,9 @@ describe("renderSummary", () => { record: "2026-10-13_142200.json", invocation: "npx @taskless/cli", }); - expect(summary).toContain("Rule a (Taskless ast-grep rule)"); + expect(summary).toContain( + "Rule a (Taskless ast-grep rule): 9 times in reviews\n" + ); expect(summary).toContain("https://tskl.es/r/abc"); expect(summary).toContain("can be deleted after review"); expect(summary).not.toContain("delete/"); @@ -641,6 +889,46 @@ describe("renderSummary", () => { expect(summary).not.toMatch(/month/i); }); + it("gives each rule its evidence totals, larger first", () => { + const report = assembleReport( + draftOf(["a", "b", "c", "d"].map((id) => localRule(id))), + factsFor([], { + rules: [ + ruleWithEvidence("a", 2, { + sources: ["reviews", "commits", "docs"], + evidence: { commits: 4, docs: 1 }, + }), + ruleWithEvidence("b", 0, { + sources: ["agent-files", "tests"], + evidence: { "agent-files": 2, tests: 3 }, + }), + ruleWithEvidence("c", 0, { sources: ["conventions"] }), + ruleWithEvidence("d", 1, { + sources: ["reviews", "agent-files"], + evidence: { "agent-files": 1 }, + }), + ], + }) + ); + const summary = renderSummary({ + report, + url: "https://tskl.es/r/abc", + expiresAt: "2026-11-12T00:00:00Z", + record: "2026-10-13_142200.json", + invocation: "npx @taskless/cli", + }); + expect(summary).toContain( + "Rule a (Taskless ast-grep rule): 6 times in reviews, 1 time in conventions\n" + ); + expect(summary).toContain( + "Rule b (Taskless ast-grep rule): 5 times in conventions\n" + ); + expect(summary).toContain("Rule c (Taskless ast-grep rule)\n"); + expect(summary).toContain( + "Rule d (Taskless ast-grep rule): 1 time in reviews, 1 time in conventions\n" + ); + }); + it("states no problem count it does not have", () => { const summary = summaryOf( draftOf(["a", "b", "c"].map((id) => localRule(id))) @@ -1128,10 +1416,12 @@ describe("taskless report", () => { // The counting and estimating guidance is on demand (#672); --full prints it. const { stdout, exitCode } = await run(["agent", "report", "--full"]); expect(exitCode).toBe(0); - expect(stdout).toMatch(/^# Topic: report {5}\(CLI v.+ \/ topic v9\)/m); + expect(stdout).toMatch(/^# Topic: report {5}\(CLI v.+ \/ topic v10\)/m); const flat = stdout.replaceAll(/\s+/g, " "); - const counting = flat.slice(flat.indexOf("## Counting surfaced")); - expect(counting).toContain("**Only review comments count.**"); + const counting = flat.slice(flat.indexOf("## Counting evidence")); + expect(counting).toContain( + "**Only review comments count toward `surfaced`.**" + ); expect(counting).toContain( "A fix-up commit, a `git blame` date, or a finding in the code today is not a review comment" ); @@ -1151,6 +1441,28 @@ describe("taskless report", () => { expect(show).toContain("which are **your estimate**"); }); + // taskless/cli#681: evidence beyond review comments had no field. + it("defines one call-out for every kind evidence counts, never paths", async () => { + const { stdout, exitCode } = await run([ + "agent", + "report", + "--section", + "counting-evidence", + ]); + expect(exitCode).toBe(0); + for (const kind of RULE_SOURCES) { + expect(stdout).toMatch(new RegExp(`^\\| \`${kind}\` +\\| \\S`, "m")); + } + const flat = stdout.replaceAll(/\s+/g, " "); + expect(flat).toContain( + "Code that merely follows a pattern is not a call-out" + ); + expect(flat).toContain("Commits count inside `window`"); + expect(flat).toContain( + "No path, file name, commit, or issue title goes in the facts" + ); + }); + // An agent grepped for the escape helper, found where escaping already // happens, and published zero for a pattern fixed four times. Zero stays a // valid answer, but only after a search for the problem itself.