Skip to content

feat(telemetry): add default-on usage analytics and detached delivery - #76

Open
littleDogWang wants to merge 5 commits into
mainfrom
codex/cli-analytics
Open

littleDogWang wants to merge 5 commits into
mainfrom
codex/cli-analytics

Conversation

@littleDogWang

@littleDogWang littleDogWang commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add usage analytics for Recipes, init and project creation across shell and MCP. When a valid collector is configured, capture defaults to enabled with a first-use stderr notice and saved/runtime opt-out controls. Observe project creation at the shared call boundary so later setup failures do not overwrite a successful creation result.

Type of change

  • New feature
  • Behavior change to an existing command
  • Breaking change to the developing telemetry config/status contract

Public-contract impact

  • Changes JSON fields and telemetry help; documented in docs/automation.md and the changelog.
  • Remove telemetryConsent from config and consent from status. enabled now reports only the saved preference; captureActive combines that preference with runtime opt-out and a valid local collector URL. transportConfigured does not prove remote availability.
    • Before: {"enabled":true,"consent":"unknown","transportConfigured":false,"captureActive":false}
    • After: {"enabled":true,"transportConfigured":false,"captureActive":false}
  • Keep business JSON framing, errors and exit codes unchanged. No new error codes.
  • Use CLI-local installation and per-operation IDs, with validated 32-hex project App IDs only after the actual target is known. Earlier events are not backfilled. No website identity/session inputs or account identity lookups.
  • Record one cli_project_create_completed event per shared createProject call, including authentication/token refresh/retry and excluding later feature/config/scaffold work. Reuse, pre-call validation failures and dry-run do not create this event. Success counts successful calls, not a complete project ledger.
  • Replace parent-process HTTP with one lazy detached sender per operation. Parent/child each queue at most 20 events; each event is limited to 4 KiB and each POST contains one event. Parent local drain is bounded to 100 ms; requests and EOF tail each have three-second budgets. No retry, replay, redirect, persistent queue or delivery guarantee. At most two children per parent, asynchronously reaped.
  • CLI retains startup policy; MCP reads preference once per tool call and skips capture on read failure. An already-started operation can finish after disable.

Test plan

  • Full go test ./... -count=1.
  • Targeted telemetry/notice race tests, plus the strengthened parent-exit regression under the race detector.
  • Creation success followed by feature/query failure, App ID retention, invalid App IDs, auth failure, refresh/retry timing, Recipe/Quickstart and shell/MCP paths.
  • First notice, quiet mode, opt-outs, JSON/MCP stdout, invalid live configuration, slow pipe, queue/child bounds, per-event size rejection, streaming beyond three seconds, HTTP/EOF budgets and detached completion after parent exit.
  • Build, go vet, gofmt, error-code audit and generated command-doc checks. make lint passed its available checks.
  • golangci-lint is not installed locally; Makefile skips that executable.
  • Linux amd64 and Windows amd64 cross-compilation.
  • Actual Linux/Windows process-detachment/handle/reaping runtime acceptance; local runtime evidence is macOS only.
  • Fresh CLI -> actual local SSP BFF -> mock PostHog HTTP receiver: 10 events, including creation/App ID, default capture, MCP isolation and opt-out. Forwarded payloads omit uuid/event_id.

Documentation

  • Changelog, README, telemetry policy, automation contract, generated help, agent summary, final design and verification record updated.
  • No new error codes or engineering/release-process changes.

Security checklist

  • Excludes credentials, certificates, user content, project names, paths, complete commands and raw errors. App IDs are deliberately included and validated.
  • Pipe payloads use a strict allowlist. Child uses no business authentication or config/session reads; stdout/stderr are null devices.
  • Installation identity and notice marker use the private config directory and restrictive permissions; notice recording never rewrites the saved disable preference.
  • No new unsafe import.

Additional notes

The embedded collector URL remains empty. Merge does not enable production capture; no deployment or real PostHog ingestion has been verified. Actual PostHog acceptance of omitted uuid and query-visible generated UUIDs must be verified before activation.

Companion collector PR: https://github.com/AgoraLab/ssp-service/pull/62.

Add anonymous core-flow events, optional webpage attribution, Agent classification and bounded collector delivery. Honor live MCP opt-out and Retry-After, classify EOF as failure, and emit each init validation stage once. Include privacy documentation and regression coverage.
@digitallysavvy

Copy link
Copy Markdown
Contributor

@littleDogWang,

Overall, I like the approach here, especially the effort to limit what's collected and exclude sensitive data.

A few things I'd like us to revisit before merging:

  1. Consent: Do we need to require an explicit opt-in for all of these events? Most of what we're collecting is basic product usage and operational metrics. I'd like to separate aggregate telemetry from persistent user/installation tracking and confirm what we can collect without interrupting the developer workflow.
  2. Web attribution: The browser-to-CLI tracking is useful for understanding conversions, but I'd treat this separately from general CLI telemetry. We should verify that we're respecting website consent and avoid passing persistent tracking IDs through shell commands or agent prompts.
  3. Performance: The five-second telemetry drain concerns me. We shouldn't potentially delay a CLI command or MCP tool call by several seconds just to deliver analytics.
  4. Collection scope: The events and metadata look reasonable. I like that we're capturing init failures and stages without sending raw errors, project names, credentials, or source code.

I'd also like to verify the BFF/PostHog privacy configuration before we enable collection in a release.

My preference is to keep basic operational analytics as frictionless as legally permissible, while giving developers clear controls and treating cross-session attribution separately.

@littleDogWang littleDogWang changed the title feat(telemetry): add opt-in usage analytics for recipes and init feat(telemetry): add default-on usage analytics and detached delivery Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants