[lvaroqui]

Export the spans and logs emitted by the benchmarked code to the CodSpeed runner, so that they can be shown next to each benchmark.

Add a telemetry mode, selected when the runner sets CODSPEED_RUNNER_MODE=telemetry. It runs each benchmark once, like the analysis modes, and:

• sends the start and end of each round to the runner (add_benchmark_timestamps). The start and stop commands of a benchmark reach the runner tens of microseconds away from the benchmarked code, so the rounds give tighter bounds to find the benchmark of each span and log;
• skips the perf map warm-up, which would emit telemetry outside the round.

When CODSPEED_OTEL_ENDPOINT is set, the plugin exports over OTLP/HTTP, with batch processors flushed when pytest exits:

• the spans of the global tracer provider. If the benchmarked code already installed an SDK TracerProvider, the export is added to it;
• the stdlib logging records, filtered by CODSPEED_TELEMETRY_FILTER (info by default, e.g. info,my_package=debug). The root logger level is lowered for the run so that the selected records reach the handler. The records of the exporter's own loggers (opentelemetry, urllib3, requests) are never exported.

Spans get codspeed.monotonic_start_ns / codspeed.monotonic_end_ns and log records codspeed.monotonic_time_ns, read from the instrument-hooks clock that the benchmark markers use.

The OpenTelemetry SDK and exporter come with a new telemetry extra (>=1.28.0). Without the extra, the plugin warns and exports nothing.

Things worth a look

• Trace ids. The plugin calls random.seed(0) before each benchmark, and the SDK's default id generator draws from the global random. Every benchmark got the same trace and span ids, and the SDK's draws changed the random numbers the benchmarks get. The ids now come from a generator with its own random.Random().
• No global LoggerProvider. Only logging records are exported, through a logging.Filter on the SDK LoggingHandler. The SDK log processor API changed between 1.28 and 1.45, so stamping records emitted through the OpenTelemetry logs API would need version-specific code. LoggingHandler is deprecated in recent SDK versions; its warning is silenced.
• service.name is the name of the pytest rootdir.

Results file

The second commit writes results/<pid>.json under the runner only in walltime mode. In the other modes, the file held no benchmarks and failed to parse as walltime results on the CodSpeed side. The other integrations already write it only in walltime mode.

Example and verification

tests/benchmarks/test_bench_telemetry.py adds benchmarks that emit spans and logs, and the benchmarks workflow now installs the telemetry extra. In simulation mode they show the cost of the calls without an exporter. Running them in telemetry mode in CI waits for the runner's telemetry mode to be released.

Checked end to end with a local build of the runner (codspeed run -m telemetry --skip-upload): it collects 3 spans and 3 logs from the example, the debug record is filtered out, each span and log falls within the round markers of its benchmark, and each benchmark has its own trace id.

Closes COD-4148

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The changes affect telemetry collection for pytest-codspeed benchmark runs.

The PR appears safe to merge, with a non-blocking fix recommended for exporter child loggers.

Findings

1.  Exporter logs bypass the filter ▶

Fix with agent prompt

### Issue 1
src/pytest_codspeed/telemetry.py:106-110
With `CODSPEED_TELEMETRY_FILTER=info,urllib3.connectionpool=debug`, `level_of()` returns the child directive before reaching the forced `urllib3=off` entry. This lets the exporter's own HTTP logs through. Exporting those messages can create more messages to export, causing repeated background requests and unrelated telemetry.

Reject `EXPORTER_LOGGERS` and their descendants before applying user directives.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a telemetry mode that exports benchmark spans and stdlib logs over OTLP/HTTP.

• Telemetry-mode benchmarks send timed spans and filtered logs to the CodSpeed runner.
• The runner’s results file is written only in walltime mode.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Runner selects telemetry mode] --> B[Pytest configures exporters]
  B --> C[Run benchmark once]
  C --> D[Send round timestamps]
  C --> E[Stamp spans and logs]
  E --> F[Batch OTLP HTTP export]
  F --> G[Runner collects telemetry]
  H[Pytest exits] --> I[Flush and stop exporters]
  I --> G

Loading

_{Reviews (1) · Last reviewed commit: "fix: write the results file only in wall..." · Reviewed by Greptile}

[greptile-apps]

Exporter logs bypass the filter

With CODSPEED_TELEMETRY_FILTER=info,urllib3.connectionpool=debug, level_of() returns the child directive before reaching the forced urllib3=off entry. This lets the exporter's own HTTP logs through. Exporting those messages can create more messages to export, causing repeated background requests and unrelated telemetry.

Reject EXPORTER_LOGGERS and their descendants before applying user directives.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/pytest_codspeed/telemetry.py
Line: 106-110

Comment:
**Exporter logs bypass the filter**

With `CODSPEED_TELEMETRY_FILTER=info,urllib3.connectionpool=debug`, `level_of()` returns the child directive before reaching the forced `urllib3=off` entry. This lets the exporter's own HTTP logs through. Exporting those messages can create more messages to export, causing repeated background requests and unrelated telemetry.

Reject `EXPORTER_LOGGERS` and their descendants before applying user directives.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[codspeed]

Merging this PR will degrade performance by 11.47%

3 warnings, 1 note

⚠️ 5 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 4 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 12 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

ℹ️ 32 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 5 improved benchmarks
❌ 7 regressed benchmarks
✅ 265 untouched benchmarks
🆕 9 new benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Walltime	test_tcp_connection[wikipedia.org-443]	14.2 ms	26.2 ms	-45.63%
❌	Walltime	test_multiprocessing_map[100000]	103 ms	151.8 ms	-32.19%
❌	Memory	test_multiprocessing_map[10]	3.3 MB	4.6 MB	-28.93%
❌	Memory	test_multiprocessing_map[100]	3.1 MB	3.8 MB	-19.04%
❌	Memory	test_multiprocessing_map[10000]	3.9 MB	4.5 MB	-14.32%
❌	Walltime	test_sum_squares_slow	308.8 µs	344.6 µs	-10.39%
❌	Walltime	test_is_complete[board0]	1.7 µs	1.9 µs	-9.88%
⚡	Walltime	test_solve_power_sum[13-2]	8.1 µs	7.2 µs	+13.2%
⚡	Memory	test_multiprocessing_map[100000]	7.3 MB	6.5 MB	+11.58%
⚡	Walltime	test_generate_sum_of_subsets_soln[nums0-9]	28.5 µs	25.9 µs	+9.99%
⚡	Walltime	test_tcp_connection[1.1.1.1-53]	948.7 µs	878 µs	+8.05%
⚡	Walltime	test_make_highpass	5.8 µs	5.5 µs	+5.23%
🆕	Walltime	test_no_telemetry	N/A	35.3 µs	N/A
🆕	Walltime	test_nested_spans_with_logs	N/A	268.4 µs	N/A
🆕	Walltime	test_log_without_span	N/A	37.2 µs	N/A
🆕	CPU Simulation	test_no_telemetry	N/A	189.5 µs	N/A
🆕	CPU Simulation	test_log_without_span	N/A	203.6 µs	N/A
🆕	CPU Simulation	test_nested_spans_with_logs	N/A	890.1 µs	N/A
🆕	Memory	test_no_telemetry	N/A	0 B	N/A
🆕	Memory	test_log_without_span	N/A	0 B	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-4148-featpython-export-the-benchmark-telemetry-from-pytest (17e57f4) with master (15d46b4)}