Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
9ace212
Tests: warnings are errors, strict pytest config, block real network …
nikolaos-paximadakis-eea Nov 19, 2026
3de8229
ruff: wider rule set (RUF, S, PL, DTZ, PTH, FBT, PT, SIM, C4, RET) an…
nikolaos-paximadakis-eea Nov 19, 2026
670c7f6
generate_resources: drop the trailing 'return None' (ruff PLR1711/RET…
nikolaos-paximadakis-eea Nov 19, 2026
1df3fe0
Regenerate resources (no trailing 'return None')
nikolaos-paximadakis-eea Nov 19, 2026
62aecb5
Name HTTP status codes with http.HTTPStatus (ruff PLR2004)
nikolaos-paximadakis-eea Nov 19, 2026
faee336
Fix: TooManyRequestsError.retry_after returned inf/nan for such Retry…
nikolaos-paximadakis-eea Nov 19, 2026
d08fa3d
Package: fix the remaining ruff findings by hand
nikolaos-paximadakis-eea Nov 19, 2026
7df56b2
Scripts, tools, tests: fix the remaining ruff findings
nikolaos-paximadakis-eea Nov 19, 2026
1fcf8c0
generate_resources: only filter out None query params when one can be…
nikolaos-paximadakis-eea Nov 19, 2026
b4c7b79
Regenerate resources (no None filter for all-required query params)
nikolaos-paximadakis-eea Nov 19, 2026
4080a15
mypy: check tests, scripts and tools too; stricter error codes
nikolaos-paximadakis-eea Nov 19, 2026
8fef957
Tests: fix the type errors mypy finds with tests in scope
nikolaos-paximadakis-eea Nov 19, 2026
d9b9384
Tests: fully annotated; remove the temporary mypy override for tests.*
nikolaos-paximadakis-eea Nov 19, 2026
e30196f
slots=True on the frozen dataclasses (handles, RetryPolicy, Match, _S…
nikolaos-paximadakis-eea Nov 19, 2026
5741267
pre-commit hooks and fuller CI: format check, strict mypy on the whol…
nikolaos-paximadakis-eea Nov 19, 2026
b68ad0c
jobs.JobError.__init__: explicit '-> None'
nikolaos-paximadakis-eea Nov 19, 2026
debff42
Revert "slots=True on the frozen dataclasses (handles, RetryPolicy, M…
nikolaos-paximadakis-eea Nov 19, 2026
e191796
Tests: tighter no-network guard (connect_ex, UDP sendto, DNS; any loo…
nikolaos-paximadakis-eea Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,40 @@ jobs:
- name: Lint (ruff)
run: uv run ruff check .

- name: Type check (mypy)
run: uv run mypy reportnet_api
- name: Format (ruff)
run: uv run ruff format --check .

- name: Test (pytest)
- name: Type check (mypy, strict; package, tests, scripts, tools)
run: uv run mypy

- name: Test (pytest; warnings are errors, no network)
run: uv run pytest

generated:
# Generated code must match its generator and spec exactly.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Install uv
uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
enable-cache: true

- name: Install dependencies (exactly as locked)
run: uv sync --all-groups --locked

- name: Regenerate models, resources and reference docs
run: |
scripts/regen_models.sh
scripts/regen_resources.sh
scripts/regen_docs.sh

- name: Fail if anything changed
run: git diff --exit-code

package:
runs-on: ubuntu-latest
steps:
Expand Down
22 changes: 22 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Run before each commit: `uvx pre-commit install` once, then it runs on `git commit`.
# Run on everything by hand: `uvx pre-commit run --all-files`.
# The hooks call the project's own locked tools through uv, so they match CI exactly.
repos:
- repo: local
hooks:
- id: ruff-check
name: ruff check
entry: uv run ruff check --fix
language: system
types: [python]
- id: ruff-format
name: ruff format
entry: uv run ruff format
language: system
types: [python]
- id: mypy
name: mypy (strict, whole project)
entry: uv run mypy
language: system
types: [python]
pass_filenames: false
12 changes: 7 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,15 @@ Thanks for helping. This is a small library, so the process is light.
## Set up

```
uv sync --all-groups # install deps + dev tools
uv run ruff check . # lint
uv run mypy reportnet_api
uv run pytest # no network needed: HTTP is mocked
uv sync --all-groups # install deps + dev tools
uv run ruff check . # lint
uv run ruff format --check .
uv run mypy # strict: package, tests, scripts, tools
uv run pytest # warnings are errors; real network access fails the test
uvx pre-commit install # optional: run ruff and mypy on every commit
```

All three must pass; CI runs the same commands on Python 3.11 to 3.14 and also builds the
All of these must pass; CI runs the same commands on Python 3.11 to 3.14 and also builds the
wheel. The docs site is built with `uv run --group docs mkdocs build --strict`.

## Generated files: do not hand-edit
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,8 +217,8 @@ the code here is MIT-licensed (see `LICENSE`).

```
uv sync --all-groups # install deps + dev tools
uv run ruff check . # lint
uv run mypy reportnet_api
uv run ruff check . && uv run ruff format --check .
uv run mypy # strict, whole project
uv run pytest
```

Expand Down
2 changes: 1 addition & 1 deletion docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ repository's workflow, so no API token is created or stored.
## Each release

1. Update `version` in `pyproject.toml` and the entry in `CHANGELOG.md`.
2. `uv run ruff check . && uv run mypy reportnet_api && uv run pytest` must pass (CI runs
2. `uv run ruff check . && uv run ruff format --check . && uv run mypy && uv run pytest` must pass (CI runs
them on Python 3.11 to 3.14).
3. **Rehearse:** run the *Release* workflow by hand (Actions tab → Release → Run workflow).
It uploads to TestPyPI. Install from there in a clean environment and import it:
Expand Down
37 changes: 34 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -67,20 +67,51 @@ target-version = "py311"
line-length = 100

[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B"]
select = [
"E", "F", "I", "B", "UP", "RUF", "S", "PL", "DTZ", "PTH", "FBT", "PT", "SIM", "C4", "RET",
]
# ANN is left out on purpose: mypy strict already enforces annotations.
ignore = [
"PLR0913", # resource methods mirror the endpoint's parameters
"PLC0415", # pandas is imported lazily so that it stays optional
"FBT001", # boolean parameters are public API; changing them is a breaking change
"FBT002", # same
"PT018", # splitting compound asserts is churn; pytest already shows both sides
"SIM117", # nested `with` blocks read fine; merging them is style only
"PLR0911", # complexity counts: splitting long, linear functions risks behaviour
"PLR0912", # changes for no correctness gain; revisit case by case
"PLR0915",
]

[tool.ruff.lint.per-file-ignores]
"reportnet_api/models/generated.py" = ["E501", "UP"]
"reportnet_api/models/generated.py" = ["ALL"]
"reportnet_api/resources/*.py" = ["E501"]
"tests/**" = ["S101", "S105", "S106", "PLR2004", "FBT003"]

[tool.mypy]
python_version = "3.11"
strict = true
warn_unused_ignores = true
files = ["reportnet_api", "tests", "scripts", "tools"]
warn_unreachable = true
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool", "possibly-undefined"]

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = ["--strict-markers", "--strict-config", "-ra"]
filterwarnings = ["error"]
xfail_strict = true

[[tool.mypy.overrides]]
module = ["pandas", "pandas.*"]
ignore_missing_imports = true

[[tool.mypy.overrides]]
# numpy's stubs use Python 3.12 syntax, which python_version 3.11 cannot parse.
module = ["numpy", "numpy.*"]
follow_imports = "skip"
follow_imports_for_stubs = true

[[tool.mypy.overrides]]
# Only scripts/make_logo.py uses these, installed ad hoc with `uv run --with`.
module = ["cairosvg", "fontTools.*"]
ignore_missing_imports = true
20 changes: 13 additions & 7 deletions reportnet_api/_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@
from dataclasses import dataclass
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
from types import MappingProxyType
from pathlib import Path
from types import FrameType, MappingProxyType
from typing import Any, BinaryIO

import httpx
Expand Down Expand Up @@ -39,11 +40,15 @@

def caller_stacklevel() -> int:
"""The `stacklevel` that points a warning at the first frame outside this package."""
package_dir = os.path.dirname(os.path.abspath(__file__)) + os.sep
level, frame = 1, sys._getframe(1)
while frame is not None and os.path.abspath(frame.f_code.co_filename).startswith(package_dir):
# os.path on purpose: Path.resolve() follows symlinks, co_filename does not.
package_dir = os.path.dirname(os.path.abspath(__file__)) + os.sep # noqa: PTH100, PTH120
level = 1
frame: FrameType | None = sys._getframe(1)
while frame is not None and os.path.abspath( # noqa: PTH100 - same reason
frame.f_code.co_filename
).startswith(package_dir):
level += 1
frame = frame.f_back # type: ignore[assignment]
frame = frame.f_back
return level


Expand Down Expand Up @@ -218,5 +223,6 @@ def stream_upload_file(
httpx reads from the file handle lazily while sending the request, so
the file is never fully buffered in memory.
"""
with open(file, "rb") as fh:
yield {field_name: (os.path.basename(str(file)), fh)}
path = Path(file)
with path.open("rb") as fh:
yield {field_name: (path.name, fh)}
35 changes: 21 additions & 14 deletions reportnet_api/_exceptions.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
from __future__ import annotations

import json
import math
import re
from http import HTTPStatus

import httpx

Expand Down Expand Up @@ -59,11 +61,12 @@ class TooManyRequestsError(ReportnetAPIError):

@property
def retry_after(self) -> float | None:
"""Seconds from a numeric `Retry-After` header, or None if absent or not a number."""
"""Seconds from a numeric `Retry-After` header, or None if absent or not a finite number."""
try:
return float(self.response.headers["Retry-After"])
seconds = float(self.response.headers["Retry-After"])
except (KeyError, ValueError):
return None
return seconds if math.isfinite(seconds) else None


class ServerError(ReportnetAPIError):
Expand Down Expand Up @@ -107,14 +110,14 @@ class ServerError(ReportnetAPIError):

def _hint(response: httpx.Response) -> str | None:
"""Short tip for 401/403 (tailored to the Authorization scheme used), 423 and 429."""
if response.status_code == 423:
if response.status_code == HTTPStatus.LOCKED:
return _LOCKED_HINT
if response.status_code == 429:
if response.status_code == HTTPStatus.TOO_MANY_REQUESTS:
return _TOO_MANY_HINT
if response.status_code not in (401, 403):
if response.status_code not in (HTTPStatus.UNAUTHORIZED, HTTPStatus.FORBIDDEN):
return None
scheme = response.request.headers.get("Authorization", "").split(" ", 1)[0].lower()
if response.status_code == 401:
if response.status_code == HTTPStatus.UNAUTHORIZED:
return _APIKEY_401_HINT if scheme == "apikey" else _BEARER_401_HINT
return _APIKEY_HINT if scheme == "apikey" else _BEARER_FORBIDDEN_HINT

Expand Down Expand Up @@ -166,7 +169,9 @@ def _explain(err: ReportnetAPIError) -> list[str]:
lines = [f"{request.method} {path} failed with HTTP {status}."]
if reason:
lines.append(f"The server said: {reason}")
if status == 500 and _is_rejected_credential_500(err.response_body):
if status == HTTPStatus.INTERNAL_SERVER_ERROR and _is_rejected_credential_500(
err.response_body
):
lines.append(
"Likely cause: the credential was rejected upstream (a revoked or expired ApiKey "
"shows up as a 500 like this, even for plain reads), or it was sent with the "
Expand All @@ -177,37 +182,39 @@ def _explain(err: ReportnetAPIError) -> list[str]:
"A Bearer token sent as an ApiKey fails the same way: Bearer tokens are bare "
'UUIDs here, not JWTs, so connect(token, scheme="Bearer").'
)
elif status == 500 and "accessReferenceEntity" in err.response_body:
elif (
status == HTTPStatus.INTERNAL_SERVER_ERROR and "accessReferenceEntity" in err.response_body
):
lines.append(
"Likely cause: the dataflow id does not exist, or is not visible to this "
"credential (the gateway reports this as a 500, not a 404)."
)
lines.append("Try: check the dataflow id (list_dataflows) and that it is not a dataset id.")
elif status == 423:
elif status == HTTPStatus.LOCKED:
lines.append(
"Likely cause: another operation (often an import job still QUEUED or "
"IN_PROGRESS) holds a lock on this dataset or dataflow; the server's own text "
"can be vague."
)
lines.append("Try: wait for that job to finish (see wait_for_job), then repeat the call.")
elif status == 403 and scheme == "bearer" and "/etl" in path:
elif status == HTTPStatus.FORBIDDEN and scheme == "bearer" and "/etl" in path:
lines.append(
"Likely cause: the ETL endpoints are gated separately from dataflow roles; a "
"Bearer user can get 403 here even as Custodian/Steward."
)
lines.append("Try: use a dataflow ApiKey for ETL calls, or ask the dataflow admins.")
elif status == 403 and scheme == "apikey":
elif status == HTTPStatus.FORBIDDEN and scheme == "apikey":
lines.append(
"Likely cause: ApiKeys are Reporter-level and work only for the dataflow they "
"were issued for, so Custodian/Steward-only routes refuse them."
)
lines.append("Try: use a Bearer token of a user holding that role on this dataflow.")
elif (hint := _hint(err.response)) is not None:
lines.append(f"Hint: {hint}.")
elif status >= 500:
elif status >= HTTPStatus.INTERNAL_SERVER_ERROR:
lines.append("Likely cause: a server-side failure; the request itself may be fine.")
lines.append("Try: check the server's message above, then retry later.")
elif status == 404:
elif status == HTTPStatus.NOT_FOUND:
lines.append("Likely cause: the id in the request does not exist for this credential.")
lines.append("Try: check dataflow and dataset ids are not mixed up.")
elif status in (400, 422):
Expand All @@ -222,7 +229,7 @@ def error_for_response(response: httpx.Response) -> ReportnetAPIError | None:
return None

exc_cls = _STATUS_EXCEPTIONS.get(response.status_code)
if exc_cls is None and response.status_code >= 500:
if exc_cls is None and response.status_code >= HTTPStatus.INTERNAL_SERVER_ERROR:
exc_cls = ServerError
if exc_cls is None:
exc_cls = ReportnetAPIError
Expand Down
5 changes: 3 additions & 2 deletions reportnet_api/async_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import asyncio
import os
from pathlib import Path
from typing import Any

import httpx
Expand Down Expand Up @@ -36,7 +37,7 @@ def __init__(
base_url: str,
token: TokenLike,
*,
token_scheme: str = "Bearer",
token_scheme: str = "Bearer", # noqa: S107 - an auth scheme name, not a secret
timeout: float | httpx.Timeout | None = DEFAULT_TIMEOUT,
retry: RetryPolicy | None = None,
) -> None:
Expand Down Expand Up @@ -99,7 +100,7 @@ async def download(
if not response.is_success:
await response.aread()
self._core.handle_response(response)
with open(dest, "wb") as fh:
with Path(dest).open("wb") as fh:
async for chunk in response.aiter_bytes():
fh.write(chunk)

Expand Down
11 changes: 7 additions & 4 deletions reportnet_api/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import sys
import tempfile
from collections.abc import Sequence
from pathlib import Path
from typing import Any, TextIO

import httpx
Expand Down Expand Up @@ -165,15 +166,17 @@ def main(argv: Sequence[str] | None = None, out: TextIO | None = None) -> int:
elif args.command == "read":
if args.out:
# Write next to the target and rename, so a failure leaves no empty file.
folder = os.path.dirname(os.path.abspath(args.out))
# abspath normalises '..' without touching the disk; Path.absolute() does not.
folder = os.path.dirname(os.path.abspath(args.out)) # noqa: PTH100, PTH120
fd, tmp = tempfile.mkstemp(dir=folder, suffix=".part")
try:
with os.fdopen(fd, "w", newline="", encoding="utf-8") as fh:
_read(client, args, fh)
os.chmod(tmp, 0o666 & ~_UMASK) # mkstemp makes 0600; use the usual mode
os.replace(tmp, args.out)
# mkstemp makes 0600; use the usual mode
Path(tmp).chmod(0o666 & ~_UMASK)
Path(tmp).replace(args.out)
except BaseException:
os.unlink(tmp)
Path(tmp).unlink()
raise
else:
_read(client, args, out)
Expand Down
5 changes: 3 additions & 2 deletions reportnet_api/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import os
import time
from pathlib import Path
from typing import Any

import httpx
Expand Down Expand Up @@ -36,7 +37,7 @@ def __init__(
base_url: str,
token: TokenLike,
*,
token_scheme: str = "Bearer",
token_scheme: str = "Bearer", # noqa: S107 - an auth scheme name, not a secret
timeout: float | httpx.Timeout | None = DEFAULT_TIMEOUT,
retry: RetryPolicy | None = None,
) -> None:
Expand Down Expand Up @@ -100,7 +101,7 @@ def download(
if not response.is_success:
response.read()
self._core.handle_response(response)
with open(dest, "wb") as fh:
with Path(dest).open("wb") as fh:
for chunk in response.iter_bytes():
fh.write(chunk)

Expand Down
Loading