You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As the service that runs these plugins, I want to ask a source for every pair in a version range, so a range nobody has read yet (an older version a user is still on) is filled in without special code per plugin.
Subtasks
core: Pairs(versions, from, to) → the consecutive pairs in a range; Backfill(ctx, source, from, to) → reads each version once and extracts each pair in order
Respect each source's limits: one fetch at a time per host, a delay between fetches, stop on 429 and say where it stopped
Idempotent: re-running a range gives the same events (fingerprints)
Test kit: backfill a saved source with four versions in memory
Acceptance criteria
Backfilling a range twice gives the same events
A range that hits a rate limit stops cleanly and resumes from where it stopped
Progress: core/versions.Pairs returns the consecutive pairs of a version list (core/versions/versions.go). Backfill itself now runs in the hosted ingestion service's worker: it reads the versions a range needs, compares each pair in order, and spaces out its fetches per host.
Not in this repo yet: Backfill(ctx, source, from, to) as a library helper, stop-on-429-and-resume, and the four-version test-kit backfill. These are needed only if backfill should also run outside the hosted service (for example in a contributor's test). Otherwise this story can be closed as covered by the worker. That's a call for a maintainer.
Activity
adssib commentedon Nov 21, 2026
Progress:
core/versions.Pairsreturns the consecutive pairs of a version list (core/versions/versions.go). Backfill itself now runs in the hosted ingestion service's worker: it reads the versions a range needs, compares each pair in order, and spaces out its fetches per host.Not in this repo yet:
Backfill(ctx, source, from, to)as a library helper, stop-on-429-and-resume, and the four-version test-kit backfill. These are needed only if backfill should also run outside the hosted service (for example in a contributor's test). Otherwise this story can be closed as covered by the worker. That's a call for a maintainer.