Skip to content

Rewrite bindings for snapshots, transaction DBs and typed errors - #1

Merged
arkkln merged 7 commits into
mainfrom
lane/k-rdbpy
Nov 3, 2026
Merged

arkkln merged 7 commits into
mainfrom
lane/k-rdbpy

Conversation

@arkkln

@arkkln arkkln commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Lane K of task-161. A clean rewrite of the bindings for RocksDB 6.29.5, shaped for what the virtuals backend and the kv contract need. Version 0.3.0 (API break).

What changed

  • One core module (_core.pyx + _rocksdb.pxd) replaces the old 3.5k-line pyx, 19 pxds, the hpp wrappers and the 1465-line stub. Diff: +4572 / -8985.
  • Surface: Options, DB, TransactionDB (pessimistic), OptimisticTransactionDB, Transaction (get, get_for_update, put, delete, iterator, set_snapshot, commit, rollback), Snapshot, Iterator (seek, seek_for_prev, first, last, bounds, prefix, reverse, keys_only, next_batch(n)), WriteBatch, destroy_db.
  • snapshot= is honoured on every read path: db get, multi_get, db iterator, txn get, get_for_update, txn iterator. Before, no read path used a snapshot at all.
  • Typed errors under one base RocksDBError: Busy, Deadlock (a Busy subtype), TimedOut, TryAgain, Expired, LockLimit, plus Corruption, IOError and InvalidArgument classes. Missing keys return None.
  • Lock timeout and deadlock detection options on TransactionDB and per transaction.
  • The GIL is released around every blocking call: open, close, reads, writes, lock waits, commits, seeks, batch iteration, flush, compaction.
  • Txn iterators clamp the transaction's own pending writes to the iterator's bounds. RocksDB 6.29 does not apply the lower bound to them.
  • DB.close() releases leftover snapshots and raises RuntimeError while another thread has a call in flight, instead of segfaulting.
  • Deleted: python callbacks, merge operators, backup, plain table, memtable factories, column families, save points, 2PC. Nothing outside rdbpy and virtuals used them.
  • Build: build-rocksdb-linux.sh wrote the rpath as RIGIN, because make ate $O. It now sets $ORIGIN on the RocksDB link only. Wheels were unaffected (auditwheel rewrites rpaths).
  • CI: test.yml now runs on pull requests across ubuntu, macos intel and macos arm, with a dependency cache.

Why

The virtuals rocksdb backend could not meet the kv contract on the old bindings. There were no snapshot reads, no GetForUpdate and no optimistic DB, and every commit failure was a generic exception classified by message text.

Testing

  • 127 tests pass in about 12s. They cover snapshots on every read path, TimedOut within lock_timeout, optimistic Busy at commit, deadlock detection, iterator seeks, prefix, bounds and reverse, txn iterator bounds over pending writes, GIL release, close during in-flight calls, and child-object lifetimes in subprocesses.
  • The virtuals full suite (1960 passed) and lane C's conformance suite run on this build (Rewrite rocksdb backend to meet the kv contract virtuals#2).

Benchmarks (red, median of 3, raw rows vary about ±30% run to run)

case before after
point put 664,280 /s 675,381 /s
point get 1,254,436 /s 1,327,322 /s
scan forward 2,694,370 /s 3,177,274 /s
write batch (1 batch of N) 1,675,638 /s 1,591,036 /s
write batch commits (10 puts) 155,706 /s 148,741 /s
txn commit (get + put) 240,915 /s 239,449 /s

Thread rules

Child objects (Snapshot, Iterator, Transaction, WriteBatch) are used by one thread at a time. Do not close or free one from a thread while another thread uses it. Documented in the README and the stub.

CI

@arkkln
arkkln marked this pull request as ready for review November 3, 2026 14:36
@arkkln
arkkln merged commit 4423dd4 into main Nov 3, 2026
8 checks passed
@arkkln
arkkln deleted the lane/k-rdbpy branch November 4, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant