Popular repositories Loading

1. carry-the-one carry-the-one Public

   Research log on length generalization in transformer addition: post + source code

   Python

2. rope-aliasing rope-aliasing Public

   Why rotary position embeddings stop extrapolating: a head-by-head study in a small addition transformer

   Python

3. sparse-branching-net-report sparse-branching-net-report Public

   Report: toy-scale study of Mixture-of-Experts self-organization (arithmetic, language model, per-token switching, shared-pool adaptive depth)

   HTML

4. adder-mech-interp adder-mech-interp Public

   Mechanistic interpretability of a small transformer that adds numbers at lengths it was never trained on: how a carry travels, why the answer drifts, and where the fix runs out.

   Python

5. flowstate-grill flowstate-grill Public

   A one-station grill game where every cook time lands on the beat.

   HTML

6. tokenization-addition tokenization-addition Public

   How digit tokenization decides whether a small transformer can add beyond its training length

   Python