length-generalization

[NeurIPS 2025] Official code for "Tropical Attention: Neural Algorithmic Reasoning for Combinatorial Algorithms"

This work provides extensive empirical results on training LMs to count. We find that while traditional RNNs trivially achieve inductive counting, Transformers have to rely on positional embeddings to count out-of-domain. Modern RNNs (e.g. rwkv, mamba) also largely underperform traditional RNNs in generalizing counting inductively.

Mamba Modulation: On the Length Generalization of Mamba Models

Modular Arithmetic Challenge. Neural induction of exact (a x b) mod p through abacus embeddings, algorithmic scratchpads and grokking, for the SAIR Foundation competition.

Kinship reasoning without a neural network: a sound, set-valued model that derives its rules from opaque symbols. CLUTRR, all six datasets.

Chromatin loop extrusion as a sequence model's routing operator: 36 runs vs attention, BiLSTM and a one-arm ablation, 8x length and 300-sequence tests, learned CTCF orientation.

A modular, config-driven PyTorch framework for conducting empirical positional encoding ablation experiments on decoder-only transformers (~51M parameters) using Google Colab T4 GPUs

An original convergence-reasoning LLM architecture: iterated reasoning emerges from end-answer labels alone and generalizes to 2.3× training depth. PyTorch, ~333K params, fully reproducible.

Mechanistic interpretability of a small transformer that adds numbers at lengths it was never trained on: how a carry travels, why the answer drifts, and where the fix runs out.

Small reproduction and extension of Zoology (Arora et al.): MQAR recall at unseen lengths and key-query distances for Attention, Based and BaseConv, with seed variation.

Code and experiments for studying length generalization in recurrent Transformers on permutation state-tracking tasks.

Transformer built from PyTorch primitives, plus a CPU-scale replication of Kazemnejad et al. (NeurIPS 2023): how NoPE, sinusoidal, RoPE and ALiBi generalize to longer sequences. Includes a KV cache and attention-head analysis.

An exact testbed for when an RL agent should check its own work under a budget: process vs. outcome rewards, observation encoders, and length generalization.

Dilution attention: a parameter-free causal attention operator that divides each query's bid by the demand its key has already absorbed. Fused Triton kernels, and a controlled study where 209M models with no position encoding retrieve at 8x their training context.