Ai2 reports Gated DeltaNet generalizes to ~10x longer sequences on StateBench

Ai2 says a flexible Gated DeltaNet handled sequences about 10 times longer than its training data on StateBench's hardest level.

In a separate test, Ai2 trained LLMs directly on StateBench and evaluated them on longer inputs. It reports that hybrid models with Gated DeltaNets were near-perfect on the first three levels, while transformers degraded as input length grew and could not solve general finite-state tasks.

Ai2 also reports a new proof that Gated DeltaNet with non-negative eigenvalues can solve some periodic tasks under fixed-precision arithmetic with rounding. The proof covers only those periodic tasks under those conditions.