Current research on artificial neural networks follows four lines that are documented in public papers: working out how much data and compute a model needs, replacing the attention layer with cheaper sequence layers, reading the internals of trained models, and testing different layer designs. None of the four has replaced the standard transformer, and each paper states its own limits. This article describes what each one claims and what remains open.
Key takeaways
- Scaling-law studies fit how loss falls as model size, data, and compute grow; a 2022 study found that parameters and training tokens should grow together.
- Mamba, a state-space model, is reported to run with about five times the throughput of transformers and to scale linearly with sequence length.
- Sparse autoencoders are used to split a model's internal activations into more interpretable features, tested so far on specific tasks.
- Kolmogorov-Arnold networks put learnable functions on connections instead of fixed activations; the authors call them promising, not a replacement.
- A research direction is a hypothesis with early evidence, not a product forecast.
Scaling laws: how much data per parameter
In January 2020, Jared Kaplan and colleagues reported that the loss of language models follows a power law in model size, dataset size, and training compute, across more than seven orders of magnitude. They also reported that width and depth matter little within a wide range, and concluded that compute-efficient training uses very large models on a modest amount of data, stopping well before convergence.
A March 2022 study from DeepMind reached a different allocation. The authors trained over 400 language models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. Their conclusion, stated in the paper's abstract, was that for every doubling of model size, the number of training tokens should also double. They tested the rule by training Chinchilla, a 70 billion parameter model, with the same compute budget as the 280 billion parameter Gopher but four times more data. In the paper, Chinchilla outperformed Gopher on a broad range of downstream tasks and reached 67.5 percent average accuracy on the MMLU benchmark.
The open question is how far such fits extend. They are empirical curves measured on language models in a given range, and two respected studies already disagree about the best split of a fixed budget.
State-space models: a different way to read long inputs
A transformer compares every position in the input with every other position. The original 2017 paper lists the cost of a self-attention layer as proportional to the square of the sequence length times the representation size. Long inputs therefore become expensive.
Albert Gu and Tri Dao proposed Mamba in December 2023, a selective state-space model that carries a compact running state along the sequence. Their abstract reports about five times the throughput of transformers, linear scaling with sequence length, and gains on real data for sequences up to a million tokens. The same abstract reports that the three billion parameter Mamba model outperforms transformers of the same size and matches transformers twice its size in pretraining and downstream evaluation. These are the authors' own comparisons, and a compressed state cannot keep every detail, which is the trade-off such designs must show they handle.
Interpretability: reading what is inside
Individual neurons in a trained network often respond to several unrelated things, a property called polysemanticity. One proposed cause is superposition, where a network stores more features than it has neurons by spreading them across directions in activation space.
In September 2023, Hoagy Cunningham and coauthors used sparse autoencoders to reconstruct a language model's activations and recover those directions. By automated measures, the features were more interpretable than those from alternative methods, and on the indirect object identification task the method located the features causally responsible for the model's behavior more precisely than earlier decompositions. The authors conclude that superposition may be resolvable with a scalable, unsupervised method. The evidence covers one kind of model and one kind of task, so a general reading of large networks remains a goal.
New layer designs: Kolmogorov-Arnold networks
A standard network places a fixed activation function on each node and learns weights on the connections. In April 2024, Ziming Liu and coauthors proposed the reverse: learnable univariate functions, parametrized as splines, on the connections, inspired by the Kolmogorov-Arnold representation theorem. They claim that smaller such networks can match or beat much larger multilayer perceptrons in data fitting and in solving partial differential equations, and that the networks are easier to interpret. The paper was accepted to ICLR 2025. The claims concern fitting and equation solving; the abstract does not report results for language or image models.
The four lines side by side
| Direction | Question asked | Reported finding | Open issue |
|---|---|---|---|
| Scaling laws | How should a fixed compute budget be split? | Model size and tokens should grow together (2022) | Earlier study recommended a different split |
| State-space models | Can long inputs be handled at lower cost? | About five times the throughput; linear in length | Compressed state versus full attention |
| Sparse autoencoders | Can internal features be made readable? | More interpretable features in a language model | Tested on limited tasks |
| Kolmogorov-Arnold networks | Can learnable edge functions fit better? | Smaller networks match larger ones on fitting tasks | No results shown for language or images |
A worked scaling example (illustrative)
The numbers here are invented to show the arithmetic of the equal-scaling rule and are not measurements. Suppose a team trains a model with 10 billion parameters on 200 billion tokens, a ratio of 20 tokens per parameter. If the model is doubled to 20 billion parameters, the rule calls for doubling the data to 400 billion tokens, which keeps the ratio at 20. Training compute then grows by roughly a factor of four, because both parameters and tokens doubled. This is why the data supply, not only hardware, limits how far a model can grow under that rule.
How to read a new architecture claim
- Is the comparison against a model of the same size and the same training data?
- Does the paper test the task the reader cares about, or a different one?
- Has anyone other than the authors reproduced the result?
- Does the abstract state limits, or only gains?
FAQ
Will a successor replace the transformer?
The papers above do not claim that. Mamba reports advantages in speed and length, and the Kolmogorov-Arnold paper calls its networks promising alternatives to multilayer perceptrons. Whether either displaces current designs depends on results that independent groups have yet to establish.
Are scaling laws guaranteed to continue?
No. They are curves fitted to measured training runs. The 2020 and 2022 studies fit different ranges and reached different advice on splitting a budget.
What does polysemantic mean?
A neuron is polysemantic when it responds to several semantically distinct inputs. That makes it hard to say what the neuron stands for, and it is the problem sparse autoencoders try to address.
Where can the original papers be read?
All the lines cited here have open preprints on arXiv, a preprint server. Preprints are not always peer reviewed, so the publication venue is worth checking.
Related reading
- Why problems in AI research are hard to spot
- Fundamentals of artificial neural networks
- Alternatives to neural networks in machine learning
- How AI is used in scientific research
More in AI Basics.
Last verified: October 2026
Sources
- Scaling Laws for Neural Language Models, Kaplan J et al., arXiv:2001.08361, 2020
- Training Compute-Optimal Large Language Models, Hoffmann J et al., arXiv:2203.15556, 2022
- Attention Is All You Need, Vaswani A et al., arXiv:1706.03762, 2017
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu A, Dao T, arXiv:2312.00752, 2023, revised 2024
- Sparse Autoencoders Find Highly Interpretable Features in Language Models, Cunningham H et al., arXiv:2309.08600, 2023
- KAN: Kolmogorov-Arnold Networks, Liu Z et al., arXiv:2404.19756, ICLR 2025