Neural networks are applied by matching the network's structure to the shape of the data: convolutional networks for grids such as images, recurrent and attention-based networks for sequences such as text, and plain fully connected networks for fixed lists of features. Published milestones show each match working, from handwritten zip code digits in 1989 to machine translation in 2017. A 2022 benchmark also shows a limit, because tree-based models still beat neural networks on medium-sized tables.
Key takeaways
- Architecture is a set of assumptions about the data; convolution assumes nearby pixels relate, and attention lets any position look at any other.
- A 1989 network trained on U.S. Postal Service zip code digits reached about one percent error while rejecting about nine percent of inputs.
- An LSTM paper from 1997 reported bridging time lags of more than 1,000 steps, a task plain recurrent training handled poorly.
- A 2022 study of 45 datasets found tree-based models still state of the art on medium-sized tabular data.
- Choosing a network starts from the data type and the amount of data, not from the newest architecture.
Structure as an assumption about the data
Every network type bakes in a guess about how inputs relate. A convolutional layer applies the same small filter at every position, which assumes that a pattern means the same thing wherever it appears in an image. A recurrent layer reads one element at a time and carries a memory forward, which assumes order matters. An attention layer lets each element weigh every other element, which assumes that relevant context can be far away. A fully connected layer assumes nothing and therefore needs more data to find structure on its own.
| Architecture | Built-in assumption | Typical input | Documented example |
|---|---|---|---|
| Fully connected | None about layout | Fixed list of features | Compared with trees on 45 tabular datasets (2022) |
| Convolutional | Local patterns repeat across positions | Images, grids | Zip code digits (1989), ImageNet classification (2012) |
| Recurrent, LSTM | Order matters; memory carries forward | Signals, text in order | Long time lags (1997) |
| Attention (transformer) | Any position may matter to any other | Text, sequences | Translation in 3.5 days on eight GPUs (2017) |
Images: from zip codes to ImageNet
Yann LeCun and colleagues applied back-propagation to digits taken from U.S. Postal Service mail and showed that building constraints from the task into the network improved generalization. The conference version, published at NIPS 1989, reported about one percent error on zip code digits with roughly nine percent of inputs rejected for a human to read, and noted that the preprocessing was minimal while the architecture was tightly designed for the task. The journal version appeared in Neural Computation in 1989.
The same idea scaled in 2012. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton trained a deep convolutional network to sort 1.3 million high-resolution images into 1,000 classes and reported top-1 and top-5 error rates of 39.7 and 18.9 percent on the test data of the 2010 ImageNet contest. To make training faster the authors used an efficient GPU implementation, and to reduce overfitting they used a new regularization method.
Sequences: memory, then attention
Training recurrent networks on long sequences was hard because the error signal fades as it travels back through time. In 1997, Sepp Hochreiter and Jürgen Schmidhuber introduced Long Short-Term Memory, a gated design whose abstract reports learning across time lags of more than 1,000 steps.
The 2017 transformer paper dropped recurrence altogether. Its Table 1 lists the cost of a self-attention layer as proportional to the square of the sequence length, but needing a constant number of sequential steps, against a number of steps equal to the sequence length for a recurrent layer. Constant sequential steps allow all positions to be computed at once on parallel hardware. The abstract reports a score of 41.8 BLEU on the WMT 2014 English-to-French translation task after 3.5 days of training on eight GPUs.
Tables: where neural networks do not always win
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux compared tree-based models such as XGBoost and random forests with deep learning on a standard benchmark of 45 datasets from varied domains. Their preprint, posted in July 2022, reports that tree-based models remain state of the art on medium-sized data, around 10,000 samples. The authors name three challenges for tabular networks: robustness to uninformative features, preserving the orientation of the data, and learning irregular functions. The finding is a reminder that a neural network is one tool among several, and that a spreadsheet of customer records may be better served by a simpler model.
Choosing: questions to ask before picking a network
- Is the input a grid, an ordered sequence, or an unordered list of features?
- How many labeled examples exist, and is that closer to 10,000 or to millions?
- Does a simpler baseline, such as a tree-based model, already solve the problem?
- Does the task need explanations that the chosen model cannot give?
- Is the compute for training available, and is the result checked on data the model never saw?
FAQ
Can one network type handle every task?
In principle a fully connected network can approximate many functions, but the table above shows why specialized structures were adopted: they make learning possible with less data and compute. The tabular benchmark shows the opposite case, where a non-neural method wins.
What does a translation score such as 41.8 measure?
It is a BLEU score, an automatic measure that compares machine output with reference translations written by people. It is useful for comparing systems on the same test set, and it does not replace reading the output for meaning.
Why does the zip code result mention rejected inputs?
Postal systems can send uncertain digits to a person. A network that declines to answer on about nine percent of inputs and errs on about one percent trades coverage for accuracy. The paper reports both figures together for that reason.
Are old results still relevant?
The tasks have changed in size, but the logic has not: structure that matches the data lets a network learn from fewer examples. Each figure above is a dated measurement on a named test set.
Related reading
- Diagram of how a neural network processes input
- Computer vision: applications and challenges
- Deep learning and where it is applied
- AI applications inside organizations
More in AI Basics.
Last verified: October 2026
Sources
- Handwritten Digit Recognition with a Back-Propagation Network, LeCun Y et al., NIPS 1989
- Backpropagation Applied to Handwritten Zip Code Recognition, LeCun Y et al., Neural Computation 1(4), 1989
- ImageNet Classification with Deep Convolutional Neural Networks, Krizhevsky, Sutskever, and Hinton, NIPS 2012
- Long Short-Term Memory, Hochreiter S, Schmidhuber J, Neural Computation 9(8), 1997
- Attention Is All You Need, Vaswani A et al., arXiv:1706.03762, 2017
- BLEU: a Method for Automatic Evaluation of Machine Translation, Papineni K et al., ACL 2002
- Why do tree-based models still outperform deep learning on tabular data?, Grinsztajn L, Oyallon E, Varoquaux G, arXiv:2207.08815, 2022