Language models handle some English grammar almost as well as people and other parts far less well. On BLiMP, a 2020 test of 67 grammar contrasts with 1,000 sentence pairs each, the best model evaluated, GPT-2, chose the grammatical sentence 83.0 percent of the time, against 88.6 percent for individual human raters. The gap was widest on rules about extraction islands, negative polarity items, and the meaning of quantifiers.
Key takeaways
- BLiMP tests grammar with minimal pairs: two sentences that differ slightly, only one of which is grammatical.
- In the original paper, GPT-2 scored 83.0 percent overall, a 5-gram baseline 61.2 percent, and humans 88.6 percent.
- Subject-verb and determiner-noun agreement came close to human level, while island effects were the hardest category for most models.
- The results describe models from 2019 and earlier. Scores for newer systems on the same test were not checked for this article.
- The authors caution that a good score on a phenomenon does not prove human-like knowledge of grammar.
How grammar knowledge is measured
A language model assigns a probability to any string of words, so grammar can be tested without asking the model to explain anything. The BLiMP paper by Alex Warstadt and colleagues, published in Transactions of the Association for Computational Linguistics in 2020, gives the model two nearly identical sentences and counts a success when the grammatical one gets the higher probability. Random guessing therefore scores 50 percent.
The pairs cover syntax, morphology, and semantics in 12 categories, generated from templates written by linguists. Examples from the paper include "Many girls insulted themselves" against "Many girls insulted herself," and "These casseroles disgust Kayla" against "These casseroles disgusts Kayla." The authors report that human annotators agreed with the intended labels about 96 percent of the time, and they estimate individual human accuracy at 88.6 percent overall.
What the scores showed
Four systems were compared: a 5-gram model, an LSTM, Transformer-XL, and GPT-2-large with 774 million parameters. Overall accuracy was 61.2, 69.8, 69.6, and 83.0 percent, with humans at 88.6. The table gives GPT-2 and human results for six of the categories, taken from the paper's Table 3.
| Category | GPT-2 (percent) | Humans (percent) |
|---|---|---|
| Anaphor agreement | 99.3 | 97.5 |
| Determiner-noun agreement | 95.8 | 92.2 |
| Subject-verb agreement | 86.4 | 90.9 |
| Quantifiers | 79.0 | 86.6 |
| Negative polarity item licensing | 76.8 | 88.1 |
| Island effects | 72.7 | 84.9 |
The paper concludes that models generally perform best and closest to humans on morphological phenomena, such as agreement, and that islands are the hardest category for most models. Only GPT-2 did well above chance there, and it stayed about 12 points under human accuracy.
What the hard categories involve
Island effects are the situations where a word cannot be questioned out of a clause. The paper's example pair is "Who has Colleen aggravated before kissing Judy?" against the ungrammatical "Who has Colleen aggravated Judy before kissing?" Negative polarity items are words like "ever," which need a licensing context: "The truck has clearly tipped over" is fine, while "The truck has ever tipped over" is not. The authors link the difficulty with such cases to abstract semantic and pragmatic conditions that may be hard to learn from text. Their reading is a hypothesis, and the paper itself notes that a forced-choice test cannot support strong conclusions about how models represent these dependencies.
An earlier test, the Corpus of Linguistic Acceptability, or CoLA, used 10,657 English sentences labeled grammatical or not. In the paper introducing it, recurrent neural networks beat earlier unsupervised models, but all models tested performed far below human level across many grammatical constructions.
Limits when reading these results
- The tested systems were trained on web text or encyclopedia text collected before the 2019 to 2020 period, so the numbers say little about current commercial assistants.
- BLiMP covers English only and uses synthetic template sentences, which differ from natural writing.
- A high score on one category does not show a human-like rule. The authors warn against reading it that way, since a model might succeed through patterns tied to specific words.
- Passing a grammaticality test is different from writing well. Style, tone, and factual accuracy are not measured here.
FAQ
Does AI make grammar mistakes?
Measured on BLiMP, GPT-2 chose the ungrammatical sentence in 17 percent of pairs overall, with higher error rates on island and negative polarity cases. Results for newer models were not checked here.
What is a minimal pair?
Two sentences that differ in a small way, where one is grammatical and the other is not. Comparing the probabilities a model gives them shows which one it prefers.
Are humans perfect on these tests?
No. Individual raters averaged 88.6 percent on BLiMP, and the paper reports about 96 percent agreement between the intended labels and human judgments.
Do the results apply to other languages?
BLiMP covers English only, so the findings cannot be assumed to hold elsewhere.
Related reading
- AI development and adoption across countries
- How much mathematics AI work needs
- Definitions of education and learning
- AI and software development
More in AI Basics.
Last verified: October 2026
Sources
- Warstadt A et al., BLiMP: The Benchmark of Linguistic Minimal Pairs for English, Transactions of the Association for Computational Linguistics 8, 2020
- BLiMP preprint, arXiv:1912.00582 (version 4, February 14, 2023), Cornell University
- Warstadt A, Singh A, Bowman SR, Neural Network Acceptability Judgments, arXiv:1805.12471, 2018 (revised 2019)