Automated writing evaluation, software that scores a learner's draft and flags problems in it, shows a medium average benefit for English learners' writing in pooled research. A three-level meta-analysis of 24 studies reported a between-group effect of g = 0.59 (g, a standardized effect size), with larger gains for vocabulary than for grammar. Most of that evidence comes from classes where English is a foreign language, so it cannot be assumed to carry over unchanged to learners in Canadian or U.S. schools.
Key takeaways
- Ngo, Chen, and Lai pooled 24 studies with 85 between-group effect sizes and found a medium overall effect of g = 0.59 on writing performance.
- The abstract reports that the tools worked better for vocabulary use than for grammar, and that medium to long use beat short use.
- One randomized trial with 190 Chinese university learners found higher scores on four writing criteria for the group that trained with an AI writing assistant.
- The pooled studies end in 2021, so they say little about today's chat-based assistants.
- Benefits were strongest in foreign-language settings, which leaves learners in English-speaking countries less well covered.
What automated writing evaluation is
Automated writing evaluation, usually shortened to AWE, covers programs that read a learner's text and return a score, error flags, or suggested rewrites. Older systems relied on rules and statistical models of grammar and word use. Newer assistants built on large language models can also rewrite whole paragraphs, which raises a different question: whether the learner or the tool did the writing.
The research summarized below mostly concerns the first kind of tool. The pooled analysis searched studies from 1993 to 2021, a period before general-purpose chatbots reached classrooms. Its results describe feedback software used alongside teaching, not chatbots used as writing partners.
What the pooled studies found
The analysis by Thuy Thi-Nhu Ngo, Howard Hao-Jan Chen, and Kyle Kuo-Wei Lai appeared in the journal Interactive Learning Environments (volume 32, issue 2, 2024). It combined 24 primary studies that compared an AWE group with a comparison group, yielding 85 between-group effect sizes, and a separate set of 34 studies that measured the same learners before and after, yielding 178 within-group effect sizes.
The between-group result was g = 0.59, which the authors call medium. The within-group result was g = 0.98, which they call large. The gap has an ordinary explanation: learners improve over a term from teaching, practice, and maturation even without software, and a before-and-after comparison counts all of that. The between-group figure is the fairer estimate of what the tool adds.
| Evidence | Design | Reported result | Main limit |
|---|---|---|---|
| Ngo, Chen, and Lai, 2024 | Three-level meta-analysis, studies from 1993 to 2021 | g = 0.59 between groups, g = 0.98 within groups | Pre-chatbot tools, mostly foreign-language classes |
| Wei, Wang, and Dong, 2023 | Randomized controlled trial, 190 Chinese university learners, repeated measures | Treatment group scored higher on task achievement, coherence and cohesion, lexicon, and grammatical accuracy | One country, one writing test format, no duration stated in the abstract |
Where the benefit was larger and where it was smaller
The authors also split the studies by features of the learners and the tools. According to the abstract, AWE worked better for vocabulary use than for grammar. Medium and long periods of use produced larger effects, while short use produced smaller effects compared with instruction without the software. Learners who studied with peers in an AWE condition appeared to gain a large effect.
By learner group, the benefits were strongest for undergraduates, for students in foreign-language contexts, and for students at an intermediate proficiency level. The last point matters for anyone choosing a tool for a beginner or an advanced writer, since the pooled average hides those differences. The abstract does not give subgroup sample sizes, so each of these splits is a pattern to test locally, not a rule.
One randomized trial in detail
A 2023 study in Frontiers in Psychology by Ping Wei, Xiaosai Wang, and Hui Dong randomly assigned 190 Chinese university students of English as a foreign language to AWE training or a control condition and measured them more than once. Writing was assessed with a sample test in the format of an international English exam, scored on task achievement, coherence and cohesion, lexicon, and grammatical accuracy. The treatment group scored higher on all four. Pre-test scores predicted three of the four outcomes, and writing self-efficacy, a learner's confidence in the skill, predicted two.
The authors conclude that the approach can help, especially at lower proficiency levels. Their wording is stronger than a single trial supports. The abstract does not report effect sizes or how many weeks the training lasted, and the sample came from one country and one educational setting. The trial fits the direction of the meta-analysis without settling how large the benefit is elsewhere.
Reading the evidence for a Canadian or U.S. classroom
English learners in Canadian and U.S. schools live in an English-speaking environment, which differs from the foreign-language classes that dominate the pooled studies. Because the meta-analysis found its strongest effects in foreign-language settings, the average cannot be read as a forecast for newcomer students in Ontario or Texas. No study located for this article measured AWE in Canadian or U.S. public school settings.
The following checks are practical habits, not findings from the studies:
- Write the draft first, and only then ask a tool for feedback.
- Compare the tool's corrections with a teacher's comments or a grammar reference, since software can be wrong about English.
- Rewrite the passage without the tool afterward to see what was learned.
- Keep a short log of repeated errors, which gives a teacher something concrete to review.
- Leave names, addresses, and immigration or health details out of anything pasted into a tool.
This article is general information about published research. It is not advice about any product, school policy, or language test, and it names no tool because the studies do not support ranking one.
FAQ
Does AI feedback improve English writing?
On average, yes, by a medium amount, according to a pooled analysis of 24 comparison studies (g = 0.59). The average covers many tools, ages, and settings, so a single learner's gain can be smaller or larger.
Is AI feedback better for grammar or vocabulary?
The meta-analysis abstract reports better results for vocabulary use than for grammar. A teacher's feedback on grammar remains worth comparing against any tool's suggestions.
Do the findings apply to newcomer students in Canada and the United States?
That is not established. The strongest effects came from foreign-language settings, and no study found for this article tested the tools in North American public schools.
Do newer chatbots work the same way as the tools studied?
Unknown. The pooled studies end in 2021, and a chatbot that rewrites text raises questions of authorship that scoring software did not.
Related reading
- AI activities for ESL students
- AI practice for speaking English
- AI-powered language learning
- AI in education: an overview
More in AI in education.
Last verified: October 2026
Sources
- Ngo TTN, Chen HHJ, Lai KKW, The effectiveness of automated writing evaluation in EFL/ESL writing: a three-level meta-analysis, Interactive Learning Environments 32(2), 727-744, 2024
- Wei P, Wang X, Dong H, The impact of automated writing evaluation on second language writing skills of Chinese EFL learners: a randomized controlled trial, Frontiers in Psychology 14, 2023