Practicing a second language with a conversational program produces a medium-sized gain on average, according to pooled studies of controlled experiments. A 2022 peer-reviewed analysis in Language Learning & Technology pooled 17 studies and found an effect of d = 0.58 on second-language proficiency, and a 2026 conference abstract covering 67 studies reported g = 0.61. Both describe averages across very different programs, and neither shows that any single app works.
Key takeaways
- Bibauw, Van den Noortgate, François, and Desmet (2022) analyzed 17 studies and 100 effect sizes and reported a significant medium effect, d = 0.58.
- A 2026 conference abstract by Wang, Bibauw, Noreillie, and Desmet pooled 67 studies from 2011 to 2025 (4,544 participants) and reported g = 0.61, with a non-significant effect on comprehension.
- Gains were established for vocabulary, grammar (morphosyntax), accuracy, and overall proficiency, and for lower-proficiency learners.
- Corrective feedback, goal-oriented tasks, and guided interaction were among the features linked to better results in the 2022 analysis.
- The 2026 result is an unreviewed conference abstract, and the technology type, including large language models, was not a significant moderator.
What the research calls it
Researchers in this field use the term dialogue-based computer-assisted language learning for systems in which a learner practices a foreign language by talking or typing with a conversational agent. The agent can be a scripted chatbot, a voice program, a robot, or, since 2022, a system built on a large language model. The pooled studies measure learning outcomes after learners practice a second language with such an agent, and their designs differ enough that the authors needed special formulas to make the effect sizes comparable.
That definition leaves out translation tools, flashcard apps without dialogue, and automatic grammar checkers, which are separate technologies with their own evidence.
The 2022 meta-analysis
The analysis in Language Learning & Technology (volume 26, issue 1, pages 1 to 24, issued August 15, 2022) found 17 studies that could be pooled. It computed 100 individual effect sizes and combined them in a multilevel random-effects model, with formulas that make results comparable across study designs. The headline is a significant medium effect, d = 0.58, on second-language proficiency.
The authors then asked which features mattered, testing the type of interaction, the modality (text or speech), constraints on what the learner may say, feedback, whether the agent has a visual embodiment, and gamification. They report that form-focused and goal-oriented systems, system-guided interactions, corrective feedback, and gamification features were effective. Effects were established for learners at lower proficiency levels and for vocabulary, morphosyntax, holistic proficiency, and accuracy.
The 2026 update after generative AI
Because chatbots changed after 2022, a team led by Zhaori Wang and Serge Bibauw presented an updated analysis at the EUROCALL 2026 conference in Belfast. The abstract in the UCLouvain repository describes 67 studies published from 2011 to 2025, 237 effect sizes, and 4,544 participants with 23 first-language backgrounds. A three-level random-effects model gave g = 0.61 with a 95 percent confidence interval of 0.48 to 0.74.
Several subgroup results are reported. Teenagers gained the most (g = 1.05), then adults (0.54) and children (0.49). Productive skills (0.65) and language knowledge (0.60), especially vocabulary (0.92), improved, while the effect on comprehension was not significant. Delayed posttests showed a larger effect (0.92) than immediate ones (0.58). The estimate for recent chatbots built on large language models was the highest (0.64), but the technology type was not a significant moderator, so the abstract does not establish that the newer systems work better.
Summary of the two studies
| Item | Bibauw et al., 2022 | Wang et al., 2026 |
|---|---|---|
| Venue | Language Learning & Technology, peer-reviewed journal | EUROCALL 2026 conference abstract |
| Studies and effect sizes | 17 studies, 100 effect sizes | 67 studies, 237 effect sizes, 4,544 participants |
| Overall effect | d = 0.58 | g = 0.61 (95 percent CI 0.48 to 0.74) |
| Skills with clear gains | Vocabulary, morphosyntax, accuracy, holistic proficiency | Productive skills, language knowledge, vocabulary |
| Weak or absent | Not stated in the abstract | Comprehension not significant |
Limits of the evidence
- The agents range from text bots to voice systems and robots, so one average covers many different kinds of program.
- Averages hide spread. A pooled effect of about 0.6 says little about one learner or one product.
- The 2026 abstract does not report risk of bias or heterogeneity, and it has not been through journal review.
- Subgroup effects, such as the figure for teenagers, may rest on a handful of studies.
- Test results measure performance on a task, which differs from using the language with other people.
In Canada and the United States, learners range from French and Spanish students in school to newcomers learning English or French. Neither analysis reports results by country or program type, so applying them to a given classroom or immigrant language class needs local judgment from teachers.
FAQ
Does talking to a chatbot improve a second language?
On average, yes, according to the pooled studies, with a medium effect. The gains were clearest for vocabulary and grammar and not significant for listening or reading comprehension in the 2026 abstract.
Are newer large language model chatbots better?
Not established. The 2026 abstract reports a slightly higher estimate for them, 0.64, but says the technology type was not a significant moderator.
Does it replace a teacher or conversation with people?
The pooled studies do not test that. The authors note that earlier reviews did not compare chatbots with human conversation partners or classroom teaching, which the 2026 work set out to examine.
Who benefits most?
The 2022 analysis established effects for lower-proficiency learners. The 2026 abstract reports the largest gain for teenagers, though that subgroup result is the least certain part of the evidence.
Related reading
- AI in English language learning
- Classroom AI activities for ESL students
- How GPT-based chatbots work
- AI across education
More in AI in education.
Last verified: October 2026
Sources
- Dialogue systems for language learning: A meta-analysis, Bibauw S, Van den Noortgate W, François T, Desmet P, Language Learning & Technology 26(1), 2022
- Meta-analysis of dialogue-based CALL after generative AI (conference abstract), Wang Z, Bibauw S, Noreillie A-S, Desmet P, EUROCALL 2026, via UCLouvain repository