Controlled trials of AI chatbot tutors have reported learning gains when the chatbot was built around a teaching design and used inside a structured program. At Harvard, 194 physics students learned more in less time with a purpose-built tutor than in an active-learning class, and in Nigeria an after-school program using a general chatbot produced a gain of about 0.3 standard deviations in six weeks. Both studies are short, and neither tested an unguided chatbot left to students on their own.
Key takeaways
- The Harvard randomized trial (Scientific Reports, June 2025) found a median post-test score of 4.5 for the AI tutor condition and 3.5 for in-class active learning.
- The Nigerian trial in Edo State covered about 800 first-year senior secondary students over six weeks of after-school English classes, with teachers guiding each session.
- In both studies the chatbot sat inside a designed lesson. The results do not describe students using a chatbot freely.
- Stated limits include a single course, short duration, self-reported engagement, and open questions about long-term and negative effects.
What the Harvard trial tested
The study, published by Kestin, Miller, Klales, Milbourne, and Ponti, took place in Harvard's introductory physics course for life sciences in fall 2023. Of 233 enrolled students, 194 were eligible. Each student took two consecutive lessons, one on surface tension and one on fluid flow. For one lesson a student attended an in-class active-learning session, and for the other worked at home with an AI tutor, with the order randomized across groups of two or three students.
The tutor was not a bare chatbot. A system prompt set guidelines on active engagement, cognitive load, and growth mindset, and the platform guided students through multi-part problems one step at a time, because the authors found a prompt alone did not reliably do so. The prompts included step-by-step solutions written in advance, so the model was not relied on to produce the answers.
Median post-test scores were 4.5 for the AI condition and 3.5 for the class, from a pre-test baseline of 2.75. The authors report an effect of 0.73 to 1.3 standard deviations from quantile regression. The median time on task in the AI group was 49 minutes. Self-reported engagement averaged 4.1 against 3.6 on a five-point scale, and motivation 3.4 against 3.1.
What the Nigerian program tested
A World Bank team ran a randomized evaluation in Edo State, Nigeria. According to the team's January 9, 2025 blog post, students who took part scored about 0.3 standard deviations higher, which the authors describe as equivalent to nearly two years of typical learning, on tests of English, AI knowledge, and digital skills, and also did better on end-of-year curricular exams. The authors state that the program outperformed 80 percent of the education interventions they compared it with.
An earlier post from the team, dated September 18, 2024, describes the setup: 800 first-year senior secondary students, after-school English classes in computer labs twice a week for six weeks in June and July 2024, using Microsoft Copilot. Teachers introduced each week's topic, opened sessions with a suggested prompt, mentored students, and led short reflections. Teachers also named risks: overreliance, hallucination, and misuse.
The two studies side by side
| Feature | Harvard physics trial | Edo State, Nigeria program |
|---|---|---|
| Learners | 194 eligible college students | About 800 first-year senior secondary students |
| Chatbot design | Custom tutor with pedagogy prompts and prewritten solutions | General-purpose Microsoft Copilot with teacher-suggested prompts |
| Teacher role | Lesson designed by instructors; tutoring done at home | Teachers led sessions in computer labs |
| Duration | Two lessons in consecutive weeks | Six weeks, twice a week |
| Reported result | Median 4.5 vs 3.5 on the post-test | About 0.3 standard deviations |
| Stated limits | One course; authors built the tutor and taught the class | Short program; long-term effects and negative effects unknown |
Limits the authors state
The Harvard authors do not assume structured AI tutoring would beat active learning everywhere, naming tasks that need complex synthesis or higher-order critical thinking as untested. The lessons covered understanding, applying, and analyzing levels of Bloom's taxonomy, and the tutor relied on prewritten answers. Results may depend on high-quality instructional videos and expert-written, question-specific prompts, and feedback on images of student work was not tested. The tutor's designers also taught the in-class lessons.
The World Bank authors note that six weeks is short, that attendance suffered from flooding, teacher strikes, and after-school work, and that long-term effects, benefits in other subjects, and any negative effects remain open. They also suggest their design may have underestimated the true impact, a point that cannot be checked from the blog post alone.
Questions a school can ask before using a chatbot tutor
- Is the chatbot tied to specific lessons and checked answers, or open-ended?
- What does a teacher do during the session?
- Which outcome was measured, over how long, and against what comparison?
- What student data does the tool collect, and which law (FERPA or provincial privacy acts) governs it?
- How are wrong answers reported and corrected?
This article summarizes published research and is not an endorsement of any product or a legal opinion on student privacy.
FAQ
Do these studies show that AI chatbots improve learning in general?
No. They show gains for particular designs in particular settings: a purpose-built physics tutor and a teacher-led English program. Other tools and subjects need their own evidence.
Was a regular chatbot used in either study?
The Nigerian program used Microsoft Copilot, a general-purpose chatbot, but inside teacher-led sessions with suggested prompts. The Harvard tutor was custom-built with prewritten solutions.
Is there Canadian trial evidence?
None is cited here. The two trials come from the United States and Nigeria, and no Canadian randomized trial was verified for this article.
What risks did teachers raise?
In Nigeria, teachers named overreliance, hallucination (false answers stated as fact), and misuse.
Related reading
- Artificial intelligence in education overview
- OpenAI's education products
- Impact and potential of AI in schools
- AI compared with machine learning
More in AI in education.
Last verified: October 2026
Sources
- AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting, Kestin G et al., Scientific Reports 15, June 2025
- From chalkboards to chatbots: Transforming learning in Nigeria, World Bank Education blog, January 9, 2025
- From chalkboards to chatbots in Nigeria: 7 lessons to pioneer generative AI for education, World Bank Education blog, September 18, 2024