Adaptive tutoring software has a longer evidence record than chatbots, and the record is mixed. A 2016 meta-analysis of 50 controlled evaluations found that intelligent tutoring systems raised test scores by a median of 0.66 standard deviations, while two large randomized field trials in U.S. schools reported much smaller effects: 0.18 standard deviations for an online math homework tool in Maine, and about eight percentile points for a high school algebra course, but only in the second year. Both tools are older rule-based systems, not generative AI.
Key takeaways
- The median effect across 50 evaluations of intelligent tutoring was 0.66 standard deviations, but the size depended heavily on whether the test was written locally or standardized.
- A randomized trial with 2,850 seventh graders in Maine found a gain of 0.18 standard deviations from an online homework tool combined with teacher training.
- A randomized trial of the Cognitive Tutor algebra course in seven states found no effect in year one and a gain of roughly eight percentile points in year two, significant for high schools only.
- Large field trials usually report smaller effects than small or locally tested studies, so a single headline number says little about a given classroom.
What a standard deviation means in these studies
Education studies report effect sizes in standard deviations, a measure of how far the average treated student moved relative to the spread of scores. An effect of 0.66 moves a typical student from the 50th to the 75th percentile, according to the meta-analysis authors. An effect of 0.18 is much smaller, though on a statewide scale it can still matter, particularly if the tool is free and the teachers already assign the homework. The number alone does not say whether a program is worth adopting. It has to be read together with the test used, the length of the study, and what the comparison group did instead.
The meta-analysis: 50 evaluations, one large median
James Kulik and J. D. Fletcher reviewed 50 controlled evaluations of intelligent computer tutoring systems and reported a median improvement of 0.66 standard deviations over conventional instruction. Their review, published in Review of Educational Research in March 2016, adds two cautions. The size of the gain depended to a great extent on whether it was measured on locally developed tests or on standardized tests, which suggests that how closely a test matches the lesson content is a major driver of results. Six evaluations with nonconventional control groups and four with flawed implementations fell outside the main analysis and showed small effects. A median taken from studies that were selected to meet design criteria is therefore a ceiling for well-run cases, not a forecast for every school.
Maine: homework with instant hints, tested in 43 schools
Jeremy Roschelle, Mingyu Feng, Robert Murphy, and Craig Mason tested ASSISTments, a web-based homework platform that gives students hints and immediate feedback and gives teachers a summary of common errors. The randomized field trial, published in AERA Open in late 2016, covered 2,850 seventh-grade mathematics students in 43 Maine schools. Maine gives every seventh grader a laptop to take home, which made home access equal across schools. Schools were paired by prior achievement and enrollment, and one school in each pair was randomly assigned to use the tool with related teacher training, while the others kept their usual homework routine. On the TerraNova standardized test, the difference of 8.84 scale points corresponded to an effect of 0.18 standard deviations, and students with low prior achievement benefited most.
Two features limit how far the result travels. The schools applied to join the study, so they were volunteers. The homework tasks were mostly procedural, and the authors cite earlier work suggesting that immediate feedback suits procedural tasks better than conceptual ones.
Cognitive Tutor algebra: nothing in year one, a gain in year two
John Pane, Beth Ann Griffin, Daniel McCaffrey, and Rita Karam studied a blended algebra course that uses personalized, mastery-based software. Schools in seven states were matched into pairs and randomly assigned to adopt the course or continue their existing algebra curriculum for two years. Published in Educational Evaluation and Policy Analysis in June 2014, the analysis found no effects in the first year and evidence of positive effects in the second. The estimate was statistically significant for high schools but not for middle schools, and in both cases the size would improve the median student by about eight percentile points. A first-year result alone would have suggested the program did nothing, which is why trial length matters.
The three results side by side
| Source | Design | Reported effect | Main caution |
|---|---|---|---|
| Kulik and Fletcher, 2016 | Meta-analysis of 50 controlled evaluations | Median 0.66 standard deviations | Larger on locally developed tests than on standardized tests |
| Roschelle and colleagues, 2016 | Randomized trial, 43 Maine schools, 2,850 students | 0.18 standard deviations on a standardized test | Volunteer schools; procedural homework; teacher training included |
| Pane and colleagues, 2014 | Randomized trial, seven states, two years | None in year one; about eight percentile points in year two | Significant for high schools only |
Questions to ask before trusting a learning-tool claim
- Was the test written by the tool's developers or by an independent party?
- Were schools or students randomly assigned, and what did the comparison group do?
- How long did the study run, and did the effect change between year one and year two?
- Did the effect include teacher training or other supports that come bundled with the software?
- Which students gained, and did the result hold for the age group in question?
FAQ
Are these studies about generative AI?
No. ASSISTments and Cognitive Tutor were built and tested before large language models were widely available. They are useful evidence about adaptive feedback in general, but results for chatbot tutors need their own trials.
Why is the meta-analysis result so much larger than the field trials?
The authors of the meta-analysis point to the test used and to the control group as drivers of the number. The meta-analysis also found that flawed implementations produced small effects, and large field trials include many schools where delivery varies.
Does a 0.18 effect mean the tool is not worth using?
Not by itself. The Maine authors present the result as a significant gain from a free web-based tool with training, with the largest benefit for students who started behind. Whether that justifies the teacher time is a local decision.
Do the findings apply in Canada?
Both field trials took place in the United States. Curriculum, language of instruction, and provincial testing differ in Canada, so a Canadian school would need its own pilot evidence before assuming the same size of effect.
Related reading
- How AI is used across the learning experience
- Impact and potential of AI in education
- Whether AI could replace teachers
- AI compared with machine learning
More in AI in education.
Last verified: October 2026
Sources
- Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review, Kulik J A, Fletcher J D, Review of Educational Research 86(1), March 2016
- Online Mathematics Homework Increases Student Achievement, Roschelle J, Feng M, Murphy R F, Mason C A, AERA Open 2(4), 2016
- Effectiveness of Cognitive Tutor Algebra at Scale, Pane J F, Griffin B A, McCaffrey D F, Karam R, Educational Evaluation and Policy Analysis 36(2), 2014