Randomized trials of AI in coaching and mentoring point in different directions, so the evidence does not support a blanket claim that AI will replace human coaches. A 2022 trial found an AI chatbot coach roughly matched human coaches on goal attainment, a 2024 trial found that AI prompts helped novice tutors teach better, and a 2026 trial of 114 coachees found measurable gains only from human coaching.
Key takeaways
- Terblanche and colleagues (2022) ran two ten-month randomized trials, one with human coaches and one with the AI chatbot coach Vici, and the coached groups did not differ significantly from each other.
- Tutor CoPilot (2024) put AI suggestions in front of 900 human tutors; their students were four percentage points more likely to master a topic.
- de Haan, Terblanche, and Nowack (2026) compared human and AI coaching directly with 114 coachees and reported substantial effects only for human coaching.
- Where AI helped most, it supported a human expert's work rather than replacing the relationship.
Three trials, three designs
| Study | What the AI did | Who took part | Reported result |
|---|---|---|---|
| Terblanche et al., PLOS ONE, June 21, 2022 | Chatbot coach (Vici) worked with each person directly | Undergraduate students; two trials of 210 and 268 people at assignment | Coached groups beat their controls on goal attainment, and the human and AI groups did not differ significantly |
| Wang et al., Tutor CoPilot, arXiv, October 2024 | Suggested expert-style responses to human tutors during live sessions | 900 tutors and 1,800 K-12 students | Four percentage points more students mastered topics; nine points for students of lower-rated tutors |
| de Haan, Terblanche, and Nowack, Human Resource Development International, February 2026 | Automated AI coaches compared with accredited human coaches | 114 coachees | Substantial effects on diverse outcomes for human coaching only |
The 2022 trial and its limits
The paper in PLOS ONE reports two longitudinal randomized trials, each lasting ten months and measuring goal attainment at eight time points. Study 1 used human coaches and Study 2, a replication years later, used the chatbot Vici. Both coached groups reached higher goal attainment than their control groups, with similar effect sizes (partial eta squared of .265 for human coaching and .269 for AI coaching). Heavier use of the chatbot went with higher attainment.
The authors note two limits: participants were undergraduate students, and goal attainment was self-reported. The two studies also ran in different years, so they are a replication design, not a head-to-head contest in one sample.
The 2026 head-to-head comparison
The 2026 study is described by its authors as the first direct randomized comparison of accredited human coaches and automated AI coaches. The abstract reports outcomes on goals, motivation, resilience, and wellbeing, and states that substantial effectiveness appeared "only for human coaching," with effect sizes in the mid-to-high range and success predicted by the coachee's initial self-efficacy and hope. The authors tie the result to a co-regulation model of coaching. Only the abstract could be read for this article, so details such as attrition and effect sizes by outcome are left out. Taken with the 2022 paper, the two results suggest that findings can depend on the sample, the chatbot, and the way outcomes are measured.
Supporting a coach instead of replacing one
The Tutor CoPilot study by Wang, Ribeiro, Robinson, Loeb, and Demszky tested a different idea: using AI to spread an expert's guidance to novices. The authors call it the first randomized controlled trial of a human-AI system in live tutoring. Beyond mastery rates, an analysis of more than 550,000 messages found that tutors with the tool more often asked guiding questions and less often gave away answers. The reported cost was about USD 20 per tutor per year. Tutors also reported problems, such as suggestions that did not fit the grade level. This is tutoring of schoolchildren, which differs from workplace mentoring, so the result is evidence for one setting.
Questions to ask about any AI coaching claim
- Was the comparison randomized, and who was in the control group?
- Were outcomes self-reported or observed by someone else?
- Did the AI work with the client directly or support a human coach?
- How many participants dropped out, and who were they?
- Do the participants resemble the people the product is sold to?
FAQ
Can AI replace a human coach?
The evidence is mixed. One randomized trial found a chatbot coach comparable to human coaches among students, while a later trial of 114 coachees reported gains only from human coaching.
What is Tutor CoPilot?
It is a research system that suggests expert-style responses to human tutors during live sessions. It was tested in a randomized trial with 900 tutors and 1,800 students.
Does AI coaching work for mental health problems?
None of the studies above addressed that. Coaching research focuses on goals and development, and questions about health belong with a licensed clinician.
Why do results differ between studies?
Participants, products, time frames, and outcome measures differed, and the 2022 study relied on self-reported goal attainment.
Related reading
- How AI chat assistants converse
- A beginner learning path for AI
- AI for study notes
- AI services in business settings
More in AI Basics.
Last verified: October 2026
Sources
- Terblanche N., Molyn J., de Haan E., Nilsson V.O., Comparing artificial intelligence and human coaching goal attainment efficacy, PLOS ONE 17(6): e0270255, 2022
- Wang R.E., Ribeiro A.T., Robinson C.D., Loeb S., Demszky D., Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise, arXiv:2410.03017, 2024
- de Haan E., Terblanche N., Nowack K., A randomised controlled comparison of the effectiveness of human and AI chatbot coaching, Human Resource Development International, 2026