The best evidence on the limits of AI in education is a field experiment with nearly 1,000 high school math students, published in PNAS in 2025. Students who practiced with an unrestricted GPT-4 interface did worse on a later exam, without the tool, than classmates who never had it, while a version built to give hints instead of answers largely removed that harm. Other limits are well documented too: language models state falsehoods fluently, and in the United States the federal government left parental consent and vendor transparency to states and districts.
Key takeaways
- Practice with unrestricted AI raised practice scores but lowered later exam results in the PNAS experiment; a hint-based tutor avoided the drop without producing a gain.
- A September 2025 paper argues that language models hallucinate partly because standard evaluations reward guessing over admitting uncertainty.
- The U.S. Department of Education's final AI priority, published April 13, 2026, declined to set federal rules on parental consent or vendor transparency.
- UNESCO's 2023 guidance asks governments to regulate data protection and to consider limits on independent use by young people.
- Evidence on long-term learning effects is still thin, so local pilots with clear measures matter.
Learning: the one large field experiment
Hamsa Bastani and five co-authors ran the study in a high school math setting, with students split among three arms. One group had a standard ChatGPT-style assistant, one had a tutor with prompts designed to protect learning, and a control group had neither. Both AI groups did better during practice. After the tools were removed, the group with the unrestricted assistant scored below the control group, and the authors attribute this to students using the tool as a crutch. The tutor group's penalty was largely eliminated, but it did not beat the control group either.
Three cautions apply. The study measured short-term outcomes in one subject. PNAS published a correction notice for the paper in August 2025, so exact percentages should be taken from the corrected article rather than from summaries. And the finding is about unguided access, not about every educational use of AI.
Accuracy: why wrong answers persist
A student who cannot yet judge an answer is poorly placed to catch a wrong one. In a paper submitted on September 4, 2025, Adam Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang argued that language models produce plausible but false statements because training and evaluation methods favor guessing over expressing doubt. They note that most benchmarks grade answers in a way that makes a confident guess pay off. Their proposed fix targets how existing leaderboards score answers, which suggests the problem will shrink only if evaluation changes.
For a classroom, the lesson is concrete. An explanation that reads smoothly can contain an invented date, a wrong formula, or a reference that does not exist, and checking against a textbook or primary source stays a human task.
Privacy and consent: who decides
The U.S. Department of Education published its final priority on advancing AI in education on April 13, 2026, after more than 300 parties commented. Some commenters urged mandatory parental notification and opt-outs under the Children's Online Privacy Protection Act and the Family Educational Rights and Privacy Act, and asked that vendors disclose whether student data is used to train models. The Department replied that how best to ensure safety and communicate about technology use is optimally decided at the state and local level, and it declined to enact federal requirements. It also noted that the priority does not itself provide AI technology to students.
UNESCO's guidance on generative AI in education and research, published on September 7, 2023, takes a different line: countries should adopt rules that legally require protection of users' personal information, institutions should validate tools before use, and policymakers should consider limits on independent conversations between young people and these platforms.
In Canada, education is a provincial matter. British Columbia's ministry says on its K-12 AI resource page that AI tools should complement, not replace, human relationships, and that teachers make the final decisions about classroom use.
Where the limits show up and what a school can do
The table pairs each documented limit with its source and a measure a school can adopt. The measures are general practice suggestions, not requirements.
| Limit | Documented in | A measure a school can adopt |
|---|---|---|
| Skill loss when AI is used as a crutch | Bastani et al., PNAS, 2025 | Prefer tools that give hints and require student attempts first; test without the tool |
| Confident but false output | Kalai et al., September 2025 | Require sources and teach checking against a textbook or primary document |
| Uneven privacy rules | U.S. Department of Education final priority, April 2026 | Set district rules on consent, data use, and vendor disclosure in writing |
| Young users on general-purpose chatbots | UNESCO guidance, 2023 | Limit independent use by age group and use school-managed accounts |
| Little long-term evidence | Bastani et al. measured short-term outcomes only; the Department's April 2026 notice leaves evidence components to each grant competition | Pilot with a comparison group and a measure fixed before the start |
Evidence is still being built
The Department's notice records that commenters asked for clear accountability in evidence building and for evaluation frameworks, and the Department answered that it will consider whether to use evidence components in each grant competition. In practice, that leaves schools to design their own checks. A pilot that names one outcome in advance, such as unaided exam performance, and compares it against classes that did not use the tool says far more than satisfaction surveys.
Claims that AI tutoring works or fails are both ahead of the data for most subjects and age groups. Until more trials exist, caution about unguided use and openness to guided use fit the current evidence better than either extreme.
FAQ
Does AI harm student learning?
It can, depending on how it is used. In the PNAS experiment, unrestricted access lowered later exam performance, while a hint-based tutor largely avoided the harm. The study covers one subject and short-term outcomes.
Why does AI give wrong answers so confidently?
A 2025 analysis argues that models learn from methods that reward guessing, and that many benchmarks never give credit for admitting that an answer is unknown. Fluent text is not evidence of accuracy.
Does U.S. federal law require parental consent for classroom AI?
The Department of Education's April 2026 priority did not add such a requirement and left the question to states and districts. Existing laws such as FERPA and COPPA still apply to student data, and a district's counsel can say how.
Is there a Canadian national rule for AI in schools?
No national classroom rule was found. Education is a provincial responsibility, and provinces such as British Columbia publish their own guidance for school leaders and teachers.
Related reading
- What AI cannot do
- Pros and cons of AI in education
- AI in education: challenges and opportunities
- The potential of AI for schools
More in AI in education.
Last verified: October 2026
Sources
- Final Priority and Definitions: Secretary's Supplemental Priority on Advancing Artificial Intelligence in Education, U.S. Department of Education, Federal Register, April 13, 2026
- Guidance for generative AI in education and research, UNESCO, September 7, 2023
- Bastani H et al., Generative AI without guardrails can harm learning: Evidence from high school mathematics, Proceedings of the National Academy of Sciences, 2025
- Kalai AT, Nachum O, Vempala SS, Zhang E, Why Language Models Hallucinate, arXiv:2509.04664, 2025
- AI resources for B.C. K-12 schools, ministry web page, Government of British Columbia, July 18, 2024