AI systems can now plan and carry out multi-step tasks without a person approving each step, but researchers disagree on whether the reasoning behind that behavior is real thinking or pattern matching that breaks under pressure. Where such systems influence decisions about people, rules tend to keep a human in the loop: Canada's Directive on Automated Decision-Making requires a human to make the final decision for federal systems rated at its two highest impact levels.
Key takeaways
- Autonomy (how much a system decides without a person) and thinking (what happens inside it) are separate questions with separate evidence.
- A June 2025 Apple research paper reported that reasoning models collapse in accuracy beyond a certain puzzle complexity, and a published reply argued that much of the collapse came from test design.
- Canada's directive sorts federal automated decision systems into four impact levels; at levels three and four, the final decision must be made by a human.
- The directive covers federal government decisions about clients, not private companies, and its deadline for older systems to comply was June 24, 2026.
Two questions inside one headline
"Can AI think for itself?" can mean two things. The first is about autonomy: can a system choose and act without a person directing each step? Software has done that in limited ways for decades, from autopilots to fraud filters. The second is about thinking: is there reasoning inside the system, or only statistical pattern matching that produces text that looks like reasoning? Evidence on the first comes from deployments and rules. Evidence on the second comes from experiments, and the experiments disagree.
The dispute over reasoning models
In June 2025, Apple researchers published "The Illusion of Thinking," which tested large reasoning models on controllable puzzles where difficulty could be raised step by step. They reported three regimes. On easy problems, standard models did as well or better than reasoning models. At medium difficulty, the extra "thinking" helped. At high difficulty, both kinds of models collapsed. The authors also reported that reasoning effort rose with difficulty and then fell, even when the token budget was sufficient, and that models failed to use explicit algorithms consistently. The paper is on Apple's machine learning research site.
A reply by Alex Lawsen, submitted on June 10, 2025, argued the collapse partly reflected the test, not the models. The comment says some Tower of Hanoi runs would exceed the models' output limits, that the automatic grading could not tell a reasoning failure from a length limit, and that some River Crossing instances with larger numbers were mathematically unsolvable but scored as failures. In preliminary tests, asking models for a short program instead of every move produced high accuracy on cases first reported as failures. Neither document ends the debate. The honest summary is that models can produce long chains of apparently logical steps, that their success depends heavily on task design, and that experts differ on what the successes and failures mean.
How Canada limits machine autonomy in government
Canada offers a concrete answer to how much a machine may decide alone. The Treasury Board's Directive on Automated Decision-Making took effect on April 1, 2019 and applies to automated systems in production that make or support administrative decisions about clients. It excludes systems used only for research or in test environments. Before launch, a department must complete and publish an algorithmic impact assessment, which assigns the system one of four impact levels. The level then sets the requirements, summarized below from the directive's appendices.
| Impact level | Expected impact of the decision | Human involvement | Approval to operate | Expert peer review |
|---|---|---|---|---|
| One | Little or none, easily reversible, brief | System may decide without direct human involvement; humans handle quality assurance and can intervene | Assistant deputy minister | None |
| Two | Moderate, likely reversible, short-term | Same as level one | Assistant deputy minister | At least one expert |
| Three | High, difficult to reverse, potentially ongoing | Final decision made by a human, who reviews the system's recommendation | Deputy head | At least one expert |
| Four | Very high, irreversible, perpetual | Same as level three | Treasury Board | At least two experts |
The directive was last shown as modified on June 24, 2025, and gave existing systems until June 24, 2026 to meet its updated requirements. It binds institutions subject to the federal Policy on Service and Digital, so provincial agencies and private firms follow other rules.
Questions that cut through autonomy claims
- Who makes the final decision, and can that person actually overrule the system?
- How reversible is the outcome if the system is wrong?
- Was the system tested on harder cases than the ones it was built on, given how sharply performance can fall as difficulty rises?
- Are results from the developer alone, or has an outside expert reviewed them and published the review?
- Is the system's behavior described by what it did in tests, or by claims about what it understands?
This article offers general information on research and public-sector rules, not legal or technical advice for any specific system.
FAQ
Does AI make decisions on its own today?
In narrow settings, yes. Systems can decide without direct human involvement when the stakes are rated low, as Canada's directive allows at its first two levels. For high-impact decisions, the same directive requires a human to make the final call.
Did Apple prove that AI cannot reason?
No. The June 2025 paper reported an accuracy collapse on complex puzzles and questioned the models' true reasoning capabilities, but a reply argued that output limits and unsolvable test cases explain part of the result. The question remains open.
Does Canada's directive apply to chatbots used by companies?
No. It applies to automated decision systems used by federal institutions that make or support administrative decisions about clients. Private-sector use falls under other laws.
What is an algorithmic impact assessment?
It is a framework departments complete and publish before a system goes into production. It rates the likely impact of the system's decisions and the risk of its data, and the result sets which requirements apply.
Related reading
- Human and machine intelligence compared
- Whether AI can feel emotions
- How AI learns and adapts
- Is AI broken?
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- Directive on Automated Decision-Making, Treasury Board of Canada Secretariat, page modified June 24, 2025
- "The Illusion of Thinking" (Shojaee P et al.), Apple Machine Learning Research, June 2025, a study of reasoning models and problem complexity
- arXiv:2506.09250, Lawsen A., rebuttal titled "Comment on The Illusion of Thinking," June 2025