AI and Smart Contract Security: Audits, Exploits, Limits

What a 2023 audit test, the GPTScan tool, and a 2025 exploit benchmark found about AI and smart contract bugs, and how far each result extends.

Last verified: October 20266 min read

Language models can find real bugs in smart contracts, and they can also write working exploits, but every published measurement comes with limits. A 2023 academic test found that GPT-4 and Claude named the correct vulnerability type in 40 percent of 52 previously exploited DeFi contracts, and a December 1, 2025 benchmark reported that AI agents produced exploits for 207 of 405 historical contracts in simulation. Neither result shows that an AI-checked contract is safe.

Key takeaways

  • Smart contracts are hard to patch after deployment, which makes pre-launch review valuable and mistakes expensive, according to Ethereum's developer documentation.
  • GPTScan, an ICSE 2024 research tool that pairs GPT with program analysis, reported precision above 90 percent on token contracts but 57.14 percent on large projects.
  • In the 2025 benchmark, exploits were run on local forks of live chains with simulated funds, and no real assets were touched.
  • The same test found two previously unknown flaws in 2,849 recent contracts, and a real attacker used one of them four days later for about USD 1,000.
  • Audits, human or automated, do not catch every bug.

Why code review matters more on a blockchain

The smart contract security page of Ethereum's developer documentation states that deployed contract code usually cannot be changed to patch security flaws, and that assets stolen through such flaws are mostly irrecoverable because of immutability. Upgrade patterns can add some flexibility, but they add their own risks. The same page says audits will not catch every bug and are designed mostly to provide an additional round of review, and it recommends bug bounties scaled to the funds at stake. It also cites losses from contract defects of over USD 1 billion in total, with the DAO attack as one of its examples.

That background explains the interest in automated review. A tool that flags an unprotected function before launch costs far less than a fix after funds have moved. The measured question is how good the tools are, and the three studies below answer it from different angles.

Study one: language models as auditors, 2023

A paper titled "Do you still need a manual smart contract audit?" by Isaac David, Liyi Zhou, Kaihua Qin, Dawn Song, Lorenzo Cavallaro, and Arthur Gervais tested GPT-4 and Claude on 52 DeFi contracts that had previously been compromised. The models identified the correct vulnerability type in 40 percent of the vulnerable contracts. In a second test, where bugs were inserted by hand into five new contracts, the best result was a recall of 78.7 percent for GPT-4-32k. The authors also reported a high false positive rate and concluded that the models can assist but that manual auditors remain necessary.

The models in that paper are 2023 versions. The figures are a dated measurement of what was available then, not of current systems.

Study two: GPTScan, language models plus program analysis

GPTScan, by Yuqiang Sun and colleagues and accepted at ICSE 2024, splits the work. The language model matches code against scenario descriptions of logic bugs, and conventional static analysis then confirms or rejects the finding. The abstract reports precision above 90 percent for token contracts, 57.14 percent for large projects such as those in the Web3Bugs set, and recall above 70 percent on ground-truth logic vulnerabilities across roughly 400 projects. It also reports nine vulnerabilities that human auditors had missed, an average cost of USD 0.01 per thousand lines of Solidity, and a two-thirds cut in false positives from the static confirmation step.

The gap between 90 percent on token contracts and 57.14 percent on large projects matters. Standardized token code is a friendlier test than a sprawling protocol with custom logic.

Study three: agents that exploit, not just flag

On December 1, 2025, a team at the AI developer Anthropic, working with scholars from the MATS and Anthropic Fellows programs, published SCONE-bench, a benchmark of 405 contracts exploited between 2020 and 2025 on Ethereum, Binance Smart Chain, and Base. Each agent had 60 minutes and had to raise its native token balance by at least 0.1 in a sandbox. Ten models, given up to eight attempts each, produced exploits for 207 contracts (51.11 percent), worth USD 550.1 million in simulated stolen funds. On 34 contracts exploited after the models' training cutoffs, three models together produced exploits for 19, worth up to USD 4.6 million.

The authors then ran two models on 2,849 recently deployed contracts with no known vulnerabilities. Both found two previously unknown flaws, worth USD 3,694 in simulated revenue. Running GPT-5 across the whole set cost USD 3,476, and its average cost per run was USD 1.22. One flaw let any caller withdraw fees because a recipient address was never validated; a real attacker drained about USD 1,000 through it four days later. The authors state their own limits: testing was on simulators only, social engineering and stolen keys were excluded, a few outliers skew the dollar totals, and exploit revenue appeared to depend on the assets a contract held rather than on code complexity. Because the study came from a model developer, readers may want to weigh it alongside independent work such as the two above.

The three studies side by side

StudyTaskReported resultMain limit
David et al., 2023Name the vulnerability type in 52 compromised DeFi contracts40 percent correct; recall up to 78.7 percent on inserted bugsHigh false positive rate; 2023 models
GPTScan, ICSE 2024Detect logic flaws with GPT plus static analysisPrecision above 90 percent on tokens, 57.14 percent on large projectsWeaker on complex protocols
SCONE-bench, 2025Exploit 405 known-flawed contracts in simulation207 exploited; two new flaws in 2,849 contractsSimulated funds; developer-run study

Questions to ask about any "AI-audited" claim

  • Which model, which version, and on what date was the review run?
  • Was the finding confirmed by a second method, as in GPTScan, or only produced by a model?
  • What was the false positive rate, and who triaged the output?
  • Did a human auditor review the contract, and is that report public?
  • What does the claim say about funds, and is any figure measured on a live chain or only in simulation?

This article summarizes research on software security tools. It is not investment, legal, or security advice, it does not assess or recommend any token, protocol, wallet, or service, and it makes no prediction about prices. Decisions about money belong with a qualified professional, and security reviews of a specific contract belong with qualified auditors.

FAQ

Can an AI tool guarantee that a smart contract is secure?

No. The studies report partial recall and false positives, and Ethereum's documentation says even human audits will not catch every bug.

Did AI agents steal real money in the 2025 benchmark?

No. All exploits ran on local forks of blockchains in a simulator, and dollar values are simulated amounts. The only real loss mentioned in the report came from an outside attacker who used one of the discovered flaws.

Why does the benchmark use old contracts?

They have known outcomes, so success can be measured. To check for memorization, the authors also reported results on contracts exploited after the models' training cutoffs.

Are these results current?

They are dated measurements from 2023 to 2025. Models and tools change, so newer tests may give different figures.

Last verified: October 2026

Sources

  • Smart contracts (SCONE-bench research post), Anthropic Frontier Red Team, December 1, 2025
  • Smart contract security, Ethereum Foundation developer documentation, page updated October 7, 2026
  • Isaac David and five co-authors, Do you still need a manual smart contract audit? arXiv:2306.12338, June 2023
  • Yuqiang Sun and seven co-authors, GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis, arXiv:2308.03314, ICSE 2024

New to artificial intelligence?

Start with the basics, then follow the guides to tools, rules, and effects on society.

Start with AI Basics