In a study published in Scientific Reports on September 14, 2023, three chatbots scored higher on average than 256 people on a standard test of divergent thinking, yet the strongest individual human answers were as original as the strongest chatbot answers in most comparisons. The test asked for unusual uses of four everyday objects. It measures one narrow kind of creative thinking, and the chatbots were the versions available in spring 2023. A January 21, 2026 study of more than 100,000 people found the same split between average and top performers with newer models, and a section below covers it.
Key takeaways
- Mika Koivisto and Simone Grassini compared 256 human participants with ChatGPT-3.5, ChatGPT-4, and Copy AI on the Alternate Uses Task, using rope, box, pencil, and candle as the objects.
- Average scores favored the chatbots on both measures: semantic distance 0.95 versus 0.91, and rater score 2.91 versus 2.47 on a five-point scale.
- Humans varied far more. Their lowest scores were much lower than any chatbot's, and their top answers were as good as or better than the chatbots' top answers in most comparisons.
- A larger study in Scientific Reports on January 21, 2026 found that some models, GPT-4 among them, match or beat the average person on word-based creativity tasks, while the most creative half of humans still outperformed every model tested.
- The authors limited the chatbots to short answers and a fixed number of ideas, and they state that results could change within about six months.
What the Alternate Uses Task asks
The Alternate Uses Task is a long-standing method for measuring divergent thinking, the ability to produce many different answers to an open question. A participant sees a common object and lists uses for it that go beyond the obvious. For a pencil, an illustrative less common answer would be "hair clip" or "bookmark". In this study, people had 30 seconds per object, and the instructions asked for quality over quantity. The median and the most common number of ideas per object was three.
Because the score depends on how unusual an idea is relative to ordinary language, the task rewards a person or system that avoids the first thing that comes to mind. That is also its weakness as a measure of creativity in general. It says nothing about whether an idea would work, whether it would be useful, or whether anyone would care about it.
Who and what was tested
The human group was recruited through the Prolific research platform and had to be native English speakers. Of 310 people who opened the study link, 279 finished and 256 passed the attention checks. The analyzed group had a mean age of 30.4 years (range 19 to 40) and lived mostly in the United Kingdom (166), the United States (79), and Canada (nine). Each was paid GBP 2 for about 13 minutes of work.
The chatbot side was ChatGPT-3.5, ChatGPT-4, and Copy AI, a writing tool built on GPT-3. They were tested between March 30 and April 5, 2023, in 11 fresh conversations per chatbot, with each conversation covering all four objects. To keep the comparison fair, the number of ideas requested from a chatbot was set to match typical human output, and replies were held to one to three words. The authors say this purposely constrained the chatbots.
How the answers were scored
Two measures were used. The first was semantic distance, computed with an open research tool called SemDis, which averages five language models to estimate how far an answer's meaning sits from the object. A larger distance means a less predictable use. The second was a rating from six trained judges who did not know whether a person or a chatbot wrote each answer, scoring on a scale from one (not at all original) to five (very original). Agreement among judges was high, with reliability values between 0.88 and 0.93 across the four objects.
The two measures moved together only moderately, with a correlation of about 0.55 for average scores. That gap matters. It means that distance in word meaning and perceived originality capture overlapping but different things, and the authors note that ChatGPT-4's high ratings were not explained by semantic distance alone.
The results in one table
The table below reproduces the descriptive statistics reported in the paper's Table 1, as read from the open-access version in the free archive of the US National Library of Medicine on October 27, 2026. A best-answer score is the top-scoring answer a participant or chatbot session produced.
| Measure | Chatbots (mean) | Humans (mean) | Chatbots (lowest) | Humans (lowest) |
|---|---|---|---|---|
| Average semantic distance | 0.95 | 0.91 | 0.90 | 0.69 |
| Best-answer semantic distance | 1.01 | 0.98 | 0.90 | 0.69 |
| Average rater score (1 to 5) | 2.91 | 2.47 | 2.21 | 1.00 |
| Best-answer rater score (1 to 5) | 3.61 | 3.18 | 2.71 | 1.00 |
Even the chatbots' lowest sessions never fell as far as the lowest human scores. The paper also reports that about seven percent of human trials had a best-answer rating below two, and no chatbot trial did.
Where the best humans stood
The averages favor the chatbots, but the title of the paper refers to the top of the distribution. For most objects and both measures, the highest human score was at least equal to the highest chatbot score. The exceptions are specific. On semantic distance, one chatbot session (Copy AI on the pencil) exceeded the top human value. On rater scores, ChatGPT-3.5 and ChatGPT-4 each reached 4.83 for the box, above the human maximum. ChatGPT-4's advantage in average ratings also did not hold for the pencil.
The authors argue that the human average is pulled down by weak or very common answers, which they link to lapses in attention, motivation, or mood in an online study. If so, the result describes the spread among ordinary people, not a ceiling on what any person can do.
What the result does not show
The paper lists its own limits. Each chatbot contributed only 11 sessions, which reduces the power to compare chatbots with one another. The sample was adults aged 19 to 40, mostly from Western countries. Chatbots are black boxes, so a high-scoring answer may come from a recombination of ideas or from text seen in training, and the authors point out that the same question can be asked of human minds. They also say the task measures one type of creative thinking, not creativity as a whole.
A correction published in February 2024 fixed a coding error in the analysis behind one figure (Figure 2D). Readers who reuse specific numbers should check the current version of the article. The data and scripts are public on the Open Science Framework.
What changed since 2023: the 2026 study of 100,000 people
The 2023 test has since been followed by a much larger comparison. On January 21, 2026, Scientific Reports published "Divergent creativity in humans and large language models", led by Karim Jerbi of the Université de Montréal and Mila, with Antoine Bellemare-Pépin, François Lespinasse, Yoshua Bengio, and Jay Olson, who developed the Divergent Association Task used in the work. The team tested several current models, including ChatGPT, Claude, and Gemini, against more than 100,000 human participants.
The press summary of the paper reports a result that echoes the 2023 title. Some models, GPT-4 among them, now match or exceed average human scores on divergent linguistic tasks. The most creative half of the human sample, however, outperformed every model tested, and the gap widened for the top 10 percent of people. The authors also report that a model's output can be adjusted through sampling temperature and that prompts built around word origins raised originality.
The same caution applies as in 2023. These tasks reward unusual word choices and short text, so they measure a thin slice of creativity, and model versions change faster than papers appear. The 2026 summary comes from a university press release; the paper itself was not readable in full for this article, so its detailed tables are not reproduced here.
A short checklist for reading creativity benchmarks
- Find out which tool and version was tested, and on what date.
- Check whether the chatbot's output length and idea count were controlled.
- Look for both average and best-case scores, since they can point in different directions.
- Check whether raters knew who or what wrote each answer.
- Ask what the test leaves out, such as usefulness, feasibility, or intent.
FAQ
Did AI beat humans at creativity in this study?
On average, yes, on this one task and with these 2023 chatbots. The best human answers were still as original as the best chatbot answers in most comparisons, so the result depends on whether average or top performance is the question.
Which chatbots were tested?
ChatGPT-3.5, ChatGPT-4, and Copy AI, a tool based on GPT-3. All were run between March 30 and April 5, 2023. Later models were not part of that study; a 2026 follow-up with newer models is described above.
How many people took part?
A total of 256 participants were analyzed after 310 opened the study and 279 finished. They were English speakers recruited online, aged 19 to 40.
Does a high score on this test mean a system is creative?
Not by itself. The test scores how unusual short answers are. It does not measure usefulness, intent, or the ability to carry an idea through, and the authors say as much.
Has the result changed with newer models?
A January 2026 study with more than 100,000 participants reports that some models now reach or pass the human average, while the top half of people, and especially the top 10 percent, still scored higher than every model tested.
Related reading
- Documented examples of creative AI output
- Whether AI can be truly creative
- Human and machine intelligence compared
- What test scores say about AI outsmarting people
More in AI Basics.
Last verified: October 2026
Sources
- Best humans still outperform artificial intelligence in a creative divergent thinking task, Koivisto and Grassini, Scientific Reports, 2023
- Correction to the same article, Scientific Reports, February 20, 2024
- Study data and scripts, Open Science Framework, project qdz3n
- News release on "Divergent creativity in humans and large language models", Université de Montréal and Scientific Reports, January 21, 2026