Large language models can draft multiple-choice questions in minutes, but published studies find that the drafts need expert review before they go on a real test. In a 2025 Hong Kong study of 100 AI-written and 100 human-written emergency medicine items, the AI items were easier and leaned toward recall. A 2026 systematic review rated the certainty of the whole evidence base as very low.
Key takeaways
- A Hong Kong cohort study found AI-drafted items easier (difficulty index 0.78 versus 0.69) with similar discrimination, and experts flagged more inappropriate difficulty in them.
- A systematic review of 15 studies found GPT-4 items rated close to human items on relevance, clarity, and distractor quality, but rated the evidence certainty very low.
- Established item-writing rules, such as focused lead-ins and plausible distractors, still decide whether a question is sound, whoever drafts it.
- Most studies come from medicine and used older models, so results may not carry over to other subjects or newer systems.
What makes a multiple-choice item sound
Item-writing research long predates language models. A 2002 review in Applied Measurement in Education by Haladyna, Downing, and Rodriguez organized 31 guidelines drawn from educational testing textbooks and research studies. The National Board of Medical Examiners (NBME) publishes a free item-writing guide for health sciences, now in its sixth edition (cover dated October 2024), that turns similar ideas into rules a drafter can apply.
Two of those rules matter most when a machine writes the first draft. The lead-in, the question sentence that follows the case description, should be focused enough that a test-taker could cover the options and still guess the answer. And all options should be homogeneous and plausible, meaning they answer the same question along one dimension, so that wrong answers cannot be dismissed on form alone. The NBME guide also separates technical flaws into two kinds: those that add irrelevant difficulty for everyone, and those that cue test-wise candidates toward the key.
The Hong Kong comparison
The most detailed head-to-head study is Law and colleagues, published in BMC Medical Education on February 8, 2025. Candidates preparing for the Hong Kong College of Emergency Medicine primary examination sat two 100-item sets about three weeks apart: a mock exam written with ChatGPT-4o, and the actual examination, written by people. Twenty-four doctors took part out of 37 invited, fewer than the 34 the authors had planned for, so the study was underpowered.
The AI items were easier, with a mean difficulty index of 0.78 against 0.69. Discrimination, which measures how well an item separates strong from weak candidates, was 0.22 for AI items and 0.26 for human items, a difference that was not statistically significant. Reliability (KR-20) was 0.75 for the AI set and 0.83 for the human set. Six expert reviewers, who knew which items were AI-written, flagged inappropriate difficulty in 14 percent of AI items against 1 percent of human items, and factual inaccuracies in 6 percent against 4 percent. The AI items mostly tested lower-order thinking. Writing the AI set took 24.5 person-hours against 96 for the human set.
What the pooled evidence says
A systematic review and network meta-analysis in PLOS One (Riehm and colleagues, published January 2, 2026) pooled 15 studies covering five tools: ChatGPT 3.5, ChatGPT 4, Llama 2, Bard 1.0, and ChatPDF. Only 5 of the 15 compared AI items with human-written ones. On a 10-point scale, ChatGPT 4 did not differ significantly from humans on relevance, clarity, or distractor quality, while Llama 2 scored lower on all three. ChatGPT 4 had a lower discrimination index than human items in one exploratory comparison.
The authors rated certainty as very low for every outcome. Quality was defined differently across studies, about half did not give the items to real students, no randomized trials were found, and most were single-institution medical studies. They also noted that rapid model development means findings may not apply to newer systems.
A review checklist for AI-drafted items
The table maps common failure points to the written rules above. It is a practical aid built from those guides, not a validated instrument.
| Check | Question to ask | Why it matters |
|---|---|---|
| Lead-in | Can the question be answered with the options covered? | Vague lead-ins force guessing and add irrelevant difficulty (NBME) |
| Distractors | Are all wrong options plausible and of the same kind as the key? | Options that differ in form let test-takers eliminate answers without the content knowledge (NBME) |
| Facts | Has a subject expert verified the key and the rationale? | Factual errors were flagged in 6 percent of AI items in the Hong Kong study |
| Cognitive level | Does the item test application, not only recall? | AI items skewed to lower-order skills in that study |
| Cues | Is the key longer, more specific, or the only grammatical fit? | Such cues reward test-taking skill over knowledge (NBME) |
| Evidence after use | Do difficulty and discrimination statistics hold up on a pilot group? | Item statistics, not authorship, show whether an item works |
Where AI drafting fits in a workflow
The Hong Kong authors concluded that ChatGPT-4o shows significant potential for efficient question generation but needs expert oversight, and they proposed a hybrid process in which AI drafts and educators review and refine. The time figures explain the appeal: a draft set took roughly a quarter of the person-hours. They also explain the risk, since the saved time is only real if review is not skipped.
Context matters too. Assessments with consequences for licensing or grades call for stricter review than low-stakes practice quizzes, and institutions often set their own policies on disclosing AI assistance. Neither source above recommends a specific tool, and results for one model or prompt style do not transfer automatically to another. Readers who write questions for school or professional exams should check the assessment policy of their own institution.
The primary sources are linked here: the Hong Kong cohort study and the NBME item-writing guide.
FAQ
Are AI-written multiple-choice questions as good as human-written ones?
The evidence is mixed. A pooled analysis found GPT-4 items close to human items on several quality ratings, while the Hong Kong cohort study found AI items easier and more often flawed in difficulty. Certainty in the pooled analysis was rated very low.
What are distractors, and why do they matter?
Distractors are the wrong options. If they are implausible, test-takers can eliminate them without knowing the content, which makes the item less useful for separating strong from weak candidates.
Does AI save time on question writing?
In the Hong Kong study, the AI set took 24.5 person-hours against 96 for the human set. That figure covers one study, one model, and one subject, and expert review time needs to be added.
Do these findings apply to subjects outside medicine?
Not necessarily. Most included studies were in medicine, and the review authors say generalizability to other fields is unclear.
Related reading
- How AI design choices shape modern technology
- AI in the conceptual design of dashboards
- AI assistance in letter writing
- AI tools in interior design
More in AI Tools and Industry.
Last verified: October 2026
Sources
- Law AKK et al., AI versus human-generated multiple-choice questions for medical education: a cohort study in a high-stakes examination, BMC Medical Education 25:208, 2025
- Riehm L et al., The use of large language models in generating multiple choice questions for health professions education: a systematic review and network meta-analysis, PLOS One 21(1), 2026
- NBME Item-Writing Guide: Constructing Written Test Questions for the Health Sciences, sixth edition, National Board of Medical Examiners, 2024
- Haladyna TM, Downing SM, Rodriguez MC, A review of multiple-choice item-writing guidelines for classroom assessment, Applied Measurement in Education 15(3), 2002