A workable AI thesis topic is a narrow question that can be answered with data a student can reach, and four published studies show what that looks like: auditing the validity of benchmarks, measuring how prompt formatting changes accuracy, re-measuring which websites block AI crawlers, and testing whether language models can stand in for survey respondents. Each started from a gap its authors documented, and each leaves room for a smaller follow-up at the master's or honors level.
Key takeaways
- A review of 445 language model benchmarks by 29 expert reviewers found recurring weaknesses in what the benchmarks measure and how they score it.
- Changing only the format of a prompt moved accuracy by up to 76 points for LLaMA-2-13B in one study.
- A 2024 audit of about 14,000 web domains found that more than 5 percent of tokens in the C4 corpus were fully restricted from AI use.
- Synthetic survey respondents matched average opinion scores but showed less variation than real people in a 2024 Political Analysis study.
- Any thesis that involves human participants needs ethics review, which in Canada follows the Tri-Council Policy Statement (TCPS 2, 2022).
What makes a thesis question workable
Broad themes such as "AI and society" cannot be finished in one term. The four studies below share three traits worth copying: a measurable claim, a public dataset or an openly documented method, and a result that can be reproduced or extended with a few weeks of compute and reading. Before committing, a student can test any topic by writing it as one sentence with a number in it, such as "Does accuracy on benchmark X change by more than five points when the prompt format changes?" If the sentence cannot be written, the topic is still a theme.
Do benchmarks measure what they claim?
In a paper submitted on November 3, 2025, Andrew Bean and colleagues reviewed 445 language model benchmarks from leading conferences with 29 expert reviewers. They argue that measuring abstract ideas such as safety or robustness needs strong construct validity, meaning the test must capture the concept, and they report recurring patterns in the phenomena, tasks, and scoring metrics that weaken benchmark claims. The authors offer eight recommendations. A thesis could pick one benchmark that is widely cited, check it against those recommendations, and report where its definition, sampling, or scoring falls short.
How much does prompt wording change results?
Melanie Sclar and co-authors tested open-source models on the same tasks with small formatting changes, such as separators and spacing. The paper, first submitted on October 17, 2023, and revised for ICLR 2024, reports a gap of up to 76 accuracy points for LLaMA-2-13B and says the sensitivity persisted as model size and the number of examples grew. A follow-up can use a newer open model and a different task, ask whether the spread has shrunk, and report it with the same prompt variants so the comparison is fair.
Which websites now block AI training?
The Data Provenance Initiative's "Consent in Crisis" audit, submitted on July 20, 2024, examined about 14,000 web domains. It reports that more than 5 percent of all tokens in C4, and more than 28 percent of the most actively maintained, critical sources in it, were fully restricted, and that 45 percent of C4 carried terms-of-service crawling restrictions. The restrictions rose quickly within a single year. A thesis can repeat part of the audit on a sample of domains today, which makes a clean before-and-after comparison.
Can a language model replace a survey sample?
Argyle and co-authors argued in a 2022 paper that a model conditioned on demographics can emulate response patterns of human subgroups, a property they call algorithmic fidelity. James Bisbee and colleagues tested a version of this claim in Political Analysis, published online on May 17, 2024. They had ChatGPT 3.5 Turbo adopt personas drawn from the 2016 and 2020 American National Election Study and rate 11 sociopolitical groups. Average scores tracked the real survey, but the synthetic responses varied less, regression coefficients often differed, results shifted with small wording changes, and they differed for the same prompt over three months. A thesis could re-run that comparison on a newer model and a Canadian survey.
Four starting points compared
| Topic | Core question | Typical design | Main risk |
|---|---|---|---|
| Benchmark validity | Does the test measure the concept it names? | Audit one benchmark against published recommendations | Judgments need a clear scoring rubric |
| Prompt sensitivity | How far does formatting move accuracy? | Run fixed task, vary format, report spread | Compute cost for many variants |
| Crawler restrictions | How many sites block AI use now? | Sample domains, read their stated rules | Rules differ between files and terms of service |
| Synthetic respondents | Do models reproduce survey variation? | Compare model personas with survey microdata | Model versions change quickly |
Ethics review and a supervisor check
Any step that recruits or observes people raises ethics review questions. In Canada, the Tri-Council Policy Statement, TCPS 2 (2022), sets out research ethics board review, consent, and privacy rules for research involving humans. The department or supervisor decides which path applies to a specific project, and that conversation belongs before data collection starts.
FAQ
How many thesis topics does a student need?
One. Shortlisting three and testing each against the one-sentence rule is enough, and the narrowest question that still matters to the supervisor is usually the one to keep.
Can a thesis repeat an existing study?
Yes. Repeating a published design on a newer model or a different population is a standard way to add evidence, as long as the original method and data are cited and any change is stated.
Do these topics need a large budget?
Not necessarily. The benchmark audit and the crawler audit need reading and scripting time, while the prompt study scales with the number of formats and models tested.
Are the results of these studies still current?
They describe the models and web at the time of each study. Newer models, new crawler rules, and revised papers can change the numbers, which is the reason a follow-up study is useful.
Related reading
- Where neural network research is heading
- Why AI research errors are hard to find
- How AI research is published and checked
- What a Harvard AI course covers, week by week
More in AI Basics.
Last verified: October 2026
Sources
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks, Bean et al., arXiv:2511.04703, November 3, 2025
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design, Sclar, Choi, Tsvetkov, Suhr, arXiv:2310.11324, ICLR 2024
- Consent in Crisis: The Rapid Decline of the AI Data Commons, Longpre et al., arXiv:2407.14933, July 2024
- Out of One, Many: Using Language Models to Simulate Human Samples, Argyle et al., arXiv:2209.06899, 2022
- Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, Bisbee et al., Political Analysis 32(4), 2024
- Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans, TCPS 2 (2022), Panel on Research Ethics, Government of Canada