A trained AI model does not hold a searchable copy of its training files, but research shows it can still reproduce some of that material word for word. Researchers extracted hundreds of verbatim text sequences from GPT-2 in a study posted in December 2020, including names, phone numbers, and email addresses, and in November 2023 a follow-up team recovered gigabytes of training data from open and production language models. Regulators treat the question as one of fact: the European Data Protection Board said on December 18, 2024 that whether a model is anonymous must be assessed case by case.
Key takeaways
- A model stores learned numerical weights, not a file archive, yet memorized passages can sometimes be pulled out by querying it.
- The 2020 GPT-2 study found extraction worked even for text that appeared in only one training document, and larger models were more vulnerable.
- A 2023 attack made ChatGPT emit training data at a rate 150 times higher than its normal behavior.
- Canada's privacy commissioners ask developers to set retention schedules for training data, system prompts, and outputs.
- Removing one person's data from a trained model is hard; a 2019 research proposal sped up unlearning by 4.63 times on one dataset, with limits.
Four places data can sit in an AI system
Questions about what AI "remembers" mix up several different things. The table separates them. It is a simplified map, not a description of any one product.
| Where | What is kept | Who decides how long | Why it matters |
|---|---|---|---|
| Training dataset | Raw text, images, or records collected before training | The developer | Source of any personal or copyrighted material |
| Model weights | Numbers adjusted during training | The developer | May encode some training examples closely enough to repeat them |
| Prompts and outputs | What users type and what the system returns | The service provider, subject to law | Can contain personal or confidential information |
| Retrieval or fine-tuning stores | Documents added later for a specific task | The organization using the tool | Adds data that the base model never saw |
Deleting one layer does not delete the others. A user who removes a chat, for example, has not changed the weights of a model that was trained earlier.
What the GPT-2 extraction study found
Nicholas Carlini and eleven co-authors published "Extracting Training Data from Large Language Models" as an arXiv preprint on December 14, 2020, revised in June 2021. They queried GPT-2, a model trained on text scraped from the public internet, and report that they could extract hundreds of verbatim text sequences from its training data. The recovered items included personally identifiable information such as names, phone numbers, and email addresses, as well as IRC conversations, code, and 128-bit UUIDs.
Two findings stand out. The attack worked even when a sequence appeared in only one training document. And larger models were more vulnerable than smaller ones, so growth in model size does not by itself reduce the risk. The authors frame the danger as most serious for models trained on private datasets, because the attack needs nothing except the ability to send queries.
Extraction from production models
A 2023 paper by Milad Nasr, Carlini, and eight other authors, "Scalable Extraction of Training Data from (Production) Language Models," moved the question from a research model to deployed ones. Submitted to arXiv on November 28, 2023, it tested open-source models (Pythia and GPT-Neo), semi-open models (LLaMA and Falcon), and the closed ChatGPT. The authors report extracting gigabytes of training data without prior knowledge of the training set. Existing techniques worked on models that had not been aligned, and for ChatGPT the team developed a "divergence attack" that made the model emit training data at a rate 150 times higher than when it behaved normally.
The result suggests that safety tuning can hide memorization from casual use without removing it. It does not show how often ordinary users meet memorized text, a figure the abstract does not give.
Is a trained model itself personal data?
The European Data Protection Board adopted Opinion 28/2024 on certain data protection aspects of processing personal data in AI models on December 18, 2024, at the request of the Irish supervisory authority. The Board states that anonymity is to be assessed case by case by data protection authorities. For a model to count as anonymous, it must be unlikely that individuals whose data was used to build it can be identified, directly or indirectly, and unlikely that personal data can be extracted from the model through queries. The opinion lists non-exhaustive methods for demonstrating this.
The extraction studies explain why that second condition carries weight. A model that can be induced to print a phone number is hard to describe as free of personal data.
Canadian expectations on retention and deletion
On December 7, 2023, the Office of the Privacy Commissioner of Canada and provincial and territorial counterparts issued "Principles for responsible, trustworthy and privacy-protective generative AI technologies." The document asks organizations to set and follow retention schedules that cover training data, system prompts, and outputs, and to end retention when information is no longer needed, while keeping it long enough for people to exercise access rights. It says organizations that use generative AI should generally not retain, reuse, or disclose prompts unless otherwise required, and it asks developers to filter personal information out of datasets before training where possible. Publicly accessible personal information still falls under privacy law, and the document cautions against scraping it.
The principles say individuals should have processes to access or correct personal information in a model, particularly where it may appear in outputs. They do not explain how information would be removed from trained weights.
Deleting what a model has learned
Retraining from scratch removes a person's data with certainty but costs time and money. Lucas Bourtoule and colleagues proposed SISA training in a paper first posted in December 2019 and published at the IEEE Symposium on Security and Privacy in 2021. The method limits how much any single data point affects the final model. They report speedups over retraining of 4.63 times on the Purchase dataset and 2.45 times on SVHN, and 1.36 times on ImageNet with some accuracy loss, which indicates that the gain from efficient unlearning varies with the task.
Regulators have also acted on models directly. On May 7, 2021, the US Federal Trade Commission finalized a settlement with Everalbum, developer of the Ever photo app, that required deleting photos and videos of users who had deactivated their accounts and the models and algorithms built from users' uploads.
Checklist before putting personal data into an AI tool
- Which of the four places in the table will the data reach?
- Does the provider state a retention period for prompts and outputs, and can it be changed?
- Is the data used to train or fine-tune a model, and can that be switched off?
- Can the organization respond to an access or deletion request that involves the model, not only the chat log?
- Has anyone tested the model for verbatim leakage of the data it was given?
This is general information, not legal advice.
FAQ
Does ChatGPT or another chatbot remember everything it was trained on?
No. Studies show that some passages are memorized and can be recovered, but the amount depends on the model, the data, and the attack. The 2023 study did not report how often ordinary use returns memorized text.
Can a person ask for their data to be removed from a trained model?
Privacy law gives access and correction rights over personal information, and the Canadian principles say processes should exist for information that may appear in outputs. Technical removal from trained weights is an open research problem, so organizations may delete datasets, filter outputs, or retrain.
Is deleting a chat the same as deleting the data from the model?
No. A deleted chat concerns the prompt and output layer. A model trained earlier keeps its weights unless the developer retrains or alters it.
Why did the FTC order models to be deleted?
In the Everalbum case, the models had been built from photos uploaded by users, and the order required deletion of those models and algorithms along with the underlying photos and videos of deactivated accounts.
Related reading
- How an AI system is developed
- How AI programs process large amounts of data
- AI and cryptography in data protection
- Whether AI can work without data
More in AI Basics.
Last verified: October 2026
Sources
- Extracting Training Data from Large Language Models, Carlini N et al., arXiv:2012.07805, 2020, revised 2021
- Scalable Extraction of Training Data from (Production) Language Models, Nasr M et al., arXiv:2311.17035, November 28, 2023
- Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models, European Data Protection Board, December 18, 2024
- Principles for responsible, trustworthy and privacy-protective generative AI technologies, Office of the Privacy Commissioner of Canada and provincial and territorial counterparts, December 7, 2023
- Machine Unlearning, Bourtoule L et al., arXiv:1912.03817 and IEEE Symposium on Security and Privacy, 2021
- FTC finalizes settlement with photo app developer related to misuse of facial recognition technology, Federal Trade Commission, May 7, 2021