Machine learning is one stage inside a data science project, not a replacement for the project. The CRISP-DM process model, released as a free guide in 2000, splits the work into six phases, and AI tools now assist with several of them. A 2024 benchmark of data science agents found the best system solved about 34 percent of its data analysis tasks, against roughly 64 percent for human participants on a sample.
Key takeaways
- CRISP-DM lists six phases: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. Modeling is only one of them.
- The phases are not strictly sequential. Projects usually move back and forth between them.
- A held-out test set is the basic guard against a model that has memorized its training data.
- On the DSBench benchmark, the strongest agent scored 34.12 percent on data analysis tasks, and human participants reached about 64 percent on a 10-challenge sample.
- In the 2025 annual developer survey, 84 percent of respondents used or planned to use AI tools, and 46 percent distrusted their accuracy.
Where machine learning sits in a data science project
CRISP-DM was produced by a consortium that included SPSS, NCR, DaimlerChrysler, and the Dutch insurer OHRA, with partial funding from the European Commission. Version 1.0 was announced on August 21, 2000, at the KDD-2000 conference in Boston, and the guide was offered as a free download. Its authors describe a cycle, not a straight line: the sequence of phases is not strict, and moving back to an earlier phase is normal. A process that continues after deployment is part of the design.
The table below pairs each phase with an illustrative reading of where AI-based tools tend to help and what a person still has to settle. The first column follows the CRISP-DM phase names; the other two columns are this article's own summary, not text from the guide.
| Phase | Where AI tools can help (illustrative) | What a person still decides |
|---|---|---|
| Business understanding | Summarizing background documents | Which question is worth answering, and what counts as success |
| Data understanding | Drafting summary statistics and charts | Whether the data actually measures what the question needs |
| Data preparation | Suggesting cleaning code, flagging odd values | How to treat missing and suspicious records |
| Modeling | Trying many model types quickly | Which features are legitimate and which leak the answer |
| Evaluation | Computing metrics on held-out data | Whether the metric matches the real cost of mistakes |
| Deployment | Generating monitoring scripts | Who is accountable when the model drifts or fails |
The pattern in the table is that software speeds up mechanical steps while the framing and checking steps stay with people. That division is a reasonable summary of current practice, but it is a judgment, and organizations draw the line in different places.
A worked example: why a test set exists
This example is illustrative, with invented numbers. Suppose a clinic-scheduling team has 1,000 past appointments, each marked as attended or missed, and wants a model that predicts missed visits. If the model is trained and scored on all 1,000 rows, a flexible model can reach 100 percent accuracy by memorizing them. That score says nothing about next month's appointments.
The standard remedy is to split the data first. The team might train on 800 rows and keep 200 untouched. If the model scores 98 percent on the 800 and 71 percent on the 200, the gap signals overfitting: it learned quirks of the training rows rather than patterns that carry over. The 71 percent is the honest estimate, and it should be compared with a trivial baseline. If 70 percent of appointments are attended, a rule that always predicts "attended" already scores 70 percent, so 71 percent is barely an improvement.
AI assistants can write this split in seconds. What they cannot do is notice that a column such as "reminder call outcome" was recorded after the appointment and therefore leaks the answer. Spotting that takes knowledge of how the data was produced.
What measurements show about AI doing the analysis
The DSBench paper, first posted on September 12, 2024, tests language models and agent systems on 466 data analysis tasks and 74 data modeling tasks drawn from competitions. The analysis questions come from a financial modeling competition and are mostly built around Excel files, long task descriptions, and several tables. The best agent, AutoGen paired with GPT-4o, solved 34.12 percent of the analysis tasks. On a sample of 10 challenges, human participants reached about 64 percent. The authors conclude that a large gap to human experts remained.
Two cautions apply. The systems tested were the ones available in 2024, and newer models may score higher, so the figure is a dated measurement, not a ceiling. The paper's own abstract and tables also report the modeling results with different metrics, so only the analysis figure is quoted here.
Survey data show how widely the tools are used. The 2025 annual developer survey reports that 84 percent of respondents used or planned to use AI tools, 46 percent distrusted the accuracy of the output, and 66 percent named "almost right, but not quite" solutions as their biggest frustration. Respondents were software developers, not data scientists specifically, so the numbers describe a neighboring group.
A checklist before trusting an AI-assisted analysis
- Was the test data separated before any cleaning or feature choices were made?
- Is there a simple baseline, such as predicting the most common outcome, to compare against?
- Does every input column exist at the moment a real prediction would be needed?
- Were the generated code and its outputs read by a person who understands the data?
- Is the metric tied to the real cost of errors, for example missed appointments versus false alarms?
- Is there a plan to monitor the model after deployment?
FAQ
Is data science the same as artificial intelligence?
No. Data science is a broad practice of drawing conclusions from data, using statistics, programming, and domain knowledge. Machine learning, a part of AI, is one family of methods used inside it. Many data science tasks, such as building a report, need no learning algorithm.
Can a language model do a data scientist's job on its own?
Current evidence says not reliably. The DSBench results show agents solving about one in three analysis tasks, and the survey data show many users distrusting the output. Tools are better described as assistants whose work needs review.
What is overfitting in plain terms?
Overfitting is when a model fits the quirks of its training data so closely that it performs worse on new data. A large gap between training and test scores is the usual warning sign.
Is CRISP-DM still used?
It remains a widely cited reference for describing project stages, though its original project website is no longer active. Teams often adapt the phases to their own tools and review steps.
Related reading
- how AI and data science differ
- AI in research and discovery
- the Deloitte AI Institute
- AI activity in New York
More in AI Basics.
Last verified: October 2026
Sources
- Jing L. et al., DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?, arXiv, 2024 (revised 2025)
- SPSS, NCR, DaimlerChrysler, and OHRA announce CRISP-DM, KDnuggets press release reprint, August 21, 2000
- Chapman P. et al., CRISP-DM 1.0 Step-by-step data mining guide, CRISP-DM consortium, 2000
- 2025 annual developer survey, AI section, 2025