A working AI system needs five things beyond the algorithm itself: a defined purpose, suitable data, a trained model, ongoing testing, and people accountable for its operation. The U.S. National Institute of Standards and Technology (NIST) lays out this chain in its AI Risk Management Framework 1.0, published in January 2023, which treats testing and evaluation as a task that runs through the whole lifecycle rather than a step at the end. Most documented failures trace back to the data or to missing evaluation, not to the model alone.
Key takeaways
- NIST's framework names four key dimensions of an AI system: application context, data and input, the model, and the task and output.
- Testing, evaluation, verification, and validation are expected throughout the lifecycle, from design to operation.
- In a 2018 audit of three commercial gender classifiers, error rates reached 34.7 percent for darker-skinned women and 0.8 percent for lighter-skinned men.
- Documentation such as datasheets for datasets and model cards makes data and model limits visible to buyers and regulators.
- Stanford's 2026 AI Index reports that the average Foundation Model Transparency Index score fell from 58 to 40.
The five components and what each requires
The table groups the components using NIST's description of who does what across the lifecycle. The grouping into five rows is a simplified reading of that framework, not NIST's own list. The quoted tasks come from its Appendix A.
| Component | What it involves | What a buyer or reviewer can ask for |
|---|---|---|
| Purpose and context | Articulating and documenting the system's concept, objectives, assumptions, context, and requirements | A written statement of intended use and of uses to avoid |
| Data | Gathering and cleaning data, and documenting the metadata and characteristics of the dataset | A datasheet describing source, composition, and collection process |
| Model | Creating, selecting, calibrating, training, and testing models or algorithms | A model card with evaluation results by group |
| Testing and validation | Test, evaluation, verification, and validation across all stages | Evidence from tests that match the real setting of use |
| Operation and oversight | Operating the system and regularly assessing its output and impacts | Named owners, a monitoring plan, and a route for complaints |
NIST adds a point about independence: ideally, the people who verify and validate a system are not the same as those who perform its testing and evaluation.
Why data decides so much
NIST observes that bias is broader than demographic balance and data representativeness, and it separates systemic, computational and statistical, and human-cognitive bias. It says computational and statistical bias in datasets often comes from systematic errors in non-representative samples.
A published audit shows the effect in numbers. In "Gender Shades," presented at the Conference on Fairness, Accountability and Transparency in 2018, Joy Buolamwini and Timnit Gebru tested three commercial gender classification systems on a new facial dataset balanced by gender and skin type. Error rates for darker-skinned women were up to 34.7 percent, while the maximum error rate for lighter-skinned men was 0.8 percent. Reporting results by subgroup, as this audit did, is what exposed the gap.
Documentation as a component
Two proposals turned documentation into a standard expectation. "Datasheets for Datasets," first posted in March 2018 and published in Communications of the ACM in December 2021, argues that every dataset should ship with a datasheet covering motivation, composition, collection process, and recommended uses, much as electronic parts ship with specification sheets. "Model Cards for Model Reporting," presented at the ACM FAT* conference in January 2019, proposes short documents reporting how a model performs across demographic, cultural, and phenotypic groups, along with its intended uses.
Transparency has not kept pace everywhere. Stanford's 2026 AI Index reports that the average Foundation Model Transparency Index score fell from 58 to 40, and that leading companies keep training code, dataset sizes, and parameter counts private. A buyer of a large model may therefore be unable to inspect the very data component this article describes.
People as part of the system
NIST uses the term AI actors for everyone who designs, builds, deploys, operates, evaluates, or is affected by a system. Its roster for design work includes data scientists, domain experts, privacy and legal specialists, and members of impacted communities; deployment work includes system integrators, end users, and operators. The framework states that teams with diverse experience, expertise, and backgrounds help surface problems and emerging risks. Its four core functions, Govern, Map, Measure, and Manage, put an organization's own policies and accountability around the technical parts, with governance described as a cross-cutting function.
The framework text is voluntary guidance, not law, and it does not prescribe a particular architecture.
Checklist before relying on an AI system
- Is the intended use written down, including contexts where the system should not be used?
- Does the vendor document data sources, and can results be broken down by group?
- Were tests run on data that resembles the real setting of use?
- Who monitors outputs after launch, and who can switch the system off?
- Is there a person with authority to review decisions that affect individuals?
FAQ
Is more data always better for an AI system?
No. NIST points to non-representative samples as a source of statistical bias, and the Gender Shades audit shows how an unbalanced evaluation set can hide large error gaps. Fit to the real population matters more than volume.
What is a model card?
A short document that reports a model's performance, including results across demographic and other groups, and states its intended uses. The idea was proposed by Margaret Mitchell and colleagues in 2018.
Does the NIST framework carry legal force?
No. It is voluntary guidance for managing AI risk. Organizations may still cite it in procurement, policy, or internal governance.
Why do testing and operation count as components?
Because a system's behavior can change when the data or the setting changes. NIST expects testing, evaluation, verification, and validation across the lifecycle, and regular assessment of outputs once the system is running.
Related reading
- How vectors represent data in AI
- AI compared with plain automation
- Benefits of AI for society
- Choosing AI tools for a business
More in AI Basics.
Last verified: October 2026
Sources
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, National Institute of Standards and Technology, January 2023
- Buolamwini J., Gebru T., Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, Proceedings of Machine Learning Research 81, 2018
- Gebru T. et al., Datasheets for Datasets, Communications of the ACM, 2021
- Mitchell M. et al., Model Cards for Model Reporting, ACM FAT* 2019
- Inside the AI Index: 12 Takeaways from the 2026 Report, Stanford HAI, April 13, 2026