Explainable AI methods fall into two main classes: self-interpretable models, such as decision trees and logistic regression, which are their own explanation, and post-hoc explanations, which are produced afterward for a model that is hard to read. Post-hoc methods are further split into local ones that explain a single decision and global ones that describe the whole model. This is the organization used in a 2021 report from the U.S. National Institute of Standards and Technology (NIST), NISTIR 8312, which also sets four principles that any explanation should meet.
Key takeaways
- The two top-level classes are self-interpretable models and post-hoc explanations; post-hoc methods divide into local and global.
- LIME and SHAP are local methods, and partial dependence plots are a global one, according to NISTIR 8312, published September 29, 2021.
- NIST's four principles are explanation, meaningful, explanation accuracy, and knowledge limits.
- An explanation can be understandable and still wrong, so accuracy of the explanation is a separate test.
- Researchers disagree on whether post-hoc tools suit high-stakes decisions, and one prominent paper argues for interpretable models instead.
The class that explains itself
A self-interpretable model can be read directly by a person. Decision trees and regression models are the usual examples, and NIST adds decision lists, decision sets, and rule lists to the group. Walking one input through such a model shows how that decision was reached, and the model's structure shows how it behaves overall, so one object gives both a local and a global explanation.
Some sources describe an accuracy-interpretability tradeoff: the readable models are said to be less accurate than opaque ones. NISTIR 8312 reports that view and also reports the objection of Cynthia Rudin and a coauthor, who argue that in many cases an interpretable model loses no accuracy. The disagreement is open, and a team choosing a model for a specific task can test both options on its own data.
Post-hoc explanations: local and global
Post-hoc methods describe or approximate a model after training. Many of them only need to query the model for outputs on chosen inputs, so they can be applied without knowing the model's internals.
Local methods explain one input at a time. LIME, introduced in a 2016 paper by Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin, queries points near a given case and fits a simple model to represent the local decision. SHAP, from a 2017 paper by Scott Lundberg and Su-In Lee, treats the input features as players in a game and assigns each a Shapley value for its contribution to the output. A counterfactual states what would have to change for the system to decide differently, and saliency maps highlight the pixels that mattered most in an image classification.
Global methods describe the entire model. Partial dependence plots show how the predicted response changes as one feature changes, and TCAV expresses a neural network's state in terms of human-friendly concepts such as color. A global explanation can also be built by summarizing many local ones.
How the terms line up
| Class | Scope | Examples named in NISTIR 8312 | What the reader gets |
|---|---|---|---|
| Self-interpretable model | Whole model and each decision | Decision tree, logistic regression, rule list | The model itself, readable as is |
| Post-hoc, local | One decision or a subset | LIME, SHAP, counterfactual, saliency map | Reasons for a particular output |
| Post-hoc, global | Whole model, approximated | Partial dependence plot, TCAV, summarized local explanations | An overall picture that may simplify the model |
The AI Risk Management Framework from the same agency separates three neighboring words: transparency says what happened in a system, explainability says how a decision was made, and interpretability says why it was made and what it means to the user.
Four principles for judging an explanation
The NIST report defines four principles. Explanation means a system supplies evidence or reasons for its outputs or processes. Meaningful means the intended recipient can understand them. Explanation accuracy means the explanation correctly reflects how the output was produced. Knowledge limits means the system operates only under the conditions it was designed for and only when its confidence is sufficient.
The first principle sets no quality bar; the other three do. The report lists regulator and legal requirements, quality control, and customer relations as situations that call for explanations, and notes that developers and end users want different things from the same system. The accuracy principle matters because explanations can be gamed. NIST cites research showing that a classifier can be built to behave differently on the odd-looking inputs that LIME generates, which hides its real behavior from the explainer, and research on "fairwashing," where interpretable stand-ins make an unfair model look fairer than it is.
The argument for avoiding post-hoc tools
In a 2019 paper in Nature Machine Intelligence, Cynthia Rudin argued that explaining black box models for high-stakes decisions can entrench harmful practices, and that the better route is to design models that are inherently interpretable. The paper names health care and criminal justice as examples. Her position is one side of a live debate, and the classes above are not ranked by the NIST report.
This article gives general information about technical terms. It is not legal or professional advice, and it does not state what any law requires for a given system; questions about obligations belong with a qualified adviser and the primary documents listed below.
FAQ
What is the difference between explainable and interpretable AI?
Usage varies by author. In NIST's AI Risk Management Framework, explainability refers to the mechanisms behind a system's operation and interpretability to the meaning of its output for its purpose. NISTIR 8312 itself uses "self-interpretable" for models a person can read directly.
Are LIME and SHAP the same method?
No. LIME fits a simple model around one case by querying nearby points. SHAP assigns each feature a contribution based on Shapley values from game theory. Both are local, post-hoc methods that give per-feature explanations.
Does a good explanation mean the model is correct?
No. An explanation describes how an output arose; it does not show that the output is right. A model can be wrong and clearly explained, and an explanation can also fail to match what the model actually did, which is why NIST lists explanation accuracy as its own principle.
Do all AI systems need explanations?
NIST states that not all AI systems may require explanations. Its principles apply to systems that are intended or required to be explainable.
Related reading
- Where Canadian AI rules stand
- Documented risks of AI systems
- When AI principles conflict
- Threat or tool, the debate
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- Four Principles of Explainable Artificial Intelligence, NISTIR 8312, National Institute of Standards and Technology, 2021
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, National Institute of Standards and Technology, 2023
- Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, Cynthia Rudin, Nature Machine Intelligence, 2019
- Explaining the Predictions of Any Classifier (the LIME paper), Ribeiro, Singh, and Guestrin, 2016
- A Unified Approach to Interpreting Model Predictions, Lundberg and Lee, 2017