The best-known research checklist for designing AI products is the 18 guidelines for human-AI interaction that Microsoft and University of Washington researchers published at the CHI 2019 conference. The guidelines fall into four phases: first contact, everyday interaction, error situations, and long-term use. In a test with 49 design practitioners across 20 AI-infused products, participants logged 313 examples of guidelines being applied and 277 of guidelines being violated.
Key takeaways
- The 18 guidelines come from more than 150 earlier design recommendations, narrowed first to 20 and then to 18 through internal review, a practitioner study, and expert evaluation.
- Guidelines 1, 4, and 12 (what the system can do, relevant information, remembering recent interactions) were the most widely applied in the study.
- Guidelines 2, 11, and 17 (how well it works, why it acted, global controls) were relevant but often violated.
- The guidelines are design advice from a research paper. They are not a law, a standard, or a certification.
What the paper is and how it was built
Saleema Amershi, Dan Weld, Eric Horvitz, and ten co-authors published "Guidelines for Human-AI Interaction" in May 2019 at the ACM CHI conference, where it received an Honorable Mention award. The team argued that designers needed concrete guidance for AI-powered features, because systems that learn and make probabilistic guesses behave differently from fixed software. Their process had four phases. First came the consolidation of more than 150 design recommendations from earlier sources into 20 guidelines. Then an internal heuristic evaluation cut the set to 18. Next, 49 participants with experience in UX or human-computer interaction used the guidelines to evaluate AI-infused products drawn from a set of 20, and the authors rephrased unclear items. A final expert review checked the revisions.
The study tested whether the guidelines are relevant and understandable, not whether following them improves outcomes for users. That second question is left to later work.
The 18 guidelines by phase
The table paraphrases the published guideline titles and keeps the paper's numbering.
| Phase | Guidelines (paraphrased) |
|---|---|
| Initially | 1. Tell users what the system is able to do. 2. Tell users how reliably it does it. |
| During interaction | 3. Choose timing to fit the context. 4. Display information that suits the task at hand. 5. Respect relevant social norms. 6. Reduce social bias in outputs. |
| When wrong | 7. Make the service easy to call up. 8. Make it easy to dismiss. 9. Make it easy to correct. 10. Narrow the service when the system is unsure. 11. Explain why the system acted as it did. |
| Over time | 12. Retain recent context. 13. Learn from what users do. 14. Change behavior cautiously. 15. Invite feedback in fine detail. 16. Show what a user's actions will lead to. 17. Offer controls that apply globally. 18. Announce changes in behavior. |
The "when wrong" group is the one most tied to everyday work with AI tools. A person who can dismiss a suggestion, edit it, or see why it appeared stays in control of the result, which is the point of those guidelines.
What the evaluation found
Across the 20 products, participants identified 785 examples of the 18 guidelines after the authors removed unusable responses: 313 applications, 277 violations, 89 neutral ratings, and 106 "does not apply." Every guideline had at least one application or violation in every product category tested. The most clearly implemented were Guidelines 1, 4, and 12. The ones the authors flagged as important yet often unmet were 2, 11, and 17. In plain terms, products were better at showing what they do than at saying how often they are wrong, why they acted, or how to change what they monitor.
Voice assistants, photo organizers, and activity trackers produced the fewest applicable ratings. The authors link this to inputs beyond text and graphical screens, such as voice, images, and sensor data. Some participants also found Guidelines 10 and 14 hard to judge in a single session, since both concern behavior that shows up over time.
Using the guidelines to review a work tool
- Does the tool state what it can and cannot do before the first use?
- Does it say how often it errs, or does it present every output with the same confidence?
- Can a user dismiss a suggestion with one action and edit it without starting over?
- When a request is ambiguous, does the tool ask or narrow its answer, or does it guess?
- Is there an explanation of why a result appeared?
- Can the user control what the tool remembers and monitors, and is a change in behavior announced?
These questions are an adaptation of the guidelines by this site and are illustrative. A tool can meet all of them and still produce poor results, and the paper does not claim otherwise.
Limits of a 2019 checklist
The study evaluated products that existed in 2018 and 2019, including music recommenders, navigation apps, web search, autocomplete, email, and voice assistants. Chat interfaces that generate long text were not among the categories the paper lists. Whether each guideline carries over to them is a judgment, not a finding from this study. The sample of 49 practitioners was also small, so the counts describe what participants noticed rather than a measured rate across the market.
FAQ
Who wrote the 18 human-AI interaction guidelines?
The lead authors are Saleema Amershi and Dan Weld, with Eric Horvitz and ten others from Microsoft and the University of Washington. The paper appeared at CHI 2019.
Are the guidelines a legal requirement?
No. They are research-based design advice. Legal duties for AI products come from separate laws and regulators, and those differ by country and state or province.
Which guideline matters most for AI mistakes?
The paper groups five guidelines under "when wrong," covering invocation, dismissal, correction, scoping when in doubt, and explanation. It does not rank them by importance, and the study found Guideline 11 (explain why) was often violated.
Do the guidelines apply to chatbots?
The paper did not test chatbots built on large language models. Several guidelines, such as stating limits, allowing correction, and offering controls, can be applied to them, but that application is a design judgment.
Related reading
- Conversational interfaces and their history
- How Netflix personalizes recommendations
- Chatbot solutions for customer communication
- AI apps on the iPhone
More in AI and Society.
Last verified: October 2026
Sources
- Guidelines for Human-AI Interaction, Amershi S, Weld D, Vorvoreanu M, et al., Proceedings of CHI 2019, ACM, May 2019