An ethical dilemma in AI is a design choice about how a system should act when every available option causes harm, and the clearest test case is an automated vehicle that cannot avoid a crash. The Moral Machine experiment, published in Nature in 2018, collected 40 million decisions from people in 233 countries and territories and found that preferences differ across three broad clusters of countries. Germany's government-appointed Ethics Commission took a different route in June 2017: it concluded that genuine dilemma decisions cannot be coded in advance so that they are ethically unquestionable, and it set limits on how a system may sort people.
Key takeaways
- Public surveys can measure what people say they prefer, but the Moral Machine authors tie those preferences to local institutions and cultural traits, which makes a single global answer hard to justify.
- Germany's 20-rule report of June 2017 prohibits any distinction based on age, gender, or physical or mental constitution in unavoidable accidents and prohibits offsetting victims against one another.
- The same report puts most of the effort on avoiding dilemmas in the first place and on documenting who is responsible at every moment.
- International and U.S. frameworks, including UNESCO's 2021 Recommendation and NIST's AI Risk Management Framework, ask organizations for oversight, impact assessment, and accountability processes in place of a coded moral rule.
The dilemma in concrete form
Philosophers have long used the trolley problem to ask whether it is right to cause one death to prevent several. Automated vehicles turned the thought experiment into an engineering question, because software that controls steering and braking will behave in some way in a rare, unavoidable crash, and someone decides that behavior in advance. Most real AI systems face quieter versions of the same problem: a hiring filter that trades fewer missed candidates against more false matches, or a triage tool that trades speed against caution. The vehicle case is useful because the stakes are visible and two serious efforts to answer it exist, one built on public opinion and one built on legal principle.
What the Moral Machine measured
The Moral Machine experiment, by Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-Francois Bonnefon, and Iyad Rahwan, appeared in Nature on October 24, 2018. The researchers built an online platform that showed visitors dilemmas faced by autonomous vehicles and asked which party should be spared. The platform gathered 40 million decisions in ten languages from millions of people in 233 countries and territories.
The paper reports a global pattern of preferences, variation by respondents' demographics, and cross-cultural variation that sorts countries into three major clusters. It also reports that these differences correlate with modern institutions and deep cultural traits. The authors argue that the results could help in developing socially acceptable principles for machine ethics. Because the data came from people who chose to use an online platform, they describe the views of participants, not a sampled survey of each country, and the study measures stated preference, not what drivers would do or what law should require.
The German commission's different answer
The Ethics Commission on Automated and Connected Driving was appointed by Germany's Federal Minister of Transport and Digital Infrastructure and published its report in June 2017 with 20 ethical rules. Several bear directly on dilemmas.
Rule 8 states that genuine dilemma decisions, such as a choice between one human life and another, depend on the specific situation and cannot be clearly standardized or coded so that they are ethically unquestionable. It adds that a human driver who kills one person to save others may act unlawfully without being culpable, and that this kind of after-the-fact judgment cannot easily be turned into advance rules in code. Rule 9 prohibits any distinction based on personal features such as age, gender, or physical or mental constitution in unavoidable accidents, and it prohibits offsetting victims against one another. The same rule says that a general design aim of reducing the number of personal injuries may be justifiable. Rule 7 gives protection of human life top priority over damage to animals or property.
The commission also pushed the question upstream. Rule 5 calls for technology designed so that critical situations, including dilemmas, do not arise, through limits on where a system operates, better sensors and braking, and warning signals for people at risk. Rule 16 asks that the allocation of responsibility between driver and system be documented and stored. The report is a set of recommendations from a commission, not a statute.
Comparing the two approaches
| Feature | Moral Machine (2018) | German commission (2017) |
|---|---|---|
| Method | Large online experiment on stated preferences | Interdisciplinary expert deliberation applying legal principles |
| Output | Patterns of preference and three country clusters | 20 ethical rules |
| View of sorting people by traits | Records that preferences vary by group and culture | Prohibits it in unavoidable accidents |
| Main limit | Preference is not a legal or moral justification | Recommendations do not bind courts or manufacturers by themselves |
What frameworks ask for instead
Broad AI ethics frameworks rarely prescribe how a system should choose between victims. The UNESCO Recommendation on the Ethics of Artificial Intelligence, adopted in November 2021 by its 193 member states, lists four values and ten principles, among them proportionality and do no harm, fairness and non-discrimination, responsibility and accountability, and human oversight and determination. Its oversight principle says AI systems should not displace ultimate human responsibility.
In the United States, the National Institute of Standards and Technology released AI RMF 1.0 on January 26, 2023 as a voluntary framework. The NIST page now says the framework is being revised as part of the White House AI Action Plan, and no revision timeline is given, so organizations should check the current version. This review found no U.S. or Canadian statute that dictates how a vehicle or other AI system must choose among people in an unavoidable harm scenario.
A checklist for teams that deploy high-stakes AI
- Name the decision the system makes and who is accountable for it at each moment, and log handovers between person and machine.
- Prohibit sorting people by protected traits unless a law or documented justification requires it, and test for it.
- Design to avoid the dilemma first: narrow where the system runs, add margins, and add warning signals.
- Run an impact assessment before launch and record the trade-offs that were accepted.
- Provide a route for outside review of incidents and publish lessons learned.
This article is general information, not legal or professional advice. Frameworks and laws change, and anyone responsible for a specific system should consult a qualified lawyer or the current text of the standard.
FAQ
Can a machine be given morals in code?
It can be built to follow rules and constraints, but the German commission concluded that genuine dilemma decisions cannot be coded in advance so that they are ethically unquestionable. Design for avoidance and human accountability does most of the work.
Did the Moral Machine find one global answer?
No. The paper reports a global pattern along with variation by demographics and three country clusters tied to institutions and cultural traits.
Is the German ethics report law?
No. It is a commission report to the transport ministry. Its rules guide policy and professional discussion.
Which framework should an organization in Canada or the U.S. start from?
UNESCO's Recommendation and NIST's voluntary AI RMF are both starting points. Neither is a legal compliance checklist, so sector rules and privacy law still need separate review.
Related reading
- global AI ethics guidance
- building trustworthy AI
- more ethical dilemmas in AI
- ethics of AI-generated art
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- The Moral Machine experiment, Awad et al., Nature, 2018
- Ethics Commission Automated and Connected Driving, Report, German Federal Ministry of Transport and Digital Infrastructure, June 2017
- Recommendation on the Ethics of Artificial Intelligence, UNESCO, 2021
- AI Risk Management Framework, National Institute of Standards and Technology, 2023, with 2026 revision notice