A voice assistant is a chain of separate parts: a wake word detector, speech-to-text, a step that decides what the user wants, and text-to-speech for the reply. Open-source tools such as OpenAI's Whisper for transcription and Piper for speech can be combined on home hardware, and Home Assistant's Assist shows one working design. Building one is mostly a question of speed, accuracy, and what happens to the recorded audio.
Key takeaways
- A typical pipeline has four stages: wake word, speech-to-text, intent handling, and text-to-speech.
- Whisper was trained on 680,000 hours of audio and is released under the MIT License, but it is slow on small computers.
- Home Assistant reports about eight seconds for Whisper on a Raspberry Pi 4 and under one second on an Intel NUC.
- A 2024 study found that roughly one percent of Whisper transcriptions contained invented phrases.
- In the United States, an FCC ruling of February 8, 2024 treats AI-generated voices in robocalls as artificial voices under the TCPA.
The four stages of a voice pipeline
A wake word detector listens for a short phrase and does nothing else until it hears it. Home Assistant's documentation says its openWakeWord model runs on the server and works only for English wake words, while a lighter model called microWakeWord can run on low-power chips such as the ESP32-S3 and on Android phones. On phones, audio is sent to the server only after the wake word is detected.
Speech-to-text then converts the audio to words. An intent step maps those words to an action, such as turning on a light or answering a question, either through fixed sentence patterns or through a language model. Text-to-speech finally turns the reply into audio. Each stage can run on a local machine or on a cloud service, and mixing them is common: for example, local wake word and transcription with a cloud model for the answer.
Speech recognition with Whisper
Whisper is a general-purpose speech recognition model. Its research paper, titled around large-scale weak supervision, was submitted to arXiv on December 6, 2022, and describes training on 680,000 hours of multilingual and multitask data. The project's repository says the code and model weights use the MIT License. Six sizes are published, trading accuracy for speed and memory.
| Model | Parameters | Approximate memory needed | Relative speed |
|---|---|---|---|
| tiny | 39 million | about 1 GB | about 10 times large |
| base | 74 million | about 1 GB | about seven times |
| small | 244 million | about 2 GB | about four times |
| medium | 769 million | about 5 GB | about twice |
| large | 1,550 million | about 10 GB | baseline |
| turbo | 809 million | about 6 GB | about eight times |
The repository gives these speeds relative to the large model, measured on an A100 graphics card with English speech. They are indicative, and results on other hardware will differ.
Speed on small computers
Home Assistant's local-assistant guide says Whisper takes around eight seconds to process a voice command on a Raspberry Pi 4 and under a second on an Intel NUC. It therefore suggests Whisper for people with powerful hardware who want open-ended requests, and a faster tool called Speech-to-Phrase for simple home control. That tool transcribes in under one second on a Raspberry Pi 4 but supports only a subset of commands.
For the voice itself, the guide describes Piper as a local neural text-to-speech system, noting that quality varies by language. The Piper repository is licensed under MIT, and its owner archived it on October 6, 2025, with development moved to a separate repository. Anyone starting a project should check which repository is current.
Errors that matter
Transcription mistakes are not just typos. A study presented at the ACM FAccT conference in June 2024 ran more than 13,000 clips from the AphasiaBank repository through Whisper and found that roughly one percent of transcriptions contained entire hallucinated phrases. Cornell's summary says speakers who pause longer between words, including people with aphasia, were more likely to trigger them. It also reports that OpenAI has updated the model since the 2023 research and the rate has fallen, so the figure describes the version tested.
A sensible build includes a confirmation step before any action with consequences, such as unlocking a door or sending a message, and a way to review what the system heard.
Voice data and the law
Recorded speech can be personal information. In Canada, the Office of the Privacy Commissioner announced on October 8, 2026 that it had updated its guidance for organizations that record customer calls. The update addresses voiceprints, described as uniquely identifying voice characteristics that count as sensitive biometric information, and says organizations covered by PIPEDA must follow it when recording calls.
In the United States, the Federal Communications Commission announced on February 8, 2024 a ruling that AI-generated voices are "artificial" under the Telephone Consumer Protection Act. The FCC said it took effect immediately and that voice cloning in robocall scams is illegal. A project that places automated calls with a synthetic voice should treat the FCC's announcement as a starting point and obtain legal advice, since consent rules turn on the type of call. This is general information, not legal advice.
A build checklist
- Decide which stages stay on the device and which use a cloud service, and write that down for users.
- Test with the target hardware before choosing a model size.
- Test speech from people with different accents and speech patterns, including long pauses.
- Keep raw audio only as long as needed, and say how long that is.
- Ask for confirmation before irreversible actions.
- Check the license of every model and voice before release.
FAQ
Can a voice assistant run without the internet?
Yes, if every stage runs locally. Home Assistant states that Assist can run fully on the user's own hardware so voice commands stay private, though local language support varies.
Does a bigger Whisper model always work better?
Larger models are generally more accurate but need more memory and time. The turbo model is an optimized version of large-v3 and is faster, but it is not trained for translation.
Is it legal to clone a voice for an assistant?
Rules depend on consent, purpose, and place. In the United States the FCC ruling covers AI voices in robocalls, and other laws can apply to commercial or impersonating uses, so permission from the person whose voice is used is the safe starting point.
Is wake word detection always listening?
The detector listens continuously, but Home Assistant says phones send audio to the server only after the wake word is heard. Server-side detection streams audio from satellites, so the design choice matters for privacy.
Related reading
- How Alexa grew into a smart home manager
- Alexa as a mass-market assistant
- The ChatGPT chatbot explained
- AI in music and lyric writing
More in AI Tools and Industry.
Last verified: October 2026
Sources
- Radford A. et al., Whisper paper on large-scale weak supervision (arXiv 2212.04356), arXiv, December 2022
- Whisper repository README, OpenAI, 2026
- Local Assist components and wake word documentation, Home Assistant, 2026
- Piper repository README, Rhasspy project, 2025
- AI speech-to-text can hallucinate violent language, Cornell Chronicle, June 11, 2024
- OPC updates guidance for organizations that record customer calls, Office of the Privacy Commissioner of Canada, October 8, 2026
- FCC Makes AI-Generated Voices in Robocalls Illegal, Federal Communications Commission, February 8, 2024