# Rebuilding my voice assistant pipeline


[[TOC]]

I [wrote an article](https://blog.platypush.tech/article/Local-voice-assistant) a couple of months ago describing how I rebuilt my old voice assistant pipeline in [Platypush](https://platypush.tech).

And I wrote earlier articles describing my journey through self-hosted voice assistants - [in 2024](https://blog.platypush.tech/article/The-state-of-voice-assistant-integrations-in-2024) (with [Picovoice](https://picovoice.ai)), [in 2020](https://blog.platypush.tech/article/Build-custom-voice-assistants) (diving into the legacy [`assistant.google`](https://docs.platypush.tech/platypush/plugins/assistant.google.html) integration, the now defunct `assistant.alexa` integration, the now defunct Snowboy integration, the now defunct DeepSpeech integration and Picovoice) and the [original article in 2019](https://blog.platypush.tech/article/Build-your-customizable-voice-assistant-with-Platypush) describing the Google Assistant integration.

For a couple of months I used the voice pipeline described in my latest article as my main driver, but after getting used to the robustness of commercial voice assistants I was unhappy with the high rates of false positives/negatives of my self-hosted pipeline, as well as its lack of robustness in detecting start/end of speech, especially in noisy environments and using cheap microphones.

After a bit of experimentation, I decided to swap and fine tune a few parts, and after a couple of days of trying the results are satisfying enough for me to write a short article about it.

The resulting pipeline is:

```mermaid
%%{init: {"flowchart": {"wrappingWidth": 250, "useMaxWidth": false}}}%%
graph TD
    A["Hotword detection<br/>- assistant.microwakeword (local)"] -->|HotwordDetectedEvent| B["Speech-to-text<br/>- assistant.openai (cloud)<br/>- assistant.vosk (local)"]
    B -->|SpeechRecognizedEvent| C["Generate response<br/>- openai (cloud or local)"]
    C --> D["Text-to-speech<br/>- tts.openai (cloud)<br/>- tts.piper (local)"]
    D -.->|follow-up| B

    classDef detect fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef process fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
    classDef respond fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
    classDef speak fill:#fff3e0,stroke:#ef6c00,color:#e65100

    class A detect
    class B process
    class C respond
    class D speak
```

## Wake word detection: a new `assistant.microwakeword` plugin

Until recently I used [`assistant.openwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.openwakeword.html), which wraps [openWakeWord](https://github.com/dscripka/openWakeWord).

It works, but the project has been quiet for a while and all of its pre-trained models are `v0.1` files from 2023, mostly trained on synthetic speech.

In practice I was getting both false positives (the hotword triggering on background noise) and false negatives (the hotword not detected unless I was close to the microphone).

I added a new plugin, [`assistant.microwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.microwakeword.html), on top of [microWakeWord](https://github.com/kahrendt/microWakeWord) through the [`pymicro-wakeword`](https://pypi.org/project/pymicro-wakeword/) Python bindings.

`microWakeWord` is the engine used by ESPHome and Home Assistant Voice, its models are actively maintained, and the built-in models (`okay_nabu`, `hey_jarvis`, `hey_mycroft`, `alexa`) ship with the package — no download required.

The current official models are provided under the [ESPHome micro-wake-word-models repository](https://github.com/esphome/micro-wake-word-models).

Custom models are supported too: drop a `microWakeWord` JSON manifest (with its `.tflite` model) in the plugin's models directory and reference it by name.

The plugin exposes the same API as `assistant.openwakeword` (same `models`, `detection_threshold`, `frame_duration`, `patience_frames`, `input_device`, `input_volume` parameters, same `list_models` action, same `HotwordDetectedEvent`), so switching is a matter of just changing the configuration to point to the new plugin:

```yaml
#   models:
#     - alexa

assistant.microwakeword:
  models:
    - alexa
```

A few design notes:

- Without an explicit `detection_threshold`, each model uses the `probability_cutoff` from its own manifest (0.9 for `alexa`), which is the value the model was validated against.
- Models already smooth their output over a sliding window, so unlike openwakeword, `patience_frames` defaults to 0 here.
- The audio thread never touches the TFLite models directly: it queues frames (and a reset marker) to the main thread, because `MicroWakeWord.reset()` reloads the interpreter and isn't safe to call concurrently.
- Like `assistant.openwakeword`, the plugin releases the audio device when another assistant plugin starts a conversation, and re-acquires it when the conversation ends.

## Speech-to-text: `assistant.openai`, with a much better ear

`assistant.openai` records audio when a conversation starts and transcribes it through Whisper.

The tricky part has always been *end-of-speech detection*: knowing when the user has finished talking. The previous implementation used a per-frame boolean from WebRTC VAD, and any single frame classified as speech would reset the silence timer. With even a bit of background noise, phrases would only end when the maximum conversation duration was hit.

The audio preprocessing layer (`AudioPreprocessor`, shared by the STT integrations) has been reworked:

- **[Silero VAD](https://github.com/snakers4/silero-vad) as the primary backend** (through [`onnxruntime`](https://onnxruntime.ai), no [PyTorch](https://pytorch.org) required), with WebRTC VAD and a plain RMS-based detector as fallbacks. To get an idea of Silero VAD's performance compared against other solutions for speech activity detection, take a look at the insightful article [_One voice detector to rule them all_](https://thegradient.pub/one-voice-detector-to-rule-them-all/). The Silero model is downloaded on first use into the Platypush workdir and checksum-verified:

```yaml
assistant.openai:
  # auto = silero -> webrtc -> energy, depending on what's installed
  vad_backend: auto
```

- **Hysteresis + sliding window**: a frame becomes speech above the VAD threshold and stays speech until its score drops by a margin, and the per-frame decisions are smoothed by majority vote over a 0.5s window. A single noise burst can no longer keep a conversation alive.

- **Adaptive noise floor**: the background noise level is tracked over time, and frames that aren't louder than `noise_floor_factor` (default 2.0, ~6 dB) times the floor are never considered as speech.

On a synthetic benchmark (Piper-generated phrases mixed with babble and burst noise), the old settings kept conversations open for 6–10 seconds after the speech had ended; the new pipeline ends them 1.0–1.6 seconds after the last word, which is the configured [`conversation_end_timeout`](https://docs.platypush.tech/platypush/plugins/assistant.openai.html#platypush.plugins.assistant.openai.AssistantOpenaiPlugin.start_conversation) plus the smoothing lag.

Since the changes on Platypush have been performed directly in the [`AudioPreprocessor`](https://git.platypush.tech/platypush/platypush/src/branch/master/platypush/plugins/assistant/_audio.py) also used by other native speech-to-text engines, other speech-to-text plugins also benefit from them - e.g. if you use [`assistant.vosk`](https://docs.platypush.tech/platypush/plugins/assistant.vosk.html) for full on-device detection.

## Responses and context: the `openai` plugin

The transcribed phrase is processed by the `openai` plugin, which handles both the API key and the conversation context. By default the assistant renders the response through the configured `tts_plugin`; if the response ends with a question mark, the conversation continues with a follow-up.

Custom hooks can intercept phrases before the default response kicks in — for example, to handle home automation commands locally without an LLM round-trip:

```python
from platypush import run, when
from platypush.message.event.assistant import SpeechRecognizedEvent

@when(SpeechRecognizedEvent, phrase='play (the)? music')
def play_music():
    run('music.mpd.play')
```

And the hotword event is wired to the conversation start with a similar hook:

```python
from platypush import run, when
from platypush.message.event.assistant import HotwordDetectedEvent

@when(HotwordDetectedEvent)
def on_hotword(**_):
    run('assistant.openai.start_conversation')
```

## Text-to-speech: `tts.openai`

Responses are rendered by [`tts.openai`](https://docs.platypush.tech/platypush/plugins/tts.openai.html), with volume and voice configurable. In my tests however [`tts.piper`](https://docs.platypush.tech/platypush/plugins/tts.piper.html) also yielded good results, and it runs entirely locally. In my tests on a RaspberryPi 5 however it added ~3-4s of latency to process the response to audio compared to `tts.openai`.

Since the TTS output goes to the speakers and the wake word detector listens on the microphone, the assistant plugins pause hotword detection while a conversation is active — otherwise the assistant would hear itself.

## The full configuration

```yaml
openai:
  api_key: YOUR_API_KEY
  # Or, for a custom setup using a model provider compatible with the OpenAI API,
  # e.g. a local Ollama instance:
  # base_url: http://localhost:8000/v1
  # model: gpt-5
  # Optional context that can be passed to initialize the assistant
  # context:
  #   role: system
  #   content: You are a useful assistant.

assistant.microwakeword:
  models:
    - alexa

assistant.openai:
  tts_plugin: tts.openai
  conversation_start_sound: ~/sounds/wake_word_detected.wav
  input_volume: 100

# Or, for a fully local setup that uses assisstant.vosk:
# assistant.vosk:
#   model: /path/to/vosk/model
#   conversation_start_sound: ~/sounds/wake_word_detected.wav
#   input_volume: 100

tts.openai:
  output_volume: 80.0

# Or, for a fully local setup that uses tts.piper:
# tts.piper:
#   output_volume: 80.0
```

## What changed on the code side

- **New [`assistant.microwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.microwakeword.html) plugin**: microWakeWord hotword detection through `pymicro-wakeword`, same API as `assistant.openwakeword`, built-in models bundled with the package, custom models via JSON manifests.
- **[`assistant.openwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.openwakeword.html) fixes**: frames are now 80ms (1280 samples, the size openWakeWord recommends) instead of 500ms — longer chunks only return the highest score over their sub-frames and can miss hotwords — plus a `patience_frames` parameter (consecutive frames above threshold) and an explicit `detection_threshold` parameter (default 0.5).
- **Shared [`AudioPreprocessor`](https://git.platypush.tech/platypush/platypush/src/branch/master/platypush/plugins/assistant/_audio.py) rework**: Silero/WebRTC/energy VAD backends with `vad_backend: auto` selection, threshold hysteresis, sliding-window majority vote, and an adaptive noise floor. Used by both [`assistant.openai`](https://docs.platypush.tech/platypush/plugins/assistant.openai.html) and [`assistant.vosk`](https://docs.platypush.tech/platypush/plugins/assistant.vosk.html).
- **New [Silero VAD wrapper](https://git.platypush.tech/platypush/platypush/src/branch/master/platypush/plugins/assistant/_vad.py)**: ONNX inference through [`onnxruntime`](https://onnxruntime.ai), recurrent state carried across frames, checksum-verified model download into the workdir.

## Caveats

The microphone on my RPi doesn't support 16 kHz natively, so audio is captured at 44.1 kHz and software-resampled — it works, but if detection feels sluggish software resampling is usually the first suspect.

The VAD and threshold defaults were tuned on synthetic audio plus a couple of my audio recordings.

Different rooms, microphones and acoustics may yield different results. You may want to adjust `detection_threshold` on the wake word plugin and `noise_floor_factor` on the STT plugins if you don't feel satisfied with your mileage.
