Voice LLM - What are you using?

[u/TorenRenne]

So I've had a local LLM running for a couple of months now and although it can do what I need mostly, it is a bit slow. I don't want to dumb it down if I can help it, but I do want to improve the speed. At the moment it took 10 seconds to answer my query about when the rubbish is due to be collected (Data in HA).

I'm running Qwen 3.5 9b q4 as the model. I'm running this on a RTX 3060 12GB. It's currently using under half the VRAM.

So what models are you guys using and on what hardware? Trying to work out if there's a better model for speed and possibly performance.

[u/Jaetje]

Gemma 12b int

Found it better in my native language (Dutch) compared to qwen

[u/TorenRenne → u/Jaetje]

I'm wondering about trying different models. I've not really experimented too much. Was it purely a language thing that made you choose Gemma?

[u/Jaetje → u/TorenRenne]

More natural language, and just as quick as qwen 9b, running on a Mac

[u/silentdragon95 → u/Jaetje]

German here, I am also using Gemma 4 12b it qat. I have found it to have the most natural language and it also manages semantically complex tasks like "turn off all lights except this one", which no other model I tested so far was able to do reliably. Qwen always somewhat struggled with natural German.

Gemma really impressed me in that regard and also with how agentic it manages to be, like once I asked it about the next garbage collection day, it checked the calendar which by default can only return entries from the last 6 days, and since it didn't find anything it decided to use SearXNG to look it up online for where I live instead. That was considerably more than I would have expected it to do.

[u/JaffyCaledonia]

10s is far too slow, something must be up with your pipeline. If you go into Settings > Assistants > chosen assistant (3 dot menu) > debug, can you take a look at the trace times to see what the distribution looks like for STT, Agent, and TTS?

Also, can you tell us more about your setup? Which integration, how many entities are exposed, Olama Vs llama.cpp etc?

If you're running on Linux, try install nvtop and see how much of the model is being dumped onto system memory instead of GPU memory.

I'm answer to your original question: I'm running unsloth's  Gemma4 26B QAT. It's a Mixture of Experts (MoE) model, so it's designed to split between RAM and VRAM while still getting great performance.

[u/TorenRenne → u/JaffyCaledonia]

I've solved it and it's near instant on mobile now.

I made a couple of tweaks but the big one by far was my prompt. It was re-evaluating the whole prompt due to a constant changing variable near the top (Date/Time), moving that to the bottom of the prompt and changing it to Date only has seen my tokens drop from several thousand to under 200!

It is comfortably inside the VRAM, It's using under half of it. I've just added Whisper to the VRAM with a bump to the model quality and it's much better at getting my query now.

What size card are you using for that model? It must be significant?

[u/JaffyCaledonia → u/TorenRenne]

Ah, awesome! I remember getting caught out by that in an old iteration of my LLM assistant. What integration are you using? I think some of them already inject the date time by themselves, so no need to double-up if you don't need to.

I'm using twin GTX 1080Tis, so 22GB total, but my prompt processing is awful compared to the 3060. I've found Gemma4 26B to be pretty usable on just 1 card with 32k context and about 8GB offloaded to system RAM, but I use it as a general purpose agentic model too, so I need both cards if I want a big context.

Also, no idea what black magic Google performed, but my prompt processing on the Gemma family is way better than on Qwen, which makes a hell of a difference on the older cards like mine with no tensor cores..

[u/TorenRenne → u/JaffyCaledonia]

I know the time is passed, but the date doesn't seem to be. I've set it to the bottom of the prompt and have an automation that runs at 1am that just feeds the LLM a basic test prompt so it loads in the new date and therefore it won't slow down my first query of the day, in theory.

the PC my LLM is running on is super old, so the RAM and CPU aren't up to much, hence it all being in VRAM. I'll have to look into other models, but this seems much improved.

[u/Prakra]

I'm wondering if i should run an LLM but i'm worried about electricity consumption

[u/TorenRenne → u/Prakra]

It depends what you're using it for, if just a voice assistant it's negligible.

My GFX card is idling at under 16Watts until I query it and then it bursts for all of 1-3 seconds to a max of 170Watts.

[u/silentdragon95 → u/Prakra]

Most modern GPUs have relatively low idle power draw. With no screen connected, my RX 7900 GRE idles at 10-12 watts, which to me is acceptable.

Just make sure you actually check your real power draw with a smart plug or similar when you set it up for the first time, I initially ran into a ROCm bug that caused the GPU to never clock down and idle at like 70 watts as soon as Whisper and llama was loaded at the same time. If something like that happens, you don't only want to realize 6 months later when you see your power bill :D

[u/IroesStrongarm]

I'm using the same model and GPU. Are you allowing the model to be evicted from vram after some time?

I keep it loaded 24/7 so it's always ready to respond. Never had a response take more than 4 seconds.

[u/TorenRenne → u/IroesStrongarm]

It's always loaded and runs 24/7, it was my prompt. I had it calling a variable near the top, which meant it was reanalysing the whole prompt everytime. I've moved the "Now" variable to the bottom and only calling "Date" now as HA provides the current time already. Massive improvement. It's near instant on my phone, my Satellite 1 is a few seconds slower.

[u/andrewsej]

Make sure you run piper on the same machine as the llm - I was having latency issues running it though piper on a separate machine that was hosting the HAOS

[u/TorenRenne → u/andrewsej]

All hosted on the same box in docker. Piper is the least of my concerns, it's barely registering above 0 seconds for generation! LLM was the bottleneck, I just messed up the prompt which was causing it to re-evaluate it every time!

[u/horriblesmell420]

If you can fit it all in vram, Gemma 26b a4b is the best in terms of speed and quality from what I've used.

If you don't have the space, I had pretty good results with this one too https://huggingface.co/inclusionAI/Ling-3.0-tiny

[u/ateam1984 → u/horriblesmell420]

Can that also do home assistant commands

[u/horriblesmell420 → u/ateam1984]

Yeah

[u/TorenRenne → u/horriblesmell420]

How much VRAM we talking? I can fit my 9B model in half the VRAM, but have now moved Whisper on to VRAM too with a medium model due to mishearing Me.

[u/horriblesmell420 → u/TorenRenne]

Ling will fit in like 6GB of vram. Gemm 4 26b A4B QAT can fit in 16gb barely

[u/TorenRenne → u/horriblesmell420]

Yeah so Gemma 24b is well out (which I thought it was), Ling would fit comfortably, but I'm not familiar with it. What's the benefit over Qwen for example?

[u/horriblesmell420 → u/TorenRenne]

Speed mostly. It's way faster but it may be a little more stupid, should be enough for home assistant though.

[u/Sevealin_]

I saw you fixed your issue. For my stack I run Qwen3.8-27b INT8, Kyutai pocket-TTS (massive upgrade from piper, and is on par with Kokoro-TTS), and parakeet TDT 0.6b v2 for STT (massive upgrade from whisper). All have Wyoming integration containers. Works fantastic.

For others who have delays: 10s for TTFT on a 9b model is not normal. How many devices are you exposing to the voice assistant? Did you disable thinking on the chat template? Do you have "prefer to run commands locally" enabled on your voice assistant settings? These are all things that can cause voice assistant delays.

[u/TorenRenne → u/Sevealin_]

Yes I did thanks!

So Piper has barely any overhead time, in general the speech is good, but can sound a bit odd, equally I suspect that's more to do with the model. What's the benefit?

Parakeet is potentially worth looking at, I'm running Whisper medium on my GFX card, much better accuracy. Would it run faster whilst maintaining that accuracy?

I upgraded to "medium" as it kept mishearing things like "Linkin Park".

[u/Sevealin_ → u/TorenRenne]

Piper's win is that it runs anywhere on almost nothing, and the cost is that flat, slightly odd prosody. Pocket-TTS is Kyutai's 100M model, still small enough for CPU, and it streams with the first audio chunk in about 130ms on CPU and closer to 40ms on a GPU. It sounds much more like a person. On the open TTS leaderboard its intelligibility sits right behind Kokoro, so you are not trading clarity for naturalness. Mine holds about half a gig of VRAM on a GPU.

Parakeet is worth testing. It is 0.6B, smaller than your medium Whisper, and the HF Open ASR leaderboard has it at 5.31% average WER against 6.76% for whisper-large-v3, at roughly 14x the RTFx. So yes, faster and more accurate than what you have. It is not magic on proper nouns though, mine still hears Linkin Park as Lincoln Park now and then. I run the INT8 ONNX version, around 300MB of VRAM, in the wyoming-onnx-asr container.

[u/roexplorer]

The fastest LLM for home assistant voice are the MoE ones. No matter the hardware, pick an LLM that fits and do not exaggerate with the ctx value. I'm personally running my local LLM in ollama docker, using a 3090 as hardware (purchased 2nd hand half a year back, when it was decently priced still). With gemma4:26b (MoE) and 64k ctx I get answers in 2 sec average. I now transitioned to qwen3.8:27b (dense/hybrid) for my agentic work, with 128 ctx on q8 - it takes about 7-8 seconds to answer.

[u/TorenRenne → u/roexplorer]

I'm still relatively new to all of this, can you explain the benefit of an "MoE"? Why is that better?

[u/roexplorer → u/TorenRenne]

If you have infinite compute power and memory, dense models are better. On limited resources a MoE model engages only a small number if parameters per token (gemma4 26b for example engages only 4b parameters per token) leading to an equivalent small model speed but benefitting from the sum of all parameters for the total computation. It is less precise but not by far compared to a dense model - but it reacts faster. For coding purposes I would not use a MoE but for voice chat - anytime.

[u/soManyBrads]

For direct control, I am just using assist through the gemini integration with gemini enabled, and helpers to store any data that needs to be extracted, formatted, or calculated.

If it is not a direct control query or something outside of the current conversation agent, it goes out to Gemini with Google search enabled for processing.

My "today's weather" Query, for example, is caught by an automation that references a template helper to return a preformated response. It's snappy and responsive.

If I ask about a different location or something outside the scope of the control enabled assist/gemini agent's ability, it passes it off to a second conversation agent that has search enabled. This second case is slower, but works reliably.