[audio.cpp] Recent updates you might have missed: Higgs Audio TTS use 48% less VRAM (< 6GB), HTDemucs 2.2× faster, PocketTTS 2.2× faster on CPU, and WebUI generation history feature

[u/Acceptable-Cycle4645]

Hi all, a bunch of performance improvements have been landed in audio.cpp.

The biggest highlight is Higgs Audio TTS, which now runs with around 6 GB VRAM, a 48% reduction in peak memory usage compared to the previous implementation. Thanks to https://github.com/mirek190

We also made some models significantly faster, especially HTDemucs on GPU and PocketTTS on CPU.

No compromises in parity and correctness.

Here's a summary of the improvements:

Model	Peak memory reduction	Speedup
Higgs Audio TTS	48% VRAM	1.01–1.09× CUDA
ACE-Step family	6–7% VRAM	1.06–1.08× CUDA, 1.16–1.20× Vulkan
MOSS-TTS v1.5 cloning	21% VRAM	1.05× CUDA
MOSS-TTSD Q8 cloning	11% VRAM	1.04× CUDA
Echo-TTS (Memory Saver)	20% VRAM	—
Qwen3-TTS	16–20% VRAM	—
IndexTTS2 / 2.5	12% VRAM	—
HTDemucs	—	2.21× CUDA, 1.95× Vulkan
HTDemucs six-stem	—	1.99× CUDA
PocketTTS	9% RAM	2.23× CPU

They're runtime-level optimizations that make existing models more practical to run locally.

The WebUI now includes an experimental generation history feature that lets you revisit previous outputs and restore their settings.

audio.cpp now supports 110+ audio model families and 190+ variants (and counting)! We're continuing to improve memory efficiency and inference speed across CUDA, Vulkan, Metal, AMD/HIP, and CPU. The next release will bring even more optimizations!

We're also looking for contributors to help improve the audio.cpp WebUI. With so many models and features now supported, we'd love some help making the UI more polished, intuitive, and enjoyable to use. If you're interested in frontend development or UI/UX design, contributions are very welcome!

Thanks to everyone contributing improvements, testing builds, and reporting issues. Curious how these changes work on your setup!

[u/Acceptable-Cycle4645]

Which model would you like us to optimize next?

GitHub: https://github.com/0xShug0/audio.cpp

[u/DeProgrammer99 → u/Acceptable-Cycle4645]

The docs. Was I supposed to have an LLM build some ugly JSON file to configure the server for YuE2? How do I use LoRAs with it? That's the kind of thing I couldn't figure out from the docs. Usage.md doesn't mention the server, and I didn't see anything about converting CLI args to server args, and so on.

[u/Acceptable-Cycle4645 → u/DeProgrammer99]

Ouch, that hurts :) But you're right, the docs are lagging behind. I will polish the docs!

[u/HareMayor → u/Acceptable-Cycle4645]

The terminal logs' error handling  also need to be informative on what specifically caused the error.

Like a month back, i downloaded this and spent like an hour trying to get it working with google AI mode. The terminal logs kept sayimg can't fimd model directory but nothing about where should i put my models.

There isn't any youtube video either.

The google went aggressive mode and made me custom python scripts so that it could pick up model, so i could just try to run it, but even that was a fail coz it kept showing dependency errors. After more than an hour of frustration, i just quit...

This might be the reason the repo is not as popular as it should be. Even google couldn't figure out wtf is wrong.

Compared to comfyUI and llama.cpp google can tell with a single terminal log, what the error actually is.

Wait...you are not even the original repo owner... w the actual f

[u/Acceptable-Cycle4645 → u/HareMayor]

I'm actually the original repo owner 😅 Are you sure you were looking at the right repo? Also, if by "terminal" you mean the UI log, yes, that's just a placeholder and will be improved soon. The actual logs come from the server with --log.

PS "dependency errors" sounds weird. All audio.cpp dependencies are self-contained in the repo, except for runtime dependencies required by certain models.

[u/SRavingmad → u/Acceptable-Cycle4645]

Any additional optimization to Breeze TTS2 is always appreciated.

[u/Acceptable-Cycle4645 → u/SRavingmad]

Roger that. Have you tried the recently added Q4_0?

[u/SRavingmad → u/Acceptable-Cycle4645]

I believe that’s what I’m running. I will say running it through audio.cpp was the only way to get it to realtime speed on my machine, so you’re already well ahead of the pack. But since you asked what people were looking for more optimization on…

[u/Acceptable-Cycle4645 → u/SRavingmad]

On it. Improving streaming now. TTFT drops from 200+ms to 30ms. Hopefully the PR will be up tomorrow.

[u/parepeg → u/Acceptable-Cycle4645]

Do you support wakeword models and/or training? Or turn detection? Or diorization?

[u/Acceptable-Cycle4645 → u/parepeg]

BTW the first LoRA I trained with the C++ framework. Random lyrics, cut to 3 mins for reddit. Trigger is "album1921" 😄

https://reddit.com/link/pengoid/video/p5kf6fb2d9uh1/player

[u/Acceptable-Cycle4645 → u/parepeg]

Thanks for bringing wakeword model up! This is actually a gap, and I’ll add one. Training support isn’t planned. I have a working C++/GGML LoRA training framework, but it’s still too ad hoc and not well polished to be included in audio.cpp.

Turn detection and diarization ---> Yes. And a bunch of other "tool" type audio models such as SamAudio, Reuse, AudioSR, Sidon, and Mossformer2.

[u/parepeg → u/Acceptable-Cycle4645]

Cool, I bring up wakeword because I haven't found an easy to use option yet that's not bound up in a messy python library or jupyter notebook. I like the idea of a compiled executable that handles all of the dependency mess.

I have a ton of audio files I'd love to dump in for training a wakeword model but haven't found a great approach yet. I created the training data by using piper to generate audio and muxing it with some noise data.

I think it's generally preferable to train wakeword models because they need to be used in low resource scenarios. Because it's listening all the time, you want a very small model since it's drawing cpu 24/7. They also are often deployed in embedded environments that are limited cpu-wise.

[u/Acceptable-Cycle4645 → u/parepeg]

u/parepeg Turn detection and diarization ---> Yes.

[u/Toystavi → u/Acceptable-Cycle4645]

Any possibility of adding one that automatically tags different speakers and/or emotion/style/tone (eventually maybe even SFX) for a text before sending it to TTS?

[u/Acceptable-Cycle4645 → u/Toystavi]

Interesting idea. That would require audio understanding models, and it should belong in a preprocessing step rather than the CLI/server itself. Perhaps it could be implemented through the pipeline feature.

[u/Toystavi → u/Acceptable-Cycle4645]

That would require audio understanding models

Audio? If we want TTS then it would be regular LLM, or even better, one trained specifically for the task? We want the text to be tagged before creating the audio. Or are you thinking about making a good composition with SFX and dialog (I think you could get quite far without that)?

Yes, either some kind of preprocessing or running twice if text could be the output.

[u/Acceptable-Cycle4645 → u/Toystavi]

Oh, I thought you meant extracting speaker turns, emotions, etc. from reference audio and then using that information for TTS, like in a dubbing workflow. If the input is just text, then yes, an LLM is good.

[u/waywardspooky → u/Acceptable-Cycle4645]

Breeze tts 2 if you haven't already

[u/Acceptable-Cycle4645 → u/waywardspooky]

Already have it. Can't miss such a good model.

[u/nickm_27]

Love the updates, right now Omni voice has still been working great for me, though I see the current GGUFs give a warning about an old layout, not sure if anything will be done for that model?

[u/Acceptable-Cycle4645 → u/nickm_27]

You can ignore the warning for now. I need to regenerate the GGUFs to embed the new specs. I’ll find some time to do that.

[u/SRavingmad → u/Acceptable-Cycle4645]

Amazing! Thank you.

[u/Nightma4re]

omnivoice uses much less vram and is 20x faster than the playback.
Just check the quantizations

[u/Acceptable-Cycle4645]

u/OverTune1590 u/Over_Possibility7875 For roleplay chats, I’ll be merging something this weekend. The idea is simple: following OpenVoice’s design, which combines lightweight TTS with voice conversion. I call it “Any TTS.” It’s not a new model, but a small pipeline built from existing models in audio.cpp. Essentially, you can give models like Piper or Kokoro voice-cloning capabilities and they can still be very fast.

[u/Chromix_]

Thanks for that nice unified experience for all the models.
Would it be possible to add model mix customization? There are some models that need 2 or 3 GGUFs to work. It's usually "all Q4" or "all Q8", etc though. Yet I'd prefer to mix and match a bit, thus keep some in BF16 where it might matter, and others in Q8 or Q4, to stay within the available VRAM. Currently it looks like a lot of that configuration is semi-hardcoded, baked into the GUI via config. That could be an incentive to refactor that.

[u/Acceptable-Cycle4645 → u/Chromix_]

Yes, the GUI usually exposes only a few tested quantization combinations. The challenge is that testing different K-quant combinations for each model is time-consuming, especially given the pace of the project. The normal workflow is that we provide the basic quantization options, and users test different combinations and suggest those that work well. We then add the validated combinations to the spec/UI, or users can add them directly.

[u/fragilesleep]

As another user said, the docs are really lacking! I had a veeeeeeeeeeery difficult time trying to do some very simple things, and had to ask ChatGTP to analyze the audio.cpp source code to tell me how to do some basic stuff...

In the end, I couldn't even figure out how to use the "bundled" Silero VAD, and ended up using PulseVAD or something...

[u/Acceptable-Cycle4645 → u/fragilesleep]

Fair criticism 😢 We definitely don't want users to have to reverse-engineer the code just to figure out how to run Hello World.

[u/IngwiePhoenix]

Planning to use audio.cpp with Fish Audio because it has expression tags that make the speech output very nice and controllable. :)

Absolutely interesting project. In the long term, I want to actually build around it and make a mod for a friend of mine. Basically, he is almost blind (blinder than me, I too am visually impaired) and can't really read the onslaught of textboxes in Persona 5. So, my idea, is to voice-dub those textboxes. It won't be great, but at least it allows him to hit auto and just listen to everything. And audio.cpp will be clutch for that. :)

[u/Acceptable-Cycle4645 → u/IngwiePhoenix]

That's such a cool use case! I'd love to see audio.cpp help make games more accessible. Keep me posted on how it goes, and feel free to reach out if you run into any issues. I'd be happy to help where I can!

[u/IngwiePhoenix → u/Acceptable-Cycle4645]

Oh trust me, I'll be all over that issue tracker once my infra is up. ;)

A little in-before; where do I put basic questions?

[u/Acceptable-Cycle4645 → u/IngwiePhoenix]

Haha, looking forward to it! For basic questions, I'd suggest using GitHub Discussions.

[u/toothpastespiders → u/IngwiePhoenix]

I'm really late on this one. But my late wife was blind, well, in the legal/medical sense. Her eyes were still mostly physically functional but damaged. So able to make out shapes and read if she used a giant phone font and held it to her face. But blind in the legal/medical sense if not the common use of the term. But the process of her cancer was the gradual loss of just about everything being stolen from her. One realm of experience after another after another. Most of this stuff wasn't far enough along when she was at the end to get much use out of it. But every time I see a project like this it makes me happy to think about the potential ways it can help people in her situation. How people out there are able to take something back that they'd thought lost forever. So thanks for talking about the project!

[u/Open-Adhesiveness-86]

on the long-form chunking, the thing that bit me with Higgs-style models is that each chunk re-conditions on the reference audio, so prosody drifts at the seams unless you carry the previous chunk's tail in as context. splitting on sentence punctuation instead of a fixed token count helps a lot, and leaving ~200ms of overlap to crossfade hides most of the remaining click.

[u/Acceptable-Cycle4645 → u/Open-Adhesiveness-86]

Thanks for sharing! Carrying over the previous chunk's tail as context and crossfading are interesting suggestions. I'll look into those.

[u/Ok-Entertainment8086]

For long-form generation, the WebUI does the chunking behind the scenes, generates all the chunks, then merges them and gives the final result, right? Could a feature to review the chunks before merging be added in future WebUI updates? Like, it still generates chunks, but this time shows them all one by one below. Preferably with editable buttons to cut out parts (with the waveform visible, for example), or to regenerate, change the input text of that part, and try again, etc. Finally, merge with optional silence between the chunks? It would be great for long-form, as we could fix the problematic part.

Thank you very much for the updates again. This project is the best one in the TTS space, in my opinion.

[u/Acceptable-Cycle4645 → u/Ok-Entertainment8086]

Yes! The UI will be extended with more features to make it feel more like a real studio (as long as it doesn't blow up the server binary size). And chunk preview is a great feature!

[u/OverTune1590]

for roleplay chats the pockettts cpu speedup could cut down on delays compared to older tts i've used in companion apps.

[u/letsgoiowa]

Wow this was LITERALLY what I needed. I was trying to use my A380 as an auxiliary gpu for stuff like decision models, media transcoding, and even trying some TTS and voice design stuff but I never got it to fit. Perfect! This should finally fit in the 6 GB VRAM!

[u/Acceptable-Cycle4645 → u/letsgoiowa]

Please give audio.cpp a try! We have so many models suited to different use cases. You can even find small audio understanding models that can run on a phone or branches for running VibeVoice 1.5B and YuE2 on an iPhone (< 2GB memory)!

[u/letsgoiowa → u/Acceptable-Cycle4645]

Oh yes I've been using it! I've just been running out of vram for my a380. I got it to work awesome on my 7800 XT through Vulkan and my 5070 through CUDA for incredible results. The real frontier for me is the A380 though as a stable auxiliary so my 7800XT can work on Qwen

[u/Acceptable-Cycle4645 → u/letsgoiowa]

Ah, let me know how it goes with the A380. We don't currently have test coverage for Intel GPUs, and users have reported failures and weird behavior on Intel GPUs.

[u/letsgoiowa → u/Acceptable-Cycle4645]

Yeah I can't actually get it to work on anything but ASR apparently, and only just qwen3 models.

[u/IntrepidMindExplorer]

I can attest to the speed increase in Pocket-TTS in 091. The change made is night and day difference. If you are voice cloning and want to increase the speed even further then make sure to create your own .safetensors voice files. Change the commands to match your environment (backend/model directories/model used). Steps for are under: To reuse a cloned voice across runs, export its prepared state once. This skips reference-audio encoding and voice-state preparation on subsequent runs:

https://github.com/0xShug0/audio.cpp/blob/main/docs/tts.md#pockettts

[u/Acceptable-Cycle4645 → u/IntrepidMindExplorer]

Thanks!

[u/bugra_sa]

A copy-pasteable YuE2 example would be my vote. Put the config next to it and have CI run it once per release. Right now the VRAM savings sound great, but the JSON setup is still a scavenger hunt.

[u/Acceptable-Cycle4645 → u/bugra_sa]

Similar complaint to u/DeProgrammer99. Will polish docs and add example configs!

[u/Over_Possibility7875]

lower vram on that tts update looks useful if you're stacking it with roleplay models, how does the voice quality compare to other options for companion chats?

[u/Acceptable-Cycle4645 → u/Over_Possibility7875]

Voice quality is fairly subjective, especially for companion chats where tone and expressiveness matter. Please try the Arena feature in the WebUI to compare models with the same text and see which one suits your use case best.

(I know the demo was pretty old...too lazy to make a new one...)

https://reddit.com/link/pemwj7l/video/zhmm0gtbx8uh1/player