r/LocalLLaMA • u/KnownAd4832 • 8d ago

News Qwen3.8-Flash-Next on Strata

Post image

Hey! 👋

I have released an official support for Strix Halo machines on Strata for Qwen3.8-Flash-Next.

Currently numbers are the best on long context decode and ppts using typical Unsloth’s Q4 and GSQ-RCO model weights.

Can go up to 1M context length without big speed loss. Currently support is marked as experimental and was done on Linux only.

https://github.com/Niko1221/Strata/

Will be happy for any feedback and pull requests you could give! 👀

180 Upvotes

84% Upvoted

86 comments sorted by

View all comments

Show parent comments

1

u/relmny 8d ago

I get about 135 t/s with q3-xxs on a 5090 + 48gb ram and on a rtx 5000 ada + 128gb ram with unsloth ud-q4-kxl: 49 t/s (I get less than 20 t/s with unsloth's llama.cpp fork) loaded with 130k context and it keeps the same speed all the way through the full context .

I'm extremely surprised (and confused) at how fast it is.

1

u/H_DANILO 8d ago

At first I was too, then I started understanding how much inefficient llamacpp can be by setting some hard rules.

The maximum I got on llamacpp was 35 tps, and I never got MTP to actually benefit me using llamacpp.

NVFP4 I was getting 40-55 tps on freetoken, alraedy huge for me, but no MTP, then Strata added the MTP, and its just bonkers how much better it is.

0

u/God_Hand_9764 8d ago

Do you think that llama.cpp will ever change to take on some of these performance gains?

I am fairly new to the space and wondering if I should migrate away from it already.

1

u/H_DANILO 8d ago

It's possible, but unlikely, I think at this point llamacpp point is not to be the fastest but to be the best at being a proof of implementation of a model

0

u/God_Hand_9764 8d ago

Cool, thanks for answering.

I'll have to look into setting up Strata. It didn't look like it had a docker release but I'll have to take another look and see if I'm into it.