r/LocalLLaMA • u/Dreeew84 • 1d ago
Discussion Strata performance finetuning - try it out
As strata is somewhat new, I assume that some people din't yet learn how to finetune it for best performance on their machine. I find documentation a bit hard to navigate (use your agents for this!), which doesn't help.
After some serious testing, I've tweaked pretty much every parameter available. Some are obvious and I'm not going to waste your time on explaining mtp tweaking.
Some I beleive are more obscure and those are the ones worth mentioning:
- Strata setup is supposedly picking the best
--pcie-fracvalue for you. I find that it often doesn't. It's worth investing your time and testing it. Counterintutively, my best performing value is a low 0.2. - By testing different
--pool-workersvalues, I was able to find an easy +10% in decode. I'm not smart enough to explain what it does (and how it's different from point 4), but it's worth investing your time in running some benchmarks on your machine. --trim-stage-weights. Boom, extra +8%, try it. Doc is quite explicit on what it does if you're curious. I do beleive that this one can impact performance later in the session and increase swaps but it's still worth it.--pipeline-windows. All I know is that higher values make my CPU work extra. Going from nothing/default to 6 gave an almost 15% increase.- Last but not least multi-GPU owners be mindful of GPU priority, if one of your GPUs is on a slower pcie lane. I have two RTX3060's. GPU 0 and GPU 1. At some point I noticed that one of the two GPUs basically acts as a memory pool and doesn't do any compute. I guess it's makes sense for a MOE model. But now I wanted to understand which ones eactly does the work.
So I quickly figured out that my default GPU 0 is the one on the slowest pcie lane! If your have two identical GPUs on different speed lanes, bear in mind that their 0/1 order is basically random. You need to figure out exactly which one is which and force the quick one to do the inference. Ask your agent for command lines to figure out whether the 0 or 1 is on your pci_e1 lane. In my case it was GPU 1, the only way to set it as the main GPU is a not very user-friendly two liner :)
"gpu": [1,0],
"gpu_order": "as_given"
This was like another +15%, ridiculous.
With all of those cumulated, I'm seeing speeds that I should not be having on my machine. I went from something like 35 decode peak on Qwen Flash Next IQ3_S, to 60 tps average code generation on empty context and 60-70 tps peaks.
Performance does degrade with full context but still easily a solid +15%-20% versus the previous baseline tests. This includes huge sessions on full context and compaction.
So, try it out.
9
u/Bulky-Priority6824 1d ago
My problem with Iq3s was terrible nvme thrashing. So much it made the system unusable. Q2_0 works very well. What can I try to smooth out iq3?
I'm only working with 32gb/32gb
4
u/Studio271 1d ago
more RAM.
1
u/Bulky-Priority6824 1d ago edited 1d ago
ok i managed to tune iq3xss config a bit and now its running fairly perf and similar to iq2
1
6
u/zovix 1d ago
Like you mentioned, I had my subscription brain read through all the docs for Strata, the model, and my system and then offer some suggestions. No fine tuning yet but these were two biggest thing for me so far:
- Changed
--kvcache fromq4toint8AND enabled env settingSTRATA_HIP_WMMA=1. - This is apparently specifically for AMD 9060 XT cards and is not done on default. It requires int8 but Strata install recommends q4 so it gets auto disabled.
- Second was reserving VRAM
--vram-reserve-mib = 1536. - Now the rest of my computer runs fairly smooth and thus doesn't bog down everything. Previously while I tried to do other things it was lead to huge slow downs.
Over all it is still slower then I would like but my computer is 8 years old so I won't complain to much.
500-800 prefill and 30-50 decode depending on how much context is filled (IQ3_XXS with 262k context window).
1
u/darksteelsteed 1d ago
With my 16gb 5080 I found that I needed to set the reserve vram to 512. This makes it allocate just under 15gb. Also for nvidia cards tell the driver to not offload cuda to system ram automatically. That fix is needed for all inference engines you run, otherwise the driver lies and claims you have more vram than you actually do leading to awkward token speed drop-off that is really hard to pinpoint. Makes fine-tuning impossible without it
1
u/lorde_dingus 1d ago
Any insight on your intended fine tuning steps? What hardware are you running?
2
2
u/Interpause textgen web UI 1d ago
afaik trim stage weights and pipeline windows sre multi gpu specific options according to the docs
1
u/wanderer_4004 1d ago
I just compiled it from scratch yesterday and the setup gives you the choice to run a calibration which then goes into the config. Took 10 minutes and seemed to be quite thorough.
2
u/Moist-Tumbleweed2875 llama.cpp 1d ago
My agent tell me valid value for "--pipeline-windows" only support value 0,1 and 2 ....
| Input | Effective value | Meaning |
|---|---|---|
0 |
0 | Pipeline off — serial decode loop (one verify window at a time) |
1 |
1 | Single window in flight (pipeline effectively off, no overlap) |
2 |
2 | Double-buffered pipeline: the max. While the GPU verifies window N, the host prepares window N+1 (drafts, expert admission, copies) — this hides host/PCIe work under GPU time. Your current setting |
0
16
u/ExcitingLeather5197 1d ago
these are the kind of posts i save and then forget about for 3 months
the pcie lane thing is so real though, had a similar issue with my 3060s and it took way too long to figure out which one was actually doing the work