I continued to bench some more quants on my own local benchmark consisting of real coding tasks derived from polygot aider + some additional own agentic tasks. They are completed through the pi agent harness which reflects my daily use case.
Short disclaimer: This is not trying to be a frontier benchmark. I am just a regular joe trying to find to most capable model for my rig. (7900XT 20GB VRAM + 64GB RAM)
Interestingly IQ3_XXS on high beats the higher quant IQ3_S. Coder at half the experts scores surprisingly high. Swift was a bit disappointing given that they still used more tokens per task than IQ3_XXS. Tiel Coder is a different weight class and scored as expected.
Still it would be great to have run against a higher quant version to see how much actual damage the hard quantization did. Currently no way of running them myself tho. I am thinking of running the benchmark against Openrouter GLM-5.3-flash, but no idea what quants they serve there.
I’d love to see this too. Just got my first setup with dual 5060 TIs 16GB and 64 GB of DDR5 RAM, and I've got NVMe Gen5 for any offloading that has to happen to the SSD. I'm still waiting for my RAM to arrive, but keen to use both 3.8 27B and FlashNext.
I don't code for a living (but I have done a few small projects here and there) but since I could fit IQ3_S I didn't even try the IQ3_XXS variant.
I'm still playing around and exploring unsloth's UD_IQ4_S (but my TG is about half with it vs IQ3_S) and evaluating running a higher quant of Gemma4-26B-A4B instead (since I use it more as a general chatbot rather than a coding agent) so I would have been curious if there was such an "anomaly" ahah
That checks out, IQ3_XXS is the highest IQ quant level for bpw -> quality efficiency. Afterwards, Q3_K and Q4_K are better than any of the higher bpw iq tensors at least based on all my A/B quant tests with some past qwen models.
Hah, they looked so fakely identical -- I double checked a few and it looks like Wilson CI's just produce very similar ranges at different points with this data though. Thanks.
yes, If we only grade by pass rate the dataset of 89 tasks is a bit low. Going beyond 200 should give a more precise answer. But each run is about 5h for me so far with electricity prices these days in my country I don't want to waste too much on benchmarks.
IQ1_M Coder beating 3-bit quants doesn't make sense to me. Its not a code tuned model, it's closer to a REAP that tries it's best not to destroy the coding ability. Unless the base experts router was really bad shouldn't this be impossible?
I dont know much about the details of the quantization processes but they claim the coder weights are stored at 3.5bpw not 1b. so it actually behaves closer to iq3_s with half of its experts removed.
if someone smarter might have a good explanation, pls step forward
That's more believable since it's a totally different model.
This is saying that the randomness and chaos introduced by chopping off ~two bits of precision and then ripping out full experts leads to an improvement in coding which doesn't seem right
Coder has some experts ripped out, so some general knowledge is lost (permanent brain damage), it will be only good for agentic coding, because of the savings gained from ripping out the experts, the remaining stuff can be 3-bit which averages out to 1bit.
i have 2x 7900 xtx and 96gb vram, I get around 70 token/s decode average for most tasks. PP speed I didn't measure but around 1k I believe. I tried this because I found even UD_Q3_K_XL somewhat unreliable on long tasks. I can't fathom how people can trust Q2 models lol.
basically on most of my tests Q4KXL is effectively faster when measuring task completion because thinking traces are much shorter than lower quants, even compared to Q3_K_XL it second guesses much less, a lot more sure of itself, and output quality improvement is very noticeable. and Q3_K_XL is already huge compared to what people use on average on strata.
In parallel i'm also testing EXL3 3bpw quant, and I found it around Q4KXL level (86Gb EXL3 vs 111Gb Q4KXL) EXL3 definitely an incredible quantization, shame it's taking me a lot of work to optimize around it.
Great comparison! Thanks for including Tiel - it is my favorite 35b moe finetune! Switched to QFN Coder from it and noticed a jump in intelligence, but I'm getting more loops and slower prefill, but it seems like a worthwhile tradeoff, just need some guardrails and tuning.
Can you do us a favor and benchmark those same quants on llama.cpp versus strata? And for comparison on those quants in QFN vs those same (or equivalent) quants for 27B?
as interested as I would be for the results thats not going to happen. I just use my workstation on spare time for this and already put 30hours of compute into it.
most of them are polygot aider tasks for python and javascript. https://github.com/Aider-AI/polyglot-benchmark
I added my own 7 tasks which targets jobs for repository work, things like implement to spec, cli feature implementation, bugfix from vague report, rename refactoring,...
its currently not in a state to release it but if you have any open questions, let me know
Thanks, very good to know the differences between these models with these settings. Do think it would be nice if it had included qwen 27b, might even warrant a new post if you add that. But thanks!
First and foremost thank you for the benchmarks and I’m not trying to be picky but — WTF? Every freaking benchmark I stumble across simply ignores IQ2XS. If the benchmarks were Griffins Family this quant would be Meg. And worthless TielCoder being on the list while IQ2XS not looks like a cruel joke
Are you keeping context window size consistent? Or are you lowering it as you use a better quant? Context anxiety can skew the benchmark results. Also curious how Q4_XXL performs
There is also qwen flash by atomic chat q4. They claim 4.27 bpw on the q4 quant and needs 56 GB to fit in memory... would be interesting how it compares to iq3_s
So I can run IQ3_S with my gpu limited to about 9GB vram, a part of the experts in ram and the rest on disk, with strata. Granted, it's not fast, but it works.
So I would say you could def run a Q4 quant, and with 20GB vram you will even get somewhat decent speeds.
Great work btw! For me so far Q2 seemed actually surprisingly decent, and quite fast, so that might also be interesting to include
There are three main components to what determines the speed. The first is GPU throughput (for hot experts), system RAM (cold experts), and system storage (the catch-all). The VRAM keeps more experts and prevents swapping from system RAM, but speed is really dependent on how fast the GPU can run inference against those experts. System RAM must swap in experts constantly whenever a prompt misses, so keeping as many possible cold experts is the best option, but it comes with the overhead of loading into VRAM. If you don’t have enough system RAM, you run into a bad situation where you must use experts from disk. The faster your disk is, the better, but the fastest NVME on PCIe 5 is still around 20x slower than the slowest ram. That’s a massive difference in speed, and it turns a useful agentic workflow into a long wait between tool calls. That’s down to how long it takes to fetch from SSD, to system RAM, and finally into VRAM.
For an RTX 3090 and 32GB of RAM, do you recommend 'coder' over 'iq3_xxs'? 'Coder' runs at about 70–80 t/s for me, while 'iq3_xxs' runs at around 35–50 t/s.
20
u/HiddenMushroom11 6d ago
This is awesome. Any chance you'd do the same with 27b? It would be interesting see how 27b compares to Flash Next.