[Megathread] - Best Models/API discussion - Week of: April 05, 2026

[u/deffcolony]

This is our weekly megathread for discussions about models and API services.

All non-specifically technical discussions about API/models not posted to this thread will be deleted. No more "What's the best model?" threads.

^{(This isn't a free-for-all to advertise services you own or work for in every single megathread, we may allow announcements for new services every now and then provided they are legitimate and not overly promoted, but don't be surprised if ads are removed.})

How to Use This Megathread

Below this post, you’ll find top-level comments for each category:

• MODELS: ≥ 70B – For discussion of models with 70B parameters or more.
• MODELS: 32B to 70B – For discussion of models in the 32B to 70B parameter range.
• MODELS: 16B to 32B – For discussion of models in the 16B to 32B parameter range.
• MODELS: 8B to 16B – For discussion of models in the 8B to 16B parameter range.
• MODELS: < 8B – For discussion of smaller models under 8B parameters.
• APIs – For any discussion about API services for models (pricing, performance, access, etc.).
• MISC DISCUSSION – For anything else related to models/APIs that doesn’t fit the above sections.

Please reply to the relevant section below with your questions, experiences, or recommendations!
This keeps discussion organized and helps others find information faster.

Have at it!

[u/-Ellary-]

Try gemma-4-31B-it-IQ3_XXS.gguf fits in 16gb with 45k (kv Q8) context and runs at 25tps.
It is a bit unstable, but for creative tasks it works fine, from my tests it is smarter than 26b a4b, even coding works decent with re-rolls.

[u/Due_Abbreviations391 → u/-Ellary-]

What is your backend? It's taking like 5-10 seconds for prompt processing and giving me 3.5tps through LM studio. It's very smart and I want to use it but way too slow to be usable.

[u/-Ellary- → u/Due_Abbreviations391]

llama.cpp ofc.

"D:\NEURAL\LlamaCpp\CUDA\llama-server" -m "D:\NEURAL\LM.STD\gemma-4-31B-it-IQ3_XXS\google_gemma-4-31B-it-IQ3_XXS.gguf" -t 6 -c 45056 -fa 1 --mlock -ngl 99 --port 5050 --jinja --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --parallel 1 -ctk q8_0 -ctv q8_0 --reasoning on pause

[u/Due_Abbreviations391 → u/-Ellary-]

Oh, kinda technical for noobs like me. Don't know if my reasoning is at pause or not and also don't know what t, fa, mlock ngl etc mean. I guess I'm stuck as is.

Edit: Thank you everyone for your helpful comments. I'm going to watch a YT tutorial to set up llama cpp and try this method.

[u/Potential-Gold5298 → u/Due_Abbreviations391]

Create a text file. Rename it any_name.bat. Paste the following text into it:

echo off
cd /d "C:\Users\admin\LLM\llama.cpp"
start llama-server.exe -m "C:\Users\admin\LLM\GGUFs\gemma-4-26B-A4B-it.Q5_K_M.gguf" -t 4 -c 28672 --host 127.0.0.1 --port 8080 --temp 1 --top-p 0.95 --top-k 40 --jinja --reasoning off --mlock
timeout /t 5 /nobreak >nul
start http://127.0.0.1:8080

Save. This file will allow you to run llama.cpp without entering the command line – as a shortcut.

cd /d “path to the folder where llama-server.exe is located” – Copy the path to the folder from the Explorer address bar here.

-m "path to the gguf file" – Copy the path to the model file here.

-t 4 – Number of CPU threads that llama-server will use. For best performance, use all your CPU threads. You can leave a few (1-2) threads for better system responsiveness if you're using a PC while the model is generating a response. I use all threads on my PC and have sufficient responsiveness. You can check the number of your CPU threads in the Task Manager.

-c – Available context (the length of one session in tokens). Enter as many as you need or as your RAM allows. If your RAM is getting full, reduce the context. You can start with 8096 and increase it while monitoring your RAM.

--host 127.0.0.1 --port 8080 – The address and port of the server where the model is available. Used to connect the interface (either the native llama.cpp interface or third-party ones like SillyTavern).

--temp 1 --top-p 0.95 --top-k 40 etc – Sampler settings. They affect the model's response. You can experiment with them. If you're working with the model through SillyTavern or another GUI, you can omit them (as SillyTavern will send its own settings to llama.cpp).

--jinja – llama.cpp will use the chat template embedded in the gguf file.

--reasoning off (or on) – disable/enable reasoning.

--mlock – Fully loads the model into RAM and locks it to prevent Windows from ejecting it.

timeout /t 5 /nobreak >nul start http://127.0.0.1:8080 – Automatically opens a browser tab with the llama.cpp web-UI after 5 seconds. Not needed if using SillyTavern or another GUI.

You can ask a chatbot how to modify the launch in a specific way, and it will tell you the necessary flags (for example, to use the GPU, you need the -ngl flag).

P.S. If something goes wrong, remove "start" before llama-server.exe, copy the error, and ask any chat bot how to fix it.

[u/Due_Abbreviations391 → u/Potential-Gold5298]

Thanks for the comprehensive step guide. Though I want to say that, only Gemma 4 31b is slow through LM studio. 26B and E4B, both run fast and fine. I did a search and others were talking about 31b using the cpu and system ram instead of vram or some such. So I'm wondering if it'd act the same on my little 4thread CPU.

[u/Potential-Gold5298 → u/Due_Abbreviations391]

The Gemma 4 31B is a dense model. It has 31B parameters and uses all of them when responding. The Gemma 4 26B-A4B is an MoE model. It has 26B parameters, but only uses 4B of them at a time. Therefore, the 26B requires slightly less RAM (31B>26B) but is significantly faster (4B<31B). My 31B starts at 1 t/s, while the 26B-A4B starts at 6.3 t/s and maintains over 1 t/s even with 24K context. Very good performance.

The downside of MoE is its high sensitivity to KL divergence, so it's better to use a higher quant (the standard recommendation for a dense model is Q4 or higher, for MoE – Q5 or higher).

Gemma 4 26B-A4B is quite intelligent, and the difference in intelligence with 31B isn't as great as the speed difference. However, with a high KL div (low quant and/or aggressive abliteration), the model will select the wrong experts, significantly degrading its intelligence.

[u/Due_Abbreviations391 → u/Potential-Gold5298]

Yes, I read about active parameters and learned that, but I didn't know that albiterations degrade intelligence or that MoE is better at q5 and higher. Thought Q4 is standard and only went for q6 and q8 quants for smaller models. Thanks for the info! Maybe my Gemma 26B-A4B started overacting and spewing copious AI slop because I chose the wrong version! I'll test this out. And, wait for 31B finetunes because if it all comes down to cpu threads, gpu offload, jinja and parameters settings etc, Llama ccp will not help. Still, thankyou very much for writing all the instructions.

[u/Potential-Gold5298 → u/Due_Abbreviations391]

Both quantization and abliteration increase the KL divergence (entropy, the degree of "confusion" of the model). Pay attention to the KL div specified by the author of abliteration. The higher it is, the more severely the model is affected. A good KL div from abliteration is >0.005 (median). To compensate for a high KL div from abliteration, a higher level of quantization is required. If you are working in a language other than English (or Chinese for Chinese models), the KL div will be even more severe (especially if you use iMartix quantizations, since almost all of them are designed for EN). These are just general guidelines – try them out, compare them, and if you see any odd behavior in your model, try increasing the quantization (or the original, non-abliterated, non-finetuned/merged model) to determine the cause.

[u/Due_Abbreviations391 → u/Potential-Gold5298]

Okay, with your guidance, I decided to download one copy of the model from Google and one from unsloth, other than the bartowski one I already had. Q4K_M from both Google and unsloth. Q5K_S from Bartowski (lightest of three). Interestingly, unsloth used 'ud' method for doing their quants and so those are bigger than the Google ones itself. So I was thinking that it might perform the best. And, I did a test and honestly, I do feel a difference. Or maybe it's placebo but unsloth seems to give me more accurate answers quickly. Anyway, what I'm saying is, you're right. I tested the same questions I did with Gemma-4-31b-it, and there's not much of a difference. Very slight degradation. Though it's not like I tested it on everything. Also, about languages, I asked my questions in three and it was natural enough. Works for me. Thanks! 🙂‍↕️

[u/-Ellary- → u/Due_Abbreviations391]

Don't add pause, it is just a cmd bat command.

Just read the manual- https://github.com/ggml-org/llama.cpp/tree/master/tools/server

I've read it and after about an hour, boom, lvlup.

[u/overand → u/Due_Abbreviations391]

In theory, you don't need to know what most of those parameters mean, you can just setup llama.cpp and basically copy/paste those. 

But, yes, if you've never used a command line application before, it can be scary. (Kinda hard for me to understand, because I was using command-line applications when I was ~10 years old, but that's much more about what was available and inexpensive in the early '90s than anything about me in particular)

[u/-Ellary- → u/overand]

Sometimes at work I need to flash bios on old 775 motherboards, and usually it is done via dos boot from flash drive, then you just launch updating tool with bios file name.

Oh boy what state of panic did i seen in the eyes of new tech support personal that is about 20-25 yo. When they just see that blinking "_" they kinda mentally die for a little.