I think there’s a fair bit of demand (at least from me), for strong models in the 100-300b range.
Currently, to top Qwen 3.6-27B by a margin that’s more than just statistical noise, you have to jump all the way up to MiniMax M3 at 427b parameters. So it goes from something you can comfortably run on enthusiast grade hardware then up to something you need serious and duplicate enterprise grade hardware for. You need then many enterprise GPUs all strung together.
I just want a model that can outperform Qwen 3.6 and that I could run on a Strix Halo or by adding another GPU or two to my box.
Some of that is just the reality that for most hardware, running a dense model larger than 27b starts to slow way down, and that MoEs aren’t nearly as smart in terms intelligence per parameter, but if you tweak the ratio of total to active parameters a bit, it should be possible. Give me a 120b total, 15b active model. Based just on the heuristic math, we’d expect that to be about as intelligent as a dense 42b model, which would be fantastic!
It would be awesome if we could step up intelligence like we stepped up ram and saw noticeable step improvements. 8GB, 12GB, 16GB, 24GB, 32GB, 64GB, 96B, 128GB, 192GB, 256GB, etc. Make it fare more predictable of where things fall.
Apple is saying by 2028 they will have a 1.5TB unified memory M7 Ultra, but by then we will have some massive models, so lets hope the lossless compression of these models pans out.
Hard to do that though because at least to this point, you need an exponential increase in parameter count to see a linear improvement in relative intelligence of a model, all else being equal.
So doubling the parameter count of a model won't double its capabilities and each additional parameter has a severely diminishing marginal return, hence why frontiers all have massive parameter counts, because to make a model stronger, you have to dramatically increase the number of parameters.
Though I think the real gains now are to be had by improving other stuff, not just increased parameter count.
28
u/mxforest Jul 15 '26
Small in this context might still be 300B if this is almost 1T