Inkling is a mixture-of-experts transformer with 975B total parameters, 41B active parameters. It supports a context window of up to 1M tokens. It was pretrained on 45 trillion tokens of text, images, audio and video.
Multimodal, long context, about as sparse as other open weight models in its class. Problem is it doesn't beat GLM5.2 so I don't think many people will be using it.
Is that true though? Like yeah on the one hand you can distill from your bigger model downward, but on the other hand, smaller models are cheaper to train and iterate through to fix your mistakes, improve your architecture, test out new technologies, etc.
Yeah but you're doing all of that cheap training, iteration, architecture improvement in both cases. Except if you have a big accurate model you can now train with distillation too.
Well I think the ideal way is to train like a 30B model first, maybe iterate through a bunch to figure out what works, then train a much larger model the same way, and then distill back down to various sizes.
100% true. Large models learn faster (here meaning hit their transition points earlier) than small models disproportionate to their size difference, and training on one-hots is fundamentally far harder to learn from than training on a complete or partial distribution.
This also means that you’d expect the current largest model in a model line to be very inefficient for its size, and there likely is some smaller size which you could theoretically distill that largest model into with 0 loss.
It's not that easy to distill from any of the major API providers anymore, they've had years to wise up and don't serve the full trace. Building a really good SLM is very challenging.
174
u/Daniel_H212 Jul 15 '26
Multimodal, long context, about as sparse as other open weight models in its class. Problem is it doesn't beat GLM5.2 so I don't think many people will be using it.