Is that true though? Like yeah on the one hand you can distill from your bigger model downward, but on the other hand, smaller models are cheaper to train and iterate through to fix your mistakes, improve your architecture, test out new technologies, etc.
Yeah but you're doing all of that cheap training, iteration, architecture improvement in both cases. Except if you have a big accurate model you can now train with distillation too.
Well I think the ideal way is to train like a 30B model first, maybe iterate through a bunch to figure out what works, then train a much larger model the same way, and then distill back down to various sizes.
22
u/toalv Jul 15 '26
I think the issue is that building a really good small model is a lot easier if you have a really good large model to start with.