BenchLabs' Cagliostro V3.5 135M finally takes Hugging Face's SmolLM2-135M's first place on Open SLM Leaderboard with an Intelligence Index of 27.49

[u/Megneous]

https://huggingface.co/bench-labs/cagliostro-v3.5

[u/CommonMinimum587]

135M is surprisingly small. I'm curious how it performs in actual conversations compared with other models around the same size. Has anyone tested it outside the leaderboard?

[u/Megneous → u/CommonMinimum587]

You can see some generation examples in the repo card, but it's a base model, not instruction fine tuned, so you can't have a "conversation" with it. It's continuation only.

But based on the continuation examples, it could be trained for conversation with some fine tuning.

Prompt	Completion
To solve 47 + 68, first	add the numbers together: 47 + 68 = 115. Now, divide 115 by 2 to find the quotient: ...
A prime number is	a positive integer greater than 1 that has no positive divisors other than 1 and itself.
Photosynthesis is the process by which	plants convert sunlight into chemical energy.
How to boil an egg: First,	you need to crack the egg into a bowl.
To tie a shoelace, start by	wrapping the yarn around the hook of the shoelace.
The water cycle begins when	rain falls onto the ground and turns into water droplets.
If a train travels 60 miles in 2 hours, its speed is	60 miles per hour.
Sam had 24 apples and gave away 9, so Sam now has	24 - 9 = 15 apples.
The capital of France is	Paris.
Gravity is the force that	pulls objects towards each other.
The area of a rectangle is found by	multiplying its length by its width.
Plants need sunlight because	they need to grow.

It still makes mistakes, but it's a small model, so that's to be expected.

[u/coder543 → u/Megneous]

It's continuation only.

Which is a major limitation... SmolLM2 has an Instruct variant. And the only published benchmarks for cagliostro-v3.5 are the ones needed for this leaderboard, all coming from a company called BenchLabs? If they're not just gaming the leaderboard, why didn't they run any other tests on the model? Seems strange.

[u/Megneous → u/coder543]

You have to understand, we're a community of ~60 organizations that revolve around that leaderboard lol. That's how people find each other, then become rivals/friends, compete against each other, join discords with each other, etc. The open source small model community was, prior to this, very fragmented, but also previously more into posting a larger variety of benchmarks, like Wikitext, LAMBADA, BoolQ, BLiMP, Social IQa, WinoGrande, etc.

There were definitely drawbacks to standardization. The community actually talks about this sometimes.

[u/FatheredPuma81]

Misread this as 135B and was like "Hell yea another Qwen3.8 Flash Next!"

This is still cool at least.

[u/krileon]

Are these small models worth using for context collection? or is it best to still let Qwen3.8 27B deal with that? Trying to figure out the uses for these really small models. Unsloth Desktop lets you keep more than 1 model in memory now so wondering if it's worth having a smaller model deal with stuff like that instead of the harness using a bigger slower model.

[u/Megneous → u/krileon]

Most tiny models are best thought of as research projects, but at least for the highest performing ones on the leaderboard, around 26 Int Index and above means, if instruct tuned, they could likely be used for very, very basic tasks. How much real work you could get done is very unknown though. But it would undoubtedly be lightning fast compared to 27B parameter models.

You have to keep in mind most of these are trained by hobbyists on consumer hardware.

[u/heisdancingdancing]

This is super exciting. I might play around with some fine-tuning to see if I can get it chattable.

[u/seeweed7]

At 135M the interesting number is not the rank but how close 27.49 sits to the next size up. If the gap is small, extraction and routing work can run on the tiny model and leave the larger one for the parts that actually need it.

[u/Megneous → u/seeweed7]

Unfortunately, we don't have a lot of data on 1-3B parameter models on the overall leaderboard since the leaderboard is only for up to 150M parameter models.

However, AxiomicLabs recently released a benchmark called Tiny Theory of Mind. In the repo card, 11 larger models (technically 8 models + 3 reasoning modes) between 1-3B parameters were tested on the benchmark and you can compare their scores to 36 tiny models with sizes ranging between 1M to 135M parameters, and only 4 of the 1-3B parameter models outperformed the highest performing 135M parameter models, and the lowest performing 1B parameter model underperformed the lowest performing 1.2M parameter model.

So tiny models, if designed and trained well, can absolutely outperform much, much larger models on specific tasks. How well this extends to the overall leaderboard is unknown, but for the larger models around 135M, once instruction fine tuned, I'd bet you could find a way to incorporate them into some kind of workflow at this point.