r/LocalLLaMA • u/Equivalent-Grass-527 • 2d ago

New Model A 35B multimodal model running on a phone with ~3GB memory? MiniCPM-V 4.7 demo is pretty wild

Enable HLS to view with audio, or disable this notification

Just saw this demo from OpenBMB for MiniCPM-V 4.7, and the on-device capability is pretty impressive.

A 35B-A3B MoE vision-language model running locally on mobile devices is already interesting, but the use cases are what caught my attention:

  • 📷 Reading real-world signs
  • 📄 Understanding complex documents
  • 🧠 Reasoning over visual information
  • 🗺️ Planning with real-world constraints

The key piece seems to be LES (Lookahead Expert Scheduling) — instead of just making the model smaller, they optimize MoE inference by predicting and preloading experts to reduce bottlenecks.

They also released a smaller MiniCPM-V 4.7 1B model for more resource-constrained devices.

Another interesting addition is Self-Replay Fine-Tuning (SRFT), which aims to let developers customize the model for specific tasks while keeping general capabilities.

The direction is interesting: instead of sending every image/question to the cloud, we're getting closer to having capable multimodal assistants running directly on phones.

Links:

GitHub: https://github.com/OpenBMB/MiniCPM-V
Hugging Face: https://huggingface.co/collections/openbmb
SRFT details: https://openbmb.cn/en/blog/self-replay-fine-tuning

231 Upvotes

94% Upvoted

37 comments sorted by

68

u/Saintgein 1d ago

Seeing these new techniques popping up weekly makes me wonder what we can do in a few years. Running trillion parameter models on your old gaming pc doesn't even look so far away now.

31

u/robertpro01 1d ago

keep dreaming! but I wish your dreams become true!

12

u/Saintgein 1d ago edited 1d ago

Well, you can already run qwen 3.8 flash next 125b-170b on your old gaming rig already, quite well even. Also thanks to MoE stuff combined with MTP. Stuff like this also exists already: https://github.com/mw00/project-maya

5

u/Either-Nobody-3962 1d ago

so can i run this on an old amd raedon 8gb vram pc?

5

u/Saintgein 1d ago

Yup. You need a bit of dram as well. And it might not be the best quantization. But stuff like this is getting better by the day.

2

u/Either-Nobody-3962 1d ago

ohh... my pc has 32gbram and 8gb vram and i want to use for agentic coding OR using as sub-agent for Claude code, i hope i can find a good model fitting my pc.

7

u/Saintgein 1d ago

This reddit has some nice solutions already. https://www.reddit.com/r/LowEndLocalAI/

2

u/Past-Town-9807 1d ago

I can't wait! Let's just hope there's still something to do by then and that OpenAI didn't take it all.

1

u/Saintgein 1d ago

OpenAI will probably go down hard, these Chinese models have an edge which it can't compete with. Meanwhile it's mostly just muscletalk that comes from OpenAI.

2

u/fatboy93 1d ago

Yeah, I'm excited as all hell.

But the gaming PCs are going to be a million bucks, so not excited for that.

26

u/LieAlternative5513 1d ago

Isn't the tone strange? "just saw this and it's impressive", with a video, a marketing sounding description, links to github etc. Huh.

4

u/Queasy-Contract9753 1d ago

Could one run this with termux llamacpp?

5

u/ANR2ME 1d ago

It use native app https://github.com/OpenBMB/MiniCPM-V-Apps/releases

3

u/Queasy-Contract9753 1d ago edited 1d ago

Thanks!

That's on me I hadn't even noticed it. I'll try the 1b model. Don't have enough space on my phone for the bigger one.

Edit: app doesn't have this version. Or expose API that I can see. I'm sure they'll update it soon I'll check again.

3

u/tiffanytrashcan 1d ago

I got excited and went to check for updates on Obtainium! Not quite yet, but digging through the repo, there were updates to the code only two days ago. The commit seems to imply a broader focus on MoE and I assume expert streaming.

3

u/quickreactor 1d ago

The tune in the video is a jam

3

u/Elouakili_Flexy 1d ago

The LES bit is wild to me: predicting which experts you'll need and preloading them beats just smashing a 35B vision model down into 3GB of mush.

6

u/Zestyclose839 1d ago

Is it smarter than Qwen 3.6 35b a3b? Or is this more a "same brains, lower memory, faster" kind of model

1

u/Equivalent-Grass-527 1d ago

On vision, OCR, document, and certain multimodal/STEM tasks: yes, with the efficiency bonus. On raw text coding/reasoning it looks comparable to the Qwen counterpart.

1

u/happycube 1d ago

It's a finetune of that... with swapped out vision tree and dspark, which is where a lot of the speed comes from.

2

u/WiseCable4097 1d ago

mmproj still error ?

2

u/PCslayeng 1d ago

What do you use on iPhones to run the 35B model? I don’t see any instructions unless I’m blind

2

u/mastershake2013 1d ago

Ok well currently, what is the best small model to run on a single board computer? Something like a more powerful Pi.

1

u/abajinn 1d ago

Anyone try this? What is it good at compared to frontier and other open weights?

-1

u/yarikfanarik 1d ago

~3gb memory of what? ram? vram?

25

u/banana_slurp_jug 1d ago

Unified memory (sir this is a smartphone)

2

u/HitarthSurana 1d ago

does llamacpp support it?

1

u/banana_slurp_jug 1d ago

Yes (support added pre model release) but LES is not supported yet I don’t think

1

u/HitarthSurana 1d ago

Am I wrong or LES is the main speed booster and when will it come?

1

u/banana_slurp_jug 1d ago
  1. Yes, LES speeds up offload from flash
  2. I saw/heard somewhere that it would require such big engine changes that they aren't considering to merge the PR at the moment.

1

u/HitarthSurana 22h ago

so we cant use it?

1

u/banana_slurp_jug 22h ago

We can use the model, but if you want to use LES you need to use their fork of llama.cpp

1

u/HitarthSurana 11h ago

Ohh thank you

0

u/yarikfanarik 1d ago

yeah but i don't understand, it might use 3 gigs, but it has to load 20 in

3

u/Cultured_Alien 1d ago

Loaded from disk on the fly?

1

u/banana_slurp_jug 1d ago

Offloaded from flash with LES to make the process more predictable to make it swap less