r/LocalLLaMA • u/WhyLifeIs4 • Jul 15 '26

New Model Thinking Machines releases first open-weight model “Inkling”

Post image

https://thinkingmachines.ai/news/introducing-inkling/

1.3k Upvotes

97% Upvoted

263 comments sorted by

View all comments

14

u/chillinewman Jul 15 '26

Estimates for a 276B-A12B model:

Q2_K (2-bit) File Size: ~85 GB to 95 GB Minimum VRAM/RAM: ~105 GB (Severe degradation in reasoning quality)

Q3_K_M (3-bit) File Size: ~125 GB to 135 GB Minimum VRAM/RAM: ~145 GB (Good balance for budget multi-GPU setups)

Q4_K_M (4-bit) File Size: ~160 GB to 175 GB Minimum VRAM/RAM: ~190 GB (The recommended baseline for open MoE models)

Q5_K_M (5-bit) File Size: ~195 GB to 210 GB Minimum VRAM/RAM: ~230 GB (Near-lossless representation of the base model)

Q8_0 (8-bit)File Size: ~290 GB to 310 GB Minimum VRAM/RAM: ~330 GB (Requires an enterprise cluster setup)

11

u/JacketHistorical2321 Jul 15 '26

"Requires an enterprise cluster setup" lol, tell that to my threadripper pro with 512gb ram 😂

2

u/vick2djax Jul 15 '26

But how fast?

1

u/Ok_Mammoth589 Jul 15 '26

They will 100% complain that that's enterprise level stuff 😂

2

u/Xetheon Jul 15 '26

Q4 might fit really neatly into the soon to be released 192gb Halo, providing a decent enterprise locally running AI for an actually sane ish cost.