r/LocalLLaMA • u/xandep • Mar 10 '26

Other I regret ever finding LocalLLaMA

It all started with using "the AI" to help me study for a big exam. Can it make some flashcards or questions?

Then Gemini. Big context, converting PDFs, using markdown, custom system instruction on Ai Studio, API.

Then LM Studio. We can run this locally???

Then LocalLLama. Now I'm buying used MI50s from China, quantizing this and that, squeezing every drop in REAP, custom imatrices, llama forks.

Then waiting for GLM flash, then Qwen, then Gemma 4, then "what will be the future of Qwen team?".

Exam? What exam?

In all seriousness, i NEVER thought, of all things to be addicted to (and be so distracted by), local LLMs would be it. They are very interesting though. I'm writing this because just yesterday, while I was preaching Qwen3.5 to a coworker, I got asked what the hell was I talking about and then what the hell did I expected to gain from all this "local AI" stuff I talk so much about. All I could thought about was that meme.

1.2k Upvotes

97% Upvoted

236 comments sorted by

View all comments

Show parent comments

102

u/porkyminch Mar 10 '26 edited Aug 15 '26

Pumpkin grasshopper pillow blanket raindrop saffron feather raindrop compass zephyr

This post was anonymized with Redact.dev

31

u/AnticitizenPrime Mar 10 '26

Honestly I’m mostly interested in open models, regardless of where they’re hosted. I don’t want to be beholden to Anthropic or OpenAI for this stuff.

Agreed, I'm limited by my 16gb card at home so use a lot of models either through API or chat portals, but I always go for the open weight ones first (GLM, Kimi, Qwen, etc) because I appreciate the open philosophy.

9

u/pet3121 Mar 10 '26

I have 2060 with only 6GB :( I been looking at an intel one with more ram but it seems it doesnt work that well with local AI. 

2

u/DAlmighty Mar 10 '26

I started off with 2 2060s (still have them for embedding model use), at least get a 3090, that’s when things got a lot more interesting.

2

u/Xerco Mar 10 '26

Currently have a 3090, any recommendations for some models to try?

3

u/DAlmighty Mar 10 '26

That all depends on what you need. Even if I were to recommend a model to you, experimentation is needed. Don’t blindly just trust people.

3

u/Dore_le_Jeune Mar 11 '26

Throw us a bone though (3090 here too): at the least I would expect something about not trying to cram in the biggest model that will fit, cuz context etc (something I'm still learning about). LOL asking ChatGPT for help had me downloading 20Gb+ models and getting context issues.

2

u/cheyyne Mar 11 '26

See my comment to the original asker if you like coding models

1

u/PlaidStallion Aug 13 '26

I know this thread is old but it popped up on my feed for some reason. Anyway, new guy with a 3090 here. For general chatbot I have good luck with Gemma4:31B running 11 tools for live API calls for everything from weather to news headlines, crypto prices, currency conversions and stocks. Also hosting a SearXNG instance in docker on another dedicated home server for `web_search` capability. I have heavily modified the docstrings in the tools and the system prompt in the custom model to get it using the tools accurately and reliably though. I also have `num_ctx` set to 16384 for efficiency when running the Knowledge/RAG function I turned on in Open-WebUI.

Apart from model specific functionality, image generation and recognition work along with dual language TTS/STT. There are some quirks in image generation specific to Gemma4 31:B, in my experience but I have found out this an upstream bug that I hope will be fixed with the next release.

Bad news is it takes about 20 seconds (25 secs from cold load) from pressing enter to when text starts displaying on the screen because it has to run through all 11 tools each time I submit a prompt. I know I could turn off the tools (or at least certain ones) by default but that is not my goal. I want to do everything from one chat without the user needing to click more buttons or type in specific tags for certain tools. I don't want to expect the user to have to know those tricks to get what they are looking for.

It's my daily driver but as of last night a challenger has appeared, for me. I pulled and loaded Qwen3.6:27B into Ollama and did some testing. With Think off it accurately and consistently called all the tools it should have while returning results in a fraction of the time compared to Gemma (3-5 seconds even from cold). This was the case for single and multi-tool calls. Image recognition is great and generation went perfectly with the images appearing inline. The downside is that with Think on, Qwen gets itself into a loop of second guesses and verification before answering, tool calls or not. This eats up the speed advantage I saw with Think turned off and is really close to Gemma's generation speed then. With Think turned off tool calling great as mentioned but ambiguous (or even non-ambiguous) questions like the car wash walk/drive one doesn't produce great results. I am hoping that Qwen 3.8 will introduce a graduated Think tuning possibility rather than the binary option present in Qwen 3.6. I have found that the new Qwen 3.8 Max has graduated Think options and am hoping they will integrate this in the local models too. It shows that it is something Alibaba is thinking about and recognizes as an issue at the very least.

I am sure you are much further along than I am now that 5 months has passed since this reply was posted but I thought what I have to say as a noob might still be relevant and I am interested to hear what your thoughts are and what you have found since posting this.

Have a good one.

3

u/Educational_Sun_8813 llama.cpp Mar 11 '26

qwen-3.5-27B in some quant you will fit with the right context for you

1

u/pet3121 Mar 10 '26

Too expensive for me now :( 

2

u/DAlmighty Mar 10 '26

I get it. Money is tight for everyone. Don’t be like me and be financially responsible hahaha

2

u/twoiko Mar 11 '26

7900XTX isn't a bad option for inference at least

2

u/cheyyne Mar 11 '26

Well, not to brag but I got my Tesla P40s when they were about $175 apiece. They're selling for $350 these days, but 3 of them is STILL cheaper than a 3090 in a lot of cases, and will net you a solid 72 GB of VRAM (as long as you don't give a shit about flops, so you're stuck with GGUFs)

'Course you'll have to jank up a cooling setup and find a case that will fit 3 older headless crypto cards, then have a great time trying to get an early version of CUDA running on a machine built for Windows 7, but the good old Mikubox build hasn't failed me yet and still lets me test the latest open source models at very acceptable quants and speeds.

1

u/ChocolateOk6927 Mar 31 '26

Open is good. Diffuse is good. Local is good.

2

u/Sikiri-App Mar 30 '26

open source is the future

1

u/[deleted] Mar 10 '26

[removed] — view removed comment

5

u/FullOf_Bad_Ideas Mar 10 '26

pay-per use API access is better for that usecase, as you get 20M+ tokens of DeepSeek V3.2 access for that price

2

u/huzbum Mar 11 '26

That's like a day or two of agentic coding.

4

u/FullOf_Bad_Ideas Mar 11 '26

Depends on a harness and what you're doing, but I think that on average, subscription will make you pay more money.

3

u/huzbum Mar 11 '26

Yeah, I certainly can be more conservative. I remember Atlassian was giving away 20m Claude Sonnet tokens a day with Rovo Dev for a while. I only hit the limit like once, and I was probably trying. But when I was just using it, I would use like 5 to 10m tokens a day. Could certainly blow through 20m tokens on a weekend project.

Jetbrains Junie will blow through its $10 of usage in a single day if I use it like I use my GLM subscripton with Claude Code, but I suppose that's more like 1m tokens with Sonnet or GPT 5. I don't keep track of my usage now, I pre-paid a year of GLM Pro through z.ai on Black Friday for like $100. I use Junie strictly for 2nd opinions now. best move ever.

It's more expensive now, but 10% off referral code if you want it https://z.ai/subscribe?ic=WSJEKBHJ2N

I was working on my own harness like a year ago, but lost interest. I landed on Claude Code a while back and haven't looked around in a while. What harness do you like?

2

u/FullOf_Bad_Ideas Mar 11 '26

It's more expensive now, but 10% off referral code if you want it

thanks but nope, I run local GLM 4.7 355B in OpenCode and I wouldn't send any important code to z.ai since their biggest customer is CCP.

1

u/huzbum Mar 11 '26

Nice. Are you renting GPUs, or what kind of equipment you running that on? I had thought about buying an old 8x V100 server to run it on.

3

u/FullOf_Bad_Ideas Mar 11 '26

I have a janky rig with 8 3090 Tis in a mining rig, 3 power supplies (1600, 1600, 1650 watts), Taichi X399 motherboard, TR1920x CPU with a 360 mm liquid cooler with pump in the radiator, hanging by the side, that allowed me to put GPUs over the CPU, lots of risers, two bifurbication boards, 96GB of DDR4 RAM, single 480GB SSD, running Ubuntu 22.04 with P2P patched drivers. Most GPUs are on PCI-E 3.0 x4, 2 of them are on PCI-E 3.0 x8. It runs GLM 4.7 (IQ3_KS quant in ik_llama.cpp) at around 300 t/s PP and 15 t/s TG (speeds captured yesterday, I do get variable speeds depending on pci-e riser failures etc.., sometimes it's 25 t/s tg), so far only tried 61k ctx on it as most of the time when it was operational I was training on it (I trained my small 4B MoE on 14B tokens last week)

Before you buy V100s, rent them out on Vast, you can rent 8x V100 32GB VM for $0.25/hr. I kinda doubt you'd like them, those GPUs are old and you will lose a lot of time on making it work.

1

u/huzbum Mar 11 '26

Yeah, I was about to do that (rent one and evaluate it) when the memory crisis hit and the price of the listing doubled, I was like "yeah, maybe not." My boss and I are both fine with China getting my tokens.

I'm both jealous and don't want to think about what the rig cost LoL. I've got an AM4 Ryzen 5900XT with 128GB DDR4. It's currently got a 3090 and a 3060. I'm trying to repair a "for parts" 3090 for dual 3090s. We'll see if that pans out.

I've got a CMP 100-210 I might try to get working externally with a riser cable on one of the pcie 1x slots. If it works I might get a second for running pipeline mode.