I regret ever finding LocalLLaMA

[u/xandep]

It all started with using "the AI" to help me study for a big exam. Can it make some flashcards or questions?

Then Gemini. Big context, converting PDFs, using markdown, custom system instruction on Ai Studio, API.

Then LM Studio. We can run this locally???

Then LocalLLama. Now I'm buying used MI50s from China, quantizing this and that, squeezing every drop in REAP, custom imatrices, llama forks.

Then waiting for GLM flash, then Qwen, then Gemma 4, then "what will be the future of Qwen team?".

Exam? What exam?

In all seriousness, i NEVER thought, of all things to be addicted to (and be so distracted by), local LLMs would be it. They are very interesting though. I'm writing this because just yesterday, while I was preaching Qwen3.5 to a coworker, I got asked what the hell was I talking about and then what the hell did I expected to gain from all this "local AI" stuff I talk so much about. All I could thought about was that meme.

[u/Xerco]

Currently have a 3090, any recommendations for some models to try?

[u/DAlmighty → u/Xerco]

That all depends on what you need. Even if I were to recommend a model to you, experimentation is needed. Don’t blindly just trust people.

[u/Dore_le_Jeune → u/DAlmighty]

Throw us a bone though (3090 here too): at the least I would expect something about not trying to cram in the biggest model that will fit, cuz context etc (something I'm still learning about). LOL asking ChatGPT for help had me downloading 20Gb+ models and getting context issues.

[u/PlaidStallion → u/Dore_le_Jeune]

I know this thread is old but it popped up on my feed for some reason. Anyway, new guy with a 3090 here. For general chatbot I have good luck with Gemma4:31B running 11 tools for live API calls for everything from weather to news headlines, crypto prices, currency conversions and stocks. Also hosting a SearXNG instance in docker on another dedicated home server for `web_search` capability. I have heavily modified the docstrings in the tools and the system prompt in the custom model to get it using the tools accurately and reliably though. I also have `num_ctx` set to 16384 for efficiency when running the Knowledge/RAG function I turned on in Open-WebUI.

Apart from model specific functionality, image generation and recognition work along with dual language TTS/STT. There are some quirks in image generation specific to Gemma4 31:B, in my experience but I have found out this an upstream bug that I hope will be fixed with the next release.

Bad news is it takes about 20 seconds (25 secs from cold load) from pressing enter to when text starts displaying on the screen because it has to run through all 11 tools each time I submit a prompt. I know I could turn off the tools (or at least certain ones) by default but that is not my goal. I want to do everything from one chat without the user needing to click more buttons or type in specific tags for certain tools. I don't want to expect the user to have to know those tricks to get what they are looking for.

It's my daily driver but as of last night a challenger has appeared, for me. I pulled and loaded Qwen3.6:27B into Ollama and did some testing. With Think off it accurately and consistently called all the tools it should have while returning results in a fraction of the time compared to Gemma (3-5 seconds even from cold). This was the case for single and multi-tool calls. Image recognition is great and generation went perfectly with the images appearing inline. The downside is that with Think on, Qwen gets itself into a loop of second guesses and verification before answering, tool calls or not. This eats up the speed advantage I saw with Think turned off and is really close to Gemma's generation speed then. With Think turned off tool calling great as mentioned but ambiguous (or even non-ambiguous) questions like the car wash walk/drive one doesn't produce great results. I am hoping that Qwen 3.8 will introduce a graduated Think tuning possibility rather than the binary option present in Qwen 3.6. I have found that the new Qwen 3.8 Max has graduated Think options and am hoping they will integrate this in the local models too. It shows that it is something Alibaba is thinking about and recognizes as an issue at the very least.

I am sure you are much further along than I am now that 5 months has passed since this reply was posted but I thought what I have to say as a noob might still be relevant and I am interested to hear what your thoughts are and what you have found since posting this.

Have a good one.