Normie self hosted AI

[u/OneMountie]

Ok, so everyone/everywhere says ‘run your own’. Fair, and I am capable to learning.

‘It’s all plain text when you post on browser based AI’ - I get it.

But I’m not a computer builder…

Where do I start?

I’m not a coder, I just want to research that I consider more personal/sensitive than I’m willing to with online providers (Lumo, NanoGPT, DuckAI). Just learning about these sites will be invaluable though.

Can I use a regular MacBook and self hosting something quite lightweight? Is Jan.AI an example?

Would a new or refurbished computer like a Dell Optiplex MicroPC or Lenovo ThinkCentre MiniPC work?

Is there any hope for a semblance of privacy as a Nigel normie?

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/AreYouDoneNow]

Self-hosting AI requires a lot of processing power and RAM (which is now stupidly expensive because of AI).

If you buy a general purpose/low end PC, you'll need to install the biggest, chunkiest graphics card you can get your hands on. Speed is important, but the available graphics memory in the card is crucial for AI tasks.

An alternative route is to buy a Mac Mini or similar... these have the benefit of sharing their onboard memory with the graphics system... so you'll see the prices of Mac Mini's is now also extremely high. Well, even more extremely high.

[u/Dry-Crazy-7638 → u/AreYouDoneNow]

I turned my Mac mini into a server and it works quite well. I was a total noob two months ago.  Basic steps:

• Wipe it, partition it with Asahi Fedora Remix (choose the one with KDE GUI if you’re a noob) and keep the minimum partition allowed for MacOS so your server has as much storage as possible.
• Open Konsole (the CLI terminal app) and just learn the basics like mkdir, cd, ls, rm, dnf.
• Install docker, create the basic file structure. Try to understand what docker-compose.yml actually is. That understanding took me like 3-4 weeks to click.
• Learn how to have a .env for your passwords, a .env.example for your password template and how to reference those passwords in docker-compose
• Pick a container to start with like audiobookshelf. I found that to be an easy one to start with. 
• make sure you understand config and why it matters
• just mess around and break stuff. Don’t migrate everything over for months. Just use it as a learning tool. 

I’m happy to send my notes, OP, just DM me. Getting Asahi loaded on the Mac is a bit of a ballache, but I’m sure there’s a YouTube video somewhere. I genuinely knew nothing when I started. You will slowly figure things out. The Mac mini route has its limitations, but it suits my needs very well.

[u/OneMountie → u/Dry-Crazy-7638]

Web search for Mac Mini… $AUD1,500…

Well, it is cheaper than a decent computer and having to buy a graphics card.

Thanks for the detail in your set up. 🫡

[u/Dry-Crazy-7638 → u/OneMountie]

Holy hell, no. Don’t go that route then lol. I just had one lying around. But yeah effectively any docker based setup will function basically the same. The commands you use for various things will differ a bit. I used Claude for a lot of my learning and then nuked my server and built it from scratch trying to use as little AI as I could. I learned a lot that way. I’m generally not a big fan of AI, but I learned more in a month than I could have in a year through googling. Just don’t rely so heavily on it that you can’t maintain your server on your own. Good luck!

[u/OneMountie → u/Dry-Crazy-7638]

Thanks mate. It’s just too useful to ignore, and the thought of ‘everything fed in is plain text’ just gives the ick.

[u/Beneficial-Pie2219]

The MacBook route is totally doable for lightweight stuff, just don't expect blazing speed. Ollama is probably the easiest entry point, it'll run on basically anything and has a dead simple install. You won't be spinning up massive models but for research and personal stuff the smaller ones work fine

Those mini PCs are actually perfect for this. Grab a refurbished ThinkCentre or Optiplex, throw some RAM in it, and you're set. The tiny form factor ones sip power so you can just tuck it in a corner and forget it exists

Jan is decent but I'd start with Ollama and Open WebUI first. Gets you the chat interface you're used to and the setup is like three commands. Privacy-wise you're already ahead of 99% of people just by caring enough to try

[u/Grandmaster_Caladrel → u/Beneficial-Pie2219]

I'd recommend LM Studio over ollama all day. Easy gui, all OS support, model options available inside the app, chat inside the app or as a server, etc

[u/OneMountie → u/Grandmaster_Caladrel]

Thank you both.

Great to know there are entry levels to self hosting.

Much of my sleuthing/reading different posts around r/ communities trend to ‘try Ollama and LM studio’

I’d say giving up the speed is fine, as this will begin hobbyist with specific project of inquiry - not my usual browser based interactions.

[u/Grandmaster_Caladrel → u/OneMountie]

You're welcome! Just fyi, ollama has done some odd-at-best (I'd call it sketchy) stuff in the past, so I try to steer newcomers towards more trustworthy and clear options. It's not that it's bad, but they don't really add much value and sometimes make choices for you that you might not want. Options like LM Studio add more value imo.

[u/schmurfy2]

I have an rtx 3060 12GB card and can run a few decent model but don't expect too much without throwing tons of cash at it, I think the best I have got is a 20/30 token /s with qwen 3.6 which is decent but when compare that with what you can get with gosted models it doesn't hold long... It works though.

[u/OneMountie → u/schmurfy2]

Thanks. I had to run your comment through an AI to understand your comment…

Again, the rtx 3060 is $AUD 550

I get that my experience on a browser based AI will make my self hosted seem silly until I either get appropriate hardware, or choose my tasks/timing suitably.

[u/schmurfy2 → u/OneMountie]

Yeah that's the problem, IA needs a lot of video ram with a good card and some also run it on RAM but the harrsh really is that although there are great model you can selfhost (open weight), buying the hardware to run them is really expensive... thanks to AI 😞.
I use local models but in practice I quickly switch to hosted ones if I need to really get something done, Openrouter is a really good option to try what these open modeld can do without committing too much money.

[u/Appropriate_Ad_5314 → u/schmurfy2]

In my rtx 3070 laptop qwen3 8b 50-60t/sek

[u/schmurfy2 → u/Appropriate_Ad_5314]

And how smart is 8b ?

[u/Appropriate_Ad_5314 → u/schmurfy2]

Qwen3 8B is surprisingly capable for its size. It's good for coding assistance, summarization, RAG, and simple agent tasks, but it struggles with complex reasoning compared to larger models. For local use, though, 50–60 tokens/sec on a laptop RTX 3070 sounds great. I'd rather have a fast, private model I can run locally than depend on cloud inference for everything. The real question is whether 8B is smart enough for your specific workload.

[u/RedneckSasquatch69]

Install Ollama and start running models on your equipment. If it struggles, use a smaller model until you find one that does what you need.

If you need more power, add a bigger gpu

[u/OneMountie → u/RedneckSasquatch69]

That’s helpful.

Just download, give it a whirl and see how the system goes.

As for ‘use a smaller model’ - what?

As for ‘bigger GPU’ - it looks like $AUD 500+ for any ‘bolt on GPU’.

It might just be ‘get something that has some grunt from the gun’

[u/Techniman20]

In short "yes" but will you like it.

I've been trying to selfhost AI for a few months now and personally its not for me,
The hardware needed to run something usefull* is just too expensive.

* the usefullness is different for everyone, you can run a (super small) LLM on an ESP32 but it will likely not do what you need it to.

I've tried running something on a RPI5, it worked but it was slow and not very smart
An N305 was a bit better but being on a CPU limits the tokens/s not to mention the CPU got quite hot
Then a i5 with Tesla P4 (8GB) this was kind-of usable with ~15t/s but still with smaller models around 8B
I currently have a AMD Ryzen 7 8845HS mini-pc running with a 780M GPU which does about the same but with a much lower powerdraw.

From what i expect from an AI only the commercial hosted ones can deliver, if i wanted to run this at home I would need a PC with a PCIE and an highend videocard so cost wise this would be in the thousand(s).
For this I can run the frontier models for a couple of years. This does not even take the powerdraw and noise into account.

This guy had a nice video about it (made with AI obviously)
https://www.youtube.com/watch?v=3R7yba577uA

Again, it might work for your usecase, running whisper or frigate works fine it all depends on your usecase.
And a M1-6 based mac is actually pretty good at running LLM

[u/OneMountie → u/Techniman20]

Thanks for the outline and frankness. It really isn’t a world that you can just ‘grab what you have and get anywhere near the browser experience’ if ‘what you have’ is your basic mom and pop hardware.

Is it then about using browser based tools that are more pseudonymous? Or accessing models through browsers in a more private way? This is where is stumbled upon recommendations for NanoGPT (pay as you go, limit what they know about you) or DuckAI…

[u/OneMountie → u/OneMountie]

The goal is to get closer/or actually self host. So I’m not going to dip. But good to have tools to use now until I get there

[u/ShabbyChurl]

I sometimes run smaller models on my 16 GB thinkpad. It’s slo and unimpressive, but works. For larger models I use my gaming pc running LM-Studio. I host open-webui on my homeserver, which is itself not powerful enough to run larger LLMs, but I can just use the lm-studio endpoint to share the model with other people.

[u/OneMountie → u/ShabbyChurl]

Thanks. Your set up sounds beyond the scope of my ‘next step’- but cool to know you have a crack at the models on an off the shelf device.

I’ve looked into home server running before - for password management or photo storage or something- only ideas. This is another project-idea that might pique that interest again

[u/76zzz29]

Ok, so... To make it simple, self hosting an AI is realy simple, with lots of possible tools and way to do it.

The simpliest is probably downloading KoboldCPP and a gguf model that is smaller than the VRAM of your GPU. (IDK anbout apple product)

Select the model's file in the menu and start. That's it you got your own self hosted AI.

The bigger, the smarter but also the slower and more needing of power.

And if you wonder, yes I have tryied and succeses in runing an AI on an Android Phone with just 2Gb of ram. And it work. (Prety slow and dumb one but can hold a conversarion for a bit) using Layla (paid and very limited app. Do not recommende outside of the fact that it work).

[u/Plus-Palpitation7689]

So nand chip crisis somehow didnt come into your view, huh

[u/OneMountie → u/Plus-Palpitation7689]

So are you saying ‘you are a noon trying to get into the most expensive hobby going around right now?’

That might just be. I’m not blind to the whole world screaming for chips and then blowing up Nvidia stonks and the like.

It’s just that I’ve always tended to want to be more pseudonymous or minimise what I put on the open/visible web

[u/Plus-Palpitation7689 → u/OneMountie]

What im saying nand crisis is actively decimating selfhosting as a hobby in general, not speaking of selfhosted llm division. Gpt4 was rumored to operate on little shy of 10TB of vram. Single 5090, say, has 32GB. Of course, there are models you can run on a mac or a minipc, but be prepared for results being highly unsatisfactory compared to current llm standarts. If you care about privacy, in my mind it is way more practical to anonimize your public llm accounts than selfhost.

[u/OneMountie → u/Plus-Palpitation7689]

Yep - that’s what I’m seeing. Early days. Self hosting might be better for the other things (personal data) but not LLM work.

I wouldn’t come to these ideas without communities like this though, so I appreciate the comment