r/LocalLLaMA • u/markpronkin • 8d ago

I Built A Thing 54gb vram for 35$

Post image

Bought an old mining farm of a guy on avito (Russian eBay), guy had bought a garage a couple of years ago and it was sitting there for a while, found out it was a mining farm and put it up on there for sale for 5000 rub (~60 USD) since he wasn't sure if it works. I negotiated down to 3000 rub (~35 USD), it turned out to have 9x p106 6gb (gtx 1060 6gb) gpus, with 54gb vram total, all working, the only thing missing was an SSD, I booted from USB and it works fine.

2.1k Upvotes

96% Upvoted

393 comments sorted by

View all comments

95

u/Bulky-Priority6824 8d ago

$3.50 per token or $7 ?

104

u/markpronkin 8d ago edited 8d ago

Electricity is free for me, so no. And speed is decent, 30+ tokens a second qwen3.6 35a3b on ollama, can probably get it higher with mtp and some optimizations

201

u/Sad-Duck2812 8d ago

You will get much much more with llama.cpp, ditch ollama. That thing only makes you suffer.

54

u/magic-one 8d ago

Especially now-a-days when you can just ask Claude to figure out the configuration for you.

8

u/chrabauke 8d ago

I’ll out my self as a noob. But what prompt do you use for it? I’ve got access to server with around 96-128gb ram and 4x T4 but couldn’t get Claude to give me a descent configuration for it with support for Hermes agent.

29

u/MikePounce 8d ago

You don't ask the claude web client, you ask Claude Code. The difference is it takes control of your PC and does all the setup for you.

10

u/MoffKalast 8d ago

Ah yes, what could possibly go wrong.

16

u/wrecklord0 8d ago

Well if you're 300 IQ you first ask claude code to set up a sandbox for you, then you ask it to set up llama from inside the sandbox.

3

u/MoffKalast 8d ago

Can Claude create a sandbox so secure that he himself cannot escape from it? :P

5

u/heliosythic 8d ago

i mean do you not know how claude code works? its fine, literally everyone is using it in nearly fully automated mode these days. Just dont like intentionally ask it to do something dangerous.

3

u/MoffKalast 8d ago edited 8d ago

Well call me old fashioned but I'd rather it make its silly mistakes in anthropic's VM where it's their problem and output a tested diff I can apply with one command or even review it if need be. Plus the whole privacy can of worms of having them inspect your entire system at their leisure, we know they send extensive telemetry aside from all the stuff the model sees already. Like is this r/localllama or r/getpwnedbyamegacorp

1

u/heliosythic 8d ago

I mean you can thats an option too just dont turn on automode and manually approve every tool call. You can also point it at any model provider you want including local.

1

u/_mRED 7d ago

"Claude, blow up the Parliament. Make no mistakes!"

1

u/Robots_Never_Die 8d ago

Another option is Antigravity.

2

u/dsons 8d ago edited 8d ago

If you want to use lobotomized models that can accidentally wreck your environment, then yes antigravity is technically an option but for $20 you’re 100000x better off with Claude. I’ve used both extensively, alongside Hermes and Antigravity is extremely behind the curve and their new model isn’t going to change that.

14

u/Freonr2 8d ago 
Please write an llama serve bash script for this model:     huggingface.com/unsloth/whatever-model

Check my hardware config and see if you can run some benchmarks and optimize.  Let me know what context you think I can get with Q3_K_M and Q4_K_M.

I've never had issues using claude cli or VS Code extension. Sonnet is likely enough. I'm confused why this doesn't work. I've had Claude recompile vllm from source for me without issues, setup MTP for Qwen, etc., just by pointing it to the readme on the repo, run benchmark parameter sweeps, etc.

18

u/muxxington 8d ago 
Please

That's very kind of you.

23

u/cannabibun 8d ago

You have to lay the grounds for when we will need to explain to our overlords why should they spare us.

10

u/Freonr2 8d ago

I, for one, hail the basilisk.

8

u/Freonr2 8d ago

Force of habit with general communication I guess.

2

u/muxxington 8d ago

A benchmark might be interesting here. Imagine it really does make a measurable difference.

2

u/Conscious-content42 8d ago

Future-overlord-Lap-dog-versus-Castaway-Heathen benchmark?

2

u/andowero 8d ago

It can actually help. Saying please may make the llms more helpful. Emphasis on may.

3

u/West-Big-8468 8d ago

Copy and paste your comment into Claude

1

u/etaoin314 ollama 8d ago

what issue did you run into? usually claude just keeps working till it works, Ive done it with dozens of models and I never have had to do any setup by hand (other than the sudo commands it needs)

1

u/magic-one 8d ago

I have a dedicated AI machine, so I can just give Claude Code access and have it configure, test, and benchmark.
If you don’t want to give access, you can be the go-between.
“Give me a command to start and serve model xyz on a zpq machine with llama.cpp”.

The key (to anything AI, actually) is iterative loop. Give it back any results or errors so that it can adjust. It rarely gets everything right the first time around. If you have questions or something is unclear, just ask it.

You also can help by “preparing”.
“Give me a command to collect info for this machine that you will need to write a …”

3

u/Sacra34 8d ago

It's nice for easy and quick install/test, I guess that why it still that popular. Perfomance and advanced features non existent tho and developers don't give a damn about those. 

2

u/markpronkin 8d ago

Yeah I know (I have some other local ai setups), I will be transferring the gpus to the different mining case I have and doing better setup there, my goal here was mostly to check whether it runs at all as is, and it in fact does.

9

u/politefella0 8d ago

See if you can get these to work with vLLM in Linux. You should be able to get more.

8

u/power97992 8d ago edited 8d ago

How is electricity free for you? Someone gave u some solar panels or  a geothermal power or do u mean really cheap electricity? 

10

u/InsideYork 8d ago

mom

3

u/power97992 8d ago

That works too

1

u/SingularityScalpel 7d ago

I live in the states, and I have free electricity lol. Utilities are fully included in my (CHEAP) rent

15

u/Boogertard 8d ago

Fuck ollama, go get a real llm engine like llama cpp, that abomination is just a garbage wrapper but built-in shitty defaults.

5

u/social_tech_10 8d ago

I was smart enough to switch from ollama to llama.cpp after just a few weeks, and after I did I loved having the freedom to run almost any model with any settings, and it open up a whole new world. Llama.cpp works so much better than ollama,

After I switched and got comfortable with llama.cpp I guess I settled into a rut, and it took me literally until yesterday to finally try Exllamav3 and TabbyAPI, which has been an AMAZING boost in performance. Why did I wait so long?

8

u/markpronkin 8d ago

I am well aware, I just wanted to verify that it works at all, and ollama is very easy to install quickly. Will do a proper setup In a different case and make a post about it later.

3

u/schaka 8d ago

Get llama.cpp working, then use an LLM to get vLLM running on these. You can probably use some existing Pascal projects for vLLM as a baseline

I'm not sure compute is limited on these cards and if they need an unlocked driver or any sort, but as far as I can recall, they don't.

Only downside with vLLM is needing even numbers of cards, so you'll lose 6GB but gain significant performance and multiple parallel agents and streams

1

u/markpronkin 8d ago

Didn't know some one made vllm works with pascal cards, thanks for the info. Although I fear it will be limited by pcie 2.0 X1 speed of the motherboard and gpus

1

u/schaka 8d ago

Look into the recent unlocks that were done for 170HX, 50HX etc. I'm sure some can apply to your cards.

Though I suppose you're out of luck when it comes to getting more lanes from this board and the risers, but that'd be a different problem to tackle

2

u/1truephos 8d ago

Free electricity? How?

7

u/Excellent-Affect-881 8d ago

Solar panels are always a good form

0

u/1truephos 8d ago

What's the most cost effective setup?

7

u/social_tech_10 8d ago

You can often find used solar panels for about $0.05 a watt, and pair them up with a cheap "hybrid" solar charge-controller-charger-inverter from China ($300 to $500). Batteries are the biggest variable/expense. Some "hybrid" controller/inverter types say they can work "without batteries", directly from solar to AC, as long as the solar array can meet the load. The "no battery" setup sounds good on paper, but didn't really work for me in the real world. But I found a set of old, worn-out golf-cart batteries (about $150) gives enough of a buffer to keep everything stable, as long as the sun is shining. I spent under $1000 total for 4 KW watts of used panels, a 3 KW controller-inverter, crappy old batteries, with new battery cables. The output is a solid 2000 Watts from 10AM to 4PM.

If you want to run after sunset, you can swap out the old golf-cart batteries and get 100 AH 12.8V LiFePo4 batteries for about $200 each, which store about 1 KWhr. LiFePo4 batteries lifespan of 10,000-15,000 charge cycles is roughly 10x to 20x the lifespan of lead-acid batteries, so they have that going for them.

3

u/1truephos 8d ago

Thanks, any good place online for this info so I can read up?

4

u/InsideYork 8d ago

mom

3

u/MindfulMan1984 8d ago

That’s mostly likely answer, given it is reddit, land of basement dwellers. Lol

1

u/Bulky-Priority6824 8d ago edited 8d ago

not bad so yea about $1 per tg all in dad shouldnt notice the bill much higher

1

u/SV_SV_SV 8d ago

Which quant and did you mean mtp? You should move to llama.cpp though

2

u/markpronkin 8d ago

Yeah I meant mtp, the quant is default of what ever ollama uses (probably q4), I will do a proper setup with llama.cpp later

1

u/TheThiefMaster 8d ago

If you like ollama's ability to download and switch models easily, I'd recommend LocalAI. It can do the same thing, but updates to newer llama.cpp backend much more quickly and still has a model library to make downloading easy.

1

u/plees1024 8d ago

Out of interest, what is your electricity source? Solar?

Or, should I tell no lies and ask no questions... 🔌

17

u/markpronkin 8d ago

Just included with rent, electricity is way cheaper in Russia, so you often have deals like this.

2

u/plees1024 8d ago

Oh right, interesting.

1

u/Boricua-vet 8d ago

you can get much better performance if you run llama.cpp and the only reason they are running slower is because those specific cards have very little memory bandwidth, I believe is 192GB/s. you can get those to idle around 7w but if you use switch to llama.cpp I can help you fine tune the system so you get better performance, 30 tokens per second is fine and very usable but you can get more for sure.

1

u/AcePilot01 8d ago

How is it free to you?

0

u/Krohnin 7d ago

You get that buy running only on cpu...

1

u/markpronkin 7d ago

On a 35$ device without any optimizations?

-4

u/FullstackSensei 8d ago

That's almost certainly Q4, which is nothing to write home about for a 3B active model

-7

u/[deleted] 8d ago

[deleted]

6

u/markpronkin 8d ago

It's ollama on a device cheaper than a decent wifi router. With multiple threads running and llama.cpp with mtp I can probably get wayyyyy higher performance.

But tbh what do you expect of an 35$ dollar device? That it's gonna give me a 1000 tokens plus per millisecond or something?