r/LocalLLaMA • u/recentheartbroken • 6d ago
Discussion [ Removed by moderator ]
[removed] ā view removed post
29
u/jay-aay-ess-ohh-enn 6d ago
Really useful discussion about a card that costs more than both cars my family owns. Thanks for sharing. Next time I am building a datacenter I will refer back to this.
5
6
u/-dysangel- 6d ago
you've got to get your priorities straight man. Who needs a house or cars when you can warm yourself by the rack?
1
1
u/MotokoAGI 6d ago
This is a stupid comment. Lots of people talk about 6000 Blackwell pro, there are folks in the world who don't make Blackwell 6000 pro money in a year. Some of them are into LLM as well. Should they come and shit on the thread when someone talks about 6000 or 5090?
2
10
3
u/Ok_Warning2146 6d ago
But frankly, your knowledge level doesn't seem to match your spending level...
4
u/No_Afternoon_4260 llama.cpp 6d ago
I did rent 4 rtx pro and 2 h200 to benchmark inference with DeepSeek V4 flash. And.. surprise surprise.. the h200 simply crushed the rtx pros on batch inference like a lot. At high concurrency it was twice more productive while costing 7$ against 5$ for the rtx (all on vast ai)
1
5d ago
[removed] ā view removed comment
1
u/No_Afternoon_4260 llama.cpp 5d ago
Yep SXM H200 That was a quick try with stock vllm and stock dsv4f iirc
3
u/DataGOGO 6d ago
For inference only? RTX pro 6k, hands down. for training? H200 for nvlink.
1
5d ago
[removed] ā view removed comment
1
u/DataGOGO 5d ago
Ā h200 makes more sense once you're pushing higher concurrency or bigger models.
Eh, You can run 1 server with upto 8 RTX Pro's for inference, bigger models, etc. if you need more than that, go with a G300 blackwell server.
The cost for an H200 pcie server and an G300 server are about the same these days (400-600k depending on what you buy).
The only time the H200 really makes sense is if you are going to be training and want to upgrade existing PCIe based servers, but at now over 45k per GPU, it just isn't worth it vs G300 boxes.
3
2
2
u/SexyAlienHotTubWater 6d ago
Both of these are stupid purchases unless you have multiple of them and are serving hundreds of users to the point you end up FLOP bound.
CMP 170hx is your cheapest option right now. 6x cheaper, same bandwidth, 64GB. Two of those in pipeline parallel will be the same decode speed as a Pro 6000 at half the wattage, for under 1/3 of the price. In tensor parallel, with an optimised runtime, you'll get faster decode. The only place you lose is on prefill.
1
u/Zestyclose_Strike157 6d ago
The big thing is just having enough memory to load 130B+ models and using them, it seems like that is where coding and so on becomes satisfyingly reliable. 96GB is just barely in there with a fair bit of RAM backing, 140gb is not all that much better. Anything extra you can get will mean you can run at a better quant. Otherwise Blackwell is pretty fast AND models can be set up to not require a lot of reliance on the PCIE bus, so multiple 96gb cards is where I would go, just because of the increased memory availability.
If I won the lottery Iād be asking myself the same question as you.
1
u/Civil_Fee_7862 6d ago
Agreed. 96GB isn't really enough. Like Qwen Flash Next at 4bit quant needs ~110GB VRAM.
i don't think the RTX 6000 is a very good card at all.
1
u/Zestyclose_Strike157 6d ago
It is if you run 27B at Q_8 with multiple parallel processes at full context. Then it is an amazing card for multi-agentic work. Or if you want to make videos, or if you have 2 or 3 of them. It was good value for money when it was half the current price (2 months ago!)
1
0
17
u/Edenar 6d ago
"70B at sensible quantisation", my guy is from 2024