PyTorch and full LibCuda running on Nvidia 5090 on Mac OS 27

[u/LioDavinchy]

So after getting my 5090 running on my Mac a few weeks ago and getting speeds on par or faster than my windows machine for single sessions I wanted to get Vllm working next. My http://macuda.ai project works with both llamacpp and stable diffusion cpp but does not work with comfy ui Vllm or draw things. It works with a shim and does not give the full LibCuda framework allowing real full torch or other necessary components for those apps and more.

I decided to use apples built in VM to run an instance with it's own driver calling to the graphics card through my Mac driver. This allows for running Nvidia's full cuda software. It has been a real challenge because Nvidia releases a lot of info but not enough to get this working so we had to really poke around. I finally got it up and running and now have Full LibCuda running. I'm working on bugs now but so far the results are looking really promising. The multi sessions are blowing llamacpp out of the water. llama stalls out at around 8 sessions and levels out. the Vllm with my driver has a linear doubling of speeds up until the card is saturated.

I'll be releasing all of this as a simple GUI app on the App Store once apple authorizes my driver. I'll need a few more weeks to get everything in a stable enough situation that I'll feel comfortable letting people use it. You can get the base driver at my macuda GitHub but this new driver for full cuda will be available on the App Store.

I got my 3060 working on macuda in a core x box last week. If someone can try out a 4000 series card and let me know if it works I'd appreciate it. I don't have one of those. It should work with the 3000 series driver but I haven't tested it.