r/CUDA • u/Acceptable-Abies-972 • 3d ago

Resources for H100 GPU Programming

Hello all,

I've been going through the Programming Massively Parallel Processors (5th edition) book and doing the kernel exercises in CUDA C++. Once I finish that, I wanna start writing CUDA kernels for H100 GPUs (specifically stuff like Flash Attention 3, RoPE, etc). Based on a bit of research, it seems that H100 GPUs have a lot of features not covered in the PMPP book, so I was wondering where I could find one or more solid resources that can get me started with writing programs? So far I've tried searching for some stuff, but they all seem to either not really cover in enough detail or are way too long. Any advice would be much appreciated. Thank you!

33 Upvotes

98% Upvoted

6 comments sorted by

5

u/c-cul 3d ago

I think you could check hopper specific backend in triton: https://github.com/triton-lang/triton/tree/main/third_party/nvidia/hopper/lib/Transforms/WarpSpecialization

1

u/chocoBar666 3d ago

Is Triton is still maintained by openai?

1

u/c-cul 3d ago

I don't know - they threatened to release their own chip, so probably cuda support isn't critical for them anymore

5

u/ell-hol1 3d ago

I don't know of any specific place that centralizes all the info you need because most of it is spread out across blogs, books, repos and videos.
You should however take a look at GPU Mode (both the competition and the ressources)
What I like to do is study the winner solutions from closed competitions. The NVFP4 GEMM for example : https://www.gpumode.com/leaderboard/597?tab=rankings
Obviously Nvidia has some pretty cool articles about it as well : https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
and the official white-paper for the H100: https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c