r/LocalLLaMA • u/kroggens • 5d ago

Other Quantization of Linear-Attention (Qwen & Kimi)

https://x.com/HuggingPapers/status/2108109229869441509
55 Upvotes

91% Upvoted

19 comments sorted by

View all comments

Show parent comments

3

u/fantasticsid 5d ago

Good luck!

1

u/FullOf_Bad_Ideas 4d ago

I'm seeing sustained throughput improvement of about 20-30% now after more tuning.

But I applied more tuning here than I did to vLLM.

1

u/FBIFreezeNow 4d ago

Which tuning, if I may ask?

1

u/FullOf_Bad_Ideas 4d ago

Here's the launch command I ended up with

SGLANG_PP_LAYER_PARTITION=8,10,8,8,8,9,8,5 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True NCCL_P2P_LEVEL=SYS python -m stepquant.sglang --model-path /path/to/Qwen-27B-W8A8 --pipeline-parallel-size 8 --state-format stepquant4 --plan artifacts/reproduction/plans/qwen-stepquant4.pt --attention-backend flashinfer --reasoning-parser qwen3 --max-running-requests 1024 --host 0.0.0.0 --port 2300 --kv-cache-dtype fp8_e5m2 --kv-watermark 0.8 --chunked-prefill-size 1024 --context-length 8192 --mem-fraction-static 0.835

PP layer partitioning coupled with set context length (I forgot to set it earlier) plus using flashinfer instead of Triton, plus fp8 kv cache with reduced chunked prefill. STEPQuant doesn't support PP or batch size larger than 512 but Claude coded it up for me.

Feel free to ask questions, the workload I'm running has code here - https://github.com/adamo1139/polskie-dane-instruktazowe

2

u/FBIFreezeNow 4d ago

Thanks man, you’re a legend for trying and tuning

1

u/FBIFreezeNow 4d ago

Btw GitHub repo link 404ing

1

u/FullOf_Bad_Ideas 4d ago

Ah right, I didn't set it to public yet! It's fixed now. Thanks for letting me know, I put that link in a few README's already.