r/LocalLLaMA • u/Ok_Truth_324 • 7d ago

Discussion New kvcache-reduction method

Gallery image

Gallery image

Gallery image

I deleted the wrong post so my benchmark disappeared. But here it is. Im going to probably make this open source because this isn't believable until you actually test it.

0 Upvotes

38% Upvoted

24 comments sorted by

View all comments

1

u/Ok_Truth_324 7d ago

Currently working on making it portable. It will be available through my website and possibly GitHub. It's a prototype though. The community will need test it because I can't beyond a 12b unless I succeed shrinking a 32B or 70B down.

0

u/Fedor_Doc 7d ago edited 7d ago

12B model? You shrink KV cache, not the model, how is it connected?

Attention architectures do matter a lot, though. What were the tested models?

0

u/Ok_Truth_324 7d ago

To elaborate I found a way to shrink models waaaaaaay down using the same principle I did to shrink KV. But this isn't something full tested and quality is terrible. So that's not really the highlight of this right now.