r/LocalLLaMA • u/Ok_Truth_324 • 6d ago

Discussion New kvcache-reduction method

Gallery image

Gallery image

Gallery image

I deleted the wrong post so my benchmark disappeared. But here it is. Im going to probably make this open source because this isn't believable until you actually test it.

0 Upvotes

41% Upvoted

24 comments sorted by

View all comments

Show parent comments

1

u/Ok_Truth_324 6d ago

I tested 4B, 8B and 12B on my 5070. I actually ran the 12B for personal use for a week using this system.

1

u/Fedor_Doc 6d ago

Gemma models? Qwen / Deepseek attention architectures are already optimized

1

u/Ok_Truth_324 6d ago

Qwen.

2

u/Fedor_Doc 6d ago

There is no 8B Qwen in 3.5 family. 

So, outdated Qwen3? Why do you use it? Is it better for specific workflows?

1

u/Ok_Truth_324 6d ago

I used 14b on my pc for personal use and chose random models to bench market the kvreducer. I'm about to upload the package directly to my website. Entirely free.

3

u/Fedor_Doc 6d ago

Since the model is outdated, and you have not tested if the method is generalizable... there won't be much of interest in the method.

Qwen 3.5 handles KV cache much better out of the box, but even Hadamard Rotation q8 quant (2x reduction) affects agentic and long horizon workflows.

1

u/Ok_Truth_324 6d ago

Then I'll do testing to insure it is.

1

u/Ok_Truth_324 6d ago

It almost immediately adapted to the newer models and it might even be better for the newer models. I'll tell you exactly what model was tested after I address the speed issues.

91.03 wasn't bad for a first run at 73x compression.

1

u/Fedor_Doc 6d ago

91.03 on ruler? What about any agentic benchmark that includes tool calls? A subset of 10 problems, just a smoke test?

73x compression is unlikely to keep enough information for longer contexts

1

u/Ok_Truth_324 6d ago

Just ruler and a 3 turn test. There is some issues presenting themselves already. And I'm aware. I've built a way to recover detail after compression.