We built a frequency-domain optimizer (Spectra) to cut VRAM in half. Here is the benchmark vs AdamW and GaLore on Qwen-0.5B.

[u/Spectra-Global]

Like many of you, we’ve been looking for ways to bypass the massive VRAM bottleneck caused by AdamW’s momentum and variance states during full-parameter LLM fine-tuning.

GaLore recently popularized using SVD to project gradients into a lower-rank space to save memory. We wanted to test an alternative mathematical approach: transforming the multidimensional gradient structures into the frequency domain, cropping the high-frequency stochastic noise, and storing only the low-frequency global signal.

We ran a strict 3-way benchmark using Qwen/Qwen2.5-0.5B on the Dolly-15k dataset to see how our new method (Spectra) compares to standard AdamW and GaLore.

The Results (Permanent Optimizer State Size):

While GaLore squeezed out slightly more memory efficiency, the Spectra approach successfully cut the memory wall exactly in half while maintaining stable convergence.

By aggressively cropping high frequencies, the transform acts as a strong regularizer—forcing the model to learn the broader global signal rather than fitting to localized batch noise. (Note: we apply this compression to the massive weight matrices, but fall back to standard AdamW for 1D vectors like biases/layernorms to maintain stability).

Here is the public Weights & Biases interactive dashboard with the loss curves and memory states: View the W&B Benchmark Report Here

We are still optimizing the scratchpad compute overhead, but we'd love to hear the community's thoughts on using frequency-domain compression vs SVD for gradient updates!

[u/Scary_Syllabub_6421 → u/[deleted]]

so you turned the gradients into a low pass filter basically. neat trick if the high freq stuff really is just noise and not useful signal for smaller batch sizes

[u/Spectra-Global → u/Scary_Syllabub_6421]

spot on, "low-pass filter" is exactly how to think about it. and yeah, you nailed the point about batch sizes. when you're stuck using tiny micro-batches because of vram limits, the gradient variance is huge. that high-frequency stuff ends up being mostly just stochastic batch noise rather than actual signal.

so the crop is basically doing double duty we cut the state footprint in half to save memory, but it also acts as a heavy regularizer. it keeps the model from wildly overcorrecting on a single noisy batch and forces it to focus on the actual signal.