Category: Machine Learning

Summary: Testing whether directly truncating weight-matrix rank reproduces the timing-sensitive generalization effects previously seen with weight decay.

Weight decay can improve generalization, but it is often unclear whether the relevant mechanism is simply shrinking weights or specifically compressing representation rank. This experiment asks whether low-rank SVD interventions can causally reproduce the same inverse critical-period pattern previously observed for weight decay.

The script applies rank truncation at different times during training and compares the resulting generalization behavior with earlier weight-decay findings. Because SVD truncation targets rank structure directly, it provides a cleaner test of whether representational compression, rather than mere magnitude reduction, explains the timing effect.

That makes the experiment a mechanistic intervention study. Instead of correlating weight decay with later outcomes, it tests whether imposing the hypothesized causal change is sufficient to recreate the observed phenomenon.

Method: Neural-network training with timed SVD-based rank truncation interventions, compared against prior timing-sensitive regularization results.

What is measured: Generalization gap, intervention-timing effect, alignment with earlier weight-decay behavior, and the effect of rank compression independent of weight magnitude.