Category: Machine Learning

Summary: Tracking how weight-matrix singular-value spectra evolve during training to test whether networks develop low-rank structure and regime changes across learning rates and architectures.

Neural networks often appear to organize their weights into structured, low-rank forms during training, but the timing and universality of that organization are still debated. This experiment asks how singular-value spectra, effective rank, and related spectral observables evolve over training across different depths, widths, and learning rates.

The script records singular-value distributions and derived rank statistics at checkpoints, along with measures such as top singular-value growth, spectral gaps, and representation similarity. That makes it possible to compare lazy-training behavior against richer feature-learning regimes using the same spectral language.

The value of the experiment is mechanistic rather than purely predictive. Instead of only reporting final accuracy, it studies how internal matrix structure emerges and whether sharp changes in that structure accompany changes in training dynamics.

Method: Checkpointed neural-network training with singular-value decompositions and rank diagnostics of weight matrices across architectures and learning rates.

What is measured: Singular-value spectra, effective rank, stable rank, top singular-value growth, spectral gaps, and representation similarity.