Category: Machine Learning

Summary: Testing whether explicitly maintaining high effective rank during training rescues compositional generalization in wide neural networks.

Wide networks can perform poorly on compositional generalization if their internal representations collapse toward low effective rank. This experiment asks whether that rank collapse is a cause of the failure or merely a symptom, by directly regularizing hidden activations to preserve higher rank during training.

The script adds a nuclear-norm-style penalty on hidden activations and compares the resulting models against unregularized baselines across widths and regularization strengths. It tracks both compositional performance and representation-rank diagnostics.

That makes the experiment a causal test, not just another regularization benchmark. If preserving rank improves out-of-distribution compositional behavior, it strengthens the claim that rank collapse is one of the operative mechanisms behind the failure.

Method: Numpy-based neural-network training with hidden-activation rank regularization, compared against baseline models across widths and penalty strengths.

What is measured: Compositional gap, effective rank of hidden activations, training accuracy, dependence on regularization strength, and width effects.