Category: Machine Learning

Summary: Testing whether orthogonality regularization can prevent representation collapse in wide networks and recover out-of-distribution compositional generalization.

Wide neural networks often fit training data easily yet generalize poorly on tasks that require recombining familiar pieces in new ways. This experiment studies whether that failure is tied to dimensional collapse inside hidden layers, where a few dominant directions crowd out a richer internal basis.

The model trains width-swept multilayer perceptrons on a synthetic compositional task while varying the strength of an orthogonality penalty on hidden weight matrices. It compares in-distribution accuracy, held-out compositional accuracy, and representation-rank diagnostics to see whether encouraging nearly orthogonal hidden features keeps wider models from collapsing into narrow internal subspaces.

That matters because pruning after training did not solve the earlier problem in this research line. If a training-time orthogonality bias rescues compositionality, it would point to a specific mechanism: wide models may fail not because they are too large in general, but because they use their capacity in an overly redundant way.

Method: Factorial MLP sweep over width and orthogonality strength, using a soft ||W^T W - I|| penalty during training on a compositional generalization task.

What is measured: In-distribution accuracy, out-of-distribution compositional accuracy, compositional gap, effective rank or width ratio, and dependence on orthogonality strength.