Category: Machine Learning

Summary: Testing whether wider networks learn more entangled internal features and whether weight decay can make those representations more disentangled and compositional.

One reason wide neural networks may generalize poorly compositionally is that their hidden units could mix underlying factors together instead of isolating them cleanly. This experiment asks whether representational entanglement increases with width, and whether weight decay can counteract that trend by pushing networks toward more factorized internal codes.

The script trains width-swept two-layer networks on a synthetic compositional task and evaluates multiple disentanglement metrics, including DCI-style scores, mutual-information-based measures, effective rank, and the usual in-distribution versus out-of-distribution accuracy gap. Weight decay is varied to test whether a simple training intervention shifts both geometry and generalization together.

The scientific value lies in connecting performance to internal structure. If disentanglement tracks compositional success across widths and regularization levels, it would provide a more interpretable explanation for why some models reuse learned factors while others mainly memorize mixtures.

Method: Width and weight-decay sweep on a synthetic compositional task, with DCI, mutual-information, and rank diagnostics computed from hidden representations.

What is measured: DCI disentanglement, completeness, informativeness, mutual-information-based disentanglement, effective rank, and compositional generalization gap.