Category: Machine Learning

Summary: Testing whether weight decay rescues compositional generalization mainly when it is applied during an early critical training window rather than throughout training.

Earlier Axiom results suggested both that wider networks struggle with compositionality and that weight decay can rescue that failure. This experiment asks whether the timing of the intervention is the key ingredient: if a critical period exists, then regularization delivered during that window should matter much more than regularization applied after it closes.

The design compares no weight decay, always-on weight decay, early-only weight decay, late-onset weight decay, and a brief narrow window of weight decay across widths 32, 64, and 128. By recording the compositional gap every five epochs, the run turns the question into a time-resolved comparison of when the rescue signal actually appears.

That matters because it reframes regularization as a schedule-design problem rather than a constant setting. If only early intervention is effective, the experiment would support a genuine critical-period interpretation of compositional training dynamics.

Method: Repeated NumPy MLP training across width and weight-decay schedules, with checkpointed compositional-gap measurements every five epochs.

What is measured: Compositional generalization gap over training time, comparison across timing schedules, width dependence of the intervention effect, and brief-window effectiveness.