Category: Machine Learning

Summary: Mapping sharp training-regime transitions as learning rate moves from near-static behavior through stable learning to noisy edge-of-chaos dynamics and outright divergence.

Learning rate is often tuned as though performance changed smoothly, but in practice training can shift abruptly between distinct regimes. This experiment asks where those boundaries lie by sweeping the learning rate over several orders of magnitude and treating the result as a phase diagram rather than a simple hyperparameter search.

For each of 50 logarithmically spaced learning rates, the script trains a small MLP for 200 epochs and records full training dynamics across three random seeds. It tracks endpoint performance, stability, divergence, convergence speed, and gradient norm behavior to separate dead-zone, Goldilocks, edge-of-chaos, and divergence-cliff regimes.

That framing matters because abrupt optimizer transitions are easier to reason about as regime changes than as small metric fluctuations. The result is meant to identify both good settings and the structure of the surrounding training landscape.

Method: Repeated NumPy MLP training across a logarithmic learning-rate sweep, with full-trajectory diagnostics and multi-seed comparisons.

What is measured: Train and test loss, train and test accuracy, stability of late loss, divergence indicators, convergence speed, and gradient norm statistics.