Category: Machine Learning

Summary: Pushing a modular-arithmetic network far enough in training to see whether memorization gives way to delayed generalization in a completed grokking transition.

Grokking describes the striking regime in which a model first memorizes its training data and only much later suddenly generalizes well. This version of the experiment follows that transition on modular addition, changing the prime size, learning rate, and logging schedule to determine whether the delayed jump in test accuracy can be observed cleanly rather than only in partial form.

The script trains a small network on one-hot encoded modular-addition pairs, logs train and test loss, test accuracy, and weight norms over long runs, and marks both the memorization epoch and the later grokking epoch when generalization crosses a high threshold. The smaller modulus and adjusted optimization settings are chosen to make the full phase transition visible within the runtime budget.

That makes the experiment a dynamics study of a specific learning phenomenon rather than a modular-arithmetic benchmark. The central question is whether the abrupt generalization phase can be completed and measured as a genuine temporal separation from memorization.

Method: Long-horizon training of a modular-addition classifier with AdamW, tracking memorization and grokking epochs under adjusted prime size and learning rate.

What is measured: Memorization epoch, grokking epoch, grokking gap, train and test loss, train and test accuracy, weight-norm trajectory, and final generalization level.