Margin-based Progress Measures for Grokking
Published in NeurIPS 2026 Workshop — Neural Network Artifacts as a New Data Modality, 2026
Grokking is the phenomenon of delayed generalization, where a network fits its training data long before it generalizes to held-out data. We study how instance margins evolve through this transition, using the logit margin in output space — which requires no labels — and the distance to the decision boundary in input or embedding space. Across modular-arithmetic tasks (MLPs, LSTMs) and a synthetic shape-classification task (ResNet-18), the mean test logit margin stays low during memorization and rises as generalization arrives; memorization separates the train and test margin distributions, and that gap shrinks as the model groks.
Presented at the Neural Network Artifacts as a New Data Modality workshop, NeurIPS 2026 (Paris, Palais des congrès). Read the paper on OpenReview.
Recommended citation: Ngnawé, J., & Tikeng Notsawo, P. J. (2026). Margin-based Progress Measures for Grokking. NeurIPS 2026 Workshop on Neural Network Artifacts as a New Data Modality.
Download Paper