Lucas Beyer says the overfitting seen with engram is plain overfitting that appears whenever a model is over-parametrized, not something specific to engram. He adds that engram, MoE, and similar methods make it easier and cheaper to reach, and says it is not necessarily bad depending on the goal.
Uhhh this is nothing special or new at all. It's plain regular overfitting.
It looks like this when you have clear epoch boundary instead of epochs shuffling into each other. And it's not specific to engram, just to how over-parametrized you are. Just that engram makes easier/cheaper to reach. As does MoE or other things.
It's also not even necessarily bad, depending on what your plan is.
engram/n-gram may markedly amplify overfitting under data repeat
Source: Lucas Beyer (bl16) · x.comPublished · added here
