Meta and MIT paper finds looping transformer layers can beat a plain model

1. 6:53 AM
   Rohan Paul@rohanpaul_aiX

   Meta and MIT find looping transformer layers can beat plain models

   AIMeta and MIT researchers found that running a transformer's layers in loops can beat a plain model at the same parameter count. They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off most when training compute was tight.

   

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.