Scaling Laws, Carefully: Early Empirical Power-Law Studies of Loss, Data and Model Size
AILil'Log examines early empirical work showing that deep learning generalization error follows power-law curves as training data and model size grow. Hestness et al. (2017) found the exponent reflects the problem domain rather than the architecture, while Rosenfeld et al. (2020) modeled loss jointly as a function of model size N and data size D, fitting parametric forms on small configurations to extrapolate to larger ones.







