John Schulman says early OpenAI doubted next-token prediction would yield intelligence
Original titleIn the early days of OpenAI, @johnschulman2 didn't think next-token prediction was going to lead to intelligence, because it would get sw...
AISummary
John Schulman reveals that in OpenAI's early days, he doubted next-token prediction would lead to intelligence because noise would swamp the signal. He adds that it has always been hard to predict in advance which techniques will elicit out-of-distribution generalization.
Source: Dwarkesh Patel · x.comPublished · added here