Mike Knoop says LLM logprobs are vanishing, yet they enable useful new patterns
Original titlei forgot that logprobs used to be universally exposed from LLM inference APIs! we're entering some market maturity phase of LLMs where it...
AISummary
Mike Knoop notes that logprobs used to be widely exposed by LLM inference APIs and sees the market maturing so that parts of the LLM stack can be packaged in new, useful ways. He links this to Bryan Helmig's post on prompting with max_tokens: 1 plus logprobs for fast, parallel judgments, which Helmig says has a lot more depth than he expected.
Source: Mike Knoop · x.comPublished · added here