Skip to content
Read the original: Simon Willison· Published 14/100AI score14/100

ttok 1.0 switches default tokenizer to GPT-5 family and GPT-6

Original titlettok 1.0

AISummary

Simon Willison released ttok 1.0, changing its default tokenizer from GPT-4 to a newer GPT-5/GPT-6 tokenizer after finding the old default was outdated. OpenAI has not confirmed that GPT-6 shares the GPT-5 tokenizer, but William Liu's experiment found all seven GPT models tested reported 44,794 tokens on a 31-fixture corpus.

Read the original simonwillison.net

Source: Simon Willison · simonwillison.netPublished · added here