Type a message to an AI model and it doesn’t see words the way you do. It sees tokens, small chunks of text turned into numbers it can process. Once you understand this, a lot of the confusing parts of AI pricing and limits suddenly make sense.
Breaking down a sentence
Take something simple: “The quick brown fox jumps.” You’d read that as six words. A tokenizer might see something closer to:
The | quick | brown | fox | jump | s | .
Notice “jumps” got split in two. That’s because tokenizers are built from patterns found in huge amounts of text, not from a dictionary of whole words. Common words often stay whole. Less common ones, or words built from smaller parts, get chopped up.
Why the same sentence counts differently depending on the model
Every provider trains its own tokenizer on its own data. That means the exact same sentence can come out to a different token count depending on whether you’re using GPT, Claude, or Gemini. A prompt that’s 100 tokens on one might be 115 on another, purely because of how each one happens to split the text.
This is one of the most overlooked reasons that comparing prices across providers can be misleading if you’re only looking at the sticker price per token. Two providers can charge the same rate and still cost you differently, just because one counts more tokens for your text than the other. Our pricing guide and provider comparison both dig into how this plays out with real numbers.
Non-English text usually costs more
Most tokenizers are trained on data where English dominates. That means text in other languages, especially ones with different scripts like Chinese, Japanese, or Arabic, often needs more tokens to say the same thing. If you’re building for a non-English audience, this is worth factoring into your cost estimates early.
The hidden tokens you never see
Beyond the actual words in your prompt, models also use invisible structural tokens behind the scenes, marking where a conversation turn starts and ends, separating instructions from user input, that kind of thing. It’s a small overhead, but it’s there on every single call whether you notice it or not.
Why any of this matters day to day
Once tokenization clicks, a few things stop being mysterious:
- Why your token count is always higher than your word count
- Why the same document costs differently depending on which provider you use
- Why a “128K context window” doesn’t map neatly onto some number of pages
- Why keeping prompts short and direct actually saves real money, something we cover in more detail in 10 Ways to Actually Cut Your AI API Costs
See it happen in real time
The easiest way to actually get a feel for this is to watch it happen. Paste any text into our free calculator and see exactly how it breaks apart across 45+ models. No signup needed. For quick answers to related questions, our FAQ page has more detail on context windows, pricing, and how the calculator works.
Leave a Reply