Token Counter
Your text never leaves your browser — tokenization runs entirely locally using the gpt-tokenizer library, loaded once from a CDN as static code.
Tokens
0
Characters
0
How to use this token counter
- Choose the encoding that matches your target model.
- Paste any text or prompt into the box above.
- The exact token count updates instantly, with token boundaries highlighted below.
What is a token?
A token is the basic unit language models process text in — often a word, part of a word, or punctuation mark, not a single character. Model context limits and API pricing are both measured in tokens, not characters, so an accurate count matters for staying within limits and estimating cost.
Why are there two different encodings?
OpenAI has used different tokenizers over time. cl100k_base powers GPT-3.5 and GPT-4, while o200k_base is the newer encoding used by GPT-4o, GPT-4o mini and the o1 models — the same text can produce a different token count depending on which one applies.
Does this work for Claude or Gemini too?
Not exactly — this counts tokens using OpenAI's tokenizers specifically. Other providers use their own tokenization, so their real counts will differ somewhat, though this is usually a reasonable ballpark estimate.
How BPE tokenization actually works
The tokenizers behind cl100k_base and o200k_base use a technique called Byte Pair Encoding (BPE): the algorithm starts by treating text as individual bytes, then repeatedly merges the most frequently occurring adjacent pair into a single token, based on statistics gathered from a huge training corpus. Over many merge steps, common sequences — whole common words, frequent prefixes and suffixes, even common multi-word phrases — end up as single tokens, while rare or unfamiliar sequences get broken down into smaller pieces, sometimes down to individual bytes for text the tokenizer has never seen anything like. This is why tokens don't map cleanly onto words: "token" itself might be one token, while an unusual word or a typo can split into two, three, or more.
Why the same text produces different token counts
cl100k_base and o200k_base are different vocabularies, trained on different data with different merge rules, so identical text can produce a noticeably different token count depending on which one is selected. o200k_base, the newer encoding behind GPT-4o and o1, was built with a larger vocabulary (roughly 200,000 tokens versus cl100k_base's ~100,000), which generally means it can represent more of the world's languages and common patterns with fewer tokens per character. This shows up clearly with non-English text: English tends to tokenize efficiently in both encodings because the training data was English-heavy, while languages like Spanish, and especially languages using non-Latin scripts, historically needed more tokens per equivalent sentence — the gap is smaller in o200k_base, but it hasn't disappeared entirely.
Why token count matters beyond context limits
Staying under a model's maximum context window is the most visible reason to count tokens, but it isn't the only one. API pricing for every major model provider is billed per token, for both the text you send and the text the model generates back, so an accurate count directly translates into a cost estimate before you make the call. Response latency also correlates loosely with token count on both ends — longer prompts take longer to process, and longer expected outputs take longer to stream back. And rate limits on most APIs are frequently expressed in tokens-per-minute rather than requests-per-minute, so a single very long prompt can consume a disproportionate share of your available quota compared to several short ones.
Common surprises: code, numbers, and non-English text
A few categories of text tokenize less efficiently than plain prose, often surprising people who estimate cost based on character count alone. Source code tends to use more tokens per character than natural language, since indentation, punctuation-heavy syntax, and unusual identifier names (camelCase or snake_case variables) don't compress as well into the trained vocabulary as common English words do. Numbers are handled inconsistently across encodings — a long number can be split into multiple 2-3 digit chunks rather than staying as one token, which matters for tasks involving lots of numeric data. And any text mixing scripts (a sentence combining English with emoji, or English with a language using a different alphabet) tends to use noticeably more tokens per visible character than uniform plain-English text.
Limitations of this tool
This tool counts tokens using OpenAI's own published tokenizers, loaded from a CDN and run entirely in your browser — it's exact for cl100k_base and o200k_base specifically, not an approximation. It is not accurate for other providers: Anthropic's Claude models, Google's Gemini models, and open-source models like Llama each use their own distinct tokenizer, and while token counts across providers are often in a similar ballpark for plain English text, they are not interchangeable and can diverge meaningfully for code, non-English text, or unusual formatting. It also counts only the raw text you provide — actual API usage typically adds a small number of extra tokens for message formatting (roles, delimiters between messages in a chat completion), which this tool doesn't attempt to simulate.