Abstract page for arXiv paper 2611.04215: Tokenizer Myths and Where To Find Them

The same text can cost several times as many tokens depending on the language. CLEAR's study grades GPT-5, Gemini, Claude, Llama and more.

Experiment with and compare different tokenizers

Hi. I can confirm that the `{` issue we found earlier is fixed on my end too: * * * I gave the published V2 Preview tokenizer a separate CPU-only check, using fixed files from the V1 repository and the new V2 repository. The particular regression that prompted the earlier discussion is repaired: the V1 tokenizer encoded an isolated opening brace as `[178, 0]`, with ID `0` being its unknown token, while the V2 tokenizer encodes `{` as the single token `[32000]` and decodes it back to `{`. That is a real and useful improvement. In the V2 Preview announcement, you said the vocabulary had become 32,001 specifically to address this, and this narrow check agrees with that description. I also tried two `tokenizers` releases, 0.22.1 and 0.23.3, in isolated CPU environments; the results agreed for the tested inputs. There is one adjacent detail that might be useful for a future tokenizer revision, though. Some strings containing the newly registered punctuation characters now acquire spaces _inside_ the decoded text. This is separate from whether the missing `{` can be represented, and it is also separate from whether the model can produce correct code or JSON. Here are a few short input/round-trip pairs from the released V2 tokenizer. I am showing the literal decoded text, including a possible leading space: Original input | Decoded after V2 encode/decode | Why it is interesting ---|---|--- `{` | `{` | The original missing character is now preserved. `{"a":1}` | `{ "a":1}` | A space appears after the opening brace, but this JSON still parses to the same value. `left{right` | ` left{ right` | An additional space appears inside ordinary text. `C:\Users\test\file.json` | ` C:\ Users\ test\ file.json` | A path-like string is no longer character-for-character identical. `https://example.org/a?b=1&c=2` | ` https://example.org/a?b=1& c=2` | A query-string-like fragment changes internally. The initial leading space in some rows needs to be interpreted separately. The published tokenizer uses a ByteLevel pre-tokenizer with `add_prefix_space=True`; its ordinary leading-space behavior already existed in V1. The interesting _new_ part here is the additional spacing at internal punctuation boundaries. The ByteLevel documentation explains the prefix-space option, while the AddedToken documentation describes the separate matching properties of registered tokens. ### A low-cost default check The cheapest next step is a small tokenizer-only regression panel: `{`, a simple JSON object, an embedded brace, a path with backslashes, and one URL or literal escape. For each, retain the token IDs and the exact decoded string. Check JSON parsing separately from exact-text equality. The source can be pinned and checked on CPU, without loading model weights or retraining. ### When this matters * If only **parseable JSON values** matter, the whitespace in the two selected JSON examples is allowed; it need not be a blocker. * If **exact code, paths, URLs, or escapes** matter, internal spaces deserve a regression check. * If preserving the **already-trained V2 checkpoint** matters, test ID compatibility before changing tokenizer settings. * If building a **new tokenizer and model together** , byte coverage and round-trip checks fit naturally _before_ training. This is a raw-tokenizer test, not a V2 generation-quality benchmark. The diagnostic inputs are deliberately selected edge cases, so their counts are not real-world error rates. The details below give pinned files, reproducible commands, the structural diff, and the limits of the conclusions. Pinned tokenizer files, environment, and a minimal reproduction (click for more details) Concrete round trips, classifications, and what the numbers mean (click for more details) Tokenizer structure, diagnostic variants, and compatibility with trained weights (click for more details) Future tokenizer design and inexpensive regression checks (click for more details) Relevant documentation, similar upstream reports, and the limits of this feedback (click for more details) Thanks again for putting the V2 Preview out and for following up on the brace issue. The concrete repair is easy to confirm, which is a nice outcome. I hope the short round-trip examples and the compatibility distinction are useful as reference points for a later revision, whenever they fit the direction you want to take the model.

Share your videos with friends, family, and the world

Abstract page for arXiv paper 2611.04482: TEMPO: Time Series Understanding via Discrete Tokenization

Hi. I can confirm that the `{` issue we found earlier is fixed on my end too: * * * I gave the published V2 Preview tokenizer a separate CPU-only check, using fixed files from the V1 repository and the new V2 repository. The particular regression that prompted the earlier discussion is repaired: the V1 tokenizer encoded an isolated opening brace as `[178, 0]`, with ID `0` being its unknown token, while the V2 tokenizer encodes `{` as the single token `[32000]` and decodes it back to `{`. That is a real and useful improvement. In the V2 Preview announcement, you said the vocabulary had become 32,001 specifically to address this, and this narrow check agrees with that description. I also tried two `tokenizers` releases, 0.22.1 and 0.23.3, in isolated CPU environments; the results agreed for the tested inputs. There is one adjacent detail that might be useful for a future tokenizer revision, though. Some strings containing the newly registered punctuation characters now acquire spaces _inside_ the decoded text. This is separate from whether the missing `{` can be represented, and it is also separate from whether the model can produce correct code or JSON. Here are a few short input/round-trip pairs from the released V2 tokenizer. I am showing the literal decoded text, including a possible leading space: Original input | Decoded after V2 encode/decode | Why it is interesting ---|---|--- `{` | `{` | The original missing character is now preserved. `{"a":1}` | `{ "a":1}` | A space appears after the opening brace, but this JSON still parses to the same value. `left{right` | ` left{ right` | An additional space appears inside ordinary text. `C:\Users\test\file.json` | ` C:\ Users\ test\ file.json` | A path-like string is no longer character-for-character identical. `https://example.org/a?b=1&c=2` | ` https://example.org/a?b=1& c=2` | A query-string-like fragment changes internally. The initial leading space in some rows needs to be interpreted separately. The published tokenizer uses a ByteLevel pre-tokenizer with `add_prefix_space=True`; its ordinary leading-space behavior already existed in V1. The interesting _new_ part here is the additional spacing at internal punctuation boundaries. The ByteLevel documentation explains the prefix-space option, while the AddedToken documentation describes the separate matching properties of registered tokens. ### A low-cost default check The cheapest next step is a small tokenizer-only regression panel: `{`, a simple JSON object, an embedded brace, a path with backslashes, and one URL or literal escape. For each, retain the token IDs and the exact decoded string. Check JSON parsing separately from exact-text equality. The source can be pinned and checked on CPU, without loading model weights or retraining. ### When this matters * If only **parseable JSON values** matter, the whitespace in the two selected JSON examples is allowed; it need not be a blocker. * If **exact code, paths, URLs, or escapes** matter, internal spaces deserve a regression check. * If preserving the **already-trained V2 checkpoint** matters, test ID compatibility before changing tokenizer settings. * If building a **new tokenizer and model together** , byte coverage and round-trip checks fit naturally _before_ training. This is a raw-tokenizer test, not a V2 generation-quality benchmark. The diagnostic inputs are deliberately selected edge cases, so their counts are not real-world error rates. The details below give pinned files, reproducible commands, the structural diff, and the limits of the conclusions. Pinned tokenizer files, environment, and a minimal reproduction (click for more details) Concrete round trips, classifications, and what the numbers mean (click for more details) Tokenizer structure, diagnostic variants, and compatibility with trained weights (click for more details) Future tokenizer design and inexpensive regression checks (click for more details) Relevant documentation, similar upstream reports, and the limits of this feedback (click for more details) Thanks again for putting the V2 Preview out and for following up on the brace issue. The concrete repair is easy to confirm, which is a nice outcome. I hope the short round-trip examples and the compatibility distinction are useful as reference points for a later revision, whenever they fit the direction you want to take the model.

Chopping text into tokens was always trivial next to the model. Then the model got fast enough to wait for it.

A flaw has been found in deepset-ai farm up to 0.8.0. This impacts the function EmbeddingTokenizer.from_pretrained of the file farm/modeling/tokenization.py of the component Tokenizer Loader. Executing a manipulation of the argument vocab_filename can lead to path traversal. The attack may be performed from remote. The exploit has been published and …