Stéphan Tulkens' Blog

The ByteLevel pre-tokenizer is largely used to avoid the possibility of an <unk> token. However, there is a problem with the continuation characters in WordPiece that prevents you from adding all o...

The definitions of terms aren’t very strict, ranging from conceptual definitions to specific practical implementations, so it’s all rather confusing… * * * You’re right: with Hugging Face fast tokenizers, the ByteLevel **pre-tokenizer** already inserts the visible space marker `Ġ`. `pre_tokenize_str(...)` shows those markers because ByteLevel replaces spaces and remaps bytes **before** the BPE model runs. That output is expected. The later **BPE model** then merges the mapped characters inside each pre-token into the final vocab tokens and ids; `decode()` applies the ByteLevel **decoder** to get back normal text. (Hugging Face) ### Where `Ġ` can appear * Pre-tokenizer stage: spaces are turned into a visible marker (`Ġ`) so merges can learn “word-start” patterns. You will see `Ġwanna`, `Ġgo`, … even in `pre_tokenize_str`. (Hugging Face Forums) * Token stage: those same strings become actual vocab tokens like `Ġwanna` and ids. `decode()` reverses the byte→Unicode mapping and space handling. (Hugging Face) ### Why your Chinese “errors” aren’t errors `'法国','的','首','都是','巴黎']` is not a grammatical analysis. It’s a sequence of **subword/byte-level tokens** chosen to compress frequent patterns. BPE is trained to minimize sequence length and handle any UTF-8 text with no `<unk>`, not to output linguistically correct words. In languages without spaces, merges can cross human “word” boundaries, e.g., the token “都是” is very frequent, so it appears as one piece even when the intended segmentation is “首都 + 是”. This is a known behavior on Chinese. ([Hugging Face) ### Why LLMs don’t use jieba for model tokenization * **Coverage and robustness:** byte-level schemes guarantee every byte sequence is representable. No OOV. Word segmenters depend on lexicons and can fail on names, slang, or mixed-script text. (Hugging Face) * **Multilingual consistency:** one tokenizer for many scripts is simpler and more stable than per-language segmenters. (Hugging Face) * **Compression vs. linguistics:** BPE optimizes token length/frequency, not grammatical boundaries. That tradeoff improves throughput and training stability even if tokens don’t align with words. (Hugging Face) ### Practical guidance * Need **human-readable per-token text** : request `return_offsets_mapping=True` and slice the original string; or decode each id separately. Both avoid mojibake. (Hugging Face) * Need **linguistic words** : run a Chinese segmenter (e.g., jieba, pkuseg, THULAC) on the original text; do not expect the model tokenizer to give you words. (Segmenters are separate tools with different goals.) * Seeing `Ġ` in **pre-tokens** is normal for GPT-2/RoBERTa-style ByteLevel pipelines; the space marker is introduced before BPE and often survives into the final tokens. (Hugging Face Forums) ### Short references * HF Tokenizers pipeline and pre-tokenization overview. Spaces → markers happen pre-BPE. (Hugging Face) * ByteLevel pre-tokenizer description: remap bytes and split into words. (Hugging Face) * `decode(...)` behavior and relation to `convert_*` helpers. (Hugging Face) * Why BPE uses visible markers like `Ġ`, and examples. (Hugging Face Forums) * On Chinese and whitespace-free scripts, why merges can cross “word” boundaries. (The Digital Orientalist) Summary: `Ġ` in your `pre_tokenize_str` is expected. Model tokenization ≠ word segmentation. Use offsets or per-id decode for readable token pieces; use dedicated Chinese segmenters if you need grammatical words.

The definitions of terms aren’t very strict, ranging from conceptual definitions to specific practical implementations, so it’s all rather confusing… * * * You’re right: with Hugging Face fast tokenizers, the ByteLevel **pre-tokenizer** already inserts the visible space marker `Ġ`. `pre_tokenize_str(...)` shows those markers because ByteLevel replaces spaces and remaps bytes **before** the BPE model runs. That output is expected. The later **BPE model** then merges the mapped characters inside each pre-token into the final vocab tokens and ids; `decode()` applies the ByteLevel **decoder** to get back normal text. (Hugging Face) ### Where `Ġ` can appear * Pre-tokenizer stage: spaces are turned into a visible marker (`Ġ`) so merges can learn “word-start” patterns. You will see `Ġwanna`, `Ġgo`, … even in `pre_tokenize_str`. (Hugging Face Forums) * Token stage: those same strings become actual vocab tokens like `Ġwanna` and ids. `decode()` reverses the byte→Unicode mapping and space handling. (Hugging Face) ### Why your Chinese “errors” aren’t errors `'法国','的','首','都是','巴黎']` is not a grammatical analysis. It’s a sequence of **subword/byte-level tokens** chosen to compress frequent patterns. BPE is trained to minimize sequence length and handle any UTF-8 text with no `<unk>`, not to output linguistically correct words. In languages without spaces, merges can cross human “word” boundaries, e.g., the token “都是” is very frequent, so it appears as one piece even when the intended segmentation is “首都 + 是”. This is a known behavior on Chinese. ([Hugging Face) ### Why LLMs don’t use jieba for model tokenization * **Coverage and robustness:** byte-level schemes guarantee every byte sequence is representable. No OOV. Word segmenters depend on lexicons and can fail on names, slang, or mixed-script text. (Hugging Face) * **Multilingual consistency:** one tokenizer for many scripts is simpler and more stable than per-language segmenters. (Hugging Face) * **Compression vs. linguistics:** BPE optimizes token length/frequency, not grammatical boundaries. That tradeoff improves throughput and training stability even if tokens don’t align with words. (Hugging Face) ### Practical guidance * Need **human-readable per-token text** : request `return_offsets_mapping=True` and slice the original string; or decode each id separately. Both avoid mojibake. (Hugging Face) * Need **linguistic words** : run a Chinese segmenter (e.g., jieba, pkuseg, THULAC) on the original text; do not expect the model tokenizer to give you words. (Segmenters are separate tools with different goals.) * Seeing `Ġ` in **pre-tokens** is normal for GPT-2/RoBERTa-style ByteLevel pipelines; the space marker is introduced before BPE and often survives into the final tokens. (Hugging Face Forums) ### Short references * HF Tokenizers pipeline and pre-tokenization overview. Spaces → markers happen pre-BPE. (Hugging Face) * ByteLevel pre-tokenizer description: remap bytes and split into words. (Hugging Face) * `decode(...)` behavior and relation to `convert_*` helpers. (Hugging Face) * Why BPE uses visible markers like `Ġ`, and examples. (Hugging Face Forums) * On Chinese and whitespace-free scripts, why merges can cross “word” boundaries. (The Digital Orientalist) Summary: `Ġ` in your `pre_tokenize_str` is expected. Model tokenization ≠ word segmentation. Use offsets or per-id decode for readable token pieces; use dedicated Chinese segmenters if you need grammatical words.

The definitions of terms aren’t very strict, ranging from conceptual definitions to specific practical implementations, so it’s all rather confusing… * * * You’re right: with Hugging Face fast tokenizers, the ByteLevel **pre-tokenizer** already inserts the visible space marker `Ġ`. `pre_tokenize_str(...)` shows those markers because ByteLevel replaces spaces and remaps bytes **before** the BPE model runs. That output is expected. The later **BPE model** then merges the mapped characters inside each pre-token into the final vocab tokens and ids; `decode()` applies the ByteLevel **decoder** to get back normal text. (Hugging Face) ### Where `Ġ` can appear * Pre-tokenizer stage: spaces are turned into a visible marker (`Ġ`) so merges can learn “word-start” patterns. You will see `Ġwanna`, `Ġgo`, … even in `pre_tokenize_str`. (Hugging Face Forums) * Token stage: those same strings become actual vocab tokens like `Ġwanna` and ids. `decode()` reverses the byte→Unicode mapping and space handling. (Hugging Face) ### Why your Chinese “errors” aren’t errors `'法国','的','首','都是','巴黎']` is not a grammatical analysis. It’s a sequence of **subword/byte-level tokens** chosen to compress frequent patterns. BPE is trained to minimize sequence length and handle any UTF-8 text with no `<unk>`, not to output linguistically correct words. In languages without spaces, merges can cross human “word” boundaries, e.g., the token “都是” is very frequent, so it appears as one piece even when the intended segmentation is “首都 + 是”. This is a known behavior on Chinese. ([Hugging Face) ### Why LLMs don’t use jieba for model tokenization * **Coverage and robustness:** byte-level schemes guarantee every byte sequence is representable. No OOV. Word segmenters depend on lexicons and can fail on names, slang, or mixed-script text. (Hugging Face) * **Multilingual consistency:** one tokenizer for many scripts is simpler and more stable than per-language segmenters. (Hugging Face) * **Compression vs. linguistics:** BPE optimizes token length/frequency, not grammatical boundaries. That tradeoff improves throughput and training stability even if tokens don’t align with words. (Hugging Face) ### Practical guidance * Need **human-readable per-token text** : request `return_offsets_mapping=True` and slice the original string; or decode each id separately. Both avoid mojibake. (Hugging Face) * Need **linguistic words** : run a Chinese segmenter (e.g., jieba, pkuseg, THULAC) on the original text; do not expect the model tokenizer to give you words. (Segmenters are separate tools with different goals.) * Seeing `Ġ` in **pre-tokens** is normal for GPT-2/RoBERTa-style ByteLevel pipelines; the space marker is introduced before BPE and often survives into the final tokens. (Hugging Face Forums) ### Short references * HF Tokenizers pipeline and pre-tokenization overview. Spaces → markers happen pre-BPE. (Hugging Face) * ByteLevel pre-tokenizer description: remap bytes and split into words. (Hugging Face) * `decode(...)` behavior and relation to `convert_*` helpers. (Hugging Face) * Why BPE uses visible markers like `Ġ`, and examples. (Hugging Face Forums) * On Chinese and whitespace-free scripts, why merges can cross “word” boundaries. (The Digital Orientalist) Summary: `Ġ` in your `pre_tokenize_str` is expected. Model tokenization ≠ word segmentation. Use offsets or per-id decode for readable token pieces; use dedicated Chinese segmenters if you need grammatical words.

BTW, for general information on tokenization, this article should also be helpful. * * * Definitions first. * **Pre-token** : an intermediate _span of the original text_ produced by the pre-tokenizer, plus its character offsets. With **ByteLevel** , the pre-tokenizer (a) remaps each UTF-8 byte to a visible Unicode placeholder and (b) **splits on whitespace** to yield “word-like” chunks; it also carries offsets so you can map back to the input. (Hugging Face) * **Token** : the result after the **model step** (BPE merges) runs _inside each pre-token_. Tokens are vocabulary strings (those mapped-byte symbols you saw) and their integer IDs. A **decoder** then inverts the byte mapping when you call `decode`/`batch_decode`. (Hugging Face) # Is it strings or bytes? * The pipeline runs on **Unicode strings** externally. Byte-level logic is handled by mapping bytes→printable Unicode during pre-tokenization, then reversing it during decoding. You interact with strings and IDs; no raw bytes are returned. Byte-level BPE is used so every UTF-8 sequence is representable without `<unk>`. (GitHub) # Your example, concretely Input: `今天天气真好,I wanna go swimming` 1. **Pre-tokenizer output (conceptual):** splits on whitespace only. So you get pre-tokens like `"今天天气真好,I", "wanna", "go", "swimming"]` with offsets over the **original** string. The “I” is attached to the first pre-token because there is **no space** before it. Punctuation does not force a split in ByteLevel; whitespace does. ([Hugging Face) 2. **Model (BPE) output:** inside each pre-token, BPE merges the byte-mapped characters into vocab tokens such as `['ä»Ĭ天','天æ°Ķ','羣','å¥1⁄2','ï1⁄4Į','I','Ġwanna','Ġgo','Ġswimming']` and maps them to IDs. The leading `Ġ` on English pieces indicates a preceding space in GPT-2–style tokenizers. (GitHub) 3. **Decoder:** `decode`/`batch_decode` inverts the byte mapping and restores normal spacing. (Hugging Face) # Clarifications to common confusions * **“Are pre-tokens a`list[str]` I can see?”** Conceptually yes (word-like chunks), but what the library _exposes by default_ are the **final tokens** and IDs. If you want to _inspect_ pre-tokens, call the underlying Rust pre-tokenizer: # deps: pip install tokenizers>=0.15 transformers>=4.44 from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("unsloth/Qwen3-14B", use_fast=True) print(tok.backend_tokenizer.pre_tokenizer.pre_tokenize_str("今天天气真好,I wanna go swimming")) # -> [(pre_token_string, (start, end)), ...] This shows the whitespace splits and their offsets. (Vinsmoke Three) * **“Where does`Ġ` come from?”** It’s a visible space marker baked into GPT-2–style vocabularies so BPE can learn merges that depend on preceding whitespace. It appears in **tokens** , not pre-tokens. (GitHub) * **“Why byte-level at all?”** Two reasons: coverage with no `<unk>` and exact reversibility. BPE expects strings, so bytes are first mapped to visible Unicode, then merged; the decoder reverses that map. (GitHub) # Mental model you can trust Unicode text → Normalizer → Pre-tokenizer (ByteLevel): bytes→visible Unicode; split on whitespace; keep offsets → Model (BPE): merge mapped chars into vocab tokens; get token strings + IDs → Post-processor: add special tokens if needed → (on decode) Decoder (ByteLevel): visible Unicode → original bytes → human text This matches Hugging Face’s Tokenizers pipeline and API terminology. (Hugging Face) # Minimal checks you can run * **Pre-tokens (whitespace splits + offsets):** use `pre_tokenize_str` as above. (Google Colab) * **Final tokens (mapped-byte strings):** `tokenizer.convert_ids_to_tokens(...)`. * **Readable per-token spans:** ask for `return_offsets_mapping=True` and slice the original string. (Hugging Face) # Short references * HF Tokenizers: **Pre-tokenizers** (ByteLevel description) and **Decoders** (ByteLevel decoder). (Hugging Face) * HF Tokenizers: **Pipeline overview** and **offset mapping**. (Hugging Face) * GPT-2 space marker `Ġ` background. (GitHub) * Byte-level BPE rationale and UTF-8 coverage. (GitHub) Summary: a **pre-token** is a whitespace-delimited span with offsets; a **token** is a BPE-merged vocab string (plus its ID). No space before `I` means the pre-tokenizer **does not** split there.

Yeah. * * * Mostly right. Two fixes: 1. The ByteLevel **pre-tokenizer** splits on whitespace and remaps bytes to printable code points, but it does not hand you a visible list of single characters. It outputs “pre-tokens” with offsets. Then the **BPE model** merges those mapped characters into vocab tokens. Decoding later inverts the byte→Unicode mapping. (Hugging Face) 2. The English space is encoded into tokens with a visible space marker (e.g., `Ġ`). That’s why you see tokens like `Ġwanna`. Offsets can therefore include the leading space. (Hugging Face) # Walk-through on your example Input: `"今天天气真好,I wanna go swimming"` ## Pre-tokenizer output (conceptual) * Operation: normalize → UTF-8 bytes → map bytes to printable Unicode → **split on whitespace** → keep offsets. * Pre-tokens (by whitespace): `["今天天气真好,", "I", "wanna", "go", "swimming"]` * Character offsets (start, end) over the original string: `[(0,7), (8,9), (10,15), (16,18), (19,27)]` These are spans in the original text, not in the mapped “mojibake” string. (Hugging Face) ## BPE model output * Runs merges **inside each pre-token** over the mapped characters. * Typical token strings and offsets (character indexes in original text): * Chinese chunk: `['ä»Ĭ天','天æ°Ķ','羣','å¥1⁄2','ï1⁄4Į']` → `[(0,2),(2,4),(4,5),(5,6),(6,7)]` which correspond to `['今天','天气','真','好',',']` * English: `['I','Ġwanna','Ġgo','Ġswimming']` → `[(8,9),(9,15),(15,18),(18,27)]` where `Ġ` indicates the preceding space is part of the token span. * Map token strings → ids, e.g. `[100644,104307,88051,52801, ...]`. * No human decoding has happened yet; these are internal symbols. Qwen uses **byte-level BPE on UTF-8** , so this behavior is expected and guarantees no OOV. (Qwen) ## Post-processor * Adds special tokens (BOS/EOS, chat template pieces) if configured. It does not “fix” readability. (Hugging Face) ## Decoder (when you call `decode` / `batch_decode`) * Inverts the byte→Unicode mapping and restores spaces, yielding normal text. * Fast tokenizers also expose `return_offsets_mapping=True` so you can slice the original string per token without decoding each id. (Hugging Face) # Quick rules to remember * `tokenize()` / `convert_ids_to_tokens()` → raw vocab strings (mapped bytes). They will look garbled for non-ASCII. Correct by design. * `decode()` / `batch_decode()` → runs the decoder → human text. * `return_offsets_mapping=True` (fast tokenizers) → character spans over the original text for each final token. * English tokens may include a leading space (`Ġ...`), so their offsets can start at the space. This depends on tokenizer settings like `add_prefix_space` and post-processing; be mindful of offset edge cases. (Hugging Face) # Minimal verification snippet # deps: # pip install --upgrade transformers>=4.44 tokenizers>=0.15 from transformers import AutoTokenizer s = "今天天气真好,I wanna go swimming" tok = AutoTokenizer.from_pretrained("unsloth/Qwen3-14B", use_fast=True) enc = tok(s, add_special_tokens=False, return_offsets_mapping=True) tokens = tok.convert_ids_to_tokens(enc["input_ids"]) spans = [s[a:b] for a,b in enc["offset_mapping"]] print(tokens) # internal strings (byte-mapped), includes Ġ for spaces print(spans) # human-readable per-token text slices print(tok.decode(enc["input_ids"])) # original text """ ['ä»Ĭ天', '天æ°Ķ', '羣', 'å¥1⁄2', 'ï1⁄4Į', 'I', 'Ġwanna', 'Ġgo', 'Ġswimming'] ['今天', '天气', '真', '好', ',', 'I', ' wanna', ' go', ' swimming'] 今天天气真好,I wanna go swimming """

BTW, for general information on tokenization, this article should also be helpful. * * * Definitions first. * **Pre-token** : an intermediate _span of the original text_ produced by the pre-tokenizer, plus its character offsets. With **ByteLevel** , the pre-tokenizer (a) remaps each UTF-8 byte to a visible Unicode placeholder and (b) **splits on whitespace** to yield “word-like” chunks; it also carries offsets so you can map back to the input. (Hugging Face) * **Token** : the result after the **model step** (BPE merges) runs _inside each pre-token_. Tokens are vocabulary strings (those mapped-byte symbols you saw) and their integer IDs. A **decoder** then inverts the byte mapping when you call `decode`/`batch_decode`. (Hugging Face) # Is it strings or bytes? * The pipeline runs on **Unicode strings** externally. Byte-level logic is handled by mapping bytes→printable Unicode during pre-tokenization, then reversing it during decoding. You interact with strings and IDs; no raw bytes are returned. Byte-level BPE is used so every UTF-8 sequence is representable without `<unk>`. (GitHub) # Your example, concretely Input: `今天天气真好,I wanna go swimming` 1. **Pre-tokenizer output (conceptual):** splits on whitespace only. So you get pre-tokens like `"今天天气真好,I", "wanna", "go", "swimming"]` with offsets over the **original** string. The “I” is attached to the first pre-token because there is **no space** before it. Punctuation does not force a split in ByteLevel; whitespace does. ([Hugging Face) 2. **Model (BPE) output:** inside each pre-token, BPE merges the byte-mapped characters into vocab tokens such as `['ä»Ĭ天','天æ°Ķ','羣','å¥1⁄2','ï1⁄4Į','I','Ġwanna','Ġgo','Ġswimming']` and maps them to IDs. The leading `Ġ` on English pieces indicates a preceding space in GPT-2–style tokenizers. (GitHub) 3. **Decoder:** `decode`/`batch_decode` inverts the byte mapping and restores normal spacing. (Hugging Face) # Clarifications to common confusions * **“Are pre-tokens a`list[str]` I can see?”** Conceptually yes (word-like chunks), but what the library _exposes by default_ are the **final tokens** and IDs. If you want to _inspect_ pre-tokens, call the underlying Rust pre-tokenizer: # deps: pip install tokenizers>=0.15 transformers>=4.44 from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("unsloth/Qwen3-14B", use_fast=True) print(tok.backend_tokenizer.pre_tokenizer.pre_tokenize_str("今天天气真好,I wanna go swimming")) # -> [(pre_token_string, (start, end)), ...] This shows the whitespace splits and their offsets. (Vinsmoke Three) * **“Where does`Ġ` come from?”** It’s a visible space marker baked into GPT-2–style vocabularies so BPE can learn merges that depend on preceding whitespace. It appears in **tokens** , not pre-tokens. (GitHub) * **“Why byte-level at all?”** Two reasons: coverage with no `<unk>` and exact reversibility. BPE expects strings, so bytes are first mapped to visible Unicode, then merged; the decoder reverses that map. (GitHub) # Mental model you can trust Unicode text → Normalizer → Pre-tokenizer (ByteLevel): bytes→visible Unicode; split on whitespace; keep offsets → Model (BPE): merge mapped chars into vocab tokens; get token strings + IDs → Post-processor: add special tokens if needed → (on decode) Decoder (ByteLevel): visible Unicode → original bytes → human text This matches Hugging Face’s Tokenizers pipeline and API terminology. (Hugging Face) # Minimal checks you can run * **Pre-tokens (whitespace splits + offsets):** use `pre_tokenize_str` as above. (Google Colab) * **Final tokens (mapped-byte strings):** `tokenizer.convert_ids_to_tokens(...)`. * **Readable per-token spans:** ask for `return_offsets_mapping=True` and slice the original string. (Hugging Face) # Short references * HF Tokenizers: **Pre-tokenizers** (ByteLevel description) and **Decoders** (ByteLevel decoder). (Hugging Face) * HF Tokenizers: **Pipeline overview** and **offset mapping**. (Hugging Face) * GPT-2 space marker `Ġ` background. (GitHub) * Byte-level BPE rationale and UTF-8 coverage. (GitHub) Summary: a **pre-token** is a whitespace-delimited span with offsets; a **token** is a BPE-merged vocab string (plus its ID). No space before `I` means the pre-tokenizer **does not** split there.

Yeah. * * * Mostly right. Two fixes: 1. The ByteLevel **pre-tokenizer** splits on whitespace and remaps bytes to printable code points, but it does not hand you a visible list of single characters. It outputs “pre-tokens” with offsets. Then the **BPE model** merges those mapped characters into vocab tokens. Decoding later inverts the byte→Unicode mapping. (Hugging Face) 2. The English space is encoded into tokens with a visible space marker (e.g., `Ġ`). That’s why you see tokens like `Ġwanna`. Offsets can therefore include the leading space. (Hugging Face) # Walk-through on your example Input: `"今天天气真好,I wanna go swimming"` ## Pre-tokenizer output (conceptual) * Operation: normalize → UTF-8 bytes → map bytes to printable Unicode → **split on whitespace** → keep offsets. * Pre-tokens (by whitespace): `["今天天气真好,", "I", "wanna", "go", "swimming"]` * Character offsets (start, end) over the original string: `[(0,7), (8,9), (10,15), (16,18), (19,27)]` These are spans in the original text, not in the mapped “mojibake” string. (Hugging Face) ## BPE model output * Runs merges **inside each pre-token** over the mapped characters. * Typical token strings and offsets (character indexes in original text): * Chinese chunk: `['ä»Ĭ天','天æ°Ķ','羣','å¥1⁄2','ï1⁄4Į']` → `[(0,2),(2,4),(4,5),(5,6),(6,7)]` which correspond to `['今天','天气','真','好',',']` * English: `['I','Ġwanna','Ġgo','Ġswimming']` → `[(8,9),(9,15),(15,18),(18,27)]` where `Ġ` indicates the preceding space is part of the token span. * Map token strings → ids, e.g. `[100644,104307,88051,52801, ...]`. * No human decoding has happened yet; these are internal symbols. Qwen uses **byte-level BPE on UTF-8** , so this behavior is expected and guarantees no OOV. (Qwen) ## Post-processor * Adds special tokens (BOS/EOS, chat template pieces) if configured. It does not “fix” readability. (Hugging Face) ## Decoder (when you call `decode` / `batch_decode`) * Inverts the byte→Unicode mapping and restores spaces, yielding normal text. * Fast tokenizers also expose `return_offsets_mapping=True` so you can slice the original string per token without decoding each id. (Hugging Face) # Quick rules to remember * `tokenize()` / `convert_ids_to_tokens()` → raw vocab strings (mapped bytes). They will look garbled for non-ASCII. Correct by design. * `decode()` / `batch_decode()` → runs the decoder → human text. * `return_offsets_mapping=True` (fast tokenizers) → character spans over the original text for each final token. * English tokens may include a leading space (`Ġ...`), so their offsets can start at the space. This depends on tokenizer settings like `add_prefix_space` and post-processing; be mindful of offset edge cases. (Hugging Face) # Minimal verification snippet # deps: # pip install --upgrade transformers>=4.44 tokenizers>=0.15 from transformers import AutoTokenizer s = "今天天气真好,I wanna go swimming" tok = AutoTokenizer.from_pretrained("unsloth/Qwen3-14B", use_fast=True) enc = tok(s, add_special_tokens=False, return_offsets_mapping=True) tokens = tok.convert_ids_to_tokens(enc["input_ids"]) spans = [s[a:b] for a,b in enc["offset_mapping"]] print(tokens) # internal strings (byte-mapped), includes Ġ for spaces print(spans) # human-readable per-token text slices print(tok.decode(enc["input_ids"])) # original text """ ['ä»Ĭ天', '天æ°Ķ', '羣', 'å¥1⁄2', 'ï1⁄4Į', 'I', 'Ġwanna', 'Ġgo', 'Ġswimming'] ['今天', '天气', '真', '好', ',', 'I', ' wanna', ' go', ' swimming'] 今天天气真好,I wanna go swimming """

BTW, for general information on tokenization, this article should also be helpful. * * * Definitions first. * **Pre-token** : an intermediate _span of the original text_ produced by the pre-tokenizer, plus its character offsets. With **ByteLevel** , the pre-tokenizer (a) remaps each UTF-8 byte to a visible Unicode placeholder and (b) **splits on whitespace** to yield “word-like” chunks; it also carries offsets so you can map back to the input. (Hugging Face) * **Token** : the result after the **model step** (BPE merges) runs _inside each pre-token_. Tokens are vocabulary strings (those mapped-byte symbols you saw) and their integer IDs. A **decoder** then inverts the byte mapping when you call `decode`/`batch_decode`. (Hugging Face) # Is it strings or bytes? * The pipeline runs on **Unicode strings** externally. Byte-level logic is handled by mapping bytes→printable Unicode during pre-tokenization, then reversing it during decoding. You interact with strings and IDs; no raw bytes are returned. Byte-level BPE is used so every UTF-8 sequence is representable without `<unk>`. (GitHub) # Your example, concretely Input: `今天天气真好,I wanna go swimming` 1. **Pre-tokenizer output (conceptual):** splits on whitespace only. So you get pre-tokens like `"今天天气真好,I", "wanna", "go", "swimming"]` with offsets over the **original** string. The “I” is attached to the first pre-token because there is **no space** before it. Punctuation does not force a split in ByteLevel; whitespace does. ([Hugging Face) 2. **Model (BPE) output:** inside each pre-token, BPE merges the byte-mapped characters into vocab tokens such as `['ä»Ĭ天','天æ°Ķ','羣','å¥1⁄2','ï1⁄4Į','I','Ġwanna','Ġgo','Ġswimming']` and maps them to IDs. The leading `Ġ` on English pieces indicates a preceding space in GPT-2–style tokenizers. (GitHub) 3. **Decoder:** `decode`/`batch_decode` inverts the byte mapping and restores normal spacing. (Hugging Face) # Clarifications to common confusions * **“Are pre-tokens a`list[str]` I can see?”** Conceptually yes (word-like chunks), but what the library _exposes by default_ are the **final tokens** and IDs. If you want to _inspect_ pre-tokens, call the underlying Rust pre-tokenizer: # deps: pip install tokenizers>=0.15 transformers>=4.44 from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("unsloth/Qwen3-14B", use_fast=True) print(tok.backend_tokenizer.pre_tokenizer.pre_tokenize_str("今天天气真好,I wanna go swimming")) # -> [(pre_token_string, (start, end)), ...] This shows the whitespace splits and their offsets. (Vinsmoke Three) * **“Where does`Ġ` come from?”** It’s a visible space marker baked into GPT-2–style vocabularies so BPE can learn merges that depend on preceding whitespace. It appears in **tokens** , not pre-tokens. (GitHub) * **“Why byte-level at all?”** Two reasons: coverage with no `<unk>` and exact reversibility. BPE expects strings, so bytes are first mapped to visible Unicode, then merged; the decoder reverses that map. (GitHub) # Mental model you can trust Unicode text → Normalizer → Pre-tokenizer (ByteLevel): bytes→visible Unicode; split on whitespace; keep offsets → Model (BPE): merge mapped chars into vocab tokens; get token strings + IDs → Post-processor: add special tokens if needed → (on decode) Decoder (ByteLevel): visible Unicode → original bytes → human text This matches Hugging Face’s Tokenizers pipeline and API terminology. (Hugging Face) # Minimal checks you can run * **Pre-tokens (whitespace splits + offsets):** use `pre_tokenize_str` as above. (Google Colab) * **Final tokens (mapped-byte strings):** `tokenizer.convert_ids_to_tokens(...)`. * **Readable per-token spans:** ask for `return_offsets_mapping=True` and slice the original string. (Hugging Face) # Short references * HF Tokenizers: **Pre-tokenizers** (ByteLevel description) and **Decoders** (ByteLevel decoder). (Hugging Face) * HF Tokenizers: **Pipeline overview** and **offset mapping**. (Hugging Face) * GPT-2 space marker `Ġ` background. (GitHub) * Byte-level BPE rationale and UTF-8 coverage. (GitHub) Summary: a **pre-token** is a whitespace-delimited span with offsets; a **token** is a BPE-merged vocab string (plus its ID). No space before `I` means the pre-tokenizer **does not** split there.

Yeah. * * * Mostly right. Two fixes: 1. The ByteLevel **pre-tokenizer** splits on whitespace and remaps bytes to printable code points, but it does not hand you a visible list of single characters. It outputs “pre-tokens” with offsets. Then the **BPE model** merges those mapped characters into vocab tokens. Decoding later inverts the byte→Unicode mapping. (Hugging Face) 2. The English space is encoded into tokens with a visible space marker (e.g., `Ġ`). That’s why you see tokens like `Ġwanna`. Offsets can therefore include the leading space. (Hugging Face) # Walk-through on your example Input: `"今天天气真好,I wanna go swimming"` ## Pre-tokenizer output (conceptual) * Operation: normalize → UTF-8 bytes → map bytes to printable Unicode → **split on whitespace** → keep offsets. * Pre-tokens (by whitespace): `["今天天气真好,", "I", "wanna", "go", "swimming"]` * Character offsets (start, end) over the original string: `[(0,7), (8,9), (10,15), (16,18), (19,27)]` These are spans in the original text, not in the mapped “mojibake” string. (Hugging Face) ## BPE model output * Runs merges **inside each pre-token** over the mapped characters. * Typical token strings and offsets (character indexes in original text): * Chinese chunk: `['ä»Ĭ天','天æ°Ķ','羣','å¥1⁄2','ï1⁄4Į']` → `[(0,2),(2,4),(4,5),(5,6),(6,7)]` which correspond to `['今天','天气','真','好',',']` * English: `['I','Ġwanna','Ġgo','Ġswimming']` → `[(8,9),(9,15),(15,18),(18,27)]` where `Ġ` indicates the preceding space is part of the token span. * Map token strings → ids, e.g. `[100644,104307,88051,52801, ...]`. * No human decoding has happened yet; these are internal symbols. Qwen uses **byte-level BPE on UTF-8** , so this behavior is expected and guarantees no OOV. (Qwen) ## Post-processor * Adds special tokens (BOS/EOS, chat template pieces) if configured. It does not “fix” readability. (Hugging Face) ## Decoder (when you call `decode` / `batch_decode`) * Inverts the byte→Unicode mapping and restores spaces, yielding normal text. * Fast tokenizers also expose `return_offsets_mapping=True` so you can slice the original string per token without decoding each id. (Hugging Face) # Quick rules to remember * `tokenize()` / `convert_ids_to_tokens()` → raw vocab strings (mapped bytes). They will look garbled for non-ASCII. Correct by design. * `decode()` / `batch_decode()` → runs the decoder → human text. * `return_offsets_mapping=True` (fast tokenizers) → character spans over the original text for each final token. * English tokens may include a leading space (`Ġ...`), so their offsets can start at the space. This depends on tokenizer settings like `add_prefix_space` and post-processing; be mindful of offset edge cases. (Hugging Face) # Minimal verification snippet # deps: # pip install --upgrade transformers>=4.44 tokenizers>=0.15 from transformers import AutoTokenizer s = "今天天气真好,I wanna go swimming" tok = AutoTokenizer.from_pretrained("unsloth/Qwen3-14B", use_fast=True) enc = tok(s, add_special_tokens=False, return_offsets_mapping=True) tokens = tok.convert_ids_to_tokens(enc["input_ids"]) spans = [s[a:b] for a,b in enc["offset_mapping"]] print(tokens) # internal strings (byte-mapped), includes Ġ for spaces print(spans) # human-readable per-token text slices print(tok.decode(enc["input_ids"])) # original text """ ['ä»Ĭ天', '天æ°Ķ', '羣', 'å¥1⁄2', 'ï1⁄4Į', 'I', 'Ġwanna', 'Ġgo', 'Ġswimming'] ['今天', '天气', '真', '好', ',', 'I', ' wanna', ' go', ' swimming'] 今天天气真好,I wanna go swimming """

Yeah. * * * Mostly right. Two fixes: 1. The ByteLevel **pre-tokenizer** splits on whitespace and remaps bytes to printable code points, but it does not hand you a visible list of single characters. It outputs “pre-tokens” with offsets. Then the **BPE model** merges those mapped characters into vocab tokens. Decoding later inverts the byte→Unicode mapping. (Hugging Face) 2. The English space is encoded into tokens with a visible space marker (e.g., `Ġ`). That’s why you see tokens like `Ġwanna`. Offsets can therefore include the leading space. (Hugging Face) # Walk-through on your example Input: `"今天天气真好,I wanna go swimming"` ## Pre-tokenizer output (conceptual) * Operation: normalize → UTF-8 bytes → map bytes to printable Unicode → **split on whitespace** → keep offsets. * Pre-tokens (by whitespace): `["今天天气真好,", "I", "wanna", "go", "swimming"]` * Character offsets (start, end) over the original string: `[(0,7), (8,9), (10,15), (16,18), (19,27)]` These are spans in the original text, not in the mapped “mojibake” string. (Hugging Face) ## BPE model output * Runs merges **inside each pre-token** over the mapped characters. * Typical token strings and offsets (character indexes in original text): * Chinese chunk: `['ä»Ĭ天','天æ°Ķ','羣','å¥1⁄2','ï1⁄4Į']` → `[(0,2),(2,4),(4,5),(5,6),(6,7)]` which correspond to `['今天','天气','真','好',',']` * English: `['I','Ġwanna','Ġgo','Ġswimming']` → `[(8,9),(9,15),(15,18),(18,27)]` where `Ġ` indicates the preceding space is part of the token span. * Map token strings → ids, e.g. `[100644,104307,88051,52801, ...]`. * No human decoding has happened yet; these are internal symbols. Qwen uses **byte-level BPE on UTF-8** , so this behavior is expected and guarantees no OOV. (Qwen) ## Post-processor * Adds special tokens (BOS/EOS, chat template pieces) if configured. It does not “fix” readability. (Hugging Face) ## Decoder (when you call `decode` / `batch_decode`) * Inverts the byte→Unicode mapping and restores spaces, yielding normal text. * Fast tokenizers also expose `return_offsets_mapping=True` so you can slice the original string per token without decoding each id. (Hugging Face) # Quick rules to remember * `tokenize()` / `convert_ids_to_tokens()` → raw vocab strings (mapped bytes). They will look garbled for non-ASCII. Correct by design. * `decode()` / `batch_decode()` → runs the decoder → human text. * `return_offsets_mapping=True` (fast tokenizers) → character spans over the original text for each final token. * English tokens may include a leading space (`Ġ...`), so their offsets can start at the space. This depends on tokenizer settings like `add_prefix_space` and post-processing; be mindful of offset edge cases. (Hugging Face) # Minimal verification snippet # deps: # pip install --upgrade transformers>=4.44 tokenizers>=0.15 from transformers import AutoTokenizer s = "今天天气真好,I wanna go swimming" tok = AutoTokenizer.from_pretrained("unsloth/Qwen3-14B", use_fast=True) enc = tok(s, add_special_tokens=False, return_offsets_mapping=True) tokens = tok.convert_ids_to_tokens(enc["input_ids"]) spans = [s[a:b] for a,b in enc["offset_mapping"]] print(tokens) # internal strings (byte-mapped), includes Ġ for spaces print(spans) # human-readable per-token text slices print(tok.decode(enc["input_ids"])) # original text """ ['ä»Ĭ天', '天æ°Ķ', '羣', 'å¥1⁄2', 'ï1⁄4Į', 'I', 'Ġwanna', 'Ġgo', 'Ġswimming'] ['今天', '天气', '真', '好', ',', 'I', ' wanna', ' go', ' swimming'] 今天天气真好,I wanna go swimming """

Hi, I have trainee by tokenizer using the model BPE and pre tokenizer as ByteLevel `tokenizer = Tokenizer(models.BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel() tokenizer.decoder = decoders.ByteLevel() ` Now, my vocabulary is saved in bytes and tokenizer.tokenize give me output in bytes too which is obvious. `tokenizer.tokenize` output is ['Ġन', 'à¥ĩ', 'प', 'ा', 'ल', 'à¥Ģ', 'Ġà¤Ń', 'ा', 'ष', 'ा', 'म', 'ा', 'Ġय', 'à¥ĭ', 'Ġà¤ıà¤ķ', 'Ġà¤īद', 'ा', 'हरण', 'Ġह', 'à¥ĭ।'] . Is there way to save my vocabulary in unicode character rather than bytes and show tokens in unicode characters too?

Researchers at Meta introduced the Byte Latent Transformer (BLT), a tokenizer-free model that matches token-based performance while slashing inference costs by up to 50%. By dynamically segmenting byt...