Tokenization and analysis pipeline for full-text search - turbopuffer/alyze

We built BPE from scratch on 83,159 words and measured it. Urdu costs 3.75x the tokens of English, and vocab size stops paying off after 5,000 merges.

Decoding the GPT-style token representation.

SentencePiece trains language-independent subword tokenizers with BPE and unigram models, reversible encoding and text normalisation. The post SentencePiece – language-independent text tokenizer appeared first on LinuxLinks.

SentencePiece is a text tokenizer and detokenizer designed for natural language processing and neural network models. It converts text into subword units that can be used as input for machine translation, language modelling and text generation systems. Unlike traditional tokenizers, SentencePiece operates directly on raw Unicode text without requiring language-specific word segmentation. This makes it particularly useful for multilingual applications and languages where words are not separated by spaces. The software supports training custom tokenization models from text corpora, allowing developers to determine the vocabulary size and segmentation method appropriate for their applications. Its tokenization process preserves whitespace information, enabling the original text to be reconstructed from its tokens. This is free and open source software. #### Key Features * Unsupervised training of subword tokenization models directly from raw text. * Byte-Pair Encoding (BPE) and unigram language model tokenization algorithms. * Reversible tokenization and detokenization preserving whitespace information. * Language-independent processing of Unicode text. * Configurable vocabulary sizes for neural language models. * Subword regularization and BPE-dropout for generating alternative text segmentations during training. * Text normalization with configurable Unicode normalization rules. * Encode text as subword strings or numerical token identifiers. * Support for user-defined symbols, control symbols and special tokens. * Byte fallback for representing characters outside the trained vocabulary. * Vocabulary restrictions and configurable character coverage. * Multithreaded tokenization and model training. * Self-contained model files storing vocabulary, normalization rules and segmentation information. * Command-line tools and APIs for integrating tokenization into other applications. * Lightweight runtime for performing tokenization with reduced dependencies. * Cross-platform support – runs on Linux, macOS, and Windows. **Website:** **github.com/google/sentencepiece** **Support:** **Developer:** Taku Kudo and contributors **License:** Apache License 2.0 SentencePiece is written in C++. Learn C++ with our recommended **free books** and **free tutorials**. * * * #### Related Software **C++ Natural Language Processing Tools** --- **text2vec**| Framework with API for text analysis and natural language processing **Moses**| Statistical machine translation system **TiMBL**| Tilburg Memory-Based Learner **MITIE**| MIT Information Extraction **MeTA**| Modern C++ data sciences toolkit **Colibri Core**| Efficient n-gram & skipgram modelling on text corpora **CRF++**| Yet Another CRF toolkit **BLLIP Parser**| Statistical natural language parser Read our verdict in the **software roundup**. * * * LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of **recommended applications** , including alternatives to popular proprietary software. We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our **Linux resources** to discover more. Know a useful free and open source Linux application we haven't covered? **Suggest a program** for inclusion. Has your software been featured on LinuxLinks? Display one of our **Featured on LinuxLinks badges** on your website or GitHub README. ---

ArXiv link for Tokenizer Myths and Where To Find Them

ArXiv link for Tokenizer Myths and Where To Find Them

Experiment with and compare different tokenizers

Hi all — I measured how many tokens Urdu costs compared with English for the same meaning, and wanted to share the numbers and ask for feedback. I built a small parallel corpus (30 sentence pairs, 17 domains, CC-BY-4.0) and tokenized both sides with three tokenizers: Tokenizer | Vocab | English tokens | Urdu tokens | Inflation ---|---|---|---|--- GPT-2 | ~50k | 320 | 1,999 | **5.39×** Mistral v0.3 | ~32k | 359 | 1,554 | **3.73×** Qwen2.5 | ~151k | 318 | 1,002 | **2.72×** Per character it’s starker: English gets ~5.35 characters per token on GPT-2, Urdu gets ~1.34. What surprised me is that vocabulary size isn’t the whole story — GPT-2 has a _larger_ vocabulary than Mistral (50k vs 32k) and does worse. What the tokenizer was trained on seems to matter at least as much. * Space (runs in-browser, shows the actual subword split): [link] * Corpus: [link] Two caveats I want to be upfront about: 30 sentences is a pilot, not a precise multiplier, and the corpus is formal register only — no conversational or Roman Urdu. Has anyone measured this for other South Asian languages? I’d like to extend the corpus to Pashto, Sindhi and Shahmukhi Punjabi.