#tokenizer
LightNews LightNews
restofworld.org
AI models can take up to 17 times more tokens to process non-Latin scripts than English, saddling users with a steep "tokenizer tax"
https://restofworld.org/2026/ai-tokenizer-tax-non-english-languages-cost-inequity/?utm_campaign=row-social&utm_source=bluesky
November 9, 2026 at 5:00 PM
restofworld.org
AI models can take up to 17 times more tokens to process non-Latin scripts than English, saddling users with a steep "tokenizer tax"
https://restofworld.org/2026/ai-tokenizer-tax-non-english-languages-cost-inequity/?utm_campaign=row-social&utm_source=bluesky
November 6, 2026 at 7:53 PM
violetbat.bsky.social
what is going on in qwen's tokenizer? What exactly is a "ĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠĠ"
November 4, 2026 at 1:52 PM
ablationlog.org
@barredspirals.comint.su A 27B weight matrix casually mapping hepatic enzyme collisions makes medical school look like an expensive tokenizer.
November 11, 2026 at 8:15 PM
throwable.bsky.social
📦 super-kernel/tokenizer 1.0.0-alpha.1

PHP class-like declaration tokenizer for SuperKernel.

🔗 https://github.com/super-kernel/tokenizer
November 10, 2026 at 6:00 AM
eev.ee
hmm, thought it might be interesting to throw a tokenizer at the pokémon emerald corpus and try to work out the most common words in the game are. main difficulty is that the script is all kana, but mecab + unidic work surprisingly well. current result is that the most common word is の, checks out
November 10, 2026 at 1:08 AM
sipirtu.com
Commercial language model APIs charge per token, so tokenizers that need more tokens for Indian languages increase cost and reduce efficiency.

Source: arXiv cs.CL
The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India
Abstract page for arXiv paper 2611.13058: The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India
arxiv.org
November 11, 2026 at 9:56 AM
restofworld.org
AI models can take up to 17 times more tokens to process non-Latin scripts than English, saddling users with a steep "tokenizer tax"
https://restofworld.org/2026/ai-tokenizer-tax-non-english-languages-cost-inequity/?utm_campaign=row-social&utm_source=bluesky
November 5, 2026 at 4:52 PM
throwable.bsky.social
📦 spiral/tokenizer 3.18.2

Static Analysis: Class and Invocation locators

🔗 https://github.com/spiral/tokenizer
November 9, 2026 at 8:21 AM
hackobar.bsky.social
Havon Multi-Word Tokenizer Outperforms GPT-4o on 161 Languages

What this means for you: You can reduce inference costs and latency for non-English users by switching to a more efficient tokenizer.
Havon Multi-Word Tokenizer Outperforms GPT-4o on 161 Languages
You can reduce inference costs and latency for non-English users by switching to a more efficient tokenizer.
arxiv.org
November 11, 2026 at 2:00 PM
osrepos.bsky.social
MOSS-TTS: Generate Long-Form Speech, Dialogue, and Sound
https://osrepos.com/repo/openmoss-moss-tts?utm_source=bluesky

MOSS-TTS is OpenMOSS’s model family for expressive speech and sound generation, built around Qwen3-based language-model architectures and the MOSS-Audio-Tokenizer.

#opensource
MOSS-TTS: Generate Long-Form Speech, Dialogue, and Sound
MOSS-TTS is OpenMOSS’s model family for expressive speech and sound generation, built around Qwen3-based language-model architectures and the MOSS-Audio-Tokenizer. It supports long-form narration, voice cloning, multi-speaker dialogue, voice design, sound effects, and real-time streaming.
osrepos.com
November 9, 2026 at 6:55 PM
🐧 **SentencePiece – language-independent text tokenizer**

SentencePiece trains language-independent subword tokenizers with BPE and unigram models, reversible encoding and text normalisation.

📰 Source: LinuxLinks
🔗 Link: https://www.linuxlinks.com/sentencepiece-language-independent-text-tokenizer/
SentencePiece – language-independent text tokenizer
SentencePiece trains language-independent subword tokenizers with BPE and unigram models, reversible encoding and text normalisation. The post SentencePiece – language-independent text tokenizer appeared first on LinuxLinks.
www.linuxlinks.com
November 10, 2026 at 6:21 PM
inautilo.bsky.social
#Business #Launches
Tokenizer Mono · A font to visualize AI tokens as you type them ilo.im/16hpob by George Mandis

_____
#OpenType #Fonts #AI #AiModels #AiTokens #Cost #Design #WebDesign #Development #WebDev
Tokenizer Mono
A monospace font that highlights LLM tokens as you type. No JavaScript, just OpenType substitution rules and a color table.
ilo.im
November 8, 2026 at 8:11 PM
throwable.bsky.social
📦 lalbert/tokenizer-engine 1.3.0

Elasticsearch-like text analysis (character filters, tokenizers, token filters) in pure PHP, UTF-8 aware.

🔗 https://github.com/lalbert/tokenizer-engine
November 9, 2026 at 7:11 PM
sebby.wsocial.eu
4/ Two runs, byte-level, 256-token vocabulary, no tokenizer at all: 195M on 690 MB Finnish Wikipedia, 501M on 1.53 GB FineWeb-2 + Wikipedia. One card, 10.5 h and 46 h.
November 10, 2026 at 4:29 PM
throwable.bsky.social
📦 lalbert/tokenizer-engine-for-wp 0.9.0

Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.

🔗 https://github.com/lalbert/tokenizer-engine-for-wp
November 10, 2026 at 9:16 AM
throwable.bsky.social
📦 lalbert/tokenizer-engine-for-wp 0.8.0

Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.

🔗 https://github.com/lalbert/tokenizer-engine-for-wp
November 9, 2026 at 7:11 PM
togelius.bsky.social
Seriously, if the tokenizer treats it as a single token, is it likely to in some sense be treated as more primitive or elemental further down the layers? If it's a single token, it's something that just exists in the world; if it's multiple tokens, you have to assemble it. Or?
November 7, 2026 at 8:30 PM
sipirtu.com
Researchers trained tokenizers from scratch on human-written text in about 200 languages. Havon v3 is a byte-level BPE tokenizer.

Source: arXiv cs.CL
A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English
Abstract page for arXiv paper 2611.13059: A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English
arxiv.org
November 11, 2026 at 10:28 AM
throwable.bsky.social
📦 lalbert/tokenizer-engine-for-wp 0.7.0

Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.

🔗 https://github.com/lalbert/tokenizer-engine-for-wp
November 9, 2026 at 3:14 PM
sebby.wsocial.eu
3/ Finnish is a good test: agglutinative, long words, bad tokenizer economics; ~2.7 bytes per token against 4.35 for English on Llama-3's tokenizer. If learned chunking were going to find morphology anywhere, it should be here.
November 10, 2026 at 4:29 PM
eev.ee
but on the other hand splitting a "word" into individual kana is not always wrong. したてんだよね is a perfectly reasonable and common way to end a sentence, and it is in fact seven words, if you count した as two (which this tokenizer does)
November 10, 2026 at 1:29 AM
throwable.bsky.social
📦 lalbert/tokenizer-engine-for-wp 0.4.1

Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.

🔗 https://github.com/lalbert/tokenizer-engine-for-wp
November 8, 2026 at 9:11 PM
throwable.bsky.social
📦 lalbert/tokenizer-engine-for-wp 0.3.1

Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.

🔗 https://github.com/lalbert/tokenizer-engine-for-wp
November 8, 2026 at 6:59 PM
throwable.bsky.social
📦 lalbert/tokenizer-engine-for-wp 0.6.0

Search, related content and internal links for WordPress, with Elasticsearch-like text analysis and no search server.

🔗 https://github.com/lalbert/tokenizer-engine-for-wp
November 8, 2026 at 11:59 PM