june 1: the tokenization of trans women of colour
June 1, 2025 at 2:22 PM
june 1: the tokenization of trans women of colour
#EMNLP has a nice set of tokenization/subword modeling papers this year.
It's a good mix of tokenization algorithms, tokenization evaluation, tokenization-free methods, and subword embedding probing. Lmk if I missed some!
Here is a list with links + presentation time (in chronological order).
It's a good mix of tokenization algorithms, tokenization evaluation, tokenization-free methods, and subword embedding probing. Lmk if I missed some!
Here is a list with links + presentation time (in chronological order).
November 11, 2024 at 10:38 PM
#EMNLP has a nice set of tokenization/subword modeling papers this year.
It's a good mix of tokenization algorithms, tokenization evaluation, tokenization-free methods, and subword embedding probing. Lmk if I missed some!
Here is a list with links + presentation time (in chronological order).
It's a good mix of tokenization algorithms, tokenization evaluation, tokenization-free methods, and subword embedding probing. Lmk if I missed some!
Here is a list with links + presentation time (in chronological order).
but in a way, it does add unnecessary stress because it can and will turn into tokenization.
blugh
blugh
May 26, 2025 at 1:40 AM
but in a way, it does add unnecessary stress because it can and will turn into tokenization.
blugh
blugh
DEI is not and has never been about tokenization. Tokenization is a tool of incompetent racists seeking to sabotage diverse communities because they rely on white privilege for their managerial positions. So they hire incompetent tokens to maintain their top-level status and avoid scrutiny.
August 14, 2026 at 5:14 PM
DEI is not and has never been about tokenization. Tokenization is a tool of incompetent racists seeking to sabotage diverse communities because they rely on white privilege for their managerial positions. So they hire incompetent tokens to maintain their top-level status and avoid scrutiny.
🚨 NEW WORKSHOP ALERT 🚨
We're thrilled to announce the first-ever Tokenization Workshop (TokShop) at #ICML2025 @icmlconf.bsky.social! 🎉
Submissions are open for work on tokenization across all areas of machine learning.
📅 Submission deadline: May 30, 2025
🔗 tokenization-workshop.github.io
We're thrilled to announce the first-ever Tokenization Workshop (TokShop) at #ICML2025 @icmlconf.bsky.social! 🎉
Submissions are open for work on tokenization across all areas of machine learning.
📅 Submission deadline: May 30, 2025
🔗 tokenization-workshop.github.io
Tokenization Workshop @ ICML 2025
tokenization-workshop.github.io
April 15, 2025 at 5:23 PM
🚨 NEW WORKSHOP ALERT 🚨
We're thrilled to announce the first-ever Tokenization Workshop (TokShop) at #ICML2025 @icmlconf.bsky.social! 🎉
Submissions are open for work on tokenization across all areas of machine learning.
📅 Submission deadline: May 30, 2025
🔗 tokenization-workshop.github.io
We're thrilled to announce the first-ever Tokenization Workshop (TokShop) at #ICML2025 @icmlconf.bsky.social! 🎉
Submissions are open for work on tokenization across all areas of machine learning.
📅 Submission deadline: May 30, 2025
🔗 tokenization-workshop.github.io
'tokenization' seems like another step into just totally transforming retail 'investing' into outright gambling
November 20, 2025 at 6:30 PM
'tokenization' seems like another step into just totally transforming retail 'investing' into outright gambling
Whoa, I thought it was a tokenization problem. But some malign force just doesn’t want us to spell strawbery.
May 29, 2026 at 4:50 AM
Whoa, I thought it was a tokenization problem. But some malign force just doesn’t want us to spell strawbery.
A curated benchmark set for protein structure tokenization put together by, of all entities, Amazon. Apparently the sweet spot for alphabet size is 256-1024 tokens arxiv.org/abs/2503.00089
Protein Structure Tokenization: Benchmarking and New Recipe
Recent years have witnessed a surge in the development of protein structural tokenization methods, which chunk protein 3D structures into discrete or continuous representations. Structure tokenization...
arxiv.org
March 10, 2025 at 1:46 PM
A curated benchmark set for protein structure tokenization put together by, of all entities, Amazon. Apparently the sweet spot for alphabet size is 256-1024 tokens arxiv.org/abs/2503.00089
The "b l u e b e r r y" miscount is the damning one, as it has nothing to do with tokenization.
August 10, 2025 at 12:06 AM
The "b l u e b e r r y" miscount is the damning one, as it has nothing to do with tokenization.
I artisanally strip stop words and do small batch tokenization
July 16, 2026 at 10:43 PM
I artisanally strip stop words and do small batch tokenization
#Tokenization is undeniably a key player in the success story of #LLMs but we poorly understand why.
I want to highlight progress we made in understanding the role of tokenization, developing the core incidents and mitigating its problems. 🧵👇
I want to highlight progress we made in understanding the role of tokenization, developing the core incidents and mitigating its problems. 🧵👇
October 30, 2024 at 6:26 PM
#Tokenization is undeniably a key player in the success story of #LLMs but we poorly understand why.
I want to highlight progress we made in understanding the role of tokenization, developing the core incidents and mitigating its problems. 🧵👇
I want to highlight progress we made in understanding the role of tokenization, developing the core incidents and mitigating its problems. 🧵👇
What happens if we tokenize cat as [ca, t] rather than [cat]?
LLMs are trained on just one tokenization per word, but they still understand alternative tokenizations. We show that this can be exploited to bypass safety filters without changing the text itself.
#AI #LLMs #tokenization #alignment
LLMs are trained on just one tokenization per word, but they still understand alternative tokenizations. We show that this can be exploited to bypass safety filters without changing the text itself.
#AI #LLMs #tokenization #alignment
March 11, 2025 at 11:13 PM
What happens if we tokenize cat as [ca, t] rather than [cat]?
LLMs are trained on just one tokenization per word, but they still understand alternative tokenizations. We show that this can be exploited to bypass safety filters without changing the text itself.
#AI #LLMs #tokenization #alignment
LLMs are trained on just one tokenization per word, but they still understand alternative tokenizations. We show that this can be exploited to bypass safety filters without changing the text itself.
#AI #LLMs #tokenization #alignment
🎥 Videos from our Tokenization Workshop are now live! Watch invited talks, panel discussions, and the best paper presentation at icml.cc/virtual/2025... #Tokenization #NLP #LLMs
Tokenization Workshop (TokShop)ICML 2025
icml.cc
August 25, 2025 at 8:36 AM
🎥 Videos from our Tokenization Workshop are now live! Watch invited talks, panel discussions, and the best paper presentation at icml.cc/virtual/2025... #Tokenization #NLP #LLMs
In our experiments, Right-to-left tokenization proved to be quite superior to the standard left-to-right tokenization for math!
The last few ablation runs are just about done, and Garreth is now writing a full blog post coming soon.
Have a nice weekend folks!
The last few ablation runs are just about done, and Garreth is now writing a full blog post coming soon.
Have a nice weekend folks!
November 24, 2024 at 11:05 AM
In our experiments, Right-to-left tokenization proved to be quite superior to the standard left-to-right tokenization for math!
The last few ablation runs are just about done, and Garreth is now writing a full blog post coming soon.
Have a nice weekend folks!
The last few ablation runs are just about done, and Garreth is now writing a full blog post coming soon.
Have a nice weekend folks!
🚀 With Meta's recent paper replacing tokenization in LLMs with patches 🩹, I figured that it's a great time to revisit how tokenization has evolved over the years using everyone's favourite medium - memes!
Let's take a trip down memory lane!
[1/N]
Let's take a trip down memory lane!
[1/N]
December 16, 2024 at 5:31 PM
🚀 With Meta's recent paper replacing tokenization in LLMs with patches 🩹, I figured that it's a great time to revisit how tokenization has evolved over the years using everyone's favourite medium - memes!
Let's take a trip down memory lane!
[1/N]
Let's take a trip down memory lane!
[1/N]
The tokenization of patriotism is another corrosive aspect of Trumpism. You can’t chant “USA” and then cheer the man who pardons J6ers and kowtows to Putin.
Just stop.
Just stop.
March 5, 2025 at 2:31 AM
The tokenization of patriotism is another corrosive aspect of Trumpism. You can’t chant “USA” and then cheer the man who pardons J6ers and kowtows to Putin.
Just stop.
Just stop.
bad tokenization in the wild
August 3, 2026 at 9:05 PM
bad tokenization in the wild
This makes me anger. Emilia Perez is deeply transphobic at its core, & while Karla is GREAT in it, her casting feels like a shield for the cis creator’s refusal to listen to trans voices. Her claim that only trans filmmakers can critique it is galling when Hollywood only supports us in tokenization.
Insanely galling way to defend your controversial movie considering 2024 is probably one of the best year for trans-led cinema and absolutely none of them are getting the blowback Emilia Pérez is?
What are Jane Schoenbrun and Vera Drew the only trans people allowed to criticise your movie?
What are Jane Schoenbrun and Vera Drew the only trans people allowed to criticise your movie?
January 7, 2025 at 12:27 AM
This makes me anger. Emilia Perez is deeply transphobic at its core, & while Karla is GREAT in it, her casting feels like a shield for the cis creator’s refusal to listen to trans voices. Her claim that only trans filmmakers can critique it is galling when Hollywood only supports us in tokenization.
The Art of Tokenization: Breaking Down Text for AI towardsdatascience.com/the-art-of-t...
The Art of Tokenization: Breaking Down Text for AI
Demystifying NLP: From Text to Embeddings
towardsdatascience.com
September 30, 2024 at 2:13 AM
The Art of Tokenization: Breaking Down Text for AI towardsdatascience.com/the-art-of-t...
This is what antiIsrael people use for tokenization
1/2
1/2
April 12, 2026 at 11:50 PM
This is what antiIsrael people use for tokenization
1/2
1/2