Well, I get why people like embeddings
July 11, 2025 at 4:35 AM
Well, I get why people like embeddings
word embeddings are a bitch
Maybe there’s something to those sentience claims.
iPhone Dictation Feature Transcribes the Word ‘Racist’ as ‘Trump’
The company said it was working to fix the problem after iPhone users began reporting the issue.
www.nytimes.com
February 25, 2025 at 9:25 PM
word embeddings are a bitch
Introducing DroPE: Extending Context by Dropping Positional Embeddings
We found embeddings like RoPE aid training but bottleneck long-sequence generalization. Our solution’s simple: treat them as a temporary training scaffold, not a permanent necessity.
arxiv.org/abs/2512.12167
pub.sakana.ai/DroPE
We found embeddings like RoPE aid training but bottleneck long-sequence generalization. Our solution’s simple: treat them as a temporary training scaffold, not a permanent necessity.
arxiv.org/abs/2512.12167
pub.sakana.ai/DroPE
January 12, 2026 at 4:07 AM
Introducing DroPE: Extending Context by Dropping Positional Embeddings
We found embeddings like RoPE aid training but bottleneck long-sequence generalization. Our solution’s simple: treat them as a temporary training scaffold, not a permanent necessity.
arxiv.org/abs/2512.12167
pub.sakana.ai/DroPE
We found embeddings like RoPE aid training but bottleneck long-sequence generalization. Our solution’s simple: treat them as a temporary training scaffold, not a permanent necessity.
arxiv.org/abs/2512.12167
pub.sakana.ai/DroPE
ICYMI, ace work from @thisisaaronland.bsky.social on shared vector embeddings for images as 'an exercise in practicing agency in our work [,] in producing tangible and reusable artifacts which have a longer shelf life' www.aaronland.info/weblog/2026/...
shared vector embeddings (fantastic futures '26)
Shared vector embeddings simply makes it possible to do something similar at a smaller, more targeted, scale with fewer resources and less coordination.
www.aaronland.info
September 24, 2026 at 1:16 PM
ICYMI, ace work from @thisisaaronland.bsky.social on shared vector embeddings for images as 'an exercise in practicing agency in our work [,] in producing tangible and reusable artifacts which have a longer shelf life' www.aaronland.info/weblog/2026/...
anyway turns out platonism is true
arxiv.org/abs/2505.12540
arxiv.org/abs/2505.12540
Harnessing the Universal Geometry of Embeddings
We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates an...
arxiv.org
May 22, 2025 at 1:34 AM
anyway turns out platonism is true
arxiv.org/abs/2505.12540
arxiv.org/abs/2505.12540
Use LLM embeddings as input to your regression and ensemble models?
Google Deepmind researches LLM embeddings when used as features for traditional regression tasks.
- LLM embeddings are dimensionally robust, i.e. regression performance can remain strong even over high-dimensional data.
Google Deepmind researches LLM embeddings when used as features for traditional regression tasks.
- LLM embeddings are dimensionally robust, i.e. regression performance can remain strong even over high-dimensional data.
November 25, 2024 at 6:44 PM
Use LLM embeddings as input to your regression and ensemble models?
Google Deepmind researches LLM embeddings when used as features for traditional regression tasks.
- LLM embeddings are dimensionally robust, i.e. regression performance can remain strong even over high-dimensional data.
Google Deepmind researches LLM embeddings when used as features for traditional regression tasks.
- LLM embeddings are dimensionally robust, i.e. regression performance can remain strong even over high-dimensional data.
Vector embeddings are just gematria that works.
May 10, 2026 at 6:52 PM
Vector embeddings are just gematria that works.
In addition to this intro to embeddings:
technicalwriting.dev/data/embeddi...
I strongly recommend this survey of stuff you can build with embeddings by @simonwillison.net
simonwillison.net/2023/Oct/23/...
And this deep dive into what embeddings are + how they work by @vickiboykis.com
technicalwriting.dev/data/embeddi...
I strongly recommend this survey of stuff you can build with embeddings by @simonwillison.net
simonwillison.net/2023/Oct/23/...
And this deep dive into what embeddings are + how they work by @vickiboykis.com
What are embeddings?
A deep-dive into machine learning embeddings.
vickiboykis.com
November 11, 2024 at 4:55 PM
In addition to this intro to embeddings:
technicalwriting.dev/data/embeddi...
I strongly recommend this survey of stuff you can build with embeddings by @simonwillison.net
simonwillison.net/2023/Oct/23/...
And this deep dive into what embeddings are + how they work by @vickiboykis.com
technicalwriting.dev/data/embeddi...
I strongly recommend this survey of stuff you can build with embeddings by @simonwillison.net
simonwillison.net/2023/Oct/23/...
And this deep dive into what embeddings are + how they work by @vickiboykis.com
NEW §
GenAI systems have to manipulate vast quantities of data. Bharani and I describe a key tool to do this - Embeddings - transforming large data blocks into numeric vectors so that embeddings near each other represent related concepts
martinfowler.com/articles/gen...
GenAI systems have to manipulate vast quantities of data. Bharani and I describe a key tool to do this - Embeddings - transforming large data blocks into numeric vectors so that embeddings near each other represent related concepts
martinfowler.com/articles/gen...
Emerging Patterns in Building GenAI Products
Patterns from our colleagues
martinfowler.com
January 29, 2025 at 4:01 PM
NEW §
GenAI systems have to manipulate vast quantities of data. Bharani and I describe a key tool to do this - Embeddings - transforming large data blocks into numeric vectors so that embeddings near each other represent related concepts
martinfowler.com/articles/gen...
GenAI systems have to manipulate vast quantities of data. Bharani and I describe a key tool to do this - Embeddings - transforming large data blocks into numeric vectors so that embeddings near each other represent related concepts
martinfowler.com/articles/gen...
Just realized BlueSky allows sharing valuable stuff cause it doesn't punish links. 🤩
Let's start with "What are embeddings" by @vickiboykis.com
The book is a great summary of embeddings, from history to modern approaches.
The best part: it's free.
Link: vickiboykis.com/what_are_emb...
Let's start with "What are embeddings" by @vickiboykis.com
The book is a great summary of embeddings, from history to modern approaches.
The best part: it's free.
Link: vickiboykis.com/what_are_emb...
November 22, 2024 at 11:13 AM
Just realized BlueSky allows sharing valuable stuff cause it doesn't punish links. 🤩
Let's start with "What are embeddings" by @vickiboykis.com
The book is a great summary of embeddings, from history to modern approaches.
The best part: it's free.
Link: vickiboykis.com/what_are_emb...
Let's start with "What are embeddings" by @vickiboykis.com
The book is a great summary of embeddings, from history to modern approaches.
The best part: it's free.
Link: vickiboykis.com/what_are_emb...
The more I think about LLMs, the more I think it's important for students to know how word embeddings work. I wonder if there's a way to use something like this to get (non-CS) students thinking about what embeddings represent. apple.github.io/embedding-at...
Embedding Atlas
Interactive visualizations for large-scale embeddings. Effortlessly explore, filter, and search through rich metadata.
apple.github.io
February 20, 2026 at 9:27 PM
The more I think about LLMs, the more I think it's important for students to know how word embeddings work. I wonder if there's a way to use something like this to get (non-CS) students thinking about what embeddings represent. apple.github.io/embedding-at...
these word embeddings are then entangled with positional embeddings, which are just a periodic function where every word is assigned to its own period of the function. "the peak of a sine wave" might be such a period, although the way this is done now is somewhat more complex than that.
October 13, 2025 at 6:46 PM
these word embeddings are then entangled with positional embeddings, which are just a periodic function where every word is assigned to its own period of the function. "the peak of a sine wave" might be such a period, although the way this is done now is somewhat more complex than that.
LLMs are injective and invertible.
They show that different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space.
Paper: www.arxiv.org/abs/2510.15511
They show that different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space.
Paper: www.arxiv.org/abs/2510.15511
October 28, 2025 at 2:05 AM
LLMs are injective and invertible.
They show that different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space.
Paper: www.arxiv.org/abs/2510.15511
They show that different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space.
Paper: www.arxiv.org/abs/2510.15511
Quick, post sparse embeddings as numpy arrays discourse while the vector db people are not here yet
November 11, 2024 at 2:46 AM
Quick, post sparse embeddings as numpy arrays discourse while the vector db people are not here yet
omg any day now computers will be able to add
May 28, 2024 at 9:18 PM
omg any day now computers will be able to add
Some similarities between our brains & LLMs: “The study revealed a remarkable alignment between the neural activity in the human brain's speech areas and the model's speech embeddings & between the neural activity in the brain’s language area and the model's language embeddings.” goo.gle/4iiUoNj
March 22, 2025 at 5:36 PM
Some similarities between our brains & LLMs: “The study revealed a remarkable alignment between the neural activity in the human brain's speech areas and the model's speech embeddings & between the neural activity in the brain’s language area and the model's language embeddings.” goo.gle/4iiUoNj
I should do more stuff with embeddings
September 4, 2025 at 6:14 PM
I should do more stuff with embeddings
i love embeddings and semantic search so much
May 10, 2026 at 11:29 PM
i love embeddings and semantic search so much
shit someone finally added character information directly into the llm. i was looking at finally rigging this up but they already did it arxiv.org/abs/2601.180...
Spelling Bee Embeddings for Language Modeling
We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on ...
arxiv.org
February 28, 2026 at 7:00 AM
shit someone finally added character information directly into the llm. i was looking at finally rigging this up but they already did it arxiv.org/abs/2601.180...
I made it by fitting a recommendation-engine style model with dot products to predict the probability that a person X follows person Y with 128-dim dimensions for "following" and "followers" and then squishing the followers embeddings into 2D with various iterations of tSNE.
February 8, 2026 at 11:04 PM
I made it by fitting a recommendation-engine style model with dot products to predict the probability that a person X follows person Y with 128-dim dimensions for "following" and "followers" and then squishing the followers embeddings into 2D with various iterations of tSNE.
👋 Hey Bsky! I spent the last year reading about embeddings, which make up large language models. When I was doing research, though, I couldn't find anything that explained them in a business, engineering, AND math context. So I wrote a paper. 🚀
https://vickiboykis.com/what_are_embeddings/
https://vickiboykis.com/what_are_embeddings/
What are embeddings?
A deep-dive into machine learning embeddings.
vickiboykis.com
June 5, 2023 at 12:50 PM
👋 Hey Bsky! I spent the last year reading about embeddings, which make up large language models. When I was doing research, though, I couldn't find anything that explained them in a business, engineering, AND math context. So I wrote a paper. 🚀
https://vickiboykis.com/what_are_embeddings/
https://vickiboykis.com/what_are_embeddings/
After 2013, word2vec embeddings started making the classifiers work better. Contextualized embeddings (ELMo, BERT) were even better, but still just fed vectors into stacks of classifiers.
September 22, 2026 at 7:57 PM
After 2013, word2vec embeddings started making the classifiers work better. Contextualized embeddings (ELMo, BERT) were even better, but still just fed vectors into stacks of classifiers.