Here's my blog on how to leverage the stochastic nature of VLMs to identify areas of potential divergence (most likely areas for errors).
Read more here! wjbmattingly.com/blog/where-t...
Read more here! wjbmattingly.com/blog/where-t...
September 24, 2026 at 3:58 PM
Here's my blog on how to leverage the stochastic nature of VLMs to identify areas of potential divergence (most likely areas for errors).
Read more here! wjbmattingly.com/blog/where-t...
Read more here! wjbmattingly.com/blog/where-t...
Unfortunately, our submission to #NeurIPS didn’t go through with (5,4,4,3). But because I think it’s an excellent paper, I decided to share it anyway.
We show how to efficiently apply Bayesian learning in VLMs, improve calibration, and do active learning. Cool stuff!
📝 arxiv.org/abs/2412.06014
We show how to efficiently apply Bayesian learning in VLMs, improve calibration, and do active learning. Cool stuff!
📝 arxiv.org/abs/2412.06014
Post-hoc Probabilistic Vision-Language Models
Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descripti...
arxiv.org
September 18, 2025 at 8:34 PM
Unfortunately, our submission to #NeurIPS didn’t go through with (5,4,4,3). But because I think it’s an excellent paper, I decided to share it anyway.
We show how to efficiently apply Bayesian learning in VLMs, improve calibration, and do active learning. Cool stuff!
📝 arxiv.org/abs/2412.06014
We show how to efficiently apply Bayesian learning in VLMs, improve calibration, and do active learning. Cool stuff!
📝 arxiv.org/abs/2412.06014
Use alt text. Generative AI models use VLMs to automatically generate labels for images they scrape. Not including alt-text does not impact them negatively.
October 31, 2024 at 5:16 AM
Use alt text. Generative AI models use VLMs to automatically generate labels for images they scrape. Not including alt-text does not impact them negatively.
I did a 1 hr speed-run on multimodal computer vision (VLMs, multimodal retrieval, zero-shot vision) in MIT AI Visions
it's up on youtube by popular demand www.youtube.com/embed/_TlhKH...
it's up on youtube by popular demand www.youtube.com/embed/_TlhKH...
November 1, 2024 at 5:51 PM
I did a 1 hr speed-run on multimodal computer vision (VLMs, multimodal retrieval, zero-shot vision) in MIT AI Visions
it's up on youtube by popular demand www.youtube.com/embed/_TlhKH...
it's up on youtube by popular demand www.youtube.com/embed/_TlhKH...
Are there conceptual directions in VLMs that transcend modality? Check out our COLM oral spotlight 🔦 paper! We use SAEs to analyze the multimodality of linear concepts in VLMs
with @chloesu07.bsky.social, @thomasfel.bsky.social, @shamkakade.bsky.social and Stephanie Gil
arxiv.org/abs/2504.11695
with @chloesu07.bsky.social, @thomasfel.bsky.social, @shamkakade.bsky.social and Stephanie Gil
arxiv.org/abs/2504.11695
September 17, 2025 at 7:12 PM
Are there conceptual directions in VLMs that transcend modality? Check out our COLM oral spotlight 🔦 paper! We use SAEs to analyze the multimodality of linear concepts in VLMs
with @chloesu07.bsky.social, @thomasfel.bsky.social, @shamkakade.bsky.social and Stephanie Gil
arxiv.org/abs/2504.11695
with @chloesu07.bsky.social, @thomasfel.bsky.social, @shamkakade.bsky.social and Stephanie Gil
arxiv.org/abs/2504.11695
🚨New preprint alert! 🚨
Do multimodal LLMs (VLMs) reason about high-level visual perception like humans do? We asked over 2000 human observers and 18 VLMs to describe scenes using 15 different tasks, ranging from general knowledge, affordances, affect, sensory experiences, and future prediction. 1/
Do multimodal LLMs (VLMs) reason about high-level visual perception like humans do? We asked over 2000 human observers and 18 VLMs to describe scenes using 15 different tasks, ranging from general knowledge, affordances, affect, sensory experiences, and future prediction. 1/
March 30, 2026 at 1:54 PM
🚨New preprint alert! 🚨
Do multimodal LLMs (VLMs) reason about high-level visual perception like humans do? We asked over 2000 human observers and 18 VLMs to describe scenes using 15 different tasks, ranging from general knowledge, affordances, affect, sensory experiences, and future prediction. 1/
Do multimodal LLMs (VLMs) reason about high-level visual perception like humans do? We asked over 2000 human observers and 18 VLMs to describe scenes using 15 different tasks, ranging from general knowledge, affordances, affect, sensory experiences, and future prediction. 1/
Our latest paper, “Visual language models show widespread visual deficits on neuropsychological tests”, is now out in Nature Machine Intelligence: www.nature.com/articles/s42...
Non-paywalled version:
arxiv.org/abs/2504.10786
Tweet thread below from first author @genetang.bsky.social...
Non-paywalled version:
arxiv.org/abs/2504.10786
Tweet thread below from first author @genetang.bsky.social...
Visual language models show widespread visual deficits on neuropsychological tests - Nature Machine Intelligence
Tangtartharakul and Storrs use standardized neuropsychological tests to compare human visual abilities with those of visual language models (VLMs). They report that while VLMs excel in high-level obje...
www.nature.com
February 9, 2026 at 2:40 AM
Our latest paper, “Visual language models show widespread visual deficits on neuropsychological tests”, is now out in Nature Machine Intelligence: www.nature.com/articles/s42...
Non-paywalled version:
arxiv.org/abs/2504.10786
Tweet thread below from first author @genetang.bsky.social...
Non-paywalled version:
arxiv.org/abs/2504.10786
Tweet thread below from first author @genetang.bsky.social...
There are now several benchmarks testing spatial reasoning and agent capabilities of LLMs and VLMs:
- arxiv.org/abs/2410.06468 (does spatial cognition ...)
- arxiv.org/abs/2307.06281 (MMBench)
- arxiv.org/abs/2411.13543 (BALROG) - additional points for the LOTR ref.
- arxiv.org/abs/2410.06468 (does spatial cognition ...)
- arxiv.org/abs/2307.06281 (MMBench)
- arxiv.org/abs/2411.13543 (BALROG) - additional points for the LOTR ref.
November 24, 2024 at 5:19 PM
There are now several benchmarks testing spatial reasoning and agent capabilities of LLMs and VLMs:
- arxiv.org/abs/2410.06468 (does spatial cognition ...)
- arxiv.org/abs/2307.06281 (MMBench)
- arxiv.org/abs/2411.13543 (BALROG) - additional points for the LOTR ref.
- arxiv.org/abs/2410.06468 (does spatial cognition ...)
- arxiv.org/abs/2307.06281 (MMBench)
- arxiv.org/abs/2411.13543 (BALROG) - additional points for the LOTR ref.
The visual world is composed of objects, and those objects are composed of features. But do VLMs exploit this compositional structure when processing multi-object scenes? In our 🆒🆕 #ICLR2026 paper, we find they do – via emergent symbolic mechanisms for visual binding. 🧵👇
February 5, 2026 at 8:55 PM
The visual world is composed of objects, and those objects are composed of features. But do VLMs exploit this compositional structure when processing multi-object scenes? In our 🆒🆕 #ICLR2026 paper, we find they do – via emergent symbolic mechanisms for visual binding. 🧵👇
What is common knowledge in your field, but shocks outsiders?
Most of us know that LLMs, VLMs, foundation models etc. are just statistical language models trained in big data and that they neither have emerged capabilities nor are they able to really reason.
Most of us know that LLMs, VLMs, foundation models etc. are just statistical language models trained in big data and that they neither have emerged capabilities nor are they able to really reason.
the field of computer vision refers to people as "objects," and you're the outlier if you find this questionable/unsettling
What is common knowledge in your field, but shocks outsiders?
You absolutely can lie via data & math & everything about how you present data is an argument for how to interpret/understand it
Everything we know is founded on best guesses & analogies
All technoscience is enmeshed with human values
You absolutely can lie via data & math & everything about how you present data is an argument for how to interpret/understand it
Everything we know is founded on best guesses & analogies
All technoscience is enmeshed with human values
June 18, 2025 at 4:05 PM
What is common knowledge in your field, but shocks outsiders?
Most of us know that LLMs, VLMs, foundation models etc. are just statistical language models trained in big data and that they neither have emerged capabilities nor are they able to really reason.
Most of us know that LLMs, VLMs, foundation models etc. are just statistical language models trained in big data and that they neither have emerged capabilities nor are they able to really reason.
Apple has published a paper & model based on Qwen 3.5 9B for processing large amounts of multi document text, compressed as images, LensVLM 9B:
huggingface.co/bartowski/Le...
github.com/apple-aiml-r...
arxiv.org/abs/2605.07019
huggingface.co/bartowski/Le...
github.com/apple-aiml-r...
arxiv.org/abs/2605.07019
LensVLM: Selective Context Expansion for Compressed Visual Representation of Text
Vision Language Models (VLMs) offer the exciting possibility of processing text as rendered images, bypassing the need for tokenizing the text into long token sequences. Since VLM image encoders map f...
arxiv.org
September 24, 2026 at 10:32 AM
Apple has published a paper & model based on Qwen 3.5 9B for processing large amounts of multi document text, compressed as images, LensVLM 9B:
huggingface.co/bartowski/Le...
github.com/apple-aiml-r...
arxiv.org/abs/2605.07019
huggingface.co/bartowski/Le...
github.com/apple-aiml-r...
arxiv.org/abs/2605.07019
Updated: 6 benchmarks testing spatial and agent reasoning of LLM/VLMs
arxiv.org/abs/2410.06468 does spatial cognition
arxiv.org/abs/2307.06281 MMBench
arxiv.org/abs/2411.13543 BALROG
arxiv.org/abs/2410.07765 GameTraversalBenchmark
3dsrbench.github.io 3DSRBenchmark
open-eqa.github.io Open-EQA
arxiv.org/abs/2410.06468 does spatial cognition
arxiv.org/abs/2307.06281 MMBench
arxiv.org/abs/2411.13543 BALROG
arxiv.org/abs/2410.07765 GameTraversalBenchmark
3dsrbench.github.io 3DSRBenchmark
open-eqa.github.io Open-EQA
November 26, 2024 at 8:25 AM
Updated: 6 benchmarks testing spatial and agent reasoning of LLM/VLMs
arxiv.org/abs/2410.06468 does spatial cognition
arxiv.org/abs/2307.06281 MMBench
arxiv.org/abs/2411.13543 BALROG
arxiv.org/abs/2410.07765 GameTraversalBenchmark
3dsrbench.github.io 3DSRBenchmark
open-eqa.github.io Open-EQA
arxiv.org/abs/2410.06468 does spatial cognition
arxiv.org/abs/2307.06281 MMBench
arxiv.org/abs/2411.13543 BALROG
arxiv.org/abs/2410.07765 GameTraversalBenchmark
3dsrbench.github.io 3DSRBenchmark
open-eqa.github.io Open-EQA
That's more a task for VLMs than LLMs, right ?
September 19, 2026 at 4:14 AM
That's more a task for VLMs than LLMs, right ?
Still wrapping my head around the fact that a consumer laptop can now run 90B VLMs without batting an eye (best spent grant funding so far)
November 20, 2024 at 9:46 AM
Still wrapping my head around the fact that a consumer laptop can now run 90B VLMs without batting an eye (best spent grant funding so far)
Roost uses LLMs.
LLMs/VLMs are the currently the state of the art for every content moderation use case.
LLMs/VLMs are the currently the state of the art for every content moderation use case.
February 11, 2025 at 4:50 AM
Roost uses LLMs.
LLMs/VLMs are the currently the state of the art for every content moderation use case.
LLMs/VLMs are the currently the state of the art for every content moderation use case.
boosters and VCs are idiots who think it will replace all labour - and on the other side you have people acting like vision transformers or VLMs don't do anything
Its frustrating because I think people don't understand what AI can and cannot do very well right now because its new technology, on the other hand, the number of people who want to actually try to answer this question are a shockingly small number of people
December 21, 2024 at 3:08 PM
boosters and VCs are idiots who think it will replace all labour - and on the other side you have people acting like vision transformers or VLMs don't do anything
It's LLMs that are providing the accessibility tools though (Well, VLMs, but those wouldn't exist without LLMs)
May 19, 2026 at 4:59 AM
It's LLMs that are providing the accessibility tools though (Well, VLMs, but those wouldn't exist without LLMs)
llama.cpp has vision language model support now! ❤️🔥
get started with sota VLMs (gemma 3, Qwen2.5VL, InternVL3 & more) and serve them wherever you want 🤩
learn more github.com/ggml-org/lla... 📖
get started with sota VLMs (gemma 3, Qwen2.5VL, InternVL3 & more) and serve them wherever you want 🤩
learn more github.com/ggml-org/lla... 📖
May 11, 2025 at 7:46 AM
llama.cpp has vision language model support now! ❤️🔥
get started with sota VLMs (gemma 3, Qwen2.5VL, InternVL3 & more) and serve them wherever you want 🤩
learn more github.com/ggml-org/lla... 📖
get started with sota VLMs (gemma 3, Qwen2.5VL, InternVL3 & more) and serve them wherever you want 🤩
learn more github.com/ggml-org/lla... 📖
🤔When combining Vision-language models (VLMs) with Large language models (LLMs), do VLMs benefit from additional genuine semantics or artificial augmentations of the text for downstream tasks?
🤨Interested? Check out our latest work at #AAAI25:
💻Code and 📝Paper at: github.com/CompVis/DisCLIP
🧵👇
🤨Interested? Check out our latest work at #AAAI25:
💻Code and 📝Paper at: github.com/CompVis/DisCLIP
🧵👇
January 8, 2025 at 3:54 PM
🤔When combining Vision-language models (VLMs) with Large language models (LLMs), do VLMs benefit from additional genuine semantics or artificial augmentations of the text for downstream tasks?
🤨Interested? Check out our latest work at #AAAI25:
💻Code and 📝Paper at: github.com/CompVis/DisCLIP
🧵👇
🤨Interested? Check out our latest work at #AAAI25:
💻Code and 📝Paper at: github.com/CompVis/DisCLIP
🧵👇
📣 New paper alert! We released a new safety benchmark for VLMs with a core focus on test cases that become unsafe by combining text and images.
TL;DR: many modern VLMs are unsafe across various types of queries and languages.
arxiv.org/abs/2501.10057
huggingface.co/datasets/fel...
TL;DR: many modern VLMs are unsafe across various types of queries and languages.
arxiv.org/abs/2501.10057
huggingface.co/datasets/fel...
Today, we are releasing MSTS, a new Multimodal Safety Test Suite for vision-language models!
MSTS is exciting because it tests for safety risks *created by multimodality*. Each prompt consists of a text + image that *only in combination* reveal their full unsafe meaning.
🧵
MSTS is exciting because it tests for safety risks *created by multimodality*. Each prompt consists of a text + image that *only in combination* reveal their full unsafe meaning.
🧵
January 22, 2025 at 1:43 PM
📣 New paper alert! We released a new safety benchmark for VLMs with a core focus on test cases that become unsafe by combining text and images.
TL;DR: many modern VLMs are unsafe across various types of queries and languages.
arxiv.org/abs/2501.10057
huggingface.co/datasets/fel...
TL;DR: many modern VLMs are unsafe across various types of queries and languages.
arxiv.org/abs/2501.10057
huggingface.co/datasets/fel...
I will present ✌️ BDU workshop papers @ NeurIPS: one by Rui Li (looking for internships) and one by Anton Baumann.
🔗 to extended versions:
1. 🙋 "How can we make predictions in BDL efficiently?" 👉 arxiv.org/abs/2411.18425
2. 🙋 "How can we do prob. active learning in VLMs" 👉 arxiv.org/abs/2412.06014
🔗 to extended versions:
1. 🙋 "How can we make predictions in BDL efficiently?" 👉 arxiv.org/abs/2411.18425
2. 🙋 "How can we do prob. active learning in VLMs" 👉 arxiv.org/abs/2412.06014
December 10, 2024 at 3:18 PM
I will present ✌️ BDU workshop papers @ NeurIPS: one by Rui Li (looking for internships) and one by Anton Baumann.
🔗 to extended versions:
1. 🙋 "How can we make predictions in BDL efficiently?" 👉 arxiv.org/abs/2411.18425
2. 🙋 "How can we do prob. active learning in VLMs" 👉 arxiv.org/abs/2412.06014
🔗 to extended versions:
1. 🙋 "How can we make predictions in BDL efficiently?" 👉 arxiv.org/abs/2411.18425
2. 🙋 "How can we do prob. active learning in VLMs" 👉 arxiv.org/abs/2412.06014
Got a digitised collection that needs OCR? uv-scripts is a set of single-file Python scripts that OCR a whole image dataset to markdown in one command — 20+ open VLMs to pick from, nothing to install but uv.
github.com/davanstrien/...
github.com/davanstrien/...
June 10, 2026 at 6:25 AM
Got a digitised collection that needs OCR? uv-scripts is a set of single-file Python scripts that OCR a whole image dataset to markdown in one command — 20+ open VLMs to pick from, nothing to install but uv.
github.com/davanstrien/...
github.com/davanstrien/...
The VLMs duty to preserve land for wildlife is a lifelong promise. Be part of this enduring mission #PreserveHabitat #Vegan @VLandMovement veganlandmovement.co...
December 15, 2024 at 7:57 PM
The VLMs duty to preserve land for wildlife is a lifelong promise. Be part of this enduring mission #PreserveHabitat #Vegan @VLandMovement veganlandmovement.co...
Introducing the smollest VLMs yet! 🤏
SmolVLM (256M & 500M) runs on <1GB GPU memory.
Fine-tune it on your laptop and run it on your toaster. 🚀
Even the 256M model outperforms our Idefics 80B (Aug '23).
How small can we go? 👀
SmolVLM (256M & 500M) runs on <1GB GPU memory.
Fine-tune it on your laptop and run it on your toaster. 🚀
Even the 256M model outperforms our Idefics 80B (Aug '23).
How small can we go? 👀
January 23, 2025 at 1:33 PM
Introducing the smollest VLMs yet! 🤏
SmolVLM (256M & 500M) runs on <1GB GPU memory.
Fine-tune it on your laptop and run it on your toaster. 🚀
Even the 256M model outperforms our Idefics 80B (Aug '23).
How small can we go? 👀
SmolVLM (256M & 500M) runs on <1GB GPU memory.
Fine-tune it on your laptop and run it on your toaster. 🚀
Even the 256M model outperforms our Idefics 80B (Aug '23).
How small can we go? 👀
🚨 🚨 Excited to share our latest paper, now on #arXiv!
🖼️ We studied how unified VLMs, trained to generate both text and images (e.g., Meta's Chameleon), exchange information between modalities, comparing them to standard VLMs.
📄 Paper: arxiv.org/abs/2412.06646
Deep dive: 👇
🖼️ We studied how unified VLMs, trained to generate both text and images (e.g., Meta's Chameleon), exchange information between modalities, comparing them to standard VLMs.
📄 Paper: arxiv.org/abs/2412.06646
Deep dive: 👇
December 10, 2024 at 8:11 PM
🚨 🚨 Excited to share our latest paper, now on #arXiv!
🖼️ We studied how unified VLMs, trained to generate both text and images (e.g., Meta's Chameleon), exchange information between modalities, comparing them to standard VLMs.
📄 Paper: arxiv.org/abs/2412.06646
Deep dive: 👇
🖼️ We studied how unified VLMs, trained to generate both text and images (e.g., Meta's Chameleon), exchange information between modalities, comparing them to standard VLMs.
📄 Paper: arxiv.org/abs/2412.06646
Deep dive: 👇