VLLMs don't "see", they remember (and they are pretty stubborn about their memories too!). I had the pleasure to supervise two master students at @itu.dk that decided to investigate this fascinating problem of VLLMs. Ayat Khudoir and Rabia Sufian just presented their work at the FAILED workshop

I wonder who's working on VLLMs.

It's the next obvious thing, right? LLMs are working out like gangbusters. Throw some zeros on the end of some hyperparameters. Start talking VLLMs. That could be super interesting.

<cat_reading_newspaper>
I should spend more time on arXiv.
</cat_reading_newspaper>

Honestly i do think robots and VLLMs could make things like electronics recycling maybe less hacky?

vision is too weak in vLLMs

I'm pretty happy to present our work on using VLLMs to achieve semantic clustering at #ica25. (Centennial A at 9.00 if you are curious ;-))

Are Vision Language Models ready for scientific research?
🧑‍🔬🧪

We compared leading VLLMs on the three pillars of chemical and material science discovery: data extraction, lab experimentation and data interpretation.
arxiv.org/abs/2411.16955

really important to keep in mind: VLLMs don't see.

It is kind of a shame that Claude Code has made making data entry portals so easy, given that LLMs and VLLMs, etc are killing the need for human RAs to punch in data for me in the first place

Clear illustration of the limitations of VLLMs. We still have a long way to go (most likely going beyond current training and model paradigms) before "solving vision".

New research shows how Vision-Language Models (VLLMs) represent image concepts within hidden layers, discovering distinct features that enhance multimodal learning. Sparse Autoencoders reveal a shared representation of images and text evolving deeper in the model. https://arxiv.org/abs/2506.04706

Two PolarVis presentations this week: (1) on the (mis)uses of VLLMs for analysing climate change visuals, today at the CCVision Network Meeting, and (2) on methods for longitudinal coordination detection applied to PolarVis data, tomorrow at SunBelt. @lrossi.bsky.social @matmagnani.bsky.social

Fine-Tuning vLLMs for Document Understanding | Towards Data Science in @towardsdatascience.com

towardsdatascience.com/vllm-fine-tu...

#computationalsocialscience Our work on using VLLMs for semantic visual clustering is out on SSCR: journals.sagepub.com/doi/10.1177/...

There is a lot happening in the space of computational analysis of visual content as we acknowledge more and more the visual nature of contemporary communication.

deep learning methods can totally read images but not generalist vLLMs and they almost certainly never will, just because there's no real reason to train them to

This iteration of AI tech (LLMs and VLLMs) has as much chance of becoming sentient as I have of actually becoming a border collie. Any suggestion otherwise is idiocy.

A future tech might but the roadblocks aren't learning or memory, they're the inability to experience reality.

That's decades away.