Lukas Edman
@lukasnlp.bsky.social
Post-Doc at TU Munich, doing cool work and that's all you need to know
Great news! Our workshop what-llms-can-not-do.org is going to COLING 2027 in Macau! Hope to see you there!
What LLMs Can(not) Do
A living survey of benchmarks that compare large language models with humans.
what-llms-can-not-do.org
October 3, 2026 at 3:11 PM
Great news! Our workshop what-llms-can-not-do.org is going to COLING 2027 in Macau! Hope to see you there!
Ever feel like it's too hard to keep track of what LLMs cannot do as well as humans? We're making your life easier over at: what-llms-can-not-do.github.io
We're compiling a list of papers testing the abilities of LLMs against humans. Check it out! And you can help contribute too!
We're compiling a list of papers testing the abilities of LLMs against humans. Check it out! And you can help contribute too!
What LLMs Can(not) Do
A living survey of benchmarks that compare large language models with humans.
what-llms-can-not-do.github.io
September 17, 2026 at 12:24 PM
Ever feel like it's too hard to keep track of what LLMs cannot do as well as humans? We're making your life easier over at: what-llms-can-not-do.github.io
We're compiling a list of papers testing the abilities of LLMs against humans. Check it out! And you can help contribute too!
We're compiling a list of papers testing the abilities of LLMs against humans. Check it out! And you can help contribute too!
Happy to announce two papers at #ACL2025!
1. We made an extension to our CUTE benchmark, EXECUTE! We find that LLMs actually *do* have the ability to do character-level manipulation, but language bias, and tokenization too, get in the way. openreview.net/forum?id=m95...
1/2
1. We made an extension to our CUTE benchmark, EXECUTE! We find that LLMs actually *do* have the ability to do character-level manipulation, but language bias, and tokenization too, get in the way. openreview.net/forum?id=m95...
1/2
EXECUTE: A Multilingual Benchmark for LLM Token Understanding
The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified...
openreview.net
May 16, 2025 at 9:49 PM
Happy to announce two papers at #ACL2025!
1. We made an extension to our CUTE benchmark, EXECUTE! We find that LLMs actually *do* have the ability to do character-level manipulation, but language bias, and tokenization too, get in the way. openreview.net/forum?id=m95...
1/2
1. We made an extension to our CUTE benchmark, EXECUTE! We find that LLMs actually *do* have the ability to do character-level manipulation, but language bias, and tokenization too, get in the way. openreview.net/forum?id=m95...
1/2