Itay Itzhak @ COLM 🍁
@itay-itzhak.bsky.social
NLProc, deep learning, and machine learning. Ph.D. student @ Technion and The Hebrew University.
https://itay1itzhak.github.io/
https://itay1itzhak.github.io/
Model preferences are a two-way street between the model’s capability and the user’s perspective.
By bridging the gap between benchmarks and real-world vibe-testing, we can evaluate AI the way humans actually use it.
arxiv.org/abs/2604.14137
technion-cs-nlp.github.io/vibe-testin...
By bridging the gap between benchmarks and real-world vibe-testing, we can evaluate AI the way humans actually use it.
arxiv.org/abs/2604.14137
technion-cs-nlp.github.io/vibe-testin...
From Feelings to Metrics: Understanding and Formalizing How Users VIBE-TEST LLMs
A paper on vibe-testing and personalized LLM evaluation, showing that personalization can change which model users prefer.
technion-cs-nlp.github.io
April 23, 2026 at 11:35 AM
Model preferences are a two-way street between the model’s capability and the user’s perspective.
By bridging the gap between benchmarks and real-world vibe-testing, we can evaluate AI the way humans actually use it.
arxiv.org/abs/2604.14137
technion-cs-nlp.github.io/vibe-testin...
By bridging the gap between benchmarks and real-world vibe-testing, we can evaluate AI the way humans actually use it.
arxiv.org/abs/2604.14137
technion-cs-nlp.github.io/vibe-testin...
Results: standard evaluation mask model quality. 🎭
Top models like GPT-5.1 were initially less preferred by beginners, but their win rate skyrocketed on personalized tasks (e.g., 9% to 94%). This shows how superior models are better for all users when evaluation is *user-aware*. 📈
Top models like GPT-5.1 were initially less preferred by beginners, but their win rate skyrocketed on personalized tasks (e.g., 9% to 94%). This shows how superior models are better for all users when evaluation is *user-aware*. 📈
April 23, 2026 at 11:35 AM
Results: standard evaluation mask model quality. 🎭
Top models like GPT-5.1 were initially less preferred by beginners, but their win rate skyrocketed on personalized tasks (e.g., 9% to 94%). This shows how superior models are better for all users when evaluation is *user-aware*. 📈
Top models like GPT-5.1 were initially less preferred by beginners, but their win rate skyrocketed on personalized tasks (e.g., 9% to 94%). This shows how superior models are better for all users when evaluation is *user-aware*. 📈
Can we scale it? We built a pipeline mirroring vibe-testing structure:
👤 Profile: Turns user descriptions into structured profiles.
✍️ Rewrite: Personalizes prompts for specific contexts.
⚖️ Judge: Compares models using user-defined criteria.
Automation meets personalization. 🤖✨
👤 Profile: Turns user descriptions into structured profiles.
✍️ Rewrite: Personalizes prompts for specific contexts.
⚖️ Judge: Compares models using user-defined criteria.
Automation meets personalization. 🤖✨
April 23, 2026 at 11:35 AM
Can we scale it? We built a pipeline mirroring vibe-testing structure:
👤 Profile: Turns user descriptions into structured profiles.
✍️ Rewrite: Personalizes prompts for specific contexts.
⚖️ Judge: Compares models using user-defined criteria.
Automation meets personalization. 🤖✨
👤 Profile: Turns user descriptions into structured profiles.
✍️ Rewrite: Personalizes prompts for specific contexts.
⚖️ Judge: Compares models using user-defined criteria.
Automation meets personalization. 🤖✨
The anatomy of a vibe-test:
👉 Input: Users adapt task complexity and context to mimic their daily life.
👉 Output: The judge is the user. What counts as "clear" or "good tone" is defined by the individual’s perspective, not a static definition.
👉 Input: Users adapt task complexity and context to mimic their daily life.
👉 Output: The judge is the user. What counts as "clear" or "good tone" is defined by the individual’s perspective, not a static definition.
April 23, 2026 at 11:35 AM
The anatomy of a vibe-test:
👉 Input: Users adapt task complexity and context to mimic their daily life.
👉 Output: The judge is the user. What counts as "clear" or "good tone" is defined by the individual’s perspective, not a static definition.
👉 Input: Users adapt task complexity and context to mimic their daily life.
👉 Output: The judge is the user. What counts as "clear" or "good tone" is defined by the individual’s perspective, not a static definition.
What does "vibe-testing" actually look like? 🕵️♂️
We analyzed public reports of vibe-tests—from YouTube to Reddit—to see how users evaluate models. Our analysis reveals vibe-testing recurring patterns, allowing us to formalize "vibe-testing" as a structured evaluation practice.
We analyzed public reports of vibe-tests—from YouTube to Reddit—to see how users evaluate models. Our analysis reveals vibe-testing recurring patterns, allowing us to formalize "vibe-testing" as a structured evaluation practice.
April 23, 2026 at 11:35 AM
What does "vibe-testing" actually look like? 🕵️♂️
We analyzed public reports of vibe-tests—from YouTube to Reddit—to see how users evaluate models. Our analysis reveals vibe-testing recurring patterns, allowing us to formalize "vibe-testing" as a structured evaluation practice.
We analyzed public reports of vibe-tests—from YouTube to Reddit—to see how users evaluate models. Our analysis reveals vibe-testing recurring patterns, allowing us to formalize "vibe-testing" as a structured evaluation practice.
Why do we "vibe-test" and ignore leaderboards? We ran a survey to find out.
Our findings:
❌ 86% said they’ve used a model that "felt" significantly better (or worse) than its reported scores.
✅ 82% of you are "vibe-testing" models through direct interaction.
Our findings:
❌ 86% said they’ve used a model that "felt" significantly better (or worse) than its reported scores.
✅ 82% of you are "vibe-testing" models through direct interaction.
April 23, 2026 at 11:35 AM
Why do we "vibe-test" and ignore leaderboards? We ran a survey to find out.
Our findings:
❌ 86% said they’ve used a model that "felt" significantly better (or worse) than its reported scores.
✅ 82% of you are "vibe-testing" models through direct interaction.
Our findings:
❌ 86% said they’ve used a model that "felt" significantly better (or worse) than its reported scores.
✅ 82% of you are "vibe-testing" models through direct interaction.
Huge thanks to my co-authors
@boknilev @GabiStanovsky!
Preprint: arxiv.org/abs/2507.07186
Webpage: itay1itzhak.github.io/planted-in-...
We’d love your thoughts, critiques, and ideas 📬
Let’s talk about building more interpretable and trustworthy LLMs!
#NLProc #Bias #CognitiveAI
@boknilev @GabiStanovsky!
Preprint: arxiv.org/abs/2507.07186
Webpage: itay1itzhak.github.io/planted-in-...
We’d love your thoughts, critiques, and ideas 📬
Let’s talk about building more interpretable and trustworthy LLMs!
#NLProc #Bias #CognitiveAI
Planted in Pretraining, Swayed by Finetuning: A Case Study on the...
Large language models (LLMs) exhibit cognitive biases -- systematic tendencies of irrational decision-making, similar to those seen in humans. Prior work has found that these biases vary across...
arxiv.org
July 15, 2025 at 1:38 PM
Huge thanks to my co-authors
@boknilev @GabiStanovsky!
Preprint: arxiv.org/abs/2507.07186
Webpage: itay1itzhak.github.io/planted-in-...
We’d love your thoughts, critiques, and ideas 📬
Let’s talk about building more interpretable and trustworthy LLMs!
#NLProc #Bias #CognitiveAI
@boknilev @GabiStanovsky!
Preprint: arxiv.org/abs/2507.07186
Webpage: itay1itzhak.github.io/planted-in-...
We’d love your thoughts, critiques, and ideas 📬
Let’s talk about building more interpretable and trustworthy LLMs!
#NLProc #Bias #CognitiveAI
🧠 Takeaway:
Cognitive biases are not introduced during instruction tuning.
They’re planted in pretraining and only surfaced by finetuning.
If we want fairer models, we need to look deeper into the pretraining pipeline.
Cognitive biases are not introduced during instruction tuning.
They’re planted in pretraining and only surfaced by finetuning.
If we want fairer models, we need to look deeper into the pretraining pipeline.
July 15, 2025 at 1:38 PM
🧠 Takeaway:
Cognitive biases are not introduced during instruction tuning.
They’re planted in pretraining and only surfaced by finetuning.
If we want fairer models, we need to look deeper into the pretraining pipeline.
Cognitive biases are not introduced during instruction tuning.
They’re planted in pretraining and only surfaced by finetuning.
If we want fairer models, we need to look deeper into the pretraining pipeline.
🔄 Step 2: Cross-tuning.
We swap instruction datasets between models with different pretraining.
Result: Biases follow the pretrained model!
PCA clearly shows models group by pretraining base, not by instruction.
The bias “signature” stays intact, no matter the finetuning!
We swap instruction datasets between models with different pretraining.
Result: Biases follow the pretrained model!
PCA clearly shows models group by pretraining base, not by instruction.
The bias “signature” stays intact, no matter the finetuning!
July 15, 2025 at 1:38 PM
🔄 Step 2: Cross-tuning.
We swap instruction datasets between models with different pretraining.
Result: Biases follow the pretrained model!
PCA clearly shows models group by pretraining base, not by instruction.
The bias “signature” stays intact, no matter the finetuning!
We swap instruction datasets between models with different pretraining.
Result: Biases follow the pretrained model!
PCA clearly shows models group by pretraining base, not by instruction.
The bias “signature” stays intact, no matter the finetuning!
🎲 Step 1: Training randomness.
We finetune the same model 3× with different seeds.
Result: Some variation in bias scores, but behavior patterns stay stable compared to MMLU variance.
✅ Aggregating across seeds reveals consistent trends.
We finetune the same model 3× with different seeds.
Result: Some variation in bias scores, but behavior patterns stay stable compared to MMLU variance.
✅ Aggregating across seeds reveals consistent trends.
July 15, 2025 at 1:38 PM
🎲 Step 1: Training randomness.
We finetune the same model 3× with different seeds.
Result: Some variation in bias scores, but behavior patterns stay stable compared to MMLU variance.
✅ Aggregating across seeds reveals consistent trends.
We finetune the same model 3× with different seeds.
Result: Some variation in bias scores, but behavior patterns stay stable compared to MMLU variance.
✅ Aggregating across seeds reveals consistent trends.
🧪 We introduce a two-step causal framework to disentangle the effects of:
- Pretraining
- Instruction tuning
- Training randomness
- 🍁 Bottom line - pretraining is the origin of bias. Finetuning? Just the messenger
#CausalInference #TrustworthyAI #NLP
- Pretraining
- Instruction tuning
- Training randomness
- 🍁 Bottom line - pretraining is the origin of bias. Finetuning? Just the messenger
#CausalInference #TrustworthyAI #NLP
July 15, 2025 at 1:38 PM
🧪 We introduce a two-step causal framework to disentangle the effects of:
- Pretraining
- Instruction tuning
- Training randomness
- 🍁 Bottom line - pretraining is the origin of bias. Finetuning? Just the messenger
#CausalInference #TrustworthyAI #NLP
- Pretraining
- Instruction tuning
- Training randomness
- 🍁 Bottom line - pretraining is the origin of bias. Finetuning? Just the messenger
#CausalInference #TrustworthyAI #NLP