Accepted to EMNLP Findings! While Large Language Models encode vast amounts of multicultural and factual information, this parametric knowledge is often unevenly accessible across languages. Standard inference frequently fails to surface localized facts, creating persistent gaps in cross-lingual knowledge transfer and consistency. We address this challenge in our paper "Cross-Lingual Exploration for Parametric Knowledge" (https://lnkd.in/drf36bV7), a collaboration between Elisha Diskind, Itamar Trainin, and Omri Abend from The Hebrew University of Jerusalem, Leshem Choshen from the Weizmann Institute of Science, alongside Uri Shaham and myself from our Google Research IL group. We formalize cross-lingual exploration as a structured search process across four core dimensions: language selection, exploration routing, answer aggregation, and inference budget. Evaluating across 17 typologically diverse languages on the ECLeKTic and CLIKE benchmarks, we show that allowing models to autono

Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prior work suggests that forcing the CoT to remain in the input language (\emph{native reasoning}) substantially degrades performance relative to allowing the model to reason in English before answering in the input language (\emph{English-pivoted reasoning}). However, most studies of this native reasoning gap rely on inference-time interventions or limited native-language training data. We revisit this comparison at a larger scale and under comparable supervision. We construct long multilingual reasoning datasets across six languages (English, French, German, Spanish, Chinese and Swahili); fine-tune specialists in both native and English-pivoted regimes on top of \texttt{Qwen/Qwen3-8B-Base}, and evaluate across mathematics, science, general knowledge, and code. In this setting, the average native reasoning gap shrinks to 1.9--3.5\% across the f

Jaavid J, Raj Dabre, Aswanth M, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

@yoavartzi @nthngdy I agree, what I say is that you talk about how much you lose of the gradient you compute, and I say, but if you just compute less, the result would lose less and still be equivalent or worse probably? More steps=more information that's true, but a bit of a separate battle right?

"Small-Scale Experiments: Are We There Yet?" Scaling laws show up even in tiny 4M parameter models, but only when their hyperparameters are tuned well enough. This Meta paper found that small models are much more hyperparameter-sensitive. As scale increases, the hyperparameter

@JumeletJ @magikarp_tokens For instance this paper shows that byte ByT5 (small vocab) outperforms subword mT5 (large vocab) on low resource translation (including Georgian pairs) https://t.co/U4DXWBLaiW

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355

Models do not improve linearly (any more). New models improve more than expected on hard tasks. Now let's think about it a bit more. It means progress is directed towards the hard tasks. It also means the easier tasks improve less than expected. Which alphaxiv.org/abs/2608.00355