Researchers propose an LLM-as-a-judge framework for evaluating math reasoning that beats rule-based symbolic comparison, fixing failures in Lighteval and SimpleRL. This enables ...
https://gentic.news/article/llm-as-a-judge-framework-fixes
Researchers propose an LLM-as-a-judge framework for evaluating math reasoning that beats rule-based symbolic comparison, fixing failures in Lighteval and SimpleRL. This enables ...
https://gentic.news/article/llm-as-a-judge-framework-fixes
* any dataset on the 🤗 Hub can become an eval task in a few lines of code: customize the prompt, metrics, parsing, few-shots, everything!
* model- and data-parallel inference
* auto batching with the new vLLM backend
* any dataset on the 🤗 Hub can become an eval task in a few lines of code: customize the prompt, metrics, parsing, few-shots, everything!
* model- and data-parallel inference
* auto batching with the new vLLM backend
🤯 If you've ever wondered how to evaluate an LLM for you problem, this is the material for you.
🤯 If you've ever wondered how to evaluate an LLM for you problem, this is the material for you.
colab.research.google.com/drive/1iQnf_...
colab.research.google.com/drive/1iQnf_...
🧵>>
🧵>>
1. Create a dataset using Argilla's UI
2. Export the dataset to the hub
3. Train a language model using the AutoTrain UI
4. Evaluate the model using lighteval
https://buff.ly/4fVcMtU
1. Create a dataset using Argilla's UI
2. Export the dataset to the hub
3. Train a language model using the AutoTrain UI
4. Evaluate the model using lighteval
https://buff.ly/4fVcMtU
Definitely will give Lighteval a try! I'm sure it will do great as we use a ton of HF packages in our org already.
Definitely will give Lighteval a try! I'm sure it will do great as we use a ton of HF packages in our org already.
We merged a huge refacto of lighteval making easier to add:
🔄 Multiturn tasks
🖼️ Multimodal tasks
📝 Plus unified logs for thorough benchmark analysis
Benchmarks guys, what evals would you like to see added ?
We merged a huge refacto of lighteval making easier to add:
🔄 Multiturn tasks
🖼️ Multimodal tasks
📝 Plus unified logs for thorough benchmark analysis
Benchmarks guys, what evals would you like to see added ?
@JoelNiklaus @_lewtun @ailozovskaya @clefourrier @alvind319 HERIUN @_EldarKurtic @mariagrandury jnanliu @qubvelx
Check out the release & try it out:
🔗 github.com/huggingface...
@JoelNiklaus @_lewtun @ailozovskaya @clefourrier @alvind319 HERIUN @_EldarKurtic @mariagrandury jnanliu @qubvelx
Check out the release & try it out:
🔗 github.com/huggingface...
Bring any model to lighteval, your backend, your rules.
Use Lighteval to benchmark it like any other supported model.
This makes evals reproducible & comparable on your backends 💥
Bring any model to lighteval, your backend, your rules.
Use Lighteval to benchmark it like any other supported model.
This makes evals reproducible & comparable on your backends 💥
Built by Hugging Face's OpenEvals team, it lets you:
Run standardized benchmarks
Compare models and backends side-by-side
Scale across backends (vLLM, HF Hub, litellm, nanotron, sglang, transformers, etc.)
Built by Hugging Face's OpenEvals team, it lets you:
Run standardized benchmarks
Compare models and backends side-by-side
Scale across backends (vLLM, HF Hub, litellm, nanotron, sglang, transformers, etc.)
Now with:
✅ Plug & play custom model inference (evaluate any backend)
📈 Tasks like AIME, GPQA:diamond, SimpleQA, and hundreds more
Details below 🧵👇
Now with:
✅ Plug & play custom model inference (evaluate any backend)
📈 Tasks like AIME, GPQA:diamond, SimpleQA, and hundreds more
Details below 🧵👇
lighteval aims to gather all benchmarks scattered across repos to make them simple to use with hundreds of popular benchmarks and many backends available for your models
github.com/huggingface...
3/N
lighteval aims to gather all benchmarks scattered across repos to make them simple to use with hundreds of popular benchmarks and many backends available for your models
github.com/huggingface...
3/N
i've been using @huggingface's lighteval and inference providers and litellm to evaluate all those models in less than a few hours 🤩
1/N
i've been using @huggingface's lighteval and inference providers and litellm to evaluate all those models in less than a few hours 🤩
1/N