Skip to content

Issues: huggingface/lighteval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

90 Open 125 Closed
90 Open 125 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[CORE] Make parsing of model args and generation parameters robust prio
#579 opened Feb 21, 2025 by lewtun
[EVAL] Add TUMLU benchmark new task
#577 opened Feb 19, 2025 by gaydmi
5
[FT] Propagate batch size control for vLLM backend feature request New feature/request
#573 opened Feb 18, 2025 by alvin319
1
Try to compute Thai finetasks score of llama 3.2 1B
#569 opened Feb 17, 2025 by Nevermetyou65
[BUG] vllm seed is always 1234 bug Something isn't working
#568 opened Feb 16, 2025 by wzunknown
[FT] LiteLLM concurrency parameters hard-coded feature request New feature/request
#567 opened Feb 16, 2025 by lhl
[BUG] LiteLLM backend fails with ifeval since prompt passes is str not List bug Something isn't working
#565 opened Feb 16, 2025 by lhl
1
[BUG] error setting tokenizer with custom generation params for vllm bug Something isn't working
#563 opened Feb 14, 2025 by rawsh
1
Issues with torch version other than 2.2.2
#562 opened Feb 14, 2025 by Nevermetyou65
Weird cuda OutOfMemoryError error
#561 opened Feb 14, 2025 by Nevermetyou65
Lighteval endpoint openai feature request New feature/request
#560 opened Feb 13, 2025 by zhaohan-alan
[BUG] Use sampling for greedy mode bug Something isn't working
#559 opened Feb 13, 2025 by tonysy
Handle errors during sympy conversion
#557 opened Feb 13, 2025 by hynky1999
@hynky1999
[BUG] Nanotron runner imports non-existant bug Something isn't working
#555 opened Feb 13, 2025 by jquesnelle
2
how can i use this "community|alghafa:meta_ar_dialects " as a task
#554 opened Feb 12, 2025 by pratim808
[BUG] tiktoken is not optional dependency bug Something isn't working
#546 opened Feb 8, 2025 by hynky1999
couldn't find it in the cached files and it looks like Elron/bleurt-tiny-512, how to set the model path?
#545 opened Feb 8, 2025 by bannima
1
More flexibility in parameters for OpenAI / LiteLLM feature request New feature/request
#544 opened Feb 7, 2025 by satpalsr
3
Better Docs feature request New feature/request
#543 opened Feb 7, 2025 by satpalsr
1
[EVAL] new task
#542 opened Feb 7, 2025 by suraj169Pi
[FT] Faster generation with TransformersModel by using less padding feature request New feature/request
#531 opened Feb 3, 2025 by rolshoven
Question about Request Count and Generative Sampling
#528 opened Jan 31, 2025 by rolshoven
3
updated pypi package with torch>=2.0,<3.0
#526 opened Jan 30, 2025 by zia1138
lighteval with llama3.2 [RuntimeError: No executable batch size found, reached zero.]
#525 opened Jan 29, 2025 by Nevermetyou65
4
[BUG] ImportError: cannot import name 'ExprExtractionConfig' from 'lighteval.metrics.dynamic_metrics' bug Something isn't working
#523 opened Jan 28, 2025 by hlzhang109
9
ProTip! Mix and match filters to narrow down what you’re looking for.