Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BFCL evaluation result reproducing details #80

Open
kennychen5411 opened this issue Jul 31, 2024 · 0 comments
Open

BFCL evaluation result reproducing details #80

kennychen5411 opened this issue Jul 31, 2024 · 0 comments

Comments

@kennychen5411
Copy link

Hi, I found this section:
https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/eval_details.md#berkeley-function-calling-leaderboard-bfcl

Benchmark results were achieved by running the open source evaluation repository ShishirPatil/gorilla on commit 7bef000 without any further changes.

I checked out the gorilla repo at this commit id, only found these items in /model_handler/handler_map.py:
"meta-llama/Meta-Llama-3-8B-Instruct": LlamaHandler,
"meta-llama/Meta-Llama-3-70B-Instruct": LlamaHandler,

in /eval_checker/eval_runner_helper.py
"meta-llama/Meta-Llama-3-8B-Instruct": [
"Meta-Llama-3-8B-Instruct (Prompt)",
"https://llama.meta.com/llama3",
"Meta",
"Meta Llama 3 Community",
],
"meta-llama/Meta-Llama-3-70B-Instruct": [
"Meta-Llama-3-70B-Instruct (Prompt)",
"https://llama.meta.com/llama3",
"Meta",
"Meta Llama 3 Community",
],

Did you assign model_id as llama3 instead add a new model_id for llama3.1 ?
And thus the model handler using the LlamaHander?

@kennychen5411 kennychen5411 changed the title BFCL evalation result reproducing details BFCL evaluation result reproducing details Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant