Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix inference quality caused by temperature parameter in bls #523

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

activezhao
Copy link

When the prompt and parameters are the same, I use APIs of ensemble and tensorrt_llm_bls, the results are different.

And the result of ensemble is expected.

I analyzed the code of bls and finally found that the inference quality dropped significantly in some scenarios, because the temperature parameters were not given.

What's more, this problem has led to many bad cases in our prod services.

After fixing the temperature problem, the scores of blue and em are close to vllm of fp16, here is the comparative data:
image

Here is the code: name_map

And I have added an issue before.
#520

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant