Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BFCL] Fix API Keys Handling #959

Merged
merged 5 commits into from
Mar 26, 2025

Conversation

catherineruoxiwu
Copy link
Contributor

The following bugs can all be reproduced from a clean, unmodified version of the repository. All of them have been addressed by this PR.

Setup 1:

  • Input: test_case_ids_to_generate.jsonrest: ["rest_0"]
  • .env: No API key provided
  • Command: bfcl generate --model gpt-4o-mini-2024-07-18 --run-ids
  1. fix check_api_key_supplied implementation in utils.py. os.getenv(var) returns None instead of a string

  2. After the previous issue is fixed, a NameError occurs when no result file existed and existing_ids was only conditionally defined inside a loop

    Exception has occurred: NameError  
    free variable 'existing_ids' referenced before assignment in enclosing scope
    
  3. Under this setup, no test should be called, but test case rest_0 was still collected unexpectedly.
    Screenshot 2025-03-25 at 2 17 05 AM

Setup 2:

  • .env: All API keys provided
  • Command: bfcl generate --model gpt-4o-mini-2024-07-18 --test-category rest
  1. API keys were replaced in the rest tests JSON file, but the credential information in the question passed to the generate_result function was not correct.

@HuanzhiMao HuanzhiMao added the BFCL-General General BFCL Issue label Mar 26, 2025
Copy link
Collaborator

@HuanzhiMao HuanzhiMao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HuanzhiMao HuanzhiMao merged commit a48d2bb into ShishirPatil:main Mar 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFCL-General General BFCL Issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants