Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Martin/sandbox #23

Open
wants to merge 47 commits into
base: main
Choose a base branch
from
Open

Martin/sandbox #23

wants to merge 47 commits into from

Conversation

MartinRiddell
Copy link
Collaborator

  • Added Spider and Squall annotations.
  • Added jupyter notebook for result processing.

niansong1996 and others added 30 commits October 10, 2022 18:42
@niansong1996
add annotation tool
44f16ee
@niansong1996
10 annotations for squall for stephen
bde0474
new ui enhancements and testing
87c8385
@StephenYin01
fixed bug where cancel still saved result, fixed bug where exec_info …
b4ebe45 
…was not being saved in jsonl, new UI color and visual enhancements
@StephenYin01
fixed annotation missing an annotation
5451568
@StephenYin01
new annotations
653af0b
@StephenYin01
new annotations again
63c2ed6
@MartinRiddell
50 squall instances
93074da
@StephenYin01
finished 50 examples
f54d0c9
@niansong1996
fix bug in initing spider executor
56a7664
@niansong1996
fix bug in executor
a0d22b6
@niansong1996
Merge pull request #16 from Yale-LILY/martin/squall_annotation
b09c70a 
50 squall instances
@niansong1996
Merge pull request #15 from Yale-LILY/stephen/annotation-dev
5e6b374 
Stephen/annotation dev
@StephenYin01
new annotations for spider (need to revise some with new execution); …
141e214 
…debugged spider execution capitalization issues and exec match issues with numpy int not being detected
@StephenYin01
fixed annotation examples
7657ae2
@niansong1996
Merge pull request #17 from Yale-LILY/stephen/spider-annotation-dev
3bfdc9f 
Spider Execution bug fixes; Added new annotations for spider (some need to be revised with spider execution change)
@StephenYin01
new examples; fixed spider execution criterion bug
834dfd7
@StephenYin01
debug spider_execution_py if table name is python reserved keyword, m…
0457619 
…ade fix for spider_answer_eq to compare sorted answers if no ORDER because sql order is implementation dependent, finished 50 examples
@niansong1996
Merge pull request #18 from Yale-LILY/stephen/spider-annotation-dev
400b98f 
Stephen/spider annotation dev
@MartinRiddell
first few annotations for spider
1c35e33
@MartinRiddell
Finished 50 annotations for spider dataset
05f3437
@MartinRiddell
100 annotations by chatgpt on successful and unsuccessful spider data…
4aee7bd 
…set questions.
@MartinRiddell
100 annotations by chatgpt on unsuccessful spider dataset
2780b01
@MartinRiddell
added dataset of codex's failures on spider
faeef72
@MartinRiddell
creating a new branch for this notebook
c912865
@MartinRiddell
playing with result processing
6702aff
@MartinRiddell
cleaned up a bit
c3fe0d3
@MartinRiddell
implemented rudimentary human evaluation for model's results on squal…
a98e6f8 
…l and spider datasets
@MartinRiddell
added doc string and a few comments in the code
5ca8911
@MartinRiddell
start_eval can now handle datasets with the same keys as the gsmath d…
e10f6ab 
…ataset
MartinRiddell and others added 17 commits April 14, 2023 17:44
@MartinRiddell
commented out print statement, and added a check for ERROR messages i…
cbecc01 
…n the code's output
@MartinRiddell
better error handling
cad6fff
@MartinRiddell
added 'big difference' and 'unclear instructions' to the reasons for …
df59973 
…failure
@MartinRiddell
better error handling. Forgot to add this in previous commit
29c7552
@MartinRiddell
began human evaluation of codex on gsmath dataset
7576f80
@MartinRiddell
added an extra error to human eval files, and finished evaluating 100…
5270c76 
… questions
@MartinRiddell
small changes to evaluation script, and eval_report can be used to su…
ac8a222 
…mmarize evaluations
@MartinRiddell
finished evaluating 100 mbpp and gsmath, 50 wtq problems answered by …
2a76777 
…codex model
@MartinRiddell
added rekey script. It's not great, but it gives the indices of the q…
e82a2fc 
…uestions in order
@StephenYin01
stephen codex cushman on gsm8k evals
17b4556
@StephenYin01
finished gpt4 evaluation on GSM8k
d924d73
@StephenYin01
finished human eval of starcoder on gsm8k
577d275
@MartinRiddell
added a few evaluations that are done. Spider_gpt4 isn't done, but pu…
8970746 
…shing it with dummy answers for stephen to rekey on
@MartinRiddell
resolving merge
77cc573
@StephenYin01
finish spider starcoder evals (2 shot)
6870fcb
@StephenYin01
finished codex cushman eval on spider
17a28c1
@MartinRiddell
added evals for gpt4 and davinci models on spider dataset
afcc894
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@MartinRiddell @niansong1996 @StephenYin01