rank based correlation

[nicolaleo]

I believe that the Pearson coefficient does not adequately represent the correlation between the two evaluation methods. In fact, observing the scatter chart it is clear that there are strong discrepancies both when gpt-4 assigns a score of 0 and when it assigns a score of 100 (llama3 covers a much wider range in both cases). Probably a correlation based on the concept of rank would better describe the relation.

import scipy
pearson=scipy.stats.pearsonr(list1, list2)
spearman=scipy.stats.spearmanr(list1, list2)
kendall=scipy.stats.kendalltau(list1, list2) # Kendall's tau
pearson,spearman,kendall

(PearsonRResult(statistic=0.8048901206421856, pvalue=6.122429296730889e-24),
SignificanceResult(statistic=0.6984056005946799, pvalue=6.613506858448206e-16),
SignificanceResult(statistic=0.5729204292295474, pvalue=1.497691422488142e-12))

[rasbt]

Thanks for the feedback, and I agree, it's probably a good idea to add rank-based correlation coefficients. The reason I used Pearson was because I wanted to compare it with Prometheus 2 (for my own curiosity), which is one of the main methods trained for rating LLM responses:

• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

But sure, I will add those! Thanks!

[rasbt]

I updated it now here: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/03_model-evaluation/scores/correlation-analysis.ipynb