Questions About the Reported Numbers in Mutox

[ASMIftekhar]

Hello @avidale,
Thanks a lot for releasing the audio toxicity benchmark. In the paper you reported performances from Mutox and ASR-Mutox. I am trying to understand their configurations. Based on my understanding Mutox only takes audio files as input and ASR-Mutox takes the whishper generated text transcripts as input, is this understanding correct?

[avidale]

Hi ASMIftekhar!

Based on my understanding Mutox only takes audio files as input and ASR-Mutox takes the whishper generated text transcripts as input

Yes, this understanding is correct.

[ASMIftekhar]

Thanks for responding quickly. Can you please point to the source code you used for evaluating the performance? In specific, how did you calculate recall at a particular precision? Did you try different threshold values and select one that gives best precision and use that threshold to calculate recall?

[avidale]

We didn't release the source code for the evaluation.
But, if I get the idea right (@mfcoria please correct me if not), we computed all possible precision-recall pairs, (e.g. using precision_recall_curve) and reported the highest recall value that corresponds to a precision no lower than max(0.3, precision of ETOX on this data part).

[mfcoria]

@avidale Yes, that's exactly how we benchmarked it.

Note that most languages are using a 0.3 precision threshold, with only spa, eng and deu working with 0.4~.

[ASMIftekhar]

Thanks a lot for the clarification, it was helpful. I can see you have provided public links to the data, unfortunately many of the links are not available anymore. Is it possible to share the extracted test/devtest data in some other ways? It would be extremely valuable to the community for creating a robust benchmark.

[avidale]

@ASMIftekhar unfortunately, it is very difficult for us to publish speech data.
However, if you extracted the audios from the links that are still alive and published them elsewhere (e.g. as a Huggingface dataset), we would be grateful for such a contribution and would include a reference to your dataset in this repo.

[ASMIftekhar]

Thanks, will try to do that.

[ASMIftekhar]

Sorry for reopenning the issue but the transcripts provided in the tsv file, are they human annotated transcripts? @avidale