raw_results

Raw Experiment Results

This folder includes all the raw experiment results.

General Evaluation

Statistics of the local KGs generated from different dataset using different models. Statistics includes:

• tokens: length of the original paper
• entities: entity count in the output kg
• mentions: mention count in the output kg
• relations_total: relation count in the output kg
• relations_normal: general relation count in the output kg
• relations_typing: taxonomic relation count in the output kg
• isolated_entities: number of entities that do no link to other entities, excluding the taxonomic relations.
• isolated_entities_typing: number of entities that do no link to other entities

./gen_ASKG_g.csv: the Five-Paper dataset, GPT

./gen_ASKG_l.csv: the Five-Paper dataset, LLaMA

./gen_SciERC_g.csv: the SciERC dataset, GPT

./gen_SciERC_l.csv: the SciERC dataset, LLaMA

Reverse Engineering Test

Similarity scores between the original documents and the synthesised documents. The synthesised documents are created from the KGs generated from different dataset using different LLMs

./re_ASKG_g.json: the Five-Paper dataset, GPT

./re_ASKG_l.json: the Five-Paper dataset, LLaMA

./re_SciERC_g.json: the SciERC dataset, GPT

./re_SciERC_l.json: the SciERC dataset, LLaMA

RAG Test

Similarity scores between the expected answer and the actual answers. Five papers are used in this test, each with ten question, and therefore, a total of 5 * 10 = 50 questions.

The 50 questions are answered in three way:

• Use the local KGs generated by GPT to answer questions through graph-based RAG.
• Use the local KGs generated by LLaMA to answer questions through graph-based RAG.
• Use the input DDM of the pipeline to answer questions through normal RAG, i.e., directly retrieve sentences similar to the query as the supplementary resource for answer the query.

./rag_LKG_g.json: using KGs generated by GPT

./rag_LKG_l.json: using KGs generated by LLaMA

./rag_DDM_l.json: using DDM