PUBLICATION
Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering
Type
Conference Paper
Year
2023
Authors
Lars-Peter Meyer
Johannes Frey
Kurt Junghanns
Felix Brei
Kirill Bulert
Michael Martin
Research Area
Event
SEMANTICS 2023 EU: 19th International Conference on Semantic Systems
Published in
SEMANTICS 2023 poster track proceedings
Download
Abstract
As the field of Large Language Models (LLMs) evolves at an accelerated pace, the critical need to assess and monitor their performance emerges. We introduce a benchmarking framework focused on knowledge graph engineering (KGE) accompanied by three challenges addressing syntax and error correction, facts extraction and dataset generation. We show that while being a useful tool, LLMs are yet unfit to assist in knowledge graph generation with zero-shot prompting. Consequently, our LLM-KG-Bench framework provides automatic evaluation and storage of LLM responses as well as statistical data and visualization tools to support tracking of prompt engineering and model performance.
Reference
Meyer, Lars-Peter; Frey, Johannes; Junghanns, Kurt; Brei, Felix; Bulert, Kirill; Gründer-Fahrer, Sabine; Martin, Michael: Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering. SEMANTICS 2023 poster track proceedings, pp. 5, 2023.
