Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing

Summary

This paper investigates how well large language models, which are good at answering questions based on knowledge graphs, can actually suggest *unexpected* but useful answers, not just the most obvious ones.

What's the problem?

Current systems using large language models for knowledge graph question answering are really good at finding answers that make sense and are directly related to the question, but they don't excel at uncovering surprising or novel connections. Essentially, they lack the ability to suggest answers that are insightful and weren't already expected. There wasn't a good way to measure how 'serendipitous' an answer was – how much it offered a surprising, yet valuable, insight.

What's the solution?

The researchers created a framework called SerenQA to specifically test this ability. They defined what 'serendipity' means in this context, looking at how relevant, new, and surprising an answer is. They also built a dataset based on medical knowledge, specifically focusing on finding new uses for existing drugs. SerenQA tests models in stages: first finding relevant information, then reasoning about connections within that information, and finally, identifying truly surprising discoveries. They then tested current large language models using this framework.

Why it matters?

This work is important because it highlights a weakness in current AI systems. While they can access and process a lot of information, they aren't very good at creative problem-solving or making unexpected connections. Improving this 'serendipity' could lead to breakthroughs in fields like medicine, where finding new uses for old drugs could save lives and resources. It also provides a benchmark and tools for other researchers to improve these models.