Combining Information Retrieval and Large Language Models for a Chatbot that Generates Reliable, Natural-style Answers
Abstract
Chatbots efficiently support users in finding relevant answers in a dialog. Traditional chatbot are mostly based on a set of rules, mapping user questions to predefined answers using a knowledge base. Users often miss an adequate adaptation of the answers to the individual needs and the concrete question. Rule-based chats are often perceived as artificial and wooden'. Large language models trained on huge collections of texts and dialog datasets promise to provide natural answers for a broad spectrum of questions. These models perform well in small talk and questions about popular general knowledge. The systems often fail in cases of domain-specific questions providing incorrect, but plausibly sounding answers. In this work we develop and evaluate a chatbot prototype for the administration of a major German city. We combine a traditional knowledge base with different Large Language Models. We evaluate the system with respect to the fluency and the correctness of the answers as well as on the resource demands and response time of the models. The evaluation of the system shows that existing Large Language Models are able to generate well-understandable answers matching the user's questions. Besides technical issues such as high resource consumption and limited scalability, adequate prompts are crucial to force the model to use reliable data from a trusted knowledge base for generating the answers to avoid both hallucinations and formulations that are not well suited for the concrete context.