Security learnings from developing the NCCoE Chatbot

The NIST National Cybersecurity Center of Excellence (NCCoE) has re-issued NIST Internal Report (IR) 8579, Developing the NCCoE Chatbot: Technical and Security Learnings from the Initial Implementation. Originally published in June, the document was revised to improve the document’s demonstration of the enhanced abilities of an RAG-based LLM tool over a generic LLM.

The public comment period for the publication has been extended and will close on September 11, 2025.

The NCCoE identified a potential application for a chatbot to support its mission and developed a secure, internal-use chatbot to assist NCCoE staff with searching and summarizing cybersecurity guidelines tailored to specific audiences or use cases.

The chatbot was built using retrieval-augmented generation (RAG)-based LLM technology. This approach combines techniques from information retrieval and natural language generation, enabling the chatbot to provide more focused, contextually relevant responses by leveraging a repository of cybersecurity knowledge, including previous NCCoE publications. Compared to search engines, LLM-based chatbots provide more contextually relevant and precise responses by understanding the nuances of natural language queries.

This report provides a point-in-time examination of the NCCoE Chatbot, outlining the NCCoE’s approach to developing the tool, as well as the NCCoE’s response to specific security challenges. In addition, this report provides an overview of the chatbot and its supporting technologies so that other organizations might consider the benefits of their use.

Comments welcome by September 11, 2025. If you have any questions, please email the team at [email protected].