Practical Guide to Crafting your first LLM-powered App Using RAG Framework

This blog post was written by Brain John Aboze as part of the Deepchecks Community Blog. If you would like to contribute your own blog post, feel free to reach out to us via We typically pay a symbolic fee for content that's accepted by our reviewers.

Practical Guide to Crafting your first LLM-powered App Using RAG Framework

Large Language Models (LLMs) have taken center stage in the burgeoning landscape of artificial intelligence, captivating the attention of technologists and businesses alike. These models, with their expansive grasp on human language, have set new benchmarks in understanding and generating text, opening up a plethora of opportunities in the realm of application development. However, integrating LLMs into functional applications presents a distinct set of challenges, often requiring a nuanced approach to leverage their capabilities effectively.

This article is designed as a cornerstone for developers and enthusiasts looking to harness the power of LLMs within their applications. The Retrieval-Augmented Generation (RAG) framework stands as a pivotal tool in this endeavor, merging the generative prowess of LLMs with the precision of information retrieval to create applications that are responsive, remarkably informed, and accurate.


Source: Lukas from Pixabay

What is RAG?

RAG represents a significant stride in natural language processing (NLP), an architectural innovation that breathes new intelligence into LLMs. Conceptualized in the Facebook
2020 paper, RAG is a hybrid model that ingeniously merges the deep, pre-trained knowledge of LLMs with a search engine’s dynamic, pinpoint accuracy. This dual-memory system empowers the model to intelligently access and utilize a vast expanse of information, much like combining the wisdom of a learned scholar with the resourcefulness of a web search.

RAG bridges static knowledge and real-time data retrieval at its core, enabling models to recall, research, and reference. This capability has proven to be a game-changer, particularly evident in knowledge-intensive tasks such as question-answering. RAG has demonstrated a marked improvement in generating contextually richer and more diverse responses. By integrating an information retrieval system, RAG allows for a tailored approach where the data steering the LLM’s responses can be meticulously controlled. This is particularly beneficial for enterprise solutions, where there is a need to channel the LLM’s focus onto specific, vectorized datasets—textual, visual, or auditory. Traditional foundation models are often restricted by the data they were trained on, oblivious to new information that emerges post-training. This often results in a plateau in their performance, especially when dealing with niche or evolving domains. RAG elegantly circumvents this limitation by dynamically fetching relevant data from an expansive external source, ensuring the model’s outputs are continually refreshed and contextually enriched.

General overview, Source: Author

The RAG framework operates in two distinct phases: retrieval and generation. During the retrieval phase, it employs algorithms to identify and gather pertinent information snippets based on the user’s input or query. This process taps into various data sources, including documents, databases, or APIs, to obtain the needed context. The gathered context is fed into a generator model, usually a sophisticated LLM. This model leverages the provided context to shape its output, ensuring the response is anchored in relevant, accurate information. The philosophy underpinning RAG is to complement the generative prowess of LLMs with the strategic incorporation of up-to-date, external data sources. This synergistic approach not only amplifies the model’s generative capabilities but also substantially elevates the quality and pertinence of the text it produces.

Benefits of RAG

RAG enhances LLMs by addressing key challenges:

  1. Reduced Hallucinations: LLMs can sometimes generate factually incorrect or irrelevant responses. RAG mitigates this using external sources, ensuring responses are more factual and relevant.
  2. Knowledge Cutoff: LLMs possess finite knowledge from their training data. RAG extends this by accessing external knowledge bases, enabling LLMs to provide more accurate and up-to-date responses.
  3. Enhanced Auditability: RAG improves the traceability of information by allowing LLMs to reference external sources, making verifying the origins of generated content easier. By drawing from verified sources, RAG enhances the reliability of its outputs.
  4. Contextual Awareness: LLMs often struggle with domain-specific queries due to a lack of context. RAG supplements LLMs with current, domain-specific data, enhancing the relevance and accuracy of responses.

Should you consider RAG?

Here are several compelling reasons:

  • Essential for Domain-Specific Knowledge: In fields where specialized knowledge is vital, RAG’s access to extensive, specific information is invaluable.
  • Critical for Fact-Based Responses: RAG’s fact-checking capabilities ensure reliability in scenarios where accuracy is paramount.
  • Beneficial for Limited Data Scenarios: RAG efficiently leverages existing data, unlike methods requiring extensive training datasets.
  • Advantageous for Dynamic Knowledge Needs: RAG stays current with rapidly changing information, a challenge for standalone LLMs.
  • Superior for Complex Reasoning: RAG facilitates advanced, multi-source reasoning.
  • Cost-Efficiency:  RAG is a resource-savvy alternative to fine-tuning LLMs, which requires a large amount of high-quality data, compute resources, and time; hence, RAG saves time and costs while maintaining high-quality outputs.
  • Increased Transparency: RAG’s ability to cite sources combats AI’s “black box” issue, fostering user trust.
  • Scalable for Any Business Size: RAG adapts to varying data and interaction volumes, making it a versatile tool for businesses of all scales.

RAG is ideal when tasks demand external knowledge integration, factual precision, and adaptability to evolving information landscapes. To clarify the distinction between RAG and fine-tuning an LLM,

consider this analogy: fine-tuning teaches your system how to think, while RAG instructs it on what to think. Fine-tuning involves customizing a pre-trained LLM for a particular task or domain, refining its thought process. In contrast, RAG supplements the LLM’s built-in knowledge by pulling in external information and providing it with additional content.

RAG Architecture

To explain the RAG architecture, consider the creation of a question-answering (QA) chatbot, integrating an LLM for document analysis, utilizing LangChain and Streamlit. LangChain is equipped with specialized components for developing RAG applications, while Streamlit facilitates the creation of the chatbot interface.
LangChain offers a user-friendly interface for working with LLMs. It is a highly popular framework for crafting LLM-driven applications. This framework simplifies the development process, allowing us to create sophisticated applications without delving into the intricacies of more complex programming layers.
Streamlit, a Python library, plays a crucial role in constructing the conversational interface of our application. It enables the creation of interactive chat elements, forming the user-facing part of the bot, thus facilitating an engaging and responsive user experience.

A standard RAG application encompasses two primary elements:

  • Indexing: This is the preliminary phase where data is ingested from a source and indexed. This process typically occurs offline and involves organizing data to facilitate efficient retrieval.
  • Retrieval and Generation: This is the core operational phase of the RAG application. Here, the user’s query is processed in real-time, triggering the retrieval of pertinent data from the indexed sources. Ranking, an optional but valuable step following retrieval, refines the order of documents initially deemed relevant. This process enhances how these documents are prioritized, ensuring that the most pertinent ones are presented first based on their relevance to the query. This data is then passed to the LLM, which generates a response based on the retrieved information, effectively combining retrieval with language model generation to produce accurate and relevant answers.
Source: Author

Source: Author

We are constructing a question-and-answer (QA) bot named RAGIntellect to implement the RAG architecture, where the foundation of this application relies on two previously introduced key components (LangChain and Streamlit).

The full code can be found on GitHub here.

This overview of the implementation covers several key steps:

  • Setting up the project environment and installing necessary components.
  • Constructing a sidebar for uploads and configuration settings.
  • Developing utility functions tailored to our application.
  • Crafting the main script integrates the RAG processes within a chat user interface.

Let’s begin!

Project Environment Setup

  • First, decide on a location on your computer where you want to store your project. Then, create a new directory for your project. You can do this using your file manager or via the command line.
  • Next, navigate to the newly created directory.
  • Once inside your project directory, create a Python virtual environment. A virtual environment is an isolated Python environment where you can install packages and run your project without affecting other Python projects or your system’s global Python setup. To create a virtual environment, use the following command: python -m venv venv
  • Before you start using the virtual environment, you need to activate it. The activation command differs slightly depending on your operating system:
    • On Windows, use: .\venv\Scripts\activate 
    • On MacOS or Linux, use: source venv/bin/activate
  • After activation, you should see the name of your virtual environment (in this case, venv) in your command prompt, indicating that it is active.
  • Next, install the LangChain and Streamlit libraries: pip install langchain streamlit
  • The project directory should be as follows:
    ├── components
    │   ├──
    │   └──
    └── venv

Practical Guide to Crafting your first LLM-powered App Using RAG Framework

  • Reduce Risk
  • Simplify Compliance
  • Gain Visibility
  • Version Comparison

Building the Application Sidebar

In building the sidebar of our Streamlit app, we implement a user-friendly interface for document upload and configuration. The provided code snippet outlines the functionality of this sidebar.

import os
import tempfile
import requests
import streamlit as st

ALLOWED_EXTENSIONS = ['.pdf', '.docx', '.doc', '.txt', '.ppt', '.csv', '.html', '.xls']

def get_saved_files_info():
    return []

def is_allowed_extension(file_name):
    return os.path.splitext(file_name)[1].lower() in ALLOWED_EXTENSIONS

def save_uploaded_file(uploaded_file):
    if is_allowed_extension(
        with tempfile.NamedTemporaryFile(delete=False, suffix=os.path.splitext([1]) as temp_file:
            st.toast(f"Saved {} to temporary storage.")
            return, os.path.splitext([1].lower()
        st.error("Unsupported file format.", icon="🚨")

def upload_file_via_url(url):
        response = requests.get(url)
        file_ext = os.path.splitext(url)[1].lower()
        if response.status_code == 200 and file_ext in ALLOWED_EXTENSIONS:
            with tempfile.NamedTemporaryFile(delete=False, suffix=file_ext) as temp_file:
                st.toast(f"Saved document from URL to temporary storage.")
                return, file_ext
            st.error('Invalid URL or unsupported content type.', icon="🚨")
    except requests.RequestException as e:
        st.error('Failed to fetch document from URL.', icon="🚨")

def sidebar():
    saved_files_info = get_saved_files_info()
    with st.sidebar:
        documents_uploads = st.file_uploader("Upload documents", accept_multiple_files=True, help="Supported formats: pdf, docx, doc, txt, ppt, csv, html, xls")
        if documents_uploads:
            for uploaded_file in documents_uploads:
                file_info = save_uploaded_file(uploaded_file)
                if file_info:

        documents_uploads_url = st.text_input("Upload documents via url", help="Supported formats: pdf, docx, doc, txt, ppt, csv, html, xls")
        submit_button = st.button('Upload link')
        if submit_button and documents_uploads_url:
            file_info = upload_file_via_url(documents_uploads_url)
            if file_info:


        with st.expander("Credentials"):
            openai_keys = st.text_input("OpenAI API Key", type='password')

        complete_button = st.button("Complete configuration", disabled=not (saved_files_info and openai_keys))

        if complete_button:
            return saved_files_info, openai_keys
            return None, None

Here’s a breakdown of its features

  • Supported File Formats: The app restricts file uploads to specific formats, including `.pdf`, `.docx`, `.doc`, `.txt`, `.ppt`, `.csv`, `.html`, and `.xls`. This ensures that the app processes only compatible file types.
  • File Upload Validation: The `is_allowed_extension` function checks if the uploaded file’s extension is among the allowed formats, enhancing data integrity and preventing irrelevant file processing.
  • Saving Uploaded Files: The `save_uploaded_file` function temporarily stores uploaded files and notifies the user with a toast message. It handles files uploaded directly from the user’s device.
  • URL-Based File Uploads: The `upload_file_via_url` function allows users to upload files via a URL. It fetches the file from the URL, saves it temporarily, and provides feedback on the success or failure of the operation.
  • Sidebar Interface:
    • Document Upload: Users can upload documents directly through the `file_uploader` interface or via URL input. This dual-upload method increases flexibility.
    • Feedback Mechanism: The app uses toast messages and error notifications to inform users about the status of their uploads.
    • Credentials Section: An expander section titled “Credentials” is provided for users to enter their OpenAI API key, ensuring secure and personalized access.
    • Configuration Completion: A “Complete configuration” button is present, becoming active only when files are uploaded and the API key is entered. This guides the user through the necessary steps before proceeding.

Essentially, this sidebar acts as the primary interaction point for users to upload documents and configure their API access, setting the stage for the app’s core functionalities.

If we call the sidebar function into our main script, ``, and execute the application with the command `streamlit run`, this will activate our application, including the sidebar functionalities as defined in the script.


To participate in this course effectively, one essential prerequisite is obtaining an OpenAI API key. This is because we will be utilizing OpenAI embeddings and Chat Models for our project. To experiment with the project or build it yourself, you can create your own API key by following the provided link. Additionally, you’ll need to install the OpenAI Python package. This package is necessary to interface with OpenAI’s API, allowing you to access embeddings and Chat Models for the project. You can install it using the following command: pip install openai

Also, you need to install openAI’s tokenizer: pip install tiktoken
Ensure you run these commands in your Python environment to have the OpenAI functionalities available for the project.
Next, with our upload functionality in place, let’s build the various backend processes as functions in our `` as follows:

import streamlit as st
    from langchain.document_loaders import (
    from langchain.document_loaders.csv_loader import CSVLoader
    from langchain.text_splitter import RecursiveCharacterTextSplitter
    from langchain.embeddings import OpenAIEmbeddings
    from langchain.vectorstores.faiss import FAISS
    from langchain.prompts import (
    from langchain.chains import ConversationalRetrievalChain
    from langchain.memory import ConversationBufferMemory
    from langchain.chat_models import ChatOpenAI

    def load_document(file_path, file_ext):
        if file_ext == '.pdf':
            return PyPDFLoader(file_path=file_path).load()
        elif file_ext == '.txt':
            return TextLoader(file_path=file_path).load()
        elif file_ext in ['.doc', '.docx']:
            return Docx2txtLoader(file_path=file_path).load()
        elif file_ext == '.ppt':
            return UnstructuredPowerPointLoader(file_path=file_path).load()
        elif file_ext == '.html':
            return UnstructuredHTMLLoader(file_path=file_path).load()
        elif file_ext == '.xls':
            return UnstructuredExcelLoader(file_path=file_path).load()
        elif file_ext == '.csv':
            return CSVLoader(file_path=file_path).load()

    def load_multiple_documents(file_infos):
        documents_to_text = []
        for file_path, file_ext in file_infos:
            documents_to_text.extend(load_document(file_path, file_ext))
        return documents_to_text

    def split_documents(documents):
        text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=20)
        text_chunks = text_splitter.split_documents(documents)
        return text_chunks

    def get_vectorstore(text_chunks, openai_keys):
        embeddings = OpenAIEmbeddings(openai_api_key=openai_keys, model='text-embedding-ada-002')
        vector_store = FAISS.from_documents(documents=text_chunks, embedding=embeddings)
        return vector_store

    def create_conservational_chain(vector_store, openai_keys):
        template = """
        As a highly competent AI assistant, your role is to analyze the documents provided by the user, Your responses should be grounded in the context contained within these documents.
        For each user query provided in the form of chat, apply the following guidelines:
        - If the answer is within the document's context, provide a detailed and precise response.
        - If the answer is not available based on the given context, clearly state that you don't have the information to answer.
        - If the query falls outside the scope of the context, politely clarify that your expertise is limited to answering questions directly related to the provided documents.

        When responding, prioritize accuracy and relevance:

        system_message_prompt = SystemMessagePromptTemplate.from_template(template=template)
        human_template = "{question}"
        human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)
        qa_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
        model = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.3, openai_api_key=openai_keys)
        memory = ConversationBufferMemory(memory_key='chat_history',return_messages=True)
        qa_chain = ConversationalRetrievalChain.from_llm(llm=model, retriever=vector_store.as_retriever(), memory=memory, combine_docs_chain_kwargs={'prompt': qa_prompt})
        return qa_chain

    def load_qa_chain(saved_files_info, openai_keys):
        loaded_docs = load_multiple_documents(saved_files_info)
        docs_splits = split_documents(loaded_docs)
        vectordb = get_vectorstore(docs_splits, openai_keys)
        return create_conservational_chain(vectordb, openai_keys)

    def initialize_state():
        if "messages" not in st.session_state:
            st.session_state.messages = [{"role": "assistant", "content": "How can I help?"}]
        if "qa_chain" not in st.session_state:
            st.session_state.qa_chain = None

The `` file is instrumental in operating a document-driven conversational AI application, since it contains critical functions for document handling, conversation flow management, and LangChain integration. Below is a detailed summary of its key components:

Import Statements: This file starts by importing necessary modules from Streamlit, LangChain, and other libraries. These include document loaders for various formats (PDF, TXT, DOC, PPT, HTML, XLS, CSV), embeddings, vector stores, prompt templates, and the conversational chain, all vital for our application.Document Loading Functions:

  • load_document: This function loads documents based on their file path and extension, utilizing appropriate LangChain document loaders for different file formats.
  • load_multiple_documents: Processes and compiles multiple documents into a list by calling load_document for each file.

Document Processing:

  • split_documents: Employs a RecursiveCharacterTextSplitter to segment documents into smaller chunks, enhancing indexing and model processing manageability. The splitter recursively uses characters like “\n\n”, “\n”, “\s” and the chuck size to handle and split large texts effectively.

Vector Store Creation:

  • get_vectorstore: This function creates a vector store by leveraging OpenAI’s embedding models, specifically using the ‘text-embedding-ada-002’ model (for a complete list of available models, see the linked documentation). It seamlessly connects with FAISS, an acclaimed open-source vector database developed by Facebook Research. Known for its exceptional scalability and robust handling of large-scale embeddings, FAISS has garnered over 25,000 GitHub stars to date. It offers implementations for both CPU and GPU, ensuring high efficiency in data retrieval – a vital component for the functionality of the conversational chain.

Conversational Chain Setup:

  • create_conversational_chain: This function establishes a conversational retrieval chain, incorporating custom prompt templates along with a chat model(Here, we are utilizing the ChatOpenAI model, particularly the gpt-3.5-turbo variant; refer to the documentation for a comprehensive list of available models). Additionally, it integrates a ConversationBufferMemory component, which is essential for ensuring continuity and coherence in conversations involving multiple queries. This is achieved by considering the context and content of past interactions within the conversation.

QA Chain Loader:

  • load_qa_chain: Merges document loading, splitting, vector store creation, and conversational chain setup. This function is key to initializing the QA functionality with provided documents and OpenAI keys.

Streamlit State Initialization:

  • initialize_state: Initializes session state variables in Streamlit, which is crucial for preserving conversational history and the state of the QA chain.

Overall, is a comprehensive toolkit for building and managing the backend of a document-based conversational AI, ensuring efficient document processing, accurate retrieval, and smooth user interactions.

Note: Several dependencies need to be installed to ensure the modules’ smooth


  • PDF Document Loader: To load PDF files into an array of documents, where each document includes page content and metadata like page number, you should install PyPDF using: pip install pypdf
  • DOCX Document Loader: For loading `.docx` and `.doc` files with the Docx2txtLoader, which converts `.docx` files into a document format, the required dependency is: pip install docx2txt
  • Vector Database API: You need to install the FAISS API for the vector database functionality. This can be done with the following command:pip install faiss-gpu # For CUDA 7.5+ Supported GPU's.
    # OR
    pip install faiss-cpu # For CPU Installation

    These installations are crucial for the respective document loaders and the vector database API to function correctly within your project.Next, we will implement our Streamlit-based web application named “RAGIntellect”, which is designed to serve as a platform for information retrieval and knowledge synthesis. The application integrates document handling capabilities and conversational AI, using OpenAI’s API for processing and generating responses.

    import streamlit as st
                    import openai
                    from components.sidebar import sidebar
                    from components.utils import (
                    def main():
                        # Set up the Streamlit page configuration and title
                        st.title("RAGIntellect: An Engine for Information Retrieval and Knowledge Synthesis")
                        # Invoke the sidebar for user inputs like file uploads and OpenAI keys
                        saved_files_info, openai_keys = sidebar()
                        st.subheader('Interaction with Documents')
                        # Initialize the session state variables
                        # Add a flag in the session state for API key validation
                        if "is_api_key_valid" not in st.session_state:
                            st.session_state.is_api_key_valid = None
                        # Load the QA chain if documents and OpenAI keys are provided, and handle OpenAI AuthenticationError
                        if saved_files_info and openai_keys and not st.session_state.qa_chain:
                                st.session_state.qa_chain = load_qa_chain(saved_files_info, openai_keys)
                                st.session_state.is_api_key_valid = True # Valid API key
                            except openai.AuthenticationError as e:
                                st.error('Please provide a valid API key. Update the API key in the sidebar and click "Complete Configuration" to proceed.', icon="🚨")
                                st.session_state.is_api_key_valid = False # Invalid API key
                        # Enable the chat section if the QA chain is loaded and API key is valid
                        if st.session_state.qa_chain and st.session_state.is_api_key_valid:
                            st.success("Configuration complete")
                            prompt = st.chat_input('Ask questions about the uploaded documents', key="chat_input")
                            # Process user prompts and generate responses
                            if prompt and (st.session_state.messages[-1]["content"] != prompt or st.session_state.messages[-1]["role"] != "user"):
                                st.session_state.messages.append({"role": "user", "content": prompt})
                                with st.spinner("Retrieving relevant information and generating output..."):
                                    response =
                                st.session_state.messages.append({"role": "assistant", "content": response})
                            # Display the conversation messages
                            for message in st.session_state.messages:
                                with st.chat_message(message["role"]):
                  "Please complete the configuration in the sidebar to proceed.")
                            # Disable the chat input if the API key is invalid
                            no_prompt = st.chat_input('Ask questions about the uploaded documents', disabled=not st.session_state.is_api_key_valid)
                    if __name__ == '__main__':

Here’s an overview of its functionalities:

Up the Streamlit Interface:

  • The set_page_config function configures the page with the title “RAGIntellect.”
  • The main title and a subheader provide a clear description of the application’s purpose.

Sidebar for User Input:

  • The sidebar function creates a sidebar in the UI, allowing users to upload documents and input their OpenAI API keys.

Initializing Session State:

  • The initialize_state function initializes variables in the Streamlit session state, which are used to store and manage the application’s state across interactions.

Handling API Key Validation:

  • A flag is_api_key_valid in the session state is used to track the validity of the OpenAI API key.

Loading the Question-Answering (QA) Chain:

  • If the necessary documents and OpenAI keys are provided, the load_qa_chain function is invoked.
  • The application tries to set up the QA chain, handling AuthenticationError from the OpenAI API. If the API key is invalid, it notifies the user to provide a valid key.

Conversational Interface:

  • The application enables a chat interface once the QA chain is set up and the API key is valid.
  • Users can input their questions or prompts related to the uploaded documents.
  • The application processes these prompts, uses the QA chain to generate responses, and appends them to the session state messages.

Displaying Conversations:

  • The application iteratively displays all messages stored in the session state, maintaining the flow of the conversation.
  • This includes user queries and AI-generated responses, facilitating an interactive chat-like experience.

Handling Incomplete Configuration:

  • If the configuration is incomplete (due to missing documents/API key or invalid API key), the application prompts the user to complete these steps. The chat input is disabled in this case.

By running this application, users can interact with RAGIntellect, which synthesizes information from uploaded documents to answer queries, making it a powerful tool for knowledge extraction and synthesis. The Streamlit framework provides an intuitive web interface, making the application accessible and user-friendly.

To run our app, we simply run:
streamlit run

Excellent, let’s test our application using the Deepchecks paper along with its welcome page as follows:

Demo, source Author

Demo, source Author

We can also observe how the system responds to questions that fall outside the document’s context, as illustrated below:


In this article, we delved into the RAG architecture, creating a QA bot adept at information retrieval and knowledge synthesis from uploaded documents, utilizing LLMs’ capabilities. This bot’s integration of a conversational AI interface and proficiency in processing and responding to document-based queries underscores its effectiveness and potential as a multifaceted tool in AI-driven data handling and user interaction.

However, this is just the beginning. We invite developers and AI enthusiasts to contribute actively to this project’s evolution. Forking the repository opens up a world of exploration, where you can dive deep into the code, grasp its intricacies, and bring your unique ideas to life. Owning a copy of the codebase unlocks limitless opportunities for enhancement and customization. Whether it’s about expanding the bot’s features, refining its existing functionalities, or adapting it for niche applications, your involvement in building and experimenting is not just an enrichment for the project but also a catalyst for your personal development in coding and AI.

Staying abreast of the latest developments in the field is crucial. AI and technology are constantly progressing, and keeping informed ensures that your contributions remain relevant and impactful. By engaging with this project, you’re part of an innovative endeavor and at the forefront of AI technology and application.


Practical Guide to Crafting your first LLM-powered App Using RAG Framework

  • Reduce Risk
  • Simplify Compliance
  • Gain Visibility
  • Version Comparison

Recent Blog Posts

Precision vs. Recall in the Quest for Model Mastery
Precision vs. Recall in the Quest for Model Mastery