RAG Document Update and Management with Qdrant and Google Drive
detail.loadingPreview
Automates RAG system updates using Qdrant vector store and Google Drive documents.
About This Workflow
This workflow automates the creation and management of a Retrieval-Augmented Generation (RAG) system. It leverages Qdrant as a vector store and Google Drive as the document source. The primary function is to enable full or incremental updates to documents within the Qdrant vector database. It also integrates with a chatbot using Google Gemini for question answering, making it a comprehensive solution for maintaining an up-to-date RAG system.
Key Features
- Automated Collection Creation: Dynamically creates Qdrant collections if they don't exist.
- Document Ingestion from Google Drive: Fetches documents from a specified Google Drive folder.
- Data Conversion: Converts Google Drive documents (like Google Docs) to plain text for processing.
- Incremental Updates: Supports deleting existing documents in Qdrant based on
file_idbefore inserting updated versions. - Full Collection Clearing: Option to clear the entire Qdrant collection for full re-indexing.
- Text Splitting: Divides documents into manageable chunks using Recursive Character Text Splitter.
- OpenAI Embeddings: Generates vector embeddings for text chunks using OpenAI.
- Qdrant Vector Storage: Stores embeddings and associated metadata in Qdrant.
- RAG Chatbot Integration: Sets up a question-answering chain using Google Gemini and Qdrant as a retriever.
- Error Handling & Waiting: Includes a wait node, potentially for asynchronous Qdrant operations or rate limiting.
How To Use
- Configure
Create collectionnode:- Replace
QDRANTURLwith your Qdrant instance URL. - Replace
COLLECTIONwith your desired collection name. - Ensure
Create collectionis executed first to set up your Qdrant collection.
- Replace
- Configure
Clear collectionnode (Optional for full re-indexing):- Ensure
QDRANTURLandCOLLECTIONare correctly set. - This node will delete all existing points in the collection. Use with caution.
- Ensure
- Configure
Get filesnode:- Ensure your
Google Drive account (n3w.it)credential is set up. - Set the
folderIdto the Google Drive folder containing the documents you want to process.
- Ensure your
- Configure
Download filesnode:- This node is triggered by
Get filesand will download each file found. - The
Google File Conversionis set totext/plainwhich is suitable for most document types.
- This node is triggered by
- Configure
Edit Fields3node:- This node is intended to set a
file_idthat will be used for updates. The current valueDRIVEFILE_IDneeds to be dynamically populated from theGet filesorDownload filesnode's output (e.g.,{{ $json.id }}).
- This node is intended to set a
- Configure
Delete single filenode:- This node is used for incremental updates. It deletes points from Qdrant where the
metadata.file_idmatches the provided ID. - Ensure
QDRANTURLandCOLLECTIONare correct. - The
{{$json.file_id}}needs to be correctly mapped from the input data.
- This node is used for incremental updates. It deletes points from Qdrant where the
- Configure
Download filenode:- This node downloads a specific file, likely used in conjunction with
Edit Fields3. - The
fileIdshould be dynamically set, e.g.,{{ $json.file_id }}.
- This node downloads a specific file, likely used in conjunction with
- Configure
Update single filenode (likelyQdrant Vector Storein a different context):- This node is responsible for inserting the updated document's vector representation into Qdrant.
- Ensure the
qdrantCollectionis set correctly. - The node connects to
Embeddings OpenAI2andDefault Data Loaderwhich process the downloaded file.
- Configure
Qdrant Vector Storenode (for initial indexing):- This node inserts embeddings into the
negozio-emporio-verdecollection. - It receives data from
Download filesand is processed byEmbeddings OpenAI1andDefault Data Loader1.
- This node inserts embeddings into the
- Configure
Recursive Character Text Splitternodes:- Adjust
chunkSizeandchunkOverlapas needed for optimal text segmentation.
- Adjust
- Configure
Embeddings OpenAInodes:- Ensure your
OpenAi accountcredential is set up correctly.
- Ensure your
- Configure
Qdrant Vector Store1node (for RAG retrieval):- This node uses the
ocr_mistral_testcollection for retrieving information.
- This node uses the
- Configure
Vector Store Retrievernode:- This node is configured to use the Qdrant vector store as a retriever.
- Configure
Google Gemini Chat Modelnode:- Ensure your
Google Gemini(PaLM) Api accountcredential is set up.
- Ensure your
- Configure
Question and Answer Chainnode:- This orchestrates the RAG process, using the retriever and language model.
- Configure
When chat message receivednode:- This acts as the entry point for chatbot queries, triggering the RAG process.
- Trigger Workflow: Use the
When clicking ‘Test workflow’node to manually run the initial setup or update process. The RAG chat functionality is triggered via theWhen chat message receivedwebhook.- Important: Replace placeholder URLs and collection names (e.g.,
http://QDRANTURL/collections/COLLECTION,negozio-emporio-verde,ocr_mistral_test) with your actual configurations.
- Important: Replace placeholder URLs and collection names (e.g.,
Apps Used
Workflow JSON
{
"id": "bf01179d-09f0-4340-9e19-00b60bf33d41",
"name": "RAG Document Update and Management with Qdrant and Google Drive",
"nodes": 14,
"category": "AI/ML",
"status": "active",
"version": "1.0.0"
}Note: This is a sample preview. The full workflow JSON contains node configurations, credentials placeholders, and execution logic.
Get This Workflow
ID: bf01179d-09f0...
About the Author
SaaS_Connector
Integration Guru
Connecting CRM, Notion, and Slack to automate your life.
Statistics
Related Workflows
Discover more workflows you might like
OpenAI Assistant for File Retrieval with Citation Formatting
Automates generating structured metadata from OpenAI assistant responses, ensuring citations and file sources are correctly identified and formatted.
RAG AI Agent with Milvus and Cohere
Automates the creation of a Retrieval-Augmented Generation (RAG) AI agent. It ingests documents from Google Drive, processes them, embeds them using Cohere, stores them in Milvus, and enables chat-based interaction for context-aware responses.
Crop Anomaly Detection Tool
Detects if an input image depicts an anomalous crop not present in the trained dataset.