RAG: Context-Aware Chunking from Google Drive to Pinecone

Overview

This workflow automates the process of ingesting documents from Google Drive, performing context-aware chunking, generating embeddings with Google Gemini, and storing these embeddings in Pinecone. This is designed for Retrieval Augmented Generation (RAG) systems, aiming to improve the accuracy and relevance of retrieved information by embedding text chunks along with their contextual surrounding.

Key Features

• Document Ingestion: Downloads documents directly from Google Drive.
• Contextual Chunking: Splits documents into sections and generates short, succinct context for each chunk.
• Embedding Generation: Utilizes Google Gemini for creating high-quality text embeddings.
• Vector Storage: Stores the embeddings and their associated context in a Pinecone vector database.
• RAG Optimization: Enhances RAG performance by including context in the embedding process.