Cloud ML - Azure AI Engineer Associate (Exam AI-102): Retrieval Augmented Generation (RAG) with Azure AI Search and Azure OpenAI

Azure_AI – Retrieval Augmented Generation (RAG) with Azure AI Search and Azure OpenAI

Azure AI-102 Study Guide: Retrieval Augmented Generation (RAG) with Azure AI Search & Azure OpenAI

What This Is

Retrieval Augmented Generation (RAG) is a hybrid AI pattern that combines information retrieval (fetching relevant documents) with generative AI (answering questions in natural language). Instead of relying solely on an LLM’s pre-trained knowledge (which may be outdated or incomplete), RAG dynamically pulls in up-to-date, domain-specific data from a knowledge base (e.g., PDFs, databases, or FAQs) to generate more accurate and context-aware responses.

Real-world scenario:
A healthcare chatbot needs to answer patient questions about symptoms, treatments, and insurance coverage. Instead of fine-tuning an LLM on medical data (expensive and risky), the team uses Azure AI Search to index clinical guidelines, drug databases, and patient records. When a user asks, "What are the side effects of Drug X?", the system retrieves the latest FDA documentation and passes it to Azure OpenAI’s GPT-4 to generate a precise, cited response—without hallucinations.

Key Terms & Services

• Azure AI Search (formerly Azure Cognitive Search):
  Microsoft’s managed vector + keyword search service. Indexes structured and unstructured data (PDFs, Word docs, databases) and supports semantic search (understanding intent) and vector search (finding similar embeddings). Best for RAG pipelines where you need fast, scalable retrieval.

• Azure OpenAI Service:
  Microsoft’s managed API for OpenAI models (GPT-4, GPT-3.5, DALL·E, etc.). Provides chat completions, embeddings, and fine-tuning (limited). Used in RAG to generate responses after retrieving relevant documents.

• Embeddings:
  Numerical representations of text (or other data) that capture semantic meaning. Azure OpenAI’s text-embedding-ada-002 model converts text into vectors for vector search in Azure AI Search.

• Vector Search:
  A search method that finds similar items by comparing embeddings (e.g., "Find documents about diabetes treatments"). Azure AI Search supports approximate nearest neighbor (ANN) search for low-latency retrieval.

• Semantic Search:
  A search method that understands context and intent (e.g., "What’s the best medication for high blood pressure?" vs. "hypertension drugs"). Azure AI Search uses Microsoft’s semantic ranking to improve relevance.

• Chunking:
  Splitting large documents into smaller chunks (e.g., paragraphs, sections) before indexing. Critical for RAG because LLMs have token limits (e.g., GPT-4 supports 8K–32K tokens). Poor chunking leads to context truncation or irrelevant retrievals.

• Prompt Engineering:
  Crafting input prompts to guide an LLM’s output. In RAG, prompts include retrieved documents + user questions (e.g., "Answer the question using only the following context: [retrieved docs]. Question: [user input]").

• Grounding:
  Ensuring an LLM’s response is based on retrieved data (not hallucinated). Azure OpenAI’s content filtering and prompt engineering help enforce grounding.

• Indexer (Azure AI Search):
  A pipeline that automatically extracts, transforms, and loads (ETL) data from sources (Blob Storage, SQL DB, etc.) into an Azure AI Search index. Supports custom skills (e.g., OCR, entity recognition).

• Skillset (Azure AI Search):
  A collection of AI enrichments (e.g., text translation, key phrase extraction, OCR) applied during indexing. Used to pre-process data before retrieval.

• Retrieval Score:
  A metric (0–1) indicating how relevant a retrieved document is to the query. Azure AI Search returns scores for keyword and vector matches.

• Hybrid Search:
  Combining keyword search (BM25) + vector search (ANN) for better recall. Azure AI Search supports hybrid queries out of the box.

Step-by-Step: Building a RAG Pipeline with Azure AI Search & Azure OpenAI

1. Prepare Your Data Source

• Store documents (PDFs, Word, HTML, JSON) in Azure Blob Storage or a database (SQL, Cosmos DB).
• Best practice: Use Azure Data Lake Storage (ADLS Gen2) for large-scale unstructured data.
• Example: Upload 10,000 PDFs of product manuals to a Blob Storage container.

2. Create an Azure AI Search Service

• In the Azure Portal, create an Azure AI Search resource.
• Choose a pricing tier (Free, Basic, Standard, or Storage Optimized). For RAG, Standard (S1) is a good starting point.
• Key settings:
• Semantic Search: Enable (adds ~$1/query but improves relevance).
• Vector Search: Enable (required for embeddings).

3. Index Your Data

Option A: Use the Azure Portal (No-Code)

• Navigate to Import Data → Select your data source (Blob Storage, SQL, etc.).
• Configure an indexer (automatically extracts text from files).
• Define an index schema (fields like id, content, metadata).
• Enable vector search: Add a field of type Collection(Edm.Single) for embeddings.

Option B: Programmatic (Python SDK)

from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import *

# Define index schema
fields = [
    SimpleField(name="id", type=SearchFieldDataType.String, key=True),
    SearchableField(name="content", type=SearchFieldDataType.String),
    SearchField(name="embedding", type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
                vector_search_dimensions=1536,  # For text-embedding-ada-002
                vector_search_profile_name="my-vector-profile")
]

# Create index
index_client = SearchIndexClient(endpoint, credential)
index = SearchIndex(name="rag-index", fields=fields)
index_client.create_index(index)

4. Generate & Store Embeddings

• Use Azure OpenAI’s text-embedding-ada-002 to convert text into vectors.
• Chunking strategy:
• Split documents into 512–1024 token chunks (GPT-4’s context window is 8K–32K tokens).
• Use overlapping chunks (e.g., 100-token overlap) to avoid cutting off context.
• Store embeddings in the vector field of your Azure AI Search index.

from openai import AzureOpenAI

client = AzureOpenAI(
    api_key="YOUR_KEY",
    api_version="2023-05-15",
    azure_endpoint="YOUR_ENDPOINT"
)

def generate_embeddings(text):
    response = client.embeddings.create(input=text, model="text-embedding-ada-002")
    return response.data[0].embedding

5. Query the Index (Retrieval)

• Use hybrid search (keyword + vector) for best results.
• Example query (Python SDK):

from azure.search.documents import SearchClient

search_client = SearchClient(endpoint, "rag-index", credential)

results = search_client.search(
    search_text="What are the side effects of Drug X?",  # Keyword search
    vector_queries=[VectorizedQuery(vector=query_embedding, k_nearest_neighbors=3, fields="embedding")],  # Vector search
    select=["content", "metadata"],
    top=3  # Return top 3 results
)

6. Generate a Response with Azure OpenAI (Generation)

• Pass the retrieved documents + user question to GPT-4.
• Prompt template:

"Answer the question using only the following context. If you don't know the answer, say 'I don't know.'

Context:
{document_1}
{document_2}
{document_3}

Question: {user_question}

Answer:"

• Example (Python):

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": prompt_template.format(
            document_1=results[0]["content"],
            document_2=results[1]["content"],
            document_3=results[2]["content"],
            user_question="What are the side effects of Drug X?"
        )}
    ]
)
print(response.choices[0].message.content)

Common Mistakes

Mistake 1: Using Only Keyword Search (No Vectors)

• Problem: Keyword search misses semantic meaning (e.g., "heart attack" vs. "myocardial infarction").
• Correction: Always use hybrid search (keyword + vector) in Azure AI Search. Enable semantic search for better ranking.

Mistake 2: Poor Chunking (Too Large or Too Small)

• Problem:
• Too large: Exceeds LLM’s token limit (e.g., 8K tokens for GPT-4).
• Too small: Loses context (e.g., a single sentence may not answer a question).
• Correction:
• Use 512–1024 token chunks (adjust based on document structure).
• Add overlap (e.g., 100 tokens) to preserve context.

Mistake 3: Not Grounding the LLM in Retrieved Data

• Problem: The LLM hallucinates answers not found in the retrieved documents.
• Correction:
• Use strict prompt engineering (e.g., "Answer using only the following context").
• Enable Azure OpenAI’s content filtering to block unsupported claims.

Mistake 4: Ignoring Retrieval Scores

• Problem: Low-scoring documents (e.g., score < 0.5) may be irrelevant, leading to bad answers.
• Correction:
• Filter results by score threshold (e.g., if score > 0.6).
• Use reranking (Azure AI Search’s semantic search improves this).

Mistake 5: Not Monitoring Latency

• Problem: Vector search can be slow if not optimized (e.g., no index tuning).
• Correction:
• Use approximate nearest neighbor (ANN) search (default in Azure AI Search).
• Monitor query latency in Azure Monitor and adjust k_nearest_neighbors.

Certification Exam Insights

1. Service Selection Traps

• Azure AI Search vs. Azure Cognitive Services (Document Intelligence):
• Use Azure AI Search for RAG (retrieval + generation).
• Use Document Intelligence for structured data extraction (e.g., invoices, forms).
• Azure OpenAI vs. Azure Machine Learning (AML) for RAG:
• Azure OpenAI is for generation (GPT-4) and embeddings.
• AML is for training custom models (not needed for RAG unless fine-tuning).

2. Key Constraints

• Token Limits:
• GPT-4: 8K–32K tokens (context window).
• text-embedding-ada-002: 8191 tokens max per input.
• Azure AI Search Limits:
• Free tier: 50 MB index, 3 indexes, no semantic search.
• Standard (S1): 200 GB index, 50 indexes, semantic search available.

3. "Which Service?" Scenarios

• Q: "A company needs to build a chatbot that answers questions about internal HR policies. They have 10,000 PDFs stored in Blob Storage. Which Azure services should they use?"
• A: Azure AI Search (for retrieval) + Azure OpenAI (for generation).

• Why? AI Search indexes the PDFs, OpenAI generates answers.

• Q: "A team wants to improve search relevance for a legal document RAG system. They need to understand query intent. Which Azure AI Search feature should they enable?"

• A: Semantic Search.
• Why? It uses deep learning to rank results by meaning, not just keywords.

4. Cost Optimization

• Vector search is expensive (charged per query). Use hybrid search to reduce costs.
• Semantic search adds ~$1 per 1000 queries—disable if not needed.
• Azure OpenAI pricing:
• GPT-4: $0.03–$0.12 per 1K tokens (input + output).
• Embeddings: $0.0001 per 1K tokens.

Quick Check Questions

1. A healthcare startup is building a RAG system to answer patient questions about clinical trials. They need to retrieve relevant studies from a database of 50,000 PDFs. Which Azure service should they use for retrieval?

• A: Azure AI Search
• Explanation: AI Search is optimized for scalable, low-latency retrieval of unstructured data (PDFs, docs) and supports vector + keyword search.

2. A company’s RAG pipeline is returning irrelevant documents. They’re using Azure AI Search with vector search only. What’s the most likely issue, and how should they fix it?

• A: They’re missing keyword search (hybrid search).
• Explanation: Vector search alone may miss exact keyword matches (e.g., "COVID-19" vs. "coronavirus"). Hybrid search (vector + keyword) improves recall.

3. A team is generating embeddings for a RAG pipeline using Azure OpenAI’s text-embedding-ada-002. Their documents are 10,000 tokens long. What should they do before generating embeddings?

• A: Split documents into smaller chunks (e.g., 512–1024 tokens).
• Explanation: text-embedding-ada-002 has an 8191-token limit. Chunking ensures all text is processed.

Last-Minute Cram Sheet

1. RAG = Retrieval (Azure AI Search) + Generation (Azure OpenAI).
2. Azure AI Search supports hybrid search (keyword + vector) and semantic search.
3. text-embedding-ada-002 = 1536-dimension embeddings for vector search.
4. Chunk documents into 512–1024 tokens to avoid truncation.
5. GPT-4 context window: 8K–32K tokens (plan prompts accordingly).
6. Enable semantic search in Azure AI Search for better relevance (costs ~$1/1000 queries).
7. ⚠️ Azure AI Search Free tier has no semantic search or vector search.
8. ⚠️ Don’t use Azure Cognitive Services (Document Intelligence) for RAG—it’s for structured extraction.
9. Monitor retrieval scores (filter out low-scoring docs to reduce hallucinations).
10. ⚠️ Azure OpenAI embeddings cost $0.0001/1K tokens—cheap, but watch large datasets.