Cloud ML - Azure AI Engineer Associate (Exam AI-102): Question Answering (QnA Maker, Custom Question Answering)

Azure_AI – Question Answering (QnA Maker, Custom Question Answering)

Azure AI-102 Study Guide: Question Answering (QnA Maker & Custom Question Answering)

What This Is

Question Answering (QnA) in Azure is a natural language processing (NLP) service that lets you build a knowledge base (KB) from structured or unstructured content (FAQs, manuals, web pages) and answer user queries conversationally. It’s critical in enterprise chatbots, customer support automation, and internal knowledge portals—think a bank’s virtual assistant answering "How do I reset my password?" or a hospital’s FAQ bot explaining insurance coverage. Unlike raw LLMs, QnA Maker (now Custom Question Answering) specializes in precise, curated responses with minimal hallucination, making it ideal for regulated industries.

Key Terms & Services

• Azure Cognitive Services (Language Service): A suite of pre-built AI APIs (including Custom Question Answering) for NLP tasks like sentiment analysis, entity recognition, and QnA.
• Custom Question Answering (CQA): The evolved version of QnA Maker, now part of Azure Language Service, enabling multi-turn conversations, active learning, and hybrid KB sources (e.g., FAQs + PDFs).
• Knowledge Base (KB): A structured repository of question-answer pairs (QnAs) that CQA uses to respond to user queries. Can be imported from URLs, PDFs, Excel, or manually authored.
• Multi-Turn Conversations: CQA’s ability to chain follow-up questions (e.g., "What’s my balance?" → "For which account?" → "Checking").
• Active Learning: CQA automatically flags low-confidence answers for human review, improving accuracy over time.
• Chit-Chat: Pre-built small talk responses (e.g., "How are you?") that can be enabled/disabled in CQA.
• Azure Bot Service: A framework to deploy CQA-powered bots to channels like Teams, Slack, or a website.
• LUIS (Language Understanding): A complementary service for intent/entity extraction (e.g., "Book a flight to Paris" → intent: BookFlight, entity: Paris). Often used alongside CQA for complex dialogs.
• Azure Cognitive Search: A search-as-a-service tool that can enhance CQA with semantic search (e.g., retrieving answers from unstructured documents).
• Confidence Score: A 0–1 metric indicating how likely CQA’s answer is correct. Used to route low-confidence queries to humans.
• Synonyms & Alternate Phrasing: CQA’s ability to map multiple question variants to the same answer (e.g., "How do I pay?" and "What are my payment options?").
• Deployment Slots: CQA supports staging vs. production environments for safe testing before release.

Step-by-Step / Process Flow

1. Create a Custom Question Answering Project

• Action: Navigate to the Azure Language Studio → Create a new "Custom question answering" project.
• Key Steps:
• Choose a resource group and Azure region (e.g., East US).
• Select a pricing tier (Free F0 for testing, Standard S for production).
• Define the default language (e.g., English).
• Why? This provisions the underlying Language Service resource and initializes your KB.

2. Populate the Knowledge Base

• Action: Add sources to your KB:
• Manual entry: Type QnA pairs directly.
• URLs: Scrape FAQs from a website (e.g., https://contoso.com/faq).
• Files: Upload PDFs, Excel, or Word docs (CQA extracts QnA pairs automatically).
• Existing KB: Import from QnA Maker (legacy) or another CQA project.
• Pro Tip: Use markdown formatting in answers (e.g., bold, lists) for better readability.
• Why? The KB is the source of truth for your bot’s responses.

3. Train & Test the KB

• Action: Click "Test" in Language Studio to:
• Simulate user queries (e.g., "How do I return a product?").
• Review confidence scores (aim for >0.7 for production).
• Add alternate phrasing (e.g., "I want to send something back" → same answer as "return").
• Key Step: Enable active learning to let CQA flag low-confidence answers for review.
• Why? Ensures your bot generalizes well to real-world queries.

4. Deploy the KB to an Endpoint

• Action: Click "Deploy" to create a REST API endpoint for your KB.
• Endpoint URL: https://<your-resource-name>.cognitiveservices.azure.com/language/:query-knowledgebases?projectName=<your-project>&api-version=2021-10-01
• Authentication: Use Ocp-Apim-Subscription-Key (found in Azure Portal under your Language Service resource).
• Why? This lets your bot or app call CQA programmatically.

5. Integrate with Azure Bot Service (Optional)

• Action: Create a Bot Service resource in Azure Portal:
• Choose "Question answering" as the bot type.
• Connect it to your CQA endpoint and subscription key.
• Deploy to channels (e.g., Teams, Web Chat, Slack).
• Why? Turns your KB into a fully functional chatbot with minimal code.

6. Monitor & Improve with Analytics

• Action: Use Azure Monitor and Application Insights to track:
• Top queries (what users ask most).
• Unanswered questions (gaps in your KB).
• Confidence scores (identify weak answers).
• Key Step: Regularly update the KB based on analytics (e.g., add missing QnAs).
• Why? Ensures your bot stays accurate and relevant.

Common Mistakes

Mistake 1: Confusing Custom Question Answering with Azure Cognitive Search

• Mistake: Assuming CQA can search unstructured documents like Cognitive Search.
• Correction: CQA only works with curated QnA pairs. For ad-hoc document search, use Cognitive Search + CQA hybrid (e.g., Cognitive Search retrieves relevant docs, CQA answers specific questions).

Mistake 2: Ignoring Multi-Turn Conversations

• Mistake: Building a KB with only single-turn QnAs (e.g., "What’s your return policy?" but not handling follow-ups like "How do I print a label?").
• Correction: Use contextual prompts in CQA to chain questions (e.g., "Would you like the return label?" after "How do I return a product?").

Mistake 3: Not Setting Up Active Learning

• Mistake: Deploying a KB without enabling active learning, leading to stagnant accuracy.
• Correction: Turn on active learning in Language Studio to automatically flag low-confidence answers for review.

Mistake 4: Overlooking Confidence Thresholds

• Mistake: Routing all queries to the bot, even those with low confidence scores (e.g., <0.5).
• Correction: Set a confidence threshold (e.g., 0.7) and escalate low-confidence queries to humans.

Mistake 5: Hardcoding Answers Instead of Using Synonyms

• Mistake: Manually adding every possible question variant (e.g., "How to pay," "Payment options," "Pay bill").
• Correction: Use synonyms and alternate phrasing in CQA to automatically map variants to the same answer.

Certification Exam Insights

1. Service Selection Traps

• Trap: The exam may ask whether to use CQA, LUIS, or Cognitive Search for a scenario.
• CQA: Best for curated QnA pairs (e.g., FAQs, manuals).
• LUIS: Best for intent/entity extraction (e.g., "Book a flight to Paris").
• Cognitive Search: Best for ad-hoc document search (e.g., "Find all contracts signed in 2023").
• Key Rule: If the question mentions "FAQs," "knowledge base," or "predefined answers," pick CQA.

2. Multi-Turn vs. Single-Turn

• Trap: The exam tests whether you know when to use multi-turn conversations.
• Single-turn: Simple QnA (e.g., "What are your hours?").
• Multi-turn: Complex dialogs (e.g., "I want to return a product" → "Which product?" → "How do I ship it back?").
• Key Rule: If the scenario involves follow-up questions, enable multi-turn in CQA.

3. Active Learning & Human-in-the-Loop

• Trap: The exam may ask how to improve answer accuracy over time.
• Wrong Answer: "Retrain the model daily."
• Correct Answer: "Enable active learning to flag low-confidence answers for human review."
• Key Rule: Active learning is the primary way to improve CQA accuracy (not retraining).

4. Deployment & Scaling

• Trap: The exam tests how to scale CQA for high traffic.
• Wrong Answer: "Use a higher-tier QnA Maker resource." (QnA Maker is legacy.)
• Correct Answer: "Deploy CQA in Standard S tier and use Azure Bot Service for load balancing."
• Key Rule: CQA scales automatically—focus on bot service configuration for high traffic.

Quick Check Questions

Question 1

A healthcare company wants to build a chatbot to answer patient FAQs about insurance coverage, with minimal hallucination. The bot must handle follow-up questions like "What’s my copay for a specialist visit?" Which Azure service should they use? - A) Azure OpenAI Service - B) Custom Question Answering (CQA) - C) LUIS - D) Azure Cognitive Search

Answer: B) Custom Question Answering (CQA)
Explanation: CQA is designed for curated QnA pairs (reducing hallucination) and supports multi-turn conversations, making it ideal for healthcare FAQs.

Question 2

A retail company’s CQA bot is getting many unanswered questions about "holiday return policies." What’s the fastest way to improve the bot’s accuracy? - A) Retrain the CQA model from scratch.
- B) Enable active learning and review flagged questions.
- C) Switch to Azure OpenAI Service.
- D) Increase the confidence threshold to 0.9.

Answer: B) Enable active learning and review flagged questions.
Explanation: Active learning automatically flags low-confidence/unanswered questions for human review, improving accuracy without retraining.

Question 3

A financial services bot needs to answer "What’s my account balance?" but must first verify the user’s identity. Which Azure services should you combine? - A) CQA + Azure Bot Service - B) CQA + LUIS - C) CQA + Azure Active Directory (AAD) - D) CQA + Azure Cognitive Search

Answer: B) CQA + LUIS
Explanation: LUIS extracts the intent ("check balance") and entities (account type), while CQA provides the answer. AAD (C) is for authentication, not dialog management.

Last-Minute Cram Sheet

1. Custom Question Answering (CQA) = Evolved QnA Maker, part of Azure Language Service.
2. KB sources: URLs, PDFs, Excel, manual entry, or import from QnA Maker (legacy).
3. Multi-turn conversations = Follow-up questions (e.g., "Which account?" after "What’s my balance?").
4. Active learning = Automatically flags low-confidence answers for human review.
5. Confidence score (0–1): >0.7 = good, <0.5 = escalate to human.
6. Synonyms & alternate phrasing = Map multiple questions to one answer (e.g., "pay bill" = "payment options").
7. Chit-chat = Pre-built small talk (e.g., "How are you?"). Disable for enterprise bots.
8. Deployment: CQA provides a REST API endpoint (use with Azure Bot Service).
9. ⚠️ Trap: CQA cannot search unstructured docs—use Cognitive Search + CQA hybrid for that.
10. Pricing: Free F0 (1,000 calls/month), Standard S (~$1 per 1,000 calls).