Cloud ML - Azure AI Engineer Associate (Exam AI-102): Image Analysis (Analyze API – Tags, Captions, Dense Captions, OCR)

Azure_AI – Image Analysis (Analyze API – Tags, Captions, Dense Captions, OCR)

Azure AI-102 Study Guide: Image Analysis (Analyze API – Tags, Captions, Dense Captions, OCR)

What This Is

The Azure AI Vision Analyze API is a pre-built, serverless service that extracts structured insights from images—such as objects, people, text, and scenes—without requiring custom ML model training. It’s critical in ML pipelines where real-time or batch image understanding is needed, such as: - Retail: Automatically tagging product images for search and recommendations (e.g., "blue sneakers, size 10, running shoes").
- Healthcare: Extracting text from medical forms (OCR) and detecting anomalies in X-rays (via custom vision if needed).
- Media & Entertainment: Generating alt-text for accessibility (captions) and moderating user-uploaded content (tags for "violence," "adult content").
- Logistics: Reading shipping labels (OCR) and verifying package contents (object detection).

Unlike Azure Custom Vision (which requires training), the Analyze API is zero-shot—you call it with an image URL or binary data, and it returns JSON with tags, captions, dense captions, and OCR results.

Key Terms & Services

• Azure AI Vision (Computer Vision) Service:
  Microsoft’s managed API for image analysis, including object detection, OCR, facial recognition, and scene understanding. Best for pre-built, low-code image insights (no training required).

• Analyze API (Image Analysis 4.0):
  The latest version of Azure’s image analysis endpoint. Returns tags, captions, dense captions, objects, people, and OCR in a single call. Optimized for low latency and high accuracy on common scenarios.

• Tags:
  Keywords describing the image (e.g., "dog," "beach," "sunset"). Useful for search indexing, content filtering, and recommendation systems.

• Captions:
  A single natural-language sentence summarizing the image (e.g., "A dog playing on the beach"). Used for accessibility (alt-text) and content moderation.

• Dense Captions:
  Multiple detailed descriptions of regions in the image (e.g., "A golden retriever running," "A blue beach ball"). Useful for fine-grained analysis (e.g., detecting multiple objects in a scene).

• OCR (Optical Character Recognition):
  Extracts printed or handwritten text from images (e.g., receipts, forms, street signs). Azure’s OCR supports 164 languages and handles rotated, skewed, or low-quality text.

• Read API (vs. OCR in Analyze API):
  A dedicated OCR endpoint for large documents (PDFs, multi-page TIFFs). Use this instead of the Analyze API’s OCR for high-volume text extraction (e.g., invoices, contracts).

• Azure AI Document Intelligence (formerly Form Recognizer):
  A specialized OCR + layout analysis service for structured documents (tables, key-value pairs). Use this for invoices, receipts, or ID cards (not general images).

• Azure Cognitive Search:
  A search service that can index and retrieve images based on tags, captions, or OCR text from the Analyze API. Useful for building image search engines.

• Azure Functions / Logic Apps:
  Serverless tools to orchestrate the Analyze API (e.g., trigger OCR on new blob uploads). Logic Apps has a built-in "Computer Vision" connector for no-code workflows.

• Azure Blob Storage:
  Where you store images before/after analysis. The Analyze API can read directly from blob URLs (no need to download locally).

• Azure Event Grid:
  Triggers real-time processing when new images are uploaded to Blob Storage (e.g., "Run OCR on this new receipt").

• Cost Model (Pay-as-you-go):
  Pricing is per 1,000 transactions (e.g., $1.50 per 1,000 images for the Analyze API). OCR is more expensive ($1.50 per 1,000 pages for Read API vs. $1.00 for Analyze API OCR).

Step-by-Step / Process Flow

Scenario: Automatically Tag and Caption Product Images for an E-Commerce Site

Goal: Process 10,000 product images daily, extract tags/captions, and store results in a database for search.

Step 1: Set Up Azure AI Vision Service

1. Create a Computer Vision resource in the Azure Portal.
2. Choose S0 tier (for production; F0 is free but rate-limited).
3. Note the endpoint URL and API key (or use Managed Identity for security).
4. Enable "Image Analysis 4.0" in the resource settings (if not default).

Step 2: Store Images in Blob Storage

1. Create an Azure Storage Account and a container (e.g., product-images).
2. Upload images (e.g., sneakers.jpg, shirt.png).
3. Set container access level to "Blob" (so the API can read images via URL).

Step 3: Call the Analyze API (Python Example)

from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from azure.cognitiveservices.vision.computervision.models import VisualFeatureTypes
from msrest.authentication import CognitiveServicesCredentials

# Authenticate
endpoint = "https://<your-resource-name>.cognitiveservices.azure.com/"
key = "<your-api-key>"
client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(key))

# Image URL (from Blob Storage)
image_url = "https://<storage-account>.blob.core.windows.net/product-images/sneakers.jpg"

# Call Analyze API (get tags, captions, OCR)
features = [
    VisualFeatureTypes.tags,
    VisualFeatureTypes.description,  # Captions
    VisualFeatureTypes.denseCaptions,
    VisualFeatureTypes.objects,
    VisualFeatureTypes.read  # OCR
]

response = client.analyze_image(image_url, features)

# Extract results
print("Tags:", [tag.name for tag in response.tags])
print("Caption:", response.description.captions[0].text)
print("Dense Captions:", [caption.text for caption in response.dense_captions.dense_captions])
print("OCR Text:", [line.text for line in response.read.result.lines])

Step 4: Store Results in a Database

1. Use Azure Cosmos DB (NoSQL) or Azure SQL Database to store:
2. image_id (Blob Storage path)
3. tags (array)
4. caption (string)
5. ocr_text (string)
6. timestamp (for auditing)
7. Example Cosmos DB document:
   json
{
"id": "sneakers.jpg",
"tags": ["sneakers", "running shoes", "blue", "outdoor"],
"caption": "A pair of blue running shoes on a white background",
"ocr_text": ["Nike Air Zoom", "Size 10", "$129.99"],
"processed_at": "2024-05-20T12:00:00Z"
}

Step 5: Automate with Azure Functions

1. Create an Azure Function (Blob Trigger):
2. Trigger: When a new blob is added to the container.
3. Code: Call the Analyze API and store results in Cosmos DB.
4. Example function (Python):
   ```python
   import azure.functions as func
   from azure.storage.blob import BlobClient
   from azure.cognitiveservices.vision.computervision import ComputerVisionClient

def main(myblob: func.InputStream):
# Get image URL
blob_url = f"https://{myblob.name.split('/')[0]}.blob.core.windows.net/{myblob.name}"

   # Call Analyze API (same as Step 3)
   response = client.analyze_image(blob_url, features)

   # Store in Cosmos DB (use Azure SDK)
   document = {
       "id": myblob.name.split('/')[-1],
       "tags": [tag.name for tag in response.tags],
       "caption": response.description.captions[0].text,
       "ocr_text": [line.text for line in response.read.result.lines]
   }
   container.upsert_item(document)

```

Step 6: Build a Search Index (Optional)

1. Use Azure Cognitive Search to index the Cosmos DB data.
2. Enable semantic search for better caption/tag matching.
3. Example query: "Find all images with 'running shoes' and 'blue'".

Common Mistakes

Mistake 1: Using the Analyze API for Large Documents

• Problem: The Analyze API’s OCR is limited to 4MB images and single pages. For multi-page PDFs, candidates often try to loop through pages, which is slow and expensive.
• Correction: Use the Read API (for large documents) or Azure AI Document Intelligence (for structured forms). The Read API handles PDFs, TIFFs, and multi-page images efficiently.

Mistake 2: Not Handling Rate Limits

• Problem: The S0 tier has a 10 TPS (transactions per second) limit. Candidates forget to implement retry logic or batch processing, leading to throttling errors.
• Correction:
• Use exponential backoff (e.g., time.sleep(2 retry_count)).
• For batch jobs, parallelize requests (e.g., Azure Functions with Durable Functions).
• Monitor metrics in Azure Portal (e.g., "Throttled Calls").

Mistake 3: Confusing Analyze API with Custom Vision

• Problem: Candidates assume the Analyze API can detect custom objects (e.g., "my company’s logo"). It cannot—it only detects pre-trained objects (e.g., "car," "person").
• Correction:
• For custom objects, use Azure Custom Vision (train a model).
• For pre-built objects, use the Analyze API (no training needed).

Mistake 4: Ignoring OCR Language Support

• Problem: The Analyze API’s OCR defaults to English. Candidates forget to set language="fr" for French text, leading to poor accuracy.
• Correction:
• Specify the language parameter (e.g., language="es" for Spanish).
• For mixed-language documents, use language="unk" (auto-detect).

Mistake 5: Storing Raw API Responses Without Processing

• Problem: Candidates dump the entire JSON response into a database, making queries inefficient (e.g., searching for "blue shoes" in a nested tags array).
• Correction:
• Flatten the response (e.g., extract tags into a separate field).
• Use Cosmos DB’s array indexing for faster queries.

Certification Exam Insights

1. Service Selection Traps

• Analyze API vs. Read API vs. Document Intelligence:
• Analyze API: Best for general images (tags, captions, OCR for single-page images).
• Read API: Best for multi-page documents (PDFs, TIFFs).
• Document Intelligence: Best for structured forms (invoices, receipts, ID cards).

• Exam Trap: If the question mentions "tables" or "key-value pairs", pick Document Intelligence, not the Analyze API.

• Azure Cognitive Search vs. Analyze API:

• Cognitive Search is for indexing and querying images (e.g., "Find all images with 'dog'").
• Analyze API is for extracting data from images (e.g., "What’s in this image?").
• Exam Trap: If the question asks about "searching images by content", pick Cognitive Search. If it asks about "extracting text/tags", pick the Analyze API.

2. Key Constraints

• Image Size Limits:
• Analyze API: Max 4MB (resize larger images before sending).
• Read API: Max 50MB (for documents).
• Rate Limits:
• F0 (Free Tier): 20 calls/minute.
• S0 (Standard): 10 TPS (transactions per second).
• OCR Accuracy:
• Printed text: ~99% accuracy.
• Handwritten text: ~80-90% accuracy (use Document Intelligence for better results).

3. "Which Feature?" Scenarios

• Question: "A retail company wants to generate alt-text for 10,000 product images. Which Analyze API feature should they use?"
• Answer: Captions (not tags or dense captions, as alt-text needs a single sentence).
• Question: "A logistics company needs to extract text from shipping labels, which may contain rotated or skewed text. Which service should they use?"
• Answer: Analyze API OCR (handles rotated text; for structured labels, use Document Intelligence).

4. Cost Optimization

• Batch vs. Real-Time:
• Batch: Use Azure Functions + Blob Trigger for cost efficiency.
• Real-Time: Use API Management to cache frequent requests.
• OCR Costs:
• Analyze API OCR: $1.00 per 1,000 images.
• Read API: $1.50 per 1,000 pages (cheaper for multi-page docs).

Quick Check Questions

Question 1

A media company needs to automatically generate alt-text for 50,000 images in their CMS. The images are stored in Azure Blob Storage. Which Azure service should they use to process the images with minimal code? - A) Azure Custom Vision - B) Azure AI Vision Analyze API - C) Azure AI Document Intelligence - D) Azure Cognitive Search

Answer: B) Azure AI Vision Analyze API
Explanation: The Analyze API’s captions feature is designed for alt-text generation, and it can read directly from Blob Storage with no custom model training.

Question 2

A logistics company scans 10,000 shipping labels daily. The labels are multi-page PDFs with rotated text. Which Azure service will provide the most accurate OCR results with the least management overhead? - A) Azure AI Vision Analyze API - B) Azure AI Vision Read API - C) Azure AI Document Intelligence - D) Azure Form Recognizer (Legacy)

Answer: B) Azure AI Vision Read API
Explanation: The Read API is optimized for multi-page documents (PDFs, TIFFs) and handles rotated text better than the Analyze API. Document Intelligence is better for structured forms (e.g., invoices).

Question 3

A retail company wants to build a product search engine where users can find items by describing them (e.g., "red dress with flowers"). Which combination of Azure services should they use? - A) Azure AI Vision Analyze API + Azure Cognitive Search - B) Azure Custom Vision + Azure Blob Storage - C) Azure AI Document Intelligence + Azure SQL Database - D) Azure Form Recognizer + Azure Cosmos DB

Answer: A) Azure AI Vision Analyze API + Azure Cognitive Search
Explanation: The Analyze API extracts tags/captions, and Cognitive Search indexes them for semantic search. Custom Vision is overkill for this zero-shot scenario.

Last-Minute Cram Sheet

1. Analyze API = Zero-shot image analysis (tags, captions, OCR, objects). No training needed.
2. Read API = OCR for multi-page documents (PDFs, TIFFs). Use instead of Analyze API for large files.
3. Document Intelligence = Structured forms (invoices, receipts, ID cards). Not for general images.
4. Captions = Single sentence (alt-text). Dense captions = Multiple region descriptions.
5. OCR supports 164 languages—set language="fr" for French (default is English).
6. Max image size: 4MB (Analyze API), 50MB (Read API). Resize larger images first.
7. Rate limits: 10 TPS (S0 tier). Use exponential backoff for retries.
8. Cost: $1.00 per 1,000 images (Analyze API OCR), $1.50 per 1,000 pages (Read API).
9. ⚠️ Analyze API cannot detect custom objects—use Custom Vision for that.
10. ⚠️ Cognitive Search indexes images by content (tags/captions), but Analyze API extracts the content.