Cloud ML - Azure AI Engineer Associate (Exam AI-102): Containerizing AI Solutions (Azure Container Instances, Kubernetes)

Azure_AI – Containerizing AI Solutions (Azure Container Instances, Kubernetes)

Azure AI-102 Study Guide: Containerizing AI Solutions (ACI & AKS)

What This Is

Containerizing AI solutions means packaging your trained models, inference code, and dependencies into lightweight, portable containers (Docker) that can run anywhere—on-prem, in the cloud, or at the edge. This is critical for scalable, reproducible, and secure ML deployments. For example: - A healthcare startup deploys a real-time X-ray classification model behind a secure API using Azure Kubernetes Service (AKS) for high availability and auto-scaling.
- A retail company runs batch inference on millions of product images using Azure Container Instances (ACI) for cost-efficient, serverless compute.
- A financial services firm deploys a fraud detection model to Azure Kubernetes Service (AKS) with GPU acceleration for low-latency predictions.

Key Terms & Services

• Azure Container Instances (ACI):
  Serverless containers for quick, low-cost, and ephemeral workloads (e.g., batch inference, testing). No orchestration—just deploy and run.

• Azure Kubernetes Service (AKS):
  Managed Kubernetes for scalable, production-grade ML deployments. Supports auto-scaling, GPU acceleration, and multi-region deployments.

• Docker:
  Industry-standard tool for building, packaging, and running containerized applications. Used to create images for ACI/AKS.

• Azure Container Registry (ACR):
  Private Docker registry for storing and managing container images. Required for deploying to ACI/AKS.

• Kubernetes (K8s):
  Open-source orchestration platform for automating deployment, scaling, and management of containerized apps. AKS is Azure’s managed K8s.

• Helm:
  Package manager for Kubernetes. Simplifies deploying complex apps (e.g., ML models + APIs) via charts (pre-configured templates).

• Azure Machine Learning (AML) Managed Endpoints:
  Fully managed inference endpoints that can automatically deploy to ACI or AKS based on traffic needs.

• GPU Acceleration in AKS:
  Enables high-performance inference for deep learning models (e.g., CNNs, LLMs) by attaching GPU nodes to AKS clusters.

• Persistent Volumes (PV) in AKS:
  Storage that persists beyond the lifecycle of a pod. Critical for stateful ML apps (e.g., feature stores, model checkpoints).

• Ingress Controllers (e.g., NGINX, Application Gateway):
  Manages external access to AKS services. Required for exposing ML APIs securely (HTTPS, load balancing).

• Azure DevOps / GitHub Actions:
  CI/CD pipelines for automating container builds and deployments to ACI/AKS.

• Azure Arc-enabled Kubernetes:
  Extends AKS management to hybrid/multi-cloud environments (e.g., on-prem + Azure).

Step-by-Step / Process Flow

1. Build & Push a Docker Image for Your ML Model

• Write a Dockerfile (e.g., Python + FastAPI + ONNX runtime for inference).
• Build the image: bash docker build -t my-ml-model:latest .
• Push to Azure Container Registry (ACR): bash az acr login --name <acr-name> docker tag my-ml-model:latest <acr-name>.azurecr.io/my-ml-model:latest docker push <acr-name>.azurecr.io/my-ml-model:latest

2. Deploy to Azure Container Instances (ACI) for Testing/Batch

• Run a container in ACI: bash az container create --resource-group <rg> --name my-aci \
--image <acr-name>.azurecr.io/my-ml-model:latest \
--cpu 2 --memory 4 --ports 8000 \
--registry-login-server <acr-name>.azurecr.io \
--registry-username <acr-username> \
--registry-password <acr-password>
• Test the endpoint (e.g., curl http://<aci-ip>:8000/predict).

3. Deploy to Azure Kubernetes Service (AKS) for Production

• Create an AKS cluster (with GPU if needed): bash az aks create --resource-group <rg> --name my-aks \
--node-count 3 --node-vm-size Standard_DS2_v2 \
--enable-addons monitoring --generate-ssh-keys
• Connect to the cluster: bash az aks get-credentials --resource-group <rg> --name my-aks
• Deploy using kubectl or Helm: bash kubectl create deployment my-ml-model --image=<acr-name>.azurecr.io/my-ml-model:latest kubectl expose deployment my-ml-model --port=8000 --type=LoadBalancer
• Scale horizontally (e.g., 5 replicas): bash kubectl scale deployment my-ml-model --replicas=5

4. Secure & Monitor the Deployment

• Enable HTTPS using an Ingress Controller (e.g., NGINX + Let’s Encrypt).
• Set up auto-scaling (HPA for CPU/memory, KEDA for event-driven scaling).
• Monitor with Azure Monitor + Prometheus/Grafana.

5. Automate with CI/CD (Optional)

• GitHub Actions workflow to build, push, and deploy on code changes.
• Azure DevOps pipeline for approval-based deployments.

Common Mistakes

Mistake 1: Using ACI for High-Traffic Production Workloads

• Correction: ACI is not designed for long-running, high-traffic workloads. Use AKS instead for auto-scaling, load balancing, and GPU support.

Mistake 2: Forgetting to Attach ACR to AKS

• Correction: AKS needs explicit permissions to pull images from ACR. Run: bash az aks update -n my-aks -g <rg> --attach-acr <acr-name>

Mistake 3: Not Configuring Resource Limits in Kubernetes

• Correction: Always set CPU/memory limits in Kubernetes manifests to prevent pods from starving other workloads: yaml resources:
limits:
cpu: "2"
memory: "4Gi"

Mistake 4: Ignoring Persistent Storage for Stateful ML Apps

• Correction: Use Azure Disk or Azure Files for persistent volumes (e.g., model checkpoints, feature stores).

Mistake 5: Exposing AKS Services Without Ingress

• Correction: Never expose services directly via LoadBalancer. Use an Ingress Controller for HTTPS, path-based routing, and security.

Certification Exam Insights

1. ACI vs. AKS: When to Use Which?

• ACI: Best for short-lived, low-cost, serverless workloads (e.g., batch inference, testing).
• AKS: Best for production, high-traffic, GPU-accelerated workloads (e.g., real-time APIs, auto-scaling).

2. Tricky Service-Selection Traps

• Azure ML Managed Endpoints vs. ACI/AKS:
• Managed endpoints automatically deploy to ACI (dev) or AKS (prod) based on traffic.
• Exam trap: If the question mentions "fully managed," pick Azure ML Managed Endpoints over raw ACI/AKS.
• AKS vs. Azure Functions for ML:
• AKS for long-running, stateful ML workloads.
• Azure Functions for event-driven, stateless inference (e.g., triggered by Blob Storage).

3. Key Constraints to Remember

• ACI limits: Max 4 vCPUs, 16GB RAM per container. No auto-scaling.
• AKS GPU nodes: Require specific VM sizes (e.g., Standard_NC6 for NVIDIA GPUs).
• ACR geo-replication: Needed for multi-region AKS deployments.

4. Helm vs. Raw kubectl

• Helm is preferred for complex deployments (e.g., ML models + APIs + monitoring).
• kubectl is fine for simple deployments (e.g., a single model API).

Quick Check Questions

1. A fintech company needs to deploy a fraud detection model that processes 10,000 transactions per second with sub-100ms latency. Which Azure service should they use?

✅ Answer: Azure Kubernetes Service (AKS) with GPU nodes and auto-scaling.
✅ Why: AKS supports high throughput, low latency, and GPU acceleration, while ACI lacks auto-scaling and GPU support.

2. A data scientist wants to test a new model in a container without managing infrastructure. Which Azure service offers the fastest setup?

✅ Answer: Azure Container Instances (ACI).
✅ Why: ACI is serverless and ephemeral, ideal for quick testing. AKS is overkill for this use case.

3. A retail company deploys a recommendation model to AKS but notices pods crash when traffic spikes. What’s the most likely cause?

✅ Answer: Missing CPU/memory limits in the Kubernetes deployment.
✅ Why: Without resource limits, pods can consume all node resources, causing crashes. Always set limits in the manifest.

Last-Minute Cram Sheet

1. ACI = Serverless, ephemeral, low-cost (max 4 vCPUs, 16GB RAM). ⚠️ Not for production.
2. AKS = Production-grade, auto-scaling, GPU support (use for high-traffic ML APIs).
3. ACR = Private Docker registry (required for ACI/AKS deployments).
4. Helm = Package manager for Kubernetes (simplifies complex deployments).
5. Ingress Controller = Required for HTTPS/load balancing in AKS (e.g., NGINX).
6. Azure ML Managed Endpoints auto-deploys to ACI (dev) or AKS (prod).
7. AKS GPU nodes need specific VM sizes (e.g., Standard_NC6).
8. Always set CPU/memory limits in Kubernetes to avoid resource starvation.
9. ACI has no auto-scaling—use AKS for dynamic workloads.
10. ⚠️ AKS requires ACR permissions (az aks update --attach-acr).