Cloud ML - Google Cloud Professional Machine Learning Engineer: Data Labeling and Quality (Vertex AI Data Labeling, CrowdCompute)

GCP_ML – Data Labeling and Quality (Vertex AI Data Labeling, CrowdCompute)

Google Cloud Professional Machine Learning Engineer Study Guide: Data Labeling and Quality (Vertex AI Data Labeling, CrowdCompute)

What This Is

Data labeling is the process of annotating raw data (images, text, audio, video) with meaningful tags or labels to train supervised ML models. High-quality labels are critical—garbage in, garbage out. Vertex AI Data Labeling and CrowdCompute (via Google Cloud’s human labeling workforce) automate and scale this process. Real-world scenario: A retail company wants to build a product recommendation system but lacks labeled images of its inventory. Using Vertex AI Data Labeling, they upload 100K product images, define labeling instructions (e.g., "Is this a shirt or pants?"), and use CrowdCompute to generate high-quality labels at scale—enabling accurate model training without manual effort.

Key Terms & Services

• Vertex AI Data Labeling: GCP’s managed service for creating labeled datasets using human labelers (via CrowdCompute) or custom ML models. Best for structured, semi-structured, and unstructured data (images, text, video, audio).
• CrowdCompute: Google’s human labeling workforce (via third-party vendors) integrated into Vertex AI Data Labeling. Scales labeling tasks to thousands of labelers with quality control (e.g., consensus scoring, expert reviews).
• Labeling Task: A job in Vertex AI Data Labeling that defines what to label (e.g., "Draw bounding boxes around cars in images") and how (human vs. model-assisted).
• Consensus Score: A metric in Vertex AI Data Labeling that measures agreement between multiple labelers. High consensus = higher label quality.
• Instruction Set: A document (PDF, HTML) provided to labelers explaining how to annotate data (e.g., "Label only visible faces, ignore blurry ones").
• Active Learning: A technique where the ML model identifies uncertain predictions and requests human labels for those samples, improving efficiency.
• Data Drift: When the distribution of input data changes over time, degrading model performance. Vertex AI Model Monitoring can detect drift in labeled data.
• Ground Truth: The "gold standard" labeled dataset used to train and evaluate models. Vertex AI Data Labeling helps generate this.
• Bounding Box: A rectangular annotation around an object in an image (e.g., for object detection).
• Semantic Segmentation: Pixel-level labeling of images (e.g., "This pixel is a road, this one is a car").
• Text Classification: Assigning categories to text (e.g., "spam" vs. "not spam").
• Entity Extraction: Identifying and labeling specific entities in text (e.g., "New York" as a location).

Step-by-Step / Process Flow

1. Set Up a Labeling Project in Vertex AI

• Action: Go to Vertex AI > Data Labeling in the GCP Console.
• Action: Click "Create Dataset" and select the data type (image, text, video, etc.).
• Action: Upload data from Cloud Storage (GCS) or BigQuery (for structured data).
• Action: Define the labeling task (e.g., "Classify sentiment in customer reviews" or "Draw bounding boxes around pedestrians").

2. Create an Instruction Set

• Action: Write clear, step-by-step instructions for labelers (e.g., "Label all visible license plates in images. Ignore blurry or obstructed plates.").
• Action: Upload the instruction set as a PDF or HTML file to GCS and link it in the labeling task.
• Action: Include examples of good/bad labels to reduce ambiguity.

3. Choose Labeling Method: Human vs. Model-Assisted

• Human Labeling (CrowdCompute):
• Best for complex or subjective tasks (e.g., sentiment analysis, medical imaging).
• Action: Select "Human Labeling" and configure:
  • Number of labelers per sample (e.g., 3 for consensus scoring).
  • Expert review (optional, for high-stakes use cases).
• Model-Assisted Labeling:
• Best for large datasets where a pre-trained model can speed up labeling (e.g., using AutoML Vision to pre-label images before human review).
• Action: Select "Model-Assisted Labeling" and choose a pre-trained model (e.g., AutoML Vision, custom TensorFlow model).

4. Run the Labeling Job

• Action: Submit the labeling task. Vertex AI distributes it to CrowdCompute labelers or runs the model.
• Action: Monitor progress in the Vertex AI Dashboard (track completed samples, consensus scores).
• Action: For human labeling, review disagreement samples (where labelers disagreed) and provide feedback.

5. Export Labeled Data

• Action: Once labeling is complete, export the dataset to:
• BigQuery (for structured data).
• GCS (for images, videos, or JSON/CSV files).
• Action: Use the labeled data to train a model in Vertex AI (AutoML, custom training) or export to TensorFlow/PyTorch.

6. Monitor Data Quality (Optional but Recommended)

• Action: Use Vertex AI Model Monitoring to detect label drift (e.g., if new data has different label distributions).
• Action: Set up alerts for data quality issues (e.g., sudden drop in consensus scores).

Common Mistakes

Mistake 1: Poor Instruction Sets

• Problem: Vague or ambiguous instructions (e.g., "Label all objects" without defining what counts as an object).
• Correction: Provide detailed examples, edge cases, and visual aids (e.g., "Label only fully visible cars. Ignore partial cars at the image edge."). Test instructions with a small batch first.

Mistake 2: Ignoring Consensus Scoring

• Problem: Using only one labeler per sample, leading to noisy labels.
• Correction: Use 3+ labelers per sample and set a consensus threshold (e.g., 70% agreement). Discard or re-label low-consensus samples.

Mistake 3: Not Using Model-Assisted Labeling for Large Datasets

• Problem: Manually labeling millions of images is slow and expensive.
• Correction: Use AutoML or a custom model to pre-label data, then have humans review uncertain samples (active learning).

Mistake 4: Storing Labeled Data in the Wrong Format

• Problem: Exporting labeled images as raw files without metadata (e.g., just the image, no bounding box coordinates).
• Correction: Export in standard formats (e.g., COCO JSON for object detection, CSV for text classification). Use TFRecords for TensorFlow pipelines.

Mistake 5: Not Monitoring for Label Drift

• Problem: Assuming labeled data stays relevant forever (e.g., product images change over time).
• Correction: Use Vertex AI Model Monitoring to detect drift in label distributions and trigger re-labeling.

Certification Exam Insights

1. Service Selection Traps

• Trap: "When should I use Vertex AI Data Labeling vs. AutoML?"
• Answer: Use Data Labeling when you need human-labeled data (e.g., subjective tasks like sentiment analysis). Use AutoML when you already have labeled data and want to train a model quickly.
• Trap: "Should I use CrowdCompute or a custom labeling team?"
• Answer: Use CrowdCompute for scalable, cost-effective labeling (e.g., 100K images). Use a custom team for highly specialized tasks (e.g., medical imaging with domain experts).

2. Key Constraints

• Vertex AI Data Labeling does not support real-time labeling (it’s batch-only).
• CrowdCompute has minimum pricing (e.g., $0.01 per image for simple tasks, but scales with complexity).
• Model-assisted labeling requires a pre-trained model (AutoML or custom).

3. "Which Service?" Scenarios

• Scenario: "A company needs to label 50K medical X-rays with bounding boxes for tumors. Which GCP service should they use?"
• Answer: Vertex AI Data Labeling with expert human labelers (medical data requires domain expertise; consensus scoring ensures accuracy).
• Scenario: "A startup wants to label 1M product images for an e-commerce site. They have a tight budget. What’s the best approach?"
• Answer: Model-assisted labeling with AutoML Vision (pre-label images, then have humans review uncertain samples to save costs).

4. Cost Optimization

• Exam Tip: Know that human labeling is more expensive than model-assisted labeling. Always prefer active learning (label uncertain samples first) to reduce costs.

Quick Check Questions

Question 1

A retail company wants to label 100K product images for an object detection model. They need high accuracy but have a limited budget. Which approach should they use? - A) Use Vertex AI Data Labeling with 5 human labelers per image.
- B) Use AutoML Vision to pre-label images, then have humans review uncertain samples.
- C) Manually label all images in-house.
- D) Use a third-party labeling tool outside GCP.

Answer: B (Model-assisted labeling with AutoML is cost-effective and scalable for large datasets.)

Question 2

A healthcare startup is labeling chest X-rays for pneumonia detection. They need FDA-compliant labels. What’s the best way to ensure label quality? - A) Use CrowdCompute with 3 labelers per image and a consensus threshold of 80%.
- B) Use a single expert radiologist to label all images.
- C) Use AutoML Vision to label all images automatically.
- D) Use Vertex AI Data Labeling with expert review for all samples.

Answer: D (Medical data requires expert review for compliance and accuracy.)

Question 3

A company notices that their sentiment analysis model’s performance is degrading over time. They suspect label drift. Which GCP service should they use to detect this? - A) Vertex AI Model Monitoring - B) Cloud Logging - C) BigQuery ML - D) AutoML Tables

Answer: A (Vertex AI Model Monitoring detects data drift, including label drift.)

Last-Minute Cram Sheet

1. Vertex AI Data Labeling = GCP’s managed service for human or model-assisted labeling.
2. CrowdCompute = Google’s human labeling workforce (scalable but not for specialized domains).
3. Consensus score = Measures agreement between labelers (aim for ≥70%).
4. Model-assisted labeling = Use AutoML or custom models to pre-label data before human review.
5. Active learning = Label uncertain samples first to improve efficiency.
6. Instruction sets must include examples, edge cases, and clear rules (test with a small batch first).
7. Export formats: COCO JSON (object detection), CSV (text), TFRecords (TensorFlow).
8. Vertex AI Model Monitoring detects label drift (set up alerts!).
9. ⚠️ Human labeling is expensive—always prefer model-assisted or active learning for large datasets.
10. ⚠️ Vertex AI Data Labeling does not support real-time labeling (it’s batch-only).