Cloud ML - Google Cloud Professional Machine Learning Engineer: Networking and Security (VPC Service Controls, CMEK, Workload Identity)

GCP_ML – Networking and Security (VPC Service Controls, CMEK, Workload Identity)

Google Cloud Professional Machine Learning Engineer Study Guide: Networking and Security (VPC Service Controls, CMEK, Workload Identity)

What This Is

This topic covers how to secure ML workloads on Google Cloud by controlling network access, encrypting data at rest, and managing identities for services. Imagine a healthcare startup training a Vertex AI model on sensitive patient records. They need to: - Block unauthorized access to training data stored in Cloud Storage (VPC Service Controls).
- Ensure encryption keys are managed by their security team (CMEK).
- Let Vertex AI Pipelines access BigQuery without hardcoding service account keys (Workload Identity).
Without these, data leaks, compliance violations, or credential theft could derail the project.

Key Terms & Services

• VPC Service Controls (VPC-SC): GCP’s perimeter security for managed services (e.g., BigQuery, Cloud Storage). Blocks data exfiltration by restricting access to a private network, even if IAM permissions are misconfigured.
• Customer-Managed Encryption Keys (CMEK): Lets you control encryption keys for GCP services (e.g., Cloud Storage, BigQuery) via Cloud Key Management Service (KMS). Required for compliance (HIPAA, GDPR) or internal security policies.
• Workload Identity: Securely grants GCP services (e.g., Vertex AI, Cloud Run) access to other GCP resources (e.g., BigQuery) using short-lived credentials tied to a Google Service Account (GSA). Replaces static service account keys.
• Private Service Connect: Enables private access to GCP services (e.g., Vertex AI, BigQuery) from your VPC without exposing them to the public internet.
• Cloud KMS: GCP’s key management service for creating, rotating, and managing encryption keys (used with CMEK).
• Service Perimeter: A VPC-SC boundary that defines which GCP services and projects can communicate. Can include "access levels" (e.g., only allow requests from corporate IPs).
• IAM Conditions: Lets you restrict IAM permissions based on attributes (e.g., "only allow access from a specific VPC").
• Data Loss Prevention (DLP) API: Scans data for sensitive info (e.g., PII) before it’s processed by ML models. Often used with VPC-SC to prevent leaks.
• Confidential Computing: Encrypts data in use (e.g., during model training) using hardware-based secure enclaves (e.g., Confidential VMs).
• Shared VPC: Allows multiple GCP projects to share a single VPC network, useful for centralized security management.

Step-by-Step / Process Flow

1. Secure Data Access with VPC Service Controls

Scenario: A fintech company wants to block unauthorized access to BigQuery datasets containing transaction records.
1. Create a service perimeter in VPC Service Controls:
- Go to Security → VPC Service Controls in the GCP Console.
- Click Create Perimeter and select the projects/services to protect (e.g., bigquery.googleapis.com).
2. Define access levels (optional):
- Restrict access to corporate IPs or specific VPCs using Access Levels (e.g., corp-ip-range).
3. Test the perimeter:
- Try accessing BigQuery from an unauthorized VPC or IP. The request should be blocked with an error like PERMISSION_DENIED: VPC Service Controls.
4. Add exceptions (if needed):
- Use Ingress/Egress Rules to allow specific projects or services (e.g., let Vertex AI access BigQuery).

2. Encrypt Data with CMEK

Scenario: A healthcare provider needs to encrypt Cloud Storage buckets storing patient X-rays with their own keys.
1. Create a key ring and key in Cloud KMS:
- Go to Security → Key Management and create a key ring (e.g., ml-keys) and a key (e.g., patient-data-key).
- Set rotation period (e.g., 90 days) for compliance.
2. Grant permissions:
- Assign the Cloud Storage Service Agent (service-[PROJECT_NUMBER]@gs-project-accounts.iam.gserviceaccount.com) the cloudkms.cryptoKeyEncrypterDecrypter role on the key.
3. Enable CMEK on the bucket:
- In Cloud Storage, create a bucket and select Customer-Managed Key under encryption.
- Choose the KMS key you created.
4. Verify encryption:
- Upload a file and check its encryption status with:
bash
gsutil ls -L gs://your-bucket-name/your-file
- Look for KMS key: projects/your-project/locations/global/keyRings/ml-keys/cryptoKeys/patient-data-key.

3. Use Workload Identity for Secure Service Access

Scenario: A Vertex AI Pipeline needs to read data from BigQuery without storing service account keys.
1. Create a Google Service Account (GSA):
- Go to IAM & Admin → Service Accounts and create a GSA (e.g., [email protected]).
- Grant it the BigQuery Data Viewer role.
2. Enable Workload Identity on the GKE cluster (if using Vertex AI Pipelines on GKE):
- Edit the cluster and enable Workload Identity under Security.
3. Bind the GSA to a Kubernetes Service Account (KSA):
- Create a KSA in your GKE namespace (e.g., vertex-ai-ksa).
- Bind the GSA to the KSA with:
bash
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:your-project.svc.id.goog[default/vertex-ai-ksa]" \
[email protected] 4. Annotate the KSA:
- Add the annotation to the KSA:
yaml
metadata:
annotations:
iam.gke.io/gcp-service-account: [email protected] 5. Test access:
- Deploy a pod using the KSA and verify it can query BigQuery without explicit key files.

Common Mistakes

Mistake 1: Assuming IAM alone is enough for security

• Correction: IAM controls who can access a resource, but VPC Service Controls control where they can access it from. Use both to prevent data exfiltration (e.g., a compromised IAM user downloading data to an external IP).

Mistake 2: Forgetting to grant KMS permissions to the service agent

• Correction: When enabling CMEK, you must grant the Cloud Storage Service Agent (or other service’s agent) the cloudkms.cryptoKeyEncrypterDecrypter role on the key. Without this, the service can’t encrypt/decrypt data.

Mistake 3: Using static service account keys instead of Workload Identity

• Correction: Static keys are a security risk (they can be leaked). Workload Identity provides short-lived credentials and is the recommended way to grant GCP services access to other resources.

Mistake 4: Not testing VPC-SC perimeters

• Correction: Always test your VPC-SC setup by attempting to access protected services from outside the perimeter (e.g., from a VM in a different VPC). Misconfigurations can silently fail.

Mistake 5: Overlooking DLP for sensitive data

• Correction: Before training models on sensitive data (e.g., PII), scan it with DLP API to identify and redact sensitive info. Combine with VPC-SC to prevent leaks.

Certification Exam Insights

1. VPC-SC vs. Private Service Connect vs. IAM Conditions

• VPC-SC: Blocks data exfiltration at the network level (e.g., prevent BigQuery data from being copied to an external bucket).
• Private Service Connect: Enables private access to GCP services (e.g., call Vertex AI from your VPC without public IPs).
• IAM Conditions: Restrict access based on attributes (e.g., "only allow access from a specific VPC").
• Trap: The exam may ask which to use for "blocking data exfiltration" (VPC-SC) vs. "enabling private access" (Private Service Connect).

2. CMEK vs. Google-Managed Keys

• CMEK: You control the keys (required for compliance, key rotation, or audit trails).
• Google-Managed Keys: GCP handles keys (simpler, but less control).
• Trap: The exam may ask which to use for "HIPAA compliance" (CMEK) or "lowest operational overhead" (Google-managed).

3. Workload Identity vs. Service Account Keys

• Workload Identity: Secure, short-lived credentials for GKE or Cloud Run.
• Service Account Keys: Static keys (risky, deprecated for most use cases).
• Trap: The exam may ask which to use for "securely granting Vertex AI access to BigQuery" (Workload Identity).

4. Confidential Computing for ML

• When to use: Encrypting data in use (e.g., training models on sensitive data in Confidential VMs).
• Trap: The exam may ask if Confidential Computing is needed for "encrypting data at rest" (no, that’s CMEK) or "encrypting data in transit" (no, that’s TLS).

Quick Check Questions

Question 1

A financial services company wants to train a Vertex AI model on transaction data stored in BigQuery. They need to: - Block unauthorized access to the data, even if IAM permissions are misconfigured.
- Ensure the data is encrypted with keys managed by their security team.
Which two GCP services should they use? Answer: VPC Service Controls (to block unauthorized access) and CMEK (to encrypt data with customer-managed keys).
Explanation: VPC-SC prevents data exfiltration at the network level, while CMEK provides control over encryption keys.

Question 2

A Vertex AI Pipeline running on GKE needs to read data from BigQuery. The security team prohibits storing service account keys in the cluster. What should you use? Answer: Workload Identity.
Explanation: Workload Identity securely grants the GKE pods access to BigQuery using short-lived credentials, without static keys.

Question 3

A healthcare startup is deploying a Vertex AI Endpoint for a model that processes patient records. They need to ensure: - The model’s predictions are encrypted in transit.
- The training data is encrypted at rest with keys they control.
- The endpoint is only accessible from their corporate network.
Which three GCP features should they configure? Answer:
1. TLS (for encrypting predictions in transit).
2. CMEK (for encrypting training data at rest with customer-managed keys).
3. Private Service Connect (to restrict endpoint access to their corporate network).
Explanation: TLS encrypts traffic, CMEK encrypts data at rest, and Private Service Connect enables private access.

Last-Minute Cram Sheet

1. VPC Service Controls (VPC-SC) blocks data exfiltration at the network level (e.g., prevent BigQuery data from being copied to an external bucket).
2. CMEK lets you control encryption keys for GCP services (required for compliance like HIPAA/GDPR).
3. Workload Identity replaces static service account keys for secure access between GCP services (e.g., Vertex AI → BigQuery).
4. Private Service Connect enables private access to GCP services (e.g., call Vertex AI from your VPC without public IPs).
5. Cloud KMS is where you create and manage CMEK keys (set rotation periods for compliance).
6. Service Perimeter defines which GCP services/projects can communicate in VPC-SC (e.g., allow Vertex AI to access BigQuery).
7. IAM Conditions restrict permissions based on attributes (e.g., "only allow access from a specific VPC").
8. DLP API scans data for sensitive info (e.g., PII) before ML processing.
9. Confidential Computing encrypts data in use (e.g., training models on sensitive data in Confidential VMs).
10. ⚠️ VPC-SC does not replace IAM – it’s an additional layer of security. Always use both!
11. ⚠️ CMEK requires granting KMS permissions to the service agent (e.g., Cloud Storage Service Agent).
12. ⚠️ Workload Identity is for GKE/Cloud Run – not for Compute Engine VMs (use OS Login or metadata-based keys instead).