TECH **Python for Data Science: Selecting, Filtering, and Indexing (loc, iloc, Boolean Masks) – Zero-Fluff Study Guide**

Python for Data Science: Selecting, Filtering, and Indexing (loc, iloc, Boolean Masks) – Zero-Fluff Study Guide

1. What This Is & Why It Matters

You’re working on a real-world data pipeline—maybe cleaning a 10GB CSV from a client, debugging a legacy ETL script, or building a dashboard that filters millions of rows in real time. Selecting, filtering, and indexing are the core operations that let you extract, transform, and analyze data efficiently.

• Without these skills, you’ll waste hours writing slow loops, hardcoding row numbers, or accidentally overwriting data.
• With them, you can:
• Slice a DataFrame in milliseconds (no loops).
• Filter rows based on complex conditions (e.g., "all customers who spent >$100 in Q3").
• Modify subsets of data without breaking the original structure.
• Debug legacy code that uses .loc and .iloc inconsistently.

Real-world scenario:
You inherit a script that processes sales data. The original dev used hardcoded row numbers (df[10:20]) to extract a subset. Now, the data format changed, and the script silently fails—returning wrong results instead of crashing. Your job: Rewrite it using .loc and boolean masks so it works even if the data shifts.

2. Core Concepts & Components

1. .loc (Label-Based Selection)

• Definition: Selects data by row/column labels (index names or column names).
• Production insight: If your DataFrame’s index isn’t meaningful (e.g., default 0,1,2,...), .loc becomes unreliable. Always reset or set a meaningful index first.
• Example:
  python df.loc["row_label", "column_name"] # Single cell df.loc[["row1", "row2"], ["col1", "col2"]] # Multiple rows/columns

2. .iloc (Position-Based Selection)

• Definition: Selects data by integer position (like array indexing).
• Production insight: Never use .iloc in production pipelines—it breaks if the data order changes. Use it only for exploratory analysis or when you explicitly need position-based access.
• Example:
  python df.iloc[0, 0] # First row, first column df.iloc[1:5, 2:4] # Rows 1-4, columns 2-3

3. Boolean Masks (Conditional Filtering)

• Definition: A True/False array used to filter rows based on a condition.
• Production insight: Boolean masks are memory-efficient (no copies) and fast (vectorized operations). Use them instead of .apply() or loops.
• Example:
  python mask = df["sales"] > 100 # Returns True/False for each row df[mask] # Only rows where sales > 100

4. Chained Indexing (⚠️ Dangerous)

• Definition: Using multiple [] operations in a single line (e.g., df[df["col"] > 0]["new_col"] = 5).
• Production insight: Avoid this at all costs. It creates temporary copies, leading to silent failures (modifications won’t persist). Use .loc instead.
• Bad:
  python df[df["age"] > 30]["status"] = "senior" # ❌ Won't work!
• Good:
  python df.loc[df["age"] > 30, "status"] = "senior" # ✅ Correct

5. .at and .iat (Fast Scalar Access)

• Definition:
• .at: Label-based single-cell access (faster than .loc for scalars).
• .iat: Position-based single-cell access (faster than .iloc for scalars).
• Production insight: Use these only when you need a single value (e.g., df.at["row1", "col1"]). For multiple values, .loc/.iloc are better.

6. .query() (SQL-Like Filtering)

• Definition: Filters DataFrames using a string expression (similar to SQL WHERE).
• Production insight: Useful for dynamic filtering (e.g., user input in a dashboard). Slower than boolean masks but more readable for complex conditions.
• Example:
  python df.query("age > 30 & sales > 100") # Equivalent to df[(df["age"] > 30) & (df["sales"] > 100)]

7. .isin() (Membership Filtering)

• Definition: Checks if values are in a list/set (e.g., df["col"].isin([1, 2, 3])).
• Production insight: Faster than multiple OR conditions (e.g., df["col"] == 1 | df["col"] == 2). Use for categorical filtering.
• Example:
  python df[df["country"].isin(["USA", "Canada"])] # Only rows where country is USA or Canada

8. .where() (Conditional Replacement)

• Definition: Replaces values where a condition is False (keeps original shape).
• Production insight: Useful for conditional imputation (e.g., replacing NaN with 0 only in certain rows). Slower than boolean masks—use sparingly.
• Example:
  python df.where(df["sales"] > 100, 0) # Replace sales <= 100 with 0

3. Step-by-Step Hands-On: Filtering a Sales Dataset

Prerequisites

• Python 3.8+ with pandas installed (pip install pandas).
• A CSV file (sales_data.csv) with columns: ["date", "product", "region", "sales", "profit"].

Task:

You need to: 1. Load the data.
2. Filter rows where sales > 1000 and region == "West".
3. Update profit to profit * 1.1 for these rows (10% bonus).
4. Save the filtered data to a new CSV.

Step-by-Step Solution

1. Load the Data

import pandas as pd

df = pd.read_csv("sales_data.csv")
print(df.head())  # Verify data loaded correctly

2. Create a Boolean Mask

mask = (df["sales"] > 1000) & (df["region"] == "West")

3. Filter and Update Profit

# ✅ Correct: Uses .loc to modify in-place
df.loc[mask, "profit"] *= 1.1

4. Save the Filtered Data

filtered_df = df[mask]  # Only rows matching the condition
filtered_df.to_csv("west_high_sales.csv", index=False)

5. Verify the Output

print(filtered_df.head())  # Check first few rows
print(f"Total rows: {len(filtered_df)}")  # Should match expected count

4. ? Production-Ready Best Practices

Performance

• Use vectorized operations (boolean masks, .isin()) instead of loops or .apply().
• Avoid chained indexing (e.g., df[df["col"] > 0]["new_col"] = 5). It creates temporary copies and fails silently.
• Use .query() for dynamic filtering (e.g., in dashboards where users input conditions).

Maintainability

• Prefer .loc over .iloc—label-based selection is more robust if data order changes.
• Reset the index if it’s meaningless (e.g., df.reset_index(drop=True)).
• Use .copy() when creating subsets to avoid SettingWithCopyWarning: python subset = df[df["sales"] > 1000].copy() # ✅ Safe

Debugging

• Check for NaN values before filtering (they can break conditions): python df["sales"].isna().sum() # Count NaN values
• Use .info() to verify dtypes (e.g., df["date"] might be a string, not datetime).
• Log filter conditions in production pipelines: python print(f"Filtering {len(df)} rows with condition: sales > 1000")

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. .loc vs .iloc:
2. "Which method would you use to select rows where the index is 'A' or 'B'?"
   Answer: .loc (label-based).

3. "Which method would you use to select the first 5 rows, regardless of index?"
   Answer: .iloc (position-based).

4. Boolean masks:

5. "How would you filter rows where sales > 100 and region == "West"?"
   Answer: df[(df["sales"] > 100) & (df["region"] == "West")].

6. Chained indexing trap:

7. "What’s wrong with this code: df[df["age"] > 30]["status"] = "senior"?"
   Answer: It creates a temporary copy and fails silently. Use .loc instead.

8. Performance:

9. "Which is faster: df[df["col"].isin([1, 2, 3])] or df[(df["col"] == 1) | (df["col"] == 2) | (df["col"] == 3)]?"
   Answer: .isin() is faster and more readable.

Key ⚠️ Trap Distinctions

• .loc vs .iloc:
• .loc → labels (can be strings, dates, etc.).
• .iloc → positions (integers only).
• & vs and:
• & → element-wise (for boolean masks).
• and → scalar (for if conditions).
• .where() vs boolean masks:
• .where() → keeps original shape (replaces False with NaN).
• Boolean masks → drops non-matching rows.

7. ? Hands-On Challenge (with Solution)

Challenge:

You have a DataFrame df with columns ["name", "age", "score"]. Write a one-liner to: 1. Filter rows where age > 25 and score > 80.
2. Update the score column to score * 1.05 for these rows.

Solution:

df.loc[(df["age"] > 25) & (df["score"] > 80), "score"] *= 1.05

Why It Works:

• .loc selects both rows and columns in one operation.
• The boolean mask (df["age"] > 25) & (df["score"] > 80) filters rows.
• score *= 1.05 updates the selected subset in-place.

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Pandas User Guide – Indexing and Selecting Data (Official docs, best for deep dives).
2. 10 Minutes to Pandas (Quick start for beginners).
3. Real Python – Pandas .loc vs .iloc (Great for visual learners).
4. Pandas Cookbook (Practical examples for real-world tasks).