TECH **Python for Data Science: Strings, Dates, and Regular Expressions (re) – Zero-Fluff Study Guide**

Python for Data Science: Strings, Dates, and Regular Expressions (re) – Zero-Fluff Study Guide

1. What This Is & Why It Matters

You’re cleaning a dataset, and half the "dates" are in MM/DD/YYYY, some in DD-MM-YY, and a few in YYYYMMDD. A column that should be numeric has "$1,200" and "1.2K" mixed in. A text field contains email addresses, but some are malformed ("[email protected]"). If you don’t handle these inconsistencies, your analysis will fail—silently.

This guide covers:
- Strings: The raw material of text data—slicing, formatting, and cleaning.
- Dates: Parsing, converting, and manipulating time data (critical for time-series analysis).
- Regular Expressions (re): The Swiss Army knife for pattern matching—extracting emails, phone numbers, or structured data from messy text.

Why this matters in production:
- Data pipelines break if you assume uniform formatting (e.g., pd.to_datetime() fails on "2023/13/01").
- Regex is 10x faster than manual string splitting for large datasets (e.g., extracting domains from 1M URLs).
- Time zones and daylight savings can skew your analysis if ignored (e.g., "Why does my daily sales report show two entries for March 12, 2023?").

Real-world scenario:
You’re building a customer analytics dashboard. The input CSV has: - A "joined_date" column with mixed formats ("Jan 5, 2020", "05-01-20", "20200105").
- A "spend" column with "$1,200", "1.2K", and "500".
- A "notes" field containing email addresses and phone numbers you need to extract.

Your job: Clean this data before feeding it to pandas or a machine learning model.

2. Core Concepts & Components

Strings

• str methods: Built-in functions like .split(), .strip(), .replace(), .lower().
  Production insight: .strip() is your first line of defense against whitespace errors (e.g., " [email protected] " vs. "[email protected]").
• String formatting (f-strings): f"User {name} joined on {date}" is cleaner and faster than + concatenation.
  Production insight: F-strings are 30% faster than .format() for logging and debugging.
• String slicing: text[0:5] extracts the first 5 characters.
  Production insight: Negative indices (text[-5:]) are great for extracting file extensions (e.g., "data.csv"[-3:] → "csv").

Dates & Time

• datetime module: datetime.datetime, datetime.date, datetime.timedelta.
  Production insight: Always store dates in UTC to avoid timezone hell (e.g., daylight savings time).
• strftime/strptime: Convert between strings and datetime objects.
  Production insight: pd.to_datetime() uses strptime under the hood—knowing the format codes (%Y-%m-%d) saves you hours of debugging.
• Time deltas: timedelta(days=1) for date arithmetic (e.g., "What was the date 7 days ago?").
  Production insight: Time deltas are critical for rolling windows in time-series analysis (e.g., "7-day moving average").

Regular Expressions (re)

• re.compile(): Pre-compile patterns for 10x speedup in loops.
  Production insight: Compiling a regex once and reusing it is faster than recompiling in every iteration.
• Common patterns:
• \d = digit, \w = word character, \s = whitespace.
• ^ = start of string, $ = end of string.
  Production insight: ^ and $ prevent partial matches (e.g., r"\d{3}" matches "123" in "a123b", but r"^\d{3}$" does not).
• re.search() vs. re.match() vs. re.findall():
• search(): Finds the first match anywhere in the string.
• match(): Only matches at the start of the string.
• findall(): Returns all matches as a list.
  Production insight: findall() is your go-to for extracting multiple emails/phone numbers from text.

3. Step-by-Step Hands-On: Cleaning a Messy Dataset

Prerequisites

• Python 3.8+ (for f-strings and datetime improvements).
• pandas installed (pip install pandas).
• A sample CSV (messy_data.csv) with: csv user_id,joined_date,spend,notes 1,Jan 5, 2020,$1,200,"Email: [email protected], Phone: 555-1234" 2,05-01-20,1.2K,"Email: [email protected], Phone: 555-5678" 3,20200105,500,"Email: [email protected], Phone: 555-9012"

Step 1: Load the Data

import pandas as pd

df = pd.read_csv("messy_data.csv")
print(df.head())

Output:

   user_id joined_date   spend                                notes
0        1  Jan 5, 2020  $1,200  Email: [email protected], Phone: 555-1234
1        2     05-01-20    1.2K  Email: [email protected], Phone: 555-5678
2        3    20200105     500   Email: [email protected], Phone: 555-9012

Step 2: Clean the joined_date Column

Problem: Mixed formats ("Jan 5, 2020", "05-01-20", "20200105").

Solution: Use pd.to_datetime() with errors="coerce" to handle failures.

df["joined_date"] = pd.to_datetime(
    df["joined_date"],
    format="mixed",  # Auto-detects formats (Python 3.11+)
    errors="coerce"   # Invalid dates become NaT (Not a Time)
)
print(df["joined_date"])

Output:

0   2020-01-05
1   2020-05-01
2   2020-01-05
Name: joined_date, dtype: datetime64[ns]

Why this works:
- format="mixed" (Python 3.11+) tries multiple formats automatically.
- errors="coerce" prevents crashes on invalid dates (e.g., "2023/13/01" becomes NaT).

Step 3: Clean the spend Column

Problem: "$1,200", "1.2K", "500" need to be converted to floats.

Solution: Use regex to extract numbers and handle "K" suffixes.

import re

def clean_spend(value):
    # Remove $ and commas
    value = re.sub(r"[$,]", "", str(value))
    # Handle "1.2K" -> 1200
    if "K" in value:
        return float(value.replace("K", "")) * 1000
    return float(value)

df["spend"] = df["spend"].apply(clean_spend)
print(df["spend"])

Output:

0    1200.0
1    1200.0
2     500.0
Name: spend, dtype: float64

Why this works:
- re.sub(r"[$,]", "", value) removes $ and , in one pass.
- "K" suffixes are converted to thousands (e.g., "1.2K" → 1200).

Step 4: Extract Emails and Phone Numbers from notes

Problem: Extract all valid emails and phone numbers from the notes column.

Solution: Use regex with re.findall().

# Extract emails
df["email"] = df["notes"].str.findall(r"[\w\.-]+@[\w\.-]+\.\w+")
# Extract phone numbers (US format: 555-1234 or 555-123-4567)
df["phone"] = df["notes"].str.findall(r"\d{3}-\d{3,4}")

print(df[["email", "phone"]])

Output:

                     email          phone
0  [[email protected]]     [555-1234]
1                 []     [555-5678]
2   [[email protected]]     [555-9012]

Why this works:
- Email regex: [\w\.-]+@[\w\.-]+\.\w+ matches [email protected].
- Phone regex: \d{3}-\d{3,4} matches 555-1234 or 555-123-4567.
- str.findall() returns a list of all matches (empty if none found).

Step 5: Save the Cleaned Data

df.to_csv("cleaned_data.csv", index=False)

4. ? Production-Ready Best Practices

Strings

• Use .strip() early: Trim whitespace before processing (e.g., df["column"] = df["column"].str.strip()).
• Avoid + for concatenation: Use f-strings (f"{var1}{var2}") or .join() for performance.
• Case consistency: Normalize case with .lower() or .upper() before comparisons (e.g., "USA" vs. "usa").

Dates

• Store in UTC: Always convert to UTC before saving to a database (e.g., df["date"] = df["date"].dt.tz_localize("UTC")).
• Use pd.to_datetime() with errors="coerce": Prevents crashes on invalid dates.
• Time deltas for rolling windows: Use timedelta(days=7) for 7-day moving averages.

Regex

• Pre-compile patterns: pattern = re.compile(r"\d{3}-\d{3}") is 10x faster in loops.
• Test regex online: Use regex101.com to debug patterns.
• Avoid greedy matches: Use .*? instead of .* to prevent over-matching (e.g., r"<.*?>" matches <tag> instead of <tag>content</tag>).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. String manipulation:

2. "How do you extract the domain from an email ([email protected] → example.com)?"
   Answer: email.split("@")[-1] or re.search(r"@(\w+\.\w+)", email).group(1).

3. Date parsing:

4. "Convert "01/02/2023" to a datetime object."
   Answer: pd.to_datetime("01/02/2023", format="%m/%d/%Y") (note: %d/%m/%Y is a common trap).

5. Regex:

6. "Write a regex to match US phone numbers (e.g., 555-123-4567)."
   Answer: r"\d{3}-\d{3}-\d{4}".

Key Trap Distinctions

• re.match() vs. re.search():
• match() only checks the start of the string.
• search() checks the entire string.
• Greedy vs. non-greedy regex:
• .* (greedy) matches as much as possible.
• .*? (non-greedy) matches as little as possible.
• Timezone awareness:
• datetime.now() is timezone-naive.
• datetime.now(timezone.utc) is timezone-aware.

7. ? Hands-On Challenge (with Solution)

Challenge:
Extract all valid email addresses from the following string:

text = "Contact us at [email protected] or [email protected]. Invalid: [email protected], @domain.com"

Solution:

import re

emails = re.findall(r"[\w\.-]+@[\w\.-]+\.\w+", text)
print(emails)  # Output: ['[email protected]', '[email protected]']

Why it works:
- [\w\.-]+ matches the username (letters, numbers, ., -).
- @[\w\.-]+ matches the domain (e.g., @example).
- \.\w+ matches the TLD (e.g., .com).

8. ? Rapid-Reference Crib Sheet

Strings

• text.strip() → Remove leading/trailing whitespace.
• text.split(",") → Split on commas.
• f"Value: {var}" → Fast string formatting.
• text[0:5] → Slice first 5 characters.
• text.lower() → Convert to lowercase.

Dates

• pd.to_datetime("2023-01-01") → Convert string to datetime.
• df["date"].dt.strftime("%Y-%m-%d") → Format as string.
• datetime.now(timezone.utc) → Current time in UTC.
• timedelta(days=1) → Add/subtract days.

Regex

• re.compile(r"\d{3}") → Pre-compile for speed.
• re.findall(r"\w+", text) → Extract all words.
• re.sub(r"\s+", " ", text) → Replace multiple spaces with one.
• ^ → Start of string, $ → End of string.
• \d → Digit, \w → Word character, \s → Whitespace.

⚠️ Exam Traps:
- re.match() only checks the start of the string.
- pd.to_datetime() defaults to errors="raise" (use errors="coerce" to avoid crashes).
- Timezone-naive datetime objects cause silent bugs (always use UTC).

9. ? Where to Go Next

1. Python datetime docs – Official reference for strftime/strptime format codes.
2. Regex101 – Interactive regex tester with explanations.
3. Pandas to_datetime() docs – Advanced date parsing options.
4. Book: "Automate the Boring Stuff with Python" (Chapter 7 – Regex) – Practical regex examples.