TECH **Python for Data Science: Summary Statistics, Correlations, and Covariance**

Python for Data Science: Summary Statistics, Correlations, and Covariance

A Hyper-Practical, Zero-Fluff Study Guide

1. What This Is & Why It Matters

You’re handed a dataset—maybe sales records, sensor logs, or customer behavior—and your boss says, "Tell me what’s going on in 10 minutes." Summary statistics, correlations, and covariance are your first line of defense against "data blindness." They let you: - Spot outliers (e.g., a single store skewing monthly revenue).
- Detect relationships (e.g., "Does ad spend actually drive sales?").
- Validate assumptions (e.g., "Are these two features redundant?").

Real-world scenario:
You inherit a legacy ML pipeline predicting customer churn. The model’s accuracy is 92%, but the business says it’s useless. Why? The dataset has leaky features (e.g., "days_since_last_purchase" correlates too strongly with "churn" because it’s derived from the target). Summary stats would’ve caught this before deployment.

What breaks if you ignore this?
- Garbage-in, garbage-out models (e.g., training on correlated features inflates accuracy).
- Wasted compute (e.g., running expensive deep learning on data with no signal).
- Wrong business decisions (e.g., "We’ll double ad spend!" based on a spurious correlation).

2. Core Concepts & Components

? Summary Statistics

• Mean (df.mean()): Average value. Production insight: Sensitive to outliers—use median for skewed data (e.g., salaries).
• Median (df.median()): Middle value. Production insight: Robust to outliers; critical for financial data (e.g., housing prices).
• Standard Deviation (df.std()): Spread of data. Production insight: High std dev = noisy data; consider normalization.
• Variance (df.var()): Squared std dev. Production insight: Used in ML loss functions (e.g., MSE).
• Min/Max (df.min(), df.max()): Range of values. Production insight: Check for impossible values (e.g., negative ages).
• Quantiles (df.quantile([0.25, 0.5, 0.75])): Percentiles. Production insight: Used in box plots to detect outliers.
• Skewness (df.skew()): Asymmetry of distribution. Production insight: Skewed data may need log transforms for ML.
• Kurtosis (df.kurtosis()): "Tailedness" of distribution. Production insight: High kurtosis = fat tails (e.g., stock returns).

? Covariance (df.cov())

• Measures linear relationship between two variables. Production insight: Positive = variables move together; negative = opposite directions.
• Problem: Scale-dependent (e.g., covariance of 100 vs. 1000 means nothing without context).

? Correlation (df.corr())

• Pearson: Linear correlation (-1 to 1). Production insight: Default in pandas; sensitive to outliers.
• Spearman: Rank-based (monotonic relationships). Production insight: Use for non-linear but consistent trends (e.g., "more ads → more sales, but not linearly").
• Kendall: Rank-based (better for small datasets). Production insight: Slower but more robust to ties.

3. Step-by-Step Hands-On: Analyzing a Dataset

Prerequisites

• Python 3.8+ with pandas, numpy, seaborn, matplotlib.
• A dataset (we’ll use the Titanic dataset for this example).

Step 1: Load and Inspect Data

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Load data
df = pd.read_csv("titanic.csv")

# Quick inspection
print(df.head())
print("\nMissing values:\n", df.isnull().sum())
print("\nData types:\n", df.dtypes)

Expected output:

   PassengerId  Survived  Pclass  ...     Fare Cabin  Embarked
0            1         0       3  ...   7.2500   NaN         S
1            2         1       1  ...  71.2833   C85         C
2            3         1       3  ...   7.9250   NaN         S

Missing values:
 PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

Data types:
 PassengerId      int64
Survived         int64
Pclass           int64
Name            object
Sex             object
Age            float64
SibSp            int64
Parch            int64
Ticket          object
Fare           float64
Cabin           object
Embarked        object
dtype: object

Step 2: Compute Summary Statistics

# Numeric columns only
numeric_cols = df.select_dtypes(include=['int64', 'float64']).columns
print(df[numeric_cols].describe())

# Skewness and kurtosis
print("\nSkewness:\n", df[numeric_cols].skew())
print("\nKurtosis:\n", df[numeric_cols].kurtosis())

Expected output:

       PassengerId    Survived      Pclass         Age       SibSp  ...       Parch        Fare
count   891.000000  891.000000  891.000000  714.000000  891.000000  ...  891.000000  891.000000
mean    446.000000    0.383838    2.308642   29.699118    0.523008  ...    0.381594   32.204208
std     257.353842    0.486592    0.836071   14.526497    1.102743  ...    0.806057   49.693429
min       1.000000    0.000000    1.000000    0.420000    0.000000  ...    0.000000    0.000000
25%     223.500000    0.000000    2.000000   20.125000    0.000000  ...    0.000000    7.910400
50%     446.000000    0.000000    3.000000   28.000000    0.000000  ...    0.000000   14.454200
75%     668.500000    1.000000    3.000000   38.000000    1.000000  ...    0.000000   31.000000
max     891.000000    1.000000    3.000000   80.000000    8.000000  ...    6.000000  512.329200

Skewness:
PassengerId    0.0
Survived       0.5
Pclass         0.0
Age            0.4
SibSp          3.7
Parch          2.8
Fare           4.8
dtype: float64

Kurtosis:
PassengerId   -1.2
Survived      -1.9
Pclass        -1.5
Age           -0.6
SibSp         14.3
Parch          8.6
Fare          25.0
dtype: float64

Key takeaways:
- Fare is highly skewed (kurtosis = 25). Log transform may help.
- Age has missing values (714/891 = 80% coverage).

Step 3: Compute Covariance and Correlation

# Covariance matrix
print("Covariance:\n", df[numeric_cols].cov())

# Correlation matrix (Pearson)
print("\nCorrelation (Pearson):\n", df[numeric_cols].corr())

# Spearman correlation
print("\nCorrelation (Spearman):\n", df[numeric_cols].corr(method='spearman'))

Expected output (Pearson):

               PassengerId  Survived    Pclass       Age     SibSp     Parch       Fare
PassengerId    6.623000e+04 -0.005007 -0.035144  0.036847 -0.057527 -0.001652  0.012658
Survived      -5.007000e+00  0.235702 -0.338481 -0.077221 -0.035322  0.081629  0.257307
Pclass        -3.514400e+01 -0.338481  0.699348  0.369226  0.083081  0.018443 -0.549500
Age            3.684700e+01 -0.077221  0.369226 210.997000  0.159651  0.081629 -0.096067
SibSp         -5.752700e+00 -0.035322  0.083081  0.159651  1.216052  0.414838  0.159651
Parch         -1.652000e+00  0.081629  0.018443  0.081629  0.414838  0.649715  0.216225
Fare           1.265800e+01  0.257307 -0.549500 -0.096067  0.159651  0.216225 2469.436846

Key takeaways:
- Pclass and Fare are strongly negatively correlated (-0.55): Higher class = higher fare.
- Survived correlates with Fare (0.26) and Pclass (-0.34): Richer passengers survived more.

Step 4: Visualize Relationships

# Pairplot for numeric columns
sns.pairplot(df[numeric_cols].dropna())
plt.show()

# Heatmap of correlations
sns.heatmap(df[numeric_cols].corr(), annot=True, cmap='coolwarm')
plt.title("Correlation Heatmap")
plt.show()

What to look for:
- Linear trends (e.g., Pclass vs. Fare).
- Outliers (e.g., a single passenger with Fare = 512).
- Clusters (e.g., Age groups).

4. ? Production-Ready Best Practices

? Data Quality

• Handle missing values: Use df.fillna() or df.dropna() before computing stats.
  python df['Age'].fillna(df['Age'].median(), inplace=True) # Robust to outliers
• Check for impossible values: E.g., negative ages, future dates.
  python assert (df['Age'] >= 0).all(), "Negative ages detected!"

⚡ Performance

• Use .select_dtypes() to avoid computing stats on non-numeric columns.
• Sample large datasets: For quick EDA, use df.sample(10000).
• Vectorize operations: Avoid loops; use df.mean() instead of sum(df[col])/len(df).

? Interpretation

• Correlation ≠ causation: A high correlation between SibSp and Parch (0.41) doesn’t mean siblings cause parents.
• Scale matters: Always normalize before comparing correlations (e.g., sklearn.preprocessing.StandardScaler).
• Non-linear relationships: Use Spearman if you suspect non-linear trends (e.g., "more ads → more sales, but diminishing returns").

? Documentation

• Save stats to a report: python stats_report = df.describe().to_markdown() with open("stats_report.md", "w") as f:
f.write(stats_report)
• Annotate outliers: python outliers = df[df['Fare'] > 200] print("High-fare outliers:\n", outliers[['PassengerId', 'Fare', 'Pclass']])

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. Interpretation:
   "Given a correlation of -0.8 between temperature and heater_usage, what does this mean?"

2. Answer: Strong negative linear relationship (higher temp → less heater use).

3. Method Selection:
   "Which correlation method should you use for ordinal data (e.g., survey responses 1-5)?"

4. Answer: Spearman (rank-based).

5. Outlier Impact:
   "A dataset has a mean of 50 and a median of 45. What does this suggest?"

6. Answer: Right-skewed data (outliers pulling mean up).

7. Covariance vs. Correlation:
   "Why is correlation preferred over covariance?"

8. Answer: Correlation is scale-independent (normalized to -1 to 1).

⚠️ Trap Distinctions

• Pearson vs. Spearman:
• Pearson = linear relationships.
• Spearman = monotonic (but not necessarily linear) relationships.
• Correlation ≠ Causation:
• "Ice cream sales and drowning deaths are correlated" → Third variable (summer heat).
• Spurious Correlations:
• "Divorce rate in Maine correlates with margarine consumption" → Coincidence.

7. ? Hands-On Challenge

Challenge: You’re given a dataset of house prices (price) and square footage (sqft). Compute the Pearson correlation between the two. Then, add a third column price_per_sqft = price / sqft and compute its correlation with sqft. What do you observe, and why?

Solution:

import pandas as pd

# Sample data
data = {
    'price': [300000, 450000, 600000, 750000],
    'sqft': [1500, 2000, 3000, 4000]
}
df = pd.DataFrame(data)
df['price_per_sqft'] = df['price'] / df['sqft']

# Correlations
print("Correlation (price vs. sqft):", df['price'].corr(df['sqft']))
print("Correlation (price_per_sqft vs. sqft):", df['price_per_sqft'].corr(df['sqft']))

Output:

Correlation (price vs. sqft): 1.0
Correlation (price_per_sqft vs. sqft): -1.0

Explanation: - price and sqft are perfectly correlated (1.0) because price_per_sqft is constant (e.g., $200/sqft).
- price_per_sqft and sqft are perfectly negatively correlated (-1.0) because dividing by sqft inverts the relationship.

Why it matters: This is a mathematical artifact—not a real-world insight. Always check if derived features (like price_per_sqft) are introducing spurious correlations.

8. ? Rapid-Reference Crib Sheet

Pandas Commands