🚢 Complete Data Cleaning Guide Using Titanic Dataset

A Step-by-Step Tutorial to Master pandas_data_cleaning.py

Author: Teslim Uthman Adeyanju

Purpose: Learn professional data cleaning techniques from beginner to pro

Case Study: Titanic Passenger Data

📚 Table of Contents

1. Introduction & Setup
2. Section 1: Column Header Cleaning
3. Section 2: Numeric Data Cleaning
4. Section 3: Duplicate Management
5. Section 4: Missing Values Handling
6. Section 5: Outlier Detection & Removal
7. Section 6: Text Data Cleaning
8. Section 7: Data Summary & Inspection
9. Section 8: Quick Clean Pipeline
10. Complete Workflow Example
11. Pro Tips & Best Practices

1. Introduction & Setup

What You’ll Learn

By the end of this guide, you’ll be able to: - Identify and fix common data quality issues - Apply appropriate cleaning techniques for different data types - Build reproducible data cleaning pipelines - Make informed decisions about data transformations - Document your cleaning process professionally

Setup Your Environment

# Import required librariesimport pandas as pd
import numpy as np
from pandas_data_cleaning import *# Load the dirty Titanic datasetdf = pd.read_csv('titanic_dirty.csv')
# Always look at your data first!print("Raw Data:")
print(df.head())
print(f"\nDataset Shape: {df.shape}")
print(f"\nColumn Names: {df.columns.tolist()}")

Common Data Quality Issues in Real Datasets

The Titanic dataset we’re using has these intentional problems: - ❌ Inconsistent column names (spaces, mixed case) - ❌ Duplicate rows - ❌ Missing values (empty strings, ‘NA’, ‘Nil’, None) - ❌ Inconsistent text formatting (mixed case, extra spaces) - ❌ Non-numeric values in numeric columns (‘#REF!’, ‘NA’) - ❌ Outliers in fare prices - ❌ Inconsistent categorical values

These are the EXACT problems you’ll face in real-world data!

2. Section 1: Column Header Cleaning

🎯 Why This Matters

Column names are your interface to the data. Bad column names lead to: - Coding errors and typos - Difficulty in automation - Problems when exporting to databases - Confusion in team collaboration

📖 What It Does

The clean_column_headers() function: 1. Converts all column names to lowercase 2. Removes leading/trailing whitespace 3. Replaces spaces with underscores 4. Removes special characters 5. Handles duplicate column names 6. Creates Python-friendly variable names

💻 Practical Example

🎓 When to Use

• ALWAYS as the first step in your cleaning pipeline
• Before any column-specific operations
• When receiving data from external sources
• When merging datasets from different sources

⚠️ Common Pitfalls

• DON’T clean headers multiple times (it’s already clean!)
• DON’T manually rename columns before using this function
• DO save a copy of original column names if needed for documentation

🔥 Pro Tips

# Save original column mapping for referenceoriginal_cols = df.columns.tolist()
df = clean_column_headers(df)
cleaned_cols = df.columns.tolist()
# Create a mapping dictionarycol_mapping = dict(zip(original_cols, cleaned_cols))
print("Column Name Changes:")
for old, new in col_mapping.items():
    if old != new:
        print(f"  '{old}' → '{new}'")

📊 Before & After Comparison

# Show the transformationcomparison = pd.DataFrame({
    'Original': original_cols,
    'Cleaned': cleaned_cols,
    'Changed': [old != new for old, new in zip(original_cols, cleaned_cols)]
})
print(comparison)

3. Section 2: Numeric Data Cleaning

🎯 Why This Matters

Numeric columns are used for calculations, statistics, and machine learning. Even one bad value can ruin your entire analysis!

📖 What It Does

The clean_numeric_column() function: 1. Replaces problematic strings (‘#REF!’, ‘Nil’, ‘NA’, etc.) with NaN 2. Converts column to numeric type 3. Fills missing values using statistical methods 4. Optionally removes negative values 5. Rounds to specified decimal places

💻 Practical Example: Cleaning Age Column

💻 Practical Example: Cleaning Fare Column

🎓 Choosing the Right Fill Method

⚠️ Common Pitfalls

🔥 Pro Tips

📊 Visualizing Numeric Cleaning Impact

4. Section 3: Duplicate Management

🎯 Why This Matters

Duplicate rows can: - Inflate your dataset size unnecessarily - Bias statistical analyses - Create incorrect counts and aggregations - Cause issues in machine learning (data leakage)

📖 What It Does

The remove_duplicates() function: 1. Identifies duplicate rows based on all or specific columns 2. Keeps first, last, or removes all duplicates 3. Resets the index for clean row numbering 4. Reports how many duplicates were removed

💻 Practical Example: Finding Duplicates

# First, let's see if we have duplicatesprint("Checking for duplicates...")
print(f"Total rows: {len(df)}")
print(f"Duplicate rows: {df.duplicated().sum()}")
# Show the duplicatesif df.duplicated().sum() > 0:
    print("\nDuplicate entries:")
    print(df[df.duplicated(keep=False)].sort_values('name'))

💻 Example 1: Remove Complete Duplicates

# Remove rows where ALL columns are identicaldf_clean = remove_duplicates(df)
# Output: ✅ Removed 2 duplicate rowsprint(f"Rows before: {len(df)}")
print(f"Rows after: {len(df_clean)}")

💻 Example 2: Remove Duplicates Based on Specific Columns

# Maybe we only care about passenger name being uniquedf_unique_names = remove_duplicates(
    df,
    subset=['name'],  # Only consider 'name' column    keep='first'  # Keep the first occurrence)
print(f"Unique passengers: {len(df_unique_names)}")

💻 Example 3: Keep Last Occurrence

# Keep the most recent entry (useful for updated records)df_keep_last = remove_duplicates(
    df,
    subset=['passenger_id'],
    keep='last'  # Keep the last occurrence)

🎓 Understanding the ‘keep’ Parameter

⚠️ Common Pitfalls

🔥 Pro Tips

📊 Duplicate Patterns Analysis

5. Section 4: Missing Values Handling

🎯 Why This Matters

Missing data is inevitable in real-world datasets. How you handle it can make or break your analysis: - Deleting rows loses information - Improper imputation introduces bias - Different columns need different strategies - ML algorithms can’t handle missing values

📖 What It Does

The fill_missing_values() function: 1. Detects missing values (NaN, None, null) 2. Fills them using various strategies 3. Supports forward/backward fill for time series 4. Allows custom values 5. Reports how many values were filled

💻 Practical Example: Understanding Missing Data Patterns

💻 Example 1: Fill Numeric Column with Statistical Method

# Fill age with median (already done in numeric cleaning)df = fill_missing_values(
    df,
    column='age',
    method='median')
# Output: ✅ Filled 2 missing values in 'age' using median

💻 Example 2: Fill Categorical Column with Mode

# Fill cabin with most common valuedf = fill_missing_values(
    df,
    column='cabin',
    method='mode')
# Output: ✅ Filled 4 missing values in 'cabin' using mode

💻 Example 3: Fill with Custom Value

# Fill embarked location with 'Unknown' for claritydf = fill_missing_values(
    df,
    column='embarked',
    method='custom',
    custom_value='Unknown')

💻 Example 4: Forward Fill for Time Series

# Imagine we have a time series of stock prices# Forward fill propagates last valid observation forwarddf_timeseries = fill_missing_values(
    df_timeseries,
    column='stock_price',
    method='forward')

🎓 Choosing the Right Filling Strategy

⚠️ Common Pitfalls

🔥 Pro Tips

📊 Missing Data Impact Analysis

🎓 Advanced: Multiple Imputation Strategies

6. Section 5: Outlier Detection & Removal

🎯 Why This Matters

Outliers can: - Distort statistical analyses (mean, standard deviation) - Reduce model accuracy - Hide patterns in your data - Sometimes indicate data entry errors - Sometimes be the most interesting data points!

KEY QUESTION: Are outliers errors or valuable extreme values?

📖 What It Does

The module provides two functions: 1. detect_outliers_iqr() - Finds outliers (doesn’t remove them) 2. remove_outliers_iqr() - Removes outliers based on IQR method

IQR Method: - IQR = Q3 - Q1 (75th percentile - 25th percentile) - Lower bound = Q1 - (factor × IQR) - Upper bound = Q3 + (factor × IQR) - Factor 1.5 = standard outliers - Factor 3.0 = extreme outliers only

💻 Practical Example: Detecting Outliers in Fare

💻 Example 1: Conservative Outlier Removal

# Use factor=3.0 for extreme outliers onlydf_conservative = remove_outliers_iqr(
    df,
    column='fare',
    factor=3.0,  # Only remove EXTREME outliers    verbose=True)
# Output: ✅ Removed 1 outliers from column 'fare'

💻 Example 2: Standard Outlier Removal

# Use factor=1.5 for standard outliersdf_standard = remove_outliers_iqr(
    df,
    column='fare',
    factor=1.5,  # Standard outlier detection    verbose=True)
# Output: ✅ Removed 3 outliers from column 'fare'

💻 Example 3: Multiple Column Outlier Detection

🎓 When to Remove Outliers (Decision Framework)

⚠️ Common Pitfalls

🔥 Pro Tips

📊 Visual Comparison: Before and After

7. Section 6: Text Data Cleaning

🎯 Why This Matters

Text data is messy by nature: - Inconsistent capitalization - Extra whitespace - Typos and variations - Multiple representations of “missing” - Makes grouping and searching difficult

Clean text data enables: - Accurate categorization - Reliable searching and filtering - Consistent analysis - Better data visualization

📖 What It Does

The clean_text_column() function: 1. Converts text to lowercase 2. Removes extra whitespace 3. Standardizes missing values 4. Handles NaN values properly 5. Preserves actual null values

💻 Practical Example: Cleaning Name Column

# First, see what we're dealing withprint("Name column issues:")
print(df['name'].unique())
# Output: Shows mixed case, extra spaces, etc.# Clean the name columndf = clean_text_column(
    df,
    column='name',
    lowercase=True,
    remove_extra_spaces=True,
    standardize_missing=True)
print("\nCleaned names:")
print(df['name'].unique())

💻 Example 1: Cleaning Embarkation Port

💻 Example 2: Preserving Case Sensitivity

# Sometimes you want to keep case (like names)df = clean_text_column(
    df,
    column='cabin',
    lowercase=False,  # Keep original case    remove_extra_spaces=True,
    standardize_missing=True)

💻 Example 3: Cleaning Multiple Text Columns

# Clean all text columns at oncetext_columns = ['name', 'cabin', 'embarked']
for col in text_columns:
    if col in df.columns:
        df = clean_text_column(df, col)
        print(f"✅ Cleaned text column: {col}")

🎓 Text Cleaning Parameters Guide

⚠️ Common Pitfalls

🔥 Pro Tips

📊 Text Data Quality Assessment

🎓 Advanced: Text Normalization Pipeline

8. Section 7: Data Summary & Inspection

🎯 Why This Matters

You can’t clean what you don’t understand! Data inspection is crucial for: - Understanding data structure and types - Identifying cleaning priorities - Tracking cleaning progress - Documenting dataset characteristics - Communicating data quality to stakeholders

“Look at your data first, clean it second!”

📖 What It Does

Two powerful functions: 1. get_data_summary() - Returns dictionary with comprehensive statistics 2. print_data_summary() - Pretty-printed formatted summary

💻 Practical Example: Initial Data Inspection

💻 Example 1: Programmatic Access to Summary

💻 Example 2: Before and After Comparison

💻 Example 3: Missing Data Dashboard

🎓 Understanding the Summary Statistics

🔥 Pro Tips

📊 Creating a Cleaning Progress Tracker

9. Section 8: Quick Clean Pipeline

🎯 Why This Matters

Sometimes you need to clean data FAST! The quick_clean() function: - Applies multiple cleaning steps at once - Uses sensible defaults - Automates common workflows - Saves time on routine cleaning tasks

Perfect for: Initial exploration, prototypes, standard datasets

Not recommended for: Production pipelines (use individual functions for control)

📖 What It Does

The quick_clean() function: 1. Cleans column headers 2. Removes duplicate rows 3. Auto-detects and cleans numeric columns 4. Auto-detects and cleans text columns 5. Prints progress messages

💻 Practical Example: Basic Quick Clean

💻 Example 1: Selective Cleaning

# Only clean headers and remove duplicatesdf_clean = quick_clean(
    df,
    clean_headers=True,
    remove_dupes=True,
    numeric_columns=[],  # Don't clean numeric    text_columns=[]      # Don't clean text)

💻 Example 2: Specific Columns Only

# Clean only specific columnsdf_clean = quick_clean(
    df,
    clean_headers=True,
    remove_dupes=True,
    numeric_columns=['age', 'fare'],  # Only these    text_columns=['name', 'cabin']     # Only these)

💻 Example 3: Before and After Comparison

🎓 When to Use Quick Clean

⚠️ Common Pitfalls

🔥 Pro Tips

10. Complete Workflow Example

🎯 Putting It All Together

Here’s a complete, professional data cleaning workflow for the Titanic dataset:

📊 Expected Output

11. Pro Tips & Best Practices

🌟 General Best Practices

1. Always Look at Your Data First

# DON'T start cleaning blindly# df = quick_clean(df)  # What am I cleaning?# DO inspect firstprint_data_summary(df)
print(df.head())
print(df.dtypes)
print(df.describe())

2. Make Copies, Don’t Modify Originals

# DON'T modify the original# clean_column_headers(df)  # Oops, can't go back!# DO create copiesdf_original = df.copy()
df_clean = clean_column_headers(df.copy())
# Now you can compare!

3. Document Your Decisions

4. Validate After Each Step

5. Use Type Hints and Documentation

🎯 Domain-Specific Tips

For Financial Data

For Medical/Healthcare Data

For E-commerce Data

🚀 Performance Tips

For Large Datasets

📝 Creating Reusable Cleaning Functions

🎓 Learning Resources

Next Steps to Become a Pro:

1. Practice with Real Datasets
• Kaggle datasets (messy data)
• UCI Machine Learning Repository
• Government open data portals
2. Learn Advanced Pandas
• Multi-indexing
• GroupBy operations
• Apply and transform functions
3. Study Data Quality Frameworks
• Data profiling
• Data lineage
• Data governance
4. Explore Related Tools
• Great Expectations (data testing)
• Pandas Profiling (automated EDA)
• Dask (large datasets)
5. Read Industry Best Practices
• “Data Cleaning” by Ihab Ilyas
• Pandas documentation
• Real-world case studies

🎉 Congratulations!

You’ve completed the comprehensive guide to data cleaning with pandas!

You now know how to: - ✅ Identify common data quality issues - ✅ Apply appropriate cleaning techniques - ✅ Build reproducible cleaning pipelines - ✅ Document your decisions - ✅ Validate your results - ✅ Handle domain-specific requirements

Remember the golden rules: 1. Look before you leap - Always inspect data first 2. Document everything - Your future self will thank you 3. Validate at each step - Catch problems early 4. Think about the domain - Context matters! 5. Keep the original - Always work on copies

📚 Quick Reference Card

Happy Data Cleaning! 🧹✨