7. Pandas Series: Missing Data Handling

1. Missing Value Detection
• df['series'].isna() → Boolean mask of missing values (alias: isnull())
• df['series'].notna() → Boolean mask of non-missing values (alias: notnull())

Sample Dataset

import pandas as pd
import numpy as np

data = {
    'temperature': [22, np.nan, 28, 24, np.nan, 23, 26, 29],
    'ice_cream_sales': [110, 150, np.nan, 130, 170, 120, np.nan, 190],
    'time_series': [10, np.nan, np.nan, 18, 14, 20, 16, 22],  # Time-series with gaps
    'inventory': [45, 50, 55, np.nan, np.nan, 60, 65, 70]    # Inventory tracking
}

df = pd.DataFrame(data)
print(df)

1. Missing Value Detection

2. Missing Value Removal

3. Missing Value Filling

Summary Table

Key Practical Scenarios

1. Time-Series Data:
• Use ffill() for sensor readings where gaps should inherit the last valid value.
• Use interpolate(method='time') for irregular timestamps.
2. Business Data:
• Replace missing sales data with the column mean:

python

Copy

df['sales'].fillna(df['sales'].mean(), inplace=True)

3. Inventory Tracking:
• Backward fill to reflect the most recent stock count:

python

Copy

df['inventory'].bfill(inplace=True)

4. Advanced Interpolation:

python

Copy

df['temperature'].interpolate(method='polynomial', order=2)  # Quadratic fit

Final Notes

• Performance: ffill/bfill are faster than interpolate for large datasets.
• Inplace Modification: Use inplace=True to modify the Series directly.
• Chaining Methods: Combine operations like:

python

Copy

df['column'].fillna(0).interpolate().bfill()