🧠

7. Pandas Series: Missing Data Handling

Status

Done

Missing Value Detection

df['series'].isna() → Boolean mask of missing values (alias: isnull())
df['series'].notna() → Boolean mask of non-missing values (alias: notnull())

2. Missing Value Removal

df['series'].dropna() → Remove missing values (returns new Series)

3. Missing Value Filling

df['series'].fillna(value) → Fill NA with specified value/method
df['series'].ffill() → Forward fill (alias: pad())
df['series'].bfill() → Backward fill (alias: backfill())
df['series'].interpolate() → Fill NA via interpolation

Key Notes:

isna/isnull and notna/notnull are identical (use whichever you prefer)
ffill/pad propagate last valid observation forward
bfill/backfill propagate next valid observation backward
interpolate offers multiple methods (linear, polynomial, etc.)

Sample Dataset

import pandas as pd
import numpy as np

data = {
    'temperature': [22, np.nan, 28, 24, np.nan, 23, 26, 29],
    'ice_cream_sales': [110, 150, np.nan, 130, 170, 120, np.nan, 190],
    'time_series': [10, np.nan, np.nan, 18, 14, 20, 16, 22],  # Time-series with gaps
    'inventory': [45, 50, 55, np.nan, np.nan, 60, 65, 70]    # Inventory tracking
}

df = pd.DataFrame(data)
print(df)

   temperature  ice_cream_sales  time_series  inventory
0         22.0            110.0         10.0       45.0
1          NaN            150.0          NaN       50.0
2         28.0              NaN          NaN       55.0
3         24.0            130.0         18.0        NaN
4          NaN            170.0         14.0        NaN
5         23.0            120.0         20.0       60.0
6         26.0              NaN         16.0       65.0
7         29.0            190.0         22.0       70.0

1. Missing Value Detection

‣

1.1 `df['series'].isna()` / `df['series'].isnull()`

‣

1.2 `df['series'].notna()` / `df['series'].notnull()`

2. Missing Value Removal

‣

2.1 `df['series'].dropna()`

3. Missing Value Filling

‣

3.1 `df['series'].fillna()`

‣

3.2 `df['series'].ffill()` / `pad()`

‣

3.3 `df['series'].bfill()` / `backfill()`

‣

3.4 `df['series'].interpolate()`

Summary Table

Method	Description	Example Use Case
`isna()` / `isnull()`	Detect missing values	`df['temp'].isna().sum()`
`notna()` / `notnull()`	Detect non-missing values	`df[df['sales'].notna()]`
`dropna()`	Remove missing values	`df['temp'].dropna()`
`fillna()`	Fill NA with value/method	`df['temp'].fillna(25)`
`ffill()` / `pad()`	Forward fill	`df['ts'].ffill()`
`bfill()` / `backfill()`	Backward fill	`df['inventory'].bfill()`
`interpolate()`	Interpolate NA	`df['temp'].interpolate()`

Key Practical Scenarios

Time-Series Data:

Use ffill() for sensor readings where gaps should inherit the last valid value.
Use interpolate(method='time') for irregular timestamps.

Business Data:

Replace missing sales data with the column mean:

python

df['sales'].fillna(df['sales'].mean(), inplace=True)

Inventory Tracking:

Backward fill to reflect the most recent stock count:

python

df['inventory'].bfill(inplace=True)

Advanced Interpolation:

python

df['temperature'].interpolate(method='polynomial', order=2)  # Quadratic fit

Final Notes

Performance: ffill/bfill are faster than interpolate for large datasets.
Inplace Modification: Use inplace=True to modify the Series directly.
Chaining Methods: Combine operations like:

python

df['column'].fillna(0).interpolate().bfill()

7. Pandas Series: Missing Data Handling

Sample Dataset

1. Missing Value Detection

1.1 df['series'].isna() / df['series'].isnull()

1.2 df['series'].notna() / df['series'].notnull()

2. Missing Value Removal

2.1 df['series'].dropna()

3. Missing Value Filling

3.1 df['series'].fillna()

3.2 df['series'].ffill() / pad()

3.3 df['series'].bfill() / backfill()

3.4 df['series'].interpolate()

Summary Table

Key Practical Scenarios

Final Notes

1.1 `df['series'].isna()` / `df['series'].isnull()`

1.2 `df['series'].notna()` / `df['series'].notnull()`

2.1 `df['series'].dropna()`

3.1 `df['series'].fillna()`

3.2 `df['series'].ffill()` / `pad()`

3.3 `df['series'].bfill()` / `backfill()`

3.4 `df['series'].interpolate()`