Hyper-parameter Tuning Guide: Updated Practice and Reference Note

Hyperparameter Tuning Guide: Updated Practice and Reference Note

This guide provides a comprehensive list of hyperparameters for tree-based models, organized by their functionality and tailored for effective tuning in machine learning projects.

1. General Workflow for Hyperparameter Tuning

1. Understand Your Model Type:
• Bagging models (e.g., Random Forest) focus on independent tree creation.
• Boosting models (e.g., XGBoost, LightGBM, CatBoost) involve sequential learning.
2. Key Objectives:
• Prevent overfitting (when the model is too complex).
• Prevent underfitting (when the model is too simple).
• Balance model complexity, generalization, and training time.
3. Steps:
• Start with default hyperparameters.
• Identify key parameters to tune based on model type and dataset size.
• Use GridSearchCV or RandomizedSearchCV for systematic exploration.

2. Parameters for Each Model Type

A. Random Forest / Extra Trees / Bagging Models

These models do not use learning_rate. Focus on parameters related to tree complexity and sampling.

B. Gradient Boosting Models (e.g., XGBoost, LightGBM, CatBoost)

These models use learning_rate and require careful balancing between tree complexity, step size, and regularization.

C. LightGBM-Specific Parameters

LightGBM has unique parameters like num_leaves and feature bagging.

D. CatBoost-Specific Parameters

CatBoost has specialized handling for categorical features and its own hyperparameters.

3. Example Param Grids

Random Forest

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, 15],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4],
    'max_features': ['sqrt', 'log2', None]
}

XGBoost

param_grid = {
    'n_estimators': [100, 200, 300],
    'learning_rate': [0.01, 0.1, 0.2],
    'max_depth': [3, 5, 7],
    'subsample': [0.6, 0.8, 1.0],
    'colsample_bytree': [0.6, 0.8, 1.0],
    'gamma': [0, 1, 5]
}

LightGBM

param_grid = {
    'n_estimators': [100, 200, 300],
    'learning_rate': [0.01, 0.05, 0.1],
    'num_leaves': [31, 63, 127],
    'feature_fraction': [0.6, 0.8, 1.0],
    'bagging_fraction': [0.6, 0.8, 1.0],
    'lambda_l1': [0, 0.1, 1],
    'lambda_l2': [0, 0.1, 1]
}

CatBoost

python
Copy code
param_grid = {
    'iterations': [100, 200, 500],
    'learning_rate': [0.01, 0.1, 0.2],
    'depth': [3, 6, 10],
    'l2_leaf_reg': [3, 5, 10],
    'random_strength': [1, 5, 10]
}

4. Summary

1. Start Simple: Use the most impactful parameters (n_estimators, max_depth, learning_rate).
2. Regularization: Use min_samples_split, min_child_weight, and lambda_l1/lambda_l2 for controlling overfitting.
3. Iterate Gradually: Expand your param grid once you identify promising ranges.
4. Specialized Models:
• Random Forest: Focus on max_depth, n_estimators, bootstrap.
• Boosting Models: Balance learning_rate, n_estimators, and regularization.

This guide integrates all major parameters to ensure effective hyperparameter tuning. Let me know if you'd like further additions or clarifications!

Choosing the optimal values for hyperparameters like learning_rate and max_depth

key difference between n_estimators and learning_rate

To decide whether to focus on max_depth or learning_rate

The num_leaves parameter is specific to LightGBM