Skip to main content
W&B Fully Connected
>
Articles
>
Community Posts

An introduction to Gradient Boosted Trees for machine learning

Explore Gradient Boosted Trees in machine learning with this comprehensive guide. Learn step-by-step implementation and key insights into XGBoost, LightGBM, and CatBoost to enhance predictive modeling.
Mostafa Ibrahim
Created on January 17|Last edited on December 20
This article explores Gradient Boosting in machine learning, highlighting its capability to handle complex datasets effectively. With examples using XGBoost, LightGBM, and CatBoost, we provide a practical and step-by-step implementation guide.


Table of conents

IntroductionWhat is gradient boosting?Key components of gradient boostingWhat are gradient boosted trees?How gradient boosting works in machine learning step-by-stepStep 1: Initial model predictionStep 2: Calculating residualsStep 3: Building a decision treeStep 4: Updating the modelStep 5: IterateStep 6: The final modelTypes of Gradient Boosting algorithms1. XGBoost (eXtreme Gradient Boosting)2. LightGBM (Light Gradient Boosting Machine)3. CatBoost (Categorical Boosting)Practical implementation of Gradient BoostingConclusion


Introduction

Gradient Boosting has transformed the landscape of machine learning, excelling in both regression and classification tasks. This method leverages gradient-boosted trees to handle complex, non-linear datasets, combining the simplicity of decision trees with the robustness of ensemble learning. Whether you're new to the field or an experienced practitioner, this article will guide you through the essentials of Gradient Boosting and its practical applications.

What is gradient boosting?

Gradient Boosting is a machine learning technique widely used for regression and classification. It builds predictive models sequentially, each improving upon the errors of its predecessor.

Key components of gradient boosting

  • Decision Trees as Base Learners: Gradient Boosting uses simple decision trees, which are built sequentially to correct errors from previous trees.
  • Gradient Descent Algorithm: This algorithm minimizes the loss function by iteratively improving predictions.
  • Loss Function Optimization: The model’s objective is to minimize the loss function, quantifying prediction errors.

What are gradient boosted trees?

Gradient-boosted trees (GBT) form the foundation of Gradient Boosting, combining multiple decision trees to create a powerful predictive model. Each tree is sequentially built to focus on correcting the residual errors from the trees before it.
This iterative process enhances the model's accuracy and robustness.
  • Building Decision Trees: These are split based on features to minimize the loss function.
  • Using Residuals: Residuals are calculated as the difference between predicted and actual values, and they guide subsequent tree-building.
  • Learning Rate: This parameter scales the contribution of each tree, enhancing model robustness.
Source
By combining these elements, Gradient Boosted Trees iteratively refine predictions, resulting in a model that balances accuracy and adaptability. In the sections ahead, we will explore how these trees are implemented step by step to tackle machine learning challenges effectively.

How gradient boosting works in machine learning step-by-step

Gradient Boosting builds models step by step to improve predictions. Here’s how the process works, using a dataset of patients’ ages and heart rates as an example:
Here's how we can do it step by step:


Step 1: Initial model prediction

The first step is to predict heart rates for each patient using the mean heart rate of the dataset as the initial prediction. This baseline serves as the foundation for further improvements.


Step 2: Calculating residuals

Residuals are calculated as the difference between the actual heart rates and the initial predictions. For example, if the actual heart rate is 72 and the initial prediction is 77.4, the residual is -5.4.



Step 3: Building a decision tree

Next, a decision tree is constructed to predict the residuals. The tree uses patient age as the feature to learn patterns in the residuals, aiming to address the errors from the initial prediction.


Step 4: Updating the model

The predictions from the decision tree are scaled by a learning rate (e.g., 0.1) and added to the initial predictions. For instance, for a residual of -5.4, the updated prediction is:
This adjustment moves the predictions closer to the actual heart rates.
Here are the results of Step 4, predicting residuals with the first decision tree:


Step 5: Iterate

Steps 2-4 are repeated for a specified number of iterations, with each new tree addressing the updated residuals. Each iteration refines the model, reducing errors and improving predictions.
Here are the results of Step 5, updating the model:


Step 6: The final model

The final model combines the initial predictions and the contributions from all trees. After three iterations, for example, the model’s predictions represent the best estimate of patients’ heart rates based on their ages. This iterative process ensures the model adjusts and improves over time.


Types of Gradient Boosting algorithms

There are multiple variations and customizations of the Gradient Boosting Algorithm all with their advantages and use cases.

1. XGBoost (eXtreme Gradient Boosting)

XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable. It implements machine learning algorithms under the Gradient Boosting framework.
Some of its key features include:
  • Speed and Performance: XGBoost is fast and known for its performance. It uses a more regularized model formalization to control overfitting, which gives it better performance.
  • Scalability and Flexibility: XGBoost scales to billions of examples and works well on a range of computing environments, including distributed systems.
  • Customizable: It supports custom objective functions and evaluation criteria, adding to its flexibility.

2. LightGBM (Light Gradient Boosting Machine)

LightGBM is a gradient-boosting framework that uses tree-based learning algorithms and is designed for distributed and efficient training, particularly on large datasets.
Some of its key advantages include:
  • Efficiency on Large Datasets: It is faster than other implementations of gradient boosting when working with large datasets.
  • Lower Memory Usage: LightGBM uses a novel technique called Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB), which reduces memory usage significantly.
  • Handles Large-scale Data: It performs well with large data and can handle a huge amount of features.
  • Differences from Others: The key difference lies in its handling of large datasets and lower memory usage, making it suitable for scenarios where computational resources are a constraint.

3. CatBoost (Categorical Boosting)

CatBoost is an open-source gradient boosting library, particularly known for its ability to handle categorical data effectively.
Some of its key advantages include:
  • Handling of Categorical Data: Unlike other gradient boosting algorithms, CatBoost doesn’t require extensive data preprocessing to convert categorical data into numerical data. It handles categorical features automatically using various statistics on combinations of categorical features and the target variable.
  • Robust and Accurate: It reduces the need for extensive hyper-parameter tuning and lowers the chances of overfitting, leading to more robust and accurate models.
  • User-Friendly: CatBoost is user-friendly and easy to integrate with deep learning frameworks.

Practical implementation of Gradient Boosting

Now let's want through a practical example of implementing the Gradient Boosting algorithm using Python and its popular machine learning library, scikit-learn. The process is divided into distinct steps, each crucial for the successful application of the algorithm.

Step 1: Importing the Necessary Libraries

from sklearn.ensemble import GradientBoostingRegressor  # or GradientBoostingClassifier for classification tasks
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error  # or accuracy_score for classification
import pandas as pd

Step 2: Loading Our Dataset

The next step is to load and prepare the dataset. In this example, a simple dataset is created using a DataFrame df, consisting of two columns: 'Age' and 'HeartRate'. These represent the features and target variables, respectively. 
data = {
    "Age": [25, 35, 45, 55, 65],
    "HeartRate": [72, 75, 78, 80, 82]
}
df = pd.DataFrame(data)
The dataset is then split into two parts: the features (X) and the target (y). X consists of the 'Age' column, while y consists of the 'HeartRate' column. Using the train_test_split function, this dataset is further divided into training and testing sets, with the test_size parameter determining the proportion of the dataset to include in the test split.
X = df[['Age']]
y = df['HeartRate']

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 3: Importing Gradient Boosting Regressor

The third step involves the instantiation of the gradient-boosting model.
The GradientBoostingRegressor is initialized with various parameters, including n_estimators, which defines the number of boosting stages to be run (essentially, the number of trees to be built), learning_rate, which scales the contribution of each tree, and max_depth, which sets the maximum depth of the individual regression estimators. The random_state parameter ensures that the splits that you generate are reproducible.
gb_model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.01, max_depth=10, random_state=42)
For classification, use GradientBoostingClassifier with similar parameters.

Step 4: Fitting the Dataset to Our Model

gb_model.fit(X_train, y_train)

Step 5: Predicting and Evaluating the Test Set Results

Finally, the model's performance is evaluated. This is achieved by predicting heart rates for the test set (X_test) and comparing these predictions (y_pred) against the actual heart rates (y_test). 
y_pred = gb_model.predict(X_test)

# Evaluate the model
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error: {mse}")
Mean Squared Error: 0.6461040090503394

Conclusion

Gradient Boosting is a versatile and powerful technique in machine learning. Its ability to sequentially build decision trees while minimizing errors makes it a go-to method for tackling complex data challenges. By exploring its practical applications, including XGBoost, LightGBM, and CatBoost, we demonstrate how Gradient Boosting can refine predictions and enhance model accuracy. Whether working on regression or classification, mastering Gradient Boosting will significantly enhance your machine learning toolkit.



Tags: Articles, Community Posts, Beginner

Created with ❤️ on Weights & Biases.

https://wandb.ai/mostafaibrahim17/ml-articles/reports/An-Introduction-to-Gradient-Boosted-Trees-for-Machine-Learning--Vmlldzo2NTQ4NzYx
Made with Weights & Biases. Sign up or log in to create reports like this one.
Iterate on AI agents and models faster. Try Weights & Biases today.
Sign upTry W&B now