Ensemble Learning in Machine Learning: Bagging, Boosting and Stacking

Quick answer: Learn ensemble learning techniques in Machine Learning including bagging, boosting, stacking, Random Forest and Gradient Boosting, with examples and comparisons.

What is Ensemble Learning?

Ensemble learning combines predictions from multiple models to produce a result that is typically more accurate and more stable than any single model alone. The core idea is that a group of reasonably good models, each making somewhat different errors, can outperform one single, even very well tuned, model.

Bagging (Bootstrap Aggregating)

Bagging trains multiple models independently on different random subsets of the training data, drawn with replacement, then averages their predictions (for regression) or takes a majority vote (for classification).

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

model = BaggingClassifier(DecisionTreeClassifier(), n_estimators=50)
model.fit(X_train, y_train)

Bagging primarily reduces variance, making it especially effective with models like decision trees that tend to overfit individually. Random Forest is the best known example, using bagging alongside random feature selection at each split.

Boosting

Boosting trains models sequentially, where each new model focuses specifically on the errors the previous models got wrong, then combines them into a weighted final prediction.

from sklearn.ensemble import GradientBoostingClassifier

model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
model.fit(X_train, y_train)

Boosting primarily reduces bias and often achieves higher accuracy than bagging, but is more prone to overfitting if not carefully tuned, particularly through the number of estimators and learning rate. XGBoost and LightGBM are popular, highly optimised implementations of gradient boosting widely used in production and competitions.

Stacking

Stacking trains several different types of models, such as a decision tree, a logistic regression and a support vector machine, then trains a final meta-model that learns how to best combine their individual predictions.

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

estimators = [('tree', DecisionTreeClassifier()), ('svm', SVC(probability=True))]
model = StackingClassifier(estimators=estimators, final_estimator=LogisticRegression())

Bagging vs Boosting

When to Use Ensemble Methods

Ensembles are worth the added complexity when a single model's accuracy plateaus and a further performance gain genuinely matters for the business problem. They add interpretability cost and computation time, so they are not the automatic first choice for every project.

Common Interview Questions

What is the main difference between bagging and boosting?

Bagging trains models independently and in parallel to reduce variance. Boosting trains models sequentially, each one correcting the previous model's errors, primarily to reduce bias.

Why is Random Forest less prone to overfitting than a single decision tree?

Each tree in a Random Forest sees a different random subset of data and features, and averaging across many such trees smooths out the tendency of any single tree to fit noise in its specific subset.