Bagging with Random Forests

Bagging with Random Forests
Bagging with Random Forests

Random Forest is one of the foremost popular and most powerful machine learning algorithms. It’s a kind of ensemble machine learning algorithm called Bootstrap Aggregation or bagging.

  • The bootstrap method for estimating statistical quantities from samples.
  • The Bootstrap Aggregation algorithm is used for creating multiple different models from one training dataset and are very important.
  • The Random Forest algorithm that creates a little tweak to Bagging and leads to a really powerful classifier.

How does the random forest model work and how is it different from bagging?

Let’s assume we use a choice tree algorithms as a base classifier for all three: Boosting, Bagging and (obviously :)) the random forest.

Why and when can we want to use any of these? Given a fixed-size number of coaching samples, our model will increasingly suffer from the “curse of dimensionality” if we increase the number of features. The challenge of individual, unpruned decision trees is that the hypothesis often finishes up being too complex for the underlying training data – decision trees are susceptible to over-fitting.

tl;dr: Bagging and random forests are “bagging” algorithms that aim to scale back the complexity of models that overfit the training data. In contrast, boosting is an approach to extend the complexity of models that suffer from high bias, that is, models that underfit the training data.

Bagging: Now, let’s take a glance at the probably “simplest” case, bagging. Here, we train variety (ensemble) of decision trees from bootstrap samples of your training set. Bootstrap sampling helps in drawing random samples from our training set with a replacement which is useful. E.g., if our training set consists of seven training samples, our bootstrap samples (here: n=7) can look as follows, where C1, C2, … Cm shall symbolise the choice tree classifiers:

After we trained your (m) decision trees, we will use them to classify new data via democracy. as an example, we’d let each decision tree make a choice and predict the category label that received more votes. Typically, this is able to end in a less complex decision boundary, and therefore the bagging classifier would have a lower variance (less overfitting) than a private decision tree. Below may be a plot comparing one decision tree (left) to a bagging classifier (right) for two variables from the Wine dataset (Alcohol and Hue).

Bootstrap Method

Before we get to Bagging, let’s take a fast check out a crucial foundation technique called the bootstrap. The bootstrap may be a powerful statistical procedure for estimating a quantity from a knowledge sample. this is often easiest to know if the number may be a descriptive statistic like a mean or a typical deviation.

Let’s assume we’ve a sample of 100 values (x) and we’d wish to get an estimate of the mean of the sample.

We can find mean directly also from the given samples:

mean(x) = 1/100 * sum(x)

We know that our sample is little which our mean has error in it. Using bootstrap we can calculate mean

Calculate the mean of every sub-sample.

Calculate the typical of all of our collected means and use that as our estimated mean for the info. For example, let’s say we used three resamples and got the mean values 2.3, 4.5 and 3.3. Taking the typical of those we could take the estimated mean of the info to be 3.367.

This process is often wont to estimate other quantities just like the variance and even quantities utilised in machine learning algorithms, like learned coefficients.

Random forests

The random forest algorithm is really a bagging algorithm: also here, we draw random bootstrap samples from your training set. However, additionally, to the bootstrap samples, we also draw random subsets of features for training the individual trees; in bagging, we offer each tree with the complete set of features. Thanks to the random feature selection, the trees are more independent of every other compared to regular bagging, which frequently leads to better predictive performance (due to raised variance-bias trade-offs) and so it is faster than bagging and very important because each tree learns only from a subset of features.


In contrast to bagging, you employ very simple classifiers as base classifiers, so-called “weak learners.” Picture these weak learners as “decision tree stumps” – decision trees with just one splitting rule. Below, we’ll ask the probably hottest example of boosting, AdaBoost. Here, we start with one decision stump (1) and “focus” on the samples it got wrong. within the next round, we train another decision stump that attempts to urge these samples right (2); we achieve this by putting a bigger weight on these training samples. Again, this 2nd classifier will likely get another sample wrong, so you’d re-adjust the weights.

n a nutshell, we will summarise “Adaboost” as “adaptive” or “incremental” learning from mistakes. Eventually, we’ll come up with a model that features a lower bias than a private decision tree (thus, it’s less likely to underfit the training data).

Difference between Bagging and Random Forests

The fundamental difference between bagging and the random forest is that in Random forests, only a subset of features is selected randomly out of the entire and therefore the best split feature from the subset is employed to separate each node during a tree, unlike in bagging where all features are considered for splitting a node.

Bagging generally is an acronym like work that’s a portmanteau of Bootstrap and aggregation. Generally, if you’re taking a bunch of bootstrapped samples of your original dataset, fit models M1, M2,…, Mb then average all b model predictions this is often bootstrap aggregation i.e. Bagging. this is often done as a step within the Random forest model algorithm. The random forest creates bootstrap samples and across observations and for every fitted decision tree a random subsample of the covariates/features/columns are utilized in the fitting process.

The choice of every covariate is completed with uniform probability within the original bootstrap paper. So if you had 100 covariates you’d select a subset of those features each have selection probability 0.01. If you simply had 1 covariate/feature you’d select that feature with probability 1. what percentage of the covariates/features you sample out of all covariates within the data set may be a tuning parameter of the algorithm. Thus this algorithm won’t generally perform well in high-dimensional data.

Bagging (Bootstrap Aggregation) is employed when our goal is to scale back the variance of a choice tree. Here idea is to make several subsets of knowledge from the training sample chosen randomly with replacement. Now, each collection of subset data is employed to coach their decision trees. As a result, we find yourself with an ensemble of various models. Average of all the predictions from different trees are used which is more robust than one decision tree.

Random Forest is an extension over bagging. It takes one extra step where additionally to taking the random subset of knowledge, it also takes the random selection of features instead of using all features to grow trees. once you have many random trees. It’s called Random Forest.

Let’s check out the steps taken to implement Random Forest:

  • Suppose there are N observations and M features in the training data set. First, a sample from the training data set is taken randomly with replacement.
  • A subset of M features are selected randomly and whichever feature gives the simplest split is employed to separate the node iteratively.
  • The tree is grown to the most important.
  • Above steps are repeated and prediction is given supported the aggregation of predictions from n number of trees.

Advantages of using Random Forest technique:

  • Handles higher dimensionality data alright.
  • Handles missing values and maintains accuracy for missing data.

Disadvantages of using Random Forest technique:

Since the final prediction is predicated on the mean predictions from subset trees, it won’t give precise values for the regression model.

Boosting is another ensemble technique to make a set of predictors. During this technique, learners are learned sequentially with early learners fitting simple models to the info then analysing data for errors. In other words, we fit consecutive trees (random sample) and at every step, the goal is to unravel for net error from the prior three.

When an input is misclassified by a hypothesis, its weight is increased in order that the next hypothesis is more likely to classify it correctly. By combining the entire set at the top converts weak learners into better performing model.

  • Gradient Boosting is an extension over the boosting method.
  • Gradient Boosting= Gradient Descent + Boosting.

It uses a gradient descent algorithm which may optimise any differentiable loss function. An ensemble of trees are always built one by one so hence it is important and individual trees are summed sequentially. The next tree always tries to recover the loss which is the difference between actual and predicted values.

Disadvantages of using Gradient Boosting technique:

  • Prone to over-fitting.
  • Requires careful tuning of various hyper-parameters.

We hope that you liked reading this article, you can read more blogs via our blog section and learn any course of your choice by checking out our course page.

By Madhav Sabharwal