An important technique under machine learning, Random Forest Algorithm is useful for both classification and regression tasks
There are many tools and techniques available under machine learning. One of these is Random Forest Algorithm, which is classified under ensemble learning. It works by combining the predictions derived from multiple individual models. This in turn helps achieve a highly accurate and reliable final output. This approach to use multiple models delivers better results in comparison to relying on a single model. Random Forest Algorithm was first introduced in 2001 by Leo Breiman. It is one of the most widely used and effective ready to use machine learning techniques.
Core idea of Random Forest Algorithm
There are two main concepts powering Random Forest Algorithm. One of these is bagging or bootstrap aggregating. This involves training each tree on a random bootstrap sample of the data. This can also be described as sampling with replacement. The second concept is ‘feature randomness’, also known as Random Subspace method. As per this concept, only a random subset of features is considered at each split in a tree.
With a dual randomness concept, the correlation between trees is reduced. This approach also helps lower variance. In effect, the Random Forest Algorithm helps remove the primary weaknesses of individual decision trees.
How Random Forest Algorithm works?
In the first step, bootstrap sampling is carried out. From a training set of size (n), a select number of bootstrap samples are drawn. Each of these are of size (n), with replacement. In terms of averages, each bootstrap sample has around 63.2% unique observations. The rest, around 36.8%, are classified as out-of-bag (OOB) samples.
In the next step, a decision tree is grown on each bootstrap sample. For this, a standard decision tree growing algorithm like CART can be used. At every node, (m) features are randomly selected from the total (p) features (m≪p). Next, the best split among those (m) features are chosen, as per purity criterion. For classification, Gini impurity or entropy is used. For regression, variance reduction is used.
In the next step, the tree is grown to its maximum depth. Or until a stopping criterion. Usually, the trees are unpruned. After this, the aggregate predictions are made. Classification is the majority vote across the (B) trees. In regression, the average of the predictions from the (B) trees is considered.
The final prediction is derived using the formula as shown below. This is the final prediction for a new point (x).
Advantages and disadvantages of Random Forest Algorithm
Like everything else, Random Forest Algorithm has its advantages and disadvantages. Talking first about advantages, the technique works well for predictive performance on many tabular datasets. Random Forest Algorithm can easily handle multi-dimensional data and mixed feature types. It has built-in feature importance and can provide OOB error estimates. It is more robust than outliers and noise. Random Forest Algorithm ensures trees are independent and works well with missing values. It requires very little tuning in comparison to other algorithms.
Coming to the disadvantages, Random Forest Algorithm is less interpretable than a single decision tree. It can take up high CPU and memory usage, especially with very large n_estimators or deep trees. There is a tendency to overfit noisy data in case the trees are too deep or if randomness is less. Prediction can take longer, especially in comparison to linear models or shallow trees. Random Forest Algorithm is usually not considered ideal for extrapolation in regression.
.png)
