Extreme Gradient Boosting

Extreme Gradient Boosting(XGB)

XGBoost is a well optimized and distributed gradient boosting library that is designed to be highly efficient, portable and even flexible. It is implemented under the Gradient Boosting machines framework also known as (GBMs).

XGBoost can be thought of as an advanced version of gradient boosting( That's Why referred to as Extreme gradient boosting). It is also a great combination of hardware and software optimization techniques which results in much better results by using less computing resources and also producing results in the shortest amount of time.

Why does XGBoost work so well?

XGBoost and Gradient Boosting Machines are both ensemble tree methods that simply apply the principle of improving or boosting weak learners like CART using the gradient descent architecture. XGBoost algorithm improves upon the base Gradient boosting machines framework through algorithmic enhancements and system optimizations.

Some of the system optimizations done to enhance the performance of XGBoost are as follows

1. Parallelization: In XGBoost the sequential tree is built using parallelized implementation. This is also possible due to the interchangeable nature of loops which are used for building base learners; the outer loop that enumerates the leaf nodes of a tree, and also the second inner loop that calculates the features. 
2. Tree Pruning: The stopping criterion for the tree splitting within the Gradient Boosting framework is also greedy in nature and it totally depends on the negative loss criterion at the point of split. XGBoost also uses some unique parameters like ‘max_depth’ parameter as specified instead of criterion first which starts pruning the trees backward. This approach is also known as ‘depth-first’ which improves computational performance very significantly.
3. Hardware Optimization: This algorithm has been designed and used  to make efficient use of hardware resources. It is accomplished by cache awareness and allocating internal buffers in each thread to store gradient statistics.Also some  further enhancements have also been done like ‘out-of-core’ computing to optimize all the available disk space while handling big data-frames which do not fit into memory.

Also some of the algorithmic enhancements in XGBoost algorithm which makes this stand out from other algorithms are as follows:

1. Cross-validation: The algorithm also comes in with a built-in cross-validation method at each iteration, taking away the need to explicitly program this search and to specify the exact number of boosting iterations required in a single run.
2. Regularization: It has the ability to penalize many complex models through both LASSO (L1) and Ridge (L2) regularization that is a very helpful approach to  prevent overfitting.
3. Weighted Quantile Sketch: XGBoost also employs the distributed weighted Quantile Sketch algorithm that can much effectively find out the optimal split points among weighted datasets.
4. Sparsity Awareness: This algorithm also naturally admits sparse features for inputs by automatically ‘learning’ best missing value depending on training loss and handles different types of sparsity patterns in the data much more efficiently than any other algorithm.

Conclusion:

XGBoost performs really well for the structured or tabular datasets on both classification as well as regression predictive modeling problems.That is why it is considered one of the best machine learning algorithms in terms of dealing with the tabular data both for regression as well as for classification.