Posts

Showing posts from September, 2014

Gradient Boosting: Fostering accuracy even further

Image
As in many real-world situations, union makes algorithms stronger. With this philosophy in mind, ensemble methods combine several weak classifiers into a massive one---in terms of accuracy. In the last post we learnt a primer with Random Forest . Therefore, the next cornerstone is gradient boosting. I mentioned Gradient Boosting many times in this blog, but I only commented the fundamental ideas, without discussing further the details. In this entry I will share my two cents. Let me introduce a little bit of history, first: recall the Kaggle-Higgs competition . The top scores in the leaderboard have been obtained by using distinct forms of gradient boosting, and XGBoost is the direct responsible of many of these. The question is, hence, how does this algorithm work ? Figure 1. A high-level description of the Gradient Boosting method I programmed. Click to enlarge. Informally, Gradient Boosting generates a sequence of classifiers in the form of an additive expansion, that i