XgboostC

Step-1 : Understanding Gradient Boosting Before delving into XGBoost, it's important to understand the basics of gradient boosting. Gradient boosting is an ensemble learning technique that combines the predictions of multiple weak learners (typically decision trees) to create a strong learner. The idea is to iteratively add new trees to the model, with each tree trying to correct the mistakes of the previous ones.

Step-2 : Objective Function The core of XGBoost is its objective function, which needs to be optimized during the training process. For binary classification, the objective function of XGBoost is the binary logistic loss, which is given by:

● n is the number of training examples.
● y_i is the true label of the i-th example(0 or 1 for binary classification).
● y_i is the predicted probability of the i-th example belonging to the positive class (obtained from the sum of predictions of all trees in the ensemble).
● The regularization term helps prevent overfitting and is controlled by hyperparameters (e.g., L1 or L2 regularization terms).

Step-3 : XGBoost Algorithm Steps The XGBoost algorithm consists of the following major steps:

1. Initialize the Model: Start by initializing the model with a simple estimator, often a decision tree with only one leaf (also known as a stump). This initial prediction will be improved in subsequent steps.
2. Compute Pseudo-Residuals: Compute the negative gradient (pseudo-residuals) of the loss function with respect to the current predictions. The pseudo-residuals indicate how much the current model's predictions need to be adjusted to better fit the true labels.
3. Fit a New Tree: Train a new decision tree using the computed pseudo-residuals as the target values. The tree is fit to minimize the objective function, considering the current model's predictions as the base prediction. The tree is added to the ensemble.
4. Update Predictions: Update the predictions of the model by adding the predictions from the newly added tree, weighted by a learning rate (to control the step size of each tree's contribution).
5. Regularization: Apply regularization to prevent overfitting. Regularization terms are added to the objective function during training.
6. Iterate: Repeat steps 2 to 5 for a predefined number of boosting rounds (iterations), or until a stopping condition is met (e.g., the objective function improvement is below a threshold).
7. Make Predictions: Once the boosting process is complete, use the final ensemble of trees to make predictions on new data. The final predictions are obtained by summing the predictions of all the trees, weighted by their respective learning rates.

Step-4 : XGBoost Hyperparameters XGBoost provides several hyperparameters to control the training process and the complexity of the model. Some important hyperparameters include:

● learning_rate: Controls the step size at each iteration (typical values: 0.01 to 0.3).
● max_depth: Maximum depth of each decision tree (higher values may lead to overfitting).
● min_child_weight: Minimum sum of instance weight (hessian) needed in a child (node).
● subsample: Fraction of training examples used for each tree (helps reduce overfitting).
● colsample_bytree: Fraction of features (columns) used for each tree.
● gamma: Minimum loss reduction required to make a further partition on a leaf node.
● lambda: L2 regularization term.
● alpha: L1 regularization term.