AdaBoostR

Step - 1 : Initialize sample weights : - Assign equal weights to all training examples. For a dataset with N samples, each sample weight (w_i) is initially set to 1/N.
Step - 2 : Boosting iterations : - For a pre-defined number of boosting iterations (denoted by T), repeat steps 3 to 6.
Step - 3 : Train a weak learner : - Train a weak learner (e.g., decision stump) on the training data. A weak learner for regression is a simple model that can predict numeric values slightly better than random guessing. Typically, a decision stump is used, which is a decision tree with a single split (one level).
Step - 4 : Compute the weighted error : - Calculate the weighted error (ε_t) of the weak learner on the training data. Weighted error is the sum of weights of the absolute differences between the predicted values and the actual target values, divided by the sum of all weights.

Weighted Error (ε_t) = Σ (w_i * |y_i - h_t(x_i)|) / Σ w_i

where y_i is the actual target value for sample i,
h_t(x_i) is the prediction of the weak learner t for input x_i,
and w_i is the sample weight.

Step - 5 : Calculate the weak learner's weight in the final regressor : - Compute the weight (α_t) of the weak learner based on its performance (weighted error) during training.

Weak Learner Weight (α_t) = 0.5 * ln((1 - ε_t) / ε_t)

Note: If the weighted error (ε_t) is very close to 0, the weak learner's weight (α_t) approaches infinity, indicating that its predictions are perfect, and it will have a significant impact on the final regression model.

Step - 6 : Update sample weights : - Update the sample weights based on the performance of the weak learner. - Increase the weights of samples with larger absolute errors, so they get more emphasis in the next iteration. - Decrease the weights of samples with smaller absolute errors to reduce their impact in the next iteration.

For sample i with larger absolute error => w_i = w_i * exp(α_t)

For sample i with smaller absolute error => w_i = w_i * exp(-α_t)

Normalize the sample weights so that they sum up to 1.

Step - 7 : Combine weak learners into a strong regressor : - Combine the predictions of all weak learners using their weights (α_t) obtained in Step 5. - The final regressor is a weighted sum of weak learner predictions:

F(x) = Σ (α_t * h_t(x))

where F(x) is the output of the final regressor, h_t(x) is the prediction of the weak learner t for input x, and α_t is the weight of the weak learner t.

Step - 8 : Make predictions : - To make predictions for new data, pass the input through the combined weak learners and their respective weights as calculated in Step 7. - The final prediction is the weighted sum of weak learner predictions.