5 Evaluating our model
When building a machine learning model, it is crucial to evaluate its performance to understand how well it generalizes to unseen data. In this section, we will discuss various evaluation metrics and techniques that can be used to assess the performance of our model.
Recall that our goal when building a machine learning model is to make good predictions for new and unseen data. To evaluate how well our model performs on unseen data, we can use a variety of metrics depending on the type of problem we are trying to solve (e.g., classification, regression).
As we have seen in previous chapters, we typically split our dataset into a training set and a test set. The training set is used to train the model, while the test set is used to evaluate its performance. It is important to ensure that the test set is representative of the data the model will encounter in real-world applications, and that it has not been used during the training process to avoid overfitting.
5.1 Evaluation Metrics for Classification
For classification problems, we often want to match the predicted class labels to the true class labels, and evaluate how well our model does this. We start by building a so-called confusion matrix (Figure 5.1), which is a table that summarizes the performance of a classification model by showing the counts of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN).
From the confusion matrix, we can derive several important metrics:
True Positive (TP): The number of instances where the model correctly predicted the positive class.
True Negative (TN): The number of instances where the model correctly predicted the negative class.
False Positive (FP): The number of instances where the model incorrectly predicted the positive class (Type I error).
False Negative (FN): The number of instances where the model incorrectly predicted the negative class (Type II error).
Accuracy: The proportion of correct predictions (both true positives and true negatives) out of the total number of predictions. This metric is sensitive to class imbalance, as it can be misleading when one class is much more frequent than the other. \[A = (TP + TN) / (TP + TN + FP + FN).\]
Balanced Accuracy: The average of the true positive rate (sensitivity) and true negative rate (specificity), providing a more balanced view of performance in the presence of class imbalance. \[BA = \frac{(TPR + TNR)}{2}.\]
Precision (Positive Predictive Value): The proportion of true positives out of all predicted positives. \[P = TP / (TP + FP).\]
Recall (Sensitivity, True Positive Rate): The proportion of true positives out of all actual positives. \[R = TP / (TP + FN).\]
F1 Score: The harmonic mean of precision and recall, providing a balance between the two metrics. \[F1 = 2 * (P * R) / (P + R).\]
Matthews Correlation Coefficient (MCC): A measure of the quality of binary classifications, taking into account true and false positives and negatives. It ranges from -1 to +1, where +1 indicates perfect prediction, 0 indicates random prediction, and -1 indicates total disagreement between prediction and observation. \[MCC = \frac{TP \cdot TN - FP \cdot FN}{ \sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}.\]
A complete diagnostic classification matrix with all these metrics and their definitions is found in Figure 5.3.
Another important metric for evaluating classification models is the Receiver Operating Characteristic (ROC) curve and the associated Area Under the Curve (AUC):
Area Under the Receiver Operating Characteristic Curve (AUC-ROC): A measure of the model’s ability to distinguish between classes (Figure 5.2 (a)). The ROC curve plots the true positive rate against the false positive rate at various threshold settings, and the AUC represents the area under this curve. The AUC ranges from 0 to 1, with higher values indicating better performance. The AUC is highly affected by class imbalance, and it can be misleading in cases where the positive class is rare. In such cases, other metrics like Precision-Recall AUC may be more informative.
Precision-Recall AUC: This metric is particularly useful when dealing with imbalanced datasets, as it focuses on the performance of the positive class. The Precision-Recall curve plots precision against recall at various threshold settings (Figure 5.2 (b)), and the area under this curve provides a single scalar value to summarize the model’s performance in terms of precision and recall. The Precision-Recall AUC ranges from 0 to 1, with higher values indicating better performance.
5.2 Evaluation Metrics for Regression
For regression problems, we typically evaluate the performance of our model using metrics that measure the difference between the predicted values and the actual values. Some common regression evaluation metrics include: - Mean Absolute Error (MAE): The average of the absolute differences between the predicted values and the actual values. \[MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|.\] - Mean Squared Error (MSE): The average of the squared differences between the predicted values and the actual values. \[MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2.\] - Root Mean Squared Error (RMSE): The square root of the mean squared error, providing a measure of the average magnitude of the errors in the same units as the target variable. \[RMSE = \sqrt{MSE}.\]
R-squared (Coefficient of Determination): A measure of how well the regression model fits the data, representing the proportion of the variance in the dependent variable that is predictable from the independent variables. \[R^2 = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y _i - \bar{y})^2}.\]
Adjusted R-squared: A modified version of R-squared that adjusts for the number of predictors in the model, providing a more accurate measure of model fit when multiple predictors are used. \[R^2_a = 1 - \left( \frac{SS_{res} / (n - p - 1)}{SS_{tot} / (n - 1 )} \right) = 1 - \left( \frac{(1 - R^2)(n - 1)}{n - p - 1} \right).\]
When evaluating regression models, it is important to consider the context of the problem and the specific requirements of the application. For example, in some cases, we may prioritize minimizing large errors (in which case RMSE may be more appropriate), while in other cases, we may want to focus on the average error (in which case MAE may be more suitable). Additionally, R-squared and Adjusted R-squared can provide insights into how well the model explains the variance in the data, but they should be interpreted with caution, especially when comparing models with different numbers of predictors.
5.3 Comprehensive Confusion Matrix and Metrics
