Overfitting , Underfitting Bias & Variance Understanding by CHIRAG

 What is Overfitting?


Overfitting means model has High accuracy score on training data but low accuracy score on test data. (That means our model best fit line satisfied all the training data points . ie. Low Training Error & High Testing Error & High Training Accuracy & Low Test Accuracy)





Reasons for Overfitting:
  1.  High variance(High Test Error) .
  2.  Low Bias(Low Training Error).
  3. The model is too complex.
  4. The size of the training data.

What is Bias?

Bias is the difference between the average prediction of our model and the correct value which we are trying to predict. A model with high bias pays very little attention to the training data and oversimplifies the model.

Let’s assume we have trained the model and are trying to predict values with input ‘x_train’. The predicted values are y_predicted.

 Bias is the error rate of y_predicted and y_train.

In simple terms. 

think of bias as the error rate of the training data.

When the error rate is high, we call it High Bias and when the error rate is low, we call it Low Bias

What is a Variance?

Variance is the variability of model prediction for a given data point or a value that tells us the spread of our data. A model with high variance pays a lot of attention to training data and does not generalize on the data which it hasn’t seen before.

Let’s assume we have trained the model and this time we are trying to predict values with input ‘x_test’. Again, the predicted values are y_predicted.

Variance is the error rate of the y_predicted and y_test

In simple terms,


 think of variance as the error rate of the testing data.

When the error rate is high, we call it High Variance and when the error rate is low, we call it Low Variance.

    Techniques to Reduce Overfitting

    1. Increase training data.
    2. Reduce model complexity.
    3. Early stopping during the training phase (have an eye over the loss over the training period as soon as loss begins to increase stop training).
    4. Ridge Regularization and Lasso Regularization.
    5. Use dropout for neural networks to tackle overfitting.

    What is Underfitting?


    Underfitting means model has low accuracy score on training data and test data both.(That Means our Trained model best fit line does not satisfy training data points as well as test data points. ie. High Training Error & High Test Error , Low Training Accuracy & Low Test Accuracy )
    Reasons for Underfitting:
    1.  High variance(High Test Error) .
    2.  High Bias(High Training Error).
    3. The model is too complex.
    4. The size of the training data.

    Techniques to Reduce Underfitting

    1. Increase model complexity.
    2. Increase the number of features, performing feature engineering.
    3. Remove noise from the data.
    4. Increase the number of epochs or increase the duration of training to get better results.
    What is a Balanced Model?

    A Balanced model is a model which performs well both on training and test set.

    A Balanced model will have low bias and low variance.

    Let’s take 3 examples to understand Overfitting, Underfitting and Balanced Model.

    1. A model with training error : 2% and test error : 20%
      Less Training Error — Low Bias
      High Test Error — High Variance
      This is an Overfitted Model.

    2. A model with training error : 30% and test error : 30%
      High Training Error — High Bias
      High Test Error — High Variance
      This is an Underfitted model.

    3. A model with training error : 4% and test error : 3%
      Less Training Error — Low Bias
      Less Test Error — Low Variance
      This is a Balanced model.

    Why Bias Variance Tradeoff?

    If our model is too simple and has very few parameters then it may have high bias and low variance. On the other hand if our model has large number of parameters then it’s going to have high variance and low bias. So we need to find the right/good balance without overfitting and underfitting the data.

    This tradeoff in complexity is why there is a tradeoff between bias and variance. An algorithm can’t be more complex and less complex at the same time.

    Total Error


    To build a good model, we need to find a good balance between bias and variance such that it minimizes the total error.

    Total Error = Bias^2 + Variance+ Irreductable Error

    An optimal balance of bias and variance would never overfit or underfit the model.

    Therefore understanding bias and variance is critical for understanding the behavior of prediction models. 

    ========================================================================

    I Hope you guys like it, stay connected to my next article.

    Thank You




    Comments

    Post a Comment

    Popular posts from this blog

    Linear Regression Understanding by CHIRAG

    Image
      Linear Regression 1 .)Linear regression is a supervised learning algorithm used when target / dependent variable continues real number. 2.)Linear regression is a type of statistical analysis used to predict the relationship between dependent and independent Variables using BEST FIT LINE. 3.) It work on the principle of ordinary least square (OLS)/ Mean square errror (MSE) Simple Linear Regression In a simple linear regression, there is one independent variable and one dependent variable. The model estimates the slope and intercept of the line of best fit, which represents the relationship between the variables. The slope represents the change in the dependent variable for each unit change in the independent variable, while the intercept represents the predicted value of the dependent variable when the independent variable is zero. Mathematically, we can represent a linear regression as: y= a0+a1x+ ε Y= Dependent Variable (Target Variable) X= Independent Variable (predictor Variab...
    Read more

    Understanding Confusion Matrix by CHIRAG

    Image
    What Is a Confusion Matrix? A confusion matrix is a performance evaluation tool in machine learning, representing the accuracy of a classification model.  It displays the number of true positives, true negatives, false positives, and false negatives. A Confusion matrix is an N x N matrix used for evaluating the performance of a classification model, where N is the total number of target classes.     For a binary classification problem, we would have a 2 x 2 matrix, as shown below, with 4 values: Let’s understanding this matrix: The target variable has two values: 0 & 1 The  columns  represent the  Predicted values  of the target variable The  rows  represent the  Actual values  of the target variable. But wait – what’s TP, FP, FN, and TN here? That’s the crucial part of a confusion matrix. Let’s understand each term below. Important Terms in a Confusion Matrix True Positive (TP)  The predicted value matches the actual value...
    Read more