• To completely and correctly learn any task follow the Learning Cycle

• The four main phases of a Learning Cycle to completely and correctly learn any task are

• • Training Phase 

• • Testing Phase

• • Applicants Phase

• • Feedback Phase

• The main goal of building a Model (Training / Learning) is to use it in doing Real-world Tasks with good Accuracy 

• • No one can perfectly predict, that if a Model performs well in Training Phase, then it will also perform well in Real-world 

• • However, before deploying a Model in the Real-world, it is important to know 

• • • How well will it perform on unseen Real-time Data?

• • To judge / estimate the performance of a Model (or h) in Real-world (Application Phase)

• • •  Evaluate the Model (or h) on large Test Data (Testing Phase)

• • If (Model Performance = Good AND Test Data = Large)
    • Then

                   Use the Model in Real-world

• • Else 

                   Refine (re-train) the Model

• Recall – Machine Learning Assumption

• • If a Model (or h) performs well on large Test Data, it will also perform well on unseen Real-world Data 

• Again, this is an assumption and we are not 100% sure that if a Model performs well on large Test Data will definitely perform well on Real-world Data

• • Therefore, it is useful to take continuous Feedback on deployed Model (Feedback Phase) and keep on improving it

• Two main advantages of Evaluating Hypothesis / Model are

• • We get the answer to an important question i.e.

• • • Should we rely on predictions of Hypothesis / Model when deployed in Real-world?

• • Machine Learning Algorithms may rely on Evaluation to refine Hypothesis (h)

• What we evaluate a Hypothesis h, we want to know 

• • How accurately will it classify future unseen instances?

• • • i.e. Estimate of Error (EoE)

• • How accurate is our Estimate of Error (EoE)?

• • • I.e. what Margin of Error (±? %) is associated with our Estimate of Error (EoE)? (we call it Error in Estimate of Error (EoE))

• True Error

• • Error computed on entire Population 

• Sample Error

• • Error computed on Sample Data 

• Since we cannot acquire entire Population

• • Therefore, we cannot calculate True Error

• Calculate Sample Error in such a way that 

• • Sample Error estimates True Error well 

• Statistical Theory tells us that Sample Error can estimate True Error well if the following two conditions are fulfilled

• • Condition 01 – the n instances in Sample S are drawn 

• • • independently of one another

• • • independently of h

• • • according to Probability Distribution D 

• • Condition 02

• • • n ≥ 30

• Estimate of Sample Error (EoSE) can be calculated as follows

• • Step 1: Randomly select a Representative Sample S from the Population

• • Step 2: Calculate Sample Error on Representative Sample S drawn in Step 1

• Estimate cannot be perfect and will contain Error 

• • Therefore, calculate Error in Estimate of Sample Error (EoSE)
• Error in Estimate of Sample Error (EoSE) can be calculated using Confidence Interval 
• The Most Probable Value of True Error is Sample Error with approximately N% Probability (Confidence Level), True Error lies in interval 

• • where n represents size of Sample

• • errors(h) represents Sample Error

• The choice of Confidence Level depends on the field of study 

• Generally, the most common Confidence Level used by Researchers is 95% 

• Our goal is to have a

• •  Small Interval with High Confidence

• The two main diseases in Machine Learning are

• • Overfitting

• • Underfitting 

• The condition when a Machine Learning Algorithm tries to remember all the Training Examples from the Training Data (Rote Learning) is known as Overfitting of the Model (h)

• • Overfitting happens when our 

• • • Model (h) has lot of features or

• • • Model (h) is too complex 

• The condition when a Machine Learning Algorithm could not learn the correlations between Attributes / Features properly then it is known as Underfitting of the Model (h)

• • Underfitting happens when our 

• • • Model misses the trends or patterns in the Training Data and could not generalize well for the Training Examples

• To overcome the problems of Overfitting and Underfitting, we 

• • Use Train-Test Split Approach

• Train-Test Split Approach, splits the Sample Data into two sets: (1) Train Set and (2) Test Set

• Two main variations of Train-Test Split Approach are

• • Random Split Approach

• • Class Balanced Split Approach

• Class Balanced Split Approach should be preferred over Random Split Approach

• Train-Test Split Ratio determines what percentage of the Sample Data will be used as Train Set and what percentage of the Sample Data will be used as Test Set

• The Train-Test Split Ratio may vary from Machine Learning Problem to Machine Learning Problem

• • e.g. 70%-30%, 80%-20%, 90%-10% etc.

• Most Common Train-Test Split Ratio

• • Use 2 / 3 of Sample Data as Train Set 

• • Use 1 / 3 of Sample Data as Test Set

• Train-Test Split Approach provides high variance in Estimate of Sample Error since 

• •  Changing which examples happens to be in Train Set can significantly change Sample Error

• To overcome the problem of high variance in Estimate of Sample Error calculated using Train-Test Split Approach, we 

• • Use K-fold Cross Validation Approach

• K-fold Cross Validation Approach works as follows

• • Step 1: Split Train Set into K equal folds (or partitions)

• • Step 2: Use one of the folds (kth fold) as the Test Set and union of remaining folds (k – 1 folds) as Training Set 

• • Step 3: Calculate Error of Model (h)

• • Step 4: Repeat Steps 2 and 3, to choose Train Sets and Test Sets from different folds, and calculate Error K-times 

• • Step 5: Calculate Average Error 

• Important Note 

• • In each fold, there must be at least 30 instances 

• • All K-folds must be disjoint i.e. instance appearing in one fold must not appear in any other fold

•  Empirical study showed that best value for 

• • K is 10

• K-fold Cross Validation Approach is a better estimator of Error since 

• •  All data is used for both Training and Testing

• K-fold Cross Validation Approach is computationally expensive since

• • we have to repeat Training and Testing Phases K-times 

• It is suitable to use Train-Test Split Approach in the following situations

• • When Training Time is Very Large 

• • Organizing International Competitions 

• • Having Very Huge Sample Data

• It is suitable to use K-fold Cross Validation Approach in the following situations

• • When Training Time is Not Very Large 

• • Having Sample Data that is Not Very Huge 

• To compare various Machine Learning Algorithms, following things must be same 

• • Train Set

• • Test Set

• • Evaluation Measure

• • Evaluation Methodology

• Important Note

• • If any of the above things are not same then it will 

• • •  Not be a valid comparison 

• Some of the most popular and widely used Evaluation Measures for Classification Problems are

• • Baseline Accuracy 

• • Accuracy

• • Precision

• • Recall

• • F1

• • Area Under the Curve (AUC)

• Baseline Accuracy (a.k.a. Majority Class Categorization (MCC)) is calculated by assigning the label of Majority Class to all the Test Instances

• BA provides a simple baseline to compare proposed Machine Learning Algorithms 

• Accuracy is defined as the proportion of correctly classified Test instances

• Accuracy = 1 – Error

• Accuracy evaluation measure is more suitable to use for evaluation of Machine Learning Algorithms when we have

• • Balanced Data 

• Two main limitations of Accuracy measure are

• • Accuracy fails to accurately evaluate a Machine Learning Algorithm when Test Data is highly unbalanced 

• • Accuracy ignores possibility of different misclassification costs 

• To overcome the limitations of Accuracy measure, we use

• • Confusion Matrix

• A Confusion Matrix is a table used to describe the performance of a Classification Model (or Classifier) on a Set of Test Examples (Test Data), whose Actual Values (or True Values) are known 

• Some of the main advantages of Accuracy measure are

• • Confusion Matrix allows us to visualize the performance of a Model / Classifier

• • Confusion Matrix allows to separately get insights into the Errors made by each Class 

• • Confusion Matrix gives insights to both 

• • •  Errors made by a Model / Classifier and 

• • •  Types of Errors made by a Model / Classifier

• • Confusion Matrix allows us to compute many different Evaluation Measures including

• • • Baseline Accuracy

• • • Accuracy

• • • True Positive Rate (or Recall)

• • • True Negative Rate 

• • • False Positive Rate 

• • • False Negative Rate 

• • • Precision

• • • F1

• Recall or True Positive Rate (TPR) or Sensitivity is the proportion of Positive cases that were correctly classified

• False Positive Rate (FPR) is the proportion of Negative cases that were incorrectly classified as Positive 

• True Negative Rate (TNR) or Specificity is defined as the proportion of Negatives cases that were classified correctly

• False Negative Rate (FNR) is the proportion of Positive cases that were incorrectly classified as Negative

• Precision (P) is the proportion of the predicted Positive cases that were correct

• F-measure is the Harmonic Mean of Precision and Recall 

• • • where controls relative weight assigned to Precision and Recall 

• F1-measure is that, When we assign same weights to Precision and Recall i.e. β = 1, the F-measure becomes F1-measure

• When considering Misclassifications

• •  Positive costs may be more or less important than misclassifying (or incorrectly predicting) Negative costs 

•  Before Training your Model, you must be very clear about the Misclassification Costs, otherwise

• • Your Model will fail to perform well in Real-world (i.e. Application Phase)

• The problem of considering different misclassification costs can be handled using

• • ROC Curves 

• • Precision-Recall Curves

• ROC Curve summarizes the trade-off between the True Positive Rate (TPR) and False Positive Rate (FPR) for a Classifier / Model using different Probability Thresholds

• ROC Curves are more suitable to use when we have Balanced Data

• Precision-Recall Curve summarizes the trade-off between Precision and Recall (or True Positive Rate) for a Classifier / Model using different Probability Thresholds

• Precision-Recall Curves are more suitable to use when we have Highly Unbalanced Data

• Area Under the ROC Curve (AUC) is defined as the probability that randomly chosen Positive instance is ranked above than the randomly selected Negative one 

• AUC tells how much Model is capable of distinguishing between Classes?

• AUC score is computed using True Positive Rate (TPR) and False Positive Rate (FPR)

• Range of AUC Score

• • Range of AUC Score is [0 – 1]

• • • 0 means All Predictions of Model are Wrong

• • • 1 means All Predictions of Model are Correct

• What is a good AUC Score?

• • AUC Score = 0.5 

• • • suggests No Discrimination (i.e., Model do not have the ability to differentiate Positive instances from Negative ones)

• • AUC Score = 0.7 to 0.8 

• • • considered Acceptable

• • AUC Score = 0.8 to 0.9 

• • • considered Excellent

• • AUC Score = Greater Than 0.9 

• • • considered Outstanding

• Some of the most popular and widely used Evaluation Measures for Regression Problems are

• • Mean Absolute Error (MAE)

• • Mean Squared Error (MSE)

• • Root Mean Squared Error (RMSE)

• • R2 or Coefficient of Determination

• • Adjusted R2

• Absolute Error (AE) is the difference between the Actual Value and the Predicted Value 

• Mean Absolute Error (MAE) is the average of all Absolute Errors

• Square Error (SE) is the Square of difference between the Actual Value and the Predicted Value 

• Mean Square Error (MSE) is the average of all Square Errors

• Root Mean Square Error (RMSE) is the Square root of all Mean Square Errors

• Some of the most popular and widely used Evaluation Measures for Sequence-to-Sequence Problems are

• • ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

• • BLEU (Bi-Lingual Evaluation Understudy)

• • METEOR (Metric for Evaluation of Translation with Explicit Ordering)

• ROUGE is a de facto standard to automatically evaluate the performance of Text Summarization Systems 

• • Normally, we calculate 

• • • ROUGE-1

• • • ROUGE-2

• • • ROUGE-L

• • Average F1 scores are reported for ROUGE-1, ROUGE-2 and ROUGE-L metrics

• BLEU is a de facto standard to automatically evaluate the performance of Machine Translation Systems 

• Normally, we calculate

• • BLEU-1

• • BLEU-2

• • BLEU-3

• • BLEU-4