Cyber crime cases

Confusion matrix and its types of error.

cyber crime

What is cyber crime?

In today’s digital world, people are trading convenience for privacy — which means they’re sharing more and more information about themselves. While we can all appreciate one-click checkouts and saved passwords, that mental “autopilot” can sometimes steer us wrong. While we were all once cautious of anyone asking for information about ourselves, now we assume that’s just the cost of participation — you ask for my favorite book, I give it to you. You show me my password has expired, I try to log in with the link you conveniently provided to me. You send me an email from my boss, you get my attention and response. But this autopilot is what cybercriminals and hackers are effortlessly turning into hundreds of millions of dollars every year from people like you and me all around the world.

Top 5 Cybercrimes cases:

1. Phishing Scams
2. Website Spoofing
3. Ransomware
4. Malware
5. IOT Hacking

Thus, detecting various cyber-attacks in a network is very necessary. The application of Machine Learning model in building an effective Intrusion Detection System (IDS) comes into play. A binary classification model can be used to identify what is happening in the network i.e., if there is any attack or not. For evaluation of model, one of the metric used is Confusion matrix.

Confusion matrix presents a table layout of different outcome of prediction and result of classification problem and helps to visualize its outcome.

Let’s understand the terms used here:

• In two-class problem, such as attack state, we assign the event normal as “positive” and anomaly as “negative“.
• “True Positive” for correctly predicted event values.
• “False Positive” for incorrectly predicted event values.
• “True Negative” for correctly predicted no-event values.
• “False Negative” for incorrectly predicted no-event values.

Confusion matrices have two types of errors: Type I and Type II

Now lets see these terms and their significance under the light of cyber attack prediction for better understanding.

IDS or Intrusion Detection System checks for any malicious activity on the system. It monitors the packets coming over internet using some ML model and predicts whether it is normal or an anomaly.

Lets say our model created the following confusion matrix for total of 165 packets it examined.

A total of 165 packets were analyzed by our model in IDS system which have been classified in the above confusion matrix.

• “Positive” -> Model predicted no attack.
• “Negative” -> Model predicted attack.
• True Negative: Out of 55 times for which model predicted attack will take place, 50 predictions were ‘True’ which means 50 times attack actually took place. Due to prediction, Security Operations Centre (SOC) will receive notification and can prevent the attack.
• False Negative: Out of 55 times for which model predicted attack will take place, 5 times the attack didn’t happen. This can be considered as “False Alarm” and also Type II error.

• True Positive: The model predicted 110 times that attack wouldn’t take place, out of which 100 times actually no attack happened. These are the correct predictions.
• False Positive: 10 times the attack actually took place when the model had predicted that no attack will happen. It is also called as Type I error.

Type I error:

This type of error can prove to be very dangerous. Our system predicted no attack but in real attack takes place, in that case no notification would have reached the security team and nothing can be done to prevent it. The False Positive cases above fall in this category and thus one of the aim of model is to minimize this value.

Type II error:

This type of error are not very dangerous as our system is protected in reality but model predicted an attack. the team would get notified and check for any malicious activity. This doesn’t cause any harm. They can be termed as False Alarm.]

spam detector:

Spam email comes in different flavors. Many are just annoying messages aiming to draw attention to a cause or spread false information. Some of them are phishing emails with the intent of luring the recipient into clicking on a malicious link or downloading a malware.

Machine learning algorithms use statistical models to classify data. In the case of spam detection, a trained machine learning model must be able to determine whether the sequence of words found in an email are closer to those found in spam emails or safe ones.

Let’s try to understand it for a binary (binary means two) response case of an email spam classification. In binary problems, we generally represent our primary target (favorable) as positive and others (unfavorable) as negative. In this case, our primary aim is to identify Spam emails. Hence, we mark it as (+ve) and Not Spam emails as (-ve). In this case, a confusion matrix represents four different combinations of predicted and actual values. As an example, assume that a total of 300 emails were used to evaluate a model. These emails were hand-labeled as either Spam or Not Spam.

Evaluating a model using Confusion Matrix

In the above example, a total of 300 emails were used to evaluate the model. Let us now see what metrics are generally used to evaluate our models using this matrix. We will later do some elementary calculations to understand this better.

Accuracy:

It says about how much we predicted correctly. It is the ratio of correct predictions to the total predictions made.

Accuracy=TP + TN TP + TN + FP + FN Accuracy=TP + TN TP + TN + FP + FN

Error Rate:

It signifies the proportion of erroneous predictions a model makes. It is the ratio of wrong predictions to the total predictions made.

Error Rate=FP + FN TP + TN + FP + FN Error Rate=FP + FN TP + TN + FP + FN

Often, we need to understand metrics that define the error in predictions of our primary target class only. Precision, Recall and F1 score do precisely the same.

Precision:

It signifies how much of total primary target predictions are correct predictions. It is the ratio of the true positive predictions to the total primary target predictions made.

Precision=TPTP + FPPrecision=TPTP + FP

Recall:

It signifies how much of the actual primary target samples were predicted as a primary target. It is the ratio of the true (+ve) predictions to the total (+ve) samples in the data.

Recall=TPTP + FN Recall=TPTP + FN

F1 score:

In most real-life cases, we get high accuracy while having low Precision and Recall. In these cases, F1 is a go-to metric. It is the weighted average of Precision and Recall.

F1 Score=2 x Precision x Recall Precision + RecallF1 Score=2 x Precision x Recall Precision + Recall

Specificity and Sensitivity:

Sensitivity is the same as Recall, which is defined only with respect to the positive responses.

Specificity, on the other hand, does the same for the negative responses. It measures the proportion of negative responses that are correctly predicted as negative.

Specificity=TN TN + FP

Let us now do some calculations on the email spam case as described above.

• Accuracy = (30+250)/300 = 0.933
• Error Rate = (12+8)/300 = 0.066
• Precision = 30/(30+12) = 0.714
• Recall (Sensitivity) = 30/(30+8) = 0.789
• F1 score = 2*0.714*0.789/(0.714+0.789) = 0.749
• Specificity = 250/(250+8) = 0.969

From the above calculations, we see that the F1 score provides a balance between Precision and Recall. It also gives a good estimate of the model performance for our target response (detecting Spam emails) despite the model’s accuracy being ~93%. High specificity and low recall mean that our model is doing just fine with detecting emails that are not spam but is unable to detect spam emails.

Conclusion:

Dealing with a machine learning problem can be a tedious task sometimes. It is equally important for us to evaluate those models using a proper evaluation metric. This choice does not come naturally in many cases. Hence, knowing a confusion matrix can be useful in identifying and targeting the specific problem in hand.

!! Thank you for reading !!

Reference: Article on IDS