How to Run Test Data and Evaluate an SVM Supervised Learning Model

By Anasse Bari, Mohamed Chaouchi, Tommy Jung

Supervised learning is a machine-learning task that learns from predictive analysis data that has been labeled. One way to think about supervised learning is that the labeling of data is done under the supervision of the modeler; unsupervised learning, by contrast, doesn’t require labeled data. Supervised learning is commonly performed using a classification algorithm.

How to run the test data

Using 10 percent of the 150 instances from the dataset gives you 15 test-data points to run through the model. Let’s see how your predictive model will perform. Type the following code listing into the interpreter:

>>> predicted = svmClassifier.predict(X_test)
>>> predictedarray([0, 0, 2, 2, 1, 0, 0, 2, 2, 1, 2, 0, 1, 2, 2])

The predict function in the first line of code is what does the prediction, as you may have guessed. It takes the test data as input and outputs the results into the variable predicted. The second line prints the output.

The last line in the code section is the output, or prediction: an array of 15 — that is, 10 percent of the sample dataset, which is the size of the test dataset. The numbers in the array represent the Iris Flower classes.

How to evaluate the model

To evaluate the accuracy of your model, you can compare the output array with the y_test array. For this small sample dataset, you can easily tell how it performed by seeing that the output array from the predict function is almost the same as the y_test array. The last line in the code is a simple equality check between the two arrays, sufficient for this simple test case.

Here’s the code:

>>> predictedarray([0, 0, 2, 2, 1, 0, 0, 2, 2, 1, 2, 0, 1, 2, 2])
>>> y_testarray([0, 0, 2, 2, 1, 0, 0, 2, 2, 1, 2, 0, 2, 2, 2])
>>> predicted == y_testarray([ True, True, True, True, True, True, True,  True, True, True, True, True, False, True,  True], dtype=bool)

Looking at the output array with all the Boolean (True and False) values, you can see that the model predicted all but one outcome. On the thirteenth data point, it predicted 1 (Versicolor) when it should have been 2 (Virginica). The False value(s) indicate that the model predicted the incorrect Iris class for that data point. The percentage of correct predictions will determine the accuracy of the predictive model.

In this case you can simply use basic division and get the accuracy:

correct outcomes / test size => 14 / 15 => 0.9333 or 93.33 percent

It’s no surprise that the model failed to predict Virginica or Versicolor; they’re clearly not separable by a straight line. A failure to predict Setosa, however, would be surprising because Setosa is clearly linearly separable. Still, the accuracy was 14 out of 15, or 93.33 percent.

For a test set with more data points, you may want to use the metrics module to do your measurements. The following code will get the accuracy of the model:

>>> from sklearn import metrics
>>> metrics.accuracy_score(y_test, predicted)0.93333333333333335

Another useful measurement tool is the confusion matrix. Yes, it’s real. It’s a matrix (tabular format) that shows the predictions that the model made on the test data. Here is the code that displays the confusion matrix:

>>> metrics.confusion_matrix(y_test, predicted)array([[5, 0, 0], 
[0, 2, 0],
[0, 1, 7]])

The diagonal line from the top-left corner to the bottom-right corner is the number of correct predictions for each row. Each row corresponds to a class of Iris.

For example: The first row corresponds to the Setosa class. The model predicted five correct test data points and had no errors predicting the Setosa class. If it had an error, a number other than zero would be present in any of the columns in that row.

The second row corresponds to the Versicolor class. The model predicted two correct test data points and no errors. The third row corresponds to the Virginica class. The model predicted seven correct test data points but also had one error.

The model mistakenly predicted one observation of Virginica for a Versicolor. You can tell that by looking at the column where the error is showing up. Column 1 belongs to Versicolor.

The accuracy of a predictive model’s results will directly affect the decision to deploy that model; the higher the accuracy, the more easily you can gather support for deploying the model.

When creating a predictive model, start by building a simple working solution quickly — and then continue to build iteratively until you get the desired outcome. Spending months building a predictive model — and not being able to show your stakeholders any results — is a sure way to lose the attention and support of your stakeholders.

Here is the full listing of the code to create and evaluate a SVM classification model:

>>> from sklearn.datasets import load_iris
>>> from sklearn.svm import LinearSVC
>>> from sklearn import cross_validation
>>> from sklearn import metrics
>>> iris = load_iris()
>>> X_train, X_test, y_train, y_test =   cross_validation.train_test_split(iris.data,   iris.target, test_size=0.10, random_state=111)
>>> svmClassifier = LinearSVC(random_state=111)
>>> svmClassifier.fit(X_train, y_train)
>>> predicted = svmClassifier.predict(X_test)
>>> predictedarray([0, 0, 2, 2, 1, 0, 0, 2, 2, 1, 2, 0, 1, 2, 2])
>>> y_testarray([0, 0, 2, 2, 1, 0, 0, 2, 2, 1, 2, 0, 2, 2, 2])
>>> metrics.accuracy_score(y_test, predicted)0.93333333333333335
>>> predicted == y_testarray([ True, True, True, True, True, True, True,  True, True, True, True, True, False, True,  True], dtype=bool)