• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

SVM using Scikit-Learn in Python

Kushashwa Ravi Shrimali
July 27, 2018 3 Comments
Machine Learning

July 27, 2018 By 3 Comments

This post explains the implementation of Support Vector Machines (SVMs) using Scikit-Learn library in Python. We had discussed the math-less details of SVMs in the earlier post.

In this post, we will show the working of SVMs for three different type of datasets:

  1. Linearly Separable data with no noise
  2. Linearly Separable data with added noise
  3. Non-linear separable data with added noise

Prerequisites

Before we begin, we need to install sklearn and matplotlib modules. This can be done using pip.

pip install -U scikit-learn
pip install -U matplotlib

We first import matplotlib.pyplot for plotting graphs. We also need svm imported from sklearn. Finally, from sklearn.model_selection we need train_test_split to randomly split data into training and test sets, and GridSearchCV for searching the best parameter for our classifier. The code below shows the imports.

 
import sys, os
import matplotlib.pyplot as plt
from sklearn import svm
from sklearn.model_selection import train_test_split, GridSearchCV

Linearly separable data with no noise

Let’s first look at the simplest cases where the data is cleanly separable linearly. In the 2D case, it simply means we can find a line that separates the data. In the 3D case, it will be a plane. For higher dimensions, it is simply a plane.

Linearly separable data
Figure 1. Linearly separable data. The dataset is randomly split into 80% training and 20% test.

Let’s see how we can use a simple binary SVM classifier based on the data above.

Download Code
To easily follow along this tutorial, please download code by clicking on the button below. It’s FREE!

Download Code

If you have downloaded the code, here are the steps for building a binary classifier

  1. Prepare data: We read the data from the files points_class_0.txt and points_class_1.txt. These files simply have x and y coordinates of points — one per line. The points in points_class_0.txt are assinged the label 0 and the points in points_class_1.txt are assigned the label 1. The dataset is then split into training (80%) and test (20%) sets. This dataset is shown in Figure 1.
     
    # Read data
    x, labels = read_data("points_class_0.txt", "points_class_1.txt")
    
    # Split data to train and test on 80-20 ratio
    X_train, X_test, y_train, y_test = train_test_split(x, labels, test_size = 0.2, random_state=0)
    
    # Plot traning and test data
    plot_data(X_train, y_train, X_test, y_test)
    
  2. Create an instance of a Linear SVM classifier: Next we create an instance of the Linear SVM classifier from scikit-learn. We use the default parameters because the problem is easy to solve and we expect the default parameters to work just fine. We only specify the SVM be linear.
    # Create a linear SVM classifier 
    clf = svm.SVC(kernel='linear')
    
  3. Train a Linear SVM classifier: Next we train a Linear SVM. In other words, based on the training data, we find the line that separates the two classes. This is simply done using the fit method of the SVM class.
    # Train classifier 
    clf.fit(X_train, y_train)
    
    # Plot decision function on training and test data
    plot_decision_function(X_train, y_train, X_test, y_test, clf)
    

    Next, we plot the decision boundary and support vectors. The decision boundary is estimated based on only the traning data. Given a new data point (say from the test set), we simply need to check which side of the line the point lies to classify it as 0 ( red ) or 1 (blue).

    Decision boundary, margins and support vectors
    Figure 2: Decision boundary (solid line) and support vectors (black dots).
  4. Test a Linear SVM classifier:
    To predict the class of a new point ( or points ) we can simply use the predict method of the SVM class. To obtain the accuracy on the test set, we can use the score method.

    # Make predictions on unseen test data
    clf_predictions = clf.predict(X_test)
    print("Accuracy: {}%".format(clf.score(X_test, y_test) * 100 ))
    

    In this easy example, the accuracy is 100%.

Linearly separable data with noise

Let’s look at a slightly more complicated case shown in Figure 3 where it is not possible to linearly separate the data, but a linear classifier still makes sense. Note that no matter what you do some red points and some blue points will be on the wrong side of the line.

SVM Linear Data with Noise
Figure 3: SVM Linear Data with Noise

The question now is which line to choose? SVM provides you with parameter called C that you can set while training. In scikit-learn, this can be done using the following lines of code

 
# Create a linear SVM classifier with C = 1
clf = svm.SVC(kernel='linear', C=1)

If you set C to be a low value (say 1), the SVM classifier will choose a large margin decision boundary at the expense of larger number of misclassifications. When C is set to a high value (say 100), the classifier will choose a low margin decision boundary and try to minimize the misclassifications. This is shown in Figure 4. The margin is the shown using dotted lines — the larger the space between the dotted lines, the larger is the margin.

SVM Parameter C
Figure 4: SVM Parameter C

You may be tempted to ask which value of C is better. The answer depends on how much noise you think there is in your data. If you think the data is very noisy, you want C to be small. On the other hand, if you think the data is less noisy, you should choose C to be large.

Non-Linearly separable data with noise

Finally, let’s look at data that is impossible to partition using a line.

Non-Linear SVM Data.
Figure 5: Non-Linear SVM Data.

Is SVM useless in such cases? Fortunately, the answer is no. We can use the Kernel Trick explained in our previous article. In scikit-learn we can specify the kernel type while instantiating the SVM class.

# Create SVM classifier based on RBF kernel. 
clf = svm.SVC(kernel='rbf', C = 10.0, gamma=0.1)

In the above example, we are using the Radial Basis Fucttion expalined in our previous post with parameter gamma set to 0.1. As you can see in Figure 6, the SVM with an RBF kernel produces a ring shaped decision boundary instead of a line.

SVM with RBF Kernel
Figure 6: SVM with RBF Kernel

Looking at Figure 6 you may be tempted to think that using some other value for C and gamma, we may be able to come up with a better decision boundary. Your intuition is right. To find the best parameters, we need to do a parameter sweep by changing values of C and gamma and picking the one that works best. Let’s see how parameters tuning in done using GridSearchCV.

Parameter Tuning using GridSearchCV

The module sklearn.model_selection allows us to do a grid search over parameters using GridSearchCV. All we need to do is specify which parameters we want to vary and by what value. In the following example, the parameters C and gamma are varied. Every combination of C and gamma is tried and the best one is chosen based. The best estimator can be accessed using clf.best_estimator_.

 
# Grid Search
# Parameter Grid
param_grid = {'C': [0.1, 1, 10, 100], 'gamma': [1, 0.1, 0.01, 0.001, 0.00001, 10]}

# Make grid search classifier
clf_grid = GridSearchCV(svm.SVC(), param_grid, verbose=1)

# Train the classifier
clf_grid.fit(X_train, y_train)

# clf = grid.best_estimator_()
print("Best Parameters:\n", clf_grid.best_params_)
print("Best Estimators:\n", clf_grid.best_estimator_)

By default, GridSearchCV performs 3-fold cross-validation. In other words, it divides the data into 3 parts and uses two parts for training, and one part for determining accuracy. This is done three times so each of the three parts is in the training set twice and validation set once. The accuracy for a given C and gamma is the average accuracy during 3-fold cross-validation.

The best parameters ( C = 1 and gamma = 0.01 ) of classifier shown in Figure 7 were found using GridSearchCV. Our intuition also confirms this shape of the decision boundary looks better than the one manually chosen.

Parameter search using GridSearchCV
Figure 7: Parameter search using GridSearchCV

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now

Tags: classification Grid Search Kernel Trick Parameter Tuning Python scikit-learn Support Vector Machine SVM

Filed Under: Machine Learning

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching
  • Classification with Localization: Convert any Keras Classifier to a Detector

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy