In our previous tutorial, we learned how to use models which were trained for Image Classification on the ILSVRC data. In this tutorial, we will discuss how to use those models as a Feature Extractor and train a new model for a different classification task.
Suppose you want to make a household robot which can cook food. The first step would be to identify different vegetables. We will try to build a model which identifies Tomato, Watermelon, and Pumpkin for this tutorial. In the previous tutorial, we saw the pre-trained models were not able to identify them because these categories were not learned by the models.
Transfer Learning vs Fine-tuning
The pre-trained models are trained on very large scale image classification problems. The convolutional layers act as feature extractor and the fully connected layers act as Classifiers.
Since these models are very large and have seen a huge number of images, they tend to learn very good, discriminative features. We can either use the convolutional layers merely as a feature extractor or we can tweak the already trained convolutional layers to suit our problem at hand. The former approach is known as Transfer Learning and the latter as Fine-tuning.
As a rule of thumb, when we have a small training set and our problem is similar to the task for which the pre-trained models were trained, we can use transfer learning. If we have enough data, we can try and tweak the convolutional layers so that they learn more robust features relevant to our problem. You can get a detailed overview of Fine-tuning and transfer learning here. We will discuss Transfer Learning in Keras in this post.
ImageNet Jargon
ImageNet is based upon WordNet which groups words into sets of synonyms (synsets). Each synset is assigned a “wnid” ( Wordnet ID ). Note that in a general category, there can be many subcategories and each of them will belong to a different synset. For example Working Dog ( sysnet = n02103406), Guide Dog ( sysnet = n02109150 ), and Police Dog ( synset = n02106854 ) are three different synsets.
The wnid’s of the 3 object classes we are considering are given below
- n07734017 -> Tomato
- n07735510 -> Pumpkin
- n07756951 -> WaterMelon
Download and prepare Data
For downloading Imagenet images by wnid, there is a nice code repository written by Tzuta Lin which is available on GitHub. You can use this to download images of a specific “wnid”. You can visit the GitHub page and follow the instructions to download the images for any of the wnid’s.
However, If you are just starting out and do not want to download full-size images, you can use another python library available through pip – imagenetscraper. It is easy to use and also provides resizing options. Installation and usage instructions are provided below. Note that it works with python3 only.
# Install imagenetscraper
pip3 install imagenetscraper
# Download the images for the three wnids and keep them in separate folders.
imagenetscraper n07756951 watermelon
imagenetscraper n07734017 tomato
imagenetscraper n07735510 pumpkin
I found that the data is very noisy, i.e. there is a lot of clutter, the objects are occluded etc. So, I shortlisted around 250 images for each class. We need to create two directories namely “train” and “validation” so that we can use the Keras functions for loading images in batches.
Load the pre-trained model
from tensorflow.keras.applications import vgg16
vgg_conv = vgg16.VGG16(weights='imagenet',
include_top=False,
input_shape=(224, 224, 3))
In the above code, we load the VGG Model along with the ImageNet weights similar to our previous tutorial. There is, however, one change – `include_top=False
. We have not loaded the last two fully connected layers which act as the classifier. We are just loading the convolutional layers. It should be noted that the last layer has a shape of 7 x 7 x 512.
Extract Features
The data is divided into 80:20 ratio and kept in separate train and validation folders. Each folder should contain 3 folders belonging to he respective classes. You can change the directory according to your system.
# each folder contains three subfolders in accordance with the number of classes
train_dir = './clean-dataset/train'
validation_dir = './clean-dataset/validation'
# the number of images for train and test is divided into 80:20 ratio
nTrain = 600
nVal = 150
We will use the ImageDataGenerator class to load the images and flow_from_directory function to generate batches of images and labels.
# load the normalized images
datagen = ImageDataGenerator(rescale=1./255)
# define the batch size
batch_size = 20
# the defined shape is equal to the network output tensor shape
train_features = np.zeros(shape=(nTrain, 7, 7, 512))
train_labels = np.zeros(shape=(nTrain,3))
# generate batches of train images and labels
train_generator = datagen.flow_from_directory(
train_dir,
target_size=(224, 224),
batch_size=batch_size,
class_mode='categorical',
shuffle=True)
Then we use model.predict() function to pass the image through the network which gives us a 7 x 7 x 512 dimensional Tensor. We reshape the Tensor into a vector. Similarly, we find the validation_features.
# iterate through the batches of train images and labels
for i, (inputs_batch, labels_batch) in enumerate(train_generator):
if i * batch_size >= nTrain:
break
# pass the images through the network
features_batch = vgg_conv.predict(inputs_batch)
train_features[i * batch_size : (i + 1) * batch_size] = features_batch
train_labels[i * batch_size : (i + 1) * batch_size] = labels_batch
# reshape train_features into vector
train_features_vec = np.reshape(train_features, (nTrain, 7 * 7 * 512))
print("Train features: {}".format(train_features_vec.shape))
Output:
Train features: (600, 25088)
Create your own model
We will create a simple feedforward network with a softmax output layer having 3 classes.
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras import Sequential, optimizers
model = Sequential()
model.add(Dense(512, activation='relu', input_dim=7 * 7 * 512))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
Train the model
Training a network in Keras is as simple as calling model.fit() function as we have seen in our earlier tutorials.
# configure the model for training
model.compile(optimizer=optimizers.RMSprop(lr=2e-4),
loss='categorical_crossentropy',
metrics=['acc'])
# use the train and validation feature vectors
history = model.fit(train_features_vec,
train_labels,
epochs=20,
batch_size=batch_size,
validation_data=(validation_features_vec,
validation_labels)
)
Check Performance
We would like to visualize which images were wrongly classified.
# get the list of all validation file names
fnames = validation_generator.filenames
# get the list of the corresponding classes
ground_truth = validation_generator.classes
# get the dictionary of classes
label2index = validation_generator.class_indices
# obtain the list of classes
idx2label = list(label2index.keys())
print("The list of classes: ", idx2label)
Output:
The list of classes: ['pumpkin', 'tomato', 'watermelon']
Let’s evaluate the number of incorrect predictions:
predictions = model.predict_classes(validation_features_vec)
prob = model.predict(validation_features_vec)
errors = np.where(predictions != ground_truth)[0]
print("Number of errors = {}/{}".format(len(errors),nVal))
We will get 14 falsely predicted classes out of 150 validation images:
Number of errors = 14/150
Let us see which images were predicted wrongly:
for i in range(len(errors)):
pred_class = np.argmax(prob[errors[i]])
pred_label = idx2label[pred_class]
print('Original label:{}, Prediction :{}, confidence : {:.3f}'.format(
fnames[errors[i]].split('/')[0],
pred_label,
prob[errors[i]][pred_class]))
original = load_img('{}/{}'.format(validation_dir,fnames[errors[i]]))
plt.axis('off')
plt.imshow(original)
plt.show()
Below we can see several examples as the result of the above code execution:

We will try to improve on the limitations of transfer learning by using another approach called Fine-tuning in our next post.
Subscribe & Download Code
If you liked this article and would like to download code (C++ and Python) and example images used in this post, please click here. Alternately, sign up to receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.References
Building powerful image classification models using very little data
Deep Learning with Python Github Repository
Don’t the shape of ground_truth and predictions need to be the same?
Yes, they should be same. Why do you think they are different in this case?
I was using a different database and they came out differently. I fixed it, thanks.
Hi,
I was running the example code but I get this error:
File “C:/Users/User/Anaconda3/envs/deeplearning/TestDeepLearning/Test4TransferLearning.py”, line 126, in
idx2label = dict((v,k) for k,v in label2index.iteritems())
AttributeError: ‘dict’ object has no attribute ‘iteritems’
For me line 126 is :
model.compile(optimizer=optimizers.RMSprop(lr=2e-4),
loss=’categorical_crossentropy’,
metrics=[‘acc’])
Can you help me with this issue?
replace iteritems with items.
getting an error with the VGG_Conv.predict line:
File “vgg16tst.py”, line 52, in
features_batch = vgg_conv.predict(inputs_batch)
NameError: name ‘vgg_conv’ is not defined
try vgg_model
It should be conv_base as it was the function name for VGG model. Also
2)Shuffle should be set to False.
3) nImages above create a model should be nTrain
I am getting this error
File “keras_vcg16_pre_train.py”, line 68, in
validation_features[i * batch_size : (i + 1) * batch_size] = features_batch
ValueError: could not broadcast input array from shape (4,7,7,512) into shape (20,7,7,512)
I think you should check your batch size and validation data first
I m getting the same issue! what do you mean to check batch size and validation data!
Hi,
I was running the example code but I get this error:
File “”, line 4, in
validation_features[i * batch_size : (i + 1) * batch_size] = [features_batch]
TypeError: ‘tuple’ object does not support item assignment
Please download the code from the download section. The code given above is for explanation purpose and only important parts are included.
Thanks alot for your help now it work
I have a problem with
”
for inputs_batch, labels_batch in validation_generator:
features_batch = vgg_conv.predict(inputs_batch)
validation_features[i * batch_size : (i + 1) * batch_size] = features_batch
validation_labels[i * batch_size : (i + 1) * batch_size] = labels_batch
i += 1
if i * batch_size >= nVal:
break
”
It returns “too many values to unpack (expected 2)”. Must have something to do with me using a single folder with images and no classes defined. My train_generator is already adapted to that and seems to be working well. But so far I can’t figure this out. For now I am loonking for other ways to loop over the Keras DirectoryIterator.
When trying to run the “Checking the Performance”, I got the following error:
NameError: name ‘validation_generator’ is not defined
What did I do wrong?
Found the error source…
train_features is of shape (nTrain, 7, 7, 512)
But the input dim is (7*7*512)
So its giving me an error
My nTrain=200
https://uploads.disquscdn.com/images/8d4f30162a29a268807c5a283c187d5970b44bd92b60c5a058fc7c5b260e9d81.jpg
What to do?
it is very nice example,Please can you share me code of rnn model on pre train model using keras,your example is related to cnn , i need example for training rnn on pre train model .
Please help me on this .