ID2223 TensorFlow Lab 2¶
Tianze Wang
Table of contents¶
- A Quick Recap
- Introduction to Computer Vision
- Introduction to Convolutions
- Convolutional Neural Networks
- More Complex Images
Section 1: A Quick Recap¶
Consider the following sets of numbers. What is the relationship between them?
| x | y |
|---|---|
| -1 | -2 |
| 0 | 1 |
| 1 | 4 |
| 2 | 7 |
| 3 | 10 |
| 4 | 13 |
Now let's look at how to train a ML model to spot the patterns between these items of data.
In [1]:
import tensorflow as tf
import numpy as np
print(tf.__version__)
Prepare training Data¶
| x | y |
|---|---|
| -1 | -2 |
| 0 | 1 |
| 1 | 4 |
| 2 | 7 |
| 3 | 10 |
| 4 | 13 |
In [2]:
# training data
X_train = np.array([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
y_train = np.array([-2.0, 1.0, 4.0, 7.0, 10.0, 13.0], dtype=float)
Define the model¶
In [4]:
model = tf.keras.Sequential([tf.keras.layers.Dense(units=1, input_shape=[1])])
model.compile(optimizer='sgd', loss='mean_squared_error')
In [6]:
# plot the model
tf.keras.utils.plot_model(model)
Out[6]:
In [5]:
# model summary
model.summary()
Train and evaluate¶
In [7]:
model.fit(X_train, y_train, epochs=500)
Out[7]:
In [8]:
print(model.predict([10]))
In [9]:
model.evaluate([10, 11], [31, 34])
Out[9]:
Section 2: Introduction to Computer Vision¶
In the previous section, we have seen how to use neural networks to map the relationship between two variables (i.e., x and y).
However, in that instance, it is a bit of overkill because it would have been easier to write the funtion $$y = 3x + 1$$ , insteal of using ML to learn the relationship bwtween $x$ and $y$ for a fixed set of values.
But what if we are dealing with a case where writing rules like $y = 3x + 1$ is much more difficult? Computre Vision problems serve as good examples in this case. Let's take a look at a scenario where we want to recognize different items of clothing, trained from a dataset containing 10 different types.
A new dataset: Fashion MNIST¶
We will train a neural network to recognize items of clothing from a common dataset called Fashion MNIST. You can learn more about this dataset here.
It contains 70,000 items of clothing in 10 different categories. Each item of clothing is in a 28x28 greyscale image. You can see some examples here:
In [10]:
# The Fashion MNIST data is available directly in the tf.keras.datasets API.
mnist_dataset = tf.keras.datasets.fashion_mnist
In [11]:
# The load_data() function of this object will give two sets of two lists, which will be the training and testing data
(X_train, y_train), (X_test, y_test) = mnist_dataset.load_data()
In [15]:
print("Shape of X_train = ", X_train.shape)
print("Shape of y_train = ", y_train.shape)
print("Shape of X_test = ", X_test.shape)
print("Shape of y_test = ", y_test.shape)
In [16]:
print(y_train[0])
print(X_train[0])
Visualizing the dataset¶
In [19]:
import matplotlib.pyplot as plt
plt.imshow(X_train[0], cmap='gray')
plt.show()
Note that the values in the image are between 0 and 255. When training a neural network, it is easier that if we can convert every values to between 0 and 1, which is a process normally referred to as normalizing.
In [18]:
print("Before normalizing: ")
print("\t min of train is {}, max of train is {}".format(np.min(X_train), np.max(X_train)))
print("\t min of test is {}, max of test is {}".format(np.min(X_test), np.max(X_test)))
X_train = X_train/255
X_test = X_test/255
print("After normalizing: ")
print("\t min of train is {}, max of train is {}".format(np.min(X_train), np.max(X_train)))
print("\t min of test is {}, max of test is {}".format(np.min(X_test), np.max(X_test)))
Define a Simple Model¶
In [20]:
# define the model
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))
Compile the Model¶
In [21]:
# compile the model
model.compile(optimizer = tf.optimizers.SGD(),
loss = 'sparse_categorical_crossentropy',
metrics=['accuracy'])
Train the model¶
In [22]:
# train for 5 epochs
model.fit(X_train, y_train, epochs=5)
Out[22]:
Prediction and evaluation¶
In [23]:
# prediction on unseen data
classifications = model.predict(X_test)
print(classifications[0])
print("True label is ", y_test[0])
print("The predicted label is ", np.argmax(classifications[0]))
In [24]:
# evaluation on test set
loss, acc = model.evaluate(X_test, y_test, verbose=0)
print("loss on test set is : ", loss)
print("accuracy on test set is : ", acc)
More Neurons!¶
In [39]:
# the data
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
X_train = X_train/255.0
X_test = X_test/255.0
In [29]:
# define the model
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(1024, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))
In [30]:
# compile the model
model.compile(optimizer = tf.optimizers.SGD(),
loss = 'sparse_categorical_crossentropy',
metrics=['accuracy'])
In [31]:
# train the model
model.fit(X_train, y_train, epochs=5)
Out[31]:
In [36]:
# a single prediction
predictions = model.predict(X_test)
print(predictions[0])
print("The predicted label is ", np.argmax(predictions[0]))
In [37]:
print("True label is ", y_test[0])
In [33]:
# evaluation on test set
loss, acc = model.evaluate(X_test, y_test, verbose=0)
print("loss on test set is : ", loss)
print("accuracy on test set is : ", acc)
Let's go DEEP! More Layers!¶
In [40]:
# get the data
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
X_train = X_train/255.0
X_test = X_test/255.0
In [42]:
# define the model
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(1024, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))
In [43]:
# compile the model
model.compile(optimizer = tf.optimizers.SGD(),
loss = 'sparse_categorical_crossentropy',
metrics=['accuracy'])
In [44]:
# train the model
model.fit(X_train, y_train, epochs=5)
Out[44]:
In [45]:
# a single prediction
classifications = model.predict(X_test)
print(classifications[0])
print("True label is ", y_test[0])
print("The predicted label is ", np.argmax(classifications[0]))
In [46]:
# evaluation on test set
loss, acc = model.evaluate(X_test, y_test, verbose=0)
print("loss on test set is : ", loss)
print("accuracy on test set is : ", acc)
Section 3: Convolutions¶
Limitations of the previous DNN¶
In the previous section, we saw how to train an image classifier for fashion items using the Fashion MNIST dataset.
This gave us a pretty accuract classifier, but there was an obvious constraint: the images were 28x28, grey scale and the item was centered in the image.
For example here are a couple of the images in Fashion MNIST
The DNN that you created simply learned from the raw pixels what made up a sweater, and what made up a boot in this context. But consider how it might classify this image?
While it's clear that there are boots in this image, the classifier would fail for a number of reasons. First, of course, it's not 28x28 greyscale, but more importantly, the classifier was trained on the raw pixels of a left-facing boot, and not the features that make up what a boot is.
That's where Convolutions are very powerful. A convolution is a filter that passes over an image, processing it, and extracting features that show a commonolatity in the image. In this lab you'll see how they work, but processing an image to see if you can extract features from it!
Generating convolutions is very simple -- we simply scan every pixel in the image and then look at it's neighboring pixels. You multiply out the values of these pixels by the equivalent weights in a filter.
So, for example, consider this:
Let's explore a bit¶
In [48]:
import cv2
import numpy as np
from scipy import misc
i = misc.ascent()
In [49]:
import matplotlib.pyplot as plt
plt.grid(False)
plt.gray()
plt.axis('off')
plt.imshow(i)
plt.show()
Now we can create a filter as a 3x3 array.
In [53]:
# This filter detects edges nicely
# It creates a convolution that only passes through sharp edges and straight
# lines.
#Experiment with different values for fun effects.
#filter = [ [0, 1, 0], [1, -4, 1], [0, 1, 0]]
# A couple more filters to try for fun!
filter = [ [-1, -2, -1], [0, 0, 0], [1, 2, 1]]
#filter = [ [-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]
# If all the digits in the filter don't add up to 0 or 1, you
# should probably do a weight to get it to do so
# so, for example, if your weights are 1,1,1 1,2,1 1,1,1
# They add up to 10, so you would set a weight of .1 if you want to normalize them
weight = 1
In [54]:
i_transformed = np.copy(i)
size_x = i_transformed.shape[0]
size_y = i_transformed.shape[1]
for x in range(1,size_x-1):
for y in range(1,size_y-1):
convolution = 0.0
convolution = convolution + (i[x - 1, y-1] * filter[0][0])
convolution = convolution + (i[x, y-1] * filter[0][1])
convolution = convolution + (i[x + 1, y-1] * filter[0][2])
convolution = convolution + (i[x-1, y] * filter[1][0])
convolution = convolution + (i[x, y] * filter[1][1])
convolution = convolution + (i[x+1, y] * filter[1][2])
convolution = convolution + (i[x-1, y+1] * filter[2][0])
convolution = convolution + (i[x, y+1] * filter[2][1])
convolution = convolution + (i[x+1, y+1] * filter[2][2])
convolution = convolution * weight
if(convolution<0):
convolution=0
if(convolution>255):
convolution=255
i_transformed[x, y] = convolution
In [55]:
# Plot the image. Note the size of the axes -- they are 512 by 512
plt.gray()
plt.grid(False)
plt.imshow(i_transformed)
#plt.axis('off')
plt.show()
So, consider the following filter values, and their impact on the image.
Using -1,0,1,-2,0,2,-1,0,1 gives us a very strong set of vertical lines:
Using -1, -2, -1, 0, 0, 0, 1, 2, 1 gives us horizontal lines:
Pooling¶
This code will show a (2, 2) pooling.Run it to see the output, and you'll see that while the image is 1/4 the size of the original, the extracted features are maintained!
In [58]:
new_x = int(size_x/2)
new_y = int(size_y/2)
newImage = np.zeros((new_x, new_y))
for x in range(0, size_x, 2):
for y in range(0, size_y, 2):
pixels = []
pixels.append(i_transformed[x, y])
pixels.append(i_transformed[x+1, y])
pixels.append(i_transformed[x, y+1])
pixels.append(i_transformed[x+1, y+1])
pixels.sort(reverse=True)
newImage[int(x/2),int(y/2)] = pixels[0]
# Plot the image. Note the size of the axes -- now 256 pixels instead of 512
plt.gray()
plt.grid(False)
plt.imshow(newImage)
#plt.axis('off')
plt.show()
Section 4: Improving accuracy using Convolutions¶
In [59]:
import tensorflow as tf
mnist = tf.keras.datasets.fashion_mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train / 255.0
X_test = X_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation=tf.nn.relu),
tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer=tf.optimizers.Adam(), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=5)
loss, acc = model.evaluate(X_test, y_test, verbose=0)
print("Loss on test set is ", loss)
print("Accuracy on test set is", acc)
In [61]:
# prepare the dataset
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
X_train = X_train.reshape(60000, 28, 28, 1)
X_train = X_train / 255.0
X_test = X_test.reshape(10000, 28, 28, 1)
X_test = X_test / 255.0
In [63]:
# define the model
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(28, 28, 1)))
model.add(tf.keras.layers.MaxPooling2D(2, 2))
model.add(tf.keras.layers.Conv2D(64, (3,3), activation='relu'))
model.add(tf.keras.layers.MaxPooling2D(2,2))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation='relu'))
model.add(tf.keras.layers.Dense(10, activation='softmax'))
In [64]:
# complie the model
model.compile(optimizer=tf.optimizers.Adam(), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
# model summary
model.summary()
In [65]:
# train the model
model.fit(X_train, y_train, epochs=5)
Out[65]:
In [66]:
loss, acc = model.evaluate(X_test, y_test, verbose=0)
print("Loss on test set is ", loss)
print("Accuracy on test set is", acc)
Section 5: Using Convolutions with Complex Images¶
In Fashion MNIST classification, the subject is always in the center of a 28x28 image.
In this section, we will take it to the next level, training to recognize features in an image where the subject can be anywhere in the image!
Building a horses-or-humans classifier that will tell you if a given image contains a horse or a human, where the network is trained to recognize features that determine which is which.
Get the dataset¶
In [70]:
!wget --no-check-certificate \
https://storage.googleapis.com/laurencemoroney-blog.appspot.com/horse-or-human.zip \
-O ./data/horse-or-human.zip
In [1]:
import os
import zipfile
local_zip = './data/horse-or-human.zip'
zip_ref = zipfile.ZipFile(local_zip, 'r')
zip_ref.extractall('./data/horse-or-human')
zip_ref.close()
# Directory with our training horse pictures
train_horse_dir = os.path.join('./data/horse-or-human/horses')
# Directory with our training human pictures
train_human_dir = os.path.join('./data/horse-or-human/humans')
In [2]:
# a pick view of the data
train_horse_names = os.listdir(train_horse_dir)
print(train_horse_names[:10])
train_human_names = os.listdir(train_human_dir)
print(train_human_names[:10])
In [3]:
print('total training horse images:', len(os.listdir(train_horse_dir)))
print('total training human images:', len(os.listdir(train_human_dir)))
Visualization¶
In [4]:
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
# Parameters for our graph; we'll output images in a 4x4 configuration
nrows = 4
ncols = 4
# Index for iterating over images
pic_index = 0
# Set up matplotlib fig, and size it to fit 4x4 pics
fig = plt.gcf()
fig.set_size_inches(ncols * 4, nrows * 4)
pic_index += 8
next_horse_pix = [os.path.join(train_horse_dir, fname)
for fname in train_horse_names[pic_index-8:pic_index]]
next_human_pix = [os.path.join(train_human_dir, fname)
for fname in train_human_names[pic_index-8:pic_index]]
In [5]:
for i, img_path in enumerate(next_horse_pix+next_human_pix):
# Set up subplot; subplot indices start at 1
sp = plt.subplot(nrows, ncols, i + 1)
sp.axis('Off') # Don't show axes (or gridlines)
img = mpimg.imread(img_path)
plt.imshow(img)
plt.show()
Data Preprocessing¶
In [6]:
from tensorflow.keras.preprocessing.image import ImageDataGenerator
# All images will be rescaled by 1./255
train_datagen = ImageDataGenerator(rescale=1/255)
# Flow training images in batches of 128 using train_datagen generator
train_generator = train_datagen.flow_from_directory(
'./data/horse-or-human/', # This is the source directory for training images
target_size=(300, 300), # All images will be resized to 150x150
batch_size=128,
# Since we use binary_crossentropy loss, we need binary labels
class_mode='binary')
Building a Small Model from Scratch¶
In [7]:
import tensorflow as tf
model = tf.keras.models.Sequential([
# Note the input shape is the desired size of the image 300x300 with 3 bytes color
# This is the first convolution
tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(300, 300, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
# The second convolution
tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# The third convolution
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# The fourth convolution
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# The fifth convolution
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# Flatten the results to feed into a DNN
tf.keras.layers.Flatten(),
# 512 neuron hidden layer
tf.keras.layers.Dense(512, activation='relu'),
# Only 1 output neuron. It will contain a value from 0-1 where 0 for 1 class ('horses') and 1 for the other ('humans')
tf.keras.layers.Dense(1, activation='sigmoid')
])
In [8]:
model.summary()
In [9]:
from tensorflow.keras.optimizers import RMSprop
# compile the model
model.compile(loss='binary_crossentropy',
optimizer=RMSprop(lr=0.001),
metrics=['acc'])
Train the model¶
In [82]:
history = model.fit_generator(
train_generator,
steps_per_epoch=8,
epochs=15,
verbose=1)
Runing the Model¶
In [83]:
import numpy as np
# from google.colab import files
from keras.preprocessing import image
# uploaded = files.upload()
for fn in uploaded.keys():
# predicting images
path = '/content/' + fn
img = image.load_img(path, target_size=(300, 300))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
images = np.vstack([x])
classes = model.predict(images, batch_size=10)
print(classes[0])
if classes[0]>0.5:
print(fn + " is a human")
else:
print(fn + " is a horse")
Visualizing Intermediate Representations¶
In [10]:
import numpy as np
import random
from tensorflow.keras.preprocessing.image import img_to_array, load_img
# Let's define a new Model that will take an image as input, and will output
# intermediate representations for all layers in the previous model after
# the first.
successive_outputs = [layer.output for layer in model.layers[1:]]
#visualization_model = Model(img_input, successive_outputs)
visualization_model = tf.keras.models.Model(inputs = model.input, outputs = successive_outputs)
# Let's prepare a random input image from the training set.
horse_img_files = [os.path.join(train_horse_dir, f) for f in train_horse_names]
human_img_files = [os.path.join(train_human_dir, f) for f in train_human_names]
img_path = random.choice(horse_img_files + human_img_files)
img = load_img(img_path, target_size=(300, 300)) # this is a PIL image
x = img_to_array(img) # Numpy array with shape (150, 150, 3)
x = x.reshape((1,) + x.shape) # Numpy array with shape (1, 150, 150, 3)
# Rescale by 1/255
x /= 255
# Let's run our image through our network, thus obtaining all
# intermediate representations for this image.
successive_feature_maps = visualization_model.predict(x)
# These are the names of the layers, so can have them as part of our plot
layer_names = [layer.name for layer in model.layers]
# Now let's display our representations
for layer_name, feature_map in zip(layer_names, successive_feature_maps):
if len(feature_map.shape) == 4:
# Just do this for the conv / maxpool layers, not the fully-connected layers
n_features = feature_map.shape[-1] # number of features in feature map
# The feature map has shape (1, size, size, n_features)
size = feature_map.shape[1]
# We will tile our images in this matrix
display_grid = np.zeros((size, size * n_features))
for i in range(n_features):
# Postprocess the feature to make it visually palatable
x = feature_map[0, :, :, i]
x -= x.mean()
x /= x.std()
x *= 64
x += 128
x = np.clip(x, 0, 255).astype('uint8')
# We'll tile each filter into this big horizontal grid
display_grid[:, i * size : (i + 1) * size] = x
# Display the grid
scale = 20. / n_features
plt.figure(figsize=(scale * n_features, scale))
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')
