Gabor layer experiment

In this quick experiment we will be training an MNIST classifier using multiple GaborLayer layers.

Library importing

import numpy as np
import matplotlib.pyplot as plt
from einops import rearrange, repeat

import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.datasets import mnist

from flayers.layers import GaborLayer

Data loading

We will be using MNIST for a simple and quick test.

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = repeat(X_train, "b h w ->  b h w c", c=1)/255.0
X_test = repeat(X_test, "b h w ->  b h w c", c=1)/255.0

X_train.shape, Y_train.shape, X_test.shape, Y_test.shape

((60000, 28, 28, 1), (60000,), (10000, 28, 28, 1), (10000,))

Definition of simple model

n_gabors = 4
sigma_i = [0.1, 0.2, 0.1, 0.1]
sigma_j = [0.2, 0.1, 0.2, 0.2]
freq = [10, 10]*2
theta = [0, np.pi/2, np.pi/4, -np.pi/4]
rot_theta = [0, 0]*2
sigma_theta = [0, 0, np.pi/4, -np.pi/4]

model = tf.keras.Sequential([
    GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
               theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20, input_shape=(28,28,1)),
    layers.MaxPool2D(2),
    GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
               theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20),
    layers.MaxPool2D(2),
    GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
               theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20),
    layers.MaxPool2D(2),
    layers.GlobalAveragePooling2D(),
    layers.Dense(10, activation="softmax")
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.summary()

2022-09-20 12:45:08.047414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5435 MB memory:  -> device: 0, name: NVIDIA GeForce GTX TITAN Black, pci bus id: 0000:83:00.0, compute capability: 3.5
2022-09-20 12:45:10.647320: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2022-09-20 12:45:10.896105: I tensorflow/core/util/cuda_solvers.cc:180] Creating CudaSolver handles for stream 0x56262da5ad20

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
gabor_layer (GaborLayer)     (None, 28, 28, 4)         1626      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 4)         0         
_________________________________________________________________
gabor_layer_1 (GaborLayer)   (None, 14, 14, 4)         1626      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 4)           0         
_________________________________________________________________
gabor_layer_2 (GaborLayer)   (None, 7, 7, 4)           1626      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 3, 3, 4)           0         
_________________________________________________________________
global_average_pooling2d (Gl (None, 4)                 0         
_________________________________________________________________
dense (Dense)                (None, 10)                50        
=================================================================
Total params: 4,928
Trainable params: 128
Non-trainable params: 4,800
_________________________________________________________________

We can show the initial Gabor filters:

model.layers[0].show_filters()

history = model.fit(X_train, Y_train, batch_size=128, epochs=2, validation_split=0.2)

Epoch 1/2

2022-09-20 12:45:31.290944: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8100
2022-09-20 12:45:31.798301: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory

375/375 [==============================] - 53s 96ms/step - loss: 2.2773 - accuracy: 0.1696 - val_loss: 2.2126 - val_accuracy: 0.2162
Epoch 2/2
375/375 [==============================] - 35s 95ms/step - loss: 2.1034 - accuracy: 0.2343 - val_loss: 1.9671 - val_accuracy: 0.2958

Showing the training dynamics:

fig, axes = plt.subplots(1,2, figsize=(9,4))
axes[0].plot(history.history['loss'][1:], label="Train")
axes[0].plot(history.history['val_loss'][1:], label="Val")
axes[1].plot(history.history['accuracy'], label="Train")
axes[1].plot(history.history['val_accuracy'], label="Val")
plt.legend()
plt.show()

Calculate the metrics in the test set:

model.evaluate(X_test, Y_test, batch_size=128)

79/79 [==============================] - 0s 4ms/step - loss: 1.9803 - accuracy: 0.2908

[1.9802665710449219, 0.290800005197525]

We can visualize the gabor filters after the training process:

model.layers[0].show_filters()

model.layers[2].show_filters()

model.layers[4].show_filters()

We can even check the atributes of the layer to inspect the change in the initial parameters:

model.layers[0].theta.numpy()*180/np.pi

array([ -2.305845,  87.13862 ,  43.73978 , -44.11806 ], dtype=float32)

model.layers[0].rot_theta.numpy()*180/np.pi

array([ 1.1697422 , -2.6115904 ,  0.03874239,  0.905848  ], dtype=float32)

model.layers[0].sigma_theta.numpy()*180/np.pi

array([ 11.119123,   8.056805,  59.319393, -73.5566  ], dtype=float32)