Layers

Functional layers that are trainable with backpropagation.

Gabor layer

Convolutional layer that forces a functional Gabor form for its filters. Every parameter of the Gabor can be learnt.

Managing dtype

Tensorflow is a bit picky when it comes to dtype, so it can be useful to define a function that will ensure that every parameter is casted to the same dtype:

a, b = tf.convert_to_tensor(1), tf.convert_to_tensor(1.1)
print(a.dtype, b.dtype)
# assert a.dtype != b.dtype
<dtype: 'int32'> <dtype: 'float32'>
2022-09-20 12:35:22.979033: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-09-20 12:35:22.979117: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: megatron
2022-09-20 12:35:22.979137: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: megatron
2022-09-20 12:35:22.979295: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 470.57.2
2022-09-20 12:35:22.979361: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.57.2
2022-09-20 12:35:22.979377: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 470.57.2
c, d = cast_all(a, b)
print(c.dtype, d.dtype)
assert c.dtype == d.dtype
<dtype: 'float32'> <dtype: 'float32'>

Creating a Gabor filter in TensorFlow

First of all we need to be able to generate Gabor filters as Tensorflow Tensor:

source

gabor_2d_tf

 gabor_2d_tf (i, j, imean, jmean, sigma_i, sigma_j, freq, theta,
              sigma_theta)
Details
i Horizontal domain
j Vertical domain
imean Horizontal mean
jmean Vertical mean
sigma_i Horizontal width
sigma_j Vertical width
freq Frecuency of the filter
theta Angle of the filter
sigma_theta Width of the angle?? Rotation of the domain??

source

create_gabor_rot_tf

 create_gabor_rot_tf (Nrows, Ncols, imean, jmean, sigma_i, sigma_j, freq,
                      theta, rot_theta, sigma_theta, fs)

Creates a rotated Gabor filter with the input parameters.

Details
Nrows Number of horizontal pixels
Ncols Number of vertical pixels
imean Horizontal mean (in degrees)
jmean Vertical mean (in degrees)
sigma_i Horizontal width (in degrees)
sigma_j Vertical width (in degrees)
freq Frequency
theta Angle
rot_theta Rotation of the domain??
sigma_theta Width of the angle?? Rotation of the domain??
fs Sampling frequency 
gabor = create_gabor_rot_tf(Nrows=20, Ncols=20, imean=0.5, jmean=0.5, sigma_i=0.1, sigma_j=0.1, freq=10, theta=0, rot_theta=0, sigma_theta=0, fs=20)
plt.imshow(gabor)
plt.show()
2022-09-20 12:35:24.252491: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)

Creating a set of Gabor filters

It can be a little bit tricky to translate plain Python for loops into tf.function.

In plain Python, if we wanted to create a set of filters we could initialize an empty array or list and fill it with the different filters generated inside a for loop, but we can’t do that inside a tf.function because Tensorflow tries to build the computational graph and starts to nest graphs inside graphs and the performance is terrible. Luckily for us, they implement a tf.TensorArray that can be used inside a tf.function to this effect.

source

create_multiple_different_rot_gabor_tf

 create_multiple_different_rot_gabor_tf (n_gabors, Nrows, Ncols, imean,
                                         jmean, sigma_i:list,
                                         sigma_j:list, freq:list,
                                         theta:list, rot_theta:list,
                                         sigma_theta:list, fs,
                                         normalize:bool=True)

Creates a set of Gabor filters.

Type Default Details
n_gabors Number of Gabor filters we want to create.
Nrows Number of horizontal pixels.
Ncols Number of vertical pixels.
imean Horizontal mean (in degrees).
jmean Vertical mean (in degrees).
sigma_i list Horizontal width (in degrees).
sigma_j list Vertical width (in degrees).
freq list Frequency.
theta list Angle.
rot_theta list Rotation of the domain??
sigma_theta list Width of the angle?? Rotation of the domain??
fs Sampling frequency.
normalize bool True Wether to normalize (and divide by n_gabors) or not the Gabors. 
n_gabors = 4
gabors = create_multiple_different_rot_gabor_tf(n_gabors=n_gabors, Nrows=20, Ncols=20, imean=0.5, jmean=0.5, sigma_i=[0.1]*n_gabors, sigma_j=[0.1]*n_gabors, freq=[10]*n_gabors, 
                                                theta=[0]*n_gabors, rot_theta=[0]*n_gabors, sigma_theta=[0]*n_gabors, fs=20)
gabors.shape
2022-09-20 12:35:26.542940: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at functional_ops.cc:374 : Internal: No function library
2022-09-20 12:35:26.545272: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at functional_ops.cc:374 : Internal: No function library
2022-09-20 12:35:26.547083: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at functional_ops.cc:374 : Internal: No function library
TensorShape([4, 20, 20])
fig, axes = plt.subplots(nrows=2, ncols=2)
for gabor_filter, ax in zip(gabors, axes.ravel()):
    ax.imshow(gabor_filter)
plt.show()

We can, as well, change the parameters of the Gabor filters independently:

n_gabors = 4
sigma_i = [0.1, 0.2, 0.3, 0.4]
sigma_j = [0.1, 0.2, 0.3, 0.4]
freq = [10, 20, 30, 40]
theta = [0, 45, 90, 135]
rot_theta = [0, 45, 90, 135]
sigma_theta = [0, 45, 90, 135]
gabors = create_multiple_different_rot_gabor_tf(n_gabors=n_gabors, Nrows=20, Ncols=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
                                                theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20)
gabors.shape
2022-09-20 12:35:28.025355: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at functional_ops.cc:374 : Internal: No function library
2022-09-20 12:35:28.026635: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at functional_ops.cc:374 : Internal: No function library
2022-09-20 12:35:28.027776: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at functional_ops.cc:374 : Internal: No function library
TensorShape([4, 20, 20])
fig, axes = plt.subplots(nrows=2, ncols=2)
for gabor_filter, ax in zip(gabors, axes.ravel()):
    ax.imshow(gabor_filter)
plt.show()

Gabor layer

Gabor layer with pre-defined values.

The Gabor filters are stored in the GaborLayer.filters attribute and while they are re-calculated at each step of training (as should be), they are not re-calculated during inference. At inference time, the last calculated filters are used.

source

GaborLayer

 GaborLayer (n_gabors, size, imean, jmean, sigma_i:list, sigma_j:list,
             freq:list, theta:list, rot_theta:list, sigma_theta:list, fs,
             **kwargs)

Pre-initialized Gabor layer that is trainable through backpropagation.

Type Details
n_gabors Number of Gabor filters
size Size of the filters (they will be square),
imean Horizontal mean (in degrees).
jmean Vertical mean (in degrees).
sigma_i list Horizontal width (in degrees).
sigma_j list Vertical width (in degrees).
freq list Frequency.
theta list Rotation of the sinusoid (rad).
rot_theta list Rotation of the domain (rad).
sigma_theta list Rotation of the envelope (rad).
fs Sampling frequency.
kwargs 
a = GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
               theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20, input_shape=(28,28,1))
# a.filters.shape

source

GaborLayer.call

 GaborLayer.call (inputs, training=False)

Build a set of filters from the stored values and convolve them with the input.

Type Default Details
inputs Inputs to the layer.
training bool False Flag indicating if we are training the layer or using it for inference.

It’s important to check if we can use it in a model:

n_gabors = 4
sigma_i = [0.1, 0.2]*2
sigma_j = [0.2, 0.1]*2
freq = [10, 10]*2
theta = [0, np.pi/2]*2
rot_theta = [0, 0]*2
sigma_theta = [0, 0]*2
model = tf.keras.Sequential([
    GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
               theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20, input_shape=(28,28,1)),
    layers.MaxPool2D(2),
    layers.GlobalAveragePooling2D(),
    layers.Dense(10, activation="softmax")
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
gabor_layer_1 (GaborLayer)   (None, 28, 28, 4)         1626      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 4)         0         
_________________________________________________________________
global_average_pooling2d (Gl (None, 4)                 0         
_________________________________________________________________
dense (Dense)                (None, 10)                50        
=================================================================
Total params: 1,676
Trainable params: 76
Non-trainable params: 1,600
_________________________________________________________________
n_gabors = 4
sigma_i = [0.1, 0.2, 0.3, 0.4]
sigma_j = [0.1, 0.2, 0.3, 0.4]
freq = [10, 20, 30, 40]
theta = [0, 45, 90, 135]
rot_theta = [0, 45, 90, 135]
sigma_theta = [0, 45, 90, 135]

gaborlayer = GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
                        theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20)
sample_input = np.random.uniform(0, 1, size=(1, 256, 256, 1))
sample_output = gaborlayer(sample_input).numpy()
assert sample_input.shape[1:3] == sample_output.shape[1:3]
assert sample_output.shape[-1] == 4
fig, axes = plt.subplots(1,2)
axes[0].imshow(sample_input.squeeze())
axes[1].imshow(sample_output.squeeze())
axes[0].set_title("Input")
axes[1].set_title("Output")
plt.show()
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

It can be helpful to implement a method to show the Gabor filters that are being used by the layer:

source

GaborLayer.show_filters

 GaborLayer.show_filters ()

Calculates and plots the filters corresponding to the stored parameters.

gaborlayer = GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
                        theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20)
gaborlayer.show_filters()
WARNING:tensorflow:5 out of the last 5 calls to <function create_multiple_different_rot_gabor_tf> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

We can check that the parameters are trainable and thus the gradient is propagated properly:

gaborlayer = GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
                        theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20)
gaborlayer = GaborLayer(n_gabors=n_gabors, size=20, imean=0.5, jmean=0.5, sigma_i=sigma_i, sigma_j=sigma_j, freq=freq, 
                        theta=theta, rot_theta=rot_theta, sigma_theta=sigma_theta, fs=20)
gaborlayer.build(())
with tf.GradientTape() as tape:
    output = gaborlayer(sample_input, training=True)
    loss = output - output**2
gradients = tape.gradient(loss, gaborlayer.trainable_variables)
gradients
WARNING:tensorflow:6 out of the last 6 calls to <function create_multiple_different_rot_gabor_tf> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[<tf.Tensor: shape=(), dtype=float32, numpy=-44429500.0>,
 <tf.Tensor: shape=(), dtype=float32, numpy=12127090.0>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=
 array([-4.42353867e+04, -8.19999920e+07, -1.21842125e+05, -1.24557440e+08],
       dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=
 array([-4.64066523e+04, -8.19875920e+07, -1.24588711e+05, -1.24616856e+08],
       dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=
 array([-8.8135223e+00, -2.6246947e+05, -1.9752870e+03, -6.3820720e+06],
       dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=
 array([ 3.5485029e-02, -9.0675000e+02,  3.3405488e+03, -8.2995800e+05],
       dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=
 array([-3.6406517e-02,  9.0700000e+02, -3.3405557e+03,  8.2996025e+05],
       dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 0.        ,  0.5       ,  0.00097656, -0.5       ], dtype=float32)>]

Random initialize a simple set of Gabor filters

Insted of defining ourselves the initial values, we can randomly initialize them. This can speed up our testing.

source

create_simple_random_set

 create_simple_random_set (n_gabors, size)

Creates a simple set of randomly initialized squared Gabor filters.

Details
n_gabors Number of Gabor filters we want to create.
size Size of the Gabor (they will be square). 
gabors = create_simple_random_set(n_gabors=4, size=20)
gabors.shape
TensorShape([4, 20, 20])
fig, axes = plt.subplots(nrows=2, ncols=2)
for gabor_filter, ax in zip(gabors, axes.ravel()):
    ax.imshow(gabor_filter)
plt.show()

RandomGabor

Actually, we can define a different layer that initializes almost all of its parameters randomly by inhereting from GaborLayer.

source

RandomGabor

 RandomGabor (n_gabors, size, **kwargs)

Randomly initialized Gabor layer that is trainable through backpropagation.

Details
n_gabors Number of Gabor filters
size Size of the filters (they will be square)
kwargs 
gaborlayer = RandomGabor(n_gabors=4, size=20)

It’s important to check if we can use it in a model:

model = tf.keras.Sequential([
    RandomGabor(n_gabors=n_gabors, size=20, input_shape=(28,28,1)),
    layers.MaxPool2D(2),
    layers.GlobalAveragePooling2D(),
    layers.Dense(10, activation="softmax")
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
random_gabor_4 (RandomGabor) (None, 28, 28, 4)         1626      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 4)         0         
_________________________________________________________________
global_average_pooling2d_1 ( (None, 4)                 0         
_________________________________________________________________
dense_1 (Dense)              (None, 10)                50        
=================================================================
Total params: 1,676
Trainable params: 76
Non-trainable params: 1,600
_________________________________________________________________

We can check that the .call() method from GaborLayer still works:

gaborlayer = RandomGabor(n_gabors=4, size=20)
sample_input = np.random.uniform(0, 1, size=(1, 256, 256, 1))
sample_output = gaborlayer(sample_input).numpy()
assert sample_input.shape[1:3] == sample_output.shape[1:3]
assert sample_output.shape[-1] == 4
fig, axes = plt.subplots(1,2)
axes[0].imshow(sample_input.squeeze())
axes[1].imshow(sample_output.squeeze())
axes[0].set_title("Input")
axes[1].set_title("Output")
plt.show()
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

Ensure that we can still plot its filters:

gaborlayer = RandomGabor(n_gabors=4, size=20)
gaborlayer.show_filters()

We can check that the parameters are trainable and thus the gradient is propagated properly:

gaborlayer = RandomGabor(n_gabors=4, size=20)
with tf.GradientTape() as tape:
    output = gaborlayer(sample_input, training=True)
    loss = output - output**2
gradients = tape.gradient(loss, gaborlayer.trainable_variables)
gradients
[<tf.Tensor: shape=(), dtype=float32, numpy=113917.734>,
 <tf.Tensor: shape=(), dtype=float32, numpy=-917516.5>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([ -5164.865 , 450326.28  ,  -3239.7764, -39325.453 ], dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=
 array([   83248.89 , -2121896.2  ,  -437151.2  ,   -11192.807],
       dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=
 array([ 1.4903282e+03,  2.7136692e+06,  7.4021333e+03, -2.4694915e+03],
       dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([-172227.47 ,  150331.88 , -100282.945,  168037.84 ], dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 149537.19,  -77730.22,  114461.95, -163637.2 ], dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 22690.408, -72601.56 , -14178.506,  -4400.712], dtype=float32)>]