Convolutional Gaussian Processes#

Mark van der Wilk (July 2019)

Here we show a simple example of the rectangles experiment, where we compare a normal squared exponential GP, and a convolutional GP. This is similar to the experiment in [1].

[1] Van der Wilk, Rasmussen, Hensman (2017). Convolutional Gaussian Processes. Advances in Neural Information Processing Systems 30.

Generate dataset#

Generate a simple dataset of rectangles. We want to classify whether they are tall or wide. NOTE: Here we take care to make sure that the rectangles don’t touch the edge, which is different to the original paper. We do this to avoid needing to use patch weights, which are needed to correctly account for edge effects.

[1]:
import time
import numpy as np
import matplotlib.pyplot as plt

import gpflow
import tensorflow as tf
import tensorflow_probability as tfp

from gpflow import set_trainable
from gpflow.ci_utils import is_continuous_integration

gpflow.config.set_default_float(np.float64)
gpflow.config.set_default_jitter(1e-4)
gpflow.config.set_default_summary_fmt("notebook")

# for reproducibility of this notebook:
np.random.seed(123)
tf.random.set_seed(42)

MAXITER = 2 if is_continuous_integration() else 100
NUM_TRAIN_DATA = (
    5 if is_continuous_integration() else 100
)  # This is less than in the original rectangles dataset
NUM_TEST_DATA = 7 if is_continuous_integration() else 300
H = W = 14  # width and height. In the original paper this is 28
IMAGE_SHAPE = [H, W]
2022-09-20 14:23:43.550358: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-20 14:23:43.709147: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-09-20 14:23:43.709167: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-09-20 14:23:43.741309: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2022-09-20 14:23:44.476806: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2022-09-20 14:23:44.476872: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2022-09-20 14:23:44.476880: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/home/circleci/project/gpflow/experimental/utils.py:42: UserWarning: You're calling gpflow.experimental.check_shapes.decorator.check_shapes which is considered *experimental*. Expect: breaking changes, poor documentation, and bugs.
  warn(
/home/circleci/project/gpflow/experimental/utils.py:42: UserWarning: You're calling gpflow.experimental.check_shapes.inheritance.inherit_check_shapes which is considered *experimental*. Expect: breaking changes, poor documentation, and bugs.
  warn(
[2]:
def affine_scalar_bijector(shift=None, scale=None):
    scale_bijector = tfp.bijectors.Scale(scale) if scale else tfp.bijectors.Identity()
    shift_bijector = tfp.bijectors.Shift(shift) if shift else tfp.bijectors.Identity()
    return shift_bijector(scale_bijector)


def make_rectangle(arr, x0, y0, x1, y1):
    arr[y0:y1, x0] = 1
    arr[y0:y1, x1] = 1
    arr[y0, x0:x1] = 1
    arr[y1, x0 : x1 + 1] = 1


def make_random_rectangle(arr):
    x0 = np.random.randint(1, arr.shape[1] - 3)
    y0 = np.random.randint(1, arr.shape[0] - 3)
    x1 = np.random.randint(x0 + 2, arr.shape[1] - 1)
    y1 = np.random.randint(y0 + 2, arr.shape[0] - 1)
    make_rectangle(arr, x0, y0, x1, y1)
    return x0, y0, x1, y1


def make_rectangles_dataset(num, w, h):
    d, Y = np.zeros((num, h, w)), np.zeros((num, 1))
    for i, img in enumerate(d):
        for j in range(1000):  # Finite number of tries
            x0, y0, x1, y1 = make_random_rectangle(img)
            rw, rh = y1 - y0, x1 - x0
            if rw == rh:
                img[:, :] = 0
                continue
            Y[i, 0] = rw > rh
            break
    return (
        d.reshape(num, w * h).astype(gpflow.config.default_float()),
        Y.astype(gpflow.config.default_float()),
    )
[3]:
X, Y = data = make_rectangles_dataset(NUM_TRAIN_DATA, *IMAGE_SHAPE)
Xt, Yt = test_data = make_rectangles_dataset(NUM_TEST_DATA, *IMAGE_SHAPE)
[4]:
plt.figure(figsize=(8, 3))
for i in range(4):
    plt.subplot(1, 4, i + 1)
    plt.imshow(X[i, :].reshape(*IMAGE_SHAPE))
    plt.title(Y[i, 0])
../../_images/notebooks_advanced_convolutional_5_0.png

Squared Exponential kernel#

[5]:
rbf_m = gpflow.models.SVGP(
    gpflow.kernels.SquaredExponential(),
    gpflow.likelihoods.Bernoulli(),
    gpflow.inducing_variables.InducingPoints(X.copy()),
)
/home/circleci/project/gpflow/experimental/utils.py:42: UserWarning: You're calling gpflow.experimental.check_shapes.checker.ShapeChecker.__init__ which is considered *experimental*. Expect: breaking changes, poor documentation, and bugs.
  warn(
2022-09-20 14:23:47.967815: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-09-20 14:23:47.968503: W tensorflow/stream_executor/cuda/cuda_driver.cc:263] failed call to cuInit: UNKNOWN ERROR (303)
2022-09-20 14:23:47.968527: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (421310ee155f): /proc/driver/nvidia/version does not exist
2022-09-20 14:23:47.969596: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[6]:
rbf_training_loss_closure = rbf_m.training_loss_closure(data, compile=True)
rbf_elbo = lambda: -rbf_training_loss_closure().numpy()
print("RBF elbo before training: %.4e" % rbf_elbo())
RBF elbo before training: -9.9408e+01
[7]:
set_trainable(rbf_m.inducing_variable, False)
start_time = time.time()
res = gpflow.optimizers.Scipy().minimize(
    rbf_training_loss_closure,
    variables=rbf_m.trainable_variables,
    method="l-bfgs-b",
    options={"disp": True, "maxiter": MAXITER},
)
print(f"{res.nfev / (time.time() - start_time):.3f} iter/s")
RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =         5152     M =           10

At X0         0 variables are exactly at the bounds

At iterate    0    f=  9.94077D+01    |proj g|=  1.77693D+01

At iterate    1    f=  8.27056D+01    |proj g|=  1.08235D+01

At iterate    2    f=  7.10286D+01    |proj g|=  2.52642D+00

At iterate    3    f=  6.96128D+01    |proj g|=  9.52086D-01

At iterate    4    f=  6.91601D+01    |proj g|=  2.98596D-01

At iterate    5    f=  6.90683D+01    |proj g|=  4.57667D-01

At iterate    6    f=  6.87999D+01    |proj g|=  8.48569D-01

At iterate    7    f=  6.87526D+01    |proj g|=  1.00002D+00

At iterate    8    f=  6.85857D+01    |proj g|=  1.28576D+00

At iterate    9    f=  6.81708D+01    |proj g|=  2.04265D+00

At iterate   10    f=  6.58595D+01    |proj g|=  1.34225D+00

At iterate   11    f=  6.47617D+01    |proj g|=  9.21923D-01

At iterate   12    f=  6.37115D+01    |proj g|=  7.98398D-01

At iterate   13    f=  6.31593D+01    |proj g|=  3.96800D-01

At iterate   14    f=  6.23801D+01    |proj g|=  1.27642D+00

At iterate   15    f=  6.17198D+01    |proj g|=  7.71734D-01

At iterate   16    f=  6.13083D+01    |proj g|=  6.72739D-01

At iterate   17    f=  6.09537D+01    |proj g|=  4.45487D-01

At iterate   18    f=  6.07507D+01    |proj g|=  5.04335D-01

At iterate   19    f=  6.06242D+01    |proj g|=  1.80232D-01

At iterate   20    f=  6.04829D+01    |proj g|=  1.79481D-01

At iterate   21    f=  6.04182D+01    |proj g|=  1.66211D-01

At iterate   22    f=  6.04028D+01    |proj g|=  1.52839D-01

At iterate   23    f=  6.03795D+01    |proj g|=  4.28437D-02

 This problem is unconstrained.
At iterate   24    f=  6.03764D+01    |proj g|=  4.87340D-02

At iterate   25    f=  6.03741D+01    |proj g|=  3.66168D-02

At iterate   26    f=  6.03735D+01    |proj g|=  1.01438D-01

At iterate   27    f=  6.03723D+01    |proj g|=  3.97105D-02

At iterate   28    f=  6.03721D+01    |proj g|=  1.89009D-02

At iterate   29    f=  6.03719D+01    |proj g|=  1.17457D-02

At iterate   30    f=  6.03716D+01    |proj g|=  1.04475D-02

At iterate   31    f=  6.03716D+01    |proj g|=  1.80645D-02

At iterate   32    f=  6.03714D+01    |proj g|=  2.73618D-03

At iterate   33    f=  6.03714D+01    |proj g|=  2.05487D-03

At iterate   34    f=  6.03714D+01    |proj g|=  2.28429D-03

At iterate   35    f=  6.03714D+01    |proj g|=  2.56206D-03

At iterate   36    f=  6.03714D+01    |proj g|=  8.80378D-04

At iterate   37    f=  6.03714D+01    |proj g|=  3.83350D-04

At iterate   38    f=  6.03714D+01    |proj g|=  3.14405D-04

At iterate   39    f=  6.03714D+01    |proj g|=  2.42994D-04

At iterate   40    f=  6.03714D+01    |proj g|=  5.85343D-04

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
 5152     40     48      1     0     0   5.853D-04   6.037D+01
  F =   60.371382131218326

CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH
31.979 iter/s
[8]:
train_acc = np.mean((rbf_m.predict_y(X)[0] > 0.5).numpy().astype("float") == Y)
test_acc = np.mean((rbf_m.predict_y(Xt)[0] > 0.5).numpy().astype("float") == Yt)
print(f"Train acc: {train_acc * 100}%\nTest acc : {test_acc*100}%")
print("RBF elbo after training: %.4e" % rbf_elbo())
Train acc: 100.0%
Test acc : 68.33333333333333%
RBF elbo after training: -6.0371e+01

Convolutional kernel#

[9]:
f64 = lambda x: np.array(x, dtype=np.float64)
positive_with_min = lambda: affine_scalar_bijector(shift=f64(1e-4))(tfp.bijectors.Softplus())
constrained = lambda: affine_scalar_bijector(shift=f64(1e-4), scale=f64(100.0))(
    tfp.bijectors.Sigmoid()
)
max_abs_1 = lambda: affine_scalar_bijector(shift=f64(-2.0), scale=f64(4.0))(tfp.bijectors.Sigmoid())

patch_shape = [3, 3]
conv_k = gpflow.kernels.Convolutional(gpflow.kernels.SquaredExponential(), IMAGE_SHAPE, patch_shape)
conv_k.base_kernel.lengthscales = gpflow.Parameter(1.0, transform=positive_with_min())
# Weight scale and variance are non-identifiable. We also need to prevent variance from shooting off crazily.
conv_k.base_kernel.variance = gpflow.Parameter(1.0, transform=constrained())
conv_k.weights = gpflow.Parameter(conv_k.weights.numpy(), transform=max_abs_1())
conv_f = gpflow.inducing_variables.InducingPatches(
    np.unique(conv_k.get_patches(X).numpy().reshape(-1, 9), axis=0)
)
[10]:
conv_m = gpflow.models.SVGP(conv_k, gpflow.likelihoods.Bernoulli(), conv_f)
[11]:
set_trainable(conv_m.inducing_variable, False)
set_trainable(conv_m.kernel.base_kernel.variance, False)
set_trainable(conv_m.kernel.weights, False)
[12]:
conv_training_loss_closure = conv_m.training_loss_closure(data, compile=True)
conv_elbo = lambda: -conv_training_loss_closure().numpy()
print("conv elbo before training: %.4e" % conv_elbo())
conv elbo before training: -8.7271e+01
[13]:
start_time = time.time()
res = gpflow.optimizers.Scipy().minimize(
    conv_training_loss_closure,
    variables=conv_m.trainable_variables,
    method="l-bfgs-b",
    options={"disp": True, "maxiter": MAXITER / 10},
)
print(f"{res.nfev / (time.time() - start_time):.3f} iter/s")
RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =         1081     M =           10

At X0         0 variables are exactly at the bounds

At iterate    0    f=  8.72706D+01    |proj g|=  3.34786D+01

At iterate    1    f=  7.06198D+01    |proj g|=  1.23134D+01
 This problem is unconstrained.

At iterate    2    f=  7.03607D+01    |proj g|=  6.56137D+00

At iterate    3    f=  6.98680D+01    |proj g|=  2.96447D+00

At iterate    4    f=  6.93739D+01    |proj g|=  2.88465D+00

At iterate    5    f=  6.88717D+01    |proj g|=  5.27202D+00

At iterate    6    f=  6.60134D+01    |proj g|=  1.11323D+01

At iterate    7    f=  6.51602D+01    |proj g|=  3.43289D+00

At iterate    8    f=  6.49795D+01    |proj g|=  9.75563D-01

At iterate    9    f=  6.48808D+01    |proj g|=  9.60694D-01

4.592 iter/s
At iterate   10    f=  6.48453D+01    |proj g|=  1.39257D+00

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
 1081     10     11      1     0     0   1.393D+00   6.485D+01
  F =   64.845328136876162

STOP: TOTAL NO. of ITERATIONS REACHED LIMIT
[14]:
set_trainable(conv_m.kernel.base_kernel.variance, True)
res = gpflow.optimizers.Scipy().minimize(
    conv_training_loss_closure,
    variables=conv_m.trainable_variables,
    method="l-bfgs-b",
    options={"disp": True, "maxiter": MAXITER},
)
train_acc = np.mean((conv_m.predict_y(X)[0] > 0.5).numpy().astype("float") == Y)
test_acc = np.mean((conv_m.predict_y(Xt)[0] > 0.5).numpy().astype("float") == Yt)
print(f"Train acc: {train_acc * 100}%\nTest acc : {test_acc*100}%")
print("conv elbo after training: %.4e" % conv_elbo())
 This problem is unconstrained.
RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =         1082     M =           10

At X0         0 variables are exactly at the bounds

At iterate    0    f=  6.48453D+01    |proj g|=  3.53996D+00

At iterate    1    f=  6.24919D+01    |proj g|=  7.35357D+00

At iterate    2    f=  6.23531D+01    |proj g|=  7.58264D+00

At iterate    3    f=  6.04166D+01    |proj g|=  1.58157D+01

At iterate    4    f=  5.88887D+01    |proj g|=  2.48005D+01

At iterate    5    f=  5.63927D+01    |proj g|=  4.28536D+01

At iterate    6    f=  5.02144D+01    |proj g|=  1.04313D+02

At iterate    7    f=  4.44685D+01    |proj g|=  8.74280D+01

At iterate    8    f=  3.92359D+01    |proj g|=  3.11481D+01

At iterate    9    f=  3.74686D+01    |proj g|=  3.06073D+01

At iterate   10    f=  3.68864D+01    |proj g|=  4.40037D+01

At iterate   11    f=  3.60668D+01    |proj g|=  7.87412D+00

At iterate   12    f=  3.58935D+01    |proj g|=  1.41535D+01

At iterate   13    f=  3.45651D+01    |proj g|=  7.04865D+01

At iterate   14    f=  3.13752D+01    |proj g|=  4.56939D+01

At iterate   15    f=  3.02012D+01    |proj g|=  2.86627D+01

At iterate   16    f=  2.96655D+01    |proj g|=  1.60272D+01

At iterate   17    f=  2.94558D+01    |proj g|=  8.24031D+00

At iterate   18    f=  2.93105D+01    |proj g|=  2.25261D+00

At iterate   19    f=  2.90992D+01    |proj g|=  5.73397D+00

At iterate   20    f=  2.87163D+01    |proj g|=  1.55953D+01

At iterate   21    f=  2.81823D+01    |proj g|=  2.54265D+01

At iterate   22    f=  2.80343D+01    |proj g|=  2.19902D+01

At iterate   23    f=  2.79432D+01    |proj g|=  1.41609D+01

At iterate   24    f=  2.78263D+01    |proj g|=  2.44911D+00

At iterate   25    f=  2.78000D+01    |proj g|=  1.18706D+00

At iterate   26    f=  2.77623D+01    |proj g|=  3.27247D+00

At iterate   27    f=  2.76356D+01    |proj g|=  5.23204D+00

At iterate   28    f=  2.74130D+01    |proj g|=  6.84226D+00

At iterate   29    f=  2.72099D+01    |proj g|=  2.39793D+00

At iterate   30    f=  2.71505D+01    |proj g|=  8.87291D-01

At iterate   31    f=  2.71225D+01    |proj g|=  1.25368D+00

At iterate   32    f=  2.70691D+01    |proj g|=  3.44551D+00

At iterate   33    f=  2.70246D+01    |proj g|=  3.99899D+00

At iterate   34    f=  2.70165D+01    |proj g|=  1.66840D+00

At iterate   35    f=  2.69736D+01    |proj g|=  1.96735D-01

At iterate   36    f=  2.69568D+01    |proj g|=  4.38151D+00

At iterate   37    f=  2.69408D+01    |proj g|=  1.61116D+00

At iterate   38    f=  2.69308D+01    |proj g|=  1.04302D+00

At iterate   39    f=  2.68977D+01    |proj g|=  1.50422D+00

At iterate   40    f=  2.68704D+01    |proj g|=  2.03393D+00

At iterate   41    f=  2.68478D+01    |proj g|=  7.74728D-01

At iterate   42    f=  2.68370D+01    |proj g|=  6.68973D-01

At iterate   43    f=  2.68312D+01    |proj g|=  1.17495D+00

At iterate   44    f=  2.68205D+01    |proj g|=  1.40366D+00

At iterate   45    f=  2.68164D+01    |proj g|=  2.93859D+00

At iterate   46    f=  2.68066D+01    |proj g|=  1.14149D+00

At iterate   47    f=  2.68038D+01    |proj g|=  2.29234D-01

At iterate   48    f=  2.68031D+01    |proj g|=  6.60556D-02

At iterate   49    f=  2.68025D+01    |proj g|=  6.92338D-01

At iterate   50    f=  2.68010D+01    |proj g|=  7.83827D-01

At iterate   51    f=  2.67992D+01    |proj g|=  2.75352D+00

At iterate   52    f=  2.67965D+01    |proj g|=  1.09880D+00

At iterate   53    f=  2.67954D+01    |proj g|=  1.38053D-01

At iterate   54    f=  2.67951D+01    |proj g|=  1.82728D-01

At iterate   55    f=  2.67943D+01    |proj g|=  2.35894D-01

At iterate   56    f=  2.67915D+01    |proj g|=  1.36174D+00

At iterate   57    f=  2.67888D+01    |proj g|=  1.44453D+00

At iterate   58    f=  2.67867D+01    |proj g|=  6.69400D-01

At iterate   59    f=  2.67852D+01    |proj g|=  2.49436D-01

At iterate   60    f=  2.67849D+01    |proj g|=  1.30110D-01

At iterate   61    f=  2.67838D+01    |proj g|=  3.45478D-01

At iterate   62    f=  2.67818D+01    |proj g|=  7.66751D-01

At iterate   63    f=  2.67784D+01    |proj g|=  9.01109D-01

At iterate   64    f=  2.67780D+01    |proj g|=  1.20388D+00

At iterate   65    f=  2.67748D+01    |proj g|=  8.05237D-01

At iterate   66    f=  2.67725D+01    |proj g|=  9.53573D-02

At iterate   67    f=  2.67716D+01    |proj g|=  2.91392D-01

At iterate   68    f=  2.67709D+01    |proj g|=  2.11336D-01

At iterate   69    f=  2.67707D+01    |proj g|=  7.81715D-01

At iterate   70    f=  2.67699D+01    |proj g|=  4.71523D-01

At iterate   71    f=  2.67687D+01    |proj g|=  7.84486D-02

At iterate   72    f=  2.67676D+01    |proj g|=  3.15235D-01

At iterate   73    f=  2.67667D+01    |proj g|=  4.34509D-01

At iterate   74    f=  2.67662D+01    |proj g|=  8.67849D-01

At iterate   75    f=  2.67653D+01    |proj g|=  1.22312D+00

At iterate   76    f=  2.67642D+01    |proj g|=  5.35684D-01

At iterate   77    f=  2.67634D+01    |proj g|=  5.13194D-02

At iterate   78    f=  2.67632D+01    |proj g|=  1.30113D-01

At iterate   79    f=  2.67626D+01    |proj g|=  1.95710D-01

At iterate   80    f=  2.67625D+01    |proj g|=  5.88159D-01

At iterate   81    f=  2.67616D+01    |proj g|=  3.65936D-01

At iterate   82    f=  2.67614D+01    |proj g|=  2.95283D-01

At iterate   83    f=  2.67611D+01    |proj g|=  1.94345D-01

At iterate   84    f=  2.67606D+01    |proj g|=  1.65529D-01

At iterate   85    f=  2.67587D+01    |proj g|=  1.14022D-01

At iterate   86    f=  2.67585D+01    |proj g|=  4.94482D-01

At iterate   87    f=  2.67577D+01    |proj g|=  3.20143D-01

At iterate   88    f=  2.67576D+01    |proj g|=  8.06103D-01

At iterate   89    f=  2.67569D+01    |proj g|=  2.29646D-01

At iterate   90    f=  2.67567D+01    |proj g|=  8.60409D-02

At iterate   91    f=  2.67563D+01    |proj g|=  2.82486D-01

At iterate   92    f=  2.67559D+01    |proj g|=  2.57009D-01

At iterate   93    f=  2.67557D+01    |proj g|=  6.98616D-01

At iterate   94    f=  2.67553D+01    |proj g|=  3.90014D-01

At iterate   95    f=  2.67549D+01    |proj g|=  1.35560D-01

At iterate   96    f=  2.67545D+01    |proj g|=  1.33563D-01

At iterate   97    f=  2.67542D+01    |proj g|=  2.03929D-01

At iterate   98    f=  2.67540D+01    |proj g|=  4.02593D-01

At iterate   99    f=  2.67536D+01    |proj g|=  3.68131D-01

At iterate  100    f=  2.67533D+01    |proj g|=  3.69556D-01

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
 1082    100    114      1     0     0   3.696D-01   2.675D+01
  F =   26.753331958808452

STOP: TOTAL NO. of ITERATIONS REACHED LIMIT
Train acc: 100.0%
Test acc : 99.0%
conv elbo after training: -2.6753e+01
[15]:
res = gpflow.optimizers.Scipy().minimize(
    conv_training_loss_closure,
    variables=conv_m.trainable_variables,
    method="l-bfgs-b",
    options={"disp": True, "maxiter": MAXITER},
)
train_acc = np.mean((conv_m.predict_y(X)[0] > 0.5).numpy().astype("float") == Y)
test_acc = np.mean((conv_m.predict_y(Xt)[0] > 0.5).numpy().astype("float") == Yt)
print(f"Train acc: {train_acc * 100}%\nTest acc : {test_acc*100}%")
print("conv elbo after training: %.4e" % conv_elbo())
 This problem is unconstrained.
RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =         1082     M =           10

At X0         0 variables are exactly at the bounds

At iterate    0    f=  2.67533D+01    |proj g|=  3.69556D-01

At iterate    1    f=  2.67533D+01    |proj g|=  2.06365D-01

At iterate    2    f=  2.67533D+01    |proj g|=  3.86139D-02

At iterate    3    f=  2.67532D+01    |proj g|=  7.55833D-02

At iterate    4    f=  2.67531D+01    |proj g|=  3.20632D-01

At iterate    5    f=  2.67530D+01    |proj g|=  5.62645D-01

At iterate    6    f=  2.67527D+01    |proj g|=  6.66727D-01

At iterate    7    f=  2.67524D+01    |proj g|=  4.89510D-01

At iterate    8    f=  2.67524D+01    |proj g|=  5.42522D-01

At iterate    9    f=  2.67522D+01    |proj g|=  1.23353D-01

At iterate   10    f=  2.67522D+01    |proj g|=  1.71282D-01

At iterate   11    f=  2.67521D+01    |proj g|=  3.07164D-01

At iterate   12    f=  2.67519D+01    |proj g|=  3.64874D-01

At iterate   13    f=  2.67517D+01    |proj g|=  2.46018D-01

At iterate   14    f=  2.67515D+01    |proj g|=  7.57820D-02

At iterate   15    f=  2.67514D+01    |proj g|=  1.00336D-01

At iterate   16    f=  2.67513D+01    |proj g|=  2.49494D-01

At iterate   17    f=  2.67512D+01    |proj g|=  3.87451D-01

At iterate   18    f=  2.67510D+01    |proj g|=  4.21678D-01

At iterate   19    f=  2.67507D+01    |proj g|=  1.64757D-01

At iterate   20    f=  2.67505D+01    |proj g|=  2.41873D-01

At iterate   21    f=  2.67504D+01    |proj g|=  3.41570D-01

At iterate   22    f=  2.67503D+01    |proj g|=  2.18225D-01

At iterate   23    f=  2.67502D+01    |proj g|=  7.63585D-02

At iterate   24    f=  2.67501D+01    |proj g|=  7.14117D-02

At iterate   25    f=  2.67501D+01    |proj g|=  7.27715D-02

At iterate   26    f=  2.67499D+01    |proj g|=  5.32219D-02

At iterate   27    f=  2.67499D+01    |proj g|=  2.55878D-01

At iterate   28    f=  2.67497D+01    |proj g|=  1.21886D-01

At iterate   29    f=  2.67496D+01    |proj g|=  4.57636D-02

At iterate   30    f=  2.67496D+01    |proj g|=  4.40242D-02

At iterate   31    f=  2.67496D+01    |proj g|=  1.94792D-02

At iterate   32    f=  2.67496D+01    |proj g|=  1.02854D-01

At iterate   33    f=  2.67496D+01    |proj g|=  6.42601D-02

At iterate   34    f=  2.67495D+01    |proj g|=  4.01797D-02

At iterate   35    f=  2.67495D+01    |proj g|=  9.46539D-02

At iterate   36    f=  2.67495D+01    |proj g|=  1.81388D-01

At iterate   37    f=  2.67494D+01    |proj g|=  4.67147D-02

At iterate   38    f=  2.67494D+01    |proj g|=  4.56626D-02

At iterate   39    f=  2.67494D+01    |proj g|=  4.03325D-02

At iterate   40    f=  2.67494D+01    |proj g|=  2.37251D-02

At iterate   41    f=  2.67494D+01    |proj g|=  3.60650D-02

At iterate   42    f=  2.67494D+01    |proj g|=  1.48639D-01

At iterate   43    f=  2.67494D+01    |proj g|=  1.04997D-01

At iterate   44    f=  2.67493D+01    |proj g|=  8.52702D-02

At iterate   45    f=  2.67493D+01    |proj g|=  2.24150D-01

At iterate   46    f=  2.67492D+01    |proj g|=  2.77971D-02

At iterate   47    f=  2.67492D+01    |proj g|=  7.01161D-02

At iterate   48    f=  2.67492D+01    |proj g|=  5.04371D-02

At iterate   49    f=  2.67492D+01    |proj g|=  5.91164D-02

At iterate   50    f=  2.67492D+01    |proj g|=  3.42234D-02

At iterate   51    f=  2.67492D+01    |proj g|=  1.35007D-01

At iterate   52    f=  2.67491D+01    |proj g|=  2.08262D-01

At iterate   53    f=  2.67490D+01    |proj g|=  2.03729D-01

At iterate   54    f=  2.67490D+01    |proj g|=  1.75642D-01

At iterate   55    f=  2.67490D+01    |proj g|=  8.25489D-02

At iterate   56    f=  2.67489D+01    |proj g|=  5.03603D-02

At iterate   57    f=  2.67489D+01    |proj g|=  9.15719D-02

At iterate   58    f=  2.67488D+01    |proj g|=  3.15185D-01

At iterate   59    f=  2.67488D+01    |proj g|=  2.94135D-01

At iterate   60    f=  2.67487D+01    |proj g|=  3.20700D-01

At iterate   61    f=  2.67485D+01    |proj g|=  1.75814D-01

At iterate   62    f=  2.67484D+01    |proj g|=  3.29942D-02

At iterate   63    f=  2.67483D+01    |proj g|=  1.17542D-01

At iterate   64    f=  2.67482D+01    |proj g|=  1.50787D-01

At iterate   65    f=  2.67481D+01    |proj g|=  3.88446D-01

At iterate   66    f=  2.67477D+01    |proj g|=  1.76959D-01

At iterate   67    f=  2.67475D+01    |proj g|=  4.17387D-02

At iterate   68    f=  2.67473D+01    |proj g|=  1.14275D-01

At iterate   69    f=  2.67472D+01    |proj g|=  1.79496D-01

At iterate   70    f=  2.67471D+01    |proj g|=  6.84332D-02

At iterate   71    f=  2.67470D+01    |proj g|=  5.70335D-02

At iterate   72    f=  2.67467D+01    |proj g|=  6.72707D-02

At iterate   73    f=  2.67466D+01    |proj g|=  3.05473D-01

At iterate   74    f=  2.67465D+01    |proj g|=  1.82233D-01

At iterate   75    f=  2.67463D+01    |proj g|=  6.06469D-02

At iterate   76    f=  2.67461D+01    |proj g|=  1.38709D-01

At iterate   77    f=  2.67456D+01    |proj g|=  3.56594D-01

At iterate   78    f=  2.67451D+01    |proj g|=  3.11022D-01

At iterate   79    f=  2.67450D+01    |proj g|=  5.90648D-01

At iterate   80    f=  2.67444D+01    |proj g|=  2.48865D-01

At iterate   81    f=  2.67443D+01    |proj g|=  7.33044D-02

At iterate   82    f=  2.67442D+01    |proj g|=  9.89315D-02

At iterate   83    f=  2.67440D+01    |proj g|=  2.31826D-01

At iterate   84    f=  2.67435D+01    |proj g|=  3.62591D-01

At iterate   85    f=  2.67428D+01    |proj g|=  6.75926D-01

At iterate   86    f=  2.67423D+01    |proj g|=  4.89902D-01

At iterate   87    f=  2.67418D+01    |proj g|=  2.96004D-01

At iterate   88    f=  2.67414D+01    |proj g|=  1.27813D-01

At iterate   89    f=  2.67407D+01    |proj g|=  1.62477D-01

At iterate   90    f=  2.67399D+01    |proj g|=  3.14672D-01

At iterate   91    f=  2.67392D+01    |proj g|=  5.17359D-01

At iterate   92    f=  2.67390D+01    |proj g|=  1.54427D-01

At iterate   93    f=  2.67388D+01    |proj g|=  9.84208D-02

At iterate   94    f=  2.67387D+01    |proj g|=  7.62589D-02

At iterate   95    f=  2.67385D+01    |proj g|=  9.58118D-02

At iterate   96    f=  2.67383D+01    |proj g|=  1.18687D-01

At iterate   97    f=  2.67382D+01    |proj g|=  1.74468D-01

At iterate   98    f=  2.67380D+01    |proj g|=  7.18287D-02

At iterate   99    f=  2.67375D+01    |proj g|=  8.26550D-02

At iterate  100    f=  2.67363D+01    |proj g|=  1.57415D-01

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
 1082    100    111      1     0     0   1.574D-01   2.674D+01
  F =   26.736287663325719

STOP: TOTAL NO. of ITERATIONS REACHED LIMIT
Train acc: 100.0%
Test acc : 99.0%
conv elbo after training: -2.6736e+01
[16]:
set_trainable(conv_m.kernel.weights, True)
res = gpflow.optimizers.Scipy().minimize(
    conv_training_loss_closure,
    variables=conv_m.trainable_variables,
    method="l-bfgs-b",
    options={"disp": True, "maxiter": MAXITER},
)
train_acc = np.mean((conv_m.predict_y(X)[0] > 0.5).numpy().astype("float") == Y)
test_acc = np.mean((conv_m.predict_y(Xt)[0] > 0.5).numpy().astype("float") == Yt)
print(f"Train acc: {train_acc * 100}%\nTest acc : {test_acc*100}%")
print("conv elbo after training: %.4e" % conv_elbo())
RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =         1226     M =           10

At X0         0 variables are exactly at the bounds

At iterate    0    f=  2.67363D+01    |proj g|=  3.15890D-01
 This problem is unconstrained.

At iterate    1    f=  2.66593D+01    |proj g|=  1.23447D+01

At iterate    2    f=  2.59601D+01    |proj g|=  1.31911D+01

At iterate    3    f=  2.38946D+01    |proj g|=  5.04322D+01

At iterate    4    f=  2.21797D+01    |proj g|=  7.72697D+00

At iterate    5    f=  2.18210D+01    |proj g|=  5.20448D+00

At iterate    6    f=  2.15484D+01    |proj g|=  2.52929D+00

At iterate    7    f=  2.09121D+01    |proj g|=  1.21088D+00

At iterate    8    f=  1.96627D+01    |proj g|=  4.42157D+00

At iterate    9    f=  1.88499D+01    |proj g|=  3.41832D+00

At iterate   10    f=  1.82255D+01    |proj g|=  3.55848D+00

At iterate   11    f=  1.79084D+01    |proj g|=  2.36303D+00

At iterate   12    f=  1.78952D+01    |proj g|=  3.36235D+00

At iterate   13    f=  1.78289D+01    |proj g|=  1.01208D+00

At iterate   14    f=  1.78061D+01    |proj g|=  1.28140D+00

At iterate   15    f=  1.77800D+01    |proj g|=  4.28107D+00

At iterate   16    f=  1.77399D+01    |proj g|=  5.38815D-01

At iterate   17    f=  1.77171D+01    |proj g|=  1.45748D+00

At iterate   18    f=  1.77002D+01    |proj g|=  8.60301D-01

At iterate   19    f=  1.76745D+01    |proj g|=  3.24458D-01

At iterate   20    f=  1.76264D+01    |proj g|=  2.78105D-01

At iterate   21    f=  1.75997D+01    |proj g|=  1.23336D-01

At iterate   22    f=  1.75696D+01    |proj g|=  7.70236D-01

At iterate   23    f=  1.75675D+01    |proj g|=  1.71359D+00

At iterate   24    f=  1.75597D+01    |proj g|=  9.01336D-01

At iterate   25    f=  1.75532D+01    |proj g|=  4.75464D-01

At iterate   26    f=  1.75458D+01    |proj g|=  7.28402D-01

At iterate   27    f=  1.75438D+01    |proj g|=  1.03329D+00

At iterate   28    f=  1.75401D+01    |proj g|=  3.09040D-01

At iterate   29    f=  1.75372D+01    |proj g|=  2.97689D-01

At iterate   30    f=  1.75283D+01    |proj g|=  3.93077D-01

At iterate   31    f=  1.75172D+01    |proj g|=  4.99012D-01

At iterate   32    f=  1.75019D+01    |proj g|=  4.42903D-01

At iterate   33    f=  1.74767D+01    |proj g|=  8.11521D-01

At iterate   34    f=  1.74668D+01    |proj g|=  1.55860D+00

At iterate   35    f=  1.74598D+01    |proj g|=  2.89704D+00

At iterate   36    f=  1.74518D+01    |proj g|=  5.30937D-01

At iterate   37    f=  1.74498D+01    |proj g|=  2.79222D-01

At iterate   38    f=  1.74485D+01    |proj g|=  3.79688D-01

At iterate   39    f=  1.74466D+01    |proj g|=  1.39104D+00

At iterate   40    f=  1.74434D+01    |proj g|=  8.27033D-01

At iterate   41    f=  1.74411D+01    |proj g|=  8.58730D-01

At iterate   42    f=  1.74314D+01    |proj g|=  1.00359D+00

At iterate   43    f=  1.74250D+01    |proj g|=  9.18368D-01

At iterate   44    f=  1.74148D+01    |proj g|=  5.05909D-01

At iterate   45    f=  1.74124D+01    |proj g|=  7.85410D-01

At iterate   46    f=  1.74061D+01    |proj g|=  1.72040D+00

At iterate   47    f=  1.74016D+01    |proj g|=  9.82946D-01

At iterate   48    f=  1.73960D+01    |proj g|=  3.56201D-01

At iterate   49    f=  1.73897D+01    |proj g|=  4.66155D-01

At iterate   50    f=  1.73790D+01    |proj g|=  8.06765D-01

At iterate   51    f=  1.73687D+01    |proj g|=  6.68134D-01

At iterate   52    f=  1.73640D+01    |proj g|=  1.59985D+00

At iterate   53    f=  1.73575D+01    |proj g|=  3.21741D-01

At iterate   54    f=  1.73540D+01    |proj g|=  5.82679D-01

At iterate   55    f=  1.73476D+01    |proj g|=  7.50497D-01

At iterate   56    f=  1.73343D+01    |proj g|=  8.32714D-01

At iterate   57    f=  1.73297D+01    |proj g|=  1.87886D+00

At iterate   58    f=  1.73082D+01    |proj g|=  1.35077D+00

At iterate   59    f=  1.72966D+01    |proj g|=  1.61551D+00

At iterate   60    f=  1.72792D+01    |proj g|=  8.95599D-01

At iterate   61    f=  1.72629D+01    |proj g|=  4.38795D-01

At iterate   62    f=  1.72463D+01    |proj g|=  1.98290D-01

At iterate   63    f=  1.72448D+01    |proj g|=  1.30648D+00

At iterate   64    f=  1.72353D+01    |proj g|=  5.29832D-01

At iterate   65    f=  1.72286D+01    |proj g|=  1.86780D-01

At iterate   66    f=  1.72173D+01    |proj g|=  5.29611D-01

At iterate   67    f=  1.72027D+01    |proj g|=  6.74830D-01

At iterate   68    f=  1.72005D+01    |proj g|=  1.23360D+00

At iterate   69    f=  1.71839D+01    |proj g|=  7.81289D-01

At iterate   70    f=  1.71796D+01    |proj g|=  7.93730D-01

At iterate   71    f=  1.71718D+01    |proj g|=  4.78583D-01

At iterate   72    f=  1.71642D+01    |proj g|=  3.34851D-01

At iterate   73    f=  1.71543D+01    |proj g|=  1.65164D+00

At iterate   74    f=  1.71522D+01    |proj g|=  2.92592D-01

At iterate   75    f=  1.71470D+01    |proj g|=  2.37492D-01

At iterate   76    f=  1.71289D+01    |proj g|=  7.56450D-01

At iterate   77    f=  1.71213D+01    |proj g|=  4.03391D-01

At iterate   78    f=  1.71175D+01    |proj g|=  1.60273D-01

At iterate   79    f=  1.71117D+01    |proj g|=  6.91271D-01

At iterate   80    f=  1.71087D+01    |proj g|=  7.43123D-01

At iterate   81    f=  1.71059D+01    |proj g|=  3.98718D-01

At iterate   82    f=  1.71032D+01    |proj g|=  2.70994D-01

At iterate   83    f=  1.71007D+01    |proj g|=  1.22893D-01

At iterate   84    f=  1.70956D+01    |proj g|=  5.84796D-01

At iterate   85    f=  1.70912D+01    |proj g|=  4.92854D-01

At iterate   86    f=  1.70883D+01    |proj g|=  7.66597D-01

At iterate   87    f=  1.70864D+01    |proj g|=  6.27289D-01

At iterate   88    f=  1.70852D+01    |proj g|=  2.05707D-01

At iterate   89    f=  1.70840D+01    |proj g|=  6.16601D-02

At iterate   90    f=  1.70824D+01    |proj g|=  2.37873D-01

At iterate   91    f=  1.70809D+01    |proj g|=  2.89050D-01

At iterate   92    f=  1.70803D+01    |proj g|=  5.73995D-01

At iterate   93    f=  1.70786D+01    |proj g|=  1.65289D-01

At iterate   94    f=  1.70773D+01    |proj g|=  1.79195D-01

At iterate   95    f=  1.70754D+01    |proj g|=  2.50309D-01

At iterate   96    f=  1.70727D+01    |proj g|=  3.16541D-01

At iterate   97    f=  1.70713D+01    |proj g|=  7.87999D-01

At iterate   98    f=  1.70686D+01    |proj g|=  4.99336D-01

At iterate   99    f=  1.70672D+01    |proj g|=  7.34938D-02

At iterate  100    f=  1.70665D+01    |proj g|=  9.01578D-02

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
 1226    100    121      1     0     0   9.016D-02   1.707D+01
  F =   17.066503593133774

STOP: TOTAL NO. of ITERATIONS REACHED LIMIT
Train acc: 100.0%
Test acc : 96.66666666666667%
conv elbo after training: -1.7067e+01
[17]:
gpflow.utilities.print_summary(rbf_m)
name class transform prior trainable shape dtype value
SVGP.kernel.variance ParameterSoftplus True () float643.56655
SVGP.kernel.lengthscalesParameterSoftplus True () float642.75129
SVGP.inducing_variable.ZParameterIdentity False (100, 196) float64[[0., 0., 0....
SVGP.q_mu ParameterIdentity True (100, 1) float64[[-5.78240e-01...
SVGP.q_sqrt ParameterFillTriangular True (1, 100, 100)float64[[[6.4475e-01, 0.0000e+00, 0.0000e+00...
[18]:
gpflow.utilities.print_summary(conv_m)
name class transform prior trainable shape dtype value
SVGP.kernel.base_kernel.variance ParameterSigmoid + Chain True () float6499.98292
SVGP.kernel.base_kernel.lengthscalesParameterSoftplus + Chain True () float640.64773
SVGP.kernel.weights ParameterSigmoid + Chain True (144,) float64[0.11865, 0.30752, 0.43216...
SVGP.inducing_variable.Z ParameterIdentity False (45, 9) float64[[0., 0., 0....
SVGP.q_mu ParameterIdentity True (45, 1) float64[[0.00999...
SVGP.q_sqrt ParameterFillTriangular True (1, 45, 45)float64[[[7.2640e-02, 0.0000e+00, 0.0000e+00...

Conclusion#

The convolutional kernel performs much better in this simple task. It demonstrates non-local generalization of the strong assumptions in the kernel.