gpflow.optimizers¶

gpflow.optimizers.NaturalGradient¶

class gpflow.optimizers.NaturalGradient(gamma, xi_transform=<gpflow.optimizers.natgrad.XiNat object>, name=None)[source]¶

Bases: keras.optimizer_v2.optimizer_v2.OptimizerV2

Implements a natural gradient descent optimizer for variational models that are based on a distribution q(u) = N(q_mu, q_sqrt q_sqrtᵀ) that is parameterized by mean q_mu and lower-triangular Cholesky factor q_sqrt of the covariance.

Note that this optimizer does not implement the standard API of tf.optimizers.Optimizer. Its only public method is minimize(), which has a custom signature (var_list needs to be a list of (q_mu, q_sqrt) tuples, where q_mu and q_sqrt are gpflow.Parameter instances, not tf.Variable).

Note furthermore that the natural gradients are implemented only for the full covariance case (i.e., q_diag=True is NOT supported).

When using in your work, please cite

@inproceedings{salimbeni18,
title={Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models}, author={Salimbeni, Hugh and Eleftheriadis, Stefanos and Hensman, James}, booktitle={AISTATS}, year={2018}

Attributes

clipnorm: float or None. If set, clips gradients to a maximum norm.
clipvalue: float or None. If set, clips gradients to a maximum value.
global_clipnorm: float or None. If set, clips gradients to a maximum norm.
iterations: Variable.
weights: Returns variables of this Optimizer based on the order created.

Methods

`add_slot`(var, slot_name[, initializer, shape])	Add a new slot variable for var.
`apply_gradients`(grads_and_vars[, name, ...])	Apply gradients to variables.
`from_config`(config[, custom_objects])	Creates an optimizer from its config.
`get_config`()	Returns the config of the optimizer.
`get_gradients`(loss, params)	Returns gradients of loss with respect to params.
`get_slot_names`()	A list of names for this optimizer's slots.
`get_weights`()	Returns the current weights of the optimizer.
`minimize`(loss_fn, var_list)	Minimizes objective function of the model.
`set_weights`(weights)	Set the weights of the optimizer.
`variables`()	Returns variables of this Optimizer based on the order created.

add_weight
get_slot
get_updates

Parameters

gamma (Union[float, Tensor, ndarray]) –
xi_transform (XiTransform) –
name (Optional[str]) –

get_config()[source]¶

Returns the config of the optimizer.

An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration.

Returns:: Python dictionary.

Return type: Dict[str, Any]

minimize(loss_fn, var_list)[source]¶

Minimizes objective function of the model. Natural Gradient optimizer works with variational parameters only.

Parameters

loss_fn (Callable[[], Tensor]) – Loss function.
var_list (Sequence[Union[Tuple[Parameter, Parameter], Tuple[Parameter, Parameter, XiTransform]]]) –
List of pair tuples of variational parameters or triplet tuple with variational parameters and ξ transformation. If ξ is not specified, will use self.xi_transform. For example, var_list could be ``` var_list = [

(q_mu1, q_sqrt1), (q_mu2, q_sqrt2, XiSqrtMeanVar())

GPflow implements the XiNat (default) and XiSqrtMeanVar transformations for parameters. Custom transformations that implement the XiTransform interface are also possible.

Return type: None

gpflow.optimizers.SamplingHelper¶

class gpflow.optimizers.SamplingHelper(target_log_prob_fn, parameters)[source]¶

Bases: object

This helper makes it easy to read from variables being set with a prior and writes values back to the same variables.

Example:

model = … # Create a GPflow model hmc_helper = SamplingHelper(model.log_posterior_density, model.trainable_parameters)

target_log_prob_fn = hmc_helper.target_log_prob_fn current_state = hmc_helper.current_state

hmc = tfp.mcmc.HamiltonianMonteCarlo(target_log_prob_fn=target_log_prob_fn, …) adaptive_hmc = tfp.mcmc.SimpleStepSizeAdaptation(hmc, …)

@tf.function def run_chain_fn():

return mcmc.sample_chain(
num_samples, num_burnin_steps, current_state, kernel=adaptive_hmc)

hmc_samples = run_chain_fn() parameter_samples = hmc_helper.convert_to_constrained_values(hmc_samples)

Attributes

current_state: Return the current state of the unconstrained variables, used in HMC.
target_log_prob_fn: The target log probability, adjusted to allow for optimisation to occur on the tracked unconstrained underlying variables.

Methods

convert_to_constrained_values(hmc_samples)

Converts list of unconstrained values in hmc_samples to constrained versions.

Parameters

target_log_prob_fn (Callable[[], Tensor]) –
parameters (Sequence[Parameter]) –

convert_to_constrained_values(hmc_samples)[source]¶

Converts list of unconstrained values in hmc_samples to constrained versions. Each value in the list corresponds to an entry in parameters passed to the constructor; for parameters that have a transform, the constrained representation is returned.

Parameters: hmc_samples (Sequence[Tensor]) –
Return type: Sequence[Tensor]

property current_state: Sequence[tensorflow.python.ops.variables.Variable]¶

Return the current state of the unconstrained variables, used in HMC.

Return type: Sequence[Variable]

property target_log_prob_fn: Callable[[...], Tuple[tensorflow.python.framework.ops.Tensor, Callable[[...], Tuple[tensorflow.python.framework.ops.Tensor, Sequence[None]]]]]¶

The target log probability, adjusted to allow for optimisation to occur on the tracked unconstrained underlying variables.

Return type: Callable[..., Tuple[Tensor, Callable[..., Tuple[Tensor, Sequence[None]]]]]

gpflow.optimizers.Scipy¶

class gpflow.optimizers.Scipy[source]¶

Bases: object

Methods

minimize(closure, variables[, method, ...])

Minimize is a wrapper around the scipy.optimize.minimize function handling the packing and unpacking of a list of shaped variables on the TensorFlow side vs.

assign_tensors
callback_func
eval_func
initial_parameters
pack_tensors
unpack_tensors

minimize(closure, variables, method='L-BFGS-B', step_callback=None, compile=True, allow_unused_variables=False, **scipy_kwargs)[source]¶

Minimize is a wrapper around the scipy.optimize.minimize function handling the packing and unpacking of a list of shaped variables on the TensorFlow side vs. the flat numpy array required on the Scipy side.

Args:

closure: A closure that re-evaluates the model, returning the loss: to be minimized.
variables: The list (tuple) of variables to be optimized: (typically model.trainable_variables)

method: The type of solver to use in SciPy. Defaults to “L-BFGS-B”. step_callback: If not None, a callable that gets called once after

each optimisation step. The callable is passed the arguments step, variables, and values. step is the optimisation step counter, variables is the list of trainable variables as above, and values is the corresponding list of tensors of matching shape that contains their value at this optimisation step.

compile: If True, wraps the evaluation function (the passed closure: as well as its gradient computation) inside a tf.function(), which will improve optimization speed in most cases.
allow_unused_variables: Whether to allow variables that are not: actually used in the closure.
scipy_kwargs: Arguments passed through to scipy.optimize.minimize: Note that Scipy’s minimize() takes a callback argument, but you probably want to use our wrapper and pass in step_callback.

Returns:

The optimization result represented as a Scipy OptimizeResult object. See the Scipy documentation for description of attributes.

Parameters

closure (Callable[[], Tensor]) –
variables (Sequence[Variable]) –
method (Optional[str]) –
step_callback (Union[Callable[[int, Sequence[Variable], Sequence[Tensor]], None], Monitor, None]) –
compile (bool) –
allow_unused_variables (bool) –
scipy_kwargs (Any) –

Return type

OptimizeResult

gpflow.optimizers.XiNat¶

class gpflow.optimizers.XiNat[source]¶

Bases: gpflow.optimizers.natgrad.XiTransform

This is the default transform. Using the natural directly saves the forward mode gradient, and also gives the analytic optimal solution for gamma=1 in the case of Gaussian likelihood.

Methods

`meanvarsqrt_to_xi`(mean, varsqrt)	Transforms the parameter mean and varsqrt to xi1, xi2
`naturals_to_xi`(nat1, nat2)	Applies the transform so that nat1, nat2 is mapped to xi1, xi2
`xi_to_meanvarsqrt`(xi1, xi2)	Transforms the parameter xi1, xi2 to mean, varsqrt

static meanvarsqrt_to_xi(mean, varsqrt)[source]¶

Transforms the parameter mean and varsqrt to xi1, xi2

Parameters

mean (Tensor) – the mean parameter (N, D)
varsqrt (Tensor) – the varsqrt parameter (D, N, N)

Return type

Tuple[Tensor, Tensor]

Returns

tuple (xi1, xi2), the xi parameters (N, D), (D, N, N)

static naturals_to_xi(nat1, nat2)[source]¶

Applies the transform so that nat1, nat2 is mapped to xi1, xi2

Parameters

nat1 (Tensor) – the θ₁ parameter
nat2 (Tensor) – the θ₂ parameter

Return type

Tuple[Tensor, Tensor]

Returns

tuple xi1, xi2

static xi_to_meanvarsqrt(xi1, xi2)[source]¶

Transforms the parameter xi1, xi2 to mean, varsqrt

Parameters

xi1 (Tensor) – the ξ₁ parameter
xi2 (Tensor) – the ξ₂ parameter

Return type

Tuple[Tensor, Tensor]

Returns

tuple (mean, varsqrt), the meanvarsqrt parameters

gpflow.optimizers.XiSqrtMeanVar¶

class gpflow.optimizers.XiSqrtMeanVar[source]¶

Bases: gpflow.optimizers.natgrad.XiTransform

This transformation will perform natural gradient descent on the model parameters, so saves the conversion to and from Xi.

Methods

`meanvarsqrt_to_xi`(mean, varsqrt)	Transforms the parameter mean and varsqrt to xi1, xi2
`naturals_to_xi`(nat1, nat2)	Applies the transform so that nat1, nat2 is mapped to xi1, xi2
`xi_to_meanvarsqrt`(xi1, xi2)	Transforms the parameter xi1, xi2 to mean, varsqrt

static meanvarsqrt_to_xi(mean, varsqrt)[source]¶

Transforms the parameter mean and varsqrt to xi1, xi2

Parameters

mean (Tensor) – the mean parameter (N, D)
varsqrt (Tensor) – the varsqrt parameter (D, N, N)

Return type

Tuple[Tensor, Tensor]

Returns

tuple (xi1, xi2), the xi parameters (N, D), (D, N, N)

static naturals_to_xi(nat1, nat2)[source]¶

Applies the transform so that nat1, nat2 is mapped to xi1, xi2

Parameters

nat1 (Tensor) – the θ₁ parameter
nat2 (Tensor) – the θ₂ parameter

Return type

Tuple[Tensor, Tensor]

Returns

tuple xi1, xi2

static xi_to_meanvarsqrt(xi1, xi2)[source]¶

Transforms the parameter xi1, xi2 to mean, varsqrt

Parameters

xi1 (Tensor) – the ξ₁ parameter
xi2 (Tensor) – the ξ₂ parameter

Return type

Tuple[Tensor, Tensor]

Returns

tuple (mean, varsqrt), the meanvarsqrt parameters

gpflow.optimizers.XiTransform¶

class gpflow.optimizers.XiTransform[source]¶

Bases: object

XiTransform is the base class that implements three transformations necessary for the natural gradient calculation wrt any parameterization. This class does not handle any shape information, but it is assumed that the parameters pairs are always of shape (N, D) and (D, N, N).

Methods

`meanvarsqrt_to_xi`(mean, varsqrt)	Transforms the parameter mean and varsqrt to xi1, xi2
`naturals_to_xi`(nat1, nat2)	Applies the transform so that nat1, nat2 is mapped to xi1, xi2
`xi_to_meanvarsqrt`(xi1, xi2)	Transforms the parameter xi1, xi2 to mean, varsqrt

abstract static meanvarsqrt_to_xi(mean, varsqrt)[source]¶

Transforms the parameter mean and varsqrt to xi1, xi2

Parameters

mean (Tensor) – the mean parameter (N, D)
varsqrt (Tensor) – the varsqrt parameter (D, N, N)

Return type

Tuple[Tensor, Tensor]

Returns

tuple (xi1, xi2), the xi parameters (N, D), (D, N, N)

abstract static naturals_to_xi(nat1, nat2)[source]¶

Applies the transform so that nat1, nat2 is mapped to xi1, xi2

Parameters

nat1 (Tensor) – the θ₁ parameter
nat2 (Tensor) – the θ₂ parameter

Return type

Tuple[Tensor, Tensor]

Returns

tuple xi1, xi2

abstract static xi_to_meanvarsqrt(xi1, xi2)[source]¶

Transforms the parameter xi1, xi2 to mean, varsqrt

Parameters

xi1 (Tensor) – the ξ₁ parameter
xi2 (Tensor) – the ξ₂ parameter

Return type

Tuple[Tensor, Tensor]

Returns

tuple (mean, varsqrt), the meanvarsqrt parameters

gpflow.optimizers.natgrad¶

gpflow.optimizers.natgrad