Divergences.jl

Divergences.jl is a Julia package that makes evaluating divergence measures between two vectors easy. The package allows for calculating the gradient and the diagonal of the Hessian of several divergences.

Supported divergences

The package defines an abstract Divergence type with the following suptypes:

• Kullback-Leibler divergence KullbackLeibler
• Chi-square distance ChiSquared
• Reverse Kullback-Leibler divergence ReverseKullbackLeibler
• Cressie-Read divergences CressieRead

These divergences differ from the equivalent ones defined in the Distances package because they are normalized. Also, the package provides methods for calculating their gradient and the (diagonal elements of the) Hessian matrix.

The constructors for the types above are straightforward

KullbackLeibler()
ChiSqaured()
ReverseKullbackLeibler()

The CressieRead type define a family of divergences indexed by a parameter alpha. The constructor for CressieRead is

CR(::Real)

The Hellinger divergence is obtained by CR(-1/2). For a certain value of alpha, CressieRead corresponds to a divergence with a defined specific type. For instance, CR(1) is equivalent to ChiSquared although the underlying code for evaluation and calculation of the gradient and Hessian are different.

Three versions of each divergence in the above list are currently implemented: a vanilla version, a modified version, and a fully modified version. These modifications extend the domain of the divergence.

The modified version takes an additional argument that specifies the point at which a convex extension modifies the divergence.

ModifiedKullbackLeibler(theta::Real)
ModifiedReverseKullbackLeibler(theta::Real)
ModifiedCressieRead(alpha::Real, theta::Real)

Similarly, the fully modified version takes two additional arguments that specify the points at which a convex extension modifies the divergence.

FullyModifiedKullbackLeibler(phi::Real, theta::Real)
FullyModifiedReverseKullbackLeibler(phi::Real, theta::Real)
FullyModifiedCressieRead(alpha::Real, phi::Real, theta::Real)

Basic usage

Divergence between two vectors

Each divergence corresponds to a divergence type. You can always compute a certain divergence between two vectors using the following syntax

d = evaluate(div, x, y)

Here, div is an instance of a divergence type. For example, the type for Kullback Leibler divergence is KullbackLeibler (more divergence types are described in some detail in what follows), then the Kullback Leibler divergence between x and y can be computed

d = evaluate(KullbackLeibler(), x, y)

We can also calculate the divergence between the vector x and the unit vector

r = evaluate(KullbackLeibler(), x)

The Divergence type is a subtype of PreMetric defined in the Distances package. As such, the divergences can be evaluated row-wise and column-wise for X::Matrix and Y::Matrix.

rowise(div, X, Y)

colwise(div, X, Y)

Gradient of the divergence

To calculate the gradient of div::Divergence with respect to x::AbstractArray{Float64, 1} the gradient method can be used

g = gradient(div, x, y)

or through its in-place version

gradient!(Array(Float64, size(x)), div, x, y)

Hessian of the divergence

The hessian method calculates the Hessian of the divergence with respect to x

h = hessian(div, x, y)

Its in-place variant is also defined

hessian!(Array(Float64, size(x)), div, x, y)

Notice that the the divergence's Hessian is sparse, where the diagonal entries are the only ones different from zero. For this reason, hessian(div, x, y) returns an Array{Float64,1} with the diagonal entries of the hessian.

List of divergences

[To be added]