CLLR/Cross Entropy Calculation

[jfiscus]

In line

PYLLR/pyllr/cllr.py

Line 28 in 4ded376

return cross_entropy(tar,non,Ptar=0.5,deriv=deriv)

, Should the output of cross_entropy be divided by log2 in cllr() rather than division occurring on

PYLLR/pyllr/cllr.py

Line 13 in 4ded376

return ( Ptar*t + (1-Ptar)*n ) / log2

?

[bsxfan]

Hi Jonathan

I think this code is doing what I intended. The division by log2 is to convert from nats to bits. The nats is because of the default use of the natural log everywhere. (I.E. we are converting from log base e to log base 2.) Since this is a binary classification problem, bits is perhaps the best scale on which to quantify information. With this scaling, 1 is a very useful reference point and we want that to hold for Cllr, but also more generally for prior-weighted cross-entropy (cross_entropy in the code) at any prior.

Cllr specializes the prior-weighted cross-entropy to the case where the prior weighting is flat (prior = 1/2). The prior and the scaling are independent choices---i.e. the prior = 1/2 and the log2 are not related.

-- Niko

[jfiscus]

Niko,

Thanks for the explanation of nats to bits - that makes sense. I'm working on a new evaluation plan that uses Cross Entropy, and Cllr. Does that mean that the cross_entropy function is actually calculating Cllr?

-Jon