Perfect precision

[trevorcampbell]

Moved (manually) from the instructor repo. Need to think about this, and if fix, fix in python too.

i have a question about a line in the textbook in section 6.3 we have this line of text as a "note":

Similarly, we can easily make a classifier that has perfect precision: never guess positive! This classifier will never incorrectly identify an obsevation as positive, but it will make a lot of false negative predictions along the way

if we classifiy eerything as "negative". iesn't the the precision 0? not 100%?
the true positive will be 0 and all the predicted positives will be 0. wouldn't it be 0/0? or am i missing something?

response

technically it's undetermined as 0/0. In cases like that in math you usually have to rule in some reasonable way as to what that means (e.g. there are limits that go to 0/0 but only if at certain rates, while others produce sensible results, and in some cases the sensible one makes more sense as a description of reality)
In this case I would say that a classifier that never guesses positive is perfectly precise; it never claims something is positive when it's actually negative

Maybe this 0/0 clarification is somethign we should clarify in the textbook?

followup:

I personally think that the interpretation of the 0/0 being indeterminant is better. Keeping in mind that precision speaks to the "trust-worthiness" of a positive prediction. In the case that the classifier truly NEVER (having 0 chance) of making a positive prediction, wouldn't then precision be irrelevant? We’d never have to judge the trust-worthiness of a non-existing positive prediction. This situation is similar to the conditional probability being undefined when the conditioning event has prob zero.

Keep in mind what we calculate using the testing data is only an estimate of true classifier precision. It is possible that the classifier happen to not make a positive prediction on that particular testing data set, despite the true prob. of a positive prediction over the whole population being non-zero. This situation would be similar to a survey sample proportion being 0 despite us knowing that the true proportion shouldn’t be 0 — we typically tell the students that we need to collect more data to generate a good estimate. Interpreting 0/0 to be perfect precision is misleading in this case because had we been given the population data, the classifier may indeed make some positive predictions (and they may even be all wrong).