---
jupyter:
  anaconda-cloud: {}
  jupytext:
    notebook_metadata_filter: all,-language_info
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.2'
      jupytext_version: 1.5.1
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

# Accuracy of the classifier

```{python tags=c("hide-cell")}
import numpy as np
# Make random number generator.
rng = np.random.default_rng()
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
import pandas as pd
pd.set_option('mode.copy_on_write', True)
```

```{python tags=c("hide-cell")}
def distance(point1, point2):
    """Returns the distance between point1 and point2
    where each argument is an array
    consisting of the coordinates of the point"""
    return np.sqrt(np.sum((point1 - point2)**2))

def all_distances(training, new_point):
    """Returns an array of distances
    between each point in the training set
    and the new point (which is a row of attributes)"""
    attributes = training.drop(columns='Class')
    def distance_from_point(row):
        return distance(np.array(new_point), np.array(row))
    return attributes.apply(distance_from_point, axis=1)

def table_with_distances(training, new_point):
    """Augments the training table
    with a column of distances from new_point"""
    out = training.copy()
    out['Distance'] = all_distances(training, new_point)
    return out

def closest(training, new_point, k):
    """Returns a table of the k rows of the augmented table
    corresponding to the k smallest distances"""
    with_dists = table_with_distances(training, new_point)
    sorted_by_distance = with_dists.sort_values('Distance')
    topk = sorted_by_distance.iloc[:k]
    return topk

def majority(topkclasses):
    ones = np.count_nonzero(topkclasses == 1)
    zeros = np.count_nonzero(topkclasses == 0)
    if ones > zeros:
        return 1
    else:
        return 0

def classify(training, new_point, k):
    closestk = closest(training, new_point, k)
    return majority(closestk['Class'])
```

You can download the data file for this page from
{download}`wine.csv <../data/wine.csv>`.

```{python tags=c("hide-cell")}
wine = pd.read_csv('wine.csv')
wine.loc[wine['Class'] != 1, 'Class'] = 0
```

## The idea of accuracy


To see how well our classifier does, we might put 50% of the data into the training set and the other 50% into the test set.  Basically, we are setting aside some data for later use, so we can use it to measure the accuracy of our classifier.  We've been calling that the *test set*. Sometimes people will call the data that you set aside for testing a *hold-out set*, and they'll call this strategy for estimating accuracy the *hold-out method*.

Note that this approach requires great discipline.  Before you start applying machine learning methods, you have to take some of your data and set it aside for testing.  You must avoid using the test set for developing your classifier: you shouldn't use it to help train your classifier or tweak its settings or for brainstorming ways to improve your classifier.  Instead, you should use it only once, at the very end, after you've finalized your classifier, when you want an unbiased estimate of its accuracy.


## Measuring the Accuracy of Our Wine Classifier


OK, so let's apply the hold-out method to evaluate the effectiveness of the $k$-nearest neighbor classifier for identifying wines.  The data set has 178 wines, so we'll randomly permute the data set and put 89 of them in the training set and the remaining 89 in the test set.

```{python}
n_wine = len(wine)
shuffled_wine = wine.sample(n_wine, replace=False)
training_set = shuffled_wine.iloc[:89]
test_set  = shuffled_wine.iloc[89:]
```

We'll train the classifier using the 89 wines in the training set, and evaluate how well it performs on the test set. To make our lives easier, we'll write a function to evaluate a classifier on every wine in the test set:

```{python}
def count_zero(array):
    """Counts the number of 0's in an array"""
    return len(array) - np.count_nonzero(array)

def count_equal(array1, array2):
    """Takes two numerical arrays of equal length
    and counts the indices where the two are equal"""
    return count_zero(array1 - array2)

def evaluate_accuracy(training, test, k):
    test_attributes = test.drop(columns='Class')
    def classify_testrow(row):
        return classify(training, row, k)
    c = test_attributes.apply(classify_testrow, axis=1)
    return count_equal(c, test['Class']) / len(test)
```

Now for the grand reveal -- let's see how we did.  We'll arbitrarily use $k=5$.

```{python}
evaluate_accuracy(training_set, test_set, 5)
```

The accuracy rate isn't bad at all for a simple classifier.


## Breast Cancer Diagnosis

Now I want to do an example based on diagnosing breast cancer.  I was inspired by Brittany Wenger, who won the Google national science fair in 2012 as a 17-year old high school student.  Here's Brittany:

![Brittany Wenger](https://i.huffpost.com/gen/701499/thumbs/o-GSF83-570.jpg?3)

[Brittany's science fair
project](https://sites.google.com/a/googlesciencefair.com/science-fair-2012-project-64a91af142a459cfb486ed5cb05f803b2eb41354-1333130785-87)
was to build a classification algorithm to diagnose breast cancer.  She won
grand prize for building an algorithm whose accuracy was almost 99%.

Let's see how well we can do, with the ideas we've learned in this course.

So, let me tell you a little bit about the data set.  Basically, if a woman has a lump in her breast, the doctors may want to take a biopsy to see if it is cancerous.  There are several different procedures for doing that.  Brittany focused on fine needle aspiration (FNA), because it is less invasive than the alternatives.  The doctor gets a sample of the mass, puts it under a microscope, takes a picture, and a trained lab tech analyzes the picture to determine whether it is cancer or not.  We get a picture like one of the following:

![benign](https://github.com/matthew-brett/unlicensed-files/raw/main/breast_fna_images/fna_benign.png)

![cancer](https://github.com/matthew-brett/unlicensed-files/raw/main/breast_fna_images/fna_malignant.png)

Unfortunately, distinguishing between benign and malignant can be tricky.  So, researchers have studied the use of machine learning to help with this task.  The idea is that we'll ask the lab tech to analyze the image and compute various attributes: things like the typical size of a cell, how much variation there is among the cell sizes, and so on.  Then, we'll try to use this information to predict (classify) whether the sample is malignant or not.  We have a training set of past samples from women where the correct diagnosis is known, and we'll hope that our machine learning algorithm can use those to learn how to predict the diagnosis for future samples.

We end up with the following data set.  For the "Class" column, 1 means malignant (cancer); 0 means benign (not cancer).

Download the file from {download}`breast-cancer.csv <../data/breast-cancer.csv>`.

```{python}
patients = pd.read_csv('breast-cancer.csv')
patients = patients.drop(columns='ID')
patients.head()
```

So we have 9 different attributes.  I don't know how to make a 9-dimensional scatterplot of all of them, so I'm going to pick two and plot them:

```{python}
patients_with_colors = patients.copy()
patients_with_colors['Color'] = 'darkblue'
patients_with_colors.loc[patients['Class'] == 0, 'Color'] = 'gold'
```

```{python}
patients_with_colors.plot.scatter(
    'Bland Chromatin',
    'Single Epithelial Cell Size',
    c=patients_with_colors['Color']);
```

Oops.  That plot is utterly misleading, because there are a bunch of points that have identical values for both the x- and y-coordinates.  To make it easier to see all the data points, I'm going to add a little bit of random jitter to the x- and y-values.  Here's how that looks:

```{python tags=c("remove-input")}
# NO CODE

def randomize_column(a):
    """ Add some random jitter to vector """
    return a + rng.normal(0.0, 0.09, size=len(a))

jittered = pd.DataFrame()
jittered['Bland Chromatin (jittered)'] = randomize_column(patients['Bland Chromatin'])
jittered['Single Epithelial Cell Size (jittered)'] = randomize_column(patients['Single Epithelial Cell Size'])
jittered['Class'] = patients['Class']
jittered['Color'] = 'darkblue'
jittered.loc[jittered['Class'] == 0, 'Color'] = 'gold'
jittered.plot.scatter(0, 1, c=jittered['Color']);
```

For instance, you can see there are lots of samples with chromatin = 2 and epithelial cell size = 2; all non-cancerous.

Keep in mind that the jittering is just for visualization purposes, to make it easier to get a feeling for the data.  We're ready to work with the data now, and we'll use the original (unjittered) data.


First we'll create a training set and a test set. The data set has 683 patients, so we'll randomly permute the data set and put 342 of them in the training set and the remaining 341 in the test set.

```{python}
n_patients = len(patients)
shuffled_patients = patients.sample(n_patients, replace=False)
training_set = shuffled_patients.iloc[:342]
test_set = shuffled_patients.iloc[342:]
```

Let's stick with 5 nearest neighbors, and see how well our classifier does.

```{python}
evaluate_accuracy(training_set, test_set, 5)
```

Over 96% accuracy.  Not bad!  Once again, pretty darn good for such a simple technique.

As a footnote, you might have noticed that Brittany Wenger did even better.  What techniques did she use? One key innovation is that she incorporated a confidence score into her results: her algorithm had a way to determine when it was not able to make a confident prediction, and for those patients, it didn't even try to predict their diagnosis.  Her algorithm was 99% accurate on the patients where it made a prediction -- so that extension seemed to help quite a bit.


{ucb-page}`Accuracy_of_the_Classifier`
