---
jupyter:
  jupytext:
    split_at_heading: true
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.1'
      jupytext_version: 1.2.4
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

# Numbers and strings

This page covers a common problem when loading data into Pandas --- when Pandas gets confused about whether values in a column are text or numbers.

## An example

```{python}
import numpy as np
import pandas as pd
pd.set_option('mode.copy_on_write', True)
```

We return to the example data file that you may have seen in the [text encoding](text_encoding) page.

You can download the data file from {download}`imdblet_latin.csv
<../data/imdblet_latin.csv>`.

```{python}
films = pd.read_csv('imdblet_latin.csv', encoding='latin1')
films.head()
```

Now imagine we are interested in the average rating across these films:

```{python tags=c("raises-exception")}
ratings = films['Rating']
ratings.mean()
```

## The problem

The problem is that we were expecting our ratings to be numbers, but in fact, they are strings.

We can see what type of thing Pandas has stored by looking at the `dtype`
attribute of a Series, or the `dtypes` attribute of a data frame.

```{python}
films.dtypes
```

```{python}
ratings.dtype
```

In fact both these bits of information say the same thing -- that the 'Rating'
column stores things in the "object" or "O" type.  This is a general type that
can store any Python value.   It is the standard type that Pandas uses when
storing text.

Why does Pandas use text for the 'Rating' column?

A quick look at the first rows gives the answer:

```{python}
ratings.head()
```

The film "Paris, Texas (1984)" has a value "N/K" for the rating. This can't be a number, so Pandas stored this column in a format that allows it to store "N/K" as text.

If that wasn't obvious, another way of checking where the problem value is, to `apply` the function `float` to the column values.

When we `apply` a function to a Series, it does this:

* For each value in the Series it:
  * Calls the function, with the value as the single argument.
  * Collects the new value returned from the function, and appends it to a new
    Series.
* Returns the new Series.

The result is a Series that is the same length as the original series, but
where each value in the new series is the result of calling the function on the
original value.

Recall that the `float` function converts the thing you pass into a floating
point value:

```{python}
v = float('3.14')
v
```

```{python}
type(v)
```

Now we try applying `float` to the problematic column:

```{python tags=c("raises-exception")}
ratings.apply(float)
```

One way of dealing with this problem is to make a *recoding* function.

A recoding function is a function that we will apply to a Series.  That means that we call the function for every value in the Series.  The function argument is the value from the series.  The function returns the new value, for a new Series.

```{python}
def recode_ratings(v):
    if v == 'N/K':  # Return missing value for 'N/K'
        return np.nan
    # Otherwise make text value into a float
    return float(v)
```

We test our function:

```{python}
recode_ratings('8.3')
```

```{python}
recode_ratings('N/K')
```

We make a new Series by calling the recode function:

```{python}
new_ratings = ratings.apply(recode_ratings)
new_ratings.head()
```

We can insert this back into a copy of the original data frame:

```{python}
films_fixed = films
films_fixed.loc[:, 'Rating'] = new_ratings
films_fixed.head()
```
