---
jupyter:
  jupyterbook:
    show_stderr: true
  jupytext:
    formats: ''
    notebook_metadata_filter: all,-language_info
    split_at_heading: true
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.2'
      jupytext_version: 1.14.1
  kernelspec:
    display_name: Python 3 (ipykernel)
    language: python
    name: python3
---

# Sorting, heads and tails

```{python}
# Load the Pandas data science library, call it 'pd'
import pandas as pd
# Turn on a setting to use Pandas more safely.
# We will discuss this setting later.
pd.set_option('mode.copy_on_write', True)

# Load the library for plotting, name it 'plt'
import matplotlib.pyplot as plt
# Make plots look a little more fancy
plt.style.use('fivethirtyeight')
```

In [basic column indexing](df_basic_col_indexing.Rmd) we found the slightly odd
result that, for rich countries, there is little relationship between GDP and
maternal mortality.

Here we investigate further, by looking at rich and poor countries separately.

In order to do that, we are going to *sort* our DataFrame using the
`.sort_values` method, and the select the first and last group of rows, using
the `.head` and `.tail` methods.

First let us return to the slightly processed DataFrame we were working on before:

```{python}
# Original data frame before dropping missing values.
gender_data = pd.read_csv('gender_stats_min.csv')
gender_data_no_na = gender_data.dropna()
labeled_gdata = gender_data_no_na.set_index('country_code')
labeled_gdata
```

Here is the plot we saw before, with the unconvincing relationship of GDP to
Maternal Mortality Rate (MMR).

```{python}
plt.scatter(labeled_gdata['gdp_us_billion'],
            labeled_gdata['mat_mort_ratio'])
plt.title('MMR as a function of GDP')
```

We wondered whether the relationship of GDP and MMR might be different for rich and poor countries.

To look at that, we can sort the DataFrame by the GDP values.

In order to do that, we use the `.sort_values` method, passing the column name containing the values we want to sort by:

```{python}
gdata_by_gdp = labeled_gdata.sort_values('gdp_us_billion')
gdata_by_gdp
```

Notice that the `.sort_values` method returned a new data frame with the rows
in ascending order of the values in the given column (here `gdp_us_billion`).
We therefore have a DataFrame where the richest countries are first and the
poorest last.

Ascending order is the default sort order, but you can ask for descending order by giving the `ascending` keyword argument a value of `False`, like this:

```{python}
gdata_by_desc_gdp = labeled_gdata.sort_values('gdp_us_billion',
                                               ascending=False)
gdata_by_desc_gdp
```

Notice that now the richest countries are first and the poorest last.

Let us go back to the poorest to richest sorted DataFrame, `gdata_by_gdp`.  The DataFrame has a `.head` method that, by default, will select the first 5 rows of the DataFrame:

```{python}
gdata_by_gdp.head()
```

Notice that the result is a new DataFrame that only has 5 rows.

In fact we often use `.head` to show a small sample of the DataFrame, and you will see that use throughout the rest of the course.

You can also give `.head` a number of rows you want.  For example to select the
125 poorest countries (in terms of GDP), you could use:

```{python}
poorest_125 = gdata_by_gdp.head(125)
poorest_125
```

Now we have the rows corresponding to the 125 poorest countries, we can repeat our GDP / MMR plot, restricted to those countries:

```{python}
plt.scatter(poorest_125['gdp_us_billion'], poorest_125['mat_mort_ratio'])
plt.title('MMR as a function of GDP, for 125 poorest countries')
```

If we sort the new DataFrame by the MMR values, we can see which of these 125
poorest countries are doing particularly well or badly in terms of MMR:

```{python}
poorest_125.sort_values('mat_mort_ratio')
```

DataFrames also have `.tail` method that, by default, gives the last 5 rows of the DataFrame.  For example, these are the 5 richest countries:

```{python}
gdata_by_gdp.tail()
```

Like `.head` we can give `.tail` a number of rows we want.  Here we are looking at the last 25 rows of the sorted DataFrame, and therefore, the 25 richest countries:

```{python}
richest_25 = gdata_by_gdp.tail(25)
richest_25
```

```{python}
plt.scatter(richest_25['gdp_us_billion'], richest_25['mat_mort_ratio'])
plt.title('MMR as a function of GDP, for 25 richest countries')
```

Again, we can sort by the MMR values to show the best and worst countries in terms of maternal health:

```{python}
richest_25.sort_values('mat_mort_ratio')
```

To investigate further, we need to do some calculations to adjust for the
population.
