Sorting, heads and tails#
# Load the Pandas data science library, call it 'pd'
import pandas as pd
# Turn on a setting to use Pandas more safely.
# We will discuss this setting later.
pd.set_option('mode.copy_on_write', True)
# Load the library for plotting, name it 'plt'
import matplotlib.pyplot as plt
# Make plots look a little more fancy
plt.style.use('fivethirtyeight')
In basic column indexing we found the slightly odd result that, for rich countries, there is little relationship between GDP and maternal mortality.
Here we investigate further, by looking at rich and poor countries separately.
In order to do that, we are going to sort our DataFrame using the
.sort_values method, and the select the first and last group of rows, using
the .head and .tail methods.
First let us return to the slightly processed DataFrame we were working on before:
# Original data frame before dropping missing values.
gender_data = pd.read_csv('gender_stats_min.csv')
gender_data_no_na = gender_data.dropna()
labeled_gdata = gender_data_no_na.set_index('country_code')
labeled_gdata
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| AFG | Afghanistan | 19.961015 | 444.00 | 32.715838 |
| AGO | Angola | 111.936542 | 501.25 | 26.937545 |
| ALB | Albania | 12.327586 | 29.25 | 2.888280 |
| ARE | United Arab Emirates | 375.027082 | 6.00 | 9.080299 |
| ARG | Argentina | 550.980968 | 53.75 | 42.976675 |
| ... | ... | ... | ... | ... |
| WSM | Samoa | 0.799887 | 54.75 | 0.192225 |
| YEM | Yemen, Rep. | 36.819337 | 399.75 | 26.246608 |
| ZAF | South Africa | 345.209888 | 143.75 | 54.177209 |
| ZMB | Zambia | 24.280990 | 233.75 | 15.633220 |
| ZWE | Zimbabwe | 15.495514 | 398.00 | 15.420964 |
179 rows × 4 columns
Here is the plot we saw before, with the unconvincing relationship of GDP to Maternal Mortality Rate (MMR).
plt.scatter(labeled_gdata['gdp_us_billion'],
labeled_gdata['mat_mort_ratio'])
plt.title('MMR as a function of GDP')
Text(0.5, 1.0, 'MMR as a function of GDP')
We wondered whether the relationship of GDP and MMR might be different for rich and poor countries.
To look at that, we can sort the DataFrame by the GDP values.
In order to do that, we use the .sort_values method, passing the column name containing the values we want to sort by:
gdata_by_gdp = labeled_gdata.sort_values('gdp_us_billion')
gdata_by_gdp
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| KIR | Kiribati | 0.177431 | 95.00 | 0.110482 |
| STP | Sao Tome and Principe | 0.314540 | 159.50 | 0.191333 |
| FSM | Micronesia, Fed. Sts. | 0.319321 | 103.25 | 0.104118 |
| TON | Tonga | 0.439179 | 129.25 | 0.105909 |
| COM | Comoros | 0.603919 | 349.50 | 0.759556 |
| ... | ... | ... | ... | ... |
| GBR | United Kingdom | 2768.864417 | 9.25 | 64.641557 |
| DEU | Germany | 3601.226158 | 6.25 | 81.281645 |
| JPN | Japan | 5106.024760 | 5.75 | 127.297102 |
| CHN | China | 10182.790479 | 28.75 | 1364.446000 |
| USA | United States | 17369.124600 | 14.00 | 318.558175 |
179 rows × 4 columns
Notice that the .sort_values method returned a new data frame with the rows
in ascending order of the values in the given column (here gdp_us_billion).
We therefore have a DataFrame where the richest countries are first and the
poorest last.
Ascending order is the default sort order, but you can ask for descending order by giving the ascending keyword argument a value of False, like this:
gdata_by_desc_gdp = labeled_gdata.sort_values('gdp_us_billion',
ascending=False)
gdata_by_desc_gdp
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| USA | United States | 17369.124600 | 14.00 | 318.558175 |
| CHN | China | 10182.790479 | 28.75 | 1364.446000 |
| JPN | Japan | 5106.024760 | 5.75 | 127.297102 |
| DEU | Germany | 3601.226158 | 6.25 | 81.281645 |
| GBR | United Kingdom | 2768.864417 | 9.25 | 64.641557 |
| ... | ... | ... | ... | ... |
| COM | Comoros | 0.603919 | 349.50 | 0.759556 |
| TON | Tonga | 0.439179 | 129.25 | 0.105909 |
| FSM | Micronesia, Fed. Sts. | 0.319321 | 103.25 | 0.104118 |
| STP | Sao Tome and Principe | 0.314540 | 159.50 | 0.191333 |
| KIR | Kiribati | 0.177431 | 95.00 | 0.110482 |
179 rows × 4 columns
Notice that now the richest countries are first and the poorest last.
Let us go back to the poorest to richest sorted DataFrame, gdata_by_gdp. The DataFrame has a .head method that, by default, will select the first 5 rows of the DataFrame:
gdata_by_gdp.head()
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| KIR | Kiribati | 0.177431 | 95.00 | 0.110482 |
| STP | Sao Tome and Principe | 0.314540 | 159.50 | 0.191333 |
| FSM | Micronesia, Fed. Sts. | 0.319321 | 103.25 | 0.104118 |
| TON | Tonga | 0.439179 | 129.25 | 0.105909 |
| COM | Comoros | 0.603919 | 349.50 | 0.759556 |
Notice that the result is a new DataFrame that only has 5 rows.
In fact we often use .head to show a small sample of the DataFrame, and you will see that use throughout the rest of the course.
You can also give .head a number of rows you want. For example to select the
125 poorest countries (in terms of GDP), you could use:
poorest_125 = gdata_by_gdp.head(125)
poorest_125
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| KIR | Kiribati | 0.177431 | 95.00 | 0.110482 |
| STP | Sao Tome and Principe | 0.314540 | 159.50 | 0.191333 |
| FSM | Micronesia, Fed. Sts. | 0.319321 | 103.25 | 0.104118 |
| TON | Tonga | 0.439179 | 129.25 | 0.105909 |
| COM | Comoros | 0.603919 | 349.50 | 0.759556 |
| ... | ... | ... | ... | ... |
| AGO | Angola | 111.936542 | 501.25 | 26.937545 |
| HUN | Hungary | 129.470864 | 16.25 | 9.868180 |
| UKR | Ukraine | 135.379275 | 24.25 | 45.302704 |
| KWT | Kuwait | 156.226123 | 4.00 | 3.752954 |
| BGD | Bangladesh | 174.545099 | 194.75 | 159.371214 |
125 rows × 4 columns
Now we have the rows corresponding to the 125 poorest countries, we can repeat our GDP / MMR plot, restricted to those countries:
plt.scatter(poorest_125['gdp_us_billion'], poorest_125['mat_mort_ratio'])
plt.title('MMR as a function of GDP, for 125 poorest countries')
Text(0.5, 1.0, 'MMR as a function of GDP, for 125 poorest countries')
If we sort the new DataFrame by the MMR values, we can see which of these 125 poorest countries are doing particularly well or badly in terms of MMR:
poorest_125.sort_values('mat_mort_ratio')
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| ISL | Iceland | 16.741585 | 3.50 | 0.327387 |
| BLR | Belarus | 64.782942 | 4.00 | 9.480348 |
| KWT | Kuwait | 156.226123 | 4.00 | 3.752954 |
| SVK | Slovak Republic | 93.894473 | 6.00 | 5.418425 |
| CYP | Cyprus | 22.347398 | 7.00 | 1.152475 |
| ... | ... | ... | ... | ... |
| SOM | Somalia | 5.785250 | 762.75 | 13.527075 |
| SSD | South Sudan | 11.480939 | 827.50 | 11.527917 |
| CAF | Central African Republic | 1.749110 | 875.75 | 4.529236 |
| TCD | Chad | 11.945942 | 892.25 | 13.574024 |
| SLE | Sierra Leone | 4.331604 | 1435.00 | 7.080112 |
125 rows × 4 columns
DataFrames also have .tail method that, by default, gives the last 5 rows of the DataFrame. For example, these are the 5 richest countries:
gdata_by_gdp.tail()
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| GBR | United Kingdom | 2768.864417 | 9.25 | 64.641557 |
| DEU | Germany | 3601.226158 | 6.25 | 81.281645 |
| JPN | Japan | 5106.024760 | 5.75 | 127.297102 |
| CHN | China | 10182.790479 | 28.75 | 1364.446000 |
| USA | United States | 17369.124600 | 14.00 | 318.558175 |
Like .head we can give .tail a number of rows we want. Here we are looking at the last 25 rows of the sorted DataFrame, and therefore, the 25 richest countries:
richest_25 = gdata_by_gdp.tail(25)
richest_25
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| NGA | Nigeria | 486.113579 | 818.50 | 176.551695 |
| BEL | Belgium | 494.221836 | 7.00 | 11.228495 |
| POL | Poland | 503.311262 | 3.00 | 38.009905 |
| SWE | Sweden | 540.626904 | 4.00 | 9.703634 |
| ARG | Argentina | 550.980968 | 53.75 | 42.976675 |
| CHE | Switzerland | 676.642359 | 5.25 | 8.185870 |
| SAU | Saudi Arabia | 707.936120 | 12.25 | 30.728077 |
| NLD | Netherlands | 819.285000 | 7.00 | 16.876547 |
| TUR | Turkey | 895.175577 | 17.50 | 77.034345 |
| IDN | Indonesia | 902.944866 | 136.75 | 255.064836 |
| MEX | Mexico | 1188.802780 | 40.00 | 124.203450 |
| ESP | Spain | 1299.724261 | 5.00 | 46.553128 |
| KOR | Korea, Rep. | 1346.751162 | 12.00 | 50.727212 |
| AUS | Australia | 1422.994116 | 6.00 | 23.444560 |
| CAN | Canada | 1708.473627 | 7.25 | 35.517119 |
| RUS | Russian Federation | 1822.691700 | 25.25 | 143.793504 |
| ITA | Italy | 2005.983980 | 4.00 | 60.378795 |
| IND | India | 2019.005411 | 185.25 | 1293.742537 |
| BRA | Brazil | 2198.765606 | 49.50 | 204.159544 |
| FRA | France | 2647.649725 | 8.75 | 66.302099 |
| GBR | United Kingdom | 2768.864417 | 9.25 | 64.641557 |
| DEU | Germany | 3601.226158 | 6.25 | 81.281645 |
| JPN | Japan | 5106.024760 | 5.75 | 127.297102 |
| CHN | China | 10182.790479 | 28.75 | 1364.446000 |
| USA | United States | 17369.124600 | 14.00 | 318.558175 |
plt.scatter(richest_25['gdp_us_billion'], richest_25['mat_mort_ratio'])
plt.title('MMR as a function of GDP, for 25 richest countries')
Text(0.5, 1.0, 'MMR as a function of GDP, for 25 richest countries')
Again, we can sort by the MMR values to show the best and worst countries in terms of maternal health:
richest_25.sort_values('mat_mort_ratio')
| country_name | gdp_us_billion | mat_mort_ratio | population | |
|---|---|---|---|---|
| country_code | ||||
| POL | Poland | 503.311262 | 3.00 | 38.009905 |
| SWE | Sweden | 540.626904 | 4.00 | 9.703634 |
| ITA | Italy | 2005.983980 | 4.00 | 60.378795 |
| ESP | Spain | 1299.724261 | 5.00 | 46.553128 |
| CHE | Switzerland | 676.642359 | 5.25 | 8.185870 |
| JPN | Japan | 5106.024760 | 5.75 | 127.297102 |
| AUS | Australia | 1422.994116 | 6.00 | 23.444560 |
| DEU | Germany | 3601.226158 | 6.25 | 81.281645 |
| BEL | Belgium | 494.221836 | 7.00 | 11.228495 |
| NLD | Netherlands | 819.285000 | 7.00 | 16.876547 |
| CAN | Canada | 1708.473627 | 7.25 | 35.517119 |
| FRA | France | 2647.649725 | 8.75 | 66.302099 |
| GBR | United Kingdom | 2768.864417 | 9.25 | 64.641557 |
| KOR | Korea, Rep. | 1346.751162 | 12.00 | 50.727212 |
| SAU | Saudi Arabia | 707.936120 | 12.25 | 30.728077 |
| USA | United States | 17369.124600 | 14.00 | 318.558175 |
| TUR | Turkey | 895.175577 | 17.50 | 77.034345 |
| RUS | Russian Federation | 1822.691700 | 25.25 | 143.793504 |
| CHN | China | 10182.790479 | 28.75 | 1364.446000 |
| MEX | Mexico | 1188.802780 | 40.00 | 124.203450 |
| BRA | Brazil | 2198.765606 | 49.50 | 204.159544 |
| ARG | Argentina | 550.980968 | 53.75 | 42.976675 |
| IDN | Indonesia | 902.944866 | 136.75 | 255.064836 |
| IND | India | 2019.005411 | 185.25 | 1293.742537 |
| NGA | Nigeria | 486.113579 | 818.50 | 176.551695 |
To investigate further, we need to do some calculations to adjust for the population.