{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Bootstrap\n",
    "A data scientist is using the data in a random sample to estimate an unknown parameter. She uses the sample to calculate the value of a statistic that she will use as her estimate. \n",
    "\n",
    "Once she has calculated the observed value of her statistic, she could just present it as her estimate and go on her merry way. But she's a data scientist. She knows that her random sample is just one of numerous possible random samples, and thus her estimate is just one of numerous plausible estimates. \n",
    "\n",
    "By how much could those estimates vary? To answer this, it appears as though she needs to draw another sample from the population, and compute a new estimate based on the new sample. But she doesn't have the resources to go back to the population and draw another sample.\n",
    "\n",
    "It looks as though the data scientist is stuck.\n",
    "\n",
    "Fortunately, a brilliant idea called *the bootstrap* can help her out. Since it is not feasible to generate new samples from the population, the bootstrap generates new random samples by a method called *resampling*: the new samples are drawn at random *from the original sample*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will see how and why the bootstrap works. In the rest of the chapter, we will use the bootstrap for inference.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Employee Compensation in the City of San Francisco\n",
    "[SF OpenData](https://data.sfgov.org) is a website where the City and County of San Francisco make some of their data publicly available. One of the data sets contains compensation data for employees of the City. These include medical professionals at City-run hospitals, police officers, fire fighters, transportation workers, elected officials, and all other employees of the City. \n",
    "\n",
    "Compensation data for the calendar year 2015 are in the table `sf2015`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "sf2015 = pd.read_csv(path_data + 'san_francisco_2015.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year Type</th>\n",
       "      <th>Year</th>\n",
       "      <th>Organization Group Code</th>\n",
       "      <th>Organization Group</th>\n",
       "      <th>Department Code</th>\n",
       "      <th>Department</th>\n",
       "      <th>Union Code</th>\n",
       "      <th>Union</th>\n",
       "      <th>Job Family Code</th>\n",
       "      <th>Job Family</th>\n",
       "      <th>...</th>\n",
       "      <th>Employee Identifier</th>\n",
       "      <th>Salaries</th>\n",
       "      <th>Overtime</th>\n",
       "      <th>Other Salaries</th>\n",
       "      <th>Total Salary</th>\n",
       "      <th>Retirement</th>\n",
       "      <th>Health/Dental</th>\n",
       "      <th>Other Benefits</th>\n",
       "      <th>Total Benefits</th>\n",
       "      <th>Total Compensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>2</td>\n",
       "      <td>Public Works, Transportation &amp; Commerce</td>\n",
       "      <td>WTR</td>\n",
       "      <td>PUC Water Department</td>\n",
       "      <td>21.0</td>\n",
       "      <td>Prof &amp; Tech Engineers - Miscellaneous, Local 21</td>\n",
       "      <td>2400</td>\n",
       "      <td>Lab, Pharmacy &amp; Med Techs</td>\n",
       "      <td>...</td>\n",
       "      <td>21538</td>\n",
       "      <td>82146.04</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>82146.04</td>\n",
       "      <td>16942.21</td>\n",
       "      <td>12340.88</td>\n",
       "      <td>6337.73</td>\n",
       "      <td>35620.82</td>\n",
       "      <td>117766.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>2</td>\n",
       "      <td>Public Works, Transportation &amp; Commerce</td>\n",
       "      <td>DPW</td>\n",
       "      <td>General Services Agency - Public Works</td>\n",
       "      <td>12.0</td>\n",
       "      <td>Carpet, Linoleum and Soft Tile Workers, Local 12</td>\n",
       "      <td>7300</td>\n",
       "      <td>Journeyman Trade</td>\n",
       "      <td>...</td>\n",
       "      <td>5459</td>\n",
       "      <td>32165.75</td>\n",
       "      <td>973.19</td>\n",
       "      <td>848.96</td>\n",
       "      <td>33987.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4587.51</td>\n",
       "      <td>2634.42</td>\n",
       "      <td>7221.93</td>\n",
       "      <td>41209.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>790.0</td>\n",
       "      <td>SEIU - Miscellaneous, Local 1021</td>\n",
       "      <td>1600</td>\n",
       "      <td>Payroll, Billing &amp; Accounting</td>\n",
       "      <td>...</td>\n",
       "      <td>41541</td>\n",
       "      <td>71311.00</td>\n",
       "      <td>5757.98</td>\n",
       "      <td>0.00</td>\n",
       "      <td>77068.98</td>\n",
       "      <td>14697.59</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>6370.06</td>\n",
       "      <td>33492.15</td>\n",
       "      <td>110561.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>351.0</td>\n",
       "      <td>Municipal Executive Association - Miscellaneous</td>\n",
       "      <td>0900</td>\n",
       "      <td>Management</td>\n",
       "      <td>...</td>\n",
       "      <td>26718</td>\n",
       "      <td>28430.25</td>\n",
       "      <td>0.00</td>\n",
       "      <td>763.07</td>\n",
       "      <td>29193.32</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4223.14</td>\n",
       "      <td>5208.51</td>\n",
       "      <td>9431.65</td>\n",
       "      <td>38624.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>2</td>\n",
       "      <td>Public Works, Transportation &amp; Commerce</td>\n",
       "      <td>MTA</td>\n",
       "      <td>Municipal Transportation Agency</td>\n",
       "      <td>790.0</td>\n",
       "      <td>SEIU - Miscellaneous, Local 1021</td>\n",
       "      <td>8200</td>\n",
       "      <td>Protection &amp; Apprehension</td>\n",
       "      <td>...</td>\n",
       "      <td>45810</td>\n",
       "      <td>7948.75</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7948.75</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2873.17</td>\n",
       "      <td>616.24</td>\n",
       "      <td>3489.41</td>\n",
       "      <td>11438.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>Public Protection</td>\n",
       "      <td>POL</td>\n",
       "      <td>Police</td>\n",
       "      <td>911.0</td>\n",
       "      <td>Police Officers' Association</td>\n",
       "      <td>Q000</td>\n",
       "      <td>Police Services</td>\n",
       "      <td>...</td>\n",
       "      <td>32906</td>\n",
       "      <td>2235.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2235.00</td>\n",
       "      <td>490.36</td>\n",
       "      <td>286.72</td>\n",
       "      <td>176.57</td>\n",
       "      <td>953.65</td>\n",
       "      <td>3188.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>791.0</td>\n",
       "      <td>SEIU - Staff and Per Diem Nurses, Local 1021</td>\n",
       "      <td>2300</td>\n",
       "      <td>Nursing</td>\n",
       "      <td>...</td>\n",
       "      <td>7506</td>\n",
       "      <td>187247.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>11704.06</td>\n",
       "      <td>198951.06</td>\n",
       "      <td>37683.66</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>11221.73</td>\n",
       "      <td>61329.89</td>\n",
       "      <td>260280.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>2</td>\n",
       "      <td>Public Works, Transportation &amp; Commerce</td>\n",
       "      <td>MTA</td>\n",
       "      <td>Municipal Transportation Agency</td>\n",
       "      <td>253.0</td>\n",
       "      <td>Transport Workers - Transit Operators, Local 2...</td>\n",
       "      <td>9100</td>\n",
       "      <td>Street Transit</td>\n",
       "      <td>...</td>\n",
       "      <td>36773</td>\n",
       "      <td>66988.54</td>\n",
       "      <td>3512.88</td>\n",
       "      <td>2770.39</td>\n",
       "      <td>73271.81</td>\n",
       "      <td>19127.22</td>\n",
       "      <td>13202.95</td>\n",
       "      <td>5455.10</td>\n",
       "      <td>37785.27</td>\n",
       "      <td>111057.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>CAT</td>\n",
       "      <td>City Attorney</td>\n",
       "      <td>311.0</td>\n",
       "      <td>Municipal Attorneys' Association</td>\n",
       "      <td>8100</td>\n",
       "      <td>Legal &amp; Court</td>\n",
       "      <td>...</td>\n",
       "      <td>12963</td>\n",
       "      <td>135189.60</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1562.50</td>\n",
       "      <td>136752.10</td>\n",
       "      <td>27501.81</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>10102.97</td>\n",
       "      <td>50029.28</td>\n",
       "      <td>186781.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>3</td>\n",
       "      <td>Human Welfare &amp; Neighborhood Development</td>\n",
       "      <td>DSS</td>\n",
       "      <td>Human Services</td>\n",
       "      <td>535.0</td>\n",
       "      <td>SEIU - Human Services, Local 1021</td>\n",
       "      <td>9700</td>\n",
       "      <td>Community Development</td>\n",
       "      <td>...</td>\n",
       "      <td>35179</td>\n",
       "      <td>70474.77</td>\n",
       "      <td>147.28</td>\n",
       "      <td>1647.24</td>\n",
       "      <td>72269.29</td>\n",
       "      <td>14650.28</td>\n",
       "      <td>10696.90</td>\n",
       "      <td>5993.11</td>\n",
       "      <td>31340.29</td>\n",
       "      <td>103609.58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Year Type  Year  Organization Group Code  \\\n",
       "0  Calendar  2015                        2   \n",
       "1  Calendar  2015                        2   \n",
       "2  Calendar  2015                        4   \n",
       "3  Calendar  2015                        4   \n",
       "4  Calendar  2015                        2   \n",
       "5  Calendar  2015                        1   \n",
       "6  Calendar  2015                        4   \n",
       "7  Calendar  2015                        2   \n",
       "8  Calendar  2015                        6   \n",
       "9  Calendar  2015                        3   \n",
       "\n",
       "                         Organization Group Department Code  \\\n",
       "0   Public Works, Transportation & Commerce             WTR   \n",
       "1   Public Works, Transportation & Commerce             DPW   \n",
       "2                          Community Health             DPH   \n",
       "3                          Community Health             DPH   \n",
       "4   Public Works, Transportation & Commerce             MTA   \n",
       "5                         Public Protection             POL   \n",
       "6                          Community Health             DPH   \n",
       "7   Public Works, Transportation & Commerce             MTA   \n",
       "8          General Administration & Finance             CAT   \n",
       "9  Human Welfare & Neighborhood Development             DSS   \n",
       "\n",
       "                               Department  Union Code  \\\n",
       "0                    PUC Water Department        21.0   \n",
       "1  General Services Agency - Public Works        12.0   \n",
       "2                           Public Health       790.0   \n",
       "3                           Public Health       351.0   \n",
       "4         Municipal Transportation Agency       790.0   \n",
       "5                                  Police       911.0   \n",
       "6                           Public Health       791.0   \n",
       "7         Municipal Transportation Agency       253.0   \n",
       "8                           City Attorney       311.0   \n",
       "9                          Human Services       535.0   \n",
       "\n",
       "                                               Union Job Family Code  \\\n",
       "0    Prof & Tech Engineers - Miscellaneous, Local 21            2400   \n",
       "1   Carpet, Linoleum and Soft Tile Workers, Local 12            7300   \n",
       "2                   SEIU - Miscellaneous, Local 1021            1600   \n",
       "3    Municipal Executive Association - Miscellaneous            0900   \n",
       "4                   SEIU - Miscellaneous, Local 1021            8200   \n",
       "5                       Police Officers' Association            Q000   \n",
       "6       SEIU - Staff and Per Diem Nurses, Local 1021            2300   \n",
       "7  Transport Workers - Transit Operators, Local 2...            9100   \n",
       "8                   Municipal Attorneys' Association            8100   \n",
       "9                  SEIU - Human Services, Local 1021            9700   \n",
       "\n",
       "                      Job Family  ... Employee Identifier   Salaries  \\\n",
       "0      Lab, Pharmacy & Med Techs  ...               21538   82146.04   \n",
       "1               Journeyman Trade  ...                5459   32165.75   \n",
       "2  Payroll, Billing & Accounting  ...               41541   71311.00   \n",
       "3                     Management  ...               26718   28430.25   \n",
       "4      Protection & Apprehension  ...               45810    7948.75   \n",
       "5                Police Services  ...               32906    2235.00   \n",
       "6                        Nursing  ...                7506  187247.00   \n",
       "7                 Street Transit  ...               36773   66988.54   \n",
       "8                  Legal & Court  ...               12963  135189.60   \n",
       "9          Community Development  ...               35179   70474.77   \n",
       "\n",
       "   Overtime  Other Salaries  Total Salary  Retirement  Health/Dental  \\\n",
       "0      0.00            0.00      82146.04    16942.21       12340.88   \n",
       "1    973.19          848.96      33987.90        0.00        4587.51   \n",
       "2   5757.98            0.00      77068.98    14697.59       12424.50   \n",
       "3      0.00          763.07      29193.32        0.00        4223.14   \n",
       "4      0.00            0.00       7948.75        0.00        2873.17   \n",
       "5      0.00            0.00       2235.00      490.36         286.72   \n",
       "6      0.00        11704.06     198951.06    37683.66       12424.50   \n",
       "7   3512.88         2770.39      73271.81    19127.22       13202.95   \n",
       "8      0.00         1562.50     136752.10    27501.81       12424.50   \n",
       "9    147.28         1647.24      72269.29    14650.28       10696.90   \n",
       "\n",
       "   Other Benefits  Total Benefits  Total Compensation  \n",
       "0         6337.73        35620.82           117766.86  \n",
       "1         2634.42         7221.93            41209.83  \n",
       "2         6370.06        33492.15           110561.13  \n",
       "3         5208.51         9431.65            38624.97  \n",
       "4          616.24         3489.41            11438.16  \n",
       "5          176.57          953.65             3188.65  \n",
       "6        11221.73        61329.89           260280.95  \n",
       "7         5455.10        37785.27           111057.08  \n",
       "8        10102.97        50029.28           186781.38  \n",
       "9         5993.11        31340.29           103609.58  \n",
       "\n",
       "[10 rows x 22 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sf2015.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one row for each of 42,979 employees. There are numerous columns containing information about City departmental affiliation and details of the different parts of the employee's compensation package. Here is the row correspoding to the late Edward Lee, the Mayor at that time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year Type</th>\n",
       "      <th>Year</th>\n",
       "      <th>Organization Group Code</th>\n",
       "      <th>Organization Group</th>\n",
       "      <th>Department Code</th>\n",
       "      <th>Department</th>\n",
       "      <th>Union Code</th>\n",
       "      <th>Union</th>\n",
       "      <th>Job Family Code</th>\n",
       "      <th>Job Family</th>\n",
       "      <th>...</th>\n",
       "      <th>Employee Identifier</th>\n",
       "      <th>Salaries</th>\n",
       "      <th>Overtime</th>\n",
       "      <th>Other Salaries</th>\n",
       "      <th>Total Salary</th>\n",
       "      <th>Retirement</th>\n",
       "      <th>Health/Dental</th>\n",
       "      <th>Other Benefits</th>\n",
       "      <th>Total Benefits</th>\n",
       "      <th>Total Compensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3335</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>MYR</td>\n",
       "      <td>Mayor</td>\n",
       "      <td>556.0</td>\n",
       "      <td>Elected Officials</td>\n",
       "      <td>1100</td>\n",
       "      <td>Administrative &amp; Mgmt (Unrep)</td>\n",
       "      <td>...</td>\n",
       "      <td>22433</td>\n",
       "      <td>288963.55</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>288963.55</td>\n",
       "      <td>58117.03</td>\n",
       "      <td>12424.5</td>\n",
       "      <td>20292.95</td>\n",
       "      <td>90834.48</td>\n",
       "      <td>379798.03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Year Type  Year  Organization Group Code  \\\n",
       "3335  Calendar  2015                        6   \n",
       "\n",
       "                    Organization Group Department Code Department  Union Code  \\\n",
       "3335  General Administration & Finance             MYR      Mayor       556.0   \n",
       "\n",
       "                  Union Job Family Code                     Job Family  ...  \\\n",
       "3335  Elected Officials            1100  Administrative & Mgmt (Unrep)  ...   \n",
       "\n",
       "     Employee Identifier   Salaries  Overtime  Other Salaries  Total Salary  \\\n",
       "3335               22433  288963.55       0.0             0.0     288963.55   \n",
       "\n",
       "      Retirement  Health/Dental  Other Benefits  Total Benefits  \\\n",
       "3335    58117.03        12424.5        20292.95        90834.48   \n",
       "\n",
       "      Total Compensation  \n",
       "3335           379798.03  \n",
       "\n",
       "[1 rows x 22 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sf2015[sf2015['Job'] == 'Mayor']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to study the final column, `Total Compensation`. That's the employee's salary plus the City's contribution towards his/her retirement and benefit plans.\n",
    "\n",
    "Financial packages in a calendar year can sometimes be hard to understand as they depend on the date of hire, whether the employee is changing jobs within the City, and so on. For example, the lowest values in the `Total Compensation` column look a little strange."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year Type</th>\n",
       "      <th>Year</th>\n",
       "      <th>Organization Group Code</th>\n",
       "      <th>Organization Group</th>\n",
       "      <th>Department Code</th>\n",
       "      <th>Department</th>\n",
       "      <th>Union Code</th>\n",
       "      <th>Union</th>\n",
       "      <th>Job Family Code</th>\n",
       "      <th>Job Family</th>\n",
       "      <th>...</th>\n",
       "      <th>Employee Identifier</th>\n",
       "      <th>Salaries</th>\n",
       "      <th>Overtime</th>\n",
       "      <th>Other Salaries</th>\n",
       "      <th>Total Salary</th>\n",
       "      <th>Retirement</th>\n",
       "      <th>Health/Dental</th>\n",
       "      <th>Other Benefits</th>\n",
       "      <th>Total Benefits</th>\n",
       "      <th>Total Compensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>27308</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>Public Protection</td>\n",
       "      <td>FIR</td>\n",
       "      <td>Fire Department</td>\n",
       "      <td>798.0</td>\n",
       "      <td>Firefighters - Miscellaneous, Local 798</td>\n",
       "      <td>H000</td>\n",
       "      <td>Fire Services</td>\n",
       "      <td>...</td>\n",
       "      <td>43833</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-423.76</td>\n",
       "      <td>-423.76</td>\n",
       "      <td>-423.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15746</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>790.0</td>\n",
       "      <td>SEIU - Miscellaneous, Local 1021</td>\n",
       "      <td>9900</td>\n",
       "      <td>Public Service Aide</td>\n",
       "      <td>...</td>\n",
       "      <td>27871</td>\n",
       "      <td>-292.40</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-292.40</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-95.58</td>\n",
       "      <td>-22.63</td>\n",
       "      <td>-118.21</td>\n",
       "      <td>-410.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24576</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>Public Protection</td>\n",
       "      <td>JUV</td>\n",
       "      <td>Juvenile Probation</td>\n",
       "      <td>790.0</td>\n",
       "      <td>SEIU - Miscellaneous, Local 1021</td>\n",
       "      <td>8300</td>\n",
       "      <td>Correction &amp; Detention</td>\n",
       "      <td>...</td>\n",
       "      <td>10517</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-159.12</td>\n",
       "      <td>-159.12</td>\n",
       "      <td>-159.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42982</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>CPC</td>\n",
       "      <td>City Planning</td>\n",
       "      <td>21.0</td>\n",
       "      <td>Prof &amp; Tech Engineers - Miscellaneous, Local 21</td>\n",
       "      <td>1000</td>\n",
       "      <td>Information Systems</td>\n",
       "      <td>...</td>\n",
       "      <td>18961</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-26.53</td>\n",
       "      <td>-26.53</td>\n",
       "      <td>-26.53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23310</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>CPC</td>\n",
       "      <td>City Planning</td>\n",
       "      <td>21.0</td>\n",
       "      <td>Prof &amp; Tech Engineers - Miscellaneous, Local 21</td>\n",
       "      <td>5200</td>\n",
       "      <td>Professional Engineering</td>\n",
       "      <td>...</td>\n",
       "      <td>19387</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-9.51</td>\n",
       "      <td>-9.51</td>\n",
       "      <td>-9.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5171</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>351.0</td>\n",
       "      <td>Municipal Executive Association - Miscellaneous</td>\n",
       "      <td>0900</td>\n",
       "      <td>Management</td>\n",
       "      <td>...</td>\n",
       "      <td>1523</td>\n",
       "      <td>256098.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>82292.31</td>\n",
       "      <td>338390.32</td>\n",
       "      <td>51977.53</td>\n",
       "      <td>11468.77</td>\n",
       "      <td>20963.32</td>\n",
       "      <td>84409.62</td>\n",
       "      <td>422799.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17805</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>2</td>\n",
       "      <td>Public Works, Transportation &amp; Commerce</td>\n",
       "      <td>AIR</td>\n",
       "      <td>Airport Commission</td>\n",
       "      <td>351.0</td>\n",
       "      <td>Municipal Executive Association - Miscellaneous</td>\n",
       "      <td>0900</td>\n",
       "      <td>Management</td>\n",
       "      <td>...</td>\n",
       "      <td>17356</td>\n",
       "      <td>326764.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>326764.01</td>\n",
       "      <td>65806.33</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>21691.23</td>\n",
       "      <td>99922.06</td>\n",
       "      <td>426686.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>ADM</td>\n",
       "      <td>General Services Agency - City Admin</td>\n",
       "      <td>164.0</td>\n",
       "      <td>Physicians and Dentists - Miscellaneous</td>\n",
       "      <td>2500</td>\n",
       "      <td>Med Therapy &amp; Auxiliary</td>\n",
       "      <td>...</td>\n",
       "      <td>13746</td>\n",
       "      <td>279311.03</td>\n",
       "      <td>9046.92</td>\n",
       "      <td>56742.56</td>\n",
       "      <td>345100.51</td>\n",
       "      <td>56211.61</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>13482.66</td>\n",
       "      <td>82118.77</td>\n",
       "      <td>427219.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13194</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>ADM</td>\n",
       "      <td>General Services Agency - City Admin</td>\n",
       "      <td>164.0</td>\n",
       "      <td>Physicians and Dentists - Miscellaneous</td>\n",
       "      <td>2500</td>\n",
       "      <td>Med Therapy &amp; Auxiliary</td>\n",
       "      <td>...</td>\n",
       "      <td>1016</td>\n",
       "      <td>279311.10</td>\n",
       "      <td>3829.36</td>\n",
       "      <td>114433.58</td>\n",
       "      <td>397574.04</td>\n",
       "      <td>56211.64</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>14299.10</td>\n",
       "      <td>82935.24</td>\n",
       "      <td>480509.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19177</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>RET</td>\n",
       "      <td>Retirement System</td>\n",
       "      <td>351.0</td>\n",
       "      <td>Municipal Executive Association - Miscellaneous</td>\n",
       "      <td>1100</td>\n",
       "      <td>Administrative &amp; Mgmt (Unrep)</td>\n",
       "      <td>...</td>\n",
       "      <td>46881</td>\n",
       "      <td>507831.60</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>507831.60</td>\n",
       "      <td>105052.98</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>23566.16</td>\n",
       "      <td>141043.64</td>\n",
       "      <td>648875.24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>42989 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Year Type  Year  Organization Group Code  \\\n",
       "27308  Calendar  2015                        1   \n",
       "15746  Calendar  2015                        4   \n",
       "24576  Calendar  2015                        1   \n",
       "42982  Calendar  2015                        6   \n",
       "23310  Calendar  2015                        6   \n",
       "...         ...   ...                      ...   \n",
       "5171   Calendar  2015                        4   \n",
       "17805  Calendar  2015                        2   \n",
       "499    Calendar  2015                        6   \n",
       "13194  Calendar  2015                        6   \n",
       "19177  Calendar  2015                        6   \n",
       "\n",
       "                            Organization Group Department Code  \\\n",
       "27308                        Public Protection             FIR   \n",
       "15746                         Community Health             DPH   \n",
       "24576                        Public Protection             JUV   \n",
       "42982         General Administration & Finance             CPC   \n",
       "23310         General Administration & Finance             CPC   \n",
       "...                                        ...             ...   \n",
       "5171                          Community Health             DPH   \n",
       "17805  Public Works, Transportation & Commerce             AIR   \n",
       "499           General Administration & Finance             ADM   \n",
       "13194         General Administration & Finance             ADM   \n",
       "19177         General Administration & Finance             RET   \n",
       "\n",
       "                                 Department  Union Code  \\\n",
       "27308                       Fire Department       798.0   \n",
       "15746                         Public Health       790.0   \n",
       "24576                    Juvenile Probation       790.0   \n",
       "42982                         City Planning        21.0   \n",
       "23310                         City Planning        21.0   \n",
       "...                                     ...         ...   \n",
       "5171                          Public Health       351.0   \n",
       "17805                    Airport Commission       351.0   \n",
       "499    General Services Agency - City Admin       164.0   \n",
       "13194  General Services Agency - City Admin       164.0   \n",
       "19177                     Retirement System       351.0   \n",
       "\n",
       "                                                 Union Job Family Code  \\\n",
       "27308          Firefighters - Miscellaneous, Local 798            H000   \n",
       "15746                 SEIU - Miscellaneous, Local 1021            9900   \n",
       "24576                 SEIU - Miscellaneous, Local 1021            8300   \n",
       "42982  Prof & Tech Engineers - Miscellaneous, Local 21            1000   \n",
       "23310  Prof & Tech Engineers - Miscellaneous, Local 21            5200   \n",
       "...                                                ...             ...   \n",
       "5171   Municipal Executive Association - Miscellaneous            0900   \n",
       "17805  Municipal Executive Association - Miscellaneous            0900   \n",
       "499            Physicians and Dentists - Miscellaneous            2500   \n",
       "13194          Physicians and Dentists - Miscellaneous            2500   \n",
       "19177  Municipal Executive Association - Miscellaneous            1100   \n",
       "\n",
       "                          Job Family  ... Employee Identifier   Salaries  \\\n",
       "27308                  Fire Services  ...               43833       0.00   \n",
       "15746            Public Service Aide  ...               27871    -292.40   \n",
       "24576         Correction & Detention  ...               10517       0.00   \n",
       "42982            Information Systems  ...               18961       0.00   \n",
       "23310       Professional Engineering  ...               19387       0.00   \n",
       "...                              ...  ...                 ...        ...   \n",
       "5171                      Management  ...                1523  256098.01   \n",
       "17805                     Management  ...               17356  326764.01   \n",
       "499          Med Therapy & Auxiliary  ...               13746  279311.03   \n",
       "13194        Med Therapy & Auxiliary  ...                1016  279311.10   \n",
       "19177  Administrative & Mgmt (Unrep)  ...               46881  507831.60   \n",
       "\n",
       "       Overtime  Other Salaries  Total Salary  Retirement  Health/Dental  \\\n",
       "27308      0.00            0.00          0.00        0.00           0.00   \n",
       "15746      0.00            0.00       -292.40        0.00         -95.58   \n",
       "24576      0.00            0.00          0.00        0.00           0.00   \n",
       "42982      0.00            0.00          0.00        0.00           0.00   \n",
       "23310      0.00            0.00          0.00        0.00           0.00   \n",
       "...         ...             ...           ...         ...            ...   \n",
       "5171       0.00        82292.31     338390.32    51977.53       11468.77   \n",
       "17805      0.00            0.00     326764.01    65806.33       12424.50   \n",
       "499     9046.92        56742.56     345100.51    56211.61       12424.50   \n",
       "13194   3829.36       114433.58     397574.04    56211.64       12424.50   \n",
       "19177      0.00            0.00     507831.60   105052.98       12424.50   \n",
       "\n",
       "       Other Benefits  Total Benefits  Total Compensation  \n",
       "27308         -423.76         -423.76             -423.76  \n",
       "15746          -22.63         -118.21             -410.61  \n",
       "24576         -159.12         -159.12             -159.12  \n",
       "42982          -26.53          -26.53              -26.53  \n",
       "23310           -9.51           -9.51               -9.51  \n",
       "...               ...             ...                 ...  \n",
       "5171         20963.32        84409.62           422799.94  \n",
       "17805        21691.23        99922.06           426686.07  \n",
       "499          13482.66        82118.77           427219.28  \n",
       "13194        14299.10        82935.24           480509.28  \n",
       "19177        23566.16       141043.64           648875.24  \n",
       "\n",
       "[42989 rows x 22 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sf2015.sort_values(by=['Total Compensation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For clarity of comparison, we will focus our attention on those who had at least the equivalent of a half-time job for the whole year. At a minimum wage of about \\$10 per hour, and 20 hours per week for 52 weeks, that's a salary of about \\$10,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sf2015 = sf2015[sf2015['Salaries'] > 10000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36569"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sf2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Population and Parameter\n",
    "Let this table of just over 36,500 rows be our population. Here is a histogram of the total compensations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28452 characters]\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sf_bins = np.arange(0, 700000, 25000)\n",
    "\n",
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "\n",
    "ax.hist(sf2015['Total Compensation'], bins = sf_bins, density=True, color='blue', alpha=0.8, ec='white')\n",
    "\n",
    "#ax.scatter(observed_distance, 0, color='red', s=40, zorder=10).set_clip_on(False)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Total Compensation'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While most of the values are below \\\\$300,000,  a few are quite a bit higher. For example, the total compensation of the Chief Investment Officer was almost \\\\$650,000. That is why the horizontal axis stretches to \\\\$700,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year Type</th>\n",
       "      <th>Year</th>\n",
       "      <th>Organization Group Code</th>\n",
       "      <th>Organization Group</th>\n",
       "      <th>Department Code</th>\n",
       "      <th>Department</th>\n",
       "      <th>Union Code</th>\n",
       "      <th>Union</th>\n",
       "      <th>Job Family Code</th>\n",
       "      <th>Job Family</th>\n",
       "      <th>...</th>\n",
       "      <th>Employee Identifier</th>\n",
       "      <th>Salaries</th>\n",
       "      <th>Overtime</th>\n",
       "      <th>Other Salaries</th>\n",
       "      <th>Total Salary</th>\n",
       "      <th>Retirement</th>\n",
       "      <th>Health/Dental</th>\n",
       "      <th>Other Benefits</th>\n",
       "      <th>Total Benefits</th>\n",
       "      <th>Total Compensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19177</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>RET</td>\n",
       "      <td>Retirement System</td>\n",
       "      <td>351.0</td>\n",
       "      <td>Municipal Executive Association - Miscellaneous</td>\n",
       "      <td>1100</td>\n",
       "      <td>Administrative &amp; Mgmt (Unrep)</td>\n",
       "      <td>...</td>\n",
       "      <td>46881</td>\n",
       "      <td>507831.6</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>507831.60</td>\n",
       "      <td>105052.98</td>\n",
       "      <td>12424.5</td>\n",
       "      <td>23566.16</td>\n",
       "      <td>141043.64</td>\n",
       "      <td>648875.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13194</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>6</td>\n",
       "      <td>General Administration &amp; Finance</td>\n",
       "      <td>ADM</td>\n",
       "      <td>General Services Agency - City Admin</td>\n",
       "      <td>164.0</td>\n",
       "      <td>Physicians and Dentists - Miscellaneous</td>\n",
       "      <td>2500</td>\n",
       "      <td>Med Therapy &amp; Auxiliary</td>\n",
       "      <td>...</td>\n",
       "      <td>1016</td>\n",
       "      <td>279311.1</td>\n",
       "      <td>3829.36</td>\n",
       "      <td>114433.58</td>\n",
       "      <td>397574.04</td>\n",
       "      <td>56211.64</td>\n",
       "      <td>12424.5</td>\n",
       "      <td>14299.10</td>\n",
       "      <td>82935.24</td>\n",
       "      <td>480509.28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Year Type  Year  Organization Group Code  \\\n",
       "19177  Calendar  2015                        6   \n",
       "13194  Calendar  2015                        6   \n",
       "\n",
       "                     Organization Group Department Code  \\\n",
       "19177  General Administration & Finance             RET   \n",
       "13194  General Administration & Finance             ADM   \n",
       "\n",
       "                                 Department  Union Code  \\\n",
       "19177                     Retirement System       351.0   \n",
       "13194  General Services Agency - City Admin       164.0   \n",
       "\n",
       "                                                 Union Job Family Code  \\\n",
       "19177  Municipal Executive Association - Miscellaneous            1100   \n",
       "13194          Physicians and Dentists - Miscellaneous            2500   \n",
       "\n",
       "                          Job Family  ... Employee Identifier  Salaries  \\\n",
       "19177  Administrative & Mgmt (Unrep)  ...               46881  507831.6   \n",
       "13194        Med Therapy & Auxiliary  ...                1016  279311.1   \n",
       "\n",
       "       Overtime  Other Salaries  Total Salary  Retirement  Health/Dental  \\\n",
       "19177      0.00            0.00     507831.60   105052.98        12424.5   \n",
       "13194   3829.36       114433.58     397574.04    56211.64        12424.5   \n",
       "\n",
       "       Other Benefits  Total Benefits  Total Compensation  \n",
       "19177        23566.16       141043.64           648875.24  \n",
       "13194        14299.10        82935.24           480509.28  \n",
       "\n",
       "[2 rows x 22 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sf2015.sort_values(by=['Total Compensation'], ascending=False).head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let the parameter be the median of the total compensations.\n",
    "\n",
    "Since we have the luxury of having all of the data from the population, we can simply calculate the parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "110305.79"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pop_median = np.percentile(sf2015['Total Compensation'], 50)\n",
    "pop_median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The median total compensation of all employees was just over \\$110,300. \n",
    "\n",
    "From a practical perspective, there is no reason for us to draw a sample to estimate this parameter since we simply know its value. But in this section we are going to pretend we don't know the value, and see how well we can estimate it based on a random sample. \n",
    "\n",
    "In later sections, we will come down to earth and work in situations where the parameter is unknown. For now, we are all-knowing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Random Sample and an Estimate\n",
    "Let us draw a sample of 500 employees at random without replacement, and let the median total compensation of the sampled employees serve as our estimate of the parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28244 characters]\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "our_sample = sf2015.sample(500, replace=False)\n",
    "\n",
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "\n",
    "ax.hist(our_sample['Total Compensation'], bins = sf_bins, density=True, color='blue', alpha=0.8, ec='white')\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Total Compensation'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "106524.32"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est_median = np.percentile(our_sample['Total Compensation'], 50)\n",
    "est_median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample size is large. By the law of averages, the distribution of the sample resembles that of the population, and consequently the sample median is not very far from the population median (though of course it is not exactly the same).\n",
    "\n",
    "So now we have one estimate of the parameter. But had the sample come out differently, the estimate would have had a different value. We would like to be able to quantify the amount by which the estimate could vary across samples. That measure of variability will help us measure how accurately we can estimate the parameter.\n",
    "\n",
    "To see how different the estimate would be if the sample had come out differently, we could just draw another sample from the population, but that would be cheating. We are trying to mimic real life, in which we won't have all the population data at hand.\n",
    "\n",
    "Somehow, we have to get another random sample without sampling from the population."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Bootstrap: Resampling from the Sample\n",
    "What we do have is a large random sample from the population. As we know, a large random sample is likely to resemble the population from which it is drawn. This observation allows data scientists to *lift themselves up by their own bootstraps*: the sampling procedure can be replicated by *sampling from the sample*. \n",
    "\n",
    "Here are the steps of *the bootstrap method* for generating another random sample that resembles the population:\n",
    "\n",
    "- **Treat the original sample as if it were the population.**\n",
    "- **Draw from the sample**, at random **with** replacement, **the same number of times as the original sample size**. \n",
    "\n",
    "It is important to resample the same number of times as the original sample size. The reason is that the variability of an estimate depends on the size of the sample. Since our original sample consisted of 500 employees, our sample median was based on 500 values. To see how different the sample could have been, we have to compare it to the median of other samples of size 500.\n",
    "\n",
    "If we drew 500 times at random *without* replacement from our sample of size 500, we would just get the same sample back. By drawing *with* replacement, we create the possibility for the new samples to be different from the original, because some employees might be drawn more than once and others not at all.\n",
    "\n",
    "Why is this a good idea? By the law of averages, the distribution of the original sample is likely to resemble the population, and the distributions of all the \"resamples\" are likely to resemble the original sample. So the distributions of all the resamples are likely to resemble the population as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "source": [
    "![Bootstrap](../../images/bootstrap_pic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Resampled Median\n",
    "Recall that when the `sample` method is used without specifying a sample size, by default the sample size equals the number of rows of the table from which the sample is drawn. That's perfect for the bootstrap! Here is one new sample drawn from the original sample, and the corresponding sample median."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year Type</th>\n",
       "      <th>Year</th>\n",
       "      <th>Organization Group Code</th>\n",
       "      <th>Organization Group</th>\n",
       "      <th>Department Code</th>\n",
       "      <th>Department</th>\n",
       "      <th>Union Code</th>\n",
       "      <th>Union</th>\n",
       "      <th>Job Family Code</th>\n",
       "      <th>Job Family</th>\n",
       "      <th>...</th>\n",
       "      <th>Employee Identifier</th>\n",
       "      <th>Salaries</th>\n",
       "      <th>Overtime</th>\n",
       "      <th>Other Salaries</th>\n",
       "      <th>Total Salary</th>\n",
       "      <th>Retirement</th>\n",
       "      <th>Health/Dental</th>\n",
       "      <th>Other Benefits</th>\n",
       "      <th>Total Benefits</th>\n",
       "      <th>Total Compensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8264</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>250.0</td>\n",
       "      <td>SEIU - Health Workers, Local 1021</td>\n",
       "      <td>2700</td>\n",
       "      <td>Housekeeping &amp; Laundry</td>\n",
       "      <td>...</td>\n",
       "      <td>2712</td>\n",
       "      <td>56531.01</td>\n",
       "      <td>621.29</td>\n",
       "      <td>2350.53</td>\n",
       "      <td>59502.83</td>\n",
       "      <td>11681.45</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>4730.12</td>\n",
       "      <td>28836.07</td>\n",
       "      <td>88338.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4685</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>250.0</td>\n",
       "      <td>SEIU - Health Workers, Local 1021</td>\n",
       "      <td>2300</td>\n",
       "      <td>Nursing</td>\n",
       "      <td>...</td>\n",
       "      <td>35035</td>\n",
       "      <td>17258.19</td>\n",
       "      <td>4907.16</td>\n",
       "      <td>1878.03</td>\n",
       "      <td>24043.38</td>\n",
       "      <td>4554.88</td>\n",
       "      <td>4696.52</td>\n",
       "      <td>1919.13</td>\n",
       "      <td>11170.53</td>\n",
       "      <td>35213.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12319</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>2</td>\n",
       "      <td>Public Works, Transportation &amp; Commerce</td>\n",
       "      <td>DPW</td>\n",
       "      <td>General Services Agency - Public Works</td>\n",
       "      <td>21.0</td>\n",
       "      <td>Prof &amp; Tech Engineers - Miscellaneous, Local 21</td>\n",
       "      <td>5200</td>\n",
       "      <td>Professional Engineering</td>\n",
       "      <td>...</td>\n",
       "      <td>4931</td>\n",
       "      <td>135421.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>135421.01</td>\n",
       "      <td>27253.50</td>\n",
       "      <td>12424.52</td>\n",
       "      <td>10116.83</td>\n",
       "      <td>49794.85</td>\n",
       "      <td>185215.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23490</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>3</td>\n",
       "      <td>Human Welfare &amp; Neighborhood Development</td>\n",
       "      <td>DSS</td>\n",
       "      <td>Human Services</td>\n",
       "      <td>21.0</td>\n",
       "      <td>Prof &amp; Tech Engineers - Miscellaneous, Local 21</td>\n",
       "      <td>1200</td>\n",
       "      <td>Personnel</td>\n",
       "      <td>...</td>\n",
       "      <td>43280</td>\n",
       "      <td>92085.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>92085.00</td>\n",
       "      <td>18979.21</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>7567.93</td>\n",
       "      <td>38971.64</td>\n",
       "      <td>131056.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15708</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>790.0</td>\n",
       "      <td>SEIU - Miscellaneous, Local 1021</td>\n",
       "      <td>9900</td>\n",
       "      <td>Public Service Aide</td>\n",
       "      <td>...</td>\n",
       "      <td>35494</td>\n",
       "      <td>37494.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>37494.70</td>\n",
       "      <td>8999.16</td>\n",
       "      <td>12177.75</td>\n",
       "      <td>3055.99</td>\n",
       "      <td>24232.90</td>\n",
       "      <td>61727.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23644</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>5</td>\n",
       "      <td>Culture &amp; Recreation</td>\n",
       "      <td>FAM</td>\n",
       "      <td>Fine Arts Museum</td>\n",
       "      <td>351.0</td>\n",
       "      <td>Municipal Executive Association - Miscellaneous</td>\n",
       "      <td>0900</td>\n",
       "      <td>Management</td>\n",
       "      <td>...</td>\n",
       "      <td>35997</td>\n",
       "      <td>118427.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>118427.00</td>\n",
       "      <td>23833.74</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>24506.58</td>\n",
       "      <td>60764.82</td>\n",
       "      <td>179191.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8710</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>791.0</td>\n",
       "      <td>SEIU - Staff and Per Diem Nurses, Local 1021</td>\n",
       "      <td>2300</td>\n",
       "      <td>Nursing</td>\n",
       "      <td>...</td>\n",
       "      <td>5225</td>\n",
       "      <td>17916.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>17916.00</td>\n",
       "      <td>3248.16</td>\n",
       "      <td>1911.46</td>\n",
       "      <td>1416.06</td>\n",
       "      <td>6575.68</td>\n",
       "      <td>24491.68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25486</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>Public Protection</td>\n",
       "      <td>POL</td>\n",
       "      <td>Police</td>\n",
       "      <td>911.0</td>\n",
       "      <td>Police Officers' Association</td>\n",
       "      <td>Q000</td>\n",
       "      <td>Police Services</td>\n",
       "      <td>...</td>\n",
       "      <td>32237</td>\n",
       "      <td>140578.74</td>\n",
       "      <td>95.73</td>\n",
       "      <td>10759.56</td>\n",
       "      <td>151434.03</td>\n",
       "      <td>28046.17</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>2569.95</td>\n",
       "      <td>43040.62</td>\n",
       "      <td>194474.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11368</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>Public Protection</td>\n",
       "      <td>CRT</td>\n",
       "      <td>Superior Court</td>\n",
       "      <td>195.0</td>\n",
       "      <td>Court Unrepresented Professionals</td>\n",
       "      <td>SCRT</td>\n",
       "      <td>SF Superior Court</td>\n",
       "      <td>...</td>\n",
       "      <td>43867</td>\n",
       "      <td>68656.96</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3000.00</td>\n",
       "      <td>71656.96</td>\n",
       "      <td>13250.87</td>\n",
       "      <td>7778.27</td>\n",
       "      <td>5949.31</td>\n",
       "      <td>26978.45</td>\n",
       "      <td>98635.41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25489</th>\n",
       "      <td>Calendar</td>\n",
       "      <td>2015</td>\n",
       "      <td>4</td>\n",
       "      <td>Community Health</td>\n",
       "      <td>DPH</td>\n",
       "      <td>Public Health</td>\n",
       "      <td>856.0</td>\n",
       "      <td>Teamsters - Miscellaneous, Local 856</td>\n",
       "      <td>2400</td>\n",
       "      <td>Lab, Pharmacy &amp; Med Techs</td>\n",
       "      <td>...</td>\n",
       "      <td>27976</td>\n",
       "      <td>107416.03</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>107416.03</td>\n",
       "      <td>22139.02</td>\n",
       "      <td>12424.50</td>\n",
       "      <td>8208.83</td>\n",
       "      <td>42772.35</td>\n",
       "      <td>150188.38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>500 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Year Type  Year  Organization Group Code  \\\n",
       "8264   Calendar  2015                        4   \n",
       "4685   Calendar  2015                        4   \n",
       "12319  Calendar  2015                        2   \n",
       "23490  Calendar  2015                        3   \n",
       "15708  Calendar  2015                        4   \n",
       "...         ...   ...                      ...   \n",
       "23644  Calendar  2015                        5   \n",
       "8710   Calendar  2015                        4   \n",
       "25486  Calendar  2015                        1   \n",
       "11368  Calendar  2015                        1   \n",
       "25489  Calendar  2015                        4   \n",
       "\n",
       "                             Organization Group Department Code  \\\n",
       "8264                           Community Health             DPH   \n",
       "4685                           Community Health             DPH   \n",
       "12319   Public Works, Transportation & Commerce             DPW   \n",
       "23490  Human Welfare & Neighborhood Development             DSS   \n",
       "15708                          Community Health             DPH   \n",
       "...                                         ...             ...   \n",
       "23644                      Culture & Recreation             FAM   \n",
       "8710                           Community Health             DPH   \n",
       "25486                         Public Protection             POL   \n",
       "11368                         Public Protection             CRT   \n",
       "25489                          Community Health             DPH   \n",
       "\n",
       "                                   Department  Union Code  \\\n",
       "8264                            Public Health       250.0   \n",
       "4685                            Public Health       250.0   \n",
       "12319  General Services Agency - Public Works        21.0   \n",
       "23490                          Human Services        21.0   \n",
       "15708                           Public Health       790.0   \n",
       "...                                       ...         ...   \n",
       "23644                        Fine Arts Museum       351.0   \n",
       "8710                            Public Health       791.0   \n",
       "25486                                  Police       911.0   \n",
       "11368                          Superior Court       195.0   \n",
       "25489                           Public Health       856.0   \n",
       "\n",
       "                                                 Union Job Family Code  \\\n",
       "8264                 SEIU - Health Workers, Local 1021            2700   \n",
       "4685                 SEIU - Health Workers, Local 1021            2300   \n",
       "12319  Prof & Tech Engineers - Miscellaneous, Local 21            5200   \n",
       "23490  Prof & Tech Engineers - Miscellaneous, Local 21            1200   \n",
       "15708                 SEIU - Miscellaneous, Local 1021            9900   \n",
       "...                                                ...             ...   \n",
       "23644  Municipal Executive Association - Miscellaneous            0900   \n",
       "8710      SEIU - Staff and Per Diem Nurses, Local 1021            2300   \n",
       "25486                     Police Officers' Association            Q000   \n",
       "11368                Court Unrepresented Professionals            SCRT   \n",
       "25489             Teamsters - Miscellaneous, Local 856            2400   \n",
       "\n",
       "                      Job Family  ... Employee Identifier   Salaries  \\\n",
       "8264      Housekeeping & Laundry  ...                2712   56531.01   \n",
       "4685                     Nursing  ...               35035   17258.19   \n",
       "12319   Professional Engineering  ...                4931  135421.01   \n",
       "23490                  Personnel  ...               43280   92085.00   \n",
       "15708        Public Service Aide  ...               35494   37494.70   \n",
       "...                          ...  ...                 ...        ...   \n",
       "23644                 Management  ...               35997  118427.00   \n",
       "8710                     Nursing  ...                5225   17916.00   \n",
       "25486            Police Services  ...               32237  140578.74   \n",
       "11368          SF Superior Court  ...               43867   68656.96   \n",
       "25489  Lab, Pharmacy & Med Techs  ...               27976  107416.03   \n",
       "\n",
       "       Overtime  Other Salaries  Total Salary  Retirement  Health/Dental  \\\n",
       "8264     621.29         2350.53      59502.83    11681.45       12424.50   \n",
       "4685    4907.16         1878.03      24043.38     4554.88        4696.52   \n",
       "12319      0.00            0.00     135421.01    27253.50       12424.52   \n",
       "23490      0.00            0.00      92085.00    18979.21       12424.50   \n",
       "15708      0.00            0.00      37494.70     8999.16       12177.75   \n",
       "...         ...             ...           ...         ...            ...   \n",
       "23644      0.00            0.00     118427.00    23833.74       12424.50   \n",
       "8710       0.00            0.00      17916.00     3248.16        1911.46   \n",
       "25486     95.73        10759.56     151434.03    28046.17       12424.50   \n",
       "11368      0.00         3000.00      71656.96    13250.87        7778.27   \n",
       "25489      0.00            0.00     107416.03    22139.02       12424.50   \n",
       "\n",
       "       Other Benefits  Total Benefits  Total Compensation  \n",
       "8264          4730.12        28836.07            88338.90  \n",
       "4685          1919.13        11170.53            35213.91  \n",
       "12319        10116.83        49794.85           185215.86  \n",
       "23490         7567.93        38971.64           131056.64  \n",
       "15708         3055.99        24232.90            61727.60  \n",
       "...               ...             ...                 ...  \n",
       "23644        24506.58        60764.82           179191.82  \n",
       "8710          1416.06         6575.68            24491.68  \n",
       "25486         2569.95        43040.62           194474.65  \n",
       "11368         5949.31        26978.45            98635.41  \n",
       "25489         8208.83        42772.35           150188.38  \n",
       "\n",
       "[500 rows x 22 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resample_1 = our_sample.sample(len(our_sample), replace=True)\n",
    "resample_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28348 characters]\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "\n",
    "ax.hist(resample_1['Total Compensation'], bins = sf_bins, density=True, color='blue', alpha=0.8, ec='white')\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Total Compensation'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "111355.89"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resampled_median_1 = np.percentile(resample_1['Total Compensation'], 50, interpolation='nearest')\n",
    "resampled_median_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By resampling, we have another estimate of the population median. By resampling again and again, we will get many such estimates, and hence an empirical distribution of the estimates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "107778.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resample_2 = our_sample.sample(len(our_sample), replace=True)\n",
    "resampled_median_2 = np.percentile(resample_2['Total Compensation'], 50, interpolation='nearest')\n",
    "resampled_median_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bootstrap Empirical Distribution of the Sample Median\n",
    "Let us define a function `bootstrap_median` that takes our original sample, the label of the column containing the variable, and the number of bootstrap samples we want to take, and returns an array of the corresponding resampled medians. \n",
    "\n",
    "Each time we resample and find the median, we *replicate* the bootstrap process. So the number of bootstrap samples will be called the number of replications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bootstrap_median(original_sample, label, replications):\n",
    "    \"\"\"Returns an array of bootstrapped sample medians:\n",
    "    original_sample: table containing the original sample\n",
    "    label: label of column containing the variable\n",
    "    replications: number of bootstrap samples\n",
    "    \"\"\"\n",
    "    just_one_column = original_sample[label]\n",
    "    medians = np.array([])\n",
    "    for i in np.arange(replications):\n",
    "        bootstrap_sample = just_one_column.sample(len(just_one_column), replace=True)\n",
    "        resampled_median = np.percentile(bootstrap_sample, 50)\n",
    "        medians = np.append(medians, resampled_median)\n",
    "        \n",
    "    return medians"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now replicate the bootstrap process 5,000 times. The array `bstrap_medians` contains the medians of all 5,000 bootstrap samples. Notice that the code takes longer to run than our previous code. It has a lot of resampling to do!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "bstrap_medians = bootstrap_median(our_sample, 'Total Compensation', 5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the histogram of the 5000 medians. The red dot is the population parameter: it is the median of the entire population, which we happen to know but did not use in the bootstrap process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 25512 characters]\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "resampled_medians = pd.DataFrame({'Bootstrap Sample Median':bstrap_medians})\n",
    "\n",
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "\n",
    "ax.hist(resampled_medians, density=True, color='blue', alpha=0.8, ec='white')\n",
    "\n",
    "ax.scatter(pop_median, 0, color='red', s=40, zorder=10).set_clip_on(False)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Bootstrap Sample Median'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to remember that the red dot is fixed: it is \\$110,305.79, the population median. The empirical histogram is the result of random draws, and will be situated randomly relative to the red dot. \n",
    "\n",
    "Remember also that the point of all these computations is to estimate the population median, which is the red dot. Our estimates are all the randomly generated sampled medians whose histogram you see above. We want those estimates to contain the parameter – it they don't, then they are off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Do the Estimates Capture the Parameter?\n",
    "How often does the empirical histogram of the resampled medians sit firmly over the red dot, and not just brush the dot with its tails? To answer this, we must define \"sit firmly\". Let's take that to mean \"the middle 95% of the resampled medians contains the red dot\". \n",
    "\n",
    "Here are the two ends of the \"middle 95%\" interval of resampled medians:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "98658.98000000001"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "left = np.percentile(bstrap_medians, 2.5, interpolation='nearest')\n",
    "left"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "114416.98999999999"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "right = np.percentile(bstrap_medians, 97.5, interpolation='nearest')\n",
    "right"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The population median of \\$110,305 is between these two numbers. The interval and the population median are shown on the histogram below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 25604 characters]\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "resampled_medians = pd.DataFrame({'Bootstrap Sample Median':bstrap_medians})\n",
    "\n",
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "\n",
    "ax.hist(resampled_medians, density=True, color='blue', alpha=0.8, ec='white', zorder=5)\n",
    "\n",
    "ax.plot(np.array([left, right]), np.array([0,0]), color='yellow', lw=8, zorder=10)\n",
    "\n",
    "ax.scatter(pop_median, 0, color='red', s=40, zorder=15).set_clip_on(False)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Bootstrap Sample Median'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"middle 95%\" interval of estimates captured the parameter in our example. But was that a fluke? \n",
    "\n",
    "To see how frequently the interval contains the parameter, we have to run the entire process over and over again. Specifically, we will repeat the following process 100 times:\n",
    "\n",
    "- Draw an original sample of size 500 from the population.\n",
    "- Carry out 5,000 replications of the bootstrap process and generate the \"middle 95%\" interval of resampled medians.\n",
    "\n",
    "We will end up with 100 intervals, and count how many of them contain the population median.\n",
    "\n",
    "**Spoiler alert:** The statistical theory of the bootstrap says that the number should be around 95. It may be in the low 90s or high 90s, but not much farther off 95 than that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total Compensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>117766.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41209.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>110561.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>38624.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>260280.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42983</th>\n",
       "      <td>61349.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42984</th>\n",
       "      <td>132788.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42986</th>\n",
       "      <td>73295.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42987</th>\n",
       "      <td>19973.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42988</th>\n",
       "      <td>55812.90</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>36569 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Total Compensation\n",
       "0               117766.86\n",
       "1                41209.83\n",
       "2               110561.13\n",
       "3                38624.97\n",
       "6               260280.95\n",
       "...                   ...\n",
       "42983            61349.71\n",
       "42984           132788.81\n",
       "42986            73295.59\n",
       "42987            19973.37\n",
       "42988            55812.90\n",
       "\n",
       "[36569 rows x 1 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_comps = sf2015[['Total Compensation']]\n",
    "total_comps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# THE BIG SIMULATION: This one takes several minutes.\n",
    "\n",
    "# Generate 100 intervals, in the table intervals\n",
    "\n",
    "left_ends = np.array([])\n",
    "right_ends = np.array([])\n",
    "\n",
    "total_comps = sf2015[['Total Compensation']]\n",
    "\n",
    "for i in np.arange(100):\n",
    "    first_sample = total_comps.sample(500, replace=False)\n",
    "    medians = bootstrap_median(first_sample, 'Total Compensation', 5000)\n",
    "    left_ends = np.append(left_ends, np.percentile(medians, 2.5))\n",
    "    right_ends = np.append(right_ends, np.percentile(medians, 97.5))\n",
    "\n",
    "intervals = pd.DataFrame(\n",
    "    {'Left':left_ends,\n",
    "    'Right':right_ends}\n",
    ")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the 100 replications, we get one interval of estimates of the median."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Left</th>\n",
       "      <th>Right</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100639.945000</td>\n",
       "      <td>113009.80850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>103596.780000</td>\n",
       "      <td>113549.15000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>106650.085125</td>\n",
       "      <td>116872.21000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>101855.280000</td>\n",
       "      <td>111880.11000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>98633.755000</td>\n",
       "      <td>113396.61500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>103042.430000</td>\n",
       "      <td>117401.49500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>97683.995000</td>\n",
       "      <td>113182.87500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>111704.995000</td>\n",
       "      <td>121917.01000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>105761.655000</td>\n",
       "      <td>117376.24225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>104795.530000</td>\n",
       "      <td>116920.50500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Left         Right\n",
       "0   100639.945000  113009.80850\n",
       "1   103596.780000  113549.15000\n",
       "2   106650.085125  116872.21000\n",
       "3   101855.280000  111880.11000\n",
       "4    98633.755000  113396.61500\n",
       "..            ...           ...\n",
       "95  103042.430000  117401.49500\n",
       "96   97683.995000  113182.87500\n",
       "97  111704.995000  121917.01000\n",
       "98  105761.655000  117376.24225\n",
       "99  104795.530000  116920.50500\n",
       "\n",
       "[100 rows x 2 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "intervals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The good intervals are those that contain the parameter we are trying to estimate. Typically the parameter is unknown, but in this section we happen to know what the parameter is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "110305.79"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pop_median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many of the 100 intervals contain the population median? That's the number of intervals where the left end is below the population median and the right end is above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "95"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(\n",
    "    intervals[\n",
    "        (intervals['Left'] < pop_median) & \n",
    "              (intervals['Right'] > pop_median)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes a few minutes to construct all the intervals, but try it again if you have the patience. Most likely, about 95 of the 100 intervals will be good ones: they will contain the parameter.\n",
    "\n",
    "It's hard to show you all the intervals on the horizontal axis as they have large overlaps – after all, they are all trying to estimate the same parameter. The graphic below shows each interval on the same axes by stacking them vertically. The vertical axis is simply the number of the replication from which the interval was generated.\n",
    "\n",
    "The red line is where the parameter is. Good intervals cover the parameter; there are about 95 of these, typically. \n",
    "\n",
    "If an interval doesn't cover the parameter, it's a dud. The duds are the ones where you can see \"daylight\" around the red line. There are very few of them – about 5, typically – but they do happen. \n",
    "\n",
    "Any method based on sampling has the possibility of being off. The beauty of methods based on random sampling is that we can quantify how often they are likely to be off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29564 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "replication_number = np.arange(1, 101)\n",
    "\n",
    "replication_number = replication_number.astype(str)\n",
    "\n",
    "intervals2 = pd.DataFrame(np.array([left_ends, right_ends]), columns=[replication_number])\n",
    "\n",
    "intervals2\n",
    "\n",
    "plt.figure(figsize=(8,8))\n",
    "for i in np.arange(100):\n",
    "    ends = intervals2.iloc[:,i]\n",
    "    plt.plot(ends, np.array([i+1, i+1]), color='gold')\n",
    "plt.plot(np.array([pop_median, pop_median]), np.array([0, 100]), color='red', lw=2)\n",
    "plt.xlabel('Median (dollars)')\n",
    "plt.ylabel('Replication')\n",
    "plt.title('Population Median and Intervals of Estimates');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize what the simulation shows, suppose you are estimating the population median by the following process: \n",
    "\n",
    "- Draw a large random sample from the population.\n",
    "- Bootstrap your random sample and get an estimate from the new random sample. \n",
    "- Repeat the above step thousands of times, and get thousands of estimates.\n",
    "- Pick off the \"middle 95%\" interval of all the estimates.\n",
    "\n",
    "That gives you one interval of estimates. Now if you repeat **the entire process** 100 times, ending up with 100 intervals, then about 95 of those 100 intervals will contain the population parameter.\n",
    "\n",
    "In other words, this process of estimation captures the parameter about 95% of the time. \n",
    "\n",
    "You can replace 95% by a different value, as long as it's not 100. Suppose you replace 95% by 80% and keep the sample size fixed at 500. Then your intervals of estimates will be shorter than those we simulated here, because the \"middle 80%\" is a smaller range than the \"middle 95%\". Only about 80% of your intervals will contain the parameter."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
