{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bootstrap_median(original_sample, label, replications):\n",
    "    \"\"\"Returns an array of bootstrapped sample medians:\n",
    "    original_sample: table containing the original sample\n",
    "    label: label of column containing the variable\n",
    "    replications: number of bootstrap samples\n",
    "    \"\"\"\n",
    "    just_one_column = original_sample[label]\n",
    "    medians = np.array([])\n",
    "    for i in np.arange(replications):\n",
    "        bootstrap_sample = just_one_column.sample(len(just_one_column), replace=True)\n",
    "        resampled_median = np.percentile(bootstrap_sample, 50)\n",
    "        medians = np.append(medians, resampled_median)\n",
    "        \n",
    "    return medians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bootstrap_mean(original_sample, label, replications):\n",
    "    \n",
    "    \"\"\"Returns an array of bootstrapped sample means:\n",
    "    original_sample: table containing the original sample\n",
    "    label: label of column containing the variable\n",
    "    replications: number of bootstrap samples\n",
    "    \"\"\"\n",
    "    \n",
    "    just_one_column = original_sample[[label]]\n",
    "    means = np.array([])\n",
    "    for i in np.arange(replications):\n",
    "        bootstrap_sample = just_one_column.sample(len(just_one_column), replace=True)\n",
    "        resampled_mean = np.mean(bootstrap_sample.iloc[:,0])\n",
    "        means = np.append(means, resampled_mean)\n",
    "        \n",
    "    return means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bootstrap_proportion(original_sample, label, replications):\n",
    "    \n",
    "    \"\"\"Returns an array of bootstrapped sample proportions:\n",
    "    original_sample: table containing the original sample\n",
    "    label: label of column containing the Boolean variable\n",
    "    replications: number of bootstrap samples\n",
    "    \"\"\"\n",
    "    \n",
    "    just_one_column = original_sample[[label]]\n",
    "    proportions = np.array([])\n",
    "    for i in np.arange(replications):\n",
    "        bootstrap_sample = just_one_column.sample(len(just_one_column), replace=True)\n",
    "        resample_array = bootstrap_sample.iloc[:,0]\n",
    "        resampled_proportion = np.count_nonzero(resample_array)/len(resample_array)\n",
    "        proportions = np.append(proportions, resampled_proportion)\n",
    "        \n",
    "    return proportions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Confidence Intervals\n",
    "A confidence interval has a single purpose – to estimate an unknown parameter based on data in a random sample. In the last section, we said that the interval (36%, 42%) was an approximate 95% confidence interval for the percent of smokers among mothers in the population. That was a formal way of saying that by our estimate, the percent of smokers among the mothers in the population was somewhere between 36% and 42%, and that our process of estimation is correct about 95% of the time.\n",
    "\n",
    "It is important to resist the impulse to use confidence intervals for other purposes. For example, recall that we calculated the interval (26.9 years, 27.6 years) as an approximate 95% confidence interval for the average age of mothers in the population. A dismayingly common misuse of the interval is to conclude that about 95% of the women were between 26.9 years and 27.6 years old. You don't need to know much about confidence intervals to see that this can't be right – you wouldn't expect 95% of mothers to all be within a few months of each other in age. Indeed, the histogram of the sampled ages shows quite a bit of variation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "baby = pd.read_csv(path_data + 'baby.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15632 characters]\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,5))\n",
    "\n",
    "ax.hist(baby['Maternal Age'], density=True, color='blue', alpha=0.8, ec='white', zorder=5)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Maternal Age'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "#plt.xticks(rotation=90)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A small percent of the sampled ages are in the (26.9, 27.6) interval, and you would expect a similar small percent in the population. The interval just estimates one number: the *average* of all the ages in the population."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, estimating a parameter by confidence intervals does have an important use besides just telling us roughly how big the parameter is. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a Confidence Interval to Test Hypotheses\n",
    "Our approximate 95% confidence interval for the average age in the population goes from 26.9 years to 27.6 years. Suppose someone wants to test the following hypotheses:\n",
    "\n",
    "**Null hypothesis.** The average age in the population is 30 years.\n",
    "\n",
    "**Alternative hypothesis.** The average age in the population is not 30 years.\n",
    "\n",
    "Then, if you were using the 5% cutoff for the P-value, you would reject the null hypothesis. This is because 30 is not in the 95% confidence interval for the population average. At the 5% level of significance, 30 is not a plausible value for the population average.\n",
    "\n",
    "This use of confidence intervals is the result of a *duality* between confidence intervals and tests: if you are testing whether or not the population mean is a particular value *x*, and you use the 5% cutoff for the P-value, then you will reject the null hypothesis if *x* is not in your 95% confidence interval for the mean. \n",
    "\n",
    "This can be established by statistical theory. In practice, it just boils down to checking whether or not the value specified in the null hypothesis lies in the confidence interval.\n",
    "\n",
    "If you were using the 1% cutoff for the P-value, you would have to check if the value specified in the null hypothesis lies in a 99% confidence interval for the population mean.\n",
    "\n",
    "To a rough approximation, these statements are also true for population proportions, *provided* the sample is large."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While we now have a way of using confidence intervals to test a particular kind of hypothesis, you might wonder about the value of testing whether or not the average age in a population is equal to 30. Indeed, the value isn't clear. But there are some situations in which a test of this kind of hypothesis is both natural and useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will study this in the context of data that are a subset of the information gathered in a randomized controlled trial about treatments for Hodgkin's disease. Hodgkin's disease is a cancer that typically affects young people. The disease is curable but the treatment can be very harsh. The purpose of the trial was to come up with dosage that would cure the cancer but minimize the adverse effects on the patients. \n",
    "\n",
    "This table ``hodgkins`` contains data on the effect that the treatment had on the lungs of 22 patients. The columns are:\n",
    "\n",
    "- Height in cm\n",
    "- A measure of radiation to the mantle (neck, chest, under arms)\n",
    "- A measure of chemotherapy\n",
    "- A score of the health of the lungs at baseline, that is, at the start of the treatment; higher scores correspond to more healthy lungs\n",
    "- The same score of the health of the lungs, 15 months after treatment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "hodgkins = pd.read_csv(path_data + 'hodgkins.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>height</th>\n",
       "      <th>rad</th>\n",
       "      <th>chemo</th>\n",
       "      <th>base</th>\n",
       "      <th>month15</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>164</td>\n",
       "      <td>679</td>\n",
       "      <td>180</td>\n",
       "      <td>160.57</td>\n",
       "      <td>87.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>168</td>\n",
       "      <td>311</td>\n",
       "      <td>180</td>\n",
       "      <td>98.24</td>\n",
       "      <td>67.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>173</td>\n",
       "      <td>388</td>\n",
       "      <td>239</td>\n",
       "      <td>129.04</td>\n",
       "      <td>133.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>157</td>\n",
       "      <td>370</td>\n",
       "      <td>168</td>\n",
       "      <td>85.41</td>\n",
       "      <td>81.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>160</td>\n",
       "      <td>468</td>\n",
       "      <td>151</td>\n",
       "      <td>67.94</td>\n",
       "      <td>79.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>170</td>\n",
       "      <td>341</td>\n",
       "      <td>96</td>\n",
       "      <td>150.51</td>\n",
       "      <td>80.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>163</td>\n",
       "      <td>453</td>\n",
       "      <td>134</td>\n",
       "      <td>129.88</td>\n",
       "      <td>69.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>175</td>\n",
       "      <td>529</td>\n",
       "      <td>264</td>\n",
       "      <td>87.45</td>\n",
       "      <td>56.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>185</td>\n",
       "      <td>392</td>\n",
       "      <td>240</td>\n",
       "      <td>149.84</td>\n",
       "      <td>106.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>178</td>\n",
       "      <td>479</td>\n",
       "      <td>216</td>\n",
       "      <td>92.24</td>\n",
       "      <td>73.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>179</td>\n",
       "      <td>376</td>\n",
       "      <td>160</td>\n",
       "      <td>117.43</td>\n",
       "      <td>101.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>181</td>\n",
       "      <td>539</td>\n",
       "      <td>196</td>\n",
       "      <td>129.75</td>\n",
       "      <td>90.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>173</td>\n",
       "      <td>217</td>\n",
       "      <td>204</td>\n",
       "      <td>97.59</td>\n",
       "      <td>76.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>166</td>\n",
       "      <td>456</td>\n",
       "      <td>192</td>\n",
       "      <td>81.29</td>\n",
       "      <td>67.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>170</td>\n",
       "      <td>252</td>\n",
       "      <td>150</td>\n",
       "      <td>98.29</td>\n",
       "      <td>55.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>165</td>\n",
       "      <td>622</td>\n",
       "      <td>162</td>\n",
       "      <td>118.98</td>\n",
       "      <td>90.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>173</td>\n",
       "      <td>305</td>\n",
       "      <td>213</td>\n",
       "      <td>103.17</td>\n",
       "      <td>79.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>174</td>\n",
       "      <td>566</td>\n",
       "      <td>198</td>\n",
       "      <td>94.97</td>\n",
       "      <td>93.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>173</td>\n",
       "      <td>322</td>\n",
       "      <td>119</td>\n",
       "      <td>85.00</td>\n",
       "      <td>41.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>173</td>\n",
       "      <td>270</td>\n",
       "      <td>160</td>\n",
       "      <td>115.02</td>\n",
       "      <td>81.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>183</td>\n",
       "      <td>259</td>\n",
       "      <td>241</td>\n",
       "      <td>125.02</td>\n",
       "      <td>97.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>188</td>\n",
       "      <td>238</td>\n",
       "      <td>252</td>\n",
       "      <td>137.43</td>\n",
       "      <td>113.20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    height  rad  chemo    base  month15\n",
       "0      164  679    180  160.57    87.77\n",
       "1      168  311    180   98.24    67.62\n",
       "2      173  388    239  129.04   133.33\n",
       "3      157  370    168   85.41    81.28\n",
       "4      160  468    151   67.94    79.26\n",
       "5      170  341     96  150.51    80.97\n",
       "6      163  453    134  129.88    69.24\n",
       "7      175  529    264   87.45    56.48\n",
       "8      185  392    240  149.84   106.99\n",
       "9      178  479    216   92.24    73.43\n",
       "10     179  376    160  117.43   101.61\n",
       "11     181  539    196  129.75    90.78\n",
       "12     173  217    204   97.59    76.38\n",
       "13     166  456    192   81.29    67.66\n",
       "14     170  252    150   98.29    55.51\n",
       "15     165  622    162  118.98    90.92\n",
       "16     173  305    213  103.17    79.74\n",
       "17     174  566    198   94.97    93.08\n",
       "18     173  322    119   85.00    41.96\n",
       "19     173  270    160  115.02    81.12\n",
       "20     183  259    241  125.02    97.18\n",
       "21     188  238    252  137.43   113.20"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hodgkins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will compare the baseline and 15-month scores. As each row corresponds to one patient, we say that the sample of baseline scores and the sample of 15-month scores are *paired* - they are not just two sets of 22 values each, but 22 pairs of values, one for each patient.\n",
    "\n",
    "At a glance, you can see that the 15-month scores tend to be lower than the baseline scores – the sampled patients' lungs seem to be doing worse 15 months after the treatment. This is confirmed by the mostly positive values in the column `drop`, the amount by which the score dropped from baseline to 15 months."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "hodgkins['drop'] = hodgkins['base'] - hodgkins['month15']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>height</th>\n",
       "      <th>rad</th>\n",
       "      <th>chemo</th>\n",
       "      <th>base</th>\n",
       "      <th>month15</th>\n",
       "      <th>drop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>164</td>\n",
       "      <td>679</td>\n",
       "      <td>180</td>\n",
       "      <td>160.57</td>\n",
       "      <td>87.77</td>\n",
       "      <td>72.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>168</td>\n",
       "      <td>311</td>\n",
       "      <td>180</td>\n",
       "      <td>98.24</td>\n",
       "      <td>67.62</td>\n",
       "      <td>30.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>173</td>\n",
       "      <td>388</td>\n",
       "      <td>239</td>\n",
       "      <td>129.04</td>\n",
       "      <td>133.33</td>\n",
       "      <td>-4.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>157</td>\n",
       "      <td>370</td>\n",
       "      <td>168</td>\n",
       "      <td>85.41</td>\n",
       "      <td>81.28</td>\n",
       "      <td>4.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>160</td>\n",
       "      <td>468</td>\n",
       "      <td>151</td>\n",
       "      <td>67.94</td>\n",
       "      <td>79.26</td>\n",
       "      <td>-11.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>170</td>\n",
       "      <td>341</td>\n",
       "      <td>96</td>\n",
       "      <td>150.51</td>\n",
       "      <td>80.97</td>\n",
       "      <td>69.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>163</td>\n",
       "      <td>453</td>\n",
       "      <td>134</td>\n",
       "      <td>129.88</td>\n",
       "      <td>69.24</td>\n",
       "      <td>60.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>175</td>\n",
       "      <td>529</td>\n",
       "      <td>264</td>\n",
       "      <td>87.45</td>\n",
       "      <td>56.48</td>\n",
       "      <td>30.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>185</td>\n",
       "      <td>392</td>\n",
       "      <td>240</td>\n",
       "      <td>149.84</td>\n",
       "      <td>106.99</td>\n",
       "      <td>42.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>178</td>\n",
       "      <td>479</td>\n",
       "      <td>216</td>\n",
       "      <td>92.24</td>\n",
       "      <td>73.43</td>\n",
       "      <td>18.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>179</td>\n",
       "      <td>376</td>\n",
       "      <td>160</td>\n",
       "      <td>117.43</td>\n",
       "      <td>101.61</td>\n",
       "      <td>15.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>181</td>\n",
       "      <td>539</td>\n",
       "      <td>196</td>\n",
       "      <td>129.75</td>\n",
       "      <td>90.78</td>\n",
       "      <td>38.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>173</td>\n",
       "      <td>217</td>\n",
       "      <td>204</td>\n",
       "      <td>97.59</td>\n",
       "      <td>76.38</td>\n",
       "      <td>21.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>166</td>\n",
       "      <td>456</td>\n",
       "      <td>192</td>\n",
       "      <td>81.29</td>\n",
       "      <td>67.66</td>\n",
       "      <td>13.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>170</td>\n",
       "      <td>252</td>\n",
       "      <td>150</td>\n",
       "      <td>98.29</td>\n",
       "      <td>55.51</td>\n",
       "      <td>42.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>165</td>\n",
       "      <td>622</td>\n",
       "      <td>162</td>\n",
       "      <td>118.98</td>\n",
       "      <td>90.92</td>\n",
       "      <td>28.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>173</td>\n",
       "      <td>305</td>\n",
       "      <td>213</td>\n",
       "      <td>103.17</td>\n",
       "      <td>79.74</td>\n",
       "      <td>23.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>174</td>\n",
       "      <td>566</td>\n",
       "      <td>198</td>\n",
       "      <td>94.97</td>\n",
       "      <td>93.08</td>\n",
       "      <td>1.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>173</td>\n",
       "      <td>322</td>\n",
       "      <td>119</td>\n",
       "      <td>85.00</td>\n",
       "      <td>41.96</td>\n",
       "      <td>43.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>173</td>\n",
       "      <td>270</td>\n",
       "      <td>160</td>\n",
       "      <td>115.02</td>\n",
       "      <td>81.12</td>\n",
       "      <td>33.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>183</td>\n",
       "      <td>259</td>\n",
       "      <td>241</td>\n",
       "      <td>125.02</td>\n",
       "      <td>97.18</td>\n",
       "      <td>27.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>188</td>\n",
       "      <td>238</td>\n",
       "      <td>252</td>\n",
       "      <td>137.43</td>\n",
       "      <td>113.20</td>\n",
       "      <td>24.23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    height  rad  chemo    base  month15   drop\n",
       "0      164  679    180  160.57    87.77  72.80\n",
       "1      168  311    180   98.24    67.62  30.62\n",
       "2      173  388    239  129.04   133.33  -4.29\n",
       "3      157  370    168   85.41    81.28   4.13\n",
       "4      160  468    151   67.94    79.26 -11.32\n",
       "5      170  341     96  150.51    80.97  69.54\n",
       "6      163  453    134  129.88    69.24  60.64\n",
       "7      175  529    264   87.45    56.48  30.97\n",
       "8      185  392    240  149.84   106.99  42.85\n",
       "9      178  479    216   92.24    73.43  18.81\n",
       "10     179  376    160  117.43   101.61  15.82\n",
       "11     181  539    196  129.75    90.78  38.97\n",
       "12     173  217    204   97.59    76.38  21.21\n",
       "13     166  456    192   81.29    67.66  13.63\n",
       "14     170  252    150   98.29    55.51  42.78\n",
       "15     165  622    162  118.98    90.92  28.06\n",
       "16     173  305    213  103.17    79.74  23.43\n",
       "17     174  566    198   94.97    93.08   1.89\n",
       "18     173  322    119   85.00    41.96  43.04\n",
       "19     173  270    160  115.02    81.12  33.90\n",
       "20     183  259    241  125.02    97.18  27.84\n",
       "21     188  238    252  137.43   113.20  24.23"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hodgkins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 16304 characters]\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,5))\n",
    "\n",
    "ax.hist(hodgkins['drop'], bins=np.arange(-20, 81, 20), density=True, color='blue', alpha=0.8, ec='white', zorder=5)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'drop'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.615909090909096"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(hodgkins['drop'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But could this be the result of chance variation? It really doesn't seem so, but the data are from a random sample. Could it be that in the entire population of patients, the average drop is just 0?\n",
    "\n",
    "To answer this, we can set up two hypotheses:\n",
    "\n",
    "**Null hypothesis.** In the population, the average drop is 0.\n",
    "\n",
    "**Alternative hypothesis.** In the population, the average drop is not 0.\n",
    "\n",
    "To test this hypothesis with a 1% cutoff for the P-value, let's construct an approximate 99% confidence interval for the average drop in the population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([16.90772727, 40.10590909])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bstrap_means = bootstrap_mean(hodgkins, 'drop', 10000)\n",
    "\n",
    "left = np.percentile(bstrap_means, 0.5, interpolation='nearest')\n",
    "right = np.percentile(bstrap_means, 99.5, interpolation='nearest')\n",
    "\n",
    "np.array([left, right])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 17312 characters]\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "resampled_means = pd.DataFrame({'Bootstrap Sample Mean': bstrap_means})\n",
    "\n",
    "unit = ''\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,5))\n",
    "\n",
    "ax.hist(resampled_means, density=True, color='blue', alpha=0.8, ec='white', zorder=5)\n",
    "\n",
    "ax.plot(np.array([left, right]), np.array([0,0]), color='yellow', lw=8, zorder=10)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Bootstrap Sample Means'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 99% confidence interval for the average drop in the population goes from about 17 to about 40. The interval doesn't contain 0. So we reject the null hypothesis.\n",
    "\n",
    "But notice that we have done better than simply concluding that the average drop in the population isn't 0. We have estimated how big the average drop is. That's a more useful result than just saying, \"It's not 0.\"\n",
    "\n",
    "**A note on accuracy.** Our confidence interval is quite wide, for two main reasons:\n",
    "- The confidence level is high (99%).\n",
    "- The sample size is relatively small compared to those in our earlier examples.\n",
    "\n",
    "In the next chapter, we will examine how the sample size affects accuracy. We will also examine how the empirical distributions of sample means so often come out bell shaped even though the distributions of the underlying data are not bell shaped at all."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Endnote\n",
    "The terminology of a field usually comes from the leading researchers in that field. [Brad Efron](https://en.wikipedia.org/wiki/Bradley_Efron), who first proposed the bootstrap technique, used a term that has [American origins](https://en.wikipedia.org/wiki/Bootstrapping). Not to be outdone, Chinese statisticians have [proposed their own method](http://econpapers.repec.org/article/eeestapro/v_3a37_3ay_3a1998_3ai_3a4_3ap_3a321-329.htm)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
