{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "np.set_printoptions(legacy='1.13')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "\n",
    "def standard_units(any_numbers):\n",
    "    \"Convert any array of numbers to standard units.\"\n",
    "    return (any_numbers - np.mean(any_numbers))/np.std(any_numbers)  \n",
    "\n",
    "def correlation(t, x, y):\n",
    "    return np.mean(standard_units(t[x])*standard_units(t[y]))\n",
    "\n",
    "def slope(table, x, y):\n",
    "    r = correlation(table, x, y)\n",
    "    return r * np.std(table[y])/np.std(table[x])\n",
    "\n",
    "def intercept(table, x, y):\n",
    "    a = slope(table, x, y)\n",
    "    return np.mean(table[y]) - a * np.mean(table[x])\n",
    "\n",
    "def fit(table, x, y):\n",
    "    a = slope(table, x, y)\n",
    "    b = intercept(table, x, y)\n",
    "    return a * table[x] + b\n",
    "\n",
    "def residual(table, x, y):\n",
    "    return table.column(y) - fit(table, x, y)\n",
    "\n",
    "def scatter_fit(table, x, y):\n",
    "    #fig, ax = plt.subplots(figsize=(7,6))\n",
    "    plt.scatter(table[x], table[y], color='darkblue', s=20)\n",
    "    \n",
    "    plt.plot(table[x], fit(table, x, y), lw=2, color='gold')\n",
    "    plt.xlabel(x)\n",
    "    plt.ylabel(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "def draw_and_compare(true_slope, true_int, sample_size):\n",
    "    x = np.random.normal(50, 5, sample_size)\n",
    "    xlims = np.array([np.min(x), np.max(x)])\n",
    "    eps = np.random.normal(0, 6, sample_size)\n",
    "    y = (true_slope*x + true_int) + eps\n",
    "    tyche = pd.DataFrame({'x':x,'y':y})\n",
    "\n",
    "    plt.figure(figsize=(6, 16))\n",
    "    \n",
    "    plt.subplot(4, 1, 1)\n",
    "    plt.scatter(tyche['x'], tyche['y'], s=20)\n",
    "    plt.plot(xlims, true_slope*xlims + true_int, lw=2, color='green')\n",
    "    plt.title('True Line, and Points Created')\n",
    "\n",
    "#---\n",
    "    plt.subplot(4, 1, 2)\n",
    "    plt.scatter(tyche['x'], tyche['y'], s=20)\n",
    "    plt.title('What We Get to See')\n",
    "    \n",
    "#---\n",
    "    plt.subplot(4, 1, 3)\n",
    "    scatter_fit(tyche, 'x', 'y')\n",
    "    plt.xlabel(\"\")\n",
    "    plt.ylabel(\"\")\n",
    "    plt.title('Regression Line: Estimate of True Line')\n",
    "#---\n",
    "    plt.subplot(4, 1, 4)\n",
    "    scatter_fit(tyche, 'x', 'y')\n",
    "    \n",
    "    xlims = np.array([np.min(tyche['x']), np.max(tyche['x'])])\n",
    "    plt.plot(xlims, true_slope*xlims + true_int, lw=2, color='green')\n",
    "    plt.ylabel(\"\")\n",
    "    plt.title(\"Regression Line and True Line\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "baby = pd.read_csv(path_data + 'baby.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference for the True Slope\n",
    "\n",
    "Our simulations show that if the regression model holds and the sample size is large, then the regression line is likely to be close to the true line. This allows us to estimate the slope of the true line.\n",
    "\n",
    "We will use our familiar sample of mothers and their newborn babies to develop a method of estimating the slope of the true line. First, let's see if we believe that the regression model is an appropriate set of assumptions for describing the relation between birth weight and the number of gestational days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37452 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scatter_fit(baby, 'Gestational Days', 'Birth Weight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.40754279338885196"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correlation(baby, 'Gestational Days', 'Birth Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By and large, the scatter looks fairly evenly distributed around the line, though there are some points that are scattered on the outskirts of the main cloud. The correlation is 0.4 and the regression line has a positive slope. \n",
    "\n",
    "Does this reflect the fact that the true line has a positive slope? To answer this question, let us see if we can estimate the true slope. We certainly have one estimate of it: the slope of our regression line. That's about 0.47 ounces per day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4665568769492164"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slope(baby, 'Gestational Days', 'Birth Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But had the scatter plot come out differently, the regression line would have been different and might have had a different slope. How do we figure out how different the slope might have been?\n",
    "\n",
    "We need another sample of points, so that we can draw the regression line through the new scatter plot and find its slope. But from where will get another sample?\n",
    "\n",
    "You have guessed it – we will *bootstrap our original sample*. That will give us a bootstrapped scatter plot, through which we can draw a regression line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bootstrapping the Scatter Plot\n",
    "We can simulate new samples by random sampling with replacement from the original sample, as many times as the original sample size. Each of these new samples will give us a scatter plot. We will call that a *bootstrapped scatter plot*, and for short, we will call the entire process *bootstrapping the scatter plot*.\n",
    "\n",
    "Here is the original scatter diagram from the sample, and four replications of the bootstrap resampling procedure. Notice how the resampled scatter plots are in general a little more sparse than the original. That is because some of the original points do not get selected in the samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 150640 characters]\n",
      "text/plain": [
       "<Figure size 576x1296 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "plt.figure(figsize=(8, 18))\n",
    "plt.subplot(5, 1, 1)\n",
    "plt.scatter(baby.iloc[:,1], baby.iloc[:,0], s=10)\n",
    "plt.xlim([150, 400])\n",
    "plt.title('Original sample')\n",
    "\n",
    "for i in np.arange(1, 5, 1):\n",
    "    plt.subplot(5,1,i+1)\n",
    "    rep = baby.sample(len(baby), replace=True)\n",
    "    plt.scatter(rep.iloc[:,1], rep.iloc[:,0], s=10)\n",
    "    plt.xlim([150, 400])\n",
    "    plt.title('Bootstrap sample '+str(i))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating the True Slope\n",
    "We can bootstrap the scatter plot a large number of times, and draw a regression line through each bootstrapped plot. Each of those lines has a slope. We can simply collect all the slopes and draw their empirical histogram. Recall that by default, the `sample` method draws at random with replacement, the same number of times as the number of rows in the table. That is, `sample` generates a bootstrap sample by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 16792 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "slopes = np.array([])\n",
    "for i in np.arange(5000):\n",
    "    bootstrap_sample = baby.sample(len(baby), replace=True)\n",
    "    bootstrap_slope = slope(bootstrap_sample, 'Gestational Days', 'Birth Weight')\n",
    "    slopes = np.append(slopes, bootstrap_slope)\n",
    "\n",
    "pd.DataFrame({'Bootstrap Slopes':slopes}).hist(bins=20, ec='white');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then construct an approximate 95% confidence interval for the slope of the true line, using the bootstrap percentile method. The confidence interval extends from the 2.5th percentile to the 97.5th percentile of the 5000 bootstrapped slopes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.38225576134670686, 0.55888349421281591)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "left = np.percentile(slopes, 2.5)\n",
    "right = np.percentile(slopes, 97.5)\n",
    "left, right"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An approximate 95% confidence interval for the true slope extends from about 0.38 ounces per day to about 0.56 ounces per day. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Function to Bootstrap the Slope\n",
    "Let us collect all the steps of our method of estimating the slope and define a function `bootstrap_slope` that carries them out. Its arguments are the name of the table and the labels of the predictor and response variables, and the desired number of bootstrap replications. In each replication, the function bootstraps the original scatter plot and calculates the slope of the resulting regression line. It then draws the histogram of all the generated slopes and prints the interval consisting of the \"middle 95%\" of the slopes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bootstrap_slope(table, x, y, repetitions):\n",
    "    \n",
    "    # For each repetition:\n",
    "    # Bootstrap the scatter, get the slope of the regression line,\n",
    "    # augment the list of generated slopes\n",
    "    slopes = np.array([])\n",
    "    for i in np.arange(repetitions):\n",
    "        bootstrap_sample = table.sample(len(table), replace=True)\n",
    "        bootstrap_slope = slope(bootstrap_sample, x, y)\n",
    "        slopes = np.append(slopes, bootstrap_slope)\n",
    "    \n",
    "    # Find the endpoints of the 95% confidence interval for the true slope\n",
    "    left = np.percentile(slopes, 2.5)\n",
    "    right = np.percentile(slopes, 97.5)\n",
    "    \n",
    "    # Slope of the regression line from the original sample\n",
    "    observed_slope = slope(table, x, y)\n",
    "    \n",
    "    # Display results\n",
    "    pd.DataFrame({'Bootstrap Slopes':slopes}).hist(bins=20, ec='white')\n",
    "    plt.plot(np.array([left, right]), np.array([0, 0]), color='yellow', lw=8);\n",
    "    print('Slope of regression line:', observed_slope)\n",
    "    print('Approximate 95%-confidence interval for the true slope:')\n",
    "    print(left, right)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we call `bootstrap_slope` to find a confidence interval for the true slope when the response variable is birth weight and the predictor is gestational days, we get an interval very close to the one we obtained earlier: approximately 0.38 ounces per day to 0.56 ounces per day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slope of regression line: 0.4665568769492164\n",
      "Approximate 95%-confidence interval for the true slope:\n",
      "0.381798869241 0.555611704253\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 14152 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bootstrap_slope(baby, 'Gestational Days', 'Birth Weight', 5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a function that automates our process of estimating the slope of the true line in a regression model, we can use it on other variables as well.\n",
    "\n",
    "For example, let's examine the relation between birth weight and the mother's height. Do taller women tend to have heavier babies?\n",
    "\n",
    "The regression model seems reasonable, based on the scatter plot, but the correlation is not high. It's just about 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28376 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scatter_fit(baby, 'Maternal Height', 'Birth Weight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.20370417718968062"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correlation(baby, 'Maternal Height', 'Birth Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we can use `bootstrap_slope` to estimate the slope of the true line in the regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slope of regression line: 1.4780193519284357\n",
      "Approximate 95%-confidence interval for the true slope:\n",
      "1.05686977213 1.91804340448\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 14248 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bootstrap_slope(baby, 'Maternal Height', 'Birth Weight', 5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A 95% confidence interval for the true slope extends from about 1 ounce per inch to about 1.9 ounces per inch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Could the True Slope Be 0?\n",
    "Suppose we believe that our data follow the regression model, and we fit the regression line to estimate the true line. If the regression line isn't perfectly flat, as is almost invariably the case, we will be observing some linear association in the scatter plot.\n",
    "\n",
    "But what if that observation is spurious? In other words, what if the true line was flat – that is, there was no linear relation between the two variables – and the association that we observed was just due to randomness in generating the points that form our sample?\n",
    "\n",
    "Here is a simulation that illustrates why this question arises. We will once again call the function ``draw_and_compare``, this time requiring the true line to have slope 0. Our goal is to see whether our regression line shows a slope that is not 0.\n",
    "\n",
    "Remember that the arguments to the function ``draw_and_compare`` are the slope and the intercept of the true line, and the number of points to be generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 53212 characters]\n",
      "text/plain": [
       "<Figure size 432x1152 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_and_compare(0, 10, 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the simulation a few times, keeping the slope of the true line 0 each time. You will notice that while the slope of the true line is 0, the slope of the regression line is typically not 0. The regression line sometimes slopes upwards, and sometimes downwards, each time giving us a false impression that the two variables are correlated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To decide whether or not the slope that we are seeing is real, we would like to test the following hypotheses:\n",
    "\n",
    "**Null Hypothesis.** The slope of the true line is 0. \n",
    "\n",
    "**Alternative Hypothesis.** The slope of the true line is not 0. \n",
    "\n",
    "We are well positioned to do this. Since we can construct a 95% confidence interval for the true slope, all we have to do is see whether the interval contains 0. \n",
    "\n",
    "If it doesn't, then we can reject the null hypothesis (with the 5% cutoff for the P-value). \n",
    "\n",
    "If the confidence interval for the true slope does contain 0, then we don't have enough evidence to reject the null hypothesis. Perhaps the slope that we are seeing is spurious."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use this method in an example. Suppose we try to estimate the birth weight of the baby based on the mother's age. Based on the sample, the slope of the regression line for estimating birth weight based on maternal age is positive, about 0.08 ounces per year. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08500766941582515"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slope(baby, 'Maternal Age', 'Birth Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though the slope is positive, it's pretty small. The regression line is so close to flat that it raises the question of whether the true line is flat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33352 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scatter_fit(baby, 'Maternal Age', 'Birth Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `bootstrap_slope` to estimate the slope of the true line. The calculation shows that an approximate 95% bootstrap confidence interval for the true slope has a negative left end point and a positive right end point – in other words, the interval contains 0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slope of regression line: 0.08500766941582515\n",
      "Approximate 95%-confidence interval for the true slope:\n",
      "-0.0983199011229 0.268808441978\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 13456 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bootstrap_slope(baby, 'Maternal Age', 'Birth Weight', 5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the interval contains 0, we cannot reject the null hypothesis that the slope of the true linear relation between maternal age and baby's birth weight is 0. Based on this analysis, it would be unwise to predict birth weight based on the regression model with maternal age as the predictor."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
