{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# HIDDEN \n",
    "\n",
    "galton = pd.read_csv(path_data + 'galton.csv')\n",
    "heights = galton[['midparentHeight', 'childHeight']]\n",
    "\n",
    "heights = heights.rename(columns={'midparentHeight':'MidParent', 'childHeight':'Child'})\n",
    "\n",
    "hybrid = pd.read_csv(path_data + 'hybrid.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "def standard_units(x):\n",
    "    return (x - np.mean(x))/np.std(x)\n",
    "\n",
    "def correlation(table, x, y):\n",
    "    x_in_standard_units = standard_units(table[x])\n",
    "    y_in_standard_units = standard_units(table[y])\n",
    "    return np.mean(x_in_standard_units * y_in_standard_units)\n",
    "\n",
    "def slope(table, x, y):\n",
    "    r = correlation(table, x, y)\n",
    "    return r * np.std(table[y])/np.std(table[x])\n",
    "\n",
    "def intercept(table, x, y):\n",
    "    a = slope(table, x, y)\n",
    "    return np.mean(table[y]) -  a * np.mean(table[x])\n",
    "\n",
    "def fit(table, x, y):\n",
    "    a = slope(table, x, y)\n",
    "    b = intercept(table, x, y)\n",
    "    return a * table[x] + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual Diagnostics\n",
    "Suppose a data scientist has decided to use linear regression to estimate values of one variable (called the response variable) based on another variable (called the predictor). To see how well this method of estimation performs, the data scientist must measure how far off the estimates are from the actual values. These differences are called *residuals*.\n",
    "\n",
    "$$\n",
    "\\mbox{residual} ~=~ \\mbox{observed value} ~-~ \\mbox{regression estimate}\n",
    "$$\n",
    "\n",
    "A residual is what's left over – the residue – after estimation. \n",
    "\n",
    "Residuals are the vertical distances of the points from the regression line. There is one residual for each point in the scatter plot. The residual is the difference between the observed value of $y$ and the fitted value of $y$, so for the point $(x, y)$,\n",
    "\n",
    "$$\n",
    "\\mbox{residual} ~~ = ~~ y ~-~\n",
    "\\mbox{fitted value of }y\n",
    "~~ = ~~ y ~-~\n",
    "\\mbox{height of regression line at }x\n",
    "$$\n",
    "\n",
    "The function `residual` calculates the residuals. The calculation assumes all the relevant functions we have already defined: `standard_units`, `correlation`, `slope`, `intercept`, and `fit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def residual(table, x, y):\n",
    "    return table[y] - fit(table, x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuing our example of using Galton's data to estimate the heights of adult children (the response) based on the midparent height (the predictor), let us calculate the fitted values and the residuals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MidParent</th>\n",
       "      <th>Child</th>\n",
       "      <th>Fitted Vlaue</th>\n",
       "      <th>Residual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>75.43</td>\n",
       "      <td>73.2</td>\n",
       "      <td>70.712373</td>\n",
       "      <td>2.487627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>75.43</td>\n",
       "      <td>69.2</td>\n",
       "      <td>70.712373</td>\n",
       "      <td>-1.512373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>75.43</td>\n",
       "      <td>69.0</td>\n",
       "      <td>70.712373</td>\n",
       "      <td>-1.712373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>75.43</td>\n",
       "      <td>69.0</td>\n",
       "      <td>70.712373</td>\n",
       "      <td>-1.712373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>73.66</td>\n",
       "      <td>73.5</td>\n",
       "      <td>69.584244</td>\n",
       "      <td>3.915756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>929</th>\n",
       "      <td>66.64</td>\n",
       "      <td>64.0</td>\n",
       "      <td>65.109971</td>\n",
       "      <td>-1.109971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>930</th>\n",
       "      <td>66.64</td>\n",
       "      <td>62.0</td>\n",
       "      <td>65.109971</td>\n",
       "      <td>-3.109971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>931</th>\n",
       "      <td>66.64</td>\n",
       "      <td>61.0</td>\n",
       "      <td>65.109971</td>\n",
       "      <td>-4.109971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>932</th>\n",
       "      <td>65.27</td>\n",
       "      <td>66.5</td>\n",
       "      <td>64.236786</td>\n",
       "      <td>2.263214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>933</th>\n",
       "      <td>65.27</td>\n",
       "      <td>57.0</td>\n",
       "      <td>64.236786</td>\n",
       "      <td>-7.236786</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>934 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     MidParent  Child  Fitted Vlaue  Residual\n",
       "0        75.43   73.2     70.712373  2.487627\n",
       "1        75.43   69.2     70.712373 -1.512373\n",
       "2        75.43   69.0     70.712373 -1.712373\n",
       "3        75.43   69.0     70.712373 -1.712373\n",
       "4        73.66   73.5     69.584244  3.915756\n",
       "..         ...    ...           ...       ...\n",
       "929      66.64   64.0     65.109971 -1.109971\n",
       "930      66.64   62.0     65.109971 -3.109971\n",
       "931      66.64   61.0     65.109971 -4.109971\n",
       "932      65.27   66.5     64.236786  2.263214\n",
       "933      65.27   57.0     64.236786 -7.236786\n",
       "\n",
       "[934 rows x 4 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heights['Fitted Vlaue'] = fit(heights, 'MidParent', 'Child')\n",
    "\n",
    "heights['Residual'] = residual(heights, 'MidParent', 'Child')\n",
    "\n",
    "heights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When there are so many variables to work with, it is always helpful to start with visualization. The function `scatter_fit` draws the scatter plot of the data, as well as the regression line. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scatter_fit(table, x, y):\n",
    "    \n",
    "    fig, ax = plt.subplots(figsize=(7,6))\n",
    "\n",
    "    ax.scatter(table[x], \n",
    "               table[y],  \n",
    "\n",
    "               color='darkblue')\n",
    "\n",
    "    ax.plot(table[x], fit(table, x, y), lw=4, color='gold')\n",
    "\n",
    "    x_label = x\n",
    "\n",
    "    y_label = y\n",
    "\n",
    "    y_vals = ax.get_yticks()\n",
    "\n",
    "    plt.ylabel(y_label)\n",
    "\n",
    "\n",
    "    plt.xlabel(x_label)\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43564 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scatter_fit(heights, 'MidParent', 'Child')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A *residual plot* can be drawn by plotting the residuals against the predictor variable. The function `residual_plot` does just that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def residual_plot(table, x, y):\n",
    "    \n",
    "    fig, ax = plt.subplots(figsize=(7,6))\n",
    "    \n",
    "    x_array = table[x]\n",
    "    \n",
    "    t = pd.DataFrame({x:x_array, 'residuals':residual(table, x, y)})\n",
    "\n",
    "    ax.scatter(t[x], t['residuals'], color='r')\n",
    "    \n",
    "    xlims = np.array([min(x_array), max(x_array)])\n",
    "    \n",
    "    ax.plot(xlims, np.array([0, 0]), color='darkblue', lw=4)\n",
    "\n",
    "    x_label = x\n",
    "\n",
    "    y_label = y\n",
    "\n",
    "    y_vals = ax.get_yticks()\n",
    "\n",
    "    plt.ylabel(y_label)\n",
    "\n",
    "\n",
    "    plt.xlabel(x_label)\n",
    "    \n",
    "    plt.title('Residual Plot')\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46900 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "residual_plot(heights, 'MidParent', 'Child')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The midparent heights are on the horizontal axis, as in the original scatter plot. But now the vertical axis shows the residuals. Notice that the plot appears to be centered around the horizontal line at the level 0 (shown in dark blue). Notice also that the plot shows no upward or downward trend. We will observe later that this is true of all regressions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression Diagnostics\n",
    "Residual plots help us make visual assessments of the quality of a linear regression analysis. Such assessments are called *diagnostics*. The function ``regression_diagnostic_plots`` draws the original scatter plot as well as the residual plot for ease of comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def regression_diagnostic_plots(table, x, y):\n",
    "    scatter_fit(table, x, y)\n",
    "    residual_plot(table, x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43564 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 46900 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "regression_diagnostic_plots(heights, 'MidParent', 'Child')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This residual plot indicates that linear regression was a reasonable method of estimation. Notice how the residuals are distributed fairly symmetrically above and below the horizontal line at 0, corresponding to the original scatter plot being roughly symmetrical above and below. Notice also that the vertical spread of the plot is fairly even across the most common values of the children's heights. In other words, apart from a few outlying points, the plot isn't narrower in some places and wider in others.\n",
    "\n",
    "In other words, the accuracy of the regression appears to be about the same across the observed range of the predictor variable. \n",
    "\n",
    "**The residual plot of a good regression shows no pattern. The residuals look about the same, above and below the horizontal line at 0, across the range of the predictor variable.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Nonlinearity\n",
    "Drawing the scatter plot of the data usually gives an indication of whether the relation between the two variables is non-linear. Often, however, it is easier to spot non-linearity in a residual plot than in the original scatter plot. This is usually because of the scales of the two plots: the residual plot allows us to zoom in on the errors and hence makes it easier to spot patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Dugong-dugon_wikipedia](../../images/Dugong_dugon.jpg)\n",
    "\n",
    "Our data are a [dataset](http://www.statsci.org/data/oz/dugongs.html)  on the age and length of dugongs, which are marine mammals related to manatees and sea cows (image from [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Dugong_dugon.jpg)). The data are in a table called `dugong`. Age is measured in years and length in meters. Because dugongs tend not to keep track of their birthdays, ages are estimated based on variables such as the condition of their teeth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Length</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.80</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.85</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.87</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.77</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.02</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.27</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.15</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2.26</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.35</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.47</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Length  Age\n",
       "0    1.80  1.0\n",
       "1    1.85  1.5\n",
       "2    1.87  1.5\n",
       "3    1.77  1.5\n",
       "4    2.02  2.5\n",
       "5    2.27  4.0\n",
       "6    2.15  5.0\n",
       "7    2.26  5.0\n",
       "8    2.35  7.0\n",
       "9    2.47  8.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dugong = pd.read_csv('http://www.statsci.org/data/oz/dugongs.txt')\n",
    "\n",
    "dugong[['Age','Length']] = dugong['Age\\tLength'].str.split('\\t',expand=True)\n",
    "\n",
    "dugong = dugong.drop(columns=['Age\\tLength'])\n",
    "\n",
    "length = dugong.pop('Length')\n",
    "\n",
    "dugong.insert(0, 'Length', length)\n",
    "\n",
    "dugong['Length'] = dugong['Length'].astype(float)\n",
    "\n",
    "dugong['Age'] = dugong['Age'].astype(float)\n",
    "\n",
    "dugong.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we could measure the length of a dugong, what could we say about its age? Let's examine what our data say. Here is a regression of age (the response) on length (the predictor). The correlation between the two variables is substantial, at 0.83."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8296474554905715"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correlation(dugong, 'Length', 'Age')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "High correlation notwithstanding, the plot shows a curved pattern that is much more visible in the residual plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20708 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 16280 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "regression_diagnostic_plots(dugong, 'Length', 'Age')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While you can spot the non-linearity in the original scatter, it is more clearly evident in the residual plot.\n",
    "\n",
    "At the low end of the lengths, the residuals are almost all positive; then they are almost all negative; then positive again at the high end of lengths. In other words the regression estimates have a pattern of being too high, then too low, then too high. That means it would have been better to use a curve instead of a straight line to estimate the ages.\n",
    "\n",
    "**When a residual plot shows a pattern, there may be a non-linear relation between the variables.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Heteroscedasticity\n",
    "\n",
    "*Heteroscedasticity* is a word that will surely be of interest to those who are preparing for Spelling Bees. For data scientists, its interest lies in its meaning, which is \"uneven spread\". \n",
    "\n",
    "Recall the table `hybrid` that contains data on hybrid cars in the U.S. Here is a regression of fuel efficiency on the rate of acceleration. The association is negative: cars that accelearate quickly tend to be less efficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29488 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 25860 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "regression_diagnostic_plots(hybrid, 'acceleration', 'mpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the residual plot flares out towards the low end of the accelerations. In other words, the variability in the size of the errors is greater for low values of acceleration than for high values. Uneven variation is often more easily noticed in a residual plot than in the original scatter plot.\n",
    "\n",
    "**If the residual plot shows uneven variation about the horizontal line at 0, the regression estimates are not equally accurate across the range of the predictor variable.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
