{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "def standard_units(any_numbers):\n",
    "    \"Convert any array of numbers to standard units.\"\n",
    "    return (any_numbers - np.mean(any_numbers))/np.std(any_numbers)  \n",
    "\n",
    "def correlation(t, x, y):\n",
    "    return np.mean(standard_units(t[x])*standard_units(t[y]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have explored ways to use multiple attributes to predict a categorical variable, let us return to predicting a quantitative variable. Predicting a numerical quantity is called regression, and a commonly used method to use multiple attributes for regression is called *multiple linear regression*.\n",
    "\n",
    "# Home Prices\n",
    "\n",
    "The following dataset of house prices and attributes was collected over several years for the city of Ames, Iowa. A [description of the dataset appears online](http://ww2.amstat.org/publications/jse/v19n3/decock.pdf). We will focus only a subset of the columns. We will try to predict the sale price column from the other columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2930"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_sales = pd.read_csv(path_data + 'house.csv')\n",
    "\n",
    "len(all_sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2002"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales1 = all_sales[all_sales['Bldg Type'] == '1Fam']\n",
    "sales2 = sales1[all_sales['Sale Condition'] == 'Normal']\n",
    "\n",
    "sales = sales2[['SalePrice', '1st Flr SF', '2nd Flr SF', \n",
    "    'Total Bsmt SF', 'Garage Area', \n",
    "    'Wood Deck SF', 'Open Porch SF', 'Lot Area', \n",
    "    'Year Built', 'Yr Sold']]\n",
    "\n",
    "sales = sales.sort_values(by=['SalePrice'])\n",
    "\n",
    "len(sales)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A histogram of sale prices shows a large amount of variability and a distribution that is clearly not normal. A long tail to the right contains a few houses that had very high prices. The short left tail does not contain any houses that sold for less than $35,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 22900 characters]\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "unit = '$'\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,5))\n",
    "\n",
    "ax.hist(sales['SalePrice'], bins=32, density=True, color='blue', alpha=0.8, ec='white', zorder=5)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'SalesPrice ($)' \n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Correlation\n",
    "\n",
    "No single attribute is sufficient to predict the sale price. For example, the area of first floor, measured in square feet, correlates with sale price but only explains some of its variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 66044 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(7,6))\n",
    "\n",
    "ax.scatter(sales['1st Flr SF'], \n",
    "           sales['SalePrice'],   \n",
    "           color='navy', \n",
    "          alpha=0.5)\n",
    "\n",
    "x_label = '1st Flr SF'\n",
    "\n",
    "y_label = 'SalePrice'\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6424662541030226"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correlation(sales, 'SalePrice', '1st Flr SF')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, none of the individual attributes have a correlation with sale price that is above 0.7 (except for the sale price itself)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation of SalePrice and SalePrice:\t 1.0000000000000018\n",
      "Correlation of 1st Flr SF and SalePrice:\t 0.6424662541030226\n",
      "Correlation of 2nd Flr SF and SalePrice:\t 0.35752189428008124\n",
      "Correlation of Total Bsmt SF and SalePrice:\t 0.6529786267571697\n",
      "Correlation of Garage Area and SalePrice:\t 0.6385944852520441\n",
      "Correlation of Wood Deck SF and SalePrice:\t 0.352698666195049\n",
      "Correlation of Open Porch SF and SalePrice:\t 0.33690941702637345\n",
      "Correlation of Lot Area and SalePrice:\t 0.29082345511576896\n",
      "Correlation of Year Built and SalePrice:\t 0.565164753713592\n",
      "Correlation of Yr Sold and SalePrice:\t 0.025948579080721384\n"
     ]
    }
   ],
   "source": [
    "for label in sales.columns:\n",
    "    print('Correlation of', label, 'and SalePrice:\\t', correlation(sales, label, 'SalePrice'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, combining attributes can provide higher correlation. In particular, if we sum the first floor and second floor areas, the result has a higher correlation than any single attribute alone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7821920556134886"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_copy = sales.copy()\n",
    "\n",
    "both_floors = sales_copy.iloc[:,1] + sales_copy.iloc[:,2]\n",
    "\n",
    "sales_copy['Both Floors'] = both_floors\n",
    "\n",
    "correlation(sales_copy, 'SalePrice', 'Both Floors')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This high correlation indicates that we should try to use more than one attribute to predict the sale price. In a dataset with multiple observed attributes and a single numerical value to be predicted (the sale price in this case), multiple linear regression can be an effective technique.\n",
    "\n",
    "## Multiple Linear Regression \n",
    "\n",
    "In multiple linear regression, a numerical output is predicted from numerical input attributes by multiplying each attribute value by a different slope, then summing the results. In this example, the slope for the `1st Flr SF` would represent the dollars per square foot of area on the first floor of the house that should be used in our prediction. \n",
    "\n",
    "Before we begin prediction, we split our data randomly into a training and test set of equal size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train / Test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1001 training and 1001 test instances.\n"
     ]
    }
   ],
   "source": [
    "sales_copy = sales.copy()\n",
    "train = sales_copy.sample(1001, replace=False)\n",
    "test = sales_copy.drop(train.index)\n",
    "\n",
    "print(len(train), 'training and', len(test), 'test instances.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### define function to create train, test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1001 training and 1001 test instances.\n"
     ]
    }
   ],
   "source": [
    "def split(self, k):\n",
    "    if not 1 <= k <= (len(self) - 1):\n",
    "        raise ValueError(\"Invalid value of k. k must be between 1 and the\"\n",
    "                         \"number of rows - 1\")\n",
    "\n",
    "    rows = np.random.permutation(len(self))\n",
    "\n",
    "    first = self.take(rows[:k])\n",
    "    rest = self.take(rows[k:])\n",
    "\n",
    "    return first, rest\n",
    "\n",
    "train, test = split(sales, 1001)\n",
    "\n",
    "print(len(train), 'training and', len(test), 'test instances.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative - scikit learn\n",
    "as an aside we could emplot the `scikit learn` function to determine the `train, test split`.\n",
    "\n",
    "[sklearn.model_selection.train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html#sklearn-model-selection-train-test-split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1001 training and 1001 test instances.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, test = train_test_split(sales_copy, test_size=0.5)\n",
    "\n",
    "print(len(train), 'training and', len(test), 'test instances.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The slopes in multiple regression is an array that has one slope value for each attribute in an example. Predicting the sale price involves multiplying each attribute by the slope and summing the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting sale price for:\n",
      "1st Flr SF        1244.0\n",
      "2nd Flr SF           0.0\n",
      "Total Bsmt SF     1244.0\n",
      "Garage Area        336.0\n",
      "Wood Deck SF         0.0\n",
      "Open Porch SF       40.0\n",
      "Lot Area         11988.0\n",
      "Year Built        1957.0\n",
      "Yr Sold           2007.0\n",
      "Name: 1922, dtype: float64\n",
      "\n",
      "Using slopes:\n",
      " [10.66266607 11.04000738 11.05823199  9.70462703  8.84114377  9.56944347\n",
      " 11.61432997 10.37252802 11.38349509]\n",
      "\n",
      "Result: 213042.62920044773\n"
     ]
    }
   ],
   "source": [
    "def predict(slopes, row):\n",
    "    return sum(slopes * np.array(row))\n",
    "\n",
    "example_row1 = test.drop(columns=['SalePrice'])\n",
    "example_row = example_row1.iloc[0]\n",
    "\n",
    "print('Predicting sale price for:')\n",
    "print(example_row)\n",
    "\n",
    "example_slopes = np.random.normal(10, 1, len(example_row))\n",
    "\n",
    "print('\\nUsing slopes:\\n', example_slopes)\n",
    "\n",
    "print('\\nResult:', predict(example_slopes, example_row))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is an estimated sale price, which can be compared to the actual sale price to assess whether the slopes provide accurate predictions. Since the `example_slopes` above were chosen at random, we should not expect them to provide accurate predictions at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual sale price: 150000\n",
      "Predicted sale price using random slopes: 213042.62920044773\n"
     ]
    }
   ],
   "source": [
    "print('Actual sale price:', test['SalePrice'].iloc[0])\n",
    "print('Predicted sale price using random slopes:', predict(example_slopes, example_row))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Least Squares Regression\n",
    "The next step in performing multiple regression is to define the least squares objective. We perform the prediction for each row in the training set, and then compute the root mean squared error (RMSE) of the predictions from the actual prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE of all training examples using random slopes: 114914.11566039815\n"
     ]
    }
   ],
   "source": [
    "train_prices = train.iloc[:,0]\n",
    "train1 = train.copy()\n",
    "train_attributes = train1.drop(train1.columns[0], axis=1)\n",
    "\n",
    "def rmse(slopes, attributes, prices):\n",
    "    errors = []\n",
    "    for i in np.arange(len(prices)):\n",
    "        predicted = predict(slopes, attributes.iloc[i])\n",
    "        actual = prices.iloc[i]\n",
    "        errors.append((predicted - actual) ** 2)\n",
    "    return np.mean(errors) ** 0.5\n",
    "\n",
    "def rmse_train(slopes):\n",
    "    return rmse(slopes, train_attributes, train_prices)\n",
    "\n",
    "print('RMSE of all training examples using random slopes:', rmse_train(example_slopes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we use the `minimize` function to find the slopes with the lowest RMSE. Since the function we want to minimize, `rmse_train`, takes an array instead of a number, we must pass the `array=True` argument to `minimize`. When this argument is used, `minimize` also requires an initial guess of the slopes so that it knows the dimension of the input array. Finally, to speed up optimization, we indicate that `rmse_train` is a smooth function using the `smooth=True` attribute. Computation of the best slopes may take several minutes.\n",
    "\n",
    "[scipy optimize.minimize](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import optimize\n",
    "\n",
    "def minimize(f, start=None, smooth=False, log=None, array=False, **vargs):\n",
    "    \"\"\"Minimize a function f of one or more arguments.\n",
    "    Args:\n",
    "        f: A function that takes numbers and returns a number\n",
    "        start: A starting value or list of starting values\n",
    "        smooth: Whether to assume that f is smooth and use first-order info\n",
    "        log: Logging function called on the result of optimization (e.g. print)\n",
    "        vargs: Other named arguments passed to scipy.optimize.minimize\n",
    "    Returns either:\n",
    "        (a) the minimizing argument of a one-argument function\n",
    "        (b) an array of minimizing arguments of a multi-argument function\n",
    "    \"\"\"\n",
    "    if start is None:\n",
    "        assert not array, \"Please pass starting values explicitly when array=True\"\n",
    "        arg_count = f.__code__.co_argcount\n",
    "        assert arg_count > 0, \"Please pass starting values explicitly for variadic functions\"\n",
    "        start = [0] * arg_count\n",
    "    if not hasattr(start, '__len__'):\n",
    "        start = [start]\n",
    "\n",
    "    if array:\n",
    "        objective = f\n",
    "    else:\n",
    "        @functools.wraps(f)\n",
    "        def objective(args):\n",
    "            return f(*args)\n",
    "\n",
    "    if not smooth and 'method' not in vargs:\n",
    "        vargs['method'] = 'Powell'\n",
    "    result = optimize.minimize(objective, start, **vargs)\n",
    "    if log is not None:\n",
    "        log(result)\n",
    "    if len(start) == 1:\n",
    "        return result.x.item(0)\n",
    "    else:\n",
    "        return result.x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best slopes for the training set:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>1st Flr SF</th>\n",
       "      <th>2nd Flr SF</th>\n",
       "      <th>Total Bsmt SF</th>\n",
       "      <th>Garage Area</th>\n",
       "      <th>Wood Deck SF</th>\n",
       "      <th>Open Porch SF</th>\n",
       "      <th>Lot Area</th>\n",
       "      <th>Year Built</th>\n",
       "      <th>Yr Sold</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>78.980021</td>\n",
       "      <td>75.902626</td>\n",
       "      <td>48.804937</td>\n",
       "      <td>48.98349</td>\n",
       "      <td>45.186775</td>\n",
       "      <td>7.047521</td>\n",
       "      <td>0.435624</td>\n",
       "      <td>538.362254</td>\n",
       "      <td>-537.790128</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  1st Flr SF 2nd Flr SF Total Bsmt SF Garage Area Wood Deck SF Open Porch SF  \\\n",
       "0  78.980021  75.902626     48.804937    48.98349    45.186775      7.047521   \n",
       "\n",
       "   Lot Area  Year Built     Yr Sold  \n",
       "0  0.435624  538.362254 -537.790128  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_slopes = minimize(rmse_train, start=example_slopes, smooth=True, array=True)\n",
    " \n",
    "train_df = pd.DataFrame(columns=[train_attributes.columns])\n",
    "\n",
    "train_df.loc[0] = best_slopes\n",
    "\n",
    "print('The best slopes for the training set:')\n",
    "\n",
    "train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE of all training examples using the best slopes: 34594.95413398824\n"
     ]
    }
   ],
   "source": [
    "print('RMSE of all training examples using the best slopes:', rmse_train(best_slopes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting Multiple Regression \n",
    "Let's interpret these results. The best slopes give us a method for estimating the price of a house from its attributes. A square foot of area on the first floor is worth about \\$75 (the first slope), while one on the second floor is worth about \\\\$70 (the second slope). The final negative value describes the market: prices in later years were lower on average.\n",
    "\n",
    "The RMSE of around \\\\$30,000 means that our best linear prediction of the sale price based on all of the attributes is off by around \\\\$30,000 on the training set, on average.  We find a similar error when predicting prices on the test set, which indicates that our prediction method will generalize to other samples from the same population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1st Flr SF</th>\n",
       "      <th>2nd Flr SF</th>\n",
       "      <th>Total Bsmt SF</th>\n",
       "      <th>Garage Area</th>\n",
       "      <th>Wood Deck SF</th>\n",
       "      <th>Open Porch SF</th>\n",
       "      <th>Lot Area</th>\n",
       "      <th>Year Built</th>\n",
       "      <th>Yr Sold</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1922</th>\n",
       "      <td>1244</td>\n",
       "      <td>0</td>\n",
       "      <td>1244.0</td>\n",
       "      <td>336.0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>11988</td>\n",
       "      <td>1957</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>832</td>\n",
       "      <td>629</td>\n",
       "      <td>832.0</td>\n",
       "      <td>384.0</td>\n",
       "      <td>0</td>\n",
       "      <td>204</td>\n",
       "      <td>10800</td>\n",
       "      <td>1949</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      1st Flr SF  2nd Flr SF  Total Bsmt SF  Garage Area  Wood Deck SF  \\\n",
       "1922        1244           0         1244.0        336.0             0   \n",
       "685          832         629          832.0        384.0             0   \n",
       "\n",
       "      Open Porch SF  Lot Area  Year Built  Yr Sold  \n",
       "1922             40     11988        1957     2007  \n",
       "685             204     10800        1949     2009  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_prices = test.iloc[:,0]\n",
    "\n",
    "test_attributes = test.drop(test.columns[0], axis=1)\n",
    "test_attributes.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set RMSE for multiple linear regression: 27254.40534406947\n"
     ]
    }
   ],
   "source": [
    "test_prices = test.iloc[:,0]\n",
    "test_attributes = test.drop(test.columns[0], axis=1)\n",
    "\n",
    "def rmse_test(slopes):\n",
    "    return rmse(slopes, test_attributes, test_prices)\n",
    "\n",
    "rmse_linear = rmse_test(best_slopes)\n",
    "print('Test set RMSE for multiple linear regression:', rmse_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the predictions were perfect, then a scatter plot of the predicted and actual values would be a straight line with slope 1. We see that most dots fall near that line, but there is some error in the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56244 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def fit(row):\n",
    "    return sum(best_slopes * np.array(row))\n",
    "\n",
    "test['Fitted'] = test_attributes.apply(fit, axis=1)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7,6))\n",
    "\n",
    "ax.scatter(test['Fitted'], \n",
    "           test['SalePrice'],   \n",
    "           color='navy', \n",
    "          alpha=0.5)\n",
    "\n",
    "x_label = 'Fitted'\n",
    "\n",
    "y_label = 'SalePrice'\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.plot([0, 5e5], [0, 5e5])\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A residual plot for multiple regression typically compares the errors (residuals) to the actual values of the predicted variable. We see in the residual plot below that we have systematically underestimated the value of expensive houses, shown by the many positive residual values on the right side of the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 59296 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test['Residual'] = test_prices-test_attributes.apply(fit, axis=1)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7,6))\n",
    "\n",
    "ax.scatter(test['SalePrice'], \n",
    "           test['Residual'],   \n",
    "           color='navy', \n",
    "          alpha=0.5)\n",
    "\n",
    "x_label = 'SalePrice'\n",
    "\n",
    "y_label = 'SalePrice'\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.plot([0, 7e5], [0, 0])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with simple linear regression, interpreting the result of a predictor is at least as important as making predictions. There are many lessons about interpreting multiple regression that are not included in this textbook. A natural next step after completing this text would be to study linear modeling and regression in further depth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nearest Neighbors for Regression\n",
    "Another approach to predicting the sale price of a house is to use the price of similar houses. This *nearest neighbor* approach is very similar to our classifier. To speed up computation, we will only use the attributes that had the highest correlation with the sale price in our original analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>1st Flr SF</th>\n",
       "      <th>2nd Flr SF</th>\n",
       "      <th>Total Bsmt SF</th>\n",
       "      <th>Garage Area</th>\n",
       "      <th>Year Built</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1137</th>\n",
       "      <td>173000</td>\n",
       "      <td>798</td>\n",
       "      <td>842</td>\n",
       "      <td>798.0</td>\n",
       "      <td>520.0</td>\n",
       "      <td>2004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>437</th>\n",
       "      <td>255500</td>\n",
       "      <td>1717</td>\n",
       "      <td>0</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>908.0</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2598</th>\n",
       "      <td>68000</td>\n",
       "      <td>715</td>\n",
       "      <td>0</td>\n",
       "      <td>715.0</td>\n",
       "      <td>660.0</td>\n",
       "      <td>1929</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      SalePrice  1st Flr SF  2nd Flr SF  Total Bsmt SF  Garage Area  \\\n",
       "1137     173000         798         842          798.0        520.0   \n",
       "437      255500        1717           0         1709.0        908.0   \n",
       "2598      68000         715           0          715.0        660.0   \n",
       "\n",
       "      Year Built  \n",
       "1137        2004  \n",
       "437         2006  \n",
       "2598        1929  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_nn = train.iloc[:,[0, 1, 2, 3, 4, 8]]\n",
    "test_nn = test.iloc[:,[0, 1, 2, 3, 4, 8]]\n",
    "train_nn.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The computation of closest neighbors is identical to a nearest-neighbor classifier. In this case, we will exclude the `'SalePrice'` rather than the `'Class'` column from the distance computation. The five nearest neighbors of the first test row are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>1st Flr SF</th>\n",
       "      <th>2nd Flr SF</th>\n",
       "      <th>Total Bsmt SF</th>\n",
       "      <th>Garage Area</th>\n",
       "      <th>Year Built</th>\n",
       "      <th>Distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1929</th>\n",
       "      <td>129500</td>\n",
       "      <td>1216</td>\n",
       "      <td>0</td>\n",
       "      <td>1216.0</td>\n",
       "      <td>336.0</td>\n",
       "      <td>1955</td>\n",
       "      <td>39.648455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1272</th>\n",
       "      <td>155000</td>\n",
       "      <td>1269</td>\n",
       "      <td>0</td>\n",
       "      <td>1269.0</td>\n",
       "      <td>308.0</td>\n",
       "      <td>1960</td>\n",
       "      <td>45.199558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1875</th>\n",
       "      <td>143000</td>\n",
       "      <td>1214</td>\n",
       "      <td>0</td>\n",
       "      <td>1214.0</td>\n",
       "      <td>318.0</td>\n",
       "      <td>1967</td>\n",
       "      <td>47.159304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1944</th>\n",
       "      <td>156000</td>\n",
       "      <td>1216</td>\n",
       "      <td>0</td>\n",
       "      <td>1216.0</td>\n",
       "      <td>371.0</td>\n",
       "      <td>1953</td>\n",
       "      <td>53.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>108538</td>\n",
       "      <td>1206</td>\n",
       "      <td>0</td>\n",
       "      <td>1206.0</td>\n",
       "      <td>312.0</td>\n",
       "      <td>1959</td>\n",
       "      <td>58.889727</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      SalePrice  1st Flr SF  2nd Flr SF  Total Bsmt SF  Garage Area  \\\n",
       "1929     129500        1216           0         1216.0        336.0   \n",
       "1272     155000        1269           0         1269.0        308.0   \n",
       "1875     143000        1214           0         1214.0        318.0   \n",
       "1944     156000        1216           0         1216.0        371.0   \n",
       "146      108538        1206           0         1206.0        312.0   \n",
       "\n",
       "      Year Built   Distance  \n",
       "1929        1955  39.648455  \n",
       "1272        1960  45.199558  \n",
       "1875        1967  47.159304  \n",
       "1944        1953  53.000000  \n",
       "146         1959  58.889727  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def distance(pt1, pt2):\n",
    "    \"\"\"The distance between two points, represented as arrays.\"\"\"\n",
    "    return np.sqrt(np.sum((pt1 - pt2) ** 2))\n",
    "    \n",
    "def row_distance(row1, row2):\n",
    "    \"\"\"The distance between two rows of a table.\"\"\"\n",
    "    return distance(np.array(row1), np.array(row2))\n",
    "\n",
    "def distances(training, example, output):\n",
    "    \"\"\"Compute the distance from example for each row in training.\"\"\"\n",
    "    dists = []\n",
    "    attributes = training.drop(columns=[output])\n",
    "    for row in range(len(attributes)):\n",
    "        dists.append(row_distance(attributes.iloc[row], example))\n",
    "    training['Distance'] = dists\n",
    "    #print(training)\n",
    "    return training\n",
    "\n",
    "def closest(training, example, k, output):\n",
    "    \"\"\"Return a table of the k closest neighbors to example.\"\"\"\n",
    "    distance = distances(training, example, output).sort_values(by=['Distance']).take(np.arange(k))\n",
    "    return distance\n",
    "\n",
    "train_nn_A = train_nn.copy()\n",
    "example_nn_row = test_nn.drop(test_nn.columns[0], axis=1).iloc[0]\n",
    "closest(train_nn_A, example_nn_row, 5, 'SalePrice')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One simple method for predicting the price is to average the prices of the nearest neighbors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1st Flr SF       1244.0\n",
       "2nd Flr SF          0.0\n",
       "Total Bsmt SF    1244.0\n",
       "Garage Area       336.0\n",
       "Year Built       1957.0\n",
       "Name: 1922, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_nn_row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "138407.6"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def predict_nn(example):\n",
    "    \"\"\"Return the majority class among the k nearest neighbors.\"\"\"\n",
    "    train_nn_B = train_nn.copy()\n",
    "    \n",
    "    col_sales_price = closest(train_nn_B, example, 5, 'SalePrice')\n",
    "    return np.average(col_sales_price['SalePrice'])\n",
    "\n",
    "predict_nn(example_nn_row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can inspect whether our prediction is close to the true sale price for our one test example. Looks reasonable!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual sale price: 150000\n",
      "Predicted sale price using nearest neighbors: 138407.6\n"
     ]
    }
   ],
   "source": [
    "print('Actual sale price:', test_nn['SalePrice'].iloc[0])\n",
    "\n",
    "print('Predicted sale price using nearest neighbors:', predict_nn(example_nn_row))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "To evaluate the performance of this approach for the whole test set, we apply `predict_nn` to each test example, then compute the root mean squared error of the predictions. Computation of the predictions may take several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "138407.6"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def predict_nn(example):\n",
    "    \"\"\"Return the majority class among the k nearest neighbors.\"\"\"\n",
    "    train_nn_B = train_nn.copy()\n",
    "    \n",
    "    col_sales_price = closest(train_nn_B, example, 5, 'SalePrice')\n",
    "    return np.average(col_sales_price['SalePrice'])\n",
    "\n",
    "predict_nn(example_nn_row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set RMSE for multiple linear regression:  27254.40534406947\n",
      "Test set RMSE for nearest neighbor regression: 28324.30879280499\n"
     ]
    }
   ],
   "source": [
    "test_nn_C = test_nn.copy()\n",
    "\n",
    "test_nn_drop = test_nn_C.drop(columns=['SalePrice'])\n",
    "\n",
    "nn_test_predictions = test_nn_drop.apply(predict_nn, axis=1)\n",
    "\n",
    "rmse_nn = np.mean((test_prices - nn_test_predictions) ** 2) ** 0.5\n",
    "\n",
    "print('Test set RMSE for multiple linear regression: ', rmse_linear)\n",
    "print('Test set RMSE for nearest neighbor regression:', rmse_nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For these data, the errors of the two techniques are quite similar! For different data sets, one technique might outperform another. By computing the RMSE of both techniques on the same data, we can compare methods fairly. One note of caution: the difference in performance might not be due to the technique at all; it might be due to the random variation due to sampling the training and test sets in the first place.\n",
    "\n",
    "Finally, we can draw a residual plot for these predictions. We still underestimate the prices of the most expensive houses, but the bias does not appear to be as systematic. However, fewer residuals are very close to zero, indicating that fewer prices were predicted with very high accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56216 characters]\n",
      "text/plain": [
       "<Figure size 504x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test['Residual'] = test_prices-nn_test_predictions\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7,6))\n",
    "\n",
    "ax.scatter(test['SalePrice'], \n",
    "           test['Residual'],   \n",
    "           color='navy', \n",
    "          alpha=0.5)\n",
    "\n",
    "x_label = 'SalePrice'\n",
    "\n",
    "y_label = 'SalePrice'\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "plt.plot([0, 7e5], [0, 0])\n",
    "\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
