{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import math\n",
    "from scipy import stats\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The SD and the Normal Curve\n",
    "\n",
    "We know that the mean is the balance point of the histogram. Unlike the mean, the SD is usually not easy to identify by looking at the histogram. \n",
    "\n",
    "However, there is one shape of distribution for which the SD is almost as clearly identifiable as the mean. That is the bell-shaped disribution. This section examines that shape, as it appears frequently in probability histograms and also in some histograms of data.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Roughly Bell-Shaped Histogram of Data\n",
    "Let us look at the distribution of heights of mothers in our familiar sample of 1,174 mother-newborn pairs. The mothers' heights have a mean of 64 inches and an SD of 2.5 inches. Unlike the heights of the basketball players, the mothers' heights are distributed fairly symmetrically about the mean in a bell-shaped curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "baby = pd.read_csv(path_data + 'baby.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heights = baby['Maternal Height']\n",
    "mean_height = np.round(np.mean(heights), 1)\n",
    "mean_height"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.5"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sd_height = np.round(np.std(heights), 1)\n",
    "sd_height"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19252 characters]\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "positions = np.arange(-3, 3.1, 1)*sd_height + mean_height\n",
    "\n",
    "unit = 'inch'\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,5))\n",
    "\n",
    "ax.hist(baby['Maternal Height'], bins=np.arange(55.5, 72.5, 1), \n",
    "        density=True, \n",
    "        color='blue', \n",
    "        alpha=0.8, \n",
    "        ec='white', \n",
    "        zorder=5)\n",
    "\n",
    "y_vals = ax.get_yticks()\n",
    "\n",
    "y_label = 'Percent per ' + (unit if unit else 'unit')\n",
    "\n",
    "x_label = 'Maternal Height (' + (unit if unit else 'unit') +')'\n",
    "\n",
    "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n",
    "\n",
    "plt.xticks(positions)\n",
    "\n",
    "plt.ylabel(y_label)\n",
    "\n",
    "plt.xlabel(x_label)\n",
    "\n",
    "plt.title('');\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last two lines of code in the cell above change the labeling of the horizontal axis. Now, the labels correspond to \"average $\\pm$ $z$ SDs\" for $z = 0, \\pm 1, \\pm 2$, and $\\pm 3$. Because of the shape of the distribution, the \"center\" has an unambiguous meaning and is clearly visible at 64."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to Spot the SD on a Bell Shaped Curve\n",
    "To see how the SD is related to the curve, start at the top of the curve and look towards the right. Notice that there is a place where the curve changes from looking like an \"upside-down cup\" to a \"right-way-up cup\"; formally, the curve has a point of inflection at which the curve changes from concavity-up to concavity-down depending upon direction of travel on curve. That point is one SD above average. It is the point $z=1$, which is \"average plus 1 SD\" = 66.5 inches.\n",
    "\n",
    "Symmetrically on the left-hand side of the mean, the point of inflection is at $z=-1$, that is, \"average minus 1 SD\" = 61.5 inches. \n",
    "\n",
    "In general, **for bell-shaped distributions, the SD is the distance between the mean and the points of inflection on either side.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The standard normal curve\n",
    "All the bell-shaped histograms that we have seen look essentially the same apart from the labels on the axes. Indeed, there is really just one basic curve from which all of these curves can be drawn just by relabeling the axes appropriately. \n",
    "\n",
    "To draw that basic curve, we will use the units into which we can convert every list: standard units. The resulting curve is therefore called the *standard normal curve*. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard normal curve has an impressive equation. But for now, it is best to think of it as a smoothed outline of a histogram of a variable that has been measured in standard units and has a bell-shaped distribution.\n",
    "\n",
    "$$\n",
    "\\phi(z) = {\\frac{1}{\\sqrt{2 \\pi}}} e^{-\\frac{1}{2}z^2}, ~~ -\\infty < z < \\infty\n",
    "$$  \n",
    "\n",
    "The graph below is the output of the function *'plot_normal_cdf'*, this function utilises the probability density function `pdf` method of the `stats` function from the `scipy` module.  \n",
    "\n",
    "[Scipy stats](https://docs.scipy.org/doc/scipy/reference/stats.html)\n",
    "\n",
    "[Scipy stats norm](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html#scipy.stats.norm)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "# The standard normal curve, a rather lengthy function\n",
    "# as with all functions - only the function name and parameter are called \n",
    "\n",
    "def plot_normal_cdf(rbound=None, lbound=None, mean=0, sd=1):\n",
    "    \"\"\"Plots a normal curve with specified parameters and area below curve shaded\n",
    "    between ``lbound`` and ``rbound``.\n",
    "    Args:\n",
    "        ``rbound`` (numeric): right boundary of shaded region\n",
    "        ``lbound`` (numeric): left boundary of shaded region; by default is negative infinity\n",
    "        ``mean`` (numeric): mean/expectation of normal distribution\n",
    "        ``sd`` (numeric): standard deviation of normal distribution\n",
    "    \"\"\"\n",
    "    shade = rbound is not None or lbound is not None\n",
    "    shade_left = rbound is not None and lbound is not None\n",
    "    inf = 3.5 * sd\n",
    "    step = 0.1\n",
    "    rlabel = rbound\n",
    "    llabel = lbound\n",
    "    if rbound is None:\n",
    "        rbound = inf + mean\n",
    "        rlabel = r\"$\\infty$\"\n",
    "    if lbound is None:\n",
    "        lbound = -inf + mean\n",
    "        llabel = r\"-$\\infty$\"\n",
    "    pdf_range = np.arange(-inf + mean, inf + mean, step)\n",
    "    plt.plot(pdf_range, stats.norm.pdf(pdf_range, loc=mean, scale=sd), color='k', lw=1)\n",
    "    cdf_range = np.arange(lbound, rbound + step, step)\n",
    "    if shade:\n",
    "        plt.fill_between(cdf_range, stats.norm.pdf(cdf_range, loc=mean, scale=sd), color='gold')\n",
    "    if shade_left:\n",
    "        cdf_range = np.arange(-inf+mean, lbound + step, step)\n",
    "        plt.fill_between(cdf_range, stats.norm.pdf(cdf_range, loc=mean, scale=sd), color='darkblue')\n",
    "    plt.ylim(0, stats.norm.pdf(0, loc=0, scale=sd) * 1.25)\n",
    "    plt.xlabel('z')\n",
    "    plt.ylabel(r'$\\phi$(z)', rotation=90)\n",
    "    plt.title(r\"Normal Curve ~ ($\\mu$ = {0}, $\\sigma$ = {1}) \"\n",
    "              \"{2} < z < {3}\".format(mean, sd, llabel, rlabel), fontsize=16)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 22948 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_normal_cdf()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 'Mechanics' behind the function\n",
    "For your interest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27156 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#pdf - probability density function\n",
    "\n",
    "#cdf - cumulative distribution function\n",
    "\n",
    "#ppf - percent point function (inverse of CDF)\n",
    "\n",
    "from scipy.stats import norm\n",
    "\n",
    "fig, ax = plt.subplots(1, 1)\n",
    "\n",
    "#calculate a few first moments\n",
    "\n",
    "mean, var, skew, kurt = norm.stats(moments='mvsk')\n",
    "\n",
    "#display the pdf\n",
    "\n",
    "x = np.linspace(norm.ppf(0.01), norm.ppf(0.99), 100)\n",
    "\n",
    "ax.plot(x, norm.pdf(x), 'r-', lw=5, alpha=0.6, label='norm pdf')\n",
    "\n",
    "ax.legend(loc='best', frameon=False)\n",
    "\n",
    "rv = norm()\n",
    "\n",
    "ax.plot(x, rv.pdf(x), 'k-', lw=2, label='frozen pdf')\n",
    "\n",
    "#Check accuracy of cdf and ppf:\n",
    "\n",
    "vals = norm.ppf([0.001, 0.5, 0.999])\n",
    "\n",
    "np.allclose([0.001, 0.5, 0.999], norm.cdf(vals))\n",
    "\n",
    "#Generate random numbers:\n",
    "\n",
    "r = norm.rvs(size=1000)\n",
    "\n",
    "#Compare the histogram:\n",
    "\n",
    "ax.hist(r, density=True, histtype='stepfilled', alpha=0.2)\n",
    "\n",
    "ax.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\", frameon=False)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always when you examine a new histogram, start by looking at the horizontal axis. On the horizontal axis of the standard normal curve, the values are standard units. \n",
    "\n",
    "Here are some properties of the curve. Some are apparent by observation, and others require a considerable amount of mathematics to establish.\n",
    "\n",
    "- The total area under the curve is 1. So you can think of it as a histogram drawn to the density scale.\n",
    "\n",
    "- The curve is symmetric about 0. So if a variable has this distribution, its mean and median are both 0.\n",
    "\n",
    "- The points of inflection of the curve are at -1 and +1. \n",
    "\n",
    "- If a variable has this distribution, its SD is 1. The normal curve is one of the very few distributions that has an SD so clearly identifiable on the histogram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are thinking of the curve as a smoothed histogram, we will want to represent proportions of the total amount of data by areas under the curve. \n",
    "\n",
    "Areas under smooth curves are often found by calculus, using a method called integration. It is a fact of mathematics, however, that the standard normal curve cannot be integrated in any of the usual ways of calculus. \n",
    "\n",
    "Therefore, areas under the curve have to be approximated. That is why almost all statistics textbooks carry tables of areas under the normal curve. It is also why all statistical systems, including a module of Python, include methods that provide excellent approximations to those areas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The standard normal \"cdf\"\n",
    "The fundamental function for finding areas under the normal curve is `stats.norm.cdf`. It takes a numerical argument and returns all the area under the curve to the left of that number. Formally, it is called the \"cumulative distribution function\" of the standard normal curve. That rather unwieldy mouthful is abbreviated as cdf.\n",
    "\n",
    "Let us use this function to find the area to the left of $z=1$ under the standard normal curve. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23192 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Area under the standard normal curve, below 1\n",
    "\n",
    "plot_normal_cdf(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The numerical value of the shaded area can be found by calling `stats.norm.cdf`.\n",
    "\n",
    "[Scipy.stats.norm.cdf( )](https://docs.scipy.org/doc/scipy/reference/tutorial/stats.html#common-methods)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8413447460685429"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.norm.cdf(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's about 84%. We can now use the symmetry of the curve and the fact that the total area under the curve is 1 to find other areas. \n",
    "\n",
    "The area to the right of $z=1$ is about 100% - 84% = 16%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 22904 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Area under the standard normal curve, above 1\n",
    "\n",
    "plot_normal_cdf(lbound=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15865525393145707"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - stats.norm.cdf(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Or** we pass the negative equivalent value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15865525393145707"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.norm.cdf(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The area between $z=-1$ and $z=1$ can be computed in several different ways.  It is the gold area under the curve below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 22868 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Area under the standard normal curve, between -1 and 1\n",
    "\n",
    "plot_normal_cdf(1, lbound=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we could calculate the area as \"100% - two equal tails\", which works out to roughly 100% - 2x16% = 68%.\n",
    "\n",
    "Or we could note that the area between $z=1$ and $z=-1$ is equal to all the area to the left of $z=1$, minus all the area to the left of $z=-1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6826894921370859"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.norm.cdf(1) - stats.norm.cdf(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By a similar calculation, we see that the area between $-2$ and 2 is about 95%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23152 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Area under the standard normal curve, between -2 and 2\n",
    "\n",
    "plot_normal_cdf(2, lbound=-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9544997361036416"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.norm.cdf(2) - stats.norm.cdf(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words, if a histogram is roughly bell shaped, the proportion of data in the range \"average $\\pm$ 2 SDs\" is about 95%. \n",
    "\n",
    "That is quite a bit more than Chebychev's lower bound of 75%. Chebychev's bound is weaker because it has to work for all distributions. If we know that a distribution is normal, we have good approximations to the proportions, not just bounds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The table below compares what we know about all distributions and about normal distributions. Notice that when $z=1$, Chebychev's bound is correct but not illuminating.\n",
    "\n",
    "| Percent in Range   | All Distributions: Bound   | Normal Distribution: Approximation |\n",
    "| :---------------   | :---------------- --| :-------------------|\n",
    "|average $\\pm$ 1 SD  | at least 0%         | about 68%           |\n",
    "|average $\\pm$ 2 SDs | at least 75%        | about 95%           |\n",
    "|average $\\pm$ 3 SDs | at least 88.888...% | about 99.73%        |"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
