{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross-Classifying by More than One Variable\n",
    "\n",
    "When individuals have multiple features, there are many different ways to classify them. For example, if we have a population of college students for each of whom we have recorded a major and the number of years in college, then the students could be classified by major, or by year, or by a combination of major and year.\n",
    "\n",
    "The `group` method also allows us to classify individuals according to multiple variables. This is called *cross-classifying*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two Variables: Counting the Number in Each Paired Category\n",
    "The table `more_cones` records the flavor, color, and price of six ice cream cones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flavor</th>\n",
       "      <th>Color</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>strawberry</td>\n",
       "      <td>pink</td>\n",
       "      <td>3.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>light brown</td>\n",
       "      <td>4.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>dark brown</td>\n",
       "      <td>5.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>strawberry</td>\n",
       "      <td>pink</td>\n",
       "      <td>5.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>dark brown</td>\n",
       "      <td>5.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bubblegum</td>\n",
       "      <td>pink</td>\n",
       "      <td>4.75</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flavor        Color  Price\n",
       "0  strawberry         pink   3.55\n",
       "1   chocolate  light brown   4.75\n",
       "2   chocolate   dark brown   5.25\n",
       "3  strawberry         pink   5.25\n",
       "4   chocolate   dark brown   5.25\n",
       "5   bubblegum         pink   4.75"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones = pd.DataFrame({\n",
    "    'Flavor':np.array(['strawberry', 'chocolate', 'chocolate', 'strawberry', 'chocolate', 'bubblegum']),\n",
    "    'Color':np.array(['pink', 'light brown', 'dark brown', 'pink', 'dark brown', 'pink']),\n",
    "    'Price':np.array([3.55, 4.75, 5.25, 5.25, 5.25, 4.75])}\n",
    ")\n",
    "\n",
    "more_cones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know how to use `group` to count the number of cones of each flavor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Color</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flavor</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bubblegum</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chocolate</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>strawberry</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Color  Price\n",
       "Flavor                  \n",
       "bubblegum       1      1\n",
       "chocolate       3      3\n",
       "strawberry      2      2"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby('Flavor').count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Or**, using the aggregation framework .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flavor</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bubblegum</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chocolate</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>strawberry</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            count\n",
       "Flavor           \n",
       "bubblegum       1\n",
       "chocolate       3\n",
       "strawberry      2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby(['Flavor']).Flavor.agg('count').to_frame('count')#.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the aggregation and resetting the index give .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flavor</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bubblegum</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>strawberry</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flavor  count\n",
       "0   bubblegum      1\n",
       "1   chocolate      3\n",
       "2  strawberry      2"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby(['Flavor']).Flavor.agg('count').to_frame('count').reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But now each cone has a color as well. To classify the cones by both flavor and color, we will pass a list of labels as an argument to `group`. The resulting table has one row for every *unique combination* of values that appear together in the grouped columns. As before, a single argument (a list, in this case, but an array would work too) gives row counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Color</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flavor</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bubblegum</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chocolate</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>strawberry</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Color  Price\n",
       "Flavor                  \n",
       "bubblegum       1      1\n",
       "chocolate       3      3\n",
       "strawberry      2      2"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby('Flavor').count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although there are six cones, there are only four unique combinations of flavor and color. Two of the cones were dark brown chocolate, and two pink strawberry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flavor</th>\n",
       "      <th>Color</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bubblegum</th>\n",
       "      <th>pink</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">chocolate</th>\n",
       "      <th>dark brown</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>light brown</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>strawberry</th>\n",
       "      <th>pink</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Price\n",
       "Flavor     Color             \n",
       "bubblegum  pink             1\n",
       "chocolate  dark brown       2\n",
       "           light brown      1\n",
       "strawberry pink             2"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby(['Flavor', 'Color']).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Or** .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flavor</th>\n",
       "      <th>Color</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bubblegum</th>\n",
       "      <th>pink</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">chocolate</th>\n",
       "      <th>dark brown</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>light brown</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>strawberry</th>\n",
       "      <th>pink</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        count\n",
       "Flavor     Color             \n",
       "bubblegum  pink             1\n",
       "chocolate  dark brown       2\n",
       "           light brown      1\n",
       "strawberry pink             2"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby(['Flavor', 'Color']).Flavor.agg('count').to_frame('count')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Or** .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flavor</th>\n",
       "      <th>Color</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bubblegum</td>\n",
       "      <td>pink</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>dark brown</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>light brown</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>strawberry</td>\n",
       "      <td>pink</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flavor        Color  count\n",
       "0   bubblegum         pink      1\n",
       "1   chocolate   dark brown      2\n",
       "2   chocolate  light brown      1\n",
       "3  strawberry         pink      2"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby(['Flavor', 'Color']).Flavor.agg('count').to_frame('count').reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two Variables: Finding a Characteristic of Each Paired Category\n",
    "A second argument aggregates all other columns that are not in the list of grouped columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flavor</th>\n",
       "      <th>Color</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bubblegum</th>\n",
       "      <th>pink</th>\n",
       "      <td>4.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">chocolate</th>\n",
       "      <th>dark brown</th>\n",
       "      <td>10.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>light brown</th>\n",
       "      <td>4.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>strawberry</th>\n",
       "      <th>pink</th>\n",
       "      <td>8.80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Price\n",
       "Flavor     Color             \n",
       "bubblegum  pink          4.75\n",
       "chocolate  dark brown   10.50\n",
       "           light brown   4.75\n",
       "strawberry pink          8.80"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby(['Flavor', 'Color']).sum()\n",
    "\n",
    "#Or ..\n",
    "\n",
    "#more_cones.groupby(['Flavor', 'Color']).Price.agg('sum').to_frame('Price sum').reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Three or More Variables.** You can use `group` to classify rows by three or more categorical variables. Just include them all in the list that is the first argument. But cross-classifying by multiple variables can become complex, as the number of distinct combinations of categories can be quite large. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pivot Tables: Rearranging the Output of `group`\n",
    "Many uses of cross-classification involve just two categorical variables, like `Flavor` and `Color` in the example above. In these cases it is possible to display the results of the classification in a different kind of table, called a *pivot table*. Pivot tables, also known as *contingency tables*, make it easier to work with data that have been classified according to two variables.\n",
    "\n",
    "Recall the use of `group` to count the number of cones in each paired category of flavor and color:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flavor</th>\n",
       "      <th>Color</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bubblegum</td>\n",
       "      <td>pink</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>dark brown</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>chocolate</td>\n",
       "      <td>light brown</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>strawberry</td>\n",
       "      <td>pink</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flavor        Color  count\n",
       "0   bubblegum         pink      1\n",
       "1   chocolate   dark brown      2\n",
       "2   chocolate  light brown      1\n",
       "3  strawberry         pink      2"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones_pivot = more_cones.groupby(['Flavor', 'Color']).Color.agg('count').to_frame('count').reset_index()\n",
    "\n",
    "more_cones_pivot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same data can be displayed differenly using the Table method `pivot`. Ignore the code for a moment, and just examine the table of outcomes.\n",
    "\n",
    "[Pandas pivot](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Flavor</th>\n",
       "      <th>bubblegum</th>\n",
       "      <th>chocolate</th>\n",
       "      <th>strawberry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Color</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dark brown</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>light brown</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pink</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Flavor       bubblegum  chocolate  strawberry\n",
       "Color                                        \n",
       "dark brown         0.0        2.0         0.0\n",
       "light brown        0.0        1.0         0.0\n",
       "pink               1.0        0.0         2.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#more_cones_pivot.pivot(index='Color', columns='Flavor', values='count').fillna(0).reset_index()\n",
    "\n",
    "#Or\n",
    "\n",
    "more_cones_pivot.pivot(index='Color', columns='Flavor', values='count').fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how this df displays all nine possible pairs of flavor and color, including pairs like \"dark brown bubblegum\" that don't exist in our data. Notice also that the count in each pair appears in the body of the DataFrame: to find the number of light brown chocolate cones, run your eye along the row `light brown` until it meets the column `chocolate`.\n",
    "\n",
    "The `groupby` method takes a list of two labels because it is flexible: it could take one or three or more. On the other hand, `pivot` always takes **two** column labels, one to determine the columns and one to determine the rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`pivot`** \n",
    "\n",
    "The `pivot` method is closely related to the `groupby` method: it groups together rows that share a combination of values. It differs from `groupby` because it organizes the resulting values in a grid. The first argument to `pivot` is the label of a column that contains the values that will be used to form new columns in the result. The second argument is the label of a column used for the rows. The result gives the count of all rows of the original table that share the combination of column and row values.\n",
    "\n",
    "Like `groupby`, `pivot` can be used with additional arguments to find characteristics of each paired category. An optional third argument called `values` indicates a column of values that will replace the counts in each cell of the grid.\n",
    "\n",
    "All of these values will not be displayed, however; the fourth argument `aggfunc` indicates how to collect them all into one aggregated value to be displayed in the cell.\n",
    "\n",
    "An example will help clarify this. Here is `pivot` being used to find the total price of the cones in each cell.  \n",
    "\n",
    "pd.pivot_table(*dataframe*, values='data upon which aggfunc acts', index= ,columns= , aggfunc=*aggregation function*).fillna(0)*replace NaN with 0*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Flavor</th>\n",
       "      <th>bubblegum</th>\n",
       "      <th>chocolate</th>\n",
       "      <th>strawberry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Color</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dark brown</th>\n",
       "      <td>0.00</td>\n",
       "      <td>10.50</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>light brown</th>\n",
       "      <td>0.00</td>\n",
       "      <td>4.75</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pink</th>\n",
       "      <td>4.75</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Flavor       bubblegum  chocolate  strawberry\n",
       "Color                                        \n",
       "dark brown        0.00      10.50         0.0\n",
       "light brown       0.00       4.75         0.0\n",
       "pink              4.75       0.00         8.8"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.pivot_table(more_cones, values='Price', index=['Color'], columns=['Flavor'], aggfunc=np.sum).fillna(0)\n",
    "\n",
    "#what happens when we add '.reset_index()' to the code?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is `group` doing the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Flavor</th>\n",
       "      <th>Color</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bubblegum</th>\n",
       "      <th>pink</th>\n",
       "      <td>4.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">chocolate</th>\n",
       "      <th>dark brown</th>\n",
       "      <td>10.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>light brown</th>\n",
       "      <td>4.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>strawberry</th>\n",
       "      <th>pink</th>\n",
       "      <td>8.80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Price\n",
       "Flavor     Color             \n",
       "bubblegum  pink          4.75\n",
       "chocolate  dark brown   10.50\n",
       "           light brown   4.75\n",
       "strawberry pink          8.80"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "more_cones.groupby(['Flavor', 'Color']).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though the numbers in both tables are the same, table produced by `pivot` is easier to read and lends itself more easily to analysis. The advantage of `pivot` is that it places grouped values into adjacent columns, so that they can be combined and compared."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Education and Income of Californian Adults\n",
    "The State of California's Open Data Portal is a rich source of information about the lives of Californians. It is our source of a [dataset](http://data.ca.gov/dataset/ca-educational-attainment-personal-income) on educational attainment and personal income among Californians over the years 2008 to 2014. The data are derived from the U.S. Census Current Population Survey.\n",
    "\n",
    "For each year, the table records the `Population Count` of Californians in many different combinations of age, gender, educational attainment, and personal income. We will study only the data for the year 2014."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Educational Attainment</th>\n",
       "      <th>Personal Income</th>\n",
       "      <th>Population Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>885</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>18 to 64</td>\n",
       "      <td>Female</td>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>H: 75,000 and over</td>\n",
       "      <td>2058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Male</td>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>H: 75,000 and over</td>\n",
       "      <td>2153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>894</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Female</td>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "      <td>4666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>895</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Female</td>\n",
       "      <td>High school or equivalent</td>\n",
       "      <td>H: 75,000 and over</td>\n",
       "      <td>7122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>896</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Female</td>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>F: 35,000 to 49,999</td>\n",
       "      <td>7261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>897</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Male</td>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "      <td>8569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>900</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>18 to 64</td>\n",
       "      <td>Female</td>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "      <td>14635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>901</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Male</td>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>F: 35,000 to 49,999</td>\n",
       "      <td>15212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>902</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Male</td>\n",
       "      <td>College, less than 4-yr degree</td>\n",
       "      <td>B: 5,000 to 9,999</td>\n",
       "      <td>15423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>903</th>\n",
       "      <td>1/1/14 0:00</td>\n",
       "      <td>65 to 80+</td>\n",
       "      <td>Female</td>\n",
       "      <td>Bachelor's degree or higher</td>\n",
       "      <td>A: 0 to 4,999</td>\n",
       "      <td>15459</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Year        Age  Gender          Educational Attainment  \\\n",
       "885  1/1/14 0:00   18 to 64  Female          No high school diploma   \n",
       "886  1/1/14 0:00  65 to 80+    Male          No high school diploma   \n",
       "894  1/1/14 0:00  65 to 80+  Female          No high school diploma   \n",
       "895  1/1/14 0:00  65 to 80+  Female       High school or equivalent   \n",
       "896  1/1/14 0:00  65 to 80+  Female          No high school diploma   \n",
       "897  1/1/14 0:00  65 to 80+    Male          No high school diploma   \n",
       "900  1/1/14 0:00   18 to 64  Female          No high school diploma   \n",
       "901  1/1/14 0:00  65 to 80+    Male          No high school diploma   \n",
       "902  1/1/14 0:00  65 to 80+    Male  College, less than 4-yr degree   \n",
       "903  1/1/14 0:00  65 to 80+  Female     Bachelor's degree or higher   \n",
       "\n",
       "         Personal Income  Population Count  \n",
       "885   H: 75,000 and over              2058  \n",
       "886   H: 75,000 and over              2153  \n",
       "894  G: 50,000 to 74,999              4666  \n",
       "895   H: 75,000 and over              7122  \n",
       "896  F: 35,000 to 49,999              7261  \n",
       "897  G: 50,000 to 74,999              8569  \n",
       "900  G: 50,000 to 74,999             14635  \n",
       "901  F: 35,000 to 49,999             15212  \n",
       "902    B: 5,000 to 9,999             15423  \n",
       "903        A: 0 to 4,999             15459  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_table = pd.read_csv(path_data + 'educ_inc.csv')\n",
    "\n",
    "ca_2014 = full_table[(full_table['Year'] == '1/1/14 0:00') & (full_table['Age'] != '00 to 17')]\n",
    "\n",
    "ca_2014.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row of the table corresponds to a combination of age, gender, educational level, and income. There are 127 such combinations in all! \n",
    "\n",
    "As a first step it is a good idea to start with just one or two variables. We will focus on just one pair: educational attainment and personal income. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Educational Attainment</th>\n",
       "      <th>Personal Income</th>\n",
       "      <th>Population Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>885</th>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>H: 75,000 and over</td>\n",
       "      <td>2058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>H: 75,000 and over</td>\n",
       "      <td>2153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>894</th>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "      <td>4666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>895</th>\n",
       "      <td>High school or equivalent</td>\n",
       "      <td>H: 75,000 and over</td>\n",
       "      <td>7122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>896</th>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>F: 35,000 to 49,999</td>\n",
       "      <td>7261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>897</th>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "      <td>8569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>900</th>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "      <td>14635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>901</th>\n",
       "      <td>No high school diploma</td>\n",
       "      <td>F: 35,000 to 49,999</td>\n",
       "      <td>15212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>902</th>\n",
       "      <td>College, less than 4-yr degree</td>\n",
       "      <td>B: 5,000 to 9,999</td>\n",
       "      <td>15423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>903</th>\n",
       "      <td>Bachelor's degree or higher</td>\n",
       "      <td>A: 0 to 4,999</td>\n",
       "      <td>15459</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Educational Attainment      Personal Income  Population Count\n",
       "885          No high school diploma   H: 75,000 and over              2058\n",
       "886          No high school diploma   H: 75,000 and over              2153\n",
       "894          No high school diploma  G: 50,000 to 74,999              4666\n",
       "895       High school or equivalent   H: 75,000 and over              7122\n",
       "896          No high school diploma  F: 35,000 to 49,999              7261\n",
       "897          No high school diploma  G: 50,000 to 74,999              8569\n",
       "900          No high school diploma  G: 50,000 to 74,999             14635\n",
       "901          No high school diploma  F: 35,000 to 49,999             15212\n",
       "902  College, less than 4-yr degree    B: 5,000 to 9,999             15423\n",
       "903     Bachelor's degree or higher        A: 0 to 4,999             15459"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "educ_inc = ca_2014[['Educational Attainment', 'Personal Income', 'Population Count']]\n",
    "\n",
    "educ_inc.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by looking at educational level alone. The categories of this variable have been subdivided by the different levels of income. So we will group the table by `Educational Attainment` and `sum` the `Population Count` in each category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Population Count sum</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Educational Attainment</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Bachelor's degree or higher</th>\n",
       "      <td>8525698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>College, less than 4-yr degree</th>\n",
       "      <td>7775497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>High school or equivalent</th>\n",
       "      <td>6294141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>No high school diploma</th>\n",
       "      <td>4258277</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                Population Count sum\n",
       "Educational Attainment                              \n",
       "Bachelor's degree or higher                  8525698\n",
       "College, less than 4-yr degree               7775497\n",
       "High school or equivalent                    6294141\n",
       "No high school diploma                       4258277"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "education = educ_inc[['Educational Attainment', 'Population Count']]\n",
    "\n",
    "educ_totals = education.groupby('Educational Attainment').sum()\n",
    "\n",
    "educ_totals = educ_totals.rename(columns={'Population Count': 'Population Count sum'})#.reset_index()\n",
    "\n",
    "educ_totals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are only four categories of educational attainment. The counts are so large that is is more helpful to look at percents. For this, we will use the function `percents` that we defined in an earlier section. It converts an array of numbers to an array of percents out of the total in the input array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def percents(array_x):\n",
    "    return np.round( (array_x/sum(array_x))*100, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the distribution of educational attainment among adult Californians. More than 30% have a Bachelor's degree or higher, while almost 16% lack a high school diploma."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Population Count sum</th>\n",
       "      <th>Population Percent</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Educational Attainment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Bachelor's degree or higher</th>\n",
       "      <td>8525698</td>\n",
       "      <td>31.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>College, less than 4-yr degree</th>\n",
       "      <td>7775497</td>\n",
       "      <td>28.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>High school or equivalent</th>\n",
       "      <td>6294141</td>\n",
       "      <td>23.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>No high school diploma</th>\n",
       "      <td>4258277</td>\n",
       "      <td>15.86</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                Population Count sum  Population Percent\n",
       "Educational Attainment                                                  \n",
       "Bachelor's degree or higher                  8525698               31.75\n",
       "College, less than 4-yr degree               7775497               28.96\n",
       "High school or equivalent                    6294141               23.44\n",
       "No high school diploma                       4258277               15.86"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "educ_distribution = educ_totals.copy()\n",
    "\n",
    "educ_distribution['Population Percent'] = percents(educ_totals.iloc[:,0])\n",
    "\n",
    "educ_distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "By using `pivot`, we can get a contingency table (a table of counts) of adult Californians cross-classified by `Educational Attainment` and `Personal Income`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Educational Attainment</th>\n",
       "      <th>Bachelor's degree or higher</th>\n",
       "      <th>College, less than 4-yr degree</th>\n",
       "      <th>High school or equivalent</th>\n",
       "      <th>No high school diploma</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Personal Income</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A: 0 to 4,999</th>\n",
       "      <td>575491</td>\n",
       "      <td>985011</td>\n",
       "      <td>1161873</td>\n",
       "      <td>1204529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B: 5,000 to 9,999</th>\n",
       "      <td>326020</td>\n",
       "      <td>810641</td>\n",
       "      <td>626499</td>\n",
       "      <td>597039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C: 10,000 to 14,999</th>\n",
       "      <td>452449</td>\n",
       "      <td>798596</td>\n",
       "      <td>692661</td>\n",
       "      <td>664607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D: 15,000 to 24,999</th>\n",
       "      <td>773684</td>\n",
       "      <td>1345257</td>\n",
       "      <td>1252377</td>\n",
       "      <td>875498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E: 25,000 to 34,999</th>\n",
       "      <td>693884</td>\n",
       "      <td>1091642</td>\n",
       "      <td>929218</td>\n",
       "      <td>464564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F: 35,000 to 49,999</th>\n",
       "      <td>1122791</td>\n",
       "      <td>1112421</td>\n",
       "      <td>782804</td>\n",
       "      <td>260579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>G: 50,000 to 74,999</th>\n",
       "      <td>1594681</td>\n",
       "      <td>883826</td>\n",
       "      <td>525517</td>\n",
       "      <td>132516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>H: 75,000 and over</th>\n",
       "      <td>2986698</td>\n",
       "      <td>748103</td>\n",
       "      <td>323192</td>\n",
       "      <td>58945</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Educational Attainment  Bachelor's degree or higher  \\\n",
       "Personal Income                                       \n",
       "A: 0 to 4,999                                575491   \n",
       "B: 5,000 to 9,999                            326020   \n",
       "C: 10,000 to 14,999                          452449   \n",
       "D: 15,000 to 24,999                          773684   \n",
       "E: 25,000 to 34,999                          693884   \n",
       "F: 35,000 to 49,999                         1122791   \n",
       "G: 50,000 to 74,999                         1594681   \n",
       "H: 75,000 and over                          2986698   \n",
       "\n",
       "Educational Attainment  College, less than 4-yr degree  \\\n",
       "Personal Income                                          \n",
       "A: 0 to 4,999                                   985011   \n",
       "B: 5,000 to 9,999                               810641   \n",
       "C: 10,000 to 14,999                             798596   \n",
       "D: 15,000 to 24,999                            1345257   \n",
       "E: 25,000 to 34,999                            1091642   \n",
       "F: 35,000 to 49,999                            1112421   \n",
       "G: 50,000 to 74,999                             883826   \n",
       "H: 75,000 and over                              748103   \n",
       "\n",
       "Educational Attainment  High school or equivalent  No high school diploma  \n",
       "Personal Income                                                            \n",
       "A: 0 to 4,999                             1161873                 1204529  \n",
       "B: 5,000 to 9,999                          626499                  597039  \n",
       "C: 10,000 to 14,999                        692661                  664607  \n",
       "D: 15,000 to 24,999                       1252377                  875498  \n",
       "E: 25,000 to 34,999                        929218                  464564  \n",
       "F: 35,000 to 49,999                        782804                  260579  \n",
       "G: 50,000 to 74,999                        525517                  132516  \n",
       "H: 75,000 and over                         323192                   58945  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "totals = pd.pivot_table(educ_inc, values='Population Count',\n",
    "                  index='Personal Income', columns='Educational Attainment', aggfunc=np.sum)\n",
    "totals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you see the power of `pivot` over other cross-classification methods. Each column of counts is a distribution of personal income at a specific level of educational attainment. Converting the counts to percents allows us to compare the four distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Personal Income</th>\n",
       "      <th>Bachelor's degree or higher</th>\n",
       "      <th>College, less than 4-yr degree</th>\n",
       "      <th>High school or equivalent</th>\n",
       "      <th>No high school diploma</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A: 0 to 4,999</td>\n",
       "      <td>6.75</td>\n",
       "      <td>12.67</td>\n",
       "      <td>18.46</td>\n",
       "      <td>28.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B: 5,000 to 9,999</td>\n",
       "      <td>3.82</td>\n",
       "      <td>10.43</td>\n",
       "      <td>9.95</td>\n",
       "      <td>14.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C: 10,000 to 14,999</td>\n",
       "      <td>5.31</td>\n",
       "      <td>10.27</td>\n",
       "      <td>11.00</td>\n",
       "      <td>15.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>D: 15,000 to 24,999</td>\n",
       "      <td>9.07</td>\n",
       "      <td>17.30</td>\n",
       "      <td>19.90</td>\n",
       "      <td>20.56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>E: 25,000 to 34,999</td>\n",
       "      <td>8.14</td>\n",
       "      <td>14.04</td>\n",
       "      <td>14.76</td>\n",
       "      <td>10.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>F: 35,000 to 49,999</td>\n",
       "      <td>13.17</td>\n",
       "      <td>14.31</td>\n",
       "      <td>12.44</td>\n",
       "      <td>6.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "      <td>18.70</td>\n",
       "      <td>11.37</td>\n",
       "      <td>8.35</td>\n",
       "      <td>3.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>H: 75,000 and over</td>\n",
       "      <td>35.03</td>\n",
       "      <td>9.62</td>\n",
       "      <td>5.13</td>\n",
       "      <td>1.38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Personal Income  Bachelor's degree or higher  \\\n",
       "0        A: 0 to 4,999                         6.75   \n",
       "1    B: 5,000 to 9,999                         3.82   \n",
       "2  C: 10,000 to 14,999                         5.31   \n",
       "3  D: 15,000 to 24,999                         9.07   \n",
       "4  E: 25,000 to 34,999                         8.14   \n",
       "5  F: 35,000 to 49,999                        13.17   \n",
       "6  G: 50,000 to 74,999                        18.70   \n",
       "7   H: 75,000 and over                        35.03   \n",
       "\n",
       "   College, less than 4-yr degree  High school or equivalent  \\\n",
       "0                           12.67                      18.46   \n",
       "1                           10.43                       9.95   \n",
       "2                           10.27                      11.00   \n",
       "3                           17.30                      19.90   \n",
       "4                           14.04                      14.76   \n",
       "5                           14.31                      12.44   \n",
       "6                           11.37                       8.35   \n",
       "7                            9.62                       5.13   \n",
       "\n",
       "   No high school diploma  \n",
       "0                   28.29  \n",
       "1                   14.02  \n",
       "2                   15.61  \n",
       "3                   20.56  \n",
       "4                   10.91  \n",
       "5                    6.12  \n",
       "6                    3.11  \n",
       "7                    1.38  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distributions = pd.DataFrame({\n",
    "                              \"Bachelor's degree or higher\":percents(totals.iloc[:,0]),\n",
    "                             'College, less than 4-yr degree':percents(totals.iloc[:,1]),\n",
    "                             'High school or equivalent':percents(totals.iloc[:,2]),\n",
    "                             'No high school diploma':percents(totals.iloc[:,3])\n",
    "                            })\n",
    "\n",
    "distributions.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a 'Barh' chart\n",
    "\n",
    "To be used as the data source in a horizontal bar chart the index must be converted to an array then included as a column in the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bachelor's degree or higher</th>\n",
       "      <th>College, less than 4-yr degree</th>\n",
       "      <th>High school or equivalent</th>\n",
       "      <th>No high school diploma</th>\n",
       "      <th>Personal Income</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Personal Income</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A: 0 to 4,999</th>\n",
       "      <td>6.75</td>\n",
       "      <td>12.67</td>\n",
       "      <td>18.46</td>\n",
       "      <td>28.29</td>\n",
       "      <td>A: 0 to 4,999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B: 5,000 to 9,999</th>\n",
       "      <td>3.82</td>\n",
       "      <td>10.43</td>\n",
       "      <td>9.95</td>\n",
       "      <td>14.02</td>\n",
       "      <td>B: 5,000 to 9,999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C: 10,000 to 14,999</th>\n",
       "      <td>5.31</td>\n",
       "      <td>10.27</td>\n",
       "      <td>11.00</td>\n",
       "      <td>15.61</td>\n",
       "      <td>C: 10,000 to 14,999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>D: 15,000 to 24,999</th>\n",
       "      <td>9.07</td>\n",
       "      <td>17.30</td>\n",
       "      <td>19.90</td>\n",
       "      <td>20.56</td>\n",
       "      <td>D: 15,000 to 24,999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E: 25,000 to 34,999</th>\n",
       "      <td>8.14</td>\n",
       "      <td>14.04</td>\n",
       "      <td>14.76</td>\n",
       "      <td>10.91</td>\n",
       "      <td>E: 25,000 to 34,999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F: 35,000 to 49,999</th>\n",
       "      <td>13.17</td>\n",
       "      <td>14.31</td>\n",
       "      <td>12.44</td>\n",
       "      <td>6.12</td>\n",
       "      <td>F: 35,000 to 49,999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>G: 50,000 to 74,999</th>\n",
       "      <td>18.70</td>\n",
       "      <td>11.37</td>\n",
       "      <td>8.35</td>\n",
       "      <td>3.11</td>\n",
       "      <td>G: 50,000 to 74,999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>H: 75,000 and over</th>\n",
       "      <td>35.03</td>\n",
       "      <td>9.62</td>\n",
       "      <td>5.13</td>\n",
       "      <td>1.38</td>\n",
       "      <td>H: 75,000 and over</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Bachelor's degree or higher  \\\n",
       "Personal Income                                    \n",
       "A: 0 to 4,999                               6.75   \n",
       "B: 5,000 to 9,999                           3.82   \n",
       "C: 10,000 to 14,999                         5.31   \n",
       "D: 15,000 to 24,999                         9.07   \n",
       "E: 25,000 to 34,999                         8.14   \n",
       "F: 35,000 to 49,999                        13.17   \n",
       "G: 50,000 to 74,999                        18.70   \n",
       "H: 75,000 and over                         35.03   \n",
       "\n",
       "                     College, less than 4-yr degree  \\\n",
       "Personal Income                                       \n",
       "A: 0 to 4,999                                 12.67   \n",
       "B: 5,000 to 9,999                             10.43   \n",
       "C: 10,000 to 14,999                           10.27   \n",
       "D: 15,000 to 24,999                           17.30   \n",
       "E: 25,000 to 34,999                           14.04   \n",
       "F: 35,000 to 49,999                           14.31   \n",
       "G: 50,000 to 74,999                           11.37   \n",
       "H: 75,000 and over                             9.62   \n",
       "\n",
       "                     High school or equivalent  No high school diploma  \\\n",
       "Personal Income                                                          \n",
       "A: 0 to 4,999                            18.46                   28.29   \n",
       "B: 5,000 to 9,999                         9.95                   14.02   \n",
       "C: 10,000 to 14,999                      11.00                   15.61   \n",
       "D: 15,000 to 24,999                      19.90                   20.56   \n",
       "E: 25,000 to 34,999                      14.76                   10.91   \n",
       "F: 35,000 to 49,999                      12.44                    6.12   \n",
       "G: 50,000 to 74,999                       8.35                    3.11   \n",
       "H: 75,000 and over                        5.13                    1.38   \n",
       "\n",
       "                         Personal Income  \n",
       "Personal Income                           \n",
       "A: 0 to 4,999              A: 0 to 4,999  \n",
       "B: 5,000 to 9,999      B: 5,000 to 9,999  \n",
       "C: 10,000 to 14,999  C: 10,000 to 14,999  \n",
       "D: 15,000 to 24,999  D: 15,000 to 24,999  \n",
       "E: 25,000 to 34,999  E: 25,000 to 34,999  \n",
       "F: 35,000 to 49,999  F: 35,000 to 49,999  \n",
       "G: 50,000 to 74,999  G: 50,000 to 74,999  \n",
       "H: 75,000 and over    H: 75,000 and over  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "personalIncome = np.array(distributions.index)\n",
    "\n",
    "distributions['Personal Income'] = personalIncome\n",
    "\n",
    "distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At a glance, you can see that over 35% of those with Bachelor's degrees or higher had incomes of $\\$75,000$ and over, whereas fewer than 10% of the people in the other education categories had that level of income. \n",
    "\n",
    "The bar chart below compares the personal income distributions of adult Californians who have no high diploma with those who have completed a Bachelor's degree or higher. The difference in the distributions is striking. There is a clear positive association between educational attainment and personal income."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 47028 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "distributions[['Personal Income',\n",
    "               \"Bachelor's degree or higher\",\n",
    "               'No high school diploma']].plot.barh('Personal Income',\n",
    "                                                    width=0.8,\n",
    "                                                    figsize=(10,8),\n",
    "                                                    color=('goldenrod','navy'))\n",
    "plt.ylabel('Personal Income')\n",
    "\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
