{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [ "remove_input" ] }, "outputs": [], "source": [ "path_data = '../../../../data/'\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.style.use('fivethirtyeight')\n", "\n", "import warnings\n", "warnings.filterwarnings('ignore')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Decisions and Uncertainty ###\n", "We have seen several examples of assessing models that involve chance, by comparing observed data to the predictions made by the models. In all of our examples, there has been no doubt about whether the data were consistent with the model's predictions. The data were either very far away from the predictions, or very close to them.\n", "\n", "But outcomes are not always so clear cut. How far is \"far\"? Exactly what does \"close\" mean? While these questions don't have universal answers, there are guidelines and conventions that you can follow. In this section we will describe some of them.\n", "\n", "But first let us develop a general framework of decision making, into which all our examples will fit.\n", "\n", "What we have developed while assessing models are some of the fundamental concepts of statistical tests of hypotheses. Using statistical tests as a way of making decisions is standard in many fields and has a standard terminology. Here is the sequence of the steps in most statistical tests, along with some terminology and examples. You will see that they are consistent with the sequence of steps we have used for assessing models." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: The Hypotheses ###\n", "\n", "All statistical tests attempt to choose between two views of the world. Specifically, the choice is between two views about how the data were generated. These two views are called *hypotheses*.\n", "\n", "**The null hypothesis.** This is a clearly defined model about chances. It says that the data were generated at random under clearly specified assumptions about the randomness. The word \"null\" reinforces the idea that if the data look different from what the null hypothesis predicts, the difference is due to *nothing* but chance.\n", "\n", "From a practical perspective, **the null hypothesis is a hypothesis under which you can simulate data.**\n", "\n", "In the example about Mendel's model for the colors of pea plants, the null hypothesis is that the assumptions of his model are good: each plant has a 75% chance of having purple flowers, independent of all other plants. \n", "\n", "Under this hypothesis, we were able to simulate random samples, by using `sample_proportions(929, [0.75, 0.25])`. We used a sample size of 929 because that's the number of plants Mendel grew.\n", "\n", "**The alternative hypothesis.** This says that some reason other than chance made the data differ from the predictions of the model in the null hypothesis.\n", "\n", "In the example about Mendel's plants, the alternative hypothesis is simply that his model isn't good." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 2: The Test Statistic ###\n", "\n", "In order to decide between the two hypothesis, we must choose a statistic that we can use to make the decision. This is called the **test statistic**.\n", "\n", "In the example of Mendel's plants, our statistic was the absolute difference between the sample percent and 75% which was predicted by his model.\n", "\n", "$$\n", "\\big{\\vert} \\text{sample percent of purple-flowering plants} - 75 \\big{\\vert}\n", "$$\n", "\n", "To see how to make the choice in general, look at the alternative hypothesis. What values of the statistic will make you think that the alternative hypothesis is a better choice than the null? \n", "- If the answer is \"big values,\" you might have a good choice of statistic. \n", "- So also if the answer is \"small values.\" \n", "- But if the answer is \"both big values and small values,\" we recommend that you look again at your statistic and see if taking an absolute value can change the answer to just \"big values\".\n", "\n", "In the case of the pea plants, a sample percent of around 75% will be consistent with the model, but percents much bigger or much less than 75 will make you think that the model isn't good. This indicates that the statistic should be the *distance* between the sample percent and 75, that is, the absolute value of the difference between them. Big values of the distance will make you lean towards the alternative.\n", "\n", "The **observed value of the test statistic** is the value of the statistic you get from the data in the study, not a simulated value. Among Mendel's 929 plants, 705 had purple flowers. The observed value of the test statistic was therefore" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.8880516684607045" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "abs ( 100 * (705 / 929) - 75)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 3: The Distribution of the Test Statistic, Under the Null Hypothesis ###\n", "\n", "The main computational aspect of a test of hypotheses is figuring out *what the values of the test statistic might be if the null hypothesis were true*. \n", "\n", "The test statistic is simulated based on the assumptions of the model in the null hypothesis. That model involves chance, so the statistic comes out differently when you simulate it multiple times.\n", "\n", "By simulating the statistic repeatedly, we get a good sense of its possible values and which ones are more likely than others. In other words, we get a good approximation to the probability distribution of the statistic, as predicted by the model in the null hypothesis.\n", "\n", "As with all distributions, it is very useful to visualize this distribution by a histogram. We have done so in all our examples." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 4. The Conclusion of the Test ###\n", "\n", "The choice between the null and alternative hypotheses depends on the comparison between what you computed in Steps 2 and 3: the observed value of the test statistic and its distribution as predicted by the null hypothesis. \n", "\n", "If the two are consistent with each other, then the observed test statistic is in line with what the null hypothesis predicts. In other words, the test does not point towards the alternative hypothesis; the null hypothesis is better supported by the data. This was the case with the assessment of Mendel's model.\n", "\n", "But if the two are not consistent with each other, as is the case in our example about Alameda County jury panels, then the data do not support the null hypothesis. That is why we concluded that the jury panels were not selected at random. Something other than chance affected their composition.\n", "\n", "If the data do not support the null hypothesis, we say that the test *rejects* the null hypothesis." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The Meaning of \"Consistent\" ###\n", "\n", "In the example about Alameda County juries, it was apparent that our observed test statistic was far from what was predicted by the null hypothesis. In the example about pea flowers, it is just as clear that the observed statistic is consistent with the distribution that the null predicts. So in both of the examples, it is clear which hypothesis to choose.\n", "\n", "But sometimes the decision is not so clear. Whether the observed test statistic is consistent with its predicted distribution under the null hypothesis is a matter of judgment. We recommend that you provide your judgment along with the value of the test statistic and a graph of its predicted distribution under the null. That will allow your reader to make his or her own judgment about whether the two are consistent.\n", "\n", "Here is an example where the decision requires judgment." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The GSI's Defense ###\n", "A Berkeley Statistics class of about 350 students was divided into 12 discussion sections led by Graduate Student Instructors (GSIs). After the midterm, students in Section 3 noticed that their scores were on average lower than the rest of the class. \n", "\n", "In such situations, students tend to grumble about the section's GSI. Surely, they feel, there must have been something wrong with the GSI's teaching. Or else why would their section have done worse than others?\n", "\n", "The GSI, typically more experienced about statistical variation, often has a different perspective: if you simply draw a section of students at random from the whole class, their average score could resemble the score that the students are unhappy about, just by chance.\n", "\n", "The GSI's position is a clearly stated chance model. We can simulate data under this model. Let's test it out. \n", "\n", "**Null Hypothesis.** The average score of the students in Section 3 is like the average score of the same number of students picked at random from the class. \n", "\n", "**Alternative Hypothesis.** No, it's too low.\n", "\n", "A natural statistic here is the average of the scores. Low values of the average will make us lean towards the alternative.\n", "\n", "Let's take a look at the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The table `scores` contains the section number and midterm score for each student in the class. The midterm scores were integers in the range 0 through 25; 0 means that the student didn't take the test." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SectionMidterm
0122
1212
2223
3214
4120
.........
354524
355216
356217
3571216
3581014
\n", "

359 rows × 2 columns

\n", "
" ], "text/plain": [ " Section Midterm\n", "0 1 22\n", "1 2 12\n", "2 2 23\n", "3 2 14\n", "4 1 20\n", ".. ... ...\n", "354 5 24\n", "355 2 16\n", "356 2 17\n", "357 12 16\n", "358 10 14\n", "\n", "[359 rows x 2 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "scores = pd.read_csv(path_data + 'scores_by_section.csv')\n", "scores" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To find the average or mean score in each section, we will use `groupby`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Midterm
Section
115.593750
215.125000
313.666667
414.766667
517.454545
615.031250
716.625000
816.310345
914.566667
1015.235294
1115.807692
1215.733333
\n", "
" ], "text/plain": [ " Midterm\n", "Section \n", "1 15.593750\n", "2 15.125000\n", "3 13.666667\n", "4 14.766667\n", "5 17.454545\n", "6 15.031250\n", "7 16.625000\n", "8 16.310345\n", "9 14.566667\n", "10 15.235294\n", "11 15.807692\n", "12 15.733333" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "section_averages = scores.groupby(['Section']).mean()\n", "\n", "section_averages" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The average score of Section 3 is 13.667, which does look low compared to the other section averages. But is it lower than the average of a section of the same size selected at random from the class? \n", "\n", "To answer this, we can select a section at random from the class and find its average. To select a section at random to we need to know how big Section 3 is, which we can by once again using `group`." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Midterm
Section
132
232
327
430
533
632
724
829
930
1034
1126
1230
\n", "
" ], "text/plain": [ " Midterm\n", "Section \n", "1 32\n", "2 32\n", "3 27\n", "4 30\n", "5 33\n", "6 32\n", "7 24\n", "8 29\n", "9 30\n", "10 34\n", "11 26\n", "12 30" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "scores.groupby('Section').count()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Section 3 had 27 students. \n", "\n", "Now we can figure out how to create one simulated value of our test statistic, the random sample average.\n", "\n", "First we have to select 27 scores at random without replacement. Since the data are already in a table, we will use the Table method `sample`.\n", "\n", "Remember that by default, `sample` draws with replacement. The optional argument `with_replacement = False` produces a random sample drawn without replacement." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "scores_only = scores.drop(columns=['Section'])" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Midterm
29623
28124
30823
3480
33915
5812
9418
7714
28017
420
9515
26117
7811
12825
6822
29925
31211
180
8313
24317
20525
1739
2011
27819
1024
21122
2390
\n", "
" ], "text/plain": [ " Midterm\n", "296 23\n", "281 24\n", "308 23\n", "348 0\n", "339 15\n", "58 12\n", "94 18\n", "77 14\n", "280 17\n", "4 20\n", "95 15\n", "261 17\n", "78 11\n", "128 25\n", "68 22\n", "299 25\n", "312 11\n", "18 0\n", "83 13\n", "243 17\n", "205 25\n", "173 9\n", "20 11\n", "278 19\n", "10 24\n", "211 22\n", "239 0" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "random_sample = scores_only.sample(27, replace=False)\n", "random_sample" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The average of these 27 randomly selected scores is" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "16.0" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.average(random_sample['Midterm'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's the average of 27 randomly selected scores. The cell below collects the code necessary for generating this random average. \n", "\n", "Now we can simulate the random sample average by repeating the calculation multple times." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "def random_sample_average():\n", " random_sample = scores_only.sample(27, replace=False)\n", " return np.average(random_sample['Midterm'])" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([15.55555556, 15.59259259, 15.51851852, ..., 15.07407407,\n", " 15.2962963 , 14.51851852])" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_averages = np.array([])\n", "\n", "repetitions = 10000\n", "for i in np.arange(repetitions):\n", " sample_averages = np.append(sample_averages, random_sample_average())\n", " \n", "sample_averages" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the histogram of the simulated averages. It shows the distribution of what the Section 3 average might have been, if Section 3 had been selected at random from the class. \n", "\n", "The observed Section 3 average score of 13.667 is shown as a red dot on the horizontal axis. You can ignore the last line of code; it just draws the dot." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbEAAAEfCAYAAADPxvgvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyGElEQVR4nO3deVQT5/4/8HdAtrKIC0Q2N0BBRW1dsCKidV9AFKxb7/XrUi1Wq23xClW0WC1upV+rSG2t3lpXBGwpt9XaK1hXrhtWi3pRpLggChgEhLAkvz/6NT8jW2ICycT36xzOaWaeeead6Tn5ODPPPCOSSCRyEBERCZCRrgMQERG9KBYxIiISLBYxIiISLBYxIiISLBYxIiISLBYxIiISLBYxIiISLBYxIiISLBaxRpKZmanrCAaPx7jx8Rg3Ph5jzbCIERGRYLGIERGRYLGIERGRYLGIERGRYLGIERGRYLGIERGRYOmsiH399dfo378/XFxc4OLigmHDhuHw4cOK9SEhIbC1tVX6Gzp0qK7iEhGRHmqmqx07OjoiMjISrq6ukMlk2Lt3L6ZNm4bU1FR069YNADBo0CBs3bpVsY2pqamu4hIRkR7SWREbM2aM0ueIiAh88803OHv2rKKImZmZQSwW6yIekV7LlYpwt1SmUR9OlkZwMOOL3UnYdFbEnlVdXY3vv/8epaWl6Nu3r2L56dOn4ebmhubNm8PHxwcRERGws7PTYVIi/XC3VIbQEw816mPDADs4mIm0lIhIN0QSiURn/xT7448/MHz4cJSXl8PS0hJff/01RowYAQBISEiAhYUF2rVrh5ycHKxatQoymQypqakwMzOrs09O4UIvg9um9lj6nyKN+ljdtzlcKh5oKRFR43B3d693vU6LWEVFBe7cuYOioiIkJSXh22+/RXJyMrp06VKjbW5uLry8vLB9+3YEBAToIK16MjMzGzz4pJmX+RifK5Rr5Uysd8v6z8Re5mPcVHiMNaPTy4mmpqbo2LEjAODVV1/FhQsXsGXLFmzevLlGWwcHBzg6OiIrK6upYxIRkZ7Sq+fEZDIZKioqal1XUFCA3NxcDvQgIiIFnZ2Jffzxxxg+fDicnJxQUlKC+Ph4nDhxAnFxcSgpKcGaNWsQEBAAsViMnJwcrFy5EnZ2dhg7dqyuIhMRkZ7RWRHLy8vDnDlz8ODBA9jY2KBr166Ij4/HkCFDUFZWhoyMDOzbtw9FRUUQi8Xw9fXFjh07YG1travIRESkZ3RWxGJjY+tcZ2FhgcTExCZMQ0REQqRX98SIiIjUwSJGRESCxSJGRESCpRfTThFR0xMZGeFcYf3zLz42tUdRYf3zIXAORtIlFjGil1R+WTWizubX20YqLYeZmbTeNpyDkXSJlxOJiEiwWMSIiEiweDmRqIlp411gZZptTmQwWMSImpg23gUW3qe1ltIQCRsvJxIRkWCxiBERkWCxiBERkWCxiBERkWCxiBERkWCxiBERkWCxiBERkWCxiBERkWCxiBERkWCxiBERkWBx2ikiNXDeQyL9orMi9vXXX2PHjh24ffs2AMDDwwOhoaEYMWIEAEAul2PNmjX49ttvIZFI0KtXL2zYsAGenp66ikzEeQ+J9IzOLic6OjoiMjISx44dQ0pKCgYOHIhp06bhypUrAICNGzciJiYGa9euxdGjR2FnZ4fx48ejuLhYV5GJiEjP6KyIjRkzBsOGDUPHjh3h5uaGiIgIWFlZ4ezZs5DL5YiNjcWiRYswbtw4dOnSBbGxsSgpKUF8fLyuIhMRkZ7Ri4Ed1dXVSEhIQGlpKfr27Ys///wTeXl5eOONNxRtLCws0L9/f6SlpekwKRER6ROdDuz4448/MHz4cJSXl8PS0hK7du1C165dFYXKzs5Oqb2dnR1yc3N1EZWIiPSQTouYu7s7jh8/jqKiIiQlJSEkJATJycmK9SKRSKm9XC6vsex5mZmZjZL1RehTFkPV1Mf4sak9pNJyjfqorKoUVB8NtXlc/BiZBQ80yvKy429F3dzd3etdr9MiZmpqio4dOwIAXn31VVy4cAFbtmxBaGgoAODBgwdwdnZWtM/Pz69xdva8hr5wU8nMzNSbLIZKF8e4qFAOMzOpRn2YNDOBmZm5IPqQSssbbGNjbQP3ls01yvIy42+FZvTinthTMpkMFRUVaNeuHcRiMVJSUhTrysvLcfr0aXh7e+swIRER6ROdnYl9/PHHGD58OJycnBSjDk+cOIG4uDiIRCKEhITgs88+g7u7O9zc3LBhwwZYWloiODhYV5GJiEjP6KyI5eXlYc6cOXjw4AFsbGzQtWtXxMfHY8iQIQCAhQsXoqysDIsXL1Y87JyYmAhra2tdRSYiIj2jchE7efIkOnfujNata59toKCgANeuXYOPj49K/cXGxta7XiQSITw8HOHh4apGJCKil4zK98T8/f2V7lE979ixY/D399dKKCIiIlWoXMTkcnm96ysqKmBkpFfjRIiIyMDVeznx8ePHKCoqUnwuLCxUTNj7LIlEgoSEBDg4OGg/IRERUR3qLWJbtmzBunXrADR8j0oulyMiIkL7CYmIiOpQbxEbNGgQzM3NIZfLsXLlSkyYMAFeXl5KbUQiEV555RW8+uqr6N27d6OGJSIiela9Raxfv37o168fAEAqlcLf3x9du3ZtkmBEREQNUXmIfVhYWGPmICIiUludRWzv3r0AgMmTJ0MkEik+N2TKlCnaSUZERNSAOovYvHnzIBKJEBQUBFNTU8ybN6/BzkQiEYsYERE1mTqL2KVLlwD8NdP8s5+JiIj0RZ1FrG3btvV+JiIi0jVOsUFERIKl1iz2qamp+Pbbb5GdnY1Hjx7VmIpKJBIhPT1dm/mIiIjqpHIRi42NxdKlS9G6dWv07t0bnp6ejZmLiIioQSoXsZiYGPj4+CAhIUEx2IOIiEiXVL4nVlBQgAkTJrCAERGR3lC5iPXs2RM5OTmNmYWIiEgtKhex1atXY8+ePfjtt98aMw8REZHKVL4nFhUVBRsbGwQGBsLV1RUuLi4wNjZWaiMSiRAXF6f1kERERLVRuYhdu3YNIpEIzs7OkEqluHHjRo02IpFIq+GIiIjqo3IRu3z5cmPmICIiUpvOZuyIjo7G4MGD4eLiAldXV0yaNAkZGRlKbUJCQmBra6v0N3ToUB0lJiIifaPymdjt27dVaufi4qJSuxMnTmDWrFl47bXXIJfL8emnnyIwMBBpaWlo0aKFot2gQYOwdetWxWcO8SfSLyIjI5wrlGnUh5OlERzM5A03JHqOykWse/fuKt3zKiwsVKm/xMREpc9bt25F27ZtcebMGYwaNUqx3MzMDGKxWNWYRNTE8suqEXU2X6M+Ngywg4MZ76mT+lQuYps3b65RxKqrq/Hnn39i3759sLe3x+zZs184SElJCWQyGWxtbZWWnz59Gm5ubmjevDl8fHwQEREBOzu7F94PEREZDpWL2LRp0+pct2jRIrzxxhsoKSl54SBhYWHw8vJC3759FcuGDh0Kf39/tGvXDjk5OVi1ahUCAgKQmpoKMzOzWvvJzMx84Qzapk9ZDFVTH+PHpvaQSss16qOyqlJQfTTURhtZHhc/RmbBA436EDL+VtTN3d293vVqzWJfFysrK0ybNg1btmzBnDlz1N7+o48+wpkzZ3Do0CGlZ8+CgoIU/921a1f07NkTXl5eOHz4MAICAmrtq6Ev3FQyMzP1Jouh0sUxLiqUw8xMqlEfJs1MYGZmLog+pNLyBttoI4uNtQ3cWzbXqA+h4m+FZrRSxADAxMQEubm5am8XHh6OxMRE/Pjjj2jfvn29bR0cHODo6IisrKwXTElERIZEK0Xs8uXL+PLLL9G5c2e1tluyZAkSExORnJyMTp06Ndi+oKAAubm5HOhBREQAtDA6saioCI8fP4aVlRViYmJU3nFoaCj279+PXbt2wdbWFnl5eQAAS0tLWFlZoaSkBGvWrEFAQADEYjFycnKwcuVK2NnZYezYsSrvh4iIDJfKRczHx6dGEROJRLC1tUXHjh0RFBRUY2RhfbZt2wYAGDdunNLyJUuWIDw8HMbGxsjIyMC+fftQVFQEsVgMX19f7NixA9bW1irvh4iIDJdab3bWJolEUu96CwuLGs+SERERPUtn004RERFpikWMiIgEi0WMiIgEi0WMiIgEi0WMiIgES6UiVl5ejrVr1+Lo0aONnYeIiEhlKhUxc3NzfP7557hz505j5yEiIlKZypcTvby8OGchERHpFZWL2PLly7Fz504cPny4MfMQERGpTOUZO7744gvY2tpiypQpcHR0RPv27WFhYaHURiQSIS4uTushiYiIaqNyEbt27RpEIhGcnZ0BADk5OTXa1DZBMBERUWNRuYhdvny5MXMQERGpjc+JERGRYKn1Uszq6mokJCTgt99+w8OHDxEREYFu3bpBIpEgJSUFr7/+Otq0adNYWYk0kisV4W6pTKM+yjTbnIi0TOUiVlRUhAkTJuDChQuwsrJCaWkp5s2bBwCwtrbG0qVLMXnyZCxfvrzRwhJp4m6pDKEnHmrUR3if1lpKQ0TaoPLlxMjISFy7dg0HDhxAeno65HK5Yp2xsTH8/f1x5MiRRglJRERUG5WL2L/+9S/MmTMHQ4cOrXUUoqurK27fvq3VcERERPVRuYhJJBJ06NChzvVyuRwVFRVaCUVERKQKlYtY27ZtkZGRUef6kydPws3NTSuhiIiIVKFyEZs4cSJ27tyJkydPKpY9vay4detWJCcnY+rUqdpPSEREVAeVi9j777+PAQMGICAgAKNGjYJIJEJYWBg8PDwQFhaG0aNHY+7cuSrvODo6GoMHD4aLiwtcXV0xadKkGmd6crkcUVFR8PDwQJs2bTBmzBhcvXpV9W9HREQGTeUiZmJigri4OHz55Zdwc3NDp06dUFVVhR49euDLL7/Ed999p9a0UydOnMCsWbNw+PBhJCUloVmzZggMDMSjR48UbTZu3IiYmBjFu8zs7Owwfvx4FBcXq/ctiYjIIKn1sDPw12XFiRMnarzjxMREpc9bt25F27ZtcebMGYwaNQpyuRyxsbFYtGgRxo0bBwCIjY2Fu7s74uPjMWPGDI0zEBGRsL3QtFNXrlzBzz//jJ9//hlXrlxRembsRZWUlEAmk8HW1hYA8OeffyIvLw9vvPGGoo2FhQX69++PtLQ0jfdHRETCp9aZWEJCAlasWIF79+4pCpdIJIKjoyNWrFih0RlaWFgYvLy80LdvXwBAXl4eAMDOzk6pnZ2dHXJzc194P0REZDhULmK7d+/G/Pnz4e7ujsjISLi5uUEul+PmzZvYuXMn5s6di4qKCkybNk3tEB999BHOnDmDQ4cOwdjYWGnd8/fZ5HJ5vffeMjMz1d5/Y9GnLIZKnWP82NQeUmm5RvurrKp86fpoqI02sjwufozMggca9SFk/K2om7u7e73rVS5i0dHR6NWrF5KTk2Fubq607u2338bo0aMRHR2tdhELDw9HYmIifvzxR7Rv316xXCwWAwAePHigeIcZAOTn59c4O3tWQ1+4qWRmZupNFkOl7jEuKpTDzEyq0T5NmpnAzMy84YYG0odUWt5gG21ksbG2gXvL5hr1IVT8rdCMyvfE7t69i4kTJ9YoYABgbm6OSZMm4d69e2rtfMmSJYiPj0dSUhI6deqktK5du3YQi8VISUlRLCsvL8fp06fh7e2t1n6IiMgwqXwm5uHhUe+9qHv37qFz584q7zg0NBT79+/Hrl27YGtrq7gHZmlpCSsrK4hEIoSEhOCzzz6Du7s73NzcsGHDBlhaWiI4OFjl/RARkeFSuYitXLkS06dPR48ePTB+/HildQkJCdi5cyd27typ8o63bdsGAIrh808tWbIE4eHhAICFCxeirKwMixcvhkQiQa9evZCYmAhra2uV90NERIZL5SK2adMmtGrVCrNmzUJYWBg6dOgAkUiErKwsPHz4EK6urvjiiy/wxRdfKLYRiUSIi4urtT+JRNLgPkUiEcLDwxVFjYiI6FkqF7Fr165BJBIpBlk8vf9lZmYGZ2dnSKVSXL9+XWkbdWbwICIiUpfKRezy5cuNmYOIiEhtLzRjBxERkT5gESMiIsFSewJgIiJtExkZ4VyhTKM+nCyN4GCm+TyuJCwsYkSkc/ll1Yg6m69RHxsG2MHBjIPJXja8nEhERILFIkZERIKlchHr0aMHfvrppzrXHzp0CD169NBKKCIiIlWoXMRycnJQWlpa5/rS0lLcvn1bK6GIiIhUodblxPpm4Lhx4wbnNCQioiZV7+jEPXv2YO/evYrPGzZswLffflujnUQiQUZGBkaMGKH9hERERHWot4iVlpYqXpECAEVFRZDJlJ/lEIlEeOWVVzB9+nSEhYU1TkoiIqJa1FvE3n77bbz99tsAgO7du2PNmjUYPXp0kwQjIiJqiMoPO//++++NmYOIiEhtas/YUVxcjDt37uDRo0eQy2tO8eLj46OVYERERA1RuYg9evQIS5YswcGDB1FdXV1jvVwuh0gkQmFhoVYDEhER1UXlIvb+++8jOTkZb7/9Nnx8fGBra9uIsYiIiBqmchH79ddfMXfuXKxevbox8xAREalM5YedTU1N4erq2phZiIiI1KJyERs3bhyOHDnSmFmIiIjUonIRW7BgAe7fv4933nkHZ8+exf379/Hw4cMaf+o4efIkJk+eDE9PT9ja2mL37t1K60NCQmBra6v0N3ToULX2QUREhkvle2K9evWCSCRCeno64uLi6mynzujE0tJSdOnSBVOmTME777xTa5tBgwZh69atis+mpqYq909ERIZN5SL2j3/8o94JgF/E8OHDMXz4cADAvHnzam1jZmYGsVis1f2S8ORKRbhbqjzl2WNTexQVqv46+jJZw22ISFhULmLh4eGNmaNOp0+fhpubG5o3bw4fHx9ERETAzs5OJ1lId+6WyhB6QvlytVRaDjMzqcp9hPdpre1YRKRjas/YAQDV1dUoKiqCjY0NmjV7oS5UMnToUPj7+6Ndu3bIycnBqlWrEBAQgNTUVJiZmdW6TWZmZqPlUZc+ZRG6x6b2kErLayyvbVldKqsq1WrPPv7SUBt9+T6Pix8js+CBRn3oCn8r6ubu7l7verUq0IULF7By5UqcPn0alZWVOHjwIPz8/FBQUICQkBC8++678PPz0yjws4KCghT/3bVrV/Ts2RNeXl44fPgwAgICat2moS/cVDIzM/UmiyEoKpTXOOv660zMXOU+TJqZqNWefah2jPXl+9hY28C9ZXON+tAF/lZoRuXRif/5z38wevRo3Lp1C5MnT1aaN7FVq1YoKSnBd9991yghn3JwcICjoyOysrIadT9ERCQMKhexTz75BK6urkhLS8Py5ctrrPf19cW5c+e0Gu55BQUFyM3N5UAPIiICoMblxAsXLmDZsmUwNzfHkydPaqx3cnJSeoGmKkpKShRnVTKZDHfu3MHvv/+OFi1aoEWLFlizZg0CAgIgFouRk5ODlStXws7ODmPHjlVrP0REZJhULmJGRkYwMqr7xC0vLw8WFhZq7fzixYvw9/dXfI6KikJUVBSmTJmC6OhoZGRkYN++fSgqKoJYLIavry927NgBa2trtfZDRIZPZGSEc4WaPUfhZGkEBzPVH9sg3VO5iPXs2ROHDh3C3Llza6yrqKjAgQMH0LdvX7V27uvrC4lEUuf6xMREtfojopdXflk1os7ma9THhgF2cDDT7vOw1LhUvif2wQcf4LfffsP8+fNx+fJlAMD9+/fx66+/IiAgALdu3cKHH37YaEGJiIiep/KZ2ODBg7F161YsXrwYe/bsAfDX3IZyuRzNmzfHtm3b0KdPn0YLSkRE9Dy1nhMLDg7G6NGjkZKSgps3b0Imk6FDhw4YMmQIrKysGisjERFRrdSebuOVV17BmDFjGiMLERGRWlS+J/bTTz9h8eLFda5fvHgxDh06pJVQREREqlC5iG3atKnW58OeKi8vx8aNG7USioiISBUqF7GMjAz07NmzzvU9evTAtWvXtJGJiIhIJSoXsaqqKpSVldW5vqysDFKp6q/FICIi0pTKRaxLly5ISkqCTFbziXiZTIakpCR4eHhoNRwREVF9VC5i77zzDs6fP48pU6YgPT0dUqkUUqkU6enpmDp1Ks6fP1/rbB5ERESNReUh9kFBQbh16xaioqJw5MgRAIBIJIJcLodIJMKSJUswadKkRgtKRET0PLWeEwsNDUVwcDB+/PFHZGdnQy6Xo0OHDvD390f79u0bKSIREVHtVCpiZWVlePPNNzFp0iS89dZbWLBgQWPnIiIiapBK98QsLCxw6dIlVFdXN3YeIiIilak8sGPAgAE4depUY2YhIiJSi8pFbO3atbhw4QIiIiKQnZ1d61B7IiKipqTywI4+ffpALpcjJiYGMTExMDIygomJiVIbkUiEe/fuaT0kERFRbVQuYuPHj4dIxDeeEhGR/lC5iMXGxjZmDiIiIrWpfE+MiIhI36hVxHJycvDee++hZ8+ecHFxwYkTJwAABQUF+PDDD5Genq7Wzk+ePInJkyfD09MTtra22L17t9J6uVyOqKgoeHh4oE2bNhgzZgyuXr2q1j6IiMhwqVzErl+/Dj8/P/zwww9wdXVFaWmp4rmxVq1a4ezZs9i2bZtaOy8tLUWXLl2wZs0aWFhY1Fi/ceNGxMTEYO3atTh69Cjs7Owwfvx4FBcXq7UfIiIyTCoXsRUrVsDa2hpnz57FV199BblcrrR++PDhOHPmjFo7Hz58OJYvX45x48bByEg5ilwuR2xsLBYtWoRx48ahS5cuiI2NRUlJCeLj49XaDxERGSaVB3acOnUKoaGhsLe3R2FhYY31Li4uyM3N1VqwP//8E3l5eXjjjTcUyywsLNC/f3+kpaVhxowZWtsXNa5cqQh3SzV7rrCMjyUSUS1ULmJVVVWwtLSsc/2jR49gbGyslVAAkJeXBwCws7NTWm5nZ1dvsczMzNRaBk3pUxZdum1qj6X/KdKoj6X9xJBKy2ssr21ZXSqrKtVqzz7+0lAboX2f+jwufozMggca9fEi+FtRN3d393rXq1zEunTpguPHj2PWrFk11snlcvz444/o2bOn2gEb8vyzaU9f/VKXhr5wU8nMzNSbLLpWVCiHmZlmb/02aWYCMzNzpWVSaXmNZer2oY0chtyHKsdYSN+nITbWNnBv2VyjPtTF3wrNqHxPLCQkBD/88APWrVunuJwok8nw3//+FzNnzsTFixe1Oru9WCwGADx4oPyvovz8/BpnZ0RE9HJS66WYt2/fxurVq7FmzRrFMgAwNjbGqlWrMGzYMK0Fa9euHcRiMVJSUvDaa68BAMrLy3H69GmsXLlSa/shIiLhUuulmIsWLUJwcDCSkpKQlZUFmUyGDh06ICAgAO3atVN75yUlJcjKygLw11ndnTt38Pvvv6NFixZwcXFBSEgIPvvsM7i7u8PNzQ0bNmyApaUlgoOD1d4XEREZngaLmFQqxU8//YTs7Gy0bNkSI0aMwLx587Sy84sXL8Lf31/xOSoqClFRUZgyZQpiY2OxcOFClJWVYfHixZBIJOjVqxcSExNhbW2tlf0TEZGw1VvE8vLyMHr0aNy6dUvxXJilpSX2798PHx8fjXfu6+sLiURS53qRSITw8HCEh4drvC8iIjI89Q7sWLVqFbKzszFv3jzs378fUVFRMDMzwz/+8Y+mykdERFSnes/Ejh49iilTpmDVqlWKZfb29pg9ezbu3r0LJyenRg9IRERUl3rPxPLy8uDt7a20rF+/fpDL5bhz506jBiMiImpIvUWsuroa5ubKDw8+/VxertmT8URERJpqcHRidnY2zp8/r/j8+PFjAH89ZW5lZVWjfa9evbQYj4iIqG4NFrGnw96f9/zgjqfTQdU2OTAREVFjqLeIxcTENFUOIiIitdVbxKZOndpUOYiIiNSm8gTARERE+oZFjIiIBItFjIiIBEutWeyJiAyZyMgI5wplGvXhZGkEBzO5lhJRQ1jEiIj+T35ZNaLO5mvUx4YBdnAwq/vt86RdvJxIRESCxSJGRESCxSJGRESCxSJGRESCxSJGRESCxSJGRESCxSJGRESCxSJGRESCpddFLCoqCra2tkp/nTp10nUsIiLSE3o/Y4e7uzuSk5MVn42NjXWY5uWTKxXhbqlm0/CUabY5EVGd9L6INWvWDGKxWNcxXlp3S2UIPfFQoz7C+7TWUhoiImV6fTkRALKzs+Hp6Ynu3btj5syZyM7O1nUkIiLSE3p9Jta7d29s2bIF7u7uyM/Px/r16zF8+HCcOXMGLVu2rHWbzMzMJk5ZN33K8qIem9pDKi3XqI/KqspG60OdfhszhyH30VAboX2fxu7jcfFjZBY8UGsbQ/itaCzu7u71rtfrIjZs2DClz71790bPnj2xZ88ezJ8/v9ZtGvrCTSUzM1NvsmiiqFAOMzOpRn2YNDOBmZm51vuQSsvV6rexchhyH6ocYyF9n6bow8baBu4tm6vc3lB+K3RF7y8nPsvKygoeHh7IysrSdRQiItIDgipi5eXlyMzM5EAPIiICoOeXE5ctW4aRI0fC2dlZcU/syZMnmDJliq6jERGRHtDrInbv3j3Mnj0bBQUFaN26NXr37o0jR46gbdu2uo5GRER6QK+L2Pbt23UdgYiI9Jig7okRERE9i0WMiIgEi0WMiIgES6/viRERCY3IyAjnClWf9fqxqT2KCuVKy5wsjeBgJq9jC3oWixgRkRbll1Uj6my+yu3/mhVFeVacDQPs4GAm0nY0g8TLiUREJFg8EzNgfBcYERk6FjEDxneBEZGh4+VEIiISLBYxIiISLBYxIiISLBYxIiISLBYxIiISLBYxIiISLBYxIiISLD4npqf4oDLRy0vd+Rdr87LMv8gipqf4oDLRy0vd+Rdr87LMv8jLiUREJFgsYkREJFgsYkREJFiCKGLbtm1D9+7dIRaL4efnh1OnTuk6EhER6QG9L2KJiYkICwvDhx9+iN9++w19+/bFxIkTcfv2bV1HIw2IZDIMOXsYH+1YjiFnD0Mk41BKIlKf3o9OjImJwdSpUzF9+nQAwPr16/Hvf/8b27dvx4oVK7S+P20MbW9ubozbtbxyXB2GPDxeJJPhi+g56H01DeaVUvifOIhznt5474OvIDfS+39XEQnCyzJMX6+LWEVFBdLT07FgwQKl5W+88QbS0tIaZZ/aGtoedUkK4MX7MeTh8W+cP6IoYABgXilFr6v/weDzR3C0zwiV+zEzM2+siPR/eIwbX2MdY20M0/9soBh3SzUrYo1dCEUSiURvy2xubi48PT3xr3/9Cz4+Porla9euxYEDB3Du3DkdpqMXZf7++zDbsaPGcunMmSiPjtZBIiISKkFcuxGJlB/Yk8vlNZaRcFQNGgS5ufK/PuXm5qgaNEg3gYhIsPS6iLVq1QrGxsZ48OCB0vL8/HzY2dnpKBVpqsrfH1UDBigKmdzcHFUDBqBq7FgdJyMiodHrImZqaoqePXsiJSVFaXlKSgq8vb11lIo0ZmSEJ3FxePLVV5DOnIknX32FJ3FxAAd1EJGa9HpgBwC8++67mDt3Lnr16gVvb29s374d9+/fx4wZM3QdjTRhZISqgABUBQToOgkRCZje/9N3woQJiIqKwvr16+Hr64szZ84gLi4Obdu21WmukydPYvLkyfD09IStrS12796tWFdZWYkVK1agf//+cHR0ROfOnTF79mw+2/YC6jvOz1u4cCFsbW2xadOmJkwofKoc4xs3buCtt95C27Zt4eDggIEDB+L69es6SCtMDR3jkpISLF68GF26dEGbNm3Qu3dvxMTE6CitsOh9EQOA2bNn4/Lly3jw4AGOHTumNFJRV0pLS9GlSxesWbMGFhYWSuuePHmCS5cuITQ0FMeOHcOePXtw9+5dBAcHo6qqSkeJham+4/ysH374ARcuXICDg0MTpjMMDR3j7OxsjBgxAu3atUNSUhJOnz6NZcuWwdLSUgdphamhY7x06VL88ssv+PLLL5GWloYPP/wQkZGR2Ldvnw7SCoteD7EXCicnJ6xbtw7Tpk2rs821a9fQr18/nDx5El27dm3CdIajruOck5ODESNG4Pvvv0dwcDDmzJlT49lCUk1tx3j27NkQiUT4+uuvdZjMcNR2jF9//XX4+/vjo48+UiwbPXo0unbtivXr1+sipmAI4kzMEBQXFwMAbG1tdRvEwFRVVWH27NkIDQ1F586ddR3H4MhkMhw6dAidO3dGUFAQXF1dMXjwYCQmJuo6mkHp168fDh06hDt37gAA0tLScOXKFQwZMkTHyfQfi1gTqKiowLJlyzBy5Eg4OTnpOo5BiYqKQosWLTBr1ixdRzFIDx8+RElJCaKjozF48GAcPHgQQUFBePvtt3Ho0CFdxzMYa9euhZeXF7p164bWrVtjzJgx+PjjjzFy5EhdR9N7ej86UeiqqqowZ84cFBUVYe/evbqOY1BOnDiBPXv24Pjx47qOYrBk/zcx8+jRozF//nwAQPfu3ZGeno5t27bxR1ZLtm7dirS0NOzduxcuLi44deoUIiIi0LZtWwwdOlTX8fQai1gjqqqqwqxZs5CRkYHk5GS0bNlS15EMyvHjx3H//n2ly4jV1dVYsWIFYmNjkZGRocN0hqFVq1Zo1qxZjUu1nTp14iVFLSkrK8PKlSvxz3/+E6NGjQIAdOvWDZcvX8amTZtYxBrAItZIKisrMXPmTFy9ehXJyckQi8W6jmRwZs+ejXHjxiktCwoKQlBQkOKtB6QZU1NTvPbaa8jMzFRafuPGDbi4uOgolWGprKxEZWUljI2NlZYbGxsrzoSpbixiL6ikpARZWVkA/rrkcufOHfz+++9o0aIFHBwcMH36dFy8eBF79+6FSCRCXl4eAMDGxqbeoeKkrL7j7OLiUmP6sWbNmkEsFsPd3V0XcQWpoWP83nvvYcaMGejfvz8GDhyI48ePIzExsd5n9khZQ8fYx8cHkZGRsLS0hIuLC06ePIl9+/YhMjJSx8n1H4fYv6Djx4/D39+/xvIpU6YgLCwMPXr0qHW7mJiYeofik7L6jnNsbGyN5V5eXhxiryZVjvHu3bsRHR2Nu3fvomPHjvjggw8QHBzc1FEFq6FjnJeXh8jISKSkpODRo0dwcXHB3//+d8yfP5+TnTeARYyIiASLQ+yJiEiwWMSIiEiwWMSIiEiwWMSIiEiwWMSIiEiwWMSIiEiwWMSImkhUVBTfYkCkZSxiJCh//PEH/ud//gdeXl4Qi8Xw8PDA6NGjERUVpetoOvXkyRM4OzvD1tYWp06d0nUcoibDIkaCcebMGQwePBjnz5/H1KlTsX79esyYMQOWlpbYsGGDruPpVHJyMsrKyuDs7Iy4uDhdxyFqMpw7kQQjOjoar7zyClJTU9GqVSuldbm5uTpKpR/i4uIwcOBA9OnTB1999RXWrVsHU1PTJs1QUVEBY2PjGhPZEjUmnomRYNy6dQuenp41ChgAODg4KH3+6aefMGnSJHh6esLe3h7dunXDihUrIJVKldqFhIRALBbj3r17mDp1KpydndG5c2ds3rwZAHDz5k0EBQXByckJnp6e2LFjh9L2x48fh62tLeLi4vDpp5/Cw8MDDg4OCAwMrDHze11SUlIwduxYODs7w9HREWPHjkVaWprKx+XBgwdITU1FUFAQgoODIZFIcPjwYcX6TZs2wdbWFrdu3aqxbW3rbt68iZkzZ8LV1RX29vbo378/du3aVef3XrduHbp164Y2bdrgzp07qKiowOrVqzFo0CC0a9cObdq0wZAhQ/DTTz/V2H95eTk++ugjuLm5wdHREePHj0dmZia8vLwQEhKi1Pbx48dYtmwZvLy8FP9PP/744xr/T+nlwiJGgtG2bVtcvnwZly9fbrDtrl27YGxsjDlz5mDt2rUYMGAANm3ahHfffbdGW5lMhuDgYNjZ2SEyMhJubm5YtmwZdu7cicDAQHTq1AmRkZEQi8V4//33cenSpRp9/O///i+SkpIwf/58vPvuuzh//jz8/f1RWFhYb874+HgEBQXB2NgYS5cuxdKlS1FYWIiAgACcO3dOpeMSHx8PIyMjjB07Fp06dUL37t2VLimOHz8eIpEICQkJNbZNSEjAq6++ig4dOgAArl+/jiFDhuDSpUt49913ERUVBRcXF8yfPx9btmypsf3nn3+O77//HnPnzkVkZCSsrKxQXFyMHTt2oE+fPoiIiMDSpUtRWVmJadOm4d///rfS9k/7HThwIFauXAknJycEBgbiyZMnSu3KysowduxYfPfdd5gwYQLWrVuHESNGYPPmzZgxY4ZKx4kMEy8nkmC89957GD9+PPz8/PDqq6/i9ddfh6+vL/z8/GBubq7Udtu2bXjllVcUn2fMmAFXV1d8+umniIyMhJOTk2JdZWUlAgMD8Y9//AMAFGdwCxcuxBdffIG//e1vAIAxY8agW7du2LVrV423FDx8+BBnz55VjD709fXFuHHjsHnzZixfvrzW71NaWorQ0FBMmjRJaUb+GTNmoF+/fli5ciWSkpIaPC5xcXEYMmSIYt/BwcFYvXo1JBIJbG1t4ezsjH79+iExMRGhoaGK7W7duoX09HR88sknimVhYWEQi8VISUlRHL9Zs2ZhxowZiIqKwvTp02FpaaloX1xcjLS0NKVl1dXV+OOPP2BmZqZYNnfuXPj6+mLTpk0YMmQIAODSpUuIj4/HzJkzER0drWi7atWqGvc4t2zZgszMTKSmpiq9oNPT0xOhoaE4deoU+vfv3+CxIsPDMzESDD8/P/z8888YOXIkrl+/js2bN2PSpEno1KlTjctdT3+AZTIZioqKUFBQgP79+0Mul9d6JvX3v/9d8d9WVlbw8PCAsbExJk+erFju4OAAJycnZGdn19h+8uTJSsPn/fz84OnpiV9++aXO75OSkgKJRII333wTBQUFir+ysjIMGjQIp0+fRmVlZb3H5L///S/S09MRFBSkWBYUFISKigr88MMPimUTJkxARkYGrl27pliWkJAAkUiE8ePHAwAkEglSU1MRGBiIsrIypUxDhw5FcXExLl68WON7P1vAgL9e5vi0gFVUVODRo0coLi6Gj48P0tPTFe1+/fVXAMCcOXOUtn/+MiIAHDx4EN7e3mjdurVSrkGDBgEAfvvtt3qPExkunomRoHh7e2PPnj2orq7GlStXcPjwYWzevBnz58+Hi4sL/Pz8AABXr17F8uXLceLECZSVlSn1UVRUpPTZxMQEbdq0UVpmY2MDsVgMExOTGsslEkmNXK6urrUuO378eJ3f5ebNmwCgKCK1KSoqQuvWretcv3//fpiYmKBr1674888/Fcu9vLywf/9+xRuuAwMDERYWhoSEBCxduhQAkJiYiH79+sHZ2VmRRy6XY+3atVi7dm2t+8vPz1f63L59+1rb7dy5E1u2bMH169chl///tz09+26s27dvQyQSoWPHjkrbtmrVqsbzdDdv3sSVK1dqPc615aKXB4sYCZKxsTF69OiBHj16wNvbG+PGjUNcXBz8/PxQVFQEf39/WFhYICIiAh06dICFhQXu3buHefPm1Xjlu5FR7Rck6lr+7I/yU7W9uLC2ds96mmPLli1wdHSstY2NjU2d28vlchw4cACVlZV4/fXXa810+/ZtxRuwBw4ciIMHD2Lp0qW4du0aMjIysH79+hp55s2bh+HDh9e6zy5duih9ru0t5fHx8XjvvfcwatQoLFy4EHZ2dmjWrBl2796NAwcO1Pl9nv9uz5LJZBg4cCA++OCDWtvXdfzI8LGIkeD16tULAHD//n0Af42cy8/PR3JyMgYMGKBol5KS0mgZbty4UWNZVlYWXFxc6tzm6WCK1q1bKy6LqeP06dPIyclBWFgYunXrprSuqqoKs2bNwoEDBxQ//BMmTMCCBQuQnp6O5ORkGBsbY9y4cYptnp5VNWvW7IXyPJWYmIj27dtjz549SsV99+7dSu1cXFwgl8uRlZUFDw8PxfKCgoIaZ8sdOnRASUmJRrnIMPGeGAnGsWPHapxFAcCRI0cAAO7u7gCgeE7p2X/Ny2QyxMTENFq2ffv2KV1mPHbsGK5evYphw4bVuc2QIUPQvHlzbNiwodZh4g1dItu/fz/Mzc2xYMECjB07VukvMDAQ3t7eSqMU/f39YWpqioMHD+LgwYPw9fWFvb29Yv3Ts7V//vOfuHPnjtp5nqrt+GdnZyM5OVmp3dChQwEAX331ldLyZwe5PDVhwgRcuHCh1mH6ZWVlKCkpUSkbGR6eiZFghIWFoaSkBGPHjkXnzp0hk8lw6dIl7N+/Hy1btlQMCOjXr5/i89y5c9GsWTMkJSU16g+dnZ0dRo4cibfeegtFRUX48ssvYW9vj/nz59e5jbW1NTZu3IhZs2ZhwIABmDhxIsRiMe7evYvjx4/D0tIS8fHxtW77dODGwIEDawyseGrUqFGIiIjApUuX0KNHD9ja2mLIkCHYvn07iouLsXDhwhrbREdHY8SIEfDx8cH06dPh6uqKgoICXLp0CUePHsXt27cbPBajRo3Cjz/+iClTpmDUqFG4d+8evvnmG7i6uuLKlSuKdj169MCECROwfft2SCQS+Pj44OLFi4qH2Z89i1uwYAF++eUX/O1vf8Obb76JXr16QSqV4saNGzh48CAOHDiAPn36NJiNDA+LGAnGJ598gqSkJBw9ehS7du2CVCpFmzZtMHHiRHz44Ydo164dAKBFixaIi4vDsmXLEBUVBUtLSwQEBGDmzJnw8fFplGyLFi1CZmYmNm/eDIlEAm9vb6xbt67WB7OfFRgYCAcHB0RHR2PLli0oKyuDWCxG7969lUZMPu/QoUOQSCQYNWpUnW1GjhyJiIgIxMXFKR4JCA4Oxs8//wwTExMEBATU2MbNzQ2pqalYt24dDhw4gPz8fLRq1QqdO3dWGopfn6lTpyI/Px/ffPMNUlNT0bFjR3z66afIyspSKmLAX/cD7e3tceDAARw6dAh9+/bFwYMHMWLECKXHJiwsLJCUlISNGzciMTERCQkJsLS0RPv27RESEqI4C6eXj0gikdR/95mI6nT8+HH4+/vjm2++URrmTi9OIpGgffv2WLZsmdJzbUS14T0xItKZ5x9/AKCY8mvgwIFNHYcEiJcTiUhnNm7ciLNnz8LPzw8WFhY4efIkvv/+ewwbNgx9+/bVdTwSABYxItIZb29vHDt2DJ9//jlKSkrQpk0bLFiwAOHh4bqORgLBe2JERCRYvCdGRESCxSJGRESCxSJGRESCxSJGRESCxSJGRESCxSJGRESC9f8AYmwpwm6NE0IAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "averages_tbl = pd.DataFrame({'Sample Average':sample_averages})\n", "\n", "observed_statistic = 13.667\n", "\n", "unit = ''\n", "\n", "fig, ax = plt.subplots()\n", "\n", "ax.hist(averages_tbl, bins=20,density=True, alpha=0.8, ec='white', zorder=5)\n", "\n", "ax.scatter(observed_statistic, 0, color='red', s=30, zorder=10).set_clip_on(False)\n", "\n", "y_vals = ax.get_yticks()\n", "\n", "y_label = 'Percent per ' + (unit if unit else 'unit')\n", "\n", "x_label = 'Sample Average'\n", "\n", "ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])\n", "\n", "plt.ylabel(y_label)\n", "\n", "plt.xlabel(x_label)\n", "\n", "plt.title('');\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we said earlier, small values of the test statistic will make us lean towards the alternative hypothesis, that the average score in the section is too low for it to look like a random sample from the class.\n", "\n", "Is the observed statistic of 13.667 \"too low\" in relation to this distribution? In other words, is the red far enough out into the left hand tail of the histogram for you to think that it is \"too far\"? \n", "\n", "It's up to you to decide! Use your judgment. Go ahead – it's OK to do so." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Conventional Cut-offs and the P-value ###\n", "If you don't want to make your own judgment, there are conventions that you can follow. These conventions tell us how far out into the tails is considered \"too far\". \n", "\n", "The conventions are based on the area in the tail, starting at the observed statistic (the red dot) and looking in the direction that makes us lean toward the alternative (the left side, in this example). If the area of the tail is small, the observed statistic is far away from the values most commonly predicted by the null hypothesis.\n", "\n", "Remember that in a histogram, area represents percent. To find the area in the tail, we have to find the percent of sample averages that were less than or equal to the average score of Section 3, where the red dot is. The array `sample_averages` contains the averages for all 10,000 repetitions of the random sampling, and `observed_statistic` is 13.667, the average score of Section 3." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.0539" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.count_nonzero(sample_averages <= observed_statistic) / repetitions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "About 5.7% of the simulated random sample averages were 13.667 or below. If we had drawn the students of Section 3 at random from the whole class, the chance that their average would be 13.667 or lower is about 5.7%.\n", "\n", "This chance has an impressive name. It is called the *observed significance level* of the test. That's a mouthful, and so it is commonly called the *P-value* of the test. \n", "\n", "**Definition:** The P-value of a test is the chance, based on the model in the null hypothesis, that the test statistic will be equal to the observed value in the sample or even further in the direction that supports the alternative.**\n", "\n", "If a P-value is small, that means the tail beyond the observed statistic is small and so the observed statistic is far away from what the null predicts. This implies that the data support the alternative hypothesis better than they support the null. \n", "\n", "How small is \"small\"? According to the conventions:\n", "\n", "- If the P-value is less than 5%, it is considered small and the result is called \"statistically significant.\"\n", "\n", "- If the P-value is even smaller – less than 1% – the result is called \"highly statistically significant.\"\n", "\n", "By this convention, our P-value of 5.7% is not considered small. So we have to conclude that the GSI's defense holds good – the average score of Section 3 is like those generated by random chance. Formally, the result of the test is not statistically significant.\n", "\n", "When you make a conclusion in this way, we recommend that you don't just say whether or not the result is statistically significant. Along with your conclusion, provide the observed statistic and the P-value as well, so that readers can use their own judgment." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Historical Note on the Conventions ###\n", "\n", "The determination of statistical significance, as defined above, has become standard in statistical analyses in all fields of application. When a convention is so universally followed, it is interesting to examine how it arose. \n", "\n", "The method of statistical testing – choosing between hypotheses based on data in random samples – was developed by Sir Ronald Fisher in the early 20th century. Sir Ronald might have set the convention for statistical significance somewhat unwittingly, in the following statement in his 1925 book *Statistical Methods for Research Workers*. About the 5% level, he wrote, \"It is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not.\"\n", "\n", "What was \"convenient\" for Sir Ronald became a cutoff that has acquired the status of a universal constant. No matter that Sir Ronald himself made the point that the value was his personal choice from among many: in an article in 1926, he wrote, \"If one in twenty does not seem high enough odds, we may, if we prefer it draw the line at one in fifty (the 2 percent point), or one in a hundred (the 1 percent point). Personally, the author prefers to set a low standard of significance at the 5 percent point ...\"\n", "\n", "Fisher knew that \"low\" is a matter of judgment and has no unique definition. We suggest that you follow his excellent example. Provide your data, make your judgment, and explain why you made it.\n", "\n", "Whether you use a conventional cutoff or your own judgment, it is important to keep the following points in mind.\n", "\n", "- Always provide the observed value of the test statistic and the P-value, so that readers can decide whether or not they think the P-value is small.\n", "- Don't look to defy convention only when the conventionally derived result is not to your liking.\n", "- Even if a test concludes that the data don't support the chance model in the null hypothesis, it typically doesn't explain *why* the model doesn't work." ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.12" } }, "nbformat": 4, "nbformat_minor": 2 }