Home Explore Help
Sign In
cmpsupport
/
data8
1
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Tree: ba88235061
Branches Tags
data8
/
_build
/
chapters
/
17
/
2
/
Training_and_Testing.html

Training_and_Testing.html 16 KB
History Raw

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352 
  1. ---
    2. 

  2. redirect_from:
    3. 

  3.   - "/chapters/17/2/training-and-testing"
    4. 

  4. interact_link: content/chapters/17/2/Training_and_Testing.ipynb
    5. 

  5. kernel_name: python3
    6. 

  6. has_widgets: false
    7. 

  7. title: |-
    8. 

  8.   Training and Testing
    9. 

  9. prev_page:
    10. 

  10.   url: /chapters/17/1/Nearest_Neighbors.html
    11. 

  11.   title: |-
    12. 

  12.     Nearest Neighbors
    13. 

  13. next_page:
    14. 

  14.   url: /chapters/17/3/Rows_of_Tables.html
    15. 

  15.   title: |-
    16. 

  16.     Rows of Tables
    17. 

  17. comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***"
    18. 

  18. ---
    19. 

  19. <div class="jb_cell tag_remove_input">
    20. 

  20. 

  21. <div class="cell border-box-sizing code_cell rendered">
    22. 

  22. 

  23. </div>
    24. 

  24. </div>
    25. 

  25. 

  26. <div class="jb_cell tag_remove_input">
    27. 

  27. 

  28. <div class="cell border-box-sizing code_cell rendered">
    29. 

  29. 

  30. </div>
    31. 

  31. </div>
    32. 

  32. 

  33. <div class="jb_cell tag_remove_input">
    34. 

  34. 

  35. <div class="cell border-box-sizing code_cell rendered">
    36. 

  36. 

  37. </div>
    38. 

  38. </div>
    39. 

  39. 

  40. <div class="jb_cell tag_remove_input">
    41. 

  41. 

  42. <div class="cell border-box-sizing code_cell rendered">
    43. 

  43. 

  44. </div>
    45. 

  45. </div>
    46. 

  46. 

  47. <div class="jb_cell">
    48. 

  48. 

  49. <div class="cell border-box-sizing code_cell rendered">
    50. 

  50. <div class="input">
    51. 

  51. 

  52. <div class="inner_cell">
    53. 

  53.     <div class="input_area">
    54. 

  54. <div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># HIDDEN </span>
    55. 

  55. <span class="n">ckd</span> <span class="o">=</span> <span class="n">Table</span><span class="o">.</span><span class="n">read_table</span><span class="p">(</span><span class="n">path_data</span> <span class="o">+</span> <span class="s1">&#39;ckd.csv&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">relabeled</span><span class="p">(</span><span class="s1">&#39;Blood Glucose Random&#39;</span><span class="p">,</span> <span class="s1">&#39;Glucose&#39;</span><span class="p">)</span>
    56. 

  56. <span class="n">ckd</span> <span class="o">=</span> <span class="n">Table</span><span class="p">()</span><span class="o">.</span><span class="n">with_columns</span><span class="p">(</span>
    57. 

  57.     <span class="s1">&#39;Hemoglobin&#39;</span><span class="p">,</span> <span class="n">standard_units</span><span class="p">(</span><span class="n">ckd</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s1">&#39;Hemoglobin&#39;</span><span class="p">)),</span>
    58. 

  58.     <span class="s1">&#39;Glucose&#39;</span><span class="p">,</span> <span class="n">standard_units</span><span class="p">(</span><span class="n">ckd</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s1">&#39;Glucose&#39;</span><span class="p">)),</span>
    59. 

  59.     <span class="s1">&#39;White Blood Cell Count&#39;</span><span class="p">,</span> <span class="n">standard_units</span><span class="p">(</span><span class="n">ckd</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s1">&#39;White Blood Cell Count&#39;</span><span class="p">)),</span>
    60. 

  60.     <span class="s1">&#39;Class&#39;</span><span class="p">,</span> <span class="n">ckd</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="s1">&#39;Class&#39;</span><span class="p">)</span>
    61. 

  61. <span class="p">)</span>
    62. 

  62. <span class="n">color_table</span> <span class="o">=</span> <span class="n">Table</span><span class="p">()</span><span class="o">.</span><span class="n">with_columns</span><span class="p">(</span>
    63. 

  63.     <span class="s1">&#39;Class&#39;</span><span class="p">,</span> <span class="n">make_array</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span>
    64. 

  64.     <span class="s1">&#39;Color&#39;</span><span class="p">,</span> <span class="n">make_array</span><span class="p">(</span><span class="s1">&#39;darkblue&#39;</span><span class="p">,</span> <span class="s1">&#39;gold&#39;</span><span class="p">)</span>
    65. 

  65. <span class="p">)</span>
    66. 

  66. <span class="n">ckd</span> <span class="o">=</span> <span class="n">ckd</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">&#39;Class&#39;</span><span class="p">,</span> <span class="n">color_table</span><span class="p">)</span>
    67. 

  67. </pre></div>
    68. 

  68. 

  69.     </div>
    70. 

  70. </div>
    71. 

  71. </div>
    72. 

  72. 

  73. </div>
    74. 

  74. </div>
    75. 

  75. 

  76. <div class="jb_cell">
    77. 

  77. 

  78. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    79. 

  79. <div class="text_cell_render border-box-sizing rendered_html">
    80. 

  80. <h3 id="Training-and-Testing">Training and Testing<a class="anchor-link" href="#Training-and-Testing"> </a></h3><p>How good is our nearest neighbor classifier? To answer this we'll need to find out how frequently our classifications are correct. If a patient has chronic kidney disease, how likely is our classifier to pick that up?</p>
    81. 

  81. <p>If the patient is in our training set, we can find out immediately. We already know what class the patient is in. So we can just compare our prediction and the patient's true class.</p>
    82. 

  82. <p>But the point of the classifier is to make predictions for <em>new</em> patients not in our training set. We don't know what class these patients are in but we can make a prediction based on our classifier. How to find out whether the prediction is correct?</p>
    83. 

  83. <p>One way is to wait for further medical tests on the patient and then check whether or not our prediction agrees with the test results. With that approach, by the time we can say how likely our prediction is to be accurate, it is no longer useful for helping the patient.</p>
    84. 

  84. <p>Instead, we will try our classifier on some patients whose true classes are known.  Then, we will compute the proportion of the time our classifier was correct.  This proportion will serve as an estimate of the proportion of all new patients whose class our classifier will accurately predict.  This is called <em>testing</em>.</p>
    85. 

  85. 

  86. </div>
    87. 

  87. </div>
    88. 

  88. </div>
    89. 

  89. </div>
    90. 

  90. 

  91. <div class="jb_cell">
    92. 

  92. 

  93. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    94. 

  94. <div class="text_cell_render border-box-sizing rendered_html">
    95. 

  95. <h3 id="Overly-Optimistic-&quot;Testing&quot;">Overly Optimistic "Testing"<a class="anchor-link" href="#Overly-Optimistic-&quot;Testing&quot;"> </a></h3><p>The training set offers a very tempting set of patients on whom to test out our classifier, because we know the class of each patient in the training set.</p>
    96. 

  96. <p>But let's be careful ... there will be pitfalls ahead if we take this path. An example will show us why.</p>
    97. 

  97. <p>Suppose we use a 1-nearest neighbor classifier to predict whether a patient has chronic kidney disease, based on glucose and white blood cell count.</p>
    98. 

  98. 

  99. </div>
    100. 

  100. </div>
    101. 

  101. </div>
    102. 

  102. </div>
    103. 

  103. 

  104. <div class="jb_cell">
    105. 

  105. 

  106. <div class="cell border-box-sizing code_cell rendered">
    107. 

  107. <div class="input">
    108. 

  108. 

  109. <div class="inner_cell">
    110. 

  110.     <div class="input_area">
    111. 

  111. <div class=" highlight hl-ipython3"><pre><span></span><span class="n">ckd</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="s1">&#39;White Blood Cell Count&#39;</span><span class="p">,</span> <span class="s1">&#39;Glucose&#39;</span><span class="p">,</span> <span class="n">group</span><span class="o">=</span><span class="s1">&#39;Color&#39;</span><span class="p">)</span>
    112. 

  112. </pre></div>
    113. 

  113. 

  114.     </div>
    115. 

  115. </div>
    116. 

  116. </div>
    117. 

  117. 

  118. <div class="output_wrapper">
    119. 

  119. <div class="output">
    120. 

  120. 

  121. <div class="jb_output_wrapper }}">
    122. 

  122. <div class="output_area">
    123. 

  123. 

  124. 

  125. 

  126. <div class="output_png output_subarea ">
    127. 

  127. <img src="../../../images/chapters/17/2/Training_and_Testing_7_0.png"
    128. 

  128. >
    129. 

  129. </div>
    130. 

  130. 

  131. </div>
    132. 

  132. </div>
    133. 

  133. </div>
    134. 

  134. </div>
    135. 

  135. 

  136. </div>
    137. 

  137. </div>
    138. 

  138. 

  139. <div class="jb_cell">
    140. 

  140. 

  141. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    142. 

  142. <div class="text_cell_render border-box-sizing rendered_html">
    143. 

  143. <p>Earlier, we said that we expect to get some classifications wrong, because there's some intermingling of blue and gold points in the lower-left.</p>
    144. 

  144. <p>But what about the points in the training set, that is, the points already on the scatter? Will we ever mis-classify them?</p>
    145. 

  145. <p>The answer is no. Remember that 1-nearest neighbor classification looks for the point <em>in the training set</em> that is nearest to the point being classified. Well, if the point being classified is already in the training set, then its nearest neighbor in the training set is itself! And therefore it will be classified as its own color, which will be correct because each point in the training set is already correctly colored.</p>
    146. 

  146. <p>In other words, <strong>if we use our training set to "test" our 1-nearest neighbor classifier, the classifier will pass the test 100% of the time.</strong></p>
    147. 

  147. <p>Mission accomplished. What a great classifier!</p>
    148. 

  148. <p>No, not so much. A new point in the lower-left might easily be mis-classified, as we noted earlier. "100% accuracy" was a nice dream while it lasted.</p>
    149. 

  149. <p>The lesson of this example is <em>not</em> to use the training set to test a classifier that is based on it.</p>
    150. 

  150. 

  151. </div>
    152. 

  152. </div>
    153. 

  153. </div>
    154. 

  154. </div>
    155. 

  155. 

  156. <div class="jb_cell">
    157. 

  157. 

  158. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    159. 

  159. <div class="text_cell_render border-box-sizing rendered_html">
    160. 

  160. <h3 id="Generating-a-Test-Set">Generating a Test Set<a class="anchor-link" href="#Generating-a-Test-Set"> </a></h3><p>In earlier chapters, we saw that random sampling could be used to estimate the proportion of individuals in a population that met some criterion.  Unfortunately, we have just seen that the training set is not like a random sample from the population of all patients, in one important respect: Our classifier guesses correctly for a higher proportion of individuals in the training set than it does for individuals in the population.</p>
    161. 

  161. <p>When we computed confidence intervals for numerical parameters, we wanted to have many new random samples from a population, but we only had access to a single sample.  We solved that problem by taking bootstrap resamples from our sample.</p>
    162. 

  162. <p>We will use an analogous idea to test our classifier. We will <em>create two samples out of the original training set</em>, use one of the samples as our training set, and <em>the other one for testing</em>.</p>
    163. 

  163. <p>So we will have three groups of individuals:</p>
    164. 

  164. <ul>
    165. 

  165. <li>a training set on which we can do any amount of exploration to build our classifier;</li>
    166. 

  166. <li>a separate testing set on which to try out our classifier and see what fraction of times it classifies correctly;</li>
    167. 

  167. <li>the underlying population of individuals for whom we don't know the true classes; the hope is that our classifier will succeed about as well for these individuals as it did for our testing set.</li>
    168. 

  168. </ul>
    169. 

  169. 

  170. </div>
    171. 

  171. </div>
    172. 

  172. </div>
    173. 

  173. </div>
    174. 

  174. 

  175. <div class="jb_cell">
    176. 

  176. 

  177. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    178. 

  178. <div class="text_cell_render border-box-sizing rendered_html">
    179. 

  179. <p>How to generate the training and testing sets? You've guessed it – we'll select at random.</p>
    180. 

  180. <p>There are 158 individuals in <code>ckd</code>. Let's use a random half of them for training and the other half for testing. To do this, we'll shuffle all the rows, take the first 79 as the training set, and the remaining 79 for testing.</p>
    181. 

  181. 

  182. </div>
    183. 

  183. </div>
    184. 

  184. </div>
    185. 

  185. </div>
    186. 

  186. 

  187. <div class="jb_cell">
    188. 

  188. 

  189. <div class="cell border-box-sizing code_cell rendered">
    190. 

  190. <div class="input">
    191. 

  191. 

  192. <div class="inner_cell">
    193. 

  193.     <div class="input_area">
    194. 

  194. <div class=" highlight hl-ipython3"><pre><span></span><span class="n">shuffled_ckd</span> <span class="o">=</span> <span class="n">ckd</span><span class="o">.</span><span class="n">sample</span><span class="p">(</span><span class="n">with_replacement</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
    195. 

  195. <span class="n">training</span> <span class="o">=</span> <span class="n">shuffled_ckd</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">79</span><span class="p">))</span>
    196. 

  196. <span class="n">testing</span> <span class="o">=</span> <span class="n">shuffled_ckd</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">79</span><span class="p">,</span> <span class="mi">158</span><span class="p">))</span>
    197. 

  197. </pre></div>
    198. 

  198. 

  199.     </div>
    200. 

  200. </div>
    201. 

  201. </div>
    202. 

  202. 

  203. </div>
    204. 

  204. </div>
    205. 

  205. 

  206. <div class="jb_cell">
    207. 

  207. 

  208. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    209. 

  209. <div class="text_cell_render border-box-sizing rendered_html">
    210. 

  210. <p>Now let's construct our classifier based on the points in the training sample:</p>
    211. 

  211. 

  212. </div>
    213. 

  213. </div>
    214. 

  214. </div>
    215. 

  215. </div>
    216. 

  216. 

  217. <div class="jb_cell">
    218. 

  218. 

  219. <div class="cell border-box-sizing code_cell rendered">
    220. 

  220. <div class="input">
    221. 

  221. 

  222. <div class="inner_cell">
    223. 

  223.     <div class="input_area">
    224. 

  224. <div class=" highlight hl-ipython3"><pre><span></span><span class="n">training</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="s1">&#39;White Blood Cell Count&#39;</span><span class="p">,</span> <span class="s1">&#39;Glucose&#39;</span><span class="p">,</span> <span class="n">group</span><span class="o">=</span><span class="s1">&#39;Color&#39;</span><span class="p">)</span>
    225. 

  225. <span class="n">plt</span><span class="o">.</span><span class="n">xlim</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">6</span><span class="p">)</span>
    226. 

  226. <span class="n">plt</span><span class="o">.</span><span class="n">ylim</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">6</span><span class="p">);</span>
    227. 

  227. </pre></div>
    228. 

  228. 

  229.     </div>
    230. 

  230. </div>
    231. 

  231. </div>
    232. 

  232. 

  233. <div class="output_wrapper">
    234. 

  234. <div class="output">
    235. 

  235. 

  236. <div class="jb_output_wrapper }}">
    237. 

  237. <div class="output_area">
    238. 

  238. 

  239. 

  240. 

  241. <div class="output_png output_subarea ">
    242. 

  242. <img src="../../../images/chapters/17/2/Training_and_Testing_13_0.png"
    243. 

  243. >
    244. 

  244. </div>
    245. 

  245. 

  246. </div>
    247. 

  247. </div>
    248. 

  248. </div>
    249. 

  249. </div>
    250. 

  250. 

  251. </div>
    252. 

  252. </div>
    253. 

  253. 

  254. <div class="jb_cell">
    255. 

  255. 

  256. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    257. 

  257. <div class="text_cell_render border-box-sizing rendered_html">
    258. 

  258. <p>We get the following classification regions and decision boundary:</p>
    259. 

  259. 

  260. </div>
    261. 

  261. </div>
    262. 

  262. </div>
    263. 

  263. </div>
    264. 

  264. 

  265. <div class="jb_cell tag_remove_input">
    266. 

  266. 

  267. <div class="cell border-box-sizing code_cell rendered">
    268. 

  268. 

  269. </div>
    270. 

  270. </div>
    271. 

  271. 

  272. <div class="jb_cell tag_remove_input">
    273. 

  273. 

  274. <div class="cell border-box-sizing code_cell rendered">
    275. 

  275. 

  276. </div>
    277. 

  277. </div>
    278. 

  278. 

  279. <div class="jb_cell tag_remove_input">
    280. 

  280. 

  281. <div class="cell border-box-sizing code_cell rendered">
    282. 

  282. 

  283. <div class="output_wrapper">
    284. 

  284. <div class="output">
    285. 

  285. 

  286. <div class="jb_output_wrapper }}">
    287. 

  287. <div class="output_area">
    288. 

  288. 

  289. 

  290. 

  291. <div class="output_png output_subarea ">
    292. 

  292. <img src="../../../images/chapters/17/2/Training_and_Testing_17_0.png"
    293. 

  293. >
    294. 

  294. </div>
    295. 

  295. 

  296. </div>
    297. 

  297. </div>
    298. 

  298. </div>
    299. 

  299. </div>
    300. 

  300. 

  301. </div>
    302. 

  302. </div>
    303. 

  303. 

  304. <div class="jb_cell">
    305. 

  305. 

  306. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    307. 

  307. <div class="text_cell_render border-box-sizing rendered_html">
    308. 

  308. <p>Place the <em>test</em> data on this graph and you can see at once that while the classifier got almost all the points right, there are some mistakes.  For example, some blue points of the test set fall in the gold region of the classifier.</p>
    309. 

  309. 

  310. </div>
    311. 

  311. </div>
    312. 

  312. </div>
    313. 

  313. </div>
    314. 

  314. 

  315. <div class="jb_cell tag_remove_input">
    316. 

  316. 

  317. <div class="cell border-box-sizing code_cell rendered">
    318. 

  318. 

  319. <div class="output_wrapper">
    320. 

  320. <div class="output">
    321. 

  321. 

  322. <div class="jb_output_wrapper }}">
    323. 

  323. <div class="output_area">
    324. 

  324. 

  325. 

  326. 

  327. <div class="output_png output_subarea ">
    328. 

  328. <img src="../../../images/chapters/17/2/Training_and_Testing_19_0.png"
    329. 

  329. >
    330. 

  330. </div>
    331. 

  331. 

  332. </div>
    333. 

  333. </div>
    334. 

  334. </div>
    335. 

  335. </div>
    336. 

  336. 

  337. </div>
    338. 

  338. </div>
    339. 

  339. 

  340. <div class="jb_cell">
    341. 

  341. 

  342. <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
    343. 

  343. <div class="text_cell_render border-box-sizing rendered_html">
    344. 

  344. <p>Some errors notwithstanding, it looks like the classifier does fairly well on the test set. Assuming that the original sample was drawn randomly from the underlying population, the hope is that the classifier will perform with similar accuracy on the overall population, since the test set was chosen randomly from the original sample.</p>
    345. 

  345. 

  346. </div>
    347. 

  347. </div>
    348. 

  348. </div>
    349. 

  349. </div>
    350. 

  350. 

  351.  
    352. 

  352.