Format: application/octet-stream
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "---\n",
8 "\n",
9 "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
10 "\n",
11 "---"
12 ]
13 },
14 {
15 "cell_type": "markdown",
16 "metadata": {},
17 "source": [
18 "# Assignment 2 - Introduction to NLTK\n",
19 "\n",
20 "In part 1 of this assignment you will use nltk to explore the Herman Melville novel Moby Dick. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling. "
21 ]
22 },
23 {
24 "cell_type": "markdown",
25 "metadata": {},
26 "source": [
27 "## Part 1 - Analyzing Moby Dick"
28 ]
29 },
30 {
31 "cell_type": "code",
32 "execution_count": 3,
33 "metadata": {},
34 "outputs": [
35 {
36 "name": "stdout",
37 "output_type": "stream",
38 "text": [
39 "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
40 "[nltk_data] Unzipping tokenizers/punkt.zip.\n",
41 "[nltk_data] Downloading package wordnet to /home/jovyan/nltk_data...\n",
42 "[nltk_data] Unzipping corpora/wordnet.zip.\n",
43 "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
44 "[nltk_data] /home/jovyan/nltk_data...\n",
45 "[nltk_data] Unzipping taggers/averaged_perceptron_tagger.zip.\n",
46 "[nltk_data] Downloading package words to /home/jovyan/nltk_data...\n",
47 "[nltk_data] Unzipping corpora/words.zip.\n"
48 ]
49 }
50 ],
51 "source": [
52 "import nltk\n",
53 "nltk.download('punkt')\n",
54 "nltk.download('wordnet')\n",
55 "nltk.download('averaged_perceptron_tagger')\n",
56 "nltk.download('words')\n",
57 "import pandas as pd\n",
58 "import numpy as np\n",
59 "\n",
60 "# If you would like to work with the raw text you can use 'moby_raw'\n",
61 "with open('moby.txt', 'r') as f:\n",
62 " moby_raw = f.read()\n",
63 " \n",
64 "# If you would like to work with the novel in nltk.Text format you can use 'text1'\n",
65 "moby_tokens = nltk.word_tokenize(moby_raw)\n",
66 "text1 = nltk.Text(moby_tokens)"
67 ]
68 },
69 {
70 "cell_type": "markdown",
71 "metadata": {},
72 "source": [
73 "### Example 1\n",
74 "\n",
75 "How many tokens (words and punctuation symbols) are in text1?\n",
76 "\n",
77 "*This function should return an integer.*"
78 ]
79 },
80 {
81 "cell_type": "code",
82 "execution_count": 2,
83 "metadata": {},
84 "outputs": [
85 {
86 "data": {
87 "text/plain": [
88 "254989"
89 ]
90 },
91 "execution_count": 2,
92 "metadata": {},
93 "output_type": "execute_result"
94 }
95 ],
96 "source": [
97 "def example_one():\n",
98 " \n",
99 " return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)\n",
100 "\n",
101 "example_one()"
102 ]
103 },
104 {
105 "cell_type": "markdown",
106 "metadata": {},
107 "source": [
108 "### Example 2\n",
109 "\n",
110 "How many unique tokens (unique words and punctuation) does text1 have?\n",
111 "\n",
112 "*This function should return an integer.*"
113 ]
114 },
115 {
116 "cell_type": "code",
117 "execution_count": 3,
118 "metadata": {},
119 "outputs": [
120 {
121 "data": {
122 "text/plain": [
123 "20755"
124 ]
125 },
126 "execution_count": 3,
127 "metadata": {},
128 "output_type": "execute_result"
129 }
130 ],
131 "source": [
132 "def example_two():\n",
133 " \n",
134 " return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))\n",
135 "\n",
136 "example_two()"
137 ]
138 },
139 {
140 "cell_type": "markdown",
141 "metadata": {},
142 "source": [
143 "### Example 3\n",
144 "\n",
145 "After lemmatizing the verbs, how many unique tokens does text1 have?\n",
146 "\n",
147 "*This function should return an integer.*"
148 ]
149 },
150 {
151 "cell_type": "code",
152 "execution_count": 4,
153 "metadata": {},
154 "outputs": [
155 {
156 "data": {
157 "text/plain": [
158 "16900"
159 ]
160 },
161 "execution_count": 4,
162 "metadata": {},
163 "output_type": "execute_result"
164 }
165 ],
166 "source": [
167 "from nltk.stem import WordNetLemmatizer\n",
168 "\n",
169 "def example_three():\n",
170 "\n",
171 " lemmatizer = WordNetLemmatizer()\n",
172 " lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]\n",
173 "\n",
174 " return len(set(lemmatized))\n",
175 "\n",
176 "example_three()"
177 ]
178 },
179 {
180 "cell_type": "markdown",
181 "metadata": {},
182 "source": [
183 "### Question 1\n",
184 "\n",
185 "What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)\n",
186 "\n",
187 "*This function should return a float.*"
188 ]
189 },
190 {
191 "cell_type": "code",
192 "execution_count": 5,
193 "metadata": {},
194 "outputs": [
195 {
196 "data": {
197 "text/plain": [
198 "0.08139566804842562"
199 ]
200 },
201 "execution_count": 5,
202 "metadata": {},
203 "output_type": "execute_result"
204 }
205 ],
206 "source": [
207 "def answer_one():\n",
208 " \n",
209 " return len(set(text1))/len(text1)\n",
210 "\n",
211 "answer_one()"
212 ]
213 },
214 {
215 "cell_type": "markdown",
216 "metadata": {},
217 "source": [
218 "### Question 2\n",
219 "\n",
220 "What percentage of tokens is 'whale'or 'Whale'?\n",
221 "\n",
222 "*This function should return a float.*"
223 ]
224 },
225 {
226 "cell_type": "code",
227 "execution_count": 6,
228 "metadata": {},
229 "outputs": [
230 {
231 "data": {
232 "text/plain": [
233 "0.4125668166077752"
234 ]
235 },
236 "execution_count": 6,
237 "metadata": {},
238 "output_type": "execute_result"
239 }
240 ],
241 "source": [
242 "def answer_two():\n",
243 " \n",
244 " dist = nltk.FreqDist(text1)\n",
245 " return (dist['whale'] + dist['Whale'])/len(text1)*100\n",
246 "\n",
247 "answer_two()"
248 ]
249 },
250 {
251 "cell_type": "markdown",
252 "metadata": {},
253 "source": [
254 "### Question 3\n",
255 "\n",
256 "What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?\n",
257 "\n",
258 "*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*"
259 ]
260 },
261 {
262 "cell_type": "code",
263 "execution_count": 7,
264 "metadata": {},
265 "outputs": [
266 {
267 "data": {
268 "text/plain": [
269 "[(',', 19204),\n",
270 " ('the', 13715),\n",
271 " ('.', 7308),\n",
272 " ('of', 6513),\n",
273 " ('and', 6010),\n",
274 " ('a', 4545),\n",
275 " ('to', 4515),\n",
276 " (';', 4173),\n",
277 " ('in', 3908),\n",
278 " ('that', 2978),\n",
279 " ('his', 2459),\n",
280 " ('it', 2196),\n",
281 " ('I', 2097),\n",
282 " ('!', 1767),\n",
283 " ('is', 1722),\n",
284 " ('--', 1713),\n",
285 " ('with', 1659),\n",
286 " ('he', 1658),\n",
287 " ('was', 1639),\n",
288 " ('as', 1620)]"
289 ]
290 },
291 "execution_count": 7,
292 "metadata": {},
293 "output_type": "execute_result"
294 }
295 ],
296 "source": [
297 "def answer_three():\n",
298 " \n",
299 " return nltk.FreqDist(text1).most_common(20)\n",
300 "\n",
301 "answer_three()"
302 ]
303 },
304 {
305 "cell_type": "markdown",
306 "metadata": {},
307 "source": [
308 "### Question 4\n",
309 "\n",
310 "What tokens have a length of greater than 5 and frequency of more than 150?\n",
311 "\n",
312 "*This function should return an alphabetically sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*"
313 ]
314 },
315 {
316 "cell_type": "code",
317 "execution_count": 26,
318 "metadata": {},
319 "outputs": [
320 {
321 "data": {
322 "text/plain": [
323 "['Captain',\n",
324 " 'Pequod',\n",
325 " 'Queequeg',\n",
326 " 'Starbuck',\n",
327 " 'almost',\n",
328 " 'before',\n",
329 " 'himself',\n",
330 " 'little',\n",
331 " 'seemed',\n",
332 " 'should',\n",
333 " 'though',\n",
334 " 'through',\n",
335 " 'whales',\n",
336 " 'without']"
337 ]
338 },
339 "execution_count": 26,
340 "metadata": {},
341 "output_type": "execute_result"
342 }
343 ],
344 "source": [
345 "def answer_four():\n",
346 " \n",
347 " dist = nltk.FreqDist(text1)\n",
348 " vocab1 = dist.keys()\n",
349 " freqwords = [w for w in vocab1 if len(w)>5 and dist[w] > 150]\n",
350 " return sorted(freqwords)\n",
351 "\n",
352 "answer_four()"
353 ]
354 },
355 {
356 "cell_type": "markdown",
357 "metadata": {},
358 "source": [
359 "### Question 5\n",
360 "\n",
361 "Find the longest word in text1 and that word's length.\n",
362 "\n",
363 "*This function should return a tuple `(longest_word, length)`.*"
364 ]
365 },
366 {
367 "cell_type": "code",
368 "execution_count": 35,
369 "metadata": {},
370 "outputs": [
371 {
372 "data": {
373 "text/plain": [
374 "(\"twelve-o'clock-at-night\", 23)"
375 ]
376 },
377 "execution_count": 35,
378 "metadata": {},
379 "output_type": "execute_result"
380 }
381 ],
382 "source": [
383 "def answer_five():\n",
384 " \n",
385 " maxlen = max(len(w) for w in text1)\n",
386 " word = [w for w in text1 if len(w) == maxlen]\n",
387 " return (word[0],maxlen)\n",
388 "\n",
389 "answer_five()"
390 ]
391 },
392 {
393 "cell_type": "markdown",
394 "metadata": {},
395 "source": [
396 "### Question 6\n",
397 "\n",
398 "What unique words have a frequency of more than 2000? What is their frequency?\n",
399 "\n",
400 "\"Hint: you may want to use `isalpha()` to check if the token is a word and not punctuation.\"\n",
401 "\n",
402 "*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*"
403 ]
404 },
405 {
406 "cell_type": "code",
407 "execution_count": 11,
408 "metadata": {},
409 "outputs": [
410 {
411 "data": {
412 "text/plain": [
413 "[(13715, 'the'),\n",
414 " (6513, 'of'),\n",
415 " (6010, 'and'),\n",
416 " (4545, 'a'),\n",
417 " (4515, 'to'),\n",
418 " (3908, 'in'),\n",
419 " (2978, 'that'),\n",
420 " (2459, 'his'),\n",
421 " (2196, 'it'),\n",
422 " (2097, 'I')]"
423 ]
424 },
425 "execution_count": 11,
426 "metadata": {},
427 "output_type": "execute_result"
428 }
429 ],
430 "source": [
431 "def answer_six():\n",
432 " dist = nltk.FreqDist(text1)\n",
433 " freqwords = [(dist[w],w) for w in set(text1) if w.isalpha() and dist[w]>2000]\n",
434 " return sorted(freqwords, key=lambda x:x[0],reverse=True)\n",
435 "\n",
436 "answer_six()"
437 ]
438 },
439 {
440 "cell_type": "markdown",
441 "metadata": {},
442 "source": [
443 "### Question 7\n",
444 "\n",
445 "What is the average number of tokens per sentence?\n",
446 "\n",
447 "*This function should return a float.*"
448 ]
449 },
450 {
451 "cell_type": "code",
452 "execution_count": 12,
453 "metadata": {},
454 "outputs": [
455 {
456 "data": {
457 "text/plain": [
458 "25.881952902963864"
459 ]
460 },
461 "execution_count": 12,
462 "metadata": {},
463 "output_type": "execute_result"
464 }
465 ],
466 "source": [
467 "def answer_seven():\n",
468 " text2 = nltk.sent_tokenize(moby_raw)\n",
469 " return len(text1)/len(text2)\n",
470 "\n",
471 "answer_seven()"
472 ]
473 },
474 {
475 "cell_type": "markdown",
476 "metadata": {},
477 "source": [
478 "### Question 8\n",
479 "\n",
480 "What are the 5 most frequent parts of speech in this text? What is their frequency?\n",
481 "\n",
482 "*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*"
483 ]
484 },
485 {
486 "cell_type": "code",
487 "execution_count": 24,
488 "metadata": {},
489 "outputs": [
490 {
491 "data": {
492 "text/plain": [
493 "[('NN', 32730), ('IN', 28657), ('DT', 25867), (',', 19204), ('JJ', 17620)]"
494 ]
495 },
496 "execution_count": 24,
497 "metadata": {},
498 "output_type": "execute_result"
499 }
500 ],
501 "source": [
502 "def answer_eight():\n",
503 " \n",
504 " import collections\n",
505 " pos_list = nltk.pos_tag(text1)\n",
506 " pos_counts = collections.Counter((subl[1] for subl in pos_list))\n",
507 " return pos_counts.most_common(5)\n",
508 "\n",
509 "answer_eight()"
510 ]
511 },
512 {
513 "cell_type": "markdown",
514 "metadata": {},
515 "source": [
516 "## Part 2 - Spelling Recommender\n",
517 "\n",
518 "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
519 "\n",
520 "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
521 "\n",
522 "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
523 "\n",
524 "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
525 ]
526 },
527 {
528 "cell_type": "code",
529 "execution_count": 4,
530 "metadata": {},
531 "outputs": [],
532 "source": [
533 "from nltk.corpus import words\n",
534 "\n",
535 "correct_spellings = words.words()"
536 ]
537 },
538 {
539 "cell_type": "markdown",
540 "metadata": {},
541 "source": [
542 "### Question 9\n",
543 "\n",
544 "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
545 "\n",
546 "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.**\n",
547 "\n",
548 "*This function should return a list of length three:\n",
549 "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
550 ]
551 },
552 {
553 "cell_type": "code",
554 "execution_count": 6,
555 "metadata": {},
556 "outputs": [
557 {
558 "name": "stderr",
559 "output_type": "stream",
560 "text": [
561 "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:9: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
562 " if __name__ == '__main__':\n"
563 ]
564 },
565 {
566 "data": {
567 "text/plain": [
568 "['corpulent', 'indecence', 'validate']"
569 ]
570 },
571 "execution_count": 6,
572 "metadata": {},
573 "output_type": "execute_result"
574 }
575 ],
576 "source": [
577 "def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):\n",
578 " \n",
579 " recommend = []\n",
580 " for entry in entries:\n",
581 " \n",
582 " # Match first letter. input_spell contains all words in correct_spellings with the same first letter.\n",
583 " input_spell = [x for x in correct_spellings if x[0] == entry[0]]\n",
584 " \n",
585 " # Find the jaccard distance between the entry word and every word in correct_spellings with the same first letter.\n",
586 " jaccard_dist = [nltk.jaccard_distance(set(nltk.ngrams(entry,n=3)), set(nltk.ngrams(x,n=3))) for x in input_spell]\n",
587 " \n",
588 " # Recommend the word in input_spell with the minimum Jaccard distance.\n",
589 " recommend.append(input_spell[np.argmin(jaccard_dist)])\n",
590 " \n",
591 " return recommend\n",
592 " \n",
593 "answer_nine()"
594 ]
595 },
596 {
597 "cell_type": "markdown",
598 "metadata": {},
599 "source": [
600 "### Question 10\n",
601 "\n",
602 "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
603 "\n",
604 "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.**\n",
605 "\n",
606 "*This function should return a list of length three:\n",
607 "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
608 ]
609 },
610 {
611 "cell_type": "code",
612 "execution_count": 7,
613 "metadata": {},
614 "outputs": [
615 {
616 "name": "stderr",
617 "output_type": "stream",
618 "text": [
619 "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:10: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
620 " # Remove the CWD from sys.path while we load stuff.\n"
621 ]
622 },
623 {
624 "data": {
625 "text/plain": [
626 "['cormus', 'incendiary', 'valid']"
627 ]
628 },
629 "execution_count": 7,
630 "metadata": {},
631 "output_type": "execute_result"
632 }
633 ],
634 "source": [
635 "def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):\n",
636 " \n",
637 " recommend = []\n",
638 " for entry in entries:\n",
639 " \n",
640 " # Match first letter. input_spell contains all words in correct_spellings with the same first letter.\n",
641 " input_spell = [x for x in correct_spellings if x[0] == entry[0]]\n",
642 " \n",
643 " # Find the jaccard distance between the entry word and every word in correct_spellings with the same first letter.\n",
644 " jaccard_dist = [nltk.jaccard_distance(set(nltk.ngrams(entry,n=4)), set(nltk.ngrams(x,n=4))) for x in input_spell]\n",
645 " \n",
646 " # Recommend the word in input_spell with the minimum Jaccard distance.\n",
647 " recommend.append(input_spell[np.argmin(jaccard_dist)])\n",
648 " \n",
649 " return recommend\n",
650 " \n",
651 "answer_ten()"
652 ]
653 },
654 {
655 "cell_type": "markdown",
656 "metadata": {},
657 "source": [
658 "### Question 11\n",
659 "\n",
660 "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
661 "\n",
662 "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**\n",
663 "\n",
664 "*This function should return a list of length three:\n",
665 "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
666 ]
667 },
668 {
669 "cell_type": "code",
670 "execution_count": 9,
671 "metadata": {},
672 "outputs": [
673 {
674 "data": {
675 "text/plain": [
676 "['corpulent', 'intendence', 'validate']"
677 ]
678 },
679 "execution_count": 9,
680 "metadata": {},
681 "output_type": "execute_result"
682 }
683 ],
684 "source": [
685 "def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):\n",
686 " \n",
687 " recommend = []\n",
688 " for entry in entries:\n",
689 " \n",
690 " # Match first letter. input_spell contains all words in correct_spellings with the same first letter.\n",
691 " input_spell = [x for x in correct_spellings if x[0] == entry[0]]\n",
692 " \n",
693 " # Find the jaccard distance between the entry word and every word in correct_spellings with the same first letter.\n",
694 " DL_dist = [nltk.edit_distance(x, entry, transpositions=True) for x in input_spell]\n",
695 " \n",
696 " # Recommend the word in input_spell with the minimum Jaccard distance.\n",
697 " recommend.append(input_spell[np.argmin(DL_dist)])\n",
698 " \n",
699 " return recommend\n",
700 " \n",
701 "answer_eleven()"
702 ]
703 },
704 {
705 "cell_type": "code",
706 "execution_count": null,
707 "metadata": {
708 "collapsed": true
709 },
710 "outputs": [],
711 "source": []
712 }
713 ],
714 "metadata": {
715 "coursera": {
716 "course_slug": "python-text-mining",
717 "graded_item_id": "r35En",
718 "launcher_item_id": "tCVfW",
719 "part_id": "NTVgL"
720 },
721 "kernelspec": {
722 "display_name": "Python 3",
723 "language": "python",
724 "name": "python3"
725 },
726 "language_info": {
727 "codemirror_mode": {
728 "name": "ipython",
729 "version": 3
730 },
731 "file_extension": ".py",
732 "mimetype": "text/x-python",
733 "name": "python",
734 "nbconvert_exporter": "python",
735 "pygments_lexer": "ipython3",
736 "version": "3.6.2"
737 }
738 },
739 "nbformat": 4,
740 "nbformat_minor": 2
741 }
742