Format: application/octet-stream
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "---\n",
8 "\n",
9 "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
10 "\n",
11 "---"
12 ]
13 },
14 {
15 "cell_type": "markdown",
16 "metadata": {},
17 "source": [
18 "# Assignment 2 - Introduction to NLTK\n",
19 "\n",
20 "In part 1 of this assignment you will use nltk to explore the Herman Melville novel Moby Dick. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling. "
21 ]
22 },
23 {
24 "cell_type": "markdown",
25 "metadata": {},
26 "source": [
27 "## Part 1 - Analyzing Moby Dick"
28 ]
29 },
30 {
31 "cell_type": "code",
32 "execution_count": 2,
33 "metadata": {},
34 "outputs": [
35 {
36 "name": "stdout",
37 "output_type": "stream",
38 "text": [
39 "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
40 "[nltk_data] Unzipping tokenizers/punkt.zip.\n"
41 ]
42 }
43 ],
44 "source": [
45 "import nltk\n",
46 "nltk.download('punkt')\n",
47 "import pandas as pd\n",
48 "import numpy as np\n",
49 "\n",
50 "# If you would like to work with the raw text you can use 'moby_raw'\n",
51 "with open('moby.txt', 'r') as f:\n",
52 " moby_raw = f.read()\n",
53 " \n",
54 "# If you would like to work with the novel in nltk.Text format you can use 'text1'\n",
55 "moby_tokens = nltk.word_tokenize(moby_raw)\n",
56 "text1 = nltk.Text(moby_tokens)"
57 ]
58 },
59 {
60 "cell_type": "markdown",
61 "metadata": {},
62 "source": [
63 "### Example 1\n",
64 "\n",
65 "How many tokens (words and punctuation symbols) are in text1?\n",
66 "\n",
67 "*This function should return an integer.*"
68 ]
69 },
70 {
71 "cell_type": "code",
72 "execution_count": 3,
73 "metadata": {},
74 "outputs": [
75 {
76 "data": {
77 "text/plain": [
78 "254989"
79 ]
80 },
81 "execution_count": 3,
82 "metadata": {},
83 "output_type": "execute_result"
84 }
85 ],
86 "source": [
87 "def example_one():\n",
88 " \n",
89 " return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)\n",
90 "\n",
91 "example_one()"
92 ]
93 },
94 {
95 "cell_type": "markdown",
96 "metadata": {},
97 "source": [
98 "### Example 2\n",
99 "\n",
100 "How many unique tokens (unique words and punctuation) does text1 have?\n",
101 "\n",
102 "*This function should return an integer.*"
103 ]
104 },
105 {
106 "cell_type": "code",
107 "execution_count": 4,
108 "metadata": {},
109 "outputs": [
110 {
111 "data": {
112 "text/plain": [
113 "20755"
114 ]
115 },
116 "execution_count": 4,
117 "metadata": {},
118 "output_type": "execute_result"
119 }
120 ],
121 "source": [
122 "def example_two():\n",
123 " \n",
124 " return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))\n",
125 "\n",
126 "example_two()"
127 ]
128 },
129 {
130 "cell_type": "markdown",
131 "metadata": {},
132 "source": [
133 "### Example 3\n",
134 "\n",
135 "After lemmatizing the verbs, how many unique tokens does text1 have?\n",
136 "\n",
137 "*This function should return an integer.*"
138 ]
139 },
140 {
141 "cell_type": "code",
142 "execution_count": 6,
143 "metadata": {},
144 "outputs": [
145 {
146 "name": "stdout",
147 "output_type": "stream",
148 "text": [
149 "[nltk_data] Downloading package wordnet to /home/jovyan/nltk_data...\n",
150 "[nltk_data] Unzipping corpora/wordnet.zip.\n"
151 ]
152 },
153 {
154 "data": {
155 "text/plain": [
156 "16900"
157 ]
158 },
159 "execution_count": 6,
160 "metadata": {},
161 "output_type": "execute_result"
162 }
163 ],
164 "source": [
165 "from nltk.stem import WordNetLemmatizer\n",
166 "nltk.download('wordnet')\n",
167 "def example_three():\n",
168 "\n",
169 " lemmatizer = WordNetLemmatizer()\n",
170 " lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]\n",
171 "\n",
172 " return len(set(lemmatized))\n",
173 "\n",
174 "example_three()"
175 ]
176 },
177 {
178 "cell_type": "markdown",
179 "metadata": {},
180 "source": [
181 "### Question 1\n",
182 "\n",
183 "What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)\n",
184 "\n",
185 "*This function should return a float.*"
186 ]
187 },
188 {
189 "cell_type": "code",
190 "execution_count": 7,
191 "metadata": {},
192 "outputs": [
193 {
194 "data": {
195 "text/plain": [
196 "0.08139566804842562"
197 ]
198 },
199 "execution_count": 7,
200 "metadata": {},
201 "output_type": "execute_result"
202 }
203 ],
204 "source": [
205 "def answer_one():\n",
206 " \n",
207 " \n",
208 " return example_two()/example_one()\n",
209 "\n",
210 "answer_one()"
211 ]
212 },
213 {
214 "cell_type": "markdown",
215 "metadata": {},
216 "source": [
217 "### Question 2\n",
218 "\n",
219 "What percentage of tokens is 'whale'or 'Whale'?\n",
220 "\n",
221 "*This function should return a float.*"
222 ]
223 },
224 {
225 "cell_type": "code",
226 "execution_count": 9,
227 "metadata": {},
228 "outputs": [
229 {
230 "data": {
231 "text/plain": [
232 "0.4125668166077752"
233 ]
234 },
235 "execution_count": 9,
236 "metadata": {},
237 "output_type": "execute_result"
238 }
239 ],
240 "source": [
241 "def answer_two():\n",
242 " \n",
243 " \n",
244 " return ((text1.vocab()['whale'] + text1.vocab()['Whale']) / example_one())*100\n",
245 "\n",
246 "answer_two()"
247 ]
248 },
249 {
250 "cell_type": "markdown",
251 "metadata": {},
252 "source": [
253 "### Question 3\n",
254 "\n",
255 "What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?\n",
256 "\n",
257 "*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*"
258 ]
259 },
260 {
261 "cell_type": "code",
262 "execution_count": 10,
263 "metadata": {},
264 "outputs": [
265 {
266 "data": {
267 "text/plain": [
268 "[(',', 19204),\n",
269 " ('the', 13715),\n",
270 " ('.', 7308),\n",
271 " ('of', 6513),\n",
272 " ('and', 6010),\n",
273 " ('a', 4545),\n",
274 " ('to', 4515),\n",
275 " (';', 4173),\n",
276 " ('in', 3908),\n",
277 " ('that', 2978),\n",
278 " ('his', 2459),\n",
279 " ('it', 2196),\n",
280 " ('I', 2097),\n",
281 " ('!', 1767),\n",
282 " ('is', 1722),\n",
283 " ('--', 1713),\n",
284 " ('with', 1659),\n",
285 " ('he', 1658),\n",
286 " ('was', 1639),\n",
287 " ('as', 1620)]"
288 ]
289 },
290 "execution_count": 10,
291 "metadata": {},
292 "output_type": "execute_result"
293 }
294 ],
295 "source": [
296 "def answer_three():\n",
297 " import operator\n",
298 " \n",
299 " \n",
300 " return sorted(text1.vocab().items(), key=operator.itemgetter(1), reverse=True)[:20]\n",
301 "\n",
302 "answer_three()"
303 ]
304 },
305 {
306 "cell_type": "markdown",
307 "metadata": {},
308 "source": [
309 "### Question 4\n",
310 "\n",
311 "What tokens have a length of greater than 5 and frequency of more than 150?\n",
312 "\n",
313 "*This function should return an alphabetically sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*"
314 ]
315 },
316 {
317 "cell_type": "code",
318 "execution_count": 11,
319 "metadata": {},
320 "outputs": [
321 {
322 "data": {
323 "text/plain": [
324 "['through',\n",
325 " 'almost',\n",
326 " 'whales',\n",
327 " 'before',\n",
328 " 'without',\n",
329 " 'should',\n",
330 " 'little',\n",
331 " 'seemed',\n",
332 " 'though',\n",
333 " 'himself',\n",
334 " 'Captain',\n",
335 " 'Queequeg',\n",
336 " 'Pequod',\n",
337 " 'Starbuck']"
338 ]
339 },
340 "execution_count": 11,
341 "metadata": {},
342 "output_type": "execute_result"
343 }
344 ],
345 "source": [
346 "def answer_four():\n",
347 " \n",
348 " \n",
349 " return [token for token, freq in text1.vocab().items() if len(token) > 5 and freq > 150]\n",
350 "\n",
351 "answer_four()"
352 ]
353 },
354 {
355 "cell_type": "markdown",
356 "metadata": {},
357 "source": [
358 "### Question 5\n",
359 "\n",
360 "Find the longest word in text1 and that word's length.\n",
361 "\n",
362 "*This function should return a tuple `(longest_word, length)`.*"
363 ]
364 },
365 {
366 "cell_type": "code",
367 "execution_count": 12,
368 "metadata": {},
369 "outputs": [
370 {
371 "data": {
372 "text/plain": [
373 "(\"twelve-o'clock-at-night\", 23)"
374 ]
375 },
376 "execution_count": 12,
377 "metadata": {},
378 "output_type": "execute_result"
379 }
380 ],
381 "source": [
382 "def answer_five():\n",
383 " import operator\n",
384 " \n",
385 " return sorted([(token, len(token))for token, freq in text1.vocab().items()], key=operator.itemgetter(1), reverse=True)[0]\n",
386 "\n",
387 "answer_five()"
388 ]
389 },
390 {
391 "cell_type": "markdown",
392 "metadata": {},
393 "source": [
394 "### Question 6\n",
395 "\n",
396 "What unique words have a frequency of more than 2000? What is their frequency?\n",
397 "\n",
398 "\"Hint: you may want to use `isalpha()` to check if the token is a word and not punctuation.\"\n",
399 "\n",
400 "*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*"
401 ]
402 },
403 {
404 "cell_type": "code",
405 "execution_count": 16,
406 "metadata": {},
407 "outputs": [
408 {
409 "data": {
410 "text/plain": [
411 "[(4515, 'to'),\n",
412 " (13715, 'the'),\n",
413 " (2978, 'that'),\n",
414 " (6513, 'of'),\n",
415 " (2196, 'it'),\n",
416 " (3908, 'in'),\n",
417 " (2459, 'his'),\n",
418 " (6010, 'and'),\n",
419 " (4545, 'a'),\n",
420 " (2097, 'I')]"
421 ]
422 },
423 "execution_count": 16,
424 "metadata": {},
425 "output_type": "execute_result"
426 }
427 ],
428 "source": [
429 "def answer_six():\n",
430 " import operator\n",
431 " \n",
432 " return sorted([(freq, token) for token, freq in text1.vocab().items() if freq > 2000 and token.isalpha()], key=operator.itemgetter(1), reverse=True)\n",
433 "answer_six()"
434 ]
435 },
436 {
437 "cell_type": "markdown",
438 "metadata": {},
439 "source": [
440 "### Question 7\n",
441 "\n",
442 "What is the average number of tokens per sentence?\n",
443 "\n",
444 "*This function should return a float.*"
445 ]
446 },
447 {
448 "cell_type": "code",
449 "execution_count": 17,
450 "metadata": {},
451 "outputs": [
452 {
453 "data": {
454 "text/plain": [
455 "25.881952902963864"
456 ]
457 },
458 "execution_count": 17,
459 "metadata": {},
460 "output_type": "execute_result"
461 }
462 ],
463 "source": [
464 "def answer_seven():\n",
465 " \n",
466 " \n",
467 " return np.mean([len(nltk.word_tokenize(sent)) for sent in nltk.sent_tokenize(moby_raw)]) \n",
468 "\n",
469 "answer_seven()"
470 ]
471 },
472 {
473 "cell_type": "markdown",
474 "metadata": {},
475 "source": [
476 "### Question 8\n",
477 "\n",
478 "What are the 5 most frequent parts of speech in this text? What is their frequency?\n",
479 "\n",
480 "*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*"
481 ]
482 },
483 {
484 "cell_type": "code",
485 "execution_count": 23,
486 "metadata": {},
487 "outputs": [
488 {
489 "data": {
490 "text/plain": [
491 "[('NN', 32730), ('IN', 28657), ('DT', 25867), (',', 19204), ('JJ', 17620)]"
492 ]
493 },
494 "execution_count": 23,
495 "metadata": {},
496 "output_type": "execute_result"
497 }
498 ],
499 "source": [
500 "def answer_eight():\n",
501 " from collections import Counter\n",
502 " import operator\n",
503 " \n",
504 " return sorted(Counter([tag for token, tag in nltk.pos_tag(text1)]).items(), key=operator.itemgetter(1), reverse=True)[:5]\n",
505 "\n",
506 "answer_eight()"
507 ]
508 },
509 {
510 "cell_type": "markdown",
511 "metadata": {},
512 "source": [
513 "## Part 2 - Spelling Recommender\n",
514 "\n",
515 "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
516 "\n",
517 "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
518 "\n",
519 "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
520 "\n",
521 "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
522 ]
523 },
524 {
525 "cell_type": "code",
526 "execution_count": 24,
527 "metadata": {},
528 "outputs": [],
529 "source": [
530 "\n",
531 "from nltk.corpus import words\n",
532 "\n",
533 "correct_spellings = words.words()"
534 ]
535 },
536 {
537 "cell_type": "markdown",
538 "metadata": {},
539 "source": [
540 "### Question 9\n",
541 "\n",
542 "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
543 "\n",
544 "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.**\n",
545 "\n",
546 "*This function should return a list of length three:\n",
547 "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
548 ]
549 },
550 {
551 "cell_type": "code",
552 "execution_count": 25,
553 "metadata": {},
554 "outputs": [
555 {
556 "data": {
557 "text/plain": [
558 "['corpulent', 'indecence', 'validate']"
559 ]
560 },
561 "execution_count": 25,
562 "metadata": {},
563 "output_type": "execute_result"
564 }
565 ],
566 "source": [
567 "def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):\n",
568 " result = []\n",
569 " import operator\n",
570 " for entry in entries:\n",
571 " spell_list = [spell for spell in correct_spellings if spell.startswith(entry[0]) and len(spell) > 2]\n",
572 " distance_list = [(spell, nltk.jaccard_distance(set(nltk.ngrams(entry, n=3)), set(nltk.ngrams(spell, n=3)))) for spell in spell_list]\n",
573 "\n",
574 " result.append(sorted(distance_list, key=operator.itemgetter(1))[0][0])\n",
575 " \n",
576 " return result \n",
577 " \n",
578 "answer_nine()"
579 ]
580 },
581 {
582 "cell_type": "markdown",
583 "metadata": {},
584 "source": [
585 "### Question 10\n",
586 "\n",
587 "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
588 "\n",
589 "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.**\n",
590 "\n",
591 "*This function should return a list of length three:\n",
592 "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
593 ]
594 },
595 {
596 "cell_type": "code",
597 "execution_count": 26,
598 "metadata": {},
599 "outputs": [
600 {
601 "data": {
602 "text/plain": [
603 "['cormus', 'incendiary', 'valid']"
604 ]
605 },
606 "execution_count": 26,
607 "metadata": {},
608 "output_type": "execute_result"
609 }
610 ],
611 "source": [
612 "def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):\n",
613 " result = []\n",
614 " import operator\n",
615 " for entry in entries:\n",
616 " spell_list = [spell for spell in correct_spellings if spell.startswith(entry[0]) and len(spell) > 2]\n",
617 " distance_list = [(spell, nltk.jaccard_distance(set(nltk.ngrams(entry, n=4)), set(nltk.ngrams(spell, n=4)))) for spell in spell_list]\n",
618 "\n",
619 " result.append(sorted(distance_list, key=operator.itemgetter(1))[0][0])\n",
620 " \n",
621 " return result\n",
622 " \n",
623 "answer_ten()"
624 ]
625 },
626 {
627 "cell_type": "markdown",
628 "metadata": {},
629 "source": [
630 "### Question 11\n",
631 "\n",
632 "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
633 "\n",
634 "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**\n",
635 "\n",
636 "*This function should return a list of length three:\n",
637 "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
638 ]
639 },
640 {
641 "cell_type": "code",
642 "execution_count": 27,
643 "metadata": {},
644 "outputs": [
645 {
646 "data": {
647 "text/plain": [
648 "['corpulent', 'intendence', 'validate']"
649 ]
650 },
651 "execution_count": 27,
652 "metadata": {},
653 "output_type": "execute_result"
654 }
655 ],
656 "source": [
657 "def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):\n",
658 " result = []\n",
659 " import operator\n",
660 " for entry in entries:\n",
661 " spell_list = [spell for spell in correct_spellings if spell.startswith(entry[0]) and len(spell) > 2]\n",
662 " distance_list = [(spell, nltk.edit_distance(entry, spell, transpositions=True)) for spell in spell_list]\n",
663 "\n",
664 " result.append(sorted(distance_list, key=operator.itemgetter(1))[0][0])\n",
665 " \n",
666 " return result \n",
667 " \n",
668 "answer_eleven()"
669 ]
670 },
671 {
672 "cell_type": "code",
673 "execution_count": null,
674 "metadata": {
675 "collapsed": true
676 },
677 "outputs": [],
678 "source": []
679 }
680 ],
681 "metadata": {
682 "coursera": {
683 "course_slug": "python-text-mining",
684 "graded_item_id": "r35En",
685 "launcher_item_id": "tCVfW",
686 "part_id": "NTVgL"
687 },
688 "kernelspec": {
689 "display_name": "Python 3",
690 "language": "python",
691 "name": "python3"
692 },
693 "language_info": {
694 "codemirror_mode": {
695 "name": "ipython",
696 "version": 3
697 },
698 "file_extension": ".py",
699 "mimetype": "text/x-python",
700 "name": "python",
701 "nbconvert_exporter": "python",
702 "pygments_lexer": "ipython3",
703 "version": "3.6.2"
704 }
705 },
706 "nbformat": 4,
707 "nbformat_minor": 2
708 }
709