{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2 - Introduction to NLTK\n",
    "\n",
    "In part 1 of this assignment you will use nltk to explore the Herman Melville novel Moby Dick. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Analyzing Moby Dick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package wordnet to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
      "[nltk_data] Downloading package words to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/words.zip.\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('words')\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# If you would like to work with the raw text you can use 'moby_raw'\n",
    "with open('moby.txt', 'r') as f:\n",
    "    moby_raw = f.read()\n",
    "    \n",
    "# If you would like to work with the novel in nltk.Text format you can use 'text1'\n",
    "moby_tokens = nltk.word_tokenize(moby_raw)\n",
    "text1 = nltk.Text(moby_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1\n",
    "\n",
    "How many tokens (words and punctuation symbols) are in text1?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "254989"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_one():\n",
    "    \n",
    "    return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)\n",
    "\n",
    "example_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2\n",
    "\n",
    "How many unique tokens (unique words and punctuation) does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20755"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_two():\n",
    "    \n",
    "    return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))\n",
    "\n",
    "example_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3\n",
    "\n",
    "After lemmatizing the verbs, how many unique tokens does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16900"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "def example_three():\n",
    "\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]\n",
    "\n",
    "    return len(set(lemmatized))\n",
    "\n",
    "example_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "\n",
    "What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08139566804842562"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    return len(set(text1))/len(text1)\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "What percentage of tokens is 'whale'or 'Whale'?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4125668166077752"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    \n",
    "    dist = nltk.FreqDist(text1)\n",
    "    return (dist['whale'] + dist['Whale'])/len(text1)*100\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?\n",
    "\n",
    "*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(',', 19204),\n",
       " ('the', 13715),\n",
       " ('.', 7308),\n",
       " ('of', 6513),\n",
       " ('and', 6010),\n",
       " ('a', 4545),\n",
       " ('to', 4515),\n",
       " (';', 4173),\n",
       " ('in', 3908),\n",
       " ('that', 2978),\n",
       " ('his', 2459),\n",
       " ('it', 2196),\n",
       " ('I', 2097),\n",
       " ('!', 1767),\n",
       " ('is', 1722),\n",
       " ('--', 1713),\n",
       " ('with', 1659),\n",
       " ('he', 1658),\n",
       " ('was', 1639),\n",
       " ('as', 1620)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_three():\n",
    "    \n",
    "    return nltk.FreqDist(text1).most_common(20)\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "What tokens have a length of greater than 5 and frequency of more than 150?\n",
    "\n",
    "*This function should return an alphabetically sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Captain',\n",
       " 'Pequod',\n",
       " 'Queequeg',\n",
       " 'Starbuck',\n",
       " 'almost',\n",
       " 'before',\n",
       " 'himself',\n",
       " 'little',\n",
       " 'seemed',\n",
       " 'should',\n",
       " 'though',\n",
       " 'through',\n",
       " 'whales',\n",
       " 'without']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    \n",
    "    dist = nltk.FreqDist(text1)\n",
    "    vocab1 = dist.keys()\n",
    "    freqwords = [w for w in vocab1 if len(w)>5 and dist[w] > 150]\n",
    "    return sorted(freqwords)\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Find the longest word in text1 and that word's length.\n",
    "\n",
    "*This function should return a tuple `(longest_word, length)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"twelve-o'clock-at-night\", 23)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "    \n",
    "    maxlen = max(len(w) for w in text1)\n",
    "    word = [w for w in text1 if len(w) == maxlen]\n",
    "    return (word[0],maxlen)\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What unique words have a frequency of more than 2000? What is their frequency?\n",
    "\n",
    "\"Hint:  you may want to use `isalpha()` to check if the token is a word and not punctuation.\"\n",
    "\n",
    "*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(13715, 'the'),\n",
       " (6513, 'of'),\n",
       " (6010, 'and'),\n",
       " (4545, 'a'),\n",
       " (4515, 'to'),\n",
       " (3908, 'in'),\n",
       " (2978, 'that'),\n",
       " (2459, 'his'),\n",
       " (2196, 'it'),\n",
       " (2097, 'I')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "    dist = nltk.FreqDist(text1)\n",
    "    freqwords = [(dist[w],w) for w in set(text1) if w.isalpha() and dist[w]>2000]\n",
    "    return sorted(freqwords, key=lambda x:x[0],reverse=True)\n",
    "\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "What is the average number of tokens per sentence?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25.881952902963864"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_seven():\n",
    "    text2 = nltk.sent_tokenize(moby_raw)\n",
    "    return len(text1)/len(text2)\n",
    "\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What are the 5 most frequent parts of speech in this text? What is their frequency?\n",
    "\n",
    "*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', 32730), ('IN', 28657), ('DT', 25867), (',', 19204), ('JJ', 17620)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eight():\n",
    "    \n",
    "    import collections\n",
    "    pos_list = nltk.pos_tag(text1)\n",
    "    pos_counts = collections.Counter((subl[1] for subl in pos_list))\n",
    "    return pos_counts.most_common(5)\n",
    "\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Spelling Recommender\n",
    "\n",
    "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
    "\n",
    "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
    "\n",
    "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
    "\n",
    "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import words\n",
    "\n",
    "correct_spellings = words.words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:9: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['corpulent', 'indecence', 'validate']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    \n",
    "    recommend = []\n",
    "    for entry in entries:\n",
    "        \n",
    "        # Match first letter. input_spell contains all words in correct_spellings with the same first letter.\n",
    "        input_spell = [x for x in correct_spellings if x[0] == entry[0]]\n",
    "        \n",
    "        # Find the jaccard distance between the entry word and every word in correct_spellings with the same first letter.\n",
    "        jaccard_dist = [nltk.jaccard_distance(set(nltk.ngrams(entry,n=3)), set(nltk.ngrams(x,n=3))) for x in input_spell]\n",
    "        \n",
    "        # Recommend the word in input_spell with the minimum Jaccard distance.\n",
    "        recommend.append(input_spell[np.argmin(jaccard_dist)])\n",
    "        \n",
    "    return recommend\n",
    "    \n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:10: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "  # Remove the CWD from sys.path while we load stuff.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['cormus', 'incendiary', 'valid']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    \n",
    "    recommend = []\n",
    "    for entry in entries:\n",
    "        \n",
    "        # Match first letter. input_spell contains all words in correct_spellings with the same first letter.\n",
    "        input_spell = [x for x in correct_spellings if x[0] == entry[0]]\n",
    "        \n",
    "        # Find the jaccard distance between the entry word and every word in correct_spellings with the same first letter.\n",
    "        jaccard_dist = [nltk.jaccard_distance(set(nltk.ngrams(entry,n=4)), set(nltk.ngrams(x,n=4))) for x in input_spell]\n",
    "        \n",
    "        # Recommend the word in input_spell with the minimum Jaccard distance.\n",
    "        recommend.append(input_spell[np.argmin(jaccard_dist)])\n",
    "        \n",
    "    return recommend\n",
    "    \n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['corpulent', 'intendence', 'validate']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    \n",
    "    recommend = []\n",
    "    for entry in entries:\n",
    "        \n",
    "        # Match first letter. input_spell contains all words in correct_spellings with the same first letter.\n",
    "        input_spell = [x for x in correct_spellings if x[0] == entry[0]]\n",
    "        \n",
    "        # Find the jaccard distance between the entry word and every word in correct_spellings with the same first letter.\n",
    "        DL_dist = [nltk.edit_distance(x, entry, transpositions=True) for x in input_spell]\n",
    "        \n",
    "        # Recommend the word in input_spell with the minimum Jaccard distance.\n",
    "        recommend.append(input_spell[np.argmin(DL_dist)])\n",
    "        \n",
    "    return recommend\n",
    "    \n",
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "r35En",
   "launcher_item_id": "tCVfW",
   "part_id": "NTVgL"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}