{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2 - Introduction to NLTK\n",
    "\n",
    "In part 1 of this assignment you will use nltk to explore the Herman Melville novel Moby Dick. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Analyzing Moby Dick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# If you would like to work with the raw text you can use 'moby_raw'\n",
    "with open('moby.txt', 'r') as f:\n",
    "    moby_raw = f.read()\n",
    "    \n",
    "# If you would like to work with the novel in nltk.Text format you can use 'text1'\n",
    "moby_tokens = nltk.word_tokenize(moby_raw)\n",
    "text1 = nltk.Text(moby_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1\n",
    "\n",
    "How many tokens (words and punctuation symbols) are in text1?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "254989"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_one():\n",
    "    \n",
    "    return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)\n",
    "\n",
    "example_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2\n",
    "\n",
    "How many unique tokens (unique words and punctuation) does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20755"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_two():\n",
    "    \n",
    "    return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))\n",
    "\n",
    "example_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3\n",
    "\n",
    "After lemmatizing the verbs, how many unique tokens does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "16900"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "nltk.download('wordnet')\n",
    "def example_three():\n",
    "\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]\n",
    "\n",
    "    return len(set(lemmatized))\n",
    "\n",
    "example_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "\n",
    "What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08139566804842562"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    \n",
    "    return example_two()/example_one()\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "What percentage of tokens is 'whale'or 'Whale'?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4125668166077752"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    \n",
    "    \n",
    "    return ((text1.vocab()['whale'] + text1.vocab()['Whale']) / example_one())*100\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?\n",
    "\n",
    "*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(',', 19204),\n",
       " ('the', 13715),\n",
       " ('.', 7308),\n",
       " ('of', 6513),\n",
       " ('and', 6010),\n",
       " ('a', 4545),\n",
       " ('to', 4515),\n",
       " (';', 4173),\n",
       " ('in', 3908),\n",
       " ('that', 2978),\n",
       " ('his', 2459),\n",
       " ('it', 2196),\n",
       " ('I', 2097),\n",
       " ('!', 1767),\n",
       " ('is', 1722),\n",
       " ('--', 1713),\n",
       " ('with', 1659),\n",
       " ('he', 1658),\n",
       " ('was', 1639),\n",
       " ('as', 1620)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_three():\n",
    "    import operator\n",
    "    \n",
    "    \n",
    "    return sorted(text1.vocab().items(), key=operator.itemgetter(1), reverse=True)[:20]\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "What tokens have a length of greater than 5 and frequency of more than 150?\n",
    "\n",
    "*This function should return an alphabetically sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['through',\n",
       " 'almost',\n",
       " 'whales',\n",
       " 'before',\n",
       " 'without',\n",
       " 'should',\n",
       " 'little',\n",
       " 'seemed',\n",
       " 'though',\n",
       " 'himself',\n",
       " 'Captain',\n",
       " 'Queequeg',\n",
       " 'Pequod',\n",
       " 'Starbuck']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    \n",
    "    \n",
    "    return [token for token, freq in text1.vocab().items() if len(token) > 5 and freq > 150]\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Find the longest word in text1 and that word's length.\n",
    "\n",
    "*This function should return a tuple `(longest_word, length)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"twelve-o'clock-at-night\", 23)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "    import operator\n",
    "    \n",
    "    return sorted([(token, len(token))for token, freq in text1.vocab().items()], key=operator.itemgetter(1), reverse=True)[0]\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What unique words have a frequency of more than 2000? What is their frequency?\n",
    "\n",
    "\"Hint:  you may want to use `isalpha()` to check if the token is a word and not punctuation.\"\n",
    "\n",
    "*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(4515, 'to'),\n",
       " (13715, 'the'),\n",
       " (2978, 'that'),\n",
       " (6513, 'of'),\n",
       " (2196, 'it'),\n",
       " (3908, 'in'),\n",
       " (2459, 'his'),\n",
       " (6010, 'and'),\n",
       " (4545, 'a'),\n",
       " (2097, 'I')]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "    import operator\n",
    "    \n",
    "    return sorted([(freq, token) for token, freq in text1.vocab().items() if freq > 2000 and token.isalpha()], key=operator.itemgetter(1), reverse=True)\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "What is the average number of tokens per sentence?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25.881952902963864"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_seven():\n",
    "    \n",
    "    \n",
    "    return np.mean([len(nltk.word_tokenize(sent)) for sent in nltk.sent_tokenize(moby_raw)]) \n",
    "\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What are the 5 most frequent parts of speech in this text? What is their frequency?\n",
    "\n",
    "*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', 32730), ('IN', 28657), ('DT', 25867), (',', 19204), ('JJ', 17620)]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eight():\n",
    "    from collections import Counter\n",
    "    import operator\n",
    "    \n",
    "    return sorted(Counter([tag for token, tag in nltk.pos_tag(text1)]).items(), key=operator.itemgetter(1), reverse=True)[:5]\n",
    "\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Spelling Recommender\n",
    "\n",
    "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
    "\n",
    "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
    "\n",
    "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
    "\n",
    "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from nltk.corpus import words\n",
    "\n",
    "correct_spellings = words.words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['corpulent', 'indecence', 'validate']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    result = []\n",
    "    import operator\n",
    "    for entry in entries:\n",
    "        spell_list = [spell for spell in correct_spellings if spell.startswith(entry[0]) and len(spell) > 2]\n",
    "        distance_list = [(spell, nltk.jaccard_distance(set(nltk.ngrams(entry, n=3)), set(nltk.ngrams(spell, n=3)))) for spell in spell_list]\n",
    "\n",
    "        result.append(sorted(distance_list, key=operator.itemgetter(1))[0][0])\n",
    "    \n",
    "    return result \n",
    "    \n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cormus', 'incendiary', 'valid']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    result = []\n",
    "    import operator\n",
    "    for entry in entries:\n",
    "        spell_list = [spell for spell in correct_spellings if spell.startswith(entry[0]) and len(spell) > 2]\n",
    "        distance_list = [(spell, nltk.jaccard_distance(set(nltk.ngrams(entry, n=4)), set(nltk.ngrams(spell, n=4)))) for spell in spell_list]\n",
    "\n",
    "        result.append(sorted(distance_list, key=operator.itemgetter(1))[0][0])\n",
    "    \n",
    "    return result\n",
    "    \n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['corpulent', 'intendence', 'validate']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    result = []\n",
    "    import operator\n",
    "    for entry in entries:\n",
    "        spell_list = [spell for spell in correct_spellings if spell.startswith(entry[0]) and len(spell) > 2]\n",
    "        distance_list = [(spell, nltk.edit_distance(entry, spell, transpositions=True)) for spell in spell_list]\n",
    "\n",
    "        result.append(sorted(distance_list, key=operator.itemgetter(1))[0][0])\n",
    "    \n",
    "    return result \n",
    "    \n",
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "r35En",
   "launcher_item_id": "tCVfW",
   "part_id": "NTVgL"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}