Format: application/octet-stream
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "---\n",
8 "\n",
9 "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
10 "\n",
11 "---"
12 ]
13 },
14 {
15 "cell_type": "markdown",
16 "metadata": {},
17 "source": [
18 "# Working With Text"
19 ]
20 },
21 {
22 "cell_type": "code",
23 "execution_count": 1,
24 "metadata": {},
25 "outputs": [
26 {
27 "data": {
28 "text/plain": [
29 "76"
30 ]
31 },
32 "execution_count": 1,
33 "metadata": {},
34 "output_type": "execute_result"
35 }
36 ],
37 "source": [
38 "text1 = \"Ethics are built right into the ideals and objectives of the United Nations \"\n",
39 "\n",
40 "len(text1) # The length of text1"
41 ]
42 },
43 {
44 "cell_type": "code",
45 "execution_count": 2,
46 "metadata": {},
47 "outputs": [
48 {
49 "data": {
50 "text/plain": [
51 "14"
52 ]
53 },
54 "execution_count": 2,
55 "metadata": {},
56 "output_type": "execute_result"
57 }
58 ],
59 "source": [
60 "text2 = text1.split(' ') # Return a list of the words in text2, separating by ' '.\n",
61 "\n",
62 "len(text2)"
63 ]
64 },
65 {
66 "cell_type": "code",
67 "execution_count": 3,
68 "metadata": {},
69 "outputs": [
70 {
71 "data": {
72 "text/plain": [
73 "['Ethics',\n",
74 " 'are',\n",
75 " 'built',\n",
76 " 'right',\n",
77 " 'into',\n",
78 " 'the',\n",
79 " 'ideals',\n",
80 " 'and',\n",
81 " 'objectives',\n",
82 " 'of',\n",
83 " 'the',\n",
84 " 'United',\n",
85 " 'Nations',\n",
86 " '']"
87 ]
88 },
89 "execution_count": 3,
90 "metadata": {},
91 "output_type": "execute_result"
92 }
93 ],
94 "source": [
95 "text2"
96 ]
97 },
98 {
99 "cell_type": "markdown",
100 "metadata": {},
101 "source": [
102 "<br>\n",
103 "List comprehension allows us to find specific words:"
104 ]
105 },
106 {
107 "cell_type": "code",
108 "execution_count": 4,
109 "metadata": {},
110 "outputs": [
111 {
112 "data": {
113 "text/plain": [
114 "['Ethics',\n",
115 " 'built',\n",
116 " 'right',\n",
117 " 'into',\n",
118 " 'ideals',\n",
119 " 'objectives',\n",
120 " 'United',\n",
121 " 'Nations']"
122 ]
123 },
124 "execution_count": 4,
125 "metadata": {},
126 "output_type": "execute_result"
127 }
128 ],
129 "source": [
130 "[w for w in text2 if len(w) > 3] # Words that are greater than 3 letters long in text2"
131 ]
132 },
133 {
134 "cell_type": "code",
135 "execution_count": 5,
136 "metadata": {},
137 "outputs": [
138 {
139 "data": {
140 "text/plain": [
141 "['Ethics', 'United', 'Nations']"
142 ]
143 },
144 "execution_count": 5,
145 "metadata": {},
146 "output_type": "execute_result"
147 }
148 ],
149 "source": [
150 "[w for w in text2 if w.istitle()] # Capitalized words in text2"
151 ]
152 },
153 {
154 "cell_type": "code",
155 "execution_count": 6,
156 "metadata": {},
157 "outputs": [
158 {
159 "data": {
160 "text/plain": [
161 "['Ethics', 'ideals', 'objectives', 'Nations']"
162 ]
163 },
164 "execution_count": 6,
165 "metadata": {},
166 "output_type": "execute_result"
167 }
168 ],
169 "source": [
170 "[w for w in text2 if w.endswith('s')] # Words in text2 that end in 's'"
171 ]
172 },
173 {
174 "cell_type": "markdown",
175 "metadata": {},
176 "source": [
177 "<br>\n",
178 "We can find unique words using `set()`."
179 ]
180 },
181 {
182 "cell_type": "code",
183 "execution_count": 7,
184 "metadata": {},
185 "outputs": [
186 {
187 "data": {
188 "text/plain": [
189 "6"
190 ]
191 },
192 "execution_count": 7,
193 "metadata": {},
194 "output_type": "execute_result"
195 }
196 ],
197 "source": [
198 "text3 = 'To be or not to be'\n",
199 "text4 = text3.split(' ')\n",
200 "\n",
201 "len(text4)"
202 ]
203 },
204 {
205 "cell_type": "code",
206 "execution_count": 8,
207 "metadata": {},
208 "outputs": [
209 {
210 "data": {
211 "text/plain": [
212 "5"
213 ]
214 },
215 "execution_count": 8,
216 "metadata": {},
217 "output_type": "execute_result"
218 }
219 ],
220 "source": [
221 "len(set(text4))"
222 ]
223 },
224 {
225 "cell_type": "code",
226 "execution_count": 9,
227 "metadata": {},
228 "outputs": [
229 {
230 "data": {
231 "text/plain": [
232 "{'To', 'be', 'not', 'or', 'to'}"
233 ]
234 },
235 "execution_count": 9,
236 "metadata": {},
237 "output_type": "execute_result"
238 }
239 ],
240 "source": [
241 "set(text4)"
242 ]
243 },
244 {
245 "cell_type": "code",
246 "execution_count": 10,
247 "metadata": {},
248 "outputs": [
249 {
250 "data": {
251 "text/plain": [
252 "4"
253 ]
254 },
255 "execution_count": 10,
256 "metadata": {},
257 "output_type": "execute_result"
258 }
259 ],
260 "source": [
261 "len(set([w.lower() for w in text4])) # .lower converts the string to lowercase."
262 ]
263 },
264 {
265 "cell_type": "code",
266 "execution_count": 11,
267 "metadata": {},
268 "outputs": [
269 {
270 "data": {
271 "text/plain": [
272 "{'be', 'not', 'or', 'to'}"
273 ]
274 },
275 "execution_count": 11,
276 "metadata": {},
277 "output_type": "execute_result"
278 }
279 ],
280 "source": [
281 "set([w.lower() for w in text4])"
282 ]
283 },
284 {
285 "cell_type": "markdown",
286 "metadata": {},
287 "source": [
288 "### Processing free-text"
289 ]
290 },
291 {
292 "cell_type": "code",
293 "execution_count": 12,
294 "metadata": {},
295 "outputs": [
296 {
297 "data": {
298 "text/plain": [
299 "['\"Ethics',\n",
300 " 'are',\n",
301 " 'built',\n",
302 " 'right',\n",
303 " 'into',\n",
304 " 'the',\n",
305 " 'ideals',\n",
306 " 'and',\n",
307 " 'objectives',\n",
308 " 'of',\n",
309 " 'the',\n",
310 " 'United',\n",
311 " 'Nations\"',\n",
312 " '#UNSG',\n",
313 " '@',\n",
314 " 'NY',\n",
315 " 'Society',\n",
316 " 'for',\n",
317 " 'Ethical',\n",
318 " 'Culture',\n",
319 " 'bit.ly/2guVelr']"
320 ]
321 },
322 "execution_count": 12,
323 "metadata": {},
324 "output_type": "execute_result"
325 }
326 ],
327 "source": [
328 "text5 = '\"Ethics are built right into the ideals and objectives of the United Nations\" \\\n",
329 "#UNSG @ NY Society for Ethical Culture bit.ly/2guVelr'\n",
330 "text6 = text5.split(' ')\n",
331 "\n",
332 "text6"
333 ]
334 },
335 {
336 "cell_type": "markdown",
337 "metadata": {},
338 "source": [
339 "<br>\n",
340 "Finding hastags:"
341 ]
342 },
343 {
344 "cell_type": "code",
345 "execution_count": 13,
346 "metadata": {},
347 "outputs": [
348 {
349 "data": {
350 "text/plain": [
351 "['#UNSG']"
352 ]
353 },
354 "execution_count": 13,
355 "metadata": {},
356 "output_type": "execute_result"
357 }
358 ],
359 "source": [
360 "[w for w in text6 if w.startswith('#')]"
361 ]
362 },
363 {
364 "cell_type": "markdown",
365 "metadata": {},
366 "source": [
367 "<br>\n",
368 "Finding callouts:"
369 ]
370 },
371 {
372 "cell_type": "code",
373 "execution_count": 14,
374 "metadata": {},
375 "outputs": [
376 {
377 "data": {
378 "text/plain": [
379 "['@']"
380 ]
381 },
382 "execution_count": 14,
383 "metadata": {},
384 "output_type": "execute_result"
385 }
386 ],
387 "source": [
388 "[w for w in text6 if w.startswith('@')]"
389 ]
390 },
391 {
392 "cell_type": "code",
393 "execution_count": 15,
394 "metadata": {
395 "collapsed": true
396 },
397 "outputs": [],
398 "source": [
399 "text7 = '@UN @UN_Women \"Ethics are built right into the ideals and objectives of the United Nations\" \\\n",
400 "#UNSG @ NY Society for Ethical Culture bit.ly/2guVelr'\n",
401 "text8 = text7.split(' ')"
402 ]
403 },
404 {
405 "cell_type": "markdown",
406 "metadata": {},
407 "source": [
408 "<br>\n",
409 "\n",
410 "We can use regular expressions to help us with more complex parsing. \n",
411 "\n",
412 "For example `'@[A-Za-z0-9_]+'` will return all words that: \n",
413 "* start with `'@'` and are followed by at least one: \n",
414 "* capital letter (`'A-Z'`)\n",
415 "* lowercase letter (`'a-z'`) \n",
416 "* number (`'0-9'`)\n",
417 "* or underscore (`'_'`)"
418 ]
419 },
420 {
421 "cell_type": "code",
422 "execution_count": 16,
423 "metadata": {},
424 "outputs": [
425 {
426 "data": {
427 "text/plain": [
428 "['@UN', '@UN_Women']"
429 ]
430 },
431 "execution_count": 16,
432 "metadata": {},
433 "output_type": "execute_result"
434 }
435 ],
436 "source": [
437 "import re # import re - a module that provides support for regular expressions\n",
438 "\n",
439 "[w for w in text8 if re.search('@[A-Za-z0-9_]+', w)]"
440 ]
441 }
442 ],
443 "metadata": {
444 "kernelspec": {
445 "display_name": "Python 3",
446 "language": "python",
447 "name": "python3"
448 },
449 "language_info": {
450 "codemirror_mode": {
451 "name": "ipython",
452 "version": 3
453 },
454 "file_extension": ".py",
455 "mimetype": "text/x-python",
456 "name": "python",
457 "nbconvert_exporter": "python",
458 "pygments_lexer": "ipython3",
459 "version": "3.6.2"
460 }
461 },
462 "nbformat": 4,
463 "nbformat_minor": 1
464 }
465