Format: application/octet-stream
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "---\n",
8 "\n",
9 "_You are currently looking at **version 1.2** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-machine-learning/resources/bANLa) course resource._\n",
10 "\n",
11 "---"
12 ]
13 },
14 {
15 "cell_type": "markdown",
16 "metadata": {},
17 "source": [
18 "# Assignment 3 - Evaluation\n",
19 "\n",
20 "In this assignment you will train several models and evaluate how effectively they predict instances of fraud using data based on [this dataset from Kaggle](https://www.kaggle.com/dalpozz/creditcardfraud).\n",
21 " \n",
22 "Each row in `fraud_data.csv` corresponds to a credit card transaction. Features include confidential variables `V1` through `V28` as well as `Amount` which is the amount of the transaction. \n",
23 " \n",
24 "The target is stored in the `class` column, where a value of 1 corresponds to an instance of fraud and 0 corresponds to an instance of not fraud."
25 ]
26 },
27 {
28 "cell_type": "code",
29 "execution_count": 1,
30 "metadata": {
31 "collapsed": true
32 },
33 "outputs": [],
34 "source": [
35 "import numpy as np\n",
36 "import pandas as pd"
37 ]
38 },
39 {
40 "cell_type": "markdown",
41 "metadata": {},
42 "source": [
43 "### Question 1\n",
44 "Import the data from `fraud_data.csv`. What percentage of the observations in the dataset are instances of fraud?\n",
45 "\n",
46 "*This function should return a float between 0 and 1.* "
47 ]
48 },
49 {
50 "cell_type": "code",
51 "execution_count": 2,
52 "metadata": {},
53 "outputs": [
54 {
55 "data": {
56 "text/plain": [
57 "0.016410823768035772"
58 ]
59 },
60 "execution_count": 2,
61 "metadata": {},
62 "output_type": "execute_result"
63 }
64 ],
65 "source": [
66 "def answer_one():\n",
67 " \n",
68 " df = pd.read_csv('fraud_data.csv')\n",
69 " seg = df.iloc[:,-1]\n",
70 " return float(seg[seg.values==1].count()/seg.count())\n",
71 "\n",
72 "answer_one()"
73 ]
74 },
75 {
76 "cell_type": "code",
77 "execution_count": 3,
78 "metadata": {},
79 "outputs": [],
80 "source": [
81 "# Use X_train, X_test, y_train, y_test for all of the following questions\n",
82 "from sklearn.model_selection import train_test_split\n",
83 "\n",
84 "df = pd.read_csv('fraud_data.csv')\n",
85 "\n",
86 "X = df.iloc[:,:-1]\n",
87 "y = df.iloc[:,-1]\n",
88 "\n",
89 "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
90 "\n",
91 "#df"
92 ]
93 },
94 {
95 "cell_type": "markdown",
96 "metadata": {},
97 "source": [
98 "### Question 2\n",
99 "\n",
100 "Using `X_train`, `X_test`, `y_train`, and `y_test` (as defined above), train a dummy classifier that classifies everything as the majority class of the training data. What is the accuracy of this classifier? What is the recall?\n",
101 "\n",
102 "*This function should a return a tuple with two floats, i.e. `(accuracy score, recall score)`.*"
103 ]
104 },
105 {
106 "cell_type": "code",
107 "execution_count": 4,
108 "metadata": {},
109 "outputs": [
110 {
111 "data": {
112 "text/plain": [
113 "(0.98303522035773561, 0.0)"
114 ]
115 },
116 "execution_count": 4,
117 "metadata": {},
118 "output_type": "execute_result"
119 }
120 ],
121 "source": [
122 "def answer_two():\n",
123 " from sklearn.dummy import DummyClassifier\n",
124 " from sklearn.metrics import recall_score\n",
125 " \n",
126 " dummy_majority = DummyClassifier(strategy='most_frequent').fit(X_train,y_train)\n",
127 " y_dummy_pred = dummy_majority.predict(X_test)\n",
128 " return (dummy_majority.score(X_train,y_train),recall_score(y_test,y_dummy_pred))\n",
129 "\n",
130 "answer_two()"
131 ]
132 },
133 {
134 "cell_type": "markdown",
135 "metadata": {},
136 "source": [
137 "### Question 3\n",
138 "\n",
139 "Using X_train, X_test, y_train, y_test (as defined above), train a SVC classifer using the default parameters. What is the accuracy, recall, and precision of this classifier?\n",
140 "\n",
141 "*This function should a return a tuple with three floats, i.e. `(accuracy score, recall score, precision score)`.*"
142 ]
143 },
144 {
145 "cell_type": "code",
146 "execution_count": 5,
147 "metadata": {},
148 "outputs": [
149 {
150 "data": {
151 "text/plain": [
152 "(0.99784866924826354, 0.375, 1.0)"
153 ]
154 },
155 "execution_count": 5,
156 "metadata": {},
157 "output_type": "execute_result"
158 }
159 ],
160 "source": [
161 "def answer_three():\n",
162 " from sklearn.metrics import recall_score, precision_score\n",
163 " from sklearn.svm import SVC\n",
164 "\n",
165 " clf = SVC(kernel='rbf').fit(X_train,y_train)\n",
166 " pred = clf.predict(X_test)\n",
167 " return (clf.score(X_train,y_train),recall_score(y_test,pred),precision_score(y_test,pred))\n",
168 "\n",
169 "answer_three()"
170 ]
171 },
172 {
173 "cell_type": "markdown",
174 "metadata": {},
175 "source": [
176 "### Question 4\n",
177 "\n",
178 "Using the SVC classifier with parameters `{'C': 1e9, 'gamma': 1e-07}`, what is the confusion matrix when using a threshold of -220 on the decision function. Use X_test and y_test.\n",
179 "\n",
180 "*This function should return a confusion matrix, a 2x2 numpy array with 4 integers.*"
181 ]
182 },
183 {
184 "cell_type": "code",
185 "execution_count": 6,
186 "metadata": {},
187 "outputs": [
188 {
189 "data": {
190 "text/plain": [
191 "array([[5320, 24],\n",
192 " [ 14, 66]])"
193 ]
194 },
195 "execution_count": 6,
196 "metadata": {},
197 "output_type": "execute_result"
198 }
199 ],
200 "source": [
201 "def answer_four():\n",
202 " from sklearn.metrics import confusion_matrix\n",
203 " from sklearn.svm import SVC\n",
204 " \n",
205 " svm = SVC(C=1e9,gamma=1e-07).fit(X_train,y_train)\n",
206 " y_pred = svm.decision_function(X_test) > -220\n",
207 " return confusion_matrix(y_test,y_pred)\n",
208 "\n",
209 "answer_four()"
210 ]
211 },
212 {
213 "cell_type": "markdown",
214 "metadata": {},
215 "source": [
216 "### Question 5\n",
217 "\n",
218 "Train a logisitic regression classifier with default parameters using X_train and y_train.\n",
219 "\n",
220 "For the logisitic regression classifier, create a precision recall curve and a roc curve using y_test and the probability estimates for X_test (probability it is fraud).\n",
221 "\n",
222 "Looking at the precision recall curve, what is the recall when the precision is `0.75`?\n",
223 "\n",
224 "Looking at the roc curve, what is the true positive rate when the false positive rate is `0.16`?\n",
225 "\n",
226 "*This function should return a tuple with two floats, i.e. `(recall, true positive rate)`.*"
227 ]
228 },
229 {
230 "cell_type": "code",
231 "execution_count": 7,
232 "metadata": {},
233 "outputs": [
234 {
235 "data": {
236 "text/plain": [
237 "(0.83, 0.94)"
238 ]
239 },
240 "execution_count": 7,
241 "metadata": {},
242 "output_type": "execute_result"
243 }
244 ],
245 "source": [
246 "def answer_five():\n",
247 " \n",
248 " #%matplotlib notebook\n",
249 " #import matplotlib.pyplot as plt\n",
250 " from sklearn.linear_model import LogisticRegression\n",
251 " from sklearn.metrics import precision_recall_curve, roc_curve\n",
252 " \n",
253 " y_scores_lr = LogisticRegression().fit(X_train,y_train).decision_function(X_test)\n",
254 " precision,recall,thresholds = precision_recall_curve(y_test,y_scores_lr)\n",
255 " fpr_lr, tpr_lr, _ = roc_curve(y_test, y_scores_lr)\n",
256 " \n",
257 " #plt.figure()\n",
258 " #plt.plot(precision,recall)\n",
259 " #plt.plot(fpr_lr,tpr_lr)\n",
260 " \n",
261 " return (0.83,0.94)\n",
262 "\n",
263 "answer_five()"
264 ]
265 },
266 {
267 "cell_type": "markdown",
268 "metadata": {},
269 "source": [
270 "### Question 6\n",
271 "\n",
272 "Perform a grid search over the parameters listed below for a Logisitic Regression classifier, using recall for scoring and the default 3-fold cross validation.\n",
273 "\n",
274 "`'penalty': ['l1', 'l2']`\n",
275 "\n",
276 "`'C':[0.01, 0.1, 1, 10, 100]`\n",
277 "\n",
278 "From `.cv_results_`, create an array of the mean test scores of each parameter combination. i.e.\n",
279 "\n",
280 "| \t| `l1` \t| `l2` \t|\n",
281 "|:----:\t|----\t|----\t|\n",
282 "| **`0.01`** \t| ?\t| ? \t|\n",
283 "| **`0.1`** \t| ?\t| ? \t|\n",
284 "| **`1`** \t| ?\t| ? \t|\n",
285 "| **`10`** \t| ?\t| ? \t|\n",
286 "| **`100`** \t| ?\t| ? \t|\n",
287 "\n",
288 "<br>\n",
289 "\n",
290 "*This function should return a 5 by 2 numpy array with 10 floats.* \n",
291 "\n",
292 "*Note: do not return a DataFrame, just the values denoted by '?' above in a numpy array. You might need to reshape your raw result to meet the format we are looking for.*"
293 ]
294 },
295 {
296 "cell_type": "code",
297 "execution_count": 8,
298 "metadata": {},
299 "outputs": [
300 {
301 "data": {
302 "text/plain": [
303 "array([[ 0.66666667, 0.76086957],\n",
304 " [ 0.80072464, 0.80434783],\n",
305 " [ 0.8115942 , 0.8115942 ],\n",
306 " [ 0.80797101, 0.8115942 ],\n",
307 " [ 0.80797101, 0.80797101]])"
308 ]
309 },
310 "execution_count": 8,
311 "metadata": {},
312 "output_type": "execute_result"
313 }
314 ],
315 "source": [
316 "def answer_six():\n",
317 " \n",
318 " from sklearn.model_selection import GridSearchCV\n",
319 " from sklearn.linear_model import LogisticRegression\n",
320 " from sklearn.model_selection import cross_val_score\n",
321 " \n",
322 " grid_values={'penalty': ['l1', 'l2'], 'C':[0.01, 0.1, 1, 10, 100]}\n",
323 " lr = LogisticRegression().fit(X_train,y_train)\n",
324 " \n",
325 " # Number of folds included as a parameter in GridSearchCV. Default cv=3.\n",
326 " lr_custom = GridSearchCV(lr,param_grid=grid_values,scoring='recall',cv=3)\n",
327 " lr_custom.fit(X_train,y_train)\n",
328 "\n",
329 " return lr_custom.cv_results_['mean_test_score'].reshape(5,2)\n",
330 "\n",
331 "answer_six()"
332 ]
333 },
334 {
335 "cell_type": "code",
336 "execution_count": 9,
337 "metadata": {},
338 "outputs": [],
339 "source": [
340 "# Use the following function to help visualize results from the grid search\n",
341 "def GridSearch_Heatmap(scores):\n",
342 " %matplotlib notebook\n",
343 " import seaborn as sns\n",
344 " import matplotlib.pyplot as plt\n",
345 " plt.figure()\n",
346 " sns.heatmap(scores.reshape(5,2), xticklabels=['l1','l2'], yticklabels=[0.01, 0.1, 1, 10, 100])\n",
347 " plt.yticks(rotation=0);\n",
348 "\n",
349 "#GridSearch_Heatmap(answer_six())"
350 ]
351 },
352 {
353 "cell_type": "code",
354 "execution_count": null,
355 "metadata": {
356 "collapsed": true
357 },
358 "outputs": [],
359 "source": []
360 }
361 ],
362 "metadata": {
363 "coursera": {
364 "course_slug": "python-machine-learning",
365 "graded_item_id": "5yX9Z",
366 "launcher_item_id": "eqnV3",
367 "part_id": "Msnj0"
368 },
369 "kernelspec": {
370 "display_name": "Python 3",
371 "language": "python",
372 "name": "python3"
373 },
374 "language_info": {
375 "codemirror_mode": {
376 "name": "ipython",
377 "version": 3
378 },
379 "file_extension": ".py",
380 "mimetype": "text/x-python",
381 "name": "python",
382 "nbconvert_exporter": "python",
383 "pygments_lexer": "ipython3",
384 "version": "3.8.8"
385 }
386 },
387 "nbformat": 4,
388 "nbformat_minor": 2
389 }
390