Format: application/octet-stream
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "---\n",
8 "\n",
9 "_You are currently looking at **version 1.2** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-machine-learning/resources/bANLa) course resource._\n",
10 "\n",
11 "---"
12 ]
13 },
14 {
15 "cell_type": "markdown",
16 "metadata": {},
17 "source": [
18 "# Assignment 3 - Evaluation\n",
19 "\n",
20 "In this assignment you will train several models and evaluate how effectively they predict instances of fraud using data based on [this dataset from Kaggle](https://www.kaggle.com/dalpozz/creditcardfraud).\n",
21 " \n",
22 "Each row in `fraud_data.csv` corresponds to a credit card transaction. Features include confidential variables `V1` through `V28` as well as `Amount` which is the amount of the transaction. \n",
23 " \n",
24 "The target is stored in the `class` column, where a value of 1 corresponds to an instance of fraud and 0 corresponds to an instance of not fraud."
25 ]
26 },
27 {
28 "cell_type": "code",
29 "execution_count": 2,
30 "metadata": {},
31 "outputs": [],
32 "source": [
33 "import numpy as np\n",
34 "import pandas as pd"
35 ]
36 },
37 {
38 "cell_type": "markdown",
39 "metadata": {},
40 "source": [
41 "### Question 1\n",
42 "Import the data from `fraud_data.csv`. What percentage of the observations in the dataset are instances of fraud?\n",
43 "\n",
44 "*This function should return a float between 0 and 1.* "
45 ]
46 },
47 {
48 "cell_type": "code",
49 "execution_count": 4,
50 "metadata": {},
51 "outputs": [
52 {
53 "data": {
54 "text/plain": [
55 "356"
56 ]
57 },
58 "execution_count": 4,
59 "metadata": {},
60 "output_type": "execute_result"
61 }
62 ],
63 "source": [
64 "def answer_one():\n",
65 " \n",
66 " df = pd.read_csv('Source/fraud_data.csv')\n",
67 " \n",
68 " \n",
69 " return len(df[df.Class==1])/len(df)\n",
70 "answer_one()"
71 ]
72 },
73 {
74 "cell_type": "code",
75 "execution_count": null,
76 "metadata": {
77 "collapsed": true
78 },
79 "outputs": [],
80 "source": [
81 "# Use X_train, X_test, y_train, y_test for all of the following questions\n",
82 "from sklearn.model_selection import train_test_split\n",
83 "\n",
84 "df = pd.read_csv('Source/fraud_data.csv')\n",
85 "\n",
86 "X = df.iloc[:,:-1]\n",
87 "y = df.iloc[:,-1]\n",
88 "\n",
89 "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
90 ]
91 },
92 {
93 "cell_type": "markdown",
94 "metadata": {},
95 "source": [
96 "### Question 2\n",
97 "\n",
98 "Using `X_train`, `X_test`, `y_train`, and `y_test` (as defined above), train a dummy classifier that classifies everything as the majority class of the training data. What is the accuracy of this classifier? What is the recall?\n",
99 "\n",
100 "*This function should a return a tuple with two floats, i.e. `(accuracy score, recall score)`.*"
101 ]
102 },
103 {
104 "cell_type": "code",
105 "execution_count": null,
106 "metadata": {
107 "collapsed": true
108 },
109 "outputs": [],
110 "source": [
111 "def answer_two():\n",
112 " from sklearn.dummy import DummyClassifier\n",
113 " from sklearn.metrics import recall_score\n",
114 " \n",
115 " # Your code here\n",
116 " \n",
117 " return # Return your answer"
118 ]
119 },
120 {
121 "cell_type": "markdown",
122 "metadata": {},
123 "source": [
124 "### Question 3\n",
125 "\n",
126 "Using X_train, X_test, y_train, y_test (as defined above), train a SVC classifer using the default parameters. What is the accuracy, recall, and precision of this classifier?\n",
127 "\n",
128 "*This function should a return a tuple with three floats, i.e. `(accuracy score, recall score, precision score)`.*"
129 ]
130 },
131 {
132 "cell_type": "code",
133 "execution_count": null,
134 "metadata": {
135 "collapsed": true
136 },
137 "outputs": [],
138 "source": [
139 "def answer_three():\n",
140 " from sklearn.metrics import recall_score, precision_score\n",
141 " from sklearn.svm import SVC\n",
142 "\n",
143 " # Your code here\n",
144 " \n",
145 " return # Return your answer"
146 ]
147 },
148 {
149 "cell_type": "markdown",
150 "metadata": {},
151 "source": [
152 "### Question 4\n",
153 "\n",
154 "Using the SVC classifier with parameters `{'C': 1e9, 'gamma': 1e-07}`, what is the confusion matrix when using a threshold of -220 on the decision function. Use X_test and y_test.\n",
155 "\n",
156 "*This function should return a confusion matrix, a 2x2 numpy array with 4 integers.*"
157 ]
158 },
159 {
160 "cell_type": "code",
161 "execution_count": null,
162 "metadata": {
163 "collapsed": true
164 },
165 "outputs": [],
166 "source": [
167 "def answer_four():\n",
168 " from sklearn.metrics import confusion_matrix\n",
169 " from sklearn.svm import SVC\n",
170 "\n",
171 " # Your code here\n",
172 " \n",
173 " return # Return your answer"
174 ]
175 },
176 {
177 "cell_type": "markdown",
178 "metadata": {},
179 "source": [
180 "### Question 5\n",
181 "\n",
182 "Train a logisitic regression classifier with default parameters using X_train and y_train.\n",
183 "\n",
184 "For the logisitic regression classifier, create a precision recall curve and a roc curve using y_test and the probability estimates for X_test (probability it is fraud).\n",
185 "\n",
186 "Looking at the precision recall curve, what is the recall when the precision is `0.75`?\n",
187 "\n",
188 "Looking at the roc curve, what is the true positive rate when the false positive rate is `0.16`?\n",
189 "\n",
190 "*This function should return a tuple with two floats, i.e. `(recall, true positive rate)`.*"
191 ]
192 },
193 {
194 "cell_type": "code",
195 "execution_count": null,
196 "metadata": {
197 "collapsed": true
198 },
199 "outputs": [],
200 "source": [
201 "def answer_five():\n",
202 " \n",
203 " # Your code here\n",
204 " \n",
205 " return # Return your answer"
206 ]
207 },
208 {
209 "cell_type": "markdown",
210 "metadata": {},
211 "source": [
212 "### Question 6\n",
213 "\n",
214 "Perform a grid search over the parameters listed below for a Logisitic Regression classifier, using recall for scoring and the default 3-fold cross validation.\n",
215 "\n",
216 "`'penalty': ['l1', 'l2']`\n",
217 "\n",
218 "`'C':[0.01, 0.1, 1, 10, 100]`\n",
219 "\n",
220 "From `.cv_results_`, create an array of the mean test scores of each parameter combination. i.e.\n",
221 "\n",
222 "| \t| `l1` \t| `l2` \t|\n",
223 "|:----:\t|----\t|----\t|\n",
224 "| **`0.01`** \t| ?\t| ? \t|\n",
225 "| **`0.1`** \t| ?\t| ? \t|\n",
226 "| **`1`** \t| ?\t| ? \t|\n",
227 "| **`10`** \t| ?\t| ? \t|\n",
228 "| **`100`** \t| ?\t| ? \t|\n",
229 "\n",
230 "<br>\n",
231 "\n",
232 "*This function should return a 5 by 2 numpy array with 10 floats.* \n",
233 "\n",
234 "*Note: do not return a DataFrame, just the values denoted by '?' above in a numpy array. You might need to reshape your raw result to meet the format we are looking for.*"
235 ]
236 },
237 {
238 "cell_type": "code",
239 "execution_count": null,
240 "metadata": {
241 "collapsed": true
242 },
243 "outputs": [],
244 "source": [
245 "def answer_six(): \n",
246 " from sklearn.model_selection import GridSearchCV\n",
247 " from sklearn.linear_model import LogisticRegression\n",
248 "\n",
249 " # Your code here\n",
250 " \n",
251 " return # Return your answer"
252 ]
253 },
254 {
255 "cell_type": "code",
256 "execution_count": null,
257 "metadata": {
258 "collapsed": true
259 },
260 "outputs": [],
261 "source": [
262 "# Use the following function to help visualize results from the grid search\n",
263 "def GridSearch_Heatmap(scores):\n",
264 " %matplotlib notebook\n",
265 " import seaborn as sns\n",
266 " import matplotlib.pyplot as plt\n",
267 " plt.figure()\n",
268 " sns.heatmap(scores.reshape(5,2), xticklabels=['l1','l2'], yticklabels=[0.01, 0.1, 1, 10, 100])\n",
269 " plt.yticks(rotation=0);\n",
270 "\n",
271 "#GridSearch_Heatmap(answer_six())"
272 ]
273 }
274 ],
275 "metadata": {
276 "coursera": {
277 "course_slug": "python-machine-learning",
278 "graded_item_id": "5yX9Z",
279 "launcher_item_id": "eqnV3",
280 "part_id": "Msnj0"
281 },
282 "kernelspec": {
283 "display_name": "Python 3",
284 "language": "python",
285 "name": "python3"
286 },
287 "language_info": {
288 "codemirror_mode": {
289 "name": "ipython",
290 "version": 3
291 },
292 "file_extension": ".py",
293 "mimetype": "text/x-python",
294 "name": "python",
295 "nbconvert_exporter": "python",
296 "pygments_lexer": "ipython3",
297 "version": "3.8.8"
298 }
299 },
300 "nbformat": 4,
301 "nbformat_minor": 2
302 }
303