Format: application/octet-stream
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "---\n",
8 "\n",
9 "_You are currently looking at **version 1.2** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-machine-learning/resources/bANLa) course resource._\n",
10 "\n",
11 "---"
12 ]
13 },
14 {
15 "cell_type": "markdown",
16 "metadata": {},
17 "source": [
18 "# Assignment 3 - Evaluation\n",
19 "\n",
20 "In this assignment you will train several models and evaluate how effectively they predict instances of fraud using data based on [this dataset from Kaggle](https://www.kaggle.com/dalpozz/creditcardfraud).\n",
21 " \n",
22 "Each row in `fraud_data.csv` corresponds to a credit card transaction. Features include confidential variables `V1` through `V28` as well as `Amount` which is the amount of the transaction. \n",
23 " \n",
24 "The target is stored in the `class` column, where a value of 1 corresponds to an instance of fraud and 0 corresponds to an instance of not fraud."
25 ]
26 },
27 {
28 "cell_type": "code",
29 "execution_count": null,
30 "metadata": {},
31 "outputs": [],
32 "source": [
33 "import numpy as np\n",
34 "import pandas as pd"
35 ]
36 },
37 {
38 "cell_type": "markdown",
39 "metadata": {},
40 "source": [
41 "### Question 1\n",
42 "Import the data from `fraud_data.csv`. What percentage of the observations in the dataset are instances of fraud?\n",
43 "\n",
44 "*This function should return a float between 0 and 1.* "
45 ]
46 },
47 {
48 "cell_type": "code",
49 "execution_count": null,
50 "metadata": {
51 "collapsed": true
52 },
53 "outputs": [],
54 "source": [
55 "def answer_one():\n",
56 " \n",
57 " # Your code here\n",
58 " \n",
59 " return # Return your answer\n"
60 ]
61 },
62 {
63 "cell_type": "code",
64 "execution_count": null,
65 "metadata": {},
66 "outputs": [],
67 "source": [
68 "# Use X_train, X_test, y_train, y_test for all of the following questions\n",
69 "from sklearn.model_selection import train_test_split\n",
70 "\n",
71 "df = pd.read_csv('fraud_data.csv')\n",
72 "\n",
73 "X = df.iloc[:,:-1]\n",
74 "y = df.iloc[:,-1]\n",
75 "\n",
76 "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
77 ]
78 },
79 {
80 "cell_type": "markdown",
81 "metadata": {},
82 "source": [
83 "### Question 2\n",
84 "\n",
85 "Using `X_train`, `X_test`, `y_train`, and `y_test` (as defined above), train a dummy classifier that classifies everything as the majority class of the training data. What is the accuracy of this classifier? What is the recall?\n",
86 "\n",
87 "*This function should a return a tuple with two floats, i.e. `(accuracy score, recall score)`.*"
88 ]
89 },
90 {
91 "cell_type": "code",
92 "execution_count": null,
93 "metadata": {},
94 "outputs": [],
95 "source": [
96 "def answer_two():\n",
97 " from sklearn.dummy import DummyClassifier\n",
98 " from sklearn.metrics import recall_score\n",
99 " \n",
100 " # Your code here\n",
101 " \n",
102 " return # Return your answer"
103 ]
104 },
105 {
106 "cell_type": "markdown",
107 "metadata": {},
108 "source": [
109 "### Question 3\n",
110 "\n",
111 "Using X_train, X_test, y_train, y_test (as defined above), train a SVC classifer using the default parameters. What is the accuracy, recall, and precision of this classifier?\n",
112 "\n",
113 "*This function should a return a tuple with three floats, i.e. `(accuracy score, recall score, precision score)`.*"
114 ]
115 },
116 {
117 "cell_type": "code",
118 "execution_count": null,
119 "metadata": {},
120 "outputs": [],
121 "source": [
122 "def answer_three():\n",
123 " from sklearn.metrics import recall_score, precision_score\n",
124 " from sklearn.svm import SVC\n",
125 "\n",
126 " # Your code here\n",
127 " \n",
128 " return # Return your answer"
129 ]
130 },
131 {
132 "cell_type": "markdown",
133 "metadata": {},
134 "source": [
135 "### Question 4\n",
136 "\n",
137 "Using the SVC classifier with parameters `{'C': 1e9, 'gamma': 1e-07}`, what is the confusion matrix when using a threshold of -220 on the decision function. Use X_test and y_test.\n",
138 "\n",
139 "*This function should return a confusion matrix, a 2x2 numpy array with 4 integers.*"
140 ]
141 },
142 {
143 "cell_type": "code",
144 "execution_count": null,
145 "metadata": {
146 "collapsed": true
147 },
148 "outputs": [],
149 "source": [
150 "def answer_four():\n",
151 " from sklearn.metrics import confusion_matrix\n",
152 " from sklearn.svm import SVC\n",
153 "\n",
154 " # Your code here\n",
155 " \n",
156 " return # Return your answer"
157 ]
158 },
159 {
160 "cell_type": "markdown",
161 "metadata": {},
162 "source": [
163 "### Question 5\n",
164 "\n",
165 "Train a logisitic regression classifier with default parameters using X_train and y_train.\n",
166 "\n",
167 "For the logisitic regression classifier, create a precision recall curve and a roc curve using y_test and the probability estimates for X_test (probability it is fraud).\n",
168 "\n",
169 "Looking at the precision recall curve, what is the recall when the precision is `0.75`?\n",
170 "\n",
171 "Looking at the roc curve, what is the true positive rate when the false positive rate is `0.16`?\n",
172 "\n",
173 "*This function should return a tuple with two floats, i.e. `(recall, true positive rate)`.*"
174 ]
175 },
176 {
177 "cell_type": "code",
178 "execution_count": null,
179 "metadata": {
180 "collapsed": true
181 },
182 "outputs": [],
183 "source": [
184 "def answer_five():\n",
185 " \n",
186 " # Your code here\n",
187 " \n",
188 " return # Return your answer"
189 ]
190 },
191 {
192 "cell_type": "markdown",
193 "metadata": {},
194 "source": [
195 "### Question 6\n",
196 "\n",
197 "Perform a grid search over the parameters listed below for a Logisitic Regression classifier, using recall for scoring and the default 3-fold cross validation.\n",
198 "\n",
199 "`'penalty': ['l1', 'l2']`\n",
200 "\n",
201 "`'C':[0.01, 0.1, 1, 10, 100]`\n",
202 "\n",
203 "From `.cv_results_`, create an array of the mean test scores of each parameter combination. i.e.\n",
204 "\n",
205 "| \t| `l1` \t| `l2` \t|\n",
206 "|:----:\t|----\t|----\t|\n",
207 "| **`0.01`** \t| ?\t| ? \t|\n",
208 "| **`0.1`** \t| ?\t| ? \t|\n",
209 "| **`1`** \t| ?\t| ? \t|\n",
210 "| **`10`** \t| ?\t| ? \t|\n",
211 "| **`100`** \t| ?\t| ? \t|\n",
212 "\n",
213 "<br>\n",
214 "\n",
215 "*This function should return a 5 by 2 numpy array with 10 floats.* \n",
216 "\n",
217 "*Note: do not return a DataFrame, just the values denoted by '?' above in a numpy array. You might need to reshape your raw result to meet the format we are looking for.*"
218 ]
219 },
220 {
221 "cell_type": "code",
222 "execution_count": null,
223 "metadata": {
224 "collapsed": true
225 },
226 "outputs": [],
227 "source": [
228 "def answer_six(): \n",
229 " from sklearn.model_selection import GridSearchCV\n",
230 " from sklearn.linear_model import LogisticRegression\n",
231 "\n",
232 " # Your code here\n",
233 " \n",
234 " return # Return your answer"
235 ]
236 },
237 {
238 "cell_type": "code",
239 "execution_count": null,
240 "metadata": {},
241 "outputs": [],
242 "source": [
243 "# Use the following function to help visualize results from the grid search\n",
244 "def GridSearch_Heatmap(scores):\n",
245 " %matplotlib notebook\n",
246 " import seaborn as sns\n",
247 " import matplotlib.pyplot as plt\n",
248 " plt.figure()\n",
249 " sns.heatmap(scores.reshape(5,2), xticklabels=['l1','l2'], yticklabels=[0.01, 0.1, 1, 10, 100])\n",
250 " plt.yticks(rotation=0);\n",
251 "\n",
252 "#GridSearch_Heatmap(answer_six())"
253 ]
254 }
255 ],
256 "metadata": {
257 "coursera": {
258 "course_slug": "python-machine-learning",
259 "graded_item_id": "5yX9Z",
260 "launcher_item_id": "eqnV3",
261 "part_id": "Msnj0"
262 },
263 "kernelspec": {
264 "display_name": "Python 3",
265 "language": "python",
266 "name": "python3"
267 },
268 "language_info": {
269 "codemirror_mode": {
270 "name": "ipython",
271 "version": 3
272 },
273 "file_extension": ".py",
274 "mimetype": "text/x-python",
275 "name": "python",
276 "nbconvert_exporter": "python",
277 "pygments_lexer": "ipython3",
278 "version": "3.5.2"
279 }
280 },
281 "nbformat": 4,
282 "nbformat_minor": 2
283 }
284