{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "_You are currently looking at **version 1.2** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-machine-learning/resources/bANLa) course resource._\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Assignment 3 - Evaluation\n", "\n", "In this assignment you will train several models and evaluate how effectively they predict instances of fraud using data based on [this dataset from Kaggle](https://www.kaggle.com/dalpozz/creditcardfraud).\n", " \n", "Each row in `fraud_data.csv` corresponds to a credit card transaction. Features include confidential variables `V1` through `V28` as well as `Amount` which is the amount of the transaction. \n", " \n", "The target is stored in the `class` column, where a value of 1 corresponds to an instance of fraud and 0 corresponds to an instance of not fraud." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Question 1\n", "Import the data from `fraud_data.csv`. What percentage of the observations in the dataset are instances of fraud?\n", "\n", "*This function should return a float between 0 and 1.* " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "356" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def answer_one():\n", " \n", " df = pd.read_csv('Source/fraud_data.csv')\n", " \n", " \n", " return len(df[df.Class==1])/len(df)\n", "answer_one()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Use X_train, X_test, y_train, y_test for all of the following questions\n", "from sklearn.model_selection import train_test_split\n", "\n", "df = pd.read_csv('Source/fraud_data.csv')\n", "\n", "X = df.iloc[:,:-1]\n", "y = df.iloc[:,-1]\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Question 2\n", "\n", "Using `X_train`, `X_test`, `y_train`, and `y_test` (as defined above), train a dummy classifier that classifies everything as the majority class of the training data. What is the accuracy of this classifier? What is the recall?\n", "\n", "*This function should a return a tuple with two floats, i.e. `(accuracy score, recall score)`.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def answer_two():\n", " from sklearn.dummy import DummyClassifier\n", " from sklearn.metrics import recall_score\n", " \n", " # Your code here\n", " \n", " return # Return your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Question 3\n", "\n", "Using X_train, X_test, y_train, y_test (as defined above), train a SVC classifer using the default parameters. What is the accuracy, recall, and precision of this classifier?\n", "\n", "*This function should a return a tuple with three floats, i.e. `(accuracy score, recall score, precision score)`.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def answer_three():\n", " from sklearn.metrics import recall_score, precision_score\n", " from sklearn.svm import SVC\n", "\n", " # Your code here\n", " \n", " return # Return your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Question 4\n", "\n", "Using the SVC classifier with parameters `{'C': 1e9, 'gamma': 1e-07}`, what is the confusion matrix when using a threshold of -220 on the decision function. Use X_test and y_test.\n", "\n", "*This function should return a confusion matrix, a 2x2 numpy array with 4 integers.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def answer_four():\n", " from sklearn.metrics import confusion_matrix\n", " from sklearn.svm import SVC\n", "\n", " # Your code here\n", " \n", " return # Return your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Question 5\n", "\n", "Train a logisitic regression classifier with default parameters using X_train and y_train.\n", "\n", "For the logisitic regression classifier, create a precision recall curve and a roc curve using y_test and the probability estimates for X_test (probability it is fraud).\n", "\n", "Looking at the precision recall curve, what is the recall when the precision is `0.75`?\n", "\n", "Looking at the roc curve, what is the true positive rate when the false positive rate is `0.16`?\n", "\n", "*This function should return a tuple with two floats, i.e. `(recall, true positive rate)`.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def answer_five():\n", " \n", " # Your code here\n", " \n", " return # Return your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Question 6\n", "\n", "Perform a grid search over the parameters listed below for a Logisitic Regression classifier, using recall for scoring and the default 3-fold cross validation.\n", "\n", "`'penalty': ['l1', 'l2']`\n", "\n", "`'C':[0.01, 0.1, 1, 10, 100]`\n", "\n", "From `.cv_results_`, create an array of the mean test scores of each parameter combination. i.e.\n", "\n", "| \t| `l1` \t| `l2` \t|\n", "|:----:\t|----\t|----\t|\n", "| **`0.01`** \t| ?\t| ? \t|\n", "| **`0.1`** \t| ?\t| ? \t|\n", "| **`1`** \t| ?\t| ? \t|\n", "| **`10`** \t| ?\t| ? \t|\n", "| **`100`** \t| ?\t| ? \t|\n", "\n", "
\n", "\n", "*This function should return a 5 by 2 numpy array with 10 floats.* \n", "\n", "*Note: do not return a DataFrame, just the values denoted by '?' above in a numpy array. You might need to reshape your raw result to meet the format we are looking for.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def answer_six(): \n", " from sklearn.model_selection import GridSearchCV\n", " from sklearn.linear_model import LogisticRegression\n", "\n", " # Your code here\n", " \n", " return # Return your answer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Use the following function to help visualize results from the grid search\n", "def GridSearch_Heatmap(scores):\n", " %matplotlib notebook\n", " import seaborn as sns\n", " import matplotlib.pyplot as plt\n", " plt.figure()\n", " sns.heatmap(scores.reshape(5,2), xticklabels=['l1','l2'], yticklabels=[0.01, 0.1, 1, 10, 100])\n", " plt.yticks(rotation=0);\n", "\n", "#GridSearch_Heatmap(answer_six())" ] } ], "metadata": { "coursera": { "course_slug": "python-machine-learning", "graded_item_id": "5yX9Z", "launcher_item_id": "eqnV3", "part_id": "Msnj0" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" } }, "nbformat": 4, "nbformat_minor": 2 }