11-EX-binary-logistic-regression-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification Using Binary Logistic Regression\n",
    "\n",
    "Many supervised learning ML problems involve learning to predict a target variable's labels that assume a finite number of discrete values.\n",
    "\n",
    "There are many different ML algorithms used for classification.  They include:\n",
    "\n",
    "* logistic regression (binomial, multinominal)\n",
    "* support vector machine (SVM)\n",
    "* Ridge, Lasso, elasticNet\n",
    "* CART (also for regression)\n",
    "* AdaBoost (a \"boosted\" ensemble classifier, also for regression)\n",
    "* RandomForest (an ensemble method that can also do regression and survival models)\n",
    "* Neural networks of various sorts\n",
    "\n",
    "We're going to start our exploration of classifiers with the simple case of predicting a target's labels that have only two values.  We'll use the famous WI breast cancer data set.  It's not large, but it's large enough for our present purposes.  It's also not that easy for classifiers to perform well on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Descent\n",
    "\n",
    "The training of classifiers is often done using a _gradient descent_ method.  The gradient is the set of partial derivatives of a cost function to be minimized w.r.t. (\"with respect to\") the parameters to be estimated during training. These parameters are often referred to as _weights_, or _coefficients_.  The cost function is minimized by iterative evaluation of the gradient evaluated the the current values of the parameters, and adjusting the parameters by adjusting them using a specified \"learning rate.\"\n",
    "\n",
    "[Stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) (\"SGD\") is a gradient descent method that uses a randomly selected datapoint to evaluate the gradient at particular values of the parameters being learned.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient for a Simple Model\n",
    "\n",
    "For example, assuming an L2 loss function, the gradient for a conventional _regression_ model could look like \n",
    "\n",
    "\\begin{align}\n",
    "\\large\n",
    "\\frac{\\partial}{\\partial w} Loss(W) =  \\frac{\\partial}{\\partial w}\\mid y - h_w (x)\\;   \\mid^2 \\\\\n",
    "\\end{align}\n",
    "\n",
    "where:\n",
    "\n",
    "y is a vector of target, or dependent variable, values;    \n",
    "W is a vector of weights to be estimated(learned);   \n",
    "h is some activation function, possibly a linear identity \"transformation\";  \n",
    "h_w(x) is the the product of a vector of weights $h_w$ and input variables (features) __x__;  \n",
    "y - $h_w$(x) is a vector of errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Gradient Descent\n",
    "\n",
    "* A models' **w's** (weights) can be solved for analytically if the model is a standard linear regression model:\n",
    "    * **y** is a continuous measure;\n",
    "    * The RHS of the model equation is _linear in its parameters_, e.g. for \"P\" predictor variables X<sub>p</sub>: \n",
    "    \n",
    "    $\\large {w_0+w_1 * X_1+w_2 * X_2 +...w_P*X_P}$ \n",
    "      \n",
    "  \n",
    "* L2 Loss is a quadratic function of the **w<sub>p</sub>'s**.\n",
    "* For pretty much all other model forms, a closed form analytical solution isn't available, and so _interative_ use of the gradient is what's done. In the simple, one **w** case,  \n",
    "    * Start with an initial value of **w_i**\n",
    "    * Initialize **w_i**\n",
    "    * Loop until Loss(**w**) is minimized:  \n",
    "      \n",
    "        * $\\large {w_i \\gets w_i - \\alpha  \\frac{\\partial Loss}{\\partial w_i}}$ \n",
    "        \n",
    "where:\n",
    "$\\alpha$ is the step size, or _**learning rate**_ . \n",
    "\n",
    "Note that the _negative_ of the Grad is used in order to point in the direction of decreasing Loss.\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross Entropy Loss \n",
    "\n",
    "A common type of cost function that ML classifiers minimize is based on [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) an _Information Theory_ concept.  It provides a way to measure of dissimilarity between the predicted probabilities of target variable class labels, and what the actual class labels are.\n",
    "\n",
    "The \"conventional\" definition of cross entropy for sets of \"i\" predicted and actual _discrete_ events (like target labels) is:\n",
    "\n",
    "\\begin{align}\n",
    "\\large\n",
    "H(p,q) = - \\sum_{i} p_i log(q_i)\n",
    "\\end{align}\n",
    "\n",
    "As applied for training binary logistic ML algorithms:\n",
    "\n",
    "$p_i$ is the \"true\" probability of observation i's class label,  \n",
    "$q_i$ is the algorithm's predicted label probability.\n",
    "\n",
    "Assuming that the target variable's labels are 0 and 1, $p_i$ for case i = 1, and 1-$p_i$ = 0. $q_i$ = the predicted probability of 1, and 1-$q_i$ = the predicted probability of 0.\n",
    "\n",
    "The cost function to be minimized can be calculated as the sum of the cross-entropies across the cases i:\n",
    "\n",
    "\\begin{align}\n",
    "\\large\n",
    "C(params~to~be~learned) = - \\frac {1}{N} \\sum_{i=1}^{N} \\Big[ y_i log(\\hat {y_i})+(1-y_i)log(1-\\hat {y_i}) \\Big]\n",
    "\\end{align}\n",
    "\n",
    "where:\n",
    "\n",
    "$y_i$ is the true class label, 0 or 1, for case i;  \n",
    "$\\hat {y_i}$ is the prediction of the class label for case i, which will be in the range [0,1], a _probability_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading Modules To Use  \n",
    "\n",
    "We'll get the data from the scikit-learn dataset collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import linear_model  \n",
    "from sklearn.model_selection import GridSearchCV, train_test_split\n",
    "from sklearn.model_selection import StratifiedKFold, StratifiedShuffleSplit\n",
    "from sklearn import preprocessing\n",
    "from sklearn.base import clone\n",
    "from sklearn import datasets\n",
    "from sklearn.metrics import accuracy_score\n",
    "import pickle\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# WI Breast CA Data\n",
    "\n",
    "We'll get them from skikit-learn's datasets collection.  What we'll be import is a sklearn \"Bunch\" data thing.  But it behaves like a Python dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sklearn.utils.Bunch"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename'])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "breastCA=datasets.load_breast_cancer()\n",
    "type(breastCA)\n",
    "breastCA.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _breast_cancer_dataset:\n",
      "\n",
      "Breast cancer wisconsin (diagnostic) dataset\n",
      "--------------------------------------------\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    :Number of Instances: 569\n",
      "\n",
      "    :Number of Attributes: 30 numeric, predictive attributes and the class\n",
      "\n",
      "    :Attribute Information:\n",
      "        - radius (mean of distances from center to points on the perimeter)\n",
      "        - texture (standard deviation of gray-scale values)\n",
      "        - perimeter\n",
      "        - area\n",
      "        - smoothness (local variation in radius lengths)\n",
      "        - compactness (perimeter^2 / area - 1.0)\n",
      "        - concavity (severity of concave portions of the contour)\n",
      "        - concave points (number of concave portions of the contour)\n",
      "        - symmetry \n",
      "        - fractal dimension (\"coastline approximation\" - 1)\n",
      "\n",
      "        The mean, standard error, and \"worst\" or largest (mean of the three\n",
      "        largest values) of these features were computed for each image,\n",
      "        resulting in 30 features.  For instance, field 3 is Mean Radius, field\n",
      "        13 is Radius SE, field 23 is Worst Radius.\n",
      "\n",
      "        - class:\n",
      "                - WDBC-Malignant\n",
      "                - WDBC-Benign\n",
      "\n",
      "    :Summary Statistics:\n",
      "\n",
      "    ===================================== ====== ======\n",
      "                                           Min    Max\n",
      "    ===================================== ====== ======\n",
      "    radius (mean):                        6.981  28.11\n",
      "    texture (mean):                       9.71   39.28\n",
      "    perimeter (mean):                     43.79  188.5\n",
      "    area (mean):                          143.5  2501.0\n",
      "    smoothness (mean):                    0.053  0.163\n",
      "    compactness (mean):                   0.019  0.345\n",
      "    concavity (mean):                     0.0    0.427\n",
      "    concave points (mean):                0.0    0.201\n",
      "    symmetry (mean):                      0.106  0.304\n",
      "    fractal dimension (mean):             0.05   0.097\n",
      "    radius (standard error):              0.112  2.873\n",
      "    texture (standard error):             0.36   4.885\n",
      "    perimeter (standard error):           0.757  21.98\n",
      "    area (standard error):                6.802  542.2\n",
      "    smoothness (standard error):          0.002  0.031\n",
      "    compactness (standard error):         0.002  0.135\n",
      "    concavity (standard error):           0.0    0.396\n",
      "    concave points (standard error):      0.0    0.053\n",
      "    symmetry (standard error):            0.008  0.079\n",
      "    fractal dimension (standard error):   0.001  0.03\n",
      "    radius (worst):                       7.93   36.04\n",
      "    texture (worst):                      12.02  49.54\n",
      "    perimeter (worst):                    50.41  251.2\n",
      "    area (worst):                         185.2  4254.0\n",
      "    smoothness (worst):                   0.071  0.223\n",
      "    compactness (worst):                  0.027  1.058\n",
      "    concavity (worst):                    0.0    1.252\n",
      "    concave points (worst):               0.0    0.291\n",
      "    symmetry (worst):                     0.156  0.664\n",
      "    fractal dimension (worst):            0.055  0.208\n",
      "    ===================================== ====== ======\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Class Distribution: 212 - Malignant, 357 - Benign\n",
      "\n",
      "    :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian\n",
      "\n",
      "    :Donor: Nick Street\n",
      "\n",
      "    :Date: November, 1995\n",
      "\n",
      "This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.\n",
      "https://goo.gl/U2Uwz2\n",
      "\n",
      "Features are computed from a digitized image of a fine needle\n",
      "aspirate (FNA) of a breast mass.  They describe\n",
      "characteristics of the cell nuclei present in the image.\n",
      "\n",
      "Separating plane described above was obtained using\n",
      "Multisurface Method-Tree (MSM-T) [K. P. Bennett, \"Decision Tree\n",
      "Construction Via Linear Programming.\" Proceedings of the 4th\n",
      "Midwest Artificial Intelligence and Cognitive Science Society,\n",
      "pp. 97-101, 1992], a classification method which uses linear\n",
      "programming to construct a decision tree.  Relevant features\n",
      "were selected using an exhaustive search in the space of 1-4\n",
      "features and 1-3 separating planes.\n",
      "\n",
      "The actual linear program used to obtain the separating plane\n",
      "in the 3-dimensional space is that described in:\n",
      "[K. P. Bennett and O. L. Mangasarian: \"Robust Linear\n",
      "Programming Discrimination of Two Linearly Inseparable Sets\",\n",
      "Optimization Methods and Software 1, 1992, 23-34].\n",
      "\n",
      "This database is also available through the UW CS ftp server:\n",
      "\n",
      "ftp ftp.cs.wisc.edu\n",
      "cd math-prog/cpo-dataset/machine-learn/WDBC/\n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "   - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction \n",
      "     for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on \n",
      "     Electronic Imaging: Science and Technology, volume 1905, pages 861-870,\n",
      "     San Jose, CA, 1993.\n",
      "   - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and \n",
      "     prognosis via linear programming. Operations Research, 43(4), pages 570-577, \n",
      "     July-August 1995.\n",
      "   - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques\n",
      "     to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994) \n",
      "     163-171.\n"
     ]
    }
   ],
   "source": [
    "# The description of this dataset:\n",
    "print(breastCA['DESCR'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['mean radius', 'mean texture', 'mean perimeter', 'mean area',\n",
       "       'mean smoothness', 'mean compactness', 'mean concavity',\n",
       "       'mean concave points', 'mean symmetry', 'mean fractal dimension',\n",
       "       'radius error', 'texture error', 'perimeter error', 'area error',\n",
       "       'smoothness error', 'compactness error', 'concavity error',\n",
       "       'concave points error', 'symmetry error',\n",
       "       'fractal dimension error', 'worst radius', 'worst texture',\n",
       "       'worst perimeter', 'worst area', 'worst smoothness',\n",
       "       'worst compactness', 'worst concavity', 'worst concave points',\n",
       "       'worst symmetry', 'worst fractal dimension'], dtype='<U23')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "array(['malignant', 'benign'], dtype='<U9')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feasure names, target_name\n",
    "breastCA['feature_names']\n",
    "breastCA['target_names']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The names of the target categories are \"malignant\" and \"benign.\"  Let's see what the corresponding codes (\"labels\") are in the \"target\" variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1] [212 357]\n"
     ]
    }
   ],
   "source": [
    "target_values, value_counts = np.unique(breastCA['target'], return_counts=True)\n",
    "print(target_values, value_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know from what's in this dataset's \"DESCR\" that there are 212 cases classified as malignant.  So the target is coded 0=malignant, 1=benign.  As we prep this data for training our binary logistic classifier, we'll reverse this coding so that our models are predicting malignancy, target=1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Doing CV'ed Binary Logistic Regression\n",
    "\n",
    "We need to to the \"usual\" creation of numpy arrays. Then we're going to go about our CV, including doing MinMax rescaling of features within CV folds.\n",
    "\n",
    "Here's where you can find the documentation on the [Logistic Regression Algorithm](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Munging the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the numpy arrays we need:\n",
    "\n",
    "X=breastCA['data']  # features\n",
    "y=breastCA['target'] # labels: 0=malignancy, 1=benign\n",
    "y=1-y                # relabelled: 0=benign, 1=malignancy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Stratified KFold CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stratification will create similar proportions of target values in the folds\n",
    "# The result may be decreased variance\n",
    "\n",
    "skf = StratifiedKFold(n_splits=20, random_state=99,shuffle=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a MinMax Scaler and a logit Regression Model Instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MinMax scaler\n",
    "\n",
    "scaler=preprocessing.MinMaxScaler()\n",
    "\n",
    "#  logistic regression algorithm\n",
    " # logreg alg instance; using defaults except explicit spec for solver\n",
    "\n",
    "logit_clf=linear_model.LogisticRegression(solver='lbfgs')   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#\n",
    "cvres=[]  # list to hold fold results\n",
    "#\n",
    "for train_ndx, test_ndx in skf.split(X, y):\n",
    "    clone_clf = clone(logit_clf)\n",
    "    X_trainS=scaler.fit_transform(X[train_ndx])\n",
    "    y_train = y[train_ndx]\n",
    "    X_testS=scaler.fit_transform(X[test_ndx])\n",
    "    y_test = y[test_ndx]\n",
    "\n",
    "    foldfit=clone_clf.fit(X_trainS, y_train)\n",
    "\n",
    "    y_pred_test=foldfit.predict(X_testS)\n",
    "    y_pred_train=foldfit.predict(X_trainS)\n",
    "    \n",
    "    trainAcc=accuracy_score(y_train,y_pred_train)\n",
    "    testAcc=accuracy_score(y_test,y_pred_test)\n",
    "    cvres.append({'train_accuracy':trainAcc,'test_accuracy':testAcc})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>20.000000</td>\n",
       "      <td>20.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.972343</td>\n",
       "      <td>0.893783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.002586</td>\n",
       "      <td>0.091995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.968519</td>\n",
       "      <td>0.642857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.970370</td>\n",
       "      <td>0.882184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.972222</td>\n",
       "      <td>0.928571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.974074</td>\n",
       "      <td>0.939017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.979630</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       train_accuracy  test_accuracy\n",
       "count       20.000000      20.000000\n",
       "mean         0.972343       0.893783\n",
       "std          0.002586       0.091995\n",
       "min          0.968519       0.642857\n",
       "25%          0.970370       0.882184\n",
       "50%          0.972222       0.928571\n",
       "75%          0.974074       0.939017\n",
       "max          0.979630       1.000000"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cvres)[['train_accuracy','test_accuracy']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice anything interesting about the results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.94682499,  1.62151943,  1.89901363,  1.59851144,  0.5736114 ,\n",
       "         0.34446353,  1.39732337,  2.06521426,  0.55338493, -0.93793789,\n",
       "         1.24467273,  0.03424528,  0.96329828,  0.81705987,  0.06979314,\n",
       "        -0.60236723, -0.25995023,  0.27001301, -0.17996763, -0.62788904,\n",
       "         2.45716617,  2.18913158,  2.23106309,  1.73189442,  1.42796693,\n",
       "         0.79268846,  1.32296286,  2.6326004 ,  1.32951989,  0.32321998]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foldfit.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pickling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's [_pickle_ ](https://wiki.python.org/moin/UsingPickle) a random split into training and test data, along with _predicted_ labels and label _probabilities_ for the training and test target. We'll use these in another notebook that's about _classifier performance measurement_.\n",
    "\n",
    "_Pickling_ is a Python method for _serializing_ (creating a nonvolitile version of) Python objects.\n",
    "\n",
    "We'll put our data and predictions into a dict, that we'll then pickle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's get our training and test split \n",
    "\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    train_test_split(X,y,stratify=y,random_state=99,shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train label proportions [0.62676056 0.37323944]\n",
      "y_test label proportions [0.62937063 0.37062937]\n"
     ]
    }
   ],
   "source": [
    "# checking to see whether training and test data have similar proportions of responses\n",
    "\n",
    "unique_ytrain, counts_ytrain = np.unique(y_train, return_counts=True)\n",
    "unique_ytest, counts_ytest = np.unique(y_test, return_counts=True)\n",
    "\n",
    "print('y_train label proportions',counts_ytrain/len((y_train)))\n",
    "print('y_test label proportions',counts_ytest/len((y_test)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  logistic regression algorithm\n",
    " # logreg alg instance; using defaults except explicit spec for solver, and max iterations=10000\n",
    "\n",
    "logit_clf=linear_model.LogisticRegression(solver='lbfgs',max_iter=10000)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get predictions\n",
    "\n",
    "LogRegM=logit_clf.fit(X_train,y_train)  # instantiate model\n",
    "yTrainPredLabels=LogRegM.predict(X_train)   # pred training labels\n",
    "yTestPredLabels=LogRegM.predict(X_test)     # pred test labels\n",
    "yTrainPredProbs=LogRegM.predict_proba(X_train) # pred training probs\n",
    "yTestPredProbs=LogRegM.predict_proba(X_test)   # pred test probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 0.96\n",
      "Test Accuracy: 0.94\n"
     ]
    }
   ],
   "source": [
    "# Get a quick look at the training and test accuracies\n",
    "\n",
    "print('Training Accuracy: {0:1.2f}'.format(accuracy_score(yTrainPredLabels,y_train)))\n",
    "print('Test Accuracy: {0:1.2f}'.format(accuracy_score(yTestPredLabels,y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Dictionary of Data to Serialize\n",
    "\n",
    "The predicted class probabilities are are N x 2 arrays, so we'll save just the col that's for the label of 1.  Then we'll create our dict.  After that, we'll write it to a pickle file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['X_train', 'y_train', 'X_test', 'y_test', 'yTestPredLabels', 'yTestPredProbs', 'yTrainPredLabels', 'yTrainPredProbs'])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yTrainPredProbs=yTrainPredProbs\n",
    "yTestPredProbs=yTestPredProbs\n",
    "\n",
    "breastCADict = {'X_train':X_train,'y_train':y_train,\n",
    "               'X_test': X_test, 'y_test':y_test,\n",
    "               'yTestPredLabels':yTestPredLabels,\n",
    "               'yTestPredProbs':yTestPredProbs,\n",
    "               'yTrainPredLabels':yTrainPredLabels,\n",
    "               'yTrainPredProbs':yTrainPredProbs}\n",
    "breastCADict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pickle it to the pwd!\n",
    "with open('DATA/ML/rBinLogData2.pkl','wb') as pickleOutFile: # write, binary format\n",
    "    pickle.dump(breastCADict,pickleOutFile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that another way to serialize several objects is using the [shelve module](https://docs.python.org/3.5/library/shelve.html).  A shelve database is like a dictionary database. Items in it are stored and retreived using keys."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding Some Regularization\n",
    "\n",
    "This logistic regression algorithm can apply some shrinkage to the weights (coefficients) that it learns when it's trained.  In this scikit-learn implementation, there's a penalty parameter C that when made smaller _increases_ the amount of regularization.  Let's do a grid search on values of this parameter to see if we can find an improved to test set data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A UDU 4U:  Grid Search for Improved Logistic Regression Accuracy\n",
    "\n",
    "Adapt the grid search code in the Ridge Regression notebook to do it.\n",
    "\n",
    "## The Ridge Regression Notebook with Grid Search: EX-Ridge-v1 \n",
    "\n",
    "To get you started, here's most of the code for the Ridge grid search.  Note that the regularization parameter, __C__, _increases_ the amount of regularization as it gets _smaller_.  So, if C=0.5, there is _more_ parameter shrinkage than if C=1.0. (In fact, at 1.0, there isn't any.)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# ridgeReg=linear_model.Ridge()  # ridge model instance\n",
    "alpha_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100,1000]}\n",
    "grid = GridSearchCV(ridgeReg, param_grid=alpha_grid, cv=10)\n",
    "grid.fit(X_train, y_train)\n",
    "print(\"Best cross-validation accuracy: {:.2f}\".format(grid.best_score_))\n",
    "print(\"Best parameters: \", grid.best_params_)\n",
    "print(\"Test set accuracy: {:.2f}\".format(grid.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Notebook: EX-Classifier-Performance-Measurement-v1"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "332.6px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}