19-NN-Intro-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Neural Networks\n",
    "\n",
    "* A category of machine learning methods.\n",
    "* Most frequent application objective is maximizing generalizing predictive accuracy.\n",
    "* Like other ML methods, not in general for theory or hypothesis testing. (Not yet, anyway.)\n",
    "* Conceptually speaking, NN's are networks of equations that include weights that are \"learned\" during training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Key Developments\n",
    "\n",
    "A history of related developments in AI and bioscience:\n",
    "\n",
    "* McCulloch & Pitts  (1943): artificial neuron\n",
    "* Hubel & Wiesel (1959): organization of central nervous system in cats, primates\n",
    "* Rosenblatt (1957), Minsky & Papert (1969, 1987): “perceptron”\n",
    "* Rumelhart & McClelland (1986, 1987): parallel distributed processing\n",
    "* Carew, Castellucci & Kandel (1971): learning at the level of the single neuron\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "<center><img src=\"images/rosenblatt-perceptron.png\" alt=\"Rosenblatt Perceptron\" width=\"800\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><h1>Biomimicry of Sorts</h1></center>  \n",
    "  \n",
    "  \n",
    "  \n",
    "<br>\n",
    "<center><img src=\"images/neurons2.png\" alt=\"neurons\" width=\"800\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A Broad Array of NN Architectures Has Developed\n",
    "\n",
    "\n",
    "<br>\n",
    "<center><img src=\"images/NN-zoo.png\" alt=\"NN Zoo\" width=\"1000\"></center>\n",
    "\n",
    "See [Van Zeen (2016), \"The Neural Network Zoo\"](http://www.asimovinstitute.org/neural-network-zoo/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#  \"Popular\" NN Architectures\n",
    "\n",
    "* Multilayer Perceptron.\n",
    "* Convolutional Neural Network.\n",
    "* Recursive Neural Network.\n",
    "* Generalizing Adversarial Network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Prototypical  Feedforward Multilayer Perceptron \n",
    "\n",
    "<br><br><br>\n",
    "<center><img src=\"images/mlp-1.png\" alt=\"multilayer perceptron\" width=\"800\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Activation Functions\n",
    "\n",
    "<center><img src=\"images/activations.png\" alt=\"activation functions\" width=\"800\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Backpropagation and Gradient Descent\n",
    "\n",
    "* NN training involves:\n",
    "    * Calculating the gradient of the loss function with respect to parameters (“weights”).\n",
    "        * Gradient is the set of partial derivatives of the loss function w.r.t. the weights.\n",
    "* \"Backprop\" is used to calculate the gradient\n",
    "    * Most loss functions are non-convex due to NN nonlinearity\n",
    "    * Implementations involve use of the diff calc chain rule; some also involve extension of the computational graph to include partial derivatives, reducing the amount of computation required.\n",
    "* Gradient descent involves adjusting the weights using the grad and a \"learning rate parameter\" to decrease the loss function.\n",
    "    * Various gradient descent versions, including batch, stochastic, mini-batch\n",
    "    * Sebastian Ruder on the variety of gradient descent methods:  \n",
    "        * [An overview of gradient descent optimization algorithms (paper)](https://arxiv.org/abs/1609.04747)  \n",
    "        * [An overview of gradient descent optimization algorithms (blog)](http://ruder.io/optimizing-gradient-descent/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1>About the Nature of NN Input Data</h1>\n",
    "\n",
    "* Computational libraries used for training NN's such as [Tensorflow](https://www.tensorflow.org/) and [Theano](http://www.deeplearning.net/software/theano/) employ a kind of matrix algebra for their computations.\n",
    "    A [\"tensor\"](https://en.wikipedia.org/wiki/Tensor) is a mathematical objects that is organized with respect to \"tensor algebra,\" a set of admissibe transformational procedures like multiplication.  Some multidimensional arrays are tensors.  \n",
    "\n",
    "* Input data that are categorical need to be transformed into variables that can be treated as continuous measures.\n",
    "\n",
    "* Methods for transforming categorical inputs include:\n",
    "\n",
    "    * Dummy, or \"one hot, encoding\n",
    "    * \"Hashing:\" \"bucketing a large number of categories\n",
    "    * \"Embedding:\"  categories representing in a possibly high dimensional vector space"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}