Tennis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tennis\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the third project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Tennis.app\"`\n",
    "- **Windows** (x86): `\"path/to/Tennis_Windows_x86/Tennis.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Tennis_Windows_x86_64/Tennis.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Tennis_Linux/Tennis.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Tennis_Linux/Tennis.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Tennis.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Tennis.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\"./Tennis.app\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, two agents control rackets to bounce a ball over a net. If an agent hits the ball over the net, it receives a reward of +0.1.  If an agent lets a ball hit the ground or hits the ball out of bounds, it receives a reward of -0.01.  Thus, the goal of each agent is to keep the ball in play.\n",
    "\n",
    "The observation space consists of 8 variables corresponding to the position and velocity of the ball and racket. Two continuous actions are available, corresponding to movement toward (or away from) the net, and jumping. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.         -6.65278625 -1.5\n",
      " -0.          0.          6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  collections  import deque\n",
    "from itertools import count\n",
    "import time\n",
    "import torch\n",
    "from ddpg_agent import Agent\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "agent1 = Agent(state_size=state_size+1, action_size=action_size, random_seed=15)\n",
    "agent2 = Agent(state_size=state_size+1, action_size=action_size, random_seed=15)\n",
    "agent2.critic_local = agent1.critic_local\n",
    "agent2.critic_target = agent1.critic_target\n",
    "agent2.critic_optimizer = agent1.critic_optimizer\n",
    "\n",
    "agent2.actor_local = agent1.actor_local\n",
    "agent2.actor_target = agent1.actor_target\n",
    "agent2.actor_optimizer = agent1.actor_optimizer\n",
    "\n",
    "agent2.memory = agent1.memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 0\t Mean Score for Agent 1: 0.000, Mean Score for Agent 2: -0.010, Mean Score of both Agents: -0.005\n",
      "Episode 100\t Mean Score for Agent 1: -0.005, Mean Score for Agent 2: -0.005, Mean Score of both Agents: -0.005\n",
      "Episode 200\t Mean Score for Agent 1: -0.003, Mean Score for Agent 2: -0.004, Mean Score of both Agents: -0.003\n",
      "Episode 300\t Mean Score for Agent 1: -0.005, Mean Score for Agent 2: -0.005, Mean Score of both Agents: -0.005\n",
      "Episode 400\t Mean Score for Agent 1: -0.005, Mean Score for Agent 2: -0.005, Mean Score of both Agents: -0.005\n",
      "Episode 500\t Mean Score for Agent 1: -0.004, Mean Score for Agent 2: -0.006, Mean Score of both Agents: -0.005\n",
      "Episode 600\t Mean Score for Agent 1: -0.002, Mean Score for Agent 2: -0.003, Mean Score of both Agents: -0.003\n",
      "Episode 700\t Mean Score for Agent 1: -0.002, Mean Score for Agent 2: -0.003, Mean Score of both Agents: -0.002\n",
      "Episode 800\t Mean Score for Agent 1: 0.001, Mean Score for Agent 2: 0.004, Mean Score of both Agents: 0.003\n",
      "Episode 900\t Mean Score for Agent 1: 0.008, Mean Score for Agent 2: 0.004, Mean Score of both Agents: 0.006\n",
      "Episode 1000\t Mean Score for Agent 1: 0.026, Mean Score for Agent 2: 0.018, Mean Score of both Agents: 0.022\n",
      "Episode 1100\t Mean Score for Agent 1: 0.061, Mean Score for Agent 2: 0.056, Mean Score of both Agents: 0.059\n",
      "Episode 1200\t Mean Score for Agent 1: 0.113, Mean Score for Agent 2: 0.110, Mean Score of both Agents: 0.112\n",
      "Episode 1300\t Mean Score for Agent 1: 0.112, Mean Score for Agent 2: 0.111, Mean Score of both Agents: 0.111\n",
      "Episode 1400\t Mean Score for Agent 1: 0.130, Mean Score for Agent 2: 0.096, Mean Score of both Agents: 0.113\n",
      "Episode 1500\t Mean Score for Agent 1: 0.292, Mean Score for Agent 2: 0.290, Mean Score of both Agents: 0.291\n",
      "Episode 1600\t Mean Score for Agent 1: 0.402, Mean Score for Agent 2: 0.391, Mean Score of both Agents: 0.396\n",
      "Episode 1700\t Mean Score for Agent 1: 0.392, Mean Score for Agent 2: 0.391, Mean Score of both Agents: 0.392\n",
      "Episode 1800\t Mean Score for Agent 1: 0.130, Mean Score for Agent 2: 0.130, Mean Score of both Agents: 0.130\n",
      "Episode 1900\t Mean Score for Agent 1: 0.650, Mean Score for Agent 2: 0.643, Mean Score of both Agents: 0.647\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xu4HFWZ7/HvCwnIVS7ZYARCQMMolzlAIjfBQRQMqAERDB7PyCCePF4QefDgBFSIM+NtBDxyEAEHBITDRQKcqHEAEQRRIkkMl5ABwk0CgSSgiRAiJr7nj6reu9O7u3d1d62q1d2/z/P009V1fbt27/VWrVW1ytwdERERgA3KDkBEROKhpCAiIoOUFEREZJCSgoiIDFJSEBGRQUoKIiIySElBREQGKSmIiMggJQURERk0quwAWjVmzBgfP3582WGIiHSVefPmrXD3gZHm67qkMH78eObOnVt2GCIiXcXMnskyn6qPRERkkJKCiIgMUlIQEZFBSgoiIjJISUFERAYpKYiIyCAlBRERGaSkICLxue8+WLCg7Cg68/rrcMUVcMMN8NJLw6ffeSdMngx33z182vLlMHMmrFgBm24KBx4IU6bA7bcHD7vrbl4TkT5w4IHJezc/Q37GDPjGN5LhQw9NkkC1ww5L3m+9dfj3PPpo+O1vhz7fd1/yvtdecPjhIaIdpDMFEZEQliwZGn7qqdaWXby4/vhNN20/noyUFEREQvjrX4eGzfJZ52ab5bOeJpQURKQ806bBlVdmn//00+H442H0aLjooqHxK1YkVS7HHjtU1VJt5Uo45hhYtiz5/MQTcNxxsGZNZ/E3U50UqquHLroITjpp/Xnf/3745CeH5muURAo4U8Ddu+o1ceJEF5EekRSDrY+vnT59+tC4Lbccvty55ybTTj89+Tx5cvJ59ux8vkc9U6YMxVRdbtV+h+rXqlXJPNttV3/6dde1HQ4w1zOUsTpTEJH+UWTDdfWZwpgx+axz553zWU8TSgoi0lvqFfyNqmNCJolG1UfNjFR9VAAlBRHJ3yWXJAXbn/6UbX4zuPrq4ePXrs2+fBZFFrrVSeG22+CLX8y+bC8mBTPbyczuNLNFZrbQzD5fZ55DzWylmS1IX2eHikdECnThhcn7s89mX+YHPxg+rrpg7Ta1Zwff/nY5cbQo5M1ra4EvuPt8M9sCmGdmt7v7IzXz3ePuHwgYh4hIbyjgDCLYmYK7L3X3+enwn4FFwA6hticiArTWphCrXqw+qmZm44F9gDl1Jh9oZg+Y2c/NbI8Gy08zs7lmNnf58uUBIxWR0tQrzPNuCK5dX2zdaPRDQ7OZbQ7MBE5z91U1k+cDO7v7fwP+D3BLvXW4+6XuPsndJw0MDIQNWES6W5YzhSIK3dgSTkZBk4KZjSZJCNe4+0210919lbu/kg7PBkabWU4X9IpIV6lXUHdbtU9eevFMwcwMuAxY5O7nN5jnTel8mNl+aTx1+pgVkZ7w0kvwyiv5r7e6EG1WoC5alP+2i1RAsgh59dE7gX8EHjKzSsfoZwHjANz9YuA44NNmthZ4DTghvR1bRHrRmDGw/fbwwgvDp+X1r99sPbfdtv7zC2Itbko8UwiWFNz910DTb+buFwIXhopBRCL04ovFb7O6kF24sH+rpTLQHc0iEoe8CupYjv47iaMX2xRERFoSS2Geh5tvhueea325CC5J1eM4RaT7xVQd5J481yGEbr6jWUSkY3mdPZTRS2o7IohHSUFEekuWgjWmM4t61KYgIlLHSIXjvHnJ4znXrWttPT/9afIewZH5eipxKimIiLThIx+BG2+Ep58uO5IhnSSaCJKUkoKIxCtrIdnukXXs1Ui11NAsIl0pgiPeTGKNU9VHIiJ1tFM46nkKHVFSEJH8dUshHOJMIY91KimIiNQRa/VOKBF8XyUFEZFuoYZmEZEWqU2hI0oKIlKOPKtKYir01aYgIhJIv/V9FAElBRGJgwroKLrOVlIQkfypgA9DDc0i0rNCJY6yE1IefR/pTEFEeko7hVonBeFIy8bUEB05JQURiUO9I+zQR/1ln1XU0pmCiIhkpjYFEZEWxXb03w6dKYhI36ktvEMWhEUWsnrIjohIDjppU2i30I+gEK5LZwoiIhJDklJSEJH8RVC4ZdJtl6qqoVlEpEWt9JKqh+wMEywpmNlOZnanmS0ys4Vm9vk685iZXWBmi83sQTPbN1Q8IlKgLIValsIzdJtCbCK4T2FUwHWvBb7g7vPNbAtgnpnd7u6PVM1zJDAhfe0PfD99FxHJrjp5lH2mkIdePFNw96XuPj8d/jOwCNihZrajgas8cR+wlZmNDRWTiBSkqMI2giPrXEWQpAppUzCz8cA+wJyaSTsAz1Z9XsLwxCEi3aqVwjqCAjEXIb9HLzQ0m9nmwEzgNHdfVTu5ziLD9qiZTTOzuWY2d/ny5SHCFJEQmhWQebQp9MoZQq1erD4CMLPRJAnhGne/qc4sS4Cdqj7vCDxfO5O7X+ruk9x90sDAQJhgRSQ/ZfaS2s3PaI6gOizk1UcGXAYscvfzG8w2C/h4ehXSAcBKd18aKiYR6TGdVtX0SpVVjkJeffRO4B+Bh8xsQTruLGAcgLtfDMwGjgIWA6uBkwLGIyJFKLLLiaeeGnn7G28Mo0IWdTW6/CE7wfaUu/+a+m0G1fM48NlQMYhIl2ungK1d5vXXk1cv6IWGZhGRuorsJbVbRHCmoKQgIt1vpIbmRtSmMIySgojEoVcKaPV9JCLSgKqEWhNBYlRSEJFwQt+81m/U0Cwi0qJuTiQRxK6kICL9K4JCuC61KYiI9IhYE01GSgoiEq9+fchOiZQURKQcERSAXUcNzSIifSSCRKmkICL9K0QhrJvXREQa6LRwi+DIud8oKYhIOJ3evJaVGppzo6QgItIt1NAsIn0ngqPljuTxkJ0SKSmISLxCF5IRFMJ1qaFZREQGk1SJyUpJQUTKoYbm1qlNQUSky+g+BRGRHJRRdRJbm0IE8SgpiEg4unmt6ygpiEg4KtRbE8H+UlIQkXLUFoC90lgcsmBXQ7OI9I0ILscUJQURiVnoh+zEloAq8ejqIxHpSb1SJdRHlBREJBx3WLYM1qypP61sMcRQLYJ4giUFM7vczJaZ2cMNph9qZivNbEH6OjtULCJSou23h8MPLzuK4nR5Q/OogOu+ArgQuKrJPPe4+wcCxiAiMfj1r7PPW12ohj5yVvXWMMHOFNz9buDlUOsXkS5QVKGrhubclN2mcKCZPWBmPzezPUqORUTyFluhKyMKWX00kvnAzu7+ipkdBdwCTKg3o5lNA6YBjBs3rrgIRSScXk0Y/fKQHTM72MxOSocHzGyXTjbs7qvc/ZV0eDYw2szGNJj3Unef5O6TBgYGOtmsiHSTCArJqMRyR7OZnQP8M3BmOmo0cHUnGzazN5kl39DM9ktjeamTdYpID2gnEfRam0KJslYffQjYh6TKB3d/3sy2aLaAmV0LHAqMMbMlwDkkyQR3vxg4Dvi0ma0FXgNOcI9gj4hIfloprPXvP6TEhuasSeF1d3czcwAz22ykBdz9oyNMv5DkklUR6VXNCvo8k0BMCSWmWNqQtU3hBjO7BNjKzP4n8AvgB+HCEpGu1EmBWO/oOHTfR7GJIKFkOlNw93PN7HBgFfB3wNnufnvQyEREZH0x3NFsZhsCt7r7ewElAhHJrp02BTU0l2rE6iN3XwesNrM3FhCPiPSSCAq5wnX5d87a0LwGeMjMbgderYx091ODRCUi3amVAjHPJ691eUE8TBdcffSz9CUiZVm8GN74RuinGzj7taG5xCSXtaH5SjPbCNgtHfWou/81XFgiMsyECbDJJrB6ddmRhBFBgRi9GBqaIXn2AXAl8DRgwE5mdmLaE6qIFOW118qOoLk8q4SKECIB5dH3URdUH50HHOHujwKY2W7AtcDEUIGJSA/ox5vXulzWm9dGVxICgLs/RtplhYhIMKEL+1iTSRecKcw1s8uAH6WfPwbMCxOSiEiLYq+mymqkdpVY2hSATwOfBU4laVO4G7goVFAi0qXyaFNo5+i93SP+EIVsrGcfGWVNCqOA77r7+TB4l/PGwaISESlCbAV4BA3NWdsU7gA2qfq8CUmneCIijXXa0Bxbod0HsiaFN1SekgaQDm8aJiQR6VrdVojHFm8E8WRNCq+a2b6VD2Y2ieTBOCIiUpSIGppPA35sZs8DDrwZmBosKhHpDXk2NBd1z0OnYoqlDU3PFMzsHWb2Jne/H3gbcD2wFvhP4KkC4hORbtZpAdmvfR9F3NB8CfB6OnwgcBbwPeCPwKUB4xKRbtRJL6kShZGqjzZ095fT4anApe4+E5hpZgvChiYiElhsiSmCeEY6U9jQzCqJ4z3AL6umZW2PEBHJLnSbwrzAnTGELNgjaGi+FviVma0gudroHgAzeyuwMnBsItJt8ughNK/1NXLZZfmvs4c0TQru/jUzuwMYC9zmPvgX2gD4XOjgRKTLFXXFUK81NJdoxCogd7+vzrjHwoQjIn2rjII9gkK4roivPhIRKUasBXSr8qhCK3FfKCmISH7yKMy6/ea1kAo4g1BSEJFwirp5rdeo+khE+k6ez3NWQ3NugiUFM7vczJaZ2cMNppuZXWBmi83sweoO90SkS+V9SWroZUMUwnmss0fPFK4AJjeZfiQwIX1NA74fMBYRkfj1ckOzu98NvNxklqOBqzxxH7CVmY0NFY+IdInqArHRcL/q8YbmHYBnqz4vSceJSK9o5Yqh556DlW12lNBrbQo9Wn00knrfuu4vyMymmdlcM5u7fPnywGGJSNs6OZp/9lnYc898t1vG2UWXn9GUmRSWADtVfd4ReL7ejO5+qbtPcvdJAwMDhQQnIiVYsqTY7XV5AR5CmUlhFvDx9CqkA4CV7r60xHhEJG/tFLr93KYQwXcM1v21mV0LHAqMMbMlwDnAaAB3vxiYDRwFLAZWAyeFikVEChJBobae2OLpVARdZ7fN3T86wnQHPhtq+yISAXVN0Zo+b2gWkV5XVMEfU4KJKZY2KCmISLxCF7BdXoCHoKQgIvmpLWQ7bWjOuq7a6pY5c9rffpkiiFdJQUTCKatN4dOfDrfuMvX4Hc0iIvmI4Ah7UKuxHHZY+8sGoKQgIuE0KuTWru1s+byWi6AQrktXH4lIT8hayI4eDT/8YX7brS1EI+htNLPI+m1SUhCRcJoVyjfc0Noy3VDAt6M6KYz0HdWmICJdraiG5jyuespLlycvJQURyU8sBWI3VR9ViyBeJQURCafTQq4fbl6rVyWkhmYRkVQeN6/FUNhnVS8B9OLjOEVESuv7qFvbFNTQLCIt+9a34JFHyo6ivlYK5yIK7m46Y6im6iMRyWz6dDjooLKjKEbom9diE0HcSgoi3ei118qOIJuyCrluuqNZN6+JSM/Ko5DN8+a1Mgr9VrfZys1rBVBSEJFw+vHmtS6npCDSjSKrcmiorPsUuj0pqKFZRKQD7RaiMSSPVqqPCohXSUFEOuMOzz03NByT2OKpJ7KzPiUFEenMuefCjjvCo48On1bU4zhjalMIefNaAZQURKQzv/hF8v7008OnddrQ3K9tCo3ojmYR6SplFcaNCsteSw5qUxDpQzfeCCtWlB1FPooqlH/60/a2Wz3funXJ0+DWrWt9+3fdVb/6LIt6Dc26+khEAHjxRTj+eDjmmLIjKU87bQovvph93kYuugg+8Qn4/vdbX/bd74a3va29bauhWUQaev315P2ZZ5rPF1lB0lA3Vd9Uzs7KPEuLYH8pKYhIfvLuJbXThuZWlu+WRBuYkoJIP/vb3+DEE+H3vy9+2yELYXeYORPOOSfcNvISWTIKmhTMbLKZPWpmi81sep3p/2Rmy81sQfr6ZMh4RKTGM8/AVVfBsceGWX+Zz1M47rjkHopmQsSQR4d4JV5NNSrUis1sQ+B7wOHAEuB+M5vl7rVPB7ne3U8JFYdIT4rs6HJQLL2kRlA3361CninsByx29yfd/XXgOuDogNsTkdjE3iFebMmjnbaQnIVMCjsAz1Z9XpKOq/VhM3vQzG40s53qrcjMppnZXDObu3z58hCxipTv/PNh3Lhs88ZWmDVSVvXRSFdvNRPzvu3yO5rrRV+7t38CjHf3vwd+AVxZb0Xufqm7T3L3SQMDAzmHKRKJ884rO4L+U13I5lXghnzITpff0bwEqD7y3xF4vnoGd3/J3f+SfvwBMDFgPCJxa6VQ6pY2hWaFWAzfIYazghj2Q5WQSeF+YIKZ7WJmGwEnALOqZzCzsVUfpwCLAsYjErfICodctFN9VD2+0XCe3OF3vxv+eaTtvfIKLFwYJqYSBUsK7r4WOAW4laSwv8HdF5rZv5jZlHS2U81soZk9AJwK/FOoeESi14tJoZkYjtIBfvQj2H//pM8pgB//OPl89dXNl5syBfbcM7nXIy8RNDQHuyQVwN1nA7Nrxp1dNXwmcGbIGES6Ri8khW6rPgJYlFZQPP548l7p2G6kDu7uvDN5j+lZDjnQHc0i3SiWArUTMRSeIe+ryKqVhuYCKCmIxKJRQb9mDfzrv7a+vp/8BO6+G664orvqvotuU6io3f/uSbvBv/0brF3beLlOq49qe3htppvvaBaRFjVKCueeC2efXX9aM1OmrP+5jKPQCI58W2Y2FPfZZ8N3vgM77ZT0EVVPp0nhN7/pbPmc6UxBJBaNksKrr4bbZp6Ftnv+vaSWyX1o37/2WuP5apNCHs9obrSOLr95TURaEeJGqpGELphjL/hHskFaRHZzcmuRkoJILMpoPG5UoB18MFx+eWvrGin+sWPX/7wow21JRSStb34zGV6zJnmvfA/3oaTQrIooxCWp7U7PgZKCSCzKSAqNCrR774WTT259fc2qj154ofV1VMdXxhF50UkhAkoKIrGI6UwhlvWHLnBHqhaq/E2UFEQkamawcmV5XVMXZd264rdZnZwrZwqrVzeev3YfVp6z3Y4I7mhWUhCJRStnCq++ClttBZdd1tk26x3lxpQoQieFkfZ5ZfpZZzWeZ8GC9T8femhHIZVNSUEkFu1UH82ePfI8zdRLAHlcUpnHuqD86qMNMhSR8+ev/3nFijDxFERJQaSbZSm0mqlXCOV5dN5OIVe9THUsRRWY9a4+auavfw0bTzVdfSTSR9o5U7jlls6OpustW3ZSqBa6+ujhh4ePqy7kW00Kzz/feL4s1KYgIoPaSQrr1sGFF2abt94Rbb3CJ6araULH8h//0Xx6lr9J9X7de+/O4hmJ7mgW6SPt/sMvWZJtvqyNyp0cned9hFvG1UcV7VQftfMM+U98ovVlAlJSEIlF6KPAegVsv1cfjaSINoU+ekaziLSiUVK44IJsy331qzB3buP5sp4pZKmyueOOpPfQkXTa0FzmHc1ZzxTOPz98LAVSUhCJXbMbp6rNmAHveEfj6fWOututPnrve+H004evqx+rjzp12mnrb7P6vQRKCiKx6KT6KEshUu8MIM/qoxA3wpVdfVRE1yPbbht+Gy1QUhCJRSePZcxS5dPsTKHRvQEjCf1ktLKTQhFnCkVsoxXu3lWviRMnukgwr7ySVIL88If1p593XjL9b3/Lb5szZlQqXoa/7r238TS9euP1wgtDwzNnJr+JQw6pP+9TT7X9MwPmuo9cxkaWokRKtnRp8t7omchnnJG8N3tmb6tmzGg87cor89uOdD/34JtQUhCplqX/fCi/WqOfnXJK2RGEUyn0Cyj8G1FSEKmWNSk88ED4WKS+jTYqO4Ly6I5mkZKMdKR2wAHFxHHvvcVsp5uMGlV2BPkq4+FKTSgpiFSrVAvF0v/Po4+WHUF8Ro9uPv273w2z3f33D7NegFtvTd5HOhhRm4JIwSoNyFmSQidP2KoYqW0izwbtXjHSkfUhh4TZ7sBAmPUC7LBDuHW3qMfOw0rw5JPw4Q/D1KkwfXrZ0YT31FMwZw6ccEJyZczhh8Ob37z+PC+9BDNnwuabw8EHw7hx609ftw6+972k4D3ggKS74dWr4eWXYfJk2G23bLGsXg1f+cpQNwO77go33AATJw6f1x0uuST5Oz3+ePJglE99Cs45B1atgq23Tr7Pk08m8y9dOlT47Ldf8l222279ZHHAAbDHHsl6N90UHnsMTjoJvvY1mDUr23cosi/+XlFWI2wR273iCvjNb4Z+h2XIct1quy9gMvAosBiYXmf6xsD16fQ5wPiR1hndfQrbbDN0DXE/GDMm+a7Llyfv++wzfJ4PfnBon4wfP3z6tdc2vmbbLHsst91Wfx31zJmTTDv+ePeBgWR41ap8rjO//fZkG7XfY4stsr2Kvi5+9Oj647fdNt/tfOAD7S87YULjaQ8/3Hjaeee5r1iR7/fYbjv3XXZp/Htr9Nptt+T9jDOS30Ojda9Z4/7yy8n/SvXvot69Ck8+mf3/owYZ71MYcYZ2X8CGwBPArsBGwAPA7jXzfAa4OB0+Abh+pPVGlxSq/2D9oPJdFy8e+lHXmjRpaL56hfwllzT/Z8rqppuyL3/XXcm0d71raL4nnsin0Lj55vX3DbgfdFD275GHmTOT7R5zTPL5m98cKpDqufTSZPrJJzde5/PPuz/7bP6xdrOpU5P99vWvt7d8q7/xyvzjxxeWFEK2KewHLHb3J939deA64OiaeY4GKnfn3Ai8xyyypvhWxNI4WYRm/cYnST68V1/NPm8lpurYli0LF8dmm+Wz7qwqja+VNog3vCF5X7Om/vx/+UvyvvHGjdc5dizsuGM+8fWKyn7bdNNy4wgoZJvCDsCzVZ+XALXN94PzuPtaM1sJbAt08OTrBm69dXivjnnbY4/4+jEJZerU5H3ZsuR7V3vkkaFh9+HT//CH5uuunb+RP/4x+/KVgvuee4bGVb5Dp844A77+9fXHFV1oVJJAJTlsscX6n2tVjr0q80k2lYOKov++le0VcMwcMinUi772EDLLPJjZNGAawLjaRsusttwSdt+9vWWb2Xxz+N3vkuE998x//bHZYoukoXm//ZLC/R/+YfhVGTvuCLfdlgwfdNDwhui3vz1piAZ461th8eKhafvsA295S/Z4VqyAu+4a+rzbbo3/zs88kzRkr1qVPJu38h1GcuaZsNde8MtfJo3hN90ECxcm++GCC5LvALDVVkkj4bHHwsknZ/8OeTjsMDjrLDj11OTzhz6U/C4/97n68598crI/vvzl4mLsBV/4QvI/cMQR7S1/xx1DXalkccstSSLYay+45hrYeef2ttsC80Cn+mZ2IDDD3d+Xfj4TwN2/UTXPrek8vzWzUcALwIA3CWrSpEk+t9mDREREZBgzm+fuk0aaL2Rdx/3ABDPbxcw2ImlIrr1ObxZwYjp8HPDLZglBRETCClZ9lLYRnALcSnIl0uXuvtDM/oWkFXwWcBnwIzNbDLxMkjhERKQkQW9ec/fZwOyacWdXDa8Bjg8Zg4iIZNcnl8qIiEgWSgoiIjJISUFERAYpKYiIyCAlBRERGRTs5rVQzGw58Eybi48hRBca+VBsrYs1LlBs7Yg1LuiN2HZ29xEfCtF1SaETZjY3yx19ZVBsrYs1LlBs7Yg1Luiv2FR9JCIig5QURERkUL8lhUvLDqAJxda6WOMCxdaOWOOCPoqtr9oURESkuX47UxARkSb6JimY2WQze9TMFpvZ9IK3vZOZ3Wlmi8xsoZl9Ph0/w8yeM7MF6euoqmXOTGN91MzeFzi+p83soTSGuem4bczsdjN7PH3fOh1vZnZBGtuDZrZvwLj+rmrfLDCzVWZ2Whn7zcwuN7NlZvZw1biW95GZnZjO/7iZnVhvWznF9m0z+690+zeb2Vbp+PFm9lrVvru4apmJ6e9gcRp/x4/5ahBby3+/vP9/G8R1fVVMT5vZgnR80fusUXlRzO8ty4Ocu/1F0nX3E8CuwEbAA8DuBW5/LLBvOrwF8BiwOzAD+F915t89jXFjYJc09g0Dxvc0MKZm3L8D09Ph6cC30uGjgJ+TPDXvAGBOgX/DF4Cdy9hvwLuAfYGH291HwDbAk+n71unw1oFiOwIYlQ5/qyq28dXz1aznd8CBadw/B44MFFtLf78Q/7/14qqZfh5wdkn7rFF5UcjvrV/OFPYDFrv7k+7+OnAdcHRRG3f3pe4+Px3+M7CI5PnUjRwNXOfuf3H3p4DFJN+hSEcDV6bDVwLHVI2/yhP3AVuZ2dgC4nkP8IS7N7txMdh+c/e7SZ75Ubu9VvbR+4Db3f1ld/8jcDswOURs7n6bu69NP94H7NhsHWl8W7r7bz0pUa6q+j65xtZEo79f7v+/zeJKj/Y/AlzbbB0B91mj8qKQ31u/JIUdgGerPi+heaEcjJmNB/YB5qSjTklP+S6vnA5SfLwO3GZm8yx5HjbA9u6+FJIfKbBdSbFVnMD6/6Qx7LdW91FZ++4TJEeSFbuY2e/N7Fdmdkg6boc0nqJia+XvV/R+OwR40d0frxpXyj6rKS8K+b31S1KoV89X+GVXZrY5MBM4zd1XAd8H3gLsDSwlOWWF4uN9p7vvCxwJfNbM3tVk3sL3pSWPc50C/DgdFct+a6RRHGXsuy8Ba4Fr0lFLgXHuvg9wOvB/zWzLgmNr9e9X9H77KOsfgJSyz+qUFw1nbRBHW/H1S1JYAuxU9XlH4PkiAzCz0SR/4Gvc/SYAd3/R3de5+9+AHzBU1VFovO7+fPq+DLg5jePFSrVQ+r6sjNhSRwLz3f3FNM4o9hut76NC40sbFj8AfCyt3iCtmnkpHZ5HUle/WxpbdRVTsNja+PsVtt/MbBRwLHB9VbyF77N65QUF/d76JSncD0wws13So84TgFlFbTyto7wMWOTu51eNr66L/xBQuRJiFnCCmW1sZrsAE0gatELEtpmZbVEZJmmgfDiNoXK1wonA/6uK7ePpFQ8HACsrp7QBrXfkFsN+q9peK/voVuAIM9s6rTI5Ih2XOzObDPwzMMXdV1eNHzCzDdPhXUn20ZNpfH82swPS3+vHq75P3rG1+vcr8v/3vcB/uftgtVDR+6xReUFRv7dOW8q75UXSQv8YSZb/UsHbPpjktO1BYEH6Ogr4EfBQOn4WMLZqmS+lsT5KDlc0NIltV5KrOR4AFlb2DbAtcAfwePq+TTregO+lsT0ETAr8RFXgAAAC+UlEQVS87zYFXgLeWDWu8P1GkpSWAn8lOQI7uZ19RFK/vzh9nRQwtsUk9cmV39vF6bwfTv/ODwDzgQ9WrWcSSQH9BHAh6c2tAWJr+e+X9/9vvbjS8VcAn6qZt+h91qi8KOT3pjuaRURkUL9UH4mISAZKCiIiMkhJQUREBikpiIjIICUFEREZpKQgfcPM1tn6va427W3TzD5lZh/PYbtPm9mYNpZ7nyU9im5tZrM7jUMki1FlByBSoNfcfe+sM7v7xSPPFdQhwJ0kPXreW3Is0ieUFKTvmdnTJN0avDsd9d/dfbGZzQBecfdzzexU4FMk/Qg94u4nmNk2wOUkNwCuBqa5+4Nmti3JzVEDJHfkWtW2/gdwKkkX0HOAz7j7upp4pgJnpus9GtgeWGVm+7v7lBD7QKRC1UfSTzapqT6aWjVtlbvvR3JX6v+us+x0YB93/3uS5ADwVeD36bizSLpOBjgH+LUnHajNAsYBmNnbgakkHRDuDawDPla7IXe/nqG+/vciuWN2HyUEKYLOFKSfNKs+urbq/Tt1pj8IXGNmtwC3pOMOJukCAXf/pZlta2ZvJKnuOTYd/zMz+2M6/3uAicD9Sfc2bMJQp2a1JpB0WwCwqSf96osEp6QgkvAGwxXvJynspwBfMbM9aN41cb11GHClu5/ZLBBLHok6BhhlZo8AYy15NOTn3P2e5l9DpDOqPhJJTK16/231BDPbANjJ3e8EvghsBWwO3E1a/WNmhwIrPOn3vnr8kSSPQoSkE7PjzGy7dNo2ZrZzbSDuPgn4GUl7wr+TdAC3txKCFEFnCtJPNkmPuCv+090rl6VubGZzSA6UPlqz3IbA1WnVkAHfcfc/pQ3RPzSzB0kamivdGn8VuNbM5gO/Av4A4O6PmNmXSZ5ytwFJD52fBeo9YnRfkgbpzwDn15kuEoR6SZW+l159NMndV5Qdi0jZVH0kIiKDdKYgIiKDdKYgIiKDlBRERGSQkoKIiAxSUhARkUFKCiIiMkhJQUREBv1/82TEZWa20sIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=4000, print_every=100, target_score=0.5):\n",
    "    scores1 = []\n",
    "    scores2 = []\n",
    "    mean_scores = []\n",
    "    for i_episode in range(0, n_episodes + 1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        state = env_info.vector_observations\n",
    "        agent1.reset()\n",
    "        agent2.reset()\n",
    "        score1 = 0\n",
    "        score2 = 0\n",
    "        while True:\n",
    "            state1 = np.concatenate([state[0], [1]]).reshape((1,state.shape[1]+1))\n",
    "            state2 = np.concatenate([state[1], [-1]]).reshape((1,state.shape[1]+1))\n",
    "            action1 = agent1.act(state1)\n",
    "            action2 = agent2.act(state2)\n",
    "            env_info = env.step([action1, action2])[brain_name]\n",
    "            next_state = env_info.vector_observations\n",
    "            reward = env_info.rewards\n",
    "            done = env_info.local_done\n",
    "            next_state1 = np.concatenate([next_state[0], [1]])\n",
    "            next_state2 = np.concatenate([next_state[1], [-1]])\n",
    "            agent1.step(state1, action1, np.mean(reward),\n",
    "                        next_state1, done[0])\n",
    "            agent2.step(state2, action2, np.mean(reward),\n",
    "                        next_state2, done[1])\n",
    "            state = next_state\n",
    "            score1 += reward[0]\n",
    "            score2 += reward[1]\n",
    "            if np.all(done):\n",
    "                break\n",
    "\n",
    "        scores1.append(score1)\n",
    "        scores2.append(score2)\n",
    "        mean_scores.append(np.mean([score1, score2]))\n",
    "        mean1 = np.mean(scores1[-100:])\n",
    "        mean2 = np.mean(scores2[-100:])\n",
    "        mean_score = np.mean(mean_scores[-100:])\n",
    "        if i_episode % print_every==0:\n",
    "            print('Episode {}\\t Mean Score for Agent 1: {:.3f}, Mean Score for Agent 2: {:.3f}, Mean Score of both Agents: {:.3f}'.format(\n",
    "            i_episode, mean1, mean2, mean_score))\n",
    "            \n",
    "        if i_episode % 100 == 0:\n",
    "            torch.save(agent1.actor_local.state_dict(), 'checkpoint_actor_1.pth')\n",
    "            torch.save(agent1.critic_local.state_dict(), 'checkpoint_critic_1.pth')\n",
    "            torch.save(agent2.actor_local.state_dict(), 'checkpoint_actor_2.pth')\n",
    "            torch.save(agent2.critic_local.state_dict(), 'checkpoint_critic_2.pth')\n",
    "        if mean_score>1:\n",
    "            break\n",
    "\n",
    "    return mean_scores\n",
    "\n",
    "scores = ddpg()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores, 'r-')\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}