notebooks/Defining-a-Heuristic-Policy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a Heuristic Policy\n",
    "In this section, we will walk through how to\n",
    "define a POMDP policy of your own. For more details on POMDPs and their\n",
    "policies, please consult Chapter 6 of the DMU textbook [1] We will define a\n",
    "simple greedy policy that takes the action that maximises the expected single-\n",
    "step reward, given the current belief state. We will also compare it against a\n",
    "policy that chooses actions at random. Please look at the documentation of\n",
    "[POMDPPolicies.jl](https://github.com/JuliaPOMDP/POMDPPolicies.jl) for more on\n",
    "the code structure of a policy object that is compatible with POMDPs.jl. We will\n",
    "use the explicit TigerPOMDP model - see\n",
    "[this](http://localhost:8888/notebooks/POMDPExamples/notebooks/Defining-a-POMDP-\n",
    "with-the-Explicit-Interface.ipynb) notebook for more on that.\n",
    "\n",
    "[1] Kochenderfer,\n",
    "Mykel J. Decision Making Under Uncertainty: Theory and Application. MIT Press,\n",
    "2015"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "1"
    }
   },
   "outputs": [],
   "source": [
    "using POMDPs\n",
    "using POMDPPolicies # For defining a policy\n",
    "using POMDPModels # For the TigerPOMDP Model\n",
    "using BeliefUpdaters # To use DiscreteUpdater\n",
    "using POMDPModelTools # For weighted_iterator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define a GreedyPolicy type that only requires the POMDP instance and the\n",
    "set of valid actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "2"
    }
   },
   "outputs": [],
   "source": [
    "struct GreedyPolicy{P<:POMDP} <: Policy\n",
    "    pomdp::P\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overriding the POMDPs.jl action function\n",
    "Now we will define a new method for\n",
    "the `action` function, which specifies the behavior of our policy. It requires\n",
    "the belief state to be represented as a `DiscreteBelief`, i.e. a Probability\n",
    "Mass Function over individual states. It computes the expected single-step\n",
    "reward for each action, given the current belief state, and chooses the maximum\n",
    "one. This is sometimes called a \"greedy\" or \"myopic\" policy. Note that we must\n",
    "use `POMDPs.action` to add a method to the `action` function of `POMDPs.jl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "4"
    }
   },
   "outputs": [],
   "source": [
    "function POMDPs.action(p::GreedyPolicy, b)\n",
    "    max_value = -Inf\n",
    "    as = actions(p.pomdp)\n",
    "    best_a = first(as)\n",
    "    for a in as\n",
    "        action_val = 0.0\n",
    "        for (state, bel) in weighted_iterator(b)\n",
    "            action_val += bel*reward(p.pomdp, state, a)\n",
    "        end\n",
    "        \n",
    "        if action_val > max_value\n",
    "            best_a = a\n",
    "            max_value = action_val\n",
    "        end\n",
    "    end\n",
    "    \n",
    "    return best_a\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmarking the Policy\n",
    "Note that unlike the other examples on using\n",
    "*solvers*, here we have already specified a *policy*. Therefore, we can just\n",
    "evaluate the policy on a problem, in this case, `TigerPOMDP`(defined in\n",
    "[POMDPModels](https://github.com/JuliaPOMDP/POMDPModels.jl)). We define the\n",
    "POMDP problem and create the policies based on it.\n",
    "\n",
    "Since we only care about the\n",
    "discounted reward, we can use the rollout simulator defined in\n",
    "[POMDPSimulators](https://github.com/JuliaPOMDP/POMDPSimulators.jl). Checkout\n",
    "this\n",
    "[notebook](https://github.com/JuliaPOMDP/POMDPExamples.jl/blob/master/notebooks/Running-Simulations.ipynb) for ways to use the other simulators as well.\n",
    "Finally, we can compare the expected discounted rewards and see how the greedy\n",
    "policy does quite better than random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TigerPOMDP(-1.0, -100.0, 10.0, 0.85, 0.95)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pomdp = TigerPOMDP()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "6"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GreedyPolicy{TigerPOMDP}(TigerPOMDP(-1.0, -100.0, 10.0, 0.85, 0.95))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greedy_pol = GreedyPolicy(pomdp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    }
   },
   "outputs": [],
   "source": [
    "# Define a random policy as a benchmark\n",
    "rand_policy = RandomPolicy(pomdp);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "8"
    }
   },
   "outputs": [],
   "source": [
    "using POMDPSimulators\n",
    "rollout_sim = RolloutSimulator(max_steps=10);\n",
    "greedy_reward = simulate(rollout_sim, pomdp, greedy_pol, DiscreteUpdater(pomdp));\n",
    "rand_reward = simulate(rollout_sim, pomdp, rand_policy, DiscreteUpdater(pomdp));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "9"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "greedy_reward = 7.867051041738277\n",
      "rand_reward = -353.43638007562487\n"
     ]
    }
   ],
   "source": [
    "@show greedy_reward;\n",
    "@show rand_reward;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.2.0",
   "language": "julia",
   "name": "julia-1.2"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}