notebooks/ibmseti_intro_to_http_api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IBM SETI Tutorial \n",
    "\n",
    "Full documentation can be found here: https://github.com/ibm-cds-labs/seti_at_ibm/blob/master/setigopublic.md\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search For Data\n",
    "\n",
    "GET `/v1/coordinates/aca` \n",
    "\n",
    "#### Interesting Target\n",
    "\n",
    "Kepler 1229b, found here http://phl.upr.edu/projects/habitable-exoplanets-catalog, is a very intersting target. This candidate has an Earth similarity index (ESI) of 0.73 (Earth = 1.0, Mars = 0.64, Jupiter = 0.12) \n",
    "and is 770 light-years away. It is the 5th highest-ranked planet by ESI. According to the catalog, it's RA/DEC coordinates are 19.832 and 46.997, respectively. \n",
    "\n",
    "Search the `/v1/coordinates/aca` endpoint of the public `https://setigopublic.mybluemix.net` data server to see if we have data for that location. \n",
    "\n",
    "Using the URL parameters, `ramin`, `ramax`, `decmin` and `decmax` for that endpoint, we define a box in the sky around that target and make our query. This ensures we find any data in the direction of the object -- allowing for a small range about its exact location.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "RA=19.832\n",
    "DEC=46.997\n",
    "box = 0.002"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      " \"returned_num_rows\": 1, \n",
      " \"skipped_num_rows\": 0, \n",
      " \"rows\": [\n",
      "  {\n",
      "   \"dec2000deg\": 46.997, \n",
      "   \"number_of_candidates\": 392, \n",
      "   \"ra2000hr\": 19.832\n",
      "  }\n",
      " ], \n",
      " \"total_num_rows\": 1\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "params = {\n",
    "'ramin':RA-box, 'ramax':RA+box, 'decmin':DEC-box, 'decmax':DEC+box\n",
    "}\n",
    "r = requests.get('https://setigopublic.mybluemix.net/v1/coordinates/aca',\n",
    "  params = params)\n",
    "\n",
    "r.raise_for_status()\n",
    "\n",
    "import json\n",
    "print json.dumps(r.json(), indent=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SignalDB Data\n",
    "\n",
    "GET `/v1/aca/meta/{ra}/{dec}`\n",
    "\n",
    "We now know that data exists for this target. There are 392 \"rows\". This means there were 392 candidate signals found in the raw data and entered as a row in the SignalDB. But this doesn't necessarily mean there are 392 data files since the SignalDB may contain multiple rows from the same raw data source. We can obtain the SignalDB data and raw data locations for each of these rows with the `/v1/aca/meta/{ra}/{dec}` endpoint. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ra = r.json()['rows'][0]['ra2000hr']\n",
    "dec = r.json()['rows'][0]['dec2000deg']\n",
    "\n",
    "r = requests.get('https://setigopublic.mybluemix.net/v1/aca/meta/{}/{}'.format(ra,dec))\n",
    "\n",
    "r.raise_for_status()\n",
    "\n",
    "#print json.dumps(r.json(), indent=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### r.json()\n",
    "\n",
    "This is the SignalDB data along with the path to the raw data file (container, objectname)\n",
    "By default only returns 200 rows, but can use `skip` keyword in request params to page through results.\n",
    "You can also increase the number of rows returned by setting `limit=2000` (2000 is the maximum) and paginate through with `skip`.\n",    
    "\n",
    "    {\n",
    "     \"returned_num_rows\": 200, \n",
    "     \"skipped_num_rows\": 0, \n",
    "     \"rows\": [\n",
    "      {\n",
    "       \"inttimes\": 94, \n",
    "       \"pperiods\": 27.36000061, \n",
    "       \"pol\": \"mixed\", \n",
    "       \"tgtid\": 150096, \n",
    "       \"sigreason\": \"PsPwrT\", \n",
    "       \"freqmhz\": 6113.461883333, \n",
    "       \"dec2000deg\": 46.997, \n",
    "       \"container\": \"setiCompAmp\", \n",
    "       \"objectname\": \"2014-05-20/act14944/2014-05-20_13-00-01_UTC.act14944.dx2016.id-0.L.archive-compamp\", \n",
    "       \"ra2000hr\": 19.832, \n",
    "       \"npul\": 3, \n",
    "       \"acttype\": null, \n",
    "       \"power\": 50.652, \n",
    "       \"widhz\": 2.778, \n",
    "       \"catalog\": \"keplerHZ\", \n",
    "       \"snr\": 32.397, \n",
    "       \"uniqueid\": \"kepler8ghz_14944_2016_0_2208930\", \n",
    "       \"beamno\": 2, \n",
    "       \"sigclass\": \"Cand\", \n",
    "       \"sigtyp\": \"Pul\", \n",
    "       \"tscpeldeg\": 78.558, \n",
    "       \"drifthzs\": -1.055, \n",
    "       \"candreason\": \"Confrm\", \n",
    "       \"time\": \"2014-05-20T12:59:55Z\", \n",
    "       \"tscpazdeg\": 307.339\n",
    "      }, \n",
    "\n",
    "      ...\n",
    "\n",
    "      \"total_num_rows\": 392\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SignalDB Analysis\n",
    "\n",
    "There are many things to explore with the SignalDB data. You don't need to access the raw data (in the next step) in order to start visualizing and analyzing SETI data. \n",
    "\n",
    "There is no limit to how much SignalDB data you can consume! So, please feel free to pull down as much as you can. It would be interesting to see a density plot of Candidate signals and \"candreason\"/\"sigreason\" versus position in the sky, the distribution of carrier wave signal frequencies (freqmhz), drift rates, and signal types for each RA/DEC position, etc. There are lots of plots to be generated from these data alone."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Raw Signal Data\n",
    "\n",
    "GET `/v1/data/url/{containter}/{objectname}`\n",
    "\n",
    "The JSON returned above contains the **container** and **objectname** of the ACA file in our Object Store service. We can retrieve the data with an HTTP request.\n",
    "\n",
    "We use the `/v1/data/url/{container}/{objectname}` endpoint to obtain a temporary URL to the data store in Object Store. This temporary URL is valid only for 24 hours. Each Bluemix user will be alloted 20,000 temporary URLs per month. If you need more, however, send us a message and we can grant you more access. \n",
    "\n",
    "### Access Token\n",
    "\n",
    "For this HTTP endpoint only, you must supply a token. This requires you to have a Bluemix or DSX account. In addition, this endpoint has a rate limit of 20,000 requests per month. (However, if you\n",
    "<a href=\"https://github.com/ibm-cds-labs/seti_at_ibm/blob/master/contact_us.md\" target=\"_blank\">make a compelling argument</a> for your analysis, this limit can be increased without a problem.)\n",
    "\n",
    "1. Click on the following link: <a href=\"https://setigopublic.mybluemix.net/token\" target=\"_blank\">https://setigopublic.mybluemix.net/token</a> (opens a new tab). \n",
    "\n",
    "2. Login to DSX/Bluemix. \n",
    "\n",
    "3. Copy the value of the token into the `params` object below. It is passed as a query parameter.\n",
    "\n",
    "4. Use this token for all future requests to the `/v1/data/url/{container}/{objectname}` endpoint. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "params = {'access_token':'123456890abcdefg'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/setiCompAmp/2014-05-20/act14944/2014-05-20_13-00-01_UTC.act14944.dx2016.id-0.L.archive-compamp?temp_url_sig=8ad75d7896c42d83d53f382baa59c3e3a2331e80&temp_url_expires=1474444840\n",
      "file size:  1061928\n"
     ]
    }
   ],
   "source": [
    "#Get Temporary URL for the first object\n",
    "cont = r.json()['rows'][0]['container']\n",
    "objname = r.json()['rows'][0]['objectname']\n",
    "\n",
    "base_url = 'https://setigopublic.mybluemix.net/v1/data/url'  #this HTTP API returns a temporary URL for the container/objectname\n",
    "r = requests.get('/'.join([base_url, cont, objname]), params=params)\n",
    "r.raise_for_status()\n",
    "\n",
    "tmp_url = r.json()['temp_url']\n",
    "print 'URL:', tmp_url\n",
    "\n",
    "r_data = requests.get(tmp_url)\n",
    "r_data.raise_for_status()\n",
    "\n",
    "rawdata = r_data.content\n",
    "print 'file size: ', len(rawdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This data is licensed by the SETI Institute under the Create Commons BY 4.0 license. IBM has been granted non-exclusive permission by the SETI Institute to reproduce, prepare derivative works, and distribute copies of data, which have been provided to IBM, or will be provided in the future, by the SETI Institute under the Creative Commons BY 4.0 license. https://github.com/ibmjstart/SETI/blob/master/setigopublic.md#data-license\n"
     ]
    }
   ],
   "source": [
    "print r.json()['license_notification']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}