em-v3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [NSF's OIR Lab](https://nationalastro.org/) Astro Data Archive: How to make an exposure map for a survey\n",
    "\n",
    "\n",
    "Knut Olsen, [Community Science and Data Center](https://nationalastro.org/programs/csdc/) (slightly modified by Steve Pothier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table Of Contents:\n",
    "* [Goals & notebook summary](#goals)\n",
    "* [Imports and setup](#init)\n",
    "* [Details](#details)\n",
    "* [Where do we look for the fields that we need?](#hdu_search)\n",
    "* [Doing the search](#search)\n",
    "* [Combining results](#combine)\n",
    "* [Computing $t_{\\rm eff}$](#teff)\n",
    "* [Making the depth map](#depth)\n",
    "* [Resources and references](#doc)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"goals\"></a>\n",
    "# Goals\n",
    "Use the Advanced Search API to make an exposure map for a survey.  We'll use the Dark Energy Survey DES) as an example.\n",
    "\n",
    "# Notebook Summary\n",
    "We show you how to use the API to search the Archive for relevant information from individual r-band HDUs as well as the primary HDUs.  We'll combine information on FWHM, transparency, and sky background to compute maps of effective exposure time, following the procedure adopted by DES."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"init\"></a>\n",
    "\n",
    "# Imports and setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using server on https://astroarchive.noirlab.edu\n",
      "About NSF's OIR Lab Astro Data Archive: https://astroarchive.noirlab.edu/about/\n"
     ]
    }
   ],
   "source": [
    "# UTILS are settings for HOST etc. that you won't need to use the API\n",
    "# but are used here to make this notebook more portable.\n",
    "%run utils.ipynb\n",
    "tic() # start tracking time of Notebook load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Python version: 3.8.5 (default, Jul 28 2020, 12:59:40) \n",
      "[GCC 9.3.0]\n",
      "\n",
      "This Jupyter Notebook uses (some of) these packages:\n",
      "\n",
      "#pandas==0.23.4\n",
      "pandas==1.1.4\n",
      "#matplotlib==3.0.2\n",
      "matplotlib==3.3.3\n",
      "#ipython==7.2.0\n",
      "jupyter==1.0.0\n",
      "#requests==2.22.0\n",
      "requests==2.25.0\n",
      "#astropy==3.2.1\n",
      "astropy==4.1\n",
      "#pytest==5.3.5\n",
      "pytest==6.1.2\n",
      "healpy==1.14.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import copy\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import math\n",
    "import requests\n",
    "import json\n",
    "import healpy as hp\n",
    "from pprint import pprint as pp  # pretty print\n",
    "\n",
    "print(f'Using Python version: {sys.version}\\n')\n",
    "with open (\"requirements.txt\", \"r\") as f:\n",
    "    data=f.readlines()\n",
    "print(f'This Jupyter Notebook uses (some of) these packages:\\n\\n{\"\".join(data)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using server on https://astroarchive.noirlab.edu\n",
      "About NSF's OIR Lab Astro Data Archive: https://astroarchive.noirlab.edu/about/\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://astroarchive.noirlab.edu/api/adv_search'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# UTILS are settings for HOST etc. that you won't need to use the API\n",
    "# but are used here to make this notebook more portable.\n",
    "%run utils.ipynb\n",
    "tic() # start tracking time of Notebook load\n",
    "\n",
    "adsurl = f'{natroot}/api/adv_search'\n",
    "adsurl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Details <a id=\"details\"/>\n",
    "\n",
    "\n",
    "The effective exposure time ratio is defined by Nielsen et al. (2016) in their equation 2:\n",
    "\n",
    "$\\tau$ = $\\eta^2 ({\\rm FWHM}/0.9\")^{-2}(b/b_{\\rm dark})^{-1}$\n",
    "\n",
    "where $\\eta$ is the atmospheric transparency, FWHM is the full width of the PSF, $b$ is the sky brightness, and $b_{\\rm dark}$ is the fiducial dark sky background. The effective exposure time, which accounts for the factors that influence the depth of an exposure, is then:\n",
    "\n",
    "$t_{\\rm eff} = \\tau \\times t_{\\rm exp}$\n",
    "\n",
    "In order to make a depth map of a survey, we will use the Advanced Search API to get the terms of this equation for individual HDUs.  We will make a Healpix map that counts $t_{\\rm eff}$ in the healpixels on the sky overlapping with those HDUs.  Our goal is to be able to adjust e.g. density maps of sources on the sky for differences in depth across a survey.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where do we look for the fields that we need? <a id=\"hdu_search\" />  \n",
    "Some of the fields that we need, in particular the locations of the HDU corners, the FWHM of the PSF, and the sky level, are written into individual HDU headers by the DECam Community Pipeline.  So, we'll look at the fields in the aux_hdu_fields service for decam instcal-processed frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Field': 'ARAWGAIN', 'Type': 'str', 'Desc': '[e/adu] Average raw gain'},\n",
       " {'Field': 'AVSIG', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'AVSKY', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'BIASFIL', 'Type': 'str', 'Desc': 'Bias'},\n",
       " {'Field': 'BPM', 'Type': 'str', 'Desc': 'Bad pixel mask'},\n",
       " {'Field': 'BPMFIL', 'Type': 'str', 'Desc': 'BPM file used to build mask'},\n",
       " {'Field': 'CCDNUM', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CD1_1', 'Type': 'str', 'Desc': 'Degrees / Pixel'},\n",
       " {'Field': 'CD1_2', 'Type': 'str', 'Desc': 'Degrees / Pixel'},\n",
       " {'Field': 'CD2_1', 'Type': 'str', 'Desc': 'Degrees / Pixel'},\n",
       " {'Field': 'CD2_2', 'Type': 'str', 'Desc': 'Degrees / Pixel'},\n",
       " {'Field': 'CENDEC1', 'Type': 'str', 'Desc': 'Mosaic CCD center'},\n",
       " {'Field': 'CENRA1', 'Type': 'str', 'Desc': 'Mosaic CCD center'},\n",
       " {'Field': 'COR1DEC1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'COR1RA1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'COR2DEC1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'COR2RA1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'COR3DEC1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'COR3RA1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'COR4DEC1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'COR4RA1', 'Type': 'str', 'Desc': 'Mosaic CCD corner'},\n",
       " {'Field': 'CRPIX1', 'Type': 'str', 'Desc': 'Reference Pixel in X'},\n",
       " {'Field': 'CRPIX2', 'Type': 'str', 'Desc': 'Reference Pixel in Y'},\n",
       " {'Field': 'CRVAL1',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'R.A. (degrees) of reference pixel'},\n",
       " {'Field': 'CRVAL2', 'Type': 'str', 'Desc': 'Declination of reference pixel'},\n",
       " {'Field': 'CTYPE1', 'Type': 'str', 'Desc': 'Coordinate Type'},\n",
       " {'Field': 'CTYPE2', 'Type': 'str', 'Desc': 'Coordinate Type'},\n",
       " {'Field': 'CUNIT1', 'Type': 'str', 'Desc': 'Axis unit'},\n",
       " {'Field': 'CUNIT2', 'Type': 'str', 'Desc': 'Axis unit'},\n",
       " {'Field': 'DATE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC', 'Type': 'str', 'Desc': 'declination (telescope)'},\n",
       " {'Field': 'DEC1', 'Type': 'str', 'Desc': 'Mosaic CCD center'},\n",
       " {'Field': 'DEC17A_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC18B_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC19A_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC19A_P', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC19A_T', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC19B_A', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC19B_C', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC19B_J', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC20A_A', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DECALS_D', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DECC1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DECC2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DECC3', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DECC4', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DEC_CENT', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DECCMAX', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DECCMIN', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DES15B_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DES16B_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DES17B_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DES18A_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESBFC', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESBIAS', 'Type': 'str', 'Desc': 'bias subtracted'},\n",
       " {'Field': 'DESBLEED', 'Type': 'str', 'Desc': 'bleed trail masking'},\n",
       " {'Field': 'DESBPM', 'Type': 'str', 'Desc': 'Construct mask from BPM'},\n",
       " {'Field': 'DESCRMSK',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Run Maskcosmics. Unable to fit sky'},\n",
       " {'Field': 'DESDCXTK',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'DECam image conversion and crosstalk corr'},\n",
       " {'Field': 'DES_EXT', 'Type': 'str', 'Desc': 'Image Extension'},\n",
       " {'Field': 'DESFIXC', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESFLAT', 'Type': 'str', 'Desc': 'flat fielded'},\n",
       " {'Field': 'DESFNAME', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESGAINC', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESILLUM', 'Type': 'str', 'Desc': 'Illumination Corrected'},\n",
       " {'Field': 'DESIMMSK', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESLINC', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESNCRAY', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESNSTRK', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESOSCN', 'Type': 'str', 'Desc': 'overscan corrected'},\n",
       " {'Field': 'DESSAT', 'Type': 'str', 'Desc': 'saturated pixels flagged'},\n",
       " {'Field': 'DESSKYSB', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESSTAR', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DETECTOR',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Detector (CCD type, photon counter, etc.)'},\n",
       " {'Field': 'DETPOS', 'Type': 'str', 'Desc': 'detector position ID'},\n",
       " {'Field': 'DETSEC', 'Type': 'str', 'Desc': 'Detector data section'},\n",
       " {'Field': 'ELLIPTIC', 'Type': 'str', 'Desc': 'Median ELLIPTICITY'},\n",
       " {'Field': 'EXTNAME', 'Type': 'str', 'Desc': 'Extension name'},\n",
       " {'Field': 'FIXCFIL', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FIXPIX', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FLATFIL', 'Type': 'str', 'Desc': 'Dome flat reference file'},\n",
       " {'Field': 'FLATMEDA', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FLATMEDB', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FPA', 'Type': 'str', 'Desc': 'focal plan array'},\n",
       " {'Field': 'FRGSCALE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FRGSCNEW', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FRINGE', 'Type': 'str', 'Desc': 'Fringe template'},\n",
       " {'Field': 'FWHM', 'Type': 'str', 'Desc': 'Median FWHM in pixels'},\n",
       " {'Field': 'FWHMP1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FZALGOR', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FZDTHRSD', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FZQMETHD', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FZQVALUE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'GAINA', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'GAINB', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'GCOUNT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'one data group (required keyword)'},\n",
       " {'Field': 'ILLMASK', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'ILLUMCOR', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'ILLUMFIL', 'Type': 'str', 'Desc': 'Illumination image'},\n",
       " {'Field': 'INHERIT', 'Type': 'str', 'Desc': 'Inherits global header'},\n",
       " {'Field': 'IRAF-TLM', 'Type': 'str', 'Desc': 'Time of last modification'},\n",
       " {'Field': 'LTM1_1', 'Type': 'str', 'Desc': 'CCD to image transformation'},\n",
       " {'Field': 'LTM1_2',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Detector to image transformation'},\n",
       " {'Field': 'LTM2_1',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Detector to image transformation'},\n",
       " {'Field': 'LTM2_2', 'Type': 'str', 'Desc': 'CCD to image transformation'},\n",
       " {'Field': 'LTV1', 'Type': 'str', 'Desc': 'CCD to image transformation'},\n",
       " {'Field': 'LTV2', 'Type': 'str', 'Desc': 'CCD to image transformation'},\n",
       " {'Field': 'MAGZERO1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'MAGZUNC1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'NAXIS1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'NAXIS2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'NBLEED', 'Type': 'str', 'Desc': 'Number of bleed trail pixels.'},\n",
       " {'Field': 'NSATPIX', 'Type': 'str', 'Desc': 'Number of saturated pixels'},\n",
       " {'Field': 'OBJECT', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'ORIGIN', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PCOUNT', 'Type': 'str', 'Desc': 'size of special data area'},\n",
       " {'Field': 'PV1_0', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_10', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_3', 'Type': 'str', 'Desc': 'PV distortion coefficient'},\n",
       " {'Field': 'PV1_4', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_5', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_6', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_7', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_8', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV1_9', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_0', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_1', 'Type': 'str', 'Desc': 'Projection parameter 1'},\n",
       " {'Field': 'PV2_10', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_3', 'Type': 'str', 'Desc': 'PV distortion coefficient'},\n",
       " {'Field': 'PV2_4', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_5', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_6', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_7', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_8', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PV2_9', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RA', 'Type': 'str', 'Desc': 'right ascension (telescope)'},\n",
       " {'Field': 'RA1', 'Type': 'str', 'Desc': 'Mosaic CCD center'},\n",
       " {'Field': 'RAC1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RAC2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RAC3', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RAC4', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RA_CENT', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RACMAX', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RACMIN', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RDNOISEA',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[electrons] Read noise for amp A'},\n",
       " {'Field': 'RDNOISEB',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[electrons] Read noise for amp B'},\n",
       " {'Field': 'REQ19A_2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SATURATA', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SATURATB', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SATURATE', 'Type': 'str', 'Desc': 'Maximum good data value (ADU)'},\n",
       " {'Field': 'SKYBRITE', 'Type': 'str', 'Desc': 'Sky Brightness (ADU)'},\n",
       " {'Field': 'SKYIM', 'Type': 'str', 'Desc': 'Sky map'},\n",
       " {'Field': 'SKYSBFIL', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYSIGMA', 'Type': 'str', 'Desc': 'Sky Sigma (ADU)'},\n",
       " {'Field': 'SKYVARA', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYVARB', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SLOT00',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'dhe board: <type> <serial> <firmware>'},\n",
       " {'Field': 'SLOT01', 'Type': 'str', 'Desc': 'Monsoon module'},\n",
       " {'Field': 'SLOT02',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'dhe board: <type> <serial> <firmware>'},\n",
       " {'Field': 'SLOT03', 'Type': 'str', 'Desc': 'Monsoon module'},\n",
       " {'Field': 'SLOT04',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'dhe board: <type> <serial> <firmware>'},\n",
       " {'Field': 'SLOT05', 'Type': 'str', 'Desc': 'Monsoon module'},\n",
       " {'Field': 'STARFIL', 'Type': 'str', 'Desc': 'Star flat correction file'},\n",
       " {'Field': 'STARMASK', 'Type': 'str', 'Desc': 'created bright star mask'},\n",
       " {'Field': 'WAT0_001', 'Type': 'str', 'Desc': 'Coordinate system'},\n",
       " {'Field': 'WAT1_001', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'WAT2_001', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'WCSAXES', 'Type': 'str', 'Desc': 'Number of WCS axes'},\n",
       " {'Field': 'WCSDIM', 'Type': 'str', 'Desc': 'WCS dimensionality'},\n",
       " {'Field': 'WTMAP', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'XTENSION', 'Type': 'str', 'Desc': 'binary table extension'},\n",
       " {'Field': 'ZDITHER0',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'dithering offset when quantizing floats'}]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get(f'{adsurl}/aux_hdu_fields/decam/instcal/').json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going through all of the fields requires some digging, but through experimentation we decide that we need these fields:\n",
    "\n",
    "* COR[1-4]RA1\n",
    "* COR[1-4]DEC1\n",
    "* FWHM\n",
    "* AVSKY\n",
    "\n",
    "However, we're still missing information on transparency.  Keywords related to transparency (e.g. AIRMASS) are generally written into the primary HDU for the full file, so we'll look at the AUX fields available in the full file search service:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Field': 'AIRMASS', 'Type': 'str', 'Desc': 'airmass'},\n",
       " {'Field': 'AOS', 'Type': 'str', 'Desc': 'AOS data available if true'},\n",
       " {'Field': 'ASTIG1', 'Type': 'str', 'Desc': '4MAPS correction 1'},\n",
       " {'Field': 'ASTIG2', 'Type': 'str', 'Desc': '4MAPS correction 2'},\n",
       " {'Field': 'ATTNUM', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'AVSIG', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'AVSKY', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'AZ', 'Type': 'str', 'Desc': 'Telescope azimuth'},\n",
       " {'Field': 'BAND', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'BCAM', 'Type': 'str', 'Desc': 'BCAM data available if true'},\n",
       " {'Field': 'BCAMAX',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] BCAM hexapod rot. about x-axis'},\n",
       " {'Field': 'BCAMAY',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] BCAM hexapod rot. about y-axis'},\n",
       " {'Field': 'BCAMAZ',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] BCAM hexapod rot. about z-axis'},\n",
       " {'Field': 'BCAMDX', 'Type': 'str', 'Desc': '[micron] BCAM hexapod x-offset'},\n",
       " {'Field': 'BCAMDY', 'Type': 'str', 'Desc': '[micron] BCAM hexapod y-offset'},\n",
       " {'Field': 'BFCFIL', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'BUNIT', 'Type': 'str', 'Desc': 'photoelectrons per second'},\n",
       " {'Field': 'CAMSHUT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Camera shutter at exposure start'},\n",
       " {'Field': 'CAMSYM', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CCDBIN1', 'Type': 'str', 'Desc': 'Binning factor along axis 1'},\n",
       " {'Field': 'CCDBIN2', 'Type': 'str', 'Desc': 'Binning factor along axis 2'},\n",
       " {'Field': 'CCDSEC', 'Type': 'str', 'Desc': 'ccd section'},\n",
       " {'Field': 'CENTDEC', 'Type': 'str', 'Desc': '[deg] DEC center of exposure'},\n",
       " {'Field': 'CENTRA', 'Type': 'str', 'Desc': '[deg] RA center of exposure'},\n",
       " {'Field': 'CONSTVER', 'Type': 'str', 'Desc': 'SISPI constants version'},\n",
       " {'Field': 'CORN1DEC', 'Type': 'str', 'Desc': '[deg] DEC corner of exposure'},\n",
       " {'Field': 'CORN1RA', 'Type': 'str', 'Desc': '[deg] RA corner of exposure'},\n",
       " {'Field': 'CORN2DEC', 'Type': 'str', 'Desc': '[deg] DEC corner of exposure'},\n",
       " {'Field': 'CORN2RA', 'Type': 'str', 'Desc': '[deg] RA corner of exposure'},\n",
       " {'Field': 'CORN3DEC', 'Type': 'str', 'Desc': '[deg] DEC corner of exposure'},\n",
       " {'Field': 'CORN3RA', 'Type': 'str', 'Desc': '[deg] RA corner of exposure'},\n",
       " {'Field': 'CORN4DEC', 'Type': 'str', 'Desc': '[deg] DEC corner of exposure'},\n",
       " {'Field': 'CORN4RA', 'Type': 'str', 'Desc': '[deg] RA corner of exposure'},\n",
       " {'Field': 'CORNDEC1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CORNDEC2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CORNDEC3', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CORNDEC4', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CORNRA1', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CORNRA2', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CORNRA3', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CORNRA4', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'CROSSRA0', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DARKTIME', 'Type': 'str', 'Desc': 'Total elapsed time in secs'},\n",
       " {'Field': 'DATASEC', 'Type': 'str', 'Desc': 'data section'},\n",
       " {'Field': 'DATE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DATE-OBS', 'Type': 'str', 'Desc': 'UTC epoch'},\n",
       " {'Field': 'DEC', 'Type': 'str', 'Desc': 'declination (telescope)'},\n",
       " {'Field': 'DES_EXT', 'Type': 'str', 'Desc': 'Image Extension'},\n",
       " {'Field': 'DESNSTRK', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DESREL', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DETSIZE', 'Type': 'str', 'Desc': 'Detector size'},\n",
       " {'Field': 'DHEFIRM', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DHEINF', 'Type': 'str', 'Desc': 'controller info'},\n",
       " {'Field': 'DIMM2SEE', 'Type': 'str', 'Desc': '[arcsec] DIMM2 Seeing'},\n",
       " {'Field': 'DIMMSEE', 'Type': 'str', 'Desc': 'Pachon DIMM seeing'},\n",
       " {'Field': 'DODX',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[um] X-decenter from donut analysis'},\n",
       " {'Field': 'DODY',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[um] Y-decenter from donut analysis'},\n",
       " {'Field': 'DODZ', 'Type': 'str', 'Desc': '[um] Delta-Z from donut analysis'},\n",
       " {'Field': 'DOMEAZ', 'Type': 'str', 'Desc': '[deg] Dome position'},\n",
       " {'Field': 'DOMEFLOR',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Dome floor temperature'},\n",
       " {'Field': 'DOMEHIGH', 'Type': 'str', 'Desc': '[deg C] High dome temperature'},\n",
       " {'Field': 'DOMELOW', 'Type': 'str', 'Desc': '[deg C] Low dome temperature'},\n",
       " {'Field': 'DONUTFN1', 'Type': 'str', 'Desc': 'Mean Wavefront'},\n",
       " {'Field': 'DONUTFN2', 'Type': 'str', 'Desc': 'Mean Wavefron'},\n",
       " {'Field': 'DONUTFN3', 'Type': 'str', 'Desc': 'Mean Wavefront'},\n",
       " {'Field': 'DONUTFN4', 'Type': 'str', 'Desc': 'Mean Wavefront'},\n",
       " {'Field': 'DONUTFS1', 'Type': 'str', 'Desc': 'Mean Wavefront'},\n",
       " {'Field': 'DONUTFS2', 'Type': 'str', 'Desc': 'Mean Wavefront'},\n",
       " {'Field': 'DONUTFS3', 'Type': 'str', 'Desc': 'Mean Wavefront'},\n",
       " {'Field': 'DONUTFS4', 'Type': 'str', 'Desc': 'Mean Wavefron'},\n",
       " {'Field': 'DOXT',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] X-theta from donut analysis'},\n",
       " {'Field': 'DOYT',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Y-theta from donut analysis'},\n",
       " {'Field': 'DTACCOUN', 'Type': 'str', 'Desc': 'observing account name'},\n",
       " {'Field': 'DTACQNAM',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'file name supplied at telescope'},\n",
       " {'Field': 'DTACQUIS',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'host name of data acquisition computer'},\n",
       " {'Field': 'DTCALDAT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'calendar date from observing schedule'},\n",
       " {'Field': 'DTCOPYRI', 'Type': 'str', 'Desc': 'copyright holder of data'},\n",
       " {'Field': 'DTINSTRU', 'Type': 'str', 'Desc': 'instrument identifier'},\n",
       " {'Field': 'DTNSANAM', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DTOBSERV', 'Type': 'str', 'Desc': 'scheduling institution'},\n",
       " {'Field': 'DTPI', 'Type': 'str', 'Desc': 'Principal Investigator'},\n",
       " {'Field': 'DTPIAFFL', 'Type': 'str', 'Desc': 'PI affiliation'},\n",
       " {'Field': 'DTPROPID',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Proposal ID validated by Archive'},\n",
       " {'Field': 'DTQUEUE', 'Type': 'str', 'Desc': 'DTS queue (1513)'},\n",
       " {'Field': 'DT_RTNAM', 'Type': 'str', 'Desc': 'NSA root name'},\n",
       " {'Field': 'DTSITE',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Telescope / observatory locale from which original raw data was taken'},\n",
       " {'Field': 'DTSTATUS', 'Type': 'str', 'Desc': 'data transport status'},\n",
       " {'Field': 'DTTELESC', 'Type': 'str', 'Desc': 'telescope identifier'},\n",
       " {'Field': 'DTTITLE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'DTUTC', 'Type': 'str', 'Desc': 'post exposure UTC epoch from DTS'},\n",
       " {'Field': 'EQUINOX', 'Type': 'str', 'Desc': 'Equinox of RA and DEC'},\n",
       " {'Field': 'ERRORS', 'Type': 'str', 'Desc': 'SISPI readout errors'},\n",
       " {'Field': 'EUPSPROD', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'EUPSVER', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'EXCLUDED',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'DECam components not used for this frame'},\n",
       " {'Field': 'EXPDUR', 'Type': 'str', 'Desc': '[s] Exposure duration'},\n",
       " {'Field': 'EXPNUM', 'Type': 'str', 'Desc': 'Exposure sequence number'},\n",
       " {'Field': 'EXPREQ', 'Type': 'str', 'Desc': '[sec] Exposure time requested'},\n",
       " {'Field': 'EXPTIME', 'Type': 'str', 'Desc': 'Exposure time in secs'},\n",
       " {'Field': 'FADX', 'Type': 'str', 'Desc': '[um] FA Delta X.'},\n",
       " {'Field': 'FADY', 'Type': 'str', 'Desc': '[um] FA Delta Y.'},\n",
       " {'Field': 'FADZ', 'Type': 'str', 'Desc': '[um] FA Delta focus.'},\n",
       " {'Field': 'FAXT', 'Type': 'str', 'Desc': '[arcsec] FA Delta X-theta.'},\n",
       " {'Field': 'FAYT', 'Type': 'str', 'Desc': '[arcsec] FA Delta Y-theta.'},\n",
       " {'Field': 'FILENAME', 'Type': 'str', 'Desc': 'fits file name'},\n",
       " {'Field': 'FILTER', 'Type': 'str', 'Desc': 'Lower filter position'},\n",
       " {'Field': 'FILTPOS', 'Type': 'str', 'Desc': 'Instrument filter position'},\n",
       " {'Field': 'FWHM', 'Type': 'str', 'Desc': 'Median FWHM in pixels'},\n",
       " {'Field': 'FZALGOR', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FZDTHRSD', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FZQMETHD', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'FZQVALUE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'G-CCDNUM',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Number of guide CCDs that remained active'},\n",
       " {'Field': 'G-FEEDBK', 'Type': 'str', 'Desc': '[%] Guider feedback'},\n",
       " {'Field': 'G-FLXVAR',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Guider mean guide star flux variances'},\n",
       " {'Field': 'G-LATENC',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[s] Guider avg. latency between exposures'},\n",
       " {'Field': 'G-MAXX',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Guider x-axis maximum offset'},\n",
       " {'Field': 'G-MAXY',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Guider y-axis maximum offset'},\n",
       " {'Field': 'G-MEANX',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Guider x-axis mean offset'},\n",
       " {'Field': 'G-MEANX2',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec2] Guider (x) 2nd moment mean offset'},\n",
       " {'Field': 'G-MEANXY',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec2] Guider (xy) 2nd moment mean offset'},\n",
       " {'Field': 'G-MEANY',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Guider y-axis mean offset'},\n",
       " {'Field': 'G-MEANY2',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec2] Guider (y) 2nd moment mean offset'},\n",
       " {'Field': 'G-MODE', 'Type': 'str', 'Desc': 'Guider operation mode'},\n",
       " {'Field': 'G-SEEING',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Guider average seeing'},\n",
       " {'Field': 'GSKYHOT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'RASICAM global sky fraction above threshold'},\n",
       " {'Field': 'GSKYPHOT', 'Type': 'str', 'Desc': 'RASICAM global sky clear flag'},\n",
       " {'Field': 'GSKYVAR',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'RASICAM global sky standard deviation'},\n",
       " {'Field': 'G-TRANSP',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Guider average sky transparency'},\n",
       " {'Field': 'GUIDER',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Guider (0-absent,1-ok,2-lost star,3-los'},\n",
       " {'Field': 'HA', 'Type': 'str', 'Desc': 'hour angle (H:M:S)'},\n",
       " {'Field': 'HDRVER', 'Type': 'str', 'Desc': 'DECam fits header version'},\n",
       " {'Field': 'HEX', 'Type': 'str', 'Desc': 'DES Hex number'},\n",
       " {'Field': 'HUMIDITY', 'Type': 'str', 'Desc': 'humidity'},\n",
       " {'Field': 'INSTANCE', 'Type': 'str', 'Desc': 'SISPI instance name'},\n",
       " {'Field': 'INSTRUME', 'Type': 'str', 'Desc': '1.5m Cassegrain spectrograph'},\n",
       " {'Field': 'IRAF-TLM', 'Type': 'str', 'Desc': 'Time of last modification'},\n",
       " {'Field': 'LINCFIL', 'Type': 'str', 'Desc': 'Nonlinearity correction file'},\n",
       " {'Field': 'LSKYHOT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'RASICAM local sky fraction above threshold'},\n",
       " {'Field': 'LSKYPHOT', 'Type': 'str', 'Desc': 'RASICAM local sky clear flag'},\n",
       " {'Field': 'LSKYPOW',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'RASICAM local sky normalized power'},\n",
       " {'Field': 'LSKYVAR',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'RASICAM local sky standard deviation'},\n",
       " {'Field': 'LUTVER', 'Type': 'str', 'Desc': 'Hexapod Lookup Table version'},\n",
       " {'Field': 'LWTRTEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Lower truss temperature'},\n",
       " {'Field': 'MAGZERO', 'Type': 'str', 'Desc': '[mag] magnitude zero point'},\n",
       " {'Field': 'MAGZPT',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[mag] Magnitude zero point per sec'},\n",
       " {'Field': 'MAGZUNC', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'MAIRTEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Mirror temperature above surface'},\n",
       " {'Field': 'MANIFEST', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'MASS2', 'Type': 'str', 'Desc': 'MASS(2) FSEE'},\n",
       " {'Field': 'MJD-END', 'Type': 'str', 'Desc': 'Exposure end'},\n",
       " {'Field': 'MJD-OBS',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'MJD=JD-2400000.5 at start of obs'},\n",
       " {'Field': 'MSURTEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Mirror surface average temperature'},\n",
       " {'Field': 'MULTIEXP',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Frame contains multiple exposures if true'},\n",
       " {'Field': 'NDONUTS',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'AOS number of donuts analyzed for this CCD'},\n",
       " {'Field': 'NEXTEND', 'Type': 'str', 'Desc': 'Number of extensions'},\n",
       " {'Field': 'NITE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'NOTE', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'NPHTMTCH', 'Type': 'str', 'Desc': 'Number of phot. stars matched'},\n",
       " {'Field': 'OBJECT', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'OBS-ELEV', 'Type': 'str', 'Desc': '[km] Observatory elevation'},\n",
       " {'Field': 'OBSERVAT', 'Type': 'str', 'Desc': 'Origin of data'},\n",
       " {'Field': 'OBSERVER', 'Type': 'str', 'Desc': 'Observers'},\n",
       " {'Field': 'OBSID', 'Type': 'str', 'Desc': 'sequence ID'},\n",
       " {'Field': 'OBS-LAT', 'Type': 'str', 'Desc': '[deg] Observatory latitude'},\n",
       " {'Field': 'OBS-LONG', 'Type': 'str', 'Desc': '[deg] Observatory longitude'},\n",
       " {'Field': 'OBSTYPE',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Type of target observed (e.g. object, dark, dome flat...)'},\n",
       " {'Field': 'ODATEOBS', 'Type': 'str', 'Desc': 'previous DATE-OBS'},\n",
       " {'Field': 'OPENSHUT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Time when shutter opened (UTC)'},\n",
       " {'Field': 'ORIGIN', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'OUTTEMP', 'Type': 'str', 'Desc': 'outside temperature'},\n",
       " {'Field': 'PHOTFLAG',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Night Photometric (1) or not (0)'},\n",
       " {'Field': 'PHOTREF',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Phototometric reference catalog'},\n",
       " {'Field': 'PIPELINE', 'Type': 'str', 'Desc': 'Name of calibration pipeline'},\n",
       " {'Field': 'PIXSCAL1',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Pixel scale for axis 1 (arcsec/pixel)'},\n",
       " {'Field': 'PIXSCAL2',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Pixel scale for axis 2 (arcsec/pixel)'},\n",
       " {'Field': 'PLARVER', 'Type': 'str', 'Desc': 'PL NSA archive version'},\n",
       " {'Field': 'PLDNAME', 'Type': 'str', 'Desc': 'PL Dataset'},\n",
       " {'Field': 'PLDSID', 'Type': 'str', 'Desc': 'PL NSA name'},\n",
       " {'Field': 'PLFNAME', 'Type': 'str', 'Desc': 'PL Filename'},\n",
       " {'Field': 'PLPROCID', 'Type': 'str', 'Desc': 'PL Processing ID'},\n",
       " {'Field': 'PLQNAME', 'Type': 'str', 'Desc': 'PL Dataset'},\n",
       " {'Field': 'PLQUEUE', 'Type': 'str', 'Desc': 'PL Queue'},\n",
       " {'Field': 'PLVER', 'Type': 'str', 'Desc': 'Pipeline version'},\n",
       " {'Field': 'PME-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Mirror east edge temperature'},\n",
       " {'Field': 'PMN-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Mirror edge surface temperature'},\n",
       " {'Field': 'PMOSTEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Primary mirror top surface temperature'},\n",
       " {'Field': 'PMS-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Mirror south edge temperature'},\n",
       " {'Field': 'PMW-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Mirror west edge temperature'},\n",
       " {'Field': 'PRESSURE', 'Type': 'str', 'Desc': 'dewar pressure (Torr) (OK)'},\n",
       " {'Field': 'PROCTYPE',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Processing Type ( Raw | Instcal | Resampled | Stack | MasterCal )'},\n",
       " {'Field': 'PRODTYPE',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Product data description ( image | mask | expmap )'},\n",
       " {'Field': 'PROGRAM', 'Type': 'str', 'Desc': 'Current observing orogram'},\n",
       " {'Field': 'PROPID', 'Type': 'str', 'Desc': 'Proposal Id'},\n",
       " {'Field': 'PROPOSER', 'Type': 'str', 'Desc': 'Proposer(s)'},\n",
       " {'Field': 'PUPILAMP', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PUPILMAX', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'PUPILSKY', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RA', 'Type': 'str', 'Desc': 'right ascension (telescope)'},\n",
       " {'Field': 'RADECSYS', 'Type': 'str', 'Desc': 'Default coordinate system'},\n",
       " {'Field': 'RADESYS',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'World coordinate reference frame'},\n",
       " {'Field': 'RADIUS', 'Type': 'str', 'Desc': '[pix] Average radius of stars'},\n",
       " {'Field': 'RADSTD',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[pix] Standard deviation of star radii'},\n",
       " {'Field': 'RECNO',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'NOAO Science Archive sequence number'},\n",
       " {'Field': 'REQNUM', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'RMCOUNT', 'Type': 'str', 'Desc': 'remediation counter'},\n",
       " {'Field': 'SB_ACCOU', 'Type': 'str', 'Desc': 'iSTB client user account'},\n",
       " {'Field': 'SB_DIR1', 'Type': 'str', 'Desc': 'level 1 directory in NSA DS'},\n",
       " {'Field': 'SB_DIR2', 'Type': 'str', 'Desc': 'level 2 directory in NSA DS'},\n",
       " {'Field': 'SB_DIR3', 'Type': 'str', 'Desc': 'level 3 directory in NSA DS'},\n",
       " {'Field': 'SB_HOST', 'Type': 'str', 'Desc': 'iSTB client host'},\n",
       " {'Field': 'SB_ID', 'Type': 'str', 'Desc': 'unique iSTB identifier'},\n",
       " {'Field': 'SB_LOCAL', 'Type': 'str', 'Desc': 'locale of iSTB daemon'},\n",
       " {'Field': 'SB_NAME', 'Type': 'str', 'Desc': 'name assigned by iSTB'},\n",
       " {'Field': 'SB_RECNO', 'Type': 'str', 'Desc': 'iSTB sequence number'},\n",
       " {'Field': 'SB_RTNAM', 'Type': 'str', 'Desc': 'NSA root name'},\n",
       " {'Field': 'SB_SITE', 'Type': 'str', 'Desc': 'iSTB host site'},\n",
       " {'Field': 'SCAMPCHI', 'Type': 'str', 'Desc': 'chi2 value from SCAMP'},\n",
       " {'Field': 'SCAMPFLG',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Flag for SCAMP; 0=no WARNING; 1=WARNING exists'},\n",
       " {'Field': 'SCAMPNUM',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'number of matched stars from SCAMP'},\n",
       " {'Field': 'SCAMPREF', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SEQID', 'Type': 'str', 'Desc': 'Sequence ID'},\n",
       " {'Field': 'SEQNUM', 'Type': 'str', 'Desc': 'Sequence Number'},\n",
       " {'Field': 'SEQTOT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Total number of images in sequence'},\n",
       " {'Field': 'SISPIVER', 'Type': 'str', 'Desc': 'SISPI software version'},\n",
       " {'Field': 'SKYORDER', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYPC00', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYPC01', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYPC02', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYPC03', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYSTAT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Cloud camera (RASICAM) available if true'},\n",
       " {'Field': 'SKYSUB', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'SKYUPDAT',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'Time of last RASICAM exposure (UTC)'},\n",
       " {'Field': 'SLOT01', 'Type': 'str', 'Desc': 'Monsoon module'},\n",
       " {'Field': 'SLOT03', 'Type': 'str', 'Desc': 'Monsoon module'},\n",
       " {'Field': 'SURVEYID', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'TELDEC', 'Type': 'str', 'Desc': 'DEC of telescope (deg)'},\n",
       " {'Field': 'TELEQUIN', 'Type': 'str', 'Desc': 'Equinox of telescope coords'},\n",
       " {'Field': 'TELESCOP', 'Type': 'str', 'Desc': 'Specific system'},\n",
       " {'Field': 'TELFOCUS', 'Type': 'str', 'Desc': 'Telescope focus'},\n",
       " {'Field': 'TELRA', 'Type': 'str', 'Desc': 'RA of telescope (hr)'},\n",
       " {'Field': 'TELSTAT', 'Type': 'str', 'Desc': 'Telescope tracking status'},\n",
       " {'Field': 'TILING', 'Type': 'str', 'Desc': 'DES Tiling number'},\n",
       " {'Field': 'TIME-OBS', 'Type': 'str', 'Desc': 'UT at start of exposure'},\n",
       " {'Field': 'time_recorded', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'TIME_RECORDED', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'TIMESYS', 'Type': 'str', 'Desc': 'Time system'},\n",
       " {'Field': 'UNITNAME', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'UPTRTEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Upper truss average temperature'},\n",
       " {'Field': 'UTE-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Upper truss temperature east'},\n",
       " {'Field': 'UTN-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Upper truss temperature north'},\n",
       " {'Field': 'UTS-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Upper truss temperature south'},\n",
       " {'Field': 'UTW-TEMP',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[deg C] Upper truss temperature west'},\n",
       " {'Field': 'VALIDA', 'Type': 'str', 'Desc': 'Data from amp A is valid'},\n",
       " {'Field': 'VALIDB', 'Type': 'str', 'Desc': 'Data from amp B is valid'},\n",
       " {'Field': 'VSUB',\n",
       "  'Type': 'str',\n",
       "  'Desc': 'True if CCD substrate voltage is on'},\n",
       " {'Field': 'WCSCAL', 'Type': 'str', 'Desc': ''},\n",
       " {'Field': 'WINDDIR', 'Type': 'str', 'Desc': 'Wind direction (degrees)'},\n",
       " {'Field': 'WINDSPD', 'Type': 'str', 'Desc': 'Wind speed (mph)'},\n",
       " {'Field': 'XTALKFIL', 'Type': 'str', 'Desc': 'Crosstalk file'},\n",
       " {'Field': 'ZD', 'Type': 'str', 'Desc': 'zenith distance (degrees)'},\n",
       " {'Field': 'ZPDELDEC',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Telescope zeropoint adjustment (DEC)'},\n",
       " {'Field': 'ZPDELRA',\n",
       "  'Type': 'str',\n",
       "  'Desc': '[arcsec] Telescope zeropoint adjustment (RA)'}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get(f'{adsurl}/aux_file_fields/decam/instcal/').json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again with some digging and experimentation, we'd like to pull out:\n",
    "* G-TRANSP (transparency as measured by the guider)\n",
    "* AIRMASS (to check that transparency correlates with airmass, as it should on clear nights)\n",
    "\n",
    "Since we're going to use information from two database tables, we'll also need a way to join them.  The unique file identifier MD5SUM (called \"fitsfile\" in the AUX HDU service) will let us do that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Doing the search <a id=\"search\" />\n",
    "So let's issue our searches to the services.  Following the examples in the Advanced Search notebook, we'll start by getting the transparency information from the full file service.  We constraint the search by specifying the instrument, proc_type, prod_type, obs_type, proposal, and filter:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Search that combines fields and filter criterial (like \"WHERE clause\") for File and Hdu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "---------------------------------------------------------------------------",
      "KeyboardInterrupt                         Traceback (most recent call last)",
      "<timed exec> in <module>\n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/requests/api.py in post(url, data, json, **kwargs)\n    117     \"\"\"\n    118 \n--> 119     return request('post', url, data=data, json=json, **kwargs)\n    120 \n    121 \n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/requests/api.py in request(method, url, **kwargs)\n     59     # cases, and look like a memory leak in others.\n     60     with sessions.Session() as session:\n---> 61         return session.request(method=method, url=url, **kwargs)\n     62 \n     63 \n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\n    540         }\n    541         send_kwargs.update(settings)\n--> 542         resp = self.send(prep, **send_kwargs)\n    543 \n    544         return resp\n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/requests/sessions.py in send(self, request, **kwargs)\n    653 \n    654         # Send the request\n--> 655         r = adapter.send(request, **kwargs)\n    656 \n    657         # Total elapsed time of the request (approximately)\n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)\n    437         try:\n    438             if not chunked:\n--> 439                 resp = conn.urlopen(\n    440                     method=request.method,\n    441                     url=url,\n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\n    697 \n    698             # Make the request on the httplib connection object.\n--> 699             httplib_response = self._make_request(\n    700                 conn,\n    701                 method,\n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)\n    443                     # Python 3 (including for exceptions like SystemExit).\n    444                     # Otherwise it looks like a bug in the code.\n--> 445                     six.raise_from(e, None)\n    446         except (SocketTimeout, BaseSSLError, SocketError) as e:\n    447             self._raise_timeout(err=e, url=url, timeout_value=read_timeout)\n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/urllib3/packages/six.py in raise_from(value, from_value)\n",
      "~/sandbox/nat-nb/venv/lib/python3.8/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)\n    438                 # Python 3\n    439                 try:\n--> 440                     httplib_response = conn.getresponse()\n    441                 except BaseException as e:\n    442                     # Remove the TypeError from the exception chain in\n",
      "/usr/lib/python3.8/http/client.py in getresponse(self)\n   1345         try:\n   1346             try:\n-> 1347                 response.begin()\n   1348             except ConnectionError:\n   1349                 self.close()\n",
      "/usr/lib/python3.8/http/client.py in begin(self)\n    305         # read until we get a non-100 response\n    306         while True:\n--> 307             version, status, reason = self._read_status()\n    308             if status != CONTINUE:\n    309                 break\n",
      "/usr/lib/python3.8/http/client.py in _read_status(self)\n    266 \n    267     def _read_status(self):\n--> 268         line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n    269         if len(line) > _MAXLINE:\n    270             raise LineTooLong(\"status line\")\n",
      "/usr/lib/python3.8/socket.py in readinto(self, b)\n    667         while True:\n    668             try:\n--> 669                 return self._sock.recv_into(b)\n    670             except timeout:\n    671                 self._timeout_occurred = True\n",
      "/usr/lib/python3.8/ssl.py in recv_into(self, buffer, nbytes, flags)\n   1239                   \"non-zero flags not allowed in calls to recv_into() on %s\" %\n   1240                   self.__class__)\n-> 1241             return self.read(nbytes, buffer)\n   1242         else:\n   1243             return super().recv_into(buffer, nbytes, flags)\n",
      "/usr/lib/python3.8/ssl.py in read(self, len, buffer)\n   1097         try:\n   1098             if buffer is not None:\n-> 1099                 return self._sslobj.read(len, buffer)\n   1100             else:\n   1101                 return self._sslobj.read(len)\n",
      "KeyboardInterrupt: "
     ]
    }
   ],
   "source": [
    "%%time\n",
    "jj = {\n",
    "        \"outfields\" : [\n",
    "            ### Files\n",
    "            \"exposure\",\n",
    "            \"ifilter\",\n",
    "            \"AIRMASS\",\n",
    "            \"G-TRANSP\",\n",
    "            ### HDUs\n",
    "            \"hdu:ra_min\",  # bounding box version of \"corners\"\n",
    "            \"hdu:ra_max\",  # bounding box version of \"corners\"\n",
    "            \"hdu:dec_min\", # bounding box version of \"corners\"\n",
    "            \"hdu:dec_max\", # bounding box version of \"corners\"\n",
    "            #\"hdu:COR1RA1\",\n",
    "            #\"hdu:COR2RA1\",\n",
    "            #\"hdu:COR3RA1\",\n",
    "            #\"hdu:COR4RA1\",\n",
    "            #\"hdu:COR1DEC1\",\n",
    "            #\"hdu:COR2DEC1\",\n",
    "            #\"hdu:COR3DEC1\",II\n",
    "            #\"hdu:COR4DEC1\",\n",
    "            \"hdu:FWHM\",\n",
    "            \"hdu:AVSKY\", \n",
    "        ],\n",
    "        \"search\" : [\n",
    "            #[\"hdu:ra_center\", -400, 400], # Exclude HDUs with no ra/dec\n",
    "            #[\"caldat\", \"2016-09-01\", \"2020-09-01\"], #@@@\n",
    "            [\"prod_type\", \"image\"],\n",
    "            [\"obs_type\", \"object\"],\n",
    "            #[\"proposal\", \"2012B-0001\"], #@@@\n",
    "            [\"ifilter\", \"r DECam\", \"startswith\"],\n",
    "            [\"instrument\", \"decam\"],\n",
    "            [\"proc_type\", \"instcal\"]\n",
    "        ]}\n",
    "dfm = None # Pandas DataFrame\n",
    "#apiurl=f'{adsurl}/find/?rectype=hdu&limit=5000000'\n",
    "apiurl=f'{adsurl}/find/?rectype=hdu&limit=1000'\n",
    "#apiurl=f'{adsurl}/find/?rectype=hdu&limit=None' #@@@\n",
    "response = requests.post(apiurl,json=jj)\n",
    "\n",
    "if response.status_code == 200:\n",
    "    dfm = pd.DataFrame(response.json()[1:])\n",
    "    print(f'File records retrieved: {dfm.shape}')\n",
    "else:\n",
    "    print(response.json()['errorMessage'])\n",
    "    print(response.json()['traceback'])  # for API developer use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'shape'",
     "output_type": "error",
     "traceback": [
      "---------------------------------------------------------------------------",
      "AttributeError                            Traceback (most recent call last)",
      "<ipython-input-7-2c64c58ac4f4> in <module>\n----> 1 dfm.shape\n",
      "AttributeError: 'NoneType' object has no attribute 'shape'"
     ]
    }
   ],
   "source": [
    "dfm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfmc = dfm.dropna()\n",
    "len(dfmc), len(dfm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "apix = 0.263 # arcsec/pixel\n",
    "sky = dfmc['hdu:AVSKY']/dfmc['exposure'] # sky rate\n",
    "tau = dfmc['G-TRANSP']**2/(dfmc['hdu:FWHM']*apix/0.9)**2/(sky/3.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = plt.hist(tau,bins=200,range=(0,1))\n",
    "plt.xlabel('tau')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tau_trim = np.clip(tau,0,1) # tau should be between 0 and 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making the depth map <a id=\"depth\" />\n",
    "Now that we have the needed quantities, we can begin to make our depth map.  Our procedure will be to create a Healpix map, use the coordinates of the HDU corners to identify which healpixels are spanned by each HDU, and add the value of $\\tau$ for those HDUs to the appropriate Healpixels.  A couple of notes:\n",
    "\n",
    "* We approximate the corners of the HDUs using its bounding box.  The bounding box values are faster to retrieve, are guaranteed to be ordered, but are less accurate the the corners.\n",
    "* We'll need to loop over all of the HDUs one at a time, which can be slow.  Parallel processing might help here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Approximate corners using bounding box\n",
    "This avoids the need to sort the corners."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "radectab_s = list(zip(\n",
    "    dfmc[['hdu:ra_min','hdu:ra_max','hdu:ra_max','hdu:ra_min']].values,\n",
    "    dfmc[['hdu:dec_max','hdu:dec_max','hdu:dec_min','hdu:dec_min']].values))\n",
    "vectab = [hp.ang2vec(ra1,dec1,lonlat=True) for ra1, dec1 in radectab_s]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the Healpix map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nside = 4096\n",
    "print('Resolution is {:5.2f} arcmin'.format(hp.nside2resol(nside,arcmin=True)))\n",
    "map = np.zeros(hp.nside2npix(nside)) # raw exposure map\n",
    "tmap = map.copy() # teff map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loop over HDUs (slow!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "for vec,exptime,tau1 in zip(vectab,dfmc['exposure'],tau_trim):\n",
    "    try:\n",
    "        ipix = hp.query_polygon(nside,vec)\n",
    "        map[ipix] += exptime\n",
    "        tmap[ipix] += tau1 * exptime\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.count_nonzero(tmap) #@@@"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show the map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(15,15))\n",
    "cmap = copy.copy(matplotlib.cm.get_cmap(\"inferno\"))\n",
    "hp.orthview(tmap,rot=(20,-30),fig=1,cmap=cmap,half_sky=True,min=0,max=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Zooming in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hp.gnomview(tmap,reso=0.75,cmap=cmap,rot=(8,-44),min=0,max=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show the map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "elapsed = toc()  # Seconds since start of Notebook load (assuming fresh run)\n",
    "print(f'Elapsed seconds={elapsed} on {natroot}')\n",
    "print(f'Completed on: {str(datetime.datetime.now())}')\n",
    "assert natroot == 'https://astroarchive.noao.edu', 'Notebook does NOT point to PRODUCTION'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources and references <a id=\"docs\" />\n",
    "* Advanced Search [notebook](https://github.com/NOAO/nat-nb)\n",
    "* Nielsen, E. et al., 2016, [FERMILAB-TM-2610-AE-CD](https://www.osti.gov/biblio/1250877-limiting-magnitude-teff-image-quality-des-year)\n",
    "* https://algorithmtutor.com/Computational-Geometry/Area-of-a-polygon-given-a-set-of-points/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}