part_1_mars_news.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Challenge\n",
    "## Deliverable 1: Scrape Titles and Preview Text from Mars News"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Splinter and BeautifulSoup\n",
    "from splinter import Browser\n",
    "from bs4 import BeautifulSoup as soup\n",
    "from webdriver_manager.chrome import ChromeDriverManager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "executable_path = {'executable_path': ChromeDriverManager().install()}\n",
    "browser = Browser('chrome', **executable_path, headless=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Visit the Website\n",
    "\n",
    "1. Use automated browsing to visit the [Mars NASA news site](https://redplanetscience.com). Inspect the page to identify which elements to scrape.\n",
    "\n",
    "      > **Hint** To identify which elements to scrape, you might want to inspect the page by using Chrome DevTools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Visit the Mars NASA news site\n",
    "url = 'https://redplanetscience.com'\n",
    "browser.visit(url)\n",
    "# Optional delay for loading the page\n",
    "browser.is_element_present_by_css('div.list_text', wait_time=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Scrape the Website\n",
    "\n",
    "Create a Beautiful Soup object and use it to extract text elements from the website."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Beautiful Soup object\n",
    "html = browser.html\n",
    "html_soup = soup(html, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<div class=\"list_text\">\n",
      "<div class=\"list_date\">March 15, 2023</div>\n",
      "<div class=\"content_title\">InSight's 'Mole' Team Peers into the Pit</div>\n",
      "<div class=\"article_teaser_body\">Efforts to save the heat probe continue.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 14, 2023</div>\n",
      "<div class=\"content_title\">NASA's New Mars Rover Is Ready for Space Lasers</div>\n",
      "<div class=\"article_teaser_body\">Perseverance is one of a few Mars spacecraft carrying laser retroreflectors. The devices could provide new science and safer Mars landings in the future.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 11, 2023</div>\n",
      "<div class=\"content_title\">Mars 2020 Unwrapped and Ready for More Testing</div>\n",
      "<div class=\"article_teaser_body\">In time-lapse video, bunny-suited engineers remove the inner layer of protective foil on NASA's Mars 2020 rover after it was relocated for testing.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 11, 2023</div>\n",
      "<div class=\"content_title\">Heat and Dust Help Launch Martian Water Into Space, Scientists Find</div>\n",
      "<div class=\"article_teaser_body\">Scientists using an instrument aboard NASA’s Mars Atmosphere and Volatile EvolutioN, or MAVEN, spacecraft have discovered that water vapor near the surface of the Red Planet is lofted higher into the atmosphere than anyone expected was possible. </div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 8, 2023</div>\n",
      "<div class=\"content_title\">NASA Administrator Statement on Moon to Mars Initiative, FY 2021 Budget</div>\n",
      "<div class=\"article_teaser_body\">Jim Bridenstine addresses NASA's ambitious plans for the coming years, including Mars Sample Return.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 3, 2023</div>\n",
      "<div class=\"content_title\">Media Get a Close-Up of NASA's Mars 2020 Rover</div>\n",
      "<div class=\"article_teaser_body\">The clean room at NASA's Jet Propulsion Laboratory was open to the media to see NASA's next Mars explorer before it leaves for Florida in preparation for a summertime launch.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 2, 2023</div>\n",
      "<div class=\"content_title\">Global Storms on Mars Launch Dust Towers Into the Sky</div>\n",
      "<div class=\"article_teaser_body\">A Mars Dust Tower Stands Out Dust storms are common on Mars. But every decade or so, something unpredictable happens: a series of runaway storms break out, covering the entire planet in a dusty haze.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 1, 2023</div>\n",
      "<div class=\"content_title\">NASA Invites Public to Share Excitement of Mars 2020 Perseverance Rover Launch</div>\n",
      "<div class=\"article_teaser_body\">There are lots of ways to participate in the historic event, which is targeted for July 30.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">March 1, 2023</div>\n",
      "<div class=\"content_title\">Robotic Toolkit Added to NASA's Mars 2020 Rover</div>\n",
      "<div class=\"article_teaser_body\">The bit carousel, which lies at the heart of the rover's Sample Caching System, is now aboard NASA's newest rover. </div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">February 27, 2023</div>\n",
      "<div class=\"content_title\">Mars InSight Lander to Push on Top of the 'Mole'</div>\n",
      "<div class=\"article_teaser_body\">Engineers have a plan for pushing down on the heat probe, which has been stuck at the Martian surface for a year.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">February 25, 2023</div>\n",
      "<div class=\"content_title\">NASA's Perseverance Mars Rover Gets Balanced</div>\n",
      "<div class=\"article_teaser_body\">The mission team performed a crucial weight-balancing test on the rover in preparation for this summer's history-making launch to the Red Planet.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">February 24, 2023</div>\n",
      "<div class=\"content_title\">NASA's Push to Save the Mars InSight Lander's Heat Probe</div>\n",
      "<div class=\"article_teaser_body\">The scoop on the end of the spacecraft's robotic arm will be used to 'pin' the mole against the wall of its hole.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">February 24, 2023</div>\n",
      "<div class=\"content_title\">NASA's Perseverance Mars Rover Gets Its Wheels and Air Brakes</div>\n",
      "<div class=\"article_teaser_body\">After the rover was shipped from JPL to Kennedy Space Center, the team is getting closer to finalizing the spacecraft for launch later this summer.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">February 19, 2023</div>\n",
      "<div class=\"content_title\">NASA's Ingenuity Mars Helicopter Recharges Its Batteries in Flight</div>\n",
      "<div class=\"article_teaser_body\">Headed to the Red Planet with the Perseverance rover, the pioneering helicopter is powered up for the first time in interplanetary space as part of a systems check.</div>\n",
      "</div>, <div class=\"list_text\">\n",
      "<div class=\"list_date\">February 15, 2023</div>\n",
      "<div class=\"content_title\">NASA Wins 4 Webbys, 4 People's Voice Awards</div>\n",
      "<div class=\"article_teaser_body\">Winners include the JPL-managed \"Send Your Name to Mars\" campaign, NASA's Global Climate Change website and Solar System Interactive.</div>\n",
      "</div>]\n"
     ]
    }
   ],
   "source": [
    "# Extract all the text elements\n",
    "text = html_soup.find_all('div', class_= 'list_text')\n",
    "\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Store the Results\n",
    "\n",
    "Extract the titles and preview text of the news articles that you scraped. Store the scraping results in Python data structures as follows:\n",
    "\n",
    "* Store each title-and-preview pair in a Python dictionary. And, give each dictionary two keys: `title` and `preview`. An example is the following:\n",
    "\n",
    "  ```python\n",
    "  {'title': \"Mars Rover Begins Mission!\", \n",
    "        'preview': \"NASA's Mars Rover begins a multiyear mission to collect data about the little-explored planet.\"}\n",
    "  ```\n",
    "\n",
    "* Store all the dictionaries in a Python list.\n",
    "\n",
    "* Print the list in your notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an empty list to store the dictionaries\n",
    "list_text = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#title = title_prev.find('div', class_=\"content_title\").text\n",
    "#title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loop through the text elements\n",
    "for title_prev in text:\n",
    "# Extract the title and preview text from the elements\n",
    "    title = title_prev.find('div', class_=\"content_title\").text\n",
    "    preview = title_prev.find('div', class_=\"article_teaser_body\").text\n",
    "# Store each title and preview pair in a dictionary\n",
    "    list_text_pair = {}\n",
    "    list_text_pair[\"title\"] = title\n",
    "    list_text_pair[\"preview\"] = preview\n",
    "# Add the dictionary to the list\n",
    "    list_text.append(list_text_pair)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'title': \"InSight's 'Mole' Team Peers into the Pit\",\n",
       "  'preview': 'Efforts to save the heat probe continue.'},\n",
       " {'title': \"NASA's New Mars Rover Is Ready for Space Lasers\",\n",
       "  'preview': 'Perseverance is one of a few Mars spacecraft carrying laser retroreflectors. The devices could provide new science and safer Mars landings in the future.'},\n",
       " {'title': 'Mars 2020 Unwrapped and Ready for More Testing',\n",
       "  'preview': \"In time-lapse video, bunny-suited engineers remove the inner layer of protective foil on NASA's Mars 2020 rover after it was relocated for testing.\"},\n",
       " {'title': 'Heat and Dust Help Launch Martian Water Into Space, Scientists Find',\n",
       "  'preview': 'Scientists using an instrument aboard NASA’s Mars Atmosphere and Volatile EvolutioN, or MAVEN, spacecraft have discovered that water vapor near the surface of the Red Planet is lofted higher into the atmosphere than anyone expected was possible. '},\n",
       " {'title': 'NASA Administrator Statement on Moon to Mars Initiative, FY 2021 Budget',\n",
       "  'preview': \"Jim Bridenstine addresses NASA's ambitious plans for the coming years, including Mars Sample Return.\"},\n",
       " {'title': \"Media Get a Close-Up of NASA's Mars 2020 Rover\",\n",
       "  'preview': \"The clean room at NASA's Jet Propulsion Laboratory was open to the media to see NASA's next Mars explorer before it leaves for Florida in preparation for a summertime launch.\"},\n",
       " {'title': 'Global Storms on Mars Launch Dust Towers Into the Sky',\n",
       "  'preview': 'A Mars Dust Tower Stands Out Dust storms are common on Mars. But every decade or so, something unpredictable happens: a series of runaway storms break out, covering the entire planet in a dusty haze.'},\n",
       " {'title': 'NASA Invites Public to Share Excitement of Mars 2020 Perseverance Rover Launch',\n",
       "  'preview': 'There are lots of ways to participate in the historic event, which is targeted for July 30.'},\n",
       " {'title': \"Robotic Toolkit Added to NASA's Mars 2020 Rover\",\n",
       "  'preview': \"The bit carousel, which lies at the heart of the rover's Sample Caching System, is now aboard NASA's newest rover. \"},\n",
       " {'title': \"Mars InSight Lander to Push on Top of the 'Mole'\",\n",
       "  'preview': 'Engineers have a plan for pushing down on the heat probe, which has been stuck at the Martian surface for a year.'},\n",
       " {'title': \"NASA's Perseverance Mars Rover Gets Balanced\",\n",
       "  'preview': \"The mission team performed a crucial weight-balancing test on the rover in preparation for this summer's history-making launch to the Red Planet.\"},\n",
       " {'title': \"NASA's Push to Save the Mars InSight Lander's Heat Probe\",\n",
       "  'preview': \"The scoop on the end of the spacecraft's robotic arm will be used to 'pin' the mole against the wall of its hole.\"},\n",
       " {'title': \"NASA's Perseverance Mars Rover Gets Its Wheels and Air Brakes\",\n",
       "  'preview': 'After the rover was shipped from JPL to Kennedy Space Center, the team is getting closer to finalizing the spacecraft for launch later this summer.'},\n",
       " {'title': \"NASA's Ingenuity Mars Helicopter Recharges Its Batteries in Flight\",\n",
       "  'preview': 'Headed to the Red Planet with the Perseverance rover, the pioneering helicopter is powered up for the first time in interplanetary space as part of a systems check.'},\n",
       " {'title': \"NASA Wins 4 Webbys, 4 People's Voice Awards\",\n",
       "  'preview': 'Winners include the JPL-managed \"Send Your Name to Mars\" campaign, NASA\\'s Global Climate Change website and Solar System Interactive.'}]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the list to confirm success\n",
    "list_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Step 4: Export the Data\n",
    "\n",
    "Optionally, store the scraped data in a file or database (to ease sharing the data with others). To do so, export the scraped data to either a JSON file or a MongoDB database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"title\": \"InSight\\'s \\'Mole\\' Team Peers into the Pit\", \"preview\": \"Efforts to save the heat probe continue.\"}, {\"title\": \"NASA\\'s New Mars Rover Is Ready for Space Lasers\", \"preview\": \"Perseverance is one of a few Mars spacecraft carrying laser retroreflectors. The devices could provide new science and safer Mars landings in the future.\"}, {\"title\": \"Mars 2020 Unwrapped and Ready for More Testing\", \"preview\": \"In time-lapse video, bunny-suited engineers remove the inner layer of protective foil on NASA\\'s Mars 2020 rover after it was relocated for testing.\"}, {\"title\": \"Heat and Dust Help Launch Martian Water Into Space, Scientists Find\", \"preview\": \"Scientists using an instrument aboard NASA\\\\u2019s Mars Atmosphere and Volatile EvolutioN, or MAVEN, spacecraft have discovered that water vapor near the surface of the Red Planet is lofted higher into the atmosphere than anyone expected was possible. \"}, {\"title\": \"NASA Administrator Statement on Moon to Mars Initiative, FY 2021 Budget\", \"preview\": \"Jim Bridenstine addresses NASA\\'s ambitious plans for the coming years, including Mars Sample Return.\"}, {\"title\": \"Media Get a Close-Up of NASA\\'s Mars 2020 Rover\", \"preview\": \"The clean room at NASA\\'s Jet Propulsion Laboratory was open to the media to see NASA\\'s next Mars explorer before it leaves for Florida in preparation for a summertime launch.\"}, {\"title\": \"Global Storms on Mars Launch Dust Towers Into the Sky\", \"preview\": \"A Mars Dust Tower Stands Out Dust storms are common on Mars. But every decade or so, something unpredictable happens: a series of runaway storms break out, covering the entire planet in a dusty haze.\"}, {\"title\": \"NASA Invites Public to Share Excitement of Mars 2020 Perseverance Rover Launch\", \"preview\": \"There are lots of ways to participate in the historic event, which is targeted for July 30.\"}, {\"title\": \"Robotic Toolkit Added to NASA\\'s Mars 2020 Rover\", \"preview\": \"The bit carousel, which lies at the heart of the rover\\'s Sample Caching System, is now aboard NASA\\'s newest rover. \"}, {\"title\": \"Mars InSight Lander to Push on Top of the \\'Mole\\'\", \"preview\": \"Engineers have a plan for pushing down on the heat probe, which has been stuck at the Martian surface for a year.\"}, {\"title\": \"NASA\\'s Perseverance Mars Rover Gets Balanced\", \"preview\": \"The mission team performed a crucial weight-balancing test on the rover in preparation for this summer\\'s history-making launch to the Red Planet.\"}, {\"title\": \"NASA\\'s Push to Save the Mars InSight Lander\\'s Heat Probe\", \"preview\": \"The scoop on the end of the spacecraft\\'s robotic arm will be used to \\'pin\\' the mole against the wall of its hole.\"}, {\"title\": \"NASA\\'s Perseverance Mars Rover Gets Its Wheels and Air Brakes\", \"preview\": \"After the rover was shipped from JPL to Kennedy Space Center, the team is getting closer to finalizing the spacecraft for launch later this summer.\"}, {\"title\": \"NASA\\'s Ingenuity Mars Helicopter Recharges Its Batteries in Flight\", \"preview\": \"Headed to the Red Planet with the Perseverance rover, the pioneering helicopter is powered up for the first time in interplanetary space as part of a systems check.\"}, {\"title\": \"NASA Wins 4 Webbys, 4 People\\'s Voice Awards\", \"preview\": \"Winners include the JPL-managed \\\\\"Send Your Name to Mars\\\\\" campaign, NASA\\'s Global Climate Change website and Solar System Interactive.\"}]'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Export data to JSON\n",
    "import json\n",
    "json.dumps(list_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export data to MongoDB\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}