chapter07.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CSS 與 XPath 選擇\n",
    "\n",
    "## 郭耀仁"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## CSS 選擇\n",
    "\n",
    "- 簡單的結構可以利用文件物件模型（Document Object Model, DOM）判斷"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading required package: xml2\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "'Hello HTML'"
      ],
      "text/latex": [
       "'Hello HTML'"
      ],
      "text/markdown": [
       "'Hello HTML'"
      ],
      "text/plain": [
       "[1] \"Hello HTML\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(rvest)\n",
    "\n",
    "raw_html <- '\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "<head>\n",
    "    <meta charset=\"utf-8\">\n",
    "    <meta name=\"viewport\" content=\"width=device-width\">\n",
    "    <title>Basic HTML Page</title>\n",
    "</head>\n",
    "<body>\n",
    "    <p>Hello HTML</p>\n",
    "</body>\n",
    "</html>\n",
    "'\n",
    "hello_html <- read_html(raw_html) %>%\n",
    "    html_nodes(css = \"body p\") %>%\n",
    "    html_text()\n",
    "hello_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（2）\n",
    "\n",
    "- 複雜的結構透過 Chrome 外掛：[SelectorGadget](https://chrome.google.com/webstore/search/selectorgadget?hl=zh-TW)\n",
    "- 我們繼續以 [IMDb](http://www.imdb.com/title/tt3783958/) 為例\n",
    "- 評等（Rating）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（3）\n",
    "\n",
    "- 點選 SelectorGadget 的外掛圖示\n",
    "- 先點選評等（Rating）\n",
    "\n",
    "![](img/ch0701.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（4）\n",
    "\n",
    "- 會有多個元素被選出（標註黃色），因為這頁有很多 `<span></span>`\n",
    "\n",
    "![](img/ch0702.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（5）\n",
    "\n",
    "- 接著點選不要的元素（標註紅色）\n",
    "\n",
    "![](img/ch0703.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（6）\n",
    "\n",
    "- 這時我們就得到正確的 CSS 選擇：`strong span`\n",
    "\n",
    "![](img/ch0704.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（7）\n",
    "\n",
    "- 這時就知道在 `html_nodes()` 函數中需要指定 `css = \"strong span\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "8.3"
      ],
      "text/latex": [
       "8.3"
      ],
      "text/markdown": [
       "8.3"
      ],
      "text/plain": [
       "[1] 8.3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(rvest)\n",
    "\n",
    "url <- \"http://www.imdb.com/title/tt3783958/\"\n",
    "lalaland <- read_html(url)\n",
    "rating <- lalaland %>%\n",
    "    html_nodes(css = \"strong span\") %>%\n",
    "    html_text() %>%\n",
    "    as.numeric()\n",
    "rating"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（8）\n",
    "\n",
    "- 再練習一次，這次我們要將這頁的演員名單（Cast）擷取出來：\n",
    "\n",
    "![](img/ch0705.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（9）\n",
    "\n",
    "- 先點選男主角 Ryan Gosling\n",
    "- 乍看之下好像 `.itemprop` 這個類別就對了\n",
    "\n",
    "![](img/ch0706.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（10）\n",
    "\n",
    "- 但是其實其他的元素也有使用 `.itemprop` 這個類別\n",
    "\n",
    "![](img/ch0707.png)\n",
    "\n",
    "![](img/ch0708.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（11）\n",
    "\n",
    "- 於是點選不要的元素（標註紅色）\n",
    "\n",
    "![](img/ch0709.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（12）\n",
    "\n",
    "- 這時我們就得到正確的 CSS 選擇：`.itemprop .itemprop`\n",
    "\n",
    "![](img/ch0710.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## CSS 選擇（13）\n",
    "\n",
    "- 這時就知道在 `html_nodes()` 函數中需要指定 `css = \".itemprop .itemprop\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'Ryan Gosling'</li>\n",
       "\t<li>'Emma Stone'</li>\n",
       "\t<li>'Amiée Conn'</li>\n",
       "\t<li>'Terry Walters'</li>\n",
       "\t<li>'Thom Shelton'</li>\n",
       "\t<li>'Cinda Adams'</li>\n",
       "\t<li>'Callie Hernandez'</li>\n",
       "\t<li>'Jessica Rothe'</li>\n",
       "\t<li>'Sonoya Mizuno'</li>\n",
       "\t<li>'Rosemarie DeWitt'</li>\n",
       "\t<li>'J.K. Simmons'</li>\n",
       "\t<li>'Claudine Claudio'</li>\n",
       "\t<li>'Jason Fuchs'</li>\n",
       "\t<li>'D.A. Wallach'</li>\n",
       "\t<li>'Trevor Lissauer'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'Ryan Gosling'\n",
       "\\item 'Emma Stone'\n",
       "\\item 'Amiée Conn'\n",
       "\\item 'Terry Walters'\n",
       "\\item 'Thom Shelton'\n",
       "\\item 'Cinda Adams'\n",
       "\\item 'Callie Hernandez'\n",
       "\\item 'Jessica Rothe'\n",
       "\\item 'Sonoya Mizuno'\n",
       "\\item 'Rosemarie DeWitt'\n",
       "\\item 'J.K. Simmons'\n",
       "\\item 'Claudine Claudio'\n",
       "\\item 'Jason Fuchs'\n",
       "\\item 'D.A. Wallach'\n",
       "\\item 'Trevor Lissauer'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'Ryan Gosling'\n",
       "2. 'Emma Stone'\n",
       "3. 'Amiée Conn'\n",
       "4. 'Terry Walters'\n",
       "5. 'Thom Shelton'\n",
       "6. 'Cinda Adams'\n",
       "7. 'Callie Hernandez'\n",
       "8. 'Jessica Rothe'\n",
       "9. 'Sonoya Mizuno'\n",
       "10. 'Rosemarie DeWitt'\n",
       "11. 'J.K. Simmons'\n",
       "12. 'Claudine Claudio'\n",
       "13. 'Jason Fuchs'\n",
       "14. 'D.A. Wallach'\n",
       "15. 'Trevor Lissauer'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] \"Ryan Gosling\"     \"Emma Stone\"       \"Amiée Conn\"       \"Terry Walters\"   \n",
       " [5] \"Thom Shelton\"     \"Cinda Adams\"      \"Callie Hernandez\" \"Jessica Rothe\"   \n",
       " [9] \"Sonoya Mizuno\"    \"Rosemarie DeWitt\" \"J.K. Simmons\"     \"Claudine Claudio\"\n",
       "[13] \"Jason Fuchs\"      \"D.A. Wallach\"     \"Trevor Lissauer\" "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(rvest)\n",
    "\n",
    "url <- \"http://www.imdb.com/title/tt3783958/\"\n",
    "lalaland <- read_html(url)\n",
    "cast <- lalaland %>%\n",
    "    html_nodes(css = \".itemprop .itemprop\") %>%\n",
    "    html_text()\n",
    "cast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 練習\n",
    "\n",
    "- 使用 CSS 選擇將 <https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e=tse> 的股票代號/名稱與成交價抓出來\n",
    "- 股票代號/名稱可能會有編碼的問題：\n",
    "    - `read_html(x, encoding = \"cp950\")`\n",
    "    - `iconv(from = \"UTF-8\", to = \"UTF-8\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## XPath 選擇\n",
    "\n",
    "- 簡單的結構可以利用文件物件模型（Document Object Model, DOM）判斷"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'Hello HTML'"
      ],
      "text/latex": [
       "'Hello HTML'"
      ],
      "text/markdown": [
       "'Hello HTML'"
      ],
      "text/plain": [
       "[1] \"Hello HTML\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(rvest)\n",
    "\n",
    "raw_html <- '\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "<head>\n",
    "    <meta charset=\"utf-8\">\n",
    "    <meta name=\"viewport\" content=\"width=device-width\">\n",
    "    <title>Basic HTML Page</title>\n",
    "</head>\n",
    "<body>\n",
    "    <p>Hello HTML</p>\n",
    "</body>\n",
    "</html>\n",
    "'\n",
    "hello_html <- read_html(raw_html) %>%\n",
    "    html_nodes(xpath = \"//body/p\") %>%\n",
    "    html_text()\n",
    "hello_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（2）\n",
    "\n",
    "- 複雜的結構透過 Chrome 外掛：[XPath Helper](https://chrome.google.com/webstore/search/xpath%20helper?hl=zh-TW)\n",
    "- 我們繼續以 [IMDb](http://www.imdb.com/title/tt3783958/) 為例\n",
    "- 評等（Rating）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（3）\n",
    "\n",
    "- 點選 XPath Helper 外掛圖示\n",
    "- 按住 shift 之後將游標移到評等（Rating）上面\n",
    "\n",
    "![](img/ch0711.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（4）\n",
    "\n",
    "- 試著將 XPath 縮短一點，可以得到 XPath 選擇：`//strong/span`\n",
    "\n",
    "![](img/ch0712.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（5）\n",
    "\n",
    "- 這時就知道在 `html_nodes()` 函數中需要指定 `xpath = \"//strong/span\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "8.3"
      ],
      "text/latex": [
       "8.3"
      ],
      "text/markdown": [
       "8.3"
      ],
      "text/plain": [
       "[1] 8.3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(rvest)\n",
    "\n",
    "url <- \"http://www.imdb.com/title/tt3783958/\"\n",
    "lalaland <- read_html(url)\n",
    "rating <- lalaland %>%\n",
    "    html_nodes(xpath = \"//strong/span\") %>%\n",
    "    html_text() %>%\n",
    "    as.numeric()\n",
    "rating"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（6）\n",
    "\n",
    "- 再練習一次，這次我們要將這頁的演員名單（Cast）擷取出來：\n",
    "\n",
    "![](img/ch0705.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（7）\n",
    "\n",
    "- 按住 shift 後點選男主角 Ryan Gosling\n",
    "\n",
    "![](img/ch0713.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（8）\n",
    "\n",
    "- 按住 shift 後點選女主角 Emma Stone\n",
    "\n",
    "![](img/ch0714.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（9）\n",
    "\n",
    "- 再點選幾個演員之後會發現 `/tbody/tr[@class='even']` 跟 `/tbody/tr[@class='odd']` 一直在變更\n",
    "- 試著將變更的部分都刪除掉，得到我們要的 XPath 選擇：`//td[@class='itemprop']/a/span[@class='itemprop']`\n",
    "\n",
    "![](img/ch0715.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## XPath 選擇（10）\n",
    "\n",
    "- 這時就知道在 `html_nodes()` 函數中需要指定 `xpath = \"//td[@class='itemprop']/a/span[@class='itemprop']\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'Ryan Gosling'</li>\n",
       "\t<li>'Emma Stone'</li>\n",
       "\t<li>'Amiée Conn'</li>\n",
       "\t<li>'Terry Walters'</li>\n",
       "\t<li>'Thom Shelton'</li>\n",
       "\t<li>'Cinda Adams'</li>\n",
       "\t<li>'Callie Hernandez'</li>\n",
       "\t<li>'Jessica Rothe'</li>\n",
       "\t<li>'Sonoya Mizuno'</li>\n",
       "\t<li>'Rosemarie DeWitt'</li>\n",
       "\t<li>'J.K. Simmons'</li>\n",
       "\t<li>'Claudine Claudio'</li>\n",
       "\t<li>'Jason Fuchs'</li>\n",
       "\t<li>'D.A. Wallach'</li>\n",
       "\t<li>'Trevor Lissauer'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'Ryan Gosling'\n",
       "\\item 'Emma Stone'\n",
       "\\item 'Amiée Conn'\n",
       "\\item 'Terry Walters'\n",
       "\\item 'Thom Shelton'\n",
       "\\item 'Cinda Adams'\n",
       "\\item 'Callie Hernandez'\n",
       "\\item 'Jessica Rothe'\n",
       "\\item 'Sonoya Mizuno'\n",
       "\\item 'Rosemarie DeWitt'\n",
       "\\item 'J.K. Simmons'\n",
       "\\item 'Claudine Claudio'\n",
       "\\item 'Jason Fuchs'\n",
       "\\item 'D.A. Wallach'\n",
       "\\item 'Trevor Lissauer'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'Ryan Gosling'\n",
       "2. 'Emma Stone'\n",
       "3. 'Amiée Conn'\n",
       "4. 'Terry Walters'\n",
       "5. 'Thom Shelton'\n",
       "6. 'Cinda Adams'\n",
       "7. 'Callie Hernandez'\n",
       "8. 'Jessica Rothe'\n",
       "9. 'Sonoya Mizuno'\n",
       "10. 'Rosemarie DeWitt'\n",
       "11. 'J.K. Simmons'\n",
       "12. 'Claudine Claudio'\n",
       "13. 'Jason Fuchs'\n",
       "14. 'D.A. Wallach'\n",
       "15. 'Trevor Lissauer'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] \"Ryan Gosling\"     \"Emma Stone\"       \"Amiée Conn\"       \"Terry Walters\"   \n",
       " [5] \"Thom Shelton\"     \"Cinda Adams\"      \"Callie Hernandez\" \"Jessica Rothe\"   \n",
       " [9] \"Sonoya Mizuno\"    \"Rosemarie DeWitt\" \"J.K. Simmons\"     \"Claudine Claudio\"\n",
       "[13] \"Jason Fuchs\"      \"D.A. Wallach\"     \"Trevor Lissauer\" "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(rvest)\n",
    "\n",
    "url <- \"http://www.imdb.com/title/tt3783958/\"\n",
    "lalaland <- read_html(url)\n",
    "cast <- lalaland %>%\n",
    "    html_nodes(xpath = \"//td[@class='itemprop']/a/span[@class='itemprop']\") %>%\n",
    "    html_text()\n",
    "cast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 練習\n",
    "\n",
    "- 使用 XPath 選擇將 <https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e=tse> 的股票代號/名稱與成交價抓出來\n",
    "- 股票代號/名稱可能會有編碼的問題：\n",
    "    - `read_html(x, encoding = \"cp950\")`\n",
    "    - `iconv(from = \"UTF-8\", to = \"UTF-8\")`"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}