-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathREADME.Rmd
106 lines (76 loc) · 3.42 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
---
output: rmarkdown::github_document
---
[](https://travis-ci.org/hrbrmstr/wayback)
[](https://codecov.io/gh/hrbrmstr/wayback)
[](https://ci.appveyor.com/project/hrbrmstr/wayback/branch/master)
# wayback
Tools to Work with Internet Archive Wayback Machine APIs
## Description
The 'Internet Archive' provides access to millions of cached sites. Methods are provided to access these cached resources through the 'APIs' provided by the 'Internet Archive' and also content from 'MementoWeb'.
## What's Inside the Tin?
The following functions are implemented:
**Memento-ish API**:
- `archive_available`: Does the Internet Archive have a URL cached?
- `cdx_basic_query`: Perform a basic/limited Internet Archive CDX resource query for a URL
- `get_mementos`: Retrieve site mementos from the Internet Archive
- `get_timemap`: Retrieve a timemap for a URL
- `read_memento`: Read a resource directly from the Time Travel MementoWeb
- `is_memento`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
- `is_first_memento`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
- `is_next_memento`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
- `is_prev_memento`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
- `is_last_memento`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
- `is_original`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
- `is_timemap`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
- `is_timegate`: Various memento-type testers (useful in `purrr` or `dplyr` contexts)
**Scrape API**
- `ia_retrieve:` Retrieve directory listings for Internet Archive objects by identifier
- `ia_scrape`: Internet Archive Scraping API Access
- `ia_scrape_has_more`: 'ia_scrape()' Pagination Helpers
- `ia_scrape_next_page`: Internet Archive Scraping API Access
## Installation
```{r eval=FALSE}
devtools::install_github("hrbrmstr/wayback")
```
```{r message=FALSE, warning=FALSE, error=FALSE, echo=FALSE}
options(width=120)
```
## Usage
```{r message=FALSE, warning=FALSE, error=FALSE}
library(wayback)
library(tidyverse)
# current verison
packageVersion("wayback")
```
### Memento-ish things
```{r avail, message=FALSE, warning=FALSE, error=FALSE}
archive_available("https://www.r-project.org/news.html")
```
```{r get_memento, message=FALSE, warning=FALSE, error=FALSE}
get_mementos("https://www.r-project.org/news.html")
```
```{r get_time, message=FALSE, warning=FALSE, error=FALSE}
get_timemap("https://www.r-project.org/news.html")
```
```{r basic_q, message=FALSE, warning=FALSE, error=FALSE}
cdx_basic_query("https://www.r-project.org/news.html", limit = 10) %>%
glimpse()
```
```{r read_mem, message=FALSE, warning=FALSE, error=FALSE}
mem <- read_memento("https://www.r-project.org/news.html")
res <- stringi::stri_split_lines(mem)[[1]]
cat(paste0(res[187:200], collaspe="\n"))
```
### Scrape API
```{r}
glimpse(
ia_scrape("lemon curry")
)
```
```{r}
(nasa <- ia_scrape("collection:nasa", count=100L))
(item <- ia_retrieve(nasa$identifier[1]))
download.file(item$link[1], file.path("man/figures", item$file[1]))
```
