```
HTML has a hierarchical structure formed by **elements** which consist of a start tag (e.g. `` (heading 1), `
` (paragraph), and `
- ` (ordered list) form the overall structure of the page.
- Inline tags like `` (bold), `` (italics), and `` (links) formats text inside block tags.
If you encounter a tag that you've never seen before, you can find out what it does with a little googling.
I recommend the [MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/HTML) which are produced by Mozilla, the company that makes the Firefox web browser.
### Contents
Most elements can have content in between their start and end tags.
This content can either be text or more elements.
For example, the following HTML contains paragraph of text, with one word in bold.
```{=html}
- This is a bulleted list
- apple & pear
- banana
- pineapple
- C-3PO is a droid that weighs 167 kg
- R2-D2 is a droid that weighs 96 kg
- Yoda weighs 66 kg
- R4-P17 is a droid
Hi! My name is Hadley.
``` The **children** of a node refers only to elements, so the `` element above has one child, the `` element.
The `` element has no children, but it does have contents (the text "name").
Some elements, like `` can't have children.
These elements depend solely on attributes for their behavior.
### Attributes
Tags can have named **attributes** which look like `name1='value1' name2='value2'`.
Two of the most important attributes are `id` and `class`, which are used in conjunction with CSS (Cascading Style Sheets) to control the visual appearance of the page.
These are often useful when scraping data off a page.
## Reading HTML with rvest
You'll usually start the scraping process with `read_html()`.
This returns a `xml_document`[^2] object which you'll then manipulate using rvest functions:
[^2]: This class comes from the [xml2](https://xml2.r-lib.org) package.
xml2 is a low-level package that rvest builds on top of.
```{r}
html <- read_html("http://rvest.tidyverse.org/")
class(html)
```
For examples and experimentation, rvest also includes a function that lets you create an `xml_document` from literal HTML:
```{r}
html <- minimal_html("
This is a paragraph
` elements. - `.title`: selects all elements with `class` "title". - `p.special`: selects all `
` elements with `class` "special". - `#title`: selects the element with the `id` attribute that equals "title". Id attributes must be unique within a document, so this will only ever select a single element. If you want to learn more CSS selectors I recommend starting with the fun [CSS dinner](https://flukeout.github.io/) tutorial and then referring to the [MDN web docs](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors). Lets try out the most important selectors with a simple example: ```{r} html <- minimal_html("
This is a heading
This is a paragraph
This is an important paragraph
") ``` In rvest you can extract a single element with `html_element()` or all matching elements with `html_elements()`. Both functions take a document[^3] and a css selector: [^3]: Or another element, more on that shortly. ```{r} html %>% html_element("h1") html %>% html_elements("p") html %>% html_elements(".important") html %>% html_elements("#first") ``` Selectors can also be combined in various ways using **combinators**. For example,The most important combinator is " ", the **descendant** combination, because `p a` selects all `` elements that are a child of a `` element. If you don't know exactly what selector you need, I highly recommend using [SelectorGadget](https://rvest.tidyverse.org/articles/selectorgadget.html), which lets you automatically generate the selector you need by supplying positive and negative examples in the browser. ## Extracting data Now that you've got the elements you care about, you'll need to get data out of them. You'll usually get the data from either the text contents or an attribute. But, sometimes (if you're lucky!), the data you need will be in an HTML table. ### Text Use `html_text2()` to extract the plain text contents of an HTML element: ```{r} html <- minimal_html("
This is a paragraph.
This is another paragraph. It has two sentences.
") ``` `html_text2()` gives you what you expect: two paragraphs of text separated by a blank line. ```{r} html %>% html_element("body") %>% html_text2() %>% cat() ``` Whereas `html_text()` returns the garbled raw underlying text: ```{r} html %>% html_element("body") %>% html_text() %>% cat() ``` ### Attributes Attributes are used to record the destination of links (the `href` attribute of `` elements) and the source of images (the `src` attribute of the `| ` (table heading), and ` | ` (table data).
Here's a simple HTML table with two columns and three rows:
```{r}
html <- minimal_html("
|
|---|