Nested data

library(tidyr)
library(dplyr)
library(purrr)

Basics

A nested data frame is a data frame where one (or more) columns is a list of data frames. You can create simple nested data frames by hand:

df1 <- tibble(
  g = c(1, 2, 3),
  data = list(
    tibble(x = 1, y = 2),
    tibble(x = 4:5, y = 6:7),
    tibble(x = 10)
  )
)

df1
#> # A tibble: 3 × 2
#>       g data            
#>   <dbl> <list>          
#> 1     1 <tibble [1 × 2]>
#> 2     2 <tibble [2 × 2]>
#> 3     3 <tibble [1 × 1]>

(It is possible to create list-columns in regular data frames, not just in tibbles, but it’s considerably more work because the default behaviour of data.frame() is to treat lists as lists of columns.)

But more commonly you’ll create them with tidyr::nest():

df2 <- tribble(
  ~g, ~x, ~y,
   1,  1,  2,
   2,  4,  6,
   2,  5,  7,
   3, 10,  NA
)
df2 %>% nest(data = c(x, y))
#> # A tibble: 3 × 2
#>       g data            
#>   <dbl> <list>          
#> 1     1 <tibble [1 × 2]>
#> 2     2 <tibble [2 × 2]>
#> 3     3 <tibble [1 × 2]>

nest() specifies which variables should be nested inside; an alternative is to use dplyr::group_by() to describe which variables should be kept outside.

df2 %>% group_by(g) %>% nest()
#> # A tibble: 3 × 2
#> # Groups:   g [3]
#>       g data            
#>   <dbl> <list>          
#> 1     1 <tibble [1 × 2]>
#> 2     2 <tibble [2 × 2]>
#> 3     3 <tibble [1 × 2]>

I think nesting is easiest to understand in connection to grouped data: each row in the output corresponds to one group in the input. We’ll see shortly this is particularly convenient when you have other per-group objects.

The opposite of nest() is unnest(). You give it the name of a list-column containing data frames, and it row-binds the data frames together, repeating the outer columns the right number of times to line up.

df1 %>% unnest(data)
#> # A tibble: 4 × 3
#>       g     x     y
#>   <dbl> <dbl> <dbl>
#> 1     1     1     2
#> 2     2     4     6
#> 3     2     5     7
#> 4     3    10    NA

Nested data and models

Nested data is a great fit for problems where you have one of something for each group. A common place this arises is when you’re fitting multiple models.

mtcars_nested <- mtcars %>%
  group_by(cyl) %>%
  nest()

mtcars_nested
#> # A tibble: 3 × 2
#> # Groups:   cyl [3]
#>     cyl data              
#>   <dbl> <list>            
#> 1     6 <tibble [7 × 10]> 
#> 2     4 <tibble [11 × 10]>
#> 3     8 <tibble [14 × 10]>

Once you have a list of data frames, it’s very natural to produce a list of models:

mtcars_nested <- mtcars_nested %>%
  mutate(model = map(data, function(df) lm(mpg ~ wt, data = df)))
mtcars_nested
#> # A tibble: 3 × 3
#> # Groups:   cyl [3]
#>     cyl data               model 
#>   <dbl> <list>             <list>
#> 1     6 <tibble [7 × 10]>  <lm>  
#> 2     4 <tibble [11 × 10]> <lm>  
#> 3     8 <tibble [14 × 10]> <lm>

And then you could even produce a list of predictions:

mtcars_nested <- mtcars_nested %>%
  mutate(pred = map(model, predict))
mtcars_nested
#> # A tibble: 3 × 4
#> # Groups:   cyl [3]
#>     cyl data               model  pred      
#>   <dbl> <list>             <list> <list>    
#> 1     6 <tibble [7 × 10]>  <lm>   <dbl [7]> 
#> 2     4 <tibble [11 × 10]> <lm>   <dbl [11]>
#> 3     8 <tibble [14 × 10]> <lm>   <dbl [14]>

This workflow works particularly well in conjunction with broom, which makes it easy to turn models into tidy data frames which can then be unnest()ed to get back to flat data frames. You can see a bigger example in the broom and dplyr vignette.