Get started with pins

The pins package helps you publish data sets, models, and other R objects, making it easy to share them across projects and with your colleagues. You can pin objects to a variety of “boards”, including local folders (to share on a networked drive or with dropbox), RStudio connect, Amazon S3, and more. This vignette will introduce you to the basics of pins.

library(pins)

Getting started

Every pin lives in a pin board, so you must start by creating a pin board. In this vignette I’ll use a temporary board which is automatically deleted when your R session is over:

board <- board_temp()

In real-life, you’d pick a board depending on how you want to share the data. Here are a few options:

board <- board_local() # share data across R sessions on the same computer
board <- board_folder("~/Dropbox") # share data with others using dropbox
board <- board_folder("Z:\\my-team\pins") # share data using a shared network drive
board <- board_rsconnect() # share data with RStudio Connect

Reading and writing data

Once you have a pin board, you can write data to it with pin_write():

mtcars <- tibble::as_tibble(mtcars)
board %>% pin_write(mtcars, "mtcars")
#> Guessing `type = 'rds'`
#> Creating new version '20120304T050607Z-2f9b5'
#> Writing to pin 'mtcars'

The first argument is the object to save (usually a data frame, but it can be any R object), and the second argument gives the “name” of the pin. The name is basically equivalent to a file name: you’ll use it when you later want to read the data from the pin. The only rule for a pin name is that it can’t contain slashes.

As you can see from the output, pins has chosen to save this data to an .rds file. But you can choose another option depending on your goals:

After you’ve pinned an object, you can read it back with pin_read():

board %>% pin_read("mtcars")
#> # A tibble: 32 × 11
#>      mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
#>    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#>  1  21       6  160    110  3.9   2.62  16.5     0     1     4     4
#>  2  21       6  160    110  3.9   2.88  17.0     0     1     4     4
#>  3  22.8     4  108     93  3.85  2.32  18.6     1     1     4     1
#>  4  21.4     6  258    110  3.08  3.22  19.4     1     0     3     1
#>  5  18.7     8  360    175  3.15  3.44  17.0     0     0     3     2
#>  6  18.1     6  225    105  2.76  3.46  20.2     1     0     3     1
#>  7  14.3     8  360    245  3.21  3.57  15.8     0     0     3     4
#>  8  24.4     4  147.    62  3.69  3.19  20       1     0     4     2
#>  9  22.8     4  141.    95  3.92  3.15  22.9     1     0     4     2
#> 10  19.2     6  168.   123  3.92  3.44  18.3     1     0     4     4
#> # … with 22 more rows

You don’t need to supply the file type when reading data from a pin because pins automatically stores the file type in the metadata, the topic of the next section.

Note that when the data lives elsewhere, pins takes care of downloading and caching so that it’s only re-downloaded when needed. That said, most boards transmit pins over HTTP, and this is going to be slow and possibly unreliable for very large pins. As a general rule of thumb, we don’t recommend using pins with files over 500 MB. If you find yourself routinely pinning data larger that this, you might need to reconsider your data engineering pipeline.

Metadata

Every pin is accompanied by some metadata that you can access with pin_meta():

board %>% pin_meta("mtcars")
#> List of 11
#>  $ file       : chr "mtcars.rds"
#>  $ file_size  : 'fs_bytes' int 3.12K
#>  $ pin_hash   : chr "2f9b568eb4365daf"
#>  $ type       : chr "rds"
#>  $ title      : chr "mtcars: a pinned 32 x 11 data frame"
#>  $ description: NULL
#>  $ created    : POSIXct[1:1], format: "2022-08-22 15:05:00"
#>  $ api_version: num 1
#>  $ user       : list()
#>  $ name       : chr "mtcars"
#>  $ local      :List of 3
#>   ..$ dir    : 'fs_path' chr "/var/folders/hv/hzsmmyk9393_m7q3nscx1slc0000gn/T/RtmpzUqxKB/pins-4a576740c8c0/mtcars/20120304T050607Z-2f9b5"
#>   ..$ url    : NULL
#>   ..$ version: chr "20120304T050607Z-2f9b5"

This shows you the metadata that’s generated by default. This includes:

When creating the pin, you can override the default description or provide additional metadata that is stored with the data:

board %>% pin_write(mtcars, 
  description = "Data extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models).",
  metadata = list(
    source = "Henderson and Velleman (1981), Building multiple regression models interactively. Biometrics, 37, 391–411."
  )
)
#> Using `name = 'mtcars'`
#> Guessing `type = 'rds'`
#> Replacing version '20120304T050607Z-2f9b5' with '20120304T050607Z-2f9b5'
#> Writing to pin 'mtcars'
board %>% pin_meta("mtcars")
#> List of 11
#>  $ file       : chr "mtcars.rds"
#>  $ file_size  : 'fs_bytes' int 3.12K
#>  $ pin_hash   : chr "2f9b568eb4365daf"
#>  $ type       : chr "rds"
#>  $ title      : chr "mtcars: a pinned 32 x 11 data frame"
#>  $ description: chr "Data extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobi"| __truncated__
#>  $ created    : POSIXct[1:1], format: "2022-08-22 15:05:00"
#>  $ api_version: num 1
#>  $ user       :List of 1
#>   ..$ source: chr "Henderson and Velleman (1981), Building multiple regression models interactively. Biometrics, 37, 391–411."
#>  $ name       : chr "mtcars"
#>  $ local      :List of 3
#>   ..$ dir    : 'fs_path' chr "/var/folders/hv/hzsmmyk9393_m7q3nscx1slc0000gn/T/RtmpzUqxKB/pins-4a576740c8c0/mtcars/20120304T050607Z-2f9b5"
#>   ..$ url    : NULL
#>   ..$ version: chr "20120304T050607Z-2f9b5"

While we’ll do our best to keep the automatically generated metadata consistent over time, I’d recommend manually capturing anything you really care about in metadata.

Versioning

In many situations it’s useful to version pins, so that writing to an existing pin does not replace the existing data, but instead adds a new copy. There are two ways to turn versioning on:

Most boards have versioning on by default. The primary exception is board_folder() since that stores data on your computer, and there’s no automated way to clean up the data your saving.

Once you have turned versioning on, every pin_write() will create a new version:

board2 <- board_temp(versioned = TRUE)

board2 %>% pin_write(1:5, name = "x", type = "rds")
#> Creating new version '20120304T050607Z-ab444'
#> Writing to pin 'x'
board2 %>% pin_write(2:6, name = "x", type = "rds")
#> Creating new version '20120304T050607Z-a077a'
#> Writing to pin 'x'
board2 %>% pin_write(3:7, name = "x", type = "rds")
#> Creating new version '20120304T050607Z-0a284'
#> Writing to pin 'x'

You can list all the available versions with pin_versions():

board2 %>% pin_versions("x")
#> # A tibble: 3 × 3
#>   version                created             hash 
#>   <chr>                  <dttm>              <chr>
#> 1 20120304T050607Z-0a284 2012-03-03 22:06:00 0a284
#> 2 20120304T050607Z-a077a 2012-03-03 22:06:00 a077a
#> 3 20120304T050607Z-ab444 2012-03-03 22:06:00 ab444

(Pins does not currently provide any tool to remove old versions, but this is likely to be implemented in the future.)

By default, pin_read() will return the most recent version:

board2 %>% pin_read("x")
#> [1] 1 2 3 4 5

But you can request an older version by supplying the version argument:

board2 %>% pin_read("x", version = "20210520T173110Z-49519")

Reading and writing files

So far we’ve focussed on pin_write() and pin_read() which work with R objects. pins also provides the lower-level pin_upload() and pin_download() which work with files on disk. You can use them to share types of data that are otherwise unsupported by pins.

pin_upload() works like pin_write() but instead of an R object you give it a vector of paths. I’ll start by creating a few files in the temp directory:

paths <- file.path(tempdir(), c("mtcars.csv", "alphabet.txt"))
write.csv(mtcars, paths[[1]])
writeLines(letters, paths[[2]])

Now I can upload those to the board:

board %>% pin_upload(paths, "example")
#> Creating new version '20120304T050607Z-e9d42'

pin_download() returns a vector of paths:

board %>% pin_download("example")
#> [1] "/var/folders/hv/hzsmmyk9393_m7q3nscx1slc0000gn/T/RtmpzUqxKB/pins-4a576740c8c0/example/20120304T050607Z-e9d42/mtcars.csv"  
#> [2] "/var/folders/hv/hzsmmyk9393_m7q3nscx1slc0000gn/T/RtmpzUqxKB/pins-4a576740c8c0/example/20120304T050607Z-e9d42/alphabet.txt"

It’s now your job to handle them. You should treat these paths as internal implementation details — never modify them and never save them for use outside of pins.

Note that you can’t pin_read() something you pinned with pin_upload():

board %>% pin_read("example")
#> Error in `object_read()`:
#> ! Pin does not declare file type so can't be automatically read
#> ℹ Retrieve uploaded paths with `pin_download()`

But you can pin_download() something that you’ve pinned with pin_write():

board %>% pin_download("mtcars")
#> [1] "/var/folders/hv/hzsmmyk9393_m7q3nscx1slc0000gn/T/RtmpzUqxKB/pins-4a576740c8c0/mtcars/20120304T050607Z-2f9b5/mtcars.rds"

Caching

The primary purpose of pins is to make it easy to share data. But pins is also designed to help you spend as little time as possible downloading data. pin_read() and pin_download() automatically cache remote pins: they maintain a local copy of the data (so it’s fast) but always check that it’s up-to-date (so your analysis doesn’t use stale data).

Wouldn’t it be nice if you could take advantage of this feature for any dataset on the internet? That’s the idea behind board_url() — you can assemble your own board from datasets, wherever they live on the internet. For example, this code creates a board containing a single pin, penguins, that refers to some fun data I found on GitHub:

my_data <- board_url(c(
  "penguins" = "https://raw.githubusercontent.com/allisonhorst/palmerpenguins/master/inst/extdata/penguins_raw.csv"
))

You can read this data by combining pin_download() with read.csv()1:

my_data %>%
  pin_download("penguins") %>% 
  read.csv(check.names = FALSE) %>% 
  tibble::as_tibble()
#> # A tibble: 344 × 17
#>    studyName Sampl…¹ Species Region Island Stage Indiv…² Clutc…³ Date …⁴ Culme…⁵
#>    <chr>       <int> <chr>   <chr>  <chr>  <chr> <chr>   <chr>   <chr>     <dbl>
#>  1 PAL0708         1 Adelie… Anvers Torge… Adul… N1A1    Yes     2007-1…    39.1
#>  2 PAL0708         2 Adelie… Anvers Torge… Adul… N1A2    Yes     2007-1…    39.5
#>  3 PAL0708         3 Adelie… Anvers Torge… Adul… N2A1    Yes     2007-1…    40.3
#>  4 PAL0708         4 Adelie… Anvers Torge… Adul… N2A2    Yes     2007-1…    NA  
#>  5 PAL0708         5 Adelie… Anvers Torge… Adul… N3A1    Yes     2007-1…    36.7
#>  6 PAL0708         6 Adelie… Anvers Torge… Adul… N3A2    Yes     2007-1…    39.3
#>  7 PAL0708         7 Adelie… Anvers Torge… Adul… N4A1    No      2007-1…    38.9
#>  8 PAL0708         8 Adelie… Anvers Torge… Adul… N4A2    No      2007-1…    39.2
#>  9 PAL0708         9 Adelie… Anvers Torge… Adul… N5A1    Yes     2007-1…    34.1
#> 10 PAL0708        10 Adelie… Anvers Torge… Adul… N5A2    Yes     2007-1…    42  
#> # … with 334 more rows, 7 more variables: `Culmen Depth (mm)` <dbl>,
#> #   `Flipper Length (mm)` <int>, `Body Mass (g)` <int>, Sex <chr>,
#> #   `Delta 15 N (o/oo)` <dbl>, `Delta 13 C (o/oo)` <dbl>, Comments <chr>, and
#> #   abbreviated variable names ¹​`Sample Number`, ²​`Individual ID`,
#> #   ³​`Clutch Completion`, ⁴​`Date Egg`, ⁵​`Culmen Length (mm)`

board_url() requires a bit of work compared to using download.file() or similar but it has a big payoff: the data will only be re-downloaded when it changes.


  1. Here I’m using read.csv() to the reduce the dependencies of the pins package. For real code I’d recommend using data.table::fread() or readr::read_csv().↩︎