We were recently building a Shiny App in which we had to load data from a very large dataframe. It was directly impacting the app initialization time, so we had to look into different ways of reading data from files to R (in this case the customer provided csv files) and identify the best one.
The goal of my post is to compare:
read.csv from utils, which was the standard way of reading csv files to R in RStudio,
read_csv from readr which replaced the former method as a standard way of doing it in RStudio,
load and readRDS from base, and
read_feather from feather and fread from data.table.
First, let’s generate some random data
and save the files on a disk to evaluate the load time. Besides the csv format we will also need feather, RDS and Rdata files.
Next, let’s check our files’ sizes:
As we can see both csv and feather format files take up much more storage space. Csv takes up 6 times and feather 4 more comparing to RDS and RData.
We will use microbenchmark library to compare the read times of the following methods:
in 10 rounds.
And the winner is… feather! However, using feather requires prior conversion of the file to the feather format.
Using load or readRDS can improve performance (second and third place in terms of speed) and has an added benefit of storing smaller/compressed file. In both cases you will first have to convert your file to the proper format.
When it comes to reading from csv format fread significantly beats read_csv and read.csv, and thus is the best option to read a csv file.
We decided to go with feather file since converting from csv to this format is just a one time job and we didn’t have a strict limitation on a storage space to consider usaging of Rds or RData format.
The final workflow was:
reading a csv file provided by our customer using fread,
writing it to feather using write_feather, and
loading a feather file on app initialization using read_feather.
The first two tasks were done once and outside of a Shiny App context.
There is also quite an interesting benchmark done by Hadley here on reading complete files to R. Unfortunately, if you use functions defined in that post, you will end up with a character type object and will have to apply string manipulations to obtain a commonly and widely used dataframe.