Jon Harmon's Blog

Contact

Sat, 29 Jun 2019 00:00:00 +0000

Email

Work email: jon.harmon@macmillan.com
Personal email: jonthegeek@gmail.com

Publications

“Measuring Microlearning in an Online Learning Environment” Jon Harmon and Rasil Warnakulasooriya. In: The 12th International Conference on Educational Data Mining, Michel Desmarais, Collin F. Lynch, Agathe Merceron, & Roger Nkambou (eds.) 2019, pp. 738 - 741

Introducing {factory}: Automate Your Automation

Sat, 11 May 2019 00:00:00 +0000

Function factories are, in my opinion, one of the cooler concepts in programming. A function factory is a function that produces a function. A straightforward example function factory (from Advanced R (second edition) by Hadley Wickham) is a power function factory, which can be used to produce functions that raise their input to a specific power. You’d probably never actually use this exact factory, but it gets the general idea across:

power1 <- function(exp) {
  function(x) {
    x ^ exp
  }
}

square <- power1(2)
cube <- power1(3)

square(3)

## [1] 9

cube(3)

## [1] 27

However, there’s a reason that particular factory is named power1 in Advanced R. That factory is fragile:

y <- 2
square <- power1(y)
y <- 3
square(2)

## [1] 8

Since y changed before we called square, square didn’t work how we expected it to work. You can get around this by forceing the input to evaluate:

power2 <- function(exp) {
  force(exp)
  function(x) {
    x ^ exp
  }
}

y <- 2
square <- power2(y)
y <- 3
square(2)

## [1] 4

That requirement makes function factories a bit more complicated to code. For an extreme example, see this pull request to the scales package, which took almost two years to get everything nailed down.

Even when you remember to force evaluation of variables inside the factory, the function produced by a factory isn’t quite what you might expect it to be:

square

## function(x) {
##     x ^ exp
##   }
## <environment: 0x00000000161e58a0>

Looking at that code, you can’t really tell what the function does. What is exp? How does the function know that it’s 2? To figure that out, you need to dig into the environment that’s displayed when you print square. It would be nicer if the output of a factory was a normal function that users could then adapt and work with like any other function.

The {rlang} package solves this problem by introducing the function new_function (as described in Advanced R by Hadley Wickham (2nd Edition), 19.7.4: Creating functions):

power3 <- function(exp) {
  rlang::new_function(
    rlang::exprs(x = ), 
    rlang::expr({
      x ^ !!exp
    }), 
    rlang::caller_env()
  )
}

y <- 2
square <- power3(y)
y <- 3
square(2)

## [1] 4

square

## function (x) 
## {
##     x^2
## }

Now the resulting function looks like any other function… but we’ve just moved the complexity from the factory user to the factory author.

My goal with {factory} is to make factory creation easy for both the author and user.

library(factory)
power4 <- build_factory(
  fun = function(x) {
    x^exp
  },
  exp
  # For the time being, you need to tell factory which arguments 
  # belong to the factory.
)

x <- 2
square <- power4(x)
x <- 3
square(2)

## [1] 4

square

## function (x) 
## {
##     x^2
## }

The function is as easy to write as in power1, and the resulting function is a “normal” function as in power3.

With help from Colin Fay, I’m currently working on adding an Rstudio addin to {factory}, to allow you to select some code (an instance of what you expect as output from a factory) and convert it into a factory. I also plan to add an addin to make it easy to use a factory while creating a package (a whole other can of worms), but that will wait for a future blog post.

Do you have any function factory use cases? Let me know here, or, more importantly, in a {factory} issue!

library(memer)
meme_get("YoDawg") %>% 
  meme_text_top(
    txt = toupper("Yo dawg I herd you like function factories"), 
    size = 24
  ) %>%  
  meme_text_bottom(
    txt = toupper(
      paste(
        "so we put a function factory in a function factory",
        "so you can automate while you automate",
        sep = "\n"
      )
    ), 
    size = 22
  )

Writing Custom Tidyverse Functions

Mon, 04 Jun 2018 00:00:00 +0000

dplyr and the other members of the tidyverse are fantastic for making exploratory data analysis quick and easy. However, that ease-of-use comes with a price: programming with the tidyverse has a steep learning curve. Let’s see if we can climb that hill together.

Packages used

We’ll need three packages for this tutorial: dplyr, rlang, and purrr. We’ll be making custom versions of a couple functions in dplyr, and we’ll use rlang (the package behind “tidy evaluation,” the way the tidyverse “thinks”) to make those changes. We’ll also use purrr::map toward the end. I’ll explicitly call out most function calls with dplyr:: or rlang:: to make it clear where each function comes from. There are four notable exceptions. The first is the pipe, %>%, which I assume you’ve seen before if you’re reading this post (if not, check out the vignette in the magrittr package). I’ll explain the other three when I get to them.

library(dplyr) # v0.7.5
library(rlang) # v0.2.1
library(purrr) # v0.2.5

I’d also like to thank Miles McBain for his excellent friendlyeval package, which helped me understand tidy evaluation faster and better.

Fake student data

I’m going to work with some fake student data, logging how students interacted with 10 questions on one of four assignments, in which the students can keep making attempts on each question until they’re scored as correct, request the solution, or move on to another question. In addition to a student_id, assignment_id, and question_number, each row specifies whether or not the student eventually reached the correct answer, whether they viewed a hint (viewed_hint), whether they gave up and requested the solution (requested_solution), and how many attempts they made on the question. If you’d like to generate the same data to play along, see the code at the end of this post.

What we’ll do

We’ll start off with a simple function that doesn’t require any tidy evaluation, to make it clear that you don’t always need to know anything special to program with tidyverse functions.
We’ll then learn how we can pass a “bare” column name into a function and get dplyr’s verbs to use it how we expect it to be used.
We’ll extend that to send in multiple column names.
We’ll see how we can pass around functions of column names.
We’ll see how we can get dplyr to treat column names as strings when we want them.
We’ll learn about the new assignment operator :=, and when we need it.

Nothing fancy: add_cfa_col

The simplest thing I want to do is to add a column indicating whether the student got the correct answer on their first attempt (“cfa”). Note: This isn’t the first thing I’d normally do, but it’s by far the simplest, so we’ll get it out of the way. I use this value a lot in my investigations, so it’ll be nice to be able to add that column quickly and easily. This isn’t that hard to do with a simple dplyr::mutate:

student_assignment_data %>% 
  dplyr::mutate(cfa = correct == 1L & attempts == 1L)

However, I do this all the time. I’d like to be able to add it by calling a custom function. Let’s call that function add_cfa_col. If we assume my column names are always the same (or I can get them there before calling the function), this can be a simple function that doesn’t rely on any tidy evaluation magic from rlang. I wanted to include this example to show you that it isn’t always hard to program with the tidyverse.

add_cfa_col <- function(.data) {
  dplyr::mutate(.data, cfa = correct == 1L & attempts == 1L)
}

student_assignment_data %>% 
  add_cfa_col()

ensym and !!: summarize_student_performance

The next-easiest thing I want to do with this data is to summarize how the students performed. The variables I include in the summary are going to be the same each time, but I want to be able to easily change what I group by: student_id, assignment_id, or question_number. Let’s take a look at the dplyr code to do what I want, then see what it will take to turn it into a function.

student_assignment_data %>% 
  add_cfa_col() %>% 
  # question_number will be passed via a parameter in the function.
  dplyr::group_by(question_number) %>% 
  dplyr::summarize(
    mean_correct = mean(correct), 
    mean_viewed_hint = mean(viewed_hint), 
    mean_requested_solution = mean(requested_solution)
  )

In my function, I want to pass in a parameter telling the function how to group the data. I’ll call this parameter group_col. The problem is, group_by already expects a “bare” symbol naming that column; if I tell it to group_by(group_col), it will look for a column named “group_col” in the data, fail to find that column, and throw an error. I need to tell it to translate the parameter I send in to the name of a column. To do this, we need two functions from rlang: ensym and !! (pronounced “bang-bang”). First we tell our function “think of this parameter as a symbol” using the ensym function, rlang::ensym(group_col). Then we’ll tell group_by to process the code we gave it rather than looking for a column named “rlang::ensym(group_col)” with !!, !!rlang::ensym(group_col). I think of !! as telling group_by “It’s not group_col, but it’s also not not group_col.”

With those two functions in place, we get this:

summarize_student_performance <- function(.data, group_col) {
  .data %>% 
    dplyr::group_by(!!rlang::ensym(group_col)) %>% 
    dplyr::summarize(
    mean_correct = mean(correct), 
    mean_viewed_hint = mean(viewed_hint), 
    mean_requested_solution = mean(requested_solution)
  )
}

student_assignment_data %>% 
  add_cfa_col() %>% 
  summarize_student_performance(question_number)

ensyms and !!!: summarize_student_performance with multiple grouping parameters

Looking at that data, what I’d really like to do is group by both question_number and assignment_id, so I can see how students performed question-by-question on each assignment. Let’s update summarize_student_performance to accept any number of grouping columns. Instead of accepting .data and a single parameter, our function will accept .data and ..., which is used in R functions to indicate a list of parameters. Since we might have more than one parameter to look at, we use the rlang function ensyms rather than ensym, and !!! (pronounced “bang-bang-bang” with formal meaning “ungroup and splice,” but just remember that !!! is connected to ...). Let’s see how that looks:

summarize_student_performance <- function(.data, ...) {
  .data %>% 
    dplyr::group_by(!!!rlang::ensyms(...)) %>% 
    dplyr::summarize(
    mean_correct = mean(correct), 
    mean_viewed_hint = mean(viewed_hint), 
    mean_requested_solution = mean(requested_solution)
  )
}

student_assignment_data %>% 
  add_cfa_col() %>% 
  summarize_student_performance(assignment_id, question_number)

enquo(s): summarize_student_performance grouped by functions of parameters

What if I want to divide my students into two groups, one if their student_id is even, another if their student_id is odd? group_by allows me to pass in a function of column names, from which it will derive a grouping. For example, dplyr::group_by(student_id %% 2) will create two groups with values 0 and 1. I can also name that group with dplyr::group_by(student_group = student_id %% 2), to make my output make more sense. Let’s adapt our function to take advantage of these options.

At this point, we’re no longer safe to assume that the parameters are meant to be treated as symbols. Moreover, it’s possible the user of our function will redefine the function they’re using in the group_by, so we need to make sure we evaluate their expression in their environment, rather than our function’s environment. To send in an expression and bring the user’s environment along for the ride, we use rlang::enquo (or rlang::enquos for ... or a list of parameters). Let’s see how that works.

summarize_student_performance <- function(.data, ...) {
  .data %>% 
    dplyr::group_by(!!!rlang::enquos(...)) %>% 
    dplyr::summarize(
    mean_correct = mean(correct), 
    mean_viewed_hint = mean(viewed_hint), 
    mean_requested_solution = mean(requested_solution)
  )
}

student_assignment_data %>% 
  add_cfa_col() %>% 
  summarize_student_performance(student_group = student_id %% 2)

quo_name: summarize_mean

Oops! I meant to include mean_cfa in these tables, but I missed it! I already have three copies of that same code, so it’s time to consider turning it into a function. Let’s add a helper function for our function, building the summarize call by passing in a list of variables for which we want to find the mean.

I want to add “mean_” to the front of each variable I pass into that function, so I’ll need to both treat the parameter as a symbol (the column for which I’ll find the mean) and a character string (the thing which will be appended to “mean_”). I can get a string representing the name of the parameter using rlang::quo_name. This would have probably been easier to demonstrate if I were trying to do something simpler, but I couldn’t think of a good, easy example, so I’ll try to walk through this code slowly.

summarize_mean <- function(.data, ...) {
  # Capture the dots into quos.
  summarize_vars <- enquos(...)
  
  # Capture the names of the dots by applying rlang::quo_name 
  # to each member of summarize_vars.
  names_of_vars <- purrr::map(summarize_vars, rlang::quo_name)
  
  # Name the list of quosures generated above.
  names(summarize_vars) <- paste0("mean_", names_of_vars)
  
  # Use dplyr::summarize_at to summarize those columns using mean.
  dplyr::summarize_at(.data, dplyr::vars(!!! summarize_vars), mean)
}

# Now call that function in the summarize_student_performance function.
summarize_student_performance <- function(.data, ...) {
  .data %>% 
    dplyr::group_by(!!!rlang::enquos(...)) %>% 
    summarize_mean(correct, viewed_hint, requested_solution, cfa)
}

student_assignment_data %>% 
  add_cfa_col() %>% 
  summarize_student_performance(student_group = student_id %% 2)

:=, a new assignment operator: mutate_logical

I mentioned above that there was something I’d like to do at the very beginning, but it was a bit complicated to explain. We’re ready to tackle that now. I’d like to convert the integer columns correct, viewed_hint, and requested_solution to logical (TRUE/FALSE) values. I can do this with three mutates:

student_assignment_data %>% 
  dplyr::mutate(
    correct = as.logical(correct), 
    viewed_hint = as.logical(viewed_hint), 
    requested_solution = as.logical(requested_solution)
  )

I could make that somewhat simpler using dplyr::mutate_at, but then it turns into almost exactly the same problem as above, and we don’t learn anything new. Instead we’ll make a function that takes .data and a single input column, and converts that column to logical. To do this, we’re going to need to use a new assignment operator, := (“colon-equals”, but I think of it as “digest then assign”). The normal argument assignment operator, =, only parses stuff on the right-hand-side. It assumes anything to the left of it is fine as-is. However, when we’re coding with tidy evaluation, that isn’t always the case. := is otherwise exactly the same as =, though, so its use is pretty straightforward. I’m also going to redefine (and simplify) my add_cfa_col function, since now the correct column will be logical. We’ll leave the summarize off this time, so we can make sure our new function works. Note: This doesn’t actually do anything useful in this case, but I like to make sure that my column types mean what I want them to mean.

mutate_logical <- function(.data, logical_col) {
  logical_col <- rlang::ensym(logical_col)
  dplyr::mutate(.data, !!logical_col := as.logical(!!logical_col))
}

# correct will now be TRUE/FALSE, so we don't have to test 
# that it's equal to 1L.
add_cfa_col <- function(.data) {
  dplyr::mutate(.data, cfa = correct & attempts == 1L)
}

student_assignment_data %>% 
  mutate_logical(correct) %>% 
  mutate_logical(viewed_hint) %>% 
  mutate_logical(requested_solution) %>% 
  add_cfa_col()

What we learned

There’s (quite a bit) more available in rlang, but these functions should cover you for a large proportion of programming with the tidyverse:

ensym tells tidyverse functions to think of a parameter as a symbol (and errors if we pass in something other than a string or a bare symbol).
!! (“bang-bang”) tells tidyverse functions to process the thing we’re giving it to get down to the bare column names they expect.
ensyms is like ensym, but works for a list of parameters (including ...).
!!! is like !!, but works for a list of parameters (!!! goes with ...).
enquo and enquos bring the environment of the parameter along for the ride, to let us work with expressions.
quo_name gives us the (character) name of something we’ve enquo’ed.
:= (“colon-equals” or “digest then assign”) lets us pass something complex (such as !!rlang:enquo(my_var)) on the left-hand-side of an assignment.

Did I make something over-complicated, or miss something important? Let me know in the comments!

Generating the data

This code will generate the data I used in this post.

set.seed(123)
student_assignment_data <- tibble(
  student_id = rep(1L:10L, 10), 
  assignment_id = rep(sample(1L:4L, 10, replace = TRUE), 10), 
  question_number = rep(1L:10L, each = 10), 
  correct = sample(0L:1L, 100, replace = TRUE), 
  viewed_hint = sample(0L:1L, 100, replace = TRUE),
  requested_solution = sample(0L:1L, 100, replace = TRUE),
  attempts = sample(1L:10L, 100, replace = TRUE, prob = c(0.2, 0.5, 0.2, rep(0.1/7, 7)))
)

The Ed.gov College Scorecard

Thu, 29 Mar 2018 00:00:00 +0000

Today I’m exploring a dataset and trying to get it into a more usable format. This post is a prequel to a video/talk for the R4Ds Online Learning Community, in which I turn the data I explore in this post into a simple data package (and explain why and how to do that).

Packages

Whenever I use a function from a package in this post, I call out the package explicitly with package::function syntax. Still, it’s friendly to let you know what you should install to play along. We’ll mostly use dplyr, but there are a couple calls to map functions from purrr, and we’ll use tidyr::fill to make some things easier to use. We’ll use readr and readxl to get the data into our session, and we’ll use stringr once or twice to clean up some text. We’ll use DT::datatable to make the tables a bit more browsable. We’ll also use rlang for some tidy evaluation (specifically !! and :=). If you aren’t familiar with tidy evaluation, this video by Hadley Wickham is an excellent introduction.

library(dplyr)
library(DT)
library(purrr)
library(readr)
library(readxl)
library(rlang)
library(stringr)
library(tidyr)

The Data

The College Scorecard is freely available from the US government (licensed under Creative Commons CCZero, meaning it is fully public domain). I’ve downloaded the full raw data locally and unzipped it. We’re going to focus our exploration on the most recent data (2015-2016).

college_scorecard_2015_2016 <- readr::read_csv(
  "CollegeScorecard_Raw_Data/MERGED2015_16_PP.csv",
  na = c("", "NA", "NULL")
)

The data includes 1805 observations (aka columns or features) about colleges and universities in the United States (7593 institutions as of 2015-2016). However, the column names can be fairly opaque (for example, “HCM2” means “under_investigation” or “heightened cash monitoring 2”, where “The Department places institutions on a Heightened Cash Monitoring (HCM) payment method to provide additional oversight of cash management. HCM2 is the type of HCM that indicates more serious financial or federal compliance issues. These data are maintained by FSA.”)

To make the data more useful, let’s rename the columns, make sure they’re typed as expected, and translate encoded values into more meaningful values. To help us out, we’ll use the data dictionary. I’ve downloaded it locally so we don’t have to deal with parsing data from the web for this project.

The Data Dictionary

The provided data dictionary is a pretty good guide to making the data more useful.

data_dictionary <- readxl::read_xlsx(
  "CollegeScorecardDataDictionary.xlsx", 
  sheet = "data_dictionary"
)
DT::datatable(head(data_dictionary), options = list(pageLength = 3))

From a quick glance through the dictionary, we can see that two columns are mostly empty: VALUE and LABEL. These columns describe sub-dictionaries for certain columns, where a given value in that column has an expanded meaning. We’ll come back to those in a bit, and use them to build a couple more tibbles of data.

The columns that are most immediately useful to us, though, are “developer-friendly name”, “API data type”, and “VARIABLE NAME”. Let’s use those to clean up the data.

column_definitions <- data_dictionary %>% 
  dplyr::select(
    ugly_column_name = `VARIABLE NAME`, 
    better_column_name = `developer-friendly name`, 
    data_type = `API data type`
  ) %>% 
  dplyr::filter(ugly_column_name != "")
DT::datatable(head(column_definitions))

Applying the Dictionary

Right away, we can see that the “data_type” column is going to need some translation (what is an “autocomplete” data type?), but it should get us closer. We’ll use this information to walk through the given data, fixing the columns. Note that the dictionary has 4 “VARIABLE NAMEs” that have lowercase characters, but the column names are all uppercase. There are also 9 columns that aren’t defined in this part of the data dictionary, so we’ll ignore those columns for now; I’ll be happy to have the other 1796 columns cleaned. While we’re at it, we’ll remove any columns that don’t vary; there are historical columns sticking around in the dataset that do not currently have any information, so let’s get rid of those.

column_definitions <- column_definitions %>%
  dplyr::mutate(ugly_column_name = stringr::str_to_upper(ugly_column_name)) %>% 
  dplyr::filter(ugly_column_name %in% colnames(college_scorecard_2015_2016)) %>% 
  dplyr::mutate(data_type = dplyr::recode(
    data_type, 
    integer = "integer",
    autocomplete = "character", 
    string = "character", 
    float = "double"
  ))
defined_columns <- dplyr::intersect(
  colnames(college_scorecard_2015_2016), 
  column_definitions$ugly_column_name
)

latest_college_scorecard <- purrr::map_dfc(defined_columns, function(this_column) {
  this_column_contents <- college_scorecard_2015_2016[[this_column]]
  # If this column has no variability, let's get rid of it.
  this_column_variability <- unique(this_column_contents)
  if(length(this_column_variability) == 1) {
    NULL
  } else {
    definition <- column_definitions %>% 
      dplyr::filter(ugly_column_name == this_column)
    data_type <- definition$data_type[[1]]
    suppressWarnings(class(this_column_contents) <- data_type)
    better_name <- definition$better_column_name[[1]]
    dplyr::tibble(
      !! better_name := this_column_contents
    )
  }
})
DT::datatable(head(latest_college_scorecard, 3))

That’s already much better. We now have 622 columns with mostly meaningful names and proper types.

Factor Columns

Now let’s look into those “VALUE” and “LABEL” columns from the data dictionary. Everything in the VALUE column is an integer, which means they’re pretty close to an index of a factor. Interesting. Watch how it lets us clean up the “degrees_awarded.predominant” column (which just contains seemingly random integers above), for example.

sub_dictionaries <- data_dictionary %>% 
  dplyr::filter(!is.na(VALUE)) %>% 
  dplyr::select(better_column_name = `developer-friendly name`, current_value = VALUE, target_value = LABEL) %>% 
  tidyr::fill(better_column_name, .direction = "down")

factor_columns <- unique(sub_dictionaries$better_column_name)
latest_college_scorecard_factored <- purrr::map_dfc(colnames(latest_college_scorecard), function(this_column) {
  if(this_column %in% factor_columns) {
    # Translate this to a factor.
    this_dictionary <- sub_dictionaries %>%
      dplyr::filter(better_column_name == this_column) %>%
      dplyr::rename(!! this_column := current_value) %>% 
      dplyr::select(-better_column_name)
    new_column <- latest_college_scorecard[this_column] %>%
      dplyr::left_join(this_dictionary) %>%
      dplyr::select(target_value) %>% 
      dplyr::mutate(target_value = as.factor(target_value))
    names(new_column) <- this_column
    new_column
  } else {
    latest_college_scorecard[this_column]
  }
})
DT::datatable(head(latest_college_scorecard_factored, 50))

And with that, I think we have a much more usable set of data!

In this video, I pull this into a simple data package and add some additional documentation, to make sure future me remembers what the columns mean.

Blogging in About Ten Minutes

Tue, 27 Feb 2018 00:00:00 +0000

Last week, I set up my blog. There were a number of excellent how-tos out there, but I still managed to take several hours to get it up and running. With this guide, you should have a blog posted in (about) five minutes.

Assumptions

You have RStudio installed and are at least vaguely familiar with how to use it.
You’re at least a little familar with R Markdown (or can read a cheat sheet)
You have admin access to the machine you want to use to blog (or can get someone to work with you to knock these steps out).
You’re over 13 and can otherwise legally sign up for a Netlify account (you don’t have to do that yet, though; we’ll get there shortly).

Enable Github in Rstudio

Create a Github account.
Install git on whatever machine you plan to use.
Enable git in RStudio (Tools > Global Options > Git/SVN: Make sure your executable is configured).

Install Blogdown

Blogdown is a fantastic package designed to make it easy to publish an R blog. There is an entire book, but you shouldn’t get sidetracked reading it and then spending an hour thinking about some side project like I did. Come back to the book after you’re up and running.

install.packages("blogdown")

Create a Github Repository

Create a new repository on Github. You’ll use this to keep track of changes to your blog, and so Netlify can see when you have something new. The name of this repo doesn’t matter, but “blog” is likely fine (unless you have multiple blogs). Make it public, and initialize it with a README. Once it’s created, copy the url from “Clone or download.”

Create an RStudio Project

In RStudio, choose File > New Project. Choose Version Control, then Git, then paste the url of your git repository. Everything else will auto-fill. Make sure it looks right, then click Create Project.

Create a New Site with Blogdown

We’re going to use the blogdown package a bit now, so we’ll library that, and use it to create a new site. We’ll also need to install hugo (the thing that stitches your files together into HTML, CSS, etc), but blogdown also makes that step easy. You’ll need admin access here.

library(blogdown)
install_hugo()
new_site()

Note: We are simply using the default template. I spent two hours digging through templates before deciding the default was at least good enough, and better than a lot of the ones I considered. If you must, you can take a side trek here to dig into templates, but from here on out I assume you chose the default.

Update the config File with Minimal Tweaks

You should now have a bunch of files, including one in the root directory of your project called config.toml. Open that file, and make these tweaks:

Name your site under title. This doesn’t have to be fancy. Mine’s called “Jon Harmon’s Blog.”
Either update the part that says https://github.com/rstudio/blogdown to your own github url, or remove that section.
Ditto for the https://twitter.com/rstudio section.
Update the description = "A website built through Hugo and blogdown." section (although I haven’t actually noticed this impacting anything yet, so maybe come back for this later).

Save that file. Keep it open, though; we’re going to fix the baseurl as soon as you have one.

Update about.md

Navigate to the content folder, then open about.md. Update the date to make it less of a lie (although that date doesn’t really impact anything), and slap something in there about yourself. You can come back to this later, so don’t stress if it isn’t perfect.

Clean Out the Post Folder

Navigate to the post folder inside content. Delete everything.

Make Sure Your Site is Built

In RStudio, click Addins (directly above your editor), then Serve Site. That’ll make sure everything you’ve done is updated in the HTML version of your site.

Commit and Push Everything

Note: You might need to set up your git info at this point to stop RStudio from bugging you about things. Fortunately, RStudio will tell you what you need to do if you try to commit and it doesn’t work, so just pay attention to the message.

Go to the Git tab in RStudio, check everything, and click Commit. Enter a commit message, along the lines of “Set up all of the basics of my blog!” Click Commit, and read what it says; it might tell you something to configure. Once that’s working properly, click “Push.”

Set Up on Netlify

Go to Netlify, and either create a new account or log into an existing account.
Click New site from Git.
Click GitHub, and follow the instructions to connect to your repo (likely still in your clipboard, but you can hop over to github and grab it if you need it). I can’t remember exactly what this looked like the first time, but it was straightforward, and gets even more straightforward if you try to set up a second Netlify site.
Set the Build command to hugo_0.19 . You’re likely using a newer version of hugo, but, at least at some point in the past, Netlify only supported through 0.19 without some fancy work, and we aren’t doing anything fancy here. Later you can read documentation and tweak this if things don’t work.
Set the Publish directory to public . This is the directory that blogdown populates with the actual files for your site.
Click Deploy site.

Netlify will assign you a random url. You almost definitely want to change this. Click Domain Settings, then the ... next to your random blog URL. If you choose something vaguely unique (such as your name), chances are it’ll be available. This is the step that you might want to pause and think about a little, but remember that it doesn’t have to be amazing. Good enough is fine for a blog url.

Tweak Your config.toml

You now have a URL! Go back to config.toml in RStudio, and update the baseurl to your netlify url (such as “https://jonthegeek-blog-demo.netlify.com/”)

Note: Your blog should now be up and running! It doesn’t have content, but it should work!

Create a Post

Go back to Addins in RStudio, and choose New Post under Blogdown. Give your post a title, some keywords, your name, etc. Choose .Rmd for the Format (this isn’t technically required, but it’s my personal preference).

This step… will take more than ten minutes, I assume. If you really want, you can always put up a “Hello world” post, and later delete it, but probably take some time here and write something real. I am choosing not to include this in the ten minutes because otherwise my title won’t be catchy.

Push All the Things

Go back to your Git tab, commit everything, and push back up to your repo. You should now have a blog! The sample I made while writing this post is currently up at jonthegeek-blog-demo.netlify.com.

Clean Up

After you get that satisfying first blog actually posted, there are some things you have left to clean up.

Create a logo! There’s a file in /themes/hugo-lithium-theme/static/images called logo.png. Right now it’s a circle that says “Li” (for lithium, the name of the theme). You should probably replace that with something more unique. I used an old photo of me. Whatever you choose, make it 50 pixels by 50 pixels (or tweak those settings in config.toml), overwrite this file, rebuild your site, and push everything up to github.
The quick-creation of the new site left some garbage behind. Get rid of /static/post/2015-07-23-r-markdown_files. Theoretically that should nuke its corresponding folder in /public once the site rebuilds, but it isn’t doing so for me, so I also killed it in /public/post/. There’s also some other noise in the /public folder. You can nuke the 2015 and 2016 folders. There’s also some noise in /public/tags/ that you can get rid of (anything that isn’t a tag on your post), as well as /public/categories/r. If you use those keywords or categories, I suspect hugo would see these and clean them up, but I’ve found no harm in cleaning them out manually. Again, you don’t have to do this, but I felt the need to clean out noise once I got through the sprint.

Be sure to push all the things again when you’re done cleaning. Netlify will see the changes, and automatically republish your site.

Let Everyone Know!

Tweet about your new blog! Post in the comments here about your new blog! Tell your friends about your new blog! If it’s an R blog, let me know here or on twitter, and I’ll take a look and let people know you’re up and running. The R community is super friendly, so you’ll likely have some eyes—and comments—in no time!

About

Mon, 19 Feb 2018 14:33:00 -0600

I’m a Data Scientist at Macmillan Learning. I’ve spent years gathering and analyzing our data, but I’m working to formalize our processes and jump into some more advanced techniques.

Disclaimer: This blog does not represent the opinions of my employer, Macmillan Learning. All views expressed are my own.

Internal Packages for Common Data Manipulations

Mon, 19 Feb 2018 00:00:00 +0000

Last week, I replied to a tweet by Jesse Maegan about what I do day to day as a working data scientist. Her reply set me on the path that (finally) led to this blog.

that sounds awesome! I've never thought of creating a package to perform data cleaning - but that makes total sense. did you have any favorite resources when you were learning to build packages?
— Jesse Mostipak (@kierisi) February 12, 2018

Today I’ll walk through how and why to build an R package for data analysis. I’ll cover some of the best practices my coworkers and I stumbled through when we first started this process.

When should you write a data manipulation package?

Writing an R package is probably easier than you think, so don’t be intimidated! However, there is some overhead involved in writing a package, so you want to check a couple things:

Will the data source be reused?

In the excellent book R for Data Science by Garrett Grolemund and Hadley Wickham, they establish a simple rule for writing a function:

“You should consider writing a function whenever you’ve copied and pasted a block of code more than twice (i.e. you now have three copies of the same code).”

Let’s adapt that to the following: If you’re connecting to a data source for a third time, you should consider writing a data manipulation package.

Do you have to manipulate the incoming data in a consistent way?

When I load data, the first thing I generally do is make sure all of the variables (columns) that I’m going to work with are typed properly. Dates might come in as strings that need to be parsed, or maybe something that obviously should be an integer is treated as a character string. The data might also have an odd way to indicate missing values, so I need to make sure I deal with those. All of these tasks are strong indicators that a data manipulation package could be helpful, ensuring that the data always comes into my analysis the same way.

Does the data change?

If you’re loading the same data repeatedly, and you have to clean it up in a consistent way, you still probably don’t need a data manipulation package if the data doesn’t change. Instead, simply save the data using save(object, file = "filename.RData") or saveRDS(object, file = "filename.rds"), and then load(file = "filename.Rdata") or new_var <- readRDS(file = "filename.rds") in the next analysis.

How should you write a data manipulation package?

If your data passes those rules, it’s time to write a package! Here are some things to do to make sure you’ll be successful.

Step 1: Read the book.

If you’ve never written an R package (or even if you have), a good first step is to read R packages by Hadley Wickham. It’s available for free online, covers all of the basics about how to get started quickly, and only takes an hour or two to read. You don’t have to absorb everything the first time through, but it was very helpful to read through it when I first got started with package development to wrap my head around how the process works—and how devtools and RStudio make it super easy to write a package. I’ll mention the relevant parts as they come up in the rest of this post, but I recommend digging into the book deeper once you begin building a package.

Step 2: Rewrite your analysis to use functions.

Before you convert your data manipulation into a package, first open up your previous analysis and rewrite any code that reads your data to use functions. For example, if you have code that looks like this:

library(dplyr)
library(readr)
library(stringr)
year_of_interest <- 1977
unemployment <- read_tsv(file = 'https://download.bls.gov/pub/time.series/ln/ln.data.1.AllData') %>% 
  filter(
    year == year_of_interest
    , str_detect(period, 'M')
  ) %>% 
  mutate(
    month = as.integer(str_extract(period, '(?<=M)\\d+'))
    , series_type = str_extract(series_id, '...')
    , series_id = as.integer(str_extract(series_id, '(?<=...)\\d+'))
  ) %>% 
  select(year, month, series_type, series_id, value)
head(unemployment)

## # A tibble: 6 x 5
##    year month series_type series_id  value
##   <int> <int> <chr>           <int>  <int>
## 1  1977     1 LNS          10000000 157688
## 2  1977     2 LNS          10000000 157913
## 3  1977     3 LNS          10000000 158131
## 4  1977     4 LNS          10000000 158371
## 5  1977     5 LNS          10000000 158657
## 6  1977     6 LNS          10000000 158928

Let’s pull that into a function. While we’re at it, we’ll explicitly declare the packages for everything other than the pipe. This will be useful when we’re converting this into a package.

Edit per a comment by Mikko Marttila: Before you turn everything into functions, be sure to clear your workspace! You can do this in RStudio with the broom icon on the Environment tab. I didn’t when I put this example together, and it led to me writing a function that doesn’t actualy work! I’ve updated the function here. It now uses !! (“bang bang”), which makes it somewhat more confusing. You can get a quick walkthrough of !! from Hadley Wickham on his YouTube channel (highly recommended), but the short version is, in this instance, you need to let dplyr::filter know that the second “year” is the variable that’s literally “year,” not the column called “year” in the incoming data. You could avoid the !! by changing the name of the variable in the function, but I find it helpful to stick with standard names throughout the package whenever possible.

get_unemployment <- function(year) {
  readr::read_tsv(file = 'https://download.bls.gov/pub/time.series/ln/ln.data.1.AllData') %>% 
  dplyr::filter(
    year == !!year # Edited per comment by Mikko Marttila
    , stringr::str_detect(period, 'M')
  ) %>% 
  dplyr::mutate(
    month = as.integer(stringr::str_extract(period, '(?<=M)\\d+'))
    , series_type = stringr::str_extract(series_id, '...')
    , series_id = as.integer(stringr::str_extract(series_id, '(?<=...)\\d+'))
  ) %>% 
  dplyr::select(year, month, series_type, series_id, value)
}

year_of_interest <- 1977
unemployment <- get_unemployment(year_of_interest)
head(unemployment)

## # A tibble: 6 x 5
##    year month series_type series_id  value
##   <int> <int> <chr>           <int>  <int>
## 1  1977     1 LNS          10000000 157688
## 2  1977     2 LNS          10000000 157913
## 3  1977     3 LNS          10000000 158131
## 4  1977     4 LNS          10000000 158371
## 5  1977     5 LNS          10000000 158657
## 6  1977     6 LNS          10000000 158928

Repeat this process anywhere you load data in your family of analyses. Reuse functions when possible, refactoring them to take additional arguments when necessary. During this process, you will likely need to refactor a few times to settle on the set of functions that makes sense for your data, and to settle on a naming convention for your functions.

Step 3: Create your package.

RStudio makes creating a package relatively painless. If you haven’t worked with projects before, this is a great time to start (and then continue, because it keeps things nice and organized). You can create a project under File > New Project. Choose “New Directory”, then “R Package”. Give your package a name. R packages has naming tips, but the most important thing here is to choose something that will clearly tell you what sort of data this package manipulates the next time you need to use the package. For example, if you’re creating a package to download and prep the labor statistics from the US Bureau of Labor Statistics, you might call your package “labordata”.

Be sure to follow the guidelines in R packages for documenting your package. This will save you headaches in a month or a year when you can’t remember why you split up the functions in that particular way.

You can likely use the expected data from your existing analyses to create test cases for your package. This will be helpful as you adapt the package for future analyses, to ensure that your past work still works.

If you’re using the tidyverse, it will be very useful to include the magrittr pipe operator in your package. To do this, create an R file in your package’s R directory called “reexort_pipe.R”, with a version of this code:

#' Pipe data
#'
#' Like dplyr, this package allows you to use the pipe function, \code{\%>\%},
#' to turn function composition into a series of imperative statements.
#'
#' @importFrom magrittr %>%
#' @name %>%
#' @rdname pipe
#' @export
#' @param lhs,rhs A vector of fields or a tibble of fields and values, and a
#'   function to apply to them
NULL

Edit per a comment by Matthias Gomolka: Also add “magrittr” to the Imports section of your DESCRIPTION file, since the pipe needs to be imported from there before your package can export it.

Step 4: Update your analysis to use your package.

Switching your analysis from your inline functions to the package should be straightforward. Be sure to do this step! This will allow you to make sure your package does what you expect it to do. You may also find additional tweaks to make to your package.

When I’m doing this, I generally have the package project open in RStudio, but open my analysis file in the same session. As I make changes to the analysis, I can update tests for the package, refactor code, rebuild the package, and make sure that the analysis still produces the expected results.

Step 5: Continue to update your package.

The first iteration of your package most likely won’t catch all of the cases that you need. When you write your first post-package analysis, you will likely find tweaks to make. Switching between projects in RStudio will allow you to quickly update your package as needed. I highly recommend implementing version control for your package by this point if you haven’t done so already.

Collaborate.

A data manipulation package can make your analyses more efficient, but this process becomes even more useful if you’re working with a team. Each time someone does something with the data source, they can use your package. If you work with a shared version control system (such as github), you can learn to make pull requests to help each other expand the package.

The first time you move your data manipulations into a package, it will likely slow down that particular analysis. However, as you work with the same data more and more, you can cut out one of the more annoying steps of analysis, leaving more time for the fun things.