Tidyr: nest fails with space in column name

[klmr]

I’ve got the following table:

x = structure(list(GO = c("GO:0002119", "GO:0016246", "GO:0008340",
"GO:0002119", "GO:0002119"), Gene = c("WBGene00000041", "WBGene00000041",
"WBGene00000041", "WBGene00000063", "WBGene00000063"), `Gene name` =
c(rep("Probable aconitate hydratase, mitochondrial", 3), "Actin-1", "Actin-1")),
class = c("tbl_df", "tbl", "data.frame"), row.names = c(NA, -5L),
.Names = c("GO", "Gene", "Gene name"))

Performing a nest operation on this table will fail. Renaming the column with the space in the name will make the operation succeed:

> library(dplyr)
> library(tidyr)
> x %>% nest(GO)
Error in parse(text = x) : <text>:1:6: unexpected symbol
1: Gene name
         ^
> x %>% rename(GeneName = `Gene name`) %>% nest(GO)
# A tibble: 2 × 3
            Gene                                    GeneName             data
           <chr>                                       <chr>           <list>
1 WBGene00000041 Probable aconitate hydratase, mitochondrial <tibble [3 × 1]>
2 WBGene00000063                                     Actin-1 <tibble [2 × 1]>

I haven’t tested this extensively but I believe that this error isn’t specific to nest but rather more general. I can trace it to a call to as.lazy.character which, when given the input "Gene name", produces the error. However, I’m not sure whether it’s the implementation or the usage of this function which is incorrect. At any rate, the above should clearly work.

> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-apple-darwin15.5.0 (64-bit)
Running under: OS X 10.11.6 (El Capitan)

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] tidyr_0.6.0.9000 dplyr_0.5.0.9000

loaded via a namespace (and not attached):
[1] lazyeval_0.2.0.9000 magrittr_1.5        R6_2.1.3
[4] assertthat_0.1      tools_3.3.1         DBI_0.5-1
[7] tibble_1.2          Rcpp_0.12.7

[klmr]

In fact the error can be reduced to the following, showing that the actual bug is in dplyr or lazyeval, not tidyr:

x %>% select_(.dots = 'Gene name')

I’ll leave this issue open for now since I’m unsure where it belongs.

[zeehio]

As a workaround you could do this:

r x %>% select_(.dots = '`Gene name`')

Or more generally:

r x %>% select_(.dots = sprintf("`%s`", 'Gene name'))

You may think "Ok, I've got it". But then one day an evil column with a backtick will surprise you and your code will fail:

data(iris)
colnames(iris)[1] <- "evil`column" # MUAHAHA

And then you will realize that the proper way is through the as.name function:

r x %>% select_(.dots = as.name("Gene name')) iris %>% select_(.dots = as.name("evil`column"))

But then the evil column will call a friend (another evil column) and you will find out that as.name is not vectorized, so your as.name approach SILENTLY fails (only the first column is selected, this is madness):

data(iris)
colnames(iris)[1] <- "evil`column"
colnames(iris)[2] <- "more evil`column" # MUAHAHA
iris %>% select_(.dots = as.name(c("evil`column", "more evil`column")))

I have been unable to find an elegant solution. My current workaround renames the columns:

# Set up a problem:
data(iris)
colnames(iris)[1] <- "evil`column"
colnames(iris)[2] <- "more evil`column" # MUAHAHA

columns_to_select <- c("evil`column", "more evil`column")

# Rename columns to something safe:
original_colnames <- colnames(iris)
safe_colnames <- make.names(original_colnames, unique = TRUE)
safe_columns_to_select <- safe_colnames[original_colnames %in% columns_to_select]
colnames(iris) <- safe_colnames

# Select on a easy problem
output <- iris %>% select_(.dots = safe_columns_to_select)
# Rename columns:
colnames(output) <- columns_to_select 

This issue is related to https://github.com/hadley/lazyeval/issues/90

[zeehio]

@klmr your "not so happy face" ("confused face" according to github) made me sad, so I found an easy and elegant solution: :smiley:

# Set up a problem:
data(iris)
colnames(iris)[1] <- "evil`column"
colnames(iris)[2] <- "more evil`column" # MUAHAHA


dplyr::select_(iris, .dots = lapply(c("evil`column", "more evil`column"), as.name))

[krlmlr]

I have proposed a solution for this problem in dplyr, happy to take your feedback there: https://github.com/hadley/dplyr/pull/2386

[klmr]

(Wrong comment submitted accidentally)

In my limited understanding of the matter, this looks like a good fix. I wish it weren’t necessary to add one more layer but it seems as if there’s fundamentally no way in dplyr currently to distinguish between a name and an expression, stored as a character string. This will inevitably lead to confusion, and using different classes for them is indeed the way to go. I’d just prefer if expressions never went the way via character strings in the first place (but changing that would break backwards compatibility).

stringlytyping.

[krlmlr]

@klmr: Thanks. I feel it's more the other way round -- everything is converted to an expression by lazyeval, but the verbs treat them differently.

CC @hadley.

[klmr]

@krlmlr The trouble is that before things arrive at lazyeval, many of them are character strings but the information whether they originate from names, or from (user-supplied) character strings designating unparsed expressions, has vanished. Your fix works around that but it would be even better (in my view) if the API wouldn’t accept strings in place of expressions. That’s the “stringly typing” part, and it subverts (R’s already weak) type system.

[krlmlr]

Agreed that mutate_() et al. don't need to accept strings. But e.g. for select_() a string is the only way to specify a column that cannot be represented in the native encoding (relevant on Windows, see hadley/dplyr#1950 for detail). Not sure about group_by_(), it has "mutate" semantics but again it's difficult to group by a UTF-8 column in a non-UTF-8 locale. The proposed PR allows specifying different rules for each verb.

[hadley]

I think the next iteration of lazyeval (now called tidyeval) will continue to move further away from accepting strings. We're likely to continue accepting them for a while for backward compatibility, but I really want to move to richer data structures (and better accompanying helpers) in the future.

[krlmlr]

Will these data structures and helpers cover all of dplyr's use cases, so that https://github.com/hadley/dplyr/pull/2386 is not necessary?

[hadley]

@krlmlr yes, that's the goal

[hadley]

Fixed, thanks to tidy eval.