Dplyr: Incorrect scoping behavior (?) with mutate + map + enquos

[ashiklom]

Possibly related to #1212 and #3019.

Functions that rlang::enquos their arguments fail to correctly unquote their arguments when called with mutate.

Consider...

f <- function(...) {
  exprs <- rlang::enquos(...)
  exprs
}
names <- c("a", "b", "c")

This works as expected, unquoting .x into the values of names:

map(names, ~f(thing = !!.))
#> [[1]]
#> <list_of<quosure>>
#> 
#> $thing
#> <quosure>
#> expr: ^"a"
#> env:  empty
#> <...>

However, this fails when called via dplyr::mutate inside a tibble:

df <- tibble(n = names)
mutate(df, y = map(n, ~f(thing = !!.)))
#> Error in quos(..., .named = TRUE): object '.' not found
mutate(df, y = map(n, ~f(thing = !!.x)))
#> Error in quos(..., .named = TRUE): object '.x' not found

I encountered this in the wild when trying to programmatically construct a series of ggplot2::geom_* calls:

mutate(df, maps = map2(n, fun_col, ~stat_function(aes(color = !!.x), fun = .y)))

...which I realize is pretty far from conventional usage, and has perfectly reasonable workarounds...but I could definitely imagine this popping up in other functions that start with rlang::enquos(...).

---

Full reprex (and session info)

library(rlang)
library(purrr)
#> 
#> Attaching package: 'purrr'
#> The following objects are masked from 'package:rlang':
#> 
#>     %@%, as_function, flatten, flatten_chr, flatten_dbl,
#>     flatten_int, flatten_lgl, flatten_raw, invoke, list_along,
#>     modify, prepend, splice
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

f <- function(...) {
  exprs <- rlang::enquos(...)
  exprs
}

names <- c("a", "b", "c")

# This works
map(names, ~f(thing = !!.))
#> [[1]]
#> <list_of<quosure>>
#> 
#> $thing
#> <quosure>
#> expr: ^"a"
#> env:  empty
#> 
#> 
#> [[2]]
#> <list_of<quosure>>
#> 
#> $thing
#> <quosure>
#> expr: ^"b"
#> env:  empty
#> 
#> 
#> [[3]]
#> <list_of<quosure>>
#> 
#> $thing
#> <quosure>
#> expr: ^"c"
#> env:  empty
map(names, ~f(thing = !!.x))
#> [[1]]
#> <list_of<quosure>>
#> 
#> $thing
#> <quosure>
#> expr: ^"a"
#> env:  empty
#> 
#> 
#> [[2]]
#> <list_of<quosure>>
#> 
#> $thing
#> <quosure>
#> expr: ^"b"
#> env:  empty
#> 
#> 
#> [[3]]
#> <list_of<quosure>>
#> 
#> $thing
#> <quosure>
#> expr: ^"c"
#> env:  empty

# But this does not
df <- tibble(n = names)
mutate(df, y = map(n, ~f(thing = !!.)))
#> Error in quos(..., .named = TRUE): object '.' not found
mutate(df, y = map(n, ~f(thing = !!.x)))
#> Error in quos(..., .named = TRUE): object '.x' not found

devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.0 (2019-04-26)
#>  os       macOS High Sierra 10.13.6   
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2019-05-01                  
#> 
#> ─ Packages ──────────────────────────────────────────────────────────────
#>  package     * version    date       lib source                          
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 3.6.0)                  
#>  backports     1.1.4      2019-04-10 [1] CRAN (R 3.6.0)                  
#>  callr         3.2.0      2019-03-15 [1] CRAN (R 3.6.0)                  
#>  cli           1.1.0      2019-03-19 [1] CRAN (R 3.6.0)                  
#>  crayon        1.3.4      2017-09-16 [1] CRAN (R 3.6.0)                  
#>  desc          1.2.0      2018-05-01 [1] CRAN (R 3.6.0)                  
#>  devtools      2.0.2      2019-04-08 [1] CRAN (R 3.6.0)                  
#>  digest        0.6.18     2018-10-10 [1] CRAN (R 3.6.0)                  
#>  dplyr       * 0.8.0.9014 2019-05-01 [1] Github (tidyverse/dplyr@b8281c3)
#>  evaluate      0.13       2019-02-12 [1] CRAN (R 3.6.0)                  
#>  fs            1.2.7      2019-03-19 [1] CRAN (R 3.6.0)                  
#>  glue          1.3.1      2019-03-12 [1] CRAN (R 3.6.0)                  
#>  highr         0.8        2019-03-20 [1] CRAN (R 3.6.0)                  
#>  htmltools     0.3.6      2017-04-28 [1] CRAN (R 3.6.0)                  
#>  knitr         1.22       2019-03-08 [1] CRAN (R 3.6.0)                  
#>  magrittr      1.5        2014-11-22 [1] CRAN (R 3.6.0)                  
#>  memoise       1.1.0      2017-04-21 [1] CRAN (R 3.6.0)                  
#>  pillar        1.3.1      2018-12-15 [1] CRAN (R 3.6.0)                  
#>  pkgbuild      1.0.3      2019-03-20 [1] CRAN (R 3.6.0)                  
#>  pkgconfig     2.0.2      2018-08-16 [1] CRAN (R 3.6.0)                  
#>  pkgload       1.0.2      2018-10-29 [1] CRAN (R 3.6.0)                  
#>  prettyunits   1.0.2      2015-07-13 [1] CRAN (R 3.6.0)                  
#>  processx      3.3.0      2019-03-10 [1] CRAN (R 3.6.0)                  
#>  ps            1.3.0      2018-12-21 [1] CRAN (R 3.6.0)                  
#>  purrr       * 0.3.2      2019-03-15 [1] CRAN (R 3.6.0)                  
#>  R6            2.4.0      2019-02-14 [1] CRAN (R 3.6.0)                  
#>  Rcpp          1.0.1      2019-03-17 [1] CRAN (R 3.6.0)                  
#>  remotes       2.0.4      2019-04-10 [1] CRAN (R 3.6.0)                  
#>  rlang       * 0.3.4.9003 2019-05-01 [1] Github (r-lib/rlang@6a232c0)    
#>  rmarkdown     1.12       2019-03-14 [1] CRAN (R 3.6.0)                  
#>  rprojroot     1.3-2      2018-01-03 [1] CRAN (R 3.6.0)                  
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.6.0)                  
#>  stringi       1.4.3      2019-03-12 [1] CRAN (R 3.6.0)                  
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 3.6.0)                  
#>  testthat      2.1.1      2019-04-23 [1] CRAN (R 3.6.0)                  
#>  tibble        2.1.1      2019-03-16 [1] CRAN (R 3.6.0)                  
#>  tidyselect    0.2.5      2018-10-11 [1] CRAN (R 3.6.0)                  
#>  usethis       1.5.0      2019-04-07 [1] CRAN (R 3.6.0)                  
#>  withr         2.1.2      2018-03-15 [1] CRAN (R 3.6.0)                  
#>  xfun          0.6        2019-04-02 [1] CRAN (R 3.6.0)                  
#>  yaml          2.2.0      2018-07-25 [1] CRAN (R 3.6.0)                  
#> 
#> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

^{Created on 2019-05-01 by the reprex package (v0.2.1)}

[ashiklom]

One possibly helpful observation is that the following doesn't error, but actually treats . as referring to the input tibble rather than the column:

df %>%
  mutate(y = map(n, ~f(thing = !!.))) %>%
  pull(y)

# [[1]]
# <list_of<quosure>>
# 
# $thing
# <quosure>
# expr: ^<tibble>
# env:  empty

Not surprisingly, calling this with .x instead of . throws the object '.x' not found error.

[krlmlr]

This is because !! is already processed by the mutate() call and never reaches f(). Note how f() is called only once here (as expected), yet the returned quosure already has the value of x unquoted.

f <- function(...) {
  exprs <- rlang::enquos(...)
  message("f() is called")
  exprs
}

x <- 1

f(f(~ !!x))
#> f() is called
#> <list_of<quosure>>
#> 
#> [[1]]
#> <quosure>
#> expr: ^f(~1)
#> env:  global

^{Created on 2019-05-11 by the reprex package (v0.2.1)}

@lionel-: Can unquoting perhaps raise a warning when moving inside a ~ or a function() definition?

[lionel-]

Can unquoting perhaps raise a warning when moving inside a ~ or a function() definition?

If we were to make a change of this type, I think ~ and function would protect against unquoting. This would solve issues of nested quasiquotation like https://stackoverflow.com/questions/56026143/r-rlang-use-x-in-map-with-quosure/56054094#56054094

This would be a highly breaking change.

[lionel-]

oh I just remember I wondered last week about having that behaviour for enquo() / enquos() but not expr(). This way we're still able to meta-program formulas and function definitions. Not sure whether the inconsistency would be confusing in practice. This should be less of a breaking change.

[moodymudskipper]

It would be really cool to be able to protect against unquoting,

I've been mixing base substitution and rlang a lot lately because I can't find a good workaround to handle nested unquoting. I would love to be able to follow official guidance for this for clearer code and vision.

What about using one or more unary + before the !! to add "layers of protection" ? It's also a breaking change but who uses unary +, especially next to !! ? Protecting with {{..}} with 2 or more { would have worked too but I believe you have other plans for it. ? would work as well, with less convenient precedence.

df %>%
  mutate(y = map(n, ~f(thing = +!!.))) %>%
  pull(y)

df %>%
  mutate(y = map(n, ~f(thing = ?!!.))) %>%
  pull(y)

[lionel-]

You can protect with !!quote()

mtcars %>% mutate(n = !!quote(map(n, ~ f(thing, !!.))))

[romainfrancois]

I think we can close this now.

[lock[bot]]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/