Reticulate: Better documentation of use_python, use_condaenv, use_miniconda

[myoung3]

It's not clear how conda_create() selects version number currently, but it seems like there's no way to specify it. The conda CLI to creating a new environment allows specification of the exact version of python. This means you need to resort to using the CLI if you want to control what version of python gets put into the environment.

Additionally, this documentation should make it more clear that if you're creating a conda environment, you want use_python to point to the python binary inside the conda environment. Results are ... unpredictable if you try to point to a python version that's not in the conda environment (i think the python version basically gets silently ignored). That a conda environment has its own python binary may be obvious to people who are familiar with conda environments, but it wasn't clear from the reticulate documentation on using conda. I think the fact that I couldn't specify a python version in conda_create() contributed to my confusion, so clearer documentationn around this would help others in the future.

EDIT: see additional suggestions for improving documentation in this comment:

[ericnewkirk]

I just ran into this as well - fyi 3.6 is used as a default in places in the initial setup. So if you use reticulate to install miniconda it will automatically create the r-reticulate environment for you and that environment will use python 3.6. I decided to leave that alone and create a separate environment with the version I needed like this:

Sys.setenv("RETICULATE_MINICONDA_PYTHON_VERSION" = "3.7")
conda_create("py_env")
use_condaenv("py_env", required = TRUE)

If you want to use the default environment but specify the python version I think this will do it, but you might have to reinstall modules etc.:

conda_remove("r-reticulate")
Sys.setenv("RETICULATE_MINICONDA_PYTHON_VERSION" = "3.7")
conda_create("r-reticulate")
use_condaenv("r-reticulate", required = TRUE)

Not suggesting this as a solution, just a short-term workaround. I only figured it out by digging around until I found where 3.6 was getting plugged in by default.

[myoung3]

Thanks @ericnewkirk
In the interest of sharing, here's my solution. It avoids relying on reticulate entirely for setup. Note that renv is basically broken for tracking python version and the contents of the environment in this context (maybe in any context that doesn't use renv's default virtual environment?), but I've described a workaround here

library(reticulate)  
library(here)
library(renv)
renv::init()

conda_env_name <- <desired environment name>
python_version <- <desired pythonn version>

if(conda_env_name %in% reticulate::conda_list()$name){
  stop("a conda environment named for the project already exists. I'd recommend deleting the conflicting environemnt first or using a different name")
}


#project-local miniconda installation
f <- here("renv","python","install.sh")
writeLines(
  c(
    "#!/bin/bash",
    paste0("cd ",here("renv","python")),
    "wget -q https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh",
    "bash miniconda.sh -b -p miniconda"
  ), 
  con=f)
Sys.chmod(f, "777")
system(f)

miniconda_loc <- here("renv","python","miniconda")
stopifnot(dir.exists(miniconda_loc))
miniconda_bin_loc <- file.path(miniconda_loc, "bin","conda")
stopifnot(file.exists(miniconda_bin_loc))

#create a conda environment in this miniconda installation
f2 <- here("renv","python","create_env.sh")
writeLines(
  c(
    "#!/bin/bash",
    paste0(miniconda_bin_loc," create -n ",conda_env_name," python=",python_version)
  ),
  con=f2
)

Sys.chmod(f2, "777")
system(f2)

##find the full filepath of the conda environment that was just created
clist <- conda_list()
conda_env_binary <- clist[clist$name==conda_env_name, "python"]
conda_env_directory <- dirname(dirname(conda_env_binary)) #drop /bin/python from the directory to get down to the environment path
stopifnot(length(conda_env_directory)==1L)


##specify the python binary that's inside the conda environment:
pythonloc <- file.path(conda_env_directory, "bin","python")

reticulate::use_python(pythonloc,required=TRUE)
reticulate::use_condaenv(conda_env_name, required=TRUE)

#also add the same information to a local .Renviron file since I have no idea what the reticulate::use_ functions even do...
#note that RETICULATE_PYTHON must point to a binary
#but RETICULATE_MINICONDA_PATH needs to point to a directory
writeLines(
  c(
    paste0("RETICULATE_PYTHON='",pythonloc,"'"),
    paste0("RETICULATE_PYTHON_ENV='",conda_env_name,"'"),
    paste0("RETICULATE_MINICONDA_PATH='",miniconda_loc,"'"),
    ""),
  con=".Renviron")

At this point, you can use reticulate to install packages into that environment as normal. e.g.:

conda_install(conda_env_name, "earthengine-api")

One warning here is that installing the intel ai toolkit will reinstall python in that environment

conda_install(conda_env_name, "intel-aikit-tensorflow", channel="intel")

In this situation, everything continues to work fine, but the version number of python in that environment might silently change. I had specified 3.8 and the default python version in that environment turned into 3.7 after installing the intel ai toolkit because there's no version 3.8 of python in the toolkit. Eventually I figured it out by closely reading the conda installation output.

[ericnewkirk]

Basically writing your own reticulate - nice! I just started with renv and I'm not using it for this project, but I can see how that would add some complications. Given my luck so far with R packages I can't even imagine trying to get it to play nice with python too.

I almost tossed this warning in the last post, but just in case you haven't stumbled across it yet:

use_condaenv("py_env")
conda_python() # just returns the first path in conda_list(), e.g. r-reticulate
py_config() # tells you which one you're actually currently using

Just a heads up as it seemed pretty counter-intuitive. I expected conda_python() to tell me which python binary I was using once I ran use_condaenv.

Also working with tensorflow, by the way. Still a long way from knowing if anything works, but setting up the environment wasn't as bad as I thought once I figured out a few of the quirks in reticulate.

[kevinushey]

It's not clear how conda_create() selects version number currently, but it seems like there's no way to specify it.

The development version of reticulate accepts a parameter python_version for specifying the Python version.

Results are ... unpredictable if you try to point to a python version that's not in the conda environment (i think the python version basically gets silently ignored).

Did you use reticulate::use_python(<pythonPath>, required = TRUE)? Otherwise the request is advisory and may be ignored. (I agree this is confusing and sub-optimal but that's the current behavior.)

[myoung3]

Thanks! Glad to hear this has already been taken care. Could you comment on the difference between calling reticulate::use_python(, require=FALSE), reticulate::use_python(, require=TRUE), versus setting RETICULATE_PYTHON in .Renviron? The documentation doesn't really say what use_python even does. Does it just set the environment variable? If so, what's the difference between required=TRUE and required=FALSE? If it doesn't set the environment variable, what does it do exactly? Is this a command you need to run every session, or does it persist between sessions? Documentation for these functions could be improved.

https://rstudio.github.io/reticulate/reference/use_python.html

[kevinushey]

Thanks; I've updated the documentation in the development version.

In short, in order of precedence: RETICULATE_PYTHON > use_python(required = TRUE) > use_python(required = FALSE).

use_python() just sets an internal list of "requested" Python installations to be used when Python bindings are initialized, and that list only lasts as long as that R session. It hence needs to be re-run on each session.

[myoung3]

Great, thanks @kevinushey! The help file on dev is much improved.

A couple more suggestions

• [ ] So from my reading of the documentation use_python is entirely ignored if RETICULATE_PYTHON is set. It might be useful for there to be a warning when calling use_python if Sys.getenv() detects a corresponding environment variable. (and similarly for the other use_ functions)
• [ ] It would be great if you could add similar description settings for the three other functions in this help file. At the very least, it would be good to indicate their corresponding environment variables and whether they follow the same order of precedence you just described for use_python.
• [ ] Also a brief sentence as to what an environment even does would be useful (given that some users of reticulate will be unfamiliar with the concept in the first place)
• [ ] It would also be good to document how use_virtualenv and use_condaenv interact (is there an order of precedence, or does whichever one was last called take precedence) and/or a recommendation to only specify one.
• [ ] It might also help to clarify that, when specifying use_condaenv, it makes the most sense to set use_python to an interpreter within that conda environment. (maybe I'm wrong on this point, and in that case some clarity is needed here but I'm not sure what the answer is).
• [ ] I don't know as much about virtual environments, but if there's an interpreter within each virtual environment then maybe the advice is the same for a conda environment.
• [ ] It was initially unclear to me what use_miniconda does. My understanding now is that this just specifies the conda installation that will be used when conda_install is called? Description of that would be great as well.
• [ ] Documentation as to whether each environment variable and/or argument to use_ should point to a binary (for the python functions) or a directory (for use_miniconda) https://github.com/rstudio/reticulate/issues/948
• [ ] whether environments should be specified by name or by directory location (or either?)

I know this is a lot, but having this information readily available will save new python users literally hours of work if they want to take greater control of their installation/environments on a system that already has python installed.

Thanks again for your work on the package. Being able to call python from R so easily is a huge accomplishment and I really respect that amount of work that has gone into this.