Conda-build: cran_mirror needs a default value (match skeleton default)

[krinsman]

Would it be possible to replace the mran.microsoft.com default CRAN mirror used by conda skeleton with another default mirror?

The mran.microsoft.com mirror often leads to HTTP 404 Not Found errors when building from the recipes generated by conda skeleton. It is usually possible to fix the error by mucking around with the url field in the recipe manually.

However, since the same fix works every time, it seems that it might be better to automate the fix into the conda skeleton source code.

I could make a pull request if you want; it is difficult for me to find which part of the source code sets the Jinja variables for meta.yaml, but if you show me where in the source it is, I could type up an initial proposal. (I don't know if this would save you time or not; I am only proposing it in case it saves you time; if not, feel free to ignore it.)

Actual Behavior

First run conda skeleton cran listenv. Everything works. Then run conda build r-listenv:

conda build r-listenv


Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for r-listenv
INFO:conda_build.metadata:Attempting to finalize metadata for r-listenv
Solving environment: ...working... done
Solving environment: ...working... done
BUILD START: ['r-listenv-0.7.0-r343h889e2dd_0.tar.bz2']
Solving environment: ...working... done

## Package Plan ##

  environment location: /opt/anaconda/anaconda3/conda-bld/r-listenv_1525267941937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_


The following NEW packages will be INSTALLED:

    _r-mutex:        1.0.0-anacondar_1  
    bwidget:         1.9.11-0           
    bzip2:           1.0.6-hd86a083_4   
    ca-certificates: 2018.03.07-0       
    cairo:           1.14.12-he60d9cb_2 
    cctools:         895-h7512d6f_0     
    clang:           4.0.1-h662ec87_0   
    clang_osx-64:    4.0.1-h1ce6c1d_10  
    clangxx:         4.0.1-hc9b4283_0   
    clangxx_osx-64:  4.0.1-h22b1bf0_10  
    compiler-rt:     4.0.1-h5487866_0   
    curl:            7.59.0-ha441bb4_0  
    fontconfig:      2.12.6-h9850fc5_0  
    freetype:        2.8-h12048fb_1     
    gettext:         0.19.8.1-h15daf44_3
    gfortran_osx-64: 4.8.5-h22b1bf0_3   
    glib:            2.56.1-h35bc53a_0  
    graphite2:       1.3.11-h337f25e_1  
    gsl:             2.4-h3efe00b_3     
    harfbuzz:        1.7.6-h171471c_1   
    icu:             58.2-h4b95b61_1    
    jpeg:            9b-he5867d9_2      
    krb5:            1.14.2-h9a779f2_6  
    ld64:            274.2-h7c2db76_0   
    libcurl:         7.59.0-hf30b1f0_0  
    libcxx:          4.0.1-h579ed51_0   
    libcxxabi:       4.0.1-hebd6815_0   
    libffi:          3.2.1-h475c297_4   
    libgfortran:     3.0.1-h93005f0_2   
    libiconv:        1.15-hdd342a3_7    
    libpng:          1.6.34-he12f830_0  
    libssh2:         1.8.0-h322a93b_4   
    libtiff:         4.0.9-h0dac147_0   
    libxml2:         2.9.8-h2866646_0   
    llvm:            4.0.1-hc748206_0   
    llvm-lto-tapi:   4.0.1-h6701bc3_0   
    llvm-openmp:     4.0.1-hda82c8b_0   
    ncurses:         6.0-hd04f020_2     
    openssl:         1.0.2o-h26aff7b_0  
    pango:           1.41.0-h3bae10e_0  
    pcre:            8.42-h378b8a2_0    
    pixman:          0.34.0-hca0a616_3  
    r-base:          3.4.3-h9b62496_0   
    readline:        7.0-hc1231fa_4     
    tk:              8.6.7-h35a86e2_3   
    tktable:         2.10-h1de35cc_0    
    xz:              5.2.3-h727817e_4   
    zlib:            1.2.11-hf3cbc9b_2  

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Solving environment: ...working... done
Source cache directory is: /opt/anaconda/anaconda3/conda-bld/src_cache
Downloading source to cache: listenv_0.7.0_6126020b11.tar.gz
Downloading https://mran.microsoft.com/snapshot/2018-01-01/src/contrib/listenv_0.7.0.tar.gz
Error: HTTP 404 NOT FOUND for url <https://mran.microsoft.com/snapshot/2018-01-01/src/contrib/listenv_0.7.0.tar.gz>
Elapsed: 00:00.612596

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
Downloading https://mran.microsoft.com/snapshot/2018-01-01/src/contrib/Archive/listenv/listenv_0.7.0.tar.gz
Error: HTTP 404 NOT FOUND for url <https://mran.microsoft.com/snapshot/2018-01-01/src/contrib/Archive/listenv/listenv_0.7.0.tar.gz>
Elapsed: 00:00.337422

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
Could not download https://mran.microsoft.com/snapshot/2018-01-01/src/contrib/Archive/listenv/listenv_0.7.0.tar.gz

Expected Behavior

This has happened with many other packages; I chose listenv as a minimal working example because it has very few dependencies.

Anyway, if one replaces the original url field in the meta.yaml file with:

  url:
    - https://cran.r-project.org/src/contrib/listenv_0.7.0.tar.gz

everything works.

I got that URL using the same manual fix that has worked for other packages, namely I went to the CRAN website of that package, went to the "Package Source" field, and then right-clicked on the file name and selected "Copy link address".

This manual fix works, but slows down the process, and I imagine that it would be possible to automatically implement a version of the fix in the conda-build source. (Because it was somewhat a matter of happenstance that I even thought to try this anyway; many users would have given up after getting the mran.microsoft.com error.)

Steps to Reproduce

conda skeleton cran listenv
conda build r-listenv

Output of conda info

conda info

     active environment : test
    active env location : /opt/anaconda/anaconda3/envs/test
            shell level : 2
       user config file : /Users/krinsman/.condarc
 populated config files : 
          conda version : 4.5.0
    conda-build version : 3.8.0
         python version : 3.6.4.final.0
       base environment : /opt/anaconda/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/osx-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /opt/anaconda/anaconda3/pkgs
                          /Users/krinsman/.conda/pkgs
       envs directories : /opt/anaconda/anaconda3/envs
                          /Users/krinsman/.conda/envs
               platform : osx-64
             user-agent : conda/4.5.0 requests/2.18.4 CPython/3.6.4 Darwin/15.6.0 OSX/10.11.6
                UID:GID : 504:20
             netrc file : None
           offline mode : False

[msarahan]

update conda-build. This is already done.

[krinsman]

You are entirely correct, after updating conda build, I did not get the same error.

A different error occurred after updating -- can you clarify if I am using conda skeleton correctly? Should I have specified the source URL when calling conda skeleton?

conda skeleton cran globals
conda build r-globals

Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for r-globals
INFO:conda_build.metadata:Attempting to finalize metadata for r-globals
Undefined Jinja2 variables remain (['cran_mirror', 'cran_mirror']).  Please enable source downloading and try again.

To confirm that I did update conda build, here is the output of conda info:

   active environment : test
    active env location : /opt/anaconda/anaconda3/envs/test
            shell level : 2
       user config file : /Users/krinsman/.condarc
 populated config files : /Users/krinsman/.condarc
          conda version : 4.5.2
    conda-build version : 3.10.1
         python version : 3.6.4.final.0
       base environment : /opt/anaconda/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/osx-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /opt/anaconda/anaconda3/pkgs
                          /Users/krinsman/.conda/pkgs
       envs directories : /opt/anaconda/anaconda3/envs
                          /Users/krinsman/.conda/envs
               platform : osx-64
             user-agent : conda/4.5.2 requests/2.18.4 CPython/3.6.4 Darwin/15.6.0 OSX/10.11.6
                UID:GID : 504:20
             netrc file : None
           offline mode : False

[msarahan]

When you build the package, conda-build is expecting you to have a cran_mirror key set. I think we can set a default there to match the skeleton's default. This is templated to allow people to easily swap between cran and mran.

See https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml#L37 for how we set it.

[msarahan]

I'm going to reopen this and change the title to "cran_mirror needs a default value"

[krinsman]

For a short-term fix, for the end-user, is it possible to set the default using conda config somehow? Sorry for the stupid questions.

Also as an update, the same hacky "fix" works with globals that worked with listenv (right-click "Copy Link Address" on the CRAN website).

[msarahan]

Yeah, that's the config file I pointed you to. You should probably read up on CB3 at https://conda.io/docs/user-guide/tasks/build-packages/variants.html

Especially the config file section at https://conda.io/docs/user-guide/tasks/build-packages/variants.html#creating-conda-build-variant-config-files

create a conda_build_config.yaml file, and put

cran_mirror:
  - https://cran.r-project.org/

in it. Follow the rules in the docs I pointed you to to make sure that conda-build picks up the file. You'll see something like this in your build output:

Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /Users/msarahan/code/AnacondaRecipesSync/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /Users/msarahan/code/AnacondaRecipesSync/conda_build_config.yaml

If you don't see the path to your config file, conda-build did not find it, and you need to figure out why.