Conda: Recent conda changes breaking Travis

[bashtage]

Not 100% sure that this is a conda issue, but it looks like changes in .27 are breaking Travis builds that
use extension modules.

In particular, the compiler is identified as

compiler: x86_64-conda_cos6-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -fPIC

Not sure why conda would be in the compiler.

And example of a failed build can be see https://travis-ci.org/statsmodels/statsmodels/builds/280537305

[msarahan]

This is because python bakes in compiler flags at build time. We have used new compilers in the 4.3.27 miniconda. These are available as gcc_linux-64, but it's better to use conda-build 3 and the new compiler jinja2 function.

Here's a recipe for statsmodels that has been modified to use conda-build 3 features: https://github.com/anacondarecipes/statsmodels-feedstock

[bashtage]

Is the solution to install gcc_linux-64 so that the normal python setup.py develop will correctly build?

[msarahan]

gcc_linux-64 will provide x86_64-conda_cos6-linux-gnu-gcc - this is a "prefixed" compiler - the stuff preceding gcc is known as the triplet, in gcc terms. Installing the package might work, but it also might not, depending on whether you activate the build environment. If you do, it should work, if you don't, it probably won't be on PATH, and won't work.

It's a bug in our python packages that things are not falling back to your system gcc. We're going to change the symlinks for miniconda-latest to point to the older release until we have this fixed. Sorry for the headache.

[bashtage]

Thanks.

I did a conda install gcc_linux-64 and conda .27 recommended upgrading a Python 3.5 to a Python 3.6 install, which seems pretty heavy handed.

[kalefranz]

I did a conda install gcc_linux-64 and conda .27 recommended upgrading a Python 3.5 to a Python 3.6 install, which seems pretty heavy handed.

This is probably a result of #5923

[lesteve]

Similar problem for scikit-learn for example https://travis-ci.org/scikit-learn/scikit-learn/jobs/280535821. Not sure why but it seems to be happening only for our Python 2.7 build using conda. I am hoping to work-around the problem by using an older miniconda installer and not do conda update conda.

[kalefranz]

The '-latest' miniconda installers have been repainted back to the previous installers with the python version that won't break you.

Sent from my iPhone

On Sep 27, 2017, at 4:45 PM, Loïc Estève [e-mail] wrote:

Similar problem for scikit-learn for example https://travis-ci.org/scikit-learn/scikit-learn/jobs/280535821. Not sure why but it seems to be happening only for our Python 2.7 build using conda. I am hoping to work-around the problem by using an older miniconda installer and not do conda update conda.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[msarahan]

As Kale said, we changed the "latest" minicondas to point at the 4.3.21 version from June. We have a fix, and are building packages. We'll upload new minicondas after testing. Sorry for the trouble.

Updating conda triggers this because updating conda pulls in packages from the new "main" channel, which is part of defaults. That channel has the new python builds that are causing this issue. Updating our python packages will be the fix.

[lesteve]

Great stuff, thanks!

[hainm]

gcc: error: unrecognized command line option ‘-fstack-protector-strong’
gcc: error: unrecognized command line option ‘-fno-plt’

I got this today :(

[twiecki]

I'm getting when travis is trying to build subprocess32.

  x86_64-conda_cos6-linux-gnu-gcc -pthread -fno-strict-aliasing -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -DNDEBUG -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda2/envs/testenv/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o

  unable to execute 'x86_64-conda_cos6-linux-gnu-gcc': No such file or directory

  error: command 'x86_64-conda_cos6-linux-gnu-gcc' failed with exit status 1

[sandys]

hi guys,
is there a workaround for this ? all our builds are failing . I installed gcc_linux-64, but i keep getting other errors (cc1plus not found, etc)

[mingwandroid]

The lastest minicondas have been temporarily replaced with the previous one
while we name a fix here.

If you want to continue with the new stuff and the new compilers (which I
would recommend to be honest) then also install gxx_linux-64 to fix your
cc1plus problem.

On Sep 28, 2017 1:03 PM, "Sandeep Srinivasa" [e-mail]
wrote:

hi guys,
is there a workaround for this ? all our builds are failing . I installed
gcc_linux-64, but i keep getting other errors (cc1plus not found, etc)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/conda/conda/issues/6030#issuecomment-332790111, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA_pdPQElDgBQKM7NC3MFjoqMfW3SddQks5sm28GgaJpZM4PmOLu
.

[lesteve]

The simple work-around I am using for scikit-learn (and for myself) is: do NOT do conda update in your root conda environment. Otherwise this picks up a conda that has the problem (4.3.27-py36h2866c0b_0 for me on Python 3.6).

Doing this you stay with conda 4.3.25 and you are not affected by this issue.

[twiecki]

@mingwandroid Still getting the same error. https://travis-ci.org/pymc-devs/pymc3/jobs/280543334

[sandys]

i may have a partial fix.
i changed my meta.yaml to the following and it kinda worked. it is still failing on test.

build:
    number: {{ build }}
    script: python setup.py install --single-version-externally-managed --record record.txt

requirements:
  host:
    - python
    - setuptools
    - cython 0.26*
    - numpy 1.12*
  build:
    - python
    - opencv 3.1*
    - numpy 1.12*
    - {{ compiler('cxx') }}
    - cython 0.26*
  run:
    - python
    - opencv 3.1*
    - numpy 1.12*
    - cython 0.26*

[bashtage]

@lesteve 's fix is working in statsmodels.

[bashtage]

@twiecki You need to pin conda<=4.3.25

From your build log during an installation:

 conda-4.3.27               |   py27hff99c7a_0         508 KB

[sandys]

Installing conda-build force upgrades conda

On 28-Sep-2017 16:48, "Kevin Sheppard" [e-mail] wrote:

@twiecki https://github.com/twiecki You need to pin conda<=4.3.35

From your build log during an installation:

conda-4.3.27 | py27hff99c7a_0 508 KB

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/conda/conda/issues/6030#issuecomment-332806360, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEsU-cpc1WJq1Zf_PYsn32pDGMJ9xRDks5sm4BpgaJpZM4PmOLu
.

[twiecki]

@bashtage I thought it was replaced by @mingwandroid:

The lastest minicondas have been temporarily replaced with the previous one while we name a fix here.

[bashtage]

AFAIU The miniconda install was changed so that the default conda is < 4.3.27, the problematic one. Your environment creation is triggering an automatic update of conda to latest, which is 4.3.27. The conda hasn't been unpublished -- only the miniconda(2/3)-latest installer has been retargetted.

[twiecki]

@bashtage I see, thanks.

[msarahan]

Quick update here: I have a "fixed" version of python 3.6 that I'm playing with. We have tried to make things compatible by copying the _sysconfig...py module from older python 3.6 packages into the new installation, and then setting _PYTHON_SYSCONFIGDATA_NAME to our new sysconfig only when we're using our new compiler.

It "works" in that the old gcc gets picked up and used. Unfortunately, it looks like this strategy is not going to work, as linking against the libraries compiled with the new compiler (and with lots of new security flags) just doesn't seem to work. Here's an excerpt of my attempt to build statsmodels with the new python package, using centos 6's gcc 4.4 compiler:

statsmodels/tsa/statespace/_statespace.c:4494: note: initialized from here
gcc -pthread -shared -L$PREFIX/lib -Wl,-rpath=$PREFIX/lib,--no-as-needed build/temp.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.o -L/opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib -L$PREFIX/lib -lnpymath -lm -lpython3.6m -o build/lib.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.cpython-36m-x86_64-linux-gnu.so
/usr/bin/ld: /opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib/libnpymath.a(npy_math_complex.o)(.text+0x2): unresolvable H�\$�H�l$�H��L�d$�H��H�GH��I��H��H�;�PxH�EI�tH��PxH�EH�{I�t$H�$H�lL�d$H�@xH����f�H�\$�H�l$�H��L�d$�H��H�GI��H��H���PhI�$H�CH��PhH�$I�DH�lL�d$H��� relocation against symbol `cabsf@@GLIBC_2.2.5'
/usr/bin/ld: BFD version 2.20.51.0.2-5.46.el6 20100205 internal error, aborting at reloc.c line 443 in bfd_get_reloc_size

/usr/bin/ld: Please report this bug.

collect2: ld returned 1 exit status
/usr/bin/ld: /opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib/libnpymath.a(npy_math_complex.o)(.text+0x2): unresolvable H‰\$èH‰l$ðH‰óL‰d$øHƒìH‹GH‰ýI‰ÔH‰ÖH‹;ÿPxH‹EItH‹ÿPxH‹EH‹{It$H‹$H‹lL‹d$H‹@xHƒÄÿàfH‰\$èH‰l$ðH‰ûL‰d$øHƒìH‹GI‰ÔH‰õH‰÷ÿPhI‰$H‹CHÿPhH‹$I‰DH‹lL‹d$HƒÄà relocation against symbol `cabsf@@GLIBC_2.2.5'
/usr/bin/ld: BFD version 2.20.51.0.2-5.46.el6 20100205 internal error, aborting at reloc.c line 443 in bfd_get_reloc_size

/usr/bin/ld: Please report this bug.

collect2: ld returned 1 exit status
error: Command "gcc -pthread -shared -L$PREFIX/lib -Wl,-rpath=$PREFIX/lib,--no-as-needed build/temp.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.o -L/opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib -L$PREFIX/lib -lnpymath -lm -lpython3.6m -o build/lib.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.cpython-36m-x86_64-linux-gnu.so" failed with exit status 1

We'll keep working here to help understand what options everyone has. We really don't want to force anyone to do drastic changes. If anyone knows if there is a resolution to these issues that maintains the security features, we'd be very interested.

[msarahan]

One other update: @jjhelmus has been working with py2.7 while I was working on py3.6. He had more luck. His changes are at https://github.com/jjhelmus/python-feedstock/tree/jjh_27_wip

He reports that he is able to build statsmodels using this python and a system gcc. I'm not sure exactly which gcc.

[lesteve]

I'm not sure exactly which gcc.

FYI Travis (Ubuntu 14.04) has gcc 4.8.4.

[virtualluke]

I hope when fixed there will be a 5.0.1 (or similar) release so that the base release has the fix. thanks

[msarahan]

It should now be safe to update conda. This is fixed in the following python packages for linux-64:

• python-3.6.2-h0b30769_14.tar.bz2
• python-2.7.14-h931c8b0_15.tar.bz2
• python-2.7.13-hac47a24_15.tar.bz2
• python-3.5.4-hc053d89_14.tar.bz2

I have verified that each of these work for statsmodels by cloning that git repo and tweaking the TravisCI configuration: https://travis-ci.org/msarahan/statsmodels/builds/281385592

The issue was that we (Anaconda, formerly Continuum) build all of our packages with a new GCC package that we created using crosstool-ng. This package does not have gcc, it has a prefixed gcc - the missing command you're seeing, x86_64-conda_cos6-linux-gnu-gcc. This gets baked into python, and any extension built with that python goes looking for that compiler. We have fixed the issue using the _PYTHON_SYSCONFIGDATA_NAME variable that was added to python 3.6. We have backported that to python 2.7 and 3.5. You'll now only ever see python using default compilers (gcc), and you must set the _PYTHON_SYSCONFIGDATA_NAME to the appropriate filename to have the new compilers used. Setting this variable is something that we'll put into the activate scripts for the compiler package, so you'll never need to worry about it. It may take us a day or two to get new compiler packages out, though, so post issues on the conda-build issue tracker if you'd like to use the new compilers and need help getting started.

Relevant code changes are at:

py27: https://github.com/anacondarecipes/python-feedstock/tree/master-2.7.14
py35: https://github.com/anacondarecipes/python-feedstock/tree/master-3.5
py36: https://github.com/anacondarecipes/python-feedstock

[msarahan]

There will be 5.0.0.1 builds of anaconda, and 4.3.27.1 builds of miniconda out soon. Those take longer to build out and run verification tests on. We'll be adding tests for this issue.

[hainm]

@msarahan did you change compiler (or compiler flag) for MacOS python build too?

I used to add clang++ -stdlib=libstdc++ flag to compile our C++ library so we can have compatibility with our Python from Miniconda. But it's not true since few days ago (cythonized .so file is linked to libc++.1.dylib ratherlibstdc++.1dylib`.

[msarahan]

@hainm yes, there's a new compiler for MacOS too. The fix for them is identical. Those packages are:

• python-2.7.14-hd74e306_15.tar.bz2
• python-2.7.13-hdada7c8_15.tar.bz2
• python-3.5.4-h821eb87_14.tar.bz2
• python-3.6.2-h9e63aee_14.tar.bz2

The new mac compilers are the clang_osx-64 and clangxx_osx-64 packages.

The recipe for the new mac compilers are at https://github.com/AnacondaRecipes/aggregate/tree/master/llvm-compilers-feedstock/recipe

They do switch from libstdc++ (from gcc) to libc++ (from llvm/clang). Keeping these consistent within a package ecosystem is extremely important, and conda-forge has demonstrated to us that libc++ is the right way to go on Mac. The main reason for confusion before now has been that people were using the gcc (version 4.8.5) package, which had gcc, g++ and gfortran all in one. They mostly used this when they needed fortran, but as a side effect, it introduced a lot of libstdc++ dependencies, and those caused problems. The gfortran package on mac is now totally separate from gcc, and there's no longer any reason to use the gcc package on mac - prefer the clang packages I mentioned above (and ideally, use conda-build 3's {{ compiler('c') }} to use them).

[msarahan]

ah, for reference, the values to set the _PYTHON_SYSCONFIGDATA_NAME to until we get things set in the compiler packages. I do this in conda_build_config.yaml:

_PYTHON_SYSCONFIGDATA_NAME:
  - _sysconfig_x86_64_conda_cos6_linux_gnu  # [linux]
  - _sysconfig_x86_64-apple-darwin13.4.0    # [osx]

Note that you need one of the new pythons I've listed above installed in your root environment before you try to do a build with these variables set. If you don't, conda will give you a nasty-looking import error when it tries to import the site module.

[hainm]

Keeping these consistent within a package ecosystem is extremely important, and conda-forge has demonstrated to us that libc++ is the right way to go on Mac.

Yes, I am very happy to see this change. Linking to libstdc++ on Mac caused us lots of trouble: https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/xaPdTycZFu0

[hainm]

@msarahan since we still need to support the old compiler (e.g: users already has Miniconda before your change), is there any reliable way to detect if Python was compiled with gcc or clang?

[hainm]

Does this?

In [7]: from distutils import sysconfig as sc

In [8]: v = sc.get_config_vars()

# new
In [9]: v['MAINCC']
Out[9]: '/usr/bin/clang'

# old
In [4]: v['MAINCC']
Out[4]: 'gcc'

[hainm]

uhm, seems not, below is from native Python shipped by Apple.

>>> v['MAINCC']
'cc'

[msarahan]

I'm not sure you need to?

The way this works (or should work, anyway) is that python is built with the new compiler. Python is all C, so that's fine as far as ABI conflicts go. Python bakes its compiler variables and such into a _sysconfigdata_something.py file. We copy that to _sysconfigdata_$HOST.py, and replace it with a _sysconfigdata.py that we retrieved from python with an older compiler. This is effectively a fallback. If you need to support the old compiler, just don't set _PYTHON_SYSCONFIGDATA_NAME - that should be all. It's more challenging to use the new compilers, but hopefully we'll take care of that by setting up the env var for you in activation scripts.

[msarahan]

One small detail I neglected to mention: we are bundling an ld executable on Linux with python. This was necessary to make the fallback work. Without it, old linkers would fail, we think due to the additional LTO optimization. We set up python's compiler flags to look in $PREFIX/bin, where we put ld, so that we're using a new enough ld to avoid issues.

[msarahan]

There was one further issue highlighted on the anaconda support mailing list by Matt Newville: having ld in $PREFIX/bin meant that it broke other normal system compiler stuff outside of python. We have moved the ld executable into a folder that is not on path, $PREFIX/compiler_compat, and pointed the python sysconfig at that folder. Hopefully that is the end of this issue. New builds are on the main channel, and new minicondas and anacondas are being built with those packages.

[msarahan]

New minicondas (4.3.27.1) are available. The "latest" symlinks have been updated to point at these. Closing this issue, but please post back if you see further issues.

[kalefranz]

Isn’t bundling ld with python kinda a bad idea? Shouldn’t we be making it its own package, and then making it a dependency of the python package?

[mingwandroid]

That is the correct way to do it, yes, but this is a quick workaround that we can easily correct later.

[TomAugspurger]

@msarahan I'm re-enabling the latest miniconda for pandas in https://github.com/pandas-dev/pandas/pull/17752

Does this failure concern you at all? https://travis-ci.org/pandas-dev/pandas/jobs/282433255#L1942. lxml is installed here

I haven't spent any time debugging this yet, so it could be entirely an issue with pandas' CI setup. I'll look more deeply tonight.

[TomAugspurger]

Actually, looking a little further, it's almost surely us mixing conda-forge and defaults. Sorry for the noise.

[mingwandroid]

We removed the libiconv package from Linux as it is provided as part of glibc. We will send PRs to conda-forge for these changes.

[dbogdanov]

I've installed the latest Minicondas 4.3.27.1 and I still get the error:

err: g++: error: unrecognized command line option ‘-fno-plt´

conda --version gives conda 4.3.27

[mingwandroid]

You need to update your python. We will release new Minicondas ASAP.

[mfeurer]

Unfortunately, there still seems to be an error related to the upgrade from .21 to .27

I ran these commands on a the latest docker image for ubuntu (ubuntu:latest):

apt-get update -y
apt-get install build-essential wget
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh 
export PATH=/root/miniconda3/bin/:$PATH
conda install gcc
conda upgrade --all
pip install psutil

and this results into:

root@c1b42aaa57b2:/# pip install psutil
Collecting psutil
  Using cached psutil-5.4.0.tar.gz
Building wheels for collected packages: psutil
  Running setup.py bdist_wheel for psutil ... error
  Complete output from command /root/miniconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-onr9l1v8/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpaz1gtiotpip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/psutil
  copying psutil/__init__.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_pssunos.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_psosx.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_psaix.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_common.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_pswindows.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_psposix.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_psbsd.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_compat.py -> build/lib.linux-x86_64-3.6/psutil
  copying psutil/_pslinux.py -> build/lib.linux-x86_64-3.6/psutil
  creating build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/__init__.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/__main__.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-3.6/psutil/tests
  copying psutil/tests/test_memory_leaks.py -> build/lib.linux-x86_64-3.6/psutil/tests
  running build_ext
  building 'psutil._psutil_linux' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/psutil
  gcc -pthread -B /root/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=540 -DPSUTIL_LINUX=1 -I/root/miniconda3/include/python3.6m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_common.o
  gcc -pthread -B /root/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=540 -DPSUTIL_LINUX=1 -I/root/miniconda3/include/python3.6m -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o
  gcc -pthread -B /root/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=540 -DPSUTIL_LINUX=1 -I/root/miniconda3/include/python3.6m -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_linux.o
  gcc -pthread -shared -B /root/miniconda3/compiler_compat -L/root/miniconda3/lib -Wl,-rpath=/root/miniconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/psutil/_psutil_common.o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o build/temp.linux-x86_64-3.6/psutil/_psutil_linux.o -L/root/miniconda3/lib -lpython3.6m -o build/lib.linux-x86_64-3.6/psutil/_psutil_linux.cpython-36m-x86_64-linux-gnu.so
  /root/miniconda3/compiler_compat/ld: cannot find -lpthread
  /root/miniconda3/compiler_compat/ld: cannot find -lc
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for psutil
  Running setup.py clean for psutil
Failed to build psutil
Installing collected packages: psutil
  Running setup.py install for psutil ... error
    Complete output from command /root/miniconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-onr9l1v8/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-izzoejmv-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/psutil
    copying psutil/__init__.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_pssunos.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_psosx.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_psaix.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_common.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_pswindows.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_psposix.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_psbsd.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_compat.py -> build/lib.linux-x86_64-3.6/psutil
    copying psutil/_pslinux.py -> build/lib.linux-x86_64-3.6/psutil
    creating build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/__init__.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/__main__.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-3.6/psutil/tests
    copying psutil/tests/test_memory_leaks.py -> build/lib.linux-x86_64-3.6/psutil/tests
    running build_ext
    building 'psutil._psutil_linux' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/psutil
    gcc -pthread -B /root/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=540 -DPSUTIL_LINUX=1 -I/root/miniconda3/include/python3.6m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_common.o
    gcc -pthread -B /root/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=540 -DPSUTIL_LINUX=1 -I/root/miniconda3/include/python3.6m -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o
    gcc -pthread -B /root/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=540 -DPSUTIL_LINUX=1 -I/root/miniconda3/include/python3.6m -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_linux.o
    gcc -pthread -shared -B /root/miniconda3/compiler_compat -L/root/miniconda3/lib -Wl,-rpath=/root/miniconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/psutil/_psutil_common.o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o build/temp.linux-x86_64-3.6/psutil/_psutil_linux.o -L/root/miniconda3/lib -lpython3.6m -o build/lib.linux-x86_64-3.6/psutil/_psutil_linux.cpython-36m-x86_64-linux-gnu.so
    /root/miniconda3/compiler_compat/ld: cannot find -lpthread
    /root/miniconda3/compiler_compat/ld: cannot find -lc
    collect2: error: ld returned 1 exit status
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/root/miniconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-onr9l1v8/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-izzoejmv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-onr9l1v8/psutil/

We're unit testing our package on travis-ci and do not observe these issues with Miniconda3-4.3.21. To check issues with the latest conda version, we use a separate branch (latest_conda) which does no longer work under python3. We're also observing build errors with a different tool called pyrfr.

[drorata]

I'm am not sure whether this is 100% related; I apologize in advance if this is not.

# Works
# FROM continuumio/anaconda3:4.4.0
# Fails
# FROM continuumio/anaconda3:5.0.0

RUN apt-get update && apt-get install -y \
  freetds-dev \
  python3-dev \
  build-essential \
  && apt-get clean -y \
  && rm -rf /var/lib/apt/lists/*

COPY requirements.txt /tmp
RUN pip install pymssql

When building using 4.4.0 everything is smooth. However, switching to 5.0.0 (currently tagged as latest) I get the following error:

  Failed building wheel for pymssql
  Running setup.py clean for pymssql
Failed to build pymssql
Installing collected packages: pymssql
  Running setup.py install for pymssql: started
    Running setup.py install for pymssql: finished with status 'error'
    Complete output from command /opt/conda/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-cjxxfq8u/pymssql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mz7gt8h4-record/install-record.txt --single-version-externally-managed --compile:
    setup.py: platform.system() => 'Linux'
    setup.py: platform.architecture() => ('64bit', '')
    setup.py: platform.linux_distribution() => ('debian', '8.5', '')
    setup.py: platform.libc_ver() => ('glibc', '2.9')
    setup.py: Not using bundled FreeTDS
    setup.py: include_dirs = ['/usr/local/include']
    setup.py: library_dirs = ['/usr/local/lib']
    running install
    running build
    running build_ext
    cythoning _mssql.pyx to _mssql.c
    warning: _mssql.pyx:143:4: Exception already a builtin Cython type
    building '_mssql' extension
    creating build
    creating build/temp.linux-x86_64-3.6
    x86_64-conda_cos6-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -fPIC -I/usr/local/include -I/opt/conda/include/python3.6m -c _mssql.c -o build/temp.linux-x86_64-3.6/_mssql.o -DMSDBLIB
    unable to execute 'x86_64-conda_cos6-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-conda_cos6-linux-gnu-gcc' failed with exit status 1

[mingwandroid]

You need to do a RUN conda update python before RUN pip install pymssql

[janssen]

Ah, this is why my builds are all broken for the past week or so!

Working:

% /local/goodstuff/bin/python   # the old working version
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> 

Broken:

% /opt/goodstuff/bin/python   # the new broken version
Python 2.7.14 |Anaconda, Inc.| (default, Oct  5 2017, 02:28:52) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

The error we're seeing is

ImportError: dlopen(/opt/goodstuff/lib/python2.7/site-packages/scipy/linalg/_fblas.so, 2): Symbol not found: _main
  Referenced from: /opt/goodstuff/lib/python2.7/site-packages/scipy/linalg/_fblas.so
  Expected in: flat namespace
 in /opt/goodstuff/lib/python2.7/site-packages/scipy/linalg/_fblas.so

Not sure why scipy wouldn't be compatible; we're using 0.19.1-py27hf01dd8f_3, selected by conda 4.3.30. OS X 10.12.6.

[terhorst]

conda.txt
I am running into an issue which I think is related to this. conda-build dies with ModuleNotFoundError: No module named '_sysconfigdata_x86_64_conda_cos6_linux_gnu'.
This occurs when trying to build on either py35 or py36. I am running the latest Miniconda and have updated conda, python, conda-build, and all the other packages I can think of.

[mingwandroid]

Please put a conda info and conda list --show-channel-urls somewhere in your script and paste the output.

[terhorst]

@mingwandroid I added it, thanks for the quick reply.

[mingwandroid]

@terhost, you are mixing conda-forge and conda defaults packages here in the same environment. We are not (yet) binary compatible. You need to use a separate environment.

But even with that I am still confused about this error since you are using python from conda-forge and that error can only come about from using conda defaults python. Can you post the full stacktrace please?

[mingwandroid]

No problem! Achieving compatibility with conda-forge is something we are trying to do.

[mingwandroid]

No problem! Achieving compatibility with conda-forge is something we are trying to do.

[terhorst]

Removing -c conda-forge from conda build fixed it. I must have added it long ago to satisfy some missing dependency, but now everything is available it default. Thanks again for the very quick response. Edit: sorry for the multiple posts

[mingwandroid]

Something seems to be going on with github today. I also posted multiply.

[nick-youngblut]

Maybe this should be a separate issue, but I'm getting x86_64-conda_cos6-linux-gnu-gcc: command not found and x86_64-conda_cos6-linux-gnu-c++: command not found when trying to install R packages in R with install.packages(). Installation of R packages works via conda (eg., conda install -r r-ggplot2, but not via install.packages().

Using conda is fine for R packages available via conda, but what about the packages that aren't available or are broken? For example, conda install -c bioconda bioconductor-phyloseq generates the following error when loading the phyloseq package:

> library(phyloseq)
Error: package or namespace load failed for ‘phyloseq’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/ebio/abt3_projects/software/dev/miniconda3/envs/phyloseq/lib/R/library/stringi/libs/stringi.so':
  libicui18n.so.58: cannot open shared object file: No such file or directory

I tried conda install gcc_linux-64 as suggested by @mingwandroid on StackOverflow, but I still get x86_64-conda_cos6-linux-gnu-c++: command not found. The only file in ./miniconda/bin/ that resembles this file is x86_64-conda_cos6-linux-gnu-c++filt. Should the "filt" be appended to the name of this file?

I'm using r-base v3.4.1 from the r channel, conda v4.3.30, Ubuntu 16.04.3

[mingwandroid]

No this is not related. To use install.packages() on any R on Linux you need compilers. To use install.packages() on Anaconda Distribution R on Linux you need Anaconda Distribution compilers. Here you are trying to compile a package that needs C++ compilers, so you need to install gxx_linux-64.

[nick-youngblut]

Sorry for asking in the wrong spot. Thanks for the clarification. For what it's worth, running the following fixed the issue:

conda install gcc_linux-64
conda install gxx_linux-64

I still get an error when installing the mnormt package:

/ebio/abt3_projects/software/dev/miniconda3/bin/x86_64-conda_cos6-linux-gnu-gfortran   -fpic  -fopenmp -march=
nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/ebio/abt3_projects
/software/dev/miniconda3/include -L/ebio/abt3_projects/software/dev/miniconda3/lib  -c biv-nt.f -o biv-nt.o
make: /ebio/abt3_projects/software/dev/miniconda3/bin/x86_64-conda_cos6-linux-gnu-gfortran: Command not found
/ebio/abt3_projects/software/dev/miniconda3/lib/R/etc/Makeconf:184: recipe for target 'biv-nt.o' failed

... but I'm guessing that conda install gfortran_linux-64 or conda install -c tsnyder gfortran_linux-cos6-x86_64 should fix it.

Thanks again for the help!

[mingwandroid]

I would not encourage people to search for packages using anaconda and install them from users they do not know. If you know and trust tsnyder and he recommended you to install his packages for specific reasons then great, but otherwise avoid installing random packages.

conda install gfortran_linux-64 will get the gfortran package we (the Anaconda Distribution team) built (provided you've not customised your channels in ~/.condarc that is).

[nick-youngblut]

Good point. conda install gfortran_linux-64 seems to have worked for installing the mnormt R package

[nick-youngblut]

I think that I spoke too soon about fixing the issues. I can move this to a separate issue if needed. I installed the following:

conda install -c r r-base
conda install gcc_linux-64
conda install gxx_linux-64
conda install gfortran_linux-64

and this allowed me to install Rcpp, mnormt, etc. However, I'm getting issues when installing devtools or igraph (installing with install.packages()). For devtools, the error is:

* installing *source* package ‘openssl’ ...
** package ‘openssl’ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=
src/tests/soname.h:1:10: fatal error: openssl/opensslv.h: No such file or directory
 #include "openssl/opensslv.h"

For igraph, the error is:

foreign-graphml.c: In function 'igraph_write_graph_graphml':
foreign-graphml.c:1408:46: error: expected ')' before 'GRAPHML_NAMESPACE_URI'
   ret=fprintf(outstream, "<graphml xmlns=\"" GRAPHML_NAMESPACE_URI "\"\n");
                                              ^~~~~~~~~~~~~~~~~~~~~
foreign-graphml.c:1412:59: error: expected ')' before 'GRAPHML_NAMESPACE_URI'
   ret=fprintf(outstream, "         xsi:schemaLocation=\"" GRAPHML_NAMESPACE_URI "\n");
                                                           ^~~~~~~~~~~~~~~~~~~~~
foreign-graphml.c:1414:38: error: expected ')' before 'GRAPHML_NAMESPACE_URI'
   ret=fprintf(outstream, "         " GRAPHML_NAMESPACE_URI "/1.0/graphml.xsd\">\n");
                                      ^~~~~~~~~~~~~~~~~~~~~
/ebio/abt3_projects/software/dev/miniconda3/envs/py2/lib/R/etc/Makeconf:160: recipe for target 'foreign-graphml.o' failed

I'm not sure if I have to install something else via conda to get these R packages to install properly.

[mingwandroid]

conda install -c r r-base

You do not need to specify -c r here anymore. r is now one of the default channels.

The Anaconda Distribution precompiles a lot of R packages so you don't need to:

conda install r-rcpp r-devtools r-igraph

In general if you want to compile things for yourself you need to be pretty software-building-savvy I am afraid and I wouldn't really recommend the 'standard' R approach of install.packages() because frankly IMHO it's a mess.

In the openssl case, you are missing the conda openssl package in your py2 env, I guess.

If you really do want to compile R packages from source you should use conda-build to generate recipes and then build those (conda skeleton cran igraph, then conda-build r-igraph for example). You need to pay attention to the SystemRequirements: section though and add the appropriate conda package to the requirements/build and or requirements/run sections in the generated meta.yaml.

I am considering modifying the Anaconda Distribution R's install.packages() to try installing a conda package first.

[nick-youngblut]

Over the past year, myself and other members of my research group have had a bunch of issues when installing R packages via conda such as:

• "package version not found" errors when trying to recreate environments from *.yaml files
• Plotting in R didn't work because of issues with r-cairo
• bioconductor packages like phyloseq fail when loading (eg., libicui18n.so.58: cannot open shared object)
• some R packages are not available via conda, which leads to a strange (and often problematic) hybrid of using conda and install.packages()

For these reasons, we've switched to installing R packages whenever possible with install.packages(). At least up until recently, it's been a LOT less problematic than installing R packages with conda. It would be great to manage all software with conda, but with R packages, it's been a huge pain.

[mingwandroid]

I will attempt to fix any problem identified to me within the Anaconda Distribution. Let's take you points one at a time.

"package version not found" errors when trying to recreate environments from *.yaml files

You are over-pinning here I guess. How did you generate the environment.yaml files? I personally do not like this approach since it probably pins exact build numbers (so you will not get bug fixes to builds) and doesn't record and re-use channel information. I would go for just conda install <r-blah> <r-blah2> <r-blah3> instead. If you do want exact versions then you need to be careful to have the exact same channels in your ~/.condarc as where you generated the environment.yaml. Pinging @kalefranz to fact check this as, as I say, I have never used this feature.

Plotting in R didn't work because of issues with r-cairo

This has been fixed for a long time.

bioconductor packages like phyloseq fail when loading (eg., libicui18n.so.58: cannot open shared object)

I assume you meant bioconductor packages installed from the bioconda or conda-forge channel(s) here? I have only very limited influence over conda-forge and bioconda. They are currently carrying broken packages I am afraid. We (the Anaconda Distribution team) are trying to attain binary compatibility with them which would include alignment of our shared library versions but we are not there yet.

some R packages are not available via conda, which leads to a strange (and often problematic) hybrid of using conda and install.packages()

You are best off building conda binary R packages the way we (Anaconda Distribution and conda-forge) do and if you run into any problems with that approach then I am more than happy to help out. You can then host such packages on anaconda.org so that other team members can re-use them.

[nick-youngblut]

Thank you for your help with all of these issues. Sorry to be such a pain, but I've spent most of my day trying to figure out how to reconfigure my group's conda environments, which incorporate python, jupyter, R, and a lot of bioinformatics software. Our previous method of installing jupyter & r-base with conda, then installing R packages via install.packages() has become very problematic recently.

In regards to recreating conda environments, I haven't tried recently, so I don't know if this problem still exists, but it made us weary of using conda to install R packages if we couldn't reproduce the environment from a *.yaml file.

The r-cairo issue is old, but again, it made us weary of using conda for installing R packages.

Yes, I did mean installing bioconductor packages with conda. Given that some package installations are broken and some are not available via conda, we often have to use install.packages(). We didn't know about building conda binary R packages, but that would have to be done repeatedly each time a package was updated on CRAN (if we want/need updated package versions), correct?

Also, members of my group that have limited computational skills will probably just keep using install.packages(), especially since documentation on installing R packages always suggests using install.packages() or biocLite() (eg., the Phyloseq package installation documentation)

I'm not sure what to do right now, given that I can't seem to install some R packages that are needed for our research. For instance, phyloseq is broken if installed via conda, but using install.packages() requires an igraph install (even if already installed via conda), and the igraph installation isn't working with install.packages(). I don't want to give up on managing R and R packages with conda, but we can't conduct our research without being able to properly manage installations of this software.

[mingwandroid]

Yes, I did mean installing bioconductor packages with conda

bioconductor is separate from bioconda. bioconda packages bioconductor software (and more). You need to be clear about where you installed broken or incompatible software from, i.e. did you use conda install -c bioconda phyloseq?)

IMHO something needs to be done upstream in the R project to fix the status quo. CRAN has had binary packages for Windows for a long time and now they've started to do that for macOS too (since 3.4), however they only support macOS 10.11 and above while we support macOS 10.9 and above. Many R packages also use something called autobrew on macOS for people who attempt to build from source. This doesn't really build from source, instead it partially builds from source, fetching static system libraries as binary bottles from Homebrew. For Linux, there are no binaries and it is up to the end-user to have the correct software (compilers and system libraries) installed. This is not something that is going to end well in general. It requires everyone using R on Linux to be an expert at software compilation and packaging.

We are actively thinking about this but have nothing to report. I honestly do believe our way is the least trouble, provided you stick to the defaults channels only and use conda skeleton cran to build the missing packages.

We didn't know about building conda binary R packages, but that would have to be done repeatedly each time a package was updated on CRAN (if we want/need updated package versions), correct?

I only do full build-outs of all of our R packages at each release of the R interpreter that we bundle. For most people there's no strong need to have the latest and greatest of everything all the time. If you need a new feature in the latest version of a package, well, you'll know about that.

[nick-youngblut]

Sorry for not being clear about the phyloseq install. I installed it from bioconda.

I appreciate all that you and others are doing with conda to create a powerful tool for managing python & other software, which greatly helps with performing reproducible bioinformatics (& data science in general).

With that said, I currently can't get some R packages installed with install.packages() (due to the compiling errors) and those packages are either not available via conda or the conda install is broken. Compiling R packages and posting them to anaconda.org could help, but this potentially produces a lot of work for my group (if many packages must be maintained due to dependencies with various bioinformatics software relying on specific package versions), and many in my group will probably argue that it will be better to just switch to local installs of R outside of conda, and keep R completely separate from conda. That is, if we could figure out how to keep R separate from conda, but still use R kernels & rpy2 with Jupyter.

It's too bad (and really frustrating) that there's such a divide between python and R (eg., Rstudio doesn't support python; Jupyter doesn't work natively with R & CRAN; not all R packages are supported by conda). This leaves bioinformaticians and data scientists trying to span the divide, and not always successfully.

[mingwandroid]

With that said, I currently can't get some R packages installed with install.packages() (due to the compiling errors) and those packages are either not available via conda or the conda install is broken

The thing is, the problems you described with install.packages() on Linux are not to do with conda. You need to install toolchains and system packages (when using conda via conda install, when using some other Linux distro via apt-get install or dnf install) so I don't think 'local installs of R outside of conda' will be any easier, ok, you might already have those toolchains and system packages installed on your system but that just leads to non-explicit dependency management (which conda always avoids) and problems when people without those system packages (or different versions of them) try to repeat what you built.

For managing R packages on Linux, my opinion is that CRAN is worse than conda because at least with conda we have built as binaries, ensured good iterop with Python and occasionally bug-fixed the build system for 340 odd R packages in a very compatible way (i.e. across different versions of our host OSes) so that you do not have to.

If you do decide to explore conda skeleton cran I am happy to assist.

[pengjingg]

I encountered this problem while installing cupy, I tried downgrade conda to several versions(including 4.3.24, 4.3.25, 4.3.27) but none of them works. My original conda version is 4.3.30 by the way. So I upgrade conda back to 4.3.30 and run command "conda update python" and it finally worked out !!!
(system: ubuntu 16.04LTS; cuda: 9.0)
2017.11.15

[armaneshaghi]

I have the same issue with the latest Anaconda (downloaded today) when installing new packages in R.

[mingwandroid]

@armaneshaghi what issue? You need to install our compilers. The package is called gxx_linux-64.

[armaneshaghi]

@mingwandroid thanks for the response, conda install gxx_linux-64 solved my issue. I just wonder why this is no longer automatic for naive users (like me)?

[mingwandroid]

Historically we didn't provide any compilers, then we provided old ones but our R packages didn't use the those old ones instead relying on system ones (when you use install.packages()). That brings it's own problems (I don't know how capable your system ones are and anything you build with them will likely not work on other systems and loading R packages built with them could prevent binaries we built from loading or working).

Forcing our compilers on everyone who installs R on Linux might be the best plan though, but actually upstream R is moving towards providing prebuilt binaries (at least for Windows and macOS).

The other complication is that you are better off not using install.packages() for most things since we already provide prebuilt binary conda packages for a lot of the most popular R packages, so conda install r-something is better.

I am considering to patch install.packages() so that it tries installing a conda package first, and failing that, installs compilers if necessary before running the normal code-path.

Overall it's very difficult to know what is best here!

[corytu]

@mingwandroid some research on the Internet led me here. I am facing difficulties when installing R packages just like @nick-youngblut did. I see your suggestion for using conda skeleton cran approach instead of using R function install.packages, and I have done conda install gxx_linux-64. However, I keep failing at the conda build step.

Specifically, I want to install an R package called "RPostgreSQL", and here is what I have done:

conda install gxx_linux-64
conda skeleton cran r-rpostgresql
conda build r-rpostgresql
# I also tried:
# conda build --R 3.4.2 r-rpostgresql
# conda build --R=3.4.2 r-rpostgresql

The error includes:

make: /home/ytu/anaconda3/conda-bld/r-rpostgresql_1519544237983/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/x86_64-conda_cos6-linux-gnu-cc: Command not found

, and

make: * [/home/ytu/anaconda3/conda-bld/r-rpostgresql_1519544237983/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/R/etc/Makeconf:160: RS-DBI.o] Error 127
ERROR: compilation failed for package ‘RPostgreSQL’

, and finally

subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/home/ytu/anaconda3/conda-bld/r-rpostgresql_1519544237983/work/conda_build.sh']' returned non-zero exit status 1.

You could find some other information in my question on Stack Overflow: Installing packages in IRkernel on Jupyter Notebook.

How should I do to solve this?

[zhou13]

I am on Archlinux and miniconda always gives me file not recognized: file format not recognized when gcc or ld is called. Removing compiler_compat/ld solves the issue.

[astrojuanlu]

@shreyasbapat and I just independently reproduced what @zhou13 mentioned on Arch Linux. I guess this is being tracked in https://github.com/ContinuumIO/anaconda-issues/issues/11152 now.

[mingwandroid]

@Juanlu001
Would it be possible to try the package mentioned in my last comment on issue 11152 please?

[astrojuanlu]

Thanks @mingwandroid, doing conda install -c rdonnelly python=3.7.4 fixes the issue :+1:

[chux0519]

@zhou13 thanks, it works