Data.table: Cannot install data.table from source due to missing symbol _omp_set_num_threads

[Serenthia]

Cannot install data.table without OpenMP on the system.

OS: macOS Sierra 10.12.5
Full output: dataTableIssue.log
Summary of errors:

openmp-utils.c:50:5: warning: implicit declaration of function 'omp_set_num_threads' is invalid in C99 [-Wimplicit-function-declaration]
omp_set_num_threads(1);
^
1 warning generated.

...

Error: package or namespace load failed for ‘data.table’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/usr/local/lib/R/3.4/site-library/data.table/libs/datatable.so':
dlopen(/usr/local/lib/R/3.4/site-library/data.table/libs/datatable.so, 6): Symbol not found: _omp_set_num_threads

Referenced from: /usr/local/lib/R/3.4/site-library/data.table/libs/datatable.so

Expected in: flat namespace in /usr/local/lib/R/3.4/site-library/data.table/libs/datatable.so

Error: loading failed

[fupangpangpang]

I got the same error. I am using macOS High Sierra 10.13

[st-pasha]

I've installed MacOS High Sierra 10.13 just yesterday, and it is able to compile & install data.table just fine (ok, maybe not too fine: it takes much longer now than it used to). I also have installed XCode Command Line tools (which is probably required), and XCode (which is probably not).

That being said, the package produced an error when running the tests:

Running test id 139      Error in strptime(xx, f <- "%Y-%m-%d", tz = "GMT") : 
  (converted from warning) unknown timezone 'default/America/Los_Angeles'
Calls: test.data.table ... as.Date -> as.Date.character -> charToDate -> strptime
Execution halted
make[1]: *** [test] Error 1

[mattdowle]

On the OpenMP fail compiling from source when OpenMP is not available, that was my mistake :
https://stackoverflow.com/questions/46677051/r-cannot-load-data-table-due-to-missing-symbol-omp-set-num-threads#comment80302991_46677051

@Serenthia @fupangpangpang Are you installing the MacOS package binary from CRAN or are you compiling yourself? In the Stack Overflow question comments I asked for the full output but you've pasted the snippet again. Please go ahead and paste _everything_.

Sorry I missed the full output you did attach in the first place. Apologies. Ok so you are installing from source. What's the command you're typing please? You need to install the MacOS binary from CRAN not from source. What's the output of sessionInfo()? You need R v3.4.0 or greater, I believe, for which CRAN supports OpenMP in the binary package.

The binaries are still working their way around CRAN mirrors. For example, right now :

So one of those MacOS binaries is available for 1.10.4-1 and the other not. I don't know how "MacOS High Sierra" relates to "El Capitan" or "Mavericks". I just wish OS vendors would use version numbers rather than names!

If you do wish to install from source then you have to follow the Mac instructions very precisely. It's easier to install the Mac package binary from CRAN though.

[mattdowle]

@st-pasha That's very strange. That's coming from base R itself. Does as.Date("2010-01-01") work in a fresh R session for you without even data.table loaded? Try OlsonNames() and see if GMT is listed.
https://stackoverflow.com/questions/4047188/unknown-timezone-name-in-r-strptime-as-posixct
The error message shows the call stack, and typing base::as.Date.character shows the line that's happening for. The as.Date() call is on a data prep line between test 139 and test 140.

[fupangpangpang]

@mattdowle Thanks for your help! I tried binary and failed (output is pasted below). But I followed your Mac instruction and have successfully installed it from source.

install.packages("/Users/wfu/Downloads/data.table_1.10.4-1.tgz")
Installing package into ‘/usr/local/lib/R/3.4/site-library’
(as ‘lib’ is unspecified)
Warning in install.packages :
package ‘/Users/wfu/Downloads/data.table_1.10.4-1.tgz’ is not available (for R version 3.4.2)

[Serenthia]

@mattdowle I'm not installing from source, I'm afraid. The above output comes from me installing from CRAN with install.packages("data.table"). I've replicated this on my second machine, which has the same OS.

The session info output is:

```R version 3.4.0 (2017-04-21)
Platform: x86_64-apple-darwin16.5.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] compiler_3.4.0 tools_3.4.0

[st-pasha]

@Serenthia can you confirm that you have XCode Command Line Tools installed? You can get them via $ xcode-select --install (in the Terminal)

[Serenthia]

@st-pasha I do have Command Line Tools installed on both machines (one running Sierra, the other High Sierra) and I have the same error on both.

[st-pasha]

I can confirm that the default clang compiler coming with MacOS High Sierra 10.13 still has no OpenMP support. So trying install it without following these instructions causes the compilation error. The error looks like this:

clang: clang: clangclangclangclangclangclang: : : : : : clang: clangclangclang: clangclang: : : : clang: clang: clang: clangclangclang: clang: : : clang: clang: clang: clang: clangclangclangclang: : : : errorerror: : error: error: error: error: errorerror: : errorerror: : error: errorerror: errorerrorerror: unsupported option '-fopenmp': : : unsupported option '-fopenmp'unsupported option '-fopenmp'errorunsupported option '-fopenmp': unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'errorerrorunsupported option '-fopenmp': : error: errorerror: : errorerrorunsupported option '-fopenmp': unsupported option '-fopenmp'unsupported option '-fopenmp': unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'errornsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'[0m: unsupported option '-fopenmp'errorerror: : unsupported option '-fopenmp'unsupported option '-fopenmp'errorerror: unsupported option '-fopenmp': unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'unsupported option '-fopenmp'

@mattdowle I checked that with R 3.4.0 running your test gives the following:

> as.Date("2010-01-01")
[1] "2010-01-01"
Warning message:
In strptime(xx, f <- "%Y-%m-%d", tz = "GMT") :
  unknown timezone 'default/America/Los_Angeles'

Same warning happens in the latest R 3.4.2 as well

[fupangpangpang]

@st-pasha, for timezone warning: that's high sierra problems. You can "fix" it by setting environmental variable Sys.setenv(TZ="America/New_York")

[mattdowle]

@fupangpangpang I believe you need to pass repos=NULL, see ?install.packages and note this paragraph starts with "If repos = NULL" ...

If repos = NULL, a character vector of file paths. These can be source directories or archives or binary package archive files (as created by R CMD build --binary). (http:// and file:// URLs are also accepted and the files will be downloaded and installed from local copies.) On a CRAN build of R for macOS these can be ‘.tgz’ files containing binary package archives. Tilde-expansion will be done on file paths.

So :
install.packages("/Users/wfu/Downloads/data.table_1.10.4-1.tgz")
is looking for the package of that name (the whole path!) and results in :
package ‘/Users/wfu/Downloads/data.table_1.10.4-1.tgz’ is not available (for R version 3.4.2)
I agree that is not very user friendly because R could know that a path with several slashes in it is not a valid package name anyway, and that you intended the install that file. I'm sure a contribution to improve R in that regard would be welcomed.

In the meantime, you must pass repos=NULL. I agree that is not obvious.
install.packages("/Users/wfu/Downloads/data.table_1.10.4-1.tgz", repos=NULL)

Even though you have successfully installed from source, please confirm installing the binary works too.

Or even just,
install.packages("data.table")
should install the binary from CRAN for you on MacOS. Did you manually download the binary because the binary is still working its way around mirrors, perhaps?

[fupangpangpang]

@mattdowle Thank you! That' really helpful! I confirm installing the downloaded binary works without following your mac instruction (maybe not exactly, I just deleted -fopenmp from ~/.R/Makevars before installation).

install.packages("data.table") still tried to install the source. I think it's probably because my macOS is high sierra (10.13).

[mattdowle]

@Serenthia, @fupangpangpang I'm not familiar with MacOS really but looking at ?install.packages I see :

type character, indicating the type of package to download and install. Will be "source" except on Windows and some macOS builds: see the section on ‘Binary packages’ for those.

What "some macOS builds" means I don't know, but the Binary packages section of the manual page is quite detailed.

Binary packages
This section applies only to platforms where binary packages are available: Windows and CRAN builds for macOS.
R packages are primarily distributed as source packages, but binary packages (a packaging up of the installed package) are also supported, and the type most commonly used on Windows and by the CRAN builds for macOS. This function can install either type, either by downloading a file from a repository or from a local file.
Possible values of type are (currently) "source", "mac.binary", "mac.binary.el-capitan" and "win.binary": the appropriate binary type where supported can also be selected as "binary".
For a binary install from a repository, the function checks for the availability of a source package on the same repository, and reports if the source package has a later version, or is available but no binary version is. This check can be suppressed by using
options(install.packages.check.source = "no")
and should be if there is a partial repository containing only binary files.
An alternative (and the current default) is "both" which means ‘use binary if available and current, otherwise try source’. The action if there are source packages which are preferred but may contain code which needs to be compiled is controlled by getOption("install.packages.compile.from.source"). type = "both" will be silently changed to "binary" if either contriburl or available is specified.
Using packages with type = "source" always works provided the package contains no C/C++/Fortran code that needs compilation. Otherwise, on macOS you need to have installed the ‘Command-line tools for Xcode’ (see the ‘R Installation and Administration Manual’) and if needed by the package a Fortran compiler, and have them in your path.

So, try passing type="binary" or type="mac.binary" perhaps. There seems to be specific "mac.binary.el-capitan" too.

The confusion may be compounded because "MacOS High Sierra" is new and you're both using it AND data.table v1.10.4-1 was on CRAN in the last few days and is still working its way around mirrors in two forms separately (source and binary).

[fupangpangpang]

@mattdowle I have to take back my confirmation earlier. Binary installation is not successful. Installation itself has no error or warning. But I cannot load data.table, which will cause R session abortion...

I tried to force type = "mac.binary.el-capitan" in install.package(), It has the same issues (R session abortion when I loaded it)

Nevertheless, your mac instruction works well. I reinstalled data.table from source and everything looks normal now.

[dagola]

I had some trouble with data.table-1.10.4.-1 as well. Following the OpenMP Instructions for Mac I was able install the data.table package but resulted in RStudio crashing whenever saving a source file.
I changed llvm to gcc and now everything is stable.

OS: macOS Sierra 10.13
R: R-3.4.2
RStudio: 1.1.383

[StanleyXu]

@dagola Thank you for sharing, I'm having the exact same issue with RStudio. Would you please share the details further about changing llvm to gcc?
I followed the OpenMP instruction and successfully installed data.table_1.10.4-2 from source. No matter how I comment on/off the ~/.R/Makevars, the RStudio keeps crashing when loading data.table in the building process of my package.
Many thanks!

[dagola]

I installed gcc (most recent version is 7.2.0) using homebrew:

brew update && brew install gcc

Then I changed ~/.R/Makevars to

CC=/usr/local/opt/gcc/bin/gcc-7 -fopenmp
CXX=/usr/local/opt/gcc/bin/g++-7
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/usr/local/opt/gettext/lib -L/usr/local/opt/gcc/lib -Wl,-rpath,/usr/local/opt/gcc/lib
CPPFLAGS=-I/usr/local/opt/gettext/include -I/usr/local/opt/gcc/include

Hope this helps @StanleyXu!