Openblas: Homebrew build of 0.3.8 fails with GCC 9, passes with native Apple clang when OpenMP enabled

Created on 15 Feb 2020 · 19Comments · Source: xianyi/OpenBLAS

EDIT: Possible dupe of #2416

I was upgrading Homebrew's OpenBLAS to 0.3.8 and the build failed with GCC 9.2.

The build errors are are all error: invalid instruction mnemonic 'dmb' in level3_thread.c.

Fortunately using native Apple Clang (which Homebrew prefers to do) resolves this issue.

Also, I'm not sure whether or not you are aware, but Homebrew switched to using OpenBLAS as the default BLAS and LAPACK implementation instead of Apple's (outdated) Accelerate framework in the last 6 months/year.

Here is a more extended log incase the Jenkins CI log gets deleted (which it will after a while).

# ...

gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_nn -DASMFNAME=_sgemm_thread_nn_ -DNAME=sgemm_thread_nn_ -DCNAME=sgemm_thread_nn -DCHAR_NAME=\"sgemm_thread_nn_\" -DCHAR_CNAME=\"sgemm_thread_nn\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DNN gemm.c -o sgemm_thread_nn.o
gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_nt -DASMFNAME=_sgemm_thread_nt_ -DNAME=sgemm_thread_nt_ -DCNAME=sgemm_thread_nt -DCHAR_NAME=\"sgemm_thread_nt_\" -DCHAR_CNAME=\"sgemm_thread_nt\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DNT gemm.c -o sgemm_thread_nt.o
gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_tn -DASMFNAME=_sgemm_thread_tn_ -DNAME=sgemm_thread_tn_ -DCNAME=sgemm_thread_tn -DCHAR_NAME=\"sgemm_thread_tn_\" -DCHAR_CNAME=\"sgemm_thread_tn\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DTN gemm.c -o sgemm_thread_tn.o
gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_tt -DASMFNAME=_sgemm_thread_tt_ -DNAME=sgemm_thread_tt_ -DCNAME=sgemm_thread_tt -DCHAR_NAME=\"sgemm_thread_tt_\" -DCHAR_CNAME=\"sgemm_thread_tt\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DTT gemm.c -o sgemm_thread_tt.o
gcc-9 -c -DTHREADED_LEVEL3 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_ssymm_thread_LU -DASMFNAME=_ssymm_thread_LU_ -DNAME=ssymm_thread_LU_ -DCNAME=ssymm_thread_LU -DCHAR_NAME=\"ssymm_thread_LU_\" -DCHAR_CNAME=\"ssymm_thread_LU\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX -ULOWER -URSIDE -DNN symm_k.c -o ssymm_thread_LU.o
gcc-9 -c -DTHREADED_LEVEL3 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_ssymm_thread_LL -DASMFNAME=_ssymm_thread_LL_ -DNAME=ssymm_thread_LL_ -DCNAME=ssymm_thread_LL -DCHAR_NAME=\"ssymm_thread_LL_\" -DCHAR_CNAME=\"ssymm_thread_LL\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX -DLOWER -URSIDE -DNN symm_k.c -o ssymm_thread_LL.o
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~level3_thread.c:354~
:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [sgemm_thread_nt.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [sgemm_thread_nn.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [ssymm_thread_LU.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [ssymm_thread_LL.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [sgemm_thread_tn.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [sgemm_thread_tt.o] Error 1
make: *** [libs] Error 1

==> Formula
Path: /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/openblas.rb
==> Configuration
HOMEBREW_VERSION: 2.2.5-56-g23ab620
ORIGIN: https://github.com/Homebrew/brew
HEAD: 23ab620cd412a5b80d300055e18e5325a8d8edee
Last commit: 9 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core/
Core tap HEAD: e0100449735763a88accb342e0fcd6e3ecaf4da6
Core tap last commit: 4 minutes ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CACHE: /Users/brew/Library/Caches/Homebrew
HOMEBREW_LOGS: /Users/brew/Jenkins/workspace/core/logs
HOMEBREW_DEV_CMD_RUN: 1
HOMEBREW_FAIL_LOG_LINES: 150
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_GIT_EMAIL: [email protected]
HOMEBREW_GIT_NAME: BrewTestBot
HOMEBREW_HOME: /Users/brew/Jenkins/workspace/core/home
HOMEBREW_NO_ANALYTICS: 1
HOMEBREW_NO_AUTO_UPDATE: 1
HOMEBREW_NO_EMOJI: 1
CPU: hexa-core 64-bit penryn
Homebrew Ruby: 2.6.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.3/bin/ruby
Clang: 11.0 build 1100
Git: 2.21.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 13.0.2, 1.8.0_242
macOS: 10.14.6-x86_64
CLT: 10.3.0.0.1.1562985497
Xcode: 11.3.1
Error: openblas 0.3.8 did not build
XQuartz: 2.7.11 => /opt/X11
==> ENV
HOMEBREW_CC: gcc-9
HOMEBREW_CXX: g++-9
MAKEFLAGS: -j6
CMAKE_PREFIX_PATH: /usr/local
CMAKE_INCLUDE_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/libxml2:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
CMAKE_LIBRARY_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_PATH: /usr/local/opt/isl/lib/pkgconfig:/usr/local/opt/mpfr/lib/pkgconfig
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14
HOMEBREW_GIT: git
HOMEBREW_SDKROOT: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk
ACLOCAL_PATH: /usr/local/share/aclocal
PATH: /usr/local/Homebrew/Library/Homebrew/shims/mac/super:/usr/local/opt/gcc/bin:/usr/bin:/bin:/usr/sbin:/sbin

Source

zbeekman

Most helpful comment

Looking good so far, currently trying to assess Power8-bigendian issues but will postpone those (and the other open topics in the milestone) to after 0.3.9 if no immediate solution presents itself.

martin-frbg on 29 Feb 2020

🚀1 🎉1

All 19 comments

No, definitely not related to #2416 - that is "only" about not linking to libopenmp, what you have here is a weird mixture of what appears to be an x86_64 build environment trying to understand an arm64 instruction. Were you trying to cross-compile for iOS by any chance ? Then this is the host compiler (or assembler) getting run instead of the arm64 cross-compiler, check your path settings.

martin-frbg on 15 Feb 2020

This is building in Homebrew's sanitized environment, so it's built exactly the same way 0.3.7 was built. It is definitely NOT trying to cross compile, intentionally or accidentally: it is building x86_64 binaries for each of the 3 currently supported macOS releases. Homebrew strips the environment down and then explicitly sets all environment variables controlling the build.

Normally packages are built with Apple's native clang compiler (and gfortran). But for OpenBLAS 0.3.7 (and previous versions) were being built with the latest stable GCC instead of clang.

Here are the environment variables that were being explicitly set specifically for OpenBLAS:

DYNAMIC_ARCH=1
USE_OPENMP=1
NO_AVX512=1
CC=/path/to/gcc-9
FC=gfortran

The normal build environment for all packages is visible at the bottom of the error log above (except the formula had been marked as fails_with :clang so normally the C/C++ compilers would be clang and clang++):

HOMEBREW_CC: gcc-9
HOMEBREW_CXX: g++-9
MAKEFLAGS: -j6
CMAKE_PREFIX_PATH: /usr/local
CMAKE_INCLUDE_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/libxml2:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
CMAKE_LIBRARY_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_PATH: /usr/local/opt/isl/lib/pkgconfig:/usr/local/opt/mpfr/lib/pkgconfig
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14
HOMEBREW_GIT: git
HOMEBREW_SDKROOT: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk
ACLOCAL_PATH: /usr/local/share/aclocal
PATH: /usr/local/Homebrew/Library/Homebrew/shims/mac/super:/usr/local/opt/gcc/bin:/usr/bin:/bin:/usr/sbin:/sbin

The commands to build and install OpenBLAS are:

system "make", "CC=gcc-9", "FC=gfortran", "libs", "netlib", "shared"
system "make", "PREFIX=/usr/local/Cellar/openblas/0.3.8", "install"

It's possible these need updating.

But simply switching to clang or removing USE_OPENMP=1 resolves the issue.

Building packages with GCC is admittedly less well-tested than with clang, but, nevertheless, gets used fairly regularly. I'll ping some of the other Homebrew maintainers and see if they have any thoughts on the matter. In the meantime I had linked the full Jenkins build logs in my original post, and if you're able to take a quick look at them to at least point me in the right direction it would be much appreciated.

zbeekman on 16 Feb 2020

Can you print the contents of config.h after make fails?

isuruf on 16 Feb 2020

https://github.com/xianyi/OpenBLAS/pull/2279/files#diff-7cf3cca9d0a67dfeb94e7187f4adba11R263-R275 is the issue here.

isuruf on 16 Feb 2020

See #2424

isuruf on 16 Feb 2020

@isuruf thanks - the intention with #2279 was to facilitate cross-compiling for iOS, removing the need to hack various files as described in the wiki. To that end I cobbled together information from said wiki entry, success reports from earlier issues and bits and pieces from stackoverflow. The result _appeared_ to work in the Travis enviroment but I admit I do not have local OSX nor iOS hardware for testing.

martin-frbg on 16 Feb 2020

I don't have iOS hardware for testing. I checked that I can cross-compile using #2424. After that PR is merged, I'll update the wiki.

isuruf on 16 Feb 2020

The result _appeared_ to work in the Travis enviroment but I admit I do not have local OSX nor iOS hardware for testing.

FWIW, I find Travis-CI somewhat unreliable for macOS. Github Actions work much more reliably in my experience.

Would you accept a PR to do nightly macOS builds of the development branch using the homebrew build formula? (With Github Actions.) That way you can see when something breaks either on your end or, possibly, on the Homebrew or macOS end.

@isuruf here is config.h:

#define OS_DARWIN   1
#define ARCH_ARM64  1
#define C_GCC   1
#define __64BIT__   1
#define FUNDERSCORE _
#define PTHREAD_CREATE_FUNC pthread_create
#define BUNDERSCORE _
#define NEEDBUNDERSCORE 1
#define HASWELL
#define L1_CODE_SIZE 32768
#define L1_CODE_ASSOCIATIVE 8
#define L1_CODE_LINESIZE 64
#define L1_DATA_SIZE 32768
#define L1_DATA_ASSOCIATIVE 8
#define L1_DATA_LINESIZE 64
#define L2_SIZE 262144
#define L2_ASSOCIATIVE 8
#define L2_LINESIZE 64
#define ITB_SIZE 2097152
#define ITB_ASSOCIATIVE 0
#define ITB_ENTRIES 8
#define DTB_SIZE 4096
#define DTB_ASSOCIATIVE 4
#define DTB_DEFAULT_ENTRIES 64
#define HAVE_CMOV
#define HAVE_MMX
#define HAVE_SSE
#define HAVE_SSE2
#define HAVE_SSE3
#define HAVE_SSSE3
#define HAVE_SSE4_1
#define HAVE_SSE4_2
#define HAVE_AVX
#define HAVE_AVX2
#define HAVE_FMA3
#define HAVE_CFLUSH
#define NUM_SHAREDCACHE 2
#define NUM_CORES 8
#define CORE_HASWELL
#define CHAR_CORENAME "HASWELL"
#define GEMM_MULTITHREAD_THRESHOLD  4

zbeekman on 17 Feb 2020

yeah, if OS_DARWIN, ARCH_ARM64 is definitely not appropriate.

zbeekman on 17 Feb 2020

@zbeekman, can you check #2424?

It's a hard to catch error. If CC=gcc or CC=clang, you wouldn't get this error. You'll get this if CC=gcc-9.

isuruf on 17 Feb 2020

Aha... so that is what triggered the CROSS=1, I believe this "ingenious" method of finding out if we are cross-compiling has been fragile from the start... I have merged #2424 now, and I think an additional PR to add nightly (or just periodic) OSX builds outside Travis would be very welcome if there is infrastructure for it. (I have not looked into the Github Actions mechanism yet)

martin-frbg on 17 Feb 2020

👍1

@isuruf

It's a hard to catch error. If CC=gcc or CC=clang, you wouldn't get this error. You'll get this if CC=gcc-9.

Yes, this is true. But, I think because of the wiki-page on installing on macOS and/or because of issues with older clangs used by Apple in the past, the formula was marked as:

fails_with :clang

Since Homebrew is a rolling release package manager, when things cannot be built with native Apple clang, they are built with the latest stable GCC, i.e., GCC 9.

@martin-frbg

I think an additional PR to add nightly (or just periodic) OSX builds outside Travis would be very welcome if there is infrastructure for it. (I have not looked into the Github Actions mechanism yet)

Cool, nearly done, should have a PR soon.

zbeekman on 17 Feb 2020

FYI @martin-frbg:

PR for adding nightly testing of develop built the way homebrew does for releases (i.e. binary "bottles" built following the package's "formula") is at https://github.com/xianyi/OpenBLAS/pull/2426

I'm waiting for the github actions tests to pass locally for me before marking it as ready for review.

zbeekman on 17 Feb 2020

FYI @martin-frbg it looks like, due to OpenMP issues, OpenBLAS in Homebrew will likely stay with GCC as the compiler. So right now I'm investigating back-porting these fixes onto version 0.3.8 but I was wondering:

Do you know (approximately) how soon the next OpenBLAS release will drop?

We plan to wait ~ 1 week for the next nwchem release because they depend on OpenBLAS and are failing to rebuild due to an unrelated issue with the current version. But if OpenBLAS' next release is far off, I'll backport the patch and upgrade Homebrew to 0.3.8. If it's around the corner, we'll wait for the next OpenBLAS release to avoid patching.

zbeekman on 20 Feb 2020

Good question, there have been several serious bugs uncovered since the release (some new, some very old) so a quick followup would be desirable. I am currently trying to sort out powerpc and it would be good if your #2428 could be fixed so give it one or two weeks ?

martin-frbg on 20 Feb 2020

I’ll check in again once nwchem is ready to ship unless you beat them to releasing.

zbeekman on 21 Feb 2020

@martin-frbg @isuruf NWChem has shipped. How close are you guys to a release? Should I patch 0.3.8?

Also, @isuruf could you link me to your conda recipe for building GCC with libomp instead of libgomp? I couldn't find it anywhere.

At any rate I would like to resurect https://github.com/Homebrew/homebrew-core/pull/50252 and get OpbenBLAS updated soon.

zbeekman on 29 Feb 2020

@zbeekman, I'm not a github committer of this project. You'll have to ask @martin-frbg.

We don't build gcc anymore in conda. Just gfortran. https://github.com/conda-forge/gfortran_impl_osx-64-feedstock/blob/master/recipe/build.sh#L38-L41

isuruf on 29 Feb 2020

Looking good so far, currently trying to assess Power8-bigendian issues but will postpone those (and the other open topics in the milestone) to after 0.3.9 if no immediate solution presents itself.

martin-frbg on 29 Feb 2020

🚀1 🎉1

Was this page helpful?

0 / 5 - 0 ratings