Julia: openblas segmentation fault

[montyvesselinov]

using julia 1.0.3, i get regularly:

OpenBLAS : Program will terminate because you tried to start too many threads.
Segmentation fault: 11

After that julia REPL quits.

[StefanKarpinski]

How many threads did you try to start?

[brianborchers]

This typically happens when the code that is calling an OpenBLAS routine is itself multithreaded. Each invocation of OpenBLAS will itself start as many threads as you have processor cores, and then you'll end up trying to start n^2 threads where n is the number of cores. Ideally, codes should be aware of how many threads they're using and be able to control the number of threads used by subroutines called from the BLAS library. One common strategy is to use multithreading at the higher level and then use only single threaded BLAS calls.

Older versions of OpenBLAS had a fixed size buffer that would overflow (leading to this error message) if you tried to call BLAS routines from a bunch of different threads. I believe that the most recent development version makes this unlimited. However, you'd still have a very significant performance problem if you tried to run n^2 threads on n cores.

There is no general solution to this problem- sometimes it's best to make the higher level code single threaded and let the BLAS routines run multithreaded, while other times it is better to run the higher level code multithreaded and run the BLAS routines single threaded. Ideally, there would be some way for the programmer to control this at the higher level and tune the code for best performance.

There's no easy way to control this the number of threads a subroutine is allowed to use if you're using pthreads multithreading. With OpenMP multithreading, it should in theory be possible for OpenBLAS to be built using OpenMP in such a way that it respects OpenMP's methods for controlling the number of threads used by called routines but I'm not sure that this is working correctly (This is properly implemented in MKL.) Of course, Julia would also have to use OpenMP to make this work.

It's unlikely that you'll get a very quick solution to this problem. In the short term, there's a good chance that your code will work (and probably work most efficiently) if you simply call the BLAS routines only from single threaded portions of your code. OpenBLAS should generally make good use of the available cores. In the medium term (weeks to months), you'll probably see a version of Julia that works (because the limit on the number of threads has been eliminated) but ends up oversubscribing your cores with n^2 threads running on n cores and doesn't perform well. In that case you'll probably find that the code performs better if it only calls BLAS routines from within single threaded Julia code.

[montyvesselinov]

I did not specify the number of threads. Checking the code performance I see that the number of threads is 10. I run this on Mac OS Mojave 10.14.1, 2.9 GHz Intel Core i9. The code runs ok until the job is completed. However, restarting the same script from REPL causes the seg fault.

I see there are many other reports about this openblas issue:

https://www.google.com/search?q=OpenBLAS+%3A+Program+will+terminate+because+you+tried+to+start+too+many+threads.+Segmentation+fault%3A+11&oq=OpenBLAS+%3A+Program+will+terminate+because+you+tried+to+start+too+many+threads.+Segmentation+fault%3A+11&aqs=chrome..69i57.334j0j7&sourceid=chrome&ie=UTF-8

My concern is that julia is crashing and quits.

[montyvesselinov]

The "same" (it is not exactly the same because the 1.0 vs 0.6 language differences but technically is the same) code runs ok on 0.6.4.

[brianborchers]

I believe that this is working with 0.6.4 is using an older version of OpenBLAS that doesn't have a particular bug that was introduced in an attempt to improve the code. My comments above deal with some broader issues that don't appear to be the particular problem here.

[StefanKarpinski]

Sigh. This seems to be an upstream issue that we may not be able to do much about until OpenBLAS fixes it. Any thoughts on how to work around the underlying OpenBLAS issue? To be clear are you using threads in Julia at all? Do you have a reproducer for this crash?

[montyvesselinov]

I will try to design a reproducer. (The code i am using is not open source yet; i will try to setup an open-source reproducer). At the moment occurs after several executes. What is the magic number i do not know yet.

I did not specify explicitly anything about the number of workers or threads in the code.

julia> Threads.nthreads()
1

julia> Base.Sys.CPU_THREADS
12

From the past, my experience was that JULIA_NUM_THREADS does not help here. But i will try to with 1 and see what happens.

[ImreSamu]

@montyvesselinov :

Lot of thread related fixes in the https://github.com/xianyi/OpenBLAS/releases
so please test with Julia v1.1.0-rc2 and julia nightly

Julia | OpenBlas
------ | -----------
V0.6.4 | v0.3.0
v1.0.3 | v0.3.2
V1.1.0-rc2 | v0.3.3
master/nightlies | v0.3.5

[montyvesselinov]

JULIA_NUM_THREADS=1 in the startup file did not help

[montyvesselinov]

will test Julia v1.1.0-rc2 and julia nightly

[montyvesselinov]

Tested extensively with Julia v1.1.0-rc2. OpenBlas segfault does not appear here. Now it all works. Thank you very much!