Julia: Julia release-0.5 fails to build on FreeBSD 10.3-RELEASE-p4

[hanzer]

Today:

sudo freebsd-update fetch
sudo freebsd-update install
sudo pkg upgrade
git config --global core.autocrlf input
git clone https://github.com/JuliaLang/julia.git
cd julia/
git checkout release-0.5
gmake MARCH=native FC=gfortran47

Ends with error:

[snip]
  CC       src/libuv_la-uv-common.lo
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c: In function 'uv_thread_setaf
finity':
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:378:3: error: unknown type na
me 'cpu_set_t'
   cpu_set_t cpuset;
   ^
In file included from /usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:26:0:
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/include/uv.h:1474:33: error: 'CPU_SETSIZE' un
declared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE

[hanzer]

Just in case it is relevant in some way, this is on an x86/32-bit machine.

[tkelman]

probably related to https://github.com/JuliaLang/libuv/pull/31? cc @kpamnany

[hanzer]

After this change to Make.inc (I lack sufficient understanding of the issues, factors, and ramifications to know if this is a reasonable thing to do, but just as a superficial experiment):

USE_SYSTEM_LIBUV:=1

and the build process moves forward a bit before the next problem:

 /usr/bin/install -c -m 644 include/libunwind-common.h '/usr/home/hanzer/build/julia/usr-staging/libunwind-1.1-julia2/usr/home/hanzer/build/julia/usr/include'
src/s_cosl.c:44:2: error: #error "Unsupported long double format"
 #error "Unsupported long double format"
  ^
gmake[2]: *** [Make.inc:66: src/s_cosl.c.o] Error 1
gmake[1]: *** [/usr/home/hanzer/build/julia/deps/openlibm.mk:11: build/openlibm-71e79eb6f74d3f04ce724195b8ef5846a70d281e/libopenlibm.so] Error 2
gmake: *** [Makefile:81: julia-deps] Error 2

Following the same pattern, changing Make.inc with:

`USE_SYSTEM_LIBUNWIND:=1`

results in:

src/s_cosl.c:44:2: error: #error "Unsupported long double format"
 #error "Unsupported long double format"
  ^
gmake[2]: *** [Make.inc:66: src/s_cosl.c.o] Error 1
gmake[1]: *** [/usr/home/hanzer/build/julia/deps/openlibm.mk:11: build/openlibm-71e79eb6f74d3f04ce724195b8ef5846a70d281e/libopenlibm.so] Error 2
gmake: *** [Makefile:81: julia-deps] Error 2

so changing Make.inc again, with the additional:

USE_SYSTEM_LIBM:=1

and the build moves forward. To speed things up, trying with dependencies supplied from the FreeBSD package system (e.g., pkg search pcre if found then sudo pkg install pcre):

USE_SYSTEM_LLVM:=1
USE_SYSTEM_LIBUNWIND:=1
USE_SYSTEM_PCRE:=1
USE_SYSTEM_LIBM:=1
USE_SYSTEM_OPENLIBM:=0
UNTRUSTED_SYSTEM_LIBM:=0
USE_SYSTEM_OPENSPECFUN:=0
USE_SYSTEM_DSFMT:=0
USE_SYSTEM_BLAS:=1
USE_SYSTEM_LAPACK:=1
USE_SYSTEM_FFTW:=1
USE_SYSTEM_GMP:=1
USE_SYSTEM_MPFR:=1
USE_SYSTEM_ARPACK:=1
USE_SYSTEM_SUITESPARSE:=1
USE_SYSTEM_LIBUV:=1
USE_SYSTEM_UTF8PROC:=1
USE_SYSTEM_MBEDTLS:=1
USE_SYSTEM_LIBSSH2:=1
USE_SYSTEM_LIBGIT2:=1
USE_SYSTEM_PATCHELF:=1

which results in:

/usr/home/hanzer/build/julia/deps/SuiteSparse_wrapper.c:2:21: fatal error: cholmod.h: No such file or directory
 #include <cholmod.h>
                     ^
compilation terminated.
gmake[1]: *** [/usr/home/hanzer/build/julia/deps/suitesparse.mk:108: /usr/home/hanzer/build/julia/usr/lib/libsuitesparse_wrapper.so] Error 1
gmake: *** [Makefile:81: julia-deps] Error 2

so:
USE_SYSTEM_SUITESPARSE:=0

compiling then:

 gmake[1]: llvm-config: Command not found

pkg info | grep llvm
llvm37-3.7.1_2                 LLVM and Clang
llvm38-3.8.0_3                 LLVM and Clang

lvm-config is installed as llvm-config37 and llvm-config38, as are the other llvm- programs. I don't suppose there is a simple top-level configuration for such things?

[tkelman]

You should be able to set an LLVM_CONFIG variable either on the gmake command line or in a Make.user file (create it at the top level and the makefiles will include it if present, useful to avoid having to patch the existing files, you can just override certain variables instead).

[hanzer]

Trying gmake MARCH=native FC=gfortran47 LLVM_CONFIG=llvm-config38

with Make.inc modified like:

USE_SYSTEM_LLVM:=1
USE_SYSTEM_LIBUNWIND:=1
USE_SYSTEM_PCRE:=1
USE_SYSTEM_LIBM:=1
USE_SYSTEM_OPENLIBM:=0
UNTRUSTED_SYSTEM_LIBM:=0
USE_SYSTEM_OPENSPECFUN:=0
USE_SYSTEM_DSFMT:=0
USE_SYSTEM_BLAS:=1
USE_SYSTEM_LAPACK:=1
USE_SYSTEM_FFTW:=1
USE_SYSTEM_GMP:=1
USE_SYSTEM_MPFR:=1
USE_SYSTEM_ARPACK:=1
USE_SYSTEM_SUITESPARSE:=0
USE_SYSTEM_LIBUV:=1
USE_SYSTEM_UTF8PROC:=1
USE_SYSTEM_MBEDTLS:=1
USE_SYSTEM_LIBSSH2:=1
USE_SYSTEM_LIBGIT2:=1
USE_SYSTEM_PATCHELF:=1

results in:

gmake[2]: ** No rule to make target '/usr/home/hanzer/build/julia/src/support/hashing.o', needed by '/usr/home/hanzer/build/julia/src/support/libsupport.a'. Stop.
gmake[1]: ** [Makefile:158: support/libsupport.a] Error 2
gmake: *** [Makefile:55: julia_flisp.boot.inc.phony] Error 2

[tkelman]

That might be the fault of USE_SYSTEM_LIBUV which isn't likely to work right now. Maybe try modifying deps/libuv.version to find an older commit of https://github.com/JuliaLang/libuv that builds on freebsd? We used to have a few contributors more actively testing on that platform and submitting patches to keep things working.

[johnmyleswhite]

Perhaps the easiest thing to do is to submit a PR that removes the line that troubles you? I suspect no one would be not reluctant to remove official claims of support for a platform that they agree is no longer actively supported.

[tkelman]

Sorry things aren't working. We don't currently have buildbots or CI for FreeBSD, or an active maintainer for it. The last time that line of the readme was touched was in 2012 (https://github.com/JuliaLang/julia/blame/5838a12ba037f40b31affe5b408edcf9ce4bed28/README.md#L53), before Julia had buildbots or CI for _any_ platform. You're correct that we should do a better job about keeping the readme up to date. It tends to get updated when a platform initially starts working (e.g. recent arm and power support), but not so much when support for a platform regresses, if no one actively notices and reports issues like this one.

[kpamnany]

Julia's libuv is required to build Julia since upstream libuv hasn't merged https://github.com/libuv/libuv/pull/597 yet (adds uv_thread_detach() and uv_thread_setaffinity()).

FreeBSD appears to have thread affinity support (http://www.freebsd.org/cgi/man.cgi?manpath=FreeBSD%208.2-RELEASE&query=pthread_setaffinity_np), but CPU set management appears to be different. Simplest answer right now is to patch libuv to exclude affinity setting on FreeBSD also. Add FreeBSD at src/uv-common.c:423:

#if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)

It might be better to invert the #if to only select those OSes we know support affinity setting rather than excluding the ones that don't...

I can't raise a PR for this anytime soon, sorry. Also, can't speak to the other issues.

Also, this whole problem should go away in the next threading runtime because I'm using hwloc there.

[hanzer]

There currently is no src/uv-common.c in my git checkout release-0.5

But find . -type f -name "uv-common.c" finds this:
./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c

A gmake cleanall followed by a gmake distcleanall
then a gmake MARCH=native FC=gfortran47 LLVM_CONFIG=llvm-config38
with this Make.inc modification:

USE_SYSTEM_LLVM:=1
USE_SYSTEM_LIBUNWIND:=1
USE_SYSTEM_PCRE:=1
USE_SYSTEM_LIBM:=1
USE_SYSTEM_OPENLIBM:=0
UNTRUSTED_SYSTEM_LIBM:=0
USE_SYSTEM_OPENSPECFUN:=0
USE_SYSTEM_DSFMT:=0
USE_SYSTEM_BLAS:=1
USE_SYSTEM_LAPACK:=1
USE_SYSTEM_FFTW:=1
USE_SYSTEM_GMP:=1
USE_SYSTEM_MPFR:=1
USE_SYSTEM_ARPACK:=1
USE_SYSTEM_SUITESPARSE:=0
USE_SYSTEM_LIBUV:=0
USE_SYSTEM_UTF8PROC:=1
USE_SYSTEM_MBEDTLS:=1
USE_SYSTEM_LIBSSH2:=1
USE_SYSTEM_LIBGIT2:=1
USE_SYSTEM_PATCHELF:=1

fails with:

config.status: executing depfiles commands
config.status: executing libtool commands
  CC       src/libuv_la-fs-poll.lo
  CC       src/libuv_la-inet.lo
  CC       src/libuv_la-threadpool.lo
  CC       src/libuv_la-uv-common.lo
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c: In function 'uv_thread_setaf
finity':
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:378:3: error: unknown type na
me 'cpu_set_t'
   cpu_set_t cpuset;
   ^
In file included from /usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:26:0:
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/include/uv.h:1474:33: error: 'CPU_SETSIZE' un
declared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE

(and there is no src/uv-common.c file)

[yuyichao]

./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c this is the src/uv-common.c

[hanzer]

Changed ./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c
to this:

    422 #else
    423 #if defined(__APPLE__) || defined(_AIX) || || defined(__FreeBSD__)
    424   return -ENOTSUP;
    425 #else
    426   int i;
    427   cpu_set_t cpuset;

The error is a bit different:

  CC       src/libuv_la-threadpool.lo
  CC       src/libuv_la-uv-common.lo
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c: In function 'uv_thread_setaf
finity':
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:378:3: error: unknown type na
me 'cpu_set_t'
   cpu_set_t cpuset;
   ^
In file included from /usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:26:0:
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/include/uv.h:1474:33: error: 'CPU_SETSIZE' un
declared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE
                                 ^
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:380:23: note: in expansion of
 macro 'UV_CPU_SETSIZE'
   assert(mask_size >= UV_CPU_SETSIZE);
                       ^

[hanzer]

err, correction: changed ./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c like this:

    374 #if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)
    423 #if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)

and the build gets to:

    CC src/jl_uv.o
    CC src/simplevector.o
    CC src/APInt-C.o
g++: error: unrecognized command line option '-Wcovered-switch-default'
g++: error: unrecognized command line option '-fcolor-diagnostics'
gmake[1]: *** [Makefile:121: APInt-C.o] Error 1
gmake: *** [Makefile:90: julia-src-release] Error 2

g++ -v

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc48/gcc/i386-portbld-freebsd10.1/4.8.5/lto-wrapper
Target: i386-portbld-freebsd10.1
Configured with: /wrkdirs/usr/ports/lang/gcc/work/gcc-4.8.5/configure --disable-multilib --disable-bootstrap --disable-nls --enable-gnu-indirect-function --libdir=/usr/local/lib/gcc48 --libexecdir=/usr/local/libexec/gcc48 --program-suffix=48 --with-as=/usr/local/bin/as --with-gmp=/usr/local --with-gxx-include-dir=/usr/local/lib/gcc48/include/c++/ --with-ld=/usr/local/bin/ld --with-pkgversion='FreeBSD Ports Collection' --with-system-zlib --with-ecj-jar=/usr/local/share/java/ecj-4.5.jar --enable-languages=c,c++,objc,fortran,java --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/info/gcc48 --build=i386-portbld-freebsd10.1
Thread model: posix
gcc version 4.8.5 (FreeBSD Ports Collection) 

[vtjnash]

It seems that the compiler used to build llvm supported those flags, but the compiler you are trying to use to build Julia does not. I think the userspace of FreeBSD may be entirely built with clang now?

[hanzer]

Yes, I believe so.

NOTE: Clang/LLVM is the system compiler on several platforms in FreeBSD 10.0 and later, and GCC is not installed by default.

https://wiki.freebsd.org/BuildingFreeBSDWithClang

[yuyichao]

In another word, since llvm-config dumps the compiler flags it is compiled with, you should compile julia with a compiler that is compatible with the compiler llvm is compiled with.

[hanzer]

To move forward, last night I set USE_SYSTEM_LLVM:=0 in Make.inc and launched the build. The results of that are:

In file included from /usr/home/hanzer/build/julia/src/support/ios.h:7:0,
                 from /usr/home/hanzer/build/julia/src/support/libsupport.h:14,
                 from /usr/home/hanzer/build/julia/src/julia.h:12,
                 from /usr/home/hanzer/build/julia/src/threading.c:22:
/usr/home/hanzer/build/julia/src/threading.c: In function 'jl_start_threads':
/usr/home/hanzer/build/julia/usr/include/uv.h:1474:33: error: 'CPU_SETSIZE' undeclared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE
                                 ^
/usr/home/hanzer/build/julia/src/threading.c:567:20: note: in expansion of macro 'UV_CPU_SETSIZE'
     char *cp, mask[UV_CPU_SETSIZE];
                    ^
/usr/home/hanzer/build/julia/usr/include/uv.h:1474:33: note: each undeclared identifier is reported only once for each function it appears in
 #  define UV_CPU_SETSIZE        CPU_SETSIZE
                                 ^
/usr/home/hanzer/build/julia/src/threading.c:567:20: note: in expansion of macro 'UV_CPU_SETSIZE'
     char *cp, mask[UV_CPU_SETSIZE];
                    ^
/usr/home/hanzer/build/julia/src/threading.c:567:15: warning: unused variable 'mask' [-Wunused-variable]
     char *cp, mask[UV_CPU_SETSIZE];
               ^
gmake[1]: *** [Makefile:117: threading.o] Error 1
gmake: *** [Makefile:90: julia-src-release] Error 2

I guess an #if defined(__FreeBSD__) is needed in usr/include/uv.h?

I am trying (blindly (I don't know what this means)):

   1471 # if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)
   1472 #  define UV_CPU_SETSIZE        8
   1473 # else
   1474 #  define UV_CPU_SETSIZE        CPU_SETSIZE
   1475 # endif
   1476 #endif

and compilation continues.

[hanzer]

Now the build ends with:

In file included from /usr/home/hanzer/build/julia/src/intrinsics.cpp:7:0,
                 from /usr/home/hanzer/build/julia/src/codegen.cpp:1504:
/usr/home/hanzer/build/julia/src/ccall.cpp: In function 'bool runtime_sym_gvs(const char*, const char*, MT&&, llvm::GlobalVariable*&, llvm::GlobalVariable*&, void**)':
/usr/home/hanzer/build/julia/src/ccall.cpp:112:17: error: 'to_string' is not a member of 'std'
         name += std::to_string(globalUnique++);
                 ^
gmake[1]: *** [Makefile:121: codegen.o] Error 1
gmake: *** [Makefile:90: julia-src-release] Error 2

[hanzer]

In another word, since llvm-config dumps the compiler flags it is compiled with, you should compile julia with a compiler that is compatible with the compiler llvm is compiled with.

Can you give an example of how this might be accomplished?

[tkelman]

Possibly need to set different combinations of USECLANG=1 and/or USEGCC=0 if those aren't already getting set by default for freebsd?

[hanzer]

The compile times are too long for [systematic] guessing, I'm giving up on this for now. The Julia RPM (Version 0.4.6) installed easily on CentOS-7 (x86_64-unknown-linux-gnu), I can explore it there.

I'll probably check periodically for any progress on the [potential(?)] FreeBSD port.

...Scientific analysis software without rigorous quality assurance and engineered/maintained releases; how many actual scientists and engineers rely on that? Should they?

[ararslan]

Scientific analysis software without rigorous quality assurance and engineered/maintained releases; how many actual scientists and engineers rely on that?

I think that's an unfair accusation to make. Julia has well maintained releases (note all of the backport tags in PRs) and excellent QA. Plus, despite being a fairly young language, Julia is used for a lot of serious science and engineering. For major platforms on which continuous integration, testing, and building is regularly run, it's known to work fantastically well. As was said previously, there are no active maintainers on FreeBSD; the only way we'll know whether there are issues on that or any platform other than Windows, OS X, and Linux is through issues such as this. So thank you very much for reporting the issue! Now it's known and can be tracked.

Should they?

If anything, I'd encourage _more_ scientists and engineers to rely on Julia. 😉

[hanzer]

...has well maintained releases [snip] and excellent QA. [snip] If anything, I'd encourage more scientists and engineers to rely on Julia.

It is probably important not to let sentiment and enthusiasm become a substitute for traction with reality. The systems engineering of high reliability tools requires sobriety and rigor. This might be a lot to ask from the mainstream computer programming technologies where a pervading goal seems to be the creation of cult-like fanatics (battology in typical software documentation is so severe one can imagine what the ancients might have meant by "idolatry").

I suspect significant advancement could be made with more sophisticated socio-technical systems supporting the measurement, analysis, and regulation of Limbic System Class Information Processing within the software development communities.

[tkelman]

Uh, what?

Try latest master, this was probably fixed by https://github.com/JuliaLang/julia/pull/18063.

[JeffBezanson]

more sophisticated socio-technical systems

In other words, adding freebsd to our CI testing matrix.

[ararslan]

Julia now builds out of the box on FreeBSD 11.0-RELEASE.