Crystal: Create a FreeBSD package

[yonas]

Crystal looks very interesting. Can we create a package for FreeBSD?

[tosmi]

i'm also interested in a freebsd package. what is the current state of freebsd support? there are packages for 0.19.x, but not for newer releases. are there any build instructions available?

[ysbaddaden]

FreeBSD is supported, just like OpenBSD. We didn't update the omnibus that builds crystal releases to compile one for FreeBSD. It's also not easy to know which version(s) of FreeBSD to support —nobody in the team uses it regularly.

You can download a 0.19.x binary and use it to build a newer release. Hoping the binary is compatible with newer FreeBSD releases.

You may try to build 0.23.x directly but it's likely 0.19 won't be capable to build it (breaking changes), so you'll may want to build some intermediate versions.

Alternatively, if you have access to a Linux or macOS, you may cross compile an object file then link it on the target host, but that's more complicated.

[tosmi]

ok, i'll use the 0.19 binaries and will try to build a newer version. i'm a freebsd user and could offer some help getting regular crystal builds working. i'm far from an expert but willing to learn...
i could also work on getting crystal into the freebsd ports system. i'm not a commiter but know the procedures necessary.

if you are interested just tell me where to start.

thanks
toni

[yonas]

@ysbaddaden I get this error when building v0.23.1 using crystal-0.19.4-2-freebsd10.1-x86_64.tar.gz on FreeBSD 11.1-RELEASE.

# gmake
Using /usr/local/bin/llvm-config39 [version=3.9.1]         
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib                                                                                                               
Error in ./src/compiler/crystal.cr:4: while requiring "./crystal/**"                                                   

require "./crystal/**"                                     
^                                                          

in ./src/compiler/crystal/command.cr:11: while requiring "json"                                                        

require "json"                                             
^                                                          

in ./src/json.cr:70: while requiring "./json/*"            

require "./json/*"                                         
^                                                          

in ./src/json/to_json.cr:96: JSON::Builder is not a module, it's a class                                               

module JSON::Builder                                       
       ^                                                   

gmake: *** [Makefile:117: .build/crystal] Error 1          

I've tried a bunch of different versions. The furthest I've got is with 0.20.1:

# gmake
Using /usr/local/bin/llvm-config39 [version=3.9.1]
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateCompileUnit':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateCompileUnit+0xb9): undefined reference to `llvm::DIBuilder::createCompileUnit(unsigned int, llvm::StringRef, llvm::StringRef, llvm::StringRef, bool, llvm::StringRef, unsigned int, llvm::StringRef, llvm::DIBuilder::DebugEmissionKind, bool)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateFunction':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateFunction+0xcd): undefined reference to `llvm::DIBuilder::createFunction(llvm::DIDescriptor, llvm::StringRef, llvm::StringRef, llvm::DIFile, unsigned int, llvm::DICompositeType, bool, bool, unsigned int, unsigned int, bool, llvm::Function*, llvm::MDNode*, llvm::MDNode*)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateLexicalBlock':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateLexicalBlock+0x20): undefined reference to `llvm::DIBuilder::createLexicalBlock(llvm::DIDescriptor, llvm::DIFile, unsigned int, unsigned int)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateSubroutineType':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateSubroutineType+0x22): undefined reference to `llvm::DIBuilder::createSubroutineType(llvm::DIFile, llvm::DITypedArray<llvm::DIRef<llvm::DIType> >, unsigned int)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateLocalVariable':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateLocalVariable+0x5e): undefined reference to `llvm::DIScope::getRef() const'
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateLocalVariable+0x6a): undefined reference to `llvm::DIRef<llvm::DIType>::DIRef(llvm::Metadata const*)'
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateLocalVariable+0xac): undefined reference to `llvm::DIBuilder::createLocalVariable(unsigned int, llvm::DIDescriptor, llvm::StringRef, llvm::DIFile, unsigned int, llvm::DIRef<llvm::DIType>, bool, unsigned int, unsigned int)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderInsertDeclareAtEnd':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderInsertDeclareAtEnd+0x20): undefined reference to `llvm::DIBuilder::insertDeclare(llvm::Value*, llvm::DIVariable, llvm::DIExpression, llvm::BasicBlock*)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateEnumerationType':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateEnumerationType+0x98): undefined reference to `llvm::DIBuilder::createEnumerationType(llvm::DIDescriptor, llvm::StringRef, llvm::DIFile, unsigned int, unsigned long, unsigned long, llvm::DITypedArray<llvm::DIDescriptor>, llvm::DIType, llvm::StringRef)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateStructType':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateStructType+0xb0): undefined reference to `llvm::DIBuilder::createStructType(llvm::DIDescriptor, llvm::StringRef, llvm::DIFile, unsigned int, unsigned long, unsigned long, unsigned int, llvm::DIType, llvm::DITypedArray<llvm::DIDescriptor>, unsigned int, llvm::DIType, llvm::StringRef)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreateMemberType':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreateMemberType+0x80): undefined reference to `llvm::DIBuilder::createMemberType(llvm::DIDescriptor, llvm::StringRef, llvm::DIFile, unsigned int, unsigned long, unsigned long, unsigned long, unsigned int, llvm::DIType)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMDIBuilderCreatePointerType':
src/llvm/ext/llvm_ext.cc:(.text.LLVMDIBuilderCreatePointerType+0x4d): undefined reference to `llvm::DIBuilder::createPointerType(llvm::DIType, unsigned long, unsigned long, llvm::StringRef)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMTemporaryMDNode':
src/llvm/ext/llvm_ext.cc:(.text.LLVMTemporaryMDNode+0x8): undefined reference to `llvm::MDNode::getTemporary(llvm::LLVMContext&, llvm::ArrayRef<llvm::Metadata*>)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `LLVMSetCurrentDebugLocation2':
src/llvm/ext/llvm_ext.cc:(.text.LLVMSetCurrentDebugLocation2+0x30): undefined reference to `llvm::DebugLoc::get(unsigned int, unsigned int, llvm::MDNode*, llvm::MDNode*)'
/usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o: In function `llvm::AtomicCmpXchgInst* llvm::IRBuilder<true, llvm::ConstantFolder, llvm::IRBuilderDefaultInserter<true> >::Insert<llvm::AtomicCmpXchgInst>(llvm::AtomicCmpXchgInst*, llvm::Twine const&) const':
src/llvm/ext/llvm_ext.cc:(.text._ZNK4llvm9IRBuilderILb1ENS_14ConstantFolderENS_24IRBuilderDefaultInserterILb1EEEE6InsertINS_17AtomicCmpXchgInstEEEPT_S8_RKNS_5TwineE[_ZNK4llvm9IRBuilderILb1ENS_14ConstantFolderENS_24IRBuilderDefaultInserterILb1EEEE6InsertINS_17AtomicCmpXchgInstEEEPT_S8_RKNS_5TwineE]+0x48): undefined reference to `llvm::SymbolTableListTraits<llvm::Instruction, llvm::BasicBlock>::addNodeToList(llvm::Instruction*)'
cc: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `cc -o "/usr/home/yonas/crystal/git/.build/crystal" "${@}"  -rdynamic  /usr/home/yonas/crystal/git/src/llvm/ext/llvm_ext.o `/usr/local/bin/llvm-config38 --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre -lm -lgc-threaded -lpthread /usr/home/yonas/crystal/git/src/ext/libcrystal.a -levent -lpthread -L/usr/lib -L/usr/local/lib`
gmake: *** [Makefile:84: .build/crystal] Error 1

Not sure if it matters, but it looks like the compiler is using both LLVM 3.8 and 3.9.

[yonas]

@ysbaddaden Which intermediate versions do you recommend we build to avoid the issue of breaking changes?

[tosmi]

i did a build of 0.23.1 on 10.3 and 11.1. i had to compile all intermediate versions that contain breaking changes (see the releases page). so 0.19.4 -> 0.20.0 -> 0.20.1 -> 0.20.3 -> 0.20.5 -> 0.21.0 -> 0.22.0 -> 0.23.1.

i'm currently working on a port of crystal. hopefully i should have a working version by the end of this week.

[tosmi]

on more note: you are using llvm 3.9 i actually had problems compiling version 0.21.0 with the 0.20.5 binary, see bug #4719. i used lvm 3.5 for all intermediate versions.

[Sija]

@tosmi llvm 3.5 is known to be buggy, see #4104.

[RX14]

--no-debug seems to work around all the issues regarding llvm assertions. I'd recommend using that and llvm 4.0.

[sdogruyol]

Can we close this? It's a known issue with LLVM

[RX14]

I'm not sure, we still don't have a freebsd package, but is this the right place for this issue?

[myfreeweb]

So --no-debug does not help with 0.20.5 building itself (or 0.21.0), with llvm 4.0.1

Using llvm-config40 [version=4.0.1]
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build --no-debug -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Using compiled compiler at .build/crystal
Assertion failed: (!FnScope || SP == FnScope->getScopeNode()), function endFunction, file /wrkdirs/usr/ports/devel/llvm40/work/llvm-4.0.1.src/lib/CodeGen/AsmPrinter/DwarfDebug.cpp, line 1213.
gmake: *** [Makefile:113: .build/crystal] Abort trap (core dumped)

Looks like this ONLY affects 0.20.5. So the path that worked for me is 0.20.3 [llvm39] → 0.20.5 [llvm35] → 0.21.0 [llvm40].

[myfreeweb]

Here's some packages!

https://unrelentingtech.s3.dualstack.eu-west-1.amazonaws.com/crystal/crystal-0.23.1-freebsd11.1.txz
https://unrelentingtech.s3.dualstack.eu-west-1.amazonaws.com/crystal/shards-0.7.1-freebsd11.1.txz

https://unrelentingtech.s3.dualstack.eu-west-1.amazonaws.com/crystal/crystal-0.23.1-freebsd12.0.txz
https://unrelentingtech.s3.dualstack.eu-west-1.amazonaws.com/crystal/shards-0.7.1-freebsd12.0.txz

And the port used to build them.

(includes https://github.com/crystal-lang/crystal/pull/5199 for 12 compatibility)

I'd like to send the port to the official ports collection eventually, so please test these :)

[yonas]

@myfreeweb Awesome work!! Works for me :+1:

[papilip]

@myfreeweb Génial ! It’s fine ! A big thank you !

With FreeBSD 11.1, as root: (You can see my gist)

pkg install llvm40
pkg install boehm-gc-threaded
wget https://unrelentingtech.s3.dualstack.eu-west-1.amazonaws.com/crystal/crystal-0.23.1-freebsd11.1.txz
wget https://unrelentingtech.s3.dualstack.eu-west-1.amazonaws.com/crystal/shards-0.7.1-freebsd11.1.txz
cd / && tar Jxvf crystal-0.23.1-freebsd11.1.txz
cd / && tar Jxvf shards-0.7.1-freebsd11.1.txz

[myfreeweb]

The ports have been committed to the official tree! r462317 r462318 Soon, there will be official binary packages so you'll be able to pkg install crystal shards.

I'll send a PR to crystal-book with installation instructions soon.

(btw @papilip these were pkg packages, you could've pkg added them instead of using tar)

This issue can be closed now :)

[RX14]

@myfreeweb A few comments:

• Are you planning to remove the bootstrap compiler after the next release, and depend on the previously-built freebsd package (i'd recommend this if at all possible)
• The pkg-message says "When building Crystal code with the --release flag, add the --no-debug flag to avoid LLVM assertion errors." but you're building crystal with LLVM 5, so that advice is outdated. Unless that bug isn't fixed on freebsd?
• I see a few patches, are they upstreamed?

[myfreeweb]

depend on the previously-built freebsd package

Yes, of course I'm planning to do that.

that advice is outdated

Maybe it's a different bug, but there are failures:

$ crystal run --release /usr/local/share/examples/crystal/tree.cr                   
Assertion failed: (cast<DISubprogram>(Scope)->describes(MF->getFunction())), function getOrCreateRegularScope, file /wrkdirs/usr/ports/devel/llvm50/work/llvm-5.0.1.src/lib/CodeGen/LexicalScopes.cpp, line 176.
zsh: abort (core dumped)  crystal run --release /usr/local/share/examples/crystal/tree.cr

I see a few patches, are they upstreamed?

Yeah, it's this one: https://github.com/crystal-lang/crystal/commit/a3f2ecc30d77b8146f49fdd3564f409495765dde (upstreamed after v0.24.1)

[RX14]

@myfreeweb that's... very strange. That's the exact same bug that we see in LLVM 3.9, but it doesn't appear to happen on LLVM 4.0+. I'm extremely surprised to see it pop up on what appears to be a compiler built on LLVM 5.0.

What's crystal -v on that specific crystal compiler?

[myfreeweb]

Crystal 0.24.0+? [2fa2a3e369e1] (2018-02-19)

LLVM: 5.0.1
Default target: x86_64-portbld-freebsd12.0

(of course it's 5.0, you can see the llvm50/work/llvm-5.0.1.src path in the error, and I can see it in ldd)

BTW, we also see LLVM assertion failures in other projects (RPCS3) that Linux users don't seem to report…

[RX14]

@myfreeweb oh well. Well that's something to investigate because at least on linux the problem is solved.

Perhaps you could bisect on linux to see which commit solved it and see why it might not have been solved on freebsd. Other than that, you're stuck with the --no-debug workaround unfortunately.

And you're right about llvm-5.0.1 being in the path, that's my bad.

[amirulalif]

Shared object "libevent-2.1.so.6" not found, required by "crystal"
i got this error on freebsd11

update: never mine it solved with pkg install libevent-2.1.8_1