Solidity: Segmentation fault getting worse...nothing compiling now.

[VoR0220]

Relates to #571 #490 #570 #502 #486 ....I currently cannot compile anything with OS X....this leads me to believe something is wrong with the OS X slave because it should be picking up these problems in the jenkins runs. Either way...we need a fix and stat.

[bobsummerwill]

It is entirely plausible that OS X tests aren't being run consistently within Jenkins.

I've been working on CircleCI, TravisCI and Appveyor alternatives which we can have running in parallel as soon as we do the repository reorgs. I will try to get test runs working on those ASAP.

[VoR0220]

Thank you @bobsummerwill ... I'm going to begin debugging this and talk with @chriseth about what exactly need to happen to get this thing working without breaking Ubuntu and Windows.

[VoR0220]

WTF!?! Just changed the cmake to debug mode...everything...and i mean everything...works fine again! GAHHHHHHHHHHH!!!!! _chucks computer out window_

[VoR0220]

okay....so from my findings

cmake set to debug = works
set to relwithdebuginfo = fails all compilation
set to release = fails all compilation

[VoR0220]

even worse...when I try to turn the sanitizer on...they all pass....creepy guys. Just plain creepy.

[tymat]

HeisenBug

[kjsujesh]

Probably llvm on Mac is doing something crazy with the stack?

[VoR0220]

@kjsujesh i would not be surprised.

[bobsummerwill]

Any valgrind experts out there? @gcolvin? @chfast? @yann300? @LianaHus? @winsvega?

Sounds like we need to hit this one with whatever analysis tools we have available to us.

[VoR0220]

@bobsummerwill what was that one debugging crew that you recommended again? The experts that said they'd be able to diagnose wtf is going on?

[bobsummerwill]

Hmm. Individuals, you mean? I don't remember saying anything so specific.

"AAA game developers in general?" :-)

[VoR0220]

Nah...there was a site that you recommended.

[tymat]

Confirmed that building with cmake -DCMAKE_BUILD_TYPE=Debug .. makes the issue go away. HeisenBug confirmed.

Has anyone tried using LLVM 3.7 to confirm that this isn't an LLVM 3.8 related issue?

[VoR0220]

I have tried LLVM 3.7 and this bug was still persisting in its earlier form.

[bobsummerwill]

Oh right, @VoR0220.

Coverity Scan, probably.

[VoR0220]

thats it!

[chfast]

• Any help with clang sanitizer? You can have release-like build with address sanitizer enabled.
• Are you sure we have tests that shows the problem?

[chriseth]

I tried to reproduce this problem on our el capitan build machine but could not. I tried the following combinations:

( Release / RelWithDebugInfo ) x ( build with XCode / build with make )

In all configurations, I used cmake with -DTESTS=0 -DGUI=0 -DETHASHCL=0 -DEVMJIT=0.

Furthermore, the --version string shows Interpreter at the end (not JIT). But I think this does not have any influence as we have seen this being reported with Interpreter builds, too.

Could anyone please check whether this is a ElCapitan/Yosemite issue?

[VoR0220]

I have not been compiling with -DEVMJIT=0..nor DETHASHCL=0....let me try those and see what comes out.

[VoR0220]

Can confirm... -DEVMJIT=0 -DETHASHCL=0 when run with all versions will enable it to build and run properly. This is going to be like finding a needle in a haystack...

[chfast]

@VoR0220 I have no idea what you have just written.

[VoR0220]

where did I lose you?

[chfast]

With -DEVMJIT=0 -DETHASHCL=0 is the bug reproducible?

[VoR0220]

no, it is not. Not on mine atleast.

[chfast]

for the buggy and the non-buggy binary, can you list all dependent shared libraries?

[VoR0220]

.....yeaaa....I was looking for that yesterday....I'm not sure how to do that easily.....

[VoR0220]

any advice on that would be welcome (I'm a cmake amateur)

[chfast]

On linux that would be ldd solc.

[boestin]

@VoR0220 any advice in which dir i should execute the cmake command?
thx

[kjsujesh]

You can use the following command in the root of the /webthree-umbrella directory where you checked out the code:

./webthree-helpers/scripts/ethbuild.sh --no-git --project solidity --all --cores 4 -DEVMJIT=0 -DTESTS=0 -DGUI=0 -DETHASHCL=0

[boestin]

@kjsujesh @VoR0220
mm.. I'm still getting the error solc: signal: segmentation fault

What did I do:
I went ethereum.org/cli because I am a Noob and I want to develop using OSX as OS.

1. brew tap ethereum/ethereum
2. brew install ethereum

No I want the solc compiler. According to ethereum.org/cli:

1. brew tap ethereum/ethereum
2. brew install cpp-ethereum
3. brew linkapps cpp-ethereum

Then I cloned, because I don't have a clue where webthree is (told you I was a Noob):

1. git clone --recursive https://github.com/ethereum/webthree-umbrella

then I went into the "just cloned" webthree-umbrella directory and executed the command:

1. ./webthree-helpers/scripts/ethbuild.sh --no-git --project solidity --all --cores 4 -DEVMJIT=0 -DTESTS=0 -DGUI=0 -DETHASHCL=0

Then I excuted the geth command:

1. geth --rpc --rpccorsdomain "*" --testnet console

In the Geth console:

1. admin.setSolc("/usr/local/bin/solc")

Still the error.

[kjsujesh]

Please have a look at this : http://solidity.readthedocs.io/en/latest/installing-solidity.html

After you build, use the solidity tests to check for errors. They are here:
/webthree-umbrella/solidity/build/test

[boestin]

@kjsujesh
ok, should I remove something first? I think I already did this.

[kjsujesh]

in your admin.setSolc("/usr/local/bin/solc") step, can you try changing to the solc that you just built?
Like
admin.setSolc("/webthree-umbrella/solidity/build/solc/solc")
and then try ?

[boestin]

@kjsujesh
I changed the solc directory to the one I just built.. same error.
Thanks for the help though.

[VoR0220]

don't worry @boestin we are working on the solution as you type.

[boestin]

@VoR0220 I know. Thanks!

[boestin]

I thought that a possibility was that Geth was causing the issue.. So I followed the instructions (https://github.com/ethereum/go-ethereum/wiki/Installation-Instructions-for-Mac) to clone and build Geth, but unfortunately the error still is here.

What I do find odd is this:
When selecting the solc within Geth using Console, OR as parameter does not seem to fill the [].
So when saying eth.getCompilers() returns [] That's strange isnt it? The same command in Windows returns "Solidity"

[boestin]

@VoR0220 @kjsujesh

I decided to remove the solc file in /usr/local/bin/solc and cp'd the build file in webthree, but that didnt help.
Something funny happened: I tried to run ./webthree-helpers/scripts/ethbuild.sh --no-git --project solidity --all --cores 4 -DEVMJIT=0 -DTESTS=0 -DGUI=0 -DETHASHCL=0.
On the end something didnt go right. Anyone?

---

-- Configuring solidity

-- CMake Version 3.5.2
-- CMAKE_BUILD_TYPE Build type RelWithDebInfo
-- TARGET_PLATFORM Target platform Darwin
--------------------------------------------------------------- features
-- PROFILING Profiling support OFF
------------------------------------------------------------- components

-- TESTS Build tests ON

-- Found jsoncpp: /usr/local/lib/libjsoncpp.dylib
-- JsonCpp headers: /usr/local/include
-- JsonCpp lib : /usr/local/lib/libjsoncpp.dylib
-- Found leveldb: /usr/local/lib/libleveldb.dylib
-- LevelDB headers: /usr/local/include
-- LevelDB lib : /usr/local/lib/libleveldb.dylib
-- Found CryptoPP: /usr/local/include (found suitable version "5.6.3", minimum required is "5.6.2")
-- CryptoPP headers: /usr/local/include
-- CryptoPP lib : /usr/local/lib/libcryptopp.a
-- Found miniupnpc: /usr/local/lib/libminiupnpc.dylib
-- miniupnpc headers: /usr/local/include
-- miniupnpc lib : /usr/local/lib/libminiupnpc.dylib
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
Eth_EVMJIT_LIBRARY
linked by target "soltest" in directory /Users/user_name/webthree-umbrella/solidity/test

[boestin]

Decided to stay up tonight and reinstall OSX. Yes. I know :)

After the install, the works:

home-brew, Xcode, all required external dependencies which are mentioned here: http://ethdocs.org/en/latest/ethereum-clients/cpp-ethereum/building-from-source/osx.html
Still waiting for it to finish.

[boestin]

Screwed up the system again, somehow I can't install cpp-ethereum using brew.

[bobsummerwill]

Maybe hitting this, @boestin?
https://github.com/ethereum/webthree-umbrella/issues/518
If so, should be fixed now, by @chfast.

[bobsummerwill]

Need to do brew install cpp-ethereum --build-from-source right now if you are on El Capitan because of a broken bottle on just that version. Because of incomplete Jenkins work in https://github.com/ethereum/webthree-umbrella/issues/454.

Sorry!

[boestin]

@bobsummerwill No worries. I thought it was me again ^^
It's running now. Then after I'll test if the segmentation error still exists.

[boestin]

Error: solc: signal: segmentation fault

[VoR0220]

@chfast so for buggy solc with command:

cmake -DCMAKE_BUILD_TYPE=Release -DEVMJIT=1 -DETHASHCL=1 .. :

vor0220@CaptainNumNutz:solidity/build ‹fixSegFault*›$ otool -L solc/solc
solc/solc:
    /usr/local/opt/jsoncpp/lib/libjsoncpp.1.dylib (compatibility version 1.0.0, current version 1.6.5)
    /usr/local/opt/leveldb/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/vor0220/webthree-umbrella/libweb3core/build/libdevcore/libdevcore.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/vor0220/webthree-umbrella/solidity/build/libsolidity/libsolidity.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/vor0220/webthree-umbrella/solidity/build/libevmasm/libevmasm.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

for

cmake -DCMAKE_BUILD_TYPE=Release -DEVMJIT=0 -DETHASHCL=0 .. :

for non buggy solc:

cmake -DCMAKE_BUILD_TYPE=Debug -DEVMJIT=0 -DETHASHCL=0 .. :

vor0220@CaptainNumNutz:build/solc ‹fixSegFault*›$ otool -L solc
solc:
    /usr/local/opt/jsoncpp/lib/libjsoncpp.1.dylib (compatibility version 1.0.0, current version 1.6.5)
    /usr/local/opt/leveldb/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/vor0220/webthree-umbrella/libweb3core/build/libdevcore/libdevcore.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/vor0220/webthree-umbrella/solidity/build/libsolidity/libsolidity.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/vor0220/webthree-umbrella/solidity/build/libevmasm/libevmasm.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

[VoR0220]

@boestin in the meantime try this

./webthree-helpers/scripts/ethbuild.sh --no-git --project solidity --all --cores 4 -DCMAKE_BUILD_TYPE=Debug

[boestin]

@VoR0220 Ok, i'll try. So to make it clear for me. The solc that is sitting in /usr/local/bin/solc remains untouched. Right? When I start geth, it will automatically points to that direction for some reason.

When ethbuild is finished, do I need to make another action to "copy" the build solc to /usr/local/bin/solc

Thanks.

[VoR0220]

nope...shouldn't need to.

[chfast]

@VoR0220 Have you tried EVMJIT=1 ETHASHCL=0 and EVMJIT=0 ETHASHCL=1?

[VoR0220]

I have not... @chfast should I start looking for dependencies elsewhere in the dylibs that solc calls from? I think we might find more information there.

[boestin]

@VoR0220 YES it's working!
-DCMAKE_BUILD_TYPE=Debug is a workaround for me!

[boestin]

@VoR0220 thanks for the help! 👍

[VoR0220]

it would appear that a fix to the way it was before is just about here. #584 when this PR is merged, you should be able to at the very least compile and grab the ABI and binaries from your contracts. There is still a remaining problem of seg faults with the gas estimator...I suppose that this is coming from a different area. But the main parts that you will care about for web3 deployment should be working. Let us know if the problem persists so that we can wrap up this issue.

[VoR0220]

if you really need the gas parameter or ast, set your cmake to the debug build.

[bobsummerwill]

Closed as dupe of "The Heisenbug" - https://github.com/ethereum/webthree-umbrella/issues/565. We can reopen if we later find there is anything unique here.