Truffle: Warn: Keccak bindings fail on first install

[wbt]

Issue

Installing the new Truffle 5 per instructions produces a one-time error about keccak bindings compilation failing.

Steps to Reproduce

First, I had a previous version of Truffle installed, and after reading the blog post etc., I decided to try the newer version. I ran npm update -g truffle which got me the response
+ [email protected]
updated 5 packages in 5.098s
I then ran npm install truffle -g as instructed.

Expected Results

+ [email protected]
updated 1 packages in 1.234s
or something like that. Numbers other than the 5 might have been higher/different.

Actual Results

C:\Users\me\AppData\Local\node\truffle -> C:\Users\me\AppData\Local\node\node_modules\truffle\build\cli.bundled.js

> [email protected] install C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak
> npm run rebuild || echo "Keccak bindings compilation fail. Pure JS implementation will be used."


> [email protected] rebuild C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak
> node-gyp rebuild


C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak>if not defined npm_config_node_gyp (node "C:\Users\me\AppData\Local\node\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Users\me\AppData\Local\node\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  addon.cc
  win_delay_load_hook.cc
..\src\addon.cc(37): warning C4244: 'initializing': conversion from 'int64_t' to 'unsigned int', possible loss of data [C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak\build\keccak.vcxproj]
..\src\addon.cc(38): warning C4244: 'initializing': conversion from 'int64_t' to 'unsigned int', possible loss of data [C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak\build\keccak.vcxproj]
..\src\addon.cc(56): warning C4244: 'initializing': conversion from 'int64_t' to 'unsigned char', possible loss of data [C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak\build\keccak.vcxproj]
  KeccakSponge.c
  KeccakP-1600-reference.c
     Creating library C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak\build\Release\keccak.lib and object C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak\build\Release\keccak.exp
  Generating code
  All 156 functions were compiled because no usable IPDB/IOBJ from previous compilation was found.
  Finished generating code
  keccak.vcxproj -> C:\Users\me\AppData\Local\node\node_modules\truffle\node_modules\keccak\build\Release\\keccak.node
+ [email protected]
added 31 packages from 17 contributors, removed 23 packages and updated 15 packages in 16.425s

Running the same command again gave the Expected Results.
Therefore, this issue may be difficult to reproduce, but maybe reading the error message triggers a recollection for somebody of a missed step (e.g. including a dependency on a particular version or later of keccak?)
This issue would seem most likely to be encountered by new users.

Environment

• Operating System: Windows 10 Pro (Username changed in output above.)
• node version (node --version): 9.3.0
• npm version (npm --version): 6.5.0

[aliaksandr-adzinets]

I tried to install Truffle 5.0.0 and got the same issue.
Rolling back to Truffle 4.1.14 works as expected.
My environments is: Windows 10, MSVS 2017, Python 2.7, node-gyp 3.8.0, npm 6.4.1, Node.js 10.14.1.

[eggplantzzz]

@wbt Looks like npm will not install a different major version when using npm update. If you want to do it cleanly you need to globally uninstall Truffle first. In other words, do npm uninstall -g truffle and then npm install -g truffle.

[stale[bot]]

Thank you for raising this issue! It has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you would like to keep this issue open, please respond with information about the current state of this problem.

[wbt]

@stalebot is getting a little too aggressive here. The issue still exists and I think could still be more of a turn-off for new users.

A duplicate was reported just 11 days ago; should we really as general practice be opening duplicates and letting earlier reports autoclose as "stale" when an issue persists? (I would recommend against adopting that.)

[stale[bot]]

Thanks for your response! This issue is no longer considered stale and someone from the Truffle team will try to respond as soon as they can.

[gnidan]

@wbt We recently made stalebot way less aggressive (180 days I believe instead of 60 or 90), but it seems like it might require further tweaking. I agree with your thoughts about duplicates. Happy to continue to evaluate stalebot practices and adjust accordingly.

[wbt]

@gnidan Can you have a comment on or reference to an item reset the timer? This was definitely not a 180 day timer.

[alanbleier182]

Yeah keccak bindings compilation are failing on my end too...

gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/home/bleier/.nvm/versions/node/v12.2.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:196:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:257:12)
gyp ERR! System Linux 4.18.0-20-generic
gyp ERR! command "/home/bleier/.nvm/versions/node/v12.2.0/bin/node" "/home/bleier/.nvm/versions/node/v12.2.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/bleier/.nvm/versions/node/v12.2.0/lib/node_modules/truffle/node_modules/keccak
gyp ERR! node -v v12.2.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] rebuild: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] rebuild script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/bleier/.npm/_logs/2019-05-21T08_08_29_306Z-debug.log
Keccak bindings compilation fail. Pure JS implementation will be used.
+ [email protected]

[taylorjdawson]

I can reproduce as well:

System Version: macOS 10.14.4 (18E226)
Kernel Version: Darwin 18.5.0

Node Version: v12.3.1

It works on node v11.10.1.

[xhyumiracle]

[email protected] installation succeed on [email protected], failed on v8.15.0 && 12.4.0

[tuler]

We need to get rid of all transitive dependencies to [email protected] and only have [email protected] in the dependency tree, because it solved the issue as can be seen here and here.

The current dependency tree as defined by yarn is this:

(develop) % yarn list --pattern keccak                                                                                                                                                                                  yarn list v1.19.2
├─ @truffle/[email protected]
│  └─ [email protected]
├─ [email protected]
│  └─ [email protected]
├─ [email protected]
│  └─ [email protected]
├─ [email protected]
│  └─ [email protected]
│     └─ [email protected]
├─ [email protected]
│  ├─ [email protected]
│  │  └─ [email protected]
│  └─ [email protected]
├─ [email protected]
│  └─ [email protected]
├─ [email protected]
│  ├─ [email protected]
│  │  └─ [email protected]
│  ├─ [email protected]
│  │  └─ [email protected]
│  ├─ [email protected]
│  │  ├─ [email protected]
│  │  │  └─ [email protected]
│  │  ├─ [email protected]
│  │  │  └─ [email protected]
│  │  └─ [email protected]
│  └─ [email protected]
│     ├─ [email protected]
│     │  └─ [email protected]
│     └─ [email protected]
│        ├─ [email protected]
│        │  └─ [email protected]
│        └─ [email protected]
├─ [email protected]
└─ [email protected]
✨  Done in 1.09s.

We can see that [email protected] is being brought by [email protected] and [email protected].

Latest ethereumjs-account is 3.0.0.
Latest ethereumjs-block is still 2.2.2.

What is strange is that I don't see keccak as a direct dependency of ethereumjs-block.

[tuler]

This is the output of yarn why keccak:

(develop)⚡ % yarn why keccak                                                                                                                                                                                           ~/Documents/workspace/cartesi/truffle
yarn why v1.19.2
[1/4] 🤔  Why do we have the module "keccak"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "[email protected]"
info Has been hoisted to "keccak"
info Reasons this module exists
   - "workspace-aggregator-3517db01-6c3f-4a79-b1d3-b5ac75f90dbd" depends on it
   - Hoisted from "_project_#ethereumjs-util#keccak"
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-util#keccak"
   - Hoisted from "_project_#@truffle#deployer#@ensdomains#ens#ganache-cli#ethereumjs-util#keccak"
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-block#ethereumjs-util#keccak"
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#eth-sig-util#ethereumjs-util#keccak"
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#web3-provider-engine#ethereumjs-util#keccak"
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-blockchain#ethereumjs-util#keccak"
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-blockchain#ethereumjs-block#ethereumjs-util#keccak"
info Disk size without dependencies: "240KB"
info Disk size with unique dependencies: "836KB"
info Disk size with transitive dependencies: "880KB"
info Number of shared dependencies: 4
=> Found "ethereumjs-vm#[email protected]"
info Reasons this module exists
   - "_project_#ethereumjs-vm#ethereumjs-util" depends on it
   - Hoisted from "_project_#ethereumjs-vm#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "@truffle/hdwallet-provider#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#hdwallet-provider#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#hdwallet-provider#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ethereumjs-vm#ethereumjs-block#[email protected]"
info Reasons this module exists
   - "_project_#ethereumjs-vm#ethereumjs-block#ethereumjs-util" depends on it
   - Hoisted from "_project_#ethereumjs-vm#ethereumjs-block#ethereumjs-util#keccak"
info Disk size without dependencies: "240KB"
info Disk size with unique dependencies: "836KB"
info Disk size with transitive dependencies: "880KB"
info Number of shared dependencies: 4
=> Found "ethereumjs-abi#[email protected]"
info Reasons this module exists
   - "_project_#eth-sig-util#ethereumjs-abi#ethereumjs-util" depends on it
   - Hoisted from "_project_#eth-sig-util#ethereumjs-abi#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ethereumjs-wallet#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#core#ethereumjs-wallet#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#core#ethereumjs-wallet#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ganache-core#ethereumjs-abi#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#ethereumjs-abi#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-abi#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ganache-core#ethereumjs-account#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#ethereumjs-account#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-account#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ganache-core#ethereumjs-vm#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ganache-core#ethereumjs-vm#ethereumjs-account#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-account#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-account#ethereumjs-util#keccak"
info Disk size without dependencies: "240KB"
info Disk size with unique dependencies: "836KB"
info Disk size with transitive dependencies: "880KB"
info Number of shared dependencies: 4
=> Found "ganache-core#ethereumjs-vm#ethereumjs-block#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-block#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-block#ethereumjs-util#keccak"
info Disk size without dependencies: "240KB"
info Disk size with unique dependencies: "836KB"
info Disk size with transitive dependencies: "880KB"
info Number of shared dependencies: 4
=> Found "ganache-core#web3-provider-engine#ethereumjs-vm#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#web3-provider-engine#ethereumjs-vm#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#web3-provider-engine#ethereumjs-vm#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ganache-core#web3-provider-engine#ethereumjs-vm#ethereumjs-block#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#web3-provider-engine#ethereumjs-vm#ethereumjs-block#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#web3-provider-engine#ethereumjs-vm#ethereumjs-block#ethereumjs-util#keccak"
info Disk size without dependencies: "240KB"
info Disk size with unique dependencies: "836KB"
info Disk size with transitive dependencies: "880KB"
info Number of shared dependencies: 4
=> Found "ganache-core#web3-provider-engine#ethereumjs-abi#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#web3-provider-engine#eth-sig-util#ethereumjs-abi#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#web3-provider-engine#eth-sig-util#ethereumjs-abi#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ethereumjs-blockchain#ethereumjs-tx#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-blockchain#ethereumjs-block#ethereumjs-tx#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#artifactor#ganache-core#ethereumjs-vm#ethereumjs-blockchain#ethereumjs-block#ethereumjs-tx#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
=> Found "ethereum-ens#[email protected]"
info Reasons this module exists
   - "_project_#@truffle#contract#ethereum-ens#web3#web3-eth#web3-eth-accounts#ethereumjs-tx#ethereumjs-util" depends on it
   - Hoisted from "_project_#@truffle#contract#ethereum-ens#web3#web3-eth#web3-eth-accounts#ethereumjs-tx#ethereumjs-util#keccak"
info Disk size without dependencies: "1.14MB"
info Disk size with unique dependencies: "1.73MB"
info Disk size with transitive dependencies: "1.77MB"
info Number of shared dependencies: 4
✨  Done in 1.07s.

[tuler]

So one of the reasons is coming from [email protected].

(master)⚡ % npm ls keccak
[email protected]
├─┬ [email protected]
│ └── [email protected]
└─┬ [email protected]
  └─┬ [email protected]
    └── [email protected]

The master branch of merkle-patricia-tree already has update of ethereum-util, but it's still unreleased. I raised this issue in this discussion.

[sambacha]

I think this may also be due to the way keccak is implemented with supporting streams. I know that this is not limited to trufflesuite, as I have seen this issue for a while come up sporadically. It can also arise from the use of sudo npm, you can read more here..

[fw-aaron]

Reproduced just now with the following Dockerfile:

from node:lts-alpire3.9

run apk add make g++ gcc python bash
run mkdir /proj && cd /proj &&  npm init --y && npm install ganache-cli
workdir /proj
run npm ls;

Very off-putting. I understand how hard it can be to get a particular version removed completely from the dependency tree!

In my case, after running the above and examining npm ls, it looks like [email protected] (which appears to be the latest: https://www.npmjs.com/package/ganache-cli) relies on [email protected] (which appears to be 2 years old!? https://www.npmjs.com/package/ethereumjs-util) which relies on [email protected].