Truffle: Differing Results with `truffle test` and `truffle test ./test/...`

[isaacsultan]

• [ ] I've asked for help in the Truffle Gitter before filing this issue.

---

Issue

When calling truffle test, I am getting different results to truffle test ./test/... on a specific file.
The test in question is here, and I am doing the setup in a separate file:

The failure I am receiving on the truffle test run mentions
shouldFailWithMessage (node_modules/openzeppelin-test-helpers/src/shouldFail.js:14:10)
in the before all hook. Which is not being called in this file. I have a feeling that there may be some kind of leak from a previous test that is causing this.

Steps to Reproduce

git clone https://github.com/partial-f/core.git
git checkout develop
npm install
truffle test 
truffle test ./test/ProductTeller.test.js

Expected Behavior

Expected the same results for the contract, regardless if called in isolation or with all of the tests.

Actual Results

truffle test

  6) Contract: ProductTeller
       "before all" hook: prepare suite:
     Uncaught AssertionError: Expected failure not received
      at shouldFailWithMessage (node_modules/openzeppelin-test-helpers/src/shouldFail.js:14:10)
      at process._tickCallback (internal/process/next_tick.js:68:7)

  7) Contract: ProductTeller
       "before each" hook: before test for "should allow an admin to set a product fee":
     TypeError: Cannot read property 'call' of undefined


  8) Contract: ProductTeller
       "after each" hook: after test for "should allow an admin to set a product fee":
     TypeError: Cannot read property 'call' of undefined

truffle test ./test/ProductTeller.test.js

Contract: ProductTeller
...
  12 passing (25s)
  1 pending
  1 failing
...

Environment

• Operating System: 10.14
• Truffle version (truffle version): v5.0.5
• node version (node --version): v10.14.2
• npm version (npm --version): 6.4.1
• ganache-cli: Ganache CLI v6.3.0 (ganache-core: 2.4.0)

[wTendera]

I recently run into similar issues with truffle tests

When truffle is running tests it's supposed to have clean room environment for each test, so there should not be a difference if I run a test on it's own or with other tests.

Unluckily that's not the case with my tests. When I run a test file on it's own every test passes nicely, but when I go for 'truffle test' on whole project suddenly the same test starts failing.

Moreover it seems really non-deterministic as with each tests execution I can get different errors, or the test fails at different point.

At first I thought it was because I upgraded to solidity 0.5.4 (and truffle/ganache as well) but even on earlier distributions it works the same.

I tried various setups with different versions of truffle/ganache/node but everytime at some point (usually random) my tests fail.

Currently I'm running everything on

Ganache CLI v6.2.5 (ganache-core: 2.3.3)
Truffle v5.0.3 (core: 5.0.3)
Solidity - 0.5.4 (solc-js)
Node v8.9.4
Any idea why is it happening?

[wTendera]

@isaacsultan Updating ganache-cli to 6.4.1 made it work for me.

Other issues
https://github.com/trufflesuite/ganache-cli/issues/624
https://github.com/trufflesuite/ganache-core/pull/329
https://github.com/trufflesuite/ganache-core/issues/358

[isaacsultan]

@wTendera Thanks, updated and fixed for me too!

[kseniya292]

Hello. I'm still running into this issue. When I run truffle test test/pathToContract on each of my files, all tests pass. When I run truffle test almost half of the tests fail. I've tried: ganache-cli v6.2.4 and v 6.4.1.

Truffle v5.0.8
Solidity - 0.5.0
Node v11.0.0

Also, a lot of the errors are reversed. For example:

1) Contract: DAO
       Dao
         constructor
           it should set master registry:

      AssertionError: expected '0xfBe5347D784aeEC17Fe45d1627fdc320A08C5Da3' to not equal '0xfBe5347D784aeEC17Fe45d1627fdc320A08C5Da3'
      + expected - actual

That is for this test:

      it('it should set master registry', async function() {
        let address = await dao.masterRegistry.call();
        assert.equal(address, masterRegistry.address);
      });

It's testing that the two addresses are equal, which they are according to the error. But the error is expecting them to not equal ??

@CruzMolina

[dougiebuckets]

Hi there - @kseniya292 (above) and I are working on the same project. For kicks, I ran all of the tests in the reverse order that they run when we use truffle test. I.e.,

truffle test file1.test.js file2.test.js file3.test.js file4.test.js file5.test.js file6.test.js

Now, approximately 20 more tests fail vs when we run truffle test. Again, when we run each test file independently, all tests pass (e.g., truffle test test/file1.test.js).

[CruzMolina]

Hey @kseniya292 & @dougiebuckets !

I'm wondering if this is another ganache-core specific bug. Can you share a repo link so I can reproduce and isolate the problem?

[kseniya292]

@CruzMolina Thanks so much for taking a look! It's a private repo so I added you on as a collaborator. Branch is upgrade/packages.
Command to run ganache:
ganache-cli --gasLimit=8000029 --gasPrice=30000000000 --defaultBalanceEther=100000000 -a 30

truffle test - 113 tests break

truffle test test/MultiSigWallet.test.js test/LottoDAO.test.js test/LottoProxyFactory.test.js test/Governance.test.js test/MasterRegistry.test.js test/LottoToken.test.js - all the tests pass. This is a random order I tried out that works for some reason.

[CruzMolina]

Awesome, thanks for the instructions @kseniya292 . Checking this out now 🧐

[CruzMolina]

Ah, okay. Looks like you are in the process of updating from truffle 4 to truffle 5.

I've invited you to a private repo w/ a branch I worked on real quick to help you along (would have just pushed my local branch but I don't think I have proper permissions from your repo). It turns out (at least in v5), we already expose chai.assert internally for running js tests, so declaring const assert = chai.assert was causing what appears to be an overloading error. Can simply start using assert() w/o needing to require the lib.

The other issue I noticed is you're using bn-chai, which (for whatever reason) wasn't written to support should(). There appears to also be an overloading error happening there as well between chai and bn-chai.

For replacing the test comparisons, I pushed an example suggestion to get you guys started.

Currently only 34 tests fail on my end, yay! 🎉

[kseniya292]

@CruzMolina Omg, thank you so much! We really appreciate you taking the time to look at the repo.
I'll make those updates.
I wonder why the overloading errors are happening inconsistently? Why is it that when you run in a specific order, they all pass, and then if you switch the order.. some of the tests fail?

[CruzMolina]

Np! The errors really threw me for a minute, so I'm glad it wasn't actually something with ganache-core/cli or truffle proper.

My best guess about the inconsistency is that maybe requiring the same libraries within truffle's test running context (mocha) causes the overloading. So maybe calling truffle test by itself loads all of the js tests within the same context w/ those requires causing funky things to happen vs creating a more isolated/separate context for each individual file when doing truffle test <file> or truffle test <file1> <file2>. I'm not quite sure and would need to look into it some more.

[CruzMolina]

Hmm... did a little more testing. I'm really not sure. The only consistency seems to be related to importing chai.assert again when it's already globally available and also using bn-chai 🤷‍♂️

[kseniya292]

@CruzMolina All the tests are finally passing! 🎉
I removed the chai.assert declaration, and removed bn-chai and refactored my tests to use should.equal rather than should.eq.BN. Thanks again for your help!

[CruzMolina]

Great! Glad to help!

[theabdulmateen]

I had a similar issue with truffle test and I was using the should.eventually.equal() method and it seems to also be causing such inconsistency with the tests.

Hope this helps.