Truffle: Testing contracts raising exceptions in Truffle

[Qqwy]

• [x] I've asked for help in the Truffle Gitter before filing this issue.

---

Issue

It is very common for a contract call to contain assert(...);, require(...); or (on older contracts) if (...) throw;, which make sure that the contract's state can only be altered if the requirements are met.

It is very important to be able to test if improper input thus makes a contract call fail.
However, I have been unable to find any way to assert that a failure has happened.

Instead, the truffle testing process crashes during testing:

     Uncaught Error: VM Exception while processing transaction: invalid opcode
      at Object.InvalidResponse (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:37022:16)
      at /home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:209743:36
      at XMLHttpRequest.request.onreadystatechange (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:208522:13)
      at XMLHttpRequestEventTarget.dispatchEvent (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210395:18)
      at XMLHttpRequest._setReadyState (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210685:12)
      at XMLHttpRequest._onHttpResponseEnd (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210840:12)
      at IncomingMessage.<anonymous> (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210800:24)
      at endReadableNT (_stream_readable.js:975:12)

Steps to Reproduce

Say you have this contract:

pragma solidity ^0.4.11;
contract NumberContract {
    uint public number;
    address public owner;

    function NumberContract(){
        owner = msg.sender;
    }

    function set(uint new_number) {
        assert (owner == msg.sender);
        number = new_number;
    }
}

Now I'd want to test it like this:

var NumberContract = artifacts.require("./NumberContract.sol");

contract("NumberContract, setting", function(accounts){
    it("Allows setting the number to a different value by owner", function(){
        return NumberContract.deployed().then(function(instance){
            instance.set(42);
            return instance;
        }).then(function(instance){
            return instance.number();
        }).then(function(num){
            assert.equal(num.toString(), '42', "Number not properly set");
        });
    });

    it("Rejects setting the number to a different value by non-owner", function(){
        return NumberContract.deployed().then(function(instance){
            instance.set(55, {from: accounts[1]});
            return instance;
            // Obviously, here we'd need to use some different assertion technique
            // to handle the failure. But what?
        }).then(function(instance){
            return instance.number();
        }).then(function(num){
            assert.equal(num.toString(), '42', "Number could be set by non-owner");
        });
    });
});

Expected Behavior

I'd expect the tests to run, and give a meaningful error, and I'd expect there to be a way to assert that a contract call failed.

Actual Results

Instead, we get an uncaught VM exception:

$ truffle test
Using network 'development'.

Compiling ./contracts/ConvertLib.sol...
Compiling ./contracts/MetaCoin.sol...
Compiling ./test/TestMetacoin.sol...
Compiling truffle/Assert.sol...
Compiling truffle/DeployedAddresses.sol...


  TestMetacoin
    ✓ testInitialBalanceUsingDeployedContract (66ms)
    ✓ testInitialBalanceWithNewMetaCoin (45ms)

  Contract: NumberContract, setting
    ✓ Allows setting the number to a different value by owner
    1) Rejects setting the number to a different value by non-owner
    ✓ Rejects setting the number to a different value by non-owner (40ms)

  Contract: MetaCoin
    > No events were emitted


  4 passing (475ms)
  1 failing

  1) Contract: NumberContract, setting Rejects setting the number to a different value by non-owner:
     Uncaught Error: VM Exception while processing transaction: invalid opcode
      at Object.InvalidResponse (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:37022:16)
      at /home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:209743:36
      at XMLHttpRequest.request.onreadystatechange (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:208522:13)
      at XMLHttpRequestEventTarget.dispatchEvent (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210395:18)
      at XMLHttpRequest._setReadyState (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210685:12)
      at XMLHttpRequest._onHttpResponseEnd (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210840:12)
      at IncomingMessage.<anonymous> (/home/qqwy/.asdf/installs/nodejs/7.8.0/.npm/lib/node_modules/truffle/build/cli.bundled.js:210800:24)
      at endReadableNT (_stream_readable.js:975:12)

Environment

• Operating System: Linux Mint Serena.

• Truffle version:

  • Truffle v3.4.3 (core: 3.4.1)
  • Solidity v0.4.11 (solc-js)

• Ethereum client: testrpc

• node version: 7.8.0
• npm version: 4.2.0 (yarn version 0.24.5)

[batjko]

I'm getting the same error for one of the tests in the pet-shop tutorial:

  function testUserCanAdoptPet() {
    uint returnedId = adoption.adopt(8);
    uint expected = 8;

    Assert.equal(returnedId, expected, "Adoption of petId should be recorded");
  }

The Adoption contract being tested is exactly as instructed by the tutorial:

pragma solidity ^0.4.4;

contract Adoption {
  address[16] public adopters;

  function adopt(uint petId) public returns (uint) {
    require(petId < 0 || petId > 15);

    adopters[petId] = msg.sender;

    return petId;
  }

  // [...]
}

[raid5]

I'm getting the same error as @batjko. I've been able to get around it by comment out the require(petId < 0 || petId > 15); line, but obviously this isn't ideal.

[batjko]

Actually, that brought me to a solution, @raid5 .

That require statement never made any sense to me. It seems to require that the petId must be less than 0 or more than 15, which is the exact opposite of what the objective was supposed to be.

So I've changed it so that petId is between 0 and 15, and tests now pass:

If this was a logical error, a test should not throw a weird, unintuitive exception like that.

[moritz-botconnect]

@batjko, Thank you! That solved the problem :-) It's obvious if you think to look there :-D

[tcoulter]

Hi there.

I believe we've fixed this issue with the Pet Shop tutorial. I'm going to close for housekeeping reasons, but please open a new ticket if you run into more issues.

Thanks!

[awgneo]

Hello all. I don't think anyone here solved the original problem. Is there a way to check for require and assert exceptions that we expect from invalid input?

[awgneo]

I may have found a decent workaround until Truffle officially supports this. The open-zeppelin project provides an assertJump function for this purpose. Check out this example. https://github.com/OpenZeppelin/zeppelin-solidity/blob/master/test/Ownable.js

[benvoi]

I have the same issue as @batjko with the following require statement

~~
function adopt(uint petId) public returns (uint) {
require(petId >= 0 && petId <= 15);
~~

~~~~
~/C/2/0/pet-shop-tutorial ❯❯❯ truffle test
Using network 'development'.

Compiling ./contracts/Adoption.sol...
Compiling ./test/TestAdoption.sol...
Compiling truffle/Assert.sol...
Compiling truffle/DeployedAddresses.sol...

...

TestAdoption
1) "before all" hook: prepare suite

0 passing (436ms)
1 failing

1) TestAdoption "before all" hook: prepare suite:
Error: VM Exception while processing transaction: invalid opcode
at Object.InvalidResponse (/usr/local/lib/node_modules/truffle/build/cli.bundled.js:37312:16)
at /usr/local/lib/node_modules/truffle/build/cli.bundled.js:220420:36
at /usr/local/lib/node_modules/truffle/build/cli.bundled.js:204149:9
at XMLHttpRequest.request.onreadystatechange (/usr/local/lib/node_modules/truffle/build/cli.bundled.js:205574:13)
at XMLHttpRequestEventTarget.dispatchEvent (/usr/local/lib/node_modules/truffle/build/cli.bundled.js:73069:18)
at XMLHttpRequest._setReadyState (/usr/local/lib/node_modules/truffle/build/cli.bundled.js:73359:12)
at XMLHttpRequest._onHttpResponseEnd (/usr/local/lib/node_modules/truffle/build/cli.bundled.js:73514:12)
at IncomingMessage. (/usr/local/lib/node_modules/truffle/build/cli.bundled.js:73474:24)
~~~~

Any idea what could go wrong?

[escoffon]

Regarding the original problem, I use the error function in the promise, which is triggered by an EVM exception. For example a snippet of a test for a token contract:

  it("should reject transfers with insufficient funds", function() {
     var account_two = accounts[2];
     var account_three = accounts[3];

     var amount = 120000;
     var tok;

     return MyToken.deployed()
         .then(function(instance) {
               tok = instance;
               return tok.transfer(account_two, amount, { from: account_three });
           })
         .then(function(r) {
               assert(false, 'transfer (1) of insufficient funds should have failed');
               return true;
           },
           function(e) {
               assert.match(e, /VM Exception[a-zA-Z0-9 ]+: invalid opcode/, "transfer (1) of insufficient funds should have raised VM exception");
           });
     });

Since I expect the transfer call to fail, I put an assert(false) in the success function, to make the test fail. And in the failure function, I assert that the error looks like it was generated by a EVM exception. The regexp works for tests running against testrpc; it may need to be tweaked for a more general use. It might also be possible to encapsulate the two functions for reuse; currently the tests as written would require a bit of cut and paste.

[mritzco]

Is there any way to test this in solidity? Every solution I found is on JS.

I feel that in this context (Ethereum/solidity) testing for edge cases and failure is even more important that testing for working cases.

In the case of the pet shop I'd like to:

• assert that if pet > 15 it fails,
• assert it fails if no pet id
• every edge case I could imagine...

[mritzco]

Any updates? Is there any way to test in solidity? Every solution I found is for JS.

I feel that in the context of (Ethereum/solidity) testing for edge cases and failure is as important as testing for working cases.

[lrascao]

@mritzco, +1, i would also like to see progress on this

[rickbatka]

+1, I would also like to test this in solidity - not javascript. For now, workaround seems to be to test from JS and assert something in the promise error - is that still the only way to accomplish this?

[fdouglis]

+1 for solidity.

[tcoulter]

You all can test for exceptions within Solidity tests using the following tutorial: http://truffleframework.com/tutorials/testing-for-throws-in-solidity-tests

The tutorial is out of date, and is written to test for throw (a predecessor to revert(), assert(), etc.)

However, the important bit is this part:

bool r = throwProxy.execute.gas(200000)();

Assert.isFalse(r, “Should be false, as it should throw”);

Note that instead of throwing, a bool is returned with the status of the execute() function. You can use Solidity tests to test for these error cases in a similar way.

[fdouglis]

Thanks, @tcoulter, for the pointer to the proxy approach. I used that and it works. It's good that if we expect a revert() we can use that to confirm it happens, and fail the test only if it doesn't.

Now if only we could see events emitted during Solidity tests without raising a test failure. My only workaround there has been to do Assert.fail() if I want to see events, but then it counts as a unit test failure.

[whoadityanawandar]

Changing <= to '<' and '>=' to '>' works.

[jackwzp]

I worked around this by catching the error in a try-catch block. However, is there a simpler/cleaner way to do it? Something like expect(tx).to.throw().

Workaround:

it('expect a revert in smart contract', async () => {
   let contract = await MyContract.deployed();
   try {
        let tx = await contract.someFunc(inputParam, {from: accounts[0]});
   } catch(err) {
        assert(true); // expected the throw
   }
})

[jackwzp]

Actually I found a solution to this problem to write cleaner tests. In a helper.js file under test/ folder, I created a helper function that wraps all the try-catch stuff. Then in my actualTest.js file under test/, I can just write a simple one line statement.

helper.js

module.exports =  async (promise) => {
    try {
        await promise;
    } catch (err) {
        return;
    }
    assert(false, 'Expected throw not received');
}

actualTest.js

// at the top of the file
var expectThrow = require('./helper.js');

// further down in test file inside a contract()
it('expect a revert in smart contract', async () => {
   let contract = await MyContract.deployed();
   // note there is no await keyword for tx as in my previous comment
   let tx = contract.someFunc(inputParam, {from: accounts[0]}); 
   await expectThrow(tx);
})

[OliveIT]

I found @rkalis has done awesome job regarding this topic.
https://ethereum.stackexchange.com/a/60574/41404
https://kalis.me/assert-reverts-solidity-smart-contract-test-truffle/