Ethers.js: Calling a function with nested structs

[naddison36]

@ricmoo do you have an example of how to use the new ABI encoder released in Solidity 0.4.19? I'm particularly interested in how to pass in nested structs as a parameter. For example, how to call addTestStruct in the following example contract using Ethers.js

pragma solidity 0.4.19;
pragma experimental ABIEncoderV2;

contract TestContract
{
    struct SubStruct {
        uint256 id;
        string description;
    }

    struct TestStruct {
        uint256 id;
        string description;
        SubStruct subStruct1;
    }

    TestStruct[] public tests;

    function addTestStruct(TestStruct testStruct) public
    {
        tests.push(testStruct);
    }
}

[ricmoo]

It doesn't support named parameter yet (mostly because I haven't written test cases yet), but you can do this:

var value = [
    someId,
    someDescription,
    [
        subId,
        subDescription
    ]
];
contract.addTestStruct(value);

It is a very simple change to add names, I just need to make sure it works, so need to test it. After that, you can do:

var value = { id: someId, description: someDescription: subStruct1: { id: subId, description: subDescription }};
contract.addTestStruct(value);

[naddison36]

Great, i’ll give it a try later tonight. Thanks @ricmoo

[naddison36]

@ricmoo I tried calling addTestStruct in the above example but interface.getParamCoder throws an invalid type exception. The parameter I was using was using to test was [100, "testing",[10, "sub testing"]]

[naddison36]

It turns out I wasn't running the latest version of the ethers-contracts sub library :-(

I can now send the transaction and receive a transaction receipt. The problem is I'm getting a response with a status of 0. Even if I change the function to just return true I get a failed transaction with a status of 0.

function addTestStruct(TestStruct testStruct) public returns (bool)
{
    return true;
}

The account that is sending the transaction has funds and I can call other functions that don't pass in a struct.

I'm running this on a local dev instance of Parity. I've also tried against a remote Geth instance.

Any ideas?

[ricmoo]

Hmmm... I've only tried returning structs so far. Maybe I'm computing the wrong signature hash for tuples? Can you paste here the function hashes the compiler spits out for that above function?

[naddison36]

"functionHashes": {
    "addTestStruct((uint256,string,(uint256,string)))": "db4063ba",
    "tests(uint256)": "de22c8d4"
},

[naddison36]

Stepping through the Ethers code I get a signature of addTestStruct(tuple(uint256,string,tuple(uint256,string))) and sighash of 0x46f8e8de

Looks like you don't need to add the tuple's

[naddison36]

@ricmoo any progress on this one? Can I help with gathering more data or running tests?

[riordant]

Also would like to know what the progress is on this.
I am replicating the above contract and deploying on a local testrpc instance - testrpc is throwing a revert error. functions with no struct parameter do not do this. code:

const solc = require('solc') 
const ethers = require('ethers');
const TestRpc = require("ethereumjs-testrpc");

var contract = "pragma solidity 0.4.19; pragma experimental ABIEncoderV2; contract TestContract{ struct SubStruct { uint256 id; string description; }struct TestStruct { uint256 id; string description; SubStruct subStruct1; }TestStruct[] public tests;function addTestStruct(TestStruct testStruct) public { tests.push(testStruct); } }"
var compiled = solc.compile(contract, 1)

var contractName = Object.keys(compiled.contracts)[0] 
var abi = compiled.contracts[contractName].interface
var bytecode = '0x' + compiled.contracts[contractName].bytecode

var wallet = new ethers.Wallet('0x2b24dcbd27cfb2e063f33c05574bdb39e94b79eef80adcbe614f893325001695'); //priv key for web3 account 0 - change as necessary

const port = 8545;

const testRpc = TestRpc.server({
  port,
  logger: console,
  blocktime: 0.5,
  network_id: 15,
  unlocked_accounts: [ wallet.address ],
  accounts: [{
    balance: '8000000000000000000000000000000000000000000000000000000000000000',
    secretKey: wallet.privateKey
  }]
});

var provider = new ethers.providers.JsonRpcProvider(`http://localhost:${port}`, { chainId: 15 });
wallet.provider = provider;

var deployTransaction = ethers.Contract.getDeployTransaction(bytecode, abi);
var transaction;
wallet.sendTransaction(deployTransaction).then(function(sendResult) {
    transaction = {
        from: sendResult.from,
        nonce: sendResult.nonce
    };
});

var contractAddress = ethers.utils.getContractAddress(transaction);

var contract = new ethers.Contract(contractAddress, abi, wallet);

contract.addTestStruct([100, "testing",[10, "sub testing"]]).then(function(result){
  console.log(result);
});

/* error message in ethers window:
 * (rejection id: 1): Error: VM Exception while processing transaction: revert
 */

/* in testrpc:
 *  Gas usage: 24326
 *  Block Number: 2
 *  Block Time: Fri Jan 05 2018 12:22:21 GMT+0000 (GMT)
 *  Runtime Error: revert
 */

[ricmoo]

I've confirmed your conclusion that the tuple word is omitted from signature hashes.

I'm building the test cases right now to make sure signature hashes are correct in general, then this should work just peachy.

[ricmoo]

It is not for the faint of heart, but here is the official repository:
https://github.com/ethereum/ens

And here is a script I have that deploys ENS to a TestRPC (with a given address set to be "god" to register anything the heart desires):
https://github.com/ethers-io/ethers-cli/blob/master/lib/test-provider.js

[ricmoo]

Running the final test cases now. Then I will npm publish.

Also, be sure to use the 4.19 compiler (the 4.18 which was used to generate test cases had a lot of bugs regarding passing in nested dynamic sized items and structs).

[ricmoo]

This is fixed in https://github.com/ethers-io/ethers.js/commit/df930103e75dd89c9f6c14acda857a07f1b0d70c with test cases added for computing method signatures in general.

[naddison36]

I can confirm Ethers is working for structs passed into public and external functions. I'm having trouble with structs in events. I'll raise a separate issue for that.

[sonaye]

I am trying to make use of this feature, but no luck so far. The output has a very odd structure to it.

pragma solidity ^0.4.19;

contract Users {
  struct User {
    bytes32 name;
  }

  User[] internal users;

  function signUp(bytes32 name) public {
    users.push(User(name));
  }

  function getUsers() public view returns (User[]) {
    return users;
  }
}

const { Contract, Wallet, providers, utils } = require('ethers');

const contract = new Contract(
  'ADDRESS',
  [
    {
      constant: true,
      inputs: [],
      name: 'getUsers',
      outputs: [
        {
          components: [{ name: 'name', type: 'bytes32' }],
          name: '',
          type: 'tuple[]'
        }
      ],
      payable: false,
      stateMutability: 'view',
      type: 'function'
    },
    {
      constant: false,
      inputs: [{ name: 'name', type: 'bytes32' }],
      name: 'signUp',
      outputs: [],
      payable: false,
      stateMutability: 'nonpayable',
      type: 'function'
    }
  ],
  new Wallet(
    'PRIVATE_KEY',
    new providers.JsonRpcProvider('http://localhost:8545')
  )
);

contract
  .signUp(utils.toUtf8Bytes('Bob'))
  .then(() => contract.getUsers())
  // .then(users => users[0].map(user => utils.toUtf8String(user.name)))
  .then(console.log)
  .catch(err => console.log(err.message));

// output is invalid
// [
//   [
//     [
//       '0x00000000000000000000000000000000000000000000000000000000000000c0',
//       name: '0x00000000000000000000000000000000000000000000000000000000000000c0'
//     ]
//   ]
// ];

Can you advise on this?

[ricmoo]

The data is correct. The result of any Solidity function is an array (or if named, array with named arguments). It might seem more obvious if you used returns (User[] users) instead, but:

var users = result[0];
// Or if you make the above change, you could also use "var users = result.users"

var firstUser = users[0];

// Access by positional value
var name = firstUser[0]

// Access by keyword value
var name = firstUser.name

Try adding more fields to the Struct and things will start looking like they make more sense. :)

Let me know if you have any more issues.

[sonaye]

@ricmoo I am already following your exact description of how it should be done. Notice how the output has the structure [ 'foo', bar: 'baz' ] which is invalid in JavaScript, is this intentional? This breaks looping through the results with map(), forEach(), filter(), etc.

And yes users[0][0].name does work, I switched to a for loop for now to handle this but can you elaborate on the idea behind this structure?

Edit: It stopped working after adding multiple records (users), getting an invalid bytes32
error, feel free to replicate and test on Ropsten.

// pragma solidity ^0.4.19;

// contract Users {
//   struct User {
//     bytes32 name;
//     uint mobile;
//   }

//   User[] internal users;

//   function signUp(bytes32 name, uint mobile) public {
//     users.push(User(name, mobile));
//   }

//   function getUsers() public view returns (User[]) {
//     return users;
//   }
// }

const { Contract, Wallet, providers, utils } = require('ethers');

const contract = new Contract(
  '0xf083dC9aB08b7572737Cf4d2Cc3e0D7e7A933297',
  [
    {
      constant: true,
      inputs: [],
      name: 'getUsers',
      outputs: [
        {
          components: [
            { name: 'name', type: 'bytes32' },
            { name: 'mobile', type: 'uint256' }
          ],
          name: '',
          type: 'tuple[]'
        }
      ],
      payable: false,
      stateMutability: 'view',
      type: 'function'
    },
    {
      constant: false,
      inputs: [
        { name: 'name', type: 'bytes32' },
        { name: 'mobile', type: 'uint256' }
      ],
      name: 'signUp',
      outputs: [],
      payable: false,
      stateMutability: 'nonpayable',
      type: 'function'
    }
  ],
  new Wallet(
    'PRIVATE_KEY',
    new providers.getDefaultProvider(providers.networks.ropsten)
  )
);

contract.signUp(utils.toUtf8Bytes('Bob'), '123');

contract
  .getUsers()
  .then(result => {
    const users = [];
    for (let i = 0; i < result[0].length; i += 1)
      users.push(utils.toUtf8String(result[0][i].name)); // utils.stripZeros() seems needed
    return users;
  })
  .then(console.log)
  .catch(err => console.log(err.message));

[ricmoo]

It is not invalid JavaScript, it is constructed by JavaScript inside the Interface object. For example:

var a = ["hello", "world"];
a.firstItem = "goodbye";
console.log(a);
// [ 'hello', 'world', firstItem: 'goodbye' ]

> a.forEach(function(x) { console.log(x); })
// hello
// world

> a.map(function(x) { return ("p:" + x); });
// [ 'p:hello', 'p:world' ]

> a.filter(function(x) { return x === 'hello'; });
[ 'hello' ]

Since arrays are just objects, they can have arbitrary properties set on them (with the exception of length, which in ethers becomes _length.

The reason for this is that Solidity results are always positional, and can optionally additionally be property based. Both the positional argument and property argument reference the identical object, so changes in one reflect in the other and no additional memory is used.

For example:

function something() returns (string foo, bytes32 bar);

returns an array of length 2. So, result.foo and result[0] are the same object, also result.bar and result[1] are the same.

So,

function something() returns (string foo);

returns an array of length 1. So, results.foo and result[0] are the same object.

Now,

function something() returns (string[] foo)

returns an array of arrays. So, results[0][0] and results.foo[0] are the same object.

Finally:

struct Foo {
    address name;
}
function something() returns (Foo[] foo);

returns an array of Food. So, results[0][0] and results.foo[0] are the same object. And to get into those object, you can user either the name or the positional argument, which means results[0][0][0] is the same item as results[0][0].name.

Make sense? Things get nested quite quickly, which is why I suspect most people will use property-based access, in which case the last example becomes result.foo[0].name.

[sonaye]

@ricmoo Thanks for the detailed explanation, much appreciated, shows a great character.

Not to take away from your efforts but truly what you are trying to explain is/was very clear to me from the beginning, there is no confusion here, I understand how we can access the props. I do now understand better that you are treating arrays here as objects (or at least taking advantage of the fact that they are), interesting usage I must say, never stumbled upon something like it before, I think I just learned a new trick!

Now that we got that out of the away, still, my implementation follows your explanation word by word, why do we still not get the desired results? If my code reflects something other than that please let me know. If you could replicate the test and still come up with valid results then the issue could be related to my environment (which I doubt is the case because I tested on multiple networks, but it is still a possibility).

Thanks again, I am actually moving away from web3 to ethers, and hopefully maybe contribute in the future, but let me get my hands steady first.

[ricmoo]

I see, I see...

First thing I notice is you are using a bytes32 to store a string. You can certainly write a method to compact/expand short strings into a bytes32, but keep in mind utils.toUtfBytes returns a bytes, not a bytes32, likewise, utils.toUtfString takes in bytes and returns a string. If you pass in a bytes32 which is largely zero-padded, you will get weird results with lots of extra zeros.

bytes32 are fixed length strings (always exactly 32 bytes), whereas bytes are length-prefixed (always at least 32 bytes; more specifically, (32 + Math.ceil(l / 32)) where l is the length of there bytes).

You should change bytes32 name to string name. If you goal is to save space, you will need to write a binary packing method, but keep in mind this means your app will need to do data validation when getting results, since depending on your binary packing method, invalid entries could be created.

For a simple packing method, you could use something similar to:

// string to bytes32
function encode(value) {
    var result = ethers.utils.hexlify(ethers.utils.toUtf8Bytes(value));
    while (result.length < 66) { result += '0'; }
    return value;
}

function decode(value) {
    value = ethers.utils.arrayify(value);
    var firstZero = 0;
    for (var firstZero = 0; firstZero < 32; firstZero++) {
        if (value[firstZero] === 0) { break; }
    }
    return ethers.utils.toUtf8String(value.slice(0, firstZero));
}

Keep in mind that this will only work for strings less than 32 bytes.

Make more sense, now that I think I'm answering the right question? :)

[sonaye]

Fantastic! we are getting somewhere.

utils.toUtfBytes returns a bytes, not a bytes32

This explains the odd behavior.

Switching to a dynamic string size is ok with me, and I guess we could do

uint nameLength = bytes(name).length;
require(nameLength > 0 && nameLength < 33); // max 32 char string

Shouldn't there be a first-class support for bytes32 though? meaning having these packing methods implemented internally, maybe under utils, just a thought.

Changing the type to a string however didn't fix things for me I am a afraid, still the same error except it's now invalid bytes instead of invalid bytes32.

contract
  .signUp('Bob')
  .then(() => contract.getUsers())
  .then(result => {
    console.log(result[0].length);
    return result[0];
  })
  .then(users => users.map(user => user.name))
  .then(console.log)
  .catch(err => console.log(err.message));

Even simply deploying through Remix and adding users there then executing

contract
  .getUsers()
  .then(console.log)
  .catch(console.log);

results the same error.

Same original contract https://github.com/ethers-io/ethers.js/issues/84#issuecomment-365480096 but with a string type.

Am I connecting to ganache properly here? I already see the transactions/calls appearing when I execute but just to double check.

new providers.JsonRpcProvider('http://localhost:8545')

[sonaye]

pragma experimental ABIEncoderV2; is the magic piece that was missing after all 😅 thanks for the help!