Web3.js: Block timestamp bigNumber coversion issue (WebSocket connection)

[Makapoh54]

Hello,

I am using "web3": "~1.0.0-beta.26" to make connection to JPMorgan Quorum (https://github.com/jpmorganchase/quorum). Quorum network is configured to use Raft consensus algorithm. Raft algorithm is making changes to block timestamp value, so it is larger (Unix nanoseconds - 1506355254821575394 e.g.) then default - https://github.com/jpmorganchase/quorum/issues/190. Due to this, when I am trying to deploy contract, using WebSocket provider, I receive error from bigNumber.js library:
Error: Number can only safely store up to 53 bits

This error is caused by https://github.com/ethereum/web3.js/blob/develop/lib/web3/formatters.js#L171; looks like Raft timestamp is too big for converting it to the number.

Timestamp value from failing block object:

{ 
"timestamp": "0x14fe06494f652193"
}

As I understood outputBlockFormatter is only used during WebSocket connection (ws provider), Rpc connection (http provider) is deploying fine. It was OK to use only RPC connection, but, with recent updates, subscription is possible only using WebSocket.

I understand, that nonstandard ethereum networks is not Your main field; however, would be cool to hear some thoughts about this issue.

[sekharkumarroy]

I'm facing the same issue. Any solution yet ?

[Freyert]

Here's a hack you can do while you wait:

Go into: node_modules/web3/packages/web-3-core-helpers and delete the line seen in the link posted by @Makapoh54:

https://github.com/ethereum/web3.js/blob/develop/lib/web3/formatters.js#L171

This also occurs when using the HTTP provider.

[Freyert]

Proposed, fix is to down sample into MS, for this formatter if it's above 53 bytes.

[Freyert]

1351 might be an even better fix.

[Freyert]

Hmmm fishy, works as of cd1cfd9db6cacb494884a1824f8562c6440f85df

> const BigNumber = require('bignumber.js')
undefined
> var x = new BigNumber('0x14fe06494f652193')
undefined
> x
BigNumber { s: 1, e: 18, c: [ 15126, 53436777472403 ] }
> x.toNumber()
1512653436777472500
> const utils = require('./lib/utils/utils')
undefined
> utils.toDecimal(x)
1512653436777472500
> utils.toDecimal('0x14fe06494f652193')
1512653436777472500

My timestamp 0x151e443edde2abcb also works.

[Freyert]

EH, scratch that.

> const BigNumber = require('bignumber.js')
undefined
> var x = new BigNumber('0x151e455f0634e16a')
undefined
> x.toNumber()
1521729998526144800
> var z = x.toNumber()
undefined
> z.toString('hex')
RangeError: toString() radix argument must be between 2 and 36
    at Number.toString (<anonymous>)
> z.toString(16)
'151e455f0634e100'
> z.toString(16) === '151e455f0634e16a'
false

toBN is enforcing safety rules for BigNumber

[Freyert]

Since Ethereum _can_ and _will_ have numbers larger than Number can handle I think a more wide spread solution is necessary here.

1. All numbers are BigNumber
2. Instead of JSON.stringify, all objects should have a serialize function that converts BigNumbers into strings.

@frozeman seems like a regular issue with a drastic solution. Do you have any maintainer insight that might simplify this?

[azeem-r00t]

A few points on this.

1. BigNumber used to have an option to turn off errors which is no longer viable.
2. The simplest way to get around this is to use web3.extend and replace the formatter for the getBlock call.

    let Web3Helpers = require('web3-core-helpers');
    let Web3Utils = require('web3-utils');

    web3.extend({
      property: 'quorum_raft',
      methods: [{
        name: 'getBlock',
        call: (args) => {
          return (_.isString(args[0]) && args[0].indexOf('0x') === 0) ? "eth_getBlockByHash" : "eth_getBlockByNumber";
        },
        params: 2,
        inputFormatter: [Web3Helpers.formatters.inputBlockNumberFormatter, function (val) { return !!val; }],
        outputFormatter: this.outputBlockFormatter
      }]
    });

An example implementation of outputBlockFormatter:

    try {
      // check to see if we have an issue with timestamp
      Web3Utils.hexToNumber(block.timestamp);
    }
    catch (err) {
      // WARNING this implementation assumes RAFT timestamp (precision is nanoseconds)
      // You should not simply assume RAFT if it is not successful rather take a consensus specific 
      // action

      // we are being extra cautious here and converting it back to the same format it was in after dropping
      // the nanoseconds (i.e. a hex string prefixed with 0x)
      block.timestamp = '0x' + Math.floor(block.timestamp / 1e6).toString(16);
    }
    return Web3Helpers.formatters.outputBlockFormatter(block);

Now, you can simply call:

web3.quorum_raft.getBlock(...)

1. This doesn't completely solve this problem though. If you create a subscription for newBlockHeaders, you will run into the same problem. There is no documented way of extending the subscription call (eth_subscribe). I managed to hack my way through it. If anyone is interested in that, please let me know (not posting here as the hack may not be valid if some of the internal details of web3 change).

2. For a long term, solution, it would be great if web3 library can make the formatters pluggable i.e. on any given call, allow the caller to specify their own formatter. That will help support other protocols and consensus algorithms that geth-type implementations may employ.

Hope it helps someone out there. :)

[thekevinbrown]

Also hitting this issue when using Truffle beta (uses Web3 1.0) with Ganache, then fast forwarding into the future.

This is the timestamp that breaks running my tests: 0x858d5aeaef564eacc

[nivida]

This will be fixed with https://github.com/ethereum/web3.js/pull/2144

[nivida]

Duplication of: https://github.com/ethereum/web3.js/issues/1905