Ethers.js: Feature request : create/import an HD wallet from xpubkey [watch only prupose]

[sulliwane]

Hello,

is there a method to instantiate a HD wallet from an extended public key? My purpose is to use getBalance() to track this HD wallet total balance (I won't have access to the private key).

Many thanks!

[24thsaint]

For Ethers.js, not yet.

But, you may use a combination of the following for the meantime:
https://docs.ethers.io/ethers.js/html/api-advanced.html#static-methods (.utils.HDNode.mnemonicToSeed)
https://www.npmjs.com/package/hdkey (hdkey.publicExtendedKey)
https://etherscan.io/apis#accounts (Get Ether Balance for multiple Addresses in a single call)

[sulliwane]

Thanks @24thsaint ,

That would be great to have a "degraded hd wallet mode" in ethers, after importing a xpubkey. I'm changing the title to reflect the feature request.

I'll look into hdkey then...! Thanks

[24thsaint]

You're welcome @sulliwane !
In the future, I would also love to see this xpubkey feature on Ethers.js, too. Along with watch-only getBalance() implementation.

[ricmoo]

Thanks @24thsaint for responding. :)

You are correct, I do not currently support xpub/xpriv, but it is something incredibly useful I always go on about how powerful BIP32 can be. The original reason I did not add it was because I didn’t want to add the bloat of base58 coding for that one case, but I totally think it makes sense to add; I will try to get it in this week. :)

[24thsaint]

@ricmoo , you're welcome.

It just so happens that I was building a special-use Ethereum wallet based on Ethers.js and I encountered a similar problem regarding xpub/xpriv. Especially fast balance inquiries, transaction history, and token balances. Would be interesting to see how Ethers.js will solve these problems.

PS. I want to extend my thanks for this wonderful library, sir! More power!

[ricmoo]

This has been added. I have not published it yet, want to try it out a bit first. Please give it a go. I'll update the docs later.

// Get an xpriv from a private HDNode
let xpriv = privateNode.extendedKey;

// Convert a private HDNode to a public HDNode
let publicNode = privateNode.neuter();

// Get an xpub from a public HDNode
let xpub = publicNode.extendedKey;

// Get a private HDNode from serialization
let privateNode = ethers.utils.HDNode.fromExtendedKey(xpriv);

// Get an public HDNode from serialization
let publicNode = ethers.utils.HDNode.fromExtendedKey(xpub);

// Note: To get an xpub from a private HDNode, just convert first:
let xpub = privateNode.neuter().extendedKey;

I've added a LOT more testing, so it takes about twice as long for tests to run now, but making sure the HDNode matches existing implementation is important. :)

[ricmoo]

This is now available in 4.0.24 and the documentation has been updated. Try it out and let me know if you have any problems.

Thanks! :)

[joshfraser]

Is it not possible to set a custom derivation path while using an xpub address?

I'm getting:
Error: invalid path - m/44'/60'/1'/0/0

[ricmoo]

Hmmm... Can you share a code snippet?

Are you using fromExtendedKey(xpub) and then trying to use derive? That won't work, the xpub specifies a location in the HD tree, but does not include the actual path. You should be able to derive specific sub-nodes though. So if the xpub was for m/44'60'/1'/0, you can derive 0.

[joshfraser]

Ah, that's exactly what I was trying to do.

let hdnode = ethers.utils.HDNode.fromExtendedKey(xpub)

for (var i = 0; i < 10; i++) {
    let node = hdnode.derivePath("m/44'/60'/1'/0/" + i);

If I can't do that, what's the best way to get a list of addresses & balances from an xpub address using a custom derivation path?

[ricmoo]

What is the path for the xpub? If you drop the m/, it should work. The m/ must be the root node (depth = 0).

[joshfraser]

The path is m/44'/60'/1'/0/0

If I drop the m/ I get throw new Error('cannot derive child of neutered node');

[ricmoo]

Oh, you also cannot derive the child of a hardened node. You can do this:

// Get the xpub (include all hardened paths you want)
let node = ethers.utils.HDNode.fromSeed("0x012345678901234567890123456789012345678901234567890123");
let child = node.derivePath("m/44'/60'/1'").neuter();
let xpub = child.extendedKey;

// Load the xpub, and derive the child paths
child = ethers.utils.HDNode.fromExtendedKey(xpub);
console.log(child.derivePath("0/0"));

[waterdrop01]

works great! Thank you

[bajian]

Oh, you also cannot derive the child of a hardened node. You can do this:

// Get the xpub (include all hardened paths you want)
let node = ethers.utils.HDNode.fromSeed("0x012345678901234567890123456789012345678901234567890123");
let child = node.derivePath("m/44'/60'/1'").neuter();
let xpub = child.extendedKey;

// Load the xpub, and derive the child paths
child = ethers.utils.HDNode.fromExtendedKey(xpub);
console.log(child.derivePath("0/0"));

How can I get the child's(child.derivePath("0/0")) private key from the master node
(let node = ethers.utils.HDNode.fromSeed("0x012345678901234567890123456789012345678901234567890123"); )

@ricmoo

Thanks!
I have found the way:node.derivePath("m/44'/60'/1'").derivePath("0/0").privateKey

[Amirhb]

hey,

will be a node.getBalance() method to return accumulated balance of all accounts in a HD wallet?

[ricmoo]

@Amirhb unfortunately no. The node object only provides a private key or address and the children private keys and addresses. There is no way to provide the total balance because there are an infinite number of keys in a given HD Node. Often a wallet will specify a certain base path and a “gap”, which is how many consecutive addresses to scan before assuming there are no more beyond that.

Basically, there is a lot of additional heuristics that need to be used to figure out what you wish to include, and it can also result in a lot of addresses. You may wish to use a balance contract for this, if you wish to try implementing it yourself. :)

[Amirhb]

@ricmoo I have something in my mind like getting the balance of the master-node address. Seems I have to calculate it manually.

[ricmoo]

@Amirhb Yes, but it is fairly simple write, depending on which heuristics you want to use. For example, if you just want the top level of the Ethereum cointype, you could use something similar to this pseudocode:

// You must choose this, depending on the heuristic you are targeting
//  but usually 20 or 40 are common
const DESIRED_GAP = 20;

async function getBalance(xpub) {
    const node = ethers.HDNode.fromExtendedKey(xpub).derive(ethers.defaultPath);
    let j = 0, gap = 0;
    let total = ethers.constants.Zero;
    while (gap < DESIRED_GAP) {
        const address = node.derivePath(String(j++)).address;
        const balance = await provider.getBalance(address);
        gap++;
        if (balance.isZero()) {
            const nonce = await provider.getTransactionCount(address);
            if (nonce) { gap = 0; }
        } else {
            gap = 0;
            total = total.add(balance);
        }
    }
    return total;
}

This is just pseudo code off the top of my head though, but that should give you a good starting point. You could do more complicated things, like building up a list of DESIRED_GAP length first, and using a balance contract to fetch many results at once, for example, which would greatly decrease the number of remote calls necessary and be much faster. If you deal with sparse or heavily used address, you may be able to use a stochastic sampling to get a wider gap to figure out what to fill in. There is a lot of possible ways to scan the infinite addresses to more quickly find the most-likely-to-contain-funds addresses. But the above code will probably be sufficient for most of the common wallets, such as MetaMask.

Hope that helps! :)

[Amirhb]

@ricmoo Thanks. It helps a lot.

[Amirhb]

@ricmoo Actually looking at m / purpose' / coin_type' / account' / change / address_index, I meant to get the balance at the account level and not the HD wallet. Still, there's no way?

[ricmoo]

@Amirhb yeah, same thing. It’s a tree, so you can always go deeper. But the above “crawler” should work well for any xpub used by a standard wallet for a given cointype. It’s actually how most wallets bootstrap the UTXO set for coins like Bitcoin when you import a new mnemonic. :)