Ethers.js: Way to link library before deployment

[Perseverance]

Hello, feel free to tell me if this is not concerning ethers, but:

I am trying to deploy contracts using the standard deploy sequence.

In web3 (truffle) There was something called .link that allowed me to add the library address in the bytecode. Is there ethers method that can help me achieve this, or should I go with regex or something like this?

[ricmoo]

I’m unfamiliar with that technique. Does it just shim the address into the byte code? How does it know where?

Is the library and on-chain contract?

Please feel free to send me links to documentation and I’ll take a look. :)

[Perseverance]

Hey @ricmoo thanks for the swift reply. I will describe the problem briefly and will give you example tomorrow morning as I am out of the office now.

The real world scenario: We have a smart contract dealing with a business logic and it requires a Linked List. The Linked List we've made into a library.

When compiling (using truffle compile) a contract to it's .json file the bytecode for the contract contains 40 symbols in the format __LinkedList________ (underscores until the 40th symbol).

Before wasting any more of your time, I've done some digging into truffle and it appears that they use this thing to leave the trace that I found:

https://github.com/trufflesuite/truffle-compile/blob/13d1a8b04bdb690cd369bd5d825d3e9f7a8e8fc4/index.js#L209 T

Question 1 (Low importance): To the best of your knowledge is this some required pattern or something they use for their convenience?

After further digging I found that on deploy time they replace these 40 symbols:

https://github.com/trufflesuite/truffle-deployer/blob/develop/src/linker.js

Question 2 (Medium importance): Is this something that concerns ethers in the light that we (ethers) already have deploy functionality. Is this something that should be handled by the developer beforehand?

Question 3 (High importance): Have you done deployment of contracts that use libraries using ethers. Any similar issues?

Question 4 (Most important): The reason we've started deploying using ethers is that truffle is, at the moment, unreliable for mainnet deployments, especially if you need to call some functions in order to "set the scene". We're starting the development of deployment tool that we want to base on ethers. What are the proper channels to ask for your permission to use this, and is there a way to communicate with you if we (quite inevitably) need your guidance?

[alcuadrado]

I've been dealing with this in Buidler, so I'm happy to help here :)

A library in solidity is just a contract whose storage is ignored. When you call a library's function in solidity, the call is compiled to a delegatecall to the librarie's contract. solc has no way to know the address of that contract in compile time. So you have to tell it which addresses to use, o it can compile with a "generic address" or symbol (ie: the __libName___) for each library. Note that the latter is pretty similar to what compilers and linkers do with dynamic libraries.

When solc compiles a contract that uses libraries, it outputs a linkReferences objects. This object acts as a table that indicates which parts of the bytecode must be replaced with each library's address.

Unluckily truffle's artifact format doesn't contain the link references, so you should do some kind of hack to get it. In the case of Buidler, as it works as a replacement of Truffle, I ensured that this information isn't lost.

Q2: I think this should be supported, but that may imply defining a self-contained artifacts format. I have this same problem with Buidler, as I want to replace web3 for something else (probably ethers).

Q3: I have don this, you can use this workaround to link the bytecode before deploying:

function link(bytecode, libName, libAddress) {
  let symbol = "__" + libName + "_".repeat(40 - libName.length - 2);
  return bytecode.split(symbol).join(libAddress.toLowerCase().substr(2))
}

Note that it will break if the library name is longer than 36 characters, or if the file containing it has other libraries/contracts.

[Perseverance]

@alcuadrado Love it! Thanks for the explanation. I will definitely check Buidler up as it seems to solve most of my concerns with truffle. Also I will definitely change web3 to ethers. Thanks guys. If you feel so close this question.

[ricmoo]

Perfect explanation, thanks!

Can you point me to some transactions on the blockchain post-link as well as the code + link data that was used? I will need to create test cases.

I’ll also look into the code to check on restricted characters and such.

[Perseverance]

@ricmoo - me or @alcuadrado ? I am not allowed to show the full code of the business logic contract, but here is the LinkedList. It's a bit specific for our use case, but it should do the trick.

It'd be trivial for you to write some simple smart contract to do this (Or if you want me to - I can do this too). Once you truffle compile it (whitout linking it in the migration script) you should see the behaviour.

pragma solidity ^0.4.21;

library LinkedList {

    modifier onlyExistingNode(Data storage self, address userAddress) {
        Node storage currentNode = self.nodes[userAddress];
        require(currentNode.nodeValue != 0x0);

        _;
    }

    modifier notSingleNode(Data storage self) {
        require(self.linkedListLength >= 2);

        _;
    }

    modifier notLastNode(Data storage self, address userAddress) {
        Node storage currentNode = self.nodes[userAddress];
        require(currentNode.nextNode != 0x0);

        _;
    }

    struct Node {
        address nodeValue; 
        address previousNode;
        address nextNode;
    }

    struct Data {
        address head;
        address tail;
        uint256 linkedListLength;
        mapping (address => Node) nodes;
    }

    function isSingleNodeList(Data storage self) public view returns(bool) {
        return self.linkedListLength <= 2;
    }

    function push(Data storage self, address userAddress) public {
        require(userAddress != address(0));

        if (self.linkedListLength == 0) {
            Node memory firstNode = Node({
                nodeValue: userAddress,
                previousNode: address(0),
                nextNode: address(0)
            });

            self.nodes[userAddress] = firstNode;
            self.head = userAddress;
            self.linkedListLength += 1;
        } else {
            Node storage previousNode = self.nodes[self.tail];
            previousNode.nextNode = userAddress;

            Node memory currentNode = Node({
                nodeValue: userAddress,
                previousNode: self.tail,
                nextNode: address(0)
            });

            self.nodes[userAddress] = currentNode;
            self.linkedListLength += 1;
        }

        self.tail = userAddress;
    }
}

In the contract:

using LinkedList for LinkedList.Data;
    LinkedList.Data linkedListData;

[kyriediculous]

If i'm not mistaken your artifact has a property called unlinked_binary which takes an address without the prefix.

Contract.unlinked_binary = Contract.unlinked_binary
          .replace('1111222233334444555566667777888899990000',
library.address.slice(2))

[ricmoo]

I'm thinking of making a utility class for this, which would take in bytecode and a replacer, which can be either a:

• an dictionary which maps names to addresses
• a function, which is called with the name and should return an address

I'm also thinking this should be asynchronous, which would allow remote or non-sync filesystem access as necessary.

[ricmoo]

There was a great audit presented during DevCon4 about this exact topic being highly exportable to create malicious code, so the format will be changing soon.

I think for now, I will close this and leave it as a exercise for the compilation and linking level for the tooling.

I'm fully open to continue discussion though, I monitor closed issues, or feel free to re-open.

Thanks! :)

[YoonjaeYoo]

For Solidity >= 5.0, you need to do it like:

const {utils} = require("ethers");

function link(bytecode, libraryName, libraryAddress) {
    const address = libraryAddress.replace('0x', '');
    const encodedLibraryName = utils
        .solidityKeccak256(['string'], [libraryName])
        .slice(2, 36);
    const pattern = new RegExp(`_+\\$${encodedLibraryName}\\$_+`, 'g');
    if (!pattern.exec(bytecode)) {
        throw new Error(`Can't link '${libraryName}'.`);
    }
    return bytecode.replace(pattern, address);
}

[blakewest]

@YoonjaeYoo that function does not work for me. I'm on sol 0.6.8. Relevant code for me looks like...

pragma solidity ^0.6.8;
pragma experimental ABIEncoderV2;

// This is the library I'm trying to import, for which there is a local copy on my machine that I deploy ahead of time, and then try to link
import "./external/FPMath.sol";  


contract MyContract is Ownable {
  // does things
 }

Then I have a little deploy script that does the following...

async function deployContract(contractName, opts={}) {
  let Contract
  if (opts.link) {
    const contractData = JSON.parse(fs.readFileSync(`${ARTIFACTS_PATH}${contractName}.json`));
    opts.link.forEach((libraryInfo) => {
      const { name, address } = libraryInfo;
      contractData.bytecode = link(contractData.bytecode, name, address);
    });
    Contract = await ethers.getContractFactory(contractData.abi, contractData.bytecode)
  } else {
    Contract = await ethers.getContractFactory(contractName);
  }
  // This sends the transaction
  const contract = await Contract.deploy();
  // This waits for it to be mined
  await contract.deployed();

  if (opts.initialize) {
    await contract.initialize(...opts.initialize);
  }

  console.log(`Successfully deployed the ${contractName}`);
  return contract;
}

And when I call that function like await deployContract("MyContract", {link: [{name: "FPMath", address: fpMath.address}]});, I get the error Can't link FPMath..

Any idea what's going on? Thanks!

[moodysalem]

Our code we used for hardhat artifacts:

import { utils } from 'ethers'

export default function linkLibraries(
  {
    bytecode,
    linkReferences,
  }: {
    bytecode: string
    linkReferences: { [fileName: string]: { [contractName: string]: { length: number; start: number }[] } }
  },
  libraries: { [libraryName: string]: string }
): string {
  Object.keys(linkReferences).forEach((fileName) => {
    Object.keys(linkReferences[fileName]).forEach((contractName) => {
      if (!libraries.hasOwnProperty(contractName)) {
        throw new Error(`Missing link library name ${contractName}`)
      }
      const address = utils.getAddress(libraries[contractName]).toLowerCase().slice(2)
      linkReferences[fileName][contractName].forEach(({ start: byteStart, length: byteLength }) => {
        const start = 2 + byteStart * 2
        const length = byteLength * 2
        bytecode = bytecode
          .slice(0, start)
          .concat(address)
          .concat(bytecode.slice(start + length, bytecode.length))
      })
    })
  })
  return bytecode
}

[valerioleo]

Our code we used for hardhat artifacts:

import { utils } from 'ethers'

export default function linkLibraries(
  {
    bytecode,
    linkReferences,
  }: {
    bytecode: string
    linkReferences: { [fileName: string]: { [contractName: string]: { length: number; start: number }[] } }
  },
  libraries: { [libraryName: string]: string }
): string {
  Object.keys(linkReferences).forEach((fileName) => {
    Object.keys(linkReferences[fileName]).forEach((contractName) => {
      if (!libraries.hasOwnProperty(contractName)) {
        throw new Error(`Missing link library name ${contractName}`)
      }
      const address = utils.getAddress(libraries[contractName]).toLowerCase().slice(2)
      linkReferences[fileName][contractName].forEach(({ start: byteStart, length: byteLength }) => {
        const start = 2 + byteStart * 2
        const length = byteLength * 2
        bytecode = bytecode
          .slice(0, start)
          .concat(address)
          .concat(bytecode.slice(start + length, bytecode.length))
      })
    })
  })
  return bytecode
}

@moodysalem what is the advantage of this over using the API provided by hardhat?

const contractFactory = await this.env.ethers.getContractFactory(
  "Example",
  {
    libraries: {
      ExampleLib: "0x..."
    }
  }
);

from: https://hardhat.org/plugins/nomiclabs-hardhat-ethers.html#library-linking

[moodysalem]

@moodysalem what is the advantage of this over using the API provided by hardhat?

if hardhat didn't compile it you can't use that (eg it came from an npm package)

[omarish]

if hardhat didn't compile it you can't use that (eg it came from an npm package)

👍, I just ran into this use case today. Thanks for the snippet, @moodysalem.

[alcuadrado]

Hey @moodysalem, @omarish, do you mind explaining a bit more about when you need to do that?

[omarish]

For sure, here's what I'm trying to do and how I'm solving it right now:

• I'm building a solidity contract that references the Uniswap V3 contracts. One of the main Uniswap contracts I need depends on a linked library (NFTDescriptor).
• In my test suite, I'm trying to simulate my contract interacting with Uniswap, so I need to deploy Uniswap's contracts on my local test chain.
• I don't want to copy the V3 source into my codebase, so instead I'm relying on the ABI/bytecode from the @uniswap/v3-{core, periphery} npm packages.
• This is what I'm doing right now: https://github.com/omarish/uniswap-v3-staker/blob/master/test/shared/fixtures.ts#L114

Does that answer your question?

[alcuadrado]

Yes, thanks @omarish. We should add a way to load artifacts from dependencies rather soon.