Solidity-coverage: Zero percent coverage with buidler plug-in (using Ethers, Waffle, Typechain)

[QYuQianchen]

Issue

Following the documentation, I tried to integrate solidity-coverage in the typescript-solidity-dev-starter-kit project. However, the coverage remains 0 % when running npx buidler coverage (see screenshot below) and no ./allFiredEvents was generated.

Configuration

• package: "solidity-coverage": "^0.7.1",

• buidler.config.ts file

import { BuidlerConfig, usePlugin } from "@nomiclabs/buidler/config";

usePlugin("@nomiclabs/buidler-waffle");
usePlugin("@nomiclabs/buidler-etherscan");
usePlugin("buidler-typechain");
usePlugin("solidity-coverage");

const INFURA_API_KEY = "";
const RINKEBY_PRIVATE_KEY = "";
const ETHERSCAN_API_KEY = "";

const config: BuidlerConfig = {
  defaultNetwork: "buidlerevm",
  solc: {
    version: "0.6.2"
  },
  networks: {
    rinkeby: {
      url: `https://rinkeby.infura.io/v3/${INFURA_API_KEY}`,
      accounts: [RINKEBY_PRIVATE_KEY]
    }
  },
  etherscan: {
    // The url for the Etherscan API you want to use.
    url: "https://api-rinkeby.etherscan.io/api",
    // Your API key for Etherscan
    // Obtain one at https://etherscan.io/
    apiKey: ETHERSCAN_API_KEY
  },
  typechain: {
    outDir: "typechain",
    target: "ethers"
  }
};

export default config;

[cgewecke]

@ChenchenYo Thanks for reporting.

The TS starter kit is tailored to BuidlerEVM and solidity-coverage currently depends on ganache-cli.

Here's a list of small changes that were necessary to get everything to work (for me):

Add a .solcover.js config file with workflow hooks for the typechain build:

const shell = require('shelljs'); // This module is already a solidity-coverage dep

module.exports = {
  onCompileComplete: async function(config){
    await run('typechain');
  },
  onIstanbulComplete: async function(config){
    shell.rm('-rf', './typechain'); // Clean up at the end
  }
}

Add a "coverage" network to buidler.config.ts . (Not sure why this is necessary - maybe something about the way waffle and coverage plugins interact)

networks: {
  ...
  coverage: {
    url: 'http://127.0.0.1:8555' // Coverage launches its own ganache-cli client
  }
},

Remove ./test from the .tsconfig include array. tsc throws errors when trying to resolve modules from dynamically generated folders like typechain/*. Removing test as a target makes the problem disappear.

-  "include": ["./scripts", "./test"],
+  "include": ["./scripts"],

Modify the counter.ts test setup (so that it is ethereum client agnostic)

-import { waffle } from "@nomiclabs/buidler";
+import { waffle, ethers } from "@nomiclabs/buidler";
+import { Wallet } from "ethers";
 import chai from "chai";
 import { deployContract, solidity } from "ethereum-waffle";

// ... some other lines of code ....

describe("Counter", () => {
   const provider = waffle.provider;

   // 2
-  let [wallet] = provider.getWallets();
+  let signers;

   // 3
   let counter: Counter;

   beforeEach(async () => {
-    counter = await deployContract(wallet, CounterArtifact) as Counter;
+    signers = await ethers.signers();
+    counter = await deployContract(<Wallet>signers[0], CounterArtifact) as Counter;
...

Run the command with flags...: (You do not need to run the build step - this is handled by the workflow hooks described above).

• --temp artifacts: The tests have hardcoded import paths and expect the artifacts to reside in the default "artifacts" folder. Usually coverage uses a temporary folder with a different name.
• --network coverage: the network needs to be specified.

npx buidler coverage --temp artifacts --network coverage

Output: (includes expected error)

Please just lmk if you're not able to get things running with these steps....

[QYuQianchen]

@cgewecke Thank you for the detailed solution! It finally works on my end.

The key points are indeed to specify a coverage network and use a client-agnostic provider.

• Add a "coverage" network to buidler.config.ts .
• Modify the counter.ts test setup (so that it is ethereum client agnostic)

I'm closing this issue.

[paulrberg]

(Not sure why this is necessary - maybe something about the way waffle and coverage plugins interact)

I know why! It's the "getWallets" function in this file from the @nomiclabs/buidler-waffle codebase:

https://github.com/nomiclabs/buidler/blob/59cc68e8f83fe36370e90e8270efac1665350798/packages/buidler-waffle/src/waffle-provider-adapter.ts#L14

Pasting it here:

if (this._buidlerNetwork.name !== "buidlerevm") {
  throw new Error(`This method only works with Buidler EVM.
You can use \`await bre.ethers.signers()\` in other networks.`);
}

So it's impossible to use solidity-coverage with waffle and the the buidler evm, because the former switches the network to "soliditycoverage" if the network is set to "buidlerevm":

https://github.com/sc-forks/solidity-coverage/blob/2b14380d3d65fb0e41e084f5f80b3fd3b435fdb3/plugins/resources/buidler.utils.js#L34

Now, I'm not sure whether adding a "coverage" network to the list makes it possible to use waffle, but I'd err on the side of caution and start using await ethers.signers() with a cast to Wallet instead of wallet.provider.getWallets, as @cgewecke showed above.

---

Just for the record, this is the error I was getting with waffle as the wallet provider in my tests:

Error: This method only works with Buidler EVM. You can use await bre.ethers.signers() in other networks.

[paulrberg]

I'd like to add yet another key point on top of the list made by @ChenchenYo above:

• Specify --temp artifacts

Without this, I am still getting zero coverage when trying to use [email protected] with [email protected], [email protected], [email protected] and [email protected].

[paulrberg]

@cgewecke, is there any way to avoid using --temp artifacts?

A side effect of doing that is that the artifacts folder gets deleted after coverage is run, which confuses Buidler's caching system.

Or maybe it's possible to avoid deleting this folder?

[alcuadrado]

A side effect of doing that is that the artifacts folder gets deleted after coverage is run, which confuses Buidler's caching system.

This must be a bug on our side. Can you elaborate on what happens?

[cgewecke]

@PaulRBerg

| is there any way to avoid using --temp artifacts

It's only required when there are hardcoded references to the artifacts folder somewhere in your project. If all the artifact loads defer to the env.paths.artifacts setting, it shouldn't be necessary.

[paulrberg]

Sure, @alcuadrado. I've actually assembled a repository so reproducing the bug is easy. Please see commit 84e3ec294c94de3a703336f40321c8d432979c20 in https://github.com/PaulRBerg/solidity-template and run these steps:

1. yarn install
2. yarn run build
3. yarn run coverage
4. yarn run compile

Buidler will erroneously complain that:

All contracts have already been compiled, skipping compilation.

In spite of a missing "artifacts" folder (deleted by the onIstanbulComplete hook from above).

This is the output when running ls artifacts:

ls: artifacts: No such file or directory

[paulrberg]

It's only required when there are hardcoded references to the artifacts folder somewhere in your project. If all the artifact loads defer to the env.paths.artifacts setting, it shouldn't be necessary.

@cgewecke, I see. Unfortunately, I'm not sure I can use buidler's config path to import the TypeChain artifacts in my tests. How could you do that for this code which is declared at the top of the file?

import GreeterArtifact from "../artifacts/Greeter.json";
import { Greeter } from "../typechain/Greeter";

[alcuadrado]

Sure, @alcuadrado. I've actually assembled a repository so reproducing the bug is easy. Please see commit 84e3ec294c94de3a703336f40321c8d432979c20 in https://github.com/PaulRBerg/solidity-template and run these steps:

1. yarn install
2. yarn run build
3. yarn run coverage
4. yarn run compile

Buidler will erroneously complain that:

All contracts have already been compiled, skipping compilation.

In spite of a missing "artifacts" folder (deleted by the onIstanbulComplete hook from above).

This is the output when running ls artifacts:

ls: artifacts: No such file or directory

Oh, wow. This is a huge bug. Thanks for putting up this reproduction repo.

[Alexangelj]

Just switched over to using buidler/ethers/waffle, have the same issue with 0 coverage.

        "@nomiclabs/buidler": "^1.3.2",
        "@nomiclabs/buidler-ethers": "^2.0.0",
        "@nomiclabs/buidler-waffle": "^2.0.0",
        "ethereum-waffle": "^3.0.1",
        "ethers": "^5.0.4",
        "solidity-coverage": "^0.7.0-beta.3",

[cgewecke]

@Alexangelj If you follow the recommendations in this comment above you should be able to generate a report.

[cgewecke]

Think this is pretty much resolved with the latest Hardhat version and solidity-coverage combination.

Everything should work without additional config or flags if you run:

npx hardhat typechain && npx hardhat coverage