Truffle: Truffle 5 always compiles all contracts before testing

[barakman]

Issue

The contracts_build_directory compiler option appears to not being handled properly on Windows.

Running truffle test always ends up converting my contracts build directory (where Truffle creates all JSON artifacts) from './<my_project_folder>/build/contracts' (which is also the default BTW) to some temporary Windows directory, for example, 'C:\\Users\\LOCALC~1\\AppData\\Local\\Temp\\test--12372-H5iUC1JGbek1' (the suffix changes on every execution).

This, in turn, causes a full rebuild upon every execution (i.e., truffle compile --all is always executed automatically before the actual test begins).

I have added some printouts in your code (file cli.bundled.js), and found out that the problem may be stemming from when function prepareConfigAndRunTests is called with this temporary directory.

I believe that the problem might be here:

      Environment.detect(config)
        .then(() => copyArtifactsToTempDir(config))
        .then(({ config, temporaryDirectory }) => {
          return prepareConfigAndRunTests({
            config,
            files,
            temporaryDirectory
          });
        })

And since I know that this problem does not occur on neither Linux nor MacOS, I believe that it possibly stems from Environment.detect.

What I don't get is, when printing all Truffle configurable directories (contracts_directory, build_directory, test_directory, contracts_build_directory, etc) in your code, why the only one which is replaced with a temporary directory is contracts_build_directory.

Any help would be very much appreciated, as I am currently left to wait on compilation before every test.

Thanks

Environment

• Operating System: Windows 10
• Truffle version (truffle version): 5.1.36
• node version (node --version): 10.16.0
• npm version (npm --version): 6.9.0

[gnidan]

Hm I just started noticing that behavior, also! Thanks for reporting this issue, we'll look into it promptly!

[gnidan]

(For the record, I'm able to reproduce this on macOS - on Truffle v5.1.41, I do truffle compile followed by another truffle compile; all contracts compile both times)

[elenadimitrova]

Confirming this behaviour on macOS as well. https://github.com/trufflesuite/truffle/issues/3277 could be related.

[gnidan]

Thanks for the added context @elenadimitrova, we will look into this!

[barakman]

@gnidan : Any progress on this, or an ETA on a fix?

[fainashalts]

Hi @barakman sorry for the delay on this! We're going to look into this asap.

[eggplantzzz]

So it looks like this bug is caused by the way that Truffle compiles during testing. I think when it compiles, it only saves the artifacts to the temporary directory that it creates for the test run. This leaves the "real" project build directory with artifacts that are out of date. It seems like Truffle should instead check to see if a compilation is necessary before it creates the temporary directory and copies the artifacts.

I would assume that if you do a regular truffle compile before you test, it will not re-compile those contracts.

I think we should be able to have a patch out for this by the end of next week. Thanks again for reporting this @barakman!

[barakman]

@eggplantzzz:

I would assume that if you do a regular truffle compile before you test, it will not re-compile those contracts.

No. As I stated explicitly:

truffle compile --all is always executed automatically before the actual test begins.

Which implies that it is executed even if I execute it manually beforehand (i.e., before executing truffle test).

This did not use to be the case on Truffle 4, and I've been experiencing that only since I upgraded to Truffle 5.

I'm pretty sure that this problem is relevant only on Windows, but other commentators here have mentioned it for other operating systems as well.

Thanks

[eggplantzzz]

@barakman Why do you say truffle compile --all is run? Internally, Truffle does not do that and only looks for sources that have been modified more recently compared with the built artifacts compilation time. If you think I'm wrong and know where it happens in the code, could you reference it here? It could do that, however, if you pass the --all flag somehow.

On mac, the behavior I can replicate is the recompilation of contracts that were out of date before running truffle test. If I modify a source, it will recompile EVERY time I run truffle test after that because the built artifact is not getting updated. This is because truffle test uses a temporary directory, recompiles modified sources, and saves them to the temporary directory used for testing. It never saves to the normal build directory in this flow. Every time you run truffle test again, it once more copies your out-of-date artifacts over to the temporary directory which causes it to recompile again.

The solution I worked out in #3438 was to compile artifacts before copying the built artifacts to the temporary directory. This will cause the copied artifacts to always be current with your source files.

[barakman]

@eggplantzzz: Thank you for your quick response.

I did not say "truffle compile --all is run" (I'm not even sure what this statement means grammatically).

As I explained in my opening post in this thread, truffle compile --all is always executed automatically before the actual test begins.

In other words, executing truffle test always triggers truffle compile internally (even if preceded by a manual execution of truffle compile).

Please read the full description in that post, at the top of this page; it contains all the technical details that I have managed to collect.

If I understand you correctly, then you're suggesting for me to copy the artifacts into the temporary directory, is that correct?
If yes, then:

1. How am I supposed to know the path of this directory?
2. Isn't it going to be a different path every time I run truffle test?

If no, then can you please rephrase your suggestion?

Also, please note that it seem a bit odd for me to deal with this problem, which was not present prior to Truffle 5.x.
I'm suspecting that it is indeed relevant only for Windows; would you be able to check this conjecture somehow?

Thank you for your help :)

[eggplantzzz]

When I said that you wrote truffle compile --all is run, I meant truffle compile --all is executed. Are you saying that your script executes it?

There is one mandatory compilation before testing: you need to at least compile the Solidity tests that you have (if you have any).

No I wasn't suggesting that you copy your artifacts, there is just a detail in the implementation of the testing infrastructure which causes truffle test to not update the artifacts during the testing compilation. So if you only ever run truffle test, all of your contracts will get compiled every run because they are never actually updated. My PR listed above should fix this problem.

Perhaps you are experiencing a different issue though. I'll see if I can find a colleague to test this for me that has a Windows machine. So just to confirm; what happens if you run truffle compile --all in your project directory and then run truffle test? Does it compile your artifacts during truffle test or just your Solidity tests?

[barakman]

@eggplantzzz:

Are you saying that your script executes it?

No, I am saying that every time I run truffle test, I see that it first compiles all contracts. It is printed in the console for a relatively long period of time. The same happens even if I manually run truffle compile beforehand.

you need to at least compile the Solidity tests that you have (if you have any)

I honestly have no idea what you mean by 'Solidity tests'. All my Truffle tests are pure JS, and there is obviously no need to compile those. Perhaps you meant the Truffle on-chain infrastructure, in that case - no, we are not using this infrastructure at all.

what happens if you run truffle compile --all in your project directory and then run truffle test? Does it compile your artifacts during truffle test or just your Solidity tests?

It compiles all of my contracts (and I don't have any Solidity tests).

Thanks again :)

[eggplantzzz]

Truffle allows you to write tests for your contracts in Solidity. These are the tests that I'm referring to. If you don't have any, then it shouldn't need that step.

I can't duplicate the behavior you mention but, like I said, I'll find a colleague with a Windows machine to see if we can reproduce your error on it. Thanks!

[akcgjc007]

I am writing all of my tests in test/test.js file. Now, If I make some changes in my test script and run the truffle test command, it is again compiling everything(which is taking some time).

My first truffle experience was not so "sweet", so I am right now doing the compilation and testing manually with my scripts.
Hope truffle will remove this bug.
I am using ubuntu 20.04.1:

Truffle v5.1.59 (core: 5.1.59)
Solidity v0.5.16 (solc-js)
Node v10.19.0
Web3.js v1.2.9