Conda-build: post build logic takes considerable time on osx

[jjhelmus]

Running the post build logic on macOS can take considerable time (over 2 hours). The slowdown seem to be predominately during detection of linkages.

For example from conda-forge/tensorflow-feedstock#80

2019-08-18T07:49:30.4994430Z New link location is None
2019-08-18T07:49:30.5192260Z 
2019-08-18T09:42:42.2436570Z    INFO (tensorflow-base,lib/python2.7/site-packages/tensorflow/contrib/tensor_forest/python/ops/_tensor_forest_ops.so): Needed DSO /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation found in the whitelist

[jjhelmus]

2291 discussed similar slow post build times in fixing linkages which was addressed in #2564

[jlehtniemi-broadsoft]

I confirm this performance degradation. In our case _fixing linking_ phase for Qt build can take over 24 hours with conda-build 3.18.9 when it takes only 20 minutes with conda-build 3.1.3.

[jlehtniemi-broadsoft]

Some additional benchmarks for Qt post-build using different conda-build versions:

| Versions | Time |
|--------------------|----------- |
|3.1.3 - 3.2.2 | 20 min |
|3.3.0 - 3.7.0 | > 2 hours |
|3.7.1 - 3.16.3 | 60 min |
|3.17.0 - 3.18.9 | > 2 hours |

To save some time, the test run was interrupted if it took more than 2 hours but from our previous experience we know that for the latest versions the post-build can take over 24 hours!

[mingwandroid]

Please conda install ripgrep in your base environment. That will considerably speed this up.

[forrestwaters]

It looks like this is still pretty slow even when ripgrep is installed in the base env. Building tensorflow 2.0 on Mac was stuck for more than 5 hours after a dozen or so lines of Fixing linking of.....

[mingwandroid]

Yeah, addressing this to the fullest extent possible (outside of not using LIEF) is in progress.

[mingwandroid]

Though actually, fixing linking of.... is done with Apple's tools at present!

[mingwandroid]

Ah, the lines after, ok, yes. That's in progress then.

[mingwandroid]

Can you point me to your latest recipe please @forrestwaters?

[forrestwaters]

Thanks Ray! Here it is: https://github.com/forrestwaters/tensorflow_recipes . Still pulling bazel 0.24.1 from my own fwaters channel for now

[mingwandroid]

Where does py_version come from here?

[forrestwaters]

Should be this: https://github.com/forrestwaters/tensorflow_recipes/blob/master/tensorflow/conda_build_config_osx_cpu.yaml#L22

[adam-urbanczyk]

I think I'm having a similar issue with the following recipe:
https://github.com/CadQuery/conda-packages/tree/master/pythonocc

I run it here:
https://dev.azure.com/cadquery/conda-packages/_build/latest?definitionId=1?branchName=master
and the post-build takes 5h+ (cannot tell exactly because the build times out at this point). With previous versions (I think around 3.14-3.16) the build was able to run till completion.

[adam-urbanczyk]

Installing ripgrep did not solve the issue (or at least did not improve it significantly enough - still getting a build timeout after 6h).

[adam-urbanczyk]

Using 3.16.3 (thanks @jlehtniemi-broadsoft ) is on MacOS a viable workaround for me - build time drops to ~ 1h.

[csosborn]

I have been running into this problem for a while now too. In my case the main culprit is this call to lief.parse() in inspect_linkages_lief, when the argument is libtensorflow.so. That one call took almost 21 minutes on my 2.4GHz i9, out of a build that's perhaps 25 minutes total.

[mgroth0]

Having the same issue trying to build tensorflow. I'm using python 3.8 which is too high for conda-build 3.16.3

edit: I created a new env with python3.7 and conda-build3.16.3. Conda-build was an order of magnitude faster (maybe 15-20 mins whereas it was over 12 hours with python3.8 and the more recent conda-build)

[mingwandroid]

12 hours is crazy-long!

Turns out this problem is exacerbated by conda-build's constant querying of statistics on memory, process and disk usage.

But the scanning itself is also in need of fixing. I apologise for not getting round to this. FIxing it is becoming critical for my workflow (I'm building Qt at present) so it should be fixed quite soon, I hope.

[mingwandroid]

Can pass try to --no-error-overlinking and --no-error-overdepending on the command line and see if that improves this. If not I can at least make sure it does and get that into the next build (Real Soon Now (tm)).

[mharradon]

Have there been improvements on this issue since? There have been a few releases and there's nothing especially obvious in the release logs.

Thanks!

[mingwandroid]

We updated lief things are better but not hugely so.

[mharradon]

OK - thank you for the update!