Conda-forge.github.io: Use ccache

[gqmelo]

This issue is intended to discuss the use of ccache when building compiled packages and the findings I've got on a experimental branch.

Introduction

ccache is a compiler cache. It speeds up recompilation by caching previous compilations and detecting when the same compilation is being done again. Supported languages are C, C++, Objective-C and Objective-C++.

From my personal experience ccache is very reliable and after a couple of years using it at work we never found any issue. Additionally adopting ccache was a huge improvement.

Why

Using ccache on conda-forge could bring a lot of benefits:

• Faster feedback for mantainers of large packages. I personally find it very demotivating trying to make a large package build on CI.
• Possibly decrease CI's queue size
• Make it possible to build packages that exceed timeout limits

Obtained results

When experimenting with llvmdev-feedstock the results were very promising.

• On CircleCI a build with 99.92% cache hit took 15m as opposed to the usual 1h30m
• On CircleCI a first-time build took about 1h instead of the usual 1h30m. This happens because on CircleCI all package variations are built on the same job.
• On Travis a build with 99.92% cache hit took about 14m instead of the usual 44m (very close to the timeout limit).

Necessary changes

TravisCI

 language: generic
 os: osx
 osx_image: xcode6.4

+cache:
+  branch: no-md5deep
+  directories:
+  - $HOME/.ccache
+
+
 env:
   matrix:

The branch: no-md5deep is a workaround for https://github.com/travis-ci/travis-ci/issues/7456

CircleCI

   # just to pull each time. #rollondockercaching
   override:
     - docker pull condaforge/linux-anvil
+    - ./ci_support/run_docker_build.sh || touch ./build_failed
+  cache_directories:
+    - "./build_artefacts/.ccache"

 test:
   override:
     # Run, test and (if we have a BINSTAR_TOKEN) upload the distributions.
-    - ./ci_support/run_docker_build.sh
+    - if test -f ./build_failed; then exit 1; fi

CircleCI cache is only done for the dependencies phase. So to cache the compilation we need to move the build command. Also, the build command cannot fail during the dependencies phase, otherwise the cache will not be saved. That is why the verification is done later.

ccache setup

This would be added preferably to toolchain activate script:

if [ $(uname) == Linux ]; then
    export CC="ccache gcc"
    export CXX="ccache g++"
    export CCACHE_DIR=/feedstock_root/build_artefacts/.ccache
    # Set max cache size so we don't carry old objects for too long
    ccache -M 400M
else
    export CC="ccache clang"
    export CXX="ccache clang++"
    # Set max cache size so we don't carry old objects for too long
    ccache -M 200M
fi
export CCACHE_BASEDIR="${SRC_DIR}"
ccache -z

Note the ccache -M commands to limit the cache size. This is important because the cache is never automatically deleted by TravisCI and CircleCI. Also we would want to set different sizes for both CI's as CircleCI builds more than one package per job.

Also ccache needs to be added as dependency, preferably on toolchain package

Recipes

There would be no necessary changes on recipes whose build systems respect CC and CXX variables. However for large packages that could exceed the timeout limit it would be desirable to avoid the timeout killing the build command. On Linux this can be achieved with something like:

timeout 5400 make -j{CPU_COUNT}

Avoiding the timeout limits is important because otherwise the cache is not saved.

Windows

On Windows there is clcache which was inspired by ccache and so is very similar. The problem is that AppVeyor cache mechanism is different. The cache is only enabled for regular branchs, not PR's. Also it limits the cache size to 1GB per account, which in my opinion make the cache useless for a large organization as conda-forge.

References

• https://circleci.com/docs/1.0/how-cache-works/
• https://docs.travis-ci.com/user/caching/
• https://www.appveyor.com/docs/build-cache/

Pending

• [ ] Make it work when someone build locally without run_docker_build.sh script
• [ ] Use conda build --no-build-id and avoid setting CCACHE_BASEDIR

[gqmelo]

cc: @jakirkham
I preferred to create an issue instead of a PR as you suggested because adding this info to the docs would not be currently useful for mantainers.

[jakirkham]

Ok, that's fine. Thanks for writing this up.

[jakirkham]

Have you tried raising an issue with AppVeyor about caching on PRs? They have tended to be pretty responsive in the past. Would be curious to see if this is something they would want to support.

[gqmelo]

Have you tried raising an issue with AppVeyor about caching on PRs? They have tended to be pretty responsive in the past. Would be curious to see if this is something they would want to support.

Yeah, I've thought about that but I was unsure if the size limit imposed by AppVeyor would be useful. I didn't want to ask for a feature we wouldn't use. Do you think it's still worth requesting it?

[msarahan]

IMHO, it's worth requesting. It's pretty compelling to say "hey, look, if you give us more cache space, we'll use much less compute"

[isuruf]

Feedstock changes can be incorporated into conda-smithy using an entry in conda-forge.yml like use-ccache which would be no by default.
What happens when we try to build locally with CCACHE_DIR set to the above non-existing directory?

[gqmelo]

What happens when we try to build locally with CCACHE_DIR set to the above non-existing directory?

Good point! I've just tested this setting CCACHE_DIR to /ccache and the compilation fails with ccache: error: Failed to create directory /ccache/tmp: Permission denied. So we need to consider this situation.

[epruesse]

Good point! I've just tested this setting CCACHE_DIR to /ccache and the compilation fails with ccache: error: Failed to create directory /ccache/tmp: Permission denied. So we need to consider this situation.

So ccache tried, but failed, to create the missing directory because it was in a non-writeable location. Should work if dirname $CCACHE_DIR points to a writeable directory.

While you are on this: it'd be really neat if local building (e.g. ./scripts/run_docker_build.sh) was also using ccache.

[gqmelo]

Should work if dirname $CCACHE_DIR points to a writeable directory.

Certainly

While you are on this: it'd be really neat if local building (e.g. ./scripts/run_docker_build.sh) was also using ccache.

Actually running ./scripts/run_docker_build.sh locally will work. What won't work is building locally without the docker script, but we should make it should work, I just didn't realize this use case.

[gqmelo]

I've created an issue asking about AppVeyor cache on PR builds: https://github.com/appveyor/ci/issues/1553

[isuruf]

How about creating a ccache-toolchain package and then check for that using conda-smithy to change the templates?
Here's a branch that checks for ccache-toolchain and does the changes, https://github.com/isuruf/conda-smithy/tree/ccache

[gqmelo]

@isuruf cool!

But given my personal experience with ccache I would say it should just be used by default for all packages with compilation. I can't see any reason why not to use it as it's pretty safe and could be easily disabled by setting CC and CXX.

Searching conda-forge recipes you can see several recipes executing just make (without -j) or nmake (which don't support parallel compilation) and people don't realize that this is wasting time. I've seen even packages near CI's time out limit being compiled with one process.
So if lots of people forget to compile in parallel, I doubt many would remember to use ccache.

But if people want to do this more slowly I think your approach is reasonable. I just don't know how to make sure a ccache-toolchain package set the CC and CXX variables after the toolchain package as the order which activate scripts are executed is undefined.

[isuruf]

But given my personal experience with ccache I would say it should just be used by default for all packages with compilation. I can't see any reason why not to use it as it's pretty safe and could be easily disabled by setting CC and CXX.

Good luck convincing the toolchain team to get this in. (I'm +1 to this change, btw)

I just don't know how to make sure a ccache-toolchain package set the CC and CXX variables after the toolchain package as the order which activate scripts are executed is undefined.

We can modify toolchain to check if CC and CXX before setting them. Then the order doesn't matter.

[gqmelo]

Good luck convincing the toolchain team to get this in. (I'm +1 to this change, btw)

hehe, I know it's polemic but I think it's worth the discussion. :)

We can modify toolchain to check if CC and CXX before setting them. Then the order doesn't matter.

duh, yeah so obvious, thanks!

[isuruf]

Ping. I'd like to see this in sagelib-feedstock

[gqmelo]

So, I'm not sure yet what would be the acceptable approach to implement this. I would like more consensus before opening PR's on multiple repositories, otherwise the PR's will just add noise.

[isuruf]

I just tried with the scipy package with,

export CC="ccache $CC"
export CXX="ccache $CXX"

and it fails. https://circleci.com/gh/isuruf/scipy-feedstock/5

I think some build systems will not be able to handle CC set to two words. A bash script wrapping ccache gcc will have to be added to fix this.

[gqmelo]

Yeah, the scipy build system is doing something weird with the variable. It's weird that for it seems to handle it right for most commands.

So I think it's better to follow the symlink approach: https://ccache.samba.org/manual.html#_run_modes, then set the variables to the symlinks, or just add the symlinks to PATH (I would prefer to avoid that as it's more obscure)

[epruesse]

The PATH approach has worked most smoothly for me. It's transparent to whatever happens to the variables in the build system, essentially replacing gcc and friends with caching capable versions.

Plus, in terms of transparency, if I'm checking the environment, the first variable to look at is PATH anyway.

[gqmelo]

Yeah, probably adding to PATH should work fine.

[epruesse]

How do we go about getting this to work? Would the first stage be having a way to add the caching directories to Travis and Circle via conda-forge.yml in a feedstock? And then a package conda-forge-ccache-toolchain to set up the environment for building with ccache?

[gqmelo]

Well, as I commented on https://github.com/conda-forge/conda-forge.github.io/issues/389#issuecomment-301265229 I'm in favour of adding ccache support as default, making proper changes to toolchain and conda-smithy and provide an easy way to disable it.

[epruesse]

It would probably be good to start with a few packages before making it the default. There are things that won't build properly with ccache, and there is the issue of CCACHE_BASEDIR to work around. Anything using __FILE__ won't cache since the build dir contains a time stamp, so __FILE__ is different in every run (and that macro is often used in logging...). Using the CCACHE_BASEDIR variable however would shorten __FILE__ to a relative path, breaking the "have enough space for install path" concept of the long build path.

[gqmelo]

Using the CCACHE_BASEDIR variable however would shorten __FILE__ to a relative path, breaking the "have enough space for install path" concept of the long build path.

I think this would not be a problem. As you said normally __FILE__ is used for debugging or logging, so it would make much difference if it's relative. And actually a lot of build systems build the files with a relative path.

Also conda's relocation feature would not replace __FILE__. On source files this macro would contains a source tree path, which is not available on the final package anyway. On header files the macro won't be expanded when the header is installed to the prefix. So conda would not detect a hard coded path on either case.

But anyway there may be some weird library that may break with this. So I think it would be better to avoid using CCACHE_BASEDIR altogether and we can do this using conda build --no-build-id.

I agree with you it's better to test with a few packages first to figure out all this little issues. And then after we are satisfied make ccache default.

I think these would be the steps to implement this:

1 - Create a ccache-toolchain package
2 - Add an option to conda-forge.yml to render the recipe with ccache stuff
3 - Test with packages using a large range of build systems and tools (distutils, cython, cmake, autotools, etc.)
4 - After testing enough, make the changes to toolchain package and make ccache-toolchain a dummy package that just depends on toolchain

[isuruf]

@gqmelo, let me know how I can help. 2. is already done here, https://github.com/isuruf/conda-smithy/tree/ccache minus the --no-build-id.

[gqmelo]

@isuruf thanks! Would you like to submit the PR? Otherwise I will create one based on yours.

[isuruf]

@gqmelo, can you first create a recipe for ccache-toolchain? Then I'll send a PR to conda-smithy.

[isuruf]

ccache-toolchain recipe is here, https://github.com/conda-forge/staged-recipes/pull/3135

[gqmelo]

Thanks!

[ogrisel]

Reviving this discussion: what are the good practices to use ccache in conda-forge conda envs with the compilers package? I would be interested both a user on my laptop (I now set CC / CXX manually) and to avoid wasting compute time on CI environments, both on the CF infrastructure and for the upstream project CI.

@jakirkham proposed to archive the ccache-toolchain feedstock: https://github.com/conda-forge/ccache-toolchain-feedstock/issues/4 .

[jakirkham]

Would just try installing ccache and see if that works.

Yeah that feedstock precedes our current compiler setup. So not sure how it would behave.

[ogrisel]

It works but you have to set the CC / CXX env variables manually whenever you conda activate an env with the compilers package, no?

[jakirkham]

If that's true, then would suggest we raise an issue on the compilers feedstock about adding an additional package output (perhaps ccache-compiler?). That way it does the right additional activation step on top of what c-compiler and cxx-compiler do.

https://github.com/conda-forge/compilers-feedstock/issues