Zcash: ZCash build system improvements to leverage native tooling and other (native) assets where applicable

[veikkoeeva]

Context

This proposal stems from a short discussion on ZCash chat (with @str4d, who asked to submit an issue for further consideration) and from the lack of native way to both develop and consume Windows binaries of ZCash (but see https://github.com/zcash/zcash/issues/1989). Having a native Windows build would make it easier to participate on ZCash development as a developer, downstream developer, end-user and in other roles.

While having Windows binaries is the most immediate goal, there other qualities that could perhaps be realized. This proposal considers using CMake and other related tooling to achieve these goals. To stage the context, some reasons for a build system quoted from CMake build system: Distribute your software easily [pdf] by Cédric Castagnède.

What problems do build system solve?

For a developer:

• reduce the time spent in the cycle “edit / compile / test” cycle
• compile only what is necessary in the source code

For a development team:

• generate packages
• run tests
• generate documentation

For a user:

• install software easily
• have understandable error during install phase
• tune installation

Features of a build system

automatic dependency management of source code

• compile only the modified sources files and tiers dependencies

software portability

• use native build environment
• determine available OS/compiler features : foo.h, libbar, strndup, -Wall, etc.
• name correctly the library: .so / .dylib / .dll

adaptability according user environment

• auto-configuration of the project
• determine the availability and location of libraries, commands, etc…

customize installation

• cross-compiling
• give some information: --help
• possibility to set information: --prefix, --libdir, --disable-shared, etc.
• have some target: make all, make install…

launch tests

• without installation: link with generated library
• after an installation: link with installed library
• give a report of the build

Goals

1. To improve the current build system to leverage native Windows tooling. Currently Autotools requires Mingw and cross-compiling to create Windows binaries and even then it looks somewhat difficult.
   1.1 Improve in general other aspects of cross-platform reach and development tooling (see for exampe cmake -E server improves Visual Studio and Qt Creator IDEs and Kitware Increases Android Support in CMake 3.7)
2. Build and improve other aspects of the build system, as described in the Context section. It looks like there would be more cross-platform expertise and tooling available on CMake.
3. To make it easier for downstream developers to reuse assets, distribute them via Conan and on Microsoft platforms via Vcpkg, see difference to Nuget and Conand) and possibly Nuget.
   <edit:
4. Remove unneeded code. Is the Qt code needed?
5. Define and/or document clear interfaces for external library integrations.

Concerns and known issues

• Needing to maintain two build systems: an Autotools (and Mingw) based one and one on CMake.
• Maintaining compatibility with Bitcoin build system (e.g. refactoring project directory layout might make sense).
• Some deeper issues from a practical project have been enumerate in From Cmake Syntax To Libstdc++ Abi Incompatibiliy: Migrations Are Always Hard.
• Microsoft tooling or build systems aren't freely available and/or require Visual Studio installed. The necessary SDKs, compilers and other tooling are freely available at Visual C++ Build Tools Standalone compiler, libraries and scripts for instance (Visual Studio, Visual Studio Code etc. are freely available also). There are also freely available CI environments with Windows buil environments, see further in this proposal.
  <edit:- The LevelDb Windows build is not well defined. There are some unfofficial libraries shared, e.g., via Nuget. Maybe https://github.com/bitcoin/bitcoin/pull/6873 should be pulled and/or other other alternatives considered?

Proposed steps

1. Start with a Windows build since there is not one currently (but see but see https://github.com/zcash/zcash/issues/1989 as far as creating executable Windows binaries goes). A first pass implementation could be basic support using 64-bit Windows 10 using the latest C++ tool-chain leveraged by VS 2017, since these have native CMake support. There should not be problems to expand this to Windows 7 and older Microsoft provided tools later.
   1.1 The first pass Windows implementation should also lay the groundwork for cross-platform builds.
2. Add native Windows build to continuous integration. I don't know how ZCash does this currently, but Travis offers free CI system for open source projects (lacks Windows support currently), a similar services is Appveyor similarly offers free accounts for open source builds.
3. Add support to either Mac OS or Linux in the desired order to manage risks and engineering efforts. Looking at other online projects, blogs and sources of information, this mightn't be a lot of effort.
4. See and check other toolig used in other projects.

Further reading and resources

• One example a project layout: Basic but working C++ project using CMake, boost and Doxygen (note, Doxygen appears to have a poor support for C++11 and newer).
• How to use CMake to add Third Party Libraries to your Project.
• CMake and CTest at ASI.
• Enhancing Software Quality with CI in the Cloud.

<edit: 2017-01-07.

[radix42]

Is there an easy way to turn an issue into a ZIP, as that's what this feels like