Drake: Identify Supported Configurations

[david-german-tri]

Drake should have a well-defined list of supported configurations, where a configuration is an operating system, a set of installed dependencies, and a set of make options selections. A supported configuration should be reproducible in CI and by developers anywhere.

Once the supported configurations are known, we can run CI for exactly those configurations, and establish a policy that PRs must not regress the supported configurations. Anything in the combinatorial explosion of unsupported configurations would have no such guarantees.

[liangfok]

Here is ROS' similar effort: http://www.ros.org/reps/rep-0003.html.

[sherm1]

Here is ROS' similar effort: http://www.ros.org/reps/rep-0003.html

Wow -- that's really nice. OSRF does a great job with documentation.

[jamiesnape]

For the record the CI currently runs:

• Ubuntu 14.04 LTS / Clang 3.7
• Ubuntu 14.04 LTS / GCC 4.9
• Windows Server 2012 R2 / MSVC 14 (32-bit and 64-bit builds)

OS X builds will be:

• OS X 10.10 / Apple Clang 7.0

Jenkins MATLAB builds use version R2015b. Java is version 7 or 8 on Jenkins. I cannot support less than Java 7 on Jenkins builds for technical and security reasons. I can potentially set up CI builds for RHEL 7.2 / SUSE 11 or 12 / Windows Server 2003 or 2008. I cannot support OS X 10.11 on Jenkins at the moment for technical reasons. Other operating systems, if possible, require more work.

[liangfok]

Maybe we should start providing & maintaining virtual machine images of the officially supported platforms allowing developers to test their changes cross-platform prior to relying on the CI infrastructure to root out cross-platform issues. I'm not sure this is even legal if the V.M. contains licensed software like OS X, Windows, SNOPT, etc. Thoughts?

[jamiesnape]

I am working on Chef scripts for the purposes of provisioning of CI, so that would be that easy for non-licensed software.

[jwnimmer-tri]

Also I've said off-list, but moving it here: in addition to the supported platform-and-compiler set, we also should enumerate the supported "make options" settings.

I recommend that we support:

• 5 compilers
  
  
  
  • Ubuntu 14.04 LTS / Clang 3.7
  
  
  • Ubuntu 14.04 LTS / GCC 4.9
  
  
  • Windows Server 2012 R2 / MSVC 14 / 32-bit
  
  
  • Windows Server 2012 R2 / MSVC 14 / 64-bit builds
  
  
  • OS X 10.10 / Apple Clang 7.0
  
  
• 2 optimization settings
  
  
  
  • debug
  
  
  • release
  
  
• 4 sets of settings for which externals / optional dependencies are enabled:
  
  
  
  • none
  
  
  • all open-source
  
  
  • all open-source + matlab
  
  
  • everything
  
  

So 5x2x4 = 40 configurations in total.

Other permutations of the "make options" settings would be on a best-effort basis only.

[jamiesnape]

We have these covered by nightly builds at moment.

Open Source:

• Ubuntu 14.04 LTS / Clang 3.7 (Debug, Release)
• Ubuntu 14.04 LTS / GCC 4.9 (Debug, Release)
• Windows Server 2012 R2 / MSVC 14 / 32-bit (Debug, Release)
• Windows Server 2012 R2 / MSVC 14 / 64-bit (Debug, Release)
• OS X 10.10 / Apple Clang 7.0 (Debug, Release)

Open Source + MATLAB:

• Ubuntu 14.04 LTS / GCC 4.9 (Release)

[david-german-tri]

@jamiesnape Just to be clear, that's with default settings only, right?

@jwnimmer-tri: To align with what works in CI right now, I'd propose a smaller list of optional dependency configs, at least initially:

• all dependencies that are enabled by default
• all dependencies that are enabled by default, plus MATLAB (release only)

I agree it would be good to add "everything" if we can sort out the licensing. I'm not sure I see a use case for "none", or for "all open-source" as distinguished from the default config.

I'll start drawing up a documentation PR.

[jamiesnape]

Yes. It may be a long time before we can do 39 (MATLAB / Clang on Linux is unsupported) or more builds nightly, or especially on pull requests.

[jamiesnape]

I agree with @david-german-tri's list. We will be working on the dependencies that live in private repositories or are otherwise licensed.

[jwnimmer-tri]

@david-german-tri I am happy to draw a distinction between _"currently-supported-by-CI"_ configurations and _"want to be supported, but not yet tested under CI"_ configurations. My proposal included both sets, but treating them separately in the interim makes sense.

I'm not sure I see a use case for "none", ...

I have two answers to this:

1. The users who want Drake-as-a-library should have the opportunity to disable Drake's _optional_ dependencies when they don't want them.
2. If make offers an option to turn a thing off, but CI never tests it, what is the point of offering such an option? It will ~never work.

...or for "all open-source" as distinguished from the default config.

I guess I'm not sure what your "default config" means. The point of my "all open-source" proposal is to get maximal CI coverage of optional dependencies, under the proviso that no specially-licensed libraries need to be used. (This represents (in CI) the use case of the many users who want to use Drake to its fullest, but don't have licenses to anything.)

@jamiesnape Good point, you're right that not all of the 40 options will make sense, e.g., matlab-clang is nixed. So the set will be smaller than 40.

In terms of CI, I was only proposing to define what we promise to support -- the question of how to cover this in CI is a separate issue, and need not have so many options tested at once (if we design it carefully). In any case, CI hardware is cheap, and even 40 build variants is well within the capability of current technologies.

[jamiesnape]

CI hardware is cheap

MATLAB licenses and OS X hardware are not, and we have limited IP addresses on the OS X side.

[jwnimmer-tri]

It is difficult for me to believe that OS X hardware cost comes anywhere near the fully-loaded cost of dozens of developers (a quick google suggests less than $3k per dedicated instance per year), but whatever. If OS X in the cloud is too expensive, we can reduce the OS X matrix accordingly, perhaps down from the 8 variants to just 2, such as OSX-Debug-None and OSX-Release-All.

On the other hand, Matlab really _is_ quite expensive (on the order of 10% of a loaded FTE), but I hope MathWorks comes through with reasonable discount pricing for the Drake CI.