Gitpod: [c/c++] Support clang formatting and add cmake

[svenefftinge]

Feedback from twitter:

• https://twitter.com/gedamore/status/1034902737857597440

try doing some level of work with my open source repo -- http://github.com/nanomsg/nng -- open any C file to see what I mean, and you have no chance to build this in the container whatsoever.

• https://twitter.com/gedamore/status/1034902482403487744

tried it... but no clang-format, no cmake, and no way to install either. No way that I can tell to tell it to understand include directories either, so most of my symbols are unknown. No, this isn't ready for "real" production C/C++ use yet.

[svenefftinge]

For the cmake addition : https://github.com/gitpod-io/workspace-images/issues/10

[svenefftinge]

https://github.com/gitpod-io/workspace-images/issues/10 is merged but not in production yet.

[simark]

Is there a single image use for all projects? It should be quite easy to include clang-format, but it would be irrelevant for projects other than C/C++ and would take a lot of unnecessary space.

[simark]

Is there a single image use for all projects? It should be quite easy to include clang-format, but it would be irrelevant for projects other than C/C++ and would take a lot of unnecessary space.

Never mind, I just tried gitpod with the nng project, and I see clang-format is there.

[svenefftinge]

Yes, the default image is a "one-size-fits-most". You can find it here
https://github.com/gitpod-io/workspace-images/blob/master/full/Dockerfile

Were you able to configure the project with include directories?

[simark]

I tried to configure the project with cmake and generate a compile_commands.json (this works, it's done through the terminal) directly in the source directory. Normally, clangd should pick it up without additional config. However, I stumbled upon the issue described in this PR:

https://github.com/theia-ide/theia/pull/2774

[jankeromnes]

This is fixed now, right?

There are still some improvements to be made to the C++ development experience (see #488), but cmake and clang-format are available in every Gitpod workspace, and Gitpod is already used by real-world C++ projects.

[aggsol]

Sadly, it is not the latest cmake version :-(

[jankeromnes]

Thanks for the feedback! Let's upgrade cmake then. 🙂

EDIT: Confirmed -- https://cmake.org/ says "CMake 3.15.1 available for download" but our workspaces have:

$ cmake --version
cmake version 3.12.1

[jankeromnes]

Unfortunately, 3.12.1 seems to be the latest available CMake version on Ubuntu Cosmic (which our workspaces are based on): https://launchpad.net/ubuntu/cosmic/+source/cmake

Nevertheless, maybe we can install a more recent CMake without apt-get, e.g. from source or via Linux Brew.

[jankeromnes]

I confirm that Linux Brew is able to install a recent cmake:

$ sudo apt-get remove -y cmake && brew install cmake

# ...

$ $(which cmake) --version
cmake version 3.15.1

However, there appears to be some broken reference to fix up:

$ cmake
bash: /usr/bin/cmake: No such file or directory

I'll try to make a PR to https://github.com/gitpod-io/workspace-images/ when I have some time (hoping for next week).

[meysholdt]

thank you @hungrybluedev for your blog post. I thoguth this issue may be related to what you wrote:

You’ll have the very decent Linux terminal at your disposal as well. You’ll have most of the build tools that you need included. Note that the version of CMake that was available in my case was just a few versions behind the latest release, so you may need to relax your minimum version requirements a little bit.

[jankeromnes]

Indeed, super cool blog post! 💯

Note that the version of CMake that was available in my case was just a few versions behind the latest release, so you may need to relax your minimum version requirements a little bit.

I believe that it's a bug when Gitpod doesn't have the latest stable version of a tool, and that developers shouldn't relax their minimum version requirements -- we should fix Gitpod instead.

Note that today, with Ubuntu Disco 19.04, we have a slightly more recent cmake:

$ cmake --version
cmake version 3.13.4

but I'll try to upgrade it to 3.15.3 anyway (which is currently the latest release).

[aggsol]

I believe that it's a bug when Gitpod doesn't have the latest stable version of a tool, and that developers shouldn't relax their minimum version requirements -- we should fix Gitpod instead.

I agree! With 3.14 support for Visual Studio 16 2019 was added, 3.15. added CMAKE_GENERATOR variable. Gitpod is a tool for developers, the developers dictate what is needed.

[hungrybluedev]

I'll update my blog post when this bug is fixed. Thanks for the support guys!

[jankeromnes]

Update: I've installed brew and used it to upgrade cmake in Gitpod's default Docker images here: https://github.com/gitpod-io/workspace-images/pull/112

$ cmake --version
cmake version 3.15.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

This was just merged, so CircleCI needs to rebuild all images, and these will then get picked up along with our next Gitpod production release (probably some time next week).

[jankeromnes]

Oh, I forgot to mention that the fix was successfully deployed recently. 🎉

This is now in Gitpod production:

$ cmake --version
cmake version 3.15.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

EDIT: And thanks to HomeBrew, it should remain up-to-date without extra efforts from our side. Thanks HomeBrew! 💯

[diegoferigo]

@jankeromnes You don't need to use HomeBrew to update CMake. Kitware provides an official CMake repository.

[jankeromnes]

@diegoferigo Thanks a lot for the tip! However, our images are based on Ubuntu 19.04 (Disco), which Kitware doesn't seem to support yet. From the page you linked:

We currently support Ubuntu 16.04 and Ubuntu 18.04 on our repository.

I did try the repo instructions with Disco, but it didn't (and still doesn't) work:

E: The repository 'https://apt.kitware.com/ubuntu disco Release' does not have a Release file.

[diegoferigo]

Did you try to use the bionic repo on disco? It might work.

Here below there is the list of dependencies of the current 3.15.4 version. The only problems I see is if on Disco those dependencies are not ABI compatible. In should work in most cases though.

❯ apt-cache show cmake=3.15.4-0kitware1
Package: cmake
Priority: optional
Section: devel
Installed-Size: 21422
Maintainer: Kitware Debian Maintainers <[email protected]>
Architecture: amd64
Version: 3.15.4-0kitware1
Depends: cmake-data (= 3.15.4-0kitware1), procps, libc6 (>= 2.16), libgcc1 (>= 1:3.0), libssl1.0.0 (>= 1.0.2~beta3), libstdc++6 (>= 5.2)
Recommends: gcc, make
Suggests: cmake-doc, ninja-build
Filename: pool/main/c/cmake/cmake_3.15.4-0kitware1_amd64.deb
Size: 3982058
MD5sum: cfa6d1620ea6bea00e38c3f70e8931cc
SHA1: ee814e7cd596360e8490e0e8760d275abb40850e
SHA256: d33747eeb69249407ab22eb0e97bd4de3ec383cbcf1f6fd49331debb6d912b3f
SHA512: d6ee24136b5c37b4348a9ad31f5144f71c016bd8cde8a17c0eb648c7c1cf4af830ce7b3571d628b0209dd6102d90cefc39b16522ef736e1c548b60548e19d150
Description: cross-platform, open-source make system
Description-md5: 47b53839da906127970f1e8c870afc2d
Multi-Arch: foreign
Homepage: https://cmake.org/

[jankeromnes]

@diegoferigo cool idea! I tried to build this proof-of-concept Dockerfile:

FROM ubuntu:disco

RUN apt-get update && \
    apt-get install -yq apt-transport-https ca-certificates gnupg software-properties-common wget && \
    wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | apt-key add - && \
    apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' && \
    apt-get update && \
    apt-get install -yq cmake && \
    cmake --version

Unfortunately, the build failed because of a libssl change:

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 cmake : Depends: libssl1.0.0 (>= 1.0.2~beta3) but it is not installable
         Recommends: gcc but it is not going to be installed
         Recommends: make
E: Unable to correct problems, you have held broken packages.

[diegoferigo]

Thanks for testing it, it was worth a try. I would not mess with dependencies for just one forward-ported package. I'm pretty sure that Kitware will support the upcoming 20.04 LTS. You can switch to the official repo (which is a cleaner solution wrt homebrew) after the LTS release.

For some while I also used the script provided by kitware, that offers a standalone installation. I found an old snippet I was using some time ago:

ARG CMAKE_VER=3.13.2
RUN cd /tmp &&\
    wget https://cmake.org/files/v${CMAKE_VER%.*}/cmake-${CMAKE_VER}-Linux-x86_64.sh &&\
    chmod +x cmake-${CMAKE_VER}-Linux-x86_64.sh &&\
    mkdir /opt/cmake &&\
    sh cmake-${CMAKE_VER}-Linux-x86_64.sh --prefix=/opt/cmake --skip-license &&\
    rm /tmp/cmake-${CMAKE_VER}-Linux-x86_64.sh
ENV PATH=/opt/cmake/bin:${PATH}

[jankeromnes]

Yes, we're planning to switch to 20.04 LTS when it's out and stabilized. Maybe this will give us an up-to-date CMake without HomeBrew.

Also, while I agree that using HomeBrew in our containers is not super "clean", it has a few key advantages for us:

• Their packages are generally super-well maintained (i.e. everything is latest stable, as opposed to Apt which usually lags behind)
• HomeBrew allows Gitpod users to install anything they want without requiring sudo, which is a huge plus (think running a quick brew install stuff vs. writing a Dockerfile, committing it, pushing it, starting a new workspace, etc.)

Also, while building from source is generally my second choice when the apt package is not available, it's not always practical. I remember trying it for CMake, but the build times were too long on CircleCI, which we rely on to iterate on our system images at https://github.com/gitpod-io/workspace-images/

TL;DR I definitely share your values, but wanted to add a bit of context on why we do some things a bit differently.

[diegoferigo]

Also, while I agree that using HomeBrew in our containers is not super "clean", it has a few key advantages for us: [...]

I definitely understand and agree with your reasons. In my experience, homebrew is quite good on average. Though, being a rolling release system, if you rely on it for many components of your systems, new releases of base libraries that break ABI compatibility cause big problems on the entire setup (and it happens more often than you might expect). This is unfortunately the price to pay.

Also, while building from source is generally my second choice when the apt package is not available, it's not always practical.

Btw the script I linked above does not compile from source if you were referring to that.

Yes, we're planning to switch to 20.04 LTS when it's out and stabilized. Maybe this will give us an up-to-date CMake without HomeBrew.

:+1:

[jankeromnes]

Oh, indeed, sorry I assumed that the script would build from source. Instead, https://cmake.org/files/v3.15/cmake-3.15.5-Linux-x86_64.sh seems to be a self-extracting script that includes a cmake binary as text? Looks like dark magic to me!

[diegoferigo]

It's just a short syntax (it could be even shorter) to download and execute an official bash script that downloads from kitware's servers and installs the selected binary version of CMake in a given prefix.

On a normal installation, I would not recommend to install files in the / that are untracked by the package manager, excluding edge cases. Though, in a docker image, this "dirty" operation becomes acceptable because typically software updates pass through an entire image rebuild rather than an update from within a container.