Intellij: Mac support: Bazel plugin doesn't work with C++ std lib and repositories

[gruszczy]

I have checked and bazel binary is set correctly in the settings.

include

Would not be detected properly by CLion. I have some repositories (both new_Http_repository and new_local_repository, for things like gtest, easylogging and llvm), neither is detected by CLion now.

Lots of warnings:

Warning:cc target //:type_inference_visitor_test does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:parser_test does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target @easylogging//:main does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:runtime does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:parser does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:classes does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target @llvm//:main does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:shwifty does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:common does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:ast does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target @gtest//:main does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:lexer does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:expression_type_visitor does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:codegen_test does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:type_inference_visitor does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:lexer_test does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:functions does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:jit does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:ast_compare_visitor does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:type does not depend on exactly 1 cc toolchain. Found 0 toolchains.
Warning:cc target //:codegen does not depend on exactly 1 cc toolchain. Found 0 toolchains.

Unfortunately there is no explanation about what this means and how to fix this. I selected the default toolchain.

Symbols get properly detected (both std lib and my local repositories) when I use hand crafted cmake, which I use specifically for CLion.

[matthias-kraft]

+1

[danieldanciu]

+1

[jvoung]

Is there an example project we could try loading?
What platform? (OSX, Linux, WIndows)?
What do some of the "intellij-info.txt" files look like? For example, with tensorflow, after bazel sync I have "bazel-bin/tensorflow/core/session_options.intellij-info.txt" for cc_library. That cc_library has a deps on target "@local_config_cc//:cc-compiler-k8", which is a cc_toolchain.

[danieldanciu]

Thanks for helping out, Jan!
I'm running on a Mac laptop (Sierra 10.12.4), don't know about the others.

I attached my *intellij-info.txt file. The cc_toolchain deps look like this:

"@local_config_cc//:cc-compiler-darwin_x86_64"

I'll attempt to repro this with a toy project and will update the issue if successful.

vmu_eval_synthetic_main.intellij-info.txt

[gruszczy]

I'm on a Mac laptop too.

[danieldanciu]

I managed to repro this with a trivial HelloWorld project, which I uploaded here:

https://github.com/danieldanciu/clion-bazel

Attached is the bazel-bin/hello.intellij-info.txt file.
hello.intellij-info.txt

Screenshot also attached.

[jvoung]

Thanks for the repro! I think the issue is that the plugin needs to be able to get toolchain info from "apple_cc_toolchain" rule kinds and not just "cc_toolchain" rule kinds.

[danieldanciu]

Thanks for looking into this, Jan. Do you have a feeling on how much work it is to get toolchain info from apple_cc_toolchain? Are there any workarounds we could use to make bazel/clion work on our macs?

[brendandouglas]

Separately to the apple_cc_toolchain issue, a quick note that we currently only support Linux.

We've heard anecdotally that the Bazel plugin already basically works on Macs, but it's not something we've tested. We do intend to support it in the future, but it's not currently a top priority.

[siddharthab]

+1 for having tested CLion integration on Mac.

[siddharthab]

I tried my hand at this, but because I don't have visibility into the true repo, I can not send an actual PR against the current code.

The combination of these two commits was enough to get the CLion plugin working on Mac with the default toolchain.
https://github.com/siddharthab/bazel/commit/e39ef01505fc14a016c437e232b7fb9c14121f1a
https://github.com/siddharthab/intellij/commit/b58e45d256a836908abea92aea4f9fc53553d89a

I understand that the aspect files have moved from the bazel repo to be shipped inside CLion, but my purpose here is to test the change in principal.

Note that a much simpler workaround is to provide a cc_toolchain (not apple_cc_toolchain) using crosstool_top build flag in your project view file (.bazelproject).

With the default toolchain, there are still some kinks. The includes and auto completes work fine, but some classes are not found inside the std namespace. See the attached screenshot.

[matthias-kraft]

Hi Siddharthab,

thanks a lot for looking into this. I have little experience using crosstool files, would you mind providing an example of how you define your cc_toolchain and include it in .bazelproject in this instance?

[jgavris]

+1 @siddharthab are you planning on contributing those changes to bazel / intellij plugin?

[siddharthab]

@jgavris I think this repo is a read-only mirror of their true repo, so I am not sure how to contribute directly, but the change is really small and should not interfere with anything else. So if the team is interested, they will be able to fix it soon.

@mkraft89 I will not be able to say how the cc_toolchain is defined as someone else in my company did it, and there are many specific customizations in it, but you can try to follow the guide from bazel team on how to define your own toolchain.

As for the .bazelproject file, you can access it for an existing project using 'Bazel > Local Project View File'. For a new project, you will be presented the option to create/edit the file as part of the configuration. Here's a minimal example:

directories:
  src/

targets:
  //src/...:all

build_flags:
  --crosstool_top=//custom_toolchain

[siddharthab]

Update on the namespace std missing issue when using the default toolchain with the suggested patch.

Looks like LLVM needs either __clang__ or __GNUC__ defined to process the std namespace correctly (example). CMake based CLion projects define both of these macros, but Bazel plugin does not set either of these. And hence, the versioned std namespace gets mapped to the global namespace in LLVM.

If I define either of these macros manually at the top of my file, CLion is able to parse the file correctly. Here's the same example from before, now working perfectly for the std::namespace.

So now we know the exact things to get CLion working on Mac, at least at first glance.

Cheers!

[Timmmm]

I was going to post a message asking for help from @siddharthab (CROSSTOOL isn't exactly easy to follow) but amazingly I actually got this to work, at least as far as I can tell.

The approach I took was to make a hello-world project, and then copy the toolchain files out of its build directory. Something like this:

mkdir foo
cd foo
touch WORKSPACE
mkdir main
echo "int main() {}" >> main/main.cpp
echo "cc_binary( name = "foo", src = ["main.cpp] )" >> main/BUILD

bazel build //main/...:all
cd bazel-foo/external/local_config_cc
ls

There are lots of useful files in there. Now create your actual project with a structure like this:

├── custom_toolchain
│  ├── BUILD
│  ├── cc_wrapper.sh
│  ├── CROSSTOOL
│  ├── dummy_toolchain.bzl
│  ├── libtool
│  ├── make_hashed_objlist.py
│  ├── osx_archs.bzl
│  ├── WORKSPACE
│  ├── wrapped_ar
│  ├── wrapped_clang
│  ├── wrapped_clang_pp
│  ├── xcode-locator-bin
│  └── xcrunwrapper.sh
├── main
│  ├── BUILD
│  └── foo.cpp
└── WORKSPACE

The main folder and WORKSPACE are as before. All of the contents of custom_toolchain are copied from local_config_cc except BUILD and CROSSTOOL. BUILD I based on the one from local_config_cc but simplified it a bit (don't care about iOS etc.).

I couldn't get the CROSSTOOL from local_config_cc to work at all - it always gives errors about %sdk_dir not being defined or something. Instead I extracted the Apple bits from this test CROSSTOOL in the bazel repo.

I also had to change osx_cc_wrapper.sh to cc_wrapper.sh (no idea why) and I added

unfiltered_cxx_flag: "-D__clang__"

to avoid the std:: issue mentioned above.

I think those were the only changes I made. BUILD and CROSSTOOL files attached. It seems to work but I presume there is a reason they added apple_cc_toolchain (and then didn't update all the places that assumed cc_toolchain, ugh.) so I guess this may have broken some Apple-related things. Who knows.

[Timmmm]

Ugh you can't attach text files apparently? BUILD:

package(default_visibility = ["//visibility:public"])

cc_toolchain_suite(
    name = "custom_toolchain",
    toolchains = {
        "darwin|compiler": ":cc-compiler-darwin_x86_64",
    }
)

cc_library(
    name = "malloc",
)

cc_library(
    name = "stl",
)

filegroup(
    name = "empty",
    srcs = [],
)

filegroup(
    name = "cc_wrapper",
    srcs = ["cc_wrapper.sh"],
)

filegroup(
    name = "osx_tools_darwin_x86_64",
    srcs = [
        ":cc_wrapper",
        ":libtool",
        ":make_hashed_objlist.py",
        ":wrapped_clang",
        ":wrapped_clang_pp",
        ":wrapped_ar",
        ":xcrunwrapper.sh",
    ],
)

cc_toolchain(
    name = "cc-compiler-darwin_x86_64",
    all_files = ":osx_tools_darwin_x86_64",
    compiler_files = ":osx_tools_darwin_x86_64",
    cpu = "darwin",
    dwp_files = ":empty",
    dynamic_runtime_libs = [":empty"],
    linker_files = ":osx_tools_darwin_x86_64",
    objcopy_files = ":empty",
    static_runtime_libs = [":empty"],
    strip_files = ":osx_tools_darwin_x86_64",
    supports_param_files = 0,
)

CROSSTOOL:

major_version: "local"
minor_version: ""
default_target_cpu: "same_as_host"

default_toolchain {
  cpu: "darwin"
  toolchain_identifier: "local_darwin"
}

toolchain {
  abi_version: "local"
  abi_libc_version: "local"
  builtin_sysroot: ""
  compiler: "compiler"
  host_system_name: "local"
  needsPic: true
  target_libc: "macosx"
  target_cpu: "darwin"
  target_system_name: "local"
  toolchain_identifier: "local_darwin"

  tool_path { name: "ar" path: "/usr/bin/libtool" }
  tool_path { name: "compat-ld" path: "/usr/bin/ld" }
  tool_path { name: "cpp" path: "/usr/bin/cpp" }
  tool_path { name: "dwp" path: "/usr/bin/dwp" }
  tool_path { name: "gcc" path: "cc_wrapper.sh" }
  cxx_flag: "-std=c++0x"
  linker_flag: "-lstdc++"
  linker_flag: "-undefined"
  linker_flag: "dynamic_lookup"
  linker_flag: "-headerpad_max_install_names"
  # TODO(ulfjack): This is wrong on so many levels. Figure out a way to auto-detect the proper
  # setting from the local compiler, and also how to make incremental builds correct.
  cxx_builtin_include_directory: "/"
  tool_path { name: "gcov" path: "/usr/bin/gcov" }
  tool_path { name: "ld" path: "/usr/bin/ld" }
  tool_path { name: "nm" path: "/usr/bin/nm" }
  tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
  objcopy_embed_flag: "-I"
  objcopy_embed_flag: "binary"
  tool_path { name: "objdump" path: "/usr/bin/objdump" }
  tool_path { name: "strip" path: "/usr/bin/strip" }

  unfiltered_cxx_flag: "-D__clang__"

  # Anticipated future default.
  unfiltered_cxx_flag: "-no-canonical-prefixes"

  # Make C++ compilation deterministic. Use linkstamping instead of these
  # compiler symbols.
  unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
  unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIME__=\"redacted\""

  # Security hardening on by default.
  # Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
  compiler_flag: "-D_FORTIFY_SOURCE=1"
  compiler_flag: "-fstack-protector"

  # Enable coloring even if there's no attached terminal. Bazel removes the
  # escape sequences if --nocolor is specified.
  compiler_flag: "-fcolor-diagnostics"

  # All warnings are enabled. Maybe enable -Werror as well?
  compiler_flag: "-Wall"
  # Enable a few more warnings that aren't part of -Wall.
  compiler_flag: "-Wthread-safety"
  compiler_flag: "-Wself-assign"

  # Keep stack frames for debugging, even in opt mode.
  compiler_flag: "-fno-omit-frame-pointer"

  # Anticipated future default.
  linker_flag: "-no-canonical-prefixes"

  compilation_mode_flags {
    mode: DBG
    # Enable debug symbols.
    compiler_flag: "-g"
  }
  compilation_mode_flags {
    mode: OPT
    # No debug symbols.
    # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or even generally?
    # However, that can't happen here, as it requires special handling in Bazel.
    compiler_flag: "-g0"

    # Conservative choice for -O
    # -O3 can increase binary size and even slow down the resulting binaries.
    # Profile first and / or use FDO if you need better performance than this.
    compiler_flag: "-O2"

    # Disable assertions
    compiler_flag: "-DNDEBUG"

    # Removal of unused code and data at link time (can this increase binary size in some cases?).
    compiler_flag: "-ffunction-sections"
    compiler_flag: "-fdata-sections"
  }
  linking_mode_flags { mode: DYNAMIC }
}

[matthias-kraft]

Hi @Timmmm thanks for sharing the CROSSTOOL and BUILD file, they worked for me and I can now happily run and build bazel projects with CLion (2017.2.3) on Mac (10.13).

Unfortunately, I cannot use the debugger yet. I encountered several errors trying to get it to work. First when clicking the debug symbol for a run configuration a small window with "unknown error" popped up. This was solved by adding a bazel before-run task in the run configuration as detailed here http://danieldanciu.blogspot.ch/2017/10/clion-with-bazel-unknown-error-when.html.

Then I received this "Unable to find Mach task port for process-id 57573: (os/kern) failure (0x5)".
(please check gdb is codesigned - see taskgated(8)). Which can be solved (or at least I think so) by following these steps https://gist.github.com/gravitylow/fb595186ce6068537a6e9da6d8b5b96d

After which I got an error message about NSUnbufferIO not being set followed by process error 255. I thought setting this variable in .bashrc to YES would help and indeed the error is gone.

Unfortunately, the debugger is still not working. Using lldb I click on debug and am left with process error 255 and nothing happening. Switching to the bundled gdb I get "During startup terminated with SIG113" and it suggests switching from gdb to lldb in the preferences. For gdb this is a known error and can be solved by adding "set startup-with-shell off" to ~/.gdbinit, but this seems to have no effect within CLion.

Does anyone have any ideas?

[Timmmm]

Good work @mkraft89. What was the variable you added to .bashrc?

According to this code exit code 255 happens when lldb-server gdbserver is run with no
extra arguments.

Although in my playing around with it, it actually prints the usage and calls exit(0). I might be being blind but I'm not sure how from the code. On the other hand it does not seem to be the highest quality code - just running this segfaults for me:

/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/lldb-server gdbserver --attach 123 foo

Maybe that is the startup-with-shell issue you mentioned? Maybe try running CLion under dtruss to see how it runs llvm-server?

[matthias-kraft]

@Timmmm. Thanks for helping out!
I added to the .bashrc export NSUnbufferedIO=YES. I will look into your suggestions, but it might be a few days before I get to it.

[RNabel]

With the most-recent published update and CLion 2017.3.3 this problem seems to be resolved on macOS 10.13.3 (for me at least).

[matthias-kraft]

@RNabel that's great to hear. Would you mind letting us know what exactly is working (running it or also debugging) and what steps you have taken to make it work?

Thanks a lot!

[RNabel]

@mkraft89 The setup consisted of Bazel syncing (top right corner B symbol), and then building (Cmd + F9). The project I'm working on is using a CROSSTOOL file already so I'm not sure whether it works without one. I also needed to create a symlink into bazel-genfiles to have generated protobuf code be available on the path and thus indexed, specifically because of -I directives in some rules.

Running and testing works no problem, debugging fails (with both LLDB and GDB) after confirming the prompt asking whether to allow the debug process to attach to another process. I have not tried disabling System Integrity Protection yet, which may fix this.

The exact error message is either:

UnbufferedIO is not set, output may be delayed

During startup program terminated with signal SIGTRAP, Trace/breakpoint trap.

macOS 10.12 has known issues running GDB due to System Integrity Protection.
Please switch the debugger to LLDB in Preferences.

OR:

...
During startup program terminated with signal ?, Unknown signal.
...

[matthias-kraft]

@RNabel thanks for letting us know! The error you describe is exactly what I encountered when I tried. I wasn't able to fix it. Let us know if you manage to (in case you are trying).

[jgavris]

Are there any efforts to getting this working on a mac without having to define a custom toolchain?

https://docs.bazel.build/versions/master/bazel-vision.html

Engineers can take build fundamentals for granted. Software developers focus on the creative process of authoring code because the mechanical process of build and test is solved. When customizing the build system to support new languages or unique organizational needs, users focus on the aspects of extensibility that are unique to their use case, without having to reinvent the basic plumbing.

The rules (and Bazel) need to interface with a widely-used IDE for the language, if there is one.

[brendandouglas]

We're not currently looking at this. Mac support for CLion is something we'd like to add, but isn't a particularly high priority.

[tomergafner]

Setting up BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 worked for me
Meaning running from the command line:

export BAZEL_USE_CPP_ONLY_TOOLCHAIN=1

before starting CLion and then opening CLion from the command line:

open /Applications/CLion.app

@jgavris

[jgavris]

@tomergafner that's very cool for CLion, I see that it actually fixes the issue, but there is no 'python console' or 'environment variables' in Android Studio as far as I am aware of, so that is not a viable solution for me.

[prinsun]

@tomergafner

File indexing OK, but got lot's of building errors.

Error:(70, 1) Couldn't build file external/com_google_protobuf/_objs/protobuf_lite/extension_set.o: C++ compilation of rule '@com_google_protobuf//:protobuf_lite' failed (Exit 1).

[Traeyee]

2020 is coming, has this problem been solved?

[immars]

+1 for this issue.

[immars]

now my IDE is able to find std::string, but not std::time_t.

Anyone encountered the same issue?