Rust-bindgen: bindgen is slow in a particular project

[caracal7]

Example project
https://github.com/caracal7/bindgentest

But for example if I make binding for subfolder uSockets everything is working as expected

OSX 10.13.6
Rust 1.39.0-nightly

[emilio]

It seems bindgen is being pretty slow, but over-all seems to be making forward progress... Do you also see forward progress if you run it like:

$ RUST_LOG=bindgen /path/to/rust-bindgen/target/debug/bindgen uWebSockets/src/App.h -- -I./uWebSockets/uSockets/src --std=c++17 -x c++

In general whitelisting the items you want may improve performance since locally it seems to spend a whole lot of time on the std lib.

But I guess gotta profile it and see what makes it blow up.

[caracal7]

I was leave bindgen working all the night without success

[emilio]

Then it may be because of different system headers, for me it finishes... Is there any chance you could create a reduced test-case as described in https://github.com/rust-lang/rust-bindgen/blob/master/CONTRIBUTING.md#using-creduce-to-minimize-test-cases?

[caracal7]

Thx for response.
I don’t know c++ and binding process is something magical for me and I only started to learn Rust. And at moment this looks very complicated.
My “super skill” is JS🤗
First I will try run binding process on other system inside docker container or virtualbox .

[emilio]

Sure, that's fine! Thanks for reporting this btw! :)

[RandomInsano]

I hope you don't mind hijacking this thread as I'm having a problem with the same general description. I'm building rust.ko on a Raspberry Pi 3 which is building one gigantic bindings.rs file out of all of the kernel headers.

I believe one run through it ran out of RAM/swap so I've increased that. It'll probably be chewing on this all week.

Are there thoughts or practices on how the process could be made re-entrant so it can pick up where it left off? Maybe having it build one rs file per header file?

[emilio]

Hmmm, that is a bit surprising... I assume the problematic bits are in https://github.com/tsgates/rust.ko/blob/f08d8cc8f742ec3dacb73f7ac65dda2d0aeb5531/std/build.rs, right?

That doesn't seem like an insane amount of headers... Though I haven't double-checked how big is the whole translation unit.

Splitting up the work in multiple headers may work, but you may end up doing more work over all of course... And you need to either de-duplicate the types that are included in both or what not, which seems a bit annoying.

In general I'm a bit surprised to hear that bindgen is choking on that given the amount of code that it has to deal with in Firefox...

One thing that would be useful to know is when in the pipeline is bindgen consuming a lot of memory / hanging...

Bindgen is split in two main phases, one taking the clang AST as an input and generating its own internal representation, the second turning that representation into code.

So I'd be curious in, for example, whether it gets to the code generation phase, and if so in which one it ends up, and so on. Maybe we get all the way to the end of that function and fail to turn everything we've computed into rust tokens / text?

Depending on that, there may be multiple solutions to the problem, like incrementally write out the output rather than keeping it in memory...

Or we may not get to the point where we even start generating the code, and maybe it's libclang what's choking on the input (I've fixed a few libclang hangs in the past, but they mostly dealt with bizarre C++).

But I'd say that there's no easy answer there, I need to know first the root cause of the problem you're dealing with in order to suggest a solution that works :)

[inferiorhumanorgans]

I can't say if it's related but I've run into a similar situation with bindgen hanging. I've traced it back to where the C++ project includes STL headers. Memory usage appears stable, one CPU core is pegged for the duration, and no progress appears to be made.

$ uname -vm
FreeBSD 12.1-RELEASE-p8 GENERIC  amd64
$ pkg info -E llvm*
llvm-90
llvm90-9.0.1_2
$ ~/rust-bindgen/target/debug/bindgen --version
bindgen 0.55.1
$ (cd ~/rust-bindgen && git rev-parse HEAD)
4608a11b5cbdcc779bd78c0e14735864a95ed893
$ cat wrapper.h
#include <vector>
$ ~/rust-bindgen/target/debug/bindgen --verbose --enable-cxx-namespaces --opaque-type 'std' wrapper.h -- -I/usr/include/c++/v1 -I/usr/local/include  -x c++ -std=c++14

🦗🦗🦗

[emilio]

Huh, that command takes about 2s for me which for a debug build is probably not terrible. @inferiorhumanorgans can you attach to the bindgen process using gdb and take a look to see how we're hanging? Can you use --dump-preprocessed-input so that we get the whole input that hangs on your system?

[inferiorhumanorgans]

@emilio Still no output

If I call bindgen directly on /usr/include/c++/v1/vector I get:

/usr/include/c++/v1/vector:291:13: warning: #pragma system_header ignored in main file [-Wpragma-system-header-outside-header], err: false

[emilio]

@inferiorhumanorgans can you attach gdb and see what is bindgen doing?

[inferiorhumanorgans]

So I'm on FreeBSD which uses clang 8.0.1 as the system compiler but does not ship with any C++ based libraries (so no libclang). I've installed clang 9.0 (which appears to use the stock c++ runtime) to build/run bindgen. Running lldb 9.0 gives the same output, the only difference being some color highlighting.

# lldb -v
lldb version 8.0.1
  clang revision 366581
# lldb -p 98051 bt
(lldb) process attach --pid 98051
^C
... Interrupted.
(lldb) bt
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #1, name = 'bindgen'
  * frame #0: 0x00000000015b3a0c bindgen`core::option::Option$LT$T$GT$::map::h6044446456cd79ac(self=Option<&bindgen::ir::ty::Type> @ 0x00007ffffffe6970, f=closure-0 @ 0x00007ffffffe6990) at option.rs:456:5
    frame #1: 0x00000000016a7dc5 bindgen`bindgen::ir::context::ItemResolver::resolve::h376a9e291964a4d0(self=<unavailable>, ctx=0x00007ffffffe9fb8) at context.rs:2655:27
    frame #2: 0x000000000157f67f bindgen`_$LT$bindgen..ir..ty..Type$u20$as$u20$bindgen..codegen..CodeGenerator$GT$::codegen::h0278f130eb999443(self=0x000000080b249308, ctx=0x00007ffffffe9fb8, result=0x00007ffffffe8598, item=0x000000080b249258) at mod.rs:768:48
    frame #3: 0x000000000159e78b bindgen`_$LT$bindgen..ir..item..Item$u20$as$u20$bindgen..codegen..CodeGenerator$GT$::codegen::h8030d9db2bd65f05(self=0x000000080b249258, ctx=0x00007ffffffe9fb8, result=0x00007ffffffe8598, _extra=0x000000000113e740) at mod.rs:468:17
    frame #4: 0x000000000165892b bindgen`_$LT$bindgen..ir..module..Module$u20$as$u20$bindgen..codegen..CodeGenerator$GT$::codegen::_$u7b$$u7b$closure$u7d$$u7d$::hbbd679bf7d53c7f1(result=0x00007ffffffe8598, found_any=0x00007ffffffe87ef) at mod.rs:490:21
    frame #5: 0x0000000001658e52 bindgen`_$LT$bindgen..ir..module..Module$u20$as$u20$bindgen..codegen..CodeGenerator$GT$::codegen::_$u7b$$u7b$closure$u7d$$u7d$::h86ada8fdffe218ba(result=0x00007ffffffe8598) at mod.rs:538:13
    frame #6: 0x00000000016585bb bindgen`bindgen::codegen::CodegenResult::inner::h41a8feff25f0d64c(self=0x00007ffffffe9148, cb=<unavailable>) at mod.rs:310:9
    frame #7: 0x00000000015a84b8 bindgen`_$LT$bindgen..ir..module..Module$u20$as$u20$bindgen..codegen..CodeGenerator$GT$::codegen::h5a9619d2696fe0fc(self=0x000000080a513df0, ctx=0x00007ffffffe9fb8, result=0x00007ffffffe9148, item=0x000000080a513d40) at mod.rs:525:27
    frame #8: 0x000000000159e7c1 bindgen`_$LT$bindgen..ir..item..Item$u20$as$u20$bindgen..codegen..CodeGenerator$GT$::codegen::h8030d9db2bd65f05(self=0x000000080a513d40, ctx=0x00007ffffffe9fb8, result=0x00007ffffffe9148, _extra=0x000000000113e740) at mod.rs:459:17
    frame #9: 0x000000000165b5e0 bindgen`bindgen::codegen::codegen::_$u7b$$u7b$closure$u7d$$u7d$::h1cca33ce69394c33((null)=closure-0 @ 0x00007ffffffe9438, context=0x00007ffffffe9fb8) at mod.rs:4026:9
    frame #10: 0x0000000001606089 bindgen`bindgen::ir::context::BindgenContext::gen::hb6b38740cdf31ff7(self=<unavailable>, cb=closure-0 @ 0x00007ffffffe9f78) at context.rs:1159:19
    frame #11: 0x0000000001696e3d bindgen`bindgen::codegen::codegen::he6dad2444fe6c93a(context=<unavailable>) at mod.rs:4000:5
    frame #12: 0x000000000160ee39 bindgen`bindgen::Bindings::generate::h88d425850f148c21(options=<unavailable>) at lib.rs:2120:32
    frame #13: 0x000000000160b08b bindgen`bindgen::Builder::generate::h7a7e207828120887(self=<unavailable>) at lib.rs:1335:9
    frame #14: 0x0000000001372be4 bindgen`bindgen::main::_$u7b$$u7b$closure$u7d$$u7d$::h6d9e371521d81f10 at main.rs:54:17
    frame #15: 0x0000000001386cfd bindgen`std::panicking::try::do_call::h56807e070c5cabb4(data="\b") at panicking.rs:348:40
    frame #16: 0x0000000001386e1d bindgen`__rust_try + 29
    frame #17: 0x0000000001386b7d bindgen`std::panicking::try::h1c9958c234e3660c(f=<unavailable>) at panicking.rs:325:15
    frame #18: 0x000000000137225d bindgen`std::panic::catch_unwind::h11525143d2d7a639(f=<unavailable>) at panic.rs:394:14
    frame #19: 0x0000000001389a77 bindgen`bindgen::main::h7862d4316b86c68d at main.rs:53:34
    frame #20: 0x0000000001375d9e bindgen`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h413cbc60c9561786 at rt.rs:67:34
    frame #21: 0x0000000001a232ea bindgen`std::rt::lang_start_internal::ha90c4c8e282f65c1 [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::hc0f35a74710e4a19 at rt.rs:52:13
    frame #22: 0x0000000001a232e2 bindgen`std::rt::lang_start_internal::ha90c4c8e282f65c1 [inlined] std::panicking::try::do_call::hf3de4c4c1dce977f at panicking.rs:348
    frame #23: 0x0000000001a232e2 bindgen`std::rt::lang_start_internal::ha90c4c8e282f65c1 [inlined] std::panicking::try::h8b2e697ba94a3253 at panicking.rs:325
    frame #24: 0x0000000001a232e2 bindgen`std::rt::lang_start_internal::ha90c4c8e282f65c1 [inlined] std::panic::catch_unwind::hc4dd4672f0cc4abb at panic.rs:394
    frame #25: 0x0000000001a232e2 bindgen`std::rt::lang_start_internal::ha90c4c8e282f65c1 at rt.rs:51
    frame #26: 0x0000000001375d82 bindgen`std::rt::lang_start::ha2bcbf320dd7bebf(main=(bindgen`bindgen::main::h7862d4316b86c68d at main.rs:44), argc=13, argv=0x00007fffffffe5f8) at rt.rs:67:5
    frame #27: 0x0000000001389dab bindgen`main + 43
    frame #28: 0x000000000136f10b bindgen`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7
(lldb)

Whoops, I missed setting RUST_LOG. With that I get plenty of output but it's the top bit that's most suspicious:

[2020-09-01T08:51:37Z INFO  bindgen] Clang Version: clang version 9.0.1 , parsed: Some((9, 0))
[2020-09-01T08:51:37Z DEBUG bindgen] Generating bindings, libclang at /usr/local/llvm90/lib/libclang.so.9
[2020-09-01T08:51:37Z DEBUG bindgen] Trying to find clang with flags: ["-x", "c++", "-std=c++14"]
[2020-09-01T08:51:37Z DEBUG bindgen] Found clang: Clang { path: "/usr/bin/clang", version: Some(CXVersion { Major: 8, Minor: 0, Subminor: 1 }), c_search_paths: None, cpp_search_paths: Some(["/usr/include/c++/v1", "/usr/lib/clang/8.0.1/include", "/usr/include"]) }

I've tried setting LLVM_CONFIG_PATH and moving /usr/bin/clang out of the way while building bindgen and get the same results. Running a release build of bindgen gives a ton of output and eventually hangs here (memory and CPU usage are constant):

[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Item as CodeGenerator>::codegen: self = Item { id: ItemId(31472), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: Some("vector<_Tp, _Alloc>"), layout: None, kind: ResolvedTypeRef(TypeId(ItemId(15547))), is_const: false }) }
[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Type as CodeGenerator>::codegen: item = Item { id: ItemId(31472), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: Some("vector<_Tp, _Alloc>"), layout: None, kind: ResolvedTypeRef(TypeId(ItemId(15547))), is_const: false }) }
[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Item as CodeGenerator>::codegen: self = Item { id: ItemId(31473), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: None, layout: Some(Layout { size: 8, align: 8, packed: false }), kind: Reference(TypeId(ItemId(31474))), is_const: false }) }
[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Type as CodeGenerator>::codegen: item = Item { id: ItemId(31473), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: None, layout: Some(Layout { size: 8, align: 8, packed: false }), kind: Reference(TypeId(ItemId(31474))), is_const: false }) }
[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Item as CodeGenerator>::codegen: self = Item { id: ItemId(31474), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: Some("vector<_Tp, _Alloc>"), layout: None, kind: ResolvedTypeRef(TypeId(ItemId(15547))), is_const: false }) }
[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Type as CodeGenerator>::codegen: item = Item { id: ItemId(31474), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: Some("vector<_Tp, _Alloc>"), layout: None, kind: ResolvedTypeRef(TypeId(ItemId(15547))), is_const: false }) }
[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Item as CodeGenerator>::codegen: self = Item { id: ItemId(31477), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: Some("size_type"), layout: None, kind: Alias(TypeId(ItemId(31477))), is_const: false }) }
[2020-09-01T09:06:16Z DEBUG bindgen::codegen] <Type as CodeGenerator>::codegen: item = Item { id: ItemId(31477), local_id: LazyCell { inner: UnsafeCell }, next_child_local_id: Cell { value: 1 }, canonical_name: LazyCell { inner: UnsafeCell }, path_for_whitelisting: LazyCell { inner: UnsafeCell }, comment: None, annotations: Annotations { opaque: false, hide: false, use_instead_of: None, disallow_copy: false, disallow_debug: false, private_fields: None, accessor_kind: None, constify_enum_variant: false, derives: [] }, parent_id: ItemId(0), kind: Type(Type { name: Some("size_type"), layout: None, kind: Alias(TypeId(ItemId(31477))), is_const: false }) }

There's no further output ~4 minutes later.

[emilio]

That smells like we have a cycle in the graph (which should never happen) and end up stuck while trying to generate code for something. Dumping the type name to stdout or such in src/codegen/mod may give some clues to try to reduce a test-case.

[inferiorhumanorgans]

Is the type not size_type?