Tarpaulin: Inconsistent segfault error (non-deterministic)

[woshilapin]

On a project I'm working on, I have this inconsistent error

[ERROR tarpaulin] Failed to get test coverage! Error: Failed to run tests: A segfault occurred while executing tests
Error: "Failed to get test coverage! Error: Failed to run tests: A segfault occurred while executing tests"
##[error]The process '/usr/share/rust/.cargo/bin/cargo' failed with exit code 1

I say _inconsistent_ because it never happens at the same time of the process. I've been able to reproduce locally at some point but then, launching the same cargo tarpaulin command over and over, it eventually works. Funnier, it seems that once it works, it always works (I even was able to do a cargo clean and make it work). So the error never happen consistently.

I'm working on Ubuntu 20.04 with the following (but this error also happened on our CI https://github.com/CanalTP/transit_model/pull/558/checks?check_run_id=716136628)

$> rustup --version
rustup 1.21.1 (7832b2ebe 2019-12-20)
$> cargo --version
cargo 1.43.0 (2cbe9048e 2020-05-03)
$> rustc --version
rustc 1.43.1 (8d69840ab 2020-05-04)
$> cargo tarpaulin --version
cargo-tarpaulin version: 0.13.2

I looked for similar issues but didn't feel like the problem was the same. Sorry if this is a duplicate.

[Enrico2]

Seeing the same problem, e.g. here https://github.com/apollographql/apollo-cli/pull/52/checks?check_run_id=721906925

[xd009642]

Interesting, once I finish the stuff I'm currently working on I'll have a look and see if I can get to the bottom of it

[woshilapin]

Interesting, once I finish the stuff I'm currently working on I'll have a look and see if I can get to the bottom of it

Thank you very much, don't hesitate to ask if you need help, information or if I can do anything to help. I'll try to do as much as possible.

[woshilapin]

Hi, any news on this issue? Is there anything I can do to help?

[xd009642]

I managed to recreate it once but haven't yet figured out what causes it. Though I admit I ended up not working on tarpaulin as much the past few weeks. I'll have a look today or tomorrow see if I can figure it out.

I also made a tarpaulin debugging tool last year which I think I'll integrate into the code to make diagnosing issues like this easier. Right now it exists in a different repo as a harness around some of the internals and needs some updating. Once I've done that I might ask you if you can use that branch to generate some debug traces and send them to me

[xd009642]

So I haven't managed to recreate it with transit_model with the latest tarpaulin, but also the traces that come out are so large I can't render them at a size where they're readable. Just trying to with apollo-cli. But @woshilapin and @Enrico2 do you still see this with the latest version or both the rust compiler (stable or beyond) and tarpaulin?

[zeeshanlakhani]

Seeing the same issue: https://github.com/capsule-rs/capsule/pull/106/checks?check_run_id=835393227. Using nightly and the args below.

Run actions-rs/cargo@v1
  with:
    command: tarpaulin
    args: -p capsule -l --count --exclude-files examples/* ffi/* macros/* --out Xml -Zpackage-features --features full
    use-cross: false

[xd009642]

ahh capsule is good, I can recreate it reliably using it and there's no so many traces that it's insurmountable to display a plot :+1: I'll start using that to investigate this issue

[woshilapin]

So I haven't managed to recreate it with transit_model with the latest tarpaulin, but also the traces that come out are so large I can't render them at a size where they're readable. Just trying to with apollo-cli. But @woshilapin and @Enrico2 do you still see this with the latest version or both the rust compiler (stable or beyond) and tarpaulin?

Hi, sorry for not answering earlier. I tried again to run the build a few minutes ago and I'm still seeing the same error (you can check the build https://github.com/CanalTP/transit_model/pull/558/checks?check_run_id=845031615). Note that this is version 0.14.1 of tarpaulin.

[woshilapin]

I tried again today with version 0.14.3 (the last try was on 0.14.1) but reached a different error than the original one in the ticket. Interestingly, I was using actions-rs/rust-tarpaulin to run tarpaulin. But then I tried to remove entirely actions-rs/rust-tarpaulin and install/run manually tarpaulin: it works!

So it might be the case that the bug has been fixed in the latest version? I'm curious if @zeeshanlakhani or @Enrico2 are still seing the problem on their sides?

[zeeshanlakhani]

@woshilapin it didn't work for me on 0.14.2, but I will try this later today with the recent version.

[xd009642]

In the latest version I found that some linker settings cause cause dwarf lines marked at address 0x00 (+ offset) which I'd never seen before and caused a segfault. I added something to get the address range of the text section and not try to instrument addresses outside of it.

I wouldn't think that would affect this as the segfault happened as soon as tarpaulin tried to place the breakpoint and happened every time... But then again linkers are part engineering, part black magic so maybe?

[zeeshanlakhani]

@xd009642 @woshilapin still failing for me on 0.14.3:

[ERROR tarpaulin] Failed to get test coverage! Error: Failed to run tests: A segfault occurred while executing tests
Error: "Failed to get test coverage! Error: Failed to run tests: A segfault occurred while executing tests"

[Enrico2]

Our code base changed significantly so it's possible that any attempt to
reproduce will not be reproducing under the same conditions. We are
currently using grcov (because it's also runnable on a Mac, for local
development), but I'm actually curious to try tarpaulin again to see if
there are changes in the coverage report. I will update once I do. Thanks!

On Wed, Sep 2, 2020, 8:47 AM Zeeshan Lakhani [e-mail]
wrote:

@xd009642 https://github.com/xd009642 @woshilapin
https://github.com/woshilapin still failing for me on 0.14.3:

[ERROR tarpaulin] Failed to get test coverage! Error: Failed to run tests: A segfault occurred while executing tests
Error: "Failed to get test coverage! Error: Failed to run tests: A segfault occurred while executing tests"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xd009642/tarpaulin/issues/461#issuecomment-685825052,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABHMIQACMS62FPSPHLOO2LSDZSJZANCNFSM4NNJGLSQ
.

[xd009642]

tarpaulin once again has refined some of the tracing and offset handling and now adds less linker flags when building your tests. I haven't tried on capsule again but it's solved a number of different issues so :crossed_fingers:

[zeeshanlakhani]

@xd009642 sadly:

Nov 14 19:11:17.023 WARN cargo_tarpaulin::statemachine::linux: Failed to find process for pid: 10764 Nov 14 19:11:17.023 WARN cargo_tarpaulin::statemachine::linux: Failed to find traces for pid: 10764 Nov 14 19:11:17.024 WARN cargo_tarpaulin::statemachine::linux: Failed to find process for pid: 10764 Nov 14 19:11:17.024 WARN cargo_tarpaulin::statemachine::linux: Failed to find traces for pid: 10764 Nov 14 19:11:17.029 ERROR cargo_tarpaulin: Failed to get test coverage! Error: Failed to run tests: Error running test - SIGILL raised in 10764 Error: "Failed to get test coverage! Error: Failed to run tests: Error running test - SIGILL raised in 10764"

[bitshifter]

I have been moving my code coverage from travis-ci to github actions and started to hit this issue. I switched from using the docker image to running cargo install cargo-tarpaulin before running coverage and the problem went away.

[xd009642]

@bitshifter that might be down to the latest release being yanked but I haven't done anything to try and yank/replace the latest tag on docker hub. I'll try and do that today to fix the docker image

[robinst]

For anyone else hitting the problem, you can pin the Docker image version in GitHub actions to 0.16.0 like this:

container:
  image: xd009642/tarpaulin:0.16.0
  options: --security-opt seccomp=unconfined