Solana: CPU Utilization On v1.3.N Stair-Steps Upwards

[brianlong]

Problem

CPU Utilization on 1.3 increases in a stair-step fashion until the CPUs are maxed-out. Other validators are seeing the same problem, so I created this issue to hold screenshots & log fragments for debugging.

[brianlong]

[brianlong]

This is a look at htop while running about 50%CPU. I will post another screenshot when CPU % steps up again.

[sakridge]

@brianlong super helpful, thanks!

@mvines can you see what send-tx-svc is doing? Or see who should look at that?

[mvines]

Interesting!

@brianlong - are you setup to easily apply patches and rebuild from source? I have a patch at https://github.com/solana-labs/solana/pull/12320 that would be useful to apply to help narrow this down a little more. Happy to wait until we step up more though and need to kill the process. Based on that there might more a little more logging we can add to help debug this more.

My guess right now: you'll see another send-tx-svc thread each time the CPU steps up. I bet some code is panicking and re-creating a new send-tx-svc

[brianlong]

@mvines I might need a little coaching, but I'll give it a try this evening.

[mvines]

Pretty sure the bug is in the BankServer's version of the SendTransactionService: https://github.com/solana-labs/solana/blob/9b866d79fb1173d837ad18273a7034ae7fb4fd18/banks-server/src/banks_server.rs#L230
I don't think it will cleanly shut down the SendTransactionService in some cases, which could cause that send-tx-svc thread to spin loop and eat up a CPU.

This would pretty much explain the CPU stepping behavior

[brianlong]

@mvines I've done sudo apt install libudev-dev and I still see this error when trying to compile from your fork & dup branch. `` error: failed to run custom build command forhidapi v1.2.3`

Caused by:
process didn't exit successfully: /home/brianlong/solana/target/release/build/hidapi-1e613a883643668a/build-script-build (exit code: 101)
--- stderr
thread 'main' panicked at 'Unable to find libudev: Command { command: ""pkg-config" "--libs" "--cflags" "libudev"", cause: Os { code: 2, kind: NotFound, message: "No such file or directory" } }', /home/brianlong/.cargo/registry/src/github.com-1ecc6299db9ec823/hidapi-1.2.3/build.rs:53:54
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
```
Any suggestions?

[brianlong]

@mvines Yes, I now see five threads for send-tx-svc. I think you're on the right track.

[t-nelson]

@brianlong try installing the full set of system dependencies listed here

[brianlong]

@t-nelson Thanks. I am now up & running on the @mvines version of 1.4.0 devbuild. I think I also just sent all of my testnet SOL to Michael ;-)

[brianlong]

I'll let this run for a while and report back when I see the CPU step-up.

[brianlong]

@mvines It just occurred to me that you might have patched a different branch, not 1.4.0. Do you want me running 1.4.0?

[mvines]

v1.3 would have been better than v1.4 but I think it's ok. Actually it's possible that on both the tip of v1.3 and v1.4/master the issue will no longer reproduce due to another bank_server bug we recently fixed.

But no matter, I'll look at fixing up what looks to be the root cause shortly

[brianlong]

I saw your change in the tip of the dup branch, so I ran with it. ;-)

[brianlong]

@mvines Here is a look at the threads this morning.

[mvines]

Ok great, two send-tx-svc threads at ~99% cpu. That fits my theory, thanks Brian!