Yugabyte-db: Core dump with "Illegal instruction: 4" on Mac

[acoliver]

For some reason this morning YB 2.2.0.0 started core dumping on my MBP. Downgraded to 2.1.8.2 and it starts without incident. Tried destroying, rebooting, updating, checked file handles...same issue.
Archive.zip

[kmuthukk]

@svarnau - Is this relevant

https://stackoverflow.com/questions/14268887/what-is-the-illegal-instruction-4-error-and-why-does-mmacosx-version-min-10

[acoliver]

I don't think so. That version of OS X is years ago. I haven't run that at any time.

[svarnau]

The crash file shows this was on macOS 10.15.6. Yugabyte was compiled on 10.14. The compile flag referenced on the web page is relevant in the reverse case, when it is running on an older version than it was compiled on.

[acoliver]

sudo dmesg        
owed, cache,  1595960181Waking up reference: 624Thread waiting on reference 624 woke upWaking up reference: 623Thread waiting on reference 623 woke upevaluation result: 624, allowed, cache,  1595446610evaluation result: 623, allowed, cache,  1595446610filecache_entry_set_syspolicy_locked: invalidating <ptr>initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libgflags.2.dylib/libgflags.2.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libgflags.2.dylib/libgflags.2.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 626Thread waiting on reference 626 woke upevaluation result: 626, allowed, cache,  1595960181Waking up reference: 625Thread waiting on reference 625 woke upevaluation result: 625, allowed, cache,  1595960181Waking up reference: 627Thread waiting on reference 627 woke upevaluation result: 627, allowed, cache,  1595446610initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libversion_info_proto.dylib/libversion_info_proto.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 628Thread waiting on reference 628 woke upevaluation result: 628, allowed, cache,  1595446610initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libev.4.dylib/libev.4.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
Waking up reference: 629Thread waiting on reference 629 woke upevaluation result: 629, allowed, cache,  1595960181initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libev.4.dylib/libev.4.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 630Thread waiting on reference 630 woke upevaluation result: 630, allowed, cache,  1595960181Waking up reference: 631Thread waiting on reference 631 woke upevaluation result: 631, allowed, cache,  1595960181Waking up reference: 632Thread waiting on reference 632 woke upevaluation result: 632, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libcrypto.1.0.0.dylib/libcrypto.1.0.0.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
Waking up reference: 633Thread waiting on reference 633 woke upevaluation result: 633, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libcrypto.1.0.0.dylib/libcrypto.1.0.0.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 614Thread waiting on reference 614 woke upevaluation result: 614, allowed, cache,  1595960181Waking up reference: 636Thread waiting on reference 636 woke upevaluation result: 636, allowed, cache,  1595446610initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libicuuc.67.dylib/libicuuc.67.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 637Thread waiting on reference 637 woke upevaluation result: 637, allowed, cache,  1595960181Waking up reference: 638Thread waiting on reference 638 woke upevaluation result: 638, allowed, cache,  1595446610initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libcrcutil.0.dylib/libcrcutil.0.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 639Thread waiting on reference 639 woke upevaluation result: 639, allowed, cache,  1595960181Waking up reference: 640Thread waiting on reference 640 woke upevaluation result: 640, allowed, cache,  1595446610initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446610 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libhistogram_proto.dylib/libhistogram_proto.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 641Thread waiting on reference 641 woke upevaluation result: 641, allowed, cache,  1595960181Waking up reference: 642Thread waiting on reference 642 woke upevaluation result: 642, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libpb_util_proto.dylib/libpb_util_proto.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 643Thread waiting on reference 643 woke upevaluation result: 643, allowed, cache,  1595960181Waking up reference: 634Thread waiting on reference 634 woke upevaluation result: 634, allowed, cache,  1595960181Waking up reference: 635Thread waiting on reference 635 woke upevaluation result: 635, allowed, cache,  1595960181Waking up reference: 644Thread waiting on reference 644 woke upevaluation result: 644, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libencryption_proto.dylib/libencryption_proto.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 645Thread waiting on reference 645 woke upevaluation result: 645, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libssl.1.0.0.dylib/libssl.1.0.0.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
Waking up reference: 646Thread waiting on reference 646 woke upevaluation result: 646, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libssl.1.0.0.dylib/libssl.1.0.0.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 647Thread waiting on reference 647 woke upevaluation result: 647, allowed, cache,  1595960181initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libssl.1.0.0.dylib/libssl.1.0.0.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 649Waking up reference: 648Thread waiting on reference 649 woke upevaluation result: 649, allowed, cache,  1595960181Thread waiting on reference 648 woke upWaking up reference: 650evaluation result: 648, allowed, cache,  1595960181Thread waiting on reference 650 woke upevaluation result: 650, allowed, cache,  1595960181Waking up reference: 651Thread waiting on reference 651 woke upevaluation result: 651, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libcurl.4.dylib/libcurl.4.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 652Thread waiting on reference 652 woke upevaluation result: 652, allowed, cache,  1595446611initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446611 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libcurl.4.dylib/libcurl.4.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
Waking up reference: 653Thread waiting on reference 653 woke upevaluation result: 653, allowed, cache,  1595446612initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446612 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libprotobuf.15.dylib/libprotobuf.15.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 654Waking up reference: 655Thread waiting on reference 654 woke upThread waiting on reference 655 woke upevaluation result: 654, allowed, cache,  1595960181evaluation result: 655, allowed, cache,  1595960181initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446612 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libprotobuf.15.dylib/libprotobuf.15.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446612 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libprotobuf.15.dylib/libprotobuf.15.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
Waking up reference: 656Thread waiting on reference 656 woke upevaluation result: 656, allowed, cache,  1595960181Waking up reference: 657Thread waiting on reference 657 woke upevaluation result: 657, allowed, cache,  1595960181Waking up reference: 658Thread waiting on reference 658 woke upevaluation result: 658, allowed, cache,  1595960181Waking up reference: 660Waking up reference: 659Thread waiting on reference 660 woke upThread waiting on reference 659 woke upevaluation result: 660, allowed, cache,  1595446612evaluation result: 659, allowed, cache,  1595446612initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446612 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libicudata.67.dylib/libicudata.67.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446612 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libicudata.67.dylib/libicudata.67.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 661Thread waiting on reference 661 woke upevaluation result: 661, allowed, cache,  1595446612initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446577 lastFileScanTime: 1595446612 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/libicudata.67.dylib/libicudata.67.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
TX(24:05:88:42:BB:3A) <Success NoACK Expired OtherErr> BE<418 0 0 0> BK<0 0 0 0> VI<0 0 0 0> VO<1 0 0 0> (5000ms)
L3 Control VO TX(24:05:88:42:BB:3A) Success=1 NoACK=0 Expired=0 OtherErr=0
TX(FF:FF:FF:FF:FF:FF) <Success NoACK Expired OtherErr> BE<0 0 0 0> (5000ms)
Waking up reference: 662Thread waiting on reference 662 woke upevaluation result: 662, allowed, cache,  1595960182Waking up reference: 664Thread waiting on reference 664 woke upevaluation result: 664, allowed, cache,  1595960182Waking up reference: 663Thread waiting on reference 663 woke upevaluation result: 663, allowed, cache,  1595960182Waking up reference: 665Thread waiting on reference 665 woke upevaluation result: 665, allowed, cache,  1595446612initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446578 lastFileScanTime: 1595446612 pid: 3177 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/CoreFoundation/CoreFoundation proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-tserver
Waking up reference: 666Thread waiting on reference 666 woke upevaluation result: 666, allowed, cache,  1595446612initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446578 lastFileScanTime: 1595446612 pid: 3183 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/CoreFoundation/CoreFoundation proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-admin
Waking up reference: 667Thread waiting on reference 667 woke upevaluation result: 667, allowed, cache,  1595446612initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446578 lastFileScanTime: 1595446612 pid: 3174 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/lib/CoreFoundation/CoreFoundation proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/bin/yb-master
Waking up reference: 668Thread waiting on reference 668 woke upevaluation result: 668, allowed, cache,  1595960182Waking up reference: 669Thread waiting on reference 669 woke upevaluation result: 669, allowed, cache,  1595960182Waking up reference: 670Thread waiting on reference 670 woke upevaluation result: 670, allowed, cache,  1595960182Waking up reference: 671Thread waiting on reference 671 woke upevaluation result: 671, allowed, cache,  1595446612initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446576 lastFileScanTime: 1595446612 pid: 3184 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/postgres/bin/initdb proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/postgres/bin/initdb
Waking up reference: 672Thread waiting on reference 672 woke upevaluation result: 672, allowed, cache,  1595960182AMFI: Process does not have a code signature, allowing core dump for pid 3177
AMFI: Process does not have a code signature, allowing core dump for pid 3174
Waking up reference: 673Thread waiting on reference 673 woke upevaluation result: 673, allowed, cache,  1595446613initiating malware scan (activeRulesVersion: 8442615140820764123 lastScanVersion: 2153796084736802050 chgtime: 1595446576 lastFileScanTime: 1595446613 pid: 3184 info_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/postgres/lib/libpq.5.11.dylib proc_path: /Users/acoliver/projects/start-yuga/yugabyte-2.2.0.0/postgres/bin/initdb
Waking up reference: 674Thread waiting on reference 674 woke upevaluation result: 674, allowed, cache,  1595960182Waking up reference: 675Thread waiting on reference 675 woke upevaluation result: 675, allowed, cache,  1595939982017179.235056 wlan0.A[3167] updateLQM@3339:No per core RSSI to report

[svarnau]

Confirmed. I can reproduce this on macOS 10.15.5

[svarnau]

This is caused due to the way a third-party dependency was compiled for macOS. It has already been fixed on the master branch (picking up a newer release of the yugabyte/yugabyte-db-thirdparty repo). Will look at potentially back-porting to the 2.2 branch.

[kmuthukk]

Thx @svarnau!

[mbautin]

@svarnau So this was not an illegal instruction, it was just the wrong -mmacosx-version-min setting, right?

[svarnau]

@mbautin I think so. The combination of the dependency being built on 10.15 without the version-min flag, and the main DB code built on 10.14 seems to run on 10.14, but fail on 10.15.

When I found that the problem did not occur on the latest master branch builds, I did a binary search through older snapshot builds to pinpoint the change that fixed the problem. Namely, the upgrade of thirdparty dependency version. It was shortly before 2.2 branched that macOS builds started using the pre-built dependencies.

[svarnau]

Fix is backported and verified, so should be in the 2.2.2 release.