Osquery: Port osquery to the AArch64 architecture

[alessandrogario]

This feature is being worked on by @artemist in the following PRs

• osquery: https://github.com/osquery/osquery/pull/6284
• osquery-toolchain: https://github.com/osquery/osquery-toolchain/pull/18

Task list

• [x] Merge the osquery-toolchain PR
• [x] Create a new release for the osquery-toolchain and upload both the x86_64 and AArch64 archives
• [ ] Update the documentation to include the supported AArch64-based Linux distributions that the static binary is compatible with
• [ ] Update the osquery.io website so that the new architecture can be displayed in the downloads section
• [x] Host any additional package/archive in the osquery organization (such as the AArch64-specific markupsafe package)
• [ ] Include the deb/rpm packages in the distro repositories
• [ ] (optional) Apply to AWS as an open source project through the Linux Foundation to request credits for a free AArch64 instance we can use for the CI

[artemist]

There are a few additional porting things to do once #6284 is merged:

• Port the system_info table
• Port the cpuid table and tests to run on AArch64
• Make sure se've properly mapped all AArch64 file, socket, and process syscalls in the audit tables

Probably more that I've forgotten

[lizthegrey]

Talked to @vielmetti about potentially getting access to N1 Neoverse Ampere via Packet rather than via AWS, as AWS's A1 boxes build a wee bit too slowly for CI purposes, and M6g is in closed preview (e.g. unlikely to be usable for CI soon)

[alessandrogario]

Talked to @vielmetti about potentially getting access to N1 Neoverse Ampere via Packet rather than via AWS, as AWS's A1 boxes build a wee bit too slowly for CI purposes, and M6g is in closed preview (e.g. unlikely to be usable for CI soon)

Thanks for looking into this!

Do you happen to know how much it would take to compile it under an A1 instance? The slowest job we have on the CI right now is Windows, so I think it can still be useful if it can complete the build process in under 38 minutes.

I have been trying to build everything under QEMU with an Intel 9900K and quickly realized how slow AArch64 emulation is! Later today I'll probably just get an EC2 instance and try it out, as it seems like it's the only option that would allow us to quickly unblock this work

[lizthegrey]

With the appropriate -jN setting, you can probably get it under 40 minutes, but you may need a 2xlarge or 4xlarge.

[artemist]

I've been testing on an a1.xlarge. I'll do a build with a cleared ccache to see how long that takes.

[artemist]

Building with ninja with tests enabled on an a1.xlarge took 1m8s for cmake, 76m35s for ninja, 36s for ninja package, and 1m11s for ninja test. An a1.4xlarge should be more than capable of getting that down to under 40 minutes

[theopolis]

Would it be possible to cross compile the aarch64 binary and tests, then use QEMU within Azure to run tests?

[alessandrogario]

Would it be possible to cross compile the aarch64 binary and tests, then use QEMU within Azure to run tests?

Yes! We can easily use cross compilation, and I think it's not that hard when using source-based libraries.

We can try with emulation, but it is extremely slow on my 9900K; I don't think the CPUs we get under Azure is really fast. It would still be a good idea to try and get the tests running on actualy hardware, due to the very different memory models of AArch64 vs x84_64

[artemist]

I think this method is _possible_, but I wouldn't recommend it. Clang can cross compile pretty easily if you get a proper binutils, libc++, and libc, but running the tests will be a bit of the problem. You'd have to use qemu-system-aarch64, not just user-mode emulation, meaning you have to get a big disk image from somewhere and find some hacky way to get the output from osquery into the image.

[vielmetti]

Thanks @lizthegrey @artemist - the best funnel into the Works on Arm program for requests is https://github.com/WorksOnArm/cluster . If you open an issue there I can set you up with either dedicated infra or info about builds using Drone Cloud or Travis CI both of which have arm64 native support.

[lizthegrey]

How are we doing with updating https://github.com/osquery/osquery/tree/aarch64-support?

[dguido]

Hey Liz,

I think this code needs to get rebased since Buck is no longer part of the project. I also noticed that a few tests are missing. Can you re-add them?

There is also a general issue with some of the libraries used not having the correct configuration, and they were regenerated on the reference system osquery uses for glibc compatibility (Ubuntu 16.04). There may be some libraries that need a second look before we can merge this PR.

While we (Trail of Bits, Alessandro, Stefano, et al) would love to help out, we are blocked on finishing up our work on eBPF. We want to fully complete that work before starting on anything new or major, like AArch64 support. If you can help push this along that would be great, otherwise we'll need to wait a bit or ask for help from elsewhere.

-Dan

[avcarrillo]

We are using OsQuery on 15k systems, mostly Windows. We have 60 remote offices we want are planning on deploying on Raspberry PI's/Jetson Nano's. I have compiled the aarch64 branch and it's working fine.

Would be great if aarch64 was official, there is a definite need.

[kshitij3199]

Hello everyone, I hope you all are fine and in good health.
I have build OSquery from source and have used it ( in Linux). Now, I wanted to use OSquery in Arch Linux so that I can compare the two. Can someone guide me on how to do so??
(I don't have arch Linux. I am thinking about taking Amazon services for same. )
Thanks in Advance.

[lizthegrey]

Aarch64 is a systems architecture by ARM holdings, and is distinct from the Linux distribution named "Arch". Sorry for the confusion, but this isn't the right place for your question.

[kshitij3199]

Thanks @lizthegrey :sweat_smile:

[AGSaidi]

I'd like to see arm64 binaries/packages hosted there and i'm willing to help. How are the artifacts for osquery.io generated? Travis CI supports Arm build hosts natively so we could use that.

[lizthegrey]

Yay! looking forward to the officially built packages containing the work

[theopolis]

Hi folks, heads up that the packages components are nearly finished but we still need the documentation changes. I do not think that will be a lot of effort, but we should add notes on how to do this here: https://github.com/osquery/osquery/blob/master/docs/wiki/development/building.md

Does someone mind taking that on?

[lizthegrey]

https://pkg.osquery.io/deb/osquery_4.5.1_1.linux.arm64.deb appears to be present in the s3 bucket but isn't added to the apt manifest. Can someone please use dput to upload it?