Beats: Add AIX support

[mostlyjason]

Describe the enhancement:
Currently, Filebeat does not support AIX but we've had several enhancement requests from customers to support it. AIX is not a common OS these days, but it does exist in great numbers in banking and insurance institutions that are highly regulated.

Additional details:
One customer has already created a version of Filebeat for AIX in version 5.5.1 but builds started to fail in 6.0. http://www.bullfreeware.com/pkg?id=4650

Go has gained support for AIX rather recently (minimum AIX 7.2), but some of our dependencies are not compiling on AIX. For example, Docker lacks support for AIX. You can see the conversation on Discuss here https://discuss.elastic.co/t/attempting-to-compile-filebeat-in-aix/143378/4 and https://discuss.elastic.co/t/disable-docker-support/178228. A fix has already been merged to use Docker only on supported platforms https://github.com/elastic/beats/pull/13509. There are additional dependencies that need support for AIX as well.

[Helflym]

Hi,
I'm a member behind the Bullfreeware website and I'm currently actively porting Filebeat, Metricbeat and Auditbeat for AIX 7.2. Filebeat is already in good shape and I'll soon start pushing a few patches to introduce AIX to the beats software. Moreover, a rpm of version 7.5.1 should be able at the same time on our website, for those who are interested in.

About dependencies, as you said a fix has been merged to disable Docker and K8s for platforms which don't support it. A few adjustments have to be made in order to fully disable the build of Docker packages (especially with tests), but it's no longer a needed dependency for AIX.
The other dependencies (for Filebeat, Metricbeat and Auditbeat) that I'm aware of are:

• github.com/elastic/gosigar: In progress https://github.com/elastic/gosigar/pull/133
• github.com/elastic/go-sysinfo: In progress https://github.com/elastic/go-sysinfo/pull/77
• github.com/fsnotify/fsnotify: In progress https://github.com/fsnotify/fsnotify/pull/326
• github.com/elastic/go-txfile: Merged https://github.com/elastic/go-txfile/pull/44. Wait for gofrs to vendor
• github.com/gofrs/flock: In progress https://github.com/gofrs/flock/pull/40
• github.com/lib/pq: Merged https://github.com/lib/pq/pull/929. Vendor possible
• github.com/insomniacslk/dhcp : Merged https://github.com/insomniacslk/dhcp/pull/352. Vendor possible
• github.com/mattn/go-isatty: Merged https://github.com/mattn/go-isatty/pull/35, Vendor possible

I'll keep posting updates on this issue.
Note, maybe the title can be changed to "Add AIX support" as Filebeat isn't the only one targeted.

Edit 2020/04/01: Update dependencies

[mostlyjason]

@Helflym Awesome thanks so much this is an amazing effort and I'm excited to see all the progress you've made. Definitely keep us up to date here as you make more progress. I also updated the title as you suggested.

I heard separately that metricbeat on AIX will require more effort due to the OS API being different. Are you working on that as well?

[Helflym]

I heard separately that metricbeat on AIX will require more effort due to the OS API being different. Are you working on that as well?

Yes of course. For the moment, it seems that porting metrics dependencies (gosigar, go-sysinfo, etc) is enough to have a working metricbeat. Maybe the data won't be fully accurate and some fields will be missing. But it will be enough for a first metricbeat, and I will have time to improve them if needed.

[Helflym]

A quick update to say that I've released a first beta version of filebeat, metricbeat and auditbeat under our website: http://www.bullfreeware.com/?searching=true&package=beat&from=&to=&libraries=false&exact=false&version=5. Any feedback about configuration and/or modules would be much appreciated.

I've started to push some patches for dependencies upstream but almost all of them are stuck right now. So, we'll have to wait to have AIX on the beats' upstream.
https://github.com/elastic/gosigar/pull/133
https://github.com/elastic/go-sysinfo/pull/77
https://github.com/shirou/gopsutil/pull/834
https://github.com/fsnotify/fsnotify/issues/9

[mostlyjason]

Thanks for this effort @Helflym

[adammike]

@Helflym are these the apache licensed filebeat/auditbeat code?

[Helflym]

@adammike I think yes. The maintainers will know more about this than me.

[sanandece]

@Helflym I want to try filbeat on AIX 7.2 to ship application logs to elastic and looking if there is a documented procedure. If there is one, can you point me to it please?

[Helflym]

@sanandece for now, the only available option to install filebeat is to use our RPM (http://www.bullfreeware.com/pkg?id=5615). We are planning to upstream the changes asap.

[Helflym]

I've updated the dependencies in my first post and added which ones have the AIX support merged in their upstream and which ones are still in progress. I've also added if they can be vendored here.

[mikemadden42]

For this effort are we targeting a set of POWER versions? It looks like Go only supports AIX 7.2.

From this article, it looks like AIX 7.2 needs POWER7 or newer CPUs. Are we going to target POWER7 & POWER8?

[Helflym]

Golang toolchain only AIX 7.2 and Power8+ is supported.
With gccgo, we might be able to build it for Power7, there will be a bit of work to do.

[inxonic]

I heard separately that metricbeat on AIX will require more effort due to the OS API being different. Are you working on that as well?

@mostlyjason @Helflym Just wanted to note that I put some effort into getting AIX specific metrics into Beats through the perfstat API a while ago. As of now that's far from complete regarding the set of metrics that could eventually be fetched. However it works for a few and the implementation could serve as an example for others. That's the code: https://github.com/WuerthIT/perfstatbeat

[Helflym]

@inxonic I've already done the work in order to have metricbeat working: https://github.com/elastic/gosigar/pull/133 and https://github.com/elastic/go-sysinfo/pull/77. However, same as you, I've only added the "mandatory" metrics for beats, some more can be added.
I'm also seen that you're building it with gccgo, did you try with the Golang Toolchain ?

[fearful-symmetry]

Just a brief update -- There's still a couple open PRs on elastic's end. We're waiting until we have an AIX dev environment until we merge them.

[serak]

can i use Linux metricbeat binary on AIX because on download page of https://www.elastic.co/downloads/beats/metricbeat i am not seeing any AIX image?

[Helflym]

@serak no, you can't for multiple various, the first one being AIX having a different executable format than Linux.
However, we've ported version 7.5.1 for AIX 7.2, it can be found there: http://www.bullfreeware.com/pkg?id=5616.

[fearful-symmetry]

Hey folks, brief update for anyone tracking. Elastic _finally_ has access to an AIX test environment, so I can start testing/developing support in metricbeat.

[fearful-symmetry]

Another update for y'all. The changes to go-sysinfo and gosigar have been merged. Working on beats itself now, starting with metricbeat.

[Helflym]

@fearful-symmetry I've checked the patches I've done when starting the port earlier this year. Most of them we'll be useless for you (the one dealing with either dependencies or magefiles) but I guess a few are still worth it.
I'm giving them as is. I mean to say that they are 6 months old so they might not be accurate anymore. Sadly I don't have time to update them right now...

0001-auditbeat-module-file_integrity-port-for-AIX.txt
0001-libbeat-processors-add_host_metadata-add-AIX.txt
0001-libbeat-processors-add_process_metadata-fix-TestBadP.txt
0001-metricbeat-add-system-metric-support-for-AIX.txt

[serak]

@serak no, you can't for multiple various, the first one being AIX having a different executable format than Linux.
However, we've ported version 7.5.1 for AIX 7.2, it can be found there: http://www.bullfreeware.com/pkg?id=5616.

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

user/# rpm -qR filebeat
/usr/bin/env
libc.a(shr_64.o)
libperfstat.a(shr_64.o)
libpthreads.a(shr_xpg5_64.o)
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1

appt1test:/opt/freeware/bin# prtconf
System Model: IBM,9117-MMC
Machine Serial Number: 064AC87
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 7
Processor Version: PV_7_Compat
Number Of Processors: 6
Processor Clock Speed: 3304 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 4 app1tst
Memory Size: 35840 MB
Good Memory Size: 35840 MB
Platform Firmware level: AM770_123
Firmware Version: IBM,AM770_123
Console Login: enable
Auto Restart: true
Full Core: false

[Helflym]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

[serak]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

[Helflym]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

[rajeshrgithub]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

In this case, Do I need to install Go toolchain separately on Power 8 or already built in function. What are the pre-requisite required inorder to start up metricbeat on Power 8.

[Helflym]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

In this case, Do I need to install Go toolchain separately on Power 8 or already built in function. What are the pre-requisite required inorder to start up metricbeat on Power 8.

Normally, you don't need anything special, not even the Go toolchain. The metricbeat provided by BullFreeware is static.

[rajeshrgithub]

i have installed this on AIX 7.2.0.0 and whenever i tried starting it using /opt/freeware/bin/filebeat -e it throws Illegal instruction(coredump) error

This filebeat is compiled using Golang toolchain which works only on Power8 and after. If you want to use filebeat on Power7, you'll need to rebuild it using gccgo.

Thank you for your quick response. the error message was useless. i have a couple of Power9 servers i will try on those. how is this issue going for a native support for aix? what is missing right now?

You're right, I should have displayed a warning when installing filebeat packages on Power7, as we do for Golang.
The problem is not with AIX itself, the Go toolchain on Power is made to work only on Power8 and later. It also impacts Linux. There was some discussions about supporting Power7 but it was refused, I don't remember why and where though.

In this case, Do I need to install Go toolchain separately on Power 8 or already built in function. What are the pre-requisite required inorder to start up metricbeat on Power 8.

Normally, you don't need anything special, not even the Go toolchain. The metricbeat provided by BullFreeware is static.

Thanks for your response. Only metricbeat-7.5.2-1.aix7.2.ppc.rpm package is enough from AIX client side right. I will be testing this package on power 8 server and metericbeat.yml should be fine to onboard in ELK dashboard.

[rajeshrgithub]

Successfully installed meteicbeat on power 8, how to start the daemon and configure.