Osquery: "docker_container_processes" Not Showing Anything on EC2 instance.

[sanghyunhong]

Hello,

I am running osquery in AWS EC2 instances, and those instances are running docker containers as well. I tried to run the command select * from docker_container_processes where id = '<container_id>' from the osqueryi command line, but anything showed up.

Can you look at this issue? the same command works at the Ubuntu environment.

• S.

[obelisk]

Were you running osqueryi as root?

[sanghyunhong]

@obelisk Yes, I did.

[sanghyunhong]

@obelisk Hey, this is the error I ran the osquery with the verbose mode
I0126 22:10:30.363524 4977 docker.cpp:537] Error getting docker container <container_id>: Invalid docker API response for /containers/<container_id>/top?ps_args=axwwo%20pid,state,uid,gid,euid,egid,suid,sgid,rss,vsz,etimes,ppid,pgrp,nlwp,nice,user,time,pcpu,pmem,comm,cmd: HTTP/1.0 500 Internal Server Error

Can you please check what can cause this issue?

[obelisk]

It seems that there is an internal server error which is unrelated to osquery (I think, I don't use docker).

@alessandrogario You were mentioning some API changes coming up with this in office hours, can you elaborate here as well?

[spasam]

Hmmm, could be that the container does not support the ps args! Which container images are you running?

[alessandrogario]

@obelisk Once my other PR is accepted (https://github.com/facebook/osquery/pull/4024), I will have everything needed to add container support to the processes table as well. This works in a generic and more robust way, by checking the user namespaces used by each process.

Once it is implemented, I think it would be best to drop the docker_container_processes table; it is best to just use processes and join on docker_containers (which is how process_open_sockets and listening_ports work inside my branch).

[sanghyunhong]

@spasam I have been running three containers (Nginx, Apache, and Ubuntu [Bash]). I have tested three environments: one is with the ubuntu as a host OS, and the others are Mac and EC2 Amazon Linux. Can the container types be cause of the issue?

@alessandrogario I wonder when your PR will be released as a stable version ... If it can take some time, can you please let me know how you join two tables (what is the key to join)?

[spasam]

Docker container processes API is a hack as it executes ps command in the container. We pass the following arguments to it: axwwo pid,state,uid,gid,euid,egid,suid,sgid,rss,vsz,etimes,ppid,pgrp,nlwp,nice,user,time,pcpu,pmem,comm,cmd

See if you can get to the containers shell and run ps command with above arguments. It looks like exec is failing. You might also find more info in syslog or where ever docker daemon logs to!

[sanghyunhong]

@spasam I was passed the wrong command. The correct output will be:

root@01ce99521bfc:/# ps -axwwo pid,state,uid,gid,euid,egid,suid,sgid,rss,vsz,etimes,ppid,pgrp,nlwp,nice,user,time,pcpu,pmem,comm,cmd
  PID S   UID   GID  EUID  EGID  SUID  SGID   RSS    VSZ ELAPSED  PPID  PGRP NLWP  NI USER         TIME %CPU %MEM COMMAND         CMD
    1 S     0     0     0     0     0     0  3252  18248     336     0     1    1   0 root     00:00:00  0.0  0.1 bash            /bin/bash
   26 R     0     0     0     0     0     0  2860  34424       0     1    26    1   0 root     00:00:00  0.0  0.1 ps              ps -axwwo pid,state,uid,gid,euid,egid,suid,sgid,rss,vsz,etimes,ppid,pgrp,nlwp,nice,user,time,pcpu,pmem,comm,cmd

[alessandrogario]

@sanghyunhong Keep in mind that as of now only process_open_sockets and listening_ports have support for user namespaces in my PR. I plan to extend it to processes (and other tables too) once it is merged - because it contains code I need.

An example query can be seen here: https://github.com/facebook/osquery/pull/4024#issuecomment-354616426

I find it is a better (and more portable approach) than trying to use specific API; the fact that you don't need extra tables is also an added bonus (IMHO).

EDIT: The feature provided by the PR is needed because the docker_container_ports table does not really show which ports are used inside containers (it only shows what - by configuration - is supposed to be forwarded from the host and nothing else).

[sanghyunhong]

@alessandrogario Just a quick question, are you assuming all the docker processes' information stored in the processes table? Otherwise, @spasam, why do we need the docker_container_processes query?

[alessandrogario]

@sanghyunhong All processes inside containers are also shown in the processes table, but since support for user namespaces is missing you have no idea whether each process is running on the primary namespace (i.e. the host) or inside a container.

EDIT: I would remove the docker_container_processes table once everything is finalized.

[sanghyunhong]

@alessandrogario Thanks for those answers :)
... However, I am sorry. Got a bit confused ...

You proposed to join the processes table and the docker_containers table to retrieve the information about the processes running inside the docker, and now, you said we cannot do that because no username space information is not in both tables. Then, what do you recommend me to use as a key to combine the two tables?

[alessandrogario]

@sanghyunhong The PR I linked contains code that lets you join process_open_sockets and listening_ports with docker_container. It also contains library code I need to add the same kind of functionality to processes and other tables. Once the PR is merged, i'll work on it.

[sanghyunhong]

@alessandrogario thanks! The last questions may be: how much time will it take usually for the both "the PR will be merged" and "the additions of other functionalities are done"?

We're really looking forward to using the functionalities ...

[alessandrogario]

@sanghyunhong don't mention it! The PR is still under review, so I can't tell you how long before it is ready. Implementing the rest is straightforward, and will not take much time.

[obelisk]

@alessandrogario We are working on reviewing your diff currently :)

[muffins]

This issue looks to be a repeat of #4070? Is that not the case? If so let's close this one out and keep notes aggregated in one issue.

[sanghyunhong]

@muffins sure, let's refer the issue #4070 and close it.