Lxd: tcpdump in unprivileged container: unable to write to stdout/stderr

Created on 22 Feb 2017  路  11Comments  路  Source: lxc/lxd

Required information

  • Distribution: ubuntu
  • Distribution version: 16.04
  • The output of "lxc info"

~~~
apiextensions:

  • id_map
    apistatus: stable
    apiversion: "1.0"
    auth: trusted
    environment:
    addresses: []
    architectures:

    • x86_64

    • i686

      certificate:

      certificatefingerprint:

      driver: lxc

      driverversion: 2.0.6

      kernel: Linux

      kernelarchitecture: x86_64

      kernelversion: 4.4.0-62-generic

      server: lxd

      serverpid: 6550

      serverversion: 2.0.8

      storage: btrfs

      storageversion: "4.4"

      config:

      images.auto_update_cached: "false"

      images.auto_update_interval: "0"

      public: false

      ~~~

Issue description

tcpdump is not working in unprivileged containers.

At first I thought this was something to do with capabilities (#2507). However tcpdump is able to capture packets and write them to a pcap file. It simply cannot write to stdout/stderr!

Steps to reproduce

  1. Create two Ubuntu 16.04 containers, one privileged, one not.
  2. ssh into each one, and then use sudo -s to get root. (Do not use lxc exec because of another issue)
  3. Inside one run tcpdump -i eth0 -nn not tcp port 22, and ping from the other.

tcpdump in the privileged container works just fine.

tcpdump in the unprivileged container does not show any output. But if I run strace on it I see errors attempting to access stdout and stderr:

~
ioctl(1, TCGETS, 0x7fff97c8d680) = -1 ENOTTY (Inappropriate ioctl for device)
...
write(2, "tcpdump: verbose output suppress"..., 75) = -1 EACCES (Permission denied)
write(2, "listening on eth0, link-type EN1"..., 74) = -1 EACCES (Permission denied)
~

This is very weird. Even more weird: the following command does capture packets:

~
tcpdump -i eth0 -nn -w foo.pcap
~

The file foo.pcap grows. This proves it's nothing to do with network capture perms.

But the following command shows no output:

~
tcpdump -r foo.pcap -nn
~

And again it's because it can't write to stdout:

~
fstat(1, 0x7ffe2fb5eb10) = -1 EACCES (Permission denied)
read(3, "", 4096) = 0
write(1, "14:34:30.618180 IP6 fe80::c609:6"..., 1740) = -1 EACCES (Permission denied)
~

I had originally thought this was to do with capabilities. But if I run capsh --print inside both containers, they both have cap_net_raw and cap_net_admin. In fact, the unprivileged container has two additional capabilities! (cap_mac_override and cap_mac_admin)

So now I suspect that apparmor is at fault.

dmesg

dmesg output generated by the following steps:

  • start tcpdump
  • wait 5 seconds
  • send 1 ping from other side
  • wait 5 seconds
  • stop tcpdump

~
[429020.685987] audit: type=1400 audit(1487774339.708:3597): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="/dev/pts/0" pid=12539 comm="tcpdump" requested_mask="wr" denied_mask="wr" fsuid=100000 ouid=101001
[429020.686000] audit: type=1400 audit(1487774339.708:3598): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="/dev/pts/0" pid=12539 comm="tcpdump" requested_mask="wr" denied_mask="wr" fsuid=100000 ouid=101001
[429020.686013] audit: type=1400 audit(1487774339.708:3599): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="/dev/pts/0" pid=12539 comm="tcpdump" requested_mask="wr" denied_mask="wr" fsuid=100000 ouid=101001
[429020.686022] audit: type=1400 audit(1487774339.708:3600): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="/dev/pts/0" pid=12539 comm="tcpdump" requested_mask="wr" denied_mask="wr" fsuid=100000 ouid=101001
[429020.716725] device eth0 entered promiscuous mode
[429020.741308] audit: type=1400 audit(1487774339.764:3601): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429020.741330] audit: type=1400 audit(1487774339.764:3602): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429021.716785] audit: type=1400 audit(1487774340.740:3603): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="r" denied_mask="r" fsuid=100000 ouid=0
[429030.630448] audit: type=1400 audit(1487774349.652:3604): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429030.630543] audit: type=1400 audit(1487774349.652:3605): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429030.630555] audit: type=1400 audit(1487774349.652:3606): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429030.630565] audit: type=1400 audit(1487774349.652:3607): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429030.630574] audit: type=1400 audit(1487774349.652:3608): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429030.630584] audit: type=1400 audit(1487774349.652:3609): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429030.630593] audit: type=1400 audit(1487774349.652:3610): apparmor="DENIED" operation="file_perm" info="Failed name lookup - disconnected path" error=-13 namespace="root//lxd-srv2-campus1_" profile="/usr/sbin/tcpdump" name="apparmor/.null" pid=12539 comm="tcpdump" requested_mask="w" denied_mask="w" fsuid=100000 ouid=0
[429030.630687] device eth0 left promiscuous mode
~

privileged container

~~~
$ lxc config show srv1-campus1
name: srv1-campus1
profiles:

  • br-cnd11
    config:
    security.privileged: "true"
    volatile.base_image: 315bedd32580c3fb79fd2003746245b9fe6a8863fc9dd990c3a2dc90f4930039
    volatile.eth0.hwaddr: 00:16:3e:25:c5:bf
    volatile.idmap.base: "0"
    volatile.idmap.next: '[]'
    volatile.last_state.idmap: '[]'
    volatile.last_state.power: RUNNING
    devices:
    root:
    path: /
    type: disk
    ephemeral: false

root@srv1:~# capsh --print
Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_syslog,cap_wake_alarm,cap_block_suspend,37+ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_syslog,cap_wake_alarm,cap_block_suspend,37
Securebits: 00/0x0/1'b0
secure-noroot: no (unlocked)
secure-no-suid-fixup: no (unlocked)
secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=0(root)
~~~

unprivileged container

~~~
$ lxc config show srv2-campus1
name: srv2-campus1
profiles:

  • br-cnd11
    config:
    volatile.base_image: 315bedd32580c3fb79fd2003746245b9fe6a8863fc9dd990c3a2dc90f4930039
    volatile.eth0.hwaddr: 00:16:3e:33:9c:29
    volatile.idmap.base: "0"
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
    volatile.last_state.power: RUNNING
    devices:
    root:
    path: /
    type: disk
    ephemeral: false

root@srv2:~# capsh --print
Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,37+ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,37
Securebits: 00/0x0/1'b0
secure-noroot: no (unlocked)
secure-no-suid-fixup: no (unlocked)
secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=0(root)
~~~

Related

LP#1641236 describes a problem with tcpdump buffering stdout when launched from lxc exec, and since it involves apparmor, it may be related.

However that issue is specifically about opening a pty on the host, and then passing it into the container. Here, the pty is created inside the container (by sshd).

picture candlerb
👍1

Most helpful comment

Yes, this does fix it, thank you:

~~~

ln -s /etc/apparmor.d/usr.sbin.tcpdump /etc/apparmor.d/disable/

apparmor_parser -R /etc/apparmor.d/disable/

~~~

picture candlerb on 22 Feb 2017
👍2

All 11 comments

The containers were built a couple of weeks ago, and I see that a new tcpdump package been released since then:

~
(New) Version: 4.9.0-1ubuntu1~ubuntu16.04.1
(Old) Version: 4.7.4-1ubuntu1
~

However, upgrading inside the container didn't make a difference.

There is also a new version of lxd (2.0.9 vs 2.0.8). Updating the host and then restarting the container also made no difference. Still have:

~
write(2, "tcpdump: verbose output suppress"..., 75) = -1 EACCES (Permission denied)
write(2, "listening on eth0, link-type EN1"..., 74) = -1 EACCES (Permission denied)
~

candlerb picture candlerb on 22 Feb 2017

What you're describing is exactly that bug you mentioned (https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1641236).

Feel free to comment in the bug to indicate that you ran into it and it'd be great if this could be fixed. Maybe we can get some more traction on the apparmor team for this.

It's effectively a kernel bug in the apparmor implementation and not something we can do anything about in LXD proper.

You do have a few workarounds available though:

  • Using ssh instead of "lxc exec" should work fine (as ssh will allocate its own pts pair)
  • Using "lxc exec" and then running a "script" session should have the same effect as ssh
  • You can turn off the tcpdump apparmor profile by symlinking the profile to /etc/apparmor.d/disable/ then reload apparmor in the container (or reboot it).

I'm closing this issue since there's nothing we can do in LXD itself and we're already tracking this apparmor problem on Launchpad. Feel free to comment and subscribe to the bug over there.

stgraber picture stgraber on 22 Feb 2017

Using ssh instead of "lxc exec" should work fine (as ssh will allocate its own pts pair)

Sorry, but that's exactly what I'm doing: I'm ssh'ing into the container and I am not using lxc exec.

So is this not a different problem altogether?

candlerb picture candlerb on 22 Feb 2017
👍1

Using "lxc exec" and then running a "script" session should have the same effect as ssh

This does not work either:

~
script foo
tcpdump -i eth0 -nn
~

... whether I first ssh into the container and sudo, or I lxc exec into the container.

candlerb picture candlerb on 22 Feb 2017

@candlerb hmm, sorry, I missed that. Yes, that would be a different apparmor bug then...

Can you file another issue at https://launchpad.net/ubuntu/+source/apparmor/+filebug
Mentioning that you tried ssh and script so it shouldn't be the inherited pts issue.

I'm assuming disabling that apparmor profile does fix the issue for you though, right?

stgraber picture stgraber on 22 Feb 2017

Yes, this does fix it, thank you:

~~~

ln -s /etc/apparmor.d/usr.sbin.tcpdump /etc/apparmor.d/disable/

apparmor_parser -R /etc/apparmor.d/disable/

~~~

candlerb picture candlerb on 22 Feb 2017
👍2

Cool. If you do file a bug on Launchpad, please post the link here and I'll subscribe the LXD team to it.

stgraber picture stgraber on 22 Feb 2017

https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1667016

candlerb picture candlerb on 22 Feb 2017

Just wanted to add that:

  • this problem still exists on Ubuntu 18.04
  • after removing the apparmor profile restarting the container helps (obvious if you know apparmor I guess)
  • tshark works out of the box.
gpatel-fr picture gpatel-fr on 21 Feb 2019

This problem persists on Ubuntu 19.04

johnbchron picture johnbchron on 13 Jun 2019

@stgraber: you appear to have marked this issue LP#1667016 as a duplicate of LP#1641236 - when you agreed earlier it was a different issue and asked me to open a separate bug report

candlerb picture candlerb on 13 Jun 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

spacekookie picture spacekookie  路  3Comments
chuegel picture chuegel  路  3Comments
killua-eu picture killua-eu  路  3Comments
tebanep picture tebanep  路  5Comments
srkunze picture srkunze  路  3Comments