Podman: failed to write to /proc/self/oom_score_adj: Permission denied

Created on 26 Apr 2019 · 28Comments · Source: containers/podman

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running rootless podman I'm getting failed to write to /proc/self/oom_score_adj: Permission denied error, but container runs after it.

Steps to reproduce the issue:

podman run -it --rm anycontainer

Describe the results you received:
The container runs, but after I get an error message in the terminal

Describe the results you expected:
The container should start without error message.

Additional information you deem important (e.g. issue happens only occasionally):
Error message: failed to write to /proc/self/oom_score_adj: Permission denied
Output of podman version:

podman version 1.2.0

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.11.6
  podman version: 1.2.0
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2.0-1.1.x86_64
    path: /usr/lib/podman/bin/conmon
    version: "failed to write to /proc/self/oom_score_adj: Permission denied, conmon
      version 1.14.0\ncommit: "
  Distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20190423"
  MemFree: 2402082816
  MemTotal: 11992973312
  OCIRuntime:
    package: runc-1.0.0~rc6-3.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      spec: 1.0.1-dev
  SwapFree: 2145845248
  SwapTotal: 2147479552
  arch: amd64
  cpus: 4
  hostname: kraken
  kernel: 5.0.8-1-default
  os: linux
  rootless: true
  uptime: 4h 43m 30.3s (Approximately 0.17 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ConfigFile: /home/dario/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/dario/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 20
  RunRoot: /tmp/1000
  VolumePath: /home/dario/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

kinbug

Source

dac73

Most helpful comment

My laptop updated cri-o this morning, 1.13.5 to 1.13.7, and the oom_score_adj warning is gone.

edsantiago on 2 May 2019

🎉2

All 28 comments

Rootless?
Is this an SELinux issue? Does it work in permissive mode?

rhatdan on 26 Apr 2019

Not an SELinux issue: I'm seeing it on my Gentoo laptop, which (I know, I know!) does not have SELinux enabled.

edsantiago on 26 Apr 2019

It's rootless.

dac73 on 26 Apr 2019

I tried on Fedora and it is allowed.
Could you try in --privileged and see if it allowed.

rhatdan on 26 Apr 2019

It it works with --privileged then try without --privileged and try
--security-opt seccomp=unconfined

rhatdan on 26 Apr 2019

Same error for both options.

dac73 on 26 Apr 2019

Same here (i.e. --privileged and --security-opt seccomp=unconfined make no difference)

edsantiago on 26 Apr 2019

I bet this is Conmon. We put an OOM adjust in there that isn't fatal if it
fails, but for some time it logged errors every time it did. I believe it's
since been changed to do so only with debug level logging.

On Fri, Apr 26, 2019, 09:19 dac[73] notifications@github.com wrote:

Same error for both options.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/3024#issuecomment-487053694,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB3AOCHFQQBJMJVN4ULDB6LPSL6MXANCNFSM4HIVPOHA
.

mheon on 26 Apr 2019

@mheon, do you know when it has been changed? The podman v1.2.0 package in openSUSE is using conmon from CRI-O 1.14.0.

vrothberg on 26 Apr 2019

I'm pretty sure @haircommander made the change in question, so I'll tag him in for that one

mheon on 26 Apr 2019

this is conmon, I didn't know any podman was shipping with that updated of a CRI-O version. We expect this debug message, though I actually think conmon's log handling is weird at the moment. (note, the change I made just made this error non fatal)

haircommander on 26 Apr 2019

I feel like this should just silently happen tbh

haircommander on 26 Apr 2019

Looking forward to c/conmon kicking off :) @sysrich, I think the openSUSE could downgrade to an older conmon for podman to quick-fix the issue.

vrothberg on 26 Apr 2019

@sysrich @vrothberg release 1.13.3 should not have this problem

haircommander on 26 Apr 2019

ah actually, we did fix this problem upstream, but haven't cut a new release on 1.14 with the updates yet. I will look into it, but for the time being I'd go back to 1.13.3 so users don't think rootless is failing

haircommander on 26 Apr 2019

My laptop updated cri-o this morning, 1.13.5 to 1.13.7, and the oom_score_adj warning is gone.

edsantiago on 2 May 2019

🎉2

are we good to close this?

baude on 29 May 2019

yes--people will need to use containers/conmon 0.2.0

haircommander on 29 May 2019

This was fixed for me with the last TW update.
Conmon is @0.2.0
Thanks.

dac73 on 3 Jun 2019

Problem returned in rootless :(

Describe the results you received:

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

Output of podman version:

Version:            1.6.3-dev
RemoteAPI Version:  1
Go Version:         go1.11.5
OS/Arch:            linux/amd64

Output of podman info --debug:

Version:            1.6.3-dev
RemoteAPI Version:  1
Go Version:         go1.11.5
OS/Arch:            linux/amd64
tobwen@pgsql:~/podman/usr/local/bin$ ./podman --tmpdir /tmp/user/1000/libpod/tmp info debug
Error: `podman system info` takes no arguments
tobwen@pgsql:~/podman/usr/local/bin$ ./podman --tmpdir /tmp/user/1000/libpod/tmp info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.11.5
  podman version: 1.6.3-dev
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /home/tobwen/podman/usr/local/bin/conmon
    version: 'conmon version 2.0.3-dev, commit: bc758d8bd98a29ac3aa4f62a886575bfec0e39a1'
  Distribution:
    distribution: debian
    version: "9"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    - container_id: 65537
      host_id: 1258512
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    - container_id: 65537
      host_id: 1258512
      size: 65536
  MemFree: 35587067904
  MemTotal: 38205444096
  OCIRuntime:
    name: runc
    package: Unknown
    path: /home/tobwen/podman/usr/local/bin/runc
    version: |-
      runc version 1.0.0-rc9+dev
      commit: 4e3701702e966b4258fbab5b92efa6418c5ae6c6
      spec: 1.0.1-dev
  SwapFree: 8586784768
  SwapTotal: 8586784768
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: pgsql
  kernel: 4.19.0-0.bpo.6-amd64
  os: linux
  rootless: true
  uptime: 29h 19m 13.88s (Approximately 1.21 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
store:
  ConfigFile: /home/tobwen/.config/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: vfs
  GraphOptions: {}
  GraphRoot: /home/tobwen/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 1
  RunRoot: /tmp/user/1000
  VolumePath: /home/tobwen/.local/share/containers/storage/volumes

tobwen on 19 Oct 2019

@tobwen what command were you executing when you saw this error? If you turn on debug logging you will still see it.

rhatdan on 20 Oct 2019

Due to bugs with the local config, I have to use this commandline

/home/tobwen/podman/usr/local/bin/podman --log-level=debug \
--tmpdir /tmp/user/1000/libpod/tmp \
--conmon /home/tobwen/podman/usr/local/bin/conmon \
--network-cmd-path /home/tobwen/podman/usr/local/bin/slirp4netns \
--runtime /home/tobwen/podman/usr/local/bin/runc \
--storage-driver overlay \
--storage-opt "overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs"

When not using _debug logging_, I don't see it.

But that's interesting:

$ ls -al /proc/self/oom_score_adj
-rw-r--r-- 1 tobwen tobwen 0 Oct 20 12:43 /proc/self/oom_score_adj

This file seems to belong to the current use. Or does _conmon_ try to read/write from within a namespace?

tobwen on 20 Oct 2019

It might be happening within a user namespace.

Both of these are successful

$ echo 1 > /proc/self/oom_score_adj 
$ podman unshare echo 1 > /proc/self/oom_score_adj 
$ podman run fedora echo 1 > /proc/self/oom_score_adj

All of these work.  But

echo -1 > /proc/self/oom_score_adj 
bash: echo: write error: Permission denied

rhatdan on 21 Oct 2019

Conmon is attempting to set this score on rootless which is not allowed.

conmon.c:#define OOM_SCORE "-999"
conmon.c:       if (write(oom_score_fd, OOM_SCORE, strlen(OOM_SCORE)) < 0) {

rhatdan on 21 Oct 2019

I'm trying to get toolbox running in silverblue 31. It fails with:

Error: unable to start container "fedora-toolbox-31": writing file '/proc/46288/gid_map': Operation not permitted
setgid(0): Invalid argument: OCI runtime permission denied error
toolbox: failed to start container fedora-toolbox-31

Elsewhere, I saw a suggestion about running with: systemd-run --scope --user podman --log-level debug start fedora-toolbox-31. When I do that, I get an error like I'm seeing in here:

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

jasonbrooks on 31 Oct 2019

@jasonbrooks that message is expected, it won't be printed without --log-level debug

haircommander on 31 Oct 2019

@haircommander Ah, so not related to my toolbox issue

jasonbrooks on 31 Oct 2019

that is correct :slightly_smiling_face:

haircommander on 31 Oct 2019

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Hitting "Error committing the finished image" "no space left on device" due to nonoptimized use of disk

oblitum · 5Comments

podman generate systemd (systemd-user) service enabled but can't autostart

junjieyuan · 4Comments

podman logs crash

yangm97 · 5Comments

export checkpoints of redis container failed

yufeifly · 4Comments

Cant run MongoDB image from dockerhub. getting permission denied on entrypoint shell script

NooneXe · 3Comments