Diagnostics: C# service not able to produce runtime coredumps from unhandled exception, if the container runs as non-root

[shivgu]

The C# service is not able to produce coredumps(generated by runtime) if the container is running with non-root user id.

Configuration

Os and version info - Linux prober-prober-784866ff9c-9nx4j 4.15.0-1091-azure #101~16.04.1-Ubuntu SMP Fri Jun 26 01:14:12 UTC 2020 x86_64 GNU/Linux

deployment.yaml file contains:

containers:
        securityContext:
          capabilities:
            add:
            - SYS_PTRACE
          runAsUser: 300
        env:
          - name: COMPlus_DbgEnableMiniDump
            value: "1"
          - name: COMPlus_DbgMiniDumpType
            value: "4"
          - name: COMPlus_DbgMiniDumpName
            value: "/cores/prober-core-%d"

[wfurt]

did you check /proc/sys/kernel/yama/ptrace_scope ?

[shirhatti]

We checked /etc/sysctl.d/10-ptrace.conf on the host.
kernel.yama.ptrace_scope = 1

[wfurt]

can you try it with 0? https://www.kernel.org/doc/Documentation/security/Yama.txt
(you can also just change that runtime once without reboot and modifying ptrace.conf)

[shirhatti]

@wfurt What's your hypothesis? I don't have an environment handy to test so we'll need to wait for @shivgu

The runtime does call prctl(PR_SET_PTRACER, ...) so it's fair assumption Yama isn't the issue here. Moreso, because the same things works outside of a container with ptrace_scope = 1

[wfurt]

I've seen cases in the past when given user would not have permission to trace own processes. (without the "predefined relationship") So I'm wondering if that could be related. Maybe somebody from the diag team will have better idea.

[mikem8361]

@dotnet/dotnet-diag

Does anybody have any ideas about this docker/createdump issue? I don't have the docker knowledge to help with any answers here.

[shivgu]

@wfurt I tried turning kernel.yama.ptrace_scope=0, but still I dont see the coredumps.

[sdmaclea]

You could also check user 300's write permissions in /cores, It sounds like a path which might be owned by root.

[shivgu]

@sdmaclea When I start the container as user 300, I see

drwxrwx--- 2 root root 4096 Oct 29 16:58 cores

[shivgu]

do we need to change it?

[sdmaclea]

So user 300 doesn't have write permissions to that directory (note the --- for user other). So you can either:

• give user 300 write permissions by chmod o+w /cores
• add user 300 to the root group
• chose to write to a different directory (where user 300 has write permissions).

I suspect you want the last one.

[shivgu]

Hi @sdmaclea I tried making a new directory within dockerfile RUN mkdir -p /Data && chmod o+w /Data.

I could make the /Data folder and the permissions on it are
drwxr-xr-x 2 root root 240 Oct 28 23:41 Data

I suppose the "w" flag should have been added to user other using this, Right? or am I missing something?

[sdmaclea]

Yeah I don't know why the permissions don't include w. I would have expected permissions drwxr-xrwx at least.
Perhaps the conditional didn't execute. Try just running chmod o+w /Data

[shivgu]

I am not able to change the permissions of a directory from within the container(running with user id 300), that is why I tried doing it from dockerfile.

Running it from container gives me:

chmod o+w /Data
chmod: changing permissions of '/Data': Operation not permitted

[shivgu]

I ran
RUN mkdir -p /Data

RUN chmod o+w /Data

and nothing changed
drwxr-xr-x 2 root root 240 Oct 28 23:41 Data

[hoyosjs]

You can try
RUN mkdir -p /Data -m762

So the user 300 can write but not read or execute from that directory.

[shirhatti]

I am not able to change the permissions of a directory from within the container(running with user id 300), that is why I tried doing it from dockerfile.

Is the directory a bind mount?

[shivgu]

I am not aware whats a bind mount, but does something like this counts as bind mount?

volumeMounts:
- name: core-dump
mountPath: /Data
readOnly: false

volumes:
    - name: core-dump
      hostPath:
        path: /var/log/core-dump

[shivgu]

okay so I changed the permissions for /var/log/core-dump on the host itself and it reflected in /cores.

drwxrwxrwx 4 root root 4096 Oct 29 19:00 cores

but still the core-dumps are not getting generated.

[mikem8361]

I think we got sidetracked here with the permissions of the directory the core dump is written too. The actual error (which isn't in the issue) is:

open(/proc/1/mem) FAILED 13 (Permission denied)

This is in createdump itself when it opens the special Linux file for reading the target runtime process memory. When createdump is launched by the runtime it does a prctl(PR_SET_PTRACER, childpid, ... call that outside of a docker container and any container I've created allows that open to succeed without requiring the runtime be running as root.

I don't have any answers yet.

[shivgu]

Hi, do we have any update here? or anything we can try?

[mikem8361]

I don't have anything. I have no idea how to setup the docker container that shows this problem. Maybe there is another capability that enables the special mem file support.

/cc: @dotnet/dotnet-diag

[shivgu]

This is a pressing issue right now for us, as this issue is blocking us from a major project delivery. I am also available for a quick call in case, someone wants to debug in the actual environment. Please let me know.

[shivgu]

Also, The container which is not able to get the dumps has

dotnet --list-runtimes
Microsoft.AspNetCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Could the runtime version be responsible for not getting the dumps?

[mikem8361]

I think this is more about the docker configuration than the runtime version, but I don't have any answers or workarounds.

[hoyosjs]

We are quickly looking at some stuff. ~This is not dotnet, as the user itself can't read procfs. Trying to figure out the capability needed to allow the reads to happen.~ turns out it can read

[hoyosjs]

This issue was resolved directly with the customer.