Gitpod: GDB isn't working

[anudeepreddy]

1) contents of dockerfile:

FROM gitpod/workspace-full:latest

USER root
RUN apt-get update && apt-get install -y \
gdb \
lldb \
fuse libfuse-dev\
&& apt-get clean && rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/* && rm -rf /tmp/* && docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined

2) contents of gitpod.yml

image:
file: Dockerfile
ports:

• port: 3000
  tasks:
  
  
  
  • init: yarn
  
  

Error:

warning: Error disabling address space randomization: Operation not permitted
warning: Could not trace the inferior process.
Error:
warning: ptrace: Operation not permitted
During startup program exited with code 127

[jankeromnes]

Hi @anudeepreddy, thanks a lot for reporting this issue!

I'd love to get gdb to work on Gitpod (and rr as well). Right now, these debuggers don't work, because ptrace is disabled in Docker and Kubernetes due to security concerns (see below for details).

I suspect that it's now safe to enable ptrace again, because the original security bug was fixed in Linux 4.8, and Gitpod seems to be using 4.15.

To allow ptrace in Gitpod workspaces, we could:

• Enable CAP_SYS_PTRACE in all Gitpod containers
• Make sure the ptrace syscall is allowed (it used to be blacklisted in Docker's default seccomp profile, for Kubernetes I don't know)
• ... and maybe the perf_event_open syscall too

(Source.)

More details:

warning: ptrace: Operation not permitted

By default, ptrace is blocked in Docker and Kubernetes. Originally, this was because of a security bug allowing people to abuse ptrace to escape out of containers into the host system. This bug was fixed in Linux kernel 4.8, so ptrace is now thought to be safe, but since it's a complicated API with a large attack surface, it remains disabled by default just to be safe. (Source.)

warning: Error disabling address space randomization: Operation not permitted

This is just a minor warning, which shouldn't prevent gdb from working. (Source)

[jankeromnes]

@geropl also raised two additional (Gitpod-specific) security concerns with enabling ptrace:

Is it possible to ptrace processes from the same user in other containers?

Are there exploitation vectors because we mount the same host filesystem into multiple containers?

[jankeromnes]

I just noticed that your dockerfile includes this command:

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined

This won't work in a dockerfile, and there is currently no way for users to configure Gitpod's container runtime options (for example we don't want to allow CAP_SYS_ADMIN at this time, in order to protect other users and our host systems).

However, as mentioned above, we could consider adding CAP_SYS_PTRACE by default, and allow syscalls like ptrace and perf_event_open, provided this doesn't put Gitpod users or servers at risk.

[ckoncz-hwx]

Hi, is GDB supposed to work as of now?

I checked it with this executable
https://github.com/ckoncz-hwx/commit-graph/blob/gitpod-nasm/nasm/Makefile#L3
but had no luck:

(gdb) break main
Breakpoint 1 at 0x401110: file hello.asm, line 8.
(gdb) r
Starting program: /workspace/commit-graph/nasm/hello 
warning: Error disabling address space randomization: Operation not permitted
warning: Could not trace the inferior process.
Error: 
warning: ptrace: Operation not permitted
During startup program exited with code 127.

[jankeromnes]

Hi @ckoncz-hwx, thanks for giving gdb a spin! As you've seen, it's now installed in Gitpod workspaces, but it's still missing the ability to use ptrace.

ptrace used to be blocked by two safety mechanisms in Docker (syscall blocked by seccomp, and privilege restricted by CAP_SYS_PTRACE).

The first blocker was removed by Docker on recent Linux Kernels, but the second one we still need to address in Gitpod, by granting the CAP_SYS_PTRACE capability to all workspaces, which has some security implications but should now be "safe enough" for us to try.

I'll try to add the SYS_PTRACE capability in Gitpod as a prototype, and update this issues with any findings.

[RossComputerGuy]

Is there a way to have the IDE debugger to work with binary programs, I think adding launch configurations for binary programs would be useful.

[weliveindetail]

Any updates regarding SYS_PTRACE? I understand the security considerations, but it would be great to have LLDB working for my project :) Also, if there's a chance it becomes possible in the future, I'd like to make a workspace for the LLVM Kaleidoscope tutorials in GitPod:
https://llvm.org/docs/tutorial/MyFirstLanguageFrontend/index.html

[jankeromnes]

Good news! We've decided to give enabling ptrace a go, in the hope to get gdb/lldb/rr debugging to work on Gitpod.

We'll start by enabling ptrace in Gitpod's staging environment, and I'll update this issue with any findings or plans to ship to production.

[anudeepreddy]

That's super amazing

[ckoncz-hwx]

The issue persists even after creating a new workspace.

(gdb) r
Starting program: /workspace/commit-graph/nasm/hello 
warning: Error disabling address space randomization: Operation not permitted
warning: Could not trace the inferior process.
Error: 
warning: ptrace: Operation not permitted
During startup program exited with code 127.
(gdb) 

Maybe the change is not deployed to production yet?

[JesterOrNot]

@jankeromnes it seems that the issue still persists

[jankeromnes]

Indeed, sorry for accidentally closing this issue too soon. We've deployed a fix in our staging environment to test it.

[weliveindetail]

Is there a way to test my workspace in the staging environment?

[jankeromnes]

Is there a way to test my workspace in the staging environment?

@weliveindetail Thank you for offering to test this. Unfortunately, our staging environment isn't public because it sees frequent, experimental deployments which often aren't ready or fully tested yet. However, we're planning to release this feature later this month. I don't have a precise ETA yet, but I'll close this issue when it goes live.

[svenefftinge]

deployed

[ckoncz-hwx]

Had a chance to test it now. GDB still outputs a scary warning, but debugging works!

(gdb) r
Starting program: /workspace/commit-graph/nasm/hello 
warning: Error disabling address space randomization: Operation not permitted

Breakpoint 1, main () at hello.asm:8
8               mov rax, 1 ; 1 = write
(gdb) c
Continuing.
hello, world[Inferior 1 (process 2156) exited normally]
(gdb) 

Thank you!

[jankeromnes]

Awesome! Thanks for the update @ckoncz-hwx 👍

GDB still outputs a scary warning, but debugging works!

Yes, the "warning: Error disabling address space randomization: Operation not permitted" message looks a bit worrying, but you can safely ignore it.

FYI, it's because GDB tries to make a few things more stable by default, because it makes some sorts of debugging a bit easier, but GDB also works really well without that. You can disable this feature with set disable-randomization off, see: https://stackoverflow.com/a/35860616/3461173