Buildkit: Implement entitlement security.unconfined

[andrewrynhard]

I would like to run privileged commands in my builds. I saw the --allow security.unconfined flag, but seeing the following when using it:

2019/04/19 05:28:51 http2: server connection error from localhost: connection error: PROTOCOL_ERROR
error: failed to solve: rpc error: code = Unknown desc = entitlement security.unconfined is not allowed
Makefile:158: recipe for target 'test' failed
make: *** [test] Error 1

[andrewrynhard]

@AkihiroSuda Are there any blockers to starting work on this? I wouldn't mind contributing.

[AkihiroSuda]

cc @kunalkushwaha

[kunalkushwaha]

@andrewrynhard please try security.insecure instead of security.unconfined

[andrewrynhard]

@kunalkushwaha

error: unknown entitlement security.insecure

Not in the latest release?

[kunalkushwaha]

is buildkitd running with --allow-insecure-entitlement security.insecure. By default it is disabled.

This feature is available with master branch. https://github.com/moby/buildkit/pull/570/

[andrewrynhard]

I am not. Looks like I need to build from master

[kunalkushwaha]

Also, No frontend support is added to it yet, So using dockerfile, this cannot be used at this moment.

[andrewrynhard]

Also, No frontend support is added to it yet, So using dockerfile, this cannot be used at this moment.

Are there examples of other ways I can use it?

[andrewrynhard]

And is there a timeline on when I can use it with the docker frontend?

[kunalkushwaha]

@andrewrynhard you can see an example here https://gist.github.com/kunalkushwaha/775b063671a62f968ff111d266764d04

Also, you can refer to tests https://github.com/moby/buildkit/blob/master/client/client_test.go#L439

And is there a timeline on when I can use it with the docker frontend?

@tonistiigi @AkihiroSuda can be in better people to tell this, since it requires to add New Dockerfile command

[andrewrynhard]

@kunalkushwaha I’d be happy to implement this for the docker frontend, but I’m afraid I’m not familiar with the code base yet. If you can give me a brief outline of the steps needed for the docker frontend, I could give it a shot.

EDIT: I posted this just after your response. Thanks for the help!

[tonistiigi]

We first need a design proposal and to agree on the syntax for the Dockerfile. (out of tree you can experiment on your own of course).

For examples, it probably makes sense to look up PRs that added things like mounts, secrets or ssh to dockerfile frontend. Eg. https://github.com/moby/buildkit/pull/655/files

Also, I believe there isn't currently a way to enable the daemon level allow rules in Moby. It might be good to push that so it may still get into 19.03 (cc @kunalkushwaha )

[kunalkushwaha]

Also, I believe there isn't currently a way to enable the daemon level allow rules in Moby. It might be >good to push that so it may still get into 19.03 (cc @kunalkushwaha )

@tonistiigi currently EntilementNetworkHost is enabled by default for builder in Moby. Which may not be great idea for enabling SecurityInsecure entitlement.

We can add allow-insecure-entitlement in BuilderConfig, which can read from docker daemon config. Will NetworkHost should also be configurable like that?

Please share views.

//cc @tiborvass

[tonistiigi]

If we don't add EntilementNetworkHost automatically we would break the old --net=host behavior (that wasn't a very brilliant idea when it was added). In some way, it would make sense to just enable securityinsecure as well(old builder allows disabling security profiles from client for example) but I guess we could use config to be on the safe side.

Maybe add key: bool in the config. Net host can be on by default but the user can set it to false if they want.

[andrewrynhard]

I'm a little confused about the release notes: https://github.com/moby/buildkit/releases/tag/v0.5.0

It mentions that security.insecure is implemented, but I have started buildkit with --allow-insecure-entitlement security.insecure, and tried to run a build that uses the mount command, but it I get operations not permitted.

Here is the buildctl command I am using:

buildctl build --opt target=test --frontend=dockerfile.v0 --local context=. --local dockerfile=. --allow security.insecure

If I understand the discussion above, it should _not_ work with the docker's buildkit feature, but _should_ in my case?

[kunalkushwaha]

buildctl build --opt target=test --frontend=dockerfile.v0 --local context=. --local dockerfile=. --allow security.insecure

@andrewrynhard This command uses the frontend as dockerfile, where support for entitlements are not yet supported in frontend. As mentioned earlier, this is only at LLB level right now.

We need to propose/add support for Dockerfile frontend something like

RUN ents=security.insecure  < command to execute in privileged container/mode>

[andrewrynhard]

Friendly ping to see if this is planned for being done soon?

[ShemTovYosef]

I found bug that images are corrupted when using this feature: owners of some files by mistake specified with other user and application is not running

I tried with the new feature of buildx:
RUN --security=unsecure hostname NEW_HOST
which works good in build time but while running container I found that image is corrupted, applications is not running, their user/group owners are wrong. Look likes bugs.
I'm using the latest docker on CentOS Linux release 7.7.1908 (Core) with the kernel 5.4.6-1.el7.elrepo.x86_64:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1695
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.6-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.64GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[AkihiroSuda]

Please open a new issue with ls -l ?