Go: os/user: LookupId panics on Linux+glibc static build

Created on 10 Apr 2018  ·  14Comments  ·  Source: golang/go

Since Go 1.10 beta, using tar.FileInfoHeader() is not possible from a static binary compiled on Linux/glibc. This is caused by commit 0564e304a6ea394a4 ("archive/tar: populate uname/gname/devmajor/devminor in FileInfoHeader") which makes use of os/user functions LookupId and LookupGroupId, which, in turn, use glibc's getpw* and getgrp* calls (in case CGO is available).

This breaks both Docker (https://github.com/moby/moby/pull/35739#issuecomment-379346230) and containerd.

This issue is a manifestation of a problem described in more details at https://github.com/golang/go/issues/23265 and remedied by commit 62f0127d8104d8266d9a3fb5a87e2f09ec8b6f5b ("os/user: add a way to enforce pure Go implementation"). Unfortunately, the remedy will only be available in Go 1.11, and requires setting a osusergo build tag for static build.

PS this report is serving merely to document the issue and maybe help others who see the same issue. I do not expect this to be fixed in Go 1.10. As a workaround, a fork of archive/tar is created with the partial revert of 0564e304a6e here: https://github.com/kolyshkin/go-tar/tree/go-1.10

What version of Go are you using (go version)?

1.10.1

Does this issue reproduce with the latest release?

yes (1.10.1)

What operating system and processor architecture are you using (go env)?

Linux/amd64

What did you do?

Switched to Go 1.10 to compile Docker (https://github.com/moby/moby/pull/35739).

What did you expect to see?

no panic

What did you see instead?

panic while running CI

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7fa8342d77f8]

runtime stack:
runtime.throw(0x183d93d, 0x2a)
        /usr/local/go/src/runtime/panic.go:616 +0x83
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:372 +0x292

goroutine 192 [syscall]:
runtime.cgocall(0x1748570, 0xc42095b5a8, 0x1839600)
        /usr/local/go/src/runtime/cgocall.go:128 +0x66 fp=0xc42095b560 sp=0xc42095b528 pc=0x402236
os/user._Cfunc_mygetpwuid_r(0x3, 0xc4207f94a0, 0x34734c0, 0x400, 0xc4200c4578, 0x0)
        _cgo_gotypes.go:170 +0x4f fp=0xc42095b5a8 sp=0xc42095b560 pc=0x97257f
os/user.lookupUnixUid.func1.1(0x7fa800000003, 0xc4207f94a0, 0x34734c0, 0x400, 0xc4200c4578, 0xc42095b638)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x141 fp=0xc42095b5e8 sp=0xc42095b5a8 pc=0x9741c1
os/user.lookupUnixUid.func1(0x10)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x52 fp=0xc42095b628 sp=0xc42095b5e8 pc=0x974262
os/user.retryWithBuffer(0xc420b01c60, 0xc42095b718, 0xc420b01c60, 0x2199980)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:253 +0x3d fp=0xc42095b688 sp=0xc42095b628 pc=0x97366d
os/user.lookupUnixUid(0x3, 0x0, 0x0, 0x0)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc42095b750 sp=0xc42095b688 pc=0x972ac2
os/user.lookupUserId(0x187f9c0, 0x1, 0x1, 0x47df00, 0x3)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:86 +0x75 fp=0xc42095b788 sp=0xc42095b750 pc=0x972955
os/user.LookupId(0x187f9c0, 0x1, 0x1, 0x0, 0x0)
        /usr/local/go/src/os/user/lookup.go:41 +0x53 fp=0xc42095b7d8 sp=0xc42095b788 pc=0x971f23
archive/tar.statUnix(0x230c640, 0xc4213e45b0, 0xc420ede2a0, 0x17ecf6e, 0x1)
        /usr/local/go/src/archive/tar/stat_unix.go:39 +0x4c3 fp=0xc42095b858 sp=0xc42095b7d8 pc=0x97dd33
archive/tar.FileInfoHeader(0x230c640, 0xc4213e45b0, 0x0, 0x0, 0xc420559360, 0x99, 0x230c640)
        /usr/local/go/src/archive/tar/common.go:699 +0x493 fp=0xc42095b9d8 sp=0xc42095b858 pc=0x9771b3
github.com/docker/docker/pkg/archive.FileInfoHeader(0xc4205593f6, 0x3, 0x230c640, 0xc4213e45b0, 0x0, 0x0, 0x7412596f, 0xf943ee04c5fd7027, 0x0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:360 +0x55 fp=0xc42095ba38 sp=0xc42095b9d8 pc=0xd73705
github.com/docker/docker/pkg/archive.(*tarAppender).addTarFile(0xc4211b70c0, 0xc420559360, 0x99, 0xc4205593f6, 0x3, 0x0, 0x0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:478 +0xd9 fp=0xc42095bad0 sp=0xc42095ba38 pc=0xd73f99
github.com/docker/docker/pkg/archive.TarWithOptions.func1.2(0xc420559360, 0x99, 0x230c640, 0xc4213e44e0, 0x0, 0x0, 0x0, 0xc42095bd28)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:888 +0x612 fp=0xc42095bca0 sp=0xc42095bad0 pc=0xd7ed42
path/filepath.walk(0xc420559360, 0x99, 0x230c640, 0xc4213e44e0, 0xc4200c74f0, 0x0, 0x0)
        /usr/local/go/src/path/filepath/path.go:361 +0xe7 fp=0xc42095bd78 sp=0xc42095bca0 pc=0x50fd17
path/filepath.walk(0xc4213e79a0, 0x97, 0x230c640, 0xc420eac8f0, 0xc4200c74f0, 0x0, 0x50)
        /usr/local/go/src/path/filepath/path.go:381 +0x2c4 fp=0xc42095be50 sp=0xc42095bd78 pc=0x50fef4
path/filepath.Walk(0xc4213e79a0, 0x97, 0xc4200c74f0, 0x17ecf6e, 0x1)
        /usr/local/go/src/path/filepath/path.go:403 +0x108 fp=0xc42095beb0 sp=0xc42095be50 pc=0x510178
github.com/docker/docker/pkg/archive.TarWithOptions.func1(0xc4202a25a0, 0x22eb160, 0xc4203cbec0, 0xc4200c5e48, 0xc4200abfb0, 0xc4203cbea0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:806 +0x2ee fp=0xc42095bfb0 sp=0xc42095beb0 pc=0xd7f42e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42095bfb8 sp=0xc42095bfb0 pc=0x45bc71
created by github.com/docker/docker/pkg/archive.TarWithOptions
        /go/src/github.com/docker/docker/pkg/archive/archive.go:748 +0x26f
FrozenDueToAge
Source
picture kolyshkin
👍8

Most helpful comment

The alternative seems to be back-porting #23265 to Go1.10.2 (if we have one)

As this looks to be the only alternative, please let me know if you want me to do a packport.

picture kolyshkin on 13 Apr 2018
❤1 🎉1 👍1

All 14 comments

This isn't really tar's fault per-say. The underlying issue is os/user, which shouldn't panic in the first place. However, you seem to indicate that os/user is already fixed. Is there anything to do here then?

dsnet picture dsnet on 10 Apr 2018

However, you seem to indicate that os/user is already fixed. Is there anything to do here then?

It is only fixed in Go 1.11 which is not here yet, and the archive/tar started to use os/user in 1.10. So, 1.11 should be fine, and 1.9 was fine, but Go 1.10 is broken in this very case.

kolyshkin picture kolyshkin on 10 Apr 2018

It is unfortunate that #23265 is only for Go1.11. We can't exactly revert the archive/tar change in the next Go1.10 point release, as that is too large of a change. The alternative seems to be back-porting #23265 to Go1.10.2 (if we have one).

dsnet picture dsnet on 10 Apr 2018

The alternative seems to be back-porting #23265 to Go1.10.2 (if we have one)

As this looks to be the only alternative, please let me know if you want me to do a packport.

kolyshkin picture kolyshkin on 13 Apr 2018
❤1 🎉1 👍1

@kolyshkin The CL to be backported would be https://golang.org/cl/92456, correct?

@ianlancetaylor any objection to cherry-picking?

andybons picture andybons on 27 Apr 2018

We can't backport that CL by itself, because it wasn't tested and didn't work (#24841, #24845). It would need at least also CLs 106837, 107299, 107304. And in my opinion the sequence of problems and failed fixes means that the risk is too high for a backport to a release branch. Not to mention the fact that this requires the user to understand the problem--it's really not obvious that a crash while using archive/tar should be fixed by using -tags osusergo.

In my opinion the latter problem means that there is still a problem to fix here. The osusergo tag is useful by itself, but as far as this bug is concerned it's only a workaround.

ianlancetaylor picture ianlancetaylor on 27 Apr 2018

I actually don't understand where this crash is coming from, and I don't understand how to recreate it. @kolyshkin Do you have step by step instructions I can use to create the problem?

ianlancetaylor picture ianlancetaylor on 27 Apr 2018

Do you have step by step instructions I can use to create the problem?

@ianlancetaylor unfortunately not. Before filing this bug I spent some time trying to create a simple reproducer but failed. Perhaps it requires some specific version of glibc, or there is something else absent from the simple reproducers I tried. What we see in docker CI is here https://github.com/moby/moby/pull/35739#issuecomment-379371309

Let me try again to create a repro

kolyshkin picture kolyshkin on 4 May 2018

It looks like the reason of segfault is indeed a glibc bug described here: https://github.com/golang/go/issues/13470

kolyshkin picture kolyshkin on 9 May 2018

and here's a reproducer, modeled after the one in C from https://github.com/golang/go/issues/13470#issuecomment-162613860

package main

import (
    "fmt"
    "os/user"
)

func main() {
    id := "3"
    wait := make(chan bool)
    ret := make(chan *user.User)
    go func() {
        <-wait
        u, _ := user.LookupId(id)
        ret <- u
    }()
    u, _ := user.LookupId(id)
    wait <- true
    fmt.Printf("%+v\n", u)
    u = <-ret
    fmt.Printf("%+v\n", u)
}

When compiled statically (i.e. go build -ldflags '-extldflags "-fno-PIC -static"' -buildmode pie), it crashes on my machine (Ubuntu 17.10, amd64, go-1.10, glibc 2.26):

&{Uid:3 Gid:3 Username:sys Name:sys HomeDir:/dev}
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7ff160f4f698]

runtime stack:
runtime.throw(0x53c045, 0x2a)
    /usr/lib/go-1.10/src/runtime/panic.go:619 +0x83
runtime.sigpanic()
    /usr/lib/go-1.10/src/runtime/signal_unix.go:372 +0x292

goroutine 5 [syscall]:
runtime.cgocall(0x48a4d0, 0xc420040d50, 0x53bdad)
    /usr/lib/go-1.10/src/runtime/cgocall.go:128 +0x66 fp=0xc420040d08 sp=0xc420040cd0 pc=0x402266
os/user._Cfunc_mygetpwuid_r(0x3, 0xc42009c000, 0x7ff14c000b10, 0x400, 0xc42009e000, 0x0)
    _cgo_gotypes.go:170 +0x4f fp=0xc420040d50 sp=0xc420040d08 pc=0x488a1f
os/user.lookupUnixUid.func1.1(0x3, 0xc42009c000, 0x7ff14c000b10, 0x400, 0xc42009e000, 0xc420035de0)
    /usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:100 +0x141 fp=0xc420040d90 sp=0xc420040d50 pc=0x489921
os/user.lookupUnixUid.func1(0xc420035db0)
    /usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:100 +0x52 fp=0xc420040dd0 sp=0xc420040d90 pc=0x4899c2
os/user.retryWithBuffer(0xc4200a0000, 0xc420040ec0, 0xc4200a0000, 0x7aa060)
    /usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:253 +0x3d fp=0xc420040e30 sp=0xc420040dd0 pc=0x48963d
os/user.lookupUnixUid(0x3, 0x0, 0x0, 0x0)
    /usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc420040ef8 sp=0xc420040e30 pc=0x488f62
os/user.lookupUserId(0x53614a, 0x1, 0x0, 0x404e4b, 0xc42007c060)
    /usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:86 +0x75 fp=0xc420040f30 sp=0xc420040ef8 pc=0x488df5
os/user.LookupId(0x53614a, 0x1, 0x0, 0x0, 0x0)
    /usr/lib/go-1.10/src/os/user/lookup.go:41 +0x53 fp=0xc420040f80 sp=0xc420040f30 pc=0x488763
main.main.func1(0xc42007c060, 0x53614a, 0x1, 0xc42007c0c0)
    /home/kir/go/src/github.com/kolyshkin/test/lookup.go:14 +0x4e fp=0xc420040fc0 sp=0xc420040f80 pc=0x48a2fe
runtime.goexit()
    /usr/lib/go-1.10/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420040fc8 sp=0xc420040fc0 pc=0x450c51
created by main.main
    /home/kir/go/src/github.com/kolyshkin/test/lookup.go:12 +0xb3

goroutine 1 [chan receive]:
main.main()
    /home/kir/go/src/github.com/kolyshkin/test/lookup.go:20 +0x164
kolyshkin picture kolyshkin on 9 May 2018

it's really not obvious that a crash while using archive/tar should be fixed by using -tags osusergo. <...>
In my opinion the latter problem means that there is still a problem to fix here. The osusergo tag is useful by itself, but as far as this bug is concerned it's only a workaround.

Producing a static build already requires a non-trivial amount of flags passed to go build, which on Linux currently amounts to something like

-ldflags '-extldflags "-fno-PIC -static"' -buildmode pie -tags 'osusergo netgo static_build'

...and this magic string keeps growing.

Perhaps the solution is to encapsulate this knowledge internally, exposing it via a new -static flag for go build and friends?

One other good thing such a flag could also do is to add --static flag to pkg-config invocations (those initiated by // #cgo pkg-config: lib lines in the source code). It will solve another issue for which a somewhat verbose workaround is currently required (for example, see ploop_link_static.go and ploop_link_dynamic.go). In fact I have already suggested it some time ago in https://github.com/golang/go/issues/12058.

kolyshkin picture kolyshkin on 9 May 2018

@kolyshkin I suggest that you open a new bug for a -static option.

I'm inclined to close this bug. I don't see what we can change in Go to fix it. There is a workaround, as you know.

ianlancetaylor picture ianlancetaylor on 9 May 2018

I suggest that you open a new bug for a -static option.

Done, see https://github.com/golang/go/issues/26492

kolyshkin picture kolyshkin on 20 Jul 2018

Closing based on https://github.com/golang/go/issues/24787#issuecomment-387822603

FiloSottile picture FiloSottile on 31 Aug 2018
🎉1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Miserlou picture Miserlou  ·  3Comments
lkarlslund picture lkarlslund  ·  3Comments
mingrammer picture mingrammer  ·  3Comments
ashb picture ashb  ·  3Comments
myitcv picture myitcv  ·  3Comments