Go: all: get standard library building with -d=checkptr

Created on 18 Oct 2019  路  72Comments  路  Source: golang/go

As a prerequisite for #34964, the standard library's tests need to all pass when -d=checkptr is enabled:

$ go test -a -short -gcflags=all=-d=checkptr std cmd

Currently, that's not the case.

  1. There are also some panics because of (*[Big]T)(p)[:n] expressions, like in package reflect. This code pattern should be recognized by cmd/compile.

  2. There are a few tests that fail because CL 201781 leads them to heap allocate when before things would only stack allocate; e.g., context.TestAllocs or database/sql.TestRawBytesAllocs. Not sure how to handle this; maybe for now the escape analysis change should only be enabled for -d=checkptr=2, and -race/-msan only enable -d=checkptr=1.

  3. In sync/atomic.hammerStoreLoadPointer, there's (in effect):

    new := uintptr(LoadPointer(addr))
new += offset
StorePointer(addr, unsafe.Pointer(new))

    This violates package unsafe's pointer rules: pointers have to be converted to uintptr and back to unsafe.Pointer in a single expression without being stored in a uintptr-typed variable, as is being done with new. (This just needs someone to grok exactly what the test is doing, and fix it to follow pointer rules correctly.)

There seem to be other failures still that need to be diagnosed. If folks want to try running the command above, claiming a failure, and then investigating what's going on, that would be great.

NeedsInvestigation help wanted
Source
picture mdempsky
👍7 ❤5

Most helpful comment

@odeke-em Thanks for looking into it.

Writing such code is a valid usecase

Regardless of the merit of the use case, if the example violates package unsafe's safety rules, then it's not valid Go.

Your repro case can be simplified to just:

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    fmt.Println(unsafe.Pointer(uintptr(0xc000000000)))
}

This is failing because 0xc000000000 is an offset into the Go heap (at least where Go tries to allocate it by default on 64-bit OSes), and package unsafe's safety rules don't allow you to fabricate a pointer into the heap out of nothing.

To fix the test, it should either:

  1. Exclusively use pointers outside of the Go heap. (This is challenging, because you don't know where the Go heap is, and allocating non-Go-heap memory in a portable way is tricky.)

  2. Alternatively, allocate a large chunk of memory, find an adequately aligned pointer within that chunk of memory, and then restrict to modifying the lower bits of the address. For example:

    buf := make([]byte, 1 << 21) // allocate 2MB
p := unsafe.Pointer(&buf[0])
p = unsafe.Pointer((uintptr(p) + 1<<20 - 1) &^ (1<<20 - 1))  // align p to 1MB boundary

    p is now 20-bit aligned, and you can change any of the lower 20 bits and be guaranteed it still points into buf.

picture mdempsky on 18 Oct 2019
👍3

All 72 comments

/cc @odeke-em @cuonglm

mdempsky picture mdempsky on 18 Oct 2019

This code in package reflect is currently failing in checkptrAlignment, and seems to violate the safety rules because of the conversion to *[16]byte (sometimes gcdata points to <16 bytes):

https://github.com/golang/go/blob/8c6876e9a481a2ea48070d3285a07163f564877b/src/reflect/type.go#L2184-L2192

(There's similar code for etype just below.)

mdempsky picture mdempsky on 18 Oct 2019

I'll cover the sync/atomic.hammer* tests fixup :)

odeke-em picture odeke-em on 18 Oct 2019
👍1

This failure looks genuinely suspicious to me:

$ go test -a -short -gcflags=all=-d=checkptr testing/quick
runtime: pointer 0xc00022a3e0 to unallocated span span.base()=0xc000222000 span.limit=0xc00023a000 span.state=3
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
mdempsky picture mdempsky on 18 Oct 2019

Change https://golang.org/cl/201841 mentions this issue: all: disable tests that fail under -d=checkptr

gopherbot picture gopherbot on 18 Oct 2019

Change https://golang.org/cl/201839 mentions this issue: cmd/compile: recognize (*[Big]T)(ptr)[:n:m] pattern for -d=checkptr

gopherbot picture gopherbot on 18 Oct 2019

Uploaded a couple CLs to improve -d=checkptr, and address some issues.

CL 201841 enumerates all of the remaining std cmd test failures that I'm aware of, but haven't investigated yet.

mdempsky picture mdempsky on 18 Oct 2019
👍1

Hey @mdempsky in regards to the sync/atomic.HammerLoad* tests that are flagged, what they do is that for 8 goroutines, run pairwise increments of the high and low parts of an int* or *uint and panic if high != low. To save the value at the end of the function, we invoke

StorePointer(addr, newValue)

where newValue is a raw uint64 value such as 0xc0000000c0 and that value is being written to addr.

checkptr tries to find the object associated with newValue but unfortunately that's a raw value that's not attached to any object and perhaps beyond Go's address space. Writing such code is a valid usecase and here is a minimal repro to test that edge case

package main

import (
    "fmt"
    "sync/atomic"
    "unsafe"
)

func main() {
    val := uint64(820338753727)
    addr := (*unsafe.Pointer)(unsafe.Pointer(&val))
    v := uintptr(atomic.LoadPointer(addr))
    atomic.StorePointer(addr, unsafe.Pointer(v+1+1<<32))

    xs := uintptr(atomic.LoadPointer(addr))
    fmt.Printf("xs: %#x\n", xs)
}

$ go run -gcflags=all=-d=checkptr main.go 
panic: (runtime.ptrArith) (0x10c09c0,0xc00000c080)

goroutine 1 [running]:
main.main()
    /Users/emmanuelodeke/Desktop/openSrc/bugs/golang/34972/main.go:13 +0x91
exit status 2

Trying to recognize cases that do plain arithmetic is going to be a tricky one :)

odeke-em picture odeke-em on 18 Oct 2019

Change https://golang.org/cl/201877 mentions this issue: syscall: fix wrong unsafe.Pointer align in anyToSockaddr

gopherbot picture gopherbot on 18 Oct 2019

Another instance in reflect:

https://github.com/golang/go/blob/8c6876e9a481a2ea48070d3285a07163f564877b/src/reflect/type.go#L1011

and some lines below:

https://github.com/golang/go/blob/8c6876e9a481a2ea48070d3285a07163f564877b/src/reflect/type.go#L1023

cuonglm picture cuonglm on 18 Oct 2019

@odeke-em Thanks for looking into it.

Writing such code is a valid usecase

Regardless of the merit of the use case, if the example violates package unsafe's safety rules, then it's not valid Go.

Your repro case can be simplified to just:

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    fmt.Println(unsafe.Pointer(uintptr(0xc000000000)))
}

This is failing because 0xc000000000 is an offset into the Go heap (at least where Go tries to allocate it by default on 64-bit OSes), and package unsafe's safety rules don't allow you to fabricate a pointer into the heap out of nothing.

To fix the test, it should either:

  1. Exclusively use pointers outside of the Go heap. (This is challenging, because you don't know where the Go heap is, and allocating non-Go-heap memory in a portable way is tricky.)

  2. Alternatively, allocate a large chunk of memory, find an adequately aligned pointer within that chunk of memory, and then restrict to modifying the lower bits of the address. For example:

    buf := make([]byte, 1 << 21) // allocate 2MB
p := unsafe.Pointer(&buf[0])
p = unsafe.Pointer((uintptr(p) + 1<<20 - 1) &^ (1<<20 - 1))  // align p to 1MB boundary

    p is now 20-bit aligned, and you can change any of the lower 20 bits and be guaranteed it still points into buf.

mdempsky picture mdempsky on 18 Oct 2019
👍3

There are also some panics because of (*[Big]T)(p)[:n] expressions, like in package reflect. This code pattern should be recognized by cmd/compile.

I saw that one coming in https://github.com/golang/go/issues/13656#issuecomment-165670266!

One could easily envision a compiler that, say, sanity-checks that all pointers to arrays point to addresses that are mapped in the program's address space.

bcmills picture bcmills on 18 Oct 2019
😄1 👍1

There are a few tests that fail because CL 201781 leads them to heap allocate when before things would only stack allocate

If so, those tests are overspecified: the language spec does not make any guarantees about allocations or escapes. They should be updated to skip (or use different thresholds) when they are not using a specific compiler _in a specific configuration_.

bcmills picture bcmills on 18 Oct 2019

They should be updated to skip (or use different thresholds) when they are not using a specific compiler in a specific configuration.

Agreed, though I don't think currently there's a way for tests to check what compiler flags were used. (Keep in mind too that different compiler flags can be used for different packages.)

mdempsky picture mdempsky on 18 Oct 2019

@mdempsky on OSX, I got a lot of:

fatal error: stack growth after fork

runtime stack:
runtime.throw(0x12e77be, 0x17)
    /Users/cuonglm/sources/go/src/runtime/panic.go:774 +0x72
runtime.newstack()
    /Users/cuonglm/sources/go/src/runtime/stack.go:928 +0xd04
runtime.morestack()
    /Users/cuonglm/sources/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 19 [running]:
runtime.checkptrAlignment(0xc00008a9e8, 0x126e840)
    /Users/cuonglm/sources/go/src/runtime/checkptr.go:14 +0x137 fp=0xc00008a8e8 sp=0xc00008a8e0 pc=0x1006177
syscall.funcPC(...)
    /Users/cuonglm/sources/go/src/syscall/syscall_darwin.go:469
syscall.forkAndExecInChild(0xc00008e120, 0xc000082ba0, 0x5, 0x5, 0xc00006a900, 0x2e, 0x2e, 0x0, 0x0, 0xc00008ac00, ...)
    /Users/cuonglm/sources/go/src/syscall/exec_darwin.go:65 +0xe6 fp=0xc00008aa60 sp=0xc00008a8e8 pc=0x10703d6
syscall.forkExec(0xc0000c6160, 0x20, 0xc000094a40, 0x4, 0x4, 0xc00008ac00, 0x1140b6b, 0x0, 0x0)
    /Users/cuonglm/sources/go/src/syscall/exec_unix.go:201 +0x35b fp=0xc00008ab70 sp=0xc00008aa60 pc=0x10718eb
syscall.StartProcess(...)
    /Users/cuonglm/sources/go/src/syscall/exec_unix.go:248
os.startProcess(0xc0000c6160, 0x20, 0xc000094a40, 0x4, 0x4, 0xc00008ad98, 0x0, 0x0, 0x0)
    /Users/cuonglm/sources/go/src/os/exec_posix.go:52 +0x2c0 fp=0xc00008ac58 sp=0xc00008ab70 pc=0x1093300
os.StartProcess(0xc0000c6160, 0x20, 0xc000094a40, 0x4, 0x4, 0xc00008ad98, 0x2d, 0x0, 0x0)
    /Users/cuonglm/sources/go/src/os/exec.go:102 +0x7c fp=0xc00008acb0 sp=0xc00008ac58 pc=0x1092d2c
os/exec.(*Cmd).Start(0xc0000bc2c0, 0xc00008af01, 0xc000082b70)
    /Users/cuonglm/sources/go/src/os/exec/exec.go:416 +0x50c fp=0xc00008adf0 sp=0xc00008acb0 pc=0x116cbac
os/exec.(*Cmd).Run(0xc0000bc2c0, 0xc000082b70, 0xc0000bc2c0)
    /Users/cuonglm/sources/go/src/os/exec/exec.go:338 +0x2b fp=0xc00008ae18 sp=0xc00008adf0 pc=0x116c63b
os/exec.(*Cmd).CombinedOutput(0xc0000bc2c0, 0x20, 0xc00008af30, 0x3, 0x3, 0xc0000bc2c0)
    /Users/cuonglm/sources/go/src/os/exec/exec.go:561 +0x91 fp=0xc00008ae48 sp=0xc00008ae18 pc=0x116d7a1
go/importer.TestForCompiler(0xc0000f0300)
    /Users/cuonglm/sources/go/src/go/importer/importer_test.go:23 +0x11d fp=0xc00008af70 sp=0xc00008ae48 pc=0x124dfed
testing.tRunner(0xc0000f0300, 0x12f26a0)
    /Users/cuonglm/sources/go/src/testing/testing.go:909 +0xc9 fp=0xc00008afd0 sp=0xc00008af70 pc=0x10f8249
runtime.goexit()
    /Users/cuonglm/sources/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc00008afd8 sp=0xc00008afd0 pc=0x105cd11
created by testing.(*T).Run
    /Users/cuonglm/sources/go/src/testing/testing.go:960 +0x351

when testing std, even after https://github.com/golang/go/commit/46aa8354fa57ab5a4fb133898baf18aafbeb2e88

Note

This error does not happen on Linux.

cuonglm picture cuonglm on 19 Oct 2019

@cuonglm Thanks for the report.

It looks like maybe checkptrAlignment and checkptrArithmetic need to be labeled //go:nosplit.

mdempsky picture mdempsky on 19 Oct 2019

Change https://golang.org/cl/202157 mentions this issue: runtime: fix unsafe.Pointer alignment on Linux

gopherbot picture gopherbot on 19 Oct 2019

@cuonglm Thanks for the report.

It looks like maybe checkptrAlignment and checkptrArithmetic need to be labeled //go:nosplit.

I tried, but still fails with go:nosplit for checkptrAlignment and checkptrArithmetic. But it does pass if findObject is marked go:nosplit, too.

cuonglm picture cuonglm on 19 Oct 2019

Ugh, right.

Okay, in that case, we should probably just make //go:nosplit imply //go:nocheckptr, like how //go:cgo_unsafe_args does. (See CL 201823.)

mdempsky picture mdempsky on 19 Oct 2019
👍1

Change https://golang.org/cl/202158 mentions this issue: cmd/compile: disable checkptr for //go:nosplit functions

gopherbot picture gopherbot on 19 Oct 2019

Change https://golang.org/cl/202177 mentions this issue: windows, unix: fix wrong unsafe.Pointer alignment in syscall

gopherbot picture gopherbot on 19 Oct 2019

Change https://golang.org/cl/202580 mentions this issue: reflect, internal/reflectlite: set capacity when slicing unsafe pointers

gopherbot picture gopherbot on 22 Oct 2019

Change https://golang.org/cl/202597 mentions this issue: sync/atomic: suppress checkptr errors for hammerStoreLoadPointer

gopherbot picture gopherbot on 22 Oct 2019

Change https://golang.org/cl/202582 mentions this issue: reflect: fix unsafe conversions reported by -d=checkptr

gopherbot picture gopherbot on 22 Oct 2019

[Edit: Filed separately as #35068, because I think this is a runtime error.]

The only remaining failure on linux/amd64 is testing/quick.

@aclements Is runtime.findObject safe to call on a pointer to the stack? I assumed it was, but it seems to reliably fail for testing/quick under -d=checkptr:

$ go test -v -short -gcflags=all=-d=checkptr testing/quick -run=TestMutuallyRecursive
=== RUN   TestMutuallyRecursive
runtime: pointer 0xc00011c3e0 to unallocated span span.base()=0xc00011c000 span.limit=0xc00012c000 span.state=3
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

goroutine 7 [running]:
runtime.throw(0x596d14, 0x3e)
    /usr/local/google/home/mdempsky/wd/go/src/runtime/panic.go:774 +0x72 fp=0xc00013c308 sp=0xc00013c2d8 pc=0x42ec12
runtime.findObject(0xc00011c3e0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/google/home/mdempsky/wd/go/src/runtime/mbitmap.go:397 +0x3b4 fp=0xc00013c358 sp=0xc00013c308 pc=0x413e04
runtime.checkptrBase(...)
    /usr/local/google/home/mdempsky/wd/go/src/runtime/checkptr.go:56
runtime.checkptrAlignment(0xc00013c3e0, 0x566020, 0x1)
    /usr/local/google/home/mdempsky/wd/go/src/runtime/checkptr.go:23 +0x76 fp=0xc00013c3c0 sp=0xc00013c358 pc=0x406306
reflect.packEface(0x561b60, 0x0, 0x19, 0xff00000000000000, 0x0)
    /usr/local/google/home/mdempsky/wd/go/src/reflect/value.go:106 +0x4c fp=0xc00013c400 sp=0xc00013c3c0 pc=0x4b1c3c
reflect.valueInterface(0x561b60, 0x0, 0x19, 0x1, 0x19, 0xc000096590)
    /usr/local/google/home/mdempsky/wd/go/src/reflect/value.go:1023 +0x12b fp=0xc00013c450 sp=0xc00013c400 pc=0x4b670b
reflect.Value.Interface(...)
    /usr/local/google/home/mdempsky/wd/go/src/reflect/value.go:993
testing/quick.sizedValue(0x5be680, 0x561b60, 0xc0000bc330, 0x32, 0xc00000eb10, 0x196, 0x7ff03c946008, 0x0)
    /usr/local/google/home/mdempsky/wd/go/src/testing/quick/quick.go:67 +0x77 fp=0xc00013c5d8 sp=0xc00013c450 pc=0x52dc77
...

The conversion in question looks okay to me:

https://github.com/golang/go/blob/9979366e07fa8aa6fac81702a70f6eacf427c431/src/reflect/value.go#L105-L106

Notably, if I force i onto the heap, then the panic goes away, which is what makes me suspect runtime.findObject doesn't like stack pointers. :(

mdempsky picture mdempsky on 22 Oct 2019

The only remaining failure on linux/amd64 is testing/quick.

It's on darwin, too.

cuonglm picture cuonglm on 22 Oct 2019
👍1

@cuonglm Thanks for the confirmation.

Also, it looks like there's still a lot of Windows code that's going to need updating:

$ git grep '\*\[.*unsafe.*:'| grep windows
cmd/vendor/golang.org/x/sys/windows/security_windows.go:    return UTF16ToString((*[256]uint16)(unsafe.Pointer(s))[:]), nil
cmd/vendor/golang.org/x/sys/windows/syscall_windows.go:     bytes := (*[10000]byte)(unsafe.Pointer(&pp.Path[0]))[0:n]
crypto/x509/root_windows.go:    simpleChains := (*[1 << 20]*syscall.CertSimpleChain)(unsafe.Pointer(simpleChain))[:]
crypto/x509/root_windows.go:    elements := (*[1 << 20]*syscall.CertChainElement)(unsafe.Pointer(lastChain.Elements))[:]
crypto/x509/root_windows.go:        encodedCert := (*[1 << 20]byte)(unsafe.Pointer(cert.EncodedCert))[:]
crypto/x509/root_windows.go:        buf := (*[1 << 20]byte)(unsafe.Pointer(cert.EncodedCert))[:]
internal/syscall/windows/registry/value.go: u := (*[1 << 29]uint16)(unsafe.Pointer(&data[0]))[:]
internal/syscall/windows/registry/value.go:         u := (*[1 << 29]uint16)(unsafe.Pointer(&r[0]))[:]
internal/syscall/windows/registry/value.go: p := (*[1 << 29]uint16)(unsafe.Pointer(&data[0]))[:len(data)/2]
internal/syscall/windows/registry/value.go: return k.setValue(name, DWORD, (*[4]byte)(unsafe.Pointer(&value))[:])
internal/syscall/windows/registry/value.go: return k.setValue(name, QWORD, (*[8]byte)(unsafe.Pointer(&value))[:])
internal/syscall/windows/registry/value.go: buf := (*[1 << 29]byte)(unsafe.Pointer(&v[0]))[:len(v)*2]
internal/syscall/windows/registry/value.go: buf := (*[1 << 29]byte)(unsafe.Pointer(&v[0]))[:len(v)*2]
net/interface_windows.go:               Name:  syscall.UTF16ToString((*(*[10000]uint16)(unsafe.Pointer(aa.FriendlyName)))[:]),
net/lookup_windows.go:  cname := syscall.UTF16ToString((*[256]uint16)(unsafe.Pointer(resolved))[:])
net/lookup_windows.go:      srvs = append(srvs, &SRV{absDomainName([]byte(syscall.UTF16ToString((*[256]uint16)(unsafe.Pointer(v.Target))[:]))), v.Port, v.Priority, v.Weight})
net/lookup_windows.go:      mxs = append(mxs, &MX{absDomainName([]byte(syscall.UTF16ToString((*[256]uint16)(unsafe.Pointer(v.NameExchange))[:]))), v.Preference})
net/lookup_windows.go:      nss = append(nss, &NS{absDomainName([]byte(syscall.UTF16ToString((*[256]uint16)(unsafe.Pointer(v.Host))[:])))})
net/lookup_windows.go:      for _, v := range (*[1 << 10]*uint16)(unsafe.Pointer(&(d.StringArray[0])))[:d.StringCount] {
net/lookup_windows.go:          s += syscall.UTF16ToString((*[1 << 20]uint16)(unsafe.Pointer(v))[:])
net/lookup_windows.go:      ptrs = append(ptrs, absDomainName([]byte(syscall.UTF16ToString((*[256]uint16)(unsafe.Pointer(v.Host))[:]))))
os/env_windows.go:      entry := (*[(1 << 30) - 1]uint16)(unsafe.Pointer(blockp))[:]
os/exec_windows.go: cmd := syscall.UTF16ToString((*[0xffff]uint16)(unsafe.Pointer(p))[:])
os/os_windows_test.go:  copy((*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:], target.pathBuf)
os/os_windows_test.go:  copy((*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:], target.pathBuf)
os/os_windows_test.go:                      n = copy((*[10000]uint16)(unsafe.Pointer(buf))[:n], s16)
os/user/lookup_windows.go:  name := syscall.UTF16ToString((*[1024]uint16)(unsafe.Pointer(i.FullName))[:])
os/user/lookup_windows.go:  entries := (*[1024]windows.LocalGroupUserInfo0)(unsafe.Pointer(p0))[:entriesRead]
os/user/lookup_windows.go:      name := syscall.UTF16ToString((*[1024]uint16)(unsafe.Pointer(entry.Name))[:])
syscall/security_windows.go:    return UTF16ToString((*[SecurityMaxSidSize]uint16)(unsafe.Pointer(s))[:]), nil
syscall/syscall_windows.go:     bytes := (*[len(pp.Path)]byte)(unsafe.Pointer(&pp.Path[0]))[0:n]

The package syscall ones are okay, and the DWORD and QWORD ones in internal/syscall/windows/registry/value.go probably are too. But the rest look suspect.

mdempsky picture mdempsky on 22 Oct 2019

Change https://golang.org/cl/202640 mentions this issue: net: fix checkptr failure on Windows

gopherbot picture gopherbot on 22 Oct 2019

I think !windows platforms are in good shape now, and -d=checkptr is now enabled automatically when -race or -msan are used on !windows platforms.

This issue could still use a Windows developer to look at the cases I identified above, and also to try running go test -a -short -gcflags=all=-d=checkptr std cmd to check for any other failures.

mdempsky picture mdempsky on 23 Oct 2019

cc @alexbrainman @mattn @zx2c4

networkimprov picture networkimprov on 23 Oct 2019

Change https://golang.org/cl/203837 mentions this issue: sha3: align (*state).storage

gopherbot picture gopherbot on 28 Oct 2019

Change https://golang.org/cl/203442 mentions this issue: internal/syscall/windows/registry: make '-gcflags=all=-d=checkptr' flag work

gopherbot picture gopherbot on 28 Oct 2019

Change https://golang.org/cl/204017 mentions this issue: windows/registry: make '-gcflags=all=-d=checkptr' flag work

gopherbot picture gopherbot on 29 Oct 2019

crypto/x509/root_windows.go: simpleChains := ([1 << 20]syscall.CertSimpleChain)(unsafe.Pointer(simpleChain))[:]
crypto/x509/root_windows.go: elements := ([1 << 20]syscall.CertChainElement)(unsafe.Pointer(lastChain.Elements))[:]
crypto/x509/root_windows.go: encodedCert := ([1 << 20]byte)(unsafe.Pointer(cert.EncodedCert))[:]
crypto/x509/root_windows.go: buf := ([1 << 20]byte)(unsafe.Pointer(cert.EncodedCert))[:]

@mdempsky checker does not complain about crypto/x509 code

c:\>go test -a -short -gcflags=all=-d=checkptr crypto/x509
ok      crypto/x509     0.393s

Why? All 4 lines deals with external to Go memory returned by Windows API. Is that why?

Thank you.

Alex

alexbrainman picture alexbrainman on 30 Oct 2019

Change https://golang.org/cl/204218 mentions this issue: syscall: revert security_windows.go change of CL 201877

gopherbot picture gopherbot on 30 Oct 2019

Why? All 4 lines deals with external to Go memory returned by Windows API. Is that why?

That sounds plausible.

The way the check currently works is when you convert p into *T, we try to check that p and p + sizeof(T) - 1 both point into the same Go variable.

For pointers into the Go heap, we have very precise information about Go variable boundaries (because we need that for GC), so that's generally a very robust check.

For pointers into Goroutine stacks or into global Go variables, we have course grained information: basically just that it points into a particular stack, or into some particular EXE or DLL's global Go variables.

For C allocated memory, we don't have any visibility at the moment. Basically anything that wasn't allocated by Go is treated uniformly as non-Go memory.

However, the reason you still need to be careful even with pointers to C memory is that while p might point into C memory, it's possible that p + sizeof([1 << 20]SomeType) - 1 might overflow into the Go heap depending on where p was allocated in memory.

mdempsky picture mdempsky on 30 Oct 2019

The way the check currently works is

Thank you for taking time to explain.

I think we do need to change crypto/x509 package.

Alex

alexbrainman picture alexbrainman on 3 Nov 2019

Change https://golang.org/cl/204621 mentions this issue: crypto/x509: make '-gcflags=all=-d=checkptr' flag work

gopherbot picture gopherbot on 3 Nov 2019

@mdempsky I need more help.

I am trying to convert this line

net/interface_windows.go: Name: syscall.UTF16ToString((([10000]uint16)(unsafe.Pointer(aa.FriendlyName)))[:]),

https://github.com/golang/go/blob/3873e5497d21979910a1ec7cf90a34577fa1f6ae/src/net/interface_windows.go#L61

But FriendlyName is 0 terminated uint16 string (see https://docs.microsoft.com/en-us/windows/win32/api/iptypes/ns-iptypes-ip_adapter_addresses_lh for the reference). There is no length of FriendlyName provided anywhere. What do we do here?

The test is also crashes

:\>u:\test -test.short
panic: runtime error: unsafe pointer conversion

goroutine 1 [running]:
net.interfaceTable(0x0, 0xa80, 0xa80, 0x66e940, 0x1, 0xc000087ae0)
        /home/a/go/src/net/interface_windows.go:61 +0xec
net.Interfaces(0x20, 0x28, 0xc0000667e0, 0x47fbb1, 0x890854)
        /home/a/go/src/net/interface.go:100 +0x35
net.loopbackInterface(0x65d0c0)
        /home/a/go/src/net/interface_test.go:20 +0x3b
net.setupTestData()
        /home/a/go/src/net/main_test.go:102 +0x5c
net.TestMain(0xc0000ac000)
        /home/a/go/src/net/main_test.go:49 +0x2d
main.main()
        _testmain.go:474 +0x13c

C:\>

Thank you.

Alex

alexbrainman picture alexbrainman on 3 Nov 2019

Change https://golang.org/cl/204117 mentions this issue: syscall: revert wrong sid string length windows

gopherbot picture gopherbot on 7 Nov 2019

@mdempsky I need more help.

Sorry, I missed this. (Well, I saw the GitHub notification emails, but I thought it was just gopherbot comments.)

There is no length of FriendlyName provided anywhere. What do we do here?

We'll need to compute the length then. E.g., a function like:

func utf16PtrToString(p *uint16) string {
    // Find NUL terminator.
    end := unsafe.Pointer(p)
    for *(*uint16)(end) != 0 {
        end = unsafe.Pointer(uintptr(end) + unsafe.Sizeof(*p))
    }

    n := (uintptr(end) - uintptr(unsafe.Pointer(p))) / unsafe.Sizeof(*p)
    s := (*[1<<30]uint16)(unsafe.Pointer(p))[:n:n]
    return string(utf16.Decode(s))
}
mdempsky picture mdempsky on 8 Nov 2019 
func utf16PtrToString(p *uint16) string {

I think this function will be useful to make other code '-gcflags=all=-d=checkptr' safe.

I could add this function to internal/syscall/windows package. But maybe we should add it to syscall package instead. We could recommend the function to be used in external code instead of syscall.UTF16ToString.

What do you think? Is it too earlier to make this decision?

Thank you.

Alex

alexbrainman picture alexbrainman on 8 Nov 2019

I could add this function to internal/syscall/windows package.

That sounds reasonable to me.

But maybe we should add it to syscall package instead.

My understanding is package syscall is frozen, and no new functions are to be added.

This seems like maybe it would be a reasonable exception? But that's not my call, so I'd probably start by just adding it to x/sys/windows instead.

mdempsky picture mdempsky on 8 Nov 2019

My understanding is package syscall is frozen, and no new functions are to be added.

This seems like _maybe_ it would be a reasonable exception?

I had to use new function in syscall package. So it cannot be in internal/syscall/windows, because internal/syscall/windows already imports syscall. I will put in syscall for now, and Ian could decide.

I went up to

os/os_windows_test.go: copy((*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:], target.pathBuf)

now. go test crashes with

panic: runtime error: unsafe pointer conversion [recovered]
        panic: runtime error: unsafe pointer conversion

goroutine 138 [running]:
testing.tRunner.func1(0xc00028e120)
        /home/a/go/src/testing/testing.go:916 +0x60b
panic(0x5afc80, 0xc000328680)
        /home/a/go/src/runtime/panic.go:961 +0x16b
os_test.createMountPoint(0xc0003394f0, 0x44, 0xc0002a5c10, 0xc00074007e, 0x3a)
        /home/a/go/src/os/os_windows_test.go:266 +0x119
os_test.TestDirectoryJunction.func1(0xc0003394f0, 0x44, 0xc00029a180, 0x3a, 0x44, 0xc0000861d0)
        /home/a/go/src/os/os_windows_test.go:285 +0xf5
os_test.testDirLinks(0xc00028e120, 0xc0004d4400, 0x3, 0x4)
        /home/a/go/src/os/os_windows_test.go:123 +0x671
os_test.TestDirectoryJunction(0xc00028e120)
        /home/a/go/src/os/os_windows_test.go:317 +0x28c
testing.tRunner(0xc00028e120, 0x5d9eb0)
        /home/a/go/src/testing/testing.go:954 +0xe3
created by testing.(*T).Run
        /home/a/go/src/testing/testing.go:1005 +0x35e

so it is definitely a problem. What is happening here, we have

package windows // import "internal/syscall/windows"

type MountPointReparseBuffer struct {
        // The integer that contains the offset, in bytes,
        // of the substitute name string in the PathBuffer array,
        // computed as an offset from byte 0 of PathBuffer. Note that
        // this offset must be divided by 2 to get the array index.
        SubstituteNameOffset uint16
        // The integer that contains the length, in bytes, of the
        // substitute name string. If this string is null-terminated,
        // SubstituteNameLength does not include the Unicode null character.
        SubstituteNameLength uint16
        // PrintNameOffset is similar to SubstituteNameOffset.
        PrintNameOffset uint16
        // PrintNameLength is similar to SubstituteNameLength.
        PrintNameLength uint16
        PathBuffer      [1]uint16
}

but PathBuffer is longer than one uint16 long - it is PrintNameLength elements long.

See MountPointReparseBuffer union part of REPARSE_DATA_BUFFER

https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_reparse_data_buffer

Windows reads or writes uint16 string starting from PathBuffer[1]. How do I write uint16 string there?

That is what we do at this moment.

https://github.com/golang/go/blob/e6fb39aa6869fa5033b87c14a9826598679cb47d/src/os/os_windows_test.go#L266

Thank you.

Alex

alexbrainman picture alexbrainman on 10 Nov 2019

@alexbrainman, changing

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:]

to

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:buflen:buflen]

might do the trick. (See CL 201839.)

bcmills picture bcmills on 10 Nov 2019

changing

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:]

to

(*[2048]uint16)(unsafe.Pointer(&buf.PathBuffer[0]))[:buflen:buflen]

might do the trick

Thank you, Bryan. That does the trick.

I knew the trick, but I forgotten about it. Moving on for now.

@mdempsky I also had to change

--- a/src/internal/syscall/windows/reparse_windows.go
+++ b/src/internal/syscall/windows/reparse_windows.go
@@ -60,8 +60,9 @@ type SymbolicLinkReparseBuffer struct {

 // Path returns path stored in rb.
 func (rb *SymbolicLinkReparseBuffer) Path() string {
-       p := (*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))
-       return syscall.UTF16ToString(p[rb.SubstituteNameOffset/2 : (rb.SubstituteNameOffset+rb.SubstituteNameLength)/2])
+       n1 := rb.SubstituteNameOffset / 2
+       n2 := (rb.SubstituteNameOffset + rb.SubstituteNameLength) / 2
+       return syscall.UTF16ToString((*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))[n1:n2:n2])
 }

 type MountPointReparseBuffer struct {
@@ -83,6 +84,7 @@ type MountPointReparseBuffer struct {

 // Path returns path stored in rb.
 func (rb *MountPointReparseBuffer) Path() string {
-       p := (*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))
-       return syscall.UTF16ToString(p[rb.SubstituteNameOffset/2 : (rb.SubstituteNameOffset+rb.SubstituteNameLength)/2])
+       n1 := rb.SubstituteNameOffset / 2
+       n2 := (rb.SubstituteNameOffset + rb.SubstituteNameLength) / 2
+       return syscall.UTF16ToString((*[0xffff]uint16)(unsafe.Pointer(&rb.PathBuffer[0]))[n1:n2:n2])
 }

because go test crashed. But these weren't listed at https://github.com/golang/go/issues/34972#issuecomment-544774978 I am worried we might miss more like it. Should I try different grep '\*\[.*unsafe.*:' expression?

Thank you.

Alex

alexbrainman picture alexbrainman on 11 Nov 2019

Change https://golang.org/cl/207199 mentions this issue: windows: revert security_windows.go change of CL 202177

gopherbot picture gopherbot on 17 Nov 2019

Change https://golang.org/cl/208617 mentions this issue: all: replace [:] slice operation involving unsafe.Pointer with [:n:n]

gopherbot picture gopherbot on 24 Nov 2019

@mdempsky one last crash remaining after I apply

https://go-review.googlesource.com/c/go/+/208617

--- FAIL: TestPacketConn (0.00s)
panic: runtime error: unsafe pointer conversion [recovered]
        panic: runtime error: unsafe pointer conversion

goroutine 203 [running]:
testing.tRunner.func1(0xc0003aab40)
        c:/users/alex/dev/go/src/testing/testing.go:916 +0x60b
panic(0x6781a0, 0xc00039dea0)
        c:/users/alex/dev/go/src/runtime/panic.go:961 +0x16b
syscall.WSASendto(0x278, 0xc0003c57b8, 0x100000000000001, 0xc0003c57a0, 0x0, 0x6ebec0, 0xc000371e80, 0xc0003c5770, 0x0, 0x0, ...)
        c:/users/alex/dev/go/src/syscall/syscall_windows.go:876 +0x9b
internal/poll.(*FD).WriteTo.func2(0xc0003c5770, 0x77, 0xc00041fc00)
        c:/users/alex/dev/go/src/internal/poll/fd_windows.go:869 +0x86
internal/poll.(*ioSrv).ExecIO(0x87fb68, 0xc0003c5770, 0x6a9b98, 0x26b54ef8, 0x4, 0x100000000000000)
        c:/users/alex/dev/go/src/internal/poll/fd_windows.go:206 +0xbb
internal/poll.(*FD).WriteTo(0xc0003c5680, 0xc0003b1360, 0xf, 0xf, 0x6ebec0, 0xc000371e80, 0x0, 0x0, 0x0)
        c:/users/alex/dev/go/src/internal/poll/fd_windows.go:868 +0x285
net.(*netFD).writeTo(0xc0003c5680, 0xc0003b1360, 0xf, 0xf, 0x6ebec0, 0xc000371e80, 0x10000001dcd6500, 0x0, 0x80)
        c:/users/alex/dev/go/src/net/fd_windows.go:187 +0x76
net.(*UDPConn).writeTo(0xc0003a6fa8, 0xc0003b1360, 0xf, 0xf, 0xc0003c33b0, 0xc0003c5900, 0xbf6e961d56f1ac28, 0x11de793c1)
        c:/users/alex/dev/go/src/net/udpsock_posix.go:80 +0xd8
net.(*UDPConn).WriteTo(0xc0003a6fa8, 0xc0003b1360, 0xf, 0xf, 0x6ec6c0, 0xc0003c33b0, 0x882200, 0xb, 0x698dac)
        c:/users/alex/dev/go/src/net/udpsock.go:167 +0x9a
net.TestPacketConn(0xc0003aab40)
        c:/users/alex/dev/go/src/net/packetconn_test.go:78 +0x62e
testing.tRunner(0xc0003aab40, 0x6aa408)
        c:/users/alex/dev/go/src/testing/testing.go:954 +0xe3
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1005 +0x35e
FAIL    net     4.355s
FAIL

I am not sure, but I suspect the problem is conversion of *SockaddrInet4 to unsafe.Pointer to *RawSockaddrAny.

https://github.com/golang/go/blob/6f7b96f6cb5f5c156cf4e69b1fa1ec0e57b01677/src/syscall/syscall_windows.go#L859

Linux does something similar but the final pointer is *byte

https://github.com/golang/go/blob/6f7b96f6cb5f5c156cf4e69b1fa1ec0e57b01677/src/syscall/syscall_linux.go#L666

, so pointer checker does not complain.

How do I change code to avoid the crash?

Thank you.

Alex

alexbrainman picture alexbrainman on 24 Nov 2019

@alexbrainman What happen if you change:

(*RawSockaddrAny)(unsafe.Pointer(rsa))

to:

(*RawSockaddrAny)(rsa)

In that call, rsa is already an unsafe.Pointer.

cuonglm picture cuonglm on 24 Nov 2019

Yes, the conversion to *RawSockaddrAny is disallowed there: a RawSockaddrAny is larger than a RawSockaddrInet4, so that conversion in general would allow a caller to make a wild write past the end of the RawSockaddrInet4.

I think the root of the problem is that we are using RawSockaddrAny to mean two different (and mutually-incompatible) things:

  1. A buffer with a large enough size (and the correct alignment) to hold any other RawSockaddr type.
  2. A pointer to any other RawSockaddr type.

For case (1), the Any struct must be _at least as large as_ the other types.
However, for case (2) it must be _no larger than_ the other types (and ideally a strict prefix of each of them).

So the uses need to be split into two separate types, but I don't see any way to do that without breaking compatibility.

I guess the other option would be to pad out the other RawSockaddr types so that they are all _exactly_ the same size as a RawSockaddrAny, but then if we ever get a RawSockaddr value from outside of the Go heap we'll still have a pointer to an object of invalid size.

bcmills picture bcmills on 25 Nov 2019

C solves this by in fact having two types: sockaddr, which is just the type prefix, and sockaddr_storage, which can fit of the other ones, such as sockaddr_in4.

zx2c4 picture zx2c4 on 25 Nov 2019
👍2

Change https://golang.org/cl/214118 mentions this issue: doc/go1.14: -d=checkptr is not yet recommended on Windows

gopherbot picture gopherbot on 9 Jan 2020

Bumping to Go 1.15 to keep working on the fixes for Windows. When those are resolved, we should also enable -d=checkptr by default when -race or -msan are enabled (currently this is the case everywhere but Windows).

aclements picture aclements on 9 Jan 2020

@mdempsky one last crash remaining after I apply https://go-review.googlesource.com/c/go/+/208617

WSASendto calls WSASendTo (which seems a little nuts to me, but okay...), which winds up just casing the *RawSockaddrAny back to a uintptr to make the syscall. What if we were to directly invoke the syscall from WSASendto with the unsafe.Pointer that came from to.sockaddr(), instead of converting it through an intermediate *RawSockaddrAny?

aclements picture aclements on 10 Jan 2020

Actually, WSASendTo (with the big 'T') is definitely an unsafe API, since it takes the byte length of the sockaddr as an argument and hence could be used to make the kernel read arbitrary memory from the Go process. We try not to expose unsafe APIs from syscall, and go to some significant lengths to make APIs safe in the UNIX syscall packages (though this is certainly not the only unsafe API in the Windows syscall package).

Can we just remove WSASendTo? syscall is not covered by Go 1 compatibility, and this would fall under the security exception anyway.

aclements picture aclements on 10 Jan 2020

WSASendto calls WSASendTo (which seems a little nuts to me, but okay...),

WSASendto was added in

https://golang.org/cl/3136042

These were the days where syscall package was free for all to do as we pleased.

What if we were to directly invoke the syscall from WSASendto with the unsafe.Pointer that came from to.sockaddr(), instead of converting it through an intermediate *RawSockaddrAny?

I will try and report it back here.

Actually, WSASendTo (with the big 'T') is definitely an unsafe API, ...

You have to complain to Microsoft

https://docs.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsasendto

Even better. You should complain to people who designed Berkeley sockets in the first place.

Can we just remove WSASendTo?

We can do anything you want. But we are using the function ourselves in net package. And other people might be using it too. So we would have to replace it with something that call real WSASendTo anyway.

Alex

alexbrainman picture alexbrainman on 16 Feb 2020

I will try and report it back here.

Thanks!

Actually, WSASendTo (with the big 'T') is definitely an unsafe API, ...

You have to complain to Microsoft
https://docs.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsasendto
Even better. You should complain to people who designed Berkeley sockets in the first place.

Yeah, it's not great. But WSASendto exposes a perfectly safe API by taking the Sockaddr interface. That's unsafe internally, but only the syscall package can implement its methods, so that unsafeness doesn't leak through the user API. Sendto on UNIX works the same way. I think the mistake was just exporting the unsafe raw WSASendTo API underlying it (and some related functions. The UNIX packages also have an unsafe sendto API, but it's not exported.

Can we just remove WSASendTo?

We can do anything you want. But we are using the function ourselves in net package. And other people might be using it too.

I may be totally missing something, but I don't see any calls to WSASendTo in net (or x/net). I do see WSASendto. In fact, I couldn't find any calls to WSASendTo in the GitHub corpus I have access to (though I'm really confused about how much coverage that has).

aclements picture aclements on 16 Feb 2020

Change https://golang.org/cl/220544 mentions this issue: syscall: fix windows WSASendto -d=checkptr violation

gopherbot picture gopherbot on 25 Feb 2020

But WSASendto exposes a perfectly safe API by taking the Sockaddr interface. That's unsafe internally, but only the syscall package can implement its methods, so that unsafeness doesn't leak through the user API.

SGTM

I may be totally missing something, but I don't see any calls to WSASendTo in net ...

I did not explain myself properly. net package calls syscall.WSASendto and syscall.WSASendto calls syscall.WSASendTo. So net package calls syscall.WSASendTo indirectly. WSASendto Windows API is needed to implement net package. Whichever way we interface to it, but we must use WSASendto Windows API.

Anyway I fixed net.TestPacketConn in

https://go-review.googlesource.com/c/go/+/220544/

Unfortunately I discovered new broken test in net (I have no idea how I missed it before). This is the output after CL 220544 is applied

c:\Users\Alex\dev\go\src>go test -a -short -gcflags=all=-d=checkptr net
fatal error: checkptr: unsafe pointer conversion

goroutine 125 [running]:
runtime.throw(0x6a5112, 0x23)
        c:/users/alex/dev/go/src/runtime/panic.go:1112 +0x79 fp=0xc00017dc40 sp=0xc00017dc10 pc=0x4379c9
runtime.checkptrAlignment(0xc00020ced0, 0x668d00, 0x1)
        c:/users/alex/dev/go/src/runtime/checkptr.go:18 +0xbe fp=0xc00017dc70 sp=0xc00017dc40 pc=0x40689e
internal/poll.(*FD).WriteMsg(0xc000237400, 0xc00020cec0, 0xc, 0xc, 0x0, 0x0, 0x0, 0x6eeec0, 0xc00020f020, 0x0, ...)
        c:/users/alex/dev/go/src/internal/poll/fd_windows.go:1130 +0x1f3 fp=0xc00017dce0 sp=0xc00017dc70 pc=0x4d3fa3
net.(*netFD).writeMsg(0xc000237400, 0xc00020cec0, 0xc, 0xc, 0x0, 0x0, 0x0, 0x6eeec0, 0xc00020f020, 0xc00020f000, ...)
        c:/users/alex/dev/go/src/net/fd_windows.go:234 +0xb1 fp=0xc00017dd78 sp=0xc00017dce0 pc=0x5644d1
net.(*UDPConn).writeMsg(0xc0002100d8, 0xc00020cec0, 0xc, 0xc, 0x0, 0x0, 0x0, 0xc000235200, 0x80, 0x80, ...)
        c:/users/alex/dev/go/src/net/udpsock_posix.go:94 +0x121 fp=0xc00017ddf0 sp=0xc00017dd78 pc=0x586621
net.(*UDPConn).WriteMsgUDP(0xc0002100d8, 0xc00020cec0, 0xc, 0xc, 0x0, 0x0, 0x0, 0xc000235200, 0xf, 0x85eea9, ...)
        c:/users/alex/dev/go/src/net/udpsock.go:186 +0xac fp=0xc00017de90 sp=0xc00017ddf0 pc=0x58477c
net.TestUDPConnSpecificMethods(0xc000228a20)
        c:/users/alex/dev/go/src/net/protoconn_test.go:143 +0x4d5 fp=0xc00017df80 sp=0xc00017de90 pc=0x5d98b5
testing.tRunner(0xc000228a20, 0x6abdd0)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3 fp=0xc00017dfd0 sp=0xc00017df80 pc=0x4ff473
runtime.goexit()
        c:/users/alex/dev/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00017dfd8 sp=0xc00017dfd0 pc=0x467b11
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1043 +0x35e

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000228a20, 0x6a155a, 0x1a, 0x6abdd0, 0x4c8701)
        c:/users/alex/dev/go/src/testing/testing.go:1044 +0x385
testing.runTests.func1(0xc00012c000)
        c:/users/alex/dev/go/src/testing/testing.go:1300 +0x7f
testing.tRunner(0xc00012c000, 0xc000071dd0)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3
testing.runTests(0xc000004680, 0x887980, 0xbc, 0xbc, 0xbf8d542e05eb6248, 0x8bb3711725, 0x88b2a0, 0x10)
        c:/users/alex/dev/go/src/testing/testing.go:1298 +0x2df
testing.(*M).Run(0xc000112000, 0x0)
        c:/users/alex/dev/go/src/testing/testing.go:1210 +0x1ae
net.TestMain(0xc000112000)
        c:/users/alex/dev/go/src/net/main_test.go:52 +0x40
main.main()
        _testmain.go:478 +0x13c

goroutine 199 [chan receive]:
testing.(*T).Parallel(0xc0001a8fc0)
        c:/users/alex/dev/go/src/testing/testing.go:867 +0x1dc
net.TestLookupGoogleSRV(0xc0001a8fc0)
        c:/users/alex/dev/go/src/net/lookup_test.go:74 +0x4a
testing.tRunner(0xc0001a8fc0, 0x6aba50)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1043 +0x35e

goroutine 200 [chan receive]:
testing.(*T).Parallel(0xc0001a90e0)
        c:/users/alex/dev/go/src/testing/testing.go:867 +0x1dc
net.TestLookupGmailMX(0xc0001a90e0)
        c:/users/alex/dev/go/src/net/lookup_test.go:123 +0x47
testing.tRunner(0xc0001a90e0, 0x6aba20)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1043 +0x35e

goroutine 274 [chan receive]:
testing.(*T).Parallel(0xc00040c240)
        c:/users/alex/dev/go/src/testing/testing.go:867 +0x1dc
net.TestCloseUnblocksRead(0xc00040c240)
        c:/users/alex/dev/go/src/net/net_test.go:505 +0x32
testing.tRunner(0xc00040c240, 0x6ab628)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1043 +0x35e

goroutine 202 [chan receive]:
testing.(*T).Parallel(0xc0001a9320)
        c:/users/alex/dev/go/src/testing/testing.go:867 +0x1dc
net.TestLookupGmailTXT(0xc0001a9320)
        c:/users/alex/dev/go/src/net/lookup_test.go:218 +0x4a
testing.tRunner(0xc0001a9320, 0x6aba30)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1043 +0x35e

goroutine 201 [chan receive]:
testing.(*T).Parallel(0xc0001a9200)
        c:/users/alex/dev/go/src/testing/testing.go:867 +0x1dc
net.TestLookupGmailNS(0xc0001a9200)
        c:/users/alex/dev/go/src/net/lookup_test.go:169 +0x47
testing.tRunner(0xc0001a9200, 0x6aba28)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1043 +0x35e

goroutine 275 [chan receive]:
testing.(*T).Parallel(0xc00040c360)
        c:/users/alex/dev/go/src/testing/testing.go:867 +0x1dc
net.TestNotTemporaryRead(0xc00040c360)
        c:/users/alex/dev/go/src/net/net_test.go:530 +0x32
testing.tRunner(0xc00040c360, 0x6abb48)
        c:/users/alex/dev/go/src/testing/testing.go:992 +0xe3
created by testing.(*T).Run
        c:/users/alex/dev/go/src/testing/testing.go:1043 +0x35e
FAIL    net     4.524s
FAIL

c:\Users\Alex\dev\go\src>

I will try and fix that too.

Alex

alexbrainman picture alexbrainman on 25 Feb 2020

Change https://golang.org/cl/222457 mentions this issue: internal/syscall/windows: change WSAMsg.Name type

gopherbot picture gopherbot on 7 Mar 2020

Change https://golang.org/cl/222855 mentions this issue: sha3: mark xorInUnaligned with go:nocheckptr

gopherbot picture gopherbot on 11 Mar 2020

I think I finished all -d=checkptr related changes. Running (on top of 801cd7c )

c:\Users\Alex\dev\go\src>go test -a -short -gcflags=all=-d=checkptr std cmd
ok      archive/tar     0.648s
ok      archive/zip     0.170s
ok      bufio   0.089s
ok      bytes   0.325s
ok      compress/bzip2  0.287s
ok      compress/flate  1.147s
ok      compress/gzip   0.250s
ok      compress/lzw    0.371s
ok      compress/zlib   0.106s
ok      container/heap  0.118s
ok      container/list  0.244s
ok      container/ring  0.082s
ok      context 0.151s
ok      crypto  0.526s
ok      crypto/aes      0.220s
ok      crypto/cipher   0.554s
ok      crypto/des      0.259s
ok      crypto/dsa      0.108s
ok      crypto/ecdsa    0.235s
ok      crypto/ed25519  0.359s
?       crypto/ed25519/internal/edwards25519    [no test files]
ok      crypto/elliptic 0.208s
ok      crypto/hmac     0.833s
?       crypto/internal/randutil        [no test files]
ok      crypto/internal/subtle  0.724s
ok      crypto/md5      0.830s
ok      crypto/rand     0.146s
ok      crypto/rc4      0.127s
ok      crypto/rsa      0.257s
ok      crypto/sha1     0.116s
ok      crypto/sha256   0.132s
ok      crypto/sha512   0.079s
ok      crypto/subtle   0.074s
ok      crypto/tls      4.314s
ok      crypto/x509     1.605s
?       crypto/x509/pkix        [no test files]
ok      database/sql    0.762s
ok      database/sql/driver     0.053s
ok      debug/dwarf     1.120s
ok      debug/elf       1.395s
ok      debug/gosym     0.157s
ok      debug/macho     0.434s
ok      debug/pe        12.299s
ok      debug/plan9obj  0.182s
?       encoding        [no test files]
ok      encoding/ascii85        0.093s
ok      encoding/asn1   0.063s
ok      encoding/base32 0.120s
ok      encoding/base64 0.098s
ok      encoding/binary 0.056s
ok      encoding/csv    0.084s
ok      encoding/gob    0.505s
ok      encoding/hex    0.495s
ok      encoding/json   0.447s
ok      encoding/pem    0.140s
ok      encoding/xml    0.096s
ok      errors  0.110s
ok      expvar  0.206s
ok      flag    0.159s
ok      fmt     0.157s
ok      go/ast  0.059s
ok      go/build        19.119s
ok      go/constant     0.113s
ok      go/doc  1.058s
ok      go/format       0.075s
ok      go/importer     0.401s
ok      go/internal/gccgoimporter       2.147s
ok      go/internal/gcimporter  2.170s
ok      go/internal/srcimporter 3.413s
ok      go/parser       0.403s
ok      go/printer      0.569s
ok      go/scanner      0.094s
ok      go/token        0.158s
ok      go/types        4.196s
ok      hash    0.051s
ok      hash/adler32    0.102s
ok      hash/crc32      0.217s
ok      hash/crc64      0.159s
ok      hash/fnv        0.160s
ok      hash/maphash    0.178s
ok      html    0.048s
ok      html/template   0.139s
ok      image   0.312s
ok      image/color     0.139s
?       image/color/palette     [no test files]
ok      image/draw      0.250s
ok      image/gif       0.747s
?       image/internal/imageutil        [no test files]
ok      image/jpeg      0.573s
ok      image/png       0.268s
ok      index/suffixarray       0.274s
?       internal/bytealg        [no test files]
?       internal/cfg    [no test files]
ok      internal/cpu    0.111s
ok      internal/fmtsort        0.096s
?       internal/goroot [no test files]
?       internal/goversion      [no test files]
?       internal/lazyregexp     [no test files]
?       internal/lazytemplate   [no test files]
?       internal/nettrace       [no test files]
?       internal/obscuretestdata        [no test files]
?       internal/oserror        [no test files]
ok      internal/poll   0.165s
?       internal/race   [no test files]
ok      internal/reflectlite    0.130s
ok      internal/singleflight   0.107s
?       internal/syscall/execenv        [no test files]
ok      internal/syscall/windows        0.085s
ok      internal/syscall/windows/registry       0.207s
?       internal/syscall/windows/sysdll [no test files]
?       internal/testenv        [no test files]
?       internal/testlog        [no test files]
ok      internal/trace  0.737s
ok      internal/xcoff  1.169s
ok      io      0.239s
ok      io/ioutil       0.302s
ok      log     0.078s
?       log/syslog      [no test files]
ok      math    0.138s
ok      math/big        1.337s
ok      math/bits       0.116s
ok      math/cmplx      0.072s
ok      math/rand       0.416s
ok      mime    0.526s
ok      mime/multipart  0.931s
ok      mime/quotedprintable    0.067s
ok      net     32.163s
ok      net/http        14.201s
ok      net/http/cgi    0.608s
ok      net/http/cookiejar      0.125s
ok      net/http/fcgi   0.891s
ok      net/http/httptest       2.162s
ok      net/http/httptrace      0.289s
ok      net/http/httputil       0.451s
ok      net/http/internal       0.299s
ok      net/http/pprof  2.254s
ok      net/internal/socktest   0.061s
ok      net/mail        0.079s
ok      net/rpc 0.357s
ok      net/rpc/jsonrpc 0.649s
ok      net/smtp        0.323s
ok      net/textproto   0.116s
ok      net/url 0.113s
ok      os      7.549s
ok      os/exec 7.068s
ok      os/signal       2.068s
ok      os/user 0.121s
ok      path    0.071s
ok      path/filepath   1.260s
ok      plugin  0.114s
ok      reflect 1.269s
ok      regexp  0.754s
ok      regexp/syntax   0.519s
ok      runtime 99.905s
?       runtime/cgo     [no test files]
ok      runtime/debug   0.096s
ok      runtime/internal/atomic 0.116s
ok      runtime/internal/math   0.060s
ok      runtime/internal/sys    0.190s
ok      runtime/pprof   13.023s
ok      runtime/pprof/internal/profile  0.085s
?       runtime/race    [no test files]
ok      runtime/trace   0.765s
ok      sort    0.101s
ok      strconv 0.690s
ok      strings 0.268s
ok      sync    0.583s
ok      sync/atomic     0.117s
ok      syscall 0.072s
ok      testing 0.737s
?       testing/internal/testdeps       [no test files]
ok      testing/iotest  0.135s
ok      testing/quick   0.270s
ok      text/scanner    0.106s
ok      text/tabwriter  0.071s
ok      text/template   0.125s
ok      text/template/parse     0.055s
ok      time    2.523s
ok      unicode 0.173s
ok      unicode/utf16   0.193s
ok      unicode/utf8    0.062s
?       unsafe  [no test files]
?       vendor/golang.org/x/crypto/chacha20     [no test files]
?       vendor/golang.org/x/crypto/chacha20poly1305     [no test files]
?       vendor/golang.org/x/crypto/cryptobyte   [no test files]
?       vendor/golang.org/x/crypto/cryptobyte/asn1      [no test files]
?       vendor/golang.org/x/crypto/curve25519   [no test files]
?       vendor/golang.org/x/crypto/hkdf [no test files]
?       vendor/golang.org/x/crypto/internal/subtle      [no test files]
?       vendor/golang.org/x/crypto/poly1305     [no test files]
?       vendor/golang.org/x/net/dns/dnsmessage  [no test files]
?       vendor/golang.org/x/net/http/httpguts   [no test files]
?       vendor/golang.org/x/net/http/httpproxy  [no test files]
?       vendor/golang.org/x/net/http2/hpack     [no test files]
?       vendor/golang.org/x/net/idna    [no test files]
?       vendor/golang.org/x/net/nettest [no test files]
?       vendor/golang.org/x/sys/cpu     [no test files]
?       vendor/golang.org/x/text/secure/bidirule        [no test files]
?       vendor/golang.org/x/text/transform      [no test files]
?       vendor/golang.org/x/text/unicode/bidi   [no test files]
?       vendor/golang.org/x/text/unicode/norm   [no test files]
ok      cmd/addr2line   20.369s
ok      cmd/api 24.461s
?       cmd/asm [no test files]
?       cmd/asm/internal/arch   [no test files]
ok      cmd/asm/internal/asm    4.127s
?       cmd/asm/internal/flags  [no test files]
ok      cmd/asm/internal/lex    0.800s
?       cmd/buildid     [no test files]
?       cmd/cgo [no test files]
ok      cmd/compile     0.624s
?       cmd/compile/internal/amd64      [no test files]
?       cmd/compile/internal/arm        [no test files]
?       cmd/compile/internal/arm64      [no test files]
ok      cmd/compile/internal/gc 18.001s
ok      cmd/compile/internal/logopt     3.718s
?       cmd/compile/internal/mips       [no test files]
?       cmd/compile/internal/mips64     [no test files]
?       cmd/compile/internal/ppc64      [no test files]
?       cmd/compile/internal/riscv64    [no test files]
?       cmd/compile/internal/s390x      [no test files]
ok      cmd/compile/internal/ssa        0.668s
ok      cmd/compile/internal/syntax     0.188s
ok      cmd/compile/internal/test       0.140s [no tests to run]
ok      cmd/compile/internal/types      0.047s
?       cmd/compile/internal/wasm       [no test files]
?       cmd/compile/internal/x86        [no test files]
ok      cmd/cover       11.431s
?       cmd/dist        [no test files]
ok      cmd/doc 0.266s
ok      cmd/fix 6.998s
go test proxy running at GOPROXY=http://127.0.0.1:62204/mod
go proxy: no archive rsc.io v1.5.2: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.1.0: file does not exist
go proxy: no archive rsc.io v1.5.1: file does not exist
go proxy: no archive example.com/newcycle v1.0.0: file does not exist
go proxy: no archive rsc.io v1.5.2: file does not exist
--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/test_race_install_cgo (1.32s)
        script_test.go:205:
            # Tests Issue #10500 (1.244s)
            > [!race] skip
            > [!darwin] ! stale cmd/cgo  # The darwin builders are spuriously stale; see #33598.
            FAIL: testdata\script\test_race_install_cgo.txt:5: cmd/cgo is unexpectedly stale


FAIL
FAIL    cmd/go  201.196s
ok      cmd/go/internal/auth    0.349s
?       cmd/go/internal/base    [no test files]
?       cmd/go/internal/bug     [no test files]
ok      cmd/go/internal/cache   5.082s
?       cmd/go/internal/cfg     [no test files]
?       cmd/go/internal/clean   [no test files]
?       cmd/go/internal/cmdflag [no test files]
?       cmd/go/internal/doc     [no test files]
?       cmd/go/internal/envcmd  [no test files]
?       cmd/go/internal/fix     [no test files]
?       cmd/go/internal/fmtcmd  [no test files]
ok      cmd/go/internal/generate        0.479s
ok      cmd/go/internal/get     0.385s
?       cmd/go/internal/help    [no test files]
ok      cmd/go/internal/imports 0.065s
?       cmd/go/internal/list    [no test files]
ok      cmd/go/internal/load    0.224s
ok      cmd/go/internal/lockedfile      0.381s
ok      cmd/go/internal/lockedfile/internal/filelock    0.321s
?       cmd/go/internal/modcmd  [no test files]
ok      cmd/go/internal/modconv 0.412s
ok      cmd/go/internal/modfetch        1.165s
ok      cmd/go/internal/modfetch/codehost       0.241s
ok      cmd/go/internal/modfetch/zip_sum_test   0.772s
?       cmd/go/internal/modget  [no test files]
?       cmd/go/internal/modinfo [no test files]
ok      cmd/go/internal/modload 0.237s
ok      cmd/go/internal/mvs     0.109s
ok      cmd/go/internal/par     0.078s
ok      cmd/go/internal/renameio        15.631s
?       cmd/go/internal/robustio        [no test files]
?       cmd/go/internal/run     [no test files]
ok      cmd/go/internal/search  0.304s
?       cmd/go/internal/str     [no test files]
ok      cmd/go/internal/test    1.983s
?       cmd/go/internal/tool    [no test files]
ok      cmd/go/internal/txtar   0.180s
?       cmd/go/internal/version [no test files]
?       cmd/go/internal/vet     [no test files]
ok      cmd/go/internal/web     0.654s
ok      cmd/go/internal/work    0.682s
ok      cmd/gofmt       1.146s
?       cmd/internal/bio        [no test files]
?       cmd/internal/browser    [no test files]
ok      cmd/internal/buildid    1.306s
?       cmd/internal/diff       [no test files]
ok      cmd/internal/dwarf      0.085s
ok      cmd/internal/edit       0.134s
?       cmd/internal/gcprog     [no test files]
ok      cmd/internal/goobj      2.878s
?       cmd/internal/goobj2     [no test files]
ok      cmd/internal/moddeps    8.717s
ok      cmd/internal/obj        0.080s
?       cmd/internal/obj/arm    [no test files]
ok      cmd/internal/obj/arm64  0.592s
?       cmd/internal/obj/mips   [no test files]
ok      cmd/internal/obj/ppc64  0.404s
ok      cmd/internal/obj/riscv  0.258s
?       cmd/internal/obj/s390x  [no test files]
?       cmd/internal/obj/wasm   [no test files]
ok      cmd/internal/obj/x86    6.724s
ok      cmd/internal/objabi     0.079s
?       cmd/internal/objfile    [no test files]
ok      cmd/internal/src        0.081s
?       cmd/internal/sys        [no test files]
ok      cmd/internal/test2json  0.789s
--- FAIL: TestDWARF (0.40s)
    dwarf_test.go:39: cmd/link is stale - run go install cmd/link
FAIL
FAIL    cmd/link        19.864s
?       cmd/link/internal/amd64 [no test files]
?       cmd/link/internal/arm   [no test files]
?       cmd/link/internal/arm64 [no test files]
ok      cmd/link/internal/ld    20.019s
?       cmd/link/internal/loadelf       [no test files]
?       cmd/link/internal/loader        [no test files]
?       cmd/link/internal/loadmacho     [no test files]
?       cmd/link/internal/loadpe        [no test files]
?       cmd/link/internal/loadxcoff     [no test files]
?       cmd/link/internal/mips  [no test files]
?       cmd/link/internal/mips64        [no test files]
?       cmd/link/internal/objfile       [no test files]
?       cmd/link/internal/ppc64 [no test files]
?       cmd/link/internal/riscv64       [no test files]
?       cmd/link/internal/s390x [no test files]
ok      cmd/link/internal/sym   0.407s
?       cmd/link/internal/wasm  [no test files]
?       cmd/link/internal/x86   [no test files]
ok      cmd/nm  15.778s
ok      cmd/objdump     5.883s
ok      cmd/pack        7.192s
?       cmd/pprof       [no test files]
?       cmd/test2json   [no test files]
ok      cmd/trace       1.206s
?       cmd/vendor/github.com/google/pprof/driver       [no test files]
?       cmd/vendor/github.com/google/pprof/internal/binutils    [no test files]
?       cmd/vendor/github.com/google/pprof/internal/driver      [no test files]
?       cmd/vendor/github.com/google/pprof/internal/elfexec     [no test files]
?       cmd/vendor/github.com/google/pprof/internal/graph       [no test files]
?       cmd/vendor/github.com/google/pprof/internal/measurement [no test files]
?       cmd/vendor/github.com/google/pprof/internal/plugin      [no test files]
?       cmd/vendor/github.com/google/pprof/internal/report      [no test files]
?       cmd/vendor/github.com/google/pprof/internal/symbolizer  [no test files]
?       cmd/vendor/github.com/google/pprof/internal/symbolz     [no test files]
?       cmd/vendor/github.com/google/pprof/internal/transport   [no test files]
?       cmd/vendor/github.com/google/pprof/profile      [no test files]
?       cmd/vendor/github.com/google/pprof/third_party/d3       [no test files]
?       cmd/vendor/github.com/google/pprof/third_party/d3flamegraph     [no test files]
?       cmd/vendor/github.com/google/pprof/third_party/svgpan   [no test files]
?       cmd/vendor/github.com/ianlancetaylor/demangle   [no test files]
?       cmd/vendor/golang.org/x/arch/arm/armasm [no test files]
?       cmd/vendor/golang.org/x/arch/arm64/arm64asm     [no test files]
?       cmd/vendor/golang.org/x/arch/ppc64/ppc64asm     [no test files]
?       cmd/vendor/golang.org/x/arch/x86/x86asm [no test files]
?       cmd/vendor/golang.org/x/crypto/ed25519  [no test files]
?       cmd/vendor/golang.org/x/crypto/ed25519/internal/edwards25519    [no test files]
?       cmd/vendor/golang.org/x/crypto/ssh/terminal     [no test files]
?       cmd/vendor/golang.org/x/mod/internal/lazyregexp [no test files]
?       cmd/vendor/golang.org/x/mod/modfile     [no test files]
?       cmd/vendor/golang.org/x/mod/module      [no test files]
?       cmd/vendor/golang.org/x/mod/semver      [no test files]
?       cmd/vendor/golang.org/x/mod/sumdb       [no test files]
?       cmd/vendor/golang.org/x/mod/sumdb/dirhash       [no test files]
?       cmd/vendor/golang.org/x/mod/sumdb/note  [no test files]
?       cmd/vendor/golang.org/x/mod/sumdb/tlog  [no test files]
?       cmd/vendor/golang.org/x/mod/zip [no test files]
?       cmd/vendor/golang.org/x/sys/unix        [no test files]
?       cmd/vendor/golang.org/x/sys/windows     [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis       [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/internal/analysisflags   [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/internal/facts        [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/asmdecl        [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/assign [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/atomic [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/bools  [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/buildtag       [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/cgocall        [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/composite      [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/copylock       [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/ctrlflow       [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/errorsas       [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/httpresponse   [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/ifaceassert    [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/inspect        [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/internal/analysisutil      [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/loopclosure    [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/lostcancel     [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/nilfunc        [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/printf [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/shift  [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/stdmethods     [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/stringintconv  [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/structtag      [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/tests  [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/unmarshal      [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/unreachable    [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/unsafeptr      [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/passes/unusedresult   [no test files]
?       cmd/vendor/golang.org/x/tools/go/analysis/unitchecker   [no test files]
?       cmd/vendor/golang.org/x/tools/go/ast/astutil    [no test files]
?       cmd/vendor/golang.org/x/tools/go/ast/inspector  [no test files]
?       cmd/vendor/golang.org/x/tools/go/cfg    [no test files]
?       cmd/vendor/golang.org/x/tools/go/types/objectpath       [no test files]
?       cmd/vendor/golang.org/x/tools/go/types/typeutil [no test files]
?       cmd/vendor/golang.org/x/xerrors [no test files]
?       cmd/vendor/golang.org/x/xerrors/internal        [no test files]
ok      cmd/vet 29.924s
FAIL

c:\Users\Alex\dev\go\src>

There are still 2 failures. But I don't think they are related to -d=checkptr.

I am not sure what to do next. Should I send windows version of

https://go-review.googlesource.com/c/go/+/201783/

?

Thank you.

Alex

alexbrainman picture alexbrainman on 2 Apr 2020
🚀1 🎉1

I think I finished all -d=checkptr related changes.

Yay!

There are still 2 failures. But I don't think they are related to -d=checkptr.

Do they also happen without checkptr? I agree they certainly don't look related to checkptr, but the dashboard is clean for Windows right now.

I am not sure what to do next. Should I send windows version of https://go-review.googlesource.com/c/go/+/201783/ ?

That would be awesome.

aclements picture aclements on 2 Apr 2020

There are still 2 failures. But I don't think they are related to -d=checkptr.

Do those resolve if you run go install cmd without -d=checkptr before running go test?

bcmills picture bcmills on 2 Apr 2020

Change https://golang.org/cl/227003 mentions this issue: cmd/compile: enable -d=checkptr even on windows

gopherbot picture gopherbot on 4 Apr 2020

There are still 2 failures. But I don't think they are related to -d=checkptr.

Do they also happen without checkptr?

Do they also happen without checkptr?

Yes. Running

go test -a -short std cmd

fails in exactly the same way as

go test -a -short -gcflags=all=-d=checkptr std cmd

command.

... Should I send windows version of https://go-review.googlesource.com/c/go/+/201783/ ?

That would be awesome.

I sent https://golang.org/cl/227003 But it fails here when I run all.bat with this error:

--- FAIL: TestLogOpt (0.63s)
    --- FAIL: TestLogOpt/Copy (0.27s)
        --- FAIL: TestLogOpt/Copy/arm (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build210684795: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/arm64 (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build080355363: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/386 (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build929204411: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/amd64 (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build049316543: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/mips (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build251041127: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/mips64 (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build593677423: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/ppc64le (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build772328963: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/s390x (0.02s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build555385451: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
        --- FAIL: TestLogOpt/Copy/wasm (0.08s)
            logopt_test.go:199: [c:\Users\Alex\dev\go\bin\go.exe tool compile -p x -json=0,file://log/opt -o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.o C:\Users\Alex\AppData\Local\Temp\TestLogOpt659564243\copy.go]
            logopt_test.go:204: go: creating work dir: mkdir C:\WINDOWS\go-build914568571: Access is denied.
            logopt_test.go:131: -json=0,file://log/opt should have succeeded
            logopt_test.go:135: -json=0,file://log/opt missing expected log file
            logopt_test.go:138:
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":3,"character":2},"end":{"line":3,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:37: did not see phrase {"range":{"start":{"line":9,"character":2},"end":{"line":9,"character":2}},"severity":3,"code":"copy","source":"go compiler","message":"128 bytes"} in
            logopt_test.go:43: expected exactly 2 occurences of "code":"copy" in
FAIL
FAIL    cmd/compile/internal/logopt     0.741s

Mind you, I created CL on current tip. And current tip has new linker code. Maybe failure is related. I did not investigate it properly.

There are still 2 failures. But I don't think they are related to -d=checkptr.

Do those resolve if you run go install cmd without -d=checkptr before running go test?

Running

go install cmd

before

go test -a -short -gcflags=all=-d=checkptr std cmd

makes things worse (I am showing just errors)

--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/test_relative_import_dash_i (0.48s)
        script_test.go:205:
            # Relative imports in go test -i
            # Run tests outside GOPATH. (0.000s)
            # Check that it's safe to pass -i (which installs dependencies in $GOPATH/pkg) to go test. (0.463s)
            > ! stale runtime # don't let test -i overwrite runtime
            FAIL: testdata\script\test_relative_import_dash_i.txt:7: runtime is unexpectedly stale


    --- FAIL: TestScript/test_race_install_cgo (2.53s)
        script_test.go:205:
            # Tests Issue #10500 (2.500s)
            > [!race] skip
            > [!darwin] ! stale cmd/cgo  # The darwin builders are spuriously stale; see #33598.
            FAIL: testdata\script\test_race_install_cgo.txt:5: cmd/cgo is unexpectedly stale


    --- FAIL: TestScript/build_package_not_stale_trailing_slash (0.16s)
        script_test.go:205:
            # Tests Issue #12690 (0.160s)
            > [gccgo] skip 'gccgo does not have GOROOT'
            > ! stale runtime
            FAIL: testdata\script\build_package_not_stale_trailing_slash.txt:5: runtime is unexpectedly stale


FAIL
FAIL    cmd/go  195.196s

and

--- FAIL: TestDWARF (1.32s)
    dwarf_test.go:39: cmd/link is stale - run go install cmd/link
FAIL
FAIL    cmd/link        16.210s

All tested on top of 801cd7c84d4.

Alex

alexbrainman picture alexbrainman on 4 Apr 2020

I sent https://golang.org/cl/227003 But it fails here when I run all.bat with this error:

--- FAIL: TestLogOpt (0.63s)
    --- FAIL: TestLogOpt/Copy (0.27s)
        --- FAIL: TestLogOpt/Copy/arm (0.02s)

TestLogOpt failure is unrelated to CL 227003. I filled #38251 for that.

Alex

alexbrainman picture alexbrainman on 5 Apr 2020

And I submitted CL 227003. So I don't see what else to do for this current issue. Leaving it for others to close.

Alex

alexbrainman picture alexbrainman on 5 Apr 2020

Thanks @alexbrainman!

mdempsky picture mdempsky on 14 Apr 2020

Change https://golang.org/cl/228858 mentions this issue: syscall, internal/syscall/windows: remove utf16PtrToString parameter

gopherbot picture gopherbot on 19 Apr 2020

Change https://golang.org/cl/225418 mentions this issue: windows: fix -d=checkptr slice failures

gopherbot picture gopherbot on 22 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gopherbot picture gopherbot  路  3Comments
bradfitz picture bradfitz  路  3Comments
ashb picture ashb  路  3Comments
mingrammer picture mingrammer  路  3Comments
lkarlslund picture lkarlslund  路  3Comments