go/types: incorrect behavior for string conversions of byte slices

[dotaheor]

What version of Go are you using (go version)?

go version go1.9.2 linux/amd64

Does this issue reproduce with the latest release?

yes

What did you do?

It looks the definitions for slice of bytes are not consistent between spec and the builtin docs.

In spec

A non-constant value x can be converted to type T in any of these cases: 
...
* x is a string and T is a slice of bytes or runes. 

In he builtin docs

The copy built-in function .... 
(As a special case, it also will copy bytes from a string to a slice of bytes.)

However:

package main

type T byte

func main() {
    var x []T
    str := "abc"
    x = []T(str) // okay
    copy(x, str) // arguments to copy have different element types: []T and string
    _ = append(x, str...) // cannot use <node SPTR> (type *uint8) as type *T in argument to runtime.memmove
}

[edit] It looks []T is treated as a slice of bytes in x = []T(str), but not in the copy and append calls.

What did you expect to see?

all fail to compile or all compile okay.

What did you see instead?

non-consistent behavior

[dotaheor]

And this is more weird (non-consistent in spec itself).

A non-constant value x can be converted to type T in any of these cases: 
...
* x is an integer or a slice of bytes or runes and T is a string type.
* x is a string and T is a slice of bytes or runes. 

.

package main

type T byte // same problem for rune

func main() {
    var x []T
    str := "abc"
    x = []T(str) // okay
    str = string(x) // cannot use x (type []T) as type []byte in argument to runtime.slicebytetostring
}

Or this is a compile bug?
It looks gc sometimes view []T as a byte slice type, sometimes not.

[edit]
So we need clarify which of the following two definitions for "slice of bytes" should be used.

1. a slice whose underlying type is []byte.
2. a slice whose element underlying type is byte.

If the first definition is adopted, then x = []T(str) shouldn't compile okay.
If the second definition is adopted, then all examples in this thread should compile okay.

[ianlancetaylor]

You say there is an inconsistency, but then you cite a section of the spec on explicit type conversions, and a section of the spec on the copy builtin. Those are two different things. The copy builtin does not do type conversions. As your examples show.

Closing because I see no bug here. I strongly encourage you to discuss these cases on golang-nuts before opening issues for them. Thanks.

[dotaheor]

It is not an issue about conversions. It is about the definition of "slice of bytes". Different official Go docs use different definitions.

[ianlancetaylor]

I see, I think.

[dotaheor]

It looks gccgo thinks a byte slice is a slice whose element underlying type is byte.
For most cases, gc think a byte slice is a slice whose underlying type is []byte.

[ianlancetaylor]

There should be no difference at all between gccgo and gc regarding byte slices. I don't see any mention of gccgo above; can you expand on what you mean?

[griesemer]

I think @dotaheor is on to something. Here's a slightly modified example based on the example above:

package main

func main() {
    x1 := []byte("foo")
    _ = string(x1)

    type T byte
    x2 := []T("foo")
    _ = string(x2)
}

go/types accepts this code w/o complaints. But gc complains with:

x.go:9:12: cannot use x2 (type []T) as type []byte in argument to runtime.slicebytetostring

So at the very least we have an inconsistency. Furthermore, if gc is correct (which I need to investigate), the error message is not very good: There's no mention of runtime in this code, so the error message shouldn't mention it either.

[griesemer]

More cases:

    x1 := []byte("foo")
    _ = string(x1)

    type T byte
    x2 := []T("foo")
    _ = string(x2)

    type S1 []byte
    x3 := S1("foo")
    _ = string(x3)

    type S2 []T
    x4 := S2("foo")
    _ = string(x4)

go/types accepts all of them. gc complains twice:

x.go:9:12: cannot use x2 (type []T) as type []byte in argument to runtime.slicebytetostring
x.go:17:12: cannot use x4 (type S2) as type []byte in argument to runtime.slicebytetostring

It looks to me that the spec is pretty clear with specific examples (https://tip.golang.org/ref/spec#Conversions). It does appear that gc is correct: A "slice of bytes" means any type whose underlying type is []byte in this case. go/types on the other hand accepts any slice type where the slice element type's underlying type is a byte, which seems incorrect according to the spec. (Whether the spec rule is too tight is a different question).

[griesemer]

I filed separate #23813 for a better gc error message.

[ianlancetaylor]

gccgo seems to accept this code without error.

[griesemer]

@ianlancetaylor The spec's examples (see 2. in https://tip.golang.org/ref/spec#Conversions) only shows examples with types with underlying type []byte, but one could also argue that perhaps examples are missing. It boils down to what exactly we mean with "slice of bytes". I filed #23814 for that.

[dotaheor]

Maybe not bad, the current gccgo implementation provides an inefficient way to convert values between []byte and []MyByte.

package main

// []byte -> []MyByte. Compiles ok for both gccgo and gc
func f1() {
    type MyByte byte
    bs := []byte{65, 66, 67}
    ts := []MyByte(string(bs))
    _ = ts
}

// []MyByte -> []byte. Only compile ok for gccgo
func f2() {
    type MyByte byte
    ts := []MyByte{65, 66, 67}
    bs := []byte(string(ts))
    _ = bs
}

func main(){}

[griesemer]

@dotaheor So does go/types. However, the spec does not explicitly permit this, it appears.

[dotaheor]

It is interesting that both gc and gccgo reflections think []MyByte and string are not ConvertibleTo each other.
So gccgo reflection implementation is inconsistent with the general routine.
And gc reflection implementation is half-inconsistent with the general routine.

```package main

import "reflect"
import "fmt"

type T byte // same problem for rune

func main() {
var x []T
str := "abc"
typA := reflect.TypeOf(str)
typB := reflect.TypeOf(x)
fmt.Println(typA.ConvertibleTo(typB)) // false
fmt.Println(typB.ConvertibleTo(typA)) // false
}
```

[dotaheor]

As Go spec mentions:

As a special case, append also accepts a first argument assignable to type []byte with a second argument of string type followed by .... This form appends the bytes of the string.

and

As a special case, copy also accepts a destination argument assignable to type []byte with a source argument of a string type. This form copies the bytes from the string into the byte slice.

The two places both clearly specify that the first argument must be of type []byte to form special cases.
So the following example clearly shows that gccgo violates the rules.

package main

func main() {
    type MyByte byte
    var bs []byte
    var mybs []MyByte
    var str = "abc"

    // gc and gccgo both accept
    copy(bs, str)
    bs = append(bs, str...)

    // gc denies, gccgo accepts
    copy(mybs, str)
    mybs = append(mybs, str...)
}

[griesemer]

@mdempsky Do you have any thoughts on this? (no rush)

[mdempsky]

@griesemer Yeah, I added some notes/comments about this issue just a few weeks ago at https://github.com/golang/go/issues/23814#issuecomment-536082226.

TL;DR: I think we should be liberal in interpreting "slice of bytes" and "slice of runes" to mean any slice type whose element type has underlying type byte or rune (respectively).