Go: encoding/json: decoding JSON into anonymous struct works in go 1.8.7 but fails silently in go 1.9.4

[DonkeyOatie]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.8.7 and 1.9.4

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="darwin"

What did you do?

When decoding json into an anonymous structure, the data is present when using an app built with go version 1.8.7 but empty when using go 1.9.7

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

https://play.golang.org/p/vPsyOX7wBea

Here is an example of how I discovered this using the req.Body, details in the README show the difference between the go 1.8 and 1.9 output. With help from eric_hill on IRC #go-nuts we narrowed it down the the above play link.

https://github.com/nkhumphreys/goreqbody

What did you expect to see?

localStr should be populated with the data decoded from the JSON

What did you see instead?

localStr was empty

[bradfitz]

With help from eric_hill on IRC #go-nuts we narrowed it down the the above play link.

You two both forgot to check for errors.

That might not necessarily matter, but it's a distracting bug report, since everybody will immediately assume the answer is in the ignored error.

[DonkeyOatie]

I can update the play link and check the error if you like, but there is not one. :)

https://play.golang.org/p/3ZxteJYqZMp

[bradfitz]

Well, we can investigate, but there's no fixing Go 1.9 at this point, and Go 1.10 is out, like, today, so probably not changing there either.

I see nothing in the Go 1.9 release notes about json or reflect.

Please test new Go releases early & often! :)

[DonkeyOatie]

Just an FYI: The same behaviour as go 1.9 is exhibited in go 1.10rc2

[pciet]

An exported field works: https://play.golang.org/p/0kdRTJY_Xp5

Capitalizing the type works: https://play.golang.org/p/4szuu5Zydg3

From https://golang.org/pkg/encoding/json/#Marshal:

Anonymous struct fields are usually marshaled as if their inner exported fields were fields in the outer struct, subject to the usual Go visibility rules amended as described in the next paragraph.

[0xmohit]

The behavior changed in b81735924936291303559fd71dabaa1aa88f57c5.

[bradfitz]

Hey @dsnet , is that expected? A pointer to a string is considered unexported?

[dsnet]

Yes, this is expected behavior.

According to the language spec:

An identifier may be exported to permit access to it from another package. An identifier is exported if both:

1. the first character of the identifier's name is a Unicode upper case letter (Unicode class "Lu"); and
2. the identifier is declared in the package block or it is a field name or method name.

All other identifiers are not exported.

Thus, embedding a builtin type results in a field name that is not exported (since all builtin have lowercase names). Since this is unexported, the json package documents that it does not handle this.

The json rule you quote does not apply here:

Anonymous struct fields are usually marshaled as if their inner exported fields were fields in the outer struct, subject to the usual Go visibility rules amended as described in the next paragraph.

The example you provided has an anonymous string, not a struct.

[dsnet]

Furthermore, in Go1.10 and on, it wouldn't be possible for json to even handle this without using unsafe since reflect reports this field as unexported.

The reason why reflect on Go1.9 and below could actually mutate this field was due to a compiler bug, fixed by @mdempsky in CL/60410.