Sqlx: Get treats json.RawMessage like db.RawBytes

[chowey]

I have a struct where one of the fields is a json.RawMessage. When I use DB.Get() to populate it, the field is volatile and changes with more calls to DB.Get().

Here is a simple program to reproduce it.

package main

import (
    "encoding/json"
    "fmt"

    _ "github.com/go-sql-driver/mysql"
    "github.com/jmoiron/sqlx"
)

type Var struct {
    Raw json.RawMessage
}

func main() {
    db, err := sqlx.Open("mysql", "root@/")
    if err != nil {
        panic(err)
    }
    defer db.Close()

    var v Var
    if err = db.Get(&v, `SELECT '{"a":"b"}' AS raw`); err != nil {
        panic(err)
    }
    // first time works great
    fmt.Printf("v: %s\n", v.Raw)

    var q Var
    if err = db.Get(&q, `SELECT 'null' AS raw`); err != nil {
        panic(err)
    }
    // "v" has changed???
    fmt.Printf("v: %s\n", v.Raw)
}

The output I get is:

v: {"a":"b"}
v: null�

[jmoiron]

I believe this is a Go issue and I've submitted an issue for it. That ticket describes the problem in more detail; rest assured that this problem exists with db.QueryRow as well. Unfortunately, for now you'll have to use []byte to get a copy of that data, though it should not involve a copy to type-convert it into json.RawMessage.

Here's a test for sqlx_test.go that shows that this behavior is consistent across any named []byte type:

func TestIssue197(t *testing.T) {
    type mybyte []byte
    type Var struct{ Raw json.RawMessage }
    type Var2 struct{ Raw []byte }
    type Var3 struct{ Raw mybyte }
    RunWithSchema(defaultSchema, t, func(db *DB, t *testing.T) {
        var err error
        var v, q Var
        if err = db.Get(&v, `SELECT '{"a": "b"}' AS raw`); err != nil {
            t.Fatal(err)
        }
        fmt.Printf("%s: v %s\n", db.DriverName(), v.Raw)
        if err = db.Get(&q, `SELECT 'null' AS raw`); err != nil {
            t.Fatal(err)
        }
        fmt.Printf("%s: v %s\n", db.DriverName(), v.Raw)

        var v2, q2 Var2
        if err = db.Get(&v2, `SELECT '{"a": "b"}' AS raw`); err != nil {
            t.Fatal(err)
        }
        fmt.Printf("%s: v2 %s\n", db.DriverName(), v2.Raw)
        if err = db.Get(&q2, `SELECT 'null' AS raw`); err != nil {
            t.Fatal(err)
        }
        fmt.Printf("%s: v2 %s\n", db.DriverName(), v2.Raw)

        var v3, q3 Var3
        if err = db.QueryRow(`SELECT '{"a": "b"}' AS raw`).Scan(&v3.Raw); err != nil {
            t.Fatal(err)
        }
        fmt.Printf("v3 %s\n", v3.Raw)
        if err = db.QueryRow(`SELECT '{"c": "d"}' AS raw`).Scan(&q3.Raw); err != nil {
            t.Fatal(err)
        }
        fmt.Printf("v3 %s\n", v3.Raw)
        t.Fail()
    })
}

Output:

postgres: v  1": "b"}
postgres: v  1l
postgres: v2 {"a": "b"}
postgres: v2 {"a": "b"}
v3  1": "b"}
v3  1": "d"}
sqlite3: v {"a": "b"}
sqlite3: v {"a": "b"}
sqlite3: v2 {"a": "b"}
sqlite3: v2 {"a": "b"}
v3 {"a": "b"}
v3 {"a": "b"}
mysql: v {"a": "b"}
mysql: v null�
mysql: v2 {"a": "b"}
mysql: v2 {"a": "b"}
v3 {"a": "b"}
v3 {"c": "d"}

[jmoiron]

I'm going to close this issue because it's not an sqlx issue, however it does make some things in sqlx/types potentially dangerous to use, so I think I will document that this bug is there for certain Go versions.

[chowey]

Okay thanks for shedding light on this. In the meanwhile I am explicitly using []byte and converting to a json.RawMessage afterwards, as you suggested.

[gofish]

@jmoiron Are you sure the warnings are appropriate for types that implement Scanner? It would seem they are always passed driver memory and are expected to copy the []byte.

[jmoiron]

You're right, the types versions are not vulnerable because they implement Scanner, and other types that implement scanner are also not vulnerable. Any type foo []byte that is not implementing Scanner is vulnerable.

[gofish]

Cool. If you haven't, see my comment on golang/go#13905 for a workaround on scanning types that are convertible to []byte. Could sqlx detect non-scannable structs (edit: struct fields) that are so convertible, e.g. under fieldsByTraversal(), then convert the pointer value to *byte[] and pass that to Row.Scan(...)?

[kevinburke]

FYI - I submitted @jmoiron's patch, which has been included in Go 1.7, so you should be safe against this problem if you are using it!