Aws-sdk-go: No way to Add to a Set with dynamodb expression package

[mitchlloyd]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

v1.13.59

Version of Go (go version)?

1.9.6

What issue did you see?

There is no apparent way to add to a set using the expression package.

Steps to reproduce

expression.Add(expression.Name("myStringSet"), expression.Value(???)),

Passing a string for ??? is not valid:

ValidationException Invalid UpdateExpression: Incorrect operand type for operator or function; operator: ADD, operand type: STRING

There is no value we can pass here to get dynamodbattribute.Marshal to produce the necessary AttributeValue:

&dynamodb.AttributeValue{
  SS: aws.StringSlice([]string{"hello"}),
}

[xibz]

Hello @mitchlloyd, thank you for reaching out to us. I believe theSET operand needs to be used in this case. Have you tried that?

Please take a look at the docs here regarding lists.

There is also an example of appending to lists. Please let us know if you have any other questions.

[mitchlloyd]

Thanks for the follow up @xibz. I took a shot at using SET rather than ADD this morning. I suppose first I need to figure out what the DynamoDB syntax would be if I use SET rather than ADD to add an item to a StringSet.

Here is what I tried:

• Using list_append as shown in the Appending Elements to a List documentation where I add an SS type:

SET #n = list_append(#n, :v)
// Error: Incorrect operand type for operator of function; operator or function; list_append, operand type: SS

• Using list_append with an L type containing an S type:

SET #n = list_append(#n, :v)
// Error: The provided expression refers to an attribute that does not exist in the item.

The issue here seems to be that #n does not exist yet. So perhaps we can add to an existing StringSet but not create a new one with this operation. In contrast, ADD handles this case:

ADD #n :v

I'm not too confident that SET would work here even if I used 2 operations since the documentation in your second link seems to stress that list_append is only for List types (not Sets).

ListAppend() only supports DynamoDB List types

The behavior I want is what I see documented in Adding Elements To A Set

Assume that the Color attribute does not exist. The following AWS CLI example sets Color to a string set with two elements...

Now that Color exists, we can add more elements to it...

Using ADD seems to be working well for me, but I can't find a way to use it with the expression package.

[xibz]

@mitchlloyd - Ah! Forgot about the sets. Yea, this may be a bug. Let me try to reproduce it on my end.

[jasdel]

I think this is related to #1639 for using Sets with the SDK. By default the SDK's attribute value marshaler from go type has no information if a slice is a set or list. The SDK defaults to list.

You can override this functionality with the stringset struct tag on the member:

type Foo struct  {
    Field []string `dynamodbav:",stringset"`
}

Or implementing the Marshaler interfere.

type ExampleMarshaler struct {
    Value int
}
func (m *ExampleMarshaler)  MarshalDynamoDBAttributeValue(av *dynamodb.AttributeValue) error {
    n := fmt.Sprintf("%v", m.Value)
    av.N = &n
    return nil
}

Docs for the marshaler used by the expression builder. https://docs.aws.amazon.com/sdk-for-go/api/service/dynamodb/dynamodbattribute/

[xibz]

@mitchlloyd - I think jasdel's comment regarding adding string sets is going to be the best way to do that. The expression package looks to lack some functionality for specifying string sets. I'm going to mark this as a feature request for the expression package. Please see jasdel's example to get unblocked. If you have any other questions, please let us know

[mitchlloyd]

Implementing the Marshaler interface is a nice solution 👍

[jasdel]

Added an example to #2360 defining a string set custom marshaler.

[LewisLebentz]

@jasdel how do you use a custom marshaler? Been struggling with this for hours.

I have a value stored in a StringSet, I just want to append an item to the end of the list - I can't for the life of me work out how to do it in Go. All the other languages look simple!

Right now I have this:

input := &dynamodb.UpdateItemInput{
        ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
            ":v": {
                SS: aws.StringSlice([]string{"TEST"}),
            },

and my expression is:

UpdateExpression: aws.String("ADD Voucher :v")}

I'm thinking it might just be easier to store as a String and convert it to a Slice when I want to use it.

[btnguyen2k]

Any update on this?
The same issue with DELETE operation: currently no way to DELETE an entry from a SET.

[mousedownmike]

In case anyone else ends up here, it is possible to ADD and DELETE from a String Set (SS). The following functions will add and delete a single item setItem from a String Set. It would be great if this were possible via expressions but that doesn't appear to be the case.

// Adds "setItem" to the StringSet (SS) identified by "setAttribute" on the record with a
// a partition key of "keyAttribute" with the value of "key" in the Dynamo table "table".
func AddToSet(db *dynamodb.DynamoDB, table, keyAttribute, key, setAttribute, setItem string) error  {
    _, err := db.UpdateItem(&dynamodb.UpdateItemInput{
        ExpressionAttributeNames: map[string]*string{
            "#0": &setAttribute,
        },
        ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
            ":0": {SS: []*string{&setItem}},
        },
        Key: map[string]*dynamodb.AttributeValue{
            keyAttribute: {S: &key},
        },
        TableName:                 &table,
        UpdateExpression:          aws.String("ADD #0 :0"),
    })
    return err
}


// Deletes "setItem" from the StringSet (SS) identified by "setAttribute" on the record with a
// a partition key of "keyAttribute" with the value of "key" in the Dynamo table "table".
func DeleteFromSet(db *dynamodb.DynamoDB, table, keyAttribute, key, setAttribute, setItem string) error  {
    _, err := db.UpdateItem(&dynamodb.UpdateItemInput{
        ExpressionAttributeNames: map[string]*string{
            "#0": &setAttribute,
        },
        ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
            ":0": {SS: []*string{&setItem}},
        },
        Key: map[string]*dynamodb.AttributeValue{
            keyAttribute: {S: &key},
        },
        TableName:                 &table,
        UpdateExpression:          aws.String("DELETE #0 :0"),
    })
    return err
}

[a-h]

@mousedownmike - I thought it was impossible with expressions too, but when I dug into the innards of the library (https://github.com/aws/aws-sdk-go/issues/3264) I found it was.

Here's a complete standalone example:

package main

import (
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/dynamodb"
    "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
    "github.com/aws/aws-sdk-go/service/dynamodb/expression"
    "github.com/google/go-cmp/cmp"
    "github.com/google/uuid"
)

const region = "eu-west-1"

func main() {
    sess, err := session.NewSession(&aws.Config{Region: aws.String(region)})
    if err != nil {
        log.Fatalf("failed to create session: %v", err)

    }
    client := dynamodb.New(sess)
    // To use DynamoDB in AWS, comment out this line.
    client.Endpoint = "http://localhost:8000"
    table, err := createTable(client)
    if err != nil {
        log.Fatalf("failed to create table: %v", err)
    }
    // Uncomment out this line to delete the table after the test.
    // defer deleteTable(client, table)

    // Run the test.

    id := "id1"
    record := Record{
        ID:     id,
        Groups: []string{"a", "b"},
    }
    err = Put(client, table, record)
    if err != nil {
        log.Printf("failed to put record: %v", err)
    }
    err = AddToGroups(client, table, id, []string{"b", "c", "d"})
    if err != nil {
        log.Printf("failed to add to groups [b, c, d]: %v", err)
    }
    err = RemoveFromGroups(client, table, id, []string{"b", "c"})
    if err != nil {
        log.Printf("failed to remove from groups [b, c]: %v", err)
    }
    actual, err := Get(client, table, id)
    if err != nil {
        log.Printf("failed to get record: %v", err)
    }
    expected := []string{"a", "d"}
    if diff := cmp.Diff(expected, actual.Groups); diff != "" {
        log.Println("Unexpected groups...")
        log.Println(diff)
    }
    fmt.Printf("%+v\n", actual)
}

type Record struct {
    ID     string   `json:"id"`
    Groups []string `json:"groups" dynamodbav:"groups,stringset"`
}

func createTable(client *dynamodb.DynamoDB) (name string, err error) {
    name = uuid.New().String()
    _, err = client.CreateTable(&dynamodb.CreateTableInput{
        AttributeDefinitions: []*dynamodb.AttributeDefinition{
            {
                AttributeName: aws.String("id"),
                AttributeType: aws.String("S"),
            },
        },
        KeySchema: []*dynamodb.KeySchemaElement{
            {
                AttributeName: aws.String("id"),
                KeyType:       aws.String("HASH"),
            },
        },
        BillingMode: aws.String(dynamodb.BillingModePayPerRequest),
        TableName:   aws.String(name),
    })
    return
}

func deleteTable(client *dynamodb.DynamoDB, name string) (err error) {
    _, err = client.DeleteTable(&dynamodb.DeleteTableInput{
        TableName: aws.String(name),
    })
    return
}

func Put(client *dynamodb.DynamoDB, tableName string, record Record) error {
    item, err := dynamodbattribute.MarshalMap(record)
    if err != nil {
        return err
    }
    _, err = client.PutItem(&dynamodb.PutItemInput{
        TableName: aws.String(tableName),
        Item:      item,
    })
    return err
}

func AddToGroups(client *dynamodb.DynamoDB, tableName, id string, groups []string) error {
    addSet := (&dynamodb.AttributeValue{}).SetSS(aws.StringSlice(groups))
    update := expression.Add(expression.Name("groups"), expression.Value(addSet))
    expr, err := expression.NewBuilder().
        WithUpdate(update).
        Build()
    if err != nil {
        return err
    }
    _, err = client.UpdateItem(&dynamodb.UpdateItemInput{
        TableName: &tableName,
        Key: map[string]*dynamodb.AttributeValue{
            "id": {S: aws.String(id)},
        },
        ExpressionAttributeNames:  expr.Names(),
        ExpressionAttributeValues: expr.Values(),
        UpdateExpression:          expr.Update(),
    })
    return err
}

func RemoveFromGroups(client *dynamodb.DynamoDB, tableName, id string, groups []string) error {
    deleteSet := (&dynamodb.AttributeValue{}).SetSS(aws.StringSlice(groups))
    update := expression.Delete(expression.Name("groups"), expression.Value(deleteSet))
    expr, err := expression.NewBuilder().
        WithUpdate(update).
        Build()
    if err != nil {
        return err
    }
    _, err = client.UpdateItem(&dynamodb.UpdateItemInput{
        TableName: &tableName,
        Key: map[string]*dynamodb.AttributeValue{
            "id": {S: aws.String(id)},
        },
        ExpressionAttributeNames:  expr.Names(),
        ExpressionAttributeValues: expr.Values(),
        UpdateExpression:          expr.Update(),
    })
    return err
}

func Get(client *dynamodb.DynamoDB, tableName, id string) (r Record, err error) {
    gio, err := client.GetItem(&dynamodb.GetItemInput{
        TableName: &tableName,
        Key: map[string]*dynamodb.AttributeValue{
            "id": {S: aws.String(id)},
        },
    })
    if err != nil {
        return
    }
    err = dynamodbattribute.UnmarshalMap(gio.Item, &r)
    return
}