Aws-sdk-go: DynamoDB attribute unmarshal for Scan api

[kfir-stratoscale]

The DynamoDB Scan api returns ScanOutput which in turn includes Items []map[string]*AttributeValue type:"list".
There is no method in unmarshal to handle it.
Unmarshal, UnmarshalMap,UnmarshalList don't support it...
Some feature to unmarshal the scan results will be helpful.

[xibz]

Hello @kfir-stratoscale, thank you for reaching out to us. I will take a look into this.

[xibz]

@kfir-stratoscale - What version of the SDK are you using? I have just tried to reproduce this and cannot. What are you doing that is causing an error?

[kfir-stratoscale]

Using version aws-sdk v1.3.0.

In aws-sdk-go/service/dynamodb/api.go:5451 we have the following definition of ScanOutput

// Represents the output of a Scan operation.
type ScanOutput struct {
    _ struct{} `type:"structure"`

    // The capacity units consumed by an operation. The data returned includes the
    // total provisioned throughput consumed, along with statistics for the table
    // and any indexes involved in the operation. ConsumedCapacity is only returned
    // if the request asked for it. For more information, see Provisioned Throughput
    // (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ProvisionedThroughputIntro.html)
    // in the Amazon DynamoDB Developer Guide.
    ConsumedCapacity *ConsumedCapacity `type:"structure"`

    // The number of items in the response.
    //
    // If you set ScanFilter in the request, then Count is the number of items
    // returned after the filter was applied, and ScannedCount is the number of
    // matching items before the filter was applied.
    //
    // If you did not use a filter in the request, then Count is the same as ScannedCount.
    Count *int64 `type:"integer"`

    // An array of item attributes that match the scan criteria. Each element in
    // this array consists of an attribute name and the value for that attribute.
    Items []map[string]*AttributeValue `type:"list"`

    // The primary key of the item where the operation stopped, inclusive of the
    // previous result set. Use this value to start a new operation, excluding this
    // value in the new request.
    //
    // If LastEvaluatedKey is empty, then the "last page" of results has been processed
    // and there is no more data to be retrieved.
    //
    // If LastEvaluatedKey is not empty, it does not necessarily mean that there
    // is more data in the result set. The only way to know when you have reached
    // the end of the result set is when LastEvaluatedKey is empty.
    LastEvaluatedKey map[string]*AttributeValue `type:"map"`

    // The number of items evaluated, before any ScanFilter is applied. A high ScannedCount
    // value with few, or no, Count results indicates an inefficient Scan operation.
    // For more information, see Count and ScannedCount (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#Count)
    // in the Amazon DynamoDB Developer Guide.
    //
    // If you did not use a filter in the request, then ScannedCount is the same
    // as Count.
    ScannedCount *int64 `type:"integer"`
}

As you can see the member Items is []map[string]*AttributeValue. That is a list of items like we can get from GetItem (Item map[string]*AttributeValue in GetItemOutput struct).

The following code is taken from a project that I'm working on to unmarshal items returned from GetItem

// DynamoObject is a generic type used to describe objects in dynamodb
type DynamoObject map[string]*dynamodb.AttributeValue

// GetItem gets an item by its key from dynamo.
// Note that the data will be placed in the given item instance.
func (s *DynamoSvc) GetItem(item interface{}) error {
    dynamoObj, err := marshalDynamoObject(item)
    if err != nil {
        return fmt.Errorf("failed to create & marshal dynamo object: %s", err)
    }

    lookupFields := make(DynamoObject, len(s.keys))
    for _, key := range s.keys {
        lookupFields[key] = dynamoObj[key]
    }

    params := &dynamodb.GetItemInput{
        Key:       lookupFields,
        TableName: s.table,
    }
    resp, err := s.db.GetItem(params)
    if err != nil {
        return fmt.Errorf("failed to get item from dynamo: %s", err)
    }

    if err = unmarshalDynamoObject(resp.Item, item); err != nil {
        return fmt.Errorf("failed to unmarshal the response from dynamo: %s", err)
    }

    return nil
}

// MarshalDynamoObject will convert a given structure to a dynamo object
func marshalDynamoObject(entry interface{}) (DynamoObject, error) {
    av, err := dynamodbattribute.Marshal(entry)
    if err != nil {
        return nil, fmt.Errorf("failed to marshal dynamo object: %s", err)
    }
    return av.M, err
}

// UnmarshalDynamoObject will convert a given dynamo object to a given structure.
func unmarshalDynamoObject(obj DynamoObject, entry interface{}) error {
    err := dynamodbattribute.Unmarshal(&dynamodb.AttributeValue{M: obj}, entry)
    return err
}

Pretty easy to use and a simple wrapper for the GetItem from dynamo.
For scan I ended up with a far more complex implementation due to the lacking feature in the unmarshaling library.

// GetAll returns all the items in the table.
// Note: if the table is big (over a few MB of data) only partial results will be returned with this call
func (s *DynamoSvc) GetAll(refItem interface{}) (interface{}, error) {
    params := &dynamodb.ScanInput{
        TableName: s.table,
    }
    resp, err := s.db.Scan(params)
    if err != nil {
        return nil, fmt.Errorf("failed to get all items from dynamo: %s", err)
    }

    refType := reflect.TypeOf(refItem)
    results := reflect.MakeSlice(reflect.SliceOf(refType), 0, int(*resp.Count))
    for _, item := range resp.Items {
        newItem := reflect.Indirect(reflect.New(refType.Elem()))
        if err = unmarshalDynamoObject(item, newItem.Addr().Interface()); err != nil {
            return nil, fmt.Errorf("failed to unmarshal the response from dynamo: %s", err)
        }
        results = reflect.Append(results, newItem.Addr())
    }

    return results.Interface(), nil
}

This implementation is not optimal but it explains my use case pretty well I think..
I just want to call Scan and get a slice of objects that use. Similar to GetItem.

[xibz]

@kfir-stratoscale - I see what you're talking about. I will mark this as a feature request. Thank you again for reaching out to us to improve the SDK! If you have any additional questions or feedback, please reach out.

[jasdel]

Thanks for bringing up this issue. Change #897 Fixes this issue by adding a UnmarshalListOfMaps utility to the dynamodbattribute package. This will make using Scan and Query easier since you'll be able to unmarshal Items now without extra logic.