Amplify-js: Data live in DynamoDB however AppSync list query doesn't show some of the data

[YikSanChan]

Describe the bug
I have an AWS Amplify app in which I add a GraghQL backend and create the model like this:

type Item @model @auth(rules: [{allow: owner}]) {
  id: ID!
  inventoryType: String!
  itemName: String!
  imgKey: String
}

Inside my app I allow users to create an item via submitting a form, by calling API.graphqlOperation(createItemMutation, newItem). This works well, because after users submit the form, I can find new items be created in DynamoDB.

After this, I go to AppSync console, I login via Cognito user pool as user A, the following query doesn't show some of created items:

query {
  listItems {
    items {
      id
      inventoryType
      itemName
    }
  }
}

What I get:

{
  "data": {
    "listItems": {
      "items": [
        {
          "itemName": "2015 book"
        },
        {
          "itemName": "Sundaya 0845"
        },
        ...
      ]
    }
  }
}

Some newly created item will be shown, while some not (e.g., one with item name "Sunday 0845 2") - and I guarantee they are from the same user so that auth is not the root cause. I cannot figure out the show/not show pattern. However, I can always show the created item by get with id:

query {
  getItem(id: "72eee5e9-c1f5-4860-8a52-d846ec7d7a54") {
    id
    inventoryType
    itemName
  }
}

What I get:

 {
  "data": {
    "getItem": {
      "id": "72eee5e9-c1f5-4860-8a52-d846ec7d7a54",
      "inventoryType": "apparels",
      "itemName": "Sunday 0845 2"
    }
  }
}

To Reproduce
I am not sure how can one reproduce. The result is somewhat un-deterministic.

You can turn on the debug mode to provide more info for us by setting window.LOG_LEVEL = 'DEBUG'; in your app.

[manueliglesias]

Hi @YikSanChan

It could be that you need to paginate through the results of your listItems query. Relations and pagination might be a good way to start.

Also take a look at this helpful medium post. It has this helpful bit:

By default the Amplify CLI configured resources on AWS AppSync will limit the scan operation to 10 items, that means DynamoDB will read 10 items from the table. The default limit is 10 to be as conservative as possible with billed operations. In case you have more than 10 notes from different users in the DynamoDB table, some of them might not be displayed in the frontend. This limit can be easily increased by updating the first line of the listNotes query resolver at the AppSync console (#set( $limit = $util.defaultIfNull($context.args.limit,10))), by passing the limit in the GraphQL query itself (e.g. “limit:50”) or you could just implement pagination in the application (see below) instead. For the sake of simplicity, we’re using the defaults.

Let us know if pagination was not the issue

[YikSanChan]

Pagination solves part of the issue, but there are still some items that don't show up in the result of list items query, even though the total number of result is less than 10.

In DynamoDB,

However in AppSync console, 3 items are missing:

query {
  listItems(filter: {inventoryType: {eq: "books"}}) {
    items {
      itemName
    }
  }
}

// response
{
  "data": {
    "listItems": {
      "items": [
        {
          "itemName": "Sunday 2015 book"
        }
      ]
    }
  }
}

Thanks for helping!

[manueliglesias]

@YikSanChan Can you paste the request mapping template for your query resolver?

[YikSanChan]

What do you mean by "request mapping template"? Can you please give me a sample?

I copy-paste the schema from AppSync console here. Let me know if it doesn't help: https://gist.github.com/YikSanChan/cc8dca2cffa410a9354bbd9112563fe3

[manueliglesias]

In the AppSync console, click on your API, then go to "Schema", and on the right side you'll see a "Resolvers" section. Look for your query in here and click on its resolver. You'll see both the request and response mapping templates

[YikSanChan]

request mapping template:

#set( $limit = $util.defaultIfNull($context.args.limit, 10) )
{
  "version": "2017-02-28",
  "operation": "Scan",
  "filter":   #if( $context.args.filter )
$util.transform.toDynamoDBFilterExpression($ctx.args.filter)
  #else
null
  #end,
  "limit": $limit,
  "nextToken":   #if( $context.args.nextToken )
"$context.args.nextToken"
  #else
null
  #end
}

response mapping template in case it is useful:

## No Static Group Authorization Rules **


## [Start] If not static group authorized, filter items **
#if( ! $isStaticGroupAuthorized )
  #set( $items = [] )
  #foreach( $item in $ctx.result.items )
    ## No Dynamic Group Authorization Rules **


    ## [Start] Owner Authorization Checks **
    #set( $isLocalOwnerAuthorized = false )
    ## Authorization rule: { allow: "owner", ownerField: "owner", identityField: "cognito:username" } **
    #set( $allowedOwners0 = $item.owner )
    #set( $identityValue = $util.defaultIfNull($ctx.identity.claims.get("username"), $util.defaultIfNull($ctx.identity.claims.get("cognito:username"), "___xamznone____")) )
    #if( $util.isList($allowedOwners0) )
      #foreach( $allowedOwner in $allowedOwners0 )
        #if( $allowedOwner == $identityValue )
          #set( $isLocalOwnerAuthorized = true )
        #end
      #end
    #end
    #if( $util.isString($allowedOwners0) )
      #if( $allowedOwners0 == $identityValue )
        #set( $isLocalOwnerAuthorized = true )
      #end
    #end
    ## [End] Owner Authorization Checks **


    #if( ($isLocalDynamicGroupAuthorized == true || $isLocalOwnerAuthorized == true) )
      $util.qr($items.add($item))
    #end
  #end
  #set( $ctx.result.items = $items )
#end
## [End] If not static group authorized, filter items **

$util.toJson($ctx.result)

[YikSanChan]

Hey, @manueliglesias I believe I figured out the reason.

I list items filtered by inventory type and I expect to apply the limit after I get all items fit the criteria ({inventoryType: {eq: "books"}}). However, dynamodb apply the limit before I get all items. Same issue as described here.

Any walkaround other than setting a very large limit and pray it works? Thanks!

[cj-l]

@manueliglesias @YikSanChan I'm new to all this so apologies if I have this confused. I have set a limit well beyond the size of my table (so that all items would be retrieved before the filter is applied), but still do not get all the results matching my filter (I get 4 of the 18 that are in the table). Would paginating results yield more items than setting a large limit, or are these the same thing?

[artista7]

This has to be a joke. Working with dynamoDb along with appsync in amplify always seem to be doing un-intuitive things. If i'm passing filter expression along with limit, i would want to limit the filtered result instead of filtering the limited result. Can anyone provide an alternate resolver mappings? @YikSanChan

Current default mapping templates in amplify are : -
query.listRounds.req.vtl:-

set( $limit = $util.defaultIfNull($context.args.limit, 10) )

set( $ListRequest = {

"version": "2017-02-28",
"limit": $limit
} )

if( $context.args.nextToken )

#set( $ListRequest.nextToken = "$context.args.nextToken" )

end

if( $context.args.filter )

#set( $ListRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") )

end

if( !$util.isNull($modelQueryExpression) && !$util.isNullOrEmpty($modelQueryExpression.expression) )

$util.qr($ListRequest.put("operation", "Query"))
$util.qr($ListRequest.put("query", $modelQueryExpression))

else

$util.qr($ListRequest.put("operation", "Scan"))

end

$util.toJson($ListRequest)

and
query.listRounds.res.vtl:-
$util.toJson($ctx.result)

[GeeeCoin]

Agreed @artista7. it's a critical and common use case that we'd want to sort by the most recent 1000 items and have that returned. Postfiltering will result in gaps in our users' histories that will make them (and us) pretty unhappy.
1) Even if we only wanted to return below max limit, say 100 items, results will still suffer from postfiltering, meaning even more gaps in history.
2) I don't want to increase coding complexity to paginate the results because I don't need to.
@manueliglesias thanks

[chris2k2]

Why is this closed? I have the same issue as @s-hood. I have set the limit way above my table size and can't find certain results. Any ideas?

[zgr024]

Same issue. I have a table with 7300 "items". I can't get any of the records from the set I need in a single query because there's a limit to how many records I can look at before applying the filter? That makes no sense.

[justhamade]

Not sure why no one else commented but this is how DynamoDB filter expressions work and has nothing to do with amplify.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.Limit

Now suppose that you add a filter expression to the Scan. In this case, DynamoDB applies the filter expression to the six items that were returned, discarding those that do not match. The final Scan result contains six items or fewer, depending on the number of items that were filtered.

[gabehollombe-aws]

This could be because AppSync limits the amount of iterations for VTL forEach loops to 1,000. So, limit values higher than this won't have any effect. Instead, you should keep limit to 1,000 and paginate through your results with the nextToken value returned in the response.

[shahid285]

Hi All,
I am new to AppSync, and i have noticed the same phenomenon while using DynamoDB as a datasource, wherein i have a total rows of 135 as a sample data.
When i am running a listAll query, i see very random data on the screen limited to 20 records.
When i investigated by adding the limit argument as 150, i am getting all the data.
If i keep the limit to 2 or 3 and trying to add a filter and query the results, i am not getting any.
When i again went on to check the Request Mapping template, i found the default value for limit is being passed as 20. Hence we see random 20 records.
"limit": $util.defaultIfNull($ctx.args.limit, 20)
So, if your filter has the record details of any of these 20 records, you will get the result. Anything else from you DynamoDB, it won't be returned.
If you remove this limit attribute from RequestMappingTemplate, you are greeted with all the data, but anytime if you pass the limit will not be accounted, and you will get all the results.

What i see from this exercise is the limit is directly associated to, something like row index.

The reason i say this is, say suppose you are running the query with filter, and you are not getting any results, but a nextToken is available.
If you query again with this nextToken, with limit, say 2, you wont be getting any results again, if these records you are looking for doesn't fall under the 20 records which Query retrieves always.

I also checked the same in the DynamoDB , i am not having any such issues, while filtering.

[jcorrea85]

I was having the same issue and solved it by getting better understanding of the difference between the list and get queries.
On my particular case I changed my query from:
listOrderss(filter:{id: {eq: $id}}, limit: 11) {
to:
getOrders(id:$id){
What you may need to understand is that listOrderss above is returning all entries, and then it is filtering the result; Instead you should aim to only return results that match your query using the get query.

This article helped me understand 'filter' much better: https://www.dynamodbguide.com/filtering

[zgr024]

Thank you

On Sun, Jul 19, 2020, 6:10 PM jcorrea85 [e-mail] wrote:

I was having the same issue and solved it by getting better understanding
of the difference between the list and get queries.
On my particular case I changed my query from:
listOrderss(filter:{id: {eq: $id}}, limit: 11) {
to:
getOrders(id:$id){
What you may need to understand is that listOrderss above is returning all
entries, and then it is filtering the result; Instead you should aim to
only return results that match your query using the get query.

This article helped me understand 'filter' much better:
https://www.dynamodbguide.com/filtering

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws-amplify/amplify-js/issues/2233#issuecomment-660717095,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFOHN575NF2HK5YJCNSSTLR4NVLZANCNFSM4GHV7YEA
.