Gatsby: Issues with sort fields

[stefanprobst]

Some issues with sort fields:

• fields linking to other nodes with the ___NODE naming convention are included as is in the sort fields enum. This means (i) we cannot sort by fields on the linked node; (ii) we also cannot sort by the foreign-key value on the ___NODE field, because we replace triple underscores with dots. We have to do this because we unnecessarily store the underscored field names as enum values. (here and here)
• sorting on arrays of objects is not possible. when collecting sort fields we assume that arrays are arrays of scalars. See also this test for an example why this is unexpected (we cannot sort on objects without also specifying a property).
• although we include fields added with the setFieldsOnGraphQLNodeType API in the sort fields enum, we cannot sort on those fields, because we don't call field resolvers on sort fields before querying (only for filter fields Fixed; the same issue is probably true for distinct and group field arg)

[sidharthachatterjee]

cc @freiksenet

[stefanprobst]

Note that all of this is working as expected in the schema refactor branch - I noticed those issues when porting over the old tests, and noted them here so there is something public to refer to.

[zslabs]

Does this recent change have anything to do with it? https://github.com/gatsbyjs/gatsby/pull/11381 gatsby": "2.0.96" works fine, but somewhere between then and now... something happened. This feels like a pretty high-priority bug since sorting by a field in graphql queries is busted.

[stefanprobst]

@zslabs Could you specify what is broken for you? The sort examples in the graphql reference docs seem to work.

[zslabs]

Hey @stefanprobst,

sort: { fields: [frontmatter___date], order: DESC } within allMarkdownRemark is what appears to be broken in the newer versions of Gatsby. 2.0.96 does not have this issue; but I also downgraded gatsby-transformer-remark": "2.2.3" to gatsby-transformer-remark": "2.2.0" as well if that helps shed any light.

[stefanprobst]

@zslabs I'll look into it.

[stefanprobst]

@zslabs Ok, this needs more investigation. In a basic hello-world starter (with [email protected]), sorting on markdown frontmatter fields does work on my end. Would you be able to share a repo to reproduce the error?

[zslabs]

@zslabs Ok, this needs more investigation. In a basic hello-world starter (with [email protected]), sorting on markdown frontmatter fields does work on my end. Would you be able to share a repo to reproduce the error?

Unfortunately no; as it's a private repo. Here's a rundown of my gatsby info though from when the bug was occurring.

  System:
    OS: macOS 10.14.2
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Shell: 3.0.0 - /usr/local/bin/fish
  Binaries:
    Node: 8.12.0 - /usr/local/bin/node
    Yarn: 1.12.1 - ~/.yarn/bin/yarn
    npm: 6.5.0 - ~/.npm-packages/bin/npm
  Browsers:
    Chrome: 71.0.3578.98
    Firefox: 65.0
    Safari: 12.0.2
  npmPackages:
    gatsby: 2.0.111 => 2.0.111
    gatsby-image: 2.0.29 => 2.0.29
    gatsby-plugin-canonical-urls: 2.0.10 => 2.0.10
    gatsby-plugin-catch-links: 2.0.10 => 2.0.10
    gatsby-plugin-feed: 2.0.13 => 2.0.13
    gatsby-plugin-google-tagmanager: 2.0.9 => 2.0.9
    gatsby-plugin-manifest: 2.0.17 => 2.0.17
    gatsby-plugin-netlify: 2.0.8 => 2.0.8
    gatsby-plugin-netlify-cms: 3.0.12 => 3.0.12
    gatsby-plugin-node-fields: 2.0.1 => 2.0.1
    gatsby-plugin-react-helmet: 3.0.6 => 3.0.6
    gatsby-plugin-sass: 2.0.10 => 2.0.10
    gatsby-plugin-sharp: 2.0.20 => 2.0.20
    gatsby-plugin-sitemap: 2.0.5 => 2.0.5
    gatsby-plugin-twitter: 2.0.9 => 2.0.9
    gatsby-remark-autolink-headers: 2.0.13 => 2.0.13
    gatsby-remark-copy-linked-files: 2.0.9 => 2.0.9
    gatsby-remark-custom-blocks: 2.0.5 => 2.0.5
    gatsby-remark-embed-video: 1.7.0 => 1.7.0
    gatsby-remark-images: 3.0.3 => 3.0.3
    gatsby-remark-prismjs: 3.2.4 => 3.2.4
    gatsby-remark-relative-images: 0.2.1 => 0.2.1
    gatsby-remark-relative-links: 0.0.3 => 0.0.3
    gatsby-remark-responsive-iframe: 2.0.9 => 2.0.9
    gatsby-source-filesystem: 2.0.20 => 2.0.20
    gatsby-transformer-remark: 2.2.3 => 2.2.3
    gatsby-transformer-sharp: 2.1.13 => 2.1.13
    gatsby-transformer-yaml: 2.1.8 => 2.1.8
  npmGlobalPackages:
    gatsby-cli: 2.4.6

[stefanprobst]

Just to clarify: broken sort means you get an Error, or it just doesn't sort (neither ASC nor DESC)?

[zslabs]

Just to clarify: broken sort means you get an Error, or it just doesn't sort (neither ASC nor DESC)?

Sorry, my comment was pretty vague; no errors; just the query was not actually returning sorted results correctly (not respecting the params of the query). This was happening with both gatsby develop and gatsby build.

[stefanprobst]

Hmm, at the moment, I'm out of ideas unfortunately -- this is working correctly for me with [email protected]. You could try cloning my example repo and check in GraphiQL if sorting works.

[zslabs]

@stefanprobst I think I've found the issue. This isn't related to the core gatsby package, but 2.2.1 of gatsby-transformer-remark; and specifically https://github.com/gatsbyjs/gatsby/pull/10924

Some of the dates are written as 2019-01-01 and others are strings, '2019-01-01' within the frontmatter; and when updating past 2.2.0 the issue appears.

[DSchau]

@zslabs seems like _we_ can confirm that, and I suspect your suspicion about strings is correct.

See #11654 for _why_ I think this. Working on fixing the underlying issue now!

[zslabs]

@DSchau Thank you very much for looking into this and appreciate that link as well!

[DSchau]

Here's a reproduction:

Repo | Live site

The posts should be sorted in descending order (newest first) but clearly that's not the case.

Upgrading to the latest gatsby-transformer-remark _does_ trigger this issue, but I'm unsure if this is something we should fix in core or whether we should just revert the removal of the date parsing/normalization. Looking into it!

[stefanprobst]

@DSchau Thanks for the reproduction!
Observations:

• in type inference, we allow fields with a mix of Date objects and Date strings as a special case.
• sorting works correctly on a second gatsby develop run, when Dates have been serialized to strings when read from .cache.
• this should not be solved just for the markdown transformer, since we could have a mixed field somewhere else as well
• What could work is calling toISOString on Date objects before sorting in run-sift.
  EDIT: I mean here something like:

.map(field => v => {
  const value = _.get(v, field)
  return value instanceof Date ? value.toISOString() : value
})

EDIT: Yes that works

[stefanprobst]

But the more correct solution is that we should manually call resolvers for querying not only for filter fields, but also for sort, group and distinct fields. This is actually the third point in the initial issue description. For Dates, they would get serialized because of this.
EDIT: Something similar to this should work.

[stefanprobst]

Ok so I included both in #11656 because calling resolvers is the more general solution, but we might still need to stringify if at some point our GraphQLDate changes to Dates instead of strings.

[rogermparent]

Hello! I'm running into a similar issue with sorting on fields added by setFieldsOnGraphQLNodeType.

I'm not sure if my issue is the same as everyone else's here, but I've noticed that only fields of type GraphQLString are sortable when added by setFieldsOnGraphQLNodeType.

I tried using the GraphQLDate from "gatsby/dist/schema/types/type-date", and while it completely works for querying, the field is not included in the sort enum. The same field shows up in the sort enum when the type is changed to GraphQLString. This is all on a custom type created in a plugin.

I hope this info helps!

[stefanprobst]

@zslabs I guess you have already seen, the Date sorting issue should be fixed in [email protected]

@rogermparent Thanks for reporting! Your issue seems to be the third point in the initial issue description (sorting should work for scalar types like String, but not when we have to call field resolvers). On second thought, sorting Date fields added with setFields... might actually work. Do you have a repo to share? I could take a look

[zslabs]

@stefanprobst Thanks! Hadn't had a chance to confirm, but things do appear to be working correctly now - thanks!

[rogermparent]

@stefanprobst I just threw together a minimal repo showcasing the problem, and in the process made a few discoveries.

• A regularly added and inferred Date field on a custom node can be sorted on totally fine.
• A Date string added by setFieldsOnGraphqlNodeType will show up in the enum and can be used, but no sorting will actually be done and the query appears unsorted.
• An actual Date-type field added by setFields will not be added to the enum, and attempting to sort on it will break at build time. (which is why it's commented out in the example)

Also, all my fields have resolvers that simply point to the real value (I'm doing it in my actual project as a janky, poor-man's nullable field for when no node has the value.)

Find the repo here: https://github.com/rogermparent/gatsby-setfields-sort-example

[marcysutton]

I just hit this fields error following the example in gatsby-plugin-feed. I read through this thread and updated as many dependencies as I could, and it's still occurring:

  Error: Cannot query field "fields" on type "MarkdownRemark".
  GraphQL request (16:15)
  15:               }
  16:               fields {
                    ^
  17:                 slug

  - internals.js:22 
    [marcysutton.com-gatsby]/[gatsby-plugin-feed]/internals.js:22:13

I don't see a fields entry when I navigate allMarkdownRemark in the graphiql explorer for my site:

Here are the relevant software versions.

{
"gatsby": "^2.1.4",
"gatsby-plugin-feed": "^2.0.13",
"gatsby-transformer-remark": "^2.2.5",
}

Not sure if this is related or not, but it seemed like it!

[stefanprobst]

@rogermparent Thanks for the repo!

It looks like there are actually two issues:

• We currently have different code paths for creating an InputObjectType for fields whose types we infer, and for fields added with the setFieldsOnGraphQLNodeType API. While the former treats Date fields like Strings, the latter does not. #11816 should fix this.
• Unfortunately even with that fix sorting on fields added by the setFields... API still won't work, because we need to call the field resolvers before sorting. This is because the nodes in the store don't have fields added in setFields.... We currently do this for all fields in a filter query, but not for sort fields. This is going to be fixed soonish, but in the meantime you can use this workaround (plus the above mentioned fix):

{
  allSortableExampleNode(
    filter: {
      dateFieldDate: { ne: null }
      stringFieldDate: { ne: null }
    }
    sort: {
      fields: [dateFieldDate]
      order: [DESC]
    }
  ) {
    edges {
      node {
        date
      }
    }
  }
}

This works because now the field resolvers for dateFieldDate and stringFieldDate will be called before querying.

[stefanprobst]

@marcysutton Did you add the slug field in onCreateNode, like in the feed example project?

[marcysutton]

@stefanprobst I missed that, it isn't in the plugin README! I made a note tonight to write a Gatsby RSS feed how-to since I haven't seen one...I'll be sure to include that detail. Thanks :)

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜