Spacex-api: Add header with total number of matched items

[GerkinDev]

Hi,

I'm writing a small app retrieving data from your API, using skip & limit, and displaying data in a paginated table.

Suppose that I want only the flight_id field for the launches endpoint, with 10 items per page. To know how many pages are available, I must do one of the following:

• Either fetch the whole set with filter=flight_id & limit=999, but it is heavy and I prefer load data lazilly
• Or do 2 requests:
  
  
  
  • one to fetch my data, with skip=0, limit=10 & filter=flight_id
  
  
  • and the other with limit=999 & filter=a_non_existent_field, to get only empty objects: lighter for both you and me.
  
  

I am currently using the 2nd solution. But it would be great:

• either to have a _query string parameter_ to ask only for the result count, to simplify & lighten the 2nd request
• or to add a header or something in a normal API response with the total number of items.

The 2nd solution would be the best, IMO. Moreover, this information could be cached if no select query is present in the request.

What do you think about it ?

[jakewmeyer]

Header is probably the best option here

Just to clarify, you want to know the total number of matched items without a limit applied?

I need to take a deeper look and see if mongo supports this kind of behavior in a single query.

[GerkinDev]

Yes exactly.

If mongo can't support it in a single query, I think you could add a parameter (query string or request header) so that the client request for the 2nd query only if it needs it.

BTW, good job for the reactivity ;)

[jakewmeyer]

Looks like we can do this in a single query. Mongo returns a cursor that has a count() method with an option to count while ignoring offset and limit params passed to the query 👍

Any preference on naming for the header? Thinking X-Count or something along those lines.

[jhpratt]

@jakewmeyer Just running across this — I recently looked into the option of a custom header for Enceladus, and ran across a couple different resources: MDN, Stack Overflow.

Basically, the IETF has formally deprecated the "X-" prefix on custom headers, on the off chance they get standardized in the future. If there's some other way to approach this topic, I highly recommend you do so. For Enceladus, this amounted to adding an additional route.

[GerkinDev]

Wow amazing! No man, seriously you're the boss on that. While it's in the
docs I'm OK. X-Count is fine

Edit : @jhpratt the RFC6684 recommends to use namespace prefixes. You could then use SPACEX-API-COUNT.

[jakewmeyer]

I've added the base logic for the header, but still need to think about how this should work with the current redis caching.

So the header is functional currently as spacex-api-count, but only set when the response isn't from cache.

Using an out of the box caching package currently, but this will probably require a custom implementation to work properly

[jakewmeyer]

@GerkinDev New branch for the custom redis caching is created and underway, might take a week or so to finish, just a heads up.

[jakewmeyer]

@GerkinDev new redis caching is merged and live, should be good to go on this issue 👍

In addition to the new header, I also made slight changes to the caching header shown when the response is served from cache. spacex-api-cache is the new header, with potential values of HIT or MISS

[GerkinDev]

Yay! I gonna try that tomorrow! Thank you very much for this :D

[GerkinDev]

Ok actually this issue isn't solved yet. There is a last (small) change to do: add an Access-Control-Expose-Headers in the response. In your case, it would be the following:

Access-Control-Expose-Headers: spacex-api-cache, spacex-api-count

[jakewmeyer]

Added 👍

[GerkinDev]

Yeeeeah that was quick! Thanks so much! 🍻