Pokeapi: Availability of named resources (for new hosting on Netlify)

[tdmalone]

This issue is to discuss supporting named resources (eg. /pokemon/bulbasaur) when we move to Netlify.

See also #373 for work on fixing named resources on our current hosting.

[sargunv]

Context: the final blocker is the build timeout on Netlify due to too many files (after adding named resources). Potential solutions include:

• Shard the netlify sites by endpoint. This requires a modification to the build script to whitelist or blacklist endpoints to build
• Generate all approx 5000 rewrite rules needed to support named resources at build time

[tdmalone]

Thanks @sargunv!

And I've e-mailed Netlify to ask if it's possible to either increasing the build time, or whether there's a particular redirect limit (I couldn't find either of these answers in their documentation).

[Naramsim]

https://pokeapi.co/api/v2/item/old-rod/

[Naramsim]

maybe I shouldn't have closed the issue...

[sargunv]

You got the wrong one :) It's https://github.com/PokeAPI/pokeapi/issues/373

[sargunv]

https://pokeapi.netlify.com/api/v2/pokemon/magikarp/encounters/

Looks like all we needed to do is disable post processing (we weren't using that anyway). So, it works!

[tdmalone]

Ohh niiice, awesome find @sargunv!!

[sargunv]

Unfortunately we have one more blocker for moving to Netlify: https://github.com/PokeAPI/api-data/issues/11

[sargunv]

I spoke too soon. Not sure why that build succeeded, but the next one failed again.

[tdmalone]

Netlify have responded to my e-mail to say the only way to increase build time is to upgrade to Enterprise, which provides a 'custom build network' with a longer timeout. Otherwise, each build command has up to 15 mins, and the entire build has up to 30 mins.

I assume Enterprise is not something they would upgrade our current Open Source plan to (which they're currently providing for free), but it can't hurt to ask, so I have gone ahead and done that.

They also didn't respond to my redirect-limit question, so I've re-asked that. 😄

[Naramsim]

A workaround could be to build on Travis and then upload from Travis to Netlify all the generated things

[sargunv]

That won't help; our build script is only taking a minute to run. It's Netlify's upload step that's taking >30min.

[sargunv]

Now we've got a build that never terminated nor timed out: https://app.netlify.com/sites/pokeapi/deploys/5ba6e595b312746d754008d9

It's been "Uploading" for nearly 20 hours.

[tdmalone]

Netlify says:

I've seen hundreds [of redirects] work without issue. Not sure about thousands. But it's a matter of performance and not a hard limit. Also, do note that once a file is uploaded, our buildbot won't try to upload it again, so in theory, if you continue to attempt your deploy, eventually, all your files (as long as they don't change between deploys) will get uploaded and a deploy will succeed.

So, I guess we could try the redirect option and see how we go.

Regarding the retries, if we _regenerate_ the deploy they'd most likely be treated as new files, but if we simply reattempt the same deploy again....

I'm triggering a new deployment now and then will try retrying it once it fails.

[tdmalone]

Ok, well that deploy seemed to succeed...

(albeit with a trace generated by babel-js when deploying the Lambda functions, but it looks like it wasn't fatal)

I am triggering again, just to make sure it wasn't a once-off. I'm also clearing the build cache between deploys.

[cmmartti]

Hmm, too bad Netlify doesn't cache build files across builds if they haven't changed. Well, it seems like they do, but only on the enterprise plan, according to that Stack Overflow answer.

[sargunv]

I guess next time we change our data it might require multiple uploads again. Atm the firebase experiment seems to be fitting all our needs and Netlify is also mostly there except for finicky but workable builds and https://github.com/PokeAPI/api-data/issues/11.

Does someone also want to put together an S3 solution in another branch or are we good deciding between Firebase and Netlify?

[cmmartti]

The data doesn't change very often, so it's not that big of a deal if we need to do multiple uploads each time, even if it requires manual intervention. But still, a totally hands-free solution would be better. Otherwise I have no preference.

What would the S3 solution look like? Would it let us keep accurate usage stats?

[sargunv]

Not sure about S3; I know next to nothing about it, even though I guess I work in that exact field. Someone else would have to take the lead on that if we want to try it.

Our Netlify solution will count resource list requests since they go through a function but not other requests. Our Firebase solution does count all requests since they all go through a Firebase function. However, neither solution is likely to provide correct data if we cache responses (which we should do).

[tdmalone]

I'm happy to work on the S3 solution - I know more about it than the other solutions - but I don't think there's a need if we're happy with either Netlify or Firebase.

S3 will give us stats - we can write access logs and then do whatever we like with them from there. But yeah, caching is going to mess with that to a point. If we cache with Cloudflare, I think they might send a HEAD request to see if the content has changed? In which case we could experiment to see whether S3 logs that. Otherwise, if we use Cloudfront instead of Cloudflare, I think we can get more detailed logs. But it's not free like Cloudflare.

I know close to nothing about Firebase (or Google Cloud in general), but is there a logging solution that we can plug into there perhaps?

[neverendingqs]

Sorry late to the game. S3 also suffers from the paginated resources issue unless we fall back to AWS functions, but I'm not sure how that would look. Can we use Edge Lambda here @tdmalone, or would we have to use API Gateway?

Not knowledgeable about GCP or Firebase to comment either, but if you get functions integration for "free", maybe it's better to side that way.

[tdmalone]

We can use Lambda@Edge, but we'd need to use Cloudfront, which is basically/kinda Cloudflare, but not for free. Lambda@Edge is also slightly more expensive than normal Lambda.

We can look into it, but probably only worth it if both Firebase & Netlify don't suit us.

[sargunv]

Any downsides with the Firebase solution atm? I haven't noticed any issues yet so if it works for everyone we can just turn on caching and move forward with that.

[neverendingqs]

However, neither solution is likely to provide correct data if we cache responses (which we should do).

Is this related to out-of-date data? Or is the cache returning a wrongly-sized paginated response?

[cmmartti]

@neverendingqs I think he means in regards to usage stats.

[cmmartti]

Looks like this was solved by moving to Firebase.