Wrangler: How to Handle Builds Where More Than 10k Pages Change?

[nickreese]

🐛 Bug Report

My SSG generates new pages each build. The first wrangler publish always works, but subsequent ones fail due to the 10k key limit.

Number of keys to delete (17397) exceeds max of 10000

Is there a workaround for this besides deleting the store ahead of time? (This works for staging but not production).

- yes | wrangler kv:namespace delete --namespace-id={ID for __static-staging-workers_sites_assets}

This said, I was reading the code for for getAssetFromKV and it appears that a possible workaround would be to:

1 - Create a new KV store for the content.
2 - Specify wrangler to use that store for the content.
3 - After deploy delete the old KV store.

Is there a way to do step 2? I've read all of the docs and can't find anything about how to specify the store for the worker to use.

Environment

• operating system: Docker: node:latest

Steps to reproduce

1 - Build a project with > 10k pages.
2 - Make a change to more than 10k of those pages.

What did you expect to see?

The build be successful.

What did you see instead?

Number of keys to delete (17397) exceeds max of 10000

[nickreese]

I’m unfamiliar with Rust but it appears that

const MAX_PAIRS: usize = 10000; in src/commands/kv/bulk/mod.rs is limiting put and delete calls.

Is there a way that wrangler can handle deletes the same way it handles uploads? (It easily uploads more than 10k files)

This is blocking me from using this in production.

Thanks. Really loving Workers/Worker Sites and stoked to be able to use them and sing their praises to the SSG community.

[ashleymichal]

hey y'all! this limit does appear in both api endpoints; i think it would be easy as pointed out to "batch" deletes the way we batch uploads. PRs welcome, but I'll have a look at this Friday if no one has taken it on.

[ashleymichal]

@nickreese the hack in the meantime would be to rename the existing site namespace in the dashboard; this is as easy as appending/prepending "old" or something to the title of the namespace. for sites, wrangler uses a naming convention for the site namespace to decide whether to create a new namespace or push changes to the old one, so if you rename your old namespace it will stay bound to your worker in production (it only changes the title, not the id which is how we bind namespaces); then when you run wrangler publish again, wrangler will look for the namespace by its old name and when it doesn't find it there, it'll create a new one, upload your assets to that, and bind the new namespace to your worker. once it's done publishing you can delete the old namespace.

[nickreese]

@ashleymichal Thanks for looking into this. I dove in this morning and wrote my own upload process in node.js that matches Wranglers upload process. (Hence my comment over here.)

All that said, I would love to be able to use wrangler instead of my own hacky code to get the job done.

Thanks!

[EverlastingBugstopper]

disregard this comment, it is inaccurate, noticed immediately after posting

~Hey @nickreese - we've released an update (i believe in wrangler 1.8.2) that optimizes this just a bit. I'm not sure if it will 100% solve your issue because i'm not sure what the actual limit is but it _might_ work a bit better now than when you first tried it out.~

[tombennet]

Hey @ashleymichal , did you get a chance to look into batching deletes? I'm currently using the hack you describe (prepending namespace so Wrangler creates a new one) but it would be awesome not to have to worry about this when we go to production. Thanks again for looking into this! 👍 🙏

[kpalmvik]

Any update on this issue @ashleymichal? This is also preventing us from using Workers Sites in production, and it would be great to know if it will be solved in the near future.

[ashleymichal]

there is a lot of work for cleaning up sites, but this needs a priority bump. we'll get this in for 1.10.0

[jspspike]

I'll take a look at this issue

[andrzejwp]

@nickreese the hack in the meantime would be to rename the existing site namespace in the dashboard; this is as easy as appending/prepending "old" or something to the title of the namespace.

Thanks for this workaround. Works like a charm.