Sapper: premature proc.kill() on `sapper export` child process - some `.json` files not getting built as a result of race condition

Created on 14 Sep 2019 · 20Comments · Source: sveltejs/sapper

Describe the bug

as above, i'm seeing dropped messages in sapper export - some .json files not getting built as a result. I don't see a pattern yet among what gets dropped, but it is somewhat consistent (i.e. i run it several times and the same .json files are missing) EDIT: I found the cause, see below, but still dont know what the solution should be

Logs

I placed tracker logs on outbound messages in the server middleware and I can see the correct outbound message: swyx sendmessage: /data/writing___ssg___rise-of-immer.json

I then also place tracker logs on received messages on the sapper export cli. I expect to see a corresponding inbound message swyx trackmessage: /data/writing___ssg___rise-of-immer.json. However, i don't.

swyx trackmessage: undefined
   2.87 kB   index.html
   2.71 kB   about/index.html
swyx sendmessage: /data/writing___ssg___writing_index.json
swyx sendmessage: /data/talks___ssg___talks_index.json
swyx sendmessage: /data/writing___ssg___svelte-static.json
swyx trackmessage: /data/writing___ssg___writing_index.json
   4.44 kB   data/writing___ssg___writing_index.json
swyx trackmessage: /data/talks___ssg___talks_index.json
   3.56 kB   data/talks___ssg___talks_index.json
swyx trackmessage: /data/writing___ssg___svelte-static.json
   9.91 kB   data/writing___ssg___svelte-static.json
   13.7 kB   writing/index.html
   11.7 kB   talks/index.html
   25.2 kB   writing/svelte-static/index.html
swyx sendmessage: /data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
swyx sendmessage: /data/writing___ssg___netlify-nextjs-ssg.json
swyx sendmessage: /data/writing___ssg___graphql-variables-why.json
swyx sendmessage: /data/writing___ssg___keyboard-first.json
swyx sendmessage: /data/writing___ssg___compilers-copout.json
swyx sendmessage: /data/writing___ssg___netlify-worlds-greatest-demo-v0.json
swyx sendmessage: /data/writing___ssg___gatsby-remark-essential-plugins.json
swyx sendmessage: /data/writing___ssg___netlify-jamstack-indiehackers.json
swyx sendmessage: /data/writing___ssg___book-poll-2019.json
swyx sendmessage: /data/writing___ssg___netlify-year-one-360-review.json
swyx sendmessage: /data/writing___ssg___react-native-web-singularity.json
swyx sendmessage: /data/writing___ssg___typescript-generics.json
swyx sendmessage: /data/writing___ssg___types-already.json
   9.27 kB   writing/keyboard-first/index.html
   5.13 kB   writing/compilers-copout/index.html
   19.3 kB   writing/graphql-variables-why/index.html
   34.3 kB   writing/netlify-worlds-greatest-demo-v0/index.html
   5.13 kB   writing/gatsby-remark-essential-plugins/index.html
swyx trackmessage: /data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
   63.5 kB   data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
swyx sendmessage: /data/writing___ssg___svelte-metaphysics.json
swyx sendmessage: /data/writing___ssg___react-not-bad.json
swyx sendmessage: /data/writing___ssg___netlify-python-ssg.json
swyx sendmessage: /data/writing___ssg___netlify-redirects-i18n.json
swyx sendmessage: /data/writing___ssg___slowmad-qna.json
swyx sendmessage: /data/writing___ssg___10000-loose-fans.json
swyx sendmessage: /data/writing___ssg___naked-promises.json
   11.5 kB   writing/book-poll-2019/index.html
   9.98 kB   writing/netlify-year-one-360-review/index.html
   12.4 kB   writing/typescript-generics/index.html
swyx sendmessage: /data/writing___ssg___netlify-env-vars.json
swyx trackmessage: /data/writing___ssg___netlify-nextjs-ssg.json
   23.8 kB   data/writing___ssg___netlify-nextjs-ssg.json
   59.1 kB   writing/netlify-nextjs-ssg/index.html
   42.8 kB   writing/react-native-web-singularity/index.html
swyx trackmessage: /data/writing___ssg___graphql-variables-why.json
   7.11 kB   data/writing___ssg___graphql-variables-why.json
swyx trackmessage: /data/writing___ssg___keyboard-first.json
   2.59 kB   data/writing___ssg___keyboard-first.json
swyx trackmessage: /data/writing___ssg___compilers-copout.json
     844 B   data/writing___ssg___compilers-copout.json
   59.6 kB   writing/netlify-jamstack-indiehackers/index.html
swyx sendmessage: /data/writing___ssg___netlify-year-one.json
swyx sendmessage: /data/writing___ssg___index.json
swyx sendmessage: /data/writing___ssg___types-or-tests.json
swyx sendmessage: /data/writing___ssg___betwixt-reason-and-results.json
swyx sendmessage: /data/writing___ssg___netlify-dev-cors.json
swyx sendmessage: /data/writing___ssg___how-to-name-things.json
swyx sendmessage: /data/writing___ssg___learning-gears.json
    4.6 kB   writing/react-not-bad/index.html
   5.24 kB   writing/netlify-env-vars/index.html
swyx trackmessage: /data/writing___ssg___netlify-worlds-greatest-demo-v0.json
   12.4 kB   data/writing___ssg___netlify-worlds-greatest-demo-v0.json
swyx trackmessage: /data/writing___ssg___gatsby-remark-essential-plugins.json
     919 B   data/writing___ssg___gatsby-remark-essential-plugins.json
swyx sendmessage: /data/writing___ssg___getting-closure-on-hooks.json
   17.5 kB   writing/svelte-metaphysics/index.html
   26.4 kB   writing/10000-loose-fans/index.html
   44.3 kB   writing/slowmad-qna/index.html
    105 kB   writing/types-already/index.html
   51.2 kB   writing/netlify-python-ssg/index.html
   56.9 kB   writing/netlify-redirects-i18n/index.html
swyx sendmessage: /data/writing___ssg___post-graduation-mindmap.json
   4.87 kB   writing/types-or-tests/index.html
swyx sendmessage: /data/writing___ssg___star-apps.json
swyx sendmessage: /data/writing___ssg___cuttlefish.json
swyx sendmessage: /data/writing___ssg___gatsby-static-dynamic.json
swyx sendmessage: /data/writing___ssg___rise-of-immer.json
swyx sendmessage: /data/writing___ssg___h1b1.json
   6.36 kB   writing/index/index.html
swyx trackmessage: /data/writing___ssg___netlify-jamstack-indiehackers.json
   26.1 kB   data/writing___ssg___netlify-jamstack-indiehackers.json
swyx trackmessage: /data/writing___ssg___book-poll-2019.json
   3.46 kB   data/writing___ssg___book-poll-2019.json
swyx sendmessage: /data/writing___ssg___specialize-in-the-new.json
   23.7 kB   writing/learning-gears/index.html
   28.4 kB   writing/betwixt-reason-and-results/index.html
swyx trackmessage: /data/writing___ssg___netlify-year-one-360-review.json
   2.96 kB   data/writing___ssg___netlify-year-one-360-review.json
swyx sendmessage: /data/writing___ssg___clone-open-source-apps.json
    155 kB   writing/netlify-gatsby-jamstack-apps-authenticated-serverless/index.html
   51.4 kB   writing/netlify-dev-cors/index.html
swyx trackmessage: /data/writing___ssg___react-native-web-singularity.json
   17.7 kB   data/writing___ssg___react-native-web-singularity.json
swyx sendmessage: /data/writing___ssg___learn-in-public.json
swyx sendmessage: /data/writing___ssg___know-your-tools.json
swyx sendmessage: /data/talks___ssg___intro-to-typescript.json
swyx sendmessage: /data/talks___ssg___ducktapes-podcast.json
swyx sendmessage: /data/talks___ssg___reactroundup-podcast.json
   14.3 kB   writing/h1b1/index.html
   5.17 kB   writing/star-apps/index.html
swyx trackmessage: /data/writing___ssg___typescript-generics.json
   3.65 kB   data/writing___ssg___typescript-generics.json
swyx sendmessage: /data/talks___ssg___programming-typescript.json
    145 kB   writing/naked-promises/index.html
   69.4 kB   writing/netlify-year-one/index.html
   69.5 kB   writing/how-to-name-things/index.html
   25.6 kB   writing/post-graduation-mindmap/index.html
   15.6 kB   writing/cuttlefish/index.html
   13.3 kB   writing/specialize-in-the-new/index.html
swyx sendmessage: /data/talks___ssg___kentcdodds-podcast.json
   12.7 kB   writing/clone-open-source-apps/index.html
swyx sendmessage: /data/talks___ssg___learn-in-public.json
   58.3 kB   writing/rise-of-immer/index.html
swyx sendmessage: /data/talks___ssg___netlify-functions-lunchnlearn.json
swyx sendmessage: /data/talks___ssg___jamstack-victory.json
swyx sendmessage: /data/talks___ssg___learn-in-public-singapore.json
swyx sendmessage: /data/talks___ssg___react-hooks.json
   11.2 kB   writing/know-your-tools/index.html
   6.61 kB   talks/ducktapes-podcast/index.html
   7.25 kB   talks/reactroundup-podcast/index.html
swyx sendmessage: /data/talks___ssg___jamstack-jumpstart.json
   17.9 kB   writing/learn-in-public/index.html
   6.32 kB   talks/programming-typescript/index.html
swyx trackmessage: /data/writing___ssg___types-already.json
   43.8 kB   data/writing___ssg___types-already.json
swyx sendmessage: /data/talks___ssg___intent-based-cli.json
swyx sendmessage: /data/talks___ssg___serverless-graphql-netlify.json
    6.6 kB   talks/kentcdodds-podcast/index.html
swyx trackmessage: /data/writing___ssg___svelte-metaphysics.json
   6.47 kB   data/writing___ssg___svelte-metaphysics.json
swyx trackmessage: /data/writing___ssg___react-not-bad.json
     517 B   data/writing___ssg___react-not-bad.json
swyx sendmessage: /data/talks___ssg___freecodecamp-podcast.json
swyx sendmessage: /data/talks___ssg___serverless-graphql.json
swyx sendmessage: /data/talks___ssg___react-hooks-lightning.json
swyx sendmessage: /data/talks___ssg___jamstack-victory-jsla.json
   26.4 kB   talks/learn-in-public/index.html
   8.41 kB   talks/netlify-functions-lunchnlearn/index.html
   5.79 kB   talks/jamstack-victory/index.html
   6.09 kB   talks/learn-in-public-singapore/index.html
swyx sendmessage: /data/talks___ssg___react-podcast.json
    188 kB   writing/getting-closure-on-hooks/index.html
swyx trackmessage: /data/writing___ssg___netlify-python-ssg.json
   21.3 kB   data/writing___ssg___netlify-python-ssg.json
swyx sendmessage: /data/talks___ssg___babel-macros.json
   25.8 kB   talks/jamstack-jumpstart/index.html
   5.69 kB   talks/serverless-graphql-netlify/index.html
swyx sendmessage: /data/talks___ssg___immutability-is-changing.json
swyx sendmessage: /data/talks___ssg___react-typescript.json
   30.2 kB   talks/intent-based-cli/index.html
swyx sendmessage: /data/talks___ssg___reacts-new-defaults.json
swyx sendmessage: /data/talks___ssg___contributing-to-open-source.json
swyx sendmessage: /data/talks___ssg___babel-graphql.json
swyx sendmessage: /data/talks___ssg___react-not-reactive.json
   6.32 kB   talks/react-hooks-lightning/index.html
   6.04 kB   talks/jamstack-victory-jsla/index.html
swyx sendmessage: /data/talks___ssg___infinite-building.json
    201 kB   writing/gatsby-static-dynamic/index.html
   19.7 kB   talks/freecodecamp-podcast/index.html
   16.3 kB   talks/serverless-graphql/index.html
   5.06 kB   talks/react-podcast/index.html
swyx trackmessage: /data/writing___ssg___netlify-redirects-i18n.json
   23.3 kB   data/writing___ssg___netlify-redirects-i18n.json
swyx sendmessage: /data/talks___ssg___creating-cra.json
    155 kB   talks/intro-to-typescript/index.html
swyx sendmessage: /data/talks___ssg___react-suspense.json
swyx sendmessage: /data/talks___ssg___never-bundle-react-again.json
   25.3 kB   talks/babel-macros/index.html
   6.56 kB   talks/react-typescript/index.html
swyx trackmessage: /data/writing___ssg___slowmad-qna.json
   19.7 kB   data/writing___ssg___slowmad-qna.json
   8.61 kB   talks/babel-graphql/index.html
    7.9 kB   talks/reacts-new-defaults/index.html
   14.2 kB   talks/contributing-to-open-source/index.html
swyx trackmessage: /data/writing___ssg___10000-loose-fans.json
   10.8 kB   data/writing___ssg___10000-loose-fans.json
swyx sendmessage: /data/talks___ssg___fullstack-heaps.json
swyx sendmessage: /data/talks___ssg___fullstack-crossbones.json
swyx sendmessage: /data/talks___ssg___fullstack-react-trip-planner.json
swyx sendmessage: /data/talks___ssg___contributing-to-react.json
   5.24 kB   talks/infinite-building/index.html
   8.13 kB   talks/creating-cra/index.html
   55.9 kB   talks/react-not-reactive/index.html
   5.09 kB   talks/react-suspense/index.html
   5.28 kB   talks/never-bundle-react-again/index.html
    101 kB   talks/immutability-is-changing/index.html
    215 kB   talks/react-hooks/index.html
   5.29 kB   talks/fullstack-react-trip-planner/index.html
   4.45 kB   talks/fullstack-heaps/index.html
   5.12 kB   talks/contributing-to-react/index.html
    5.3 kB   talks/fullstack-crossbones/index.html
swyx trackmessage: /data/writing___ssg___naked-promises.json
   58.9 kB   data/writing___ssg___naked-promises.json
swyx trackmessage: /data/writing___ssg___netlify-env-vars.json
     848 B   data/writing___ssg___netlify-env-vars.json
swyx trackmessage: /data/writing___ssg___netlify-year-one.json
     31 kB   data/writing___ssg___netlify-year-one.json
swyx trackmessage: /data/writing___ssg___index.json
   1.46 kB   data/writing___ssg___index.json
swyx trackmessage: /data/writing___ssg___types-or-tests.json
     721 B   data/writing___ssg___types-or-tests.json
swyx trackmessage: /data/writing___ssg___betwixt-reason-and-results.json
   11.9 kB   data/writing___ssg___betwixt-reason-and-results.json
   2.63 kB   service-worker-index.html

As you can see the message is always sent, but not always received.

To Reproduce

repro is going to be difficult for this one since its almost certainly a scale problem. my work is here: https://github.com/sw-yx/swyxdotio/tree/moveToSapper and you will note the logs above what the result of the code execution is.

Expected behavior

as explained above in the logs - messages that are sent ought to be received.

Information about your Sapper Installation:

Your browser and the version: n/a
Your operating system: osx
Your hosting environment: n/a
Sapper version: 0.27.8
Svelte version: n/a
If it is an exported (npm run export) or dynamic application: exported
Whether your application uses Webpack or Rollup: rollup

Severity

very severe. can't trust sapper export if it doesnt reliably work.

Source

sw-yx

👍2

Most helpful comment

Fixed in 0.27.14, with the long-awaited merge of 0.27.14.

Conduitry on 9 Jun 2020

❤2 🎉2

All 20 comments

possibly related? https://github.com/sveltejs/sapper/pull/869 IO backpressure from @khrome83

a reader also pointed out that all posts from "Netlify Year One" on down on https://www.swyx.io/writing/ are the ones missing the .json. a huge clue! though I have no idea what to do with this information.

sw-yx on 14 Sep 2019

got it. overeager proc.kill: https://github.com/sveltejs/sapper/blob/910d28e3419409b5498933e8c13d37cc03b652a0/src/api/export.ts#L238

commenting out this one line fixes it.

how can we avoid killing that child process while it still has work left to do?

EDIT: proof: i injected a line where yootils.queue.close is called, which is supposed to block execution so that we dont proceed to the deadly proc.kill.. but it doesn't do its job. look at all the json files being written thereafter

> Crawling http://localhost:3000/
   2.87 kB   index.html
   4.42 kB   data/writing___ssg___writing_index.json
   2.71 kB   about/index.html
   9.91 kB   data/writing___ssg___svelte-static.json
   3.56 kB   data/talks___ssg___talks_index.json
   13.7 kB   writing/index.html
   25.2 kB   writing/svelte-static/index.html
   11.7 kB   talks/index.html
   23.8 kB   data/writing___ssg___netlify-nextjs-ssg.json
   9.27 kB   writing/keyboard-first/index.html
   5.13 kB   writing/compilers-copout/index.html
   5.13 kB   writing/gatsby-remark-essential-plugins/index.html
   7.11 kB   data/writing___ssg___graphql-variables-why.json
   19.3 kB   writing/graphql-variables-why/index.html
   34.2 kB   writing/netlify-worlds-greatest-demo-v0/index.html
   59.1 kB   writing/netlify-nextjs-ssg/index.html
   11.5 kB   writing/book-poll-2019/index.html
   9.98 kB   writing/netlify-year-one-360-review/index.html
   12.4 kB   writing/typescript-generics/index.html
    4.6 kB   writing/react-not-bad/index.html
   17.4 kB   writing/svelte-metaphysics/index.html
   59.6 kB   writing/netlify-jamstack-indiehackers/index.html
   42.8 kB   writing/react-native-web-singularity/index.html
   5.24 kB   writing/netlify-env-vars/index.html
   63.5 kB   data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
   2.59 kB   data/writing___ssg___keyboard-first.json
   26.4 kB   writing/10000-loose-fans/index.html
   6.36 kB   writing/index/index.html
    155 kB   writing/netlify-gatsby-jamstack-apps-authenticated-serverless/index.html
   51.2 kB   writing/netlify-python-ssg/index.html
   44.3 kB   writing/slowmad-qna/index.html
   12.3 kB   data/writing___ssg___netlify-worlds-greatest-demo-v0.json
     844 B   data/writing___ssg___compilers-copout.json
     919 B   data/writing___ssg___gatsby-remark-essential-plugins.json
    105 kB   writing/types-already/index.html
   56.8 kB   writing/netlify-redirects-i18n/index.html
   4.87 kB   writing/types-or-tests/index.html
   69.4 kB   writing/netlify-year-one/index.html
   28.4 kB   writing/betwixt-reason-and-results/index.html
   23.7 kB   writing/learning-gears/index.html
   25.6 kB   writing/post-graduation-mindmap/index.html
   5.17 kB   writing/star-apps/index.html
   26.1 kB   data/writing___ssg___netlify-jamstack-indiehackers.json
   3.46 kB   data/writing___ssg___book-poll-2019.json
   2.96 kB   data/writing___ssg___netlify-year-one-360-review.json
   51.4 kB   writing/netlify-dev-cors/index.html
    145 kB   writing/naked-promises/index.html
   15.6 kB   writing/cuttlefish/index.html
   14.3 kB   writing/h1b1/index.html
   13.3 kB   writing/specialize-in-the-new/index.html
   12.7 kB   writing/clone-open-source-apps/index.html
   11.2 kB   writing/know-your-tools/index.html
   17.7 kB   data/writing___ssg___react-native-web-singularity.json
   69.5 kB   writing/how-to-name-things/index.html
   3.65 kB   data/writing___ssg___typescript-generics.json
   58.3 kB   writing/rise-of-immer/index.html
   17.9 kB   writing/learn-in-public/index.html
   6.61 kB   talks/ducktapes-podcast/index.html
   7.25 kB   talks/reactroundup-podcast/index.html
   6.32 kB   talks/programming-typescript/index.html
    6.6 kB   talks/kentcdodds-podcast/index.html
   5.79 kB   talks/jamstack-victory/index.html
   26.4 kB   talks/learn-in-public/index.html
   43.8 kB   data/writing___ssg___types-already.json
   8.41 kB   talks/netlify-functions-lunchnlearn/index.html
   6.09 kB   talks/learn-in-public-singapore/index.html
   6.47 kB   data/writing___ssg___svelte-metaphysics.json
     517 B   data/writing___ssg___react-not-bad.json
    188 kB   writing/getting-closure-on-hooks/index.html
   25.8 kB   talks/jamstack-jumpstart/index.html
   5.69 kB   talks/serverless-graphql-netlify/index.html
   30.2 kB   talks/intent-based-cli/index.html
   16.3 kB   talks/serverless-graphql/index.html
   19.7 kB   talks/freecodecamp-podcast/index.html
   21.3 kB   data/writing___ssg___netlify-python-ssg.json
   6.04 kB   talks/jamstack-victory-jsla/index.html
   6.32 kB   talks/react-hooks-lightning/index.html
   5.06 kB   talks/react-podcast/index.html
    201 kB   writing/gatsby-static-dynamic/index.html
   6.56 kB   talks/react-typescript/index.html
    7.9 kB   talks/reacts-new-defaults/index.html
   14.2 kB   talks/contributing-to-open-source/index.html
    155 kB   talks/intro-to-typescript/index.html
   25.3 kB   talks/babel-macros/index.html
   23.3 kB   data/writing___ssg___netlify-redirects-i18n.json
   8.61 kB   talks/babel-graphql/index.html
   5.24 kB   talks/infinite-building/index.html
   5.28 kB   talks/never-bundle-react-again/index.html
   8.13 kB   talks/creating-cra/index.html
   5.09 kB   talks/react-suspense/index.html
   19.7 kB   data/writing___ssg___slowmad-qna.json
   4.45 kB   talks/fullstack-heaps/index.html
   5.12 kB   talks/contributing-to-react/index.html
   10.8 kB   data/writing___ssg___10000-loose-fans.json
    101 kB   talks/immutability-is-changing/index.html
   55.9 kB   talks/react-not-reactive/index.html
    5.3 kB   talks/fullstack-crossbones/index.html
   5.29 kB   talks/fullstack-react-trip-planner/index.html
    215 kB   talks/react-hooks/index.html
   58.9 kB   data/writing___ssg___naked-promises.json
     848 B   data/writing___ssg___netlify-env-vars.json
     31 kB   data/writing___ssg___netlify-year-one.json
   1.46 kB   data/writing___ssg___index.json
     721 B   data/writing___ssg___types-or-tests.json
   11.9 kB   data/writing___ssg___betwixt-reason-and-results.json
   2.63 kB   service-worker-index.html

swyx: close called { pending: 0 }

> Finished in 8.4s. Type npx serve __sapper__/export to run the app.
   20.8 kB   data/writing___ssg___netlify-dev-cors.json
   29.4 kB   data/writing___ssg___how-to-name-things.json
   9.37 kB   data/writing___ssg___learning-gears.json
   76.8 kB   data/writing___ssg___getting-closure-on-hooks.json
   8.68 kB   data/writing___ssg___post-graduation-mindmap.json
     810 B   data/writing___ssg___star-apps.json
   5.64 kB   data/writing___ssg___cuttlefish.json
   81.9 kB   data/writing___ssg___gatsby-static-dynamic.json
   23.8 kB   data/writing___ssg___rise-of-immer.json
    5.1 kB   data/writing___ssg___h1b1.json
   4.84 kB   data/writing___ssg___specialize-in-the-new.json
   4.44 kB   data/writing___ssg___clone-open-source-apps.json
   3.83 kB   data/writing___ssg___know-your-tools.json
   6.79 kB   data/writing___ssg___learn-in-public.json
   62.2 kB   data/talks___ssg___intro-to-typescript.json
   1.03 kB   data/talks___ssg___ducktapes-podcast.json
   1.58 kB   data/talks___ssg___reactroundup-podcast.json
     677 B   data/talks___ssg___programming-typescript.json
    1.2 kB   data/talks___ssg___kentcdodds-podcast.json
   9.15 kB   data/talks___ssg___learn-in-public.json
     865 B   data/talks___ssg___jamstack-victory.json
   1.57 kB   data/talks___ssg___netlify-functions-lunchnlearn.json
     705 B   data/talks___ssg___learn-in-public-singapore.json
   86.6 kB   data/talks___ssg___react-hooks.json
   9.88 kB   data/talks___ssg___jamstack-jumpstart.json
   10.5 kB   data/talks___ssg___intent-based-cli.json
     890 B   data/talks___ssg___serverless-graphql-netlify.json
   6.57 kB   data/talks___ssg___freecodecamp-podcast.json
   4.92 kB   data/talks___ssg___serverless-graphql.json
     985 B   data/talks___ssg___react-hooks-lightning.json
     971 B   data/talks___ssg___jamstack-victory-jsla.json
     575 B   data/talks___ssg___react-podcast.json
   8.42 kB   data/talks___ssg___babel-macros.json
   39.3 kB   data/talks___ssg___immutability-is-changing.json
   1.01 kB   data/talks___ssg___react-typescript.json
   1.61 kB   data/talks___ssg___reacts-new-defaults.json
   4.23 kB   data/talks___ssg___contributing-to-open-source.json
   1.78 kB   data/talks___ssg___babel-graphql.json
   23.3 kB   data/talks___ssg___react-not-reactive.json
     625 B   data/talks___ssg___infinite-building.json
   1.24 kB   data/talks___ssg___creating-cra.json
     451 B   data/talks___ssg___never-bundle-react-again.json
     397 B   data/talks___ssg___react-suspense.json
     475 B   data/talks___ssg___contributing-to-react.json
     339 B   data/talks___ssg___fullstack-heaps.json
     510 B   data/talks___ssg___fullstack-crossbones.json
     461 B   data/talks___ssg___fullstack-react-trip-planner.json

sw-yx on 14 Sep 2019

a reader also pointed out

You’re welcome!

rheinardkorf on 14 Sep 2019

👍1

Ok I am officially stumped. i know -why- this is happening: yootils.queue is made to deal with async queues of work... except:

(in queue) the work is not a blocker because it is a fast fetch request
(in queue) the data saving is triggered from the child process via middleware - there is NO WAY for this process to pause execution and wait for a message back from the parent telling us it is ok to proceed (as far as i can tell)
(NOT in queue) that work is triggered asynchronously via message passing and takes time to complete because it is synchronous I/O
(in queue) we kill the process once the queue is done, even though there is ongoing work to save that file

but i dont know what the solution is here. hoping @Rich-Harris see this and has an idea.

sw-yx on 14 Sep 2019

A definite improvement we can make, that doesn't solve this but helps, is to make the I/O non-blocking. rewriting this writeFileSync code to promisified async helps more data to be output, but we are still killing the child process prematurely.

sw-yx on 14 Sep 2019

@sw-yx I have a disappearing json file on Netlify, and no where near as many file exports as thou...

arxpoetica on 19 Sep 2019

repro?

sw-yx on 20 Sep 2019

@sw-yx I figured out that my bug wasn't related at all to json file generation (that's working fine), but static export crawling when using something other than this.fetch, so my issue is unrelated. See https://github.com/sveltejs/sapper/issues/908

Sorry for the noise on this ticket.

arxpoetica on 20 Sep 2019

i thought that merging https://github.com/sveltejs/sapper/pull/869 would fix this issue, but after checking futher it doesn't. :( still unsolved.

sw-yx on 3 Oct 2019

🎉2

update: i think i have a janky fix... here's what needs to happen:

the proc.on callback calls save as though it were a cheap function, but it is not!
as a result, Sapper calls proc.kill too early and so the files are not getting written out because they are blocked.

To fix, either spin out the file writing to yet another child process (wasteful) or...

    // swyx: solve race condition where saving in child process is prematurely stopped by proc.kill
    // https://github.com/sveltejs/sapper/issues/893#issuecomment-538811020
    let latestUrl = null; 
    proc.on('message', message => {
        if (!message.__sapper__ || message.event !== 'file') return;
        latestUrl = message.url;
        const saveresult = save(message.url, message.status, message.type, message.body);
        if (saveresult) saveresult.then(() => checkWritingDone(message.url));
    });

    // swyx: introduced to solve race condition
    // https://github.com/sveltejs/sapper/issues/893#issuecomment-538811020
    function checkWritingDone(currentUrl) {
        if (latestUrl != currentUrl || queue.getLength() > 0) ; else {
            proc.kill();
        }
    }

and comment out the bottom proc.kill.

i'm implementing this in my sapper fork. let me know if you want this in the main repo.

sw-yx on 7 Oct 2019

👍2

just swimming by to confirm that @nealalpert has fixed this once and for all by including the save() method in the export queue: https://github.com/nealalpert/sapper/commit/3247d504cb4c7cf3f71e57614697e2b559b40138

i really think this belongs in core sapper. others will run into what we ran into.

sw-yx on 8 Oct 2019

👍3

Hitting the same problem. Any chance of getting #869 merged soon?

bummzack on 16 Jan 2020

👍3

@sw-yx I'm assuming that https://github.com/sveltejs/sapper/pull/869 now fixes this issue, based on your previous comment about addSave?

antony on 20 Feb 2020

yes, he updated it after i reported my issues.

sw-yx on 20 Feb 2020

Fixed in 0.27.14, with the long-awaited merge of 0.27.14.

Conduitry on 9 Jun 2020

❤2 🎉2

This issue still seems to be unsolved with 0.27.16.
Creating a fresh sapper project and making the src/blog/index.json.js waiting for data does not export all pages.

Build output with default setup (everything gets exported):
Screenshot 2020-07-13 at 12 06 20

Testing with a delay of 5 seconds on the blog data:
Screenshot 2020-07-13 at 12 06 46

Example code: https://github.com/bummzack/sapper-wtf/commit/b0d9f8bd6ff6f537b91933e83adfd7868fcc8a82

Running sapper export --legacy --no-timeout does not generate any pages at all.

bummzack on 13 Jul 2020

Nevermind, increasing the timeout via --timeout=10000 does the trick. Although I still get some weird problems with some json files not being exported. I'll try to reproduce it in a simple test-case.

bummzack on 13 Jul 2020

Currently in export.ts onDone of the queue is sometimes called before the callback in proc.on('message', cb).

I commented out proc.kill() in the onDone callback to get some logs:

Another log showing 2 files missed:

It seems like how things work now there's no way to know we've waited long enough for all process.send calls to be done. One simple solution is to add a delay of half a second in handleFetch, this slows things down though and is not gaurunteed to work. A better solution would be to have some way to know we're not waiting on process.send, maybe the server could always use process.send to signal that it's done.

DylanVann on 5 Oct 2020

I can confirm that this bug still exists...
i have a huge _posts.js with a lot of data in it, and Sapper will not export all json files!
the hint from @zhammer really solved it for me until the real thing is solved

So everybody who still has this problem should try

"export": "sapper export --legacy --concurrent 1",

ingomc on 9 Nov 2020

❤1

@ingomc i'm glad that comment on my commit came in handy! yeah, my first step for an exported sapper project is now to add --concurrent 1 to the export package.json command