Gatsby: [gatsby-transformer-sqip] - current state & todo

Created on 2 May 2018 · 10Comments · Source: gatsbyjs/gatsby

This issue gathers all information about the current state of the SQIP implementation in Gatsby. It is a follow up to #4205.

First of all: It got merged, a very first version is published and can already be used in your website. 🎉

Wanna see it in action? Click here for the demo

What is SQIP?

It is a svg based implementation of low quality image previews (LQIP)

What? I understand nothing

An algorithm calculates a primitive representation of your images based on simple shapes like circles, ellipses, triangles and more. These will be embedded in your initial HTML payload.

This will help your users to get a feeling of how the pictures will look like, even __before__ they got loaded by their (probably) slow connection.

39412713-35581ae2-4c21-11e8-8fd3-5a6392ecac02

Current State

It is feature complete
It is published as a v0.x.x and can be used by early adopters
Everything seems to work fine already, but performance and other things are not optimal yet. Decide on your own if it is ready for production.
We have an example which demonstrates how to integrate it into your Gatsby website.

Open todos & problems

You are very welcome to support me here, just do a PR, mention this issue. Feel free to contact me for questions via Twitter, the Gatsby Discord or just leave a comment here.

👨‍💻 👩‍💻 Code (@todo, help is very welcome)

[ ] something still goes wrong with the sharp focus. The actual image and the calculated SQIP do sometimes not share the same center
[ ] It currently implements its own queue. We should change this to use the queue from gatsby-transformer-sqip for proper progress indication and to avoid performance bottlenecks.

📜 Documentation (Adressed in #5287)

[ ] Readme & Example page need some more love, some important documentation is missing (Use using-sqip example as a reference for now)
[ ] It seriously needs proper caching between builds when using services like Netlify. We need to make this very clear to the users
[ ] We should give some recommended settings. These depend a lot on your use case, current website payload size and more. I did some research about this at https://axe312ger.github.io/embedded-svg-loading-impact-research/

👋 ,
Benedikt

stale?

Source

axe312ger

🎉6 ❤1

Most helpful comment

You are right @martynhoyer, builds on Netlify are limited to a 30 minute duration. Typically that is waaaay bigger than people need. We make exceptions for those on Enterprise plans where we provide dedicated build infrastructure. Even there, exceeding 30 mins is not very common.

As @KyleAMathews mentions, there is no _documented_ way for people to access a build cache to stash things between builds.

However, it is possible. \o/

It's not currently documented because managing a cache between builds can get complex and the levels of support requests we see to help people manage their own internal build caching logic can quickly spiral.

However, if you are happy to shoulder the responsibility of managing your own intra-build cache on Netlify, this will likely help the scenario you are describing here.

I put together a basic proof of concept to illustrate this:

https://github.com/philhawksworth/content-shards

This example lets you run your static site generator to target one section of a site or another, and then use a previously built set of assets for the rest of the site.

The area of particular interest will be int he gulp file where the build cache is accessed:

https://github.com/philhawksworth/content-shards/blob/master/gulpfile.js#L62-L69

and also where the cache location is defined:

https://github.com/philhawksworth/content-shards/blob/master/gulpfile.js#L62-L69

Since every build is containerized, you can't damage the environment for anyone else, or for your other deploys... but by messing with that /opt/build/cache/ directory you might get yourself in a pickle in the build unless you are careful.

I hope this helps!

philhawksworth on 8 May 2018

❤3

All 10 comments

On Netlify — I found out recently they don't let you set directories to cache so actually none of Gatsby's caches work atm 🤷‍♂️

KyleAMathews on 3 May 2018

Would you like to write a blog post introducing the technique/plugin as well?

KyleAMathews on 3 May 2018

👍2

Yeah, thats a pity. Do you have contacts to Netlify to talk about this? Would speed up Gatsby deployments a lot 🚀

About the blog post: Already working on it, my research is done and as the PR is now merged in Gatsby, I have more or less everything on the code side ready. Just writing the actual blog post is missing 💪

axe312ger on 3 May 2018

👍1

You could try @philhawksworth at Netlify (DevRel)

I've just run into what looks like Netlify issues myself while deploying the results of my testing - the site seems to hit the build timeout limit (around 30 minutes?). It sounds like you're aware of this issue @axe312ger and have managed to get around it somehow - any tips?

The build log doesn't report much:

10:50:23 PM: success extract queries from components — 0.327 s
11:18:54 PM: Build exceeded maximum allowed runtime

EDIT:

Local build with a clean directory doesn't take very long (there's only 3 images to process on this test branch), so I'm not sure what Netlify is tripping up on. This is the local build output:

success delete html and css files from previous builds — 0.009 s
success open and validate gatsby-config — 0.005 s
success copy gatsby files — 0.014 s
success onPreBootstrap — 0.785 s
success source and transform nodes — 0.175 s
success building schema — 0.176 s
success createLayouts — 0.009 s
success createPages — 0.043 s
success createPagesStatefully — 0.011 s
success onPreExtractQueries — 0.002 s
success update schema — 0.110 s
success extract queries from components — 0.149 s
Generating image thumbnails [==============================] 3/3 8.6 secs 100%
success run graphql queries — 15.623 s
success write out page data — 0.007 s
success write out redirect data — 0.001 s
Generating image thumbnails [==============================] 42/42 16.1 secs 100%

info bootstrap finished - 19.88 s

success onPostBootstrap — 0.003 s
success Building CSS — 6.964 s
success Building production JavaScript bundles — 37.007 s
success Building static HTML for pages — 26.221 s
Total precache size is about 776 kB for 5 resources.
info Done building in 90.111 sec

martynhoyer on 8 May 2018

As @KyleAMathews mentions, there is no _documented_ way for people to access a build cache to stash things between builds.

However, it is possible. \o/

However, if you are happy to shoulder the responsibility of managing your own intra-build cache on Netlify, this will likely help the scenario you are describing here.

I put together a basic proof of concept to illustrate this:

https://github.com/philhawksworth/content-shards

This example lets you run your static site generator to target one section of a site or another, and then use a previously built set of assets for the rest of the site.

The area of particular interest will be int he gulp file where the build cache is accessed:

https://github.com/philhawksworth/content-shards/blob/master/gulpfile.js#L62-L69

and also where the cache location is defined:

https://github.com/philhawksworth/content-shards/blob/master/gulpfile.js#L62-L69

I hope this helps!

philhawksworth on 8 May 2018

❤3

there are actually gatsby-plugin-sharp and axios as dependency missing

edit: adressed in #5469

axe312ger on 17 May 2018

@philhawksworth - Thanks for your extended reply and that demo repository. Will put this somehow in the documentation at #5287 😄

@martynhoyer - My fellow coworker @stefanjudis combined SQIP with the Netlify build cache in his recent article: https://dev.to/stefanjudis/faster-static-site-builds-part-1--process-only-what-you-need-2j70 🚀

axe312ger on 18 May 2018

Looks like Safari has issues to display the svgs as img tag dataURI. Just getting black boxes. Anybody can confirm?

Edit: okay, that was just the black background and the images had opacity: 0. But the fade animation is failing in safari, resulting in these black boxes instead of a nice fade. Will investigate.

axe312ger on 21 May 2018

Old issues will be closed after 30 days of inactivity. This issue has been quiet for 20 days and is being marked as stale. Reply here or add the label "not stale" to keep this issue open!