Gridsome: Post images optimized by <g-image> on Wordpress Source

[dominiquedutra]

Summary

Have the wordpress post images available on GraphQL layer so they can be passed to g-image component.

During build / develop command:

• Wordpress Post HTML content is parsed;
• Posts are split into fragments being either html or img fragments;
• All images are downloaded from the remote source and stored locally if they haven't been before;
• Images are added to graphql
• You can now pass the images to g-image and have the magic happen (optimization, lazy-loading and progressive image)

Basic example

I've created this fork and successfully implemented a working version of this feature.
Please note that I am new to nodejs, webpack, and javascript in general. Mostly I work with php and vue. Let me know if I should pursue this and submit a PR.

https://github.com/dominiquedutra/gridsome/commits/source-wordpress-get-post-images

Simply clone the branch and start a new clean WordPress starter.
Modify your WordPressPost.vue file:

<template>
  <Layout>
    <h1 v-html="$page.wordPressPost.title"/>
        <template v-if="$page.wordPressPost.postFragments" v-for="fragment in $page.wordPressPost.postFragments">
            <!-- Fragment is a html block -->
            <template v-if="fragment.type == 'html'">
                <div v-html="fragment.fragmentData.html"></div>
            </template>

            <!-- Fragment is a image -->
            <template v-if="fragment.type == 'img' && fragment.fragmentData.image">
                <g-image :src="fragment.fragmentData.image" width="200" />
            </template>
        </template>
  </Layout>
</template>

<page-query>
query Post ($path: String!) {
    wordPressPost (path: $path) {
        title
        content
        excerpt
        link
        postFragments {
            type
            fragmentData {
                image
                html
            }
        }
        categories {
            id
            title
            path
            slug
        }
    }
}
</page-query>

<script>
export default {
  metaInfo () {
    return {
      title: this.$page.wordPressPost.title
    }
  }
}
</script>

<style>
  //Up to you
</style>

Set options for Wordpress Source on gridsome.config.js:

plugins: [
    {
      use: '@gridsome/source-wordpress',
      options: {
        baseUrl: '', // required
        typeName: 'WordPress', // GraphQL schema name (Optional)
        perPage: 100, // How many posts to load from server per request (Optional)
        concurrent: 10, // How many requests to run simultaneously (Optional)
        splitPostsIntoFragments: true, //Split html posts into fragments representing html blocks or images
        downloadRemoteImagesFromPosts: true, //Download remote images
        postImagesLocalPath: '/Users/username/this/must/be/full/path/', //Full path with '/' in the end
        routes: {
          post: '/:slug', //adds route for "post" post type (Optional)
          post_tag: '/tag/:slug' // adds route for "post_tag" post type (Optional)
        }
      }
    }
  ]

Run gridsome build and use http-server (yarn global add http-server) to serve dist folder (cd dist && http-server)

Tested on a production WordPress instance with about 250 posts and 1200 images. It takes some time to download and process the images but not nearly as much as I thought it would.

Motivation

I am sure some people out there want to use Wordpress (and other headless cms) that behave like WordPress regarding post images. Not having post images optimized is bad.
We could use this as boilerplate to modify any source plugins.

Todo (I can easily invest time on this if the idea is on track)

• ~Check why building to Zeit now won't work~;
• ~Extract image alt and description and have those properties available on graphql~;
• Replicate WordPress folder structure locally wp-content/uploads/{year}/{month} - currently, all images are being stored on a single dir;
• ~Download featured images as well~.
• [ISSUE] sometimes, when syncing a lot of images, some images get downloaded partially.

Please let me know what you think.
Let me know if you need a staging WordPress endpoint to test this.

@hjvedvik my fork branch is really dirty. If you think this is an idea that should be pursued I can clean it up and prepare a PR.
Also, we could move further and split postFragments in:

HTML fragments -> ends in v-html
IMG fragments -> ends up in g-image
HREF fragments -> ends up in g-link

[dominiquedutra]

Been thinking about this.

I guess we could have g-image take a remoteSrc property and have the image processing pipeline download it. The source plugins would be responsible for determining which images need to be downloaded and processed locally, and not responsible for processing it.

[milosodalovic]

@dominiquedutra - I'm trying to solve the same problem. The commit on your fork doesn't exist, or it's not publicly accessible... Did you have any progress after your last post?

[hjvedvik]

@dominiquedutra Downloading and processing is indeed on our roadmap and would be great if you want to look into it :) I have been thinking about how it could be solved, but haven't done any testing yet. One approach would be to just let the source plugins download all its images when you start the project. But that will download lots of unnecessary images in development.

Another approach could be to have a download argument for g-image and image fields in the schema. Gridsome would then download the image and process it only when they were visited in development mode.

<g-image src="https://example.com/image.png" download />

query Post {
  image(download: true)
}

In the build process, we could somehow try to skip the GraphQL resolvers for external images while executing all the queries and run them in another step later in the process. Because downloading all the images while executing queries will be very slow. When downloading the images, we generate the base64 string and add the image to the process queue. Then patch the results from the GraphQL query in the previous step before the data file is written to disk.

Images in HTML content like the Wordpress post body could also be downloaded and processed in the same way, but just needs more logic to extract them from the HTML etc. I'm working on a GraphQL API which could let plugins do it instead of having each source plugin doing it manually. For example, a plugin could add a @downloadAssets directive to the schema which would process any String field.

type WordPressPost implements Node {
  title: String!
  content: String! @downloadAssets
}

Just a few ideas if they make any sense :)

[u12206050]

I would recommend a slightly different approach in that Gridsome only generates and stores the low quality svg placeholder for the image, and then loading the image from the source (be that an image cdn or file server, should not matter).

This way there is less time spent on downloading and processing images whilst still getting the benefit of the svg placeholders.

[milosodalovic]

Makes sense @hjvedvik - thanks!

I've been already working on a source plugin that would download all images during development (although aware of the possible issues). I managed to successfully download images and store them locally in the src/assets/images directory. The plan was to download one by one and as soon as the image is downloaded add it to GraphQL. Some pseudo code example:

api.loadSource(async store => {
      const images = store.addContentType('CustomPostTypeImage');
      ....
      const { data } = await axios.get( `${baseUrl}/wp-json/wp/v2/custom_post_type`);

      for (const post of data) {
        post.acf.gallery.forEach(async imgObj => {
          const url = imgObj.sizes.medium;
          const imgName = createImageName( imgObj.filename );
          await this.downloadImage(url, "src/assets/images", imgName);

          images.addNode({
            image: imgName,
            local_path: path.resolve('src/assets/images'),
            post: store.createReference(
              "CustomPostTypeTypeName",
              post.id
            )
          });
        });

However, addNode doesn't add nodes as I would expect - there is only id field added on the node and no other fields are available...
What I'm missing? I would rather not dig deeper onto GraphQL layer :D

Thanks

[u12206050]

@milosodalovic the mysql-source plugin for gridsome also has this feature implemented. The value of the field in GraphQL should be the full path to the image eg. /Users/john/projects/wp/temp-images/cool.jpg

Gridsome will pick that up as an image and do the rest for you.

[u12206050]

@milosodalovic You will find that you'll need to queue image downloads, otherwise you'll get some buffer issues.
You can reuse this helper code: file.js which has some helper functions for downloading, generating full path etc. Look at the plugin to see the rest of the implementation.

[milosodalovic]

Thanks @u12206050, I'm checking mysql-source plugin now.
I know that path should be full (absolute). I updated my comment above... Also aware that image downloads should be queued. However, I first tried to make a proof of concept by very simple version of the plugin working in just one specific case, before covering edge cases, polishing up and making it available for the rest of the community. Unfortunately, I stuck and don't see the reason at the moment...

In short, I have post added in graphql and I had expected the logic bellow to work and add image into graphql with the reference to the post. The node is added but with no fields except the id.

 const images = store.addContentType('CustomPostTypeImage');
   images.addNode({
            image: imgName,
            local_path: path.resolve('src/assets/images'),
            post: store.createReference(
              customPostTypeTypeName,
              postId
            )
     });

If I have added reference like this, it would appear in the list of available fields in graphql:
images.addReference("post", customPostTypeTypeName); Other fields are missing anyways.
Looks I'm missing something really obvious but cant figure it out...

[dominiquedutra]

I've been away for a few days and this issue escalated quickly :)

@milosodalovic I am currently using a custom gridsome.server.js instead of the Wordpress Source Plugin.

https://gist.github.com/dominiquedutra/8d48018c62b8270f51630de1eacc2711

That should work with 0.6.4. I have it on production at https://tudocondo.com.br.

---

@u12206050 In my case, I also need the image resizing capabilities of gridsome.
I believe that is one strong point of any static website generator.
My content team can upload the largest image possible to the backend for archiving and the transpiler will eat it up.

[milosodalovic]

Thanks @dominiquedutra! I'm checking ...
I had started the same way, by extending Wordpress Source Plugin, and keeping my version in the gridsome.server.js. Then I decided to make an independent plugin, and change things a bit. I've got all major problems solved now and just need to polish things and make it more flexible to be able to cover more use cases.

Thanks everyone for the help!

[sofyanlabs]

Hello, how to use this?
help me

[mjatharvest]

question for you @milosodalovic @dominiquedutra, with your implementations, did g-image optimize and resize the images, giving you srcset and lazy loading?

I have not found a way to get lazy loading to work with g-image and graphQL data and was wondering if the full system file path was the issue

Could you post a sample of what your graphQL data looks like from Wordpress?

[milosodalovic]

Hi @mjatharvest.
Yes - when you manage to store images locally and add them into graphql properly, g-image will pick it up and does all the rest for you.

You'll need full system path... read comments from @u12206050 and check out his repo, it think it should help you.

[sofyanlabs]

I've been away for a few days and this issue escalated quickly :)

@milosodalovic I am currently using a custom gridsome.server.js instead of the Wordpress Source Plugin.

https://gist.github.com/dominiquedutra/8d48018c62b8270f51630de1eacc2711

That should work with 0.6.4. I have it on production at https://tudocondo.com.br.

@u12206050 In my case, I also need the image resizing capabilities of gridsome.
I believe that is one strong point of any static website generator.
My content team can upload the largest image possible to the backend for archiving and the transpiler will eat it up.

@dominiquedutra i use your gridsome server, but some images download is corrupted.
how to fix it?

[u12206050]

I've added it as a PR #568

[mjatharvest]

@milosodalovic thank you, using the full system path did the trick

[dominiquedutra]

Glad to see this got added to the source-plugin.
Closing it.