Gatsby: Gatsby wordpress (DIVI Theme)

[anishanair6]

Hi Team,

I have a wordpress website which is being builded using DIVI theme. I have used divi page builder to create entire website and now i want to make my website headless. I understand gatsby fetches wordpress CMS contents using WP Rest API. But the page contents which is builded using DIVI is saved in the form of DIVI shortcodes in database. Which can be only rendered by divi modules. Since the WP Rest API also returns the shortcode, how can i build a gatsby website using this ?

[shrthnd]

@anishanair6, I think the best path forward here is to configure your WordPress project to output the expected markup from the get-go.

Generally speaking, you need the WP REST API to apply the do_shortcode() function to your post $content and $excerpt server-side, before returning the results to Gatsby through the API. I haven't tested this proposed solution, but here's a link to a stackoverflow answer which might get you started.

[anishanair6]

Hi @shrthnd ,

Thanks for the reply. I had tried the above proposed solution before

First of all what i have observed is , the above solution returns is a part of body element in html format. And i was able to create a full html by combining the output of header and footer. And another thing is the API tooks lot of time to process the request.

Now i have two concerns

1. Slowness in API
2. Is it a good idea to send entire html in API response and render in gatsby. Or only that body part i have to send ? If that is the case how can i incorporate all DIVI theme dependencies (css and javascript) in gatsby ?

[jasonbahl]

@anishanair6 Thank you so much for using Gatsby with WordPress! We're really excited for the future of this stack and excited to see how folks like you are innovating with it.

Quick Aside:

Before getting too deep into things, I want to make sure you're aware of is that we're working on a new WordPress source plugin for Gatsby that will use WPGraphQL (https://github.com/wp-graphql/wp-graphql) on the WordPress server. Gatsby will communicate with your WordPress site via GraphQL to fetch and cache data as Gatsby Nodes for use in your Gatsby site.

You can read more about this here: https://github.com/gatsbyjs/gatsby/issues/19292

Some Ideas

The way Divi and other page builders work in a "classic" WordPress theme is to render the shortcodes to HTML and output the HTML as content.

Then, styles and JS are enqueued to the page.

If we can query for the content rendered as HTML, _and_ query for the enqueued assets, we might be able to apply those assets to the template in Gatsby.

We cannot do this out of the box right now, but I imagine using WPGraphQL we _might_ be able to get something working like so:

{
  post( id: "hello-world" idType: URI ) {
    id
    title
    content # <- The HTML rendered by the Divi Page Builder would be output here
    enqueuedAssets {
       nodes {
          url # <- The url of any assets enqueued to this particular page
       }
    }
  }
}

If we were able to query enqueuedAssets for any given node from WordPress, we could use those assets in Gatsby and apply them to the template.

This could allow for Divi to work in a headless context.

I'll see if I can work up a proof of concept for how this might work with WPGraphQL and report back.

[anishanair6]

Thanks @jasonbahl for your valuable feedback.

[shrthnd]

@anishanair6, @jasonbahl's response answers many of _my_ questions*, but I'm following up with your original questions just in case:

1. Slowness in API

Not sure where exactly you are seeing slowness so I'll make an assumption (correct me if I'm wrong):

During Gatsby's bootstrap and build process the gatsby-source-wordpress plugin makes a request to the WP REST API for each page in your website.

When the request is made, WordPress queries the database and performs other server-side operations which can be a bottleneck depending on: the number of pages in your website, and the complexity of the page-layouts you have built with DIVI (every shortcode needs to be processed/interpreted), along with the resources available to your web-server, network speed, etc. (After the initial request to the REST API, Gatsby fetches image/file as well.)

If the slowness you're experiencing is with these initial API requests, an immediate solution may be to install a plugin to enable caching the WP REST API. I see a few options available on Github and the plugin directory but can't make a recommendation at this time.

1. Is it a good idea to send entire html in API response and render in gatsby. Or only that body part i have to send ?

It may not be ideal to request/return all of this HTML but is unavoidable when using page-builders like DIVI.

As @jasonbahl pointed out, the WPGraphql API currently mirrors the WP REST API by returning the standard WP $post->content object. You don't have access to assets defined in the < head > or above the

1. If that is the case how can i incorporate all DIVI theme dependencies (css and javascript) in gatsby ?

I'm not too familiar with DIVI theme standards in particular but am under the impression each page has a unique bundle of CSS/JS references enqueued through WordPress theme API. Being able to retrieve a list of enqueued assets from WPGraphql feels like the ideal solution and answer to your question but we'll have to wait patiently...

In the meanwhile, it's not elegant but, if you can infer a persistent url/naming convention for these assets, perhaps by installing an appropriate WP caching plugin and forcing them into "bundles", you may be able to consume them in your Gatsby project.

Does this make sense? Even if it does I'm not sure it's a good idea, and I think you'll still have to wrestle with loading/unloading DIVI-based JS inside your Gatsby application.

*is the WP Gatsby plugin available for testing gatsby-source-wordpress-experimental?

[anishanair6]

@shrthnd , Your assumption regarding slowness is right and initial build is taking too much time for me. My site having around 81 pages and which is medium complex. So i will try out your recommendation and update.

And regarding DIVI dependencies, i have tried by customizing html.js of gatsby and injected divi dependencies in head ? Is it a right way to do ? Can you elaborate more about your statement

by installing an appropriate WP caching plugin and forcing them into "bundles", you may be able to consume them in your Gatsby project

Also one more challenge i have faced is regarding the below configuration in the plugin

    searchAndReplaceContentUrls: {
      sourceUrl: "http://{source}",
      replacementUrl: "https://{replacement}",
    }

Due to this all my resource urls (ex: image urls configured in DIVI got replaced and giving 404 errors). Is there anyway to limit this configuration only for page routing ?

[jacobarriola]

@anishanair6 out of curiosity, what are your motivations for moving headless? If it's performance, I'm curious as to how much of a boost you'll get in this scenario given that you'll get all of the rendered markup (which I've found is pretty superfluous with page builders) + it's seems like you'll need to make render-blocking requests for the assets.

I'd love to see how the progress is going, since I'm running into some projects that want to hold on to their page builders, yet want to see significant bumps in their perf.

Thanks!

[anishanair6]

@jacobarriola If you ask me motivation for moving headless, the real reason is performance and security and ofcourse my client wants headless CMS with page builder capabilities. I had been gone through lot of bad situations interms of performance and security in wordpress on my previous projects and i wanted to have more control over front end and back end. So i need a decoupled architecture for this.

[shrthnd]

... initial build is taking too much time for me.

Does the process slow down significantly after enabling the do_shortcode function you implemented previously? Do you have a lot of images embedded in your content? If possible, watch your WordPress server logs while running the build process and see if you notice anything unusual.

If your WordPress installation or web-server is underperforming, or Gatsby has to process large amounts of content/image data after making the initial requests, I would expect to see some impact on performance during build. Network speed will also affect the experience if your WordPress website is not being run locally.

... regarding DIVI dependencies, i have tried by customizing html.js of gatsby and injected divi dependencies in head ? Is it a right way to do ?

This might work but, to @jacobarriola's point, you'll be missing out some of the core front-end performance gains (like lazy loading images, async script/style loading, dependency-injection, etc.)

What @jasonbahl described is the ideal scenario:

If we were able to query enqueuedAssets for any given node from WordPress, we could use those assets in Gatsby and apply them to the template.

This is _almost_ possible out of the box with WPGraphql (in a custom plugin or functions.php):

add_action( 'graphql_register_types', function() {
    global $wp_scripts;
    global $wp_styles;

    register_graphql_field( 'Page', 'enqueuedScripts', [
        'type' => [ 'list_of' => 'String' ],
        'description' => __( 'Page Scripts', 'wp-graphql' ),
        'resolve' => function( $post ) {
            $enqueued_scripts = array('scripts');
            foreach( $wp_scripts->queue as $handle ) {
                    $enqueued_scripts[] = $wp_scripts->registered[$handle]->src;
            }
            return $enqueued_scripts;
        }
    ] );

    register_graphql_field( 'Page', 'enqueuedStyles', [
        'type' => [ 'list_of' => 'String' ],
        'description' => __( 'Page Styles', 'wp-graphql' ),
        'resolve' => function( $post ) {
            $enqueued_styles = array('styles');
            foreach( $wp_styles->queue as $handle ) {
                    $enqueued_styles[] = $wp_styles->registered[$handle]->src;
            }
            return $enqueued_styles;
        }
    ] );
}, 10 );

If WordPress provided a mechanism to query enqueuedAssets for any given node from WordPress adding this feature would probably take no time at all. The problem is, when the graphql_register_types action is run, $wp_scripts or $wp_styles are empty because these objects are context-specific and constructed when the page is requested via a URL.

Based on this proposed solution I tried/failed to actually fetch the page/post and parse the contents for the relevant details. I didn't pursue this too much because it would add add'l requests to the process and will probably slow your experience further.

Otherwise, I've made an initial attempt to extend wp-graphql to include enqueuedAssets (scripts/styles) but it's not working yet. Not sure if/when I'll get to revisit this, or if it's even working in the right direction, but it might get you started.

I don't personally have a solution for you, @anishanair6, but wanted to share a record of my failures in case it helps you or others ultimately work through things.

Good luck!

[anishanair6]

@shrthnd , Thank you so much for your support. As i have explained in my initial post, i will be able to get entire html of a page and by using some scripts i could be able to segregate script urls/inline script and css urls/ inline css. And as a new rest API field i could be able to send back as page response along with content attribute.

1. How should i consume this new custom field from graphql in gatsby ?

2. How can i apply this scripts/css to dynamically created pages ?

[shrthnd]

No worries, @anishanair6--thanks for your patience.

How should i consume this new custom field from graphql in gatsby ?

Use the createRemoteFileNode helper from gatsby-source-filesystem inside gatsby-node.js to fetch and create local copies of your styles/scripts during build.

You can then reference these file nodes in your templates, similar to this example for preprocessing external images. Here is another example pulled from the gatsby-source-wordpress plugin.

How can i apply this scripts/css to dynamically created pages ?

useEffect hooks can be used to add/remove <script /> and <link /> elements from the page when your page/component is loaded/unloaded:

import { useEffect } from 'react';

export const useScript = url => {
  useEffect(() => {
    const script = document.createElement('script');

    script.src = url;
    script.async = true;

    document.body.appendChild(script);

    return () => {
      document.body.removeChild(script);
    }
  }, [url]);
};

export const useStyle = url => {
  useEffect(() => {
    const link = document.createElement('link');

    link.href = url;
    link.rel = "stylesheet";

    document.body.appendChild(link);

    return () => {
      document.body.removeChild(link);
    }
  }, [url]);
};

Import these hooks and use them in your page/post templates (to add bootstrap and jquery from a CDN, for example):

import { useStyle, useScript } from "../hooks"
// ...
export default ({ children }) => {
  useStyle("https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css")
  useScript("https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.0/jquery.min.js")
  useScript("https://code.jquery.com/jquery-3.4.1.slim.min.js")
  useScript("https://cdn.jsdelivr.net/npm/[email protected]/dist/umd/popper.min.js")
  useScript("https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/js/bootstrap.min.js")

  return (
  <div>
    {children}
  </div>
  )
}

[anishanair6]

@shrthnd Thanks for your reply !!!

using createRemoteFileNode downloading multiple urls and create node for each is working fine for me. When i tried to create a node with an array of file node ids, i am getting error from gastby during build. I am pasting my sample code below...

`const js_local_source = [];

node.resource_urls.script.external.forEach(function(js_url) {

    if (js_url) {

        let fileNode =  createRemoteFileNode({
            url: js_url,
            store,
            cache,
            createNode,
            parentNodeId: node.id,
            createNodeId
        })
        .catch(err => {
            console.log("caught error:", err)
        });

        if (fileNode) {
            js_local_source.push(fileNode.id);
        }

    }
});

if (js_local_source.length > 0) {
    node.localFilejs___NODE = js_local_source;
}`

[anishanair6]

@shrthnd The above issue is sorted out, but i am facing a challenge with slowness and which is eating lot of my time to figure out the cause. It seems to me that common.js is takes too long to load (Around 33 seconds it tooks for loading and site becomes extremely slow now)

Not getting the exact cause. Could you please help me to sort this out

[shrthnd]

Glad you pushed through, @anishanair6! 33 seconds is a problem. How big is the file you mentioned? Or, is the website making requests to the original, wordpress powered website during load?

Otherwise, is it slow when running in develop or also after running gatsby build && gatsby serve?

Without digging into your code it's hard to debug. If you could share the entire contents of your gatsby-node.js and gatsby-config.js along with the output from gatsby info I can see if the issue is reproducible on my end.

[anishanair6]

@shrthnd The file size is around 2.2 MB , yes the website is making request to this file during load. So i can clearly see an initial delay of 30 to 40 seconds (a blank screen) before the original content appears in the website. This delay is nothing but the load time of common.js and didn't understand whats make the size to increase, because initially file size was less and loads faster.

And site is slow when running on develop, i haven't tried buildand serve.

Please find which you have asked from below link :

gatsby-node.js gatsby-config.js gatsby info

[shrthnd]

Thank you for sharing, @anishanair6. This is basically what I was expecting to see. Would you mind sharing the contents of ./src/templates/BlogPost.js and/or any other components these assets into which these assets are being imported?

Update: could you follow these steps to make a reproducible test case?

Working from a public repository we can actively contribute to the same solution. If it helps, I can add the code you have shared to a hello-world starter and publish this on github, but I think it's to your benefit to show proof of work. My apologies for taking so long to bring us to this point!

[anishanair6]

@shrthnd Apologies for the late response. I have created a redproducable test case, please find github link for the same from below url

Gatsby-Wordpress

[shrthnd]

All good, @anishanair6. Thank for pulling this together. My gut says the reactHook approach may only be appropriate for external assets but I'm also not convinced it was the right suggestion on my part to begin with. If you haven't tried a full gatsby build yet I'd be interested to know if the results are the same.

[shrthnd]

@anishanair6, you might also consider signing into the Gatsby chat to talk through the issue with other devs in real time. Between the work you've put together, and our notes here, someone out there may be able to unblock you sooner than I.

Otherwise, my next step is to get a local instance of wordpress set up and connected to this example. I'll report back if my work leads to a breakthrough (or any identifiable next steps.)

[jasonbahl]

I wanted to follow up on what I said above:

It's not super easy or straightforward to do what I suggested at the moment. There's some changes that will need to be made to core WPGraphQL processing (specifically the model layer) to make this work.

But, I've made some good progress:

See: https://github.com/wp-graphql/wp-graphql/issues/1308#issuecomment-626902632

Once this is working, you'd be able to query for the content of a post (or page, or whatever) and also query for the scripts (and styles) enqueued by WordPress to apply to the specific template for the page.

The scripts would likely need to be loaded only when the window is available, as most scripts in WP are things like jQuery that assume the window is available.

PSEUDO CODE BELOW

I think there could easily be a package that abstracts this to a Fragment and Component. So you could potentially use in your Gatsby site like so (pseudo code):

// Import the fragment and component
import { EnqueuedScriptsFragment }, EnqueuedScripts from 'gatsby-theme-wordpress'

// Your page template
const MyPageTemplate = ({ title, enqueuedScripts }) => (
  <div>
     <h1>{title}</h1>
     <div>{content}</div>
     <EnqueuedScripts scripts={enqueuedScripts} /> # <-- Use the component, pass down the enqueuedScripts
  </div>
);

export pageQuery = gql`
{
  post( id: $id ) {
    id
    title
    content
    ...EnqueuedScriptsFragment # <-- Use the imported fragment
  }
}
${EnqueuedScriptsFragment} <-- Use the imported fragment
`;

[shrthnd]

Thank you, @jasonbahl. Based on your feedback I started working in this direction but ran into an issue interpolating the EnqueuedScriptsFragment:

Here is the relevant portion of my functions.php (note: I've verified these scripts do load in the front-end):

add_action( 'wp_enqueue_scripts', function() {

  global $post;

  // Query to single posts in the "mouse" category
  if ( is_single( $post ) && in_category('mouse', $post->ID ) ) {
    wp_enqueue_script( 'only-single-post-mouse-category', get_template_directory_uri() . '/js/single-post-mouse.js' );
  }

  // Query to all single posts
  if ( is_single() ) {
    wp_enqueue_script( 'only-single-post', get_template_directory_uri() . '/js/single-post.js' );
  }

  // Enqueue to all pages
  if ( is_page() ) {
    wp_enqueue_script( 'only-page', get_template_directory_uri() . '/js/single-page.js' );
  }

} );

My alpine docker dev environment does not work well and may be the issue (I have to manually restart to see changes at this point but this is not your problem. My machines just not loving me right now.)

See any major issues here?

Unrelated: the wp-graphiql plugin does not work for me off-the-shelf (404s on a couple of assets). Is this intentional?

@anishanair6, I began to modify your code to begin testing the performance of EnqueuedScriptsFragment.

Since you have this partially working on your side, you might be able to npm install gatsby-theme-wordpress and take a look at this pull request to take this a step further on your end.

These aren't major change but I did reformat your code in vscode with prettier (shift-option-f) to make this a little easier to read. If I get myself unstuck with the issue noted above I will certainly revisit.

[anishanair6]

Thank you @shrthnd

All good, @anishanair6. Thank for pulling this together. My gut says the reactHook approach may only be appropriate for external assets but I'm also not convinced it was the right suggestion on my part to begin with. If you haven't tried a full gatsby build yet I'd be interested to know if the results are the same.

Yes definitely i will update you the results once i done full build !!!

Since you have this partially working on your side, you might be able to npm install gatsby-theme-wordpress and take a look at this pull request to take this a step further on your end.

Yes i will update on this

[shrthnd]

Looking over my notes today, I think I need to register enqueuedAssets with the graphql_register_types action I mentioned in this comment (which calls register_graphql_field.)

I'll see how this works when I have a moment and will share any meaningful results.

[jasonbahl]

@shrthnd I'm still working on this in WPGraphQL. It's not been merged yet. Hoping to have a PR soon

[shrthnd]

Appreciate the effort, @jasonbahl. This is going to be great. Please let me know if I can test or otherwise support this work.

[jasonbahl]

@shrthnd I have a PR with a lot of initial work on this including heaps of tests.

check it out: https://github.com/wp-graphql/wp-graphql/issues/1308#issuecomment-628224050

[LekoArts]

@jasonbahl Can this issue be closed as the initial question(s) were answered? And refer to issues/PRs on your repository?

[jasonbahl]

@LekoArts @shrthnd @anishanair6 I think based on the conversations above and the recent work done in WPGrapQL, there are paths to resolving this.

I'm not sure we can officially say "Gatsby supports Divi", but there are paths to use the content Divi creates within Gatsby.

If you're using WPGraphQL + Gatsby, you can know fetch enqueued assets and use them in Gatsby. So you can query the rendered content (how Divi shortcodes are converted to HTML) and use that along with the enqueued JS and CSS that Divi provides and apply that to pages created in Gatsby.

If you're using Gatsby + the WP REST API, you could try to follow suit of the work I've done in WPGraphQL to expose enqueued assets in REST and then use the assets in Gatsby.

Another option would be to attempt to expose the actual Divi modules as JSON in either WP REST API or WPGraphQL and query the data that makes up the modules and output in custom components in Gatsby.

There are a lot of options.