Reactgo: How to load multiple data with preRenderMiddleware?

[k1tzu]

It looked like this before:

static need = [  // eslint-disable-line
        fetchData,
         fetchAnotherData
     ];

And now I'm trying to do:
<IndexRoute component={Dashboard} fetchData={fetchData, fetchAnotherData} />
and it doesn't work. It returns only "fetchAnotherData" results.

I'm completely stuck, help me please!

[StefanWerW]

Have a look on this #655 . I use this modified version of preRenderMiddleware and it does the job.

Advantages:

• Can handle multiple services
• fetchData can be either falsy (null, undefined etc), an action (function, Promise) or an array of either
• Reducer handle specific data fetched (ex: topics) and not in a generic way.
• Data fetchers can use params and query for specific page (ex: https://www.youtube.com/results?search_query=react --> fetch data for query search_query:react )

[k1tzu]

I can't understand something about your code:
server.jsx

preRenderMiddleware(store.dispatch, props)
      .then(data => {
                           store.dispatch({ type: types.FETCH_DATA_SUCCESS, data });

so we're looking forward here to get some data from the database as the result of preRenderMiddleware? Right?

app/middlewares/preRenderMiddleware.js

return Promise.all(flatDataFetchers.map(dataFetcher => dispatch(dataFetcher({ ...params, query }))))

It returns only when all fetchers are complete.

But if we look into the function it calls:
app/fetch-data/fetchVoteData.js

const fetchData = () => {
  return {
    type: types.GET_TOPICS,
    promise: voteService.getTopics()
  }
}

We'll see that it returns an object with 'type:' and 'promise:' keys. But not the data from the database. Is it normal?

      .then(data => { // {type: types.GET_TOPICS, promise: function }
                           store.dispatch({ type: types.FETCH_DATA_SUCCESS, data });

[k1tzu]

Ok I got it to return data, not the {pending} promises.

import {flatten} from 'lodash'

//Dispatch is from the store
//FetchData is either falsy (null, undefined etc), an action (function, Promise) or an array of either:
export default(dispatch, {routes, params, location: {query}}) => {
    // get all routes (that has fetchData) as a flat array
    const flatRoutes = flatten(routes.filter(({fetchData}) => !!fetchData));

    // get each routes fetchData
    const dataFetchers = flatRoutes.map(fr => fr.fetchData);

    // flatten dataFetchers in case of arrays (<Route fetchData={[a, b, c]})/>)
    const flatDataFetchers = flatten(dataFetchers);

    let dataFetchersObjects = flatDataFetchers.map(dataFetcher => {
        return dispatch(dataFetcher({...params, query}))
    });
    let promises = [];
    for (let i = 0; i < Object.keys(dataFetchersObjects).length; i++) {
        promises.push(dataFetchersObjects[i].promise);
    }

    return Promise.all(promises)
}

and app/fetch-data/fetchVoteData.js

import { voteService } from 'services';
import * as types from 'types';

const fetchData = () => {
  return {
    type: types.GET_TOPICS,
    promise: voteService.getTopics().then(res => res.data)
  }
};

export default fetchData;

[k1tzu]

I've ended up with:
app/fetch-data/fetchVoteData.js

...
  return {
    type: types.GET_TOPICS,
    promise: voteService.getTopics().then(res => {return {type: 'GET_TOPICS_SUCCESS', data: res.data}})
  }
...

server.jsx

...
      preRenderMiddleware(store.dispatch, props)
      .then(data => {
          data.forEach(returningCall => {
            store.dispatch({ type: returningCall.type, data: returningCall.data});
          });
        const html = pageRenderer(store, props);
        res.status(200).send(html);
      })
...

[montezume]

Thanks for the tips @k1tzu and @StefanWerW. Would be great if maintainers would accept a PR from one of you guys adding this (necessary!) functionality to the boilerplate.

[choonkending]

Sorry for taking so long to get back on this thread.

I love the discussion + hard work from contributors over here! And I agree that this is a necessary functionality for this repo. (I just added to the project board https://github.com/reactGo/reactGo/projects/1).

However, I am holding back in commenting/merging the PRs because I need to get into the proper headspace and understand all the concerns - possibly even suggest a combination of the different approaches. There's also another discussion https://github.com/reactGo/reactGo/issues/527 hehe. 🎅

[githubjosh]

spent days racking my brain on a related issue without success.

On fetchData I need to access redux store data in my action creator. The current implementation does not allow calling an action creator that returns a function with dispatch and getState as arguments: e.g. fn() { return (dispatch, getState()) => {}...

I really thik this should be baked in: i.e. all routing passes through actions where dispatch and getState are exposed. Any insight would be greatly appreciated.

[githubjosh]

@k1tzu your implementation above saved the day. Just needed some nominal refactoring in server.jsx. boo ya!

[choonkending]

I've been thinking pretty hard about this problem as I agree with @montezume that this is a critical feature that we should support in reactGo.

I'd like to involve @ZeroCho @StefanWerW @k1tzu @GGAlanSmithee @DraconPern (and anyone who is interested or left out!) who have all done excellent work with proposing changes.

Sorry for taking so long to get around to this.

I've summarised #487, #527 and #655 and will state the different possible approaches. Please raise your concern here if I've missed out something important.

What we want

• Fetch data from multiple services
• Populate redux store with data returned from fetch

In the PRs and issues raised, I've classified them in terms of data-fetching responsibility and broke them down further to different approaches.

Which module is responsible for fetching data from multiple services?

1. Fetch-data modules

Individual fetch-data modules, e.g. fetchPost or fetchVoteData handles data fetching from multiple services.

For example:

// fetchPost
const fetchPost = params => {
   const p1 = loadPost(params.id);
   const p2 = loadComments(params.id);
   return Promise.all([p1, p2])
                            .then(values => {
                                      return { post: values[0], comment: values[1] };
                            });
};

Approach I : Create page level reducer that handles returned data

The Redux store can have a page level reducer which contains other reducers, e.g. post, comment.

// reducers
import post from 'postReducer';
import comment from 'commentReducer';

const pageData = (initialState = {}, action) => {
   switch(action.type) {
      ... 
      case REQUEST_SUCCESS:
           return {
               post(action.data.post),
               comment(action.data.comment)
           };
   }
}

const rootReducer = combineReducers({
   pageData,
   isFetching
});

Advantage

• Adding extra services do not affect server.jsx and preRenderMiddleware.
  
  
  
  • dispatch actions are still handled on a page level by server.jsx
  
  
• fetchPost is only responsible for fetching data without redux knowledge

Disadvantage

• Redux reducers and store are coupled to fetchPost
  
  
  
  • fetchPost needs to return data in a certain format for pageData reducer to work.
  
  
  • If we change how we fetch return our data in fetchPost, our store will break
  
  

Approach II : Dispatch actions from individual fetch-data modules

Suggested by @DraconPern

• Pass in store or store.dispatch to preRenderMiddleware.

// preRenderMiddleware
function preRenderMiddleware({ routes, params }, store) {
  const matchedRoute = routes[routes.length - 1];
  const fetchDataHandler = matchedRoute.fetchData || defaultFetchData;
  return fetchDataHandler(params, store);
}

// fetchPost.js
const fetchPost = (params, store) => {
   const p1 = loadPost(params.id, store);
   const p2 = loadComments(params.id, store);
   return Promise.all([p1, p2]);
};

// Responsible for populating store with post data
const loadPost = (id , store) => {
   store.dispatch({ type: "GET_POSTS", data });
    return getPostService()
                   .then(data => {
                              store.dispatch({ type: "LOAD_POSTS", data });
                              return data;
                    })
                    .catch(err => {
                                return store.dispatch({ type: "RESET_POSTS", data });
                     });
};

// Similar thing for loadComments.js
// Responsible for populating store with comment data

Advantage

• Each fetch module, e.g. loadPost is responsible for initialising and loading store
• Adding services does not affect preRenderMiddleware or server.jsx
• fetchPost is not coupled to the redux store structure

Disadvantage

• Although preRenderMiddleware itself doesn't use redux, it's fetch-data methods still relies on redux. Ideally I would like our data fetching mechanism to just do one thing, fetch data, without having to load the store as well.

2. preRenderMiddleware fetches data and dispatches actions

Raised in https://github.com/reactGo/reactGo/pull/655 by @StefanWerW and also from @k1tzu

• fetchData methods are redux actions that are dispatched
• preRenderMiddleware finds all fetchData methods
  
  
  
  • dispatch all actions
  
  
  • redux middleware handles data fetching and populating store
  
  

Advantages

• Reducer handle specific data fetched (ex: topics) and not in a generic way.

Disadvantages

• data fetching is coupled to redux + redux specific middleware
• promiseMiddleware has been removed from our repo so will not apply

Other Proposals

• Rename preRenderMiddleware to something more meaningful and intention-revealing.
  
  
  
  • preRenderMiddleware is not a clear name, i.e. it's something that executes before rendering, but what is it for?
  
  
  • I think we should go with a clearer name, e.g. fetchDataForScreens or fetchDataForRoutes. That alone could be a source of debate lol.
  
  

Note: I've left out the flattening of fetch data methods for now.

So far I am leaning towards approach II.

In an ideal world, I would like to separate the fetching of data and the populating of the store (by individual reducers) as separate and clear steps. Each function should do one thing.

I will attempt approach II and see how it goes.

PS: This might be getting too long - perhaps I should make this a blog post on Medium. Let me know what you think!

[choonkending]

After speaking with @psimyn, who mentioned that in Approach I we could

• store all the data in pageData reducer
• use reselect to grab data when a Container component needs certain information from the redux store

The disadvantage of coupling fetchPost to the reducer would not exist if pageData is just a raw store. We would derive relevant information from the Redux Store with selectors.

If that is the case, I believe Approach I will work well. However, given that both Approach I and Approach II have their own merits I will attempt to spike out two solutions and see which fit better.

In the meantime, I have created https://github.com/reactGo/reactGo/issues/804 to rename preRenderMiddleware to eliminate confusion.

[choonkending]

I've gone ahead and did a quick spike on Approach II over here.

High level changes:

• fetchData handlers will define fetchSuccess and fetchError handlers which receive a store as an argument
• extract getMatchedRoute out of fetchDataForRoute
• A bit cleaner - fetch data doesnt need to receive store, it does need to have access to it.

I'm not sure if this is a good solution.

Approach I: Making a page level reducer works for all routes, however, reduces flexibility
Approach II: Pushes more responsibility to the Route component itself.

Thoughts?

[choonkending]

Any thoughts @psimyn @StefanWerW @k1tzu @ZeroCho (or anyone interested) ^

[psimyn]

still not hugely into passing the store through all the handlers just so it can dispatch. I'd base a decision on what the usage of each option looks like rather than the code change.

[Cleanshooter]

@choonkending Just reviewed your spike... still not clear on how this solves the "fetch data from multiple services" issue (#527)... I like the idea of having success and error handlers here though.

[omidahourai]

Has this been solved/ documented somewhere..?

[slavab89]

Basically, here you perform the actual fetchData logic. It returns a promise so you can do as many actions/fetches that you need. Just do something like

Promise.all([
  fetchA(),
  fetchB(),
  fetchC(),
]);

[omidahourai]

@slavab89 works but it maps all the results to the topics and i'm not seeing how/ why

edit- figured it out (via middleware.js)

[szkristof2h]

Disclaimer: I'm a bit new to javascript, so I'm not sure about the quality of my code.

Basically, there are only one or two things that vary every time you try to fetch data: the request url and the name of the action whose request and success/failure you will want to dispatch. Thus I think the simplest way would be to make it so that each route would have a url and an action (or a list of urls and actions if you want multiple fetches) and then pass these into the same function instead of making several different ones.

render/middleware.js

fetchDataForRoute({ props }, api, store.dispatch)
  .then(() => {
    const html = pageRenderer(store, props);
    res.status(200).send(html);
  })
  .catch(err => {
    console.error(err);
    res.status(500).json(err);
  });

I removed the two lines where the request and success action were dispatched and passed store.dispatch down fetchDataForRoute.

utils/fetchDataForRoute.js

function fetchDataForRoute({ props: { routes, params } }, api, dispatch) {
  const urls = routes[routes.length - 1].urls;

  return urls ? fetchData(api, urls, params, dispatch) : defaultFetchData();
}

The only thing fetchDataForRoute is doing is checking if the current route has urls to fetch data from and if yes, it passes them and the dispatch down the fetchData function.

fetch-data/fetchData.js

import { get } from '../actions/api';

const actionNames = {
  '/topic': 'GET_TOPIC',
  '/comments': 'GET_COMMENTS'
};
const paramsList = {
  '/topic': ['id'],
  '/comments': ['pId']
};

export function fetchData(api, urls, params, dispatch){
  const addPromise = function add(url) {
    let n = '';
    paramsList[url].forEach(p => {
      if(params.hasOwnProperty(p)) n += params[p] + '/';
    });
    return get(api, actionNames[url], url, '/' + n, dispatch);
  };

  const actions = urls.map(addPromise);
  const results = Promise.all(actions);

  return results.catch(err => { 
    console.log('Err: ' + err);
  });
}

The fetchData function takes the urls and creates a get request for each of them (taking the name of the actions from the actionNames object) and waits for all of them to be finished with Promise.all.

actions/api.js

export function get(api, action, url, params, dispatch) {
  dispatch({type: action + '_REQUEST'});

  return api.get(url + params)
  .then(res => {
    dispatch({
      type: action + '_SUCCESS',
      data: res.data,
    });
  })
  .catch((err) => {
    dispatch({type: action + '_FAILURE', err});
  });
}

I think this function is quite straightforward, it dispatches a request action and then a success/failure action.

So, let's say you're adding a new route for viewing an article and the comments belonging to it. This way all you have to do is to add an array of urls (e.g. ['/article', '/comments']) to the route and then update your actionNames object in fetchData.js with the url + action name pair, instead of having to write a new loadArticle() and loadComments() function.

Edit: I completely forgot about params (probably because I was only testing it with get requests that had none), so I updated my code to work with them.

The fetchDataForRoute function now passes down params to the fetchData function where it filters them with the help of paramsList object for each get requests. Although I doubt there'd be many occasions when you'd use multiple get requests with their own params for a route -- as if you wanted to get an article from the db with its comments you'd probably just use populate -- but I wanted to make it so that the option exists if it's ever needed.