Material-ui: [docs] /versions page rate limited issue with GitHub API

[fatso83]

The versions page of the docs does not work. It is caused by a 403 to the Github API, followed by some shoddy error handling.

(index):1 GET https://api.github.com/repos/mui-org/material-ui-docs/branches 403 (Forbidden)
versions.js:1 Uncaught (in promise) TypeError: n.map is not a function
    at n.<anonymous> (versions.js:1)
    at w (commons.9038b76362190d37f37e.js:87)
    at Generator._invoke (commons.9038b76362190d37f37e.js:87)
    at Generator.e.(/versions/anonymous function) [as next] (https://material-ui.com/_next/static/chunks/commons.9038b76362190d37f37e.js:87:12869)
    at n (commons.9038b76362190d37f37e.js:77)
    at u (commons.9038b76362190d37f37e.js:77)

We are on version 1.0 of MUI, so the newest docs doesn't work that well for us ...

• [x] This is not a v0.x issue.
• [x] I have searched the issues of this repository and believe that this is not a duplicate.

Steps to Reproduce 🕹

Go to https://material-ui.com/versions/.
Observe blank box in the middle and no info on versions.

[oliviertassinari]

@fatso83 You might have been rate-limited. We are happy to accept a pull request to handle the fetched network request error.

[fatso83]

Well, the handling is something else, but a few minutes after you commented the page is mysteriously back up again, which points to you being right the rate limiting (either that or someone did some magic in the background :smile_cat:).

Since this issue changed, I'm updating the title to reflect it and I'll see if I can smack together a working fix and test (using nise,fake-fetch or whatever).

Thanks!

[Android3000]

I get this on the home page as well (https://material-ui.com) due to a GET against https://api.github.com/repos/mui-org/material-ui being performed (maybe for the Star count or something?).

[oliviertassinari]

I think that we can spend our time on problems with a better ROI (using the latest version should be preferred). I propose we ignore this one.

[scottc-netflix]

@oliviertassinari I don't think you should ignore this, because especially over time, I think you will find that most users can't "just use the latest version". I don't think I have to explain to you that as a project grows it becomes more costly to upgrade libraries with many breaking changes.

It's really great that you put this page to find old documentation (unlike some people, Webpack developers, I'm looking at you!). However, many of your users will see an empty page even when going even for the first time, because they are using it at work and many people are on the same IP address. Netflix is now using YOUR project internally as the basis for tons of our internal apps, so now about 90% of the time I try to go to that page, even once a day, it doesn't work for me. So essentially you are telling your really serious users who use it for work and are the most active that you will no longer make it easy to find old documentation so they can do their jobs. I know you provide this for free so maybe you don't care, but if you want people to keep using it you should care. If you want people to use the latest version, the first thing you need to do is make the documentation accessible, so they can compare API between versions and tell how to translate props or CSS api or whatever.

A simple solution would be to just generate a static page once a day instead of having every user spamming the api. It's actually incredibly useful for a ton of people with very little work. Probably the single best thing you could do for your project actually. You could probably even do something hacky like cache it for the user in local storage and serve the cached version if the request fails.

FYI I noticed that even if I don't go to the version page it still tries to call the github api, not sure if for the same purpose or not so the situation might be greatly improved if you just didn't call it unnecessarily and only on the version page.

[oliviertassinari]

@scottc-netflix Thanks for the feedback, I like the idea of generating the version page once, on the server-side, during the getInitialProps method call. Do you want to work on it? :) I would also like to force the version page navigation to land on the main domain.

We do care about the DX, the next step for the project is to provide advanced enterprise components. The median project lifetime in an enterprise context is above 3 years (source: Sencha website). Even if we only release a breaking version once a year, the company's projects will likely lag behind. They should feel confident they can always access the documentation. So yes, we have a monetary incentive to care.

[oliviertassinari]

Since #18155, the documentation links https://material-ui.com/versions/ instead of /versions/. This means that we will (for new versioned documentation) have a single page that lists the different versions that are available instead of many. Basically, like https://www.ag-grid.com/archive/. The call to the GitHub API was introduced in the first place to provide flexibility. We might have even more flexibility with this structural change.

@scottc-netflix So, if you have some time to contribute a fix to your pain point, you are clear to go. We can move the call to the GitHub API to the getInitialProps method of Next.js. This means one call during the Netlify build vs one for each developer page display.

[fatso83]

@oliviertassinari If you are looking for help, does it make sense add a "help appreciated" tag to a closed issue? I'd assume most people looking to help would browse the open issues :thinking: But maybe you have some kind of policy here I don't know about :)

[oliviertassinari]

@mbrookes Do you want to reopen it?

[scottc-netflix]

@oliviertassinari Sure, I'll take a crack at it!

[oliviertassinari]

@scottc-netflix Cool, did you make progress? I think that we could get something to work with:

diff --git a/docs/pages/_app.js b/docs/pages/_app.js
index 8c21428fe..04f399d2d 100644
--- a/docs/pages/_app.js
+++ b/docs/pages/_app.js
@@ -336,6 +336,10 @@ export default class MyApp extends App {
   render() {
     const { Component, pageProps } = this.props;

+    if (pageProps.versions) {
+      global.__VERSION__ = pageProps.versions;
+    }
+
     return (
       <AppWrapper pageProps={pageProps}>
         <Component {...pageProps} />
@@ -344,9 +348,13 @@ export default class MyApp extends App {
   }
 }

-MyApp.getInitialProps = ({ ctx }) => {
+MyApp.getInitialProps = async ({ ctx, Component }) => {
   let pageProps = {};

+  if (Component.getInitialProps) {
+    pageProps = await Component.getInitialProps(ctx);
+  }
+
   if (!process.browser) {
     const redux = initRedux({
       options: {
@@ -354,6 +362,7 @@ MyApp.getInitialProps = ({ ctx }) => {
       },
     });
     pageProps = {
+      ...pageProps,
       // No need to include other initial Redux state because when it
       // initialises on the client-side it'll create it again anyway
       reduxServerState: redux.getState(),
diff --git a/docs/pages/versions.js b/docs/pages/versions.js
index 86b836c95..dd665d26b 100644
--- a/docs/pages/versions.js
+++ b/docs/pages/versions.js
@@ -1,4 +1,6 @@
 import React from 'react';
+import orderBy from 'lodash/orderBy';
+import sortedUniqBy from 'lodash/sortedUniqBy';
 import MarkdownDocs from 'docs/src/modules/components/MarkdownDocs';

 const req = require.context('docs/src/pages/versions', false, /\.(md|js|tsx)$/);
@@ -8,3 +10,46 @@ const reqPrefix = 'pages/versions';
 export default function Page() {
   return <MarkdownDocs req={req} reqSource={reqSource} reqPrefix={reqPrefix} />;
 }
+
+let cacheBranches = null;
+
+async function getBranches() {
+  try {
+    if (!cacheBranches) {
+      const result = await fetch('https://api.github.com/repos/mui-org/material-ui-docs/branches');
+      cacheBranches = await result.json();
+    }
+  } catch (err) {
+    // Swallow the exceptions.
+  }
+
+  cacheBranches = cacheBranches || [];
+  return cacheBranches;
+}
+
+Page.getInitialProps = async () => {
+  const FILTERED_BRANCHES = ['latest', 'staging', 'l10n', 'next'];
+
+  const branches = await getBranches();
+  let versions = branches.map(n => n.name);
+  versions = versions.filter(value => FILTERED_BRANCHES.indexOf(value) === -1);
+  versions = versions.map(version => ({
+    version,
+    // Replace dot with dashes for Netlify branch subdomains
+    url: `https://${version.replace(/\./g, '-')}.material-ui.com`,
+  }));
+  // Current version.
+  versions.push({
+    version: `v${process.env.LIB_VERSION}`,
+    url: 'https://material-ui.com',
+  });
+  // Legacy documentation.
+  versions.push({
+    version: 'v0',
+    url: 'https://v0.material-ui.com',
+  });
+  versions = orderBy(versions, 'version', 'desc');
+  versions = sortedUniqBy(versions, 'version');
+
+  return { versions };
+};
diff --git a/docs/src/pages/versions/StableVersions.js b/docs/src/pages/versions/StableVersions.js
index 1ba5d998b..fd2172fb3 100644
--- a/docs/src/pages/versions/StableVersions.js
+++ b/docs/src/pages/versions/StableVersions.js
@@ -12,7 +12,6 @@ import Typography from '@material-ui/core/Typography';
 import Link from 'docs/src/modules/components/Link';

 const GITHUB_RELEASE_BASE_URL = 'https://github.com/mui-org/material-ui/releases/tag/';
-const FILTERED_BRANCHES = ['latest', 'staging', 'l10n', 'next'];

 const styles = {
   root: {
@@ -22,53 +21,15 @@ const styles = {
   },
 };

-let cacheBranches = null;
-
-async function getBranches() {
-  try {
-    if (!cacheBranches) {
-      const result = await fetch('https://api.github.com/repos/mui-org/material-ui-docs/branches');
-      cacheBranches = await result.json();
-    }
-  } catch (err) {
-    // Swallow the exceptions.
-  }
-
-  cacheBranches = cacheBranches || [];
-  return cacheBranches;
-}
-
 function StableVersions(props) {
   const { classes } = props;
-  const [docs, setDocs] = React.useState([]);
+  const [docs, setDocs] = React.useState(global.__VERSION__);

-  React.useEffect(() => {
-    (async () => {
-      const branches = await getBranches();
-      let newDocs = branches.map(n => n.name);
-      newDocs = newDocs.filter(value => FILTERED_BRANCHES.indexOf(value) === -1);
-      newDocs = newDocs.map(version => ({
-        version,
-        // Replace dot with dashes for Netlify branch subdomains
-        url: `https://${version.replace(/\./g, '-')}.material-ui.com`,
-      }));
-      // Current version.
-      newDocs.push({
-        version: `v${process.env.LIB_VERSION}`,
-        url: document.location.origin,
-      });
-      // Legacy documentation.
-      newDocs.push({
-        version: 'v0',
-        url: 'https://v0.material-ui.com',
-      });
-      newDocs = orderBy(newDocs, 'version', 'desc');
-      newDocs = sortedUniqBy(newDocs, 'version');
-      // The latest version is always using the naked domain.
-      newDocs[0].url = 'https://material-ui.com';
-      setDocs(newDocs);
-    })();
-  }, []);
+  // React.useEffect(() => {
+  //   (async () => {
+  //     setDocs(global.__VERSION__);
+  //   })();
+  // }, []);

   return (
     <div className={classes.root}>
diff --git a/packages/material-ui/src/InputAdornment/InputAdornment.js b/packages/material-ui/src/InputAdornment/InputAdornment.js
index 6c05f8ab9..6c6149270 100644
--- a/packages/material-ui/src/InputAdornment/InputAdornment.js
+++ b/packages/material-ui/src/InputAdornment/InputAdornment.js
@@ -12,6 +12,7 @@ export const styles = {
@@ -12,6 +12,7 @@ export const styles = {
     height: '0.01em', // Fix IE 11 flexbox alignment. To remove at some point.
+  versions = orderBy(versions, 'version', 'desc');
+  versions = sortedUniqBy(versions, 'version');
+
+  return { versions };
+};
diff --git a/docs/src/pages/versions/StableVersions.js b/docs/src/pages/versions/StableVersions.js
index 1ba5d998b..fd2172fb3 100644
--- a/docs/src/pages/versions/StableVersions.js
+++ b/docs/src/pages/versions/StableVersions.js
@@ -12,7 +12,6 @@ import Typography from '@material-ui/core/Typography';
 import Link from 'docs/src/modules/components/Link';

 const GITHUB_RELEASE_BASE_URL = 'https://github.com/mui-org/material-ui/releases/tag/';
-const FILTERED_BRANCHES = ['latest', 'staging', 'l10n', 'next'];

 const styles = {
   root: {
@@ -22,53 +21,15 @@ const styles = {
   },
 };

-let cacheBranches = null;
-
-async function getBranches() {
-  try {
-    if (!cacheBranches) {
-      const result = await fetch('https://api.github.com/repos/mui-org/material-ui-docs/branches');
-      cacheBranches = await result.json();
-    }
-  } catch (err) {
-    // Swallow the exceptions.
-  }
-
-  cacheBranches = cacheBranches || [];
-  return cacheBranches;
-}
-
 function StableVersions(props) {
   const { classes } = props;
-  const [docs, setDocs] = React.useState([]);
+  const [docs, setDocs] = React.useState(global.__VERSION__);

-  React.useEffect(() => {
-    (async () => {
-      const branches = await getBranches();
-      let newDocs = branches.map(n => n.name);
-      newDocs = newDocs.filter(value => FILTERED_BRANCHES.indexOf(value) === -1);
-      newDocs = newDocs.map(version => ({
-        version,
-        // Replace dot with dashes for Netlify branch subdomains
-        url: `https://${version.replace(/\./g, '-')}.material-ui.com`,
-      }));
-      // Current version.
-      newDocs.push({
-        version: `v${process.env.LIB_VERSION}`,
-        url: document.location.origin,
-      });
-      // Legacy documentation.
-      newDocs.push({
-        version: 'v0',
-        url: 'https://v0.material-ui.com',
-      });
-      newDocs = orderBy(newDocs, 'version', 'desc');
-      newDocs = sortedUniqBy(newDocs, 'version');
-      // The latest version is always using the naked domain.
-      newDocs[0].url = 'https://material-ui.com';
-      setDocs(newDocs);
-    })();
-  }, []);

   return (
     <div className={classes.root}>