Next.js: Support locating non-page js/jsx files with page files

[zenflow]

Original title: Support having non-page js/jsx files in pages dir

This is an often requested feature. See issues #3508 #1914 #1689 #1545 #988 and there are likely more. edit: Also #3183.

This issue was fixed in PR #3195 but somewhere along the way the pagesGlobPattern config became broken, and was cleaned out in PR #3578.

This feature would allow us to place files that support specific pages, with the pages that they support. These "support" files could be tests, components, utilities, and more. This would improve the application's file organization, especially for large projects.

[zenflow]

For clarification

This is the folder structure we want to avoid, especially if it is to scale to have more than 3 pages in a hierarchy more than 1 deep:

• components/
  
  
  
  • chat/
  
  
  • timeline/
  
  
  • (... index components)
  
  
• helpers/
  
  
  
  • chat/
  
  
  • timeline/
  
  
  • (... index helpers)
  
  
• pages/
  
  
  
  • chat.js
  
  
  • timeline.js
  
  
  • index.js
  
  
• tests/
  
  
  
  • chat/
  
  
  • timeline/
    
    (... index tests)
  
  

This would be more maintainable at scale, since we don't duplicate the (root->(chat, timeline)) structure for each file type:

• pages/
  
  
  
  • __tests__/
  
  
  • lib/
  
  
  • components/
  
  
  • helpers/
  
  
  • chat/
  
  
  • __tests__/
  
  
  • lib/
    
    
    
    
    
    • components/
    
    
    
    • helpers/
    
    
    
  
  
  • index.js
  
  
  • timeline/
  
  
  • __tests__/
  
  
  • lib/
    
    
    
    
    
    • components/
    
    
    
    • helpers/
    
    
    
  
  
  • index.js
  
  
  • index.js
  
  

[zenflow]

Don't see any harm in adding this 👍 I do want to keep this as 'internal' (undocumented) cause it's not a practice we endorse. @timneutkens in PR #3195

@timneutkens Why would you not endorse this practice?

[zenflow]

I see two solutions:

1. Re-implement pagesGlobPattern config, but this time:
   
   
   
   • remove the 'pages/' bit from the default, and just prefix it before using, since this cannot be changed (pages must be in the pages/ dir, otherwise this solution becomes more complicated)
   
   
   • add tests to make sure there is not a regression
   
   
2. Take a convention-over-configuration approach, and make all lib/, __tests__/, and __mocks__/ folders ignored. __test__/ and __mock__/ (singular) too

[zenflow]

@timneutkens @sergiodxa Would you be open to a PR for either of the above solutions? Or perhaps a different solution?

[moaxaca]

This really is a deal breaker for people designing large more enterprise applications.

Some notable solutions that allow similar functionality:
https://www.gatsbyjs.org/ Static Generation
https://github.com/jaredpalmer/after.js SSR - React Router

If I had the community support, I would consider making my fork something more modularized & flexible then pushing it to npm something like flext.

[timneutkens]

You could actually try to do this:

1. Make a directory, eg components
2. Put all the code that you want to put into pages into components
3. Use symlinks to link the actual pages to the pages directory

This gives more granular control over routes that are accessible to the outside world. And is safer and clearer than providing exclude/includes.

but somewhere along the way the pagesGlobPattern config became broken

To be clear, pagesGlobPattern was always broken. It only accounted for next build scenario. And I removed it because it was obsolete because of that.

This really is a deal breaker for people designing large more enterprise applications.

We're continuously adding features/improvements to face bigger scale, as our own app grows (currently over a thousand components), and as the community gives feedback.

[moaxaca]

@timneutkens That's clever, I will give it a shot. Hopefully, the windows guys on my team don't hate me for committing a symlink if it works!

This being said, I would prefer to see a roadmap or a consensus from the maintainers on pivotal features.

[rauchg]

@moaxaca the only reason we're hesitant to support the glob is precisely because we are concerned that at scale it could open critical security concerns.

We basically would be encouraging people to put into pages/ things that are not meant to be routable, that they have to carefully exclude (as opposed to opt into and include explicitly).

For example, a side effect of not excluding __tests__ in the example above would be to execute code that results in denial of service, because it's not meant to be run in the context of a HTTP request.

I don't consider it "enterprise ready" to go back on what's otherwise a very clear and secure pattern, but we're always listening to the community for different approaches, in case one is suggested here.

In fact, it suffices to do some light research into the vast literature of PHP vulnerabilities to find many that fall under "I exposed a .php file to the end user that was not meant to be world-accessible".

[zenflow]

@timneutkens I am a Windows guy (don't judge) so the symlink approach will not work for me, and besides, there should be a portable solution. But thank you very much for the suggestion!

[zenflow]

@rauchg

but we're always listening to the community for different approaches, in case one is suggested here.

What about the second solution that I proposed above?

Take a convention-over-configuration approach, and make all lib/, __tests__/, and __mocks__/ folders ignored. __test__/ and __mock__/ (singular) too

It seems that this would be less error-prone; users would not have to carefully exclude the files (i.e. write a glob pattern), they just have to make sure that they don't misspell lib or __tests__ or __test__.

This would also be a very clear pattern I think, no?

[zenflow]

If this is still too error-prone, maybe we could still work with this idea, and extend it with something like "print a warning if a dir like lbi is detected", or something else

[gihrig]

I worked around this issue as follows:

Think of ./pages as "routes", move existing pages content to another directory.

Create ./containers directory
Move contents of ./pages into ./containers
Create 'link files' in pages. e.g. ./pages/index.js:

import Home from '../containers/Home'

export default Home

This allows for any complex folder structure in ./pages for matching complex url's and does not risk unwanted exposure of content that should not be public.

[zenflow]

Beautiful solution @gihrig! 🎉 This solves the problem!

Though I still would prefer not to have to write modules like:

import Foo from '../../../../../path/to/deeply/nested/container/Foo'

export default Foo

and if it was approved I would still make a PR implementing what I proposed above.

[rauchg]

By the way, I also think we do need a solution to not having to write so many ../../, but I think that can be done with babel? It's nice to be able to register a "root", and be able to use it anywhere like so:

import "containers/…"

That's also generally useful throughout the codebase, so maybe @gihrig's solution has to be extended with some babel plugin in case you have too many nested paths.

I believe @hanford is using such a plugin successfully already.

tl;DR: @gihrig's solution is sufficient. It can be combined with the "root" import solution which is orthogonal and generally useful beyond pages/

[hanford]

@rauchg @zenflow - We're using babel-plugin-module-resolver at Eaze, and I'm a huge fan

[zenflow]

In the same way that writing so many ../../ is irritating and less-than-elegant, so is this

// pages/some/deeply/nested/Foo.js
import Foo from "containers/some/deeply/nested/Foo"
export default Foo

note the repetition of "some/deeply/nested/Foo"

Also

1. This type of Babel plugin breaks code completion in most (if not all) IDEs (as well as other code intelligence features like ctrl+clicking the import path to open the imported file)
2. The path is ambiguous. It could also refer to <project root>/node_modules/containers/some/deeply/nested/Foo.js. And this is what a casual observer would probably assume it's referring to.

@rauchg Why not make some particular folder names ignored, as I proposed? Even if it is only a single folder name (like lib) that would be enough since you could place everything (tests, components, utilities, etc.) there.

[zenflow]

@hanford Can you confirm # 1 in my above comment?

edit: I see here that with configuration the code completion in the popular IDEs can still work with babel-plugin-module-resolver https://github.com/tleunen/babel-plugin-module-resolver#editors-autocompletion

[gihrig]

Though I still would prefer not to have to write modules like:

import Foo from '../../../../../path/to/deeply/nested/container/Foo'

export default Foo

Add module-resolver to .babelrc and it becomes

import Foo from 'containers/Foo'

export default Foo

.babelrc:

...
  "plugins": [
...
    [
      "module-resolver",
      {
        "root": ["./"],
        "alias": {
          "components": "./src/components",
          "containers": "./src/containers",
          "graphql": "./src/graphql",
          "lib": "./src/lib",
          "pages": "./src/pages",
          "src": "./src"
        }
      }
    ]
  ]

I keep my 'source' under ./src, make adjustments as appropriate to your configuration.

@hanford beat me by a few, but the details may still be helpful... 😄

[gihrig]

@zenflow I'm working with VS Code:

Re: 1. I have not noticed any problems. I have a pretty strict ESLint setup an it has always told me when I have an unresolved module.

Re: 2. I avoid module nesting more that two level below components or containers. I usually match nesting between containers and pages.

And yeah, it would be super awesome if Next could provide a way to avoid the need for the boilerplate in pages 😄

[zenflow]

Sorry @gihrig could you clarify your comment about # 2? I'm not sure I understand what you mean, or if I do I don't see how it relates to that issue.

[gihrig]

@zenflow Yeah, I was commenting on problems with deeply nested folders (I try to avoid that).

On closer reading, I see you were referring to an 'absolute' import being expected to be under node_modules and this possibly confusing those unfamiliar with the setup.

That's a valid point, but since import Foo from 'folder/Foo' under node_modules is less common (though not unheard of) I don't see it as a big issue. Once learned, I don't think devs will have an issue with it, but there is a learning curve, to be sure.

[moaxaca]

@rauchg that makes sense and I completely understand how this could be a loaded gun handed to the masses. I have a ton of empathy for library maintainers that protect their users from themselves.

But, (and there's always a butt) the current implementation is very similar to page controller pattern (Talk about some ancient PHP!), and assuming stick with the file system delegating routes, issues like this will always arise.

This being said, A NextServer Router that allows route definitions could be optimal.

I currently use a Koa instance to handle all this dynamic routing, and while its excellent on the initial hit, it quickly falls apart on the frontend router. I relish the idea of having a router could resolve against my koa definitions while allowing me to structure my app in a modular fashion.

Also just to mention, we use Lerna and would love to package pieces our interface in different bundles. A router that could be composed would be amazing. We currently use a similar pattern to the one mentioned above where we import container then just manually link them to root pages.

Also if you guys ever need help. I would be happy to contribute some of these features.

[zenflow]

@gihrig Yeah, it's just a minor issue. # 1 is a minor issue as well, since we will not be working in these "link files" often, but I think it's still an issue, since people use a variety of IDE's, each with it's own way of supporting code completion with babel-plugin-module-resolver.

[zenflow]

@moaxaca Let's try to keep this issue concentrated on "Support locating non-page js/jsx files with page files".. I might suggest opening a separate issue for "file-system based routing is not flexible". (btw I think the resolution of issue #257 might help with your modularization issue since you will be able to run multiple apps with common dependencies on the same domain)

[moaxaca]

@zenflow Ok, but I am going to leave my comment. I genuinely feel the root of this issue is in the routing mechanics.

Also ty for the PR I didn't consider multiple apps 👍.

[zenflow]

The only downside I see with my proposal is that you can no longer have routes containing /lib/, /__tests__/, etc. (unless you are using one of the router extensions), but I think that is a reasonable sacrifice considering how often those strings are used in routes.

@moaxaca @gihrig @rauchg Do you see any other downsides or problems?

[moaxaca]

@zenflow I agree with @timneutkens I feel like this may create a false sense of security and be prone to exposing too much accidentally.

If I were to take a convention over configuration approach, I would prefer a solution where I explicitly define my pages through a .pages.js|jsx|ts extension. They share a common naming convention and don't require blacklisted words like __tests__ __mocks__ lib.

[arunoda]

@zenflow I don't think we don't need any huge customization for what you wanted.
Our pages directory is something specially. It's the entry point of for a page.
You don't need to put your all the files inside this.

Try using the following approach:

• pages

  • home.js
  • app.js
  • index.js

• modules

  • home
    
    
    
    • components
    
    
    • containers
    
    
    • helpers
    
    
    • actions
    
    
  • app
    
    
    
    • components
    
    
    • containers
    
    
    • helpers
    
    
    • actions
    
    

[zenflow]

@moaxaca Do you think people would be prone to misspelling lib? Yes/no?... Like I said: "Even if it is only a single folder name (like lib) that would be enough since you could place everything (tests, components, utilities, etc.) there."

btw it was @rauchg that said that and I believe (I could be wrong) he was talking only about the pagesGlobPattern approach, hence "For example, a side effect of not excluding __tests__ in the example above" which does not apply to the convention over configuration (2nd) approach.

I would personally be OK with a .page.js extension, but I don't think (edit) I'm not sure it's a good idea because of the reasons here. Particularly, it would be a breaking change for every single Next.js app out there.

[arunoda]

Adding to above my comment:

• pages directory in the routing system of Next.js
• You can do advance routing with our custom routing API and completely disable file system based routing.

What we are going to do next is allowing to use custom routers like ReactRouter easily. So, that's all you need.

If you need simplicity, you can use our pages system. Or if you are familiar with RR, go with that.

[zenflow]

@arunoda Thanks for the suggestion. This seems to be basically the same solution that @gihrig stated above https://github.com/zeit/next.js/issues/3728#issuecomment-363964953 .. It is a good workable solution, but even combined with babel-plugin-module-resolver, it is not very ergonomic or convenient (see above discussion). And I don't see why we can't just always ignore lib folders by convention. I do not think "lib" is prone to being misspelled, which would open security vulnerabilities. Thoughts? Would "ignored lib folders" be a "huge customization" in terms of implementing (just to be clear)?

If you need simplicity, you can use our pages system. Or if you are familiar with RR, go with that.

I need simplicity. I also need "locating non-page js/jsx files with page files" (for simplicity in organization). If we can't implement "ignored lib folders" then the simplest way to achieve "locating non-page js/jsx files with page files" will be to use @gihrig's solution (without babel-plugin-module-resolver) which will work, but is less than ideal.

[arunoda]

@zenflow

And I don't see why we can't just always ignore lib folders by convention

Basically we can't do this. It's your preference. But not others. (Including myself)
But I understand you.

There are two ways to solve this problem (as others suggest):

1. Use pages directory as your routing end points. And use another directory called modules to build your application architecture.
2. You can change our webpack's entry options to ignore lib or __tests__ directories. Use our webpack config option in next.config.js (You can even create a Next.js plugin out of it)

Above are the current solution to this problem. To me, both seems viable and the first option looks nice.

As a long term solution, we are going to support React Router (and other similar routers). So you don't need to use the pages directory at all.

I'm closing this issue because we've some decent solutions and we are on the way of adding a long term solution.

But feel free to discuss.

[zenflow]

Thank you @arunoda for your very sensible response!

Basically we can't do this. It's your preference. But not others. (Including myself) But I understand you.

This makes sense, and I must agree. I just don't understand why this is not the preference of others 😆 I am guessing it's because you would not be able to have /lib/ contained in any routes? oh well..

You can change our webpack's entry options to ignore lib or __tests__ directories. Use our webpack config option in next.config.js (You can even create a Next.js plugin out of it)

This option had not been suggested, and I had not considered it. It looks like I will have to rely on some internals, but I really want a solution without all the boilerplate of # 1 if possible, so I will look into a way to do this reliably and efficiently in a plugin.

Thank you everyone!

[arunoda]

@zenflow

Here's how you can modify entries. This is the content for next.config.js

module.exports = {
  webpack(cfg, { isServer }) {
    const originalEntry = cfg.entry
    cfg.entry = async () => {
      const entries = await originalEntry()
      // Add something to main.js
      entries['main.js'].unshift('./client/bootstrap.js')
      // Add or remove entries as you want
      // ...

      return entries
    }

    return cfg
  }
}

[zenflow]

@arunoda Ah! No reliance on internals! Beautiful! Why didn't I think of that? Thank you!

[moaxaca]

@arunoda The ability to use a custom router like ReactRouter is a dream. 👍
Thanks, guys!

[stipsan]

I made a plugin for this: https://github.com/stipsan/next-without-jest

Dropping the link in case anyone stops by (even though the issue is closed) 😄

[timneutkens]

@stipsan note that this will still break commons chunk config, since every .js file inside pages counts as an entrypoint.

[stipsan]

@timneutkens ouch, definitely don't want that!

I am using this reducer to try and filter out both entrypoints and their chunks:
https://github.com/stipsan/next-without-jest/blob/f764e180f7d45803307476b1008f5f5b3614169e/index.js#L33-L47

Is there a way I can modify this to handle commons chunk config? 🤔

[dcalhoun]

@timneutkens would you be willing to expand on your commons chunk config comment please? What exactly breaks? What code breaks it? I am also interested in configuring next.config.js to ignore test files within the pages directory.

[timneutkens]

commons config uses totalPages * 0.5 for minChunks, pages are found using a glob similar to pages/**/*.js.

[dcalhoun]

@timneutkens ah, ok. I can understand how that would break things. Thank you.

I noticed you referenced the removal of the undocumented pagesGlobPattern before, but I would love to see the functionality reinstated in some form. Colocating tests seem to be a fairly common practice now.

[stipsan]

I would also appreciate some clarity here as well @timneutkens. Is colocated tests a pattern that something next.js supports by configuration or is it a hard no?
I don’t want to encourage people to use next this way, with stuff like my plugin, if that’s the case. Your response would help me here and be greatly appreciated 🙂

[austincondiff]

Sorry to open up an old issue but have we gotten anywhere with this? What if all file and folder names prefixed with an underscore were ignored? e.g. pages/my-page/_queries.js or pages/my-page/_components/MyComponent.jsx.

Using this approach, you could also do something like,

pages/
  my-page/
    _tests/
      myTest.js
    _components/
      MyComponent.jsx
    _queries.js
    index.js

Or even...

pages/
  my-page/
    _lib/
      tests/
        myTest.js
      components/
        MyComponent.jsx
      queries.js
    index.js

[shynome]

@austincondiff
maybe we can use a config like this, and rename index.tsx to index.page.tsx to ignore files which is not a page, it work at Next.js version ^9.0.4

module.exports = {
  pageExtensions: ["page.tsx"],
}

[namesphill]

How is this not something Next supports? IMO, that's a ironically Back(wards)...

[timneutkens]

How is this not something Next supports? IMO, that's a ironically Back(wards)...

You can read this whole thread, it's clearly explained, in particular https://github.com/zeit/next.js/issues/3728#issuecomment-363946415

[namesphill]

Yes, I understand your concerns, but isn't that argument a bit of a slippery slope. I mean, one cannot refuse to give users the option of something exclusively because they could use it agains themselves.
In such a case, why enable people to put any code anywhere, they could always put an API key somewhere.
What I mean, is that despite your intentions being aligned with the best interests of the developer, people should just get the option.
Do you really see anyone posting something like "I was hacked because of next" because they accidentally posted something very sensitive under pages and did not exclude it?

[egemon]

If someone is passing by, my summary would be like this:
1) https://github.com/zeit/next.js/issues/3728#issuecomment-363964953 - this is the best option which available now
2) https://github.com/zeit/next.js/issues/3728#issuecomment-523789071 - this is the best options that still may be implemented
Also I heard that with custom server u can use next as middleware and do magic in custom server. I am not ready to spend time on that and this is not silver bullet for all.

[ziserman]

Just want to add to solution with ./containers directory.

Instead of ./containers folder you can use ./src/pages directory to build any structure (components, tests etc.) and import pages to root ./pages for routing since:

1. we have another usage for ./containers folder time to time (I mean React Container Component)
2. src/pages will be ignored if pages is present in the root directory (Next.js docs).

[blujedis]

@rauchg I don't see this request (it's not an issue) going away any time soon. While I largely agree as to the security related issues, I would argue that if Next is smart enough to look for "pages" in root or "src/pages" and it's smart enough to discriminate or ignore other files/dirs it stands to reason this could be solved with a convention, even if a bit opinionated. While there are workarounds, I'll be honest this really does drive me bonkers at times ;)

[johot]

Thank you for your suggestion @gihrig

My current go to solution is very simple and becomes very nice to work with, this is ofc a matter of personal taste but instead of creating a folder called features, modules, container etc I simply use the following sample structure:

// files in pages just render a component from the `src` folder and are basically empty/very think except for data fetching (which can also be moved into `src` folder if you want to)
pages\...
src\common\...
src\start\...
src\profile\...
src\contact\...

It might feel odd to see the pages folder outside the src directory but it really makes it easier to work with as you don't mix them up as easily. I almost see the pages directory as a configuration folder and the real meat of the application lives inside the src folder. By doing this I could also avoid coming up with a good name for the "features" root folder 😄