Create-react-app: add a simple test runner?

[threepointone]

An important target group for this app (imo) are beginners and prototype builders. From my experience, both these groups would benefit greatly from having simple test infrastructure bundled with this starter.

From my own experience, I usually have a script.js that I'd include into the main module, but that's pretty hacky until I finally setup (whatever flavor of test infra is popular). The value I get is from being able to write code that'll run on every edit-save _separately_ from my app, but that later morphs into a unit test suite or so on. For beginners, they'd build their app, but write tests for smaller parts of the app they're building, while learning to write tests for their code.

To be clear, I'm not advocating any specific framework/suite/assert libs; I'd just like to be able to run even a if(true) throw new Error script on every change.

Thoughts?

Thanks for the great tool!

[gaearon]

I agree we should ship a test runner (Jest, ava, mocha, whatever) as long as we can stick with “no configuration” as the guiding design principle.

Other thoughts:

• We’re probably not going to include anything for running tests in browser because of the complexity it involves.
• We’re probably not going to include jsdom either.
• Testing app logic with a simple Node runner using Jasmine (or what do people use these days?) seems like a good idea.

[modernserf]

How about tape? It's super simple and it already _almost_ works -- if you restrict yourself to the subset of ES6 that your version of node supports you can use it now; all one would need to get it working is running it in babel-node; i.e. react-scripts test would run babel-node src/test.js.

[threepointone]

With you on zero config; running src/tests.js in a 'watch' mode on npm start (and once on npm run build?) would be good. You sure about not running it in a 'browser' though? Phantomjs works pretty nicely, and wouldn't need any config/ explicit invocation/require iirc.

[ffxsam]

Huge vote for Mocha + Enzyme + Sinon. This probably requires JSdom, at least according to Enzyme docs.

[aaronplanell]

I'm agree with Threepointone. Maybe PhantomJS with Mocha.

[selfup]

@ffxsam Enzyme would keep things nice and simple.

Thanks adding it to the conversation!

[threepointone]

also headless chrome is round the corner, so that'll be a great solution with no 'dependencies'

[aweary]

I'd be happy to help and/or answer any questions about using Enzyme here!

[ffxsam]

Is PhantomJS really needed though? It seems a bit overkill, when Enzyme can easily simulate events on components. If I'm not mistaken, PhantomJS & Selenium are more for E2E testing, not unit testing.

[wdhorton]

@ffxsam Agree on Enzyme (it's great), and we wouldn't need jsdom if we do shallow rendering.

[ffxsam]

jsdom is only required for mount and render, correct?

[tristanoneil]

Another vote for Enzyme, simplifies testing considerably, also the official React docs kind of recommend it now 😄

[gaearon]

To make it clear, the tool currently only abstracts build dependencies so it shouldn’t matter too much what library you use to make assertions / mount tree / etc. Choosing the runner and conventions is what we need to do.

[wdhorton]

I'd vote Mocha then

[AdamBrodzinski]

Personally I think a "no config" setup would make sense with Jasmine, as it has all the things you need (assertions, mocking, etc..) with no configs. Where as Mocha might require chai and other things.

[wdhorton]

That's a good point, but based on what @gaearon said above, I don't think we're as concerned with the assertion library because it's not a build dependency. I'd still favor Mocha just because I think it's more commonly used, but I have no problem with Jasmine if that's the way we want to go.

[ffxsam]

If many people will be using Enzyme, I think Mocha makes more sense and it's quite easy to set up. Enzyme was designed to be used with Mocha, though of course it can be used with any test-runner.

Enzyme+Jasmine is a separate repo, FYI.

[wdhorton]

A few questions about conventions:
Can we put the tests in a test folder on the same level as src?
Should we include a template test, since there's a template component?
Do people prefer explicit extensions on the test files (i.e. '.spec.js') or just run everything in the test folder?

I'm thinking something like:

my-app/
  src/
    App.js
    ...
  test/
    App[.spec?].js

[gaearon]

I'd err on the side of having __tests__ folders anywhere in the tree. It's very painful to write imports that go many folders up and then many folders down. Also colocating related code helps comprehension.

[ariporad]

Hi all,

I have no qualifications whatsoever, so feel free to completely ignore this comment.

I personally would advocate for considering anything ending with .test.js or .spec.js as a test, because it allows you to either put your tests next to your code (as I prefer to do), or throughout the tree in __tests__. It would also be really easy to look for tests in a tests directory on the same level as src as well.

[Place1]

I'd err on the side of having tests folders anywhere in the tree. It's very painful to write imports that go many folders up and then many folders down. Also colocating related code helps comprehension.

I agree that imports begin to look ugly. In past projects i've usually used webpack's resolve.root feature to resolve imports from the src/ folder as well. This could be a solution

[nickpresta]

Is there any chance at using Karma as the test runner? It is fairly agnostic to assertion libraries, has a rich plugin ecosystem for everything, supports console (via PhantomJS, SlimerJS, jsdom) and browser testing. The major downside is not way for test splitting/parallelism/concurrency.

My concern would be that providing no support for browser testing would make this less useful for the majority of application (but perhaps less so for library authors). Not everything can, or should, be shallowlyy rendered or run in jsdom.

[lacker]

I am not sure that we should ship a test runner. We need to avoid "boilerplate disease" where we add more dependencies because a feature is useful, but many people don't like those dependencies and would prefer to customize it, so in the end most people dislike that boilerplate. Maybe it would be possible to have some addon system like npm run add karma or npm run add tape and so on. It does need to be possible to run tests without ejecting though, somehow.

[kripod]

Ava is as simple as tape, with an unopinionated design that could accommodate any kind of project. In combination with nyc, code coverage could also be measured without hassle.

[bravely]

This ends up being a really large decision, considering that whatever test framework is chosen, we end up essentially anointing it as official and as a result encourage its use in the community. While there are certainly cases of framework tooling that was eventually dropped in favor of something better(minitest => rspec), that is the exception, and not the rule.

So in my eyes, a testing framework that is forward looking and encourages the kinds of design principles that the core team sees as the way forward for React should be respected.

I believe this points towards something like Ava, because of how it encourages a more functional style(which in turn is easier to keep concurrent), much like how Redux is encouraging such a paradigm.

[gaearon]

We need to avoid "boilerplate disease" where we add more dependencies because a feature is useful, but many people don't like those dependencies and would prefer to customize it, so in the end most people dislike that boilerplate.

Agreed. I think as long as it’s unobtrusive it could work though. I’m really interested in how well new snapshot testing works in Jest btw. It could plausibly be a good default for React apps.

[vutran]

I am not sure that we should ship a test runner. We need to avoid "boilerplate disease" where we add more dependencies because a feature is useful, but many people don't like those dependencies and would prefer to customize it, so in the end most people dislike that boilerplate. Maybe it would be possible to have some addon system like npm run add karma or npm run add tape and so on. It does need to be possible to run tests without ejecting though, somehow.

Perhaps extending the create-react-app command with custom flags like so:

$ create-react-app --with-tests

This allows the tool to still retain its simplicity. Some beginners may not be writing any tests yet so it may be best to avoid having them learning a testing framework as well.

[timjacobi]

I'd like to make an argument against including any test framework in this. Everyone has an opinion on which test framework works best and most of them are valid. I feel like the scope of this package is perfect at the moment. Yes it is opinionated to some extend but not in a too invasive fashion.

• react Most obvious one. If you don't want to use React you probably don't need this package
• babel It works in the background and doesn't dictate how I write my code. Also configuration is done for me.
• webpack Same as Babel

If we include a test framework we dictate how the user should write their tests. We might then go ahead and let the user choose which test framework they want to use by adding a selection dialog in the CLI but at that point the user might as well use a Yeoman generator. Also adding your favorite test runner really doesn't take long.

I think we should focus on optimizing what is currently there so we can guarantee highest possible quality and best user experience.

In short: Let's leave this package un-opinionated around testing and let the user decide.

[mxstbr]

I'm kinda :+1: for what @timjacobi said, adding Mocha to a generated app is a single command and an entry in a JSON file:

npm install mocha

{
  "scripts": "mocha *.test.js"
}

[gaearon]

It's not so simple, you'd have to configure it to be compatible with our set of Babel plugins.

[Place1]

Originally I was excited to include testing support but @timjacobi makes a very compelling argument that has changed my opinion. I think it's almost impossible to include testing without forcing the user into a specific set of tools/config which they may not want. Additionally I agree that the primary use case for this is for new users who aren't likely to be experimenting with testing (as much as we all wish they would! :P) simply because the react learning curve is the first thing people are looking to solve.

I think the 'eject' feature is where this project really shines for users who want to do things like set up automated testing.

[just-boris]

If we include a test framework we dictate how the user should write their tests.

Actually, most popular testing framworks are following the describe/it convention. There are some differences in implementation (Jasmine, for example has built-in assertions), but the code structure will be the same and it will be possible to mirgate from one to another easily.

[ForbesLindesay]

The difficulty with expecting users to do this themselves is that it requires running babel and (to some extent) webpack. We make that even harder by hiding the babel and webpack configs from the user.

I think the least opinionated thing we can do here is provide a way to run a JavaScript file (or set of JavaScript files) in node.js with webpack + babel done for the user. This would mean providing something like react-scripts run *.test.js. It would leave the user free to choose a library for structuring tests and assertions.

[mxstbr]

That sounds very sensible @ForbesLindesay!

[wdhorton]

@timjacobi I disagree with your reasoning about how opinionated this project already is. Despite you saying that it shouldn't tell you how to write your code, it already includes ESLint, a tool whose only purpose is to tell you how to write your code. And babel also dictates how you can write your code, based on the specific plugins that have been chosen. Similar to the Mocha/Ava/Karma/Jest debate we're having here, there are bundlers besides Webpack that are commonly used by JS developers, like Browserify and Rollup, and yet @gaearon chose to use one of those over all the others. (For the record, I love Webpack and think it was the right choice.) So I don't really see how choosing a test runner to include would go beyond the bounds of what has already been established by this project.

I think the least opinionated thing we can do here is provide a way to run a JavaScript file (or set of
JavaScript files) in node.js with webpack + babel done for the user. This would mean providing
something like react-scripts run *.test.js. It would leave the user free to choose a library for
structuring tests and assertions.

@ForbesLindesay This might be a good idea, but it doesn't really solve the problem for anybody who wants to use a test runner. For example, mocha runs with its own CLI, so even if you had this set up to handle the webpack + babel stuff, you wouldn't be able to do react-scripts run mocha *.test.js. I think the same is true of karma and ava.

[timjacobi]

@wdhorton I never said it doesn't dictate you how to write your code. I said it wasn't too invasive in doing so. Perhaps I should have been more clear about the framework aspect. If you introduce a test framework you dictate the user to use it. Babel doesn't dictate you to use it since you can still write ES5 code an it will happily transpile it for you even though this arguably turns into a no-op. Same for Webpack. No one forces you to run npm build at anytime. I actually think that it would be great to have tooling like enzyme be part of this but we should try to find a more generic way to do so.

[vutran]

Despite you saying that it shouldn't tell you how to write your code, it already includes ESLint, a tool whose only purpose is to tell you how to write your code. And babel also dictates how you can write your code, based on the specific plugins that have been chosen.

Having webpack, and eslint is fine in my opinion. The user is not really forced to learning anything extra on that end. There's no additional API/frameworks they need to learn. It's just JavaScript in the end and the only framework they require to learn is React.

I'm going to have to agree with @timjacobi on not including any testing framework. Adding these testing framework can be beneficial but that requires a new user to learn those frameworks on top of learning React itself. We should be asking who the target audience is? Should we assume users to already know a testing framework? If so, should we assume they know x framework or y framework?

[gaearon]

Adding these testing framework can be beneficial but that requires a new user to learn those frameworks on top of learning React itself.

Doesn’t “require”, you can keep writing your app without any tests. However most apps need at least _some_ tests pretty soon.

Should we assume users to already know a testing framework? If so, should we assume they know x framework or y framework?

Knowledge of testing frameworks is super portable because most just use describe / it for declaring tests. So in many cases it’s possible to use e.g. Mocha, and then easily move to Jest, or vice versa.

[ariporad]

I personally vote for a test runner (probably mocha) to be included, but, at least as an interim step, we should make sure that the webpack config is enzyme compatible.

[AdamBrodzinski]

I think what's important here is that we _decrease the friction_ for creating tests as much as possible.

Elixir does this with all new projects and having that passing test ready to go makes me write 10x more tests compared to JS projects because frankly it's chore to get everything setup.

[gaearon]

at least as an interim step, we should make sure that the webpack config is enzyme compatible.

Please file an issue if there’s something about that config that breaks enzyme.

[wdhorton]

I added a PR (#216) to show a basic test setup. I used Mocha, but the most important parts are framework-agnostic, and we should be able to easily swap in another framework if one pulls ahead significantly.

[SpencerCDixon]

I think what's important here is that we decrease the friction for creating tests as much as possible.

I've gotten a lot of feedback from new React developers that setting up a test environment is a PITA. A production ready test environment that properly deals with CSS/images, mocking/stubbing/spys, shallow rendering, etc. is not a trivial thing to set up and would probably be very beneficial to abstract away.

Elixir does this with all new projects and having that passing test ready to go makes me write 10x more tests compared to JS projects because frankly it's chore to get everything setup.

rails new does this as well. I think an important aspect is to allow users to disable the 'default' test config so they can use their own if they're opinionated.

For example, often in rails projects I pass the -T flag to NOT use minitest and then set up RSpec myself b/c that is the framework I prefer.

Maybe a good compromise is to include the simplest testing setup possible that will be compatible with this project and then allow users to _disable_ instead of _enable_ the testing setup if they decide they want to use a different framework or something more complex. That way people are automatically opted-in to doing something they should be doing (think pit of success) and have to make an active choice to opt out of having a testing environment setup included in their new projects.

[ariporad]

@gaearon: @wdhorton already fixed the bug in #216 here.

[mpj]

Jest aaaaalmost works out of the box with create-react-app if you install babel-jest, the only problem is that create-react-app doesn't put a .babelrc file in the root.

[lacker]

@mpj Does babel-jest handle non-js imports like import "foo.css" and import "foo.png"?

I am working on the react-scripts run as described by @ForbesLindesay & discussed in https://github.com/facebookincubator/create-react-app/issues/218 . That should be useful for some quick-and-dirty testing and maybe some lighter-weight test frameworks can actually just be invoked through it.

[cpojer]

A few people asked me to contribute a pull request with an initial Jest integration: https://github.com/facebookincubator/create-react-app/pull/250

If you'd like to find out more about snapshot testing, please read our new blog post and documentation.

[Marlena]

I'm a big fan of TDD and testing which means I always want it to be easy for people to include tests or to learn more about writing tests.

If a BDD framework such as Mocha or Jasmine were included, it would be nice, however, it is kind of an extra.

I'm thinking back to the first React app I wrote at a Railsbridge (yes, it was Railsbridge, they do js, front-end workshops too). I was wrapping my head around building components and using a store instead of using templates and models. I didn't write any tests. Maybe it's not as important if this is a setup for an initial Hello Components! react app.

Also, there are a lot of opinions that come with testing. The one I see in this thread is folder structure. If I'm encouraging someone to do more testing, they need to be able to arrange their tests in a way that suits them.

My team downloaded create-react-app and found it pretty easy to add tests in a separate test folder. We had to include babel but it was nbd.

I'm with @timjacobi that it seems early to add in testing, but it's not too early for a pointer to anyone who wants to do a test setup. Also I hope people _are_ encouraged to set up tests with this project and it makes me so happy to see this thread!

[giorgian]

I'll add my two cents. Contrary to what I expected, it took me just a few minutes to setup mocha, enzyme, expect.js (mostly spent googling, as I had never used any of them).

The inconvenience is that I have two terminal tabs, one for create-react-app, the other for mocha watch. right now, for me this is already good enough; I'm willing to trade a little inconvenience for the ability to ditch, say, mocha for Jasmine whenever I see fit, or to use a single test dir instead of many __tests__ dirs—tastes, as they say.

I'm no beginner, though, and a beginner cannot be expected to know how to hack a testing solution on an existing project, or whether to use Mocha or Jasmine or Karma or whatnot.

So, my proposal:

• have exactly one full testing solution nicely integrated (opt out) in create-react-app, with sample tests, pointers to docs, etc.—I have personally no real preferences here, I'm used to Jasmine but I'll use whatever works, they are all very similar after all.
• leave room for the integration of alternative solutions—No idea how hard it would be!
• have pointers to a few how to docs for other solutions.

This should help beginners to start using React with proper tests without hassle, and should also allow everyone to opt for a different testing setup with reasonable effort.

[cpojer]

250 was merged and should fulfill most of the wishes people expressed in this thread. If you'd like additional features, please feel free to create new issues and cc me on them :)

[gaearon]

0.3.0 is out with support for testing.
Read the usage guide and the migration instructions.

💜