Ember-cli: utilize ember server during testem run to get HTTP mocks

[stefanpenner]

this should be possible since: https://github.com/airportyh/testem/commit/f42ce8c93ff7fec0a3e1f3c4fd3847150547a058

[rwjblue]

Related to or duplicate of https://github.com/stefanpenner/ember-cli/pull/1423.

[danfinlay]

This would be a valuable step towards making testing feel right in ember-cli.

Right now if I want to take advantage of airportyh/testem@f42ce8c in combination with the /server/index.js mock server, I would have to still write my own script to launch an express server and pass it to the server/index module.

This sounds like a short term hack to me, but one I'm willing to do in the meanwhile.. would there be anything else I'd need to do to the express server? I guess give it the port declared in the package.json?

Also, this would be a good issue to discuss the ideal api for declaring "mocks during tests" in. Since we're already declaring proxyURL and proxyPath in the package.json, maybe we should just add a proxyInTesting boolean key, to tell ember-cli whether it should launch the mock server during testing or not.

[hhff]

:+1:

I could use this! I feel like this is something that should be enabled on an environment basis

[hhff]

ENV.mockApiEnabled = true

or something

[seawatts]

@hhff Any idea when this will get merged to master?

[hhff]

@seawatts haven't had a spare second to look over this. My approach need a complete rejigger as we now have testem middleware being injected (that's where we should enable the mock server).

Doubt I'll be able to look into this until the new year

[taras]

I'm going to take a crack at it. I'll give an update on this when I have something to show.

[taras]

PR #2969 adds http mocks to testem by making _./server_ testem's middleware.

I marked it as WIP because I don't know if that functionality should be enabled/disabled by default.

Thoughts?

[hhff]

@taras - this is rad!!

Personally - I think it should be enabled/disabled via environment setting. Default currently is that it's enabled in development by default if ./server exists - I think that should be the same for test at least as new developers to the ember-cli will be confused if ember test --server behaves differently to development :heart:

[seawatts]

I agree with @hhff

[taras]

Default currently is that it's enabled in development by default if ./server exists

This is how the implementation works currently.

enabled/disabled via environment setting

Since it's enabled by _./server_ directory, then maybe we should only have an option to explicitly disable this functionality via environment settings.

var app = new EmberApp({
   httpMock: {
     disableInTests: false
   }
});

What do you think about the above?

[hhff]

@taras looks good to me!!

[seawatts]

@taras I also like it. Does httpMock have any other properties that can be set on it?

[taras]

Not at the moment, but as I'm looking into, I'm starting to think that there is an opportunity to make this functionality more generic. In it's current state it's not very DRY or flexible.

It's not DRY because I copied server loading code from ExpressServer#processAppMiddlewares into TestTask#addonMiddlewares.

It's not flexible because server root is hardcoded to './server' in two separate files.

I want to refactor this server code into a single file and make it possible to configure the server root via environment variables.

With refactoring, it would look like this:

var app = new EmberApp({
  server: {
    root: './some-dir', // default to './server',
    inTesting: false, // default to true
    inDevelopment: false // default to true
  }
});

This could become an entry point for further server functionality.

[seawatts]

Omg that is amazing! I've been trying to figure out how to structure my code to also run my own express server within the same project folder. Which ended up being named something like expressServer or something stupid like that. If anything I think the default should be mock-server

[taras]

mock-server === good idea.

I'll get cracking on this.

[seawatts]

thanks @taras!

[cibernox]

Isn't in general considered a better practice to use pretender rather than having a real server, even if its a server that returns fixtures?

Maybe I'me being too purist, I don't know.

[taras]

@cibernox that was my initial reaction as well, but I think it doesn't hurt to have consistency here.

[go1dfish]

@taras I like this approach and will be watching this PR with great anticipation.

I agree that something like pretender is the better way to go, but there are many cases where it still makes sense to have express based mocks in more complex cases.

For me, I have a mock system that currently runs as express middleware in ember server mocks and works great for in browser testing, but isn't usable for CI.

https://github.com/trek/pretender/pull/48 will make pretender much more capable.

[alexlafroscia]

If it's useful to anyone that's working with CLI version 0.1.7 but wants to try out @taras's ./server middleware, I've got a branch that applies the same changes but from the newer version

https://github.com/alexlafroscia/ember-cli/tree/testem-http-mock

[taras]

I just pushed an updated version of the PR #2969.

It includes the following changes:

• Custom server root can be specified with ember serve --server-root=./some/directory and ember test --server --server-root=./some/directory
• Refactor of _server_ command which removes mock server directory watcher
• Refactor of Express Server which removes mock server directory watcher
• Mock Server code is moved into an addon in /lib/tasks/server/middleware/mock-server
• Watcher is now instantiated by the Mock Server addon
• [BREAKING] addon.testemMiddleware is now called same way as addon.serverMiddleware is called

I want to get some early feedback, while I fix the tests.

[taras]

I need some advice on how deeply to go with this integration. I realized that the Server Watcher actually applies to any configuration that happens in server root directory, which means Proxy & Mocks. Also, it would also be useful on ember test --server.

Making server watcher work on ember test --server is more complicated.

• How many people actually ember test --server?
• Should I just ignore ember test --server?

[taras]

@stefanpenner @rwjblue what do you guys think about ^^?

[seawatts]

I use ember test --server almost exclusively
On Jan 19, 2015 6:59 AM, "Taras Mankovski" [email protected] wrote:

@stefanpenner https://github.com/stefanpenner @rwjblue
https://github.com/rwjblue what do you guys think about ^^?

—
Reply to this email directly or view it on GitHub
https://github.com/ember-cli/ember-cli/issues/1763#issuecomment-70506332
.

[ulisesrmzroche]

The name 'server' is too generic for one that is trying to tackle with
mocks and proxies. Does this mean that the express server is not exposed
for development anymore?
ᐧ

On Mon, Jan 19, 2015 at 9:50 AM, Chris Watts [email protected]
wrote:

I use ember test --server almost exclusively
On Jan 19, 2015 6:59 AM, "Taras Mankovski" [email protected]
wrote:

@stefanpenner https://github.com/stefanpenner @rwjblue
https://github.com/rwjblue what do you guys think about ^^?

—
Reply to this email directly or view it on GitHub
<
https://github.com/ember-cli/ember-cli/issues/1763#issuecomment-70506332>
.

—
Reply to this email directly or view it on GitHub
https://github.com/ember-cli/ember-cli/issues/1763#issuecomment-70515086
.

[taras]

@ulisesrmzroche The name 'server' is too generic for one that is trying to tackle with
mocks and proxies. Does this mean that the express server is not exposed
for development anymore?

I'm sorry, I don't understand your questions. Can you please ask another way?

[backspace]

I also almost only use ember test --server.

[ulisesrmzroche]

Yeah, so in the past, what is now 'ember g server' used to be the api-stub.
Basically, the same idea behind renaming it to 'mock-server'. The server
directory exposed the built-in express server to allow for custom
development. Does this still work right now?
ᐧ

On Mon, Jan 19, 2015 at 10:43 AM, Buck Doyle [email protected]
wrote:

I also almost only use ember test --server.

—
Reply to this email directly or view it on GitHub
https://github.com/ember-cli/ember-cli/issues/1763#issuecomment-70523902
.

[taras]

@ulisesrmzroche yes, except _./server/index.js_ is not just mock-server, it's also proxy-server configuration. I'm trying to understand wether I should refactor in such a way that would allow Testem Express Server to also benefit of the Server Watcher which reconfigures the Express server when a file _./server_ directory changes.

[ulisesrmzroche]

Ah, sweet. Got it!
ᐧ

On Mon, Jan 19, 2015 at 11:21 AM, Taras Mankovski [email protected]
wrote:

@ulisesrmzroche https://github.com/ulisesrmzroche yes, except
_./server/index.js_ is not just mock-server, it's also proxy-server
configuration. I'm trying to understand wether I should refactor in such a
way that would allow Testem Express Server to also benefit of the Server
Watcher which reconfigures the Express server when a file _./server_
directory changes.

—
Reply to this email directly or view it on GitHub
https://github.com/ember-cli/ember-cli/issues/1763#issuecomment-70529581
.

[taras]

@abuiles I'm leaning towards making server watcher work with ember test --server. What do you think?

[taras]

I wanted to give everyone an update on this issue. I spent some time speaking to @stefanpenner about this issue. We can to an agreement that mocks are useful for development but are an anti-pattern in testing. We decided that instead of spending time making mocks working in ember test --server we are going to show how to do proper mocking in testing using Pretender.

Stefan showed me an example of mocking with Pretender and the test code is much more readable than having mock server with responses hidden away in the ./server configuration. I'm going to spend some time this weekend looking at what can be scavenged from related PR. Before closing this issue, I'll write up an example of how to use Pretender to mock a server response.

I hope this is not too much of a disappointment for those who've been looking forward to this issue being resolved. I believe that if you use Pretender in tests, you will have better and faster tests.

[ulisesrmzroche]

Nice, I totally agree, and I'd like to add it also gets in the way of the express-server, which is usually used as a static file server or an API.

[djmitchella]

A counterpoint -- when deployed, our app is served up as a subfolder of the main site, and needs to access files outside the main Ember folder. When deployed this is fine, but when developing using "ember serve" we can't get at those files because they're above the root of where "ember serve" is running. I'm using http-mocks to let us retrieve these files, but I can't run tests on this functionality with "ember test" because the mock server isn't running there. What's the suggested way to make this work in CI builds? Pretender doesn't seem to solve this problem, because we still need to get the data in for our tests. Previously I used a symbolic link to redirect things, but that was not self-contained in the same way that http-mock is -- but not being able to use http-mock in CI builds is very awkward.

[alexlafroscia]

@taras is the code that @stefanpenner showed you available somewhere? I've been working on getting Pretender set up with the Ember CLI tests this past week and am curious how different things are

[stefanpenner]

my codez: https://gist.github.com/stefanpenner/1a3b3467059fc9b2f7be#file-asdf-test-js-L40-L72

[bjornstar]

Please finish implementing this, there's no reason why you can't support mocks in test, _especially_ since you support them in development.

Pretender may be perfect for you, but other people need something a little more robust. http-mocks and pretender are not mutually exclusive, having one does not prevent the other from existing.

[stefanpenner]

there's no reason why you can't support mocks in test,

Time, and pretender works exceptionally well so when weighing this against other work to be done it unfortunately has a much lower priority. If you are motivated I would encourage you to explore @taras's WIP and maybe you can help speed things up.

If we had infinite time, this would already be taken care of :)

[akhomchenko]

Have someone launched pretender with ember-mocha? It restarts instantly. Thx.

[rwjblue]

Have someone launched pretender with ember-mocha? It restarts instantly. Thx.

@gagoman - This seems completely unrelated to the issue being discussed here. Please open a new issue (most likely in ember-cli-mocha) with details (how to reproduce) and possibly a reproduction repo that we can use to diagnose to get attention on your issue.

[lessless]

@taras can you please shed light on the reason why using mocks in testing is anti-patterns?
Using both Mocks and Pretender is a code duplication.

[alexlafroscia]

@lessless I think, as a general rule, using Pretender is a better solution because you're mocking the entire request instead of mocking the response to the request. With a lot of tests, you probably don't want to actually wait for the requests to go to the Express server and back again.

[taras]

@lessless it's an anti pattern because it's fragile and doesn't scale. Pretender allows you to include your mock data in your test - which makes it obvious what response is expected. It doesn't scale because it's much slower than using Pretender. It's ok when you have 10 tests, but when you're 6 months into the project and you have 50 you're going to regret using Express mocks. Your test suite will run much slower and the only way to fix it will be to replace the Express server with Pretender.

[zyllorion]

@stefanpenner @taras The decision to migrate all mock code to Pretender does not make sense when the documentation has guided us towards using http-mocks. Please either deprecate http-mocks or allow use of them in a full CI environment. I understand the speed advantage but ember-cli is not guiding people towards this and Pretender is a third party dependency not included with core ember-cli. Documentation should be added asap. Even if mocks are bad at scaling, they are currently a good way to isolate server side logic in the ember file hierarchy.

The best, and one of very few, videos/tutorials for TDD / testing in general is https://www.youtube.com/watch?v=2b1vcg_XSR8 by @toranb (thanks) but this has nothing useful on setting up pretender / http-mocks.

Can someone who has some experience create a testing guide that covers:

1) Initial configuration of testem/phantomjs/chrome/ff etc.
2) Writing of tests with pretender and qunit.
3) Preparation to integrate with Travis or Bamboo.

Can someone who knows pretender well create ember-cli specific documentation for using it?

I think it is critical for Ember's credibility that tests are given more focus.

Thanks

[stefanpenner]

Please either deprecate http-mocks or allow use of them in a full CI environment. I understand the speed advantage but ember-cli is not guiding people towards this and Pretender is a third party dependency not included with core ember-cli. Documentation should be added asap. Even if mocks are bad at scaling, they are currently a good way to isolate server side logic in the ember file hierarchy.

We welcome PR's to improve the mocks, and the docs. Unfortunately this is somewhat low priority right now, and will likely linger for some time without its own champion.

[pedrokost]

I designed my app running tests in the browser, using the http-mock as it is described on the website.

Now, I wanted to configure it to run in CI, and discovered this thread, as well as many related ones, which started a long time ago. It is annoying because the ember-cli.com website says in extra large letters: "Note: Mocks are just for development and testing.".

@taras if you were able to configure Pretender, I would be very happy if you shared how you did it.

[lessless]

@pedrokost have you tried https://github.com/samselikoff/ember-cli-mirage ?

[pedrokost]

Hi @lessless, this project looks very promising! I won't try this on the current project yet, because the project is wrapping up, and it makes no sense to include an addon that is under heavy development.
However, I will use it in the next Ember project.

[seawatts]

It looks like ember-cli-mirage handles this problem exactly.
On Apr 10, 2015 1:56 AM, "Pedro Kostelec" [email protected] wrote:

Hi @lessless https://github.com/lessless, this project looks very
promising! I won't try this on the current project yet, because the project
is wrapping up, and it makes no sense to include an addon that is under
heavy development.
However, I will use it in the next Ember project.

—
Reply to this email directly or view it on GitHub
https://github.com/ember-cli/ember-cli/issues/1763#issuecomment-91483433
.

[abuiles]

Closing in favor of https://github.com/samselikoff/ember-cli-mirage

[danfinlay]

It seems clear to me that ember-cli-mirage is taking a very specific approach to this general problem, and does not close this issue.

The issue is that we should be able to use the normal http-mocks in cli testing. Mirage has an alternative way to set up both.

Please reopen.

[taras]

@flyswatter can you describe what use case http-mocks solves that ember-cli-mirage can not?

[danfinlay]

Right now http-mocks are a dead-simple method of declaring a mock server, supported with their own ember-cli-generator. Since it's been supported by a generator for a while, it's fair to believe that lots of people already have mock servers ready to go in this format.

This issue is saying "We should be able to use that same configuration in a cli-testing environment".

I'm not saying we couldn't all rewrite all our mocks in a mirage format. I'm saying it should be simple, given an already working mock server, to serve it during testem runs using testem's proxy feature.

[taras]

The difficulty with this issue is that it would require a non trivial refactor of the code base to enable an _unrecommended_ path.

Practically speaking, EmberCLI Mirage is better than HTTP Mocks in every way. Mocking with EmberCLI Mirage is easier, it allows the test suite to run faster and provides a robust fixture and mocks implementation.

For these reasons, there is very little interest in implementing something that some users will prefer but is from many prospectives an inferior solution.

[danfinlay]

If this path is in fact _unrecommended_, then I would suggest removing mocks and fixtures from the user guide, this seems mis-guiding.

I'm curious why this requires a non-trivial refactor of the code base. To me it seems like it just needs to call the express mock server before running testem, and close it after. Is there something I'm missing there?

I just spent a week building http-mocks for a new large project, so forgive my reluctance to rewrite the mocks again.

[taras]

If this path is in fact unrecommended, then I would suggest removing mocks and fixtures from the user guide, this seems mis-guiding.

You're right. This needs to be addressed, ASAP.

I'm curious why this requires a non-trivial refactor of the code base. To me it seems like it just needs to call the express mock server before running testem, and close it after. Is there something I'm missing there?

Last time I looked at this was about 6 months ago, so I might be mistaking, but one of the features that's built into the HTTP mocking is restarting the express server when mocks change. To make this work reliably in all of the different modes that the tests can run would require adding a cross platform process manager which we don't have at the moment. This improvement needs be made either way but it hasn't been a huge priority.

I just spent a week building http-mocks for a new large project, so forgive my reluctance to rewrite the mocks again.

I'm very sorry that our documentation was misleading. I hope in-spite of the wasted time, you'll find EmberCLI Mirage migration to be worthwhile.

@stefanpenner @rwjblue we should replace mocks and fixtures guides in favour of ember-cli-mirage if you agree, then I'll put together a PR.

[ghedamat]

@flyswatter FWIW for now, you can already configure testem to work against your mock api
https://github.com/airportyh/testem#api-proxy
it's just up to you to run the mock server while testem is running.
that said, as @taras explained it's probably not the recommended path for a bunch of reasons (aside form the current limitations of ember-cli):

as an example even mirage suggests to use factories instead of fixtures (that are basically mocks) during your tests because this way it's way easier to ensure that you have a determined "state of the world" between your tests, mocks instead always return the same data regardless of the test you're running and that can easily leave you in inconsistent situations...

[odarbelaeze]

Holy moly iI spent my whole afternoon after this issue, I'll check the guide to see if the http-moks section is being replaced, otherwise I'll contribute that. Thanks @taras for clarifying, you should comment as well in your WIP pull request addressing this issue.

[sudrao]

While I am using ember-cli-mirage for testing, it appears that ember-cli-blanket for test coverage needs the mock server (express js) running in test mode. Test coverage reports are written via http request to the mock/proxy server on the back end. How do I enable express js in test for coverage? @taras @stefanpenner

[cibernox]

@sudrao I use ember-cli-blanket and I don't have the express server running in testing, neither I've seen any reference to express in the addon. Are you sure that is the problem?

[sudrao]

@cibernox From the ember-cli-blanket README: "When tests are complete an HTTP POST request is sent to the ember-cli express server to initiate the report writing process."

[cibernox]

@sudrao I see. Probably I don't needed it because I use it with ember s, but I don't do coverage when I run ember test (and I should)

[jasonmit]

I worked around this with the following now that there is a testemMiddleware hook exposed to addons:

ember g in-repo-addon testem-middleware

// lib/testem-middleware/index.js
/*jshint node:true*/

var path = require('path');

module.exports = {
  name: 'testem-middleware',

  testemMiddleware: function(app) {
    // invoke the server/index.js that the http-mock blueprint creates
    require(path.join(this.project.root, 'server'))(app);
  }
};

Update: published a simple addon to handle this
https://github.com/jasonmit/ember-cli-testem-http-mocks

[sudrao]

@jasonmit Post to server did work in test. There was a bug in Pretender that I fixed to make coverage work.

[trek]

@sudrao is there a link to the matching Pretender issue/pr? Thanks!