Tools: If setup() throws, the test still passes with great success
Consider this very simple test:
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<script src="web-component-tester/browser.js"></script>
</head>
<body>
<script>
suite('the-suite', () => {
setup(() => {
console.log('setup running');
//throw 'oops';
});
test('test1', () => {
console.log('test1 running');
});
});
</script>
</body>
</html>
Running this gives:
joco@joco:~/test$ wct test.html
Installing and starting Selenium server for local browsers
Selenium server running on port 58264
Web server running on port 2000 and serving from /usr/local/google/home/joco/test
chrome 51 Beginning tests via http://localhost:2000/components/test/generated-index.html?cli_browser_id=0
chrome 51 Tests passed
Test run ended with great success
chrome 51 (1/0/0)
Now if you uncomment that throw sentence, it gives this:
joco@joco:~/test$ wct test.html
Installing and starting Selenium server for local browsers
Selenium server running on port 58461
Web server running on port 2000 and serving from /usr/local/google/home/joco/test
chrome 51 Beginning tests via http://localhost:2000/components/test/generated-index.html?cli_browser_id=0
chrome 51 Tests passed
Test run ended with great success
chrome 51 ()
Test success/failure numbers are not shown anymore, but the final result is still success, which is very very bad. It should fail both because the suite is apparently empty and also because something threw an error.
fejesjoco
All 11 comments
I came across this too and it caused much head scratching...
dgl
on 18 Jul 2016
Me too! Any ideas??
ppalladino
on 2 Aug 2016
My example:
suite('some-test', function() {
let item;
setup(function(done){
item = 'is defined';
// Never call done(), should timeout then throw
// done();
});
test('item defined', function() {
assert.isDefined(item, 'item defined');
});
});
outputs:
$ wct
...
$ firefox 46 Tests passed
$ Test run ended with great success
$ firefox 46 ()
I discovered one way this could be addressed. Take a look at the event listeners set in runner/clireporter.js.
In the case where the setup function fails for a test suite, that emitter object will emit events like so: sub-suite-start, test-start, sub-suite-end, sub-suite-end...
So it won't emit test-end events in that suite at all. It will just keep emitting sub-suite-end until it's done with that suite.
So a solution is to expect that for a test to be passing, a test-start event must be immediately followed by a test-end event. That would probably mean tracking state in clireporter.js unless somebody has a better idea.
Pilatch
on 2 Aug 2016
Here's some debugging output from a simple test with only one test suite, and three test cases therein.
Setup working:
event name sub-suite-start
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 undefined
arg3 { status: 'running', passing: 0, pending: 0, failing: 0 }
event name test-start
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 { test:
[ 'app/elements/my-element/test/my-element.html',
'styling',
'sets the icon background color' ] }
arg3 { status: 'running', passing: 0, pending: 0, failing: 0 }
event name test-end
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 { state: 'passing',
test:
[ 'app/elements/my-element/test/my-element.html',
'styling',
'sets the icon background color' ],
duration: 1 }
arg3 { status: 'running', passing: 1, pending: 0, failing: 0 }
event name test-start
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 { test:
[ 'app/elements/my-element/test/my-element.html',
'styling',
'does not throw if no owner is passed to _setStyle' ] }
arg3 { status: 'running', passing: 1, pending: 0, failing: 0 }
event name test-end
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 { state: 'passing',
test:
[ 'app/elements/my-element/test/my-element.html',
'styling',
'does not throw if no owner is passed to _setStyle' ],
duration: 0 }
arg3 { status: 'running', passing: 2, pending: 0, failing: 0 }
event name test-start
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 { test:
[ 'app/elements/my-element/test/my-element.html',
'styling',
'does not change the icon background color if null owner is set' ] }
arg3 { status: 'running', passing: 2, pending: 0, failing: 0 }
event name test-end
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 { state: 'passing',
test:
[ 'app/elements/my-element/test/my-element.html',
'styling',
'does not change the icon background color if null owner is set' ],
duration: 1 }
arg3 { status: 'running', passing: 3, pending: 0, failing: 0 }
event name sub-suite-end
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 {}
arg3 { status: 'running', passing: 3, pending: 0, failing: 0 }
event name sub-suite-end
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54927 },
id: 0 }
arg2 {}
arg3 { status: 'running', passing: 3, pending: 0, failing: 0 }
Setup failing:
event name sub-suite-start
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54788 },
id: 0 }
arg2 undefined
arg3 { status: 'running', passing: 0, pending: 0, failing: 0 }
event name test-start
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54788 },
id: 0 }
arg2 { test:
[ 'app/elements/my-element/test/my-element.html',
'styling',
'sets the icon background color' ] }
arg3 { status: 'running', passing: 0, pending: 0, failing: 0 }
event name sub-suite-end
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54788 },
id: 0 }
arg2 {}
arg3 { status: 'running', passing: 0, pending: 0, failing: 0 }
event name sub-suite-end
arg1 { browserName: 'firefox',
version: '46',
firefox_binary: '/Applications/Firefox.app/Contents/MacOS/firefox',
url: { hostname: '127.0.0.1', port: 54788 },
id: 0 }
arg2 {}
arg3 { status: 'running', passing: 0, pending: 0, failing: 0 }
My crude diff of runner/clireporter.js to show how we can detect a setup failure. Yes, I know the source is TypeScript. Gimmie a break already. Sheesh.
All it does right now is error log when it detects a setup failure. It does not affect the "great success" outcome yet.
--- clireporter.original.js 2016-08-02 16:40:19.000000000 -0400
+++ clireporter.js 2016-08-02 16:47:31.000000000 -0400
@@ -43,6 +43,7 @@
const STATUS_PAD = 38;
class CliReporter {
constructor(emitter, stream, options) {
+ var testEndEventExpectedNext = false;
this.prettyBrowsers = {};
this.browserStats = {};
this.emitter = emitter;
@@ -70,7 +71,11 @@
this.log(browser, 'Beginning tests via', chalk.magenta(data.url));
this.updateStatus();
});
+ emitter.on('test-start', (browser, data, stats) => {
+ testEndEventExpectedNext = true;
+ });
emitter.on('test-end', (browser, data, stats) => {
+ testEndEventExpectedNext = false;
this.browserStats[browser.id] = stats;
if (data.state === 'failing') {
this.writeTestError(browser, data);
@@ -80,6 +85,11 @@
}
this.updateStatus();
});
+ emitter.on('sub-suite-end', (browser, error, stats) => {
+ if (testEndEventExpectedNext) {
+ this.log(chalk.red, browser, 'A test setup failed!');
+ }
+ });
emitter.on('browser-end', (browser, error, stats) => {
this.browserStats[browser.id] = stats;
if (error) {
Note there is a working fix in #351, although I haven't had a chance to update that since the tests got moved to Typescript, but the actual fix should apply cleanly still.
dgl
on 3 Aug 2016
Thanks man. We may rely on that specific version of your repo for a bit until it gets merged in.
Pilatch
on 3 Aug 2016
Any ETA on when this will be fixed? Blocker issue when your testing framework passes tests that should fail.
chiefcll
on 8 Aug 2016
@rictic WDYT?
tjsavage
on 4 Oct 2016
This was fixed by https://github.com/Polymer/web-component-tester/pull/415
fejesjoco
on 18 Nov 2016
Related issues
pmaudsley
路
3Comments
idoshamun
路
3Comments
rwatts3
路
3Comments
web-padawan
路
4Comments
MaKleSoft
路
3Comments
Most helpful comment
Any ETA on when this will be fixed? Blocker issue when your testing framework passes tests that should fail.