Testcafe: Testcafe tests run perfectly fine locally but fail on CI (Gitlab CI)

[ananddharne]

Are you requesting a feature or reporting a bug?

Reporting a bug

What is the current behavior?

I am trying to run tests on CI. They fail giving a weird script error. They run fine Tried using the stable(alpha) as well as the latest version of testcafe. Using this Docker to run on CI

Locally:

testcafe chrome:headless __tests__/Sequences1_testcafe.js
  Using locally installed version of TestCafe.
  Running tests in:
  - HeadlessChrome 0.0.0 / Linux 0.0.0
 Getting Started
 ✓ Sequences1
 1 passed (54s)

In docker, the tests pass too.

testcafe chrome:headless __tests__/Sequences1_testcafe.js
using locally installed version of TestCafe.
 Running tests in:
 - HeadlessChrome 0.0.0 / Linux 0.0.0
 Getting Started
 ✓ Sequences1
 1 passed (58s)
`

On gitlab CI with the same image up there I get this
CI error On the CI , it runs on chrome 63.0.3239 and other times it runs on 0.0.0 ( what is this?) what kind of chrome version is this?

What is the expected behavior?

The tests should run on CI given they run locally as well as the docker image I have been using to run on CI

How would you reproduce the current behavior (if this is a bug)?

Cannot reporoduce since this is a company private repo and a private server

Specify your

• operating system: Linux
• testcafe version: 0.18.0-alpha2
• node.js version: 6.0.0

Any help would be greatly appreciated! Thank you

[AlexanderMoskovkin]

Hi @ananddharne,

it runs on chrome 63.0.3239 and other times it runs on 0.0.0

Before TestCafe detected headless chrome version as 0.0.0.0. Now it works properly (with the latest Chrome and TestCafe versions). I'm not sure when it was exactly fixed (it could be changes in Chrome or TestCafe submodules). So it possible your issue depends on the browser and TestCafe versions.
Additionally it possible we've already fixed it in the latest TestCafe version.

Could you please try to update TestCafe to the latest version (0.18.6) and run your tests on the CI. Please let me know does it work?

[ananddharne]

@AlexanderMoskovkin I got it to work, on both the versions. However the problem seems to be something different.
My build process is as follows,

1. Run the node server in the docker
2. Run the tests

So the tests are done on the app that runs on the local server in the docker. So all my tests start with

import { Selector } from 'testcafe'

fixture `Outreach`
    .page `localhost:4000/login`

test('My first test', async t => {
  const drag = Selector('#app > div.app-container > div.main-container > div.panel.settings-panel > div > div.kanban-column-aligner > div:nth-child(2) > div:nth-child(1) > span.candidate-name-container > div:nth-child(1) > span')
  const drop = Selector('#app > div.app-container > div.main-container > div.panel.settings-panel > div > div.kanban-column-aligner > div:nth-child(1)')
  const statuschanged = Selector('#app > div.modal-collection-container > span > div > div > div > div > div > div.comments-panel-aligner > div.candidate-activity-container > div.candidate-activity-list > div:nth-child(1) > p:nth-child(4)').withText('was changed to new ')
.
.
.
await t
        // Login
        .typeText('#auth0-lock-container-1 > div > div.auth0-lock-center > form > div > div > div:nth-child(3) > span > div > div > div > div > div > div > div > div > div:nth-child(3) > div.auth0-lock-input-block.auth0-lock-input-email > div > input', '[email protected]')
        .typeText('#auth0-lock-container-1 > div > div.auth0-lock-center > form > div > div > div:nth-child(3) > span > div > div > div > div > div > div > div > div > div:nth-child(3) > div.auth0-lock-input-block.auth0-lock-input-password > div > input', 'happie123')
        .click('#auth0-lock-container-1 > div > div.auth0-lock-center > form > div > div > button > span')
        .dragToElement(drag, drop)

This didnt work and gave me an error. However changing the page to a live production site seemed to work.
This works

import { Selector } from 'testcafe'

fixture `Outreach`
    .page `https://talent-staging.gethappie.me/outreach/index.html#/login`

I have got the tests running but I would really like to do tests on the local server. Any help is appreciated!

[ananddharne]

@AlexanderMoskovkin Also when I curl the page localhost , I get a response back and the server runs fine it seems

[AndreyBelym]

Hello @ananddharne!

Unfortunately, I can't figure out the problem without access to your site. I will try to fix it ASAP if you provide me a link to your repository.

I also suggest you to try the following steps:

1. Make sure that the error doesn't happen when you browse locally served website without TestCafe.

2. If you can, install a debug TestCafe build by executing 'npm i https://github.com/AndreyBelym/testcafe/releases/download/debug-20180306-1/testcafe-0.19.1-dev20180305.tgz' or addding "testcafe": "https://github.com/AndreyBelym/testcafe/releases/download/debug-20180306-1/testcafe-0.19.1-dev20180305.tgz" into dependencies in your package.json. Start your tests and open DevTools in the browser. When the error occurs, the string 'Error message:' and a dump of two objects will be printed to the browser's console. Please, send this print to me as a screenshot. Pay attention to the filename, lineno. colno properties in the first object, and to the stack property in the second. I hope you will be able to locate a piece of code that causes the exception by using information form the properties. Please send screenshot of the code to me.

3. You can get rid of the error by using the --skip-js-errors or -e command-line flags or the skipJsErrors property in the Runner.run method. But you won't be able to catch a real exception in your JS scripts when using these options.

[AlexanderMoskovkin]

I'm closing this since there were no activity here for a long time. Feel free to reopen the issue if it's still needed.
Also, you can try to run your tests on the latest TestCafe version.

[ivanzamoraarias]

Hi I have the same issue , it is like selectors can not see the DOM fields when it is on headless chrome.

[Lwdthe1]

I have the same issue as @ivanzamoraarias . All works fine in chrome and chrome:headless locally, but fails immediately on CI. The test I'm running is to count the number of visible elements by a selector. It returns the right number locally, but it returns 0 on Heroku CI.

// -- fixture

test('Check screen for logged out user', async (t) => {
  let pageModel = PubsPageModel.instance(t)

  await pageModel.hasLoginNavButton() // pass
  await pageModel.testMeta({numPubs: numAdvertisedPubs}) // pass
  await pageModel.hasContributeRequestButtons(numAdvertisedPubs) // FAIL!
})

// -- PubsPageModel

'use strict';

// see https://alligator.io/angular/e2e-testing-testcafe/

const BasePageModel = require('../../testCafe/pageModel/BasePageModel')
const {Selector, ClientFunction} = require('testcafe')


class PubsPageModel extends BasePageModel {
    constructor(testCafe) {
        super(...arguments)

        this.navbarRightActions = Selector('.js-navBarTopMainRightActions')
        this.navLoginButton = this.navbarRightActions.find('a').withText('Login With Medium')

        this.contributeRequestButtons = Selector(`.js-siteMain .js-mainPubsList .js-pubSendContributeRequest`)
    }

    testMeta({numPubs}) {
        let title = `${numPubs} Publications`
        let description = 'My website'
        let imageUrl = sm.const.App.Main.Seo.imageUrl
        let siteName = 'My Website'

        return this.metaEquals({
            description: description,
            ogTitle: title,
            ogDescription: description,
            ogImageUrl: imageUrl,
            ogImageSecureUrl: imageUrl,
            ogSiteName: siteName,
            twitterTitle: title,
            twitterDescription: description,
            twitterImageUrl: imageUrl,
            twitterSite: siteName,
            twitterCard: 'summary_large_image',
        }, title)
    }

    async hasContributeRequestButtons(numPubs) {
        await this.printHtml()
        console.log(await this.contributeRequestButtons.count) // 23 locally, 0 on CI
        let selector = await this.filterVisible(this.contributeRequestButtons)
        console.log(await selector.count) // 23 locally, 0 on CI
        return await this.t
            .expect(selector.count)
                .eql(numPubs, 'Should have enough contribute request buttons', { timeout: 2000 })
    }

    async hasLoginNavButton() {
        await this.navLoginButton.visible
        return this.t
            .expect(this.navLoginButton.getAttribute('ng-click')).eql('loginMediumer($event)')
    }

    static instance(t) {
        return new PubsPageModel(t)
    }
}

module.exports = PubsPageModel

// --- BasePageModel

'use strict';

// See https://devexpress.github.io/testcafe/documentation/test-api/test-code-structure.html

const Q = require('q')
const {Selector, ClientFunction} = require('testcafe')

class BasePageModel {
    constructor(testCafe) {
        this.t = testCafe

        this.head = {
            title: Selector('head title'),
            meta: {
                description: Selector('meta[name="description"]'),
                // -- og uses meta[property=]
                ogTitle: Selector('meta[property="og:title"]'),
                ogDescription: Selector('meta[property="og:description"]'),
                ogImageUrl: Selector('meta[property="og:image"]'),
                ogImageSecureUrl: Selector('meta[property="og:image:secure_url"]'),
                ogSiteName: Selector('meta[property="og:site_name"]'),
                // -- twitter uses meta[name=]
                twitterTitle: Selector('meta[name="twitter:title"]'),
                twitterDescription: Selector('meta[name="twitter:description"]'),
                twitterImageUrl: Selector('meta[name="twitter:image"]'),
                twitterSite: Selector('meta[name="twitter:site"]'),
                twitterCard: Selector('meta[name="twitter:card"]'),
            }
        }
    }

    async titleEquals(expected) {
        return await this.t
            .expect(this.head.title.innerText).eql(expected)
    }

    async metaEquals(meta, title) {
        if (title) await this.titleEquals(title)

        let result = this.t
        Object.keys(meta).forEach(async (key) => {
            await result.expect(this.head.meta[key].getAttribute('content')).eql(meta[key])
        })

        return await result
    }

    async filterVisible(selector){
        return await selector.with({ visibilityCheck: true }).filter((node) => {
            return $(node).is(':visible')
        })
    }

    async printHtml() {
        const getPageHTML = ClientFunction(() => document.documentElement.outerHTML)
        console.log(await getPageHTML())
    }

    async debug() {
        return await t.debug()
    }
}

module.exports = BasePageModel

[Lwdthe1]

Hmmm, after looking closer, it looks like the test wasn't waiting for angular to render the view that I'm trying to count

        <div id="site-main" class="site-main surface-container js-siteMain">
            <div>
                <!-- Dynamically render angular.js templates -->
                <div id="ng-controller-view" ng-view="">
                </div>
            </div>
        </div>

Why wouldn't it render the angular view on CI, but succeed to do so locally?

[miherlosev]

Why wouldn't it render the angular view on CI, but succeed to do so locally?

It's difficult to understand the cause of the problem without an example to reproduce.
Could you please provide a full example to reproduce (site + tests)?

[jwineman]

I was able to fix this by adding maximizeWindow(). My assumption was that the element was visible but off screen in CI which is why I was getting this error in CI but not locally:
The element that matches the specified selector is not visible.

[jelling]

Like @Lwdthe1 I am also using Heroku and calling testcafe chrome:headless from inside of the dyno and having weird issues. My test runs from local against the remote dyno and works. However, if I run it inside the dyno then most of the time it fails but it has also passed with the same code and config.

Previously, I had an issue running locally where Angular wasn't updating the view unless I added a call to detectChanges() in my test. So I suspect there might be performance interactions with Test Cafe and large angular applications. Running inside of a dyno likely makes that worse.

Edit: Heroku runs CI tests on a "performance-m" dyno by default which is fairly high-powered so I'm less certain server performance is the issue.

[jelling]

I fixed my issue and once again the cause was that change detection wasn't running automatically. This didn't happen locally on my Macbook Pro but it happened both in Heroku's CI environment (~2gb mem) and also on the VMs Crossbrowsertesting.com uses for automated testing.

So there's definitely an Angular <> Test Cafe performance issue. I'm not certain how to troubleshoot that but the fix is to manually invoke the change detector. I have to do it after every form input right now but only once the user is 90% through the flow.

[eaorak]

I've added a manual scrolling to the page due to this bug (see my comment there please):

https://github.com/DevExpress/testcafe/issues/2601

const scroll = ClientFunction((pixels) => window.scrollBy(0, pixels))

It works fine in UI mode but fails on headless. Is there a solution for this?

[steveblue]

I am seeing this issue as well trying to run TestCafe in CircleCI to test an Angular application.

#!/bin/bash -eo pipefail
npm run e2e:ci

> testcafe 'chrome:headless --no-sandbox' e2e/test/*.e2e-spec.ts

 Running tests in:
 - HeadlessChrome 0.0.0 / Linux 0.0.0

testcafe ^0.23.3
node 8.10.0

Every test except the first that navigates to the homepage fails. Cannot reproduce locally on a Mac.

[cvolpe-ecobee]

f

I fixed my issue and once again the cause was that change detection wasn't running automatically. This didn't happen locally on my Macbook Pro but it happened both in Heroku's CI environment (~2gb mem) and also on the VMs Crossbrowsertesting.com uses for automated testing.

So there's definitely an Angular <> Test Cafe performance issue. I'm not certain how to troubleshoot that but the fix is to manually invoke the change detector. I have to do it after every form input right now but only once the user is 90% through the flow.

@jelling How would I go about manually invoking the change detector in testcafe?

[jelling]

@cvolpe-ecobee

Inject the change detector service into the constructor of your component then expose the changeDetector service to the window level scope like this:

  constructor(private cd: ChangeDetectorRef) { 
     window['__cd'] = cd; 
}

The changeDetector object can then be called inside of your TestCafe script like this:

const detectChanges = ClientFunction(() => {
    // @ts-ignore
    return __cd.detectChanges(); // calls window function
});

await t
    .doStuff();

detectChanges();

await t
    .doMoreStuff();

...

[Lwdthe1]

@jelling Thanks for tagging me. I'm going to give it another shot with your suggestions.

[Lwdthe1]

I'm using AngularJs so I tried calling $timeout() to run the digest, but not much luck. maximizeWindow() also breaks in headless chrome.

[Lwdthe1]

I'm giving up on this after spending two days on it. It works perfectly locally but fails on Heroku CI. Please add support for Heroku CI or at least documentation like you have for other CI clients given the popularity of Heroku.

[AlexKamaev]

@Lwdthe1
Please create a separate feature request about Heroku CI using this form.
We would also appreciate it if you share your project details with us (full project code, a website url, test code).