Amplify-console: How to get my simple ng test command running with chrome headless

[blomm]

* Which Category is your question related to? *
Deployment pipeline

* Provide additional details e.g. code snippets *

in my package.json:
"test-ci": "ng test --no-watch --no-progress --browsers=ChromeHeadlessCI",
in my karma.conf:

browsers: ['Chrome', 'ChromeHeadless', 'ChromeHeadlessCI'],
    customLaunchers: {
      ChromeHeadlessCI: {
        base: 'ChromeHeadless',
        flags: ['--no-sandbox']
      }
    }

my test script is:
ng test --no-watch --no-progress --browsers=ChromeHeadlessCI
in my build settings file:

test:
  phases:
    preTest:
      commands:
        - wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
        - yum install -y ./google-chrome-stable_current_*.rpm
        - npm ci
    test:
      commands:
        - npm run test-ci
  artifacts:
    baseDirectory: dist/my-tests
    files:
      - '**/*'

ERROR MESSAGE:
ChromeHeadless have not captured in 60000 ms, killing

I guess the chrome install is not being found on the Linux Image/Machine?
do I need to set chrome_bin in the environment variables? if Yes, what is it?

Thanks for any help

[Joycehao19]

Hello @blomm , do you have a sample repo that we could test?

[blomm]

Hi @Joycehao19 , not a publicly available one, unfortunately.
All the documentation relates to end to end cypress testing, but I just want to run my unit tests, should be simple.

[Joycehao19]

Hi @blomm I can't reproduce this issue. I got Tests successfully completed when I tried.

1. Did you edit files: ['dist/my-tests/**/*.js'] in karma.conf.js (or something related)?
2. For test script: Why did you put ng test --no-watch --no-progress --browsers=ChromeHeadless instead of ng test --no-watch --no-progress --browsers=ChromeHeadlessCI ?

There are configuration files for both the Karma JavaScript test runner and Protractor end-to-end testing tool, which you must adjust to start Chrome without sandboxing.

• https://angular.io/guide/testing)

[blomm]

Hi @Joycehao19 ,

1. I haven't edited this, I don't think the problem is related to finding the test files, actually, this my-tests folder doesn't exist, I was trying to figure out what to put for my test artifacts (in fact it appears there are no artifacts for _ng test_ unless I run with _coverage=true_, so I'm not sure what to put for test>artifacts>baseDirectory ? what have you put for this value?)
2. Apologies, this was a typo, I am running with ng test --no-watch --no-progress --browsers=ChromeHeadlessCI

But I'm pretty sure my problem is related to the build machine not finding Chrome... how are you installing Chrome? I've seen people mention installing Puppeteer, did you do this?

> ng test --no-watch --no-progress  --browsers=ChromeHeadlessCI
2019-11-19T07:10:57.374Z [INFO]: 19 11 2019 07:10:57.370:INFO [karma-server]: Karma v4.1.0 server started at http://0.0.0.0:9876/
2019-11-19T07:10:57.376Z [INFO]: 19 11 2019 07:10:57.374:INFO [launcher]: Launching browsers ChromeHeadlessCI with concurrency unlimited
2019-11-19T07:10:57.456Z [INFO]: 19 11 2019 07:10:57.456:INFO [launcher]: Starting browser ChromeHeadless
2019-11-19T07:11:57.460Z [INFO]: 19 11 2019 07:11:57.458:WARN [launcher]: ChromeHeadless have not captured in 60000 ms, killing.
2019-11-19T07:11:59.461Z [INFO]: 19 11 2019 07:11:59.460:WARN [launcher]: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
2019-11-19T07:12:01.461Z [INFO]: 19 11 2019 07:12:01.461:WARN [launcher]: ChromeHeadless was not killed by SIGKILL in 2000 ms, continuing.

[Joycehao19]

@blomm ,

Q1: I put the folder which contains my test.js
Q2: I didn't use Puppeteer.

I followed this link. [You need to have Chrome 59+.]
In karma.conf.js:

    browsers: ['Chrome', 'ChromeHeadless', 'ChromeHeadlessCI'],
    customLaunchers: {
      ChromeHeadlessCI: {
        base: 'ChromeHeadless',
        flags: ['--no-sandbox']
      }
    },
    files: ['test/**/*.js'],

In amplify.yml

test:
  phases:
    preTest:
      commands:
        - wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
        - yum install -y ./google-chrome-stable_current_*.rpm
        - npm ci
    test:
      commands:
        - npm run test:ci
  artifacts:
    baseDirectory: test
    files:
      - '**/*'

When you test your app locally, could it work?
When I search ChromeHeadless have not captured in 60000 ms, killing. , I found some similar issues. You could also take a look and ask them for help: https://github.com/karma-runner/karma-chrome-launcher/issues

[blomm]

@Joycehao19 thanks for your continued support
Running the test locally, if I execute '_npm run test-ci_' the tests all pass, but I get

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test-ci: `ng test --no-watch --no-progress --browsers ChromeHeadlessCI`
npm ERR! Exit status 1

If instead I execute _ng test --no-watch --no-progress --browsers ChromeHeadlessCI_ the test will pass and not error. This is very strange, that running through npm scripts causes an error.

So in my amplify build settings I chose to execute that base command _ng test --no-watch --no-progress --browsers ChromeHeadlessCI_ rather than _npm run test-ci_
I also set the test>artifacts>baseDirectory to be _src_ because this doc says that: _artifacts>baseDirectory - The directory from which tests are run._
And my tests are in many different folders, not just a test folder. Really though, this baseDirectory appears to be define the folder that I am allowed to download via the '_download artifacts_' button.
Somehow, the test stage passes. Logs show that the pretest steps ran, but there is no indication of tests actually running, the log file just ends abruptly:

But now the deploy stage fails, no indication of the problem. (Deploy works fine if I don't run tests)

[Joycehao19]

Hi @blomm , Can you send me you appID and region? Then I can check the build logs of your app.

For MaxListenersExceededWarning, you could try something like eventEmitter.once().

[blomm]

Hi @Joycehao19 , thanks very much:
eu-west-1
dbwksz5yaroo9

[Joycehao19]

@blomm Did your app deploy successfully? Build 089 seems like successfully deployed.

[blomm]

Hi @Joycehao19 ,
I have my build successfully running, but it's without the test section of the buildSettings.
I've re-added the test section in build 94, chrome headless is not being found, test phase doesn't fail (but doesn't run any tests), but deploy does fail.
Question: What is the artifacts>baseDirectory for in the test section? does it define where the tests are located? or where the output from running tests will be dumped?

[blomm]

Update: so as I had been saying, the Linux build server does not find chrome.
adding this line in prebuild of test secion resolved that, and now I do not get the karma timeout, and my tests run:
export CHROME_BIN=/usr/bin/google-chrome

however, my deploy is still failing with the same 3 error lines, (unless I remove the test phase, then it deploys)

2019-11-24T16:02:04 [INFO]: Starting Deployment
2019-11-24T16:02:04 [ERROR]: Failed to deploy
2019-11-24T16:02:04 [INFO]: Deployment finished.

^^ this was build 98 by the way... I've since removed the test section again to get my build completing.

[Joycehao19]

@blomm
Question: What is the artifacts>baseDirectory for in the test section? does it define where the tests are located? or where the output from running tests will be dumped?

It defines where the tests are located.

Could you please try re-connect the app and edit amplify.yml (something like)

test:
  artifacts:
    baseDirectory: tests
    files:
      - "**/*.png"

The “artifacts:” sub-config needs a baseDirectory, and at least one glob. It can be all - */ - but that may have undesired effects, as it will attempt to upload the entire test directory, the above example just searches for any .png files, and if there are none, none will be uploaded.

[blomm]

Hi @Joycehao19, tests folder fails because this does not exist in my project. If I set src as the baseDirectory, the tests will pass and run, but my deploy fails with the 3 non-descript error messages.

As an aside, setting src as the folder will also mean that on the console, when I click the 'download artifacts' button i will be able to download everything in src folder.

[david-wallace-croft]

I am getting the same error during my deploy phase:

2020-01-16T21:08:20 [INFO]: Starting Deployment
2020-01-16T21:08:20 [ERROR]: Failed to deploy
2020-01-16T21:08:20 [INFO]: Deployment finished.

I have a single Protractor test which runs in Chrome headless and succeeds right before the deploy fails. I am able to download the test artifact zip file and read the JSON file output inside.

My build configuration test section looks like this:

test:
  phases:
    preTest:
      commands:
        - wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
        - yum install -y ./google-chrome-stable_current_*.rpm
        - npm ci
    test:
      commands:
        - if [ "${AWS_BRANCH}" = "master" ]; then npm run e2e-prod; fi
        - if [ "${AWS_BRANCH}" = "develop" ]; then npm run e2e; fi
  artifacts:
    baseDirectory: test-results
    files:
      - '**/*'

My Protractor configuration file looks like this:

// Protractor configuration file, see link for more information
// https://github.com/angular/protractor/blob/master/lib/config.ts

const { SpecReporter } = require('jasmine-spec-reporter');

exports.config = {
  allScriptsTimeout: 11000,
  resultJsonOutputFile: 'test-results/e2e/result.json',
  specs: [
    './src/**/*.e2e-spec.ts'
  ],
  capabilities: {
    'browserName': 'chrome',
    chromeOptions: {
      args: [ "--headless", "--no-sandbox", "--disable-gpu" ]
    },
  },
  directConnect: true,
  baseUrl: 'http://localhost:4200/',
  framework: 'jasmine',
  jasmineNodeOpts: {
    showColors: true,
    defaultTimeoutInterval: 30000,
    print: function() {}
  },
  onPrepare() {
    require('ts-node').register({
      project: require('path').join(__dirname, './tsconfig.e2e.json')
    });
    jasmine.getEnv().addReporter(new SpecReporter({ spec: { displayStacktrace: true } }));
  }
};

[nordinh]

I'm getting the same error as well when adding the test section to the build script:

test:
  phases:
    preTest:
      commands:
        - wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
        - yum install -y ./google-chrome-stable_current_*.rpm
        - export CHROME_BIN=/usr/bin/google-chrome
        - npm ci
    test:
      commands:
        - npm run test -- --no-watch --no-progress --browsers=ChromeHeadlessCI --reporters=coverage-istanbul,html
        - npm run e2e -- --protractor-config=e2e/protractor-ci.conf.js
  artifacts:
    baseDirectory: test-results
    files:
      - '**/*'

Tests pass but deploy fails:

2020-04-03T19:39:27 [INFO]: Starting Deployment
2020-04-03T19:39:27 [ERROR]: Failed to deploy
2020-04-03T19:39:27 [INFO]: Deployment finished.

[Joycehao19]

Hi @nordinh , @david-wallace-croft , would you mind provide your appId and region? (App Setting => General => App ARN => arn:aws:amplify:${Region}:xxxxxxx:apps/${appId}).

[david-wallace-croft]

arn:aws:amplify:us-east-1:xxxxxxxxxxxx:apps/dd0x2t9ylfft9

I created a new clean project:
ng new amplify-protractor

Changes to e2e.protractor.conf.js:

exports.config = {
  [...]
  resultJsonOutputFile: 'test-results/e2e/result.json',
  [...]
  capabilities: {
    [...]
    chromeOptions: {
      args: [ "--headless", "--no-sandbox", "--disable-gpu" ]
    },
  },
  [...]

Created test-results/e2e/.gitkeep
Added test-results to .gitignore
No other changes to project

Running tests locally succeeds:
npm run e2e

Added test section to default amplify.yml:

version: 0.1
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: dist/amplify-protractor
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
        - yum install -y ./google-chrome-stable_current_*.rpm
        - npm ci
    test:
      commands:
        - npm run e2e
  artifacts:
    baseDirectory: test-results
    files:
      - '**/*'

Amplify Console reports that test phase succeeded but deploy phase failed:

2020-04-14T19:09:26 [INFO]: Starting Deployment
2020-04-14T19:09:26 [ERROR]: Failed to deploy
2020-04-14T19:09:26 [INFO]: Deployment finished.

Failure might actually be with the preceding test phase since the test phase console log ends in a way that makes it look like it failed to complete:

2020-04-14T19:09:15.277Z [INFO]: ℹ ｢wdm｣: Compiled successfully.
2020-04-14T19:09:15.500Z [INFO]: [19:09:15] I/launcher - Running 1 instances of WebDriver
2020-04-14T19:09:15.501Z [INFO]: [19:09:15] I/direct - Using ChromeDriver directly...
2020-04-14T19:09:17.158Z [INFO]: Jasmine started

[sintaro]

Any update?

[david-wallace-croft]

I figured it out. The main problem was a missing configFilePath in the test artifacts section. Excluding configFilePath causes the test phase to appear to succeed including generating successful test artifacts but with the deploy phase reported as failing for no apparent reason. I am not sure why configFilePath is required or what it is used for in the context of Angular Protractor tests but it is described here:
https://docs.aws.amazon.com/amplify/latest/userguide/running-tests.html

Another lesser issue that I was having was that I needed to first create the directory where the test-results were stored. My updated amplify.yml test section follows:

test:
  phases:
    preTest:
      commands:
        - wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
        - yum install -y ./google-chrome-stable_current_*.rpm
        - npm ci
        - mkdir -p test-results/e2e
    test:
      commands:
        - npm run e2e
    postTest:
      commands:
        - echo postTest started on `date`
  artifacts:
    baseDirectory: .
    configFilePath: '**/result.json'
    files:
      - 'test-results/e2e/result.json'

[ddouras]

There is no need to create a test target to your amplify file if you just want to run unit tests. I got this working with following steps:

Set your amplify.yml to be :

frontend:
  phases:
    preBuild:
      commands:
        - npm ci
        - wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
        - yum install -y ./google-chrome-stable_current_*.rpm
        - npm run test:ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: dist
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Add to your karma.conf.js options:

browsers: ['ChromeHeadless'],
customLaunchers: {
 ChromeHeadlessCI: {
   base: 'ChromeHeadless',
   flags: ['--no-sandbox'],
 },
},

Add this script to package.json:

"test:ci": "ng test --no-watch --no-progress --browsers ChromeHeadlessCI"