Nx: Azure Dev Ops regression with nx 8.4 affected:test task doesn't finish

[Tre665]

_Please make sure you have read the submission guidelines before posting an issue_

Prerequisites

Please answer the following questions for yourself before submitting an issue.
YOU MAY DELETE THE PREREQUISITES SECTION.

• [x] I am running the latest version
• [x] I checked the documentation and found no answer
• [x] I checked to make sure that this issue has not already been filed
• [x] I'm reporting the issue to the correct repository (not related to Angular, AngularCLI or any dependency)

Expected Behavior

run yarn affected test sends a done signal that is recognized by azure dev ops

Current Behavior

After all tests are completed the test task hangs forever.

Context

Please provide any relevant information about your setup:

• version of Nx used
  8.4.1
• version of Angular CLI used
  8.1.1

Hi,

I just upgraded our dependencies to the newest NX version via yarn update.
On my local machine (MacOs) all works perfectly.

But unfortunately our CI Build (runs on ubuntu) hangs on the test task.
To run the tests for our PR triggered builds we ran the following command:

yarn affected:test --parallel --base=origin/master --head=HEAD

This worked flawlessly before the update.
Now all tests run (and pass) but the test doesn't finish. The output on the console is the following

NX SUCCESS Running target "test" for affected projects succeeded

On my local machine it is followed with this message:
✨ Done in 137.25s.

I am not sure if the ✨ is new. Could that be the issue?
Unfortunately we can't upgrade to this behaviour.

[Tre665]

I tried different configurations and flags, but unfortunately nothing seems to help.
The last thing that we tried is:

yarn affected:test --runInBand --ci --base=origin/master --head=HEAD

The version that we used previously (and currently for development due this problem) is 8.1.
Unfortunately this prevents us from upgrading. Any help would be highly appreciated.

[Tre665]

I update the nrwl libraries to the newest version 8.4.3. But still without luck :-(

[vsavkin]

You could try to disable the new colorful output by setting the CI env variable to true. Something analogous to export CI=true.

Could you try it and let us know if it helps?

[Tre665]

Thank you for your response!

I am sorry but I am not sure if I did set the environment variable correct.
What I just tried was to run this command (this is set on the test step)

yarn affected:test -- --ci --base=origin/master --head=HEAD

The YAML for this step

steps:
- task: geeklearningio.gl-vsts-tasks-yarn.yarn-task.Yarn@2
  displayName: 'Run Unit Tests'
  inputs:
    Arguments: 'affected:test -- --ci --base=origin/master --head=HEAD'

And also set an environment variable in Azure CI with the value true.

Unfortunately with the same result.

This is the Output of the run (stripped to the important parts. If something important is missing, please let me know)

At the start

[command]/opt/hostedtoolcache/yarn/1.17.3/x64/yarn-v1.17.3/bin/yarn affected:test -- --ci --base=origin/master --head=HEAD
yarn run v1.17.3
warning From Yarn 1.0 onwards, scripts don't require "--" for options to be forwarded. In a future version, any explicit "--" will be forwarded as-is to the scripts.
$ ./node_modules/.bin/nx affected:test --ci --base=origin/master --head=HEAD

>  NX  Running target test for projects:

  - affected project 1
  - affected project 2
  (about 25 projects)

  With flags: --ci=true

The output of test run for one of the libs

warning From Yarn 1.0 onwards, scripts don't require "--" for options to be forwarded. In a future version, any explicit "--" will be forwarded as-is to the scripts.
$ ng test shared-ui-notification --ci=true
PASS libs/shared/ui/notification/src/lib/notification-large/notification-large.component.spec.ts (6.315s)
PASS libs/shared/ui/notification/src/lib/notification-small/notification-small.component.spec.ts
PASS libs/shared/ui/notification/src/lib/notification.service.spec.ts
PASS libs/shared/ui/notification/src/lib/badge/badge.component.spec.ts
PASS libs/shared/ui/notification/src/lib/notification-list/notification-list.component.spec.ts
PASS libs/shared/ui/notification/src/lib/notification-list-item/notification-list-item.component.spec.ts
PASS libs/shared/ui/notification/src/lib/notification-icon/notification-icon.component.spec.ts
PASS libs/shared/ui/notification/src/lib/notification-button/notification-button.component.spec.ts
PASS libs/shared/ui/notification/src/lib/shared-ui-notification.module.spec.ts

Test Suites: 9 passed, 9 total
Tests:       35 passed, 35 total
Snapshots:   0 total
Time:        11.898s
Ran all test suites.

And this is on the end of the test suite (where I have to cancel it manually)

———————————————————————————————————————————————


>  NX   SUCCESS  Running target "test" for affected projects succeeded

[cjanz]

I encountered the described problem as well, not with Azure DevOps but with Jenkins: All nx affected:* tasks with the --parallel flag run successfully and print the success message, but the process does not complete so that the CI build hangs forever.

After some investigation today I found the (potential) cause of the problem and a workaround: I have attached a NodeJS debugger to the hanging "nx affected:lint ..." process. Then I could evaluate process._getActiveHandles() which returns a single Socket instance. This Socket avoids that the NodeJS process terminates. As it turns out this socket is related to process.stdin. After calling process.stdin.unref() the nx process terminates gracefully.

So as a workaround I supply /dev/null as std input to the nx process in CI build, e.g.

nx affected:lint --base=origin/master --head=HEAD --parallel < /dev/null

With this workaround the CI build runs the parallel tasks without problems 🥳

Maybe the problem is caused by passing process.stdin to runAll in https://github.com/nrwl/nx/blob/6644cb7a24eb494989c395b0e08e786b0fa3c1df/packages/workspace/src/command-line/affected.ts#L237 but that's only a guess and can be completely wrong.

BTW: Thank you, Nrwl, for building Nx. It's really great and improves our development workflow a lot 🙏🏻

[vsavkin]

Folks, thank you for troubleshooting.

@Tre665 just to confirm:

• everything worked on ci before the upgrade. what version did you upgrade from?
• do you have the same issue for other affected commands (e.g., lint)?
• is it the same issue with and without --parallel

[Tre665]

I currently tried to replicate the workaround from @cjanz but unfortunately didn't find a way to append < /dev/null in azure dev ops yet. But I am currently investigating if this is somehow possible in Azure Devops.

@vsavkin

everything worked on ci before the upgrade. what version did you upgrade from?

Yes, we hadn't any issues before. The version before the update was pinned to 8.1.0

do you have the same issue for other affected commands (e.g., lint)

I feel a little bit ashamed, but currently we don't run lint on our CI server. The only other task that we run is the code format check via this command
format:check --base=origin/master --head=HEAD
This works also on the new version. But I will add a linting task to our pipeline and update this post to let you know if this has issues too.

is it the same issue with and without --parallel

Yes, the issues occurs with and without the --parallel flag (also --runInBand seems to do nothing with this issue)

[cjanz]

Thanks for the quick responses.

Just to clarify and add further information to the issue: The behavior that we have encountered is not exact the same as the problem that @Tre665 has:

• The CI build only hangs when using the --parallel flag. Without the flag everything works fine
• I could test the issue with the --parallel flag for the affected commands "lint", "test" and "build". The workaround also works for the different commands (in our scenario)

I hope this information helps.

[Tre665]

@vsavkin

Ok I added the lint step in our pipeline and can confirm that this step will also hangs after it has successfully finished. In my case it also doesn't matter if I add the --parallel flag or not.

An additional sidenote (don't know if it's useful) on the first run of

yarn affected:lint --base=origin/master --head=HEAD --parallel

We had some lint issues which made the command finish with errors. In error case the Azure DevOp task exits as expected and doesn't hang.

[ubergeoff]

Definitely a bug.

All CI builds (Bamboo for us) are stuck - once successful - from version 8.4.0 - when the new logging was introduced..?

However: Everything works locally.

[vsavkin]

We repro the issue and fixed it in 8.4.4.

Thank you folks for troubleshooting it.

[cjanz]

@vsavkin Thank you very much for fixing the bug 🙏🏻

[Tre665]

@vsavkin I also can confirm that the bug is fixed. Thank you for your quick response and the good work!

[marcus-sa]

I cannot confirm that the error is fixed, since my jobs are stuck and never finishes even though the linting/testing is done.

[saleemmalikraja]

@Tre665 - nx commands is not working in azure devops and it says "nx is not recognized as internal or external command" !!. Can you help me how did you solve this issue . Definitely i could see you where able to run affected test in azure devops . awaiting for your response... thanks in advance :)

[marcus-sa]

@saleemmalikraja Azure doesn't respect the binaries in node_modules, so you'd have to call them explicitly using yarn or npm

Like so yarn run nx, $(yarn bin)/nx or $(npm bin)/nx

Your question isn't really related to the issue btw.

OBS: everything is working perfectly using the latest release of Nx, I was just using an older version.

[saleemmalikraja]

@marcus-sa - Thanks for your input marcus.! I could find that nx command is working for others in azure devops but not working for me and its showing the below error . It would be an great help , if you could tell me how affected test has been set up

[marcus-sa]

@saleemmalikraja I just told you?
It's quite hard to help someone who doesn't want to read the answers being provided.
I'd recommend you to read the documentation on the tools you're using.

If you're using NPM then run npx nx affected:test

[saleemmalikraja]

@marcus-sa - The problem was with the hosted agent . When my hosted agent is windows , nx command is not working and when my hosted agent is ubuntu then nx commands are working .!! I couldn't see this answer anywhere . I appreciate your help but at the same time if it works for you the same may not be the case for all .It would be good if you can understand the problem statement completely and i m glad to give more details about the context incase if the problem description is not elaborate!!

[EvtK]

testing with Jest?

make sure to set isolatedModules: true otherwise testing takes forever on Windows machines.

I had to make a hack to get this to work, see here for more info.