Salesforcedx-vscode: Apex: Unit test results show only 2,000 results when more results exist

[jwalke]

Summary

When an org has over 2,000 local test methods, and all local tests are run, only 2,000 results will be included in the test output file.

Steps To Reproduce:

Repository to reproduce:
dreamhouse-lwc-unit-test-issue.zip

1. Run the bin/install-scratch.sh script
2. Run sfdx force:apex:test:run --resultformat human --wait 60 --outputdir ./tests/apex

Expected result

The results file in /tests/apex will contain 2157 results

Actual result

The results file in /tests/apex will contain 2000 results

Additional information

The sample project is the vanilla lwc dreamhouse project with the addition of one test class that contains over 2k test methods.

Navigating to the test execution history in the Setup UI in the scratch org will show the correct number of test results:

We believe this issue started with the 51.7.1 (April 8, 2021) - CLI 7.96.1 release

SFDX CLI Version(to find the version of the CLI engine run sfdx --version):

7.99.0

SFDX plugin Version(to find the version of the CLI plugin run sfdx plugins --core)

51.12.0

OS and version:

macOS 10.15.7 (19H524)

[github-actions[bot]]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[NVandenBossche]

Not 100% sure if this is related, but we're also getting issues with executing large amounts of tests using the CLI. But for us it doesn't even create the output files, the command just goes into error. Seems like something to do with the streaming API.

Haven't tested the number of tests yet, but it's also failing for more than 2000 tests vs. succeeding for +- 500 tests.

[uip-robot]

This issue has been linked to a new work item: W-9250725

[AnanyaJha]

Hi @jwalke thanks for reporting this! We've been able to repro this issue and will work on getting a fix out there- we'll update this thread once we do.

[NVandenBossche]

Workaround is to revert back to v96 of sfdx-cli.

[AnanyaJha]

Hi @jwalke! This issue has been addressed by release v0.2.1 of the Apex plugin. To access these changes, please run sfdx plugins:install [email protected]. The plugin version can be verified by running sfdx plugins.

[brpowell]

This was also addressed in v51.15.0 of the extensions

[jwalke]

Thank you! I was able to confirm that the issue appears to be resolved for us now.