Testcafe: Screenshots in Chrome and Firefox on MacOS with `--concurrency` are missing

[beyondcompute]

Are you requesting a feature or reporting a bug?

A bug.

What is the current behavior?

If you are taking screenshots while running with --concurrency 2 (or more) in Chrome or Firefox (Safari works fine!) on Mac OS 10.12 Sierra, only one window takes screenshots. Others end up reporting:

 Warnings (1):
 --
  Was unable to take a screenshot due to an error.

  Command failed: /something/testcafe/node_modules/testcafe-browser-tools/bin/mac/generate-thumbnail
  /some-path/2018-xx-xx_xx-xx-xx/test-2/Firefox_57.0.0_Mac_OS_X_10.12.0/1.png
  /some-path/2018-xx-xx-xx-xx/test-2/Firefox_57.0.0_Mac_OS_X_10.12.0/thumbnails/1.png 240 130
  Error 78: failed to open file for reading

Not only thumbnails are missing but actual screenshots do not get created as well.

What is the expected behavior?

Screenshots to be taken, no warnings issued.

How would you reproduce the current behavior (if this is a bug)?

1. Modify examples/basic/test.js with the patch:

Patch

diff --git a/examples/basic/test.js b/examples/basic/test.js
index b5809a48..14ab9b8e 100644
--- a/examples/basic/test.js
+++ b/examples/basic/test.js
@@ -1,7 +1,12 @@
 import Page from './page-model';

+const takeScreenshot = async (t) => {
+  await t.takeScreenshot();
+};
+
 fixture `A set of examples that illustrate how to use TestCafe API`
-    .page `https://devexpress.github.io/testcafe/example/`;
+    .page `https://devexpress.github.io/testcafe/example/`
+    .afterEach(takeScreenshot);

 // Page model
 const page = new Page();

>

1. Run as:

./bin/testcafe-with-v8-flag-filter.js 'chrome --disable-infobars' examples/basic/test.js --concurrency 2 --screenshots-on-fails --screenshots shots

Provide the test code and the tested page URL (if applicable)

See above

Specify your

• operating system: macOS Sierra 10.12.6
• testcafe version: 0.18.6 and 0.18.7-dev20180124
• node.js version: 8.6.0

Some more info

It appears that bin/mac/find-window.scpt is able to locate only windows of the last open “application”. I.e., the code:

repeat with bundleId in bundleIds

        tell application id bundleId

always “tells” to the last opened application with certain bundelId and gets only its window information

So this.windowDescriptors here end up looking like:

{ '~jkMw2z': 
   { bundleId: 'com.google.Chrome',
     windowId: '1',
     cocoaId: '81092' },
  '5AraLjG': null }

(if we run two instances)

Interesting, that in Firefox cocoaId seems to match windowId (In Chrome they are different):

{ VthVVxH: 
   { bundleId: 'org.mozilla.firefox',
     windowId: '82804',
     cocoaId: '82804' },
  O8IaHio: null }

Because of this, screenshot gets created only for non-null descriptor windows (neither bin/mac/find-window-cocoa nor bin/mac/screenshot get called for the null one). This all goes silently and only in the end bin/mac/generate-thumbnail is executed for all screenshots and fails in those cases where screenshots were not created in the first place.

Now, I spent some time trying to modify the src/natives/find-window/mac/find-window.applescript to handle the situation when many apps are running with the same bundleId but to no avail (I am no Applescript expert). It is possible to get names of all windows of all process using tell application "System Events" but System Events has no information about any “window ids”. Also this requires Assistive Permissions!

I was able to modify the find-window-cocoa.m

Approximately like this:

#import <Cocoa/Cocoa.h>

int main (int argc, char **argv)
{
   if (argc < 2) {
     fprintf(stderr, "Usage: %s stringToSearch\n", argv[0]);
     return 1;
   }

   NSString* searchNamePart = [NSString stringWithUTF8String:argv[1]];

   NSArray *windows = (NSArray *)CGWindowListCopyWindowInfo(kCGWindowListExcludeDesktopElements, kCGNullWindowID);
   for (NSDictionary *window in windows) {
      NSString* windowName = [window objectForKey:(NSString *)kCGWindowName];

      if ([windowName length] == 0) {
        continue;
      }

      NSRange textRange = [windowName rangeOfString:searchNamePart options:NSCaseInsensitiveSearch];
      if (textRange.location != NSNotFound) {
        int windowNum = [[window objectForKey:(NSString *)kCGWindowNumber] intValue];
        int ownerPid = [[window objectForKey:(NSString *)kCGWindowOwnerPID] intValue];
        NSRunningApplication* app = [NSRunningApplication runningApplicationWithProcessIdentifier:(pid_t) ownerPid];
        printf("%s\n%d\n", [[app bundleIdentifier] UTF8String], windowNum);
        break;
      }
   }

   return 0;
}

>

so that it returns bundleId and cocoaId of the first window, whose name contains certain text (I am no Objective C expert either). But I am not aware of the way of obtaining the “Applescript’s” windowId this way. So it renders natives/close/mac/close.applescript and others useless. Because:

tell application id bundleId to close window id windowId

will not work with cocoaId (and in case of Chrome they are different).

[AndreyBelym]

@beyondcompute, really thank you for the excellent research! :clap: Yes, Systems Events likely can help in this case, but we try to avoid requesting Assistive Permissions as much as possible.

As a workaround, you can try to use Headless mode for both Chrome and Firefox. TestCafe uses DevTools and Marionette debug protocols to capture screenshots in headless browsers, so they shouldn't be affected by problems with Applescripts.

[AndreyBelym]

I've finally found the way to get screenshots in concurrency working on macOS even without requiring Assistive Perimssions. There is a method SBApplication.applicationWithProcessIdentifier for getting an AppleScript application object by a specified UNIX process ID. Unfortunately, it's only accessible from Objective C, and it means that I have to rewrite all our AppleScripts in Objective C, because each script needs access to an application object.

[beyondcompute]

Just wanted to say thank you to @AndreyBelym and all the TestCafe contributors for the incredible work you people are doing! 🙌 🎉

[lock[bot]]

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.