Webdriveragent: Run XCUITest bundle from command line (without xcworkspace) ?

[sapieneptus]

Is it possible to invoke the XCUITest bundle (TestRunner.app) for physical devices from the command line directly, i.e. _without_ needing the xcodeproject/workspace?

E.g., could I email someone just the TestRunner.app bundle and have them be able to use the driver from the command line as though they had the full project?

I am aware of xctool and it seems like this _might_ be able to do it, but it appears to only work for simulators.

---

Note, if I install the TestRunner bundle to device and launch it manually, the server will still respond to /status, but am unable to start a new /session. I assume the bundle needs to be run with some certain environment configuration or needs to be run in the context of something like testmanagerd.

[marekcirkos]

Hi, unfortunately not yet, however currently we are working on this. /session command does not work, because xctest framework needs to be connected (via XPC) to testmanagerd deamon to execute taping, searching for elements, launching apps etc. As you may guess it will never work by launching TestRunner.app. There is some extra work needed to launch testmanagerd and connect them on host machine.

[marekcirkos]

Good news, we released XCTestBootstrap as a part of FBSimulatorControl and fbsimsctl.

You can run it from command line having WebDriverAgentRunner.xctest build.

./fbsimctl --state=booted LaunchXCTest [path]/WebDriverAgentRunner.xctest com.apple.mobilesafari --port 8200 -- listen

or trigger it from code by using https://github.com/facebook/FBSimulatorControl/blob/master/FBSimulatorControl/Interactions/FBSimulatorInteraction%2BXCTest.h

[sapieneptus]

Hey @marekcirkos, is there plan for device support as well? We've been working on this as well (but unsure of what I can say publicly...).

[marekcirkos]

XCTestBootstrap works with any platform given that you know how to operate the device. Check it out https://github.com/facebook/FBSimulatorControl/tree/master/XCTestBootstrap

[marekcirkos]

Probably you might want to use device preparation strategy https://github.com/facebook/FBSimulatorControl/blob/master/XCTestBootstrap/Strategies/FBDeviceTestPreparationStrategy.m

[sapieneptus]

@marekcirkos Truthfully, we've reverse engineered the whole thing in-house already, but it's great to see another implementation. We are still deciding on how we want to approach supporting XCUITest-based driver for clients (I work for Xamarin TestCloud/Microsoft/Calabash, btw), but fbsimctl tooling looks an interesting option for tooling :) Thanks for the info!

[marekcirkos]

Did you reverse engineered same path? Or different approach? Did you OpenSourced it? I am curious to see different approach :)

[sapieneptus]

We did not open source it. Unclear what the plan is...

But if/when we are able to share details, I will let you know! In general, it seems fairly similar. We more or less tried to just recreate the same flow used by OS X tooling for test bundle launching, so I don't imagine there can be _too_ much variance in implementations.

[marekcirkos]

We finally have Wiki page that describes hot to do that https://github.com/facebook/WebDriverAgent/wiki/Starting-WebDriverAgent :)

[sapieneptus]

Hey @marekcirkos , Just wanted to double check that command you posted for fbsimctl:

When I run

fbsimctl --state=booted \
launch_xctest /path/to/WebDriverAgentRunner.xctest com.apple.mobilesafari --port 8100 -- \
listen

I get

Failed to Parse Command '["--state=booted", "launch_xctest", "/path/to/WebDriverAgentRunner.xctest", "com.apple.mobilesafari", "--port", "8100", "--", "listen"]'
etc...

I just built fbsimctl today, is the command/wiki outdated already?

[marekcirkos]

It looks correct, I'll have a look when I am back from holiday

[sapieneptus]

@marekcirkos Ok, let me know! Just FYI, I was able to get it to work by using an explicit sim ID, so perhaps it is user error.

[lawrencelomax]

@sapieneptus It's probably because /path/to/WebDriverAgent.xctest doesn't exist, it needs to be a real path to a test bundle. If you're having issues with this still, I suggest opening an issue on the FBSimulatorControl repo.

[nghiadhd]

@marekcirkos

I would like to use XCTestBootstrap to invoke the test on real device without xcbuild
Currently I use idevicedebug to invoke the test but as you said, seem xctest doesn't connect to testmanagerd so I cannot execute any command

I read FBSimulatorControl code to find the way to use XCTestBootstrap but seem FBDeviceControl has not finished yet. So could you give me some hint to deal with it

[marekcirkos]

Probably you should post that on FBSimulatorControl repo ;). FBDeviceControl can execute xctest on iOS 9.x devices, however we do not have command wrapper (as fbsimctl for FBSimulatorControl) yet. You should be able to do it code by using FBDeviceTestPreparationStrategy and FBXCTestRunStrategy directly.

[marekcirkos]

You can also try using Xcode's 8 xcodebuild as it supports running tests without project file.

[nghiadhd]

Thanks @marekcirkos
I'll try with xCode 8 first, however I hope FBSimulatorControl (FBDeviceControl) can support run the test on real device without xcode (using lockdown protocol) or xcodebuild

[marekcirkos]

@nghiadhd it does support running tests on devices. Tho there is no command line support yet.

[nghiadhd]

@marekcirkos
After investigate the fbsimctl sources and flows. I'm starting to port fbsimctl to fbdevicectl. However I'm stuck at FBXCTestPreparationStrategy

• (instancetype)strategyWithApplicationPath:(NSString *)applicationPath
  applicationDataPath:(NSString *)applicationDataPath
  testBundlePath:(NSString *)testBundlePath;

Could I confirm sth?

• applicationPath: is path to *.app which is generated by Xcode after build test
• applicationDataPath: is path to .xcappdata which is download by Xcode -> Window -> Devices -> DownloadContainer
• testBundlePath: is path to *.xctest that is subfolder in *.app

Otherwise, in testBundlePath for real device doesn't have .xctestconfiguration

Here is structure of *.app

WebDriverAgentRunner-Runner.app
    - _CodeSignature
    - embedded.mobileprovision
    - Frameworks
    - Info.plish
    - PkgInfo
    - Plugins
        - WebDriverAgentRunner.xctest
            - _CodeSignature
            - Frameworks
            - Info.plist
            - WebDriverAgentRunner
    - XCTRunner

[marekcirkos]

Moving to https://github.com/facebook/FBSimulatorControl/issues/274