Detox: Run on hardware iOS device

[saikrishna321]

[LeoNatan]

There are several challenges we need to overcome before we can achieve this.
Please know there there will be some limitations to what will be possible on the device, such as permission changing.

[simonracz]

Here is how we achieved this recently.

• We took the "Copy Detox Framework" run script from the demo-react-native example project.
• As I ran into code signing issues, I replaced it by manually embedding the Detox.framework into the project.
• I added a new script that recursively code signs each framework and dylib inside Detox.framework.
• I copied and embedded the XCTest.framework also in the project as Earlgrey was complaining about it.
• I set up the detoxServer and detoxSessionId in the launch arguments of the scheme.
• The device and laptop was on the same network of course. The laptop's IP was set up correctly in the appdelegate and in the detoxServer launch argument.
• I set up the same port and sessionId in the detox config (in package.json)
• Also added an "ios.none" config there.
• I ran "detox run-server" manually
• Then, I ran "detox test --configuration ios.none".

And now, about the problems. After setting these all up, the app was running, it connected to the server, the tests started to run. And each of them failed. :|

During investigating what could be wrong, it suddenly started to work. Without any code change. The same pattern repeated on one of my colleague's machine. (It failed the tests first, then it passed them half an hour later.)

[rotemmiz]

@simonracz did you run a react-native app? in debug mode ? Loading the bundle from the debug node server may take time, and detox may reach the default timeout and fail.

[simonracz]

If you mean the Mocha runner time out, then no, it wasn't that.

We tried both release and debug mode. It was the demo-react-native sample. No modifications were made to it, apart from the project file modifications mentioned above.

The first test line is "await expect(element(by.id('welcome'))).toBeVisible();". This failed with an Earlgrey assertion. It found the view with the id welcome, however it failed the visibility test.

I started to debug the runtime view hierarchy. For example, I stopped the process and called "po [GREYElementHierarchy hierarchyStringForAllUIWindows]".

For this view it showed the AX.id correctly, but the AX.frame was (0,0),(0,0). Sadly, I don't have saved logs.

After it started to work, the AX.frame was correct.

I wasn't able to reproduce it since then.

[simonracz]

I was able to reproduce this "Earlgrey fails the visibility test" issue.

In fact I spent half-a-day on chasing this. I was able to narrow it down to the following.

The matched view is rejected, because it's accessibilityFrame is zero. It's frame is correct. It is an RCTView, which inherits from UIView. From to the docs "The default value for this property is CGRectZero unless the receiver is a UIView object or is a subclass of UIView, in which case the value is the frame of the view."

So, this shouldn't happen unless someone manually touches the accessibilityFrame. I looked at the react-native source-code, they never touch the accessibilityFrame. Neither does Earlgrey (except in it's tests).

I restarted, cleared, rebuilt, killed everything. Nothing helped.

I don't think this is a Detox issue. I wrote it down only that in case you encounter it, let's share our findings.

[simonracz]

After spending more than a day on this bug I found the issue.

Since some iOS version (don't know yet exactly when, but 10.3.2 and 10.3.3 are affected) Apple introduced a bug/feature around accessibility.

The accessibilityFrame is not set, nothing isAccessible etc. unless the user is really using some accessibility features, e.g. voiceover.

Earlgrey heavily depends on accessibility. I don't know why haven't anyone else found this, opened a bug about it yet.

I'll open a bug report on Earlgrey and also open radar. I want to determine first the exact iOS version when this was introduced.

[LeoNatan]

@simonracz
Interesting. XCUITest also uses accessibility for its operation. It’s worth investigating how the system allows that for those cases but doesn’t for ours. Most likely an env var.

[simonracz]

Made a radar about it. http://www.openradar.me/radar?id=5043342748418048

[LeoNatan]

@simonracz
This is probably an intentional change. There is no need to calculate these values on a real device, running production software (I don't mean a release build). We need to find out how the system differentiates between running from Xcode and otherwise. I will take a look soon.

[LeoNatan]

I have found a way to force accessibility.
I will push soon to Detox.

[saikrishna321]

@LeoNatan is it possible to run on real device now ? I couldn't find any docs on it

[pachun]

@LeoNatan This will be _so_ helpful to have! Thanks for putting in the work. Can't wait to get my hands on it.

[LeoNatan]

Running on devices is not supported yet.

[LeoNatan]

@pachun Please do not unassign members of the team from issues. Leave the issue management to us. Thanks.

[pachun]

@LeoNatan I have no idea how that happened. Sincerest apology. 100% unintentional. All I meant to do was thank you.

[LeoNatan]

Cheers!
We plan to support Detox on device soon.

[sonianin]

@LeoNatan, are there any updates on the iOS real devices support? We are really looking forward to this functionality...

[LeoNatan]

Nothing to announce yet. It’s in our backlog and we intend to investigate this eventually, but it’s not on the top of our priorities. There are some challenges that we haven’t solved yet.

[mike623]

Really looking forward of this too. My app need turn on the real device to check the camera.

[fdnhkj]

@LeoNatan Don't you think the following note in README should be adapted then?

Runs on Devices: Gain confidence to ship by testing your app on a device/simulator just like a real user.

IMO, it is a bit misleading. I would mention it is not yet supported on iOS.

[DanielMSchmidt]

@fdnhkj I think you are right, could you go ahead and make a PR?

[fdnhkj]

@DanielMSchmidt Sure I can!

Do you know if running on physical device on Android works? I have seen android.attached config in the docs but I didn't test it so I'm not sure.

I'll adapt the README accordingly.

[DanielMSchmidt]

AFAIK it should

Fidan Hakaj [e-mail] schrieb am Mo., 8. Jan. 2018 um
15:49 Uhr:

@DanielMSchmidt https://github.com/danielmschmidt Sure I can!

Do you know if running on physical device on Android works? I have seen
android.attached config in the docs
https://github.com/wix/detox/blob/master/docs/Introduction.Android.md#4-add-android-configuration
but I didn't test it so I'm not sure.

I'll adapt the README accordingly.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/wix/detox/issues/95#issuecomment-355986544, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABRm1t2L4xtCOM1DmKekI5Qa5isrXHd0ks5tIisBgaJpZM4MVY2O
.

[LeoNatan]

I think I've solved the first hurdle (but a big one), which is setting environment variables when running on device. 🎉
Using a modified ios-deploy, I was able to embed environment variables.

Steps ahead: sign Detox dylib with the same signature as the user's app and send to device storage, then try to inject it using DYLD_INSERT_LIBRARIES environment variable.

[LeoNatan]

Could anyone please walk me through how signing is managed on test clouds?

[Karniej]

Any information when the testing on real android devices will be ready?

[swaminator]

Checking in to see if there's been any progress with this?

[LeoNatan]

No new progress. We've done a POC and it's technically feasible.
An actual implementation is planned, but there is no timeline.

[IljaDaderko]

@LeoNatan since it can run on a real device will it be possible to somehow integrate Detox tests into firebase test lab? Especially now with iOS support announced, could be very powerful.

[LeoNatan]

Unfortunately we have not yet implemented real device support.

[aknackiron]

I work for Bitbar (mobile devops cloud) and we tried out Detox in our cloud successfully. We even had a post about it (link below). We believe real device execution is faster and more robust to running mobile tests. We are interested in helping to implement the cloud solution for iOS real device support once you have something working.

https://bitbar.com/its-time-to-detox-your-mobile-tests-on-bitbar-cloud/

[rotemmiz]

Hey @aknackiron,
This is exciting!
Please contact me at [email protected] so we can continue discussing this offer seriously.

[DusanDimitric]

Guys, any news on this?
We're having a difficult time testing React Native \w Appium and are looking for alternative test frameworks/SaaS offerings which we can use to test our iOS/Android apps on a real device farm.
Thanks

[aknackiron]

We tried out WIX's solution and our cloud supports Detox on real Android and iOS devices. Android side is out of the box, iOS needs some extra setup from our side. Not to make this into a sales post, please get in touch with me directly at niko.[email protected].
Our cloud supports also almost any other frameworks out there, not only Detox, Appium or native frameworks.

[EdwardDrapkin]

@LeoNatan since this appears to still be in TODO, can you release the POC so we (the community) can work on it and maybe PR it back upstream without having to reinvent your wheels?

[DanielMSchmidt]

Maybe on a similar topic just to share it upfront, we are trying to move detox into a position where is it more extensible. The goal is to allow you to solve this on your own, depending on the needs of your platform. I wrote an issue for that to track the progress: https://github.com/wix/Detox/issues/1091

[LeoNatan]

@DanielMSchmidt

While extensibility is very important, the magnitude of work required for this to work on a real device is beyond the scope of plugins. It involves touching all parts of detox and some JS extensions will not suffice. This is one of these cases which should come build-in into Detox. I am not sure extensibility like that is possible or feasible. I don't think an entirely new category of device will be able to be added as an extension.

@EdwardDrapkin

My POC was to test the current delivery method of Detox on a real device.

Right now, we use an environment variable DYLD_INSERT_LIBRARIES to inject the Detox library without requiring the user's project be changed or changing it ourselves. That's good for many reasons:

• Detox should not be uploaded to the app store, so a clear solution is always better
• A lot of our users are not familiar with Xcode, so making changes to the project is difficult
• Changing the IPA on behalf of the user requires resigning of the package, which we cannot do
• The same binary can be checked with newer versions of Detox without requiring rebuild of the binary

Doing this on the simulator is relatively easy, as Xcode provides the simctl utility, which we can use to install and run binaries on the device.

For a real device, no such official utility exists, despite there being support for it (Xcode does it, obviously, as well as iTunes). So I tested a third-party utility called ios-deploy, which allows for some similar capabilities. One hurdle I found right away was providing an environment variable, which is not normally possible to do on a device. But again, Xcode is able to do it, and ios-deploy has an option to launch in a debugger mode (which means it's a controlled launch, just like Xcode does). So I was able to modify the source of ios-deploy to add the environment variable and was able to launch an unmodified app and load our framework.

This is a big step, but only the first one. There are many features that would break. applesimtutils obviously can't run on a real device, so permission dialogs cannot be handled. Going to background and foreground will probably not be possible. Since my POC, some features have been added which make assumptions that the tested process is running on the same machine as the tester and just go to the disk and take files; that will clearly not work, so all those systems need to be rewritten to use file access APIs which we don't have in Detox yet. No one has tested how Earl Grey works on arm yet with real UIKit (simulator and device UIKit have internal differences). That's a lot of hurdles to overcome before we can say we support real devices.

But, with ios-deploy, it is probably possible to achieve some sort of running POC at some capacity with relatively not much work. But from this point to a point where we can feel confident to write "we support real iOS devices" is far.

[timoisalive]

@aknackiron
@LeoNatan

Out of curiosity, how is it possible Bitbar (apparently) has Detox running on iOS devices in their device farm, but Detox alone is not (even close, apparently) supporting iOS devices?

Were/are you working together on this?

[LeoNatan]

I don't see anywhere that it runs on a real device?
Even if so, the project is open source and they could have done the necessary to make it work. As I have outlined, this is not that hard.

[aknackiron]

Using Leo's setup we're able to host customer dedicated iOS devices for Detox testing.

[vincenzovitale]

@aknackiron, could you share that back to the community?

[AlexK777]

I highly doubt @aknackiron will share his implementation if bitbar uses it, as that is a paid service.

Maybe I am wrong :smile:. But even if that is the case I would be personally wary to use that implementation until it was "blessed" by some of the main contributors here. It is exciting to see the ball move forward nevertheless.

[LeoNatan]

My apron is ready for blessing.

[aknackiron]

@aknackiron, could you share that back to the community?

As mentioned earlier, we have not added anything to Leo's implementation. I don't know if these steps have been shared elsewhere than in Slack.

[hammadj]

Any update on this? Running on real devices is crucial for us to setup automated performance testing. If this can be working reliably on Bitbar our team would love to use it. @aknackiron Do you have a guideline for setting all this up? What are the limitations of testing on device using Bitbar?

[aknackiron]

@hammadj
Detox's real iOS device support is not ready for prime time and is currently not on Detox roadmap. We have not developed anything on top of Detox.

Know that at current state (last we checked), you don't get screenshots and you can't relaunch app with Detox on real iOS devices.

If you still want to try out, we can do a project with you, but you'll have to get in touch with our sales... :-/

[LeoNatan]

Guys, please keep the discussion civil. If you'd like to see device support in the open source project, you are welcome to contribute; we will guide you. But please do not include rude comments.

[AlexK777]

@LeoNatan fair. But the unwillingness to share a repro (that many have asked for in this thread) and instead directing you to contact a sales department does not feel like "contributing to the open source project". 🤷‍♂️

[LeoNatan]

Detox is licensed under the MIT license. This is a permissive license. As long as there is attribution, anyone is free to clone the project privately and extend it as they see fit; if they decide not to share their extensions, that is part of the contract too. While I understand your frustration, your comment was out of line and unacceptable in this community.

[muraliseveneleven]

Hey Guys ..any direction would be helpful. We have a huge framework written in Java/Appium and one our goal is to execute tests on real devices. We are trying to transition everything to detox( as our new app is react native and all the backend is NodeJs). we successfully ran few tests and works well with our complete simulator setup and found it little more powerful than appium at various instances , Now we wanted to run this on real device( test devices which we have ) as cloud is out of the financial bucket to spend for this activity. Do we start and try what @simonracz had suggested or any other workable workarounds for this to make it happen today. Thanks

[bvodola]

@LeoNatan my company would greatly benefit from this feature.
And I am interested in helping. I could dig into it and create a PR once I have something.
Could you guide me or maybe give me some hints on this, so I could get started? Thanks!

[LeoNatan]

This comment mentions everything I've done in POC to get it to work on a real device, as well as challenges I've met or think will be difficult to tackle. Here is more info:

The general gist of how we inject Detox into apps on simulators is as follows:

1. Take a user's .app and install it on simulator
2. Launch app with DYLD_INSERT_LIBRARIES environment variable to ask the operating system to load the Detox dylib
3. We pass launch arguments to the process to control how Detox functions.

We believe this is a good process that requires nothing from "Xcode-challenged" users and we would like to keep as much as possible of it. So, from a possible solution, we need the following requirements:

1. Be able to install an app on a device
2. Be able to inject an environment variable—only possible when running a process through debugserver
3. Be able to pass launch arguments—only possible when running a process through debugserver

Other considerations:
• On the simulator we are able to load the Detox dylib from any folder on the Mac machine—this is not possible on iOS obviously, so need to be able to somehow send the Detox dylib somewhere accessible within the app sandbox
• In order for the iOS kernel to allow the loading of dylib, it has to be signed with the same codesign identity as the hosting app
• Permissions will not be editable (no Apple~Simulator~DeviceUtils yet :stuck_out_tongue_winking_eye:)

I did a POC to check what is needed in general to run on device. With my own fork of ios-deploy I was able to install a test app on a device, set environment variables before launch and then launch the process with arguments. In my POC, I used a script that took the original .app, extracted the signing information, embedded the Detox dylib inside the bundle and resigned it. This obviously requires the certificate private key, which is not optimal for a testing cloud. So a research has to be done to see how to pass this dylib somehow without modifying the app bundle. Perhaps ios-deploy can do this and I didn't see. If this hurdle is solved, we are in a good shape to have a device driver for real devices.

[rawqing]

It seems that you have a practical solution, but it is troublesome. Could you first provide a manual configuration solution, let us study together, and then consider the adaptation of the cloud test platform? I can't wait to try!
Your contribution is admirable!

[lawrencelomax]

@LeoNatan these are indeed similar constraints to what idb has to work with.

Essentially the constraints about going through a debugserver to get anything done w.r.t app launching on devices holds on iOS Devices.

On the simulator we are able to load the Detox dylib from any folder on the Mac machine—this is not possible on iOS obviously, so need to be able to somehow send the Detox dylib somewhere accessible within the app sandbox

If the detox lib is part of the App bundle, then you are able to dynamically link it, even using preloads like DYLD_INSERT_LIBRARIES, it's essentially a directive to dyld like any linker directive in the main app image.

The regular use-case here is splitting up an application into a smaller main executable that is loaded on launch and then other dylibs for features within an application. This is used to improve launch time so that that only the main dylib needs to be loaded when the app is launched and additional features can be linked only when they are needed.

[LeoNatan]

@lawrencelomax We decided to go that route to ease the setup and potential deployment issues of our users. They don't change their project and the system automagically adds the required Detox code in runtime.

Once we are able to sign applications with Detox for running on device (in order to add the Detox dylib inside the bundle), it will be trivial to modify the signed entitlements to allow debugger attach.

[maxkomarychev]

hey @LeoNatan I was wondering, how big are changes which you are trying to save users from in this

We decided to go that route to ease the setup and potential deployment issues of our users. They don't change their project and the system automagically adds the required Detox code in runtime.

scenario?

There's amazing xcodeproj project which lets you do pretty much anything you want with xcode project.

So is there another problem on they way if we can implement a solution which manipulates over user's project in the cloud w/o bothering a user to configure detox manually and then maintain parallel versions of the app (with and w/o detox libs).

update

user's project in the cloud w/o bothering a user to configure detox manually and then maintain parallel versions of the app

*or actually before uploading to the cloud

[LeoNatan]

@maxkomarychev Yes, I am familiar with that project, thank you. We used to use xcodeproj before settling to the current system. In some ways, it was terrible (not xcodeproj, but our previous solution). Detox would make a change in the user's project; since a lot of our users are ... "Xcode challenged" (to be as politically correct as I possibly can in this late hour), they would not understand the changes, and revert some or make changes of their own. We used to have problems with people complaining about changes. Changes done using xcodeproj were hard to change if we wanted to make a change in Detox. Big mess. We won't be going back to that system, sorry.

In your cloud proposed method, signing would still be an issue. We could sign it with an internal certificate, but that cannot guarantee a 100% functioning app (for example, is an app uses iOS extensions, using application groups and accessing files in the group container, etc.). Also, since Detox does not have a centralized backend, every user that would like to run Detox on a device would require them setting up this cloud on their own. I find it easier to just do everything locally (including signing with their own certificate).

[pvinis]

Is there any work happening on this one?

[LeoNatan]

No work, sorry.
We are working on rewriting large portions of the project, which should in theory make running on hardware device easier.

[pvinis]

Is this happening in open source or is it more of an internal wix effort?

[LeoNatan]

Detox is an open source project, so the changes will be available once ready.

[fjmorant]

Hi,

How is this feature going? Do you have any updates?

Thanks

[LeoNatan]

This is not a priority for us. No updates right now.

[fjmorant]

But it is part of the roadmap

https://github.com/wix/Detox/blob/master/docs/More.Roadmap.md#ios-physical-device-support

is there any plan to make it available at some point?

Thx

[rotemmiz]

Referencing @LeoNatan's answer https://github.com/wix/Detox/issues/95#issuecomment-541563023

This is not a priority for us as we still see Simulators as a good enough tool for our testing. The Detox iOS rewrite will make this potentially easier to support, but we'll still need to put some effort into it as a separate task once the rewrite is done.
Re the roadmap, you are right, but it needs a refresh.

[LeoNatan]

The roadmap document isn't really maintained. As part of the rewrite, we need to go over documents and remove some of the more esoteric ones which we don't actively maintain (such as the roadmap one).

[fjmorant]

Hi @rotemmiz and @LeoNatan ,

Thanks for your replies and for giving me a bit of the context with your current situation, I will keep an eye to the latest releases.

Kind Regards

[LeoNatan]

With the pending rewrite, some of the more peculiar deployment issues should be solved (no need for ios-deploy and ability to easily set an environment variable), but some issues will still remain, such as deploying the Detox dylib on device and signing it correctly so it can be loaded on device. Those aren’t particularly hard to solve but require thinking and perhaps additional requirements from the user. Once we finish the rewrite effort, we can revisit this topic.

[TammyTorres]

@LeoNatan Is not clear for me if till today is possible or not to run on real devices from documentation https://github.com/wix/Detox/blob/master/docs/APIRef.Configuration.md I'm not seeing how to do it

[LeoNatan]

Well, clearly it is not possible, as evident by the discussion here.

[TammyTorres]

thanks! just want to be sure . Is something that would be possible to provide soon? @LeoNatan

[LeoNatan]

Not currently planned, as stated many times before. We are working on overhauling Detox’ internals for iOS, which will make it easier.