AWS Device Farm – iOS Learnings with XC UI Testing

I’ve been playing with AWS Device Farm for the last few days. It’s a neat little service provided by Amazon, especially for those who are interested in automated testing and continuous testing. They make it really easy to go ahead and automate your tests against a number of different device / OS flavors. While useful on iOS, this becomes especially useful on Android where different devices and manufactured have a very wide range of possible configurations.

But this post isn’t so much to detail the usefulness of the tool – it probably speaks for itself. It’s written with the intent of surfacing a few gotchas in setting up and executing tests on AWS Device Farm that kind of slip through the cracks with Amazon’s official (and sometimes difficult to surface) documentation / blogs.

Generating a Test Runner App – Fiddling with your Build Schemes

First thing you’ll need to do, and hopefully it’ll save a lot of headache for you, is to setup your Xcode project settings so that it actually builds and exports your Test-Runner.app that you use for your XC UI Tests:

  1. Go to Product -> Scheme -> Edit Scheme
  2. Click on Build in the left nav tab
  3. Select your XCTests and XC UI Tests targets and enable then for ‘Run’ as below
    1. XCTests and XCUITEsts
    2. I won’t go into detail about how you generate XCTest and XCTest UI targets – you can find plenty of stuff elsewhere on that. But what is useful to know here is that by default, just running your program / build won’t auto generate the .app files that you’re looking for.

Generating your .IPA Files for your App + TestRunner App

There are a bunch of ways to generate a .IPA file for your app. One of the earliest mistakes I made was to go ahead and do the normal Build -> Archive -> Export IPA flow. This takes FOREVER. So digging deeper into the documentation on Amazon, I found that they expect a different flow.

  1. Change your Build Scheme to ‘Generic Device’
    1. Generic Device
  2. Go to Product -> Build For -> Running
    1. Build For Running
  3. Great! Now go to where your products are stored. The easiest would be to select your .app file from the Products folder in the file hierarchy and then view it under Identity and Type in the file inspector. Click on the little arrow under Full Path and it’ll take you right there.
    1. File and Type Identify Inspector
  4. Here you’ll see the two .app files of interest.
    1. File Runner!
  5. Okay, now what about your IPA files? You have .App files! Well according to AWS Device Farm, this is what you do:
    1. Create a folder named Payload
    2. Drag and drop the .app file (either the application if you’re uploading the application, or the test runner if you’re uploading an XC UI Test) into the Payload folder
    3. Zip the folder
    4. Change the extension to .ipa and confirm the change

Fin

The rest is pretty straightforward. Basically you’ll just go ahead and follow the wizard walkthrough and upload things as need be. Hopefully you found this bit pretty helpful. If you’re still running into random issues – check out their forums and blog posts listed below!

Useful Resources

https://mobile.awsblog.com/post/TxE7RHRA4JF4B/Test-User-Interfaces-in-iOS-Apps-with-XCTest-UI-and-AWS-Device-Farm

http://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest-ui.html#test-types-ios-xctest-ui-prepare

https://forums.aws.amazon.com/index.jspa?categoryID=1