Welcome to Emcee project, an ultimate solution for running iOS tests in parallel locally and across many Macs.
Emcee allows you to run UI tests on many physical machines, distributing the work and getting the results of the test run faster. Shared queue manages the order of test execution. Emcee workers execute tests and maintain lifecycle of their simulators automatically. Emcee can generate the Junit and trace reports to make you see how the test run behaved on different machines.
Rich test plans using simple JSON file format
Automatic simulator lifecycle management
Per-test timeouts, simulator settings, environment variables
Single test queue to run tests from multiple parallel pull requests
Prioritized jobs and job groups for different kinds of test runs
Load balancing of worker machines to achieve optimal parallelization performance
On-the-go maintenance of the workers
Integration into existing test management systems via plugins
Easy to use command line interface
Rich test discovery mechanism
Swift Package for using and extending Emcee the way you want
In this guide will demonstrate how to use Emcee. We will use two MacOS machines to run unit and UI tests from a sample project. You can also use a single machine to try out Emcee to see if it works for your project. In this case, a single machine will act as a
queue and a
worker simultaneously. Alternatively, you can scale this guide to as many machines as you have.
If you encounter any issues while proceeding through the guide, please open an issue or reach out via https://t.me/emcee_ios.
You will need to grant SSH access to your machines.
We will be using two machines:
ios-build-machine77will be a worker and a queue - it will provide workers with tests to execute and execute some of those tests.
ios-build-machine78will be a worker - it will only execute tests.
Both machines are set up with a standard non-administrator user
emcee and a
Install Xcode and
sudo xcode-select --switch /Applications/Xcode.app on all of your machines.
We will use
Xcode 13.0 (13A233) and the
iOS 15.0 simulator runtime bundled with this Xcode. If you want to use a specific version of simulator runtime, proceed to
Xcode -> Preferences... -> Components -> Simulators and install the runtime on all the worker machines, where you want the tests to execute with the specific runtime version.
Emcee uses ssh to deploy itself to the machines specified as
workers. Enable SSH in your
System Preferences -> Sharing -> Remote Login. To open this pane execute:
$ open "x-apple.systempreferences:com.apple.preferences.sharing?Services_RemoteLogin"
Now make sure that machines are accessible by ssh. For example:
If your machines are not accessible by DNS, use their IP addresses instead. You can check IP address in
System Preferences -> Sharing. Please note IP addresses may change over time. To open this pane execute:
$ open "x-apple.systempreferences:com.apple.preferences.sharing"
In this step, we will build a sample project that features different types of tests. Xcode and
xcodebuild will produce build artifacts in derived data.
You can run this step from either machine. Clone the sample project:
cd ~ git clone https://github.com/avito-tech/Emcee.git cd Emcee/Samples/EmceeSample
To build the project, create a simulator:
xcrun simctl create '15.0' 'iPhone X' 'iOS15.0'
Now run xcodebuild:
xcodebuild build-for-testing \ -project EmceeSample.xcodeproj \ -destination "platform=iOS Simulator,name=15.0,OS=15.0" \ -scheme AllTests \ -derivedDataPath derivedData
Xcodebuild will place the build products in:
Now that the machines are ready, and the project is built, download Emcee on the same machine where you built the project by running:
curl -L https://github.com/avito-tech/Emcee/releases/download/16.0.0/Emcee -o Emcee && chmod +x Emcee
If you download Emcee using a browser you will need to clear attributes and set the executable bit:
xattr -c Emcee && chmod +x Emcee
With Emcee installed it is finally time to run the tests. The sample project includes 3 test types:
Let's first run tests that don't require a host application. We will be using the
./Emcee runTests \ --queue ssh://emcee:qwerty@ios-build-machine77 \ --worker ssh://emcee:qwerty@ios-build-machine77 \ --worker ssh://emcee:qwerty@ios-build-machine78 \ --device "iPhone X" \ --runtime "15.0" \ --test-bundle derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleTestsWithoutHost.xctest \ --junit tests_without_host_junit.xml
Here is what these options stand for:
--queue- is a URL of a machine that will serve workers with tests
--worker- is a URL of a machine that will execute tests that it queries from the
--runtime- are options that specify which simulators will run the tests
--test-bundle- is a path to the xctest bundle
--junit- is a path to the JUnit xml that will contain the result of the test run
You can find more about all options accepted by
runTests and how to specify them using
Emcee runTests -h.
After the test finishes, Emcee will create a
tests_without_host_junit.xml file. The JUnit report contains four
testcase entries matching the four test methods from the
EmceeSampleTestsWithoutHost.xctest test bundle.
<?xml version="1.0" encoding="UTF-8"?> <testsuites name="xctest" tests="4" failures="1"> <testsuite name="EmceeSampleTestsWithoutHost" tests="4" failures="1"> <testcase classname="EmceeSampleTestsWithoutHost" name="test_0___from_tests_without_host___that_always_succeeds" timestamp="2021-12-29T01:15:16+03:00" time="0.012196063995361328"></testcase> <testcase classname="EmceeSampleTestsWithoutHost" name="test_1___from_tests_without_host___that_always_succeeds" timestamp="2021-12-29T01:15:27+03:00" time="0.02156198024749756"></testcase> <testcase classname="EmceeSampleTestsWithoutHost" name="test_2___from_tests_without_host___that_always_succeeds" timestamp="2021-12-29T01:15:36+03:00" time="0.021990060806274414"></testcase> <testcase classname="EmceeSampleTestsWithoutHost" name="test___from_tests_without_host___that_always_fails" timestamp="2021-12-29T01:15:31+03:00" time="0.1255110502243042"> <failure message="failed - Failure from tests without host">/Users/emcee/Emcee/SampleProject/EmceeSampleTestsWithoutHost/EmceeSampleTestsWithoutHost.swift:17</failure> <failure message="failed - Failure from tests without host">/Users/emcee/Emcee/SampleProject/EmceeSampleTestsWithoutHost/EmceeSampleTestsWithoutHost.swift:17</failure> </testcase> </testsuite> </testsuites>
For a more sophisticated test reporting mechanism such as Allure, check out the Plugins documentation.
Now let's try running tests that require a host application. Host application path is specified using the
--app option. For example:
./Emcee runTests \ --queue ssh://emcee:qwerty@ios-build-machine77 \ --worker ssh://emcee:qwerty@ios-build-machine77 \ --worker ssh://emcee:qwerty@ios-build-machine78 \ --device "iPhone X" \ --runtime "15.0" \ --app derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app \ --test-bundle derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app/PlugIns/EmceeSampleHostedTests.xctest \ --junit tests_with_host_junit.xml
To get a visual confirmation that Emcee is running the tests, you can open the Simulator app on the worker machines:
open "$(xcode-select -p)"/Applications/Simulator.app
Finally, we will run XCUI tests by adding a
--runner option and changing the
--test-bundle option to the XCUI test bundle:
./Emcee runTests \ --queue ssh://emcee:qwerty@ios-build-machine77 \ --worker ssh://emcee:qwerty@ios-build-machine77 \ --worker ssh://emcee:qwerty@ios-build-machine78 \ --device "iPhone X" \ --runtime "15.0" \ --runner derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app \ --app derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app \ --test-bundle derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app/PlugIns/EmceeSampleUITests.xctest \ --junit ui_tests_junit.xml
This is how the test run will look:
Complete documentation is available in our Wiki.
runTests command allows you to get Emcee up and running quickly; however, it doesn't allow for a lot of configuration. On the other hand,
runTestsOnRemoteQueue command allows for fine-grained control of how your tests execute. To get started with
runTestsOnRemoteQueue check out the Queue Server Configuration and Test Arg File wiki pages.
Emcee uses Swift Package Manager for building, testing and exposing the Swift packages.
To start exploring code open
Package.swift in Xcode 13 or execute
make open to generate and open Xcode project.
We are happy to accept your pull requests. If something does not work for you, please let us know by submitting an issue. Read the docs and suggest improvements to them as well!
General commands that help you with a development workflow:
Package.swift file is generated automatically. You must update it before submitting a pull request (run
make package). CI checks will fail if you forget to do so.
brew install libssh2
|Last commit: 4 hours ago|
We've conducted a bunch of interviews with teams who use Emcee. We were provided with nice feedback which we used to improve our product. In this release, we focus on ease of use and an easy first try experience.
It is available right at README.
We introduce a new command -
runTests. A very similar one to a famous
runTestsOnRemoteQueue but it is dedicated to opening the world of simplicity and clear setup for most Emcee users. It has the following syntax and options:
$ Emcee runTests -h
--queue: describes where should a queue be started. This argument may be repeated multiple times, in which case Emcee will use hosts one by one until it manages to start the queue. You must provide at least one value here.
--worker: describes where worker should be started. This argument may be repeated multiple times. It is required to provide at least one value.
--device: Device to run test on, e.g.
iPhone X. Required. Note:
iPhone SEgenerations might be tricky to use!
--runtime: Runtime to run test on, e.g.
--test-bundle: Location of an
.xctestbundle which tests will be executed. Required.
--app: Location of an
.appbundle. Optional. Please refer to our guide to learn more about tests with host app.
--runner: Location of an
--test: Test to execute, e.g.
ClassName/testMethod. Optional. If not specified, Emcee will run all tests. You can specify multiple tests to execute, e.g.
--test Class/test1 --test Class/test2
--retries: How many retries to attempt to run each test successfully. Optional. By default it will retry tests once if failure happens.
--test-timeout: Maximum test execution duration. Optional. Default test duration limit is 180 seconds.
--junit: Path where the Junit report file should be created. Optional.
--trace: Path where the Chrome trace file should be created. Optional.
--worker arguments. These are URLs which describe how to connect to a machine. Currently only SSH is supported. Some examples:
--queue ssh://emcee:firstname.lastname@example.org/Users/emcee/emceequeue.noindex: it means to run Emcee queue on a
queue.example.comhost, by authenticating with
passpassword. The working directory will be located at
--worker ssh://email@example.com/Users/emcee/emceeworker.noindex?custom_rsa: it means the queue will start its worker on
worker.example.com. Worker will use
/Users/emcee/emceeworker.noindexas its working directory. Also, Emcee will use
emceeas username and
~/.ssh/custom_rsakey to auth against a worker host. This key is expected to be present on queue host.
--worker ssh://firstname.lastname@example.org/Users/emcee/emceeworker.noindex#/absolute/path/to/custom_rsa: same as above, but queue will use a key from
Emcee now contains a built-in HTTP server that will cover most of your needs. This significantly simplifies the flow and lowers the entry threshold.
You can pass local paths into
runTests command. Provided local
.xctest bundles will be hosted via a built-in HTTP server transparently, allowing all workers to download artifacts in order to run tests.
You can also provide local paths inside test arg file which you pass into
runTestsOnRemoteQueue, those will also be hosted via the built-in server. Handy!
And finally, you can provide a local queue server configuration file into
runTestsOnRemoteQueue. Now there is no need to upload it to a HTTP server.
You can now get help for any command by typing
-h next to it, e.g.
Emcee -h or
Emcee runTests -h. Sounds like Emcee is ready for such kind of hi-tech.
Should you provide an incorrect test arg file or queue configuration JSON file, Emcee will now print human-readable errors. You will be amazed. Here is one error for your consideration:
Failed to decode value for key "entries" in "test arg file": key "xcTestBundle" not found at entries.buildArtifacts
That is a crystal clear explanation of what went wrong, is it not?
It wasn't clear for some of our users that
jobId field in a test arg file should be unique. Now you can omit it, Emcee will generate a random
jobId for you.
You can now specify how you'd like your tests to be retried if they fail. In your test arg file's
entries there is a new field
testRetryMode. Possible values:
retryThroughQueue — if a test fails on a worker, it will be returned back to the queue. A queue will then retry this test on other workers, up to the specified number of retries. This is the default behavior. This allows the test to be executed on different hosts, potentially eliminating the problem of a broken environment (because every macOS installation in Universe is unique).
retryOnWorker — if the test fails, the same worker will perform retry.
There was a bug (OMG) that resulted in redundant simulator patching (read: apply simulator settings). Now Emcee won't patch simulators if everything is patched.
There is a way to disable worker cache. In queue server configuration, in worker specific settings, next to
numberOfSimulators field there are two more fields:
maximumCacheSize— maximum cache size in bytes.
maximumCacheTTL— maximum TTL for any cached item, in seconds.
You can set
0 these fields, and it will disable cache entirely. This is handy if you want to debug something.
You can now stop workers from removing all test artifacts automatically after a test finishes and plugins terminate. In a test arg file's
entries there is a new field
runnerWasteCleanupPolicy. Possible values:
clean— default value, it means everything will be deleted
keep— all created files will be preserved.
You can now control what logs should be captured during a test run. These logs are obtained from
xcresult bundles after the tests finish. In the test arg file's
entries there is a new field
logCapturingMode. Possible values:
allLogs— all log messages will be captured
onlyCrashLogs— only logs that look like crash logs will be captures
noLogs— no logs will be captured
You can now insert dynamic libraries into a test. In a test arg file's
entries there is a new field
userInsertedLibraries. This is an array of paths. These paths will be passed as
DYLD_INSERT_LIBRARIES env into the test. Combine it with state of the art macOS imaging features, with a fact that tests are being executed via
xcodebuild (so it can expand
__TESTBUNDLE__ and other strings - look at
man xcodebuild.xctestrun), and this will provide you some additional flexibility for your test execution process.
libssh, so you can use it with ease.
Emceebinary from GitHub, do not forget to remove quarantine attribute:
xattr -c Emcee
Full Changelog: https://github.com/avito-tech/Emcee/compare/15.0.0...16.0.0