Swiftpack.co - Package - avito-tech/Emcee

Emcee Banner

Welcome to Emcee project, an ultimate solution for running iOS tests in parallel locally and across many Macs.

Emcee allows you to run UI tests on many physical machines, distributing the work and getting the results of the test run faster. Shared queue manages the order of test execution. Emcee workers execute tests and maintain lifecycle of their simulators automatically. Emcee can generate the Junit and trace reports to make you see how the test run behaved on different machines.

Using Emcee

Up to date documentation is available on Wiki.

Features

  • Rich test plans using simple JSON file format

  • Automatic simulator lifecycle management

  • Per-test timeouts, simulator settings, environment variables

  • Single test queue to run tests from multiple parallel pull requests

  • Prioritized jobs and job groups for different kinds of test runs

  • Load balancing of worker machines to achieve optimal parallelization performance

  • On-the-go maintenance of the workers

  • Integration into existing test management systems via plugins

  • Easy to use command line interface

  • Rich test discovery mechanism

  • Swift Package for using and extending Emcee the way you want

Publications

Getting Around the Code

Emcee uses Swift Package Manager for building, testing and exposing the Swift packages.

To start exploring code open Package.swift in Xcode 11 or execute make open to generate and open Xcode project.

Contributing

We are happy to accept your pull requests. If something does not work for you, please let us know by submitting an issue. Read the docs and suggest improvements to them as well!

General commands that help you with a development workflow:

  • Generating an Xcode project: make open
  • Generating Package.swift based on template and import statements in source code: make gen
  • Building the binary at .build/debug/Emcee: make build
  • Running unit tests: make test

Since Package.swift file is generated automatically, you must update it before submitting a pull request. CI checks will fail if you forget to do so.

Dependencies

libssh2

brew install libssh2

Github

link
Stars: 118

Used By

Total: 0

Releases

Reduce - 2020-08-19 08:30:24

Worker Sharing is ON

By default. Read more about this feature here.

Worker Capabilties

New feature! Allows to schedule tests to workers that do meet specific requirements. Read more about this feature here. Currently only available Xcode versions are exposed, but this can be extended. Submit your ideas.

Kickstarting Silent Workers

New feature! Allows you to send a kickstart command to the queue via REST to make it attempt revive the specific workers. Useful if you kill EmceeWorker process for machine maintenance and then need to get it back to the working state.

Metrics

All metrics now have emcee version in them

Cleaning up

  • Lots of fields in test arg file now optional and have default values. The shortest valid test arg file can be very small now:
{
    "jobId": "jobId",
    "entries": [
    {
        "testsToRun": ["all"],
        "testDestination": {"deviceType": "iPhone X", "runtime": "11.3"},
        "testType": "uiTest",
        "buildArtifacts": {
            "appBundle": "http://example.com/App.zip#MyApp/MyApp.app",
            "runner": "http://example.com/App.zip#Tests/UITests-Runner.app",
            "xcTestBundle": "http://example.com/App.zip#Tests/UITests-Runner.app/PlugIns/UITests.xctest"
        }
    }
    ]
}
  • Made --emcee-version non optional - this is generated automatically when you do make build.

  • Removed --queue-server-destination command line arg from runTestsOnRemoteQueue command. This file has been merged into --queue-server-configuration. By the way, --queue-server-run-configuration became --queue-server-configuration (no stupid run anymore).

  • --job-id, --job-group-id, --priority, --job-group-priority arguments has been removed. They all migrated to a test arg file.

  • Models target has been removed from Swift package.

Fixes

Demangling buffer size (this is related to test discovery) has been increased from 1K to 10K. Should be "enough for everyone". If it fires, let us know, we can implement dynamic buffer size.

Side Project

EmceeAdmin allows you to observe and control your Emcee workers via GUI. The latest release supports matching Emcee workers and TeamCity agents.

Publicity - 2020-07-05 17:16:03

APIs

  • We've exposed EmceeCommunications library, a set of APIs to discover and talk to Emcee queue.

Other

  • enableWorker command correctly enables worker now

  • Emcee cache size is now limited to 20Gb by default. In your plugins, if you use cache, set this value to 20Gb as well.

  • Log files are cleaned every 3 hours instead of cleaning them on each Emcee invocation.

Cold Fix - 2020-06-23 16:33:52

  • Improve log contents for better readability and usefulness

  • Fix partial data parsing from test runner

How It Should Have Been Done - 2020-06-17 17:32:17

Simulator Settings are Working Correctly πŸ˜₯

This feature has been implemented for fbxctest previously, and didn't work for xcodebuild test runner. Now this has changed! Emcee correctly patches simulator plist files, and restart affected daemons to ensure patched settings are applied. There is even a new keyboard setting for you to patch - didShowContinuousPathIntroduction, which allows to skip annoying gesture-based input hint.

Synced Simulator State πŸš‰

Emcee now gets simulator for its actual state instead of keeping it in memory. This allows to avoid some nasty bugs, e.g. when Emcee attempts to shutdown simulator, but the operation takes too long, so Emcee kills xcrun simctl shutdown process in order to proceed, but actually simulator is eventully getting torn down by the Apple mechanics. This led to inconsistent state; not anymore.

New Metrics! 🐎

  • test.between_tests.duration - provides a duration between "test has finished" and "test started" events in the same bucket.
  • simulator.allocation.duration - provides a duration of how long simulator gets prepared to be used for testing
  • test.preflight - provides a duration of how long test runner (e.g. xcodebuild) takes to prepare itself and start running a first test. More precisely, this is a time from when test runner process starts till the time when first test starts.
  • test.postflight - this is a duration between test finish event and the termination of test runner process. Describes situation when test runner process finishes running all tests, but does not terminate, wrapping up its work.

Logging Improvements πŸ“„

  • Queue server has less redundant logs
  • Logs of all launched subprocesses are stored inside ~/Library/Logs/ru.avito.emcee.logs/ like logs of Emcee itself. This is useful for investigation. Emcee cleans all logs that are older than 30 days when it launches, but you may clean up these logs yourself if you don't need them.
  • When running tests using runTestsOnRemoteQueue, Emcee now prints number of enqueued and dequeued tests instead of buckets, as well as what tests each worker is executing. πŸ”₯

Bug Fixing Too 🐞

  • Emcee now accepts streamable test failures from XCTestJSON. This is our secret technology, yet to be announced later. γŠ™οΈ
  • Emcee correctly processed localized xcodebuild output.
  • Silent workers can become alive, and the whole process of silent detection has been simplified.
  • Old log files are deleted in parallel to speed up Emcee launch.
  • Emcee now correctly tracks test timeouts when you use xcodebuild as a test runner.

xcodebuild, disabling & enabling workers - 2020-05-21 19:53:03

xcodebuild

In our environment Emcee was able to perform our complete set of about ~2k unit, ~1.2k application, and ~600 UI tests using pure xcodebuild test runner. This means you can stop providing fbxctest to Emcee and use "testRunnerTool": {"type": "xcodebuild"} in test arg file.

This is still in beta though: errors are being parsed from xcodebuild output, line by line. If error has multiline description, only the first line will be captured by Emcee. We are aiming to fix this in future releases of Emcee.

If you use xcodebuild with Emcee, you have to use insideUserLibrary location for simulators.

Enabling and Disabling Workers on the Fly

Emcee now has two new commands that allow you to disable specified worker, and to re-enable it later. This is useful for maintenance purposes.

$ Emcee disableWorker --worker-id "worker-machine-01" --queue-server "queue-host:41000"
$ Emcee enableWorker --worker-id "worker-machine-01" --queue-server "queue-host:41000"

There is also a corresponding REST handlers which queue server accepts.

Generation of Package.swift File

Package.swift for highly modularized project quickly becomes a mess. Now this file is generated when you invoke make open and make gen. New changes to it's contents should be applied to Package.swift.template.

Xcode 11.4

can finally be used to develop Emcee.

Race condition fix - 2020-05-12 07:01:24

After performing test discovery, Emcee issues a request to store the obtained result into remote cache. Previously it didn't wait for it to complete. Sometimes this led to request not being completed. Now Emcee waits for request to finish.

Move Over from the fbsimctl Needle to the simctl Face - 2020-04-29 08:21:51

Runtime Dump in Application Test Mode without Booting Simulator

Now Emcee allows you to speed up application test-like runtime dump. Previously, Emcee would boot the simulator and then launch your xctest bundle with fake test. While this mechanism is flexible, it is slow.

Now Emcee allows to perform runtime dump but without booting the simulator.

  • Use new runtimeExecutableLaunch test discovery mode when specifying xcTestBundle build artifact in test arg file. You also must specify appBundle.
  • Emcee will start start your app without booting simulator (this is magic!). main() will be invoked as usual.
  • Inside your main() you should check for the environments:
    • if EMCEE_XCTEST_BUNDLE_PATH is provided, the value will contain a path to xctest bundle on local file system.
    • EMCEE_RUNTIME_TESTS_EXPORT_PATH will contain a path where to write out the resulting JSON with runtime dump, this is as usual
    • Either both envs will be provided, or both will be missing.
  • If you detect EMCEE_XCTEST_BUNDLE_PATH, you can then load it using let bundle = Bundle(path: xctestBundlePath); bundle.load(). Then you'll be able to access your bundle's principal class, and perform the runtime dump as usual, and export it to EMCEE_RUNTIME_TESTS_EXPORT_PATH.

To use new way of discovering tests, pass testDiscoveryMode: "runtimeExecutableLaunch".

Xcode 11.4 Support

Emcee is now buildable in Xcode 11.4. This required some dancing around Ruby-lang based technologies like brewery@home, and some bugs in SPM. More in this thread.

Define the Location where Simulators should be Created

Emcee now allows to specify where to create simulators: either in its own private folder like it has been doing up to this point (insideEmceeTempFolder) or in default shared location ~/Library/Developer/CoreSimulator/Devices (insideUserLibrary). The latter allows to use the simulators with tools like xcodebuild and Xcode.app. This is required in order to start running tests using xcodebuild (which is still wasn't implmeneted though).

{
    "simulatorControlTool": {
        "location": "insideEmceeTempFolder",
        "tool": {
            "toolType": "fbsimctl",
            "location": "http://example.com/fbsimctl.zip"
        }
    }
}

As of now, you can safely use insideEmceeTempFolder to keep things unchanged.

simctl

Emcee had support for simctl for a while now, but now it has been tested and enabled for all builds in Avito. No problems so far, and you can switch to it as well. One less dependency to run the tests! In your arg file:

{
    "simulatorControlTool": {
        "location": "insideEmceeTempFolder",
        "tool": {
            "toolType": "simctl"
        }
    }
}

Speed for Need - 2020-04-09 13:19:24

Metrics

  • Queue server now reports notRegistered worker state for those workers that were expected to register with queue but haven't done so.

Job Grouping

  • --run-id argument has been renamed to --job-id

  • Emcee allows to group jobs. runTestsOnRemoteQueue accepts new optional arguments: --job-group-id and --job-group-priority. Test queue sorts job groups by: job group priority, job group creation time (for job groups with same priority); jobs are sorted by priority and by job creation time (for jobs with same priority) within their groups.

Test Discovery

Runtime dump feature is now part (and implementation detail) of Test Discovery feature. Test arg file has been updated: runtimeDumpMode became testDiscoveryMode, logicTest and appTest were renamed to runtimeLogicTest and runtimeAppTest correspondingly, allProvidedByRuntimeDump became allDiscoveredTests.

Quicker but Unsafer Test Discovery

Emcee now has a new way of discovering tests. When you pass parseFunctionSymbols as a value of testDiscoveryMode in test arg file, Emcee will run nm -j -U on provided .xctest bundle, demangle all symbols using libswiftDemangle.dylib, and then find all @objc .test...() methods (throwable and not, with () return types).

This approach is much quicker than performing a runtime dump. This discovery method does not require any test source code modification.

There are some risks though. Emcee may find excess tests, or may miss some tests. Although in Avito we checked this for our 50 unit test bundles, we still might miss something, so use with care. Also, the lack of runtime support makes this test discovery method less flexible: Emcee won't be able to locate tests that you might be generating dynamically in runtime; Emcee won't export any data, e.g. tags, if you attach them to your tests. For those kind of things you must still use logic runtime dump (runtimeLogicTest) which is quite fast, or runtime with a host app (runtimeAppTest) and by defining NSPrincipalClass in your test bundle plists.

If you experience any inconsistency between parseFunctionSymbols and runtime dump based test discovery, please file an issue.

Totally Makes Sense - 2020-03-27 18:50:27

This release adds new features and tunes existing behavior. The changes totally make sense.

Automatic Simulator Deletion

This new feature allows to enable the following behavior: when simulators is automatically shutdown after being idle, and stays idle for a given period of time, Emcee will delete it to free up disk space. [commit]

Removal of Blocking Functionality

  • Previously, Emcee workers reported to the queue their state with buckets they are executing. Now Emcee worker implemented REST, and the queue polls its workers instead. [commit]

  • Emcee had a feature: if worker would switch from executing a bucket of tests to another bucket without sending back results in time, Emcee would consider these buckets as 'stuck' and re-schedule tests back to the queue; but then, if Emcee worker would send bucket results back to the queue, Emcee queue would decide that worker behaves incorrectly and block it since it sends back unexpected results. This blocking functionality has been removed. [commit]

Bug Fix & Clean Up - 2020-03-27 18:37:17

  • Removed some models from Swift codebase

  • Fixed a bug: queue has been starting workers twice, once for each deployed file. Now queue starts workers only once, when all files has been deployed to the worker machines. [commit]

  • Emcee now expects a --version argument to be passed. This is rather a controversial way of versioning Emcee binary. If you update Emcee, you should update the value of this argument as well. [commit]

  • Emcee does not create new simulators when you update fbxctest. [commit]

  • Fixed a bug that would sometimes result to a corrupted cache state. When unzipping the downloaded artifacts, Emcee now will use temporary folder, and then move extracted data to a final location. [commit]

  • SimulatorVideoRecorder class has been updated to work with Xcode 11. [commit]

  • Emcee now passes TMPDIR environment to simulator control tool (e.g. fbsimctl) [commit]

Runtime Dump Cache, Simulator Timeout & Automatic Shutdown Configuration - 2020-02-21 10:36:55

Runtime Dump Cache

Emcee can now store results of runtime dump in a shared writable cache and re-use them to speed up runtime dumps on the same input. There are some requirements for this to work correctly:

  • You must pass an additional config to support shared cache via --remote-cache-config arg. You can read more about its contents here.

  • URL is used as a key to runtime dump cache entries. Thus, if you change the contents of xctest bundle behind some URL and attempt to dump it again, Emcee will re-use cached entry if it is available. This may lead to wrong or unexpected runtime dump results.

  • Runtime dump should output exactly the same results regardless of execution environment or kind. iOS runtime, runtime dump kind, time of the day, or any environment variable values should not affect the result of runtime dump.

Simulator Timeout Configuration & Automatic Shutdown

Test arg file now expects to have simulatorOperationTimeouts object provided. This object allows you to control timeouts for various simulator operations like creating, booting, shutting down and deleting, as well as automatic shutdown. The latter allows you to specify the time after which idle simulators will be shutdown.

Shutdown simulators allow them to be booted faster than booting a newly created simulator, but reduces RAM, swap, and max_proc/max_files system resources utilization. This is useful for CI where you might be running tests on various iOS versions. Updating Emcee is also safer now because the amount of running simulators now decrease when simulators are idle.

Models Target has been Refactored

Models target previously contained a lot of models used by the Emcee. Now most of them moved to corresponding modules, like SimulatorPoolModels, RunnerModels, TestArgFile, and so on. Emcee exposes all model targets as EmceeInterfaces product for convenience.

Stability Increase

Emcee workers are much stable and do not crash when fetching contents of URLs (fix FYI)

simctl, Plugins Lifecycle - 2020-02-21 09:33:09

simctl

Emcee now correctly uses xcrun simctl to control simulators, making it unnecessary to provide a location of fbsimctl tool.

To use xcrun simctl, pass the following values via test arg file:

    "toolResources": {
        "simulatorControlTool": {
            "toolType": "simctl"
        },
    }

To switch between Xcode versions, provide developerDir value using toolchainConfiguration field in test arg file.

You can still use fbsimctl if you prefer it more. fbxctest tool is still required to run tests, support for xcrun xcoebuild is still in WIP status.

Plugins

Implemented plugin lifecycle. Previously, Emcee would start plugins on both server and workers, and you defined plugins as part of queue server run configuration. Now this has changed:

  • You define plugins for a job using test arg file. This allows you to update plugins easily.
  • Plugins are started before executing tests, and terminated afterwards.
  • Plugins are expected to start within 30 seconds. Previously the timeout was 170 seconds (for unknown reason)

Bug fix

Fixed a nasty bug. Sometimes, when workers fetch a bucket for execution and then fail to execute it, they won't return it back to the queue. This would look like some worker took a bucket and keep executing it infinitely.

Now, if failure happens during bucket execution, worker will report back a test failure with a description of what happened.

Feature

Junit reports now have hostname and timestamp fields set for tests.

- 2020-01-09 12:22:43

Simulator settings are now passed as data structures via test arg file instead of URLs to some JSON file with not very obvious contents.

These are the new models: