Swiftpack.co - Swift Packages by avito-tech

The markdown parsing is broken/disabled for release notes. Sorry about that, I'm chasing the source of a crash that's been bringing this website down for the last couple of days. # What's Changed ## Crossplatform - Meet test history service! It tracks test results and time consumption for individual tests and allows other services to use this information to optimally schedule new test run. See more in doc ([iOS](https://github.com/avito-tech/Emcee/wiki/Test-History), [Android](https://docs.emcee.cloud/on-premise/android/queue_configuration/#testhistoryservice)) - There is a new `timeWeighted` test splitter that works hand in hand with test history service and provides the shortest possible time for entire test run. See more in doc ([iOS](https://github.com/avito-tech/Emcee/tree/master/Sources/ScheduleStrategy#time-weighted), [Android](https://docs.emcee.cloud/on-premise/android/launch_tests/)) ## iOS - Minimal supported target is macOS 12.0 - Get chrome tracing for the entire test run by using `--trace` option. See more in `emceeFree runTests --help` ## Android - Added readable Artifactory uploading error description - Fixed possible crash in clients because of redundant internal checks - Fixed possible zero test time bug - Added check for desired sdk version before test run start - Added fix for endless CLI process after test run - Updated internal emulators version to 33.0.2. Other internal tools were also updated - Added timeout for emulator boot complete - Fixed wrong test result when test ended instantly # What's Changed ## iOS - By default buckets divide strategy is `equally divided` - By default tests restart strategy is `retry on worker` - By default queue is watching for workers and restart them if needed. See `silentWorkerAutomaticRestartFeatureConfig` - Simulator setup before tests run is optimized. Now it requires less time - Emcee now writes less logs by default. The log level can be configured by environment - Improved plugins errors handling in workers. Memory consumption for plugin support is reduced - Added Emcee metrics Grafana dashboard example ## Android - Improved test run result info output for clients (Gradle plugin and CLI) - Added ability to set total tests execution timeout for clients (Gradle plugin and CLI) - Worker now starts on 41001 port (was 80) - Added Android metrics Grafana dashboard example - Documentation was improved - Android documentation now on the [separate site](https://docs.emcee.cloud/on-premise/android/intro/) - Added ability to configure Queue analytics and metrics for containerized Queue - Fixed bug with Allure report downloading on sdk level 29 and lower - Fixed test start time event # What's Changed Common: - Added support for skipped tests on all platforms iOS: - There is new [tests discoverer](https://github.com/avito-tech/Emcee/wiki/Test-Discovery#runtimexctest-test-discovery-mode) - `runtimeXCTest`. Use it for discovering dynamically added tests with frameworks like Quick, Kiwi, etc - `scheduleTests` command dispatches tests to previously deployed queue. It's useful in systems where Emcee gets deployed with external platform operator - default file verbosity has been changed to debug (was trace before change) - better device management and improved cleaning routines that lead to reduced disk and memory consumption Android: - Another option to run tests - command line utility. [Give it a try!](https://github.com/avito-tech/Emcee/wiki/Android-documentation#%D0%B7%D0%B0%D0%BF%D1%83%D1%81%D0%BA-%D1%82%D0%B5%D1%81%D1%82%D0%BE%D0%B2-%D1%81-%D0%BF%D0%BE%D0%BC%D0%BE%D1%89%D1%8C%D1%8E-cli) - Get video recorded during test run by using `videoRecordStrategy` - Documentation refined Cloud: - Integrate Emcee test runner to your GitHub pipeline with [Emcee.cloud github action](https://github.com/marketplace/actions/emcee-cloud-action) - Now Emcee offers alpha support for Appium cloud web server platform! Try it out - https://docs.emcee.cloud/cloud/appium/ Minor fixes: - `retryOnWorker` tests retry mode does not cause excessive retries on queue ## Announcement Since version 20.0.0 Emcee we will change a minor version for ongoing regular releases. Major versions will be increased with significant changes or broad refactoring. We are excited to announce that Emcee now supports the Android platform. Run Emcee queue and workers from container images in Docker or your Kubernetes cluster to shorten testing time. Keep calm, we continue to develop iOS part as well. # Android Android workers, as well as their iOS counterparts, receive test jobs from queue. It means you get smart bucket slicing strategies, test retries, and better hardware utilization out of the box. - Workers ship Android runtimes on board, on demand start and manage necessary devices with required sdk version. As for now we support sdk levels 24, 27, 29, 31 and will add more on a regular basis. - It's possible to configure Android worker to send rich analytics. Use these insights when making decisions on equipment purchase or problem investigation. [Try out Android workers now!](https://github.com/avito-tech/Emcee/wiki/Android-documentation) # iOS - Xcode test plans are now supported. Build your project with test plan support `xcodebuild build-for-testing -testPlan <name>` and then feed the resulting xctestrun file to Emcee via `--xctestrun`. See more in `Emcee runTests --help`. - Set http/https proxies through command line invocation or json config. - Test discoverer now finds tests with swift concurrency. - Minor improvements: fixed problems when Emcee might lose some analytics events, worker does better dynamic libraries lookup. 18th release is concentrated on fixing bugs, speeding up things, and separating the functionality between features which become available after purchasing a licence and always-free version. # Emcee now has paid version Starting from version 17, Emcee is distributed as-is as a closed source project and available in a form of prebuilt binaries. Version 18 goes beyond this and introduces two distinct products: free Emcee and paid Emcee. Free version (available as a download on Github) allows you to test the product with up to 3 workers, however, you'll need to buy licenses for some richer functionality like: - Allowing to use more than 3 workers - For a technical support - For EmceeAdmin GUI app - For getting things done that you want to see in Emcee sooner than later. Main functionality remains available in free version: - For using plugins in your environment - For using analytics, such as grafana, statsd, and kibana - For a bit richer test discovery flows, such as discovering via runtime ## Some notes about the current Github repository The current source code has been frozen from April 2022, which was version 16. We will keep it available as is, and we're not planning on removing it. All made forks will still be present obviously as well, as we can't control them. Version 17 is closed source, yet fully functional. You can use it without any limitations as well. We haven't developed any plans on upstreaming changes from v17 into open source yet, however, we are considering it. ## Automatic Job Deletion This is huge one and helpful. Emcee now tracks if job is needed at all, and automatically deletes it if it is not needed to run anymore. This helps on CI: when you cancel the test run, job gets deleted automatically. Tech note: previously Emcee client process issued a `/jobDelete` request which sometimes could get lost; now queue itself tracks all jobs. ## Worker Resurrection Emcee now attempts to restart its workers in case if they die. So if you decided to reboot your machine, or if it kernel panics, it will become a worker automatically. ## Simulator Cloning Emcee worker can now clone simulators instead of booting a new one. This speeds up creation quite a bit, and saves disk space. Emcee will clone any previously shut down simulator it booted. If Emcee never booted suitable simulator, it will create one instead. We advise to use smaller values of `automaticSimulatorShutdown` and `automaticSimulatorDelete` settings, like 120 seconds, to have better effect of simulator cloning. As an example, booting a new simulator can take 1-2 minutes easily, but cloning and booting takes around 40 seconds. ## Plugins are back! You can use the new, a little simplified plugin SDK which is available here: https://github.com/avito-tech/EmceePluginSupport It's easier to wire up everything, as there are only two exposed targets now: EmceePlugin and EmceePluginModels. Boilerplate code has been reduced just to: ``` let plugin = try Plugin { (event: PluginAppleTestEvent) in // process an event } plugin.join() ``` ## Bugs and small features and changes - Emcee now uses socket API to talk to graphite and statsd instead of state of the art OutputStream APIs. Users complained about crashes related to graphite, and this change should help. - You can query global queue state via `/globalQueueState`. Maybe this will be helpful for you to debug how queue is being formed when you submit your jobs. - Worker ID is exposed as capability now, allowing you to pin your tests to a specific machine. Capability name is `emcee.workerid` and the value is actually worker ID. - `--device` and `--runtime` arguments now accept only fully qualified Core Simulator IDs, e.g. `com.apple.CoreSimulator.SimRuntime.iOS-16-4` and `com.apple.CoreSimulator.SimDeviceType.iPhone-14`. Please refer to `xcrun simctl list` for more examples. - If you want to set environment variables for a test or app while using `runTests` command, set them in the calling environment with a `EMCEE_` prefix. - CPU architecture is now exposed as worker capability, allowing you to pin your tests to the machine with specific CPU arch. The key is `emcee.cpu.architecture`, the value is CPU arch returned by `uname -a`, e.g. `arm64`. - `testFinished` event is now correctly delivered to plugins when tests times out. - If plugin fails to start, error message will contain its standard output and error for debugging purposes. - Emcee now deletes its temporary xcresult storage after test run finishes. - You can now specify range of ports that workers can utilize. Queue server config has a new field `workerPortRange` which controls exactly that. You can also set queue port range via `queuePortRange`. - New command `discoverAppleTests` lets you run test discovery much easier than before, without composing a test arg file - Speaking of test arg files, they are gone from a free version of Emcee. We decided to keep them for paid release only, as they provide more flexibility. - We are happy to say Version 18 supports macOS Ventura. Version 17 does not support Ventura. ## Join our Telegram channels! A decent place to discuss knowledge about Emcee and obtain a licence! - Russian: https://t.me/emcee_ios - English: https://t.me/emcee_ios_en # Preambula This release brings long awaited feature - generating a resulting `xcresult` file which you can open in Xcode right after running your tests. Just use `runTests` command and provide `--result-bundle <path>` to generate your result bundle. You can also control the lifetime of `xcresult` attachments via `TestArgFile.entries.testAttachmentLifetime` - it can be `deleteOnSuccess`, `keepAlways`, `keepNever`. Just like `xcodebuild`, huh! ## Caveats - This release **DOES NOT SUPPORT PLUG-INS**. You won't be able to compile them against Emcee v17. Please wait until we release further update for that! - Worker sharing is not working between this Emcee version and previous versions. This is because we messed up key and value for HTTP field: it used to be `application/json: Content-Type` which swift-nio thing forcefully rejects; now it is set correctly to `Content-Type: application/json`. ## Smaller Details - Less memory leaks and more speed thanks to updating HTTP server to Vapor - Less logs are generated! Again! - Emcee now kills `xcodebuild` with more suitable signal `SIGINT` which leads to less result bundle corruptions. - `libssh` freedom has arrived! Emcee does not require it to be installed on your system. Also, it is adviced to use key-based authentication. - We got rid of duplicate errors from junit. We got rid of duplicate errors from junit. - Help (e.g. `Emcee runTests -h`) got even better thanks to more information provided. - You can now configure simulator count on each worker, locale, keyboard, language via `runTests` command. Please refer to `Emcee runTests -h`. ## Join our Telegram channels! A decent place to discuss and share your knowledge about Emcee! - Russian: https://t.me/emcee_ios - English: https://t.me/emcee_ios_en We've conducted a bunch of interviews with teams who use Emcee. We were provided with nice feedback which we used to improve our product. In this release, we focus on ease of use and an easy first try experience. # Getting Started Guide with Pictures and Sample Project 🥩 It is available right at [README](https://github.com/avito-tech/Emcee#getting-started). # Just Run These Tests 🤬 We introduce a new command - `runTests`. A very similar one to a famous `runTestsOnRemoteQueue` but it is dedicated to opening the world of simplicity and clear setup for most Emcee users. It has the following syntax and options: `$ Emcee runTests -h` * `--queue`: describes where should a queue be started. This argument may be repeated multiple times, in which case Emcee will use hosts one by one until it manages to start the queue. You must provide at least one value here. * `--worker`: describes where worker should be started. This argument may be repeated multiple times. It is required to provide at least one value. * `--device`: Device to run test on, e.g. `iPhone X`. Required. Note: `iPhone SE` generations might be tricky to use! * `--runtime`: Runtime to run test on, e.g. `15.0`. Required. * `--test-bundle`: Location of an `.xctest` bundle which tests will be executed. Required. * `--app`: Location of an `.app` bundle. Optional. Please refer to [our guide](https://github.com/avito-tech/Emcee#tests-with-a-host-application-) to learn more about tests with host app. * `--runner`: Location of an `XX-Runner.app`. Optional. * `--test`: Test to execute, e.g. `ClassName/testMethod`. Optional. If not specified, Emcee will run all tests. You can specify multiple tests to execute, e.g. `--test Class/test1 --test Class/test2` * `--retries`: How many retries to attempt to run each test successfully. Optional. By default it will retry tests once if failure happens. * `--test-timeout`: Maximum test execution duration. Optional. Default test duration limit is 180 seconds. * `--junit`: Path where the Junit report file should be created. Optional. * `--trace`: Path where the Chrome trace file should be created. Optional. Let's explain `--queue` and `--worker` arguments. These are URLs which describe how to connect to a machine. Currently only SSH is supported. Some examples: - `--queue ssh://emcee:[email protected]/Users/emcee/emceequeue.noindex`: it means to run Emcee queue on a `queue.example.com` host, by authenticating with `emcee` username and `pass` password. The working directory will be located at `/Users/emcee/emceequeue.noindex`. - `--worker ssh://[email protected]/Users/emcee/emceeworker.noindex?custom_rsa`: it means the queue will start its worker on `worker.example.com`. Worker will use `/Users/emcee/emceeworker.noindex` as its working directory. Also, Emcee will use `emcee` as username and `~/.ssh/custom_rsa` key to auth against a worker host. This key is expected to be present on queue host. - `--worker ssh://[email protected]/Users/emcee/emceeworker.noindex#/absolute/path/to/custom_rsa`: same as above, but queue will use a key from `/absolute/path/to/custom_rsa`. # Free HTTP Server for Everyone! 🌏 Emcee now contains a built-in HTTP server that will cover most of your needs. This significantly simplifies the flow and lowers the entry threshold. You can pass local paths into `runTests` command. Provided local `.app` and `.xctest` bundles will be hosted via a built-in HTTP server transparently, allowing all workers to download artifacts in order to run tests. You can also provide local paths inside test arg file which you pass into `runTestsOnRemoteQueue`, those will also be hosted via the built-in server. Handy! And finally, you can provide a local queue server configuration file into `runTestsOnRemoteQueue`. Now there is no need to upload it to a HTTP server. # Other Goodies 🦚 ## Help me Dr. `-h` 🍡 You can now get help for any command by typing `-h` next to it, e.g. `Emcee -h` or `Emcee runTests -h`. Sounds like Emcee is ready for such kind of hi-tech. ## Better errors 💢 Should you provide an incorrect test arg file or queue configuration JSON file, Emcee will now print human-readable errors. You will be amazed. Here is one error for your consideration: ``` Failed to decode value for key "entries" in "test arg file": key "xcTestBundle" not found at entries[0].buildArtifacts ``` That is a crystal clear explanation of what went wrong, is it not? ## Optional `jobId` ☃️ It wasn't clear for some of our users that `jobId` field in a test arg file should be unique. Now you can omit it, Emcee will generate a random `jobId` for you. ## Test Retry Mode 👯‍♀️ You can now specify how you'd like your tests to be retried if they fail. In your test arg file's `entries` there is a new field `testRetryMode`. Possible values: - `retryThroughQueue` — if a test fails on a worker, it will be returned back to the queue. A queue will then retry this test on other workers, up to the specified number of retries. This is the default behavior. This allows the test to be executed on different hosts, potentially eliminating the problem of a broken environment (because every macOS installation in Universe is unique). - `retryOnWorker` — if the test fails, the same worker will perform retry. ## Simulator Patching is Better 📲 There was a bug (OMG) that resulted in redundant simulator patching (read: apply simulator settings). Now Emcee won't patch simulators if everything is patched. ## Cacheless Workers are Now Possible 💳 There is a way to disable worker cache. In queue server configuration, in worker specific settings, next to `numberOfSimulators` field there are two more fields: - `maximumCacheSize` — maximum cache size in bytes. - `maximumCacheTTL` — maximum TTL for any cached item, in seconds. You can set `0` these fields, and it will disable cache entirely. This is handy if you want to debug something. ## Prevent Emcee from Cleaning Runner Output 🏃 You can now stop workers from removing all test artifacts automatically after a test finishes and plugins terminate. In a test arg file's `entries` there is a new field `runnerWasteCleanupPolicy`. Possible values: - `clean` — default value, it means everything will be deleted - `keep` — all created files will be preserved. ## Log Capturing Mode 📝 You can now control what logs should be captured during a test run. These logs are obtained from `xcresult` bundles after the tests finish. In the test arg file's `entries` there is a new field `logCapturingMode`. Possible values: - `allLogs` — all log messages will be captured - `onlyCrashLogs` — only logs that look like crash logs will be captures - `noLogs` — no logs will be captured ## Where without `dyld` You can now insert dynamic libraries into a test. In a test arg file's `entries` there is a new field `userInsertedLibraries`. This is an array of paths. These paths will be passed as `DYLD_INSERT_LIBRARIES` env into the test. Combine it with state of the art macOS imaging features, with a fact that tests are being executed via `xcodebuild` (so it can expand `__TESTBUNDLE__` and other strings - look at `man xcodebuild.xctestrun`), and this will provide you some additional flexibility for your test execution process. # Try it Today! 🧨 - Below is a universal binary (arm64 & x64) for your convenience! It even has built-in `libssh`, so you can use it with ease. - If you download `Emcee` binary from GitHub, do not forget to remove quarantine attribute: `xattr -c Emcee` - Please rebuild your plugins. v15 API is not compatible with v16. - - - **Full Changelog**: https://github.com/avito-tech/Emcee/compare/15.0.0...16.0.0 Today we celebrate a bit delayed Emcee v15 release 🥳 # Data Centre Edition 🦚 Emcee queues can be run on multiple hosts now, allowing even more graceful degradation when necessary. For example, if you set your queue host on maintenance (turn it off, or if aliens destroy your machine), queue may now be started on a spare host(s), allowing you to keep running your tests. To use this, just provide multiple queue hosts in `queue_config.json` via `queueServerDeploymentDestinations` array. Emcee iterate over the hosts in provided order. # Simulator Settings 📱 - You can now import *certificates* 📜 into simulator keychain! Thanks to @EvgenyIv96 for implementing this feature. - `xcodebuild` now deletes all attachments 📎 from resulting `xcresult` bundle if test was successful ✅, reducing a load on the workers. Thanks again to @EvgenyIv96 for implementing this. - New iOS 15 keyboard gesture banner thing ⌨️ can be hidden by setting `didShowContinuousPathIntroduction` to `true` in your test arg file. # It Just Became Much Better Now 📈 - Emcee now parses `xcresult` files and extracts test failures from there in addition to parsing result stream. This improves test error capturing significantly. 🌡 - Progressive schedule strategy has been optimized for smaller worker count. 👷🏻‍♀️👩🏼‍🌾👩🏽‍🏭👩‍🚀 - New schedule strategy with **fixed bucket size** allows to (obvsly) set a fixed bucket size. This is useful for larger worker count. E.g. we use this strategy in Avito with our 80+ workers. 🖥 - New `Emcee initTestArgFile` command creates a sample test arg file which you can fill with your settings. Useful for quick-start! 🥩 - New `Emcee initQueueServerConfig` command creates a sample queue config. Again, useful for initial setup. 🌱 - Swift symbol demangling now uses dynamic buffer. This eliminates troubles with test discovery for some users of Emcee. 🦅 - Finally, as a reminder, Emcee does not require `sudo`. You can create a standard user with limited rights and make Emcee use it. 🔒 For this scenario, Emcee now checks if `~/L/D/CoreSimulator` folder exists, and creates it if it is missing. # Meet prebuilt binary! 🤡 Just don't forget to install `libssh2` via excellent package manager which uses chemistry, water supply, liquid manipulation, and drink-related domain terms to describe its all possible actions and processes. ☢️ Also, do not forget about clearly understandable fact that Apple Silicon chips have special destination path for installable things in that honorable package manager, and that it is different on Intel chips. Thus, it is unlikely you will be able to use Intel binary on Apple Silicon Mac, which is very due to extreme quality of **any** modern software. 💎 Please render this fine emoji in red in your mind. - **Full Changelog**: https://github.com/avito-tech/Emcee/compare/14.0.0...15.0.0 # Humaney - If simulators fail to boot, there will be a better, a more human readable error about this. This came as a replacement for "xcrun process failed with exit code 69". # `xcresult`-eys into Plugins - @venigreat implemented a feature that allows to access `xcresult` bundles from plugins, allowing to process them on the go. `TestContext.environment` now contains a new entry which points to a `xcresult` bundle. You can and should use [`XcodebuildTestRunnerConstants.envXcresultPath`](https://github.com/avito-tech/Emcee/blob/master/Sources/XcodebuildTestRunnerConstants/XcodebuildTestRunnerConstants.swift) to get the name of that env. - By the way, there is an [open source Allure plugin for Emcee](https://github.com/TinkoffCreditSystems/AllurePlugin.emceeplugin/) # Securitey - @venigreat implemented the following feature: Emcee now allows to access SSH by using private key files. Previously it only used password based auth. You specify auth mechanism via queue configuration. - @venigreat implemented a third feature: Emcee now can add arbitrary HTTP headers to its network requests. This is useful if your storage (e.g. artifactory, nexus, etc.) requires authentication via headers, or if your back-end wants to track its clients via some `X-Fancy-Header` value. In this release, we focused on adding support for distributed logging system, and we used Kibana for these purposes. Thus, we decided to document how Emcee logging system works, and provide in-depth explanation of what metrics Emcee reports. # Test Arg File Changes and More Flexible Analytics ## Global Analytics Emcee now respects two analytics configurations. One is a **global** one. It gets captured by the **queue** when it starts ([`queueServerConfiguration.globalAnalyticsConfiguration`](https://github.com/avito-tech/Emcee/wiki/Queue-Server-Configuration#globalAnalyticsConfiguration)), and it is used to report some **global events** which do not relate to any specific job. Some examples are: - [Status of worker sharing](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueueworkersutilizablecountemcee_versionqueue_hostreservedreservedreserved) feature - [Individual worker statuses](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueueworkerstatusworker_idstatusemcee_versionreserved), as seen by the queue (e.g. alive, silent, never started) - Global number of [enqueued](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuestateenqueuedqueue_hostemcee_versionreservedreservedreserved) and [dequeued](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuestatedequeuedqueue_hostemcee_versionreservedreservedreserved) buckets and tests - [Global counter](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuejobscountqueue_hostemcee_versionreservedreservedreserved) of current jobs on the queue - Simulator operations metrics - [allocation duration](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixsimulatorallocationdurationworker_hostallocation_outcomeemcee_versionreservedreservedreserved), [action duration](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixsimulatoractiondurationactionworker_hostdevice_typeruntimeis_successemcee_versionreservedreservedreserved) (these are durations of create, boot, shutdown and delete operations) ## Specific Analytics Another analytics configuration is a **specific** one. You provide a configuration for it via test arg file, and it is used to report job-specific analytic events. Some, if not all, examples: - Job [processing durations](https://github.com/avito-tech/Emcee/wiki/Graphite#statsd-prefixjobdurationqueue_hostemcee_versionpersistent_metrics_job_idreservedreservedreservedreservedreserved) - Job [preparation duration](https://github.com/avito-tech/Emcee/wiki/Graphite#statsd-prefixjobpreparationqueue_hostemcee_versionpersistent_metrics_job_idsuccessfailurereservedreservedreservedreservedreserved) - how long to takes to prepare job, i.e. discover all tests and enqueue them - Bucket [dequeue](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuebucketsdequeueworker_idemcee_versionqueue_hostreservedreserved) and [enqueue](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuebucketsenqueueemcee_versionqueue_hostreservedreserved) events - Test [dequeue](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuetestsdequeueworker_idemcee_versionqueue_hostreservedreserved) and [enqueue](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuetestsenqueueemcee_versionqueue_hostreservedreserved) events - [Bucket processing duration](https://github.com/avito-tech/Emcee/wiki/Graphite#statsd-prefixbucketdurationqueue_hostemcee_versionpersistent_metrics_job_idreservedreservedreservedreservedreserved). Each job consists of tests, and tests are grouped into buckets. This metric is used to report the durations of these buckets. - [Stuck buckets](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuebucketsstuckreasonworker_idemcee_versionqueue_hostreservedreserved) in job - Job queue states - number of [enqueued](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuejobsstateenqueuedqueue_hostjob_idemcee_versionreservedreservedreserved) and [dequeued](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuejobsstatedequeuedqueue_hostjob_idemcee_versionreservedreservedreserved) buckets, [time to dequeue a bucket](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixqueuebucketstime_to_dequeueemcee_versionqueue_hostreservedreserved) - Test discovery events - [test case count](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixruntime_dumptest_case_counttest_bundle_nameemcee_versionreservedreservedreserved), [test count](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixruntime_dumptest_counttest_bundle_nameemcee_versionreservedreservedreserved), and [aggregated test discovery duration](https://github.com/avito-tech/Emcee/wiki/Graphite#statsd-prefixtestdiscoverydurationhostemcee_versionpersistent_metrics_job_idsuccessfailurereservedreservedreserved) - [Time to start a test](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixteststime_to_starttest_class_nametest_method_nameemcee_versionqueue_hostreservedreserved) metric - [Test preflight](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixtestpreflightworker_hostemcee_versionreservedreservedreserved) duration - [Test started event](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixteststartedworker_hosttest_classtest_methodemcee_versionreservedreserved) - [Time between tests](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixtestbetween_testsdurationworker_hostemcee_versionreservedreservedreserved) - [Test finished event](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixtestfinishedresultworker_hosttest_classtest_methodemcee_versionreservedreserved) - [Concrete test duration](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixtestdurationresultworker_hosttest_classtest_methodemcee_versionreservedreserved) - [Aggregated test duration](https://github.com/avito-tech/Emcee/wiki/Graphite#statsd-prefixtestdurationhostemcee_versionpersistent_metrics_job_idresultreservedreservedreservedreserved) - [Test postflight](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixtestpostflightworker_hostemcee_versionreservedreservedreserved) duration - [Useless test runner invocation duration](https://github.com/avito-tech/Emcee/wiki/Graphite#prefixtestuseless_runner_invocationhostemcee_versionreservedreservedreservedreserved) metric **It is worth noting that specific analytics configuration does NOT inherit anything from global configuration**. You must pass complete configurations via both queue configuration and via test arg file. One thing to keep in mind is that test arg file should contain a job specific analytics configuration, so at least `analyticsConfiguration.persistentMetricsJobId` can (and probably should) be different depending on your job (e.g. unit tests, ui tests, full regression suite, etc.). As usual, it is better to refer to the unit tests to see all possible ways how test arg file can be configured. Tests for root test arg file structure [are here](https://github.com/avito-tech/Emcee/blob/master/Tests/TestArgFileTests/TestArgFileTests.swift), and tests for test arg file entries [are here](https://github.com/avito-tech/Emcee/blob/master/Tests/TestArgFileTests/TestArgFileEntryTests.swift). # Kibana Emcee now supports sending all its logs to a common Kibana end point(s). Kibana is configured [via `[global]AnalyticsConfiguration.kibanaConfiguration`](https://github.com/avito-tech/Emcee/wiki/Analytics#kibana). Emcee will add the following values to logged messages: - `hostname` — host which emitted a log message - `emceeVersion` — version of Emcee - `emceeCommand` — command being executed, e.g. `dump`, `runTestsOnRemoteQueue`, `distWork`, etc. - `workerId` — worker which emitted a log message. - `processId` and `processName` — pid and process name of Emcee itself, e.g. `3224` and `EmceeWorker_ab33da`. - `subprocessId` and `subprocessName` — pid and process name of a subprocess which Emcee executed - `xcrunToolName` — tool name which has been executed via `xcrun`, e.g. for `xcrun xcodebuild` this key will have `xcodebuild` value - `persistentMetricsJobId` — a job id from test arg file - `subprocessPipe` — standard output pipe name for messages logged by subprocesses, e.g. `stdout` or `stderr`. Currently Emcee logs subprocess messages during test discovery process, but more cases may be adopted in the future. In addition to default keys that Emcee adds automatically, you may add your keys via `[global]analyticsConfiguration.metadata`. On our CI, for instance, we add the following keys to help us track down **job-specific** logs (so we append these values to **specific** configuration via testargfile): `ciBranch`, `ciBuildNumber`, `ciPullRequestId`. # Handy Pendy - `--temp-folder` is now optional - Emcee now removes test and test runner artifacts after executing a bucket. Plugins still have time to process test artifacts (see [`TestsWorkingDirectorySupport`](https://github.com/avito-tech/Emcee/wiki/Plugins#test-working-directory)), but as soon as all plugins terminate, Emcee worker will clean up all files. This fixes an issue with a leaking disk space. - Emcee parses `xcodebuild -resultStreamPath` file contents by default now and even attempts to add crash logs (if `xcodebuild` will emit them). - Emcee now also collects the global test exceptions that happen before any test starts. Now, when test bundle crashes, Emcee will append these exceptions to all tests that were expected to run. - `xcrun simctl spawn` calls are now limited by a hardcoded timeout of 30 seconds to prevent potential stalls. 30 seconds is more than enough, but if it is not, [reach out to us](https://github.com/avito-tech/Emcee/discussions). # Nice to Know - Emcee now tracks test timeouts that may happen before any test starts, after the last test finishes, and in between two test runs. This timeout value is controlled by `testArgFile.testTimeoutConfiguration.testRunnerMaximumSilenceDuration`(https://github.com/avito-tech/Emcee/wiki/Test-Arg-File#testtimeoutconfiguration). - Plugin APIs didn't change much, but we've updated docs: https://github.com/avito-tech/Emcee/wiki/Plugins - Emcee works well with `xcodebuild` test runner. Support for `fbxctest` has been removed entirely. Also, we don't parse `xcodebuild` logs anymore, we always use the secret power of `-resultStreamPath` flag. - We didn't use Sentry + Emcee combination, and apparently nobody did, and codebase required tuning and polishing anyways. So we removed support for Sentry. - We've updated some docs about configuring analytics: https://github.com/avito-tech/Emcee/wiki/Queue-Server-Configuration#globalanalyticsconfiguration - We've updated some docs about what analytic events Emcee sends. This page now describes statsd metrics too: https://github.com/avito-tech/Emcee/wiki/Graphite - A lot of targets have been migrated from Emcee project up to the [CommandLineToolkit SPM project](http://github.com/avito-tech/CommandLineToolkit). You may need to fix your plugins build errors because of this change. Some APIs changed too.

The markdown parsing is broken/disabled for release notes. Sorry about that, I'm chasing the source of a crash that's been bringing this website down for the last couple of days. This releases fixes compatibility with Xcode 10.2.1 Basically this release should fix this issue: https://github.com/avito-tech/Mixbox/issues/12 It also contains tons of other changes, refactorings, fixes. It is still not production ready for external developers, so I will not describe changes.