Skip to content

Easy Peasy
Compare
Choose a tag to compare
@beefon beefon released this 17 Jan 17:58
16.0.0
a665ac0

We've conducted a bunch of interviews with teams who use Emcee. We were provided with nice feedback which we used to improve our product. In this release, we focus on ease of use and an easy first try experience.

Getting Started Guide with Pictures and Sample Project 🥩

It is available right at README.

Just Run These Tests 🤬

We introduce a new command - runTests. A very similar one to a famous runTestsOnRemoteQueue but it is dedicated to opening the world of simplicity and clear setup for most Emcee users. It has the following syntax and options:

$ Emcee runTests -h

  • --queue: describes where should a queue be started. This argument may be repeated multiple times, in which case Emcee will use hosts one by one until it manages to start the queue. You must provide at least one value here.
  • --worker: describes where worker should be started. This argument may be repeated multiple times. It is required to provide at least one value.
  • --device: Device to run test on, e.g. iPhone X. Required. Note: iPhone SE generations might be tricky to use!
  • --runtime: Runtime to run test on, e.g. 15.0. Required.
  • --test-bundle: Location of an .xctest bundle which tests will be executed. Required.
  • --app: Location of an .app bundle. Optional. Please refer to our guide to learn more about tests with host app.
  • --runner: Location of an XX-Runner.app. Optional.
  • --test: Test to execute, e.g. ClassName/testMethod. Optional. If not specified, Emcee will run all tests. You can specify multiple tests to execute, e.g. --test Class/test1 --test Class/test2
  • --retries: How many retries to attempt to run each test successfully. Optional. By default it will retry tests once if failure happens.
  • --test-timeout: Maximum test execution duration. Optional. Default test duration limit is 180 seconds.
  • --junit: Path where the Junit report file should be created. Optional.
  • --trace: Path where the Chrome trace file should be created. Optional.

Let's explain --queue and --worker arguments. These are URLs which describe how to connect to a machine. Currently only SSH is supported. Some examples:

  • --queue ssh://emcee:[email protected]/Users/emcee/emceequeue.noindex: it means to run Emcee queue on a queue.example.com host, by authenticating with emcee username and pass password. The working directory will be located at /Users/emcee/emceequeue.noindex.
  • --worker ssh://[email protected]/Users/emcee/emceeworker.noindex?custom_rsa: it means the queue will start its worker on worker.example.com. Worker will use /Users/emcee/emceeworker.noindex as its working directory. Also, Emcee will use emcee as username and ~/.ssh/custom_rsa key to auth against a worker host. This key is expected to be present on queue host.
  • --worker ssh://[email protected]/Users/emcee/emceeworker.noindex#/absolute/path/to/custom_rsa: same as above, but queue will use a key from /absolute/path/to/custom_rsa.

Free HTTP Server for Everyone! 🌏

Emcee now contains a built-in HTTP server that will cover most of your needs. This significantly simplifies the flow and lowers the entry threshold.

You can pass local paths into runTests command. Provided local .app and .xctest bundles will be hosted via a built-in HTTP server transparently, allowing all workers to download artifacts in order to run tests.

You can also provide local paths inside test arg file which you pass into runTestsOnRemoteQueue, those will also be hosted via the built-in server. Handy!

And finally, you can provide a local queue server configuration file into runTestsOnRemoteQueue. Now there is no need to upload it to a HTTP server.

Other Goodies 🦚

Help me Dr. -h 🍡

You can now get help for any command by typing -h next to it, e.g. Emcee -h or Emcee runTests -h. Sounds like Emcee is ready for such kind of hi-tech.

Better errors 💢

Should you provide an incorrect test arg file or queue configuration JSON file, Emcee will now print human-readable errors. You will be amazed. Here is one error for your consideration:

Failed to decode value for key "entries" in "test arg file": key "xcTestBundle" not found at entries[0].buildArtifacts

That is a crystal clear explanation of what went wrong, is it not?

Optional jobId ☃️

It wasn't clear for some of our users that jobId field in a test arg file should be unique. Now you can omit it, Emcee will generate a random jobId for you.

Test Retry Mode 👯‍♀️

You can now specify how you'd like your tests to be retried if they fail. In your test arg file's entries there is a new field testRetryMode. Possible values:

  • retryThroughQueue — if a test fails on a worker, it will be returned back to the queue. A queue will then retry this test on other workers, up to the specified number of retries. This is the default behavior. This allows the test to be executed on different hosts, potentially eliminating the problem of a broken environment (because every macOS installation in Universe is unique).

  • retryOnWorker — if the test fails, the same worker will perform retry.

Simulator Patching is Better 📲

There was a bug (OMG) that resulted in redundant simulator patching (read: apply simulator settings). Now Emcee won't patch simulators if
everything is patched.

Cacheless Workers are Now Possible 💳

There is a way to disable worker cache. In queue server configuration, in worker specific settings, next to numberOfSimulators field there are two more fields:

  • maximumCacheSize — maximum cache size in bytes.
  • maximumCacheTTL — maximum TTL for any cached item, in seconds.

You can set 0 these fields, and it will disable cache entirely. This is handy if you want to debug something.

Prevent Emcee from Cleaning Runner Output 🏃

You can now stop workers from removing all test artifacts automatically after a test finishes and plugins terminate. In a test arg file's entries there is a new field runnerWasteCleanupPolicy. Possible values:

  • clean — default value, it means everything will be deleted
  • keep — all created files will be preserved.

Log Capturing Mode 📝

You can now control what logs should be captured during a test run. These logs are obtained from xcresult bundles after the tests finish. In the test arg file's entries there is a new field logCapturingMode. Possible values:

  • allLogs — all log messages will be captured
  • onlyCrashLogs — only logs that look like crash logs will be captures
  • noLogs — no logs will be captured

Where without dyld

You can now insert dynamic libraries into a test. In a test arg file's entries there is a new field userInsertedLibraries. This is an array of paths. These paths will be passed as DYLD_INSERT_LIBRARIES env into the test. Combine it with state of the art macOS imaging features, with a fact that tests are being executed via xcodebuild (so it can expand __TESTBUNDLE__ and other strings - look at man xcodebuild.xctestrun), and this will provide you some additional flexibility for your test execution process.

Try it Today! 🧨

  • Below is a universal binary (arm64 & x64) for your convenience! It even has built-in libssh, so you can use it with ease.
  • If you download Emcee binary from GitHub, do not forget to remove quarantine attribute: xattr -c Emcee
  • Please rebuild your plugins. v15 API is not compatible with v16.

Full Changelog: 15.0.0...16.0.0

Assets 3
Loading