Swiftpack.co - Package - SeanROlszewski/muter

Muter

Swift 5 Build Status

Automated mutation testing for Swift inspired by Stryker, PITest, and Mull.

Muter can be run within Xcode

Use this mode to rapidly diagnose areas where you can begin improving your test code

Muter running inside Xcode

Muter can be run from the command line

Use this mode to get detailed information about the health and quality of your entire test suite

Muter running from the commandline

What Is Muter?

Muter is a mutation testing utility that is used to help you determine the quality of your test suite.

With Muter, you can make sure your test suite is meeting all your requirements, fails meaningfully and clearly, and remains stable in the face of unexpected or accidental code changes.

If you're interested in checking out more about mutation testing, you can check out this link.

Why Should I Use This?

Muter can strengthen your test suite and shine a light on weaknesses that you were unaware existed. It does this by generating a mutation score (expanded on below), which will show you both the areas you may want to improve in your test suite, as well as the areas that are performing well.

Specifically, a mutation score can help you:

  • Find gaps in fault coverage from your test suite by identifying missing groups of tests, assertions, or test cases from your test suite
  • Determine if you are writing meaningful and effective assertions that withstand different code than what the test was originally written against
  • Assess how many tests fail as a result of one code change

How Does It Work?

Muter will introduce changes into your source code based on the logic contained in your app. The changes introduced by Muter are called mutants which it generates using mutation operators.

You can view the list of available mutation operators here.

NOTE: Muter does all of its work on a complete copy of your codebase, so it's not possible for it to accidentally leave anything behind.

Mutation Score

A mutation score is provided at the end of every run of Muter. The score is the ratio of the number of mutants your test suite killed versus the total number of mutants introduced.

mutation score = number of mutants killed / total number of mutants

For example, if your test suite killed 50 mutants of the 75 introduced by Muter, your score would be 67%. A well-engineered test suite should strive to be as close to 100% as possible.

Muter not only provides a mutation score for your entire test suite, but it also generates individual scores for the files it has mutated.

If you're curious about how a mutation score is different than test code coverage, then check out this document.

Example Test Report

There's an example of the test report that Muter generates hosted in this repository.

Check out this example to familiarize yourself with what a report looks like.

Installation

Muter is available through Homebrew. Run the following command to install Muter:

brew install seanrolszewski/formulae/muter

Setup

Muter's Configuration

To get started using Muter, run muter init in the root of your project directory. Muter will take its best guess at a configuration that will work for your project. Muter supports generating configurations for the following build systems:

  • Xcode Projects & Workspace
  • Swift Package Manager

It saves its configuration into a file named muter.conf.json, which you should keep in the root directory of your project. You should version control your configuration file as well.

After running muter init, you should look at the generated configuration and ensure that it will run your project. We recommend trying the settings it generates in your terminal, and verifying those commands run your tests.

Should you need to modify any of the options, you can use the list below to understand what each configuration option does.

Configuration Options

  • executable - the absolute path to the program which can run your test suite (like xcodebuild, swift, fastlane, make, etc.)

  • arguments - any command line arguments the executable needs to run your test suite

  • exclude - a list of paths, file extensions, or names you want Muter to ignore. By default, Muter ignores all non-Swift files, and any files or paths containing the following phrases:

    • .build
    • .framework
    • .swiftdep
    • .swiftmodule
    • Build
    • Carthage
    • muter_tmp
    • Pods
    • Spec
    • Test

    The exclude option is optional.

NOTE: Muter uses a substring match to determine if a file should be excluded from mutation testing. You should not use glob expressions (like **/*Model.swift) or regex.

Below is an example pulled directly from the ExampleApp project. The configuration file will end up looking something like this:

{
    "executable": "/usr/bin/xcodebuild",
    "arguments": [
        "-project",
        "ExampleApp.xcodeproj",
        "-scheme",
        "ExampleApp",
        "-sdk",
        "iphonesimulator",
        "-destination",
        "platform=iOS Simulator,name=iPhone 8",
        "test"
    ],
    "exclude": ["AppDelegate.swift"]
}

Check out the muter.conf.json in the root directory of this repository for another example.

Xcode Setup

Setting up Muter to run within Xcode is simple. After creating your configuation:

  1. Create a new Aggregate Build Target in the Xcode project of the codebase you're mutation testing. We suggest calling it "Mutation Test"

  2. Add a run script step to the newly created aggregate build target.

  3. Add the Muter Xcode command to the build step:

    muter --output-xcode

Running Muter

From the command line

Once you've created your configuration file, simply run muter in your terminal from any directory of the project you're mutation testing. Muter will take it from there.

Within Xcode

Build (Cmd + B) your aggregate build target and let Muter run. The mutants which survive testing will be called out in the issue navigator. Once the target finishes building, testing has completed.

Limitations

  • Muter assumes you always put spaces around your operators. For example, it expects an equality check to look like

    a == b (Muter will mutate this)

    not like:

    a==b (Muter won't mutate this)

  • Muter assumes you aren't putting multiple expressions on one line (and we have the opinion you shouldn't be doing this anyway). Basically, if you aren't using semicolons in your code then Muter shouldn't have an issue mutating it.

Best Practices

  • Commit your muter.conf.json
  • It's possible for Muter to cause compile time warnings. As a result of this, we recommend you don't treat Swift warnings as errors while mutation testing by adding the argument SWIFT_TREAT_WARNINGS_AS_ERRORS=NO to your muter.conf.json if you're using xcodebuild.
  • Disable or relax linting rules that would cause a build error as a consequence of a code change not matching your project's style. Muter operates on your source code and then rebuilds it, and the change it introduces could trigger your linter if it's part of your build process.
  • Running Muter can be a lengthy process, so be sure to allocate enough time for the test to finish.
  • Because Muter can take a while to run, it is recommend to exclude UI or journey tests from your test suite. We recommend creating a separate schemes or targets for mutation testing. However, you should feel free to run these kinds of tests if you're okay with the longer feedback cycle.
  • Don’t be dogmatic about your mutation score - in practice, 100% is not always possible.

FAQ

What platforms does Muter support?

Muter supports any platform that compiles and tests using xcodebuild, which includes iOS, macOS, tvOS, and watchOS.

Muter can run only on macOS 10.13 or higher.

Does Muter support UI test suites?

Yes! However, these can be very lengthy test suites, and mutation testing can take a long time. I recommend you start using Muter only on your unit tests. Once you have a feel for interpreting mutation scores, you can then ease into incorporating your longer-running tests.

Does Muter support Objective-C?

No, not at this time. Objective-C support will come at a later time. Until then, Muter only supports Swift code. Any bridging code that's written in Swift, but ultimately calls down to Objective-C, is compatible with Muter.

Is Muter self-hosted?

Yes! Very early on I made the decision to make sure that Muter was able to provide insight into its development and test suite. After all, since Muter is providing a form of automated testing, it must be as thorough and robust as possible. :P

This is all pretty cool, but I'm nervous about running this on my own code. I mean, you're putting bugs into my work, and how do I know you're not stealing my source code?

This is an understandable concern. If you would like to get a feel for what mutation testing is like, and how Muter performs it, I recommend cloning this repository, installing Muter, and then running Muter on the included example project and Muter itself.

Additionally, because Muter is parsing, analyzing, and modifying your source code, a decision was made to give it no network access - Muter collects no analytics, and never phones home. Feel free to look at its source code if you have concerns about this, or open an issue if you would like to have a discussion.

And lastly, make sure you look at and follow Muter's best practices to ensure the best possible experience while using Muter.

Github

link
Stars: 111
Help us keep the lights on

Used By

Total:

Releases

v10 - May 28, 2019

This release of Muter updates it to Swift 5. In addition to this, it contains a handful of new features we're quite excited about!

πŸ†• What's new? πŸ†•

  • There is a new mutation operator, InvertLogicalConnector! This operator is useful for driving out testing issues that don't catch all the branches that your logical operators can take. Read more about it here.
  • Running muter init will now generate a configuration file that is filled out with Muter's best guess at a configuration that will work for your project. This means that it can be as simple as muter init && muter to get your project up and running! Right now, the automatic configuration generation supports projects that are using Xcode projects and workspace, as well as SPM projects. If you don't use any of these for your build and test system, Muter is still configurable manually inside of its muter.conf.json.

⬆️ What's improved? ⬆️

  • The error messaging for when Muter detects a missing or invalid configuration has been improved to help you troubleshoot what might be going wrong.

Many thanks to @rakaramos, @ZevEisenberg, and a many others for their suggestions on what to improve, and their help getting this release together. A particular amount of gratitude is due to @mdiasdev for his help getting a CI set up for this project. Thank you all! :D

v8 - Mar 4, 2019

This release of Muter contains many suggestions and improvements that came from you all, the community!

πŸ†• What's new? πŸ†•

  • Muter is now formally licensed under the GNU Affero General Public License v3.0 (GNU AGPL v3). You can see the rationale behind the licensing choice here. (Thanks for reminding us of the importance of this, @ZevEisenberg)

⬆️ What's improved? ⬆️

  • Muter's reporting of mutation test results now avoids using the language of passing and failing. Instead, it uses the domain language of having killed mutants or surviving mutants, with an accompanying reason for how a mutant was killed. Check out the new test report format here. This was in response to many of you finding the previous language confusing, which hindered your ability to digest the report and make changes based on it.
  • Muter's commandline output will now concatenate the filename and line number where a mutation operator was applied, which allows you to more rapidly look it up in your editor of choice. Previously, these were separate columns. Now it's formatted as fileName.swift:1. (Thanks for the suggestion, @fcanas)
  • Muter will now ignore operator declarations when discovering where it can apply the negate conditionals operator. This change eliminates a category of build errors that Muter caused by modifying operator declarations. Muter will now run faster on code bases which have custom operator declarations. (Thanks again @fcanas!)
  • Muter now gracefully handles an edge case were a division by zero would occur if your test report only contained build errors.

v7 - Feb 10, 2019

This release of Muter contains improvements to Muter's XCTest integration, a new report file format, and a new command line argument.

πŸ†• What's new? πŸ†•

  • Muter now has the option of conditionally exporting a JSON report of all of the information contained in its test report. You can enable this by running Muter with the new commandline flag - --output-json. The report will be written to a file named muterReport.json. Stay tuned for some interesting features we have planned which will take advantage of this report format. ;)
  • Muter will now abort testing if it notices 5 consecutive build errors during mutation testing. This is a fail-fast feature intended to help callout anomalous test runs to you. This makes it faster to diagnose any potential issues Muter may have running your test suite.

⬆️ What's improved? ⬆️

  • Muter is now able to identify more cases of test failures within XCTest's test logs. This improves the accuracy of the mutation scores generated by Muter.

πŸ“£πŸŽ‰ Shout outs πŸŽ‰πŸ“£ I'm excited to shout out @rakaramos for his work on adding the new commandline argument and fail-fast/fail-early features that are included with this release. He worked on the bulk of the features included in this release, and is a generally awesome person to collaborate with. πŸ‘

I'm also excited to shout out @reedr3, who I paired with on writing Muter's new regression test suite. The work we did together makes sure that future releases of Muter can't adversely affect its test reporting, which is essential for depending on it as a testing tool. ✌️

Thank you both for all of your work, and here's to many more releases!

v6 - Jan 28, 2019

This release includes several updates that affect Muter's handling of build errors, run-time errors, and mutation scores.

Specifically:

  • Muter performs an initial test suite run which will prevent mutation testing if it fails. In previous versions, it would continue performing its tests even if it was unable to compile your application or test suite. This would produce meaningless results and waste time.
  • Muter is able to identify when a build error occurs as a result of a mutant it creates, and no longer treats a build error as a passing mutation test. This improves the signal of generated mutation scores.
  • Muter now identifies run-time errors that lead to a crash, and differentiates them from test failures.
  • Muter's mutation score calculations now include failing tests, run-time errors, and passing tests, but explicitly exclude any build errors.

Additionally, there's been an improvement to the messaging when Muter doesn't discover source files; it's output now includes some suggestions to help you troubleshoot.

And lastly, there are a few minor style and copy changes to the mutation scores to make them a little bit easier to read.

v5 - Jan 21, 2019

This release introduces the first breaking change within Muter's configuration file schema.

What was previously called blacklist is now called exclude. This change was made to better reflect the intent of the option, which is to exclude files from the mutation testing that Muter performs.

What's new? In addition to the change to the configuration file called out above:

  • In the event that Muter discovers no changes it can apply to your codebase, it will stop early and display some suggestions of what to do.
  • Muter's source file discovery is now stricter, and will ignore files that do not end in .swift. Before this change, files that had .swift anywhere in their file path would be considered for mutation testing.
  • Muter now displays what version you're running in its header whenever you invoke it from the command line.
  • Muter now swallows all of the logs from xcodebuild. It dumps them into log files which you can find inside the temporary directory Muter generates for testing. A future version will be focused on making these files more discoverable.

What's been fixed? There was a bug introduced in v4 that would cause pollution between mutation test passes, which would adversely affect mutation scoring. This issue has been resolved.