Swiftpack.co - Package - typelift/SwiftCheck

Carthage compatible Build Status Gitter chat

SwiftCheck

QuickCheck for Swift.

For those already familiar with the Haskell library, check out the source. For everybody else, see the Tutorial Playground for a beginner-level introduction to the major concepts and use-cases of this library.

Introduction

SwiftCheck is a testing library that automatically generates random data for testing of program properties. A property is a particular facet of an algorithm or data structure that must be invariant under a given set of input data, basically an XCTAssert on steroids. Where before all we could do was define methods prefixed by test and assert, SwiftCheck allows program properties and tests to be treated like data.

To define a program property the forAll quantifier is used with a type signature like (A, B, C, ... Z) -> Testable where A : Arbitrary, B : Arbitrary ... Z : Arbitrary. SwiftCheck implements the Arbitrary protocol for most Swift Standard Library types and implements the Testable protocol for Bool and several other related types. For example, if we wanted to test the property that every Integer is equal to itself, we would express it as such:

func testAll() {
    // 'property' notation allows us to name our tests.  This becomes important
    // when they fail and SwiftCheck reports it in the console.
    property("Integer Equality is Reflexive") <- forAll { (i : Int) in
        return i == i
    }
}

For a less contrived example, here is a program property that tests whether Array identity holds under double reversal:

property("The reverse of the reverse of an array is that array") <- forAll { (xs : [Int]) in
    // This property is using a number of SwiftCheck's more interesting 
    // features.  `^&&^` is the conjunction operator for properties that turns
    // both properties into a larger property that only holds when both sub-properties
    // hold.  `<?>` is the labelling operator allowing us to name each sub-part
    // in output generated by SwiftCheck.  For example, this property reports:
    //
    // *** Passed 100 tests
    // (100% , Right identity, Left identity)
    return
        (xs.reversed().reversed() == xs) <?> "Left identity"
        ^&&^
        (xs == xs.reversed().reversed()) <?> "Right identity"
}

Because SwiftCheck doesn't require tests to return Bool, just Testable, we can produce tests for complex properties with ease:

property("Shrunken lists of integers always contain [] or [0]") <- forAll { (l : [Int]) in
    // Here we use the Implication Operator `==>` to define a precondition for
    // this test.  If the precondition fails the test is discarded.  If it holds
    // the test proceeds.
    return (!l.isEmpty && l != [0]) ==> {
        let ls = self.shrinkArbitrary(l)
        return (ls.filter({ $0 == [] || $0 == [0] }).count >= 1)
    }
}

Properties can even depend on other properties:

property("Gen.one(of:) multiple generators picks only given generators") <- forAll { (n1 : Int, n2 : Int) in
    let g1 = Gen.pure(n1)
    let g2 = Gen.pure(n2)
    // Here we give `forAll` an explicit generator.  Before SwiftCheck was using
    // the types of variables involved in the property to create an implicit
    // Generator behind the scenes.
    return forAll(Gen.one(of: [g1, g2])) { $0 == n1 || $0 == n2 }
}

All you have to figure out is what to test. SwiftCheck will handle the rest.

Shrinking

What makes QuickCheck unique is the notion of shrinking test cases. When fuzz testing with arbitrary data, rather than simply halt on a failing test, SwiftCheck will begin whittling the data that causes the test to fail down to a minimal counterexample.

For example, the following function uses the Sieve of Eratosthenes to generate a list of primes less than some n:

/// The Sieve of Eratosthenes:
///
/// To find all the prime numbers less than or equal to a given integer n:
///    - let l = [2...n]
///    - let p = 2
///    - for i in [(2 * p) through n by p] {
///          mark l[i]
///      }
///    - Remaining indices of unmarked numbers are primes
func sieve(_ n : Int) -> [Int] {
    if n <= 1 {
        return []
    }

    var marked : [Bool] = (0...n).map { _ in false }
    marked[0] = true
    marked[1] = true

    for p in 2..<n {
        for i in stride(from: 2 * p, to: n, by: p) {
            marked[i] = true
        }
    }

    var primes : [Int] = []
    for (t, i) in zip(marked, 0...n) {
        if !t {
            primes.append(i)
        }
    }
    return primes
}

/// Short and sweet check if a number is prime by enumerating from 2...⌈√(x)⌉ and checking 
/// for a nonzero modulus.
func isPrime(n : Int) -> Bool {
    if n == 0 || n == 1 {
        return false
    } else if n == 2 {
        return true
    }
    
    let max = Int(ceil(sqrt(Double(n))))
    for i in 2...max {
        if n % i == 0 {
            return false
        }
    }
    return true
}

We would like to test whether our sieve works properly, so we run it through SwiftCheck with the following property:

import SwiftCheck

property("All Prime") <- forAll { (n : Int) in
    return sieve(n).filter(isPrime) == sieve(n)
}

Which produces the following in our testing log:

Test Case '-[SwiftCheckTests.PrimeSpec testAll]' started.
*** Failed! Falsifiable (after 10 tests):
4

Indicating that our sieve has failed on the input number 4. A quick look back at the comments describing the sieve reveals the mistake immediately:

- for i in stride(from: 2 * p, to: n, by: p) {
+ for i in stride(from: 2 * p, through: n, by: p) {

Running SwiftCheck again reports a successful sieve of all 100 random cases:

*** Passed 100 tests

Custom Types

SwiftCheck implements random generation for most of the types in the Swift Standard Library. Any custom types that wish to take part in testing must conform to the included Arbitrary protocol. For the majority of types, this means providing a custom means of generating random data and shrinking down to an empty array.

For example:

import SwiftCheck
 
public struct ArbitraryFoo {
    let x : Int
    let y : Int

    public var description : String {
        return "Arbitrary Foo!"
    }
}

extension ArbitraryFoo : Arbitrary {
    public static var arbitrary : Gen<ArbitraryFoo> {
        return Gen<(Int, Int)>.zip(Int.arbitrary, Int.arbitrary).map(ArbitraryFoo.init)
    }
}

class SimpleSpec : XCTestCase {
    func testAll() {
        property("ArbitraryFoo Properties are Reflexive") <- forAll { (i : ArbitraryFoo) in
            return i.x == i.x && i.y == i.y
        }
    }
}

There's also a Gen.compose method which allows you to procedurally compose values from multiple generators to construct instances of a type:

public static var arbitrary : Gen<MyClass> {
    return Gen<MyClass>.compose { c in
        return MyClass(
            // Use the nullary method to get an `arbitrary` value.
            a: c.generate(),

            // or pass a custom generator
            b: c.generate(Bool.suchThat { $0 == false }),

            // .. and so on, for as many values and types as you need.
            c: c.generate(), ...
        )
    }
}

Gen.compose can also be used with types that can only be customized with setters:

public struct ArbitraryMutableFoo : Arbitrary {
    var a: Int8
    var b: Int16
    
    public init() {
        a = 0
        b = 0
    }
    
    public static var arbitrary: Gen<ArbitraryMutableFoo> {
        return Gen.compose { c in
            var foo = ArbitraryMutableFoo()
            foo.a = c.generate()
            foo.b = c.generate()
            return foo
        }
    }
}

For everything else, SwiftCheck defines a number of combinators to make working with custom generators as simple as possible:

let onlyEven = Int.arbitrary.suchThat { $0 % 2 == 0 }

let vowels = Gen.fromElements(of: [ "A", "E", "I", "O", "U" ])

let randomHexValue = Gen<UInt>.choose((0, 15))

let uppers = Gen<Character>.fromElements(in: "A"..."Z")
let lowers = Gen<Character>.fromElements(in: "a"..."z")
let numbers = Gen<Character>.fromElements(in: "0"..."9")
 
/// This generator will generate `.none` 1/4 of the time and an arbitrary
/// `.some` 3/4 of the time
let weightedOptionals = Gen<Int?>.frequency([
    (1, Gen<Int?>.pure(nil)),
    (3, Int.arbitrary.map(Optional.some))
])

For instances of many complex or "real world" generators, see ComplexSpec.swift.

System Requirements

SwiftCheck supports OS X 10.9+ and iOS 7.0+.

Setup

SwiftCheck can be included one of two ways:

Using The Swift Package Manager

  • Add SwiftCheck to your Package.swift file's dependencies section:
.package(url: "https://github.com/typelift/SwiftCheck.git", from: "0.8.1")

Using Carthage

  • Add SwiftCheck to your Cartfile
  • Run carthage update
  • Drag the relevant copy of SwiftCheck into your project.
  • Expand the Link Binary With Libraries phase
  • Click the + and add SwiftCheck
  • Click the + at the top left corner to add a Copy Files build phase
  • Set the directory to Frameworks
  • Click the + and add SwiftCheck

Using CocoaPods

  • Add our Pod to your podfile.
  • Run $ pod install in your project directory.

Framework

  • Drag SwiftCheck.xcodeproj into your project tree as a subproject
  • Under your project's Build Phases, expand Target Dependencies
  • Click the + and add SwiftCheck
  • Expand the Link Binary With Libraries phase
  • Click the + and add SwiftCheck
  • Click the + at the top left corner to add a Copy Files build phase
  • Set the directory to Frameworks
  • Click the + and add SwiftCheck

License

SwiftCheck is released under the MIT license.

Github

link
Stars: 1220

Dependencies

Used By

Total: 0

Releases

Fivel - 2019-03-28 17:36:17

⚠️ Breaking Changes Ahead ⚠️

  • Update to Swift 5, 5-style package format, and upgrade the Xcode project to compile in Swift 5 mode
  • Fix a bug where covered properties yielded a false positive if the condition was never true

In the coming release, we will be reworking the coverage system to better match QuickCheck's current functionality.

Four Two The Floor - 2018-09-19 19:06:08

⚠️ Breaking Changes Ahead ⚠️

  • Update to the Swift 4.2 tooling, 4.2-style package format, and upgrade the Xcode project to compile in the 4.2 mode of the Swift compiler.
  • Removed the previously-deprecated Gen.map forms. Please use the corresponding Gen.zip function call itself
  • RemovedquickCheck(_ :name:). Use one of quickCheck(asserting:) or quickCheck(reporting:) instead.

Impredicativity - 2018-09-19 18:50:33

  • Remove self-referential Arbitrary instances for lazy collections
  • Updates to documentation

⚠️ Breaking Changes Ahead ⚠️

This is the last release of SwiftCheck that supports Xcode 9 and its tooling.

Higher-Order Checkanery - 2018-04-12 03:10:43

Swift turns 4.1, and so SwiftCheck marches onward. This release brought with it a raft of improvements:

  • Non-operator spellings of the standard property() <- test pattern is provided in the form of the new quickCheck functions.
  • Conditional conformances for a raft of Standard Library types obviate most modifier types. As such, ArrayOf<T>, SetOf<T>, DictionaryOf<K, V>, and OptionalOf<T> have been removed. You can now directly use the natural types [T], Set<T>, [K: V], and T? respectively.
  • Documentation improvements and simplifications

Uppercut - 2017-11-02 02:45:08

  • Silence warnings introduced by Xcode 9.1.

Chubby Checker - 2017-10-20 21:51:31

SwiftCheck has internally upgraded to Swift 4.x.

⚠️ Breaking Changes Ahead ⚠️

  • Generation of floating point values specified an incorrect mask which could lead to decreased diversity of test values. Calculation of the mask has been corrected (h/t @sebastiangrail). Seeds replaying tests involving floating point numbers may need to be recalculated.

  • Gen.fromElements(in:), Gen.fromElements(of:), Gen.choose(_:) and Gen.chooseAny() have been updated to take fuller advantage of type inference and may no longer require explicit specialization. In addition, incorrect specializations may now be diagnosed as errors.

  • Gen.map(...) overloads have been deprecated and renamed Gen.zipWith. These have been given a similar overhaul to the above.

Pointless Pointilism - 2017-09-20 16:13:40

⚠️ Breaking Changes Ahead ⚠️

SwiftCheck now targets Xcode 9 and Swift 3.2. This is the last version of SwiftCheck that will support Swift 3.

Swift Three, We Hardly Knew Ye - 2017-03-29 03:07:24

SwiftCheck now builds with Swift 3.1

I-Triple-E-To-The-One-To-The-Three - 2017-02-08 20:21:53

Fixes an issue where generating with the range (Int.max - 512, Int.max) could cause Builtin integer conversions to fail in the middle of random number generation.

Arrowized Materialism - 2016-12-01 15:02:17

Fixes an issue where extending the lifetime of ArrowOf<T, U> or IsoOf<T, U> could cause the underlying data table to be prematurely deallocated. (h/t @broomburgo)

Target Practice - 2016-10-09 23:26:15

  • Fixes a compatibility issue with the Swift Package Manager
  • Thanks to gyb, zip and map now extend all the way out to arity 22.
  • The default RawRepresentable arbitrary instance has been removed.

The Seven Chekras - 2016-09-16 23:21:03

⚠️ Breaking Changes Ahead ⚠️

SwiftCheck now fully supports Swift 3.0 and Xcode 8.0. That entails a bit of breakage:

  • The operator equivalents for functional combinators have been removed - these are, <^>, <*>, and >>-. Please use their nominal equivalents .map, .ap, and .flatMap respectively.
  • For long Applicative-style chains a <^> b <*> c <*> ... please use Gen.compose.
  • The little-known and barely-used exhaustive requirement of Arbitrary has been removed. Please use .once and .again.

In addition, we have a number of framework improvements:

  • Gen.withBounds and Gen.chooseAny have been added to make interacting with RandomType and LatticeType generators even easier.
  • Fixes a potential crash when interacting with randomInRange for large unsigned values.
  • Documentation coverage and robustness continues to increase with the addition of doc comments.
  • SwiftCheck now builds without dependencies.

Thank you to everybody who contributed to this release.

@griotspeak @bgerstle @kouky @EuAndreh @kballard @gfontenot @kykim

❤️

Scrap Your [Applicative] Boilerplate - 2016-08-29 16:54:18

This release includes support for Swift 2.3. In addition, we've completely revamped the way you can interact with Gen with a new combinator Gen.compose.

Most users of this framework have to generate structures that involve significant amounts of setup and state. Previously, this meant you would have to go through the Applicative generator dance much like this

MyClass.create
  <^> Int.arbitrary
  <*> String.arbitrary
  <*> Float.arbitrary
  <*> // ...

This pattern, while terse, can lead to code bloat, bad compile times, and an overall bad experience with the framework. Instead, let's rewrite this in a more natural way [and infer all the types as a bonus!]

Gen<MyClass>.compose { c in
    return MyClass(
        // Use the nullary method to get an `arbitrary` value.
        a: c.generate(),

       // or pass a custom generator
       b: c.generate(Bool.suchThat { $0 == false }),

       // .. and so on, for as many values and types as you need.
       c: c.generate(), ...
    )
}

We're going to begin the process of deprecating the old pattern and replacing it with Gen.compose.

The Exchequer - 2016-05-06 23:41:28

This is an exciting release for us, led by a ton of helpful new features and patterns to make the framework easier more Swifty. Let's get right to it:

  • We know it can be a huge pain to work with the Applicative operators. To that end, we have included an instance of Monoidal Functors (rather than the old Applicative one) for Gen that provides an overload of zip up to 10 parameters. This means code that previously looked like this:
public struct ArbitraryFoo {
    let x : Int
    let y : Int

    public static func create(x : Int) -> Int -> ArbitraryFoo {
        return { y in ArbitraryFoo(x: x, y: y) }
    }
}

extension ArbitraryFoo : Arbitrary {
    public static var arbitrary : Gen<ArbitraryFoo> {
        return ArbitraryFoo.create <^> Int.arbitrary <*> Int.arbitrary
    }
}

Can instead scrap the horrific initializer boilerplate and move straight to zip then map like so:

public struct ArbitraryFoo {
    let x : Int
    let y : Int
}

extension ArbitraryFoo : Arbitrary {
    public static var arbitrary : Gen<ArbitraryFoo> {
        return Gen<(Int, Int)>.zip(Int.arbitrary, Int.arbitrary).map(ArbitraryFoo.init)
    }
}

If you ever run out of parameters, try flatMaping generators together.

  • Enums that conform to RawRepresentable now have a default instance of Arbitrary if you ask for it. To do so, simply declare your conformance to both RawRepresentable and Arbitrary and we'll handle the rest (thanks @bgerstle!).
  • zip and ap now have much, much faster implementations.
  • Fixed a bug (h/t @bgerstle) where replay seeds and sizes would fail to print if the test failed through coverage.
  • The framework's overall documentation and code coverage continue to improve.

Check Yourself Before You Wreck Yourself - 2016-03-22 02:26:18

  • SwiftCheck now fully supports Swift 2.2 and all that entails including
    • Support for the Swift Package Manager
    • Support for building SwiftCheck on Linux
  • The order of Gen.sequence and Rose.sequence have been fixed.
  • Documentation has been added to more parts of the framework.
  • Our own test suite is now more robust.

Arbiter of Perfection - 2016-03-17 04:28:37

  • Adjusts internal RNG code for integers so it isn't subject to integer overflow conditions. This fixes #151 and may, as a side-effect, cause existing seed values created before this release to report different values as the calculations have become more robust.

Carthaginian Angst - 2016-01-22 21:54:20

  • Fixes compatibility issues with the iOS Carthage scheme (h/t @samritchie).

Harry Potter and the Half-Blood print()s - 2016-01-08 10:15:10

:warning: Breaking Changes Ahead :warning:

For SwiftCheck's 5th birthday, we sped up all the insides and moved the furniture around a bit.

  • The insides of Gen move an order of magnitude quicker because of framework-wide simplifications.
  • We now match Swift's naming convention for some big combinators (esp. map and flatMap). Update your code accordingly!
  • Currying syntax has been removed framework-wide and labels have been made more semantically meaningful.
  • Gen.fromShufflingElementsOf is unbelievably fast now.
  • pseudo-Equatable instances for modifier types have been removed.
  • More things have been documented and what documentation there is has been improved.
  • Quantifiers and property binders have more informative @warn_unused_result messages.

Finally, for the pièce de résistance, cover has been made a first-class citizen of the framework. For those times when a particular property of a spec should hold a certain percentage of the time, you can ask SwiftCheck how well it's been covered. If we don't rise to meet your set goals, the test fails!

screen shot 2016-01-08 at 2 53 31 am

Production Rights - 2016-01-04 05:21:43

We fought xcodebuild tooth and nail to bring you tvOS support. Enjoy!

Effectric Slide - 2015-12-28 22:22:46

  • Framework build times continue to improve.
  • Fixes an issue where building the framework in release mode would cause strange crashes (h/t @kykim).

We Gon' Be Alright - 2015-12-16 05:41:56

  • Fixes an issue where 32-bit devices could cause the random number generator to crash with an overflow exception.
  • Compile times for the entire framework have been significantly improved (h/t @robrix)

Pines of the Ap'ian way - 2015-11-21 20:29:16

This release fixes a major issue caused by <*> not re-jiggering its generators. This could cause tests in the previous release (v0.4.0) that use the same call in the traditional applicative form to generate the same values in many cases (h/t @QF5690 for the report).

As a consequence, generators for all types are now significantly more robust and have instances rewritten and optimized for the new deterministic RNG.

Reproductive Rights - 2015-11-19 00:30:39

SwiftCheck now has a new in-library pseudo-RNG that enables tests to be replayed argument-for-argument with just two seed values.

For example, given this property:

property("Obviously wrong") <- forAll { (x : Int, y : Int, c : Int) in (x > y) ==> x + c < y + c }

SwiftCheck takes care of printing all the nitty-gritty details.

screen shot 2015-11-18 at 7 20 33 pm

So all you have to do is feed those values back into the testing loop:

let (seedl, seedr) = (618383975, 8314)
let replayArgs = CheckerArguments(replay: .Some(StdGen(seedl, seedr), 1))
property("Obviously wrong", arguments: replayArgs) <- forAll { (x : Int, y : Int, c : Int) in (x > y) ==> x + c < y + c }.verbose

In addition, this release comes with a number of bug fixes and improvements:

  • SwiftCheck now uses the @exported attribute with XCTest so you can just import SwiftCheck without needing to import XCTest.
  • Test cases that throw exceptions now also print counterexamples. Whoops!
  • The CheckerArguments structure is now easier to create and use (h/t @kballard).
  • The StdGen structure is now public.

Exceptionally Gifted - 2015-10-15 20:31:46

With this release, all functions now catch exceptions thrown inside their respective testing blocks.

Naturally, a thrown exception is treated as an automatic failure of the test case. If you wish to treat it as a passing case, see Property.invert and Property.expectFailure.

We've also added support for the existential quantifier exists in addition to the universal forAll.

Operational Semantics - 2015-09-14 05:40:24

  • Updates operators to maintain compatibility with popular frameworks.

Check Your Privilege - 2015-09-12 18:52:27

Good News Everyone!

SwiftCheck now fully supports Swift 2.0. Other changes include:

  • The syntax for testing has changed slightly.
property["Numa Numa"] = forAll { //... }

now becomes

property("Numa Numa") <- forAll { //... }
  • We have a shiny new tutorial playground where you can explore SwiftCheck to your heart's content.
  • Support for tvOS has been added!
  • Handling of signs in signed integer generation has been fixed (h/t @brentleyjones).
  • SwiftCheck is now fully integrated with XCTest. Failures will display on the properties and lines that caused them.
  • You get an Arbitrary instance, and you get an Arbitrary instance. Everybody gets an Arbitrary instance!
  • Those classes and structures that could not be made Arbitrary now have Arbitrary-like extensions for use with forAllShrink.
  • The WitnessedArbitrary protocol has been added for higher-order types like [Arbitrary] and Set<Arbitrary>.
  • The IsoOf modifier has been added. It works like ArrowOf, but in 2 directions instead of just 1.
  • Operators have been split into their own package and standardized across TypeLift.
  • shrink is now an optional requirement for an Arbitrary instance. If none is provided, SwiftCheck will assume you don't want to shrink at all.
  • conjamb, the non-deterministic conjunction combinator, has been added.
  • The Large modifier has been added. It allows integers to be generated in their entire range rather than be bounded by their Generator's size.
  • Arguments that modify the testing loop can now be passed in a property call.
                                                  // Re-checks all tests with this generator and size.
let args = CheckerArguments(  replay: Optional.Some((newStdGen(), 10))
                                                 // Doubles the required number of tests to pass 
                        , maxAllowableSuccessfulTests: 200
                                                 // We can no longer discard.
                        , maxAllowableDiscardedTests: 0
                                                 // 10x the default size of the test case.
                        , maxTestCaseSize: 1000
                        )

property("Discards forbidden", arguments: args) <- forAll { (x : UInt) in
    return Discard()
}.expectFailure

Lazy Sunday - 2015-06-01 18:48:02

  • Makes the postcondition for ==> (Implication) lazy.

The Last Arrow Bender - 2015-06-01 03:40:44

  • ArrowOf now memoizes its arguments and behaves like a proper function.
  • Many more types have CoArbitrary instances so they can be used with ArrowOf.

Recursion: See Recursion - 2015-05-31 07:08:07

  • Adds a new combinator, Gen.weighted. It's like frequency but doesn't require so many Gen.pure calls.
  • The constraints on Gen.frequency and Gen.weighted are now relaxed to allow any SequenceType
  • The shrinker loop is thinner, quicker, and much much less memory hungry than before for complex shrinks.
  • Fixes a bug in the shrinker where tests would not report the number of shrinks they'd gone through.
  • Propagates expected failures and outputs the correct Result type for them.
  • Compile times have been significantly improved for some files (h/t @robrix).

10x Property Testing - 2015-05-29 04:17:18

  • Fixed iOS deployment target [and therefore the Carthage build]
  • Introduced 4 new operators to make complex testing possible (==>, ====, <?>, ^&&^, ^||^; Implication, Verbose Equality, Conjunction, Disjunction)
  • Generator Combinators are now static methods instead of top-level functions.
  • Numerous internal fixes to the testing loop
  • Removal of extraneous methods and data structures.
  • Array shrinks now consume less memory and complete much faster
  • Documentation has been added to every user-facing method
  • forAllShrink(_:_:_:) has a relaxed generic parameter for easier integration with custom testing
  • Verbose checks are now possible. Using verbose checks, SwiftCheck will dump all test cases to console with the result of each test.
  • When labelling properties, SwiftCheck will output a distribution graph of the percentage of passing test cases a property has gone through.
  • Conjunct and Disjunct properties used with labelling will report distribution graphs for each individual property