Introduction

Testing is widely accepted practice in software industry. I am an iOS Engineer and have been writing tests, like most of us. The way I approach testing changed radically a few years back. And I have used and shared this new technique for a few years within Zalando and outside. In this post, I will explain what is wrong with most test cases and how to apply randomized input to improve tests.

This is our sample code under test:

struct DomainStore {
    private let internalStorage = UserDefaults.standard

    func set(value: String, for key: String) {
        internalStorage.set(value, for: key)
    }

    func get(for key: String) -> String? {
        internalStorage.value(for: key) as? String
    }
}

The usual testing approach

func test_setValueCanBeRetrieved() {
    let storage = DomainStore()

    storage.set(value: "Zalando", for: "companyName")
    let obtained = storage.get(for: "companyName")!
    XCTAssertEqual("Zalando", obtained)
}

Imagine someone opens your code a few months down the road and modifies the code under test ever so slightly.

struct DomainStore {
    private let internalStorage = UserDefaults.standard

    func set(value: String, for key: String) {
        internalStorage.set(value, for: key)
    }

    func get(for key: String) -> String? {
        return "Zalando"        // Note
    }
}

This diligent test runs on your machine or on CI and it passes. Does it mean the production code works fine? Of course not. Most Test Driven Development (TDD) practitioners would move past this DomainStore but, should you? How can we reveal similar quality issues and address them?

Fundamentally we are testing using constant String while the production method suggests it can take any String.

When we check this function signature.

func set(value: String, for key: String)

It tells it can take any String instance. Not just "Zalando". However, our previous test asserted on only 1 instance of String type.

Better approach: Feed Randomized Input to test cases

The fundamental idea of this technique is never to feed test cases hand typed constants. What do we feed in then? Welcome randomness.

This is our fixed test case.

func test_setValueCanBeRetrieved() {
      let storage = DomainStore()

      let value = String.random  // Note
      let key = String.random

      storage.set(value: value, for: key)
      let obtained = storage.get(for: key)!
      XCTAssertEqual(value, obtained)
}

Note:

• String.random produces random instance of a String. At Zalando, we use this Randomizer library for generating random inputs. It covers most the used types in the Standard Library.
• If Randomizer doesn’t fit your need, feel free to extend it or add your custom conformance to Random protocol requirement.

Now the above tempered code will not pass through this test case. Unless we run it, we don’t know ahead of time what values we are going to test with. And these values are different across runs. Effectively exercising our production code with many permutations of possible values. This is the essence of randomized input tests (sometimes referred to as permutation tests).

Going beyond a simple case

Here’s one example test case from our module. The code below creates random label component and sets random accessibility options on model layer, then asserts if the rendered view has correct accessibility information.

func test_whenAccessibilityProvided_andComponentHasTapAction_thenAccessibilityIsSet() {
        let props = LabelProps.random
        let accessibilityModel = APIAccessibility.random
        let component = LabelComponent(
          componentId: .random,
          flex: .random,
          actions: .random,
          props: props,
          accessibility: Accessibility(with: accessibilityModel, componentType: .label(props)),
          debugProps: DebugProps()
        )
        let node = MockNode()
        component.actions = [EventType.tap: [ComponentAction(.random, .log(.random))]]

        component.updateAccessibility(node)

        XCTAssertTrue(node.isAccessibilityElement)
        XCTAssertEqual(node.accessibilityLabel, accessibilityModel.label)
        XCTAssertEqual(node.accessibilityHint, accessibilityModel.hint)
        XCTAssertTrue(node.accessibilityTraits.contains(.staticText))
        XCTAssertTrue(node.accessibilityTraits.contains(.button))
}

Note:

• User defined types (usually Structs) are composed of standard library types and predefined custom types. We can extend user defined types in our test target to conform to Random. An example conformance of LabelProps is as below:

struct LabelProps: Codable, Hashable {

    let text: String
    let backgroundColor: String?
    let font: FontProps

}

extension LabelProps: Random {
    public static var random: LabelProps {
        return LabelProps(text: .random, backgroundColor: .random, font: .random)
    }
}

• We could do code generation on build phase to synthesize the Random conformance. Although this is out of scope of this post, its how Equatable conformance works.
• Due to Swift’s type inference; .random will use the exact type’s random conformance.
• For cases where we need to compare against input value, we can store the generated model into a local property. Like we did for accessibilityModel.
• There are times when function under tests expects Email, URL, Deeplink or PhoneNumbers. These data types are often represented by String. However, String.random is not good enough on this case. There are 2 ways of tackling this. One is to extend String to have String.randomEmail. Another is to create concrete type which conforms to Random.

Conclusion

This technique was not my realization. I grasped the phrase “Don’t use constants on tests” from Jorge Ortiz during his workshop on Clean Architecture on Swift Averio, 2017. It then changed the way I write tests. I hope this technique will help you too.

The technique of permutation testing by using random input applies to all software testing; not just iOS development. The only requirement is Type.random.

We're hiring! Do you like working in an ever evolving organization such as Zalando? Consider joining our teams as a Mobile Engineer!