Property based testing for domain models

One of the challenges that we face building a non trivial domain model is to write proper tests that verify the domain rules that the model implements. The domain rules can be quite complex, may have a number of edge cases which the developer himself may fail to take care of. When you use an implementation language that supports a decent type system, many of the rules and invariants can be encoded statically within the type system itself. This makes it impossible for the programmer to write any code that violates those constraints. But you can only encode some of the domain rules as part of your static type based constraints - you need to complement them with a testing procedure that validates the semantic behavior of the model.

If you are doing testing manually, you are doing it wrong. And if you use xUnit based testing procedures, there's enough scope for you to grow up towards better frameworks that offer true automation not only with respect to executing your tests, but generating the data as well.

Frameworks like QuickCheck in Haskell or ScalaCheck in Scala allows you to write property specifications that the model should satisfy and then generate data to guide evaluation of test executions for correctness as well as completeness. Here's what Bryan O'Sullivan et al mentions when discussing property based testing in their book Real World Haskell ..

Property-based testing encourages a high level approach to testing in the form of abstract invariants functions should satisfy universally, with the actual test data generated for the programmer by the testing library. In this way code can be hammered with thousands of tests that would be infeasible to write by hand, often uncovering subtle corner cases that wouldn't be found otherwise.

I have been doing lots of domain modeling on securities trading system. The model is a quite complex one and like the most of us, I started with xUnit based testing to verify some of the invariants and constraints that the system needs to honor. But very quickly I found that properties offer a more succinct way to abstract the constraints and invariants of my model. Hence using a tool like ScalaCheck makes it much simpler once I give it a proper data generator as input. Consider the simple property in the following example ..

A trade needs to be enriched in its lifecycle with tax/fee values and other attributes. We can then compute its net value which should be a positive numeric quantity.

property("Enrichment of a trade should result in netvalue > 0") {
  forAll((a: Trade) =>
    enrichTrade(a).netAmount.get should be > (BigDecimal(0)))
}

Here I don't need to construct concrete data by hand (in fact this is what makes xUnit based frameworks a sophisticated manual testing system - it only automates the execution part of your testing). Instead what I supply is a trade generator which gives ScalaCheck enough information based on which it can generate loads of trades. A typical simplified version of the trade generator is the following ..

implicit lazy val arbTrade: Arbitrary[Trade] =
  Arbitrary {
    for {
      a <- Gen.oneOf("acc-01", "acc-02", "acc-03", "acc-04")
      i <- Gen.oneOf("ins-01", "ins-02", "ins-03", "ins-04")
      r <- Gen.oneOf("r-001", "r-002", "r-003")
      m <- arbitrary[Market]
      u <- Gen.oneOf(BigDecimal(1.5), BigDecimal(2), BigDecimal(10))
      q <- Gen.oneOf(BigDecimal(100), BigDecimal(200), BigDecimal(300))
    } yield Trade(a, i, r, m, u, q)
  }

I can make more sophisticated properties and ask the system to check whether the model satisfies the property or not ..

property("Enrichment should mean netValue equals principal + taxes") {
  forAll((a: Trade) => {
    val et = enrichTrade(a)
    et.netAmount should equal (et.taxFees.map(_.foldLeft(principal(et))((a, b) => a + b._2)))
  })
}

This is a direct encoding of the business rule, which the domain model needs to satisfy. And using properties we can encode the verification in a more declarative fashion, and that too without bothering about how to generate concrete data for all the test cases.

Crosscutting Property Verification

When we talk about properties of the model, we can go all the way and write specifications for properties at all levels of granularity. It can be a local property limited to one module or it can be a system wide property, an invariant that holds good across multiple components. What I want to say is that property based testing can be very effective in system testing (or integration testing) as well.

In our system, we have Client Orders coming in that get executed in the stock exchanges resulting in broker side executions, which then get allocated to client accounts resulting in client trades. A succinct implementation of this entire flow can be the following ..

def tradeGeneration(market: Market, broker: Account, clientAccounts: List[Account]) =
  // client orders           executed at market by broker        & allocated to client accounts
  kleisli(clientOrders) >=> kleisli(execute(market)(broker)) >=> kleisli(allocate(clientAccounts))

We can use property based specification to test this entire lifecycle using ScalaCheck. Let's check for one of the invariants in this entire process -

Invariant: "The total quantity of order will be equal to the total quantity traded"

and here's the specification of the property in ScalaCheck ..

property("Client trade allocation in the trade pipeline should maintain quantity invariant") {
  forAll { (clientOrders: List[ClientOrder], args: (Market, Account, List[Account])) =>
    whenever (clientOrders.size > 0 && args._2.size > 0 && args._3.size > 0) {
      val trades = tradeGeneration(args._1, args._2, args._3)(clientOrders)
      trades.size should be > 0
      (trades.sequence[({type λ[a]=Validation[NonEmptyList[String],a]})#λ, Trade]) match {
        case Success(l) => {
          val tradeQuantity = l.map(_.quantity).sum
          val orderQuantity = fromClientOrders(clientOrders).map(_.items).flatten.map(_.qty).sum
          tradeQuantity should equal(orderQuantity)
        }
        case _ => fail("should get a list of size > 0")
      }
    }
  }
}

The properties are the domain rules and these can be prepared by the domain experts. I am a firm believer in collaborative involvement of domain experts in building the model. I have advocated the use of domain specific languages as a thin linguistic abstraction on top of a domain model that should be built iteratively with the domain experts. Property based testing is the third piece that completes the solution framework of developing complete domain models. Property based testing strengthens this view of increased involvement of the domain people in building the software. At the end of the day, leave everything to the respective experts - the domain people specifies properties and invariants, the developer implements the specification and the underlying test framework does the heavy lifting of generating enough data and automate the execution process.