Friday, February 01, 2013

Modular Abstractions in Scala with Cakes and Path Dependent Types

I have been trying out various options of implementing the Cake pattern in Scala, considered to be one of the many ways of doing dependency injection without using any additional framework. There are other (more functional) ways of doing the same thing, one of which I blogged about before and also talked about in a NY Scala meetup. But I digress ..

Call it DI or not, the Cake pattern is one of the helpful techniques to implement modular abstractions in Scala. You weave your abstract components (aka traits), layering on the dependencies and commit to implementations only at the end of the world. I was trying to come up with an implementation that does not use self type annotations. It's not that I think self type annotations are kludgy or anything but I don't find them used elsewhere much besides the Cake pattern. And of course mutually recursive self annotations are a code smell that makes your system anti-modular.

In the following implementation I use path dependent types, which have become a regular feature in Scala 2.10. Incidentally it was there since long back under the blessings of an experimental feature, but has come out in public only in 2.10. The consequence is that instead of self type annotations or inheritance I will be configuring my dependencies using composition.

Let me start with some basic abstractions of a very simple domain model. The core component that I will build is a service that reports the portfolio of clients as a balance. The example has been simplified for illustration purposes - the actual real life model has a much more complex implementation.

A Portfolio is a collection of Balances. A Balance is a position of an Account in a specific Currency as on a particular Date. Expressing this in simple terms, we have the following traits ..

// currency
sealed trait Currency
case object USD extends Currency
case object EUR extends Currency
case object AUD extends Currency

case class Account(no: String, name: String, openedOn: Date, status: String)

trait BalanceComponent {
  type Balance

  def balance(amount: Double, currency: Currency, asOf: Date): Balance
  def inBaseCurrency(b: Balance): Balance

The interesting point to note is that the actual type of Balance has been abstracted in BalanceComponent, since various services may choose to use various representations of a Balance. And this is one of the layers of the Cake that we will mix finally ..

Just a note for the uninitiated, a base currency is typically considered the domestic currency or accounting currency. For accounting purposes, a firm may use the base currency to represent all profits and losses. So we may have some service or component that would like to have the balances reported in base currency.

trait Portfolio {
  val bal: BalanceComponent
  import bal._

  def currentPortfolio(account: Account): List[Balance]

Portfolio uses the abstract BalanceComponent and does not commit to any specific implementation. And the Balance in the return type of the method currentPortfolio is actually a path dependent type, made to look nice through the object import syntax.

Now let's have some standalone implementations of the above components .. we are still not there yet to mix the cake ..

// report balance as a TUPLE3 - simple
trait SimpleBalanceComponent extends BalanceComponent {
  type Balance = (Double, Currency, Date)

  override def balance(amount: Double, currency: Currency, asOf: Date) = 
    (amount, currency, asOf)
  override def inBaseCurrency(b: Balance) = 
    ((b._1) * baseCurrencyFactor.get(b._2).get, baseCurrency, b._3)

// report balance as an ADT
trait CustomBalanceComponent extends BalanceComponent {
  type Balance = BalanceRep

  // balance representation
  case class BalanceRep(amount: Double, currency: Currency, asOf: Date)

  override def balance(amount: Double, currency: Currency, asOf: Date) = 
    BalanceRep(amount, currency, asOf)
  override def inBaseCurrency(b: Balance) = 
    BalanceRep((b.amount) * baseCurrencyFactor.get(b.currency).get, baseCurrency, b.asOf)

And a sample implementation of ClientPortfolio that adds logic without yet commiting to any concrete type for the BalanceComponent.

trait ClientPortfolio extends Portfolio {
  val bal: BalanceComponent
  import bal._

  override def currentPortfolio(account: Account) = {
    //.. actual impl will fetch from database
      balance(1000, EUR, Calendar.getInstance.getTime),
      balance(1500, AUD, Calendar.getInstance.getTime)

Similar to ClientPortfolio, we can have multiple implementations of Portfolio reporting that reports balances in various forms. So our cake has started taking shape. We have the Portfolio component and the BalanceComponent already weaved in without any implementation. Let's add yet another layer to the mix, maybe for fun - a decorator for the Portfolio.

We add Auditing as a component which can decorate *any* Portfolio component and report the balance of an account in base currency. Note that Auditing needs to abstract implementations of BalanceComponent as well as Portfolio since the idea is to decorate any Portfolio component using any of the underlying BalanceComponent implementations.

Many cake implementations use self type annotations (or inheritance) for this. I will be using composition and path dependent types.

trait Auditing extends Portfolio {
  val semantics: Portfolio
  val bal: semantics.bal.type
  import bal._

  override def currentPortfolio(account: Account) = {
    semantics.currentPortfolio(account) map inBaseCurrency

Note how the Auditing component uses the same Balance implementation as the underlying decorated Portfolio component, enforced through path dependent types.

And we have reached the end of the world without yet committing to any implementation of our components .. But now let's do that and get a concrete service instantiated ..

object SimpleBalanceComponent extends SimpleBalanceComponent
object CustomBalanceComponent extends CustomBalanceComponent

object ClientPortfolioAuditService1 extends Auditing {
  val semantics = new ClientPortfolio { val bal = SimpleBalanceComponent }
  val bal: semantics.bal.type = semantics.bal

object ClientPortfolioAuditService2 extends Auditing {
  val semantics = new ClientPortfolio { val bal = CustomBalanceComponent }
  val bal: semantics.bal.type = semantics.bal

Try out in your Repl and see how the two services behave the same way abstracting away all implementations of components from the user ..

scala> ClientPortfolioAuditService1.currentPortfolio(Account("100", "dg", java.util.Calendar.getInstance.getTime, "a"))
res0: List[(Double, com.redis.cake.Currency, java.util.Date)] = List((1300.0,USD,Thu Jan 31 12:58:35 IST 2013), (1800.0,USD,Thu Jan 31 12:58:35 IST 2013))

scala> ClientPortfolioAuditService2.currentPortfolio(Account("100", "dg", java.util.Calendar.getInstance.getTime, "a"))
res1: List[com.redis.cake.ClientPortfolioAuditService2.bal.Balance] = List(BalanceRep(1300.0,USD,Thu Jan 31 12:58:46 IST 2013), BalanceRep(1800.0,USD,Thu Jan 31 12:58:46 IST 2013))

The technique discussed above is inspired from the paper Polymoprhic Embedding of DSLs. I have been using this technique for quite some time and I have discussed a somewhat similar implementation in my book DSLs In Action while discussing internal DSL design in Scala.

And in case you are interested in the full code, I have uploaded it on my Github.


Andreas Flierl said...

That looks like a great approach. Im just slightly worried that most of the types in a programm would end up as path-dependent. The deeper the nesting, the longer the paths would become which might be kind of hard to understand.

Anyway, I'll have to try it out. Thanks for sharing!

Alois Cochard said...

That's an interesting way to avoid self annotation, even if I don't consider them to be an issue as long they are not mutually recursive.

But my main issue with cake pattern, comes from path-dependent type, I really like them when designing DSL, but I faced situation where using them for modularization went bad... I'll try to explain.

My first concern, is that if I want to now create a typeclass for BalanceComponent.Balance, I cant' do it the same way I design usually type class.

I mean the typeclass can't be declared at top level, it need to be 'in the cake', it's not a big deal but that give the feeling that something could be done in a better and more flexible way.

More generally any abstraction you'll want on a type which is part of the cake, will have to be in the cake too, and I found that frustrating and make me generate a good amount of boilerplate.

My second concern, is that you can end in situation where your abstract type become incompatible while they concrete definition are actually the same.

Let's say I have three components which consume BalanceComponent (like Portoflio): A, B and C.
They each use they own BalanceComponent, but A consume B and C as well.

Maybe I missed something in my design, but I've run to the issue where even if the concrete Balance type was the same, I wasn't able to exchange value between A/B/C because the path is different and it was not compiling.

I confess this could be seen as somewhat complex use case, but this happened to me multiple time. As for now I prefer to stay away from cake as much as possible and prefer using implicit to share collaborators (and with a bit of to ease the pain).

Unfortunatly as soon as I use a library which abuse of cake (Akka/Slick/...), I can't escape myself from entering the cake.

Would love to know what is your opinion about these kinds of issues I've run into?

pascal said...

Interesting idea!

@alois : I agree about your concerns about path-dep types potential difficulties. Need to study further. Just about your 1st concern, would it be heretic to do the following?

trait BalanceAbstract
case class BalanceCustom(amount: Double, cur: Currency, date: Date) extends BalanceAbstract

and then constrain the Balance type:
trait BalanceComponent {
type Balance <: BalanceAbstract

Then you could build your typeclass on BalanceAbstract.

(BTW I'm @mandubian on twitter ;))

Andreas Textor said...

Thank you for the interesting article. Just a quick note on a typo: In the 3rd code block, line 21, b._2 should probably be: b.currency.
Also looking forward to your reply on Alois' comment!

Debasish Ghosh said...

@Andreas Textor - Thanks for pointing out the typo. Now fixed. Will post my observations on Alois' comment soon ..

Debasish Ghosh said...

@Alois - Balance is a type abstracted within BalanceComponent. So by design I don't want the Balance type to escape the component. In this design I would like to have multiple implementations of BalanceComponent, which I can do very well as shown in the post. I think I am missing the rationale for a typeclass for Balance here.

The idea that this design espouses is that we can have multiple BalanceComponent implementations that can be flexibly plugged into Portfolio implementations without the Portfolio getting to know the details of the underlying types of Balance and yet being able to access through path-dependent types. I guess this is what we also call existential types.

Regarding your second concern, isn't the Auditing abstraction an example ? Auditing is an implementation of Portfolio and it also consumes another Portfolio. So effectively it has two BalanceComponents. But by doing val bal: semantics.bal.type (as in the code), we prevent the inconsistency. Auditing is bound to use the same variant of BalanceComponent (and hence the Balance type) of the Portfolio that it consumes. I really found this part quite illuminating in the design.

Unknown said...

Hi, interesting, that’s indeed the pattern that is used in LMS. But I don’t think it differs much from the use of selftype annotations.

Alois Cochard said...

@Pascal: yes this work as long as BalanceAbsract is *out* of the cake, what frustrated me with some cake libraries I used is that some types which doesn't use anything from the cake was part of it, making what you describe unfortunately not possible. one should be very careful about not putting everything in the cake without thikning about consequence imo.

@Debasish: About my first concern, I confess my example was a bit weird because Balance is existential, but let say you want to contstrain it using something like "BalanceLike", then it could make sense to have a typeclass for BalanceLike... that's the kind of concrete issue I had.

Regarding the second one, yes that's solve this issue, but it still quite verbose in my opinion. And in your example you have only 2 'consumers', but when you start to have a complex set of collaborators it might be tricky to define type for the one that need to be able to exchange data. Maybe it's not... I should probably experiment more.

It sounds to me we are fixing a problem we introduced by trying to solve an other one, and we end doing this "Bakery of Doom", I have the feeling there should be a better way of doing that...

Debasish Ghosh said...

@Unknown - True, not much different. However I like how val bal: semantics.bal.type in Auditing enforces that the decorator and the decorated use the same representation of Balance type. Don't think that's possible without a path dependent type.

Debasish Ghosh said...

@Alois - I am still a bit confused when you talk about typeclass for Balance or BalanceLike. For typeclass we need to have a type constructor and here we r using Balance as an abstract type within BalanceComponent. But surely you can constrain the Balance type - I think the Precog blog post that Daniel wrote used that pattern.

Alois Cochard said...

@Debasish yes my bad sorry, I meant 'type Balance[T]' / 'type Balance[T] <: BalanceLike[T]'

Adam Warski said...

Unless I'm mistaken, if there were two clients which used the BalanceComponent, the concerete instance would have to be specified in both of them, as a val?

This is quite similar to a simplistic class-with-constructor dependency injection.

In a normal cake pattern, this only needs to be done once.

rintcius said...

Nice write up, Debashish. Thanks for the food for thought! I have a couple of questions:

1) You must have another idea of the cake pattern than I have, since you say you are still applying the cake pattern. To me the basic parts of the cake pattern is the use of self-type annotations and its use of components. So in my world you would *not* apply the cake pattern here, since you are not using self-type annotations. I think I would call what you describe here "type injection" analogous to "value injection" which I described here (and I described it as moving away from the cake pattern):
As an attempt to get to a clearer set of definitions, I am curious what exactly the cake pattern is to you (cannot find an exact definition of it anywhere).

2) Do you see any use-case to give bal another value than semantics.bal in extensions of Auditing? Or in other words, what about assigning
"val bal: semantics.bal.type = semantics.bal"
in Auditing instead of ClientPortfolioAuditService1 and ClientPortfolioAuditService2?

Debasish Ghosh said...

@Adam Warski - true. In case of self annotations also you need to specify the annotation in both the clients. Admit having to declare the abstract val is a bit more of a typing. But the real value that the path dependent type based approach adds is that it allows you to be more consistent with types in the use case of a decorator as in Auditing implementation of the post. Auditing is an implementation of Portfolio and it also consumes another Portfolio. So effectively it has two BalanceComponents. But by doing val bal: semantics.bal.type (as in the code), we prevent the inconsistency. Auditing is bound to use the same variant of BalanceComponent (and hence the Balance type) of the Portfolio that it consumes. I really found this part quite illuminating in the design.

Debasish Ghosh said...

@rintcius -
1. I see the Cake pattern as an idiom where you build stuff separately and delay the mixing as late as possible. Self annotation is only one of the neat ways to do that. So I would like to view "self annotations" as one of the implementation vehicles for the pattern.

2. Yes, I could do that. But the general philosophy of the Cake pattern is to delay the concretization of instances as long as possible. Maybe in this use case it doesn't seem natural to give bal another value than semantics.bal. But there may be other use cases where it makes sense. Consider a decorator that just aggregates various semantics and adds logic on top. In that case the decorator may have some different value. The bottomline is I usually follow the principle of keeping values and types abstract as long as I can.

rintcius said...

Re 1: Is there any form of DI that you would not label as the cake pattern then?

Re 2: Yeah indeed, there's a trade-off between keeping things DRY and keeping things abstract.

Debasish Ghosh said...

@rintcius - Think functional .. DI as function currying which I blogged at and with Reader monad on which I talked at ..

rintcius said...

Hmm ... if I interpret your definition of cake pattern correctly, then it seems that what Jonas Boner described here ( as *alternatives* to the cake pattern (structural typing, implicit declarations & using Google Guice) would be part of your definition of the cake pattern.

Debasish Ghosh said...

@rintcius - Absolutely - I at least think so. If you look at the traditional definition of a design pattern, it's never tied to a particular implementation technique. I can implement a strategy either through runtime polymorphism w/ subtyping or with generics and parameteric polymorphism. So I would like to see all of these as various implementation variations of the Cake pattern.

rintcius said...

I agree and would go with your definition of the cake pattern, if only it didn't contradict with the non-cake examples that Jonas Boner gave in his article (and he seems to have coined the name, at least I haven't found an earlier reference).