Open Classes in Ruby - Too easy to be misused ?

I read this ..

Open classes makes invention possible in real time. No committee. No waiting for the next language revision. If you think of it, you can try it. Does Symbol#to_proc make it worthwhile? How about andand? Why don’t you tell me? I was serious when I asked you what you think of it.

and soon after .. this ..

I am all with Raganwald that "dangerous features enable bottom-up language evolution". My main concern, however, is that, I have seen too much of Ruby code where people tend to open up classes without even considering other design choices. On many of these occasions did I feel that mixins could have been a more pragmatic alternative. Maybe, one reason could be that, open classes in Ruby are too easy to implement. And developers give in to the temptation of using this "dangerous feature", without giving the adequate amount of thought that the design decision deserves. Scala's lexically scoped alternative is less sexy, but looks more pragmatic.

Implementation Inheritance with Mixins - Some Thoughts

Eugene's post It is safer not to invent safe hash map / Java once again brings forth one of the most controversial topics in the OO space - public inheritance, in general and implementation inheritance, in particular. Inheritance is a hard problem to solve in OO programming. Anton Van Straaten says in this LtU thread ..

".. inheritance definitely seems to be an "unsolved problem" in OO languages. As already mentioned, it involves so many competing forces: typing, interface vs. implementation, single vs. multiple, etc. As a result, languages have typically picked some workable compromise, and lived with the limitations this produces."

And, so has Java. In Java, the recommended practice is to make heavy use of interfaces and design concrete classes using interface inheritance. But there is no elegant way to reuse interface implementations, unless you resort to tools like AOP that works on byte code weaving. People often try to use public inheritance on concrete classes in Java to add new behaviors or override existing ones and lands up losing extensibility in orthogonal directions.

Consider the abstraction java.util.Map<K,V>. The following are the various collaborations that the abstraction participates in.

• It has subinterfaces like SortedMap<K,V>, ConcurrentMap<K,V>, which implies linear extension of the contract. That is, these subinterfaces add additional behavioral contracts (no implementation) to the generic map.


• It has multiple implementations like HashMap<K,V>, TreeMap<K,V> etc.

Hence any added behavior to a generic component like Map<K,V> needs to work across all extension points of the abstraction. This is where the suggested implementations by the original authors of SafeHashMap<K,V> falls flat. As Eugene points out, extending java.util.HashMap<K,V> will result in the additional behavior NOT being implemented in other variants of Map<K,V>. While adopting a wrapper based implementation will not scale with SortedMap<K,V> and ConcurrentMap<K,V>.

And, of course, implementing the contract of Map<K,V< ground up with additional behavior will force a blatant copy-paste based implementation.

Eugene's implementation, based on static methods and camoflaged with static imports gives it the best possible shape in Java.

Can we do better ?

No, not in Java, unless you use tools like aspects, which may seem too magical to many, and of course, is never a substitute to better language features.

What we need is the ability to compose independent granular abstractions that can seamlessly mix in the main abstraction additively to introduce new behavior or override existing ones. CLOS refers to this technique as mixin based programming - Gilad Bracha, in his OOPSLA 90 paper defines a mixin as an abstract subclass that may be used to specialize the behavior of a variety of parent classes. It often does this by defining new methods that perform some actions and then calls the corresponding parent methods.

The example in CLOS, which the paper illustrates is ..

(defclass Graduate-mixin () (degree))

(defmethod display ((self Graduate-mixin))
  (call-next-method)
  (display (slot-value self 'degree)))


In the above example, the mixin method display() invokes call-next-method despite the fact that the mixin does not have any explicit parent. The parent comes in implicitly when the mixin class is mixed-in with another abstraction that supports the same method display().

(defclass Graduate (Graduate-mixin Person)())


Here Person is the superclass of Graduate and Graduate-mixin mixes in with the hierarchy to provide the additional behavior. The mixin has no independent existence and cannot be instantiated on its own. It comes to life only when it is mixed in with an existing hierarchy. Hence the term abstract subclass.

Mixins provide a notion of compositional inheritance that allow sharing of implementations along with pure interface inheritance, without the noise of an intermediate abstract class - it is like being able to provide implementation to Java interfaces. There have been a few proposals in the recent past for adding mixins in Java.

Till we have mixins in Java 7 ..

Scala provides mixin implementations as traits. Reusable behaviors can be modeled as traits in Scala and mixed in with concrete classes during class definitions as well as object creation. The most commonly demonstrated use of a mixin in Scala is the use the Ordered trait, which implements total ordering of data for the class with which it is mixed in.

class Person(val lastName: String, val firstName: String)
extends Ordered[Person] {
  def compare(that: Person): Int = {
    //.. implementation
  }
  //..
}


The parent class has to implement the compare method, but gets convenient comparison operators for free by mixing in with the trait. This is implementation inheritance without the abstract class. And the class designer can stack in multiple behaviors simultaneously by mixing in with many traits at the same time.

Scala traits also offer implementation inheritance also on an object basis, through mixins during object creation.

Consider adding a synchronized get method to Scala Maps as an additional behavior overriding the existing api. We can define a trait for this behavior ..

trait SynchronizedGet[A,B] extends Map[A, B] {
  abstract override def get(key: A): Option[B] = synchronized {
    super.get(key)
  }
}


which overrides the get method of Map with a synchronized variant.

Now we can mix in this behavior with any variant of a Scala Map, irrespective of the underlying implementation ..

// Scala HashMap
val names = new HashMap[String, List[String]]
              with SynchronizedGet[String, List[String]]

//.. use names

// wrapper for Java Map
val stuff = new scala.collection.jcl.LinkedHashMap[String, List[String]]
              do SynchronizedGet[String, List[String]]

//.. use stuff


In fact Scala provides a complete trait SynchronizedMap[A,B] that synchronizes the Map function of the class into which it is mixed in.

Mixins provide the flexibility to add behaviors to existing abstractions through implementation inheritance without some of the drawbacks of existing mainstream languages as discussed above. Unlike CLOS mixins, Scala mixins do not support chaining of methods without a context. The above mixin SynchronizedGet needs to inherit from Map in order to have invocation of super in the overridden method. This takes away some of the flexibility and makes the mixin reusable only in the context of the abstraction that it inherits from. However, like CLOS, Scala mixins are also based on the linearization technique that allows users the flexibility to manipulate the order in which the behavior chains get invoked.

Overall, I think mixins are a useful feature to have in your language and mixins in Scala offer a better way to use implementation inheritance for code reuse than Java.

Safe HashMap - The Scala Way

Erik extends java.util.HashMap to implement null safe gets and puts in Java. The following piece of code is the native Java snippet that he compacts using SafeHashMap. The code iterates through a List of objects and populates a HashMap of Lists.

HashMap<String, List<String>> h = new HashMap<String, List<String>>();

for(Thing v : list) {
    List<String> ar = h.get(v.foo);
    if (ar == null) {
        ar = new ArrayList<String>();
        h.put(v.foo, ar);
    }
    ar.add(v.bar);
}


and his version using SafeHashMap ..

SafeHashMap<String, List<String>> h = new SafeHashMap<String, List<String>>(new ArrayList<String>());

for(Thing v : list) {
    h.get(v.foo).add(v.bar);
}


Obviously his solution is verbose enough (the code for SafeHashMap has been left out for brevity) because of the lack of nullable types in Java. Scala handles this nicely through the Option data type. Here is how I would implement the same in Scala ..

val list = List(...)
val h = new HashMap[String, List[String]]()

list foreach(v => names.put(v.foo, v.bar :: h.get(v.foo).getOrElse(List[String]())))


Scala's Option data type helps model optional values. An Option can be of the form Some(x) where x is the actual value, or the None object, which represents a missing value. And the Option class also supports a method getOrElse, that allows users to return the result of evaluating a default expression in case the value is empty. The above snippet uses getOrElse to construct the List in case of missing values. Also, the default expression in getOrElse uses call-by-name and hence evaluated only if the value is empty.

Is this idiomatic enough ?

Why I like Scala's Lexically Scoped Open Classes

Ruby allows you to open up any class definition and plug in your methods into the guts of the existing class. John Carter opens up Integer and adds factorial ..

class Integer
  def factorial
    return 1 if self <= 1
    self * (self-1).factorial
  end
end


This allows him to write

puts 10.factorial


While this can lead to designing nice looking expressions of "pleasing symmetry", that read consistently from left-to-right (5.succ.factorial.succ.odd), it looks scary to me. Not the openness of the class, but the global context in which it is open.

Matz, replying to a thread, on the difference between ruby and python, mentions ..

"open class" is so strong (often too strong), we can break things easily. In other word, Ruby trust you to give you sharp knives, where Python don't. From the Python point of view, it's wrong, I guess.

In a way, I think the implementation of globally open class in Ruby brings up similar problems as the obliviousness property of aspects. You really never know what has been added to a class and whether the new implementation that you are plugging in, breaks apart any of the methods added by another developer. And it is only because all extensions in Ruby classes take place on the global namespace without any context whatsoever. Check out the discussion between Reginald Braithwaite and Avi Bryant in the comments section of the above Raganwald post.

Scalability problems in large projects ?

Scala has put in better thoughts in designing lexically scoped open classes. They call it "implicits" and allow extension of existing classes through a lexically scoped implicit conversion. Taking the example from Martin Odersky's blog, in order to add the append() method to Scala's built-in Array, define a class as ..

class RichArray[T](value: Array[T]) {
    def append(other: Array[T]): Array[T] = {
        val result = new Array[T](value.length + other.length)
        Array.copy(value, 0, result, 0, value.length)
        Array.copy(other, 0, result, value.length, other.length)
        result
    }
}


and add an implicit conversion from plain arrays to rich arrays:

implicit def enrichArray[T](xs: Array[T]) = new RichArray[T]


Now we can apply the append() method to plain arrays as well. It's not as organic as Ruby's open classes, but gives you a much better control towards evolution of APIs as your application codebase scales up. Let's see how ..

I love to program in languages that offer extensibility of abstractions - no, not the inheritance way .. public inheritance is possibly the second most tightest coupling between abstractions (guess what the first one is .. correct! .. the friend class in C++ ..). Scala offers extensibility even to classes written in Java, and with a rich repertoire of features. It almost obviates the necessity of plugging in your favorite Dependency Injection framework. And implicits offer features to add extensions to existing class structures.

Multidimensionally Open Classes ..

The core APIs that any class exposes abstract its prime responsibility, which are invariant across the entire lifecycle of the domain. This is the minimalist view of designing a class (when in doubt, leave it out !). Whether the large Array class of Ruby is the ideal approach of designing an abstraction, has been beaten to death. But definitely it is the result of a mindset that offers open classes - if you do not put assoc in Array, someone else will.

When we consider extensibility of abstractions, typically we want to extend an abstraction in multiple dimensions. A core domain object often needs to be extended with behaviors that help developers design smart APIs. But, more often than not, these behaviors make sense in a specific context of the application and may seem irrelevant or redundant in other contexts. In John Carter's earlier example, extending Integer with factorial may make sense when you are designing smart APIs for mathematical calculations. But why should an Integer class generally be bothered about computing it's own factorial ? This way, adding all possible math calculations in the global class object, can only lead to a big big bloat violating all good principles of OO design. The extension has to be in a specific context of the class and should be invisible to any other context, for which it seems like a noise.

Have a look at how Scala handles this wreck effect through a simple example ..

I have a class Person, which is a domain object in the core package ..

package org.dg.biz.core;

class Person(val lastName: String,val firstName: String,val age: Int) {
    //..
    //.. details

    override def toString(): String = {
        lastName + " " + firstName + " " + age
    }
}


While working with my UI classes, I would like to have an API person.toLabel() which will display a JLabel Swing component for rendering a person's details on a frame. And while working on the messaging part of the application, I would like to have a person.toXML(), which will help me generate an XML message out of the Person object. But obviously I would not like to have any javax.swing imports in my messaging component.

I would like to extend the same core abstraction (Person), but along two mutually orthogonal dimensions in a share nothing mode. The messaging component should not be able to invoke person.toLabel(), even though it is the same core abstraction that it is extending. The extensions have to honor the context in which they are being used. Then only can we ensure proper separation of concerns and the right modularity in designing application component boundaries.

In Ruby, class objects are global variables, and indiscriminate extension methods plugged into existing classes can lead to reduced maintenability and long term reliability of class structures. Not with Scala *implicits* though ..

Within the ui package, I define an enhancer class RichPerson, which provides all extensions to my Person class *only* for the UI context ..

package org.dg.biz.ui;

import org.dg.biz.core._
import javax.swing._

class RichPerson(person: Person) {
    def toLabel(): JLabel = {
        new JLabel(person.toString)
    }
    //.. other UI context extensions
}


and define a mixin that provides me the implicit conversion ..

package org.dg.biz.ui;

import org.dg.biz.core._

trait UIFramework {
    implicit def enrichPerson(person: Person): RichPerson = {
        new RichPerson(person)
    }
}


When I write my UI components, the extension designed for the UI context kicks in and serves me the UI-only view of the extended core abstraction. I can use the nice person.xxx() syntax on the core abstraction itself, the compiler does the magic underneath through invocation of the conversion function ..

package org.dg.biz.ui;

import org.dg.biz.core._
import java.awt.event.WindowAdapter
import javax.swing._

object Main extends UIFramework {

    def main(args: Array[String]) = {
        val p = new Person("ghosh", "debasish", 35)
        val frame = new JFrame()
        frame.addWindowListener(new WindowAdapter(){})
        frame.getContentPane().add(p.toLabel) // nice person.toLabel() usage
        frame.pack()
        frame.setVisible(true)
    }
}


Trying to use person.toXML() will result in a syntax error ! In fact the IDE can only display the available extensions as part of auto-completion features.

Similarly for the messaging component, I define the following extensions in the messaging package ..

package org.dg.biz.msg;

import org.dg.biz.core._

class RichPerson(person: Person) {
    def toXML(): scala.xml.Elem = {
        <person>
            <name>
                <lastname>{person.lastName}</lastname>
                <firstname>{person.firstName}</firstname>
            </name>
            <age>{person.age}</age>
        </person>
    }
}


and the mixin ..

package org.dg.biz.msg;

import org.dg.biz.core._;

trait MessageFramework {

    implicit def enrichPerson(person: Person): RichPerson = {
        new RichPerson(person)
    }
}


and my messaging application code ..

package org.dg.biz.msg;

import org.dg.biz.core._

object Main extends MessageFramework {

    def save(person: scala.xml.Elem) = {
        scala.xml.XML.saveFull("person.xml", person, "UTF8",
                true, null)
    }

    def main(args: Array[String]) = {
        val p = new Person("ghosh", "debasish", 41)
        save(p.toXML)
    }
}


and with Controlled Visibility ..

We extend our Person class in different dimensions for two orthogonal concerns and both of the extensions are mutually exclusive of each other. We cannot access the UI extension from within the messaging part of the application and vice versa. The visibility of the *implicit* conversion functions, along with Scala mixin techniques, ensure that the UI component *only* gets access to the RichPerson class meant for UI extension. Hence the global namespace does not get polluted and the class structure does not get bloated. Yet we have the power of open classes.

Scala - To DI or not to DI

People have been discussing about dependency injection frameworks in languages that offer powerful abstraction techniques for construction and composition of objects. Scala mailing list points towards various approaches of implementing DI capabilities using the abstraction mechanisms over types and values and great composition techniques of traits. Gilad Bracha talks about his language, Newspeak, organized around namespaces, modules and lexically scoped nestable classes. Newspeak offers powerful class instantiation and module composition features that alleviate the necessity of writing explicit static factories or magical instantiation of objects through DI techniques.

I have also been thinking about the relevance of Dependency Injection in Scala and taking cue from the ideas discussed in the Scala community, tried out some techniques in a Scala application.

One of the features that a DI framework offers is the ability to decouple explicit concrete dependencies from the abstraction of the component. This is what Martin Odersky calls the service oriented software component model, where the actual component uses the services of other cooperating components without being statically dependent on their implementations. This composition of services is typically done using a DSL like module that sets up the wiring between components by injecting all relevant dependencies into the component graph.

Do I need a separate DI framework in Scala ?

With powerful mechanisms of object construction (and built-in factory method apply() with companion objects), abstraction over types and values and flexible composition using traits, Scala offers more power than Java in decoupling of concrete implementations from the abstract services. Every component can have separate configuration units that will inject the concrete type and data members that wire up non-intrusively to deliver the runtime machinery for the particular service.

Here is an example Scala class for computing the salary sheet of employees ..

abstract class SalaryCalculationEngine {
    trait Context {
        val calculator: Calculator;
        val calendar: Calendar;
    }

    protected val ctx: Context
    type E <: Employee

    def calculate(dailyRate: Double): Double = {
        ctx.calculator.calculate(dailyRate, ctx.calendar.noOfDays)
    }

    def payroll(employees: List[E]) = {
        employees.map(_.getDailyRate).foreach(s => println(calculate(s)))
    }
}


where Calculator is defined as a trait that mixes in ..

trait Calculator {
    def calculate(basic: Double): Double
}


Notice the use of the bounded abstract type and the abstract data members as configurable points of the abstraction. A concrete implementation of the above class will inject these abstract members to complete the runtime machinery. The configurable abstract data members form the context of the abstraction and has been encapsulated into another trait. This will be mixed in with a concrete implementation as part of the configuration module. Before going into the usage of the abstraction SalaryCalculationEngine, we need to configure the abstract types and data members. Let us define a module for Class1 Employees that will supply the exact concrete configuration parameters ..

trait Class1SalaryConfig {
    val calculator = new DefaultCalculator
    val calendar = new DefaultCalendar

    class DefaultCalculator extends Calculator {
        def calculate(basic: Double, noOfDays: Int): Double = basic * noOfDays
    }

    class DefaultCalendar extends Calendar {
        def noOfDays: Int = 30
    }
}


and use this configuration to instantiate the abstraction for generating salary sheet ..

val emps = List[Employee](..

val sal = new SalaryCalculationEngine {
    type E = Employee
    protected val ctx = new Class1SalaryConfig with Context
}
sal.payroll(emps)


Note how the concrete configuration (Class1SalaryConfig) mixes-in with the Context defined in the abstraction to inject the dependencies.

We can easily swap out the current implementation by mixing in with another configuration - the MockConfiguration ..

trait MockSalaryConfig {
    type T = MockCalendar
    val calculator = new MockCalculator
    val calendar = new MockCalendar

    class MockCalculator extends Calculator {
        def calculate(basic: Double, noOfDays: Int): Double = 0
    }

    class MockCalendar extends Calendar {
        def noOfDays: Int = 10
    }
}


and the application ..

val mock = new SalaryCalculationEngine {
    type E = Employee
    protected val ctx = new MockSalaryConfig with Context
}
mock.payroll(emps)


Extensibility ..

Traits make the above scheme very extensible. If we add another dependency in the Context, then we just need to provide a configuration for it in the implementation of the config trait. All sites of usage do not need to change since the Context mixes in dynamically with the configuration.

Powerful abstraction techniques of the Scala language help us achieve easy composability of services enable swapping in and out of alternative implementations in a fairly non-intrusive manner - one of the main features thet DI frameworks offer. The configurations can be easily reused at a much more coarse level of granularity and can be kept decoupled from the main flow of the application. From this point of view, these can act similar to the Modules of Guice, as they serve as the repository for binding information.

However, standard dependency injection frameworks shine in a couple of other ways which the above abstraction techniques fail to achieve :

• DI frameworks offer a container of their own that abstracts the object creation and injection services in a manner completely decoupled from the main business logic of the application. To the application code, the developer gets a bunch of unit testable classes with all dependencies injected transparently through declarative configuration based on DSLs. Language based techniques are not that non-intrusive and often are found to tangle with the main logic of the application.


• DI frameworks like Guice and Spring have a separate lifecycle of their own and perform lots of work upfront during application initialization. This is called the bootstrapping process, when all published dependencies are analysed and recursively injected starting from the root class. Hence runtime performance is greatly enhanced, since we have the injector as well as all bindings completely set up. In the case with Scala, every class asks for the configurations - hence it is more like a Service Locator pattern.


• Modules in Guice provide a nice way to decouple compile time independent components and act as the central repository for all published dependencies that need to be injected. The Spring application context does the same thing during startup. You create a dependency one time and use it in many places - hence development scalability improves. The configuration specification in Scala is at a finer level of granularity and is strictly not centrally managed. I would love to have a DSL that allows me to manage all configurations declaratively in a centralized repository.


• Another great feature that DI frameworks provide is integration with Web components and frameworks that allow objects to be created with specialized scopes, e.g. an object per http request, or an object per user session. These facilities come out of the box and provide great productivity boost to the developers.


• Interceptors and AOP are another area where the DI frameworks shine. The following snippet from Guice manual applies a transcation interceptor to all methods annotated with @Transaction ..

binder.bindInterceptor(
    any(),                              // Match classes.
    annotatedWith(Transactional.class), // Match methods.
    new TransactionInterceptor()        // The interceptor.
);


Despite all powerful abstraction techniques present in the Scala language, I think a separate dependency injection framework has a lot to offer. Most importantly it addresses the object construction, injection and interception of lifecycle services as a completely separate concern from the application logic, leading to more modular software construction.