Sunday, June 21, 2009

scouchdb Views now interoperable with Scala Objects

In one of the mail exchanges that I had with Dick Wall before the scouchdb demonstration at JavaOne ScriptBowl, Dick asked me the following ..

"Can I return an actual car object instead of a string description? It would be killer if I can actually show some real car sale item objects coming back from the database instead of the string description."

Yes, Dick, you can, now. scouchdb now offers APIs for returning Scala objects directly from couchdb views. Here's an example with Dick's CarSaleItem object model ..

// CarSaleItem class
@BeanInfo
case class CarSaleItem(make : String, model : String, 
  price : BigDecimal, condition : String, color : String) {

  def this(make : String, model : String, 
    price : Int, condition : String, color : String) =
    this(make, model, BigDecimal.int2bigDecimal(price), condition, color)

  private [db] def this() = this(null, null, 0, null, null)

  override def toString = "A " + condition + " " + color + " " + 
    make + " " + model + " for $" + price
}


The following map function returns the car make as the key and the car price as the value ..

// map function
val redCarsPrice =
  """(doc: dispatch.json.JsValue) => {
        val (id, rev, car) = couch.json.JsBean.toBean(doc, 
          classOf[couch.db.CarSaleItem]);
        if (car.color.contains("Red")) List(List(car.make, car.price)) else Nil
  }"""


This is exciting. The following map function returns the car make as the key and the car object as the value ..

// map function
val redCars =
  """(doc: dispatch.json.JsValue) => {
        val (id, rev, car) = couch.json.JsBean.toBean(doc, 
          classOf[couch.db.CarSaleItem]);
        if (car.color.contains("Red")) List(List(car.make, car)) else Nil
  }"""


And now some regular view setup code that registers the views in the CouchDB design document.

// view definitions
val redCarsView = new View(redCars, null)
val redCarsPriceView = new View(redCarsPrice, null)

// handling design document stuff
val cv = DesignDocument("car_views", null, Map[String, View]())
cv.language = "scala"

val rcv = 
  DesignDocument(cv._id, null, 
    Map("red_cars" -> redCarsView, "red_cars_price" -> redCarsPriceView))
rcv.language = "scala"
couch(Doc(carDb, rcv._id) add rcv)


The following query returns JSON corresponding to the car objects being returned from the view ..

val ls1 = couch(carDb view(
  Views builder("car_views/red_cars") build))


On the client side, we can do a simple map over the collection that converts the returned collection into a collection of the specific class objects .. Here we have a collection of CarSaleItem objects ..

import dispatch.json.Js._;
val objs =
  ls1.map { car =>
    val x = Symbol("value") ? obj
    val x(x_) = car
    JsBean.toBean(x_, classOf[CarSaleItem])._3
  }
objs.size should equal(3)
objs.map(_.make).sort((e1, e2) => (e1 compareTo e2) < 0) 
  should equal(List("BMW", "Geo", "Honda"))


But it gets better than this .. we can now have direct Scala objects being fetched from the view query directly through scouchdb API ..

// ls1 is now a list of CarSaleItem objects
val ls1 = couch(carDb view(
  Views builder("car_views/red_cars") build, classOf[CarSaleItem]))
ls1.map(_.make).sort((e1, e2) => (e1 compareTo e2) < 0) 
  should equal(List("BMW", "Geo", "Honda"))


Note the class being passed as an additional parameter in the view API. Similar stuff is also being supported for views having reduce functions. This makes scouchdb more seamless for interoperability between JSON storage layer and object based application layer.

Have a look at the project home page and the associated test case for details ..

Thursday, June 18, 2009

Scala/Lift article available as a podcast

Myself and Steve Vinoski's article in IEEE Internet Computing (May/June issue) titled "Scala and Lift - Functional Recipes for the Web" is now available as a podcast. Here it goes ..

Thanks Steve, for the effort ..

Sunday, June 14, 2009

Code Reading for fun and profit

I still remember those days when APIs were not so well documented, and we didn't have the goodness that Javadocs bring us today. I was struggling to understand the APIs of the C++ Standard Library by going through the source code. Before that my only exposure to code reading was a big struggle to pile through reams of Cobol code that we were trying to migrate to the RDBMS based platforms. Code reading was not so enjoyable (at least to me) those days. Still I found it a more worthwhile exercise than trying to navigate through inconsistent pieces of crappy paperwork and half-assed diagrams that project managers passed on in the name of design documentation.

Exploratory Code Reading ..

C++ Standard library and Boost changed it all. C++ was considered to be macho enough those days, particularly if you can boast of your understandability of the template meta-programming that Andrei Alexandrescu first brought to the mainstream through his columns in C++ Report and his seemingly innocuously titled Modern C++ Design. Code reading became a pleasure to me, code understanding was more satisfying, particularly if you could reuse some of those code snippets in your own creations. It was the first taste of how dense C++ code could be, it was as if every sentence had some hidden idioms that you're trying to unravel. That was exploratory code reading - as if I was trying to explore the horizons of the language and its idioms as the experts documented with great care. I subscribed to the view that Code is the Design.

Collaborating with xUnit ..

Then came unit testing and the emergence of xUnit frameworks that proved to be the most complete determinants of the virtues of code reading. Code reading changed from being a passive learning vehicle to an active reification of thoughts. Just fire up your editor, load the unit testing framework and validate your understanding through testXXX() methods. It was then that I realized the wonders of code reading through collaboration with unit testing frameworks. It was as if you are doing pair programming with xUnit - together you and your xUnit framework are trying to understand the library that you're exploring. TDD was destined to be the next step, the only change being that instead of code understanding you're now into real world code writing.

Code Reading on the GO ..

Sometimes I enjoy reading code when I'm traveling or in a long commute. It's not painstaking, you do not have any specific agenda or you're not working against a strict timeline for the project. I found this habit very productive and in fact learnt quite a few tricks of the trade in some of these sessions. I still remember how I discovered the first instance of how to implement the Strategy pattern through Java enums browsing through Guice code in one of the flights to Portland.

Code Reading towards Polyglotism ..

When you're learning a new language, it helps a lot looking at existing programs in languages that you've been programming for long. And think how you could model it in the new language that you're learning. It's not a transliteration, often it results in knowing new idioms and lots of aha! moments as you explore through your learning process. This is one of the most invaluable side-effects of code reading - reading programs in language X makes you a better programmer in language Y. Stuart Halloway in his book on Clojure programming gives a couple of excellent examples of how thinking functionally while reading Java code makes you learn lots of idioms of the new paradigm.

Reading bad code ..

This is important too, since it makes you aware you of the anti-patterns of a language. It's a common misconception that using recursion in functional programs makes them more idiomatic. Recursion has its own problems, and explicit recursions are best hidden within the language offered combinators and libraries. Whenever you see explicit recursion in non trivial code snippets that can potentially get a large data set, think twice. You may be better off refactoring it some other way, particular when you have an underlying runtime that does not support tail call optimization. Code that do not read well, are not communicative to users. Code reading makes you aware of the importance of expressiveness, you realize that you'd not write code that you cannot read well.

Well, that was a drunken rant .. that I wrote as a side-effect in the midst of reading the Scala source for 2.8 Collections ..

Sunday, June 07, 2009

scouchdb Scala View Server gets "reduce"

scouchdb View Server gets reduce. After a fairly long hiatus, I finally got some time to do some hacking on scouchdb over the weekend. And this is what came out of a brief stint on Saturday evening ..

map was already supported in version 0.3. You could define map functions in Scala as ..

val mapfn = """(doc: dispatch.json.JsValue) => {
  val it = couch.json.JsBean.toBean(doc, classOf[couch.json.TestBeans.Item_1])._3;
  for (st <- it.prices)
    yield(List(it.item, st._2))
}"""


Now you can do reduce too ..

val redfn = """(key: List[(String, String)], values: List[dispatch.json.JsNumber], rereduce: Boolean) => {
  values.foldLeft(BigDecimal(0.00))
    ((s, f) => s + (match { case dispatch.json.JsNumber(n) => n }))
}"""


attach the map and reduce functions to a view ..

val view = new View(mapfn, redfn)


and finally fetch using the view query ..

val ls1 =
  couch(test view(
    Views.builder("big/big_lunch")
         .build))
ls1.size should equal(1)


reduce, by default returns only one row through a computation on the result set returned by map. The above query does not use grouping and returns 1 row as the result. You can also use view results grouping and return rows grouped by keys ..

val ls1 =
  couch(test view(
    Views.builder("big/big_lunch")
         .options(optionBuilder group(true) build) // with grouping
         .build))
ls1.size should equal(3)


For a more detailed discussion and examples have a look at the project home page documentation or browse through the test script ScalaViewServerSpec.

The current trunk is 0.3.1. The previous version has been tagged as 0.3 and available in tags folder.

Next up ..

  • JPA like collections of objects directly from scouchdb views

  • more capable reduce options (rereduce, collations etc.)

  • replication

  • advanced exception management with new dbDispatch


.. and lots of other features ..

Stay tuned!

Wednesday, June 03, 2009

scouchdb @ JavaOne

JavaOne script bowl was organized as a panel session to show off different scripting languages on the JVM. Tha languages considered were Jython, Groovy, Scala, JRuby and Clojure. As part of the Scala show, Dick Wall demonstrated scouchdb, the Scala driver for CouchDB. Cool .. and thanks Dick for choosing scouchdb ..

Alex Miller has more details here ..

Monday, June 01, 2009

Prototypal Inheritance in Javascript - Template Method meets Strategy

I have been reading some of the papers on Self, a programming environment that models computation exclusively in terms of objects. However unlike the classical object-oriented approach, Self is a classless language, where everything is an object. An object has slots - each slot has a name and a value. The slot name is always a String, while the value can be any other Self object. The slot can point to methods as well, consisting of code. A special designated slot points to the parent object in the hierarchy. Hence each object is consistently designed for extensibility through inheritance. But since we don't have class structures, everything is dynamic and runtime. Objects interact through messages - when an object receives a message, it looks up into its slot for a match. If the matching message is not found, the search continues up the chain through successive parent pointers, till the root is reached.

Prototype based languages offer a different way of implementing objects, and hence require a different thinking for structuring your programs. They make you think more in terms of messages that your objects will receive, and how the messages get propagated up the inheritance chain.

Javascript follows an almost identical architecture, where the hierarchies of objects are constructed through prototypes. This post is not about Self or, for that matter, about the Javascript language. Some time back I had blogged about how the Template Method design pattern gets subsumed into higher order functions and closures when implemented using functional programming languages.

In a class based language, template method pattern is implemented in terms of inheritance, which makes the structure of the pattern static and makes the derivatives of the hierarchy statically coupled to the base abstraction. Closures liberate the pattern structure from this compile time coupling and make it dynamic. But once we take off the class inheritance part and use higher order functions to plug in the variable parts of the algorithm, what we end up with closely matches the Strategy pattern. Have a look at James Iry's insightful comments in my earlier post.

James also hinted at another level of subsumption which is more interesting - the case of the two patterns implemented in a prototype based language like Javascript. Here is how it looks ..

// the template function at the base object
// defines the generic flow
// uses hooks to be plugged in by derived objects

var processor = {
  process: function() {
    this.doInit();
    this.doProcess();
    this.doEnd();
    return true;
  }
};


We construct another object that inherits from the base object. The function beget is the one that Douglas Crockford defines as a helper to create a new object using another object as the prototype.

if (typeof Object.beget !== 'function') {
  Object.beget = function(o) {
    var F = function() {};
    F.prototype = o;
    return new F();
  };
}

var my_processor  = Object.beget(processor);


The new object now implements the variable parts of the algorithm.

my_processor.doInit = function() {
  //..
};
my_processor.doProcess = function() {
  //..
};
my_processor.doEnd = function() {
  //..
};


and we invoke the function from the base object ..

my_processor.process();

If we need to define another specialization of the algorithm that only has to override a single variable part, we do it likewise by supplying the object my_processor as the prototype ..

var your_processor= Object.beget(my_processor);
your_processor.doEnd = function() {
  //.. another specialization
};

your_processor.process();


So what we get is a dynamic version of the Template Method pattern with no static coupling - thanks to prototypal inheritance of Javascript. Is this a Template Method pattern or a Strategy pattern ? Both get subsumed into the prototypal nature of the language.