Monday, April 23, 2007

Executable XML (aka Lisp)

In a project that I have been working on for quite some time, the back office system receives XML messages from the front and middle office systems for processing. It is a securities trading and settlement system for one of the big financial houses of the world - typical messages are trades, settlements, position etc. which reach the back-office after the trade is made. Like any sane architect we have designed the system based on the Java EE stack (nowadays you never get fired for choosing Java ..) centered around a message oriented middleware transporting XML messages with gay abandon. The system has gone live for many implementations and has been delivering satisfactory throughput all over.

No complaints whatsoever, on the architecture, on the Java EE backbone, on the multitudes of XML machinery that goes behind the engineering harness of the millions of messages generated every day. If I were to architect the same system today (the existing one had been architected 3 years back), I, possibly would have gone for a similar stack, just for the sheer stability and robustness of the XML based technology and the plethora of toolset that XML offers today.

Why am I writing this blog then ?

Possibly I have been having extra caffeine of late, which has been taking away most of my sleep at night. It is 1 AM in the morning and I am still glued to two of my newest possessions in my bookshelf.

I had read some parts of SICP long back - rereading it now is like a rediscovery of many of the Aha! moments that I had last time and of course, lots of ruminations and discoveries this time as well. Based on the new found lights of Lispy (and s-expy) knowledge, I hope that some day I will be able to infuse my today's dreams and rumblings into a real life architecture. I am not sure if we will ever reach the stage of human evolution when Lisp and Scheme will be considered the bricks and mortar of enterprise architecture. Till then they will exist as the sexy counterparts of Java and C++, and will continue to allure all developers who have once committed the sin of being there and done that.

The XML Message - Today's Brick and Mortar

Here is how a sample trade message (simplified for clarity) looks in our system :

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE trd01 SYSTEM "trd01.dtd">

We use XML parsers to parse this message, use all sorts of XPath expressions, XQuery and XSLT transformations to do all processing, querying and tearing apart the hierarchical structures that embody an XML message. The above XML message is *data* and we have tonnes of Java code processing the XML data, transforming them into business logic and persisting them in the database. So, we have the *code* (in Java) strictly separated from the *data* (in XML) using a small toolbox comprising of a bundle of XML parsers, XPath expressions, XSLT transformations, all packaged in a couple of dozens of third party jars (aka frameworks). The entire exercise is meant to make these *data* executable.

Executable Data - aka Lisp

Why not use a power that allows you to execute your data directly instead of creating unnecessary abstraction barriers in the name of OOP ? Steve Yeggey summarises it with elan :
The whole nasty "configuration" problem becomes incredibly more convenient in the Lisp world. No more stanza files, apache-config, .properties files, XML configuration files, Makefiles — all those lame, crappy, half-language creatures that you wish were executable, or at least loaded directly into your program without specialized processing. I know, I know — everyone raves about the power of separating your code and your data. That's because they're using languages that simply can't do a good job of representing data as code. But it's what you really want, or all the creepy half-languages wouldn't all evolve towards being Turing-complete, would they?

In Lisp, I can make the above data much more readable, clearer in intent, easier for the eyes and at the same time make it executable ..

  (id 10234)
  (trade_type "equity")
  (trade_date "2005-02-21T18:57:39")
  (instrument "IBM")
  (value 10000)
  (trade_ccy "usd")
    (settle_date "2005-02-24T18:57:39")
    (settle_ccy "usd"))))

Lisp is a language which was intended to be small with *no* syntax, where you have the power of macros to create your own syntax and roll it out into the language. I made each of the above tags separate Scheme functions (Oh! I was using Scheme btw), each one of which is capable of transforming itself into the desired functionality. As a result, the above data is also my code and directly executes. Another of those Aha! moments.

But my entire system is based on Java! Surely you are not telling me to change all of the guts to Scheme - are you ? My job will be at stake and I will never be able to convince my pointy haired boss that the trading back office system is running on Lisp. In fact, many people who dare to use Lisp in daytime projects are often careful to keep this a secret in the industry. And unless you have PG on your company board or blessed enough to get the favor of Y, this is a very useful tip.

Enter SISC

SISC is a lightweight, platform independent Scheme system targetting the Java Virtual Machine. It comes as a lightweight distribution (the core jar is 233 KB) and offers Scheme as a scripting language for Java. In SISC bridging is accomplished by a Java API for executing Scheme code and evaluating Scheme expressions, and a module that provides Scheme-level access to Java objects and implementation of Java interfaces in Scheme.

I can write Scheme modules and load it using Java api from my Java code, once I have bootstrapped the SISC runtime. Here is how I can initialize SISC from my Java application so as to enable my application use the Scheme functions :

// bootstrapping the SISC runtime
SeekableInputStream heap = new MemoryRandomAccessInputStream(
AppContext ctx = new AppContext();
Interpreter interpreter = Context.enter(ctx);

and then I can use on the fly evaluation of Scheme functions as follows :

interpreter.eval("(load \"trd01.scm\")");
String s = interpreter.eval("(trd01 (id 10234) (trade_type "equity") ...)").toString();

There are quite a few variants of eval() that SISC offers, along with multiple modes of execution of Scheme code from the Java environment. For details have a look at their documentation. SISC also offers calling Java code from Scheme accessing Java classes through the extensible type system of SISC.

I do not dream about using SISC in any production code in the near foreseeable future. But just wanted to share my rants with all of you. In today's world, it is really raining programming languages - all scripting languages like Ruby, Python, JRuby, Groovy etc. are making inroads as the preferred glue language of today's enterprise architecture. But Lisp stands out as a solid robust language, with the exceptionally powerful code-as-data paradigm - I always felt Lisp was way ahead of its time. Possibly this is the time when Lisp needs to be reincarnated, all incompatibilities should be rubbed off the numerous versions and dialects of Lisp. Lisp code is executable data - it makes perfect sense to replace all frameworks that execute reams of code to process hierarchical structures as data by a single language.


Anonymous said...


Fabien said...

If you want code-as-data while keeping a regular lua/python/ruby/groovy-like language, you might be interested by metalua.

It adds code-as-data and compile-time macros to Lua, whose semantics is already very close to Scheme's (except that it didn't enjoy macros until now).

The ML-inspired structural pattern matching sample provided with it is a good base to develop a developer-friendly XLST replacement: you can look at OCamlDuce if you want inspiration about adapting ML pattern matching to XML munching.

Anonymous said...

Charles Miller Confluence lead dev oops architect had a strikingly similar blog entry nearly four years ago

J. Aaron Farr said...

another solution for the JVM is Scala which is functional and has great XML support. See:


manish said...

nice :) I haven't worked in lisp. But your idea seems nice.. I realized that something similar could also be done in JavaScript. Where any XML can be easily transformed to JS objects and while transforming functions could be added to these objects which do the functionality u want it to do.. Hence, once this JS code gets evaluated.. it will do what u want it to do.. what say?

Unknown said...'s_Tenth_Rule

Anonymous said...

You might also find Dynamic XML of interest where you can create a dynamic DOM.
Checkout this technology:

Hierarchical Inherited Rule-Interpreted XML

Andrew Sazonov said...

Well, it seems that what are you dreaming of is very close to JSON (JavaScript object notation) in which data are stored in JavaScript object in XML like way.

Btw, I suppose that it's possible to have such a trick with self-executable messages in Java too - it's possible to generate Java class which contains data and processing code for it dynamically, than compile it on the fly using JDK compiler and, finally, send that class file to another place for further execution (like some agent).

It's simple scheme, it could be code d quite fast, but I suppose that it has some serious security limitations.

Alex Wei said...

I don't know Lisp at all. However, if you want code-as-data, have you looked into Groovy Builder? A good examples showing off what Groovy Builder can do is the domain class constraint in Grails...

Unknown said...

[for manish]: I admit similar things can be done using Javascript, XML, transforming XML into JS objects etc. But Lisp offers me more power, a uniform code-as-data semantics - why go into the trouble of transformation at all ? The Lisp data (as I showed in the post) is much more soothing for the eyes than XML, much more readable, and on top of that, it executes!


Unknown said...

[for fabien]: I have been deperately looking for some excuses to learn Lua. Now with metalua, I have one .. I will definitely look into it. Thanks for the pointer.

Unknown said...

This entry has been picked up by programming.reddit. Follow the interesting comments posted here.

Anonymous said...

The .NET platform offers custom expression types in config files, which can be piped right into runtime compilation stuff (CodeDOM) or handled as desired.

3.0 will also allow the "reverse", code can be trapped and handed to you as an expression tree instead of treated normally. You can then optimize it and hand it on to .NET or you can handle it with your own interpreter.

Anonymous said...

Unotal  is like S-expressions enriched with explicit attributes and types. The Java-Implementation is called Junotal. A Junotal-Tutorial is half-finished.

Anonymous said...

With Lisp you actually can make XML executable by turning Lisp (via readers and printers) into Lisp expressions. We call them X-expressions. XMLisp will automagically turn X-expressions into CLOS objects and turn CLOS objects into XML.

Anonymous said...