Sunday, September 07, 2008

More Erlang with Disco

Over the weekend I was having a look at Disco, an open source map-reduce framework, built atop Erlang/OTP that allows users to write mapper/reducer jobs in Python. Disco does not mandate any Erlang knowledge on part of the user, who can use all the expressiveness of Python to implement map/reduce jobs.

One more addition to the stack of using Erlang as middleware.

As a programmer, you can concentrate on composing map/reduce jobs using all the expressiveness of Python. Disco master receives jobs from the clients, adds them to the job queue, and makes them run on Erlang powered clusters. Each node of the cluster runs the usual supervisor-worker hierarchies of Erlang processes that fires up the concurrent processing of all client jobs. Server crash does not affect job execution, new servers can be added on the fly and high availability can be ensured through a multiple Disco master configuration.

Disco has quite a bit of overlap of functionalities with CouchDb, one of the earliest adopters of Erlang-at-the-backend with Javascript and optionally a host of your favorite languages for view processing and REST APIs.

As I had mentioned before, Erlang as middleware is catching up ..

6 comments:

Anonymous said...

Maybe I'm missing something about Disco and CouchDb, but where is the overlap of functionality? I thought Disco was a MapReduce framework while CouchDb was a document store. Seems different to me.

The only thing sort of similar is that CouchDb uses MapReduces internally to help gather results of a db query/view.

Debasish said...

I was referring to CouchDB "views", which act as the "map" part in a map-reduce system.

Anonymous said...

Ok, just checking because I thought maybe I was missing something real big here between Disco/CouchDb.

The one view "map" simililarity is a far cry from "Disco has quite a bit of overlap of functionalities with CouchDB."

Debasish said...

Maybe I should have toned down :-) actually I was too euphoric about the Erlang stuff and the fact that both use Erlang lightweight processes for the map/reduce implementation.

Anonymous said...

No, I understand the euphoria...trust me. I'm loving Erlang right now. I really just wanted to check I wasn't missing something here. :)

cyn0n said...

get a room you two!

I'm personally enjoying my erlang derldo...