Friday, March 20, 2009

Now serving - Message Queuing in Web Applications

Looks like asynchronous messaging is becoming quite a common paradigm of distribution in Web applications and large Web sites. The general theory is to do the absolutely essential user requirement stuff as part of the synchronous HTTP request/response cycle. And delegate the rest offline for the asynchronous queuing service to process. This leads to faster page reloads, optimal resource utilization and much better user experience. Gojko Azdic has a great post clarifying the misconceptions around messaging systems, as being only useful for big investment banks.

And here are some real life use cases ..

  • Flickr uses offline queuing systems to process notifications to contacts and third party partners when a user uploads a photo. Informing the user is of prime importance which Flickr does as part of the synchronous request/response processing, while the other notifications are processed by job queues running in parallel.

  • Digg uses Gearman to farm out similar processing of jobs in parallel while serving the absolute essential stuff synchronously.

  • And as recently presented in QCon by Evan Weaver, Twitter is becoming more and more centered around messaging middleware. Every incoming tweet triggers the messaging system for processing notifications to all followers, and this is done asynchronously through Kestrel, an implementation of a message queue written in Scala.

All these use cases lead to the conclusion that message queuing is becoming an increasingly potent medium of distribution even in Web based applications. The moot point is eventual consistency, that leads to a better scalability model. And, by the way, it looks to be about time that open source messaging platforms start getting more steam, so that every house does not have to reinvent the wheel and develop its own message queuing infrastructure.


grove said...

Which do you think are the hottest open source candidates?

victori said...

We use activemq on Works well when you have a user sending out a PM to his or her 10,000 or so friends. I honestly don't see why the ruby camp had to re-invent the wheel with starling (twitter) or beanstalkd. ActiveMQ is cross-platform, supports multiple languages via the STOMP protocol.

Just my two cents.

Anonymous said...

This article is bordering on the silly given the obviousness.

Thanks for the tip. Whats alot more interesting is to look at systems that use alot of JMS and what a nightmare they are to maintain.

And also the complete and utter failure of message bus systems enterprise vendors have been hawking for years.

Truly asynchronous processing (as opposed to the simple batch examples you provide) is a necessary evil, but its not cool, fun or even to be described as "good".

Justin Camerer said...

i also have not been convinced by any of the current open source queues that i have come across. between this and me wanting to continue learn erlang, i have started a restful queue in erlang. it is definitely not anywhere near stable, but i hope to make it so before too long. have any bright ideas or comments?

Spock said...

rabbitMQ ?