Twitter jilts Ruby for Scala

Web 2.0 Expo Famously, when Twitter's app was suddenly embraced by digerati+dog in late 2007, its original Ruby on Rails architecture had keeping up the endless stream of digital solipsism.

But in mid-2008, the web's startup-of-the-moment quietly ported some of its core code to Scala, and according to Twitter developer Alex Payne, the switch should stand as a lesson to cutting-edge coders everywhere.

Addressing developers at the annual Web 2.0 Expo in San Francisco, California, Payne argued that Scala - not Ruby on Rails or PHP or Python - should be the language of choice for the modern web startup.

Today, Payne said, most of the hip dev set codes in Ruby or PHP or Python because they're perceived as "agile" languages suited to quick site changes - but also because the deverati grew bored with languages like Java and C++. The trouble, he insisted, is that the so-called Web 2.0 languages aren't always as efficient as they need to be, especially in an economic environment threatening to bring Web 2.0 to its knees.

"Investors now want to know that you're not going to be paying tons and tons of money for servers because you decided to build in a stack that isn't as efficient as possible," he said.

"In a down market, you may have to make choices between technologies you love to work with and technologies that will keep the lights on. The question is... whether we can afford to build things that we love with tools that we love. The answer is 'Yes'. But it means adopting a different set of tools."

For Payne, who oversees Twitter's API development, your ticket to the web promised land is Scala, the next-generation language developed by Martin Odersky at Switzerland's École Polytechnique Fédérale de Lausanne.

Sometime in 2007, Twitter came to the realization that its Ruby-based message queuing system couldn't keep up with all that Web2.0rhea. "It grew with us for a number of months, and then, at a certain point, it just hit a wall," Payne said. "And that wall was not so much it code but in limitations in Ruby virtual machine. There's a lot of things that Ruby is great at, but long running processes? Particularly memory intensive ones? Not so much."

By mid-2008, one of these Ruby message queues completely crashed and developers needed two and a half hours to shove the dropped Tweets back through the system. When your game is micro-blogging, that's a lifetime. Then, in his spare time, one developer ported the code to Scala. According to Payne, the Scala queue could process the same message backlog in 20 seconds.

The new system was in place before the end of last year, Payne told The Reg, just in time for the Twittering chaos of the Barack Obama's inauguration.

By the end of this year, Payne said, Twitter hopes to have its entire middleware infrastructure and its APIs ported to the new language. Ruby will remain, but only on the front end. "We're still happy with Rails for building user facing features... performance-wise, it's fine for people clicking around web pages. It's the heavy lifting, asynchronous processing type of stuff that we've moved away from."

For this sort of heavy lifting, languages like Java and C++ may be as fast as Scala. But Payne wants more than speed. He prefers Scala because it combines functional and object-oriented techniques. "Functions are objects, and objects are functions," he said. "Scala asks can you have the best of these two worlds together? And you can."

But he also prefers Scala because it's, well, beautiful. "It's a fuzzy thing. But we like writing beautiful code, code that you're proud of, code that you can show to non-programmers and they get it."

When Twitter made the Scala switch, it wasn't using Java Virtual Machines (JVMs). But, Payne argues, the language is that much more attractive if you're already running Java.

"The biggest selling point for certain organizations is that Scala is 100 per cent compatible with your existing JVM code. There's no wrappers. There's no bullshit. You just load up your JVM code and call in to it, and there's no performance penalty. If you have a big investment in Java infrastructure, Scala is the ideal language."

But if you don't, Payne will tell you much the same thing. Clearly, he's in love. And he's pitching his new Scala book. And his new Scala book is published by O'Reilly. And O'Reilly runs the Web 2.0 Expo. ®