More on Twitter

by Felix Francis
[Crossposted at SpreadingScience]
Twitter’s growing pains:
[Via Buzzworthy]

It’s hardly news that Twitter is experiencing growing pains, but a couple of items have appeared in recent days that shed some new light on just how bad they’re getting.
[More]

As mentioned below, some of the problems Twitter is having while trying to scale are rooted in its basic communication paradigm. It is much more complex than a system based on the telephone company. It is almost as if every phone call was a 5 or 6 person conference call.

Difficult to do with the best experts. But it sounds like Twitter was somewhat surprised by the direction its technology took and was not prepared for the type of growth it sustained. It is still a very small company and one that may not have had onboard all the engineering help it needed.

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency’s sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we’ve tried to make our system behave like a messaging system as much as possible, but that’s introduced a great deal of complexity and unpredictability. When we’re in crisis mode, adding more instrumentation to help us navigate the web of interdependencies in our current architecture is often our primary recourse. This is, clearly, not optimal.

Twitter broke ground on a new manner of using Web 2.0 tools. Time will tell if it is able to maintain its initial success.There are some very difficult problems that have to be solved. But there will be somebody who solves the scaling problem because this tool is just too useful.

Google was not the first search engine., just the best one so far. Web 2.0 works by allowing rapid prototyping of new tools as one works towards perfection. Twitter was able to accomplish a lot with really very little. It has hit a barrier now. It will be interesting to see how this problem get solved. It is not too unlikely that a user who is really knowledgeable will propose a solution.

We have kept an eye on the public discussions about what our architecture should be. Our favorite post from the community is by someone who’s actually tried to build a service similar to Twitter. Many of the best practices in scalability are inapplicable to the peculiar problem space of social messaging. Many off-the-shelf technologies that seem like intuitive fits do not, on closer inspection, meet our needs. We appreciate the creativity that the technical community has offered up in thinking about our issues, but our issues won’t be resolved in an afternoon’s blogging.

We’d like people to know that we’re motivated by the community discussion around our architecture. We’re immersed in ideas about improving our system, and we have a clear direction forward that takes into account many of the bright suggestions that have emerged from the community.

To those taking the time to blog about our architecture, I encourage you to check out our jobs page. If you want to make Twitter better, there’s no more direct way than getting involved in our engineering efforts. We love kicking around ideas, but code speaks louder than words.

That would be the Web 2.0 way.

Technorati Tags: Knowledge Creation, Social media, Web 2.0

Twitter – Good and Bad

by kevindooley
[Crossposted at SpreadingScience]
Am I Done with Facebook? Twitter FTW!:
[Via Phil Windley's Technometria]

I got a message from Facebook today saying that someone had friended me. I realized I didn’t care. Not that I didn’t care about the person who’d friended me–I didn’t care about Facebook. It’s been weeks since I was there and my life is pretty much the same.
I think the reason is Twitter. Twitter is much more social, much more interesting, and the plethora of clients (including any mobile phone with SMS) means that I don’t have to remember to go check the site to see what’s happening. Twitterific displays a solid stream of the 140 character thoughts of my friends.

Because of Twitter, today I know:
There were tornados in Denver and Laramie
Twitter posted an article about their architecture on their blog
There’s a blogger dinner tonight in Salt Lake City
@tylerwhitaker and @bradbaldwin aren’t going to carpool to the blogger dinner

I like that.

Twitter has scaling problems even though their user base is reportedly quite small. As Nik Cubrilovic points out, Twitter isn’t like WordPress or Digg. Twitter is a group forming network (GFN). When a Metcalfeian network adds another user, the number of potential connections goes from N² to (N+1)². When a GFN adds one more user, the number of potential connections goes from 2^N to 2^(N+1). In case it’s been a while since you’d done that math–it’s a big difference.

There are three important ‘Laws’ dealing with networks, social or otherwise: Sarnoff’s, Metcalfe’s and Reed’s. Sarnoff’s Law states that the value of a network (David Sarnoff started NBC) is proportional to the number of nodes. Metcalfe’s Law states that the value of a network is proportional to the square of the number of nodes. Reed’s Law states that the value of a network increases exponentially with the number of nodes.

Sarnoff’s is linear. It simply demonstrates how much NBC would make if another viewer joined the network. Since the network communicates in only one direction from a single source, the number of connections does not change much with increasing numbers of nodes.

Metcalf’s assumes the nodes can communicate with each other, resulting in multiple connections. This is where network effects come into play. One phone is not useful. Two are a little better, but a network of 10 can be very useful. Metcalf first showed this using this figure:

The line corresponds to Sarnoff’s Law. Metcalf’s, which was derived from the first Ethernet networks, shows that the value for small numbers in a communication network is not great. But this increases rapidly with larger networks.

Now Reed’s law looks at the number of groups that can be formed in a network. So take all the nodes 2 at a time, 3 at a time and so on. This results in the number growing at a rate proportional to 2^N. This is much faster growth than Metcalf’s. You can be a member of several different groups, some which have members in common and some that do not.

What this means is that in some social media settings, the number of groups can increase much faster than the number of connections. Here is a consequence of this:

To make this more real, consider TechCrunch’s twitter account. When TechCrunch, with almost 18000 followers, sends a message, that results in 18000 messages–one to each follower. This is like the phone system with infinite, always-on conference call capability. Sure, you can do things internally to collapse some messages, but you’re still dealing with exponential growth.

What is happening with Twitter, that is making it have problems scaling, is that the number of groups substantially increases the number of possible connections and messages it might have to maintain. With email, everyone on the list is sent a copy of the email, that sits on their computer and take up space. Twitter sends messages to phones, for example, possibly 18000 of them in this example.

That is a lot of wasted effort. Twitter may not be the best way to communicate with a large number of people in several different groups.

Technorati Tags: Social media, Web 2.0