Monday, July 26, 2010

Your Gold Dollars at Work


Two and a half weeks ago, we put out a call for help, in the form of reddit gold pay-what-you-want subscriptions. Last week we solidified a price point. But throughout the whole process, many of you asked, "Where is my money going?" Some of you came up with your own theories (or conspiracies), but today I am here to tell you what actually happened in what we hope is just the first of many "Your Gold Dollars at Work" posts.

Cassandra

As many of you know, we had a few problems with Cassandra recently. The best Cassandra description I've heard yet was when we visited Digg: Distributed win means distributed fail. Truer words have never been spoken. Such is life when using beta software.

Many of those problems, we believe, are because we were running a cluster that was too small for what we are doing with it. With the reddit gold revenue coming in, we were able to get approval to double the size of our Cassandra cluster, from 6 to 12 machines. We are also in the process of more than doubling the amount of storage each node has, from 200GB to 500GB.

Some of this expansion was to support the new profile page sorts. However, most of the expansion was just to make the site faster and more stable for everyone. By doubling the number of nodes, it means that read loads are spread out better, and it also means we are more resilient to failure.

Application servers

Before the Cassandra expansion, we undertook a different expansion. Just four days after the announcement of reddit gold, we added three new application servers, increasing our capacity by 13%. This expansion was done for the benefit of everyone, since none of the reddit gold features had even been written yet!

New furniture

We figured you all wouldn't mind if we sit in comfort, so we also picked up four solid gold chairs for our office. Actually, that's not true. Besides the huge slap in the face that would be to everyone, and how uncomfortable that would be for us, we would also be breaking our promise. We promised that we would use the reddit gold money for improving the site, and that is what we intend to do, mostly through increased server budget.

The Plan

Our vision for reddit gold is to make it, first and foremost, an early beta program. Yes, there will be a few things that are for subscribers only (like turning off ads), and perhaps a few more that could never scale to everyone (like .. well, we haven't thought up anything like that yet), but mostly, it'll be things that we have to release to a limited set of folks first so that we can figure out HOW to scale it.

In other words, you don't have to worry about us turning into some crappy paywall site. We're not shooting to have all our of active members turn into paying subscribers. We're not even trying to get half that. In fact, what we'd absolutely love is for about 2% of our eight million active users to subscribe to reddit gold. That would be an annual income stream of almost $5 million, which would solve all of our problems many times over.

Please note that it would also mean that 98% of our users would not be subscribers. We're not going to leave 7.8 million people in the dust. Our promise to you is that if you don't subscribe, reddit's going to remain pretty much the same as it already is. And in a lot of ways, it'll get better.

What's next?

We have a lot more improvements still to make under the hood. The first thing is to expand our storage for our vote data. Apparently a lot of you are really into this whole "voting" thing, and 500GB just isn't enough storage for that anymore. Beyond that, it depends a lot on you guys, and what pain point you find in our system next!

But reddit is still slow!! You're a bunch of liars!

Obviously, we haven't solved all of our problems in just two weeks. As usual, you guys keep helping us find new and unusual ways for our site to break, and today happens to be one of those days (although we think we fixed it by rebalancing our server pools a little bit). We'll keep doing our best to make things as stable and quick as possible, and we'll be able to respond a lot faster now that we have a bit of money to throw at the problem. Please do continue to tell us when we're letting you down.

So where are we now?

Since a lot of people ask me this often, here are some quick stats on our cluster right now:

80 Server Instances
388 Virtual Cores
815 GB of RAM
24.3 TB of storage

For those of you that know Amazon's EC2 instance types, that would be:

28 c1.xlarge
23 m1.large
29 m1.xlarge

We hope you enjoyed this installment of "Your gold dollars at work", and if things go well, there should be more in the future.

Let the insults begin! And if by chance we inspired you, you can purchase your subscription here! :)
discuss this post on reddit