Wednesday, May 26, 2010

It's a scavenger hunt!

Yesterday Wired magazine released their brand new iPad app. It's the same price per issue as the print edition ($4.99) and comes with a bunch of slick features. Now, normally we don't pimp out products from our sister site, but we made an exception for this one because we honestly think it's a breakthrough. Also they gave us brownies. But still, we really do think it represents the future of magazines. Read through the list of features or watch the video to see what we mean.

There is one other reason we like Wired on the iPad — there is a reddit alien in it! Yep, hidden in the very first edition of the Wired tablet edition is a reddit alien.

We thought it might be fun to thank Wired for doing this by having a little contest for people who buy the app. Here are the instructions:

- Buy an iPad (check first to see if you already have one).
- Install the Wired iPad app.
- Find the alien. It is hidden, but not that hard to find.
- Take a picture (be creative!), and upload it to imgur (but don't tell anyone!)
- Email us the link to the picture at

Your picture must have the following in it:

1. You
2. An iPad with the Wired tablet edition currently being displayed and on the page with the alien (in a way that is not clearly a lame Photoshop job)
3. Your username written on a piece of tape somewhere

The first ten valid entries will each receive either a pack of reddit buttons or an official sticker pack (winner's choice). These are a $9-$10 value, before even taking shipping into account.

Then we'll pick our favorite, most creative, picture out of all the valid entries, and that person will get a signed reddit Evolution Poster, and a special pre-release reddit T-shirt.

Lastly, all of the submissions will be submitted to a special reddit so that everyone can see the entries and vote on and discuss their favorites, and all of the successful entries will win a reddit award.

You can ask questions or talk about this contest on reddit.

Good luck!

Legally we're required to state that one way or another, the contest will be considered over by June 15, even if we haven't gotten as many submissions as we would have liked. Also, if you would like a means of entering without buying the app, please draw a picture of yourself holding an iPad with the reddit alien on it and mail it to reddit. Contact the admins for the mailing address and other details. Limit one prize per household.

Tuesday, May 25, 2010

Call for interns

How'd you like to work for reddit? For the first time ever, we're looking to bring on summer interns (or, for that matter, interns for any future quarter). One position is open in San Francisco and one in New York City. Responsibilities will include:
  • San Francisco: Manage relationships with merchandise providers, manage licensing agreements, help negotiate new licensing agreements, manage workflow of merchandise production from end to end, assist with site and advertiser feedback. Hang out with the reddit engineers in our terrarium conveniently located inside the Wired Magazine offices (overlooking South Park in SOMA, close to BART and CalTrain). Expected commitment is 20 hours a week.

  • New York: Business development and marketing duties would include finding and managing relationships with potential advertising sponsors and publishing partners. Expected to work on reddit projects for 20 hours a week. Can work from home or school, but weekly in-person meetings will be required at the famous Conde Nast headquarters in Times Square ... or some swanky coffee shop.

  • Requirements for both internships: smart, active reddit addict user, self-starter, good at pub trivia, willing to play or learn to play strategy board games (Carcassone, Settlers, etc.) in the off times (there aren't many). Extra points for willingness to try Starcraft 2 (if you can believe the release dates). Level 60+ WoW character a plus. Please include character race, class, and spec (girl characters are a-okay, but hunters need not apply).
Along with your resume please include the answers to the following questions:
  • What is your user name?
  • What is your karma?
  • What is your favorite color?
  • What quarter(s) are you available?
  • What city are you applying for?
  • Have you ever run a marketing campaign for anything (club, social event, etc.)?
  • Do you have any familiarity with the production of T-shirts or other such merchandise? (It's okay if you don't.)
Internships are unpaid (or at least not paid very well -- lunch plus commute time) and intended for course credit. However, you get a rare look at the inside of reddit (and Conde Nast / Wired), and you'll be doing actual real stuff, not sitting around filing uparrows or running out for coffee. If you're interested, please contact us at Please check with your college on eligibility requirements: we're legally required to ask you for a Letter of Credit.

Tuesday, May 18, 2010

The admins never do what you want? Now it is easier than ever for you to help!

EDIT (6 Apr. 2011): We've deprecated the pre-built VMs. The versions available on the site are now based on an out-of-date version of the code. There is now an install script that will turn a blank copy of Ubuntu 10.04 into a reddit install. See its reddit wiki page for more information.

We're proud to announce another major release of our code (only 6 months in the making...), now available from our code repository, and this time we took the extra step of building a VM with reddit already up and running on it. We broke this release into multiple chunks (each about a month's worth of development time) to keep it organized and, frankly, to make sure we didn't crash trac like last time.

We're hoping to have a bunch of discussions about new features on redditdev, so keep an eye on that reddit for up-to-date patches and bugfixes. For now, here are the highlights:
Cassandra Support
to replace memcachedb as a persistent caching layer for listings.

Self-service sponsored links
including our own python library for talking to to take care of credit card payments

Improved Messaging
including threading and moderator messaging.

replacing our python markdown interpreter

Included jobs/daemons
Part of properly setting up reddit is to get all of the services and cron jobs set up and configured properly to keep everything consistent. The code repos now include a "srv" and "scripts" directory for daemons and jobs, respectively. We also annotated the example.ini file.
Many of these additions require updates to the installed site-packages, so make sure to run "python develop" when you update your code repos.

We're also planning on a much more sane release schedule for future patches (much closer to "weekly" rather than "epoch modulo 10Ms").

Also, we know that reddit can be a bit of a pain to set up from scratch. A lot of it has grown organically without much thought to distribution, and keeping a handle on what services and scripts to run to keep it happy can be a bit hairy.

To put the focus on open source development rather than installation, we decided to build and distribute a VM along with this release. Here's a link to the torrent*. The VM is a VMWare image running Ubuntu 10.04 with the full reddit stack already built and configured (including memcached, postgres, cassandra, and haproxy). To try it out, load the app in VMWare, figure out the IP address it comes up with, and set your /etc/hosts file to resolve that IP as "reddit.local".

The admin account that ships with the VM is "reddit"/"password" so you probably are going to want to change that first. The admin account on the reddit install is the same, and all of the test users conform to the "my username is my password" standard.

TLDR: New code released to our open source project, and this time we're also providing a Virtual Machine that already has reddit built and ready to go on it. Enjoy.

*Yes, it is theoretically possible to intuit the direct download link from that URL. Please, we beg you, use the torrent! It's 832MB, and we pay for every byte.

Tuesday, May 11, 2010

reddit's May 2010 "State of the Servers" report

We promised we'd make a blog post about last week's issues, but every time we sat down to write it, a new issue popped up. It's been a crappy week. Anyway, although we're still knee-deep in looking for ways to make the site more stable, it's probably time to write up that, uh, writeup.

Warning: this post is dense with needlessly technical information. But there's a cartoon if you make it to the end.

she who entangled programmers

The primary cause of the initial slowness and later the all-out outage was the amount of time we spent blocking on Cassandra. Before you blame Cassandra outright, we've been running on Cassandra for our persistent cache for a while now, and when we first deployed it we deployed three machines to the cluster, which at the time was plenty. For performance we stuck a memcached cluster in front of it (because using it to replace memcacheDB was an emergency and we didn't want to wait for Cassandra's 0.6 release, which ships with in-core row-caching).

(Still with me?)

Since then the amount of data that we've been storing in it has significantly increased and so has the effective load on the memcached/Cassandra pair. memcached is blindingly fast, so we didn't notice that effective load was becoming way too high for just Cassandra to be able to keep up on only three nodes, as it was when we first deployed it. On Wednesday, we changed our memcached hashing implementation (from modulo to ketama, actually to avoid exactly this kind of problem in the future), effectively emptying the memcaches. We've done this before, and the resultant load is usually a bit high for an hour or so, but we've always carried through it just fine. The load on the Cassandra cluster rose instantly and dramatically (which is normal), and stayed high (which is not). Our Cassandra cluster was now so underprovisioned for the load that it wasn't able to catch up. Requests hitting Cassandra started failing with a TimeoutException, and didn't stop.

Normally the load would go down as more keys are requested and stored in memcached, so repeated requests for the same key would succeed as soon as the first one completes. Since our data in Cassandra has very high locality this would alleviate the load very quickly, but a pair of quirks kept this from happening. First, Cassandra has an internal queue of work to do. When it times out a client (10s by default), it still leaves the operation in the queue of work to complete (even though the person that asked for the read is no longer even holding the socket), which given a constant stream of requests makes the amount of pending work snowball effectively infinitely (specifically, ROW-READ-STAGE's PENDING operations grow unbounded). And second, since the request has timed out, reddit isn't able to take the answer and put it in memcached. Combined, these keep us from ever catching up once we fall behind.

The way you resolve this with Cassandra is by adding more nodes. But the node-bootstrapping process is particularly expensive (it incurs a process called anti-compaction, which duplicates most of the data on the node being split to make room for the new one) and when the node gets this far behind (that is, when you most need to be able to bootstrap a new node) it actually makes things worse. Adding new nodes too quickly can make things worse yet: the way a bootstrap works is by finding the node in the ring with the highest "load" (that is, the most disk-space used). The new node takes responsibility for half of his ring-space, has the data shipped to himself, and starts serving it up. But the old node doesn't delete the now obsolete data until it's told to execute a cleanup operation, and continues to count that obsolete, unreachable data towards his own load until that time, so new bootstrapping nodes overestimate his load. With a small enough ring this can cause the load of the nodes to become severely imbalanced, which when you're already imbalanced and loaded exacerbates the issue.

We eventually worked around this by adding a temporary hack to the code that if the data was not found in memcached, the app should assume that it doesn't exist instead of going on to check Cassandra. This removed the read-load from Cassandra but has the side-effect that if you vote or submit a link in that time, when we go to re-order the listings in which the voted item appears we end up fetching an empty listing, adding the voted item, and storing a new 1-item list, overwriting the non-empty list that used to be stored. This is what caused some listings to become empty or inconsistent. Because we receive so many votes this tends to keep the active listings (like /r/ very close to consistent but less active ones (like smaller reddits and user-pages) then have a huge gap of data. As we write this a mapreduce job is running to recalculate those listings from the canonical store at about 20/sec, and should be done by the end of the week. During this time no links, comments, or votes were lost, just the persisted cache of what goes into what listing.

With the read-load gone from Cassandra we were able to finally bring up some new nodes. This is already a long and expensive process but in our case because of some application quirks also required some hand-holding, which took all Wednesday night and into Thursday until around noon. Some particular members of the #cassandra room on Freenode IRC were there for a lot of time walking us through many of the required processes and Cassandra internals to help us keep the downtime as short and the data-loss as small as possible. In particular, the users driftx, jbellis, ericflo, and benblack were all especially helpful, and we owe you guys a case of beer some time.

Here are some of the lessons we have learnt about Cassandra so far:
  • It scales up nicely, but doesn't scale down nearly as well. At only three nodes we weren't able to take advantage of most of Cassandra's safeguards for ending up in this situation.
  • After bootstrapping a new node, the old ones don't delete old data until you run the cleanup, which can unbalance your cluster.
  • The pending queue is a good thing to monitor
  • It's significantly preferable to be over-provisioned than underprovisioned, and you need good statistics on your growth to project the number of nodes you'll need to handle your load as it increases so that you don't fall behind the curve. We've learnt that this is true of all things in our system, but our monitoring was skewed by memcached masking the load issue.

Cassandra is saved by the Amazonians

To make things worse, during the outage we had some initial problems getting the capacity from EC2 that we needed to quickly scale up Cassandra. Sometimes a hedge fund or biotech company will come through and fire up 1000 instances in a single EC2 zone to do some number crunching. A properly architected application will be able to work around this problem by simply coming up in another zone. We hope to one day have our application at that point, but right now we're sensitive to internal latency and so operating cross-zone is a serious performance hit. Since the bulk of our application is in a zone that had this capacity problem on Thursday, we had trouble getting the instance sizes we needed alongside them. To Amazon's credit, they did help us get what we needed quickly once we told them about the problem.

she must have been pregnant*

After finally recovering from the Cassandra failure and preparing to head off for some much-needed sleep, our internal message bus (rabbitmq) died, which added about an hour to the downtime. It dies like this pretty often at 2am or at other especially bad times. Usually it doesn't cause any data-loss, just sleep-loss (its queues are persisted and the apps just build up their own queues until it comes back up), but in this case it decided to crash in a way that corrupted its database of persisted queues beyond repair. rabbitmq accounts for the only unrecoverable data-loss incurred, which was about 400 votes. As far as we can tell, these were entirely unlinked events. Coincidentally, rabbitmq crashed twice more that day and a few more times into the weekend. For now we've upgraded to the latest version of Erlang (rabbitmq is written in Erlang) since R13B-4 is rumoured to have significantly better memory management which can act as a temporary stopgap for the apparent reasons for some of the crashes, but not all of them. Things have improved thus far, but replacing rabbitmq is at the top end of our extremely long list of things to do.

broken listings

Last week's woes only got in the way of fixing our aforementioned data integrity problems. Right before the proverbial shit / fan interaction last week, we had an unrelated caching bug appear where a cache node went into an indeterminate state where it wasn't down so much as returning empty results (in this case, empty precomputed listings). This normally isn't bad, except when those corrupted listings start having updates and inserts applied to them, and are subsequently cached mostly empty (the same symptom as during the Cassandra downtime but for unrelated reasons). Since all listings are stored in the same caches, this affected some random sampling of listings all over the site. The mapreduce job will fix instances of this inconsistency regardless of the cause.

email verification irony

We've also discovered that a lot of the verification emails we've been sending out haven't been going through. It seems that the mail server admins at some popular domains (e.g.,,,, and have their servers configured to consider all mail from reddit to be spam. This is because Trend Micro has marked Amazon's entire EC2 network as a "dial-up pool", and the aforementioned domains subscribe to Trend Micro's list and block all mail from anyone on said list. We've written to Trend Micro explaining that we're actually neither a spammer nor an individual end user, but rather an honest website that's kind of a big deal, and they sent us a form letter explaining how to configure Outlook Express and encouraging us to ask our ISP for further information. We'll try to figure something out as soon as time allows.

moving forward

We think the twine and gum holding our servers together should last for the time being, but just in case things fall apart again, we've gone ahead and updated our failure aliens (or, "failiens") to a new set provided by SubaruKayak.

We hope you don't get to see them for a long, long time, but here's a sneak peek at the new 404 page:

We invite all reddit artists to contribute your own images of the alien in strange, exotic locales. You can get the template here. (Kinda looks like Homsar, doesn't he?)

in conclusion

Despite these problems, our traffic keeps growing, so thank you again for your continued patience -- even if it means we'll have the usual problems of scaling up every few weeks.

We know that this site would be nothing without its community, and we thank you for your support.

Friday, May 07, 2010

From the reddit archives

Salutary acclamations to aviopticus for unearthing a scrivening of the reddit front page circa the month of November, Eighteen-Hundred and Sixty-Five: