Thursday, January 28, 2010

What a day.

It was a hell of a day for reddit's Engineering team.

25% of the department (i.e., Chris) was in New York on Conde Nast business -- I imagine him smoking cigars with bigwigs while everyone's riding polo horses. The rest of us (David, Jeremy, and me) were preparing for a road trip to the Googleplex to videotape the Peter Norvig AMA.

Right when our trio was getting ready to hit the road, news broke that some reddit users were seeing a pop-up ad right smack in the middle of our site. This was a surprise to us, and wasn't even supposed to be possible, but we had an appointment to get to, so we pulled Chris away from, I presume, the squash court to handle things. I'll pass the mic to him to explain how it went:

So, honestly, the first thing I thought when I saw that post on the front page was, "oh &#*%. I'm going to have to bludgeon myself to death with my own keyboard, because if I don't someone is going to remind me that I said I would" About half a second later I thought, "well, I'm doomed, but I can at least save the site from further abuse," so I shut off all site-wide ads post haste and hoped that this would give us a chance to figure out what had happened.

Soon after, we figured out that this was what we will refer to as a "whoopsie" on the part of our ad systems, and that we had accidentally started running ads from another of our parent company's sites (yes, that's what advertising looks like on other parts of the internet. It is considered "normal". Please buy a T-shirt...!) The issue was fixed before we had noticed it, but the damage was done. We've decided internally that the best possible fix for this will be to run the ad unit from an <iframe> on a separate domain which will prevent it from busting out by browser security rules (kind of like what we do currently on video embeds). There was nothing malicious involved. This was, apparently, a "feature" of that ad campaign, though it was not intended for us.

This was all done in the bit of time I had between meetings. [Oh, and before I forget, don't believe what your manservant tells you: polo is overrated.]

Mike here again. Meanwhile, back in Mountain View, the rest of the team had met up with Peter and, after a terrific lunch on Google's dime, grabbed a conference room and began the interview. As Murphy's Law would have it, the site starting acting funny right around the time Jeremy pressed Record.

We noticed that our silenced phones seemed to be vibrating a lot more than usual, and so David and I quietly checked things out as best we could while Jeremy ran the show. (When we post the video, you may notice the sound of some frantic off-camera typing.)

As soon as the taping was finished, we started discussing the anomaly. Within a minute or two of us flipping our ringers back on, all of our phones started simultaneously making the "reddit is in *really* big trouble" noise. The site was down; we didn't even have time to give our hosts a proper goodbye, but instead asked if we could sit in the parking lot and mooch some more WiFi.

Google's hospitality, however, was exemplary: they invited us to sit on couches in their lobby for as long as we wanted and help ourselves to all the WiFi we needed. We set up an emergency base of operations next to the reception desk and got Chris on the horn as well. David and Jeremy had brought their laptops along, but mine was 45 minutes away back at reddit HQ, so I had to commandeer one from a generous passerby. (You know who you are. Thanks!)

After an hour of rushed coast-to-coast debugging, we had patched things up well enough to survive the trip home. It had been an exciting day, and the Googleplex is every bit the nerd utopia it's reported to be. It's the sort of place where one can leave a laptop sitting on the floor unattended and know that not only is it perfectly safe, but that nobody's going to bat an eyelash over it.

For more information, see:

Tuesday, January 26, 2010

You asked, DirectRelief answered - learn about how your dollars are helping Haiti

If you've visited reddit at all since the earthquake that rocked Haiti two weeks ago, you've undoubtedly noticed the outpouring of support from the reddit community.

In fact, we've convinced digg to compete with reddit in fundraising for DirectRelief, which means great things for the people of Haiti; finally a chance for us to use this community rivalry for some good.

That said, they've got some catching up to do.

Our fundraiser for DirectRelief is nearing $180,000 -- a remarkable feat -- and our non-profit partners took some time to answer your top 10 questions from a recent AMA interview. Here are their answers.

Links mentioned in the Q&A:

UN Logistics Cluster for the Haiti response

Thursday, January 21, 2010

Messaging updates, round 2: conversations and an "unread" tab

When we added the ability to mark messages as unread last month, one of the maintenance problems that became apparent was a need to be able to find those unread messages once they fell off the first page, never to be seen again. So, we added an unread tab to the inbox page. Priming that listing required a little bit of db churn (enough to postpone this announcement for a couple of days actually), and we think we got everyone, but let us know if your unread tab looks sub-par.

The other half of this upgrade is to include something like threaded messages to keep track of your conversations. All of the new UI lives in the messages tab, which now looks like this:
In this way, we'll show both the messages you've received and sent in a given conversation, in order of their creation. You can reply in place and change their read/unread status just like anywhere else in the inbox. Also, in this view, messages are collapsable, and they'll retain that state from one load to the next. To conserve resources (screen, CPU, bandwidth, etc.), we've also allowed for collapsed conversations to only show the most recent response with a link to load the rest of the conversation (think gmail).

Other than this your inbox should look generally unchanged except:

  • We swapped the title and author line on comments. This makes it a little more consistent with the comment layouts we already use, and also helps with the threading layout.

  • Messages now have a "permalink" button which bring you to the messages page with their position in a conversation highlighted (just like comment permalinks).

We're still working on getting the UI right, and are open to suggestions (not like I don't think you'd be providing them anyway).

And, for those who don't like the change, no worry: new preferences:
The first will disable the new rendering style, unless you go to a message permalink. The second will auto-collapse read messages to keep your inbox nice and condensed.

And, last of all, with all of these changes to the inbox, we realized it was time to upgrade our API a little bit. There really hasn't been a great way via JSON to check if you have mail, and we broke some of the schemes/workarounds that have been developed when we last upgraded messaging. So, we added a "has_mail" attr to /user/username/about.json which will return true if you have mail and are logged in as that user.

Ask DirectRelief Anything

Submit your questions here! As you've probably heard, reddit is fundraising for DirectRelief in Haiti -- over $155,000 raised so far! And we couldn't be more impressed with the spirit of the reddit community, which has obliterated all of our goals and has now created its own, $185,229.69. Some are even calling for a jaw-dropping $314,159.26! No matter where this final number ends up, we've not only been humbled by the reddit response, but so has DirectRelief. I visited their office yesterday and photos/video from that journey will be posted to /r/haiti in the coming days. In the meantime, they've asked to be the subject of a reddit Ask Me Anything interview.

In line with their record of striking transparency and remarkable efficiency, ask them anything! On hand to respond to the top 10 voted questions as of 5pm Pacific on Friday will be a host of DirectRelief employees to make sure everything is answered by someone with the expertise in that particular area.

Remember: only questions in this comment thread will be considered (don't bother posting them in the comments of this blog entry).

About Jim & Andrew from Direct Relief:
Jim Prosser is Direct Relief's spokesperson, manager of media relations, and resident redditor (jimprosser). Jim has been with Direct Relief for nearly four years, helping build awareness and spread information about various ongoing medical aid programs in 70 countries as well as immediate response during emergencies. He graduated with a BA in Government from the University of Virginia in 2005.

Andrew Schroeder is Direct Relief's director of research. Andrew joined Direct Relief in 2008 from the University of Michigan's Ford School of Public Policy. He holds a Ph.D. in social and cultural analysis from NYU. Andrew's role in Direct Relief's response in Haiti involves the organization's GIS mapping and public health analysis.

Wednesday, January 20, 2010

Congratulations to reddit's Best of 2009!

The results are in -- congratulations to the winners of reddit's Best of 2009 awards:Honorable mentions go to all our finalists -- and in fact, even the unabridged list of all nominees is great reading material, whether you missed it the first time or not.

Tuesday, January 19, 2010

Haiti: One week later

It has been a week since a 7.0 earthquake hit Haiti. Since then, the world community has gathered to offer their support (well, mostly), and the reddit community was no exception.

When we asked you to help out our friends at DirectRelief, we were overwhelmed by the response. In just five hours, you managed to reach our first goal of $31,415, and in 12 hours you reached our second goal of $62,830. Even more amazing is the fact that over 2900 of you donated. Through your generosity, reddit became the single largest donor to DirectRelief International.

Saydrah has now challenged the community to raise $10pi and many of you responded with a parade of upvotes and donations. A team of volunteers is even being organized to spread the word - so even if you can't give, send Saydrah a message letting her know you'd like to help.

Since we asked for your donations, others in the Conde Nast family have followed suit.

When the first shipment departed for Haiti, DirectRelief sent us photos, and will continue to keep us updated through the /r/Haiti community, so be sure to subscribe to see where every dollar is going.

In that vein, members of the DirectRelief team have agreed to do an Ask Me Anything interview. Alexis is at their offices today and thanking all of these sleep-deprived but ever-enthusiastic employees and workers -- all of whom have had nothing but thankful about what reddit has done for them, despite being very confused about Alexis' Coat of Arms t-shirt.

To help out our cause, Zach Weiner, creator of SMBC Comics, challened his community to get us to break 100K in donations. He promised an extra comic if they did so. Well, we broke 100K, and keeping true to his word, Zach provided us with this artwork to commemorate that achievement:

On a personal note, we here at reddit HQ are humbled by your amazing response, and by what this community can do. It is this type of generosity and community effort that keeps us coming to work each day, and we thank you for reaffirming our belief that the community can do great work if they want to.

If you haven't already, please pass around the donation link to any friends and family who may be interested but don't know how to help.

Monday, January 18, 2010

Ask Chris Anderson (TED Curator) Anything!

Chris Anderson became the curator of TED Conferences in 2002 and hasn't looked back since. Now's your chance to Ask him anything! (comments on this blog will not be considered)

Few people (myself included) even knew about TED back then -- few did -- but Chris has aimed to change that in the last eight years. Now hundreds of thousands of TED Talks are viewed every day and TEDx events spring up in cities around the world, hosted by TEDsters who've taken the model for sharing ideas back to their hometowns.

And Chris has actively encouraged this growth every step of the way, even reaching out to the reddit community to form partnerships like this one to ask anything of TED speakers (Sir Ken Robinson, Hans Rosling, and Karen Armstrong to name a few).

Now it's your turn to ask Chris Anderson anything. You'll be able to find him in around the /r/TED reddit and elsewhere around reddit under the name TEDChris. In fact, he agreed to this interview in a reddit thread this past December.

The top 10 questions on this thread as of January 22nd at noon Eastern will be asked.

Thursday, January 14, 2010

Ask Anil Dash Anything

Anil Dash is a pioneering blogger, technologist, and entrepreneur, and the first employee of Six Apart. He recently became the director of Expert Labs, a non-profit, independent group (largely run/funded by AAAS - American Association for the Advancement of Science) to facilitate feedback by the experts to the Obama Administration and other government officials. Read the press release, a recent interview with the New York Times, and a video.

ASK HIM ANYTHING HERE (questions int he blogger comments will not be considered)

We will ask Anil the top ten questions as of 12pm ET this Saturday, January 16th, while at ScienceOnline2010, an annual conference on science and the Web.

Here are some more links to inspire questions:

Top scoring posts on reddit from Anil Dash (
Best of
Follow Anil on Twitter @anildash

Helping Haiti (because we ought to do more than just change our logo)

Zoinks! We hit our original, delicious goal of $31,415 in just five hours (reddit, you're awesome, that is, truly awe-inspiring).

But then we realized e^(pi * i) is -1 and we don't want to be negative; alternatively, we'd rather say that we "e^(2pi * i)." Everyone likes a winner* so let's raise $62,830 for Haiti!

UPDATE: Hot oatmeal! You did that in 12 hours!

Absolutely remarkable, reddit, you're nearing you've broken $100,000 in donations for Haiti! Keep piling it on and subscribe to /r/Haiti to watch the results of your awesomeness.

We've all been shocked and saddened by the earthquake that's hit Haiti and based on the trending reddit links since the quake hit, it's clear many of you are anxious to help. I got to talking with a redditor at DirectRelief, a non-profit specializing in just this kind of disaster recovery and sending 100% of donations to help Haiti - redditors needn't worry about any money siphoned off for administrative or fundraising costs.

And as we saw on the reddit frontpage today, sending anything but money is detrimental to the relief effort right now.

Here's the reddit DirectRelief fundraising page. We've set an ambitious goal of US$31,415 because we know you're up to the challenge, reddit. There's $1,000 in the pot to get things started, but team reddit will personally match redditors for another $1,000.

Update! We hit the redditor-donated $1,000 mark in a few minutes! You'll now find the 'team reddit' matching donation of $1,000 on the donation page. Thank you, everyone! Now we're going to test the limits of the DirectRelief fundraising counter!

Even if you can't donate right now, you're helping with an upvote.

And here's the best part, we'll be getting photo/video/text updates from the field on the /r/Haiti reddit so you can see just how your money is being used as well as keep up with their work in the days, weeks, and months to come.

*e^(2pi * i) = 1. Don't you wish you'd studied harder in Pre Calc?

Tuesday, January 12, 2010

Best of 2009: Final round (go vote!)

Last Monday, we asked you to submit nominees for reddit's Best of 2009 awards. The response has been terrific, distilling the past year's cream of the crop into, like, a thickshake or something.

The whole list of nominees is worth reading through (you can do so via the bulleted list in last week's post), but if you're short on time, here are the finalists:Please vote!

Important note:Other, less important notes:
  • Voting will end at 5pm California time on the 19th.
  • You can vote for more than one finalist in a particular category -- in fact, you're encouraged to. Please don't downvote, though.
  • Finalists are presented in random order, so don't consider their relative positions to be at all meaningful.

Thursday, January 07, 2010

Why did we take reddit down for 71 minutes?

As most of you know, we moved reddit to EC2 back in May of 2009. Our experience there has been excellent so far. Since we moved to EC2, the number of unique users has gone up 50%, and pageviews are up more than 100%. To support this growth, we have added 30% more ram and 50% more CPU, yet because of Amazon's constant price reductions, we are actually paying less per month now than when we started.

So why am I singing the praises of Amazon and EC2? Mainly to dispel the opinion that the site getting slower since the move is in any way related to Amazon. Our experience with EC2 so far has been excellent, and when we do hit a bump in the road, their support staff is extremely helpful, competent and technically knowledgeable. Any slowness that reddit has been experiencing is our fault, not theirs.

So, why did we take the site down for 71 minutes yesterday? As you probably noticed, our site has been unacceptably slow during peak times since before the holidays, and unusably slow for the last week. To fix this we upgraded the disks for one piece of our system.

Nerd alert -- this section gets technical
Let me start from the beginning… Amazon offers a service they call an Elastic Block Store (EBS), which is a device that you can attach to your running instance that looks like a SCSI disk. Under the hood these disks are accessed over the network and are built from RAID systems, but from our point of view, they are roughly as fast as a single disk fast SCSI disk.

We use a lot of the EBS disks. All of our databases were each using one EBS. This worked really well for us up until a week or so ago. Then all of you came back from holiday and decided that work was just too boring or something, and our traffic spiked, essentially breaking the camel's back, if you will.

In response, we started upgrading some of our databases to use a software RAID of EBS disks, which gives drastically increased performance (at a higher cost of course). This worked really well, but there was still one missing piece of the puzzle.

Part of our setup uses what we call a "permacache", which uses Memcachedb. Memcachedb is Memcached with a built-in permanent storage system using BDB. One of the "features" of this system is that it saves up its disk writes and then bursts them to the disk. Unfortunately, the single EBS volumes they were on could not handle these bursting writes. Memcachedb also has another feature that blocks all reads while it writes to the disk. These two things together would cause the site to go down for about 30 seconds every hour or so lately.

Last night, we upgraded the disks that these caches use to be the same RAID setup that we are moving the databases to. We had to take the site down because while we call them "caches", they really are just another database. We call them a cache because all the info they hold can be recreated from the main database, but not quickly or easily.
End Nerd Alert

The maintenance itself went smoothly, and we only had one small issue where one of the five machines had a slight performance problem building the RAID.

As always, if you have any questions, you can ask them on the comments to this blog post.

Tuesday, January 05, 2010

Christopher Hitchens Answers Your Questions

Author and well-earned contrarian Christopher Hitchens answers your top questions in a 30 min interview. Watch on youtube, or embedded below. After the embedded video are all the questions, where you can also go directly to the response for each question.

10. PSteak
Dear Mr. Hitchens,
what historical figures, events, movements, or books do you feel have been ignored, or under emphasized, in the public education of young people?
Watch Response
A Struggle For Power by Theodore Draper (The book he recommends)

9. Scariot
From what I've read it seems you initially supported US led military action in Iraq and Afghanistan; do you believe that US foreign policy in Iraq and Afghanistan has had a positive or negative impact on the growth and exposure of Islamic extremism? Also, given that the countries are still plagued with problems many years after the initial invasions what direction d o you think US foreign policy should take now?
Watch Response

8. BoredGreg
Where do you get your news?
Watch Response

7. OmegaMoose
Do you believe in some kind of free will or do you subscribe to determinism/ incompatibilism?
Sorry. Question was accidentally skipped. My fault - not Christopher's.

6. droberts1982
You've stated that the litmus test for the Obama administration is Iran. How is the president doing in this area?
Watch Response

5. Callidor
You and your fellow horsemen (Dawkins Dennett and Harris) are sometimes referred to collectively as "New Atheists." What does this term mean to you? Do you embrace it, or do you hold that there is nothing particularly "new" about your breed of atheism? Also, in god is not Great you briefly mention your disapproval of Dawkins and Dennett's "Brights" movement. Are there other significant points on which you disagree with the rest of the "New Atheists?"
Watch Response

4. 1984WasNotAManual
If you were the Prime Minister of the UK, what would you do to combat religious extremism? Also, can and should the UK government try to encourage atheism, and if so, how?
Watch Response

3. dingledog
I'm a nationally-ranked policy debater in college, and despite years of debating, practice, and research, I am occasionally stumped by a question asked by my opponent. Has there ever been a question asked for which you had no good answer? And if so, what is your typical strategy in dealing with these situations?
Watch Response

2. adlayormoffer
You've called yourself a Marxist, but say you no longer consider yourself a socialist. This issue was addressed in a reason article a while ago, but could you elaborate more? For instance, is the power of the unaccountable corporation no longer a major concern for you? You've also been eerily silent on the health care debate (as far as I know), why?
*palsh7 has identified the essence of the question: "what consensus exist[s] between Socialism and Libertarianism?"
Watch Response

1. neilk
Your speaking style is very unlike the norm today -- elevated yet accessible, aggressive but still entertaining. What goes into achieving this effect? Are there any other speakers or schools of rhetoric you draw from especially?
What do you think of the state of rhetoric and public debate in America?
Watch Response

Huge thanks to Christopher Hitchens for sharing so much of his time with reddit! If you haven't checked it out already, make sure you read his recent Slate article on the TSA and airline travel.

Monday, January 04, 2010

Who would you nominate for the "best of reddit 2009"?

Welcome back to work, everyone. Now that last year is officially behind us, it's time to honor the best of 2009. That's right, we're kicking off the second annual "best of reddit" awards.

To nominate someone or something, click the name of the category and post a top-level comment:Got it? Nominations go on the official threads linked above. Don't accidentally submit them to this blog post's comments page, but feel free to use that space to come up with a better name than "best of reddit awards", to make suggestions for categories we didn't think of, and to complain that awards and trophies are stupid and dumb.

We'll be accepting nominees until 11:59pm RST* on the 11th, and will announce the finalists sometime on the 12th -- probably in the late afternoon, since we expect to spend the first half of the day filtering out all the cheating. Then there'll be a runoff... but we're getting ahead of ourselves. Go nominate! Be sure to check your saved page, /r/bestof, tldr, and this list of noteworthy reddit events for ideas.

*reddit standard time, which is like PDT except we never turned our clocks back.