Thursday, March 31, 2011

reddit mold is now live!

Depending on your luck, you may have noticed that reddit looks a little different today. If you go to the site and the logo looks like this...

...then congratulations, somebody signed you up for reddit mold. Here are the features you'll be enjoying today:

New comment limits

You know how you can load up to 500 comments at once? Say goodbye to that limit! Every time somebody gives you reddit mold, it goes down by 10. So if you get mold fifty times, you won't see any comments at all. Think of the time you'll save.

Streamlined posting

Got a keyboard with a broken letter E? No problem! With reddit mold, you're no longer allowed to use that letter anyway — just like a modern-day Ernest Vincent Wright. In fact, every time someone gives you reddit mold, you lose another letter. (Note: the letters M, P, and H are excluded, so that you can always say, "mmph! MMMPH!")

Mark of Honor

All reddit mold subscribers will be distinguished with a number:

Each time you get another helping of mold, the number goes up. This way, everyone will know how awesome you are. And if they go to your userpage, they'll see exactly which letters you're still allowed to use. There's also a beautiful trophy courtesy, as always, of licenseplate.

Exclusive community

When you sign up for reddit gold, you get the option of subscribing to a secret members-only lounge. Oops, so much for the "secret" part. Anyway, we wanted to do something similar for reddit mold, so we created /r/basement, a special place for mold subscribers. Or anyone else, in fact; it's a public reddit that anyone can post to. However, there's a special bonus if you have mold: for your convenience, you get to see /r/basement links on your front page even if you accidentally try to unsubscribe from it. Stay informed!

Gift notes

Every reddit mold subscription comes with a gift note. You'll be reminded of your benefactor's words every time you visit a page on reddit. Don't forget to thank them!

And, if you see any good April Fools pranks on reddit or elsewhere, submit them to:

Don't forget that if you buy a reddit gold creddit, you will get a mold spore so you can retaliate return the favor. It seems to take about 20 minutes for them to tell us about the purchase, so be patient. :)

* If you bought reddit gold (and therefore mold spores) any time after our last blog post, but haven't gotten it yet, please mark your payment as disputed so we can address the problem. It appears that a few of the purchases didn't go through. Sorry, and thanks!

Better Have a Backup Plan. World Backup Day!

Recently, the fine reddit users had the outstanding idea to declare March 31st World BackUp Day. The idea grew like a W32/Yayin-A Worm with redditors creating, and an official twitter account @worldbackupday and #worldbackupday hashtag. They have even partnered with some backup providers and services to provide some discounts and related contests to redditors, so you have no excuse not to backup your data anymore. GigaOM and ReadWriteWeb have even helped spread the news about World Backup Day, so this looks to be well on it's way to becoming an annual event.

Share your backup questions, horror stories, and advice to celebrate this brand new important holiday. If any redditors work for a backup provider or service, reply in the comments with details about any deals you have for redditors or message hueypriest and we'll add a link in this blog post.

"Remember to check your data backups. Backup your memories and financial information and check your old backup restores." - WorldBackupDay.Net

After all, reddit mold is coming and there's no telling what it might do ;)

Wednesday, March 30, 2011

Rep. Anthony Weiner Answers Your Top Questions on Video, and A Stats Breakdown for a Top IAMA

Last week New York Congressman Anthony Weiner posted "IAmA Democrat Who Fights, Rep. Anthony Weiner (D-NY). AMA." He responded in the IAMA thread to many of the questions, but also wanted to respond on video to the top 5 voted questions. His response can be seen HERE or on the embedded video below.

Why do you unconditionally support the state of Israel? Why have you insisted in the past that the Palestinian Liberation Organization is a terrorist organization, but you let Israel slide when it kicks Palestinians out of their homes to build new settlements?
I thoroughly appreciate your hard-hitting style on the issues I agree with you on, but to me this is one of the primary unrecognized (by the U.S.) human rights crises of our day.
Thank you for your time, as I know you are a busy man. Take care.

About voting "No" to extend the Patriot Act

Do you think its time to legalize marijuana for recreational purposes and what do you feel about the War on Drugs in general?

What's going on with Campaign Finance Reform? Is there any hope we'll see an attempt at it any time soon?

First off, thank you for fighting so hard for all of us. If there were more Democrats like you, we wouldn't be in the shit we are.
As a congressional Democrat who is putting up a solid fight, you have a better understanding than any of us how this process works. A lot of us on the outside are getting pretty disappointed by President Obama's actions, but we have no understanding of the process or the give and take that accompanies those actions.
Given that you have a much better vantage point than us, how do you feel about Obama's performance so far?
Have you ever voted against your conscience because of the politics involved? What votes were they?

Since this IAMA made it all the way into our Top 10 IAMAs of All Time, we thought we'd break down what that means in terms of traffic and activity.

The Congressman's IAMA received the following:
  • 2,554 total points (upvotes - downvotes)
  • 4,564 total comments
  • 62,887 total votes on comments
  • 159,062 Pageviews
  • 133,178 Unique Views
  • 00:03:34 Time on Page
That's a lot of votes, comments, and visitors for one day. If you'd like to do an IAMA on reddit or would like to request a famous IAMA on reddit, contact hueypriest.

Tuesday, March 29, 2011

give the gift of reddit mold

Here at reddit, we like to make a big deal about April Fools Day. In years past, we've ██████ed forbidden words, given the site a horrendous rebranding, and even made everybody an admin. Of course, those all turned out to be hoaxes.

This year, there won't be any deception. In fact, we're going to announce our prank three days in advance. It's a new service we'd like to call "reddit mold", and it's going to be a real thing. It's like reddit gold in that you can give it to other people. It's unlike reddit gold in that they probably aren't going to thank you.

So what (mis-)features will they get? We're keeping that under wraps for now, but we will say that if you give someone reddit mold, you get to leave a gift note, which will appear in the right sidebar just like an ad. So one of the features will be that they get to look at whatever you wrote — all day long, on every page. (We should mention that everything goes back to normal on April 2, so you don't have to worry about long-term implications.)

So how do you give reddit mold? You'll need a spore, of course. Each spore lets you sign one unlucky redditor up for mold. Throughout the day, we're going to be randomly distributing spores to lucky users. Also, the admins are going to be handing them out like candy in this comment thread. Tell us why you should get one. Beware, though: if your request gets downvoted, we might give you mold instead. Finally, if you still don't have a spore by the end of the day, you can also get one for free when you buy a reddit gold creddit (limited time offer!)

Once you get your spore, just head over to, name your victim, and write a taunt.

Thursday, March 24, 2011

Pimping and Other Ways to Find New reddits

One of the secrets of reddit's success is that anyone is allowed to create any new reddit they can imagine. This allows for redditors to dream up ways to use the reddit platform that we at reddit could have never imagined. When the first (sub)reddit(/r/programming) was created, no one would have predicted that a few years later there would be reddits for IAMAs, Weird Bug Identification, First World Problems, Imaginary Monsters, Lady Boners (Mostly SFW), Brazilian Jiu-Jitsu, Space Porn (SFW), Forever Alone Dating, and yes even a reddit for Vogon Poetry - Ohfreddled gruntbuggly!

So, with so many amazing reddits out there, how does one find new reddits to subscribe to, and if you've created a brilliant new reddit how do you let other redditors know about it? You can always subscribe to /r/newreddits & /r/shamelessplugs and you can always try your luck with /r/random (it's fun, roll the dice). Eventually some of the brilliant new programmers we're hiring will come up with an elegant new reddit finding system, but in the meantime, we're going to start featuring new and upcoming reddits in the sidebar ads and in sponsored headlines.

If you are the mod or just a fan of a reddit you think is worthy of some free promotion, then submit it to /r/PimpMyReddit, which is being run by Cupcake1713. Each week we'll select some new or unknown reddits to get some free ads. To increase your favorite reddit's chances of being pimped*, please include as much of the following as you can:

- reddit name
- number of subcribers (if any),
- sponsored headlines examples
- links to potential 300 x 250 ads, or ideas for a 300x250 ad (see examples below)
- similar reddits that exist,
- why we should promote this

If you are a graphic designer, keep an eye out in /r/forhire where we'll be hiring designers to create some of the ads for new reddits. Everyone else, keep an eye on the sidebar for some new reddits to +frontpage.

*All reddit pimping decisions will be the sole discretion of the Pimp Counsel.

Thursday, March 17, 2011

Why reddit was down for 6 of the last 24 hours.

As most of you are probably aware, we had some serious downtime with the site today. Now that the dust is beginning to settle and we have finally gotten some sleep, we will attempt to explain what happened.

As you will see, the blame was partly ours and partly Amazon's (our hosting provider). But you probably don't care who is to blame, and we aren't here to assign blame. We just want to tell you what happened.

Begin nerd talk

At approximately 1AM PDT today, we noticed that load was simultaneously shooting up on a large number of our Postgres and Cassandra servers. Within the next 10 minutes, we determined that I/O had ground to a complete halt on nearly every server which was using Amazon's Elastic Block Storage(EBS) service in one particular Availability Zone (their version of a data center). When we say "complete halt", we really do mean it. It was taking minutes to read or write a single 512-byte sector. Since replication everywhere was severely degraded due to this issue, we decided to take the site down to prevent further issues.

We immediately got in contact with Amazon and supplied them with a multitude of data. Jedberg even resorted to pasting data to AWS every 30 seconds as it came in. After a couple hours of digging, they put the following status message up:

2:45 AM PDT We are currently investigating increased latencies for a subset of EBS volumes in a single Availability Zone.

Note that when you are the size of Amazon, a "small subset" can mean a lot of things. In our particular case, it meant the majority of EBS disks on most of our Postgres and Cassandra servers.

EBS outages affect us particularly badly, because we use a whole lot of them, so we have a large exposure to their problems.

Nearly an hour later, Amazon acknowledged that the issue started right around the same time we started seeing issues:

3:36 AM PDT A small subset of EBS volumes began experiencing increased latencies in the US-EAST-1 region beginning at 01:00 AM PDT. We are working to restore normal operation to the effected EBS volumes.

At this point, the AWS engineers informed us that they were manually repairing the issue on the EBS disks. They asked us "which disks are most critical" to repair first, to which we replied "all of them".

Nearly an hour after that, Amazon was still working to resolve the issue:

4:33 AM PDT We are continuing to work to restore normal operation to the small number of EBS volumes still experiencing elevated latencies.

At around 5am, most of our disks were repaired, and we brought the site completely up.

We monitored the site for the next hour or so to ensure things were stable, and even had to wake Spladug at 5am for some quick help with some corrupted listings. Once we had cleaned up those issues ,we decided to get some much needed sleep.

What Happened... again

In a extremely cruel twist of fate, we started to notice issues with our EBS volumes again at around 10am this morning. We immediately contacted AWS about the issue, and they began to work on repairing the bad volumes. The issue was not to as large of an extent as it was previously, so no AWS status post was made at that time, as it was only effecting a few volumes -- ours.

Then, something really bad happened. Something which made the earlier outage a comparative walk in the park.

Part of reddit's database backend is a handful of Postgres replication clusters. We are using master-slave replication across multiple different masters using a program called londiste.

Shortly after noticing the EBS issues this morning, our database replication took a severe turn for the worse. Data which had been committed to the slaves was not committed to the masters. In a normal replication scenario, this should never, ever happen. The master commits the data, then tells the slave it is safe to commit the same data.

We are still investigating as to why replication failed. All we know is that it definitely broke when the EBS disks on the masters started having issues. We could make some speculation about the disks possibly losing writes when Postgres flushed commits to disk, but we have no proof to determine what happened.

The replication issue resulted in key conflicts on some of our slave databases. If you work in RDBMS at all, you know this is an extremely bad thing. Since there was inconstant data in the cluster, we were forced to bring the site down to prevent further inconsistencies. You can see a graphical representation of this here.

There was no easy way to untangle the mess that the broken replication had left behind, and our only option was to partially rebuild our slaves by dropping and then re-replicating the affected tables. We opted to do this rebuild while we waited for Amazon to migrate the data on the master to better hardware, since we already had the site down. This data moving process took several grueling hours, during which the site was completely down. At approximately 1:30PM PDT, the data migration and slave rebuild both completed (coincidentally just 10 minutes apart) and we were able to bring the site back online.

What we are doing about it and what we could have done better.

Nothing wears on our fragile sanity more than when reddit goes down. Like you, we never, ever want to have this happen again.

Amazon's Elastic Block Service is an extremely handy technology. It allows us to spin up volumes and attach them to any of our systems very quickly. It allows us to migrate data from one cluster to another very quickly. It is also considerably cheaper than getting a similar level of technology out of a SAN.

Unfortunately, EBS also has reliability issues. Even before the serious outage last night, we suffered random disks degrading multiple times a week. While we do have protections in place to mitigate latency on a small set of disks by using raid-0 stripes, the frequency of degradation has become highly unpalatable. To Amazon's credit, they are working very closely with us to try and determine the root cause of the problem and implement a fix.

Over the course of the past few weeks, we have been working to completely move Cassandra off of EBS and onto the local storage which is directly attached to the EC2 instances. This move will be executed within the month. While the local storage has much less functionality than EBS, the reliability of local storage outweighs the benefits of EBS. After the outage today, we are going to be investigating doing the same for our Postgres clusters.

One last change that we will make is fixing a mistake we made a long time ago. When we first started using Amazon's EC2, there were no "best practices" yet. One mistake we made was using a single EBS disk to back some of our older master databases (the ones that hold links, accounts and comments). Fixing this issue has been on our todo list for quite a while, but will take time and require a scheduled site outage. This task has just moved to the top of our list.

Some answers to common questions

Q: No other AWS site appeared to be having issues. Why was reddit affected so severely, while other sites stayed up?

A: This outage affected a specific product(EBS), on a specific AWS availbility zone. Unfortunately, a disproportionate amount of our services rely on this exact product, in this exact availability zone. We also use EBS more heavily than sites similar to us.

Q: Why is reddit tied so tightly to the affected availability zone?

A: When we started with Amazon, our code was written with the assumption that there would be one data center. We have been working towards fixing this since we moved two years ago. Unfortunately, progress has been slow in this area. Luckily, we are currently in a hiring round which will increase the technical staff by 200% :) These new programmers will help us address this issue.

Q: Why was reddit taken down by something as simple as a disk issue? Don't you RAID?!?!

A: Disks always fail eventually. We have standard protections in place to prevent problems from disk failures. However, there is very little that can be done to prevent issues when such a large amount of disks fail, like what happened today. And in this particular case, it hit one of the few servers where we don't use a RAID.

Q: Why are you using Amazon as a scapegoat?

A: We'll certainly admit that more could be done from our side to prevent hosting issues from affecting us so gravely. However this was a very serious outage which affected a large proportion of our disks. We would be lying if we said Amazon didn't have some fault here.


Yes, the site was down for two separate, long periods of time today. The downtime was catalyzed by an outage with Amazon's EBS service. Jedberg, spladug, and myself are working hard to prevent future issues. We take great pride in the site, and whenever it goes down it seriously hurts for us, as much as it hurts the community.

Wednesday, March 16, 2011

So long, and thanks for all the postcards

Today was my last day at reddit.

Well, I should clarify that. I mean, I'm still going to spend half my life on this site and maybe I'll even try building up my karma again. But from now on, I'll be doing it as an amateur.

My time here has been wild. I've made friends with startup founders and web cartoonists and have been lucky enough to shake hands with redditors from Adam Savage to Marc Andreessen to the thousands of creative and wonderful people who joined us in DC last October. I've watched the site grow from a mere link-aggregation portal into a true community — the best forum on the Internet, if you ask me.

When I started in November 2008, revenue was a trickle. The site survived on t-shirt sales and Conde Nast's love. Today, reddit is signing big advertising deals, does brisk business every day via the sponsored links system, and (thanks to all you reddit gold subscribers out there) has proven that endless advertising isn't the only way for a website to pay its bills. Traffic is skyrocketing, articles are written about reddit nearly every day, and the four new programmers joining the team this spring should have a huge impact on speed, stability, and features.

I'm so proud of what our scrappy, brilliant skeleton crew has been able to accomplish, and I know that the best is yet to come for reddit, but I nonetheless feel that it's time for me to move on. At this point in my career, I want to work at a place where the programmers go all the way to the top, where the policies coming from HR and IT and everyone else are designed with nerds in mind, not magazine publishers.

And so when Google approached me recently with a dream offer, I realized that it was time for me to make my pilgrimage. (I think that every programmer should work there at least once in his or her career.) If you'd like to hear more about this adventure as it unfolds and keep in touch with me as I exit the reddit spotlight, you can follow me on Twitter. (This is my first time tweeting, so let me know if I do it wrong.)

Me with some of the toys, art, and postcards you sent to reddit from 81 countries on all seven continents

Friday, March 11, 2011

Are you going to be at SXSW?

Our Community Manager Erik Martin (hueypriest) will be doing a workshop at SXSW on Saturday March 12th at 11am. Come heckle him and fellow panelists Drew Curtis from Fark, Kelly McBride from The Poynter Institute, and Tucker Max as they discuss why comments on most sites suck so much.

Community Engagement Strategies: Rational Debate or Herding Cats?
Saturday March 12
11:00AM @ Sheraton - Capitol E-H
701 East 11th St

Also at SXSW will be ketralnis, who will be representing his new company Hipmunk. Yes, it turns out ketralnis is leaving us to rejoin reddit's founders on their latest adventure, and we wish him all the best. He will certainly be missed. He has this to say for himself:

It's been my time to go for a while but now that hiring is so far along and there's so much great news about reddit's future coming in, I feel like I can leave the site in good shape and in good hands. I know that you guys will do great things without me, and I'm genuinely excited to see them. I'll still be hanging around the site of course, but I'll be going back to just posting pictures of cats with captions on them.

I'm looking forward to rejoining my friends at Hipmunk and helping them build a company to make travel search suck a little less. I'll be tagging along with them at sxsw so if you see us there make sure to come say hi!

On a more positive note, our massive engineering expansion is going really well. There are so many terrific submissions that we're just now wrapping up the initial review process and will be requesting resumes soon. And yes, the departure of ketralnis does mean that we'll be hiring even more people than previously anticipated.

Saturday, March 05, 2011

Millions Ask Anything

It's a scary world out there. It takes courage to share anything intimate with the online world these days and even more to offer yourself up to a huge audience saying "ask me anything." This past week three IAMA posts received so many points & comments that they are in our top ten of all time: Lucidending, Ken Jennings, Congressman Garamendi, and one from a 4-year old, which also received thousands of comments. The IAMA reddit, an organic creation of the reddit community, has seen a huge growth over the past few months (see graphs), outpacing even reddit's incredible growth. What makes this format so compelling?

My theory at least is that by opening themselves up to questions about anything, from anyone, anywhere, IAMA subjects take a big step toward their audience. They show that they are willing to jump into the fast and fluid conversation, and that neither interviewer nor interviewee really knows where the discussion is eventually going to go. Everyone is along for the ride. The format works not only because the community (and voting) creates a welcoming forum and keeps things civil, but also because it's big enough and unpredictable enough to keep things interesting for both subject and audience. What do you think makes IAMAs so fascinating?

Check out the IAMAs going on today, or even share your story. See the how to or contact hueypriest if you have questions. Or create your own community. Who knows, it could be the next IAMA.