Monday, September 28, 2009

We had some bugs, and it hurt us.

As many of you noticed last night, or heard this morning, we had a bug in reddit that allowed someone to start a comment bomb. Specifically, we had two bugs.

The bugs have been squashed, and it is perfectly safe to open your inboxes again.

It is important to point out here that as a site that gets all of its content from users, we take sanitization very seriously. We sanitize both input and output. In this particular case, our output sanitizer was broken in a non-obvious way. As a matter of fact, these bugs were only exploitable because we are open source. The worm author had to scour the source of our output filter to find these holes. We cannot hide behind security though obscurity, and we like it that way. We also rely on our users reporting security bugs in a responsible manner.

We have spoken to the worm author, and he has apologized for his actions and admitted that what he did was irresponsible. He has promised that he will follow the path of responsible disclosure in the future.

We would also like to take this opportunity to thank the mods who spent time banning all of the malicious comments in their reddits.

And lastly, a special thanks to everyone in the IRC channel who helped us track down and fix this problem, with an extra special thanks to chromakode for his help last night.

Technical Details

The first bug wasn't really a bug, but a feature of markdown that we hadn't removed. This feature allowed one to specify a variable for replacement later on. This feature alone however was not enough to carry out the exploit.

The second bug was also in the markdown library. To prevent double escaping of certain characters, they are run through MD5 after being escaped once, and then the MD5 is undone at the end. Since the MD5 is the same every time, someone figured out that if you just put the MD5 into your comment, it would be unescaped at the end.

Putting the two holes together allowed a user to create a comment that had javascript in it that would run via onmouseover and automatically post new comments on the user's behalf.

To fix the first hole, we disabled that feature, which probably should have been disabled before. To fix the second, we added a salt for the MD5, so that it would not be predictable. You can see the patch here:

http://code.reddit.com/changeset/1f1f0606f5b6bf14a0db55a28cfd03e1e42e3550
discuss this post on reddit