Usually I illustrate blog posts with little drawings. Unfortunately, this one didn't lend itself to illustration, so I'm sorry if the drawings don't really relate to the material.
reddit has a new comment sorting algorithm, the "Best" ranking (joining "Top", "Hot", "New", "Controversial", and "Old"). It affects only comments, not stories. davean (the xkcd sysadmin) wrote the patch, the system has been in trial for a week or so, and as of this morning has become the default way comments are sorted.This is a very, very good thing. Most of the time, you won't notice that there's anything obviously different (it doesn't affect threading or anything -- don't worry!), but it should improve the quality of the top comments immensely.
The algorithm is detailed here, in an article by Evan Miller (which I found last year on reddit). The article explains why each of the common systems for ranking things based on votes is flawed in a different way, and if you go there you'll see an explanation of the algorithm plus some examples of flawed voting systems. If you want to jump to the example of the new system working, scroll to the screenshots at the bottom. But if you're interested in the nerdy details, here's an outline of reddit-specific problems this fixes:
reddit is heavily biased toward comments posted early. When a mediocre joke gets posted in the first hour a story is up, it will become the top comment if it's even slightly funny. (I know this particularly well because I have dozens of reddit fake identities and with them have posted hundreds of mediocre jokes.) The reason for this bias is that once a comment gets a few early upvotes, it's moved to the top. The higher something is listed, the more likely it is to be read (and voted on), and the more votes the comment gets. It's a feedback loop that cements the comment's position, and a comment posted an hour later has little chance of overtaking it -- even if people reading it are upvoting it at a much higher rate.There are periodically big threads with subjects like "what's your favorite joke/best advice/worst secret/weirdest habit?" A few weeks ago, I took one of these stories when it was 8 hours old and did a count of the top (root) comments. Of the top 10 comments, ALL were posted either "7 hours ago" or "8 hours ago" -- that is, in the first hour or two the story had been up. The "top" list was simply a list of the best jokes from within the first hour. Not a single joke from the last six hours (when most of the comments had been posted) made it near the top. And they never got a chance to; the story fell off the frontpage and they all stopped accumulating votes. They may have been getting upvotes from everyone who saw them, but that didn't let them catch up to the older comments at the top. One effect of this bias, which you may have seen, is posts saying "sorry to hijack your top comment, but I need to post some important information that refutes the main article." They know that no matter how good their comment is, it won't make it to the top.
There's a solution, and it's the new 'Best' ranking. When a few people have voted on a comment, you get a rough idea of its quality. The more people who vote on it, the better an idea you get of where it 'should' ultimately end up. With this algorithm, you quantify exactly how much you can tell about a comment from a limited number of votes.
If everyone got a chance to see a comment and vote on it, it would get some proportion of upvotes to downvotes. This algorithm treats the vote count as a statistical sampling of a hypothetical full vote by everyone, much as in an opinion poll. It uses this to calculate the 95% confidence score for the comment. That is, it gives the comment a provisional ranking that it is 95% sure it will get to. The more votes, the closer the 95% confidence score gets to the actual score.
If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom. (Picky readers might observe that some comments probably get a higher rate of votes, up or down, than others, which this system doesn't explicitly model. However, any bias which that introduces is tiny in comparison to the time bias which the system removes, and comments which get fewer overall votes will stay a bit lower anyway due to lower confidence.)In practice, this system works splendidly. Here's an example from just today (when I wrote this). A redditor, in search of someone from a bar, posted this:
Now, it took a little while for the photographer to find this thread. People posted jokes, and it was only some hours later that the picture was found and the question was answered. Under the old system, her post (which was being furiously upvoted) never made it to the top:
Even if you collapse the top comment and its children, there is another below. The girl's post, after eight hours, was stuck in third place -- despite being linked to heavily.
Now, here's the same comment thread ordered with the 'Best' system:
It's hard to make a better argument for the new system than that.
This new reddit should be smarter and better. I especially look forward to the new iterations of the 'post your best <X>' threads, when most people are using this system and seeing a better sample on which to vote. I hope you enjoy it! See you there.
