Thursday, February 24, 2011

Today RedditGifts announces its first ever individual reddit gift exchange for r/books!

Over 17,000 of you in 92 countries participated in SecretSanta 2010. RedditGifts knows you don't want to wait until next year or even Arbitrary Day to start the gentle stalking and gift procurement, so they have added a new feature. Individual reddits will now be able to hold their own gift exchanges, starting with /r/books.

Wednesday, February 23, 2011

IBM Watson Research Team Answers Your Questions

Below are answers to your top 10 questions, along with some bonus common ones as well. Thanks for taking the time to answer, Watson Team!

1. Could you give an example of a question (or question style) that Watson always struggled with? (Chumpesque)

Any questions that require the resolution of very opaque references especially to everyday knowledge that no one might have written about in an explicit way. For example, “If you're standing, it's the direction you should look to check out the wainscoting.”

Or questions that require a resolving and linking opaque and remote reference, for example “A relative of this inventor described him as a boy staring at the tea kettle for an hour watching it boil.”
The answer is James Watt, but he might have many relatives and there may be very many ways in which one of them described him as studying tea boil. So first, find every possible inventor (and there may be 10,000's of inventors), then find each relative, then what they said about the inventor (which should express that he stared at boiling tea). Watson attempts to do exactly this kind of thing but there are many possible places to fail to build confident evidence in just a few seconds.

2. What was the biggest technological hurdle you had to overcome in the development of Watson? (this_is_not_the_cia)

Accelerating the innovation process – making it easy to combine, weigh evaluate and evolve many different independently developed algorithms that analyze language form different perspectives.

Watson is a leap in computers being able to understand natural language, which will help humans be able to find the answers they need from the vast amounts of information they deal with everyday. Think of Watson as a technology that will enable people to have the exact information they need at their fingertips
3. Can you walk us through the logic Watson would go through to answer a question such as, "The antagonist of Stevenson's Treasure Island." (Who is Long John Silver?) (elmuchoprez)

Step One: Parses sentence to get some logical structure describing the answer
X is the answer.
antagonist_of(X, Stevenson's Treasure Island).
modifies_possesive(Stevenson, Treasure Island).
modifies(Treasure, Island)

Step Two: Generates Semantic Assumptions
island(Treasure Island)
location(Treasure Island)
resort(Treasure Island)
book(Treasure Island)
movie(Treasure Island)



Step Three: Builds different semantic queries based on phrases, keywords and semantic assumptions.

Step Four: Generates 100s of answers based on passage, documents and facts returned
from 3. Hopefully Long-John Silver is one of them.

Step Five: For each answer formulates new searches to find evidence in support or
refutation of answer -- score the evidence.
Positive Examples:
Long-John Silver the main character in Treasure Island.....
The antagonist in Treasure Island is Long-John Silver
Treasure Island, by Stevenson was a great book.
One of the great antagonists of all time was Long-John Silver
Richard Lewis Stevenson's book, Treasure Island features many great
characters, the greatest of which was Long-John Silver.

Step Six: Generate, get evidence and score new assumptions
Positive Examples: (negative examples would support other characters,
people, books, etc associated with any Stevenson, Treasure or Island)
Stevenson = Richard Lewis Stevenson
"by Stevenson" --> Stevenson's
main character --> antagonist

Step Seven: Combine all the evidence and their scores
Based on analysis of evidence for all possible answer compute a final
confidence and link back to the evidence.
Watson's correctness will depend on evidence collection, analysis and
scoring algorithms and the machine learning used to weight and combine the

4. What is Watson’s strategy for seeking out Daily Doubles, and how did it compute how much to wager on the Daily Doubles and the final clue? (AstroCreep5000)

Watson’s strategy for seeking out Daily Doubles is the same as humans -- Watson hunts around the part of the grid where they typically occur.

In order to compute how much to wager, Watson uses input like its general confidence, the current state of the game (how much ahead or behind), its confidence in the category and prior clues, what is at risk and known human betting behaviors. We ran Watson through many, many simulations to learn the optimal bet for increasing chances of winning.

5. It seems like Watson had an unfair advantage with the buzzer. How did Jeopardy! and IBM try to level the playing field? (Raldi)

Jeopardy! and IBM tried to ensure that both humans and machines had equivalent interfaces to the game. For example, they both had to press down on the same physical buzzer. IBM had to develop a mechanical device that grips and physically pushes the button. Any given player however has different strengths and weakness relative to his/her/its competitors. Ken had a fast hand relative to his competitors and dominated many games because he had the right combination of language understanding, knowledge, confidence, strategy and speed. Everyone knows you need ALL these elements to be a Jeopardy! champion.

Both machine and human got the same clues at the same time -- they read differently, they think differently, they play differently, they buzz differently but no player had an unfair advantage over the other in terms of how they interfaced with the game. If anything the human players could hear the clue being read and could anticipate when the buzzer would enable. This allowed them the ability to buzz in almost instantly and considerably faster than Watson's fastest buzz. By timing the buzz just right like this, humans could beat Watson's fastest reaction. At the same time, one of Watson's strength was its consistently fast buzz -- only effective of course if it could understand the question in time, compute the answer and confidence and decide to buzz in before it was too late.

The clues are in English -- Brad and Ken's native language; not Watson's. Watson analyzes the clue in natural language to understand what the clue is asking for. Once it has done that, it must sift through the equivalent of one million books to calculate an accurate response in 2-3 seconds and determine if it's confident enough to buzz in, because in Jeopardy! you lose money if you buzz in and respond incorrectly. This is a huge challenge, especially because humans tend to know what they know and know what they don't know. Watson has to do thousands of calculations before it knows what it knows and what it doesn't. The calculating of confidence based on evidence is a new technological capability that is going to be very significant in helping people in business and their personal lives, as it means a computer will be able to not only provide humans with suggested answers, but also provide an explanation of where the answers came from and why they seem correct.

6. What operating system does Watson use? What language is he written in? (RatherDashing)

Watson is powered by 10 racks of IBM Power 750 servers running Linux, and uses 15 terabytes of RAM, 2,880 processor cores and is capable of operating at 80 teraflops. Watson was written in mostly Java but also significant chunks of code are written C++ and Prolog, all components are deployed and integrated using UIMA.

Watson contains state-of-the-art parallel processing capabilities that allow it to run multiple hypotheses – around one million calculations – at the same time. Watson is running on 2,880 processor cores simultaneously, while your laptop likely contains four cores, of which perhaps two are used concurrently. Processing natural language is scientifically very difficult because there are many different ways the same information can be expressed. That means that Watson has to look at the data from scores of perspectives and combine and contrast the results. The parallel processing power provided by IBM Power 750 systems allows Watson to do thousands of analytical tasks simultaneously to come up with the best answer in under three seconds.

7. Are you pleased with Watson's performance on Jeopardy!? Is it what you were expecting? (eustis)

We are pleased with Watson's performance on Jeopardy! While at times, Watson did provide the wrong response to the clues, such as its Toronto response, it is still a giant leap in a computer’s understanding of natural human language; in its ability to understand what the Jeopardy! clue was asking for and respond with the correct response the majority of the time.

8. Will Watson ever be available public [sic] on the Internet? (i4ybrid)

We envision Watson-like cloud services being offered by companies to consumers, and we are working to create a cloud version of Watson's natural language processing. However, IBM is focused on creating technologies that help businesses make sense of data in order to enable companies to provide the best service to the consumer.

So, we are first focused on providing this technology to companies so that those companies can then provide improved services to consumers. The first industry we will provide the Watson technology to is the healthcare industry, to help physicians improve patient care.

Consider these numbers:
  • Primary care physicians spend an average of only 10.7 - 18.7 minutes face-to-face with each patient per visit.
  • Approximately 81% average 5 hours or less per month – or just over an hour a week -- reading medical journals.
  • An estimated 15% of diagnoses are inaccurate or incomplete.

In today’s healthcare environment, where physicians are often working with limited information and little time, the results can be fragmented care and errors that raise costs and threaten quality. What doctors need is an assistant who can quickly read and understand massive amounts of information and then provide useful suggestions.

In terms of other applications we’re exploring, here are a few examples of how Watson might some day be used:
  • Watson technology offered through energy companies could teach us about our own energy consumption. People querying Watson on how they might improve their energy management would draw on extensive knowledge of detailed smart meter data, weather and historical information.
  • Watson technology offered through insurance companies would allow us to get the best recommendations from insurance agents and help us understand our policies more easily. For our questions about insurance coverage, the question answering system would access the text for that person’s actual policy, the other policies that they might have purchased, and any exclusions, endorsements, and riders.
  • Watson technology offered through travel agents would more easily allow us to plan our vacations based on our interests, budget, desired temperature, and more. Instead of having to do lots of searching, Watson-like technology could help us quickly get the answers we need among all of the information that is out there on the Internet about hotels, destinations, events, typical weather, etc, to plan our travel faster.

9. How raw is your source data? I am sure that you distilled down whatever source materials you were using into something quick to query, but I noticed that on some of the possible answers Watson had, it looked like you weren't sanitizing your sources too much; for example, some words were in all caps, or phrases included extraneous and unrelated bits. Did such inconsistencies not cause you any problems? Couldn't Watson trip up an answer as a result? (knorby)

Some of the source data was very messy and we did several things to clean it up. It was relatively rare, less than 1% of the time that this issue overtly surfaced in a confident answer. Evidentiary passages might have been weighed differently if they were cleaner, however. We did not measure how much of problem messy data effected evidence assessment.

10. I'm interested in how Watson is able to (sometimes) use object-specific questions like "Who is --" or "Where is --". In the training/testing materials I saw, it seemed to be limited to "What is--" regardless of what is being talked about ("What is Shakespeare?"), which made me think that words were only words and Watson had no way of telling if a word was a person, place, or thing. Then in the Jeopardy challenge, there was plenty of "Who is--." Was there a last-minute change to enable this, or was it there all along and I just never happened to catch it? I think that would help me understand the way that Watson stores and relates data. (wierdaaron)

Watson does distinguish between and people, things, dates, events, etc. certainly for answering questions. It does not do it perfectly of course, there are many ambiguous cases where it struggles to resolve. When formulating a response, however, since "What is...." was acceptable regardless, early on in the project, we did not make the effort to classify the answer for the response. Later in the project, we brought more of the algorithms used in determining the answer to help formulate the more accurate response phrase. So yes, there was a change in that we applied those algorithms, or the results there-of, to formulate the "who"/"what" response.

11. Now that both Deep Blue and Watson have proven to be successful, what is IBM's next "great challenge"? (xeones)

We don’t assign grand challenges, grand challenges arrive based on our scientists' insights and inspiration. One of the great things about working for IBM Research is that we have so much talent that we have ambitious projects going on in a wide variety of areas today. For example:
  • We are working to make computing systems 1,000 times more powerful than they are today from the petascale to the exascale.
  • We are working to make nanoelectronic devices 1,000 times smaller than they are today, moving us from an era of nanodevices to nanosystems. One of those systems we are working on is a DNA transistor, which could decode a human genome for under $1000, to help enable personalized medicine to become reality.
  • We are working on technologies that move from an era of wireless connectivity -- which we all enjoy today -- to the Internet of Things and people, where all sorts of unexpected things can be connected to the Internet.

12. Can we have Watson itself / himself do an AMA? If you give him traditional questions, ie not phrased in the form they are on jeopardy, how well will he perform- how tailored is he to those questions, and how easy would it be to change that? Would it be unfeasible to hook him up to a website and let people run queries?

At this point, all Watson can do is play Jeopardy and provide responses in the Jeopardy format. However, we are collaborating with Nuance, Columbia University Medical Center and the University of Maryland School of Medicine to apply Watson technology to healthcare. You can read more about that here:

13. After seeing the description of how Watson works, I found myself wondering whether what it does is really natural language processing, or something more akin to word association. That is to say, does Watson really need to understand syntax and meaning to just search its database for words and phrases associated with the words and phrases in the clue? How did Waston's approach differ from simple phrase association (with some advanced knowledge of how Jeopardy clues work, such as using the word "this" to mean "blank"), and what would the benefit/drawback have been to taking that approach? (ironicsans)

Watson performs deep parsing on questions and on background content to extract the syntactic structure of sentences (e.g., grammatical and logical structure) and then assign semantics (e.g., people, places, time, organization, actions, relationship etc). Watson does this analysis on the Jeopardy! clue, but also on hundreds of millions of sentences from which it abstracts propositional knowledge about how different things relate to one another. This is necessary to generate plausible answers or to relate an evidentiary passage to a question even if they are expressed with different words or structures. Consider more complex clues like: “A relative of this inventor described him as a boy staring at the tea kettle for an hour watching it boil.”
Sometimes, of course, Jeopardy questions are best answered based on the weight of a simple word associations. For example, "Got ___ !" – well if "Milk" occurs mostly frequently in association with this phrase in everything Watson processed, then Watson should answer "Milk". It’s a very quick and direct association based on the frequency of exposure to that context.

Other questions require a much deeper analysis. Watson has to try many different techniques, some deeper than others, for almost all questions and all at the same time to learn which produces the most compelling evidence. That is how it gets its confidence scores for its best answer. So even the ones that might have been answered based on word-association evidence, Watson also tried to answer other ways requiring much deeper analysis. If word association evidence produced strong evidence (high confidence scores) then that is what Watson goes with. We imagine this is to the way a person might quickly peruse many different paths toward an answer simultaneously but then will provide the answer they are most confident in being correct.

14. In the time it takes a human to even know they are hearing something (about .2 seconds) Watson has already read the question and done several million computations. It's got a huge head start. Do you agree or disagree with that assessment? (robotpirateninja)

The clues are in English -- Brad and Ken's native language; not Watson's. Watson must calculate its response in 2-3 seconds and determine if it's confident enough to buzz in, because as you know, you lose money if you buzz in and respond incorrectly. This is a huge challenge, especially because humans tend to know what they know and know what they don't know. Watson has to do thousands of calculations before it knows what it knows and what it doesn't. The calculating of confidence based on evidence is a new technological capability that is going to be very significant in helping people in business and their personal lives, as it means a computer will be able to not only provide humans with suggested answers, but also provide an explanation of where the answers came from and why they seem correct. This will further human ability to make decisions.

Tuesday, February 08, 2011

reddit undergoes a dramatic expansion

Citizens of reddit, we're in a very good place right now, and it's all because of you.

You return again and again, contributing and voting and building the site into something that millions want to be a part of. It gives us eye-popping traffic numbers. And the percentage of you who block our ads is astoundingly small for such a tech-savvy audience. Put together, these factors allow us to ink the big advertising deals which keep this site afloat.

But not all ad revenue comes from big-ticket clients. Through our sponsored link system, you've been promoting your startups and small businesses... even redditors with nothing to sell have been using it to give their blogs and opinions and YouTube videos a boost in traffic and views. These deals are a lot smaller (they start at $20 / day), but there are so many of them that they add up to be another major pillar of our support.

Finally, more and more of you continue to sign up for reddit gold, or give it to each other as gifts. This is very important because it proves that a site like ours can cover the costs of its own growth without being forced to resort to ever-more-annoying ads. Every time someone subscribes to reddit gold, this point gets stronger.

What's that, you say? It sounds like, "Great, you're making money. Whoop-de-do. When are you going to fix the site?"

Welcome, Jason!

Well, this should help: We have a new member of the reddit technical staff. His name is Jason Harvey — a.k.a. alienth — and his arrival doubles the size of our systems administration team to the unprecedented headcount of "two."

In his first week, Jason's already upgraded our monitoring system and built too many RAIDs to count. Going forward, he'll begin addressing a number of infrastructure issues that have been put off for far too long. The impact on you will be a faster and more stable reddit experience. If you have an idea for what title we should give him, let us know in the comments. We were thinking "server monkey" but HR didn't have a code for it.

If you don't recognize that thing on his shirt or that thing in his hand, just check the comments on this blog post. Clearly you've wasted your life studying things other than science fiction cartoons and vexillology.

But wait, there's more!

We're also growing in ways that are more behind-the-scenes. For example, our sales department will also be doubling in size, thanks to Jena Donlin, who will be transferring from another, more boring area of Conde Nast to work full time on securing big contracts for reddit. We promise to keep things unobtrusive in exchange for your continued lack of adblocking, and the money Jena brings in will help pay for the next round of servers and engineers. If you have any advertising ideas or happen to work in the media industry yourself, drop her a line and we'll have her people (i.e., her) talk to your people (i.e., you).

On top of that, two longtime users and overall reddit celebrities are joining the fun. They'll be keeping their day jobs but pitching in for reddit on the side: krispykrackers is going to help us keep up with our endless feedback and customer support queues, and qgyh2 will help manage our Amazon Associates links.

And this expansion isn't over; we still need more programmers, and we're going to address this problem immediately. Stay tuned.

In conclusion...

...we'd like to welcome these new additions to our team and we tip our hat to everyone who made it possible, be it in the form of submissions, community moderation, t-shirt ownership, adblock-not-doing, sponsored-link-buying, or reddit-gold-subscribing.

TL;DR: Four new hires plus more to come. You make reddit grow, and we love you for it.

Monday, February 07, 2011

Charting the Super Bowl through our traffic graph

Last night I was sitting in front of the TV watching the Super Bo--- oops, can't call it that. I was watching a certain athletic sportsmanship contest, and at the same time keeping an eye on our monitoring to make sure reddit was staying up, as I am apt to do any time I'm in front of the computer.

I noticed a fun little trend in our traffic, and I posted it to the sports reddit:

At that point I got up to eat and stopped watching the traffic, but at the end of the game, I came back and took another screenshot of the entire day's traffic:

Apparently a good chunk of you either had a lot of pent up redditing to do, or you wanted to talk about Christina Aguilera or the Black Eyed Peas, the two most popular topics on reddit right after the game, because we saw the absolute highest number of requests we've ever seen.

So thank you all for helping us prove conclusively that people care more about the commercials than the game!

Wednesday, February 02, 2011

reddit: billions served

Our monthly Google Analytics page has a new comma that we'd like to show off:

Never has a single punctuation mark brought us so much joy. There are only about 100 sites on the entire Internet that get a billion pageviews in a single month, and now reddit can put on its smoking jacket and join that exclusive club. The New York Times isn't on the membership list, nor is Expedia,,, or Fox News. In your face, meteorologists!

This is an accomplishment that all redditors should take pride in, because people wouldn't keep arriving in droves — and coming back — if not for the community that you've created here. To see how fast our little site's been growing, contrast this post from July or this one from last month. Oh, and in case anyone was wondering, the billionth pageview was this submission.

Play us off, the Fat Boys!

Tuesday, February 01, 2011

reddit joins the Free Software Foundation! Help us design an ad for FSF.

reddit, like many of you, supports the work of the Free Software Foundation to promote and protect computer user freedom, and fight against threats such as software patents and DRM. In order to show our support, we've joined the FSF as a corporate patron, joining the likes of Google, IBM and Hewlett Packard in our dedication to the cause.

To celebrate this, we've teamed up with the FSF to help design a new logo for their associate membership program. Associate members make up the vast majority of funding for the FSF, and associate members are computer users who, like us, acknowledge and appreciate the importance of free software and the work being done by the FSF. When you join the FSF associate membership program, you join a society of ethical computer users, who value freedom and support efforts to stamp out DRM and software patents. In addition, associate members receive some cool benefits from the FSF:

* Bootable USB membership card
* Free entrance to the LibrePlanet 2011 conference in March
* 20% discount on all FSF merchandise

In order to better define what the FSF's associate membership program is, and to advertise it, we're looking to you, dear redditors, for the design. Come up with the best design, using the reddit alien, the FSF logo, and whatever takes your fancy. The winner will be featured on the FSF website, reddit house ads, and who knows where else. The winner and two runners-up will also receive a copy of Richard Stallman's latest book of essays, Free Software, Free Society, or his biography, Free As In Freedom.

To enter: Simply post up a low-res PNG or JPEG of your design to imgur and link it in your reply to this thread over in r/gnu and let the community judge the best ones. We will take the top ten entries from that thread and create a new thread next Tuesday February 8th at 12pm ET for final voting (This will use the using the same random order/hidden vote totals code we used for Best Of 2010 Voting. Winners will be asked to supply their final design in SVG format where possible.

Download FSF logo HERE and reddit logos HERE

The FSF is also having its annual conference next month. LibrePlanet 2011 will take place on March 19th 2011 in Boston, MA. More information can be found at LibrePlanet.