25% of the department (i.e., Chris) was in New York on Conde Nast business -- I imagine him smoking cigars with bigwigs while everyone's riding polo horses. The rest of us (David, Jeremy, and me) were preparing for a road trip to the Googleplex to videotape the Peter Norvig AMA.
Right when our trio was getting ready to hit the road, news broke that some reddit users were seeing a pop-up ad right smack in the middle of our site. This was a surprise to us, and wasn't even supposed to be possible, but we had an appointment to get to, so we pulled Chris away from, I presume, the squash court to handle things. I'll pass the mic to him to explain how it went:
So, honestly, the first thing I thought when I saw that post on the front page was, "oh &#*%. I'm going to have to bludgeon myself to death with my own keyboard, because if I don't someone is going to remind me that I said I would" About half a second later I thought, "well, I'm doomed, but I can at least save the site from further abuse," so I shut off all site-wide ads post haste and hoped that this would give us a chance to figure out what had happened.Soon after, we figured out that this was what we will refer to as a "whoopsie" on the part of our ad systems, and that we had accidentally started running ads from another of our parent company's sites (yes, that's what advertising looks like on other parts of the internet. It is considered "normal". Please buy a T-shirt...!) The issue was fixed before we had noticed it, but the damage was done. We've decided internally that the best possible fix for this will be to run the ad unit from an <iframe> on a separate domain which will prevent it from busting out by browser security rules (kind of like what we do currently on video embeds). There was nothing malicious involved. This was, apparently, a "feature" of that ad campaign, though it was not intended for us.
This was all done in the bit of time I had between meetings. [Oh, and before I forget, don't believe what your manservant tells you: polo is overrated.]
Mike here again. Meanwhile, back in Mountain View, the rest of the team had met up with Peter and, after a terrific lunch on Google's dime, grabbed a conference room and began the interview. As Murphy's Law would have it, the site starting acting funny right around the time Jeremy pressed Record.We noticed that our silenced phones seemed to be vibrating a lot more than usual, and so David and I quietly checked things out as best we could while Jeremy ran the show. (When we post the video, you may notice the sound of some frantic off-camera typing.)
As soon as the taping was finished, we started discussing the anomaly. Within a minute or two of us flipping our ringers back on, all of our phones started simultaneously making the "reddit is in *really* big trouble" noise. The site was down; we didn't even have time to give our hosts a proper goodbye, but instead asked if we could sit in the parking lot and mooch some more WiFi.
Google's hospitality, however, was exemplary: they invited us to sit on couches in their lobby for as long as we wanted and help ourselves to all the WiFi we needed. We set up an emergency base of operations next to the reception desk and got Chris on the horn as well. David and Jeremy had brought their laptops along, but mine was 45 minutes away back at reddit HQ, so I had to commandeer one from a generous passerby. (You know who you are. Thanks!)
After an hour of rushed coast-to-coast debugging, we had patched things up well enough to survive the trip home. It had been an exciting day, and the Googleplex is every bit the nerd utopia it's reported to be. It's the sort of place where one can leave a laptop sitting on the floor unattended and know that not only is it perfectly safe, but that nobody's going to bat an eyelash over it.For more information, see:
- Our official statement on pop-over ads
- Your official statement on pop-over ads (warning: dirty words)
