Easy processing of massive data sets- July 31, 2008 Ronnie Chaiken, Bob Jenkins, Per-Ake Larson, Bill Ramsey, Darren Shakib, Simon Weaver, and Jingren Zhou have an upcoming paper at VLDB 2008, "SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets" (PDF), that describes a parallel data processing tool "being used daily ... inside Microsoft" on "large clusters ... of thousands of commodity servers" over "petabytes of data".Scope is similar to Yahoo's Pig, which is a higher level language on top of Hadoop, or Google's Sawzall, which...http://glinden.blogspot.com/2008/07/easy-processing-of-massive-data-sets.html Kai-Fu Lee keynote at SIGIR- July 21, 2008 Googler Kai-Fu Lee gave a keynote talk yesterday at SIGIR 2008 on "The Google China Experience".The Google China experience has been fraught with difficulties. As Kai-Fu said, Google was expecting that their search engine could succeed in China with the same interface and approach that had worked so well elsewhere, but was "humbled" by the lack of uptake.Why did a Google that had worked well elsewhere not succeed in ChinaKai-Fu argued that two unusual features of Chinese internet users, their..http://glinden.blogspot.com/2008/07/kai-fu-lee-keynote-at-sigir.html Learning diversity when learning to rank- July 16, 2008 Filip Radlinski, Robert Kleinberg, and Thorsten Joachims have a ICML 2008 paper, "Learning Diverse Rankings with Multi-Armed Bandits" (PDF), that attempts to "directly learn a diverse ranking of documents based on users' clicking behavior."An excerpt:We show how clickthrough data can be used to learn rankings maximizing the probability that any new user will find at least one relevant document high in the ranking .... even though web queries often have different meanings for different users.We..http://glinden.blogspot.com/2008/07/learning-diversity-when-learning-to.html Automatic optimization on large Hadoop clusters- July 16, 2008 Chris Olston, Benjamin Reed, Adam Silberstein, and Utkarsh Srivastava at Yahoo Research had a USENIX 2008 paper, "Automatic Optimization of Parallel Dataflow Programs" (PDF), that looks at optimizations for large-scale Hadoop clusters using Pig.The paper says that it is only attempting "to suggest some jumping-off points for optimization work." With much of the paper spending a fair amount of time on "textbook ... optimizations" such as early projection and filtering, operator rewrites, and...http://glinden.blogspot.com/2008/07/automatic-optimization-on-large-hadoop.html Social peer-to-peer and Tribler- July 16, 2008 I finally got to a fun paper I have been meaning to read for some time, "Tribler: A social-based peer-to-peer system" (PDF).What is interesting about the paper is that it proposes a combination of social networking and P2P for content sharing that not only is designed to produce recommendations from friends and users with similar tastes in the network, but also uses those friend relationships to speed downloads, reduce free-riding, and encourage good behavior.An excerpt:Most current P2P...http://glinden.blogspot.com/2008/07/social-peer-to-peer-and-tribler.html Going to SIGIR- July 15, 2008 I will be at SIGIR 2008 next week in Singapore. If you make it to what is very nearly the antipode for those of us in the US, please say hello!http://glinden.blogspot.com/2008/07/going-to-sigir.html Google Toolbar data and the actual surfer model- July 7, 2008 There were a few interesting developments in the past couple weeks from Google that appear to relate to their ability to track much of the movements on the web.As MG Siegler from VentureBeat noted, among many others, Google Trends now offers a feature that shows the traffic many websites get, much like Alexa does using data from the Alexa toolbar. The new functionality also shows similar pages and queries, people who visited site X also visited Y and people who visited site X also searched for.http://glinden.blogspot.com/2008/07/google-toolbar-data-and-actual-surfer.html Black, white, and gray spam- July 7, 2008 Scott Yih, Robert McCann, and Alek Kolcz had a paper at CEAS 2007 on "Improving Spam Filtering by Detecting Gray Mail" (PDF).The paper focuses on training a classifier to detect unwanted e-mail, which they called gray mail, but what excites me more is the broader idea of shades of gray in what we think of as spam. The ideas behind the paper seem to suggest that spam should be more of a continuum. Documents vary from annoying to interesting, with much falling in the middle.Part of my...http://glinden.blogspot.com/2008/07/black-white-and-gray-spam.html Video recommendations on YouTube- July 7, 2008 A WWW 2008 paper out of Google, "Video Suggestion and Discovery for YouTube: Taking Random Walks Through the View Graph" (PDF) presents "a novel method based on the analysis of the entire user-video graph to provide personalized video suggestions for users."Not sure if the same technique was used, but YouTube recently launched (1 2) recommendations.Update: Ten days later, Googler Sapna Mehta announces video recommendations on Google Video.These recommendations appear to be quite different than..http://glinden.blogspot.com/2008/07/video-recommendations-on-youtube.html Google, the press, and tearing down your heroes- July 5, 2008 The press seems to have a pattern reporting on successful technology companies. First, these companies can do no wrong, the heroes of our time, bringing us clever new ways of doing things that promise to dramatically change our lives.Then, reporters start nipping away at the edges. Maybe the hero has flaws Is our hero really good At the first sign of blood, the fangs sink in, and a frenzied attack tears down the hero the press itself had once created.This happened dramatically when I was at.http://glinden.blogspot.com/2008/07/google-press-and-tearing-down-your.html |