Being Smart About Aggregated User Content: Lessons Learned from Analyzing One of the Largest Repositories of User Generated Content Ever Assembled

This presentation will show what we’ve learned taking on huge collections of user generated content (“UGC”), with the aim of discovering actionable insights by applying text-mining and semantic analysis. For over a year, we’ve analyzed the entire data set of DISQUS (by far the largest third-party commenting platform, with over 70% market-share and over 300 million active users).

Specifically we will explain and illustrate, by our own case-studies, each of the following “lessons learned”:

When there is no “wisdom of the crowds,” wisdom can be gleaned from data about the crowds
The self-selecting nature of UGC is a blessing or a curse depending on how refined your filters are
In evaluative commenting, what users say is their bottom-line, often isn’t
In forward-looking comments, what users wish would happen can be a better predictor than their actual predictions
It’s a myth that user commentary on articles is either just band-wagon or flame-war; many comments add relevant breadth and depth, and there is a way to harness that

Besides showing a concrete example of each of the above, we’ll give tips on how to effectively manage the massive data set and the fact that it changes around the clock. We will focus on how to uncover insights that are valuable to a variety of stakeholders including marketers, merchandisers, and publishers. We’ll clearly outline the components that anyone needs for “distilling the essence” of massive user generated content.

Tim has worked on cognitive semantics, data science and natural language processing since the late 1990's, resulting in a number of product launches, publications and patents. Tools developed by his teams have generated and/or filtered high volumes of content for numerous leading digital media companies, including Yahoo, CNET, Federated Media, Gannett, Disqus, and many more. Tim’s current company, Temnos, provides best-of-breed content intelligence services to leading content networks and their marketing partners. Tim’s specialty is crossing the chasm between technology and business, recognizing how real-world business needs can be satisfied by synthesizing and productizing research results from diverse disciplines, including not only mainstream artificial intelligence but also philosophy and linguistics. Meanwhile Tim has twice led a startup through growth, acquisition, and integration with a new parent company. A Pepperdine University alumnus, Tim holds a PhD from Claremont.