case study: digital agency turbocharges social listening and insights with the infochimps platform

© 2012 Infochimps, Inc. All rights reserved. 1

CASE STUDY

Today’s top contenders in the digital marketing and advertising market have a new set of challenges in creating and managing brands. Now, with an abundance of data at their fingertips, agencies are learning how to leverage their customers’ internal data to message and communicate brands and products more effectively. To further strengthen their efforts, they are tapping into technologies that allow them to leverage massive data streams from social and other web sources.

Meet Agency X*, one of the world’s top ad agencies with global brand management of a number of Fortune 500 accounts. Agency X has a proprietary social media listening and insights offering, powered by technologies like Radian6 among others that allows them to acquire and process data to visualize and present brand performance metrics and statistics. Agency X’s social media listening gathers information from social sites, like Facebook, Twitter, Tumblr and more, while the insights offering aggregates metrics from a number of sources including Google Analytics, Facebook Insights and Omniture. Ultimately, the organization was looking to unify their data platforms and offer their team members push-button reporting.

We had realized that the most profitable 20% of our projects were reporting and data related.

It was clear we needed to double down in this area and we quickly discov-ered Infochimps was the right partner to build it out.

- VP - Analytics, Agency X

Digital Agency Turbocharges Social Listening and Insights with the Infochimps Platform

*Agency X’s name has been protected in this case study to protect their competitive advantage and intellectual property.


CASE STUDY

Vision

Diversification. Like many of the top agencies, Agency X is always looking for ways to offer more services to differentiate themselves from their peers and create more value. The best and largest agencies don’t focus on just one aspect of the advertising landscape, but have a whole portfolio of tools and services they can bring to bear - creating more value for their clients, and greater revenue opportunities for the agency, beyond hourly billing for creative work.

Early on Agency X realized the opportunity to differentiate themselves and serve their cli-ents better by creating a social media practice. Social media strategy and reporting are by now just one of the many ways Agency X operates a diversified portfolio of services that can help their clients. After clients started asking for more metrics and measurement around their social media campaigns, Agency X adopted Radian6 as a third party tool to help them with reports. Realizing the power of such a system to gener-ate real value for themselves and their clients, Agency X began looking for ways to have their own proprietary solution which they could fur-ther customize and give themselves greater capabilities.

Repeatability. Agency X is considered a leader in out-of-the-box thinking when it comes to new projects, it’s their bread and butter. However, this can put a lot of stress on the technical de-velopment team responsible for implementing these projects.

Every project seems like it’s breaking new technical ground and there’s always a tight deadline, which means code gets written in a one-off fashion, without long-term, repeated use in mind. And because actions can only be done ad-hoc and by a technical resource, the account teams can usually be left on their own when it comes to creating new reports through one esoteric tool after another.

Led by Agency X’s senior management, and spearheaded by its CTO, the organization de-cided it needed to develop a Platform - a con-sistent set of tools and processes for executing similar projects. Its data and reporting projects were a natural first place to start, as many of the workflows and scripts involved were ripe for consolidation and automation. It was clear that clients and account teams would benefit greatly from push-button reporting and analyti-cal capabilities, saving the technical teams the effort, and getting insights to their clients in near real-time.


CASE STUDY

Profitability. With every new digital project that Agency X took on, technical and human risk had the potential creep up and eat into that project’s margins. Stacking enough of these together meant Agency X could face signifi-cant barriers to growing revenues and profits, as resources were tied up in projects that took longer than they should have and suck up more resources than budgeted.

Revenues and profits are always on senior management’s mind at Agency X. Leading up to working with Infochimps, Agency X realized that a significant portion of their repeatable profits came from data pipelines and reporting related projects. By Agency X’s own account, these were the 20% of projects that generated 80% of their profits. It was clear to them they should double their efforts in this area.

Creativity. At Agency X, one argument against bringing technology platforms in-house was that the focus on locking down toolsets and process-es could restrict the creative possibilities for the organization and the kinds of projects they could take on. Certain employees at Agency X had experience with using legacy data plat-forms, being familiar with Business Intelligence cubes and Relational Database Management Systems, and bemoaned their rigidity and the negative effects it would have on their client projects.

The CTO at Agency X knew that philosophi-cally, these arguments were wrong. Embrac-ing a few constraints and simplifications in one area of a business meant that other areas can be even more freely creative in their solutions - they can depend on the repeatable results and build around them. On a practical level the CTO knew the old-school Business Intelligence tools wouldn’t work for his needs, and he began evaluating the various big data technologies out there to see what could offer the flexibility he would still need within a solution.


CASE STUDY

Challenges

Beyond the complex organizational challenges in the way of moving Agency X into the future, there were a number of technical challenges that needed solved. As an example, for their social media reporting services their clients began asking for greater and greater insight and simplification, including real-time analytics and sentiment analysis. Their teams struggled with scaling the underlying technology to deliver these solutions.

Key challenges preventing the scaling of these systems for Agency X were:

• SourcingandSustainability. Agency X relied on building custom scrapers and data integrations and soon found that sustaining these custom connections with analytics and application layers was messy and ad-hoc at best. Regardless of how an agency obtained their data, whether powered by homemade or third-party aggregators like Gnip and DataSift, the components and collected data were nearly impossible to maintain indefinitely as customer and project priorities shifted.

• ScalabilityandElasticity. As content increased to “Big Data” scale, existing SQL databases were pushed to their limits.

• AnalyticsFeatureDevelopment. Senti-ment analysis, influence scores, and other derivative data are important features that would greatly add to the overall value of the listener and insights platform. However, there wasn’t a common platform and tool set for these features to be developed and scaled out.

• Workflows. Much of the work that Agency X’s existing data platforms perform were done with manual tasks. Agency X needed a solution that enabled the automation of particular data workflows, starting with Senti-ment Analysis and Influencer Reporting.

• ClientCustomization. Customizing and preparing reporting and analytics for any given customer was largely a manual pro-cess. The ability to create customizations easily and efficiently, as well as cater analyt-ics on a per-customer basis, would greatly enhance the efficiency of Agency X teams and increase the speed with which teams could deliver insights to customers.


CASE STUDY

Solution

Infochimps helped Agency X strengthen and extend their listening and insights platform by leveraging the Infochimps Platform, tools, and expertise to unify data sources, process data streams, and visualize insights in real-time, all while satisfying on-demand internal reporting needs.

• SystemUnificationwithinaFlexibleBigDataInfrastructure. By taking advantage of Big Data tools and Ironfan as the provision-ing, configuration, and updates foundation, Agency X can scale to any volume of ap-plications, reporting, and data collection. All applications, reporting, and data are con-nected as part of one unified, modular sys-tem.

• PeaceofMind. Infochimps hosts, main-tains, and monitors the underlying system architecture and tools to greatly minimize the operational resources and mindshare that Agency X must allocate to infrastruc-ture support. Simply put - it just works.

• PushButtonReporting. By leveraging the Hadoop suite of tools for batch analyt-ics, as well as the Infochimps Data Delivery System (based on Apache Flume) for real-time streaming analytics, Agency X can develop analysis and subsequent reports on the fly for high customer responsiveness. The power and simplicity of the Infochimps analytics tools make it easier to develop and productionize analysis and reporting.

• ModularAnalyticsDevelopment. By us-ing the Infochimps Data Delivery Service (based on Apache Flume) and “decorators,” Agency X can add any number of additional analytics enhancements as their feature set grows. These enhancements are modular, meaning Agency X can customize features depending on the customer’s needs, and new analytics can be added without disrup-tion to previously developed metrics.


CASE STUDY

Infochimps Platform Benefits

The Infochimps Platform provided these key benefits to Agency X:

• Diminishedoperationsburdenallowingthemtofocusonbuildingproducts,notmachines• Sustainable, scalable systems powered

by best-in-class technologies like Ha-doop, Flume, Elasticsearch, and more.

• Infochimps hosting, maintenance and administration, support, skills transfer, and Big Data strategy and methodology.

• Asystemthatiseasytochange,grow,andmanage• Built on a foundation of open-source

technologies.• Agile infrastructure development - their

infrastructure is defined with descriptive code. This enables fast iterations, test-ability, and flexibility.

• Modularity and customizability, allowing them to respond dynamically to changing business or customer requirements.

• Powerfulanalyticscapabilities• Batch Big Data analytics with Hadoop

and related tools.• Streaming (real-time) Big Data analytics

powered by Apache Flume and enhance-ments added by Infochimps’ Data Deliv-ery Service.

Working with Infochimps

Infochimps has built a plug-and-play platform, provides all the data you could want from our data marketplace at your fingertips, and we’ve got the experts to make it all just work.

BigDataExperts. We built a uniquely powerful data system that could ingest and deliver mas-sive quantities of data while performing com-plex, resource-intensive analytics in order to create and maintain our market-leading catalog of thousands of data sets and APIs. Now, com-panies looking to gain insights big data come to us to leverage our platform, experience and expertise. Underlying those offerings are a set of cutting edge Infochimps’ technologies and tools, including Ironfan and Wukong, as well as key partnerships.

Integrated. Infochimps’ experience and tool de-velopment in all the other aspects of Big Data, from ingestion and conditioning to low-friction delivery (in the form of our API platform), means that the team understands and is equipped with the tools to successfully navigate and trouble-shoot the entire Big Data ecosystem of an organization.


CASE STUDY

FlexibleCost. Infochimps helps companies manage costs with infrastructure agnosticism. Want to pay for just what you need? We can leverage the flexibility of Infrastructure as a Service (IaaS) providers such as Amazon Web Services and the Rackspace Cloud. Now, all infrastructure costs can be operating expenses (use what you need) and not capital expendi-tures (pay for infrastructure whether you need it or not). Switching from CapEx to OpEx can dramatically lower the funding barrier to adopt-ing Big Data internally in an enterprise. Got bare metal? We’ll set you up with OpenStack and Ironfan so you can create your own flexible, highly-efficient infrastructure environment. Context. Perhaps best of all, the Infochimps Platform can be used to provide context to enterprise’s internal data, whether through public opinion mining (via social networks), geo-located information of many varieties, word corpus training for machine learning, and other commonly useful (but difficult to accumulate) data. All those capabilities combine to make us a great choice for providing Big Data services to the budget and process-conscious enterprise customer.


About Infochimps

Ourmissionistomaketheworld’sdatamoreaccessible.Infochimps helps companies understand their data. We provide tools and services that connect their internal data, leverage the power of cloud computing and new technologies such as Hadoop, and provide a wealth of external datasets, which organizations can connect to their own data.

ContactUsInfochimps, Inc.1214 W 6th St. Suite 202 Austin, TX 78703

1-855-DATA-FUN (1-855-328-2386)

[email protected]

Twitter: @infochimps

Get a free Big Data consultationLet’s talk Big Data in the enterprise!

Get a free conference with the leading big data experts regarding your enterprise big data project. Meet with leading data scientists Flip Kromer and/or Dhruv Bansal to talk shop about your project objectives, design, infrastructure, tools, etc. Find out how other compa-nies are solving similar problems. Learn best practices and get recommendations — free.