Towards Methods for Systematic Research On Big Data

Manirupa Das, Renhao Cui, David R. Campbell, Gagan Agrawal, Rajiv Ramnath

2

• Motivation

• Case Studies

• Characterizing data driven research

• Methodologies and Processes

• How to make data-driven research more systematic

• Conclusion and Questions

3

Motivation

• Big Data characterized by five V’s — of Volume, Velocity, Variety, Veracity and Value.

• Research on Big Data — practice of gaining insights• challenges intellectual, process, and computational limits of an

enterprise. • exploratory and often ad-hoc nature of analytic demands

• Distinct lack of established processes and methodologies• difficult for Big Data teams to set expectations or even create valid

project plans.

4[Source: McKinsey Global Survey Results 2012]

Digital business ranks high on the agenda for many companies

5

Executives do want companies to focus on generating customer insights

[Source: McKinsey Global Survey Results 2012]

6

Motivation

• “Data science” science of extraction of "actionable knowledge", usually from “Big Data”

large volumes of data generated by systems, sensors or devices, orpersonal and social digital traces of information from people.

• While database querying asks, “What data satisfy this pattern (query)?” data-driven discovery asks, “What patterns satisfy this data?”

• Data-driven analytics pipelines thus often comprise the following activities:

(i) Descriptive Analytics (What happened?), (ii) Diagnostic Analytics (Why did it happen?), (iii) Predictive Analytics (What will happen?) and (iv) Prescriptive Analytics (How can we make a desired effect happen?).

7

Motivation

• Traditional database-focused methods optimized for fast access and summarization of data given what the user knows to ask (query)

• But what if we don’t know what exactly to ask of the data?

• Software applications for data science very different from traditional database systems

provide probabilistic answers, and hardware architectures are designed for exploration at scale

• We present our perspectives on characterizing data-driven research

tools, methodologies and processes to make data-driven research more systematic, exemplars of several projects using large, heterogeneous, complex data sets providing ad-hoc tools to query a dataset to answer larger business or research question

8

Case Studies

A. Maximum Entropy Churn Prediction Using Topic Models

B. Mining Emotion-Word Correlations in a Large Blog Corpus

C. Using Latent Semantic Analysis to Identify Successful Bloggers

D. Brand specific tweet classification with user provided topics

9

Project and Characteristics

A – Churn Prediction

B – Mining Emotion-Word Correlations

C – LSA to identify successful bloggers

D – Brand specific tweet classification

10

Project and Characteristicscontd..

A – Churn Prediction

B – Mining Emotion-Word Correlations

C – LSA to identify successful bloggers

D – Brand specific tweet classification

11

Characterizing data driven research

• Typical research starts from a pre-determined goal

• Then collects data and validates and builds models to achieve the goal.

• A data-driven research project, starts from the data, and tries to reveal the pattern or information stored

• Data-driven research is atypical — no clear purpose or outcome at outset, but evolve this in an iterative fashion.

• Highlight certain key considerations to characterize primary research activities

12

Key considerations in primary research activities

A. Clarity About Purpose D. Type of Experiments

B. Methods considerations E. Type of Analytics

C. Type and Availability of Data F. Infrastructure/System considerations

13

Methodologies

Agile — a group of software development methods in which solutions evolve through collaboration between self-organizing, cross-functional teams

Promotes adaptive planning, evolutionary development, early delivery, continuous improvement, and encourages rapid and flexible response to change

Key characteristic criteria for Agile Analytics

Challenges in data-driven research

1. Informal use of language and creation of new words 2. Noise and redundant information

3. Inadaptable methodology 4.High-frequency data generation

1-Iterative, incremental, evolutionary 2-Value-driven development

3-Production quality 4-Barely sufficient processes

5-Automation, automation, automation 6-Collaboration

7-Self-organizing, self-managing teams

14

How to make data-driven research more systematic

Agile analytic steps for data-driven planning and execution of the research pipeline:

1. Information extraction and cleaning

2. Preliminary data analysis

3. Research goal or Hypothesis generation

4. Research data design

5. Model and feature selection

6. Output evaluation

7. Visualization

8. Iterate in value-drive chunks (Agile)

A generalized dataset tries to expand the chance that a single data extraction step can work for multiple projects and purposes.

This does not necessarily mean increasing the volume of the data, but enlarges the coverage of the data

Standardized data processing focuses on some common process of the data that can be abstracted to make it reusable for multiple proj

15

A Process for Systematic Data-driven Research

16

Thoughts

We believe that in today's competitive research and business landscape, it is heterogeneity (variety), the speed at which data is being generated (velocity) and the inconsistency and incompleteness (veracity) of the data, that are the most cross-cutting aspects of Big Data, touching organizations of nearly every type and size. We thus provide a fairly comprehensive overview of research methods and key considerations in characterizing data-driven research, drawing from our experiences in conducting data-driven research projects, including types of available data and experiments.

Using the same, we recommend a process for performing systematic research on Big Data, akin to Agile methodology for software development.

17

Conclusions

While Big data technologies developed mostly out of the large web companies to come up with strategies to process voluminous data, every organization with data, regardless of a global user base, needs to run efficiently and provide the best possible service to its customers

This can be done by harnessing systematic processes to better channel data, in order to provide data-driven value – the most important of the V’s

Questions

How can Big Data serve to support studies designed to perform causal inference,given that the two have opposing starting and ending points?

How does Agile fit into different stages of data driven research?

Can Big Data be a part of a new mixed methods approach where we try to find the individual stories that support the data?

18

The Hidden Biases in Big Data

The Business and the Science worlds are focused on how large [quantitative] datasets can give insight on previously intractable challenges. The hype becomes problematic when it leads to “data fundamentalism”, the notion that correlation always indicates causation and that massive datasets and predictive analytics always reflects objective truth.

Data and data sets are not objective; they are creations of human design. We give numbers their voice, draw inferences from them, and define their meaning through our interpretations. Hidden biases in both the collection and analysis stages present considerable risks, and are as important to the big-data equation as the numbers themselves.

– Kate Crawford(2013, HBR)

19

Thank you

20

Key considerations

A. Clarity About Purpose

1-Basic research: Contributes to fundamental knowledge and theory

2-Applied research: Illuminates a societal concern or problem in search for solutions

3-Summative evaluation: Determines if a solution (policy or program) works

4-Formative evaluation: Improves a policy or program as it is being implemented

5-Action research: Understands and solve a problem as quickly as possible.

21

Key considerations

B. Methods considerations – Qualitative and Quantitative Approaches and Outcomes

“The key to making good forecasts is weighing quantitative and qualitative information appropriately” – Nate Silver

Qualitative - When we want to explore a problem in depth.

Quantitative - Well suited for the testing of theories and hypotheses.

C. Type and Availability of Data • Heterogeneous and Complex Data • Data Ownership and Distribution

22

Key considerations

D. Type of Experiments Field experiments (in natural conditions, e.g., in space),

Laboratory experiments (in artificial conditions)

Qualitative and Quantitative experiments,

Computer simulation experiments

Retrospection , a review of the past events.

Forecasting, a special scientific study of concrete development prospects of an object.

E. Type of Analytics a. Predictive tasks: Classification, Regression, Recommendation

b. Descriptive tasks: Cluster analysis, Anomaly detection, Association Analysis ( used to discover patterns that describe strongly associated features)

F. Infrastructure/System considerations:Scalability, High dimensionality

23

Case Studies

A. Maximum Entropy Churn Prediction Using Topic Models • 1.2 million subscribers, 1.5 years worth of news/blog data from 13

websites, 3.4TB of Server logs.

• Explore structured and unstructured data to come up with predictive models for customer churn

• using features mined from transactional databases or Web-based textual data to determine which factors most impact user engagement

• unique dataset normalization and modeling approach to carve out a future timeframe from the present data for prediction

• topic and metadata features reveal engagement patterns

24

Case Studies

B. Mining Emotion-Word Correlations in a Large Blog Corpus

• Spinn3r dataset for the ICWSM 2009 data challenge - 44 million blog posts spanning 62 days covering some big news events such as the 2008 Olympics, both 2008 US presidential nominating conventions, and the beginnings of the financial crisis. Total size - 142 GB uncompressed

• exploratory study, try to determine whether the words people choose correlate well with categories drawn from a basic theory on emotion

• If successful, can use this information to better predict how blog entries might cluster based on emotion, leading to improved models of information retrieval for blogs

25

Case StudiesC. Using Latent Semantic Analysis to Identify Successful

Bloggers • Spinn3r dataset• In this work, we hypothesized that there may exist characteristics of

language use by informal writers, such as vocabulary or word choice, that are directly associated with successful communication.

• Specifically, we hypothesized a relationship between the vocabulary of a blog and comment density

• used latent semantic analysis (LSA) to reduce the dimensionality of a term-document matrix for each blog in a collection (where a blog is a concatenated set of blog entries).

• Two experiments performed . (i) using an unsupervised clustering approach to see if relationships to comment density naturally emerge, (ii) a supervised classification method to identify high and low comment density blogs, by using two complimentary models built through LSA.

26

Case Studies

D. Brand specific tweet classification with user provided topics

• Many companies take feedback information on their products from Twitter, from where a large number of tweets are collected for data analysis for the purpose of market research

• define some rules containing keywords and simple logic to be able to label some tweets into certain bins of interest

• starting from the brand specific data, and the simple keyword-based logic rules, we build a system that is able to label many more tweets with a certain confidence level using LLDA model

• We collect the mentioning tweets for 5 brands and build a topic model for each of them to better be able to predict tweets for each bin with high precision ~85-90% considering the size of the data.