cio considerations for big data: obtaining real business value from big data

An Enterprise Strategy Program paper

CIO considerations for Big Data

Obtaining real business value from Big Data

Abstract

This paper describes the Microsoft vision for Big Data, discusses key industry trends, and evaluates how to

obtain real business value from investments in Big Data. To highlight the potential that Big Data holds for

organizations of all sizes, this paper presents common business scenarios that include customer analytics,

risk management, science and research, and IT business innovation.

Author

Achim Granzen, Architect, Microsoft Services

Publication date

November 2012

Version

1.0

We welcome your feedback on this paper. Please send your comments to the Microsoft Services Enterprise

Architecture IP team at [email protected]

mailto:[email protected]?subject=Feedback%20on%20CIO%20considerations%20for%20Big%20Data



Microsoft Proprietary and Confidential Information Page ii

Acknowledgments

The author would like to thank the following people who contributed to, reviewed, and helped improve this

paper.

Contributors

Susan Conway, Ulrich Homann, Mike Wise

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date

of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on

the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this

document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic,

mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft

Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in

this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not

give you any license to these patents, trademarks, copyrights, or other intellectual property.

The descriptions of other companies’ products in this document, if any, are provided only as a convenience to you. Any such references

should not be considered an endorsement or support by Microsoft. Microsoft cannot guarantee their accuracy, and the products may

change over time. Also, the descriptions are intended as brief highlights to aid understanding, rather than as thorough coverage. For

authoritative descriptions of these products, please consult their respective manufacturers.

© 2012 Microsoft Corporation. All rights reserved. Any use or distribution of these materials without express authorization of Microsoft

Corp. is strictly prohibited.

Microsoft and Windows are either registered trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.



Microsoft Proprietary and Confidential Information Page iii

Table of contents 1 Executive summary ..................................................................................................................................... 1

2 Microsoft vision - Democratize Big Data ................................................................................................. 2

3 Business motivation .................................................................................................................................... 3

3.1 Big Data opportunities ................................................................................................................................................... 3 3.2 The evolving Big Data platform .................................................................................................................................. 4

4 Key trends ..................................................................................................................................................... 5

5 Enabling business value out of Big Data investments............................................................................ 6

5.1 Sensemaking – Gaining insight from Big Data ...................................................................................................... 6 5.2 Insight decisioning .......................................................................................................................................................... 7 5.3 Innovating your business with Big Data .................................................................................................................. 9

6 Key scenarios .............................................................................................................................................. 10

6.1 Customer analytics ......................................................................................................................................................... 11 6.2 Risk and performance management ........................................................................................................................ 11 6.3 Science and research .................................................................................................................................................... 12 6.4 IT business innovation .................................................................................................................................................. 12

7 Call to action: Getting started with a Big Data strategy ...................................................................... 13

7.1 Our approach to Big Data and Business Intelligence ........................................................................................ 13

8 References and resources ......................................................................................................................... 14



Microsoft Proprietary and Confidential Information Page 1

1 Executive summary Enterprises can now generate, capture, and buy large volumes of data from a variety of sources and then use

analytical methods to gain business insight, support business processes, improve competitive advantage,

and generate profit. The convergence of data availability and analytical innovation is commonly referred to

as Big Data, and many organizations are now starting to use it to produce insights that are relevant to

tactical and strategic business issues.

However, more often than not the use of Big Data starts with a lab approach in IT departments that is driven

by the eagerness of IT staff to examine and understand the latest cutting edge technology. To obtain real

business value out of investments in Big Data, a strategic approach is required to join technical possibilities

with business goals. The Microsoft Enterprise Strategy Program makes resources and offerings available that

can help you get started.




2 Microsoft vision - Democratize Big Data Microsoft envisions all business users having the ability to gain actionable insights from virtually any data,

including insights that were previously hidden in unstructured data.

Microsoft is expanding the vision of Business Intelligence to provide business insight as a service layer for

applications to increase the richness and variety of the Big Data experience. This layer can serve as a new

platform to provide insight into structured and unstructured data of any volume by creating unified and

intuitive approaches to discovering, gathering, storing, indexing, exploring, analyzing, and performing self-

service visualization of Big Data.

Figure 1. Aspects of Big Data

The Microsoft vision of democratizing Big Data aims to provide your organization with the following three

key capabilities:

Manage data of any type or size

▪ Flexible data management layer that supports all data types—structured, semi-structured, and

unstructured data at rest or in motion

▪ Data management and analysis function that can be performed on-premises, in the cloud, or using a

hybrid approach

Enrich your data with the world’s data

▪ Enrichment layer for discovering, transforming, sharing, and governing data

▪ Deeper insights that combine an organization’s data with data and services from external sources

Gain insight from any data

▪ Compelling suite of tools to help users gain insight from analytics

▪ Enable insight decision making for everyone from data scientists to casual users




3 Business motivation

Figure 2. Overview of the Big Data Customer Analytics Framework

3.1 Big Data opportunities Big Data delivers game-changing benefits using a new approach to data acquisition, management, and

visualization for the emerging Big Data platforms.

Organizations have always produced significant amounts of unstructured data from sources such as medical

images, blogs, radio-frequency identification (RFID) tags, and locality sensors. Historically, organizations

threw away most of the data they could collect to avoid what were once considered excessive costs of

managing such a data deluge.

Spurred by plummeting storage and computation costs coupled with a new understanding of the inherent

value of previously discarded data, organizations are demanding new types of business insight from every

bit of data they can access using cost-effective and scalable methods. Examples of new insights include:

▪ Understanding user behavior and online interactions

▪ Identifying trends and popular topics in social media sentiment analytics

▪ Optimizing and targeting advertising campaigns

▪ Discovering medical epidemiological trends (such as identifying the next flu outbreak)

▪ Identifying financial fraud within public sector transactions

Such insights are critical in providing competitive advantages to organizations as well as improving tactical

decisions and controlling costs. To achieve such insights, organizations must invest to create a platform that

accommodates the requirements of Big Data.




3.2 The evolving Big Data platform The initiatives launched in the IT labs of Web 2.0 companies are now evolving into platforms with potential

for broad adoption in all industries. The design, operation, and use of a state of the art Big Data platform has

become easier, and the user circle has widened from data scientists in IT labs to business analysts,

information workers, and, increasingly, information consumers. Eventually, nearly everyone in your

organization can analyze and make more informed decisions with the right tools, as described in the

following example.

Example: Using social media to conduct brand research and promotion

Organizations can conduct brand research and promotion using social media to understand and gain

insight into the opinions of customers and the market (public) about the organization and its products and

services. This research can produce immediate, mid-term, and long-term impact on the top line, such as

market research, brand building, brand protection, product development, and customer service.

Insight obtained from brand research can be used at many levels:

▪ Executive level. Determine new markets, customer segments, products, and services.

▪ Marketing director/CMO level. Develop an organizational brand and overall marketing message.

▪ Marketing campaign managers level. Optimize segments, channels, and campaign for highest returns.

▪ Call center agents. Provide better service to customers by better understanding their motivations and

opinions.

Although its initial uses are mostly tactical, Big Data has the potential to drive real business innovation. After

integrating Big Data analytics into operations to improve day-to-day decision making, your organization will

be poised to start innovating with Big Data (see the “Innovating your business with Big Data” section later in

this paper).




4 Key trends

Figure 3. The volume and impact of data is rapidly increasing

A set of broad industry trends are putting pressure on traditional data management and Business

Intelligence platforms and tools. These trends are as follows:

Increasing data volumes

The annual growth of worldwide information volume is roughly 50 percent and continues to rise. This

explosion of new data is driven by a full range of traditional and nontraditional sources that include sensors,

devices, and tools that monitor and catalog content on the Internet, such as bots and crawlers. According to

an IDC study,1 the amount of digital information created and replicated is forecasted to hit 1.8 zettabytes

(1.8 trillion gigabytes) in 2011—and is predicted to grow by a factor of 44 during the 2009-2020 forecast

period.

Increasing complexity of data and analysis

The real growth in data comes from unstructured data in a wide variety of documents, streaming data, and

click-stream data. The success of search engine providers and e-retailers who have unlocked the value of

unstructured data has debunked the presumption that 80 percent of such data has no value. The business

requirement to store, analyze, and mine a combination of structured and unstructured data is becoming the

new norm.

Changing economics and emerging technologies

Cloud computing and commodity hardware have dramatically reduced the acquisition cost of

computational and storage capacity and is fundamentally changing the economics of data processing. To

create platforms for tackling massive data processing tasks, architects are complementing commodity

hardware with new, distributed parallel processing frameworks (such as Hadoop and MapReduce) and a rich

ecosystem of tools.

These trends provide a variety of opportunities for organizations to obtain business insight that supports

effective decision making and innovation in business processes, products, and services.

1 IDC Digital Universe Study, sponsored by EMC, June 2010.




5 Enabling business value out of Big Data

investments

5.1 Sensemaking – Gaining insight from Big Data To obtain pertinent insight, traditional data platforms require you to pre-identify and structure data of

interest, and to define and apply a data model. However, with the increasing volume and types of data,

people cannot reliably anticipate which data will be valuable or decide which data can be discarded without

risking the loss of insight.

With the explosion of potentially meaningful data—some structured, some unstructured (such as signals,

streaming, social, interaction, and transactions)—new types of analyses are needed. The cycle that holds the

greatest promise for gaining knowledge from Big Data is based on the concept of sensemaking developed

by Pirolli and Card in 2005 within the intelligence analysis community.

Figure 4. Obtaining value from all data sources




Figure 5. The sensemaking cycle is useful for identifying valuable data and questions.

Sensemaking uses continuous feedback and defines interdependent relationships to create a context that

provides information about data and supports analysis of the data. Big Data expands this cycle to the

unmodeled domains of unstructured content.2

One of the most significant aspects of sensemaking is how it transforms traditional analytical processes:

▪ Traditionally, data comes to you. The data is then structured through an information management

process (such as extract, transform, and load) after which enterprise stakeholders can analyze it.

▪ Now, knowledge and insight come to you. You define your business objectives and then allow the

sensemaking process to map them to data in real time, drawing out knowledge that may form the basis

for immediate action, such as modifying product prices in real time.

Sensemaking is emerging within Big Data environments as a powerful method for culling promising

information from torrents of data, as well as helping identify questions that business decision makers should

ask to meet their business objectives.

5.2 Insight decisioning The connections between an organization, its data, and its processes defines the technologies that support

traditional structured data as well as unstructured data and advanced analytics within an insight decisioning

platform. The following figure illustrates the conceptual model for an insight decisioning platform.

2 Klein, et al, “A Data/Frame Theory of Sensemaking.” June, 2008




Figure 6. Conceptual model of an insight decisioning platform

An insight decisioning platform consists of four different types of components: consumption, insight,

integration, and analysis.

The nature of the insight decisioning process calls for Big Data consumption components from cloud

environments as well as internal business data. The insight platform may use a hybrid environment or a pure

cloud environment that processes and transfers refined Big Data results into an on-premises analysis

environment. Resources that are available in the cloud enable organizations of all sizes to enrich their data

sources, often using a pay-as-you-go model that allows them to apply analytics to selected sets of data,

without incurring significant cost.

Insight components provide the following features: Hot stream data (alert information such as customer

credit issues); cold stream data (latent information, such as unstructured data from search engines that track

banking service inquiries); and multiple decision points about a sale or upsell. Alert features, such as those

provided by StreamInsight have the ability to process time series data in near real-time.

Integration components combine results from the Big Data interrogation with results of similar

interrogation of structured business data. The data analyst facilitates the process by defining the mappings

of the unstructured and structured findings to create an integrated result for presentation and general

consumption.

Analysis components provide a business analyst with results to review, such as a customer’s ability to make

loan payments, or upsell opportunities.




5.3 Innovating your business with Big Data A typical approach for innovating your business with Big Data might consist of the following three

components.

Big Data foundation

A foundational infrastructure is necessary to use, manage, and maintain Big Data. The infrastructure can be

integrated into existing Business Intelligence and operational systems, and may be deployed on-premises, in

the cloud, or as a combination of the two. The infrastructure should also include a sandboxed environment

for employees to acquire new skills and experiment with data.

Key benefits include:

▪ Enable subsequent Big Data activities

▪ Enable development of innovative, customer-specific approaches and solutions

▪ Enable ad hoc analysis and discovery for simple, tactical use cases

Big Data operational analytics

Big Data enriches predictive analytics to improve day-to-day operations and tactical business decision

making. Big Data operational analytics use near real-time data (hot stream) and non-real-time data (cold

stream) to help ensure agile and well-founded business action.


▪ Improve operations in all targeted business areas, building on richer, more accurate and detailed

business insight (for example, achieving high-yield campaigns using customer social media behavior)

▪ Become more agile in business operations and tactical decision making

Big Data innovation

Big Data drives business innovation. Using closed-loop marketing techniques, Big Data can enhance

branding and drive sales. Big Data can also reveal opportunities for market-driven products and services,

and identify new business areas and markets for an organization to pursue.


▪ Widen the idea pipeline

▪ Shorten the reaction time to changing business and sales conditions

▪ Increase the agility of decision making processes and strategic decision making




6 Key scenarios

There are four main business scenarios in which Big Data is starting to be used as an extension to the

traditional Business Intelligence and analytical systems:

Figure 7. Big Data scenarios




There are additional business scenarios for Big Data, particularly in online search, advertising and managing

social networks. However, this section discusses business scenarios beyond web analysis scenarios.

For overview purposes, this section presents cross-industry scenarios, although some scenarios might be

more common in certain industries. For example, customer analytics is especially important for retail and

telecommunications companies, whereas financial services organizations focused more on risk and

performance management. Also, science and research is a hot scenario for the energy industry to support

the search for natural resources. And pharmaceutical companies might consider all four scenarios of equal

priority.

6.1 Customer analytics Going beyond classic customer analytics that use internal, structured

data, Big Data customer analytics can create a multifaceted view of a

single customer, with actionable business insights from such diverse

sources as point-of-sale transactions, loyalty data, online/web activity,

lifestyle information, market research, demographic data, marketing

channel responses, direct communications, and social media.

Classic customer analytics tools (such as segmentation, churn, and

cross-/up-sell) are enriched and complemented by analysis of text, sentiment, marketing, and advertising to

enable closed-loop processes for optimizing marketing techniques and portfolios, predicting retail behavior,

improving customer service and satisfaction, conducting brand research, protecting brand value, and

developing new products and services.

Sample use cases involving customer analytics include:

▪ Multichannel campaign management and predictive customer analytics

▪ Social media brand promotion and sentiment analytics

▪ New product and services development

6.2 Risk and performance management Risk and corporate performance management involves primarily

financial and operational risk/portfolios with a primary goal of obtaining

an optimal state that yields the highest benefits (financial, safety, and so

on). A key application is trading and investment portfolio management,

specifically in high-frequency trading and short-term investment, and in

financial risk management for mid and long-term (insurance).

Sample use cases:

▪ Energy trading and risk management (ETRM)

▪ Improved actuarial analysis

▪ Computerized high-frequency trading and portfolio optimization




6.3 Science and research Science and research involves both the public sector and private sector and uses

varied data source such as meteorological data, fundamental research such as

subatomic research, astronomy, medical/pharmaceutical/genetics research, and

natural resources exploration and exploitation. In a wider sense it also deals with

areas such as real-time monitoring in manufacturing and production.

Sample use cases of science and research include:

▪ Natural resources exploration and prospecting

▪ Pharmaceutical clinical trials

▪ Meteorology and the natural disaster research

6.4 IT business innovation IT organizations across all industries are constantly being evaluated

against the services and benefits they provide to their enterprises. Simply

“keeping the lights on” has become an outdated metric for many IT

organizations: they are now under pressure to innovate and participate

as a business enabler.

Big Data and cloud technologies provide unique opportunities for IT

organizations to examine and redefine their current services, establishing new creative ways for helping

employees do their business better while enhancing operational efficiency.

Sample use cases of IT business innovation include:

▪ Develop new innovative IT services to enable internal/external business clients

▪ Streamline IT operations and modernize technology platforms to exploit today’s opportunities




7 Call to action: Getting started with a

Big Data strategy

Microsoft helps customers assess their approaches to Big Data as part of the Enterprise Strategy Program

(ESP). This program focuses on helping enterprises realize value from their IT investments and consists of

four core elements:

▪ Enterprise architects. Dedicated to the customer and charged with accelerating customers toward their

business goals.

▪ Microsoft network. Subject matter experts from across all areas of Microsoft, including product groups,

research and development, internal IT resources, and Microsoft Research.

▪ Value Realization Framework (VRF). A framework and methodology designed to deliver on the value

proposition of the program.

▪ Library. A collection of exclusive intellectual property including comprehensive guidance, reference

architectures, implementation information, and worked examples from Microsoft engagements.

7.1 Our approach to Big Data and Business Intelligence Our approach focuses on how you can programmatically extend your traditional Business Intelligence

infrastructure to leverage Big Data techniques, aligning the right technologies to help enable and obtain

measurable benefits from your Big Data strategy.

The Big Data Workshop is specifically designed to help you get started with a strategic, business value-

focused approach to Big Data. On a high level, the workshop provides the following:

▪ Rapidly models the Big Data environment in your enterprise

▪ Identifies gaps in your current Big Data/Business Intelligence infrastructure

▪ Finds opportunities to generate value using Big Data techniques and tools

▪ Proposes initiatives to address those opportunities

▪ Augments the initiatives with metrics that measure adoption and realization of opportunity value

For more information about the Microsoft Enterprise Strategy Program, contact your Microsoft account

representative or visit www.microsoft.com/GoESP.

http://www.microsoft.com/GoESP




8 References and resources Sources referenced in this white paper include:

▪ IDC Digital Universe Study, sponsored by EMC, June 2010.

▪ Klein, et al, “A Data/Frame Theory of Sensemaking.” June, 2008

Additional resources from Microsoft include:

▪ Conway, Susan. “Obtaining Insight from Big Data.” Microsoft, 2011

▪ Conway, Susan. “Making Decisions with Insight: A process and platform for generating business insight.”

Microsoft, 2012.

▪ Wise, Mike. “CIO considerations for Business Intelligence: Obtaining competitive insight.” Microsoft, 2012.

cio considerations for big data: obtaining real business value from big data

Technology

natural resources

obtaining

traditional

conduct brand

big data strategy

key benefits

insight decisioning

business intelligence