2015 analytics & bi survey

reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

reports.informationweek.com

December 2014 $99

2015 Analytics & BI SurveyA new spirit of experimentation is in the air: For the first time in years, more companies

are trying out multiple analytical tools rather than standardizing on a few favored ones.

Data quality and ease of use remain the toughest challenges. The key question: Will

buyers look for simpler or smarter tools for big data analytics?

By Doug Henschen

Report ID: R8301214

http://reports.informationweek.com

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

CONT

ENTS

TABLE OF

reports Analytics & BI Survey

reports.informationweek.com December 2014 2

3 Author’s Bio

4 Executive Summary

5 Research Synopsis

6 Analytics Showdown: Simpler, Or Smarter?

8 Tableau Goes Mobile

9 Salesforce.com Shoots For Simple

10 School Of Smarts

12 Which Approach Wins?

14 Big Data: Time For Some New Tools

18 Big Data Example No. 1: Utility Companies

20 Big Data Example No. 2: Schema-On-Read

33 Related Reports

Figures

6 Users Accessing Analytics and BI Tools

7 Barriers to Analytics and BI Success

8 Extent of Technology Use

9 Analytics and BI Deployment

11 Current or Planned Use of Analytics and BI

12 Interest in Analytics and BI Technologies

13 Information Management Technologies

14 Factors Driving Interest in Big Data Analysis

16 Factors Driving Interest in NoSQL Databases

18 Big Data Concerns

19 Factors Driving Interest in Hadoop

22 T echnologies Used to Share Analytic and

BI Insights

23 Utilization of Analytics and BI

24 Factors Driving Interest in Advanced

Analytics

25 Analytics and BI Vendors

26 Current or Planned Use of Analytics and

BI Vendors

27 Impediments to Information Management

Success

28 Extent of Information Management

Technology Use

29-32 Survey Demographics


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Want More?

Never Miss a Report!

FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

Doug HenschenInformationWeek Reports

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersec-tion of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of Transform Magazine, and Executive Editor at DM News. He has covered IT and data-driven marketing for more than 15 years.

© 2014 InformationWeek, Reproduction Prohibited


https://www.facebook.com/pages/InformationWeek-Reports/149825495070501

https://twitter.com/IW_Reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

SUM

MAR

Y The world of analytics, business intelligence, and information management is in flux, with old-guard vendors and conventional technologies losing steam and companies enter-taining new options for the big data era. Our InformationWeek 2015 Analytics, Business Intelligence, and Information Management Survey finds:>> 28% of the 297 respondents responsible for analytics and BI software selections say their organizations have “standardized on one or a few analytics and BI products de-ployed throughout the organization.” That’s down from 35% in our 2014 report.>> 21% say their firms use “many analytics and BI products,” up from 16% in 2014.>> 48% of analytics and BI decision makers and influencers say “ease-of-use challenges with complex software or less technically savvy employees” creates a barrier to success, a close second to “data quality problems” for the third year in a row.>> 26% of the 374 respondents involved in selecting or recommending information management technologies say their firms are using NoSQL databases, up from 19% in our 2014 survey.>> 22% say their firms are using Hadoop, up from 15% in our 2014 survey.In this report, we:>> Examine two approaches to addressing ease-of-use in analytics and BI software. One camp favors software simplicity, the other focuses on software intelligence.>> Explore the growing use of Hadoop and consider whether this new platform requires new BI and analytics tools. Respondent breakdown: 35% have 5,000 or more employees; 25% have more than 10,000. Manufacturing, financial services, government, education, and healthcare are well-represented; and 45% are IT director/manager, IT executive management (C-level/VP) level or non-IT executive managers.

EXECUTIVE


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

SYNO

PSIS

RESEARCH



ABOUT USInformationWeek Reports’

analysts arm business

technology decision-makers

with real-world perspective

based on qualitative and

quantitative research, business

and technology assessment

and planning tools, and

adoption best practices

gleaned from experience.

OUR STAFFLorna Garey Content Director [email protected]

Heather Vallis Managing Editor, Research

[email protected]

Find all of our reports at

reports.informationweek.com.

Survey Name InformationWeek 2015 Analytics, Business Intelligence and Information Management Survey

Survey Date October 2014

Region North America

Number of Respondents 384

Purpose To examine adoption trends and strategies around analytics, business intelligence and information management

Methodology InformationWeek surveyed business technology decision-makers at North American companies. The survey was conducted online, and respondents were recruited via an email invitation containing an embedded link to the survey. The email invitation was sent to qualified InformationWeek subscribers.



Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Analytics Showdown: Should Apps Be Simpler, Or Smarter?

The latest software releases from IBM, Qlik, Salesforce, SAS, and Tableau Soft-ware demonstrate different approaches to making data analysis more accessible. Think of the showdown as Simpler versus Smarter. Which strategy will win?

With both approaches, vendors of ana-lytics and business intelligence software are injecting new energy into solving de-cades-old complaints about the complex-ity of this type of software.

We’ve been polling technology profes-sionals on this topic since 2010, and the 2015 edition of our InformationWeek Ana-lytics, Business Intelligence, and Informa-tion Management Survey once again finds that “ease-of-use challenges with complex software or less technically savvy employ-ees” is the second most-cited barrier to success in data analysis (see chart below). For the third year in a row, only data-quali-ty is cited more often as an obstacle.

So what are vendors trying to do about

Figure 1

2015 2014

Which of the following users have access to or utilize analytics and BI today? Users Accessing Analytics and BI Tools

Business analysts

IT management

Data analysts or data scientists

C-level corporate executives (including VPs)

Line-of-business managers

Financial managers

Knowledge workers

Note: Multiple responses allowed Base: 297 respondents in October 2014 and 248 in October 2013 at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek Analytics, Business Intelligence, and Information Management Survey of business technology professionals

55%61%

18%18%

67%68%

18%19%

51%55%

13%17%

42%43%

3%1%

35%40%

43%40%

52%51%

5%4%

9%9%

R8241114/2

Sales force

Customer service reps

Customers

External suppliers or partners

All employees

Every employee and partner


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

the complexity of their software?“There’s a school that believes that

if you can make the software smart enough, it can do the analysis,” says An-thony Deighton, CTO at Qlik. “Another school thinks that if you make the tools easy to use, people can handle even so-phisticated analyses.”

Qlik falls into the latter, “simpler” group, says Deighton, and you can put Saleforce.com and Tableau in that camp as well. Qlik and Tableau have been the fastest-grow-ing vendors in the analytics and BI field in recent years, and both vendors (and their customers) attribute their success to sim-plicity and ease of use.

Qlik’s QlikView software provides an in-memory data-discovery environment while Tableau is known for its data-visual-ization software. Big BI incumbents such as IBM Cognos, MicroStrategy, Oracle, SAP and others have all validated the Qlik and Tableau approaches by adding self-ser-vice data-discovery and data-visualization modules of their own.

Figure 2

What are the biggest barriers to successful analytics or BI initiatives?Barriers to Analytics and BI Success


R8241114/7

2015 2014

Data quality problems

Ease-of-use challenges with complex software or less technically savvy employees

Integration and compatibility issues with existing or multiple platforms

Challenges scaling the technology across the entire organization

Challenges getting constituents to agree on standardized product(s)

No clear ROI

Software licenses are too expensive

46%48%

14%19%

59%51%

26%27%

42%44%

16%18%

37%36%

6%5%

34%34%

2%4%

30%26%

39%40%

10%7%

43%43% 17%

8%

Talent is too scarce or expensive to hire

Training internal staff too time-intensive and costly

Overlap with other products

Lack of industry standards

Lower-than-expected analytic value

No need for BI capabilities throughout our enterprise

Other

None


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Now Qlik and Tableau are upping the ante with even simpler tools — Qlik Sense and Tableau Elastic — aimed at a still-broader base of users. QlikView is used first by analysts and power users who build data-discovery apps that less tech-savvy users can use to answer data-driven questions. Qlik Sense, the vendor’s new application, is geared to self-service data-visualization without the aid of analysts or power users.

“The problem with many data-visualiza-tion tools is that they still have an analyst-centric model,” says Deighton, meaning they require tech-savvy data analysts or business analysts to set things up. “Our fo-cus with Qlik Sense has been to make the tools so easy to use, people can discover the insights for themselves.”

Tableau Goes MobileTableau, one of Qlik’s biggest rivals, ac-

knowledges that its Tableau Desktop soft-ware tends to be used by more data-savvy users, but the reports that those users

Used extensively Used on a limited basis Planned use No current/planned use

To what extent are the following technologies used to share analytic and BI insights within your organization?

Extent of Technology Use

Spreadsheets

Reports (formatted PDF or HTML sent by email or accessed online)

Dashboards (interactive data visualization interfaces)

Scorecards (comparing performance to predefined goals)

Alerts (email, SMS, etc., for exceptions or thresholds)

Data discovery and visualization tools

Query and analysis software (e.g., in-memory what-if planning, OLAP cubes)

Embedded BI (data visualizations within business apps or portals)

Predictive analytics apps

Mobile (smartphone- or tablet-based) dashboards or data visualizations

R8241114/3_Analytics_BI_Information_Management_Survey

R8241114/3

69% 22% 6% 3%

53% 35% 10% 2%

35% 40% 19% 6%

27% 34% 23% 16%

23% 38% 19% 20%

19% 45% 25% 11%

18% 34% 28% 20%

12% 42% 25% 21%

9% 25% 42% 24%

7% 28% 42% 23%

Base: 297 respondents at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek 2015 Analytics, Business Intelligence, and Information Management Survey of 384 business technology professionals, October 2014

Figure 3


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

publish on Tableau Server or Tableau On-line can be “used by almost anyone,” says Ellie Fields, Tableau’s VP of product mar-keting. Using filters and other interactive controls, people ranging from salespeople and doctors to teachers and “other folks you wouldn’t consider to be data analysts” are working with reports on Tableau Serv-er and Tableau Online, Fields says.

Tableau’s attempt to reach a broader base of users is Tableau Elastic, but it’s not a dumbed-down version of Tableau Desktop. Elastic is a tablet app designed for a particular use case — such as when somebody emails you a spreadsheet, but you’re not at your desk and wouldn’t want to fire up a laptop. “If you’re on the go you may not have much time, but you want to do better than reading columns and rows,” Fields says. “With data visual-ization, you can absorb information in seconds that’s not readily apparent look-ing at a spreadsheet.”

Tableau isn’t trying to replace Micro-soft Excel, the most broadly used data-

analysis tool in the world, but it does see an opportunity for a tablet app that pro-vides a visual view of the data stored in spreadsheets.

Salesforce.com Shoots For SimpleThe biggest news in BI and analytics this

year has been Salesforce.com’s Wave Ana-lytics Cloud, which the company has out

Figure 4

2015 2014

Which of the following best describes the way your organization deploys, or plans to deploy, analytics and BI technologies?

Analytics and BI Deployment

We deploy analytics and BI as part of other technology initiatives

We deploy analytics and BI on a project-by-project basis

We have standardized on one or a few analytics and BI products deployed throughout the organization

We have many analytics and BI products scattered throughout departments, operations, and locations

None of the above

Base: 297 respondents in October 2014 and 248 in October 2013 at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek Analytics, Business Intelligence, and Information Management Survey of business technology professionals

28%28%

18%20%

35%28%

21%16%

3%3%

R8241114/1



Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

in limited release. It’s big news because Salesforce has more than 100,000 corpo-rate customers and roughly 25 million us-ers, stats that few BI/analytics vendors can match. In contrast to the biggest players in this arena — such as IBM, Oracle, and SAP, all of which have broad and mostly tradi-tional (and IT-support-centric) BI suites — Salesforce is promising a simple, mobile-first approach.

“The fact that there are only four but-tons on the Wave mobile app shows that we want to reach every user,” says Keith Bigelow, a BusinessObjects veteran who is now GM and senior VP of big data and ana-lytics at Salesforce. “Our customers already have the competitors’ products, but those products weren’t designed to be used by every salesperson. We want our app to be viral, and that’s not the typical approach.”

Wave beta customer GE Capital, for in-stance, “had lots of BI tools to choose from,” Bigelow says, “but they were not able to build an app that was consumable by a salesperson.”

Usability for all is in Salesforce.com’s bottom-line interest. “The single metric that SaaS vendors care about more than anything is adoption,” says Bigelow. “It’s a monthly subscription, and if you don’t have usage, you don’t have revenue.”

However, as simple and usable as the mobile apps may be on the front end, get-ting real value out of Wave Analytics will still require back-end data-integration work and developer/power user decisions about what data to combine and expose to end users.

School Of SmartsAmong the vendors I would put in the

“tools must be smarter” school of thought are IBM and SAS. IBM recently introduced Watson Analytics, a cloud-based service that promises to streamline high-end data analysis. Currently in beta and available (by invite only) after registering online, Watson Analytics makes heavy use of natural-lan-guage interaction. You start by uploading files, and you can then explore the data us-

ing visual tools. Alternatively you can type questions into an Internet-search-style window, or apply use-case templates such as “Improve Campaign Effectiveness” or “Prevent Employee Attrition.”

Despite the Watson branding, the ser-vice is not the same as IBM’s cognitive computing technology, so you won’t have to go through laborious training. Behind the scenes, the service combines technol-ogies that IBM has been working on for some time. InfoSphere Data Refinery ser-vices, for example, are used to assess the data that you upload and automatically suggest corrections, such as deleting du-plicates or merging records that appear to be related. The suggestions are presented in plain English, and users can accept or decline the changes.

Other IBM R&D at work includes Catalyst Engine, a tool that comes out of IBM’s SPSS acquisition that’s designed to automati-cally find correlations and trends in data. Watson Analytics also uses technologies from an IBM Labs effort called Project Neo


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

that automatically suggest which type of visualization to use to best illustrate a trend, outlier, correlation, or other trait.

It’s not as if IBM isn’t thinking about making analysis simpler, since that is what all these natural-language-inter-action and auto-suggestion features de-liver. But given the breadth of this online service, with multiple data-loading and data-cleansing options followed by mul-tiple data-analysis options and use-case scenarios, it feels more like a data-ana-lyst-oriented studio or suite rather than a simple data-visualization tool.

SAS Visual Analytics is another broad option with built-in smarts. Introduced in 2012, this on-premises software also spots trends and patterns of interest within data sets and then suggests the types of visualizations that best expose those traits. In a Visual Analytics up-grade announced in October, SAS added “guided analytics” whereby users can de-fine business goals, such as unit sales or profitability, and then see how other vari-

Figure 5

How do you utilize or plan to utilize analytics or business intelligence? Current or Planned Use of Analytics and BI


R8241114/6

2015 2014

Business activity monitoring

Forecasting

Financial analysis

Predictive analysis

Customer relationship management

Operational process optimization

Sales tracking

Risk management

86%91%

86%92%

90%90%

79%75%

71%68%

70%63%

80%75%

82%81%

68%67%

59%67%

68%65%

55%50%

36%NA

56%49%

61%58%

Product marketing

Competitive analysis

Corporate governance

Product development

Social media sentiment analysis

Fraud prevention

Internet of Things-style connected-device applications


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

ables, such as marketing spend, material costs or other inputs, will have to change in order to meet that goal.

This goal-seeking approach provides a new twist on widely available what-if analysis capabilities. Rather than wast-ing time manipulating data variables and then looking at the impact, you simply input the business outcome you’re after — like 25%-plus profit margin — and the software shows you what conditions or combinations of conditions it will take to reach that goal.

Which Approach Wins?Everyone wants software that’s both

simple and smart, no matter how tech sav-vy they might be. IBM and SAS of course both argue that their smart features make their services and software simpler and easier to use, even for novices.

I put this idea of simplicity versus smarts in front of one super-experienced BI and analytics practitioner — Steve Rimar, se-nior staff IT architect at Qualcomm. The

Figure 6

20151 5

2014

Please rate the level of interest within your organization in the following leading-edge analytics or BI technologies. Please use a scale of 1 to 5, where 1 is "not interested" and 5 is "extremely interested."

Interest in Analytics and Business Intelligence Technologies

Advanced analytics (predictive or statistical analysis, etc.)

Advanced data visualization capabilities (sparklines, treemaps, heat maps, etc.)

Self-service data-discovery and -analysis tools

Embedded BI (reports or visualizations deployed within enterprise apps, portals, etc.)

Analysis of big data, particularly unstructured or nonrelational data

Collaborative BI (tools promoting broad sharing and input on analyses)

In-database analysis for predictive or statistical modeling

In-memory BI or analytics (fast analysis or what-if planning on large data sets)

Mobile BI (alerts, reports, or visualizations delivered to smartphones or other devices)

Software-as-a-service or cloud computing-based BI or analytics

Social media or social network analysis (sentiment analysis, customer influencer, or behavior analysis)

Note: Mean average ratingsBase: 297 respondents in October 2014 and 248 in October 2013 at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek Analytics, Business Intelligence, and Information Management Survey of business technology professionals

3.53.6

3.53.5

NA3.5

3.53.4

3.33.3

3.33.3

3.23.2

3.23.1

3.13.0

2.82.8

2.62.6

R8241114/8

1 5


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

company has used a variety of different tools over the years, including Cognos, Mi-crosoft, and BusinessObjects, but QlikView and Oracle OBIEE have emerged as Qual-comm’s dominant corporate standards.

Qualcomm recently decided it needed something even simpler than QlikView to meet the needs of human-resources and business executives. After an extensive re-view that also included Tableau Software, Qualcomm chose Qlik Sense for that role, and it’s in the midst of big deployment. Familiarity with Qlik and the opportunity to swap and balance QlikView and Qlik Sense licenses had a lot to do with the selection, Rimar admits. But Qualcomm is keeping its options open and it’s also doing a proof-of-concept project on Wat-son Analytics.

What’s Rimar’s take on software simplic-ity versus smarts?

“I always tell people that the future of BI is Google; it’s a text box where you put questions in and get answers,” Rimar says. “But today, no tool is smart enough

Figure 7

2015 2014

Which of the following systems and technologies are used on an extensive or limited basis within your organization?Information Management Technologies in Use

On-premises document or record repository

On-premises data mart(s) or data warehouse(s)

Document imaging or capture (scanning and optical character recognition)

Data integration software (ETL)

Data cleansing or data quality tools

Cloud-based document or record repository

Complex event processing technology

Cloud-based data mart(s) or warehouses

High-scale data mart or data warehouse systems supporting massively parallel processing

Trickle feed or change data capture technologies

NoSQL database(s)

Hadoop

Note: Multiple responses allowed Base: 374 respondents in October 2014 and 298 in October 2013 involved with information management technologies Data: InformationWeek Analytics, Business Intelligence, and Information Management Survey of business technology professionals

27%35%

61%61%

64%58%

62%58%

59%57%

50%47%

30%36%

15%22%

19%26%

25%26%

25%31%

24%34%

R8241114/16



Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

to just tell you the right answers. You have to minimize potential errors by having power users pull the right data together so self-service business, finance, and sales users — or whoever they may be — can draw conclusions from clean data sets.”

The future will undoubtedly bring more new products that combine simple, in-tuitive interfaces with built-in auto-sug-gestion features. Simplicity and software smarts both help, but for the foreseeable future, human expertise still will be re-quired to put the right data together and then explore and visualize it to come up with valid conclusions.

Big Data: Time For Some New ToolsHHadoop is steadily gaining adoption

as an enterprise platform for capturing high-scale and highly variable data that’s not easy or economically viable to store in relational databases. What’s less clear is just how companies are going to ana-lyze all this data.

A recent Forrester report declared that

Hadoop is “no longer optional” for large enterprises. Our data suggests that train hasn’t left the station just yet: Just 4% of

companies use Hadoop extensively, while 18% say they use it on a limited basis, ac-cording to our just-released 2015 Informa-

Figure 8

2015 2014

What data sources or challenges are driving, or would drive, your organization's interest in doing big data analysis?Factors Driving Interest in Big Data Analysis

Finding correlations across multiple, disparate data sources (clickstreams, geospatial, transactions, etc.)

Predicting customer behavior

Predicting product or service sales

Predicting fraud or financial risk

Analyzing social network comments for consumer sentiment

Note: Multiple responses allowedBase: 297 respondents in October 2014 and 248 in October 2013 at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek Analytics, Business Intelligence, and Information Management Survey of business technology professionals

44%46%

43%48%

36%40%

32%27%

29%24%

29%25%

28%23%

24%26%

1%3%

12%14%

R8241114/10


Analyzing high-scale machine data from sensors, web logs, etc.

Identifying computer security risks

Analyzing web clickstreams

Other

Big data analytics is not of interest to my organization


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

tionWeek Analytics, Business Intelligence, and Information Management Survey. That is up from the 3% reporting exten-sive use and 12% reporting limited use of Hadoop in our survey last year. Another 20% plan to use Hadoop, though that still leaves 58% with no plans to use it.

But there’s no doubt that interest in Hadoop is rising. The top draw is the platform’s “ability to store and process semi-structured, unstructured, and vari-able data,” cited by 31% of the 374 re-spondents to our survey involved with information management technology. Another 30% cited Hadoop’s ability to handle “massive volumes of data,” while 25% said it’s Hadoop’s “lower hardware and storage scaling costs” as compared to conventional relational database man-agement systems.

That’s the IT, data-management per-spective on the need for Hadoop. But why is the business looking to capture and analyze big data in the first place? The top driver, cited by 48% of respon-

dents using or planning to deploy data analytics, BI, or statistical analysis soft-ware, is finding correlations across mul-tiple, disparate data sources, like Internet clickstreams, geospatial data, and cus-tomer-transaction data. Next in line are predicting customer behavior, cited by 46%, and predicting product or service sales, cited by 40% of respondents (mul-tiple responses allowed, see chart below). Other motivations include predicting fraud and financial risks, analyzing social network comments for customer senti-ment, and identifying security risks.

In each of these examples, companies are searching for insight by analyzing big data sets that they couldn’t discover parsing the same old data they’ve long held in transactional systems alone. Cap-turing and analyzing clickstreams, server log files, social network streams, and geo-spatial data from mobile apps is a recent, big-data-era phenomenon for most or-ganizations attempting it, and they’re gaining insights and seeing correlations

that just weren’t available in the enter-prise data warehouse.

But pulling insight out of this new data will require some new tools, ones that work alongside Hadoop — which is, at its core, nothing more than a highly dis-tributed file system. Here are the three categories of options associated with Ha-doop, along with product examples:

Hadoop-native data-processing and analysis options: These include Apache Hive (provides SQL-like data access — think data warehousing meets Hadoop); Apache Mahout (supports machine learning on top of Hadoop — think find-ing patterns in data); Apache MapReduce (for searching, filtering, sorting, and forms of processing large data sets in Hadoop — ways to boil down really big data to find the useful nuggets); and Apache Pig (a language for writing MapReduce jobs).

Alternative SQL access/analysis op-tions: Hive is slow by relational database standards, and it doesn’t support all SQL-analysis capabilities. These alternatives


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

are designed to make BI professionals feel more at home, giving them accus-tomed performance, SQL- or SQL-like querying, and compatibility with current BI tools. Examples include Actian Analyt-ics Platform SQL Hadoop Edition, Apache Drill, Cloudera Impala, HP Vertica For SQL on Hadoop, IBM Big SQL, Microsoft SQL Server Polybase, Oracle Big Data SQL, Piv-otal HAWQ, and Teradata Query Grid.

Analytics and BI options designed to run on Hadoop: These tools blend SQL and BI-type querying with big-data-oriented and advanced analytics capa-bilities. Examples include Apache Spark, Apache Storm, Datameer, Platfora, and SAS Visual Analytics. Many of these anal-ysis engines now run on Hadoop 2.0’s YARN resource-management system.

The first thing to note is that the SQL and SQL-like options — including Hive, Impala, Drill, the various relational data-bases ported to run on Hadoop (Actian, HP, Pivotal), and the various SQL-access options (Microsoft, Oracle, Teradata) —

Figure 9

2015 2014

What factors are driving, or would drive, your organization's interest in using NoSQL databases?

Factors Driving Interest in NoSQL Databases

Ability to deal with variable data and fast-changing data models

Interest in simpler or easier management than possible with relational databases

Interest in faster, more flexible development than achievable with relational databases

Lower software and deployment cost than commercial products

Ability to manage high-scale web and mobile applications

Lower hardware and storage scaling cost than commercial products

Other

NoSQL databases are not a priority for my organization

31%27%

17%21%

20%21%

19%18%

21%24%

26%24%

R8241114/18

1%1%

49%50%



Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

give you the basics of SQL query and analysis, but these are not alternatives to analytics workbenches or business intelligence suites. As noted, a key point of these query and access tools is mak-ing Hadoop compatible with incumbent SQL-connected products like Busines-sObjects, Cognos, MicroStrategy, OBIEE, Tableau Software, and so on.

Businesses are demanding compatibil-ity with tools that they already have on hand. This helps explain why there were so many SQL-on-Hadoop announce-ments from both Hadoop vendors (Clou-dera, Hortonworks, MapR) and database incumbents (Actian, Hewlett-Packard, Oracle) over the last year.

But companies need more than SQL. The value in big data analysis is often in finding correlations among disparate data sets or insights hidden in semi-structured or highly variable data sourc-es, such as social networks, log files, click-streams, and even images or sound files. The very name “structured query lan-

guage” underscores that SQL was born for analysis of data that fits neatly into columns and rows, and that’s the stuff most businesses were already collecting in their data warehouses.

As for that interest in finding correla-tions among multiple, disparate data sources; predicting customer behavior; and predicting product or service sales — those types of analyses aren’t in SQL’s wheelhouse.

Understanding such large and variable data sets and doing prediction requires non-SQL approaches such as machine learning, graph analysis, and various ad-vanced analytical algorithms used in pre-dictive modeling and text mining. Even if you have separate tools that support these techniques, the next question is whether they can handle Hadoop-scale data sets. If not, you’ll have to rely on SQL connectors or laborious, two-step move-ment of boiled-down data sets from Ha-doop into relational databases.

The downside of many tools that are

part of Hadoop — like MapReduce, Pig, and Mahout — is that they’re complex and unfamiliar to most IT pros, even data analysts who are used to working with advanced analytics workbenches. The third category of products noted earlier — analytics and BI that run on Hadoop — is designed, not only to run on top of Ha-doop, but also to bring multiple analysis options to nontechnical users who aren’t up to (or interested in) the rigors of cod-ing MapReduce jobs from scratch.

Apache Spark, for example, is an open-source platform that aims to deliver multiple analysis libraries — including machine learning, SQL analysis, R al-gorithms, graph analysis, and stream-ing analysis — that can all run against a single, in-memory execution engine on top of Hadoop. Datameer combines data-integration and preparation tools (for bringing data into Hadoop) with a spreadsheet-style interface for analysis, basic SQL-like join and group-by func-tions, and even predictive analytics and


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

machine learning options, including k-means clustering, decision trees, and so on. Platfora’s strength is in providing fast, in-memory data-exploration and data-visualization on top of Hadoop. SAS, meanwhile, has ported its Visual Analyt-ics product to run on Hadoop and as a standalone clustered-server platform.

The bottom line is big data is not just data warehousing as we know it at high scale. So don’t expect to really tap into new data sources and learn new things with SQL tools alone.

Big Data Example No. 1: UtilitiesAs we await more widespread adop-

tion of Hadoop, good places to look for innovative adoption of tools designed to work with Hadoop are relatively young companies that are using Hadoop exten-sively. Think of these as companies whose businesses are built on big data.

Opower is one such business — a utility analytics company that gath-ers meter-reading data and household

data from more than 100 gas and elec-tric companies. It adds in weather data, property data, and more to get a clearer

understanding of the size and character of homes and the number of people in each household, since those are big driv-

Figure 10

2015 2014

What are your primary concerns about using big data software? Big Data Concerns

Big data expertise is scarce and expensive

We aren't sure how big data analytics will create business opportunities

Our data's not accurate

Hadoop and NoSQL databases aren't compatible with my existing analytics or BI tool

Hadoop and NoSQL technologies are hard to learn


35%36%

47%50%

27%25%

19%19%

19%18%

19%20%

14%17%

14%13%

1%4%

16%15%

R8241114/11

Analytical tools are lacking for big data platforms like Hadoop and NoSQL databases

We don't have enough data

Hadoop and NoSQL technologies lack management features

Other

I have no concerns about big data analytics


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

ers of energy use. The idea is to use that analysis to help homeowners better un-derstand their energy usage so they can reduce consumption and choose more attractive rate plans.

Opower uses Hadoop-native tools, in-cluding MapReduce and Hive, and it uses Datameer for data-transformation and summarization work. Datameer helps power users make clearly tagged and defined data sets available to internal users such as customer-engagement managers and product managers. These business users then use Platfora to build visualizations that help them analyze customer engagement, energy con-sumption, and other trends across vari-ous customer segments.

Opower wanted tools that work na-tively on top of Hadoop, so that the com-pany could look at data across multiple utilities. “If it’s a separate [database] envi-ronment, that’s more infrastructure you have to maintain,” says Mehgann Lomas, a product manager at Opower.

Figure 11

2015 2014

What factors are driving, or would drive, your organization's interest in using Hadoop? Factors Driving Interest in Hadoop

Ability to store and process semistructured, unstructured, and variable data

Ability to store and process massive volumes of data

Lower hardware and storage scaling cost than commercial products

Lower software and deployment cost than commercial products

Interest in new insights, such as social media analysis

Other

Hadoop is not a priority for my organization


29%31%

1%1%

18%16%

21%22%

23%25%

30%30%

R8241114/17


50%51%


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Another concern is ending up with multiple versions of the truth. Even when reporting on something as simple as how many customers received a home-ener-gy report, “I don’t want that stored in too many different places, because you run the risk of having different calculations depending on where you pull that data from,” Lomas explains.

Big Data Example No. 2: The Schema-On-Read Advantage

Vivint is in the home-automation busi-ness, selling a system that lets custom-ers monitor and control security, safety, heating, and air conditioning. The system includes a touchscreen control panel in-side the house and mobile apps through which customers can remotely adjust heating or air conditioning, lock and un-lock doors, and control lights or applianc-es. Sensor options include security cam-eras, thermostats, electronic door locks, door and window sensors, appliance-control switches, motion detectors, and

smoke and CO alarms.Vivint uses Hadoop to store the data

from the more than 800,000 customers it serves. It chose Datameer and Platfora because “Hadoop isn’t user friendly and lacks an intuitive interface,” says Brandon Bunker, Vivint’s senior director of custom-er intelligence and analytics. Without this software, “we would need data scientist and PhD types” to access and make sense of the data.

Many traditional BI tools have connec-tors to Hadoop, but they’re best “when you know what questions you’re going to ask in advance,” Bunker says. “When businesspeople ask me new questions, I get same-day answers without having to create a new schema.”

This schema-on-read capability pro-vides one of the game-changing advan-tages of Hadoop. In a traditional database environment, you have to build a data model in advance, picking and choosing what data to include that you think might be relevant based on the questions that

you suspect you will want to ask. Hadoop lets you store everything without having to fit it to a predefined data model.

Vivint has used this open-ended ex-ploration of the data to help reduce false alarms, a problem that has dogged secu-rity companies for decades.

“You can imagine that there’s a signifi-cant amount of data that can go into un-derstanding when and why false alarms occur,” Bunker says. With Hadoop, you can consider all the data, because you don’t have to make assumptions about causes in advance.

Using Datameer to ingest, summarize, transform, join, and group data; Platfora for data visualization; and algorithms written in R and Matlab for prediction, Vivint can analyze all the data available in Hadoop, as opposed to limiting the scope of analysis by building a data mod-el based on preconceived assumptions about the causes of false alarms.

“We’re able to work with summary data as well as very granular data, so you can

22%of companies use Hadoop

extensively or on a limited

basis

FAST FACT


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Like This Report?

Share it!FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

FollowFollow

Follow

TweetTweet

LikeLike

Follow

ShareShare

Next

let the data do the talking, as opposed to relying on preconceived notions to solve that problem,” Bunker says. Vivint consid-ers its false-alarm analysis a trade secret and a source of competitive advantage.

Vivint has no shortage of tools at its dis-posal; it also uses Tableau Software and is experimenting with Spark (particularly its MLLib machine learning component). Tableau makes data visualization “easy for any user,” Bunker says, but it slows down with larger data sets. When Vivint is ana-lyzing truly big data, it relies on Datameer for its efficient data transformation, and Platfora to provide the equivalent of “an incredibly large OLAP cube in memory.”

It’s not a surprise that companies like Opower and Vivint, which do the bulk of their data work in Hadoop, are more inclined to use software that’s part of or designed to work with that platform. Companies that use Hadoop on a lim-ited basis and that have big investments in relational database management sys-tems and conventional, SQL-oriented BI

and analytics suites will naturally want to make the most of those tools.

But even if you’re in the second camp, key takeaways here should be that not all big data questions can be easily an-swered with SQL, and not all tools de-veloped for small data can cope with data variety or data at high scale. Keep these points in mind before you draw the wrong conclusions about early big data failures. It could be that your failures are due to using the wrong tools or starting with preconceived assumptions about what the data might tell you.


http://www.linkedin.com/pub/ubm-tech/87/293/4b5

http://www.facebook.com/sharer/sharer.php?u= http://reports.informationweek.com/abstract/81/12544/Business-Intelligence-and-Information-Management/2015-analytics-bi-survey.html&t=2015+Analytics+&+BI+Survey&src=sp

http://twitter.com/home/?status=2015+Analytics+BI+Survey +http://ubm.io/1sA7jue @IW_Reports

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next


reports.informationweek.com Decmeber 2014 22

Table of Contents

APPE

NDIX 2015 2014

Which of the following technologies are used on an extensive or limited basis to share analytic and BI insights within your organization?

Technologies Used to Share Analytic and BI Insights

Spreadsheets

Reports (formatted PDF or HTML sent by email or accessed online)

Dashboards (interactive data visualization interfaces)

Data discovery and visualization tools

Alerts (email, SMS, etc., for exceptions or thresholds)


86%88%

88%91%

75%75%

61%64%

54%55%

52%

35%34%

34%47%

65%

61%64%

64%NA

R8241114/4


Scorecards (comparing performance to predefined goals)

Embedded BI (data visualizations within business apps or portals)

Query and analysis software

Mobile dashboards or data visualizations

Predictive analytics apps

Figure 12


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 13

Current use Planned use No current/planned use

How do you utilize or plan to utilize analytics or business intelligence? Utilization of Analytics and BI

Financial analysis

Business activity monitoring

Forecasting

Sales tracking

Customer relationship management

Operational process optimization

Risk management

Corporate governance

Predictive analysis

Competitive analysis

Product marketing

Product development

Fraud prevention

Social media sentiment analysis

Internet of Things-style connected-device applications


R8241114/5Base: 297 respondents at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek 2015 Analytics, Business Intelligence, and Information Management Survey of 384 business technology professionals, October 2014

71% 19% 10% 34%

65% 27% 8%

56% 35% 9%

48% 23% 29%

46% 34% 20%

37% 42% 21%

34% 36% 30%

33% 31% 35%

48% 18%

33% 34% 33%

32% 35% 33%

31% 30% 39%

22% 33% 45%

18% 38% 44%

8% 28% 64%


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 14

2015 2014

What factors are driving, or would drive, your organization's interest in using advanced analytics?

Factors Driving Interest in Advanced Analytics

Desire to optimize business operations (sales, pricing, profitability, efficiency, etc.)

Need to accurately forecast financial or operating results while there's still time to adjust plans

Desire to identify business risk (customer churn, fraud, default, etc.)

Desire to predict promising new business opportunities (e.g., upsell, cross-sell, best new-customer prospects, promising new products)

Need to improve customer understanding and marketing segmentation

Need to stay in compliance with laws or regulatory requirements (money laundering, fair banking, fraud management, etc.)

Other

Advanced analytics is not a priority for my organization


49%51%

67%67%

50%50%

43%44%

2%3%

5%5%

32%31%

43%39%

R8241114/9


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 15

Are you using, planning to use, or evaluating analytics or BI products from the following vendors?

Analytics and BI Vendors

Base: 297 respondents at organizations using or planning to deploy data analytics, BI, or statistical analysis software Data: InformationWeek 2015 Analytics, Business Intelligence, and Information Management Survey of 384 business technology professionals, October 2014

R8241114/12


Using41%30%23%21%20%15%13%7%6%6%4%4%2%2%2%2%2%2%1%1%1%1%1%1%0%0%0%

No use/evaluation

27%43%47%60%54%63%60%76%76%77%78%80%80%88%82%86%86%85%90%88%89%90%82%89%91%91%89%

Evaluated, but not selected for use

8%11%14%9%11%11%12%11%10%6%9%8%8%3%8%6%5%7%4%6%5%6%5%6%3%6%7%

Planning to use10%6%4%3%6%3%4%2%2%4%3%3%4%4%3%3%3%2%2%1%2%2%3%1%2%1%2%

Evaluating14%10%12%7%9%8%11%4%6%7%6%5%6%3%5%3%4%4%3%4%3%1%9%3%4%2%2%

MicrosoftOracleIBM CognosSASSAP BusinessObjects (including Lumira)IBM SPSSTableau SoftwareMicroStrategyQlik (formerly QlikTech)Information BuildersTibco Jaspersoft and SpotfireActuatePentahoAlteryxBirstAdaptive Insights (formerly Adaptive Planning)DatameerLogi AnalyticsArcplanGoodDataPlatforaYellowfin BICloud9 AnalyticsHost AnalyticsKarmaspherePanoramaSisense


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 16

2015 2014

Which of the following analytics or BI vendors' products are in use or planned for use at your organization?Current or Planned Use of Analytics and BI Vendors

Microsoft

Oracle

IBM Cognos

SAP BusinessObjects (including Lumira)

SAS

IBM SPSS

Tableau Software

Information Builders

MicroStrategy


51%51%

9%8%

NA4%

36%36%

NA7%

4%3%

26%27%

6%7%

3%3%

27%24%

NA6%

4%2%

3%5%

2%2%

4%5%

4%5%

6%4%

4%2%

2%2%

2%1%

20%18%

19%17%

9%9%

6%10%

34%26%

4%6%

NA3%

R8241114/13

Qlik (formerly QlikTech)

Tibco Jaspersoft and Spotfire

Actuate

Pentaho

Alteryx

Birst

Datameer

Adaptive Insights (formerly Adaptive Planning)

Cloud9 Analytics

Logi Analytics

Arcplan

Platfora

Yellowfin BI

GoodData

Sisense

Host Analytics

Karmasphere

Panorama


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 17

2015 2014

With your organization's experience in mind, what are the biggest impediments to success related to information management?

Impediments to Information Management Success

Accessing relevant, timely, or reliable data

Integrating data (e.g., extract, transform, load, or data federation)

Accessing or managing content such as Word files, email messages, and presentations

Maintaining reliable and responsive data marts or data warehouses

Organizing and maintaining data models and/or taxonomies

Coping with rapidly increasing volumes of data and/or content


47%49%

58%59%

44%49%

35%32%

33%44%

30%33%

29%24%

27%24%

26%26%

18%12%

5%4%

R8241114/14


Cleansing, deduping, or ensuring consistent data

Reducing data latency and supporting faster decision-making

Extracting data or transactional information from paper-based forms and documents

Processing high-velocity data streams (e.g., financial trade or shipping data)

Other


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 18

Used extensively Used on a limited basis Planned use No current/planned use

To what extent are the following systems or technologies used within your organization?

Extent of Information Management Technology Use

On-premises data mart(s) or data warehouse(s)

On-premises document or record repository

Data integration software (ETL)

Document imaging and capture (scanning and optical character recognition)

Data cleansing and data quality tools

High-scale data mart or data warehouse systems supporting massively parallel processing

Complex event processing technology

Cloud-based document and record repository

Cloud-based data mart(s) or data warehouses

Trickle feed or change data capture technologies

NoSQL database(s)

Hadoop


R8241114/15Base: 374 respondents involved with information management technologies Data: InformationWeek 2015 Analytics, Business Intelligence, and Information Management Survey of 384 business technology professionals, October 2014

36% 22% 15% 27%

36% 25% 12% 27%

32% 25% 15% 28%

27% 31% 13% 29%

16% 31% 22% 31%

14% 17% 19% 50%

12% 23% 15% 50%

11% 25% 19% 45%

10% 24% 16% 50%

7% 19% 15% 59%

6% 20% 18% 56%

4% 18% 20% 58%


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 19

Which of the following best describes your job title?

6%

8%10%

28%

34%

Job Title

Data: InformationWeek 2015 Analytics, Business Intelligence, and Information Management Survey of 384 business technology professionals, October 2014

R8241114/19


IT executive management (C-level/VP)

7%

7%

IT director/manager

IT/IS staff

Non-IT executive management

Line-of-business management

Consultant

Other


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 20

Which of the following dollar ranges includes the annual revenue of your entire organization?

Revenue


R8241114/20


7%

12%

13%

7%

8%

Less than $6 million

13%

15%

13%

12%

$6 million to $49.9 million




$1 billion to $4.9 billion

$5 billion or more

Government/nonprofit

Don't know/decline to say


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 21

What is your organization's primary industry? Industry

Biotech/biomedical/pharmaceutical

Consulting and business services

Education

Electronics

Financial services

Government

Healthcare/medical

Insurance/HMOs


R8241114/21


2%

7%

8%

4%

7%

3% 11%

12%

13%

2%

10%

4%

3%

2%

3%

9%IT vendors

Logistics/transportation

Manufacturing/industrial, noncomputer

Media/entertainment

Retail/e-commerce

Telecommunications/ISPs

Utilities

Other


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next



Table of Contents

Figure 22

Approximately how many employees are in your organization?

25% 6%8%

20%

21%

Company Size


R8241114/22


Fewer than 50

10%

10%

50-99

100-499

500-999

1,000-4,999

5,000-9,999

10,000 or more


Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next


reports.informationweek.com

Table of Contents

MOR

ELIKE THIS

NewsletterWant to stay current on all new InformationWeek Reports? Subscribe to our weekly newsletter and never miss a beat.

Previous Next

Previous Next

DownloadDownload

RegisterRegister

SubscribeSubscribe

Previous Next

Previous Next

Want More Like This?

InformationWeek creates more than 150 reports like this each year, and they’re all free to registered users. We’ll help you sort through vendor claims, justify IT projects and implement new systems by providing analysis and advice from IT professionals. Right now on our site you’ll find:

Big Data in a Loosely Coupled World: Feeling a bit out of control when it comes to your big data plat-form? It is possible to serve the needs of multiple departments while still giving business units autonomy over their own operational software, securely. 2015 State of End User Computing: The 404 respondents to our InformationWeek 2015 Consumeriza-tion of IT Survey have accepted the infiltration of consumer-class tech into enterprise networks. But that doesn’t mean they’re happy about it: just 22% encourage employees to contact tech support for assis-tance with personally owned devices. 2014 US IT Salary Survey: BI & Analytics: Business intelligence, data integration, and data warehousing types are treading water even as they get more productive, but the rare data scientist may be bucking the salary trends. PLUS: Find signature reports, such as the InformationWeek Salary Survey, InformationWeek 500 and the annual State of Security report; full issues; and much more.

December 2014 33

http://reports.informationweek.com/abstract/83/12542/IT-Business-Strategy/Big-Data-in-a-Loosely-Coupled-World.html

http://reports.informationweek.com/abstract/18/12541/Mobility-Wireless/2015-State-of-End-User-Computing.html

http://reports.informationweek.com/abstract/166/12516/Professional-Development-and-Salary-Data/2014-US-IT-Salary-Survey:-BI---Analytics.html




http://links.techwebnewsletters.com/servlet/SignUpForm?f=491839

2015 analytics & bi survey

Documents