silobreakers: knocking down the barriers to data insight · warehouse, but seven, eight sources or...
TRANSCRIPT
Silobreakers:Knocking down the barriers to data insight
2
Businesses today have access to more data than ever, in more places than ever.
They want to use this data to understand their customers, and make smart
decisions, but it’s scattered in silos across the company. That’s the dilemma of
the financial analyst: surrounded by data, with no easy way to get value from it.
By some estimates, analysts waste half their time manually preparing data
instead of interpreting it. To solve this problem, businesses need to knock down
their silos and give analysts the power to choose their data sources, define their
own calculations and analyze data using the tools they prefer. But to do that,
they first need to understand the challenges they face at every step in the data-
insight lifecycle, and the best practices that can drive a truly effective data-
access solution.
Abstract
RENEWAL OPPORTUNITY
LOW
HIGH
LOW
HIGH
MEDIUM
MEDIUM
HIGH
LOW
20
80
25
70
60
50
90
10
HEALTH
45,000
28,000
15,000
25,000
30,000
20,000
20,000
14,000
USAGE
$80,000
$65,000
$110,510
$75,000
$90,000
$10,000
$25,000
$50,000
ARR
Verizon
Pepsi
GE
Pfizer
Delta
Boeing
AMEX
Apple
CUSTOMER
7
6
8
9
5
4
3
2
1
A B C D E F
=Kloud("SF_saleforce_customer_revenue”)
Customer Renewal Report
Frequency: Daily
Timezone: Los Angeles PST
8 00 AM
Run Report
STEP 3: Autorefresh
STEP 2: Filters
US Region
Customer Renewal Report
STEP 1: Select a Report
Kloudio Run a Report
=Kloud(“Salesforce Report”,“customer name”,A2)
3
Ask any executive what it takes to make good decisions. Judgment would rank
up there. Humility. Experience. All important.
But the definitive factor in business today is data. Businesses strive toward
“data-driven decision making,” leveraging data to make smarter business
decisions and gain a 360-degree view of the customer.
It’s an unmissable opportunity—and a frustrating dilemma.
First the opportunity. After studying the relationship between data-driven
decisions and business outcomes, researchers at the MIT Center for Digital
Business found that firms that moved toward data and analytics significantly
improved their productivity and profitability.
The dilemma is what financial analysts struggle with every day.
Volume is part of it. Everything that happens online—shopping, streaming,
banking—generates data. Our applications generate data. The devices they run
on generate data. The networks that carry the data generate data.
According to the most recent IDC report, what it calls the “global data sphere”
will reach 175 zettabytes, or 175 trillion gigabytes, by 2025. For financial
analysts, that includes data on product usage, support, feature requests,
renewals and much, much more.
The data-insight dilemma
4
AND REPEAT…
MANUALUPLOAD
COPY / PASTE & PIVOT
MERGING & FILTERING DATA
ANALYSTENGINEER
SEPARATE DATABASES
SQL QUERIES
CUSTOMCODE
But volume isn’t the biggest problem. The biggest problem is that the data
analysts need comes from vastly more data sources than it used to.
In the past, a business might have worked with SAP, Oracle or Microsoft for all
their data needs. Now it might use one application for back-office accounting,
another for CRM, yet another for subscription billing.
IDC found that data workers use an average of 6 data sources and 40M rows of
data for each activity. It’s the digital version of the proverb about the blind man
and the elephant. Each source tells part of the story. None, by itself, tells much
at all.
The finance team is expected to derive insights from all this customer data, in all
these different stores, but the engineering team is the only one empowered to
access it. Only engineering has the tools and the expertise needed to cobble
together disparate datasets through custom coding.
Analysts don’t want to learn SQL or write code. They just want to answer
questions about the business. Who is renewing? What does our churn rate look
like? What are our CACs? Which customers are escalating to higher tiers of
support? How heavily are our products being used?
5
Analysts, in other words, just want to get value out of their data.
Unfortunately “getting value” means haggling with IT, writing specs, filing
support tickets and manually cutting, pasting and tweaking to get the data to
line up in a spreadsheet.
That’s the heart of the data-insight dilemma: the more digital and data-oriented
business becomes, the more tedious and frustrating the analyst’s job.
According to IDC, data workers like financial analysts spend 90% of workweek
on data-related activities, Which sounds good until you learn that 73% of that
time is spent searching and preparing data and only 23% on analysis and insight.
A study by the American Productivity & Quality Center found FP&A analysts
mired in data prep. Respondents reported spending 75% of their time gathering
data and navigating processes.
Inefficiencies like these undoubtedly contribute to the $3.1 trillion that IBM
estimates poor-quality data costs companies each year.
The upshot
Business Insights27%
Silos of Data73%
DATA GATHERING
SUPPORT COST?
GROWTH POTENTIAL?
CHURN RISK?
6
If you manage data analysts or are one yourself, the data-insight dilemma is
your day-to-day reality. And you’re probably looking for a solution.
The good news is, the silos keeping you from insight can be broken. But you
can’t do it halfway. You need to break them at both a technological and
organizational level by asking these questions:
Breaking silos
From a technological standpoint, how do you remove the custom coding
and SQL queries needed to connect and consolidate data from multiple
sources?
From an organizational standpoint, how do you enable analysts to access
the data they need without waiting on IT? How can analysts build off each
other’s work and collaborate on getting insights needed by the business?
Finally, how do you publish the results of analyses throughout the
organization in the format preferred by your target audiences?
We call this end-to-end process the ‘data-insight lifecycle’ and it has 5 phases:
Connection, Consolidation, Analysis, Collaboration and Publication.
Break silos in one part of the process without breaking them in the rest, and
you’ll find yourself cutting, pasting and filing tickets as much as ever.
The trick is knowing what to knock down and what to build up. Here’s what you
need to keep in mind at each stage of the process.
VISUALIZE
SHARE
SHARE
RENEWAL OPPORTUNITY
LOW
HIGH
LOW
HIGH
MEDIUM
MEDIUM
HIGH
LOW
20
80
25
70
60
50
90
10
HEALTH
45,000
28,000
15,000
25,000
30,000
20,000
20,000
14,000
USAGE
$80,000
$65,000
$110,510
$75,000
$90,000
$10,000
$25,000
$50,000
ARR
Verizon
Pepsi
GE
Pfizer
Delta
Boeing
AMEX
Apple
CUSTOMER
7
6
8
9
5
4
3
2
1
A B C D E F
=Kloud("SF_saleforce_customer_revenue”)
Customer Renewal Report
Frequency: Daily
Timezone: Los Angeles PST
8 00 AM
Run Report
STEP 3: Autorefresh
STEP 2: Filters
US Region
Customer Renewal Report
STEP 1: Select a Report
Kloudio Run a Report
=Kloud(“Salesforce Report”,“customer name”,A2)
7
The data-insight lifecycleConnec3on: accessing data
Before you can use data, you have to get at it.
From a technological standpoint, connecting to data sources and pulling data is
easy—as long as you’re happy navigating APIs, RDBMSs and SQL. Most
connection tools are designed for IT and database experts, not analysts and
other line-of-business types.
That’s why the Connection phase of the data-insight lifecycle is usually off-
limits to analysts. It’s why analysts need to file a ticket to add a connection or a
new piece of software.
This practice is not only inconvenient, it’s a major source of delay. The longer it
takes to make decisions, the longer it takes to get business value. Separating
analysts from the data sources they draw from has real costs—both in terms of
opportunity and the bottom line.
20
80
60
50
90
10
45,000
28,000
30,000
20,000
20,000
14,000
$80,000
$65,000
$90,000
$10,000
$25,000
$50,000
Verizon
Pepsi
Delta
Boeing
AMEX
Apple
CUSTOMERRENEWAL OPPORTUNITY
LOW
HIGH
MEDIUM
MEDIUM
HIGH
LOW
8
Consolida7on: merging, filtering and sor7ng
Connection is about making data available. Consolidation is about making it
useful. Also known as data munging, data wrangling or data transformation,
Consolidation is the process of combining siloed data from different sources
and arranging it in a specified format.
Think about what goes into a renewals analysis. The financial analyst needs to
know how many licenses to renew. That data lives in Salesforce. Then the
analyst needs to know if the customer is using the product. That data lives in
Segment. Then, she needs to know if the customer is satisfied. That data lives in
Gainsight, Zendesk, and possibly Jira. One “simple” analysis—as many as five
different data sources to pull together.
Everything about Consolidation depends on the analyst’s needs. But from the
analyst’s perspective, the process is a black box. The analyst needs to write
everything down in an IT ticket that specifies the data required, the sources it
comes from, and the format it needs to be delivered. Then the engineers—when
they get to it—write a query to pull in the data and combine it into a single
report, often a gigantic file in comma-separated-value (CSV) format.
Believe it or not, that’s the better of the two common scenarios. In the
alternative, which is even worse, financial analysts consolidate data themselves,
cutting and pasting raw data dumps into a spreadsheet, then painstakingly
linking them into some semblance of order. It’s crazy, wasteful, mind-numbing
work. But it’s common practice at some of the world’s largest companies.
Remember, analysts are no longer managing data from one master data
warehouse, but seven, eight sources or more: databases in the data center,
services in the cloud, database services in the cloud. Pulling data from that
many sources is a chore. Consolidating it—especially on an ongoing basis—is an
expensive, time-consuming pain in the neck.
The solution: Give analysts the ability to manage Consolidation themselves,
using tools that don’t require them to write code. They’ll thank you, and so will
the engineering team.
9
Analysis: refining data in real-1me
After data is pulled and consolidated, it needs to be refined. That’s the purpose
of the Analysis phase. This is where calculations are performed on the raw data
to produce the metrics the analyst needs.
For instance, suppose an analyst wants to report on customer acquisition cost
(CAC). The calculation requires two pieces of raw data: the cost of acquiring
each customer over a certain period of time, and the number of customers
acquired over the same period, with the former stat divided by the latter. The
analyst doesn’t need to see the raw data, just the overall, aggregated CAC.
Typically, the analyst receives a metric like CAC one of two ways: as a static file,
or within a business-intelligence tool like Tableau or Looker.
The static-file option is the less desirable option. In this scenario, the analyst
receives the metrics from engineering and manually loads them into a pivot
table for slicing and dicing. Aside from inconvenience, the big problem with this
approach is that metrics go obsolete as soon as they’re generated.
Businesses—especially SaaS businesses—increasingly rely on time-sensitive
consumption-based metrics like cohort analysis. The value of these metrics to
the board, say, or the management team, starts to decline as soon as they’re
calculated. Ideally the analyst needs metrics to update in real time, as the
underlying data changes.
Business intelligence tools can provide real-time metrics. But they still depend
on engineers to connect data sources and implement the analyst’s calculations.
It’s the same sort of inefficiency we discussed in the last phase, but worse.
10
Whenever one group implements the business logic of another, it’s a recipe for
trouble. Engineers don’t know how to calculate CAC, or any other financial
metric. The analyst has to define it in a spec, send it to the engineers and hope
they implement it properly.
Maybe they will, but often they don’t. Messages get garbled. Data gets
miscalculated. Failures happen silently. Numbers arrive on time, but no one
notices they’re wrong until the quarterly report comes out — or worse, after.
And it’s not just about errors, but delay. Analysis is supposed to be an open-
ended, iterative process. But when engineering has to hard-code calculations,
every little change an analyst might want to explore — a different time span, a
different geography, a different product — means another ticket for IT.
Under those circumstances, how likely are analysts to explore new hypotheses?
Dig into anomalies? Try to do anything beyond what they’ve been tasked with?
One thing is for sure: it’s no way to make data-driven decisions.
As a rule, analysts need to control their calculations, and those calculations
need to update in real time.
RENEWAL OPPORTUNITY
LOW
HIGH
LOW
HIGH
MEDIUM
MEDIUM
HIGH
LOW
20
80
25
70
60
50
90
10
HEALTH
45,000
28,000
15,000
25,000
30,000
20,000
20,000
14,000
USAGE
$80,000
$65,000
$110,510
$75,000
$90,000
$10,000
$25,000
$50,000
APR
Verizon
Pepsi
GE
Pfizer
Delta
Boeing
AMEX
Apple
CUSTOMER
7
6
8
9
5
4
3
2
1
A B C D E
=Kloud(“Salesforce Report”,“customer name”,A2)
Customer Renewal Report
RENEWAL OPPORTUNITY
LOW
HIGH
LOW
HIGH
MEDIUM
MEDIUM
HIGH
LOW
20
80
25
70
60
50
90
10
HEALTH
45,000
28,000
15,000
25,000
30,000
20,000
20,000
14,000
USAGE
$80,000
$65,000
$110,510
$75,000
$90,000
$10,000
$25,000
$50,000
ARR
Verizon
Pepsi
GE
Pfizer
Delta
Boeing
AMEX
Apple
CUSTOMER
7
6
8
9
5
4
3
2
1
A B C D E
=Kloud(“Salesforce Report”,“customer name”,A2)
Customer Renewal Report
11
Collabora'on: sharing data across analyst teams
Financial analysts work as a team, especially in larger companies. HQ is the hub.
The regions are the spokes. But all too often, they aren’t on the same page.
During the budgeting process, for example, the FP&A group at HQ consolidates
and analyzes data, and generates core numbers for the rest of the company.
Analysts in different regions (e.g. APAC, EMEA) then add their own figures,
basing their calculations on the official numbers.
The numbers from HQ usually circulate as a static spreadsheet—-a seemingly
innocuous practice that leads to all sorts of trouble. Suppose HQ needs to
correct the original numbers and sends out an updated spreadsheet. APAC gets
the message, but it slips by one of the analysts on the EMEA team. Now APAC is
using the new numbers, EMEA the old, or even worse, a mix of old and new. It
doesn’t take long for the entire budgeting process to descend into chaos.
This is a good example of how static data creates organizational silos. Each
region has its own copy of the “truth,” and these copies decay for all sorts of
reasons: cut-and-paste errors, overwritten formulas, mixed-up files. The more
copies of the data circulate, the more error creeps in.
Smashing your silos means giving every financial analyst on your team access to
a single source of truth that’s updated in real-time.
M A R K E T I N G
O P S
S A L E SS I N G L E S O U RC E O F T RU T H
F I N A N C E
12
Publica(on:
Information wants to be free, and reports are no exception. The Publication
phase of the data-insight lifecycle is where the data is consumed by the
organization at large.
The challenge is that reports may live in multiple places: email, Box, Dropbox,
Excel, Slack and Sharepoint, in all sorts of tools: BI tools, presentation tools,
PDF viewers.Even more challenging, the reports are used by multiple groups
We’ve just seen how static data can create silos within an FP&A team. Now
apply that same principle to an entire organization, where Ops, Marketing,
Engineering, or the C-Suite need to accessing the same report.
Keeping a team on the same page is important. Keeping the whole company on
the same page is critical. Once again, that means connecting reports to a single
source of truth that’s updated in real-time.
There’s one more way to break down silos in the Publication phase: giving
analysts the ability to ‘write back’ data in the same tool they use to analyze it.
The data-insight lifecycle doesn’t always end with the generation of a report.
Often, the analyst needs to reenter the results of the report—the budget, for
example—into back-end systems like Oracle Financials
VISUALIZEPUBLISHSTORESHARE
.Normally that’s a manual data-entry job, with all
the time and risk of error involved. It requires yet
another tool, creating one last technological silo.
Financial analysts, like everyone, have their
favorite tools. By a wide margin, their favorite
tool is the spreadsheet. Wherever else the data is
published, it will eventually end up in an
application like Google Sheets or Excel.
Everything needed to complete the data-insight
lifecycle is in the spreadsheet. Why add another
step?
Rather than risk Instead, close the loop by
allowing analysts to update data sources within
the spreadsheet they’re already using.
13
Silobreakers
From this survey of the data-insight lifecycle, we can derive a number of best
practices. These are your silobreakers. Some are technological, others
organizational. Together, they represent the necessary steps to transform your
company into a truly data-driven business.
Give analysts control over their data stream, from the data source to the
pivot table
Choose data-access solutions designed for human end-users, not
applications
Automate everything: eliminate any process that depends on IT tickets
Make every phase of the reporting process self-service
Establish a single source of truth for reports that updates in real time as
the underlying data changes
Sync data directly to the analyst’s spreadsheet, and write it back from the
spreadsheet to the data source
VISUALIZE
SHARE
SHARE
RENEWAL OPPORTUNITY
LOW
HIGH
LOW
HIGH
MEDIUM
MEDIUM
HIGH
LOW
20
80
25
70
60
50
90
10
HEALTH
45,000
28,000
15,000
25,000
30,000
20,000
20,000
14,000
USAGE
$80,000
$65,000
$110,510
$75,000
$90,000
$10,000
$25,000
$50,000
APR
Verizon
Pepsi
GE
Pfizer
Delta
Boeing
AMEX
Apple
CUSTOMER
7
6
8
9
5
4
3
2
1
A B C D E F
=Kloud("SF_saleforce_customer_revenue”)
Customer Renewal Report
Frequency: Daily
Timezone: Los Aangeles PST
8 00 AM
Run Report
STEP 3: Autorefresh
STEP 2: Filters
US Region
Customer Renewal Report
STEP 1: Select a Report
Kloudio Run a Report
=Kloud(“Salesforce Report”,“customer name”,A2)
14
The elusive end-to-end solu/on
With all the trouble and expense involved with reporting, you would expect
vendors to devise solutions that incorporate these silobreaking principles. And
they have. Hundreds of them.
The trouble is, each of them addresses parts of the data-insight lifecycle, but
not the whole thing. They offer connectors and ETL tools for IT. Visualization
apps with front ends for executives and analysts, and back ends for data
engineers. A no-code tool for data queries. Another for visualization. They solve
problems within silos, or they break some silos and leave others intact.
A few products claim to offer end-to-end solutions for data access. But they
force analysts onto new tools rather than letting them work the way they liked,
in the tools they prefer. As we’ve seen, that just creates new silos even as it
breaks down old ones.
It raises the question: what would an effective end-to-end data access solution
look like? From what we’ve seen:
It would bring data together in a consistent automated fashion, using the
tools analysts preferred
It would provide secure access to data connections, allowing analysts to
add or change data sources without filing IT tickets
It would make designing reports and defining calculations drag-and drop
simple, so no one has to write SQL code
It would automatically sync data sources with the analysts’ spreadsheet
and recalculate equations, on demand or on a predefined schedule
It would enable analysts to ‘write back’ data to any source from their
favorite spreadsheet
It would automatically enforce compliance with regulations and standards
such as HIPAA, PCI and DSS.
ANALYSTENGINEER
15
That’s the vision behind the new data-access solution Kloudio: give analysts the
data they need in the tools they like to use, without getting IT involved.
It’s not easy to pull off, which is why point solutions dominate the data access
market. But the benefits are unmistakable.
One Kloudio customer, Netflix, reported saving 10,000 hours of data
engineering time and 6,000 hours of analyst time by giving analysts self-serve
access to their data. Massive manual exports (2M rows+) that used to crash
devices now happened automatically in minutes. Time spent prepping data for
financial reports dropped from 7 days to zero.
“We used to waste 50% of our time getting data out of our systems, and the
first week of every month bogged down in FP&A data prep,” said Ranjit Patel,
Finance Products, Netflix. “With Kloudio we focus more on insights, less on
data.”
What can Kloudio do for you?
RENEWAL OPPORTUNITY
LOW
HIGH
LOW
HIGH
MEDIUM
MEDIUM
HIGH
LOW
20
80
25
70
60
50
90
10
HEALTH
45,000
28,000
15,000
25,000
30,000
20,000
20,000
14,000
USAGE
$80,000
$65,000
$110,510
$75,000
$90,000
$10,000
$25,000
$50,000
ARR
Verizon
Pepsi
GE
Pfizer
Delta
Boeing
AMEX
Apple
CUSTOMER
7
6
8
9
5
4
3
2
1
A B C D E F
=Kloud("SF_saleforce_customer_revenue”)
Customer Renewal Report
Frequency: Daily
Timezone: Los Angeles PST
8 00 AM
Run Report
STEP 3: Autorefresh
STEP 2: Filters
US Region
Customer Renewal Report
STEP 1: Select a Report
Kloudio Run a Report
=Kloud(“Salesforce Report”,“customer name”,A2)
kloud.io