the 25 predictions about the future of big data

The 25 Predictions About The Future Of Big Data – A New Currency?

Robert J. Abate, CBIP, CDMP © 2016 – All Rights Reserved Page: 1 of 10

The 23 25 Predictions About The Future Of Big Data

Prelude – A New Currency? In the past, I published on the value of information, big data, advanced analytics and the Abate Information Triangle and have recently been asked to give my humble opinion on the future of Big Data.

I have been fortunate to have been on three panels recently which discussed this very question with such industry thought leaders as: Bill Franks (CTO,Teradata), Louis DiModugno (CDAO, AXA US), Zhongcai Zhang, (CAO, NY Community Bank), Dewey Murdick, (CAO, Department Of Homeland Security), Dr. Pamela Bonifay Peele (CAO, UPMC Insurance Services), Dr. Len Usvyat (VP Integrated Care Analytics, FMCNA), Jeffrey Bohn (Chief Science Officer, State Street), Kenneth Viciana (Business Analytics Leader, Equifax) and others.

Each brought their unique perspective to the challenges of Big Data and their insights into their “premonitions” as to the future of the field. I would like to surmise their thoughts adding in color to the discussion while foretelling that Data is being monetized.

Recent Article By Bernard Marr If you haven’t had the opportunity, I believe that a recent article published by Bernard Marr (Forbes) entitled: “17 Predictions About the Future of Big Data Everyone Should Read” was a great start [http://www.forbes.com/sites/bernardmarr/2016/03/15/17-predictions-

about-the-future-of-big-data-everyone-should-read/#3f47b5bf157c]. Many of the industry thought leaders that I mentioned above had hit on these points. He had listed the following:

1. Data volumes will continue to grow . There’s absolutely no question that we will continue generating larger and larger volumes of data, especially considering that the number of handheld devices and Internet-connected devices is expected to grow exponentially.

2. Ways to analyse data will improve . While SQL is still the standard, Spark is emerging as a complementary tool for analysis and will continue to grow, according to Ovum.

3. More tools for analysis (without the analyst) will emerge . Microsoft MSFT -1.57% and Salesforce both recently announced features to let non-coders create apps to view business data.

4. Prescriptive analytics will be built in to business analytics software . IDC predicts that half of all business analytics software will include the intelligence where it’s needed by 2020.

5. In addition, real-time streaming insights into data will be the hallmarks of data winners going forward, according to Forrester. Users will want to be able to use data to make decisions in real time with programs like Kafka and Spark.

6. Machine learning is a top strategic trend for 2016 , according to Gartner. And Ovum predicts that machine learning will be a necessary element for data preparation and predictive analysis in businesses moving forward.

7. Big data will face huge challenges around privacy, especially with the new privacy regulation by the European Union. Companies will be forced to address the ‘elephant in the room’ around their privacy controls and procedures. Gartner predicts that by 2018, 50% of business ethics violations will be related to data.

8. More companies will appoint a chief data officer . Forrester believes the CDO will see a rise in prominence — in the short term. But certain types of businesses and even generational differences will see less need for them in the future.



9. “Autonomous agents and things” will continue to be a huge trend , according to Gartner, including robots, autonomous vehicles, virtual personal assistants, and smart advisers.

10. Big data staffing shortages will expand from analysts and scientists to include architects and experts in data management according to IDC.

11. But the big data talent crunch may ease as companies employ new tactics. The International Institute for Analytics predicts that companies will use recruiting and internal training to get their personnel problems solved.

12. The data-as-a-service business model is on the hori zon . Forrester suggests that after IBM IBM -1.11%’s acquisition of The Weather Channel, more businesses will attempt to monetize their data.

13. Algorithm markets will also emerge . Forrester surmises that businesses will quickly learn that they can purchase algorithms rather than program them and add their own data. Existing services like Algorithmia, Data Xu, and Kaggle can be expected to grow and multiply.

14. Cognitive technology will be the new buzzword . For many businesses, the link between cognitive computing and analytics will become synonymous in much the same way that businesses now see similarities between analytics and big data.

15. “All companies are data businesses now ,” according to Forrester. More companies will attempt to drive value and revenue from their data.

16. Businesses using data will see $430 billion in prod uctivity benefits over their competition not using data by 2020, according to International Institute for Analytics.

17. “Fast data” and “actionable data” will replace big data , according to some experts. The argument is that big isn’t necessarily better when it comes to data, and that businesses don’t use a fraction of the data they have access too. Instead, the idea suggests companies should focus on asking the right questions and making use of the data they have — big or otherwise

What Was Missing… I agree with all of Bernard’s listing but I believe that he missed some predictions that the industry has called out. I would like to add the following:

18. Data Governance and Stewardship around Master Data and Reference Data is rapidly becoming the key area where focus is required as data volumes and in turn insights grow.

19. Data Visualization is the key to understanding the overwhelming V’s of Big Data (IBM data scientists break big data into four dimensions: volume, variety, velocity and veracity) and in turn the advanced analytics and is an area where much progress is being made with new toolsets.

20. Data Fabrics will become the key delivery mechanism to the enterprise by providing a “single source of the truth” with regard to the right data source. Today the enterprise is full of “spreadmarts” where people get their “trusted information” and this will have to change.

21. More than one human sensory input source (multiple screens, 3D, sound, etc.) is required to truly capture the information that is b eing conveyed by big data today . The human mind has so many ways to compare information sources that it requires more feeds today in order to find correlations and find clusters of knowledge. We have to provide a "single source of the truth" and eliminate the pervasive sharing of information from untrusted sources.

22. Empowerment of business partners is the key to getting information into the hands of decision makers and self-service cleansed and governed data sources and visualization toolsets (such as provided by Tableau, ClickView, etc.) will become the norm of delivery. We have to provide a "single source of the truth" and eliminate the pervasive sharing of information from untrusted sources.

23. Considering Moore's Law (our computing power is increasing rapidly) and the technologies to look thought vast quantities of data is improving with each passing year, our analytical capabilities and in turn insights are starting to grow exponentially and will



soon change organizations to become more data drive n and less "business instinct" driven .

24. Data is going to become the next global currency (l ate addition) and is already being globally monetized by corporations.

25. Data toolsets will become more widely used by corpo rations to both discover, profile and govern data assets within the confines of a data fabric or marketplace. Toolsets will include the management of metadata and automatic classification of assets and liabilities (i.e.: Global IDs, etc.).

The Four V’s Of Big Data IBM recently presented a slide that discussed the myriad of challenges when facing Big Data – it is mostly self explanatory and hits many of the points that were mentioned in Bernard’s article:

What this infographic exemplifies is that there is a barrage of data coming at businesses today and this has changed the information landscape for good. No longer are enterprises (or even small businesses for that matter) living with mostly internal data, the shift has happened where data is now primarily coming from external sources and at a pace that would make any organizations head spin.



Today's Best Practice “Data Insights Process” Today, external data sources (SFDC, POS, Market-share, Consumer demographics, psychographics, Census data, CDC, Bureau of labor, etc.) provide much more than half of the information into the enterprise with the norm to create value in weeks. How is this done you may ask? Let’s call this the Data Insights process. The best practice today has turned upside down the development of business intelligence solutions, this process is:

� Identify a number of disparate data sources of interest to start the investigation

� Connect them together (data integration using common keys)

� Cleanse the data (as Data Governance has not been applied) creating your own master and reference data



� Learn about what the data is saying and visualize it (what insight or trend has been uncovered

� Create a model that gives you answers

� Formalize data source (cleanse and publish) to the myriad of enterprise data consumers with governance (if applicable)

� Use the answers to change your business

� Repeat (adding new sources, creating new models, etc.)

This process utilizes data experts to find data sources of value (1 to 2 weeks), Quickly connect together and scan to determine suitability and eliminating information which is incomplete or lacking value/connection to other sources (integrating and cleansing takes about 2 weeks), Visualize what value these sources provide using data visualization toolsets - find interesting value statements or features of the data to pursue {like store clustering and customer segmentation} (1 to 2 weeks), Develop a model or advanced analytic to see what your value statement found using a Data Scientist (2 weeks), and Then present to business to determine next steps. This whole process happens in about 6-8 weeks and usually creates the "interest" in the business to invest in developing into a data warehouse or BI solutions.

Yes, the new process is completely reusable – as what is learned can be turned into a data source (governed data store or warehouse which is part of a data fabric) for future usage in BI and in turn for self-service; but what is important is that we now go from data to insights in weeks rather than months, and it forms the foundation for our business requirements – yes, I said that.

The long term investment of a BI solution (often six months or more) is proven rapidly and then the formal process of capturing the business requirements and rules (transformations in ETL language can be taken from rapid prototyping tools like Alteryx) has a head start and typically has the added advantage of cutting down the BI process into 3-4 months.



Recent Advances In Data Engineering We can thank recent technological advancements for the changes in delivery of information with the advent of a number of toolsets providing self-service to tech-savvy business partners.

But there is still a need for managing the information using architectural principals and this process is not going away. I will elaborate further in the paragraph below.



The Need For Enterprise Information Management The myriad of data sources is changing the way we as business intelligence and analytics experts behave and likewise it has created a demand for data management and governance (with Master data and in turn Reference data) – so this element was added to the predictions. It's a very important piece of the puzzle and should not be overlooked or downplayed. It was even added to my latest information triangle which appears below.

The role of enterprise data management in IT has been evolving from “A Single Source of Truth” into becoming “The Information Assurance Flexible Delivery Mechanism”. Back in March of 2008 I published at the DAMA International Symposium the needs for a flexible information delivery environment including:

� Metadata management for compliance enforcement, audit support, analysis, and reporting

� Master data integration and control

� Near-real time business information

� Source data management for controlling data quality at the transaction level

� Effective governance for a successful managed data environment

� Integration of analytics, reporting, and transaction control

� Control of business processes and information usage

A flexible structure is just as important today as business needs are changing at an accelerating pace and it allows IT to be responsive in meeting new business requirements, hence the need for an information architecture for ingestion, storage, and consumption of data sources.



The Need For Knowing Where Your Data Is Coming From (And Going To) One of the challenges facing enterprises today is that they have an ERP (like SAP, Oracle, etc.), internal data sources, external data sources and what ends up happening is that “spread-marts” (commonly referred to as Excel Spreadsheets) start proliferating data. Different resources download data from differing (and sometimes the same) sources creating dissimilar answers to the same question. This proliferation of data within the enterprise utilizes precious storage that is already overflowing - causing duplication and wasted resources without standardized or common business rules.

Not to mention that these end up being passed around as inputs to other’s work – without knowledge of the data lineage. This is where many organizations are today - many disparate data sets with little to no knowledge of if this is a "trusted" data source.



Enterprise Data Fabric (or Data Marketplace) An enterprise data fabric or marketplace (I've used both terms) is one location that everyone in the enterprise can go to get their data – providing quality, semantic consistency and security. This can be accomplishing with data lakes, data virtualization or a number of integration technologies (like API’s, services, etc.). The point is to give a common point of access to the enterprise for data that has been cleansed and is ready for use with master data. Here are a couple of reasons why you should consider this approach:

� Business mandate to obtain more value out of the data (get answers)

� Need to adapt and become agile to information and industry-wide changes

� Variety of sources, amount and granularity of data that customers want to integrate is growing exponentially

� Need to shrink the latency between the business event and the data availability for analysis and decision-making

Image from EMC Corporation – Dr. David Reiner Autho r



Summation – Data Is The New Global Currency In summation, consider that increasingly information is produced outside the enterprise, combined with information across a set of partners, and consumed by ever more participants so data is the new currency of the information age and we all pass around currency – so let’s get cracking at delivering this to our enterprise (or it will go elsewhere to find it).

To the point, Big Data is an old acronym and the new one is “Smart Data” if you ask me.

I would welcome any comments or input into the above - let's start a dialog around best-practices in today's information age...

Robert J. Abate, CBIP, CDMP

[email protected]

http://www.linkedin.com/in/robertjabate

the 25 predictions about the future of big data

Documents