t22.fujitsu world tour india 2016-business intelligence and data analytics in banking, financial,...
TRANSCRIPT
0 Copyright 2016 FUJITSU
Fujitsu for Business Intelligence and Data Analytics Pankaj Sharma – Solution Architect – HPC & Analytics
Fujitsu World Tour 2016
1 Copyright 2016 FUJITSU
Why Big Data?
2 Copyright 2016 FUJITSU
Big Data
Big is a relative term – depends upon the context
When we have to deploy a cluster to do store and analyze
When we take analysis of the data to the realms of High Performance Computing
Structured and unstructured Data
3 Copyright 2016 FUJITSU
Big Data continued ….
We cannot escape BigData – Either analyze in parallel or perish
Abundance of unstructured Data – difficult to use any DB
Getting the data to processor bottleneck
Parallel and Cluster computing on data
4 Copyright 2016 FUJITSU
Where? Fujitsu’s Foot steps
5 Copyright 2016 FUJITSU
Use Cases.....Examples
Customer Intimacy Operational Efficiency
Risk Management Innovation
Categories
Market Basket Analysis
Predictive Maintenance
Enterprise Data Hub
IoT Applications
Monitoring and Analysis of unused data
360° View
Targeted Advertisement
Sentiment Analysis
Recommendations
Cybersecurity
Fraud Detection
Compliance Auditing
6 Copyright 2016 FUJITSU
Cybersecurity Target Industries: Large Public Networks, Government, Financial, Telecom, Utilities, etc.
Real Time
Detect
Attacks in
Real Time
Prevent
Threats in
Real Time
7 Copyright 2016 FUJITSU
Fraud Detection Target Industries: Financial, Insurance, Government, etc.
Patterns
Words
External
Information
Internal
Information
8 Copyright 2016 FUJITSU
Targeted Advertising Target Industries: Retail, Manufacturing, Service, Financial, etc.
Personalized Offers to Consumers
Data sources: • Social networks • CRM • Sales Orders • POS • Web Logs • Online profiles • etc.
Social Media
Data
Multi-
Structured
Data
9 Copyright 2016 FUJITSU
Trace and interprets online discussions in social media to Gather customer feelings on: Your organization
Your Brand
Your Products and Services
Your Competition
Other events that may impact corporate performance: politics, disasters, large-scale events, etc.
Sentiment Analysis
or ?
Decide /
Improve
Target Industries: Retail, Manufacturing, Service, Financial, etc.
Like
Dislike
Want
Need
10 Copyright 2016 FUJITSU
Real Time
Recommendations
Widely in the retail sector
Looks at customers’ purchase behavior of products, and offer a "best next offer" recommendation when they purchase a product
Not only checking which combinations are most likely, but also, identifying a closely related peer consumer group
Target Industries: Retail, Service, Social Networks, etc.
Web logs, Cart Data, User Data
Recommendation Profiles Web Logs
11 Copyright 2016 FUJITSU
Market Basket Analysis
Match products purchased together for cross-selling, up-selling and promotions
Big data adds more context: Time of day
Music played in a store
Store visit duration
Store traffic
Weather, etc.
Target Industries: Retail
Multi-
Structured
Data
Social Media
Operational
Data
Other Data
12 Copyright 2016 FUJITSU
Predictive Maintenance Target Industries: Manufacturing, Natural Resources, Utilities, Telecommunication
Real Time
Improve Operations, Reduce Waste,
Identify Bottlenecks
Control Manufactured Product Quality
Monitor Asset Performance
Avoid Operations Downtime
Predict Asset Failure
Machine / Sensor Data
Predictive Models/Machine Learning
13 Copyright 2016 FUJITSU
Usage Monitoring Target Industries: Manufacturing , Natural Resources, Transport, Utilities
Monetize Data
Monitor/Control
Optimize
Machine / Sensor Data
14 Copyright 2016 FUJITSU
DW Optimization
Offload and accelerate ELT/ETL workloads
Improve Storage TCO, store more data, and tackle data growth challenges
Scale-out to meet performance and/or capacity with low cost commodity hardware
Insight Enrichment
Perform analytics on all your data assets and discover new insights
Enable Enterprise Data Hub and eliminate information silos
Keep all your data assets virtually “forever”
Offer business users self-service analytics and reduce time-to-information, and development costs
Use Case: Enterprise Data Hub BI/Data Warehouse Augmentation and Optimization
Target Industries: Generic
15 Copyright 2016 FUJITSU
Fujitsu Big Data Technologies
16 Copyright 2016 FUJITSU
Apache Hadoop Ecosystem
Fujitsu Infra Unified, Elastic, Resilient, Secure
Workload Management (YARN)
Batch
Processing (MR, Pig, Hive)
Analytic
SQL (Hive, Impala)
Search
Engine Solr
Machine
Learning (Mahout, Spark)
Stream
Processing (Spark)
3rd Party
Apps (Datameer,
Tableau,
elasticsearch,…)
DA
TA
MA
NA
GE
ME
NT
(H
adoop, F
alc
on, O
ozie
…)
SY
ST
EM
MA
NA
GE
ME
NT
(Z
oo
ke
ep
er, A
mb
ari, …
)
File System Online NoSQL
Da
ta A
cce
ss
Secu
red
Pla
tform
Integration on open source stack (Flume, Storm, Sqoop, Falcon, WebHDFS, [Pentaho, Informatica, Other Connectors])
17 Copyright 2016 FUJITSU
Cluster with Hadoop
• Holds the metadata for the HDFS NameNode
• Performs housekeeping functions for the NameNode
Secondary NameNode
• Stores the actual HDFS data blocks and runs analytic compute DataNode
• Manages MapReduce jobs JobTracker YARN
• Monitors individual Map and Reduce tasks TaskTracker
18 Copyright 2016 FUJITSU
Multi-Structured Data Storage and Batch Processing
Structured Data
•Databases •Transaction •CRM •ERP •…
Unstructured Data
•Machine Data •Web Data •Social Media •Documents •E-Mails •Audio/Visual •…
Multi-Function
•Self-Service Analytics •Advanced Analytics •DW Landing + ETL Stage •Dynamic Archival •…
19 Copyright 2016 FUJITSU
Real-Time/In-Memory Data Analytics
Real-Time Data Analysis Non Real-Time Data Lake
and Batch Processing
Network Logs
Web Logs
Machine Data
Market Data
Hadoop
Real-Time Data Collection
20 Copyright 2016 FUJITSU
Fujitsu Big Data Professional Services
21 Copyright 2016 FUJITSU
Why Fujitsu?
The world‘s fourth-largest IT services provider and No.1 in Japan*
Committed to deliver local service globally
We do everything in ICT, and work with you to deliver end-to-end solutions
Global best-practice-based delivery of Business Transformation and Information Technology Projects
A one-stop shop for Big Data where organizations can get optimized Big Data infrastructure solutions including hardware, software and services – all from a single source
PC/Mobile
Services
Networks Software
Servers
22 Copyright 2016 FUJITSU
Big Data Professional Services
• FUJITSU Big Data Assessment Workshop • FUJITSU Big Data Strategy Consulting Service • FUJITSU Services for Hadoop • FUJITSU Big Data Analytics Services
Consulting Services
• FUJITSU Integrated System PRIMEFLEX for Hadoop • Other Platforms on request
Integration Services
23 Copyright 2016 FUJITSU
Fujitsu Big Data Strategy Consulting Service
Offers the expertise and skills required to build your Big Data Strategy Identify and document your Big Data requirements
Identify the most promising best practices, use cases, and technologies
Assess and align Big Data to organizational assets and strategic objectives
Identify gaps
Document strategic recommendations for Big Data initiatives and their implementation plan
Service Deliverables Big Data Strategy Plan for the organization
Recommendation on the best-of-breed solution and reference architecture
Strategy implementation road map
Service description Best Practice-based service following a 5-step approach
The Benefit
Helps you identify, implement, and validate use cases and their outcomes
Remove Big Data technical and analytical skill gaps
Provide Big Data analytics development services for data transformation, analysis, and reporting
Duration: Scope Dependent
1- Requirement Definition and Project Planning
2- Organizational Assets assessment
3- Gap Analysis
4- Reference Architecture definition
5- Strategy and Road Map Documentation
24 Copyright 2016 FUJITSU
Fujitsu PRIMEFLEX for Hadoop Deployment Service
Introduction to Fujitsu Integrated System PRIMEFLEX for Hadoop
Overview of the hardware and software stack
Definition and configuration of the required infrastructure resources: Servers
Network
OS
Definition of Hadoop Cluster Configuration Configuration of nodes
Configuration of networking parameters
Definition of Enterprise Integration of Hadoop Cloudera Manager cluster configuration
Base services and security
Define Testing
Definition of Datameer configuration Datameer application configuration
Base Security and Tuning
Enterprise Integration
The service delivers a formal configuration report, subsequent implementation of the configuration in the IT infrastructure environment, and concludes with a handover of the running solution to the Customer
25 Copyright 2016 FUJITSU