einsatz von big data technologien im geo-marketing · 21/9/2017 · •umfassende data management...
TRANSCRIPT
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Einsatz von Big Data Technologienim Geo-MarketingDDS Data Days 2017
Hans ViehmannPorduct Manager EMEA
Heidelberg, September 21, 2017
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Marketingim Wandel
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 3
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 4
DISCOVERWEB
DIGITAL ADS
WORD OF MOUTH
SOCIAL
TV
NEED
RESEARCH
RETAIL STORE
WEB SHOP
REVIEWS
SEARCHKW/ADS
RECEIVEOFFER
CONTACT CENTER
WEB SHOP
TRACK ORDER
SHOP AND BUY
RECEIVE PACKAGE
MISSINGITEM
WEB SHOP
CONTACT CENTER
SUPPORT PORTAL
SETUPPHONE
POSTREVIEW
JOINGROUPS
NETWORKISSUE
PHONEDAMAGE
TERMINATESERVICE
BILLINGISSUE
SOCIAL
RETAIL STORE
SOCIAL
MAKEPAYMENT
CONTACT CENTER
SUPPORT PORTAL
CHANGEADDRESS
CONTACT CENTER
RECEIVEOFFER
RESTARTSERVICE
CONTACT CENTER
RETAIL STORE
WORD OF MOUTH
SOCIAL
REFERFRIENDS
BRANDEDCOMMUNITY
Mehr InteraktionMehr Daten
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Technologische LösungsansätzeDrei Möglichkeiten
1. Einsatz „intelligenterer“ Software
2. Umstellung auf leistungsfähigere Hardware
3. Vollständig neue Architektur
5
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Technologische Lösungsansätze (2017)Drei Möglichkeiten
1. Einsatz „intelligenterer“ Software
2. Umstellung auf leistungsfähigere Hardware in der Cloud
3. Vollständig neue Architektur
6
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Einsatz „intelligenterer“ SoftwareNicht jedes Problem erfordert eine neue Architektur
7
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Anwendungsbeispiel Geomarketing
• Vorbereitung von Massen-Mailing im Handel
• Zuordnung von ca. 1 Mio. Adressdatensätzen zur nächstgelegenen Filiale
• Bestimmung nach Fahrstrecke, nicht Luftlinie
8
Filialstandort
Wohnort Kunde
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Anwendungsbeispiel GeomarketingMögliche Lösungsansätze
• Berechnung der Fahrstrecke jedes Kunden zu jeder Filiale
– Aufwändige Berechnung, erfordert auch bei guter Parallelisierung viel Zeit
• Aufbau von Voronoi-Diagrammen zur Gruppierung von Kunden und Filialen
– Vorverarbeitung zeitintensiv
• Geschickte Ausnutzung von Netzwerk-Datenmodell in der Datenbank– Bestimmung von Straßensegment aus Geocoding
– Berechnung aller Straßensegmente im Einzugsgebiet der jeweiligen Filiale
– Verknüpfung von Kunde und Filiale in einer einzigen SQL-Abfrage
9
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Anwendungsbeispiel GeomarketingVorgehensweise
• Straßennetz in der Datenbank
– In diesem Falle HERE Daten für die USA im Oracle Data Format (ODF)
• Verwendung des selben Straßennetzes für Geocoding und Routing
– Damit ID des Straßensegments und ID der zugehörigen Kante im Router übereinstimmt
• Ausnutzung von Parallelisierung beim Geocoding– Bulk Geocoding mittels Parallel Pipelined Table Function
• Umgebung als VirtualBox Image für Workshops verfügbar
– samt detaillierter Anleitung
10
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Umstellung auf leistungsfähigere Hardware in der Cloud
11
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Massiver Wandel bei CX Plattformen
• Verbindung von eigenen Systemen mit Cloud-basierten Services
– Neue Projekte fast ausschließlich in der Cloud
• Zusammenführung eigener Daten mit Daten von Partnern und Datendienstleistern
• Umfassende Data Management Plattformen verfügbar
–Wesentlicher Bestandteil des Oracle CX Ökosystems
– Zusammenführung der Kundenhistorie mit Verhaltensdaten
– Pool von ca. 5 Mrd. anonymen Profilen, verknüpft über ID Graph
• Cloud-basiertes Data Warehouse oder Big Data Platform naheliegend
– „Elastische“ Kapazität, reduzierte Kosten, kürzere time-to-market
12
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Vollständig andere ArchitekturAusnutzung von Big Data Technologien
13
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 14
Addressing Big Data Challenges
VOLUME VELOCITY VARIETY VALUE
SOCIAL
BLOG
SMARTMETER
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
4th Generation Data Architecture for Big Data
WarehouseData FactoryData Lake
Events and Streaming
Data Platform
Data Discovery Lab
AnalyticTools
Execution
Innovation
ActionableInformation
ActionableInsights
ActionableEvents
ActionableDiscoveries
Enterprise Data
Other Data Sources
Data Streams
BusinessData
Social/LogData
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 16
Using Hadoop to Address Big Data Challenges
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Benefit of Big Data Technologies
• Low cost and high horizontal scalability infrastructure
• Allowing storage of more data, more details over longer time periods
• Cost-effective way to analyse huge amounts of data
• Dealing with variable data by means of „schema-on-read“ capability
• Complementary to existing data warehouse technologies
17
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Database technologies
– Designed for „high-density“ data
– Transactionality and data validation
– High availability
– Comprehensive security and access control
– Existing tooling and workflows
–Operational experience
• Big Data Technologies
– Designed for „sparse“ data
– Low cost and high horizontal scalability
– Allowing storage of more data, more details, over longer time periods
– Cost-effective way to analyze huge amounts of data
– Rapid ingestion, not necessarily consistent
– „Schema-on-read“ capability for variable data
18
Differentiation against conventional databases
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Working with geospatial data on Big Data platforms
• Requires support for geometric functions, topological operations
– eg. determining the centroid
– Needs to deal with projections, complex operations such as buffering, ...
• More complex processing usually requires spatial indexing
– eg. spatial joins
• Spatial data usually comes in specific formats (Shapefiles, GeoJSON, ...)
• Needs to cope with location information which is only included implicitly
– requires geo-enrichment
• Visualization is very valuable for inspection of source data and results
19
Location Intelligence has specific requirements
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Categorization and filtering based on location and proximity
Preparation, validation and cleansing of Spatial and Raster data
Visualizing and displaying results on a map
Spatial querying and analysis of Hadoop data with SQL
What problems can Big Data Spatial analysis address?
Data Harmonization using any location attribute (address, postal code, lat/long, placename, etc).
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Data enrichment service API using GeoNames and geometry hierarchy data
SPARK and MapReduceroutines for distance calculations, PointInPolygon, buffer creation, Categorization, KMeansClustering, Binning
Spatial processing using of data stored in HDFS or NoSQL. Raster processing operations: Mosaic and sub-set operations. Geodetic and Cartesian data
HTML5 Map Visualization API
Hive SQL APIQuery from Oracle DB with Big Data SQL & Oracle SQL Connectors for Hadoop
Big Data Spatial Features
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Example: Development of new insurance products
• Analyzing customer behaviour in Data Discovery Lab
• Using data from in-vehicle telematics box
• Exploratory stage
– Determining data quality
– Interpreting location data
–Merging data with customer profile date from other internal sources
– Developing possible factors for premiums
• Next step: converting to production workflow
– Automated data processing and derivation of actionable results
22
Pay-as-you-drive model at Italian insurance company
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Related topic: Graph analysis
• Graphs are probably familiar from network models
– Reachability
– Shortest Path (eg. Dijkstra algorithm)
• Concept is increasingly popular and widely applicable–Modeling entities as vertices
–Modeling relationships as edges
• Graph algorithms enable topological analysis
– Detecting behavioural patterns
– Creating knowledge graphs, potentially in conjunction with Machine Learning
23
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Examples for Graph Analytics
• Community detection and influencer analysis
– Churn risk analysis/targeted marketing, HR Turnover analysis
• Product recommendation
– Collaborative filtering, clustering
• Anomaly detection
– Social Network Analysis (spam detection), fraud detection in healthcare
• Path analysis and reachability
– Outage analysis in utilities networks, vulnerability analysis in IP networks, „Panama Papers“
• Pattern matching
– Tax fraud detection, data extraction
24
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Trend: Machine Learning replacing Big Data
• KDnuggets studyin April 2017
– Google trends analysis
26
Source: Geospatialmedia.net
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Technologische Lösungsansätze (2017)Drei Möglichkeiten
1. Einsatz „intelligenterer“ Software
2. Umstellung auf leistungsfähigere Hardware in der Cloud
3. Vollständig neue Architektur
27
Copyright © 2017 Oracle and/or its affiliates. All rights reserved. |
Wirklich eine vollständig neue Architektur?SO
UR
CES
DATA RESERVOIR DATA WAREHOUSE
Oracle Database
Oracle IndustryModels
Oracle Advanced Analytics
Oracle Spatial & Graph
Big Data Appliance
Apache Flume
OracleGoldenGate
Oracle Event Processing
Cloudera Hadoop
Oracle Big Data SQL
Oracle NoSQL
Oracle R Distribution
Oracle Big Data Spatial and Graph
Oracle Database
In-Memory, Multi-tenant
Oracle Industry Models
Oracle Advanced Analytics
Oracle Spatial & Graph
Exadata
OracleGoldenGate
Oracle EventProcessing
Oracle DataIntegrator
Oracle Big DataConnectors
Oracle DataIntegrator
28
Copyright © 2016 Oracle and/or its affiliates. All rights reserved. |
Concluding remarks
29
Copyright © 2017 Oracle and/or its affiliates. All rights reserved. |
The Forrester Wave™: Geospatial Analytics Tools And Platforms, Q3 2016
“While hardcore GIS professionals may start their work in other applications, when they want to solve spatial problems in production and with web- and IoT- scale data, Oracle gives themthe platform to do so.”
Confidential – Oracle Internal/Restricted/Highly Restricted 30
Analysts: Rowan Curran with Holger Kisker, Ph.D. and Emily Miller September 1, 2016
The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical representation of Forrester's call on a market and is plotted using a detailed spreadsheet with exposed scores, weightings, and comments. Forrester does not endorse any vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Interested in project experience, best practices, networking?Spatial and Graph Summit
• IOUG Business Intelligence, Warehousing and Analytics SIG have established annual BIWA Summit
– Planned for March 20-22, 2018 at OracleHQ, Silicon Valley, California
• Spatial and Graph Summit is separate track
– Lots of interesting material from previous years available on OTN
• Opportunity for interaction withSpatial and Graph product team
31
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
More resources
• Further information on oracle.com
– www.oracle.com/goto/spatial
• Blogs
– https://blogs.oracle.com/oraclespatial
• Developer forums on OTN
– https://community.oracle.com/community/database/oracle-database-options/spatial
• LinkedIn community
– „Oracle Spatial and Graph“ group
• Google+ community
– „Oracle Spatial and Graph SIG“
• Follow us on Twitter – @SpatialHannes, @agodfrin, @JeanIhm
32
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Q&A
33
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 34
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Cloud Computing enabling Big Data Analytics
• Scaleable resources readily available
• No setup cost
• No hardware investment
• Pay-as-you-go model
• e.g. Oracle Big Data Cloud Service
– Hadoop Platform, SPARK technology
– Data Integration
– R Distribution
–Oracle Big Data Spatial and Graph
35
Database
Backup
Big Data
General Purpose
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 36
Cloud Service – Management Console