building the next-gen digital meter platform for fluvius...fluvius’ data portfolio drives...
TRANSCRIPT
Building the next-gen digital meter platform for FluviusMaarten HerthogeTeam Lead Data Science & Engineering, delaware BeLux
64% 29% 17%got measurable results is successful reaches production
It’s easy to understand why so many projects fail
Deep Learning
ETL vs ELT
PaaS vs IaaS
Data Visualisation
Data Quality
Master Data
Big Data
Databricks
SMP vs MPP
Data Catalog
Spark
Storm
Data Mart
No-SQL
Cloud vs On-prem
Velocity, Variety and VolumeSemantic Layer
Predictive
Prescriptive
IoTStreaming
AI
Start with the problem,
not with the technology
The core data platform needs to support various use cases
Portal access to 3 years of history
Interfaces to allow scalable sharing of data
Digital meters collect data… a lot of data
7+ million IoT devices after full roll-out 7M+
LEGAL
12TB Sending up to 12 TB of data every day
GDPR and strict government regulations
The “golden hammer” does not exist,
use a portfolio approach
Think about the types of use cases, processing flexibility and enablement you require
Data consumers
LOB
CRM
ERP
#1 Increasing volume
Think about the types of use cases, processing flexibility and enablement you require
Data consumers
LOB
CRM
ERP
#2 Need to collect and combine any data
Think about the types of use cases, processing flexibility and enablement you require
Data consumers
LOB
CRM
ERP
#3 From data consumer to data explorer
Think about the types of use cases, processing flexibility and enablement you require
Data consumers
LOB
CRM
ERP
#4 Real-time becomes the standard
Think about the types of use cases, processing flexibility and enablement you require
Data consumers
LOB
CRM
ERP
#1 Increasing volume
#3 From data consumer to data explorer
#2 Need to collect and combine any data
#4 Real-time becomes the standard
Prioritize software-as-a-service to create a data portfolio
Managing all data
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PROCESS SERVE
Analyzing all data
Integrating all data
From GB’s
to TB’s
Seamless
scaling
1000+ API
requests/sec7M+ devices
Avoid the data lake pitfall
Data lakes often presented as the go-to solution in these cases
Massive scale
Optimized performance
Integration flexibility
Cost effectiveness
But data lake implementations fail in 60% of the cases
Massive scale
Optimized performance
Integration flexibility
Cost effectiveness
Becomes data swamp
Capabilities mismatch
Schema-on-read
Misused as “strategy”
Focus on collaboration
Azure Databricks
Aim for one unified analytics processing engine
Cloud storage
Data warehouses
Big data storage
IoT / streaming dataML models
BI tools
Data exports
Data warehouses
Collaborative Workspace
DATA
ENGINEER
DATA
SCIENTISTBUSINESS
ANALYST
Deploy and manage production jobs & workflows
ETL / ELT JOB SCHEDULER MONITORING DEVOPS
Single Runtime Foundation
DATA LAKE
MANAGEMENT
SERVERLESS
/ SaaSSCALE WITHOUT
LIMITS
PAY-AND-SCALE-
AS-YOU-GO
Narrow the scope and decouple storage from serve
INGEST STORE PROCESS SERVE
Why is this successful?
Unified, consistent data ingest path
Cost efficient for full detail
Why is this successful?
Allows for workload-specific platforms
tailored at specific cases independent
from the raw datastore
SQL DB Service Fabric
Data Lake Gen2& Delta
Databricks acts as glue between a data lake and various workload specific application platforms
INGEST STORE PROCESS SERVE
Databricks
Why is this successful?
Unified runtime and collaboration
One-click setup
Native integration with Azure services
Enterprise security & SLA
SQL DB Service Fabric
Data Lake Gen2& Delta
Fluvius’ data portfolio drives innovationon the Flemish energy market
Bringing consumption insights and awareness to millions of households
Lower barrier for data- and market-driven change
Portfolio projects TCO 20x lower compared to “golden hammer”
Focus on collaboration
Avoid the data lake pitfall
The “golden hammer” does not exist,
use a portfolio approach
Start with the problem,
not with the technology
Interested in more?Visit delaware.ai
Focus on collaboration
Avoid the data lake pitfall
The “golden hammer” does not exist,
use a portfolio approach
Start with the problem,
not with the technology
Thank you!Maarten HerthogeTL DS&E, delaware BeLux
Feedback
Your feedback is important to us.
Don’t forget to rateand review the sessions.