1

MOBILE DATABASES AND SOALucas Jellema (AMIS, The Netherlands)

Workshop “Big Data and Its Impact” – 16 & 17 March 2013for the Military College of Telecommunication Engineering

2

THE PRESENTER:LUCAS JELLEMA• Lives in The Netherlands

(close to Amsterdam)• 1994-2002 Oracle Consulting Global Center

of Excellence for Internet Development• Joined AMIS in 2002 – now working as CTO,

Consultant (Architect & Technical Lead) & Trainer

• Oracle ACE (2005) & ACE Director (2006)• Author of ‘Oracle SOA Suite 11g Handbook’

(Oracle Press, 2010)• Presenter at Oracle OpenWorld, JavaOne,

OTN Yathra & more User Group Conferences• Frequent blogger at

http://technology.amis.nl• Active with SQL & PL/SQL, Java EE & ADF,

SOA, BPM & more Fusion Middleware

3

OVERVIEW

• Overview of mobile devices• Data is presented, collected and edited in

devices• Handle ‘challenged server data connection’• Introduce the Mobile Database • Mobile Database challenges

– Big Data, Fast Data, Fresh Data, Distributed Database System, Security, Notifications

• Overview of core SOA concepts and objectives

• How SOA helps to enable the Mobile Database

4

MY CAR – IS A DATA COLLECTOR

• Collecting metrics– Some for big data analysis for all cars (of this

type)– Some for car analysis of my car when serviced– Some for real time (traffic jam, local

temperatures)

5

FORMULA ONE TELE METRICS

6

CAR NAVIGATION SYSTEM

7

LIVE TRAFFIC INFORMATION

8

LIVE? TRAFFIC INFORMATION

9

ZOOMING TRAFFIC INFORMATION

10

LIVE ON-ROAD TRAFFIC INFORMATION SYSTEM == MOBILE DEVICE

11

ON SITE CRASH INSPECTION

12

PLETHORA OF MOBILE DEVICES & USAGES

13

MOBILE AGENTS

Sensors

Cars

Navigation Systems

Cameras

Tablets

Mobile Phones

Security Badge Scanners

Barcode Scanners &Handheld Devices

Push Buttons

Satellites

Report Read & Edit

Read & Refresh (Alert)

Microphones & Speakers

Security Gates

ATM

Audio GuideTraffic Information

14

DATA AND MOBILE DEVICES

• Some data requirements depend on the actual location of the device

• Some data has to be on the device even when no data connection is available

• Some data to be presented has to be fetched in (near) real time from a central service

• Some (big) data is collected for analysis at a much later point in time

• Some data is unstructured • Some data needs to be sent from the device

to a central service in real time• Most data will need to be refreshed at some

point in time – although the refresh rate differs

• Sometimes the user needs to be (very) aware when data is not recently refreshed

15

DATA USED ON MOBILE DEVICES

• Originates from/to be sent to a back end (“the server”)

• Through limited bandwidth• Sometimes entirely disconnected

– No reception– Not permitted to use device– Server side unavailable

16

NEED FOR ON-DEVICE-DATABASE

• Provide quick response for user• Work when off-line• Handle bursts of fast-data and piles of big

data• Take load off back end servers

17

REQUIREMENTS FOR MOBILE DATABASE• Able to handle right types of data – including

unstructured data• Easily accessible to apps

– Support for SQL • Right scalability, right performance• Right security

– Encrypted, Recoverable, Remote Wipe• Maintainable/upgradable

– Database software and database model– Pre-populated data (“firmware”)

• Able to fetch and process data from back-end– Compact, secure, smart (incremental data

updates)• Smart upload to server, also Fast and Big Data

– pre-aggregated, pre-filtered, compacted

18

TRANSPARENT TO USER AND APPS

• The local database should be ‘invisible’ to parties: they interact with data service to query or report data– However: it should be clear when data cannot

guaranteed to be fresh or complete

19

SECURITY

• Confidentiality – no unauthorized access– Encryption of data in storage and transport– Authorization on server access

• Integrity – no tampering, constraint compliant

• Availability – no data loss, right-time-access and freshness

20

HOW DOES DATA ENTER THE MOBILE DATABASE?• Initial load when app is first installed• Fetched from back end• Entered by mobile app(s)

• When– During install or upgrade of app– During use of the application (by the user or

device features such as GPS, Bar Code scanner, Camera or Recorder)

– When cache freshness is in doubt– When cache completeness

is in doubt– Synch upon re-connect:

submit collected new and changed data

21

SOME GENERIC CHARACTERISTICS

• Pre loading• Lazy loading (background)• Write behind (lazy posting)• Data Freshness and Completeness

management• Local fallback (next best thing)

22

DATA FRESHNESS AND COMPLETENESS MANAGEMENT• Based on actual time and geo-location…

– And possibly user, running app(s) and other context

• … assess if locally available data is fresh & complete

• When data is no longer fresh– Try to negotiate a smart refresh with the server– Depending on type of data: hide stale data,

show stale data with warning, continue to show stale data

• Incomplete data should be complemented– Different location/area, zoom level, …

23

PUSH NOTIFICATIONS

• The server side can process messages, updates and events from many directions

• Sometimes a mobile device should be informed of this new information

• The server needs to know about the relevance of information to a device (subscription)

• The server needs to be able topush information (or poll-basedsemi-push)

• The mobile database & app has to process the pushed information

push

24

MOBILE DATABASE(S) & BACK END SYSTEMS IS A …

25

DISTRIBUTED DATABASE SYSTEM

26

THE 12 OBJECTIVES FOR DISTRIBUTED DATABASE SYSTEMS

27

MOBILE DATABASES DO NOT SUPPORT DISTRIBUTED TRANSACTIONS

• Server side updates happen independent from clients– Refresh of Mobile Database may take place at

any point in the future• Data manipulation in Mobile Database is

committed locally on the device– Upload to server happens later: ‘write (far)

behind’• This means that at some point after the

transaction is complete, the transactionis ‘replicated’ to other databases

• During this replication,conflicts may occur

28

DATA REPLICATION – STAGE ONE

• Updates on the client are made through local apps an stored in the mobile database

• Local validations apply – based on the information available in the mobile device at the time of the transaction – Without consulting the back end

29

DATA REPLICATION (‘WRITE BEHIND’) CONFLICTS – STAGE TWO• Updates on client violate server side integrity

rules– Invalid data is not tolerated by the back end– The mobile client has previously accepted the

transaction – How is the app/user engaged to correct/resolve?

30

DATA REPLICATION (‘WRITE BEHIND’) CONFLICTS• Updates on client collide with changes in the

server– The client attempts to update a record that has

also been updated in some other way– How are conflicts detected?– How should conflicts be resolved?– How should parties be notified

about ‘lost conflicts’?

31

SOA = BAD

32

SOA =BusinessAgility through

Decoupling

33

DECOUPLING≈MANAGING DEPENDENCIES

minimize impact of change while maximizing reusability

34

COUPLING

A B

35

B

DECOUPLING

A B

36

B

DECOUPLING

A BC

37

B

ADDITIONAL DECOUPLING THROUGH INTERMEDIARY

A BCESB

38

B

COMPOSITE SERVICES

AD

E

ESB

39

B

TRANSFORMATION/ADAPTION OF PROTOCOL AND FORMAT

A

ESB

JSONHTTPSSL

SQLJDBC over SQL*Net

No encryptionXML according to canonical model

D

E

EJB over RMI

B

XML/SOAPHTTP

WS-Security

40

B

ORCHESTRATION, POTENTIALLY LONG RUNNING, STATEFUL, ASYNCHRONOUS

AD

E

Process(BPM/BPEL)

41

B

IMPLEMENTATION OF SERVICES

A ESB

Q

PL/SQL

Mail Server

42

B

EXTREME DECOUPLING USING EVENTS

A

D

Event Handler E

F

subscription

B

43Se

rvic

e Bu

s

SOA AND MOBILE DATABASE

44

SOA AND MOBILE DATABASE

• Services that hide underlying complexity– Technology, formats, protocols, data structure of

back end systems• Services that handle two-way data

synchronization– Possibly conflicting changes on both ends– Violation of server side data constraints– Smart refresh of data on mobile device

• ESB that supports subscriptions and push to send alerts/notifications/updates to mobile devices

• ESB that exposes services in appropriate protocol and format (e.g. http/json)

• ESB that handles security in transport and message contents

45Se

rvic

e Bu

s

SOA AND MOBILE DATABASE

46

SOA AND MOBILE DATABASE

• ESB ensuring the right availability – possibly exceeding the service window of back-end systems

• ESB dealing with potentially large number of devices– Scalability, Throttling (DoS protection)

• (Simulated) Device-to-Device-Push• Upgrade of meta-data, apps and data-

structure• Bulk upload of Big Data from mobile device• Initialization/Update of

pre-populated data on device

47

SUMMARY

• Mobile Database is a necessity to ensure apps can run– With acceptable performance– At all (in off-line situations)

• Mobile Database is a local cache that contains a snapshot of the ‘back end data’ at a certain timestamp– Cache is not complete and is not current– Cache contains data collected and edited on device

• Mobile Database is part of distributed database system– Without real-time distributed transactions– Periodic synchronization between Mobile Database and

Server is required (to really complete the transaction)– Data conflicts need to be resolved

• SOA can help to hide complexity of back end and offer available services with fitting protocol and data format– Providing security, handling peak loads, supporting push,

understanding delta-based updates, …

48

RESOURCES

• Blog: technology.amis.nl– On Oracle, SQL, Java, SOA, BPM & more

• Email: lucas.jellema@amis.nl

• : lucasjellema

• Slideshare: www.slideshare.net/lucasjellema

• : www.amis.nl, info@amis.nl +31 306016000 Edisonbaan 15, Nieuwegein

The Netherlands