microservices architecture with oracle database · and ultra-reliable distributed database •state...

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|

TRN5515Industrial-strengthMicroservice ArchitecturewithNext-GenerationOracleDatabase

WeiHu,VicePresidentofDevelopmentAnilMadan,DirectorofDevelopmentDominicGiles, MasterProductManager

OracleDatabaseDevelopment

October22,2018


SafeHarborStatement

Thefollowingisintended tooutline ourgeneralproductdirection. Itisintended forinformationpurposes only,andmaynotbe incorporatedintoanycontract.Itisnotacommitment todeliver anymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions. Thedevelopment, release, timing, andpricingofanyfeaturesorfunctionality described forOracle’sproductsmaychangeandremainsatthesolediscretion ofOracleCorporation.

2


ProgramAgenda

Microservice Architecture– Promise&Challenges

DataManagementforMicroservices

CustomerCaseStudies

TransactionalMessagingSystemusingAdvancedQueuing

WrapUp

1

2

3

4

5

3


ThoughtExperiment:TrueorFalse?

1. RunningyourapplicationinDockercontainers2. OrchestratingyourapplicationusingKubernetes3. UsingKafkaforeventprocessing4. UsingNoSQL5. Alloftheabove?6. Noneoftheabove?

Microservices Require…

4

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.| 5

Container 2

OracleRAC DBInstance - 1 OracleRAC DBInstance - 2

Private Interconnect

Container 1

Container 3

SCANIP


Microservices• Organizeapplicationassetofwell-defined,loosely-coupledservices

– Similar insomeformtoSOAdevelopment

• Makesdevelopmentandtheresultantapplicationmoremodular– Improvestheparallelization ofdevelopment foranapplication– Makestheapplication easier tounderstand– Allowsteamstodeploytheirservices independently ofotherteamswork– Abletoleveragethebestofbreedsolutions todevelop/update theapplication– Workswellinacontinuousdeliverymodel

• Basically isthebestthingever… Untilthenextbestthing

6


Microservice Architecture• ArchitecturePattern,not justputtingcodeintoDocker

containers• Eachmicroservice canruninacontainer

– Privatedatabase anddatamodelforeachmicroservice

• Microservices communicateusingasynchronousmessagingviasomeeventqueuingsystem– Decoupled formaximum resiliency andscalability– Services do nottalktoeachother directly – only viaevent

queuing service– Insulated fromslowdown orfailure ofothermicroservices

7

ShippingService


Microservice Challenges• Eachservice getsitsowndatabasethatneedstobemaintained, patched,upgraded,

madeHA,secured…• Hardtoshareandquerydataacrossservices

– Analyticsrequiresthefederationofdatafromvariousservices– Evenwithinthesameapplication!

• Needfortransactional eventqueuing/messaging system– Microservices arelooselycoupledandcommunicateviaEvents– Reliableeventqueuingsystem(at-most-once,at-least-once,exactly-once,…)

• Transactions thatspanmultiple microservices– Microservice AcallsBthatcallsC.WorkflowonlycommitswhenCcompletes– HowtoundoA&Bshould Cfail– ApproachofusingSAGAsrequirescreationofcompensatingtransactions

8


ProgramAgenda



CustomerCaseStudies


WrapUp

1

2

3

4

5

9


AnACIDcompliant enterprise classdatabase

Really nicetohave’sforMicroservices DataArchitecture

Supportfordifferentdatamodels i.e.Relational, Document,Graph,KeyValue

10

Atransactionally consistent messaging servicetosupportMicroservices events

Materialized viewstosupportcachingofdatafromotherservices tosupportcrossserviceanalytics


MultitenantDatabaseforMicroservices

• Microservices canusePDBs– Eachmicroservice allocates itsowndataset

(canbeaPDB,Schema,orasubset)– Eachmicroservice hasprivatedatamodel

• Multitenant database containersdeliver– Managemanydatabases asone– Secureseparation ofdata– Easysharingandqueryingofdataacross

PDBs• Databaseviewscanalsoisolatewhateach

microservice sees

11

Reducethecostandcomplexityofdatamanagementfornewapps


PDBSharding forMicroservices

• Wantcentralizeddatabase(CDB)withultra-highavailability andscalability – Exadata isgreatforthis

• 19calsosupportsPDBSharding– EachPDBcanbesharded individuallyacrossmultipleCDBs

• Providesfault isolation andgeo-distribution formicroservices– LossofanentireCDBmakesonlypartofaPDBunavailable

• Alsoallowseachmicroservice toscale itsPDBindividually– Moreefficientuseofresourcescomparedtoscalinga

monolithicapplication(CDB).

12

Scalability,faultisolationandgeo-distribution Shard-1 Shard-1Shard-1

Shard-2 Shard-2

Shard-3

Recomm

ProductCatalog

ProductCatalog

ProductCatalog

CheckOut

CheckOut


ImportanceofMessaging• Theterm‘microservice’mayimplythatyoushouldlookattheservicesfirst• Infact,thebestwaytodesignistothinkofthemessagesfirst• Amicroservices-basedarchitectureisdescribedbytheinteractionof

messages.Thisprovidestheabstractionthatallowseachmicroservice tobedevelopedandevolvedindependently– Providedthemessages remainthesame,youcanreplaceaservicebyoneormore

otherservices transparently.Thisgivesyouresiliency andscalability

• Themessagingsystemalsosimplifiesthearchitecture.Insteadoffiguringoutwhichmicroservice talkstowhichothermicroservice,theyallusemessagingtopublish/subscribetomessages/events

13


OracleAdvancedQueuing

• Microservices send/receiveeventsusingJavaMessageService(JMS)overAdvancedQueuing(AQ)– JMS+AQprovidereliableandrichasync messagingwithinOracleDB– RESTisnotasgoodbecauseoftightcouplingbetween

sender/receiver

• Providesalltheresiliencyandscalabilityofmessagingandeventprocessing,AND

• Unifiedtransactionacrossmessagesandtableswithouttwo-phase commit

• PowerfulSQLqueriesovermessages– Important fordebugging enterprise-scale microservices

• Highperformance:12cSharded Queuesrearchitected for10xhigherthroughput

14

TransactionalEventQueuingSystemforMicroservices


ProgramAgenda



CustomerCaseStudies


WrapUp

1

2

3

4

5

15


• StateGridisbiggestelectricalutilityintheworld– serves 1.1billionpeople– #2inFortuneGlobal500(2nd largestcompanygloballybyrevenue)

• StateGridisadoptingMicroservice architecture forthegrowingbusiness– Microservice architectureprovidesresiliencythroughloosecouplingandflexibilitytomeet

growingandchangingbusinessneeds

• Microservice architecture inturnrequiresahighlyscalable,loosely-coupled,andultra-reliable distributeddatabase

• StateGridwanted toprovethattheirbusiness-critical orderingservice systemcanleverage OracleSharding tomeetthese requirements– Orderingservicecurrentlyhas200-300millionusersdistributedacross26subsidiaries.Goalisto

buildcentralizedorderingservice

Challenges

16


FederationService

(SAML2)

IDCSInfrastructureMicroServices

Web SecurityService

(OAuth2)

Identity Service(SCIM)

CacheJobScheduler

CloudIdentityPlatformSuite(IDCS)

IDCSPlatformMicroServices

IDCSOperationalData

Messaging SSO

AuthenticationService

(OpenID Connect)

Storage Reports Node.js Keys/Creds

CustomerData

Scalability

KeyService

NotificationService(ATOM)

Compute Storage Notifications

OracleCloudIaaS

AnalyticsService

CloudIdentityAppsSuite

CloudSSOService

B2BExchangeService

PrivilegedAccountsService

AuthenticationService

IDCSDataServices

Monitoring Logging

ServiceManager

GovernanceService

Availability

CloudPortalIntegration IaaS Integration PaaS Integration SaaS Integration

queuing

Globality

CaseStudy:OracleIdentityCloudService(IDCS)Largestproductionmicroservices-basedapplicationinOracle

17


WhyIdentityCloudServiceusesAQ1. Statelessmicroservices:Keeping stateofmessagesincentralizedlocation

(DB)helpsmicroservices tobetrulystateless2. Nomessageloss:Dequeued messagesareretaineduntilexplicitcommit.

Microservices cansendeventsasynchronouslywithoutworryingaboutlostactions

3. LocalordistributedTransaction: Commit/rollbackhelpstohandledataandmessageinsametransactionlocallyoracrossmultiplemicroservices.Hardwithnon-DBbasedpub/subsystem.

4. Handles largemessagebacklogs: Databasemessaginghelpssmoothoutspikyloadsandtoleratesservicesthataretemporarilydown

18


ProgramAgenda



CustomerCaseStudies


WrapUp

1

2

3

4

5

19



IntroductiontoDatabaseAdvancedQueuing(AQ)

HowAQhelpsbuildresilientMicroservices

HowtwocriticalOracleCloudServicesuseAdvancedQueue

What’snewinAdvancedQueuein18cand19c?

a

b

c

d

20

4


AdvancedQueuingOverview

Transformation & Rules Engine

Publish Advanced Queues

Subscribe Priority > 2

Subscribe corrid= ‘ORDER_ID’

ORACLE

Advanced Queues

Propagation

PYTHONJMSJDBCPL/SQL

Queue Tables

ORACLE

SOAP WebServices/XML

IBM MQ MessageGateway

OCI

TIBCO

Ø ProvidesareliableandrobustmessaginginfrastructurewithinOracleDBØ StoresmessagesinabstractstorageunitscalledqueuesØ TwoQueues– ClassicQueuesandSharded Queues(12cOnwards)

Application 2

Application 3

Application 1


HowAQhelpsbuildresilientMicroservices

ØAllowsreliablecommunicationwithoutcouplingØAllowsstatelessmicroservicesØTransactedsessionguaranteesnolossofmessageandexactlyonce

deliveryØSupportslocalanddistributedtransactionsØSupportsachangenotificationmechanismonDMLsØLocalatomictransactionsinvolvingqueueoperationsandDML

22


HowAQhelpsbuildMicroservices Contd..ØProvidestransparencyØWorksforanydataØWorksforPDBsandnon-CDBs

ØSupportschemaversioningusingEditionBasedRedefinitionØHandleslargemessagebacklogsØProvidesQueuelevelaccessprivilegesØHighavailabilityanddisasterrecoverysupport(LogicalStandbyandOracle

GoldenGate)

23


AQPL/SQLCode

24

/* Update the status of the order in the order status table: */ CREATE OR REPLACE PROCEDURE update_status( new_status IN VARCHAR2, order_msg IN BOLADM.ORDER_TYP) BEGIN

/* Query old status from the table: */ SELECT st.status INTO old_status FROM order_status_table st

WHERE st.customer_order.orderno = order_msg.orderno;

IF new_status = 'SHIPPED_ORDER' THEN IF old_status = 'BILLED_ORDER' THEN

return; /* message about a previous state */ END IF;

END IF;

/* Update the order status */ UPDATE order_status_table st

SET st.customer_order = order_msg, st.status = new_status;

/* Enqueue the message into Shipping Queue */dbms_aq.enqueue(‘SHIPPING_QUEUE’, enqopt, msgprop,order_msg, enq_msgid); COMMIT;

………

END; /


IdentityCloudServicebasedonAQ

IDCSService

AQDB

Messaging-Service

Audit Handler

WorkerGroup#1

WorkerGroup#2

Audit Handler

Audit Handler

Audit Handler

Notify Handler

Notify Handler

Notify Handler

Notify Handler

JMSListener

JMSListener

JMSListener

JMSListener

JMSListener

JMSListener

JMSListener

JMSListener

EventManager

JMSProducer

JMS PropertiesTopic : AdminAudit : trueNotify : false

JMS BodyJSON{

“attrName”:“attrVal”,…}

Q2

Q1

Q2_A

Publish Subscribe

JMSMessage

IDCSusageofAQ:--PersistAuditpublished byvariousMicroservices-Managenotifications(Email,SMSatal.)-Provisionapplications basedonuser/groupactivity

Q1_A– Subscription for Q1,Audit handlerQ1_N – Subscription forQ1,Notification handlerTotal connections =8 (2Queues x2Worker groups x2Handlers)


ADWC&PDBCSasynchronousjobexecutionframeworkbasedonAQ

CentralCDB

Worker Pool forJob Type1 Worker Pool forJob TypeN

Worker2

Worker1

WorkerK

POD1

PODN

Worker1

Worker2

WorkerK

RoundRobinrouter

Subscriber

BrokerJob Table

JobQueue1

JobQueue2

JobQueueN

Subscriber

RoundRobinrouter

RESTFULRequests

PODAPI CDBCloudScripts

PODAPICDBCloud

Scripts

.

.

.

.

.

.

.....


• Sharded Queuessupportwasintroducedin12.1.0.2–Asinglelogicalqueuewithmany“shards”–“Sharding” obtainshigherconcurrencyandthroughputviahorizontal

partitioning.–Automaticmanagementofsessionaffinitytoshards–Automaticmanagementoftablepartitionstoavoidcontention–Automaticmanagementofpartitioninstanceaffinity– Integratedwiththedatabasetooptimizeperformance–12.1.0.2onlysupportsJMSinterface

Sharded Queues


Sharded QueuesArchitectureforScalabilityandPerformance

Instance1

PartShardENQ DEQ

PartShardENQ DEQ

PartShardENQ DEQ

Instance2

Instance3

SinglelogicalqueuemodelPhysicalQueue:partitionsmappedtoshardswithinstanceaffinityonRAC

ENQ

ENQ

ENQ

DEQ

DEQ

DEQ


• AutomaticManagementofShardsandPartitions• OracleRealApplicationClusterOptimizations• MessageCache• NativeJMSSupportfrom12.1onwards• Enqueue /Dequeue Optimizations• RulesEngine• SubscriberManagement• Event-basedListener

Sharded QueuesKeyfunctionalities


• PL/SQL supportforSharded Queues– PL/SQLEnqueue, Dequeue andNotification– JMS,ADTandRAWmessage payloads– Propagationbetween sharded queues– Message delay– Exception queue

• MessagecacheadvisorforSharded Queues• Event-basedJMSlistenerforSharded Queues• LongIdentifiersforClassicandSharded Queues

Sharded Queues:Contd…Keyfunctionalities


Sharded Queues

PROCEDURECREATE_SHARDED_QUEUE (queue_name INVARCHAR2,storage_clause INVARCHAR2DEFAULTNULL,multiple_consumers INBOOLEANDEFAULTFALSE,max_retries INNUMBER DEFAULTNULL,comment INVARCHAR2DEFAULTNULL);

ExistingAQapplicationsdon’tneedtobechanged


• Higherthroughput• Lesssystemresourceconsumption• Largenumberofsubscribers• Event-basedlistenerwithfewerdatabaseconnections• Manyconcurrentenqueuers anddequeuers acrossmultipleRACinstances• BackwardsCompatibleforStandardJMSandAQPL/SQL based

applications– justrecreatetheAQinthedatabase

Sharded QueuesKeybenefits

1,024

2,048

4,096

8,192

16,384

32,768

65,536

131,072

1 2 4 8 16 32 64 128

dequ

eues

/sec

ond

dequeuer threads

Nonsharded - 1node

Sharded - 4nodes

JMS Queue withsame number of enq and deq threadscommitting after every10 operations


• AQ-OGGsupportforClassicQueue– exceptSharded Queues, PropagationandArrayinterface.

• LogicalStandbysupportforSharded Queues• PL/SQL andJMSArrayinterfaceforSharded Queues• PythoninterfaceforClassicQueues(RawQueues)• Keybasedmessaging(Comingin19c)• RESTAPIs(Admin,Monitoringcoming in19c)

AdvancedQueue:Contd…Keyfunctionalities


Sharded QueuesRestrictions Restrictions

No message retention JDBC Thin Driver only

No Transformation No ODP .NET support

No Enterprise Manager support No OGG procedural replication support

No transactional grouping No recipients support

No JMS-AQ extensions

Note:ClassicQueuesupportsallthesefeatures


ProgramAgenda



CustomerCaseStudies


WrapUp

1

2

3

4

5

35


• Multitenant Architecture allowsmicroservices tohaveseparatedatabases (PDBs)butstillmanagethemasone(CDB)

• OracleAdvancedQueuingsimplifies microservices designbyprovidingtransactionalmessaging forevents

• OracleAdvancedQueuingalsosimplifies SAGAdesign byallowingrowandmessagingcommits asasingle transaction

• OracleDatabaseprovidesatrueMultiModel enginesupporting Relational, OLAP,JSONDocument,XML,Graph,Spatial

• OracleMaterialized Views,multi-shardandcross-PDBqueries supportrealtime analysis

• Industryleading security,scalability, analytics, andhighavailability

36

OracleDatabaseasaplatformforMicroservices


OracleDatabasesimplifiesdeploymentofMicroservices

37

ANALYTICS RETURNS

ORDERSEVENT

EVENT

REST CALLCUSTOMERS WAREHOUSE

MICROSERVICESAPPLICATION

AQMSG AQMSG

ANALYTICSRETURNSORDERS CUSTOMERS WAREHOUSE

MICROSERVICESAPPLICATION

CROSSPDBQUERIES

RESTAPI

Sharded DatabaseforUltraHighAvailability


OtherMicroservices SessionsofInterest

TIP4071:Microservices, Containers, Databases, andPersistence ModelsKuassiMensah,Director,ProductManagement,OracleChristopherJones,SeniorProductManager,OraclePaulParkinson,ConsultingMemberofTechnicalStaff,Oracle10/23/20185:45pm-6:30pmMoscone West3010

TIP4175:DataManagement inaMicroservices WorldGeraldVenzl,SeniorPrincipalProductManager,OracleMonday10/225:45pm-6:30pmMoscone West3003

38

microservices architecture with oracle database · and ultra-reliable distributed database •state...

Documents