micro- services devops distributed systems
Post on 13-Feb-2017
221 Views
Preview:
TRANSCRIPT
MartinFowler’sDefinition
• Developasingleapplicationasasuiteofsmallservices• Eachservicerunsinitsownprocess• Services communicatewithlightweightmechanisms
• “Often anHTTPresourceAPI”• Butthathassomeproblems• Messagingisbetter.
• Theseservicesarebuiltaroundbusinesscapabilities• Independentlydeployablebyfullyautomateddeploymentmachinery.• Minimumofcentralizedmanagementoftheseservices,
• Maybewrittenindifferentprogramminglanguages• Mayusedifferentdatastoragetechnologies.
http://martinfowler.com/articles/microservices.html
MonolithicApplications
•aclient-sideuserinterface•adatabase•aserver-sideapplication.
Enterprise applicationsthree tieredarchitecture
•handleHTTPrequests,•executedomainlogic,•retrieveandupdatedatafromthedatabase,•selectandpopulateHTMLviewstobesenttothebrowser.
Server-side Application
•Asinglelogicalexecutable.•Changesrequirebuildinganddeployinganewversionoftheserver-sideapplication.
Thisserver-sideapplicationisamonolith
OtherQuestionableMonolithicPatterns
• Theserver-sideapplicationinteractswithasingledatabase• NoteverythingneedstofitintooneDBsystem• NotjustCAPchoices:usetheDBthatfitseachsubsytem’s problemandyourdevelopmentpatterns
• Splittingteamsintotiers• TheDBteam,theUXteam,theServerAppteam• Thedeploymentandoperationsteam
CAP:Consistency-Availability-PartitionTolerance
Browser
Web InterfaceServer
Application Server
ServerSDK
Client SDK
Karst:MOAB/Torque
ResourcePlugins
Stampede:SLURM
Comet:SLURM Jureca:SLURM
HTTPS
HTTPorTCP/IP
RecalltheGatewayOctopusDiagram
“Super”SchedulingandResourceManagement
Inmicro-servicearch,thesealsoneedscheduling
APIServer
ApplicationManager
MetadataServerApplication
ManagerApplicationManagerApplicationManagerApplicationManagerApplicationManager
MetadataServerMetadataServerMetadataServerMetadataServerMetadataServer
APIServerAPIServerAPIServer
APIServer
ApplicationManager
MetadataServer
DecouplingtheApp
Server,Take1
MySuggestedList(1/2)
• APIServer• Highavailability,load-balancedEntryPoint
• Scheduler• HighlyavailableleaderOrchestrator
• Resources,Applications• Highlyavailable?
StatelessMetadataServer
MySuggestedList(2/2)
• Experiment stateStatefulMetadataServer
• Submitsjobs• FireandforgetApplicationManager
• MonitorsjobsonremoteresourcesMonitorService
• Centralizeyourservice logsLogging
ThoughtExercise
• AssumeAmazonCloudandabigbudget• Howdothesecomponentsinteract?• HowcanIdistributecopiesofcomponentsacrossmultipleavailabilityzones?
• Whatkindsoffailurescanoccur?• HowcanIhandlefailuresofdifferentcomponents?
LastThoughts:AreMonolithicApplicationsBad?• Amonolithicserverisanaturalwaytoapproachbuildingmanysystems.• Monolithicapplicationscanandshouldstillbemodular• Allyourlogicforhandlingarequestrunsinasingleprocess
• Youcanusethebasicfeaturesofyourlanguage todivideuptheapplicationintoclasses,functions,andnamespaces.
• MonolithscanstilluseDevOps• Youcanrunandtesttheapplicationonadeveloper'slaptop,• Useadeploymentpipeline toensurethatchangesareproperlytestedanddeployedintoproduction.
• Youcanhorizontallyscalethemonolithbyrunningmanyinstancesbehindaload-balancer.
• DBCAPproblemsdon’thitmostapplications
MonolithicApplications:TraditionalSoftwareReleases• Softwarereleasesoccurindiscreteincrements• Runonclients’systems• Releasesmaybefrequentbuttheyarestilldistinct
• Firefox• OSsystemupgrades
• Traditionalreleasecycles• Extensive testing• Alpha,beta,release candidates,andfullreleases
• Extensiverecompilingandtestingrequiredaftercodechanges• Codechangesrequiretheentirereleasecycletoberepeated
MicroServices:SoftwareasaService
• Doesyoursoftwarerunasaservice?• Doyourunthisserviceyourself?Ordoesanotherpartofyourorganizationruntheservicethatyourteamdevelops?
• Traditionalreleasecyclesdon’tworkwell• Test-release-deploy takestoolong• Maymakereleasesmanytimesperday
• Youcanbealittlemoretolerantofbugsdiscoveredafterreleaseifyoucanfixquicklyorrollbackquickly.
• Getnewfeaturesandimprovementsintoproductionquickly.
Microservices:StartwithModularity
• Thinkaboutyourcurrentapplication.• Howcanyoubreakitintomodularcomponentparts?• Whataretheinterfaces?Whatarethemessagesthatyouneedtosendbetweentheparts?
• Yourcodeshouldbemodularanyway.• OGSIandSpringdothisforsingleJVMJavaapplications• Youcanmaintainamodulewithouttouchingothercode.
Shouldapersonorgroup“own”asubsetofmodulesormicroservices?
• Teammemberscomeandgo• Theyaren’talwaysavailableanyway• Work-lifebalancewilleventuallybecomeimportanttomost• Nooneshouldbeindispensable
Forsmallteams,NOinmyexperience.
• “Possessiveownership”culture• Opaquecode• “Notmyproblem”ifithappens inanothermodule
Moduleownershipleadstootherproblems
Ownership and Conway’s Law
• Anyorganizationthatdesignsasystem(definedbroadly)willproduceadesignwhosestructureisacopyoftheorganization'scommunication structure.–MelvinConway
InOtherWords
• If youseparateyourorganizationinto• Systemsspecialists,• DBadmins,• Softwaredevelopers,• Userexperiencespecialists, etc,
• Andaskthemtobuildsomething,• Then
• thesystemwilllookliketheorgchart.• ThisistheoppositeoftheDevOps approach• Createsthebureaucraticinfightingthatplagueorganizations.
Products,NotProjects
• Thealternative:organizearoundproducts• Aproductteamhasalltheexpertise itneeds• JimGrayinterviewsWernerVogels:https://queue.acm.org/detail.cfm?id=1142065
• “Youbuildit,youownit”
• Problemswiththisapproach:• Someonemayworkonmultipleproducts• Whoisthatperson’sboss?
RulesofThumbYouMightEncounter
• Amazon’stwo-pizzarule• Teamsshouldn’tbelargerthanwhattwopizzascanfeed
• “Agoodprogrammershouldbeabletorewriteamicroservice inacoupleofweeks”
• BoundedContextsfromDomainDrivenDesign
CanWeMakeThisMoreRigorous?
Messagingpatternsbetweencomponents
•REST•CommoninInternetapplicationsbutyouwouldneverdothiswithincodeinasingleJVM
Request-response
•CommoninsingleJVMapplication:Observer,listenerpatterns•Butnot socommonindecoupledapplications.•Moregenerally,asynchronousmessagingisalittlescary.
Push
•Thecomponentknowswhichcomponentneedstogetthenewinformation.One-to-one
•Componentsdon’tneedtoknowwhichothercomponentsneedstheinformation.Theyjustbroadcastit.Many-to-many
HowBigIsaComponent?
• Oneanswer:theoptimalmessagepatternshouldnotchangewhengoingfromsingleJVMtodistributedcomponents.
• Telescope inandtelescopeout
• Componentsthatruninthesameprocesscanbechattyandsendlotsofmessages.
• Messages canbepointerstoshareddataobjects• Messages arealwaysreceived• Thisistrueforbothrequestandpushmessaging.
Pan-GalacticComponents
• Componentsthatrunacrosstheuniverseneeddifferentmessages,eitherinfrequentorterse.
• Noshareddataobjects topointto.• Distributedcomponentsmustalsoassumeawidevarietyofmessagefailures
• Lostconnections• Recipients neverreceived themessage• Recipients receivedmessages inthewrongorder• Recipients crashedwhileprocessing• ...
• Microservices aredistributedsystems• ShouldknowanduseDSideas
Microservices AreNotDistributedObjects
• AcomponentisnotanobjectintheOOsense.• AcomponentshouldbeaserviceintheSOAsense• RecallRESTarchitecture:services shouldbeasidempotentaspossible.
• Ifyousendthesamemessagemorethanonce, theserverstateafterprocessingthemessageisthesame.
• Ex:Ifacomponentreceives10identicalmessagesayingsubmitajob,thecomponentshouldonlysubmitthejobonce.
• Howdoyouknowthemessageisreally identical?Timestamps,otheridentifiers,etc
• “Sharednothing”betweenservices• Messageprocessingshouldhavenosideeffectsthatneedtobeexposedtoothercomponents
• Anti-pattern:allthoseoutputfilesgeneratedbyscientificapplications
DecentralizedDataManagement
• TheRegistrycomponent,backedbyasingleDBsystem,holdsallthedata
Thisisanopenchallengefor
ApacheAiravata
• Onedatabasehasall theinformationfortheapplication• Limitsyourchoiceofdatastores:youhavetousethesameDBtechnologyforeverything.
• Microservices constantlyneedtomakeremoteDBcalls• Performanceandsecurity
MicroserviceAntipattern
SettingDataBoundaries
• Thisiseasywhenthedataisn’tconnectedorlightlyconnected.• ButnoteasytosplitupatraditionallydesignedDBschema.• Sciencegatewaydataboundaries
• Youmaywanttosplitstateless andstateful data
• Ifyousplitevenlightlyconnecteddataacrossservices,youwillneedtoconsiderconsistency
• Strong:requiresdistributedtransactions,whichcanbetricky• Weak:eventualconsistency.Canyoulivewiththis?
• Whatdataboundariesdosciencegatewayshave?
Howdoservicescommunicate?
• Ideally,alwaysdothesamething.• Yourdevelopmentenvironmentshouldbeaminiatureversionofyourdeploymentenvironment:containerization
• Encapsulateandabstract• ThistouchesDevOps CI/CD,sothoseareseparate lectures
• Keepthecommunicationpatternsthesame• Push,pull• 1-1,orN-N
• We’lllookatmessagingsystemsindetailnext• Pushmessaging,many-to-many,reliabledelivery,replay,etc
IfImakeeverythingdistributed,howcanItest?• Unittestswithincomponentsaresimple• Buttestingtheentiresystembecomesanoperationaltest
• Thisisnotabadthingifyoucanscriptyouroperationtests
• Andyoushould
• Monolith:testyourcodewhilebuildingwithMaven
• BringupsimpletestDBslikeSQLiteorDerby• Onebigbuilddoeseverything
Microservice Testing
• Microservice:testyourcodeinanoperation-likesetting• Don’tbuildtheentirecodeeverytime• Buildonlythemicroservice youareworkingon.• Soyourbuildshouldbemodular
• Somaybeyoudon’tneedonebigbuildsystemafterall• Andmaybeyoudon’tneedonegit repoforeverything...• Andmaybeyoudon’tneedoneversion#thatcoversallyourcode...
• Soreleasesbecomeincremental• Testing,monitoring,andloggingbecomemorecontinuous
• ThisisatouchpointbetweenMicroservices andDevOps
DesigningforFailure
• Microservices needtobebuilttotolerateerrors.• Whatkindsoferrorswillsciencegatewaymicroservices encounter?• Howdoyoudetectthesefailures?• Whattoolsdoyouneed?• Whatstrategiescanyouusetodosystemtesting?
Zab:AtomicBroadcasting
ClientsSeeSpaces
Zookeeper’sclientAPIresemblesastructuredversionoftheTupleSpacesconceptofparallelprogramming.
AMQP:NetworkProtocolandArchitecture,NotAnAPI• ManyImplementations
• RabbitMQ• ApacheActiveMQ• ApacheQpid• SwiftMQ• Andothers:• Thiscontradictsmyearlierdistinctionbetweenhorizontalandverticalstandardizationefforts
• AtleastforVersion0-9-1• EachimplementationcanhaveitsownAPI• I’llfocusonVersion0-9-1• ThisisnotanAMQPtutorial
AnAMQPServer(orBroker)
• Exchange• Message Queue• Exchanges caninteract withmultiple message queues
Twomainparts
• Accepts producermessages• SendstotheMessage QueueExchange
• Routesmessages todifferentconsumers depending onarbitrarycriteria• Buffersmessages when consumers arenotabletoaccept them fastenough.
MessageQueue
ProducersandConsumers
• ProducersonlyinteractwithExchanges• ConsumersinteractwithMessageQueues• Consumersaren’tpassive
• Cancreateanddestroymessagequeues
• Thesameapplicationcanactasbothapublisherandaconsumer• Youcanimplement Request-Response withAMQP• Exceptthepublisherdoesn’tblock
• Ex:yourapplicationmaywantanACKorNACKwhenitpublishes• Thisisareplyqueue
MessageQueue
• Typesofqueueproperties:• Privateorshared• Durableortemporary• Client-named orserver- named,etc.
• Store-and-forward queue:holdsmessagesanddistributesthesebetweenconsumersonaround-robinbasis.
• Durable andsharedbetweenmultipleconsumers.• Privatereplyqueue:holdsmessagesandforwardsthesetoasingleconsumer.
• Replyqueuesaretypicallytemporary,server-named, andprivatetooneconsumer.• Privatesubscriptionqueue:holdsmessagescollectedfromvarious"subscribed"sources,andforwardsthesetoasingleconsumer.
ConsumersandMessageQueues
• AMQPConsumerscancreatetheirownqueuesandbindthemtoExchanges
• Thishasaninterestingimplication• Queuescanhavemorethanoneattachedconsumer• AMQPqueuesareFIFO
• AMQPallowsonlyoneconsumerperqueue toreceivethemessage.• Useround-robindeliveryif>1attachedconsumer.• Thisgreatlysimplifiesthings
• Ifyouneed>1consumertoreceiveamessage,youcangiveeachconsumertheirownqueue.
• Usetopicmatchingtoroutemessages• ComparethistoZab andatomicbroadcastprotocols,generally.
• AMQPmakessomesimplifications
TheExchange
• Receivesmessages• Inspectsamessageheader,body,andproperties• Routesmessagestoappropriatemessagequeues• Routingusuallydonewithroutingkeys inthemessagepayload
• Forpoint-to-pointmessages,theroutingkeyisthenameofthemessagequeue
• Forpub-subrouting,theroutingkeyisthenameofthetopic• Topicscanbehierarchical
Publish-SubscribePatterns
• Usefulformany-to-manymessaging• Inmicroservice-basedsystems,severaldifferenttypesofcomponentsmaywanttoreceivethesamemessage
• Buttakedifferentactions• Ex:youcanalwaysaddaloggerservice
• Youcanalwaysdothiswithexplicitlynamedroutingkeys.• Youmayalsowanttousehierarchical(namespace)keynamesandpatternmatching.
• gateway.jobs.jobtype.gromacs• Gateway.jobs.jobtype.*
TheMessagePayload
• Readthespecificationformoredetails.• IngeneralAMQPfollowsthehead-bodyformat• Themessagebodypayloadisbinary• AMQPassumesthecontentishandledbyconsumers
• Itistheconsumer’sjobtodosomethingwiththecontent.• UnlikeJMS
• YoucouldserializeyourcontentwithJSONorThriftanddeserialize ittodirectlysendobjects.
DirectExchange
• Apublishersendsamessagetoanexchangewithaspecificroutingkey.
• Theexchangeroutesthistothemessagequeueboundtotheroutingkey.
• Oneormoreconsumersreceivemessagesiflisteningtothequeue.
• Default:round-robinqueuingtodelivertomultiplesubscribersofsamequeue
Queue.Declare queue=app.svc01 Basic.Consume queue=app.svc01 Basic.Publish routing-key=app.svc01
Fanout Exchange
• MessageQueuebindstoanExchangewithnoargument
• PublishersendsamessagetotheExchange
• TheExchangesendsthemessagetotheMessageQueue
• AllconsumerslisteningtoallMessageQueuesassociatedwithanExchangegetthemessage
TopicExchange
• MessageQueuesbindusingroutingpatternsinsteadofroutingkeys.
• APublishersendsamessagewitharoutingkey.
• ExchangewillroutetoallMessageQueuesthatmatchtheroutingkey’spattern
MoreExamples
• TheRabbitMQ tutorialpagehasseveralniceexamplesofclassicmessageexchangepatterns.
• https://www.rabbitmq.com/getstarted.html
SomeUsefulCapabilitiesofMessagingSystemsfor
MicroservicesOverarchingRequirement:Itshouldsupportyoursystem’s
distributedstatemachine
Let’sbrainstormsome
UsefulCapabilities:MyList(1/2)
• Supportsbothpushandpullmessaging• Delivermessagesinorder• Successfullydeliveredmessagesaredeliveredexactlyonce
• MultiplerecipientsOK• Delivermessagestooneormorelistenersbasedonpre-definedtopics.
• Storemessagespersistently• Therearenoactiverecipients.• Allrecipientsarebusy
• Determineifcriticalmessagesweredeliveredcorrectly
UsefulCapabilities:MyList(2/2)
• Redelivermessagesthatweren’tcorrectlyreceived• Corruptedmessages,norecipients,etc• Recipientcanchange
• Redelivermessagesonrequest• Helpsclientsresynchtheirstates
• Allowothercomponentstoinspectmessagedeliverymetadata.• Supportselasticity, faulttolerance
• Prioritymessaging?• QualitiesofService
• Security,faulttolerance
WhichMessagingSoftwaretoChoose?
• AMQPdoesnotcoverallthecapabilitieslistedabove.• Itcanbeextendedtocoverthese inmanycases
• AMQPmessagingsystemimplementationsarenotnecessarilycloud-ready
• Theyhavetobeconfiguredashighlyavailableservices.• Primary+failover
• Nofancyleaderelections, etc asused inZookeeper+Zab• Havescalinglimitations,althoughthesemaynotmatteratourscales.
• Othermessagingsystems(Kafka,HedWig)arealternatives
SimpleWorkQueue
• Queueupworktobedone.• Publisher:pushesarequestforworkintothequeue
• Queue shouldbeasimpleDirectExchange• MessageQueueshouldimplement“onlydelivermessageoncetoonceconsumer”.
• Round-robinscheduling.• RabbitMQ doesthisoutofthebox
• Consumer:SendsanACKaftercompletingthetask• IfaQueue-ClientclosesbeforeanACK,resendmessagetoanewconsumer.
• RabbitMQ detectsthesetypesoffailures.
GatewayAPIServer
AppMan Worker
AppMan Worker
AppMan Worker
Broker
MSG1, 2,3MSG2
MSG1
MSG3
ACK
ACK
ACK
• 3Job requests come totheAPIServer• APIServer publishes toBroker(Exchange+Message
Queue)• Brokersends toworkersinround-robin• Workers sendACKswhendone• IfBrokerdetectsaclosed connection beforeanACK, sends
MSGtoadifferentworker
SimpleWorkQueue
WhatCouldPossiblyGoWrong?
• Jobstakealongtimetofinish,soACKsmaynotcomeforhours.• DurableconnectionsneededbetweenConsumersandMessageQueues• Alternatively,theACKcouldcomefromadifferentprocess
• Jobsmayactuallygetlaunchedontheexternalsupercomputer,soyoudon’twanttolaunchtwicejustbecauseofamissingACK
• Clientshavetoimplementtheirownqueues• Couldgetanotherworkrequestwhiledoingwork.
WorkQueue,TakeTwo
• Orchestratorpushesworkintoaqueue.• Haveworkers requestworkwhentheyarenotbusy.
• RabbitMQ supportsthisas“prefetchCount”• Useround-robinbutdon’tsendworktobusyworkerswithoutstandingACKs.• Workers donotreceiveworkrequestswhentheyarebusy.
• WorkersendsACKaftersuccessfullysubmittingthejob.• Thisonlymeansthejobhasbeensubmitted• Worker cantakemorework
• Aseparateprocesshandlesthestatechangesonthesupercomputer• Publishes”queued”, “executing”, ”completed”or“failed”messages
• Whenjobisdone,Orchestratorcreatesa“cleanup”job• Anyworker availablecantakethis.
WorkQueue,Take2
Orchestrator Broker
AppMan Worker
EmailListener
1a.Ready
2.RunJob
3.ACK
AppMan Worker
5.Done
1b.Ready
8.CleanUp
9.ACK
0.RunJob
ExperimentMetadataManager
4.,6.,10.JobStatus
7.Cleanup
WhatCouldPossiblyGoWrong?
• AWorkermaynotbeabletosubmitthejob• Remotesupercomputer isunreachable, forexample• WeneedaNACK
• TheOrchestratorandExperimentMetadatacomponentsarealsoconsumers.
• ShouldsendACKstomakesuremessagesaredelivered.• OrchestratorandExperimentMetadataManagermayalsodieandgetreplaced.
• UnlikeAppMan workers,OrchestratorandEMMmayneed aleader-followerimplementation
• Brokercrashes• RabbitMQ providessomedurabilityforrestarting• Possibletolosecachedmessages thathaven’tgonetopersistentstorage
WhatElseCouldGoWrong?
• Lotsofthings.• Howdoyoudebugunexpectederrors?
• Logs
• AloggerlikeLogStash shouldbeoneofyourconsumers• Noone-to-onemessagesanymore.• Everythinghasatleast2subscribers
• Yourlogservice• Themaintarget
• OryoucoulduseFanout
Summary
• Microservices areapplicationsbuiltoutofdistributedcomponents• ImportantifyourunSoftwareasaService• ConnectiontoDevOps:
• Acceleratethedevelop-test-deploycycle• ContinuousIntegrationandContinuousDeliveryareessential
• Servicesneedtocommunicate• Asynchronousmessaging ispowerfulmechanism• Push,many-to-many• Caneasilyextendtoaddmorecomponents
top related