rabbitmq for pcf - pivotal · documentation for rabbitmq for pcf v1.7 and earlier only describes a...

RabbitMQforPCF®

Documentation

Version1.12

Published:5March2019

16212428415657646869737678808297

100105107

TableofContents

TableofContentsRabbitMQforPCFRabbitMQ®forPCFReleaseNotesUnlockingthePowerofOn-DemandRabbitMQforPCFOn-DemandServiceArchitectureDeployingtheRabbitMQPre-ProvisionedServiceInstallingandConfiguringtheOn-DemandServiceInstallingandConfiguringthePre-ProvisionedServiceSmokeTestsMonitoringandKPIsforPre-ProvisionedRabbitMQforPCFSettingLimitsforOn-DemandServiceInstancesControllingAccesstoServicePlansbyOrgIsolatingClusterswiththeRabbitMQforPCFReplicatorSettingDefaultPoliciesfortheRabbitMQServiceUsingtheRabbitMQManagementDashboardClusteringandNetworkPartitionsUpgradingRabbitMQforPCFTroubleshootingandFAQsforOn-DemandRabbitMQforPCFFrequentlyAskedQuestionsforPre-ProvisionedRabbitMQforPCFUsingOn-DemandRabbitMQforPCFRabbitMQEnvironmentVariablesTroubleshootingInstances

©CopyrightPivotalSoftwareInc,2013-2019 2 1.12

RabbitMQforPCF

AboutRabbitMQforPCFRabbitMQforPivotalCloudFoundry(PCF)enablesPCFappdeveloperstoprovisionandusetheRabbitMQmessagebrokerwithasinglecommand.

RabbitMQforPCFv1.8andlatersupporttwotypesofservice,anon-demandserviceandapre-provisionedservice.Thistablesummarizesthemaindifferencesbetweenthetwo:

AvailableSince VMsitRunsOn HowVMsareCreated MetricsName

Prefix

On-DemandService v1.8

DedicatedVMthatservesasingleserviceinstance.Seethistopicfordetails.

PCFcreateseachVMon-demandwhenappdevelopercreatesserviceinstance

p.rabbitmq

(withadot)

Pre-ProvisionedService

v1.2Multi-tenantVMssharedbyappsacrossPCFdeployment

PCFcreatesallVMswhenoperatordeploysorupdatesservice

p-rabbitmq

(withadash)

ThisRabbitMQforPCFdocumentationdescribesbothservicetypes.DocumentationforRabbitMQforPCFv1.7andearlieronlydescribesapre-provisionedservice.

WhatareOn-DemandInstancesInRabbitMQforPCFversionsbeforev1.8.0,theRabbitMQserviceinstancescorrespondtoauniqueRabbitMQvhostonthemulti-tenantRabbitMQcluster.RabbitMQforPCFv1.8.0introducedOn-DemandBroker(ODB) support.Thatmeansthatanew,single-tenant,clustercanbecreatedanddedicatedtoasingleapp.

Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCFandOn-DemandServiceArchitecture.

AboutRabbitMQRabbitMQisafastanddependableopen-sourcemessageserver,whichsupportsawiderangeofusecasesincludingreliableintegration,content-basedroutingandglobaldatadelivery,andhigh-volumemonitoringanddataingestion.

Emergingasthedefactostandardforcloudmessaging,RabbitMQisusedforefficientcommunicationbetweenservers,appsanddevices,andcreateslastingvaluebyenablingrapiddevelopmentofmoderndecentralizedappanddataarchitecturesthatcanscalewithyourbusinessneeds.

ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutRabbitMQforPCF.

Element Details

Version v1.12.15

Releasedate December28,2018

Softwarecomponentversions RabbitMQOSSv3.6.16andRabbitMQOSSv3.7.9

CompatibleOpsManagerversions v2.1.xandv2.2.x

CompatiblePivotalApplicationServiceversions v2.1.xandv2.2.x

IaaSsupport AWS,Azure,GCP,OpenStack,andvSphere

IPsecsupport No

Note:RabbitMQforPCFv1.12isnolongersupported.Thesupportperiodforv1.12hasexpired.Tostayup-to-datewiththelatestsoftwareandsecurityupdates,upgradetoasupportedversion.

Features

On-DemandCreateupto5differenton-demandRabbitMQplanswhichcanbeprovisionedthroughthemarketplace

Choosewhetheraplanhasone,three,five,orsevennodes

Defaultresourcesizesinplanstoguideselection

Morecontroloverwhichorgsandspaceshavevisibilityofeachconfiguredplan

Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding

Managementdashboardaccesstoappdevelopers

Deploymentintoanavailabilityzonespecifiedbytheplan

AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seereleasenotesfordowntimerequirements)

RabbitMQSyslogforwardingconfigurationinheritedfromthepre-provisionedconfiguration

RabbitMQmetricsareexposedontheFirehose

Runsmoketestsforon-demandplansonplan1

ErrandsarerunoncolocatedVMstodecreasedeploymenttimes

Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCF.

Pre-ProvisionedProvisionaninstanceoftheRabbitMQservice,whichcorrespondstoauniqueRabbitMQvhost(virtualhost)

Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding

ManagementdashboardaccesstoPCFOperatorsandappdevelopers

Deploymentacrossmultipleavailabilityzones,withnodesstripedacrosstheAZsautomatically

EnableSSL(SecureSocketsLayer)fortheAMQP,MQTT,STOMPprotocols

HAProxyloadbalanceracrossallnodestobalanceconnections

Pluginconfigurationcanbeeasilychangedatanytimeandtheclusterredeployedandupdated

Theclustertopologycanbechangedandeasilyscaledout

AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seeDowntimeWhenUpgradingfordowntimerequirements)

ConfiguretheendpointfortheRabbitMQSyslog

RabbitMQandHAProxymetricsareexposedonthefirehose

Syslogforwardingonbydefault

ErrandsarerunoncolocatedVMstodecreasedeploymenttimes

ReleaseNotesandKnownIssuesCheckthereleasenotesforyourreleaseversionforimportantinformationandknownissues.Toseereleasenotesforanotherversion,selecttheversionfromthedropdownlistatthetopofthepage.

RabbitMQforPCFandOtherPCFServicesSomePCFservicesofferon-demandserviceplans.Theseplansletdevelopersprovisionserviceinstanceswhentheywant.

Thesecontrastwiththemorecommonpre-provisionedserviceplans,whichrequireoperatorstoprovisiontheserviceinstancesduringinstallationandconfigurationthroughtheservicetileUI.

ThefollowingPCFservicesofferon-demandserviceplans:

MySQLforPCFv2.0andlater

RabbitMQforPCF

RedisforPCF

PivotalCloudCache(PCC)

Theseservicespackageanddelivertheiron-demandserviceofferingsdifferently.Forexample,someservices,likeRedisforPCF,haveonetile,andyouconfigurethetiledifferentlydependingonwhetheryouwanton-demandserviceplansorpre-provisionedserviceplans.

Forotherservices,likePCCandMySQLforPCF,onlyon-demandserviceplansareavailable.

ThefollowingtablelistsandcontraststhedifferentwaysthatPCFservicespackageon-demandandpre-provisionedserviceofferings.

PCFservicetile Standaloneproductrelatedtotheservice Versionssupportingondemand Versionssupportingpre-provisioned

RabbitMQforPCF PivotalRabbitMQ v1.8andlater Allversions

RedisforPCF Redis v1.8andlater Allversions

MySQLforPCF MySQL v2.x NA

PCC PivotalGemFire Allversions NA

Pleaseprovideanybugs,featurerequests,orquestionstothePCFFeedbacklist.

RabbitMQ®forPCFReleaseNotes

UpgradetotheLatestVersionPivotalrecommendsthatyouupgradetothelatestversionofyourcurrentminorline,thenupgradetothelatestavailableversionofthenewminorline.Forexample,ifyouuseanolderv1.11.xversion,upgradetothelatestv1.11.xversionbeforeupgradingtothelatestv1.12.xversion.

Forproductversionsandupgradepaths,seetheProductCompatibilityMatrix .

UseRabbitMQv3.7On-DemandPlansRabbitMQv3.7plansareavailable.EnsurethatallappsusetheRabbitMQv3.7on-demandplansinsteadoftheRabbitMQv3.6on-demandplans.

Formoreinformation,seethefollowing:

AboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7

TheANNRabbitMQ3.6.xsupporttimeline postontheRabbitMQusersforum

v1.12.15ReleaseDate:December28,2018

FeaturesNewfeaturesandchangesinthisrelease:

Requiresstemcell 3586.65

ScriptstodropandrestoreAMQP(S)traffic.Formoreinformation,seeDropandRestoreAMQP(S)TraffictoaRabbitMQInstance.

FixedIssuesThisreleasefixesthefollowingissue:

InRabbitMQforPCFv1.12.14andv1.12.13,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.

KnownIssuesThisreleasehasthefollowingissues:

[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .

IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.

ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.

Note:PivotalnolongerrecommendswaitingforSpringCloudServices(SCS)v3.0,wherethedependencyonpre-provisionedRabbitMQforPCFwillberemoved,beforeupgradingfromRabbitMQforPCFv1.12tov1.13.ThisisduetotheriskofgoingoutofsupportonyourRabbitMQforPCFtile,PivotalApplicationService(PAS),andothercomponents.

WARNING:PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1oronPCFv2.2.6orlater.

ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.

PackagesOSSRabbitMQv3.6.16

OSSRabbitMQv3.7.9

Erlangv20.3.8.15

HAProxyv1.6.13

v1.12.14ReleaseDate:December20,2018

FeaturesRequiresstemcell 3586.60

Setswapto1GBforRabbitMQnodes

SupportstheLogCachecfCLIpluginthatenablesdeveloperstoaccesslogsforanon-demandserviceinstanceusingthecommand cf tail .

Formoreinformationaboutthisfeature,seeAccessRabbitMQMetricsforOn-DemandServiceInstances.

KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .

IfyouupgradetoRabbitMQforPCFtothisversion,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.Thisissuedoesnotaffectnewinstallations.

OSSRabbitMQv3.7.9

Erlangv20.3.8.15

HAProxyv1.6.13

v1.12.13ReleaseDate:November22,2018

SecurityFixesThisreleaseincludesthefollowingsecurityfix:

CriticalCVE-2018-15759:OnDemandServicesSDKTimingAttackVulnerability

FeaturesAddsmutualTLSbetweenservicemetricsandtheloggregatorsystem

RemovescredentialsfromthedashboardURLinthepre-provisionedservice

IfyouupgradetoRabbitMQforPCFtothisversion,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.Thisissuedoesnotaffectnewinstallations.

OSSRabbitMQv3.7.8

Erlangv20.3.8.10

HAProxyv1.6.13

v1.12.12ReleaseDate:November1,2018

Smoketestsarenowmoreresilientwhenusingexternalloadbalancers.

KnownIssues

[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .

OSSRabbitMQv3.7.8

Erlangv20.3.8.9

HAProxyv1.6.13

v1.12.10ReleaseDate:October5,2018

RabbitMQ3.7updatedto3.7.8

Erlangupdatedtov20.3.8.9

OSSRabbitMQv3.7.8

Erlangv20.3.8.9

HAProxyv1.6.13

v1.12.9ReleaseDate:September13,2018

Smoketestsnowwaitupto5minutesforatestapptodeploy.

OSSRabbitMQv3.7.7

Erlangv20.3.8.6

HAProxyv1.6.13

v1.12.8ReleaseDate:August20,2018

FeaturesRequiresstemcell3468.64

Thereisanissuewithupgradingfromv1.12.xtov1.13.xonPCFv2.2.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1.

YoucannotupgradefromRabbitMQforPCFv.12toRabbitMQforPCFv1.13onPCFv2.2.0-2.2.5.YoumustupgradeonPCFv2.1oronPCFv2.2.6orlater.

OSSRabbitMQv3.7.7

Erlangv20.3.8.1

HAProxyv1.6.13

v1.12.7ReleaseDate:July13,2018

FixesbuginthesmoketestswhenCredHubisenabled

Thereisanissuewithupgradingfromv1.12.xtov1.13.xonPCFv2.2.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1.

YoucannotupgradefromRabbitMQforPCFv.12toRabbitMQforPCFv1.13onPCFv2.2.0-2.2.5.YoumustupgradeonPCFv2.1oronPCFv2.2.6orlater.

OSSRabbitMQv3.7.7

Erlangv20.3.8.1

HAProxyv1.6.13

v1.12.6

ReleaseDate:June28,2018

Requiresstemcell3468.51 .

UpdatesOSSRabbitMQServerandErlangpackages

OSSRabbitMQv3.7.6

Erlangv20.3.8.1

HAProxyv1.6.13

v1.12.5ReleaseDate:June5,2018

cf-cligolangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .

FixedIssuesThisreleasefixesthefollowingissue:

Smoketestssometimesfailedbecauseoflowtime-outvalues.

OSSRabbitMQv3.7.4

Erlangv19.3.6.4

HAProxyv1.6.13

v1.12.4ReleaseDate:May16,2018

Timestampsaddedtologentriesthatdidnothavethem.

Golangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .

Smoketestsmightfailbecausetime-outvaluesinthesmoketestsaretoolow.Formoreinformationabouttroubleshootingsmoketests,seeSmokeTests.

OSSRabbitMQv3.7.4

Erlangv19.3.6.4

HAProxyv1.6.13

v1.12.3ReleaseDate:April19,2018

On-Demand

ThefollowingerrandsarecolocatedwiththeirrespectivebrokertodecreaseerrandruntimeandVMfootprint:

Register/DeregisterOnDemandServiceBrokerOnDemandInstanceSmokeTestsUpgradeAllServiceInstancesDeleteAllServiceInstances

Formoreinformationabouterrands,seeErrands .

[Beta]ServiceSharingallowsRabbitMQdeploymentstobesharedacrossCloudFoundryspacesandorgs.Formoreinformationaboutsharingserviceinstances,seeCloudFoundryDocumentation .

[Beta]Supportforsecurecredentials(CredHub).Credentialsarestoredinacentralrepositoryandarerestrictedtocomponentsthatactuallyneedthem.

FormoreinformationaboutCredHub,seeCloudFoundryDocumentation .

Pre-Provisioned

ThefollowingerrandsarecolocatedwiththeirrespectivebrokertodecreaseerrandruntimeandVMfootprint:

BrokerRegistrar/DeregistrarSmokeTestserrands

Formoreinformationabouterrands,seeErrands .

OSSRabbitMQv3.7.4

Erlangv19.3.6.4

HAProxyv1.6.13

ViewReleaseNotesforAnotherVersionToviewthereleasenotesforanotherproductversion,selecttheversionfromthedrop-downlistatthetopofthispage.

UnlockingthePowerofOn-DemandRabbitMQforPCF

IntroductionRabbitMQforPivotalCloudFoundry(PCF)respondstothedemandsofPCFoperatorstoofferaRabbitMQon-demandclusterfortheirapplicationteams,inadditiontotheexistingsingle-nodeon-demandplan.Theon-demandclusterplanisaimedatworkloadsthatrequirethesameresiliencerequirementsasthePre-Provisionedoffering,butalsorequiretheirworkloadsbeisolated.

TheplatformoperationsteamcannowconfigureaRabbitMQforPCFclustertomeettheirbusinessrequirementsandempowerappdevelopmentteamstoself-servetheirownRabbitMQcluster.

RabbitMQforPCFalsoprovidessmoketestsfortheon-demandplanssothatoperationsteamscanvalidatetheappdeveloperworkflowforon-demandservices.SeeDedicatedInstanceSmokeTestProcess.

PlatformoperatorscannowoffertheirappdevelopersthreetypesofRabbitMQforPCFserviceplans:

Pre-provisioned—Forlighttomoderatemessagingneeds,thisserviceisfullyoperatedandmanagedbyplatformoperatorsasaservice.

On-demandsinglenode—Forapplicationteamsrequiringgreaterisolationthanprovidedbythepre-provisionedapproach.Appdevelopmentteamscanhavefullaccesstotheirownmessagebrokertoadapttheruntimeparameterstotheirrequirements.Formoreinformationontheseparameters,seeParametersandPolicies intheRabbitMQdocumentation.

On-demandcluster—Foranincreasedlevelofmessageresilienceandclusteravailability,aswellasthebenefitsofworkloadisolationmentionedabove.

Thistopicexplainshowtobenefitfromthetwoon-demandplansabove.

Forinformationaboutthepre-provisionedplan,seeDeployingtheRabbitMQPre-ProvisionedService.Forinformationonusingpre-provisionedplanstoisolateworkloads,seeCreatingIsolationwiththeTileReplicator.

On-DemandSingleNodePlanUsingRabbitMQ3.7Thisplanisdesignedtobesimpletoconfigure,deploy,anduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingopensourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.

Thisplancansuithigh-performanceworkloadsrequiringmessagingresilienceandasynchronousmessagingreplication.RabbitMQcopiesmessagestodiskforresilienceandallowsasynchronousmessagingreplicationthroughtheRabbitMQFederationplug-in.

Thisplanoffers:

FastaccesstoanisolatedinstanceofRabbitMQscopedfortheapplicationteams

OrgandSpaceAdministratoraccesstotheRabbitMQManagementUIsoapplicationteamscanhavefullcontroloverthenode

Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes

MessageresilienceprovidedthroughRabbitMQexchange,queueFederation,andShovelplugins.

ToBeDeprecated:On-DemandSingleNodePlanUsingRabbitMQ3.6Thisplanprovidesthesamebenefitsasdescribedabove,butusesRabbitMQ3.6forbackwardscompatibility.

RabbitMQ3.6isreachingendofsupport.ConsidermovingtoRabbitMQ3.7assoonaspossible.FormoreinformationaboutsupportforRabbitMQ3.6,seetheRabbitMQ3.6.xsupporttimelineannouncement .

IfyouarealreadyusingRabbitMQforPCF,youcanchoosetoprovideserviceplanswithRabbitMQ3.6forbackwardscompatibility.However,youshouldalsoenableRabbitMQ3.7planstoallowdeveloperstomigratetothenewversion.

Fornewdeployments,PivotalrecommendsdisablingallRabbitMQ3.6plansandusingRabbitMQ3.7toavoidtheneedtoupgradeinthenearfuture.

On-DemandClusterPlanUsingRabbitMQ3.7

Likethesinglenodeplan,thisplanisdesignedtobesimpletoconfigure,deployanduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingOpenSourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.

Thisplancansuithighperformanceworkloadsrequiringmessagingresilience(copiedtodisk)andasynchronousmessagingreplicationthroughtheRabbitMQFederationplugin.Withthisplan,however,youalsoscaleoutRabbitMQforPCFtomultiplenodes.

Thisplanoffers:

Fastaccesstoanisolated,clusteredinstanceofRabbitMQscopedtotheapplicationteamOrgsandSpaces

AdministratoraccesstotheRabbitMQManagementUItogiveapplicationteamsfullcontroloverthecluster

Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes.

MessageresilienceprovidedbymirroringqueuesacrossRabbitMQnodes,andtheoptiontousetheFederationandShovelplugins.

ToBeDeprecated:On-DemandClusterPlanUsingRabbbitMQ3.6Thisplanprovidesthesamebenefitsasdescribedabove,butusesRabbitMQ3.6forbackwardscompatibility.

RabbitMQ3.6isreachingendofsupport.ConsidermovingtoRabbitMQ3.7assoonaspossible.FormoreinformationaboutsupportforRabbitMQ3.6,seetheRabbitMQ3.6.xsupporttimelineannouncement .

IfyouarealreadyusingRabbitMQforPCF,youcanchoosetoprovideserviceplanswithRabbitMQ3.6forbackwardscompatibility.However,youshouldalsoenableRabbitMQ3.7planstoallowdeveloperstomigratetothenewversion.

Fornewdeployments,PivotalrecommendsdisablingallRabbitMQ3.6plansandusingRabbitMQ3.7toavoidtheneedtoupgradeinthenearfuture.

GeneralPrinciplesoftheClusterPlanThefollowingaresomegeneralprinciplestobeawareofwhenconfiguringtheclusterplan:

DesignedforConsistency

RabbitMQclusteringisnotprimarilyasolutionforincreasedavailability.Instead,itisdesignedforconsistencyandpartitiontolerance,asdescribedintheCAPtheorem .RabbitMQclusteringprovidesincreasedmessageconsistencythroughqueuemirroring.Thismeansthatmessagesaccessedinonequeueareexactlythesameasinanotherqueue.Formoreinformation,seeConsistencyorAvailabilityTradeoff.

Otheroptionscanbeusedforavailabilityrequirements,suchastheuseoffederationbetweenexchangesorqueues.

ForadetaileddescriptionofdistributedRabbitMQbrokers,seetheRabbitMQdocumentation .

NumberofNodes

Everynodeintheon-demandclustermaintainsacompletedatabaseofallmetadata,andallchangestothemetadataareconfirmedbyeverynodeinthecluster.Therefore,goingbeyondsevennodescanhaveasignificantnegativeimpactonperformance.Foroptimumresilienceandperformance,Pivotalrecommendsthreenodesformostworkloads.

NetworkLatency

RabbitMQclustersareonlyrecommendedfordeploymentinlowlatencynetworks,whichnormallymeansthatitisnotadvisabletodeploytheseclustersacrossavailabilityzones(AZs).ThestabilityandperformanceoftheRabbitMQclusterisheavilyinfluencedbytheworkloadonthenodes,replicationchoices,andnetworklatency.

Forthisreason,PivotalrecommendsthatyoudeployRabbitMQclustersintoasingleOpsManagerAZ.However,wheredifferentAZsareinthesamedatacenter,withreliablelowlatencylinks,spanningAZscanbeused.

ForcloudIaaSdeployments,Pivotaldoesnotrecommendthatdeploymentsspanregions.Forexample,inAmazonWebServices(AWS)terms,deployingaRabbitMQclusteracrossAZswithinaregionshouldprovidehighenoughnetworkperformancetopreventimpactingclusterstability.However,deployingacrossAWSregionsislikelytoleadtoclusterinstability.Formoreinformation,seetheAWSdocumentation .

ConsistencyorAvailabilityTradeoffInadistributedmessagingsystem,atradeoffmustbemadebetweenavailabilityorconsistencywhenanetworkpartitioneventoccursandoneormorenodesarenotabletocommunicatewitheachother.TheclusterplanletsoperatorsdecidehowtheywanttheRabbitMQclustertoreactintheeventofanetworkpartition.

Pivotalrecommendskeepingthedefaultclusterpartitionoptionof pause_minority becausethissatisfiesmostusecases.Choosingthe pause_minority

partition-handlingstrategyfavorsmessageconsistencyoveravailability.Formoreinformationabouttheoptionsforhandlingpartitions,seetheRabbitMQdocumentation .ForadetaileddescriptionoftheoptionsavailableinRabbitMQforPCF,seeClusteringandNetworkPartitions.

Hereisanexampleofhow pause_minority works.IfyoucreateaRabbitMQclusterwiththreenodesandonenodebecomesunabletocommunicatewiththeothertwo,thisnodeisintheminority.Thenodethatisintheminorityispaused,andtheothertwonodescontinueservingtraffic.Ifeachofthenodeslosesconnectivitywiththeothertwo,thentheentireclusterispausedtopreservedatasincenomajoritycanbeestablished.Theclusterhealswhentwoormorenodesareabletocommunicatewitheachother.

RabbitMQQueueAvailabilityItisimportanttobeawarethatmessagequeueavailabilityisdifferentfromclusteravailability.So,havingclusteravailabilitydoesnotmeanthatallofthemessageswithinthequeuesarealsoavailable.

Bydefault,queueswithinaRabbitMQclusterarelocatedonasinglenode—thenodeonwhichtheywerefirstdeclared.However,queuescanbeconfiguredtomirroracrossmultiplenodes,sothatanymessagepublishedtothequeueisreplicatedtoallmirrors.Enablingmirroringcanhaveanegativeimpactonqueueperformancebecausemessagesmustbecopiedtoallmirrorsbeforebeingacknowledged.

Eachmirroredqueueconsistsofonemasterandoneormoremirrors,withtheoldestmirrorbeingpromotedtothenewmasteriftheoldmasterdisappearsforanyreason.Consumersareconnectedtothemasterregardlessofwhichnodetheyconnectto,andmirrorsdropmessagesthathavebeenacknowledgedatthemaster.Queuemirroringenhancesqueueavailability,butdoesnotdistributeloadacrossnodesbecauseeachoftheparticipatingnodesmuststilldoallthework.

Appdevelopersmustdecideiftheywanttousequeuemirroringanddeterminethepolicytheywanttoapplytotheirqueues.Thesechoiceshavesignificantimpactontheavailabilityoftheirqueues.Formoreinformation,seetheRabbitMQdocumentation .

Unlikethepre-provisionedplan,theclusterplandoesnotshipwithadefaultloadbalancer.Therefore,developersmustconfiguretheirapptousethearrayofhostsprovidedin VCAP_SERVICES .Ifdevelopersenablequeuemirroring,theymustalsoensuretheirappshavere-trylogicandreconnectionlogicthatiteratesovertherangeofhostsprovided.MostcommonRabbitMQclientshavethislogicbuiltintothem.Formoreinformation,seetheSpringAMQPdocumentation .

Becausetheclusterplanisdesignedtoenableapplicationteamstoself-serve,nothavingaloadbalancerinfrontoftheRabbitMQclusterhasthesebenefits:

Manageresourcesbetter,asfewerVMsareneeded.

Helpwithtroubleshooting.ClientIPisnowtheIPofthesourcecontainerandnottheHAProxy.

Reducethenumberofhopsbetweenappsandbroker.Thishelpswithlatency.

Determinequeueplacement.Thismakessenseforlargerscaledeployments.

Empowerapplicationteamstomanagetheirclusterinthebestwayfortheirapp.

Requirere-trylogicinanappifitneedsHAaccesstoaqueue.Thus,allnodescanroutetoaqueueifitisavailable.

ManagingOn-DemandResourcesThroughPlansInconfiguringeachplan,thereareanumberofoperationalcontrolsthatplatformoperationsteamscanusetomanagetheresourcesconsumedbyon-demandRabbitMQ:

ControlAccess—Operatorscanchoosetheappdevelopmentorgsandspacesforwhichtheplansareavailableandvisible.Eachplancanbeenabledordisabled,andserviceaccessandvisibilitycaneitherbeglobal,orenabledperorgandspacethroughthecommandline.

Forexample,youmaydecidetoenablethesinglenodeon-demandplanacrossallapplicationteamstomeettheirdemandtoisolatetheirworkload.Youmaythenchoosetooffertheon-demandclusterplanonlytoasubsetofapplicationteamswhorequiretheextraresources.

SetQuotas—Youcansetaglobalquotaforallon-demandinstancesthattakesprecedenceovereachplanquota.Thisletsyouguardagainsttheriskofover-committingresources,butallowstheflexibilityofover-committingeachplan,soyoucanmeetthefluctuatingdemandsofyourappdevelopers.

ControlResourceConsumption—Eachplanoffersmorefine-grainedcontroloverindividualplanresourceconsumption.Atthehighestlevel,youcanusetheplanquotatocontrolthenumberofinstancesthatcanbedeployedwithinafoundation.Foreachplan,youcanalsoconfigurethenumberof

nodesthatconstituteacluster(3,5,or7),theinstancetype,andpersistentdiskstoragesizetobestsuityourrequirements.

Monitor—Youcanmonitorthenumberofinstancesthathavebeendeployedagainstthequotayouhavesetsothatyoucanplanfutureresourcerequirements.

CustomizingPlanOptionsTheRabbitMQforPCFon-demandplansexposeanumberofconfigurationoptions.Inmostcases,thedefaultconfigurationsmeetmostappdemands.However,itisimportantforanoperationsteamtoconsidertheoptionstoensurethattheyprovidethebestservicetotheirappdevelopers.Thissectionexplainstheseoptions.

ConfigurationOptions

SingleNodeandClusterPlansEnable/Disableplan

Determinewhichorgsandspacescanseeandaccesstheplan

SetServiceInstanceQuota

SelectAZplacement(whereapplicable)

SetRabbitMQinstancesize(CPUandMemory)

Setpersistentdisksize(PersistedMessageStore)fortheRabbitMQinstance.Ensurethesizeofthepersistentdiskisatleasttwiceaslargeastheinstancememory.

ClusterPlanOnlySetnumberofnodesto3,5,or7

Determinenetworkpartitionbehavior.SeeConsistencyorAvailabilityTradeoffabove.

ThingsThatArePreconfiguredThefollowingarepreconfiguredforboththesinglenodeandtheclusterplans:

RabbitMQVMType—WheninstallingonPCFv2.0orlater,eachRabbitMQnodeisconfiguredtohavethefollowingproperties:

CPUs:2RAM:8GBEphemeraldisk:16GB

YoucanchangethesesettingsintheServicePlanConfigurationpage.Changingthesesettingsaffectsallnodes.

PersistentDiskType—WheninstallingonPCFv2.0orlater,eachRabbitMQnodeisconfiguredtohave30GBofpersistentdiskspace.

YoucanchangethissettingintheServicePlanConfigurationpage.PivotalrecommendsyousetthisvaluetobetwicetheamountofRAMoftheselectedRabbitMQVMType.

Metrics—EmittedtotheLoggregatorFirehoseforallon-demandinstances.ThepollingintervalissetintheOpsManager,intheMetricspollingintervalfield,inthePre-ProvisionedRabbitMQtaboftheRabbitMQforPCFtile.Duetotheimpactofsomeoftheclustersettingsdetailedbelow,PivotalstronglyrecommendsthatyoumonitortheexposedmetricsandconfigurealarmsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF .SeealsoMonitoringOn-DemandRabbitMQClustersbelow.

Logs—RabbitMQon-demandinstancelogsareforwardedusingthesameconfigurationascontainedintheSyslogtaboftheRabbitMQforPCFtile.

Diskfreespacelimit—Thediskfreespacelimitissetto150%ofRAMoftheinstancetypeyouselect.Forexample,ifyouselectaninstancetypewith10GBofRAM,thediskfreespacelimitissetto15GB.Acluster-widealarmistriggerediftheamountoffreediskspacedropsbelowthis,andallpublishersareblocked.InstancesmustbeconfiguredtohavepersistentdisksthatareatleasttwicethesizeofinstanceRAM.Formoreinformation,seetheRabbitMQdocumentation .

Memorythresholdfortriggeringflowcontrol—Thresholdatwhichflowcontrolistriggeredissetto40%oftheinstanceRAM.Thismeansthatwhen

Note:Aloadbalancer,suchasHAProxy,isnotdeployedwithon-demandclusterplans.

thealarmistriggered,allconnectionspublishingmessagesareblockedcluster-wideuntilthealarmiscleared.

Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan4GBofmemoryisused,allpublishingconnectionsareblocked.Formoreinformation,seeMemoryAlarms intheRabbitMQdocumentation.

Memorypagingthreshold—ThisisthelevelatwhichRabbitMQtriestofreeupmemorybyinstructingqueuestopagetheircontentsouttodisk.Thisisdonetotrytoavoidreachingthehighwatermarkandblockingpublishers.Thisthresholdissetto50%oftheconfiguredhighwatermark,whichis20%ofconfiguredmemory.Formoreinformationonmemorycalculation,seeChangestoMemoryAllocationwhenUpgrading .

Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan2GBofmemoryisused,allqueuesstartwritingallqueuecontentstodisk.Formoreinformation,seetheRabbitMQdocumentation .

MonitoringOn-DemandRabbitMQClustersItisimportanttomonitorandcomparethenumberofinstancesthathavebeendeployedagainstthequotayousetviathemetricexposedontheFirehose.

Eachinstanceispre-configuredtoemitmetricstotheFirehoseandcanbeidentifiedbythe deployment tag,whichhastheserviceinstanceID.ItisimportanttomonitorthesemetricsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF .

On-DemandServiceArchitectureThistopicdescribesthearchitectureforon-demandRabbitMQ®forPivotalCloudFoundry(PCF).

Forinformationaboutarchitectureoftheolder,pre-provisionedservice,seeDeployingtheRabbitMQ®Service.

ServiceNetworkRequirementWhenyoudeployPCF,youmustcreateastaticallydefinednetworktohostthecomponentvirtualmachinesthatconstitutethePCFinfrastructure.

PCFcomponents,liketheCloudControllerandUAA,runonthisinfrastructurenetwork.On-demandPCFservicesmayrequirethatyouhostthemonanetworkthatrunsseparatelyfromthisnetwork.Youcanalsodeploytilesonseparateservicenetworkstomeetyourownsecurityrequirement.

PCFv2.0andEarlierInPCFv2.0andearlier,cloudoperatorspre-provisionserviceinstancesfromOpsManager.Foreachservice,OpsManagerallocatesandrecoversstaticIPaddressesfromapre-definedblockofaddresses.

Toenableon-demandservicesinPCFv2.0andearlier,operatorsmustcreateaservicenetworksinBOSHDirectorandselecttheServiceNetworkcheckbox.Operatorsthencanselecttheservicenetworktohoston-demandserviceinstanceswhentheyconfigurethetileforthatservice.

PCFv2.1andLaterPCFv2.1andlaterincludedynamicnetworking.InPCFv2.1andlater,operatorscanusedynamicnetworkingwithasynchronousserviceprovisioningtodefinedynamically-provisionedservicenetworks.Formoreinformation,seeDefaultNetworkandServiceNetwork.

InPCFv2.1andlater,on-demandservicesareenabledbydefaultonallnetworks.OperatorscancreateseparatenetworkstohostservicesinBOSHDirector,butdoingsoisoptional.Operatorsselectwhichnetworkhostson-demandserviceinstanceswhentheyconfigurethetileforthatservice.

DefaultNetworkandServiceNetworkOn-demandPCFservicesrelyontheBOSH2.0abilitytodynamicallydeployVMsinadedicatednetwork.Theon-demandservicebrokerusesthiscapabilitytocreatesingle-tenantserviceinstancesinadedicatedservicenetwork.

On-demandservicesusethedynamically-provisionedservicenetworktohostthesingle-tenantworkerVMsthatrunasserviceinstanceswithindevelopmentspaces.ThisarchitectureletsdevelopersprovisionIaaSresourcesfortheirserviceinstancesatcreationtime,ratherthantheoperatorpre-provisioningafixedquantityofIaaSresourceswhentheydeploytheservicebroker.

Bymakingservicessingle-tenant,whereeachinstancerunsonadedicatedVMratherthansharingVMswithunrelatedprocesses,on-demandserviceseliminatethe“noisyneighbor”problemwhenoneapphogsresourcesonasharedcluster.Single-tenantservicescanalsosupportregulatorycompliancewheresensitivedatamustbecompartmentalizedacrossseparatemachines.

Anon-demandservicesplitsitsoperationsbetweenthedefaultnetworkandtheservicenetwork.Sharedcomponentsoftheservice,suchasexecutivecontrollersanddatabases,runcentrallyonthedefaultnetworkalongwiththeCloudController,UAA,andotherPCFcomponents.Theworkerpooldeployedtospecificspacesrunsontheservicenetwork.

ThediagrambelowshowsworkerVMsinanon-demandserviceinstancerunningonaseparateservicesnetwork,whileothercomponentsrunonthedefaultnetwork.

RequiredNetworkingRulesforOn-DemandServicesBeforedeployingaservicetilethatusestheon-demandservicebroker(ODB),requesttheneedednetworkconnectionstoallowcomponentsofPivotalCloudFoundry(PCF)tocommunicatewithODB.

ThespecificsofhowtoopenthoseconnectionsvariesforeachIaaS.

Seethefollowingtableforkeycomponentsandtheirresponsibilitiesinanon-demandarchitecture.

KeyComponents TheirResponsibilities

BOSHDirector

CreatesandupdatesserviceinstancesasinstructedbyODB.

BOSHAgentIncludesanagentoneveryVMthatitdeploys.TheagentlistensforinstructionsfromtheBOSHDirectorandcarriesoutthoseinstructions.TheagentreceivesjobspecificationsfromtheBOSHDirectorandusesthemtoassignarole,orjob,totheVM.

BOSHUAA IssuesOAuth2tokensforclientstousewhentheyactonbehalfofBOSHusers.

PAS Containstheappsthatareconsumingservices

ODB InstructsBOSHtocreateandupdateservices,andconnectstoservicestocreatebindings.

Deployedserviceinstance

Runsthegivendataservice.Forexample,thedeployedRedisforPCFserviceinstancerunstheRedisforPCFdataservice.

Regardlessofthespecificnetworklayout,theoperatormustensurenetworkrulesaresetupsothatconnectionsareopenasdescribedinthetablebelow.

Thiscomponent…Mustcommunicatewith…

DefaultTCPPort Communicationdirection(s) Notes

BOSHDirector

BOSHUAA

8443 One-way Thedefaultportsarenotconfigurable.

ODBDeployedserviceinstances

15672(RabbitMQManagementUI)

One-wayThisconnectionisforadministrativetasks.Avoidopeninggeneraluse,app-specificportsforthisconnection.

ODBPAS(orElastic 8443 One-way Thedefaultportisnotconfigurable.

Runtime)

ErrandVMs

PAS(orElasticRuntime)

DeployedServiceInstances

5671-2(AMQP/AMQPS)

One-way Thedefaultportisnotconfigurable.

BOSHAgentBOSHDirector

4222 Two-way

TheBOSHAgentrunsoneveryVMinthesystem,includingtheBOSHDirectorVM.TheBOSHAgentinitiatestheconnectionwiththeBOSHDirector.Thedefaultportisnotconfigurable.

DeployedappsonPAS(orElasticRuntime)

Deployedserviceinstances

5671-2(AMQP/AMQPS)

61613-4(STOMP/STOMPS)

1883,8883(MQTT/MQTTS)

One-wayThisconnectionisforgeneraluse,app-specifictasks.Avoidopeningadministrativeportsforthisconnection.

PAS(orElasticRuntime)

ODB 8080 One-wayThisportmaybedifferentforindividualservices.Thisportmayalsobeconfigurablebytheoperatorifallowedbythetiledeveloper.

DeployedappsonPAS

RuntimeCredHub

8844 One-wayThisportisneededifsecureserviceinstancecredentialsareenabled.Forinformation,see(Beta)OnDemand-SecureServiceInstanceCredentialswithRuntimeCredHub.

DeployingtheRabbitMQPre-ProvisionedService

DefaultDeploymentDeployingRabbitMQforPivotalCloudFoundry (PCF)throughOpsManagerdeploysaRabbitMQclusterof3nodesbydefault.

Thedeploymentincludesasingleloadbalancer haproxy whichspreadsconnectionsonallofthedefaultports,foralloftheshippedpluginsacrossallofthemachineswithinthecluster.

Thedeploymentoccursinasingleavailabilityzone(AZ).

ThedefaultconfigurationisfortestingpurposesonlyandPivotalrecommendsthatcustomershaveaminimumof3RabbitMQnodesand2HAProxynodes

ConsiderationsforthisdeploymentProvideshighavailabilityfortheRabbitMQcluster

Queuesmustbeconfiguredtobehighavailabilityastheyareplacedononenodebydefault

Customersshoulddecidewhichpartitionbehaviorisbestsuitedtotheirusecase.Fortwonodes‘automatic’ispreferred.

HAProxyisasinglepointoffailure(SPOF)

TheentiredeploymentisinasingleAZ,whichdoesnotprotectagainstexternalfailuresfromfailuresinhardware,networking,etc.

RecommendedDeployment

PivotalrecommendsthatRabbitMQforPCFisdeployedacrossatleasttwoAZs.ScaleRabbitMQservernodestoanoddnumberthatisgreaterthanorequaltothree.

Onlyusereplicationofqueueswhererequiredasitcanhaveabigimpactonsystemperformance.

TheHAProxyjobinstancecountshouldalsobeincreasedtomatchthenumberofAZstoensurethereisaHAProxylocatedineachAZ.ThisremovestheHAProxySPOFandprovidesfurtherredundancy.

ThediagramaboveshowsthatyoucannowsufferthefailureofasingleHAProxyandsingleRabbitMQnodeandstillkeepyourclusteronlineandapplicationsconnected.

UpgradingtothisdeploymentfromasingleAZdeploymentItisnotpossibletoupgradetothissetupfromthedefaultdeploymentacrossasingleAZ.

ThisisbecausetheAZsetupcannotbechangedafterthetilehasbeendeployedforthefirsttime.ThisistoprotectagainstdatalosswhenmovingjobsbetweenAZs.

UpgradingtothisdeploymentfromamultiAZdeploymentIfyouhavedeployedthetileacrosstwoAZs,butwithasingleHAProxyinstance,youcanmigratetothissetupbydeployinganadditionalHAProxyinstancethroughOpsManager.Neworre-boundapplicationstotheRabbitMQserviceseetheIPsofbothHAProxiesimmediately.Existingboundapplicationswillcontinuetowork,butonlyusingthepreviouslydeployedHAProxyIPAddress.Theycanbere-boundasrequiredatyourdiscretion.

ConsiderationsforthisdeploymentRequiresIaaSconfigurationforAZsaheadofdeployingtheRabbitMQtile

ApplicationdevelopersarehandedtheIPsofeachdeployedHAProxyintheirenvironmentvariables

Queuesmustbeconfiguredtobehighavailabilityastheyareplacedononenodebydefault

Customersshoulddecideonwhichpartitionbehaviorisbestsuitedtotheirusecase.Forthreeormorenodes'pause_minority’ispreferred.

AdvancedDeploymentThisdeploymentbuildsupontheaboverecommendeddeploymentandsofollowsthesameupgradepaths.

ThisallowsyoutoreplacetheuseofHAProxywithyourownexternalloadbalancer.

YoumightchoosetodothistoremoveanyknowledgeofthetopologyoftheRabbitMQsetupfromapplicationdevelopers.

Advantages

ApplicationdevelopersdonotneedtohandlemultipleIPsfortheHAProxyjobsintheirapplications

Disadvantages

TheloadbalancerneedstobeconfiguredwiththeIPsoftheRabbitMQNodes.Theseareonlybeknownafterthedeploymentisfinished.TheIPsshouldremainthesameduringsubsequentdeploymentsbutthereisarisktheymightchange.

UpgradingtothisdeploymentfromtherecommendeddeploymentItispossibletofirstdeploywithmultipleHAProxyjobs,aspertherecommendeddeployment,andlateruseyourownexternalloadbalancer.

Thiscanbeachievedwithoutdowntimetoyourapplications.Followthesestepstodoso:

1. ConfigureyourexternalloadbalancertopointtotheRabbitMQNodeIPs.

2. ConfiguretheDNSnameorIPaddressfortheexternalloadbalancer(ELB)ontheRabbitMQtileinOpsManager.

3. Deploythechanges.AnynewinstancesoftheRabbitMQserviceoranyre-boundconnectionswillusetheDNSnameorIPaddressoftheELBintheirVCAP_SERVICES .AnyexistinginstanceswillcontinuetousetheHAProxyIPaddressesintheir VCAP_SERVICES

4. Phasethere-bindingofexistingapplicationstoupdatetheirenvironmentvariables.

5. Afterallapplicationsareupdated,reducetheinstancecountofthe HAProxy jobinOpsManagerto1.

6. Deploythechanges.

Thisapproachworksasanyexistingboundapplicationshavetheir VCAP_SERVICES informationcachedinCloudControllerandareonlyupdatedbyare-

bindrequest.

DowngradingfromthisdeploymenttotherecommendeddeploymentIfyouarecurrentlyusinganexternalloadbalancer,thenyoucanmovebacktousingHAProxiesinstead.

Youcanachievethisbyfollowingtheabovestepsinreverseorderandre-instatingtheHAProxyjobs.

ResourcerequirementsThefollowingtableshowsthedefaultresourceandIPrequirementsforinstallingthetile:

Product Resource Instances CPU Ram Ephemeral Persistent StaticIP DynamicIP

RabbitMQ RabbitMQnode 3 2 8192 16384 30720 1 0

RabbitMQ HAProxyforRabbitMQ 1 1 2048 4096 0 1 0

RabbitMQ RabbitMQservicebroker 1 1 2048 4096 0 1 0

RabbitMQ BrokerRegistrar 1 1 1024 2048 0 0 1

RabbitMQ BrokerDeregistrar 1 1 1024 2048 0 0 1

RabbitMQ SmokeTests 1 1 1024 2048 0 0 1

RabbitMQ RabbitMQon-demandbroker 1 1 1024 8192 1024 0 1

RabbitMQ RegisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1

RabbitMQ DeregisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1

RabbitMQ DeleteAllServiceInstances 1 1 1024 2048 0 0 1

RabbitMQ UpgradeAllServiceInstances 1 1 1024 2048 0 0 1

Notes:Thenumberof RabbitMQ Node canbeincreasedifrequired.

ChangingthenumberofRabbitMQnodeswhentheerlangcookieisnotdefinedwillrestartthecluster.Checkhereformoreinformation.

InstallingandConfiguringtheOn-DemandServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideon-demandservice.

TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .

Role-BasedAccessinOpsManagerOpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.

FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .

PrerequisitesforDeployingtheOn-DemandServiceBeforedeployingRabbitMQforPCFasanon-demandservice,youmustensurethattherequirednetworkrulesareinplacetoallowvariouscomponentstocommunicate.

SeeRequiredNetworkingRulesforOn-DemandServicesfordetailsonthenetworkconnectionsthatmustbeopentoenabletheon-demandservice.

Forinformationabouttheon-demandservicearchitecture,seeOn-DemandServiceArchitecture.

DownloadandInstallRabbitMQforPCF1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.

3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.

4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.

ConfigureOn-DemandRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.

Note:Forinstructionsonhowtoinstall,configure,anddeploytheRabbitMQforPCFtileasapre-provisionedservice,seeInstallingandConfiguringthePre-ProvisionedService.

WhichSettingsTabstoConfigurefortheOn-DemandServiceConfigurethefollowingtabsfortheon-demandservice:

RabbitMQSettingsTab Instructions

AssignAZsandNetworks ConfigureAZsandNetworks

Pre-ProvisionedRabbitMQ ConfigureAdminCredentialsandMetricsPollingInterval

Networking Networking

Syslog SetupSyslogForwarding

GlobalSettingsforOn-DemandPlans ConfigureGlobalSettings

OnDemandInstance:Plann ConfiguretheServicePlan

Errands Errands

Stemcell VerifytheStemcell

ConfigureAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.

1. ClickAssignAZsandNetworks.

2. ConfigurethefieldsontheAssignAZsandNetworksasfollows:

Field Instructions

Placesingletonjobsin

SelecttheregionthatyouwantforsingletonVMs.PCFcreatestheRabbitMQbrokerinthisAZ.

Balanceotherjobsin

Selectadditionalregion.Thisselectiondoesnotaffecttheon-demandRabbitMQforPCFservice.

Network

SelectanetworkfortheRabbitMQOn-DemandBroker.

ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.FormoreinformationabouttheDefaultNetwork,seeDefaultNetworkandServiceNetwork.

Typically,youselectthenetworkusedforthePivotalApplicationService(PAS)orElasticRuntimecomponents.

ServiceNetwork

Selectaseparatenetworkthattheon-demandserviceinstancesrunon.

Atypicalpracticeistoputallon-demandservicesonasinglenetwork,separatefromthenetworkthatPivotalApplicationService(PAS)orElasticRuntimeandtheOn-DemandBrokerrunon.ForinformationabouttheServiceNetwork,seeDefaultNetworkandServiceNetwork.

Thisfieldisalsorequiredforthepre-provisionedservice,thoughinthatcase,itdoesn’tmatterwhichnetworkyouselect.

3. ClickSave.

ConfigureAdminCredentialsandMetricsPollingIntervalOnthePre-ProvisionedRabbitMQtab,youmustconfiguretwoitemsonlyfortheondemandservice:

SpecifyadmincredentialsforusingtheRabbitMQManagementDashboard.

Enterametricspollinginterval.

SpecifyAdminUserCredentials

IntheRabbitMQadminusercredentialsfieldofthePre-ProvisionedRabbitMQtab,enteranadminusernameandpassword:

Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesintheApplyChangesfromYourConfigurationbelow.

WARNING:ChangingtheNetworkorServiceNetworkafteryouhaveconfiguredthem,orchangingtheirIPconfigurations,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

Youcanuseacombinationofupperorlowercasealphanumericsandsupportedspecialcharacters: []^_!"#$%&()*+,-./\:;<=>? .

ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.

EnteraMetricsPollingInterval

IntheMetricspollingintervalfield,setametricspollinginterval:

Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.

Changingthissettingaffectsalldeployedinstances.

Formoreinformation,seeWhatareMetrics .

ConfigureLoggingtoMonitorRabbitMQforPCFPivotalrecommendsthatyouconfigureloggingtomonitorthehealthofRabbitMQforPCF.FollowSetUpSyslogForwarding toconfigurelogging.

NetworkingTospecifyastaticIPaddressfortheOnDemandServiceBroker,dothefollowing:

1. SelectNetworking.

2. EnteranIPaddressintheOnDemandServiceBrokerStaticIPfield.ThisIPaddressisassignedtoyourOnDemandServiceBrokernode.BOSHallocatesanIPaddressifthefieldisleftblank.

3. ClickSave.

ConfigureGlobalSettingsFollowthestepsbelowtoconfigureglobalsettings.

1. ClickGlobalSettingsforOn-DemandPlans.

Note:Youcannotusebackquote ` orsinglequote ' .

Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.

2. Configurethefollowing:

Serviceinstancequotamin:0,max:50setthetotalnumberofon-demandserviceinstanceswhichcanbedeployed.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.

VMoptions:

Allowoutboundinternetaccess(IaaS-dependent).Thischeckboxmustbetickedtoallowexternallogforwarding,sendingbackupartifactstoexternaldestinations,andcommunicatingwithanexternalBOSHblobstore.

(Beta)ShareableInstances:ClickYestoenablethebetafeatureforsharinginstances.

Note:OutboundnetworktrafficrulesalsodependonyourIaaSsettings.ConsultyournetworkorIaaSadministratortoensurethatyourIaaSallowsoutboundtraffictotheexternalnetworksyouneed.

Note:ThisisabetafeaturebasedonanexperimentalfeatureinCloudFoundry.Usethefeatureatyourownriskinnon-productionenvironments.Ifyoutrythisfeature,pleasesendyourcommentsandfeedbacktoPCFFeedbackList.

Sharingaserviceinstancebetweenspaces,allowsappsindifferentspacestosharedatabases,messagingqueues,andmanyothertypesofservices.Formoreinformation,seeSharingServiceInstances(Beta) .

*(Beta)OnDemand-SecureServiceInstanceCredentialswithRuntimeCredHub:Foron-demandservicesinstances,clickYestosecurecredentialswithCredHub.

3. ClickSave.

ConfiguretheServicePlanToenabletheon-demandservice,youmustconfigureatleastoneon-demandplan.

Youcanconfigureuptofiveon-demandplans:OnDemandInstance:Plan1–OnDemandInstance:Plan5.

Allon-demandplanscanbeconfiguredtohave1,3,5,or7RabbitMQnodes.

Iftheon-demandserviceisnotenabled,theon-demandbrokerisdeployedalongsidetheRabbitMQinstallation,butitisnotavailableintheMarketplace.

1. Choosetheon-demandserviceinstanceyouwanttoconfigure:

OnDemandInstance:Plan1(completerequiredfieldsevenifyoudisablethisplan):

Note1:ForthisfeaturetoworkyoumustalsoenableitinPAS.Forinstructions,seeStep1:ConfigurethePASTile .Afterdeployingthetile,notifydevelopersthattheymustunbindandrebindexistingserviceinstancestosecuretheircredentialswithCredHub.

Note2:Thisisabetafeature.Useitatyourownriskinnon-productionenvironments.Ifyoutrythisfeature,pleasesendyourcommentsandfeedbacktoPCFFeedbackList.

Note:YoumustfullyconfigureOnDemandInstance:Plan1evenifyoudisableaccesstothisplan(seeCFServiceAccessinthetablebelow).

OnDemandInstance:Plan2,3,4,and5:

2. Configurethefieldsasfollows:

Field Instructions

EnableThisPlan

(Plans2-5only)Toenable,selectPlanEnable.

CFServiceAccess

Enableordisableaccesstothisplan,orleaveaccessunchanged.

IfyouenablePlan1,thedefaultsettingforPlans2-5isEnableServiceAccess.Ifyouchangethisdefaultsetting,thesmoketestsfail.Therefore,ifyouenablePlan1andwanttochangethisdefault,beforedoingso,settheOn-DemandInstanceSmokeTestserrandtoOff.Formoreinformation,seeErrands.

EnableServiceAccess—Givesaccesstoallorgs,anddisplaystheserviceplantoalldevelopersintheMarketplace.DisableServiceAccess—Disablesaccesstoallorgs,andhidestheserviceplantoalldevelopersintheMarketplace.ThissettingcannotbeselectedatalatertimeintheUI.LeaveServiceAccessUnchanged—Keepsanyexistingaccesssettings,andonlydisplaystheserviceplanintheMarketplacetomembersoforgsthathaveaccesstotheplan.YoucanchangetheaccesssettingslaterusingthecfCLI.Forinstructions,seeControllingAccesstoServicePlansbyOrg.

PlanName Acceptthedefaultorenteraname.ThisisthenamethatappearsintheMarketplace.

PlanDescription

Acceptthedefaultorenteradescription.ThisdescriptionappearsintheMarketplace.

PlanFeatures

Acceptthedefaultorenteradescription.ThisdescriptionappearsinAppsManager.

ServiceInstanceQuota

Enterthemaximumnumberofon-demandserviceinstancesthatcanbeavailableatonetime.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.

NumberofNodes

Enter1,3,5or7.Thissettingonlyaffectsnewserviceinstances.Previouslydeployedserviceinstancesarenotupdated.

NetworkPartitionBehaviour

Select pause_minority or autoheal .Pivotalrecommendsusingpauseminority.Formoreinformation,seeConsistencyorAvailabilityTradeoff.

RabbitMQVMType

SelectalargeVMtype.Theplancreatesaserviceinstanceofthissize.Formoreinformation,seeUnderstandingRabbitMQVMTypesandPersistentDiskSizebelow.

PersistentDiskType

ThisiswhereRabbitMQpagesmessagestodisk.ServiceinstancedeploymentsfailifthisvalueislessthantwicethevolumeofRAMoftheselectedRabbitMQVMType.Formoreinformation,seeUnderstandingRabbitMQVMTypesandPersistentDiskSizebelow.

AZPlacement

ThisfieldisavailableafteryoucompletetheAssignAZsandNetworkspage.

Forasingle-nodeplan,selectoneormoreAZs.Foraplancontainingmultiplenodes,selectonlyoneAZ.Pivotalrecommendsthisformulti-nodeplanstominimizerisksduetonetworklatencyandpartitions.SeeNetworkLatencyandConsistencyorAvailabilityTradeofffordetails.

Ifyouchangethisselectionafterdeployment,existinginstancesarenotaffectedbythechange.SeeDeterminewhichAZsaServiceInstanceUses.

3. ClickSave.

DeterminewhichAZsaServiceInstanceUses

TodeterminewhichAZsaserviceinstanceisplacedin,dooneofthefollowing:

RetrievetheserviceGUIDusingthe cf service SERVICE_INSTANCE --guid commandandthenruntheBOSH instances commandfortheservice-instance_GUID deployment.

Withsyslogforwardingenabled,inspecttheservicebrokerlogswhenrunningtheUpgradeAllServiceInstanceserrand.Foreachexistingservice

Important:IfyouchangethisconfigurationafteryouhaveselectedAZsanddeployedserviceinstances,existinginstancesarenotplacedinthenewlyconfiguredAZswhentheUpgradeAllServiceInstanceserrandisrun.Thispreventsre-creationoftheVMsindifferentAZs,whichcanleadtodataloss.Allnewserviceinstances,however,willbecreatedinthenewlyconfiguredAZs.

instance,thelogmessageincludestheserviceinstanceGUIDandtheAZstheserviceinstanceisrunningin.

UnderstandingRabbitMQVMTypesandPersistentDiskSize

TheRabbitMQVMTypeandPersistentdisktypearerequiredfieldsontheserviceplanconfigurationpages.IfyouareinstallingonPCFv2.0orlater,thesepropertiesarepre-configuredbydefault.

PivotalrecommendsthatthevalueofPersistentdisktypebetwicetheamountofRAMoftheselectedRabbitMQVMType.

YoucanchangetheRabbitMQVMtypeandthesizeofthepersistentdiskthatisattachedtotheRabbitMQinstances.Forexample,ifyouarerunningoutofdiskspaceyoumightwanttoincreasethepersistentdisksizebychangingthePersistentdisktypefield.Ifyoumakechanges,ensurethatthepersistentdisksizeisstilltwicethesizeoftheRAMoftheRabbitMQVMtype.

RabbitMQraisesalarmswhendiskspacedropsbelowtheconfiguredlimit.Incorrectdisksizesmightcausethedeployedinstancenottostart.RabbitMQdeclinestostartifthereisnotenoughspaceavailableaccordingtothethreshold.

On-Demandinstancesareconfiguredwithathresholdsettothe150%ofthememory(RAM)oftheVM.Usethefollowingtableasaguidewhenselectingthesizeofthepersistentdisk.

ThefollowingtableshowsanexampleofpossibleRAMvalues,absoluteminimalvaluebelowwhichRabbitMQdeclinestostart,andthedisksizesuggestedforanaverageusecase.

RAM Freediskalarmthreshold(1.5xRAM) Suggesteddisksize(2xRAM)

10GB 15GB 20GB

16GB 24GB 32GB

32GB 48GB 64GB

MinimumresourcesrequiredforeachRabbitMQVM:

RAM:1GB

Ephemeraldisk:2GB

Persistentdisk:4GB

Formoreinformation,seethefollowing:

MemoryandDiskAlarms

DiskAlarms

Forinformationonallpreconfiguredsettings,seeThingsthatarePreconfigured.

VerifytheStemcellForOpsManagerv2.0:

1. ClickStemcell.

2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .

ForOpsManagerv2.1:

1. FollowtheprocedureoutlinedinImportingandManagingStemcells .

ApplyChangesfromYourConfigurationYourinstallationisnotcompleteuntilyouapplyyourconfigurationchanges.Followthestepsbelow:

1. ReturntotheOpsManagerInstallationDashboard.

2. ClickApplyChanges.

ErrandsWhendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofPost-DeployErrands.Anexampleisthe Smoke

Testserrand,

whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.

Youcandecidewhethertorunerrandsbytogglingthemonoroffbeforeanupdate.Thisisaone-timesettingontheinstallationdashboard:

However,ifnecessary,youcanchangethesedefaultsbyclickingErrandsintheRabbitMQforPCFSettingstab.

Formoreinformationonerrandrunrules,seeErrandRunRules .

RabbitMQforPCFerrandsarecolocatedwiththeirbrokerstodecreaseerrandruntimeandVMfootprint.Inearlierreleases,anewVMwasdeployedforeacherrand.Formoreinformationabouterrands,seeErrands .

Important:AsofRabbitMQforPCFv1.9.0,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,becausethesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmightfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.

Post-DeployErrands

Errand Description

BrokerRegistrar

Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace

SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTests.

RegisterOn-DemandServiceBroker

Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.

On-DemandInstanceSmokeTests

Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTestsbelow.

UpgradeAllServiceInstances

On-Demandinstancesareupdatedandredeployediftherearechangestoon-demandplansettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoon-demandplansettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.

Pre-DeleteErrands

Errand Description

BrokerDeregistrarRemovesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings

DeleteAllServiceInstancesUnbindsanddeletesexistingon-demandserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.

DeregisterOn-DemandServiceBroker

Removestheon-demandRabbitMQservicefromtheMarketplace

On-DemandInstanceSmokeTestsSmoketestsonlyrunagainstPlan1.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.

CreateanAdminUserforaServiceInstanceIfyouwanttogiveappdevelopersadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstanceandsharetheusercredentialswithappdevelopers.

Bothoperatorsandappdeveloperscanusethisprocedure.

TocreateanadminuseronaRabbitMQinstancedothefollowing:

1. Runthiscommandtocreateaservicekey:

cfcreate-service-keySERVICE_INSTANCESERVICE_KEY-c'{"tags":"administrator"}'

where:

SERVICE_INSTANCE isthenameyousuppliedwhenyouran cfcreate-service .SERVICE_KEY isanameyouchoosetoidentifytheservicekey.

Forexample:

$cfcreate-service-keymy-instancemy-admin-key-c'{"tags":"administrator"}'

Creatingservicekeymy-admin-keyforserviceinstancemy-instanceasuser@example.com...OK

2. Runthiscommandtogettheadminusercredentials:

cfservice-keySERVICE_INSTANCESERVICE_KEY wherethevariablesarethesameasabove.

ThisreturnsaDashboardURLcontainingtheadmincredentials,whichcanbeusedtoaccessthemanagementUI.Forexample:

$cfservice-keymy-instancemy-admin-key

Gettingkeymy-admin-keyforserviceinstancemy-instanceasuser@example.com...{"dashboard_url":"https://my-instance.bosh-lite.com/#/login/admin-username/admin-password","username":"admin-username","password":"admin-password",...}

RabbitMQServerSettingsThatCannotBeDisabledThefollowingpluginsareenabledbydefaultandcannotbedisabled:

rabbitmq_management

rabbitmq_federation

rabbitmq_federation_management

rabbitmq_shovel

rabbitmq_shovel_management

InstallingandConfiguringthePre-ProvisionedServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideapre-provisionedservice.

TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .

Role-BasedAccessinOpsManagerOpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.

FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .

DownloadandInstalltheTile1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.

3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.

4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.

ConfigurePre-ProvisionedRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.

Note:Forinstructionsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtileasanon-demandservice,seeInstallingandConfiguringRabbitMQforPCFasanOn-DemandService.

WhichSettingsTabstoConfigureforthePre-ProvisionedServiceConfigurethefollowingtabsforthepre-provisionedservice:

RabbitMQSettingsTab Instructions

AssignAZsandNetworks AssignAZsandNetworks

Pre-ProvisionedRabbitMQ RabbitMQ

Networking Networking

Syslog SyslogGlobalSettingsforOn-DemandPlans GlobalSettings

Errands Errands

Stemcell Stemcell

AssignAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.

1. IntheSettingsscreen,clickAssignAZsandNetworks.

2. ConfigurethefieldsontheAssignAZsandNetworksasfollows.Allfieldsarerequired,thoughsomedonotapplytothepre-provisionedservice.

RequiredFields Instructions

Placesingletonjobsin

Selectaregion.Thisselectiononlyaffectstheon-demandservice.

Balanceotherjobsin

Selectadditionalregion(s).Thisselectiononlyaffectsthepre-provisionedservice.

Network

SelectanetworkfortheRabbitMQBroker.

ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.ThisnetworkisrepresentedbytheDefaultNetwork,describedinDefaultNetworkandServiceNetwork.Typically,youselectthenetworkusedforthePivotalApplicationService(PAS)orElasticRuntimecomponents.

ServiceNetwork

Selectanetwork.Thisselectiononlyaffectstheon-demandservice.

3. ClickSave.

RabbitMQToconfigurethefollowingsectionsinthePre-ProvisionedRabbitMQtab,intheSettingsscreen,clickPre-ProvisionedRabbitMQ.

RabbitMQAdminUserCredentialsIntheRabbitMQadminusercredentialsfieldoftheRabbitMQpane,enteranadminusernameandpassword:

Youcanuseacombinationofupperorlowercasealphanumericsandsupportedspecialcharacters: -=_+":;/?.><,~ .

ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.

Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesinthefinalstepbelow.

WARNING:ChangingtheNetworkafteryouhaveconfiguredit,orchangingtheIPconfiguration,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

Note:Youcannotusebackquote ` orsinglequote ' .

PluginsChoosewhichpluginsyouwanttoenableinthissectionofthePre-ProvisionedRabbitMQtab.

Youmustleavetherabbitmq_managementpluginenabledforthisproducttowork.

FormoreinformationaboutRabbitMQplugins,seetheRabbitMQdocumentation .

HAProxyPortsEntertheportsHAProxyshouldloadbalancetotheRabbitMQnodesinthissectionofthePre-ProvisionedRabbitMQtab:

Allthedefaultportsofalltheavailablepluginsareload-balancedbydefault.However,ifyouinstallextraprotocolplugins,orprovideacustomconfigurationthatchangestheportsRabbitMQlistenson,thenyoumustupdatethelistofload-balancedports.

Youmustleavethemanagementpluginlisteningonport 15672 andloadbalancethatport.

IfyouchangethetopologyofyourRabbitMQcluster,theHAProxyisautomaticallyreconfiguredduringthedeployment.

SSL(Optional)ProvideSSLcertificatesandkeysforusebytheRabbitMQclusterinthissectionofthePre-ProvisionedRabbitMQtab:

Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.

SSLissimultaneouslyprovidedforAMQPS,STOMPandMQTT.NootherpluginsareautomaticallyconfiguredforusewithSSL.

IfyouprovideSSLkeysandcertificates,non-SSLsupportisdisabled.IfyoupreviouslydeployedthisservicewithoutSSLsupportandhaveappsconnectedtotheservice,theseappslosetheirconnectionsandmustreconnectusingSSL.

SSLsettingsareappliedequallyacrossallVMsinthecluster.

YoucanprovidemorethenoneCAcertificate.

FormoreinformationaboutSSLsupport,seetheRabbitMQdocumentation .

ErlangCookie(Optional)ProvideanErlangcookietobeusedbytheclusterinthissectionofthePre-ProvisionedRabbitMQtab.ThisisusefulifyouwanttoconnectdirectlytotheRabbitMQcluster,forexamplewith rabbitmqctl ,ortoconnectotherVMsrunningErlang.

KnownIssueswithErlangCookie

TherearethreeknownissuesassociatedwiththeErlangcookie:

SecurityIssuewiththeTileGeneratedErlangCookie

ClusterScalingKnownIssue

ChangingtheErlangCookieValueKnownIssue

Note:Leavingthisfieldblankisasecurityrisk.SeeSecurityIssuewiththeTileGeneratedErlangCookiebelow.

SecurityIssuewiththeTileGeneratedErlangCookie

IfyouleavetheErlangcookiefieldblank,thetilegeneratesthecookieinawaythatcanbereverse-engineered.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .

Toavoidthisissue,settheErlangcookietoasecurepasswordvalue.Thisrequiresclusterdowntime,seeChangingtheErlangCookieValueKnownIssuebelow.

ClusterScalingKnownIssue

IfyouhavenotsettheErlangcookieandyouwanttoscaleoutyourclustersizewithoutdowntime,followthesesteps:

1. Followthestepsinthelinkbelow,uptoandincludingthesectionLogintotheBOSHDirector:AdvancedTroubleshootingwiththeBOSHCLI

2. Runthefollowingcommand:boshsshrabbitmq-server/0

3. Runthefollowingcommands:

sudo-iecho$(cat/var/vcap/store/rabbitmq/.erlang.cookie)

4. PastethevaluereturnedfromthelastcommandintotheErlangcookiefieldinthePre-ProvisionedRabbitMQtab.Thisfieldisshownabove.

5. Toincreasethesizeofyourcluster,navigatetotheResourceConfigtaband,inthefirstrow,raisethevalueforthenumberofInstancesoftheRabbitMQnode.

6. ReturntotheOpsManagerInstallationDashboard,andclickApplyChanges.

ChangingtheErlangCookieValueKnownIssue

ChangingtheErlangcookievaluerequiresclusterdowntime.Pivotalstronglyrecommendsthatyoudonotchangeanythingelseduringthistime,becauseitispossiblefortheconfigurationtobeinconsistentlyappliedduringthisprocess.

Thedeploymentmightfailafterthisprocess.Ifso,redeployingfixestheissue.

RabbitMQConfiguration(Optional)Provideafull rabbitmq.config filebypastingitscontentsintheRabbitMQconfigurationfieldinthePre-ProvisionedRabbitMQtab.Thisrabbitmq.config fileisthenprovidedtoallthenodesinthecluster.

TheinputinthisfieldmustbeBase64encoded.

Forexample,supposeyouwanttoconfigurethe rates_mode ofthe rabbitmq_management statsbelow:

[{rabbitmq_management,[{rates_mode,detailed}]}].

Note:BOSHtellsyouthatthecookiehaschanged—thisisbecausethedefaultvalueinthemanifestisempty,whichresultsinanauto-generatedcookie.However,thevalueofthecookieontheserverremainsthesame,sotheknownissuebelowdoesnotapply.

1. EncodethefileintoBase64:

WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K

2. PastetheaboveintotheRabbitMQconfigurationfield:

Youcanseeanexample rabbitmq.config filehere .FormoreinformationabouttheRabbitMQconfiguration,seetheRabbitMQdocumentation .

TLSSupportConfigureTLSinthissectionofthePre-ProvisionedRabbitMQtab:

TLSv1.0isdisabledbydefault,duetosecurityissues.

TLSv1.1andv1.2areenabledbydefaultandcanbeturnedonandoff.

ExternalLoadBalancer(Optional)EnteraDNSnameorIPaddressofanexternalloadbalancertobereturnedinthebindingcredentials( VCAP_SERVICES )toappdevelopers.EnterthisinthissectionofthePre-ProvisionedRabbitMQtab:

Ifyouconfigureanexternalloadbalancer,toavoidanunnecessaryVMdeployment,intheResourceConfigtabsettheHAProxyforRabbitMQinstancecountto0.

MetricsPollingIntervalThemetricspollingintervalissetinthissectionofthePre-ProvisionedRabbitMQtab:

Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.

Changingthissettingaffectsalldeployedinstances.

DiskFreeAlarmLimitChoosehowmuchdiskspaceRabbitMQattemptstokeepfreeatanygiventimeinthissectionofthePre-ProvisionedRabbitMQtab:

RabbitMQperiodicallychecksifthereissufficientfreespaceondisk.Ifthereisnot,RabbitMQtemporarilystopsacceptingnewmessages.Thisgivesyourappstimetoconsumeexistingmessages,andthusfreeupsomediskspace.TheRabbitMQtileprovidesfouroptionsforthisvalue:

50MBistheminimumvalue.(NotRecommended)Selecting50MBisnotrecommendedandcancausedataloss.Formoreinformation,seeDangersofSettingThisValueTooLowandWhentoUsethe50MBValuebelow.

100%MemoryensuresthatatthetimewhenRabbitMQcheckstheavailabledisk,theremustbeenoughspaceforRabbitMQtopageallmemory-basedmessagesouttodisk.

150%Memoryisrecommended.Thisisbecauseitispossiblethatinbetweendisk-spacechecks,RabbitMQmay:

Writepersistentmessagestodisk(usingupsomediskspace).Acceptmorememory-basedmessagesintovariousqueues.Pageallmemory-basedmessagestodisk.

Intheabovesituations,RabbitMQmightrequiremorefreediskthanithasmemory.

200%MemoryisaconservativevalueusedwhentheoperatorwantshigherconfidencethatRabbitMQneverrunsoutofdiskspace.

Formoreinformationaboutdiskalarms,seetheRabbitMQdocumentation .

DangersofSettingThisValueTooLow

IfthediskofagivenRabbitMQnodecompletelyfillswhileRabbitMQisrunning,thatnodecrashes.Thiscanleadtodataloss,andlossofavailability.

RabbitMQreservestherighttopageanyandallmessagesinmemory(eventransientmessages)todiskatanytime.YoumustsetyourDiskfreealarmlimithighenoughtoensurethatRabbitMQalwayshasatleastenoughspacetodothis.

DisadvantagesofSettingThisValueTooHigh

IfyousetyourDiskfreealarmlimittoavaluelargerthanthesizeofyourpersistentdisk,thenRabbitMQisnotabletofreeupenoughdiskspacetoacceptnewmessages.EnsurethatyouhavealargeenoughdisktopersistallthemessagesyouintendtopersistwhilealsoleavingenoughspacefreetosatisfytheDiskfreealarmlimitthatyouchoose.

WhentoUsethe50MBValue

Pivotaldoesnotrecommendusingthisvalueinproduction.However,ifyouareexperimentingwithadevelopmentenvironmentyoumightwanttouseasmalldisktokeepdowncosts,thoughthisincreasesthepossibilitythatRabbitMQcrashesandlosesdata.

NetworkingTospecifystaticIPaddresses,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.

2. IntheSettingstab,selectNetworking.

3. Configurethefieldsasfollows:

Pre-ProvisionedHAProxyStaticIPs:Enteracomma-delimitedlistofIPaddressesgroupedintheorderofAZsconfigured.TheseIPaddressesareassignedtoyourPre-ProvisionedHAproxynodes.

Pre-ProvisionedRabbitMQServerStaticIPs:Enteracomma-delimitedlistofIPaddressesgroupedintheorderofAZsconfigured.TheseIPaddressesareassignedtoyourPre-ProvisionedRabbitMQServernodes.

Pre-ProvisionedServiceBrokerStaticIP:EnteranIPaddress.ThisaddressmustbeinthenetworkrangeofthenetworknamespecifiedintheNetworkdrop-downlistintheAssignAZsandNetworkstabandmustnotbeintheServiceNetwork.ThisIPaddressisassignedtoyourPre-ProvisionedServiceBrokernode.

4. ClickSave.

SyslogToenablemonitoringforRabbitMQforPCF,operatorsforwardthesyslogbydesignatinganexternalsyslogendpointforRabbitMQcomponentlogmessages.ThisendpointservesastheinputtoamonitoringplatformsuchasDatadog,Papertrail,orSumoLogic.

TospecifythedestinationforRabbitMQforPCFlogmessages,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.

2. IntheRabbitMQtile,clicktheSettingstab.

Note:Ifanyoftheabovefieldsareleftblank,BOSHallocatesanIPaddress.

3. ClickSyslog.

4. ConfigurethefieldsontheSyslogpaneasfollows:

Option Description

Syslogaddress IPorDNSaddressofthesyslogserver

Syslogport Portofthesyslogserver

Transportprotocol

Transportprotocolofthesyslogserver.Oneof udp , tcp , relp .

Formatforlogs Formatforlogs.Pivotalrecommends RFC 5424 ,but Legacy Format canbeusedforcompatibilityreasons.

EnableTLS EnableTLStothesyslogserver.

PermittedPeerIfthereareseveralpeerserversthatcanrespondtoremotesyslogconnections,thenyoumayprovideawildcardinthedomain,suchas *.example.com .

CustomCACertificate

Iftheservercertificateisnotsignedbyaknownauthority,forexample,aninternalsyslogserver,providetheCAcertificateofthelogmanagementserviceendpoint.

5. ClickSave.

6. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoredeploywiththechanges.

GlobalSettingsFollowthestepsbelowtoenabletheshareableinstancesbetafeature.Sharingaserviceinstancebetweenspaces,allowsappsindifferentspacestosharedatabases,messagingqueues,andmanyothertypesofservices.Formoreinformation,seeSharingServiceInstances(Experimental) .

1. ClickGlobalSettings.

2. Select(Beta)ShareableInstancestoenablethebetafeatureforsharinginstances.

3. ClickSave.

Errands(Optional)IntheErrandstab,choosethedefaultsforwhenerrandsrun.

Errandscanbethoughtofastasks.Forexample,whendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofpost-deployerrands.AnexampleistheSmokeTestserrand,whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.

Youcandecidewhethertorunpost-deployerrandsbytogglingthemonoroffbeforeyouclickApplyChangestoupdateaconfigurationintheOpsManagerInstallationDashboard:

Thisisaone-timeactionbeforeanupdate.YoucanchangetheabovedefaultsintheErrandstab,aswellasthedefaultsforpre-deleteerrands.

WARNING:ThisisabetafeaturebasedonanexperimentalfeatureinCloudFoundry.Usethefeatureatyourownriskinnon-productionenvironments.

Important:InRabbitMQforPCFv1.9.0andlater,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,because

Formoreinformationonerrandrunrules,seeErrandRunRules .

Post-DeployErrands

Errand Description

BrokerRegistrar

Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace

SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTestsbelow.

RegisterOn-DemandServiceBroker

Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.

On-DemandInstanceSmokeTests

Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTests.

On-demandinstancesareupdatedandredeployediftherearechangestotheDedicatedInstancesettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoDedicatedInstancesettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.

Pre-ProvisionedInstanceSmokeTestsSmoketestsrunasapost-deploymenterrand.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.

Pre-DeleteErrandsPre-deleteerrandsrunafteranoperatorchoosestodeleteaproductintheOpsManagerInstallationDashboard,butbeforeOpsManagerfinishesdeletingtheproduct.

Errand Description

BrokerDeregistrar Removesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings

DeleteAllServiceInstancesUnbindsanddeletesexistingdedicatedserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.

DeregisterOn-DemandServiceBroker

Removestheon-demandRabbitMQservicefromtheMarketplace

StemcellForOpsManagerv2.0:

1. ClickStemcell.

2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .

ForOpsManagerv2.1:

1. FollowtheprocedureoutlinedinImportingandManagingStemcells .

ApplyConfigurationandCompletetheInstallation

thesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmightfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.

ReturntotheOpsManagerInstallationDashboardandclickApplyChangestocompletetheinstallationofRabbitMQforPCF.

OtherConfigurationTopics

ConnectingtoaHighlyAvailableRabbitMQClusterTheRabbitMQtile,allowsforahighlyavailableclusterthroughmultipleHAProxynodes.The hostnames , uris and hosts propertieshavebeenaddedandshouldbeusedinpreferenceovertheequivalentsingularproperties.Thesingularpropertiesaremaintainedforbackwardscompatibilityandalwayscontainthefirstvaluefromtheequivalentpluralproperty.Thesingularpropertieswilleventuallybedeprecated.

Forexample,withtwoHAProxyjobsdeployed,thefollowingpropertieswillbepresent:

"hostname":"10.0.0.41","hostnames":["10.0.0.41","10.0.0.51"]

Porttoprotocolmappings15672=Managementdashboard

5672=RabbitMQ

5671=RabbitMQSSL

1883=MQTT

8883=MQTTSSL

61613=STOMP

61614=STOMPSSL

15674=WebSTOMP

4567=RabbitMQServiceBroker

3457-3459=CFLoggregator

SecurityGroupsToenableaccesstotheRabbitMQtileservice,youmustensureyoursecuritygroupallowsaccesstotheHAProxyandRabbitMQServiceBrokerVMsconfiguredinyourdeployment.YoucanobtaintheIPaddressesforthesefromtheOpsManagerStatuspagefortheRabbitMQtile.EnsurethefollowingportsareenabledforthoseVMs:

3457-3459

ThefollowingisatemplateforconfiguringyourCloudFoundrysecuritygroups:[{"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"},{"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"}]

ApplicationSecurityGroupsToallowthisservicetohavenetworkaccess,youmustcreateApplicationSecurityGroups (ASGs).

ApplicationContainerNetworkConnectionsApplicationcontainersthatuseinstancesoftheRabbitMQservicerequirethefollowingoutboundnetworkconnections:

Destination Ports Protocol Reason

HAProxy IPs 5672 tcp ApplicationcontainersusingAMQP

HAProxy IPs 5671 tcp ApplicationcontainersusingAMQPoverSSL

HAProxy IPs 1883 tcp ApplicationcontainersusingMQTT

HAProxy IPs 8883 tcp ApplicationcontainersusingMQTToverSSL

HAProxy IPs 61613 tcp ApplicationcontainersusingSTOMP

HAProxy IPs 61614 tcp ApplicationcontainersusingSTOMPoverSSL

HAProxy IPs 61613 tcp ApplicationcontainersusingWebSTOMP

CreateanASGnamed rabbitmq-app-containers withtheaboveconfigurationandbindittoeither:

Theappropriatespace

The default-running ASGsetifyouwanttoprovideaccesstoallstartedapps.Thenrestartyourapps.

Ifyouareusinganexternalloadbalancer,orhavemorethanoneIPaddressforHAProxy,youmustalsocreateegressrulesforthese.Forexample:

[{"ports":"5671-5672","protocol":"tcp","destination":"10.10.10.10/32"}]

AssignedIPsRabbitMQforPCFdoesnotsupportchangingtheIPaddresseswhichhavebeenassignedtotheRabbitMQdeployments.Forexample,youcannotchangethesubnetintowhichtheRabbitMQclusterwasoriginallyprovisioned.Doingsocausesthedeploymenttofail.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

PreservingDynamicallyAssignedIPsYoucannotswitchfromdynamicallyassignedIPaddressestoadifferentsetofstaticIPaddresses.However,youcanconfigureOpsManagersothecurrentsetofdynamicallyassignedIPaddressesalwayscontinuetobeused.Thismightbeusefulwhenupgrading.

Todothis,followthesesteps:

1. GototheStatuspageintheRabbitMQtile.

2. TakenoteoftheIPaddressesfortheRabbitMQServerandHAProxyforRabbitMQjobs,intheordernodesappearintheUI.

3. GototheSettingspage,andclickNetworking.

4. EntertheIPaddressesyougotfromtheStatuspageasacomma-separatedlist.

Note:TheserviceisunusablewithoutApplicationSecurityGroups.

5. ClickSave.

RabbitMQServerSettingsthatCannotbeOverwrittenInallcases:

rabbit halt_on_upgrade_failure false

rabbitmq_mqtt subscription_ttl 1800000

log_levels [{connection,info}]

halt_on_upgrade_failure false

{rabbit, [ {collect_statistics_interval, 60000}] }

{rabbitmq_management, [ {rates_mode, none}] }

WhenSSLisenabled:

rabbit tcp_listeners []

rabbit ssl_listeners [5671]

rabbitmq_management listener [{port,15672},{ssl,false}]

rabbitmq_mqtt ssl_listeners [8883]

rabbitmq_stomp ssl_listeners [61614]

SmokeTestsRabbitMQforPCFrunsasetofsmoketestsduringinstallationtoconfirmsystemhealth.

SmokeTestStepsThesmoketestsperformthefollowingforeachavailableserviceplan:

1. Targetstheorg system andcreatesaspacetorunthetests.

2. DeploysaninstanceoftheCFRabbitMQExampleApp tothisspace

3. CreatesaRabbitMQserviceinstanceandbindsittotheCFRabbitMQExampleApp

4. ChecksthattheCFRabbitMQExampleAppcanwritetoandreadfromtheRabbitMQserviceinstance

5. Cleansupalldeployedapplicationandallitsservicebindings.Finally,thecfspaceisdeleted.

TroubleshootingIferrorsoccurwhilethesmoketestsrun,theyaresummarizedattheendoftheerrandlogoutput.Detailedlogscanbefoundwherethefailureoccurs.

Whenencounteringanerrorwhenrunningsmoketests,itcanbehelpfultosearchthelogforotherinstancesoftheerrorsummaryprintedattheendofthetests,forexample, FailedtotargetCloudFoundry .Lookoutfor TIP:... inthelogsnexttoanyerroroutputforfurthertroubleshootinghints.

Note:Smoketestsfailunlessyouenableglobaldefaultapplicationsecuritygroups(ASGs).YoucanenableglobaldefaultASGsbybindingtheASGtothe system orgwithoutspecifyingaspace.ToenableglobaldefaultASGs,use cfbind-running-security-

group.

MonitoringandKPIsforPre-ProvisionedRabbitMQforPCFThistopicexplainshowtomonitorthehealthofthepre-provisionedversionoftheRabbitMQforPivotalCloudFoundry(PCF)serviceusingthelogs,metrics,andKeyPerformanceIndicators(KPIs)generatedbyRabbitMQforPCFcomponentVMs.

Pre-provisionedRabbitMQforPCFcomponentsgeneratemanyofthesamemetrics astheon-demandRabbitMQservicecomponents.

SeeLoggingandMetrics forgeneralinformationaboutloggingandmetricsinPCF.

SettingupSyslogForwardingOperatorscanenablelogforwardingbyconfiguringanexternalsyslogendpointforRabbitMQcomponentlogmessages.Forinstructionsonsettingupsyslogforwarding,seeSyslog.

Ifsyslogforwardingisenabled,logentrieswithtimestampsareavailablelocallyin /var/log/messages .Logsareavailableunder /var/vcap/sys/log/ whethersyslogforwardingisenabledornot.

LoggingFormatsWithpre-provisionedRabbitMQforPCFloggingconfigured,threetypesofcomponentgeneratelogs:theRabbitMQmessageservernodes,theservicebroker,andHAProxy.IfyouhavemultipleserverorHAProxynodes,youcanidentifylogsfromindividualnodesbytheirindex,whichcorrespondstotheindexoftheRabbitMQVMinstancesdisplayedinOpsManager:

ThelogsforRabbitMQservernodesfollowtheformat [job=rabbitmq-server-partition-GUID index=X]

ThelogsforHAProxynodesfollowtheformat [job=rabbitmq-haproxy-partition-GUID index=X]

ThelogsfortheRabbitMQservicebrokerfollowtheformat [job=rabbitmq-broker-partition-GUID index=X]

RabbitMQandHAProxyserverslogatthe info levelandcaptureerrors,warnings,andinformationalmessages.

Forusersfamiliarwithdocumentationforpreviousversionsofthetile,thetagweusedtocallthe app_name isnowcalledthe program_name .

Thegenericlogformatisasfollows:

<PRI>TIMESTAMPIP_ADDRESSPROGRAM_NAME[job=NAMEindex=JOB_INDEXid=JOB_ID]MESSAGE

Therawlogslooksimilartothefollowing:

<7>2017-06-28T16:06:10.733560+00:0010.244.16.133vcap.agent[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]2017/06/2816:06:10CEF:0|CloudFoundry|BOSH|1|agent_api|ssh|1|duser=director.be5a66bb-a9b4-459f-a0d3-1fc5c9c3ed79.be148cc6-91ef-4eed-a788-237b0b8c63b7src=10.254.50.4spt=4222shost=5ae233e0-ecc5-4868-9ae0-f9767571251b<86>2017-06-28T16:06:16.704572+00:0010.244.16.133useradd[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]newgroup:name=bosh_ly0d2rbjr,GID=1003<86>2017-06-28T16:06:16.704663+00:0010.244.16.133useradd[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]newuser:name=bosh_ly0d2rbjr,UID=1001,GID=1003,home=/var/vcap/bosh_ssh/bosh_ly0d2rbjr,shell=/bin/bash<86>2017-06-28T16:06:16.736932+00:0010.244.16.133usermod[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]add'bosh_ly0d2rbjr'togroup'admin'<86>2017-06-28T16:06:16.736964+00:0010.244.16.133usermod[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]add'bosh_ly0d2rbjr'togroup'vcap'

LogssenttoexternalloggingtoolssuchasPapertrailmaybepresentedinadifferentformat.

Thefollowingtabledescribestheloggingtagsusedinthistemplate:

Tag Description

PRI Thisisavaluewhichinfuturewillbeusedtodescribetheseverityofthelogmessageandwhichfacilityitcamefrom.

TIMESTAMPThisisthetimestampofwhenthelogisforwarded,forexample, 2016-08-24T05:14:15.000003Z .Thetimestampvalueistypicallyslightlyafterwhenthelogmessagewasgenerated.

IP_ADDRESS TheinternalIPaddressofserveronwhichthelogmessageoriginated

PROGRAM_NAMEProcessnameoftheprogramthegeneratedthemessage.Sameas app_name beforev1.9.0.Formoreinformationaboutprogramname,seeRabbitMQProgramNamesbelow.

NAME TheBOSHinstancegroupname(forexample, rabbitmq_server )

JOB_INDEX BOSHjobindex.Usedtodistinguishbetweenmultipleinstancesofthesamejob.

JOB_IDBOSHVMGUID.ThisisdistinctfromtheCIDdisplayedintheOpsManagerStatustab,whichcorrespondstotheVMIDassignedbytheinfrastructureprovider.

MESSAGE Thelogmessagethatappears

RabbitMQProgramNames

ProgramName Description

rabbitmq_server_cluster_check ChecksthattheRabbitMQclusterishealthy.Runsaftereverydeploy.

rabbitmq_server_node_check ChecksthattheRabbitMQnodeishealthy.Runsaftereverydeploy.

rabbitmq_route_registrar_stderrRegisterstherouteforthemanagementAPIwiththeGorouterinyourPivotalApplicationService(PAS)orElasticRuntimedeployment.

rabbitmq_route_registrar_stdout RegisterstherouteforthemanagementAPIwiththeGorouterinyourPASorElasticRuntimedeployment.

rabbitmq_server TheErlangVMandRabbitMQapps.Logsmayspanmultiplelines.

rabbitmq_server_drain ShutsdowntheErlangVMandRabbitMQapps.RunsaspartoftheBOSHlifecycle.

rabbitmq_server_http_api_access AccesstotheRabbitMQmanagementUI.

rabbitmq_server_init StartstheErlangVMandRabbitMQ.

rabbitmq_server_post_deploy_stderr Runsthenodecheckandclustercheck.Runsaftereverydeploy.

rabbitmq_server_post_deploy_stdout Runsthenodecheckandclustercheck.Runsaftereverydeploy.

rabbitmq_server_pre_start Runsbeforetherabbitmq-serverjobisstarted.

rabbitmq_server_sasl Supervisor,progress,andcrashreportingfortheErlangVMandRabbitMQapps.

rabbitmq_server_shutdown_stderr StopstheRabbitMQappandErlangVM.

rabbitmq_server_shutdown_stdout StopstheRabbitMQappandErlangVM.

rabbitmq_server_startup_stderr StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.

rabbitmq_server_startup_stdout StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.

rabbitmq_server_upgrade ShutsdownErlangVMandRabbitMQappifrequiredduringanupgrade.

MetricsMetricsareregularly-generatedlogmessagesthatreportmeasuredcomponentstates.Themetricspollingintervaldefaultsto30seconds.ThemetricspollingintervalisaconfigurationoptionontheRabbitMQtile(Settings>RabbitMQ).Theintervalsettingappliestoallcomponentsdeployedbythetile.

Metricsarelong,singlelinesoftextthatfollowtheformat:

origin:"p-rabbitmq"eventType:ValueMetrictimestamp:1441188462382091652deployment:"cf-rabbitmq"job:"cf-rabbitmq-node"index:"0"ip:"10.244.3.46"valueMetric:<name:"/p-rabbitmq/rabbitmq/system/memory"value:1024unit:"MB">

PartitionIndicatorAnewmetrichasbeenintroducedtohelptoidentifynetworkpartitions.Essentiallyitexposeshowmanynodeseachnodeknows.Whenanodeisinpartitiontheonlynodethatitrecognizesisitselfandthatisagoodindicationthatthatnodemightbeinapartition.

Anexampleofthatmetricsis:

origin:"p-rabbitmq"eventType:ValueMetrictimestamp:1441188462382091652deployment:"cf-rabbitmq"job:"cf-rabbitmq-node"index:"0"ip:"10.244.3.46"valueMetric:<name:"/p-rabbitmq/rabbitmq/erlang/reachable_nodes"value:3unit:"count">

Monitorscanbecreatedtoemitalertsincaseaclusterseemstobeinapartition.Ametricsisemittedforeachnodeinthecluster.Forexample:inathree-nodeclusteramonitorcanexpecttohaveatotalof9(nine)sinceeachnodeisexpectedtoemit3(2reachablenodesanditself).Otherwise,analertcanbesenttotheteam.

RecoveringfromanetworkpartitionPleaserefertotheoficialRabbitMQguidetounderstandhowtorecoverfromanetworkpartition:https://www.rabbitmq.com/partitions.html

KeyPerformanceIndicatorsKeyPerformanceIndicators(KPIs)forRabbitMQforPCFaremetricsthatoperatorsfindmostusefulformonitoringtheirRabbitMQservicetoensuresmoothoperation.KPIsarehigh-signal-valuemetricsthatcanindicateemergingissues.KPIscanberawcomponentmetricsorderivedmetricsgeneratedbyapplyingformulastorawmetrics.

PivotalprovidesthefollowingKPIsasgeneralalertingandresponseguidancefortypicalRabbitMQforPCFinstallations.Pivotalrecommendsthatoperatorscontinuetofine-tunethealertmeasurestotheirinstallationbyobservinghistoricaltrends.Pivotalalsorecommendsthatoperatorsexpandbeyondthisguidanceandcreatenew,installation-specificmonitoringmetrics,thresholds,andalertsbasedonlearningfromtheirowninstallations.

ForalistofallRabbitMQforPCFrawcomponentmetrics,seeComponentMetricsReference.

ComponentHeartbeatsKeyRabbitMQforPCFcomponentsperiodicallyemitheartbeatmetrics:theRabbitMQservernodes,HAProxynodes,andtheServiceBroker.TheheartbeatsareBooleanmetrics,where 1 meansthesystemisavailable,and 0 ortheabsenceofaheartbeatmetricmeanstheserviceisnotrespondingandshouldbeinvestigated.

ServiceBrokerHeartbeat

p-rabbitmq.service_broker.heartbeat

Description

RabbitMQServiceBroker is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:IftheServiceBrokerdoesnotemitheartbeats,thisindicatesthatitisoffline.TheServiceBrokerisrequiredtocreate,update,anddeleteserviceinstances,whicharecriticalfordependenttilessuchasSpringCloudServicesandSpringCloudDataFlow.

Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)

Recommendedmeasurement Averageoverlast5minutes

RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1

RecommendedresponseChecktheRabbitMQServiceBrokerlogsforerrors.YoucanfindthisVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningthefollowingcommand:bosh -d service-instance_GUID vms

HAProxyHeartbeat

p-rabbitmq.haproxy.heartbeat

Description

RabbitMQHAProxy is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:IftheHAProxydoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireHAProxy.

ChecktheRabbitMQHAProxylogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwith

Recommendedresponse BOSHandrunningthefollowingcommand,whichlists HAProxy_GUID :bosh -d service-instance_GUID vms

ServerHeartbeat

p-rabbitmq.rabbitmq.heartbeat

Description

RabbitMQServer is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:Iftheserverdoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireRabbitMQServer.

RecommendedresponseChecktheRabbitMQServerlogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommands,whichlists rabbitmq :bosh -d service-instance_GUID vms

RabbitMQServerKPIsThefollowingKPIsfromtheRabbitMQservercomponent:

FileDescriptors

p-rabbitmq.rabbitmq.system.file_descriptors

Description

Filedescriptorsconsumed.

Use:Ifthenumberoffiledescriptorsconsumedbecomestoolarge,theVMmaylosetheabilitytoperformdiskIO,whichcancausedataloss.

Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)

RecommendedalertthresholdsYellowwarning:>250000Redcritical:>280000

Recommendedresponse

Thedefault ulimit forRabbitMQforPCFis300000.Ifthismetricismetorexceededforanextendedperiodoftime,consideroneofthefollowingactions:

ScalingtherabbitnodesinthetileResourceConfigpane.

Reducetheloadontheserver

ErlangProcesses

p-rabbitmq.rabbitmq.erlang.erlang_processes

Erlang processesconsumedbyRabbitMQ,whichrunsonanErlangVM.

Note:Thisassumesnon-persistentmessagesarehandledbyretriesorsomeotherlogicbytheproducers.

DescriptionUse:Thisisthekeyindicatoroftheprocessingcapabilityofanode.

Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)

RecommendedresponseThedefaultErlangprocesslimitinRabbitMQforPCFv1.6andlateris1,048,816.Ifthismetricmeetsorexceedstherecommendedthresholdsforextendedperiodsoftime,considerscalingtheRabbitMQnodesinthetileResourceConfigpane.

BOSHSystemHealthMetricsTheBOSHlayerthatunderliesPCFgenerates healthmonitor metricsforallVMsinthedeployment.AsofPCFv2.0,thesemetricsareincludedintheLoggregatorFirehosebydefault.Formoreinformation,seeBOSHSystemMetricsAvailableinLoggregatorFirehose inPivotalApplicationService(PAS)ReleaseNotes.

AllBOSH-deployedcomponentsgeneratethesystemhealthmetricsbelow.ThesecomponentmetricsarefromRabbitMQforPCFcomponents,andserveasKPIsfortheRabbitMQforPCFservice.

system.mem.percent

Description

RAMbeingconsumedbythe p-rabbitmq VM.

Use:RabbitMQisconsideredtobeinagoodstatewhenithaslittleornomessages.Inotherwords,“anemptyrabbitisahappyrabbit.”Alertingonthismetriccanindicatethattherearetoofewconsumersorappsthatreadmessagesfromthequeue.

HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsRAMforthepasttenminutes.

Origin:BOSHHMType:percentFrequency:30s(default),10s(configurableminimum)

Recommendedresponse Addmoreconsumerstodrainthequeueasfastaspossible.

system.cpu.percent

Description

CPUbeingconsumedbythe p-rabbitmq VM.

Use:AnodethatexperiencescontextswitchingorhighCPUusagewillbecomeunresponsive.Thisalsoaffectstheabilityofthenodetoreportmetrics.

HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.

Recommendedalertthresholds Yellowwarning:>60Redcritical:>75

Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.

EphemeralDisk

system.disk.percent

Description

EphemeralDiskbeingconsumedbythe p-rabbitmq VM.

Use:Ifsystemdiskfillsup,therearetoofewconsumers.

PersistentDisk

persistent.disk.percent

Description

PersistentDiskbeingconsumedbythe p-rabbitmq VM.

Use:Ifsystemdiskfillsup,therearetoofewconsumers.

ComponentMetricReferenceRabbitMQforPCFcomponentVMsemitthefollowingrawmetrics.Thefullnameofthemetricfollowstheformat: /p-rabbitmq/COMPONENT/METRIC-NAME

RabbitMQServerMetricsRabbitMQforPCFmessageservercomponentsemitthefollowingmetrics.

FullName Unit Description

/p-rabbitmq.rabbitmq.heartbeat boolean IndicateswhethertheRabbitMQserverisavailableandabletorespondtorequests

/p-rabbitmq/rabbitmq/erlang/erlang_processes

count ThenumberofErlangprocesses

/p-rabbitmq/rabbitmq/system/memory MB ThememoryinMBusedbythenode

/p-rabbitmq/rabbitmq/system/mem_alarm boolean Indicatesifthememoryalarmwentoff

/p-rabbitmq/rabbitmq/system/disk_free_alarm boolean Indicatesifthediskfreealarmwentoff

/p-rabbitmq/rabbitmq/system/disk_free MB Thediskspaceavailableonthenode

/p-rabbitmq/rabbitmq/connections/count count Thetotalnumberofconnectionstothenode

/p-rabbitmq/rabbitmq/consumers/count count Thetotalnumberofconsumersregisteredinthenode

/p-rabbitmq/rabbitmq/messages/delivered count Thetotalnumberofmessageswiththestatus deliver_get onthenode

/p-rabbitmq/rabbitmq/messages/delivered_noack

count Thenumberofmessageswiththestatus deliver_noack onthenode

/p-rabbitmq/rabbitmq/messages/delivered_rate

rateTheratepersecondatwhichmessagesarebeingdeliveredtoconsumersorclientsonthenode

/p-rabbitmq/rabbitmq/messages/published count Thetotalnumberofmessageswiththestatus publish onthenode

/p-rabbitmq/rabbitmq/messages/published_rate

rate Theratepersecondatwhichmessagesarebeingpublishedbythenode

/p-rabbitmq/rabbitmq/messages/redelivered count Thetotalnumberofmessageswiththestatus redeliver onthenode

/p-rabbitmq/rabbitmq/messages/redelivered_rate rate

Theratepersecondatwhichmessagesaregettingthestatus redeliver onthenode

/p-rabbitmq/rabbitmq/messages/get_no_ack count Thenumberofmessageswiththestatus get_no_ack onthenode

/p-rabbitmq/rabbitmq/messages/get_no_ack_rate

rate Theratepersecondatwhichmessagesgetthestatus get_no_ack onthenode

/p-rabbitmq/rabbitmq/messages/pending count Thenumberofmessageswiththestatus messages_unacknowledged onthenode

/p-rabbitmq/rabbitmq/messages/depth countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready onthenode

/p-rabbitmq/rabbitmq/system/file_descriptors

count Thenumberofopenfiledescriptorsonthenode

/p-rabbitmq/rabbitmq/exchanges/count count Thetotalnumberofexchangesonthenode

/p-rabbitmq/rabbitmq/messages/available count Thetotalnumberofmessageswiththestatus messages_ready onthenode

/p-rabbitmq/rabbitmq/queues/count count Thenumberofqueuesonthenode

/p-rabbitmq/rabbitmq/channels/count count Thenumberofchannelsonthenode

/p-rabbitmq/rabbitmq/vhosts/count count Thenumberofvhosts

/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/consumers

count Thenumberofconsumerspervirtualhostperqueue

/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/depth

countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready pervirtualhostperqueue

HAProxyMetricsRabbitMQforPCFHAProxycomponentsemitthefollowingmetrics.

NameSpace Unit Description

/p-rabbitmq.haproxy.heartbeat booleanIndicateswhethertheRabbitMQHAProxycomponentisavailableandabletorespondtorequests

/p-rabbitmq/haproxy/health/connections

count Thetotalnumberofconcurrentfront-endconnectionstotheserver

/p-rabbitmq/haproxy/backend/qsize/amqp

size ThetotalsizeoftheAMQPqueueontheserver

/p-rabbitmq/haproxy/backend/retries/amqp count ThenumberofAMQPretriestotheserver

/p-rabbitmq/haproxy/backend/ctime/amqp

time ThetotaltimetoestablishtheTCPAMQPconnectiontotheserver

SettingLimitsforOn-DemandServiceInstancesOn-demandprovisioningisintendedtoaccelerateappdevelopmentbyeliminatingtheneedfordevelopmentteamstorequestandwaitforoperatorstocreateaserviceinstance.However,tocontrolcosts,operationsteamsandadministratorsmustensureresponsibleuseofresources.

Thereareseveralwaystocontroltheprovisioningofon-demandserviceinstancesbysettingvariousquotasattheselevels:

Global

Afteryousetquotas,youcan:

ViewCurrentOrgandSpace-levelQuotas

MonitorQuotaUseandServiceInstanceCount

CalculateResourceCostsforOn-DemandPlans

CreateGlobal-levelQuotasEachPivotalCloudFoundry(PCF)servicehasaseparateservicebroker.Aglobalquotaattheservicelevelsetsthemaximumnumberofserviceinstancesthatcanbecreatedbyagivenservicebroker.Ifaservicehasmorethanoneplan,thenthenumberofserviceinstancesforallplanscombinedcannotexceedtheglobalquotafortheservice.

TheoperatorsetsaglobalquotaforeachPCFserviceindependently.Forexample,ifyouhaveRedisforPCFandRabbitMQforPCF,youmustsetaseparateglobalservicequotaforeachofthem.

Whentheglobalquotaisreachedforaservice,nomoreinstancesofthatservicecanbecreatedunlessthequotaisincreased,orsomeinstancesofthatservicearedeleted.

CreatePlan-levelQuotasAservicemayofferoneormoreplans.Youcansetaseparatequotaperplansothatinstancesofthatplancannotexceedtheplanquota.Foraservicewithmultipleplans,thetotalnumberofinstancescreatedforallplanscombinedcannotexceedtheglobalquotafortheservice.

Whentheplanquotaisreached,nomoreinstancesofthatplancanbecreatedunlesstheplanquotaisincreasedorsomeinstancesofthatplanaredeleted.

CreateandSetOrg-levelQuotasAnorg-levelquotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesanorganizationcancreatewithinPCF.Forexample,ifyousetyourorg-levelquotato100,developerscancreateupto100serviceinstancesinthatorgusinganycombinationofPCFservices.

Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedintheorgunlessthequotaisincreasedorsomeserviceinstancesaredeleted.

Tocreateandsetanorg-levelquota,dothefollowing:

1. Runthiscommandtocreateaquotaforserviceinstancesattheorglevel:

cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans

Where:

QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg

Forexample:

cfcreate-quotamyquota-m1024mb-i16gb-r30-s50--allow-paid-service-plans

2. Associatethequotayoucreatedabovewithaspecificorgbyrunningthefollowingcommand:

cf set-quota ORG-NAME QUOTA-NAME

Forexample:

cfset-quotadev_orgmyquota

Formoreinformationonmanagingorg-levelquotas,seeCreatingandModifyingQuotaPlans .

CreateandSetSpace-levelQuotasAspace-levelservicequotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesthatcanbecreatedwithinagivenspaceinPCF.Forexample,ifyousetyourspace-levelquotato100,developerscancreateupto100serviceinstancesinthatspaceusinganycombinationofPCFservices.

Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedinthespaceunlessthequotaisupdatedorsomeserviceinstancesaredeleted.

Tocreateandsetaspace-levelquota,dothefollowing:

1. Runthefollowingcommandtocreatethequota:

cf create-space-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans

Where:

QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg

Forexample:

cfcreate-space-quotamyspacequota-m1024mb-i16gb-r30-s50--allow-paid-service-plans

2. Associatethequotayoucreatedabovewithaspecificspacebyrunningthefollowingcommand:

cf set-space-quota SPACE-NAME QUOTA-NAME

Forexample:

cfset-space-quotamyspacemyspacequota

Formoreinformationonmanagingspace-levelquotas,seeCreatingandModifyingQuotaPlans .

ViewCurrentOrgandSpace-levelQuotasTovieworgquotas,runthefollowingcommand.

cforgORG-NAME

Toviewspacequotas,runthefollowingcommand:

cfspaceSPACE-NAME

Formoreinformationonmanagingorgandspace-levelquotas,seetheCreatingandModifyingQuotaPlans .

MonitorQuotaUseandServiceInstanceCountService-levelandplan-levelquotause,andtotalnumberofserviceinstances,areavailablethroughtheon-demandbrokermetricsemittedtoLoggregator.Thesemetricsarelistedbelow:

MetricName Description

on-demand-broker/SERVICE-NAME/quota_remaining Quotaremainingforallinstancesacrossallplans

on-demand-broker/SERVICE-NAME/PLAN-NAME/quota_remaining

Quotaremainingforaspecificplan

on-demand-broker/SERVICE-NAME/total_instances Totalinstancescreatedacrossallplans

on-demand-broker/SERVICE-NAME/PLAN-NAME/total_instances

Totalinstancescreatedforaspecificplan

CalculateResourceCostsforOn-DemandPlansOn-demandplansusededicatedVMs,disks,andvariousotherresourcesfromanIaaS,suchasAWS.Tocalculatemaximumresourcecostforplansindividuallyorcombined,youmultiplythequotabythecostoftheresourcesselectedintheplanconfiguration(s).ThespecificcostsdependonyourIaaS.

TheimagebelowshowsanexamplethatincludestheVMtypeandpersistentdiskselectedfortheserverVMs,aswellasthequotaforthisplan.

Note:Quotametricsarenotemittedifnoquotahasbeenset.

Important:Althoughoperatorscanlimiton-demandinstanceswithplanquotasandaglobalquota,asdescribedintheabovetopics,IaaSresourceusagestillvariesbasedonthenumberofon-demandinstancesprovisioned.

CalculateMaximumResourceCostPerOn-DemandPlanTocalculatethemaximumcostofVMsandpersistentdiskforeachplan,dothefollowingcalculation:

planquotaxcostofselectedresources

Forexample,ifyouselectedtheoptionsintheaboveimage,youhaveselectedaVMtypemicroandapersistentdisktype20GB,andtheplanquotais15.TheVMandpersistentdisktypeshaveanassociatedcostfortheIaaSyouareusing.Therefore,tocalculatethemaximumcostofresourcesforthisplan,multiplythecostoftheresourcesselectedbytheplanquota:

(15xcostofmicroVMtype)+(15xcostof20GBpersistentdisk)=maxcostperplan

CalculateMaximumResourceCostforAllOn-DemandPlansTocalculatethemaximumcostforallplanscombined,addtogetherthemaximumcostsforeachplan.Thisassumesthatthesumofyourindividualplanquotasislessthantheglobalquota.

Hereisanexample:

(plan1quotaxplan1resourcecost)+(plan2quotaxplan2resourcecost)=maxcostforallplans

CalculateActualResourceCostofallOn-DemandPlansTocalculatethecurrentactualresourcecostacrossallyouron-demandplans:

1. Findthenumberofinstancescurrentlyprovisionedforeachactiveplanbylookingatthe total_instance metricforthatplan.

2. Multiplythe total_instance countforeachplanbythatplan’sresourcecosts.Recordthecostsforeachplan.

3. AddupthecostsnotedinStep2togetyourtotalcurrentresourcecosts.

Forexample:

(plan1total_instancesxplan1resourcecost)+(plan2total_instancesxplan2resourcecost)=currentcostforallplans

ControllingAccesstoServicePlansbyOrgIfyouwanttorestrictaccesstoaserviceplantoaspecificorg,followtheinstructionsbelow.

Youcanalsolimitthenumberofserviceinstancesbysettingquotas—forinstructions,seeSettingLimitsforOn-DemandInstances.

ChangeAccesstoServicePlans

Torestrictaccesstoaplanforaspecificorg,runthiscommand:

cfenable-service-accessp.rabbitmq-pPLAN_NAME-oORG_NAME

Forexample:

$cfenable-service-accessp.rabbitmq-pmy-cluster-plan-omy-dev-org

Formoreinformationabouttheabovecommand,seeAccessControl .

Note:Iftheplanyouarerestrictingiscurrentlyenabledforallorgs,youmustfirstDisableServiceAccessfortheplan,thengrantaccesstotheplantospecificorgs.UsetheCFServiceAccessfieldtodisableaccessintheserviceplanconfiguration.

IsolatingClusterswiththeRabbitMQforPCFReplicator

OverviewRabbitMQforPCFReplicatorisatoolthatallowsyoutoinstallmultipleRabbitMQforPivotalCloudFoundry(PCF)tilesinasingleOpsManagerenvironment.Thisletsyourunmultiplepre-provisionedRabbitMQclustersthatareisolatedfromeachother.

Forexample,youmaywanttoisolatetheclusterservingSpringCloudServices(SCS)fromtheclusterservingappsintheMarketplace.Oryoumaywanttogiveacertainteamtheirowndedicated,pre-provisionedclusterthatyoumanageforthem.Forinformationonhowtoaccomplishthesescenarios,seeCommonUseCases.

CommonUseCasesTheimagebelowillustrateshowtoisolateSCSonRabbitMQforPCFfromclusteredandsinglenodeserviceinstancesdedicatedtodifferentteams.Inthisusecase:

TheunreplicatedRabbitMQforPCFtiledeploystwotypesofservices:

Apre-provisionedserviceisusedasabackingserviceforSCSAnon-demandserviceisusedtocreatethreecompletelyisolatedsinglenodeserviceinstances

Tworeplicatilesareusedtocreatetwodedicatedpre-provisionedclusters,witheachonededicatedtoaspecificteam.

ThesescenariosareexplainedbelowinRunningSCSonaDedicatedRabbitMQClusterandProvidingaPre-ProvisionedDedicatedCluster.

RunningSCSonaDedicatedRabbitMQClusterReplicaRabbitMQtilescannotbeusedtoprovideabackingserviceforSpringCloudServices(SCS)becauseSCSexpectsthattheserviceiscalledp-rabbitmq .Therefore,ifyouwanttoisolatetheRabbitMQclusterthatisusedbySCSfromothertenants,youcanreservetheunreplicatedRabbitMQforPCFtileforSCS,asshowninthediagrambelow.YoucanthenaddreplicaRabbitMQclustersforusebyappsintheMarketplace.

PivotalrecommendsthatyouusetheunreplicatedRabbitMQforPCFtilesolelyforSCStoavoidcontentionbetweenappsusingSCS,andappsusingRabbitMQforPCFintheMarketplace.

ToreservetheunreplicatedtileforSCS,turnofftheBrokerRegistrarerrandtopreventthebrokerfrombeingexposedintheMarketplace.Formoreinformation,seeErrands.

ToofferRabbitMQasacloudmessagingserviceintheMarketplace,createoneorseveralreplicas,installtheminOpsManager,andeitherallowthebrokerregistrarerrandtorun,orregistertheservicemanuallyusingCF .

ProvidingaPre-ProvisionedDedicatedClusterToreserveaRabbitMQclusterforusebyaspecificteam,disabletheBrokerRegistrarerrandinOpsManager.ThispreventsserviceregistrationintheMarketplace.Formoreinformation,seeErrands.

Afteryoudeploythetile,manuallyexposetheservicebrokertoyourdesiredorgsandspaces.Forinstructions,seeRegisteraBroker .

UsingReplicasWhileOfferingtheOn-DemandRabbitMQServiceTheOn-Demandserviceisnotofferedinreplicatiles,sincethepurposeofthereplicatoristocreateadditionalpre-provisionedclusters.Ifyouwishtoofferon-demandserviceplans,usetheunreplicatedRabbitMQforPCFtileasshowninthediagramabove.

Blue-GreenUpgrades(Advanced)Inordertodoblue-greenstyleupgradestominimizedowntime,youcanstandupanewclusterandmigratedataandusersovertothenewclusteroveraperiodoftime.SpeakwithyourPlatformArchitectabouthowtoenablethisworkflow.

GeneratingReplicaTilesThistopicdescribeshowtoinstallthereplicatorandgeneratereplicatilesofRabbitMQforPCF.

PrerequisitesRabbitMQforPCFv1.8.xorv1.9.x

2.5GBoffreediskspace

DownloadtheReplicatorTheRabbitMQforPCFReplicatoriscurrentlyavailablefromPivotalNetwork .Searchforanddownloadthisarchive,andthenruntheenclosedbinary.

GenerateReplicaTilesThefollowingisthesyntaxforgeneratingareplicatile:

./rabbitmq-replicator-darwin\--nameYOUR_DESIRED_TILE_NAME\--pathPATH_TO_TILE\--outputDESIRED_FILE_NAME.pivotal

Thefollowingaretheparametersexpectedintheabovesyntax:

Parameter Description

nameThedesireduniqueidentifierforthereplicatile,whichisusedtogeneratemanifests,deploymentnames,andMarketplacenamefortheservice.Onlyalphanumericcharactersand - areacceptedbythetool.

path Thelocationoftheoriginal,unreplicated,RabbitMQforPCFsourcetilethatyoudownloaded

output Thedesiredfilenameandpathforthereplicatile

NamingConventionsinOriginalandReplicaTilesThetablebelowshowsthenamingconventionsforvariouscomponentsrelatedtotheoriginalRabbitMQforPCFtileandtothereplicatile.

Forthepurposesofthisexample,assumethatwhenyougenerateareplicatileasshownabove,inthe name fieldyouprovidethestring finance\ .Thentheattributesfortheoriginalandreplicatilesareasfollows:

Component NamewithOriginalTile NamewithReplicaTile

Brokername p-rabbitmq p-rabbitmq-finance

BrokerURL pivotal-rabbitmq-broker.YOUR_CF_DOMAIN pivotal-rabbitmq-broker-finance.YOUR_CF_DOMAIN

Servicename p-rabbitmq p-rabbitmq-finance

URLfortheRabbitMQManagementUIDashboard pivotal-rabbitmq.YOUR_CF_DOMAIN pivotal-rabbitmq-finance.YOUR_CF_DOMAIN

TiledisplaynameinOpsManager RabbitMQ RabbitMQ(finance)

TilenameusedinternallybyOpsManager p-rabbitmq p-rabbitmq-finance

Metrics/LoggingOrigin p-rabbitmq p-rabbitmq-finance

InstallingReplicaTilesAfteryouhavegeneratedareplicatile,youcanuploadittoOpsManagerasyouwouldanyothertile.Afteryouhaveuploadedit,followtheinstructionsforInstallingandConfiguringRabbitMQforPCFasaPre-ProvisionedService.TheOn-Demandserviceisnotofferedonreplicatiles.

LimitingAccesstoReplicaTilestoSpecificOrgsWhenyoureplicateRabbitMQforPCF,thereplicatilehastheBrokerRegistrarerrandsettoOnbydefault.ThisfieldappearsintheErrandstabinthetile:

Withanytile,iftheBrokerRegistrarerrandissettoOn,itrunsautomaticallywhenyoufinishinstallingthetileandcausesthetiletobeavailabletoallCForgs.

Ifyouwanttolimitaccesstothetiletoaspecificorg,followthesesteps:

1. SetthebrokerregistrarerrandtoOff,andapplyyourchanges.

2. ManuallyregisterthetilewithaspecificCForgusingthefollowingcommand.Seetheabovetablefor BROKER_NAME , BROKER_URL ,andSERVICE_NAME :

cfcreate-service-brokerBROKER_NAMEBROKER_USERNAMEBROKER_PASSWORDBROKER_URL

3. Togiveaccesstotheorg,usethefollowingcommandandrepeatforeachadditionalorg:

cfenable-service-accessSERVICE_NAME-oORG_NAME

UpgradingReplicaTilesYoucanupgradereplicatileslikeregulartileswithoneimportantdifference.YoumustgenerateareplicaofthenewerversionoftheRabbitMQforPCFtile,usingthereplicator,andgivethenewreplicathesame name astheexistingreplica.Thisisshownintheexampleworkflowbelow.

ExampleofanIn-PlaceUpgradeofaReplicaSupposeyouusedthereplicatortogenerateareplicaofv1oftheRabbitMQforPCFtile,withthe name trading-team,andyouinstalleditinOpsManager.Hereisthesamplereplicatorcommandyouusedfortheinitialinstallation:

./rabbitmq-replicator-darwin\--nametrading-team\--path/download/p-rabbitmq-v1.pivotal\--output/output/p-rabbitmq-v1-trading-team.pivotal

Toupgradetov2,followthesesteps:

1. DownloadthenewRabbitMQforPCFv2.

2. Runthereplicatorcommand,usingthepathtothenewRabbitMQforPCFv2tile,andsupplythesame name ,trading-team,asshownbelow.

./rabbitmq-replicator-darwin\--nametrading-team\--path/download/p-rabbitmq-v2.pivotal\--output/output/p-rabbitmq-v2-trading-team.pivotal

3. Afteryouhavethereplicatilep-rabbitmq-v2-trading-team.pivotal,uploadittoOpsManager.Thisupgradesthev1replicatileinplace.

Youcanthenproceedwithupgradingthecluster.

Ifyouwanttodoablue-greenstyleupgrade,seeBlue-GreenUpgrades.

LimitationsTheOn-Demandserviceisnotofferedonreplicatiles.

SettingDefaultPoliciesfortheRabbitMQService

UnderstandingaRabbitMQPolicyYoucansetadefaultqueueandanexchangepolicyintheRabbitMQforPivotalCloudFoundry(PCF)tiletobeappliedtotheRabbitMQcluster.Afteryoudeploythetile,PivotalrecommendsthatyouusetheRabbitMQManagementInterfacetomakeconfigurationchanges.

FormoreinformationaboutRabbitMQpolicies,seetheRabbitMQdocumentation .

RulesforPoliciesSetintheTileThefollowingrulesapplytopolicessetthroughtheRabbitMQforPCFtile:

Anewpolicy,oranupdatetoapolicy,onlyappliestonewinstances(vhosts).Existinginstancesarenotaffectedbythepolicy.

ThepolicycanonlybedeletedmanuallyfromtheRabbitMQnodes.

PoliciescanbeaddeddynamicallyusingtheRabbitMQManagementInterface.

Itisnotpossibletousepatternmatchingwithpolicies.Policieswillbeappliedtoallqueuesandexchanges.Forgranularpolicysettings,PivotalrecommendsusingtheRabbitMQManagementUI.Seta prioritynumber lowerthan 50 ,thedefault prioritynumber

appliedthroughtheOpsManagerconfiguration.

AnExamplePolicy:MirroronTwoNodesHereisanexamplepolicythatensuresmessagesaremirroredontwonodes:

{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}

Operatorsshouldconsidersomeoftheperformanceimplicationsofmakingqueuesandexchangeshighlyavailable.Formoreinformationabouthighlyavailablequeues,seetheRabbitMQdocumentation .

BestPracticeforSyncingQueuesWhenaqueuesyncsallitsmessages,theyareloadedintomemory.Whenqueuesaresyncing,theycanuseasmuchmemoryasthetotalsizeofallmessages.Thisappliestobothnodes—thenodewherethequeueleaderruns(fromnode)andthenodewherethequeuefollowerruns(tonode),butonlyappliestonewlycreatedqueuefollowers.

Thisbehaviorisespeciallyrelevantwhenanychangeaffectsthedeployment,forexample:stemcellupdates,deploymentconfigurationchanges,andnetworkchanges.Verifythatyouhaveenoughmemoryanddiskavailabletosupportallmessages.

Forexample:

Thereare5GBofmessagesinamirroredqueuethatissettoautomaticsync.Whenthisqueueneedstosync,thenodewherethequeueleaderrunscanuseupto5GBofextramemory.Thesameappliestothenodewherethenewqueuefolloweriscreated.

SettingorChangingthePolicyTosettheRabbitMQpolicy,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQforPCFtileandthenclickPre-ProvisionedRabbitMQPolicy.

2. SelectEnablecustompolicyonnewinstances.

3. InthePolicyfornewinstancesfield,pastethepolicy.ThepolicymustbevalidJSONandshouldmeetvalidRabbitMQpolicycriteria.

4. IntheSelectthenetworkpartitionbehavioroftheRabbitMQcluster,choosethedesiredbehavior:pause_minorityorautoheal.

FormoreinformationabouttheseoptionsandonRabbitMQclustersandnetworkpartitions,seetheRabbitMQdocumentation .

Forproductionpurposes,PivotalrecommendsthatcustomershaveatleastthreeRabbitMQservernodesandtwoHAProxiesspreadacrosslowlatencyavailabilityzones.

ViewingPoliciesintheRabbitMQManagementDashboardYoucanviewRabbitMQpoliciesintheRabbitMQManagementDashboard,shownbelow.TheexamplepolicyenteredintheRabbitMQforPCFtileaboveisappliedtoallqueuesandgivenaPriorityof50.Thisallowsyoutooverrideitbydefininganotherpolicywithahigherpriority.

IntheQueuessectionshownbelow,youcanseethatanynewqueuescreatedhavethepolicyautomaticallyapplied.

Note:Nopolicyvalidationoccursduringthedeployment,anderrorscancausethedeploymenttofailorpoliciestobeappliedincorrectly.

Note:DeveloperscanobtaintheURLofthepolicyfrom VCAP_SERVICES forappdevelopers.

UsingtheRabbitMQManagementDashboard

RabbitMQManagementDashboard

AdminUserTogainaccesstothemanagementdashboardasthe admin user,visit http://pivotal-rabbitmq.SYS-DOMAIN .Toretrieveyoursystemdomain,navigatetoyourPivotalApplicationService(PAS)orElasticRuntimetileandlocatetheSystemDomainfieldoftheDomainssection.

TheusernameandpasswordistheusernameandpasswordyouprovidedintheRabbitMQconfigurationinOpsManager,whichisalsoshownintheCredentialstab.

ApplicationDeveloperUsersofCloudFoundrywhocreateinstancesviatheAppsManagerorthecfCLIalsogetaccesstotheManagementUI.Thisisdoneusingcredentialsthatprovideaccessonlytotheirparticularvhost.

TheappropriateURLisaccessibleviatheManagebuttonwithintheAppsManager.

Oritisalsoinjectedintothe VCAP_SERVICES environmentvariableprovidedtoappsrunningonCloudFoundry.ThiscanalsobefoundviatheCLIusingcfenv<yourappname>

LoggingATCPSyslogendpointcanbeconfiguredinOpsManager.LogsarecurrentlyonlyforwardedfortheRabbitMQcluster.

RabbitMQCLIIfyouwishtoruncommandssuchas rabbitmqctl thenyouhavetwooptions:

SSHintooneofthemachinesrunningtherabbitmq-server.IPscanbefoundfromtheStatustabandaccesscredentialsfromtheCredentialstabwithintheRabbitMQcomponentoftheinstaller.FromthereyouneedtobringRabbitMQandErlangintoyourenvironmentandfromthereyoucanuserabbitmqctl :

bash-4.1#exportPATH=$PATH:/var/vcap/packages/rabbitmq-server/binbash-4.1#exportPATH=$PATH:/var/vcap/packages/erlang/binbash-4.1#rabbitmqctlcluster_statusClusterstatusofnoderabbit@node0...[{nodes,[{disc,[rabbit@node0,rabbit@node1,rabbit@node2,rabbit@node3]}]},{running_nodes,[rabbit@node3,rabbit@node2,rabbit@node1,rabbit@node0]},{partitions,[]}]...done.

Alternatively,installRabbitMQandErlangonamachineofyourchoice.Besuretomatchversionsofbothtothecluster:theManagementUIshowsboththeversionofRabbitMQandErlang.

Thensetyour ~/.erlang.cookie tomatchthecookieusedinthecluster(youmayhavesuppliedthisaspartoftheinstallation;seeabove).

Youwillneedtosetupyour /etc/hosts filetomatchtheRabbitMQnodes.

ClusteringandNetworkPartitions

ClusteringinRabbitMQforPCFInRabbitMQforPCF,theRabbitMQ®brokerisalwaysdeployedasaclusterofoneormorevirtualmachines(nodes).ARabbitMQbrokerisalogicalgroupingofoneorseveralErlangnodes,eachrunningtheRabbitMQapplicationandsharingusers,virtualhosts,queues,exchanges,bindings,andruntimeparameters.

WhatisReplicatedbetweennodesinaRabbitMQcluster?

Alldata/staterequiredfortheoperationofaRabbitMQbrokerisreplicatedacrossallnodes.Anexceptiontothisaremessagequeues,whichbydefaultresideononenode,thoughtheyarevisibleandreachablefromallnodes.ThismeansthattheRabbitMQclustermaybeavailableandservingrequests,whileanindividualqueueresidingonasinglenodeisoffline.

Replicatingmessagequeuesacrossnodesisanexpensiveoperationandshouldonlybedonetotheextentneededbytheapplication.Tounderstandmoreaboutreplicatingqueuesacrossnodesinacluster,seethedocumentation onhighavailability.

AutomaticNetworkPartitionBehaviorsinRabbitMQClustersTheRabbitMQ®tileusesthe pause_minority optionforhandlingclusterpartitionsbydefault.Thisensuresdataintegritybypausingthepartitionoftheclusterintheminority,andresumesitwiththedatafromthemajoritypartition.Youmustmaintainmorethantwonodes.Ifthereisapartitionwhenyouonlyhavetwonodes,bothnodesimmediatelypause.

Youcanalsochoosethe autoheal optioninthePre-ProvisionedRabbitMQPolicytab.Inthismode,ifapartitionoccurs,RabbitMQautomaticallydecidesonawinningpartition,andrestartsallnodesthatarenotinthewinningpartition.Thisoptionallowsyoutocontinuetoreceiveconnectionstobothpartsofpartitions.

DetectingaNetworkPartitionWhenanetworkpartitionoccurs,alogmessageiswrittentotheRabbitMQnodelog:

=ERRORREPORT====15-Oct-2012::18:02:30===Mnesia(rabbit@da3be74c053640fe92c6a39e2d7a5e46):**ERROR**mnesia_eventgot{inconsistent_database,running_partitioned_network,rabbit@21b6557b73f343201277dbf290ae8b79}

Youcanalsorunthe rabbitmqctlcluster_status commandonanyoftheRabbitMQnodestoseethenetworkpartition.Torun rabbitmqctlcluster_status ,dothefollowing:

1. $ sudo su -

2. $ cd /var/vcap/packages

3. $ export ERL_DIR=$PWD/erlang/bin/

4. $ cd rabbitmq-server/bin/

5. $./rabbitmqctlcluster_status

[...{partitions,[{rabbit@da3be74c053640fe92c6a39e2d7a5e46,[rabbit@21b6557b73f343201277dbf290ae8b79]}]}]

RecoveringBecausetheRabbitMQtileusesthe pause_minority option,minoritynodesrecoverautomaticallyafterthepartitionisresolved.Afteranoderecovers,itresumesaccessingthequeuealongwithdatafromthequeuesontheothernodes.However,ifyourqueuesuse ha-mode:all ,theyonlysynchronizefully

afterconsumingallthemessagescreatedwhilethenodewasdown.Thisissimilartohowmessagessynchronizewhenyoucreateanewqueue.

ManuallySynchronizingafteraPartitionAfteranetworkpartition,aqueueonaminoritynodesynchronizesafterconsumingallthemessagescreatedwhileitwasdown.Youcanalsorunthesync_queue commandtosynchronizeaqueuemanually.Torun sync_queue ,dothefollowingoneachnode:

1. $ sudo su -

2. $ cd /var/vcap/packages

3. $ export ERL_DIR=$PWD/erlang/bin/

4. $ cd rabbitmq-server/bin/

5. $ ./rabbitmqctl list_queues

6. $ ./rabbitmqctl sync_queue name

UpgradingRabbitMQforPCFThisproductenablesautomatedupgradesbetweenversionsoftheproductandisdeployedthroughOpsManager.Insomecases,youmightberequiredtotaketheclusteroffline.Whenthisisnecessary,itisclearlynotedinthereleasenotesforthatversion.

TheupgradepathsforeachversionaredetailedatPivotalNetwork-RabbitMQforPCFpage .

Toupgradeon-demandinstancesfromRabbitMQv3.6tov3.7,seeAboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7.

DowntimeWhenUpgradingAguidefordowntimeduringupgradedeploymentsisshowninthetablebelow.Insomecases,theclusterremainsavailableduringatileupgrade,butindividualqueuesonclusternodesmaybetakenoffline.

Thisisonlyaguide,sobeforeupgrading,checkthereleasenotesfortheversionyouareupgradingto.

TheRabbitMQclusterbecomesunavailableonlywhenupgradingbetweenspecificversionsofErlangorRabbitMQ.Thisisstatedinthereleasenotesforthoseversions.

UpgradeType WillDowntimeBeRequiredForThisUpgrade/Update

MajorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.

MinorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.

PatchTileVersion

Normallythesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.Therearespecificmigrationpathsthatrequiredowntime,whichareidentifiedinthereleasenotesforthatversion.

Stemcell-OnlyPatchTileVersion

Wherethepatchupdateisonlyanewstemcellversionthesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.

NotesontheUpgradeProcessReviewthefollowingbeforestartinganupgradeofRabbitMQforPCF:

Upgradingtoanewerversionoftheproductdoesnotcauseanylossofdataorconfiguration.

ItmaytakebusyRabbitMQnodesalongtimetoshutdownduringtheupgradeandyoumustnotinterruptthisprocess.

ThebenefityougetfromstemcellrollingupgradesdependsonhowyouhaveconfigurednetworkpartitionhandlingandtheResourceConfigtab.AnHAProxyinstancecountof2andaRabbitMQnodecountof3arerequiredforrollingstemcellupgrades.Asofv1.7.7,thesecountsarethedefault.Formoreinformation,seeClusteringandNetworkPartitions.

ThelengthofthedowntimedependsonwhetherthereisastemcellupdatetoreplacetheoperatingsystemimageoriftheexistingVMcanjusthavetheRabbitMQsoftwareupdated.StemcellupdatesincuradditionaldowntimewhiletheIaaScreatesthenewVM.

OpsManagerensurestheinstancesareupdatedwiththenewpackagesandanyconfigurationchangesareappliedautomatically.

ForissueswithupgradingRabbitMQforPCF,seeTroubleshootingOn-DemandRabbitMQforPCF.

BeforeUpgradingRabbitMQorErlangEnsuretheclusterishealthyusingtheRabbitMQManagementUI.YoucannotrelyontheBOSH instances outputbecausethatreflectsthestateofErlangVM,notRabbitMQ.

UpgradeRabbitMQforPCFToupgradetheproduct,followthesesteps:

1. DownloadthelatestversionoftheproductfromPivotalNetwork .

2. Uploadthenew.pivotalfiletoOpsManager.

3. Uploadthestemcellassociatedwiththeupdate(ifrequired).

4. Updateanynewmandatoryconfigurationparameters(ifrequired).

5. ClickApplychangesintheOpsManagerInstallationDashboard.Therestoftheprocessisautomated.

AboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7BeforeRabbitMQforPCFv1.12,on-demandserviceinstancesweredeployedusingRabbitMQv3.6.RabbitMQforPCFv1.12provideson-demandserviceplanswithRabbitMQv3.6andv3.7.

Appdeveloperscaneitherstartfreshwithav3.7on-demandinstanceortheycanmigratetheirRabbitMQv3.6instancestoRabbitMQv3.7usingblue-greenappdeploymentswithoutdowntime.

Toenablethemigrationfromon-demandv3.6instancestov3.7instances,theon-demandv3.7plansshouldbeconfiguredtomirrortheexistingv3.6plans.Formoreinformation,seetheblogBlue-GreenApplicationDeploymentswithRabbitMQ andthevideoBlue-GreenDeploymentofApplicationsleveragingRabbitMQ .

RabbitMQv3.6on-demandplanswillberemovedinanupcomingRabbitMQforPCFtilerelease.Encouragealldeveloperstomovetheirappsawayfromtheseplans.

ReleasePolicyWhenanewversionofRabbitMQisreleased,anewversionofRabbitMQforPCFisreleasedsoonafter.

FormoreinformationaboutthePCFreleasepolicy,seeReleasePolicy .

TroubleshootingandFAQsforOn-DemandRabbitMQforPCFThistopicprovidesoperatorswithbasictroubleshootingtechniquesandFAQsforon-demandRabbitMQforPivotalCloudFoundry(PCF).

HowtoRetrieveaServiceinstanceGUIDYouneedtheGUIDofyourserviceinstancetorunsomeBOSHcommands.ToretrievetheGUID,runthecommand:

cfserviceSERVICE-INSTANCE-NAME--guid

Ifyoudonotknowthenameoftheserviceinstance,run cfservices toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthenamecolumn.

TroubleshootingErrorsStarthereifyou’rerespondingtoaspecificerrororerrormessages.

FailedInstall1. Certificateissues:Theon-demandbroker(ODB)requiresvalidcertificates.Ensurethatyourcertificatesarevalidandgeneratenewones if

necessary.

2. Deployfails:Deployscanfailforavarietyofreasons.ViewthelogsusingOpsManagertodeterminewhythedeployisfailing.

3. Networkingproblems:

CloudFoundrycannotreachtheRabbitMQforPCFservicebrokerCloudFoundrycannotreachtheserviceinstancesTheservicenetworkcannotaccesstheBOSHdirector

4. Registerbrokererrandfails.

5. Thesmoketesterrandfails.

6. Resourcesizingissues:TheseoccurwhentheresourcesizesselectedforagivenplanarelessthantheRabbitMQforPCFservicerequirestofunction.CheckyourresourceconfigurationinOpsManagerandensurethattheconfigurationmatchesthatrecommendedbytheservice.

7. Otherservice-specificissues.

CannotCreateorDeleteServiceInstancesIfdevelopersreporterrorssuchas:

Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:create

Followthesesteps:

1. IftheBOSHerrorshowsaproblemwiththedeploymentmanifest,openthemanifestinatexteditortoinspectit.

2. Tocontinuetroubleshooting,LogintoBOSH andtargettheRabbitMQforPCFserviceinstanceusingtheinstructionsonparsingaCloudFoundryerrormessage.

3. RetrievetheBOSHtaskIDfromtheerrormessageandrunthefollowingcommand:

boshtaskTASK-ID

4. Ifyouneedmoreinformation,accessthebrokerlogsandusethe broker-request-id fromtheerrormessageabovetosearchthelogsformoreinformation.Checkfor:

AuthenticationerrorsNetworkerrorsQuotaerrors

BrokerRequestTimeoutsIfdevelopersreporterrorssuchas:

Servererror,statuscode:504,errorcode:10001,message:Therequesttotheservicebrokertimedout:https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b

Followthesesteps:

1. ConfirmthatCloudFoundry(CF)isconnectedtotheservicebroker.

2. ChecktheBOSHqueuesize:

a. LogintoBOSHasanadmin.b. Run bosh tasks .

3. Iftherearealargenumberofqueuedtasks,thesystemmaybeundertoomuchload.BOSHisconfiguredwithtwoworkersandonestatusworker,whichmaynotbesufficientresourcesforthelevelofload.Adviseappdeveloperstotryagainoncethesystemisunderlessload.

CannotBindtoorUnbindfromServiceInstances

InstanceDoesNotExist

Ifdevelopersreporterrorssuchas:

Servererror,statuscode:502,errorcode:10001,message:Servicebrokererror:instancedoesnotexist`

Followthesesteps:

1. ConfirmthattheRabbitMQforPCFserviceinstanceexistsinBOSHandobtaintheGUIDCFbyrunning:

cfserviceMY-INSTANCE--guid

2. UsingtheGUIDobtainedabove,thefollowingBOSHCLIcommand:

bosh-dservice-instance_GUIDvms

IftheBOSHdeploymentisnotfound,ithasbeendeletedfromBOSH.ContactPivotalsupportforfurtherassistance.

OtherErrors

Servererror,statuscode:502,errorcode:10001,message:Servicebrokererror:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:example-service,service-instance-guid:8d69de6c-88c6-4283-b8bc-1c46103714e2,broker-request-id:15f4f87e-200a-4b1a-b76c-1c4b6597c2e1,operation:bind

Tofindouttheexactissuewiththebindingprocess:

1. Accesstheservicebrokerlogs.

2. Searchthelogsforthe broker-request-id stringlistedintheerrormessageabove.

3. ContactPivotalsupportforfurtherassistanceifyouareunabletoresolvetheproblem.

4. Checkfor:

AuthenticationerrorsNetworkerrors

CannotConnecttoaServiceInstanceIfdevelopersreportthattheirappcannotuseserviceinstancesthattheyhavesuccessfullycreatedandbound:

Asktheusertosendapplicationlogsthatshowtheconnectionerror.Iftheerrorisoriginatingfromtheservice,thenfollowRabbitMQforPCF-specificinstructions.Iftheissueappearstobenetwork-related,then:

1. Checkthatapplicationsecuritygroups areconfiguredcorrectly.Accessshouldbeconfiguredfortheservicenetworkthatthetileisdeployedto.

2. EnsurethatthenetworkthePivotalApplicationService(PAS)tileisdeployedtohasnetworkaccesstotheservicenetwork.YoucanfindthenetworkdefinitionforthisservicenetworkintheBOSHDirectortile.

3. InOpsManagergointotheservicetileandseetheservicenetworkthatisconfiguredinthenetworkstab.

4. InOpsManagergointothePAStileandseethenetworkitisassignedto.Makesurethatthesenetworkscanaccesseachother.

UpgradeAllServiceInstancesFailsIfthe upgrade-all-service-instances errandfails,lookattheerrandoutputintheOpsManagerlog.

Ifaninstancefailstoupgrade,debugandfixitbeforerunningtheerrandagaintopreventanyfailureissuesfromspreadingtootheron-demandinstances.

OncetheOpsManagerlognolongerliststhedeploymentas failing ,re-runtheerrandtoupgradetherestoftheinstances.

MissingLogsandMetricsIfnologsarebeingemittedbytheon-demandbroker,checkthatyoursyslogforwardingaddressiscorrectinOpsManager.

1. Ensureyouhaveconfiguredsyslogforthetile.

2. Ensurethatyouhavenetworkconnectivitybetweenthenetworksthatthetileisusingandthesyslogdestination.Ifthedestinationisexternal,youneedtousethepublicip VMextensionfeatureavailableinyourOpsManagertileconfigurationsettings.

3. VerifythattheFirehoseisemittingmetrics:

a. Installthe cf nozzle plugin.Forinstructions,seethefirehoseplugin GitHubrepository.b. Tofindlogsfromyourserviceinthe cfnozzle output,runthefollowing:

cfnozzle-fValueMetric|grep--line-buffered"on-demand-broker/MY-SERVICE"

Ifnometricsappearwithinfiveminutes,verifythatthebrokernetworkhasaccesstotheLoggregatorsystemonallrequiredports.

ContactPivotalsupportifyouareunabletoresolvetheissue.

FailedDeploymentonUpgradeorafterApplyChangesIfthedeploymentfailsaftereditingtheAssignAZsandNetworkspaneoftheRabbitMQforPCFtile,itmightbeduetoachangetotheIPaddressesassignedtothe RabbitMQ Server job.RabbitMQforPCFrequiresthattheseIPaddressesdonotchangeonceassigned.Ifyouchangethem,thedeploymentfails.Thisincludeschangesmadetoyourcurrentinstallationorduringanupgrade.Todiagnoseandsolvethisissue,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

TroubleshootingComponentsGuidanceoncheckingforandfixingissuesinon-demandservicecomponents.

BOSHproblems

LargeBOSHQueue

On-demandservicebrokersaddtaskstotheBOSHrequestqueue,whichcanbackupandcausedelayunderheavyloads.AnappdeveloperwhorequestsanewRabbitMQforPCFinstancesees createinprogress intheCloudFoundryCommandLineInterface(cfCLI)untilBOSHprocessesthequeuedrequest.

OpsManagercurrentlydeploystwoBOSHworkerstoprocessitsqueue.FutureversionsofOpsManagerwillletusersconfigurethenumberofBOSHworkers.

Configuration

Serviceinstancesinfailingstate

YoumayhaveconfiguredaVM/DisktypeintileplanpageinOpsManagerthatisinsufficientlylargefortheRabbitMQforPCFserviceinstancetostart.Seetile-specificguidanceonresourcerequirements.

Authentication

UAAChanges

IfyouhaverotatedanyUAAusercredentialsthenyoumayseeauthenticationissuesintheservicebrokerlogs.

Toresolvethis,redeploytheRabbitMQforPCFtileinOpsManager.Thisprovidesthebrokerwiththelatestconfiguration.

Note:YoumustensurethatanychangestoUAAcredentialsarereflectedintheOpsManager credentials tabofthePivotalApplicationService(PAS)tile.

NetworkingCommonissueswithnetworkinginclude:

Issue Solution

LatencywhenconnectingtotheRabbitMQforPCFserviceinstancetocreateordeleteabinding.

Tryagainorimprovenetworkperformance.

FirewallrulesareblockingconnectionsfromtheRabbitMQforPCFservicebrokertotheserviceinstance.

OpentheRabbitMQforPCFtileinOpsManagerandcheckthetwonetworksconfiguredintheNetworkspane.Ensurethatthesenetworksallowaccesstoeachother.

FirewallrulesareblockingconnectionsfromtheservicenetworktotheBOSHdirectornetwork.

EnsurethatserviceinstancescanaccesstheDirectorsothattheBOSHagentscanreportin.

Appscannotaccesstheservicenetwork.ConfigureCloudFoundryapplicationsecuritygroupstoallowruntimeaccesstotheservicenetwork.

ProblemsaccessingBOSH’sUAAortheBOSHdirector.FollownetworktroubleshootingandcheckthattheBOSHdirectorisonline

ValidateServiceBrokerConnectivitytoServiceInstances

Tovalidateconnectivity,dothefollowing:

1. ToSSHintotheRabbitMQforPCFservicebroker,runthefollowingcommnand:

bosh-dservice-instance_GUIDssh

2. IfnoBOSH task-id appearsintheerrormessage,lookinthebrokerlogusingthe broker-request-id fromthetask.

ValidateAppAccesstoServiceInstance

Use cfssh toaccesstotheappcontainer,thentryconnectingtotheRabbitMQforPCFserviceinstanceusingthebindingincludedintheVCAP_SERVICES environmentvariable.

Quotas

PlanQuotaissues

Message:Servicebrokererror:Thequotaforthisserviceplanhasbeenexceeded.PleasecontactyourOperatorforhelp.

1. Checkyourcurrentplanquota.

2. Increasetheplanquota.

3. LogintoOpsManager.

4. Reconfigurethequotaontheplanpage.

5. Deploythetile.

6. Findwhoisusingtheplanquotaandtaketheappropriateaction.

GlobalQuotaIssues

Message:Servicebrokererror:Thequotaforthisservicehasbeenexceeded.PleasecontactyourOperatorforhelp.

1. Checkyourcurrentglobalquota.

2. Increasetheglobalquota.

3. LogintoOpsManager.

4. Reconfigurethequotaontheon-demandsettingspage.

5. Deploythetile.

6. Findoutwhoisusingthequotaandtaketheappropriateaction.

FailingjobsandunhealthyinstancesTodeterminewhetherthereisanissuewiththeRabbitMQforPCFservicedeployment,inspecttheVMs.Todoso,runthefollowingcommand:

bosh-dservice-instance_GUIDvms--vitals

Foradditionalinformation,runthefollowingcommand:

boshinstances--ps--vitals

IftheVMisfailing,followtheservice-specificinformation.Anyunadvisedcorrectiveactions(suchasrunningBOSH restart onaVM)cancauseissuesintheserviceinstance.

TechniquesforTroubleshootingThissectioncontainsinstructionsoninteractingwiththeon-demandservicebrokerandon-demandserviceinstanceBOSHdeployments,andonperforminggeneralmaintenanceandhousekeepingtasks.

ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cfserviceINSTANCE-NAME .

$cfservicemyservice

Serviceinstance:myserviceService:super-dbBoundapps:Tags:Plan:dedicated-vmDescription:DedicatedInstanceDocumentationurl:Dashboard:

LastOperationStatus:createfailedMessage:Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:createStarted:2017-03-13T10:16:55ZUpdated:2017-03-13T10:17:58Z

Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.

The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe boshtaskTASK-ID .

The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping boshlogsbroker0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.

AccessBrokerandInstanceLogsandVMsBeforefollowingtheproceduresbelow,logintothecfCLI andtheBOSHCLI .

AccessBrokerLogsandVM(s)

YoucanaccesslogsusingOpsManager byclickingontheLogstabinthetileanddownloadingthebrokerlogs.

ToaccesslogsusingtheBOSHCLI,dothefollowing:

1. Identifytheon-demandbroker(ODB)deploymentbyrunningthefollowingcommand:

boshdeployments

2. ViewVMsinthedeploymentbyrunningthefollowingcommand:

bosh-dDEPLOYMENT-NAMEinstances

3. SSHontotheVMbyrunningthefollowingcommand:

4. Downloadthebrokerlogsbyrunningthefollowingcommand:

bosh-dservice-instance_GUIDlogs

ThearchivegeneratedbyBOSHorOpsManagerincludesthefollowinglogs:

LogName Description

broker.logRequeststotheon-demandbrokerandtheactionsthebrokerperformswhileorchestratingtherequest(e.g.generatingamanifest

andcallingBOSH).Startherewhentroubleshooting.broker_ctl.log Controlscriptlogsforstartingandstoppingtheon-demandbroker.

post-start.stderr.log

Errorsthatoccurduringpost-startverification.

post-start.stdout.log

Post-startverification.

drain.stderr.log Errorsthatoccurwhilerunningthedrainscript.

AccessServiceInstanceLogsandVMs1. Totargetanindividualserviceinstancedeployment,retrievetheGUIDofyourserviceinstancewiththefollowingcfCLIcommand:

cfserviceMY-SERVICE--guid

2. ToviewVMsinthedeployment,runthefollowingcommand:

bosh-dDEPLOYMENT-NAMEinstances

3. ToSSHintoaVM,runthefollowingcommand:

4. Todownloadtheinstancelogs,runthefollowingcommand:

bosh-dservice-instance_GUIDlogs

RunServiceBrokerErrandstoManageBrokersandInstancesFromtheBOSHCLI,youcanrunservicebrokererrandsthatmanagetheservicebrokersandperformmassoperationsontheserviceinstancesthatthebrokerscreated.Theseservicebrokererrandsinclude:

register-broker registersabrokerwiththeCloudControllerandlistsitintheMarketplace.

deregister-broker deregistersabrokerwiththeCloudControllerandremovesitfromtheMarketplace.

upgrade-all-service-instances upgradesexistinginstancesofaservicetoitslatestinstalledversion.

delete-all-service-instances deletesallinstancesofservice.

orphan-deployments detects“orphan”instancesthatarerunningonBOSHbutnotregisteredwiththeCloudController.

Torunanerrand,runthefollowingcommand:

bosh-dDEPLOYMENT-NAMErun-errandERRAND-NAME

Forexample:

bosh-dmy-deploymentrun-errandderegister-broker

RegisterBroker

The register-broker errandregistersthebrokerwithCloudFoundryandenablesaccesstoplansintheservicecatalog.Runthiserrandwheneverthebrokerisre-deployedwithnewcatalogmetadatatoupdatetheCloudFoundrycatalog.

Planswithdisabledserviceaccessarenotvisibletonon-adminCloudFoundryusers,includingOrgManagersandSpaceManagers.AdminCloudFoundry

userscanseeallplansincludingthosewithdisabledserviceaccess.

Theerranddoesthefollowing:

RegisterstheservicebrokerwithCloudController.

Enablesserviceaccessforanyplansthathavetheradiobuttonsetto enabled inthetileplanpage.

Disablesserviceaccessforanyplansthathavetheradiobuttonsetto disabled inthetileplanpage.

Doesnothingforanyforanyplansthathavetheradiobuttonsetto manual .

Toruntheerrand,runthefollowingcommand:

bosh-dDEPLOYMENT-NAMErun-errandregister-broker

DeregisterBroker

ThiserrandderegistersabrokerfromCloudFoundry.

Theerranddoesthefollowing:

DeletestheservicebrokerfromCloudController

Failsifthereareanyserviceinstances,withorwithoutbindings

UsetheDeleteAllServiceInstanceserrandtodeleteanyexistingserviceinstances.

bosh-dDEPLOYMENT-NAMErun-errandderegister-broker

IfyouhavemadechangestotheplandefinitionoruploadedanewtileintoOpsManager,youmightwanttoupgradealltheRabbitMQforPCFserviceinstancestothelatestsoftwareorplandefinition.

The upgrade-all-service-instances erranddoesthefollowing:

Collectsalloftheserviceinstancestheon-demandbrokerhasregistered

Foreachinstancetheerranddoesthefollowingserially

Issuesanupgradecommandtotheon-demandbrokerRegeneratestheserviceinstancemanifestbasedonitslatestconfigurationfromthetileDeploysthenewmanifestfortheserviceinstanceWaitsforthisoperationtocomplete,thenproceedstothenextinstance

AddstoaretrylistanyinstancesthathaveongoingBOSHtasksatthetimeofupgrade

Retriesanyinstancesintheretrylistuntilallareupgraded

Ifanyinstancefailstoupgrade,theerrandfailsimmediately.Thispreventssystemicproblemsfromspreadingtotherestofyourserviceinstances.

Toruntheerrand,dooneofthefollowing:

SelecttheerrandthroughtheOpsManagerUIandhaveitrunwhenyouclickApplyChanges.

Runthefollowingcommand.

bosh-dDEPLOYMENT-NAMErun-errandupgrade-all-service-instances

DeleteAllServiceInstances

ThiserrandusestheCloudControllerAPItodeleteallinstancesofyourbroker’sserviceofferingineveryCloudFoundryorgandspace.ItonlydeletesinstancestheCloudControllerknowsabout.ItdoesnotdeleteorphanBOSHdeployments.

The delete-all-service-instances erranddoesthefollowing:

1. Unbindsallappsfromtheserviceinstances.

2. Deletesallserviceinstancessequentially.Eachserviceinstancedeletionincludes:

a. Runninganypre-deleteerrandsb. DeletingtheBOSHdeploymentoftheserviceinstancec. RemovinganyODB-managedsecretsfromCredhubd. Checkingforinstancedeletionfailure,whichresultsintheerrandfailingimmediately

3. Determineswhetheranyinstanceshavebeencreatedwhiletheerrandwasrunning.Ifnewinstancesaredetected,theerrandreturnsanerror.Inthiscase,Pivotalrecommendsrunningtheerrandagain.

bosh-dservice-instance_GUIDdelete-deployment

DetectOrphanedInstancesServiceInstances

Aserviceinstanceisdefinedas“orphaned”whentheBOSHdeploymentfortheinstanceisstillrunning,buttheserviceisnolongerregisteredinCloudFoundry.

The orphan-deployments errandcollatesalistofservicedeploymentsthathavenomatchingserviceinstancesinCloudFoundryandreturnthelisttotheoperator.ItisthenuptotheoperatortoremovetheorphanedBOSHdeployments.

bosh-dDEPLOYMENT-NAMErun-errandorphan-deployments

Iforphandeploymentsexist—Theerrandscriptdoesthefollowing:

Exitwithexitcode10

Outputalistofdeploymentnamesundera [stdout] header

Provideadetailederrormessageundera [stderr] header

Forexample:

Note:OrphanBOSHdeploymentsdonotcorrespondtoaknownserviceinstance.Whilerare,orphandeploymentscanoccur.Usetheorphan-deployments errandtoidentifythem.

WARNING:Useextremecautionwhenrunningthiserrand.Youshouldonlyuseitwhenyouwanttototallydestroyalloftheon-demandserviceinstancesinanenvironment.

[stdout][{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]

[stderr]OrphanBOSHdeploymentsdetectedwithnocorrespondingserviceinstanceinCloudFoundry.BeforedeletinganydeploymentitisrecommendedtoverifytheserviceinstancenolongerexistsinCloudFoundryandanydataissafetodelete.

Errand'orphan-deployments'completedwitherror(exitcode10)

ThesedetailswillalsobeavailablethroughtheBOSH /tasks/ APIendpointforuseinscripting:

$curl'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result'|jq.{"exit_code":10,"stdout":"[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n","stderr":"OrphanBOSHdeploymentsdetectedwithnocorrespondingserviceinstanceinCloudFoundry.BeforedeletinganydeploymentitisrecommendedtoverifytheserviceinstancenolongerexistsinCloudFoundryandanydataissafetodelete.\n","logs":{"blobstore_id":"d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"}}

Ifnoorphandeploymentsexist—Theerrandscriptdoesthefollowing:

Exitwithexitcode0

Stdoutwillbeanemptylistofdeployments

Stderrwillbe None

[stdout][]

[stderr]None

Errand'orphan-deployments'completedsuccessfully(exitcode0)

Iftheerrandencountersanerrorduringrunning—Theerrandscriptdoesthefollowing:

Exitwithexit1

Stdoutwillbeempty

Anyerrormessageswillbeunderstderr

Tocleanuporphanedinstances,runthefollowingcommandoneachinstance:

boshdelete-deploymentservice-instance_SERVICE-INSTANCE-GUID

GetAdminCredentialsforaServiceInstance1. IdentifytheservicedeploymentbyGUID.

2. LogintoBOSH .

3. Openthemanifestinatexteditor.

4. Lookinthemanifestforthecredentials.

WARNING:RunningthiscommandmayleaveIaaSresourcesinanunusablestate.

ReinstallaTileToreinstallatileinthesameenvironmentwhereitwaspreviouslyuninstalled:

1. Ensurethattheprevioustilewascorrectlyuninstalledasfollows:

a. Loginasanadminbyrunning:

cf login

b. ConfirmthattheMarketplacedoesnotlistRabbitMQforPCFbyrunning:

c. LogintoBOSHasanadminbyrunning:

bosh log-in

d. DisplayyourBOSHdeploymentstoconfirmthattheoutputdoesnotshowRabbitMQforPCFdeploymentbyrunning:

bosh deployments

e. Runthe“delete-all-service-instances”errandtodeleteeveryinstanceoftheservice.f. Runthe“deregister-broker”errandtodeletetheservicebroker.g. DeletetheservicebrokerBOSHdeploymentbyrunning:

bosh delete-deployment BROKER-DEPLOYMENT-NAME

h. Reinstallthetile.

ViewResourceSaturationandScalingToviewusagestatisticsforanyservice,dothefollowing:

1. Runthefollowingcommand:

bosh-dDEPLOYMENT-NAMEvms--vitals

2. Toviewprocess-levelinformation,run:

bosh-dDEPLOYMENT-NAMEinstances--ps

IdentifyServiceInstanceOwnerIfyouwanttoidentifywhichappsareusingaspecificserviceinstancefromtheBOSHdeploymentsname,youcanrunthefollowingsteps:

1. Takethedeploymentnameandstripthe service-instance_ leavingyouwiththeGUID.

2. LogintoCFasanadmin.

3. Obtainalistofallservicebindingsbyrunningthefollowing:

cfcurl/v2/service_instances/GUID/service_bindings

4. Theoutputfromtheabovecurlgivesyoualistof resources ,witheachitemreferencingaservicebinding,whichcontainsthe APP-URL .Tofindthename,org,andspacefortheapp,runthefollowing:

a. cf curl APP-URL andrecordtheappnameunder entity.name .b. cf curl SPACE-URL toobtainthespace,usingthe entity.space_url fromtheabovecurl.Recordthespacenameunder entity.name .c. cf curl ORGANIZATION-URL toobtaintheorg,usingthe entity.organization_url fromtheabovecurl.Recordtheorganizationname

under entity.name .

MonitorQuotaSaturationandServiceInstanceCountQuotasaturationandtotalnumberofserviceinstancesareavailablethroughODBmetricsemittedtoLoggregator.Themetricnamesareshownbelow:

MetricName Description

on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining globalquotaremainingforallinstancesacrossallplans

on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/quota_remaining quotaremainingforaparticularplan

on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances totalinstancescreatedacrossallplans

on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/total_instances totalinstancescreatedforagivenplan

DropandRestoreAMQP(S)TraffictoaRabbitMQInstanceWhiledebuggingaRabbitMQinstance,youcanpreventappsfromsendingandreceivingmessages,forexample,todecreasetheserverload.Youcanuse drop-amqp-traffic and restore-amqp-traffic scripts,whichrunthenecessary iptables commandstoachievethat.

TostopandthenrestoretraffictoaRabbitMQinstance,dothefollowing:1. TostopallAMQP(S)traffictoaRabbitMQinstance,runthefollowingcommand:

bosh -d service-instance_GUID ssh rabbitmq-server "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/drop-amqp-traffic"

2. Afterperformingthetroubleshootingsteps,restorethetraffic.Todothis,runthefollowingcommand:

bosh -d service-instance_GUID ssh rabbitmq-server "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/restore-amqp-traffic"

Alternatively,youcanrunthesescriptsonindividualnodes:1. bosh ssh toarabbitmq-serverinstance.

2. sudo -s togainrootprivileges.

3. Execute drop-amqp-traffic todropallAMQP(S)traffictothisinstance,or restore-amqp-traffic tostartacceptingtrafficagain.

FrequentlyAskedQuestions

Note:Whenrunning cfcurl ensurethatyouqueryallpages,becausetheresponsesarelimitedtoacertainnumberofbindingsperpage.Thedefaultis50.Tofindthenextpagecurlthevalueunder next_url .

Note:Quotametricsarenotemittedifnoquotahasbeenset.

WhatshouldIcheckbeforedeployinganewversionofthetile?EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.YoucannotrelysolelyontheBOSH instances outputasthatreflectsthestateoftheErlangVMusedbyRabbitMQandnottheRabbitMQapp.

WhatisthecorrectwaytostopandstartRabbitMQinPCF?OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapp.

Forexample:

bosh stop rabbitmq-server and bosh start rabbitmq-server .

ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningthestopcommandandspecifying JOB [index] .

WhathappenswhenIrunboshstoprabbitmq-server?BOSHstartstheshutdownsequencefromthebootstrapinstance.

WestartbytellingtheRabbitMQapptoshutdownandthenshutdowntheErlangVMwithinwhichitisrunning.Ifthissucceeds,werunthefollowingcheckstoensurethattheRabbitMQappandErlangVMhavestopped:

1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.NoticethatwearetrackingtheErlangPIDandnottheRabbitMQPID.

2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID.

Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.Allremainingrabbitmq-serverinstanceswillbestoppedonebyone.

Whathappenswhenboshstoprabbitmq-serverfails?IftheBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:

result:1of1drainscriptsfailed.FailedJobs:rabbitmq-server.

WhatdoIdowhenboshstoprabbitmq-serverfails?

Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.

First,BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monitstartrabbitmq-server

.Youwillnotbeableto

startthejobviaBOSH start asthisalwaysrunsthedrainscriptfirstandwillfailsincethedrainscriptisfailing.

Oncerabbitmq-serverjobisrunning(confirmthisvia monitstatus ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.

HowcanImanuallybackupthestateoftheRabbitMQcluster?

Note:Donotuse monitstoprabbitmq-server

asthisdoesnotcallthedrainscripts

ItispossibletobackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.

BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.

2. Selectexportdefinitionsfromthemainpage.

BackupandRestorewithaScript

UsetheAPItorunscriptswithcodesimilartothefollowing:

Forthebackup:

curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"-o "$BACKUP-FOLDER/rabbit-backup.json"

Fortherestore:

curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"-X POST -H "Content-Type: application/json" -d"@$BACKUP-FOLDER/rabbit-backup.json"

Whatpre-upgradechecksshouldIdo?

BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:

1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.

2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.

3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.

KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .

FileaSupportTicketYoucanfileaticketwithPivotalSupport .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .

Toexpeditetroubleshooting,provideyourservicebrokerlogsandyourserviceinstancelogs.Ifyour cfserviceYOUR-SERVICE-INSTANCE outputincludesatask-id ,providetheBOSHtaskoutput.

FrequentlyAskedQuestionsforPre-ProvisionedRabbitMQforPCFThistopiclistsfrequentlyaskedquestionsthatapplytotheRabbitMQforPivotalCloudFoundry(PCF)pre-provisionedservice.

FrequentlyAskedQuestions

WhatshouldIcheckbeforedeployinganewversionofthetile?EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.YoucannotrelysolelyontheBOSH instances outputasthatreflectsthestateoftheErlangVMusedbyRabbitMQandnottheRabbitMQapplication.

WhatisthecorrectwaytostopandstartRabbitMQinPCF?OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapp.

Forexample:

boshstoprabbitmq-server

and boshstartrabbitmq-server

ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningthestopcommandandspecifying JOB[index]

WhathappenswhenIrunboshstoprabbitmq-server?BOSHstartstheshutdownsequencefromthebootstrapinstance.

WestartbytellingtheRabbitMQapplicationtoshutdownandthenshutdowntheErlangVMwithinwhichitisrunning.Ifthissucceeds,werunthefollowingcheckstoensurethattheRabbitMQapplicationandErlangVMhavestopped:

1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.NoticethatwearetrackingtheErlangPIDandnottheRabbitMQPID.

2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID

Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.Allremainingrabbitmq-serverinstanceswillbestoppedonebyone.

Whathappenswhenboshstoprabbitmq-serverfails?IftheBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:

result:1of1drainscriptsfailed.FailedJobs:rabbitmq-server.

WhatdoIdowhenboshstoprabbitmq-serverfails?Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.

Note:Donotuse monitstoprabbitmq-server

asthisdoesnotcallthedrainscripts

First,BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monitstartrabbitmq-server

.Youwillnotbeableto

startthejobviaBOSH start asthisalwaysrunsthedrainscriptfirstandwillfailsincethedrainscriptisfailing.

Oncerabbitmq-serverjobisrunning(confirmthisvia monitstatus ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.

HowcanImanuallybackupthestateoftheRabbitMQcluster?ItispossibletobackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.

BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.

2. Selectexportdefinitionsfromthemainpage.

BackupandRestorewithaScript

UsetheAPItorunscriptswithcodesimilartothefollowing:

1. Forthebackup:

curl-u"$USERNAME:$PASSWORD""http://$RABBIT_ADDRESS:15672/api/definitions"-o"$BACKUP_FOLDER/rabbit-backup.json"

2. Fortherestore:

curl-u"$USERNAME:$PASSWORD""http://$RABBIT_ADDRESS:15672/api/definitions"-XPOST-H"Content-Type:application/json"-d"@$BACKUP_FOLDER/rabbit-backup.json"

Whatpre-upgradechecksshouldIdo?BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:

1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.

2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.

3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.

FileaSupportTicketYoucanfileaticketwithPivotalSupport .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .

Toexpeditetroubleshooting,provideyourservicebrokerlogsandyourserviceinstancelogs.Ifyour cfserviceYOUR-SERVICE-INSTANCE outputincludesatask-id ,providetheBOSHtaskoutput.

UsingOn-DemandRabbitMQforPCFThistopicprovidesinstructionsfordevelopersusingtheon-demandRabbitMQservicefortheirPivotalCloudFoundry(PCF)apps.RabbitMQenablesmessagingbetweencloud-basedservers,appsanddevices.

TheseproceduresusetheCloudFoundryCommand-LineInterface(cfCLI).YoucanalsouseAppsManager toperformthesametasksusingagraphicalUI.

Forgeneralinformation,seeManagingServiceInstanceswiththecfCLI .

PrerequisitesTouseon-demandRabbitMQforPCFwithyourPCFapps,youneed:

APCFinstallationwithRabbitMQforPCF installedandlistedintheMarketplace

ASpaceDeveloper orAdminaccountonthePCFinstallation

Alocalmachinewiththefollowinginstalled:

abrowserashelltheCloudFoundryCommand-LineInterface (cfCLI)theLinuxwatch command

Tologinto theorgandspacecontainingyourapp

DeveloperGuide

EntriesintheVCAP_SERVICESEnvironmentVariableAppsrunninginCloudFoundrygainaccesstotheboundserviceinstancesviaanenvironmentvariablecredentialshashcalledVCAP_SERVICES .Anexamplehashisshownbelow:

{ "p-rabbitmq": [{ "credentials": { "dashboard_url": "http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "password": "tavk86pnnns1ddiqpsdtbchurn", "protocols": { "amqp": { "password": "tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7" ] } } } }]}

Formoreinformationabouttheenvironmentvariable VCAP_SERVICES ,seeRabbitMQEnvironmentVariables.

TheCreate-BindProcessBecauseeveryappandserviceinPCFisscopedtoaspace ,anappcanonlyuseaserviceifaninstanceoftheserviceexistsinthesamespace.

TouseRabbitMQinaPCFapp:

1. UsethecfCLI orAppsManager tologintotheorgandspacethatcontainstheapp.

2. MakesureaninstanceoftheRabbitMQforPCFserviceexistsinthesamespaceastheapp.

IfthespacedoesnotalreadyhaveaRabbitMQforPCFinstance,createone.IfthespacealreadyhasaRabbitforPCFinstance,youcanbindyourapptotheexistinginstanceorcreateanewinstancetobindtoyourapp.

3. BindtheapptotheRabbitMQforPCFserviceinstance,toenabletheapptouseRabbitMQ.

ConfirmServiceAvailabilityForanapptouseaservice,1)theservicemustbeavailableintheMarketplaceforitsspaceand2)aninstanceoftheservicemustexistinitsspace.

YoucanconfirmbothoftheseusingthecfCLIasfollows.

1. TofindoutifOn-DemandRabbitMQforPCFserviceisavailableintheMarketplace:

a. Enter cf marketplaceb. Iftheoutputlists ondemand-rabbitmq inthe service column,on-demandRabbitMQforPCFisavailable.Ifitisnotavailable,askyour

operatortoinstallit.

$cfmarketplaceGettingservicesfrommarketplaceinorgmy-org/spacemy-spaceasuser@example.com...OKserviceplansdescription[...]ondemand-rabbitmqSoloRabbitMQService[...]

2. ToconfirmthatanOn-DemandRabbitMQforPCFinstanceisrunninginthespace

a. Enter cf servicesb. Any ondemand-rabbitmq listingsinthe service columnareserviceinstancesofon-demandRabbitMQinthespace.

$cfservicesGettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded

Youcanbindyourapptoanexistinginstanceorcreateanewinstancetobindtoyourapp.

CreateaServiceInstanceUnlikepre-provisionedservices,on-demandservicesarecreatedasynchronously,notimmediately.The watch commandshowsyouwhenyourserviceinstanceisreadytobindanduse.

Tocreateaninstanceoftheon-demandRabbitMQforPCFservice,run cfcreate-service

1. Enter cfcreate-serviceondemand-rabbitmqSoloSERVICE_INSTANCE

Where SERVICE_INSTANCE isanameyouchoosetoidentifytheserviceinstance.Thisnamewillappearunder service [sic]inoutputfrom cf

services .

2. Enter watchcfservices andwaitforthe lastoperation foryourinstancetoshowas createsucceeded .

$cfcreate-serviceondemand-rabbitmqSolomy-instance

Creatingservicemy-instanceinorgmy-org/spacemy-spaceasuser@example.com...OK

$watchcfservices

Gettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded

Ifyougetanerror,seeTroubleshootingInstances.

BindaServiceInstancetoYourAppForanapptouseaservice,youmustbindittoaserviceinstance.Dothisafteryoupushorre-pushtheappusing cfpush .

TobindanapptoaRabbitMQinstancerun $cfbind-service

1. Enter cfbind-serviceAPPSERVICE_INSTANCE

Where APP istheappyouwanttousetheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cfcreate-

service .

$cfbind-servicemy-appmy-instance

Bindingservicemydbtomy-appinorgmy-org/spacetestasuser@example.com...OKTIP:Use'cfpush'toensureyourenvvariablechangestakeeffect

UsetheRabbitMQServiceinYourAppToaccesstheRabbitMQservicefromyourapp:

1. Run cfenvAPP_NAME withthenameoftheappboundtotheRabbitMQforPCFinstance.

2. Intheoutput,notetheconnectionstringslistedinthe VCAP_SERVICES > credentials objectfortheapp.

3. Inyourappcode,calltheRabbitMQserviceusingtheconnectionstrings.

ForhowtocodeyourapptouseRabbitMQmessaging,seeAboutUsingPivotalRabbitMQ>ClientDocumentationintheRabbitMQdocumentation .

UpdatingaServiceInstanceIfyoubindanewserviceorchangetheservicebindings,youneedtorun cf

restarttoupdatethe VCAP_SERVICES environmentvariableintheapplication

container.

1. Enter cfrestart-appAPP

Where APP istheappyouwanttousetheupdatedserviceinstance.

$cfrestartmy-app

Pushingnewversionofanappautomaticallyrestagesandrestartstheapponanyserviceinstancesitisboundto.

UnbindaServiceInstancetoYourAppTostopanappfromusingaserviceitnolongerneeds,unbinditfromtheserviceinstanceusing cfunbind-

service.

1. Enter cfunbind-serviceAPPSERVICE_INSTANCE

Where APP istheappyouwanttostopusingtheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf

create-service .

$cfunbind-servicemy-appmy-instance

Unbindingappmy-appfromservicemy-instanceinorgmy-org/spacemy-spaceasuser@example.com...OK

DeleteaServiceInstanceTodeleteaserviceinstance,run cfdelete-

service.

1. Enter cfdelete-serviceSERVICE_INSTANCE

Where SERVICE_INSTANCE isthenameoftheservicetodelete.

$cfdelete-servicemy-instance

Areyousureyouwanttodeletetheservicemy-instance?yDeletingservicemy-serviceinorgmy-org/spacemy-spaceasuser@example.com...OK

2. Enter watchcfserviceSERVICE_INSTANCE andwaitfora Serviceinstancenotfound errorindicatingthattheinstancenolongerexists.

Youcannotdeleteaserviceinstancethatanappisboundto.

CreateanAdminUserforaServiceInstanceIfyouwanttogetadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstance,andobtainusercredentialsthatyoucansharewithotherappdevelopers.

Bothoperatorsandappdeveloperscanusethisprocedure.Forinstructions,seeCreateanAdminUserforaServiceInstance.

SharingServiceInstancesInordertoshareserviceinstances thefeature-flag service_instance_sharing mustbeenabledbyyourOperator.YoucanthenfollowthedocumentationtoshareyourserviceinstancesacrossCloudFoundryOrganizationsandSpaces.

FederateExchangesandQueuesYoucanfederateexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.

Tofederateexchangesandqueues,dothefollowing:

1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.

Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.

2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:

{..."uri":"amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris":["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}

Forexample:

{..."uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}

3. Setupfederationasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.

Forinstructionsonfederation,seetheRabbitMQdocumentation .

ShovelExchangesandQueuesYoucanshovelexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.

Toshovelexchangesandqueues,dothefollowing:

1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.

Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.

2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:

{..."uri":"amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris":["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}

Forexample:

{..."uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}

3. Setupshovelasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.

Forshovelinstructions,seetheRabbitMQdocumentation .Atthemoment,RabbitMQforPCFonlysupportsDynamicShovels.

RabbitMQEnvironmentVariablesThistopicprovidesareferencefortheenvironmentvariablesthatPivotalCloudFoundry(PCF)storesforRabbitMQforPCFserviceinstances.Thesevariablesincludethecredentialsthatappsusetoaccesstheserviceinstances.

VCAP_SERVICESAppsrunninginPCFgainaccesstotheboundserviceinstancesthroughanenvironmentvariablecredentialshashcalled VCAP_SERVICES .Anexamplehashisshowbelow:

{"p-rabbitmq":[{"label":"p-rabbitmq","name":"my-rabbit-service-instance","plan":"standard","tags":["rabbitmq","messaging","message-queue","amqp","pivotal"],"credentials":{"dashboard_url":"http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn","username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","vhost":"62e5ab21-7b38-44ac-b139-6aa97af01cd7","password":"#passwordexample123456789","ssl":false,"hostname":"10.0.0.41","hostnames":["10.0.0.41","10.0.0.51"],"uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.51/62e5ab21-7b38-44ac-b139-6aa97af01cd7"],"http_api_uri":"http://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41:15672/api","http_api_uris":["http://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41:15672/api","http://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.51:15672/api"],"protocols":{"amqp":{"password":"passwordexample123456789","port":5672,"ssl":false,"username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","vhost":"62e5ab21-7b38-44ac-b139-6aa97af01cd7","host":"10.0.0.41","hosts":["10.0.0.41","10.0.0.51"],"uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},"management":{"username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","password":"passwordexample123456789","path":"/api","port":15672,"ssl":false,"host":"10.0.0.41","hosts":["10.0.0.41","10.0.0.51"],"uri":"http://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:15672/api","uris":["http://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:15672/api","http://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.51:15672/api"]}}}}]}

Youcansearchforyourservicebyits name ,givenwhencreatingtheserviceinstance,ordynamicallythroughthe tags or label properties.Thecredentials propertycanbeusedasfollows:

Thetoplevelproperties uri , uris , vhost , username , password , hostname ,and hostnames provideaccesstotheAMQP0.9.1protocol.

Amoreflexibleapproachisprovidedbythe credentials.protocols property,whichhasakeyperenabledprotocol.Thepossiblekeysare amqp ,management , mqtt ,and stomp .IfSSLisenabled,thenthekeysare amqp+ssl , management+ssl , mqtt+ssl ,and stomp+ssl respectively.

Thevaluesassociatedwitheachofthesekeysgivesaccesscredentialsspecifictoeachprotocol.Inallcases,URIsareprovided,alongwiththeindividualcomponents.

ChangingEnabledPluginsandProtocols

IfyouadjustthepluginsandprotocolsenabledforRabbitMQ,youmightneedtoforceallapp’sVCAP_SERVICES environmentvariabletoberegenerated.Addingandremovingthefollowingpluginsrequireboundappstoberestaged:

rabbitmq_management

rabbitmq_stomp

rabbitmq_mqtt

rabbitmq_amqp1_0

IncommonwithallservicesinPCF,the VCAP_SERVICES environmentvariableforanappisonlymodifiedwhentheappisboundtoaserviceinstance.Usersneedto cfunbind-

service, cfbind-

service,and cfrestage theirappinthisscenario.

Note:Removingoraddingplugins/protocolsmightcauseappsboundwithRabbitMQtobreak.

TroubleshootingInstancesThistopicprovidesbasicinstructionsforappdeveloperstroubleshootingOn-DemandRabbitMQ®forPCF.

ErrorsYoumayseeanerrorwhenusingtheCloudFoundryCommand-LineInterface(cfCLI)toperformbasicoperationsonaRabbitMQforPCFserviceinstance:

cf create

cf update

cf bind

cf unbind

cf delete

ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cfserviceINSTANCE-NAME .

$cfservicemyservice

Serviceinstance:myserviceService:super-dbBoundapps:Tags:Plan:dedicated-vmDescription:DedicatedInstanceDocumentationurl:Dashboard:

LastOperationStatus:createfailedMessage:Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:createStarted:2017-03-13T10:16:55ZUpdated:2017-03-13T10:17:58Z

Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.

The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe boshtaskTASK-ID .

The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping boshlogsbroker0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.

RetrieveServiceInstanceInformation1. Logintothespacecontainingtheinstanceorfailedinstance.

$cflogin

2. Ifyoudonotknowthenameoftheserviceinstance,run cfservices toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthe name column.

$cfservicesGettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded

3. Run cfserviceSERVICE-INSTANCE-NAME toretrievemoreinformationaboutaspecificinstance.

4. Run cfserviceSERVICE-INSTANCE-NAME--guid toretrievetheGUIDoftheinstance,whichisusefulfordebugging.

RetrieveRabbitMQInstanceCredentialsIfyouwanttoaccesstheManagementDashboardortheRabbitMQserverfortroubleshooting,youcancreateanewservice-keytoretrieveRabbitMQinstancecredentials.Pivotalrecommendsthatyouusethiskeyfortroubleshootingonly,andthatyoudeletethekeyaftertroubleshooting.Toretrievethecredentials,dothefollowing:

1. Createaservice-keyforyourRabbitMQinstanceusingthecommand cfcreate-service-keyINSTANCE-NAMESERVICE-KEY-NAME .

2. Retrievethecredentialsusingthecommand cfservice-keyINSTANCE-NAMESERVICE-KEY-NAME .

Forexample:

$cfcreate-service-keymy-rmq-instancemy-keyCreatingservicekeymy-keyforserviceinstancemy-rmq-instanceasadmin...OK$cfservice-keymy-rmq-instancemy-keyGettingkeymy-keyforserviceinstancemy-rmq-instanceasadmin...{"host":"10.0.8.4","password":"","port":6379}

FileaSupportTicketYoucanfileasupporttickethere .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .

Toexpeditetroubleshooting,ifpossible,provideyourservicebrokerlogs,serviceinstancelogs,andBOSHtaskoutput.Yourcloudoperatorshouldbeabletoobtainthesefromyourerrormessage.

DeleteRabbitMQInstancesOn-DemandBrokerprovidesaBOSHcommandtodeletealltheOn-DemandBrokerdeployedinstances.Todeletetheinstances,dothefollowingprocedure:

1. RunthefollowingcommandtodeleteallinstancesoftheOn-DemandBroker:

boshrun-erranddelete-sub-deployments

WARNING:Thiscommanddeletesdeploymentinstancesserially.Itisverydestructiveandcannotbeundone.

rabbitmq for pcf - pivotal · documentation for rabbitmq for pcf v1.7 and earlier only describes a...

Documents

integrating rabbitmq with php

pcf feb2010

learning rabbitmq - sample chapter

rabbitmq on openvms

celery-rabbitmq documentation

spring rabbitmq

intro to rabbitmq

integrating postgresql with rabbitmq

rabbitmq in sprayer

php, rabbitmq, and you

rabbitmq fairly-indepth

rabbitmq oxford geek night

scaling rabbitmq to 11

becs rabbitmq connector documentation · becs rabbitmq...

the rabbitmq message broker

rabbitmq for perl mongers

reducing load with rabbitmq

interoperability with rabbitmq

improvements in rabbitmq

rabbitmq hands on