RabbitMQforPCF®
Documentation
Version1.12
Published:5March2019
©2019PivotalSoftware,Inc.AllRightsReserved.
236
16212428415657646869737678808297
100105107
TableofContents
TableofContentsRabbitMQforPCFRabbitMQ®forPCFReleaseNotesUnlockingthePowerofOn-DemandRabbitMQforPCFOn-DemandServiceArchitectureDeployingtheRabbitMQPre-ProvisionedServiceInstallingandConfiguringtheOn-DemandServiceInstallingandConfiguringthePre-ProvisionedServiceSmokeTestsMonitoringandKPIsforPre-ProvisionedRabbitMQforPCFSettingLimitsforOn-DemandServiceInstancesControllingAccesstoServicePlansbyOrgIsolatingClusterswiththeRabbitMQforPCFReplicatorSettingDefaultPoliciesfortheRabbitMQServiceUsingtheRabbitMQManagementDashboardClusteringandNetworkPartitionsUpgradingRabbitMQforPCFTroubleshootingandFAQsforOn-DemandRabbitMQforPCFFrequentlyAskedQuestionsforPre-ProvisionedRabbitMQforPCFUsingOn-DemandRabbitMQforPCFRabbitMQEnvironmentVariablesTroubleshootingInstances
©CopyrightPivotalSoftwareInc,2013-2019 2 1.12
RabbitMQforPCF
AboutRabbitMQforPCFRabbitMQforPivotalCloudFoundry(PCF)enablesPCFappdeveloperstoprovisionandusetheRabbitMQmessagebrokerwithasinglecommand.
RabbitMQforPCFv1.8andlatersupporttwotypesofservice,anon-demandserviceandapre-provisionedservice.Thistablesummarizesthemaindifferencesbetweenthetwo:
AvailableSince VMsitRunsOn HowVMsareCreated MetricsName
Prefix
On-DemandService v1.8
DedicatedVMthatservesasingleserviceinstance.Seethistopicfordetails.
PCFcreateseachVMon-demandwhenappdevelopercreatesserviceinstance
p.rabbitmq
(withadot)
Pre-ProvisionedService
v1.2Multi-tenantVMssharedbyappsacrossPCFdeployment
PCFcreatesallVMswhenoperatordeploysorupdatesservice
p-rabbitmq
(withadash)
ThisRabbitMQforPCFdocumentationdescribesbothservicetypes.DocumentationforRabbitMQforPCFv1.7andearlieronlydescribesapre-provisionedservice.
WhatareOn-DemandInstancesInRabbitMQforPCFversionsbeforev1.8.0,theRabbitMQserviceinstancescorrespondtoauniqueRabbitMQvhostonthemulti-tenantRabbitMQcluster.RabbitMQforPCFv1.8.0introducedOn-DemandBroker(ODB) support.Thatmeansthatanew,single-tenant,clustercanbecreatedanddedicatedtoasingleapp.
Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCFandOn-DemandServiceArchitecture.
AboutRabbitMQRabbitMQisafastanddependableopen-sourcemessageserver,whichsupportsawiderangeofusecasesincludingreliableintegration,content-basedroutingandglobaldatadelivery,andhigh-volumemonitoringanddataingestion.
Emergingasthedefactostandardforcloudmessaging,RabbitMQisusedforefficientcommunicationbetweenservers,appsanddevices,andcreateslastingvaluebyenablingrapiddevelopmentofmoderndecentralizedappanddataarchitecturesthatcanscalewithyourbusinessneeds.
ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutRabbitMQforPCF.
Element Details
Version v1.12.15
Releasedate December28,2018
Softwarecomponentversions RabbitMQOSSv3.6.16andRabbitMQOSSv3.7.9
CompatibleOpsManagerversions v2.1.xandv2.2.x
CompatiblePivotalApplicationServiceversions v2.1.xandv2.2.x
IaaSsupport AWS,Azure,GCP,OpenStack,andvSphere
IPsecsupport No
Note:RabbitMQforPCFv1.12isnolongersupported.Thesupportperiodforv1.12hasexpired.Tostayup-to-datewiththelatestsoftwareandsecurityupdates,upgradetoasupportedversion.
©CopyrightPivotalSoftwareInc,2013-2019 3 1.12
Features
On-DemandCreateupto5differenton-demandRabbitMQplanswhichcanbeprovisionedthroughthemarketplace
Choosewhetheraplanhasone,three,five,orsevennodes
Defaultresourcesizesinplanstoguideselection
Morecontroloverwhichorgsandspaceshavevisibilityofeachconfiguredplan
Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding
Managementdashboardaccesstoappdevelopers
Deploymentintoanavailabilityzonespecifiedbytheplan
AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seereleasenotesfordowntimerequirements)
RabbitMQSyslogforwardingconfigurationinheritedfromthepre-provisionedconfiguration
RabbitMQmetricsareexposedontheFirehose
Runsmoketestsforon-demandplansonplan1
ErrandsarerunoncolocatedVMstodecreasedeploymenttimes
Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCF.
Pre-ProvisionedProvisionaninstanceoftheRabbitMQservice,whichcorrespondstoauniqueRabbitMQvhost(virtualhost)
Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding
ManagementdashboardaccesstoPCFOperatorsandappdevelopers
Deploymentacrossmultipleavailabilityzones,withnodesstripedacrosstheAZsautomatically
EnableSSL(SecureSocketsLayer)fortheAMQP,MQTT,STOMPprotocols
HAProxyloadbalanceracrossallnodestobalanceconnections
Pluginconfigurationcanbeeasilychangedatanytimeandtheclusterredeployedandupdated
Theclustertopologycanbechangedandeasilyscaledout
AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seeDowntimeWhenUpgradingfordowntimerequirements)
ConfiguretheendpointfortheRabbitMQSyslog
RabbitMQandHAProxymetricsareexposedonthefirehose
Syslogforwardingonbydefault
ErrandsarerunoncolocatedVMstodecreasedeploymenttimes
ReleaseNotesandKnownIssuesCheckthereleasenotesforyourreleaseversionforimportantinformationandknownissues.Toseereleasenotesforanotherversion,selecttheversionfromthedropdownlistatthetopofthepage.
RabbitMQforPCFandOtherPCFServicesSomePCFservicesofferon-demandserviceplans.Theseplansletdevelopersprovisionserviceinstanceswhentheywant.
Thesecontrastwiththemorecommonpre-provisionedserviceplans,whichrequireoperatorstoprovisiontheserviceinstancesduringinstallationandconfigurationthroughtheservicetileUI.
ThefollowingPCFservicesofferon-demandserviceplans:
MySQLforPCFv2.0andlater
RabbitMQforPCF
©CopyrightPivotalSoftwareInc,2013-2019 4 1.12
RedisforPCF
PivotalCloudCache(PCC)
Theseservicespackageanddelivertheiron-demandserviceofferingsdifferently.Forexample,someservices,likeRedisforPCF,haveonetile,andyouconfigurethetiledifferentlydependingonwhetheryouwanton-demandserviceplansorpre-provisionedserviceplans.
Forotherservices,likePCCandMySQLforPCF,onlyon-demandserviceplansareavailable.
ThefollowingtablelistsandcontraststhedifferentwaysthatPCFservicespackageon-demandandpre-provisionedserviceofferings.
PCFservicetile Standaloneproductrelatedtotheservice Versionssupportingondemand Versionssupportingpre-provisioned
RabbitMQforPCF PivotalRabbitMQ v1.8andlater Allversions
RedisforPCF Redis v1.8andlater Allversions
MySQLforPCF MySQL v2.x NA
PCC PivotalGemFire Allversions NA
Pleaseprovideanybugs,featurerequests,orquestionstothePCFFeedbacklist.
©CopyrightPivotalSoftwareInc,2013-2019 5 1.12
RabbitMQ®forPCFReleaseNotes
UpgradetotheLatestVersionPivotalrecommendsthatyouupgradetothelatestversionofyourcurrentminorline,thenupgradetothelatestavailableversionofthenewminorline.Forexample,ifyouuseanolderv1.11.xversion,upgradetothelatestv1.11.xversionbeforeupgradingtothelatestv1.12.xversion.
Forproductversionsandupgradepaths,seetheProductCompatibilityMatrix .
UseRabbitMQv3.7On-DemandPlansRabbitMQv3.7plansareavailable.EnsurethatallappsusetheRabbitMQv3.7on-demandplansinsteadoftheRabbitMQv3.6on-demandplans.
Formoreinformation,seethefollowing:
AboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7
TheANNRabbitMQ3.6.xsupporttimeline postontheRabbitMQusersforum
v1.12.15ReleaseDate:December28,2018
FeaturesNewfeaturesandchangesinthisrelease:
Requiresstemcell 3586.65
ScriptstodropandrestoreAMQP(S)traffic.Formoreinformation,seeDropandRestoreAMQP(S)TraffictoaRabbitMQInstance.
FixedIssuesThisreleasefixesthefollowingissue:
InRabbitMQforPCFv1.12.14andv1.12.13,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.
KnownIssuesThisreleasehasthefollowingissues:
[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.
Note:PivotalnolongerrecommendswaitingforSpringCloudServices(SCS)v3.0,wherethedependencyonpre-provisionedRabbitMQforPCFwillberemoved,beforeupgradingfromRabbitMQforPCFv1.12tov1.13.ThisisduetotheriskofgoingoutofsupportonyourRabbitMQforPCFtile,PivotalApplicationService(PAS),andothercomponents.
WARNING:PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1oronPCFv2.2.6orlater.
©CopyrightPivotalSoftwareInc,2013-2019 6 1.12
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.9
Erlangv20.3.8.15
HAProxyv1.6.13
v1.12.14ReleaseDate:December20,2018
FeaturesRequiresstemcell 3586.60
Setswapto1GBforRabbitMQnodes
SupportstheLogCachecfCLIpluginthatenablesdeveloperstoaccesslogsforanon-demandserviceinstanceusingthecommand cf tail .
Formoreinformationaboutthisfeature,seeAccessRabbitMQMetricsforOn-DemandServiceInstances.
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
IfyouupgradetoRabbitMQforPCFtothisversion,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.Thisissuedoesnotaffectnewinstallations.
ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.9
Erlangv20.3.8.15
HAProxyv1.6.13
©CopyrightPivotalSoftwareInc,2013-2019 7 1.12
v1.12.13ReleaseDate:November22,2018
SecurityFixesThisreleaseincludesthefollowingsecurityfix:
CriticalCVE-2018-15759:OnDemandServicesSDKTimingAttackVulnerability
FeaturesAddsmutualTLSbetweenservicemetricsandtheloggregatorsystem
RemovescredentialsfromthedashboardURLinthepre-provisionedservice
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
IfyouupgradetoRabbitMQforPCFtothisversion,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.Thisissuedoesnotaffectnewinstallations.
ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.8
Erlangv20.3.8.10
HAProxyv1.6.13
v1.12.12ReleaseDate:November1,2018
FeaturesRequiresstemcell 3468.78
Smoketestsarenowmoreresilientwhenusingexternalloadbalancers.
KnownIssues
©CopyrightPivotalSoftwareInc,2013-2019 8 1.12
[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.8
Erlangv20.3.8.9
HAProxyv1.6.13
v1.12.10ReleaseDate:October5,2018
FeaturesRequiresstemcell 3468.73
RabbitMQ3.7updatedto3.7.8
Erlangupdatedtov20.3.8.9
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.8
Erlangv20.3.8.9
©CopyrightPivotalSoftwareInc,2013-2019 9 1.12
HAProxyv1.6.13
v1.12.9ReleaseDate:September13,2018
FeaturesRequiresstemcell 3468.69
Smoketestsnowwaitupto5minutesforatestapptodeploy.
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.7
Erlangv20.3.8.6
HAProxyv1.6.13
v1.12.8ReleaseDate:August20,2018
FeaturesRequiresstemcell3468.64
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
Thereisanissuewithupgradingfromv1.12.xtov1.13.xonPCFv2.2.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1.
©CopyrightPivotalSoftwareInc,2013-2019 10 1.12
YoucannotupgradefromRabbitMQforPCFv.12toRabbitMQforPCFv1.13onPCFv2.2.0-2.2.5.YoumustupgradeonPCFv2.1oronPCFv2.2.6orlater.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.7
Erlangv20.3.8.1
HAProxyv1.6.13
v1.12.7ReleaseDate:July13,2018
FeaturesNewfeaturesandchangesinthisrelease:
FixesbuginthesmoketestswhenCredHubisenabled
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
Thereisanissuewithupgradingfromv1.12.xtov1.13.xonPCFv2.2.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1.
YoucannotupgradefromRabbitMQforPCFv.12toRabbitMQforPCFv1.13onPCFv2.2.0-2.2.5.YoumustupgradeonPCFv2.1oronPCFv2.2.6orlater.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.7
Erlangv20.3.8.1
HAProxyv1.6.13
v1.12.6
©CopyrightPivotalSoftwareInc,2013-2019 11 1.12
ReleaseDate:June28,2018
FeaturesNewfeaturesandchangesinthisrelease:
Requiresstemcell3468.51 .
UpdatesOSSRabbitMQServerandErlangpackages
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
OSSRabbitMQv3.7.6
Erlangv20.3.8.1
HAProxyv1.6.13
v1.12.5ReleaseDate:June5,2018
FeaturesNewfeaturesandchangesinthisrelease:
Requiresstemcell3468.46 .
cf-cligolangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .
FixedIssuesThisreleasefixesthefollowingissue:
Smoketestssometimesfailedbecauseoflowtime-outvalues.
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
©CopyrightPivotalSoftwareInc,2013-2019 12 1.12
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.15
OSSRabbitMQv3.7.4
Erlangv19.3.6.4
HAProxyv1.6.13
v1.12.4ReleaseDate:May16,2018
FeaturesNewfeaturesandchangesinthisrelease:
Requiresstemcell3468.42 .
Timestampsaddedtologentriesthatdidnothavethem.
Golangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
Smoketestsmightfailbecausetime-outvaluesinthesmoketestsaretoolow.Formoreinformationabouttroubleshootingsmoketests,seeSmokeTests.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.15
OSSRabbitMQv3.7.4
Erlangv19.3.6.4
HAProxyv1.6.13
©CopyrightPivotalSoftwareInc,2013-2019 13 1.12
v1.12.3ReleaseDate:April19,2018
FeaturesNewfeaturesandchangesinthisrelease:
On-Demand
ThefollowingerrandsarecolocatedwiththeirrespectivebrokertodecreaseerrandruntimeandVMfootprint:
Register/DeregisterOnDemandServiceBrokerOnDemandInstanceSmokeTestsUpgradeAllServiceInstancesDeleteAllServiceInstances
Formoreinformationabouterrands,seeErrands .
[Beta]ServiceSharingallowsRabbitMQdeploymentstobesharedacrossCloudFoundryspacesandorgs.Formoreinformationaboutsharingserviceinstances,seeCloudFoundryDocumentation .
[Beta]Supportforsecurecredentials(CredHub).Credentialsarestoredinacentralrepositoryandarerestrictedtocomponentsthatactuallyneedthem.
FormoreinformationaboutCredHub,seeCloudFoundryDocumentation .
Pre-Provisioned
ThefollowingerrandsarecolocatedwiththeirrespectivebrokertodecreaseerrandruntimeandVMfootprint:
BrokerRegistrar/DeregistrarSmokeTestserrands
Formoreinformationabouterrands,seeErrands .
KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.15
OSSRabbitMQv3.7.4
Erlangv19.3.6.4
HAProxyv1.6.13
ViewReleaseNotesforAnotherVersionToviewthereleasenotesforanotherproductversion,selecttheversionfromthedrop-downlistatthetopofthispage.
©CopyrightPivotalSoftwareInc,2013-2019 14 1.12
©CopyrightPivotalSoftwareInc,2013-2019 15 1.12
UnlockingthePowerofOn-DemandRabbitMQforPCF
IntroductionRabbitMQforPivotalCloudFoundry(PCF)respondstothedemandsofPCFoperatorstoofferaRabbitMQon-demandclusterfortheirapplicationteams,inadditiontotheexistingsingle-nodeon-demandplan.Theon-demandclusterplanisaimedatworkloadsthatrequirethesameresiliencerequirementsasthePre-Provisionedoffering,butalsorequiretheirworkloadsbeisolated.
TheplatformoperationsteamcannowconfigureaRabbitMQforPCFclustertomeettheirbusinessrequirementsandempowerappdevelopmentteamstoself-servetheirownRabbitMQcluster.
RabbitMQforPCFalsoprovidessmoketestsfortheon-demandplanssothatoperationsteamscanvalidatetheappdeveloperworkflowforon-demandservices.SeeDedicatedInstanceSmokeTestProcess.
PlatformoperatorscannowoffertheirappdevelopersthreetypesofRabbitMQforPCFserviceplans:
Pre-provisioned—Forlighttomoderatemessagingneeds,thisserviceisfullyoperatedandmanagedbyplatformoperatorsasaservice.
On-demandsinglenode—Forapplicationteamsrequiringgreaterisolationthanprovidedbythepre-provisionedapproach.Appdevelopmentteamscanhavefullaccesstotheirownmessagebrokertoadapttheruntimeparameterstotheirrequirements.Formoreinformationontheseparameters,seeParametersandPolicies intheRabbitMQdocumentation.
On-demandcluster—Foranincreasedlevelofmessageresilienceandclusteravailability,aswellasthebenefitsofworkloadisolationmentionedabove.
Thistopicexplainshowtobenefitfromthetwoon-demandplansabove.
Forinformationaboutthepre-provisionedplan,seeDeployingtheRabbitMQPre-ProvisionedService.Forinformationonusingpre-provisionedplanstoisolateworkloads,seeCreatingIsolationwiththeTileReplicator.
On-DemandSingleNodePlanUsingRabbitMQ3.7Thisplanisdesignedtobesimpletoconfigure,deploy,anduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingopensourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.
Thisplancansuithigh-performanceworkloadsrequiringmessagingresilienceandasynchronousmessagingreplication.RabbitMQcopiesmessagestodiskforresilienceandallowsasynchronousmessagingreplicationthroughtheRabbitMQFederationplug-in.
Thisplanoffers:
FastaccesstoanisolatedinstanceofRabbitMQscopedfortheapplicationteams
OrgandSpaceAdministratoraccesstotheRabbitMQManagementUIsoapplicationteamscanhavefullcontroloverthenode
Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes
MessageresilienceprovidedthroughRabbitMQexchange,queueFederation,andShovelplugins.
ToBeDeprecated:On-DemandSingleNodePlanUsingRabbitMQ3.6Thisplanprovidesthesamebenefitsasdescribedabove,butusesRabbitMQ3.6forbackwardscompatibility.
RabbitMQ3.6isreachingendofsupport.ConsidermovingtoRabbitMQ3.7assoonaspossible.FormoreinformationaboutsupportforRabbitMQ3.6,seetheRabbitMQ3.6.xsupporttimelineannouncement .
IfyouarealreadyusingRabbitMQforPCF,youcanchoosetoprovideserviceplanswithRabbitMQ3.6forbackwardscompatibility.However,youshouldalsoenableRabbitMQ3.7planstoallowdeveloperstomigratetothenewversion.
Fornewdeployments,PivotalrecommendsdisablingallRabbitMQ3.6plansandusingRabbitMQ3.7toavoidtheneedtoupgradeinthenearfuture.
On-DemandClusterPlanUsingRabbitMQ3.7
©CopyrightPivotalSoftwareInc,2013-2019 16 1.12
Likethesinglenodeplan,thisplanisdesignedtobesimpletoconfigure,deployanduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingOpenSourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.
Thisplancansuithighperformanceworkloadsrequiringmessagingresilience(copiedtodisk)andasynchronousmessagingreplicationthroughtheRabbitMQFederationplugin.Withthisplan,however,youalsoscaleoutRabbitMQforPCFtomultiplenodes.
Thisplanoffers:
Fastaccesstoanisolated,clusteredinstanceofRabbitMQscopedtotheapplicationteamOrgsandSpaces
AdministratoraccesstotheRabbitMQManagementUItogiveapplicationteamsfullcontroloverthecluster
Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes.
MessageresilienceprovidedbymirroringqueuesacrossRabbitMQnodes,andtheoptiontousetheFederationandShovelplugins.
ToBeDeprecated:On-DemandClusterPlanUsingRabbbitMQ3.6Thisplanprovidesthesamebenefitsasdescribedabove,butusesRabbitMQ3.6forbackwardscompatibility.
RabbitMQ3.6isreachingendofsupport.ConsidermovingtoRabbitMQ3.7assoonaspossible.FormoreinformationaboutsupportforRabbitMQ3.6,seetheRabbitMQ3.6.xsupporttimelineannouncement .
IfyouarealreadyusingRabbitMQforPCF,youcanchoosetoprovideserviceplanswithRabbitMQ3.6forbackwardscompatibility.However,youshouldalsoenableRabbitMQ3.7planstoallowdeveloperstomigratetothenewversion.
Fornewdeployments,PivotalrecommendsdisablingallRabbitMQ3.6plansandusingRabbitMQ3.7toavoidtheneedtoupgradeinthenearfuture.
GeneralPrinciplesoftheClusterPlanThefollowingaresomegeneralprinciplestobeawareofwhenconfiguringtheclusterplan:
DesignedforConsistency
RabbitMQclusteringisnotprimarilyasolutionforincreasedavailability.Instead,itisdesignedforconsistencyandpartitiontolerance,asdescribedintheCAPtheorem .RabbitMQclusteringprovidesincreasedmessageconsistencythroughqueuemirroring.Thismeansthatmessagesaccessedinonequeueareexactlythesameasinanotherqueue.Formoreinformation,seeConsistencyorAvailabilityTradeoff.
Otheroptionscanbeusedforavailabilityrequirements,suchastheuseoffederationbetweenexchangesorqueues.
ForadetaileddescriptionofdistributedRabbitMQbrokers,seetheRabbitMQdocumentation .
NumberofNodes
Everynodeintheon-demandclustermaintainsacompletedatabaseofallmetadata,andallchangestothemetadataareconfirmedbyeverynodeinthecluster.Therefore,goingbeyondsevennodescanhaveasignificantnegativeimpactonperformance.Foroptimumresilienceandperformance,Pivotalrecommendsthreenodesformostworkloads.
NetworkLatency
RabbitMQclustersareonlyrecommendedfordeploymentinlowlatencynetworks,whichnormallymeansthatitisnotadvisabletodeploytheseclustersacrossavailabilityzones(AZs).ThestabilityandperformanceoftheRabbitMQclusterisheavilyinfluencedbytheworkloadonthenodes,replicationchoices,andnetworklatency.
Forthisreason,PivotalrecommendsthatyoudeployRabbitMQclustersintoasingleOpsManagerAZ.However,wheredifferentAZsareinthesamedatacenter,withreliablelowlatencylinks,spanningAZscanbeused.
ForcloudIaaSdeployments,Pivotaldoesnotrecommendthatdeploymentsspanregions.Forexample,inAmazonWebServices(AWS)terms,deployingaRabbitMQclusteracrossAZswithinaregionshouldprovidehighenoughnetworkperformancetopreventimpactingclusterstability.However,deployingacrossAWSregionsislikelytoleadtoclusterinstability.Formoreinformation,seetheAWSdocumentation .
©CopyrightPivotalSoftwareInc,2013-2019 17 1.12
ConsistencyorAvailabilityTradeoffInadistributedmessagingsystem,atradeoffmustbemadebetweenavailabilityorconsistencywhenanetworkpartitioneventoccursandoneormorenodesarenotabletocommunicatewitheachother.TheclusterplanletsoperatorsdecidehowtheywanttheRabbitMQclustertoreactintheeventofanetworkpartition.
Pivotalrecommendskeepingthedefaultclusterpartitionoptionof pause_minority becausethissatisfiesmostusecases.Choosingthe pause_minority
partition-handlingstrategyfavorsmessageconsistencyoveravailability.Formoreinformationabouttheoptionsforhandlingpartitions,seetheRabbitMQdocumentation .ForadetaileddescriptionoftheoptionsavailableinRabbitMQforPCF,seeClusteringandNetworkPartitions.
Hereisanexampleofhow pause_minority works.IfyoucreateaRabbitMQclusterwiththreenodesandonenodebecomesunabletocommunicatewiththeothertwo,thisnodeisintheminority.Thenodethatisintheminorityispaused,andtheothertwonodescontinueservingtraffic.Ifeachofthenodeslosesconnectivitywiththeothertwo,thentheentireclusterispausedtopreservedatasincenomajoritycanbeestablished.Theclusterhealswhentwoormorenodesareabletocommunicatewitheachother.
RabbitMQQueueAvailabilityItisimportanttobeawarethatmessagequeueavailabilityisdifferentfromclusteravailability.So,havingclusteravailabilitydoesnotmeanthatallofthemessageswithinthequeuesarealsoavailable.
Bydefault,queueswithinaRabbitMQclusterarelocatedonasinglenode—thenodeonwhichtheywerefirstdeclared.However,queuescanbeconfiguredtomirroracrossmultiplenodes,sothatanymessagepublishedtothequeueisreplicatedtoallmirrors.Enablingmirroringcanhaveanegativeimpactonqueueperformancebecausemessagesmustbecopiedtoallmirrorsbeforebeingacknowledged.
Eachmirroredqueueconsistsofonemasterandoneormoremirrors,withtheoldestmirrorbeingpromotedtothenewmasteriftheoldmasterdisappearsforanyreason.Consumersareconnectedtothemasterregardlessofwhichnodetheyconnectto,andmirrorsdropmessagesthathavebeenacknowledgedatthemaster.Queuemirroringenhancesqueueavailability,butdoesnotdistributeloadacrossnodesbecauseeachoftheparticipatingnodesmuststilldoallthework.
Appdevelopersmustdecideiftheywanttousequeuemirroringanddeterminethepolicytheywanttoapplytotheirqueues.Thesechoiceshavesignificantimpactontheavailabilityoftheirqueues.Formoreinformation,seetheRabbitMQdocumentation .
Unlikethepre-provisionedplan,theclusterplandoesnotshipwithadefaultloadbalancer.Therefore,developersmustconfiguretheirapptousethearrayofhostsprovidedin VCAP_SERVICES .Ifdevelopersenablequeuemirroring,theymustalsoensuretheirappshavere-trylogicandreconnectionlogicthatiteratesovertherangeofhostsprovided.MostcommonRabbitMQclientshavethislogicbuiltintothem.Formoreinformation,seetheSpringAMQPdocumentation .
Becausetheclusterplanisdesignedtoenableapplicationteamstoself-serve,nothavingaloadbalancerinfrontoftheRabbitMQclusterhasthesebenefits:
Manageresourcesbetter,asfewerVMsareneeded.
Helpwithtroubleshooting.ClientIPisnowtheIPofthesourcecontainerandnottheHAProxy.
Reducethenumberofhopsbetweenappsandbroker.Thishelpswithlatency.
Determinequeueplacement.Thismakessenseforlargerscaledeployments.
Empowerapplicationteamstomanagetheirclusterinthebestwayfortheirapp.
Requirere-trylogicinanappifitneedsHAaccesstoaqueue.Thus,allnodescanroutetoaqueueifitisavailable.
ManagingOn-DemandResourcesThroughPlansInconfiguringeachplan,thereareanumberofoperationalcontrolsthatplatformoperationsteamscanusetomanagetheresourcesconsumedbyon-demandRabbitMQ:
ControlAccess—Operatorscanchoosetheappdevelopmentorgsandspacesforwhichtheplansareavailableandvisible.Eachplancanbeenabledordisabled,andserviceaccessandvisibilitycaneitherbeglobal,orenabledperorgandspacethroughthecommandline.
Forexample,youmaydecidetoenablethesinglenodeon-demandplanacrossallapplicationteamstomeettheirdemandtoisolatetheirworkload.Youmaythenchoosetooffertheon-demandclusterplanonlytoasubsetofapplicationteamswhorequiretheextraresources.
SetQuotas—Youcansetaglobalquotaforallon-demandinstancesthattakesprecedenceovereachplanquota.Thisletsyouguardagainsttheriskofover-committingresources,butallowstheflexibilityofover-committingeachplan,soyoucanmeetthefluctuatingdemandsofyourappdevelopers.
ControlResourceConsumption—Eachplanoffersmorefine-grainedcontroloverindividualplanresourceconsumption.Atthehighestlevel,youcanusetheplanquotatocontrolthenumberofinstancesthatcanbedeployedwithinafoundation.Foreachplan,youcanalsoconfigurethenumberof
©CopyrightPivotalSoftwareInc,2013-2019 18 1.12
nodesthatconstituteacluster(3,5,or7),theinstancetype,andpersistentdiskstoragesizetobestsuityourrequirements.
Monitor—Youcanmonitorthenumberofinstancesthathavebeendeployedagainstthequotayouhavesetsothatyoucanplanfutureresourcerequirements.
CustomizingPlanOptionsTheRabbitMQforPCFon-demandplansexposeanumberofconfigurationoptions.Inmostcases,thedefaultconfigurationsmeetmostappdemands.However,itisimportantforanoperationsteamtoconsidertheoptionstoensurethattheyprovidethebestservicetotheirappdevelopers.Thissectionexplainstheseoptions.
ConfigurationOptions
SingleNodeandClusterPlansEnable/Disableplan
Determinewhichorgsandspacescanseeandaccesstheplan
SetServiceInstanceQuota
SelectAZplacement(whereapplicable)
SetRabbitMQinstancesize(CPUandMemory)
Setpersistentdisksize(PersistedMessageStore)fortheRabbitMQinstance.Ensurethesizeofthepersistentdiskisatleasttwiceaslargeastheinstancememory.
ClusterPlanOnlySetnumberofnodesto3,5,or7
Determinenetworkpartitionbehavior.SeeConsistencyorAvailabilityTradeoffabove.
ThingsThatArePreconfiguredThefollowingarepreconfiguredforboththesinglenodeandtheclusterplans:
RabbitMQVMType—WheninstallingonPCFv2.0orlater,eachRabbitMQnodeisconfiguredtohavethefollowingproperties:
CPUs:2RAM:8GBEphemeraldisk:16GB
YoucanchangethesesettingsintheServicePlanConfigurationpage.Changingthesesettingsaffectsallnodes.
PersistentDiskType—WheninstallingonPCFv2.0orlater,eachRabbitMQnodeisconfiguredtohave30GBofpersistentdiskspace.
YoucanchangethissettingintheServicePlanConfigurationpage.PivotalrecommendsyousetthisvaluetobetwicetheamountofRAMoftheselectedRabbitMQVMType.
Metrics—EmittedtotheLoggregatorFirehoseforallon-demandinstances.ThepollingintervalissetintheOpsManager,intheMetricspollingintervalfield,inthePre-ProvisionedRabbitMQtaboftheRabbitMQforPCFtile.Duetotheimpactofsomeoftheclustersettingsdetailedbelow,PivotalstronglyrecommendsthatyoumonitortheexposedmetricsandconfigurealarmsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF .SeealsoMonitoringOn-DemandRabbitMQClustersbelow.
Logs—RabbitMQon-demandinstancelogsareforwardedusingthesameconfigurationascontainedintheSyslogtaboftheRabbitMQforPCFtile.
Diskfreespacelimit—Thediskfreespacelimitissetto150%ofRAMoftheinstancetypeyouselect.Forexample,ifyouselectaninstancetypewith10GBofRAM,thediskfreespacelimitissetto15GB.Acluster-widealarmistriggerediftheamountoffreediskspacedropsbelowthis,andallpublishersareblocked.InstancesmustbeconfiguredtohavepersistentdisksthatareatleasttwicethesizeofinstanceRAM.Formoreinformation,seetheRabbitMQdocumentation .
Memorythresholdfortriggeringflowcontrol—Thresholdatwhichflowcontrolistriggeredissetto40%oftheinstanceRAM.Thismeansthatwhen
Note:Aloadbalancer,suchasHAProxy,isnotdeployedwithon-demandclusterplans.
©CopyrightPivotalSoftwareInc,2013-2019 19 1.12
thealarmistriggered,allconnectionspublishingmessagesareblockedcluster-wideuntilthealarmiscleared.
Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan4GBofmemoryisused,allpublishingconnectionsareblocked.Formoreinformation,seeMemoryAlarms intheRabbitMQdocumentation.
Memorypagingthreshold—ThisisthelevelatwhichRabbitMQtriestofreeupmemorybyinstructingqueuestopagetheircontentsouttodisk.Thisisdonetotrytoavoidreachingthehighwatermarkandblockingpublishers.Thisthresholdissetto50%oftheconfiguredhighwatermark,whichis20%ofconfiguredmemory.Formoreinformationonmemorycalculation,seeChangestoMemoryAllocationwhenUpgrading .
Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan2GBofmemoryisused,allqueuesstartwritingallqueuecontentstodisk.Formoreinformation,seetheRabbitMQdocumentation .
MonitoringOn-DemandRabbitMQClustersItisimportanttomonitorandcomparethenumberofinstancesthathavebeendeployedagainstthequotayousetviathemetricexposedontheFirehose.
Eachinstanceispre-configuredtoemitmetricstotheFirehoseandcanbeidentifiedbythe deployment tag,whichhastheserviceinstanceID.ItisimportanttomonitorthesemetricsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF .
©CopyrightPivotalSoftwareInc,2013-2019 20 1.12
On-DemandServiceArchitectureThistopicdescribesthearchitectureforon-demandRabbitMQ®forPivotalCloudFoundry(PCF).
Forinformationaboutarchitectureoftheolder,pre-provisionedservice,seeDeployingtheRabbitMQ®Service.
ServiceNetworkRequirementWhenyoudeployPCF,youmustcreateastaticallydefinednetworktohostthecomponentvirtualmachinesthatconstitutethePCFinfrastructure.
PCFcomponents,liketheCloudControllerandUAA,runonthisinfrastructurenetwork.On-demandPCFservicesmayrequirethatyouhostthemonanetworkthatrunsseparatelyfromthisnetwork.Youcanalsodeploytilesonseparateservicenetworkstomeetyourownsecurityrequirement.
PCFv2.0andEarlierInPCFv2.0andearlier,cloudoperatorspre-provisionserviceinstancesfromOpsManager.Foreachservice,OpsManagerallocatesandrecoversstaticIPaddressesfromapre-definedblockofaddresses.
Toenableon-demandservicesinPCFv2.0andearlier,operatorsmustcreateaservicenetworksinBOSHDirectorandselecttheServiceNetworkcheckbox.Operatorsthencanselecttheservicenetworktohoston-demandserviceinstanceswhentheyconfigurethetileforthatservice.
PCFv2.1andLaterPCFv2.1andlaterincludedynamicnetworking.InPCFv2.1andlater,operatorscanusedynamicnetworkingwithasynchronousserviceprovisioningtodefinedynamically-provisionedservicenetworks.Formoreinformation,seeDefaultNetworkandServiceNetwork.
InPCFv2.1andlater,on-demandservicesareenabledbydefaultonallnetworks.OperatorscancreateseparatenetworkstohostservicesinBOSHDirector,butdoingsoisoptional.Operatorsselectwhichnetworkhostson-demandserviceinstanceswhentheyconfigurethetileforthatservice.
DefaultNetworkandServiceNetworkOn-demandPCFservicesrelyontheBOSH2.0abilitytodynamicallydeployVMsinadedicatednetwork.Theon-demandservicebrokerusesthiscapabilitytocreatesingle-tenantserviceinstancesinadedicatedservicenetwork.
On-demandservicesusethedynamically-provisionedservicenetworktohostthesingle-tenantworkerVMsthatrunasserviceinstanceswithindevelopmentspaces.ThisarchitectureletsdevelopersprovisionIaaSresourcesfortheirserviceinstancesatcreationtime,ratherthantheoperatorpre-provisioningafixedquantityofIaaSresourceswhentheydeploytheservicebroker.
Bymakingservicessingle-tenant,whereeachinstancerunsonadedicatedVMratherthansharingVMswithunrelatedprocesses,on-demandserviceseliminatethe“noisyneighbor”problemwhenoneapphogsresourcesonasharedcluster.Single-tenantservicescanalsosupportregulatorycompliancewheresensitivedatamustbecompartmentalizedacrossseparatemachines.
Anon-demandservicesplitsitsoperationsbetweenthedefaultnetworkandtheservicenetwork.Sharedcomponentsoftheservice,suchasexecutivecontrollersanddatabases,runcentrallyonthedefaultnetworkalongwiththeCloudController,UAA,andotherPCFcomponents.Theworkerpooldeployedtospecificspacesrunsontheservicenetwork.
ThediagrambelowshowsworkerVMsinanon-demandserviceinstancerunningonaseparateservicesnetwork,whileothercomponentsrunonthedefaultnetwork.
©CopyrightPivotalSoftwareInc,2013-2019 21 1.12
RequiredNetworkingRulesforOn-DemandServicesBeforedeployingaservicetilethatusestheon-demandservicebroker(ODB),requesttheneedednetworkconnectionstoallowcomponentsofPivotalCloudFoundry(PCF)tocommunicatewithODB.
ThespecificsofhowtoopenthoseconnectionsvariesforeachIaaS.
Seethefollowingtableforkeycomponentsandtheirresponsibilitiesinanon-demandarchitecture.
KeyComponents TheirResponsibilities
BOSHDirector
CreatesandupdatesserviceinstancesasinstructedbyODB.
BOSHAgentIncludesanagentoneveryVMthatitdeploys.TheagentlistensforinstructionsfromtheBOSHDirectorandcarriesoutthoseinstructions.TheagentreceivesjobspecificationsfromtheBOSHDirectorandusesthemtoassignarole,orjob,totheVM.
BOSHUAA IssuesOAuth2tokensforclientstousewhentheyactonbehalfofBOSHusers.
PAS Containstheappsthatareconsumingservices
ODB InstructsBOSHtocreateandupdateservices,andconnectstoservicestocreatebindings.
Deployedserviceinstance
Runsthegivendataservice.Forexample,thedeployedRedisforPCFserviceinstancerunstheRedisforPCFdataservice.
Regardlessofthespecificnetworklayout,theoperatormustensurenetworkrulesaresetupsothatconnectionsareopenasdescribedinthetablebelow.
Thiscomponent…Mustcommunicatewith…
DefaultTCPPort Communicationdirection(s) Notes
ODB
BOSHDirector
BOSHUAA
25555
8443 One-way Thedefaultportsarenotconfigurable.
ODBDeployedserviceinstances
15672(RabbitMQManagementUI)
One-wayThisconnectionisforadministrativetasks.Avoidopeninggeneraluse,app-specificportsforthisconnection.
ODBPAS(orElastic 8443 One-way Thedefaultportisnotconfigurable.
©CopyrightPivotalSoftwareInc,2013-2019 22 1.12
Runtime)
ErrandVMs
PAS(orElasticRuntime)
ODB
DeployedServiceInstances
8443
8080
15672(RabbitMQManagementUI)
5671-2(AMQP/AMQPS)
One-way Thedefaultportisnotconfigurable.
BOSHAgentBOSHDirector
4222 Two-way
TheBOSHAgentrunsoneveryVMinthesystem,includingtheBOSHDirectorVM.TheBOSHAgentinitiatestheconnectionwiththeBOSHDirector.Thedefaultportisnotconfigurable.
DeployedappsonPAS(orElasticRuntime)
Deployedserviceinstances
15672(RabbitMQManagementUI)
5671-2(AMQP/AMQPS)
61613-4(STOMP/STOMPS)
1883,8883(MQTT/MQTTS)
One-wayThisconnectionisforgeneraluse,app-specifictasks.Avoidopeningadministrativeportsforthisconnection.
PAS(orElasticRuntime)
ODB 8080 One-wayThisportmaybedifferentforindividualservices.Thisportmayalsobeconfigurablebytheoperatorifallowedbythetiledeveloper.
DeployedappsonPAS
RuntimeCredHub
8844 One-wayThisportisneededifsecureserviceinstancecredentialsareenabled.Forinformation,see(Beta)OnDemand-SecureServiceInstanceCredentialswithRuntimeCredHub.
©CopyrightPivotalSoftwareInc,2013-2019 23 1.12
DeployingtheRabbitMQPre-ProvisionedService
DefaultDeploymentDeployingRabbitMQforPivotalCloudFoundry (PCF)throughOpsManagerdeploysaRabbitMQclusterof3nodesbydefault.
Thedeploymentincludesasingleloadbalancer haproxy whichspreadsconnectionsonallofthedefaultports,foralloftheshippedpluginsacrossallofthemachineswithinthecluster.
Thedeploymentoccursinasingleavailabilityzone(AZ).
ThedefaultconfigurationisfortestingpurposesonlyandPivotalrecommendsthatcustomershaveaminimumof3RabbitMQnodesand2HAProxynodes
ConsiderationsforthisdeploymentProvideshighavailabilityfortheRabbitMQcluster
Queuesmustbeconfiguredtobehighavailabilityastheyareplacedononenodebydefault
Customersshoulddecidewhichpartitionbehaviorisbestsuitedtotheirusecase.Fortwonodes‘automatic’ispreferred.
HAProxyisasinglepointoffailure(SPOF)
TheentiredeploymentisinasingleAZ,whichdoesnotprotectagainstexternalfailuresfromfailuresinhardware,networking,etc.
RecommendedDeployment
©CopyrightPivotalSoftwareInc,2013-2019 24 1.12
PivotalrecommendsthatRabbitMQforPCFisdeployedacrossatleasttwoAZs.ScaleRabbitMQservernodestoanoddnumberthatisgreaterthanorequaltothree.
Onlyusereplicationofqueueswhererequiredasitcanhaveabigimpactonsystemperformance.
TheHAProxyjobinstancecountshouldalsobeincreasedtomatchthenumberofAZstoensurethereisaHAProxylocatedineachAZ.ThisremovestheHAProxySPOFandprovidesfurtherredundancy.
ThediagramaboveshowsthatyoucannowsufferthefailureofasingleHAProxyandsingleRabbitMQnodeandstillkeepyourclusteronlineandapplicationsconnected.
UpgradingtothisdeploymentfromasingleAZdeploymentItisnotpossibletoupgradetothissetupfromthedefaultdeploymentacrossasingleAZ.
ThisisbecausetheAZsetupcannotbechangedafterthetilehasbeendeployedforthefirsttime.ThisistoprotectagainstdatalosswhenmovingjobsbetweenAZs.
UpgradingtothisdeploymentfromamultiAZdeploymentIfyouhavedeployedthetileacrosstwoAZs,butwithasingleHAProxyinstance,youcanmigratetothissetupbydeployinganadditionalHAProxyinstancethroughOpsManager.Neworre-boundapplicationstotheRabbitMQserviceseetheIPsofbothHAProxiesimmediately.Existingboundapplicationswillcontinuetowork,butonlyusingthepreviouslydeployedHAProxyIPAddress.Theycanbere-boundasrequiredatyourdiscretion.
ConsiderationsforthisdeploymentRequiresIaaSconfigurationforAZsaheadofdeployingtheRabbitMQtile
ApplicationdevelopersarehandedtheIPsofeachdeployedHAProxyintheirenvironmentvariables
Queuesmustbeconfiguredtobehighavailabilityastheyareplacedononenodebydefault
©CopyrightPivotalSoftwareInc,2013-2019 25 1.12
Customersshoulddecideonwhichpartitionbehaviorisbestsuitedtotheirusecase.Forthreeormorenodes'pause_minority’ispreferred.
AdvancedDeploymentThisdeploymentbuildsupontheaboverecommendeddeploymentandsofollowsthesameupgradepaths.
ThisallowsyoutoreplacetheuseofHAProxywithyourownexternalloadbalancer.
YoumightchoosetodothistoremoveanyknowledgeofthetopologyoftheRabbitMQsetupfromapplicationdevelopers.
Advantages
ApplicationdevelopersdonotneedtohandlemultipleIPsfortheHAProxyjobsintheirapplications
Disadvantages
TheloadbalancerneedstobeconfiguredwiththeIPsoftheRabbitMQNodes.Theseareonlybeknownafterthedeploymentisfinished.TheIPsshouldremainthesameduringsubsequentdeploymentsbutthereisarisktheymightchange.
UpgradingtothisdeploymentfromtherecommendeddeploymentItispossibletofirstdeploywithmultipleHAProxyjobs,aspertherecommendeddeployment,andlateruseyourownexternalloadbalancer.
Thiscanbeachievedwithoutdowntimetoyourapplications.Followthesestepstodoso:
1. ConfigureyourexternalloadbalancertopointtotheRabbitMQNodeIPs.
2. ConfiguretheDNSnameorIPaddressfortheexternalloadbalancer(ELB)ontheRabbitMQtileinOpsManager.
3. Deploythechanges.AnynewinstancesoftheRabbitMQserviceoranyre-boundconnectionswillusetheDNSnameorIPaddressoftheELBintheirVCAP_SERVICES .AnyexistinginstanceswillcontinuetousetheHAProxyIPaddressesintheir VCAP_SERVICES
4. Phasethere-bindingofexistingapplicationstoupdatetheirenvironmentvariables.
5. Afterallapplicationsareupdated,reducetheinstancecountofthe HAProxy jobinOpsManagerto1.
6. Deploythechanges.
Thisapproachworksasanyexistingboundapplicationshavetheir VCAP_SERVICES informationcachedinCloudControllerandareonlyupdatedbyare-
©CopyrightPivotalSoftwareInc,2013-2019 26 1.12
bindrequest.
DowngradingfromthisdeploymenttotherecommendeddeploymentIfyouarecurrentlyusinganexternalloadbalancer,thenyoucanmovebacktousingHAProxiesinstead.
Youcanachievethisbyfollowingtheabovestepsinreverseorderandre-instatingtheHAProxyjobs.
ResourcerequirementsThefollowingtableshowsthedefaultresourceandIPrequirementsforinstallingthetile:
Product Resource Instances CPU Ram Ephemeral Persistent StaticIP DynamicIP
RabbitMQ RabbitMQnode 3 2 8192 16384 30720 1 0
RabbitMQ HAProxyforRabbitMQ 1 1 2048 4096 0 1 0
RabbitMQ RabbitMQservicebroker 1 1 2048 4096 0 1 0
RabbitMQ BrokerRegistrar 1 1 1024 2048 0 0 1
RabbitMQ BrokerDeregistrar 1 1 1024 2048 0 0 1
RabbitMQ SmokeTests 1 1 1024 2048 0 0 1
RabbitMQ RabbitMQon-demandbroker 1 1 1024 8192 1024 0 1
RabbitMQ RegisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1
RabbitMQ DeregisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1
RabbitMQ DeleteAllServiceInstances 1 1 1024 2048 0 0 1
RabbitMQ UpgradeAllServiceInstances 1 1 1024 2048 0 0 1
Notes:Thenumberof RabbitMQ Node canbeincreasedifrequired.
ChangingthenumberofRabbitMQnodeswhentheerlangcookieisnotdefinedwillrestartthecluster.Checkhereformoreinformation.
©CopyrightPivotalSoftwareInc,2013-2019 27 1.12
InstallingandConfiguringtheOn-DemandServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideon-demandservice.
TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .
Role-BasedAccessinOpsManagerOpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.
FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .
PrerequisitesforDeployingtheOn-DemandServiceBeforedeployingRabbitMQforPCFasanon-demandservice,youmustensurethattherequirednetworkrulesareinplacetoallowvariouscomponentstocommunicate.
SeeRequiredNetworkingRulesforOn-DemandServicesfordetailsonthenetworkconnectionsthatmustbeopentoenabletheon-demandservice.
Forinformationabouttheon-demandservicearchitecture,seeOn-DemandServiceArchitecture.
DownloadandInstallRabbitMQforPCF1. DownloadtheproductfilefromPivotalNetwork .
2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.
3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.
4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.
ConfigureOn-DemandRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.
Note:Forinstructionsonhowtoinstall,configure,anddeploytheRabbitMQforPCFtileasapre-provisionedservice,seeInstallingandConfiguringthePre-ProvisionedService.
©CopyrightPivotalSoftwareInc,2013-2019 28 1.12
WhichSettingsTabstoConfigurefortheOn-DemandServiceConfigurethefollowingtabsfortheon-demandservice:
RabbitMQSettingsTab Instructions
AssignAZsandNetworks ConfigureAZsandNetworks
Pre-ProvisionedRabbitMQ ConfigureAdminCredentialsandMetricsPollingInterval
Networking Networking
©CopyrightPivotalSoftwareInc,2013-2019 29 1.12
Syslog SetupSyslogForwarding
GlobalSettingsforOn-DemandPlans ConfigureGlobalSettings
OnDemandInstance:Plann ConfiguretheServicePlan
Errands Errands
Stemcell VerifytheStemcell
ConfigureAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.
1. ClickAssignAZsandNetworks.
2. ConfigurethefieldsontheAssignAZsandNetworksasfollows:
Field Instructions
Placesingletonjobsin
SelecttheregionthatyouwantforsingletonVMs.PCFcreatestheRabbitMQbrokerinthisAZ.
Balanceotherjobsin
Selectadditionalregion.Thisselectiondoesnotaffecttheon-demandRabbitMQforPCFservice.
Network
SelectanetworkfortheRabbitMQOn-DemandBroker.
ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.FormoreinformationabouttheDefaultNetwork,seeDefaultNetworkandServiceNetwork.
Typically,youselectthenetworkusedforthePivotalApplicationService(PAS)orElasticRuntimecomponents.
ServiceNetwork
Selectaseparatenetworkthattheon-demandserviceinstancesrunon.
Atypicalpracticeistoputallon-demandservicesonasinglenetwork,separatefromthenetworkthatPivotalApplicationService(PAS)orElasticRuntimeandtheOn-DemandBrokerrunon.ForinformationabouttheServiceNetwork,seeDefaultNetworkandServiceNetwork.
Thisfieldisalsorequiredforthepre-provisionedservice,thoughinthatcase,itdoesn’tmatterwhichnetworkyouselect.
3. ClickSave.
ConfigureAdminCredentialsandMetricsPollingIntervalOnthePre-ProvisionedRabbitMQtab,youmustconfiguretwoitemsonlyfortheondemandservice:
SpecifyadmincredentialsforusingtheRabbitMQManagementDashboard.
Enterametricspollinginterval.
SpecifyAdminUserCredentials
IntheRabbitMQadminusercredentialsfieldofthePre-ProvisionedRabbitMQtab,enteranadminusernameandpassword:
Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesintheApplyChangesfromYourConfigurationbelow.
WARNING:ChangingtheNetworkorServiceNetworkafteryouhaveconfiguredthem,orchangingtheirIPconfigurations,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
©CopyrightPivotalSoftwareInc,2013-2019 30 1.12
Youcanuseacombinationofupperorlowercasealphanumericsandsupportedspecialcharacters: []^_!"#$%&()*+,-./\:;<=>? .
ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.
EnteraMetricsPollingInterval
IntheMetricspollingintervalfield,setametricspollinginterval:
Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.
Changingthissettingaffectsalldeployedinstances.
Formoreinformation,seeWhatareMetrics .
ConfigureLoggingtoMonitorRabbitMQforPCFPivotalrecommendsthatyouconfigureloggingtomonitorthehealthofRabbitMQforPCF.FollowSetUpSyslogForwarding toconfigurelogging.
NetworkingTospecifyastaticIPaddressfortheOnDemandServiceBroker,dothefollowing:
1. SelectNetworking.
2. EnteranIPaddressintheOnDemandServiceBrokerStaticIPfield.ThisIPaddressisassignedtoyourOnDemandServiceBrokernode.BOSHallocatesanIPaddressifthefieldisleftblank.
3. ClickSave.
ConfigureGlobalSettingsFollowthestepsbelowtoconfigureglobalsettings.
1. ClickGlobalSettingsforOn-DemandPlans.
Note:Youcannotusebackquote ` orsinglequote ' .
Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.
©CopyrightPivotalSoftwareInc,2013-2019 31 1.12
2. Configurethefollowing:
Serviceinstancequotamin:0,max:50setthetotalnumberofon-demandserviceinstanceswhichcanbedeployed.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.
VMoptions:
Allowoutboundinternetaccess(IaaS-dependent).Thischeckboxmustbetickedtoallowexternallogforwarding,sendingbackupartifactstoexternaldestinations,andcommunicatingwithanexternalBOSHblobstore.
(Beta)ShareableInstances:ClickYestoenablethebetafeatureforsharinginstances.
Note:OutboundnetworktrafficrulesalsodependonyourIaaSsettings.ConsultyournetworkorIaaSadministratortoensurethatyourIaaSallowsoutboundtraffictotheexternalnetworksyouneed.
Note:ThisisabetafeaturebasedonanexperimentalfeatureinCloudFoundry.Usethefeatureatyourownriskinnon-productionenvironments.Ifyoutrythisfeature,pleasesendyourcommentsandfeedbacktoPCFFeedbackList.
©CopyrightPivotalSoftwareInc,2013-2019 32 1.12
Sharingaserviceinstancebetweenspaces,allowsappsindifferentspacestosharedatabases,messagingqueues,andmanyothertypesofservices.Formoreinformation,seeSharingServiceInstances(Beta) .
*(Beta)OnDemand-SecureServiceInstanceCredentialswithRuntimeCredHub:Foron-demandservicesinstances,clickYestosecurecredentialswithCredHub.
3. ClickSave.
ConfiguretheServicePlanToenabletheon-demandservice,youmustconfigureatleastoneon-demandplan.
Youcanconfigureuptofiveon-demandplans:OnDemandInstance:Plan1–OnDemandInstance:Plan5.
Allon-demandplanscanbeconfiguredtohave1,3,5,or7RabbitMQnodes.
Iftheon-demandserviceisnotenabled,theon-demandbrokerisdeployedalongsidetheRabbitMQinstallation,butitisnotavailableintheMarketplace.
1. Choosetheon-demandserviceinstanceyouwanttoconfigure:
OnDemandInstance:Plan1(completerequiredfieldsevenifyoudisablethisplan):
Note1:ForthisfeaturetoworkyoumustalsoenableitinPAS.Forinstructions,seeStep1:ConfigurethePASTile .Afterdeployingthetile,notifydevelopersthattheymustunbindandrebindexistingserviceinstancestosecuretheircredentialswithCredHub.
Note2:Thisisabetafeature.Useitatyourownriskinnon-productionenvironments.Ifyoutrythisfeature,pleasesendyourcommentsandfeedbacktoPCFFeedbackList.
Note:YoumustfullyconfigureOnDemandInstance:Plan1evenifyoudisableaccesstothisplan(seeCFServiceAccessinthetablebelow).
©CopyrightPivotalSoftwareInc,2013-2019 33 1.12
OnDemandInstance:Plan2,3,4,and5:
©CopyrightPivotalSoftwareInc,2013-2019 34 1.12
©CopyrightPivotalSoftwareInc,2013-2019 35 1.12
2. Configurethefieldsasfollows:
Field Instructions
EnableThisPlan
(Plans2-5only)Toenable,selectPlanEnable.
CFServiceAccess
Enableordisableaccesstothisplan,orleaveaccessunchanged.
IfyouenablePlan1,thedefaultsettingforPlans2-5isEnableServiceAccess.Ifyouchangethisdefaultsetting,thesmoketestsfail.Therefore,ifyouenablePlan1andwanttochangethisdefault,beforedoingso,settheOn-DemandInstanceSmokeTestserrandtoOff.Formoreinformation,seeErrands.
EnableServiceAccess—Givesaccesstoallorgs,anddisplaystheserviceplantoalldevelopersintheMarketplace.DisableServiceAccess—Disablesaccesstoallorgs,andhidestheserviceplantoalldevelopersintheMarketplace.ThissettingcannotbeselectedatalatertimeintheUI.LeaveServiceAccessUnchanged—Keepsanyexistingaccesssettings,andonlydisplaystheserviceplanintheMarketplacetomembersoforgsthathaveaccesstotheplan.YoucanchangetheaccesssettingslaterusingthecfCLI.Forinstructions,seeControllingAccesstoServicePlansbyOrg.
PlanName Acceptthedefaultorenteraname.ThisisthenamethatappearsintheMarketplace.
PlanDescription
Acceptthedefaultorenteradescription.ThisdescriptionappearsintheMarketplace.
PlanFeatures
Acceptthedefaultorenteradescription.ThisdescriptionappearsinAppsManager.
ServiceInstanceQuota
Enterthemaximumnumberofon-demandserviceinstancesthatcanbeavailableatonetime.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.
NumberofNodes
Enter1,3,5or7.Thissettingonlyaffectsnewserviceinstances.Previouslydeployedserviceinstancesarenotupdated.
NetworkPartitionBehaviour
Select pause_minority or autoheal .Pivotalrecommendsusingpauseminority.Formoreinformation,seeConsistencyorAvailabilityTradeoff.
RabbitMQVMType
SelectalargeVMtype.Theplancreatesaserviceinstanceofthissize.Formoreinformation,seeUnderstandingRabbitMQVMTypesandPersistentDiskSizebelow.
PersistentDiskType
ThisiswhereRabbitMQpagesmessagestodisk.ServiceinstancedeploymentsfailifthisvalueislessthantwicethevolumeofRAMoftheselectedRabbitMQVMType.Formoreinformation,seeUnderstandingRabbitMQVMTypesandPersistentDiskSizebelow.
AZPlacement
ThisfieldisavailableafteryoucompletetheAssignAZsandNetworkspage.
Forasingle-nodeplan,selectoneormoreAZs.Foraplancontainingmultiplenodes,selectonlyoneAZ.Pivotalrecommendsthisformulti-nodeplanstominimizerisksduetonetworklatencyandpartitions.SeeNetworkLatencyandConsistencyorAvailabilityTradeofffordetails.
Ifyouchangethisselectionafterdeployment,existinginstancesarenotaffectedbythechange.SeeDeterminewhichAZsaServiceInstanceUses.
3. ClickSave.
DeterminewhichAZsaServiceInstanceUses
TodeterminewhichAZsaserviceinstanceisplacedin,dooneofthefollowing:
RetrievetheserviceGUIDusingthe cf service SERVICE_INSTANCE --guid commandandthenruntheBOSH instances commandfortheservice-instance_GUID deployment.
Withsyslogforwardingenabled,inspecttheservicebrokerlogswhenrunningtheUpgradeAllServiceInstanceserrand.Foreachexistingservice
Important:IfyouchangethisconfigurationafteryouhaveselectedAZsanddeployedserviceinstances,existinginstancesarenotplacedinthenewlyconfiguredAZswhentheUpgradeAllServiceInstanceserrandisrun.Thispreventsre-creationoftheVMsindifferentAZs,whichcanleadtodataloss.Allnewserviceinstances,however,willbecreatedinthenewlyconfiguredAZs.
©CopyrightPivotalSoftwareInc,2013-2019 36 1.12
instance,thelogmessageincludestheserviceinstanceGUIDandtheAZstheserviceinstanceisrunningin.
UnderstandingRabbitMQVMTypesandPersistentDiskSize
TheRabbitMQVMTypeandPersistentdisktypearerequiredfieldsontheserviceplanconfigurationpages.IfyouareinstallingonPCFv2.0orlater,thesepropertiesarepre-configuredbydefault.
PivotalrecommendsthatthevalueofPersistentdisktypebetwicetheamountofRAMoftheselectedRabbitMQVMType.
YoucanchangetheRabbitMQVMtypeandthesizeofthepersistentdiskthatisattachedtotheRabbitMQinstances.Forexample,ifyouarerunningoutofdiskspaceyoumightwanttoincreasethepersistentdisksizebychangingthePersistentdisktypefield.Ifyoumakechanges,ensurethatthepersistentdisksizeisstilltwicethesizeoftheRAMoftheRabbitMQVMtype.
RabbitMQraisesalarmswhendiskspacedropsbelowtheconfiguredlimit.Incorrectdisksizesmightcausethedeployedinstancenottostart.RabbitMQdeclinestostartifthereisnotenoughspaceavailableaccordingtothethreshold.
On-Demandinstancesareconfiguredwithathresholdsettothe150%ofthememory(RAM)oftheVM.Usethefollowingtableasaguidewhenselectingthesizeofthepersistentdisk.
ThefollowingtableshowsanexampleofpossibleRAMvalues,absoluteminimalvaluebelowwhichRabbitMQdeclinestostart,andthedisksizesuggestedforanaverageusecase.
RAM Freediskalarmthreshold(1.5xRAM) Suggesteddisksize(2xRAM)
10GB 15GB 20GB
16GB 24GB 32GB
32GB 48GB 64GB
MinimumresourcesrequiredforeachRabbitMQVM:
CPU:2
RAM:1GB
Ephemeraldisk:2GB
Persistentdisk:4GB
Formoreinformation,seethefollowing:
MemoryandDiskAlarms
DiskAlarms
Forinformationonallpreconfiguredsettings,seeThingsthatarePreconfigured.
VerifytheStemcellForOpsManagerv2.0:
1. ClickStemcell.
©CopyrightPivotalSoftwareInc,2013-2019 37 1.12
2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .
ForOpsManagerv2.1:
1. FollowtheprocedureoutlinedinImportingandManagingStemcells .
ApplyChangesfromYourConfigurationYourinstallationisnotcompleteuntilyouapplyyourconfigurationchanges.Followthestepsbelow:
1. ReturntotheOpsManagerInstallationDashboard.
2. ClickApplyChanges.
ErrandsWhendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofPost-DeployErrands.Anexampleisthe Smoke
Testserrand,
whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.
Youcandecidewhethertorunerrandsbytogglingthemonoroffbeforeanupdate.Thisisaone-timesettingontheinstallationdashboard:
However,ifnecessary,youcanchangethesedefaultsbyclickingErrandsintheRabbitMQforPCFSettingstab.
Formoreinformationonerrandrunrules,seeErrandRunRules .
RabbitMQforPCFerrandsarecolocatedwiththeirbrokerstodecreaseerrandruntimeandVMfootprint.Inearlierreleases,anewVMwasdeployedforeacherrand.Formoreinformationabouterrands,seeErrands .
Important:AsofRabbitMQforPCFv1.9.0,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,becausethesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmightfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.
©CopyrightPivotalSoftwareInc,2013-2019 38 1.12
Post-DeployErrands
Errand Description
BrokerRegistrar
Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace
SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTests.
RegisterOn-DemandServiceBroker
Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.
On-DemandInstanceSmokeTests
Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTestsbelow.
UpgradeAllServiceInstances
On-Demandinstancesareupdatedandredeployediftherearechangestoon-demandplansettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoon-demandplansettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.
Pre-DeleteErrands
Errand Description
BrokerDeregistrarRemovesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings
DeleteAllServiceInstancesUnbindsanddeletesexistingon-demandserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.
DeregisterOn-DemandServiceBroker
Removestheon-demandRabbitMQservicefromtheMarketplace
On-DemandInstanceSmokeTestsSmoketestsonlyrunagainstPlan1.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.
CreateanAdminUserforaServiceInstanceIfyouwanttogiveappdevelopersadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstanceandsharetheusercredentialswithappdevelopers.
Bothoperatorsandappdeveloperscanusethisprocedure.
TocreateanadminuseronaRabbitMQinstancedothefollowing:
1. Runthiscommandtocreateaservicekey:
cfcreate-service-keySERVICE_INSTANCESERVICE_KEY-c'{"tags":"administrator"}'
where:
SERVICE_INSTANCE isthenameyousuppliedwhenyouran cfcreate-service .SERVICE_KEY isanameyouchoosetoidentifytheservicekey.
Forexample:
$cfcreate-service-keymy-instancemy-admin-key-c'{"tags":"administrator"}'
Creatingservicekeymy-admin-keyforserviceinstancemy-instanceasuser@example.com...OK
2. Runthiscommandtogettheadminusercredentials:
©CopyrightPivotalSoftwareInc,2013-2019 39 1.12
cfservice-keySERVICE_INSTANCESERVICE_KEY wherethevariablesarethesameasabove.
ThisreturnsaDashboardURLcontainingtheadmincredentials,whichcanbeusedtoaccessthemanagementUI.Forexample:
$cfservice-keymy-instancemy-admin-key
Gettingkeymy-admin-keyforserviceinstancemy-instanceasuser@example.com...{"dashboard_url":"https://my-instance.bosh-lite.com/#/login/admin-username/admin-password","username":"admin-username","password":"admin-password",...}
RabbitMQServerSettingsThatCannotBeDisabledThefollowingpluginsareenabledbydefaultandcannotbedisabled:
rabbitmq_management
rabbitmq_federation
rabbitmq_federation_management
rabbitmq_shovel
rabbitmq_shovel_management
©CopyrightPivotalSoftwareInc,2013-2019 40 1.12
InstallingandConfiguringthePre-ProvisionedServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideapre-provisionedservice.
TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .
Role-BasedAccessinOpsManagerOpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.
FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .
DownloadandInstalltheTile1. DownloadtheproductfilefromPivotalNetwork .
2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.
3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.
4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.
ConfigurePre-ProvisionedRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.
Note:Forinstructionsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtileasanon-demandservice,seeInstallingandConfiguringRabbitMQforPCFasanOn-DemandService.
©CopyrightPivotalSoftwareInc,2013-2019 41 1.12
WhichSettingsTabstoConfigureforthePre-ProvisionedServiceConfigurethefollowingtabsforthepre-provisionedservice:
RabbitMQSettingsTab Instructions
AssignAZsandNetworks AssignAZsandNetworks
Pre-ProvisionedRabbitMQ RabbitMQ
Networking Networking
©CopyrightPivotalSoftwareInc,2013-2019 42 1.12
Syslog SyslogGlobalSettingsforOn-DemandPlans GlobalSettings
Errands Errands
Stemcell Stemcell
AssignAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.
1. IntheSettingsscreen,clickAssignAZsandNetworks.
2. ConfigurethefieldsontheAssignAZsandNetworksasfollows.Allfieldsarerequired,thoughsomedonotapplytothepre-provisionedservice.
RequiredFields Instructions
Placesingletonjobsin
Selectaregion.Thisselectiononlyaffectstheon-demandservice.
Balanceotherjobsin
Selectadditionalregion(s).Thisselectiononlyaffectsthepre-provisionedservice.
Network
SelectanetworkfortheRabbitMQBroker.
ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.ThisnetworkisrepresentedbytheDefaultNetwork,describedinDefaultNetworkandServiceNetwork.Typically,youselectthenetworkusedforthePivotalApplicationService(PAS)orElasticRuntimecomponents.
ServiceNetwork
Selectanetwork.Thisselectiononlyaffectstheon-demandservice.
3. ClickSave.
RabbitMQToconfigurethefollowingsectionsinthePre-ProvisionedRabbitMQtab,intheSettingsscreen,clickPre-ProvisionedRabbitMQ.
RabbitMQAdminUserCredentialsIntheRabbitMQadminusercredentialsfieldoftheRabbitMQpane,enteranadminusernameandpassword:
Youcanuseacombinationofupperorlowercasealphanumericsandsupportedspecialcharacters: -=_+":;/?.><,~ .
ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.
Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesinthefinalstepbelow.
WARNING:ChangingtheNetworkafteryouhaveconfiguredit,orchangingtheIPconfiguration,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
Note:Youcannotusebackquote ` orsinglequote ' .
©CopyrightPivotalSoftwareInc,2013-2019 43 1.12
PluginsChoosewhichpluginsyouwanttoenableinthissectionofthePre-ProvisionedRabbitMQtab.
Youmustleavetherabbitmq_managementpluginenabledforthisproducttowork.
FormoreinformationaboutRabbitMQplugins,seetheRabbitMQdocumentation .
HAProxyPortsEntertheportsHAProxyshouldloadbalancetotheRabbitMQnodesinthissectionofthePre-ProvisionedRabbitMQtab:
Allthedefaultportsofalltheavailablepluginsareload-balancedbydefault.However,ifyouinstallextraprotocolplugins,orprovideacustomconfigurationthatchangestheportsRabbitMQlistenson,thenyoumustupdatethelistofload-balancedports.
Youmustleavethemanagementpluginlisteningonport 15672 andloadbalancethatport.
IfyouchangethetopologyofyourRabbitMQcluster,theHAProxyisautomaticallyreconfiguredduringthedeployment.
SSL(Optional)ProvideSSLcertificatesandkeysforusebytheRabbitMQclusterinthissectionofthePre-ProvisionedRabbitMQtab:
Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.
©CopyrightPivotalSoftwareInc,2013-2019 44 1.12
SSLissimultaneouslyprovidedforAMQPS,STOMPandMQTT.NootherpluginsareautomaticallyconfiguredforusewithSSL.
IfyouprovideSSLkeysandcertificates,non-SSLsupportisdisabled.IfyoupreviouslydeployedthisservicewithoutSSLsupportandhaveappsconnectedtotheservice,theseappslosetheirconnectionsandmustreconnectusingSSL.
SSLsettingsareappliedequallyacrossallVMsinthecluster.
YoucanprovidemorethenoneCAcertificate.
FormoreinformationaboutSSLsupport,seetheRabbitMQdocumentation .
ErlangCookie(Optional)ProvideanErlangcookietobeusedbytheclusterinthissectionofthePre-ProvisionedRabbitMQtab.ThisisusefulifyouwanttoconnectdirectlytotheRabbitMQcluster,forexamplewith rabbitmqctl ,ortoconnectotherVMsrunningErlang.
KnownIssueswithErlangCookie
TherearethreeknownissuesassociatedwiththeErlangcookie:
SecurityIssuewiththeTileGeneratedErlangCookie
ClusterScalingKnownIssue
ChangingtheErlangCookieValueKnownIssue
Note:Leavingthisfieldblankisasecurityrisk.SeeSecurityIssuewiththeTileGeneratedErlangCookiebelow.
©CopyrightPivotalSoftwareInc,2013-2019 45 1.12
SecurityIssuewiththeTileGeneratedErlangCookie
IfyouleavetheErlangcookiefieldblank,thetilegeneratesthecookieinawaythatcanbereverse-engineered.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .
Toavoidthisissue,settheErlangcookietoasecurepasswordvalue.Thisrequiresclusterdowntime,seeChangingtheErlangCookieValueKnownIssuebelow.
ClusterScalingKnownIssue
IfyouhavenotsettheErlangcookieandyouwanttoscaleoutyourclustersizewithoutdowntime,followthesesteps:
1. Followthestepsinthelinkbelow,uptoandincludingthesectionLogintotheBOSHDirector:AdvancedTroubleshootingwiththeBOSHCLI
2. Runthefollowingcommand:boshsshrabbitmq-server/0
3. Runthefollowingcommands:
sudo-iecho$(cat/var/vcap/store/rabbitmq/.erlang.cookie)
4. PastethevaluereturnedfromthelastcommandintotheErlangcookiefieldinthePre-ProvisionedRabbitMQtab.Thisfieldisshownabove.
5. Toincreasethesizeofyourcluster,navigatetotheResourceConfigtaband,inthefirstrow,raisethevalueforthenumberofInstancesoftheRabbitMQnode.
6. ReturntotheOpsManagerInstallationDashboard,andclickApplyChanges.
ChangingtheErlangCookieValueKnownIssue
ChangingtheErlangcookievaluerequiresclusterdowntime.Pivotalstronglyrecommendsthatyoudonotchangeanythingelseduringthistime,becauseitispossiblefortheconfigurationtobeinconsistentlyappliedduringthisprocess.
Thedeploymentmightfailafterthisprocess.Ifso,redeployingfixestheissue.
RabbitMQConfiguration(Optional)Provideafull rabbitmq.config filebypastingitscontentsintheRabbitMQconfigurationfieldinthePre-ProvisionedRabbitMQtab.Thisrabbitmq.config fileisthenprovidedtoallthenodesinthecluster.
TheinputinthisfieldmustbeBase64encoded.
Forexample,supposeyouwanttoconfigurethe rates_mode ofthe rabbitmq_management statsbelow:
[{rabbitmq_management,[{rates_mode,detailed}]}].
Note:BOSHtellsyouthatthecookiehaschanged—thisisbecausethedefaultvalueinthemanifestisempty,whichresultsinanauto-generatedcookie.However,thevalueofthecookieontheserverremainsthesame,sotheknownissuebelowdoesnotapply.
©CopyrightPivotalSoftwareInc,2013-2019 46 1.12
1. EncodethefileintoBase64:
WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K
2. PastetheaboveintotheRabbitMQconfigurationfield:
Youcanseeanexample rabbitmq.config filehere .FormoreinformationabouttheRabbitMQconfiguration,seetheRabbitMQdocumentation .
TLSSupportConfigureTLSinthissectionofthePre-ProvisionedRabbitMQtab:
TLSv1.0isdisabledbydefault,duetosecurityissues.
TLSv1.1andv1.2areenabledbydefaultandcanbeturnedonandoff.
ExternalLoadBalancer(Optional)EnteraDNSnameorIPaddressofanexternalloadbalancertobereturnedinthebindingcredentials( VCAP_SERVICES )toappdevelopers.EnterthisinthissectionofthePre-ProvisionedRabbitMQtab:
Ifyouconfigureanexternalloadbalancer,toavoidanunnecessaryVMdeployment,intheResourceConfigtabsettheHAProxyforRabbitMQinstancecountto0.
MetricsPollingIntervalThemetricspollingintervalissetinthissectionofthePre-ProvisionedRabbitMQtab:
Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.
©CopyrightPivotalSoftwareInc,2013-2019 47 1.12
Changingthissettingaffectsalldeployedinstances.
DiskFreeAlarmLimitChoosehowmuchdiskspaceRabbitMQattemptstokeepfreeatanygiventimeinthissectionofthePre-ProvisionedRabbitMQtab:
RabbitMQperiodicallychecksifthereissufficientfreespaceondisk.Ifthereisnot,RabbitMQtemporarilystopsacceptingnewmessages.Thisgivesyourappstimetoconsumeexistingmessages,andthusfreeupsomediskspace.TheRabbitMQtileprovidesfouroptionsforthisvalue:
50MBistheminimumvalue.(NotRecommended)Selecting50MBisnotrecommendedandcancausedataloss.Formoreinformation,seeDangersofSettingThisValueTooLowandWhentoUsethe50MBValuebelow.
100%MemoryensuresthatatthetimewhenRabbitMQcheckstheavailabledisk,theremustbeenoughspaceforRabbitMQtopageallmemory-basedmessagesouttodisk.
150%Memoryisrecommended.Thisisbecauseitispossiblethatinbetweendisk-spacechecks,RabbitMQmay:
Writepersistentmessagestodisk(usingupsomediskspace).Acceptmorememory-basedmessagesintovariousqueues.Pageallmemory-basedmessagestodisk.
Intheabovesituations,RabbitMQmightrequiremorefreediskthanithasmemory.
200%MemoryisaconservativevalueusedwhentheoperatorwantshigherconfidencethatRabbitMQneverrunsoutofdiskspace.
Formoreinformationaboutdiskalarms,seetheRabbitMQdocumentation .
DangersofSettingThisValueTooLow
IfthediskofagivenRabbitMQnodecompletelyfillswhileRabbitMQisrunning,thatnodecrashes.Thiscanleadtodataloss,andlossofavailability.
RabbitMQreservestherighttopageanyandallmessagesinmemory(eventransientmessages)todiskatanytime.YoumustsetyourDiskfreealarmlimithighenoughtoensurethatRabbitMQalwayshasatleastenoughspacetodothis.
DisadvantagesofSettingThisValueTooHigh
IfyousetyourDiskfreealarmlimittoavaluelargerthanthesizeofyourpersistentdisk,thenRabbitMQisnotabletofreeupenoughdiskspacetoacceptnewmessages.EnsurethatyouhavealargeenoughdisktopersistallthemessagesyouintendtopersistwhilealsoleavingenoughspacefreetosatisfytheDiskfreealarmlimitthatyouchoose.
WhentoUsethe50MBValue
Pivotaldoesnotrecommendusingthisvalueinproduction.However,ifyouareexperimentingwithadevelopmentenvironmentyoumightwanttouseasmalldisktokeepdowncosts,thoughthisincreasesthepossibilitythatRabbitMQcrashesandlosesdata.
NetworkingTospecifystaticIPaddresses,dothefollowing:
1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.
2. IntheSettingstab,selectNetworking.
©CopyrightPivotalSoftwareInc,2013-2019 48 1.12
3. Configurethefieldsasfollows:
Pre-ProvisionedHAProxyStaticIPs:Enteracomma-delimitedlistofIPaddressesgroupedintheorderofAZsconfigured.TheseIPaddressesareassignedtoyourPre-ProvisionedHAproxynodes.
Pre-ProvisionedRabbitMQServerStaticIPs:Enteracomma-delimitedlistofIPaddressesgroupedintheorderofAZsconfigured.TheseIPaddressesareassignedtoyourPre-ProvisionedRabbitMQServernodes.
Pre-ProvisionedServiceBrokerStaticIP:EnteranIPaddress.ThisaddressmustbeinthenetworkrangeofthenetworknamespecifiedintheNetworkdrop-downlistintheAssignAZsandNetworkstabandmustnotbeintheServiceNetwork.ThisIPaddressisassignedtoyourPre-ProvisionedServiceBrokernode.
4. ClickSave.
SyslogToenablemonitoringforRabbitMQforPCF,operatorsforwardthesyslogbydesignatinganexternalsyslogendpointforRabbitMQcomponentlogmessages.ThisendpointservesastheinputtoamonitoringplatformsuchasDatadog,Papertrail,orSumoLogic.
TospecifythedestinationforRabbitMQforPCFlogmessages,dothefollowing:
1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.
2. IntheRabbitMQtile,clicktheSettingstab.
Note:Ifanyoftheabovefieldsareleftblank,BOSHallocatesanIPaddress.
©CopyrightPivotalSoftwareInc,2013-2019 49 1.12
3. ClickSyslog.
4. ConfigurethefieldsontheSyslogpaneasfollows:
Option Description
Syslogaddress IPorDNSaddressofthesyslogserver
Syslogport Portofthesyslogserver
Transportprotocol
Transportprotocolofthesyslogserver.Oneof udp , tcp , relp .
Formatforlogs Formatforlogs.Pivotalrecommends RFC 5424 ,but Legacy Format canbeusedforcompatibilityreasons.
EnableTLS EnableTLStothesyslogserver.
PermittedPeerIfthereareseveralpeerserversthatcanrespondtoremotesyslogconnections,thenyoumayprovideawildcardinthedomain,suchas *.example.com .
CustomCACertificate
Iftheservercertificateisnotsignedbyaknownauthority,forexample,aninternalsyslogserver,providetheCAcertificateofthelogmanagementserviceendpoint.
©CopyrightPivotalSoftwareInc,2013-2019 50 1.12
5. ClickSave.
6. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoredeploywiththechanges.
GlobalSettingsFollowthestepsbelowtoenabletheshareableinstancesbetafeature.Sharingaserviceinstancebetweenspaces,allowsappsindifferentspacestosharedatabases,messagingqueues,andmanyothertypesofservices.Formoreinformation,seeSharingServiceInstances(Experimental) .
1. ClickGlobalSettings.
2. Select(Beta)ShareableInstancestoenablethebetafeatureforsharinginstances.
3. ClickSave.
Errands(Optional)IntheErrandstab,choosethedefaultsforwhenerrandsrun.
Errandscanbethoughtofastasks.Forexample,whendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofpost-deployerrands.AnexampleistheSmokeTestserrand,whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.
Youcandecidewhethertorunpost-deployerrandsbytogglingthemonoroffbeforeyouclickApplyChangestoupdateaconfigurationintheOpsManagerInstallationDashboard:
Thisisaone-timeactionbeforeanupdate.YoucanchangetheabovedefaultsintheErrandstab,aswellasthedefaultsforpre-deleteerrands.
WARNING:ThisisabetafeaturebasedonanexperimentalfeatureinCloudFoundry.Usethefeatureatyourownriskinnon-productionenvironments.
Important:InRabbitMQforPCFv1.9.0andlater,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,because
©CopyrightPivotalSoftwareInc,2013-2019 51 1.12
Formoreinformationonerrandrunrules,seeErrandRunRules .
Post-DeployErrands
Errand Description
BrokerRegistrar
Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace
SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTestsbelow.
RegisterOn-DemandServiceBroker
Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.
On-DemandInstanceSmokeTests
Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTests.
UpgradeAllServiceInstances
On-demandinstancesareupdatedandredeployediftherearechangestotheDedicatedInstancesettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoDedicatedInstancesettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.
Pre-ProvisionedInstanceSmokeTestsSmoketestsrunasapost-deploymenterrand.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.
Pre-DeleteErrandsPre-deleteerrandsrunafteranoperatorchoosestodeleteaproductintheOpsManagerInstallationDashboard,butbeforeOpsManagerfinishesdeletingtheproduct.
Errand Description
BrokerDeregistrar Removesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings
DeleteAllServiceInstancesUnbindsanddeletesexistingdedicatedserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.
DeregisterOn-DemandServiceBroker
Removestheon-demandRabbitMQservicefromtheMarketplace
StemcellForOpsManagerv2.0:
1. ClickStemcell.
2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .
ForOpsManagerv2.1:
1. FollowtheprocedureoutlinedinImportingandManagingStemcells .
ApplyConfigurationandCompletetheInstallation
thesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmightfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.
©CopyrightPivotalSoftwareInc,2013-2019 52 1.12
ReturntotheOpsManagerInstallationDashboardandclickApplyChangestocompletetheinstallationofRabbitMQforPCF.
OtherConfigurationTopics
ConnectingtoaHighlyAvailableRabbitMQClusterTheRabbitMQtile,allowsforahighlyavailableclusterthroughmultipleHAProxynodes.The hostnames , uris and hosts propertieshavebeenaddedandshouldbeusedinpreferenceovertheequivalentsingularproperties.Thesingularpropertiesaremaintainedforbackwardscompatibilityandalwayscontainthefirstvaluefromtheequivalentpluralproperty.Thesingularpropertieswilleventuallybedeprecated.
Forexample,withtwoHAProxyjobsdeployed,thefollowingpropertieswillbepresent:
"hostname":"10.0.0.41","hostnames":["10.0.0.41","10.0.0.51"]
Porttoprotocolmappings15672=Managementdashboard
5672=RabbitMQ
5671=RabbitMQSSL
1883=MQTT
8883=MQTTSSL
61613=STOMP
61614=STOMPSSL
15674=WebSTOMP
4567=RabbitMQServiceBroker
3457-3459=CFLoggregator
SecurityGroupsToenableaccesstotheRabbitMQtileservice,youmustensureyoursecuritygroupallowsaccesstotheHAProxyandRabbitMQServiceBrokerVMsconfiguredinyourdeployment.YoucanobtaintheIPaddressesforthesefromtheOpsManagerStatuspagefortheRabbitMQtile.EnsurethefollowingportsareenabledforthoseVMs:
15672
5672
5671
1883
8883
61613
61614
15674
4567
3457-3459
ThefollowingisatemplateforconfiguringyourCloudFoundrysecuritygroups:[{"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"},{"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"}]
©CopyrightPivotalSoftwareInc,2013-2019 53 1.12
ApplicationSecurityGroupsToallowthisservicetohavenetworkaccess,youmustcreateApplicationSecurityGroups (ASGs).
ApplicationContainerNetworkConnectionsApplicationcontainersthatuseinstancesoftheRabbitMQservicerequirethefollowingoutboundnetworkconnections:
Destination Ports Protocol Reason
HAProxy IPs 5672 tcp ApplicationcontainersusingAMQP
HAProxy IPs 5671 tcp ApplicationcontainersusingAMQPoverSSL
HAProxy IPs 1883 tcp ApplicationcontainersusingMQTT
HAProxy IPs 8883 tcp ApplicationcontainersusingMQTToverSSL
HAProxy IPs 61613 tcp ApplicationcontainersusingSTOMP
HAProxy IPs 61614 tcp ApplicationcontainersusingSTOMPoverSSL
HAProxy IPs 61613 tcp ApplicationcontainersusingWebSTOMP
CreateanASGnamed rabbitmq-app-containers withtheaboveconfigurationandbindittoeither:
Theappropriatespace
The default-running ASGsetifyouwanttoprovideaccesstoallstartedapps.Thenrestartyourapps.
Ifyouareusinganexternalloadbalancer,orhavemorethanoneIPaddressforHAProxy,youmustalsocreateegressrulesforthese.Forexample:
[{"ports":"5671-5672","protocol":"tcp","destination":"10.10.10.10/32"}]
AssignedIPsRabbitMQforPCFdoesnotsupportchangingtheIPaddresseswhichhavebeenassignedtotheRabbitMQdeployments.Forexample,youcannotchangethesubnetintowhichtheRabbitMQclusterwasoriginallyprovisioned.Doingsocausesthedeploymenttofail.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
PreservingDynamicallyAssignedIPsYoucannotswitchfromdynamicallyassignedIPaddressestoadifferentsetofstaticIPaddresses.However,youcanconfigureOpsManagersothecurrentsetofdynamicallyassignedIPaddressesalwayscontinuetobeused.Thismightbeusefulwhenupgrading.
Todothis,followthesesteps:
1. GototheStatuspageintheRabbitMQtile.
2. TakenoteoftheIPaddressesfortheRabbitMQServerandHAProxyforRabbitMQjobs,intheordernodesappearintheUI.
3. GototheSettingspage,andclickNetworking.
4. EntertheIPaddressesyougotfromtheStatuspageasacomma-separatedlist.
Note:TheserviceisunusablewithoutApplicationSecurityGroups.
©CopyrightPivotalSoftwareInc,2013-2019 54 1.12
5. ClickSave.
RabbitMQServerSettingsthatCannotbeOverwrittenInallcases:
rabbit halt_on_upgrade_failure false
rabbitmq_mqtt subscription_ttl 1800000
log_levels [{connection,info}]
halt_on_upgrade_failure false
{rabbit, [ {collect_statistics_interval, 60000}] }
{rabbitmq_management, [ {rates_mode, none}] }
WhenSSLisenabled:
rabbit tcp_listeners []
rabbit ssl_listeners [5671]
rabbitmq_management listener [{port,15672},{ssl,false}]
rabbitmq_mqtt ssl_listeners [8883]
rabbitmq_stomp ssl_listeners [61614]
©CopyrightPivotalSoftwareInc,2013-2019 55 1.12
SmokeTestsRabbitMQforPCFrunsasetofsmoketestsduringinstallationtoconfirmsystemhealth.
SmokeTestStepsThesmoketestsperformthefollowingforeachavailableserviceplan:
1. Targetstheorg system andcreatesaspacetorunthetests.
2. DeploysaninstanceoftheCFRabbitMQExampleApp tothisspace
3. CreatesaRabbitMQserviceinstanceandbindsittotheCFRabbitMQExampleApp
4. ChecksthattheCFRabbitMQExampleAppcanwritetoandreadfromtheRabbitMQserviceinstance
5. Cleansupalldeployedapplicationandallitsservicebindings.Finally,thecfspaceisdeleted.
TroubleshootingIferrorsoccurwhilethesmoketestsrun,theyaresummarizedattheendoftheerrandlogoutput.Detailedlogscanbefoundwherethefailureoccurs.
Whenencounteringanerrorwhenrunningsmoketests,itcanbehelpfultosearchthelogforotherinstancesoftheerrorsummaryprintedattheendofthetests,forexample, FailedtotargetCloudFoundry .Lookoutfor TIP:... inthelogsnexttoanyerroroutputforfurthertroubleshootinghints.
Note:Smoketestsfailunlessyouenableglobaldefaultapplicationsecuritygroups(ASGs).YoucanenableglobaldefaultASGsbybindingtheASGtothe system orgwithoutspecifyingaspace.ToenableglobaldefaultASGs,use cfbind-running-security-
group.
©CopyrightPivotalSoftwareInc,2013-2019 56 1.12
MonitoringandKPIsforPre-ProvisionedRabbitMQforPCFThistopicexplainshowtomonitorthehealthofthepre-provisionedversionoftheRabbitMQforPivotalCloudFoundry(PCF)serviceusingthelogs,metrics,andKeyPerformanceIndicators(KPIs)generatedbyRabbitMQforPCFcomponentVMs.
Pre-provisionedRabbitMQforPCFcomponentsgeneratemanyofthesamemetrics astheon-demandRabbitMQservicecomponents.
SeeLoggingandMetrics forgeneralinformationaboutloggingandmetricsinPCF.
SettingupSyslogForwardingOperatorscanenablelogforwardingbyconfiguringanexternalsyslogendpointforRabbitMQcomponentlogmessages.Forinstructionsonsettingupsyslogforwarding,seeSyslog.
Ifsyslogforwardingisenabled,logentrieswithtimestampsareavailablelocallyin /var/log/messages .Logsareavailableunder /var/vcap/sys/log/ whethersyslogforwardingisenabledornot.
LoggingFormatsWithpre-provisionedRabbitMQforPCFloggingconfigured,threetypesofcomponentgeneratelogs:theRabbitMQmessageservernodes,theservicebroker,andHAProxy.IfyouhavemultipleserverorHAProxynodes,youcanidentifylogsfromindividualnodesbytheirindex,whichcorrespondstotheindexoftheRabbitMQVMinstancesdisplayedinOpsManager:
ThelogsforRabbitMQservernodesfollowtheformat [job=rabbitmq-server-partition-GUID index=X]
ThelogsforHAProxynodesfollowtheformat [job=rabbitmq-haproxy-partition-GUID index=X]
ThelogsfortheRabbitMQservicebrokerfollowtheformat [job=rabbitmq-broker-partition-GUID index=X]
RabbitMQandHAProxyserverslogatthe info levelandcaptureerrors,warnings,andinformationalmessages.
Forusersfamiliarwithdocumentationforpreviousversionsofthetile,thetagweusedtocallthe app_name isnowcalledthe program_name .
Thegenericlogformatisasfollows:
<PRI>TIMESTAMPIP_ADDRESSPROGRAM_NAME[job=NAMEindex=JOB_INDEXid=JOB_ID]MESSAGE
Therawlogslooksimilartothefollowing:
<7>2017-06-28T16:06:10.733560+00:0010.244.16.133vcap.agent[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]2017/06/2816:06:10CEF:0|CloudFoundry|BOSH|1|agent_api|ssh|1|duser=director.be5a66bb-a9b4-459f-a0d3-1fc5c9c3ed79.be148cc6-91ef-4eed-a788-237b0b8c63b7src=10.254.50.4spt=4222shost=5ae233e0-ecc5-4868-9ae0-f9767571251b<86>2017-06-28T16:06:16.704572+00:0010.244.16.133useradd[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]newgroup:name=bosh_ly0d2rbjr,GID=1003<86>2017-06-28T16:06:16.704663+00:0010.244.16.133useradd[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]newuser:name=bosh_ly0d2rbjr,UID=1001,GID=1003,home=/var/vcap/bosh_ssh/bosh_ly0d2rbjr,shell=/bin/bash<86>2017-06-28T16:06:16.736932+00:0010.244.16.133usermod[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]add'bosh_ly0d2rbjr'togroup'admin'<86>2017-06-28T16:06:16.736964+00:0010.244.16.133usermod[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]add'bosh_ly0d2rbjr'togroup'vcap'
LogssenttoexternalloggingtoolssuchasPapertrailmaybepresentedinadifferentformat.
Thefollowingtabledescribestheloggingtagsusedinthistemplate:
Tag Description
PRI Thisisavaluewhichinfuturewillbeusedtodescribetheseverityofthelogmessageandwhichfacilityitcamefrom.
TIMESTAMPThisisthetimestampofwhenthelogisforwarded,forexample, 2016-08-24T05:14:15.000003Z .Thetimestampvalueistypicallyslightlyafterwhenthelogmessagewasgenerated.
IP_ADDRESS TheinternalIPaddressofserveronwhichthelogmessageoriginated
PROGRAM_NAMEProcessnameoftheprogramthegeneratedthemessage.Sameas app_name beforev1.9.0.Formoreinformationaboutprogramname,seeRabbitMQProgramNamesbelow.
NAME TheBOSHinstancegroupname(forexample, rabbitmq_server )
JOB_INDEX BOSHjobindex.Usedtodistinguishbetweenmultipleinstancesofthesamejob.
JOB_IDBOSHVMGUID.ThisisdistinctfromtheCIDdisplayedintheOpsManagerStatustab,whichcorrespondstotheVMIDassignedbytheinfrastructureprovider.
©CopyrightPivotalSoftwareInc,2013-2019 57 1.12
MESSAGE Thelogmessagethatappears
RabbitMQProgramNames
ProgramName Description
rabbitmq_server_cluster_check ChecksthattheRabbitMQclusterishealthy.Runsaftereverydeploy.
rabbitmq_server_node_check ChecksthattheRabbitMQnodeishealthy.Runsaftereverydeploy.
rabbitmq_route_registrar_stderrRegisterstherouteforthemanagementAPIwiththeGorouterinyourPivotalApplicationService(PAS)orElasticRuntimedeployment.
rabbitmq_route_registrar_stdout RegisterstherouteforthemanagementAPIwiththeGorouterinyourPASorElasticRuntimedeployment.
rabbitmq_server TheErlangVMandRabbitMQapps.Logsmayspanmultiplelines.
rabbitmq_server_drain ShutsdowntheErlangVMandRabbitMQapps.RunsaspartoftheBOSHlifecycle.
rabbitmq_server_http_api_access AccesstotheRabbitMQmanagementUI.
rabbitmq_server_init StartstheErlangVMandRabbitMQ.
rabbitmq_server_post_deploy_stderr Runsthenodecheckandclustercheck.Runsaftereverydeploy.
rabbitmq_server_post_deploy_stdout Runsthenodecheckandclustercheck.Runsaftereverydeploy.
rabbitmq_server_pre_start Runsbeforetherabbitmq-serverjobisstarted.
rabbitmq_server_sasl Supervisor,progress,andcrashreportingfortheErlangVMandRabbitMQapps.
rabbitmq_server_shutdown_stderr StopstheRabbitMQappandErlangVM.
rabbitmq_server_shutdown_stdout StopstheRabbitMQappandErlangVM.
rabbitmq_server_startup_stderr StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.
rabbitmq_server_startup_stdout StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.
rabbitmq_server_upgrade ShutsdownErlangVMandRabbitMQappifrequiredduringanupgrade.
MetricsMetricsareregularly-generatedlogmessagesthatreportmeasuredcomponentstates.Themetricspollingintervaldefaultsto30seconds.ThemetricspollingintervalisaconfigurationoptionontheRabbitMQtile(Settings>RabbitMQ).Theintervalsettingappliestoallcomponentsdeployedbythetile.
Metricsarelong,singlelinesoftextthatfollowtheformat:
origin:"p-rabbitmq"eventType:ValueMetrictimestamp:1441188462382091652deployment:"cf-rabbitmq"job:"cf-rabbitmq-node"index:"0"ip:"10.244.3.46"valueMetric:<name:"/p-rabbitmq/rabbitmq/system/memory"value:1024unit:"MB">
PartitionIndicatorAnewmetrichasbeenintroducedtohelptoidentifynetworkpartitions.Essentiallyitexposeshowmanynodeseachnodeknows.Whenanodeisinpartitiontheonlynodethatitrecognizesisitselfandthatisagoodindicationthatthatnodemightbeinapartition.
Anexampleofthatmetricsis:
origin:"p-rabbitmq"eventType:ValueMetrictimestamp:1441188462382091652deployment:"cf-rabbitmq"job:"cf-rabbitmq-node"index:"0"ip:"10.244.3.46"valueMetric:<name:"/p-rabbitmq/rabbitmq/erlang/reachable_nodes"value:3unit:"count">
Monitorscanbecreatedtoemitalertsincaseaclusterseemstobeinapartition.Ametricsisemittedforeachnodeinthecluster.Forexample:inathree-nodeclusteramonitorcanexpecttohaveatotalof9(nine)sinceeachnodeisexpectedtoemit3(2reachablenodesanditself).Otherwise,analertcanbesenttotheteam.
RecoveringfromanetworkpartitionPleaserefertotheoficialRabbitMQguidetounderstandhowtorecoverfromanetworkpartition:https://www.rabbitmq.com/partitions.html
©CopyrightPivotalSoftwareInc,2013-2019 58 1.12
KeyPerformanceIndicatorsKeyPerformanceIndicators(KPIs)forRabbitMQforPCFaremetricsthatoperatorsfindmostusefulformonitoringtheirRabbitMQservicetoensuresmoothoperation.KPIsarehigh-signal-valuemetricsthatcanindicateemergingissues.KPIscanberawcomponentmetricsorderivedmetricsgeneratedbyapplyingformulastorawmetrics.
PivotalprovidesthefollowingKPIsasgeneralalertingandresponseguidancefortypicalRabbitMQforPCFinstallations.Pivotalrecommendsthatoperatorscontinuetofine-tunethealertmeasurestotheirinstallationbyobservinghistoricaltrends.Pivotalalsorecommendsthatoperatorsexpandbeyondthisguidanceandcreatenew,installation-specificmonitoringmetrics,thresholds,andalertsbasedonlearningfromtheirowninstallations.
ForalistofallRabbitMQforPCFrawcomponentmetrics,seeComponentMetricsReference.
ComponentHeartbeatsKeyRabbitMQforPCFcomponentsperiodicallyemitheartbeatmetrics:theRabbitMQservernodes,HAProxynodes,andtheServiceBroker.TheheartbeatsareBooleanmetrics,where 1 meansthesystemisavailable,and 0 ortheabsenceofaheartbeatmetricmeanstheserviceisnotrespondingandshouldbeinvestigated.
ServiceBrokerHeartbeat
p-rabbitmq.service_broker.heartbeat
Description
RabbitMQServiceBroker is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:IftheServiceBrokerdoesnotemitheartbeats,thisindicatesthatitisoffline.TheServiceBrokerisrequiredtocreate,update,anddeleteserviceinstances,whicharecriticalfordependenttilessuchasSpringCloudServicesandSpringCloudDataFlow.
Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1
RecommendedresponseChecktheRabbitMQServiceBrokerlogsforerrors.YoucanfindthisVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningthefollowingcommand:bosh -d service-instance_GUID vms
HAProxyHeartbeat
p-rabbitmq.haproxy.heartbeat
Description
RabbitMQHAProxy is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:IftheHAProxydoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireHAProxy.
Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1
ChecktheRabbitMQHAProxylogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwith
©CopyrightPivotalSoftwareInc,2013-2019 59 1.12
Recommendedresponse BOSHandrunningthefollowingcommand,whichlists HAProxy_GUID :bosh -d service-instance_GUID vms
ServerHeartbeat
p-rabbitmq.rabbitmq.heartbeat
Description
RabbitMQServer is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:Iftheserverdoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireRabbitMQServer.
Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1
RecommendedresponseChecktheRabbitMQServerlogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommands,whichlists rabbitmq :bosh -d service-instance_GUID vms
RabbitMQServerKPIsThefollowingKPIsfromtheRabbitMQservercomponent:
FileDescriptors
p-rabbitmq.rabbitmq.system.file_descriptors
Description
Filedescriptorsconsumed.
Use:Ifthenumberoffiledescriptorsconsumedbecomestoolarge,theVMmaylosetheabilitytoperformdiskIO,whichcancausedataloss.
Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>250000Redcritical:>280000
Recommendedresponse
Thedefault ulimit forRabbitMQforPCFis300000.Ifthismetricismetorexceededforanextendedperiodoftime,consideroneofthefollowingactions:
ScalingtherabbitnodesinthetileResourceConfigpane.
Reducetheloadontheserver
ErlangProcesses
p-rabbitmq.rabbitmq.erlang.erlang_processes
Erlang processesconsumedbyRabbitMQ,whichrunsonanErlangVM.
Note:Thisassumesnon-persistentmessagesarehandledbyretriesorsomeotherlogicbytheproducers.
©CopyrightPivotalSoftwareInc,2013-2019 60 1.12
DescriptionUse:Thisisthekeyindicatoroftheprocessingcapabilityofanode.
Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>900000Redcritical:>950000
RecommendedresponseThedefaultErlangprocesslimitinRabbitMQforPCFv1.6andlateris1,048,816.Ifthismetricmeetsorexceedstherecommendedthresholdsforextendedperiodsoftime,considerscalingtheRabbitMQnodesinthetileResourceConfigpane.
BOSHSystemHealthMetricsTheBOSHlayerthatunderliesPCFgenerates healthmonitor metricsforallVMsinthedeployment.AsofPCFv2.0,thesemetricsareincludedintheLoggregatorFirehosebydefault.Formoreinformation,seeBOSHSystemMetricsAvailableinLoggregatorFirehose inPivotalApplicationService(PAS)ReleaseNotes.
AllBOSH-deployedcomponentsgeneratethesystemhealthmetricsbelow.ThesecomponentmetricsarefromRabbitMQforPCFcomponents,andserveasKPIsfortheRabbitMQforPCFservice.
RAM
system.mem.percent
Description
RAMbeingconsumedbythe p-rabbitmq VM.
Use:RabbitMQisconsideredtobeinagoodstatewhenithaslittleornomessages.Inotherwords,“anemptyrabbitisahappyrabbit.”Alertingonthismetriccanindicatethattherearetoofewconsumersorappsthatreadmessagesfromthequeue.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsRAMforthepasttenminutes.
Origin:BOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>40Redcritical:>50
Recommendedresponse Addmoreconsumerstodrainthequeueasfastaspossible.
CPU
system.cpu.percent
Description
CPUbeingconsumedbythe p-rabbitmq VM.
Use:AnodethatexperiencescontextswitchingorhighCPUusagewillbecomeunresponsive.Thisalsoaffectstheabilityofthenodetoreportmetrics.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:BOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
©CopyrightPivotalSoftwareInc,2013-2019 61 1.12
Recommendedalertthresholds Yellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
EphemeralDisk
system.disk.percent
Description
EphemeralDiskbeingconsumedbythe p-rabbitmq VM.
Use:Ifsystemdiskfillsup,therearetoofewconsumers.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:BOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
PersistentDisk
persistent.disk.percent
Description
PersistentDiskbeingconsumedbythe p-rabbitmq VM.
Use:Ifsystemdiskfillsup,therearetoofewconsumers.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:BOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
ComponentMetricReferenceRabbitMQforPCFcomponentVMsemitthefollowingrawmetrics.Thefullnameofthemetricfollowstheformat: /p-rabbitmq/COMPONENT/METRIC-NAME
RabbitMQServerMetricsRabbitMQforPCFmessageservercomponentsemitthefollowingmetrics.
FullName Unit Description
/p-rabbitmq.rabbitmq.heartbeat boolean IndicateswhethertheRabbitMQserverisavailableandabletorespondtorequests
/p-rabbitmq/rabbitmq/erlang/erlang_processes
count ThenumberofErlangprocesses
/p-rabbitmq/rabbitmq/system/memory MB ThememoryinMBusedbythenode
©CopyrightPivotalSoftwareInc,2013-2019 62 1.12
/p-rabbitmq/rabbitmq/system/mem_alarm boolean Indicatesifthememoryalarmwentoff
/p-rabbitmq/rabbitmq/system/disk_free_alarm boolean Indicatesifthediskfreealarmwentoff
/p-rabbitmq/rabbitmq/system/disk_free MB Thediskspaceavailableonthenode
/p-rabbitmq/rabbitmq/connections/count count Thetotalnumberofconnectionstothenode
/p-rabbitmq/rabbitmq/consumers/count count Thetotalnumberofconsumersregisteredinthenode
/p-rabbitmq/rabbitmq/messages/delivered count Thetotalnumberofmessageswiththestatus deliver_get onthenode
/p-rabbitmq/rabbitmq/messages/delivered_noack
count Thenumberofmessageswiththestatus deliver_noack onthenode
/p-rabbitmq/rabbitmq/messages/delivered_rate
rateTheratepersecondatwhichmessagesarebeingdeliveredtoconsumersorclientsonthenode
/p-rabbitmq/rabbitmq/messages/published count Thetotalnumberofmessageswiththestatus publish onthenode
/p-rabbitmq/rabbitmq/messages/published_rate
rate Theratepersecondatwhichmessagesarebeingpublishedbythenode
/p-rabbitmq/rabbitmq/messages/redelivered count Thetotalnumberofmessageswiththestatus redeliver onthenode
/p-rabbitmq/rabbitmq/messages/redelivered_rate rate
Theratepersecondatwhichmessagesaregettingthestatus redeliver onthenode
/p-rabbitmq/rabbitmq/messages/get_no_ack count Thenumberofmessageswiththestatus get_no_ack onthenode
/p-rabbitmq/rabbitmq/messages/get_no_ack_rate
rate Theratepersecondatwhichmessagesgetthestatus get_no_ack onthenode
/p-rabbitmq/rabbitmq/messages/pending count Thenumberofmessageswiththestatus messages_unacknowledged onthenode
/p-rabbitmq/rabbitmq/messages/depth countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready onthenode
/p-rabbitmq/rabbitmq/system/file_descriptors
count Thenumberofopenfiledescriptorsonthenode
/p-rabbitmq/rabbitmq/exchanges/count count Thetotalnumberofexchangesonthenode
/p-rabbitmq/rabbitmq/messages/available count Thetotalnumberofmessageswiththestatus messages_ready onthenode
/p-rabbitmq/rabbitmq/queues/count count Thenumberofqueuesonthenode
/p-rabbitmq/rabbitmq/channels/count count Thenumberofchannelsonthenode
/p-rabbitmq/rabbitmq/vhosts/count count Thenumberofvhosts
/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/consumers
count Thenumberofconsumerspervirtualhostperqueue
/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/depth
countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready pervirtualhostperqueue
HAProxyMetricsRabbitMQforPCFHAProxycomponentsemitthefollowingmetrics.
NameSpace Unit Description
/p-rabbitmq.haproxy.heartbeat booleanIndicateswhethertheRabbitMQHAProxycomponentisavailableandabletorespondtorequests
/p-rabbitmq/haproxy/health/connections
count Thetotalnumberofconcurrentfront-endconnectionstotheserver
/p-rabbitmq/haproxy/backend/qsize/amqp
size ThetotalsizeoftheAMQPqueueontheserver
/p-rabbitmq/haproxy/backend/retries/amqp count ThenumberofAMQPretriestotheserver
/p-rabbitmq/haproxy/backend/ctime/amqp
time ThetotaltimetoestablishtheTCPAMQPconnectiontotheserver
©CopyrightPivotalSoftwareInc,2013-2019 63 1.12
©CopyrightPivotalSoftwareInc,2013-2019 64 1.12
SettingLimitsforOn-DemandServiceInstancesOn-demandprovisioningisintendedtoaccelerateappdevelopmentbyeliminatingtheneedfordevelopmentteamstorequestandwaitforoperatorstocreateaserviceinstance.However,tocontrolcosts,operationsteamsandadministratorsmustensureresponsibleuseofresources.
Thereareseveralwaystocontroltheprovisioningofon-demandserviceinstancesbysettingvariousquotasattheselevels:
Global
Plan
Org
Space
Afteryousetquotas,youcan:
ViewCurrentOrgandSpace-levelQuotas
MonitorQuotaUseandServiceInstanceCount
CalculateResourceCostsforOn-DemandPlans
CreateGlobal-levelQuotasEachPivotalCloudFoundry(PCF)servicehasaseparateservicebroker.Aglobalquotaattheservicelevelsetsthemaximumnumberofserviceinstancesthatcanbecreatedbyagivenservicebroker.Ifaservicehasmorethanoneplan,thenthenumberofserviceinstancesforallplanscombinedcannotexceedtheglobalquotafortheservice.
TheoperatorsetsaglobalquotaforeachPCFserviceindependently.Forexample,ifyouhaveRedisforPCFandRabbitMQforPCF,youmustsetaseparateglobalservicequotaforeachofthem.
Whentheglobalquotaisreachedforaservice,nomoreinstancesofthatservicecanbecreatedunlessthequotaisincreased,orsomeinstancesofthatservicearedeleted.
CreatePlan-levelQuotasAservicemayofferoneormoreplans.Youcansetaseparatequotaperplansothatinstancesofthatplancannotexceedtheplanquota.Foraservicewithmultipleplans,thetotalnumberofinstancescreatedforallplanscombinedcannotexceedtheglobalquotafortheservice.
Whentheplanquotaisreached,nomoreinstancesofthatplancanbecreatedunlesstheplanquotaisincreasedorsomeinstancesofthatplanaredeleted.
CreateandSetOrg-levelQuotasAnorg-levelquotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesanorganizationcancreatewithinPCF.Forexample,ifyousetyourorg-levelquotato100,developerscancreateupto100serviceinstancesinthatorgusinganycombinationofPCFservices.
Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedintheorgunlessthequotaisincreasedorsomeserviceinstancesaredeleted.
Tocreateandsetanorg-levelquota,dothefollowing:
1. Runthiscommandtocreateaquotaforserviceinstancesattheorglevel:
cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans
Where:
QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg
©CopyrightPivotalSoftwareInc,2013-2019 65 1.12
Forexample:
cfcreate-quotamyquota-m1024mb-i16gb-r30-s50--allow-paid-service-plans
2. Associatethequotayoucreatedabovewithaspecificorgbyrunningthefollowingcommand:
cf set-quota ORG-NAME QUOTA-NAME
Forexample:
cfset-quotadev_orgmyquota
Formoreinformationonmanagingorg-levelquotas,seeCreatingandModifyingQuotaPlans .
CreateandSetSpace-levelQuotasAspace-levelservicequotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesthatcanbecreatedwithinagivenspaceinPCF.Forexample,ifyousetyourspace-levelquotato100,developerscancreateupto100serviceinstancesinthatspaceusinganycombinationofPCFservices.
Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedinthespaceunlessthequotaisupdatedorsomeserviceinstancesaredeleted.
Tocreateandsetaspace-levelquota,dothefollowing:
1. Runthefollowingcommandtocreatethequota:
cf create-space-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans
Where:
QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg
Forexample:
cfcreate-space-quotamyspacequota-m1024mb-i16gb-r30-s50--allow-paid-service-plans
2. Associatethequotayoucreatedabovewithaspecificspacebyrunningthefollowingcommand:
cf set-space-quota SPACE-NAME QUOTA-NAME
Forexample:
cfset-space-quotamyspacemyspacequota
Formoreinformationonmanagingspace-levelquotas,seeCreatingandModifyingQuotaPlans .
ViewCurrentOrgandSpace-levelQuotasTovieworgquotas,runthefollowingcommand.
cforgORG-NAME
Toviewspacequotas,runthefollowingcommand:
cfspaceSPACE-NAME
©CopyrightPivotalSoftwareInc,2013-2019 66 1.12
Formoreinformationonmanagingorgandspace-levelquotas,seetheCreatingandModifyingQuotaPlans .
MonitorQuotaUseandServiceInstanceCountService-levelandplan-levelquotause,andtotalnumberofserviceinstances,areavailablethroughtheon-demandbrokermetricsemittedtoLoggregator.Thesemetricsarelistedbelow:
MetricName Description
on-demand-broker/SERVICE-NAME/quota_remaining Quotaremainingforallinstancesacrossallplans
on-demand-broker/SERVICE-NAME/PLAN-NAME/quota_remaining
Quotaremainingforaspecificplan
on-demand-broker/SERVICE-NAME/total_instances Totalinstancescreatedacrossallplans
on-demand-broker/SERVICE-NAME/PLAN-NAME/total_instances
Totalinstancescreatedforaspecificplan
CalculateResourceCostsforOn-DemandPlansOn-demandplansusededicatedVMs,disks,andvariousotherresourcesfromanIaaS,suchasAWS.Tocalculatemaximumresourcecostforplansindividuallyorcombined,youmultiplythequotabythecostoftheresourcesselectedintheplanconfiguration(s).ThespecificcostsdependonyourIaaS.
TheimagebelowshowsanexamplethatincludestheVMtypeandpersistentdiskselectedfortheserverVMs,aswellasthequotaforthisplan.
Note:Quotametricsarenotemittedifnoquotahasbeenset.
Important:Althoughoperatorscanlimiton-demandinstanceswithplanquotasandaglobalquota,asdescribedintheabovetopics,IaaSresourceusagestillvariesbasedonthenumberofon-demandinstancesprovisioned.
©CopyrightPivotalSoftwareInc,2013-2019 67 1.12
CalculateMaximumResourceCostPerOn-DemandPlanTocalculatethemaximumcostofVMsandpersistentdiskforeachplan,dothefollowingcalculation:
planquotaxcostofselectedresources
Forexample,ifyouselectedtheoptionsintheaboveimage,youhaveselectedaVMtypemicroandapersistentdisktype20GB,andtheplanquotais15.TheVMandpersistentdisktypeshaveanassociatedcostfortheIaaSyouareusing.Therefore,tocalculatethemaximumcostofresourcesforthisplan,multiplythecostoftheresourcesselectedbytheplanquota:
(15xcostofmicroVMtype)+(15xcostof20GBpersistentdisk)=maxcostperplan
CalculateMaximumResourceCostforAllOn-DemandPlansTocalculatethemaximumcostforallplanscombined,addtogetherthemaximumcostsforeachplan.Thisassumesthatthesumofyourindividualplanquotasislessthantheglobalquota.
Hereisanexample:
(plan1quotaxplan1resourcecost)+(plan2quotaxplan2resourcecost)=maxcostforallplans
CalculateActualResourceCostofallOn-DemandPlansTocalculatethecurrentactualresourcecostacrossallyouron-demandplans:
1. Findthenumberofinstancescurrentlyprovisionedforeachactiveplanbylookingatthe total_instance metricforthatplan.
2. Multiplythe total_instance countforeachplanbythatplan’sresourcecosts.Recordthecostsforeachplan.
3. AddupthecostsnotedinStep2togetyourtotalcurrentresourcecosts.
Forexample:
(plan1total_instancesxplan1resourcecost)+(plan2total_instancesxplan2resourcecost)=currentcostforallplans
©CopyrightPivotalSoftwareInc,2013-2019 68 1.12
ControllingAccesstoServicePlansbyOrgIfyouwanttorestrictaccesstoaserviceplantoaspecificorg,followtheinstructionsbelow.
Youcanalsolimitthenumberofserviceinstancesbysettingquotas—forinstructions,seeSettingLimitsforOn-DemandInstances.
ChangeAccesstoServicePlans
Torestrictaccesstoaplanforaspecificorg,runthiscommand:
cfenable-service-accessp.rabbitmq-pPLAN_NAME-oORG_NAME
Forexample:
$cfenable-service-accessp.rabbitmq-pmy-cluster-plan-omy-dev-org
Formoreinformationabouttheabovecommand,seeAccessControl .
Note:Iftheplanyouarerestrictingiscurrentlyenabledforallorgs,youmustfirstDisableServiceAccessfortheplan,thengrantaccesstotheplantospecificorgs.UsetheCFServiceAccessfieldtodisableaccessintheserviceplanconfiguration.
©CopyrightPivotalSoftwareInc,2013-2019 69 1.12
IsolatingClusterswiththeRabbitMQforPCFReplicator
OverviewRabbitMQforPCFReplicatorisatoolthatallowsyoutoinstallmultipleRabbitMQforPivotalCloudFoundry(PCF)tilesinasingleOpsManagerenvironment.Thisletsyourunmultiplepre-provisionedRabbitMQclustersthatareisolatedfromeachother.
Forexample,youmaywanttoisolatetheclusterservingSpringCloudServices(SCS)fromtheclusterservingappsintheMarketplace.Oryoumaywanttogiveacertainteamtheirowndedicated,pre-provisionedclusterthatyoumanageforthem.Forinformationonhowtoaccomplishthesescenarios,seeCommonUseCases.
CommonUseCasesTheimagebelowillustrateshowtoisolateSCSonRabbitMQforPCFfromclusteredandsinglenodeserviceinstancesdedicatedtodifferentteams.Inthisusecase:
TheunreplicatedRabbitMQforPCFtiledeploystwotypesofservices:
Apre-provisionedserviceisusedasabackingserviceforSCSAnon-demandserviceisusedtocreatethreecompletelyisolatedsinglenodeserviceinstances
Tworeplicatilesareusedtocreatetwodedicatedpre-provisionedclusters,witheachonededicatedtoaspecificteam.
ThesescenariosareexplainedbelowinRunningSCSonaDedicatedRabbitMQClusterandProvidingaPre-ProvisionedDedicatedCluster.
©CopyrightPivotalSoftwareInc,2013-2019 70 1.12
RunningSCSonaDedicatedRabbitMQClusterReplicaRabbitMQtilescannotbeusedtoprovideabackingserviceforSpringCloudServices(SCS)becauseSCSexpectsthattheserviceiscalledp-rabbitmq .Therefore,ifyouwanttoisolatetheRabbitMQclusterthatisusedbySCSfromothertenants,youcanreservetheunreplicatedRabbitMQforPCFtileforSCS,asshowninthediagrambelow.YoucanthenaddreplicaRabbitMQclustersforusebyappsintheMarketplace.
PivotalrecommendsthatyouusetheunreplicatedRabbitMQforPCFtilesolelyforSCStoavoidcontentionbetweenappsusingSCS,andappsusingRabbitMQforPCFintheMarketplace.
ToreservetheunreplicatedtileforSCS,turnofftheBrokerRegistrarerrandtopreventthebrokerfrombeingexposedintheMarketplace.Formoreinformation,seeErrands.
ToofferRabbitMQasacloudmessagingserviceintheMarketplace,createoneorseveralreplicas,installtheminOpsManager,andeitherallowthebrokerregistrarerrandtorun,orregistertheservicemanuallyusingCF .
ProvidingaPre-ProvisionedDedicatedClusterToreserveaRabbitMQclusterforusebyaspecificteam,disabletheBrokerRegistrarerrandinOpsManager.ThispreventsserviceregistrationintheMarketplace.Formoreinformation,seeErrands.
Afteryoudeploythetile,manuallyexposetheservicebrokertoyourdesiredorgsandspaces.Forinstructions,seeRegisteraBroker .
UsingReplicasWhileOfferingtheOn-DemandRabbitMQServiceTheOn-Demandserviceisnotofferedinreplicatiles,sincethepurposeofthereplicatoristocreateadditionalpre-provisionedclusters.Ifyouwishtoofferon-demandserviceplans,usetheunreplicatedRabbitMQforPCFtileasshowninthediagramabove.
Blue-GreenUpgrades(Advanced)Inordertodoblue-greenstyleupgradestominimizedowntime,youcanstandupanewclusterandmigratedataandusersovertothenewclusteroveraperiodoftime.SpeakwithyourPlatformArchitectabouthowtoenablethisworkflow.
GeneratingReplicaTilesThistopicdescribeshowtoinstallthereplicatorandgeneratereplicatilesofRabbitMQforPCF.
PrerequisitesRabbitMQforPCFv1.8.xorv1.9.x
2.5GBoffreediskspace
DownloadtheReplicatorTheRabbitMQforPCFReplicatoriscurrentlyavailablefromPivotalNetwork .Searchforanddownloadthisarchive,andthenruntheenclosedbinary.
GenerateReplicaTilesThefollowingisthesyntaxforgeneratingareplicatile:
./rabbitmq-replicator-darwin\--nameYOUR_DESIRED_TILE_NAME\--pathPATH_TO_TILE\--outputDESIRED_FILE_NAME.pivotal
Thefollowingaretheparametersexpectedintheabovesyntax:
©CopyrightPivotalSoftwareInc,2013-2019 71 1.12
Parameter Description
nameThedesireduniqueidentifierforthereplicatile,whichisusedtogeneratemanifests,deploymentnames,andMarketplacenamefortheservice.Onlyalphanumericcharactersand - areacceptedbythetool.
path Thelocationoftheoriginal,unreplicated,RabbitMQforPCFsourcetilethatyoudownloaded
output Thedesiredfilenameandpathforthereplicatile
NamingConventionsinOriginalandReplicaTilesThetablebelowshowsthenamingconventionsforvariouscomponentsrelatedtotheoriginalRabbitMQforPCFtileandtothereplicatile.
Forthepurposesofthisexample,assumethatwhenyougenerateareplicatileasshownabove,inthe name fieldyouprovidethestring finance\ .Thentheattributesfortheoriginalandreplicatilesareasfollows:
Component NamewithOriginalTile NamewithReplicaTile
Brokername p-rabbitmq p-rabbitmq-finance
BrokerURL pivotal-rabbitmq-broker.YOUR_CF_DOMAIN pivotal-rabbitmq-broker-finance.YOUR_CF_DOMAIN
Servicename p-rabbitmq p-rabbitmq-finance
URLfortheRabbitMQManagementUIDashboard pivotal-rabbitmq.YOUR_CF_DOMAIN pivotal-rabbitmq-finance.YOUR_CF_DOMAIN
TiledisplaynameinOpsManager RabbitMQ RabbitMQ(finance)
TilenameusedinternallybyOpsManager p-rabbitmq p-rabbitmq-finance
Metrics/LoggingOrigin p-rabbitmq p-rabbitmq-finance
InstallingReplicaTilesAfteryouhavegeneratedareplicatile,youcanuploadittoOpsManagerasyouwouldanyothertile.Afteryouhaveuploadedit,followtheinstructionsforInstallingandConfiguringRabbitMQforPCFasaPre-ProvisionedService.TheOn-Demandserviceisnotofferedonreplicatiles.
LimitingAccesstoReplicaTilestoSpecificOrgsWhenyoureplicateRabbitMQforPCF,thereplicatilehastheBrokerRegistrarerrandsettoOnbydefault.ThisfieldappearsintheErrandstabinthetile:
Withanytile,iftheBrokerRegistrarerrandissettoOn,itrunsautomaticallywhenyoufinishinstallingthetileandcausesthetiletobeavailabletoallCForgs.
Ifyouwanttolimitaccesstothetiletoaspecificorg,followthesesteps:
©CopyrightPivotalSoftwareInc,2013-2019 72 1.12
1. SetthebrokerregistrarerrandtoOff,andapplyyourchanges.
2. ManuallyregisterthetilewithaspecificCForgusingthefollowingcommand.Seetheabovetablefor BROKER_NAME , BROKER_URL ,andSERVICE_NAME :
cfcreate-service-brokerBROKER_NAMEBROKER_USERNAMEBROKER_PASSWORDBROKER_URL
3. Togiveaccesstotheorg,usethefollowingcommandandrepeatforeachadditionalorg:
cfenable-service-accessSERVICE_NAME-oORG_NAME
UpgradingReplicaTilesYoucanupgradereplicatileslikeregulartileswithoneimportantdifference.YoumustgenerateareplicaofthenewerversionoftheRabbitMQforPCFtile,usingthereplicator,andgivethenewreplicathesame name astheexistingreplica.Thisisshownintheexampleworkflowbelow.
ExampleofanIn-PlaceUpgradeofaReplicaSupposeyouusedthereplicatortogenerateareplicaofv1oftheRabbitMQforPCFtile,withthe name trading-team,andyouinstalleditinOpsManager.Hereisthesamplereplicatorcommandyouusedfortheinitialinstallation:
./rabbitmq-replicator-darwin\--nametrading-team\--path/download/p-rabbitmq-v1.pivotal\--output/output/p-rabbitmq-v1-trading-team.pivotal
Toupgradetov2,followthesesteps:
1. DownloadthenewRabbitMQforPCFv2.
2. Runthereplicatorcommand,usingthepathtothenewRabbitMQforPCFv2tile,andsupplythesame name ,trading-team,asshownbelow.
./rabbitmq-replicator-darwin\--nametrading-team\--path/download/p-rabbitmq-v2.pivotal\--output/output/p-rabbitmq-v2-trading-team.pivotal
3. Afteryouhavethereplicatilep-rabbitmq-v2-trading-team.pivotal,uploadittoOpsManager.Thisupgradesthev1replicatileinplace.
Youcanthenproceedwithupgradingthecluster.
Ifyouwanttodoablue-greenstyleupgrade,seeBlue-GreenUpgrades.
LimitationsTheOn-Demandserviceisnotofferedonreplicatiles.
©CopyrightPivotalSoftwareInc,2013-2019 73 1.12
SettingDefaultPoliciesfortheRabbitMQService
UnderstandingaRabbitMQPolicyYoucansetadefaultqueueandanexchangepolicyintheRabbitMQforPivotalCloudFoundry(PCF)tiletobeappliedtotheRabbitMQcluster.Afteryoudeploythetile,PivotalrecommendsthatyouusetheRabbitMQManagementInterfacetomakeconfigurationchanges.
FormoreinformationaboutRabbitMQpolicies,seetheRabbitMQdocumentation .
RulesforPoliciesSetintheTileThefollowingrulesapplytopolicessetthroughtheRabbitMQforPCFtile:
Anewpolicy,oranupdatetoapolicy,onlyappliestonewinstances(vhosts).Existinginstancesarenotaffectedbythepolicy.
ThepolicycanonlybedeletedmanuallyfromtheRabbitMQnodes.
PoliciescanbeaddeddynamicallyusingtheRabbitMQManagementInterface.
Itisnotpossibletousepatternmatchingwithpolicies.Policieswillbeappliedtoallqueuesandexchanges.Forgranularpolicysettings,PivotalrecommendsusingtheRabbitMQManagementUI.Seta prioritynumber lowerthan 50 ,thedefault prioritynumber
appliedthroughtheOpsManagerconfiguration.
AnExamplePolicy:MirroronTwoNodesHereisanexamplepolicythatensuresmessagesaremirroredontwonodes:
{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}
Operatorsshouldconsidersomeoftheperformanceimplicationsofmakingqueuesandexchangeshighlyavailable.Formoreinformationabouthighlyavailablequeues,seetheRabbitMQdocumentation .
BestPracticeforSyncingQueuesWhenaqueuesyncsallitsmessages,theyareloadedintomemory.Whenqueuesaresyncing,theycanuseasmuchmemoryasthetotalsizeofallmessages.Thisappliestobothnodes—thenodewherethequeueleaderruns(fromnode)andthenodewherethequeuefollowerruns(tonode),butonlyappliestonewlycreatedqueuefollowers.
Thisbehaviorisespeciallyrelevantwhenanychangeaffectsthedeployment,forexample:stemcellupdates,deploymentconfigurationchanges,andnetworkchanges.Verifythatyouhaveenoughmemoryanddiskavailabletosupportallmessages.
Forexample:
Thereare5GBofmessagesinamirroredqueuethatissettoautomaticsync.Whenthisqueueneedstosync,thenodewherethequeueleaderrunscanuseupto5GBofextramemory.Thesameappliestothenodewherethenewqueuefolloweriscreated.
SettingorChangingthePolicyTosettheRabbitMQpolicy,dothefollowing:
1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQforPCFtileandthenclickPre-ProvisionedRabbitMQPolicy.
©CopyrightPivotalSoftwareInc,2013-2019 74 1.12
2. SelectEnablecustompolicyonnewinstances.
3. InthePolicyfornewinstancesfield,pastethepolicy.ThepolicymustbevalidJSONandshouldmeetvalidRabbitMQpolicycriteria.
4. IntheSelectthenetworkpartitionbehavioroftheRabbitMQcluster,choosethedesiredbehavior:pause_minorityorautoheal.
FormoreinformationabouttheseoptionsandonRabbitMQclustersandnetworkpartitions,seetheRabbitMQdocumentation .
Forproductionpurposes,PivotalrecommendsthatcustomershaveatleastthreeRabbitMQservernodesandtwoHAProxiesspreadacrosslowlatencyavailabilityzones.
ViewingPoliciesintheRabbitMQManagementDashboardYoucanviewRabbitMQpoliciesintheRabbitMQManagementDashboard,shownbelow.TheexamplepolicyenteredintheRabbitMQforPCFtileaboveisappliedtoallqueuesandgivenaPriorityof50.Thisallowsyoutooverrideitbydefininganotherpolicywithahigherpriority.
IntheQueuessectionshownbelow,youcanseethatanynewqueuescreatedhavethepolicyautomaticallyapplied.
Note:Nopolicyvalidationoccursduringthedeployment,anderrorscancausethedeploymenttofailorpoliciestobeappliedincorrectly.
©CopyrightPivotalSoftwareInc,2013-2019 75 1.12
Note:DeveloperscanobtaintheURLofthepolicyfrom VCAP_SERVICES forappdevelopers.
©CopyrightPivotalSoftwareInc,2013-2019 76 1.12
UsingtheRabbitMQManagementDashboard
RabbitMQManagementDashboard
AdminUserTogainaccesstothemanagementdashboardasthe admin user,visit http://pivotal-rabbitmq.SYS-DOMAIN .Toretrieveyoursystemdomain,navigatetoyourPivotalApplicationService(PAS)orElasticRuntimetileandlocatetheSystemDomainfieldoftheDomainssection.
TheusernameandpasswordistheusernameandpasswordyouprovidedintheRabbitMQconfigurationinOpsManager,whichisalsoshownintheCredentialstab.
ApplicationDeveloperUsersofCloudFoundrywhocreateinstancesviatheAppsManagerorthecfCLIalsogetaccesstotheManagementUI.Thisisdoneusingcredentialsthatprovideaccessonlytotheirparticularvhost.
TheappropriateURLisaccessibleviatheManagebuttonwithintheAppsManager.
©CopyrightPivotalSoftwareInc,2013-2019 77 1.12
Oritisalsoinjectedintothe VCAP_SERVICES environmentvariableprovidedtoappsrunningonCloudFoundry.ThiscanalsobefoundviatheCLIusingcfenv<yourappname>
.
LoggingATCPSyslogendpointcanbeconfiguredinOpsManager.LogsarecurrentlyonlyforwardedfortheRabbitMQcluster.
RabbitMQCLIIfyouwishtoruncommandssuchas rabbitmqctl thenyouhavetwooptions:
SSHintooneofthemachinesrunningtherabbitmq-server.IPscanbefoundfromtheStatustabandaccesscredentialsfromtheCredentialstabwithintheRabbitMQcomponentoftheinstaller.FromthereyouneedtobringRabbitMQandErlangintoyourenvironmentandfromthereyoucanuserabbitmqctl :
bash-4.1#exportPATH=$PATH:/var/vcap/packages/rabbitmq-server/binbash-4.1#exportPATH=$PATH:/var/vcap/packages/erlang/binbash-4.1#rabbitmqctlcluster_statusClusterstatusofnoderabbit@node0...[{nodes,[{disc,[rabbit@node0,rabbit@node1,rabbit@node2,rabbit@node3]}]},{running_nodes,[rabbit@node3,rabbit@node2,rabbit@node1,rabbit@node0]},{partitions,[]}]...done.
Alternatively,installRabbitMQandErlangonamachineofyourchoice.Besuretomatchversionsofbothtothecluster:theManagementUIshowsboththeversionofRabbitMQandErlang.
Thensetyour ~/.erlang.cookie tomatchthecookieusedinthecluster(youmayhavesuppliedthisaspartoftheinstallation;seeabove).
Youwillneedtosetupyour /etc/hosts filetomatchtheRabbitMQnodes.
©CopyrightPivotalSoftwareInc,2013-2019 78 1.12
ClusteringandNetworkPartitions
ClusteringinRabbitMQforPCFInRabbitMQforPCF,theRabbitMQ®brokerisalwaysdeployedasaclusterofoneormorevirtualmachines(nodes).ARabbitMQbrokerisalogicalgroupingofoneorseveralErlangnodes,eachrunningtheRabbitMQapplicationandsharingusers,virtualhosts,queues,exchanges,bindings,andruntimeparameters.
WhatisReplicatedbetweennodesinaRabbitMQcluster?
Alldata/staterequiredfortheoperationofaRabbitMQbrokerisreplicatedacrossallnodes.Anexceptiontothisaremessagequeues,whichbydefaultresideononenode,thoughtheyarevisibleandreachablefromallnodes.ThismeansthattheRabbitMQclustermaybeavailableandservingrequests,whileanindividualqueueresidingonasinglenodeisoffline.
Replicatingmessagequeuesacrossnodesisanexpensiveoperationandshouldonlybedonetotheextentneededbytheapplication.Tounderstandmoreaboutreplicatingqueuesacrossnodesinacluster,seethedocumentation onhighavailability.
AutomaticNetworkPartitionBehaviorsinRabbitMQClustersTheRabbitMQ®tileusesthe pause_minority optionforhandlingclusterpartitionsbydefault.Thisensuresdataintegritybypausingthepartitionoftheclusterintheminority,andresumesitwiththedatafromthemajoritypartition.Youmustmaintainmorethantwonodes.Ifthereisapartitionwhenyouonlyhavetwonodes,bothnodesimmediatelypause.
Youcanalsochoosethe autoheal optioninthePre-ProvisionedRabbitMQPolicytab.Inthismode,ifapartitionoccurs,RabbitMQautomaticallydecidesonawinningpartition,andrestartsallnodesthatarenotinthewinningpartition.Thisoptionallowsyoutocontinuetoreceiveconnectionstobothpartsofpartitions.
DetectingaNetworkPartitionWhenanetworkpartitionoccurs,alogmessageiswrittentotheRabbitMQnodelog:
=ERRORREPORT====15-Oct-2012::18:02:30===Mnesia(rabbit@da3be74c053640fe92c6a39e2d7a5e46):**ERROR**mnesia_eventgot{inconsistent_database,running_partitioned_network,rabbit@21b6557b73f343201277dbf290ae8b79}
Youcanalsorunthe rabbitmqctlcluster_status commandonanyoftheRabbitMQnodestoseethenetworkpartition.Torun rabbitmqctlcluster_status ,dothefollowing:
1. $ sudo su -
2. $ cd /var/vcap/packages
3. $ export ERL_DIR=$PWD/erlang/bin/
4. $ cd rabbitmq-server/bin/
5. $./rabbitmqctlcluster_status
[...{partitions,[{rabbit@da3be74c053640fe92c6a39e2d7a5e46,[rabbit@21b6557b73f343201277dbf290ae8b79]}]}]
RecoveringBecausetheRabbitMQtileusesthe pause_minority option,minoritynodesrecoverautomaticallyafterthepartitionisresolved.Afteranoderecovers,itresumesaccessingthequeuealongwithdatafromthequeuesontheothernodes.However,ifyourqueuesuse ha-mode:all ,theyonlysynchronizefully
©CopyrightPivotalSoftwareInc,2013-2019 79 1.12
afterconsumingallthemessagescreatedwhilethenodewasdown.Thisissimilartohowmessagessynchronizewhenyoucreateanewqueue.
ManuallySynchronizingafteraPartitionAfteranetworkpartition,aqueueonaminoritynodesynchronizesafterconsumingallthemessagescreatedwhileitwasdown.Youcanalsorunthesync_queue commandtosynchronizeaqueuemanually.Torun sync_queue ,dothefollowingoneachnode:
1. $ sudo su -
2. $ cd /var/vcap/packages
3. $ export ERL_DIR=$PWD/erlang/bin/
4. $ cd rabbitmq-server/bin/
5. $ ./rabbitmqctl list_queues
6. $ ./rabbitmqctl sync_queue name
©CopyrightPivotalSoftwareInc,2013-2019 80 1.12
UpgradingRabbitMQforPCFThisproductenablesautomatedupgradesbetweenversionsoftheproductandisdeployedthroughOpsManager.Insomecases,youmightberequiredtotaketheclusteroffline.Whenthisisnecessary,itisclearlynotedinthereleasenotesforthatversion.
TheupgradepathsforeachversionaredetailedatPivotalNetwork-RabbitMQforPCFpage .
Toupgradeon-demandinstancesfromRabbitMQv3.6tov3.7,seeAboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7.
DowntimeWhenUpgradingAguidefordowntimeduringupgradedeploymentsisshowninthetablebelow.Insomecases,theclusterremainsavailableduringatileupgrade,butindividualqueuesonclusternodesmaybetakenoffline.
Thisisonlyaguide,sobeforeupgrading,checkthereleasenotesfortheversionyouareupgradingto.
TheRabbitMQclusterbecomesunavailableonlywhenupgradingbetweenspecificversionsofErlangorRabbitMQ.Thisisstatedinthereleasenotesforthoseversions.
UpgradeType WillDowntimeBeRequiredForThisUpgrade/Update
MajorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.
MinorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.
PatchTileVersion
Normallythesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.Therearespecificmigrationpathsthatrequiredowntime,whichareidentifiedinthereleasenotesforthatversion.
Stemcell-OnlyPatchTileVersion
Wherethepatchupdateisonlyanewstemcellversionthesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.
NotesontheUpgradeProcessReviewthefollowingbeforestartinganupgradeofRabbitMQforPCF:
Upgradingtoanewerversionoftheproductdoesnotcauseanylossofdataorconfiguration.
ItmaytakebusyRabbitMQnodesalongtimetoshutdownduringtheupgradeandyoumustnotinterruptthisprocess.
ThebenefityougetfromstemcellrollingupgradesdependsonhowyouhaveconfigurednetworkpartitionhandlingandtheResourceConfigtab.AnHAProxyinstancecountof2andaRabbitMQnodecountof3arerequiredforrollingstemcellupgrades.Asofv1.7.7,thesecountsarethedefault.Formoreinformation,seeClusteringandNetworkPartitions.
ThelengthofthedowntimedependsonwhetherthereisastemcellupdatetoreplacetheoperatingsystemimageoriftheexistingVMcanjusthavetheRabbitMQsoftwareupdated.StemcellupdatesincuradditionaldowntimewhiletheIaaScreatesthenewVM.
OpsManagerensurestheinstancesareupdatedwiththenewpackagesandanyconfigurationchangesareappliedautomatically.
ForissueswithupgradingRabbitMQforPCF,seeTroubleshootingOn-DemandRabbitMQforPCF.
BeforeUpgradingRabbitMQorErlangEnsuretheclusterishealthyusingtheRabbitMQManagementUI.YoucannotrelyontheBOSH instances outputbecausethatreflectsthestateofErlangVM,notRabbitMQ.
UpgradeRabbitMQforPCFToupgradetheproduct,followthesesteps:
1. DownloadthelatestversionoftheproductfromPivotalNetwork .
©CopyrightPivotalSoftwareInc,2013-2019 81 1.12
2. Uploadthenew.pivotalfiletoOpsManager.
3. Uploadthestemcellassociatedwiththeupdate(ifrequired).
4. Updateanynewmandatoryconfigurationparameters(ifrequired).
5. ClickApplychangesintheOpsManagerInstallationDashboard.Therestoftheprocessisautomated.
AboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7BeforeRabbitMQforPCFv1.12,on-demandserviceinstancesweredeployedusingRabbitMQv3.6.RabbitMQforPCFv1.12provideson-demandserviceplanswithRabbitMQv3.6andv3.7.
Appdeveloperscaneitherstartfreshwithav3.7on-demandinstanceortheycanmigratetheirRabbitMQv3.6instancestoRabbitMQv3.7usingblue-greenappdeploymentswithoutdowntime.
Toenablethemigrationfromon-demandv3.6instancestov3.7instances,theon-demandv3.7plansshouldbeconfiguredtomirrortheexistingv3.6plans.Formoreinformation,seetheblogBlue-GreenApplicationDeploymentswithRabbitMQ andthevideoBlue-GreenDeploymentofApplicationsleveragingRabbitMQ .
RabbitMQv3.6on-demandplanswillberemovedinanupcomingRabbitMQforPCFtilerelease.Encouragealldeveloperstomovetheirappsawayfromtheseplans.
ReleasePolicyWhenanewversionofRabbitMQisreleased,anewversionofRabbitMQforPCFisreleasedsoonafter.
FormoreinformationaboutthePCFreleasepolicy,seeReleasePolicy .
©CopyrightPivotalSoftwareInc,2013-2019 82 1.12
TroubleshootingandFAQsforOn-DemandRabbitMQforPCFThistopicprovidesoperatorswithbasictroubleshootingtechniquesandFAQsforon-demandRabbitMQforPivotalCloudFoundry(PCF).
HowtoRetrieveaServiceinstanceGUIDYouneedtheGUIDofyourserviceinstancetorunsomeBOSHcommands.ToretrievetheGUID,runthecommand:
cfserviceSERVICE-INSTANCE-NAME--guid
Ifyoudonotknowthenameoftheserviceinstance,run cfservices toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthenamecolumn.
TroubleshootingErrorsStarthereifyou’rerespondingtoaspecificerrororerrormessages.
FailedInstall1. Certificateissues:Theon-demandbroker(ODB)requiresvalidcertificates.Ensurethatyourcertificatesarevalidandgeneratenewones if
necessary.
2. Deployfails:Deployscanfailforavarietyofreasons.ViewthelogsusingOpsManagertodeterminewhythedeployisfailing.
3. Networkingproblems:
CloudFoundrycannotreachtheRabbitMQforPCFservicebrokerCloudFoundrycannotreachtheserviceinstancesTheservicenetworkcannotaccesstheBOSHdirector
4. Registerbrokererrandfails.
5. Thesmoketesterrandfails.
6. Resourcesizingissues:TheseoccurwhentheresourcesizesselectedforagivenplanarelessthantheRabbitMQforPCFservicerequirestofunction.CheckyourresourceconfigurationinOpsManagerandensurethattheconfigurationmatchesthatrecommendedbytheservice.
7. Otherservice-specificissues.
CannotCreateorDeleteServiceInstancesIfdevelopersreporterrorssuchas:
Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:create
Followthesesteps:
1. IftheBOSHerrorshowsaproblemwiththedeploymentmanifest,openthemanifestinatexteditortoinspectit.
2. Tocontinuetroubleshooting,LogintoBOSH andtargettheRabbitMQforPCFserviceinstanceusingtheinstructionsonparsingaCloudFoundryerrormessage.
©CopyrightPivotalSoftwareInc,2013-2019 83 1.12
3. RetrievetheBOSHtaskIDfromtheerrormessageandrunthefollowingcommand:
boshtaskTASK-ID
4. Ifyouneedmoreinformation,accessthebrokerlogsandusethe broker-request-id fromtheerrormessageabovetosearchthelogsformoreinformation.Checkfor:
AuthenticationerrorsNetworkerrorsQuotaerrors
BrokerRequestTimeoutsIfdevelopersreporterrorssuchas:
Servererror,statuscode:504,errorcode:10001,message:Therequesttotheservicebrokertimedout:https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b
Followthesesteps:
1. ConfirmthatCloudFoundry(CF)isconnectedtotheservicebroker.
2. ChecktheBOSHqueuesize:
a. LogintoBOSHasanadmin.b. Run bosh tasks .
3. Iftherearealargenumberofqueuedtasks,thesystemmaybeundertoomuchload.BOSHisconfiguredwithtwoworkersandonestatusworker,whichmaynotbesufficientresourcesforthelevelofload.Adviseappdeveloperstotryagainoncethesystemisunderlessload.
CannotBindtoorUnbindfromServiceInstances
InstanceDoesNotExist
Ifdevelopersreporterrorssuchas:
Servererror,statuscode:502,errorcode:10001,message:Servicebrokererror:instancedoesnotexist`
Followthesesteps:
1. ConfirmthattheRabbitMQforPCFserviceinstanceexistsinBOSHandobtaintheGUIDCFbyrunning:
cfserviceMY-INSTANCE--guid
2. UsingtheGUIDobtainedabove,thefollowingBOSHCLIcommand:
bosh-dservice-instance_GUIDvms
IftheBOSHdeploymentisnotfound,ithasbeendeletedfromBOSH.ContactPivotalsupportforfurtherassistance.
OtherErrors
©CopyrightPivotalSoftwareInc,2013-2019 84 1.12
Ifdevelopersreporterrorssuchas:
Servererror,statuscode:502,errorcode:10001,message:Servicebrokererror:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:example-service,service-instance-guid:8d69de6c-88c6-4283-b8bc-1c46103714e2,broker-request-id:15f4f87e-200a-4b1a-b76c-1c4b6597c2e1,operation:bind
Tofindouttheexactissuewiththebindingprocess:
1. Accesstheservicebrokerlogs.
2. Searchthelogsforthe broker-request-id stringlistedintheerrormessageabove.
3. ContactPivotalsupportforfurtherassistanceifyouareunabletoresolvetheproblem.
4. Checkfor:
AuthenticationerrorsNetworkerrors
CannotConnecttoaServiceInstanceIfdevelopersreportthattheirappcannotuseserviceinstancesthattheyhavesuccessfullycreatedandbound:
Asktheusertosendapplicationlogsthatshowtheconnectionerror.Iftheerrorisoriginatingfromtheservice,thenfollowRabbitMQforPCF-specificinstructions.Iftheissueappearstobenetwork-related,then:
1. Checkthatapplicationsecuritygroups areconfiguredcorrectly.Accessshouldbeconfiguredfortheservicenetworkthatthetileisdeployedto.
2. EnsurethatthenetworkthePivotalApplicationService(PAS)tileisdeployedtohasnetworkaccesstotheservicenetwork.YoucanfindthenetworkdefinitionforthisservicenetworkintheBOSHDirectortile.
3. InOpsManagergointotheservicetileandseetheservicenetworkthatisconfiguredinthenetworkstab.
4. InOpsManagergointothePAStileandseethenetworkitisassignedto.Makesurethatthesenetworkscanaccesseachother.
UpgradeAllServiceInstancesFailsIfthe upgrade-all-service-instances errandfails,lookattheerrandoutputintheOpsManagerlog.
Ifaninstancefailstoupgrade,debugandfixitbeforerunningtheerrandagaintopreventanyfailureissuesfromspreadingtootheron-demandinstances.
OncetheOpsManagerlognolongerliststhedeploymentas failing ,re-runtheerrandtoupgradetherestoftheinstances.
MissingLogsandMetricsIfnologsarebeingemittedbytheon-demandbroker,checkthatyoursyslogforwardingaddressiscorrectinOpsManager.
1. Ensureyouhaveconfiguredsyslogforthetile.
2. Ensurethatyouhavenetworkconnectivitybetweenthenetworksthatthetileisusingandthesyslogdestination.Ifthedestinationisexternal,youneedtousethepublicip VMextensionfeatureavailableinyourOpsManagertileconfigurationsettings.
3. VerifythattheFirehoseisemittingmetrics:
a. Installthe cf nozzle plugin.Forinstructions,seethefirehoseplugin GitHubrepository.b. Tofindlogsfromyourserviceinthe cfnozzle output,runthefollowing:
cfnozzle-fValueMetric|grep--line-buffered"on-demand-broker/MY-SERVICE"
©CopyrightPivotalSoftwareInc,2013-2019 85 1.12
Ifnometricsappearwithinfiveminutes,verifythatthebrokernetworkhasaccesstotheLoggregatorsystemonallrequiredports.
ContactPivotalsupportifyouareunabletoresolvetheissue.
FailedDeploymentonUpgradeorafterApplyChangesIfthedeploymentfailsaftereditingtheAssignAZsandNetworkspaneoftheRabbitMQforPCFtile,itmightbeduetoachangetotheIPaddressesassignedtothe RabbitMQ Server job.RabbitMQforPCFrequiresthattheseIPaddressesdonotchangeonceassigned.Ifyouchangethem,thedeploymentfails.Thisincludeschangesmadetoyourcurrentinstallationorduringanupgrade.Todiagnoseandsolvethisissue,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
TroubleshootingComponentsGuidanceoncheckingforandfixingissuesinon-demandservicecomponents.
BOSHproblems
LargeBOSHQueue
On-demandservicebrokersaddtaskstotheBOSHrequestqueue,whichcanbackupandcausedelayunderheavyloads.AnappdeveloperwhorequestsanewRabbitMQforPCFinstancesees createinprogress intheCloudFoundryCommandLineInterface(cfCLI)untilBOSHprocessesthequeuedrequest.
OpsManagercurrentlydeploystwoBOSHworkerstoprocessitsqueue.FutureversionsofOpsManagerwillletusersconfigurethenumberofBOSHworkers.
Configuration
Serviceinstancesinfailingstate
YoumayhaveconfiguredaVM/DisktypeintileplanpageinOpsManagerthatisinsufficientlylargefortheRabbitMQforPCFserviceinstancetostart.Seetile-specificguidanceonresourcerequirements.
Authentication
UAAChanges
IfyouhaverotatedanyUAAusercredentialsthenyoumayseeauthenticationissuesintheservicebrokerlogs.
Toresolvethis,redeploytheRabbitMQforPCFtileinOpsManager.Thisprovidesthebrokerwiththelatestconfiguration.
Note:YoumustensurethatanychangestoUAAcredentialsarereflectedintheOpsManager credentials tabofthePivotalApplicationService(PAS)tile.
©CopyrightPivotalSoftwareInc,2013-2019 86 1.12
NetworkingCommonissueswithnetworkinginclude:
Issue Solution
LatencywhenconnectingtotheRabbitMQforPCFserviceinstancetocreateordeleteabinding.
Tryagainorimprovenetworkperformance.
FirewallrulesareblockingconnectionsfromtheRabbitMQforPCFservicebrokertotheserviceinstance.
OpentheRabbitMQforPCFtileinOpsManagerandcheckthetwonetworksconfiguredintheNetworkspane.Ensurethatthesenetworksallowaccesstoeachother.
FirewallrulesareblockingconnectionsfromtheservicenetworktotheBOSHdirectornetwork.
EnsurethatserviceinstancescanaccesstheDirectorsothattheBOSHagentscanreportin.
Appscannotaccesstheservicenetwork.ConfigureCloudFoundryapplicationsecuritygroupstoallowruntimeaccesstotheservicenetwork.
ProblemsaccessingBOSH’sUAAortheBOSHdirector.FollownetworktroubleshootingandcheckthattheBOSHdirectorisonline
ValidateServiceBrokerConnectivitytoServiceInstances
Tovalidateconnectivity,dothefollowing:
1. ToSSHintotheRabbitMQforPCFservicebroker,runthefollowingcommnand:
bosh-dservice-instance_GUIDssh
2. IfnoBOSH task-id appearsintheerrormessage,lookinthebrokerlogusingthe broker-request-id fromthetask.
ValidateAppAccesstoServiceInstance
Use cfssh toaccesstotheappcontainer,thentryconnectingtotheRabbitMQforPCFserviceinstanceusingthebindingincludedintheVCAP_SERVICES environmentvariable.
Quotas
PlanQuotaissues
Ifdevelopersreporterrorssuchas:
Message:Servicebrokererror:Thequotaforthisserviceplanhasbeenexceeded.PleasecontactyourOperatorforhelp.
1. Checkyourcurrentplanquota.
2. Increasetheplanquota.
3. LogintoOpsManager.
©CopyrightPivotalSoftwareInc,2013-2019 87 1.12
4. Reconfigurethequotaontheplanpage.
5. Deploythetile.
6. Findwhoisusingtheplanquotaandtaketheappropriateaction.
GlobalQuotaIssues
Ifdevelopersreporterrorssuchas:
Message:Servicebrokererror:Thequotaforthisservicehasbeenexceeded.PleasecontactyourOperatorforhelp.
1. Checkyourcurrentglobalquota.
2. Increasetheglobalquota.
3. LogintoOpsManager.
4. Reconfigurethequotaontheon-demandsettingspage.
5. Deploythetile.
6. Findoutwhoisusingthequotaandtaketheappropriateaction.
FailingjobsandunhealthyinstancesTodeterminewhetherthereisanissuewiththeRabbitMQforPCFservicedeployment,inspecttheVMs.Todoso,runthefollowingcommand:
bosh-dservice-instance_GUIDvms--vitals
Foradditionalinformation,runthefollowingcommand:
boshinstances--ps--vitals
IftheVMisfailing,followtheservice-specificinformation.Anyunadvisedcorrectiveactions(suchasrunningBOSH restart onaVM)cancauseissuesintheserviceinstance.
TechniquesforTroubleshootingThissectioncontainsinstructionsoninteractingwiththeon-demandservicebrokerandon-demandserviceinstanceBOSHdeployments,andonperforminggeneralmaintenanceandhousekeepingtasks.
ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cfserviceINSTANCE-NAME .
©CopyrightPivotalSoftwareInc,2013-2019 88 1.12
$cfservicemyservice
Serviceinstance:myserviceService:super-dbBoundapps:Tags:Plan:dedicated-vmDescription:DedicatedInstanceDocumentationurl:Dashboard:
LastOperationStatus:createfailedMessage:Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:createStarted:2017-03-13T10:16:55ZUpdated:2017-03-13T10:17:58Z
Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.
The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe boshtaskTASK-ID .
The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping boshlogsbroker0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.
AccessBrokerandInstanceLogsandVMsBeforefollowingtheproceduresbelow,logintothecfCLI andtheBOSHCLI .
AccessBrokerLogsandVM(s)
YoucanaccesslogsusingOpsManager byclickingontheLogstabinthetileanddownloadingthebrokerlogs.
ToaccesslogsusingtheBOSHCLI,dothefollowing:
1. Identifytheon-demandbroker(ODB)deploymentbyrunningthefollowingcommand:
boshdeployments
2. ViewVMsinthedeploymentbyrunningthefollowingcommand:
bosh-dDEPLOYMENT-NAMEinstances
3. SSHontotheVMbyrunningthefollowingcommand:
bosh-dservice-instance_GUIDssh
4. Downloadthebrokerlogsbyrunningthefollowingcommand:
bosh-dservice-instance_GUIDlogs
ThearchivegeneratedbyBOSHorOpsManagerincludesthefollowinglogs:
LogName Description
broker.logRequeststotheon-demandbrokerandtheactionsthebrokerperformswhileorchestratingtherequest(e.g.generatingamanifest
©CopyrightPivotalSoftwareInc,2013-2019 89 1.12
andcallingBOSH).Startherewhentroubleshooting.broker_ctl.log Controlscriptlogsforstartingandstoppingtheon-demandbroker.
post-start.stderr.log
Errorsthatoccurduringpost-startverification.
post-start.stdout.log
Post-startverification.
drain.stderr.log Errorsthatoccurwhilerunningthedrainscript.
AccessServiceInstanceLogsandVMs1. Totargetanindividualserviceinstancedeployment,retrievetheGUIDofyourserviceinstancewiththefollowingcfCLIcommand:
cfserviceMY-SERVICE--guid
2. ToviewVMsinthedeployment,runthefollowingcommand:
bosh-dDEPLOYMENT-NAMEinstances
3. ToSSHintoaVM,runthefollowingcommand:
bosh-dservice-instance_GUIDssh
4. Todownloadtheinstancelogs,runthefollowingcommand:
bosh-dservice-instance_GUIDlogs
RunServiceBrokerErrandstoManageBrokersandInstancesFromtheBOSHCLI,youcanrunservicebrokererrandsthatmanagetheservicebrokersandperformmassoperationsontheserviceinstancesthatthebrokerscreated.Theseservicebrokererrandsinclude:
register-broker registersabrokerwiththeCloudControllerandlistsitintheMarketplace.
deregister-broker deregistersabrokerwiththeCloudControllerandremovesitfromtheMarketplace.
upgrade-all-service-instances upgradesexistinginstancesofaservicetoitslatestinstalledversion.
delete-all-service-instances deletesallinstancesofservice.
orphan-deployments detects“orphan”instancesthatarerunningonBOSHbutnotregisteredwiththeCloudController.
Torunanerrand,runthefollowingcommand:
bosh-dDEPLOYMENT-NAMErun-errandERRAND-NAME
Forexample:
bosh-dmy-deploymentrun-errandderegister-broker
RegisterBroker
The register-broker errandregistersthebrokerwithCloudFoundryandenablesaccesstoplansintheservicecatalog.Runthiserrandwheneverthebrokerisre-deployedwithnewcatalogmetadatatoupdatetheCloudFoundrycatalog.
Planswithdisabledserviceaccessarenotvisibletonon-adminCloudFoundryusers,includingOrgManagersandSpaceManagers.AdminCloudFoundry
©CopyrightPivotalSoftwareInc,2013-2019 90 1.12
userscanseeallplansincludingthosewithdisabledserviceaccess.
Theerranddoesthefollowing:
RegisterstheservicebrokerwithCloudController.
Enablesserviceaccessforanyplansthathavetheradiobuttonsetto enabled inthetileplanpage.
Disablesserviceaccessforanyplansthathavetheradiobuttonsetto disabled inthetileplanpage.
Doesnothingforanyforanyplansthathavetheradiobuttonsetto manual .
Toruntheerrand,runthefollowingcommand:
bosh-dDEPLOYMENT-NAMErun-errandregister-broker
DeregisterBroker
ThiserrandderegistersabrokerfromCloudFoundry.
Theerranddoesthefollowing:
DeletestheservicebrokerfromCloudController
Failsifthereareanyserviceinstances,withorwithoutbindings
UsetheDeleteAllServiceInstanceserrandtodeleteanyexistingserviceinstances.
Toruntheerrand,runthefollowingcommand:
bosh-dDEPLOYMENT-NAMErun-errandderegister-broker
UpgradeAllServiceInstances
IfyouhavemadechangestotheplandefinitionoruploadedanewtileintoOpsManager,youmightwanttoupgradealltheRabbitMQforPCFserviceinstancestothelatestsoftwareorplandefinition.
The upgrade-all-service-instances erranddoesthefollowing:
Collectsalloftheserviceinstancestheon-demandbrokerhasregistered
Foreachinstancetheerranddoesthefollowingserially
Issuesanupgradecommandtotheon-demandbrokerRegeneratestheserviceinstancemanifestbasedonitslatestconfigurationfromthetileDeploysthenewmanifestfortheserviceinstanceWaitsforthisoperationtocomplete,thenproceedstothenextinstance
AddstoaretrylistanyinstancesthathaveongoingBOSHtasksatthetimeofupgrade
Retriesanyinstancesintheretrylistuntilallareupgraded
Ifanyinstancefailstoupgrade,theerrandfailsimmediately.Thispreventssystemicproblemsfromspreadingtotherestofyourserviceinstances.
Toruntheerrand,dooneofthefollowing:
SelecttheerrandthroughtheOpsManagerUIandhaveitrunwhenyouclickApplyChanges.
Runthefollowingcommand.
©CopyrightPivotalSoftwareInc,2013-2019 91 1.12
bosh-dDEPLOYMENT-NAMErun-errandupgrade-all-service-instances
DeleteAllServiceInstances
ThiserrandusestheCloudControllerAPItodeleteallinstancesofyourbroker’sserviceofferingineveryCloudFoundryorgandspace.ItonlydeletesinstancestheCloudControllerknowsabout.ItdoesnotdeleteorphanBOSHdeployments.
The delete-all-service-instances erranddoesthefollowing:
1. Unbindsallappsfromtheserviceinstances.
2. Deletesallserviceinstancessequentially.Eachserviceinstancedeletionincludes:
a. Runninganypre-deleteerrandsb. DeletingtheBOSHdeploymentoftheserviceinstancec. RemovinganyODB-managedsecretsfromCredhubd. Checkingforinstancedeletionfailure,whichresultsintheerrandfailingimmediately
3. Determineswhetheranyinstanceshavebeencreatedwhiletheerrandwasrunning.Ifnewinstancesaredetected,theerrandreturnsanerror.Inthiscase,Pivotalrecommendsrunningtheerrandagain.
Toruntheerrand,runthefollowingcommand:
bosh-dservice-instance_GUIDdelete-deployment
DetectOrphanedInstancesServiceInstances
Aserviceinstanceisdefinedas“orphaned”whentheBOSHdeploymentfortheinstanceisstillrunning,buttheserviceisnolongerregisteredinCloudFoundry.
The orphan-deployments errandcollatesalistofservicedeploymentsthathavenomatchingserviceinstancesinCloudFoundryandreturnthelisttotheoperator.ItisthenuptotheoperatortoremovetheorphanedBOSHdeployments.
Toruntheerrand,runthefollowingcommand:
bosh-dDEPLOYMENT-NAMErun-errandorphan-deployments
Iforphandeploymentsexist—Theerrandscriptdoesthefollowing:
Exitwithexitcode10
Outputalistofdeploymentnamesundera [stdout] header
Provideadetailederrormessageundera [stderr] header
Forexample:
Note:OrphanBOSHdeploymentsdonotcorrespondtoaknownserviceinstance.Whilerare,orphandeploymentscanoccur.Usetheorphan-deployments errandtoidentifythem.
WARNING:Useextremecautionwhenrunningthiserrand.Youshouldonlyuseitwhenyouwanttototallydestroyalloftheon-demandserviceinstancesinanenvironment.
©CopyrightPivotalSoftwareInc,2013-2019 92 1.12
[stdout][{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]
[stderr]OrphanBOSHdeploymentsdetectedwithnocorrespondingserviceinstanceinCloudFoundry.BeforedeletinganydeploymentitisrecommendedtoverifytheserviceinstancenolongerexistsinCloudFoundryandanydataissafetodelete.
Errand'orphan-deployments'completedwitherror(exitcode10)
ThesedetailswillalsobeavailablethroughtheBOSH /tasks/ APIendpointforuseinscripting:
$curl'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result'|jq.{"exit_code":10,"stdout":"[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n","stderr":"OrphanBOSHdeploymentsdetectedwithnocorrespondingserviceinstanceinCloudFoundry.BeforedeletinganydeploymentitisrecommendedtoverifytheserviceinstancenolongerexistsinCloudFoundryandanydataissafetodelete.\n","logs":{"blobstore_id":"d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"}}
Ifnoorphandeploymentsexist—Theerrandscriptdoesthefollowing:
Exitwithexitcode0
Stdoutwillbeanemptylistofdeployments
Stderrwillbe None
[stdout][]
[stderr]None
Errand'orphan-deployments'completedsuccessfully(exitcode0)
Iftheerrandencountersanerrorduringrunning—Theerrandscriptdoesthefollowing:
Exitwithexit1
Stdoutwillbeempty
Anyerrormessageswillbeunderstderr
Tocleanuporphanedinstances,runthefollowingcommandoneachinstance:
boshdelete-deploymentservice-instance_SERVICE-INSTANCE-GUID
GetAdminCredentialsforaServiceInstance1. IdentifytheservicedeploymentbyGUID.
2. LogintoBOSH .
3. Openthemanifestinatexteditor.
4. Lookinthemanifestforthecredentials.
WARNING:RunningthiscommandmayleaveIaaSresourcesinanunusablestate.
©CopyrightPivotalSoftwareInc,2013-2019 93 1.12
ReinstallaTileToreinstallatileinthesameenvironmentwhereitwaspreviouslyuninstalled:
1. Ensurethattheprevioustilewascorrectlyuninstalledasfollows:
a. Loginasanadminbyrunning:
cf login
b. ConfirmthattheMarketplacedoesnotlistRabbitMQforPCFbyrunning:
cf m
c. LogintoBOSHasanadminbyrunning:
bosh log-in
d. DisplayyourBOSHdeploymentstoconfirmthattheoutputdoesnotshowRabbitMQforPCFdeploymentbyrunning:
bosh deployments
e. Runthe“delete-all-service-instances”errandtodeleteeveryinstanceoftheservice.f. Runthe“deregister-broker”errandtodeletetheservicebroker.g. DeletetheservicebrokerBOSHdeploymentbyrunning:
bosh delete-deployment BROKER-DEPLOYMENT-NAME
h. Reinstallthetile.
ViewResourceSaturationandScalingToviewusagestatisticsforanyservice,dothefollowing:
1. Runthefollowingcommand:
bosh-dDEPLOYMENT-NAMEvms--vitals
2. Toviewprocess-levelinformation,run:
bosh-dDEPLOYMENT-NAMEinstances--ps
IdentifyServiceInstanceOwnerIfyouwanttoidentifywhichappsareusingaspecificserviceinstancefromtheBOSHdeploymentsname,youcanrunthefollowingsteps:
1. Takethedeploymentnameandstripthe service-instance_ leavingyouwiththeGUID.
2. LogintoCFasanadmin.
3. Obtainalistofallservicebindingsbyrunningthefollowing:
cfcurl/v2/service_instances/GUID/service_bindings
©CopyrightPivotalSoftwareInc,2013-2019 94 1.12
4. Theoutputfromtheabovecurlgivesyoualistof resources ,witheachitemreferencingaservicebinding,whichcontainsthe APP-URL .Tofindthename,org,andspacefortheapp,runthefollowing:
a. cf curl APP-URL andrecordtheappnameunder entity.name .b. cf curl SPACE-URL toobtainthespace,usingthe entity.space_url fromtheabovecurl.Recordthespacenameunder entity.name .c. cf curl ORGANIZATION-URL toobtaintheorg,usingthe entity.organization_url fromtheabovecurl.Recordtheorganizationname
under entity.name .
MonitorQuotaSaturationandServiceInstanceCountQuotasaturationandtotalnumberofserviceinstancesareavailablethroughODBmetricsemittedtoLoggregator.Themetricnamesareshownbelow:
MetricName Description
on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining globalquotaremainingforallinstancesacrossallplans
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/quota_remaining quotaremainingforaparticularplan
on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances totalinstancescreatedacrossallplans
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/total_instances totalinstancescreatedforagivenplan
DropandRestoreAMQP(S)TraffictoaRabbitMQInstanceWhiledebuggingaRabbitMQinstance,youcanpreventappsfromsendingandreceivingmessages,forexample,todecreasetheserverload.Youcanuse drop-amqp-traffic and restore-amqp-traffic scripts,whichrunthenecessary iptables commandstoachievethat.
TostopandthenrestoretraffictoaRabbitMQinstance,dothefollowing:1. TostopallAMQP(S)traffictoaRabbitMQinstance,runthefollowingcommand:
bosh -d service-instance_GUID ssh rabbitmq-server "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/drop-amqp-traffic"
2. Afterperformingthetroubleshootingsteps,restorethetraffic.Todothis,runthefollowingcommand:
bosh -d service-instance_GUID ssh rabbitmq-server "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/restore-amqp-traffic"
Alternatively,youcanrunthesescriptsonindividualnodes:1. bosh ssh toarabbitmq-serverinstance.
2. sudo -s togainrootprivileges.
3. Execute drop-amqp-traffic todropallAMQP(S)traffictothisinstance,or restore-amqp-traffic tostartacceptingtrafficagain.
FrequentlyAskedQuestions
Note:Whenrunning cfcurl ensurethatyouqueryallpages,becausetheresponsesarelimitedtoacertainnumberofbindingsperpage.Thedefaultis50.Tofindthenextpagecurlthevalueunder next_url .
Note:Quotametricsarenotemittedifnoquotahasbeenset.
©CopyrightPivotalSoftwareInc,2013-2019 95 1.12
WhatshouldIcheckbeforedeployinganewversionofthetile?EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.YoucannotrelysolelyontheBOSH instances outputasthatreflectsthestateoftheErlangVMusedbyRabbitMQandnottheRabbitMQapp.
WhatisthecorrectwaytostopandstartRabbitMQinPCF?OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapp.
Forexample:
bosh stop rabbitmq-server and bosh start rabbitmq-server .
ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningthestopcommandandspecifying JOB [index] .
WhathappenswhenIrunboshstoprabbitmq-server?BOSHstartstheshutdownsequencefromthebootstrapinstance.
WestartbytellingtheRabbitMQapptoshutdownandthenshutdowntheErlangVMwithinwhichitisrunning.Ifthissucceeds,werunthefollowingcheckstoensurethattheRabbitMQappandErlangVMhavestopped:
1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.NoticethatwearetrackingtheErlangPIDandnottheRabbitMQPID.
2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID.
Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.Allremainingrabbitmq-serverinstanceswillbestoppedonebyone.
Whathappenswhenboshstoprabbitmq-serverfails?IftheBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:
result:1of1drainscriptsfailed.FailedJobs:rabbitmq-server.
WhatdoIdowhenboshstoprabbitmq-serverfails?
Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.
First,BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monitstartrabbitmq-server
.Youwillnotbeableto
startthejobviaBOSH start asthisalwaysrunsthedrainscriptfirstandwillfailsincethedrainscriptisfailing.
Oncerabbitmq-serverjobisrunning(confirmthisvia monitstatus ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.
HowcanImanuallybackupthestateoftheRabbitMQcluster?
Note:Donotuse monitstoprabbitmq-server
asthisdoesnotcallthedrainscripts
©CopyrightPivotalSoftwareInc,2013-2019 96 1.12
ItispossibletobackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.
BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.
2. Selectexportdefinitionsfromthemainpage.
BackupandRestorewithaScript
UsetheAPItorunscriptswithcodesimilartothefollowing:
Forthebackup:
curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"-o "$BACKUP-FOLDER/rabbit-backup.json"
Fortherestore:
curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"-X POST -H "Content-Type: application/json" -d"@$BACKUP-FOLDER/rabbit-backup.json"
Whatpre-upgradechecksshouldIdo?
BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:
1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.
2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.
3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.
KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .
FileaSupportTicketYoucanfileaticketwithPivotalSupport .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .
Toexpeditetroubleshooting,provideyourservicebrokerlogsandyourserviceinstancelogs.Ifyour cfserviceYOUR-SERVICE-INSTANCE outputincludesatask-id ,providetheBOSHtaskoutput.
©CopyrightPivotalSoftwareInc,2013-2019 97 1.12
FrequentlyAskedQuestionsforPre-ProvisionedRabbitMQforPCFThistopiclistsfrequentlyaskedquestionsthatapplytotheRabbitMQforPivotalCloudFoundry(PCF)pre-provisionedservice.
FrequentlyAskedQuestions
WhatshouldIcheckbeforedeployinganewversionofthetile?EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.YoucannotrelysolelyontheBOSH instances outputasthatreflectsthestateoftheErlangVMusedbyRabbitMQandnottheRabbitMQapplication.
WhatisthecorrectwaytostopandstartRabbitMQinPCF?OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapp.
Forexample:
boshstoprabbitmq-server
and boshstartrabbitmq-server
.
ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningthestopcommandandspecifying JOB[index]
WhathappenswhenIrunboshstoprabbitmq-server?BOSHstartstheshutdownsequencefromthebootstrapinstance.
WestartbytellingtheRabbitMQapplicationtoshutdownandthenshutdowntheErlangVMwithinwhichitisrunning.Ifthissucceeds,werunthefollowingcheckstoensurethattheRabbitMQapplicationandErlangVMhavestopped:
1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.NoticethatwearetrackingtheErlangPIDandnottheRabbitMQPID.
2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID
Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.Allremainingrabbitmq-serverinstanceswillbestoppedonebyone.
Whathappenswhenboshstoprabbitmq-serverfails?IftheBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:
result:1of1drainscriptsfailed.FailedJobs:rabbitmq-server.
WhatdoIdowhenboshstoprabbitmq-serverfails?Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.
Note:Donotuse monitstoprabbitmq-server
asthisdoesnotcallthedrainscripts
©CopyrightPivotalSoftwareInc,2013-2019 98 1.12
First,BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monitstartrabbitmq-server
.Youwillnotbeableto
startthejobviaBOSH start asthisalwaysrunsthedrainscriptfirstandwillfailsincethedrainscriptisfailing.
Oncerabbitmq-serverjobisrunning(confirmthisvia monitstatus ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.
HowcanImanuallybackupthestateoftheRabbitMQcluster?ItispossibletobackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.
BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.
2. Selectexportdefinitionsfromthemainpage.
BackupandRestorewithaScript
UsetheAPItorunscriptswithcodesimilartothefollowing:
1. Forthebackup:
curl-u"$USERNAME:$PASSWORD""http://$RABBIT_ADDRESS:15672/api/definitions"-o"$BACKUP_FOLDER/rabbit-backup.json"
2. Fortherestore:
curl-u"$USERNAME:$PASSWORD""http://$RABBIT_ADDRESS:15672/api/definitions"-XPOST-H"Content-Type:application/json"-d"@$BACKUP_FOLDER/rabbit-backup.json"
Whatpre-upgradechecksshouldIdo?BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:
1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.
2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.
3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.
KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .
FileaSupportTicketYoucanfileaticketwithPivotalSupport .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .
Toexpeditetroubleshooting,provideyourservicebrokerlogsandyourserviceinstancelogs.Ifyour cfserviceYOUR-SERVICE-INSTANCE outputincludesatask-id ,providetheBOSHtaskoutput.
©CopyrightPivotalSoftwareInc,2013-2019 99 1.12
©CopyrightPivotalSoftwareInc,2013-2019 100 1.12
UsingOn-DemandRabbitMQforPCFThistopicprovidesinstructionsfordevelopersusingtheon-demandRabbitMQservicefortheirPivotalCloudFoundry(PCF)apps.RabbitMQenablesmessagingbetweencloud-basedservers,appsanddevices.
TheseproceduresusetheCloudFoundryCommand-LineInterface(cfCLI).YoucanalsouseAppsManager toperformthesametasksusingagraphicalUI.
Forgeneralinformation,seeManagingServiceInstanceswiththecfCLI .
PrerequisitesTouseon-demandRabbitMQforPCFwithyourPCFapps,youneed:
APCFinstallationwithRabbitMQforPCF installedandlistedintheMarketplace
ASpaceDeveloper orAdminaccountonthePCFinstallation
Alocalmachinewiththefollowinginstalled:
abrowserashelltheCloudFoundryCommand-LineInterface (cfCLI)theLinuxwatch command
Tologinto theorgandspacecontainingyourapp
DeveloperGuide
EntriesintheVCAP_SERVICESEnvironmentVariableAppsrunninginCloudFoundrygainaccesstotheboundserviceinstancesviaanenvironmentvariablecredentialshashcalledVCAP_SERVICES .Anexamplehashisshownbelow:
{ "p-rabbitmq": [{ "credentials": { "dashboard_url": "http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "password": "tavk86pnnns1ddiqpsdtbchurn", "protocols": { "amqp": { "password": "tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7" ] } } } }]}
Formoreinformationabouttheenvironmentvariable VCAP_SERVICES ,seeRabbitMQEnvironmentVariables.
TheCreate-BindProcessBecauseeveryappandserviceinPCFisscopedtoaspace ,anappcanonlyuseaserviceifaninstanceoftheserviceexistsinthesamespace.
TouseRabbitMQinaPCFapp:
1. UsethecfCLI orAppsManager tologintotheorgandspacethatcontainstheapp.
©CopyrightPivotalSoftwareInc,2013-2019 101 1.12
2. MakesureaninstanceoftheRabbitMQforPCFserviceexistsinthesamespaceastheapp.
IfthespacedoesnotalreadyhaveaRabbitMQforPCFinstance,createone.IfthespacealreadyhasaRabbitforPCFinstance,youcanbindyourapptotheexistinginstanceorcreateanewinstancetobindtoyourapp.
3. BindtheapptotheRabbitMQforPCFserviceinstance,toenabletheapptouseRabbitMQ.
ConfirmServiceAvailabilityForanapptouseaservice,1)theservicemustbeavailableintheMarketplaceforitsspaceand2)aninstanceoftheservicemustexistinitsspace.
YoucanconfirmbothoftheseusingthecfCLIasfollows.
1. TofindoutifOn-DemandRabbitMQforPCFserviceisavailableintheMarketplace:
a. Enter cf marketplaceb. Iftheoutputlists ondemand-rabbitmq inthe service column,on-demandRabbitMQforPCFisavailable.Ifitisnotavailable,askyour
operatortoinstallit.
$cfmarketplaceGettingservicesfrommarketplaceinorgmy-org/[email protected][...]ondemand-rabbitmqSoloRabbitMQService[...]
2. ToconfirmthatanOn-DemandRabbitMQforPCFinstanceisrunninginthespace
a. Enter cf servicesb. Any ondemand-rabbitmq listingsinthe service columnareserviceinstancesofon-demandRabbitMQinthespace.
$cfservicesGettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded
Youcanbindyourapptoanexistinginstanceorcreateanewinstancetobindtoyourapp.
CreateaServiceInstanceUnlikepre-provisionedservices,on-demandservicesarecreatedasynchronously,notimmediately.The watch commandshowsyouwhenyourserviceinstanceisreadytobindanduse.
Tocreateaninstanceoftheon-demandRabbitMQforPCFservice,run cfcreate-service
:
1. Enter cfcreate-serviceondemand-rabbitmqSoloSERVICE_INSTANCE
Where SERVICE_INSTANCE isanameyouchoosetoidentifytheserviceinstance.Thisnamewillappearunder service [sic]inoutputfrom cf
services .
2. Enter watchcfservices andwaitforthe lastoperation foryourinstancetoshowas createsucceeded .
$cfcreate-serviceondemand-rabbitmqSolomy-instance
Creatingservicemy-instanceinorgmy-org/[email protected]
$watchcfservices
Gettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded
Ifyougetanerror,seeTroubleshootingInstances.
©CopyrightPivotalSoftwareInc,2013-2019 102 1.12
BindaServiceInstancetoYourAppForanapptouseaservice,youmustbindittoaserviceinstance.Dothisafteryoupushorre-pushtheappusing cfpush .
TobindanapptoaRabbitMQinstancerun $cfbind-service
.
1. Enter cfbind-serviceAPPSERVICE_INSTANCE
Where APP istheappyouwanttousetheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cfcreate-
service .
$cfbind-servicemy-appmy-instance
Bindingservicemydbtomy-appinorgmy-org/[email protected]:Use'cfpush'toensureyourenvvariablechangestakeeffect
UsetheRabbitMQServiceinYourAppToaccesstheRabbitMQservicefromyourapp:
1. Run cfenvAPP_NAME withthenameoftheappboundtotheRabbitMQforPCFinstance.
2. Intheoutput,notetheconnectionstringslistedinthe VCAP_SERVICES > credentials objectfortheapp.
3. Inyourappcode,calltheRabbitMQserviceusingtheconnectionstrings.
ForhowtocodeyourapptouseRabbitMQmessaging,seeAboutUsingPivotalRabbitMQ>ClientDocumentationintheRabbitMQdocumentation .
UpdatingaServiceInstanceIfyoubindanewserviceorchangetheservicebindings,youneedtorun cf
restarttoupdatethe VCAP_SERVICES environmentvariableintheapplication
container.
1. Enter cfrestart-appAPP
Where APP istheappyouwanttousetheupdatedserviceinstance.
$cfrestartmy-app
Pushingnewversionofanappautomaticallyrestagesandrestartstheapponanyserviceinstancesitisboundto.
UnbindaServiceInstancetoYourAppTostopanappfromusingaserviceitnolongerneeds,unbinditfromtheserviceinstanceusing cfunbind-
service.
1. Enter cfunbind-serviceAPPSERVICE_INSTANCE
Where APP istheappyouwanttostopusingtheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf
create-service .
$cfunbind-servicemy-appmy-instance
Unbindingappmy-appfromservicemy-instanceinorgmy-org/[email protected]
DeleteaServiceInstanceTodeleteaserviceinstance,run cfdelete-
service.
©CopyrightPivotalSoftwareInc,2013-2019 103 1.12
1. Enter cfdelete-serviceSERVICE_INSTANCE
Where SERVICE_INSTANCE isthenameoftheservicetodelete.
$cfdelete-servicemy-instance
Areyousureyouwanttodeletetheservicemy-instance?yDeletingservicemy-serviceinorgmy-org/[email protected]
2. Enter watchcfserviceSERVICE_INSTANCE andwaitfora Serviceinstancenotfound errorindicatingthattheinstancenolongerexists.
Youcannotdeleteaserviceinstancethatanappisboundto.
CreateanAdminUserforaServiceInstanceIfyouwanttogetadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstance,andobtainusercredentialsthatyoucansharewithotherappdevelopers.
Bothoperatorsandappdeveloperscanusethisprocedure.Forinstructions,seeCreateanAdminUserforaServiceInstance.
SharingServiceInstancesInordertoshareserviceinstances thefeature-flag service_instance_sharing mustbeenabledbyyourOperator.YoucanthenfollowthedocumentationtoshareyourserviceinstancesacrossCloudFoundryOrganizationsandSpaces.
FederateExchangesandQueuesYoucanfederateexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.
Tofederateexchangesandqueues,dothefollowing:
1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.
Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.
2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:
{..."uri":"amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris":["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}
Forexample:
{..."uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}
3. Setupfederationasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.
Forinstructionsonfederation,seetheRabbitMQdocumentation .
©CopyrightPivotalSoftwareInc,2013-2019 104 1.12
ShovelExchangesandQueuesYoucanshovelexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.
Toshovelexchangesandqueues,dothefollowing:
1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.
Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.
2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:
{..."uri":"amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris":["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}
Forexample:
{..."uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}
3. Setupshovelasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.
Forshovelinstructions,seetheRabbitMQdocumentation .Atthemoment,RabbitMQforPCFonlysupportsDynamicShovels.
©CopyrightPivotalSoftwareInc,2013-2019 105 1.12
RabbitMQEnvironmentVariablesThistopicprovidesareferencefortheenvironmentvariablesthatPivotalCloudFoundry(PCF)storesforRabbitMQforPCFserviceinstances.Thesevariablesincludethecredentialsthatappsusetoaccesstheserviceinstances.
VCAP_SERVICESAppsrunninginPCFgainaccesstotheboundserviceinstancesthroughanenvironmentvariablecredentialshashcalled VCAP_SERVICES .Anexamplehashisshowbelow:
{"p-rabbitmq":[{"label":"p-rabbitmq","name":"my-rabbit-service-instance","plan":"standard","tags":["rabbitmq","messaging","message-queue","amqp","pivotal"],"credentials":{"dashboard_url":"http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn","username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","vhost":"62e5ab21-7b38-44ac-b139-6aa97af01cd7","password":"#passwordexample123456789","ssl":false,"hostname":"10.0.0.41","hostnames":["10.0.0.41","10.0.0.51"],"uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7"],"http_api_uri":"http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","http_api_uris":["http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api"],"protocols":{"amqp":{"password":"passwordexample123456789","port":5672,"ssl":false,"username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","vhost":"62e5ab21-7b38-44ac-b139-6aa97af01cd7","host":"10.0.0.41","hosts":["10.0.0.41","10.0.0.51"],"uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},"management":{"username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","password":"passwordexample123456789","path":"/api","port":15672,"ssl":false,"host":"10.0.0.41","hosts":["10.0.0.41","10.0.0.51"],"uri":"http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","uris":["http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api"]}}}}]}
Youcansearchforyourservicebyits name ,givenwhencreatingtheserviceinstance,ordynamicallythroughthe tags or label properties.Thecredentials propertycanbeusedasfollows:
Thetoplevelproperties uri , uris , vhost , username , password , hostname ,and hostnames provideaccesstotheAMQP0.9.1protocol.
Amoreflexibleapproachisprovidedbythe credentials.protocols property,whichhasakeyperenabledprotocol.Thepossiblekeysare amqp ,management , mqtt ,and stomp .IfSSLisenabled,thenthekeysare amqp+ssl , management+ssl , mqtt+ssl ,and stomp+ssl respectively.
Thevaluesassociatedwitheachofthesekeysgivesaccesscredentialsspecifictoeachprotocol.Inallcases,URIsareprovided,alongwiththeindividualcomponents.
©CopyrightPivotalSoftwareInc,2013-2019 106 1.12
ChangingEnabledPluginsandProtocols
IfyouadjustthepluginsandprotocolsenabledforRabbitMQ,youmightneedtoforceallapp’sVCAP_SERVICES environmentvariabletoberegenerated.Addingandremovingthefollowingpluginsrequireboundappstoberestaged:
rabbitmq_management
rabbitmq_stomp
rabbitmq_mqtt
rabbitmq_amqp1_0
IncommonwithallservicesinPCF,the VCAP_SERVICES environmentvariableforanappisonlymodifiedwhentheappisboundtoaserviceinstance.Usersneedto cfunbind-
service, cfbind-
service,and cfrestage theirappinthisscenario.
Note:Removingoraddingplugins/protocolsmightcauseappsboundwithRabbitMQtobreak.
©CopyrightPivotalSoftwareInc,2013-2019 107 1.12
TroubleshootingInstancesThistopicprovidesbasicinstructionsforappdeveloperstroubleshootingOn-DemandRabbitMQ®forPCF.
ErrorsYoumayseeanerrorwhenusingtheCloudFoundryCommand-LineInterface(cfCLI)toperformbasicoperationsonaRabbitMQforPCFserviceinstance:
cf create
cf update
cf bind
cf unbind
cf delete
ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cfserviceINSTANCE-NAME .
$cfservicemyservice
Serviceinstance:myserviceService:super-dbBoundapps:Tags:Plan:dedicated-vmDescription:DedicatedInstanceDocumentationurl:Dashboard:
LastOperationStatus:createfailedMessage:Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:createStarted:2017-03-13T10:16:55ZUpdated:2017-03-13T10:17:58Z
Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.
The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe boshtaskTASK-ID .
The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping boshlogsbroker0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.
RetrieveServiceInstanceInformation1. Logintothespacecontainingtheinstanceorfailedinstance.
$cflogin
2. Ifyoudonotknowthenameoftheserviceinstance,run cfservices toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthe name column.
©CopyrightPivotalSoftwareInc,2013-2019 108 1.12
$cfservicesGettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded
3. Run cfserviceSERVICE-INSTANCE-NAME toretrievemoreinformationaboutaspecificinstance.
4. Run cfserviceSERVICE-INSTANCE-NAME--guid toretrievetheGUIDoftheinstance,whichisusefulfordebugging.
RetrieveRabbitMQInstanceCredentialsIfyouwanttoaccesstheManagementDashboardortheRabbitMQserverfortroubleshooting,youcancreateanewservice-keytoretrieveRabbitMQinstancecredentials.Pivotalrecommendsthatyouusethiskeyfortroubleshootingonly,andthatyoudeletethekeyaftertroubleshooting.Toretrievethecredentials,dothefollowing:
1. Createaservice-keyforyourRabbitMQinstanceusingthecommand cfcreate-service-keyINSTANCE-NAMESERVICE-KEY-NAME .
2. Retrievethecredentialsusingthecommand cfservice-keyINSTANCE-NAMESERVICE-KEY-NAME .
Forexample:
$cfcreate-service-keymy-rmq-instancemy-keyCreatingservicekeymy-keyforserviceinstancemy-rmq-instanceasadmin...OK$cfservice-keymy-rmq-instancemy-keyGettingkeymy-keyforserviceinstancemy-rmq-instanceasadmin...{"host":"10.0.8.4","password":"","port":6379}
KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .
FileaSupportTicketYoucanfileasupporttickethere .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .
Toexpeditetroubleshooting,ifpossible,provideyourservicebrokerlogs,serviceinstancelogs,andBOSHtaskoutput.Yourcloudoperatorshouldbeabletoobtainthesefromyourerrormessage.
DeleteRabbitMQInstancesOn-DemandBrokerprovidesaBOSHcommandtodeletealltheOn-DemandBrokerdeployedinstances.Todeletetheinstances,dothefollowingprocedure:
1. RunthefollowingcommandtodeleteallinstancesoftheOn-DemandBroker:
boshrun-erranddelete-sub-deployments
WARNING:Thiscommanddeletesdeploymentinstancesserially.Itisverydestructiveandcannotbeundone.
©CopyrightPivotalSoftwareInc,2013-2019 109 1.12