user's guide pcf jmx bridge - pivotal · the jmx bridge tool is a jmx extension for elastic...

PCFJMXBridge®

Version1.8

User'sGuide

©2018PivotalSoftware,Inc.

234

141720212526

TableofContents

TableofContentsPivotalCloudFoundryJMXBridgeDeployingJMXBridgeUsingJMXBridgeUsingSSLwithaSelf-SignedCertificateinJMXBridgeJMXBridgeResourcesTroubleshootingandUninstallingJMXBridgeApplicationSecurityGroupsReleaseNotesandKnownIssues

©CopyrightPivotalSoftwareInc,2013-2018 2 1.8

Pivotal Cloud Foundry JMX Bridge

ThePivotalCloudFoundry(PCF)JMXBridgecollectsandexposessystemdatafromCloudFoundrycomponentsviaaJMXendpoint.Youcanusethissystemdatatomonitoryourinstallationandassistintroubleshooting.

TheJMXBridgetooliscomposedofthefollowingtwovirtualmachines:

TheJMXprovider

ANozzlefortheLoggregatorFirehose

ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutPCFJMXBridge:

Element Details

Version v1.8.23

Releasedate August10,2017

CompatibleOpsManagerversion(s) v1.8.x,v1.9.x,v1.10.x

CompatibleElasticRuntimeversion(s) v1.8.1orlater,v1.9.x,v1.10.x

IaaSsupport AWS,Azure,GCP,OpenStack,andvSphere

IPsecsupport Yes

JMXBridgeUserGuideDeployingJMXBridge

UsingJMXBridge

UsingSSLwithaSelf-SignedCertificateinJMXBridge

Resources

TroubleshootingandUninstallingJMXBridge

ApplicationSecurityGroups

ReleaseNotesandKnownIssues

WARNING:AsofPivotalCloudFoundry(PCF)v2.1,JMXBridgev1.8isdeprecated.


http://docs.pivotal.io/pivotalcf/1-8/loggregator/architecture.html

Deploying JMX BridgePage last updated:

TheJMXBridgetoolisaJMXextensionforElasticRuntime.FollowtheinstructionsbelowtodeployJMXBridgeusingthePivotalCloudFoundry (PCF)OperationsManager.

Step1:InstalltheJMXBridgeTile

1. DownloadJMXBridge .

2. ImportJMXBridgeintoOpsManagerbyfollowingtheinstructionsforAddingandImportingProducts .

3. OntheInstallationDashboard,clicktheJMX Bridgetile.

TheorangebarontheJMX Bridgetileindicatesthattheproductrequiresconfiguration.

Step2:AssignAvailabilityZonesandNetworks1. SelectAssign AZs and Networks.ThissectionshowstheAvailabilityZonesthatyouCreate whenconfiguringtheBOSHDirector.

2. (vSphere and Amazon Web Services Only)SelectanAvailabilityZoneunderPlace singleton jobs in.ThisAvailabilityZoneiswhereOpsManagerrunsMetricsjobsthatonlyhaveasingleinstance.

3. (vSphere and Amazon Web Services Only)SelectoneormoreAvailabilityZonesunderBalance other jobs in.OpsManagerbalancesMetricsjobswithmorethanoneinstanceacrosstheAvailabilityZonesthatyouspecify.

Note:TousetheFirehoseNozzle,youmustinstallElasticRuntime beforeinstallingJMXBridge.JMXBridgerequiresthecomponentstobeinstalledinthisorder.


https://network.pivotal.io/products/pcf-metrics

https://network.pivotal.io/products/elastic-runtime

https://network.pivotal.io/products/ops-metrics

https://docs.pivotal.io/pivotalcf/1-7/customizing/add-delete.html#add-import

https://docs.pivotal.io/pivotalcf/1-7/customizing/vsphere-config.html#create-az

4. SelectaNetworkfromthedropdownmenu.

5. ClickSave.

Step3:ConfigureJMXProvider1. SelectJMX Provider.

2. EnteranewusernameandpasswordintotheJMX Provider credentialsusernameandpasswordfields.

Note:JMXBridgeusesthedefaultAssignedNetworkifyoudonotselectadifferentnetwork.

Note:Whenyousavethisform,averificationerrordisplays.Youcanignorethiserror.ItonlyappearsbecausethePCFsecuritygroupblocksICMP.


3. Recordthesecredentials.YouusethesetoconnectJMXclientstotheJMXProvider.

(Optional)Step4:EnableNATSupport1. SelecttheEnableradiobutton.NATsupportisdisabledbydefault.EnablingNATsupportallowsyoutosettheNATIPasthehostIP.Bydefault,the

internalIPaddressoftheJMXProviderVMissetasthehostIP.

2. EntertheNATIPastheExternalIPaddressintheform 0.0.0.0

3. Ifyouhaveenabledordisabled NATSupport ,clickSave.

(Optional)Step5:EnableSecurityLogging

Note:ToconnecttotheJMXProviderafterinstall,youmustusethespecifiedIPaddress.TheIPaddressdisplayedinthe Status tabalwaysreflectstheinternalIPaddressoftheJMXProviderVM,nottheexternalIPaddress.


1. SelecttheEnable Security Loggingcheckbox.AccesstotheJMXendpointisloggedtoSTDOUTbydefault.YoucanenablesecurityloggingintheJMXBridgetileconfigurationbyselectingthischeckbox,ordisableitdeselectingthischeckbox.Securityloggingisenabledbydefault.

2. IfyoumadechangestoSecurityLogging,clickSave.

(Optional)Step6:ConfigureSSL1. SelecttheEnable SSLcheckbox.IfyouenableSSL,JMXclientsareforcedtouseSSLtoconnecttotheJMXProvider.

Note:RelatedlogoutputismadeavailablebyinitiatingaJMXProviderlogsdownloadfromtheJMXBridgetileconfigurationstatustab,thenfetchingthedownloadfromthelogstab.


IfyouselecttheEnable SSLcheckbox,youmustalsoprovideanSSLcertificateandprivatekey.TherearetwowaystoprovideanSSLcertificateandprivatekey:

Ifyouareusingasignedcertificate,pasteanX.509certificateintheCertificate PEMfieldandaPKCS#1privatekeyinthePrivate Keyfield.IfyouwanttouseSSLbutdonotwanttouseasignedcertificate,youmustperformthefollowingactions:

1. Generateaself-signedcertificateontheserver.2. Importtheself-signedcertificatetoatruststoreontheclient.3. StartjConsole,oranothermonitoringtool,withthetruststore.

Formoreinformation,seeUsingSSLwithaSelf-SignedCertificate.


2. ClickSave.

(Optional)Step7:ConfigureErrandsErrandsarescriptsthatOpsManagerrunstoautomatetasks.Bydefault,OpsManagerrunsthepost-installerrandslistedbelowwhenyoudeployPCFJMXBridge.However,youcanpreventaspecificpost-installerrandfromrunningbydeselectingitscheckboxontheErrandspage.


SelectSmoke tests for JMX BridgetocausetheJMXBridgetoverifythefollowing:

IftheFirehoseNozzleisenabled,JMXBridgeverifiesthattheNozzleisreceivingmetricsandthattheproductisnotaslowconsumerIfBOSHMetricsareenabled,JMXBridgeverifiesthattheproductisreceivingappropriatehealthmetrics

(Optional)Step8:ResourceConfiguration

TodisabletheFirehoseNozzleorstopreceivingElasticRuntime (includingDiego)metrics,modifytheinstancecountoftheOpenTSDB FirehoseNozzlefrom 1 to 0 .

Step9:ApplyChanges1. NavigatetothePCFOpsManagerInstallationDashboard.

2. InthePendingChangesview,clickApply ChangestoinstallJMXBridge.

Note:Iferrorsoccurduringtheinstallduetosmoketests,refertothetroubleshootingdocumentationformoreinformation.

Note:TheFirehoseNozzleisenabledbydefaultandrequiresElasticRuntime .




Afterinstallationcompletes,amessageappearsstatingthatthechangeshavebeenapplied.

Step10:FindtheIPAddressoftheJMXProvider1. ClickReturn to Product Dashboard.

2. ClicktheJMX BridgetileandselecttheStatustab.

3. RecordtheIPaddressoftheJMX Provider.

Step11:ConfiguretheMetricsIPAddress1. ReturntotheInstallation Dashboard.ClicktheBOSH DirectortileandselectDirector Config.

Note:Afterinstallation,yourJMXclientconnectstothisIPaddressatport44444usingthecredentialsthatyousupplied.AlsoensurethatTCPport44445isopen.


2. IntheMetrics IP Addressfield,entertheIPaddressoftheJMXProvider.ClickSave.

Step12:CompleteInstallation1. InthePendingChangesview,clickApply Changes.

Whencomplete,amessageappearsstatingthatthechangeshavebeenapplied.

2. ClickReturn to Product Dashboard.JMXBridgeisnowinstalledandconfigured.

Afterinstallationandconfiguration,metricsforCloudFoundrycomponentsautomaticallyreporttotheJMXendpoint.


Using JMX BridgePage last updated:

JMXBridgeisaJavaManagementExtensions(JMX)toolforElasticRuntime.Tohelpyoumonitoryourinstallationandassistintroubleshooting,JMXBridgecollectsandexposessystemdatafromCloudFoundrycomponentsviaaJMXendpoint.

CloudControllerMetricsJMXBridgereportsthenumberofCloudControllerAPIrequestscompletedandtherequestssentbutnotcompleted.

Thenumberofrequestssentbutnotcompletedrepresentsthependingactivityinyoursystem,andcanbehigherunderload.Thisnumberwillvaryovertime,andtherangeitcanvaryoverdependsonspecificsofyourenvironmentsuchashardware,OS,processorspeeds,load,etc.Inanygivenenvironment,though,youcanestablishatypicalrangeofvaluesandmaximumforthisnumber.

UsetheCloudControllermetricstoensurethattheCloudControllerisprocessingAPIrequestsinatimelymanner.Ifthependingactivityinyoursystemincreasessignificantlypastthetypicalmaximumandstaysatanelevatedlevel,CloudControllerrequestsmaybefailingandadditionaltroubleshootingmaybenecessary.

ThefollowingtableshowsthenameoftheCloudControllermetric,whatthemetricrepresents,andthemetrictype(datatype).

METRIC NAME DEFINITION METRIC TYPE (DATATYPE)

cc.requests.completedNumberofCloudControllerAPIrequestscompletedsincethisinstanceofCloudControllerstarted

Counter(Integer)

cc.requests.outstandingNumberofCloudControllerAPIrequestsmadebutnotcompletedsincethisinstanceofCloudControllerstarted

Counter(Integer)

SeetheCloudController topicformoreinformationabouttheCloudController.

RouterMetricsJMXBridgereportsthenumberofsentrequestsandthenumberofcompletedrequestsforeachCloudFoundrycomponent.

Thedifferencebetweenthesetwometricsisthenumberofrequestsmadetoacomponentbutnotcompleted,andrepresentsthependingactivityforthatcomponent.Thenumberforeachcomponentcanvaryovertime,andistypicallyhigherunderload.Inanygivenenvironment,though,youcanestablishatypicalrangeofvaluesandmaximumforthisnumberforeachcomponent.

UsethesemetricstoensurethattheRouterispassingrequeststoothercomponentsinatimelymanner.Ifthependingactivityforaparticularcomponentincreasesignificantlypastthetypicalmaximumandstaysatanelevatedlevel,additionaltroubleshootingofthatcomponentmaybenecessary.Ifthependingactivityformostorallcomponentsincreasessignificantlyandstaysatelevatedvalues,troubleshootingoftheroutermaybenecessary.

ThefollowingtableshowsthenameoftheRoutermetric,whatthemetricrepresents,andthemetrictype(datatype).

METRIC NAME DEFINITION METRIC TYPE (DATATYPE)

gorouter.requests[component=c]

NumberofrequeststherouterhasreceivedforcomponentcsincethisinstanceoftherouterhasstartedccanbeCloudControllerorroute-emitter

Counter(Integer)

gorouter.responses[status=s,component=c]

NumberofrequestscompletedbycomponentcsincethisinstanceoftherouterhasstartedccanbeCloudControllerorroute-emittersishttpstatusfamily:2xx,3xx,4xx,5xx,andother

Counter(Integer)

SeetheRouter topicformoreinformationabouttheRouter.

Note:IfusingJMXBridgev1.8withPCFv1.10,pleaseseethefollowingrecommendedKeyPerformanceIndicators .


https://docs.pivotal.io/pivotalcf/1-10/monitoring/kpi.html

https://docs.pivotal.io/pivotalcf/concepts/architecture/cloud-controller.html

https://docs.pivotal.io/pivotalcf/concepts/architecture/router.html

DiegoMetricsPivotalJMXBridgereportsmetricsfortheDiegocellsandfromtheDiegoBulletinBoardSystem(BBS).ThefollowingtablesshowthenameoftheDiegometric,whatthemetricrepresents,andthemetrictype(datatype).

ForgeneralinformationaboutDiego,seetheDiegoArchitecture topic.

DiegoCellMetricsPivotalJMXBridgereportsthefollowingmetricsforeachDiegocell.Ifyouhavemultiplecells,JMXBridgereportsmetricsforeachcellindividually.Themetricsarenotsummedacrosscells.

Usethesemetricstodeterminethesizeofyourdeploymentorwhentoscaleupadeployment,andtotrackthestatusofLongRunningProcesses(LRP)intheDiegolifecycle.

METRIC NAME DEFINITION METRIC TYPE (DATA TYPE)

rep.CapacityTotalMemory Totalamountofmemoryavailableforthiscelltoallocatetocontainers Gauge(Float)

rep.CapacityRemainingMemory Remainingamountofmemoryavailableforthiscelltoallocatetocontainers Gauge(Float)

rep.CapacityTotalDisk Totalamountofdiskavailableforthiscelltoallocatetocontainers Gauge(Float)

rep.CapacityRemainingDisk Remainingamountofdiskavailableforthiscelltoallocatetocontainers Gauge(Float)

rep.ContainerCount Numberofcontainershostedonthecell Gauge(Integer)

DiegoBBSMetricsPivotalJMXBridgereportsthesemetricsfromtheDiegoBBS,andaredeployment-widemetrics.Usethesemetricstoinspectthestateoftheappsrunningonthedeploymentasawhole.

METRIC NAME DEFINITION METRIC TYPE (DATA TYPE)

bbs.CrashedActualLRPs TotalnumberofLRPinstancesthathavecrashed Gauge(Integer)

bbs.LRPsRunning TotalnumberofLRPinstancesthatarerunningoncells Gauge(Integer)

bbs.LRPsUnclaimed TotalnumberofLRPinstancesthathavenotyetbeenclaimedbyacell Gauge(Integer)

bbs.LRPsClaimed TotalnumberofLRPinstancesthathavebeenclaimedbysomecell Gauge(Integer)

bbs.LRPsDesired TotalnumberofLRPinstancesdesiredacrossallLRPs Gauge(Integer)

bbs.LRPsExtra TotalnumberofLRPinstancesthatarenolongerdesiredbutstillhaveaBBSrecord Gauge(Integer)

bbs.LRPsMissing TotalnumberofLRPinstancesthataredesiredbuthavenorecordintheBBS Gauge(Integer)

VirtualMachineMetricsJMXBridgereportsdataforeachvirtualmachine(VM)inadeployment.UsethesemetricstomonitorthehealthofyourVirtualMachines.

ThefollowingtableshowsthenameoftheVirtualMachinemetric,whatthemetricrepresents,andthemetrictype(datatype).

METRIC NAME DEFINITION METRIC TYPE(DATA TYPE)

system.cpu.sys AmountofCPUspentinsystemprocesses Gauge(Float)

system.cpu.user AmountofCPUspentinuserprocesses Gauge(Float)

system.cpu.wait AmountofCPUspentinwaitingprocesses Gauge(Float)

system.disk.ephemeral.percent PercentageofephemeraldiskusedontheVM Gauge(Float,0-100)

system.disk.ephemeral.inode.percent Percentageofinodesconsumedbytheephemeraldisk Gauge(Float,0-100)

system.disk.persistent.percent PercentageofpersistentdiskusedontheVM Gauge(Float,0-100)

system.disk.persistent.inode.percent Thepercentageofinodesconsumedbythepersistentdisk Gauge(Float,0-100)

system.disk.system.percent PercentageofsystemdiskusedontheVM Gauge(Float,0-100)


https://docs.pivotal.io/pivotalcf/concepts/diego/diego-architecture.html

system.healthy IndicateswhetheraVMsystemishealthy.`1`meansthesystemishealthy,and`0`meansthesystemisnothealthy

Gauge(Float,0-1)

system.load.1m Amountofloadthesystemisunder,averagedoveroneminute Gauge(Float)

system.mem.percent PercentageofmemoryusedontheVM Gauge(Float)

system.swap.kb AmountofswapusedontheVMinKB Gauge(Float)

system.swap.percent PercentageofswapusedontheVM Gauge(Float,0-100)


Using SSL with a Self-Signed Certificate in JMX BridgePage last updated:

SecureSocketLayer(SSL)isastandardprotocolforestablishinganencryptedlinkbetweenaserverandaclient.TocommunicateoverSSL,aclientneedstotrusttheSSLcertificateoftheserver.

ThistopicexplainshowtouseSSLwithaself-signedcertificateinJMXBridge(formerlyOpsMetrics).ThisSSLlayersecurestrafficbetweenJMXBridgeandtheuser,andisseparatefromtheSSLlayerconfiguredbetweenElasticRuntime andtherestoftheOpsManagerenvironment.

TherearetwokindsofSSLcertificates:signedandself-signed.

Signed:ACertificateAuthority(CA)signsthecertificate.ACAisatrustedthirdpartythatverifiesyouridentityandcertificaterequest,thensendsyouadigitallysignedcertificateforyoursecureserver.Clientcomputersautomaticallytrustsignedcertificates.Signedcertificatesarealsocalledtrustedcertificates.

Self-signed:Yourownservergeneratesandsignsthecertificate.Clientsdonotautomaticallytrustself-signedcertificates.TocommunicateoverSSLwithaserverprovidingaself-signedcertificate,aclientmustbeexplicitlyconfiguredtotrustthecertificate.

ThefollowingprocedureconfiguresaJMXuserclienttotrustaself-signedcertificatebyimportingthecertificatetoitstruststore,aninternalkeystore.TouseatrustedcertificatesignedbyaCA,youonlyneedtopastetheCertificateandKeyintothefieldsintheOpsManagerJMXBridgetile,asshowninStep1,Option2,below.

Step1:SupplySSLCertificate

Option1:GenerateSelf-SignedCertificateFollowthestepsbelowtogenerateaself-signedcertificateonyourserver:

1. InPivotalOpsManager,clicktheJMX Bridgetile.

2. CheckEnable SSL.

3. ClickGenerate Self-Signed RSA CertificateandchecktheTrust Self-Signed Certificatesbox.

4. Enteryoursystemandapplicationdomainsinwildcardformat.Optionally,alsoaddanycustomdomainsinwildcardformat.ClickGenerate.

Note:CertificatesgeneratedinElasticRuntimearesignedbytheOperationsManagerCertificateAuthority.Theyarenottechnicallyself-signed,buttheyarereferredtoas‘Self-SignedCertificates’intheOpsManagerGUIandthroughoutthisdocumentation.


https://docs.pivotal.io/pivotalcf/1-7/opsguide/security_config.html

5. Selectandcopythecertificate.

6. Pastethecertificateintoatextfileandsaveasa .cer file,suchas MY-JMX-BRIDGE.cer .

Option2:UseanExistingSelf-SignedCertificate1. InPivotalOpsManager,clicktheJMX Bridgetile.

2. CheckEnable SSL.

3. Pasteyourcertificateandprivatekeyintotheappropriateboxes.ThisisyourX.509certificateandPKCS#1privatekey.

Step2:ImporttheSelf-signedCertificatetoaTruststoreFollowthestepsbelowtoimporttheself-signedcertificatetoyourclient:

1. Copyyourcertificatefile MY-JMX-BRIDGE.cer fromyourservertoyourclient.

2. Navigatetotheclientdirectorywhereyoucopiedthesavedcertificate.

3. Use keytool-import toimportthecertificatewithanaliasof ops-metrics-ssl tothetruststore localhost.truststore :


$keytool-import-aliasops-metrics-ssl-fileMY-JMX-BRIDGE.cer-keystorelocalhost.truststore

If localhost.truststore alreadyexists,apasswordpromptappears.Enterthekeystorepasswordthatyourecordedinapreviousstep.If localhost.truststore doesnotexist,youmustcreateapassword.

4. Verifythedetailsoftheimportedcertificate.

Step3:StartaMonitoringToolwiththeTruststoreOnceyouimporttheself-signedcertificateto localhost.truststore ontheclient,configureyourmonitoringtool,suchasJconsole,tousethetruststore.Youdothisfromacommandline,bystartingyourmonitoringtoolwiththelocationandpasswordofthetruststore.

1. Passinthelocationof localhost.truststore toyourmonitoringtoolwiththe javax.net.ssl.trustStore property,anditspasswordwiththejavax.net.ssl.trustStorePassword property.Forexample,youwouldinvokejConsolewith:

$jconsole-J-Djavax.net.ssl.trustStore=/lib/home/jcert/localhost.truststore-J-Djavax.net.ssl.trustStorePassword=KEYSTORE_PASSWORD

2. IntheRemote Processfield,enterthefullyqualifiedhostnameoftheMaximusserver,portnumber 44444 .

3. TocompletetheUsernameandPasswordfields,refertotheCredentialstaboftheJMXBridgetileinPivotalOpsManager.Bydefault,thesecredentialsare admin and admin .

YourmonitoringtoolshouldnowcommunicatewithyourserverthroughtheSSLconnection.


JMX Bridge Resources

ResourceRequirementsThefollowingtableshowsthedefaultresourceandIPrequirementsforinstallingthetile:

Product Resource Instances CPU RAM Ephemeral Persistent Static IP Dynamic IP

JMXBridge JMXProvider 1 2 4GB 8GB 1GB 1 0

JMXBridge OpenTSDBFirehoseNozzle 1 2 4GB 8GB 1GB 1 0

JMXBridge SmokeTests 1 4 4GB 8GB 0 1 0

GuidelinesIfyouanticipatealargevolumeofmetricscomingfromtheFirehose,thenscaleupthenumberofOpenTSDBFirehoseNozzleinstancesaccordingly.


Troubleshooting and Uninstalling JMX BridgePage last updated:

ThistopicdescribeshowtoresolvecommonissueswiththeJMXBridgeforPivotalCloudFoundry(PCF)tileandhowtouninstallthetileifnecessary.

TroubleshootThefollowingsectionsprovidehelpwithtroubleshootingJMXBridgeforPCF.

DependencyErrorDuringInstallYoumightseethefollowingerrorifyourinstalledversionofElasticRuntimeisnotcompatiblewiththeversionofJMXBridgethatyouareattemptingtoinstall: JMXBridgerequires'cf'version'~>1.8.0.0'asa

dependency.

FollowtheinstructionsbelowthatcorrespondtoyourinstalledElasticRuntimeversion:

Elastic Runtime v1.8.0 to v1.8.16:1. Downloadversionv1.8.17orlateroftheElasticRuntimetile andimportitintoyourOpsManagerInstallation Dashboard.2. Clickthenewlyaddedtiletoreviewanyconfigurableoptions.Formoreinformation,seeInstallingPivotalCloudFoundry andchooseyour

IaaS.3. ClickApply ChangestoinstallbothElasticRuntimeandJMXBridge.

Elastic Runtime v1.8.17 or later:UpgradetheJMXBridgetiletov1.8.9orlater.

CyclicDependencyErrorDuringInstallYoumightseethefollowingerrorifyouhaveElasticRuntimeandJMXBridgeinstalled.

JMX Bridge v1.8.7 or later; Elastic Runtime v1.8:1. TheCollectorcomponentisnolongercompatiblewithJMXBridge.AllmetricsthatwerepreviouslyflowingthroughtheCollectorarenowflowing

throughtheFirehoseinElasticRuntimev1.8.2. ScaledowntheCollectorin Elastic Runtime > Resource Config to 03. ClickApply ChangesandcontinueyourJMXBridgeinstallation.

MissingMetricsfromPCFInstallationorFirehoseIfyoudonotseeexpectedmetricsfromElasticRuntimeintheJMXprovider,verifythatyouinstalledElasticRuntimebeforeJMXBridge.IfyouinstalledJMXBridgefirst,performthefollowingsteps:

1. SSHintotheopentsdb-firehose-nozzleVM.ForinformationabouthowtousetheBOSHCLItoSSHintoaVM,seeAdvancedTroubleshooting

Note:AknownissueexistswitholderversionsofJMXBridgeandthenewerversionsofElasticRuntime.JMXBridgev1.8.2throughv1.8.8onlyworkwithElasticRuntimev1.8.0throughv1.8.16.

Note:ThisisonlyanissueforElastic Runtime v1.8.TheCollectordoesnotexistinElasticRuntimev1.9.



https://docs.pivotal.io/pivotalcf/installing/index.html

https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html

withtheBOSHCLI .

2. Grantsudoaccesstothemachine:

$sudo-i

3. Restartthe opentsdb-firehose-nozzle job.

$monitrestartopentsdb-firehose-nozzle

MissingBOSHMetricsIfyoudonotseeexpectedmetricsfromBOSH,trythefollowingsteps:

1. MakesuretheIPaddressinJMX Bridge > Status > JMX ProvidermatchesthevalueenteredinBOSH Director > Director Config > MetricsIP Address.

2. IftheJMXProviderIPaddressmatchestheMetricsIPaddressandyouseenoBOSHmetricsinthesystem,contactPivotalSupport forhelp.

ValidatingJMXBridgeMBeansIfyoudonotseemetricsfromJMXBridgeinyourthird-partytoolingintegrationasexpected,firsttrythefollowingstepstoquicklydebugwhetherthereisanissuewiththeJMXBridgeproductoriftheissueiswiththetoolingintegration:

1. VerifyJava6+ isinstalled.

2. Run jconsole :

$jconsole

3. SelectRemote ProcessandentertheIPoftheJMX ProviderVMwithport 44444 .

4. FillintheusernameandpasswordfortheJMX ProviderthatwasenteredduringinstallationofJMXBridge.

5. ClickConnect.

6. AllowInsecure connectionifSSLwasnotenabled.


https://support.pivotal.io/

http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html

YoucannowviewallMBeansemittedbyJMXBridge.

SetUpPortForwardingforJMX

Ifyouareconnectingtojconsolefromalocationdifferentfromtheinstalllocation(forexample,deployedonAWSorGCP),youhavetosetupportforwardingtoaccesstheMBeans.

1. Setupportforwardingononetabofyourconsoleandkeepitopen:

[email protected]

2. Start jconsole inanewtabandsetupthe socksProxyPort totheforwardedport:

jconsole-J-DsocksProxyHost=localhost-J-DsocksProxyPort=7777

3. Navigate jconsole asnormal.

SmokeTests

Note:IfyouhaveenabledSSL,seeUsingSSLwithaSelf-SignedCertificateinJMXBridge.


Iferrorsoccurwhenthesmoketestsrun,youcanfindtheerrorsintheChangeLogfortheinstallation.Somecommonfailuresarelistedbelow.

Error internalMetricsAreSent() Fails

Cause TheJMXProviderdidnotreceiveinternalhealthmetricsfromtheOpenTSDBFirehoseNozzle.

Solution RestarttheOpenTSDBFirehoseNozzleVMandcheckthelogstoverifyitisrunningcorrectly.

Error receivingFirehoseMetrics() Fails

Cause TheOpenTSDBFirehoseNozzleisnotreceivingmetricsfromtheFirehose.

SolutionRestarttheOpenTSDBFirehoseNozzleVMandcheckthelogstoverifyitisconnectedtotheFirehose.IfyouseealotofreconnectattemptsinthelogsthenyoulikelyneedtoscaleupthenumberofOpenTSDBFirehoseNozzleinstancesintheResource Configtab.

UninstallTouninstalltheJMXBridgeforPCFtile,seeDeletingaProduct .


https://docs.pivotal.io/pivotalcf/customizing/add-delete.html#del-prod

Application Security GroupsPCFapplicationsdonotinteractdirectlywiththePCFJMXBridgetile.Therefore,youdonotneedtocreateApplicationSecurityGroups(ASGs)tointeractwiththebridgefromanexternalapplication.


Release Notes and Known Issues

Version1.8.24Release Date: October 13, 2017

ReleaseNotesMaintenanceupdateofthefollowingproductdependencies:

OpenJDKnowv1.8.0.144Golangnowv1.9.1

Thestemcellforv1.8.24remainsv3363

KnownissuesNone

Version1.8.23Release Date: August 10, 2017


OpenJDKnowv1.8.0.141Golangnowv1.8.3Guavanowv23.0BouncyCastlenowv1.57


KnownissuesNone

Version1.8.22Release Date: May 25, 2017

ReleaseNotesImprovementstoOpenTSDBFirehoseNozzlebehaviorduringconnectivityissues

WhentheOpenTSDBFirehosenozzlehadaconnectivityissuetothemaximusvm,themetricsqueuecouldfilltocapacity,leavingthenozzleunabletosufficientlyrecoveroncetheendpointconnectionwasrestoredBecausetheproductemphasizestransmissionofonlythelatestmetricvalues,inthecaseofabackupduetoaninabilitytowritetomaximus,thefirehosenozzlewillnowdropbacked-upmessagesandwillwriteoutanerrorwiththenumberofmessagesbeingdropped.Thiswillallowthe

Note:Beforev1.7.X,JMXBridgewasknownasOpsMetrics.ForOpsMetricsreleasenotesandknownissues,seeVersion1.6.X .


https://docs.pivotal.io/pivotalcf/1-6/pcf-release-notes/p1-v1.6/opsmetrics_rn_1_6.html

nozzletobetterrecoveruponrecoveryoftheendpointconnection

Examplelogmessageoutput 2017/05/22 18:40:29 Could not write to maximus VM. Dropping 2942 messages.


KnownissuesNone



OpenJDKnowv1.8.0.131Golangnowv1.8.1SLF4Jnowv1.7.25Logbacknowv1.1.11GRPC-ALLnowv1.3.0Guavanowv21.0BouncyCastlebcprov-jdk15onnowv1.56


KnownissuesNone


ReleaseNotesUpdatesthestemcellrequiredtov3363

KnownissuesNone

Version1.8.18Release Date: March 09, 2017


UpdatestheOpenJDKversionusedintheproducttov1.8.0.121


Updatesthegolangversionusedintheproducttov1.8


KnownissuesNone

Version1.8.14Release Date: January 18, 2017

ReleaseNotesJMXBridgev1.8.14—BUGFIX:Fixestheissueidentifiedinearlierv1.8.xversionswheretheJMXBridgesmoketestswouldshowanSSLconnectionerrorwhenrunningwithSSLenabled.


KnownissuesNone

Version1.8.12Release Date: January 5, 2017

ReleaseNotesJMXBridgev1.8.12—ThismaintenancepatchmakesanimprovementtohowthefirehoseconnectiontotheJMXprovideristested.ThepriorSlowConsumer metricwasnotaconsistentenoughindicatorthatthefirehosenozzlemayneedtoscale.Thispatchintroducesametricopentsdb.nozzle.totalFirehoseDisconnects thatincrementsthenumberofJMXBridgefirehosedisconnectsandlogseachdisconnecttoSTDOUT.

Thismetriccanbeusedtohelpindicateaneedtoscaletheproductfirehosenozzle.Whenexperiencingtruncatedordroppedmessages,firstrestarttheFirehoseNozzleVMandchecklogstovalidateitisconnected.Ifseveraldisconnectsarelogged,youshouldincreasethenumberofOpenTSDBFirehoseNozzleinstancesinthetileResourceConfig.Firehosereconnectattemptswillshowadelayof5secondsbetweenattemptsinordertoreducetheriskofexcessivecallstoUAA.Formoreinfoontroubleshootingthefirehoseconnection,seetroubleshooting.


KnownissuesBUGIDENTIFIED:TheintegrationtestswillgivenanSSLconnectionerrorwhenrunningwithSSLenabled.Theresolutiontothisissueistodisablethesmoketests,ortoupgradetov1.8.14whichcontainsafixtothisbug.

Version1.8.11Release Date: December 15, 2016

ReleaseNotesJMXBridgev1.8.11—Enhancement:AnewoptionalfeaturetoenablesupportofanExternalIPinaNAT’dcustomerenvironmenthasbeenaddedforoperatorswithinthetileconfigunderJMXProvider


OperatorsareabletosettheNATIPasthehostIP,atwhichpointtheJMXProvidercanbeconnectedtoatthespecifiedIPThisenhancementaddressesusageofJMXBridgeinaNAT’dnetwork,wheretheproductprovidedendpointwouldbecomeunavailable.Whileamanualconfigurationwaspossibletoovercomethis,themanualchangewouldnotholdthroughsubsequentenvironmentupdatesThisfeatureisdisabledbydefault.Onceenabled,itwillcontinuetopersistthroughfutureupdatesuntiloperatordisabledSeehowtoenable

Releasealsoincludesamaintenanceupdateofthefollowingproductdependencies:

SLF4Jnowv1.7.21Logbacknowv1.1.7Guavanowv20.0BouncyCastlebcprov-jdk15onnowv1.55



Version1.8.9Release Date: November 29, 2016

ReleaseNotesJMXBridgev1.8.9—EarlierversionsofElasticRuntimev1.8hadanissuewiththe provides_product_version property.ElasticRuntimev1.8.17andlatercorrectthisissue,whichnecessitatedanupdatetohowJMXBridgev1.8handlestheproperty.

WerecommendyouupgradetoElasticRuntime1.8.17orlater,howeverJMXBridgev1.8.9cansupportElasticRuntimev1.8.0orlaterEarlierversionsofJMXBridge(v1.8.2throughv1.8.8)willnowonlyworkwithElasticRuntimev1.8.0throughv1.8.16.




ReleaseNotesJMXBridgev1.8.8—BUGFIX:Fixestheissueidentifiedinv1.8.7wherethenewintegrationtestswouldfailifthedopplerinstancesweregreaterthan1.Asofthisversion,theintegrationtestsnowcorrectlyhandlemultipledopplerinstances.

Stemcellforv1.8.8remainsv3263


JMXBridgev1.8.2throughv1.8.8willonlyworkwithElasticRuntimev1.8.0throughv1.8.16



ReleaseNotesJMXBridgev1.8.7-NowenforcesthedependencystatedintheproductdocumentationtoinstallElasticRuntime beforeinstallingJMXBridge

EnforcingthecorrectinstallationorderensuresagoodconnectionbetweenJMXBridgeandthefirehoseIfinstallationofJMXBridgeisattemptedbeforeElasticRuntime,theuserwillnowreceiveaninstallationerrorstatingtherequireddependency

Enhancement:Thefirehosenozzleisnowenabledbydefaultoninstallation

Thisenhancementaddressescommonfeedbackthatthemajorityofusecasesrequirethefirehosetobeenabled,andthereforeeliminatingthenozzleenablementstepspeedsproductinstallationForproductusecasesthatdonotincludefirehosenozzleusage,theOpenTSDBnozzlemaystillbedisabledOpenTSDBnozzleenabled/disabledpreferencewillcontinuetoberetainedthroughfutureupgrades

Enhancement:ThetilenowincludesSmokeTestswhichareexecutedviaerranduponinstallation/upgrade

Anyerrorsresultingfromthesmoketestswillbesummarizedattheendofthelogoutput,andcanbereferencedagainsttheproducttroubleshootingdocumentation

UpdatestheOpenJDKversiontov1.8.0.111


KnownissuesBUGIDENTIFIED:Theintegrationtestswillfailifthedopplerinstancesaregreaterthan1.Theresolutiontothisissueistodisablethesmoketests,ortoupgradetov1.8.8whichcontainsafixtothisbug.

BUGIDENTIFIED:TheintegrationtestswillgivenanSSLconnectionerrorwhenrunningwithSSLenabled.Theresolutiontothisissueistodisablethesmoketests,ortoupgradetov1.8.14whichcontainsafixtothisbug.


Version1.8.6Release Date: October 11, 2016

ReleaseNotesJMXBridgev1.8.6—BUGFIX:AsofPCFv1.8,thefirehosebegantosendthedropsondeenvelopeindexasaGUIDinsteadofaninteger.TheOpenTSDBnozzle,ifenabled,continuedtoassumeparsingbyintegercausingsomenullvaluestobeoutput.Asofv1.8.6,theOpenTSDBnozzlenowcorrectlyparsestheindexfieldasaGUID.


KnownissuesThereisnothingthatpreventstheuserfromturningontheNozzledeploymentwhenElasticRuntimeisnotpresent.

EnablingthenozzlewhenElasticRuntimeisnotdeployedorenabledwillproducethefollowingerror:“RuntimeError—unknownproduct‘cf’in((..cf.cloud_controller.system_domain.value))”Tofixthis,reducethenumberofnozzlecountstozerountilElasticRuntimeisenabled.

BecausedeployingtheOpenTSDBfirehosenozzleisoptionalanddisabledbydefault,smoketestsforElasticRuntimehavebeendisableduntilalaterrelease.




Version1.8.4Release Date: September 22, 2016

ReleaseNotesJMXBridgev1.8.4—nowwithstemcellv3263

KnownissuesBUGIDENTIFIED:PriorversionsofJMXv1.8.xbeforev1.8.6didnotproperlyaccountforaFirehosechange(appearingasofPCFv1.8),wheretheFirehose(akaLoggregator)begantosendthedropsondeenvelopeindexasaGUIDinsteadofaninteger.Whenenabled,theOpenTSDBnozzleparsedtheindexfieldasaninteger,causingparsingfailure.AsofJMXv1.8.6andlater,thisOpenTSDBparsingnowassumesGUIDinsteadofinteger.

ThereisnothingthatpreventstheuserfromturningontheNozzledeploymentwhenElasticRuntimeisnotpresent.

EnablingthenozzlewhenElasticRuntimeisnotdeployedorenabledwillproducethefollowingerror:“RuntimeError—unknownproduct'cf’in((..cf.cloud_controller.system_domain.value))”Tofixthis,reducethenumberofnozzlecountstozerountilElasticRuntimeisenabled.




MajorFeaturesAccesstotheJMXendpointisnowloggedtoSTDOUTbydefault.Thissecurityloggingcanbeenabled/disabledintheJMXBridgetileconfigurationbychecking/unchecking“EnableSecurityLogging”underJMXProvider.

RelatedlogoutputismadeavailablebyinitiatingaJMXProviderlogsdownloadfromtheJMXBridgetileconfigurationstatustab,thenfetchingthedownloadfromthelogstab.

ReleaseNotesJMXBridgev1.8.3—BUGFIX:Correctsissueinv1.8.2wherethenewsecurityloggingfeaturewasnotavailabletotheoperatorinthetileconfig.


KnownissuesBUGIDENTIFIED:PriorversionsofJMXv1.8.xbeforev1.8.6didnotproperlyaccountforaFirehosechange(appearingasofPCFv1.8),wheretheFirehose(akaLoggregator)begantosendthedropsondeenvelopeindexasaGUIDinsteadofaninteger.Whenenabled,theOpenTSDBnozzleparsedtheindexfieldasaninteger,causingparsingfailure.AsofJMXv1.8.6andlater,thisOpenTSDBparsingnowassumesGUIDinsteadofinteger.







ReleaseNotesJMXBridgev1.8.2istargetedforPCFv1.8.X

Stemcellforv1.8.2isv3262

KnownissuesBUGIDENTIFIED:ThenewsecurityloggingfeatureexpectedinJMXBridgev1.8.Xwasnotavailabletotheoperatorinthetileconfiginversionv1.8.2.JMXBridgev1.8.3correctsthis.

BUGIDENTIFIED:PriorversionsofJMXv1.8.xbeforev1.8.6didnotproperlyaccountforaFirehosechange(appearingasofPCFv1.8),wheretheFirehose(akaLoggregator)begantosendthedropsondeenvelopeindexasaGUIDinsteadofaninteger.Whenenabled,theOpenTSDBnozzleparsedtheindexfieldasaninteger,causingparsingfailure.AsofJMXv1.8.6andlater,thisOpenTSDBparsingnowassumesGUIDinsteadofinteger.





MetricNameChangesinPCFv1.8InJMXBridgev1.8,metrics fromaPCFdeploymenttravelthroughtwopathstotheJMXBridgeendpoint:theFirehoseandtheBOSHDirector.

InPCFv1.7andJMXBridgev1.7,thecollectorwasstillametricsflowpathforCloudController,Gorouter,andETCDmetrics.WithPCFv1.8andJMXBridgev1.8,thecollectorwasretiredandthesecomponentsnowbroadcastviatheFirehose.

Someoldermetricsmayhavebeenretiredorhadtheirnamesupdated,seePCFv1.8metrics formoreinformationontheindividualcomponentmetrics.

InJMXBridge,allmetricsthattravelthroughtheJMXBridgeFirehosenozzleareprependedwiththenamingconvention opentsdb.nozzle.<metric>

PastMinorVersion1.7.XReleaseNotesforv1.7.Xreleasescanbefoundhere .


https://docs.pivotal.io/pivotalcf/1-8/loggregator/all_metrics.html

https://docs.pivotal.io/pivotalcf/1-8/loggregator/all_metrics.html

http://docs.pivotal.io/jmx-bridge/1-7/rn-ki.html

user's guide pcf jmx bridge - pivotal · the jmx bridge tool is a jmx extension for elastic...

Documents