spring cloud data flow for pivotal cloud foundry · spring cloud® data flow for pcf about spring...

SpringCloudDataFlowfor

PivotalCloudFoundry®

Documentation

Version1.1.4

Published:2April2019

©2019PivotalSoftware,Inc.AllRightsReserved.

SpringCloud®DataFlowforPCF

AboutSpringCloudDataFlowSpringCloudDataFlowisanopen-sourcetoolkitfordataintegrationandreal-timedataprocessing(seetheprojecthomepage ).TheSpringCloudDataFlow(SCDF)serverdeployspipelinescomposedofSpringCloudStream orSpringCloudTask applications.Adomain-specificlanguage(DSL)makesiteasytodescribetheapplicationsinapipelineandhowtheyareconnected.

TheSCDFserverexposesaRESTAPIforcomposinganddeployingdatapipelinesandadashboardwithagraphicalpipelineeditor.ItalsoincludesashellforworkingwiththeAPIfromthecommandline.

AboutSpringCloudDataFlowforPCFSpringCloudDataFlowforPivotalCloudFoundry(PCF)automatesthedeploymentofSCDFanditsdependentservicessothatdeveloperscanuseAppsManagertodeploytheirownSCDFinstances.TheycanthenaccesstheirDataFlowserver,eitherviatheSCDFdashboardUIorusingtheirPCFcredentialswiththeSCDFshell.

KeyFeaturesSpringCloudDataFlowforPCFincludesthefollowingkeyfeatures:

AdditionoftheSpringCloudDataFlowservertotheMarketplaceasamanagedservice

AutomaticintegrationwithdependentPCFservices:

MySQLforPCF(orauser-providedrelationaldatabaseservice)forapps,pipelines,andtaskhistoryRabbitMQforPCF(orauser-providedRabbitMQorKafkaservice)foreventmessagingRedisforPCF(orauser-providedRedisservice)forcapturinganalyticsdata

IntegrationofDataFlowserverwithPCF’sUAAsecuritymodel

ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutSpringCloudDataFlowforPCF.

Element Details

Version v1.1.4

Releasedate September7,2018

Softwarecomponentversion v1.5.1.RELEASE

CompatibleOpsManagerversion(s) v1.10.x,v1.11.x,v1.12.x,v2.0.x,andv2.1.x

CompatibleElasticRuntimeversion(s) v1.10.x,v1.11.x,v1.12.x,v2.0.x,andv2.1.x

IaaSsupport AllsupportedbyPCF

RequirementsSpringCloudDataFlowforPCFhasthefollowingrequirements:

JavaCloudFoundrybuildpack version3.8orlater(seethePrerequisitessectionofInstallationformoreinformation)

MySQLforPCFv2 ,oranalternativerelationaldatabaseservice

RabbitMQforPCF ,oranalternativeRabbitMQorKafkaservice

RedisforPCF ,oranalternativeRedisservice

CloudFoundryCommandLineInterface (cfCLI)

OptionalTousetheSpringCloudDataFlowshellinterfacewithSpringCloudDataFlowforPCFserviceinstances,installthefollowingcfCLIplugins:

SpringCloudDataFlowforPCFcfCLIplugin.Toinstalltheplugin,runthefollowingcommand:

$cfinstall-plugin-rCF-Community"spring-cloud-dataflow-for-pcf"

JavaRuntimeEnvironment (JRE).RequiredtoruntheDataFlowshell.YoucandownloadtheJREfromtheJavawebsite .

ServiceInstanceLoggingcfCLIplugin .Toinstalltheplugin,runthefollowingcommand:

Note:SpringCloudDataFlowforPCFv1.1isnolongersupported.Thesupportperiodforv1.1hasexpired.Tostayuptodatewiththelatestsoftwareandsecurityupdates,upgradetoasupportedversion.

©CopyrightPivotalSoftwareInc,2013-2019 2 1.1

http://cloud.spring.io/spring-cloud-dataflow/

http://cloud.spring.io/spring-cloud-stream/

http://cloud.spring.io/spring-cloud-task/

https://github.com/cloudfoundry/java-buildpack

https://network.pivotal.io/products/pivotal-mysql

https://network.pivotal.io/products/pivotal-rabbitmq-service

https://network.pivotal.io/products/p-redis

https://github.com/cloudfoundry/cli

https://github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin

https://java.com/en/


https://github.com/pivotal-cf/service-instance-logs-cli-plugin

$cfinstall-plugin-rCF-Community"ServiceInstanceLogging"

Pleaseprovideanybugs,featurerequests,orquestionstothePivotalCloudFoundryFeedbacklist.


mailto:[email protected]

SpringCloudDataFlowReleaseNotesReleasenotesforSpringCloudDataFlowforPCF

v1.1.4ReleaseDate:September7,2018

Fixesincludedinthisrelease:

Fixedanissuethatpreventedthe“run-tests”errandfromcompletingwhentheSpringCloudDataFlowtilewasconfiguredtonotallowinternetaccess.

v1.1.3ReleaseDate:August6,2018

Enhancementsincludedinthisrelease:

Anew“DonotallowpublicInternetaccessfromserviceinstances”optionintheDataFlowtileconfigurationpreventsserviceinstancesfromaccessingtheInternet.

The“run-tests”errandnowcomplieswiththeUAApasswordpolicywhencreatingitstemporarytestuser.


TheJavabuildpacksettingconfiguredintheDataFlowtileorforaparticularserviceinstanceisnowhonored.

v1.1.2ReleaseDate:July16,2018

Enhancementsincludedinthisrelease:

Serviceinstancesnowallow spring.cloud.dataflow.applicationProperties.* tobesetvia -c propertiesto cf createandupdatecommands

v1.1.1ReleaseDate:June25,2018


Thebrokernowproceedswithanupdateoperationforaserviceinstancethatpreviouslyfailedtoupdate.

v1.1.0ReleaseDate:June18,2018

Featuresincludedinthisrelease:

ServiceinstancesarenowbasedonSpringCloudDataFlow1.5.1.RELEASE.

ServiceinstancesnowacceptMavenconfigurationpropertiesprovidedbySpringCloudDataFlowOSS.

The“run-tests”lifecycleerrandnolongerrequiresaccesstotheInternet.


InstallingandConfiguringSpringCloud®DataFlowforPCFFollowthebelowstepstoinstallSpringCloudDataFlowforPivotalCloudFoundry(PCF).

PrerequisitesSpringCloudDataFlowforPCFisbuiltusingSpringBoot1.5,whichrequiresversion3.8orlateroftheJavaCloudFoundrybuildpack.ThedefaultJavabuildpack—thebuildpackatthelowestpositionofallJavabuildpacks—onyourPCFinstallationmustthereforebeatversion3.8orlater.

YoucanusetheCloudFoundryCommandLineInterfacetool(cfCLI)toseetheversionoftheJavabuildpackthatiscurrentlyinstalled.

$cfbuildpacksGettingbuildpacks...

buildpackpositionenabledlockedfilenamejava_buildpack_offline1truefalsejava-buildpack-offline-v3.8.1.zipruby_buildpack2truefalseruby_buildpack-cached-v1.6.19.zipnodejs_buildpack3truefalsenodejs_buildpack-cached-v1.5.15.zipgo_buildpack4truefalsego_buildpack-cached-v1.7.10.zip

IfthedefaultJavabuildpackisolderthanversion3.8,youcandownloadanewerversionfromPivotalNetwork andupdatePivotalCloudFoundrybyfollowingtheinstructionsintheManagingCustomBuildpacks topic.ToensurethatthenewerbuildpackisthedefaultJavabuildpack,youmaydeleteordisabletheolderbuildpackormakesurethatthenewerbuildpackisinalowerposition.

DependentServicesSpringCloudDataFlowreliesonotherPCFservicesforitsservicebroker,fortheDataFlowservercreatedforeachserviceinstance,andfortheSpringCloudSkipperapplicationcreatedforeachserviceinstance.Youcanconfiguretheseinthetilesettings.Whenfollowingtheinstallationstepsbelow,reviewallofthesesettings,ensuringthatyouhavepreviouslyinstalledthespecifiedservicesandthatthespecifiedserviceplansareactive.

InstallationSteps1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.

3. UndertheImportaProductbutton,click+nexttotheversionnumberofSpringCloud®DataFlowforPCF.Thisaddsthetiletoyourstagingarea.

4. ClickthenewlyaddedSpringCloudDataFlowtile.IntheSettingstab,clickAssignAZsandNetworks.

Selecttheavailabilityzonesforthetiletouse.IntheNetworksection,selectthePAS(orERT)network.

5. IntheSettingstab,clickServiceBroker.

Important:IfanyoftheservicesorserviceplansconfiguredintheDataFlowtilesettingsarenotavailableinyourPCFdeployment,thetileinstallationmayfail.


https://network.pivotal.io/products/buildpacks

http://docs.pivotal.io/pivotalcf/adminguide/buildpacks.html

https://network.pivotal.io/

EntertherelationaldatabaseservicenameandplannamefortheSpringCloudDataFlowtiletouseforstoringitsservicebroker’sserviceinstancedata.

YoucanalsoconfiguretheJavabuildpacktousefordeployingthebrokerandserviceinstanceapplications(bydefault,thetilewillusethehighest-priorityJavabuildpack).

6. IntheSettingstab,clickDataFlowServer.

ConfigurethedefaultservicesandserviceplansusedbytheSpringCloudDataFlowserverthatisdeployedforeachDataFlowserviceinstance.ThesevaluescanbeoverriddenwhencreatingaDataFlowserviceinstance.

7. StillintheSettingstab,clickSkipper.

ConfiguretherelationaldatabaseservicenameandplanusedbytheSpringCloudSkipperapplicationdeployedforeachDataFlowserviceinstance.

8. StillintheSettingstab,clickErrands.


SpringCloudDataFlowhasfourlifecycleerrands.InPCF2.2andlater,eacherrandcanbesettoalwaysrun(On)ortoneverrun(Off).InPCF2.1andearlier,errandscanalsobesettorunconditionally(WhenChanged).PivotalrecommendsthatallSpringCloudDataFlowlifecycleerrandsbesettoalwaysrun(On).

9. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoinstalltheSpringCloud®DataFlowtile.


TileConfigurationSeebelowforinformationaboutconfiguringoptionsintheSpringCloud®DataFlowforPCFtilesettings.

TileConfigurationOptionsTheSpringCloudDataFlowtileincludessettingsforvariousoptions.YoucanconfigurethesebyvisitingtheInstallationDashboardofPivotalCloudFoundry®OperationsManagerandclickingtheSpringCloudDataFlowtile.

ConfigureBrokerDatabaseServiceandPlanBydefault,theSpringCloudDataFlowproductusestheMySQLforPCFv2productanditsdefault db-small serviceplantoprovisionadatabaseserviceinstancefortheDataFlowservicebroker.IfyouareusingtheMySQLforPCFv2productanditdoesnothaveanactive db-small planorifyouwishtouseanalternativeservice,youmustconfiguretheSpringCloudDataFlowproductfortheserviceandserviceplanyouwishtouse.YoucanconfigureanalternativerelationaldatabaseserviceandserviceplanintheServiceBrokerpaneoftheSpringCloudDataFlowsettings.

Thebroker’sdatabaseserviceshouldbeconfiguredonlyduringtheSpringCloudDataFlowtileinstallationprocess.Ifyouhavealreadycompletedthetileinstallationprocess,donotalterthatsetting(youmaystillconfigurethebroker’sdatabaseserviceplanafterthetileinstallation).

Seebelowformoreinformationabouttheresultsofconfiguringtheservicebroker’srelationaldatabase.

Ifyou andchange then

installthetile

theservicenameorplan

thetileusesthespecifiedserviceandplan.Youmayalsousethedefaultvaluesof p.mysql and db-small .

updatethetile

theserviceplan thetile’srelationaldatabaseserviceinstanceischangedtousethespecifiedplan.Nodataislost.

updatethetile

theservicename

theservicenamechangeisignored.Ifyouhavealsochangedtheserviceplan,thismayleavethebrokerinaninoperativestate.Toresolvethisissue,revertyourchanges.

ConfigureBuildpackforBrokerandServiceInstancesBydefault,theSpringCloudDataFlowservicebrokerapplicationandserviceinstancebackingapplicationsusethebuildpackchosenbytheplatform’sbuildpackdetection;normally,thiswillbethehighest-priorityJavabuildpack.TocausetheseapplicationstouseaparticularJavabuildpackregardlessofpriority,youcansetthenameofthebuildpacktouseintheServiceBrokerpaneoftheSpringCloudDataFlowtilesettings.

IntheBuildpackfield,enterthenameofthedesiredJavabuildpack.ClickSave,returntotheInstallationDashboard,andapplyyourchanges.Thebrokerapplicationandserviceinstanceapplicationswillnowusetheselectedbuildpack.

ConfigureDataFlowServerDependentServicesBydefault,theSpringCloudDataFlowproductusestheMySQLforPCFv2,RabbitMQforPCF,andRedisforPCFproductstoprovisiondependentservice

Note:OpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .

warning:Configuringtheservicebroker’sdependentrelationaldatabaseserviceafterthetileinstallationhasfinishedcanresultinorphanedDataFlowserviceinstancesorinmultipledatasetsandcorruptionofthebrokerdatabase’sdata.


https://docs.pivotal.io/pivotalcf/2-0/opsguide/config-rbac.html#about

instancesfortheDataFlowserverapplicationdeployedforeachDataFlowserviceinstance.Ifyouwishtouseotherservicesortochangetheserviceplansused,youmustconfiguretheservicesandserviceplansyouwishtousefortheDataFlowserver.YoucanconfigurealternateservicesorserviceplansintheDataFlowServerpaneoftheSpringCloudDataFlowsettings.

DisableDataFlowServerInternetAccessYoucandisableinternetaccessforallSpringCloudDataFlowserviceinstances.Todothis,selectthe“DisablepublicInternetaccess”checkboxintheDataFlowServerpaneoftheSpringCloudDataFlowsettings.

ConfigureSkipperDatabaseServiceandPlanBydefault,theSpringCloudDataFlowproductusestheMySQLforPCFv2productanditsdefault db-small serviceplantoprovisionadatabaseserviceinstancefortheSpringCloudSkipperapplicationdeployedforeachDataFlowserviceinstance.IfyouareusingtheMySQLforPCFv2productanditdoesnothaveanactive db-small planorifyouwishtouseanalternativeservice,youmustconfiguretheserviceandserviceplanyouwishtouseforSkipper.YoucanconfigureanalternativerelationaldatabaseserviceandserviceplanintheSkipperpaneoftheSpringCloudDataFlowsettings.

EnableorDisableGlobalMarketplaceAccessBydefault,theSpringCloudDataFlowproductenablesaccesstoitsserviceoffering,called p-dataflow ,acrossallorgsinthePCFdeploymentaspartofthetileinstallationprocess.Ifyouwishtomanuallygrantserviceaccesstospecificorganizations,youcanconfigurethedefaultglobalaccessintheServiceAccesspaneoftheSpringCloudDataFlowsettings.


Todisablethedefaultofserviceaccessenabledforallorgs,cleartheEnableglobalaccesstoplansofservicep_dataflowcheckbox.ClickSave,returntotheInstallationDashboard,andapplyyourchanges.YoucannowenableordisableaccesstotheDataFlowserviceofferingforspecificorgs.


TheServiceBrokerandInstancesSeebelowforinformationaboutSpringCloud®DataFlow’sdeploymentmodelandotherinformationwhichmaybeusefulinadministeringDataFlowserviceinstancesordeployedapplications.

TheServiceBrokerSpringCloudDataFlowprovidesaSpringCloudDataFlowserverasaManagedService onPivotalCloudFoundry (PCF).ItusesCloudFoundry’sServiceBrokerAPI tomanagethisservice.SeebelowforinformationaboutSpringCloudDataFlow’sbrokerimplementation.

TheSpringCloudDataFlowservicebroker’sfunctionalityiscontainedinthefollowingSpringBootapplicationinstance,whichisdeployedinthe“system”organizationtothe“p-dataflow”space.

p-dataflow-[version]:ImplementstheServiceBrokerAPItoactonprovision,deprovision,bind,andunbindrequests.

ThebrokerreliesontheMySQLforPivotalCloudFoundryv2 productforthefollowingserviceinstance.

p-dataflow-db:AMySQLdatabaseusedasabackingstorefortheservicebroker.

ServiceBrokerUpgradesTheSpringCloudDataFlowproductupgradeprocesschecksbeforeredeployingtheservicebrokertoseewhethertheproduct’sversionhaschanged.Iftheversionhasnotchanged,theupgradeprocesswillcontinuewithoutredeployingtheservicebroker.

Theservicebrokerapplicationisdeployedusingablue-greendeploymentstrategy .Duringanupgradeoftheservicebroker,thebrokerwillcontinueprocessingrequeststoprovision,deprovision,bind,andunbindserviceinstances,withoutdowntime.

AccessViaAppsManagerToviewthebrokerapplicationinPivotalCloudFoundry®AppsManager,logintoAppsManagerasanadminuserandselectthe“system”org.

Note:OpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .

Note:Youcanconfigureanalternaterelationaldatabaseserviceforthebrokertouse.SeetheConfigureBrokerDatabaseServiceandPlan

sectionoftheTileConfiguration topic.


https://docs.pivotal.io/pivotalcf/2-0/opsguide/config-rbac.html#about

https://docs.cloudfoundry.org/services/

https://network.pivotal.io/products/pivotal-cf

https://docs.cloudfoundry.org/services/api.html

https://network.pivotal.io/products/pivotal-mysql

https://docs.pivotal.io/scdf/tile-configuration.html#configure-broker-database-service-and-plan

https://docs.pivotal.io/scdf/tile-configuration.html

http://martinfowler.com/bliki/BlueGreenDeployment.html

Theapplicationisdeployedinthe“p-dataflow”space.

GetBrokerBuildInformationTheSpringCloudDataFlowbrokerprovidesbuildinformationusingtheSpringBootActuator info endpoint,whichismappedto /info .Youcanaccessthisendpointbyappending /info tothepathoftheSpringCloudDataFlowbroker.

IftheSpringCloudDataFlowservicebrokerapplicationislocatedatthefollowingURL:

https://p-dataflow.apps.wise.com

thenyoucanaccessthe info endpointbyvisiting:

https://p-dataflow.apps.wise.com/info

TheservicebrokerreturnsaJSONresponse,asinthefollowingexample.

{"git":{"commit":{"time":1526673193000,"id":"740fb80"},"branch":"740fb80a6e1d83de033a4b81bf29c8b4cfa83b74"},"build":{"version":"1.1.0-build.19","artifact":"scdf-for-pcf-service-broker","name":"scdf-for-pcf-service-broker","group":"io.pivotal.springcloud.dataflow","time":1526673593000}}

Theresponsecontainsinformationaboutthebuildoftheservicebrokerapplication,includingtheMavenprojectcoordinatesandbuildtime.ItalsocontainsinformationabouttheGitrepositoryforSpringCloudDataFlowforPCFatbuildtime.

ServiceInstanceArchitecture

Note:FieldssuchasthoseforGitrepositoryinformationarefordiagnosticpurposesandintendedtoprovidePivotalSupport withinformationtohelpintroubleshooting.


https://pivotal.io/support

ForeachSpringCloudDataFlowserviceinstancecreated,theservicebrokerprovisionsthefollowingresources,allwithinthespacefromwhichtheserviceinstancewascreated(“theuserspace”)unlessnotedotherwise.

Anewspacewithinthe“p-dataflow”org,namedusingtheserviceinstanceGUIDandcontaining:

ADataFlowserverapplication.ADataFlowmetricsapplication.ASpringCloudSkipperpackagemanagementapplication.Arelationaldatabaseservice,boundtotheDataFlowserverapplication.

Arelationaldatabaseservice,boundtotheSkipperpackagemanagementapplication.Amessagingdataservice,boundtotheDataFlowmetricsapplication.

ARedisdatabaseservice,boundtotheDataFlowserverapplication.

Thefollowingresourcesarecreatedintheoriginatinguserspacewheretheserviceinstancecommandwastargetedat:

A“p-dataflow”serviceinstance.A“p-dataflow”relationaldatabaseserviceinstance(providingaccesstotherelationaldatabaseservicecreatedintheserviceinstance’sspacewithinthe“p-dataflow”org).A“p-dataflow”messagingserviceinstance(providingaccesstothemessagingdataservicecreatedintheserviceinstance’sspacewithinthe“p-dataflow”org).A“p-dataflow”analyticsserviceinstance(providingaccesstotheanalyticsservicecreatedintheserviceinstance’sspacewithinthe“p-dataflow”org).

CapacityRequirementsBelowaretheusagerequirementsoftheSpringCloudDataFlowservicebroker.

Application MemoryAllocated DiskAllocation

ServiceBroker 2GB 1GB

Theservicebrokerisboundtoarelationaldatabaseserviceinstance,whichstoresdatarelatingtothebroker’sserviceinstances.Therelationaldatabaseservicetouseisconfigurable,asdescribedinTheServiceBrokersectionabove.

BelowaretheusagerequirementsoftheDataFlowserverandmetricsapplicationsthatbackeachSpringCloudDataFlowserviceinstance.

BackingApplication MemoryAllocation/AppInstance DiskAllocation/AppInstance

DataFlowServer 2GB 2GB

DataFlowMetrics 1GB 1GB

Note:Thisrelationaldatabaseserviceisa“p.mysql”serviceinstancebydefault.Youcanconfigureanalternaterelationaldatabasewhenyoucreatetheserviceinstance.

Note:Thismessagingdataserviceisa“p-rabbitmq”serviceinstancebydefault.Youcanconfigureanalternatemessagingservicewhenyoucreatetheserviceinstance.

Note:ThisRedisdatabaseserviceisa“p-redis”serviceinstancebydefault.YoucanconfigureanalternateRedisdatabasewhenyoucreatetheserviceinstance.


SpringCloudSkipper 1GB 2GB

BackingApplication MemoryAllocation/AppInstance DiskAllocation/AppInstance

SpringCloudDataFlowserviceinstancesarealsobackedbyinstancesofotherPCFservices.TheseareeitherservicesfromPCFdataserviceproductsorcustomservicesprovidedtoaDataFlowserviceinstanceatcreatetime.Thesedataservicesincludearelationaldatabaseservice,amessagingservice,andananalyticsdatabaseserviceinstanceforeachDataFlowserviceinstancecreated.TheSkipperbackingapplicationusesaMySQLdatabaseserviceforitsbackingstore.


GettingStartedwithSpringCloud®DataFlowforPCF

SeebelowforstepstogetstartedusingSpringCloudDataFlowforPCF.TheexamplesbelowuseSpringCloudDataFlowforPCFtoquicklycreateadatapipeline.

CreatingaDataPipelineUsingtheShellCreateaSpringCloudDataFlowserviceinstance(seetheCreatinganInstancesectionoftheManagingServiceInstancestopic).IfyouusethedefaultbackingdataservicesofMySQLforPCFv2,RabbitMQforPCF,andRedisforPCF,youcanthenimporttheSpringCloudDataFlowOSS“RabbitMQ+Maven”streamappstarters .

StarttheDataFlowshellusingthe cfdataflow-shell commandaddedbytheSpringCloudDataFlowforPCFcfCLIplugin:

$cfdataflow-shelldata-flow...WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>

ImportthestreamappstartersusingtheDataFlowshell’s appimport command:

dataflow:>appimporthttps://bit.ly/Celsius-SR2-stream-applications-rabbit-mavenSuccessfullyregistered65applicationsfrom[source.sftp,source.mqtt.metadata,sink.mqtt.metadata,...processor.scriptable-transform.metadata]

Withtheappstartersimported,youcanusethree–the http source,the split processor,andthe log sink–tocreateastreamthatconsumesdataviaanHTTPPOSTrequest,processesitbysplittingitintowords,andoutputstheresultsinlogs.

CreatethestreamusingtheDataFlowshell’s streamcreate command:

dataflow:>streamcreate--namewords--definition"http|splitter--expression=payload.split('')|log"Creatednewstream'words'

Next,deploythestream,usingthe streamdeploy

command:

dataflow:>streamdeploywordsDeploymentrequesthasbeensentforstream'words'

CreatingaDataPipelineUsingtheDashboardCreateaSpringCloudDataFlowserviceinstance(seetheCreatinganInstancesectionoftheManagingServiceInstancestopic).IfyouusethedefaultbackingdataservicesofMySQLforPCFv2,RabbitMQforPCF,andRedisforPCF,youcanthenimporttheSpringCloudDataFlowOSS“RabbitMQ+Maven”streamappstarters .

InAppsManager,visittheSpringCloudDataFlowserviceinstance’spageandclickManagetoaccessitsdashboard.

Thiswilltakeyoutothedashboard’sAppstab,whereyoucanimportapplications.

ConsiderinstallingtheSpringCloudDataFlowforPCFandServiceInstanceLogscfCLIplugins.SeetheUsingtheShellandViewingServiceInstanceLogstopics.

TheexamplesinthistopicusetheSpringCloudDataFlowforPCFcfCLIplugin.


https://cloud.spring.io/spring-cloud-stream-app-starters/#stream-app-starters-and-spring-cloud-data-flow-

https://cloud.spring.io/spring-cloud-stream-app-starters/#stream-app-starters-and-spring-cloud-data-flow-

ClickBulkImportApplication(s)toimportthe“RabbitMQ+Maven”streamappstarters.IntheUrifield,enterhttps://bit.ly/Celsius-SR2-stream-applications-rabbit-maven .ThenclickImportTheApplication(s).

Withtheappstartersimported,visittheStreamstab.Youcanusethreeoftheimportedstarterapplications—the http source,the split processor,andthe log sink—tocreateastreamthatconsumesdataviaanHTTPPOSTrequest,processesitbysplittingitintowords,andoutputstheresultsinlogs.

ClickCreateAStreamtoenterthestreamcreationview.Intheleftsidebar,searchforthe http sourceapplication.Clickitanddragitontothecanvastobegindefiningastream.


Searchforandaddthe splitter processorapplicationand log sinkapplication.

Clickthe splitter application,thenclickthegeariconbesideittoedititsproperties.Intheexpressionfield,enter payload.split('') .ClickOK.

Clickanddragbetweentheoutputandinputportsontheapplicationstoconnectthemandcompletethestream.


ClicktheCreateStreambutton.Typethename“words”,thenclickCreateTheStream.

TheStreamstabnowdisplaysthenewstream.Clickthe▶buttontodeploythestream.

ClickDeployTheStream.


UsingtheDeployedDataPipelineYoucanrunthe cfapps commandtoseetheapplicationsdeployedaspartofthestream:

$cfappsGettingappsinorgmyorg/spacedevasuser...OK

namerequestedstateinstancesmemorydiskurlsRWSDZgk-words-http-v1started1/11G1GRWSDZgk-words-http-v1.apps.wise.comRWSDZgk-words-log-v1started1/11G1GRWSDZgk-words-log-v1.apps.wise.comRWSDZgk-words-splitter-v1started1/11G1GRWSDZgk-words-splitter-v1.apps.wise.com

Runthe cflogs commandonthe RWSDZgk-words-log-v1 application:

$cflogsRWSDZgk-words-log-v1RetrievinglogsforappRWSDZgk-words-log-v1inorgmyorg/spacedevasuser...

Then,inaseparatecommandline,usetheDataFlowshell(startedusingthe cfdataflow-shell command)tosendaPOSTrequesttotheRWSDZgk-words-http-v1 application:

$cfdataflow-shelldataflow...WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>httppost--targethttp://RWSDZgk-words-http-v1.apps.wise.com--data"Thisisatest">POST(text/plain)http://RWSDZgk-words-http-v1.apps.wise.comThisisatest>202ACCEPTED

Watchfortheprocesseddatainthe RWSDZgk-words-log-v1 application’slogs:

2018-06-07T16:47:08.80-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.808INFO16---[plitter.words-1]RWSDZgk-words-log-v1:This2018-06-07T16:47:08.81-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.810INFO16---[plitter.words-1]RWSDZgk-words-log-v1:is2018-06-07T16:47:08.82-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.820INFO16---[plitter.words-1]RWSDZgk-words-log-v1:a2018-06-07T16:47:08.82-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.822INFO16---[plitter.words-1]RWSDZgk-words-log-v1:test


ManagingServiceInstances

SeebelowforinformationaboutmanagingDataFlowserviceinstancesusingtheCloudFoundryCommandLineInterfacetool(cfCLI).YoucanalsomanageDataFlowserviceinstancesusingPivotalCloudFoundry®AppsManager.

CreatinganInstance

Beginbytargetingthecorrectorgandspace.

$cftarget-omyorg-sdevelopmentapiendpoint:https://api.system.wise.comapiversion:2.75.0user:userorg:myorgspace:development

YoucanviewplandetailsfortheDataFlowproductusing cfmarketplace-s

.

$cfmarketplaceGettingservicesfrommarketplaceinorgmyorg/spacedevelopmentasuser...OK

serviceplansdescriptionp-dataflowstandardDeploysSpringCloudDataFlowserverstoorchestratedatapipelinesp-dataflow-mysqlproxyProxiestotheSpringCloudDataFlowMySQLserviceinstancep-dataflow-rabbitmqproxyProxiestotheSpringCloudDataFlowRabbitMQserviceinstancep-dataflow-redisproxyProxiestotheSpringCloudDataFlowRedisserviceinstance

TIP:Use'cfmarketplace-sSERVICE'toviewdescriptionsofindividualplansofagivenservice.

$cfmarketplace-sp-dataflowGettingserviceplaninformationforservicep-dataflowasuser...OK

serviceplandescriptionfreeorpaidstandardStandardPlanfree

SettingtheBuildpackEachDataFlowserviceinstancecanbegiventhenameofabuildpacktousefordeployingstreamandtaskapps.Youcansetthebuildpackfortheserviceinstanceusinga buildpack parametergivento cfcreate-

service.Tocreateaserviceinstancethatusesabuildpacknamed custom-java-buildpack todeploy

apps,youmightrun:

$cfcreate-servicep-dataflowstandarddata-flow-c'{"buildpack":"custom-java-buildpack"}'

SettingDependentServicesEachDataFlowserviceinstanceusesthreedependentdataservices.Defaultsfortheseservicescanbeconfiguredinthetilesettings,andthesedefaultscanbeoverriddenforeachindividualserviceinstanceatcreatetime.

GeneralparametersusedtoconfiguredependentdataservicesforaDataFlowserviceinstancearelistedbelow.

Parameter Function

relational-data-service.name

ThenameoftheservicetouseforarelationaldatabasethatstoresSpringCloudDataFlowmetadataandtaskhistory.

relational-data-service.plan

Thenameoftheserviceplantousefortherelationaldatabaseservice.

messaging-data-service.name ThenameoftheservicetouseforaRabbitMQorKafkaserverthatfacilitateseventmessaging.

messaging-data-service.plan ThenameoftheserviceplantousefortheRabbitMQorKafkaservice.

analytics-data-service.name ThenameoftheservicetouseforaRedisserverthatstoresanalytics.

analytics-data-service.plan ThenameoftheserviceplantousefortheRedisserver.

skipper-relational.name ThenameoftheservicetouseforarelationaldatabaseusedbytheSkipperapplication.

skipper-relational.plan ThenameoftheserviceplantouseforarelationaldatabaseusedbytheSkipperapplication.

Important:IfyouareusingtheRedisforPCF productfortheSpringCloudDataFlowanalyticsstore,youcannotcreatemoreDataFlowserviceinstancesthanthesettingoftheRedisproduct’sServiceInstanceLimit(thedefaultServiceInstanceLimitforRedisforPCFis5).SeetheShared-VMPlan sectionoftheInstallingandUpgradingRedisforPCF topicintheRedisforPCFdocumentation forinformationaboutconfiguringthislimit.

Note:Theserviceofferingswiththeplan proxy areproxyservicesusedbySpringCloudDataFlowforPCFserviceinstances.TheSpringCloudDataFlowservicebrokercreatesanddeletesinstancesoftheseservicesautomaticallyalongwitheachSpringCloudDataFlowserviceinstance.Donotmanuallycreateordeleteinstancesoftheseservices.


https://network.pivotal.io/products/p-redis/

http://docs.pivotal.io/redis/installing.html#shared-vm-config

http://docs.pivotal.io/redis/installing.html

http://docs.pivotal.io/redis/

SettingMavenPropertiesEachDataFlowserviceinstancecanoptionallyspecifyMavenconfigurationproperties.Forthecompletelistofpropertiesthatcanbespecified,seethe“Maven”sectionintheOSSSpringCloudDataFlowdocumentation .

MavenconfigurationpropertiescanbesetforeachDataFlowserviceinstanceusingparametersgivento cfcreate-service

.Tosetthe

maven.remote-repositories.repo1.url property,youmightuseacommandsuchasthefollowing:

$cfcreate-servicep-dataflowstandarddata-flow-c'{"maven.remote-repositories.repo1.url":"https://repo.spring.io/libs-snapshot"}'

CreatingtheInstanceCreatetheserviceinstanceusing cfcreate-

service.TocreateaDataFlowservicethatusesaRedisCloudserviceavailablefromyourPCFmarketplaceand

setstheMaven maven.remote-repositories.repo1.url propertyto https://repo.spring.io/release ,youmightrun:

$cfcreate-servicep-dataflowstandarddata-flow-c'{"analytics-data-service":{"name":"rediscloud","plan":"30mb"},"maven.remote-repositories.repo1.url":"https://repo.spring.io/libs-snapshot"}'Creatingserviceinstancedata-flowinorgmyorg/spacedevelopmentasuser...OK

Createinprogress.Use'cfservices'or'cfservicedata-flow'tocheckoperationstatus.

Asthecommandoutputsuggests,youcanusethe cfservices or cfservice commandstocheckthestatusoftheserviceinstance.Whentheserviceinstanceisready,the cfservice commandwillgiveastatusof create

succeeded:

$cfservicedata-flow

Serviceinstance:data-flowService:p-dataflowBoundapps:Tags:Plan:standardDescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesDocumentationurl:http://cloud.spring.io/spring-cloud-dataflow/Dashboard:https://p-dataflow.apps.wise.com/instances/f09e5c77-e526-4f49-86d6-721c6b8e2fd9/dashboard

LastOperationStatus:createsucceededMessage:CreatedStarted:2017-07-20T18:24:14ZUpdated:2017-07-20T18:26:17Z

UpgradinganInstanceAfteranupgradeoftheSpringCloudDataFlowforPCFproduct,youcanusethe cfupdate-service commandtoupgradeindividualDataFlowserviceinstances.



Youcanviewallserviceinstancesinthespaceusing cfservices .

$cfservicesGettingservicesinorgmyorg/spacedevelopmentasuser...OK

nameserviceplanboundappslastoperationdata-flowp-dataflowstandardcreatesucceededmysql-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-mysqlproxycreatesucceededrabbitmq-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-rabbitmqproxycreatesucceededredis-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-redisproxycreatesucceeded

UpgradetheDataFlowserviceinstanceusing cfupdate-service ,passingthe -c flagtosetthe upgrade parameterto true .

$cfupdate-servicedata-flow-c'{"upgrade":true}'Updatingserviceinstancedata-flowasuser...OK

Updateinprogress.Use'cfservices'or'cfservicedata-flow'tocheckoperationstatus.

Astheoutputfromthe cfupdate-service commandsuggests,youcanusethe cfservices or cfservice commandstocheckthestatusoftheserviceinstance.WhentheDataFlowserviceinstancehasbeenupgraded,the cfservice commandwillgiveastatusof update

succeeded:


https://docs.spring.io/spring-cloud-dataflow/docs/1.5.0.RELEASE/reference/htmlsingle/#configuration-maven

$cfservicedata-flowShowinginfoofservicedata-flowinorgmyorg/spacedevasuser...

name:data-flowservice:p-dataflowboundapps:tags:plan:standarddescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesdocumentation:dashboard:https://p-dataflow.apps.wise.com/instances/1cf8ff5b-4a65-469d-bee7-36e6541ac241/dashboard

Showingstatusoflastoperationfromservicedata-flow...

status:updatesucceededmessage:Updatedstarted:2018-06-19T19:26:09Zupdated:2018-06-19T19:29:17Z

DeletinganInstanceDeletingaDataFlowserviceinstancewillresultindeletionofallofitsdependentserviceinstances.



Youcanviewallserviceinstancesinthespaceusing cfservices .


nameserviceplanboundappslastoperationdata-flowp-dataflowstandardcreatesucceededmysql-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-mysqlproxycreatesucceededrabbitmq-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-rabbitmqproxycreatesucceededredis-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-redisproxycreatesucceeded

DeletetheDataFlowserviceinstanceusing cfdelete-service

.Whenprompted,enter y toconfirmthedeletion.

$cfdelete-servicedata-flow

Reallydeletetheservicedata-flow?>yDeletingservicedata-flowinorgmyorg/spacedevelopmentasuser...OK

Deleteinprogress.Use'cfservices'or'cfservicedata-flow'tocheckoperationstatus.

ThedependentserviceinstancesfortheDataFlowserverserviceinstancearedeletedfirst,andthentheDataFlowserverserviceinstanceitselfisdeleted.

Astheoutputfromthe cfdelete-service

commandsuggests,youcanusethe cfservices or cfservice commandstocheckthestatusoftheserviceinstance.

WhentheDataFlowserviceinstanceanditsdependentserviceinstanceshavebeendeleted,the cfservices commandwillnolongerlisttheserviceinstance:


Noservicesfound


UsingUser-ProvidedServiceInstances

Bydefault,aSpringCloudDataFlowserviceinstanceusesMySQLforPCFforitsbackingrelationaldatabaseservice,RabbitMQforPCFforitsbackingmessagingservice,andRedisforPCFforitsbackinganalyticsservice.WhencreatingaSpringCloudDataFlowserviceinstance,youcanconfigureittouseCloudFoundryuser-providedservices foritsdependentdataservicesinstead.SeebelowforinformationaboutconfiguringaDataFlowserviceinstancetouseanalternativedependentdataservice.

DependentServiceParametersYoucanspecifythataDataFlowserviceinstanceshoulduseauser-providedserviceinstancebypassingconfigurationparameterstothe cfcreate-

service

commandinJSONusingthe -c flag.TheparametersusedtoconfigurealternativedependentdataservicesforaDataFlowserviceinstancearelistedbelow.

Parameter Function

relational-data-service.user-provided

AJSONobjectcontainingconnectioninformationfortherelationaldatabaseserviceusedtostoretaskhistory.

messaging-data-service.user-provided

AJSONobjectcontainingconnectioninformationforthemessagingserviceusedforeventmessaging.

analytics-data-service.user-provided

AJSONobjectcontainingconnectioninformationfortheRedisserviceusedforananalyticsstore.

Seethefollowingsectionsforinformationaboutusingtheseparameters.

User-ProvidedRelationalDatabaseServiceYoucanusethe relational-data-service parametertosupplytheconfigurationforauser-providedrelationaldatabaseservice.ThisparametercontainsaJSONobject user-provided ,withfieldsforeachoftheconnectionvaluesneededtoprovisionaCloudFoundryuser-providedrelationaldatabaseserviceinstance.

TherelationaldatabaseservicecanbethedefaultMySQLforPCFv2serviceoranyotherrelationaldatabaseservice.Itmustprovidearelationaldatabase.

Anexampleoffieldscontainedinthe user-provided objectforarelationaldatabaseserviceisshownbelow.

{"uri":"mysql://kempercrabb:[email protected]:3306/vigil","username":"kempercrabb","password":"fake-password","dbname":"vigil","host":"123.234.456.879","port":3306,"tags":["mysql"]}

TocreateaSpringCloudDataFlowserviceinstanceusingtheseconnectionvaluesfortheserviceinstance’srelationaldatabaseserviceinstance:

$cfcreate-servicep-dataflowstandarddata-flow-c'{"relational-data-service":{"user-provided":{"uri":"mysql://kempercrabb:[email protected]:3306/vigil","username":"kempercrabb","password":"fake-password","dbname":"vigil","host":"123.234.456.879","port":3306,"tags":["mysql"]}}}'

User-ProvidedMessagingServiceYoucanusethe messaging-data-service parametertosupplytheconfigurationforauser-providedmessagingservice.ThisparametercontainsaJSONobject

user-provided ,withfieldsforeachoftheconnectionvaluesneededtoprovisionaCloudFoundryuser-providedmessagingserviceinstance.

ThemessagingdataservicecanbethedefaultRabbitMQforPCFserviceoranothermessagingservice.ItmustprovideeitheraRabbitMQserveroraKafkaserver.

Anexampleoffieldscontainedinthe user-provided objectforamessagingserviceisshownbelow.

{"dashboard_url":"https://api.cloudamqp.com/console/a-GUID-would-go-here/details","username":"buckstorm","vhost":"buckstorm","password":"fake-password","ssl":false,"hostname":"stonehill.rmq.cloudamqp.com","uri":"amqp://buckstorm:[email protected]/buckstorm","http_api_uri":"http://buckstorm:[email protected]:1883/api","tags":["rabbitmq"]}

TocreateaSpringCloudDataFlowserviceinstanceusingtheseconnectionvaluesfortheserviceinstance’smessagingserviceinstance:

$cfcreate-servicep-dataflowstandarddata-flow-c'{"messaging-data-service":{"user-provided":{"dashboard_url":"https://api.cloudamqp.com/console/a-GUID-would-go-here/details","username":"buckstorm","vhost":"buckstorm","password":"fake-password","ssl":false,"hostname":"stonehill.rmq.cloudamqp.com","uri":"amqp://buckstorm:[email protected]/buckstorm","http_api_uri":"http://buckstorm:[email protected]:1883/api","tags":["rabbitmq"]}}}'


https://docs.pivotal.io/pivotalcf/devguide/services/user-provided.html

User-ProvidedAnalyticsServiceYoucanusethe analytics-data-service parametertosupplytheconfigurationforauser-providedanalyticsservice.ThisparametercontainsaJSONobject

user-provided ,withfieldsforeachoftheconnectionvaluesneededtoprovisionaCloudFoundryuser-providedRedisserviceinstance.

TheanalyticsservicecanbethedefaultRedisforPCFserviceoranyotherRedisservice.ItmustprovideaRedisdatabase.

Anexampleoffieldscontainedinthe user-provided objectforananalyticsserviceisshownbelow.

{"host":"garrels.gce.cloud.redislabs.com","uri":"redis://:[email protected]:11781/boars-head","port":11781,"dbname":"boars-head","password":"fake-password","tags":["redis"]}

TocreateaSpringCloudDataFlowserviceinstanceusingtheseconnectionvaluesfortheserviceinstance’sanalyticsserviceinstance:

$cfcreate-servicep-dataflowstandarddata-flow-c'{"analytics-data-service":{"user-provided":{"host":"garrels.gce.cloud.redislabs.com","uri":"redis://:[email protected]:11781/boars-head","port":11781,"dbname":"boars-head","password":"fake-password","tags":["redis"]}}}'


ViewingServiceInstanceLogs

SpringCloudDataFlowforPCFprovidesaccesstothelogsgeneratedbyeachDataFlowserverserviceinstance,includinglogsforeachofthethreebackingapplications(DataFlowserverapplication,metricsapplication,andSkipperapplication)foreachinstance.YoucanviewtheselogseitherusingtheServiceInstanceLogscfCLIpluginorbyvisitingthedashboardoftheSpringCloudDataFlowservicebroker.

UsingthecfCLIPluginAfterinstallingtheServiceInstanceLogscfCLIplugin (seetheinstructionsintheInstalling sectionoftheplugin’sREADME ),youcanusethe

service-logs commandtotaillogsordumprecentlogsforaserviceinstance.

TotaillogsforaDataFlowserviceinstance,run cfservice-logsSERVICE_NAME ,where SERVICE_NAME isthenameoftheserviceinstance:

$cfservice-logsdata-flow

Todumprecentlogsfortheinstance,usethe --recent flag:

$cfservice-logs--recentdata-flow

IfyourPivotalCloudFoundrydeploymentusesaself-signedcertificate,youmustusethe --skip-ssl-validation flagtodisablethedefaultvalidationoftheplatform’sSSLcertificate:

$cfservice-logs--skip-ssl-validationdata-flow

UsingtheServiceBrokerDashboard

VisittheSpringCloudDataFlowservicebroker’sdashboard.YoucanaccessitatthefollowingURL,whereapps.wise.com istheapplicationdomainofyourPCFdeployment:

https://p-dataflow.apps.wise.com/

Thedashboardshowsthename,org,andspaceofeachserviceinstance,aswellasalinktoviewlogsfortheinstance.

ClicktheLogslinktoviewlogsforaparticularserviceinstance’sbackingapplication.

Youcanstreamcurrentlogsfortheinstancebyclickingthe▶button.

Note:Toaccesstheservicebrokerdashboard,youmustbeaSpaceDeveloperinthebrokerapplication’sspace(thisistypicallythe system organd p-dataflow space).


https://github.com/pivotal-cf/service-instance-logs-cli-plugin/

https://github.com/pivotal-cf/service-instance-logs-cli-plugin/blob/master/README.md#installing

https://github.com/pivotal-cf/service-instance-logs-cli-plugin/blob/master/README.md

ReadingAggregatedLogsThelogsretrievedbytheServiceInstanceLogscfCLIpluginaggregatelogsfromthreebackingapplications:aSpringCloudDataFlowserverapplication,ametricsapplication,andaSpringCloudSkipperapplication.Thefollowingexcerptshowslogsafterdeployingastream:

2018-02-09T11:12:02.45-0600[RTR/dataflow0]OUTdataflow-11f71dd3-f902-4e31-b631-fbbbfdb82459.apps.americancanyon.cf-app.com-[2018-02-09T17:12:02.429+0000]"GET/metrics/streams?names=httptestHTTP/1.1"200014765"https://dataflow-11f71dd3-f902-4e31-b631-fbbbfdb82459.apps.americancanyon.cf-app.com/dashboard/""Mozilla/5.0(Macintosh;IntelMacOSX10_13_2)AppleWebKit/537.36(KHTML,likeGecko)Chrome/63.0.3239.132Safari/537.36""130.211.0.191:63089""10.0.4.8:61018"x_forwarded_for:"71.10.174.240,35.201.75.216,130.211.0.191"x_forwarded_proto:"https"vcap_request_id:"adc3ee08-a97a-41ae-71d4-5fc2dff4a82c"response_time:0.025969938app_id:"7e61bf00-2a8c-4cc9-833a-9dd92442ff47"app_index:"0"x_b3_traceid:"6862348a02fa27db"x_b3_spanid:"6862348a02fa27db"x_b3_parentspanid:"-"

2018-02-09T11:12:02.45-0600[RTR/metrics0]OUTdf-metrics-11f71dd3-f902-4e31-b631-fbbbfdb82459.apps.americancanyon.cf-app.com-[2018-02-09T17:12:02.444+0000]"GET/collector/metrics/streamsHTTP/1.1"200022261"-""Apache-HttpClient/4.5.3(Java/1.8.0_144)""130.211.3.158:53467""10.0.4.35:61004"x_forwarded_for:"35.184.235.101,35.201.75.216,130.211.3.158"x_forwarded_proto:"https"vcap_request_id:"87c87336-863e-424d-6ebe-3fa85f6545e0"response_time:0.007074433app_id:"369ae87c-b9a6-4801-a4c5-61b779ff4f71"app_index:"0"x_b3_traceid:"60c018a7f484a97e"x_b3_spanid:"60c018a7f484a97e"x_b3_parentspanid:"-"

2018-02-09T11:12:02.60-0600[APP/PROC/WEB/skipper0]OUT2018-02-0917:12:02.603INFO15---[ry-client-nio-3]o.s.c.d.s.c.CloudFoundryAppDeployer:Successfullycomputedstatus[deployed]forhttptest-http-v12018-02-09T11:12:02.94-0600[APP/PROC/WEB/skipper0]OUT2018-02-0917:12:02.940INFO15---[ry-client-nio-3]o.s.c.d.s.c.CloudFoundryAppDeployer:Successfullycomputedstatus[deployed]forastream-log-v42018-02-09T11:12:03.06-0600[APP/PROC/WEB/skipper0]OUT2018-02-0917:12:03.068INFO15---[ry-client-nio-1]o.s.c.d.s.c.CloudFoundryAppDeployer:Successfullycomputedstatus[deployed]forastream-time-v4

TheDataFlowserverapplication’slogsareidentifiedasbelongingtothe dataflow application.Themetricsapplicationsimilarlyiscalled metrics .TheSpringCloudSkipperapplicationiscalled skipper .


UsingtheDashboard

Tofindthedashboard,navigateinPivotalCloudFoundry®AppsManagertotheDataFlowserviceinstance’sspace,clickthelistingfortheserviceinstance,andthenclickManage.

Ifyouareusingversion6.8.0orlateroftheCloudFoundryCommandLineInterfacetool(cfCLI),youcanalsouse cfserviceSERVICE_NAME ,whereSERVICE_NAME isthenameoftheDataFlowserviceinstance:

$cfservicedata-flowShowinginfoofservicedata-flowinorgmyorg/spacedevasuser...

name:data-flowservice:p-dataflowboundapps:tags:plan:standarddescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesdocumentation:dashboard:https://p-dataflow.apps.wise.com/instances/2c9a66d3-99d8-458c-860d-01e7ac64eb1d/dashboard


status:createsucceededmessage:Createdstarted:2018-05-17T19:02:04Zupdated:2018-05-17T19:08:14Z

VisittheURLgivenfor“Dashboard”.

DashboardInformationThedashboardprovidesanoverviewofregisteredapplications,streamandtaskdefinitions,andbatchjobs.Italsoprovidescontrolsfordeployingstreams,launchingtasks,andrestartingbatchjobexecutions.

Forcompleteinformationaboutthedashboardanditsprovidedfunctionality,seetheOSSSpringCloudDataFlowproject’sdocumentation .


https://docs.spring.io/spring-cloud-dataflow/docs/1.5.0.RELEASE/reference/htmlsingle/#dashboard

UsingtheShell

Theopen-sourceSpringCloudDataFlow projectprovidesashell ,whichcanbeusedtointeractlocallywithaDataFlowserviceinstancedeployedonPivotalCloudFoundry(PCF).

YoucanusetheshellwithSpringCloudDataFlowforPCFserviceinstancesineitheroftwoways:

TheSpringCloudDataFlowforPCFCloudFoundryCLIplugin(recommended)

Theopen-sourceshellbinary,withmanually-configuredcommand-lineoptions

TheSpringCloudDataFlowforPCFCloudFoundryCLIpluginwascreatedtoeasetheuseoftheSpringCloudDataFlowshellwithSpringCloudDataFlowserviceinstancesonPCF.

UsingtheCloudFoundryCLIPluginThepluginaddsa dataflow-shell command,whichattachestheDataFlowshelltoaSpringCloudDataFlowforPCFserviceinstance.Givenanexistingserviceinstancenamed data-flow ,thefollowingcommandwilldownloadtheappropriateshellversionandattachittotheserviceinstance:

$cfdataflow-shelldata-flowAttachingshelltodataflowservicedata-flowinorgmyorg/spacedevasuser...LaunchingdataflowshellJAR___________/___|______(_)______/___||_______||\___\|'_\|'__||'_\/_`|||||/_\||||/_`|___)||_)|||||||(_||||___||(_)||_||(_|||____/|.__/|_||_|_||_|\__,|\____|_|\___/\__,_|\__,_|____|_|___|___/__________|_\___||____|___||_______\\\\\\||||/_`|__/_`|||_||/_\\/\//\\\\\\||_||(_||||(_|||_|||(_)\VV///////|____/\__,_|\__\__,_||_||_|\___/\_/\_//_/_/_/_/_/

1.5.0.RELEASE

WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>

FormoreinformationabouttheSpringCloudDataFlowshell,seeitsdocumentation .

UsingtheShellManuallyIfyouwouldliketodownloadandconfiguretheDataFlowshellmanually,downloadtheshellJARfilefromtheSpringReleasesMavenrepository.SpringCloudDataFlowforPCFcanbeusedwithversion1.5.0.RELEASEorlateroftheSpringCloudDataFlowshell.

TotargetaDataFlowserviceinstance’sserverwiththeshell,youmustobtaintheserver’sURL.Run cfserviceSERVICE_NAME ,where SERVICE_NAME isthenameoftheserviceinstance:

$cfservicedata-flow

Serviceinstance:data-flowShowinginfoofservicedata-flowinorgmyorg/spacedevasuser...

name:data-flowservice:p-dataflowboundapps:tags:plan:standarddescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesdocumentation:dashboard:https://p-dataflow.apps.wise.com/instances/2f6ec0c6-c828-45bb-905a-4779ce50552b/dashboard


status:createsucceededmessage:Createdstarted:2018-05-17T14:43:38Zupdated:2018-05-17T14:49:43Z

VisittheURLgivenfor“Dashboard”andauthenticateusingyourPCFcredentials.Whenredirectedtotheserviceinstance’sdashboard,copythedashboard’sdomainname(theURLminusthepathfollowingthedomain).

Forexample,giventhefollowingURL:

https://dataflow-2f6ec0c6-c828-45bb-905a-4779ce50552b.apps.wise.com/dashboard/#/apps

Copythefollowing:

Note:ToruntheSpringCloudDataFlowshell,youmusthaveaJavaRuntimeEnvironment(JRE)installed.YoucandownloadtheJREfromtheJavawebsite .

Note:BeforeconnectingtoaDataFlowserviceinstancewiththeSpringCloudDataFlowshell,besuretologintothePCFdeploymentusingtheCloudFoundryCommandLineInterfacetool(cfCLI)andtargettheorgandspaceoftheDataFlowserviceinstance.TheDataFlowshellusesthecfCLItoauthenticatetoPCF.


https://cloud.spring.io/spring-cloud-dataflow/

https://docs.spring.io/spring-cloud-dataflow/docs/1.5.0.RELEASE/reference/htmlsingle/#shell

https://github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin


https://docs.spring.io/spring-cloud-dataflow/docs/1.5.0.RELEASE/reference/htmlsingle/#shell

http://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-shell/1.5.0.RELEASE/spring-cloud-dataflow-shell-1.5.0.RELEASE.jar

https://dataflow-2f6ec0c6-c828-45bb-905a-4779ce50552b.apps.wise.com

FromthedirectorycontainingtheshellJARfile,runtheshellfromthecommandlineusingacommandasshownbelow.

$java-jarJAR_NAMESKIP_VALIDATION--dataflow.uri=SERVER_URL--dataflow.credentials-provider-command="cfoauth-token"--dataflow.mode=skipper

Inthiscommand,replacethefollowingplaceholdersasshownbelow.

JAR_NAME withthenameofthedownloadedSpringCloudDataFlowshellJARfile

SKIP_VALIDATION withtheflagandvalue --dataflow.skip-ssl-validation=true ifyourPCFinstallationisusingself-signedSSLcertificates;otherwiseremove

SERVER_URL withtheDataFlowserviceinstance’sserverURL(thedomaincopiedabove)

Thecompletecommandmaylooksomethinglikethefollowing:

$java-jarspring-cloud-dataflow-shell-1.5.0.RELEASE.jar\--dataflow.skip-ssl-validation=true\--dataflow.uri=https://dataflow-2f6ec0c6-c828-45bb-905a-4779ce50552b.apps.wise.com\--dataflow.credentials-provider-command="cfoauth-token"--dataflow.mode=skipper

Whenyourunthe java-jar

command,theshellwillinitializeandgivea dataflow:> prompt.

WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>

Foralistofavailableshellcommands,run help fromwithintheshell.Forinformationabouttheflagswhichcanbeprovidedtotheshellonstartup,seetheOSSSpringCloudDataFlowproject’sdocumentation .


https://docs.spring.io/spring-cloud-dataflow/docs/1.5.0.RELEASE/reference/htmlsingle/#shell-options

spring cloud data flow for pivotal cloud foundry · spring cloud® data flow for pcf about spring...

Documents