pivotal container service (pks) documentation › pdfs › pks-1-0.pdfnsx-t 2.1 advanced edition...
TRANSCRIPT
-
PivotalContainerService
(PKS)
Version1.0
Published:16Oct2018
©2018PivotalSoftware,Inc.AllRightsReserved.
-
2471718212224252729303234404456575864667881848591102103105107109110111112114115116117120121122124125126127128129130
TableofContents
TableofContentsPivotalContainerService(PKS)PKSReleaseNotesPKSConceptsPKSClusterManagementPKSAPIAuthenticationLoadBalancersinPKSPKSPrerequisitesInstallingthePKSCLIInstallingtheKubernetesCLIPreparingtoInstallPKSonvSpherevSpherePrerequisitesandResourceRequirementsFirewallPortsandProtocolsRequirementsforvSpherewithNSX-TPreparingtoDeployPKSonvSphereDeployingOpsManagertovSphereConfiguringOpsManageronvSpherePreparingtoInstallPKSonGCPGCPPrerequisitesandResourceRequirementsPreparingtoDeployPKSonGCPDeployingOpsManagertoGCPConfiguringOpsManageronGCPConfiguringaGCPLoadBalancerforthePKSAPIConfiguringaGCPLoadBalancerforPKSClustersInstallingPKSInstallingandConfiguringPKSInstallingandConfiguringPKSwithNSX-TIntegrationUpgradingPKSWhatHappensDuringPKSUpgradesUpgradePKSMaintainWorkloadUptimeConfiguretheUpgradePipelineManagingPKSConfigurePKSAPIAccessManageUsersinUAAManagePKSDeploymentswithBOSHAddCustomWorkloadsDownloadClusterLogsServiceInterruptionsDeletePKSUsingPKSCreateaClusterRetrieveClusterCredentialsandConfigurationViewClusterListViewClusterDetailsViewClusterPlansUsingDynamicPersistentVolumesScaleExistingClustersAccessDashboard
©CopyrightPivotalSoftwareInc,2013-2019 2 1.0
-
131133134135136137138139140143148
DeployandAccessBasicWorkloadsDeleteaClusterLogOutofthePKSEnvironmentUsingHelmwithPKSConfigureTillerInstallConcourseUsingHelmDiagnosingandTroubleshootingPKSDiagnosticToolsTroubleshootingPKSCLIPKSSecurityDisclosureandReleaseProcess
©CopyrightPivotalSoftwareInc,2013-2019 3 1.0
-
PivotalContainerService(PKS)Pagelastupdated:
PivotalContainerService(PKS)enablesoperatorstoprovision,operate,andmanageenterprise-gradeKubernetesclustersusingBOSHandPivotalOpsManager.
OverviewPKSusestheOn-DemandBroker todeployCloudFoundryContainerRuntime ,aBOSHreleasethatoffersauniformwaytoinstantiate,deploy,andmanagehighlyavailableKubernetesclustersonacloudplatformusingBOSH.
AfteroperatorsinstallthePKStileontheOpsManagerInstallationDashboard,developerscanprovisionKubernetesclustersusingthePKSCommandLineInterface(PKSCLI),andruncontainer-basedworkloadsontheclusterswiththeKubernetesCLI,kubectl.
PKSisavailableaspartofPivotalCloudFoundry orasastand-aloneproduct.
WhatPKSAddstoKubernetesThefollowingtabledetailsthefeaturesthatPKSaddstotheKubernetesplatform.
Feature IncludedinK8s IncludedinPKS
Singletenantingress ✓ ✓
Securemulti-tenantingress ✓
Statefulsetsofpods ✓ ✓
Multi-containerpods ✓ ✓
Rollingupgradestopods ✓ ✓
Rollingupgradestoclusterinfrastructure ✓
Podscalingandhighavailability ✓ ✓
Clusterprovisioningandscaling ✓
MonitoringandrecoveryofclusterVMsandprocesses ✓
Persistentdisks ✓ ✓
Securecontainerregistry ✓
Embedded,hardenedoperatingsystem ✓
FeaturesPKShasthefollowingfeatures:
KubernetesCompatibility:ConstantcompatibilitywithcurrentstablereleaseofKubernetes
Production-ready:Highlyavailablefromapplicationstoinfrastructure,withnosinglepointsoffailure
BOSHadvantages:Built-inhealthchecks,scaling,auto-healingandrollingupgrades
Fullyautomatedoperations:Fullyautomateddeploy,scale,patch,andupgradeexperience
Multi-cloud:Consistentoperationalexperienceacrossmultipleclouds
GCPAPIsaccess:TheGoogleCloudPlatform(GCP)ServiceBrokergivesapplicationsaccesstotheGoogleCloudAPIs,andGoogleContainerEngine(GKE)consistencyenablesthetransferofworkloadsfromortoGCP
OnvSphere,PKSsupportsdeployingandrunningKubernetesclustersinair-gappedenvironments.
PKSComponents
©CopyrightPivotalSoftwareInc,2013-2019 4 1.0
https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://docs-kubo.cfapps.io/https://docs.pivotal.io
-
ThePKScontrolplanecontainsthefollowingcomponents:
AnOn-DemandBroker thatdeploysCloudFoundryContainerRuntime (CFCR),anopen-sourceprojectthatprovidesasolutionfordeployingandmanagingKubernetes clustersusingBOSH .
AServiceAdapter
ThePKSAPI
FormoreinformationaboutthePKScontrolplane,seePKSClusterManagement.
ForadetailedlistofcomponentsandsupportedversionsbyaparticularPKSrelease,seethePKSReleaseNotes.
PKSConceptsForconceptualinformationaboutPKS,seePKSConcepts.
PKSPrerequisitesForinformationabouttherequirementsforinstallingPKS,seePKSPrerequisites.
PreparingtoInstallPKSToinstallPKS,youmustdeployOpsManagerv2.0orv2.1.YouuseOpsManagertoinstallandconfigurePKS.
IfyouareinstallingPKStovSphere,youcanalsoconfigureintegrationwithNSX-TandHarbor.
Consultthefollowingtableforcompatibilityinformation:
IaaS OpsManagerv2.0 NSX-T Harbor
vSphere Required Available Available
GCP Required NotAvailable NotAvailable
ForinformationaboutpreparingyourenvironmentbeforeinstallingPKS,seethetopicthatcorrespondstoyourcloudprovider:
PreparingtoInstallPKSonvSphere
PreparingtoInstallPKSonGCP
InstallingPKSForinformationaboutinstallingPKS,seeInstallingandConfiguringPKS.
UpgradingPKSForinformationaboutupgradingthePKStileandPKS-deployedKubernetesclusters,seeUpgradingPKS.
ManagingPKSForinformationaboutconfiguringauthentication,creatingusers,andmanagingyourPKSdeployment,seeManagingPKS.
UsingPKSForinformationaboutusingthePKSCLItocreateandmanageKubernetesclusters,seeUsingPKS.
©CopyrightPivotalSoftwareInc,2013-2019 5 1.0
https://docs.pivotal.io/svc-sdk/odb/https://docs-kubo.cfapps.iohttps://kubernetes.io/docs/home/https://bosh.io/docs
-
DiagnosingandTroubleshootingPKSForinformationaboutdiagnosingandtroubleshootingissuesinstallingorusingPKS,seeDiagnosingandTroubleshootingPKS.
©CopyrightPivotalSoftwareInc,2013-2019 6 1.0
mailto:[email protected]
-
PKSReleaseNotesPKS(PivotalContainerService)isusedtocreateandmanageon-demandKubernetesclustersviathePKSCLI.
v1.0.4ReleaseDate:May21,2018
UpgradeProcedure
ToupgradetoPKSv1.0.4,followtheproceduresinUpgradePKS.
FeaturesUpdatesKubernetestov1.9.7.
ComponentVersionsPKSv1.0.4includesorsupportsthefollowingcomponentversions:
ProductComponent VersionSupported Notes
PivotalCloudFoundryOperationsManager(OpsManager)
2.0.Xand2.1.X SeparatedownloadavailablefromPivotalNetwork
vSphere
6.5,6.5U1,and6.5U2-EditionsvSphereEnterprisePlusEdition
vSpherewithOperationsManagementEnterprisePlus
vSphereversionssupportedforPivotalContainerService(PKS)
VMwareHarborRegistry 1.4.2 SeparatedownloadavailablefromPivotalNetwork
NSX-T 2.1AdvancedEdition AvailablefromVMware
Stemcell 3468.XFloatingstemcelllineavailabletodownloadfromPivotalNetwork
Kubernetes 1.9.7* PackagedinthePKSTile(CFCR)
CFCR(Kubo) 0.13 PackagedinthePKSTile
Golang 1.9.5 PackagedinthePKSTile
NCP 2.1.3 PackagedinthePKSTile
KubernetesCLI 1.9.7*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
PKSCLI 1.0.3-build.15SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
UAA 55
*Componentsmarkedwithanasteriskhavebeenpatchedtoresolvesecurityvulnerabilitiesorfixcomponentbehavior.
KnownIssuesThissectionincludesknownissueswithPKSv1.0.4andcorrespondingworkarounds.
Note:UpgradetoPKSv1.0.4fromeitherPKSv1.0.2orPKSv1.0.3.DonotupgradePKSv1.0.0directlytov1.0.4.Instead,upgradetov1.0.2,thenv1.0.4.Alternatively,doauniqueinstallofPKSv1.0.4.
©CopyrightPivotalSoftwareInc,2013-2019 7 1.0
-
AccesstotheKubernetesAPIisUnavailableDuringUpgrades
PKSupgradesincludeupgradestothemasternode.Whilethemasternodeisundergoinganupgrade,theKubernetesAPIisunavailable.
IfyouattempttoaccesstheAPIduringanupgrade,youwillnotbeabletoconnect.
StemcellUpdatesCauseAutomaticVMUpgrading
EnablingtheUpgradeallclusterserrandallowsautomaticupgradingforVMsinyourdeployment.PivotalrecommendsenablingthiserrandtoensurethatalldeployedclusterVMsarepatched.
WhenyouenabletheUpgradeallclusterserrand,thefollowingactionscancausedowntime:
UpdatingthePKStilewithanewstemcelltriggersupdatingeachVMineachcluster.
UpdatingothertilesinyourdeploymentwithnewstemcellscausestheupgradingofthePKStile.
UpgradeErrandFailswithFailedDeployments
TheUpgradeallclusterserrandfailsifanydeploymentsareinafailedstate.
Toworkaroundthisissue,deletethefailedclusterusingthePKSCLIorredeploythefailedclusterwiththeBOSHCLItoensuretheclusterisinasuccessfulstate.
PodsLoseNetworkConnectivityAfterVMColdMigration
WhenaKubernetesclusterworkerVMgoesthroughcoldmigrationinvSphere,newlyprovisionedpodslosenetworkconnectivity.
Thisissuecanoccurunderthefollowingconditions:
WhentheVMispoweredoffandissubjecttocoldmigration,andtheVMmovestoadifferentESXihost
WhentheVMispoweringonandissubjecttoDistributedResourceScheduler(DRS)beforethepowerupcompletes
WhenthevNICoftheVMisdetachedandreattached
Toworkaroundthisissue,deletetheworkerVM.BOSHrecreatestheworkerVMandrestoresnetworkconnectivity.
KubernetesClusterCreationFailsifNSX-TManagerPasswordBeginswithCertainSpecialCharacters
IfyouselectNSX-TasaContainerNetworkTypeinPKSandyourNSX-TManagerpasswordbeginswithan @ , $ , ^ , ' ,orspacecharacter,Kubernetesclustercreationfails.Toresolvethisissue,resetyourNSX-TManagerpasswordsothatitdoesnotbeginwithanyofthesecharacters.AfterresettingyourNSX-TManagerpassword,reconfigureyourNSX-TManagercredentialsinthePKStilewiththeupdatedpassword.
v1.0.3ReleaseDate:May4,2018
UpgradeProcedure
ToupgradetoPKSv1.0.3,performthefollowingsteps:
1. Downloadthelatest3468.xstemcellfromPivotalNetwork andconfigurethePKStilewiththestemcell.
2. Createanewworkernodeserviceaccount.
TocreatetheserviceaccountonGCP,seeCreatetheWorkerNodeServiceAccount.
Note:TheonlysupportedupgradepathforPKSv1.0.3isfromPKSv1.0.2.DonotupgradePKSv1.0.0directlytov1.0.3.Instead,upgradetov1.0.2,thenv1.0.3.Alternatively,doauniqueinstallofPKSv1.0.3.
©CopyrightPivotalSoftwareInc,2013-2019 8 1.0
https://network.pivotal.io/products/stemcells
-
TocreatetheserviceaccountonvSphere,seeCreatetheWorkerNodeServiceAccount.
3. FollowtheproceduresinUpgradePKS.WhenconfiguringtheKubernetesCloudProviderconfigurationscreeninthePKStile,configurethenewworkernodecredentialsorserviceaccountkeyasappropriateforyourIaaS.
FeaturesSeparatesthemasterandworkernodecredentials.
UpdatesKubernetestov1.9.6.
UpdatesGolangtov1.9.5.
ComponentVersionsPKSv1.0.3includesorsupportsthefollowingcomponentversions:
ProductComponent VersionSupported Notes
PivotalCloudFoundryOperationsManager(OpsManager)
2.0.Xand2.1.X SeparatedownloadavailablefromPivotalNetwork
vSphere
6.5and6.5U1-EditionsvSphereEnterprisePlusEdition
vSpherewithOperationsManagementEnterprisePlus
vSphereversionssupportedforPivotalContainerService(PKS)
VMwareHarborRegistry 1.4.1 SeparatedownloadavailablefromPivotalNetwork
NSX-T 2.1AdvancedEdition AvailablefromVMware
Stemcell 3468.XFloatingstemcelllineavailabletodownloadfromPivotalNetwork
Kubernetes 1.9.6* PackagedinthePKSTile(CFCR)
CFCR(Kubo) 0.13 PackagedinthePKSTile
Golang 1.9.5* PackagedinthePKSTile
NCP 2.1.3* PackagedinthePKSTile
KubernetesCLI 1.9.6*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
PKSCLI 1.0.3-build.15*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
UAA 55*
*Componentsmarkedwithanasteriskhavebeenpatchedtoresolvesecurityvulnerabilitiesorfixcomponentbehavior.
KnownIssuesThissectionincludesknownissueswithPKSv1.0.3andcorrespondingworkarounds.
AccesstotheKubernetesAPIisUnavailableDuringUpgrades
PKSupgradesincludeupgradestothemasternode.Whilethemasternodeisundergoinganupgrade,theKubernetesAPIisunavailable.
IfyouattempttoaccesstheAPIduringanupgrade,youwillnotbeabletoconnect.
StemcellUpdatesCauseAutomaticVMUpgrading
EnablingtheUpgradeallclusterserrandallowsautomaticupgradingforVMsinyourdeployment.PivotalrecommendsenablingthiserrandtoensurethatalldeployedclusterVMsarepatched.
©CopyrightPivotalSoftwareInc,2013-2019 9 1.0
-
WhenyouenabletheUpgradeallclusterserrand,thefollowingactionscancausedowntime:
UpdatingthePKStilewithanewstemcelltriggersupdatingeachVMineachcluster.
UpdatingothertilesinyourdeploymentwithnewstemcellscausestheupgradingofthePKStile.
UpgradeErrandFailswithFailedDeployments
TheUpgradeallclusterserrandfailsifanydeploymentsareinafailedstate.
Toworkaroundthisissue,deletethefailedclusterusingthePKSCLIorredeploythefailedclusterwiththeBOSHCLItoensuretheclusterisinasuccessfulstate.
PodsLoseNetworkConnectivityAfterVMColdMigration
WhenaKubernetesclusterworkerVMgoesthroughcoldmigrationinvSphere,newlyprovisionedpodslosenetworkconnectivity.
Thisissuecanoccurunderthefollowingconditions:
WhentheVMispoweredoffandissubjecttocoldmigration,andtheVMmovestoadifferentESXihost
WhentheVMispoweringonandissubjecttoDistributedResourceScheduler(DRS)beforethepowerupcompletes
WhenthevNICoftheVMisdetachedandreattached
Toworkaroundthisissue,deletetheworkerVM.BOSHrecreatestheworkerVMandrestoresnetworkconnectivity.
StatefulSetsPodFailureAfterRecreatingaVM
WhenusingvSpherewithNSX-Tintegration,ifyourecreateanodethathostsaStatefulSetspod,thepodcangetstuckina ContainerCreating state.Thepodemitsawarningeventwitha FailedCreatePodSandBox reason.ThisissueaffectsStatefulSetspodscreatedbeforePKSv1.0.3.
AfixforthisbugisincludedinPKSv1.0.3,butthefixappliesonlytoStatefulSetscreatedusingPKSv1.0.2orlater.AfterupgradingPKStov1.0.3,manuallydeletingandrecreatingallpreexistingStatefulSetspodsisrecommended,eveniftheyareinarunningstate.
TogetallStatefulSetspods,runthefollowingcommandoneveryKubernetesclusterusingtheKubernetesadminuserpermissions:
$ kubectl get pods -l "statefulset.kubernetes.io/pod-name" \-o wide --all-namespaces
Foreachresult,deletethepodbyrunningthefollowingcommand:
$ kubectl delete pod POD-NAME -n POD-NAMESPACE
Youdonotneedtomanuallyrecreatethedeletedpods.KubernetesdetectsaStatefulSetwithmissingpodsandautomaticallyrecreatesthepods.
[KubernetesBug]UpgradingaClusterAffectsPersistentWorkloadUptime
Duringanupgradetov1.0.3onvSphere,persistentstoragevolumesdonotreattachtopodsuntilallworkernodeshavebeenupgraded,whichresultsinworkloaddowntimeuntiltheentireclusterisupgraded.
Thisissueoccurswhenyoudeployapodwithpersistentstorageattached,drainthenode,andthenimmediatelydeletethenodeVM.
TheexpectedbehaviorisforpersistentdiskstoreattachtotheupgradedVMsafterthepodisrestored.However,aKubernetesbugpreventsthediskfromreattaching.PKSv1.0.3worksaroundthisbugbyattachingthevolumesafterallworkersareupgraded.
Formoreinformation,seetheKubernetesissueonGitHub .
Inrarecases,podswithpersistentvolumescanstayin ContainerCreating state.Ifyouseetheerror FailedMountUnabletomountvolumesforpodPOD-NAME
,
performthefollowingsteps:
1. Findtheproblemnodebyrunning kubectl describe pod POD-NAME .
©CopyrightPivotalSoftwareInc,2013-2019 10 1.0
https://github.com/kubernetes/kubernetes/issues/61707
-
2. Preventschedulingonthenodethatrunsthepodbyrunning kubectl cordon NODE-NAME .
3. Deletepodbyrunning kubectl delete pod POD-NAME .
4. Waitforpodtoberescheduledandenter Running state.Thismaytakeseveralminutes.
5. Resumeschedulingonthenodethatrunsthepodbyrunning kubectl uncordon NODE-NAME .
KubernetesClusterCreationFailsifNSX-TManagerPasswordBeginswithCertainSpecialCharacters
IfyouselectNSX-TasaContainerNetworkTypeinPKSandyourNSX-TManagerpasswordbeginswithan @ , $ , ^ , ' ,orspacecharacter,Kubernetesclustercreationfails.Toresolvethisissue,resetyourNSX-TManagerpasswordsothatitdoesnotbeginwithanyofthesecharacters.AfterresettingyourNSX-TManagerpassword,reconfigureyourNSX-TManagercredentialsinthePKStilewiththeupdatedpassword.
v1.0.2ReleaseDate:April12,2018
UpgradeProcedureToupgradetoPKSv1.0.2,performthefollowingsteps:
1. Downloadthedocker_ctl script.
2. Downloadthedocker_ctl_update.sh script.
3. LogintotheBOSHDirectorbyrunning bosh -e MY-ENVIRONMENT log-in fromaVMthatcanaccessyourPKSdeployment.ReplaceMY-ENVIRONMENT withtheBOSHaliasforyourPKSenvironment.SeeManagePKSDeploymentswithBOSHformoreinformation.
IfyouchoosetologinfromtheOpsManagerVM,performthefollowingsteps:
a. Run sudo apt-get update .b. Run sudo apt-get install jq .
4. Run export BOSH_ENVIRONMENT=MY-ENVIRONMENT .Replace MY-ENVIRONMENT withtheBOSHaliasforyourPKSenvironment.
5. Runthe docker_ctl_update.sh script.ThisscriptcontainsthefixtocorrectlyunmountDockeroverlays.Seethecorrespondingknownissueformoreinformation.
6. Downloadthelatest3468.xstemcellfromPivotalNetwork andconfigurethePKStilewiththestemcell.
7. FollowtheproceduresinUpgradePKS.
FeaturesUpdatesKubernetestov1.9.5.
UpdatesGolangtov1.9.4.
FixedIssues
GeneralWorkernodesarenowdrainedbeforetheystopinordertominimizeworkloaddowntimeduringarollingupgrade.
UAAcredentialsandvCenterpasswordsnolongerappearinBOSHlogs.
BOSHDNSnolongercausesworkernodestofailafteramanualrestart.
TheKubernetesControllerManagercertificatenolongercontainsadditionalwhitespace.
©CopyrightPivotalSoftwareInc,2013-2019 11 1.0
http://localhost:9292/runtimes/pks/1-0/1.0.2/stemcell/docker_ctlhttp://localhost:9292/runtimes/pks/1-0/1.0.2/stemcell/docker_ctl_update.shhttps://network.pivotal.io/products/stemcells
-
Drainusernowhasadditionalpermissionstoremovereplicationcontroller-ownedpods.
UnmountingDockeroverlayvolumesnolongercausesBOSHunmountfailures.
Addressesupgradeissuesinconstrainedenvironments.
vSpherevSphereNSX-TintegrationnowworkswithBOSHstemcellv3468.25andlater.
ForvSpherewithNSX-T,thepodlogicalswitchport(LSP)isnowupdatedwhenyourecreatetheVMthathoststhepod.SeeStatefulSets intheKubernetesdocumentationandtheknownissuebelowformoreinformation.
Addedsupportforspecialcharacters # , & , ; , " , ' , ^ , \ ,space( ), % ,and ! invCenterpasswordsintheKubernetesCloudProvidertileconfigurationpage.
DrainscriptnowdeletesnodestofixavSphereissuewherenodenameschangedbetween1.9.2and1.9.5.
ComponentVersionsPKSv1.0.2includesorsupportsthefollowingcomponentversions:
ProductComponent VersionSupported Notes
PivotalCloudFoundryOperationsManager(OpsManager)
2.0.Xand2.1.X SeparatedownloadavailablefromPivotalNetwork
vSphere
6.5and6.5U1-EditionsvSphereEnterprisePlusEdition
vSpherewithOperationsManagementEnterprisePlus
vSphereversionssupportedforPivotalContainerService(PKS)
VMwareHarborRegistry 1.4.1 SeparatedownloadavailablefromPivotalNetwork
NSX-T 2.1AdvancedEdition AvailablefromVMware
Stemcell 3468.X*FloatingstemcelllineavailabletodownloadfromPivotalNetwork
Kubernetes 1.9.5* PackagedinthePKSTile(CFCR)
CFCR(Kubo) 0.13 PackagedinthePKSTile
Golang 1.9.4* PackagedinthePKSTile
NCP 2.1.2* PackagedinthePKSTile
KubernetesCLI 1.9.5*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
PKSCLI 1.0.2-build.4*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
*Componentsmarkedwithanasteriskhavebeenpatchedtoresolvesecurityvulnerabilitiesorfixcomponentbehavior.
KnownIssuesThissectionincludesknownissueswithPKSv1.0.2andcorrespondingworkarounds.
AccesstotheKubernetesAPIisUnavailableDuringUpgrades
PKSupgradesincludeupgradestothemasternode.Whilethemasternodeisundergoinganupgrade,theKubernetesAPIisunavailable.
IfyouattempttoaccesstheAPIduringanupgrade,youwillnotbeabletoconnect.
VolumeUnmountFailureAfterStemcellUpgrade
DuringanupgradetoPKSv1.0.2,BOSHcanfailtounmountthe /var/vcap/store volumeonworkernodes.ThisisduetoanissuewiththeDockerBOSH
©CopyrightPivotalSoftwareInc,2013-2019 12 1.0
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
-
releaseinstalledbythePKSv1.0.0tile.
InthisversionoftheBOSHrelease,Dockeroccasionallyfailstounmountalloverlayswhenstoppinganode.WhenyouupgradethestemcellforthePKStile,BOSHrecreatesVMsandcanfailtocorrectlyunmountDockeroverlays.
Toavoidthisissue,followthestepsintheUpgradeProceduresectionwhenyouupgradethePKStile.The docker_ctl_update.sh scriptcorrectlyunmountsDockeroverlaysbyreplacingthe docker_ctl scriptonallworkernodesthathaveDockerdeployed.
StemcellUpdatesCauseAutomaticVMUpgrading
EnablingtheUpgradeallclusterserrandallowsautomaticupgradingforVMsinyourdeployment.PivotalrecommendsenablingthiserrandtoensurethatalldeployedclusterVMsarepatched.
WhenyouenabletheUpgradeallclusterserrand,thefollowingactionscancausedowntime:
UpdatingthePKStilewithanewstemcelltriggersupdatingeachVMineachcluster.
UpdatingothertilesinyourdeploymentwithnewstemcellscausestheupgradingofthePKStile.
UpgradeErrandFailswithFailedDeployments
TheUpgradeallclusterserrandfailsifanydeploymentsareinafailedstate.
Toworkaroundthisissue,deletethefailedclusterusingthePKSCLIorredeploythefailedclusterwiththeBOSHCLItoensuretheclusterisinasuccessfulstate.
PodsLoseNetworkConnectivityAfterVMColdMigration
WhenaKubernetesclusterworkerVMgoesthroughcoldmigrationinvSphere,newlyprovisionedpodslosenetworkconnectivity.
Thisissuecanoccurunderthefollowingconditions:
WhentheVMispoweredoffandissubjecttocoldmigration,andtheVMmovestoadifferentESXihost
WhentheVMispoweringonandissubjecttoDistributedResourceScheduler(DRS)beforethepowerupcompletes
WhenthevNICoftheVMisdetachedandreattached
Toworkaroundthisissue,deletetheworkerVM.BOSHrecreatestheworkerVMandrestoresnetworkconnectivity.
StatefulSetsPodFailureAfterRecreatingaVM
WhenusingvSpherewithNSX-Tintegration,ifyourecreateanodethathostsaStatefulSetspod,thepodcangetstuckina ContainerCreating state.Thepodemitsawarningeventwitha FailedCreatePodSandBox reason.ThisissueaffectsStatefulSetspodscreatedbeforePKSv1.0.2.
AfixforthisbugisincludedinPKSv1.0.2,butthefixappliesonlytoStatefulSetscreatedusingPKSv1.0.2orlater.AfterupgradingPKStov1.0.2,manuallydeletingandrecreatingallpreexistingStatefulSetspodsisrecommended,eveniftheyareinarunningstate.
TogetallStatefulSetspods,runthefollowingcommandoneveryKubernetesclusterusingtheKubernetesadminuserpermissions:
$ kubectl get pods -l "statefulset.kubernetes.io/pod-name" \-o wide --all-namespaces
Foreachresult,deletethepodbyrunningthefollowingcommand:
$ kubectl delete pod POD-NAME -n POD-NAMESPACE
Youdonotneedtomanuallyrecreatethedeletedpods.KubernetesdetectsaStatefulSetwithmissingpodsandautomaticallyrecreatesthepods.
[KubernetesBug]UpgradingaClusterAffectsPersistentWorkloadUptime
Duringanupgradetov1.0.2onvSphere,persistentstoragevolumesdonotreattachtopodsuntilallworkernodeshavebeenupgraded,whichresultsin
©CopyrightPivotalSoftwareInc,2013-2019 13 1.0
-
workloaddowntimeuntiltheentireclusterisupgraded.
Thisissueoccurswhenyoudeployapodwithpersistentstorageattached,drainthenode,andthenimmediatelydeletethenodeVM.
TheexpectedbehaviorisforpersistentdiskstoreattachtotheupgradedVMsafterthepodisrestored.However,aKubernetesbugpreventsthediskfromreattaching.PKSv1.0.2worksaroundthisbugbyattachingthevolumesafterallworkersareupgraded.
Formoreinformation,seetheKubernetesissueonGitHub .
Inrarecases,podswithpersistentvolumescanstayin ContainerCreating state.Ifyouseetheerror FailedMountUnabletomountvolumesforpodPOD-NAME
,
performthefollowingsteps:
1. Findtheproblemnodebyrunning kubectl describe pod POD-NAME .
2. Preventschedulingonthenodethatrunsthepodbyrunning kubectl cordon NODE-NAME .
3. Deletepodbyrunning kubectl delete pod POD-NAME .
4. Waitforpodtoberescheduledandenter Running state.Thismaytakeseveralminutes.
5. Resumeschedulingonthenodethatrunsthepodbyrunning kubectl uncordon NODE-NAME .
KubernetesClusterCreationFailsifNSX-TManagerPasswordBeginswithCertainSpecialCharacters
IfyouselectNSX-TasaContainerNetworkTypeinPKSandyourNSX-TManagerpasswordbeginswithan @ , $ , ^ , ' ,orspacecharacter,Kubernetesclustercreationfails.Toresolvethisissue,resetyourNSX-TManagerpasswordsothatitdoesnotbeginwithanyofthesecharacters.AfterresettingyourNSX-TManagerpassword,reconfigureyourNSX-TManagercredentialsinthePKStilewiththeupdatedpassword.
v1.0.0ReleaseDate:February8,2018
FeaturesCreate,resize,delete,list,andshowclustersthroughthePKSCLI
NativesupportforNSX-TandFlannel
Easilyobtainkubeconfigstouseeachcluster
UsekubectltoviewtheKubernetesdashboard
Defineplansthatpre-configureVMsize,authentication,defaultnumberofworkers,andaddonswhencreatingKubernetesclusters
User/AdminconfigurationsforaccesstoPKSAPI
Centralizedloggingthroughsyslog
ComponentVersionsPKSv1.0.0includesorsupportsthefollowingcomponentversions:
ProductComponent VersionSupported Notes
PivotalCloudFoundryOperationsManager(OpsManager)
2.0.0-2.0.5 SeparatedownloadavailablefromPivotalNetwork
vSphere
6.5and6.5U1-EditionsvSphereEnterprisePlusEdition
vSpherewithOperationsManagementEnterprisePlus
vSphereversionssupportedforPivotalContainerService(PKS)
VMwareHarborRegistry 1.4.1 SeparatedownloadavailablefromPivotalNetwork
NSX-T 2.1AdvancedEdition AvailablefromVMware
©CopyrightPivotalSoftwareInc,2013-2019 14 1.0
https://github.com/kubernetes/kubernetes/issues/61707
-
Stemcell 3468.21 SeparatedownloadavailablefromPivotalNetwork
Kubernetes 1.9.2 PackagedinthePKSTile(CFCR)
CFCR(Kubo) 0.13 PackagedinthePKSTile
NCP 2.1.0.1 PackagedinthePKSTile
KubernetesCLI 1.9.2SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
PKSCLI 1.0.0-build.3SeparatedownloadavailablefromthePKSsectionofPivotalNetwork
KnownIssuesThissectionincludesknownissueswithPKSv1.0.0andcorrespondingworkarounds.
AccesstotheKubernetesAPIisUnavailableDuringUpgrades
PKSupgradesincludeupgradestothemasternode.Whilethemasternodeisundergoinganupgrade,theKubernetesAPIisunavailable.
IfyouattempttoaccesstheAPIduringanupgrade,youwillnotbeabletoconnect.
SpecialCharacters
InPKSv1.0.0,specialcharacters,suchas # , & , ; , " , ' , ^ , \ ,space( ), ! ,and % cannotbeusedinvCenterpasswords.Toresolvethisissue,resetyourpasswordsothatitdoesnotincludeanyofthespecialcharacterslistedabove.AfterresettingyourpasswordinvCenter,reconfigureyourcredentialsinthePKStilewiththeupdatedpassword.PKSv1.0.2addssupportforthespecialcharacterslistedabove.
StemcellIncompatibilitywithNSX-T
WhendeployingPKSv1.0.0usingNSX-Tasthenetworkinglayerwithastemcellotherthan3468.21,Kubernetesclusterdeploymentsfail.PKSv1.0.2addssupportforstemcellsv3468.25andlater.
StemcellUpdatesCauseAutomaticVMUpgrading
EnablingtheUpgradeallclusterserrandallowsautomaticupgradingforVMsinyourdeployment.PivotalrecommendsenablingthiserrandtoensurethatalldeployedclusterVMsarepatched.
WhenyouenabletheUpgradeallclusterserrand,thefollowingactionscancausedowntime:
UpdatingthePKStilewithanewstemcelltriggerstherollingofeachVMineachcluster.
UpdatingothertilesinyourdeploymentwithnewstemcellscausestherollingofthePKStile.
UpgradeErrandFailswithFailedDeployments
TheUpgradeallclusterserrandfailsifanydeploymentsareinafailedstate.
Toworkaroundthisissue,deletethefailedclusterusingthePKSCLIorredeploythefailedclusterwiththeBOSHCLItoensuretheclusterisinasuccessfulstate.
SyslogSecurityRecommendations
BOSHDirectorlogscontainsensitiveinformationthatshouldbeconsideredprivileged.Forexample,theselogsmaycontaincloudprovidercredentialsinPKSv1.0.0.Ifyouchoosetoforwardlogstoanexternalsyslogendpoint,usingTLSencryptionisstronglyrecommendedtopreventinformationfrombeinginterceptedbyathirdparty.
©CopyrightPivotalSoftwareInc,2013-2019 15 1.0
-
©CopyrightPivotalSoftwareInc,2013-2019 16 1.0
mailto:[email protected]
-
PKSConceptsPagelastupdated:
ThistopicdescribesPivotalContainerService(PKS)concepts.Seethefollowingsections:
PKSClusterManagement
PKSAPIAuthentication
LoadBalancersinPKS
©CopyrightPivotalSoftwareInc,2013-2019 17 1.0
mailto:[email protected]
-
PKSClusterManagementThistopicdescribeshowPivotalContainerService(PKS)managesthedeploymentofKubernetesclusters.
OverviewUsersinteractwithPKSandPKS-deployedKubernetesclustersintwoways:
DeployingKubernetesclusterswithBOSHandmanagingtheirlifecycle.ThesetasksareperformedusingthePKScommandlineinterface(CLI)andthePKScontrolplane.
Deployingandmanagingcontainer-basedworkloadsonKubernetesclusters.ThesetasksareperformedusingtheKubernetesCLI, kubectl .
ClusterLifecycleManagementThePKScontrolplaneenablesuserstodeployandmanageKubernetesclusters.
ForcommunicatingwiththePKScontrolplane,PKSprovidesacommandlineinterface,thePKSCLI.SeeInstallingthePKSCLIforinstallationinstructions.
PKSControlPlaneOverviewThePKScontrolplanemanagesthelifecycleofKubernetesclustersdeployedusingPKS.ThecontrolplaneallowsuserstodothefollowingthroughthePKSCLI:
Viewclusterplans
Createclusters
Viewinformationaboutclusters
Obtaincredentialstodeployworkloadstoclusters
Scaleclusters
Deleteclusters
Inaddition,thePKScontrolplanecanupgradeallexistingclustersusingtheUpgradeallclustersBOSHerrand.Formoreinformation,seeUpgradeKubernetesClustersinUpgradePKS.
PKSControlPlaneArchitectureThePKScontrolplaneisdeployedonasingleVMthatincludesthefollowingcomponents:
ThePKSAPIserver
ThePKSBroker
AUserAccountandAuthentication(UAA)server
Formoreinformationabouthowthesecomponentsinteract,seethefollowingdiagram:
©CopyrightPivotalSoftwareInc,2013-2019 18 1.0
-
UAA
WhenauserlogsintoorlogsoutofthePKSAPIthroughthePKSCLI,thePKSCLIcommunicateswithUAAtoauthenticatethem.ThePKSAPIpermitsonlyauthenticateduserstomanageKubernetesclusters.Formoreinformationaboutauthenticating,seePKSAPIAuthentication.
UAAmustbeconfiguredwiththeappropriateusersanduserpermissions.Formoreinformation,seeManageUsersinUAA.
PKSAPI
ThroughthePKSCLI,usersinstructthePKSAPIservertodeploy,scaleup,anddeleteKubernetesclustersaswellasshowclusterdetailsandplans.ThePKSAPIcanalsowriteKubernetesclustercredentialstoalocalkubeconfigfile,whichenablesuserstoconnecttoaclusterthrough kubectl .
ThePKSAPIsendsallclustermanagementrequests,exceptread-onlyrequests,tothePKSBroker.
PKSBroker
WhenthePKSAPIreceivesarequesttomodifyaKubernetescluster,itinstructsthePKSBrokertomaketherequestedchange.
ThePKSBrokerconsistsofanOn-DemandServiceBroker andaServiceAdapter.ThePKSBrokergeneratesaBOSHmanifestandinstructstheBOSHDirectortodeployordeletetheKubernetescluster.
ClusterWorkloadManagementPKSusersmanagetheircontainer-basedworkloadsonKubernetesclustersthrough kubectl .
Formoreinformationabout kubectl ,seeOverviewofkubectl intheKubernetesdocumentation.
©CopyrightPivotalSoftwareInc,2013-2019 19 1.0
https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://kubernetes.io/docs/reference/kubectl/overview/
-
©CopyrightPivotalSoftwareInc,2013-2019 20 1.0
mailto:[email protected]
-
PKSAPIAuthenticationPagelastupdated:
ThistopicdescribeshowthePivotalContainerService(PKS)APIworkswithUserAccountandAuthentication(UAA)tomanageauthenticationandauthorizationinyourPKSdeployment.
AuthenticatingPKSAPIRequestsBeforeuserscanloginandusethePKSCLI,youmustconfigurePKSAPIaccesswithUAA.YouusetheUAACommandLineInterface(UAAC)totargettheUAAserverandrequestanaccesstokenfortheUAAadminuser.Ifyourrequestissuccessful,theUAAserverreturnstheaccesstoken.TheUAAadminaccesstokenauthorizesyoutomakerequeststothePKSAPIusingthePKSCLIandgrantclusteraccesstoneworexistingusers.
WhenauserwithclusteraccesslogsintothePKSCLI,theCLIrequestsanaccesstokenfortheuserfromtheUAAserver.Iftherequestissuccessful,theUAAserverreturnsanaccesstokentothePKSCLI.WhentheuserrunsPKSCLIcommands,forexample, pksclusters ,theCLIsendstherequesttothePKSAPIserverandincludestheuser’sUAAtoken.
ThePKSAPIsendsarequesttotheUAAservertovalidatetheuser’stoken.IftheUAAserverconfirmsthatthetokenisvalid,thePKSAPIusestheclusterinformationfromthePKSbrokertorespondtotherequest.Forexample,iftheuserruns pksclusters ,theCLIreturnsalistoftheclustersthattheuserisauthorizedtomanage.
RoutingtothePKSAPIControlPlaneVMThePKSAPIserverandtheUAAserverusedifferentportnumbersonthecontrolplaneVM.Forexample,ifyourPKSAPIdomainis api.pks.example.com ,youcanreachyourPKSAPIandUAAserversatthefollowingURLs:
Server URL
PKSAPI api.pks.example.com:9021
UAA api.pks.example.com:8443
RefertoOpsManager>PivotalContainerService>UAA>UAAURLforyourPKSAPIdomain.
WhenyouinstallthePKStile,youconfigurealoadbalancerforthePKSAPI.ThisloadbalancerallowsyoutorunPKSCLIcommandsfromyourlocalworkstation.Formoreinformation,seetheConfigureExternalLoadBalancersectionofInstallingandConfiguringPKS.
©CopyrightPivotalSoftwareInc,2013-2019 21 1.0
mailto:[email protected]
-
LoadBalancersinPKSPagelastupdated:
ThistopicdescribesthetypesofloadbalancersthatareusedinPivotalContainerService(PKS).
Youcanconfigureloadbalancersforthefollowing:
PKSAPI:ConfiguringthisloadbalancerallowsyoutorunPKSCommandLineInterface(CLI)commandsfromyourlocalworkstation.
KubernetesClusters:ConfiguringaloadbalancerforeachnewclusterallowsyoutorunKubernetesCLI(kubectl)commandsonthecluster.
Workloads:Configuringaloadbalancerforyourapplicationworkloadsallowsexternalaccesstotheservicesthatrunonyourcluster.
ThefollowingdiagramshowswhereeachoftheaboveloadbalancerscanbeusedwithinyourPKSdeployment:
IfyouuseeithervSpherewithNSX-TorGCP,youcancreateloadbalancerswithinyourcloudproviderconsole.
Ifyourcloudproviderdoesnotofferloadbalancing,youcanuseanyexternalTCPorHTTPSloadbalancerofyourchoice.
AboutthePKSAPILoadBalancerTheloadbalancerforthePKSAPIallowsyoutoaccessthePKSAPIfromoutsidethenetwork.Forexample,configuringaloadbalancerforthePKSAPIallowsyoutorunPKSCLIcommandsfromyourlocalworkstation.
ForinformationaboutconfiguringthePKSAPIloadbalancer,seetheConfigureExternalLoadBalancersectionofInstallingandConfiguringPKS.
©CopyrightPivotalSoftwareInc,2013-2019 22 1.0
-
AboutKubernetesClusterLoadBalancersWhenyoucreateacluster,youmustconfigureexternalaccesstotheclusterbycreatinganexternalTCPorHTTPSloadbalancer.TheloadbalancerallowstheKubernetesCLItocommunicatewiththecluster.
Ifyoucreateaclusterinanon-productionenvironment,youcanchoosenottousealoadbalancer.Toallowkubectltoaccesstheclusterwithoutaloadbalancer,youcandooneofthefollowing:
CreateaDNSentrythatpointstothecluster’smasterVM.Forexample:
my-cluster.example.com A 10.0.0.5
Ontheworkstationwhereyourunkubectlcommands,addthemasterIPaddressofyourclusterand kubo.internal tothe /etc/hosts file.Forexample:
10.0.0.5 kubo.internal
Forinformationaboutconfiguringaclusterloadbalancer,seeCreateaCluster.
AboutWorkloadLoadBalancersToallowexternalaccesstoyourapp,youcaneithercreatealoadbalancerorexposeastaticportonyourworkload.
Forinformationaboutconfiguringaloadbalancerforyourappworkload,seeDeployandAccessBasicWorkloads.
©CopyrightPivotalSoftwareInc,2013-2019 23 1.0
mailto:[email protected]
-
PKSPrerequisitesPagelastupdated:
ThistopicdescribestheprerequisitesforinstallingPivotalContainerService(PKS)onvSphereorGoogleCloudPlatform(GCP).
GeneralPKSPrerequisitesPKSrequiresthePKSCommandLineInterface(PKSCLI)andtheKubernetesCLI(kubectl).SeethefollowingtopicsforinformationaboutinstallingeachCLI:
InstallingthePKSCLI
InstallingtheKubernetesCLI
ResourceRequirementsForinformationabouttheresourcerequirementsforinstallingPKS,seethetopicthatcorrespondstoyourcloudprovider:
vSpherePrerequisitesandResourceRequirements
GCPPrerequisitesandResourceRequirements
©CopyrightPivotalSoftwareInc,2013-2019 24 1.0
mailto:[email protected]
-
InstallingthePKSCLIPagelastupdated:
ThistopicdescribeshowtoinstallthePivotalContainerServiceCommandLineInterface(PKSCLI).
ToinstallthePKSCLI,followtheproceduresforyouroperatingsystemtodownloadthePKSCLIfromPivotalNetwork .Binariesareonlyprovidedfor64-bitarchitectures.
MacOSX1. NavigatetoPivotalNetwork andlogin.
2. ClickPivotalContainerService(PKS).
3. ClickPKSCLI.
4. ClickPKSCLI-MactodownloadtheMacOSXbinary.
5. Renamethedownloadedbinaryto pks .
6. Onthecommandline,runthefollowingcommandtomakethePKSbinaryexecutable:
$chmod+xpks
7. Movethebinaryintoyour PATH .
Forexample:
$mvpks/usr/local/bin/pks
Linux1. NavigatetoPivotalNetwork andlogin.
2. ClickPivotalContainerService(PKS).
3. ClickPKSCLI.
4. ClickPKSCLI-LinuxtodownloadtheLinuxbinary.
5. Renamethedownloadedbinaryto pks .
6. Onthecommandline,runthefollowingcommandtomakethePKSbinaryexecutable:
$chmod+xpks
7. Movethebinaryintoyour PATH .
Forexample:
$mvpks/usr/local/bin/pks
Windows1. NavigatetoPivotalNetwork andlogin.
2. ClickPivotalContainerService(PKS).
©CopyrightPivotalSoftwareInc,2013-2019 25 1.0
https://network.pivotal.io/products/pivotal-container-servicehttps://network.pivotal.io/https://network.pivotal.io/https://network.pivotal.io/
-
3. ClickPKSCLI.
4. ClickPKSCLI-WindowstodownloadtheWindowsexecutablefile.
5. Renamethedownloadedbinaryto pks.exe .
6. Movethebinaryintoyour PATH .
LogintoPKSCLIOnthecommandline,runthefollowingcommandtologintothePKSCLI:
pks login -a PKS_API -u USERNAME -p PASSWORD --ca-cert CERT-PATH
Replacetheplaceholdervaluesinthecommandasfollows:
PKS_API isthedomainnameyouenteredinOpsManager>PivotalContainerService>UAA>UAAURL.Forexample, api.pks.example.com .
USERNAME and PASSWORD belongtotheaccountyoucreatedintheGrantClusterAccesstoaUserstepinManageUsersinUAA.
CERT-PATH isthepathtoyourrootCAcertificate.ProvidethecertificatetovalidatethePKSAPIcertificatewithSSL.
Forexample:
$pkslogin-aapi.pks.example.com-ualana\--ca-cert/var/tempest/workspaces/default/root_ca_certificate
Ifyouareloggingintoatrustedenvironment,youcanuse -k toskipSSLverificationinsteadof --ca-certCERT-PATH .
Forexample:
$pkslogin-aapi.pks.example.com-ualana-k
Uponsuccessfullogin,thePKSCLIgeneratesa creds.yml filecontainingtheAPIendpoint,CAcertificate(ifapplicable),refreshtoken,andaccesstoken.
Bydefault, creds.yml issavedinthe ~/.pks directory.Youcanusethe PKS_HOME environmentvariabletooverridethislocationanduse creds.yml fromanydirectory.
©CopyrightPivotalSoftwareInc,2013-2019 26 1.0
mailto:[email protected]
-
InstallingtheKubernetesCLIPagelastupdated:
ThistopicdescribeshowtoinstalltheKubernetesCommandLineInterface(kubectl).
Toinstallkubectl,followtheproceduresforyouroperatingsystemtodownloadkubectlfromPivotalNetwork .Binariesareonlyprovidedfor64-bitarchitectures.
MacOSX1. NavigatetoPivotalNetwork andlogin.
2. ClickPivotalContainerService(PKS).
3. ClickKubectlCLIs.
4. ClickkubectlCLI-Mactodownloadthekubectlbinary.
5. Renamethedownloadedbinaryto kubectl .
6. Onthecommandline,runthefollowingcommandtomakethekubectlbinaryexecutable:
$chmod+xkubectl
7. Movethebinaryintoyour PATH .Forexample:
$mvkubectl/usr/local/bin/kubectl
Linux1. NavigatetoPivotalNetwork andlogin.
2. ClickPivotalContainerService(PKS).
3. ClickKubectlCLIs.
4. ClickkubectlCLI-Linuxtodownloadthekubectlbinary.
5. Renamethedownloadedbinaryto kubectl .
6. Onthecommandline,runthefollowingcommandtomakethekubectlbinaryexecutable:
$chmod+xkubectl
7. Movethebinaryintoyour PATH .Forexample:
$mvkubectl/usr/local/bin/kubectl
Windows1. NavigatetoPivotalNetwork andlogin.
2. ClickPivotalContainerService(PKS).
3. ClickKubectlCLIs.
4. ClickkubectlCLI-Windowstodownloadthekubectlexecutablefile.
©CopyrightPivotalSoftwareInc,2013-2019 27 1.0
https://network.pivotal.io/products/pivotal-container-servicehttps://network.pivotal.io/https://network.pivotal.io/https://network.pivotal.io/
-
5. Renamethedownloadedbinaryto kubectl.exe .
6. Movethebinaryintoyour PATH .
©CopyrightPivotalSoftwareInc,2013-2019 28 1.0
mailto:[email protected]
-
PreparingtoInstallPKSonvSphereThistopicoutlinesthestepsforpreparingtoinstallPivotalContainerService(PKS)onvSphere.Seethefollowingsections:
vSpherePrerequisitesandResourceRequirements
FirewallPortsandProtocolsRequirementsforvSpherewithNSX-T
PreparingtoDeployPKStovSphere
DeployingOpsManagertovSphere
ConfiguringOpsManageronvSphere
InstallingandIntegratingVMwareHarborRegistrywithPKS
©CopyrightPivotalSoftwareInc,2013-2019 29 1.0
https://docs.pivotal.io/partners/vmware-harbor/index.htmlmailto:[email protected]
-
vSpherePrerequisitesandResourceRequirementsPagelastupdated:
ThistopicdescribestheprerequisitesandresourcerequirementsforinstallingPivotalContainerService(PKS)onvSpherewithorwithoutNSX-Tintegration.
PKSsupportsair-gappeddeploymentsonvSpherewithorwithoutNSX-Tintegration.
YoucanalsoconfigureintegrationwiththeHarbortile,anenterprise-classregistryserverforcontainerimages.Formoreinformation,seetheVMwareHarborRegistry documentation.
ComponentVersionRequirements
vSphereVersionRequirementsPKSonvSpheresupportsthefollowingvSpherecomponentversions:
Versions Editions
VMwarevSphere6.5GA
VMwarevSphere6.5U1
vSphereEnterprisePlus
vSpherewithOperationsManagementEnterprisePlus
NSX-TIntegrationVersionRequirementsDeployingNSX-Trequirestheadditionalfollowingcomponentversions:
Component Version
VMwareNSX-T 2.1
ResourceRequirementsInstallingPKSdeploysthefollowingtwovirtualmachines(VMs):
VM CPU RAM Storage
PivotalContainerService 1 4GB 20GB
PivotalOpsManager 1 8GB 160GB
EachKubernetesclusterprovisionedthroughPKSdeploystheVMslistedbelow.IfyoudeploymorethanoneKubernetescluster,youmustscaleyourallocatedresourcesappropriately.
VMName Number CPUCores RAM EphemeralDisk PersistentDisk
master 1 2 4GB 8GB 5GB
worker 1 2 4GB 8GB 10GB
NSX-TIntegrationResourceRequirementsDeployingNSX-TrequirestheadditionalfollowingresourcesfromyourvSphereenvironment:
NSX-TComponent InstanceCount MemoryperInstance vCPUperInstance DiskSpaceperInstance
NSXManagerAppliance 1 16GB 4 140GB
NSXControllers 3 16GB 4 120GB
NSX-TEdge 1upto8 16GB 8 120GB
©CopyrightPivotalSoftwareInc,2013-2019 30 1.0
https://docs.pivotal.io/partners/vmware-harbor/index.html
-
InstallingPKSonvSpherewithNSX-TForinformationaboutthefirewallportsandprotocolsrequirementsforusingPKSonvSpherewithNSX-T,seeFirewallPortsandProtocolsRequirementsforvSpherewithNSX-T.
ToinstallandconfigurePKSwithNSX-Tintegration,followtheproceduresbelow:
1. InstallingandConfiguringPKSwithNSX-TIntegration
2. (Optional)InstallingandIntegratingVMwareHarborRegistrywithPKS
InstallingPKSonvSpherewithoutNSX-TToinstallPKSonvSpherewithoutNSX-Tintegration,followtheproceduresbelow:
1. PreparingtoDeployPKStovSphere
2. DeployingOpsManagertovSphere
3. ConfiguringOpsManageronvSphere
4. InstallingandConfiguringPKS
5. (Optional)InstallingandIntegratingVMwareHarborRegistrywithPKS
AboutDeployingPASandPKSThePivotalApplicationService(PAS)andPKSruntimeplatformsarebothdeployedbyOpsManagerusingBOSH.YoucandeploybothPASandPKSusingthesameOpsManagerinstanceinadevelopmentortestenvironment,butwerecommendthatyoudeployproductioninstallationsofPASandPKStoseparateOpsManagerinstances.Forincreasedsecurity,werecommenddeployingeachOpsManagerinstanceusingauniquecloudprovideraccount.
SeparateinstallationsofOpsManagerallowyoutocustomizeandtroubleshootruntimetilesindependently.YoumaychoosetoconfigureOpsManagerwithdifferentsettingsforyourPASandPKSdeployments.Forexample,PKSandmanyPASfeaturesdependonBOSHDNS.
IfyoudeployPAStoaseparateOpsManagerinstance,youcandisableBOSHDNSfortroubleshootingpurposes.PAScanrunwithoutBOSHDNS,butkeyfeaturessuchassecureservicecredentialswithCredHub,servicediscoveryforcontainer-to-containernetworking,andNSX-TintegrationdonotworkwhenBOSHDNSisdisabled.
IfyoudeployPASandPKStothesameOpsManagerinstance,youcannotdisableBOSHDNSwithoutbreakingyourPKSinstallationalongwiththePASfeaturesthatdependonBOSHDNS.
©CopyrightPivotalSoftwareInc,2013-2019 31 1.0
https://docs.pivotal.io/partners/vmware-harbor/https://docs.pivotal.io/partners/vmware-harbor/mailto:[email protected]
-
FirewallPortsandProtocolsRequirementsforvSpherewithNSX-TPagelastupdated:
ThistopicdescribesthefirewallportsandprotocolsrequirementsforusingPivotalContainerService(PKS)onvSpherewithNSX-Tintegration.
Inenvironmentswithstrictinter-networkaccesscontrolpolicies,firewallsoftenrequireconduitstopasscommunicationbetweensystemcomponentsonadifferentnetworkorallowinterfacingwithexternalsystemssuchaswithenterpriseapplicationsorthepublicInternet.
ForPKS,therecommendationistodisablesecuritypoliciesthatfiltertrafficbetweenthenetworkssupportingthesystem.Whenthatisnotanoption,refertothefollowingtable,whichidentifiestheflowsbetweensystemcomponentsinatypicalPKSdeployment.
SourceComponent DestinationComponent DestinationProtocol DestinationPort Service
ApplicationUser K8sClusterWorkerNodes TCP 30000-32767 k8snodeport
ApplicationUser K8sLoad-Balancers TCP/UDP varies varies
ApplicationUser K8sIngress-Controllers TCP/UDP varies varies
CloudFoundryBOSHDirector DomainNameServer UDP 53 dns
CloudFoundryBOSHDirector vCenterServer TCP 443 https
CloudFoundryBOSHDirector vSphereESXIMgmt.vmknic TCP 443 https
CompilationJobVMs DomainNameServer UDP 53 dns
Developer HarborPrivateImageRegistry TCP 4443 notary
Developer HarborPrivateImageRegistry TCP 443 https
Developer HarborPrivateImageRegistry TCP 80 http
Developer K8sClusterMaster/EtcdNodes TCP 8443 uaaauth
Developer K8sClusterWorkerNodes TCP 30000-32767 k8snodeport
Developer K8sLoad-Balancers TCP/UDP varies varies
Developer K8sIngress-Controllers TCP/UDP varies varies
DomainNameServer vCenterServer UDP 1433 ms-sql-server
HarborPrivateImageRegistry DomainNameServer UDP 53 dns
HarborPrivateImageRegistry PublicCVESourceDatabase TCP 443 https
HarborPrivateImageRegistry PublicCVESourceDatabase TCP 80 http
K8sClusterMaster/EtcdNodes CloudFoundryBOSHDirector TCP 4222 boshnatsserver
K8sClusterMaster/EtcdNodes CloudFoundryBOSHDirector TCP 25250 boshblobstore
K8sClusterMaster/EtcdNodes DomainNameServer UDP 53 dns
K8sClusterMaster/EtcdNodes NSXManagerServer TCP 443 https
K8sClusterMaster/EtcdNodes vCenterServer TCP 443 https
K8sClusterWorkerNodes CloudFoundryBOSHDirector TCP 4222 boshnatsserver
K8sClusterWorkerNodes CloudFoundryBOSHDirector TCP 25250 boshblobstore
K8sClusterWorkerNodes DomainNameServer UDP 53 dns
K8sClusterWorkerNodes HarborPrivateImageRegistry TCP 8853 boshdnshealth
K8sClusterWorkerNodes HarborPrivateImageRegistry TCP 443 https
K8sClusterWorkerNodes NSXManagerServer TCP 443 https
K8sClusterWorkerNodes vCenterServer TCP 443 https
NSXControllers NetworkTimeServer UDP 123 ntp
NSXEdgeManagement NSXEdgeTEPvNIC UDP 3784 bfd
NSXManagerServer DomainNameServer UDP 53 dns
NSXManagerServer SFTPBackupServer TCP 22 ssh
Operator HarborPrivateImageRegistry TCP 443 https
Operator HarborPrivateImageRegistry TCP 80 http
Operator K8sLoad-Balancers TCP 80 http
Operator NSXManagerServer TCP 443 https
©CopyrightPivotalSoftwareInc,2013-2019 32 1.0
-
Operator NSXManagerServer TCP 443 https
Operator PCFOperationsManager TCP 22 ssh
Operator PCFOperationsManager TCP 443 https
Operator PCFOperationsManager TCP 80 http
Operator PKSController TCP 8443 uaaauth
Operator PKSController TCP 9021 pksapiserver
Operator vCenterServer TCP 443 https
Operator vCenterServer TCP 80 http
Operator vSphereESXIMgmt.vmknic TCP 22 ssh
PCFOperationsManager DomainNameServer UDP 53 dns
PCFOperationsManager K8sClusterWorkerNodes TCP 22 ssh
PCFOperationsManager NetworkTimeServer UDP 123 ntp
PCFOperationsManager vCenterServer TCP 443 https
PCFOperationsManager vSphereESXIMgmt.vmknic TCP 443 https
PKSController DomainNameServer UDP 53 dns
PKSController K8sClusterMaster/EtcdNodes TCP 8443 uaaauth
PKSController NSXManagerServer TCP 443 https
PKSController vCenterServer TCP 443 https
vCenterServer DomainNameServer UDP 53 dns
vCenterServer NetworkTimeServer UDP 123 ntp
vCenterServer vSphereESXIMgmt.vmknic TCP 8080 vsanvp
vCenterServer vSphereESXIMgmt.vmknic TCP 9080 iofilterstorage
vCenterServer vSphereESXIMgmt.vmknic TCP 443 https
vCenterServer vSphereESXIMgmt.vmknic TCP 902 ideafarm-door
SourceComponent DestinationComponent DestinationProtocol DestinationPort Service
Note:Youhavetheoptiontoexposecontainerizedapplications,runninginaKubernetescluster,forexternalconsumptionthroughvariousportsandmethods.YoucanenableexternalaccesstoapplicationsbywayofKubernetesNodePorts,load-balancers,andingress.EnablingaccesstoapplicationsviaKubernetesload-balancersandingresscontrollertypesallowforspecificportandprotocoldesignations,whileNodePortofferstheleastcontrolanddynamicallyallocatesportsfromapre-definedrangeofports.
©CopyrightPivotalSoftwareInc,2013-2019 33 1.0
mailto:[email protected]
-
PreparingtoDeployPKSonvSpherePagelastupdated:
BeforeyouinstallPivotalContainerService(PKS)onvSpherewithoutNSX-Tintegration,youmustprepareyourvSphereenvironment.InadditiontofulfillingtheprerequisitesspecifiedinvSpherePrerequisitesandResourceRequirements,youmustcreatethefollowingtwoserviceaccountsinvSphere:
MasterNodeServiceAccount:YoumustcreateaserviceaccountforKubernetesclustermasterVMs.
BOSH/OpsManagerServiceAccount:YoumustcreateaserviceaccountforBOSHandOpsManager.
Afteryoucreatetheserviceaccountslistedabove,youmustgrantthemprivilegesinvSphere.Pivotalrecommendsconfiguringeachserviceaccountwiththeleastpermissiveprivilegesanduniquecredentials.
Forthemasternodeserviceaccount,youcancreateacustomroleinvSpherebasedonyourstorageconfiguration.KubernetesmasternodeVMsrequirestoragepermissionstocreateloadbalancersandattachpersistentdiskstopods.CreatingacustomroleallowsvSpheretoapplythesameprivilegestoallKubernetesmasternodeVMsinyourPKSinstallation.
WhenyouconfiguretheKubernetesCloudProviderpaneofthePKStile,youenterthemasternodeserviceaccountcredentialsinthevSphereMasterCredentialsfields.Formoreinformation,seetheKubernetesCloudProvidersectionofInstallingandConfiguringPKS.
FortheBOSH/OpsManagerserviceaccount,youcanapplyprivilegesdirectlytotheserviceaccountwithoutcreatingarole.YoucanalsoapplythedefaultVMwareAdministratorSystemRole totheserviceaccounttoachievetheappropriatepermissionlevel.
Step1:CreatetheMasterNodeServiceAccount1. FromthevCenterconsole,createaserviceaccountforKubernetesclustermasterVMs.
2. GrantthefollowingVirtualMachineObjectpermissionstotheserviceaccount:
Privilege(UI) Privilege(API)
Advanced VirtualMachine.Configuration.Advanced
Settings VirtualMachine.Configuration.Settings
Step2:GrantAdditionalStoragePermissionsKubernetesmasternodeVMserviceaccountsrequirethefollowing:
Readaccesstothefolder,host,anddatacenteroftheclusternodeVMs
PermissiontocreateanddeleteVMswithintheresourcepoolwherePKSisdeployed
Grantthesepermissionstothemasternodeserviceaccountbasedonyourstorageconfigurationusingoneoftheproceduresbelow:
StaticOnlyPersistentVolumeProvisioning
DynamicPersistentVolumeProvisioning(withStoragePolicy-BasedVolumePlacement)
DynamicPersistentVolumeProvisioning(withoutStoragePolicy-BasedVolumePlacement)
SeevSphereStorageforKubernetes intheVMwaredocumentationformoreinformation.
StoragePermissionsforServiceAccountsThefollowingtablesdescribetheminimumpermissionsrequiredbythemasternodeserviceaccountbasedonyourstorageconfiguration.
StaticOnlyPersistentVolumeProvisioning
Roles Privileges Entities PropagatetoChildren
Note:IfyourKubernetesclustersspanmultiplevCenters,youmustsettheserviceaccountprivilegescorrectlyineachvCenter.
©CopyrightPivotalSoftwareInc,2013-2019 34 1.0
http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.wssdk.pg.doc/PG_Authenticate_Authorize.8.6.html#1110514https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/index.html
-
manage-k8s-node-vms
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddNewDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.RemoveDisk
VMFolder Yes
manage-k8s-volumes Datastore.FileManagement(Lowlevelfileoperations) Datastore No
Read-only(pre-existingdefaultrole)
System.Anonymous
System.Read
System.View
vCenter
Datacenter
DatastoreCluster
DatastoreStorageFolder
No
DynamicPersistentVolumeProvisioning(withStoragePolicy-BasedVolumePlacement)
Roles Privileges Entities PropagatetoChildren
manage-k8s-node-vms
Resource.AssignVMToPool
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddNewDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.RemoveDisk
VirtualMachine.Inventory.Create
VirtualMachine.Inventory.Delete
Cluster
Hosts
VMFolderYes
manage-k8s-volumes
Datastore.AllocateSpace
Datastore.FileManagement(Lowlevelfileoperations)
Datastore No
k8s-system-read-and-spbm-profile-view
StorageProfile.View(Profile-drivenstorageview) vCenter No
Read-only(pre-existingdefaultrole)
System.Anonymous
System.Read
System.View
Datacenter
DatastoreCluster
DatastoreStorageFolder
No
DynamicVolumeProvisioning(withoutStoragePolicy-BasedVolumePlacement)
Roles Privileges Entities PropagatetoChildren
manage-k8s-node-vms
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddNewDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.RemoveDisk
VMFolder Yes
manage-k8s-volumes
Datastore.AllocateSpace
Datastore.FileManagement(Lowlevelfileoperations)
Datastore No
System.AnonymousvCenter
Datacenter
Note:Datastore.FileManagementisonlyrequiredfortherole manage-k8s-volumes ifaPersistentVolumeClaim(PVC)iscreatedtobindwithastaticallyprovisionedPersistentVolume(PV),andthereclaimpolicysettodelete.WhenthePVCisdeleted,thestaticallyprovisionedPVisalsodeleted.
©CopyrightPivotalSoftwareInc,2013-2019 35 1.0
-
Read-only(pre-existingdefaultrole) System.Read
System.View
DatastoreCluster
DatastoreStorageFolder
No
Step3:CreatetheBOSH/OpsManagerServiceAccount1. FromthevCenterconsole,createaserviceaccountforBOSHandOpsManager.
2. GrantthepermissionsbelowtotheBOSHandOpsManagerserviceaccount.
vCenterRootPrivilegesGrantthefollowingprivilegesontherootvCenterserverentitytotheserviceaccount:
Privilege(UI) Privilege(API)
Read-only System.Anonymous
System.Read
System.View
Managecustomattributes Global.ManageCustomFields
vCenterDatacenterPrivilegesGrantthefollowingprivilegesonanyentitiesinadatacenterwhereyoudeployPKS:
RoleObject
Privilege(UI) Privilege(API)
UsersinherittheRead-OnlyrolefromthevCenterrootlevel System.Anonymous
System.Read
System.View
DatastoreObject
Grantthefollowingprivilegesmustatthedatacenterleveltouploadanddeletevirtualmachinefiles:
Privilege(UI) Privilege(API)
Allocatespace Datastore.AllocateSpace
Browsedatastore Datastore.Browse
Lowlevelfileoperations Datastore.FileManagement
Removefile Datastore.DeleteFile
Updatevirtualmachinefiles Datastore.UpdateVirtualMachineFiles
FolderObject
Privilege(UI) Privilege(API)
Deletefolder Folder.Delete
Note:TheprivilegeslistedinthissectiondescribetheminimumrequiredpermissionstodeployBOSH.YoucanalsoapplythedefaultVMwareAdministratorSystemRole totheserviceaccounttoachievetheappropriatepermissionlevel,butthedefaultroleincludesmoreprivilegesthanthoselistedbelow.
©CopyrightPivotalSoftwareInc,2013-2019 36 1.0
http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.wssdk.pg.doc/PG_Authenticate_Authorize.8.6.html#1110514
-
Createfolder Folder.CreateMovefolder Folder.Move
Renamefolder Folder.Rename
GlobalObject
Privilege(UI) Privilege(API)
Setcustomattribute Global.SetCustomField
HostObject
Privilege(UI) Privilege(API)
Modifycluster Host.Inventory.EditCluster
InventoryServiceObject
Privilege(UI) Privilege(API)
vSphereTagging>CreatevSphereTag InventoryService.Tagging.CreateTag
vSphereTagging>DeletevSphereTag InventoryService.Tagging.EditTag
vSphereTagging>EditvSphereTag InventoryService.Tagging.DeleteTag
NetworkObject
Privilege(UI) Privilege(API)
Assignnetwork Network.Assign
ResourceObject
Privilege(UI) Privilege(API)
Assignvirtualmachinetoresourcepool Resource.AssignVMToPool
Migratepoweredoffvirtualmachine Resource.ColdMigrate
Migratepoweredonvirtualmachine Resource.HotMigrate
vAppObject
Granttheseprivilegesattheresourcepoollevel.
Privilege(UI) Privilege(API)
Import VApp.Import
vAppapplicationconfiguration VApp.ApplicationConfig
VirtualMachineObject
Configuration
Privilege(UI) Privilege(API)
Addexistingdisk VirtualMachine.Config.AddExistingDisk
Addnewdisk VirtualMachine.Config.AddNewDisk
Addorremovedevice VirtualMachine.Config.AddRemoveDevice
©CopyrightPivotalSoftwareInc,2013-2019 37 1.0
-
Advanced VirtualMachine.Config.AdvancedConfigChangeCPUcount VirtualMachine.Config.CPUCount
Changeresource VirtualMachine.Config.Resource
ConfiguremanagedBy VirtualMachine.Config.ManagedBy
Diskchangetracking VirtualMachine.Config.ChangeTracking
Disklease VirtualMachine.Config.DiskLease
Displayconnectionsettings VirtualMachine.Config.MksControl
Extendvirtualdisk VirtualMachine.Config.DiskExtend
Memory VirtualMachine.Config.Memory
Modifydevicesettings VirtualMachine.Config.EditDevice
Rawdevice VirtualMachine.Config.RawDevice
Reloadfrompath VirtualMachine.Config.ReloadFromPath
Removedisk VirtualMachine.Config.RemoveDisk
Rename VirtualMachine.Config.Rename
Resetguestinformation VirtualMachine.Config.ResetGuestInfo
Setannotation VirtualMachine.Config.Annotation
Settings VirtualMachine.Config.Settings
Swapfileplacement VirtualMachine.Config.SwapPlacement
Unlockvirtualmachine VirtualMachine.Config.Unlock
GuestOperations
Privilege(UI) Privilege(API)
GuestOperationProgramExecution VirtualMachine.GuestOperations.Execute
GuestOperationModifications VirtualMachine.GuestOperations.Modify
GuestOperationQueries VirtualMachine.GuestOperations.Query
Interaction
Privilege(UI) Privilege(API)
Answerquestion VirtualMachine.Interact.AnswerQuestion
ConfigureCDmedia VirtualMachine.Interact.SetCDMedia
Consoleinteraction VirtualMachine.Interact.ConsoleInteract
Defragmentalldisks VirtualMachine.Interact.DefragmentAllDisks
Deviceconnection VirtualMachine.Interact.DeviceConnection
GuestoperatingsystemmanagementbyVIXAPI VirtualMachine.Interact.GuestControl
Poweroff VirtualMachine.Interact.PowerOff
Poweron VirtualMachine.Interact.PowerOn
Reset VirtualMachine.Interact.Reset
Suspend VirtualMachine.Interact.Suspend
VMwareToolsinstall VirtualMachine.Interact.ToolsInstall
Inventory
Privilege(UI) Privilege(API)
Createfromexisting VirtualMachine.Inventory.CreateFromExisting
Createnew VirtualMachine.Inventory.Create
Move VirtualMachine.Inventory.Move
Register VirtualMachine.Inventory.Register
Remove VirtualMachine.Inventory.Delete
Unregister VirtualMachine.Inventory.Unregister
©CopyrightPivotalSoftwareInc,2013-2019 38 1.0
-
Provisioning
Privilege(UI) Privilege(API)
Allowdiskaccess VirtualMachine.Provisioning.DiskRandomAccess
Allowread-onlydiskaccess VirtualMachine.Provisioning.DiskRandomRead
Allowvirtualmachinedownload VirtualMachine.Provisioning.GetVmFiles
Allowvirtualmachinefilesupload VirtualMachine.Provisioning.PutVmFiles
Clonetemplate VirtualMachine.Provisioning.CloneTemplate
Clonevirtualmachine VirtualMachine.Provisioning.Clone
Customize VirtualMachine.Provisioning.Customize
Deploytemplate VirtualMachine.Provisioning.DeployTemplate
Markastemplate VirtualMachine.Provisioning.MarkAsTemplate
Markasvirtualmachine VirtualMachine.Provisioning.MarkAsVM
Modifycustomizationspecification VirtualMachine.Provisioning.ModifyCustSpecs
Promotedisks VirtualMachine.Provisioning.PromoteDisks
Readcustomizationspecifications VirtualMachine.Provisioning.ReadCustSpecs
SnapshotManagement
Privilege(UI) Privilege(API)
Createsnapshot VirtualMachine.State.CreateSnapshot
Removesnapshot VirtualMachine.State.RemoveSnapshot
Renamesnapshot VirtualMachine.State.RenameSnapshot
Revertsnapshot VirtualMachine.State.RevertToSnapshot
NextStepsToinstallPKSonvSphere,followtheproceduresinDeployingOpsManagertovSphere.
©CopyrightPivotalSoftwareInc,2013-2019 39 1.0
mailto:[email protected]
-
DeployingOpsManagertovSpherePagelastupdated:
ThistopicprovidesinstructionsfordeployingOpsManagertoVMwarevSphere.
1. Beforestarting,refertotheknownissuesinthePCFOpsManagerReleasev2.0ReleaseNotes .
2. DownloadthePivotalCloudFoundry (PCF)OpsManager .ova fileatPivotalNetwork .ClickthePivotalCloudFoundryregiontoaccessthePCFproductpage.UsethedropdownmenutoselectanOpsManagerrelease.
3. LogintovCenter.
4. SelecttheVMandTemplatesview.
5. RightclickonyourdatacenterandselectNewFolder.
Note:WithvSphere6.5andNSX-T2.1,wheninitiallydeployingtheOperationsManagerOVF,youcannotconnectdirectlytoanNSX-Tlogicalswitch.YoumustfirstconnecttoavSphereStandard(vSS)orvSphereDistributedSwitch(vDS).AsuggestedapproachistoconnecttoaVSSorVDSwhendeployingtheOVF,butdonotpowertheVMon.AftertheOVFdeploymenthascompleted,youcanthenconnectthenetworkinterfacetotheappropriateNSX-TlogicalswitchandpowertheVMontoproceedwiththeinstall.ThisissueisresolvedinVMwarevCenterServer6.7.Formoreinformationaboutthisissue,seetheVMwareKnowledgeBase .
©CopyrightPivotalSoftwareInc,2013-2019 40 1.0
https://kb.vmware.com/kb/54142http://docs.pivotal.io/pivotalcf/2-0/pcf-release-notes/opsmanager-rn.htmlhttps://network.pivotal.io/products/pivotal-cfhttps://network.pivotal.io
-
6. Namethefolder pivotal_cf andselectit.
7. SelectFile>DeployOVFTemplate.
8. Selectthe.ovafileandclickNext.
9. ReviewtheproductdetailsandclickNext.
10. AcceptthelicenseagreementandclickNext.
11. NamethevirtualmachineandclickNext.
12. SelectavSphereclusterandclickNext.
13. Ifprompted,selectaresourcepoolandclickNext.
14. Ifprompted,selectahostandclickNext.
Note:Theselectedfolderistheoneyoucreated.
Note:IfyourvSpherehostdoesnotsupportVT-X/EPT,hardwarevirtualizationmustbe**off**.Formoreinformation,seePCFonvSphereRequirements .
©CopyrightPivotalSoftwareInc,2013-2019 41 1.0
https://docs.pivotal.io/pivotalcf/customizing/vsphere.html#vsphere-reqs
-
15. SelectastoragedestinationandclickNext.
16. SelectadiskformatandclickNext.Formoreinformationaboutdiskformats,seeProvisioningaVirtualDiskonvSphere .
17. SelectanetworkfromthedropdownlistandclickNext.
18. EnternetworkinformationandpasswordsfortheOpsManagerVMadminuser.
19. IntheAdminPasswordfield,enteradefaultpasswordfortheubuntuuser.Ifyoudonotenteradefaultpassword,yourOpsManagerwillnotbootup.
20. ClickNext.
21. CheckthePoweronafterdeploymentcheckboxandclickFinish.OncetheVMboots,theinterfaceisavailableattheIPaddressyouspecified.
Warning:OpsManagerv2.0requiresaDirectorVMwithatleast8GBmemory.
Note:Recordthisnetworkinformation.TheIPAddresswillbethelocationoftheOpsManagerinterface.
Note:ItisnormaltoexperienceabriefdelaybeforetheinterfaceisaccessiblewhilethewebserverandVMstartup.
©CopyrightPivotalSoftwareInc,2013-2019 42 1.0
https://docs.pivotal.io/pivotalcf/customizing/disk-format.html
-
22. CreateaDNSentryfortheIPaddressthatyouusedforOpsManager.YoumustusethisfullyqualifieddomainnamewhenyoulogintoOpsManagerinInstallingPivotalCloudFoundryonvSphere .
NextStepsAfteryoucompletethisprocedure,followtheinstructionsinConfiguringOpsManageronvSphere.
Note:OpsManagersecurityfeaturesrequireyoutocreateafullyqualifieddomainnametoaccessOpsManagerduringtheinitialconfiguration.
©CopyrightPivotalSoftwareInc,2013-2019 43 1.0
https://docs.pivotal.io/pivotalcf/customizing/vsphere.html#paashttps://docs.pivotal.io/pivotalcf/customizing/vsphere.html#paasmailto:[email protected]
-
ConfiguringOpsManageronvSpherePagelastupdated:
ThistopicdescribeshowtoconfigureOpsManagerforVMwarevSphere.
IfyouareinstallingPivotalContainerService(PKS)tovSpherewithoutNSX-Tintegration,beforeyoubeginthisprocedure,ensurethatyouhavesuccessfullycompletedallofthestepsinDeployingOpsManagertovSphere.
Step1:SetUpOpsManager1. NavigatetothefullyqualifieddomainofyourOpsManagerinawebbrowser.
2. ThefirsttimeyoustartOpsManager,youmustchooseoneofthefollowing:
UseanIdentityProvider:IfyouuseanIdentityProvider,anexternalidentityservermaintainsyouruserdatabase.InternalAuthentication:IfyouuseInternalAuthentication,PCFmaintainsyouruserdatabase.
UseanIdentityProvider(IdP)1. LogintoyourIdPconsoleanddownloadtheIdPmetadataXML.Optionally,ifyourIdPsupportsmetadataURL,youcancopythemetadataURLinsteadoftheXML.
Note:YoucanalsoperformtheproceduresinthistopicusingtheOpsManagerAPI.Formoreinformation,seeUsingtheOpsManagerAPI .
©CopyrightPivotalSoftwareInc,2013-2019 44 1.0
https://docs.pivotal.io/pivotalcf/customizing/ops-man-api.html
-
2. CopytheIdPmetadataXMLorURLtotheOpsManagerUseanIdentityProviderloginpage.
3. EnteryourDecryptionpassphrase.ReadtheEndUserLicenseAgreement,andselectthecheckboxtoaccepttheterms.
4. YourOpsManagerloginpageappears.Enteryourusernameandpassword.ClickLogin.
5. DownloadyourSAMLServiceProvidermetadata(SAMLRelyingPartymetadata)bynavigatingtothefollowingURLs:
5a.OpsManagerSAMLserviceprovidermetadata: https://OPS-MAN-FQDN:443/uaa/saml/metadata5b.BOSHDirectorSAMLserviceprovidermetadata: https://BOSH-IP-ADDRESS:8443/saml/metadata
6. ConfigureyourIdPwithyourSAMLServiceProvidermetadata.ImporttheOpsManagerSAMLprovidermetadatafromStep5aabovetoyourIdP.IfyourIdPdoesnotsupportimporting,providethevaluesbelow.
SinglesignonURL: https://OPS-MAN-FQDN:443/uaa/saml/SSO/alias/OPS-MAN-FQDNAudienceURI(SPEntityID): https://OP-MAN-FQDN:443/uaaNameID:EmailAddressSAMLauthenticationrequestsarealwayssigned
7. ImporttheBOSHDirectorSAMLprovidermetadatafromStep5btoyourIdP.IftheIdPdoesnotsupportanimport,providethevaluesbelow.
SinglesignonURL: https://BOSH-IP:8443/saml/SSO/alias/BOSH-IPAudienceURI(SPEntityID): https://BOSH-IP:8443NameID:EmailAddressSAMLauthenticationrequestsarealwayssigned
8. ReturntotheOpsManagerDirectortile,andcontinuewiththeconfigurationstepsbelow.
InternalAuthentication1. WhenredirectedtotheInternalAuthenticationpage,youmustcompletethefollowingsteps:
EnteraUsername,Password,andPasswordconfirmationtocreateanAdminuser.EnteraDecryptionpassphraseandtheDecryptionpassphraseconfirmation.ThispassphraseencryptstheOpsManagerdatastore,andisnotrecoverable.IfyouareusinganHTTPproxyorHTTPSproxy,followtheinstructionsinConfiguringProxySettingsfortheBOSHCPI .ReadtheEndUserLicenseAgreement,andselectthecheckboxtoaccepttheterms.
Note:ThesameIdPmetadataURLorXMLisappliedfortheBOSHDirector.IfyouuseaseparateIdPforBOSH,copythemetadataXMLorURLfromthatIdPandenteritintotheBOSHIdPMetadatatextboxintheOpsManagerloginpage.
Note:Toretrieveyour BOSH-IP-ADDRESS ,navigatetotheOpsManagerDirectortile>Statustab.RecordtheOpsManagerDirectorIPaddress.
©CopyrightPivotalSoftwareInc,2013-2019 45 1.0
https://docs.pivotal.io/pivotalcf/customizing/pcf-director-proxy-settings.html
-
Step2:vCenterConfigPage1. LogintoOpsManagerwiththeAdminusernameandpasswordyoucreatedinthepreviousstep.
2. ClicktheOpsManagerDirectortile.
3. SelectvCenterConfig.
©CopyrightPivotalSoftwareInc,2013-2019 46 1.0
-
4. Enterthefollowinginformation:
vCenterHost:ThehostnameofthevCenterthatmanagesESXi/vSphere.vCenterUsername:AvCenterusernamewithcreateanddeleteprivilegesforvirtualmachines(VMs)andfolders.vCenterPassword:ThepasswordforthevCenteruserspecifiedabove.DatacenterName:ThenameofthedatacenterasitappearsinvCenter.VirtualDiskType:TheVirtualDiskTypetoprovisionforallVMs.Forguidanceonselectingavirtualdisktype,seeProvisioningaVirtualDiskinvSphere .EphemeralDatastoreNames(commadelimited):ThenamesofthedatastoresthatstoreephemeralVMdisksdeployedbyOpsManager.PersistentDatastoreNames(commadelimited):ThenamesofthedatastoresthatstorepersistentVMdisksdeployedbyOpsManager.VMFolder:ThevSpheredatacenterfolder(default: pcf_vms )whereOpsManagerplacesVMs.TemplateFolder:ThevSpheredatacenterfolder(default: pcf_templates )whereOpsManagerplacesVMs.
©CopyrightPivotalSoftwareInc,2013-2019 47 1.0
https://docs.pivotal.io/pivotalcf/customizing/disk-format.html
-
DiskpathFolder:ThevSpheredatastorefolder(default: pcf_disk )whereOpsManagercreatesattacheddiskimages.Youmustnotnestthisfolder.
5. SelectStandardvCenterNetworking.ThisisthedefaultoptionwhenupgradingOpsManager.ThisconfigurationisutilizedforPASonly.YouconfigureNSX-TintegrationforPKSwithinthePKStile.
6. ClickSave.
Step3:DirectorConfigPage1. SelectDirectorConfig.
2. IntheNTPServers(commadelimited)field,enteryourNTPserveraddresses.
3. LeavetheJMXProviderIPAddressfieldblank.
4. LeavetheBoshHMForwarderIPAddressfieldblank.
5. SelecttheEnableVMResurrectorPlugintoenableOpsManagerResurrectorfunctionality.
6. SelectEnablePostDeployScriptstorunapost-deployscriptafterdeployment.Thisscriptallowsthejobtoexecuteadditionalcommandsagainstadeployment.
Note:Afteryourinitialdeployment,youwillnotbeabletoedittheVMFolder,TemplateFolder,andDiskpathFoldernames.
Note:StartingfromPCFv2.0,BOSH-reportedcomponentmetricsareavailableintheLoggregatorFirehosebydefault.Therefore,ifyoucontinuetousePCFJMXBridgeforconsumingthemoutsideoftheFirehose,youmayreceiveduplicatedata.Topreventthis,leavetheJMXProviderIPAddressfieldblank.
Note:StartingfromPCFv2.0,BOSH-reportedcomponentmetricsareavailableintheLoggregatorFirehosebydefault.Therefore,ifyoucontinuetousetheBOSHHMForwarderforconsumingthem,youmayreceiveduplicatedata.Topreventthis,leavetheBoshHMForwarderIPAddressfieldblank.
©CopyrightPivotalSoftwareInc,2013-2019 48 1.0
-
7. SelectRecreateallVMstoforceBOSHtorecreateallVMsonthenextdeploy.Thisprocessdoesnotdestroyanypersistentdiskdata.
8. SelectEnableboshdeployretriesifyouwantOpsManagertoretryfailedBOSHoperationsuptofivetimes.
GCSBlobstore:SelectthisoptiontouseanexternalGoogleCloudStorage(GCS)endpoint.TocreateaGCSbucket,youwillneedaGCSaccount.FollowtheproceduresinCreatingStorageBuckets intheGCPdocumentation.AfteryouhavecreatedaGCSbucket,completethefollowingsteps:
1. BucketName:EnterthenameofyourGCSbucket.2. StorageClass:SelectthestorageclassforyourGCSbucket.Formoreinformation,seeStorageClasses intheGCPdocumentation.3. ServiceAccountKey:FollowthestepsintheCreateServiceAccountssectiontodownloadaJSONfilewithaprivatekey,andthenenterthecontentsoftheJSONfileintothefield.
9. Bydefault,PCFdeploysandmanagesanInternaldatabaseforyou.IfyouchoosetouseanExternalMySQLDatabase,completetheassociatedfieldswithinformationobtainedfromyourexternalMySQLDatabaseprovider:Host,Port,Username,Password,andDatabase.
Note:Youmustenablepost-deployscriptstoinstallPKS.
©CopyrightPivotalSoftwareInc,2013-2019 49 1.0
https://cloud.google.com/storage/docs/creating-bucketshttps://cloud.google.com/storage/docs/storage-classes
-
10. (Optional)DirectorWorkerssetsthenumberofworkersavailabletoexecuteDirectortasks.Thisfielddefaultsto 5 .
11. (Optional)MaxThreadssetsthemaximumnumberofthreadsthattheOpsManagerDirectorcanrunsimultaneously.ForvSphere,thedefaultvalueis 32 .Leavethefieldblanktousethisdefaultvalue.PivotalrecommendsthatyouusethedefaultvalueunlessdoingsoresultsinratelimitingorerrorsonyourIaaS.
12. LeavetheDirectorHostnamefieldblank.
13. EnsuretheDisableBOSHDNSserverfortroubleshootingpurposescheckboxisnotselected.
14. Optional:TosetacustombannerthatusersseewhenloggingintotheDirectorusingSSH,entertextintheCustomSSHBannerfield.
15. ClickSave.
Step4:CreateAvailabilityZonePageOpsManagerAvailabilityZonescorrespondtoyourvCenterclustersandresourcepools.MultipleAvailabilityZonesallowyoutoprovidehigh-availabilityandloadbalancingtoyourapplications.Whenyourunmorethanoneinstanceofanapplication,OpsManagerbalancesthoseinstancesacrossalloftheAvailabilityZonesassignedtotheapplication.Atleastthreeavailabilityzonesarerecommendedforahighlyavailableinstallationofyourchosenruntime.
Note:BOSHDNSmustbeenabledinallPKSdeployments.IfPASandPKSarerunningonthesameinstanceofOpsManager,youcannotusetheopt-outfeatureofBOSHDNSforyourPASwithoutbreakingPKS.IfyouwanttooptoutofBOSHDNSinyourPASdeployment,installthetileonaseparateinstanceofOpsManager.FormoreinformationaboutoptingoutofBOSHDNS,seeDisablingorOptingOutofBOSHDNSinPCF(PivotalKnowledgeBasearticle) andBOSHDNSServiceDiscovery(Beta)andOpt-OutOption intheOpsManagerv2.0ReleaseNotes.
Note:Afteryourinitialdeployment,youwillnotbeabletoedittheBlobstoreandDatabaselocations.
©CopyrightPivotalSoftwareInc,2013-2019 50 1.0
https://discuss.pivotal.io/hc/en-us/articles/115015720428-Disabling-or-Opting-Out-of-BOSH-DNS-in-PCFhttps://docs.pivotal.io/pivotalcf/2-0/pcf-release-notes/opsmanager-rn.html#bosh-dns
-
1. SelectCreateAvailabilityZones.
2. UsethefollowingstepstocreateoneormoreAvailabilityZonesforyourapplicationstouse:
ClickAdd.EnterauniqueNamefortheAvailabilityZone.EnterthenameofanexistingvCenterClustertouseasanAvailabilityZone.(Optional)EnterthenameofaResourcePoolinthevCenterclusterthatyouspecifiedabove.ThejobsrunninginthisAvailabilityZonesharetheCPUandmemoryresourcesdefinedbythepool.(Optional)ClickAddClustertocreateanothersetofClusterandResourcePoolfields.Youcanaddmultipleclusters.Clickthetrashicontodeleteacluster.Thefirstclustercannotbedeleted.
3. ClickSave.
Step5:CreateNetworksPage1. SelectCreateNetworks.
2. SelectEnableICMPcheckstoenableICMPonyournetworks.OpsManagerusesICMPcheckstoconfirmthatcomponentswithinyournetworkarereachable.
3. ClickAddNetworkandcreatethefollowingnetworks:
pks-infrastructure :forOpsManager,theBOSHDirector,thePKSbroker,andthePKSAPI.Ifyouhavealargedeploymentwithmultipletiles,youcanchoosetodeploythePKSbrokerandPKSAPItoaseparatenetworknamed pks-main .Seethetablebelowformoreinformation.pks-services :forcreatingthemasterandworkerVMsforKubernetesclusters.
Usethevaluesfromthefollowingtableasaguidewhenyoucreateeachnetwork,replacingtheIPaddresseswithrangesthatareavailableinyourvSphereenvironment:
Field Configuration
Name
Note:FormoreinformationaboutusingavailabilityzonesinvSphere,seeUnderstandingAvailabilityZonesinVMwareInstallations .
Note:IfyouaredeployingPKSwithNSX-Tintegration,seethenetworkconfigurationtableintheConfigureOpsManagersectionofInstallingandConfiguringPKSwithNSX-TIntegration.
©CopyrightPivotalSoftwareInc,2013-2019 51 1.0
https://docs.pivotal.io/pivotalcf/customizing/understand-az.html
-
InfrastructureNetwork
pks-infrastructureServiceNetwork LeaveServiceNetworkunchecked.
vSphereNetworkName MY-PKS-virt-net/MY-PKS-subnet-infrastructure
CIDR 192.168.101.0/26
ReservedIPRanges 192.168.101.1-192.168.101.9
DNS 192.168.101.2
Gateway 192.168.101.1
MainNetwork(Optional)
Field Configuration
Name pks-main
ServiceNetwork LeaveServiceNetworkunchecked.
vSphereNetworkName MY-PKS-virt-net/MY-PKS-subnet-pks
CIDR 192.168.16.0/26
ReservedIPRanges 192.168.16.1-192.168.16.9
DNS 192.168.16.2
Gateway 192.168.16.1
ServiceNetwork
Field Configuration
Name pks-services
ServiceNetwork SelecttheServiceNetworkcheckbox.
vSphereNetworkName MY-PKS-virt-net/MY-PKS-subnet-services
CIDR 192.168.20.0/22
ReservedIPRanges 192.168.20.1-192.168.20.9
DNS 192.168.20.2
Gateway 192.168.20.1
4. SelectwhichAvailabilityZonestousewiththenetwork.
5. ClickSave.
Step6:AssignAZsandNetworksPage1. SelectAssignAZsandNetworks.
Note:MultiplenetworksallowyoutoplacevCenteronaprivatenetworkandtherestofyourdeploymentonapublicnetwork.IsolatingvCenterinthismannerdeniesaccesstoitfromoutsidesourcesandreducespossiblesecurityvulnerabilities.
Note:IfyouareusingtheCiscoNexus1000vSwitch,seemoreinformationinUsingtheCiscoNexus1000vSwitchwithOpsManager .
©CopyrightPivotalSoftwareInc,2013-2019 52 1.0
https://docs.pivotal.io/pivotalcf/customizing/nexus-switch.html
-
2. Usethedrop-downmenutoselectaSingletonAvailabilityZone.TheOpsManagerDirectorinstallsinthisAvailabilityZone.
3. Usethedrop-downmenutoselectaNetworkforyourOpsManagerDirector.
4. ClickSave.
Step7:SecurityPage1. SelectSecurity.
2. InTrustedCertificates,enteracustomcertificateauthority(CA)certificatetoinsertintoyourorganization’scertificatetrustchain.ThisfeatureenablesallBOSH-deployedcomponentsinyourdeploymenttotrustacustomrootcertificate.IfyouwanttouseDockerRegistriesforrunningappinstancesinDockercontainers,usethisfieldtoenteryourcertificateforyourprivateDockerRegistry.Formoreinformation,seeUsingDockerRegistries .
©CopyrightPivotalSoftwareInc,2013-2019 53 1.0
https://docs.pivotal.io/pivotalcf/opsguide/docker-registry.html
-
3. ChooseGeneratepasswordsorUsedefaultBOSHpassword.PivotalrecommendsthatyouusetheGeneratepasswordsoptionforincreasedsecurity.
4. ClickSave.ToviewyoursavedDirectorpassword,clicktheCredentialstab.
Step8:SyslogPage1. SelectSyslog.
2. (Optional)TosendBOSHDirectorsystemlogstoaremoteserver,selectYes.
3. IntheAddressfield,entertheIPaddressorDNSnamefortheremoteserver.
4. InthePortfield,entertheportnumberthattheremoteserverlistenson.
5. IntheTransportProtocoldropdownmenu,selectTCP,UDP,orRELP.Thisselectiondetermineswhichtransportprotocolisusedtosendthelogstotheremoteserver.
6. (Optional)MarktheEnableTLScheckboxtouseTLSencryptionwhensendinglogstotheremoteserver.
InthePermittedPeerfield,entereitherthenameorSHA1fingerprintoftheremotepeer.IntheSSLCertificatefield,entertheSSLcertificatefortheremoteserver.
©CopyrightPivotalSoftwareInc,2013-2019 54 1.0
-
7. ClickSave.
Step9:ResourceConfigPage1. SelectResourceConfig.
2. Adjustanyvaluesasnecessaryforyourdeployment.UndertheInstances,PersistentDiskType,andVMTypefields,chooseAutomaticfromthedrop-downmenutoallocatetherecommendedresourcesforthejob.IfthePersistentDiskTypefieldreadsNone,thejobdoesnotrequirepersistentdiskspace.
3. ClickSave.
Step10:CompletetheOpsManagerInstallation1. ClicktheInstallationDashboardlinktoreturntotheInstallationDashboard.
2. ClickApplyChangesontherightnavigation.
NextStepsToinstallPKSonvSpherewithNSX-Tintegration,performtheproceduresinInstallingandConfiguringPKSwithNSX-TIntegration.
ToinstallPKSonvSpherewithoutNSX-Tintegration,performtheproceduresinInstallingandConfiguringPKS.
TouseHarbortostoreandmanagecontainerimages,seeInstallingandIntegratingVMwareHarborRegistrywithPKS .
Note:OpsManagerrequiresaDirectorVMwithatleast8GBmemory.
Note:IfyousetafieldtoAutomaticandtherecommendedresourceallocationchangesinafutureversion,OpsManagerautomaticallyusestheupdatedrecommendedallocation.
©CopyrightPivotalSoftwareInc,2013-2019 55 1.0
https://docs.pivotal.io/partners/vmware-harbormailto:[email protected]
-
PreparingtoInstallPKSonGCPThistopicoutlinesthestepsforpreparingtoinstallPivotalContainerService(PKS)onGCP.Seethefollowingsections:
GCPPrerequisitesandResourceRequirements
PreparingtoDeployPKSonGCP
DeployingOpsManagertoGCP
ConfiguringOpsManageronGCP
ConfiguringaGCPLoadBalancerforthePKSAPI
ConfiguringaGCPLoadBalancerforPKSClusters
©CopyrightPivotalSoftwareInc,2013-2019 56 1.0
mailto:[email protected]
-
GCPPrerequisitesandResourceRequirementsPagelastupdated:
ThistopicdescribestheprerequisitesandresourcerequirementsforinstallingPivotalContainerService(PKS)onGoogleCloudPlatform(GCP).
ResourceRequirementsInstallingPKSdeploysthefollowingtwovirtualmachines(VMs):
VM CPU RAM Storage
PivotalContainerService 1 4GB 20GB
PivotalOpsManager 1 8GB 160GB
EachKubernetesclusterprovisionedthroughPKSdeploystheVMslistedbelow.IfyoudeploymorethanoneKubernetescluster,youmustscaleyourallocatedresourcesappropriately.
VMName Number CPUCores RAM EphemeralDisk PersistentDisk
master 1 2 4GB 8GB 5GB
worker 1 2 4GB 8GB 10GB
InstallingPKSonGCPToinstallPKSonGCP,followtheproceduresbelow:
1. PreparingtoDeployPKSonGCP
2. DeployingOpsManagertoGCP
3. ConfiguringOpsManageronGCP
4. InstallingandConfiguringPKS
AboutDeployingPASandPKSThePivotalApplicationService(PAS)andPKSruntimeplatformsarebothdeployedbyOpsManagerusingBOSH.YoucandeploybothPASandPKSusingthesameOpsManagerinstanceinadevelopmentortestenvironment,butwerecommendthatyoudeployproductioninstallationsofPASandPKStoseparateOpsManagerinstances.Forincreasedsecurity,werecommenddeployingeachOpsManagerinstanceusingauniquecloudprovideraccount.
SeparateinstallationsofOpsManagerallowyoutocustomizeandtroubleshootruntimetilesindependently.YoumaychoosetoconfigureOpsManagerwithdifferentsettingsforyourPASandPKSdeployments.Forexample,PKSandmanyPASfeaturesdependonBOSHDNS.
IfyoudeployPAStoaseparateOpsManagerinstance,youcandisableBOSHDNSfortroubleshootingpurposes.PAScanrunwithoutBOSHDNS,butkeyfeaturessuchassecureservicecredentialswithCredHub,servicediscoveryforcontainer-to-containernetworking,andNSX-TintegrationdonotworkwhenBOSHDNSisdisabled.
IfyoudeployPASandPKStothesameOpsManagerinstance,youcannotdisableBOSHDNSwithoutbreakingyourPKSinstallationalongwiththePASfeaturesthatdependonBOSHDNS.
©CopyrightPivotalSoftwareInc,2013-2019 57 1.0
mailto:[email protected]
-
PreparingtoDeployPKSonGCPPagelastupdated:
ThisguidedescribesthepreparationstepsrequiredtoinstallPivotalContainerService(PKS)onGoogleCloudPlatform(GCP).
InadditiontofulfillingtheprerequisiteslistedintheGCPPrerequisitesandResourceRequirementstopic,youmustcreateresourcesinGCPsuchasanewnetwork,firewallrules,loadbalancers,andaserviceaccountbeforedeployingPKS.FollowtheseprocedurestoprepareyourGCPenvironment.
Step1:EnableGoogleCloudAPIsOpsManagermanagesGCPresourcesusingtheGoogleComputeEngineandCloudResourceManagerAPIs.ToenabletheseAPIs,performthefollowingsteps:
1. LogintotheGoogleDevelopersconsoleathttps://console.developers.google.com .
2. Intheconsole,navigatetotheGCPprojectwhereyouwanttoinstallPKS.
3. SelectEnableAPIs&ServicestoaccesstheAPILibrary.
4. Inthesearchfield,enter Compute Engine API andpressEnter.
5. OntheGoogleComputeEngineAPIpage,clickEnable.
6. Inthesearchfield,enter Cloud Resource Manager API andpressEnter.
7. OntheGoogleCloudResourceManagerAPIpage,clickEnable.
8. ToverifythattheAPIshavebeenenabled,performthefollowingsteps:
a. LogintoGCP:
$gcloudauthlogin
b. Listyourprojects:
$gcloudprojectslistPROJECT_IDNAMEPROJECT_NUMBERmy-project-idmy-project-name##############
ThiscommandliststheprojectswhereyouenabledGoogleCloudAPIs.
Step2:CreateServiceAccountsInorderforKubernetestocreateloadbalancersandattachpersistentdiskstopods,youmustcreateserviceaccountswithsufficientpermissions.
YouneedseparateserviceaccountsforKubernetesclustermasterandworkernodeVMs,andathirdaccountforBOSHandOpsManager.Pivotalrecommendsconfiguringeachserviceaccountwiththeleastpermissiveprivilegesanduniquecredentials.
CreatetheMasterNodeServiceAccount1. FromtheGCPConsole,selectIAM&admin>Serviceaccounts.
2. ClickCreateServiceAccount.
3. Enteranamefortheserviceaccount,andaddthefollowingroles:
ComputeEngine
StorageAdminNetworkAdminSecurityAdmin
©CopyrightPivotalSoftwareInc,2013-2019 58 1.0
https://console.developers.google.com
-
InstanceAdmin(v1)ComputeViewer
IAM
ServiceAccountUser
4. SelectFurnishanewprivatekeyandselectJSON.
5. ClickCreate.YourbrowserautomaticallydownloadsaJSONfilewithaprivatekeyforthisaccount.Savethisfileinasecurelocation.
CreatetheWorkerNodeServiceAccount1. FromtheGCPConsole,selectIAM&admin>Serviceaccounts.
2. ClickCreateServiceAccount.
3. Enteranamefortheserviceaccount,andaddtheComputeEngine>ComputeViewerrole.
4. SelectFurnishanewprivatekeyandselectJSON.
5. ClickCreate.YourbrowserautomaticallydownloadsaJSONfilewithaprivatekeyforthisaccount.Savethisfileinasecurelocation.
CreatetheBOSH/OpsManagerServiceAccount1. FromtheGCPConsole,selectIAM&admin>Serviceaccounts.
2. ClickCreateServiceAccount.
3. Enteranamefortheserviceaccount,andaddthefollowingroles:
ServiceAccounts
ServiceAccountUserServiceAccountTokenCreator
ComputeEngine
ComputeInstanceAdmin(v1)ComputeNetworkAdminComputeStorageAdmin
Storage
StorageAdmin
4. SelectFurnishanewprivatekeyandselectJSON.
5. ClickCreate.YourbrowserautomaticallydownloadsaJSONfilewithaprivatekeyforthisaccount.Savethisfileinasecurelocation.
Step3:CreateaGCPNetworkwithSubnets1. LogintotheGCPConsole .
2. NavigatetotheGCPprojectwhereyouwanttoinstallPKS.
3. SelectVPCnetwork,thenCREATEVPCNETWORK.
4. IntheNamefield,enter your-pks-virt-net . your-pks isalower-caseprefixtohelpyouidentifyresourcesforthisPKSdeploymentintheGCPconsole.
Note:PivotalrecommendsconfirmingthepermissionsofyourMasterNodeServiceAccount,WorkerNodeServiceAccount,andBOSH/OpsManagerServiceAccountafteryoucreatethem.Toverifytheseaccountpermissions,runthe gcloudauth
listcommand.Formoreinformation,see
gcloudaut