case study: echostar transforms complex software releases to customer network set top boxes with ca...

World®’16

EchoStarTransformsComplexSoftwareReleasestoCustomerNetworkSetTopBoxeswithCAReleaseAutomationContinuousDeliveryEditionMonicaTanguay,EngineeringManager,SWConfigurationManagement,EchoStarDonBuckalew,StaffEngineeringProgramManager,EchoStar

D05X12S

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Agenda

INTRODUCINGECHOSTAR

SCM/DCMISSUESANDCHALLENGESTHEYPOSED

FUTUREENHANCEMENTSBUILTONCARELEASEAUTOMATIONSOLUTIONS

TACKLINGDCMISSUESINPHASE1

AUTOMATINGSTREAMBUILDSWITHINDCMWITHCARELEASEAUTOMATIONSOLUTIONS

STEPPINGTHROUGHIMPLEMENTATION

1

2

3

4

5

6


AboutEchoStar

Industryleaderinearthtosatellitecommunicationstotransferdatatoover10Mset-topboxcustomersinMexico,UnitedStatesandCanada

Weareaglobalpowerhouse§ Multibillion-dollarpubliccompany(NASDAQ:SATS)§ BuiltandlaunchedDISHthe2nd largestdirecttohome

entertainmentcontentprovider

Weshapethefutureofourindustry§ InventorofHopperwhole-homeHDDVRfamilyproductline

§ DevelopedfirstsatelliteTVDigitalVideoRecorder(DVR)

⎻ Inventorofthecommercialsatelliteverysmallapertureterminal(VSAT)

⎻ FirsttoimplementUHF/RFremotecontroltechnology


Let’sstepthroughhowSCMandDCMworkstodelivercontenttoyoursettopbox

OurSatelliteCommunicationsandSoftwareTransferWereBeingImpactedbyOngoingSCM/DCMIssues

SoftwareConfigurationManagement(SCM)§ Buildsset-top-boxsoftware§ Verifiesthereproducibilityofthebuild§ Auditsthechangesincludedinthebuild§ Encryptsthesoftware§ Distributestoothergroups

DownloadConfigurationManagement(DCM)§ TakesthesoftwareSCMbuilds/audits§ CreatesanMPEGstream§ Passesthestreamtotest§ Routestheapproval§ SendstotheUplinkforspoolingtothe

generalpopulation

SCM output triggers DCM process


OurmanualDCMflowintroducedmanyopportunitiesforhumanerrorandmistakes

DownloadConfigurationManagementProcessFlow


SoftwareConfigurationManagementProcessFlow

Manymanualsteps,highpotentialformistakesandlengthybuildtime


§ SCM output triggers DCM process but today, the communication between them is not automated

§ Each group has over 40 manual processes

§ Prone to redundancy errors – copy/paste

§ Tools out of date – some 10 years old§ Difficult to track work load – no metrics

§ Limited internal development support

SummingUptheChallengesPosedbyDCM/SCMProcesses


§ Start with download configuration management (DCM) = Phase 1⎻ DCM errors are more visible than SCM errors to our stakeholders⎻ Impact on customer experience is greater because DCM delivers code to the viewer

§ Find a way to automate as many of the manual processes as possible§ Alleviate the simple errors§ Create new expandable/adaptable tools§ Have the system integrate with other organizations within EchoStar§ Track metrics

EstablishedClearObjectivestoTackleSCM/DCMChallenges


PinpointedCorrective/PreventativeActionstoAddressHighErrorRateDuringDRBuildProcess

Identifiedseveralissuesthat,if

addressed,couldresolveasmuchas79%ofissuesforDownloadRequest

(DR)Builds14

5 4 4 3 3 2 25

33%

45% 55%

64% 71%

79% 83% 88% 100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

05101520253035404550

ApprovalDoc/Targeting

Table

Label/Set-up Network-Stream/zipFile

PoingStream datFile AppVersion BuildStreamTransactions

Passcode Other

XiP DRFailuresbyDRBuildProcess1/5/15– 8/20/15


§ Improveefficiency

§ Removemiscommunication

§ Informationcentralizedintoasingledatastore

ImperativetoAutomateMostofDCMProcessFlow

Phase CurrentProcess ProcessType ProposedProcess ProcessType

InitiateRequest

RequestCreated Manual CreatePackageRequestAssigned ManualReviewBetaList Manual

BuildFiles

EnterintoStatussheet Manual StatusSheetPopUpCreateDirectories Manual SystemDetermineStreamType Manual WebPageDropDownObtainSW Manual ManualValidateSW(md5) Manual SystemToolsAddtomd5file Manual SystemTools

CreateDocuments

CreateDocumentation Manual PackageCreateDeterminereleasetype Manual Selector/WebPage ManualModifyExcelfiles Manual PopUpStartApprovalDoc Manual WebPageDropDown

StreamBuild

BuildStreamfile Manual WebPagePopUpCreatezipfile Manual SystemUpdateStatussheet Manual WebPageObtainzippassword Manual WebPagePopUpCreateTeststreams Manual SystemPeerReview Manual WebPageLogic

TestStream

SubmittoTest Manual WebPageLogicAttachFiles Manual SystemToolsAttachApprovalDoc Manual PackageCreateAttachTestDocs Manual PackageCreateAttachdatafile Manual WebPage

DLApproval

Dispositionrequest Manual WebPageAuditComplete Manual InterfaceCreateSharepointfolder Manual SystemToolsUploaddocstoSharepoint Manual SystemToolsValidateSWonFTP Validation WebPageLogicSendApprovalemail Manual WebPageLogic

SendToSpool

CreateUplinkemail Manual WebPageLogicAttachApprovaldocs Manual PackageCreateVerifyspooling Manual InterfacewithUplinkSendSpoolingupdateemail Manual InterfacewithUplinkUpdateStatussheet Manual WebPageLogicRequestcomplete N/A WebPageNotification


§ Anewfrontendsystem

§ ThecurrenthomegrownsystemwasbasedonPHP

§ Internalsupportwasslowingdown

§ Atooltorunprocessesautomatically

§ Adaptabilityinintegrationwithothersystems

§ Capturemetrics

OurHigh-LevelCriteriaforAutomation&Tooling

WENEEDED


ConsideredSeveralTechnologyOptions

JiraMongoDB– Openstack

ASP.netCA

ReleaseAutomation

CAReleaseAutomationContinuousDeliveryEdition


JIRABenefits§ Providesdashboard

reporting§ Provides

visualizationofworkflow

§ Standardtoolsetusedincompany

§ Limitedtalentpool

ASP.NetBenefits§ Canbemodifiedforotherfacetsof

company(e.g.Muxbuilder)§ TalentPool(in-house)§ Wellunderstoodtechnology§ UpdatetechnologyofDSB§ MuxExpress(notRequired)§ Reductioninerrorrate§ Futureexpandability

CAReleaseAutomationBenefits§ Singlelogictoolfor

workflow§ Easeofuse§ Underlyingcode

maintainedbyvendor§ In-housetalentpool

(Limited)

(ASP.NET)- DataCollectionmodule

SolutionATechnologyCombination

JIRAManagementOversight/Report

CAReleaseManagement(RA)-AutomatedTask

Execution


CDE/RABenefits§ Providespre-built

dashboards§ Providesvisualizationofflow§ Allowsformorecomplex

logic§ IntegratedwithRA(RESTSvc)§ VendorresponsibleforCode

ASP.NetBenefits§ Canbemodifiedforotherfacetsof

company(e.g.Muxbuilder)§ Talentpool(in-house)§ Wellunderstoodtechnology§ UpdatetechnologyofDSB§ MuxExpress(notRequired)§ Reductioninerrorrate§ Futureexpandability

RAAutomationBenefits§ Singlelogictoolfor

workflow§ Easeofuse§ Vendorresponsible

forcode§ In-houseTalent

pool(Limited)

(ASP.NET)- DataCollectionmodule

SolutionBTechnologyCombination

CAReleaseAutomationContinuousDeliveryEdition(CDE)ManagementOversight/Reporting

CAReleaseManagement(RA)-AutomatedTask

Execution


DeeperDiveintoDevelopmentRequirementsforSolutionB

Buildacustomfrontend,with“RestServices”pulled/pushingdatabetweenEngineeringapplicationsandCATechnologiesapplications

UtilizingCAReleaseAutomationContinuousDeliveryEditionandCAReleaseAutomationapplications,allowedauto-deliveryof“Reviews”and“Approval”documentswithseniormanagement

DevelopedexecutioncomponentswhichallowedforautomaticdeliveryofsoftwarepackagestoEchoStarUplinkCenters

RequiredutilizingteamsfromCATechnologies,EchoStarConfigurationManagementteamandASP.NETdevelopers


DCMAutomationCollectorCreatedbyEchoStarDatacollectionhubforDownloadRequests

IntegratedwithCAReleaseAutomation(deploymentengine)andCAReleaseAutomationContinuousDeliveryEdition(releasemanagement)

Enables§ CreationandconfigurationofDATfiledataforbuildingstreams§ Providesmechanismstosupportthebuildprocessthroughoutcommunication,managementreview&approval,testingandreal-timeandhistoricalreporting.

Thisapplicationisindevelopment.


CAReleaseAutomationContinuousDeliveryEditionIndividualreleaseautomaticallygeneratedforeachDownloadRequest

CAReleaseAutomationCDEdition

CAReleaseAutomation(deploymentengine)

ReleaseDashboardofCAReleaseAutomationCDEdition

DCMAutomationCollectorTriggersMonitors

JohnSmith

JohnSmith

JohnSmith

JohnSmith

CAReleaseAutomation(deploymentengine)constantlymonitorsdatabaseforstatusupdatesofdownloadrequests(DR).

IttriggersCAReleaseAutomationCDEditiontoinstantaneouslycreateareleaseforeachDR.


DeeperDiveintoCAReleaseAutomationCDEditionFlowforeachindividualreleasecorrespondingtoaDownloadRequest

CAReleaseAutomationCDEditiontakesover:

§ TriggersautomateddeploymentinCAReleaseAutomationtodelivertherelease

§ Monitors“Collector”todetermineifcanproceedwithnexttaskinCDEphase

§ CancelsreleaseinCDEifataskfails

StatusofTasksUpdates

CAReleaseAutomation(DeploymentEngine)

CAReleaseAutomationCDEditionDCMAutomationCollector

TriggersMonitors


CAReleaseAutomationDashboardDeepdiveintoautomateddeploymentprocessesbehindeachrelease

CAReleaseAutomationCDEditionleveragesCAReleaseAutomationasthedeploymentenginetocompletethereleaseoftheDownloadRequest


CAReleaseAutomationCDEditionDashboardReal-timevisibility&troubleshootingofindividualreleases

CAReleaseAutomationCDEditionprovidesvisibilityinto

§ ProgressofreleasecorrespondingtoaDownloadRequestthroughpipeline

§ Timetakentocompletearelease

§ Resourceconflicts&bottlenecks

§ Pointwherefailuresoccurring


PHASE1 PHASE2

AutomationofStreamBuildwithinDCM

LaunchedSeptember2016

AutomationofDocumentationandValidationforSCM

LaunchDecember2016 Q12017 Future

OptimizationofSCM/DCMProcesses

Integratewithotherorganizations§ Test§ SoftwarePMs

§ Betaorganization§ Uplinkcenters

CARELEASEAUTOMATIONCARELEASEAUTOMATIONCONTINUOUSDELIVERYEDITION

§ Reducerework§ Maintainquality§ Improveproductivity

§ Improvereporting

§ Reducerework§ Maintainquality§ Improveproductivity

§ Improvereporting

§ Streamlineflows§ Opencommunication

withreducedemail

§ Greaterefficiency

BenefitsBenefits

Benefits

ThisIsJusttheBeginning.We’reonaRoll.

PHASE3 PHASE4(andbeyond)


ProposedEnhancementsinPhase2– 4BuiltonCAReleaseAutomationandCAReleaseAutomationContinuousDeliveryEdition

§ SCMautomationofworkflow(re-usecomponentsfromDCM)

§ Automationof“Push”SpoolingdirectlytoSatellite,

§ AutomatingintakeprocessintoSCM/DCM– Projectmanagement/engineeringintake

process– toolsintegration

§ Mobiledevicescapabilitiesforworkflowapproval

§ Advancedreporting(Auto-generatedreportstomanagement)

§ Customerreporting/synchronization– (Auto-generatedreportingtoDISH,Channel

Master,othercustomers)

§ UpdatingDSBapplicationtonewertechnology– Allowingforon-goingsupportcapabilitiesof

application(PHPtoASP.net)

§ CansetthestageforfurtherdataanalyticswithothercomponentsofEchoStar

– E.g.,howlongdidasoftwarepatchtakefromrequest-throughuplink?


WrapUp

Anewindustrystandard:Pioneeringinnovationindeliveringsoftwaretosettopboxes

CAReleaseAutomationandCAReleaseAutomationContinuousDeliveryEditionarepivotaltechnologiesforautomatingprocessestominimizeerrors,increaseefficiencyandenforcestandards

Phase1– ChoseDCMprocessbecauseofvisibilityandgreaterimpactonviewerexperience

Freesupteamtoperformmoreanalysisandhigherendworkindevelopment

ProviderealtimeviewsfordevelopmentmanagementteamonprogressbyDCM/SCMteams

FuturephasescontinuetobuildoffCARA/CDE

Advancedanalytics&real-timevisibilityforstakeholders

Automaticallyupdatecustomersystems,eliminatingemailupdates


Questions?