connected corridors: i-210 pilot integrated corridor management … · 2019-12-17 · partners for...

PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY OF CALIFORNIA, BERKELEY

Connected Corridors: I-210 Pilot Integrated Corridor Management System

Core System High-Level Design

November 29, 2017 DRAFT

Partners for Advanced Transportation Technology works with researchers, practitioners, and industry to implement transportation research and innovation, including products and services that improve the efficiency, safety, and security of the transportation system.

I-210Pilot:CoreSystemHigh-LevelDesign

ii

Thispageleftblankintentionally


iii


iv

TableofContents

1. Introduction ...................................................................................................................................1

1.1. PurposeofDocument ............................................................................................................. 1

1.2. RelationtoSystemsEngineeringProcess................................................................................ 1

1.3. IntendedAudience.................................................................................................................. 2

1.4. DocumentOrganization .......................................................................................................... 2

2. SystemPrimaryObjectivesandPurpose.........................................................................................4

2.1. ProjectGoalsandObjectives .................................................................................................. 5

2.2. TechnicalCapabilitiesSought.................................................................................................. 7

3. HighLevelDesignObjectives,Constraints,andPrinciples ...............................................................8

4. CoreSystemHighLevelDesign .....................................................................................................10

4.1. MajorComponents ............................................................................................................... 10

4.2. FieldElements ....................................................................................................................... 11

4.3. DataHub ............................................................................................................................... 12

4.4. DecisionSupport ................................................................................................................... 13

4.5. CorridorManagement .......................................................................................................... 14

4.6. Primaryprocessflow............................................................................................................. 17

5. DataHubDesign...........................................................................................................................18

5.1. DataSources ......................................................................................................................... 21

5.2. DataPipelines........................................................................................................................ 23

5.2.1. SensingDataPipeline .......................................................................................... 235.2.2. CaltransFreewayTrafficDataPipeline................................................................ 245.2.3. HeterogeneousDataPipeline.............................................................................. 255.2.4. HomogenousDataPipeline ................................................................................. 275.2.5. PipelineControl ................................................................................................... 275.2.6. PipelineStatusandLogging................................................................................. 295.2.7. CorridorManagementSystem-DecisionSupportSystem(CMS-DSS)Pipeline ... 305.2.8. CMSCommandStatusPipeline ........................................................................... 34

5.3. ExternalInterface/DataGateway.......................................................................................... 35

5.3.1. Subscriptionendpoints........................................................................................ 375.3.2. Request/Responseendpoints ............................................................................. 37

5.4. DataHubCommandGateway............................................................................................... 37

5.4.1. Conductor............................................................................................................ 38


v

5.4.2. Camel................................................................................................................... 395.4.3. ActiveMQWorkflowStatusTopic ....................................................................... 395.4.4. ActiveMQWorkflowTaskTopic .......................................................................... 395.4.5. Monitor ............................................................................................................... 39

6. DecisionSupportSystemDesign...................................................................................................39

6.1. DSShighleveldesign............................................................................................................. 40

6.2. ResponsePlanManagement................................................................................................. 41

6.3. RulesEngine .......................................................................................................................... 43

6.4. Modeling ............................................................................................................................... 43

6.4.1. Modelingtechniques........................................................................................... 43


1

1. INTRODUCTION

ThisConnectedCorridorsHighLevelDesigndocumentprovidesthehighlevelsystemarchitectureforthesystemtobedeployedontheI-210corridor.ThesystemarchitecturedescribedhereisadirectresultoftheConnectedCorridorsSystemRequirementsdocumentandtheworkdoneatUCBerkeleyintrafficmodelingandcontrol.Thisdocumentprovidesthesystemarchitecture,highleveldesignoftheprimarysubsystems,thedecisions,assumptions,constraints,andreasoningbehindthatarchitecture,andcriticalfunctionseachsubsystemprovidesforthesystem.

Thesystem,tobepilotedalongasectionoftheI-210corridorintheSanGabrielValleyareaofLosAngelesCounty,aimstoimproveoverallcorridorperformanceduringincidents,unscheduledevents,andplannedevents.Thisistobeachievedbymoreefficientlymanagingexistingsystemsandinfrastructures,promotingcross-jurisdictionaloperations,andusingmulti-modaltrafficanddemandmanagementstrategiesthatconsiderallrelevantmodesoftransportation.

1.1. PURPOSEOFDOCUMENT

Thisdocumentprovidesthehighleveldesign,servingastheidentificationoftheprimarysubsystemsandmajorcomponentsaswellasthebasisfortheselection,development,andintegrationoftheseintoasystemthatsatisfiesthesystemrequirementsasdefinedintheSystemsRequirementsDocument.ThishighleveldesignwillgovernthetechnologyplatformanddirectionoftheI-210PilotICMSystemandserveasthebasisforotherCaltrans-ledICMeffortsstatewide.

1.2. RELATIONTOSYSTEMSENGINEERINGPROCESS

ThedevelopmentofhighleveldesignispartofthesystemsengineeringprocessthattheFederalHighwayAdministration(FHWA)requiresbefollowedfordevelopingIntelligentTransportationSystem(ITS)projectswhenfederalfundsareinvolved.Whilenotrequiredforprojectsonlyusingstateorlocalfunds,useofthesystemsengineeringprocessisstillencouragedinsuchcases.

TheoverallsystemsengineeringprocessisillustratedinFigure1-1.DevelopinghighleveldesignrepresentsthenextstepoftheSystemDefinitionandDesignphaseofaproject(Phase2inthefigure)followingthecompletionoftheSystemRequirements.HighLevelDesignistypicallyderivedfromtherequirements.Theresultingdesignelementsareinturnusedtoinformandguidethemoredetaileddesignofthevarioussystemandsubsystemcomponents.


2

Figure1-1–SystemRequirementsSpecificationwithinSystemsEngineeringProcess

1.3. INTENDEDAUDIENCE

TheprimaryaudiencefortheSystemHighLevelDesigndocumentincludespersonnelresponsiblefordesigningandimplementingtheICMsystem.TheaudiencealsoincludesindividualsfromCaltransDistrict7,CaltransHeadquarters,andtheUniversityofCalifornia,Berkeley,taskedwithprojectmanagementduties.

1.4. DOCUMENTORGANIZATION

Theremainderofthisdocumentisorganizedasfollows:

• Section2summarizestheprimarysystemobjectivesidentifiedwithintheSystemRequirementsthatshapethesystemdesign.

• Section3presentstheprimaryguidingdesignprinciplesandbaseassumptionsthatshapethesystemdesign.

• Section4presentsthekeysystemdesigncomponentsandprimarydataflows.


3

• Section 5 presents the key system components of the Data Hub including data sources,interfaces/gatewaysandpipelines.

• Section6presentsthekeysystemcomponentsoftheDecisionSupportSystem(DSS)includingtherulesengine,modelinginterfaces,andresponseplangeneration.

• Section7presentskeysecuritydesignissuesandimplementationplans.

• Section8providesdesigninformationforthemessagingsystemsanddescribeshowinformationispassedbetweensubsystems.

• Section9describesthesysteminterfaces.

Inaddition,othersupportingdocumentsareavailableintheDocumentLibraryoftheI-210Pilotwebsiteathttp://ccdocs.berkeley.edu/content/document-library:


4

2. SYSTEMPRIMARYOBJECTIVESANDPURPOSE

TheoverridingpurposeoftheI-210PilotistoreducecongestionandimprovemobilityalongasectionoftheI-210corridorinLosAngelesCountythroughthecoordinatedmanagementofitsmajornetworks:theI-210freeway,keysurroundingarterials,andlocalandregionaltransitservices.Thegoalistoenableallcorridor“actors”—transportationsystemmanagersandoperators,controlsystems,vehicles,andtravelers—toworktogetherinanefficientandcoordinatedway.

TheseimprovementswillbeachievedbydevelopinganddeployingtheICMsystemdescribedinthishighleveldesigndocument.AttheheartoftheproposedsystemwillbeaDecisionSupportSystem(DSS)designedtohelpcorridorsystemoperatorsmanageincidents,unscheduledevents,andplannedeventsmoreeffectively.Thissystemwilluseinformationgatheredfrommonitoringsystemsandprovidedbypredictiveanalyticaltoolstoestimatecurrentandnear-futureoperationalperformance.Theinformationwillbeusedtodeveloprecommendedcoursesofactiontoaddressproblemscausedbyidentifiedincidentsandevents.Morespecifically,thissystemisexpectedto:

• Improvereal-timemonitoringoftravelconditionswithinthecorridor• Enableoperatorstobettercharacterizetravelpatternswithinthecorridorandacrosssystems• Providepredictivetrafficandsystemperformancecapabilities• Beabletoevaluatealternativesystemmanagementstrategiesandrecommenddesiredcourses

ofactioninresponsetoplannedevents,unscheduledevents,andincidents• Improvedecision-makingbytransportationsystemmanagers• Improvecollaborationamongagenciesoperatingtransportationsystemsinthecorridor• Improvetheutilizationofexistinginfrastructureandsystems• Moreefficientlyusesparecapacitytoaddressnon-recurringcongestion• Reducedelaysandtraveltimesalongfreewaysandarterials• Improvetraveltimereliability• Helpreducethenumberofaccidentsoccurringalongthecorridor• Reducetheperiodduringwhichthecongestionresultingfromanincidentoreventaffects

corridoroperations• Reducegreenhousegasemissions• Generatehighertravelersatisfactionrates• IncreasetheoveralllivabilityofcommunitiesinandaroundtheI-210corridor

WhiledevelopmentoftheproposedsystemisunderthefinancialsponsorshipofCaltransHeadquarters,thesystemwillbedevelopedprimarilybythelocaltransportationagenciesthathaveagreedtoparticipateinitsoperation,incoordinationwithPATH.Projectactivitieswillincludethedesign,development,installation,testing,andoperationofvariouscomponentsoftheICMsystem,aswellasthedevelopmentofinterfaceswithexistingmonitoringandcontrolsystems.Forexample,theICMCoreSystemwillbeinterfacedwithtrafficmanagementsystemsownedbyCaltrans,suchastheAdvancedTrafficManagementSystem(ATMS).


5

2.1. PROJECTGOALSANDOBJECTIVES

TheprimarygoaloftheI-210PilotICMprojectistoimproveoverallcorridorperformancealongasectionoftheI-210corridor.Thistranslatesintothefollowingspecificgoals:

1. Improveoperationalsituationalawareness2. Promotecollaborationamongcorridorstakeholders3. Improveresponsetoincidentsandevents4. Improvetravelreliability5. Improveoverallcorridormobility6. Empowertravelerstomakeinformedtraveldecisions7. Facilitatemulti-modalmovementsacrosstheregion8. Promotetransportationsustainabilitybyreducingimpactsontheenvironment9. Improvecorridorsafety

Foreachofthesegoals,Table2-1furtheridentifiesthemainoperationalobjectives.Manyoftheobjectivesaresimilartothoseoftraditionaltransportationimprovementprojects.Many,however,alsofocusonimplementingmorecomprehensivetravelandsystemstatusmonitoringsystems,improvedoperationalforecasting,improvedinformationdisseminationtotravelers,enhanceddata-sharingcapabilities,demandmanagementapproaches,andimprovedcollaborationamongtransportationsystemoperators.

Table2-1–ICMSystemGoalsandObjectives

Goals Objectives

1.Improvesituationalawareness

• Establishminimumrequirementsfordatacollectiontosupportsystemmanagement• Increasedatacollectionopportunitiesfromarterialsandlocalroads• Improvethecollectionofreal-timeoperationaldatafromnon-traditionalsources,suchasprobevehicles

• Developacomprehensivecorridorinformationaldatabasecoveringallrelevanttravelmodeswithinthecorridor

• Improvethequality,accuracy,andvalidationprocessofcollecteddata• Increasetheabilitytoestimatetraveldemandpatternsinamulti-modalenvironment• Improvetheabilitytoforecastnear-futuretravelconditionsbasedonknownincidents,roadconditions,weather,andlocalevents

• Developperformancemetricsconsideringallavailabletravelmodes

2.Promotecollaborationamongcorridorstakeholders

• Strengthenexistingcommunicationchannelsamongthecorridor’sinstitutionalstakeholders• Exploreopportunitiesfornewcommunicationlinksbetweencorridorstakeholders• Improvecooperationandcollaborationamongcorridorstakeholders• Developregional/jointoperationsconcepts• Identifynewmethodsofcollaboration• Extendcorridorperformancemetricstothenetworklevel• Investigatenewtypesofagreementsbetweenparticipatingagencies


6

Goals Objectives

3.Improveresponsetoincidentsandunexpectedevents

• Reducethetimeneededtoidentifytheexistenceofanincidentorunexpectedevent• Reducethetimeneededtorespondtoincidentsorunscheduledevents• Enhancethecoordinationofactivitiesamongfirstresponders,trafficmanagementagencies,andtransitagenciestominimizeimpactsonsystemoperations

• Reducethetimeneededtoimplementcontrolactionstoaddresscongestionresultingfromanincidentorevent

• Reducethetimeneededtodisseminaterecommendeddetoursaroundanincidentorevent

4.Improvetravelreliability

• Improvetraveltimepredictabilityalongthecorridor• Reducetheimpactsofincidentsandeventsonnetworkoperations• Improveincident/eventnotificationforfirstrespondersandnetworkoperators• Improveincident/eventnotificationtotravelersandfleetoperators• Providetravelersandcommercialvehicleoperatorsaffectedbyanincidentoreventanenhancedabilitytoseekalternateroutesormodeoftransportation

5.Improveoverallcorridormobility

• Reducedelaysincurredbytravelers• Reducetheimpactsofincidentsandeventsonnetworkoperations• Efficientlyusesparecapacityalongcorridorroadwaystoplannecessarydetoursaroundincidentsorevents

• Promotestrategiestoinducedesirabletraveldemandpatterns• Coordinatethemanagementoffreewayandarterialbottlenecks• Promoteincreasesinvehicleoccupancy• Promoteincreasesintransitridership

6.Empowersystemuserstomakeinformedtraveldecisions

• Improvethedisseminationofreal-time,multi-modaltravelinformation• Enhancetheuseofinfrastructure-basedinformationaldevices(freewayCMS,arterialtrailblazersigns,kiosks,etc.)toprovideen-routeinformationtotravelers

• Enableindividualstoreceivetravelinformationonconnectedmobiledevices• Makearchivedhistoricaldataavailableto511servicesandinformationserviceproviders• Supportthedisseminationoftravelinformationby511servicesandthird-partyproviders

7.Facilitateregionalmulti-modalmovements

• Promotetheintegrationofcommuterrailandbusserviceswithcorridoroperations• Facilitatetransfersacrossmodesduringincidentsandevents• Providerelevantregionaltravelinformationtotravelers• Directtravelerstopark-and-ridefacilitieswithavailablespaces

8.Promotetransportationsustainability

• Reducefuelconsumption• Reducevehicleemissions• Identifyfinanciallysustainablesolutionsforlong-termsystemoperationsandmaintenance• Encouragetheuseoftransit,walking,andbicyclingwhereappropriate• Supportlocallypreferredalternativescompatiblewithcorridorobjectives• Developandimplementperformancemetricsreflectingenvironmentalgoals

9.Improvecorridorsafety

• Reducecollisionrates• Reducetheseverityofcollisions• Reducethenumberoffatalities• Reducetheimpactsofprimaryandsecondaryincidentsonnetworkoperationsthroughimprovedincidentmanagement

• Improvesafetyforbicycles,pedestrians,andtransit


7

2.2. TECHNICALCAPABILITIESSOUGHT

Tohelpmanagetravelactivitieswithinthecorridorduringincidents,unscheduledevents,andplannedevents,theprojectisseekingthefollowingtechnicalcapabilitiestosupportthegoalsandobjectivesidentifiedinSection2.1:

• Gatherandarchiveinformationcharacterizingtrafficoperations,transitoperations,andtheoperationalstatusofrelevantcontroldeviceswithintheI-210corridor.

• IdentifyunusualtravelconditionsontheI-210freewayornearbyarterialsbasedonmonitoringdataprovidedbyvarioustraffic,transit,andtravelmonitoringsystems.

• Identifysituationsinwhichanincidentonroadwaysortransitfacilitiessignificantlyaffectstravelconditionswithinthecorridor.

• Providecorridor-wideoperationalevaluationstotrafficmanagers,transitdispatchers,andotherrelevantsystemmanagers,includingprojectedassessmentsofnear-futuresystemoperationsundercurrentandalternatecontrolscenarios.

• Identifyrecommendeddetoursaroundincidentsorroutesleadingtothesiteofanevent,consideringobservedtravelconditionswithinthecorridor.Dependingontheneed,andfinalsystemcapabilities,specificdetoursmayberecommendedformotoristsandtransitvehicles.

• Identifyrecommendedsignaltimingplanstouseatsignalizedintersectionstoimproveand/oraccommodatetrafficflowinfluxduringincidentsandeventsandimproveoverallcorridormobility.

• IdentifyrecommendedrampmeteringratestouseonindividualI-210freewayon-rampsandconnectorstomaintainoverallcorridormobility.

• IdentifymessagestopostonavailablefreewayandarterialCMSstoinformmotoristsofincidentsandevents.

• ProvideguidancetomotoristsontheI-210freewayandsurroundingarterialsusingavailablefreewayCMSs,arterialCMSs,andarterialdynamictrailblazersignsregardingwhichdetourtotaketogoaroundanincidentorwhichroutetofollowtoreachthesiteofanevent.

• Provideinformationtomotoristsabouttheavailabilityofparkingandtransitservicestohelptravelersmakealternatemode-choicedecisions.

• Provideuniformtrafficmanagementstrategiesacrossjurisdictionalboundariesduringincidentsandevents.

• Provideinformationtomotoriststhroughthird-partyoutlets,suchas511services,navigationapplicationproviders,etc.


8

3. HIGHLEVELDESIGNOBJECTIVES,CONSTRAINTS,ANDPRINCIPLES

Thecoresystemdesignisgovernedbyafewkeyprimarydesignobjectivesdictatedbytheprojectandtherequirements:

• Realtimeoperation–Thesystemmustoperateinanearrealtimeenvironment.Thetimebetweenanincidentoccurringandaresponseplanimplementationisacriticaltimeperiod.Responsetimesshouldbedictatedprimarilybythetimetonotificationandconfirmationoftheincident,andthetimetofullyimplementadecision.Thetimerequiredbythesystemtoprocessinformationshouldbeminimizedsoastohavethemaximumpositiveimpactoncorridoroperations.

• Speedtodecision–Thereisasignificantamountofdataprocessingrequiredtomaintainbothoperatorawarenessandtoensurethemodelingwithinthedecisionsupportsystemisupdatedwiththelatestavailableinformation.Dataprocessinganddecisionprocessingtimeshouldbeminimized.

• Qualityofdecision–Thequalityoftheresponseplansisadirectresultofthetrafficestimationquality,trafficpredictionquality,dataprocessingtime,dataquality,andrulesusedtogenerateresponseplans.

• Abilitytomeasureoutcomes–Theremustbeahighlevelofconfidenceinthesystemoutcomes,andtheseoutcomesmustbecontinuouslymeasuredandmonitoredwithprocessesinplacetoimprovethesystemseffectivenessovertime.

• Incrementaldeployment–Thesystemmustbeabletobedeployedincrementally,addingnewcapabilitiesovertime.

• Systemflexibility–Thesystemmustbebuilttobeflexible,asitisintendedasapilotthatisadjustedandmodifiedasexperienceisgainedwithoperationsandgrowasnewcapabilitiesareadded.Inaddition,thisflexibilitywillbekeytofutureimplementationsinothercorridorswithdifferentintegrationneedsanddifferentrequirements.Itisalsoexpectedthattransportationitselfwillbechangingradicallyoverthelifetimeofthesystem,soflexibilityiskeytoitslongtermviability.

• Secureoperations–Asthesystemhasthecapabilitytorequestchangestotrafficcontrolsacrossalarge,complexurbancorridor,securityofthesystemiscriticaltoitssuccess.

Coresystemdesignislimitedbythefollowingconstraints:

• AbilitytobeoperatedandmaintainedbyCaltranswithoutsignificantlicensingcosts–Thesystemmustbedesignedwithopensourcecomponentsasmuchaspossible.Itisnotdesiredtohavesignificantlicensingcostsrequiredforsubsequentdeploymentsforfuturecorridorsoroverlongperiodsoftime.ItisintendedthatCaltranswillbecapableofdeploying,maintaining,andoperatingthisandfuturedeployments.

• Limitedtimetodeploy–Thedeploymentscheduleisaggressive,creatingtheneedtoensuresignificantreusabilitywithinthedesignsothatmultipledatasourcescanbeincorporatedintothesystemwithoutdesigningnewdatapipelinesforeachone.


9

• Constraineddevelopmentresources–Caltransprovidedafixedamountoffundingsothesystemdesignmustnotexceedourresourceswithcomplexityandadditionalfeatures.

Thesecoredesignobjectivesandconstraintsareimplementedwiththefollowingdesignprinciples:

• Clouddevelopment,deployment,andoperations–Inordertoachievemaximumflexibilityinbothsystemdevelopmentandfutureconfigurations,minimizeresourceutilizationindevelopmentanddeployment,andminimizedeploymenttime,thecoresystemisdesignedfor100%cloudoperationswithinanAmazonAWScloudenvironment.

• Thesystemisapilotsystemanditexpectedthataswelearnfromthedeploymentofthesystemchangeswillbeneededandrecommended.

• ThesystemshallbecapableofbeingsupportedbyCaltransinternalresources.• Decisionsproducedintheformofresponseplansshallbebasedoncurrentcorridorinformation

withlimiteddelaysininformationprocessing.• Datamaintainedwithinthecoresystemwillbelimitedtothedatarequiredforsystem

operation.LongtermdatastorageshallbeafunctionofPeMS.• Thesystemwillbedesignedforfuturegrowth,incrementalimprovements,future

transportationsystemchanges,changesintransportationitself,geographicchanges,andnewandupdatedinformationsourcesandneeds.

• Easeofdeployment• Easeofmaintenance• Portabilityofthesystemdesignandcomponentstoothercorridors• Highlydecoupleddesignforflexibility,scalability,redundancy,andreliability• Useavailabledatastandardswheneverpossible• Standardizedexternalinterfacesforeaseofportabilityandinclusionofnewsubsystems• Maintenanceofaseparationofconcernsbetweendatamanagement,decisionsupport,and

controlfunctions.• Designwithstate,regional,andlocallayersforfuturescalabilityandstandardizeddeployment

acrossthestate.


10

4. CORESYSTEMHIGHLEVELDESIGN

ThecoresystemhighleveldesignisillustratedinFigure4-1.Thishighleveldesignprovidesclearseparationbetweenexternalsystemsprovidingdataandreceivingcontrolrequests(green),thedatahubreceivingandprocessinginformationfromthoseexternalsystems(red),decisionsupportprovidingresponseplansforincidents(blue),andthecorridormanagementsystemprovidinguserinterface,responseplanselectionandapproval,andsendingofcontrolrequeststoexternalsystems(purple).

Figure4-1CoreSystemHighLevelDesign

4.1. MAJORCOMPONENTS

AsshowninFigure4-1,theICMCoreSystemiscomposedofthreemajorsubsystems:theDataHub,theDecisionSupportSystem,andtheCorridorManagementsubsystem.Thesesubsystems,plustheexternalfieldelementstheCoreSysteminteractswith,aresummarizedinthefollowingtable(color-codedtomatchFigure4-1)anddescribedindetailinthefollowingpages.

Component Description

DataHub Providesforreceiptandprocessingofallcorridordataandamethodofcommunicationamongthethreesubsystems,usingacommonsetofdatadefinitions.Providesoperationaldatapersistenceandretrieval.

DecisionSupport Providescurrenttrafficstate,responseplandevelopment,andresponseplanperformancepredictiontoprovideoneormorerecommendedresponseplansforincidentsandevents.

CorridorManagement Providestheprimaryuserinterfaceforsystemoperators,amethodforusersto


11

monitorcurrentcorridorandcorridorassetstates,interactwiththeDecisionSupportSystem,review,evaluate,select,andapproveresponseplans,andinteractwithexternalsystems,particularlyforexecutionandmanagementofresponseplans.

FieldElements Representexternalprovidersandconsumersofinformationandrequestsforactionsuchasintersectionsignals,rampmeters,corridorsensing,andotherelements,usuallythroughrespectivetransportationmanagementsystemsandTMCs

ThedesignfortheI-210Pilotwillnothavealayeredgeographicapproach,butthedesignisintendedtoallowsuchanapproachforfutureuse.Therearethreegeographiclayerstothedesign:state,regional,andlocal.TheintentisthattheDataHuboperatesonaregionallevel(Caltransdistrict),withthepotentialforservingmultiplecorridorswithinaregion.DatawouldbeconsolidatedandaggregatedatastatelevelformultipleregionsviaPeMS,tosupportlargerscalearchiving,businessintelligenceanalysis,andcontinuedimprovementofcorridorandICMsystemcapabilitiesatthestatelevel.TheDecisionSupportandCorridorManagementcomponentsoperateonalocallevel,withinasinglecorridor.Withstateconsolidationofdata,regionaldatacommunication,andlocaldecisionsupportandresponseplanexecution,amethodofsharinginformationbetweencorridorsisenabled,whileensuringlocaljurisdictioncontroloftrafficeventandincidentresponse.

4.2. FIELDELEMENTS

GreenelementsinFigure4-1representthevariouscorridorfielddatasourcesandfieldelementcontrols,includingvariousstate,regional,andlocaltransportationsystems,regionaltransportationdatanetworks,andprivateinformationproviders.

• Ontheleftsideofthediagram,theseelementsrepresentthevariousdatasourcesforinformationconsumedandprocessedbytheICMsystem.

• Ontherightside,theserepresentthevariouscontrolinterfacestoexecuteresponseplanelementsselectedbyoperatorsoftheICMsystem.

ThesystemsthatwillbeprovidingthedatawithintheI-210corridorinclude:

Table4-1–FIELDSYSTEMS

Source InformationTypePasadenaTMC* PasadenaintersectionsignalsArcadiaTMC* Arcadiaintersectionsignals

LACountyintersectionsignalsDuarteintersectionsignals

CountyTMC*

MonroviaintersectionsignalsFreewayloopsensing(traffic)Rampmeters

CaltransATMS**

Dynamicmessagesystem


12

IncidentinformationCaltransLCS* LaneclosuresCaltransTSMSS* IntersectionsignalTrailblazerSystem* Dynamicmessagesystemforrerouting

LocalvideoRIITSCaltransvideoEnvironmentalsensingCalPolyODSTransit

HSRLaneClosure* CitylaneclosuresNextBus Goldlinetransit

*Indicatesfieldsystemthatacceptscontrolmessages

**Caltrans’ATMSsystemacceptscontrolmessagesandprovideslimitedICMcoresystemcontrol

Futurefieldsystemsmayincludeprobedataproviders(GPSlocation/speed),additionaltransitinformation,parkinginformation,andothers.

`

4.3. DATAHUB

DatafromeachofthefieldsourcesarereceivedbytheICMDataHub,representedinredinFigure4-1.TheprimaryfunctionsoftheDataHubare:

• Receivedatafromfieldelementsviaexistingcorridortrafficmanagementcentersandregionaldatanetworks:VariousdatareceiversreceivedataandprepareitforprocessingbytheICMsystem.Thesereceiversaregenerallyexpectedtobebuiltforthespecificinterfacesdefinedbyeachfielddatasource,bothintransmissionmethod(RESTorSOAPwebservices,socket-basedstreaming,file-based,orothers)andtheintendedinitialpathrequiredforsystemdataprocessing.ThepreferredmethodofinformationtransferforthesystemistheTrafficManagementDataDictionary(TMDD)standarddevelopedbytheInstituteforTrafficEngineers(ITE),currentlyatversion3.03d(withcertainmodifications).

• Processdatareceivedfromfieldelements:Datafromfieldelementsmustbevalidatedforcompletenessanddataqualitypriortousebydownstreamsystemcomponents.Withsuchavarietyofdatasources,oftenforthesametypeoffieldelements,datamustbetransformedintoacommonsetofdatasemantics.TheDataHubprocessesallincomingdataintoastandardizedformat,TMDDfortransportationassets,GTFSfortransitinformation,orothersdependinguponthedatabeingreceived,withacommonsetofdatadefinitionsaswell(suchasasinglenamingstandardforallstreetswithinthecorridor).However,itiscriticaltonotethatthistransformationintoastandardizedformatforprocessingwithinthedatahub,whileitmaintainsthedatastructureswithinTMDD,doesnotgenerallymaintainanXMLdataformatandinternalcommunicationswithinthedatahub(andtheDSS)donotuseSOAPprotocols.ThedatahubandDSSuseJMSprotocols,eitherwithinanActiveMQorKafkamessagingsystem.


13

Additionalprocessingisalsocompletedwithinthedatahub,eitherforspecificmetrics,calculatedparameters,orpredictiveanalytics.

• Datamessagingandcommunications:Dataprovidedtodownstreamsystems,aswellasinternalsystemcommand,control,andstatusdata—indeed,alldatawithintheICMsystem—ismadeavailableviaaninternaldatabus.Themethodofdatatransportisviadatamessagingtechnologies,namelyActiveMQJMSmessagingorKafkadatamessagingsystems.Thespecificmessagetechnologyisbasedonthetype,size,frequency,andmessagepersistencerequirementsofthedata,dataproducer,anddataconsumers.Exceptionstothesemethodswithinthedatahubarelimited,andaregenerallyusedforlargeblockbulkdataarchiving/datatransportbetweenpersistencestores.InordertoaccommodatemultipleCMSvendors,andtoprovidecommunicationbetweenthedatahub’sKafkamessagingsystemandtheDSSActiveMQmessagingsystem,anApacheCamelbasedinterfacebetweenthedatahubandtheDSSandCMSsystemsisincludedwithinthedatahub.ThisinterfaceprovidestheabilitytoprovideSOAPandRESTbasedinterfacesasnecessary,andtranslationbetweenmessagingsystems.

• Datapersistence:TheDataHubalsoprovidesdatapersistencecapabilities,allowingforpersistenceofrawandprocesseddata,systemcommand/control/status,andothersysteminformationinacentralrepository.Thisdatapersistenceisbrokenintodifferenttimelayers,includinglivereal-timedata,youngdata(0-90days),aggregateddata,andarchiveddata.ItincludesleveragingstateEDWandBIresourcesasacomponentofthedatapersistencecapabilities,specificallyasthesingledatastorefornon-operationaldata(dataolderthan90days).Sincedatapersistencewithinthedatahubislimitedtooperationaluses,andthearchitectureisbasedonapatternofcoreservicesconnectedbymessaging,multipledatapersistencetechnologiesareutilizedspecifictotheneedsofthesystemandthedatabeingstored.Largetimeseriescollectionsofdata,suchassensordata,ismaintainedinaCassandradatabase;largerelationalstructureswithsmallerupdatefrequenciesaremaintainedwithinaPostgresdatabase,andMongoDBisusedwithindatatranslationpipelineswhenmultiplesourcesprovidethesametypeofinformationindifferingformatsandimplementations.NOTE:MongoDBmaybeusedindevelopmentandearlyversionsofthesysteminsteadofPostgresforsomeoftherelationaldatastores.

4.4. DECISIONSUPPORT

TheDecisionSupportSystem,portrayedinblueinFigure4-1,providesthefollowingcapabilities:

• Corridortrafficstatedetermination:TheDecisionSupportSystemprovidescorridortrafficstateestimation,providingbothgeospatialtrafficstateinformationandtrafficstatemetrics.Separatearterialandfreewaytrafficmodelsareusedandmergedtoprovidefullcorridortrafficstateatalltimes.

• Corridortrafficstateprediction:TheDecisionSupportSystemprovidescorridortrafficstatepredictionsforusebycorridoroperatorsandforpredictionofincidentandeventresponseplanperformance.Acommercialsimulationengine(TSSAimsun)isusedtoprovidetraffic


14

predictions.Trafficpredictionsaregeneratedusingthetrafficestimatedstateasaninitialtrafficstate,andcurrentcorridorassetstatefromthedatahub.

• Responseplandevelopmentandevaluation:TheDecisionSupportSystem,uponnotificationofaconfirmedincident,willusearules-basedapproachalongwithtrafficstateestimationandpredictioncapabilitiestodevelop,evaluate,andrankresponseplansforusebycorridoroperators.

4.5. CORRIDORMANAGEMENT

TheCorridorManagementsubsystem,portrayedinpurpleinFigure4-1,shallprovidethefollowingprimarycapabilities:

• Presentationofcorridorstate:Displayofcorridorstate,includingcorridorassets(signals,ramps,cameras,dynamicmessaging,transit,etc.),trafficestimatedstate,trafficvisualization(cameraoutput),humanassets,physicalassets(vehicles,responsecrews).Assetstateincludescurrentoperationalcapabilities,currentoperatingstate,currentfunctionalstate(operating,failed,degradedstates),timeavailability,controllability,etc.

• Controlofcorridorassets:TheCorridorManagementsubsystemprovidesthecapabilitytocontrolcorridorassets,takingresponseplansapprovedforexecutionandexecutingeachoftheindividualresponseplanelementsbysendingcommandstotheappropriatestate,regional,andlocalsystemsandentities.TheCorridorManagementsubsystemdoesnotdirectlycontrolcorridorassets,butonlysendscommandstothestate,regional,orlocalsystemsthatdirectlycommandcorridorassets.

• Presentationofresponseplans:TheCorridorManagementsubsystempresentseachresponseplandevelopedbytheDecisionSupportSystem,displayinginmeaningfulwaysthevariousresponseplanelements,howtheywillchangefromthecurrentcorridorstate,howtheychangeduringresponseplanexecution,whatthepredictedoutcomesareforeachresponseplan,whattheactualoutcomesareforaresponseplan,aswellasthevariousmetricsonhowtheresponseplansareevaluatedandtheireffectivenessmeasured.Thispresentationisexpectedtobeinmultipledomains,includinggeospatial,tabular,comparative,graphical,andothersolution-specificmethods.

• Responseplanlifecyclemanagement:Eachresponseplandevelopedbythesystemhasaspecificlifecyclethatincludes:

1. Initiation2. Development3. Evaluation4. Selection5. Approval

6. Execution7. Monitoring8. Close9. Post-incident/eventanalysis


15

TheCorridorManagementsubsystemprovidesamanagementinterfaceforthislifecycle:

Initiation TheCorridorManagementsubsystempresentsincidentandeventinformationreceivedfromtheDataHubforreview,edit,andconfirmationbythecorridoroperators.

Development TheCorridorManagementsystemprovidesdisplayofstatusinformationreceivedfromtheDecisionSupportSystem(DSS)regardingitsdecisiontodevelopasetofresponseplanstoanincidentorevent,aswellasthedevelopmentofthoseresponseplans.Inaddition,itcansendcommandstotheDecisionSupportSystemtoinitiatespecificDSSfunctions,suchasmockincidentevaluation,manualresponseplandevelopmentorevaluation,orotherfunctionsrequiredfortheICMsystem.

Evaluation TheCorridorManagementsubsystemreceivestheresultsofresponseplanevaluationandtrafficstatepredictionanddisplaysthoseresults,alongwiththeresponseplans,tothecorridoruser.Displayshallincludeindividualplananalysis,aswellasrankingandcomparativeanalysisofresponseplans.

Selection TheCorridorManagementsubsystemprovidestheuserthecapabilitytoselectaspecificresponseplanforsubsequentapprovalandimplementation.

Modification TheCorridorManagementsubsystemshallprovideuserstheabilitytomakeminormodificationstoresponseplans.NOTE:Initialversionsofthesoftwarewillnothavethiscapability.

Approval TheCorridorManagementsubsystemprovidesaselectedresponseplantothecorridorstakeholdersatthestate,regional,andlocallevelsforreviewandapproval.Theseapprovalsshallincludeminormodificationstotheresponseplanandmayincludereevaluationandresubmittalforapproval.

Execution TheCorridorManagementsubsystemisthesolecomponentinthesystemcapableofsendingcommandstothevarioussystemsincontrolofcorridorassetsforindividualresponseplancomponentexecution.Itshallhavedisplaysthatallowcontrolandmonitoringofcorridorassetsviathestate,regional,andlocaltrafficmanagementsystemsandassets.

Monitoring Giventhecapabilitytodisplayassetstate,managecorridorassets,andexecuteresponseplanelements,theCorridorManagementsubsystemshallprovideanintegrateddisplayofresponseplanmonitoringoncetheexecutioncommandsaresenttothefieldsystems.Monitoringshallincludedisplayingwhencommandshavebeenexecutedandarecompleteatthefieldassetlevel,alertinguserswhencommandsorassetshavefailed,identifyinganddisplayingvariancesbetweenexpectedandactualtrafficstate,andtheabilitytotakeactionintheeventofresponseplanelementfailureforanyspecificcorridorasset.

Close TheCorridorManagementsubsystemshallbeabletoreturncorridorassetstotheirnormalstateoncetraffichasreturnedtoanormalstate.

Post-incident/eventanalysis

TheCorridorManagementsubsystemshalldisplayacorridorpost-eventanalysisforeachincidentorevent.TheanalysisshallincludeinformationfromtheCorridorManagementsubsystemitself,especiallywithregardtothecorridorassetresponseplanexecution,responseplanexecutionissuesand


16

failures,trafficanalysis(expectedvs.actual),andothersystemperformancemeasures.Thepost-eventanalysisisprimarilylimitedtoinformationusedbytheICMsystemanditscomponents,aswellasmetricscalculatedpostincident/eventregardingresponseplanandsystemperformance.Itisnotintendedtobeanexhaustiveanalysiswithextensiveadditionalmodelinganalysisand“what-if”scenarios.

• ICMSystemmanagement:TheCorridorManagementsubsystemprovidesmanagementcapabilitiesfortheICMsystem,including:

Rulesmanagement TheCorridorManagementsubsystemprovidesauserinterfaceforthemanagementoftherulesusedwithintherulesengine.Thiswillincludetheabilitytocreatebasicrulesandrulesets,editrulesandrulesets,archiverulesandrulesets,provideconfigurationmanagementforrulesandrulesets,makerulesactive,executerulesandrulesets,andimportrulesandrulesets.Generally,rulesthatcanbeeditedwillbesimplerrulesandrulessets,asmorecomplexrulesarelikelytorequireprofessionaldevelopmenteffortandintegration.NOTE:Theinitialimplementationofthesystemwillnotincludethiscapability.

Responseplanmanagement

TheCorridorManagementsubsystemprovidesuserstheabilitytoupdatetheavailableresponseplansandresponseplanelements.Thisincludesadding,archiving,updating,andactivationordeactivationofresponseplansorresponseplanelementsforusewithinthecorridor.

Systemmonitoring TheCorridorManagementsubsystemprovidesuserstheabilitytomonitortheICMsystemandcomponents.Systemslogs,alerts,statusinformation,controlandexecutionofsystemfunctionsshallbeaccomplishedbytheCorridorManagementsubsystem.

Security TheCorridorManagementsubsystemprovidesuserauthenticationandauthorizationfortheCorridorManagementsubsystemandallfunctionsavailablewithintheCorridorManagementsubsystemthataffectotherICMsystemfunctions,includingDecisionSupportandtheDataHub.ThisdoesnotincludeuserauthenticationandauthorizationofindividualcomponentssuchasCassandra,Postgres,ormessaging;rather,itprovidestheseservicesforthefunctionsthatusethosecomponents.

Configuration TheCorridorManagementsubsystemprovidesmethodsandinterfaceforallowinguserstochangeanyandallconfigurationelementswithintheICMsystem.Aswithsecurity,thisdoesnotincludeconfigurationofindividualsystemcomponentssuchasCassandraandPostgres,butratherforthefunctionsthatusethesecomponents.NOTE:Theinitialimplementationofthesystemwillhaveminimalcapabilitiesforthisfunction.


17

4.6. PRIMARYPROCESSFLOW

Figure4-2providestheprimarysystemworkflowillustratingthebasicintegrationbetweeneachofthesubsystems.Whilethisillustrationisnotintendedtocaptureallofthedetailsoftheinteractionsbetweensystems,andtheprocesslifetimesarenotintendedtobeaccuratewithinthissequencediagram,itdoesprovidethebasicsequenceofeventsandprimarycommunicationchannelsfortheprimaryworkflowwithinthesystem,aresponsetoanincidentonthecorridor.Secondaryactionssuchaspersistence,logging,communicationdialogs,andothersarenotdepictedinthisdiagram.Alsonotethat,asdescribedinFigure4-1,allcommunicationbetweentheDSSandCMSisviatheDataHub’sdatabus,althoughthespecificsofthecommunicationchannelarenotillustratedinFigure4-2.

Figure4-2PrimarySystemIncidentFlow(Subsystem)

Thebasicworkflowdescribedinthisdiagramisasfollows:

• Anincidentoccurswithinthecorridor.Thediagramshowstheincidentinformationbeingprocessedthroughthedatahub.

• TheincidentisconfirmedbyanoperatorwithintheCMSorCaltran’sATMS.Theinitialpilotsystemwillalwaysuseanoperatortoconfirmanincident.


18

• TheincidentinformationispassedtotheDSS’sresponseplandevelopmentcomponent.• Theresponseplandevelopmentcomponentrequestsapredictionoftheimpactoftheincident

onthecorridor.• Thepredictionengineusesthecurrentestimatedtrafficstateandcorridortrafficstateto

initializeandrunapredictionwithnoresponseplanexecution(“donothing”prediction).• Theresponseplandevelopmentcomponentusesthe“donothing”predictionandtherules

enginetodetermineifaresponseplanshouldbedeveloped.Iftheresultisnoresponseplanshouldbedeveloped,executionoftheworkflowwouldendwithnotificationtotheCMSofthatresult.

• Iftheresultisthataresponseplanshouldbedeveloped,theresponseplandevelopmentcomponentagainusestherulesengine,resultsofthe“do-nothing”prediction,currentcorridorstate,andasetoffixedresponseplancomponentstodevelopalimitednumberofresponseplansforevaluation.Itsubmitsthoseresponseplanstothepredictionengine.

• Thepredictionenginerunsapredictionforeachresponseplan,alongwithanother“donothing”prediction”basedoncurrentcorridorconditions.Itprovidestheresultsofthosepredictionstotheresponseplandevelopmentcomponent.

• Theresponseplandevelopmentcomponentusestheresultsofeachprediction,currentcorridorstate,andtherulesenginetoevaluate,rank,andselectarecommendedresponseplan.

• TheresultsofresponseplanpredictionsandevaluationisprovidedtotheCMSforoperatorselection.

• TheCMSprovidestheresultstoanoperator,whoselectstheresponseplantobeimplementedandobtainsapprovalfortheresponseplanimplementation.

• Iftheresponseplanisapproved,theCMSexecutestheresponseplanbysubmittingC2CcommandstotheTMCsandothersystemsrequiredtoexecutecommandstoindividualcorridorassets.

• TheindividualTMCsandothersystemsexecutingtheresponseplancommandsreportthesuccessorfailuretoexecutethedesiredresponseplanelements,andcontinuetoreportcorridorassetstate.

Uponcompletionofthisprimaryworkflow,furtherprocessesinvolvedinresponseplanlifecyclemanagementwilloccur,suchasresponseplanevaluation,responseplangenerationtoadjusttocurrentconditions,responseplancancellation,andresponseplancloseout.

5. DATAHUBDESIGN

Thedatahubhasfiveprimaryroleswithinthesystem:

1. Receiveinformationfromexternalprovidersofcorridorstate2. Processinformationreceivedfromprovidersofcorridorstate,evaluatingthedataforquality,

providingcommonmetricsandanalysisofthedatareceived,conductingpredictiveanalyticstosupportestimationandpredictionwithintheDSS,andstandardizingtheformatandcontentfordownstreamconsumptionbytheDSSandCMSsystems.

3. Persistinformationreceived,resultsofprocessing,andoverallsystemstateandcommandinformation.Persistoperationallyrequiredinformationlocallyandsendtoarchiveolder


19

informationforlongertermstorage.Retrieveinformationstoredupondemand(operationaldataforimmediateretrieval,archiveddataforlater,scheduledretrieval).

4. Secureinformationreceived,processedandstored.5. ProvidedatacommunicationsbetweentheprimarysystemsoftheICMcoresystem.

Inordertofulfilltheseroles,thedatahubprovidesaseriesofdatapipelinestoreceiveinformation,processtheinformation,persisttheinformation,andsecuretheinformation.ThepipelinesuseKafkaandActiveMQmessagingsystemstotransmitinformationthatcanbetappedbypersistenceworkerstopersistinformationtothevariousdatastoreswithinthedatahub,andretrievetheinformationandplacetheinformationbackonthemessagingsystemsfordownstreamconsumption.Externalinterfaces,usingApacheCamel,provideinformationandcommunicationbetweenthedatahub,theDSS,andCMSsystems.

Figure5-1DataHubHighLevelDesign

Thisdesignconsistsofindividualservicesconnectedviamessaging.Itisahighlydecoupleddesignbasedonagroupofindependentservicesconnectedbytwomessagingplatforms.Figure5-1providesageneralizeddiagramthatillustratesthreebasictypesofdatapipelines.Sensordataisgenerallyahighfrequency,largerdatavolumepipelinethathandlesalargernumberofsmallermessages.Theheterogeneousdatapipelinetypehandleslargersizedmessagesatlowermessageprocessingfrequencies.Heterogeneousdataischaracterizedbymultiplesourcesprovidingthesametypeofdatainvaryingdegreesofformatandcontentdifferences.Thehomogeneousdatapipelinesaresimilartotheheterogeneousdatapipelines,butgenerallyarebuiltforasingledatasourcethatprovidesallofa


20

specifictypeofdata.Notethattheheterogeneousandhomogenousdatapipelinesprovidesomedataintothesensordatapipelinewhensensordataispartofthedatareceivedwithinthesepipelines.

Adescriptionofthedifferenttypesofcomponentsisprovidedbelow:

Reader–Readersarethebeginningofthedatapipeline.Theirroleissimple–toconnectandgatherthedatafromadatasourceusingtheprotocolandformatnativetothatdatasource,andtoplacetheinformationwithinadatamessagingchannelfordownstreamprocessing.Ingeneral,itdoeslittleornoprocessingofthedata,withanyprocessinglimitedtosimpleparsingoflargebatchesintoindividualmessages.

Spark–SparkisanApacheSparkinstancethatprovideshighspeed,highvolume,realtimedataprocessingforsensordata.Itsroleistoprovidedatatransformation,dataquality,dataprocessing,andpredictiveanalyticsforsensordata.Itreceivesdatafromreaders,processesthatdata,andplacesthedataintoKafkamessagetopicsforusebydownstreamprocesses.

Cassandra–CassandraisanApacheCassandrainstancethatprovidesstoragefortimeseriesdata,primarilysensingdata.Cassandraprovideshighspeed,highvolume,clusteredstorageforsensingdata.Thisdataislimitedtooperationaldatastorageandisnotintendedasalong-termdatastore.

Postgres–Postgresisanopensourcerelationaldatabaseinstancethatprovidesstorageforrelationalandgeospatialdata,suchasintersectionsignalplans,rampmeterplans,roadnetworks,assetinventories,organizationalinformation.Itisanoperationaldatastoreandnotintendedasalongtermdatastore.

Persistenceworkers–PersistenceworkersprovidebothstorageandretrievalservicesforbothPostgresandCassandradatastores.Persistenceworkerslistentoassignedmessagetopicsandqueues,bothKafkaandActiveMQ,andstoretheinformationontheappropriatedatastore.Requestscanbesenttoapersistenceworkerviaacommandmessagechanneltoretrievedataandplacethatdataonaspecifiedmessagetopicorqueue.

Processors–Processorsareusedinthehomogeneousandheterogeneousdatapipelinestoprocessdatareceivedfromthevariousdatasources.Processorsprovidedatatransformation,quality,anddataprocessingservicesfordatareceivedfromthesesources.

Mongo–MongoisaninstanceofaMongoDBdatabaseandisusedwithintheprocesstotransformdatareceivedonaheterogeneousdatapipeline.ReadersstorethedatareceivedfromasourceintoMongoDB.ProcessorsthanqueryMongofortheinformationrequiredforprocessinganddownstreamprocessesincludingdecisionsupportandcorridormanagementsystems.MongoDBprovidesanidealplatformforstoringinformationretrievedinanativeformatandfastqueryingofthisinformationwithminimalmaintenancerequiredwhensourceformattingischanged.

Datahublogger(DHlogger)–ThedatahubloggerisaninstanceofGreylogthatcollectsandindexesallloggingforeachindividualcomponentwithinthedatahub,andlistenstomessagingtocapturecontrolandprocessloggingmessages.Itsroleistocapturesystemstateanderrormessagesforsystemstatusandtroubleshootingneeds.


21

Datahubcontrolgateway(DHcontrolgateway)–ThedatahubcontrolgatewayisanAPIgatewaythatprovidesroutingandmanagementofcontrolandstatusmessagesforthedatahub.ItsroleistoreceivemessagesroutedfromtheinterfacesfortheDSSandCMS,aswellasinternaldatahubcontrolchannels,androutethosetotheappropriatechannelsforthereceivingprocess.Thisprovidesencapsulationofindividualservicesandensuresthatnewservices,servicechanges,andmessagesystemmodificationscanbeaccomplishedwithoutimpactingothersystems,andiskeytothedecouplingofthesystemservicesfortheentireICMsystem.ThisgatewayusesanimplementationofApacheCamel.

Externalinterface/Datagateway–Theexternalinterface/datagatewayencapsulatesthefunctionsofthedatahub,providingasortofswitchboardforusebytheCMSandDSSsystems.CMSandDSSsystemsarenotrequiredtoknowtheinternalsofthedatahubmessaging,butinsteadrelyontheexternalinterface/datagatewaytoprovideprotocolspecificcommunicationportsthatareappropriatefortheCMSandDSSandthentoroutemessagestoandfromthedatahubforthem.Thisexternalinterface/datagatewayusesApachecameltoprovidebothroutingandtranslationservices.Internaldatahubmessages(KafkaorActiveMQ)aretranslatedintotheprotocolsusedbytheCMSandDSS(ActiveMQ,RESTservices,orSOAPservices).CommonTMDDandGTFSmessageformatsareusedinmostcommunications,sonotranslationofdataformatsisrequiredbythegateway.

PeMS–PeMSisastatesystemthatcurrentlyprovidesstatetransportationdataservices.PeMSwillprovidelongtermstorageforallsystemdata,archivingservices,andreportingservicesfornon-realtimedataandreportingneeds.

BusinessIntelligence–BIserviceswillbeprovidedbytheCaltrans’OBIEEimplementation.

5.1. DATASOURCES

ThefollowingdatasourceshavebeendefinedfortheI-210ICMproject:

Source InformationType

System Vendor Product C2CTMDD?

Pasadena Intersectionsignal

PasadenaTMC

McCain Transparity Planned

Duarte Intersectionsignal

CountyTMC KimleyHorn

KITS Planned

Monrovia Intersectionsignal


KITS Planned

Arcadia Intersectionsignal

ArcadiaTMC Transcore Transuite Planned

CaltransFWTraffic Loopsensing CaltransATMS Parsons Custom N


22

CaltransFWRamps Rampmeters CaltransATMS Parsons Custom N

CaltransFWCMS DMS CaltransATMS Parsons Custom N

CorridorTrailblazer DMS Notyetidentified

Custom Planned

CaltransIntersections Intersectionsignal

TSMSS Transcore Transuite Planned

Bluetoothtraffic1 Traveltime Vendor Iteris/? Unknown

Bluetoothtraffic2 Traveltime CountyTMC ? Unknown

Environmentalsensing Environmental ? CalPoly ODS ?

RIITSTransit Transit ? CalPoly ODS ?

RIITSVideo* Video RIITS

CaltransVideo* Video viaRIITS

CaltransFWLaneclosure Lanestatus LCS

CityLaneclosure Lanestatus StateHSRsystem

StateofCA Custom N

LACounty Intersectionsignal


KITS Planned

CHPCAD Incident CAD manualinputbyoperator

Caltransincident Incident CaltransATMS Parsons Custom Planned

Goldlinetransit Transit NextBus NextBus


23

*Videoisnotpassedthroughorstoredwithinthedatahub.

5.2. DATAPIPELINES

5.2.1. SENSINGDATAPIPELINE

ThesensingdatapipelinedesignisprovidedinFigure5-2.Inthisillustration,thedifferentsensorfeedscanbeseenontheleft.Thisillustrationshowsthereaderreceivingdatafromadatasource,placingthatdataonaKafkatopic,SparkconsumingthatdataandprocessingthedatautilizingtheMLLibandStreamingSparklibraries,andthenplacingtheprocesseddataonaKafkatopic.DecisionSupportandCorridorManagementsystemscanaccessthatprocesseddatadirectlyviathedatahubdatainterfaces.SparkalsostoresresultsontheCassandracluster.Persistenceworkerscanbeusedtoretrieveprocesseddatawhenrequested,anddatathatisstoredinPostgres,suchassensorinventories,isavailabletoSparkifneededforprocessingthatmayrequiresuchdata.

Figure5-2SensingDataPipelineDesign

ThedatasourceslistedinSection5.1thatwillbeprocessedusingthistypeofpipelineinclude:

Source InformationType

System DataType C2CTMDD?

Pasadena Intersectionsignal

PasadenaTMC

SignalSensing Planned


24

Duarte Intersectionsignal

CountyTMC SignalSensing Planned

Monrovia Intersectionsignal


Arcadia Intersectionsignal

ArcadiaTMC SignalSensing Planned

CaltransFWTraffic Loopsensing CaltransATMS

FWsensing N

CaltransFWRamps Rampmeters CaltransATMS

Rampsensing N

CaltransIntersections Intersectionsignal

TSMSS SignalSensing Planned

Bluetoothtraffic1 Traveltime Vendor Traveltime Unknown

Bluetoothtraffic2 Traveltime CountyTMC Traveltime Unknown

RIITSEnvironmentalsensing

Environmental ? EnvironmentalSensing

RIITSTransit Transit ? Transitlocation(future)

LACounty Intersectionsignal


Goldlinetransit Transit NextBus Transitlocation(future)

5.2.2. CALTRANSFREEWAYTRAFFICDATAPIPELINE

TheCaltransFreewaytrafficdatapipelineisasensingdatapipeline.Itconsistsofallofthestandardsensingpipelinecomponents:

• Kafkatopicsfordatatransmission• Readerforretrievingdata• Sparkfordatatransformation,qualitychecks,machinelearning• Cassandraforpersistence


25

Figure5-3illustratesthisdatapipeline.

Figure5-3FreewaySensingDataPipeline(PeMS)

PeMS30secondfilesareaflatformatdatafile.DataisretrievedbythereaderviaFTP.Thereaderutilizesaconfigurableschedulerthatwillpollfordatafiles.Whenoneormoredatafilesisavailable,thereaderprocessesthefilesintimeorder,withtheoldestfilefirst.Eachrowwithinthefilecontainstheinformationforasinglesensoratasingletimestep.Allsensorsforasingletimesteparecontainedwithinthefile.Thetimeofeachreadingislocatedinthefilename.ThereaderparseseachrowofthefileandcreatesasingleKafkamessageforeachsensorandtimestep,appendingthetimesteptotheinformationpassedwithinthemessage.Uponcompletionofthefileprocessing,astatusmessageforthefile,includingthetimestepandtheKafkapositionofeachmessageissenttoastatustopic.

Uponreceivingthestatusmessage,Sparkprocessesthemessages,completingthefollowingsteps:

1. ThemessagesaretransformedintoaTMDDdatastructure(Note:thisisnotaTMDDformat,andKafkamessagesareJSON,notXML,anditisnotpassedviaaSOAPprotocol.However,thedatastructureisidenticaltoaTMDDstructure.)

2. Aninventorycheckiscompletedforeachtimestep.3. Aflowbalancecheckiscompletedforallsensorson4and6hourrollingtimewindowsanda24

hourstatictimewindow.4. Ademandpredictioniscompletedforthenexthourforeachfreewayramp.

SparkstorestheresultofitsprocessinginCassandraandpassesthesensinginformationtotheDH_pems_processedtopic.

5.2.3. HETEROGENEOUSDATAPIPELINE

TheheterogeneousdatapipelinedesignisprovidedinFigure5-4.Inthisillustration,thedifferentintersectionsignalfeedscanbeseenontheleft.

ThisillustrationshowsthereaderreceivingdatafromadatasourceandinsertingthatdatainarawmessageformatintoMongoDB.Thereaderretrievesdatafromthedatasourceusingthedatasource’s


26

nativeprotocolsandformats.ItisdesiredthattheformatbeTMDDformat,andtheprotocolisastandardSOAPprotocol,eitherarequestreceiveorsubscriptionmethodasspecifiedintheTMDDspecification.ThereadertransformsthemessagefromaSOAP/XMLformattoaJSONformatpriortoinsertingintoMongo.

Amessageissentfromthereadertotheprocessortoinformtheprocessoroftheavailabilityofnewdata.

TheprocessorthenqueriesMongotoobtainadesireddocumentwiththespecificattributesrequiredtomakeaTMDDmessagecontainingastandardizedsetofinformationrequiredfordownstreamprocessing.Theprocessoralsoutilizesastandarddictionaryofdatavaluessothatdatafromdifferentsourcesthatmaybedifferentinspecificdefinition,butrepresentthesamevaluearestandardizedfordownstreamconsumption.Forexample,ifonesourceabbreviatesboulevardasBlvd,andaseconddoesnotabbreviatetheword,theprocessorwillstandardizetoasingledefinitionthatcanbeunderstoodbythedownstreamprocesses.Theprocessoralsoconductsdataqualitychecksandanyotherprocessingrequired.

OncetheTMDDtransformation,qualitychecks,andanyprocessingiscompleted,theprocessorconstructsthefinalTMDDstructuredJSONmessageandplacesitontheoutgoingtopic.Persistenceworkerslisteningtothetopicpersisttheinformationforlateranalysisorreplay,andtheCMSandDSSsystemsmayconsumethemessagedirectlyfromtheprocesseddatatopic.

Forintersectionsthatprovideintersectionsensing,theprocessorwillretrievethesensinginformationandsenditviaaseparatetopicforfurtherprocessingwithinthesensing/Sparkstream.

Figure5-4HeterogeneousDataPipeline


27

5.2.4. HOMOGENOUSDATAPIPELINE

ThehomogenousdatapipelinedesignisprovidedinFigure5-5.Inthisillustration,arampmeterinventoryandstatesourceisusedastheexamplefeedontheleft.Thehomogenousdatapipelineisidenticaltotheheterogeneousdatapipelinewithonlyonedifference.Homogenousdatapipelinesareusedwhenthereisonlyasingledatasourceforthetypeofinformationbeingprocessed.Whenthisoccurs,thereisnoneedtostorethedocumentsreceivedina“schema-less”database,sincetherearenotmultipledatastandardsandimplementationstoaccommodate.Becauseofthis,thehomogenousdatasourceremovestheMongoDBinstallationfromthepipeline,andthereaderandprocessordirectlycommunicateforanydatatransformation,qualitychecks,orprocessingrequired.

Figure5-5HomogenousDataPipeline

5.2.5. PIPELINECONTROL

Allpipelines,regardlessofpipelinetype,areprovidedwiththesamebasecontrolset.Thecontrolsavailableforeachpipelineinclude:

1. Startpipelineprocessing2. Stoppipelineprocessing3. Pipelinestatus4. Replaydata

Allpipelinecontrolrequestsarehandledbythedatahubcontrolgateway.Requestsreceivedbythegatewayareroutedtoanappropriateendpoint.

Startpipelineprocessing–thestartpipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:


28

1. Ifthepipelineiscurrentlynotstarted,thereaderwillbeginprocessingdataandinsertingitintothepipeline.Thedataprocessedwillbebasedontheconfigurationofthespecificreader.

2. Ifthepipelineisalreadystarted,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystarted.

3. Ifthepipelinecannotbestartedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereaderorifnoreaderinstanceexists.

4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.

Stoppipelineprocessing–thestoppipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:

1. Ifthepipelineiscurrentlyrunning,thereaderwillstopprocessingdataandwillcompleteinsertionofanycurrentlyprocessingdataintothepipeline.

2. Ifthepipelineisnotcurrentlyrunning,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystopped.

3. Ifthepipelinecannotbestoppedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereader.

4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.

Pipelinestatus–thepipelinestatusrequestisarequestforasubscriptiontopipelinestatusmessagesthatareproducedatalltimesbyeachprocessingpointwithinapipeline,includingreaders,spark,processors,messagingsystems,andworkers.Thisincludestimesforwhichthepipelineisnotcurrentlyrunning,butthecomponentsarerunning(forinstancewhenthepipelineiscurrentlystopped,butavailableforprocessing).Allpipelinecomponentsprovideataminimumaheartbeatindicatingtheyarecurrentlyupandrunning,evenifnodataisbeingprocessed.

Foreachpipelinestatusrequest,thedatahubcontrolgatewaywillprovideachannelforasubscriptiontoakafkatopiccontainingallstatusmessagesforaspecificpipeline.Pipelinestatusrequestsshallalwaysincludewithintheresponsethetopicidentifier,andmostrecentkafkamessageoffset.Requestersmaythenconnecttothekafkachannelprovidedbythedatahubcontrolgatewayandsetupaconsumerforstatusmessages.

Thefollowingresponsesarepossiblewhenrequestingapipelinestatus:

1. Aresponsewithakafkatopicidentifierandaddress,alongwiththecurrentkafkamessageoffsetandcurrenttime.

2. Ifthestatusmessagesarenotcurrentlyavailable,anerrorwillbereturned.3. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.


29

Figure5-6StreamingPrimaryControlLayer

5.2.6. PIPELINESTATUSANDLOGGING

Allpipelinecomponentsprovideloggingandstatusinformation,regardlessofpipelinetypeorcomponent.LoggingincludestheloggingattheEC2orapplicationlevelthatisprovidedtotheEC2logsattheOSlevelandispublishedtotheGrayloginstancewithinthedatahub.StatusincludesstatusmessagespublishedtostatusKafkatopics,againcollectedbythedatahubGrayloginstance.

Loggingshallbeconfigurableandsetoncomponentstartandshallincludeataminimumthefollowinglogginglevels:

1. All2. Debug3. Fatal4. Error5. Warn6. Info

Logginglevelwillbesetuponcomponentinitializationwhenitisstarted.Ingeneral,duringnormaloperation,thefollowinglevelsshallreport–Error,Fatal,Warn,andInfo.TheinformationgeneratedbyloggingshallonlybeavailablethrougheitherinspectionoftheEC2instancewheretheyweregeneratedandviathedatahubGrayloginstance.

StatusinformationisproducedbythecomponentsandpublishedtoapipelinestatusKafkatopic.Thisinformationisavailabletoexternalconsumersviathedatahubcontrolgateway(usingadatahubstatusrequest)orthedatahubGrayloginstance.Datapipelinestatusmessagesshallbeprovidedforthefollowingataminimum:


30

1. Pipelinecomponentheartbeat–astatusmessageindicatingthecomponentisrunningsentataregularinterval.

2. Pipelineerrormessage–astatusmessagesentindicatingthatanerrorhasoccurred.Generally,thiswillincludeFatalorErrorlogmessages.

3. Pipelinewarningmessage–astatusmessagesentprovidingawarningindicatingpotentialissuesinoperation.Generally,thiswillincludeallWarnloggingmessages.

4. Pipelineprocessingmessage–thisincludesmessagesindicatingprocessingofasetofdata,oftenatatimesteporfilelevel.Forinstance,inaTMDDsubscription,aprocessingmessagewouldbesetfromeachpipelinecomponentindicatingwhentheinformationfromamessageresultingfromthesubscriptionhasbeenprocessed.Itisnotguaranteedthatdifferentpipelineprocesseswillprocessinformationatthesamegranularity,however,eachstatusmessagewillprovidethecorridorentityidentifiersandtheoriginalmessageidentifierfromtheoriginatingsystemthatstartedthedataprocessing.

Pipelinecomponentsmayincludeotherstatusmessagesasneeded.

Figure5-7Status/LoggingLayer

5.2.7. CORRIDORMANAGEMENTSYSTEM-DECISIONSUPPORTSYSTEM(CMS-DSS)PIPELINE

TheCorridorManagementSystem–DecisionSupportSystem(CMS-DSS)PipelineprovidesthecommunicationbetweentheDecisionSupportSystem,DataHubandtotheCorridorManagementSystem:

• Trafficestimationresults,includinggeospatialinformationandtrafficmetrics• Trafficprediction,includinggeospatialinformationandtrafficmetrics• Responseplansandresponseplanevaluations• DSSStatus• CMSStatus


31

• DSSControl• Trafficestimationandpredictionscenarioinformation

Inaddition,thepipelineprovidespersistencewithinthedatahubofallDSS–CMScommunications.ThepipelinedoesnotincludeanycommunicationsfromtheCorridorManagementSystemthatdonotinvolvetheDecisionSupportSystem.

AllCMS-DSSpipelinessupportTMDDcompliantSOAPdialogsinaccordancewiththeConnectedCorridorsDataCommunicationSpecification.SOAPendpointsarepresentattheCMSDataHubGatewaytosupporttheseconversationsandareexposedtotheCMS.ThedatafortheseSOAPinterfacesarecommunicatedtotheDSSviatheDataHubanditsKafkaandActiveMQmessagingsystem.AllcommandrequestsandresponsesarealsoprovidedtotheDataHubControlgatewayforanydatahubcontrolactionsrequired.WhileonlytobeprovidedasnecessarytosupportthedifferentCMSvendors,REST,ActiveMQmessaging,andKafkamessagingmayalsobeprovidedasinterfacestotheCMSinsteadofSOAPinterfaces.

AllCMS-DSSpipelinesarealwaysonpipelines.Forfutureuse,datahubswillsupportmultipleCMSandDSSinstances,soallindividualpipelinesarecreatedforeachCMS/DSSpair.AllmessagesbetweenCMSandDSSinstancesshallbecommunicatedthroughonlypipelinesdesignatedforthatspecificCMS-DSSpairandshallincludewithinthemessagesthesourceandtargetsystemidentifiers.

Figure5-8providesanillustrationoftheCMS–DSSpipeline,anditsassociatedprimarydatatopics,including:

• Incidents• ResponsePlans• DecisionResults• StatusRequest• StatusResponse• EstimationRequest• EstimationMetrics• EstimationResults• EstimationNetwork• EstimationScenario• PredictionRequest• PredictionResult• PredictionMetrics• PredictionNetwork• PredictionScenario


32

Figure5-8CorridorManagement-DecisionSupportSystemPipeline

5.2.7.1. Incident

TheincidentpipelineprovidesconfirmedincidentinformationfromtheCMStotheDSS.ThispipelinemaystartasaRESTrequest,Kafkamessage,ActiveMQmessage,orSOAPincident.AlsonotethatthispipelinerepresentsaTMDDeventclassdialogsetwithbothrequests,responses,andsubscriptions.Subscriptionsarethepreferredmethodofcommunication.TheincidentpipelineincludesanincidentsubscriptionrequesttotheCMS,aswellasthecorrespondingsubscriptionresponses.IncidentrequestsareoriginatedfromtheCMSDataGateway.TheresponsesareprovidedtotheDSSviathedatahubasActiveMQmessages.

5.2.7.2. ResponsePlan

TheresponseplanpipelineprovidesresponseplanstotheCMS.AsthereisnoTMDDdialogforresponseplans,thisisaConnectedCorridorscustomcommunication.Responseplansareprovidedasan


33

ActiveMQmessageviatheCMSDataGateway.ResponseplansareprovidedinaccordancewiththeConnectedCorridorsDataSpecification,andincludetheresponseplanelements,event/incidentindexidentifier,andassociatedperformanceindicatorsandmetrics.

5.2.7.3. DecisionResult

ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.

5.2.7.4. StatusRequest

ThestatusrequestisacommunicationchannelforrequestingDSSanddatahubstatus.ItisanActiveMQqueue/message.Requestsmustincludeanidentifierfortherequestingsystem.Aresponsewillincludeanaddressforreceivingstatusmessages.Statusrequestsareessentiallyasubscriptionrequest,withtheresponsesimplyalocationforsubscribingtoacontinuousstreamofstatusmessages.

5.2.7.5. Status

ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.DatahubandCMSstatusmessagesareprovidedonthesametopicspecifictotheCMS-DSSpair.

5.2.7.6. EstimationRequest

TheestimationrequestpipelineprovidesamethodfortheCMStorequestanestimation.Ifanestimationiscurrentlyrunning,theresponsetotherequestshallindicatetheaddresswhereestimationresultsarepublished.Iftheestimationisnotcurrentlyrunning,theDSSshallstartanestimationandtheresponseshallincludeamessageindicatingtheDSSisbeginninganestimation,andtheaddresswhereestimationresultsarepublished.

5.2.7.7. EstimationResult

Theestimationresultpipelineprovidestherawresultsofanestimationrequest,includingforfreewaystheestimateddensity,flow,andspeedforeachnetworklinkwithinthefreewayestimationnetwork,andforarterials,theestimateddensityforeachnetworklinkwithinthearterialestimationnetwork.

5.2.7.8. EstimationMetrics

Theestimationmetricspipelineprovidesestimationmetricresults,includingtraveltimeandcurrentdelayestimatesforthefreewayandarterialnetworks.


34

5.2.7.9. EstimationNetwork

TheestimationnetworkpipelineprovidesthecurrentestimationnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.

5.2.7.10. EstimationScenario

Theestimationscenariopipelineprovidesthecurrentestimationscenariodatastructure(arterialandfreeway).Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.

5.2.7.11. PredictionRequest

ThepredictionrequestpipelineprovidesamethodfortheCMStorequestaprediction.Theresponsetotherequestshallindicatetheaddresswherepredictionresultsarepublished.Intheinitialversionofthesoftware,predictionsshallonlybeprovidedeitherasaresponsetoanincidentaspartoftheDSSprocess,orshallbeprovidedasareplaytoapreviousprediction.

5.2.7.12. PredictionResult

Thepredictionresultpipelineprovidestherawresultsofapredictionrequest.TODO:determinewhatfromAimsunwillbeprovided.

5.2.7.13. PredictionMetrics

Thepredictionmetricspipelineprovidespredictionmetricresults,includedtraveltimeandcurrentdelaypredictions.

5.2.7.14. PredictionNetwork

ThepredictionnetworkpipelineprovidesthecurrentpredictionnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.

5.2.7.15. PredictionScenario

ThepredictionscenariopipelineprovidesthecurrentAimsunpredictionfile.Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.

5.2.8. CMSCOMMANDSTATUSPIPELINE

CommunicationsbetweentheCMSandexternalsystemsorcentersarecapturedbytheDataHub.TheyarealsoprovidedtotheCMScontrolgatewaysothatthedatahubmayrespondtocommandsissuedbytheCMSsystemtoexternalsystems.Figure5-9providesanillustrationoftheCMSCommandStatuspipeline.


35

Figure5-9CMSCommandStatusPipeline

ThisCMSCommandStatuspipelineisasimpleimplementationofanActiveMQtopicthatisprovidedacopyofallCMScommunicationswithanyexternalcenterorsystem.AllsuchcommunicationsareprovidedviaActiveMQmessagecontainingatextrepresentationofanymessagesentorreceived.Messagesarekeyedwiththeexternalsystemorcentercommunicatedwith,aswellaseithersendorreceivetoindicatewhetherthemessagewassenttotheexternalcenterorsystem(send)orreceivedfromtheexternalcenterorsystem(receive)aswellasthetimeofmessagesentorreceipt,asappropriate.AllmessagesaresavedviapersistenceworkertoaPostgresdatabase.TheDSSdatahubcontrolgatewayalsosubscribestothedatahubCMScommandstatustopicandmayrespondwithspecificcontrolactionswithinthedatahub.

Initially,thestoreforcommandswillbeGraylog,notPostgres.Graylogprovidesauserinterfaceandqueryenginethatallowsuserstointerpretandreviewsystemprocessesthroughthecapturedcommandstatusmessages.

5.3. EXTERNALINTERFACE/DATAGATEWAY

Thedatagatewayprovidesanexternalinterfacetothedatahubforthecorridormanagementsystemandthedecisionsupportsystem.Thepurposeofthisdatagatewayis:

• Providefortwo-waycommunicationbetweentheDataHubandtheDecisionSupportSystem• Providefortwo-waycommunicationbetweentheDataHubandtheCorridorManagement

System• EncapsulatethefunctionsofthedatahubfromotherICMcomponentsystems• Provideasecureinterfacetothedatahub• AllowforthechangefromKafkaorActiveMQprotocolsusedwithinthedatahubtoRESTor

SOAPbasedservicesaswellasKafkaorActiveMQthatmayberequiredbyotherICMcomponentsystems

• AllowforaswitchboardlikefunctionalitythatcanbeconfiguredfordifferentCorridorManagementSystemprovidersorDecisionSupportSystems

• Provideformultipleoutputchannelstopublishinformationtomultipledownstreamconsumers


36

TheDataHubDataGatewayusesApacheCameltoprovidetheseservices.TwoprimarydesignpatternsareusedasshowninFigure5-10,firstforcommunicationtoexternalActiveMQbrokers,andsecondtoexternalSOAPorRESTwebservicesendpoints.

Figure5-10DataHubDataGateway–ActiveMQandWebServicesDesignPatterns

EachofthetwodesignsusesApachecamelandimplementsaKafkaconsumertoretrievemessagesfromadefinedKafkatopic.Eachalsoprovidesforaprocessorfortransformationandroutingofmessages.ForpassingdatausinganActiveMQtopicorqueuehostedbyanexternalbroker,Camel’sSedacomponentisusedforasynchronousmessagingandmessageflowregulation.ForpassingdataviaSOAPorRESTwebservices,anappropriateSOAPorRESTendpointisprovidedforcommunicationwithanexternalSOAPorRESTendpoint.

NOTE:AllinternaldatacommunicationswithinthedatahubareconductedviaKafkaorActiveMQdatamessagingandJSONformatting.NativedatacommunicationsareimplementedusingTMDDorotherappropriatestandardformatswhenavailable.WhilethesestandardsmaynotimplementJSONformatting,themessageswithinthedatahubthatcontainthisinformationareJSONformatted,retainingthedatastructuresandrelationshipswithintheapplicablestandard.Whenexternallyexposed,thesedatamessagesareserializedbacktotheappropriatedataformatdictatedbythestandardused(TMDDorother).


37

5.3.1. SUBSCRIPTIONENDPOINTS

Subscribe/Notifypattern(http://www.enterpriseintegrationpatterns.com/patterns/conversation/SubscribeNotify.html)-usealeasetoendifnorenew/continuationmessagereceived?

5.3.2. REQUEST/RESPONSEENDPOINTS

BothActiveMQandKafka–CMSgetsconnectionviacamel–camelmaintainssubscriptiontotopic/queue,ActiveMQusesaLastImageSubscriptionRecoveryPolicysubscriptionrecoverypolicy.Getmostrecentmessage.Kafka–getmostrecentmessage

5.4. DATAHUBCOMMANDGATEWAY

ThedatahubcommandgatewayisaninternaldatahubcomponentthatmanagesallcommandcommunicationandcoordinationactivitiesofthedatahubandorchestratescommunicationandcontrolofallcoreICMsystemcomponents.SincetheDSSandCMSdonotcommunicatedirectly,thedatahuborchestratesworkflowsandcommunicationsbetweenthetwocomponentsandtheinternalcomponentsofthedatahub.Similartothedatahubdatagateway,itusesApacheCameltorouteandprocessCoreICMSystemcommands.Inaddition,itusesaworkflowcomponentdesignedforamicroservicearchitectureusedbyNetflixinitsownproductionenvironmentcalledConductor.Itprovidesthefollowingcapabilities:

• Receiveallcoresystemrequestsforservices• Provideexecutionmanagementofallrequestsfordatahubservices• Provideexecutionmanagementofworkflowprocessingforrequestsinvolvingsimplesinglestep

workflowsorcomplexmulti-stepworkflows• Persistservicerequestsandtheiroutcomes(successorfailure)• Manageinternaloperationsofthedatahub(starting/stoppingservices,restartonfailure,etc)• OrchestrateactionsbetweentheDSS,CMS,anddatahub.

Notethatthedatahubcommandgatewayisaninternaldatahubcomponent.Atnotimearethemessagequeuesandtopicsthatitcommunicateswithdirectlyconnectedtoexternalsystems.Allqueuesandtopicsthatcommunicatewiththiscomponentareinternalqueuesandtopicsconnectedtodatahubcomponentsorthedatahubdatagateway(forcommunicationwithexternalsystems).


38

Figure5-11DataHubCommandGateway

Thedatahubcommandgatewayconsistsofjustafewprimarycomponenttypes.Theseinclude:

• Conductorandrelatedworkflowcomponents• Camelandrelatedroutingandtransformationcomponents• ActiveMQworkflowstatustopic• ActiveMQworkflowtasktopic• Monitor

5.4.1. CONDUCTOR

Conductorisusedfororchestrationoftheindividualpipelinesandbothinternalandexternalrequestsforservices.ConductorisanopensourceprojectdevelopedbyNetflixtomanagesomeofitsownproductionworkflows,particularlywithamicro-servicearchitecture.ThereareanumberofsimilaritiesbetweentheusecasesforwhichConductorisdesignedandtheorchestrationoftheindividualpipelinesandtherequestsforservicesfromtheDSSandCMS,aswellasorchestrationofthecommunicationsbetweenDSSandCMSthatmakeitwellsuitedforuse.

Conductorisusedinan“as-is”configuration“out-of-the-box”.ThedatahubimplementationusestheRESTinterfacesandin-memorydatabasethatcomeswithConductor.NomodificationsaremadetoConductor.Itprovidesworkflowmanagementandcontrol,withworkflowsdescribedinJSONformat.


39

5.4.2. CAMEL

ToadaptConductortothedatahubinterfaces,CamelisusedasafacadetotheActiveMQandKafkadatahubinterfaces.CamelprovidestheinterfacebetweenConductor’snativeRESTinterfaceandthedatahub’sActiveMQandKafkadatabuselements.

5.4.3. ACTIVEMQWORKFLOWSTATUSTOPIC

TheActiveMQworkflowstatustopicisusedtoallowindividualdatahubcomponentstoprovideworkflowstatusmessagestoConductor,viatheCamelinterface.Italsoisusedtoprovidegeneralsystemandcomponentstatusinformationforuseinworkflowmanagement.Thedatahubdatagatewayalsoprovidesworkflowandexternalsystemstatus(CMSandDSS)messagesviatheworkflowstatustopic.TheCMSandDSSsystemsareneverrequired(orallowed)todirectlymessageviatheWorkflowstatustopic,butinstead,theircommunicationsaremanagedbythedatahubdatagateway.

5.4.4. ACTIVEMQWORKFLOWTASKTOPIC

TheActiveMQworkflowtasktopicishowworkflowtasksaredistributedfromConductortotheindividualdatahubcomponents,ortheDSSandCMSsystems(viathedatahubdatagateway).Aswiththeworkflowstatustopic,DSSandCMSsystemsareneverdirectlyallowedtocommunicatewiththeworkflowtasktopic.Instead,thedatahubdatagatewaymanagesthedistributionoftasksandtheresultingresponsesviatheDSSandCMSAPIs.

5.4.5. MONITOR

Themonitorlistenstotheworkflowstatustopicforsystemandcomponentstatusmessages,alwaysprovidingdatahub,DSS,andCMSstateandstatusinformationtotheConductorworkflowmanager.ThisallowsConductortorespondtodegradedsystemstate,eitherbydisablingspecificworkflowsorstartingotherworkflowstorespondtosystemcomponentfailures.

6. DECISIONSUPPORTSYSTEMDESIGN

Thedecisionsupportsystem’s(DSS)primaryrolesinclude:

• Provideresponseplansfollowingreceiptofaconfirmedincident• Evaluateresponseplansandrankthembasedonuserdefinedcriteria• ProvideoneormorerecommendationstocorridoroperatorsandstakeholdersviatheCorridor

ManagementSystem,whethertoimplementaresponseplanandwhichresponseplantoimplementforeachreceivedconfirmedincident

• Evaluateimplementedresponseplaneffectivenessandrecommendnewresponseplanswhenappropriate

• ProvideresponseplanevaluationresultsforcorridoroperatorsreviewviatheCorridorManagementSystem


40

TheDSSaccomplishesthiswiththreeprimarycomponents:

• ResponsePlanManagement• RulesEngine• Modeling

Theresponseplanmanagementcomponentreceivesincidentinformationandcoordinatesthedevelopmentandevaluationofresponseplans.Therulesengineprovidesconfigurablelogic/rulesforthedecisiontocreateresponseplans,responseplancreation,andresponseplanranking.Themodelingcomponentprovidestrafficestimationandpredictioncapabilitiestosupportresponseplancreation,evaluation,andranking.

6.1. DSSHIGHLEVELDESIGN

TheDSS’sthreeprimarycomponentsaredescribedinFigure6-1.TheDSSisoneofthethreeprimaryICMsystemsubsystems,andcommunicateswiththeICMsystemviathedatahub’sdatagateway.

Figure6-1DSSArchitecture

Allcorridorinformationisreceivedfromthedatahubdatagateway.Themodelingandresponseplanmanagementcomponentsreceiveallcorridordataandsystemstatusviaaseriesofdatachannelsavailableonthedatahubdatagateway.


41

AllcommandstotheDSSandrequestsforDSSstatusarereceivedviaasetofcommandchannelsexposedonthedatahubdatagateway.AllcommandrequestsfromentitiesexternaltotheDSSarereceivedbytheresponseplanmanagementcomponent.Requeststothemodelingcomponentareoriginatedbytheresponseplanmanagementcomponent.DirectrequeststothemodelingcomponentfromsystemsexternaltotheDSSarenotallowed,andmustbemanagedviathecommandsexposedbytheresponseplanmanagementcomponent.

AllcommunicationswiththeDSSviatheDataHubDataGatewayaremanagedviaActiveMQmessaging.CommunicationsbetweentheResponsePlanManagementandModelingareenabledviaRESTorActiveMQmessaging.TheresponseplanmanagementandrulesenginearetightlycoupledviatheirjavabasedAPIs.

OutputfromtheDSSconsistsprimarilyofresponseplansandtheirevaluationsfromtheresponseplanmanagementsystem,andestimationandpredictionresultsfromthemodelingsystem.

Figure6-2DecisionSupportSystemHighLevelDesign

Figure6-2providesaslightlymoredetailedviewoftheDSS.

6.2. RESPONSEPLANMANAGEMENT

Responseplanmanagementprovidesfunctionsforthedevelopmentandselectionofresponseplans,includingcoordinationoftheactionsoftheresponseplanmanagement,rulesengine,andthemodelingcomponentsoftheDSS.

Theresponseplanmanagementcomponentisprimarilyaneventbased,asynchronousservicethatlistensforaconfirmedincidentfromthedatahub,anduponreceipt,orchestratesactionsbetweenitself,therulesengine,andmodelingcomponentstodetermineifaresponseplanmightpositivelyimpacttrafficconditionsandifso,developseveralresponseplans,evaluatethem,andrecommenda


42

responseplanforimplementation.Theresponseplanmanagementcomponentalsohasalimitedselectionofmethodsimplementedsothatotherspecificrequestscanbemadeviathedatahub.

Theresponseplanmanagementalsomonitorsthedecisionsupportsystemanditscomponents.IthaslimitedcapabilitiestoperformspecificstartupactivitiesrequiredforDSSoperation,monitoreachDSScomponent,andperformlimitedactionsshouldastartupactivityfail(forinstance,ifthemodelingestimationprocessshouldstop,theresponseplanmanagementcomponentshallattempttorestarttheestimationprocess).

Figure6-3ResponsePlanManagementHighLevelDesign

ThecircledareaofFigure6-3illustratesthefundamentalcomponentsoftheresponseplanmanagementcomponent.

Ontheleftoftheresponseplanmanagementcomponentinthefigureisthedatahubinterface.ThisisanActiveMQbasedinterfacethatsubscribestothedatahubdatainterfaceandprovidesassetinformation,routeinformation,andincidentinformationtotheresponseplanmanagementcomponent.Thisinformationismaintainedinalocaldatastore.ItalsolistensforspecificrequestssenttotheDSSviathedatahub.

Theresponsemanager,uponreceiptofaspecificrequestorincidentmanagesthespecificworkflowsrequiredtorespondtotherequestortoprovidearesponseplanrecommendation.Thisincludesrequestingdecisionsandresultsfromtherulesengine,informationregardingcurrenttrafficstatefromthemodelingsystem,orspecificpredictionsforresponseplansreceivedfromtherulesengine.Theresponseplanmanageralsoassemblesthecompleteresponseplan,andprovidesitasaresponsetothedatahubforstorageanddistributiontotheCMS.


43

6.3. RULESENGINE

TherulesengineisacombinationofjavacomponentsandtheopensourceDroolsrulesenginedesignedtorespondtoalimitednumberofquestionsthatcanbeaskedbytheresponseplanmanager.Theselimitedquestionsinclude:

• Doesthecurrentconfirmedincident,withtheresultsofa“do-nothing”prediction,requiredevelopmentandevaluationofresponseplans?

• Whatresponseplansshouldbeevaluated?Createthoseresponseplansfromalimitedsetofresponseplancomponents.

• Whichresponseplan,giventheresponseplansdevelopedandtheirrespectivepredictions,shouldberecommended,listedinarankedorder?

TherulesengineisaJavalibrarythatisincludedintheresponseplanmanagercomponentwithadefinedapispecifictoansweringthesedefinedquestions,giventhecurrentstateofthecorridorassets,definedlimitswithinasetofuserspecifiedrules(suchas“donotusethisroutebetweenthehoursof3and5pm”),currenttrafficstate,currenttime,andincidentinformation.

Itisintendedthatthefinaluserinterfacefordeveloping,editing,testing,andevaluatingrulesandtheirperformancewillbeintheCMS.However,initially,thiswillbelimitedtorequiringITsupporttoassistwithrulesdevelopment,anduploadingofspreadsheetstoaknownlocationwhererulescanbeaddedorupdated.

6.4. MODELING

Themodelingsystemprovidestwoprimaryservices;trafficstateestimationandtrafficprediction.

6.4.1. MODELINGTECHNIQUES

Threetrafficmodelsareusedwithinthemodelingsystem.Theseinclude:

• AcustomfreewaytrafficestimationmodeldevelopedbyUCBerkeleyusingacelltransmissionmodel(CTM)method.

• AcustomarterialtrafficestimationmodeldevelopedbyUCBerkeleyusinganintersectionqueuemodelmethod.

• AcommercialpredictionmodelusingTSS’sAimsunmodelingsystemandamicro-modelingbehavioralmodel.


44

Figure6-4-ModelingComponentDesign

Thesystemiscomposedofseveralprimarysubsystems–• Datamanagement(inblue)–receivesandprocessestrafficandtrafficinfrastructuredataso

thatitcanbeconsumedbythemodelingsoftware.• Estimationmodelingsystem(inpink)–buildsandrunsestimations,producesoutputof

estimatedtrafficstate.• Predictionmodelingsystem(inorange)–buildsandrunspredictedtrafficstate,inparticular

predictedresponseplanperformance,includingperformancemetrics.• Calculation/metrics(inlightorange)–consumesestimatedtrafficstateandproducescurrent

trafficandnetworkperformancemeasures.• ProjectManager(inbrown)–providesoverallsystemorchestration,cloudscalingandEC2

management,workflow,andexternalRESTinterfaces.Auserinterfaceisavailableforestimationmodeldevelopment.ThisiscurrentlyusedforinternaltestingandQA.

• Persistence–(green/gray)–persiststraffic/trafficinfrastructuredata,modelandmodelbuildingdefinitions,rawandprocessedresults.

CommunicationwithexternalsystemsandthemodelingsystemisviaaRESTAPIexposedbytheProjectManager.DatafeedsfromthedatahubareprovideddataviaActiveMQdatatopics.InternalcomponentcommunicationbetweenmodelingsystemcomponentsisviaActiveMQtopics(data)orqueues(command).


45

Technologystack:• PrimarycomponentsarebuiltusingJavawiththeSpringframework.Hibernateisusedwith

componentsrequiringPostgresaccess.• PersistenceisbuiltwithJavabasedpersistenceworkers(bothforstoreandretrieveoperations)

andPostgresv9.6.2orCassandrav3.10forpersistence.• MessagingviaActiveMQv5.14.5• ProjectManagerRESTserviceshostedonTomcatv8.5.15• LogManagementviaGraylog• AmazonWebServices,includingthefollowingAWSservices

o EC2o RDS(Postgres)o S3o VPCo IAMo Otherservicesareusedforcoderepository,build,anddeploymentservices

Platform/ApplicationServers/OS• MostanyversionofLinuxOSisacceptable,Ubuntu14.04isthecurrentstandard• AmazonWebServices• Tomcatv8.5.15

GeneralArchitecturalApproach:

Theapproachusedhasbeentodevelopspecificapplicationcomponentstargetedtospecifictasks,connectthemtoeachotherviamessagingorRESTservicesdependingupontheirapplicationrole(backendprocessingorservicedelivery),andprovidescalingcomponentsthatalloweachindividualcomponenttodetectload(CPU/threadutilizationanditsfeedingqueuestate)andscalethenumberofinstancesoftheresourcetypebasedondemand.

Primarycomponentdescriptions:

ModelEngine–Therearetwoversionsofthemodelengine.Thefirstisusedforfreewayestimation.ThefreewaymodelengineisthecorecomponentofthesystemthatimplementstheCTMalgorithm.Thisimplementationisasimulationusingauserdefinedroadnetwork,accompanyinginfrastructureelementsincludingrampmetersandsensors,trafficdemand,trafficbehaviorinformation(splitratiosatintersectionsandfreewayramps),androad/trafficcharacteristicsrequiredbytheCTMalgorithm(fundamentaldiagrams).Thesimulationcanbeusedtoprovideestimationsofcurrentstateorpredictionsoffuturestate,butisusedonlyforfreewaycurrenttrafficstatewithintheICMsystem.

Thesecondversionofthemodelengineisusedforarterialestimation.Thisarterialmodelengineusesanalgorithmusingintersectionsensingandsignaltimingandcycleinformationtoproduceanestimatedtrafficstateofthecorridorarterials.

Bothtypesofmodelenginereceivearequestforamodelruncontainingthescenariotoberunandthedesiredconfigurationparameters.Themodelenginethenrunsthemodelandoutputsthelinkstate(trafficdensityandspeed)foreachroadnetworklink.


46

ThemodelengineisamultithreadedJavaapplication,buttheprimaryworkcurrentlyremainssingle-threaded.

Predictionengine–ThepredictionengineisanimplementationofTSS’sAimsunsoftware.ItincludestheAimsunsimulationsoftwareaswellasacomponentthatbuildssimulationmodelsforuseinthepredictionworkflow.Thesimulationbuildcomponentcombinesabasesimulationmodelwithdatafromthedatahubindicatingcurrentcorridorassetstate(signals,ramps,signs,etc),currenttrafficstatefromthefreewayandarterialmodelengines,andtheresponseplansfromtheresponseplanmanagementsystemtocreateasetofoneormoremodelsthatcanbeusedtoevaluatearesponseplan.Italsoprovidesthepredictionmetricsrequiredforpresentationtotheuserandtherulesengineforresponseplanrecommendationandselection.

ProjectManager/ProjectManagerApp-Theprojectmanagerappallowsexternalsystemstointeractwiththemodelingsystem.ProjectmanagerappprovidesRESTservicesandmessagingtoinitiateactionsbytherestofthesystem.TheprojectmanagerappisservedviaaTomcatapplicationserver.

ScenarioBuilder-Thescenariobuilderisusedtocreateascenariotoberun.Itisrarelyused,butcanbeusedwhencorridorinfrastructureelementsaremodified,suchasarampmeter.Itisaback-endcomponentthatreceivesmodelcomponents(roadnetworks,intersectionsignals,rampmeters,trafficdemandandsplitsets,etc)fromtheprojectmanagerappandbuildsamodelsuitableforrunninginthefreewaymodelengine.

Calculators-Thecalculatorsareusedtotakethemodelenginelinkstatedataandprocessitforconsumptioneitherasspecificmetricsforanalysis,orasvisualizationinformationfortheCMS.Eachcalculatorjobissinglethreadedandisdesignedforaspecificcalculation.

PersistenceWorker-Persistenceworkersarecomponentsthataredesignedtoeithertakedatafromaqueueortopic(modelenginedata,model/scenarioinformation,orcalculatoroutput)andpersistitinoneofthedatastores(PostgresorCassandra)or,uponrequestretrievethedatafromoneofthedatastoresandplaceinontheappropriatequeueortopicforconsumptionbydownstreamprocesses.

ReadersandProcessors-Thesecomponentsareusedtoretrieveoracceptpusheddatafromthedatahubandplaceitintoaqueueortopicforconsumption(reader)andtoretrievedatafromaqueueortopicandprocessitforconsumptionbythemodel(processor)orpredictionengine.

connected corridors: i-210 pilot integrated corridor management … · 2019-12-17 · partners for...

Documents