onboard automation with eem
TRANSCRIPT
OnboardAutomationwithEmbeddedEventManager
ShailaSharminSeniorEngineer,Core&IPNetworkBanglalion CommunicationsLTD
Index
• EmbeddedEventManager(EEM)Overview• UnderstandingEEMEventDetectors• EEMPolicieswithsamplescripts• EEMDemos
Introduction:EEM(EmbeddedEventmanager)EEMisasoftwarecomponentofciscoIOS,XR,andNX-OSthatallowsyoutorunascriptorasetofcommandsuponaneventandmakeslifeeasierforadministratorsbytrackingandclassifyingeventsthattakeplaceonarouterandprovidingnotificationoptionsforthoseevents..
Therearetwoindependentpieces:AppletsandScripting->AppletsareacollectionofCLIcommands->ScriptsareactionscodedupinTCL(interpreterlanguage)
EEMCoreEventDetectors&Components
EEMdetectorscanbe1)Syslog2)CLIevents3)Counter4)Timers5)SNMP6)IPSLAandNetflows events.7)None:simply"eventmanagerrun"command.
EEMComponents:EEMserverEEMpublisher(detector)EEMsubscriber(policy)
DeterminingtheVersionofEEMCISCOACCESSROUTERS- CurrentModels
CISCOACCESSROUTERS- OldModels
DeterminingtheVersionofEEMCISCOSERVICEAGGREGATION/COREROUTERS
CISCOCATALYSTSWITCHES
CreatinganEEMapplet
TherearethreestepstocreatingthisEEMapplet.1:CreatetheappletandgiveitanameR6(config)#eventmanagerappletIntf_Management
2:TelltheappletwhattolookoutforR6(config-applet)#eventsyslogpattern"%LINK-5-CHANGED:InterfaceLoopback0,changedstatetoadministrativelydown“
3:Whatdoyouwanttheroutertodowhenitseeswhatyouhavedefined instep2– simple!R6(config-applet)#action1.0clicommand“enable”R6(config-applet)#action2.0clicommand“conf t”R6(config-applet)#action3.0clicommand"interfaceloopback0"R6(config-applet)#action4.0clicommand“noshut”R6(config-applet)#action5.0clicommand"end"R6(config-applet)#action6.0clicommand"who"R6(config-applet)#action7.0mailserver"58.97.254.49"to"[email protected]"from "[email protected]"subject"ISP1_Interface_loopback0SHUTDOWN" body"Currentusers$_cli_result"R6(config-applet)#end
SampleEEMapplet
EEMEventDetector– SyslogED
FunctionalityTriggersEventonMatchesforSyslogMessagesBasedonRegularExpression
Exampleeventsyslogpattern"%LINK-5-CHANGED:InterfaceLoopback0,changedstatetoadministrativelydown“
UseCaseTroubleshooting,AutomaticFaultDetectionandAlert
SyslogEDExampleSyslogmessagesarethemessagesthatshowupbydefaultonconsole.Thisexampleshowsthesyslogeventdetector.Configuration:
SMTPServerisreachable,Loopback0isup.
SyslogEDExamplecontinue..WhentheLoopback0 interfacehasbeenshutdown,thebelowappletautomaticallyrunstoturnontheinterfaceandsendthealerttospecificemailaddressincludingloggedinuserinformation.
Todisplay theEmbedded EventManagereventsthathavebeentriggeredinR1,usethefollowing command:
Samplemailreceivedbynetworkadministrator
EEMEventDetector– CLIED
FunctionalityTriggersSynchronousorAsynchronousEventsWhenCertainCLIIsExecuted.AllowCustomCLICreation(EEM3.0).
Exampleeventclipattern"reload"syncnoskipyesoccurs1
UseCaseConfigManagement,Security,FeatureCustomization
CLIED EDExampleItcantakeactionbasedoncommandsthatareusedontheCLIConfiguration:
Insteadoflookingforapatterninsyslog,thistimewe’rewaitingforapatternenteredontotheCLI.breakdown:eventclipattern:DefinestheeventcriteriatoinitializetheEEMapplet.sync: SpecifiesifthepolicyshouldbeexecutedsynchronouslybeforetheCLIcommandsexecutesskip: IndicatesiftheCLIcommandsshouldbeexecutedoccurs:IndicatesthenumberofoccurrencesbeforetheEEMappletistriggers.
Whenweattempttoreloadtherouter,theresultsareasexpected
EEMEventDetector– InterfaceEDFunctionalityTriggersEventWhenInterfaceCountersCrossThreshold.22CountersSupported,Includinginput_error,interface_reset,transmit_rate,etc.
Exampleeventtagif_1interfacenameFa0/0parameterinput_errors_crc entry-opge entry-val 10entry-typeincrementpoll-interval60
UseCaseRealTimeAlertandRecoveryofInterfaceError
InterfaceEDExampleMonitorCRCerrorsonmultipleWANinterfacesandnotifytheoperator(viae-mail) whenaninterfacehasmorethantwoerrorsperminute.Configuration:
eventmanagerappletmultiple_ifeventtagif_1interfacenameFa0/0parameterinput_errors_crc entry-opge entry-val 2entry-typeincrementpoll-interval60eventtagif_2interfacenameGi01/0parameterinput_errors_crc entry-opge entry-val 2entry-typeincrementpoll-interval60triggercorrelateeventif_1oreventif_2action1.0syslogmsg "CRCfailureleasedline$_interface_name"action2.0mailserver"58.97.254.49"to"[email protected]"from"[email protected]"subject"CRCproblemson
$_info_routername interface$_interface_name"body"CRCfailureshaveexceededthethreshold“
ToviewtheregisteredpoliciesonrouterR1,youcanusethefollowingcommand:
EEMEventDetector– TimerED
FunctionalityTriggersEventsonWatchdog,CountDown,cron andAbsoluteTimer
Exampleeventtimercron cron-entry"019**0-7"eventtimerwatchdogtime300
UseCaseSystemMonitoringviaPeriodicAction,PeriodicDataCollectionandReporting
TimerEDExampleThisappletisputintoCron,whichwillbetriggeredattheexactspecifictime.Itiscomposedof5valuesseparatedbyaspace.Minuteshoursdaymonth{dayofweek(0-6,0isSunday)}Configuration:
EEMEventDetector– SNMPEDFunctionalityTriggersEventBasedonSNMPOIDValueCrossingPredefinedThreshold
Exampleeventsnmp oid " 1.3.6.1.4.1.9.9.109.1.1.1.1.5"get-typeexactentry-opge entry-val50exit-opleexit-val 5poll-interval5
UseCaseSystemStatsMonitoringandAlerting,e.g.CPUandMemoryUtilization
SNMPEDExampleFollowingEEMscriptrunthecommandwhentheCPUgoesaboveacertainvalue.Configuration:eventmanagerapplethighcpueventsnmp oid " 1.3.6.1.4.1.9.9.109.1.1.1.1.5"get-typeexactentry-opge entry-val 50exit-opleexit-val 5poll-interval5action1.0clicommand"enable"action2.0clicommand"showproc cpu sorted"action3.0mailserver"58.97.254.49"to"[email protected]" from"[email protected]" subject "HighCPUAlert"body"$_cli_result"End
ThiswillpollthefivesecondCPUutilizationoftherouteprocessoreveryfiveseconds.Iftheutilizationisatorabove50%,theeventwillfire.TheeventwillnotfireagainuntiltheCPUdropsbelow5%,thengoesbackto50%.Thedefinitionsofvariablesare:highcpu - nameoftheeventmanagerapplet/script1.3.6.1.4.1.9.9.109.1.1.1.1.5 /cpmCPUTotal5min- Objectidentifier(OID)forpollingthetotalCPUutilizationoftherouteprocessor(RP)entry-val 50 - CPUutilizationthattriggersthescriptpoll-interval0.5- Frequency(every0.5seconds)thescriptmonitorstheCPU
EEMEventDetector– IPSLAEDFunctionalityTriggerEventsWhenIPSLATestResultsCrossCertainThreshold.IntegratedwithAutoIPSLAGrouptoMonitorLargeNumberofIPSLAOperationResults
Exampleeventmanagerappletwatch-jittereventipsla operation-id1reaction-type jitterAvgaction001cli command"enable"action002if$_ipsla_measured_threshold_value >$_ipsla_threshold_risingaction003cli command"config t"action004cli command"iproute10.10.20.0255.255.255.0192.168.15.1"action005cli command"end“
UseCaseLinkFailureDetection,DiagnosticsandRecovery
Thedefinitionsofvariablesare:operation-id - SpecifiestheIPSLAsoperationID.
operation-id-value- Numberintherangefrom1to2147483647.
reaction-type- SpecifiesthereactiontobetakenforthespecifiedIPSLAsoperation.
jitterAvg JitterAverageinboth thedirections
EEMTCL-BasedPolicyExample
EEMscriptsarewrittenusingTCL.TCL(ToolControlLanguage)isascriptinglanguageused byCiscofortestingandautomatingofvariousfunctionsintheIOS.Inthisexample,smallTCLscriptconfiguredtocheckreachabilityoffewIPfromtheCorerouter.
EEMDemos
TheProblem: AnEnterprisenetworkconnectedwithtwoISP.WhileloadsharingtrafficwithbothISP,ifonelinktoISPfailsthentrafficshouldshifttoanotherISP.ButNATtranslationsarenotclearingaftertheprimarylinkfails.Whentheprimarylinkrecovers,trafficstillgoingovertheback-up link.
TheSolution:UsingIPSLAandEEMapplettofailoverthetrafficusingNAT.
1. DualISP:NATProblem
Topology
EEMActioneventmanagerappletlink-ISP-1-Downeventsyslogpattern"1ip sla 1reachability Up->Down"action1.0cli command"enable"action1.1cli command"configureterminal"action1.2cli command"noip nat insidesourcelist101interfaceGigabitEthernet1/0overload"action1.3cli command"noip nat insidesourcelist102interfaceGigabitEthernet2/0overload"action1.4cli command"ip nat insidesourcelist100interfaceGigabitEthernet2/0overload"eventmanagerappletlink-ISP-1-UPeventsyslogpattern"1ip sla 1reachability Down->Up"action1.0cli command"enable"action1.1cli command"configureterminal"action1.2cli command"noip nat insidesourcelist100interfaceGigabitEthernet2/0overload"action1.3cli command"ip nat insidesourcelist102interfaceGigabitEthernet2/0overload"action1.4cli command"ip nat insidesourcelist101interfaceGigabitEthernet1/0overload"eventmanagerappletlink-ISP-2-Downeventsyslogpattern"2ip sla 2reachability Up->Down"action1.0cli command"enable"action1.1cli command"configureterminal"action1.2cli command"noip nat insidesourcelist101interfaceGigabitEthernet1/0overload"action1.3cli command"noip nat insidesourcelist102interfaceGigabitEthernet2/0overload"action1.4cli command"ip nat insidesourcelist100interfaceGigabitEthernet1/0overload"eventmanagerappletlink-ISP-2-UPeventsyslogpattern"2ip sla 2reachability Down->Up"action1.0cli command"enable"action1.1cli command"configureterminal"action1.2cli command"noip nat insidesourcelist100interfaceGigabitEthernet1/0overload"action1.3cli command"ip nat insidesourcelist102interfaceGigabitEthernet2/0overload"action1.4cli command"ip nat insidesourcelist101interfaceGigabitEthernet1/0overload"!
Resources
• Support forums for this technology are GREAT• “Living” document at https://supportforums.cisco.com/docs/DOC-12757 Contains helpful tips
and tricks to get the most out of EEM . • For reading material and further resources for this session, visit www.pearson-
books.com/CLMilan2014. • https://networklessons.com/network-management/cisco-ios-embedded-event-manager/. • http://www.techtutsonline.com/cisco-ios-embedded-event-manager/• http://www.cisco.com/c/en/us/td/docs/ios/netmgmt/configuration/guide/12_2sx/nm_12_2sx_bo
ok/nm_eem_overview.html for basic info• http://www.cisco.com/c/en/us/td/docs/ios/netmgmt/configuration/guide/12_2sx/nm_12_2sx_bo
ok/nm_eem_policy_cli.html for Policies Using the Cisco IOS CLI• http://www.cisco.com/c/en/us/td/docs/ios/netmgmt/configuration/guide/12_2sx/nm_12_2sx_bo
ok/nm_eem_policy_tcl.html for Policies Using Tcl
Questions?