improving the ip telephony experience: how to troubleshoot converged networks with voip monitoring...
DESCRIPTION
Watch the full OnDemand Webcast: http://bit.ly/WihOoX IT and Telecom departments are realizing that the performance of the underlying infrastructure is paramount to successful VoIP implementation with Unified Communications (UC). To get the real benefits of UC, your network needs to perform at optimal levels. In this webcast, you will receive a framework that allows you to answer the following questions: How Do You Deal with the Challenges of Jitter, Packet Loss, Echo/Delay, and Voice Signal to Noise? How Do You Balance High-speed, Bursty Data Requirements with Requirements of High Quality Voice Calls? How Do You Create Help Desk Guidelines to Correctly Direct Problems to a Voice or LAN/WAN Subject Matter Expert? How Do You Make Sure that When Adding UC to the Mix that Your Users are Receiving the QoS that They Deserve?TRANSCRIPT
www.wildpackets.com © WildPackets, Inc.
www.wildpackets.com © WildPackets, Inc. WildPackets Seminar Series
© WildPackets, Inc.
Corporate Background
• Experts in network monitoring, analysis, and troubleshooting
Founded: 1990 / Headquarters: Walnut Creek, CA
Offices throughout the US, EMEA, and APAC
• Our customers are leading edge organizations
Mid-market, and enterprise lines of business
Financial, manufacturing, ISPs, major federal agencies,
state and local governments, and universities
Over 7,000 customers / 60+ countries / 80% of Fortune 1,000
• Award-winning solutions that improve network performance
Internet Telephony, Network Magazine, Network Computing Awards
United States Patent 5,787,253 issued July 28, 1998 • Different approach to maintaining availability of network services
WildPackets Seminar Series 3
© WildPackets, Inc.
What We Do
• Provide network visibility and intelligence … WatchPoint, OmniPeek, OmniEngines
• Expert systems – we find the problems for you
• Superior drill-down capability – trouble-shoot from anywhere
• Flexible, customizable, extensible – leverage your investment
Professional services, training, best practices
• For all network segments … Data center to desktop to remote office
LAN, WAN, Wireless …
HTTP, Email, Database, VoIP, Video …
• To … Network engineers; IT Management; Developers
WildPackets Seminar Series 4
© WildPackets, Inc.
With accurate visibility
into the network…
IT staff can improve:
• End-user Productivity
• Network Performance
• Application Performance
• Security
• Compliance
WildPackets Seminar Series 5
© WildPackets, Inc.
Select WildPackets Customers
WildPackets Seminar Series 6
Mid-Market / Enterprise Government & Education
© WildPackets, Inc.
WildPackets Delivers Network Visibility
WildPackets Seminar Series 7
www.wildpackets.com © WildPackets, Inc.
© WildPackets, Inc.
Feel the Rush…
• Your network is
running great!
• Packets enjoy a
speed-limit ride
on the wire!
• Performance is
awesome!
• You have few
complaints from
users!
Are You Dreaming?
© WildPackets, Inc.
Or Feel the Jam!
• Does your network really look more like this…?
CAMP IT Pinpointing the Problem 10
© WildPackets, Inc.
The BIG Questions
• DID YOU (OR WILL YOU) DETERMINE THE HEALTH
OF YOU NETWORK BEFORE DEPLOYING VOIP?
• DID YOU (OR WILL YOU) RID YOUR NETWORK OF
VoIP-KILLING TROUBLES BEFORE INSTALLATION?
• CAN YOU SUCCESSFULLY ASSESS YOUR
NETWORK WHEN VOIP-KILLING TROUBLES ARISE?
© WildPackets, Inc.
Troubleshooting?
• The formal definition of troubleshooting is… “the act of shooting or killing troubles”
CAMP IT Pinpointing the Problem 12
When troubles are small, they can
seem so innocent and harmless,
but…
© WildPackets, Inc.
Troubleshooting?
• The formal definition of troubleshooting is… “the act of shooting or killing troubles”
CAMP IT Pinpointing the Problem 13
You’ve got to kill them when they’re
young, or…
© WildPackets, Inc.
Troubleshooting?
• The formal definition of troubleshooting is… “the act of shooting or killing troubles”
CAMP IT Pinpointing the Problem 14
They will come
back to get you!
© WildPackets, Inc.
Troubleshooting: Not Just Reactive!
• When we use the word troubleshooting, most folks
immediately think about reacting to a problem
• But proactive troubleshooting identifies troubles when they are small and are having minimal impact!
• The concept is simple…
Proactive
Troubleshooting
Reactive
Troubleshooting =
© WildPackets, Inc.
What Troubles Are We Shooting for VoIP?
• Before VoIP, network
troubleshooting focused on
factors like application
response time and latency
• With VoIP, we’ve
learned that latency
is just one part of a
three-headed
monster…
CAMP IT Pinpointing the Problem 16
Jitter Latency
Packet
Loss
The monster attacks RTP with one or more of its weapons!
© WildPackets, Inc.
Identifying Troubles: The First Step
• Before you begin worry about statistics or packets,
take time to listen to representative calls
• Hearing VoIP troubles is the most natural way to recognize them
Use analysis application that can playback call audio • Playback of individual RTP streams
• Playback of complete call
Listen for the telltale of signs of latency, jitter, and packet loss
© WildPackets, Inc.
Understanding the Monster: Latency
• The time it takes for packets to travel across the
network is based on several factors: Distance latency – unavoidable – a fact of physics
Queue latency
Decision latency
Encryption/decryption
Codec operations
© WildPackets, Inc.
Understanding the Monster: Latency
Queue Latency &
Decision Latency
Network
Propagation
Delay
Encoding / Decoding
Compression / Decompression
Jitter Buffer Latency
© WildPackets, Inc.
Latency Tolerance
0 ms
100 ms
200 ms
300 ms
400 ms
500 ms
600 ms
700 ms
800 ms
High Quality
Required for
VoIP
Satellite
Quality
Fax Relay
Broadcast
Quality
The ITU
recommends a
maximum one-way
delay of 150 ms
for VoIP
Roundtrip latency > 250 ms will be
noticeable for call participants!
Latency is much
more critical for
VoIP systems than for
traditional data
applications
© WildPackets, Inc.
Understanding the Monster: Latency
• Excessive latency is a major enemy of VoIP Often caused by network congestion in the absence of adequate
QoS provisions
For some network segments, especially WAN circuits, elevated
latency may be a way of life
Excessive latency may be one-way or roundtrip, depending on
how traffic is routed through the network
© WildPackets, Inc.
Latency's Effects: Talkover
• “Talkover” occurs when excessive latency delays
audio
– Caller A speaks, but his words are slow to reach Caller B
– As a result, Caller B is slow to respond
– Caller A believes that his words were unheard, so he begins to
speak again, often just as Caller B begins his response
– Caller A is speaking as he begins to hear Caller B
– Caller B may still be speaking when Caller A’s second set of
words begin to arrive
• Conversation cadence is not natural or comfortable
• Callers feel as if they must “push to talk” or say
“over” to control the conversation
© WildPackets, Inc.
Latency's Effects: Echo
• In some cases, excessive latency may produce an
echo effect The speaker’s voice feeds back into the listener’s microphone
The speaker then hears his own voice returning from the
listener’s end, but delayed due to latency
• Most callers find it difficult to maintain normal speech when echo delay is prolonged
• Some VoIP systems attempt to cancel echo, but are
not always successful
High latency may also cause additional troubles such as loss of
synchronization between audio and video for multimedia sessions.
© WildPackets, Inc.
Understanding the Monster: Jitter
• Closely related to latency Jitter is really nothing more than variable latency preventing on-
time delivery of RTP packets
Example • G.711 needs an RTP packet to be delivered every 20 ms to provide
accurate audio reconstruction
• If the delta time between one RTP packet and the next is 24 ms,
then the jitter is 4 ms
• VoIP devices employ jitter buffers to smooth packet
delivery Jitter up to about 100 ms may be managed by the buffer
Packets with jitter greater than the jitter buffer are dropped
An large jitter buffer increases latency
© WildPackets, Inc.
Jitter's Effects
• Jitter causes weird “sound effects” that vary with
jitter severity and environmental factors
• Examples include: Static
Stuttering or uneven audio – abnormal speech rhythm
For multimedia systems, video may be “jerky” or irregular
• If jitter levels are high, packet loss can result In some cases, severe jitter may sound similar to packet loss,
even if no packets are actually dropped
© WildPackets, Inc.
Measuring Jitter
• Jitter can be measured as instantaneous jitter The difference in actual packet arrival time vs. expected arrival time
• Each packet is the jitter reference for the next packet
Gives a rather “jerky” view of jitter that may overstate its effects • However, it does correctly depict the “jerkiness” of a call
• Smoothed jitter is an improved metric defined in RFC 3550 Applies a “filter” to smooth out the instantaneous jitter trend, which
provides a much more useful and accurate view of jitter over time
For smoothed and instantaneous jitter, minimum, maximum, average,
and standard deviation values are very meaningful.
Graphs of these metrics provide good insight into call quality.
© WildPackets, Inc.
Measuring Jitter
• Absolute jitter uses the first packet in the stream as a
constant reference, which catches “clock skew” Clock skew is the difference in the clocks on the involved VoIP
stations and the measuring device
Absolute jitter graphs can reveal when packets are dropped
due to clock skew between VoIP stations
Calculating maximum, minimum, average, and standard deviation for absolute
jitter is not very helpful because of the clock skew factor.
© WildPackets, Inc.
Understanding the Monster: Packet Loss
• Most commonly caused by: Packet dropped due to physical layer corruption
Congestion without adequate QoS provisions
Jitter buffer discards due to excessive latency
• Causes missing sounds, syllables, words, or phrases DSP algorithms may compensate for up to 30 ms of missing data
More than 30 ms of missing audio (e.g. 2 RTP packets for G.
711) is noticeable by listeners
© WildPackets, Inc.
Packet Loss Effects
• An average person speaks at a rate of about 200
words per minute Do the math – that’s 3.33 words/sec = 300 ms per word
For G.711, we would need to lose 15 consecutive RTP packets
to lose a whole word
Dropping 15 packets/sec for G.711 would be a loss rate of 30%
• But losing only a few packets can still be very noticeable
Loss of more than 2 consecutive packets will be heard
Loss rates 2% will have a strong impact on quality
Losses of 5 – 10% make calls all but intolerable
Bursty periods of packet loss are worse than more dispersed
loss
© WildPackets, Inc.
Packet Loss Bursts
• A packet loss “burst” is a period of time that begins
and ends with loss in which the number of
consecutive received packets is less than the minimum number needed (G
min) to maintain adequate
quality G
min for VoIP = 16
Gmin
for video services 64 - 128
• The more “bursty” the packet loss, the worse the
quality of the call
• VoIP quality scoring standards consider Burst length (ms) of the bursts for a given RTP stream
Burst density (% of missing packets) for the RTP stream
© WildPackets, Inc.
Assessing the Monster's Impact
• While traditional network applications are very tolerant of
jitter, latency, and even some degree of packet loss, VoIP is
very sensitive to these troubles
• Levels of jitter, latency, and packet loss that would be easily
tolerated on a data network can be devastating on a
converged VoIP network
• Pre- and post-deployment network assessment are critical
– You must understand your network’s ability to accommodate VoIP
– Current latency, jitter, and packet loss
– QoS capabilities
– Current bandwidth utilization (is there any room for VoIP)
– You must maintain a constant vigil after deployment to watch for
imminent troubles
© WildPackets, Inc.
Network Traffic: Quantitative Analysis
• Most network folks are concerned about the amount
of traffic on their networks Utilization (percentage of bandwidth)
Throughput (bits or bytes per second)
• You also need to be concerned about individual utilization components
How much bandwidth and throughput can be attributed to each
application or process? • Clarifies which application traffic may need to be tuned or controlled
How well or poorly will the baseline (trended) behavior of each
application interact with VoIP • Don’t forget to also consider the reverse case – VoIP’s impact on
existing applications
© WildPackets, Inc.
The Impact of "Just One More Call"
• Although a network link may be able to support a
number of concurrent calls, one additional call is
often enough to cause quality problems…
x1113
x2111
x1112
x1111
1st Call
2nd Call
3rd Callx2112
x2113
Example: The WAN can support 2 simultaneous calls.
What happens when a third call is attempted???
© WildPackets, Inc.
Network Traffic: Qualitative Analysis
• The quality of your network traffic is potentially more
important than its quantity when it comes to VoIP
• Many traffic streams are “bursty” in nature Burstiness my occur over long period of time, or may consist of
rapid, recurring traffic spikes
Prolonged rises in utilization may decrease the number of calls
that can occur simultaneously
Sharp spikes may cause very noticeable quality issues with
ongoing calls
• Your baseline monitoring should consider not only
averages and long-term trends, but also the short-
term peaks and dips that characterize your traffic
flow
© WildPackets, Inc.
Troubleshooting for Latency,
Jitter, and Packet Loss • Proactive and reactive assessment of VoIP troubles
can be easily accomplished using common utilities
and/or analysis tools: Performance and protocol analysis applications
Network management systems
PING, Traceroute, etc. • Use packet sizes that reflect your actual RTP packets
• For example, 218 byte RTP packets are typical for G.711
For reactive assessment (post-deployment) include VoIP stats,
quality scores, and audio evaluation
© WildPackets, Inc. 36
ping –l 218 –t 206.169.32.70
Pinging 206.169.32.70 with 218 bytes of data:
Reply from 206.169.32.70: bytes=218 time=88ms TTL=57 Reply from 206.169.32.70: bytes=218 time=83ms TTL=57 Reply from 206.169.32.70: bytes=218 time=92ms TTL=57 Reply from 206.169.32.70: bytes=218 time=88ms TTL=57 Reply from 206.169.32.70: bytes=218 time=88ms TTL=57 Reply from 206.169.32.70: bytes=218 time=102ms TTL=57 Reply from 206.169.32.70: bytes=218 time=87ms TTL=57
Watch for
excessive
latency.
Watch for
unsteady
latency (jitter).
Watch for
variable
routing.
Watch for
packet
loss.
Ping statistics for 206.169.32.70: Packets: Sent = 7, Received = 7, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 83ms, Maximum = 102ms, Average = 90ms
C:\>
© WildPackets, Inc.
Traceroute can show the hop-by-
hop components of latency to help
locate the source of troubles.
C:\>tracert 206.169.32.70
Tracing route to 206.169.32.70 over a maximum of 30 hops:
1 1 ms <1 ms <1 ms 10.71.0.1 2 3 ms 4 ms 3 ms 69.46.168.254 3 3 ms 2 ms 2 ms 172.33.37.78 4 4 ms 3 ms 4 ms 66.163.28.157 5 6 ms 2 ms 2 ms 209.167.101.217 6 8 ms 3 ms 2 ms 152.63.133.74 7 15 ms 13 ms 14 ms 152.63.128.117 8 14 ms 14 ms 14 ms 152.63.66.65 9 17 ms 102 ms 68 ms 64.215.195.137 10 17 ms 14 ms 14 ms 67.17.109.118 11 14 ms 15 ms 14 ms 207.200.10.11 12 73 ms 73 ms 72 ms 66.192.254.213 13 87 ms 83 ms 107 ms 206.169.32.70
Trace complete.
C:\>
© WildPackets, Inc.
A Picture is Worth a 1000 Words
• A graph of latency,
jitter, or packet loss
can speak volumes
about network
health, either for
proactive or reactive
troubleshooting
• Overlaying this
graph with a graph
of utilization or total
throughput can
reveal even more
about the causes of
VoIP troubles
© WildPackets, Inc.
Quality Scoring for VoIP
• One of the best initial troubleshooting tools for VoIP traffic
• Mean Opinion Score (MOS) – several flavors Algorithmic simulation of subjective audio assessment Most commonly used varieties are MOS-LQ (listening quality) and MOS-CQ (conversational quality) Possible range of 1 (poor) to 5 (excellent) Maximum possible MOS = 4.4 with G.711 Typical range in most networks is 3.5 – 4.2
• R-Factor – several flavors Based on latency, jitter, packet loss, bit rate, and signal-to-noise ratio, codec effects (for low bit-rate codecs), recency
• The ITU algorithms consider about 20 quality inputs
Possible range of 0 (poor) to 100 (excellent) Provides LQ, CQ, and other score variants
© WildPackets, Inc.
Quality Score Trending
• Isolated scores are useful for validating single call
complaints, but overall VoIP health is best seen by
graphing long-term trends
Overlaying VoIP trends
with network utilization,
errors, or other metrics may
reveal previously unseen
performance relationships!
© WildPackets, Inc.
Got QoS?
• One of the most potent weapons for fighting VoIP troubles is provision of Quality of Service (QoS) parameters
• QoS enables network devices to prioritize and give preference to packet streams that are sensitive to delay, packet loss, jitter, and other performance inhibitors
• Standards-based QoS methods include: RSVP (nearly antiquated) IP Differentiated Services (DiffServ) MAC Layer QoS with IEEE 802.1p VLANs
• QoS may be obtained or supplemented via proprietary means, such as traffic shaping via various flow processing algorithms
© WildPackets, Inc.
QoS the Old-Fashioned Way with RSVP
RSVP is fading away since it manages traffic on a flow-by-flow basis, a method
that is not scalable to enterprise and carrier grade networks.
© WildPackets, Inc.
IP Differentiated Services
• Using DiffServ, routers and other devices no longer have to worry about individual flows that are identified by IP addresses and port numbers
• Instead, devices only need to examine 6 bits to know how to classify and manage traffic
The DiffServ bits provide 64 DSCPs, of which 8 give
backward compatibility with the older IP Type of Service
(ToS) field. Only 32 DSCPs are now in common use.
DiffServ Code
Point (DSCP)
© WildPackets, Inc.
DiffServ at the Data Link Layer
• IEEE’s 802.1p specification enables packet prioritization via
3-bit field in 802.1q VLAN tags
• This field provides 8 levels of precedence, and only requires
switches to read 3 bits to classify and manage traffic
• Each 802.1p-aware switch allocates 8 different queues to
separate handling for each priority level
• Network administrators must map priority levels with handling
methods
VLAN Tag (802.1q)
© WildPackets, Inc.
Ready for QoS?
• QoS provisions are based on the “weakest link”
concept If any device in a data path does not support QoS, then media
streams will not be afforded the preference they require for good
performance
• Pre-deployment assessment must ensure that ALL devices can recognize and respond to QoS
parameters in packet headers Switches, routers, firewalls, proxies, and any other devices that
touch RTP packets must be “VoIP-friendly”
© WildPackets, Inc.
Summary
• The main threats to VoIP are latency, jitter, and packet loss
The presence of these “monsters” may remain unnoticed in a data network, but will become very obvious and ugly in VoIP systems
• Troubleshooting VoIP is both reactive and proactive Prudence dictates that you test your network before installing a VoIP system to identify and correct performance troubles
• You can see the three-headed monster even before VoIP is installed
After VoIP deployment, constant monitoring will • Validate QoS operations
• Reveal network traffic pattern changes that adversely affect VoIP • Provide alerts when VoIP performance declines
• Proactive troubleshooting and monitoring is a way of life – a job that is never done!
www.wildpackets.com © WildPackets, Inc. WildPackets Seminar Series
© WildPackets, Inc.
OmniPeek Enterprise Packet Capture, Decode and Analysis
• 10/100/1000 Ethernet, Wireless, WAN, 10Gbe
• Portable Capture and OmniEngine Console
• VoIP Analysis and Call Playback
OmniEngine / Omnipliance Distributed Enterprise Network Forensics
• Packet Capture and real-time analysis
• Stream-to-Disk with Data Mining
• Integrated OmniAdapter network analysis cards
Product Line Overview
WildPackets Seminar Series 48
© WildPackets, Inc.
OmniPeek Product Family Architecture
• OmniEngines / Omnipliance OmniEngines collect and process
data throughout the network.
• Platform Services Filters, alarms, authentication and
communication.
• Interfaces Packets captured from Ethernet
NICs, 802.11 NICs, OmniAdapter
Gigabit and WAN analyzer cards.
• OmniPeek Analyzers OmniPeek Analyzers perform local
network analysis on a portable basis and function as consoles to
multiple remote engines.
• Intelligent Data Transport OmniPeek analyzes network traffic
at the engine and intelligently
coordinate analysis results that are exchanged with the console for
maximum efficiency.
WildPackets Seminar Series 49
© WildPackets, Inc.
Omnipliance
• Full Line Rate support, Windows or Linux
• Customer upgradable hardware
• Representative Specs 3U, 2x Intel Dual Xeon 3.3 Ghz,
8x 500 Gigabyte SATA or 8x 150 Gigabyte SAS, etc.
Packet filtering in hardware
1U Omnipliance Edge available
WildPackets Seminar Series 50
© WildPackets, Inc.
WatchPoint Architecture
• WatchPoint Server
Web server
Centralized data collector management
• OmniFlow Collector
• NetFlow Collector
• sFlow Collector
• Scalable architecture
• Collectors can be separated for
increased performance
• Published API and SDK
for easy extension
WildPackets Seminar Series 51
© WildPackets, Inc.
WildPackets Key Differentiators
• Visual Expert Intelligence with Intuitive Drill-down
Let computer do the hard work, and return results, real-time
Packet / Payload Visualizers are faster than packet-per-packet diagnostics
Experts and analytics can be memorized and automated
• Automated Capture Analytics
Filters, triggers, scripting and advanced alarming system combine to provide
automated network problem detection 24x7
• Multiple Issue Network Forensics
Can be tracked by one or more people simultaneously
Real-time or post capture
• User-Extensible Platform
Plug-in architecture and SDK
• Aggregated Network Views and Reporting
NetFlow, sFlow, and OmniFlow
WildPackets Seminar Series 52
www.wildpackets.com © WildPackets, Inc.
WildPackets, Inc.
1340 Treat Boulevard, Suite 500
Walnut Creek, CA 94597
(925) 937-3200