case study: debugging multicast problems from an applications perspective
DESCRIPTION
Case Study: Debugging Multicast Problems from an Applications Perspective. Steven Senger, Ph.D. Dept. of Computer Science University of Wisconsin - La Crosse. HAVnet Project. Parvati Dev, PI, Stanford SUMMIT National Library of Medicine, NGI & SII programs since 1999. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/1.jpg)
Case Study: Debugging Multicast Problems from an Applications
PerspectiveSteven Senger, Ph.D.
Dept. of Computer ScienceUniversity of Wisconsin - La Crosse
![Page 2: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/2.jpg)
HAVnet Project
• Parvati Dev, PI, Stanford SUMMIT• National Library of Medicine, NGI & SII
programs since 1999.• Applications of high-performance networks to
anatomical and surgical education.• http://havnet.stanford.edu• http://visu.uwlax.edu
![Page 3: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/3.jpg)
Immersive Segmentation
![Page 4: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/4.jpg)
Remote Stereo Viewer
![Page 5: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/5.jpg)
Nomadic Anatomy Viewer
![Page 6: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/6.jpg)
Other Apps and Components
• Information Channels– Multicast based announcement/discovery
mechanism.– Supports other app requirements such as
logging.• Access Grid
![Page 7: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/7.jpg)
Testbed
![Page 8: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/8.jpg)
Network/App Monitoring
![Page 9: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/9.jpg)
Potholes Along the Way
• Stanford / CENIC– Multicast setup delay
• WiscNet– Conflict between sender and receiver
• Michigan / Merit– Multicast setup delay– Inbound flow stops after 209 secs
![Page 10: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/10.jpg)
Stanford / CENIC …
• Longstanding problem (observed in ‘01).• Large delays (~15 min) in multicast setup.• Stanford / La Crosse / NLM
– Significant delays except for La Crosse / NLM
• Originally thought to be at Stanford Border and RP.
• 04 hardware/ios upgrades at Stanford.• Situation improved.
![Page 11: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/11.jpg)
Stanford / CENIC …
• Only Michigan to Stanford delayed, ~6 mins. • Oct 04, Phone calls, Stanford, CENIC,
Vendor support, La Crosse. Escalate through 3 layers of vendor support.
• Test/Debug every couple of weeks through March ‘05.
• Identified as MSDP propagation delay related to encap/unencap data received by MSDP.
![Page 12: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/12.jpg)
Stanford / CENIC
• Delay occurred at each CENIC router. • At some point problem had been internally
found and resolved by vendor.• Solution: upgrade OS on CENIC routers.
![Page 13: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/13.jpg)
La Crosse / WiscNet …
• First observed spring 05 using AccessGrid.• La Crosse sender and Stanford receiver OK.• Starting a La Crosse receiver breaks the flow.• WiscNet identified problem router.• Vendor support engaged.• Discovered rpd restart sufficient to fix.• Reoccurs every 2 months.
![Page 14: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/14.jpg)
La Crosse / WiscNet …
• When failing– Upstream interface on router gets set to
unreasonable value.– Sender continues to send data in
encapsulated PIM-register messages.– Router never sends register-stop
messages.
![Page 15: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/15.jpg)
La Crosse / WiscNet
• Problem has survived router chassis upgrade. • No solution as yet.
![Page 16: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/16.jpg)
U. Michigan / Merit …
• Discovered after CENIC problem solved.• Small delay in setup for Michigan to Stanford.• Varies between 0 and 60 sec.• Similar behavior for Milwaukee to Stanford.• Does not appear to be in CENIC?
![Page 17: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/17.jpg)
![Page 18: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/18.jpg)
U. Michigan / Merit …
• Presence of other receivers seems to change the setup delay.
• Merit engaged in isolating problem. • No solution as yet.
![Page 19: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/19.jpg)
U. Michigan / Merit
• Discovered Jan ‘06 using AccessGrid.• Traffic from Stanford to MCBI/Merit starts
correctly but stops after 208 seconds. • When stopped IPLSng shows as pruned.• Merit identified problem with a switch in
Chicago not allowing streams to setup correctly.
• Problem resolved with OS upgrade.
![Page 20: Case Study: Debugging Multicast Problems from an Applications Perspective](https://reader036.vdocument.in/reader036/viewer/2022070502/56814bf5550346895db8e8c0/html5/thumbnails/20.jpg)
Diagnostic Help
• Debugging strategies• Tools• Monitoring