netgrove: new tools for network optimization

NetGrove: New Tools for Network OptimizationTomas VykrutaSoftware Development EngineerXNA Publishing PlatformMicrosoft

Speed of Light Still Constant

Latency• Speed of light = 186,282 miles per second• Seattle to L.A.

• 960 miles = 5 milliseconds• Seattle to England

• 4,799 miles = 26 milliseconds• Speed of light in fiber or copper slows to 60%• Each router adds 5 to 50 ms• DSL or cable modem adds 10 ms

Bandwidth on the Rise

Thanks, Bungie!

Also on the Rise• Player expectations

• More players in a session together• More voice, video, and background

downloads• Network-using features in your game,

such as data-feeds and spectator modes

• Network-using features in the platform

• Other home devices consuming bandwidth

Building a successful title• Optimizing game networking is still

very important!• Have a Plan:

1. Define max bandwidth goal2. Monitor bandwidth during development

**DO NOT LEAVE UNTIL SUBMISSION**

Defining Maximum Bandwidth• Host = worst case scenario• Let’s target top 80% of users

• 80% have 175ms or better latency• And 176kb/s down, 226kb/s up or better

bandwidth• From “The Real World: Console Latency

and Bandwidth on Xbox LIVE” white paper on xds.xbox.com

• Subtract reservation (i.e. voice chat)• Divide remaining bandwidth by “max # of

players – 1”

Bandwidth Calculation

HOST

CLIENT

CLIENT

CLIENT

CLIENT

56kbps

14kbps

14kbps

14kbps

14kbps

17kbps

17kbps

17kbps

17kbps

5 playergame

68kb

ps

Max Bandwidth cont’d• Scenario: theoretical 16 player game with

30 kbps reservation for voice chat• Host is updating 15 players

• And receiving data from 15 players• Max Bandwidth = 9 kbps down, 13 kbps

up per client max

• Also simulate 175 ms latency for playtests• Monitor regularly!

Max Bandwidth Worksheet Host Bandwidth 175 kbps   Host Bandwidth 226 kbps Reservation 30 kbps  Reservation 30 kbps Effective 145 kbps  Effective 196 kbps

Players Maximum Kbps Players Maximum kbps2 145.00 2 196.004 48.33 4 65.338 20.71 8 28.00

12 13.18 12 17.8216 9.67 16 13.0720 7.63 20 10.3224 6.30 24 8.5228 5.37 28 7.2632 4.68 32 6.3264 2.30 64 3.11

Bandwidth Data 2007-2008Table 1: 2007 Latencies Table 2: 2007 BandwidthPercentage of Consoles with 

this Latency or LessRound Trip Latency Between 

Consoles (ms)

10% 3220% 4530% 5740% 7150% 84 (median)60% 10270% 13080% 17590% 25095% 32097% 38098% 43099% 54099.5% 64099.9% 770

Percentage of Consoles with this Bandwidth or Less

Downstream Bandwidth Between 

Consoles (kb/s)

Upstream Bandwidth Between Consoles (kb/s)

0.1% 20 240.5% 40 441% 64 642% 72 803% 80 885% 104 12010% 144 17620% 176 22430% 200 25640% 248 28050% 336 (median) 352 (median)60% 480 52870% 904 92080% 1760 222490% 5184 5544

Latency and Bandwidth Data• 2008 bandwidth stats same as 2007• Median latency 84 ms• Only 10% of consoles are 32 ms or less• Worst case latency 500 ms• 336 kb/s median downstream bandwidth• 352 kb/s median upstream bandwidth• Typical packet loss 2%• Worst case packet loss 10%

The Real World• From real-world title network reviews

• Best case = 4 kbps per client in AAA FPS title

• Worst case = 370 kbps+ for the host in a 4 player game of a different AAA title!

From a real network review• “Stopped moving” message sent every frame• Many invisible objects simulated• Entire animation state sent (100 bytes+)• Uncompressed data• Position + Orientation + LookAt sent each frame =

12 + 36 + 36 bytes = 84 bytes• Host not coalescing data• NPCs simulated like players• Many more…• End result after optimizations = 5 kbps per client!

Run-time Bandwidth Monitoring• Fps, Memory Usage, why not peak

bandwidth?• Display on screen, always present• Visual alert to indicate packet loss

Tools• Instrumentation in game code• Subjective playtest experience• Packet sniffing

• Tried and true• Black box and low barrier to entry• Many sophisticated sniffers available

Network Monitor 3.2• Process Tracking to identify rogue applications• New engine for improved capture rate in high-

speed networks• Find conversations to view TCP streams, HTTP

flows etc. • Extensive parsers for over 300 protocols! • Network Monitor API: Create your own

applications that capture, parse, and analyze network traffic

Network Monitor 3.2

Quick NetMon Tour• Conversations• Find frame (CTRL+F)• Display and capture filters

• Declarative language with Intellisense• Color filters• Ex: XSP (by itself) filters down to Xbox LIVE

traffic• NMCAP for command-line captures

NetMon Protocols• Extensive script-based protocol

parser language• Documented in NetMon\Help\ParserLanguage.doc

• Create your own!• Trade with your friends! • Sure would be nice if the LIVE

protocol…

LIVE Secure Packet Parsing• Xbox 360 SDK and Games for Windows – LIVE SDK

both provide a NetMon Parser• XSP.NPL

• Automatically installs with XDK• Encrypted data is still encrypted

• XNET_STARTUP_DISABLE_PEER_ENCRYPTION to send unencrypted data while debugging

• XSP parser + NetMon filtering + no encryption =

see what’s actually happening on the wire

DISABLING ENCRYPTION• XNET_STARTUP_BYPASS_SECURITY added in

November 2008• Disables encryption, view unencrypted XSP

data in NetGrove and NetMon• Flag must be enabled on all consoles

• Silently disabled on RETAIL hardware• Stop using XNET_STARTUP_BYPASS_SECURITY

• Disables authentication and XSP completely

LIVE Server Traffic

Traffic to LIVE Statistics / Profile Server

Traffic to LIVE Presence Server

PIX – System MonitorPIX providescounters for LIVE servertraffic Traffic to LIVE Statistics

Server

So What Do You Do with This?• Debug pathological network conditions

• Excessive or unexpected LIVE traffic (TCR)• Mapping LIVE API calls to actual network traffic

• Optimize and tune?• Not really• No great analysis features• Easy to lose serious problems in the weeds

Enter the NetGrove• Ships in the XDK as of March 2008

• Coming soon for Games for Windows – LIVE

• Uses capture files from NetMon 2.0+• Sophisticated analysis of capture files• Flexible and functional GUI• Warning system

NetGrove Key Features• Interactive network topology view• Packet traffic view• Enhanced packet information display• Graphical analysis of packet statistics• Text-based output and command-line

usage

NetGrove Window

Network Topology View

Packet Traffic View

Packet Information

Graphical Analysis of Packet Statistics

Warnings Window

Text-based output

But wait, there’s more!• Includes built-in tutorial and sample

capture file

So What Do You Do with This?• Seriously!• First• Some• Discussion• About• Network• Optimization• (An aside in many parts)

The Story of Goldilocks• The Goldilocks principle of packet size

• 50 < Payload Size < 1264• Coalesce!

• The Goldilocks principle of frequency• This frequency is JUST RIGHT: 15–20

packets/s• Varies slightly by game type

Other Naïve Optimizations• Avoid XSP padding!

• This includes using port 1000 • Avoid struct padding!

• VDP always, UDP rarely, TCP almost never

• Don’t send packets you don’t need to send• For example, don’t send muted voice

Slightly More Complex Choices• Do not lock send rate to framerate• Use peer-to-peer networking when

feasible• Don’t need authoritative server• Can’t rely on one server to take high CPU

and network bandwidth load• Definitely send voice peer-to-peer

Voice and Video• Always use peer-to-peer networking• Don’t send muted voice• Partition traffic by

• Proximity to player• Team channel• “Importance” rating

Compressing Data• Quantize data• Pack Booleans as bit flags• Use polar coordinates (2 parameters) rather

than 3D position (3 parameters)• Use random number seed to send

simulation state• Do not send strings!• For larger chunks, use lossless compression

Modern Implementations• Delta compression• Client-side prediction• Lag correction and interpolation• Contextual bandwidth scaling• Separate guaranteed and non-

guaranteed data

Sequential Packet Delivery• Use with great caution• Separate guaranteed and non-guaranteed

data into different packets and queues• Can cause serious lag even under ideal

network conditions when a packet is dropped

• Also applies to in-house reliable packet delivery protocols

HOST

Sequential Packet Delivery

CLIENT

PACKET 0PACKET 1PACKET 2PACKET 3PACKET 4PACKET 5

STALL

DROPPED

PACKET 1TIMEOUT

REQUESTRESEND PACKET 1

PACKET 1 (R)

GAME

Setup Diagram`

Development PCRunning NetMon

Hub

`

GFWL client

*Series of Tubes

*

Living With Latency• Embrace quantum uncertainty

• You can never know where an object is• Only where it used to be• ... and how fast it was moving

• The current state is a probability field• Is the cat alive or dead?

• Each player has their own parallel universe• Our goal is to keep the universes similar

• No red pill

Prediction Relativity• Remember the parallel universes• Send information relative to the

recipient• “Fired toward position (x, y, z)”

• What if players are in different places?• “Shot at Shawn, missed 10° to the

left”• Robust even if player positions differ

Prediction Paradox• Larger packets = less bandwidth• Position + velocity + controller input• More data makes prediction work

better• Allows lower packet send rate• Fewer packet headers

Setup• Download and install NetMon 3.2• Run NetMon as Administrator• Enable P-Mode• Use a hub! Switches and routers do

not work.

Profiling Your Title• Start with a 2 player capture, then

gradually add more players• Capture with voice and/or video

enabled• Capture lobby, eliminate redundant

traffic• Capture gameplay when all players

are standing still, look for redundant packets

• Capture worst-case scenario (heavy battle)

Automated Monitoring• If you already have an automated

nightly runtime “monkey” test• NetGrove.exe -report Results.txt

mygame.cap• Generates detailed report (text

format):

Outbound Inbound Voice Out Voice InSystem name pkt/s kbit/s pkt/s kbit/s kbit/s kbit/s-------------------------------------- ----- ------ ----- ------ --------- --------192.168.1.3 31.3 32.7 36.9 66.6 0.0 0.0192.168.1.2 37.0 66.7 31.3 32.7 0.0 0.0

NetGrove in Action

Real World Fail Case• Title is ready for final cert• Was only tested with ideal network

conditions throughout development• Tested with 100 ms+ latency for first

time• Terrible lag causing random deaths,

bad user experience• Too late to fix = negative reviews

Common Pitfalls• Common pitfalls from real title reviews

• Idle network chatter• Uncompressed data• Unused data• Complex state• High update frequency

(10–15 Hz is sufficient)• Network simulating invisible objects• Lack of prediction• Bugs

Resources• Network Performance Optimization

white paper• Networking Best Practices white paper• Network Monitor 3.2 Connect Site

© 2009 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

http://www.xna.com