21-jan-2008dren ipv6 update1 dren ipv6 implementation update techs in paradise, 2008 jan 21, 2008...
TRANSCRIPT
21-Jan-2008 DREN IPv6 Update 1
DREN IPv6 Implementation Update
Techs in Paradise, 2008Jan 21, 2008Honolulu, HI
Ron BroersmaDREN Chief Engineer
High Performance Computing Modernization [email protected]
21-Jan-2008 DREN IPv6 Update 2
Background
• WAN provider for the DoD R&D community
• Also serving as DoD’s IPv6 pilot implementation of the DoD CIO June ’08 mandate
• Deploying IPv6 where possible in a production environment– See what works and what’s broken– See what’s missing– Share lessons learned
21-Jan-2008 DREN IPv6 Update 3
Previously• Reported to you previously:
– Most serious problem is lack of IPv6 support in many security products (firewall, IDS, IPS, VPNs, web proxy, etc).
• Major inhibitor to deployment of IPv6 in protected enclaves– Numerous bugs
• hurts deployment and require workarounds– Increased complexities with running dual-stack– Vista missing some v6 support– Some things getting better
• Important fixes in key software components• Some incentives from Asian demands
– The scare over RH0, now deprecated– Router meltdown– Proponents and providers aren’t eating their own
dogfood
21-Jan-2008 DREN IPv6 Update 4
What’s new• Still finding bugs, but they are getting harder
to diagnose• Increased campus deployment efforts• Expanded peering• Testbed downsizing• Tunnel brokers updated• www.v6.dren.net web site restored and
updated– Moved to new server (virtual)– Fedora 7– Some cleanup and restructuring of content
• DHPCv6 interoperability testing• Vendors still aren’t eating their own dogfood
21-Jan-2008 DREN IPv6 Update 5
DREN IPv6 Testbed
• Starting to shut down DRENv6 testbed– 3 (of 9) Core nodes shut down
• ERDC• NAVO• AFRL Kirtland
– Moving Peering from testbed to production network
• SPRINT• UUNET• NTT/Verio• … and others in the works
21-Jan-2008 DREN IPv6 Update 6
DRENv6 “testbed”Logical Topology
Dayton
San Diego
Albuquerque
Wash D.C.
Stennis
Vicksburg
Aberdeen
ATM PVC (OC-3)
tunnel
HICv6
(Hawaii)
GlobalCrossing
HurricaneElectric
LAVAnet
SPRINT
vBNS+
6TAP
SSC CharlestonSSAPAC
SSC San Diego
WCISD
AOL
NRL
ARLWPAFB
ERDC
NAVO
C&W
Cisco
NTTComVerio
AFRLKirtland AFB
Abilene
SD-NAPSDSC
Core Router
“site”
IXP
ISP orBGP Neighbor
FIX-West Abilene
HP
AIX-v6
TIC
JITC
Tunnel broker
21-Jan-2008 DREN IPv6 Update 7
Peering• Upgraded all NIDS at peering locations to insure visibility
of IPv6 traffic (security).• Renewed effort to improve peering
– Production network only had 8 operational as of 2 months ago.
– Move peers from testbed to production• Production network used testbed for transit to most peers.
– Add new peers• UUNET (WAE and HAY)• TWAREN (Seattle, working on Starlight)• SPRINT (MAEwest)• NTT/Verio (vastly improved performance to Japan sites0• NLR (Seattle, MAEwest)• Hurricane Electric (WAE tunnel 1/22/08, MAE west cross connect being worked)• UH (last week)• HIX and LavaNet in the works• USGS (this week)• Qwest (working final draft of agreement)
• Had to adjust policy– Waive requirement to peer at 3 locations– Waive requirement for high bandwidth native peering
21-Jan-2008 DREN IPv6 Update 8
Path from UH to DREN
dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.nettraceroute6 to narf.v6.dren.net (2001:480:10:1050::5) from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets 1 2607:f278:5:3::1 2.873 ms 3.163 ms 5.251 ms 2 2607:f278:0:5::2 19.21 ms 6.222 ms 12.63 ms 3 so-2-1-0.bb1.a.syd.aarnet.net.au 95.954 ms 103.687 ms 102.948 ms 4 ge-0-0-0.bb1.b.syd.aarnet.net.au 107.493 ms 95.859 ms 100.499 ms 5 so-2-1-0.bb1.b.sea.aarnet.net.au 264.709 ms 259.346 ms 259.667 ms 6 dren-1-lo-jmb-706.sttlwa.pacificwave.net 157.101 ms 160.606 ms 170.337 ms 7 so12-0-0-0.sandiego.dren.net 189.137 ms 189.147 ms 186.446 ms 8 narf.v6.dren.net 186.873 ms 190.175 ms 207.715 ms
dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.nettraceroute6 to narf.v6.dren.net (2001:480:10:1050::5) from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets 1 2607:f278:5:3::1 2.927 ms 2.413 ms 4.273 ms 2 2607:f278:0:1::1 9.538 ms 18.885 ms 2.41 ms 3 2001:480:0:c81::1 17.07 ms 2.736 ms 3.02 ms 4 2001:480:0:c2f::1 54.571 ms 57.151 ms 54.373 ms 5 so12-0-0-0.sandiego.dren.net 88.173 ms 90.586 ms 84.086 ms 6 narf.v6.dren.net 84.489 ms 84.001 ms 84.304 ms
Before…
After…
21-Jan-2008 DREN IPv6 Update 9
Recent Frustrations
• Products still not doing IPv6– Juniper NSM - slipped 18 months– Tipping Point - slipped 18 months– Bluecoat web/proxy - still no beta code, although
promised a year ago
• Bugs, bugs, bugs– Netscreen ND is broken– BIND sometimes stops responding when using IPv6
transport– Losing first packet breaks NTP– ping6 annoyance– … and more
21-Jan-2008 DREN IPv6 Update 10
Netscreen ND bug
No. Time Source Destination Protocol Info 1 0.000000 fe80::211:92ff:fe10:ca90 ff02::1:ff00:1 ICMPv6 Neighbor solicitation 2 0.001721 fe80::212:1eff:feaf:9906 fe80::211:92ff:fe10:ca90 ICMPv6 Neighbor advertisement
Internet Control Message Protocol v6 Type: 136 (Neighbor advertisement) Code: 0 Checksum: 0x5e5b [correct] Flags: 0x00000060 0... .... .... .... .... .... .... .... = Not Router .0.. .... .... .... .... .... .... .... = Not Solicited ..0. .... .... .... .... .... .... .... = Not Override Target: 2001:480:822:1f01::1 ICMPv6 Option (Target link-layer address) Type: Target link-layer address (2) Length: 8 Link-layer address: 00:12:1e:af:99:06
21-Jan-2008 DREN IPv6 Update 11
Neighbor Advertisement
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Code | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |R|S|O| Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + Target Address + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options ... +-+-+-+-+-+-+-+-+-+-+-+-
S Solicited flag. When set, the S-bit indicates that the advertisement was sent in response to a Neighbor Solicitation from the Destination address. The S-bit is used as a reachability confirmation for Neighbor Unreachability Detection. It MUST NOT be set in multicast advertisements or in unsolicited unicast advertisements.
21-Jan-2008 DREN IPv6 Update 12
Switch loses first packet
• Foundry BigIron will lose first packet after period of no activity
When the mac address of ins1.sd is not in the mac table of the BigIron....
[[email protected] ~]$ ping6 ins1.sdPING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=0.778 ms64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=2 ttl=64 time=1.87 ms
When the mac address is in the table....
[[email protected] ~]$ ping6 ins1.sdPING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=0 ttl=64 time=0.789 ms64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=4.85 ms
Notice the first time that it loses sequence 0.
21-Jan-2008 DREN IPv6 Update 13
Losing first packet
First there is the IPv6 neighbor discovery, which makes it through and is answered because it is sent to a ethernet multicast address..
17:57:26.913312 00:0b:db:93:b5:73 > 33:33:ff:00:00:02, 2001:480:10:4::3 > ff02::1:ff00:2: icmp6: neighbor sol: who has 2001:480:10:4::217:57:26.915433 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3: icmp6: neighbor adv: tgt is 2001:480:10:4::2
By now the mac address (00:0b:db:93:b5:85) should be in the mac table (but I think it isn't yet, due to some internal delay). The next packet is the one that never makes it through the BigIron...
17:57:26.915457 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 0
That echo request is never replied to. But, the next one makes it OK...
17:57:27.912144 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 117:57:27.912901 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3: icmp6: echo reply seq 1
Is it possible that 24 microseconds (between the NA and the echo request packet) isn't enough time for the BigIron to get the mac table updated?
21-Jan-2008 DREN IPv6 Update 14
Losing first packet: impact
• NTP fails– Normally backs off to 1024 second updates– Single UDP packet sent on update– Client never sees packet, declares server
down, and restart at 64 seconds, backs off, then fails again
21-Jan-2008 DREN IPv6 Update 15
ping6 annoyance
• ping6 does a DNS lookup on EVERY packet– Bug in addrconf patch– The value was never
copied to the cache
--- iputils/ping.c.addrcache 2002-09-20 17:08:11.000000000 +0200 +++ iputils/ping.c 2003-05-15 16:41:19.000000000 +0200 @@ -1124,6 +1124,12 @@ { struct hostent *hp; static char buf[4096]; + static __u32 addr_cache = 0; + + if ( addr == addr_cache ) + return buf; + + addr_cache = addr;
if ((options & F_NUMERIC) || !(hp = gethostbyaddr((char *)&addr, 4, AF_INET))) --- iputils/ping6.c.addrcache 2002-09-20 17:08:11.000000000 +0200 +++ iputils/ping6.c 2003-05-15 16:41:19.000000000 +0200 @@ -893,7 +893,14 @@ */ char * pr_addr(struct in6_addr *addr) { - struct hostent *hp = NULL; + static struct hostent *hp = NULL; + static struct in6_addr addr_cache = {{{0,0,0,0}}}; + + if ( addr->s6_addr32[0] == addr_cache.s6_addr32[0] && + addr->s6_addr32[1] == addr_cache.s6_addr32[1] && + addr->s6_addr32[2] == addr_cache.s6_addr32[2] && + addr->s6_addr32[3] == addr_cache.s6_addr32[3] ) + return hp ? hp->h_name : pr_addr_n(addr);
if (!(options&F_NUMERIC)) hp = gethostbyaddr((__u8*)addr, sizeof(struct in6_addr), AF_INET6);
21-Jan-2008 DREN IPv6 Update 16
Expanded deployment• Many campus LANs now 100% IPv6 enabled• One site has achieved dual-stacking ALL desktops
and servers (except 2)– Others have active programs to do the same, along with
getting help desk trained and tooled for supporting it.
• Remove “A” record from DNS for servers– Possible when all clients for that server are dual-stack
• Servers running v6-only (no IPv4 address on any interface, nor on network it is connected to)– Fedora 7 - some manual fiddling of the config is
required, but works– DNS issues (BIND sometimes stops responding when
doing v6 transport queries)
21-Jan-2008 DREN IPv6 Update 17
Fedora 7 IPv6-only
• Connect to IPv6-only LAN, so no cheating.
• Configure from GUI, like any point-and-click sysadmin would do.
• Getting it to actually work…– Manual hacking of ifcfg-eth0
• Delete IPV6ADDR and IPV6PREFIX• Set ONBOOT = yes
– Fix broken /etc/hosts– Configure sendmail to listen on ::1– Fix /etc/yum.repos.d/* files to point to an
IPv6-capable mirror
21-Jan-2008 DREN IPv6 Update 18
IPv6-enablement milestones
and scoring (proposed)1. IPv6 address allocation and address/routing plan developed2. LAN (wired and wireless) is fully v6-enabled (all routers do v6 on all interfaces) and is
connected to the IPv6 Internet (WAN).– The implication is that any v6-enabled device can connect anywhere in the LAN and get IPv6 Internet
connectivity. – (worry about routing implementation, make sure address plan is right, think about security and
performance, play with multicast, make sure network staff is trained to support it, etc)3. Internal infrastructure services (DNS, NTP, DHCP, SMTP, etc) are IPv6-enabled4. Public facing services (public DNS, MXs, public web site) are IPv6-enabled5. Network management infrastructure (management LAN, SNMP, SYSLOG, MIBs, management
access to switches/routers/etc) is IPv6-enabled6. Most workstations and servers are v6-enabled
– (behind this is the support infrastructure, i.e. help desk and other IT support, and a message to sys admins to turn on IPv6 where possible, and servers have AAAA records in DNS, and your help desk tools/scripts for things like host registration and update are upgraded to handle IPv6 addresses
7. Once critical mass is reached on the client side, remove "A" records for servers (resulting in final incentive and some pain for those that didn't dual-stack their workstations).
– You don't need to do them all at once, just one at a time as their clients all become dual-stack8. Migrate some network segments to IPv6-only, with no IPv4 addresses anywhere
– (this forces one to figure out how to make hosts operate in a v6-only environment, helps the organization figure out what's missing, forces the implementation of some kind of transition mechanism to bridge to the IPv4-only world, plus adds continued incentive to get more stragglers upgraded since they can't even get there by typing the IP address
9. Deploy advanced features (mobile-ip, v6 multicast, etc.), provide remote IPv6 access (travellers, telecommuters, home, etc) from v4-only environments, cleanup and reduce complexity (readdressing network if necessary), ....
10. Declare victory• you claim a perfect score in public, and are willing to stand up to scrutiny
Scoring: Scale of 0 to 10. 1 point for any milestone that is 95% complete.
21-Jan-2008 DREN IPv6 Update 19
Commitment to IPv6?
• Are network product vendors really committed to IPv6 support?
• Are they using it in their production networks?
• Do they have an IPv6 presence on the Internet?
• Do they follow the “eat your own dogfood” principle?
• A survey…
21-Jan-2008 DREN IPv6 Update 20
Vendor scorecard
• Looked in DNS to see if there were AAAA records for www, MX, and DNS.
• Quick sampling of major computer and network companies showed no public facing IPv6.
• 6 months ago, and then today.
Jan 2008
June 2007
21-Jan-2008 DREN IPv6 Update 21
Scorecard – IPv6 Summit Sponsors (March ’07)
• Grand and Gold Sponsors of 2007 IPv6 Summit.
• Only one has an IPv6 presence at their corporate “front door”
21-Jan-2008 DREN IPv6 Update 22
Summary: Situation Today
• DREN has been successfully using IPv6 in a production environment, with many dual-stack systems and services, for years– Modern operating systems just work, out of the box
(MacOSX, Linux, Vista, Solaris 10, etc)
• Most urgent needs from our perspective:– Need parity with IPv4 in all implementations– Enabling IPv6 must NOT break things– Need to make security stacks fully IPv6 capable
• Firewalls, IDS, proxies, IDP/IPS, ACLs– Eat your own dogfood
• The rest of DoD will not make the June ’08 deadline.
• The US is falling far behind Asia and other regions.– Prevalent attitude: IPv6 just another GOSIP, or ADA, or ….– A call to action
21-Jan-2008 DREN IPv6 Update 23
END