simplifying data center migrations using lisp, from 42...

Simplifying Data Center migrations using LISP, from 42 years to 2 years

CCSDCT-1100

Patrice Bellagamba, Distinguished System Engineer

Santiago Freitas, Customer Solutions Architect

© 2014 Cisco and/or its affiliates. All rights reserved. CCSDCT-1100 Cisco Public

Session Objectives •  Explain how a Customer was able to reduce by 95% the migration

window time required to migrate applications from one DC to another.

•  Provide an overview of how the technology used works: LISP enabled on ASR 1000.

•  Cover the benefits of using a Layer 3-based migration technique.

•  Share the Testing Results including Scale, Latency, Convergence Time and Performance.

2


Agenda •  Problem statement

•  The impact of LISP on Data Center Migrations

•  Technical Overview: LISP enabled on ASR 1000

•  Implementation Details

•  Testing Results –  Scale, Latency, Convergence Time and Performance

•  Conclusion

3


DC Migration without IP mobility / IP Retention Application Migration – Moving resources WITH changing IP addressing

4

•  DNS may not always help with legacy hard coded applications

•  Applications local and remote may need to be amended

•  Firewalls need to be amended

§  Takes longer to start moving servers due to data gathering / documentation of legacy application interfaces

§  Risk – has an interface been overlooked?

•  Risk - Can you be sure you have the complete picture?


DC Migration without IP mobility / IP Retention

Application Migration – Moving resources WITHOUT changing IP addressing

5

Without IP Mobility, it requires to move all server of a subnet together Problem with Affinity Groups:

Move requires understanding of server VLAN cross-patching / affinity groups if smaller units of servers to be migrated in one event

Subnet A Subnet B Subnet C


DC Migration – Server Physical Move

Move of Affinity Groups – The Reality

6

Move the server and patch into new infrastructure: §  Takes longer to start moving servers due to data gathering and

understanding of virtual server network interfaces. §  Requires understanding of server VLAN cross-patching & affinity groups

if smaller units of servers to be migrated in one event §  Conflicting VLAN numbering in switch blocks – virtual server VLAN re-

configuration required during migration event §  Risk – has a server or VLAN cross connection been overlooked? §  Server virtualisation / platform refresh is a follow on project


DC Migration – Big Bang

Every Server Migrated in Single Migration Event (Physical Move or Re-build)

7

Without a solution that enables IP mobility with IP Retention for each server then ‘Big Bang’ approach implies:

§  Years in Planning - takes longer to start moving servers due to data gathering and move planning

§  Longer storage migration cycle that requires keeping a large data set in synch over WAN (or other methods)

§  High risk / large service outage during migration event §  Cast of thousands / large workforce required


A solution that delivers IP mobility / IP Retention Cisco Locator/ID Separation Protocol (LISP)

WAN

Brownfield Data Center Greenfield Data Center

IP

•  Pluggable, Non-Disruptive Migration Network •  Keeps the Same IP address on the Server during the migration •  Works for all types of Servers (i.e. x86 and all others) – Physical or Virtual (Hypervisor Agnostic) •  Removes affinity group constraints

IP 10.1.1.5 IP 10.1.1.6 IP 10.1.1.7

WAN


DC Migration with IP mobility / IP Retention

•  Customer can perform the server migrations in much smaller waves which lowers the risk of the project.

•  Server Migration can begin much faster, as soon as the data for that server is available on Customer’s DC.

•  The amount of data to be kept in synch is minimized, reducing risk and WAN requirements.

•  Path optimization from the user to the application is possible, eliminating latency concerns and reducing WAN bandwidth requirements.

•  Simplicity: Repeatable, easy to implement with pre-defined price.

Enabled by Cisco Locator/ID Separation Protocol (LISP)

Customer reduced the migration window by 95%


LISP operations Registration “Push Mode”

West-DC East-DC

X Z Y

Map Server

A B C D

LISP Site

Map

-Reg

iste

r

ETR


LISP Operations Resolution “Pull Mode”

West-DC East-DC

X Z Y

A B C D

LISP Site

Map-Request

Map Server

ETR

ITR


LISP Operations Data-plane “Connectionless tunnel = Overlay”

West-DC East-DC

X Z Y

A B C D

LISP Site


Generic Deployment LISP with NO LAN Extension

L3 LISP tunnel

User

DC-1 DC-2

eTR eTR

iTR

1

2

3

iTR exists on User site (ie. Branch office) Redirect the end-user request to DC-2

Subnet A Subnet B

Model1

User

DC-1 DC-2

eTR eTR

PiTR

1

2

3

No iTR on User site (ie. Internet User) PTR on SP WAN intercepts request and redirect to DC-2

Subnet A Subnet B

Model 2

User

DC-1 DC-2

PxTR eTR

2

PTR on primary Site Intercepts end-User request, and redirect to DC-2

Subnet A Subnet B

3

1

Model 3


Generic Deployment LISP with NO LAN Extension

L3 LISP tunnel

User

DC-1 DC-2

eTR eTR

iTR

2

3

iTR exists on User site (ie. Branch office) Redirect the end-user request to DC-2

Subnet A Subnet B

Model1

User

DC-1 DC-2

eTR eTR

PiTR

2

3

No iTR on User site (ie. Internet User) PTR on SP WAN intercepts request and redirect to DC-2

Subnet A Subnet B

Model 2

User

DC-1 DC-2

PxTR eTR

2

PTR on primary Site Intercepts end-User request, and redirect to DC-2

Subnet A Subnet B

3

1

Model 3

§  Simplicity of Deployment

§  Address the Server team needs

§  Start here and evolve to model 2 or 1


PxTR

ASR

Introduced on a “S2ck” – non-‐intrusive;

Brownfield DC

Mobility needed on 10.1.1.0/24

LISP for Data Center Migration

10.1.1.5 10.1.1.6

L3 L2

Greenfield

10.1.1.0/24 advertised to the WAN from Brownfield, no change on routing required.

WAN

LISP xTR on a s2ck Default gateway for the moved traffic (op2onal) Does not receive any traffic before the move

10.1.1.0/24

xTR MS/MR

ASR

Any VLAN and

Any STP


ETR MS/MR

Brownfield DC


PxTR

ASR

10.1.1.5 10.1.1.6

L3 L2

ASR

Greenfield

10.1.1.0/24

Moving a resource – what happens?

WAN

LISP Mapping DB

Server -> Location 10.1.1.5 -> 2.2.2.2/3.3.3.3 10.1.1.6 -> 2.2.2.2/3.3.3.3

Server Moves – “Cold” Migration

2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5

10.1.1.5

IP/ARP


ETR MS/MR

Brownfield DC


PxTR

ASR

10.1.1.6

L3 L2

ASR

Greenfield

10.1.1.0/24

WAN

LISP Mapping DB

Server -> Location 10.1.1.5 -> 4.4.4.4/5.5.5.5 10.1.1.6 -> 2.2.2.2/3.3.3.3

2.2.2.2 3.3.3.3 4.4.4.4

10.1.1.5

The packet flow can be symmetric to allow Firewalls on Brownfield DC before the WAN

Packet Flow from Client to Server in Greenfield North-South Traffic after the migration

5.5.5.5 LISP-encapsulated packets


ETR MS/MR

Brownfield DC


PxTR

ASR

10.1.1.6

L3 L2

ASR

Greenfield

10.1.1.0/24

WAN

LISP Mapping DB

Server -> Location 10.1.1.5 -> 4.4.4.4/5.5.5.5 10.1.1.6 -> 2.2.2.2/3.3.3.3

2.2.2.2 3.3.3.3 4.4.4.4

Packet Flow between Servers in Brownfield and Greenfield DCs West-East Intra-subnet Traffic after the migration

LISP delivers intra subnet mobility. The servers communication is routed.

10.1.1.5

5.5.5.5 LISP-encapsulated packets


Comparison between OTV and LISP for Data Center Migration

Capability OTV LISP Comments

Non-Intrusive Insertion Yes Yes on Brownfield. In Greenfield, usually is the Default Gateway

VLAN Translation Yes Yes

Spanning-Tree Isolation Yes Yes

Share the ASR between multiple blocks No Yes LISP potentially requires less Hardware.

Broadcast Issues Protection No Yes

Support for stretched Cluster requiring Layer 2 Yes No

“Cold” migration Yes Yes

Live (“Hot”) migration Yes No

Share an ASR for OTV and LISP Yes Yes

Convergence on Router Failure ASR 3.11 up to 50 secs ASR 3.13 target below 10 sec

Up to 10 seconds N7K below 5 seconds

Route when you can, bridge when you must


Why Layer-3 based DC Migration?

•  Provides the ability to move servers between DC while keeping the same IP address and without extending the failure domain.

•  It allows a routed (Layer-3) connection between the sites.

•  Multiple customers mandate Layer-3 connection for long distance

•  Total isolation of broadcast (Layer-2) domains between the sites.

•  Also, LISP is having local significant VLAN allowing to support multiple aggregation blocks with overlapping

IP mobility without Layer 2 extension

20


Agenda •  Problem statement

•  The impact of LISP on Data Center Migrations

•  Technical Overview: LISP enabled on ASR 1000

•  Implementation Details

•  Testing Results –  Scale, Latency, Convergence Time and Performance

•  Conclusion

21


Physical Connectivity

4 X ASR1002-X

PxTR-1

PxTR-2

MS/MR-1

MS/MR-2


HSRP active for all Vlans

HSRP Standby for all Vlans

Legacy 6500s will remain as the default-gateways for hosts

MSMRs will be the default-gateways for hosts that migrate to the new DC

using LISP

The PxTRs will use a separate HSRP group

number to what is already in use on the legacy 6500s

HSRP Configuration


HSRP active for all Vlans

HSRP Standby for all Vlans

MSMRs will be the default-gateways for hosts that migrate to the new DC

using LISP

The PxTRs will use a separate HSRP group

number to what is already in use on the legacy 6500s

HSRP Configuration PxTR-01 interface GigabitEthernet0/0/0.2000 encapsulation dot1Q 500 ip address 11.10.0.4 255.255.255.0 ip pim sparse-mode standby delay minimum 180 reload 300 standby 20 ip 11.10.0.6 standby 20 timers 1 3 standby 20 priority 150 standby 20 preempt standby 20 track 6 decrement 100 standby 20 track 9 decrement 100 no lisp mobility liveness test lisp mobility LISP2000 PxTR-02 interface GigabitEthernet0/0/0.2000 encapsulation dot1Q 500 ip address 11.10.0.5 255.255.255.0 ip pim sparse-mode standby 20 ip 11.10.0.6 standby 20 timers 1 3 standby 20 preempt no lisp mobility liveness test lisp mobility LISP2000

MSMR-01 interface TenGigabitEthernet0/3/0.2000 encapsulation dot1Q 2000 ip address 11.10.0.2 255.255.255.0 ip pim sparse-mode standby delay minimum 180 reload 300 standby 1 ip 11.10.0.1 standby 1 timers 1 3 standby 1 priority 150 standby 1 preempt standby 1 track 5 decrement 100 no lisp mobility liveness test lisp mobility LISP2000 MSMR-02 interface TenGigabitEthernet0/3/0.2000 encapsulation dot1Q 2000 ip address 11.10.0.3 255.255.255.0 ip pim sparse-mode standby 1 ip 11.10.0.1 standby 1 timers 1 3 standby 1 preempt no lisp mobility liveness test lisp mobility LISP2000


Routing on ASRs

OSPF Process 1 Area 0

OSPF process 1 used only for RLOC reachability

between the ASRs

OSPF Process 1 will be used to advertise the RLOC IP addresses (loopback0) between the ASRs Completely isolated from routing on existing environment.

The MSMRs will have static default routes with the RLOC addresses of the PxTRs as the next hop. This is required for the use-PeTR function. ip route 0.0.0.0 0.0.0.0 <PxTR-1 loopback> ip route 0.0.0.0 0.0.0.0 <PxTR-2 loopback>

MSMRs use PeTR to send traffic to the PxTRs destined for non-LISP sites (WAN traffic).


LISP Multicast Map-Notify Messages

LISP uses multicast map-notify messages between the active and standby xTR to keep the LISP dynamic EID table in sync.


LISP Multicast Map-Notify Messages

- Multicast routing needs to enabled globally. - Separate Loopback interface 1 used as the RP address. - PIM sparse-mode enabled on each of the LISP mobility subnet sub-interfaces. - Each LISP mobility subnet uses a separate multicast group address for the map-notify messages. - Statically define each xTR with itself as the RP.

ip multicast-routing distributed ! interface Loopback1 description <Lo1> Multicast RP ip address 10.119.255.242 255.255.255.255 ip pim sparse-mode ! interface GigabitEthernet0/0/0.2000 encapsulation dot1Q 500 ip address 11.10.0.4 255.255.255.0 ip pim sparse-mode lisp mobility LISP2000 ! router lisp eid-table default instance-id 0 dynamic-eid LISP2000 database-mapping 11.10.0.0/24 locator-set DC1 map-notify-group 239.0.0.100 ! ip pim rp-address 10.119.255.242 override


Detecting EIDs on the existing Data Center PxTR is not the default gateway on source DC

•  LISP PxTRs need to receive a packet to detect hosts on the local segments to build up the EID table.

•  Hosts have already ARP’d for the default GW, so unlikely that the PxTR will receive a packet (ARP request broadcast) from hosts in production.

•  PxTRs may never learn about hosts in source DC…

28

•  Use a TCL script which sends unicast pings to each IP address within the LISP mobility subnets.

•  Even if the hosts do not respond to pings they will reply to the ARP that is sent by the xTR before the ping, and so the xTR will learn about the hosts on the segments.


Detecting EIDs on the existing Data Center PxTR is not the default gateway on source DC

29

tclsh puts "Enter the subnet 1st 3 octets of subnet:" flush stdout gets stdin subnet for {set i 1} {$i <= 254} {incr i} { set var $subnet append var $i ping $var rep 2 time 1}

This script can be copied into a text editor and saved it as a .tcl file. In this example it was saved as SWEEPING.tcl. The script should then be copied onto the flash on the ASR. Note the script is only needed on the PxTR-1.

alias exec detect tclsh flash:SWEEPING.tcl An alias command can be created on the ASR.

xTR-01#detect Enter the subnet 1st 3 octets of subnet: 10.10.14. Type escape sequence to abort. Sending 2, 100-byte ICMP Echos to 10.10.14.1, timeout is 1 seconds: !! Success rate is 100 percent (2/2), round-trip min/avg/max = 1/1/1 ms Type escape sequence to abort. Sending 2, 100-byte ICMP Echos to 10.10.14.2, timeout is 1 seconds: !!

Then to run, type ‘detect’. It will prompt you to enter the 1st 3 octets of the subnet address.


Cisco Validation Services

•  Confirmed that the solution met requirements for availability, security, and reliability before implementing it in the production network.

•  Vital on the success of this deployment.

•  Gives customers the experience and confidence.

•  Test overlap with rollout.

•  Reduction of risk of introducing new technology.

Enhanced Customer Aligned Testing Services

30

http://www.cisco.com/web/services/portfolio/documents/validation-service-overview.pdf


HW and SW Used on the validation testing The same HW and SW that was used in production

31

Qty Part # Hardware Platform

Software Version Role

4 ASR1002-X Cisco ASR1002-X Chassis, 6 built-in GE, Dual P/S, 4GB DRAM

IOS XE 3.10.0S LISP Enabled Routers / Devices Under Test

2 WS-C6509-E 6500-E chassis 12.2(33)SXH4 New Environment Core / Non-DUT 2 WS-C6509-E 6500-E chassis 12.2(33)SXH4 New Environment Aggregation / Non-DUT 4 WS-C6509 8.5(3) Aggregation Legacy / Non-DUT

4 WS-X6K-SUP1A-2GE 1000BaseX Supervisor Supervisor used on Legacy Agg Switches

ASR 1002-X with LISP enabled was connected to End of Life Catalyst 6500 with Sup1A, which proves this solution works with any network.


Test Results – Scale and Performance

32

Scale Parameter Number Comments

Number of subnets with LISP mobility 120 120 Tested at ECATS; 256 Supported with XE 3.10

Number of Dynamic EID (IP addresses) 2000 2000 Tested at ECATS;

5000 Supported with XE 3.10.

Performance

Platform Frame size (Byte)

Throughput (Mpps)

Throughput (Gbps)

Latency (us)

ASR1001 300 0.70 1.70 111.072

ASR1002-X 300 7.09 17.03 83.16

ASR 1006 or 1013 With RP2/ESP40

300 10.54 25.3 62.1


This is the latency added by the ASR on Legacy and ASR on New Data Center. It does not include the latency that may be added by the WAN link, this depends on the distance.

Latency Before and After the migration LISP on ASR adds 60 Microseconds for a migrated server = Negligible

33

Legacy <-> WAN

New <-> WAN

Legacy: Within Block

Legacy: Between

Block Legacy <->

New Within New

Server migration, diff VLANs, same block - Before 66 µs - 45 µs 55 µs 115 µs -

Server migration, diff VLANs, same block - After - 120 µs - - 115 µs 72 µs

Server migration, diff block - Before 73 µs - 45 µs 59 µs 120 µs -

Server migration, diff block - After - 123 µs - - 118 µs 39 µs

Server migration, Same VLAN - Before 55 µs - 38 µs 43µs 110 µs -

Server migration, Same VLAN - After - 118 µs - - 115 µs 55 µs

Server migration, Average - Before 64 µs - 42 µs 52 µs 115 µs -

Server migration, Average - After - 120 µs - - 116 µs 55 µs


The 28 seconds convergence when the 6500 Aggregation Switch on Legacy DC was powered off was caused by Spanning Tree reconvergence. Legacy DC was using Per-VLAN Spanning Tree.

Converge Times A / B : A = Convergence after failure; B = Convergence after recovery

Leg <--> WAN New <--> WAN Leg: Within Blk Leg: Between Blk Leg <--> New Within New

Router failure Tests Power Failure, Leg - Stby - - - - - - Power Failure, Leg - Act 3 sec. / - 3 sec. / - - - 3.1 sec. / - - Power Failure, New - Stby - - - - - - Power Failure, New - Act 3 sec. / - 3 sec. / - - - 6.5 sec. / - 3.3 sec. / - Reload, Leg - Stby - - - - - - Reload, Leg - Act - 0.3 Sec. / - - - - - Reload, New - Stby - - - - - - Reload, New - Act - 3 sec. / - - - 3.1 sec. / - - Sim. Crash, Leg - Act - - - - - - Sim. Crash, New - Act - - - - - - Sim. Crash, Leg- Stby - - - - - - Sim. Crash, New - Stby - - - - - - Power failure, Agg, Leg - Stby* 7.2 sec / - - 7.2 sec / * 7.2 sec / - 7.2 sec / - -

Power failure, Agg, Leg - Act* 9.7 sec. / 28 sec. 0.1 sec. / - * / * 9.7 sec. / 28 sec. 9.7 sec. / 28 sec. -

LISP Convergence below 10 seconds and faster for most cases


ETR MS/MR

Brownfield DC


PxTR

ASR

10.1.1.6

L3 L2 Any VLAN

and Any STP

ASR

L3 L2 Any VLAN

and Any STP Greenfield

10.1.1.0/24

WAN

LISP Mapping DB

Server -> Location 10.1.1.5 -> 4.4.4.4/5.5.5.5 10.1.1.6 -> 4.4.4.4/5.5.5.5

2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5

10.1.1.5

All the servers for a subnet have been migrated to new DC Goal is to avoid 'trombone' of all traffic

35

LISP-encapsulated packets

HSRP Configured


ETR MS/MR

Brownfield DC


PxTR

ASR

10.1.1.6

L3 L2 Any VLAN

and Any STP

ASR

L3 L2 Any VLAN


10.1.1.0/24

WAN

LISP Mapping DB

Server -> Location 10.1.1.5 -> 4.4.4.4/5.5.5.5 10.1.1.6 -> 4.4.4.4/5.5.5.5

2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5

10.1.1.5

All the servers for a subnet have been migrated to new DC Step 1 – Add Agg Switches to same HSRP group as ASR

36


HSRP Configured

1.  Add VLAN interface on the aggregation switch in the New DC

2.  Use the same HSRP group as on ASR 1K

3.  HSRP Virtual IP will be the same as on the ASR

4.  Use Lower HSRP priority on Agg than the sub-interfaces on the ASR

5.  Enable the vlan interfaces (no shut) and make sure they go into the HSRP listen state.

No Service Disruption


ETR MS/MR

Brownfield DC


PxTR

ASR

10.1.1.6

L3 L2 Any VLAN

and Any STP

ASR

L3 L2 Any VLAN


10.1.1.0/24

WAN

LISP Mapping DB

Server -> Location 10.1.1.5 -> 4.4.4.4/5.5.5.5 10.1.1.6 -> 4.4.4.4/5.5.5.5

2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5

10.1.1.5

All the servers for a subnet have been migrated to new DC Step 2 and 3 – Move the Gateway to the Agg Switches

37


HSRP Configured

1.  Increase the HSRP priority of the 6500s so that they preempt the ASRs and become HSRP active and standby.

2.  Remove HSRP configuration on the ASR1k’s sub-interface on New DC.

Those steps must be done in quick succession. This is to avoid traffic black holing.

No Service Disruption


ETR MS/MR

Brownfield DC


PxTR

ASR

10.1.1.6

L3 L2 Any VLAN

and Any STP

ASR

L3 L2 Any VLAN


10.1.1.0/24

WAN

LISP Mapping DB

Server -> Location 10.1.1.5 -> 4.4.4.4/5.5.5.5 10.1.1.6 -> 4.4.4.4/5.5.5.5

2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5

10.1.1.5

All the servers for a subnet have been migrated to new DC Step 4 and 5 – Start advertising the subnet from New DC

38


HSRP Configured

1.  Advertise the subnet to the WAN via the aggregation switches in New DC.

2.  Shutdown the corresponding interface on the aggregation switches on the Legacy DC.

Those steps must be done in quick succession. If they are done within 5 seconds, then 2 seconds

of traffic interruption.


Key Takeaways

•  Servers and Application Teams are increasingly demanding that the network allows migrations to happen with IP Preservation (no changes on the server).

•  LISP running on ASR 1000 has been validated by Cisco to meet this requirement.

–  It works with Any Network on Source and Destination Sites.

•  Solution deployed by large outsourcing provider for a multinational insurance company and was able to reduce by 95% the migration window time required to migrate applications from one DC to another.

39


More Information Detailed presentation, workshop, demo, test results, training, discuss your project

40

Santiago Freitas Customer Solutions Architect [email protected]

Patrice Bellagamba Distinguished Systems Engineer [email protected]


Complete Your Online Session Evaluation

•  Give us your feedback and you could win fabulous prizes. Winners announced daily.

•  Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

41

simplifying data center migrations using lisp, from 42...

Documents