ip nat troubleshooting solutions · ip nat source list 100 interface ethernet0/1 overload then...

28
IP NAT Troubleshooting Solutions Luke Bibby, CCIE #45527

Upload: others

Post on 14-Oct-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

IP NAT Troubleshooting

Solutions

Luke Bibby, CCIE #45527

Page 2: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Quick Overview of Scenario Solutions

Scenario #1

R2’s E0/0 should be NAT inside not NAT outside

ACL 100 is configured incorrectly

NAT policy missing the inside keyword

Scenario #2

RIP is not enabled on 1.0.0.0 on R1

NAT policy missing the inside keyword

NAT policy has the inside local and inside global addresses around the wrong way

Scenario #3

R1’s Loopback0 needs to be configured as NAT inside

NAT policies are referencing non-existant ACLs; should be referencing route maps

The route maps have the interfaces configured incorrectly

Page 3: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Scenario #1 Solutions

The first thing we should test is that the connectivity between R1 and R3 is infact actually not

working using ping.

R1#ping 160.0.23.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 160.0.23.3, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

Do we have NLRI for 160.0.23.3 or a default?

R1#show ip route 160.0.23.3

% Network not in table

R1#show ip route 0.0.0.0

Routing entry for 0.0.0.0/0, supernet

Known via "static", distance 1, metric 0, candidate default path

Routing Descriptor Blocks:

* 10.0.12.2

Route metric is 0, traffic share count is 1

And is the next-hop reachable?

R1#ping 10.0.12.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.0.12.2, timeout is 2 seconds:

.!!!!

Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms

Page 4: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

At this point we can deduce that there is no apparent problem between R1 and R2 so let’s now focus

on R2. Let’s make sure the inside and outside interfaces are up according to the diagram.

R2#show ip interface brief | exclude una|down

Any interface listed with OK? value "NO" does not have a valid configuration

Interface IP-Address OK? Method Status

Protocol

Ethernet0/0 10.0.12.2 YES manual up up

Ethernet0/1 160.0.23.2 YES manual up up

Let’s eliminate a connectivity problem between R2 and R3 using ping.

R2#ping 160.0.23.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 160.0.23.3, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

And lastly let’s confirm that R3 has no NLRI for the internal private network of R1-R2.

R3#show ip route | begin Gateway

Gateway of last resort is not set

160.0.0.0/16 is variably subnetted, 2 subnets, 2 masks

C 160.0.23.0/24 is directly connected, Ethernet0/0

L 160.0.23.3/32 is directly connected, Ethernet0/0

Because we are trying to route traffic from a network with a private address space out towards the

provider (R3 in this case) we need NAT to translate the private IP addresses into the public IP

address space allocated by the provider, 160.0.23.2 in this case. Obviously from the title of this

workbook the problem will be with NAT but in the real lab exam Cisco is not going to tell you what

Page 5: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

the problem is so it is always good practice to go through the process to identify where the problem

might be even if you are told what the problem actually is.

The most basic component of a NAT configuration is specifying which interfaces are the inside

interfaces and which ones are the outside interfaces (unless NVI is used in which case you just simply

enable NAT under the interface). The following command will confirm which interfaces are enabled

for NAT.

R2#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 0

Outside interfaces:

Ethernet0/0, Ethernet0/1

Inside interfaces:

Hits: 0 Misses: 0

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

-- Outside Destination

[Id: 1] access-list 100 interface Ethernet0/1 refcount 0

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

Both interfaces are specified as being NAT outside interfaces? Doesn’t it make sense that E0/0

should be an inside interface and E0/1 is the outside interface rather than them both being NAT

outside interfaces? Let’s hypothesize that this is part of the problem and test whether changing the

E0/0 configuration to be a NAT inside interface fixes the problem.

R2(config)#int e0/0

R2(config-if)#no ip nat outside

Page 6: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

R2(config-if)#ip nat inside

Then verify our configuration.

R2#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 0

Outside interfaces:

Ethernet0/1

Inside interfaces:

Ethernet0/0

Hits: 0 Misses: 0

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

-- Outside Destination

[Id: 1] access-list 100 interface Ethernet0/1 refcount 0

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

This looks better, let’s test out hypothesis from R1. You may be tempted to look for more problems

for each ticket on the real lab but in my opinion you should try to do the minimum amount of

changes to get solve the problem. Don’t go looking for faults that aren’t there.

R1#ping 160.0.23.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 160.0.23.3, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

Page 7: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Still nothing. Let’s run a debug on R2 for NAT (note that this command wasn’t supported for me on

CSR1KV but it works on IOU).

R2#debug ip nat

IP NAT debugging is on

Then try to ping from R1 again.

R1#ping 160.0.23.3 repeat 1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 160.0.23.3, timeout is 2 seconds:

.

Success rate is 0 percent (0/1)

However no debug output showed up on R2 meaning that R2 could not identify a translation policy

that matched R1’s traffic to R3. Let’s look at the policy on R2.

R2#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 0

Outside interfaces:

Ethernet0/1

Inside interfaces:

Ethernet0/0

Hits: 0 Misses: 0

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

-- Outside Destination

[Id: 1] access-list 100 interface Ethernet0/1 refcount 0

Page 8: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

If you don’t look at this output very often you may miss it but the problem has become apparently

when we looked at what the configured NAT translation policy is doing. The configured one is for

“Outside Destination” whereas we want to translate the source IP address from inside local to inside

global. If you need a refresher on these terms you can look at the following document.

http://www.cisco.com/c/en/us/support/docs/ip/network-address-translation-nat/4606-8.html

Before we create a new NAT policy (or change the existing policy), let’s confirm that the ACL 100 is

matching the traffic that we want.

R2#show access-list 100

Extended IP access list 100

10 permit ip 10.0.23.0 0.0.0.255 any

Isn’t the R1-R2 network 10.0.12.0/24 though? This ACL will never match that traffic. Let’s create a

new ACL that matches the traffic we actually want, which is IP traffic going from 10.0.12.0/24 to any

destination. We could parse the config for any more references to this ACL (class maps, packet

filtering, etc) to confirm that this ACL is not used by any other IOS functions, or we could simply just

create a new ACL. Before creating it, let’s confirm that we are not adding to an existing ACL as

follows.

R2#show access-list 101

Then create the new ACL.

R2(config)#access-list 101 permit ip 10.0.12.0 0.0.0.255 any

Page 9: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Seeing as we now believe the problem to be with the NAT policy configuration, we can use a show

run to grab the existing the NAT policy, copy it to notepad (incase it is not actually the fix and you

need to reapply the configuration), remove it, and add our new configuration.

R2#show run | s ip nat

ip nat inside

ip nat outside

ip nat source list 100 interface Ethernet0/1 overload

Then remove the configuration.

R2(config)#no ip nat source list 100 interface Ethernet0/1 overload

Then think about what we want to do: we want to translate the source IP address of inside traffic,

which is 10.0.12.0/24, going to any destination out of the outside interface, which is E0/1 (we could

also use a pool but its extra configuration that is unnecessary to fix this problem. Using these key

pieces of information, we can configure the following (given that we have already configured a new

ACL).

R2(config)#ip nat inside source list 101 interface Ethernet0/1 overload

Let’s run the debug on R2 again.

R2#debug ip nat

IP NAT debugging is on

Then try to ping from R1 to R3.

R1#ping 160.0.23.3 repeat 1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 160.0.23.3, timeout is 2 seconds:

Page 10: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

!

Success rate is 100 percent (1/1), round-trip min/avg/max = 5/5/5 ms

And on R2 we notice the following debug output.

*Jan 17 02:08:36.626: NAT*: s=10.0.12.1->160.0.23.2, d=160.0.23.3 [17]

*Jan 17 02:08:36.628: NAT*: s=160.0.23.3, d=160.0.23.2->10.0.12.1 [17]

This is a good because it is showing us that R2 is translating 10.0.12.1 (inside local) to 160.0.23.2

(inside global), and then back again on the return packet.

At this point we can consider this ticket solved!

Page 11: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Scenario #2 Solutions

The acceptance criteria of this ticket from the perspective of R3 is that it can connect to 160.0.23.2

on TCP port 80 and have the web server present some HTTP response (in this case, we can simply

just use GET for the document root). Let’s try to test this acceptance criteria as follows.

R3#telnet 160.0.23.2 80

Trying 160.0.23.2, 80 ...

% Connection refused by remote host

The remote end (R2 in this case) sent back a RST so we can already confirm that connectivity is

working to R2, so we can assume basic things like routing, ARP, etc, is working okay. Let’s now

confirm on R1 that the HTTP service is running and it has NLRI to the outside network.

R1#show ip route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

+ - replicated route, % - next hop override

Gateway of last resort is 10.0.12.2 to network 0.0.0.0

R* 0.0.0.0/0 [120/1] via 10.0.12.2, 00:00:24, Ethernet0/0

1.0.0.0/32 is subnetted, 1 subnets

C 1.1.1.1 is directly connected, Loopback0

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C 10.0.12.0/24 is directly connected, Ethernet0/0

L 10.0.12.1/32 is directly connected, Ethernet0/0

Page 12: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

And now check that the HTTP service is running.

R1#show control-plane host open-ports

Active internet connections (servers and established)

Prot Local Address Foreign Address Service State

tcp *:23 *:0 Telnet LISTEN

tcp *:8080 *:0 HTTP CORE LISTEN

The above command didn’t work on CSR1KV but did on IOU, so an alternative command you can use

to verify that the port is listening is as follows.

R1#show ip http server status

HTTP server status: Enabled

HTTP server port: 8080

HTTP server active supplementary listener ports:

HTTP server authentication method: enable

HTTP server digest algorithm: md5

HTTP server access class: 0

HTTP server base path:

HTTP server help root:

Maximum number of concurrent server connections allowed: 5

Server idle time-out: 180 seconds

Server life time-out: 180 seconds

Maximum number of requests allowed on a connection: 1

HTTP server active session modules: ALL

HTTP secure server capability: Present

HTTP secure server status: Disabled

HTTP secure server port: 443

HTTP secure server ciphersuite: 3des-ede-cbc-sha des-cbc-sha rc4-128-md5 rc4-128-

sha

HTTP secure server client authentication: Disabled

HTTP secure server trustpoint:

Page 13: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

HTTP secure server active session modules: ALL

At this point we know two things:

1. R1 is listening on TCP port 8080

2. R3 is attempting to connect to the web service on TCP port 80 using the IP address

160.0.23.3.

From this information we can deduce that R2 needs is configured to translate the destination IP

address of 160.0.23.2 and TCP port 8080 to R1 on TCP port 80. This is referred to as port address

redirection or port forwarding.

As we did with the last scenario, let’s confirm that there are NAT inside and outside interfaces, or

that NVI is configured.

R2#show ip nat statistics

Total active translations: 1 (1 static, 0 dynamic; 1 extended)

Peak translations: 1, occurred 00:00:06 ago

Outside interfaces:

Ethernet0/1

Inside interfaces:

Ethernet0/0

Hits: 0 Misses: 0

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

Page 14: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

According to the diagram and the original HTTP connectivity test we did at the start, this looks okay.

We also notice that a static NAT policy exists so let’s see what that looks like as follows.

R2#show ip nat translations

Pro Inside global Inside local Outside local Outside global

tcp --- --- 1.1.1.1:8080 160.0.23.2:80

This is saying that R2 is configured to translate between the outside local address of 1.1.1.1 and TCP

port 8080 to the outside global address of 160.0.23.3 and TCP port 80. Referring to the

documentation and remembering our NAT terms for inside, outside, local, and global, does this

configuration make sense? Aren’t the addresses 1.1.1.1 (R1’s loopback) and 160.0.23.3 (R2’s

interface connected to the R2-R3 network) both considered inside from the perspective of NAT? At

this point we can deduce that there is a mistake in the configuration and I am going to purposely use

show run here to show why this is often a bad idea to troubleshoot using that command.

R2#show run | s ip nat

ip nat inside

ip nat outside

ip nat outside source static tcp 160.0.23.2 80 1.1.1.1 8080 extendable

Your instincts might be to assume that the sole problem of this configuration is the outside

keyword in the NAT policy and that the fix is to replace that keyword with inside. This is because

you what you are translating from is usually listed first in the configuration and what you are

translating to is listed second.

R2(config)#no ip nat outside source static tcp 160.0.23.2 80 1.1.1.1 8080

extendable

R2(config)#ip nat inside source static tcp 160.0.23.2 80 1.1.1.1 8080 extendable

Now we verify our configuration.

R2#show ip nat translations

Pro Inside global Inside local Outside local Outside global

tcp 1.1.1.1:8080 160.0.23.2:80 --- ---

Page 15: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

But this doesn’t make sense because 1.1.1.1 is not an address assigned by a provider that represents

one or more private hosts to the outside world – 160.0.23.2 is. More specifically, 1.1.1.1 is not an

inside global address and 160.0.23.2 is not an inside local address; 1.1.1.1 is the inside local address

and 160.0.23.2 is the inside global address. Once again, if you used the configuration solely to isolate

this problem then you may miss this.

The correct configuration is actually as follows.

R2(config)#no ip nat inside source static tcp 160.0.23.2 80 1.1.1.1 8080 extendable

R2(config)#ip nat inside source static tcp ?

A.B.C.D Inside local IP address

R2(config)#ip nat inside source static tcp 1.1.1.1 8080 ?

A.B.C.D Inside global IP address

interface Specify interface for global address

R2(config)#ip nat inside source static tcp 1.1.1.1 8080 160.0.23.2 80 extendable

Now verify the configuration.

R2#show ip nat translations

Pro Inside global Inside local Outside local Outside global

tcp 160.0.23.2:80 1.1.1.1:8080 --- ---

This is looking much better. Let’s run the following debug on R2 before testing it out from R3.

R2#debug ip nat

IP NAT debugging is on

Then test from R3.

Page 16: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

R3#telnet 160.0.23.2 80

Trying 160.0.23.2, 80 ...

% Connection timed out; remote host not responding

This is different to what we saw earlier; we are timing out the connection request rather being

refused. Let’s see what the debugs on R2 look like.

R2#

*Jan 17 03:03:07.213: NAT*: TCP s=48361, d=80->8080

*Jan 17 03:03:07.213: NAT*: s=160.0.23.3, d=160.0.23.2->1.1.1.1 [42624]

R2#

*Jan 17 03:03:09.213: NAT*: TCP s=48361, d=80->8080

*Jan 17 03:03:09.213: NAT*: s=160.0.23.3, d=160.0.23.2->1.1.1.1 [42624]

R2#

*Jan 17 03:03:13.213: NAT*: TCP s=48361, d=80->8080

*Jan 17 03:03:13.213: NAT*: s=160.0.23.3, d=160.0.23.2->1.1.1.1 [42624]

R2#

*Jan 17 03:03:21.213: NAT*: TCP s=48361, d=80->8080

*Jan 17 03:03:21.213: NAT*: s=160.0.23.3, d=160.0.23.2->1.1.1.1 [42624]

We already confirmed earlier that R1 should has a default route so reachability back to 160.0.23.3

shouldn’t be a problem. Can we reach R1’s loopback from R2?

R2#show ip route 1.1.1.1

% Network not in table

R2#show ip route 0.0.0.0

% Network not in table

We could add a static route to R1’s loopback but the scenario already told us that we couldn’t do

that. We also noticed earlier that R1 was receiving a default route via RIP from R2 so let’s see what

R1 itself is advertising via RIP.

Page 17: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

R1#show ip protocols

*** IP Routing is NSF aware ***

Routing Protocol is "rip"

Outgoing update filter list for all interfaces is not set

Incoming update filter list for all interfaces is not set

Sending updates every 30 seconds, next due in 6 seconds

Invalid after 180 seconds, hold down 180, flushed after 240

Redistributing: rip

Default version control: send version 2, receive version 2

Interface Send Recv Triggered RIP Key-chain

Ethernet0/0 2 2

Automatic network summarization is not in effect

Maximum path: 4

Routing for Networks:

10.0.0.0

Routing Information Sources:

Gateway Distance Last Update

10.0.12.2 120 00:00:03

Distance: (default is 120)

As it turns out, RIP is the only IGP we are running and it is not advertising the 1.1.1.1/32 network,

therefore R2 has no knowledge of how to get to 1.1.1.1 after it translates the destination IP address

from 160.0.23.2 to 1.1.1.1. Therefore, it is important to understand here that the NAT operation

works but the routing does not. Let’s enable RIP on all subnets in 1.0.0.0/8.

R1(config)#router rip

R1(config-router)#network 1.0.0.0

Wait for the flash update to occur on R1 and for R2 to process and act on the update and once this

occurs we can confirm that R2 has NLRI to 1.1.1.1/32

Page 18: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

R2#show ip route 1.1.1.1

Routing entry for 1.1.1.1/32

Known via "rip", distance 120, metric 1

Redistributing via rip

Last update from 10.0.12.1 on Ethernet0/0, 00:00:19 ago

Routing Descriptor Blocks:

* 10.0.12.1, from 10.0.12.1, 00:00:19 ago, via Ethernet0/0

Route metric is 1, traffic share count is 1

Now we can retry the test on R3.

R3#telnet 160.0.23.2 80

Trying 160.0.23.2, 80 ... Open

GET /

HTTP/1.1 400 Bad Request

Date: Sat, 17 Jan 2015 03:16:20 GMT

Server: cisco-IOS

Connection: close

Accept-Ranges: none

400 Bad Request

[Connection to 160.0.23.2 closed by foreign host]

At this point we can consider this ticket solved too!

Page 19: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Scenario #3 Solutions

The troubleshooting ticket has already indicated that basic connectivity between R1 to R2, and from

R1 to R3 is working correctly. The problem occurs when R1 sources a ping from its loopback to either

of these routers and we are not allowed to simply advertise the 10.0.1.0/24 through a routing

protocol or use static routing to fix the problem. More specifically, the problem occurs when we are

trying to source the ping from a private network to a public address space. We can confirm this as

follows.

R1#ping 2.2.2.2 source loopback0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

.....

Success rate is 0 percent (0/5)

R1#ping 3.3.3.3 source loopback0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

.....

Success rate is 0 percent (0/5)

Just like in the other scenarios, we first verify which interfaces are configured as NAT inside and

outside, or whether NVI has been configured.

R1#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 0

Outside interfaces:

Ethernet0/0, Ethernet0/1

Inside interfaces:

Hits: 0 Misses: 0

Page 20: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

-- Inside Source

[Id: 1] access-list ISPR2 pool ISPR2_POOL refcount 0

pool ISPR2_POOL: netmask 255.255.255.0

start 160.0.12.1 end 160.0.12.1

type generic, total addresses 1, allocated 0 (0%), misses 0

[Id: 2] access-list ISPR3 pool ISPR3_POOL refcount 0

pool ISPR3_POOL: netmask 255.255.255.0

start 160.0.13.1 end 160.0.13.1

type generic, total addresses 1, allocated 0 (0%), misses 0

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

Immediately we see a problem: there are no inside interfaces. We need to first enable R1’s

loobpack0 interface as a NAT inside interface.

R1(config)#interface loop0

R1(config-if)#ip nat inside

Then verify the result of our configuration.

R1#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 0

Outside interfaces:

Ethernet0/0, Ethernet0/1

Page 21: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Inside interfaces:

Loopback0

Hits: 0 Misses: 0

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

-- Inside Source

[Id: 1] access-list ISPR2 pool ISPR2_POOL refcount 0

pool ISPR2_POOL: netmask 255.255.255.0

start 160.0.12.1 end 160.0.12.1

type generic, total addresses 1, allocated 0 (0%), misses 0

[Id: 2] access-list ISPR3 pool ISPR3_POOL refcount 0

pool ISPR3_POOL: netmask 255.255.255.0

start 160.0.13.1 end 160.0.13.1

type generic, total addresses 1, allocated 0 (0%), misses 0

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

Let’s see if this has fixed the problem using a general NAT debug and ping.

R1#debug ip nat

IP NAT debugging is on

R1#ping 2.2.2.2 source loopback0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

.....

Success rate is 0 percent (0/5)

Page 22: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

However we saw nothing in the debug output so this is leaning us towards a problem with the NAT

policy configuration. You are probably getting sick of this command already but it is useful for

troubleshooting NAT!

R1#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 0

Outside interfaces:

Ethernet0/0, Ethernet0/1

Inside interfaces:

Loopback0

Hits: 0 Misses: 0

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

-- Inside Source

[Id: 1] access-list ISPR2 pool ISPR2_POOL refcount 0

pool ISPR2_POOL: netmask 255.255.255.0

start 160.0.12.1 end 160.0.12.1

type generic, total addresses 1, allocated 0 (0%), misses 0

[Id: 2] access-list ISPR3 pool ISPR3_POOL refcount 0

pool ISPR3_POOL: netmask 255.255.255.0

start 160.0.13.1 end 160.0.13.1

type generic, total addresses 1, allocated 0 (0%), misses 0

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

Page 23: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

If we consider the first one isolation (ID of 1), it is referencing an ACL called ISPR2 for matching

traffic and is translating it to an IP address in the pool called ISPR2_POOL. We can already see from

above that the ISPR2_POOL has the range of 160.0.12.1 to 160.0.12.1, which is the IP address

configured on R1’s E0/0 interface. Let’s confirm the ACL now.

R1#show access-list ISPR2

Noting shows up. Do we have any access-lists at all (i.e. maybe they are configured but are

referenced incorrectly)?

R1#show access-lists

So we have multiple NAT pools but do not have any criteria to select which one to use. The answer

to this fix is route maps. The NAT policy configuration should be referencing route maps here,

selecting on the ACLs or the outgoing interface. Because we cannot configure any new ACLs, we

must use route maps where we can match on the outgoing interface or next-hop of the traffic to

select which NAT pool to use. Let’s confirm whether any route maps have been configured.

R1#show route-map

route-map ISPR2, permit, sequence 10

Match clauses:

interface Ethernet0/1

Set clauses:

Policy routing matches: 0 packets, 0 bytes

route-map ISPR3, permit, sequence 10

Match clauses:

interface Ethernet0/0

Set clauses:

Policy routing matches: 0 packets, 0 bytes

Page 24: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

We can see what the network administrator has done here. They have referenced ISPR2 and ISPR3

as an ACL when they should have been referenced as a route map. Let’s grab the configuration and

change the reference to an ACL to reference a route map.

R1#show run | s ip nat

ip nat inside

ip nat outside

ip nat outside

ip nat pool ISPR2_POOL 160.0.12.1 160.0.12.1 netmask 255.255.255.0

ip nat pool ISPR3_POOL 160.0.13.1 160.0.13.1 netmask 255.255.255.0

ip nat inside source list ISPR2 pool ISPR2_POOL

ip nat inside source list ISPR3 pool ISPR3_POOL

Then remove the existing policies.

R1(config)#no ip nat inside source list ISPR2 pool ISPR2_POOL

R1(config)#no ip nat inside source list ISPR3 pool ISPR3_POOL

Then we simply use the same commands as above except that we change the list keyword to

route-map.

R1(config)#ip nat inside source route-map ISPR2 pool ISPR2_POOL

R1(config)#ip nat inside source route-map ISPR3 pool ISPR3_POOL

Now we run a generalised NAT debug and try to test the original pings again.

R1#debug ip nat

IP NAT debugging is on

R1#ping 2.2.2.2 source loopback0 repeat 1

Type escape sequence to abort.

Page 25: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Sending 1, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

.

Success rate is 0 percent (0/1)

*Jan 17 03:58:04.926: NAT: s=10.0.1.1->160.0.13.1, d=2.2.2.2 [20]

Hmm, that didn’t work. Let’s see what happens when we try to ping R3.

R1#ping 3.3.3.3 source loopback0 repeat 1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

.

Success rate is 0 percent (0/1)

*Jan 17 03:59:34.822: NAT: s=10.0.1.1->160.0.12.1, d=3.3.3.3 [21]

Can you see what the problem is? If you aren’t used to the above debug output maybe the following

will make it clearer on R2.

R2#debug ip icmp

ICMP packet debugging is on

Then run the test again on R1 to R2.

R1#ping 2.2.2.2 source loopback0 repeat 1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

.

Success rate is 0 percent (0/1)

Page 26: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

And observe the debug output on R2.

R2#

*Jan 17 04:01:48.323: ICMP: echo reply sent, src 2.2.2.2, dst 160.0.13.1, topology

BASE, dscp 0 topoid 0

The source IP address on R1 is being translated from 1.1.1.1 to 160.0.13.1, which is the IP address on

the R1-R3 segment, and a similar thing is happening when R1 pings R3 because the source IP address

is being translated from 1.1.1.1 to 160.0.12.1 which is the IP address configured on R1’s E0/0

interface on the R1-R2 segment. That is, the route map is configured incorrectly.

R1#show route-map

route-map ISPR2, permit, sequence 10

Match clauses:

interface Ethernet0/1

Set clauses:

Policy routing matches: 0 packets, 0 bytes

route-map ISPR3, permit, sequence 10

Match clauses:

interface Ethernet0/0

Set clauses:

Policy routing matches: 0 packets, 0 bytes

If we compare this to the IP addresses configured on the interfaces, which are as follows.

R1#show ip interface brief | exc una|down

Interface IP-Address OK? Method Status

Protocol

Ethernet0/0 160.0.12.1 YES manual up up

Ethernet0/1 160.0.13.1 YES manual up up

Loopback0 10.0.1.1 YES manual up up

NVI0 10.0.1.1 YES unset up up

Page 27: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

We can see that the route maps have the wrong outgoing interfaces. The ISPR2 route map should be

referencing E0/0 and the ISPR3 route map should be referencing E0/1. Let’s fix this as follows.

R1(config)#route-map ISPR2 permit 10

R1(config-route-map)#no match interface

R1(config-route-map)#match interface E0/0

R1(config)#no route-map ISPR3 permit 10

R1(config)#route-map ISPR3 permit 10

R1(config-route-map)#match interface E0/1

Note that there are a few ways you can change the configuration. Let’s now verify the configuration

on R1 when it tries to ping R2 and R3 sourcing the packet from its loopback interface.

R1#ping 2.2.2.2 source loopback0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

R1#ping 3.3.3.3 source loopback0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:

Packet sent with a source address of 10.0.1.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

Then as a final check verify the NAT translation table as follows.

R1#show ip nat translations

Page 28: IP NAT Troubleshooting Solutions · ip nat source list 100 interface Ethernet0/1 overload Then remove the configuration. R2(config)#no ip nat source list 100 interface Ethernet0/1

Pro Inside global Inside local Outside local Outside global

icmp 160.0.12.1:11 10.0.1.1:11 2.2.2.2:11 2.2.2.2:11

icmp 160.0.13.1:12 10.0.1.1:12 3.3.3.3:12 3.3.3.3:12

Which looks good!