Icmp error does not match an existing connection

Solution ID	sk119312
Technical Level
Product	Quantum Security Gateways
Version	All
OS	Gaia
Platform / Model	All
Date Created	2017-08-03 00:00:00.0
Last Modified	2019-01-24 00:00:19.0

Symptoms

Cause

Solution

Introduction

I have been labbing up this particular technology in an effort to understand it better.  It is my personal opinion that the documentation is rather “lacking” in this particular area. The topic of interest is ASA ICMP error inspection.  Note, we are NOT talking about standard ICMP inspection, but the inspection of ICMP error messages.  Let’s look at our test bed for today:

What we have here is a pretty simple setup. To test our ICMP error inspection, we need to make it mildly more complicated. We will add a static NAT on the ASA such that the test PC at 10.10.3.100 is seen on the outside as 100.100.100.100. Note, OSPF has been configured on R2, R3 and the ASA for end to end IP connectivity. The ASA redistributes the outside 100.100.100.0/24 subnet into OSPF. NAT control is NOT being enforced. Let’s also make a simple outside ACL to permit our UNIX style traceroute. While we are at it we will allow ICMP echo-reply from the outside as well so we can ping R1 from the inside if need be for testing

static (inside,outside) 100.100.100.100 10.10.3.100 netmask 255.255.255.255
!
access-list OUTSIDE_IN extended permit udp host 100.100.100.1 any range 33434 33464
access-list OUTSIDE_IN extended permit icmp host 100.100.100.1 any echo-reply
access-group OUTSIDE_IN in interface outside

Default Behavior (no inspect icmp error)

Let’s give this a try and see what happens by default.

R1#trace 100.100.100.100

Type escape sequence to abort.
Tracing the route to 100.100.100.100

  1 100.100.100.100 232 msec 252 msec 140 msec
  2 100.100.100.100 532 msec 480 msec 612 msec
  3 100.100.100.100 744 msec 584 msec 628 msec

Uhhhhh, OK…that is interesting. We have not enabled any kind of fancy inspection of any kind, yet we have some fairly strange output for our trace. All three hops show as 100.100.100.100, the global IP address of our server. Why? Well, we have to understand how traceroute normally works before we can understand how the ASA modifies it. Normally, the process would look something like this assuming no ASA is in the middle.

•  R1 sends three UDP packets to 10.10.3.100 with a TTL of 1. These UDP packets are destined for ports 33434,33435 and 33436.

• As the next hop router R2 receives these packets, it decrements the TTL to 0 and thus has to send an ICMP time-exceeded message back to the source.  R2 sends an ICMP time-exceeded sourced from 10.10.10.2 back to 100.100.100.1, one for each packet.

• R1 receives these three ICMP time-exceeded messages and now we now that R2 is the first “hop” towards the server.  The first “hop” of our traceroute is complete

• The process repeats, but R1 sends three packets this time with a TTL of 2.  Once the packets get to R3, R3 decrements the TTL to 0 and the same thing happens again.

• Eventually, we trace the entire path to the server and the server sends back an ICMP unreachable message. With no firewall in the middle, our trace looks like this assuming we had a route to 10.10.3.0/24 from R1

R1#trace 10.10.3.100

Type escape sequence to abort.
Tracing the route to 10.10.3.100

  1 10.10.10.2 232 msec 252 msec 140 msec
  2 10.10.23.3 532 msec 480 msec 612 msec
  3 10.10.3.100 744 msec 584 msec 628 msec

So, what happened? By default, the ASA doesn’t like to reveal any more than it has to.  It does some crafty packet manipulation to hide the real IP address of the end destination we are tracing to AS WELL AS any intermediate hop routers between the firewall and the end destination. To fully understand what happens, we need to look at what an ICMP error packet looks like

The most interesting thing about this packet is that in the ICMP error payload, there is a record of the packet that caused the error in the first place.  As you can see, the ICMP error payload carries the IP header of the original packet that triggered the error.  This makes life a little more difficult when you have NAT involved in the process.

When R1 sends that first UDP packet in the traceroute, by the time the packet hits R2, the destination IP address has already been altered by the ASA due to the static NAT.  When it hits R2 the packet is destined to 10.10.3.100.  That means when R2 sends back the ICMP time-exceeded error message, in the payload of that ICMP message, the source and destination of the original message will be 100.100.100.1 –> 10.10.3.100.

What the ASA does by default is a couple of things.  First, when R2 sends the ICMP time-exceeded back to R1, the ASA actually looks into that packet.  It looks specifically at the ICMP payload, and it looks at the original IP packet source and destination as well as the original UDP source and destination port numbers. If those source/destination pairs match an existing connection flow that was setup through the ASA, a few different things happen. What do I mean by if it matches an existing connection?  When R1 sent that first UDP packet, a connection was created in the ASA for 100.100.100.1/abcde going to 10.10.3.100 port 33434.  When the ICMP error message coming back is looked into and the ASA sees that the original source/destination was 100.100.100.1/abcde –> 10.10.3.100/33434 it knows inherently that this is “return” traffic and it makes some changes to account for the NAT process.  What happens if there is no match or there is a mismatch?  The traffic is dropped.  What happens if we were using Microsoft tracert instead, which does not use UDP, but ICMP?  The same concept, but instead of tracking UDP ports it can look at ICMP sequence number pairs.  So, what exactly happens?

The ASA changes the source IP address of that message to 100.100.100.100.  This “hides” R2 as being an intermediate hop.  We still have a problem with our secrecy though, because in the ICMP payload, the true IP address of our server, 10.10.3.100 is revealed in the “Original IP destination” field.  To mask this behavior, the ASA also rewrites the ICMP payload such that the original IP destination address is changed back to 100.100.100.100.  That way, from the perspective of R1 on the outside, it has no idea that there was a NAT in the process. The same thing happens as each other hop behind the ASA sends back the time-exceeded messages.  That is why every hop comes back the same by default

Enabling ICMP error inspection

We have seen that by default, the ASA “hides” the real IP address of the host you are tracing to, and it also hides the IP addresses of any intermediate hops.  It does this by manipulating information in the IP and ICMP header as the ICMP error messages pass through it back to the source.  What if we don’t want that?  What if we actually want to see intermediate hop routers along the way to our destination?  That is where ICMP error inspection comes into play.  We can enable this using the legacy command fixup protocol icmp error or by enabling it in the MPF like so:

policy-map global_policy
 class inspection_default
  inspect icmp error

Let’s enable that on our ASA and see what happens.

ASA(config)# policy-map global_policy
ASA(config-pmap)#  class inspection_default
ASA(config-pmap-c)# inspect icmp error
ASA(config-pmap-c)#

R1#trace 100.100.100.100

Type escape sequence to abort.
Tracing the route to 100.100.100.100

  1 10.10.10.2 464 msec 372 msec 308 msec
  2 10.10.23.3 484 msec 324 msec 372 msec
  3 100.100.100.100 728 msec 724 msec 404 msec

OK, we see all the hops! Keep in mind the ASA still does not show up as a hop itself. This is normal, as the ASA does NOT decrement the TTL when it routes packets through itself (clever!). Now, what happened? Well, for starters we no longer are changing the source IP address in the IP header to 100.100.100.100. That is the key that makes the “real” hops show up. In our case we do not have any other NAT going on, but if we had NAT translations for those intermediate hops, the translated addresses would show up in our traceroute instead of the real addresses. For example, if we translated 10.10.10.2 on the inside to 100.100.100.2 on the outside, our first hop here would have been 100.100.100.2. However, if we examine the actual ICMP payload we will see that the original IP destination field is STILL set to 100.100.100.100. The ASA still changes this field. The net result is that the intermediate hop routers are now visible to our trace, but ultimately the REAL IP address of our server remains a mystery to R1 and anybody else on the outside of our network. Here is a wireshark capture of the first ICMP time-exceeded message from R2 to R1 arriving at R1 AFTER the ASA has had it’s way with it.

Let’s step through a few things.  We have enabled ICMP error inspection, so the source IP address is the REAL IP address of R2, 10.10.10.2.  Good.  Recall that when we have the feature disabled, the source IP was actually changed to the global IP of the server.  The next important thing is the ICMP payload.  As you see, it contains the IP header of the “original” packet that triggered the ICMP time-exceeded message.  Notice that the DST field is set to 100.100.100.100.  When the ASA received this packet from R2 that field was naturally set to 10.10.3.100, but the ASA altered it, thus hiding the real IP address of the server.

That about does it for ICMP error inspection on the ASA