Author: David Monnier
What’s the value of a packet? How about three packets?
In this post I’ll show how you can identify potential reconnaissance being conducted on a network, including identifying the potential target, by taking specific note of one type of ICMP packet being produced by your border device.
ICMP, or Internet Control Message Protocol, is one of the unsung heroes of the Internet and cyber threat detection. The fabric of the Internet and the communications it contains and facilitates is largely informed by ICMP. It serves as the over-arching management mechanism that allows devices to be aware of congestion, missing routes, unavailable hosts, and a plethora of other functions. Today we’ll specifically cover ICMP type 11, otherwise known as Time To Live Exceeded, or TTL for short, and how its presence in your flows may indicate someone attempting to map the devices between your host and theirs — a common step when conducting recon against a cyber attack target.
First, let’s cover how TTL is used in IP networking. When a packet is created and sent across the network it is given an expiration of sorts. To ensure it doesn’t “live” forever, it sets a field that determines how many hops the packet may be forwarded before reaching its intended target. Various operating systems use different values for this field, with some as low as 64 and some as high as 255. Regardless of the value, a packet will be forwarded along a route until one of two things happens; the packet will either reach its intended recipient (ie; destination IP) or it will follow the path towards the recipient and have each router along the way reduce the TTL value inside by 1 and then send the packet on towards the next hop in the route. When a packet reaches a TTL value of 0 it cannot continue and is assumed an orphan resulting in the source being notified via ICMP (11) by the router that set the value to zero that the packet life has been exceeded. The source host receives this packet and understands that the packet it made never made it to the intended recipient and abandons the effort.
There are a few ways that TTL gets exceeded in real networking. Sometimes it’s due to misconfigurations in routing tables or by loops and other anomalies, and very often it’s on purpose. On purpose? Yes, on purpose. You see, being able to manipulate that field in a packet and receiving a reply from the router that last routed the packet is very useful to understand what the topology between the source and destination hosts look like. So useful, in fact, that the tool Traceroute was designed specifically for that purpose.
Here’s the description taken from the Traceroute manual page from a popular GNU/Linux distribution:
traceroute(8) – Linux man page
Name
traceroute – print the route packets trace to network host
Description
traceroute tracks the route packets taken from an IP network on their way to a given host. It utilizes the IP protocol’s time to live (TTL) field and attempts to elicit an ICMP TIME_EXCEEDED response from each gateway along the path to the host.
traceroute6 is equivalent to traceroute -6
The only required parameter is the name or IP address of the destination host. The optional packet_len‘gth is the total size of the probing packet (default 60 bytes for IPv4 and 80 for IPv6). The specified size can be ignored in some situations or increased up to a minimal value.
This program attempts to trace the route an IP packet would follow to some internet host by launching probe packets with a small ttl (time to live) then listening for an ICMP “time exceeded” reply from a gateway. We start our probes with a ttl of one and increase by one until we get an ICMP “port unreachable” (or TCP reset), which means we got to the “host”, or hit a max (which defaults to 30 hops). Three probes (by default) are sent at each ttl setting and a line is printed showing the ttl, address of the gateway and round trip time of each probe. The address can be followed by additional information when requested. If the probe answers come from different gateways, the address of each responding system will be printed. If there is no response within a 5.0 seconds (default), an “*” (asterisk) is printed for that probe.
After the trip time, some additional annotation can be printed: !H, !N, or !P (host, network or protocol unreachable), !S (source route failed), !F (fragmentation needed), !X (communication administratively prohibited), !V (host precedence violation), !C (precedence cutoff in effect), or !<num> (ICMP unreachable code <num>). If almost all the probes result in some kind of unreachable, traceroute will give up and exit.
We don’t want the destination host to process the UDP probe packets, so the destination port is set to an unlikely value (you can change it with the -p flag). There is no such a problem for ICMP or TCP tracerouting (for TCP we use half-open technique, which prevents our probes to be seen by applications on the destination host).
In the modern network environment the traditional traceroute methods can not be always applicable, because of widespread use of firewalls. Such firewalls filter the “unlikely” UDP ports, or even ICMP echoes. To solve this, some additional tracerouting methods are implemented (including tcp), see LIST OF AVAILABLE METHODS below. Such methods try to use particular protocol and source/destination port, in order to bypass firewalls (to be seen by firewalls just as a start of allowed type of a network session).
What does this mean in practice? Typically (aka, by default, but not only) this command, when run against a remote host, will result in a sequence of UDP datagrams sent in groups of 3 to the intended target where the value of the TTL field is increased by one and repeated until the intended host responds. The routers along the path will reply with a TTL exceeded message and that return message is used as a discovery step to trace the path.
For example, this is the traceroute to one of the time servers run by Apple from an AWS host:
$ traceroute 17.253.14.251
traceroute to 17.253.14.251 (17.253.14.251), 64 hops max
1 216.182.224.174 12.138ms 21.740ms 11.849ms
2 100.65.89.32 47.270ms 8.610ms 8.180ms
3 100.66.10.98 19.421ms 21.891ms 21.887ms
4 100.66.7.17 26.423ms 16.807ms 21.897ms
5 100.66.5.109 10.435ms 19.606ms 21.836ms
6 100.66.5.161 19.503ms 11.643ms 62.394ms
7 52.93.28.175 3.437ms 0.767ms 0.760ms
8 100.100.4.0 0.922ms 1.734ms 1.468ms
9 17.1.144.64 1.176ms 1.115ms 1.156ms
10 17.1.0.66 9.425ms 15.509ms 8.336ms
11 17.1.2.188 11.599ms 11.806ms 11.923ms
12 17.1.2.181 7.494ms 7.883ms 7.792ms
13 17.1.2.165 14.564ms 6.647ms 6.605ms
14 17.0.147.9 6.448ms 6.388ms 6.402ms
15 17.0.147.9 6.401ms 6.435ms 6.346ms
16 17.253.14.9 6.644ms 6.642ms 6.509ms
17 17.253.14.251 6.470ms 6.575ms 6.565ms
Here’s what it looks like at the network level using the tool tcpdump:
$ sudo tcpdump -n proto 1 or proto 17
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:20:52.693457 IP 172.26.7.235.50135 > 17.253.14.251.33434: UDP, length 9
19:20:52.705648 IP 216.182.224.174 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.705833 IP 172.26.7.235.50135 > 17.253.14.251.33434: UDP, length 9
19:20:52.727576 IP 216.182.224.174 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.727727 IP 172.26.7.235.50135 > 17.253.14.251.33434: UDP, length 9
19:20:52.739579 IP 216.182.224.174 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.739730 IP 172.26.7.235.50135 > 17.253.14.251.33435: UDP, length 9
19:20:52.786972 IP 100.65.89.32 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.787108 IP 172.26.7.235.50135 > 17.253.14.251.33435: UDP, length 9
19:20:52.795691 IP 100.65.89.32 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.795809 IP 172.26.7.235.50135 > 17.253.14.251.33435: UDP, length 9
19:20:52.803988 IP 100.65.89.32 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.804146 IP 172.26.7.235.50135 > 17.253.14.251.33436: UDP, length 9
19:20:52.823566 IP 100.66.10.98 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.823685 IP 172.26.7.235.50135 > 17.253.14.251.33436: UDP, length 9
19:20:52.845547 IP 100.66.10.98 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.845668 IP 172.26.7.235.50135 > 17.253.14.251.33436: UDP, length 9
19:20:52.867551 IP 100.66.10.98 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.867693 IP 172.26.7.235.50135 > 17.253.14.251.33437: UDP, length 9
19:20:52.889024 IP 100.66.7.17 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.894227 IP 172.26.7.235.50135 > 17.253.14.251.33437: UDP, length 9
19:20:52.911017 IP 100.66.7.17 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.911129 IP 172.26.7.235.50135 > 17.253.14.251.33437: UDP, length 9
19:20:52.933022 IP 100.66.7.17 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.933163 IP 172.26.7.235.50135 > 17.253.14.251.33438: UDP, length 9
19:20:52.943593 IP 100.66.5.109 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.943700 IP 172.26.7.235.50135 > 17.253.14.251.33438: UDP, length 9
19:20:52.963293 IP 100.66.5.109 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.963394 IP 172.26.7.235.50135 > 17.253.14.251.33438: UDP, length 9
19:20:52.985210 IP 100.66.5.109 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:52.985365 IP 172.26.7.235.50135 > 17.253.14.251.33439: UDP, length 9
19:20:53.004860 IP 100.66.5.161 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.004968 IP 172.26.7.235.50135 > 17.253.14.251.33439: UDP, length 9
19:20:53.016601 IP 100.66.5.161 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.016705 IP 172.26.7.235.50135 > 17.253.14.251.33439: UDP, length 9
19:20:53.079088 IP 100.66.5.161 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.079259 IP 172.26.7.235.50135 > 17.253.14.251.33440: UDP, length 9
19:20:53.082694 IP 52.93.28.175 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.082792 IP 172.26.7.235.50135 > 17.253.14.251.33440: UDP, length 9
19:20:53.083553 IP 52.93.28.175 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.083650 IP 172.26.7.235.50135 > 17.253.14.251.33440: UDP, length 9
19:20:53.084404 IP 52.93.28.175 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.084524 IP 172.26.7.235.50135 > 17.253.14.251.33441: UDP, length 9
19:20:53.085444 IP 100.100.4.0 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.085549 IP 172.26.7.235.50135 > 17.253.14.251.33441: UDP, length 9
19:20:53.087284 IP 100.100.4.0 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.087360 IP 172.26.7.235.50135 > 17.253.14.251.33441: UDP, length 9
19:20:53.088840 IP 100.100.4.0 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.088950 IP 172.26.7.235.50135 > 17.253.14.251.33442: UDP, length 9
19:20:53.090130 IP 17.1.144.64 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.090208 IP 172.26.7.235.50135 > 17.253.14.251.33442: UDP, length 9
19:20:53.091330 IP 17.1.144.64 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.091405 IP 172.26.7.235.50135 > 17.253.14.251.33442: UDP, length 9
19:20:53.092558 IP 17.1.144.64 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.092658 IP 172.26.7.235.50135 > 17.253.14.251.33443: UDP, length 9
19:20:53.102080 IP 17.1.0.66 > 172.26.7.235: ICMP time exceeded in-transit, length 148
19:20:53.102162 IP 172.26.7.235.50135 > 17.253.14.251.33443: UDP, length 9
19:20:53.117665 IP 17.1.0.66 > 172.26.7.235: ICMP time exceeded in-transit, length 148
19:20:53.117747 IP 172.26.7.235.50135 > 17.253.14.251.33443: UDP, length 9
19:20:53.126080 IP 17.1.0.66 > 172.26.7.235: ICMP time exceeded in-transit, length 148
19:20:53.126181 IP 172.26.7.235.50135 > 17.253.14.251.33444: UDP, length 9
19:20:53.137761 IP 17.1.2.188 > 172.26.7.235: ICMP time exceeded in-transit, length 152
19:20:53.137875 IP 172.26.7.235.50135 > 17.253.14.251.33444: UDP, length 9
19:20:53.149668 IP 17.1.2.188 > 172.26.7.235: ICMP time exceeded in-transit, length 152
19:20:53.149786 IP 172.26.7.235.50135 > 17.253.14.251.33444: UDP, length 9
19:20:53.161705 IP 17.1.2.188 > 172.26.7.235: ICMP time exceeded in-transit, length 152
19:20:53.161848 IP 172.26.7.235.50135 > 17.253.14.251.33445: UDP, length 9
19:20:53.169336 IP 17.1.2.181 > 172.26.7.235: ICMP time exceeded in-transit, length 148
19:20:53.169439 IP 172.26.7.235.50135 > 17.253.14.251.33445: UDP, length 9
19:20:53.177315 IP 17.1.2.181 > 172.26.7.235: ICMP time exceeded in-transit, length 148
19:20:53.177413 IP 172.26.7.235.50135 > 17.253.14.251.33445: UDP, length 9
19:20:53.185199 IP 17.1.2.181 > 172.26.7.235: ICMP time exceeded in-transit, length 148
19:20:53.185329 IP 172.26.7.235.50135 > 17.253.14.251.33446: UDP, length 9
19:20:53.199863 IP 17.1.2.165 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.199990 IP 172.26.7.235.50135 > 17.253.14.251.33446: UDP, length 9
19:20:53.206643 IP 17.1.2.165 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.206749 IP 172.26.7.235.50135 > 17.253.14.251.33446: UDP, length 9
19:20:53.213339 IP 17.1.2.165 > 172.26.7.235: ICMP time exceeded in-transit, length 36
19:20:53.213466 IP 172.26.7.235.50135 > 17.253.14.251.33447: UDP, length 9
19:20:53.219896 IP 17.0.147.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.219999 IP 172.26.7.235.50135 > 17.253.14.251.33447: UDP, length 9
19:20:53.226380 IP 17.0.147.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.226482 IP 172.26.7.235.50135 > 17.253.14.251.33447: UDP, length 9
19:20:53.232865 IP 17.0.147.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.232985 IP 172.26.7.235.50135 > 17.253.14.251.33448: UDP, length 9
19:20:53.239391 IP 17.0.147.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.239472 IP 172.26.7.235.50135 > 17.253.14.251.33448: UDP, length 9
19:20:53.245888 IP 17.0.147.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.245986 IP 172.26.7.235.50135 > 17.253.14.251.33448: UDP, length 9
19:20:53.252340 IP 17.0.147.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.2524 49 IP 172.26.7.235.50135 > 17.253.14.251.33449: UDP, length 9
19:20:53.259082 IP 17.253.14.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.259180 IP 172.26.7.235.50135 > 17.253.14.251.33449: UDP, length 9
19:20:53.265814 IP 17.253.14.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.265915 IP 172.26.7.235.50135 > 17.253.14.251.33449: UDP, length 9
19:20:53.272417 IP 17.253.14.9 > 172.26.7.235: ICMP time exceeded in-transit, length 45
19:20:53.272570 IP 172.26.7.235.50135 > 17.253.14.251.33450: UDP, length 9
19:20:53.279032 IP 17.253.14.251 > 172.26.7.235: ICMP 17.253.14.251 udp port 33450 unreachable, length 45
19:20:53.279126 IP 172.26.7.235.50135 > 17.253.14.251.33450: UDP, length 9
19:20:53.285628 IP 17.253.14.251 > 172.26.7.235: ICMP 17.253.14.251 udp port 33450 unreachable, length 45
19:20:53.285775 IP 172.26.7.235.50135 > 17.253.14.251.33450: UDP, length 9
19:20:53.292271 IP 17.253.14.251 > 172.26.7.235: ICMP 17.253.14.251 udp port 33450 unreachable, length 45
Note the repeated pattern. Three (3) UDP packets out with the same TTL, and three (3) replies from the router that reduced the TTL to zero (0). The source host then increases the TTL value by one (1) and repeats the process. Eventually every host along the path replies with ICMP 11 or the final host responds.
Operationally this allows us to create workflows around the presence of ICMP type 11 in our router logs or netflow records. Seeing your router send ICMP type 11 responses to hosts in groups of three (3) almost certainly means the tool Traceroute was used to map the network between your host and the destination of that ICMP packet. To know what host in your network was targeted, you can look for inbound UDP datagrams, in groups of three, destined to your network that have a source IP that matches the destination IP of the ICMP packets. The destination IP of the traceroute never changes and will be present in every routers logs or flow records along the path.
Note: There are a few probe types with the Traceroute tool. Probes may come as ICMP type 8 (ECHO REQUEST). The tracert command as part of the Windows operating systems will also do the probe using ICMP type 8. Regardless of the protocol, the TTL manipulation and corresponding ICMP type 11 response are always present.
Special note for netflow users: ICMP doesn’t use ports! Netflow will show you the ICMP message type in the DST Port field of the flow records. This will show up as port 11 OR 2816 depending on your flow format. BOTH types can be found in our global cyber threat analyzer, Augury and can be specified when running a flow query by specifying the destination port in the query to “11,2816”.