debugging ipsec nat traversal again, was working

[journeyman]

Having trouble with 2 out of 6 ipsec tunnels, all were working previously. Continues from my previous post debugging ipsec with nat traversal. I have seen enmoc' s blog post on debugging and tried to work through it. Comment - at each step, what does a good result look like? We have a central FGT60C connecting via nat-t ipsec to 6 FGT60C remotes. The tunnels are interface based / routed and running ospf. All remote sites connect via 3G (the tunnel is our backup path to the remote sites; the 3G router is the NAT device). All tunnels are configured from the same template. We previously had an error with the NAT configuration which has been rectified at all remote sites (link above). The central unit failed recently and was replaced. All 6 tunnels came back up. Subsequently 3 tunnels failed and have been continuously attempting to establish. One tunnel came up two nights ago and has been up since. I don' t know any event to explain why it came up. So we currently have 4 tunnels up and 2 tunnels down. - configurations checked, - PSK re-entered anyway (no reason to suspect this is the problem), - confirmed traffic sent and received on UDP:500 and UDP:4500 both ends, - establishment traffic is constant (approx 1kbps in both directions for each of the down tunnels), - packet capture shows most of the traffic is UDP:500 but there is some UDP:4500, - diag vpn ike gateway list shows both tunnels with approx zero age, - syslog shows that both tunnels do occasionally come up for 20 seconds at a time, at random intervals between 2 - 15 minutes apart - diag debug app ike -1 shows a message being sent from remote:500 to central:4500. This was not allowed in the NAT router config. I have now allowed this traffic. Each attempt to establish the tunnel includes the following traffic.

wan2 -- central.4500 -> remote.4500: udp 112
 wan2 -- remote.4500 -> central.4500: udp 80
 wan2 -- remote.4500 -> central.4500: udp 304
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.4500: udp 304
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.4500: udp 96
 wan2 -- remote.4500 -> central.4500: udp 304
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.4500: udp 96
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.4500: udp 304
 wan2 -- remote.4500 -> central.4500: udp 96
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.500: udp 148
 wan2 -- remote.4500 -> central.4500: udp 96

This exchange lasts for 20 seconds and corresponds to the tunnel up syslog. Debug at both ends of a tunnel shows identical traffic (nothing is blocked anywhere). The central unit begins the exchange on 4500 but never replies again. I guess the fact that the central never relpies on 4500 is important? The remote tries 4500 > 500 which also seems odd. What should be my next step to fault find this issue, any suggestions please?

[ede_pfau]

Glad it' s working. Actually you don' t see ESP packets neither on the ' internal' nor ' tunnelXYZ' interfaces. I found them using the ' any' interface designator showing that the ' ppp1' interface is reporting them (" diag deb ena; diag deb sni pa any ' proto 50' 4' " ). So the FGT does see ESP traffic but not where you would expect it. As for tracing sniffer output, you can have it with full timestamps: diag deb sniffer packet any ' proto 50' 4 999 a options for the third parameter: <tsformat> format of timestamp a: absolute UTC time, yyyy-mm-dd hh:mm:ss.ms otherwise: relative to the start of sniffing, ss.ms

[journeyman]

Thank you, adding UTC time format will be very useful, I do a fair bit of sniffing. I try to avoid sniffing the any interface, and didn' t even think of it. While things were not working I saw traffic from the remote with source port 4500 and destination port 500. As mentioned above I have allowed it, but can that be removed? It seems odd. Thanks again

[journeyman]

Just to add to the absolute timestamp comment, some tricks I found. The overall syntax is

diagnose sniffer packet interface ' filter'  [[[verbosity] count] a]

The last three parameters are optional but must be present in order if desired. A count of 0 means " sniffing continuously (ctrl C interrupts)" - this is the default.