Technical Tip: Configuring and verifying a GRE over an IPsec tunnel

sha-1_FTNT · ‎03-10-2017

Description

This article describes how to configure and troubleshoot a GRE over an IPsec tunnel between a FortiGate and a Cisco router.

Scope

Support for GRE tunneling and GRE over IPsec in tunnel-mode is available as of FortiOS 3.0.
Support for IPsec in transport-mode is available as of FortiOS 4.0 MR2.

Solution

Diagram

The following topology is used:

PC1(.1) - 10.1.1.0/24-port2-[ FGT ]-port1----(198.51.100.1) =======
             I
     (Internet)     I GRE over IPsec tunnel
         I
PC2(.2) - 10.2.2.0/24-gi0/0-[ Cisco_RTR ]-gi1/0-(192.0.2.2) =======

Design

This setup is designed in the following ways:

- The goal is to establish a GRE over IPsec tunnel between a FortiGate and a Cisco router to be able to reach each remote LAN 10.x.x.x.

- IPsec in transport mode is used since data packets are already tunneled in GRE.

- OSPF is also used as a dynamic routing protocol (with multicast traffic, hence the need for GRE-IPsec with some vendors).

Why a GRE is being used over IPsec tunnel instead of a plain IPsec tunnel

Common reasons are:

1) Multicast over IPsec:

Some vendors do not support multicast traffic (such as with OSPF or streaming) directly inside an IPsec tunnel.
The multicast traffic is therefore tunneled in GRE, which itself is protected by IPsec. Only the unicast GRE traffic between the GRE endpoints is exposed to IPsec.

FortiOS supports multicast traffic directly inside IPsec.
There is therefore no requirement to use GRE-IPsec to carry multicast traffic between two FortiGates.

2) Traffic selector simplification:

Some vendors do not support negotiating wildcard traffic selectors (namely any-any selectors: src-subnet=0.0.0.0/0 and dst-subnet=0.0.0.0/0).
The vendors either require an exhaustive list of all local-subnets and all remote-subnets, or they require specification of all the possible combinations of (local <-> remote) subnets.

FortiOS supports any-any selectors (src-subnet=0.0.0.0/0 and dst-subnet=0.0.0.0/0).

There is therefore no requirement to use GRE-IPsec to simplify the traffic selector configuration between two FortiGates.

GRE over IPsec configuration with FortiOS

The GRE over IPsec configuration in this article is based on the independent configuration of GRE settings and IPsec settings. A tighter integration between GRE and IPsec ('encapsulation gre') is available as of FortiOS 5.4.

FortiOS 5.4 'encapsulation gre' has some limitations, however:

- Only IPsec in tunnel-mode is supported (no support for IPsec in transport-mode).
- The traffic selectors cannot be restricted to the GRE endpoints.
- Dynamic routing is not supported.

All of the above 'encapsulation gre' limitations are removed as of FortiOS 5.6:

- IPsec is supported in both transport-mode and tunnel-mode.
- Traffic selectors can be restricted to the GRE endpoints' addresses and GRE protocol (ip/47).
- Dynamic routing is supported.

The scenario covered in this article can also be used with the 'encapsulation gre' configuration. See the Related Articles section.

Limitations

- The inner GRE traffic cannot be hardware offloaded to NPU (NP6, NP4).

- IPsec in transport-mode cannot be offloaded to NPU (NP6, NP4).

Configuration

CLI configuration of the FortiGate

In the following, Port1 is the Internet-facing interface and Port2 is the LAN interface:

# config system interface
    edit "port1"
        set ip 198.51.100.1 255.255.255.0
        set alias "Internet"
    next
    edit "port2"
        set ip 10.1.1.254 255.255.255.0
        set alias "LAN"
    next
end

IPsec VPN used to protect the GRE traffic:

# config vpn ipsec phase1-interface
    edit "ipsec"
        set interface "port1"
        set proposal aes128-sha1
        set dhgrp 14
        set remote-gw 192.0.2.2
        set psksecret fortinet
    next
end

# config vpn ipsec phase2-interface
    edit "ipsec"
        set phase1name "ipsec"
        set proposal aes128-sha1
        set dhgrp 14
        set protocol 47                     // restrict traffic selectors to GRE protocol (ip/47)
        set auto-negotiate enable
        set encapsulation transport-mode    // transport-mode (GRE is already tunneled)
    next
end

Configure a GRE tunnel:

# config system gre-tunnel
    edit "toCisco"
        set interface "ipsec"      // the GRE tunnel is protected by IPsec
        set remote-gw 192.0.2.2
        set local-gw 198.51.100.1
    next
end

Local and remote overlay IP addresses over the GRE tunnel:

# config system interface
    edit "toCisco"
        set ip 10.255.255.1 255.255.255.255
        set remote-ip 10.255.255.2
    next
end

Firewall policies:

# config firewall address
    edit "10.1.1.0/24"
        set comment "Local LAN"
        set subnet 10.1.1.0 255.255.255.0
    next
    edit "10.2.2.0/24"
        set comment "Remote LAN"
        set subnet 10.2.2.0 255.255.255.0
    next
end

Allow traffic between the local LAN (port2) and the remote LAN (GRE):

Note: the following configuration in this section is intended as one configuration. It is periodically interrupted by additional context written with an indent.

# config firewall policy
   edit 1
      set name "to remote LAN"
      set srcintf "port2"
      set dstintf "toCisco"
      set srcaddr "10.1.1.0/24"
      set dstaddr "10.2.2.0/24"
      set action accept
      set schedule "always"
      set service "ALL"
      set comments "local LAN to remote LAN"
   next
   edit 2
      set name "from remote LAN"
      set srcintf "toCisco"
      set dstintf "port2"
      set srcaddr "10.2.2.0/24"
      set dstaddr "10.1.1.0/24"
      set action accept
      set schedule "always"
      set service "ALL"
      set comments "remote LAN to local LAN"
   next

(GRE traffic to be IPsec-protected is self-originated: it is not received on an interface. No forward-policy is therefore needed to allow GRE traffic to enter or leave the IPsec interface. However, in FortiOS design, a forward-policy is required to allow an IPsec negotiation to take place. An arbitrary forward-policy (e.g., from and to the IPsec interface itself) is therefore used to 'activate' IPsec:)

edit 3

set name "Enable IPsec"

set srcintf "ipsec"

set dstintf "ipsec"

set srcaddr "all"

set dstaddr "all"

set action accept

set schedule "always"

set service "ALL"

set comments "Just an \'activator\' for IPsec negotiation. No traffic flowing through this policy since IPsec is used to protect self-originated GRE traffic."

If the remote LAN subnet (10.2.2.0/24) is missing in the routing table (such as if OSPF adjacency is down), packets destined to 10.2.2.0/24 would match the default-route and the ‘Internet Access’ policy. This ‘Deny Internet’ policy ensures that packets destined to the remote LAN never match the ‘Internet Access’ policy.

edit 4
   set name "Deny Internet"
   set srcintf "port2"
   set dstintf "port1"
   set srcaddr "all"
   set dstaddr "10.2.2.0/24"
   set schedule "always"
   set service "ALL"
   set logtraffic all
   set comments "Prevent remote LAN access to leak over the Internet"

next
edit 5
   set name "Internet Access"
   set srcintf "port2"
   set dstintf "port1"
   set srcaddr "10.1.1.0/24"
   set dstaddr "all"
   set action accept
   set schedule "always"
   set service "ALL"
   set comments "Internet Access"
   set nat enable

OSPF routing:

# config router ospf

set router-id 10.1.1.254

config area

edit 0.0.0.0

config network

edit 1

set prefix 10.1.1.254 255.255.255.255 // enable OSPF on the LAN interface

set prefix 10.255.255.1 255.255.255.255 // enable OSPF on the GRE tunnel

end

Static routes:

# config router static

edit 1

set gateway 198.51.100.254

set device "port1"

set comment "default-route to Internet ISP"

edit 2

set dst 192.0.2.2 255.255.255.255

set device "ipsec"

set comment "Reach GRE endpoint via IPsec tunnel"

CLI configuration of the Cisco Router

IPsec configuration

# crypto isakmp policy 10

encr aes

authentication pre-share

group 14

# crypto isakmp key fortinet address 198.51.100.1

# crypto ipsec transform-set aes128-sha1-transport esp-aes esp-sha-hmac mode transport

ip access-list extended encryptionDomain permit gre host 192.0.2.2 host 198.51.100.1

# crypto map gre_over_ipsec 10 ipsec-isakmp

set peer 198.51.100.1

set transform-set aes128-sha1-transport

set pfs group14

match address encryptionDomain

GRE tunnel interface:

# interface Tunnel0
ip address 10.255.255.2 255.255.255.252 <-- overlay subnet over the GRE tunnel
tunnel source GigabitEthernet1/0
tunnel destination 198.51.100.1

LAN:

# interface GigabitEthernet0/0
ip address 10.2.2.254 255.255.255.0
ip nat inside

Internet:

# interface GigabitEthernet1/0

ip address 192.0.2.2 255.255.255.0

ip nat outside

crypto map gre_over_ipsec <-- apply IPsec to the traffic matching the crypto map

OSPF:

# router ospf 1
router-id 10.2.2.254
network 10.2.2.254 0.0.0.0 area 0
network 10.255.255.2 0.0.0.0 area 0

SNAT for internet access:

# ip nat inside source list natAcl interface GigabitEthernet1/0 overload

# ip access-list extended natAcl

permit ip 10.2.2.0 0.0.0.255 any

Static route: default-route to Internet ISP.

# ip route 0.0.0.0 0.0.0.0 192.0.2.253

Verification

On the FortiGate, verify the routing table (RIB). Example output is attached:

# get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S*      0.0.0.0/0 [10/0] via 198.51.100.254, port1
C       10.1.1.0/24 is directly connected, port2
O       10.2.2.0/24 [110/101] via 10.255.255.2, toCisco, 00:06:10
O       10.255.255.0/30 [110/1100] via 10.255.255.2, toCisco, 00:06:10
C       10.255.255.1/32 is directly connected, toCisco
C       10.255.255.2/32 is directly connected, toCisco
C       172.16.31.0/24 is directly connected, port10
S       192.0.2.2/32 [10/0] is directly connected, ipsec
C       198.51.100.0/24 is directly connected, port1

To verify the GRE tunnel status, run the following command. Once more, example output is attached.

# diagnose sys gre keepalive

GRE tunnel: toCisco, Status: alive

period: 5, retries 2/10

Packets sent: 49632 Packets received: 12234

Verify that PC1 and PC2 can ping each other by running the following commands as the root users of PC1 and then PC2, respectively. In this case, 10.2.2.2 is the IP of PC2 and 10.1.1.1 is the IP of PC1.

From PC1:

# ping -c 5 10.2.2.2
PING 10.2.2.2 (10.2.2.2) 56(84) bytes of data.
64 bytes from 10.2.2.2: icmp_seq=1 ttl=62 time=40.7 ms
64 bytes from 10.2.2.2: icmp_seq=2 ttl=62 time=53.5 ms
64 bytes from 10.2.2.2: icmp_seq=3 ttl=62 time=50.0 ms
64 bytes from 10.2.2.2: icmp_seq=4 ttl=62 time=47.1 ms
64 bytes from 10.2.2.2: icmp_seq=5 ttl=62 time=44.9 ms

--- 10.2.2.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 40.769/47.296/53.577/4.379 ms

From PC2:

# ping 10.1.1.1
84 bytes from 10.1.1.1 icmp_seq=1 ttl=62 time=87.241 ms
84 bytes from 10.1.1.1 icmp_seq=2 ttl=62 time=80.711 ms
84 bytes from 10.1.1.1 icmp_seq=3 ttl=62 time=46.863 ms
84 bytes from 10.1.1.1 icmp_seq=4 ttl=62 time=46.857 ms
84 bytes from 10.1.1.1 icmp_seq=5 ttl=62 time=46.881 ms

5 packets transmitted, 5 received, 0% packet loss

Troubleshooting

This section contains possible troubleshooting steps to identify issues with the connection.

Verify the GRE tunnel interface status in the FortiGate CLI:

# diag sys gre list

vd=0 devname=toCisco devindex=15 ifindex=20
saddr=198.51.100.1 daddr=192.0.2.2 ref=0
key=0/0 flags=0/0

total tunnel = 1

# diag netlink interface list | grep -A1 "toCisco"
if=toCisco family=00 type=778 index=20 mtu=1430 link=0 master=0
ref=17 state=off start fw_flags=0 flags=up p2p run noarp multicast

# get sys interface | grep -A1 "toCisco"
== [ toCisco ]
name: toCisco ip: 10.255.255.1 255.255.255.255 status: up netbios-forward: disable type: tunnel netflow-sampler: disable sflow-sampler: disable scan-botnet-connections: disable explicit-web-proxy: disable explicit-ftp-proxy: disable mtu-override: disable wccp: disable

Verify OSPF status on the FortiGate:

# get router info ospf status
Routing Process "ospf 0" with ID 10.1.1.254
Process uptime is 2 minutes
Process bound to VRF default
Conforms to RFC2328, and RFC1583Compatibility flag is disabled
Supports only single TOS(TOS0) routes
Supports opaque LSA
Do not support Restarting
SPF schedule delay 5 secs, Hold time between two SPFs 10 secs
Refresh timer 10 secs
Number of incomming current DD exchange neighbors 0/5
Number of outgoing current DD exchange neighbors 0/5
Number of external LSA 0. Checksum 0x000000
Number of opaque AS LSA 0. Checksum 0x000000
Number of non-default external LSA 0
External LSA database is unlimited.
Number of LSA originated 4
Number of LSA received 21
Number of areas attached to this router: 1
    Area 0.0.0.0 (BACKBONE)
        Number of interfaces in this area is 2(2)
        Number of fully adjacent neighbors in this area is 1
        Area has no authentication
        SPF algorithm last executed 00:01:35.330 ago
        SPF algorithm executed 21 times
        Number of LSA 2. Checksum 0x0024d5

# get router info ospf interface
port2 is up, line protocol is up

Internet Address 10.1.1.254/24, Area 0.0.0.0, MTU 1500
Process ID 0, Router ID 10.1.1.254, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 10.1.1.254, Interface Address 10.1.1.254
No backup designated router on this network
Timer intervals configured, Hello 10.000, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:09
Neighbor Count is 0, Adjacent neighbor count is 0
Crypt Sequence Number is 2983
Hello received 0 sent 16, DD received 0 sent 0
LS-Req received 0 sent 0, LS-Upd received 0 sent 0
LS-Ack received 0 sent 0, Discarded 0
toCisco is up, line protocol is up

Internet Address 10.255.255.1/32, Area 0.0.0.0, MTU 1476
Process ID 0, Router ID 10.1.1.254, Network Type POINTOPOINT, Cost: 100
Transmit Delay is 1 sec, State Point-To-Point
Timer intervals configured, Hello 10.000, Dead 40, Wait 40, Retransmit 5
    Hello due in 00:00:05
Neighbor Count is 1, Adjacent neighbor count is 1
Crypt Sequence Number is 2983
Hello received 15 sent 16, DD received 5 sent 6
LS-Req received 2 sent 1, LS-Upd received 5 sent 9
LS-Ack received 5 sent 4, Discarded 0

# get router info ospf neighbor

OSPF process 0:
Neighbor ID     Pri   State           Dead Time   Address         Interface
10.2.2.254        1   Full/ -         00:00:30    10.255.255.2    toCisco

# get router info ospf database brief

                Router Link States (Area 0.0.0.0)

Link ID         ADV Router      Age Seq#     CkSum Flag Link count
10.1.1.254      10.1.1.254      130 80000005 10f5 0031 4
10.2.2.254      10.2.2.254      144 80000003 13e0 0002 3

# get router info ospf route
C 10.1.1.0/24 [1] is directly connected, port2, Area 0.0.0.0
O 10.2.2.0/24 [101] via 10.255.255.2, toCisco, Area 0.0.0.0
O 10.255.255.0/30 [1100] via 10.255.255.2, toCisco, Area 0.0.0.0
C 10.255.255.1/32 [100] is directly connected, toCisco, Area 0.0.0.0

Verify the routing table (RIB):

# get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S*      0.0.0.0/0 [10/0] via 198.51.100.254, port1
C       10.1.1.0/24 is directly connected, port2
O       10.2.2.0/24 [110/101] via 10.255.255.2, toCisco, 00:06:10
O       10.255.255.0/30 [110/1100] via 10.255.255.2, toCisco, 00:06:10
C       10.255.255.1/32 is directly connected, toCisco
C       10.255.255.2/32 is directly connected, toCisco
C       172.16.31.0/24 is directly connected, port10
S       192.0.2.2/32 [10/0] is directly connected, ipsec
C       198.51.100.0/24 is directly connected, port1

Verify the kernel routes (FIB):

# get router info kernel
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->10.1.1.0/32 pref=10.1.1.254 gwy=0.0.0.0 dev=4(port2)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->10.1.1.254/32 pref=10.1.1.254 gwy=0.0.0.0 dev=4(port2)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->10.1.1.255/32 pref=10.1.1.254 gwy=0.0.0.0 dev=4(port2)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->10.255.255.1/32 pref=10.255.255.1 gwy=0.0.0.0 dev=20(toCisco)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.0.0.0/32 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.0.0.0/8 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.0.0.1/32 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.255.255.255/32 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->172.16.31.0/32 pref=172.16.31.1 gwy=0.0.0.0 dev=12(port10)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->172.16.31.1/32 pref=172.16.31.1 gwy=0.0.0.0 dev=12(port10)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->172.16.31.255/32 pref=172.16.31.1 gwy=0.0.0.0 dev=12(port10)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->198.51.100.0/32 pref=198.51.100.1 gwy=0.0.0.0 dev=3(port1)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->198.51.100.1/32 pref=198.51.100.1 gwy=0.0.0.0 dev=3(port1)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->198.51.100.255/32 pref=198.51.100.1 gwy=0.0.0.0 dev=3(port1)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=198.51.100.254 dev=3(port1)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->10.1.1.0/24 pref=10.1.1.254 gwy=0.0.0.0 dev=4(port2)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->10.2.2.0/24 pref=0.0.0.0 gwy=10.255.255.2 dev=20(toCisco)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->10.255.255.0/30 pref=0.0.0.0 gwy=10.255.255.2 dev=20(toCisco)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->10.255.255.2/32 pref=10.255.255.1 gwy=0.0.0.0 dev=20(toCisco)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->172.16.31.0/24 pref=172.16.31.1 gwy=0.0.0.0 dev=12(port10)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->192.0.2.2/32 pref=0.0.0.0 gwy=0.0.0.0 dev=15(ipsec)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->198.51.100.0/24 pref=198.51.100.1 gwy=0.0.0.0 dev=3(port1)

Verify the IPsec tunnel status.

Phase1 IKE SA:

# diagnose vpn ike gateway list

vd: root/0
name: ipsec
version: 1
interface: port1 3
addr: 198.51.100.1:500 -> 192.0.2.2:500
created: 1598s ago
auto-discovery: 0
IKE SA: created 1/1 established 1/1 time 7230/7230/7230 ms
IPsec SA: created 1/1 established 1/1 time 7380/7380/7380 ms

id/spi: 4 637dd492a91aa3aa/7fce7e98f4817222
direction: initiator
status: established 1598-1591s ago = 7230ms
proposal: aes128-sha1
key: 60ceeb19c9ff8e7f-29d9ea3fc2be4e2a
lifetime/rekey: 86400/84508
DPD sent/recv: 00000000/00000000

Phase2 IPsec SA:

# diagnose vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=ipsec ver=1 serial=1 198.51.100.1:0->192.0.2.2:0
bound_if=3 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/0
proxyid_num=1 child_num=0 refcnt=18 ilast=6 olast=6 auto-discovery=0
stat: rxp=191 txp=231 rxb=29240 txb=22352
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=3
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=ipsec proto=47 sa=1 ref=2 serial=2 auto-negotiate transport-mode
src: 47:0.0.0.0/0.0.0.0:0
dst: 47:0.0.0.0/0.0.0.0:0
SA: ref=3 options=27 type=00 soft=0 mtu=1454 expire=1979/0B replaywin=2048 seqno=e8 esn=0 replaywin_lastseq=000000c9
life: type=01 bytes=0/0 timeout=3576/3600
dec: spi=6ede198b esp=aes key=16 11ed2d9b5665a96f64569a9db743bb8a
ah=sha1 key=20 ab1074130590c886585d7aebfe319c1bd077eeb0
enc: spi=e837e17f esp=aes key=16 676c2881a5ea4fb4bb824401da7543f0
ah=sha1 key=20 b2f5985d9b248acd04e095570ec6fec924be0e28
dec:pkts/bytes=191/16384, enc:pkts/bytes=231/32536

Verify the sniffer trace when PC1 attempts to ping PC2.

ICMP traffic between PC1 and PC2:

# diag sniffer packet any 'host 10.2.2.2 and icmp' 4
interfaces=[any]
filters=[host 10.2.2.2 and icmp]

2.831172 port2 in 10.1.1.1 -> 10.2.2.2: icmp: echo request
2.831287 toCisco out 10.1.1.1 -> 10.2.2.2: icmp: echo request
2.868716 toCisco in 10.2.2.2 -> 10.1.1.1: icmp: echo reply
2.868764 port2 out 10.2.2.2 -> 10.1.1.1: icmp: echo reply

3.831141 port2 in 10.1.1.1 -> 10.2.2.2: icmp: echo request
3.831185 toCisco out 10.1.1.1 -> 10.2.2.2: icmp: echo request
3.857989 toCisco in 10.2.2.2 -> 10.1.1.1: icmp: echo reply
3.858025 port2 out 10.2.2.2 -> 10.1.1.1: icmp: echo reply

4.831918 port2 in 10.1.1.1 -> 10.2.2.2: icmp: echo request
4.831944 toCisco out 10.1.1.1 -> 10.2.2.2: icmp: echo request
4.867633 toCisco in 10.2.2.2 -> 10.1.1.1: icmp: echo reply
4.867658 port2 out 10.2.2.2 -> 10.1.1.1: icmp: echo reply

5.833020 port2 in 10.1.1.1 -> 10.2.2.2: icmp: echo request
5.833055 toCisco out 10.1.1.1 -> 10.2.2.2: icmp: echo request
5.856450 toCisco in 10.2.2.2 -> 10.1.1.1: icmp: echo reply
5.856489 port2 out 10.2.2.2 -> 10.1.1.1: icmp: echo reply

6.833319 port2 in 10.1.1.1 -> 10.2.2.2: icmp: echo request
6.833359 toCisco out 10.1.1.1 -> 10.2.2.2: icmp: echo request
6.855880 toCisco in 10.2.2.2 -> 10.1.1.1: icmp: echo reply
6.855910 port2 out 10.2.2.2 -> 10.1.1.1: icmp: echo reply

20 packets received by filter
0 packets dropped by kernel

GRE traffic (protocol 47) sent and received by the FortiGate:

# diagnose sniffer packet any 'ip proto 47' 4
interfaces=[any]
filters=[ip proto 47]

1.920502 ipsec out 198.51.100.1 -> 192.0.2.2: gre: length 88 proto-800
1.976693 ipsec in 192.0.2.2 -> 198.51.100.1: gre: length 88 proto-800

2.920556 ipsec out 198.51.100.1 -> 192.0.2.2: gre: length 88 proto-800
2.958866 ipsec in 192.0.2.2 -> 198.51.100.1: gre: length 88 proto-800

3.921789 ipsec out 198.51.100.1 -> 192.0.2.2: gre: length 88 proto-800
3.972762 ipsec in 192.0.2.2 -> 198.51.100.1: gre: length 88 proto-800

4.922061 ipsec out 198.51.100.1 -> 192.0.2.2: gre: length 88 proto-800
4.960529 ipsec in 192.0.2.2 -> 198.51.100.1: gre: length 88 proto-800

5.922551 ipsec out 198.51.100.1 -> 192.0.2.2: gre: length 88 proto-800
5.957651 ipsec in 192.0.2.2 -> 198.51.100.1: gre: length 88 proto-800

10 packets received by filter
0 packets dropped by kernel

IPsec traffic (ESP) sent and received by the FortiGate:

# diagnose sniffer packet any 'esp' 4
interfaces=[any]
filters=[esp]

3.315417 port1 out 198.51.100.1 -> 192.0.2.2: ip-proto-50 132
3.364389 port1 in 192.0.2.2 -> 198.51.100.1: ip-proto-50 132

4.316114 port1 out 198.51.100.1 -> 192.0.2.2: ip-proto-50 132
4.363084 port1 in 192.0.2.2 -> 198.51.100.1: ip-proto-50 132

5.317221 port1 out 198.51.100.1 -> 192.0.2.2: ip-proto-50 132
5.360981 port1 in 192.0.2.2 -> 198.51.100.1: ip-proto-50 132

6.318920 port1 out 198.51.100.1 -> 192.0.2.2: ip-proto-50 132
6.359161 port1 in 192.0.2.2 -> 198.51.100.1: ip-proto-50 132

7.319719 port1 out 198.51.100.1 -> 192.0.2.2: ip-proto-50 132
7.373217 port1 in 192.0.2.2 -> 198.51.100.1: ip-proto-50 132

10 packets received by filter
0 packets dropped by kernel

Verify the debug flow when PC1 attempts to ping PC2.

Clear the filter:

# diag debug flow filter clear

Show function names:

# diag debug flow show function-name enable

Show trace messages about iprope:

# diag debug flow show iprope enable

# diag debug flow filter proto 1

# diag debug flow filter addr 10.2.2.2

# diag debug flow show console enable

Show trace messages from the console:

# diag debug flow trace start 1000

# diag debug enable

Example ICMP echo-request from PC1 to PC2:

id=20085 trace_id=9 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 10.1.1.1:172->10.2.2.2:2048) from port2. type=8, code=0, id=172, seq=1."
id=20085 trace_id=9 func=init_ip_session_common line=4944 msg="allocate a new session-000003d5"
id=20085 trace_id=9 func=iprope_dnat_check line=4659 msg="in-[port2], out-[]"
id=20085 trace_id=9 func=iprope_dnat_check line=4672 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=20085 trace_id=9 func=vf_ip_route_input_common line=2586 msg="find a route: flag=04000000 gw-10.255.255.2 via toCisco"
id=20085 trace_id=9 func=iprope_fwd_check line=636 msg="in-[port2], out-[toCisco], skb_flags-02000000, vid-0"
id=20085 trace_id=9 func=__iprope_check line=2049 msg="gnum-100004, check-ffffffffa001e70e"
id=20085 trace_id=9 func=__iprope_check_one_policy line=1823 msg="checked gnum-100004 policy-1, ret-matched, act-accept"
id=20085 trace_id=9 func=__iprope_user_identity_check line=1648 msg="ret-matched"
id=20085 trace_id=9 func=__iprope_check line=2049 msg="gnum-4e20, check-ffffffffa001e70e"
id=20085 trace_id=9 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=9 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=9 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=9 func=__iprope_check line=2068 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=20085 trace_id=9 func=__iprope_check_one_policy line=2020 msg="policy-1 is matched, act-accept"
id=20085 trace_id=9 func=__iprope_check line=2068 msg="gnum-100004 check result: ret-matched, act-accept, flag-08010000, flag2-00004000"
id=20085 trace_id=9 func=iprope_fwd_auth_check line=688 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1"
id=20085 trace_id=9 func=fw_forward_handler line=697 msg="Allowed by Policy-1:"
id=20085 trace_id=9 func=ipsecdev_hard_start_xmit line=157 msg="enter IPsec interface-ipsec"
id=20085 trace_id=9 func=esp_output4 line=859 msg="IPsec encrypt/auth"
id=20085 trace_id=9 func=ipsec_output_finish line=498 msg="send to 198.51.100.254 via intf-port1"

ICMP echo-reply from PC2 to PC1:

id=20085 trace_id=10 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 10.2.2.2:172->10.1.1.1:0) from toCisco. type=0, code=0, id=172, seq=1."
id=20085 trace_id=10 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-000003d5, reply direction"
id=20085 trace_id=10 func=vf_ip_route_input_common line=2586 msg="find a route: flag=00000000 gw-10.1.1.1 via port2"

Verify the session:

# diag sys session filter clear

# diag sys session filter dst 10.2.2.2

# diag sys session filter proto 1

# diag sys session list

session info: proto=1 proto_state=00 duration=4 expire=55 timeout=0 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
ha_id=0 policy_dir=0 tunnel=ipsec/ vlan_cos=0/255
state=may_dirty
statistic(bytes/packets/allow_err): org=84/1/1 reply=84/1/1 tuples=2
tx speed(Bps/kbps): 19/0 rx speed(Bps/kbps): 19/0
orgin->sink: org pre->post, reply pre->post dev=4->20/20->4 gwy=10.255.255.2/10.1.1.1
hook=pre dir=org act=noop 10.1.1.1:172->10.2.2.2:8(0.0.0.0:0)
hook=post dir=reply act=noop 10.2.2.2:172->10.1.1.1:0(0.0.0.0:0)
misc=0 policy_id=1 auth_info=0 chk_client_info=0 vd=0
serial=000003d5 tos=ff/ff app_list=0 app=0 url_cat=0
dd_type=0 dd_mode=0
total session 1

Below are example of decrypted GRE over IPsec packets containing PC1’s Echo-Request.

The ESP (IPsec) packet:

Ethernet II, Src: MS-NLB-PhysServer-09_69:5c:04:02 (02:09:69:5c:04:02), Dst: MS-NLB-PhysServer-09_69:5c:04:01 (02:09:69:5c:04:01)

    Destination: MS-NLB-PhysServer-09_69:5c:04:01 (02:09:69:5c:04:01)
    Source: MS-NLB-PhysServer-09_69:5c:04:02 (02:09:69:5c:04:02)
    Type: IPv4 (0x0800)

Internet Protocol Version 4, Src: 198.51.100.1, Dst: 192.0.2.2

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 152
    Identification: 0x5c6c (23660)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 64
    Protocol: Encap Security Payload (50)

    Header checksum: 0xf190 [correct]
    Source: 198.51.100.1
    Destination: 192.0.2.2

Encapsulating Security Payload
    ESP SPI: 0xe837e17f (3895976319)
    ESP Sequence: 367
    ESP IV: 17271258c2b5ebda8ca6dda8b4bfa956
    Pad: 010203040506
    ESP Pad Length: 6
    Next header: Generic Routing Encapsulation (0x2f)

Authentication Data [correct]

The original IP packet carried inside the GRE packet:

Generic Routing Encapsulation (IP)
Flags and Version: 0x0000
Protocol Type: IP (0x0800)

Internet Protocol Version 4, Src: 10.1.1.1, Dst: 10.2.2.2

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 84
    Identification: 0x3bec (15340)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 63
    Protocol: ICMP (1)

    Header checksum: 0xe8b7 [correct]
    Source: 10.1.1.1
    Destination: 10.2.2.2

Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xbb92 [correct]
    Identifier (BE): 174 (0x00ae)
    Identifier (LE): 44544 (0xae00)
    Sequence number (BE): 1 (0x0001)
    Sequence number (LE): 256 (0x0100)
    Data (48 bytes)

Troubleshooting scenario

In some cases, after creating the static route for the public interface, the IPsec tunnel goes down.

This may occur under conditions of a staticroute for the public interface being configured with an Administrative distance AD of 1 (the default), while the AD value on the static route for the IPsec tunnel interface (installed by the configuration of ip address overlapping where the IPsec tunnel interface) gets assigned an ip address on the same subnet as the public interface:

# config system settings

set allow-subnet-overlap enable

end

In this scenario, the solution is to change the AD distance on the route for the public interface to match the same value on the route for the IPsec tunnel.