Technical Note : IPSec VPN between FortiGate in Transparent mode (TP) on different subnets - Configuration example and troubleshooting

rmetzger · ‎11-17-2009

Purpose
This article provides a configuration example for IPSec VPN tunnels between two FortiGate in Transparent Mode (TP) on different subnets, as well as some troubleshooting steps.

Scope
All FortiOS 3.0 and 4.0

Diagram
The following network scenario is used to illustrate this example :

Expectations, Requirements
Requirements for this example :

Because both FortiGate are not in the same L2 network (separated by routers), the hosts on each side must be on different subnets.
FortiGate management IP addresses must be in the same subnet as the local hosts
The default gateways (router1 and router2) for PC1 and PC2 must be behind port2 in order for the FortiGate to match the appropriate Encrypt firewall policy (port1 --> port2)

PC1 and PC2 which are in different subnets will be able to communicate via the IPSec tunnel.

Configuration
Only relevant parts of configuration are provided

FortiGate 1

config system settings
    set opmode transparent
    set manageip 10.1.1.100/255.255.255.0
end

config router static
    edit 1
        set gateway 10.1.1.254
    next
end

config firewall address
    edit "10.1.1.0/24"
        set subnet 10.1.1.0 255.255.255.0
    next
    edit "10.2.2.0/24"
        set subnet 10.2.2.0 255.255.255.0
    next
end

config vpn ipsec phase1
    edit "to_FGT2"
        set proposal 3des-sha1 aes128-sha1 des-md5
        set remote-gw 10.2.2.100
        set psksecret fortinet
    next
end

config vpn ipsec phase2
    edit "to_FGT2"
        set keepalive enable
        set phase1name "to_FGT2"
        set proposal 3des-sha1 aes128-sha1
        set dst-subnet 10.2.2.0 255.255.255.0
        set src-subnet 10.1.1.0 255.255.255.0
    next
end

config firewall policy
    edit 1
        set srcintf "port1"
        set dstintf "port2"
            set srcaddr "10.1.1.0/24"
            set dstaddr "10.2.2.0/24"
        set action ipsec
        set schedule "always"
            set service "ANY"
        set inbound enable
        set outbound enable
        set vpntunnel "to_FGT2"
    next
end

FortiGate 2

config system settings
    set opmode transparent
    set manageip 10.2.2.100/255.255.255.0
end

config router static
    edit 1
        set gateway 10.2.2.254
    next
end

config firewall address
    edit "10.1.1.0/24"
        set subnet 10.1.1.0 255.255.255.0
    next
    edit "10.2.2.0/24"
        set subnet 10.2.2.0 255.255.255.0
    next
end

config vpn ipsec phase1
    edit "to_FGT1"
        set nattraversal disable
        set proposal 3des-sha1 aes128-sha1 des-md5
        set remote-gw 10.1.1.100
        set psksecret fortinet
    next
end

config vpn ipsec phase2
    edit "to_FGT1"
        set keepalive enable
        set phase1name "to_FGT1"
        set proposal 3des-sha1 aes128-sha1
        set dst-subnet 10.1.1.0 255.255.255.0
        set src-subnet 10.2.2.0 255.255.255.0
    next
end

config firewall policy
    edit 1
        set srcintf "port1"
        set dstintf "port2"
            set srcaddr "10.2.2.0/24"
            set dstaddr "10.1.1.0/24"
        set action ipsec
        set schedule "always"
            set service "ANY"
        set inbound enable
        set outbound enable
        set vpntunnel "to_FGT1"
    next
end

Troubleshooting

Steps given when PC1 pings PC2

1- Verify if IPSec tunnels are up

FGT1 # diagnose vpn tunnel list

list all ipsec tunnel in vd 0
------------------------------------------------------
name=to_FGT2 ver=0 serial=1 10.1.1.100:0->10.2.2.100:0 lgwy=dyn tun=tunnel mode=
auto bound_if=0
proxyid_num=1 child_num=0 refcnt=7 ilast=0 olast=0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=active on=1 idle=5000ms retry=3 count=0 seqno=1455
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=to_FGT2 proto=0 sa=0 ref=1 auto_negotiate=0 serial=5
src: 10.1.1.0/255.255.255.0:0
dst: 10.2.2.0/255.255.255.0:0

The above tunnel is down (output given as example) !

FGT2 # diagnose vpn tunnel list

list all ipsec tunnel in vd 0
------------------------------------------------------
name=to_FGT1 10.2.2.100:0->10.1.1.100:0 lgwy=dyn tun=tunnel mode=auto bound_if=0
proxyid_num=1 child_num=0 refcnt=7 ilast=1 olast=1
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=active on=1 idle=5000ms retry=3 count=0 seqno=21
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=to_FGT1 proto=0 sa=1 ref=2 auto_negotiate=0 serial=1
src: 10.2.2.0/255.255.255.0:0
dst: 10.1.1.0/255.255.255.0:0
SA: ref=3 options=00000009 type=00 soft=0 mtu=1436 expire=1771 replaywin=0 seq
no=1
life: type=01 bytes=0/0 timeout=1773/1800
dec: spi=c1a8e951 esp=3des key=24 9213fdf22b150e01abb3535d1a647044eebf772b92f2
f7ee
ah=sha1 key=20 66a38bf99f0b2d234f64b5a05187995c4f56f6bb
enc: spi=322067b4 esp=3des key=24 720e5680329937fb3630b7ed70bd41bb3114d3c269ae
8b61
ah=sha1 key=20 e316113eb6ea03b014b3a5f9c1a3bd386637801a

The above tunnel is up !

2- Verify that destination local hosts are seen in the ARP table (necessary for IPSec despite being in TP mode)

FGT2 # get system arp

Address           Age(min)   Hardware Addr      Interface
10.2.2.10         2          00:50:56:00:76:04 root.b
10.2.2.254        0          00:09:0f:30:29:e4 root.b

3- Using the debug flow command on the initiator side (example on FortiGate1)

FGT1 # diagnose debug flow filter addr 10.1.1.10
FGT1 # diagnose debug flow show console enable
FGT1 # diagnose debug enable
FGT1 # diagnose debug flow trace start 50

FGT1 # id=36870 trace_id=615 msg="vd-root received a packet(proto=1, 10.1.1.10:512->10.2.2.10:8) from port1."
id=36870 trace_id=615 msg="allocate a new session-00000636"
id=36870 trace_id=615 msg="Allowed by Policy-1: encrypt"
id=36870 trace_id=615 msg="enter IPsec tunnel-to_FGT2"
id=36870 trace_id=615 msg="SA is not ready yet, drop"
id=36870 trace_id=616 msg="vd-root received a packet(proto=1, 10.1.1.10:512->10.2.2.10:8) from port1."
id=36870 trace_id=616 msg="Find an existing session, id-00000636, original direction"
id=36870 trace_id=616 msg="enter IPsec tunnel-to_FGT2"
id=36870 trace_id=616 msg="encrypted, and send to 10.2.2.100 with source 10.1.1.100"
id=36870 trace_id=616 msg="send out via dev-port2, dst-mac-00:09:0f:30:29:e0"
id=36870 trace_id=617 msg="vd-root received a packet(proto=1, 10.1.1.10:512->10.2.2.10:8) from port1."
id=36870 trace_id=617 msg="Find an existing session, id-00000636, original direction"
id=36870 trace_id=617 msg="enter IPsec tunnel-to_FGT2"
id=36870 trace_id=617 msg="encrypted, and send to 10.2.2.100 with source 10.1.1.100"
id=36870 trace_id=617 msg="send out via dev-port2, dst-mac-00:09:0f:30:29:e0"
id=36870 trace_id=618 msg="vd-root received a packet(proto=1, 10.2.2.10:512->10.1.1.10:0) from port2."
id=36870 trace_id=618 msg="Find an existing session, id-00000636, reply direction"
id=36870 trace_id=618 msg="send out via dev-port1, dst-mac-00:50:56:00:76:03"
id=36870 trace_id=619 msg="vd-root received a packet(proto=1, 10.1.1.10:512->10.2.2.10:8) from port1."
id=36870 trace_id=619 msg="Find an existing session, id-00000636, original direction"
id=36870 trace_id=619 msg="enter IPsec tunnel-to_FGT2"
id=36870 trace_id=619 msg="encrypted, and send to 10.2.2.100 with source 10.1.1.100"
id=36870 trace_id=619 msg="send out via dev-port2, dst-mac-00:09:0f:30:29:e0"
id=36870 trace_id=620 msg="vd-root received a packet(proto=1, 10.2.2.10:512->10.1.1.10:0) from port2."
id=36870 trace_id=620 msg="Find an existing session, id-00000636, reply direction"
id=36870 trace_id=620 msg="send out via dev-port1, dst-mac-00:50:56:00:76:03"

Note : the message "id=36870 trace_id=615 msg="SA is not ready yet, drop" simply means that the tunnel was not up yet.

4- Using the debug flow command on the receiver side (example on FortiGate2)

FGT2 # diagnose debug flow filter addr 10.1.1.10
FGT2 # diagnose debug flow show console enable
FGT2 # diagnose debug enable
FGT2 # diagnose debug flow trace start 50

FGT2 # id=36870 trace_id=51 msg="vd-root received a packet(proto=1, 10.1.1.10:512->10.2.2.10:8) from port2."
id=36870 trace_id=51 msg="allocate a new session-00000435"
id=36870 trace_id=51 msg="Allowed by Policy-1:"
id=36870 trace_id=51 msg="send out via dev-port1, dst-mac-00:50:56:00:76:04"
id=36870 trace_id=52 msg="vd-root received a packet(proto=1, 10.2.2.10:512->10.1.1.10:0) from port1."
id=36870 trace_id=52 msg="Find an existing session, id-00000435, reply direction"
id=36870 trace_id=52 msg="enter IPsec tunnel-to_FGT1"
id=36870 trace_id=52 msg="encrypted, and send to 10.1.1.100 with source 10.2.2.100"
id=36870 trace_id=52 msg="send out via dev-port2, dst-mac-00:09:0f:30:29:e4"

5- Using the sniffer trace (example on FortiGate2)

FGT2 # diagnose sniffer packet any "host 10.2.2.10" 4

9.460021 root.b out arp who-has 10.2.2.10 tell 10.2.2.100
9.460028 port2 out arp who-has 10.2.2.10 tell 10.2.2.100
9.460034 port1 out arp who-has 10.2.2.10 tell 10.2.2.100
9.460462 port1 in arp reply 10.2.2.10 is-at 0:50:56:0:76:4
9.460462 root.b in arp reply 10.2.2.10 is-at 0:50:56:0:76:4
[...]
49.477368 port2 in 10.1.1.10 -> 10.2.2.10: icmp: echo request
49.477444 port1 out 10.1.1.10 -> 10.2.2.10: icmp: echo request
49.477898 port1 in 10.2.2.10 -> 10.1.1.10: icmp: echo reply
50.510023 port2 in 10.1.1.10 -> 10.2.2.10: icmp: echo request
50.510079 port1 out 10.1.1.10 -> 10.2.2.10: icmp: echo request
50.510524 port1 in 10.2.2.10 -> 10.1.1.10: icmp: echo reply

Note : the above ARP process in Transparent mode with IPSec is allowing to Fortigate to :
- Identify the MAC address of the destination device 10.2.2.10
- Populate the MAC table (see below), which in turn will give a destination interface and allow a Firewall policy look-up

6 - Check the FDB entries for the destination

FGT_2 # diagnose netlink brctl name host root.b

port no device devname mac addr ttl
[..]
1 2 port1 00:50:56:00:76:04 0