Help
FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
sha-1_FTNT
Staff
Staff

Created on ‎01-31-2017 08:57 AM Edited on ‎06-09-2022 09:24 PM By Moderator

Technical Tip: Building a Layer-2 VPN with VxLAN over IPsec

Description

This article describes how to build a Layer-2 VPN between two FortiGates using VxLAN over IPsec.


Scope

Support for VxLAN over IPsec was added in FortiOS 5.4


Solution

 

Diagram

 

 

The following topology is used:
 
PC1(.1) – 192.168.100.0/24 - port2-[ FG1 ]-port1-(198.51.100.1) ====
                                                                    I
                                                   (Internet)       I VxLAN-over-IPsec tunnel

 

                                                                    I
PC2(.2) - 192.168.100.0/24 - port2-[ FG2 ]-port1-(203.0.113.2) =====

  

Design

 

  • Layer-2 VPN (aka Ethernet-VPN, EVPN) subnet 192.168.100.0/24 spans over two sites which are connected via a VxLAN-IPsec tunnel

     

  • A software switch is configured to bridge Ethernet frames between the local LAN and the VxLAN-IPsec tunnel

     

  • Ethernet frames forwarded to the remote site are encapsulated in UDP (VxLAN) then protected with IPsec (VxLAN over IPsec)

     

 

Limitations

 

  • In FortiOS 5.4, VXLAN is only supported as an encapsulation method within the configuration of an IPsec tunnel. Native VXLAN tunnel cannot be configured in FortiOS 5.4.
  • Support for native VxLAN is available as of FortiOS 5.6
  • IPsec processing cannot be hardware offloaded to NPU (NP6, NP4) when IPsec is configured with VxLAN encapsulation
  • FortiOS does not send back an ICMP “destination unreachable, fragmentation needed and DF set” to the source when an IP packet with the DF bit set and a size greater than the tunnel MTU cannot be forwarded inside the VxLAN-IPsec tunnel.

The workaround is to stop honoring the DF bit:

 

config system global

 

    set honor-df disable

 

end

 

 

  • FortiOS 5.4 VxLAN encapsulation within IPsec only supports forwarding untagged frames
The forwarding of tagged frames (802.1Q) is not supported.
More specifically, tagged frames with a size greater than the tunnel MTU are not forwarded/fragmented even if the inner IP packet has DF=0.
This kind of configuration is therefore NOT supported:

     config system interface

 

         edit "port2"                        // connected to a 802.1Q trunk port

 

             set vlanforward enable          // to forward Vlans undeclared in FortiOS

         next

 

         edit "VxLan-IPsec"

 

             set vlanforward enable

         next

 

     end

 

 

     config system switch-interface

 

         edit "soft_switch"

 

             set member "port2" "VxLan-IPsec"

 

               next
           end

Configuration

 

  

CLI configuration of FortiGate 1

 

 
#
# Port1 is the Internet-facing interface
# Port2 is the LAN interface which is placed in a soft-switch
#
 
config system interface
    edit "port1"
        set ip 198.51.100.1 255.255.255.0
        set alias "Internet"
    next
    edit "port2"
        set alias "LAN"
    next
end
 
 
#
# Configure an IPsec tunnel with VxLAN encapsulation
#
 
config vpn ipsec phase1-interface
    edit "VxLan-IPsec"
        set interface "port1"
        set peertype any
        set proposal aes128-sha1
        set encapsulation vxlan    // data packets are subject to VxLAN encapsulation prior to IPsec processing
        set remote-gw 203.0.113.2
        set psksecret someSecureKey
    next
end
 
config vpn ipsec phase2-interface
    edit "VxLan-IPsec"
        set phase1name "VxLan-IPsec"
        set proposal aes128-sha1
    next
end
 
#
# Define the soft-switch used to bridge traffic between the local LAN (port2) and the remote LAN (VxLan-IPsec)
#
config system switch-interface
    edit "soft_switch"
        set vdom root
        set member "port2" "VxLan-IPsec"
        set intra-switch-policy explicit   // (optional) mandates firewall policy lookup for bridged traffic
    next
end
 
#
# Allow traffic between the local LAN (port2) and the remote LAN (VxLan-IPsec)
# This is mandatory since “set intra-switch-policy explicit” was used in the soft-switch
#
 
config firewall policy
    edit 1
        set srcintf "port2"
        set dstintf "VxLan-IPsec"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
    edit 2
        set srcintf "VxLan-IPsec"
        set dstintf "port2"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
end
 
#
# Internet Access
#
 
config router static
    edit 1
        set gateway 198.51.100.253
        set device "port1"
        set comment "default-route to Internet ISP"
    next
end
 
 
//
// Alternative scenario: The LAN side could be a Vlan (sub-)interface.
//

 

config system interface
    edit "vlan1234"
        set interface "port2"
        set vlanid 1234
    next
end

 

config system switch-interface
    edit "soft_switch"
        set vdom root
        set member "vlan1234" "VxLan-IPsec"
        set intra-switch-policy explicit
    next
end

 

config firewall policy
    edit 1
        set srcintf "vlan1234"
        set dstintf "VxLan-IPsec"
        …
    next
    edit 2
        set srcintf "VxLan-IPsec"
        set dstintf "vlan1234"
        …
    next
end

 

This alternative configuration does not conflict with the fact that FortiOS only supports the forwarding of untagged frames. The 802.1Q tag has already been stripped off when a frame reaches the vlan1234 (sub-)interface. 802.1Q tagged frames are received on the physical interface (here, port2), not on the vlan1234 (sub-)interface.
 

 

 

CLI configuration of FortiGate 2

 


config system interface
    edit "port1"
        set ip 203.0.113.2 255.255.255.0
        set alias "Internet"
    next
    edit "port2"
        set alias "LAN"
    next
end
 
config vpn ipsec phase1-interface
    edit "VxLan-IPsec"
        set interface "port1"
        set proposal aes128-sha1
        set encapsulation vxlan
        set remote-gw 198.51.100.1
        set psksecret someSecureKey
    next
end
 
config vpn ipsec phase2-interface
    edit "VxLan-IPsec"
        set phase1name "VxLan-IPsec"
        set proposal aes128-sha1
    next
end
 
config system switch-interface
    edit "soft_switch"
        set member "port2" "VxLan-IPsec"
        set intra-switch-policy explicit
    next
end
 
config firewall policy
    edit 1
        set srcintf "port2"
        set dstintf "VxLan-IPsec"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
    edit 2
        set srcintf "VxLan-IPsec"
        set dstintf "port2"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
end
 
config router static
    edit 1
        set gateway 203.0.113.254
        set device "port1"
        set comment "default-route to Internet ISP"
    next
end

  

Verification

 

 
Verify that PC1 can ping PC2

 

root@PC1:~# ping -c 5 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=10.1 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=8.45 ms
64 bytes from 192.168.100.2: icmp_seq=3 ttl=64 time=17.9 ms
64 bytes from 192.168.100.2: icmp_seq=4 ttl=64 time=8.28 ms
64 bytes from 192.168.100.2: icmp_seq=5 ttl=64 time=15.3 ms
 
--- 192.168.100.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4010ms
rtt min/avg/max/mdev = 8.282/12.042/17.964/3.905 ms
 
 
Verify the IPsec tunnel

 

FG1 # diag vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=VxLan-IPsec ver=1 serial=3 198.51.100.1:0->203.0.113.2:0
bound_if=3 lgwy=static/1 tun=intf/0 mode=auto/1 encap=VXLAN/6 options[0006]=edg
encap-addr: 198.51.100.1->203.0.113.2
proxyid_num=1 child_num=0 refcnt=14 ilast=16 olast=317 auto-discovery=0
stat: rxp=10 txp=11 rxb=1784 txb=1022
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=843
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=VxLan-IPsec proto=0 sa=1 ref=2 serial=1
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA: ref=3 options=2e type=00 soft=0 mtu=1390 expire=42473/0B replaywin=2048 seqno=c esn=0 replaywin_lastseq=0000000b
  life: type=01 bytes=0/0 timeout=43147/43200
  dec: spi=e1677820 esp=aes key=16 199f8427aaede3b3ae875320a76ce4c7
       ah=sha1 key=20 1698424168fb6834e68dfc57e927ca77c60da93c
  enc: spi=a8d52b6c esp=aes key=16 5e78151deb161b9c541b7fa2938b2a35
       ah=sha1 key=20 e94b8f491e5c519c5ea19519665d161421019c00
  dec:pkts/bytes=10/1284, enc:pkts/bytes=11/1968
 
 
Verify the soft-switch forwarding table

 

FG1 # diagnose netlink brctl name host soft_switch
show bridge control interface soft_switch host.
fdb: size=2048, used=4, num=4, depth=1
Bridge soft_switch host table
port no device  devname       mac addr            ttl     attributes
  2     4       port2         02:09:0f:00:02:02    2        Hit(2)      ß The MAC address of PC1 (local)
  1     21      VxLan-IPsec   02:09:0f:00:06:03    2        Hit(2)      ß The MAC address of PC2 (remote)
  2     4       port2         02:09:0f:00:02:03    0       Local Static
  1     21      VxLan-IPsec   d2:ff:26:ba:45:9c    0       Local Static

 

 

Troubleshooting

 

 
Verify that ARP traffic can flow through the VxLAN-IPsec tunnel
 
FG1 # diag sniffer packet any 'host 192.168.100.2 and arp' 4
interfaces=[any]
filters=[host 192.168.100.2 and arp]
12.601491 VxLan-IPsec in arp who-has 192.168.100.1 tell 192.168.100.2
12.601508 port2 out arp who-has 192.168.100.1 tell 192.168.100.2
12.601882 port2 in arp reply 192.168.100.1 is-at 2:9:f:0:2:2
12.601888 VxLan-IPsec out arp reply 192.168.100.1 is-at 2:9:f:0:2:2
 
4 packets received by filter
0 packets dropped by kernel
 
 
Verify the debug flow when PC1 attempts to ping PC2

 

FG1 # diag debug flow filter clear
FG1 # diag debug flow show function-name enable
show function name
 
FG1 # diag debug flow show iprope enable
show trace messages about iprope
 
FG1 # diag debug flow filter proto 1
FG1 # diag debug flow filter addr 192.168.100.2
FG1 # diag debug flow show console enable
show trace messages on console
 
FG1 # diag debug flow trace start 1000
FG1 # diag debug enable
 

## ICMP echo-request from PC1 to PC2

 

id=20085 trace_id=51 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 192.168.100.1:2871->192.168.100.2:2048) from port2. type=8, code=0, id=2871, seq=1."

id=20085 trace_id=51 func=init_ip_session_common line=4944 msg="allocate a new session-00002434"
id=20085 trace_id=51 func=iprope_dnat_check line=4659 msg="in-[port2], out-[]"
id=20085 trace_id=51 func=iprope_dnat_check line=4672 msg="result: skb_flags-06000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=20085 trace_id=51 func=iprope_fwd_check line=636 msg="in-[port2], out-[VxLan-IPsec], skb_flags-06000000, vid-0"
id=20085 trace_id=51 func=__iprope_check line=2049 msg="gnum-100004, check-ffffffffa001e70e"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-100004 policy-1, ret-matched, act-accept"
id=20085 trace_id=51 func=__iprope_user_identity_check line=1648 msg="ret-matched"
id=20085 trace_id=51 func=__iprope_check line=2049 msg="gnum-4e20, check-ffffffffa001e70e"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=51 func=__iprope_check line=2068 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=20085 trace_id=51 func=__iprope_check_one_policy line=2020 msg="policy-1 is matched, act-accept"
id=20085 trace_id=51 func=__iprope_check line=2068 msg="gnum-100004 check result: ret-matched, act-accept, flag-08010000, flag2-00004000"
id=20085 trace_id=51 func=iprope_fwd_auth_check line=688 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1"
id=20085 trace_id=51 func=br_fw_forward_handler line=537 msg="Allowed by Policy-1:"
id=20085 trace_id=51 func=__if_queue_push_xmit line=365 msg="send out via dev-VxLan-IPsec, dst-mac-02:09:0f:00:06:03"
id=20085 trace_id=51 func=ipsecdev_hard_start_xmit_vxlan line=185 msg="enter IPsec interface-VxLan-IPsec"
id=20085 trace_id=51 func=esp_output4 line=859 msg="IPsec encrypt/auth"
id=20085 trace_id=51 func=ipsec_output_finish line=498 msg="send to 198.51.100.253 via intf-port1"
 

## ICMP echo-reply from PC2 to PC1

 

id=20085 trace_id=52 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 192.168.100.2:2871->192.168.100.1:0) from VxLan-IPsec. type=0, code=0, id=2871, seq=1."
id=20085 trace_id=52 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-00002434, reply direction"
id=20085 trace_id=52 func=br_ipv4_fast_cb line=68 msg="enter fast path"
id=20085 trace_id=52 func=__if_queue_push_xmit line=365 msg="send out via dev-port2, dst-mac-02:09:0f:00:02:02"
 
 
 
Verify the session
 
FG1 # diag sys session filter clear
 
FG1 # diag sys session filter dst 192.168.100.2
 
FG1 # diag sys session filter proto 1
 
FG1 # diag sys session list
 
session info: proto=1 proto_state=00 duration=8 expire=51 timeout=0 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
ha_id=0 policy_dir=0 tunnel=VxLan-IPsec/ vlan_cos=0/255
state=may_dirty br
statistic(bytes/packets/allow_err): org=84/1/1 reply=84/1/1 tuples=2
tx speed(Bps/kbps): 10/0 rx speed(Bps/kbps): 10/0
orgin->sink: org pre->post, reply pre->post dev=4->21/21->4 gwy=0.0.0.0/0.0.0.0
hook=pre dir=org act=noop 192.168.100.1:2871->192.168.100.2:8(0.0.0.0:0)
hook=post dir=reply act=noop 192.168.100.2:2871->192.168.100.1:0(0.0.0.0:0)
misc=0 policy_id=1 auth_info=0 chk_client_info=0 vd=0
serial=00002434 tos=ff/ff app_list=0 app=0 url_cat=0
dd_type=0 dd_mode=0
 
total session 1
 
 
Example of a decrypted VxLAN-IPsec packet containing PC1’s Ethernet frame
 

 

## The ESP (IPsec) packet
 
Ethernet II, Src: MS-NLB-PhysServer-09_0f:00:04:02 (02:09:0f:00:04:02), Dst: MS-NLB-PhysServer-09_0f:00:04:01 (02:09:0f:00:04:01)
    Destination: MS-NLB-PhysServer-09_0f:00:04:01 (02:09:0f:00:04:01)
    Source: MS-NLB-PhysServer-09_0f:00:04:02 (02:09:0f:00:04:02)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 198.51.100.1, Dst: 203.0.113.2

 

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 184
    Identification: 0xc28d (49805)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 63
    Protocol: Encap Security Payload (50)
    Header checksum: 0x524f [correct]
    Source: 198.51.100.1
    Destination: 203.0.113.2
Encapsulating Security Payload
    ESP SPI: 0xa8d52b6d (2832542573)
    ESP Sequence: 12
    ESP IV: 5b8204e860038607
    ESP Pad Length: 0
    Next header: IPIP (0x04)
    Authentication Data [correct]

## The VxLAN packet carried inside the ESP (IPsec) packet

 

 
Internet Protocol Version 4, Src: 198.51.100.1, Dst: 203.0.113.2

 

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 134
    Identification: 0xc28b (49803)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 63
    Protocol: UDP (17)
    Header checksum: 0x52a4 [correct]
    Source: 198.51.100.1
    Destination: 203.0.113.2
User Datagram Protocol, Src Port: 4789, Dst Port: 4789

 

    Source Port: 4789
    Destination Port: 4789
    Length: 114
Virtual eXtensible Local Area Network
    Flags: 0x0000
        0... .... .... .... = GBP Extension: Not defined
        .... .... .0.. .... = Don't Learn: False
        .... 0... .... .... = VXLAN Network ID (VNI): False
        .... .... .... 0... = Policy Applied: False
        .000 .000 0.00 .000 = Reserved(R): 0x0000
    Group Policy ID: 0
    VXLAN Network Identifier (VNI): 0
    Reserved: 0
 
## PC1’s frame carried inside the VxLAN packet
 
Ethernet II, Src: MS-NLB-PhysServer-09_0f:00:02:02 (02:09:0f:00:02:02), Dst: MS-NLB-PhysServer-09_0f:00:06:03 (02:09:0f:00:06:03)

 

    Destination: MS-NLB-PhysServer-09_0f:00:06:03 (02:09:0f:00:06:03)
    Source: MS-NLB-PhysServer-09_0f:00:02:02 (02:09:0f:00:02:02)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.100.1, Dst: 192.168.100.2

 

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 84
    Identification: 0xcfff (53247)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 64
    Protocol: ICMP (1)
    Header checksum: 0x2155 [correct]
    Source: 192.168.100.1
    Destination: 192.168.100.2
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xc1c9 [correct]
    [Checksum Status: Good]
    Identifier (BE): 2888 (0x0b48)
    Identifier (LE): 18443 (0x480b)
    Sequence number (BE): 1 (0x0001)
    Sequence number (LE): 256 (0x0100)
    Data (48 bytes)



 

Labels:
51506
0 Kudos
Contributors

Contact Us