Created on 01-31-2017 08:57 AM Edited on 06-09-2022 09:24 PM By Anonymous
Description
This article describes how to build a Layer-2 VPN between two FortiGates using VxLAN over IPsec.
Scope
Support for VxLAN over IPsec was added in FortiOS 5.4
Solution
Diagram
I
Design
Limitations
The workaround is to stop honoring the DF bit:
config system global
set honor-df disable
end
config system interface
edit "port2" // connected to a 802.1Q trunk port
set vlanforward enable // to forward Vlans undeclared in FortiOS
next
edit "VxLan-IPsec"
set vlanforward enable
next
end
config system switch-interface
edit "soft_switch"
set member "port2" "VxLan-IPsec"
Configuration
CLI configuration of FortiGate 1
# # Port1 is the Internet-facing interface # Port2 is the LAN interface which is placed in a soft-switch # config system interface edit "port1" set ip 198.51.100.1 255.255.255.0 set alias "Internet" next edit "port2" set alias "LAN" next end # # Configure an IPsec tunnel with VxLAN encapsulation # config vpn ipsec phase1-interface edit "VxLan-IPsec" set interface "port1" set peertype any set proposal aes128-sha1 set encapsulation vxlan // data packets are subject to VxLAN encapsulation prior to IPsec processing set remote-gw 203.0.113.2 set psksecret someSecureKey next end config vpn ipsec phase2-interface edit "VxLan-IPsec" set phase1name "VxLan-IPsec" set proposal aes128-sha1 next end # # Define the soft-switch used to bridge traffic between the local LAN (port2) and the remote LAN (VxLan-IPsec) # config system switch-interface edit "soft_switch" set vdom root
set member "port2" "VxLan-IPsec"
set intra-switch-policy explicit // (optional) mandates firewall policy lookup for bridged traffic next end # # Allow traffic between the local LAN (port2) and the remote LAN (VxLan-IPsec) # This is mandatory since “set intra-switch-policy explicit” was used in the soft-switch # config firewall policy edit 1 set srcintf "port2" set dstintf "VxLan-IPsec" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" next edit 2 set srcintf "VxLan-IPsec" set dstintf "port2" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" next end # # Internet Access # config router static edit 1 set gateway 198.51.100.253 set device "port1" set comment "default-route to Internet ISP" next end //
// Alternative scenario: The LAN side could be a Vlan (sub-)interface. // config system interface edit "vlan1234" set interface "port2" set vlanid 1234 next end config system switch-interface edit "soft_switch" set vdom root
set member "vlan1234" "VxLan-IPsec" set intra-switch-policy explicit next end config firewall policy edit 1 set srcintf "vlan1234" set dstintf "VxLan-IPsec" … next edit 2 set srcintf "VxLan-IPsec" set dstintf "vlan1234" … next end This alternative configuration does not conflict with the fact that FortiOS only supports the forwarding of untagged frames. The 802.1Q tag has already been stripped off when a frame reaches the vlan1234 (sub-)interface. 802.1Q tagged frames are received on the physical interface (here, port2), not on the vlan1234 (sub-)interface. |
|
CLI configuration of FortiGate 2
config system interface edit "port1" set ip 203.0.113.2 255.255.255.0 set alias "Internet" next edit "port2" set alias "LAN" next end config vpn ipsec phase1-interface edit "VxLan-IPsec" set interface "port1" set proposal aes128-sha1 set encapsulation vxlan set remote-gw 198.51.100.1 set psksecret someSecureKey next end config vpn ipsec phase2-interface edit "VxLan-IPsec" set phase1name "VxLan-IPsec" set proposal aes128-sha1 next end config system switch-interface edit "soft_switch" set member "port2" "VxLan-IPsec" set intra-switch-policy explicit next end config firewall policy edit 1 set srcintf "port2" set dstintf "VxLan-IPsec" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" next edit 2 set srcintf "VxLan-IPsec" set dstintf "port2" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" next end config router static edit 1 set gateway 203.0.113.254 set device "port1" set comment "default-route to Internet ISP" next end |
Verification
root@PC1:~# ping -c 5 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data. 64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=10.1 ms 64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=8.45 ms 64 bytes from 192.168.100.2: icmp_seq=3 ttl=64 time=17.9 ms 64 bytes from 192.168.100.2: icmp_seq=4 ttl=64 time=8.28 ms 64 bytes from 192.168.100.2: icmp_seq=5 ttl=64 time=15.3 ms --- 192.168.100.2 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4010ms rtt min/avg/max/mdev = 8.282/12.042/17.964/3.905 ms |
FG1 # diag vpn tunnel list
list all ipsec tunnel in vd 0 ------------------------------------------------------ name=VxLan-IPsec ver=1 serial=3 198.51.100.1:0->203.0.113.2:0 bound_if=3 lgwy=static/1 tun=intf/0 mode=auto/1 encap=VXLAN/6 options[0006]=edg encap-addr: 198.51.100.1->203.0.113.2 proxyid_num=1 child_num=0 refcnt=14 ilast=16 olast=317 auto-discovery=0 stat: rxp=10 txp=11 rxb=1784 txb=1022 dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=843 natt: mode=none draft=0 interval=0 remote_port=0 proxyid=VxLan-IPsec proto=0 sa=1 ref=2 serial=1 src: 0:0.0.0.0/0.0.0.0:0 dst: 0:0.0.0.0/0.0.0.0:0 SA: ref=3 options=2e type=00 soft=0 mtu=1390 expire=42473/0B replaywin=2048 seqno=c esn=0 replaywin_lastseq=0000000b life: type=01 bytes=0/0 timeout=43147/43200 dec: spi=e1677820 esp=aes key=16 199f8427aaede3b3ae875320a76ce4c7 ah=sha1 key=20 1698424168fb6834e68dfc57e927ca77c60da93c enc: spi=a8d52b6c esp=aes key=16 5e78151deb161b9c541b7fa2938b2a35 ah=sha1 key=20 e94b8f491e5c519c5ea19519665d161421019c00 dec:pkts/bytes=10/1284, enc:pkts/bytes=11/1968 |
FG1 # diagnose netlink brctl name host soft_switch
show bridge control interface soft_switch host. fdb: size=2048, used=4, num=4, depth=1 Bridge soft_switch host table port no device devname mac addr ttl attributes 2 4 port2 02:09:0f:00:02:02 2 Hit(2) ß The MAC address of PC1 (local) 1 21 VxLan-IPsec 02:09:0f:00:06:03 2 Hit(2) ß The MAC address of PC2 (remote) 2 4 port2 02:09:0f:00:02:03 0 Local Static 1 21 VxLan-IPsec d2:ff:26:ba:45:9c 0 Local Static |
Troubleshooting
FG1 # diag sniffer packet any 'host 192.168.100.2 and arp' 4
interfaces=[any] filters=[host 192.168.100.2 and arp] 12.601491 VxLan-IPsec in arp who-has 192.168.100.1 tell 192.168.100.2 12.601508 port2 out arp who-has 192.168.100.1 tell 192.168.100.2 12.601882 port2 in arp reply 192.168.100.1 is-at 2:9:f:0:2:2 12.601888 VxLan-IPsec out arp reply 192.168.100.1 is-at 2:9:f:0:2:2 4 packets received by filter 0 packets dropped by kernel |
FG1 # diag debug flow filter clear
FG1 # diag debug flow show function-name enable show function name FG1 # diag debug flow show iprope enable show trace messages about iprope FG1 # diag debug flow filter proto 1 FG1 # diag debug flow filter addr 192.168.100.2 FG1 # diag debug flow show console enable show trace messages on console FG1 # diag debug flow trace start 1000 FG1 # diag debug enable ## ICMP echo-request from PC1 to PC2
id=20085 trace_id=51 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 192.168.100.1:2871->192.168.100.2:2048) from port2. type=8, code=0, id=2871, seq=1." id=20085 trace_id=51 func=init_ip_session_common line=4944 msg="allocate a new session-00002434"id=20085 trace_id=51 func=iprope_dnat_check line=4659 msg="in-[port2], out-[]" id=20085 trace_id=51 func=iprope_dnat_check line=4672 msg="result: skb_flags-06000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=51 func=iprope_fwd_check line=636 msg="in-[port2], out-[VxLan-IPsec], skb_flags-06000000, vid-0" id=20085 trace_id=51 func=__iprope_check line=2049 msg="gnum-100004, check-ffffffffa001e70e" id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-100004 policy-1, ret-matched, act-accept" id=20085 trace_id=51 func=__iprope_user_identity_check line=1648 msg="ret-matched" id=20085 trace_id=51 func=__iprope_check line=2049 msg="gnum-4e20, check-ffffffffa001e70e" id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=51 func=__iprope_check line=2068 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=51 func=__iprope_check_one_policy line=2020 msg="policy-1 is matched, act-accept" id=20085 trace_id=51 func=__iprope_check line=2068 msg="gnum-100004 check result: ret-matched, act-accept, flag-08010000, flag2-00004000" id=20085 trace_id=51 func=iprope_fwd_auth_check line=688 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1" id=20085 trace_id=51 func=br_fw_forward_handler line=537 msg="Allowed by Policy-1:" id=20085 trace_id=51 func=__if_queue_push_xmit line=365 msg="send out via dev-VxLan-IPsec, dst-mac-02:09:0f:00:06:03" id=20085 trace_id=51 func=ipsecdev_hard_start_xmit_vxlan line=185 msg="enter IPsec interface-VxLan-IPsec" id=20085 trace_id=51 func=esp_output4 line=859 msg="IPsec encrypt/auth" id=20085 trace_id=51 func=ipsec_output_finish line=498 msg="send to 198.51.100.253 via intf-port1" ## ICMP echo-reply from PC2 to PC1
id=20085 trace_id=52 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 192.168.100.2:2871->192.168.100.1:0) from VxLan-IPsec. type=0, code=0, id=2871, seq=1."
id=20085 trace_id=52 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-00002434, reply direction" id=20085 trace_id=52 func=br_ipv4_fast_cb line=68 msg="enter fast path" id=20085 trace_id=52 func=__if_queue_push_xmit line=365 msg="send out via dev-port2, dst-mac-02:09:0f:00:02:02" |
FG1 # diag sys session filter clear
FG1 # diag sys session filter dst 192.168.100.2 FG1 # diag sys session filter proto 1 FG1 # diag sys session list session info: proto=1 proto_state=00 duration=8 expire=51 timeout=0 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=3 origin-shaper= reply-shaper= per_ip_shaper= ha_id=0 policy_dir=0 tunnel=VxLan-IPsec/ vlan_cos=0/255 state=may_dirty br statistic(bytes/packets/allow_err): org=84/1/1 reply=84/1/1 tuples=2 tx speed(Bps/kbps): 10/0 rx speed(Bps/kbps): 10/0 orgin->sink: org pre->post, reply pre->post dev=4->21/21->4 gwy=0.0.0.0/0.0.0.0 hook=pre dir=org act=noop 192.168.100.1:2871->192.168.100.2:8(0.0.0.0:0) hook=post dir=reply act=noop 192.168.100.2:2871->192.168.100.1:0(0.0.0.0:0) misc=0 policy_id=1 auth_info=0 chk_client_info=0 vd=0 serial=00002434 tos=ff/ff app_list=0 app=0 url_cat=0 dd_type=0 dd_mode=0 total session 1 |
## The ESP (IPsec) packet
Ethernet II, Src: MS-NLB-PhysServer-09_0f:00:04:02 (02:09:0f:00:04:02), Dst: MS-NLB-PhysServer-09_0f:00:04:01 (02:09:0f:00:04:01)
Destination: MS-NLB-PhysServer-09_0f:00:04:01 (02:09:0f:00:04:01)
Source: MS-NLB-PhysServer-09_0f:00:04:02 (02:09:0f:00:04:02) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: 198.51.100.1, Dst: 203.0.113.2 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 184 Identification: 0xc28d (49805) Flags: 0x00 Fragment offset: 0 Time to live: 63 Protocol: Encap Security Payload (50) Header checksum: 0x524f [correct] Source: 198.51.100.1 Destination: 203.0.113.2 Encapsulating Security Payload ESP SPI: 0xa8d52b6d (2832542573) ESP Sequence: 12 ESP IV: 5b8204e860038607 ESP Pad Length: 0 Next header: IPIP (0x04) Authentication Data [correct]
## The VxLAN packet carried inside the ESP (IPsec) packet
Internet Protocol Version 4, Src: 198.51.100.1, Dst: 203.0.113.2 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 134 Identification: 0xc28b (49803) Flags: 0x00 Fragment offset: 0 Time to live: 63 Protocol: UDP (17) Header checksum: 0x52a4 [correct] Source: 198.51.100.1 Destination: 203.0.113.2 User Datagram Protocol, Src Port: 4789, Dst Port: 4789 Source Port: 4789 Destination Port: 4789 Length: 114 Virtual eXtensible Local Area Network Flags: 0x0000 0... .... .... .... = GBP Extension: Not defined .... .... .0.. .... = Don't Learn: False .... 0... .... .... = VXLAN Network ID (VNI): False .... .... .... 0... = Policy Applied: False .000 .000 0.00 .000 = Reserved(R): 0x0000 Group Policy ID: 0 VXLAN Network Identifier (VNI): 0 Reserved: 0 ## PC1’s frame carried inside the VxLAN packet
Ethernet II, Src: MS-NLB-PhysServer-09_0f:00:02:02 (02:09:0f:00:02:02), Dst: MS-NLB-PhysServer-09_0f:00:06:03 (02:09:0f:00:06:03) Destination: MS-NLB-PhysServer-09_0f:00:06:03 (02:09:0f:00:06:03) Source: MS-NLB-PhysServer-09_0f:00:02:02 (02:09:0f:00:02:02) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: 192.168.100.1, Dst: 192.168.100.2 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 84 Identification: 0xcfff (53247) Flags: 0x02 (Don't Fragment) Fragment offset: 0 Time to live: 64 Protocol: ICMP (1) Header checksum: 0x2155 [correct] Source: 192.168.100.1 Destination: 192.168.100.2 Internet Control Message Protocol Type: 8 (Echo (ping) request) Code: 0 Checksum: 0xc1c9 [correct] [Checksum Status: Good] Identifier (BE): 2888 (0x0b48) Identifier (LE): 18443 (0x480b) Sequence number (BE): 1 (0x0001) Sequence number (LE): 256 (0x0100) Data (48 bytes) |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.