VXLAN over IPSec tunnel issues

[steveballantyne]

Hello all, I have an MPLS circuit and I want to run an encrypted end to end connection over it using two Fortigate 60E boxes. I am trying to follow a cookbook recipe from the KB on using a virtual-wire and an IPSec tunnel. Its been challenging because the examples do *not work* out of the box.

This is the most useful cookbook / KB I have found: https://docs.fortinet.com/document/fortigate/6.2.0/cookbook/821119/vxlan-over-ipsec-tunnel

Namely the issues that I have had is that the ports being used for this configuration must first be removed from the "internal" switch group. I have to wonder if that is causing me problems because there is some sort of other policy applied to those ports? Also, all of the notes refer to these are "port1" and "port2", etc. but my ports are named "internal1" and "internal2". Again, maybe just a code revision as the KB I am reading was for 6.2.0 and I am running 6.4.1

Also, when I get to the step of configuring the virtual switch I get an error about the member not being in the dataset. I found another forum post where someone suggested you delete all references to that member. The only references I had were the firewall rules from an earlier step. So I deleted all the firewall rules. Then created the virtual switch. Then re-created those same rules, and that seemed to work? But ... it doesn't function at all.

My end result is that the VPN tunnel is not establishing over port1 (internal1 in my case) and I cannot ping the other firewall's interface.

Can someone take a look and see what I might be doing wrong?  :) I am attaching one of the sanitized configs (cant seem to attach more than file here?).

Note that I am attempting to send ALL of my L2 traffic from one end to the other. The example was created for a single subnet. So that may also be my issue is that I changed the firewall rules to any/any?

[echo]

Have you got this working? I also want to start using it in my environment.

Port names depend on Fortigate model, some have portx, some have internalx with x=1,2,...

From your configuration, I see that you have software-switch called "VXLAN-HQ2" with members internal2 and to_HQ2 (IPSEC-tunnel interface) as in the manual but where's the internal IP-address defined? In the example the network is 10.1.100.0/24, in your configuration the dmz has 10.10.10.1/24. Unclear from the example is this: where is the default gateway of 10.1.100.0/24 defined. Maybe one has to choose the side, HQ1 or HQ2 and use it for either dmz or port9 (possibly on bigger HQ side). Another theoretical option would be to use this address on software switch interface but I don't know if this is the case for VXLAN's.

In Fortinet's example, I see that maybe using IPSEC-interface to_HQ2 in firewall rules is not correct or not sufficient: maybe the software switch's interface VXLAN-HQ2 has to be used instead or in addition to the one shown in the example. Check the logs! If this is the case then that may be the reason why the tunnel does not come up. Using any/any firewall rules as you have done (for a test at least) let's you more easily to narrow down the problem.

I understand that when one wants to use one single network/VLAN going over the IPSEC tunnel then it is OK for using encapsulation VXLAN on the IPSEC settings but when one wants to select which VLANs spread over the tunnel this way then another configuration has to be done, maybe something like this: https://docs.fortinet.com/document/fortigate/6.2.0/new-features/392860/vlan-inside-vxlan

But maybe this is not what you want to achieve. For me, this what I want to achieve and I'd like to find a way how it works.

[RMirkowski]

I worked on the same case from few days but still not have worked configuration. I use this link https://docs.fortinet.com...xlan-over-ipsec-tunnel In my environment I use Internet connection from one site to another, IPSEC works and even I see MAC addresses on software switch whitch point to local port and VPN-interface but ping don't work. Maybe it's ARP issue. In fact I don't get this configuration also, where is for example VTEP address or something related with VxLAN...

[nbanba]

Hi

I got this working. 

For my part, the extend LAN is a VLAN , so I had to let the vlan interface in layer2 on the fortigate (no objects, no route , no ip address, ONLY vlan tag and interface name).

For the moment, in this configuration, I didn't find the way to pass a tagged vlan over ipsec.

(The documentation provide is for a LAN extended by VXLAN over IPSEC , or for a  VLAN inside a VXLAN , but not for a VLAN extended by VXLAN over IPSEC. Maybe it 's possible to mount a regular IPSEC tunnel with routed phases 2 and to configure a VLAN inside a VXLAN over this layer3 connection, but I didn't test)

I have a support case open at the TAC, I can let you know the solution

Regards,

nbanba

[echo]

Somebody describes here how tagged traffic goes over the tunnel using loopback interfaces:

https://forum.fortinet.com/tm.aspx?m=160442

[echo]

Based on https://docs.fortinet.com/document/fortigate/6.2.0/new-features/392860/vlan-inside-vxlan

and https://kb.fortinet.com/kb/documentLink.do?externalID=FD47557

could this work? 1. vxlan inside ipsec-tunnel interface, 2. vlan inside port1, 3. vlan and vxlan bridged together. The second part of config system interface seems strange though but it is taken from the first link. I think that second part is just wrong, there is not "set type vlan" but "set type vxlan" automatically and then also hardly ever you could write set vlanid 100 directly into a vxlan interface configuration. So probably those two lines are a copy-paste error and the question is if without those two the rest works or not. (Unless it is really possible to define two different kind of interfaces with the same vlan id.)

config system vxlan
 edit vxlan1
 set interface ipsec-tun
 set vni 1000
 set remote-ip 172.16.200.3
 next
end

config system interface
   edit vlan100
     set vdom root
     set vlanid 100
     set interface port1
   next
   edit vxlan100
     set type vlan
     set vlanid 100
     set vdom root
     set interface vxlan1
   next
end

config system switch-interface
  edit sw1
    set vdom root
    set member vlan100 vxlan100
  next
end

[echo]

I made a test environment with two Fortigate 60F having the last FortiOS v6.4.5 and I got it working and so far only for the ping to go through. To test other protocols, I will need to improve the test environment with new devices.

And it turned out the above-mentioned https://docs.fortinet.com/document/fortigate/6.2.0/new-features/392860/vlan-inside-vxlan actually works. The vlan-interface having name vxlan100 is just a convention to refer to the vlan number 100 that is used over the ipsec tunnel.

My conf is as follows in the FGT1, relevant parts only. Interface dmz is used for "internet" connectivity but in my test environment it is in the same switch with FGT2's dmz interface (untagged vlan 1) for direct connectivity for the ipsec tunnel. Internal1 is the internal interface under which the vlan10 is made and is in use. Comments that are not present in the config file but in this post only are written after ///.

config system interface     edit "dmz" /// Internet         set vdom "root"         set ip 10.10.10.1 255.255.255.0         set allowaccess ping https ssh http fgfm fabric         set type physical         set role dmz         set snmp-index 3     next     edit "internal1" /// Internal single interface for LAN         set vdom "root"         set type physical         set snmp-index 8     next     edit "vxlan1" /// Formal interface for logical connection between ipsec tunnel and vxlan         set vdom "root"         set type vxlan         set snmp-index 10         set interface "ipsectun01"     next     edit "vxlan10" /// VLAN interface that refers to the real VLAN10 but reflects the VXLAN usage in its name         set vdom "root"         set device-identification enable         set role lan         set snmp-index 11         set interface "vxlan1"         set vlanid 10     next     edit "vlan10" /// VLAN10 interface for tagged traffic for LAN switch         set vdom "root"         set device-identification enable         set role lan         set snmp-index 12         set interface "internal1"         set vlanid 10     next     edit "softsw" /// This appears here later after the software switch has been created; default gateway for LAN         set vdom "root"         set ip 10.0.1.1 255.255.255.0         set allowaccess ping         set type switch         set snmp-index 13     next end config system switch-interface     edit "softsw"         set vdom "root"         set member "vlan10" "vxlan10" /// Binding together the LAN's VLAN10 and logical VXLAN's VLAN10     next end The IPSEC tunnel is standard, made in GUI (which added the "set net-device enable" row automatically):

config vpn ipsec phase1-interface     edit "ipsectun01"         set interface "dmz"         set ike-version 2         set peertype any         set net-device enable         set proposal aes256-sha512         set dhgrp 21         set nattraversal disable         set remote-gw 10.10.10.2         set psksecret ENC ---     next end config vpn ipsec phase2-interface     edit "ipsectun01"         set phase1name "ipsectun01"         set proposal aes256-sha512         set dhgrp 21     next end

config router static     edit 1         set gateway 10.10.10.100 /// In my test environment this doesn't exist but it is there because in real environments the default route always exists and I felt it is better to have it even if it is not really working this way         set device "dmz"     next end

The important difference is that there is no route to the network behind the tunnel as it usually is because the same L2 network has to work on both sides of the tunnel.

For FGT2, all this is the same, just the .1 is replaced with .2, eg dmz is 10.10.10.2 and softsw is 10.0.1.2/24 (and this is used as default gateway for the device behind FGT2). I also used certain IP-addresses on ipsectun01 interfaces on both sides but maybe that is not important.

And now firewall policies. This is ODD! It turned out that it didn't matter if the switch-interface configuration had either "set intra-switch-policy explicit" or "set intra-switch-policy implicit": in the second case the existence or nonexistence of firewall policies between softsw interfaces vlan10 and vxlan10 didn't change anything: the ping continued to work! Looks like a bug.

And even more: I had set the ipsectun01 interface into vpn zone, the empty/down interface _wan1_ into trust zone and dmz interface into untrust zone and I made firewall rules from trust to vpn and backwards allowing all. The GUI showed warning on the firewall rules because wan1 interface is down. I added wan1 to trust only for the sake of reconfiguration and deletion of softsw interface only (disabling intra-switch-policy required to delete the softsw interface and recreate it). But with this wrong trust zone membership the ping continued to work and it stopped working when I disabled that rule! This tells me that the policy between zones is only formally needed (reverse route check?) and it doesn't even matter what's actually in the zone. I think it is another bug. When I saw this first, I thought my test environment is problematic: the VLAN10 connection somehow went directly through dmz interfaces. But when I removed different cables and changed configuration it still turned out that if I disabled the ipsec tunnel then the ping also stopped working so that VLAN10 actually went over the ipsec tunnel.

And third oddity: in both cases with "set intra-switch-policy explicit" or "set intra-switch-policy implicit" plus the trust -> vpn and vpn -> trust firewall policies: the GUI shows no traffic! Bytes: 0, Last used: empty, Sessions: 0. But ping works. I only saw the real traffic information from Dashboard>Network>IPsec (expand) in Incoming Data and Outgoing Data columns of ipsectun01.

I pinged 10.0.1.20 from FGT1 and this IP belongs to 8-port HP switch behind FGT2 port internal1 (tagged) and the connected switch port has VLAN10 tagged. From FGT1 I saw this:

FGT1 # diag ip arp list index=7 ifname=dmz 10.10.10.2 04:d5:90:43:48:10 state=00000004 use=132 confirm=14778352 update=41 ref=2 index=7 ifname=dmz 10.10.10.100 state=00000001 use=219 confirm=14778633 update=277 ref=3 index=31 ifname=softsw 10.0.1.20 38:63:bb:1e:d8:80 state=00000002 use=2630 confirm=2630 update=3035 ref=2 index=31 ifname=softsw 10.0.1.2 04:d5:90:43:48:11 state=00000002 use=132 confirm=132 update=234 ref=2 index=18 ifname=root 0.0.0.0 00:00:00:00:00:00 state=00000040 use=5981 confirm=11981 update=5981 ref=1 index=24 ifname=ipsectun01 0.0.0.0  state=00000040 use=14795469 confirm=14795469 update=14873573 ref=1

Here I can't really see where the 10.0.1.20 lies but another command reveals the connection with vxlan (this and next debug commands taken from https://kb.fortinet.com/kb/documentLink.do?externalID=FD40170) :

FGT1 # diagnose netlink brctl name host softsw show bridge control interface softsw host. fdb: size=2048, used=4, num=4, depth=1, ageing-time=300, simple=switch Bridge softsw host table port no device  devname mac addr                ttl     attributes   1     21      vlan10  04:d5:90:43:22:4b       0       Local Static   2     26      vxlan10 04:d5:90:43:48:11       9        Hit(9) /// This is the VLAN10 tagged interface of FGT2   2     26      vxlan10 38:63:bb:1e:d8:80       38       Hit(38) /// This is the remote device behind FGT2 (also happens to be tagged because it is a switch)

I also checked with "diag debug flow" what I see there:

id=20085 trace_id=12 func=print_pkt_detail line=5693 msg="vd-root:0 received a packet(proto=1, 10.0.1.1:2560->10.0.1.20:2048) from local. type=8, code=0, id=2560, seq=0." id=20085 trace_id=12 func=init_ip_session_common line=5864 msg="allocate a new session-00006528" id=20085 trace_id=12 func=iprope_dnat_check line=5005 msg="in-[], out-[softsw]" id=20085 trace_id=12 func=iprope_dnat_check line=5018 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=12 func=__iprope_check line=2164 msg="gnum-100004, check-ffffffbffc03d7c0" id=20085 trace_id=12 func=__iprope_check_one_policy line=1920 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=12 func=__iprope_check_one_policy line=1920 msg="checked gnum-100004 policy-2, ret-no-match, act-drop" id=20085 trace_id=12 func=__iprope_check_one_policy line=1920 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=12 func=__iprope_check line=2183 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=12 func=iprope_policy_group_check line=4450 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=12 func=ipd_post_route_handler line=490 msg="out softsw vwl_zone_id 0, state2 0x0, quality 0. " id=20085 trace_id=12 func=__if_queue_push_xmit line=392 msg="send out via dev-vxlan10, dst-mac-38:63:bb:1e:d8:80" id=20085 trace_id=13 func=print_pkt_detail line=5693 msg="vd-root:0 received a packet(proto=1, 10.0.1.20:2560->10.0.1.1:0) from softsw. type=0, code=0, id=2560, seq=0."

64 bytes from 10.0.1.20: icmp_seq=0 ttl=64 time=1.1 ms /// Only this typical reply appears 5 times when the debug messages are not set to show in console

id=20085 trace_id=13 func=resolve_ip_tuple_fast line=5774 msg="Find an existing session, id-00006528, reply direction" id=20085 trace_id=13 func=vf_ip_route_input_common line=2584 msg="find a route: flag=84000000 gw-10.0.1.1 via root" id=20085 trace_id=14 func=print_pkt_detail line=5693 msg="vd-root:0 received a packet(proto=1, 10.0.1.1:2560->10.0.1.20:2048) from local. type=8, code=0, id=2560, seq=1." id=20085 trace_id=14 func=resolve_ip_tuple_fast line=5774 msg="Find an existing session, id-00006528, original direction" id=20085 trace_id=14 func=ipd_post_route_handler line=490 msg="out softsw vwl_zone_id 0, state2 0x0, quality 0. " id=20085 trace_id=14 func=__if_queue_push_xmit line=392 msg="send out via dev-vxlan10, dst-mac-38:63:bb:1e:d8:80"

(And it repeats 4 times.)

This is only ping so far. The next thing to test when considering the branch office scenario over the IPSEC tunnel is if I set the def gw over the IPSEC tunnel and/or set up DHCP because we have central DHCP server already that the clients access over the IPSEC tunnel and this has to work somehow. What would be the def gw for them in the branch, either local .2 or remote .1, I am no yet sure.

And of course the performance / speed with file copy over the tunnel using different protocols needs to be tested too (plus wifi AP's connection to controller, VoIP calls, central printserver connection etc.) needs to be tested later.

[maskboy]

My scenerio look like that. But twice site vlan gateway on the HQ diffrent L3 interface. When I try to ping from Branch to HQ Vlan gateway (I dont use ip address on software switch, I give ip address to diffrent L3 interface at HQ) it fail!! I think PC dont resolve mac address of vlan gateway at HQ ip address. When I insert mac address of vlan gateway static at the PC (Branch site) ping was success. Any idea about that issue?