IPsec hub-and-spoke VPN only working in one direction

[Zylan]

Hello,

I'm trying to connect 3 FortiGates (1 hub, 2 spokes) together with this topology:

                     ┌──────────────────────┐
                     │                      │
                     │  Hub                 │
                     │  OS 6.4.9            │
                     │                      │
                     │  Tunnel              │
┌──────────────┐     │  10.255.254.254      │     ┌──────────────┐
│              │     │                      │     │              │
│ Spoke        │     │  Public static IPv4  │     │ Spoke        │
│ Site A       │     │                      │     │ Site B       │
│ OS 7.0.6     │     └───────────┬──────────┘     │ OS 6.0.14    │
│              │                 │                │              │
│ Tunnel       │                 │                │ Tunnel       │
│ 10.255.254.1 │     ┌───────────┴──────────┐     │ 10.255.254.4 │
│              ├─────┤          WAN         ├─────┤              │
└──────────────┘     └──────────────────────┘     └──────────────┘

Both spoke FortiGates are behind NAT. The weird problem I'm seeing is that while site B is able to reach every other device in the VPN subnet, neither site A nor the hub can reach site B.

SiteB # exec traceroute 10.255.254.254
traceroute to 10.255.254.254 (10.255.254.254), 32 hops max, 3 probe packets per hop, 72 byte packets
1  10.255.254.254  2.789 ms  2.567 ms  2.377 ms

SiteB # exec traceroute 10.255.254.1
traceroute to 10.255.254.1 (10.255.254.1), 32 hops max, 3 probe packets per hop, 72 byte packets
1  10.255.254.254  2.594 ms  2.891 ms  3.148 ms
2  10.255.254.1  4.284 ms  4.165 ms  5.277 ms

Hub # exec ping 10.255.254.1
PING 10.255.254.1 (10.255.254.1): 56 data bytes
64 bytes from 10.255.254.1: icmp_seq=0 ttl=255 time=2.2 ms
--- 10.255.254.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 2.2/2.2/2.2 ms

Hub # exec ping 10.255.254.4
PING 10.255.254.4 (10.255.254.4): 56 data bytes
--- 10.255.254.4 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

There is a policy on the hub to allow both spoke-to-spoke and spoke-to-hub traffic, as well as policies on the spoke devices themselves to allow traffic to go through and the tunnel to be up. All IPsec interfaces are showing "up".

Here are the relevant configurations for all devices:

Hub

config vpn ipsec phase1-interface
    edit "hub"
        set type dynamic
        set interface "wan1"
        set ike-version 2
        set authmethod signature
        set peertype peergrp
        set net-device disable
        set proposal aes256-sha256 aes256gcm-prfsha384
        set add-route disable
        set dpd on-idle
        set dhgrp 18
        set certificate "s2s-hub-certificate"
        set peergrp "s2s-clients"
        set dpd-retryinterval 5
    next
end
config vpn ipsec phase2-interface
    edit "hub"
        set phase1name "hub"
        set proposal aes256-sha512 aes256-sha256 aes256gcm
        set dhgrp 18
    next
end
config system interface
    edit "hub"
        set ip 10.255.254.254 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.255.254.253 255.255.255.0
        set mtu-override enable
        set mtu 1260
        set interface "wan1"
    next
end

Site A

config vpn ipsec phase1-interface
    edit "site-a"
        set interface "wan1"
        set ike-version 2
        set authmethod signature
        set peertype any
        set net-device enable
        set proposal aes256gcm-prfsha384
        set dpd on-idle
        set dhgrp 18
        set remote-gw xxx.xxx.xxx.xxx
        set certificate "s2s-site-a-cert"
        set dpd-retryinterval 5
    next
end
config vpn ipsec phase2-interface
    edit "site-a"
        set phase1name "site-a"
        set proposal aes256gcm
        set dhgrp 18
        set auto-negotiate enable
    next
end
config system interface
    edit "site-a"
        set ip 10.255.254.1 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.255.254.254 255.255.255.0
        set interface "wan1"
    next
end

Site B

config vpn ipsec phase1-interface
    edit "site-b"
        set interface "wan1"
        set ike-version 2
        set authmethod signature
        set peertype any
        set proposal aes256gcm-prfsha384
        set dpd on-idle
        set dhgrp 18
        set remote-gw xxx.xxx.xxx.xxx
        set certificate "s2s-site-b-cert"
        set dpd-retryinterval 5
    next
end
config vpn ipsec phase2-interface
    edit "site-b"
        set phase1name "site-b"
        set proposal aes256-sha512
        set dhgrp 18
        set auto-negotiate enable
    next
end
config system interface
    edit "site-b"
        set ip 10.255.254.4 255.255.255.255
        set allowaccess ping
        set type tunnel
        set tcp-mss 1260
        set remote-ip 10.255.254.254 255.255.255.0
        set interface "wan1"
    next
end

What could be causing this problem?

[Zylan]

Just to confirm, net-device should be set on the hub, is this correct? I've tried it and the result was that I couldn't ping any tunnel interface IP from anywhere (hub, site A, or site B). On the hub, with net-device enabled, the routing table shows:

Hub # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

Routing table for VRF=0
C       10.255.254.253/32 is directly connected, hub_0
                          is directly connected, hub_1
C       10.255.254.254/32 is directly connected, hub_0
                          is directly connected, hub_1

(I'm not sure if this is something that needs to be enabled on both sides - both spoke and hub - but if so, while this looks possible with site A (OS 7.0), this doesn't seem to be possible with site B's legacy FGT (OS 6.0))

Yes, I'm just testing things out. Eventually, BGP will be used to route subnets behind the tunnel interface IPs, but I would think that IPs within the VPN subnet needs to be reachable first for BGP to work - please correct me if I'm wrong (I did try configuring BGP, but no routes were exchanged with an error of "malformed AS-PATH").

As for multiple phase1 interfaces, I haven't tried this out yet but would that not create multiple logical interfaces that then needs security policies applied to all of them? Either way, I'm trying to avoid having to configure too many things at the hub if possible...

[Toshi_Esumi]

The remote side is "static" ipsec. The net-device config doesn't do anything. Only in case dynamic/dialup is configured (hub), it would allow creating those hub_0, hub_1 dynamic interface as you saw in the routing table. Since you are configuring one remote-ip on the tunnel interface, it's showing on both side as a connected route. I don't think dynamic/dialup IPsec config on the hub side allows you to configure BGP neighbors. Because you don't know which one becomes hub_0 and which becomes hub_1.

I think you have to have separate phase1-interfaces for each then you can specify a remote IP like 10.255.254.1 or .4 on each interface. Then you can configure BGP neighbors.

I might be wrong and there might be a way with dialup(one phase1-interface on hub) but somebody from FTNT should be able to tell if that's the case.

<edit>

For the policies on the hub if you have individual phase1s, you could put two remote interfaces in a zone then allow "intrazone" traffic if you don't want to restrict accesses each others, or deny it if no remote-to-remote is allowed. For hub<->remotes, you probably have to have some kind of policies anyway so you at least need two policies for both directions.

</edit>

Toshi

[Toshi_Esumi]

Looks like your set up is quite similar to ADVPN, which I don't have any experience with. I'll stand down and let somebody else helping you. The doc below says you should "disable" net-device at the hub.

https://docs.fortinet.com/document/fortigate/6.2.11/cookbook/820072/advpn-with-bgp-as-the-routing-pr...

Toshi

[Zylan]

Yes, my setup is based on guides for ADVPN, although I later learned that ADVPN doesn't work with IKEv2. That is indeed where the "disable net-device" configuration came from.

Thanks for the help though! Hopefully someone with more experience with ADVPN-type configuration will be able to point me in the right direction.

Also, if it helps, each site has a different BGP ASN, as the end-use case is to link multiple organizations together.