IPSEC Tunnel dropping its route

TWH · ‎02-18-2016

Hello All,

Sorry for the long explanation - trying to answer foreseeable questions in advance!

We have a pair of FG-100Ds in a HA Active-Passive configuration, running v5.2.3. (Would like to upgrade to v5.2.5, but no one wants to approve the change!)

Connected to this are 22 small sites (couple of devices each) connected via IPSec tunnels, each with Cisco 887 (ADSL plus 3G Failover) and 4 x Phase 2 selectors.

The Fortigate has a private IP address on the WAN1 port, so all tunnels are enabled for NAT-T on both sides.

Head office has a range of 172 and 10 subnets, and each site has both a 10.x.x.0/28 and 172.x.x.0/28 subnet.

These tunnels were all configured by a third party in August 2015, and have been quite robust.

Needed to alter the Interesting Traffic selectors, after testing one site successfully, rolled out the change to each of the Cisco routers and on its Fortigate tunnel.

Then I started to notice semi-frequent dropouts in links from HQ to individual sites. Most sites drop 1 or 2 times a day, some up to 5 times a day, and some have never dropped, despite the fact configs have been checked 50 times and are identical.

When I check the Fortigate, I find the tunnels are reporting as up, but the route specific to that P2 has vanished.

VPN, Monitor, IPsec Monitor, Select the appropriate Phase 2, right click, Bring Down will resolve the issue within a second or two.

My thinking has been P1 / P2 timeouts.

P1 was correct - both sides 86,400 seconds - 1 day.

Then I discovered P2 was default on the Cisco (3,600 seconds, 4,608,000 Kbytes) and set to 43,200 on the Fortigate. Bingo!

Reconfigured all the P2s (all 88 of them) on the Fortigate to Key Lifetime "Both" Seconds 3,600 and Kilobytes 4,608,000 to match the defaults on the Cisco.

Reliability appears better, but still experiencing dropouts.

Some of the Cisco's have been rebooted, but the Fortigate has not.

DPD enabled both sides. NAT-T configured both sides. I believe MTU configuration on the Cisco is good.

Spent most of the last two weeks on this! We have site monitoring which pings every site every 30 minutes. Before the change this was stable. Now getting 30 or 40 alerts a day. Manually bringing down the tunnel whenever a site is unreachable resolves it, but sites are offline for ~15-20 minutes before I give it a kick, which is unacceptable.

If I don't intervene, it does come back by itself within an hour, again seeming to indicate timeout related.

The biggest thing I don't understand in all of this is how the tunnels were stable previously with the Phase 2 lifetimes being incorrect. I suppose since they were lower on the Cisco, it just re-established the P2, and the Fortigate said "Oh, OK!".

The fortigate hasn't been restarted in over 200 days, probably not since these tunnels were commissioned.

Screen grab of snippets of the interfaces view and routes view are attached.

Would like to create an explicit static route for each of the sites on the Fortigate pointing to their tunnel. But new route doesn't list the tunnels as a selectable interface. Setting in the CLI also fails. The routes from that screen shot are generated automatically and vanish when the issue occurs.

I feel like I need a way to completely reset each tunnel on the Fortigate (rebooting it has been tempting!) - perhaps just dropping the tunnel isn't resetting the lifetime back to 0 for all phases on both the Fortigate and Cisco?

Need help with advanced troubeshooting / monitoring CLI commands for the Fortigate. Have done most of my debug on the Cisco's thus far, and haven't found anything.

Help!

TIA

Tony

TWH · ‎02-22-2016

Here is a snippet of the Fortigate config pertaining to one of our sites, Parry.

Hopefully I have included everything pertinent!

The definition of RR_VPN_allowed is messy with overlapping subnets, but that is how I inherited it. I don't think that would be an issue for a firewall traffic definition..... ?

config system interface

edit "wan1"

set vdom "root"

set ip 172.25.26.1 255.255.255.0

set allowaccess ping auto-ipsec

set type physical

set alias "Internet VLAN1004"

set snmp-index 1

set vdom "root"

set type tunnel

set snmp-index 27

set interface "wan1"

edit "RRLan1"

set uuid 35b53b82-2532-51e5-1e68-4e12f9c7ad29

set subnet 10.0.0.0 255.0.0.0

set uuid 434365d0-2532-51e5-8c01-3d3f45b0514a

set subnet 172.25.31.0 255.255.255.0

set uuid 60de0802-4abc-51e5-8b39-e8af024ce3c2

set subnet 172.26.32.0 255.255.252.0

set uuid 700447a6-4abc-51e5-3ce6-f98e8336aa4a

set subnet 172.27.26.0 255.255.254.0

set uuid 5368b170-5040-51e5-6f65-d5615fdde652

set comment "RR Lan 5"

set subnet 172.16.0.0 255.240.0.0

set uuid afe75a02-5043-51e5-736d-5aba728b7f94

set subnet 172.26.31.0 255.255.255.0

set uuid c2226042-b7f2-51e5-8f64-c6785f7eb758

set subnet 172.25.29.0 255.255.255.0

set uuid 3e277088-3cbf-51e5-6611-3c888255465e

set visibility disable

set subnet 10.26.32.32 255.255.255.240

set uuid 4dc467e4-3cbf-51e5-9a94-565640679102

set visibility disable

set subnet 172.26.32.32 255.255.255.240

edit "Parry"

set uuid 7c44929c-3cbf-51e5-a1fc-369091c65fe9

set member "Parry_Data" "Parry_Voice"

set comment "Parry_House"

edit "RR_VPN_allowed"

set uuid 552711a2-2532-51e5-0777-aadca12dc890

set member "RRLan1" "RRLan2" "RRLan3" "RRLan4" "RRLan5" "RRLan6" "RRLan7"

edit "ParryVPN"

set type dynamic

set interface "wan1"

set mode aggressive

set peertype one

set proposal 3des-md5 3des-sha1 3des-sha256

set dhgrp 2

set peerid "parry.royalrehab.local"

set psksecret ENC REDACTED

edit "ParryVPN_1"

set phase1name "ParryVPN"

set proposal 3des-md5 3des-sha256

set dhgrp 2

set keepalive enable

set keylife-type both

set keylifeseconds 3600

set keylifekbs 4608000

set src-subnet 10.0.0.0 255.0.0.0

set dst-subnet 10.26.32.32 255.255.255.240

set phase1name "ParryVPN"

set proposal 3des-md5 3des-sha256

set dhgrp 2

set keepalive enable

set keylife-type both

set keylifeseconds 3600

set keylifekbs 4608000

set src-subnet 10.0.0.0 255.0.0.0

set dst-subnet 172.26.32.32 255.255.255.240

set phase1name "ParryVPN"

set proposal 3des-md5 3des-sha256

set dhgrp 2

set keepalive enable

set keylife-type both

set keylifeseconds 3600

set keylifekbs 4608000

set src-subnet 172.16.0.0 255.240.0.0

set dst-subnet 10.26.32.32 255.255.255.240

set phase1name "ParryVPN"

set proposal 3des-md5 3des-sha256

set dhgrp 2

set keepalive enable

set keylife-type both

set keylifeseconds 3600

set keylifekbs 4608000

set src-subnet 172.16.0.0 255.240.0.0

set dst-subnet 172.26.32.32 255.255.255.240

edit 5

set uuid 9d48ddea-3cbf-51e5-5244-8af2cafae703

set srcintf "ParryVPN" "port1"

set dstintf "ParryVPN" "port1"

set srcaddr "Parry" "RR_VPN_allowed"

set dstaddr "Parry" "RR_VPN_allowed"

set action accept

set schedule "always"

set service "ALL"

set logtraffic disable

TWH · ‎02-22-2016

And here is the Cisco config, again for one of our sites, Parry.

Thanks everyone!

!

hostname parry

!

ip dhcp pool pool 2202

import all

network 10.26.32.32 255.255.255.240

default-router 10.26.32.33

!

ip dhcp pool pool 2502

import all

network 172.26.32.32 255.255.255.240

default-router 172.26.32.33

!

multilink bundle-name authenticated

!

chat-script lte "" "AT!CALL" TIMEOUT 20 "OK"

!

cts logging verbose

license udi pid C887VAG-4G-GA-K9 sn REDACTED

!

vtp mode transparent

username admin privilege 15 secret 5 REDACTED

!

controller VDSL 0

!

controller Cellular 0

lte modem link-recovery rssi onset-threshold -110

lte modem link-recovery monitor-timer 20

lte modem link-recovery wait-timer 10

lte modem link-recovery debounce-count 6

!

vlan 2202,2502

!

track 1 ip sla 1 reachability

!

ip ssh version 2

!

crypto isakmp policy 1

encr 3des

hash md5

authentication pre-share

group 2

!

crypto isakmp policy 10

crypto isakmp identity hostname

crypto isakmp keepalive 10

!

crypto isakmp peer address PUBLIC-IP-FORTIGATE

set aggressive-mode password REDACTED

set aggressive-mode client-endpoint user-fqdn parry.DOMAIN

!

crypto ipsec transform-set FG esp-3des esp-md5-hmac

mode tunnel

!

crypto ipsec profile FGipsec

set transform-set FG

set pfs group2

!

crypto map CMAP 10 ipsec-isakmp

set peer PUBLIC-IP-FORTIGATE

set transform-set FG

set pfs group2

match address 115

!

interface Loopback30

description Monitoring address

ip address 10.224.62.159 255.255.255.255

!

interface ATM0

description Phy_ADSL

no ip address

no atm ilmi-keepalive

!

interface ATM0.1 point-to-point

mtu 1452

pvc 8/35

pppoe-client dial-pool-number 1

!

interface Cellular0

ip address negotiated

ip nat outside

ip virtual-reassembly in

ip virtual-reassembly out

encapsulation slip

dialer in-band

dialer string optus

dialer string hspa

dialer-group 1

async mode interactive

crypto map CMAP

!

interface Ethernet0

no ip address

shutdown

!

interface FastEthernet0

no ip address

!

interface FastEthernet1

description Data

switchport access vlan 2202

no ip address

!

interface FastEthernet2

description Voice

switchport access vlan 2502

no ip address

!

interface FastEthernet3

description Data

switchport access vlan 2202

no ip address

!

interface Vlan1

no ip address

!

interface Vlan2202

description Data

ip address 10.26.32.33 255.255.255.240

ip access-group DATA_VLAN_INBOUND in

ip access-group DATA_VLAN_OUTBOUND out

ip nat inside

ip virtual-reassembly in

!

interface Vlan2502

description Voice

ip address 172.26.32.33 255.255.255.240

ip access-group VOICE_VLAN_INBOUND in

ip access-group VOICE_VLAN_OUTBOUND out

ip nat inside

ip virtual-reassembly in

!

interface Dialer1

description Dialer interface for ADSL2

mtu 1492

ip address negotiated

no ip redirects

no ip unreachables

no ip proxy-arp

ip nbar protocol-discovery

ip flow ingress

ip flow egress

ip nat outside

ip virtual-reassembly in max-reassemblies 64

encapsulation ppp

ip tcp adjust-mss 1452

dialer pool 1

dialer-group 1

ppp chap hostname REDACTED

ppp chap password 0 REDACTED

ppp pap sent-username REDACTED

ppp ipcp route default

no cdp enable

crypto map CMAP

!

ip forward-protocol nd

no ip http server

no ip http secure-server

!

ip nat inside source route-map NAT interface Cellular0 overload

ip nat inside source route-map NAT1 interface Dialer1 overload

ip route 0.0.0.0 0.0.0.0 Dialer1 track 1

ip route 0.0.0.0 0.0.0.0 Cellular0 254

ip route 8.8.4.4 255.255.255.255 Dialer1

!

ip access-list extended DATA_VLAN_INBOUND

permit ip 10.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

permit ip 10.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

permit ip host 10.26.32.34 any

permit ip host 10.26.32.35 any

permit udp any eq bootpc any eq bootps

ip access-list extended DATA_VLAN_OUTBOUND

permit ip 10.0.0.0 0.255.255.255 10.26.32.32 0.0.0.15

permit ip 172.16.0.0 0.15.255.255 10.26.32.32 0.0.0.15

permit ip any host 10.26.32.34

permit ip any host 10.26.32.35

ip access-list extended VOICE_VLAN_INBOUND

permit ip 172.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

permit ip 172.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

permit udp any eq bootpc any eq bootps

permit ip host 172.26.32.34 any

ip access-list extended VOICE_VLAN_OUTBOUND

permit ip 10.0.0.0 0.255.255.255 172.26.32.32 0.0.0.15

permit ip 172.16.0.0 0.15.255.255 172.26.32.32 0.0.0.15

permit ip any host 172.26.32.34

!

ip sla auto discovery

ip sla 1

icmp-echo 8.8.4.4 source-interface Dialer1

threshold 2

timeout 1000

frequency 3

ip sla schedule 1 life forever start-time now

dialer-list 1 protocol ip permit

!

route-map NAT permit 10

match ip address 111

match interface Cellular0

!

route-map NAT1 permit 10

match ip address 110

match interface Dialer1

!

snmp-server community RRHouseRO RO

snmp-server community RRHouseRW RW

snmp-server community MACQread RO

access-list 110 deny ip 10.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

access-list 110 deny ip 10.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

access-list 110 deny ip 172.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

access-list 110 deny ip 172.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

access-list 110 permit ip 10.26.32.32 0.0.0.15 any

access-list 110 permit ip 172.26.32.32 0.0.0.15 any

access-list 111 deny ip 10.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

access-list 111 deny ip 10.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

access-list 111 deny ip 172.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

access-list 111 deny ip 172.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

access-list 111 permit ip 10.26.32.32 0.0.0.15 any

access-list 111 permit ip 172.26.32.32 0.0.0.15 any

access-list 115 permit ip 10.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

access-list 115 permit ip 10.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

access-list 115 permit ip 172.26.32.32 0.0.0.15 10.0.0.0 0.255.255.255

access-list 115 permit ip 172.26.32.32 0.0.0.15 172.16.0.0 0.15.255.255

!

control-plane

!

mgcp behavior rsip-range tgcp-only

mgcp behavior comedia-role none

mgcp behavior comedia-check-media-src disable

mgcp behavior comedia-sdp-force disable

!

mgcp profile default

!

line con 0

no modem enable

line aux 0

line 2

no activation-character

no exec

transport preferred none

transport input all

stopbits 1

line 3

script dialer lte

no exec

line vty 0 4

login local

transport input ssh

!

scheduler allocate 20000 1000

end

rdesilva_FTNT · ‎03-10-2016

The dropouts were triggered due to the multiple ACLs on the Cisco routers.

Fortigate was dropping the route as part of the tear down of a certain P2 that was requested by Cisco GW/client. But the deleted route was required by a another Phase2/tunnel between the two gateways.

Please use ACLs in the following format at the client end, eg:

1. 10.26.32.32 0.0.0.15 0.0.0.0 0.0.0.0 2. 172.26.32.32 0.0.0.15 0.0.0.0 0.0.0.0

The remote IPSec dial-up client should not have more than one Phase2/ACL for it's local subnet when connecting to the FGT(Dial-up Server), in this example.