Problem with LinkMonitor

[AfonsoAndrade]

Helo, I am with problem with link monitor in the FGT 100D 5.2.9. I did in of all, but the FGT no identify link is down.

Test:

### I disconnected the cable in the port1

# config system link-monitor edit "Link1" set srcintf "port1" set server "8.8.8.8" "200.221.2.45" set timeout 5 set failtime 3 set recoverytime 3 set update-cascade-interface disable next edit "Link2" set srcintf "wan1" set server "8.8.8.8" "200.221.2.45" set timeout 5 set failtime 3 set recoverytime 3 set update-cascade-interface disable next edit "Link3" set srcintf "wan2" set server "8.8.8.8" "200.221.2.45" set timeout 5 set failtime 3 set recoverytime 3 set update-cascade-interface disable next end

# diag test application lnkmtd 3 now_jiffies=448297941 'dmz': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'ha1': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'ha2': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'mgmt': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'modem': link=no, brought_up=0, brought_down=0, signal_sent=0, broughtup_jiffies=0, broughtdown_jiffies=0 'port1': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=297687, broughtdown_jiffies=0 'port10': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port11': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port12': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port13': link=ok, brought_up=0, brought_down=0, signal_sent=0, broughtup_jiffies=0, broughtdown_jiffies=0 'port14': link=ok, brought_up=0, brought_down=0, signal_sent=0, broughtup_jiffies=0, broughtdown_jiffies=0 'port15': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port16': link=ok, brought_up=0, brought_down=0, signal_sent=0, broughtup_jiffies=0, broughtdown_jiffies=0 'port2': link=no, brought_up=0, brought_down=0, signal_sent=0, broughtup_jiffies=0, broughtdown_jiffies=0 'port3': link=no, brought_up=0, brought_down=0, signal_sent=0, broughtup_jiffies=0, broughtdown_jiffies=0 'port4': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port5': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port6': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port7': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port8': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'port9': link=no, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=3171, broughtdown_jiffies=0 'wan1': link=ok, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=295287, broughtdown_jiffies=0 'wan2': link=ok, brought_up=1, brought_down=0, signal_sent=0, broughtup_jiffies=296487, broughtdown_jiffies=0 lnkmtd::ping_epoll_callback(142): ping response 10.50.50.2, buf-sz=28

# get system link-monitor

== [ port1] name: WCS timeout: 5 == [ wan1] name: GVT timeout: 5 == [ wan2] name: CTBC timeout: 5

# diagnose sys link-monitor interface port1

Interface(port1): state(up, since Wed Jan 4 16:07:38 2017 ), bandwidth(27236), session count(0) latency(0.00), jitters(0.00).

# diagnose sys link-monitor st

PORT1 Status: alive Create time: Wed Jan 4 16:07:38 2017 Source interface: port1 (7) Source IP: XXX.XXX.XXX.170 Gateway: XXX.XXX.XXX.169 Interval: 5, Timeout 5 Fail times: 0/3 Send times: 0 Peer: 200.221.2.45(200.221.2.45) Source IP(XXX.XXX.XXX.170) protocol: ping, state: alive Latency(recent/average): 0.00/0.00 ms Jitter: 0.00 Recovery times(0/3) Continuous sending times after the first recovery time 0 Packet sent: 0 Packet received: 0 Peer: 8.8.8.8(8.8.8.8) Source IP(XXX.XXX.XXX.170) protocol: ping, state: alive Latency(recent/average): 0.00/0.00 ms Jitter: 0.00 Recovery times(0/3) Continuous sending times after the first recovery time 0 Packet sent: 0 Packet received: 0

-----------------------------------------------------------------------------------

the last time that i had this error, i removed the config at linkMonitor, reboot the FGT and I did configuration the linkMonitor again. After that, It came back work. But the problem retorn with the time.

[emnoc]

dumb questions

1: is port1  really up

2: does it have a  routes install on it

3: can you set the source-ip and next-hop

4: have  query the logs and the logdesc 

e.g

 execute  log  filter  field  logdesc "Link monitor status"

 execute  log  filter  cat 1

 execute log dis

[AfonsoAndrade]

1: is port1  really up

--- Now It is UP, but when it down the status no change.

2: does it have a  routes install on it

---- Yes, It has. I have 3 links and that link has very router.

---- in the test up I executed the command line "execute route restart". after I remove the cable in the port1.

3: can you set the source-ip and next-hop

---- Excuse, but I didn't understand. Do you want a test "execute ping-options source" ?

---- Look bellow, Is that?

edit "port1" set vdom "root" set mode static set dhcp-relay-service disable set ip XXX.XXX.XXX.170 255.255.255.248 set allowaccess ping https ssh snmp capwap set fail-detect disable set arpforward enable set broadcast-forward disable set bfd global set l2forward disable set icmp-redirect enable set vlanforward enable set stpforward disable set ips-sniffer-mode disable set ident-accept disable set ipmac disable set subst disable set status up set netbios-forward disable set wins-ip 0.0.0.0 set type physical set netflow-sampler disable set sflow-sampler disable set sample-rate 2000 set polling-interval 20 set sample-direction both set explicit-web-proxy disable set explicit-ftp-proxy disable set tcp-mss 0 set inbandwidth 0 set outbandwidth 0 set spillover-threshold 0 set weight 0 set external disable set description set alias "PORT1" set security-mode none set device-identification disable set lldp-transmission vdom set listen-forticlient-connection enable set broadcast-forticlient-discovery disable set vrrp-virtual-mac disable set snmp-index 1 set secondary-IP disable config ipv6 set ip6-mode static unset ip6-allowaccess set ip6-reachable-time 0 set ip6-retrans-time 0 set ip6-hop-limit 0 set ip6-address ::/0 set ip6-send-adv disable set autoconf disable set dhcp6-relay-service disable end unset dhcp-relay-ip set dhcp-relay-type regular set speed auto set mtu-override disable set wccp disable set drop-overlapped-fragment disable set drop-fragment disable

edit "PORT1" set srcintf "port1" set server "8.8.8.8" "200.221.2.45" set protocol ping set gateway-ip XXX.XXX.XXX.169 set source-ip XXX.XXX.XXX.170 set interval 5 set timeout 5 set failtime 3 set recoverytime 3 set ha-priority 1 set update-cascade-interface disable set update-static-route enable set status enable

4: have  query the logs and the logdesc

----- result: 

0 logs found.

0 logs returned. 6.8% of logs has been searched.

I asked to remove the network cable, again. --- At Interface/port1 was change status down. --- At Log/System has the log "Link Monitor: Interface port1 was turned down --- At linkMonitor the status UP

as the interface port1 is not status down in link Monitor, the firewall doesn't move the session to other port.

[emnoc]

hmm...

Can you ensure logging is enabled for appliance and retest. Also it would not hurt to ensure that pings are being sent from  src x.x.x.x to the targets

CLI

diag sniffer packet port1 "src host XXX.XXX.XXX.170 and dust host 8.8.8.8" 4

If you have no  packets being sent and on the same  interval, the LinkMon is not functional. With out a logged event  that makes it harder to isolate if the monitor is working 100% 

ken

[rwpatterson]

diag sniffer packet port1 "src host XXX.XXX.XXX.170 and dest host 8.8.8.8" 4

I believe there was a typo here

[AfonsoAndrade]

Ok,

I did test, follow the result:

### diag sniffer packet port1 "src host XXX.XXX.XXX.170 and dst host 8.8.8.8" 4

856.325075 port1 -- XXX.XXX.XXX.170.54102 -> 8.8.8.8.53: udp 39 856.528675 port1 -- XXX.XXX.XXX.170 -> 8.8.8.8: icmp: echo request 856.749863 port1 -- XXX.XXX.XXX.170.58123 -> 8.8.8.8.53: udp 52 856.750880 port1 -- XXX.XXX.XXX.170.58888 -> 8.8.8.8.53: udp 52 856.800530 port1 -- XXX.XXX.XXX.170.59035 -> 8.8.8.8.53: udp 59 857.440236 port1 -- XXX.XXX.XXX.170.59069 -> 8.8.8.8.53: udp 44 857.933399 port1 -- XXX.XXX.XXX.170.58253 -> 8.8.8.8.53: udp 51 858.123693 port1 -- XXX.XXX.XXX.170.60522 -> 8.8.8.8.53: udp 47 858.792391 port1 -- XXX.XXX.XXX.170.59030 -> 8.8.8.8.53: udp 42 858.810396 port1 -- XXX.XXX.XXX.170.59889 -> 8.8.8.8.53: udp 45 858.873365 port1 -- XXX.XXX.XXX.170.59447 -> 8.8.8.8.53: udp 61 858.887392 port1 -- XXX.XXX.XXX.170.60205 -> 8.8.8.8.53: udp 59 858.964232 port1 -- XXX.XXX.XXX.170.59174 -> 8.8.8.8.53: udp 46 859.319181 port1 -- XXX.XXX.XXX.170.57249 -> 8.8.8.8.53: udp 63 859.325328 port1 -- XXX.XXX.XXX.170.52361 -> 8.8.8.8.53: udp 49 859.703808 port1 -- XXX.XXX.XXX.170.60059 -> 8.8.8.8.53: udp 40 859.726694 port1 -- XXX.XXX.XXX.170.59628 -> 8.8.8.8.53: udp 47 859.750761 port1 -- XXX.XXX.XXX.170.58608 -> 8.8.8.8.53: udp 37

<<<<< I disconnected the link in the interface Port1 >>>>>>>

1139.470509 port1 -- XXX.XXX.XXX.170.56964 -> 8.8.8.8.53: udp 46 1139.473081 port1 -- XXX.XXX.XXX.170.60515 -> 8.8.8.8.53: udp 54 1139.473235 port1 -- XXX.XXX.XXX.170.58652 -> 8.8.8.8.53: udp 54 1139.473262 port1 -- XXX.XXX.XXX.170.58383 -> 8.8.8.8.53: udp 63 1139.475072 port1 -- XXX.XXX.XXX.170.52944 -> 8.8.8.8.53: udp 34 1139.475094 port1 -- XXX.XXX.XXX.170.51938 -> 8.8.8.8.53: udp 45 1139.475360 port1 -- XXX.XXX.XXX.170.59863 -> 8.8.8.8.53: udp 45 1139.475378 port1 -- XXX.XXX.XXX.170.53629 -> 8.8.8.8.53: udp 29 1139.475422 port1 -- XXX.XXX.XXX.170.51836 -> 8.8.8.8.53: udp 39 1139.475472 port1 -- XXX.XXX.XXX.170.53176 -> 8.8.8.8.53: udp 35 1139.475543 port1 -- XXX.XXX.XXX.170.52820 -> 8.8.8.8.53: udp 36 1139.475620 port1 -- XXX.XXX.XXX.170.53260 -> 8.8.8.8.53: udp 51 1139.475650 port1 -- XXX.XXX.XXX.170.52415 -> 8.8.8.8.53: udp 41 1139.475671 port1 -- XXX.XXX.XXX.170.52688 -> 8.8.8.8.53: udp 26 1139.475738 port1 -- XXX.XXX.XXX.170.52525 -> 8.8.8.8.53: udp 33 1139.475808 port1 -- XXX.XXX.XXX.170.53969 -> 8.8.8.8.53: udp 37 1139.475834 port1 -- XXX.XXX.XXX.170.53893 -> 8.8.8.8.53: udp 36 1139.524702 port1 -- XXX.XXX.XXX.170 -> 8.8.8.8: icmp: echo request

### Ping to port1 (on Windows), IP XXX.XXX.XXX.170:

Resposta de XXX.XXX.XXX.170: bytes=32 tempo=11ms TTL=24 Resposta de XXX.XXX.XXX.170: bytes=32 tempo=5ms TTL=246 Resposta de XXX.XXX.XXX.170: bytes=32 tempo=5ms TTL=246 Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Esgotado o tempo limite do pedido. Resposta de XXX.XXX.XXX.170: bytes=32 tempo=5ms TTL=246 Resposta de XXX.XXX.XXX.170: bytes=32 tempo=5ms TTL=246 Resposta de XXX.XXX.XXX.170: bytes=32 tempo=7ms TTL=246 Resposta de XXX.XXX.XXX.170: bytes=32 tempo=5ms TTL=246

### Command Line: diagnose sys link-monitor interface port1

Interface(port1): state(up, since Wed Jan 4 16:07:38 2017 ), bandwidth(83), session count(5828) latency(0.00), jitters(0.00). FG100D-CBBW02 # diagnose sys link-monitor interface port1 Interface(port1): state(up, since Wed Jan 4 16:07:38 2017 ), bandwidth(79), session count(5972) latency(0.00), jitters(0.00). #### I don't know if that help, but follow the rotas with link down:

get router info routing-table details Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default S* 0.0.0.0/0 [50/0] via XXX.XXX.XXX.82, wan2, [30/0] [50/0] via XXX.XXX.XXX.73, wan1, [31/0] [50/0] via XXX.XXX.XXX.169, port1, [32/0] !!!( This router keeping UP)!!!! C 10.50.50.0/29 is directly connected, port13 S 10.70.85.0/24 [10/0] is directly connected, ascenty-wan1 S 10.100.0.0/16 [10/0] via 169.254.248.29, Amazon-IKE-CTBC S 10.200.200.0/22 [10/0] via 192.168.11.248, port16 C 10.254.248.0/21 is directly connected, port14 C 169.254.248.29/32 is directly connected, Amazon-IKE-CTBC C 169.254.248.30/32 is directly connected, Amazon-IKE-CTBC C 177.69.189.80/28 is directly connected, wan2 C 177.99.242.72/29 is directly connected, wan1 C 191.240.145.168/29 is directly connected, port1 C 192.168.10.0/23 is directly connected, port16 is directly connected, port16 S 192.168.13.0/24 [10/0] via 192.168.10.4, port16 S 192.168.60.0/24 [10/0] via 192.168.10.50, port16 S 192.168.70.0/24 [10/0] via 192.168.10.50, port16 S 192.168.80.0/24 [10/0] via 192.168.10.50, port16

[emnoc]

yes that was a typo ;)

So what does the diag sys link status show now?

On the  monitor, when when you disable the  link does it show a failure and logged event ?

e.g

cmd cli

execute  log   filter  reset

execute  log  filter  view-lines  1000

exec  log filter field logdesc "Link monitor status"

exec  log filter  category  1

execute  log  display

It should show the monitor dying and restarting

Make sure to re-exec a "execute  log   filter  reset" after your finish.

Ken

[emnoc]

e.g log output

1: date=2017-01-10 time=07:04:19 logid=0100022922 type=event subtype=system level=notice vd="OPMG" logdesc="Link monitor status" name="NXDNS" interface="port1" probeproto="ping " msg="Link Monitor changes state from failed to ok, protocol: ping " 

2: date=2017-01-10 time=07:04:04 logid=0100022922 type=event subtype=system level=notice vd="OPMG" logdesc="Link monitor status" name="NXDNS" interface="port1" probeproto="ping " msg="Link Monitor changes state from ok to failed, protocol: ping " 

Also from cli provide a show sys link   output so we can see the full configuration .

[AfonsoAndrade]

Yes, 

at log show events in the link down.

### show sys link 

edit "port1" set srcintf "port1" set server "8.8.8.8" "200.221.2.45" set gateway-ip 191.240.145.169 set source-ip 191.240.145.170 set timeout 5 set failtime 3 set recoverytime 3 set update-cascade-interface disable next edit "wan1" set srcintf "wan1" set server "8.8.8.8" "200.221.2.45" set timeout 5 set failtime 3 set recoverytime 3 set update-cascade-interface disable next edit "wan2" set srcintf "wan2" set server "8.8.8.8" "200.221.2.45" set timeout 5 set failtime 3 set recoverytime 3 set update-cascade-interface disable next end

[AfonsoAndrade]

image Port1